2024-04-12T04:46:13.588 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-12T04:46:13.588 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627 --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627/orig.config.yaml 2024-04-12T04:46:13.646 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T04:46:13.740 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627 branch: reef-release description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: null first_in_suite: false job_id: '7652627' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific ' name: yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: reef-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef-release sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2064 sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: reef-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 targets: smithi113.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOuzn2QoBu1JU0pSxKraKaKiJGW6M8Q2m/AYPeBQyjrZZlCjDV3uF0Ib9qe3ZfRBGzHzH5X/+dMe/TdBp3nWoGE= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJci6ze4WnN5nmpQkZPWaN69kYBi6vSuQ0l7tFn5JES2IS0STsffymi0mVGzvcR+4IwSf1EK4nvAMDIrlmGqDfs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: pacific exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: pacific cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:pacific roleless: true - print: '**** done end installing pacific cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-11_20:32:36 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-12T04:46:13.740 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa; will attempt to use it 2024-04-12T04:46:13.741 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks 2024-04-12T04:46:13.741 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-12T04:46:13.743 INFO:teuthology.task.internal:Checking packages... 2024-04-12T04:46:13.771 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd540ebaca6b131a1dd560e7f69e024b133bbaa42' 2024-04-12T04:46:13.771 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T04:46:13.772 INFO:teuthology.packaging:ref: None 2024-04-12T04:46:13.772 INFO:teuthology.packaging:tag: None 2024-04-12T04:46:13.772 INFO:teuthology.packaging:branch: reef-release 2024-04-12T04:46:13.772 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T04:46:13.772 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef-release 2024-04-12T04:46:13.932 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1187.gd540ebac 2024-04-12T04:46:13.933 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-12T04:46:13.940 INFO:teuthology.task.internal:no buildpackages task found 2024-04-12T04:46:13.940 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-12T04:46:13.948 INFO:teuthology.task.internal:Saving configuration 2024-04-12T04:46:13.972 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-12T04:46:13.978 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T04:46:14.009 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 04:40:26.943143', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOuzn2QoBu1JU0pSxKraKaKiJGW6M8Q2m/AYPeBQyjrZZlCjDV3uF0Ib9qe3ZfRBGzHzH5X/+dMe/TdBp3nWoGE='} 2024-04-12T04:46:14.040 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 04:40:26.944967', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJci6ze4WnN5nmpQkZPWaN69kYBi6vSuQ0l7tFn5JES2IS0STsffymi0mVGzvcR+4IwSf1EK4nvAMDIrlmGqDfs='} 2024-04-12T04:46:14.040 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-12T04:46:14.046 INFO:teuthology.task.internal:roles: ubuntu@smithi113.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-12T04:46:14.046 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-12T04:46:14.046 INFO:teuthology.run_tasks:Running task console_log... 2024-04-12T04:46:14.134 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f00b45690d0>, signals=[15]) 2024-04-12T04:46:14.134 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-12T04:46:14.140 INFO:teuthology.task.internal:Opening connections... 2024-04-12T04:46:14.140 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T04:46:14.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:46:14.243 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-04-12T04:46:14.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:46:14.327 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-12T04:46:14.333 DEBUG:teuthology.orchestra.run.smithi113:> uname -m 2024-04-12T04:46:14.355 INFO:teuthology.orchestra.run.smithi113.stdout:x86_64 2024-04-12T04:46:14.356 DEBUG:teuthology.orchestra.run.smithi113:> cat /etc/os-release 2024-04-12T04:46:14.417 INFO:teuthology.orchestra.run.smithi113.stdout:NAME="CentOS Stream" 2024-04-12T04:46:14.417 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION="8" 2024-04-12T04:46:14.417 INFO:teuthology.orchestra.run.smithi113.stdout:ID="centos" 2024-04-12T04:46:14.418 INFO:teuthology.orchestra.run.smithi113.stdout:ID_LIKE="rhel fedora" 2024-04-12T04:46:14.418 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION_ID="8" 2024-04-12T04:46:14.418 INFO:teuthology.orchestra.run.smithi113.stdout:PLATFORM_ID="platform:el8" 2024-04-12T04:46:14.418 INFO:teuthology.orchestra.run.smithi113.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-12T04:46:14.418 INFO:teuthology.orchestra.run.smithi113.stdout:ANSI_COLOR="0;31" 2024-04-12T04:46:14.418 INFO:teuthology.orchestra.run.smithi113.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-12T04:46:14.418 INFO:teuthology.orchestra.run.smithi113.stdout:HOME_URL="https://centos.org/" 2024-04-12T04:46:14.418 INFO:teuthology.orchestra.run.smithi113.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T04:46:14.418 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-12T04:46:14.418 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-12T04:46:14.419 INFO:teuthology.lock.ops:Updating smithi113.front.sepia.ceph.com on lock server 2024-04-12T04:46:14.450 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-04-12T04:46:14.473 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-04-12T04:46:14.473 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-04-12T04:46:14.534 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="8" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="8" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el8" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-12T04:46:14.535 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-12T04:46:14.536 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-04-12T04:46:14.575 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-12T04:46:14.584 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-12T04:46:14.589 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-12T04:46:14.589 DEBUG:teuthology.orchestra.run.smithi113:> test '!' -e /home/ubuntu/cephtest 2024-04-12T04:46:14.591 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-04-12T04:46:14.612 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-12T04:46:14.618 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-12T04:46:14.618 DEBUG:teuthology.orchestra.run.smithi113:> test -z $(ls -A /var/lib/ceph) 2024-04-12T04:46:14.652 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-04-12T04:46:14.684 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-12T04:46:14.761 INFO:teuthology.run_tasks:Running task kernel... 2024-04-12T04:46:14.776 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-12T04:46:14.776 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-12T04:46:14.776 DEBUG:teuthology.orchestra.run.smithi113:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T04:46:14.777 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T04:46:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T04:46:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T04:46:14.798 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-04-12T04:46:14.798 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2024-04-12T04:46:14.858 INFO:teuthology.orchestra.run.smithi192.stdout:4.18.0-489.el8.x86_64 2024-04-12T04:46:14.858 INFO:teuthology.task.kernel:Running kernel on smithi192: 4.18.0-489.el8.x86_64 2024-04-12T04:46:14.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2024-04-12T04:46:14.859 INFO:teuthology.orchestra.run.smithi113.stdout:4.18.0-489.el8.x86_64 2024-04-12T04:46:14.859 INFO:teuthology.task.kernel:Running kernel on smithi113: 4.18.0-489.el8.x86_64 2024-04-12T04:46:14.860 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum install -y kernel 2024-04-12T04:46:18.807 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - AppStream 21 MB/s | 28 MB 00:01 2024-04-12T04:46:18.878 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - AppStream 21 MB/s | 28 MB 00:01 2024-04-12T04:46:24.687 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-04-12T04:46:24.715 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-04-12T04:46:26.774 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - Extras 40 kB/s | 18 kB 00:00 2024-04-12T04:46:26.862 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2024-04-12T04:46:27.188 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - Extras common packages 36 kB/s | 7.7 kB 00:00 2024-04-12T04:46:27.233 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 7.7 kB 00:00 2024-04-12T04:46:27.415 INFO:teuthology.orchestra.run.smithi192.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-04-12T04:46:27.459 INFO:teuthology.orchestra.run.smithi113.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-04-12T04:46:28.440 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2024-04-12T04:46:29.010 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-12T04:46:32.273 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 530 kB/s | 24 kB 00:00 2024-04-12T04:46:32.734 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 431 kB/s | 24 kB 00:00 2024-04-12T04:46:35.304 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-12T04:46:35.304 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-12T04:46:35.423 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:46:35.424 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:46:35.425 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-12T04:46:35.425 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:46:35.425 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:46:35.425 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-12T04:46:35.425 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-12T04:46:35.425 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-12T04:46:35.425 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-12T04:46:35.425 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:46:35.426 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:46:35.426 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:46:35.426 INFO:teuthology.orchestra.run.smithi192.stdout:Install 3 Packages 2024-04-12T04:46:35.426 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:46:35.426 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 90 M 2024-04-12T04:46:35.426 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 96 M 2024-04-12T04:46:35.426 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:46:35.806 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-12T04:46:35.806 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-12T04:46:35.939 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout:Install 3 Packages 2024-04-12T04:46:35.940 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:46:35.941 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 90 M 2024-04-12T04:46:35.941 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 96 M 2024-04-12T04:46:35.941 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:46:36.352 INFO:teuthology.orchestra.run.smithi192.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 14 MB/s | 10 MB 00:00 2024-04-12T04:46:37.027 INFO:teuthology.orchestra.run.smithi113.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 11 MB/s | 10 MB 00:00 2024-04-12T04:46:37.500 INFO:teuthology.orchestra.run.smithi192.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 19 MB/s | 36 MB 00:01 2024-04-12T04:46:38.144 INFO:teuthology.orchestra.run.smithi113.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 18 MB/s | 36 MB 00:02 2024-04-12T04:46:38.200 INFO:teuthology.orchestra.run.smithi192.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 17 MB/s | 44 MB 00:02 2024-04-12T04:46:38.200 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:46:38.201 INFO:teuthology.orchestra.run.smithi192.stdout:Total 33 MB/s | 90 MB 00:02 2024-04-12T04:46:38.761 INFO:teuthology.orchestra.run.smithi113.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 17 MB/s | 44 MB 00:02 2024-04-12T04:46:38.761 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:46:38.762 INFO:teuthology.orchestra.run.smithi113.stdout:Total 32 MB/s | 90 MB 00:02 2024-04-12T04:46:38.892 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:46:39.330 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:46:39.330 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:46:39.450 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:46:39.866 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:46:39.866 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:46:40.591 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:46:40.613 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:46:41.113 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:46:41.138 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:46:43.390 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:46:43.946 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:46:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-12T04:46:44.482 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-12T04:46:47.470 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-12T04:46:47.895 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-12T04:46:48.050 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-12T04:46:48.440 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-12T04:46:51.328 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-12T04:46:51.635 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-12T04:46:51.873 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-12T04:46:52.162 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-12T04:47:13.664 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-12T04:47:13.690 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-12T04:47:14.135 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-12T04:47:14.159 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-12T04:47:14.322 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-12T04:47:14.322 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-12T04:47:14.322 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-12T04:47:14.505 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-12T04:47:14.505 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-12T04:47:14.505 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-12T04:47:14.836 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-12T04:47:14.836 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:47:14.836 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:47:14.836 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-12T04:47:14.836 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-12T04:47:14.836 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:47:14.836 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:47:14.991 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T04:47:15.026 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-12T04:47:15.026 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:47:15.026 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:47:15.026 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-12T04:47:15.026 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-12T04:47:15.026 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:47:15.026 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:47:15.182 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T04:47:15.238 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-12T04:47:15.239 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-12T04:47:15.239 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-12T04:47:15.239 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-12T04:47:15.239 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-12T04:47:15.239 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-12T04:47:15.239 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2024-04-12T04:47:15.240 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T04:47:15.442 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-12T04:47:15.443 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-12T04:47:15.443 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-12T04:47:15.443 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-12T04:47:15.443 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-12T04:47:15.443 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-12T04:47:15.443 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi113.front.sepia.ceph.com, path=None, version=distro) 2024-04-12T04:47:15.443 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T04:47:15.451 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-12T04:47:15.452 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-12T04:47:15.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo rpm -qi grub2-tools 2024-04-12T04:47:15.485 INFO:teuthology.orchestra.run.smithi192.stdout:Name : grub2-tools 2024-04-12T04:47:15.485 INFO:teuthology.orchestra.run.smithi192.stdout:Epoch : 1 2024-04-12T04:47:15.485 INFO:teuthology.orchestra.run.smithi192.stdout:Version : 2.02 2024-04-12T04:47:15.485 INFO:teuthology.orchestra.run.smithi192.stdout:Release : 129.el8 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: x86_64 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Group : System Environment/Base 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Size : 9265586 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:License : GPLv3+ 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Relocations : (not relocatable) 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Packager : CentOS BuildSystem 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Vendor : CentOS 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-12T04:47:15.486 INFO:teuthology.orchestra.run.smithi192.stdout:Summary : Support tools for GRUB. 2024-04-12T04:47:15.487 INFO:teuthology.orchestra.run.smithi192.stdout:Description : 2024-04-12T04:47:15.487 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:47:15.487 INFO:teuthology.orchestra.run.smithi192.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-12T04:47:15.487 INFO:teuthology.orchestra.run.smithi192.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-12T04:47:15.487 INFO:teuthology.orchestra.run.smithi192.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-12T04:47:15.487 INFO:teuthology.orchestra.run.smithi192.stdout:hardware devices. 2024-04-12T04:47:15.487 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:47:15.487 INFO:teuthology.orchestra.run.smithi192.stdout:This subpackage provides tools for support of all platforms. 2024-04-12T04:47:15.488 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-12T04:47:15.488 INFO:teuthology.task.kernel:Updating grub on smithi192 to boot 4.18.0-552.el8.x86_64 2024-04-12T04:47:15.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-12T04:47:15.654 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-12T04:47:15.654 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-12T04:47:15.654 DEBUG:teuthology.orchestra.run.smithi113:> sudo rpm -qi grub2-tools 2024-04-12T04:47:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:Name : grub2-tools 2024-04-12T04:47:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:Epoch : 1 2024-04-12T04:47:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:Version : 2.02 2024-04-12T04:47:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:Release : 129.el8 2024-04-12T04:47:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:Architecture: x86_64 2024-04-12T04:47:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-12T04:47:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:Group : System Environment/Base 2024-04-12T04:47:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:Size : 9265586 2024-04-12T04:47:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:License : GPLv3+ 2024-04-12T04:47:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-12T04:47:15.688 INFO:teuthology.orchestra.run.smithi113.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:Relocations : (not relocatable) 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:Packager : CentOS BuildSystem 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:Vendor : CentOS 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:Summary : Support tools for GRUB. 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:Description : 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:hardware devices. 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:47:15.689 INFO:teuthology.orchestra.run.smithi113.stdout:This subpackage provides tools for support of all platforms. 2024-04-12T04:47:15.690 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-12T04:47:15.690 INFO:teuthology.task.kernel:Updating grub on smithi113 to boot 4.18.0-552.el8.x86_64 2024-04-12T04:47:15.691 DEBUG:teuthology.orchestra.run.smithi113:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-12T04:47:16.940 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2024-04-12T04:47:17.491 INFO:teuthology.orchestra.run.smithi113.stderr:Generating grub configuration file ... 2024-04-12T04:47:19.204 INFO:teuthology.orchestra.run.smithi192.stderr:done 2024-04-12T04:47:19.206 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-04-12T04:47:19.236 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.I4D568xb21 2024-04-12T04:47:19.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.I4D568xb21 2024-04-12T04:47:19.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.I4D568xb21 2024-04-12T04:47:19.435 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.I4D568xb21 is 6KB 2024-04-12T04:47:19.477 INFO:teuthology.orchestra.run.smithi113.stderr:done 2024-04-12T04:47:19.479 DEBUG:teuthology.orchestra.run.smithi113:> mktemp 2024-04-12T04:47:19.483 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.I4D568xb21 2024-04-12T04:47:19.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo /bin/ls /boot/loader/entries || true 2024-04-12T04:47:19.502 INFO:teuthology.orchestra.run.smithi113.stdout:/tmp/tmp.QSYv8PxQ2l 2024-04-12T04:47:19.502 DEBUG:teuthology.orchestra.run.smithi113:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QSYv8PxQ2l 2024-04-12T04:47:19.569 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-12T04:47:19.569 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-12T04:47:19.569 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-12T04:47:19.569 INFO:teuthology.orchestra.run.smithi192.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-12T04:47:19.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-12T04:47:19.575 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0666 /tmp/tmp.QSYv8PxQ2l 2024-04-12T04:47:19.712 DEBUG:teuthology.orchestra.remote:smithi113:/tmp/tmp.QSYv8PxQ2l is 6KB 2024-04-12T04:47:19.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2024-04-12T04:47:19.760 DEBUG:teuthology.orchestra.run.smithi113:> rm -fr /tmp/tmp.QSYv8PxQ2l 2024-04-12T04:47:19.777 DEBUG:teuthology.orchestra.run.smithi113:> sudo /bin/ls /boot/loader/entries || true 2024-04-12T04:47:19.856 INFO:teuthology.orchestra.run.smithi113.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-12T04:47:19.856 INFO:teuthology.orchestra.run.smithi113.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-12T04:47:19.856 INFO:teuthology.orchestra.run.smithi113.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-12T04:47:19.856 INFO:teuthology.orchestra.run.smithi113.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-12T04:47:19.857 DEBUG:teuthology.orchestra.run.smithi113:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-12T04:47:20.024 DEBUG:teuthology.orchestra.run.smithi113:> sudo shutdown -r now 2024-04-12T04:47:49.763 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-12T04:47:49.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-12T04:47:49.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:47:50.027 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-12T04:47:50.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-12T04:47:50.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:48:08.223 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-04-12T04:48:08.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-04-12T04:48:17.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-12T04:48:17.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:48:17.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-12T04:48:17.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:48:20.322 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-04-12T04:48:20.546 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-04-12T04:48:32.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-12T04:48:32.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:48:32.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-12T04:48:32.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:48:35.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-04-12T04:48:50.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-12T04:48:50.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:49:04.786 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-04-12T04:49:06.001 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-12T04:49:06.001 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-552.el8.x86_64"... 2024-04-12T04:49:06.001 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-04-12T04:49:06.031 INFO:teuthology.orchestra.run.smithi192.stdout:4.18.0-552.el8.x86_64 2024-04-12T04:49:06.031 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-12T04:49:06.031 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-12T04:49:06.031 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-12T04:49:07.032 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-12T04:49:07.032 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-12T04:49:07.088 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-04-12T04:49:07.112 DEBUG:teuthology.parallel:result is None 2024-04-12T04:49:09.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-04-12T04:49:27.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-12T04:49:27.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:49:27.598 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-04-12T04:49:27.870 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-12T04:49:27.870 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-552.el8.x86_64"... 2024-04-12T04:49:27.870 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2024-04-12T04:49:27.889 INFO:teuthology.orchestra.run.smithi113.stdout:4.18.0-552.el8.x86_64 2024-04-12T04:49:27.889 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-12T04:49:27.889 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-12T04:49:27.889 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-12T04:49:28.890 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-12T04:49:28.890 DEBUG:teuthology.orchestra.run.smithi113:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-12T04:49:28.955 INFO:teuthology.orchestra.run.smithi113.stdout:ttyS1 2024-04-12T04:49:28.971 DEBUG:teuthology.parallel:result is None 2024-04-12T04:49:28.972 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-12T04:49:28.980 INFO:teuthology.task.internal:Creating test directory... 2024-04-12T04:49:28.981 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T04:49:28.983 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T04:49:29.005 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-12T04:49:29.016 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-12T04:49:29.032 INFO:teuthology.task.internal:Creating archive directory... 2024-04-12T04:49:29.033 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T04:49:29.043 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T04:49:29.078 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-12T04:49:29.085 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-12T04:49:29.086 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T04:49:29.116 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T04:49:29.157 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T04:49:29.166 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T04:49:29.168 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T04:49:29.178 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T04:49:29.179 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-12T04:49:29.188 INFO:teuthology.task.internal:Configuring sudo... 2024-04-12T04:49:29.188 DEBUG:teuthology.orchestra.run.smithi113:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T04:49:29.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T04:49:29.244 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-12T04:49:29.255 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-12T04:49:29.255 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T04:49:29.282 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T04:49:29.304 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T04:49:29.394 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T04:49:29.438 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T04:49:29.533 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T04:49:29.566 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T04:49:29.566 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T04:49:29.632 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T04:49:29.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T04:49:29.736 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T04:49:29.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T04:49:29.865 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T04:49:29.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T04:49:29.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo service rsyslog restart 2024-04-12T04:49:29.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-04-12T04:49:29.989 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T04:49:30.045 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T04:49:30.470 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-12T04:49:30.477 INFO:teuthology.task.internal:Starting timer... 2024-04-12T04:49:30.477 INFO:teuthology.run_tasks:Running task pcp... 2024-04-12T04:49:30.488 INFO:teuthology.run_tasks:Running task selinux... 2024-04-12T04:49:30.498 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-12T04:49:30.498 DEBUG:teuthology.orchestra.run.smithi113:> sudo service auditd rotate 2024-04-12T04:49:30.625 INFO:teuthology.orchestra.run.smithi113.stdout:Rotating logs: 2024-04-12T04:49:30.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-04-12T04:49:30.724 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-04-12T04:49:30.726 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-12T04:49:30.726 DEBUG:teuthology.orchestra.run.smithi113:> /usr/sbin/getenforce 2024-04-12T04:49:30.770 INFO:teuthology.orchestra.run.smithi113.stdout:Permissive 2024-04-12T04:49:30.770 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-04-12T04:49:30.824 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-04-12T04:49:30.825 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi113.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-04-12T04:49:30.825 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T04:49:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T04:49:30.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T04:49:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T04:49:30.881 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-12T04:49:30.881 DEBUG:teuthology.orchestra.run.smithi113:> sudo /usr/sbin/setenforce permissive 2024-04-12T04:49:30.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-04-12T04:49:30.948 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-12T04:49:30.957 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-12T04:49:30.973 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-12T04:49:30.974 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi113.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-12T04:55:20.085 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi113.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-04-12T04:55:20.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-12T04:55:20.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:55:20.170 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-04-12T04:55:20.236 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-04-12T04:55:20.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-12T04:55:20.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T04:55:20.313 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-04-12T04:55:20.380 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-12T04:55:20.381 INFO:teuthology.run_tasks:Running task clock... 2024-04-12T04:55:20.391 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-12T04:55:20.391 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T04:55:20.391 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T04:55:20.393 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T04:55:20.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T04:55:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T04:55:20.436 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T04:55:20.453 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T04:55:20.467 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T04:55:20.501 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: ntpd: command not found 2024-04-12T04:55:20.511 INFO:teuthology.orchestra.run.smithi113.stdout:506 Cannot talk to daemon 2024-04-12T04:55:20.525 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T04:55:20.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-04-12T04:55:20.538 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T04:55:20.540 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-04-12T04:55:20.553 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T04:55:20.566 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T04:55:20.608 INFO:teuthology.orchestra.run.smithi113.stderr:bash: ntpq: command not found 2024-04-12T04:55:20.612 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T04:55:20.612 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-04-12T04:55:20.612 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T04:55:20.612 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T04:55:20.612 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T04:55:20.612 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T04:55:20.629 INFO:teuthology.orchestra.run.smithi192.stderr:bash: ntpq: command not found 2024-04-12T04:55:20.634 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T04:55:20.634 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-04-12T04:55:20.634 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T04:55:20.634 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T04:55:20.634 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T04:55:20.634 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T04:55:20.635 INFO:teuthology.run_tasks:Running task pexec... 2024-04-12T04:55:20.643 INFO:teuthology.task.pexec:Executing custom commands... 2024-04-12T04:55:20.643 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T04:55:20.644 DEBUG:teuthology.orchestra.run.smithi113:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-12T04:55:20.644 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi192.front.sepia.ceph.com 2024-04-12T04:55:20.644 DEBUG:teuthology.orchestra.run.smithi192:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-12T04:55:21.402 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:23 ago on Fri 12 Apr 2024 04:53:58 AM UTC. 2024-04-12T04:55:21.431 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:23 ago on Fri 12 Apr 2024 04:53:58 AM UTC. 2024-04-12T04:55:23.505 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:55:23.506 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:55:23.506 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-12T04:55:23.506 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:55:23.506 INFO:teuthology.orchestra.run.smithi113.stdout:Resetting modules: 2024-04-12T04:55:23.506 INFO:teuthology.orchestra.run.smithi113.stdout: container-tools 2024-04-12T04:55:23.506 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:55:23.506 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:55:23.506 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:55:23.506 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:55:23.524 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:55:23.525 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:55:23.525 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-12T04:55:23.525 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:55:23.525 INFO:teuthology.orchestra.run.smithi192.stdout:Resetting modules: 2024-04-12T04:55:23.525 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools 2024-04-12T04:55:23.526 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:55:23.526 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:55:23.526 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:55:23.526 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:55:23.767 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:55:23.782 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:55:24.601 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:26 ago on Fri 12 Apr 2024 04:53:58 AM UTC. 2024-04-12T04:55:24.620 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:26 ago on Fri 12 Apr 2024 04:53:58 AM UTC. 2024-04-12T04:55:26.722 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:55:26.744 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:55:26.759 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:55:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-12T04:55:26.759 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout:Installing group/module packages: 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2024-04-12T04:55:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-12T04:55:26.761 INFO:teuthology.orchestra.run.smithi192.stdout:Installing module profiles: 2024-04-12T04:55:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools/common 2024-04-12T04:55:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:55:26.761 INFO:teuthology.orchestra.run.smithi192.stdout:Enabling module streams: 2024-04-12T04:55:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools rhel8 2024-04-12T04:55:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:55:26.761 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:55:26.761 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:55:26.761 INFO:teuthology.orchestra.run.smithi192.stdout:Install 13 Packages 2024-04-12T04:55:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:55:26.762 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 27 M 2024-04-12T04:55:26.762 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 91 M 2024-04-12T04:55:26.762 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:55:26.781 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:55:26.781 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-04-12T04:55:26.781 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:55:26.781 INFO:teuthology.orchestra.run.smithi113.stdout:Installing group/module packages: 2024-04-12T04:55:26.781 INFO:teuthology.orchestra.run.smithi113.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-12T04:55:26.781 INFO:teuthology.orchestra.run.smithi113.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-12T04:55:26.781 INFO:teuthology.orchestra.run.smithi113.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-12T04:55:26.781 INFO:teuthology.orchestra.run.smithi113.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-12T04:55:26.781 INFO:teuthology.orchestra.run.smithi113.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-12T04:55:26.781 INFO:teuthology.orchestra.run.smithi113.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout:Installing weak dependencies: 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout:Installing module profiles: 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout: container-tools/common 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout:Enabling module streams: 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout: container-tools rhel8 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:55:26.782 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:55:26.783 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:55:26.783 INFO:teuthology.orchestra.run.smithi113.stdout:Install 13 Packages 2024-04-12T04:55:26.783 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:55:26.783 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 27 M 2024-04-12T04:55:26.783 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 91 M 2024-04-12T04:55:26.783 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:55:27.255 INFO:teuthology.orchestra.run.smithi192.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.5 MB/s | 689 kB 00:00 2024-04-12T04:55:27.266 INFO:teuthology.orchestra.run.smithi113.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.6 MB/s | 689 kB 00:00 2024-04-12T04:55:27.288 INFO:teuthology.orchestra.run.smithi192.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.1 MB/s | 1.0 MB 00:00 2024-04-12T04:55:27.307 INFO:teuthology.orchestra.run.smithi113.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.2 MB/s | 1.0 MB 00:00 2024-04-12T04:55:27.330 INFO:teuthology.orchestra.run.smithi192.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 502 kB/s | 38 kB 00:00 2024-04-12T04:55:27.340 INFO:teuthology.orchestra.run.smithi113.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 504 kB/s | 38 kB 00:00 2024-04-12T04:55:27.391 INFO:teuthology.orchestra.run.smithi113.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 3.0 MB/s | 258 kB 00:00 2024-04-12T04:55:27.391 INFO:teuthology.orchestra.run.smithi192.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.5 MB/s | 258 kB 00:00 2024-04-12T04:55:27.466 INFO:teuthology.orchestra.run.smithi113.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.3 MB/s | 180 kB 00:00 2024-04-12T04:55:27.473 INFO:teuthology.orchestra.run.smithi192.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.2 MB/s | 180 kB 00:00 2024-04-12T04:55:27.541 INFO:teuthology.orchestra.run.smithi113.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 330 kB/s | 25 kB 00:00 2024-04-12T04:55:27.548 INFO:teuthology.orchestra.run.smithi192.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 331 kB/s | 25 kB 00:00 2024-04-12T04:55:27.600 INFO:teuthology.orchestra.run.smithi113.stdout:(7/13): netavark-1.10.3-1.module_el8+951+32019c 16 MB/s | 4.1 MB 00:00 2024-04-12T04:55:27.657 INFO:teuthology.orchestra.run.smithi192.stdout:(7/13): buildah-1.33.6-2.module_el8+976+1136073 12 MB/s | 9.6 MB 00:00 2024-04-12T04:55:27.682 INFO:teuthology.orchestra.run.smithi192.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 702 kB/s | 94 kB 00:00 2024-04-12T04:55:27.700 INFO:teuthology.orchestra.run.smithi113.stdout:(8/13): buildah-1.33.6-2.module_el8+976+1136073 11 MB/s | 9.6 MB 00:00 2024-04-12T04:55:27.725 INFO:teuthology.orchestra.run.smithi113.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 510 kB/s | 94 kB 00:00 2024-04-12T04:55:27.749 INFO:teuthology.orchestra.run.smithi192.stdout:(9/13): netavark-1.10.3-1.module_el8+951+32019c 9.8 MB/s | 4.1 MB 00:00 2024-04-12T04:55:27.826 INFO:teuthology.orchestra.run.smithi113.stdout:(10/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 20 MB/s | 2.5 MB 00:00 2024-04-12T04:55:27.849 INFO:teuthology.orchestra.run.smithi192.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 526 kB/s | 53 kB 00:00 2024-04-12T04:55:27.851 INFO:teuthology.orchestra.run.smithi113.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 420 kB/s | 53 kB 00:00 2024-04-12T04:55:27.900 INFO:teuthology.orchestra.run.smithi192.stdout:(11/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 12 MB/s | 2.5 MB 00:00 2024-04-12T04:55:27.901 INFO:teuthology.orchestra.run.smithi113.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 545 kB/s | 41 kB 00:00 2024-04-12T04:55:27.924 INFO:teuthology.orchestra.run.smithi192.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 546 kB/s | 41 kB 00:00 2024-04-12T04:55:28.035 INFO:teuthology.orchestra.run.smithi113.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 20 MB/s | 8.8 MB 00:00 2024-04-12T04:55:28.035 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:55:28.035 INFO:teuthology.orchestra.run.smithi113.stdout:Total 22 MB/s | 27 MB 00:01 2024-04-12T04:55:28.092 INFO:teuthology.orchestra.run.smithi192.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 20 MB/s | 8.8 MB 00:00 2024-04-12T04:55:28.092 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:55:28.092 INFO:teuthology.orchestra.run.smithi192.stdout:Total 21 MB/s | 27 MB 00:01 2024-04-12T04:55:28.303 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:55:28.324 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:55:28.325 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:55:28.363 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:55:28.384 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:55:28.384 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:55:28.592 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:55:28.595 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:55:28.648 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:55:28.651 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:55:29.288 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:55:29.341 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:55:30.270 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-12T04:55:30.277 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-12T04:55:30.499 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-12T04:55:30.507 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-12T04:55:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-12T04:55:30.702 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-12T04:55:30.898 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-12T04:55:30.943 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-12T04:55:31.112 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-12T04:55:31.162 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-12T04:55:31.679 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-12T04:55:31.720 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-12T04:55:32.646 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-12T04:55:32.676 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-12T04:55:32.856 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-12T04:55:32.903 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-12T04:55:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-12T04:55:33.149 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-12T04:55:33.544 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-12T04:55:33.711 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-12T04:55:33.764 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-12T04:55:33.961 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-12T04:55:33.998 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-12T04:55:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-12T04:55:34.180 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-12T04:55:34.367 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-12T04:55:34.586 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-12T04:55:34.586 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-12T04:55:34.586 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-12T04:55:34.586 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-12T04:55:34.586 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-12T04:55:34.586 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-12T04:55:34.586 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-12T04:55:34.586 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-12T04:55:34.586 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-12T04:55:34.586 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-12T04:55:34.586 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-12T04:55:34.587 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-12T04:55:34.587 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-12T04:55:34.803 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-12T04:55:34.803 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-12T04:55:34.803 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-12T04:55:34.803 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-12T04:55:34.804 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-12T04:55:34.804 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-12T04:55:34.804 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-12T04:55:34.804 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-12T04:55:34.804 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-12T04:55:34.804 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-12T04:55:34.804 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-12T04:55:34.804 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-12T04:55:34.804 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-12T04:55:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-12T04:55:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-12T04:55:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-12T04:55:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-12T04:55:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-12T04:55:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:55:35.083 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:55:35.333 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-12T04:55:35.334 INFO:teuthology.orchestra.run.smithi113.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-12T04:55:35.335 INFO:teuthology.orchestra.run.smithi113.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-12T04:55:35.335 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:55:35.335 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:55:35.391 DEBUG:teuthology.parallel:result is None 2024-04-12T04:55:35.566 DEBUG:teuthology.parallel:result is None 2024-04-12T04:55:35.566 INFO:teuthology.run_tasks:Running task install... 2024-04-12T04:55:35.574 DEBUG:teuthology.task.install:project ceph 2024-04-12T04:55:35.575 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}} 2024-04-12T04:55:35.575 DEBUG:teuthology.task.install:config {'branch': 'pacific', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'} 2024-04-12T04:55:35.575 INFO:teuthology.task.install:Using flavor: default 2024-04-12T04:55:35.582 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-12T04:55:35.582 INFO:teuthology.task.install:extra packages: [] 2024-04-12T04:55:35.582 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-12T04:55:35.582 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T04:55:35.582 INFO:teuthology.packaging:ref: None 2024-04-12T04:55:35.582 INFO:teuthology.packaging:tag: None 2024-04-12T04:55:35.582 INFO:teuthology.packaging:branch: pacific 2024-04-12T04:55:35.583 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T04:55:35.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-04-12T04:55:35.584 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-12T04:55:35.584 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T04:55:35.585 INFO:teuthology.packaging:ref: None 2024-04-12T04:55:35.585 INFO:teuthology.packaging:tag: None 2024-04-12T04:55:35.585 INFO:teuthology.packaging:branch: pacific 2024-04-12T04:55:35.585 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T04:55:35.585 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-04-12T04:55:35.848 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/ 2024-04-12T04:55:35.848 INFO:teuthology.task.install.rpm:Package version is 16.2.15-0 2024-04-12T04:55:35.888 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-12T04:55:35.888 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T04:55:35.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-12T04:55:35.916 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-12T04:55:35.916 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T04:55:35.917 INFO:teuthology.packaging:ref: None 2024-04-12T04:55:35.917 INFO:teuthology.packaging:tag: None 2024-04-12T04:55:35.917 INFO:teuthology.packaging:branch: pacific 2024-04-12T04:55:35.917 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T04:55:35.917 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-12T04:55:35.940 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/ 2024-04-12T04:55:35.940 INFO:teuthology.task.install.rpm:Package version is 16.2.15-0 2024-04-12T04:55:35.977 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-12T04:55:35.977 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T04:55:35.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-12T04:55:35.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-12T04:55:36.006 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-12T04:55:36.006 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T04:55:36.006 INFO:teuthology.packaging:ref: None 2024-04-12T04:55:36.006 INFO:teuthology.packaging:tag: None 2024-04-12T04:55:36.006 INFO:teuthology.packaging:branch: pacific 2024-04-12T04:55:36.006 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T04:55:36.006 DEBUG:teuthology.orchestra.run.smithi113:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-12T04:55:36.062 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-12T04:55:36.073 DEBUG:teuthology.orchestra.run.smithi113:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-12T04:55:36.129 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-04-12T04:55:36.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-04-12T04:55:36.149 DEBUG:teuthology.orchestra.run.smithi113:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-12T04:55:36.215 INFO:teuthology.orchestra.run.smithi113.stdout:check_obsoletes = 1 2024-04-12T04:55:36.216 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum clean all 2024-04-12T04:55:36.389 INFO:teuthology.orchestra.run.smithi192.stdout:76 files removed 2024-04-12T04:55:36.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw 2024-04-12T04:55:36.482 INFO:teuthology.orchestra.run.smithi113.stdout:76 files removed 2024-04-12T04:55:36.509 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-radosgw 2024-04-12T04:55:36.900 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 970 kB/s | 76 kB 00:00 2024-04-12T04:55:36.994 INFO:teuthology.orchestra.run.smithi113.stdout:ceph packages for x86_64 958 kB/s | 76 kB 00:00 2024-04-12T04:55:37.017 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 192 kB/s | 15 kB 00:00 2024-04-12T04:55:37.111 INFO:teuthology.orchestra.run.smithi113.stdout:ceph noarch packages 193 kB/s | 15 kB 00:00 2024-04-12T04:55:37.118 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-04-12T04:55:37.220 INFO:teuthology.orchestra.run.smithi113.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-04-12T04:55:38.637 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - AppStream 20 MB/s | 28 MB 00:01 2024-04-12T04:55:38.810 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - AppStream 17 MB/s | 28 MB 00:01 2024-04-12T04:55:44.795 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-04-12T04:55:45.052 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - BaseOS 7.6 MB/s | 10 MB 00:01 2024-04-12T04:55:46.979 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2024-04-12T04:55:47.213 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - Extras 32 kB/s | 18 kB 00:00 2024-04-12T04:55:47.429 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 7.7 kB 00:00 2024-04-12T04:55:47.689 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 7.7 kB 00:00 2024-04-12T04:55:48.605 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 8 - PowerTools 4.6 MB/s | 5.0 MB 00:01 2024-04-12T04:55:48.757 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 8 - PowerTools 5.1 MB/s | 5.0 MB 00:00 2024-04-12T04:55:49.724 INFO:teuthology.orchestra.run.smithi192.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-04-12T04:55:49.825 INFO:teuthology.orchestra.run.smithi113.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-04-12T04:55:50.759 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2024-04-12T04:55:51.483 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 16 MB 00:01 2024-04-12T04:55:54.517 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 528 kB/s | 24 kB 00:00 2024-04-12T04:55:55.282 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 378 kB/s | 24 kB 00:00 2024-04-12T04:55:58.131 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:55:58.134 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:55:58.134 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-04-12T04:55:58.134 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:55:58.134 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:55:58.134 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw x86_64 2:16.2.15-0.el8 ceph 12 M 2024-04-12T04:55:58.134 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrading: 2024-04-12T04:55:58.134 INFO:teuthology.orchestra.run.smithi113.stdout: librados2 x86_64 2:16.2.15-0.el8 ceph 3.7 M 2024-04-12T04:55:58.134 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1 x86_64 2:16.2.15-0.el8 ceph 4.2 M 2024-04-12T04:55:58.134 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base x86_64 2:16.2.15-0.el8 ceph 6.1 M 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common x86_64 2:16.2.15-0.el8 ceph 24 M 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux x86_64 2:16.2.15-0.el8 ceph 26 k 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2 x86_64 2:16.2.15-0.el8 ceph 825 k 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1 x86_64 2:16.2.15-0.el8 ceph 494 k 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2 x86_64 2:16.2.15-0.el8 ceph 3.8 M 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse x86_64 2:16.2.15-0.el8 ceph 45 k 2024-04-12T04:55:58.135 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common x86_64 2:16.2.15-0.el8 ceph 99 k 2024-04-12T04:55:58.136 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs x86_64 2:16.2.15-0.el8 ceph 215 k 2024-04-12T04:55:58.136 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-12T04:55:58.136 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados x86_64 2:16.2.15-0.el8 ceph 389 k 2024-04-12T04:55:58.136 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd x86_64 2:16.2.15-0.el8 ceph 369 k 2024-04-12T04:55:58.136 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw x86_64 2:16.2.15-0.el8 ceph 114 k 2024-04-12T04:55:58.136 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:55:58.136 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:55:58.136 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:55:58.136 INFO:teuthology.orchestra.run.smithi113.stdout:Install 21 Packages 2024-04-12T04:55:58.136 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrade 2 Packages 2024-04-12T04:55:58.136 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:55:58.137 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 58 M 2024-04-12T04:55:58.137 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:55:58.694 INFO:teuthology.orchestra.run.smithi113.stdout:(1/23): ceph-base-16.2.15-0.el8.x86_64.rpm 46 MB/s | 6.1 MB 00:00 2024-04-12T04:55:58.719 INFO:teuthology.orchestra.run.smithi113.stdout:(2/23): ceph-selinux-16.2.15-0.el8.x86_64.rpm 1.0 MB/s | 26 kB 00:00 2024-04-12T04:55:58.778 INFO:teuthology.orchestra.run.smithi113.stdout:(3/23): libcephfs2-16.2.15-0.el8.x86_64.rpm 14 MB/s | 825 kB 00:00 2024-04-12T04:55:58.819 INFO:teuthology.orchestra.run.smithi113.stdout:(4/23): libradosstriper1-16.2.15-0.el8.x86_64.r 12 MB/s | 494 kB 00:00 2024-04-12T04:55:58.880 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:55:58.883 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:55:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-12T04:55:58.883 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:55:58.883 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:55:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:16.2.15-0.el8 ceph 12 M 2024-04-12T04:55:58.883 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-04-12T04:55:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:16.2.15-0.el8 ceph 3.7 M 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:16.2.15-0.el8 ceph 4.2 M 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:16.2.15-0.el8 ceph 6.1 M 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:16.2.15-0.el8 ceph 24 M 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:16.2.15-0.el8 ceph 26 k 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:16.2.15-0.el8 ceph 825 k 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:16.2.15-0.el8 ceph 494 k 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:16.2.15-0.el8 ceph 3.8 M 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-12T04:55:58.884 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:16.2.15-0.el8 ceph 45 k 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:16.2.15-0.el8 ceph 99 k 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:16.2.15-0.el8 ceph 215 k 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:16.2.15-0.el8 ceph 389 k 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:16.2.15-0.el8 ceph 369 k 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:16.2.15-0.el8 ceph 114 k 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout:Install 21 Packages 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 2 Packages 2024-04-12T04:55:58.885 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:55:58.886 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 58 M 2024-04-12T04:55:58.886 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:55:58.912 INFO:teuthology.orchestra.run.smithi113.stdout:(5/23): librgw2-16.2.15-0.el8.x86_64.rpm 41 MB/s | 3.8 MB 00:00 2024-04-12T04:55:58.937 INFO:teuthology.orchestra.run.smithi113.stdout:(6/23): python3-ceph-argparse-16.2.15-0.el8.x86 1.8 MB/s | 45 kB 00:00 2024-04-12T04:55:58.962 INFO:teuthology.orchestra.run.smithi113.stdout:(7/23): python3-ceph-common-16.2.15-0.el8.x86_6 3.9 MB/s | 99 kB 00:00 2024-04-12T04:55:58.987 INFO:teuthology.orchestra.run.smithi113.stdout:(8/23): python3-cephfs-16.2.15-0.el8.x86_64.rpm 8.4 MB/s | 215 kB 00:00 2024-04-12T04:55:59.021 INFO:teuthology.orchestra.run.smithi113.stdout:(9/23): python3-rados-16.2.15-0.el8.x86_64.rpm 11 MB/s | 389 kB 00:00 2024-04-12T04:55:59.054 INFO:teuthology.orchestra.run.smithi113.stdout:(10/23): python3-rbd-16.2.15-0.el8.x86_64.rpm 11 MB/s | 369 kB 00:00 2024-04-12T04:55:59.080 INFO:teuthology.orchestra.run.smithi113.stdout:(11/23): python3-rgw-16.2.15-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-04-12T04:55:59.250 INFO:teuthology.orchestra.run.smithi113.stdout:(12/23): ceph-radosgw-16.2.15-0.el8.x86_64.rpm 17 MB/s | 12 MB 00:00 2024-04-12T04:55:59.531 INFO:teuthology.orchestra.run.smithi113.stdout:(13/23): ceph-common-16.2.15-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-04-12T04:55:59.600 INFO:teuthology.orchestra.run.smithi192.stdout:(1/23): ceph-base-16.2.15-0.el8.x86_64.rpm 35 MB/s | 6.1 MB 00:00 2024-04-12T04:55:59.626 INFO:teuthology.orchestra.run.smithi192.stdout:(2/23): ceph-selinux-16.2.15-0.el8.x86_64.rpm 1.0 MB/s | 26 kB 00:00 2024-04-12T04:55:59.659 INFO:teuthology.orchestra.run.smithi192.stdout:(3/23): libcephfs2-16.2.15-0.el8.x86_64.rpm 24 MB/s | 825 kB 00:00 2024-04-12T04:55:59.693 INFO:teuthology.orchestra.run.smithi192.stdout:(4/23): libradosstriper1-16.2.15-0.el8.x86_64.r 15 MB/s | 494 kB 00:00 2024-04-12T04:55:59.731 INFO:teuthology.orchestra.run.smithi113.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.0 MB/s | 677 kB 00:00 2024-04-12T04:55:59.756 INFO:teuthology.orchestra.run.smithi113.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 195 kB/s | 44 kB 00:00 2024-04-12T04:55:59.824 INFO:teuthology.orchestra.run.smithi113.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 452 kB/s | 259 kB 00:00 2024-04-12T04:55:59.826 INFO:teuthology.orchestra.run.smithi192.stdout:(5/23): ceph-radosgw-16.2.15-0.el8.x86_64.rpm 29 MB/s | 12 MB 00:00 2024-04-12T04:55:59.852 INFO:teuthology.orchestra.run.smithi192.stdout:(6/23): python3-ceph-argparse-16.2.15-0.el8.x86 1.7 MB/s | 45 kB 00:00 2024-04-12T04:55:59.877 INFO:teuthology.orchestra.run.smithi192.stdout:(7/23): python3-ceph-common-16.2.15-0.el8.x86_6 3.9 MB/s | 99 kB 00:00 2024-04-12T04:55:59.949 INFO:teuthology.orchestra.run.smithi113.stdout:(17/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 216 kB/s | 47 kB 00:00 2024-04-12T04:55:59.962 INFO:teuthology.orchestra.run.smithi192.stdout:(8/23): python3-cephfs-16.2.15-0.el8.x86_64.rpm 2.5 MB/s | 215 kB 00:00 2024-04-12T04:56:00.178 INFO:teuthology.orchestra.run.smithi192.stdout:(9/23): python3-rados-16.2.15-0.el8.x86_64.rpm 1.8 MB/s | 389 kB 00:00 2024-04-12T04:56:00.211 INFO:teuthology.orchestra.run.smithi192.stdout:(10/23): python3-rbd-16.2.15-0.el8.x86_64.rpm 11 MB/s | 369 kB 00:00 2024-04-12T04:56:00.228 INFO:teuthology.orchestra.run.smithi192.stdout:(11/23): python3-rgw-16.2.15-0.el8.x86_64.rpm 6.8 MB/s | 114 kB 00:00 2024-04-12T04:56:00.257 INFO:teuthology.orchestra.run.smithi113.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 611 kB/s | 306 kB 00:00 2024-04-12T04:56:00.283 INFO:teuthology.orchestra.run.smithi113.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 394 kB/s | 181 kB 00:00 2024-04-12T04:56:00.295 INFO:teuthology.orchestra.run.smithi192.stdout:(12/23): librgw2-16.2.15-0.el8.x86_64.rpm 6.3 MB/s | 3.8 MB 00:00 2024-04-12T04:56:00.300 INFO:teuthology.orchestra.run.smithi113.stdout:(20/23): liboath-2.6.2-3.el8.x86_64.rpm 168 kB/s | 59 kB 00:00 2024-04-12T04:56:00.325 INFO:teuthology.orchestra.run.smithi113.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2024-04-12T04:56:00.417 INFO:teuthology.orchestra.run.smithi113.stdout:(22/23): librados2-16.2.15-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-04-12T04:56:00.476 INFO:teuthology.orchestra.run.smithi113.stdout:(23/23): librbd1-16.2.15-0.el8.x86_64.rpm 24 MB/s | 4.2 MB 00:00 2024-04-12T04:56:00.478 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:56:00.478 INFO:teuthology.orchestra.run.smithi113.stdout:Total 25 MB/s | 58 MB 00:02 2024-04-12T04:56:00.519 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:56:00.565 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:56:00.565 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:56:00.579 INFO:teuthology.orchestra.run.smithi192.stdout:(13/23): ceph-common-16.2.15-0.el8.x86_64.rpm 21 MB/s | 24 MB 00:01 2024-04-12T04:56:00.763 INFO:teuthology.orchestra.run.smithi192.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2024-04-12T04:56:00.796 INFO:teuthology.orchestra.run.smithi192.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 203 kB/s | 44 kB 00:00 2024-04-12T04:56:00.880 INFO:teuthology.orchestra.run.smithi192.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 444 kB/s | 259 kB 00:00 2024-04-12T04:56:00.903 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:56:00.906 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:56:00.947 INFO:teuthology.orchestra.run.smithi192.stdout:(17/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 256 kB/s | 47 kB 00:00 2024-04-12T04:56:01.031 INFO:teuthology.orchestra.run.smithi192.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2024-04-12T04:56:01.089 INFO:teuthology.orchestra.run.smithi192.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2024-04-12T04:56:01.106 INFO:teuthology.orchestra.run.smithi192.stdout:(20/23): liboath-2.6.2-3.el8.x86_64.rpm 370 kB/s | 59 kB 00:00 2024-04-12T04:56:01.132 INFO:teuthology.orchestra.run.smithi192.stdout:(21/23): leveldb-1.22-1.el8.x86_64.rpm 720 kB/s | 181 kB 00:00 2024-04-12T04:56:01.207 INFO:teuthology.orchestra.run.smithi192.stdout:(22/23): librados2-16.2.15-0.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2024-04-12T04:56:01.299 INFO:teuthology.orchestra.run.smithi192.stdout:(23/23): librbd1-16.2.15-0.el8.x86_64.rpm 22 MB/s | 4.2 MB 00:00 2024-04-12T04:56:01.301 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:56:01.302 INFO:teuthology.orchestra.run.smithi192.stdout:Total 24 MB/s | 58 MB 00:02 2024-04-12T04:56:01.345 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:56:01.392 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:56:01.392 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:56:01.749 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:56:01.752 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:56:01.925 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:56:02.150 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-04-12T04:56:02.378 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T04:56:02.762 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T04:56:02.779 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:56:02.987 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librados2-2:16.2.15-0.el8.x86_64 3/25 2024-04-12T04:56:02.989 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-04-12T04:56:03.083 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:16.2.15-0.el8.x86_64 3/25 2024-04-12T04:56:03.181 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T04:56:03.326 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rados-2:16.2.15-0.el8.x86_64 4/25 2024-04-12T04:56:03.520 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs2-2:16.2.15-0.el8.x86_64 5/25 2024-04-12T04:56:03.573 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T04:56:03.747 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:16.2.15-0.el8.x86_64 3/25 2024-04-12T04:56:03.848 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.15-0.el8.x86_64 3/25 2024-04-12T04:56:03.926 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephfs2-2:16.2.15-0.el8.x86_64 5/25 2024-04-12T04:56:04.102 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:16.2.15-0.el8.x86_64 4/25 2024-04-12T04:56:04.103 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librbd1-2:16.2.15-0.el8.x86_64 6/25 2024-04-12T04:56:04.164 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:16.2.15-0.el8.x86_64 6/25 2024-04-12T04:56:04.272 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:16.2.15-0.el8.x86_64 5/25 2024-04-12T04:56:04.412 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-04-12T04:56:04.714 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:16.2.15-0.el8.x86_64 5/25 2024-04-12T04:56:04.876 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-04-12T04:56:04.889 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:16.2.15-0.el8.x86_64 6/25 2024-04-12T04:56:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.15-0.el8.x86_64 6/25 2024-04-12T04:56:05.045 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librgw2-2:16.2.15-0.el8.x86_64 9/25 2024-04-12T04:56:05.131 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librgw2-2:16.2.15-0.el8.x86_64 9/25 2024-04-12T04:56:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-04-12T04:56:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-04-12T04:56:05.968 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-04-12T04:56:06.151 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-argparse-2:16.2.15-0.el8.x86_64 11/25 2024-04-12T04:56:06.273 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:16.2.15-0.el8.x86_64 9/25 2024-04-12T04:56:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:16.2.15-0.el8.x86_64 9/25 2024-04-12T04:56:06.357 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cephfs-2:16.2.15-0.el8.x86_64 12/25 2024-04-12T04:56:06.514 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-04-12T04:56:06.571 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rgw-2:16.2.15-0.el8.x86_64 13/25 2024-04-12T04:56:06.695 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:16.2.15-0.el8.x86_64 11/25 2024-04-12T04:56:06.831 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rbd-2:16.2.15-0.el8.x86_64 14/25 2024-04-12T04:56:06.868 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:16.2.15-0.el8.x86_64 12/25 2024-04-12T04:56:06.994 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libradosstriper1-2:16.2.15-0.el8.x86_64 15/25 2024-04-12T04:56:07.053 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:16.2.15-0.el8.x86_64 13/25 2024-04-12T04:56:07.067 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libradosstriper1-2:16.2.15-0.el8.x86_64 15/25 2024-04-12T04:56:07.271 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:16.2.15-0.el8.x86_64 14/25 2024-04-12T04:56:07.330 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-12T04:56:07.405 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:16.2.15-0.el8.x86_64 15/25 2024-04-12T04:56:07.502 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:16.2.15-0.el8.x86_64 15/25 2024-04-12T04:56:07.511 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-04-12T04:56:07.699 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-12T04:56:07.725 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-12T04:56:07.927 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-04-12T04:56:07.944 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-common-2:16.2.15-0.el8.x86_64 19/25 2024-04-12T04:56:08.120 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-12T04:56:08.272 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:16.2.15-0.el8.x86_64 19/25 2024-04-12T04:56:10.463 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:16.2.15-0.el8.x86_64 20/25 2024-04-12T04:56:10.711 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-common-2:16.2.15-0.el8.x86_64 20/25 2024-04-12T04:56:10.821 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:16.2.15-0.el8.x86_64 20/25 2024-04-12T04:56:11.081 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:16.2.15-0.el8.x86_64 20/25 2024-04-12T04:56:11.424 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:16.2.15-0.el8.x86_64 20/25 2024-04-12T04:56:11.699 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-base-2:16.2.15-0.el8.x86_64 21/25 2024-04-12T04:56:11.805 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:16.2.15-0.el8.x86_64 20/25 2024-04-12T04:56:11.833 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-base-2:16.2.15-0.el8.x86_64 21/25 2024-04-12T04:56:11.976 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-selinux-2:16.2.15-0.el8.x86_64 22/25 2024-04-12T04:56:12.088 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:16.2.15-0.el8.x86_64 21/25 2024-04-12T04:56:12.187 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:16.2.15-0.el8.x86_64 21/25 2024-04-12T04:56:12.338 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:16.2.15-0.el8.x86_64 22/25 2024-04-12T04:56:30.852 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-selinux-2:16.2.15-0.el8.x86_64 22/25 2024-04-12T04:56:30.852 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /sys 2024-04-12T04:56:30.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /proc 2024-04-12T04:56:30.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /dev 2024-04-12T04:56:30.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /run 2024-04-12T04:56:30.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /mnt 2024-04-12T04:56:30.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /var/tmp 2024-04-12T04:56:30.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /home 2024-04-12T04:56:30.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /tmp 2024-04-12T04:56:30.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /dev 2024-04-12T04:56:30.853 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:56:31.462 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:16.2.15-0.el8.x86_64 22/25 2024-04-12T04:56:31.462 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-04-12T04:56:31.462 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-04-12T04:56:31.462 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /dev 2024-04-12T04:56:31.462 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /run 2024-04-12T04:56:31.462 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-04-12T04:56:31.462 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-04-12T04:56:31.462 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-04-12T04:56:31.462 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-04-12T04:56:31.463 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /dev 2024-04-12T04:56:31.463 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:56:32.044 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-radosgw-2:16.2.15-0.el8.x86_64 23/25 2024-04-12T04:56:32.117 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-radosgw-2:16.2.15-0.el8.x86_64 23/25 2024-04-12T04:56:32.117 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T04:56:32.357 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T04:56:32.357 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T04:56:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:16.2.15-0.el8.x86_64 23/25 2024-04-12T04:56:32.729 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:16.2.15-0.el8.x86_64 23/25 2024-04-12T04:56:32.729 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T04:56:32.962 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T04:56:32.962 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T04:56:33.099 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T04:56:33.099 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-base-2:16.2.15-0.el8.x86_64 1/25 2024-04-12T04:56:33.099 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-common-2:16.2.15-0.el8.x86_64 2/25 2024-04-12T04:56:33.099 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-radosgw-2:16.2.15-0.el8.x86_64 3/25 2024-04-12T04:56:33.099 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-selinux-2:16.2.15-0.el8.x86_64 4/25 2024-04-12T04:56:33.099 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs2-2:16.2.15-0.el8.x86_64 5/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libradosstriper1-2:16.2.15-0.el8.x86_64 6/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librgw2-2:16.2.15-0.el8.x86_64 7/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-argparse-2:16.2.15-0.el8.x86_64 8/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-common-2:16.2.15-0.el8.x86_64 9/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cephfs-2:16.2.15-0.el8.x86_64 10/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rados-2:16.2.15-0.el8.x86_64 11/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rbd-2:16.2.15-0.el8.x86_64 12/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rgw-2:16.2.15-0.el8.x86_64 13/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-04-12T04:56:33.100 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-04-12T04:56:33.101 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-04-12T04:56:33.101 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:16.2.15-0.el8.x86_64 22/25 2024-04-12T04:56:33.101 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-12T04:56:33.101 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:16.2.15-0.el8.x86_64 24/25 2024-04-12T04:56:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T04:56:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:16.2.15-0.el8.x86_64 1/25 2024-04-12T04:56:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:16.2.15-0.el8.x86_64 2/25 2024-04-12T04:56:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:16.2.15-0.el8.x86_64 3/25 2024-04-12T04:56:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:16.2.15-0.el8.x86_64 4/25 2024-04-12T04:56:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:16.2.15-0.el8.x86_64 5/25 2024-04-12T04:56:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:16.2.15-0.el8.x86_64 6/25 2024-04-12T04:56:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:16.2.15-0.el8.x86_64 7/25 2024-04-12T04:56:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:16.2.15-0.el8.x86_64 8/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:16.2.15-0.el8.x86_64 9/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:16.2.15-0.el8.x86_64 10/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:16.2.15-0.el8.x86_64 11/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:16.2.15-0.el8.x86_64 12/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:16.2.15-0.el8.x86_64 13/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-04-12T04:56:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-04-12T04:56:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-04-12T04:56:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.15-0.el8.x86_64 22/25 2024-04-12T04:56:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-12T04:56:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.15-0.el8.x86_64 24/25 2024-04-12T04:56:34.050 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T04:56:34.050 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:56:34.050 INFO:teuthology.orchestra.run.smithi113.stdout:Upgraded: 2024-04-12T04:56:34.050 INFO:teuthology.orchestra.run.smithi113.stdout: librados2-2:16.2.15-0.el8.x86_64 librbd1-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.050 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:56:34.050 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: leveldb-1.22-1.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.051 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.052 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-12T04:56:34.052 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.052 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.052 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.052 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:56:34.052 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:56:34.395 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-test 2024-04-12T04:56:34.562 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T04:56:34.562 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:56:34.562 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-04-12T04:56:34.562 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:16.2.15-0.el8.x86_64 librbd1-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.562 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:56:34.562 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.562 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: leveldb-1.22-1.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.563 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.564 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.564 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-12T04:56:34.564 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.564 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.564 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:16.2.15-0.el8.x86_64 2024-04-12T04:56:34.564 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:56:34.564 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:56:34.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test 2024-04-12T04:56:35.130 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:00:41 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:56:35.602 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:40 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:56:36.749 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:56:36.750 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:56:36.750 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-12T04:56:36.750 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:56:36.750 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:56:36.750 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test x86_64 2:16.2.15-0.el8 ceph 52 M 2024-04-12T04:56:36.750 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-04-12T04:56:36.750 INFO:teuthology.orchestra.run.smithi113.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-12T04:56:36.751 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite x86_64 2:16.2.15-0.el8 ceph 174 k 2024-04-12T04:56:36.751 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-12T04:56:36.751 INFO:teuthology.orchestra.run.smithi113.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-12T04:56:36.751 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-12T04:56:36.751 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:56:36.751 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:56:36.751 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:56:36.751 INFO:teuthology.orchestra.run.smithi113.stdout:Install 6 Packages 2024-04-12T04:56:36.751 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:56:36.752 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 53 M 2024-04-12T04:56:36.752 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 197 M 2024-04-12T04:56:36.752 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:56:37.009 INFO:teuthology.orchestra.run.smithi113.stdout:(1/6): libcephsqlite-16.2.15-0.el8.x86_64.rpm 870 kB/s | 174 kB 00:00 2024-04-12T04:56:37.225 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:56:37.226 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:16.2.15-0.el8 ceph 52 M 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:16.2.15-0.el8 ceph 174 k 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:56:37.227 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:56:37.228 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-04-12T04:56:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:56:37.228 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 53 M 2024-04-12T04:56:37.228 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 197 M 2024-04-12T04:56:37.228 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:56:37.353 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): libcephsqlite-16.2.15-0.el8.x86_64.rpm 2.5 MB/s | 174 kB 00:00 2024-04-12T04:56:37.559 INFO:teuthology.orchestra.run.smithi113.stdout:(2/6): ceph-test-16.2.15-0.el8.x86_64.rpm 70 MB/s | 52 MB 00:00 2024-04-12T04:56:37.584 INFO:teuthology.orchestra.run.smithi113.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 262 kB/s | 203 kB 00:00 2024-04-12T04:56:37.685 INFO:teuthology.orchestra.run.smithi113.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 708 kB/s | 71 kB 00:00 2024-04-12T04:56:37.835 INFO:teuthology.orchestra.run.smithi113.stdout:(5/6): oniguruma-6.8.2-3.el8.x86_64.rpm 228 kB/s | 188 kB 00:00 2024-04-12T04:56:37.918 INFO:teuthology.orchestra.run.smithi113.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 901 kB/s | 323 kB 00:00 2024-04-12T04:56:37.919 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:56:37.920 INFO:teuthology.orchestra.run.smithi113.stdout:Total 45 MB/s | 53 MB 00:01 2024-04-12T04:56:37.960 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:56:37.981 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:56:37.981 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:56:38.037 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): ceph-test-16.2.15-0.el8.x86_64.rpm 69 MB/s | 52 MB 00:00 2024-04-12T04:56:38.062 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 262 kB/s | 203 kB 00:00 2024-04-12T04:56:38.087 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 256 kB/s | 188 kB 00:00 2024-04-12T04:56:38.146 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 846 kB/s | 71 kB 00:00 2024-04-12T04:56:38.281 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:56:38.284 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:56:38.420 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 844 kB/s | 323 kB 00:00 2024-04-12T04:56:38.422 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:56:38.422 INFO:teuthology.orchestra.run.smithi192.stdout:Total 44 MB/s | 53 MB 00:01 2024-04-12T04:56:38.463 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:56:38.484 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:56:38.484 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:56:38.781 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:56:38.781 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:56:38.784 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:56:39.038 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-12T04:56:39.226 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-12T04:56:39.290 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:56:39.403 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-12T04:56:39.481 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-12T04:56:39.504 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-12T04:56:39.676 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-12T04:56:39.678 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-12T04:56:39.834 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-12T04:56:39.870 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephsqlite-2:16.2.15-0.el8.x86_64 5/6 2024-04-12T04:56:39.908 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-12T04:56:40.106 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-12T04:56:40.292 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:16.2.15-0.el8.x86_64 5/6 2024-04-12T04:56:44.505 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephsqlite-2:16.2.15-0.el8.x86_64 5/6 2024-04-12T04:56:44.743 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-test-2:16.2.15-0.el8.x86_64 6/6 2024-04-12T04:56:44.895 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:16.2.15-0.el8.x86_64 5/6 2024-04-12T04:56:45.025 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-test-2:16.2.15-0.el8.x86_64 6/6 2024-04-12T04:56:45.025 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-test-2:16.2.15-0.el8.x86_64 1/6 2024-04-12T04:56:45.026 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephsqlite-2:16.2.15-0.el8.x86_64 2/6 2024-04-12T04:56:45.026 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-12T04:56:45.026 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-12T04:56:45.026 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-12T04:56:45.165 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:16.2.15-0.el8.x86_64 6/6 2024-04-12T04:56:45.441 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-12T04:56:45.441 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:56:45.441 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:56:45.441 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test-2:16.2.15-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-12T04:56:45.441 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite-2:16.2.15-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-04-12T04:56:45.441 INFO:teuthology.orchestra.run.smithi113.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-04-12T04:56:45.441 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:56:45.441 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:56:45.451 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-2:16.2.15-0.el8.x86_64 6/6 2024-04-12T04:56:45.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:16.2.15-0.el8.x86_64 1/6 2024-04-12T04:56:45.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:16.2.15-0.el8.x86_64 2/6 2024-04-12T04:56:45.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-12T04:56:45.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-12T04:56:45.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-12T04:56:45.672 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph 2024-04-12T04:56:45.901 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-12T04:56:45.901 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:56:45.901 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:56:45.901 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:16.2.15-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-12T04:56:45.901 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:16.2.15-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-04-12T04:56:45.901 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-04-12T04:56:45.901 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:56:45.902 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:56:46.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph 2024-04-12T04:56:46.405 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:00:52 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:56:46.848 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:51 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:56:48.043 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:56:48.046 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:56:48.046 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-04-12T04:56:48.046 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:56:48.046 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:56:48.046 INFO:teuthology.orchestra.run.smithi113.stdout: ceph x86_64 2:16.2.15-0.el8 ceph 6.3 k 2024-04-12T04:56:48.046 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-04-12T04:56:48.046 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds x86_64 2:16.2.15-0.el8 ceph 2.4 M 2024-04-12T04:56:48.046 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr x86_64 2:16.2.15-0.el8 ceph 1.6 M 2024-04-12T04:56:48.046 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core noarch 2:16.2.15-0.el8 ceph-noarch 245 k 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon x86_64 2:16.2.15-0.el8 ceph 5.2 M 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd x86_64 2:16.2.15-0.el8 ceph 19 M 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-12T04:56:48.047 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-12T04:56:48.048 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-12T04:56:48.049 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-12T04:56:48.049 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:56:48.049 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:56:48.049 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:56:48.049 INFO:teuthology.orchestra.run.smithi113.stdout:Install 32 Packages 2024-04-12T04:56:48.049 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:56:48.049 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 32 M 2024-04-12T04:56:48.050 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 104 M 2024-04-12T04:56:48.050 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:56:48.492 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:56:48.495 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:56:48.495 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-12T04:56:48.495 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:56:48.495 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:16.2.15-0.el8 ceph 6.3 k 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:16.2.15-0.el8 ceph 2.4 M 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:16.2.15-0.el8 ceph 1.6 M 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:16.2.15-0.el8 ceph-noarch 245 k 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:16.2.15-0.el8 ceph 5.2 M 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:16.2.15-0.el8 ceph 19 M 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-12T04:56:48.496 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-12T04:56:48.497 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-12T04:56:48.498 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-12T04:56:48.498 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-12T04:56:48.498 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-12T04:56:48.498 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:56:48.498 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:56:48.498 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:56:48.498 INFO:teuthology.orchestra.run.smithi192.stdout:Install 32 Packages 2024-04-12T04:56:48.498 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:56:48.499 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 32 M 2024-04-12T04:56:48.499 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 104 M 2024-04-12T04:56:48.499 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:56:48.579 INFO:teuthology.orchestra.run.smithi113.stdout:(1/32): ceph-16.2.15-0.el8.x86_64.rpm 142 kB/s | 6.3 kB 00:00 2024-04-12T04:56:48.680 INFO:teuthology.orchestra.run.smithi113.stdout:(2/32): ceph-mon-16.2.15-0.el8.x86_64.rpm 52 MB/s | 5.2 MB 00:00 2024-04-12T04:56:48.713 INFO:teuthology.orchestra.run.smithi113.stdout:(3/32): ceph-mgr-16.2.15-0.el8.x86_64.rpm 9.1 MB/s | 1.6 MB 00:00 2024-04-12T04:56:48.755 INFO:teuthology.orchestra.run.smithi113.stdout:(4/32): ceph-mds-16.2.15-0.el8.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-04-12T04:56:48.780 INFO:teuthology.orchestra.run.smithi113.stdout:(5/32): ceph-mgr-modules-core-16.2.15-0.el8.noa 3.6 MB/s | 245 kB 00:00 2024-04-12T04:56:48.856 INFO:teuthology.orchestra.run.smithi192.stdout:(1/32): ceph-16.2.15-0.el8.x86_64.rpm 107 kB/s | 6.3 kB 00:00 2024-04-12T04:56:48.907 INFO:teuthology.orchestra.run.smithi192.stdout:(2/32): ceph-mgr-16.2.15-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-12T04:56:48.957 INFO:teuthology.orchestra.run.smithi192.stdout:(3/32): ceph-mds-16.2.15-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-04-12T04:56:48.991 INFO:teuthology.orchestra.run.smithi192.stdout:(4/32): ceph-mgr-modules-core-16.2.15-0.el8.noa 7.2 MB/s | 245 kB 00:00 2024-04-12T04:56:49.048 INFO:teuthology.orchestra.run.smithi113.stdout:(6/32): ceph-osd-16.2.15-0.el8.x86_64.rpm 51 MB/s | 19 MB 00:00 2024-04-12T04:56:49.066 INFO:teuthology.orchestra.run.smithi192.stdout:(5/32): ceph-mon-16.2.15-0.el8.x86_64.rpm 25 MB/s | 5.2 MB 00:00 2024-04-12T04:56:49.072 INFO:teuthology.orchestra.run.smithi113.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 133 kB/s | 39 kB 00:00 2024-04-12T04:56:49.107 INFO:teuthology.orchestra.run.smithi113.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 447 kB/s | 157 kB 00:00 2024-04-12T04:56:49.198 INFO:teuthology.orchestra.run.smithi113.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 454 kB/s | 57 kB 00:00 2024-04-12T04:56:49.299 INFO:teuthology.orchestra.run.smithi113.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.3 MB/s | 457 kB 00:00 2024-04-12T04:56:49.324 INFO:teuthology.orchestra.run.smithi192.stdout:(6/32): ceph-osd-16.2.15-0.el8.x86_64.rpm 45 MB/s | 19 MB 00:00 2024-04-12T04:56:49.332 INFO:teuthology.orchestra.run.smithi113.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 364 kB/s | 103 kB 00:00 2024-04-12T04:56:49.350 INFO:teuthology.orchestra.run.smithi192.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 137 kB/s | 39 kB 00:00 2024-04-12T04:56:49.375 INFO:teuthology.orchestra.run.smithi192.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 408 kB/s | 157 kB 00:00 2024-04-12T04:56:49.484 INFO:teuthology.orchestra.run.smithi192.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 426 kB/s | 57 kB 00:00 2024-04-12T04:56:49.508 INFO:teuthology.orchestra.run.smithi113.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 805 kB/s | 249 kB 00:00 2024-04-12T04:56:49.559 INFO:teuthology.orchestra.run.smithi192.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.4 MB/s | 457 kB 00:00 2024-04-12T04:56:49.567 INFO:teuthology.orchestra.run.smithi113.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 466 kB/s | 109 kB 00:00 2024-04-12T04:56:49.601 INFO:teuthology.orchestra.run.smithi192.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 375 kB/s | 103 kB 00:00 2024-04-12T04:56:49.675 INFO:teuthology.orchestra.run.smithi113.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-04-12T04:56:49.768 INFO:teuthology.orchestra.run.smithi192.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 876 kB/s | 249 kB 00:00 2024-04-12T04:56:49.825 INFO:teuthology.orchestra.run.smithi113.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 137 kB/s | 44 kB 00:00 2024-04-12T04:56:49.835 INFO:teuthology.orchestra.run.smithi192.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 467 kB/s | 109 kB 00:00 2024-04-12T04:56:49.918 INFO:teuthology.orchestra.run.smithi192.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 290 kB/s | 44 kB 00:00 2024-04-12T04:56:49.960 INFO:teuthology.orchestra.run.smithi192.stdout:(15/32): python3-cryptography-3.2.1-7.el8.x86_6 1.6 MB/s | 641 kB 00:00 2024-04-12T04:56:50.036 INFO:teuthology.orchestra.run.smithi192.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 923 kB/s | 185 kB 00:00 2024-04-12T04:56:50.059 INFO:teuthology.orchestra.run.smithi113.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 376 kB/s | 185 kB 00:00 2024-04-12T04:56:50.061 INFO:teuthology.orchestra.run.smithi192.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.2 MB/s | 173 kB 00:00 2024-04-12T04:56:50.086 INFO:teuthology.orchestra.run.smithi192.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2024-04-12T04:56:50.126 INFO:teuthology.orchestra.run.smithi113.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 383 kB/s | 173 kB 00:00 2024-04-12T04:56:50.136 INFO:teuthology.orchestra.run.smithi192.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 233 kB/s | 18 kB 00:00 2024-04-12T04:56:50.151 INFO:teuthology.orchestra.run.smithi113.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2024-04-12T04:56:50.176 INFO:teuthology.orchestra.run.smithi113.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2024-04-12T04:56:50.195 INFO:teuthology.orchestra.run.smithi192.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 449 kB/s | 49 kB 00:00 2024-04-12T04:56:50.212 INFO:teuthology.orchestra.run.smithi192.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 781 kB/s | 59 kB 00:00 2024-04-12T04:56:50.227 INFO:teuthology.orchestra.run.smithi113.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-04-12T04:56:50.246 INFO:teuthology.orchestra.run.smithi192.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.3 MB/s | 384 kB 00:00 2024-04-12T04:56:50.252 INFO:teuthology.orchestra.run.smithi113.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 484 kB/s | 49 kB 00:00 2024-04-12T04:56:50.262 INFO:teuthology.orchestra.run.smithi192.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2024-04-12T04:56:50.288 INFO:teuthology.orchestra.run.smithi192.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2024-04-12T04:56:50.302 INFO:teuthology.orchestra.run.smithi113.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 468 kB/s | 59 kB 00:00 2024-04-12T04:56:50.305 INFO:teuthology.orchestra.run.smithi192.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 318 kB/s | 19 kB 00:00 2024-04-12T04:56:50.330 INFO:teuthology.orchestra.run.smithi192.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 355 kB/s | 24 kB 00:00 2024-04-12T04:56:50.355 INFO:teuthology.orchestra.run.smithi192.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 435 kB/s | 29 kB 00:00 2024-04-12T04:56:50.361 INFO:teuthology.orchestra.run.smithi113.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2024-04-12T04:56:50.389 INFO:teuthology.orchestra.run.smithi192.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 328 kB/s | 27 kB 00:00 2024-04-12T04:56:50.414 INFO:teuthology.orchestra.run.smithi192.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.0 MB/s | 255 kB 00:00 2024-04-12T04:56:50.428 INFO:teuthology.orchestra.run.smithi113.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 149 kB/s | 19 kB 00:00 2024-04-12T04:56:50.440 INFO:teuthology.orchestra.run.smithi192.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.9 MB/s | 251 kB 00:00 2024-04-12T04:56:50.453 INFO:teuthology.orchestra.run.smithi113.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2024-04-12T04:56:50.465 INFO:teuthology.orchestra.run.smithi192.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2024-04-12T04:56:50.478 INFO:teuthology.orchestra.run.smithi113.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 204 kB/s | 24 kB 00:00 2024-04-12T04:56:50.490 INFO:teuthology.orchestra.run.smithi192.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2024-04-12T04:56:50.491 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:56:50.492 INFO:teuthology.orchestra.run.smithi192.stdout:Total 16 MB/s | 32 MB 00:01 2024-04-12T04:56:50.537 INFO:teuthology.orchestra.run.smithi113.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 268 kB/s | 29 kB 00:00 2024-04-12T04:56:50.562 INFO:teuthology.orchestra.run.smithi113.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2024-04-12T04:56:50.570 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:56:50.607 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:56:50.607 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:56:50.612 INFO:teuthology.orchestra.run.smithi113.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.9 MB/s | 255 kB 00:00 2024-04-12T04:56:50.663 INFO:teuthology.orchestra.run.smithi113.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 853 kB/s | 85 kB 00:00 2024-04-12T04:56:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.4 MB/s | 251 kB 00:00 2024-04-12T04:56:50.730 INFO:teuthology.orchestra.run.smithi113.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 196 kB/s | 23 kB 00:00 2024-04-12T04:56:50.732 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:56:50.732 INFO:teuthology.orchestra.run.smithi113.stdout:Total 12 MB/s | 32 MB 00:02 2024-04-12T04:56:50.813 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:56:50.852 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:56:50.852 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:56:50.918 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:56:50.921 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:56:51.169 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:56:51.173 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:56:52.058 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:56:52.271 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-12T04:56:52.313 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:56:52.531 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-12T04:56:52.548 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-12T04:56:52.798 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-12T04:56:52.833 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-12T04:56:53.021 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-12T04:56:53.110 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-12T04:56:53.228 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-12T04:56:53.357 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-12T04:56:53.408 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-12T04:56:53.555 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-12T04:56:53.576 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-12T04:56:53.754 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-12T04:56:53.768 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-12T04:56:53.961 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-12T04:56:53.969 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-12T04:56:54.172 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-12T04:56:54.249 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-12T04:56:54.374 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-12T04:56:54.488 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-12T04:56:54.628 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-12T04:56:54.763 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-12T04:56:54.882 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-12T04:56:55.156 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-12T04:56:55.171 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-12T04:56:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-12T04:56:55.538 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-12T04:56:55.573 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-12T04:56:55.786 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-12T04:56:55.814 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-12T04:56:56.007 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-12T04:56:56.040 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-12T04:56:56.282 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-12T04:56:56.366 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-12T04:56:56.517 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-12T04:56:56.587 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-12T04:56:56.800 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-12T04:56:56.845 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-12T04:56:56.978 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-12T04:56:57.097 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-12T04:56:57.228 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-12T04:56:57.294 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-12T04:56:57.433 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-12T04:56:57.505 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-12T04:56:57.641 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-12T04:56:57.753 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-12T04:56:57.956 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-12T04:56:57.976 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-12T04:56:58.206 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-12T04:56:58.267 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-12T04:56:58.571 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-12T04:56:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:16.2.15-0.el8.noarch 27/32 2024-04-12T04:56:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:16.2.15-0.el8.x86_64 28/32 2024-04-12T04:56:58.911 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-12T04:56:59.266 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-modules-core-2:16.2.15-0.el8.noarch 27/32 2024-04-12T04:56:59.480 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-2:16.2.15-0.el8.x86_64 28/32 2024-04-12T04:57:00.394 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:16.2.15-0.el8.x86_64 28/32 2024-04-12T04:57:00.548 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:16.2.15-0.el8.x86_64 29/32 2024-04-12T04:57:00.990 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:16.2.15-0.el8.x86_64 29/32 2024-04-12T04:57:01.000 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-2:16.2.15-0.el8.x86_64 28/32 2024-04-12T04:57:01.166 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-osd-2:16.2.15-0.el8.x86_64 29/32 2024-04-12T04:57:01.189 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:16.2.15-0.el8.x86_64 30/32 2024-04-12T04:57:01.405 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:16.2.15-0.el8.x86_64 30/32 2024-04-12T04:57:01.607 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-osd-2:16.2.15-0.el8.x86_64 29/32 2024-04-12T04:57:01.631 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:16.2.15-0.el8.x86_64 31/32 2024-04-12T04:57:01.648 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:16.2.15-0.el8.x86_64 31/32 2024-04-12T04:57:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:16.2.15-0.el8.x86_64 32/32 2024-04-12T04:57:01.790 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mon-2:16.2.15-0.el8.x86_64 30/32 2024-04-12T04:57:02.006 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mon-2:16.2.15-0.el8.x86_64 30/32 2024-04-12T04:57:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-2:16.2.15-0.el8.x86_64 32/32 2024-04-12T04:57:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:16.2.15-0.el8.x86_64 1/32 2024-04-12T04:57:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:16.2.15-0.el8.x86_64 2/32 2024-04-12T04:57:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:16.2.15-0.el8.x86_64 3/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:16.2.15-0.el8.x86_64 4/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:16.2.15-0.el8.x86_64 5/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:16.2.15-0.el8.noarch 6/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-12T04:57:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-12T04:57:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-12T04:57:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-12T04:57:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-12T04:57:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-12T04:57:02.207 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mds-2:16.2.15-0.el8.x86_64 31/32 2024-04-12T04:57:02.224 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mds-2:16.2.15-0.el8.x86_64 31/32 2024-04-12T04:57:02.348 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-2:16.2.15-0.el8.x86_64 32/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-2:16.2.15-0.el8.x86_64 32/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-2:16.2.15-0.el8.x86_64 1/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mds-2:16.2.15-0.el8.x86_64 2/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-2:16.2.15-0.el8.x86_64 3/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mon-2:16.2.15-0.el8.x86_64 4/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-osd-2:16.2.15-0.el8.x86_64 5/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-modules-core-2:16.2.15-0.el8.noarch 6/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-12T04:57:02.768 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-12T04:57:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-12T04:57:02.770 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-12T04:57:02.770 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-12T04:57:02.770 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-12T04:57:02.770 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-12T04:57:02.770 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-12T04:57:02.770 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-12T04:57:02.771 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:16.2.15-0.el8.noarch 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-12T04:57:02.996 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-2.6-1.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-12T04:57:02.997 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-12T04:57:02.998 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-12T04:57:02.998 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-12T04:57:02.998 INFO:teuthology.orchestra.run.smithi192.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-12T04:57:02.998 INFO:teuthology.orchestra.run.smithi192.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-12T04:57:02.998 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-12T04:57:02.998 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-12T04:57:02.998 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-12T04:57:02.998 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-12T04:57:02.998 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:02.998 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:57:03.251 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base 2024-04-12T04:57:03.638 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-12T04:57:03.638 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:03.638 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:57:03.638 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:03.638 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:03.638 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:03.638 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core-2:16.2.15-0.el8.noarch 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-12T04:57:03.639 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend-2.6-1.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:03.640 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:57:03.954 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-base 2024-04-12T04:57:03.990 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:08 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:57:04.690 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:10 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:57:05.561 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:57:05.610 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:57:05.611 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-12T04:57:05.611 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:57:05.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install cephadm 2024-04-12T04:57:06.270 INFO:teuthology.orchestra.run.smithi113.stdout:Package ceph-base-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:57:06.319 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:57:06.320 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-04-12T04:57:06.320 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:57:06.435 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install cephadm 2024-04-12T04:57:06.461 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:11 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:57:07.157 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:13 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:57:08.073 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:57:08.073 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-12T04:57:08.073 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:08.073 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:57:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:16.2.15-0.el8 ceph-noarch 85 k 2024-04-12T04:57:08.073 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:08.073 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:57:08.074 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:08.074 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-12T04:57:08.074 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:08.074 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 85 k 2024-04-12T04:57:08.074 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 352 k 2024-04-12T04:57:08.074 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:57:08.124 INFO:teuthology.orchestra.run.smithi192.stdout:cephadm-16.2.15-0.el8.noarch.rpm 1.7 MB/s | 85 kB 00:00 2024-04-12T04:57:08.125 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:08.125 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.6 MB/s | 85 kB 00:00 2024-04-12T04:57:08.125 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:57:08.132 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:57:08.132 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:57:08.134 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:57:08.136 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:57:08.179 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:57:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:16.2.15-0.el8.noarch 1/1 2024-04-12T04:57:08.793 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm noarch 2:16.2.15-0.el8 ceph-noarch 85 k 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 85 k 2024-04-12T04:57:08.794 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 352 k 2024-04-12T04:57:08.795 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:57:08.841 INFO:teuthology.orchestra.run.smithi113.stdout:cephadm-16.2.15-0.el8.noarch.rpm 1.8 MB/s | 85 kB 00:00 2024-04-12T04:57:08.841 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:08.841 INFO:teuthology.orchestra.run.smithi113.stdout:Total 1.8 MB/s | 85 kB 00:00 2024-04-12T04:57:08.842 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:57:08.848 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:57:08.848 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:57:08.850 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:57:08.853 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:57:08.859 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:16.2.15-0.el8.noarch 1/1 2024-04-12T04:57:08.908 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:57:09.085 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:16.2.15-0.el8.noarch 1/1 2024-04-12T04:57:09.386 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephadm-2:16.2.15-0.el8.noarch 1/1 2024-04-12T04:57:09.478 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:16.2.15-0.el8.noarch 1/1 2024-04-12T04:57:09.478 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:09.479 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:57:09.479 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:16.2.15-0.el8.noarch 2024-04-12T04:57:09.479 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:09.479 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:57:09.599 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephadm-2:16.2.15-0.el8.noarch 1/1 2024-04-12T04:57:09.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache 2024-04-12T04:57:09.823 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephadm-2:16.2.15-0.el8.noarch 1/1 2024-04-12T04:57:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephadm-2:16.2.15-0.el8.noarch 1/1 2024-04-12T04:57:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:10.188 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:57:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm-2:16.2.15-0.el8.noarch 2024-04-12T04:57:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:10.188 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:57:10.426 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-immutable-object-cache 2024-04-12T04:57:10.469 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:15 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:57:11.164 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:17 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:57:12.083 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:57:12.083 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:12.083 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-04-12T04:57:12.084 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:12.084 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:57:12.084 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:16.2.15-0.el8 ceph 161 k 2024-04-12T04:57:12.084 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:12.084 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:57:12.084 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:12.084 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-12T04:57:12.084 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:12.085 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 161 k 2024-04-12T04:57:12.085 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 470 k 2024-04-12T04:57:12.085 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:57:12.253 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-16.2.15-0.el8.x86_6 956 kB/s | 161 kB 00:00 2024-04-12T04:57:12.254 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:12.254 INFO:teuthology.orchestra.run.smithi192.stdout:Total 949 kB/s | 161 kB 00:00 2024-04-12T04:57:12.254 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:57:12.265 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:57:12.265 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:57:12.311 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:57:12.315 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:57:12.478 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:57:12.663 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:57:12.800 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:57:12.800 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repo Size 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache x86_64 2:16.2.15-0.el8 ceph 161 k 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 161 k 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 470 k 2024-04-12T04:57:12.801 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:57:12.846 INFO:teuthology.orchestra.run.smithi113.stdout:ceph-immutable-object-cache-16.2.15-0.el8.x86_6 3.5 MB/s | 161 kB 00:00 2024-04-12T04:57:12.846 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:12.847 INFO:teuthology.orchestra.run.smithi113.stdout:Total 3.5 MB/s | 161 kB 00:00 2024-04-12T04:57:12.847 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:57:12.857 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:57:12.857 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:57:12.903 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:57:12.907 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:57:13.058 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:57:13.060 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:57:13.246 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:57:13.430 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:57:13.430 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:13.430 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:57:13.430 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:13.430 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:13.430 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:57:13.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr 2024-04-12T04:57:13.656 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:57:14.029 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:57:14.030 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:14.030 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:57:14.030 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 2024-04-12T04:57:14.030 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:14.030 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:57:14.268 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-mgr 2024-04-12T04:57:14.376 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:19 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:57:15.001 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:20 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:57:15.955 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-mgr-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:57:16.004 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:57:16.005 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-12T04:57:16.005 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:57:16.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-dashboard 2024-04-12T04:57:16.589 INFO:teuthology.orchestra.run.smithi113.stdout:Package ceph-mgr-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:57:16.637 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:57:16.639 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-04-12T04:57:16.639 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:57:16.752 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-mgr-dashboard 2024-04-12T04:57:16.846 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:21 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:57:17.478 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:23 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:57:18.468 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:57:18.469 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:18.469 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-12T04:57:18.469 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:18.469 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:57:18.469 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:16.2.15-0.el8 ceph-noarch 3.1 M 2024-04-12T04:57:18.469 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-12T04:57:18.470 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:16.2.15-0.el8 ceph-noarch 24 k 2024-04-12T04:57:18.470 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:16.2.15-0.el8 ceph-noarch 15 k 2024-04-12T04:57:18.470 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-12T04:57:18.470 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-12T04:57:18.470 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-12T04:57:18.470 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:18.470 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:57:18.470 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:18.470 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-04-12T04:57:18.470 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:18.471 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.4 M 2024-04-12T04:57:18.471 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 22 M 2024-04-12T04:57:18.471 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:57:19.048 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): ceph-prometheus-alerts-16.2.15-0.el8.noa 230 kB/s | 15 kB 00:00 2024-04-12T04:57:19.065 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): ceph-grafana-dashboards-16.2.15-0.el8.no 299 kB/s | 24 kB 00:00 2024-04-12T04:57:19.107 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:57:19.108 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:19.108 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-04-12T04:57:19.108 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:19.108 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:57:19.108 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard noarch 2:16.2.15-0.el8 ceph-noarch 3.1 M 2024-04-12T04:57:19.108 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-04-12T04:57:19.108 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards noarch 2:16.2.15-0.el8 ceph-noarch 24 k 2024-04-12T04:57:19.109 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts noarch 2:16.2.15-0.el8 ceph-noarch 15 k 2024-04-12T04:57:19.109 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-12T04:57:19.109 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-12T04:57:19.109 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-12T04:57:19.109 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:19.109 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:57:19.109 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:19.109 INFO:teuthology.orchestra.run.smithi113.stdout:Install 6 Packages 2024-04-12T04:57:19.109 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:19.110 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 3.4 M 2024-04-12T04:57:19.110 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 22 M 2024-04-12T04:57:19.110 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:57:19.118 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): ceph-mgr-dashboard-16.2.15-0.el8.noarch. 24 MB/s | 3.1 MB 00:00 2024-04-12T04:57:19.257 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 204 kB/s | 43 kB 00:00 2024-04-12T04:57:19.307 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 139 kB/s | 34 kB 00:00 2024-04-12T04:57:19.349 INFO:teuthology.orchestra.run.smithi113.stdout:(1/6): ceph-grafana-dashboards-16.2.15-0.el8.no 488 kB/s | 24 kB 00:00 2024-04-12T04:57:19.366 INFO:teuthology.orchestra.run.smithi113.stdout:(2/6): ceph-prometheus-alerts-16.2.15-0.el8.noa 222 kB/s | 15 kB 00:00 2024-04-12T04:57:19.449 INFO:teuthology.orchestra.run.smithi113.stdout:(3/6): ceph-mgr-dashboard-16.2.15-0.el8.noarch. 21 MB/s | 3.1 MB 00:00 2024-04-12T04:57:19.483 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 542 kB/s | 196 kB 00:00 2024-04-12T04:57:19.486 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:19.486 INFO:teuthology.orchestra.run.smithi192.stdout:Total 3.4 MB/s | 3.4 MB 00:01 2024-04-12T04:57:19.498 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:57:19.509 INFO:teuthology.orchestra.run.smithi113.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2024-04-12T04:57:19.509 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:57:19.509 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:57:19.542 INFO:teuthology.orchestra.run.smithi113.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.1 MB/s | 196 kB 00:00 2024-04-12T04:57:19.584 INFO:teuthology.orchestra.run.smithi113.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 182 kB/s | 43 kB 00:00 2024-04-12T04:57:19.586 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:19.586 INFO:teuthology.orchestra.run.smithi113.stdout:Total 7.2 MB/s | 3.4 MB 00:00 2024-04-12T04:57:19.596 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:57:19.605 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:57:19.605 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:57:19.669 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:57:19.673 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:57:19.761 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:57:19.764 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:57:19.963 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:57:20.057 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:57:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-12T04:57:20.295 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-12T04:57:20.412 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-12T04:57:20.581 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-12T04:57:20.614 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-12T04:57:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:16.2.15-0.el8.noarch 4/6 2024-04-12T04:57:20.780 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-12T04:57:20.934 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-prometheus-alerts-2:16.2.15-0.el8.noarch 4/6 2024-04-12T04:57:21.788 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:16.2.15-0.el8.noarch 5/6 2024-04-12T04:57:22.055 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-grafana-dashboards-2:16.2.15-0.el8.noarch 5/6 2024-04-12T04:57:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:16.2.15-0.el8.noarch 6/6 2024-04-12T04:57:22.358 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-dashboard-2:16.2.15-0.el8.noarch 6/6 2024-04-12T04:57:22.375 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.15-0.el8.noarch 6/6 2024-04-12T04:57:22.375 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:16.2.15-0.el8.noarch 1/6 2024-04-12T04:57:22.375 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:16.2.15-0.el8.noarch 2/6 2024-04-12T04:57:22.375 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:16.2.15-0.el8.noarch 3/6 2024-04-12T04:57:22.375 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-12T04:57:22.375 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-12T04:57:22.639 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.15-0.el8.noarch 6/6 2024-04-12T04:57:22.640 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-grafana-dashboards-2:16.2.15-0.el8.noarch 1/6 2024-04-12T04:57:22.640 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-dashboard-2:16.2.15-0.el8.noarch 2/6 2024-04-12T04:57:22.640 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-prometheus-alerts-2:16.2.15-0.el8.noarch 3/6 2024-04-12T04:57:22.640 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-12T04:57:22.640 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-12T04:57:22.811 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-12T04:57:22.811 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:22.811 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:57:22.811 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:16.2.15-0.el8.noarch 2024-04-12T04:57:22.811 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:16.2.15-0.el8.noarch 2024-04-12T04:57:22.811 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:16.2.15-0.el8.noarch 2024-04-12T04:57:22.811 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-12T04:57:22.811 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-12T04:57:22.811 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-12T04:57:22.811 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:22.811 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:57:22.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-12T04:57:23.186 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-12T04:57:23.186 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:23.186 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:57:23.186 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards-2:16.2.15-0.el8.noarch 2024-04-12T04:57:23.186 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard-2:16.2.15-0.el8.noarch 2024-04-12T04:57:23.186 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts-2:16.2.15-0.el8.noarch 2024-04-12T04:57:23.186 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-12T04:57:23.186 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-12T04:57:23.186 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-12T04:57:23.187 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:23.187 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:57:23.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-12T04:57:23.698 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:28 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:57:24.243 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:30 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:57:25.323 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:57:25.324 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-04-12T04:57:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-12T04:57:25.324 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.15-0.el8 ceph-noarch 8.6 M 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-04-12T04:57:25.325 INFO:teuthology.orchestra.run.smithi192.stdout:Install 8 Packages 2024-04-12T04:57:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:25.326 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 37 M 2024-04-12T04:57:25.326 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 210 M 2024-04-12T04:57:25.326 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:57:25.721 INFO:teuthology.orchestra.run.smithi192.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.15-0. 56 MB/s | 8.6 MB 00:00 2024-04-12T04:57:25.880 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:57:25.881 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-04-12T04:57:25.881 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.15-0.el8 ceph-noarch 8.6 M 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:57:25.882 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-04-12T04:57:25.883 INFO:teuthology.orchestra.run.smithi113.stdout:Install 8 Packages 2024-04-12T04:57:25.883 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:25.883 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 37 M 2024-04-12T04:57:25.883 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 210 M 2024-04-12T04:57:25.883 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:57:26.304 INFO:teuthology.orchestra.run.smithi192.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.6 MB/s | 4.9 MB 00:00 2024-04-12T04:57:26.388 INFO:teuthology.orchestra.run.smithi192.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 5.8 MB/s | 4.7 MB 00:00 2024-04-12T04:57:26.431 INFO:teuthology.orchestra.run.smithi113.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.15-0. 50 MB/s | 8.6 MB 00:00 2024-04-12T04:57:26.463 INFO:teuthology.orchestra.run.smithi192.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.0 MB/s | 3.7 MB 00:00 2024-04-12T04:57:26.488 INFO:teuthology.orchestra.run.smithi192.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.2 MB/s | 225 kB 00:00 2024-04-12T04:57:26.872 INFO:teuthology.orchestra.run.smithi192.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 29 MB/s | 14 MB 00:00 2024-04-12T04:57:26.897 INFO:teuthology.orchestra.run.smithi192.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 422 kB/s | 172 kB 00:00 2024-04-12T04:57:26.981 INFO:teuthology.orchestra.run.smithi113.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 6.5 MB/s | 4.7 MB 00:00 2024-04-12T04:57:26.998 INFO:teuthology.orchestra.run.smithi192.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2024-04-12T04:57:26.998 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:26.998 INFO:teuthology.orchestra.run.smithi192.stdout:Total 22 MB/s | 37 MB 00:01 2024-04-12T04:57:27.048 INFO:teuthology.orchestra.run.smithi113.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.2 MB/s | 4.9 MB 00:00 2024-04-12T04:57:27.073 INFO:teuthology.orchestra.run.smithi113.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-04-12T04:57:27.140 INFO:teuthology.orchestra.run.smithi113.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.3 MB/s | 3.7 MB 00:00 2024-04-12T04:57:27.229 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:57:27.247 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:57:27.247 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:57:27.399 INFO:teuthology.orchestra.run.smithi113.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 668 kB/s | 172 kB 00:00 2024-04-12T04:57:27.432 INFO:teuthology.orchestra.run.smithi113.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 645 kB 00:00 2024-04-12T04:57:27.593 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:57:27.596 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:57:27.608 INFO:teuthology.orchestra.run.smithi113.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 25 MB/s | 14 MB 00:00 2024-04-12T04:57:27.608 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:27.608 INFO:teuthology.orchestra.run.smithi113.stdout:Total 21 MB/s | 37 MB 00:01 2024-04-12T04:57:27.837 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:57:27.855 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:57:27.855 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:57:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:57:28.195 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:57:28.198 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:57:28.310 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-12T04:57:28.442 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-12T04:57:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-12T04:57:28.763 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:57:29.026 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-12T04:57:29.187 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-12T04:57:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-12T04:57:29.342 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-12T04:57:29.526 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-12T04:57:29.931 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-12T04:57:30.117 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-12T04:57:30.139 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-12T04:57:30.301 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-12T04:57:30.725 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-12T04:57:30.867 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-12T04:57:31.014 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-12T04:57:31.362 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T04:57:31.562 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-12T04:57:31.618 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T04:57:31.878 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T04:57:32.142 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T04:57:33.367 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T04:57:33.887 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T04:57:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-12T04:57:34.883 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.15-0.el8.noarch 8/8 2024-04-12T04:57:35.203 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.15-0.el8.noarch 8/8 2024-04-12T04:57:35.203 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.15-0.el8.noarch 1/8 2024-04-12T04:57:35.203 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-12T04:57:35.203 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-12T04:57:35.204 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-12T04:57:35.204 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T04:57:35.204 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-12T04:57:35.204 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-12T04:57:35.636 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-12T04:57:35.636 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:35.636 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:57:35.636 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:16.2.15-0.el8.noarch 2024-04-12T04:57:35.636 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-12T04:57:35.636 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-12T04:57:35.636 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-12T04:57:35.637 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-12T04:57:35.637 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-12T04:57:35.637 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-12T04:57:35.637 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-12T04:57:35.637 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:35.637 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:57:35.758 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-12T04:57:35.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-rook 2024-04-12T04:57:35.941 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.15-0.el8.noarch 8/8 2024-04-12T04:57:36.291 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.15-0.el8.noarch 8/8 2024-04-12T04:57:36.291 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.15-0.el8.noarch 1/8 2024-04-12T04:57:36.291 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-12T04:57:36.291 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-12T04:57:36.291 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-12T04:57:36.291 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T04:57:36.291 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-12T04:57:36.291 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-12T04:57:36.508 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:41 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:57:36.692 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-12T04:57:36.692 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:36.692 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:57:36.693 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local-2:16.2.15-0.el8.noarch 2024-04-12T04:57:36.693 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-12T04:57:36.693 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-12T04:57:36.693 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-12T04:57:36.693 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-12T04:57:36.693 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-12T04:57:36.693 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-12T04:57:36.693 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-12T04:57:36.693 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:36.693 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:57:36.817 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-mgr-rook 2024-04-12T04:57:37.539 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:43 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:57:38.150 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:57:38.152 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:38.152 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-12T04:57:38.152 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:38.152 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:57:38.152 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:16.2.15-0.el8 ceph-noarch 27 k 2024-04-12T04:57:38.152 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:57:38.153 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:38.154 INFO:teuthology.orchestra.run.smithi192.stdout:Install 13 Packages 2024-04-12T04:57:38.154 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:38.154 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.9 M 2024-04-12T04:57:38.154 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 24 M 2024-04-12T04:57:38.154 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:57:38.655 INFO:teuthology.orchestra.run.smithi192.stdout:(1/13): ceph-mgr-rook-16.2.15-0.el8.noarch.rpm 526 kB/s | 27 kB 00:00 2024-04-12T04:57:38.788 INFO:teuthology.orchestra.run.smithi192.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 108 kB/s | 20 kB 00:00 2024-04-12T04:57:38.813 INFO:teuthology.orchestra.run.smithi192.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 126 kB/s | 27 kB 00:00 2024-04-12T04:57:38.939 INFO:teuthology.orchestra.run.smithi192.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 445 kB/s | 126 kB 00:00 2024-04-12T04:57:38.982 INFO:teuthology.orchestra.run.smithi192.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 567 kB/s | 110 kB 00:00 2024-04-12T04:57:39.073 INFO:teuthology.orchestra.run.smithi192.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 598 kB/s | 155 kB 00:00 2024-04-12T04:57:39.123 INFO:teuthology.orchestra.run.smithi192.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 232 kB/s | 43 kB 00:00 2024-04-12T04:57:39.194 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:57:39.196 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:39.196 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-04-12T04:57:39.196 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:39.196 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:57:39.196 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook noarch 2:16.2.15-0.el8 ceph-noarch 27 k 2024-04-12T04:57:39.196 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-04-12T04:57:39.196 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-12T04:57:39.196 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-12T04:57:39.196 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-12T04:57:39.196 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-12T04:57:39.196 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout:Install 13 Packages 2024-04-12T04:57:39.197 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:39.198 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 1.9 M 2024-04-12T04:57:39.198 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 24 M 2024-04-12T04:57:39.198 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:57:39.207 INFO:teuthology.orchestra.run.smithi192.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 147 kB/s | 33 kB 00:00 2024-04-12T04:57:39.290 INFO:teuthology.orchestra.run.smithi192.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 75 kB/s | 16 kB 00:00 2024-04-12T04:57:39.441 INFO:teuthology.orchestra.run.smithi192.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 260 kB/s | 81 kB 00:00 2024-04-12T04:57:39.466 INFO:teuthology.orchestra.run.smithi192.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 366 kB/s | 64 kB 00:00 2024-04-12T04:57:39.524 INFO:teuthology.orchestra.run.smithi192.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 3.7 MB/s | 1.2 MB 00:00 2024-04-12T04:57:39.549 INFO:teuthology.orchestra.run.smithi192.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 564 kB/s | 61 kB 00:00 2024-04-12T04:57:39.552 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:39.553 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-04-12T04:57:39.609 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:57:39.625 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:57:39.625 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:57:39.696 INFO:teuthology.orchestra.run.smithi113.stdout:(1/13): ceph-mgr-rook-16.2.15-0.el8.noarch.rpm 416 kB/s | 27 kB 00:00 2024-04-12T04:57:39.713 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:57:39.716 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:57:39.825 INFO:teuthology.orchestra.run.smithi113.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 102 kB/s | 20 kB 00:00 2024-04-12T04:57:39.917 INFO:teuthology.orchestra.run.smithi113.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 92 kB/s | 27 kB 00:00 2024-04-12T04:57:39.984 INFO:teuthology.orchestra.run.smithi113.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 438 kB/s | 126 kB 00:00 2024-04-12T04:57:40.010 INFO:teuthology.orchestra.run.smithi113.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 596 kB/s | 110 kB 00:00 2024-04-12T04:57:40.126 INFO:teuthology.orchestra.run.smithi113.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 283 kB/s | 33 kB 00:00 2024-04-12T04:57:40.185 INFO:teuthology.orchestra.run.smithi113.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 212 kB/s | 43 kB 00:00 2024-04-12T04:57:40.218 INFO:teuthology.orchestra.run.smithi113.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 528 kB/s | 155 kB 00:00 2024-04-12T04:57:40.236 INFO:teuthology.orchestra.run.smithi113.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 150 kB/s | 16 kB 00:00 2024-04-12T04:57:40.336 INFO:teuthology.orchestra.run.smithi113.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 541 kB/s | 81 kB 00:00 2024-04-12T04:57:40.370 INFO:teuthology.orchestra.run.smithi113.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 481 kB/s | 64 kB 00:00 2024-04-12T04:57:40.375 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:57:40.403 INFO:teuthology.orchestra.run.smithi113.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 919 kB/s | 61 kB 00:00 2024-04-12T04:57:40.445 INFO:teuthology.orchestra.run.smithi113.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.0 MB/s | 1.2 MB 00:00 2024-04-12T04:57:40.446 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:40.446 INFO:teuthology.orchestra.run.smithi113.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2024-04-12T04:57:40.502 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:57:40.518 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:57:40.518 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:57:40.602 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-12T04:57:40.605 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:57:40.608 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:57:40.824 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-12T04:57:41.027 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-12T04:57:41.152 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:57:41.231 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-12T04:57:41.384 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-12T04:57:41.422 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-12T04:57:41.606 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-12T04:57:41.653 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-12T04:57:41.833 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-12T04:57:41.902 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-12T04:57:42.079 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-12T04:57:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-12T04:57:42.312 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-12T04:57:42.560 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-12T04:57:42.820 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-12T04:57:43.077 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-12T04:57:43.170 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-12T04:57:43.448 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-12T04:57:43.648 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-12T04:57:43.856 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-12T04:57:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:16.2.15-0.el8.noarch 13/13 2024-04-12T04:57:44.113 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-12T04:57:44.302 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.15-0.el8.noarch 13/13 2024-04-12T04:57:44.302 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:16.2.15-0.el8.noarch 1/13 2024-04-12T04:57:44.302 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-12T04:57:44.302 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-12T04:57:44.302 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-12T04:57:44.302 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-12T04:57:44.302 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-12T04:57:44.302 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-12T04:57:44.302 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-12T04:57:44.302 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-12T04:57:44.302 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-12T04:57:44.303 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-12T04:57:44.303 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-12T04:57:44.429 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-12T04:57:44.654 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-12T04:57:44.777 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-12T04:57:44.777 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:44.777 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:57:44.777 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:16.2.15-0.el8.noarch 2024-04-12T04:57:44.777 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-12T04:57:44.777 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-12T04:57:44.777 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-12T04:57:44.777 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-12T04:57:44.778 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-12T04:57:44.778 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-12T04:57:44.778 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-12T04:57:44.778 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-12T04:57:44.778 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-12T04:57:44.778 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-12T04:57:44.778 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-12T04:57:44.778 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-12T04:57:44.778 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:44.778 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:57:44.855 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-12T04:57:44.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-cephadm 2024-04-12T04:57:45.047 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-rook-2:16.2.15-0.el8.noarch 13/13 2024-04-12T04:57:45.340 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.15-0.el8.noarch 13/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-rook-2:16.2.15-0.el8.noarch 1/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-12T04:57:45.341 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-12T04:57:45.621 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:50 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:57:45.795 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-12T04:57:45.795 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:45.795 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:57:45.795 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook-2:16.2.15-0.el8.noarch 2024-04-12T04:57:45.795 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-12T04:57:45.795 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:45.796 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:57:45.927 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-mgr-cephadm 2024-04-12T04:57:46.649 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:52 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:57:47.256 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:16.2.15-0.el8 ceph-noarch 116 k 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-04-12T04:57:47.257 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:47.258 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:57:47.258 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:47.258 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-04-12T04:57:47.258 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:47.258 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 5.9 M 2024-04-12T04:57:47.258 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 24 M 2024-04-12T04:57:47.259 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:57:47.816 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): ceph-mgr-cephadm-16.2.15-0.el8.noarch.rp 2.9 MB/s | 116 kB 00:00 2024-04-12T04:57:47.949 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 159 kB/s | 21 kB 00:00 2024-04-12T04:57:48.141 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.8 MB/s | 355 kB 00:00 2024-04-12T04:57:48.175 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.4 MB/s | 578 kB 00:00 2024-04-12T04:57:48.208 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 733 kB/s | 49 kB 00:00 2024-04-12T04:57:48.256 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:57:48.257 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:48.257 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-04-12T04:57:48.257 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:48.257 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:57:48.257 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm noarch 2:16.2.15-0.el8 ceph-noarch 116 k 2024-04-12T04:57:48.257 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-04-12T04:57:48.257 INFO:teuthology.orchestra.run.smithi113.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-04-12T04:57:48.257 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-12T04:57:48.257 INFO:teuthology.orchestra.run.smithi113.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-04-12T04:57:48.257 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-12T04:57:48.257 INFO:teuthology.orchestra.run.smithi113.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-04-12T04:57:48.258 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:48.258 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:57:48.258 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:48.258 INFO:teuthology.orchestra.run.smithi113.stdout:Install 6 Packages 2024-04-12T04:57:48.258 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:48.259 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 5.9 M 2024-04-12T04:57:48.259 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 24 M 2024-04-12T04:57:48.259 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:57:48.425 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.4 MB/s | 4.8 MB 00:00 2024-04-12T04:57:48.426 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:48.427 INFO:teuthology.orchestra.run.smithi192.stdout:Total 5.0 MB/s | 5.9 MB 00:01 2024-04-12T04:57:48.479 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:57:48.489 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:57:48.489 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:57:48.528 INFO:teuthology.orchestra.run.smithi113.stdout:(1/6): ceph-mgr-cephadm-16.2.15-0.el8.noarch.rp 2.3 MB/s | 116 kB 00:00 2024-04-12T04:57:48.609 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:57:48.612 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:57:48.754 INFO:teuthology.orchestra.run.smithi113.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 94 kB/s | 21 kB 00:00 2024-04-12T04:57:48.883 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:57:48.895 INFO:teuthology.orchestra.run.smithi113.stdout:(3/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.4 MB/s | 578 kB 00:00 2024-04-12T04:57:49.021 INFO:teuthology.orchestra.run.smithi113.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.3 MB/s | 355 kB 00:00 2024-04-12T04:57:49.129 INFO:teuthology.orchestra.run.smithi113.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.3 MB/s | 4.8 MB 00:00 2024-04-12T04:57:49.180 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-04-12T04:57:49.196 INFO:teuthology.orchestra.run.smithi113.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 163 kB/s | 49 kB 00:00 2024-04-12T04:57:49.198 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:49.199 INFO:teuthology.orchestra.run.smithi113.stdout:Total 6.3 MB/s | 5.9 MB 00:00 2024-04-12T04:57:49.252 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:57:49.263 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:57:49.263 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:57:49.364 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:57:49.367 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:57:49.417 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-04-12T04:57:49.668 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:57:49.928 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-04-12T04:57:50.223 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-04-12T04:57:50.348 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-04-12T04:57:50.734 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-04-12T04:57:50.973 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-04-12T04:57:51.183 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-04-12T04:57:51.190 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:16.2.15-0.el8.noarch 6/6 2024-04-12T04:57:51.472 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.15-0.el8.noarch 6/6 2024-04-12T04:57:51.473 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:16.2.15-0.el8.noarch 1/6 2024-04-12T04:57:51.473 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-04-12T04:57:51.473 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-04-12T04:57:51.473 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-04-12T04:57:51.473 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-04-12T04:57:51.575 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-04-12T04:57:51.854 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-04-12T04:57:51.865 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-04-12T04:57:51.865 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:51.866 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:57:51.866 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:16.2.15-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-04-12T04:57:51.866 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-04-12T04:57:51.866 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-04-12T04:57:51.866 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:51.866 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:57:51.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse 2024-04-12T04:57:52.045 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-cephadm-2:16.2.15-0.el8.noarch 6/6 2024-04-12T04:57:52.330 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.15-0.el8.noarch 6/6 2024-04-12T04:57:52.330 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-cephadm-2:16.2.15-0.el8.noarch 1/6 2024-04-12T04:57:52.330 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-04-12T04:57:52.330 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-04-12T04:57:52.331 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-04-12T04:57:52.331 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-04-12T04:57:52.714 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:57 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:57:52.755 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-04-12T04:57:52.755 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:52.755 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:57:52.755 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm-2:16.2.15-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-04-12T04:57:52.755 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-04-12T04:57:52.755 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-04-12T04:57:52.755 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:52.755 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:57:52.881 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-fuse 2024-04-12T04:57:53.607 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:01:59 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:57:54.347 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:57:54.348 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:54.348 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-12T04:57:54.348 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:54.348 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:57:54.348 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:16.2.15-0.el8 ceph 911 k 2024-04-12T04:57:54.348 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-12T04:57:54.348 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-12T04:57:54.349 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:54.349 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:57:54.349 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:54.349 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-04-12T04:57:54.349 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:54.349 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 994 k 2024-04-12T04:57:54.349 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.8 M 2024-04-12T04:57:54.349 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:57:54.626 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): ceph-fuse-16.2.15-0.el8.x86_64.rpm 9.7 MB/s | 911 kB 00:00 2024-04-12T04:57:54.792 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 322 kB/s | 83 kB 00:00 2024-04-12T04:57:54.793 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:54.793 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.2 MB/s | 994 kB 00:00 2024-04-12T04:57:54.803 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:57:54.823 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:57:54.823 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:57:54.909 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:57:54.912 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:57:55.080 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:57:55.240 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:57:55.241 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:55.241 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-12T04:57:55.241 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:55.241 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:57:55.241 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse x86_64 2:16.2.15-0.el8 ceph 911 k 2024-04-12T04:57:55.241 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-04-12T04:57:55.241 INFO:teuthology.orchestra.run.smithi113.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-12T04:57:55.241 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:55.242 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:57:55.242 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:57:55.242 INFO:teuthology.orchestra.run.smithi113.stdout:Install 2 Packages 2024-04-12T04:57:55.242 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:55.242 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 994 k 2024-04-12T04:57:55.242 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 2.8 M 2024-04-12T04:57:55.242 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:57:55.537 INFO:teuthology.orchestra.run.smithi113.stdout:(1/2): ceph-fuse-16.2.15-0.el8.x86_64.rpm 3.7 MB/s | 911 kB 00:00 2024-04-12T04:57:55.647 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-12T04:57:55.766 INFO:teuthology.orchestra.run.smithi113.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 178 kB/s | 83 kB 00:00 2024-04-12T04:57:55.766 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:55.767 INFO:teuthology.orchestra.run.smithi113.stdout:Total 1.9 MB/s | 994 kB 00:00 2024-04-12T04:57:55.777 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:57:55.798 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:57:55.798 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:57:55.856 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:16.2.15-0.el8.x86_64 2/2 2024-04-12T04:57:55.884 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:57:55.887 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:57:56.064 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:57:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-2:16.2.15-0.el8.x86_64 2/2 2024-04-12T04:57:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:16.2.15-0.el8.x86_64 1/2 2024-04-12T04:57:56.343 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-12T04:57:56.595 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-fuse-2:16.2.15-0.el8.x86_64 2/2 2024-04-12T04:57:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-12T04:57:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:56.676 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:57:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:16.2.15-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-12T04:57:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:56.676 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:57:56.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel 2024-04-12T04:57:56.977 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-fuse-2:16.2.15-0.el8.x86_64 2/2 2024-04-12T04:57:56.978 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-fuse-2:16.2.15-0.el8.x86_64 1/2 2024-04-12T04:57:57.396 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-12T04:57:57.396 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:57.396 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:57:57.396 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse-2:16.2.15-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-12T04:57:57.396 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:57:57.397 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:57:57.631 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install librados-devel 2024-04-12T04:57:57.665 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:02 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:57:58.377 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:04 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:57:59.308 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:57:59.308 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:59.309 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-12T04:57:59.309 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:59.309 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:57:59.309 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:16.2.15-0.el8 ceph 131 k 2024-04-12T04:57:59.309 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:59.309 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:57:59.309 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:57:59.309 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-12T04:57:59.309 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:57:59.310 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 131 k 2024-04-12T04:57:59.310 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 432 k 2024-04-12T04:57:59.310 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:57:59.356 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-16.2.15-0.el8.x86_64.rpm 2.7 MB/s | 131 kB 00:00 2024-04-12T04:57:59.357 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:57:59.357 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.7 MB/s | 131 kB 00:00 2024-04-12T04:57:59.357 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:57:59.363 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:57:59.363 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:57:59.412 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:57:59.415 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:57:59.521 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:57:59.738 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:00.007 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:00.008 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:00.008 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-12T04:58:00.008 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:00.008 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:58:00.008 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel x86_64 2:16.2.15-0.el8 ceph 131 k 2024-04-12T04:58:00.009 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:00.009 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:58:00.009 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:00.009 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-04-12T04:58:00.009 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:00.009 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 131 k 2024-04-12T04:58:00.009 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 432 k 2024-04-12T04:58:00.010 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:58:00.034 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:00.053 INFO:teuthology.orchestra.run.smithi113.stdout:librados-devel-16.2.15-0.el8.x86_64.rpm 2.9 MB/s | 131 kB 00:00 2024-04-12T04:58:00.053 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:58:00.054 INFO:teuthology.orchestra.run.smithi113.stdout:Total 2.9 MB/s | 131 kB 00:00 2024-04-12T04:58:00.054 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:58:00.059 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:58:00.059 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:58:00.108 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:58:00.111 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:58:00.244 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:58:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:00.424 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:58:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:16.2.15-0.el8.x86_64 2024-04-12T04:58:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:00.424 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:00.451 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librados-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:00.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2 2024-04-12T04:58:00.744 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:01.423 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:06 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:58:01.533 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:01.533 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:01.533 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:58:01.533 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel-2:16.2.15-0.el8.x86_64 2024-04-12T04:58:01.533 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:01.533 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:02.215 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install libcephfs2 2024-04-12T04:58:02.946 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:08 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:58:03.020 INFO:teuthology.orchestra.run.smithi192.stdout:Package libcephfs2-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:03.068 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:58:03.070 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-12T04:58:03.070 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:03.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs-devel 2024-04-12T04:58:03.928 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:08 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:58:04.534 INFO:teuthology.orchestra.run.smithi113.stdout:Package libcephfs2-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:04.582 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:04.583 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-04-12T04:58:04.583 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:04.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install libcephfs-devel 2024-04-12T04:58:05.411 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:11 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:58:05.558 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:58:05.559 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-12T04:58:05.559 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:05.559 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:58:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:16.2.15-0.el8 ceph 25 k 2024-04-12T04:58:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:05.560 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:58:05.560 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:05.560 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-12T04:58:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:05.560 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 25 k 2024-04-12T04:58:05.560 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 107 k 2024-04-12T04:58:05.560 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:58:05.595 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs-devel-16.2.15-0.el8.x86_64.rpm 719 kB/s | 25 kB 00:00 2024-04-12T04:58:05.596 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:58:05.596 INFO:teuthology.orchestra.run.smithi192.stdout:Total 695 kB/s | 25 kB 00:00 2024-04-12T04:58:05.596 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:58:05.600 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:58:05.600 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:58:05.602 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:58:05.605 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:58:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:58:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:06.128 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:06.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:06.513 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:06.513 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:58:06.513 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:16.2.15-0.el8.x86_64 2024-04-12T04:58:06.513 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:06.513 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:06.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2 2024-04-12T04:58:07.038 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:07.039 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:07.039 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-12T04:58:07.040 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:07.040 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:58:07.040 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel x86_64 2:16.2.15-0.el8 ceph 25 k 2024-04-12T04:58:07.040 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:07.040 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:58:07.040 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:07.040 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-04-12T04:58:07.040 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:07.040 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 25 k 2024-04-12T04:58:07.040 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 107 k 2024-04-12T04:58:07.040 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:58:07.083 INFO:teuthology.orchestra.run.smithi113.stdout:libcephfs-devel-16.2.15-0.el8.x86_64.rpm 595 kB/s | 25 kB 00:00 2024-04-12T04:58:07.083 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:58:07.084 INFO:teuthology.orchestra.run.smithi113.stdout:Total 579 kB/s | 25 kB 00:00 2024-04-12T04:58:07.084 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:58:07.087 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:58:07.088 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:58:07.089 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:58:07.093 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:58:07.139 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:58:07.366 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:07.374 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:12 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:58:07.629 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephfs-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:08.687 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:08.687 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:08.687 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:58:08.687 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel-2:16.2.15-0.el8.x86_64 2024-04-12T04:58:08.687 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:08.687 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:08.820 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install librados2 2024-04-12T04:58:08.953 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:09.001 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:58:09.003 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-12T04:58:09.003 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:09.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1 2024-04-12T04:58:09.550 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:15 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:58:09.847 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:14 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:58:11.137 INFO:teuthology.orchestra.run.smithi113.stdout:Package librados2-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:11.186 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:11.187 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-04-12T04:58:11.187 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:11.288 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install librbd1 2024-04-12T04:58:11.420 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:11.469 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:58:11.470 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-12T04:58:11.470 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:11.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados 2024-04-12T04:58:12.022 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:18 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:58:12.304 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:17 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:58:13.581 INFO:teuthology.orchestra.run.smithi113.stdout:Package librbd1-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:13.629 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:13.631 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-04-12T04:58:13.631 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:13.829 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install python3-rados 2024-04-12T04:58:13.885 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rados-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:13.933 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:58:13.935 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-12T04:58:13.935 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:14.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw 2024-04-12T04:58:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:20 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:58:14.774 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:19 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:58:16.148 INFO:teuthology.orchestra.run.smithi113.stdout:Package python3-rados-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:16.197 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:16.198 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-04-12T04:58:16.199 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:16.306 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install python3-rgw 2024-04-12T04:58:16.376 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rgw-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:16.426 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:58:16.427 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-12T04:58:16.428 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:16.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs 2024-04-12T04:58:17.045 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:23 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:58:17.255 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:22 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:58:18.629 INFO:teuthology.orchestra.run.smithi113.stdout:Package python3-rgw-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:18.676 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:18.677 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-04-12T04:58:18.677 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:18.790 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install python3-cephfs 2024-04-12T04:58:18.836 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-cephfs-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:18.884 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:58:18.885 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-12T04:58:18.885 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:18.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd 2024-04-12T04:58:19.520 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:25 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:58:19.728 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:24 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:58:21.113 INFO:teuthology.orchestra.run.smithi113.stdout:Package python3-cephfs-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:21.161 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-04-12T04:58:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:21.276 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install python3-rbd 2024-04-12T04:58:21.302 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rbd-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:21.351 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:58:21.353 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-12T04:58:21.353 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:21.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse 2024-04-12T04:58:22.005 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:28 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:58:22.179 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:27 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:58:23.580 INFO:teuthology.orchestra.run.smithi113.stdout:Package python3-rbd-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-12T04:58:23.628 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:23.629 INFO:teuthology.orchestra.run.smithi113.stdout:Nothing to do. 2024-04-12T04:58:23.629 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:23.733 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install rbd-fuse 2024-04-12T04:58:23.816 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:58:23.817 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:23.817 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-12T04:58:23.817 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:23.817 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:58:23.817 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:16.2.15-0.el8 ceph 88 k 2024-04-12T04:58:23.817 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:23.817 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:58:23.817 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:23.817 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-12T04:58:23.818 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:23.818 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 88 k 2024-04-12T04:58:23.818 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 240 k 2024-04-12T04:58:23.818 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:58:23.859 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-16.2.15-0.el8.x86_64.rpm 2.1 MB/s | 88 kB 00:00 2024-04-12T04:58:23.860 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:58:23.861 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.0 MB/s | 88 kB 00:00 2024-04-12T04:58:23.861 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:58:23.870 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:58:23.870 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:58:23.920 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:58:23.923 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:58:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:58:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:24.460 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:30 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:58:24.509 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:24.988 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:58:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:16.2.15-0.el8.x86_64 2024-04-12T04:58:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:24.988 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:25.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror 2024-04-12T04:58:25.946 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:30 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:58:26.093 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:26.094 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:26.094 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-12T04:58:26.094 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:26.094 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:58:26.094 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse x86_64 2:16.2.15-0.el8 ceph 88 k 2024-04-12T04:58:26.094 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:26.094 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:58:26.095 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:26.095 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-04-12T04:58:26.095 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:26.095 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 88 k 2024-04-12T04:58:26.095 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 240 k 2024-04-12T04:58:26.095 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:58:26.149 INFO:teuthology.orchestra.run.smithi113.stdout:rbd-fuse-16.2.15-0.el8.x86_64.rpm 1.6 MB/s | 88 kB 00:00 2024-04-12T04:58:26.149 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:58:26.150 INFO:teuthology.orchestra.run.smithi113.stdout:Total 1.6 MB/s | 88 kB 00:00 2024-04-12T04:58:26.150 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:58:26.159 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:58:26.159 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:58:26.208 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:58:26.294 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:58:26.311 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:58:26.520 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-fuse-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:26.811 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-fuse-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:27.206 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-fuse-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:27.206 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:27.207 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:58:27.207 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse-2:16.2.15-0.el8.x86_64 2024-04-12T04:58:27.207 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:27.207 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:27.442 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install rbd-mirror 2024-04-12T04:58:27.582 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:58:27.582 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:27.583 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-12T04:58:27.583 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:27.583 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:58:27.583 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:16.2.15-0.el8 ceph 3.7 M 2024-04-12T04:58:27.583 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:27.583 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:58:27.583 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:27.583 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-12T04:58:27.583 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:27.584 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.7 M 2024-04-12T04:58:27.584 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 15 M 2024-04-12T04:58:27.584 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:58:27.680 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-16.2.15-0.el8.x86_64.rpm 38 MB/s | 3.7 MB 00:00 2024-04-12T04:58:27.681 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:58:27.681 INFO:teuthology.orchestra.run.smithi192.stdout:Total 38 MB/s | 3.7 MB 00:00 2024-04-12T04:58:27.681 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:58:27.693 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:58:27.693 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:58:27.755 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:58:27.758 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:58:28.179 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:34 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:58:28.206 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:58:28.390 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:28.841 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:29.203 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:58:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:16.2.15-0.el8.x86_64 2024-04-12T04:58:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:29.203 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:29.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd 2024-04-12T04:58:29.805 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:29.805 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror x86_64 2:16.2.15-0.el8 ceph 3.7 M 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 3.7 M 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 15 M 2024-04-12T04:58:29.806 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:58:29.902 INFO:teuthology.orchestra.run.smithi113.stdout:rbd-mirror-16.2.15-0.el8.x86_64.rpm 38 MB/s | 3.7 MB 00:00 2024-04-12T04:58:29.903 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:58:29.903 INFO:teuthology.orchestra.run.smithi113.stdout:Total 38 MB/s | 3.7 MB 00:00 2024-04-12T04:58:29.904 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:58:29.915 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:58:29.915 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:58:29.978 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:58:29.981 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:58:30.192 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:35 ago on Fri 12 Apr 2024 04:55:55 AM UTC. 2024-04-12T04:58:30.587 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:58:30.829 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-mirror-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:31.242 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-mirror-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:31.654 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-mirror-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:31.655 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:31.655 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:58:31.655 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror-2:16.2.15-0.el8.x86_64 2024-04-12T04:58:31.655 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:31.655 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:31.832 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-12T04:58:31.833 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:31.833 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-12T04:58:31.833 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:31.833 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-12T04:58:31.833 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:16.2.15-0.el8 ceph 175 k 2024-04-12T04:58:31.834 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:31.834 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-12T04:58:31.834 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-12T04:58:31.834 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-12T04:58:31.834 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:31.834 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 175 k 2024-04-12T04:58:31.834 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 523 k 2024-04-12T04:58:31.834 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-12T04:58:31.868 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install rbd-nbd 2024-04-12T04:58:31.875 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-16.2.15-0.el8.x86_64.rpm 4.2 MB/s | 175 kB 00:00 2024-04-12T04:58:31.875 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:58:31.876 INFO:teuthology.orchestra.run.smithi192.stdout:Total 4.1 MB/s | 175 kB 00:00 2024-04-12T04:58:31.876 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-12T04:58:31.886 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-12T04:58:31.886 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-12T04:58:31.936 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-12T04:58:31.939 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-12T04:58:32.056 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-12T04:58:32.249 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:32.609 INFO:teuthology.orchestra.run.smithi113.stdout:Last metadata expiration check: 0:02:38 ago on Fri 12 Apr 2024 04:55:54 AM UTC. 2024-04-12T04:58:32.916 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:32.916 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:32.916 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-12T04:58:32.916 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:16.2.15-0.el8.x86_64 2024-04-12T04:58:32.916 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T04:58:32.917 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-12T04:58:33.098 DEBUG:teuthology.parallel:result is None 2024-04-12T04:58:34.232 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-04-12T04:58:34.232 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:34.233 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-04-12T04:58:34.233 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:34.233 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-04-12T04:58:34.233 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd x86_64 2:16.2.15-0.el8 ceph 175 k 2024-04-12T04:58:34.233 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:34.233 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-04-12T04:58:34.233 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-04-12T04:58:34.233 INFO:teuthology.orchestra.run.smithi113.stdout:Install 1 Package 2024-04-12T04:58:34.233 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:34.233 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 175 k 2024-04-12T04:58:34.233 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 523 k 2024-04-12T04:58:34.234 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-04-12T04:58:34.297 INFO:teuthology.orchestra.run.smithi113.stdout:rbd-nbd-16.2.15-0.el8.x86_64.rpm 2.7 MB/s | 175 kB 00:00 2024-04-12T04:58:34.298 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-04-12T04:58:34.298 INFO:teuthology.orchestra.run.smithi113.stdout:Total 2.6 MB/s | 175 kB 00:00 2024-04-12T04:58:34.299 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-04-12T04:58:34.309 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-04-12T04:58:34.309 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-04-12T04:58:34.358 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-04-12T04:58:34.361 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-04-12T04:58:34.468 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-04-12T04:58:34.651 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-nbd-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:34.931 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-nbd-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:35.331 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-nbd-2:16.2.15-0.el8.x86_64 1/1 2024-04-12T04:58:35.331 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:35.331 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-04-12T04:58:35.331 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd-2:16.2.15-0.el8.x86_64 2024-04-12T04:58:35.331 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T04:58:35.331 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-04-12T04:58:35.563 DEBUG:teuthology.parallel:result is None 2024-04-12T04:58:35.563 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T04:58:35.563 INFO:teuthology.packaging:ref: None 2024-04-12T04:58:35.563 INFO:teuthology.packaging:tag: None 2024-04-12T04:58:35.563 INFO:teuthology.packaging:branch: pacific 2024-04-12T04:58:35.564 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T04:58:35.564 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-04-12T04:58:35.798 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-12T04:58:35.820 INFO:teuthology.orchestra.run.smithi113.stdout:16.2.15-0.el8 2024-04-12T04:58:35.821 INFO:teuthology.packaging:The installed version of ceph is 16.2.15-0.el8 2024-04-12T04:58:35.821 INFO:teuthology.task.install:The correct ceph version 16.2.15-0 is installed. 2024-04-12T04:58:35.823 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T04:58:35.823 INFO:teuthology.packaging:ref: None 2024-04-12T04:58:35.823 INFO:teuthology.packaging:tag: None 2024-04-12T04:58:35.823 INFO:teuthology.packaging:branch: pacific 2024-04-12T04:58:35.823 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T04:58:35.823 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-04-12T04:58:36.051 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-12T04:58:36.078 INFO:teuthology.orchestra.run.smithi192.stdout:16.2.15-0.el8 2024-04-12T04:58:36.079 INFO:teuthology.packaging:The installed version of ceph is 16.2.15-0.el8 2024-04-12T04:58:36.079 INFO:teuthology.task.install:The correct ceph version 16.2.15-0 is installed. 2024-04-12T04:58:36.080 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-12T04:58:36.080 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T04:58:36.080 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T04:58:36.108 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T04:58:36.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T04:58:36.152 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-12T04:58:36.152 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T04:58:36.152 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T04:58:36.179 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T04:58:36.245 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T04:58:36.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T04:58:36.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T04:58:36.347 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-12T04:58:36.348 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T04:58:36.348 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T04:58:36.374 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T04:58:36.440 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T04:58:36.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T04:58:36.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T04:58:36.541 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-12T04:58:36.542 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T04:58:36.542 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T04:58:36.567 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T04:58:36.634 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T04:58:36.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T04:58:36.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T04:58:36.732 INFO:teuthology.run_tasks:Running task print... 2024-04-12T04:58:36.741 INFO:teuthology.task.print:**** done install task... 2024-04-12T04:58:36.741 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-12T04:58:36.828 INFO:tasks.cephadm:Config: {'cephadm_branch': 'pacific', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:pacific', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'} 2024-04-12T04:58:36.828 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:pacific 2024-04-12T04:58:36.828 INFO:tasks.cephadm:Cluster fsid is 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:58:36.829 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-12T04:58:36.829 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-12T04:58:36.829 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi113': '172.21.15.113', 'mon.smithi192': '172.21.15.192'} 2024-04-12T04:58:36.829 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-12T04:58:36.829 DEBUG:teuthology.orchestra.run.smithi113:> sudo hostname $(hostname -s) 2024-04-12T04:58:36.856 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2024-04-12T04:58:36.884 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref pacific)... 2024-04-12T04:58:36.885 DEBUG:teuthology.orchestra.run.smithi113:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-12T04:58:37.097 INFO:teuthology.orchestra.run.smithi113.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Apr 12 04:58 /home/ubuntu/cephtest/cephadm 2024-04-12T04:58:37.097 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-12T04:58:37.348 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Apr 12 04:58 /home/ubuntu/cephtest/cephadm 2024-04-12T04:58:37.348 DEBUG:teuthology.orchestra.run.smithi113:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-12T04:58:37.367 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-12T04:58:37.396 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:pacific on all hosts... 2024-04-12T04:58:37.396 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific pull 2024-04-12T04:58:37.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific pull 2024-04-12T04:58:38.626 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:pacific... 2024-04-12T04:58:38.659 INFO:teuthology.orchestra.run.smithi113.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:pacific... 2024-04-12T04:59:14.441 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T04:59:14.441 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_version": "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)", 2024-04-12T04:59:14.441 INFO:teuthology.orchestra.run.smithi113.stdout: "image_id": "727f386679cb5962a8bf6fbf5ad40cabb5457725c5b7aee81a1eb45b35ff08b0", 2024-04-12T04:59:14.441 INFO:teuthology.orchestra.run.smithi113.stdout: "repo_digests": [ 2024-04-12T04:59:14.441 INFO:teuthology.orchestra.run.smithi113.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:a917adc3f5c2fb88f02feae5d8f667631e1c7fa0b341817d32f1d6fd5cd1bec1" 2024-04-12T04:59:14.441 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-04-12T04:59:14.441 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T04:59:18.357 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-12T04:59:18.357 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)", 2024-04-12T04:59:18.357 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "727f386679cb5962a8bf6fbf5ad40cabb5457725c5b7aee81a1eb45b35ff08b0", 2024-04-12T04:59:18.357 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2024-04-12T04:59:18.357 INFO:teuthology.orchestra.run.smithi192.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:a917adc3f5c2fb88f02feae5d8f667631e1c7fa0b341817d32f1d6fd5cd1bec1" 2024-04-12T04:59:18.357 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-12T04:59:18.357 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-12T04:59:18.378 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /etc/ceph 2024-04-12T04:59:18.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2024-04-12T04:59:18.446 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 777 /etc/ceph 2024-04-12T04:59:18.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2024-04-12T04:59:18.513 INFO:tasks.cephadm:Writing seed config... 2024-04-12T04:59:18.514 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-12T04:59:18.515 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-12T04:59:18.516 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-12T04:59:18.517 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-12T04:59:18.517 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-12T04:59:18.517 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-12T04:59:18.517 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-12T04:59:18.517 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-12T04:59:18.517 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-12T04:59:18.517 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-12T04:59:18.517 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-12T04:59:18.517 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-12T04:59:18.517 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-12T04:59:18.517 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-12T04:59:18.518 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T04:59:18.518 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-12T04:59:18.566 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 52321e2c-f889-11ee-bc8c-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-12T04:59:18.567 DEBUG:teuthology.orchestra.run.smithi113:mon.smithi113> sudo journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi113.service 2024-04-12T04:59:18.609 INFO:tasks.cephadm:Bootstrapping... 2024-04-12T04:59:18.609 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific -v bootstrap --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.113 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-12T04:59:18.637 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:-- Logs begin at Fri 2024-04-12 04:49:01 UTC. -- 2024-04-12T04:59:18.814 INFO:teuthology.orchestra.run.smithi113.stderr:-------------------------------------------------------------------------------- 2024-04-12T04:59:18.814 INFO:teuthology.orchestra.run.smithi113.stderr:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:pacific', '-v', 'bootstrap', '--fsid', '52321e2c-f889-11ee-bc8c-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.113', '--skip-admin-label'] 2024-04-12T04:59:18.836 INFO:teuthology.orchestra.run.smithi113.stderr:/bin/podman: 4.9.4-dev 2024-04-12T04:59:18.851 INFO:teuthology.orchestra.run.smithi113.stderr:Verifying podman|docker is present... 2024-04-12T04:59:18.870 INFO:teuthology.orchestra.run.smithi113.stderr:/bin/podman: 4.9.4-dev 2024-04-12T04:59:18.877 INFO:teuthology.orchestra.run.smithi113.stderr:Verifying lvm2 is present... 2024-04-12T04:59:18.878 INFO:teuthology.orchestra.run.smithi113.stderr:Verifying time synchronization is in place... 2024-04-12T04:59:18.884 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-12T04:59:18.890 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: inactive 2024-04-12T04:59:18.896 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: enabled 2024-04-12T04:59:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: active 2024-04-12T04:59:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:Unit chronyd.service is enabled and running 2024-04-12T04:59:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:Repeating the final host check... 2024-04-12T04:59:18.921 INFO:teuthology.orchestra.run.smithi113.stderr:/bin/podman: 4.9.4-dev 2024-04-12T04:59:18.932 INFO:teuthology.orchestra.run.smithi113.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-12T04:59:18.932 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl is present 2024-04-12T04:59:18.932 INFO:teuthology.orchestra.run.smithi113.stderr:lvcreate is present 2024-04-12T04:59:18.937 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-12T04:59:18.942 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: inactive 2024-04-12T04:59:18.948 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: enabled 2024-04-12T04:59:18.954 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: active 2024-04-12T04:59:18.954 INFO:teuthology.orchestra.run.smithi113.stderr:Unit chronyd.service is enabled and running 2024-04-12T04:59:18.955 INFO:teuthology.orchestra.run.smithi113.stderr:Host looks OK 2024-04-12T04:59:18.955 INFO:teuthology.orchestra.run.smithi113.stderr:Cluster fsid: 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:59:18.955 INFO:teuthology.orchestra.run.smithi113.stderr:Acquiring lock 140456923302376 on /run/cephadm/52321e2c-f889-11ee-bc8c-c7b262605968.lock 2024-04-12T04:59:18.955 INFO:teuthology.orchestra.run.smithi113.stderr:Lock 140456923302376 acquired on /run/cephadm/52321e2c-f889-11ee-bc8c-c7b262605968.lock 2024-04-12T04:59:18.955 INFO:teuthology.orchestra.run.smithi113.stderr:Verifying IP 172.21.15.113 port 3300 ... 2024-04-12T04:59:18.955 INFO:teuthology.orchestra.run.smithi113.stderr:Verifying IP 172.21.15.113 port 6789 ... 2024-04-12T04:59:18.956 INFO:teuthology.orchestra.run.smithi113.stderr:Base mon IP(s) is [172.21.15.113:3300, 172.21.15.113:6789], mon addrv is [v2:172.21.15.113:3300,v1:172.21.15.113:6789] 2024-04-12T04:59:18.959 INFO:teuthology.orchestra.run.smithi113.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.113 metric 100 2024-04-12T04:59:18.959 INFO:teuthology.orchestra.run.smithi113.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.113 metric 100 2024-04-12T04:59:18.962 INFO:teuthology.orchestra.run.smithi113.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-12T04:59:18.962 INFO:teuthology.orchestra.run.smithi113.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-12T04:59:18.962 INFO:teuthology.orchestra.run.smithi113.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-04-12T04:59:18.964 INFO:teuthology.orchestra.run.smithi113.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-12T04:59:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-12T04:59:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-12T04:59:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-12T04:59:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d7c1/64 scope link 2024-04-12T04:59:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-12T04:59:18.965 INFO:teuthology.orchestra.run.smithi113.stderr:Mon IP `172.21.15.113` is in CIDR network `172.21.0.0/20` 2024-04-12T04:59:18.966 INFO:teuthology.orchestra.run.smithi113.stderr:Mon IP `172.21.15.113` is in CIDR network `172.21.0.0/20` 2024-04-12T04:59:18.966 INFO:teuthology.orchestra.run.smithi113.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-12T04:59:18.966 INFO:teuthology.orchestra.run.smithi113.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-12T04:59:18.967 INFO:teuthology.orchestra.run.smithi113.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:pacific... 2024-04-12T04:59:19.043 INFO:teuthology.orchestra.run.smithi113.stderr:/bin/podman: Trying to pull quay.ceph.io/ceph-ci/ceph:pacific... 2024-04-12T04:59:19.128 INFO:teuthology.orchestra.run.smithi113.stderr:/bin/podman: Getting image source signatures 2024-04-12T04:59:19.128 INFO:teuthology.orchestra.run.smithi113.stderr:/bin/podman: Copying blob sha256:efbf2990d792dd99354c95f0e06c6d4330413e38e584cb30666e2a0d35649b94 2024-04-12T04:59:19.128 INFO:teuthology.orchestra.run.smithi113.stderr:/bin/podman: Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-04-12T04:59:19.128 INFO:teuthology.orchestra.run.smithi113.stderr:/bin/podman: Copying config sha256:727f386679cb5962a8bf6fbf5ad40cabb5457725c5b7aee81a1eb45b35ff08b0 2024-04-12T04:59:20.220 INFO:teuthology.orchestra.run.smithi113.stderr:/bin/podman: Writing manifest to image destination 2024-04-12T04:59:20.959 INFO:teuthology.orchestra.run.smithi113.stderr:/bin/podman: 727f386679cb5962a8bf6fbf5ad40cabb5457725c5b7aee81a1eb45b35ff08b0 2024-04-12T04:59:21.478 INFO:teuthology.orchestra.run.smithi113.stderr:ceph: ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable) 2024-04-12T04:59:22.143 INFO:teuthology.orchestra.run.smithi113.stderr:Ceph version: ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable) 2024-04-12T04:59:22.143 INFO:teuthology.orchestra.run.smithi113.stderr:Extracting ceph user uid/gid from container image... 2024-04-12T04:59:22.701 INFO:teuthology.orchestra.run.smithi113.stderr:stat: 167 167 2024-04-12T04:59:23.329 INFO:teuthology.orchestra.run.smithi113.stderr:Creating initial keys... 2024-04-12T04:59:23.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-authtool: AQCrvxhmPX22LhAA5WvozbtsPVTgeR04JncI/A== 2024-04-12T04:59:24.963 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-authtool: AQCsvxhmKswQNBAA6ytqjKvI6co7FRp7qU9Z6g== 2024-04-12T04:59:26.212 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-authtool: AQCuvxhmOzBWBxAAobBolH3eoz0BRiW0AJSmww== 2024-04-12T04:59:26.765 INFO:teuthology.orchestra.run.smithi113.stderr:Creating initial monmap... 2024-04-12T04:59:27.260 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-12T04:59:27.260 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:59:27.260 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-12T04:59:27.829 INFO:teuthology.orchestra.run.smithi113.stderr:monmaptool for smithi113 [v2:172.21.15.113:3300,v1:172.21.15.113:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-12T04:59:27.830 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/monmaptool: set fsid to 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:59:27.830 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-12T04:59:27.830 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-12T04:59:27.830 INFO:teuthology.orchestra.run.smithi113.stderr:Creating mon... 2024-04-12T04:59:28.393 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.275+0000 7f7d5f865880 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-12T04:59:28.393 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.275+0000 7f7d5f865880 1 imported monmap: 2024-04-12T04:59:28.393 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: epoch 0 2024-04-12T04:59:28.393 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: fsid 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:59:28.393 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: last_changed 2024-04-12T04:59:27.208775+0000 2024-04-12T04:59:28.393 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: created 2024-04-12T04:59:27.208775+0000 2024-04-12T04:59:28.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-04-12T04:59:28.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-04-12T04:59:28.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.smithi113 2024-04-12T04:59:28.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.275+0000 7f7d5f865880 0 /usr/bin/ceph-mon: set fsid to 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:59:28.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: RocksDB version: 6.8.1 2024-04-12T04:59:28.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Compile date Mar 4 2024 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: DB SUMMARY 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi113/store.db dir, Total Num: 0, files: 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi113/store.db: 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.error_if_exists: 0 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.create_if_missing: 1 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.paranoid_checks: 1 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.env: 0x55cdc1e62080 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.fs: Posix File System 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.info_log: 0x55cdc4585300 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-12T04:59:28.422 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.statistics: (nil) 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.use_fsync: 0 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_log_file_size: 0 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.allow_fallocate: 1 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.use_direct_reads: 0 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.create_missing_column_families: 0 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.db_log_dir: 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi113/store.db 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T04:59:28.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_subcompactions: 1 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_background_flushes: -1 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.advise_random_on_open: 1 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.write_buffer_manager: 0x55cdc458b950 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.rate_limiter: (nil) 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-12T04:59:28.424 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.unordered_write: 0 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.row_cache: None 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.wal_filter: None 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.preserve_deletes: 0 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.two_write_queues: 0 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.manual_wal_flush: 0 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.atomic_flush: 0 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T04:59:28.425 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.log_readahead_size: 0 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_background_jobs: 2 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_background_compactions: -1 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_total_wal_size: 0 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_open_files: -1 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.bytes_per_sync: 0 2024-04-12T04:59:28.426 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Compression algorithms supported: 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: kZSTD supported: 0 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: kXpressCompression supported: 0 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: kLZ4Compression supported: 1 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: kBZip2Compression supported: 0 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: kZlibCompression supported: 1 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: kSnappyCompression supported: 1 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.427 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi113/store.db/MANIFEST-000001 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.merge_operator: 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_filter: None 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_filter_factory: None 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cdc449bd18) 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: index_type: 0 2024-04-12T04:59:28.428 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: checksum: 1 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: block_cache: 0x55cdc44d0d10 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-04-12T04:59:28.429 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: format_version: 2 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: block_align: 0 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compression: NoCompression 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-12T04:59:28.430 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.num_levels: 7 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compression_opts.level: 32767 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T04:59:28.431 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compression_opts.enabled: false 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T04:59:28.432 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.arena_block_size: 4194304 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T04:59:28.433 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.table_properties_collectors: 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.inplace_update_support: 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.bloom_locality: 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.max_successive_merges: 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.force_consistency_checks: 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.ttl: 2592000 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.277+0000 7f7d5f865880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.278+0000 7f7d5f865880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi113/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T04:59:28.434 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.278+0000 7f7d5f865880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.279+0000 7f7d5f865880 4 rocksdb: DB pointer 0x55cdc4599800 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.279+0000 7f7d486d8700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.279+0000 7f7d486d8700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-04-12T04:59:28.435 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T04:59:28.436 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T04:59:28.437 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T04:59:28.438 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-04-12T04:59:28.438 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T04:59:28.438 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T04:59:28.438 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T04:59:28.438 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.438 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-04-12T04:59:28.438 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: 2024-04-12T04:59:28.438 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.280+0000 7f7d5f865880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-04-12T04:59:28.438 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.280+0000 7f7d5f865880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-04-12T04:59:28.438 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph-mon: debug 2024-04-12T04:59:28.280+0000 7f7d5f865880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi113 for mon.smithi113 2024-04-12T04:59:28.945 INFO:teuthology.orchestra.run.smithi113.stderr:create mon.smithi113 on 2024-04-12T04:59:28.960 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-12T04:59:29.053 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-12T04:59:29.146 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-52321e2c-f889-11ee-bc8c-c7b262605968.target → /etc/systemd/system/ceph-52321e2c-f889-11ee-bc8c-c7b262605968.target. 2024-04-12T04:59:29.146 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-52321e2c-f889-11ee-bc8c-c7b262605968.target → /etc/systemd/system/ceph-52321e2c-f889-11ee-bc8c-c7b262605968.target. 2024-04-12T04:59:29.347 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: Failed to reset failed state of unit ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi113.service: Unit ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi113.service not loaded. 2024-04-12T04:59:29.354 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: Created symlink /etc/systemd/system/ceph-52321e2c-f889-11ee-bc8c-c7b262605968.target.wants/ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi113.service → /etc/systemd/system/ceph-52321e2c-f889-11ee-bc8c-c7b262605968@.service. 2024-04-12T04:59:29.481 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:29 smithi113 systemd[1]: Starting Ceph mon.smithi113 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T04:59:30.028 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: disabled 2024-04-12T04:59:30.035 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: inactive 2024-04-12T04:59:30.036 INFO:teuthology.orchestra.run.smithi113.stderr:firewalld.service is not enabled 2024-04-12T04:59:30.036 INFO:teuthology.orchestra.run.smithi113.stderr:Not possible to enable service . firewalld.service is not available 2024-04-12T04:59:30.036 INFO:teuthology.orchestra.run.smithi113.stderr:Waiting for mon to start... 2024-04-12T04:59:30.036 INFO:teuthology.orchestra.run.smithi113.stderr:Waiting for mon... 2024-04-12T04:59:30.320 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:30 smithi113 bash[39165]: 8fe3e1ff58ff2deda614de488b439cfc25cb4068304b5971318a4b18bcf62dd9 2024-04-12T04:59:30.320 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:30 smithi113 systemd[1]: Started Ceph mon.smithi113 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T04:59:30.320 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: cluster 2024-04-12T04:59:30.005142+0000 mon.smithi113 (mon.0) 0 : [INF] mkfs 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:59:30.320 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: cluster 2024-04-12T04:59:29.992603+0000 mon.smithi113 (mon.0) 1 : cluster [INF] mon.smithi113 is new leader, mons smithi113 in quorum (ranks 0) 2024-04-12T04:59:30.623 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.618+0000 7f02b4913700 1 Processor -- start 2024-04-12T04:59:30.623 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.618+0000 7f02b4913700 1 -- start start 2024-04-12T04:59:30.623 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.618+0000 7f02b4913700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00ffc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:30.623 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.618+0000 7f02b4913700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0100180 con 0x7f02b0071c00 2024-04-12T04:59:30.623 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.619+0000 7f02ae59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00ffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:30.623 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.619+0000 7f02ae59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00ffc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38090/0 (socket says 172.21.15.113:38090) 2024-04-12T04:59:30.623 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.619+0000 7f02ae59c700 1 -- 172.21.15.113:0/1308046030 learned_addr learned my addr 172.21.15.113:0/1308046030 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:30.623 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.619+0000 7f02ae59c700 1 -- 172.21.15.113:0/1308046030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b01002c0 con 0x7f02b0071c00 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.619+0000 7f02ae59c700 1 --2- 172.21.15.113:0/1308046030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00ffc90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0298009730 tx=0x7f02980090f0).ready entity=mon.0 client_cookie=9c9fb1d87e2d9e48 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.619+0000 7f02ad59a700 1 -- 172.21.15.113:0/1308046030 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f029800b6e0 con 0x7f02b0071c00 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.619+0000 7f02ad59a700 1 -- 172.21.15.113:0/1308046030 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029800b840 con 0x7f02b0071c00 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.619+0000 7f02ad59a700 1 -- 172.21.15.113:0/1308046030 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f029800bb10 con 0x7f02b0071c00 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.620+0000 7f02b4913700 1 -- 172.21.15.113:0/1308046030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 msgr2=0x7f02b00ffc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.620+0000 7f02b4913700 1 --2- 172.21.15.113:0/1308046030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00ffc90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0298009730 tx=0x7f02980090f0).stop 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.620+0000 7f02b4913700 1 -- 172.21.15.113:0/1308046030 shutdown_connections 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.620+0000 7f02b4913700 1 --2- 172.21.15.113:0/1308046030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00ffc90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.620+0000 7f02b4913700 1 -- 172.21.15.113:0/1308046030 >> 172.21.15.113:0/1308046030 conn(0x7f02b00f76f0 msgr2=0x7f02b00f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.620+0000 7f02b4913700 1 -- 172.21.15.113:0/1308046030 shutdown_connections 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.620+0000 7f02b4913700 1 -- 172.21.15.113:0/1308046030 wait complete. 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.620+0000 7f02b4913700 1 Processor -- start 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.620+0000 7f02b4913700 1 -- start start 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02b4913700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00fd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02b4913700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b00fbb90 con 0x7f02b0071c00 2024-04-12T04:59:30.625 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02ae59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00fd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:30.625 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02ae59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00fd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38102/0 (socket says 172.21.15.113:38102) 2024-04-12T04:59:30.625 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02ae59c700 1 -- 172.21.15.113:0/2552238752 learned_addr learned my addr 172.21.15.113:0/2552238752 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:30.625 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02ae59c700 1 -- 172.21.15.113:0/2552238752 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0298009370 con 0x7f02b0071c00 2024-04-12T04:59:30.625 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02ae59c700 1 --2- 172.21.15.113:0/2552238752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00fd710 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0298000c00 tx=0x7f0298017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:30.625 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02a77fe700 1 -- 172.21.15.113:0/2552238752 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0298011cc0 con 0x7f02b0071c00 2024-04-12T04:59:30.625 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02a77fe700 1 -- 172.21.15.113:0/2552238752 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0298011e20 con 0x7f02b0071c00 2024-04-12T04:59:30.625 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02b4913700 1 -- 172.21.15.113:0/2552238752 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b00fbd30 con 0x7f02b0071c00 2024-04-12T04:59:30.625 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02a77fe700 1 -- 172.21.15.113:0/2552238752 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f029801d590 con 0x7f02b0071c00 2024-04-12T04:59:30.626 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.621+0000 7f02b4913700 1 -- 172.21.15.113:0/2552238752 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b00fc140 con 0x7f02b0071c00 2024-04-12T04:59:30.626 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.622+0000 7f02a77fe700 1 -- 172.21.15.113:0/2552238752 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f029801da20 con 0x7f02b0071c00 2024-04-12T04:59:30.626 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.622+0000 7f02a77fe700 1 -- 172.21.15.113:0/2552238752 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0298026b10 con 0x7f02b0071c00 2024-04-12T04:59:30.626 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.622+0000 7f02b4913700 1 -- 172.21.15.113:0/2552238752 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b0046ce0 con 0x7f02b0071c00 2024-04-12T04:59:30.627 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.625+0000 7f02a77fe700 1 -- 172.21.15.113:0/2552238752 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f0298021030 con 0x7f02b0071c00 2024-04-12T04:59:30.695 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.691+0000 7f02b4913700 1 -- 172.21.15.113:0/2552238752 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f02b0105180 con 0x7f02b0071c00 2024-04-12T04:59:30.695 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.691+0000 7f02a77fe700 1 -- 172.21.15.113:0/2552238752 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f029802f430 con 0x7f02b0071c00 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: cluster: 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: id: 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: services: 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi113 (age 0.691848s) 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: data: 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: pgs: 2024-04-12T04:59:30.696 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.697 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.692+0000 7f02b4913700 1 -- 172.21.15.113:0/2552238752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 msgr2=0x7f02b00fd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:30.697 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.692+0000 7f02b4913700 1 --2- 172.21.15.113:0/2552238752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00fd710 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0298000c00 tx=0x7f0298017000).stop 2024-04-12T04:59:30.697 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.692+0000 7f02b4913700 1 -- 172.21.15.113:0/2552238752 shutdown_connections 2024-04-12T04:59:30.697 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.692+0000 7f02b4913700 1 --2- 172.21.15.113:0/2552238752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02b0071c00 0x7f02b00fd710 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:30.697 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.692+0000 7f02b4913700 1 -- 172.21.15.113:0/2552238752 >> 172.21.15.113:0/2552238752 conn(0x7f02b00f76f0 msgr2=0x7f02b00f9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:30.697 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.692+0000 7f02b4913700 1 -- 172.21.15.113:0/2552238752 shutdown_connections 2024-04-12T04:59:30.697 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:30.692+0000 7f02b4913700 1 -- 172.21.15.113:0/2552238752 wait complete. 2024-04-12T04:59:31.259 INFO:teuthology.orchestra.run.smithi113.stderr:mon is available 2024-04-12T04:59:31.259 INFO:teuthology.orchestra.run.smithi113.stderr:Assimilating anything we can from ceph.conf... 2024-04-12T04:59:31.261 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: cluster 2024-04-12T04:59:30.000630+0000 mon.smithi113 (mon.0) 2 : cluster [INF] mon.smithi113 is new leader, mons smithi113 in quorum (ranks 0) 2024-04-12T04:59:31.261 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: cluster 2024-04-12T04:59:30.002611+0000 mon.smithi113 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi113=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} removed_ranks: {} 2024-04-12T04:59:31.261 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: cluster 2024-04-12T04:59:30.010330+0000 mon.smithi113 (mon.0) 4 : cluster [DBG] fsmap 2024-04-12T04:59:31.261 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: cluster 2024-04-12T04:59:30.018422+0000 mon.smithi113 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-12T04:59:31.261 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: cluster 2024-04-12T04:59:30.019021+0000 mon.smithi113 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-04-12T04:59:31.261 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: audit 2024-04-12T04:59:30.692440+0000 mon.smithi113 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.113:0/2552238752' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-12T04:59:31.816 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.814+0000 7fb555e44700 1 Processor -- start 2024-04-12T04:59:31.817 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.814+0000 7fb555e44700 1 -- start start 2024-04-12T04:59:31.817 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.814+0000 7fb555e44700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb5500fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:31.817 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.814+0000 7fb555e44700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5500fc7e0 con 0x7fb5500fbed0 2024-04-12T04:59:31.817 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.814+0000 7fb54f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb5500fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.814+0000 7fb54f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb5500fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38104/0 (socket says 172.21.15.113:38104) 2024-04-12T04:59:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.814+0000 7fb54f7fe700 1 -- 172.21.15.113:0/2995915190 learned_addr learned my addr 172.21.15.113:0/2995915190 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.814+0000 7fb54f7fe700 1 -- 172.21.15.113:0/2995915190 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5500fc920 con 0x7fb5500fbed0 2024-04-12T04:59:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.814+0000 7fb54f7fe700 1 --2- 172.21.15.113:0/2995915190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb5500fc2f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb540009730 tx=0x7fb5400090f0).ready entity=mon.0 client_cookie=da268167c35d1052 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.815+0000 7fb54e7fc700 1 -- 172.21.15.113:0/2995915190 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb54000b6e0 con 0x7fb5500fbed0 2024-04-12T04:59:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.815+0000 7fb54e7fc700 1 -- 172.21.15.113:0/2995915190 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54000b840 con 0x7fb5500fbed0 2024-04-12T04:59:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.815+0000 7fb54e7fc700 1 -- 172.21.15.113:0/2995915190 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb54000bb10 con 0x7fb5500fbed0 2024-04-12T04:59:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.815+0000 7fb555e44700 1 -- 172.21.15.113:0/2995915190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 msgr2=0x7fb5500fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.815+0000 7fb555e44700 1 --2- 172.21.15.113:0/2995915190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb5500fc2f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb540009730 tx=0x7fb5400090f0).stop 2024-04-12T04:59:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.815+0000 7fb555e44700 1 -- 172.21.15.113:0/2995915190 shutdown_connections 2024-04-12T04:59:31.819 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.815+0000 7fb555e44700 1 --2- 172.21.15.113:0/2995915190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb5500fc2f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.815+0000 7fb555e44700 1 -- 172.21.15.113:0/2995915190 >> 172.21.15.113:0/2995915190 conn(0x7fb5500f76f0 msgr2=0x7fb5500f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.815+0000 7fb555e44700 1 -- 172.21.15.113:0/2995915190 shutdown_connections 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.815+0000 7fb555e44700 1 -- 172.21.15.113:0/2995915190 wait complete. 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.816+0000 7fb555e44700 1 Processor -- start 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.816+0000 7fb555e44700 1 -- start start 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.816+0000 7fb555e44700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb55010e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.816+0000 7fb555e44700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb550071ce0 con 0x7fb5500fbed0 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.816+0000 7fb54f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb55010e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.816+0000 7fb54f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb55010e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38118/0 (socket says 172.21.15.113:38118) 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.816+0000 7fb54f7fe700 1 -- 172.21.15.113:0/305218040 learned_addr learned my addr 172.21.15.113:0/305218040 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.817+0000 7fb54f7fe700 1 -- 172.21.15.113:0/305218040 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb540009370 con 0x7fb5500fbed0 2024-04-12T04:59:31.820 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.817+0000 7fb54f7fe700 1 --2- 172.21.15.113:0/305218040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb55010e280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fb540000c00 tx=0x7fb540017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:31.821 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.817+0000 7fb54cff9700 1 -- 172.21.15.113:0/305218040 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb540011cc0 con 0x7fb5500fbed0 2024-04-12T04:59:31.821 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.817+0000 7fb54cff9700 1 -- 172.21.15.113:0/305218040 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540011e20 con 0x7fb5500fbed0 2024-04-12T04:59:31.821 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.817+0000 7fb555e44700 1 -- 172.21.15.113:0/305218040 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb550071ee0 con 0x7fb5500fbed0 2024-04-12T04:59:31.821 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.817+0000 7fb54cff9700 1 -- 172.21.15.113:0/305218040 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb54001d590 con 0x7fb5500fbed0 2024-04-12T04:59:31.821 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.817+0000 7fb555e44700 1 -- 172.21.15.113:0/305218040 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55010e930 con 0x7fb5500fbed0 2024-04-12T04:59:31.821 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.817+0000 7fb54cff9700 1 -- 172.21.15.113:0/305218040 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7fb54001da20 con 0x7fb5500fbed0 2024-04-12T04:59:31.821 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.817+0000 7fb54cff9700 1 -- 172.21.15.113:0/305218040 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb540026b50 con 0x7fb5500fbed0 2024-04-12T04:59:31.821 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.818+0000 7fb555e44700 1 -- 172.21.15.113:0/305218040 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb550046ce0 con 0x7fb5500fbed0 2024-04-12T04:59:31.822 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.820+0000 7fb54cff9700 1 -- 172.21.15.113:0/305218040 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7fb540021030 con 0x7fb5500fbed0 2024-04-12T04:59:31.885 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.883+0000 7fb555e44700 1 -- 172.21.15.113:0/305218040 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb5500fd210 con 0x7fb5500fbed0 2024-04-12T04:59:31.891 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.886+0000 7fb54cff9700 1 -- 172.21.15.113:0/305218040 <== mon.0 v2:172.21.15.113:3300/0 7 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fb540019840 con 0x7fb5500fbed0 2024-04-12T04:59:31.891 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.886+0000 7fb54cff9700 1 -- 172.21.15.113:0/305218040 <== mon.0 v2:172.21.15.113:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+456 (secure 0 0 0) 0x7fb540036430 con 0x7fb5500fbed0 2024-04-12T04:59:31.891 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.891 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: [global] 2024-04-12T04:59:31.891 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: fsid = 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:59:31.891 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: log_to_journald = false 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.113:3300,v1:172.21.15.113:6789] 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: [mgr] 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: [osd] 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-12T04:59:31.892 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.887+0000 7fb555e44700 1 -- 172.21.15.113:0/305218040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 msgr2=0x7fb55010e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:31.893 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.887+0000 7fb555e44700 1 --2- 172.21.15.113:0/305218040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb55010e280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fb540000c00 tx=0x7fb540017000).stop 2024-04-12T04:59:31.893 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.887+0000 7fb555e44700 1 -- 172.21.15.113:0/305218040 shutdown_connections 2024-04-12T04:59:31.893 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.887+0000 7fb555e44700 1 --2- 172.21.15.113:0/305218040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5500fbed0 0x7fb55010e280 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:31.893 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.887+0000 7fb555e44700 1 -- 172.21.15.113:0/305218040 >> 172.21.15.113:0/305218040 conn(0x7fb5500f76f0 msgr2=0x7fb5500fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:31.893 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.887+0000 7fb555e44700 1 -- 172.21.15.113:0/305218040 shutdown_connections 2024-04-12T04:59:31.893 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:31.887+0000 7fb555e44700 1 -- 172.21.15.113:0/305218040 wait complete. 2024-04-12T04:59:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: audit 2024-04-12T04:59:31.884600+0000 mon.smithi113 (mon.0) 8 : audit [INF] from='client.? 172.21.15.113:0/305218040' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-12T04:59:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: audit 2024-04-12T04:59:31.887674+0000 mon.smithi113 (mon.0) 9 : audit [INF] from='client.? 172.21.15.113:0/305218040' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-12T04:59:32.569 INFO:teuthology.orchestra.run.smithi113.stderr:Generating new minimal ceph.conf... 2024-04-12T04:59:33.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.070+0000 7fd16ec85700 1 Processor -- start 2024-04-12T04:59:33.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.070+0000 7fd16ec85700 1 -- start start 2024-04-12T04:59:33.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.070+0000 7fd16ec85700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd1680ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:33.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.070+0000 7fd16ec85700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd168100160 con 0x7fd168071c20 2024-04-12T04:59:33.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.071+0000 7fd16ca21700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd1680ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:33.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.071+0000 7fd16ca21700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd1680ffc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38128/0 (socket says 172.21.15.113:38128) 2024-04-12T04:59:33.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.071+0000 7fd16ca21700 1 -- 172.21.15.113:0/2521831020 learned_addr learned my addr 172.21.15.113:0/2521831020 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:33.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.071+0000 7fd16ca21700 1 -- 172.21.15.113:0/2521831020 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1681002a0 con 0x7fd168071c20 2024-04-12T04:59:33.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.071+0000 7fd16ca21700 1 --2- 172.21.15.113:0/2521831020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd1680ffc70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd15c009730 tx=0x7fd15c0090f0).ready entity=mon.0 client_cookie=eb07199e922d4688 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:33.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.071+0000 7fd1677fe700 1 -- 172.21.15.113:0/2521831020 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd15c00b770 con 0x7fd168071c20 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.071+0000 7fd1677fe700 1 -- 172.21.15.113:0/2521831020 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fd15c00bd20 con 0x7fd168071c20 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.071+0000 7fd1677fe700 1 -- 172.21.15.113:0/2521831020 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd15c011cc0 con 0x7fd168071c20 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.072+0000 7fd16ec85700 1 -- 172.21.15.113:0/2521831020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 msgr2=0x7fd1680ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.072+0000 7fd16ec85700 1 --2- 172.21.15.113:0/2521831020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd1680ffc70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd15c009730 tx=0x7fd15c0090f0).stop 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.072+0000 7fd16ec85700 1 -- 172.21.15.113:0/2521831020 shutdown_connections 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.072+0000 7fd16ec85700 1 --2- 172.21.15.113:0/2521831020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd1680ffc70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.072+0000 7fd16ec85700 1 -- 172.21.15.113:0/2521831020 >> 172.21.15.113:0/2521831020 conn(0x7fd1680f76f0 msgr2=0x7fd1680f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.072+0000 7fd16ec85700 1 -- 172.21.15.113:0/2521831020 shutdown_connections 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.072+0000 7fd16ec85700 1 -- 172.21.15.113:0/2521831020 wait complete. 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.072+0000 7fd16ec85700 1 Processor -- start 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd16ec85700 1 -- start start 2024-04-12T04:59:33.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd16ec85700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd16810be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd16ec85700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16810c320 con 0x7fd168071c20 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd16ca21700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd16810be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd16ca21700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd16810be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38134/0 (socket says 172.21.15.113:38134) 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd16ca21700 1 -- 172.21.15.113:0/1460671986 learned_addr learned my addr 172.21.15.113:0/1460671986 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd16ca21700 1 -- 172.21.15.113:0/1460671986 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c009370 con 0x7fd168071c20 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd16ca21700 1 --2- 172.21.15.113:0/1460671986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd16810be30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd15c011630 tx=0x7fd15c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd165ffb700 1 -- 172.21.15.113:0/1460671986 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd15c011ae0 con 0x7fd168071c20 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd165ffb700 1 -- 172.21.15.113:0/1460671986 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fd15c00be90 con 0x7fd168071c20 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd16ec85700 1 -- 172.21.15.113:0/1460671986 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16810c520 con 0x7fd168071c20 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.073+0000 7fd165ffb700 1 -- 172.21.15.113:0/1460671986 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd15c01aaf0 con 0x7fd168071c20 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.074+0000 7fd16ec85700 1 -- 172.21.15.113:0/1460671986 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16810c930 con 0x7fd168071c20 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.074+0000 7fd165ffb700 1 -- 172.21.15.113:0/1460671986 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7fd15c023660 con 0x7fd168071c20 2024-04-12T04:59:33.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.074+0000 7fd165ffb700 1 -- 172.21.15.113:0/1460671986 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd15c0223b0 con 0x7fd168071c20 2024-04-12T04:59:33.115 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.075+0000 7fd16ec85700 1 -- 172.21.15.113:0/1460671986 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd168046ce0 con 0x7fd168071c20 2024-04-12T04:59:33.115 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.077+0000 7fd165ffb700 1 -- 172.21.15.113:0/1460671986 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7fd15c01e020 con 0x7fd168071c20 2024-04-12T04:59:33.143 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.141+0000 7fd16ec85700 1 -- 172.21.15.113:0/1460671986 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd16810de80 con 0x7fd168071c20 2024-04-12T04:59:33.144 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.142+0000 7fd165ffb700 1 -- 172.21.15.113:0/1460671986 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fd15c011400 con 0x7fd168071c20 2024-04-12T04:59:33.145 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.142+0000 7fd16ec85700 1 -- 172.21.15.113:0/1460671986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 msgr2=0x7fd16810be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:33.145 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.142+0000 7fd16ec85700 1 --2- 172.21.15.113:0/1460671986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd16810be30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd15c011630 tx=0x7fd15c0090f0).stop 2024-04-12T04:59:33.145 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.142+0000 7fd16ec85700 1 -- 172.21.15.113:0/1460671986 shutdown_connections 2024-04-12T04:59:33.145 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.142+0000 7fd16ec85700 1 --2- 172.21.15.113:0/1460671986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd168071c20 0x7fd16810be30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:33.146 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.142+0000 7fd16ec85700 1 -- 172.21.15.113:0/1460671986 >> 172.21.15.113:0/1460671986 conn(0x7fd1680f76f0 msgr2=0x7fd1680f9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:33.146 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.142+0000 7fd16ec85700 1 -- 172.21.15.113:0/1460671986 shutdown_connections 2024-04-12T04:59:33.146 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:33.143+0000 7fd16ec85700 1 -- 172.21.15.113:0/1460671986 wait complete. 2024-04-12T04:59:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: audit 2024-04-12T04:59:33.142866+0000 mon.smithi113 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.113:0/1460671986' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T04:59:33.781 INFO:teuthology.orchestra.run.smithi113.stderr:Restarting the monitor... 2024-04-12T04:59:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:33 smithi113 systemd[1]: Stopping Ceph mon.smithi113 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T04:59:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:33 smithi113 bash[39504]: Error: no container with name or ID "ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon.smithi113" found: no such container 2024-04-12T04:59:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: debug 2024-04-12T04:59:34.026+0000 7f0b84670700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi113 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-12T04:59:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39179]: debug 2024-04-12T04:59:34.026+0000 7f0b84670700 -1 mon.smithi113@0(leader) e1 *** Got Signal Terminated *** 2024-04-12T04:59:34.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:34 smithi113 bash[39543]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113 2024-04-12T04:59:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:34 smithi113 systemd[1]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi113.service: Succeeded. 2024-04-12T04:59:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:34 smithi113 systemd[1]: Stopped Ceph mon.smithi113 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T04:59:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:34 smithi113 systemd[1]: Starting Ceph mon.smithi113 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T04:59:35.481 INFO:teuthology.orchestra.run.smithi113.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-12T04:59:35.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.418+0000 7fc49b22a880 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-12T04:59:35.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.418+0000 7fc49b22a880 0 ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable), process ceph-mon, pid 7 2024-04-12T04:59:35.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.418+0000 7fc49b22a880 0 pidfile_write: ignore empty --pid-file 2024-04-12T04:59:35.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.422+0000 7fc49b22a880 0 load: jerasure load: lrc load: isa 2024-04-12T04:59:35.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: RocksDB version: 6.8.1 2024-04-12T04:59:35.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-12T04:59:35.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Compile date Mar 4 2024 2024-04-12T04:59:35.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: DB SUMMARY 2024-04-12T04:59:35.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: CURRENT file: CURRENT 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: IDENTITY file: IDENTITY 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi113/store.db dir, Total Num: 1, files: 000004.sst 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi113/store.db: 000006.log size: 100030 ; 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.error_if_exists: 0 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.create_if_missing: 0 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.paranoid_checks: 1 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.env: 0x55bf4bf6a080 2024-04-12T04:59:35.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.fs: Posix File System 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.info_log: 0x55bf4d320560 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.statistics: (nil) 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.use_fsync: 0 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_log_file_size: 0 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.allow_fallocate: 1 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.use_direct_reads: 0 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T04:59:35.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.create_missing_column_families: 0 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.db_log_dir: 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi113/store.db 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_subcompactions: 1 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_background_flushes: -1 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.advise_random_on_open: 1 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.write_buffer_manager: 0x55bf4e0123f0 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T04:59:35.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.rate_limiter: (nil) 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.unordered_write: 0 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.row_cache: None 2024-04-12T04:59:35.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.wal_filter: None 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.preserve_deletes: 0 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.two_write_queues: 0 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.manual_wal_flush: 0 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.atomic_flush: 0 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.log_readahead_size: 0 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_background_jobs: 2 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_background_compactions: -1 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T04:59:35.680 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_total_wal_size: 0 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_open_files: -1 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.bytes_per_sync: 0 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Compression algorithms supported: 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T04:59:35.681 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: kZSTD supported: 0 2024-04-12T04:59:35.682 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: kXpressCompression supported: 0 2024-04-12T04:59:35.682 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-12T04:59:35.682 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: kLZ4Compression supported: 1 2024-04-12T04:59:35.682 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: kBZip2Compression supported: 0 2024-04-12T04:59:35.682 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: kZlibCompression supported: 1 2024-04-12T04:59:35.682 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: kSnappyCompression supported: 1 2024-04-12T04:59:35.682 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T04:59:35.683 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi113/store.db/MANIFEST-000005 2024-04-12T04:59:35.683 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.683 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-04-12T04:59:35.683 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.merge_operator: 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_filter: None 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_filter_factory: None 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bf4d2c5d10) 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cache_index_and_filter_blocks: 1 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: pin_top_level_index_and_filter: 1 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: index_type: 0 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: data_block_index_type: 0 2024-04-12T04:59:35.684 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: index_shortening: 1 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: data_block_hash_table_util_ratio: 0.750000 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: hash_index_allow_collision: 1 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: checksum: 1 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: no_block_cache: 0 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: block_cache: 0x55bf4d2faf10 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: block_cache_name: BinnedLRUCache 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: block_cache_options: 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: capacity : 536870912 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: num_shard_bits : 4 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: strict_capacity_limit : 0 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: high_pri_pool_ratio: 0.000 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: block_cache_compressed: (nil) 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: persistent_cache: (nil) 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: block_size: 4096 2024-04-12T04:59:35.685 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: block_size_deviation: 10 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: block_restart_interval: 16 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: index_block_restart_interval: 1 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: metadata_block_size: 4096 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: partition_filters: 0 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: use_delta_encoding: 1 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: whole_key_filtering: 1 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: verify_compression: 0 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: read_amp_bytes_per_bit: 0 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: format_version: 2 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: enable_index_compression: 1 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: block_align: 0 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.686 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compression: NoCompression 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.num_levels: 7 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T04:59:35.687 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T04:59:35.688 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T04:59:35.688 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T04:59:35.688 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compression_opts.level: 32767 2024-04-12T04:59:35.688 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-12T04:59:35.688 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T04:59:35.688 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T04:59:35.688 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compression_opts.enabled: false 2024-04-12T04:59:35.688 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T04:59:35.691 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T04:59:35.691 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T04:59:35.691 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-12T04:59:35.691 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T04:59:35.691 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T04:59:35.691 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T04:59:35.691 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.arena_block_size: 4194304 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-12T04:59:35.692 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.table_properties_collectors: 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.inplace_update_support: 0 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T04:59:35.693 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.bloom_locality: 0 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.max_successive_merges: 0 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.force_consistency_checks: 0 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.ttl: 2592000 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.423+0000 7fc49b22a880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.424+0000 7fc49b22a880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi113/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.424+0000 7fc49b22a880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.424+0000 7fc49b22a880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712897975426070, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-04-12T04:59:35.694 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.424+0000 7fc49b22a880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.424+0000 7fc49b22a880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.425+0000 7fc49b22a880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712897975426927, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 95409, "table_properties": {"data_size": 93504, "index_size": 382, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12918, "raw_average_key_size": 49, "raw_value_size": 86398, "raw_average_value_size": 332, "num_data_blocks": 12, "num_entries": 260, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712897975, "oldest_key_time": 3, "file_creation_time": 0}} 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.425+0000 7fc49b22a880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.425+0000 7fc49b22a880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712897975427342, "job": 1, "event": "recovery_finished"} 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.426+0000 7fc49b22a880 4 rocksdb: DB pointer 0x55bf4d3c3800 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.426+0000 7fc48201c700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.426+0000 7fc48201c700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: ** DB Stats ** 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T04:59:35.695 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: ** Compaction Stats [default] ** 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: L0 2/0 94.87 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.1 0.00 0.00 1 0.001 0 0 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Sum 2/0 94.87 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.1 0.00 0.00 1 0.001 0 0 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.1 0.00 0.00 1 0.001 0 0 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: ** Compaction Stats [default] ** 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 163.1 0.00 0.00 1 0.001 0 0 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T04:59:35.696 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: AddFile(Keys): cumulative 0, interval 0 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Cumulative compaction: 0.00 GB write, 33.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Interval compaction: 0.00 GB write, 33.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: ** File Read Latency Histogram By Level [default] ** 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: ** Compaction Stats [default] ** 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: L0 2/0 94.87 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.1 0.00 0.00 1 0.001 0 0 2024-04-12T04:59:35.697 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Sum 2/0 94.87 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.1 0.00 0.00 1 0.001 0 0 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: ** Compaction Stats [default] ** 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 163.1 0.00 0.00 1 0.001 0 0 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: AddFile(Keys): cumulative 0, interval 0 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Cumulative compaction: 0.00 GB write, 33.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T04:59:35.698 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: ** File Read Latency Histogram By Level [default] ** 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.426+0000 7fc49b22a880 0 starting mon.smithi113 rank 0 at public addrs [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] at bind addrs [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi113 fsid 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.427+0000 7fc49b22a880 1 mon.smithi113@-1(???) e1 preinit fsid 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.427+0000 7fc49b22a880 0 mon.smithi113@-1(???).mds e1 new map 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.427+0000 7fc49b22a880 0 mon.smithi113@-1(???).mds e1 print_map 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: e1 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: legacy client fscid: -1 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.699 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: No filesystems configured 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.427+0000 7fc49b22a880 0 mon.smithi113@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.427+0000 7fc49b22a880 0 mon.smithi113@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.427+0000 7fc49b22a880 0 mon.smithi113@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.427+0000 7fc49b22a880 0 mon.smithi113@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:33.142866+0000 mon.smithi113 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.113:0/1460671986' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T04:59:35.428+0000 7fc49b22a880 1 mon.smithi113@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 bash[39664]: e27d7c02f24223757b21bd66ab4d52f0c7dbb321f89372351af85115ae3b429f 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 systemd[1]: Started Ceph mon.smithi113 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:35.445622+0000 mon.smithi113 (mon.0) 1 : cluster [INF] mon.smithi113 is new leader, mons smithi113 in quorum (ranks 0) 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:35.445680+0000 mon.smithi113 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi113=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} removed_ranks: {} 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:35.446039+0000 mon.smithi113 (mon.0) 3 : cluster [DBG] fsmap 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:35.446066+0000 mon.smithi113 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-12T04:59:35.700 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:35.446407+0000 mon.smithi113 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-04-12T04:59:35.980 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.976+0000 7f892bb32700 1 Processor -- start 2024-04-12T04:59:35.985 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.977+0000 7f892bb32700 1 -- start start 2024-04-12T04:59:35.985 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.977+0000 7f892bb32700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240ffc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:35.985 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.977+0000 7f892bb32700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8924100120 con 0x7f8924071c60 2024-04-12T04:59:35.985 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.977+0000 7f89298ce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240ffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:35.985 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.977+0000 7f89298ce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240ffc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38144/0 (socket says 172.21.15.113:38144) 2024-04-12T04:59:35.985 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.977+0000 7f89298ce700 1 -- 172.21.15.113:0/34786390 learned_addr learned my addr 172.21.15.113:0/34786390 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:35.985 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.978+0000 7f89298ce700 1 -- 172.21.15.113:0/34786390 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8924100260 con 0x7f8924071c60 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.978+0000 7f89298ce700 1 --2- 172.21.15.113:0/34786390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240ffc30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8914009730 tx=0x7f89140090f0).ready entity=mon.0 client_cookie=5dd5372fbae0eab9 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.978+0000 7f89288cc700 1 -- 172.21.15.113:0/34786390 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f891400b6e0 con 0x7f8924071c60 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.978+0000 7f89288cc700 1 -- 172.21.15.113:0/34786390 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f891400bc90 con 0x7f8924071c60 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.978+0000 7f89288cc700 1 -- 172.21.15.113:0/34786390 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8914011c70 con 0x7f8924071c60 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.978+0000 7f892bb32700 1 -- 172.21.15.113:0/34786390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 msgr2=0x7f89240ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.978+0000 7f892bb32700 1 --2- 172.21.15.113:0/34786390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240ffc30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8914009730 tx=0x7f89140090f0).stop 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f892bb32700 1 -- 172.21.15.113:0/34786390 shutdown_connections 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f892bb32700 1 --2- 172.21.15.113:0/34786390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240ffc30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f892bb32700 1 -- 172.21.15.113:0/34786390 >> 172.21.15.113:0/34786390 conn(0x7f89240f76f0 msgr2=0x7f89240f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f892bb32700 1 -- 172.21.15.113:0/34786390 shutdown_connections 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f892bb32700 1 -- 172.21.15.113:0/34786390 wait complete. 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f892bb32700 1 Processor -- start 2024-04-12T04:59:35.986 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f892bb32700 1 -- start start 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f892bb32700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240fd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f892bb32700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240fbb30 con 0x7f8924071c60 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f89298ce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240fd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f89298ce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240fd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38146/0 (socket says 172.21.15.113:38146) 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.979+0000 7f89298ce700 1 -- 172.21.15.113:0/225350868 learned_addr learned my addr 172.21.15.113:0/225350868 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.980+0000 7f89298ce700 1 -- 172.21.15.113:0/225350868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8914009370 con 0x7f8924071c60 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.980+0000 7f89298ce700 1 --2- 172.21.15.113:0/225350868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240fd6b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f891400c010 tx=0x7f89140090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.980+0000 7f891affd700 1 -- 172.21.15.113:0/225350868 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f891401a610 con 0x7f8924071c60 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.980+0000 7f891affd700 1 -- 172.21.15.113:0/225350868 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f891401abc0 con 0x7f8924071c60 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.980+0000 7f892bb32700 1 -- 172.21.15.113:0/225350868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89240fbcd0 con 0x7f8924071c60 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.980+0000 7f891affd700 1 -- 172.21.15.113:0/225350868 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8914022a20 con 0x7f8924071c60 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.980+0000 7f892bb32700 1 -- 172.21.15.113:0/225350868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89240fc0e0 con 0x7f8924071c60 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.980+0000 7f891affd700 1 -- 172.21.15.113:0/225350868 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f8914023450 con 0x7f8924071c60 2024-04-12T04:59:35.987 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.980+0000 7f891affd700 1 -- 172.21.15.113:0/225350868 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8914019390 con 0x7f8924071c60 2024-04-12T04:59:35.988 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.981+0000 7f892bb32700 1 -- 172.21.15.113:0/225350868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8924046ce0 con 0x7f8924071c60 2024-04-12T04:59:35.988 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:35.983+0000 7f891affd700 1 -- 172.21.15.113:0/225350868 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f8914028060 con 0x7f8924071c60 2024-04-12T04:59:36.051 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:36.048+0000 7f892bb32700 1 -- 172.21.15.113:0/225350868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f8924105bd0 con 0x7f8924071c60 2024-04-12T04:59:36.053 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:36.050+0000 7f891affd700 1 -- 172.21.15.113:0/225350868 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f891401e070 con 0x7f8924071c60 2024-04-12T04:59:36.054 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:36.050+0000 7f892bb32700 1 -- 172.21.15.113:0/225350868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 msgr2=0x7f89240fd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:36.054 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:36.050+0000 7f892bb32700 1 --2- 172.21.15.113:0/225350868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240fd6b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f891400c010 tx=0x7f89140090f0).stop 2024-04-12T04:59:36.054 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:36.051+0000 7f892bb32700 1 -- 172.21.15.113:0/225350868 shutdown_connections 2024-04-12T04:59:36.054 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:36.051+0000 7f892bb32700 1 --2- 172.21.15.113:0/225350868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8924071c60 0x7f89240fd6b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:36.055 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:36.051+0000 7f892bb32700 1 -- 172.21.15.113:0/225350868 >> 172.21.15.113:0/225350868 conn(0x7f89240f76f0 msgr2=0x7f89240f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:36.055 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:36.051+0000 7f892bb32700 1 -- 172.21.15.113:0/225350868 shutdown_connections 2024-04-12T04:59:36.055 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:36.051+0000 7f892bb32700 1 -- 172.21.15.113:0/225350868 wait complete. 2024-04-12T04:59:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:36.052019+0000 mon.smithi113 (mon.0) 6 : audit [INF] from='client.? 172.21.15.113:0/225350868' entity='client.admin' 2024-04-12T04:59:37.713 INFO:teuthology.orchestra.run.smithi113.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-12T04:59:37.713 INFO:teuthology.orchestra.run.smithi113.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-12T04:59:37.713 INFO:teuthology.orchestra.run.smithi113.stderr:Creating mgr... 2024-04-12T04:59:37.714 INFO:teuthology.orchestra.run.smithi113.stderr:Verifying port 9283 ... 2024-04-12T04:59:37.845 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: Failed to reset failed state of unit ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mgr.smithi113.ctsazg.service: Unit ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mgr.smithi113.ctsazg.service not loaded. 2024-04-12T04:59:37.851 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: Created symlink /etc/systemd/system/ceph-52321e2c-f889-11ee-bc8c-c7b262605968.target.wants/ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mgr.smithi113.ctsazg.service → /etc/systemd/system/ceph-52321e2c-f889-11ee-bc8c-c7b262605968@.service. 2024-04-12T04:59:42.173 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: disabled 2024-04-12T04:59:42.181 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: inactive 2024-04-12T04:59:42.182 INFO:teuthology.orchestra.run.smithi113.stderr:firewalld.service is not enabled 2024-04-12T04:59:42.182 INFO:teuthology.orchestra.run.smithi113.stderr:Not possible to enable service . firewalld.service is not available 2024-04-12T04:59:42.189 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: disabled 2024-04-12T04:59:42.197 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: inactive 2024-04-12T04:59:42.198 INFO:teuthology.orchestra.run.smithi113.stderr:firewalld.service is not enabled 2024-04-12T04:59:42.198 INFO:teuthology.orchestra.run.smithi113.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-12T04:59:42.198 INFO:teuthology.orchestra.run.smithi113.stderr:Waiting for mgr to start... 2024-04-12T04:59:42.199 INFO:teuthology.orchestra.run.smithi113.stderr:Waiting for mgr... 2024-04-12T04:59:45.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.512+0000 7f3febdbd700 1 Processor -- start 2024-04-12T04:59:45.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.512+0000 7f3febdbd700 1 -- start start 2024-04-12T04:59:45.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.512+0000 7f3febdbd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe40ffc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:45.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.512+0000 7f3febdbd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe4100180 con 0x7f3fe4071c00 2024-04-12T04:59:45.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.512+0000 7f3fe9b59700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe40ffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:45.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.512+0000 7f3fe9b59700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe40ffc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38900/0 (socket says 172.21.15.113:38900) 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.512+0000 7f3fe9b59700 1 -- 172.21.15.113:0/2772728965 learned_addr learned my addr 172.21.15.113:0/2772728965 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.513+0000 7f3fe9b59700 1 -- 172.21.15.113:0/2772728965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe41002c0 con 0x7f3fe4071c00 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.513+0000 7f3fe9b59700 1 --2- 172.21.15.113:0/2772728965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe40ffc90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f3fe0009730 tx=0x7f3fe00090f0).ready entity=mon.0 client_cookie=8625916500450b57 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.513+0000 7f3fe8b57700 1 -- 172.21.15.113:0/2772728965 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe000b6e0 con 0x7f3fe4071c00 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.513+0000 7f3fe8b57700 1 -- 172.21.15.113:0/2772728965 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f3fe000bc90 con 0x7f3fe4071c00 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.513+0000 7f3fe8b57700 1 -- 172.21.15.113:0/2772728965 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe0011c70 con 0x7f3fe4071c00 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.514+0000 7f3febdbd700 1 -- 172.21.15.113:0/2772728965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 msgr2=0x7f3fe40ffc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.514+0000 7f3febdbd700 1 --2- 172.21.15.113:0/2772728965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe40ffc90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f3fe0009730 tx=0x7f3fe00090f0).stop 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.514+0000 7f3febdbd700 1 -- 172.21.15.113:0/2772728965 shutdown_connections 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.514+0000 7f3febdbd700 1 --2- 172.21.15.113:0/2772728965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe40ffc90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.514+0000 7f3febdbd700 1 -- 172.21.15.113:0/2772728965 >> 172.21.15.113:0/2772728965 conn(0x7f3fe40f76f0 msgr2=0x7f3fe40f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.514+0000 7f3febdbd700 1 -- 172.21.15.113:0/2772728965 shutdown_connections 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.514+0000 7f3febdbd700 1 -- 172.21.15.113:0/2772728965 wait complete. 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.514+0000 7f3febdbd700 1 Processor -- start 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.514+0000 7f3febdbd700 1 -- start start 2024-04-12T04:59:45.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.514+0000 7f3febdbd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe410be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.514+0000 7f3febdbd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe410c340 con 0x7f3fe4071c00 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3fe9b59700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe410be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3fe9b59700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe410be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38910/0 (socket says 172.21.15.113:38910) 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3fe9b59700 1 -- 172.21.15.113:0/3536191463 learned_addr learned my addr 172.21.15.113:0/3536191463 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3fe9b59700 1 -- 172.21.15.113:0/3536191463 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe0009370 con 0x7f3fe4071c00 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3fe9b59700 1 --2- 172.21.15.113:0/3536191463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe410be50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3fe0009330 tx=0x7f3fe00090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3fdaffd700 1 -- 172.21.15.113:0/3536191463 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe001a610 con 0x7f3fe4071c00 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3fdaffd700 1 -- 172.21.15.113:0/3536191463 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f3fe001abc0 con 0x7f3fe4071c00 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3febdbd700 1 -- 172.21.15.113:0/3536191463 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe410c540 con 0x7f3fe4071c00 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3fdaffd700 1 -- 172.21.15.113:0/3536191463 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe0022a30 con 0x7f3fe4071c00 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3febdbd700 1 -- 172.21.15.113:0/3536191463 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe410c950 con 0x7f3fe4071c00 2024-04-12T04:59:45.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3fdaffd700 1 -- 172.21.15.113:0/3536191463 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f3fe0023450 con 0x7f3fe4071c00 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.515+0000 7f3fdaffd700 1 -- 172.21.15.113:0/3536191463 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3fe0019390 con 0x7f3fe4071c00 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.516+0000 7f3febdbd700 1 -- 172.21.15.113:0/3536191463 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe4046ce0 con 0x7f3fe4071c00 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.518+0000 7f3fdaffd700 1 -- 172.21.15.113:0/3536191463 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f3fe0028060 con 0x7f3fe4071c00 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.587+0000 7f3febdbd700 1 -- 172.21.15.113:0/3536191463 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3fe4105d10 con 0x7f3fe4071c00 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.588+0000 7f3fdaffd700 1 -- 172.21.15.113:0/3536191463 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1250 (secure 0 0 0) 0x7f3fe001e070 con 0x7f3fe4071c00 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.588+0000 7f3febdbd700 1 -- 172.21.15.113:0/3536191463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 msgr2=0x7f3fe410be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.588+0000 7f3febdbd700 1 --2- 172.21.15.113:0/3536191463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe410be50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3fe0009330 tx=0x7f3fe00090f0).stop 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.588+0000 7f3febdbd700 1 -- 172.21.15.113:0/3536191463 shutdown_connections 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.588+0000 7f3febdbd700 1 --2- 172.21.15.113:0/3536191463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3fe4071c00 0x7f3fe410be50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.588+0000 7f3febdbd700 1 -- 172.21.15.113:0/3536191463 >> 172.21.15.113:0/3536191463 conn(0x7f3fe40f76f0 msgr2=0x7f3fe40f91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.589+0000 7f3febdbd700 1 -- 172.21.15.113:0/3536191463 shutdown_connections 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.589+0000 7f3febdbd700 1 -- 172.21.15.113:0/3536191463 wait complete. 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: { 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "fsid": "52321e2c-f889-11ee-bc8c-c7b262605968", 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "health": { 2024-04-12T04:59:45.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "checks": {}, 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "mutes": [] 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "quorum": [ 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 0 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: ], 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "smithi113" 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: ], 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "monmap": { 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:45.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "osdmap": { 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "pgmap": { 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-12T04:59:45.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "fsmap": { 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "available": false, 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "modules": [ 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "iostat", 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "nfs", 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "restful" 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: ], 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "services": {} 2024-04-12T04:59:45.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:45.879 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "servicemap": { 2024-04-12T04:59:45.879 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T04:59:45.879 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "modified": "2024-04-12T04:59:30.005485+0000", 2024-04-12T04:59:45.879 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "services": {} 2024-04-12T04:59:45.879 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:45.879 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-12T04:59:45.879 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: } 2024-04-12T04:59:45.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:45.588665+0000 mon.smithi113 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.113:0/3536191463' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T04:59:46.437 INFO:teuthology.orchestra.run.smithi113.stderr:mgr not available, waiting (1/15)... 2024-04-12T04:59:46.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:46.507894+0000 mon.smithi113 (mon.0) 8 : cluster [INF] Activating manager daemon smithi113.ctsazg 2024-04-12T04:59:46.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:46.510403+0000 mon.smithi113 (mon.0) 9 : cluster [DBG] mgrmap e2: smithi113.ctsazg(active, starting, since 0.00262791s) 2024-04-12T04:59:46.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:46.511414+0000 mon.smithi113 (mon.0) 10 : audit [DBG] from='mgr.14100 172.21.15.113:0/2965319854' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T04:59:46.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:46.511537+0000 mon.smithi113 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.113:0/2965319854' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T04:59:46.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:46.511651+0000 mon.smithi113 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.113:0/2965319854' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T04:59:46.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:46.511755+0000 mon.smithi113 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.113:0/2965319854' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T04:59:46.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:46.512054+0000 mon.smithi113 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.113:0/2965319854' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi113.ctsazg", "id": "smithi113.ctsazg"}]: dispatch 2024-04-12T04:59:46.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:46.513898+0000 mon.smithi113 (mon.0) 15 : cluster [INF] Manager daemon smithi113.ctsazg is now available 2024-04-12T04:59:46.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:46.522991+0000 mon.smithi113 (mon.0) 16 : audit [INF] from='mgr.14100 172.21.15.113:0/2965319854' entity='mgr.smithi113.ctsazg' 2024-04-12T04:59:46.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:46.528415+0000 mon.smithi113 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.113:0/2965319854' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T04:59:46.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:46.530931+0000 mon.smithi113 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.113:0/2965319854' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/trash_purge_schedule"}]: dispatch 2024-04-12T04:59:46.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:46.533203+0000 mon.smithi113 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.113:0/2965319854' entity='mgr.smithi113.ctsazg' 2024-04-12T04:59:46.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:46.535451+0000 mon.smithi113 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.113:0/2965319854' entity='mgr.smithi113.ctsazg' 2024-04-12T04:59:48.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:47.514087+0000 mon.smithi113 (mon.0) 21 : cluster [DBG] mgrmap e3: smithi113.ctsazg(active, since 1.00631s) 2024-04-12T04:59:49.039 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.036+0000 7f0983192700 1 Processor -- start 2024-04-12T04:59:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.036+0000 7f0983192700 1 -- start start 2024-04-12T04:59:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.036+0000 7f0983192700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c0fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.036+0000 7f0983192700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c0fc7e0 con 0x7f097c0fbed0 2024-04-12T04:59:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.037+0000 7f0980f2e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c0fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.037+0000 7f0980f2e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c0fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38960/0 (socket says 172.21.15.113:38960) 2024-04-12T04:59:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.037+0000 7f0980f2e700 1 -- 172.21.15.113:0/2669567681 learned_addr learned my addr 172.21.15.113:0/2669567681 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.037+0000 7f0980f2e700 1 -- 172.21.15.113:0/2669567681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c0fc920 con 0x7f097c0fbed0 2024-04-12T04:59:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.037+0000 7f0980f2e700 1 --2- 172.21.15.113:0/2669567681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c0fc2f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f0970009730 tx=0x7f09700090f0).ready entity=mon.0 client_cookie=916a5521f24f6288 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.037+0000 7f097b7fe700 1 -- 172.21.15.113:0/2669567681 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f097000b6e0 con 0x7f097c0fbed0 2024-04-12T04:59:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.037+0000 7f097b7fe700 1 -- 172.21.15.113:0/2669567681 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f097000bc90 con 0x7f097c0fbed0 2024-04-12T04:59:49.040 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.037+0000 7f097b7fe700 1 -- 172.21.15.113:0/2669567681 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0970011c70 con 0x7f097c0fbed0 2024-04-12T04:59:49.043 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.038+0000 7f0983192700 1 -- 172.21.15.113:0/2669567681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 msgr2=0x7f097c0fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:49.043 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.038+0000 7f0983192700 1 --2- 172.21.15.113:0/2669567681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c0fc2f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f0970009730 tx=0x7f09700090f0).stop 2024-04-12T04:59:49.043 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.038+0000 7f0983192700 1 -- 172.21.15.113:0/2669567681 shutdown_connections 2024-04-12T04:59:49.043 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.038+0000 7f0983192700 1 --2- 172.21.15.113:0/2669567681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c0fc2f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:49.043 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.038+0000 7f0983192700 1 -- 172.21.15.113:0/2669567681 >> 172.21.15.113:0/2669567681 conn(0x7f097c0f76f0 msgr2=0x7f097c0f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:49.043 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.038+0000 7f0983192700 1 -- 172.21.15.113:0/2669567681 shutdown_connections 2024-04-12T04:59:49.043 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.038+0000 7f0983192700 1 -- 172.21.15.113:0/2669567681 wait complete. 2024-04-12T04:59:49.043 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.038+0000 7f0983192700 1 Processor -- start 2024-04-12T04:59:49.043 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.038+0000 7f0983192700 1 -- start start 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.039+0000 7f0983192700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c1020f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.039+0000 7f0983192700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c1005e0 con 0x7f097c0fbed0 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.039+0000 7f0980f2e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c1020f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.039+0000 7f0980f2e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c1020f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38968/0 (socket says 172.21.15.113:38968) 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.039+0000 7f0980f2e700 1 -- 172.21.15.113:0/2603269966 learned_addr learned my addr 172.21.15.113:0/2603269966 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.039+0000 7f0980f2e700 1 -- 172.21.15.113:0/2603269966 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0970009370 con 0x7f097c0fbed0 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.039+0000 7f0980f2e700 1 --2- 172.21.15.113:0/2603269966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c1020f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0970011870 tx=0x7f09700090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.039+0000 7f0979ffb700 1 -- 172.21.15.113:0/2603269966 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f097001a610 con 0x7f097c0fbed0 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.039+0000 7f0979ffb700 1 -- 172.21.15.113:0/2603269966 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f097001abc0 con 0x7f097c0fbed0 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.039+0000 7f0983192700 1 -- 172.21.15.113:0/2603269966 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097c1007e0 con 0x7f097c0fbed0 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.039+0000 7f0979ffb700 1 -- 172.21.15.113:0/2603269966 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0970022a30 con 0x7f097c0fbed0 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.040+0000 7f0983192700 1 -- 172.21.15.113:0/2603269966 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097c100bf0 con 0x7f097c0fbed0 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.040+0000 7f0979ffb700 1 -- 172.21.15.113:0/2603269966 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 38515+0+0 (secure 0 0 0) 0x7f0970022ca0 con 0x7f097c0fbed0 2024-04-12T04:59:49.044 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.040+0000 7f0979ffb700 1 --2- 172.21.15.113:0/2603269966 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f096c0324e0 0x7f096c0349a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:49.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.040+0000 7f0979ffb700 1 -- 172.21.15.113:0/2603269966 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f09700445f0 con 0x7f097c0fbed0 2024-04-12T04:59:49.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.041+0000 7f097bfff700 1 --2- 172.21.15.113:0/2603269966 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f096c0324e0 0x7f096c0349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:49.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.041+0000 7f0983192700 1 -- 172.21.15.113:0/2603269966 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097c046ce0 con 0x7f097c0fbed0 2024-04-12T04:59:49.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.041+0000 7f097bfff700 1 --2- 172.21.15.113:0/2603269966 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f096c0324e0 0x7f096c0349a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0968009780 tx=0x7f0968006cc0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:49.048 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.046+0000 7f0979ffb700 1 -- 172.21.15.113:0/2603269966 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0970018970 con 0x7f097c0fbed0 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.272+0000 7f0983192700 1 -- 172.21.15.113:0/2603269966 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f097c0fdc20 con 0x7f097c0fbed0 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.273+0000 7f0979ffb700 1 -- 172.21.15.113:0/2603269966 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f097c0fdc20 con 0x7f097c0fbed0 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: { 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "fsid": "52321e2c-f889-11ee-bc8c-c7b262605968", 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "health": { 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "checks": {}, 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "mutes": [] 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-12T04:59:49.277 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "quorum": [ 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 0 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: ], 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "smithi113" 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: ], 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "quorum_age": 13, 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "monmap": { 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "osdmap": { 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-12T04:59:49.278 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "pgmap": { 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-12T04:59:49.283 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "fsmap": { 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "available": true, 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "modules": [ 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "iostat", 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "nfs", 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "restful" 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: ], 2024-04-12T04:59:49.284 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "services": {} 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "servicemap": { 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "modified": "2024-04-12T04:59:30.005485+0000", 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "services": {} 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: }, 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: } 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.275+0000 7f0983192700 1 -- 172.21.15.113:0/2603269966 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f096c0324e0 msgr2=0x7f096c0349a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.275+0000 7f0983192700 1 --2- 172.21.15.113:0/2603269966 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f096c0324e0 0x7f096c0349a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0968009780 tx=0x7f0968006cc0).stop 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.275+0000 7f0983192700 1 -- 172.21.15.113:0/2603269966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 msgr2=0x7f097c1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.275+0000 7f0983192700 1 --2- 172.21.15.113:0/2603269966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c1020f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0970011870 tx=0x7f09700090f0).stop 2024-04-12T04:59:49.285 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.275+0000 7f0983192700 1 -- 172.21.15.113:0/2603269966 shutdown_connections 2024-04-12T04:59:49.286 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.275+0000 7f0983192700 1 --2- 172.21.15.113:0/2603269966 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f096c0324e0 0x7f096c0349a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:49.286 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.275+0000 7f0983192700 1 --2- 172.21.15.113:0/2603269966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f097c0fbed0 0x7f097c1020f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:49.286 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.275+0000 7f0983192700 1 -- 172.21.15.113:0/2603269966 >> 172.21.15.113:0/2603269966 conn(0x7f097c0f76f0 msgr2=0x7f097c071890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:49.286 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.275+0000 7f0983192700 1 -- 172.21.15.113:0/2603269966 shutdown_connections 2024-04-12T04:59:49.286 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:49.275+0000 7f0983192700 1 -- 172.21.15.113:0/2603269966 wait complete. 2024-04-12T04:59:49.615 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:49.273640+0000 mon.smithi113 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.113:0/2603269966' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T04:59:49.935 INFO:teuthology.orchestra.run.smithi113.stderr:mgr is available 2024-04-12T04:59:50.488 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.482+0000 7f277e773700 1 Processor -- start 2024-04-12T04:59:50.488 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.482+0000 7f277e773700 1 -- start start 2024-04-12T04:59:50.488 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.482+0000 7f277e773700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f27780fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:50.488 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.482+0000 7f277e773700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27780fc7e0 con 0x7f27780fbed0 2024-04-12T04:59:50.488 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.482+0000 7f2777fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f27780fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:50.488 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.482+0000 7f2777fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f27780fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58818/0 (socket says 172.21.15.113:58818) 2024-04-12T04:59:50.488 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.482+0000 7f2777fff700 1 -- 172.21.15.113:0/4140658224 learned_addr learned my addr 172.21.15.113:0/4140658224 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:50.488 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.483+0000 7f2777fff700 1 -- 172.21.15.113:0/4140658224 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27780fc920 con 0x7f27780fbed0 2024-04-12T04:59:50.488 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.483+0000 7f2777fff700 1 --2- 172.21.15.113:0/4140658224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f27780fc2f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f2760009730 tx=0x7f27600090f0).ready entity=mon.0 client_cookie=b8377455a0926d03 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:50.488 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.483+0000 7f2776ffd700 1 -- 172.21.15.113:0/4140658224 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f276000b6e0 con 0x7f27780fbed0 2024-04-12T04:59:50.488 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.483+0000 7f2776ffd700 1 -- 172.21.15.113:0/4140658224 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f276000bc90 con 0x7f27780fbed0 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.483+0000 7f2776ffd700 1 -- 172.21.15.113:0/4140658224 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2760011c70 con 0x7f27780fbed0 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.483+0000 7f277e773700 1 -- 172.21.15.113:0/4140658224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 msgr2=0x7f27780fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.483+0000 7f277e773700 1 --2- 172.21.15.113:0/4140658224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f27780fc2f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f2760009730 tx=0x7f27600090f0).stop 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.484+0000 7f277e773700 1 -- 172.21.15.113:0/4140658224 shutdown_connections 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.484+0000 7f277e773700 1 --2- 172.21.15.113:0/4140658224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f27780fc2f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.484+0000 7f277e773700 1 -- 172.21.15.113:0/4140658224 >> 172.21.15.113:0/4140658224 conn(0x7f27780f76f0 msgr2=0x7f27780f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.484+0000 7f277e773700 1 -- 172.21.15.113:0/4140658224 shutdown_connections 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.484+0000 7f277e773700 1 -- 172.21.15.113:0/4140658224 wait complete. 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.484+0000 7f277e773700 1 Processor -- start 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.484+0000 7f277e773700 1 -- start start 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.484+0000 7f277e773700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f2778102160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.484+0000 7f277e773700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27781005e0 con 0x7f27780fbed0 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.485+0000 7f2777fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f2778102160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:50.489 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.485+0000 7f2777fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f2778102160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58832/0 (socket says 172.21.15.113:58832) 2024-04-12T04:59:50.490 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.485+0000 7f2777fff700 1 -- 172.21.15.113:0/907554030 learned_addr learned my addr 172.21.15.113:0/907554030 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:50.490 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.485+0000 7f2777fff700 1 -- 172.21.15.113:0/907554030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2760009370 con 0x7f27780fbed0 2024-04-12T04:59:50.490 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.485+0000 7f2777fff700 1 --2- 172.21.15.113:0/907554030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f2778102160 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f27600115c0 tx=0x7f27600090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:50.490 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.485+0000 7f27757fa700 1 -- 172.21.15.113:0/907554030 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f276001a610 con 0x7f27780fbed0 2024-04-12T04:59:50.490 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.485+0000 7f277e773700 1 -- 172.21.15.113:0/907554030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2778100780 con 0x7f27780fbed0 2024-04-12T04:59:50.490 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.485+0000 7f27757fa700 1 -- 172.21.15.113:0/907554030 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f276001abc0 con 0x7f27780fbed0 2024-04-12T04:59:50.490 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.485+0000 7f27757fa700 1 -- 172.21.15.113:0/907554030 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2760022a40 con 0x7f27780fbed0 2024-04-12T04:59:50.490 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.485+0000 7f277e773700 1 -- 172.21.15.113:0/907554030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2778100b90 con 0x7f27780fbed0 2024-04-12T04:59:50.491 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.486+0000 7f27757fa700 1 -- 172.21.15.113:0/907554030 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 4) v1 ==== 38621+0+0 (secure 0 0 0) 0x7f2760022d00 con 0x7f27780fbed0 2024-04-12T04:59:50.491 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.486+0000 7f27757fa700 1 --2- 172.21.15.113:0/907554030 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2764032590 0x7f2764034a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:50.491 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.486+0000 7f27757fa700 1 -- 172.21.15.113:0/907554030 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2760044790 con 0x7f27780fbed0 2024-04-12T04:59:50.491 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.486+0000 7f27777fe700 1 --2- 172.21.15.113:0/907554030 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2764032590 0x7f2764034a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:50.491 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.486+0000 7f277e773700 1 -- 172.21.15.113:0/907554030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f277805a310 con 0x7f27780fbed0 2024-04-12T04:59:50.491 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.487+0000 7f27777fe700 1 --2- 172.21.15.113:0/907554030 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2764032590 0x7f2764034a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2768009780 tx=0x7f2768006cc0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:50.493 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.491+0000 7f27757fa700 1 -- 172.21.15.113:0/907554030 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f276001e070 con 0x7f27780fbed0 2024-04-12T04:59:50.654 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.652+0000 7f277e773700 1 -- 172.21.15.113:0/907554030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f2778046ce0 con 0x7f27780fbed0 2024-04-12T04:59:50.658 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.653+0000 7f27757fa700 1 -- 172.21.15.113:0/907554030 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7f2760018b70 con 0x7f27780fbed0 2024-04-12T04:59:50.658 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.658 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: [global] 2024-04-12T04:59:50.658 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: fsid = 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T04:59:50.658 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: log_to_journald = false 2024-04-12T04:59:50.658 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-12T04:59:50.658 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-12T04:59:50.658 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: [mgr] 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: [osd] 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.654+0000 7f277e773700 1 -- 172.21.15.113:0/907554030 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2764032590 msgr2=0x7f2764034a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.655+0000 7f277e773700 1 --2- 172.21.15.113:0/907554030 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2764032590 0x7f2764034a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2768009780 tx=0x7f2768006cc0).stop 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.655+0000 7f277e773700 1 -- 172.21.15.113:0/907554030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 msgr2=0x7f2778102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:50.659 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.655+0000 7f277e773700 1 --2- 172.21.15.113:0/907554030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f2778102160 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f27600115c0 tx=0x7f27600090f0).stop 2024-04-12T04:59:50.660 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.655+0000 7f277e773700 1 -- 172.21.15.113:0/907554030 shutdown_connections 2024-04-12T04:59:50.660 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.655+0000 7f277e773700 1 --2- 172.21.15.113:0/907554030 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2764032590 0x7f2764034a50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:50.660 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.655+0000 7f277e773700 1 --2- 172.21.15.113:0/907554030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27780fbed0 0x7f2778102160 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:50.660 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.655+0000 7f277e773700 1 -- 172.21.15.113:0/907554030 >> 172.21.15.113:0/907554030 conn(0x7f27780f76f0 msgr2=0x7f2778071890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:50.660 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.655+0000 7f277e773700 1 -- 172.21.15.113:0/907554030 shutdown_connections 2024-04-12T04:59:50.660 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:50.655+0000 7f277e773700 1 -- 172.21.15.113:0/907554030 wait complete. 2024-04-12T04:59:50.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:49.513329+0000 mon.smithi113 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi113.ctsazg(active, since 3s) 2024-04-12T04:59:51.342 INFO:teuthology.orchestra.run.smithi113.stderr:Enabling cephadm module... 2024-04-12T04:59:51.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:50.653614+0000 mon.smithi113 (mon.0) 24 : audit [INF] from='client.? 172.21.15.113:0/907554030' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-12T04:59:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.857+0000 7f2c42552700 1 Processor -- start 2024-04-12T04:59:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.857+0000 7f2c42552700 1 -- start start 2024-04-12T04:59:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.857+0000 7f2c42552700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c0fc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.857+0000 7f2c42552700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c0fc820 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.858+0000 7f2c3bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c0fc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.858+0000 7f2c3bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c0fc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58842/0 (socket says 172.21.15.113:58842) 2024-04-12T04:59:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.858+0000 7f2c3bfff700 1 -- 172.21.15.113:0/3960185812 learned_addr learned my addr 172.21.15.113:0/3960185812 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.858+0000 7f2c3bfff700 1 -- 172.21.15.113:0/3960185812 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c0fc960 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.858+0000 7f2c3bfff700 1 --2- 172.21.15.113:0/3960185812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c0fc330 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f2c2c009730 tx=0x7f2c2c0090f0).ready entity=mon.0 client_cookie=fb527245d652ae2a server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.858+0000 7f2c3affd700 1 -- 172.21.15.113:0/3960185812 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c2c00b6e0 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.862 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.858+0000 7f2c3affd700 1 -- 172.21.15.113:0/3960185812 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f2c2c00bc90 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.863 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.858+0000 7f2c3affd700 1 -- 172.21.15.113:0/3960185812 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c2c011c70 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.863 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.859+0000 7f2c42552700 1 -- 172.21.15.113:0/3960185812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 msgr2=0x7f2c3c0fc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:51.863 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.859+0000 7f2c42552700 1 --2- 172.21.15.113:0/3960185812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c0fc330 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f2c2c009730 tx=0x7f2c2c0090f0).stop 2024-04-12T04:59:51.863 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.859+0000 7f2c42552700 1 -- 172.21.15.113:0/3960185812 shutdown_connections 2024-04-12T04:59:51.863 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.859+0000 7f2c42552700 1 --2- 172.21.15.113:0/3960185812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c0fc330 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:51.863 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.859+0000 7f2c42552700 1 -- 172.21.15.113:0/3960185812 >> 172.21.15.113:0/3960185812 conn(0x7f2c3c0f7750 msgr2=0x7f2c3c0f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:51.863 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.859+0000 7f2c42552700 1 -- 172.21.15.113:0/3960185812 shutdown_connections 2024-04-12T04:59:51.863 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.859+0000 7f2c42552700 1 -- 172.21.15.113:0/3960185812 wait complete. 2024-04-12T04:59:51.863 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.860+0000 7f2c42552700 1 Processor -- start 2024-04-12T04:59:51.863 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.860+0000 7f2c42552700 1 -- start start 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.860+0000 7f2c42552700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c10ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.860+0000 7f2c42552700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c071ca0 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.860+0000 7f2c3bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c10ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.860+0000 7f2c3bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c10ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58852/0 (socket says 172.21.15.113:58852) 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.860+0000 7f2c3bfff700 1 -- 172.21.15.113:0/2971294252 learned_addr learned my addr 172.21.15.113:0/2971294252 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.860+0000 7f2c3bfff700 1 -- 172.21.15.113:0/2971294252 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c009370 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.860+0000 7f2c3bfff700 1 --2- 172.21.15.113:0/2971294252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c10ea70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f2c2c002f40 tx=0x7f2c2c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.861+0000 7f2c397fa700 1 -- 172.21.15.113:0/2971294252 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c2c01a610 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.861+0000 7f2c42552700 1 -- 172.21.15.113:0/2971294252 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3c071ea0 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.861+0000 7f2c397fa700 1 -- 172.21.15.113:0/2971294252 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f2c2c01abc0 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.861+0000 7f2c397fa700 1 -- 172.21.15.113:0/2971294252 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c2c0229d0 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.861+0000 7f2c42552700 1 -- 172.21.15.113:0/2971294252 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3c10f120 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.861+0000 7f2c397fa700 1 -- 172.21.15.113:0/2971294252 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 4) v1 ==== 38621+0+0 (secure 0 0 0) 0x7f2c2c022c90 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.862+0000 7f2c397fa700 1 --2- 172.21.15.113:0/2971294252 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2c280325c0 0x7f2c28034a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:51.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.862+0000 7f2c397fa700 1 -- 172.21.15.113:0/2971294252 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2c2c044750 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.862+0000 7f2c42552700 1 -- 172.21.15.113:0/2971294252 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3c046ce0 con 0x7f2c3c0fbf10 2024-04-12T04:59:51.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.862+0000 7f2c3b7fe700 1 --2- 172.21.15.113:0/2971294252 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2c280325c0 0x7f2c28034a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:51.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.862+0000 7f2c3b7fe700 1 --2- 172.21.15.113:0/2971294252 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2c280325c0 0x7f2c28034a80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2c24009780 tx=0x7f2c24006cc0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:51.868 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:51.866+0000 7f2c397fa700 1 -- 172.21.15.113:0/2971294252 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2c2c018970 con 0x7f2c3c0fbf10 2024-04-12T04:59:52.069 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.067+0000 7f2c42552700 1 -- 172.21.15.113:0/2971294252 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f2c3c0fdd50 con 0x7f2c3c0fbf10 2024-04-12T04:59:52.523 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.520+0000 7f2c397fa700 1 -- 172.21.15.113:0/2971294252 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 5) v1 ==== 38632+0+0 (secure 0 0 0) 0x7f2c2c0198b0 con 0x7f2c3c0fbf10 2024-04-12T04:59:52.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.521+0000 7f2c397fa700 1 -- 172.21.15.113:0/2971294252 <== mon.0 v2:172.21.15.113:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f2c2c0219b0 con 0x7f2c3c0fbf10 2024-04-12T04:59:52.527 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.524+0000 7f2c42552700 1 -- 172.21.15.113:0/2971294252 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2c280325c0 msgr2=0x7f2c28034a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:52.527 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.524+0000 7f2c42552700 1 --2- 172.21.15.113:0/2971294252 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2c280325c0 0x7f2c28034a80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2c24009780 tx=0x7f2c24006cc0).stop 2024-04-12T04:59:52.527 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.524+0000 7f2c42552700 1 -- 172.21.15.113:0/2971294252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 msgr2=0x7f2c3c10ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:52.527 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.524+0000 7f2c42552700 1 --2- 172.21.15.113:0/2971294252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c10ea70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f2c2c002f40 tx=0x7f2c2c0090f0).stop 2024-04-12T04:59:52.527 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.524+0000 7f2c42552700 1 -- 172.21.15.113:0/2971294252 shutdown_connections 2024-04-12T04:59:52.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.524+0000 7f2c42552700 1 --2- 172.21.15.113:0/2971294252 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f2c280325c0 0x7f2c28034a80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:52.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.524+0000 7f2c42552700 1 --2- 172.21.15.113:0/2971294252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c3c0fbf10 0x7f2c3c10ea70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:52.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.524+0000 7f2c42552700 1 -- 172.21.15.113:0/2971294252 >> 172.21.15.113:0/2971294252 conn(0x7f2c3c0f7750 msgr2=0x7f2c3c0f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:52.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.524+0000 7f2c42552700 1 -- 172.21.15.113:0/2971294252 shutdown_connections 2024-04-12T04:59:52.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:52.524+0000 7f2c42552700 1 -- 172.21.15.113:0/2971294252 wait complete. 2024-04-12T04:59:52.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:52.068581+0000 mon.smithi113 (mon.0) 25 : audit [INF] from='client.? 172.21.15.113:0/2971294252' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-12T04:59:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.805+0000 7fc856c30700 1 Processor -- start 2024-04-12T04:59:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.805+0000 7fc856c30700 1 -- start start 2024-04-12T04:59:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.805+0000 7fc856c30700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc8500ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.805+0000 7fc856c30700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc850071fa0 con 0x7fc850071b80 2024-04-12T04:59:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.805+0000 7fc8549cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc8500ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.805+0000 7fc8549cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc8500ffd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58880/0 (socket says 172.21.15.113:58880) 2024-04-12T04:59:53.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.806+0000 7fc8549cc700 1 -- 172.21.15.113:0/3796890849 learned_addr learned my addr 172.21.15.113:0/3796890849 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:53.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.806+0000 7fc8549cc700 1 -- 172.21.15.113:0/3796890849 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc850100200 con 0x7fc850071b80 2024-04-12T04:59:53.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.806+0000 7fc8549cc700 1 --2- 172.21.15.113:0/3796890849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc8500ffd10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc844009730 tx=0x7fc8440090f0).ready entity=mon.0 client_cookie=b9c08cb239aaced3 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:53.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.806+0000 7fc84f7fe700 1 -- 172.21.15.113:0/3796890849 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc84400b6e0 con 0x7fc850071b80 2024-04-12T04:59:53.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.806+0000 7fc84f7fe700 1 -- 172.21.15.113:0/3796890849 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fc84400bc90 con 0x7fc850071b80 2024-04-12T04:59:53.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.806+0000 7fc84f7fe700 1 -- 172.21.15.113:0/3796890849 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc844011c70 con 0x7fc850071b80 2024-04-12T04:59:53.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.807+0000 7fc856c30700 1 -- 172.21.15.113:0/3796890849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 msgr2=0x7fc8500ffd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:53.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.807+0000 7fc856c30700 1 --2- 172.21.15.113:0/3796890849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc8500ffd10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc844009730 tx=0x7fc8440090f0).stop 2024-04-12T04:59:53.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.807+0000 7fc856c30700 1 -- 172.21.15.113:0/3796890849 shutdown_connections 2024-04-12T04:59:53.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.807+0000 7fc856c30700 1 --2- 172.21.15.113:0/3796890849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc8500ffd10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:53.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.807+0000 7fc856c30700 1 -- 172.21.15.113:0/3796890849 >> 172.21.15.113:0/3796890849 conn(0x7fc8500f76f0 msgr2=0x7fc8500f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:53.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.807+0000 7fc856c30700 1 -- 172.21.15.113:0/3796890849 shutdown_connections 2024-04-12T04:59:53.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.807+0000 7fc856c30700 1 -- 172.21.15.113:0/3796890849 wait complete. 2024-04-12T04:59:53.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.808+0000 7fc856c30700 1 Processor -- start 2024-04-12T04:59:53.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.808+0000 7fc856c30700 1 -- start start 2024-04-12T04:59:53.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.808+0000 7fc856c30700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc85010be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:53.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.808+0000 7fc856c30700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85010c2f0 con 0x7fc850071b80 2024-04-12T04:59:53.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.808+0000 7fc8549cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc85010be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:53.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.808+0000 7fc8549cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc85010be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58894/0 (socket says 172.21.15.113:58894) 2024-04-12T04:59:53.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.808+0000 7fc8549cc700 1 -- 172.21.15.113:0/3740841434 learned_addr learned my addr 172.21.15.113:0/3740841434 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:53.813 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.808+0000 7fc8549cc700 1 -- 172.21.15.113:0/3740841434 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844009370 con 0x7fc850071b80 2024-04-12T04:59:53.813 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.808+0000 7fc8549cc700 1 --2- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc85010be00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fc84400c010 tx=0x7fc8440090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:53.813 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.809+0000 7fc84dffb700 1 -- 172.21.15.113:0/3740841434 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc84401a610 con 0x7fc850071b80 2024-04-12T04:59:53.813 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.809+0000 7fc84dffb700 1 -- 172.21.15.113:0/3740841434 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fc84401abc0 con 0x7fc850071b80 2024-04-12T04:59:53.813 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.809+0000 7fc84dffb700 1 -- 172.21.15.113:0/3740841434 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc844022a20 con 0x7fc850071b80 2024-04-12T04:59:53.814 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.809+0000 7fc856c30700 1 -- 172.21.15.113:0/3740841434 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85010c4f0 con 0x7fc850071b80 2024-04-12T04:59:53.814 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.809+0000 7fc856c30700 1 -- 172.21.15.113:0/3740841434 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85010c900 con 0x7fc850071b80 2024-04-12T04:59:53.814 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.810+0000 7fc84dffb700 1 -- 172.21.15.113:0/3740841434 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 38632+0+0 (secure 0 0 0) 0x7fc844022c90 con 0x7fc850071b80 2024-04-12T04:59:53.814 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.810+0000 7fc84dffb700 1 --2- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7fc840032660 0x7fc840034b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:53.814 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.810+0000 7fc84dffb700 1 -- 172.21.15.113:0/3740841434 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc844044820 con 0x7fc850071b80 2024-04-12T04:59:53.814 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.810+0000 7fc84ffff700 1 -- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7fc840032660 msgr2=0x7fc840034b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/3275382936 2024-04-12T04:59:53.814 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.810+0000 7fc84ffff700 1 --2- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7fc840032660 0x7fc840034b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-04-12T04:59:53.814 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.810+0000 7fc856c30700 1 -- 172.21.15.113:0/3740841434 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc850046ce0 con 0x7fc850071b80 2024-04-12T04:59:53.817 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:53.815+0000 7fc84dffb700 1 -- 172.21.15.113:0/3740841434 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc84401a770 con 0x7fc850071b80 2024-04-12T04:59:53.906 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:52.522478+0000 mon.smithi113 (mon.0) 26 : audit [INF] from='client.? 172.21.15.113:0/2971294252' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-12T04:59:53.906 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:52.522525+0000 mon.smithi113 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi113.ctsazg(active, since 6s) 2024-04-12T04:59:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.011+0000 7fc84ffff700 1 -- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7fc840032660 msgr2=0x7fc840034b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/3275382936 2024-04-12T04:59:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.011+0000 7fc84ffff700 1 --2- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7fc840032660 0x7fc840034b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-04-12T04:59:54.067 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.064+0000 7fc856c30700 1 -- 172.21.15.113:0/3740841434 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc85010d8d0 con 0x7fc850071b80 2024-04-12T04:59:54.068 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.065+0000 7fc84dffb700 1 -- 172.21.15.113:0/3740841434 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fc84401e030 con 0x7fc850071b80 2024-04-12T04:59:54.068 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: { 2024-04-12T04:59:54.068 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-12T04:59:54.068 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "available": true, 2024-04-12T04:59:54.068 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "active_name": "smithi113.ctsazg", 2024-04-12T04:59:54.068 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-12T04:59:54.068 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: } 2024-04-12T04:59:54.069 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.066+0000 7fc856c30700 1 -- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7fc840032660 msgr2=0x7fc840034b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T04:59:54.070 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.067+0000 7fc856c30700 1 --2- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7fc840032660 0x7fc840034b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T04:59:54.070 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.067+0000 7fc856c30700 1 -- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 msgr2=0x7fc85010be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:54.070 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.067+0000 7fc856c30700 1 --2- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc85010be00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fc84400c010 tx=0x7fc8440090f0).stop 2024-04-12T04:59:54.070 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.067+0000 7fc856c30700 1 -- 172.21.15.113:0/3740841434 shutdown_connections 2024-04-12T04:59:54.070 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.067+0000 7fc856c30700 1 --2- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7fc840032660 0x7fc840034b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T04:59:54.070 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.067+0000 7fc856c30700 1 --2- 172.21.15.113:0/3740841434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc850071b80 0x7fc85010be00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:54.070 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.067+0000 7fc856c30700 1 -- 172.21.15.113:0/3740841434 >> 172.21.15.113:0/3740841434 conn(0x7fc8500f76f0 msgr2=0x7fc8500f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:54.070 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.067+0000 7fc856c30700 1 -- 172.21.15.113:0/3740841434 shutdown_connections 2024-04-12T04:59:54.070 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:54.067+0000 7fc856c30700 1 -- 172.21.15.113:0/3740841434 wait complete. 2024-04-12T04:59:54.722 INFO:teuthology.orchestra.run.smithi113.stderr:Waiting for the mgr to restart... 2024-04-12T04:59:54.722 INFO:teuthology.orchestra.run.smithi113.stderr:Waiting for mgr epoch 5... 2024-04-12T04:59:54.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:54.065872+0000 mon.smithi113 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.113:0/3740841434' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-12T04:59:55.299 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.295+0000 7f06c8b7d700 1 Processor -- start 2024-04-12T04:59:55.299 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.295+0000 7f06c8b7d700 1 -- start start 2024-04-12T04:59:55.299 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.296+0000 7f06c8b7d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40ffc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:55.299 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.296+0000 7f06c8b7d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c4100180 con 0x7f06c4071c00 2024-04-12T04:59:55.300 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.296+0000 7f06c259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40ffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:55.300 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.296+0000 7f06c259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40ffc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58910/0 (socket says 172.21.15.113:58910) 2024-04-12T04:59:55.300 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.296+0000 7f06c259c700 1 -- 172.21.15.113:0/2761316272 learned_addr learned my addr 172.21.15.113:0/2761316272 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:55.300 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.296+0000 7f06c259c700 1 -- 172.21.15.113:0/2761316272 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c41002c0 con 0x7f06c4071c00 2024-04-12T04:59:55.300 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.296+0000 7f06c259c700 1 --2- 172.21.15.113:0/2761316272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40ffc90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f06b4009730 tx=0x7f06b40090f0).ready entity=mon.0 client_cookie=12e1f49195777b94 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:55.300 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.296+0000 7f06c159a700 1 -- 172.21.15.113:0/2761316272 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06b400b6e0 con 0x7f06c4071c00 2024-04-12T04:59:55.300 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.296+0000 7f06c159a700 1 -- 172.21.15.113:0/2761316272 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f06b400bc90 con 0x7f06c4071c00 2024-04-12T04:59:55.300 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.296+0000 7f06c159a700 1 -- 172.21.15.113:0/2761316272 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06b4011c70 con 0x7f06c4071c00 2024-04-12T04:59:55.300 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.297+0000 7f06c8b7d700 1 -- 172.21.15.113:0/2761316272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 msgr2=0x7f06c40ffc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:55.300 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.297+0000 7f06c8b7d700 1 --2- 172.21.15.113:0/2761316272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40ffc90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f06b4009730 tx=0x7f06b40090f0).stop 2024-04-12T04:59:55.302 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.297+0000 7f06c8b7d700 1 -- 172.21.15.113:0/2761316272 shutdown_connections 2024-04-12T04:59:55.302 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.297+0000 7f06c8b7d700 1 --2- 172.21.15.113:0/2761316272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40ffc90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.297+0000 7f06c8b7d700 1 -- 172.21.15.113:0/2761316272 >> 172.21.15.113:0/2761316272 conn(0x7f06c40f76f0 msgr2=0x7f06c40f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.297+0000 7f06c8b7d700 1 -- 172.21.15.113:0/2761316272 shutdown_connections 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.297+0000 7f06c8b7d700 1 -- 172.21.15.113:0/2761316272 wait complete. 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.298+0000 7f06c8b7d700 1 Processor -- start 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.298+0000 7f06c8b7d700 1 -- start start 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.298+0000 7f06c8b7d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40fd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.298+0000 7f06c8b7d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40fbb90 con 0x7f06c4071c00 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.298+0000 7f06c259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40fd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.298+0000 7f06c259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40fd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58922/0 (socket says 172.21.15.113:58922) 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.298+0000 7f06c259c700 1 -- 172.21.15.113:0/850336348 learned_addr learned my addr 172.21.15.113:0/850336348 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.298+0000 7f06c259c700 1 -- 172.21.15.113:0/850336348 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b4009370 con 0x7f06c4071c00 2024-04-12T04:59:55.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.299+0000 7f06c259c700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40fd710 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f06b4009330 tx=0x7f06b40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:55.304 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.299+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06b401a610 con 0x7f06c4071c00 2024-04-12T04:59:55.304 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.299+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f06b401abc0 con 0x7f06c4071c00 2024-04-12T04:59:55.304 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.299+0000 7f06c8b7d700 1 -- 172.21.15.113:0/850336348 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c40fbd30 con 0x7f06c4071c00 2024-04-12T04:59:55.304 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.299+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06b4022a30 con 0x7f06c4071c00 2024-04-12T04:59:55.304 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.299+0000 7f06c8b7d700 1 -- 172.21.15.113:0/850336348 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c40fc140 con 0x7f06c4071c00 2024-04-12T04:59:55.304 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.300+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 38632+0+0 (secure 0 0 0) 0x7f06b4022c60 con 0x7f06c4071c00 2024-04-12T04:59:55.304 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.300+0000 7f06bb7fe700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 0x7f06b0034af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:55.304 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.300+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 --> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f06b00351b0 con 0x7f06b0032630 2024-04-12T04:59:55.304 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.300+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f06b4044ac0 con 0x7f06c4071c00 2024-04-12T04:59:55.304 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.300+0000 7f06c1d9b700 1 -- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 msgr2=0x7f06b0034af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/3275382936 2024-04-12T04:59:55.304 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.300+0000 7f06c1d9b700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 0x7f06b0034af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-04-12T04:59:55.503 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.501+0000 7f06c1d9b700 1 -- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 msgr2=0x7f06b0034af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/3275382936 2024-04-12T04:59:55.503 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.501+0000 7f06c1d9b700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 0x7f06b0034af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-04-12T04:59:55.904 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.902+0000 7f06c1d9b700 1 -- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 msgr2=0x7f06b0034af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/3275382936 2024-04-12T04:59:55.904 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:55.902+0000 7f06c1d9b700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 0x7f06b0034af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-04-12T04:59:56.705 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:56.703+0000 7f06c1d9b700 1 -- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 msgr2=0x7f06b0034af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/3275382936 2024-04-12T04:59:56.705 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:56.703+0000 7f06c1d9b700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 0x7f06b0034af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-04-12T04:59:57.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:57.863+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mgrmap(e 6) v1 ==== 38456+0+0 (secure 0 0 0) 0x7f06b40198b0 con 0x7f06c4071c00 2024-04-12T04:59:57.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:57.863+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 msgr2=0x7f06b0034af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T04:59:57.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:57.863+0000 7f06bb7fe700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 0x7f06b0034af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T04:59:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:57.862701+0000 mon.smithi113 (mon.0) 29 : cluster [INF] Active manager daemon smithi113.ctsazg restarted 2024-04-12T04:59:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:57.862917+0000 mon.smithi113 (mon.0) 30 : cluster [INF] Activating manager daemon smithi113.ctsazg 2024-04-12T04:59:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:57.865688+0000 mon.smithi113 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-04-12T04:59:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:57.865808+0000 mon.smithi113 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi113.ctsazg(active, starting, since 0.00296557s) 2024-04-12T04:59:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.866790+0000 mon.smithi113 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T04:59:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.867528+0000 mon.smithi113 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi113.ctsazg", "id": "smithi113.ctsazg"}]: dispatch 2024-04-12T04:59:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.868139+0000 mon.smithi113 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T04:59:58.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.868284+0000 mon.smithi113 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T04:59:58.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.868402+0000 mon.smithi113 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T04:59:58.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:57.870362+0000 mon.smithi113 (mon.0) 38 : cluster [INF] Manager daemon smithi113.ctsazg is now available 2024-04-12T04:59:58.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.881959+0000 mon.smithi113 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T04:59:58.867 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.865+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 7) v1 ==== 38526+0+0 (secure 0 0 0) 0x7f06b4019390 con 0x7f06c4071c00 2024-04-12T04:59:58.867 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.865+0000 7f06bb7fe700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f06b0035ff0 0x7f06b00383e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T04:59:58.867 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.865+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f06b00351b0 con 0x7f06b0035ff0 2024-04-12T04:59:58.868 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.865+0000 7f06c1d9b700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f06b0035ff0 0x7f06b00383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T04:59:58.868 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.866+0000 7f06c1d9b700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f06b0035ff0 0x7f06b00383e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f06ac003850 tx=0x7f06ac006ea0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T04:59:58.869 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.867+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f06b00351b0 con 0x7f06b0035ff0 2024-04-12T04:59:58.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.873+0000 7f06c8b7d700 1 -- 172.21.15.113:0/850336348 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f06c405a310 con 0x7f06b0035ff0 2024-04-12T04:59:58.875 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.873+0000 7f06bb7fe700 1 -- 172.21.15.113:0/850336348 <== mgr.14114 v2:172.21.15.113:6800/2567243935 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f06c405a310 con 0x7f06b0035ff0 2024-04-12T04:59:58.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: { 2024-04-12T04:59:58.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-12T04:59:58.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "initialized": true 2024-04-12T04:59:58.876 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: } 2024-04-12T04:59:58.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.873+0000 7f06c8b7d700 1 -- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f06b0035ff0 msgr2=0x7f06b00383e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:58.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.873+0000 7f06c8b7d700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f06b0035ff0 0x7f06b00383e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f06ac003850 tx=0x7f06ac006ea0).stop 2024-04-12T04:59:58.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.873+0000 7f06c8b7d700 1 -- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 msgr2=0x7f06c40fd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T04:59:58.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.873+0000 7f06c8b7d700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40fd710 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f06b4009330 tx=0x7f06b40090f0).stop 2024-04-12T04:59:58.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.873+0000 7f06c8b7d700 1 -- 172.21.15.113:0/850336348 shutdown_connections 2024-04-12T04:59:58.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.873+0000 7f06c8b7d700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f06b0035ff0 0x7f06b00383e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:58.877 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.873+0000 7f06c8b7d700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06c4071c00 0x7f06c40fd710 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T04:59:58.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.873+0000 7f06c8b7d700 1 --2- 172.21.15.113:0/850336348 >> [v2:172.21.15.113:6800/3275382936,v1:172.21.15.113:6801/3275382936] conn(0x7f06b0032630 0x7f06b0034af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T04:59:58.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.873+0000 7f06c8b7d700 1 -- 172.21.15.113:0/850336348 >> 172.21.15.113:0/850336348 conn(0x7f06c40f76f0 msgr2=0x7f06c40f91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T04:59:58.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.874+0000 7f06c8b7d700 1 -- 172.21.15.113:0/850336348 shutdown_connections 2024-04-12T04:59:58.878 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T04:59:58.874+0000 7f06c8b7d700 1 -- 172.21.15.113:0/850336348 wait complete. 2024-04-12T04:59:59.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T04:59:57.879368+0000 mgr.smithi113.ctsazg (mgr.14114) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-04-12T04:59:59.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.966007+0000 mon.smithi113 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T04:59:59.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.966933+0000 mon.smithi113 (mon.0) 41 : audit [DBG] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T04:59:59.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.968299+0000 mon.smithi113 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T04:59:59.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.970287+0000 mon.smithi113 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T04:59:59.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.984498+0000 mon.smithi113 (mon.0) 44 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T04:59:59.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:57.986616+0000 mon.smithi113 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/trash_purge_schedule"}]: dispatch 2024-04-12T04:59:59.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:58.868484+0000 mon.smithi113 (mon.0) 46 : cluster [DBG] mgrmap e7: smithi113.ctsazg(active, since 1.00563s) 2024-04-12T04:59:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:mgr epoch 5 is available 2024-04-12T04:59:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:Setting orchestrator backend to cephadm... 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.106+0000 7f03835dc700 1 Processor -- start 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.106+0000 7f03835dc700 1 -- start start 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.107+0000 7f03835dc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c0ffc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.107+0000 7f03835dc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c100120 con 0x7f037c071c60 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.107+0000 7f0381378700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c0ffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.107+0000 7f0381378700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c0ffc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58968/0 (socket says 172.21.15.113:58968) 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.107+0000 7f0381378700 1 -- 172.21.15.113:0/1973318855 learned_addr learned my addr 172.21.15.113:0/1973318855 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.107+0000 7f0381378700 1 -- 172.21.15.113:0/1973318855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c100260 con 0x7f037c071c60 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.107+0000 7f0381378700 1 --2- 172.21.15.113:0/1973318855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c0ffc30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f036c009730 tx=0x7f036c0090f0).ready entity=mon.0 client_cookie=5541ec1be0e1f225 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.107+0000 7f0373fff700 1 -- 172.21.15.113:0/1973318855 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f036c00b6e0 con 0x7f037c071c60 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.107+0000 7f0373fff700 1 -- 172.21.15.113:0/1973318855 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f036c00bc90 con 0x7f037c071c60 2024-04-12T05:00:00.112 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.107+0000 7f0373fff700 1 -- 172.21.15.113:0/1973318855 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f036c011c70 con 0x7f037c071c60 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.108+0000 7f03835dc700 1 -- 172.21.15.113:0/1973318855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 msgr2=0x7f037c0ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.108+0000 7f03835dc700 1 --2- 172.21.15.113:0/1973318855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c0ffc30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f036c009730 tx=0x7f036c0090f0).stop 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.108+0000 7f03835dc700 1 -- 172.21.15.113:0/1973318855 shutdown_connections 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.108+0000 7f03835dc700 1 --2- 172.21.15.113:0/1973318855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c0ffc30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.108+0000 7f03835dc700 1 -- 172.21.15.113:0/1973318855 >> 172.21.15.113:0/1973318855 conn(0x7f037c0f76f0 msgr2=0x7f037c0f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.108+0000 7f03835dc700 1 -- 172.21.15.113:0/1973318855 shutdown_connections 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.108+0000 7f03835dc700 1 -- 172.21.15.113:0/1973318855 wait complete. 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.108+0000 7f03835dc700 1 Processor -- start 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.108+0000 7f03835dc700 1 -- start start 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f03835dc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c10be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f03835dc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c10c2f0 con 0x7f037c071c60 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f0381378700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c10be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f0381378700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c10be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58980/0 (socket says 172.21.15.113:58980) 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f0381378700 1 -- 172.21.15.113:0/532272607 learned_addr learned my addr 172.21.15.113:0/532272607 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:00.113 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f0381378700 1 -- 172.21.15.113:0/532272607 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c009370 con 0x7f037c071c60 2024-04-12T05:00:00.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f0381378700 1 --2- 172.21.15.113:0/532272607 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c10be00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f036c000c00 tx=0x7f036c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:00.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f03727fc700 1 -- 172.21.15.113:0/532272607 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f036c01a610 con 0x7f037c071c60 2024-04-12T05:00:00.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f03727fc700 1 -- 172.21.15.113:0/532272607 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f036c01abc0 con 0x7f037c071c60 2024-04-12T05:00:00.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f03835dc700 1 -- 172.21.15.113:0/532272607 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037c10c4f0 con 0x7f037c071c60 2024-04-12T05:00:00.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f03727fc700 1 -- 172.21.15.113:0/532272607 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f036c022a20 con 0x7f037c071c60 2024-04-12T05:00:00.114 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.109+0000 7f03835dc700 1 -- 172.21.15.113:0/532272607 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037c10c900 con 0x7f037c071c60 2024-04-12T05:00:00.115 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.110+0000 7f03727fc700 1 -- 172.21.15.113:0/532272607 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7f036c022ce0 con 0x7f037c071c60 2024-04-12T05:00:00.115 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.110+0000 7f03727fc700 1 --2- 172.21.15.113:0/532272607 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0368032610 0x7f0368034ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:00.115 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.110+0000 7f03727fc700 1 -- 172.21.15.113:0/532272607 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f036c044e80 con 0x7f037c071c60 2024-04-12T05:00:00.115 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.110+0000 7f03835dc700 1 -- 172.21.15.113:0/532272607 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037c046ce0 con 0x7f037c071c60 2024-04-12T05:00:00.115 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.111+0000 7f0380b77700 1 --2- 172.21.15.113:0/532272607 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0368032610 0x7f0368034ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:00.115 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.111+0000 7f0380b77700 1 --2- 172.21.15.113:0/532272607 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0368032610 0x7f0368034ad0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0378009780 tx=0x7f0378006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:00.117 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.115+0000 7f03727fc700 1 -- 172.21.15.113:0/532272607 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f036c018500 con 0x7f037c071c60 2024-04-12T05:00:00.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:58.867486+0000 mgr.smithi113.ctsazg (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-12T05:00:00.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 04:59:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T04:59:58.874075+0000 mgr.smithi113.ctsazg (mgr.14114) 3 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-12T05:00:00.292 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.290+0000 7f03835dc700 1 -- 172.21.15.113:0/532272607 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f037c10d100 con 0x7f0368032610 2024-04-12T05:00:00.299 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.297+0000 7f03727fc700 1 -- 172.21.15.113:0/532272607 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f037c10d100 con 0x7f0368032610 2024-04-12T05:00:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.299+0000 7f03835dc700 1 -- 172.21.15.113:0/532272607 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0368032610 msgr2=0x7f0368034ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.299+0000 7f03835dc700 1 --2- 172.21.15.113:0/532272607 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0368032610 0x7f0368034ad0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0378009780 tx=0x7f0378006cc0).stop 2024-04-12T05:00:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.299+0000 7f03835dc700 1 -- 172.21.15.113:0/532272607 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 msgr2=0x7f037c10be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.299+0000 7f03835dc700 1 --2- 172.21.15.113:0/532272607 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c10be00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f036c000c00 tx=0x7f036c0090f0).stop 2024-04-12T05:00:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.299+0000 7f03835dc700 1 -- 172.21.15.113:0/532272607 shutdown_connections 2024-04-12T05:00:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.299+0000 7f03835dc700 1 --2- 172.21.15.113:0/532272607 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0368032610 0x7f0368034ad0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.299+0000 7f03835dc700 1 --2- 172.21.15.113:0/532272607 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f037c071c60 0x7f037c10be00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.299+0000 7f03835dc700 1 -- 172.21.15.113:0/532272607 >> 172.21.15.113:0/532272607 conn(0x7f037c0f76f0 msgr2=0x7f037c0f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.300+0000 7f03835dc700 1 -- 172.21.15.113:0/532272607 shutdown_connections 2024-04-12T05:00:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:00.300+0000 7f03835dc700 1 -- 172.21.15.113:0/532272607 wait complete. 2024-04-12T05:00:01.262 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T04:59:59.968601+0000 mon.smithi113 (mon.0) 47 : cluster [DBG] mgrmap e8: smithi113.ctsazg(active, since 2s) 2024-04-12T05:00:01.262 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:00.291930+0000 mgr.smithi113.ctsazg (mgr.14114) 4 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:01.262 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:00.298238+0000 mon.smithi113 (mon.0) 48 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:01.262 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:00.346002+0000 mon.smithi113 (mon.0) 49 : audit [DBG] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:00:01.584 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.580+0000 7fe01c5ed700 1 Processor -- start 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.580+0000 7fe01c5ed700 1 -- start start 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.581+0000 7fe01c5ed700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe0140fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.581+0000 7fe01c5ed700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140fc7e0 con 0x7fe0140fbed0 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.581+0000 7fe01a389700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe0140fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.581+0000 7fe01a389700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe0140fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49074/0 (socket says 172.21.15.113:49074) 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.581+0000 7fe01a389700 1 -- 172.21.15.113:0/2181673231 learned_addr learned my addr 172.21.15.113:0/2181673231 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.581+0000 7fe01a389700 1 -- 172.21.15.113:0/2181673231 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0140fc920 con 0x7fe0140fbed0 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.581+0000 7fe01a389700 1 --2- 172.21.15.113:0/2181673231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe0140fc2f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe008009730 tx=0x7fe0080090f0).ready entity=mon.0 client_cookie=a545b609725031f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.581+0000 7fe019387700 1 -- 172.21.15.113:0/2181673231 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe00800b6e0 con 0x7fe0140fbed0 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.581+0000 7fe019387700 1 -- 172.21.15.113:0/2181673231 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fe00800bc90 con 0x7fe0140fbed0 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.581+0000 7fe019387700 1 -- 172.21.15.113:0/2181673231 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe008011c70 con 0x7fe0140fbed0 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.582+0000 7fe01c5ed700 1 -- 172.21.15.113:0/2181673231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 msgr2=0x7fe0140fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.582+0000 7fe01c5ed700 1 --2- 172.21.15.113:0/2181673231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe0140fc2f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe008009730 tx=0x7fe0080090f0).stop 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.582+0000 7fe01c5ed700 1 -- 172.21.15.113:0/2181673231 shutdown_connections 2024-04-12T05:00:01.585 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.582+0000 7fe01c5ed700 1 --2- 172.21.15.113:0/2181673231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe0140fc2f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:01.586 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.582+0000 7fe01c5ed700 1 -- 172.21.15.113:0/2181673231 >> 172.21.15.113:0/2181673231 conn(0x7fe0140f76f0 msgr2=0x7fe0140f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:01.586 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.582+0000 7fe01c5ed700 1 -- 172.21.15.113:0/2181673231 shutdown_connections 2024-04-12T05:00:01.586 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.582+0000 7fe01c5ed700 1 -- 172.21.15.113:0/2181673231 wait complete. 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.583+0000 7fe01c5ed700 1 Processor -- start 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.583+0000 7fe01c5ed700 1 -- start start 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.583+0000 7fe01c5ed700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe01410be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.583+0000 7fe01c5ed700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01410c2f0 con 0x7fe0140fbed0 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.583+0000 7fe01a389700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe01410be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.583+0000 7fe01a389700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe01410be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49076/0 (socket says 172.21.15.113:49076) 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.583+0000 7fe01a389700 1 -- 172.21.15.113:0/3192407630 learned_addr learned my addr 172.21.15.113:0/3192407630 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.583+0000 7fe01a389700 1 -- 172.21.15.113:0/3192407630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe008009370 con 0x7fe0140fbed0 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.583+0000 7fe01a389700 1 --2- 172.21.15.113:0/3192407630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe01410be00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe008011870 tx=0x7fe0080090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.584+0000 7fe0077fe700 1 -- 172.21.15.113:0/3192407630 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe00801a610 con 0x7fe0140fbed0 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.584+0000 7fe0077fe700 1 -- 172.21.15.113:0/3192407630 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fe00801abc0 con 0x7fe0140fbed0 2024-04-12T05:00:01.588 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.584+0000 7fe01c5ed700 1 -- 172.21.15.113:0/3192407630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01410c4f0 con 0x7fe0140fbed0 2024-04-12T05:00:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.584+0000 7fe0077fe700 1 -- 172.21.15.113:0/3192407630 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe008022a30 con 0x7fe0140fbed0 2024-04-12T05:00:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.584+0000 7fe01c5ed700 1 -- 172.21.15.113:0/3192407630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01410c900 con 0x7fe0140fbed0 2024-04-12T05:00:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.584+0000 7fe0077fe700 1 -- 172.21.15.113:0/3192407630 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7fe008022cf0 con 0x7fe0140fbed0 2024-04-12T05:00:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.585+0000 7fe0077fe700 1 --2- 172.21.15.113:0/3192407630 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe000032610 0x7fe000034ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.585+0000 7fe0077fe700 1 -- 172.21.15.113:0/3192407630 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fe008044c40 con 0x7fe0140fbed0 2024-04-12T05:00:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.585+0000 7fe019b88700 1 --2- 172.21.15.113:0/3192407630 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe000032610 0x7fe000034ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.585+0000 7fe01c5ed700 1 -- 172.21.15.113:0/3192407630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe014046ce0 con 0x7fe0140fbed0 2024-04-12T05:00:01.589 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.585+0000 7fe019b88700 1 --2- 172.21.15.113:0/3192407630 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe000032610 0x7fe000034ad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe010009780 tx=0x7fe010006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:01.592 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.590+0000 7fe0077fe700 1 -- 172.21.15.113:0/3192407630 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe00801e070 con 0x7fe0140fbed0 2024-04-12T05:00:01.756 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.753+0000 7fe01c5ed700 1 -- 172.21.15.113:0/3192407630 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fe014105d40 con 0x7fe000032610 2024-04-12T05:00:01.756 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.754+0000 7fe0077fe700 1 -- 172.21.15.113:0/3192407630 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fe014105d40 con 0x7fe000032610 2024-04-12T05:00:01.756 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: value unchanged 2024-04-12T05:00:01.759 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.756+0000 7fe01c5ed700 1 -- 172.21.15.113:0/3192407630 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe000032610 msgr2=0x7fe000034ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:01.759 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.756+0000 7fe01c5ed700 1 --2- 172.21.15.113:0/3192407630 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe000032610 0x7fe000034ad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe010009780 tx=0x7fe010006cc0).stop 2024-04-12T05:00:01.759 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.756+0000 7fe01c5ed700 1 -- 172.21.15.113:0/3192407630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 msgr2=0x7fe01410be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:01.759 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.756+0000 7fe01c5ed700 1 --2- 172.21.15.113:0/3192407630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe01410be00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe008011870 tx=0x7fe0080090f0).stop 2024-04-12T05:00:01.759 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.756+0000 7fe01c5ed700 1 -- 172.21.15.113:0/3192407630 shutdown_connections 2024-04-12T05:00:01.759 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.756+0000 7fe01c5ed700 1 --2- 172.21.15.113:0/3192407630 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe000032610 0x7fe000034ad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:01.759 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.756+0000 7fe01c5ed700 1 --2- 172.21.15.113:0/3192407630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0140fbed0 0x7fe01410be00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:01.759 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.756+0000 7fe01c5ed700 1 -- 172.21.15.113:0/3192407630 >> 172.21.15.113:0/3192407630 conn(0x7fe0140f76f0 msgr2=0x7fe0140f82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:01.759 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.756+0000 7fe01c5ed700 1 -- 172.21.15.113:0/3192407630 shutdown_connections 2024-04-12T05:00:01.760 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:01.756+0000 7fe01c5ed700 1 -- 172.21.15.113:0/3192407630 wait complete. 2024-04-12T05:00:02.418 INFO:teuthology.orchestra.run.smithi113.stderr:Generating ssh key... 2024-04-12T05:00:02.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:01.754949+0000 mgr.smithi113.ctsazg (mgr.14114) 5 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.963+0000 7fc1bfd67700 1 Processor -- start 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.964+0000 7fc1bfd67700 1 -- start start 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.964+0000 7fc1bfd67700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b80ffc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.964+0000 7fc1bfd67700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b8100120 con 0x7fc1b8071c60 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.964+0000 7fc1bdb03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b80ffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.964+0000 7fc1bdb03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b80ffc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49082/0 (socket says 172.21.15.113:49082) 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.964+0000 7fc1bdb03700 1 -- 172.21.15.113:0/138443208 learned_addr learned my addr 172.21.15.113:0/138443208 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.964+0000 7fc1bdb03700 1 -- 172.21.15.113:0/138443208 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b8100260 con 0x7fc1b8071c60 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.964+0000 7fc1bdb03700 1 --2- 172.21.15.113:0/138443208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b80ffc30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fc1a8009730 tx=0x7fc1a80090f0).ready entity=mon.0 client_cookie=de859805af6c787a server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.964+0000 7fc1bcb01700 1 -- 172.21.15.113:0/138443208 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1a800b6e0 con 0x7fc1b8071c60 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.965+0000 7fc1bcb01700 1 -- 172.21.15.113:0/138443208 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fc1a800bc90 con 0x7fc1b8071c60 2024-04-12T05:00:02.968 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.965+0000 7fc1bcb01700 1 -- 172.21.15.113:0/138443208 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1a8011c70 con 0x7fc1b8071c60 2024-04-12T05:00:02.969 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.965+0000 7fc1bfd67700 1 -- 172.21.15.113:0/138443208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 msgr2=0x7fc1b80ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:02.969 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.965+0000 7fc1bfd67700 1 --2- 172.21.15.113:0/138443208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b80ffc30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fc1a8009730 tx=0x7fc1a80090f0).stop 2024-04-12T05:00:02.969 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.965+0000 7fc1bfd67700 1 -- 172.21.15.113:0/138443208 shutdown_connections 2024-04-12T05:00:02.969 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.965+0000 7fc1bfd67700 1 --2- 172.21.15.113:0/138443208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b80ffc30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:02.969 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.965+0000 7fc1bfd67700 1 -- 172.21.15.113:0/138443208 >> 172.21.15.113:0/138443208 conn(0x7fc1b80f76f0 msgr2=0x7fc1b80f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:02.969 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.965+0000 7fc1bfd67700 1 -- 172.21.15.113:0/138443208 shutdown_connections 2024-04-12T05:00:02.969 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.965+0000 7fc1bfd67700 1 -- 172.21.15.113:0/138443208 wait complete. 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.966+0000 7fc1bfd67700 1 Processor -- start 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.966+0000 7fc1bfd67700 1 -- start start 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.966+0000 7fc1bfd67700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b810be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.966+0000 7fc1bfd67700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b810c2f0 con 0x7fc1b8071c60 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.966+0000 7fc1bdb03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b810be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.966+0000 7fc1bdb03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b810be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49094/0 (socket says 172.21.15.113:49094) 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.966+0000 7fc1bdb03700 1 -- 172.21.15.113:0/2213945445 learned_addr learned my addr 172.21.15.113:0/2213945445 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.967+0000 7fc1bdb03700 1 -- 172.21.15.113:0/2213945445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a8009370 con 0x7fc1b8071c60 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.967+0000 7fc1bdb03700 1 --2- 172.21.15.113:0/2213945445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b810be00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fc1a8000c00 tx=0x7fc1a80090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.967+0000 7fc1aeffd700 1 -- 172.21.15.113:0/2213945445 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1a801a610 con 0x7fc1b8071c60 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.967+0000 7fc1aeffd700 1 -- 172.21.15.113:0/2213945445 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fc1a801abc0 con 0x7fc1b8071c60 2024-04-12T05:00:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.967+0000 7fc1bfd67700 1 -- 172.21.15.113:0/2213945445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b810c4f0 con 0x7fc1b8071c60 2024-04-12T05:00:02.972 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.967+0000 7fc1aeffd700 1 -- 172.21.15.113:0/2213945445 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1a8022a20 con 0x7fc1b8071c60 2024-04-12T05:00:02.972 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.967+0000 7fc1bfd67700 1 -- 172.21.15.113:0/2213945445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b810c900 con 0x7fc1b8071c60 2024-04-12T05:00:02.972 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.968+0000 7fc1aeffd700 1 -- 172.21.15.113:0/2213945445 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7fc1a8022c90 con 0x7fc1b8071c60 2024-04-12T05:00:02.972 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.968+0000 7fc1bfd67700 1 -- 172.21.15.113:0/2213945445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b8046ce0 con 0x7fc1b8071c60 2024-04-12T05:00:02.972 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.968+0000 7fc1aeffd700 1 --2- 172.21.15.113:0/2213945445 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fc1a40326b0 0x7fc1a4034b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:02.972 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.968+0000 7fc1aeffd700 1 -- 172.21.15.113:0/2213945445 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fc1a8044b30 con 0x7fc1b8071c60 2024-04-12T05:00:02.972 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.968+0000 7fc1bd302700 1 --2- 172.21.15.113:0/2213945445 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fc1a40326b0 0x7fc1a4034b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:02.972 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.969+0000 7fc1bd302700 1 --2- 172.21.15.113:0/2213945445 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fc1a40326b0 0x7fc1a4034b70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc1b4009780 tx=0x7fc1b4006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:02.975 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:02.973+0000 7fc1aeffd700 1 -- 172.21.15.113:0/2213945445 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc1a801ad30 con 0x7fc1b8071c60 2024-04-12T05:00:03.137 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.135+0000 7fc1bfd67700 1 -- 172.21.15.113:0/2213945445 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1b810d100 con 0x7fc1a40326b0 2024-04-12T05:00:03.667 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.665+0000 7fc1aeffd700 1 -- 172.21.15.113:0/2213945445 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc1b810d100 con 0x7fc1a40326b0 2024-04-12T05:00:03.670 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.667+0000 7fc1bfd67700 1 -- 172.21.15.113:0/2213945445 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fc1a40326b0 msgr2=0x7fc1a4034b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:03.670 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.667+0000 7fc1bfd67700 1 --2- 172.21.15.113:0/2213945445 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fc1a40326b0 0x7fc1a4034b70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc1b4009780 tx=0x7fc1b4006cc0).stop 2024-04-12T05:00:03.670 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.667+0000 7fc1bfd67700 1 -- 172.21.15.113:0/2213945445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 msgr2=0x7fc1b810be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:03.670 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.667+0000 7fc1bfd67700 1 --2- 172.21.15.113:0/2213945445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b810be00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fc1a8000c00 tx=0x7fc1a80090f0).stop 2024-04-12T05:00:03.670 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.667+0000 7fc1bfd67700 1 -- 172.21.15.113:0/2213945445 shutdown_connections 2024-04-12T05:00:03.670 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.667+0000 7fc1bfd67700 1 --2- 172.21.15.113:0/2213945445 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fc1a40326b0 0x7fc1a4034b70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:03.670 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.667+0000 7fc1bfd67700 1 --2- 172.21.15.113:0/2213945445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b8071c60 0x7fc1b810be00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:03.670 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.667+0000 7fc1bfd67700 1 -- 172.21.15.113:0/2213945445 >> 172.21.15.113:0/2213945445 conn(0x7fc1b80f76f0 msgr2=0x7fc1b80f9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:03.670 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.667+0000 7fc1bfd67700 1 -- 172.21.15.113:0/2213945445 shutdown_connections 2024-04-12T05:00:03.670 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:03.667+0000 7fc1bfd67700 1 -- 172.21.15.113:0/2213945445 wait complete. 2024-04-12T05:00:03.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:03.136556+0000 mgr.smithi113.ctsazg (mgr.14114) 6 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:03.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:03.136777+0000 mgr.smithi113.ctsazg (mgr.14114) 7 : cephadm [INF] Generating ssh key... 2024-04-12T05:00:04.864 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.859+0000 7f98d0854700 1 Processor -- start 2024-04-12T05:00:04.864 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.859+0000 7f98d0854700 1 -- start start 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.859+0000 7f98d0854700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc0ffc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.859+0000 7f98d0854700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc100120 con 0x7f98cc071c60 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.859+0000 7f98ca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc0ffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.859+0000 7f98ca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc0ffc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49100/0 (socket says 172.21.15.113:49100) 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.859+0000 7f98ca59c700 1 -- 172.21.15.113:0/828507828 learned_addr learned my addr 172.21.15.113:0/828507828 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.859+0000 7f98ca59c700 1 -- 172.21.15.113:0/828507828 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc100260 con 0x7f98cc071c60 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.860+0000 7f98ca59c700 1 --2- 172.21.15.113:0/828507828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc0ffc30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f98bc009730 tx=0x7f98bc0090f0).ready entity=mon.0 client_cookie=1324b2e8502905ff server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.860+0000 7f98c959a700 1 -- 172.21.15.113:0/828507828 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98bc00b6e0 con 0x7f98cc071c60 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.860+0000 7f98c959a700 1 -- 172.21.15.113:0/828507828 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f98bc00bc90 con 0x7f98cc071c60 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.860+0000 7f98c959a700 1 -- 172.21.15.113:0/828507828 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98bc011c70 con 0x7f98cc071c60 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.860+0000 7f98d0854700 1 -- 172.21.15.113:0/828507828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 msgr2=0x7f98cc0ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.860+0000 7f98d0854700 1 --2- 172.21.15.113:0/828507828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc0ffc30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f98bc009730 tx=0x7f98bc0090f0).stop 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.860+0000 7f98d0854700 1 -- 172.21.15.113:0/828507828 shutdown_connections 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.860+0000 7f98d0854700 1 --2- 172.21.15.113:0/828507828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc0ffc30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:04.865 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.860+0000 7f98d0854700 1 -- 172.21.15.113:0/828507828 >> 172.21.15.113:0/828507828 conn(0x7f98cc0f76f0 msgr2=0x7f98cc0f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.860+0000 7f98d0854700 1 -- 172.21.15.113:0/828507828 shutdown_connections 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.861+0000 7f98d0854700 1 -- 172.21.15.113:0/828507828 wait complete. 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.861+0000 7f98d0854700 1 Processor -- start 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.861+0000 7f98d0854700 1 -- start start 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.861+0000 7f98d0854700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc10be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.861+0000 7f98d0854700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc10c2f0 con 0x7f98cc071c60 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.862+0000 7f98ca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc10be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.862+0000 7f98ca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc10be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49116/0 (socket says 172.21.15.113:49116) 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.862+0000 7f98ca59c700 1 -- 172.21.15.113:0/3781705721 learned_addr learned my addr 172.21.15.113:0/3781705721 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.862+0000 7f98ca59c700 1 -- 172.21.15.113:0/3781705721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc009370 con 0x7f98cc071c60 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.862+0000 7f98ca59c700 1 --2- 172.21.15.113:0/3781705721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc10be00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f98bc000c00 tx=0x7f98bc0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.862+0000 7f98bb7fe700 1 -- 172.21.15.113:0/3781705721 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98bc01a610 con 0x7f98cc071c60 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.862+0000 7f98bb7fe700 1 -- 172.21.15.113:0/3781705721 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f98bc01abc0 con 0x7f98cc071c60 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.862+0000 7f98bb7fe700 1 -- 172.21.15.113:0/3781705721 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98bc022a20 con 0x7f98cc071c60 2024-04-12T05:00:04.866 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.862+0000 7f98d0854700 1 -- 172.21.15.113:0/3781705721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc10c4f0 con 0x7f98cc071c60 2024-04-12T05:00:04.867 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.862+0000 7f98d0854700 1 -- 172.21.15.113:0/3781705721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc10c900 con 0x7f98cc071c60 2024-04-12T05:00:04.867 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.863+0000 7f98bb7fe700 1 -- 172.21.15.113:0/3781705721 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7f98bc022c90 con 0x7f98cc071c60 2024-04-12T05:00:04.868 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.863+0000 7f98bb7fe700 1 --2- 172.21.15.113:0/3781705721 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f98b4032660 0x7f98b4034b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:04.868 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.863+0000 7f98bb7fe700 1 -- 172.21.15.113:0/3781705721 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f98bc044b20 con 0x7f98cc071c60 2024-04-12T05:00:04.868 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.863+0000 7f98c9d9b700 1 --2- 172.21.15.113:0/3781705721 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f98b4032660 0x7f98b4034b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:04.868 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.863+0000 7f98d0854700 1 -- 172.21.15.113:0/3781705721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc046ce0 con 0x7f98cc071c60 2024-04-12T05:00:04.868 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.863+0000 7f98c9d9b700 1 --2- 172.21.15.113:0/3781705721 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f98b4032660 0x7f98b4034b20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f98c0009780 tx=0x7f98c0006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:04.870 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:04.868+0000 7f98bb7fe700 1 -- 172.21.15.113:0/3781705721 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f98bc01e070 con 0x7f98cc071c60 2024-04-12T05:00:04.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:03.663478+0000 mon.smithi113 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:04.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:03.665217+0000 mon.smithi113 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:05.033 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.031+0000 7f98d0854700 1 -- 172.21.15.113:0/3781705721 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f98cc105ca0 con 0x7f98b4032660 2024-04-12T05:00:05.034 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.032+0000 7f98bb7fe700 1 -- 172.21.15.113:0/3781705721 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f98cc105ca0 con 0x7f98b4032660 2024-04-12T05:00:05.034 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXs+Pr8AGjugkNdzu1VpDNMF4LYzc73Sw+ke1QTQK/W6wHyIyvUN5gij1EQjKBN0xzHSaDXkDFpkrbRHkU7+taoE80W5SJTrFnJxOXV7a2Ngi82sC11bHjtrkTTosv51rPtR4q8CT6N+ZZYyaVa3ZipMwwXNNZGCfppd6elv/E/neYYCYn1nIExE/O+htrAklvBrx2qXEFOifvUSv311H3ti761gMJ74JKLc82A2H3Ah/BwIoBi83tUWJAud2rsTmbms31h7mmnDdi+BoQM+ga3HzNb6+rs3/FhfbTpAkzHX9pOi0L1CynVWISrr5VmOhdvbgHeqrpcXG3kQfAf12VN2uOGL+1vffXT0DmrzLmHjAf/b8pv92ZNqdlErTwn79/qWIMLZ5thX9V+zVrt/jm17NrQqgViK3MF22cWBy6e3yNgjRVsIb9nGG/bX+YXmHBquMAYAeKRVRML65+yjwdS28+97DhMJK6YfXNmELhOuFSNUxnvrwLd+HUsANcoYk= ceph-52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T05:00:05.036 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.033+0000 7f98d0854700 1 -- 172.21.15.113:0/3781705721 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f98b4032660 msgr2=0x7f98b4034b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:05.036 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.033+0000 7f98d0854700 1 --2- 172.21.15.113:0/3781705721 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f98b4032660 0x7f98b4034b20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f98c0009780 tx=0x7f98c0006cc0).stop 2024-04-12T05:00:05.037 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.033+0000 7f98d0854700 1 -- 172.21.15.113:0/3781705721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 msgr2=0x7f98cc10be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:05.037 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.033+0000 7f98d0854700 1 --2- 172.21.15.113:0/3781705721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc10be00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f98bc000c00 tx=0x7f98bc0090f0).stop 2024-04-12T05:00:05.037 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.033+0000 7f98d0854700 1 -- 172.21.15.113:0/3781705721 shutdown_connections 2024-04-12T05:00:05.037 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.033+0000 7f98d0854700 1 --2- 172.21.15.113:0/3781705721 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f98b4032660 0x7f98b4034b20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:05.037 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.033+0000 7f98d0854700 1 --2- 172.21.15.113:0/3781705721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc071c60 0x7f98cc10be00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:05.037 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.033+0000 7f98d0854700 1 -- 172.21.15.113:0/3781705721 >> 172.21.15.113:0/3781705721 conn(0x7f98cc0f76f0 msgr2=0x7f98cc0f9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:05.037 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.034+0000 7f98d0854700 1 -- 172.21.15.113:0/3781705721 shutdown_connections 2024-04-12T05:00:05.037 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:05.034+0000 7f98d0854700 1 -- 172.21.15.113:0/3781705721 wait complete. 2024-04-12T05:00:05.690 INFO:teuthology.orchestra.run.smithi113.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-12T05:00:05.690 INFO:teuthology.orchestra.run.smithi113.stderr:Adding key to root@localhost authorized_keys... 2024-04-12T05:00:05.691 INFO:teuthology.orchestra.run.smithi113.stderr:Adding host smithi113... 2024-04-12T05:00:05.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:05.032718+0000 mgr.smithi113.ctsazg (mgr.14114) 8 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:06.222 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.215+0000 7f527faab700 1 Processor -- start 2024-04-12T05:00:06.222 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.215+0000 7f527faab700 1 -- start start 2024-04-12T05:00:06.222 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.215+0000 7f527faab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780ffc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:06.222 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.215+0000 7f527faab700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5278100120 con 0x7f5278071c60 2024-04-12T05:00:06.222 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.215+0000 7f527d847700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780ffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:06.222 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.215+0000 7f527d847700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780ffc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49118/0 (socket says 172.21.15.113:49118) 2024-04-12T05:00:06.222 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.215+0000 7f527d847700 1 -- 172.21.15.113:0/2878331759 learned_addr learned my addr 172.21.15.113:0/2878331759 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:06.222 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.216+0000 7f527d847700 1 -- 172.21.15.113:0/2878331759 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5278100260 con 0x7f5278071c60 2024-04-12T05:00:06.222 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.216+0000 7f527d847700 1 --2- 172.21.15.113:0/2878331759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780ffc30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5274009730 tx=0x7f52740090f0).ready entity=mon.0 client_cookie=5ebbee590d4c1a25 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:06.222 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.216+0000 7f527c845700 1 -- 172.21.15.113:0/2878331759 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f527400b6e0 con 0x7f5278071c60 2024-04-12T05:00:06.222 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.216+0000 7f527c845700 1 -- 172.21.15.113:0/2878331759 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f527400bc90 con 0x7f5278071c60 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.216+0000 7f527c845700 1 -- 172.21.15.113:0/2878331759 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5274011c70 con 0x7f5278071c60 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.216+0000 7f527faab700 1 -- 172.21.15.113:0/2878331759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 msgr2=0x7f52780ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.216+0000 7f527faab700 1 --2- 172.21.15.113:0/2878331759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780ffc30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5274009730 tx=0x7f52740090f0).stop 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.217+0000 7f527faab700 1 -- 172.21.15.113:0/2878331759 shutdown_connections 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.217+0000 7f527faab700 1 --2- 172.21.15.113:0/2878331759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780ffc30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.217+0000 7f527faab700 1 -- 172.21.15.113:0/2878331759 >> 172.21.15.113:0/2878331759 conn(0x7f52780f76f0 msgr2=0x7f52780f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.217+0000 7f527faab700 1 -- 172.21.15.113:0/2878331759 shutdown_connections 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.217+0000 7f527faab700 1 -- 172.21.15.113:0/2878331759 wait complete. 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.217+0000 7f527faab700 1 Processor -- start 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.217+0000 7f527faab700 1 -- start start 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.217+0000 7f527faab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780fd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.217+0000 7f527faab700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52780fbb30 con 0x7f5278071c60 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.218+0000 7f527d847700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780fd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.218+0000 7f527d847700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780fd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49126/0 (socket says 172.21.15.113:49126) 2024-04-12T05:00:06.223 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.218+0000 7f527d847700 1 -- 172.21.15.113:0/1143760893 learned_addr learned my addr 172.21.15.113:0/1143760893 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.218+0000 7f527d847700 1 -- 172.21.15.113:0/1143760893 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5274009370 con 0x7f5278071c60 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.218+0000 7f527d847700 1 --2- 172.21.15.113:0/1143760893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780fd6b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5274000c00 tx=0x7f52740090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.218+0000 7f526ef78700 1 -- 172.21.15.113:0/1143760893 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f527401a610 con 0x7f5278071c60 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.218+0000 7f526ef78700 1 -- 172.21.15.113:0/1143760893 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7f527401abc0 con 0x7f5278071c60 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.218+0000 7f527faab700 1 -- 172.21.15.113:0/1143760893 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52780fbcd0 con 0x7f5278071c60 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.218+0000 7f526ef78700 1 -- 172.21.15.113:0/1143760893 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5274022a20 con 0x7f5278071c60 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.218+0000 7f527faab700 1 -- 172.21.15.113:0/1143760893 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52780fc0e0 con 0x7f5278071c60 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.219+0000 7f527faab700 1 -- 172.21.15.113:0/1143760893 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5278046ce0 con 0x7f5278071c60 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.219+0000 7f526ef78700 1 -- 172.21.15.113:0/1143760893 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7f5274022c90 con 0x7f5278071c60 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.219+0000 7f526ef78700 1 --2- 172.21.15.113:0/1143760893 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f5264032660 0x7f5264034b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.219+0000 7f526ef78700 1 -- 172.21.15.113:0/1143760893 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f5274044b50 con 0x7f5278071c60 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.219+0000 7f527d046700 1 --2- 172.21.15.113:0/1143760893 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f5264032660 0x7f5264034b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:06.224 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.220+0000 7f527d046700 1 --2- 172.21.15.113:0/1143760893 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f5264032660 0x7f5264034b20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5268009780 tx=0x7f5268006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:06.226 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.224+0000 7f526ef78700 1 -- 172.21.15.113:0/1143760893 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f527401ad30 con 0x7f5278071c60 2024-04-12T05:00:06.390 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:06.388+0000 7f527faab700 1 -- 172.21.15.113:0/1143760893 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi113", "addr": "172.21.15.113", "target": ["mon-mgr", ""]}) v1 -- 0x7f5278105d20 con 0x7f5264032660 2024-04-12T05:00:06.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:06.389790+0000 mgr.smithi113.ctsazg (mgr.14114) 9 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi113", "addr": "172.21.15.113", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:07.005 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:07.002+0000 7f526ef78700 1 -- 172.21.15.113:0/1143760893 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5278105d20 con 0x7f5264032660 2024-04-12T05:00:07.006 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: Added host 'smithi113' with addr '172.21.15.113' 2024-04-12T05:00:07.008 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:07.004+0000 7f527faab700 1 -- 172.21.15.113:0/1143760893 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f5264032660 msgr2=0x7f5264034b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:07.008 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:07.004+0000 7f527faab700 1 --2- 172.21.15.113:0/1143760893 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f5264032660 0x7f5264034b20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5268009780 tx=0x7f5268006cc0).stop 2024-04-12T05:00:07.008 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:07.004+0000 7f527faab700 1 -- 172.21.15.113:0/1143760893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 msgr2=0x7f52780fd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:07.008 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:07.004+0000 7f527faab700 1 --2- 172.21.15.113:0/1143760893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780fd6b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5274000c00 tx=0x7f52740090f0).stop 2024-04-12T05:00:07.008 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:07.005+0000 7f527faab700 1 -- 172.21.15.113:0/1143760893 shutdown_connections 2024-04-12T05:00:07.008 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:07.005+0000 7f527faab700 1 --2- 172.21.15.113:0/1143760893 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f5264032660 0x7f5264034b20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:07.008 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:07.005+0000 7f527faab700 1 --2- 172.21.15.113:0/1143760893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5278071c60 0x7f52780fd6b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:07.008 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:07.005+0000 7f527faab700 1 -- 172.21.15.113:0/1143760893 >> 172.21.15.113:0/1143760893 conn(0x7f52780f76f0 msgr2=0x7f52780f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:07.008 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:07.005+0000 7f527faab700 1 -- 172.21.15.113:0/1143760893 shutdown_connections 2024-04-12T05:00:07.009 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:07.005+0000 7f527faab700 1 -- 172.21.15.113:0/1143760893 wait complete. 2024-04-12T05:00:07.662 INFO:teuthology.orchestra.run.smithi113.stderr:Deploying mon service with default placement... 2024-04-12T05:00:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.213+0000 7fabd8759700 1 Processor -- start 2024-04-12T05:00:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.213+0000 7fabd8759700 1 -- start start 2024-04-12T05:00:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.213+0000 7fabd8759700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00ffcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.213+0000 7fabd8759700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01001a0 con 0x7fabd0071be0 2024-04-12T05:00:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.214+0000 7fabd64f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00ffcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.214+0000 7fabd64f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00ffcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49140/0 (socket says 172.21.15.113:49140) 2024-04-12T05:00:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.214+0000 7fabd64f5700 1 -- 172.21.15.113:0/344332400 learned_addr learned my addr 172.21.15.113:0/344332400 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.214+0000 7fabd64f5700 1 -- 172.21.15.113:0/344332400 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd01002e0 con 0x7fabd0071be0 2024-04-12T05:00:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.214+0000 7fabd64f5700 1 --2- 172.21.15.113:0/344332400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00ffcb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fabcc009730 tx=0x7fabcc0090f0).ready entity=mon.0 client_cookie=84fc425686dfc475 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.214+0000 7fabd54f3700 1 -- 172.21.15.113:0/344332400 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabcc00b6e0 con 0x7fabd0071be0 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.214+0000 7fabd54f3700 1 -- 172.21.15.113:0/344332400 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fabcc00bc90 con 0x7fabd0071be0 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.214+0000 7fabd54f3700 1 -- 172.21.15.113:0/344332400 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabcc011c70 con 0x7fabd0071be0 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd8759700 1 -- 172.21.15.113:0/344332400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 msgr2=0x7fabd00ffcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd8759700 1 --2- 172.21.15.113:0/344332400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00ffcb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fabcc009730 tx=0x7fabcc0090f0).stop 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd8759700 1 -- 172.21.15.113:0/344332400 shutdown_connections 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd8759700 1 --2- 172.21.15.113:0/344332400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00ffcb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd8759700 1 -- 172.21.15.113:0/344332400 >> 172.21.15.113:0/344332400 conn(0x7fabd00f76f0 msgr2=0x7fabd00f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd8759700 1 -- 172.21.15.113:0/344332400 shutdown_connections 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd8759700 1 -- 172.21.15.113:0/344332400 wait complete. 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd8759700 1 Processor -- start 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd8759700 1 -- start start 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd8759700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00fd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd8759700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00fbbb0 con 0x7fabd0071be0 2024-04-12T05:00:08.266 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.215+0000 7fabd64f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00fd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.216+0000 7fabd64f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00fd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49154/0 (socket says 172.21.15.113:49154) 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.216+0000 7fabd64f5700 1 -- 172.21.15.113:0/2797953436 learned_addr learned my addr 172.21.15.113:0/2797953436 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.216+0000 7fabd64f5700 1 -- 172.21.15.113:0/2797953436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc009370 con 0x7fabd0071be0 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.216+0000 7fabd64f5700 1 --2- 172.21.15.113:0/2797953436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00fd730 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fabcc00c010 tx=0x7fabcc0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.216+0000 7fabc77fe700 1 -- 172.21.15.113:0/2797953436 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabcc01a610 con 0x7fabd0071be0 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.216+0000 7fabc77fe700 1 -- 172.21.15.113:0/2797953436 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fabcc01abc0 con 0x7fabd0071be0 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.216+0000 7fabd8759700 1 -- 172.21.15.113:0/2797953436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd00fbd50 con 0x7fabd0071be0 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.216+0000 7fabc77fe700 1 -- 172.21.15.113:0/2797953436 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabcc022a20 con 0x7fabd0071be0 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.216+0000 7fabd8759700 1 -- 172.21.15.113:0/2797953436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd00fc160 con 0x7fabd0071be0 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.217+0000 7fabc77fe700 1 -- 172.21.15.113:0/2797953436 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7fabcc022c90 con 0x7fabd0071be0 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.217+0000 7fabc77fe700 1 --2- 172.21.15.113:0/2797953436 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fabbc032660 0x7fabbc034b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.217+0000 7fabd8759700 1 -- 172.21.15.113:0/2797953436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd0046ce0 con 0x7fabd0071be0 2024-04-12T05:00:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.217+0000 7fabc77fe700 1 -- 172.21.15.113:0/2797953436 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fabcc044bc0 con 0x7fabd0071be0 2024-04-12T05:00:08.268 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.217+0000 7fabd5cf4700 1 --2- 172.21.15.113:0/2797953436 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fabbc032660 0x7fabbc034b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:08.268 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.217+0000 7fabd5cf4700 1 --2- 172.21.15.113:0/2797953436 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fabbc032660 0x7fabbc034b20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fabc0009780 tx=0x7fabc0006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:08.268 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.222+0000 7fabc77fe700 1 -- 172.21.15.113:0/2797953436 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fabcc01a770 con 0x7fabd0071be0 2024-04-12T05:00:08.398 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.395+0000 7fabd8759700 1 -- 172.21.15.113:0/2797953436 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fabd00fc3a0 con 0x7fabbc032660 2024-04-12T05:00:08.402 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.400+0000 7fabc77fe700 1 -- 172.21.15.113:0/2797953436 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fabd00fc3a0 con 0x7fabbc032660 2024-04-12T05:00:08.402 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-12T05:00:08.405 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.402+0000 7fabd8759700 1 -- 172.21.15.113:0/2797953436 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fabbc032660 msgr2=0x7fabbc034b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:08.405 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.402+0000 7fabd8759700 1 --2- 172.21.15.113:0/2797953436 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fabbc032660 0x7fabbc034b20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fabc0009780 tx=0x7fabc0006cc0).stop 2024-04-12T05:00:08.405 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.402+0000 7fabd8759700 1 -- 172.21.15.113:0/2797953436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 msgr2=0x7fabd00fd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:08.405 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.402+0000 7fabd8759700 1 --2- 172.21.15.113:0/2797953436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00fd730 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fabcc00c010 tx=0x7fabcc0090f0).stop 2024-04-12T05:00:08.405 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.402+0000 7fabd8759700 1 -- 172.21.15.113:0/2797953436 shutdown_connections 2024-04-12T05:00:08.405 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.402+0000 7fabd8759700 1 --2- 172.21.15.113:0/2797953436 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fabbc032660 0x7fabbc034b20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:08.405 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.402+0000 7fabd8759700 1 --2- 172.21.15.113:0/2797953436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0071be0 0x7fabd00fd730 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:08.405 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.402+0000 7fabd8759700 1 -- 172.21.15.113:0/2797953436 >> 172.21.15.113:0/2797953436 conn(0x7fabd00f76f0 msgr2=0x7fabd00f91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:08.405 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.402+0000 7fabd8759700 1 -- 172.21.15.113:0/2797953436 shutdown_connections 2024-04-12T05:00:08.405 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:08.402+0000 7fabd8759700 1 -- 172.21.15.113:0/2797953436 wait complete. 2024-04-12T05:00:08.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:07.003026+0000 mon.smithi113 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:08.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:07.003301+0000 mgr.smithi113.ctsazg (mgr.14114) 10 : cephadm [INF] Added host smithi113 2024-04-12T05:00:08.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:07.028547+0000 mon.smithi113 (mon.0) 53 : audit [DBG] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:00:09.005 INFO:teuthology.orchestra.run.smithi113.stderr:Deploying mgr service with default placement... 2024-04-12T05:00:09.604 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.600+0000 7fb5fc79b700 1 Processor -- start 2024-04-12T05:00:09.604 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.600+0000 7fb5fc79b700 1 -- start start 2024-04-12T05:00:09.604 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.601+0000 7fb5fc79b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40ffc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:09.604 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.601+0000 7fb5fc79b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f4100120 con 0x7fb5f4071c60 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.601+0000 7fb5fa537700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40ffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.601+0000 7fb5fa537700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40ffc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49170/0 (socket says 172.21.15.113:49170) 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.601+0000 7fb5fa537700 1 -- 172.21.15.113:0/4053338768 learned_addr learned my addr 172.21.15.113:0/4053338768 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.601+0000 7fb5fa537700 1 -- 172.21.15.113:0/4053338768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f4100260 con 0x7fb5f4071c60 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.601+0000 7fb5fa537700 1 --2- 172.21.15.113:0/4053338768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40ffc30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb5e4009730 tx=0x7fb5e40090f0).ready entity=mon.0 client_cookie=bbbb3ea8153323f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.601+0000 7fb5f9535700 1 -- 172.21.15.113:0/4053338768 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e400b6e0 con 0x7fb5f4071c60 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.601+0000 7fb5f9535700 1 -- 172.21.15.113:0/4053338768 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fb5e400bc90 con 0x7fb5f4071c60 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.601+0000 7fb5f9535700 1 -- 172.21.15.113:0/4053338768 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e4011c70 con 0x7fb5f4071c60 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.602+0000 7fb5fc79b700 1 -- 172.21.15.113:0/4053338768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 msgr2=0x7fb5f40ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.602+0000 7fb5fc79b700 1 --2- 172.21.15.113:0/4053338768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40ffc30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb5e4009730 tx=0x7fb5e40090f0).stop 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.602+0000 7fb5fc79b700 1 -- 172.21.15.113:0/4053338768 shutdown_connections 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.602+0000 7fb5fc79b700 1 --2- 172.21.15.113:0/4053338768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40ffc30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.602+0000 7fb5fc79b700 1 -- 172.21.15.113:0/4053338768 >> 172.21.15.113:0/4053338768 conn(0x7fb5f40f76f0 msgr2=0x7fb5f40f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.602+0000 7fb5fc79b700 1 -- 172.21.15.113:0/4053338768 shutdown_connections 2024-04-12T05:00:09.605 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.602+0000 7fb5fc79b700 1 -- 172.21.15.113:0/4053338768 wait complete. 2024-04-12T05:00:09.607 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5fc79b700 1 Processor -- start 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5fc79b700 1 -- start start 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5fc79b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40fd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5fc79b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f40fbb30 con 0x7fb5f4071c60 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5fa537700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40fd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5fa537700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40fd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49176/0 (socket says 172.21.15.113:49176) 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5fa537700 1 -- 172.21.15.113:0/915212878 learned_addr learned my addr 172.21.15.113:0/915212878 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5fa537700 1 -- 172.21.15.113:0/915212878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4009370 con 0x7fb5f4071c60 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5fa537700 1 --2- 172.21.15.113:0/915212878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40fd6b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb5e4000c00 tx=0x7fb5e40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5eb7fe700 1 -- 172.21.15.113:0/915212878 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e401a610 con 0x7fb5f4071c60 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5eb7fe700 1 -- 172.21.15.113:0/915212878 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 982+0+0 (secure 0 0 0) 0x7fb5e401abc0 con 0x7fb5f4071c60 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5eb7fe700 1 -- 172.21.15.113:0/915212878 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e4022a20 con 0x7fb5f4071c60 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.603+0000 7fb5fc79b700 1 -- 172.21.15.113:0/915212878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f40fbcd0 con 0x7fb5f4071c60 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.604+0000 7fb5fc79b700 1 -- 172.21.15.113:0/915212878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f40fc0e0 con 0x7fb5f4071c60 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.604+0000 7fb5fc79b700 1 -- 172.21.15.113:0/915212878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f4046ce0 con 0x7fb5f4071c60 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.605+0000 7fb5eb7fe700 1 -- 172.21.15.113:0/915212878 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7fb5e4022ce0 con 0x7fb5f4071c60 2024-04-12T05:00:09.608 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.605+0000 7fb5eb7fe700 1 --2- 172.21.15.113:0/915212878 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb5e0032610 0x7fb5e0034ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:09.609 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.605+0000 7fb5eb7fe700 1 -- 172.21.15.113:0/915212878 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fb5e4044f90 con 0x7fb5f4071c60 2024-04-12T05:00:09.609 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.605+0000 7fb5f9d36700 1 --2- 172.21.15.113:0/915212878 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb5e0032610 0x7fb5e0034ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:09.609 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.605+0000 7fb5f9d36700 1 --2- 172.21.15.113:0/915212878 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb5e0032610 0x7fb5e0034ad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb5f0009780 tx=0x7fb5f0006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:09.611 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.609+0000 7fb5eb7fe700 1 -- 172.21.15.113:0/915212878 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb5e40223b0 con 0x7fb5f4071c60 2024-04-12T05:00:09.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:08.397335+0000 mgr.smithi113.ctsazg (mgr.14114) 11 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:09.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:08.398498+0000 mgr.smithi113.ctsazg (mgr.14114) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-04-12T05:00:09.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:08.400709+0000 mon.smithi113 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:09.793 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.791+0000 7fb5fc79b700 1 -- 172.21.15.113:0/915212878 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5f4105cd0 con 0x7fb5e0032610 2024-04-12T05:00:09.798 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.796+0000 7fb5eb7fe700 1 -- 172.21.15.113:0/915212878 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb5f4105cd0 con 0x7fb5e0032610 2024-04-12T05:00:09.798 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-12T05:00:09.801 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.798+0000 7fb5fc79b700 1 -- 172.21.15.113:0/915212878 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb5e0032610 msgr2=0x7fb5e0034ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:09.801 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.798+0000 7fb5fc79b700 1 --2- 172.21.15.113:0/915212878 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb5e0032610 0x7fb5e0034ad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb5f0009780 tx=0x7fb5f0006cc0).stop 2024-04-12T05:00:09.801 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.798+0000 7fb5fc79b700 1 -- 172.21.15.113:0/915212878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 msgr2=0x7fb5f40fd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:09.802 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.798+0000 7fb5fc79b700 1 --2- 172.21.15.113:0/915212878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40fd6b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb5e4000c00 tx=0x7fb5e40090f0).stop 2024-04-12T05:00:09.802 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.798+0000 7fb5fc79b700 1 -- 172.21.15.113:0/915212878 shutdown_connections 2024-04-12T05:00:09.802 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.798+0000 7fb5fc79b700 1 --2- 172.21.15.113:0/915212878 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb5e0032610 0x7fb5e0034ad0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:09.802 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.798+0000 7fb5fc79b700 1 --2- 172.21.15.113:0/915212878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5f4071c60 0x7fb5f40fd6b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:09.802 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.798+0000 7fb5fc79b700 1 -- 172.21.15.113:0/915212878 >> 172.21.15.113:0/915212878 conn(0x7fb5f40f76f0 msgr2=0x7fb5f40f9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:09.802 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.799+0000 7fb5fc79b700 1 -- 172.21.15.113:0/915212878 shutdown_connections 2024-04-12T05:00:09.802 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:09.799+0000 7fb5fc79b700 1 -- 172.21.15.113:0/915212878 wait complete. 2024-04-12T05:00:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:09.793167+0000 mgr.smithi113.ctsazg (mgr.14114) 13 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:09.794066+0000 mgr.smithi113.ctsazg (mgr.14114) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-04-12T05:00:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:09.796474+0000 mon.smithi113 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:11.244 INFO:teuthology.orchestra.run.smithi113.stderr:Deploying crash service with default placement... 2024-04-12T05:00:12.164 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:10.807384+0000 mon.smithi113 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.686+0000 7f0aaafae700 1 Processor -- start 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.686+0000 7f0aaafae700 1 -- start start 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.686+0000 7f0aaafae700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa40feb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.686+0000 7f0aaafae700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa40feff0 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.687+0000 7f0aa8d4a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa40feb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.687+0000 7f0aa8d4a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa40feb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42328/0 (socket says 172.21.15.113:42328) 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.687+0000 7f0aa8d4a700 1 -- 172.21.15.113:0/1217560392 learned_addr learned my addr 172.21.15.113:0/1217560392 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.687+0000 7f0aa8d4a700 1 -- 172.21.15.113:0/1217560392 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa40ff130 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.687+0000 7f0aa8d4a700 1 --2- 172.21.15.113:0/1217560392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa40feb00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0a98009730 tx=0x7f0a980090f0).ready entity=mon.0 client_cookie=36f2898cfe7fa56b server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.687+0000 7f0aa37fe700 1 -- 172.21.15.113:0/1217560392 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a9800b6e0 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.687+0000 7f0aa37fe700 1 -- 172.21.15.113:0/1217560392 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f0a9800bcd0 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.726 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.687+0000 7f0aa37fe700 1 -- 172.21.15.113:0/1217560392 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a98011cb0 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.688+0000 7f0aaafae700 1 -- 172.21.15.113:0/1217560392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 msgr2=0x7f0aa40feb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.688+0000 7f0aaafae700 1 --2- 172.21.15.113:0/1217560392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa40feb00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0a98009730 tx=0x7f0a980090f0).stop 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.688+0000 7f0aaafae700 1 -- 172.21.15.113:0/1217560392 shutdown_connections 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.688+0000 7f0aaafae700 1 --2- 172.21.15.113:0/1217560392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa40feb00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.688+0000 7f0aaafae700 1 -- 172.21.15.113:0/1217560392 >> 172.21.15.113:0/1217560392 conn(0x7f0aa40f6e60 msgr2=0x7f0aa40f9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.688+0000 7f0aaafae700 1 -- 172.21.15.113:0/1217560392 shutdown_connections 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.688+0000 7f0aaafae700 1 -- 172.21.15.113:0/1217560392 wait complete. 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.688+0000 7f0aaafae700 1 Processor -- start 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.688+0000 7f0aaafae700 1 -- start start 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.688+0000 7f0aaafae700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa410be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.688+0000 7f0aaafae700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa410c330 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.689+0000 7f0aa8d4a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa410be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.689+0000 7f0aa8d4a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa410be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42340/0 (socket says 172.21.15.113:42340) 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.689+0000 7f0aa8d4a700 1 -- 172.21.15.113:0/216134466 learned_addr learned my addr 172.21.15.113:0/216134466 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:12.727 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.689+0000 7f0aa8d4a700 1 -- 172.21.15.113:0/216134466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a98009370 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.689+0000 7f0aa8d4a700 1 --2- 172.21.15.113:0/216134466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa410be40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0a98002f40 tx=0x7f0a980090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.689+0000 7f0aa1ffb700 1 -- 172.21.15.113:0/216134466 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a9801a590 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.689+0000 7f0aaafae700 1 -- 172.21.15.113:0/216134466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa410c530 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.689+0000 7f0aa1ffb700 1 -- 172.21.15.113:0/216134466 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f0a9801ab80 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.689+0000 7f0aa1ffb700 1 -- 172.21.15.113:0/216134466 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a98022aa0 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.689+0000 7f0aaafae700 1 -- 172.21.15.113:0/216134466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa410c940 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.690+0000 7f0aa1ffb700 1 -- 172.21.15.113:0/216134466 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7f0a98019390 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.690+0000 7f0aa1ffb700 1 --2- 172.21.15.113:0/216134466 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0a94032610 0x7f0a94034ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.690+0000 7f0aaafae700 1 -- 172.21.15.113:0/216134466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa4046ce0 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.690+0000 7f0aa1ffb700 1 -- 172.21.15.113:0/216134466 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f0a98044de0 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.690+0000 7f0aa3fff700 1 --2- 172.21.15.113:0/216134466 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0a94032610 0x7f0a94034ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.690+0000 7f0aa3fff700 1 --2- 172.21.15.113:0/216134466 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0a94032610 0x7f0a94034ad0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0a90009780 tx=0x7f0a90006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:12.728 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.695+0000 7f0aa1ffb700 1 -- 172.21.15.113:0/216134466 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0a9802a300 con 0x7f0aa40fe6e0 2024-04-12T05:00:12.908 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.905+0000 7f0aaafae700 1 -- 172.21.15.113:0/216134466 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f0aa410d140 con 0x7f0a94032610 2024-04-12T05:00:12.912 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.910+0000 7f0aa1ffb700 1 -- 172.21.15.113:0/216134466 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f0aa410d140 con 0x7f0a94032610 2024-04-12T05:00:12.912 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: Scheduled crash update... 2024-04-12T05:00:12.914 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.911+0000 7f0aaafae700 1 -- 172.21.15.113:0/216134466 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0a94032610 msgr2=0x7f0a94034ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.911+0000 7f0aaafae700 1 --2- 172.21.15.113:0/216134466 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0a94032610 0x7f0a94034ad0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0a90009780 tx=0x7f0a90006cc0).stop 2024-04-12T05:00:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.911+0000 7f0aaafae700 1 -- 172.21.15.113:0/216134466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 msgr2=0x7f0aa410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.911+0000 7f0aaafae700 1 --2- 172.21.15.113:0/216134466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa410be40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0a98002f40 tx=0x7f0a980090f0).stop 2024-04-12T05:00:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.912+0000 7f0aaafae700 1 -- 172.21.15.113:0/216134466 shutdown_connections 2024-04-12T05:00:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.912+0000 7f0aaafae700 1 --2- 172.21.15.113:0/216134466 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f0a94032610 0x7f0a94034ad0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.912+0000 7f0aaafae700 1 --2- 172.21.15.113:0/216134466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0aa40fe6e0 0x7f0aa410be40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.912+0000 7f0aaafae700 1 -- 172.21.15.113:0/216134466 >> 172.21.15.113:0/216134466 conn(0x7f0aa40f6e60 msgr2=0x7f0aa4100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.912+0000 7f0aaafae700 1 -- 172.21.15.113:0/216134466 shutdown_connections 2024-04-12T05:00:12.915 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:12.912+0000 7f0aaafae700 1 -- 172.21.15.113:0/216134466 wait complete. 2024-04-12T05:00:13.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:12.280729+0000 mon.smithi113 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:13.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:12.910581+0000 mon.smithi113 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:13.720 INFO:teuthology.orchestra.run.smithi113.stderr:Deploying prometheus service with default placement... 2024-04-12T05:00:14.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.311+0000 7f939d153700 1 Processor -- start 2024-04-12T05:00:14.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.311+0000 7f939d153700 1 -- start start 2024-04-12T05:00:14.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.312+0000 7f939d153700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f93981020b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:14.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.312+0000 7f939d153700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93980fdaa0 con 0x7f9398071be0 2024-04-12T05:00:14.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.312+0000 7f9396d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f93981020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:14.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.312+0000 7f9396d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f93981020b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42344/0 (socket says 172.21.15.113:42344) 2024-04-12T05:00:14.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.312+0000 7f9396d9d700 1 -- 172.21.15.113:0/2252950999 learned_addr learned my addr 172.21.15.113:0/2252950999 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:14.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.312+0000 7f9396d9d700 1 -- 172.21.15.113:0/2252950999 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93981025a0 con 0x7f9398071be0 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.312+0000 7f9396d9d700 1 --2- 172.21.15.113:0/2252950999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f93981020b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9388009730 tx=0x7f93880090f0).ready entity=mon.0 client_cookie=2fc48daa634fe715 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.312+0000 7f9395d9b700 1 -- 172.21.15.113:0/2252950999 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f938800b6e0 con 0x7f9398071be0 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.312+0000 7f9395d9b700 1 -- 172.21.15.113:0/2252950999 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f938800bcd0 con 0x7f9398071be0 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.312+0000 7f9395d9b700 1 -- 172.21.15.113:0/2252950999 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9388011cb0 con 0x7f9398071be0 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.313+0000 7f939d153700 1 -- 172.21.15.113:0/2252950999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 msgr2=0x7f93981020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.313+0000 7f939d153700 1 --2- 172.21.15.113:0/2252950999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f93981020b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9388009730 tx=0x7f93880090f0).stop 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.313+0000 7f939d153700 1 -- 172.21.15.113:0/2252950999 shutdown_connections 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.313+0000 7f939d153700 1 --2- 172.21.15.113:0/2252950999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f93981020b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.313+0000 7f939d153700 1 -- 172.21.15.113:0/2252950999 >> 172.21.15.113:0/2252950999 conn(0x7f93980f76f0 msgr2=0x7f93980f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.313+0000 7f939d153700 1 -- 172.21.15.113:0/2252950999 shutdown_connections 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.313+0000 7f939d153700 1 -- 172.21.15.113:0/2252950999 wait complete. 2024-04-12T05:00:14.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.314+0000 7f939d153700 1 Processor -- start 2024-04-12T05:00:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.314+0000 7f939d153700 1 -- start start 2024-04-12T05:00:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.314+0000 7f939d153700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f939810dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.314+0000 7f939d153700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939810e4c0 con 0x7f9398071be0 2024-04-12T05:00:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.314+0000 7f9396d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f939810dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.314+0000 7f9396d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f939810dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42356/0 (socket says 172.21.15.113:42356) 2024-04-12T05:00:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.314+0000 7f9396d9d700 1 -- 172.21.15.113:0/2218020241 learned_addr learned my addr 172.21.15.113:0/2218020241 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.315+0000 7f9396d9d700 1 -- 172.21.15.113:0/2218020241 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9388009370 con 0x7f9398071be0 2024-04-12T05:00:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.315+0000 7f9396d9d700 1 --2- 172.21.15.113:0/2218020241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f939810dfd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f93880118b0 tx=0x7f93880090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.315+0000 7f937ffff700 1 -- 172.21.15.113:0/2218020241 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f938800be40 con 0x7f9398071be0 2024-04-12T05:00:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.315+0000 7f939d153700 1 -- 172.21.15.113:0/2218020241 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939810e6c0 con 0x7f9398071be0 2024-04-12T05:00:14.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.315+0000 7f937ffff700 1 -- 172.21.15.113:0/2218020241 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f938801aad0 con 0x7f9398071be0 2024-04-12T05:00:14.321 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.315+0000 7f937ffff700 1 -- 172.21.15.113:0/2218020241 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9388023a80 con 0x7f9398071be0 2024-04-12T05:00:14.321 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.315+0000 7f939d153700 1 -- 172.21.15.113:0/2218020241 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939810ead0 con 0x7f9398071be0 2024-04-12T05:00:14.321 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.316+0000 7f937ffff700 1 -- 172.21.15.113:0/2218020241 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7f9388023cc0 con 0x7f9398071be0 2024-04-12T05:00:14.321 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.316+0000 7f937ffff700 1 --2- 172.21.15.113:0/2218020241 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f9380032610 0x7f9380034ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:14.321 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.316+0000 7f937ffff700 1 -- 172.21.15.113:0/2218020241 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f9388044ae0 con 0x7f9398071be0 2024-04-12T05:00:14.321 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.316+0000 7f939659c700 1 --2- 172.21.15.113:0/2218020241 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f9380032610 0x7f9380034ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:14.321 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.316+0000 7f939d153700 1 -- 172.21.15.113:0/2218020241 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939805a310 con 0x7f9398071be0 2024-04-12T05:00:14.321 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.317+0000 7f939659c700 1 --2- 172.21.15.113:0/2218020241 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f9380032610 0x7f9380034ad0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f938c009780 tx=0x7f938c006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:14.323 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.321+0000 7f937ffff700 1 -- 172.21.15.113:0/2218020241 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f938801e070 con 0x7f9398071be0 2024-04-12T05:00:14.497 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.494+0000 7f939d153700 1 -- 172.21.15.113:0/2218020241 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f9398107e90 con 0x7f9380032610 2024-04-12T05:00:14.501 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.499+0000 7f937ffff700 1 -- 172.21.15.113:0/2218020241 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f9398107e90 con 0x7f9380032610 2024-04-12T05:00:14.501 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-04-12T05:00:14.504 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.500+0000 7f939d153700 1 -- 172.21.15.113:0/2218020241 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f9380032610 msgr2=0x7f9380034ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:14.504 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.500+0000 7f939d153700 1 --2- 172.21.15.113:0/2218020241 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f9380032610 0x7f9380034ad0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f938c009780 tx=0x7f938c006cc0).stop 2024-04-12T05:00:14.504 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.500+0000 7f939d153700 1 -- 172.21.15.113:0/2218020241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 msgr2=0x7f939810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:14.504 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.500+0000 7f939d153700 1 --2- 172.21.15.113:0/2218020241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f939810dfd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f93880118b0 tx=0x7f93880090f0).stop 2024-04-12T05:00:14.504 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.500+0000 7f939d153700 1 -- 172.21.15.113:0/2218020241 shutdown_connections 2024-04-12T05:00:14.504 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.500+0000 7f939d153700 1 --2- 172.21.15.113:0/2218020241 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f9380032610 0x7f9380034ad0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:14.504 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.500+0000 7f939d153700 1 --2- 172.21.15.113:0/2218020241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9398071be0 0x7f939810dfd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:14.504 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.500+0000 7f939d153700 1 -- 172.21.15.113:0/2218020241 >> 172.21.15.113:0/2218020241 conn(0x7f93980f76f0 msgr2=0x7f93980f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:14.504 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.501+0000 7f939d153700 1 -- 172.21.15.113:0/2218020241 shutdown_connections 2024-04-12T05:00:14.505 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:14.501+0000 7f939d153700 1 -- 172.21.15.113:0/2218020241 wait complete. 2024-04-12T05:00:14.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:12.907239+0000 mgr.smithi113.ctsazg (mgr.14114) 15 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:14.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:12.908265+0000 mgr.smithi113.ctsazg (mgr.14114) 16 : cephadm [INF] Saving service crash spec with placement * 2024-04-12T05:00:15.229 INFO:teuthology.orchestra.run.smithi113.stderr:Deploying grafana service with default placement... 2024-04-12T05:00:15.834 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.829+0000 7f67769a2700 1 Processor -- start 2024-04-12T05:00:15.834 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.829+0000 7f67769a2700 1 -- start start 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.829+0000 7f67769a2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f67700fb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.829+0000 7f67769a2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67700fbd30 con 0x7f67700fb400 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.830+0000 7f676ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f67700fb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.830+0000 7f676ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f67700fb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42358/0 (socket says 172.21.15.113:42358) 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.830+0000 7f676ffff700 1 -- 172.21.15.113:0/1651656976 learned_addr learned my addr 172.21.15.113:0/1651656976 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.830+0000 7f676ffff700 1 -- 172.21.15.113:0/1651656976 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67700fbe70 con 0x7f67700fb400 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.830+0000 7f676ffff700 1 --2- 172.21.15.113:0/1651656976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f67700fb840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f6760009730 tx=0x7f67600090f0).ready entity=mon.0 client_cookie=23d376d790f148d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.830+0000 7f676effd700 1 -- 172.21.15.113:0/1651656976 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f676000b6e0 con 0x7f67700fb400 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.830+0000 7f676effd700 1 -- 172.21.15.113:0/1651656976 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f676000bcd0 con 0x7f67700fb400 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.830+0000 7f676effd700 1 -- 172.21.15.113:0/1651656976 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6760011cb0 con 0x7f67700fb400 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.831+0000 7f67769a2700 1 -- 172.21.15.113:0/1651656976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 msgr2=0x7f67700fb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.831+0000 7f67769a2700 1 --2- 172.21.15.113:0/1651656976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f67700fb840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f6760009730 tx=0x7f67600090f0).stop 2024-04-12T05:00:15.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.831+0000 7f67769a2700 1 -- 172.21.15.113:0/1651656976 shutdown_connections 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.831+0000 7f67769a2700 1 --2- 172.21.15.113:0/1651656976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f67700fb840 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.831+0000 7f67769a2700 1 -- 172.21.15.113:0/1651656976 >> 172.21.15.113:0/1651656976 conn(0x7f67700f6ec0 msgr2=0x7f67700f92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.831+0000 7f67769a2700 1 -- 172.21.15.113:0/1651656976 shutdown_connections 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.831+0000 7f67769a2700 1 -- 172.21.15.113:0/1651656976 wait complete. 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.831+0000 7f67769a2700 1 Processor -- start 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.831+0000 7f67769a2700 1 -- start start 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f67769a2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f677010be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f67769a2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677010c310 con 0x7f67700fb400 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f676ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f677010be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f676ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f677010be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42372/0 (socket says 172.21.15.113:42372) 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f676ffff700 1 -- 172.21.15.113:0/4235959452 learned_addr learned my addr 172.21.15.113:0/4235959452 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:15.836 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f676ffff700 1 -- 172.21.15.113:0/4235959452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6760009370 con 0x7f67700fb400 2024-04-12T05:00:15.837 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f676ffff700 1 --2- 172.21.15.113:0/4235959452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f677010be20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f6760012fc0 tx=0x7f67600090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:15.837 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f676d7fa700 1 -- 172.21.15.113:0/4235959452 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f676001a640 con 0x7f67700fb400 2024-04-12T05:00:15.837 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f67769a2700 1 -- 172.21.15.113:0/4235959452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677010c510 con 0x7f67700fb400 2024-04-12T05:00:15.838 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f676d7fa700 1 -- 172.21.15.113:0/4235959452 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f676001ac30 con 0x7f67700fb400 2024-04-12T05:00:15.838 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f676d7fa700 1 -- 172.21.15.113:0/4235959452 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6760023c90 con 0x7f67700fb400 2024-04-12T05:00:15.838 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.832+0000 7f67769a2700 1 -- 172.21.15.113:0/4235959452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677010c920 con 0x7f67700fb400 2024-04-12T05:00:15.838 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.833+0000 7f676d7fa700 1 -- 172.21.15.113:0/4235959452 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7f676002c3f0 con 0x7f67700fb400 2024-04-12T05:00:15.838 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.833+0000 7f67769a2700 1 -- 172.21.15.113:0/4235959452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6770046ce0 con 0x7f67700fb400 2024-04-12T05:00:15.838 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.833+0000 7f676d7fa700 1 --2- 172.21.15.113:0/4235959452 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6758032610 0x7f6758034ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:15.838 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.833+0000 7f676d7fa700 1 -- 172.21.15.113:0/4235959452 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f6760044cd0 con 0x7f67700fb400 2024-04-12T05:00:15.838 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.834+0000 7f676f7fe700 1 --2- 172.21.15.113:0/4235959452 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6758032610 0x7f6758034ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:15.838 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.834+0000 7f676f7fe700 1 --2- 172.21.15.113:0/4235959452 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6758032610 0x7f6758034ad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f6764009780 tx=0x7f6764006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:15.842 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:15.840+0000 7f676d7fa700 1 -- 172.21.15.113:0/4235959452 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6760021910 con 0x7f67700fb400 2024-04-12T05:00:15.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:14.496441+0000 mgr.smithi113.ctsazg (mgr.14114) 17 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:15.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:14.497342+0000 mgr.smithi113.ctsazg (mgr.14114) 18 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-04-12T05:00:15.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:14.499553+0000 mon.smithi113 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:16.043 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.041+0000 7f67769a2700 1 -- 172.21.15.113:0/4235959452 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f677010d120 con 0x7f6758032610 2024-04-12T05:00:16.047 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.045+0000 7f676d7fa700 1 -- 172.21.15.113:0/4235959452 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f677010d120 con 0x7f6758032610 2024-04-12T05:00:16.047 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-04-12T05:00:16.050 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.047+0000 7f67769a2700 1 -- 172.21.15.113:0/4235959452 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6758032610 msgr2=0x7f6758034ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:16.050 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.047+0000 7f67769a2700 1 --2- 172.21.15.113:0/4235959452 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6758032610 0x7f6758034ad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f6764009780 tx=0x7f6764006cc0).stop 2024-04-12T05:00:16.050 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.047+0000 7f67769a2700 1 -- 172.21.15.113:0/4235959452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 msgr2=0x7f677010be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:16.050 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.047+0000 7f67769a2700 1 --2- 172.21.15.113:0/4235959452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f677010be20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f6760012fc0 tx=0x7f67600090f0).stop 2024-04-12T05:00:16.050 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.047+0000 7f67769a2700 1 -- 172.21.15.113:0/4235959452 shutdown_connections 2024-04-12T05:00:16.050 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.047+0000 7f67769a2700 1 --2- 172.21.15.113:0/4235959452 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6758032610 0x7f6758034ad0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:16.050 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.047+0000 7f67769a2700 1 --2- 172.21.15.113:0/4235959452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700fb400 0x7f677010be20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:16.051 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.047+0000 7f67769a2700 1 -- 172.21.15.113:0/4235959452 >> 172.21.15.113:0/4235959452 conn(0x7f67700f6ec0 msgr2=0x7f67700f88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:16.051 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.047+0000 7f67769a2700 1 -- 172.21.15.113:0/4235959452 shutdown_connections 2024-04-12T05:00:16.051 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:16.047+0000 7f67769a2700 1 -- 172.21.15.113:0/4235959452 wait complete. 2024-04-12T05:00:16.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:15.556694+0000 mon.smithi113 (mon.0) 60 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:16.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:16.045779+0000 mon.smithi113 (mon.0) 61 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:17.048 INFO:teuthology.orchestra.run.smithi113.stderr:Deploying node-exporter service with default placement... 2024-04-12T05:00:17.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:16.042527+0000 mgr.smithi113.ctsazg (mgr.14114) 19 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:17.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:16.043362+0000 mgr.smithi113.ctsazg (mgr.14114) 20 : cephadm [INF] Saving service grafana spec with placement count:1 2024-04-12T05:00:17.918 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.708+0000 7f13813e8700 1 Processor -- start 2024-04-12T05:00:17.923 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.708+0000 7f13813e8700 1 -- start start 2024-04-12T05:00:17.923 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.708+0000 7f13813e8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c0fc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.708+0000 7f13813e8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c0fc7f0 con 0x7f137c0fbee0 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.708+0000 7f137affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c0fc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.709+0000 7f137affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c0fc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42378/0 (socket says 172.21.15.113:42378) 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.709+0000 7f137affd700 1 -- 172.21.15.113:0/2576307525 learned_addr learned my addr 172.21.15.113:0/2576307525 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.709+0000 7f137affd700 1 -- 172.21.15.113:0/2576307525 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137c0fc930 con 0x7f137c0fbee0 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.709+0000 7f137affd700 1 --2- 172.21.15.113:0/2576307525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c0fc300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f136c009e50 tx=0x7f136c009840).ready entity=mon.0 client_cookie=566d78b2006958b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.709+0000 7f1379ffb700 1 -- 172.21.15.113:0/2576307525 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f136c00ba20 con 0x7f137c0fbee0 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.709+0000 7f1379ffb700 1 -- 172.21.15.113:0/2576307525 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f136c00bb80 con 0x7f137c0fbee0 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.709+0000 7f1379ffb700 1 -- 172.21.15.113:0/2576307525 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f136c007400 con 0x7f137c0fbee0 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.709+0000 7f13813e8700 1 -- 172.21.15.113:0/2576307525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 msgr2=0x7f137c0fc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.709+0000 7f13813e8700 1 --2- 172.21.15.113:0/2576307525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c0fc300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f136c009e50 tx=0x7f136c009840).stop 2024-04-12T05:00:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.710+0000 7f13813e8700 1 -- 172.21.15.113:0/2576307525 shutdown_connections 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.710+0000 7f13813e8700 1 --2- 172.21.15.113:0/2576307525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c0fc300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.710+0000 7f13813e8700 1 -- 172.21.15.113:0/2576307525 >> 172.21.15.113:0/2576307525 conn(0x7f137c0f7720 msgr2=0x7f137c0f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.710+0000 7f13813e8700 1 -- 172.21.15.113:0/2576307525 shutdown_connections 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.710+0000 7f13813e8700 1 -- 172.21.15.113:0/2576307525 wait complete. 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.710+0000 7f13813e8700 1 Processor -- start 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.710+0000 7f13813e8700 1 -- start start 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.710+0000 7f13813e8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c10ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.710+0000 7f13813e8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c10b1f0 con 0x7f137c0fbee0 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.711+0000 7f137affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c10ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.711+0000 7f137affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c10ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42388/0 (socket says 172.21.15.113:42388) 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.711+0000 7f137affd700 1 -- 172.21.15.113:0/2650634035 learned_addr learned my addr 172.21.15.113:0/2650634035 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.711+0000 7f137affd700 1 -- 172.21.15.113:0/2650634035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c009ae0 con 0x7f137c0fbee0 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.711+0000 7f137affd700 1 --2- 172.21.15.113:0/2650634035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c10ad00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f136c0072f0 tx=0x7f136c009840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.711+0000 7f1373fff700 1 -- 172.21.15.113:0/2650634035 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f136c007680 con 0x7f137c0fbee0 2024-04-12T05:00:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.711+0000 7f13813e8700 1 -- 172.21.15.113:0/2650634035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137c10b3f0 con 0x7f137c0fbee0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.711+0000 7f1373fff700 1 -- 172.21.15.113:0/2650634035 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f136c007c70 con 0x7f137c0fbee0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.711+0000 7f1373fff700 1 -- 172.21.15.113:0/2650634035 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f136c01caf0 con 0x7f137c0fbee0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.711+0000 7f13813e8700 1 -- 172.21.15.113:0/2650634035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137c10b800 con 0x7f137c0fbee0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.712+0000 7f1373fff700 1 -- 172.21.15.113:0/2650634035 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7f136c003040 con 0x7f137c0fbee0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.712+0000 7f1373fff700 1 --2- 172.21.15.113:0/2650634035 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f13680325e0 0x7f1368034aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.712+0000 7f1373fff700 1 -- 172.21.15.113:0/2650634035 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f136c0440b0 con 0x7f137c0fbee0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.712+0000 7f137a7fc700 1 --2- 172.21.15.113:0/2650634035 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f13680325e0 0x7f1368034aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.712+0000 7f137a7fc700 1 --2- 172.21.15.113:0/2650634035 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f13680325e0 0x7f1368034aa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1364009780 tx=0x7f1364006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.712+0000 7f13813e8700 1 -- 172.21.15.113:0/2650634035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137c046ce0 con 0x7f137c0fbee0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.717+0000 7f1373fff700 1 -- 172.21.15.113:0/2650634035 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f136c01ce00 con 0x7f137c0fbee0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.889+0000 7f13813e8700 1 -- 172.21.15.113:0/2650634035 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f137c10ccc0 con 0x7f13680325e0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.896+0000 7f1373fff700 1 -- 172.21.15.113:0/2650634035 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f137c10ccc0 con 0x7f13680325e0 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.897+0000 7f13813e8700 1 -- 172.21.15.113:0/2650634035 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f13680325e0 msgr2=0x7f1368034aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:17.926 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.897+0000 7f13813e8700 1 --2- 172.21.15.113:0/2650634035 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f13680325e0 0x7f1368034aa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1364009780 tx=0x7f1364006cc0).stop 2024-04-12T05:00:17.927 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.897+0000 7f13813e8700 1 -- 172.21.15.113:0/2650634035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 msgr2=0x7f137c10ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:17.927 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.897+0000 7f13813e8700 1 --2- 172.21.15.113:0/2650634035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c10ad00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f136c0072f0 tx=0x7f136c009840).stop 2024-04-12T05:00:17.927 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.898+0000 7f13813e8700 1 -- 172.21.15.113:0/2650634035 shutdown_connections 2024-04-12T05:00:17.927 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.898+0000 7f13813e8700 1 --2- 172.21.15.113:0/2650634035 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f13680325e0 0x7f1368034aa0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:17.927 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.898+0000 7f13813e8700 1 --2- 172.21.15.113:0/2650634035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f137c0fbee0 0x7f137c10ad00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:17.927 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.898+0000 7f13813e8700 1 -- 172.21.15.113:0/2650634035 >> 172.21.15.113:0/2650634035 conn(0x7f137c0f7720 msgr2=0x7f137c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:17.927 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.898+0000 7f13813e8700 1 -- 172.21.15.113:0/2650634035 shutdown_connections 2024-04-12T05:00:17.927 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:17.898+0000 7f13813e8700 1 -- 172.21.15.113:0/2650634035 wait complete. 2024-04-12T05:00:17.927 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-04-12T05:00:18.859 INFO:teuthology.orchestra.run.smithi113.stderr:Deploying alertmanager service with default placement... 2024-04-12T05:00:19.135 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:17.869520+0000 mgr.smithi113.ctsazg (mgr.14114) 21 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:00:19.135 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:17.891139+0000 mgr.smithi113.ctsazg (mgr.14114) 22 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:19.135 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:17.892027+0000 mgr.smithi113.ctsazg (mgr.14114) 23 : cephadm [INF] Saving service node-exporter spec with placement * 2024-04-12T05:00:19.135 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:17.896634+0000 mon.smithi113 (mon.0) 62 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:19.414 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.410+0000 7fb857031700 1 Processor -- start 2024-04-12T05:00:19.414 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.410+0000 7fb857031700 1 -- start start 2024-04-12T05:00:19.414 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.410+0000 7fb857031700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb8500ffc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.410+0000 7fb857031700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb850100170 con 0x7fb850071c20 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.410+0000 7fb854dcd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb8500ffc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.410+0000 7fb854dcd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb8500ffc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42398/0 (socket says 172.21.15.113:42398) 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.410+0000 7fb854dcd700 1 -- 172.21.15.113:0/2979480597 learned_addr learned my addr 172.21.15.113:0/2979480597 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.411+0000 7fb854dcd700 1 -- 172.21.15.113:0/2979480597 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8501002b0 con 0x7fb850071c20 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.411+0000 7fb854dcd700 1 --2- 172.21.15.113:0/2979480597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb8500ffc80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb844009730 tx=0x7fb8440090f0).ready entity=mon.0 client_cookie=cc396db2ebbfb0b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.411+0000 7fb84f7fe700 1 -- 172.21.15.113:0/2979480597 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb84400b770 con 0x7fb850071c20 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.411+0000 7fb84f7fe700 1 -- 172.21.15.113:0/2979480597 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fb84400bd60 con 0x7fb850071c20 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.411+0000 7fb84f7fe700 1 -- 172.21.15.113:0/2979480597 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb844011d90 con 0x7fb850071c20 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.411+0000 7fb857031700 1 -- 172.21.15.113:0/2979480597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 msgr2=0x7fb8500ffc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.411+0000 7fb857031700 1 --2- 172.21.15.113:0/2979480597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb8500ffc80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb844009730 tx=0x7fb8440090f0).stop 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.411+0000 7fb857031700 1 -- 172.21.15.113:0/2979480597 shutdown_connections 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.411+0000 7fb857031700 1 --2- 172.21.15.113:0/2979480597 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb8500ffc80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.411+0000 7fb857031700 1 -- 172.21.15.113:0/2979480597 >> 172.21.15.113:0/2979480597 conn(0x7fb8500f76c0 msgr2=0x7fb8500f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:19.415 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.412+0000 7fb857031700 1 -- 172.21.15.113:0/2979480597 shutdown_connections 2024-04-12T05:00:19.416 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.412+0000 7fb857031700 1 -- 172.21.15.113:0/2979480597 wait complete. 2024-04-12T05:00:19.416 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.412+0000 7fb857031700 1 Processor -- start 2024-04-12T05:00:19.416 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.412+0000 7fb857031700 1 -- start start 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb857031700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb85010bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb857031700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85010c3b0 con 0x7fb850071c20 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb854dcd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb85010bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb854dcd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb85010bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42414/0 (socket says 172.21.15.113:42414) 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb854dcd700 1 -- 172.21.15.113:0/1914611465 learned_addr learned my addr 172.21.15.113:0/1914611465 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb854dcd700 1 -- 172.21.15.113:0/1914611465 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb844009370 con 0x7fb850071c20 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb854dcd700 1 --2- 172.21.15.113:0/1914611465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb85010bec0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fb844009f00 tx=0x7fb8440090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb84dffb700 1 -- 172.21.15.113:0/1914611465 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb844011bb0 con 0x7fb850071c20 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb84dffb700 1 -- 172.21.15.113:0/1914611465 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fb84400bbc0 con 0x7fb850071c20 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb857031700 1 -- 172.21.15.113:0/1914611465 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85010c5b0 con 0x7fb850071c20 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb84dffb700 1 -- 172.21.15.113:0/1914611465 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb84401ab20 con 0x7fb850071c20 2024-04-12T05:00:19.418 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.413+0000 7fb857031700 1 -- 172.21.15.113:0/1914611465 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85010c9c0 con 0x7fb850071c20 2024-04-12T05:00:19.419 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.414+0000 7fb84dffb700 1 -- 172.21.15.113:0/1914611465 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7fb8440223b0 con 0x7fb850071c20 2024-04-12T05:00:19.419 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.414+0000 7fb857031700 1 -- 172.21.15.113:0/1914611465 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb850046ce0 con 0x7fb850071c20 2024-04-12T05:00:19.419 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.414+0000 7fb84dffb700 1 --2- 172.21.15.113:0/1914611465 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb8400325e0 0x7fb840034aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:19.419 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.414+0000 7fb84dffb700 1 -- 172.21.15.113:0/1914611465 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fb844044f40 con 0x7fb850071c20 2024-04-12T05:00:19.419 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.414+0000 7fb84ffff700 1 --2- 172.21.15.113:0/1914611465 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb8400325e0 0x7fb840034aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:19.419 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.415+0000 7fb84ffff700 1 --2- 172.21.15.113:0/1914611465 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb8400325e0 0x7fb840034aa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fb83c009780 tx=0x7fb83c006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:19.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.419+0000 7fb84dffb700 1 -- 172.21.15.113:0/1914611465 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb844018ad0 con 0x7fb850071c20 2024-04-12T05:00:19.593 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.591+0000 7fb857031700 1 -- 172.21.15.113:0/1914611465 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fb850105e20 con 0x7fb8400325e0 2024-04-12T05:00:19.597 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.595+0000 7fb84dffb700 1 -- 172.21.15.113:0/1914611465 <== mgr.14114 v2:172.21.15.113:6800/2567243935 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fb83c0113f0 con 0x7fb8400325e0 2024-04-12T05:00:19.597 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-04-12T05:00:19.600 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.597+0000 7fb857031700 1 -- 172.21.15.113:0/1914611465 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb8400325e0 msgr2=0x7fb840034aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:19.600 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.597+0000 7fb857031700 1 --2- 172.21.15.113:0/1914611465 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb8400325e0 0x7fb840034aa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fb83c009780 tx=0x7fb83c006cc0).stop 2024-04-12T05:00:19.600 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.597+0000 7fb857031700 1 -- 172.21.15.113:0/1914611465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 msgr2=0x7fb85010bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:19.600 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.597+0000 7fb857031700 1 --2- 172.21.15.113:0/1914611465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb85010bec0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fb844009f00 tx=0x7fb8440090f0).stop 2024-04-12T05:00:19.600 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.597+0000 7fb857031700 1 -- 172.21.15.113:0/1914611465 shutdown_connections 2024-04-12T05:00:19.600 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.597+0000 7fb857031700 1 --2- 172.21.15.113:0/1914611465 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fb8400325e0 0x7fb840034aa0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:19.600 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.597+0000 7fb857031700 1 --2- 172.21.15.113:0/1914611465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb850071c20 0x7fb85010bec0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:19.600 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.597+0000 7fb857031700 1 -- 172.21.15.113:0/1914611465 >> 172.21.15.113:0/1914611465 conn(0x7fb8500f76c0 msgr2=0x7fb8500f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:19.600 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.597+0000 7fb857031700 1 -- 172.21.15.113:0/1914611465 shutdown_connections 2024-04-12T05:00:19.600 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:19.597+0000 7fb857031700 1 -- 172.21.15.113:0/1914611465 wait complete. 2024-04-12T05:00:20.871 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.863+0000 7f0a19d3d700 1 Processor -- start 2024-04-12T05:00:20.871 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.863+0000 7f0a19d3d700 1 -- start start 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.863+0000 7f0a19d3d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.863+0000 7f0a19d3d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a14100160 con 0x7f0a14071c20 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.864+0000 7f0a137fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.864+0000 7f0a137fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140ffc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36020/0 (socket says 172.21.15.113:36020) 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.864+0000 7f0a137fe700 1 -- 172.21.15.113:0/23229416 learned_addr learned my addr 172.21.15.113:0/23229416 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.864+0000 7f0a137fe700 1 -- 172.21.15.113:0/23229416 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a141002a0 con 0x7f0a14071c20 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.864+0000 7f0a137fe700 1 --2- 172.21.15.113:0/23229416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140ffc70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f0a08009730 tx=0x7f0a080090f0).ready entity=mon.0 client_cookie=c07d8e8f6441b845 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.864+0000 7f0a127fc700 1 -- 172.21.15.113:0/23229416 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a0800b770 con 0x7f0a14071c20 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.864+0000 7f0a127fc700 1 -- 172.21.15.113:0/23229416 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f0a0800bd60 con 0x7f0a14071c20 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.864+0000 7f0a127fc700 1 -- 172.21.15.113:0/23229416 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a08011d90 con 0x7f0a14071c20 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.865+0000 7f0a19d3d700 1 -- 172.21.15.113:0/23229416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 msgr2=0x7f0a140ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.865+0000 7f0a19d3d700 1 --2- 172.21.15.113:0/23229416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140ffc70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f0a08009730 tx=0x7f0a080090f0).stop 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.865+0000 7f0a19d3d700 1 -- 172.21.15.113:0/23229416 shutdown_connections 2024-04-12T05:00:20.872 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.865+0000 7f0a19d3d700 1 --2- 172.21.15.113:0/23229416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140ffc70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.865+0000 7f0a19d3d700 1 -- 172.21.15.113:0/23229416 >> 172.21.15.113:0/23229416 conn(0x7f0a140f76f0 msgr2=0x7f0a140f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.865+0000 7f0a19d3d700 1 -- 172.21.15.113:0/23229416 shutdown_connections 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.865+0000 7f0a19d3d700 1 -- 172.21.15.113:0/23229416 wait complete. 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.865+0000 7f0a19d3d700 1 Processor -- start 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a19d3d700 1 -- start start 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a19d3d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140fd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a19d3d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a140fbb70 con 0x7f0a14071c20 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a137fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140fd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a137fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140fd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36036/0 (socket says 172.21.15.113:36036) 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a137fe700 1 -- 172.21.15.113:0/3972569488 learned_addr learned my addr 172.21.15.113:0/3972569488 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a137fe700 1 -- 172.21.15.113:0/3972569488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a08009370 con 0x7f0a14071c20 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a137fe700 1 --2- 172.21.15.113:0/3972569488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140fd6f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0a08009f00 tx=0x7f0a080090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a10ff9700 1 -- 172.21.15.113:0/3972569488 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a08011bb0 con 0x7f0a14071c20 2024-04-12T05:00:20.873 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a10ff9700 1 -- 172.21.15.113:0/3972569488 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f0a0800bbc0 con 0x7f0a14071c20 2024-04-12T05:00:20.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a19d3d700 1 -- 172.21.15.113:0/3972569488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a140fbd10 con 0x7f0a14071c20 2024-04-12T05:00:20.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a10ff9700 1 -- 172.21.15.113:0/3972569488 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a0801ab20 con 0x7f0a14071c20 2024-04-12T05:00:20.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.866+0000 7f0a19d3d700 1 -- 172.21.15.113:0/3972569488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a140fc120 con 0x7f0a14071c20 2024-04-12T05:00:20.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.867+0000 7f0a10ff9700 1 -- 172.21.15.113:0/3972569488 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7f0a080223b0 con 0x7f0a14071c20 2024-04-12T05:00:20.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.867+0000 7f0a10ff9700 1 --2- 172.21.15.113:0/3972569488 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f09fc032630 0x7f09fc034af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:20.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.867+0000 7f0a10ff9700 1 -- 172.21.15.113:0/3972569488 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f0a08044ed0 con 0x7f0a14071c20 2024-04-12T05:00:20.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.868+0000 7f0a12ffd700 1 --2- 172.21.15.113:0/3972569488 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f09fc032630 0x7f09fc034af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:20.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.868+0000 7f0a19d3d700 1 -- 172.21.15.113:0/3972569488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1405a310 con 0x7f0a14071c20 2024-04-12T05:00:20.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.868+0000 7f0a12ffd700 1 --2- 172.21.15.113:0/3972569488 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f09fc032630 0x7f09fc034af0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0a04009780 tx=0x7f0a04006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:20.874 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:20.872+0000 7f0a10ff9700 1 -- 172.21.15.113:0/3972569488 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0a0801e070 con 0x7f0a14071c20 2024-04-12T05:00:20.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:19.592800+0000 mgr.smithi113.ctsazg (mgr.14114) 24 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:20.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:19.593609+0000 mgr.smithi113.ctsazg (mgr.14114) 25 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-04-12T05:00:20.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:19.595990+0000 mon.smithi113 (mon.0) 63 : audit [INF] from='mgr.14114 172.21.15.113:0/3770707279' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:20.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:19.869951+0000 mgr.smithi113.ctsazg (mgr.14114) 26 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:00:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.034+0000 7f0a19d3d700 1 -- 172.21.15.113:0/3972569488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f0a14105c10 con 0x7f0a14071c20 2024-04-12T05:00:21.042 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.040+0000 7f0a10ff9700 1 -- 172.21.15.113:0/3972569488 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0a08018cf0 con 0x7f0a14071c20 2024-04-12T05:00:21.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.042+0000 7f0a19d3d700 1 -- 172.21.15.113:0/3972569488 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f09fc032630 msgr2=0x7f09fc034af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:21.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.042+0000 7f0a19d3d700 1 --2- 172.21.15.113:0/3972569488 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f09fc032630 0x7f09fc034af0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0a04009780 tx=0x7f0a04006cc0).stop 2024-04-12T05:00:21.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.042+0000 7f0a19d3d700 1 -- 172.21.15.113:0/3972569488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 msgr2=0x7f0a140fd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:21.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.042+0000 7f0a19d3d700 1 --2- 172.21.15.113:0/3972569488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140fd6f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0a08009f00 tx=0x7f0a080090f0).stop 2024-04-12T05:00:21.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.042+0000 7f0a19d3d700 1 -- 172.21.15.113:0/3972569488 shutdown_connections 2024-04-12T05:00:21.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.042+0000 7f0a19d3d700 1 --2- 172.21.15.113:0/3972569488 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f09fc032630 0x7f09fc034af0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:21.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.042+0000 7f0a19d3d700 1 --2- 172.21.15.113:0/3972569488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a14071c20 0x7f0a140fd6f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:21.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.042+0000 7f0a19d3d700 1 -- 172.21.15.113:0/3972569488 >> 172.21.15.113:0/3972569488 conn(0x7f0a140f76f0 msgr2=0x7f0a140f91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:21.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.042+0000 7f0a19d3d700 1 -- 172.21.15.113:0/3972569488 shutdown_connections 2024-04-12T05:00:21.045 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:21.042+0000 7f0a19d3d700 1 -- 172.21.15.113:0/3972569488 wait complete. 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.346+0000 7f3cf1c38700 1 Processor -- start 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.346+0000 7f3cf1c38700 1 -- start start 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.346+0000 7f3cf1c38700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec0fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.346+0000 7f3cf1c38700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0fc7e0 con 0x7f3cec0fbed0 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.347+0000 7f3ceb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec0fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.347+0000 7f3ceb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec0fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36050/0 (socket says 172.21.15.113:36050) 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.347+0000 7f3ceb7fe700 1 -- 172.21.15.113:0/2752897385 learned_addr learned my addr 172.21.15.113:0/2752897385 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.347+0000 7f3ceb7fe700 1 -- 172.21.15.113:0/2752897385 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec0fc920 con 0x7f3cec0fbed0 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.347+0000 7f3ceb7fe700 1 --2- 172.21.15.113:0/2752897385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec0fc2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f3ce4009730 tx=0x7f3ce40090f0).ready entity=mon.0 client_cookie=62ac81c14c44f23f server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.347+0000 7f3cea7fc700 1 -- 172.21.15.113:0/2752897385 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce400b6e0 con 0x7f3cec0fbed0 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.347+0000 7f3cea7fc700 1 -- 172.21.15.113:0/2752897385 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f3ce400bcd0 con 0x7f3cec0fbed0 2024-04-12T05:00:22.362 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.347+0000 7f3cea7fc700 1 -- 172.21.15.113:0/2752897385 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce4011cb0 con 0x7f3cec0fbed0 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.348+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2752897385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 msgr2=0x7f3cec0fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.348+0000 7f3cf1c38700 1 --2- 172.21.15.113:0/2752897385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec0fc2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f3ce4009730 tx=0x7f3ce40090f0).stop 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.348+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2752897385 shutdown_connections 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.348+0000 7f3cf1c38700 1 --2- 172.21.15.113:0/2752897385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec0fc2f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.348+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2752897385 >> 172.21.15.113:0/2752897385 conn(0x7f3cec0f76f0 msgr2=0x7f3cec0f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.348+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2752897385 shutdown_connections 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.348+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2752897385 wait complete. 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.348+0000 7f3cf1c38700 1 Processor -- start 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.348+0000 7f3cf1c38700 1 -- start start 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3cf1c38700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec10eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3cf1c38700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec071ca0 con 0x7f3cec0fbed0 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3ceb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec10eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3ceb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec10eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36056/0 (socket says 172.21.15.113:36056) 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3ceb7fe700 1 -- 172.21.15.113:0/2734932486 learned_addr learned my addr 172.21.15.113:0/2734932486 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3ceb7fe700 1 -- 172.21.15.113:0/2734932486 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce4009370 con 0x7f3cec0fbed0 2024-04-12T05:00:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3ceb7fe700 1 --2- 172.21.15.113:0/2734932486 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec10eaf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f3ce4002f40 tx=0x7f3ce40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3ce8ff9700 1 -- 172.21.15.113:0/2734932486 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce401a590 con 0x7f3cec0fbed0 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3ce8ff9700 1 -- 172.21.15.113:0/2734932486 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f3ce401ab80 con 0x7f3cec0fbed0 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3ce8ff9700 1 -- 172.21.15.113:0/2734932486 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce4022aa0 con 0x7f3cec0fbed0 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2734932486 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cec071ea0 con 0x7f3cec0fbed0 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.349+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2734932486 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cec10f1a0 con 0x7f3cec0fbed0 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.350+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2734932486 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cec046ce0 con 0x7f3cec0fbed0 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.351+0000 7f3ce8ff9700 1 -- 172.21.15.113:0/2734932486 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7f3ce4022d10 con 0x7f3cec0fbed0 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.351+0000 7f3ce8ff9700 1 --2- 172.21.15.113:0/2734932486 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f3cd4032660 0x7f3cd4034b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.351+0000 7f3ce8ff9700 1 -- 172.21.15.113:0/2734932486 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f3ce4045130 con 0x7f3cec0fbed0 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.351+0000 7f3ceaffd700 1 --2- 172.21.15.113:0/2734932486 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f3cd4032660 0x7f3cd4034b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.351+0000 7f3ceaffd700 1 --2- 172.21.15.113:0/2734932486 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f3cd4032660 0x7f3cd4034b20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f3ce0009780 tx=0x7f3ce0006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:22.364 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.355+0000 7f3ce8ff9700 1 -- 172.21.15.113:0/2734932486 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3ce402b710 con 0x7f3cec0fbed0 2024-04-12T05:00:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:21.040778+0000 mon.smithi113 (mon.0) 64 : audit [INF] from='client.? 172.21.15.113:0/3972569488' entity='client.admin' 2024-04-12T05:00:22.518 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.514+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2734932486 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f3cec0fdd10 con 0x7f3cec0fbed0 2024-04-12T05:00:22.520 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.517+0000 7f3ce8ff9700 1 -- 172.21.15.113:0/2734932486 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f3ce4028070 con 0x7f3cec0fbed0 2024-04-12T05:00:22.523 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.519+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2734932486 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f3cd4032660 msgr2=0x7f3cd4034b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:22.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.519+0000 7f3cf1c38700 1 --2- 172.21.15.113:0/2734932486 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f3cd4032660 0x7f3cd4034b20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f3ce0009780 tx=0x7f3ce0006cc0).stop 2024-04-12T05:00:22.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.519+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2734932486 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 msgr2=0x7f3cec10eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:22.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.519+0000 7f3cf1c38700 1 --2- 172.21.15.113:0/2734932486 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec10eaf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f3ce4002f40 tx=0x7f3ce40090f0).stop 2024-04-12T05:00:22.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.520+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2734932486 shutdown_connections 2024-04-12T05:00:22.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.520+0000 7f3cf1c38700 1 --2- 172.21.15.113:0/2734932486 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f3cd4032660 0x7f3cd4034b20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:22.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.520+0000 7f3cf1c38700 1 --2- 172.21.15.113:0/2734932486 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cec0fbed0 0x7f3cec10eaf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:22.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.520+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2734932486 >> 172.21.15.113:0/2734932486 conn(0x7f3cec0f76f0 msgr2=0x7f3cec0f9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:22.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.520+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2734932486 shutdown_connections 2024-04-12T05:00:22.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:22.520+0000 7f3cf1c38700 1 -- 172.21.15.113:0/2734932486 wait complete. 2024-04-12T05:00:23.174 INFO:teuthology.orchestra.run.smithi113.stderr:Enabling the dashboard module... 2024-04-12T05:00:23.782 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.778+0000 7fe8e556a700 1 Processor -- start 2024-04-12T05:00:23.782 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.778+0000 7fe8e556a700 1 -- start start 2024-04-12T05:00:23.783 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.778+0000 7fe8e556a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e00fd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:23.783 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.778+0000 7fe8e556a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e00fdd80 con 0x7fe8e00fd470 2024-04-12T05:00:23.783 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.778+0000 7fe8deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e00fd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:23.783 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.778+0000 7fe8deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e00fd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36060/0 (socket says 172.21.15.113:36060) 2024-04-12T05:00:23.783 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.778+0000 7fe8deffd700 1 -- 172.21.15.113:0/130471761 learned_addr learned my addr 172.21.15.113:0/130471761 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:23.783 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.778+0000 7fe8deffd700 1 -- 172.21.15.113:0/130471761 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e00fdec0 con 0x7fe8e00fd470 2024-04-12T05:00:23.783 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.778+0000 7fe8deffd700 1 --2- 172.21.15.113:0/130471761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e00fd890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe8d0009730 tx=0x7fe8d00090f0).ready entity=mon.0 client_cookie=76bbc3c5677669a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:23.783 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.779+0000 7fe8ddffb700 1 -- 172.21.15.113:0/130471761 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8d000b6e0 con 0x7fe8e00fd470 2024-04-12T05:00:23.783 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.779+0000 7fe8ddffb700 1 -- 172.21.15.113:0/130471761 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fe8d000bcd0 con 0x7fe8e00fd470 2024-04-12T05:00:23.785 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.779+0000 7fe8ddffb700 1 -- 172.21.15.113:0/130471761 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8d0011cb0 con 0x7fe8e00fd470 2024-04-12T05:00:23.785 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.779+0000 7fe8e556a700 1 -- 172.21.15.113:0/130471761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 msgr2=0x7fe8e00fd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.779+0000 7fe8e556a700 1 --2- 172.21.15.113:0/130471761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e00fd890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe8d0009730 tx=0x7fe8d00090f0).stop 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.779+0000 7fe8e556a700 1 -- 172.21.15.113:0/130471761 shutdown_connections 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.779+0000 7fe8e556a700 1 --2- 172.21.15.113:0/130471761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e00fd890 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.779+0000 7fe8e556a700 1 -- 172.21.15.113:0/130471761 >> 172.21.15.113:0/130471761 conn(0x7fe8e00f6ec0 msgr2=0x7fe8e00f92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.779+0000 7fe8e556a700 1 -- 172.21.15.113:0/130471761 shutdown_connections 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.779+0000 7fe8e556a700 1 -- 172.21.15.113:0/130471761 wait complete. 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.780+0000 7fe8e556a700 1 Processor -- start 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.780+0000 7fe8e556a700 1 -- start start 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.780+0000 7fe8e556a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e010bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.780+0000 7fe8e556a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e010c280 con 0x7fe8e00fd470 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.780+0000 7fe8deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e010bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.780+0000 7fe8deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e010bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36062/0 (socket says 172.21.15.113:36062) 2024-04-12T05:00:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.780+0000 7fe8deffd700 1 -- 172.21.15.113:0/449961132 learned_addr learned my addr 172.21.15.113:0/449961132 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.781+0000 7fe8deffd700 1 -- 172.21.15.113:0/449961132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0009370 con 0x7fe8e00fd470 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.781+0000 7fe8deffd700 1 --2- 172.21.15.113:0/449961132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e010bd90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe8d0006e30 tx=0x7fe8d00090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.781+0000 7fe8c7fff700 1 -- 172.21.15.113:0/449961132 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8d001a590 con 0x7fe8e00fd470 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.781+0000 7fe8c7fff700 1 -- 172.21.15.113:0/449961132 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fe8d001ab80 con 0x7fe8e00fd470 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.781+0000 7fe8e556a700 1 -- 172.21.15.113:0/449961132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e010c480 con 0x7fe8e00fd470 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.781+0000 7fe8c7fff700 1 -- 172.21.15.113:0/449961132 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8d0022b30 con 0x7fe8e00fd470 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.781+0000 7fe8e556a700 1 -- 172.21.15.113:0/449961132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e010c890 con 0x7fe8e00fd470 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.782+0000 7fe8c7fff700 1 -- 172.21.15.113:0/449961132 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 8) v1 ==== 38632+0+0 (secure 0 0 0) 0x7fe8d0019390 con 0x7fe8e00fd470 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.782+0000 7fe8c7fff700 1 --2- 172.21.15.113:0/449961132 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe8c8032610 0x7fe8c8034ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.782+0000 7fe8e556a700 1 -- 172.21.15.113:0/449961132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e0046ce0 con 0x7fe8e00fd470 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.782+0000 7fe8c7fff700 1 -- 172.21.15.113:0/449961132 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fe8d0044b10 con 0x7fe8e00fd470 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.782+0000 7fe8de7fc700 1 --2- 172.21.15.113:0/449961132 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe8c8032610 0x7fe8c8034ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.782+0000 7fe8de7fc700 1 --2- 172.21.15.113:0/449961132 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe8c8032610 0x7fe8c8034ad0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe8d4009780 tx=0x7fe8d4006cc0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:23.789 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.786+0000 7fe8c7fff700 1 -- 172.21.15.113:0/449961132 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe8d002a300 con 0x7fe8e00fd470 2024-04-12T05:00:23.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:21.870175+0000 mgr.smithi113.ctsazg (mgr.14114) 27 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:00:23.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:22.519508+0000 mon.smithi113 (mon.0) 65 : audit [INF] from='client.? 172.21.15.113:0/2734932486' entity='client.admin' 2024-04-12T05:00:23.993 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:23.990+0000 7fe8e556a700 1 -- 172.21.15.113:0/449961132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fe8e0105e50 con 0x7fe8e00fd470 2024-04-12T05:00:24.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.520+0000 7fe8c7fff700 1 -- 172.21.15.113:0/449961132 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 9) v1 ==== 38645+0+0 (secure 0 0 0) 0x7fe8d00223b0 con 0x7fe8e00fd470 2024-04-12T05:00:24.524 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.521+0000 7fe8c7fff700 1 -- 172.21.15.113:0/449961132 <== mon.0 v2:172.21.15.113:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fe8d0018ad0 con 0x7fe8e00fd470 2024-04-12T05:00:24.527 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.523+0000 7fe8e556a700 1 -- 172.21.15.113:0/449961132 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe8c8032610 msgr2=0x7fe8c8034ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:24.527 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.523+0000 7fe8e556a700 1 --2- 172.21.15.113:0/449961132 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe8c8032610 0x7fe8c8034ad0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fe8d4009780 tx=0x7fe8d4006cc0).stop 2024-04-12T05:00:24.527 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.523+0000 7fe8e556a700 1 -- 172.21.15.113:0/449961132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 msgr2=0x7fe8e010bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:24.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.523+0000 7fe8e556a700 1 --2- 172.21.15.113:0/449961132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e010bd90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe8d0006e30 tx=0x7fe8d00090f0).stop 2024-04-12T05:00:24.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.523+0000 7fe8e556a700 1 -- 172.21.15.113:0/449961132 shutdown_connections 2024-04-12T05:00:24.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.523+0000 7fe8e556a700 1 --2- 172.21.15.113:0/449961132 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe8c8032610 0x7fe8c8034ad0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:24.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.523+0000 7fe8e556a700 1 --2- 172.21.15.113:0/449961132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8e00fd470 0x7fe8e010bd90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:24.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.523+0000 7fe8e556a700 1 -- 172.21.15.113:0/449961132 >> 172.21.15.113:0/449961132 conn(0x7fe8e00f6ec0 msgr2=0x7fe8e00fb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:24.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.524+0000 7fe8e556a700 1 -- 172.21.15.113:0/449961132 shutdown_connections 2024-04-12T05:00:24.528 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:24.524+0000 7fe8e556a700 1 -- 172.21.15.113:0/449961132 wait complete. 2024-04-12T05:00:24.826 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:23.992825+0000 mon.smithi113 (mon.0) 66 : audit [INF] from='client.? 172.21.15.113:0/449961132' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-12T05:00:25.826 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.811+0000 7f6241e0c700 1 Processor -- start 2024-04-12T05:00:25.826 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.811+0000 7f6241e0c700 1 -- start start 2024-04-12T05:00:25.826 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.811+0000 7f6241e0c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c102090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:25.826 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.811+0000 7f6241e0c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c0fda80 con 0x7f623c071c00 2024-04-12T05:00:25.826 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.812+0000 7f623b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:25.826 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.812+0000 7f623b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c102090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36094/0 (socket says 172.21.15.113:36094) 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.812+0000 7f623b7fe700 1 -- 172.21.15.113:0/3631979060 learned_addr learned my addr 172.21.15.113:0/3631979060 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.812+0000 7f623b7fe700 1 -- 172.21.15.113:0/3631979060 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c102580 con 0x7f623c071c00 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.812+0000 7f623b7fe700 1 --2- 172.21.15.113:0/3631979060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c102090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f622c009730 tx=0x7f622c0090f0).ready entity=mon.0 client_cookie=91847220d6b24159 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.812+0000 7f623a7fc700 1 -- 172.21.15.113:0/3631979060 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f622c00b6e0 con 0x7f623c071c00 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.812+0000 7f623a7fc700 1 -- 172.21.15.113:0/3631979060 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f622c00bcd0 con 0x7f623c071c00 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.812+0000 7f623a7fc700 1 -- 172.21.15.113:0/3631979060 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f622c011cb0 con 0x7f623c071c00 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.813+0000 7f6241e0c700 1 -- 172.21.15.113:0/3631979060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 msgr2=0x7f623c102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.813+0000 7f6241e0c700 1 --2- 172.21.15.113:0/3631979060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c102090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f622c009730 tx=0x7f622c0090f0).stop 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.813+0000 7f6241e0c700 1 -- 172.21.15.113:0/3631979060 shutdown_connections 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.813+0000 7f6241e0c700 1 --2- 172.21.15.113:0/3631979060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c102090 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.813+0000 7f6241e0c700 1 -- 172.21.15.113:0/3631979060 >> 172.21.15.113:0/3631979060 conn(0x7f623c0f76f0 msgr2=0x7f623c0f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.813+0000 7f6241e0c700 1 -- 172.21.15.113:0/3631979060 shutdown_connections 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.813+0000 7f6241e0c700 1 -- 172.21.15.113:0/3631979060 wait complete. 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.813+0000 7f6241e0c700 1 Processor -- start 2024-04-12T05:00:25.827 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f6241e0c700 1 -- start start 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f6241e0c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c10e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f6241e0c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c10e540 con 0x7f623c071c00 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f623b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c10e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f623b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c10e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36106/0 (socket says 172.21.15.113:36106) 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f623b7fe700 1 -- 172.21.15.113:0/1176126681 learned_addr learned my addr 172.21.15.113:0/1176126681 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f623b7fe700 1 -- 172.21.15.113:0/1176126681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c009370 con 0x7f623c071c00 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f623b7fe700 1 --2- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c10e050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f622c000c00 tx=0x7f622c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f6238ff9700 1 -- 172.21.15.113:0/1176126681 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f622c01a640 con 0x7f623c071c00 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f6241e0c700 1 -- 172.21.15.113:0/1176126681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623c10e740 con 0x7f623c071c00 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f6238ff9700 1 -- 172.21.15.113:0/1176126681 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f622c01ac30 con 0x7f623c071c00 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f6238ff9700 1 -- 172.21.15.113:0/1176126681 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f622c0239a0 con 0x7f623c071c00 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.814+0000 7f6241e0c700 1 -- 172.21.15.113:0/1176126681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623c10eb50 con 0x7f623c071c00 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.815+0000 7f6238ff9700 1 -- 172.21.15.113:0/1176126681 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 9) v1 ==== 38645+0+0 (secure 0 0 0) 0x7f622c023c10 con 0x7f623c071c00 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.815+0000 7f6238ff9700 1 --2- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6224032660 0x7f6224034b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.815+0000 7f6241e0c700 1 -- 172.21.15.113:0/1176126681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623c046ce0 con 0x7f623c071c00 2024-04-12T05:00:25.828 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.815+0000 7f623affd700 1 -- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6224032660 msgr2=0x7f6224034b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/2567243935 2024-04-12T05:00:25.829 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.815+0000 7f6238ff9700 1 -- 172.21.15.113:0/1176126681 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f622c044c50 con 0x7f623c071c00 2024-04-12T05:00:25.829 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.815+0000 7f623affd700 1 --2- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6224032660 0x7f6224034b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:00:25.829 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:25.820+0000 7f6238ff9700 1 -- 172.21.15.113:0/1176126681 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f622c01e030 con 0x7f623c071c00 2024-04-12T05:00:25.885 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:24.523306+0000 mon.smithi113 (mon.0) 67 : audit [INF] from='client.? 172.21.15.113:0/449961132' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-12T05:00:25.886 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:24.523368+0000 mon.smithi113 (mon.0) 68 : cluster [DBG] mgrmap e9: smithi113.ctsazg(active, since 26s) 2024-04-12T05:00:26.019 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.016+0000 7f623affd700 1 -- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6224032660 msgr2=0x7f6224034b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/2567243935 2024-04-12T05:00:26.019 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.016+0000 7f623affd700 1 --2- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6224032660 0x7f6224034b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-04-12T05:00:26.069 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.065+0000 7f6241e0c700 1 -- 172.21.15.113:0/1176126681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f623c10fc00 con 0x7f623c071c00 2024-04-12T05:00:26.069 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.065+0000 7f6238ff9700 1 -- 172.21.15.113:0/1176126681 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f622c03b070 con 0x7f623c071c00 2024-04-12T05:00:26.069 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: { 2024-04-12T05:00:26.069 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-12T05:00:26.069 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "available": true, 2024-04-12T05:00:26.069 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "active_name": "smithi113.ctsazg", 2024-04-12T05:00:26.069 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-12T05:00:26.069 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: } 2024-04-12T05:00:26.070 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.067+0000 7f6241e0c700 1 -- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6224032660 msgr2=0x7f6224034b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:00:26.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.067+0000 7f6241e0c700 1 --2- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6224032660 0x7f6224034b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:00:26.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.067+0000 7f6241e0c700 1 -- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 msgr2=0x7f623c10e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:26.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.067+0000 7f6241e0c700 1 --2- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c10e050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f622c000c00 tx=0x7f622c0090f0).stop 2024-04-12T05:00:26.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.067+0000 7f6241e0c700 1 -- 172.21.15.113:0/1176126681 shutdown_connections 2024-04-12T05:00:26.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.067+0000 7f6241e0c700 1 --2- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7f6224032660 0x7f6224034b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:00:26.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.067+0000 7f6241e0c700 1 --2- 172.21.15.113:0/1176126681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f623c071c00 0x7f623c10e050 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:26.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.067+0000 7f6241e0c700 1 -- 172.21.15.113:0/1176126681 >> 172.21.15.113:0/1176126681 conn(0x7f623c0f76f0 msgr2=0x7f623c0f9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:26.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.067+0000 7f6241e0c700 1 -- 172.21.15.113:0/1176126681 shutdown_connections 2024-04-12T05:00:26.072 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:26.067+0000 7f6241e0c700 1 -- 172.21.15.113:0/1176126681 wait complete. 2024-04-12T05:00:26.754 INFO:teuthology.orchestra.run.smithi113.stderr:Waiting for the mgr to restart... 2024-04-12T05:00:26.754 INFO:teuthology.orchestra.run.smithi113.stderr:Waiting for mgr epoch 9... 2024-04-12T05:00:26.756 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:26.067332+0000 mon.smithi113 (mon.0) 69 : audit [DBG] from='client.? 172.21.15.113:0/1176126681' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-12T05:00:27.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.311+0000 7fe12210b700 1 Processor -- start 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.311+0000 7fe12210b700 1 -- start start 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.311+0000 7fe12210b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0ffcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.311+0000 7fe12210b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c1001a0 con 0x7fe11c071be0 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.311+0000 7fe11b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0ffcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.311+0000 7fe11b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0ffcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36118/0 (socket says 172.21.15.113:36118) 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.311+0000 7fe11b7fe700 1 -- 172.21.15.113:0/1209257120 learned_addr learned my addr 172.21.15.113:0/1209257120 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.311+0000 7fe11b7fe700 1 -- 172.21.15.113:0/1209257120 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c1002e0 con 0x7fe11c071be0 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.312+0000 7fe11b7fe700 1 --2- 172.21.15.113:0/1209257120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0ffcb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe10c009730 tx=0x7fe10c0090f0).ready entity=mon.0 client_cookie=bc1de3222be11da0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.312+0000 7fe11a7fc700 1 -- 172.21.15.113:0/1209257120 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe10c00b6e0 con 0x7fe11c071be0 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.312+0000 7fe11a7fc700 1 -- 172.21.15.113:0/1209257120 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fe10c00bcd0 con 0x7fe11c071be0 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.312+0000 7fe11a7fc700 1 -- 172.21.15.113:0/1209257120 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe10c011cb0 con 0x7fe11c071be0 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.312+0000 7fe12210b700 1 -- 172.21.15.113:0/1209257120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 msgr2=0x7fe11c0ffcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.312+0000 7fe12210b700 1 --2- 172.21.15.113:0/1209257120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0ffcb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe10c009730 tx=0x7fe10c0090f0).stop 2024-04-12T05:00:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.312+0000 7fe12210b700 1 -- 172.21.15.113:0/1209257120 shutdown_connections 2024-04-12T05:00:27.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.312+0000 7fe12210b700 1 --2- 172.21.15.113:0/1209257120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0ffcb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:27.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.312+0000 7fe12210b700 1 -- 172.21.15.113:0/1209257120 >> 172.21.15.113:0/1209257120 conn(0x7fe11c0f76f0 msgr2=0x7fe11c0f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:27.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.312+0000 7fe12210b700 1 -- 172.21.15.113:0/1209257120 shutdown_connections 2024-04-12T05:00:27.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.313+0000 7fe12210b700 1 -- 172.21.15.113:0/1209257120 wait complete. 2024-04-12T05:00:27.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.313+0000 7fe12210b700 1 Processor -- start 2024-04-12T05:00:27.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.313+0000 7fe12210b700 1 -- start start 2024-04-12T05:00:27.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.313+0000 7fe12210b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0fd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:27.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.313+0000 7fe12210b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c0fbbb0 con 0x7fe11c071be0 2024-04-12T05:00:27.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.314+0000 7fe11b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0fd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.314+0000 7fe11b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0fd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36120/0 (socket says 172.21.15.113:36120) 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.314+0000 7fe11b7fe700 1 -- 172.21.15.113:0/564846861 learned_addr learned my addr 172.21.15.113:0/564846861 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.314+0000 7fe11b7fe700 1 -- 172.21.15.113:0/564846861 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c009370 con 0x7fe11c071be0 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.314+0000 7fe11b7fe700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0fd730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe10c0118b0 tx=0x7fe10c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.314+0000 7fe118ff9700 1 -- 172.21.15.113:0/564846861 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe10c00be40 con 0x7fe11c071be0 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.314+0000 7fe118ff9700 1 -- 172.21.15.113:0/564846861 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fe10c01aad0 con 0x7fe11c071be0 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.314+0000 7fe12210b700 1 -- 172.21.15.113:0/564846861 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c0fbd50 con 0x7fe11c071be0 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.314+0000 7fe118ff9700 1 -- 172.21.15.113:0/564846861 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe10c023a80 con 0x7fe11c071be0 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.314+0000 7fe12210b700 1 -- 172.21.15.113:0/564846861 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c0fc160 con 0x7fe11c071be0 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.315+0000 7fe118ff9700 1 -- 172.21.15.113:0/564846861 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 9) v1 ==== 38645+0+0 (secure 0 0 0) 0x7fe10c023c70 con 0x7fe11c071be0 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.315+0000 7fe118ff9700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 0x7fe104034b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:27.320 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.315+0000 7fe12210b700 1 -- 172.21.15.113:0/564846861 --> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe11c05a310 con 0x7fe1040326b0 2024-04-12T05:00:27.321 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.315+0000 7fe118ff9700 1 -- 172.21.15.113:0/564846861 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fe10c0448b0 con 0x7fe11c071be0 2024-04-12T05:00:27.321 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.315+0000 7fe11affd700 1 -- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 msgr2=0x7fe104034b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/2567243935 2024-04-12T05:00:27.321 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.315+0000 7fe11affd700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 0x7fe104034b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:00:27.519 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.516+0000 7fe11affd700 1 -- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 msgr2=0x7fe104034b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/2567243935 2024-04-12T05:00:27.520 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.516+0000 7fe11affd700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 0x7fe104034b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-04-12T05:00:27.920 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.917+0000 7fe11affd700 1 -- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 msgr2=0x7fe104034b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/2567243935 2024-04-12T05:00:27.920 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:27.917+0000 7fe11affd700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 0x7fe104034b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-04-12T05:00:28.721 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:28.718+0000 7fe11affd700 1 -- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 msgr2=0x7fe104034b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/2567243935 2024-04-12T05:00:28.721 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:28.718+0000 7fe11affd700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 0x7fe104034b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-04-12T05:00:29.833 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:29.829+0000 7fe118ff9700 1 -- 172.21.15.113:0/564846861 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mgrmap(e 10) v1 ==== 38469+0+0 (secure 0 0 0) 0x7fe10c02cc90 con 0x7fe11c071be0 2024-04-12T05:00:29.833 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:29.829+0000 7fe118ff9700 1 -- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 msgr2=0x7fe104034b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:00:29.833 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:29.829+0000 7fe118ff9700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 0x7fe104034b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:00:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:29.828878+0000 mon.smithi113 (mon.0) 70 : cluster [INF] Active manager daemon smithi113.ctsazg restarted 2024-04-12T05:00:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:29.829190+0000 mon.smithi113 (mon.0) 71 : cluster [INF] Activating manager daemon smithi113.ctsazg 2024-04-12T05:00:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:29.832214+0000 mon.smithi113 (mon.0) 72 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-04-12T05:00:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:29.832315+0000 mon.smithi113 (mon.0) 73 : cluster [DBG] mgrmap e10: smithi113.ctsazg(active, starting, since 0.003265s) 2024-04-12T05:00:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:29.833281+0000 mon.smithi113 (mon.0) 74 : audit [DBG] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:00:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:29.833896+0000 mon.smithi113 (mon.0) 75 : audit [DBG] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi113.ctsazg", "id": "smithi113.ctsazg"}]: dispatch 2024-04-12T05:00:30.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:29.834767+0000 mon.smithi113 (mon.0) 76 : audit [DBG] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:00:30.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:29.835003+0000 mon.smithi113 (mon.0) 77 : audit [DBG] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:00:30.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:29.835226+0000 mon.smithi113 (mon.0) 78 : audit [DBG] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:00:30.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:29.849111+0000 mon.smithi113 (mon.0) 79 : cluster [INF] Manager daemon smithi113.ctsazg is now available 2024-04-12T05:00:30.834 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.830+0000 7fe118ff9700 1 -- 172.21.15.113:0/564846861 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 11) v1 ==== 38539+0+0 (secure 0 0 0) 0x7fe10c01a810 con 0x7fe11c071be0 2024-04-12T05:00:30.834 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.830+0000 7fe118ff9700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fe104035fb0 0x7fe1040383a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:30.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.830+0000 7fe118ff9700 1 -- 172.21.15.113:0/564846861 --> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe11c05a310 con 0x7fe104035fb0 2024-04-12T05:00:30.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.831+0000 7fe11affd700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fe104035fb0 0x7fe1040383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:30.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.831+0000 7fe11affd700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fe104035fb0 0x7fe1040383a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe110003850 tx=0x7fe110009350).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:30.835 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.832+0000 7fe118ff9700 1 -- 172.21.15.113:0/564846861 <== mgr.14154 v2:172.21.15.113:6800/1349703403 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fe11c05a310 con 0x7fe104035fb0 2024-04-12T05:00:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.837+0000 7fe12210b700 1 -- 172.21.15.113:0/564846861 --> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe11c046ce0 con 0x7fe104035fb0 2024-04-12T05:00:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe118ff9700 1 -- 172.21.15.113:0/564846861 <== mgr.14154 v2:172.21.15.113:6800/1349703403 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fe11c046ce0 con 0x7fe104035fb0 2024-04-12T05:00:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: { 2024-04-12T05:00:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-12T05:00:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: "initialized": true 2024-04-12T05:00:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: } 2024-04-12T05:00:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe12210b700 1 -- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fe104035fb0 msgr2=0x7fe1040383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe12210b700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fe104035fb0 0x7fe1040383a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe110003850 tx=0x7fe110009350).stop 2024-04-12T05:00:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe12210b700 1 -- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 msgr2=0x7fe11c0fd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe12210b700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0fd730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe10c0118b0 tx=0x7fe10c0090f0).stop 2024-04-12T05:00:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe12210b700 1 -- 172.21.15.113:0/564846861 shutdown_connections 2024-04-12T05:00:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe12210b700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fe104035fb0 0x7fe1040383a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe12210b700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe11c071be0 0x7fe11c0fd730 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe12210b700 1 --2- 172.21.15.113:0/564846861 >> [v2:172.21.15.113:6800/2567243935,v1:172.21.15.113:6801/2567243935] conn(0x7fe1040326b0 0x7fe104034b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:00:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe12210b700 1 -- 172.21.15.113:0/564846861 >> 172.21.15.113:0/564846861 conn(0x7fe11c0f76f0 msgr2=0x7fe11c0f9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe12210b700 1 -- 172.21.15.113:0/564846861 shutdown_connections 2024-04-12T05:00:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:30.838+0000 7fe12210b700 1 -- 172.21.15.113:0/564846861 wait complete. 2024-04-12T05:00:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:29.942409+0000 mon.smithi113 (mon.0) 80 : audit [DBG] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:00:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:29.943734+0000 mon.smithi113 (mon.0) 81 : audit [DBG] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:00:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:29.969572+0000 mon.smithi113 (mon.0) 82 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:00:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:29.972659+0000 mon.smithi113 (mon.0) 83 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/trash_purge_schedule"}]: dispatch 2024-04-12T05:00:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:30.834874+0000 mon.smithi113 (mon.0) 84 : cluster [DBG] mgrmap e11: smithi113.ctsazg(active, since 1.00582s) 2024-04-12T05:00:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:mgr epoch 9 is available 2024-04-12T05:00:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:Generating a dashboard self-signed certificate... 2024-04-12T05:00:32.416 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.411+0000 7f1e3a530700 1 Processor -- start 2024-04-12T05:00:32.417 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.411+0000 7f1e3a530700 1 -- start start 2024-04-12T05:00:32.417 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.411+0000 7f1e3a530700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e340ffc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:32.417 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.411+0000 7f1e3a530700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e34100180 con 0x7f1e34071c00 2024-04-12T05:00:32.417 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.411+0000 7f1e33fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e340ffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:32.417 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.411+0000 7f1e33fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e340ffc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46824/0 (socket says 172.21.15.113:46824) 2024-04-12T05:00:32.417 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.411+0000 7f1e33fff700 1 -- 172.21.15.113:0/2906589784 learned_addr learned my addr 172.21.15.113:0/2906589784 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:32.417 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e33fff700 1 -- 172.21.15.113:0/2906589784 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e341002c0 con 0x7f1e34071c00 2024-04-12T05:00:32.417 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e33fff700 1 --2- 172.21.15.113:0/2906589784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e340ffc90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1e1c009730 tx=0x7f1e1c0090f0).ready entity=mon.0 client_cookie=5ca57a2af55ab70f server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:32.419 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e32ffd700 1 -- 172.21.15.113:0/2906589784 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e1c00b6e0 con 0x7f1e34071c00 2024-04-12T05:00:32.419 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e32ffd700 1 -- 172.21.15.113:0/2906589784 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f1e1c00bcd0 con 0x7f1e34071c00 2024-04-12T05:00:32.419 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e32ffd700 1 -- 172.21.15.113:0/2906589784 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e1c011cb0 con 0x7f1e34071c00 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e3a530700 1 -- 172.21.15.113:0/2906589784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 msgr2=0x7f1e340ffc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e3a530700 1 --2- 172.21.15.113:0/2906589784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e340ffc90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1e1c009730 tx=0x7f1e1c0090f0).stop 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e3a530700 1 -- 172.21.15.113:0/2906589784 shutdown_connections 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e3a530700 1 --2- 172.21.15.113:0/2906589784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e340ffc90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e3a530700 1 -- 172.21.15.113:0/2906589784 >> 172.21.15.113:0/2906589784 conn(0x7f1e340f76f0 msgr2=0x7f1e340f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e3a530700 1 -- 172.21.15.113:0/2906589784 shutdown_connections 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.412+0000 7f1e3a530700 1 -- 172.21.15.113:0/2906589784 wait complete. 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.413+0000 7f1e3a530700 1 Processor -- start 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.413+0000 7f1e3a530700 1 -- start start 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.413+0000 7f1e3a530700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e3410bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.413+0000 7f1e3a530700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3410c3b0 con 0x7f1e34071c00 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.413+0000 7f1e33fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e3410bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.414+0000 7f1e33fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e3410bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46836/0 (socket says 172.21.15.113:46836) 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.414+0000 7f1e33fff700 1 -- 172.21.15.113:0/2625516737 learned_addr learned my addr 172.21.15.113:0/2625516737 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:32.420 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.414+0000 7f1e33fff700 1 -- 172.21.15.113:0/2625516737 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c009370 con 0x7f1e34071c00 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.414+0000 7f1e33fff700 1 --2- 172.21.15.113:0/2625516737 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e3410bec0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1e1c000c00 tx=0x7f1e1c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.414+0000 7f1e317fa700 1 -- 172.21.15.113:0/2625516737 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e1c01a640 con 0x7f1e34071c00 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.414+0000 7f1e317fa700 1 -- 172.21.15.113:0/2625516737 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f1e1c01ac30 con 0x7f1e34071c00 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.414+0000 7f1e3a530700 1 -- 172.21.15.113:0/2625516737 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3410c5b0 con 0x7f1e34071c00 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.414+0000 7f1e317fa700 1 -- 172.21.15.113:0/2625516737 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e1c0239a0 con 0x7f1e34071c00 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.414+0000 7f1e3a530700 1 -- 172.21.15.113:0/2625516737 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e3410c9c0 con 0x7f1e34071c00 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.416+0000 7f1e3a530700 1 -- 172.21.15.113:0/2625516737 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e34046ce0 con 0x7f1e34071c00 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.416+0000 7f1e317fa700 1 -- 172.21.15.113:0/2625516737 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 12) v1 ==== 38645+0+0 (secure 0 0 0) 0x7f1e1c01a7a0 con 0x7f1e34071c00 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.416+0000 7f1e317fa700 1 --2- 172.21.15.113:0/2625516737 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1e20032660 0x7f1e20034b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.416+0000 7f1e317fa700 1 -- 172.21.15.113:0/2625516737 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1e1c044d00 con 0x7f1e34071c00 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.416+0000 7f1e337fe700 1 --2- 172.21.15.113:0/2625516737 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1e20032660 0x7f1e20034b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.417+0000 7f1e337fe700 1 --2- 172.21.15.113:0/2625516737 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1e20032660 0x7f1e20034b20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1e24009780 tx=0x7f1e24006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:32.423 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.421+0000 7f1e317fa700 1 -- 172.21.15.113:0/2625516737 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1e1c048030 con 0x7f1e34071c00 2024-04-12T05:00:32.604 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.602+0000 7f1e3a530700 1 -- 172.21.15.113:0/2625516737 --> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e3410d150 con 0x7f1e20032660 2024-04-12T05:00:32.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:30.833636+0000 mgr.smithi113.ctsazg (mgr.14154) 1 : audit [DBG] from='client.14158 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-12T05:00:32.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:30.839939+0000 mgr.smithi113.ctsazg (mgr.14154) 2 : audit [DBG] from='client.14158 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-12T05:00:32.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:31.392521+0000 mon.smithi113 (mon.0) 85 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:32.773 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-12T05:00:32.773 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.770+0000 7f1e317fa700 1 -- 172.21.15.113:0/2625516737 <== mgr.14154 v2:172.21.15.113:6800/1349703403 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f1e3410d150 con 0x7f1e20032660 2024-04-12T05:00:32.775 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.772+0000 7f1e3a530700 1 -- 172.21.15.113:0/2625516737 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1e20032660 msgr2=0x7f1e20034b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:32.775 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.772+0000 7f1e3a530700 1 --2- 172.21.15.113:0/2625516737 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1e20032660 0x7f1e20034b20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1e24009780 tx=0x7f1e24006cc0).stop 2024-04-12T05:00:32.775 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.772+0000 7f1e3a530700 1 -- 172.21.15.113:0/2625516737 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 msgr2=0x7f1e3410bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:32.775 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.772+0000 7f1e3a530700 1 --2- 172.21.15.113:0/2625516737 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e3410bec0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1e1c000c00 tx=0x7f1e1c0090f0).stop 2024-04-12T05:00:32.775 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.773+0000 7f1e3a530700 1 -- 172.21.15.113:0/2625516737 shutdown_connections 2024-04-12T05:00:32.775 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.773+0000 7f1e3a530700 1 --2- 172.21.15.113:0/2625516737 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1e20032660 0x7f1e20034b20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:32.775 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.773+0000 7f1e3a530700 1 --2- 172.21.15.113:0/2625516737 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e34071c00 0x7f1e3410bec0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:32.776 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.773+0000 7f1e3a530700 1 -- 172.21.15.113:0/2625516737 >> 172.21.15.113:0/2625516737 conn(0x7f1e340f76f0 msgr2=0x7f1e340f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:32.776 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.773+0000 7f1e3a530700 1 -- 172.21.15.113:0/2625516737 shutdown_connections 2024-04-12T05:00:32.776 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:32.773+0000 7f1e3a530700 1 -- 172.21.15.113:0/2625516737 wait complete. 2024-04-12T05:00:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:32.393807+0000 mon.smithi113 (mon.0) 86 : cluster [DBG] mgrmap e12: smithi113.ctsazg(active, since 2s) 2024-04-12T05:00:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:32.769206+0000 mon.smithi113 (mon.0) 87 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:32.771100+0000 mon.smithi113 (mon.0) 88 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:34.463 INFO:teuthology.orchestra.run.smithi113.stderr:Creating initial admin user... 2024-04-12T05:00:34.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:32.603888+0000 mgr.smithi113.ctsazg (mgr.14154) 3 : audit [DBG] from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:35.087 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.084+0000 7f2e1e7cc700 1 Processor -- start 2024-04-12T05:00:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.084+0000 7f2e1e7cc700 1 -- start start 2024-04-12T05:00:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.084+0000 7f2e1e7cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e180fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.084+0000 7f2e1e7cc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e180fc7e0 con 0x7f2e180fbed0 2024-04-12T05:00:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.084+0000 7f2e17fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e180fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.084+0000 7f2e17fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e180fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46852/0 (socket says 172.21.15.113:46852) 2024-04-12T05:00:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.084+0000 7f2e17fff700 1 -- 172.21.15.113:0/560514256 learned_addr learned my addr 172.21.15.113:0/560514256 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.085+0000 7f2e17fff700 1 -- 172.21.15.113:0/560514256 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e180fc920 con 0x7f2e180fbed0 2024-04-12T05:00:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.085+0000 7f2e17fff700 1 --2- 172.21.15.113:0/560514256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e180fc2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f2e08009730 tx=0x7f2e080090f0).ready entity=mon.0 client_cookie=bf5bcb42c98edbd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.085+0000 7f2e16ffd700 1 -- 172.21.15.113:0/560514256 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e0800b6e0 con 0x7f2e180fbed0 2024-04-12T05:00:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.085+0000 7f2e16ffd700 1 -- 172.21.15.113:0/560514256 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f2e0800bcd0 con 0x7f2e180fbed0 2024-04-12T05:00:35.088 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.085+0000 7f2e16ffd700 1 -- 172.21.15.113:0/560514256 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e08011cb0 con 0x7f2e180fbed0 2024-04-12T05:00:35.091 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.086+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/560514256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 msgr2=0x7f2e180fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:35.091 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.086+0000 7f2e1e7cc700 1 --2- 172.21.15.113:0/560514256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e180fc2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f2e08009730 tx=0x7f2e080090f0).stop 2024-04-12T05:00:35.091 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.086+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/560514256 shutdown_connections 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.086+0000 7f2e1e7cc700 1 --2- 172.21.15.113:0/560514256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e180fc2f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.086+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/560514256 >> 172.21.15.113:0/560514256 conn(0x7f2e180f76f0 msgr2=0x7f2e180f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.086+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/560514256 shutdown_connections 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.086+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/560514256 wait complete. 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.086+0000 7f2e1e7cc700 1 Processor -- start 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.086+0000 7f2e1e7cc700 1 -- start start 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.087+0000 7f2e1e7cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e1810eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.087+0000 7f2e1e7cc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18071ca0 con 0x7f2e180fbed0 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.087+0000 7f2e17fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e1810eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.087+0000 7f2e17fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e1810eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46862/0 (socket says 172.21.15.113:46862) 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.087+0000 7f2e17fff700 1 -- 172.21.15.113:0/3052250361 learned_addr learned my addr 172.21.15.113:0/3052250361 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.087+0000 7f2e17fff700 1 -- 172.21.15.113:0/3052250361 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e08009370 con 0x7f2e180fbed0 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.087+0000 7f2e17fff700 1 --2- 172.21.15.113:0/3052250361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e1810eb10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2e08002f40 tx=0x7f2e080090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:35.092 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.087+0000 7f2e157fa700 1 -- 172.21.15.113:0/3052250361 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e0801a590 con 0x7f2e180fbed0 2024-04-12T05:00:35.093 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.088+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/3052250361 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e18071ea0 con 0x7f2e180fbed0 2024-04-12T05:00:35.093 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.088+0000 7f2e157fa700 1 -- 172.21.15.113:0/3052250361 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f2e0801ab80 con 0x7f2e180fbed0 2024-04-12T05:00:35.093 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.088+0000 7f2e157fa700 1 -- 172.21.15.113:0/3052250361 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e08022aa0 con 0x7f2e180fbed0 2024-04-12T05:00:35.093 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.088+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/3052250361 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1810f1c0 con 0x7f2e180fbed0 2024-04-12T05:00:35.093 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.088+0000 7f2e157fa700 1 -- 172.21.15.113:0/3052250361 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 12) v1 ==== 38645+0+0 (secure 0 0 0) 0x7f2e08019390 con 0x7f2e180fbed0 2024-04-12T05:00:35.093 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.088+0000 7f2e157fa700 1 --2- 172.21.15.113:0/3052250361 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f2e00032660 0x7f2e00034b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:35.093 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.089+0000 7f2e157fa700 1 -- 172.21.15.113:0/3052250361 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2e08044b90 con 0x7f2e180fbed0 2024-04-12T05:00:35.093 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.089+0000 7f2e177fe700 1 --2- 172.21.15.113:0/3052250361 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f2e00032660 0x7f2e00034b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:35.093 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.089+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/3052250361 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e18046ce0 con 0x7f2e180fbed0 2024-04-12T05:00:35.093 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.089+0000 7f2e177fe700 1 --2- 172.21.15.113:0/3052250361 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f2e00032660 0x7f2e00034b20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2e0c009780 tx=0x7f2e0c006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:35.096 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.094+0000 7f2e157fa700 1 -- 172.21.15.113:0/3052250361 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2e0801e020 con 0x7f2e180fbed0 2024-04-12T05:00:35.275 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.273+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/3052250361 --> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2e1805a130 con 0x7f2e00032660 2024-04-12T05:00:35.492 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.490+0000 7f2e157fa700 1 -- 172.21.15.113:0/3052250361 <== mgr.14154 v2:172.21.15.113:6800/1349703403 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f2e1805a130 con 0x7f2e00032660 2024-04-12T05:00:35.492 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$xwHIn7Xhclh9O4meYLq3ruMGrw4O0mot9VsRumaKhncT4o9FLfWwm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712898035, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-12T05:00:35.495 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.492+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/3052250361 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f2e00032660 msgr2=0x7f2e00034b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:35.495 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.492+0000 7f2e1e7cc700 1 --2- 172.21.15.113:0/3052250361 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f2e00032660 0x7f2e00034b20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2e0c009780 tx=0x7f2e0c006cc0).stop 2024-04-12T05:00:35.495 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.492+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/3052250361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 msgr2=0x7f2e1810eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:35.495 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.492+0000 7f2e1e7cc700 1 --2- 172.21.15.113:0/3052250361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e1810eb10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2e08002f40 tx=0x7f2e080090f0).stop 2024-04-12T05:00:35.495 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.492+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/3052250361 shutdown_connections 2024-04-12T05:00:35.495 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.492+0000 7f2e1e7cc700 1 --2- 172.21.15.113:0/3052250361 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f2e00032660 0x7f2e00034b20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:35.495 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.492+0000 7f2e1e7cc700 1 --2- 172.21.15.113:0/3052250361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2e180fbed0 0x7f2e1810eb10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:35.495 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.492+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/3052250361 >> 172.21.15.113:0/3052250361 conn(0x7f2e180f76f0 msgr2=0x7f2e180f9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:35.495 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.492+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/3052250361 shutdown_connections 2024-04-12T05:00:35.495 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:35.492+0000 7f2e1e7cc700 1 -- 172.21.15.113:0/3052250361 wait complete. 2024-04-12T05:00:36.163 INFO:teuthology.orchestra.run.smithi113.stderr:Fetching dashboard port number... 2024-04-12T05:00:36.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.766+0000 7f1b6d26b700 1 Processor -- start 2024-04-12T05:00:36.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.766+0000 7f1b6d26b700 1 -- start start 2024-04-12T05:00:36.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.766+0000 7f1b6d26b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b680722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:36.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.766+0000 7f1b6d26b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b68074120 con 0x7f1b68073d00 2024-04-12T05:00:36.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.766+0000 7f1b66ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b680722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.766+0000 7f1b66ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b680722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46878/0 (socket says 172.21.15.113:46878) 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.766+0000 7f1b66ffd700 1 -- 172.21.15.113:0/2811533712 learned_addr learned my addr 172.21.15.113:0/2811533712 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.766+0000 7f1b66ffd700 1 -- 172.21.15.113:0/2811533712 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b68072790 con 0x7f1b68073d00 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.766+0000 7f1b66ffd700 1 --2- 172.21.15.113:0/2811533712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b680722a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f1b58009730 tx=0x7f1b580090f0).ready entity=mon.0 client_cookie=dcf2396251657119 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.767+0000 7f1b65ffb700 1 -- 172.21.15.113:0/2811533712 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b5800b6e0 con 0x7f1b68073d00 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.767+0000 7f1b65ffb700 1 -- 172.21.15.113:0/2811533712 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f1b5800bcd0 con 0x7f1b68073d00 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.767+0000 7f1b65ffb700 1 -- 172.21.15.113:0/2811533712 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b58011cb0 con 0x7f1b68073d00 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.767+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2811533712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 msgr2=0x7f1b680722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.767+0000 7f1b6d26b700 1 --2- 172.21.15.113:0/2811533712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b680722a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f1b58009730 tx=0x7f1b580090f0).stop 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.767+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2811533712 shutdown_connections 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.767+0000 7f1b6d26b700 1 --2- 172.21.15.113:0/2811533712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b680722a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.767+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2811533712 >> 172.21.15.113:0/2811533712 conn(0x7f1b680f8810 msgr2=0x7f1b680fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:36.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.767+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2811533712 shutdown_connections 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.767+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2811533712 wait complete. 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.768+0000 7f1b6d26b700 1 Processor -- start 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.768+0000 7f1b6d26b700 1 -- start start 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.768+0000 7f1b6d26b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b6810dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.768+0000 7f1b6d26b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6810e4e0 con 0x7f1b68073d00 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.768+0000 7f1b66ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b6810dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.768+0000 7f1b66ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b6810dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46890/0 (socket says 172.21.15.113:46890) 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.768+0000 7f1b66ffd700 1 -- 172.21.15.113:0/2161011114 learned_addr learned my addr 172.21.15.113:0/2161011114 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.768+0000 7f1b66ffd700 1 -- 172.21.15.113:0/2161011114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b58009370 con 0x7f1b68073d00 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.769+0000 7f1b66ffd700 1 --2- 172.21.15.113:0/2161011114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b6810dff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f1b58006bc0 tx=0x7f1b580090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.769+0000 7f1b4ffff700 1 -- 172.21.15.113:0/2161011114 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b5801a590 con 0x7f1b68073d00 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.769+0000 7f1b4ffff700 1 -- 172.21.15.113:0/2161011114 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f1b5801ab80 con 0x7f1b68073d00 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.769+0000 7f1b4ffff700 1 -- 172.21.15.113:0/2161011114 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b58022af0 con 0x7f1b68073d00 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.769+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2161011114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6810e6e0 con 0x7f1b68073d00 2024-04-12T05:00:36.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.769+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2161011114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6810eaf0 con 0x7f1b68073d00 2024-04-12T05:00:36.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.769+0000 7f1b4ffff700 1 -- 172.21.15.113:0/2161011114 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7f1b58022db0 con 0x7f1b68073d00 2024-04-12T05:00:36.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.770+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2161011114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b68046ce0 con 0x7f1b68073d00 2024-04-12T05:00:36.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.770+0000 7f1b4ffff700 1 --2- 172.21.15.113:0/2161011114 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1b50032700 0x7f1b50034bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:36.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.770+0000 7f1b4ffff700 1 -- 172.21.15.113:0/2161011114 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1b58044be0 con 0x7f1b68073d00 2024-04-12T05:00:36.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.770+0000 7f1b667fc700 1 --2- 172.21.15.113:0/2161011114 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1b50032700 0x7f1b50034bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:36.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.770+0000 7f1b667fc700 1 --2- 172.21.15.113:0/2161011114 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1b50032700 0x7f1b50034bc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f1b5c009780 tx=0x7f1b5c006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:36.812 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.774+0000 7f1b4ffff700 1 -- 172.21.15.113:0/2161011114 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1b58018730 con 0x7f1b68073d00 2024-04-12T05:00:36.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:35.274977+0000 mgr.smithi113.ctsazg (mgr.14154) 4 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:36.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:35.490856+0000 mon.smithi113 (mon.0) 89 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:36.939 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.937+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2161011114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f1b681031b0 con 0x7f1b68073d00 2024-04-12T05:00:36.940 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.937+0000 7f1b4ffff700 1 -- 172.21.15.113:0/2161011114 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f1b5801e020 con 0x7f1b68073d00 2024-04-12T05:00:36.940 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 8443 2024-04-12T05:00:36.942 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.939+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2161011114 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1b50032700 msgr2=0x7f1b50034bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:36.943 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.939+0000 7f1b6d26b700 1 --2- 172.21.15.113:0/2161011114 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1b50032700 0x7f1b50034bc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f1b5c009780 tx=0x7f1b5c006cc0).stop 2024-04-12T05:00:36.943 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.939+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2161011114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 msgr2=0x7f1b6810dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:36.943 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.939+0000 7f1b6d26b700 1 --2- 172.21.15.113:0/2161011114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b6810dff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f1b58006bc0 tx=0x7f1b580090f0).stop 2024-04-12T05:00:36.943 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.939+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2161011114 shutdown_connections 2024-04-12T05:00:36.943 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.939+0000 7f1b6d26b700 1 --2- 172.21.15.113:0/2161011114 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f1b50032700 0x7f1b50034bc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:36.943 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.939+0000 7f1b6d26b700 1 --2- 172.21.15.113:0/2161011114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b68073d00 0x7f1b6810dff0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:36.943 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.939+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2161011114 >> 172.21.15.113:0/2161011114 conn(0x7f1b680f8810 msgr2=0x7f1b680fa240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:36.943 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.939+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2161011114 shutdown_connections 2024-04-12T05:00:36.943 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:36.939+0000 7f1b6d26b700 1 -- 172.21.15.113:0/2161011114 wait complete. 2024-04-12T05:00:37.660 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:36.492468+0000 mon.smithi113 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi113.ctsazg(active, since 6s) 2024-04-12T05:00:37.660 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:36.938589+0000 mon.smithi113 (mon.0) 91 : audit [DBG] from='client.? 172.21.15.113:0/2161011114' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-12T05:00:37.666 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: disabled 2024-04-12T05:00:37.674 INFO:teuthology.orchestra.run.smithi113.stderr:systemctl: inactive 2024-04-12T05:00:37.674 INFO:teuthology.orchestra.run.smithi113.stderr:firewalld.service is not enabled 2024-04-12T05:00:37.675 INFO:teuthology.orchestra.run.smithi113.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-12T05:00:37.677 INFO:teuthology.orchestra.run.smithi113.stderr:Ceph Dashboard is now available at: 2024-04-12T05:00:37.678 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-12T05:00:37.678 INFO:teuthology.orchestra.run.smithi113.stderr: URL: https://smithi113.front.sepia.ceph.com:8443/ 2024-04-12T05:00:37.678 INFO:teuthology.orchestra.run.smithi113.stderr: User: admin 2024-04-12T05:00:37.678 INFO:teuthology.orchestra.run.smithi113.stderr: Password: i3h0553jxh 2024-04-12T05:00:37.678 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-12T05:00:37.678 INFO:teuthology.orchestra.run.smithi113.stderr:Enabling autotune for osd_memory_target 2024-04-12T05:00:38.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.310+0000 7f40049f5700 1 Processor -- start 2024-04-12T05:00:38.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.310+0000 7f40049f5700 1 -- start start 2024-04-12T05:00:38.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.310+0000 7f40049f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f4000102090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:38.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.310+0000 7f40049f5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40000fda80 con 0x7f4000071c00 2024-04-12T05:00:38.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.310+0000 7f3ffe59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f4000102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:38.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.310+0000 7f3ffe59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f4000102090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46902/0 (socket says 172.21.15.113:46902) 2024-04-12T05:00:38.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.310+0000 7f3ffe59c700 1 -- 172.21.15.113:0/907464752 learned_addr learned my addr 172.21.15.113:0/907464752 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:38.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.310+0000 7f3ffe59c700 1 -- 172.21.15.113:0/907464752 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4000102580 con 0x7f4000071c00 2024-04-12T05:00:38.316 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.311+0000 7f3ffe59c700 1 --2- 172.21.15.113:0/907464752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f4000102090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f3ff0009730 tx=0x7f3ff00090f0).ready entity=mon.0 client_cookie=5e508eabfc4ca90f server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.311+0000 7f3ffd59a700 1 -- 172.21.15.113:0/907464752 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ff000b6e0 con 0x7f4000071c00 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.311+0000 7f3ffd59a700 1 -- 172.21.15.113:0/907464752 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f3ff000bcd0 con 0x7f4000071c00 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.311+0000 7f3ffd59a700 1 -- 172.21.15.113:0/907464752 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ff0011cb0 con 0x7f4000071c00 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.311+0000 7f40049f5700 1 -- 172.21.15.113:0/907464752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 msgr2=0x7f4000102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.311+0000 7f40049f5700 1 --2- 172.21.15.113:0/907464752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f4000102090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f3ff0009730 tx=0x7f3ff00090f0).stop 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.311+0000 7f40049f5700 1 -- 172.21.15.113:0/907464752 shutdown_connections 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.311+0000 7f40049f5700 1 --2- 172.21.15.113:0/907464752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f4000102090 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.311+0000 7f40049f5700 1 -- 172.21.15.113:0/907464752 >> 172.21.15.113:0/907464752 conn(0x7f40000f76f0 msgr2=0x7f40000f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.311+0000 7f40049f5700 1 -- 172.21.15.113:0/907464752 shutdown_connections 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.311+0000 7f40049f5700 1 -- 172.21.15.113:0/907464752 wait complete. 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.312+0000 7f40049f5700 1 Processor -- start 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.312+0000 7f40049f5700 1 -- start start 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f40049f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f400010e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f40049f5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400010e550 con 0x7f4000071c00 2024-04-12T05:00:38.317 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f3ffe59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f400010e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:38.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f3ffe59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f400010e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46910/0 (socket says 172.21.15.113:46910) 2024-04-12T05:00:38.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f3ffe59c700 1 -- 172.21.15.113:0/3254656801 learned_addr learned my addr 172.21.15.113:0/3254656801 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:38.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f3ffe59c700 1 -- 172.21.15.113:0/3254656801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff0009370 con 0x7f4000071c00 2024-04-12T05:00:38.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f3ffe59c700 1 --2- 172.21.15.113:0/3254656801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f400010e060 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f3ff00118b0 tx=0x7f3ff00090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:38.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f3fef7fe700 1 -- 172.21.15.113:0/3254656801 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ff000be40 con 0x7f4000071c00 2024-04-12T05:00:38.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f40049f5700 1 -- 172.21.15.113:0/3254656801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400010e750 con 0x7f4000071c00 2024-04-12T05:00:38.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f3fef7fe700 1 -- 172.21.15.113:0/3254656801 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f3ff001aad0 con 0x7f4000071c00 2024-04-12T05:00:38.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f3fef7fe700 1 -- 172.21.15.113:0/3254656801 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ff00239e0 con 0x7f4000071c00 2024-04-12T05:00:38.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.313+0000 7f40049f5700 1 -- 172.21.15.113:0/3254656801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400010eb60 con 0x7f4000071c00 2024-04-12T05:00:38.318 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.314+0000 7f3fef7fe700 1 -- 172.21.15.113:0/3254656801 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7f3ff0023ca0 con 0x7f4000071c00 2024-04-12T05:00:38.319 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.314+0000 7f3fef7fe700 1 --2- 172.21.15.113:0/3254656801 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3fe80326d0 0x7f3fe8034b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:38.319 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.314+0000 7f3fef7fe700 1 -- 172.21.15.113:0/3254656801 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3ff0045050 con 0x7f4000071c00 2024-04-12T05:00:38.319 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.314+0000 7f3ffdd9b700 1 --2- 172.21.15.113:0/3254656801 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3fe80326d0 0x7f3fe8034b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:38.319 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.314+0000 7f40049f5700 1 -- 172.21.15.113:0/3254656801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400005a310 con 0x7f4000071c00 2024-04-12T05:00:38.319 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.315+0000 7f3ffdd9b700 1 --2- 172.21.15.113:0/3254656801 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3fe80326d0 0x7f3fe8034b90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f3ff4009780 tx=0x7f3ff4006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:38.322 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.319+0000 7f3fef7fe700 1 -- 172.21.15.113:0/3254656801 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3ff002b530 con 0x7f4000071c00 2024-04-12T05:00:38.484 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.482+0000 7f40049f5700 1 -- 172.21.15.113:0/3254656801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f4000046ce0 con 0x7f4000071c00 2024-04-12T05:00:38.484 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.482+0000 7f3fef7fe700 1 -- 172.21.15.113:0/3254656801 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f3ff002b350 con 0x7f4000071c00 2024-04-12T05:00:38.487 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.484+0000 7f40049f5700 1 -- 172.21.15.113:0/3254656801 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3fe80326d0 msgr2=0x7f3fe8034b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:38.487 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.484+0000 7f40049f5700 1 --2- 172.21.15.113:0/3254656801 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3fe80326d0 0x7f3fe8034b90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f3ff4009780 tx=0x7f3ff4006cc0).stop 2024-04-12T05:00:38.487 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.484+0000 7f40049f5700 1 -- 172.21.15.113:0/3254656801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 msgr2=0x7f400010e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:38.487 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.484+0000 7f40049f5700 1 --2- 172.21.15.113:0/3254656801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f400010e060 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f3ff00118b0 tx=0x7f3ff00090f0).stop 2024-04-12T05:00:38.487 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.484+0000 7f40049f5700 1 -- 172.21.15.113:0/3254656801 shutdown_connections 2024-04-12T05:00:38.487 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.484+0000 7f40049f5700 1 --2- 172.21.15.113:0/3254656801 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3fe80326d0 0x7f3fe8034b90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:38.487 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.484+0000 7f40049f5700 1 --2- 172.21.15.113:0/3254656801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4000071c00 0x7f400010e060 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:38.487 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.484+0000 7f40049f5700 1 -- 172.21.15.113:0/3254656801 >> 172.21.15.113:0/3254656801 conn(0x7f40000f76f0 msgr2=0x7f40000f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:38.487 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.484+0000 7f40049f5700 1 -- 172.21.15.113:0/3254656801 shutdown_connections 2024-04-12T05:00:38.487 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:38.484+0000 7f40049f5700 1 -- 172.21.15.113:0/3254656801 wait complete. 2024-04-12T05:00:39.806 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.800+0000 7f5402d81700 1 Processor -- start 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.800+0000 7f5402d81700 1 -- start start 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.801+0000 7f5402d81700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc0fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.801+0000 7f5402d81700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc0fc7e0 con 0x7f53fc0fbed0 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.801+0000 7f5400b1d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc0fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.801+0000 7f5400b1d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc0fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46926/0 (socket says 172.21.15.113:46926) 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.801+0000 7f5400b1d700 1 -- 172.21.15.113:0/3517338296 learned_addr learned my addr 172.21.15.113:0/3517338296 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.801+0000 7f5400b1d700 1 -- 172.21.15.113:0/3517338296 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc0fc920 con 0x7f53fc0fbed0 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.801+0000 7f5400b1d700 1 --2- 172.21.15.113:0/3517338296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc0fc2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f53f0009730 tx=0x7f53f00090f0).ready entity=mon.0 client_cookie=b55dec697d231a39 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.801+0000 7f53fb7fe700 1 -- 172.21.15.113:0/3517338296 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53f000b6e0 con 0x7f53fc0fbed0 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.801+0000 7f53fb7fe700 1 -- 172.21.15.113:0/3517338296 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f53f000bcd0 con 0x7f53fc0fbed0 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.801+0000 7f53fb7fe700 1 -- 172.21.15.113:0/3517338296 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53f0011cb0 con 0x7f53fc0fbed0 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.802+0000 7f5402d81700 1 -- 172.21.15.113:0/3517338296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 msgr2=0x7f53fc0fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:39.807 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.802+0000 7f5402d81700 1 --2- 172.21.15.113:0/3517338296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc0fc2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f53f0009730 tx=0x7f53f00090f0).stop 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.802+0000 7f5402d81700 1 -- 172.21.15.113:0/3517338296 shutdown_connections 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.802+0000 7f5402d81700 1 --2- 172.21.15.113:0/3517338296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc0fc2f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.802+0000 7f5402d81700 1 -- 172.21.15.113:0/3517338296 >> 172.21.15.113:0/3517338296 conn(0x7f53fc0f76f0 msgr2=0x7f53fc0f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.802+0000 7f5402d81700 1 -- 172.21.15.113:0/3517338296 shutdown_connections 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.802+0000 7f5402d81700 1 -- 172.21.15.113:0/3517338296 wait complete. 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.802+0000 7f5402d81700 1 Processor -- start 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f5402d81700 1 -- start start 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f5402d81700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f5402d81700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc10a0f0 con 0x7f53fc0fbed0 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f5400b1d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f5400b1d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc109c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46938/0 (socket says 172.21.15.113:46938) 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f5400b1d700 1 -- 172.21.15.113:0/2900703412 learned_addr learned my addr 172.21.15.113:0/2900703412 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f5400b1d700 1 -- 172.21.15.113:0/2900703412 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f0009370 con 0x7f53fc0fbed0 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f5400b1d700 1 --2- 172.21.15.113:0/2900703412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc109c00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f53f0002f40 tx=0x7f53f00090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f53f9ffb700 1 -- 172.21.15.113:0/2900703412 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53f001a590 con 0x7f53fc0fbed0 2024-04-12T05:00:39.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f53f9ffb700 1 -- 172.21.15.113:0/2900703412 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f53f001ab80 con 0x7f53fc0fbed0 2024-04-12T05:00:39.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f5402d81700 1 -- 172.21.15.113:0/2900703412 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53fc10a2f0 con 0x7f53fc0fbed0 2024-04-12T05:00:39.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f53f9ffb700 1 -- 172.21.15.113:0/2900703412 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53f0022aa0 con 0x7f53fc0fbed0 2024-04-12T05:00:39.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.803+0000 7f5402d81700 1 -- 172.21.15.113:0/2900703412 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53fc10a700 con 0x7f53fc0fbed0 2024-04-12T05:00:39.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.804+0000 7f53f9ffb700 1 -- 172.21.15.113:0/2900703412 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7f53f0022d20 con 0x7f53fc0fbed0 2024-04-12T05:00:39.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.804+0000 7f53f9ffb700 1 --2- 172.21.15.113:0/2900703412 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f53ec034950 0x7f53ec036e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:39.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.804+0000 7f53f9ffb700 1 -- 172.21.15.113:0/2900703412 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f53f0045b20 con 0x7f53fc0fbed0 2024-04-12T05:00:39.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.804+0000 7f53fbfff700 1 --2- 172.21.15.113:0/2900703412 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f53ec034950 0x7f53ec036e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:39.809 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.805+0000 7f5402d81700 1 -- 172.21.15.113:0/2900703412 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53fc05a310 con 0x7f53fc0fbed0 2024-04-12T05:00:39.810 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.805+0000 7f53fbfff700 1 --2- 172.21.15.113:0/2900703412 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f53ec034950 0x7f53ec036e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f53e8009780 tx=0x7f53e8006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:39.811 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:39.809+0000 7f53f9ffb700 1 -- 172.21.15.113:0/2900703412 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f53f001e070 con 0x7f53fc0fbed0 2024-04-12T05:00:40.066 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.063+0000 7f5402d81700 1 -- 172.21.15.113:0/2900703412 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f53fc0fdc20 con 0x7f53fc0fbed0 2024-04-12T05:00:40.068 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.066+0000 7f53f9ffb700 1 -- 172.21.15.113:0/2900703412 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v29)=0 set mgr/dashboard/cluster/status v29) v1 ==== 153+0+0 (secure 0 0 0) 0x7f53f0018970 con 0x7f53fc0fbed0 2024-04-12T05:00:40.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.067+0000 7f5402d81700 1 -- 172.21.15.113:0/2900703412 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f53ec034950 msgr2=0x7f53ec036e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:40.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.067+0000 7f5402d81700 1 --2- 172.21.15.113:0/2900703412 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f53ec034950 0x7f53ec036e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f53e8009780 tx=0x7f53e8006cc0).stop 2024-04-12T05:00:40.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.067+0000 7f5402d81700 1 -- 172.21.15.113:0/2900703412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 msgr2=0x7f53fc109c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:40.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.067+0000 7f5402d81700 1 --2- 172.21.15.113:0/2900703412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc109c00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f53f0002f40 tx=0x7f53f00090f0).stop 2024-04-12T05:00:40.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.067+0000 7f5402d81700 1 -- 172.21.15.113:0/2900703412 shutdown_connections 2024-04-12T05:00:40.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.068+0000 7f5402d81700 1 --2- 172.21.15.113:0/2900703412 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f53ec034950 0x7f53ec036e10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:40.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.068+0000 7f5402d81700 1 --2- 172.21.15.113:0/2900703412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53fc0fbed0 0x7f53fc109c00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:40.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.068+0000 7f5402d81700 1 -- 172.21.15.113:0/2900703412 >> 172.21.15.113:0/2900703412 conn(0x7f53fc0f76f0 msgr2=0x7f53fc100c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:40.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.068+0000 7f5402d81700 1 -- 172.21.15.113:0/2900703412 shutdown_connections 2024-04-12T05:00:40.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: 2024-04-12T05:00:40.068+0000 7f5402d81700 1 -- 172.21.15.113:0/2900703412 wait complete. 2024-04-12T05:00:40.071 INFO:teuthology.orchestra.run.smithi113.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-12T05:00:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-12T05:00:40.752 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-12T05:00:40.752 INFO:teuthology.orchestra.run.smithi113.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-12T05:00:40.752 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-12T05:00:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:Or, if you are only running a single cluster on this host: 2024-04-12T05:00:40.752 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-12T05:00:40.752 INFO:teuthology.orchestra.run.smithi113.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-12T05:00:40.752 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-12T05:00:40.753 INFO:teuthology.orchestra.run.smithi113.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-12T05:00:40.753 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-12T05:00:40.753 INFO:teuthology.orchestra.run.smithi113.stderr: ceph telemetry on 2024-04-12T05:00:40.753 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-12T05:00:40.753 INFO:teuthology.orchestra.run.smithi113.stderr:For more information see: 2024-04-12T05:00:40.753 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-12T05:00:40.753 INFO:teuthology.orchestra.run.smithi113.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-04-12T05:00:40.753 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-04-12T05:00:40.753 INFO:teuthology.orchestra.run.smithi113.stderr:Bootstrap complete. 2024-04-12T05:00:40.789 INFO:tasks.cephadm:Fetching config... 2024-04-12T05:00:40.789 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:00:40.789 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-12T05:00:40.809 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-12T05:00:40.809 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:00:40.809 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-12T05:00:40.867 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-12T05:00:40.867 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:00:40.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/keyring of=/dev/stdout 2024-04-12T05:00:40.934 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-12T05:00:40.934 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:00:40.934 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-12T05:00:40.991 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-12T05:00:40.992 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXs+Pr8AGjugkNdzu1VpDNMF4LYzc73Sw+ke1QTQK/W6wHyIyvUN5gij1EQjKBN0xzHSaDXkDFpkrbRHkU7+taoE80W5SJTrFnJxOXV7a2Ngi82sC11bHjtrkTTosv51rPtR4q8CT6N+ZZYyaVa3ZipMwwXNNZGCfppd6elv/E/neYYCYn1nIExE/O+htrAklvBrx2qXEFOifvUSv311H3ti761gMJ74JKLc82A2H3Ah/BwIoBi83tUWJAud2rsTmbms31h7mmnDdi+BoQM+ga3HzNb6+rs3/FhfbTpAkzHX9pOi0L1CynVWISrr5VmOhdvbgHeqrpcXG3kQfAf12VN2uOGL+1vffXT0DmrzLmHjAf/b8pv92ZNqdlErTwn79/qWIMLZ5thX9V+zVrt/jm17NrQqgViK3MF22cWBy6e3yNgjRVsIb9nGG/bX+YXmHBquMAYAeKRVRML65+yjwdS28+97DhMJK6YfXNmELhOuFSNUxnvrwLd+HUsANcoYk= ceph-52321e2c-f889-11ee-bc8c-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-12T05:00:41.070 INFO:teuthology.orchestra.run.smithi113.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXs+Pr8AGjugkNdzu1VpDNMF4LYzc73Sw+ke1QTQK/W6wHyIyvUN5gij1EQjKBN0xzHSaDXkDFpkrbRHkU7+taoE80W5SJTrFnJxOXV7a2Ngi82sC11bHjtrkTTosv51rPtR4q8CT6N+ZZYyaVa3ZipMwwXNNZGCfppd6elv/E/neYYCYn1nIExE/O+htrAklvBrx2qXEFOifvUSv311H3ti761gMJ74JKLc82A2H3Ah/BwIoBi83tUWJAud2rsTmbms31h7mmnDdi+BoQM+ga3HzNb6+rs3/FhfbTpAkzHX9pOi0L1CynVWISrr5VmOhdvbgHeqrpcXG3kQfAf12VN2uOGL+1vffXT0DmrzLmHjAf/b8pv92ZNqdlErTwn79/qWIMLZ5thX9V+zVrt/jm17NrQqgViK3MF22cWBy6e3yNgjRVsIb9nGG/bX+YXmHBquMAYAeKRVRML65+yjwdS28+97DhMJK6YfXNmELhOuFSNUxnvrwLd+HUsANcoYk= ceph-52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T05:00:41.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXs+Pr8AGjugkNdzu1VpDNMF4LYzc73Sw+ke1QTQK/W6wHyIyvUN5gij1EQjKBN0xzHSaDXkDFpkrbRHkU7+taoE80W5SJTrFnJxOXV7a2Ngi82sC11bHjtrkTTosv51rPtR4q8CT6N+ZZYyaVa3ZipMwwXNNZGCfppd6elv/E/neYYCYn1nIExE/O+htrAklvBrx2qXEFOifvUSv311H3ti761gMJ74JKLc82A2H3Ah/BwIoBi83tUWJAud2rsTmbms31h7mmnDdi+BoQM+ga3HzNb6+rs3/FhfbTpAkzHX9pOi0L1CynVWISrr5VmOhdvbgHeqrpcXG3kQfAf12VN2uOGL+1vffXT0DmrzLmHjAf/b8pv92ZNqdlErTwn79/qWIMLZ5thX9V+zVrt/jm17NrQqgViK3MF22cWBy6e3yNgjRVsIb9nGG/bX+YXmHBquMAYAeKRVRML65+yjwdS28+97DhMJK6YfXNmELhOuFSNUxnvrwLd+HUsANcoYk= ceph-52321e2c-f889-11ee-bc8c-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-12T05:00:41.126 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXs+Pr8AGjugkNdzu1VpDNMF4LYzc73Sw+ke1QTQK/W6wHyIyvUN5gij1EQjKBN0xzHSaDXkDFpkrbRHkU7+taoE80W5SJTrFnJxOXV7a2Ngi82sC11bHjtrkTTosv51rPtR4q8CT6N+ZZYyaVa3ZipMwwXNNZGCfppd6elv/E/neYYCYn1nIExE/O+htrAklvBrx2qXEFOifvUSv311H3ti761gMJ74JKLc82A2H3Ah/BwIoBi83tUWJAud2rsTmbms31h7mmnDdi+BoQM+ga3HzNb6+rs3/FhfbTpAkzHX9pOi0L1CynVWISrr5VmOhdvbgHeqrpcXG3kQfAf12VN2uOGL+1vffXT0DmrzLmHjAf/b8pv92ZNqdlErTwn79/qWIMLZ5thX9V+zVrt/jm17NrQqgViK3MF22cWBy6e3yNgjRVsIb9nGG/bX+YXmHBquMAYAeKRVRML65+yjwdS28+97DhMJK6YfXNmELhOuFSNUxnvrwLd+HUsANcoYk= ceph-52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T05:00:41.138 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-12T05:00:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:40.066796+0000 mon.smithi113 (mon.0) 92 : audit [INF] from='client.? 172.21.15.113:0/2900703412' entity='client.admin' 2024-04-12T05:00:44.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.512+0000 7f711bff2700 1 -- 172.21.15.113:0/1917651492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71140f64b0 msgr2=0x7f71140f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:44.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.512+0000 7f711bff2700 1 --2- 172.21.15.113:0/1917651492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71140f64b0 0x7f71140f68d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f7104004770 tx=0x7f7104009cb0).stop 2024-04-12T05:00:44.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.512+0000 7f711bff2700 1 -- 172.21.15.113:0/1917651492 shutdown_connections 2024-04-12T05:00:44.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.512+0000 7f711bff2700 1 --2- 172.21.15.113:0/1917651492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71140f64b0 0x7f71140f68d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:44.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.512+0000 7f711bff2700 1 -- 172.21.15.113:0/1917651492 >> 172.21.15.113:0/1917651492 conn(0x7f71140f1c10 msgr2=0x7f71140f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:44.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.512+0000 7f711bff2700 1 -- 172.21.15.113:0/1917651492 shutdown_connections 2024-04-12T05:00:44.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.512+0000 7f711bff2700 1 -- 172.21.15.113:0/1917651492 wait complete. 2024-04-12T05:00:44.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f711bff2700 1 Processor -- start 2024-04-12T05:00:44.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f711bff2700 1 -- start start 2024-04-12T05:00:44.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f711bff2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71140f64b0 0x7f711410a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:44.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f711bff2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7114061190 con 0x7f71140f64b0 2024-04-12T05:00:44.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f7119d8e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71140f64b0 0x7f711410a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:44.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f7119d8e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71140f64b0 0x7f711410a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43436/0 (socket says 172.21.15.113:43436) 2024-04-12T05:00:44.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f7119d8e700 1 -- 172.21.15.113:0/4207426296 learned_addr learned my addr 172.21.15.113:0/4207426296 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:44.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f7119d8e700 1 -- 172.21.15.113:0/4207426296 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7104005040 con 0x7f71140f64b0 2024-04-12T05:00:44.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f7119d8e700 1 --2- 172.21.15.113:0/4207426296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71140f64b0 0x7f711410a540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f7104009300 tx=0x7f7104009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:44.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f710affd700 1 -- 172.21.15.113:0/4207426296 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7104018070 con 0x7f71140f64b0 2024-04-12T05:00:44.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f710affd700 1 -- 172.21.15.113:0/4207426296 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f7104009890 con 0x7f71140f64b0 2024-04-12T05:00:44.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f710affd700 1 -- 172.21.15.113:0/4207426296 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f710401c400 con 0x7f71140f64b0 2024-04-12T05:00:44.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.513+0000 7f711bff2700 1 -- 172.21.15.113:0/4207426296 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7114061390 con 0x7f71140f64b0 2024-04-12T05:00:44.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.514+0000 7f711bff2700 1 -- 172.21.15.113:0/4207426296 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f711410abf0 con 0x7f71140f64b0 2024-04-12T05:00:44.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.515+0000 7f710affd700 1 -- 172.21.15.113:0/4207426296 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7f710401d420 con 0x7f71140f64b0 2024-04-12T05:00:44.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.515+0000 7f711bff2700 1 -- 172.21.15.113:0/4207426296 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f711405a310 con 0x7f71140f64b0 2024-04-12T05:00:44.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.515+0000 7f710affd700 1 --2- 172.21.15.113:0/4207426296 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f7100032750 0x7f7100034c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:44.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.515+0000 7f710affd700 1 -- 172.21.15.113:0/4207426296 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f71040446a0 con 0x7f71140f64b0 2024-04-12T05:00:44.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.515+0000 7f711958d700 1 --2- 172.21.15.113:0/4207426296 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f7100032750 0x7f7100034c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:44.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.515+0000 7f711958d700 1 --2- 172.21.15.113:0/4207426296 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f7100032750 0x7f7100034c10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7110009780 tx=0x7f7110006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:44.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.520+0000 7f710affd700 1 -- 172.21.15.113:0/4207426296 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f710401b370 con 0x7f71140f64b0 2024-04-12T05:00:44.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.683+0000 7f711bff2700 1 -- 172.21.15.113:0/4207426296 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f7114046ce0 con 0x7f71140f64b0 2024-04-12T05:00:44.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.686+0000 7f710affd700 1 -- 172.21.15.113:0/4207426296 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f710401b370 con 0x7f71140f64b0 2024-04-12T05:00:44.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.688+0000 7f711bff2700 1 -- 172.21.15.113:0/4207426296 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f7100032750 msgr2=0x7f7100034c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:44.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.688+0000 7f711bff2700 1 --2- 172.21.15.113:0/4207426296 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f7100032750 0x7f7100034c10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7110009780 tx=0x7f7110006cc0).stop 2024-04-12T05:00:44.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.688+0000 7f711bff2700 1 -- 172.21.15.113:0/4207426296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71140f64b0 msgr2=0x7f711410a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:44.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.688+0000 7f711bff2700 1 --2- 172.21.15.113:0/4207426296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71140f64b0 0x7f711410a540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f7104009300 tx=0x7f7104009cb0).stop 2024-04-12T05:00:44.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.688+0000 7f711bff2700 1 -- 172.21.15.113:0/4207426296 shutdown_connections 2024-04-12T05:00:44.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.688+0000 7f711bff2700 1 --2- 172.21.15.113:0/4207426296 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f7100032750 0x7f7100034c10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:44.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.688+0000 7f711bff2700 1 --2- 172.21.15.113:0/4207426296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71140f64b0 0x7f711410a540 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:44.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.688+0000 7f711bff2700 1 -- 172.21.15.113:0/4207426296 >> 172.21.15.113:0/4207426296 conn(0x7f71140f1c10 msgr2=0x7f71140f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:44.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.688+0000 7f711bff2700 1 -- 172.21.15.113:0/4207426296 shutdown_connections 2024-04-12T05:00:44.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:43.688+0000 7f711bff2700 1 -- 172.21.15.113:0/4207426296 wait complete. 2024-04-12T05:00:44.671 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-12T05:00:44.672 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-12T05:00:44.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:43.687297+0000 mon.smithi113 (mon.0) 93 : audit [INF] from='client.? 172.21.15.113:0/4207426296' entity='client.admin' 2024-04-12T05:00:44.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:44.452658+0000 mon.smithi113 (mon.0) 94 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:46.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:44.798690+0000 mon.smithi113 (mon.0) 95 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd/host:smithi113", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:00:46.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:44.801829+0000 mon.smithi113 (mon.0) 96 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:46.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:44.811889+0000 mgr.smithi113.ctsazg (mgr.14154) 5 : cephadm [INF] Deploying daemon alertmanager.smithi113 on smithi113 2024-04-12T05:00:47.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.373+0000 7efc11e6b700 1 -- 172.21.15.113:0/3958405356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0c0f64b0 msgr2=0x7efc0c0f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:47.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.373+0000 7efc11e6b700 1 --2- 172.21.15.113:0/3958405356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0c0f64b0 0x7efc0c0f68d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7efc00004770 tx=0x7efc00009cb0).stop 2024-04-12T05:00:47.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.373+0000 7efc11e6b700 1 -- 172.21.15.113:0/3958405356 shutdown_connections 2024-04-12T05:00:47.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.373+0000 7efc11e6b700 1 --2- 172.21.15.113:0/3958405356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0c0f64b0 0x7efc0c0f68d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:47.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.373+0000 7efc11e6b700 1 -- 172.21.15.113:0/3958405356 >> 172.21.15.113:0/3958405356 conn(0x7efc0c0f1c10 msgr2=0x7efc0c0f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:47.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.373+0000 7efc11e6b700 1 -- 172.21.15.113:0/3958405356 shutdown_connections 2024-04-12T05:00:47.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.373+0000 7efc11e6b700 1 -- 172.21.15.113:0/3958405356 wait complete. 2024-04-12T05:00:47.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.373+0000 7efc11e6b700 1 Processor -- start 2024-04-12T05:00:47.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.374+0000 7efc11e6b700 1 -- start start 2024-04-12T05:00:47.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.374+0000 7efc11e6b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0c0f64b0 0x7efc0c10a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:47.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.374+0000 7efc11e6b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c061190 con 0x7efc0c0f64b0 2024-04-12T05:00:47.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.374+0000 7efc0b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0c0f64b0 0x7efc0c10a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:47.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.374+0000 7efc0b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0c0f64b0 0x7efc0c10a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43456/0 (socket says 172.21.15.113:43456) 2024-04-12T05:00:47.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.374+0000 7efc0b7fe700 1 -- 172.21.15.113:0/4030935908 learned_addr learned my addr 172.21.15.113:0/4030935908 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:47.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.374+0000 7efc0b7fe700 1 -- 172.21.15.113:0/4030935908 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc00005040 con 0x7efc0c0f64b0 2024-04-12T05:00:47.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.374+0000 7efc0b7fe700 1 --2- 172.21.15.113:0/4030935908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0c0f64b0 0x7efc0c10a540 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7efc00009300 tx=0x7efc00009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:47.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.374+0000 7efc08ff9700 1 -- 172.21.15.113:0/4030935908 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc00018070 con 0x7efc0c0f64b0 2024-04-12T05:00:47.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.375+0000 7efc08ff9700 1 -- 172.21.15.113:0/4030935908 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7efc00009890 con 0x7efc0c0f64b0 2024-04-12T05:00:47.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.375+0000 7efc11e6b700 1 -- 172.21.15.113:0/4030935908 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0c061390 con 0x7efc0c0f64b0 2024-04-12T05:00:47.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.375+0000 7efc08ff9700 1 -- 172.21.15.113:0/4030935908 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc0001c400 con 0x7efc0c0f64b0 2024-04-12T05:00:47.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.375+0000 7efc11e6b700 1 -- 172.21.15.113:0/4030935908 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0c10abf0 con 0x7efc0c0f64b0 2024-04-12T05:00:47.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.375+0000 7efc08ff9700 1 -- 172.21.15.113:0/4030935908 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7efc0001d420 con 0x7efc0c0f64b0 2024-04-12T05:00:47.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.375+0000 7efc11e6b700 1 -- 172.21.15.113:0/4030935908 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0c05a310 con 0x7efc0c0f64b0 2024-04-12T05:00:47.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.376+0000 7efc08ff9700 1 --2- 172.21.15.113:0/4030935908 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efbf4032750 0x7efbf4034c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:47.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.376+0000 7efc08ff9700 1 -- 172.21.15.113:0/4030935908 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efc00044580 con 0x7efc0c0f64b0 2024-04-12T05:00:47.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.376+0000 7efc0affd700 1 --2- 172.21.15.113:0/4030935908 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efbf4032750 0x7efbf4034c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:47.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.376+0000 7efc0affd700 1 --2- 172.21.15.113:0/4030935908 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efbf4032750 0x7efbf4034c10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7efbfc009780 tx=0x7efbfc006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:47.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.380+0000 7efc08ff9700 1 -- 172.21.15.113:0/4030935908 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7efc00025b40 con 0x7efc0c0f64b0 2024-04-12T05:00:47.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.563+0000 7efc11e6b700 1 -- 172.21.15.113:0/4030935908 --> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7efc0c0f2d70 con 0x7efbf4032750 2024-04-12T05:00:47.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.566+0000 7efc08ff9700 1 -- 172.21.15.113:0/4030935908 <== mgr.14154 v2:172.21.15.113:6800/1349703403 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efc0c0f2d70 con 0x7efbf4032750 2024-04-12T05:00:47.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.568+0000 7efc11e6b700 1 -- 172.21.15.113:0/4030935908 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efbf4032750 msgr2=0x7efbf4034c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.568+0000 7efc11e6b700 1 --2- 172.21.15.113:0/4030935908 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efbf4032750 0x7efbf4034c10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7efbfc009780 tx=0x7efbfc006cc0).stop 2024-04-12T05:00:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.568+0000 7efc11e6b700 1 -- 172.21.15.113:0/4030935908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0c0f64b0 msgr2=0x7efc0c10a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.568+0000 7efc11e6b700 1 --2- 172.21.15.113:0/4030935908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0c0f64b0 0x7efc0c10a540 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7efc00009300 tx=0x7efc00009cb0).stop 2024-04-12T05:00:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.568+0000 7efc11e6b700 1 -- 172.21.15.113:0/4030935908 shutdown_connections 2024-04-12T05:00:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.568+0000 7efc11e6b700 1 --2- 172.21.15.113:0/4030935908 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efbf4032750 0x7efbf4034c10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.568+0000 7efc11e6b700 1 --2- 172.21.15.113:0/4030935908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc0c0f64b0 0x7efc0c10a540 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.568+0000 7efc11e6b700 1 -- 172.21.15.113:0/4030935908 >> 172.21.15.113:0/4030935908 conn(0x7efc0c0f1c10 msgr2=0x7efc0c0f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.568+0000 7efc11e6b700 1 -- 172.21.15.113:0/4030935908 shutdown_connections 2024-04-12T05:00:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:47.568+0000 7efc11e6b700 1 -- 172.21.15.113:0/4030935908 wait complete. 2024-04-12T05:00:48.655 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2024-04-12T05:00:48.655 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T05:00:48.655 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2024-04-12T05:00:48.677 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T05:00:48.677 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:00:48.739 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2024-04-12T05:00:48.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph orch host add smithi192 2024-04-12T05:00:48.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:47.564761+0000 mgr.smithi113.ctsazg (mgr.14154) 6 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:48.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:47.567251+0000 mon.smithi113 (mon.0) 97 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:49.836241+0000 mgr.smithi113.ctsazg (mgr.14154) 7 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:00:51.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.654+0000 7f8055d2c700 1 -- 172.21.15.113:0/1178839258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80500611f0 msgr2=0x7f80500fa190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.654+0000 7f8055d2c700 1 --2- 172.21.15.113:0/1178839258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80500611f0 0x7f80500fa190 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8044004770 tx=0x7f8044009cb0).stop 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.654+0000 7f8055d2c700 1 -- 172.21.15.113:0/1178839258 shutdown_connections 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.654+0000 7f8055d2c700 1 --2- 172.21.15.113:0/1178839258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80500611f0 0x7f80500fa190 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.654+0000 7f8055d2c700 1 -- 172.21.15.113:0/1178839258 >> 172.21.15.113:0/1178839258 conn(0x7f80500f1c10 msgr2=0x7f80500f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.654+0000 7f8055d2c700 1 -- 172.21.15.113:0/1178839258 shutdown_connections 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.654+0000 7f8055d2c700 1 -- 172.21.15.113:0/1178839258 wait complete. 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.655+0000 7f8055d2c700 1 Processor -- start 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.655+0000 7f8055d2c700 1 -- start start 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.655+0000 7f8055d2c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80500611f0 0x7f80500f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.655+0000 7f8055d2c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80500f6090 con 0x7f80500611f0 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.655+0000 7f804f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80500611f0 0x7f80500f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:51.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.655+0000 7f804f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80500611f0 0x7f80500f7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57476/0 (socket says 172.21.15.113:57476) 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.655+0000 7f804f7fe700 1 -- 172.21.15.113:0/1720543884 learned_addr learned my addr 172.21.15.113:0/1720543884 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.655+0000 7f804f7fe700 1 -- 172.21.15.113:0/1720543884 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8044005040 con 0x7f80500611f0 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.655+0000 7f804f7fe700 1 --2- 172.21.15.113:0/1720543884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80500611f0 0x7f80500f7c10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8044004740 tx=0x7f8044009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.655+0000 7f804cff9700 1 -- 172.21.15.113:0/1720543884 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8044018070 con 0x7f80500611f0 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.656+0000 7f804cff9700 1 -- 172.21.15.113:0/1720543884 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f8044013e30 con 0x7f80500611f0 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.656+0000 7f8055d2c700 1 -- 172.21.15.113:0/1720543884 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80500f6230 con 0x7f80500611f0 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.656+0000 7f804cff9700 1 -- 172.21.15.113:0/1720543884 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f804401d600 con 0x7f80500611f0 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.656+0000 7f8055d2c700 1 -- 172.21.15.113:0/1720543884 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80500f66c0 con 0x7f80500611f0 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.656+0000 7f804cff9700 1 -- 172.21.15.113:0/1720543884 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7f80440125d0 con 0x7f80500611f0 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.656+0000 7f8055d2c700 1 -- 172.21.15.113:0/1720543884 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805005a310 con 0x7f80500611f0 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.656+0000 7f804cff9700 1 --2- 172.21.15.113:0/1720543884 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f8038032700 0x7f8038034bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.657+0000 7f804cff9700 1 -- 172.21.15.113:0/1720543884 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8044044680 con 0x7f80500611f0 2024-04-12T05:00:51.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.657+0000 7f804effd700 1 --2- 172.21.15.113:0/1720543884 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f8038032700 0x7f8038034bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:51.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.657+0000 7f804effd700 1 --2- 172.21.15.113:0/1720543884 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f8038032700 0x7f8038034bc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f8040009780 tx=0x7f8040006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:51.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.661+0000 7f804cff9700 1 -- 172.21.15.113:0/1720543884 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f80440246f0 con 0x7f80500611f0 2024-04-12T05:00:51.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:51.830+0000 7f8055d2c700 1 -- 172.21.15.113:0/1720543884 --> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}) v1 -- 0x7f80500f6f10 con 0x7f8038032700 2024-04-12T05:00:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:52.469+0000 7f804cff9700 1 -- 172.21.15.113:0/1720543884 <== mgr.14154 v2:172.21.15.113:6800/1349703403 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f80500f6f10 con 0x7f8038032700 2024-04-12T05:00:52.472 INFO:teuthology.orchestra.run.smithi113.stdout:Added host 'smithi192' with addr '172.21.15.192' 2024-04-12T05:00:52.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:52.471+0000 7f8055d2c700 1 -- 172.21.15.113:0/1720543884 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f8038032700 msgr2=0x7f8038034bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:52.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:52.471+0000 7f8055d2c700 1 --2- 172.21.15.113:0/1720543884 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f8038032700 0x7f8038034bc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f8040009780 tx=0x7f8040006cc0).stop 2024-04-12T05:00:52.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:52.471+0000 7f8055d2c700 1 -- 172.21.15.113:0/1720543884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80500611f0 msgr2=0x7f80500f7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:52.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:52.471+0000 7f8055d2c700 1 --2- 172.21.15.113:0/1720543884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80500611f0 0x7f80500f7c10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8044004740 tx=0x7f8044009cb0).stop 2024-04-12T05:00:52.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:52.471+0000 7f8055d2c700 1 -- 172.21.15.113:0/1720543884 shutdown_connections 2024-04-12T05:00:52.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:52.471+0000 7f8055d2c700 1 --2- 172.21.15.113:0/1720543884 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f8038032700 0x7f8038034bc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:52.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:52.471+0000 7f8055d2c700 1 --2- 172.21.15.113:0/1720543884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80500611f0 0x7f80500f7c10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:52.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:52.471+0000 7f8055d2c700 1 -- 172.21.15.113:0/1720543884 >> 172.21.15.113:0/1720543884 conn(0x7f80500f1c10 msgr2=0x7f80500f25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:52.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:52.471+0000 7f8055d2c700 1 -- 172.21.15.113:0/1720543884 shutdown_connections 2024-04-12T05:00:52.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:52.471+0000 7f8055d2c700 1 -- 172.21.15.113:0/1720543884 wait complete. 2024-04-12T05:00:53.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:51.832313+0000 mgr.smithi113.ctsazg (mgr.14154) 8 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:00:53.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:51.839780+0000 mgr.smithi113.ctsazg (mgr.14154) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:00:53.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:52.469964+0000 mon.smithi113 (mon.0) 98 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:53.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:52.470265+0000 mgr.smithi113.ctsazg (mgr.14154) 10 : cephadm [INF] Added host smithi192 2024-04-12T05:00:54.050 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph orch host ls --format=json 2024-04-12T05:00:55.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:53.840050+0000 mgr.smithi113.ctsazg (mgr.14154) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:00:56.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.655+0000 7f336de3f700 1 -- 172.21.15.113:0/1477986216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33680f64b0 msgr2=0x7f33680f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.655+0000 7f336de3f700 1 --2- 172.21.15.113:0/1477986216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33680f64b0 0x7f33680f68d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f3358004770 tx=0x7f3358009cb0).stop 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.655+0000 7f336de3f700 1 -- 172.21.15.113:0/1477986216 shutdown_connections 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.655+0000 7f336de3f700 1 --2- 172.21.15.113:0/1477986216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33680f64b0 0x7f33680f68d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.655+0000 7f336de3f700 1 -- 172.21.15.113:0/1477986216 >> 172.21.15.113:0/1477986216 conn(0x7f33680f1c10 msgr2=0x7f33680f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.655+0000 7f336de3f700 1 -- 172.21.15.113:0/1477986216 shutdown_connections 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.655+0000 7f336de3f700 1 -- 172.21.15.113:0/1477986216 wait complete. 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.656+0000 7f336de3f700 1 Processor -- start 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.656+0000 7f336de3f700 1 -- start start 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.656+0000 7f336de3f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33680f64b0 0x7f3368107820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.656+0000 7f336de3f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3368107d10 con 0x7f33680f64b0 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.656+0000 7f33677fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33680f64b0 0x7f3368107820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.656+0000 7f33677fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33680f64b0 0x7f3368107820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57502/0 (socket says 172.21.15.113:57502) 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.656+0000 7f33677fe700 1 -- 172.21.15.113:0/2222607352 learned_addr learned my addr 172.21.15.113:0/2222607352 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:00:56.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.657+0000 7f33677fe700 1 -- 172.21.15.113:0/2222607352 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3358005040 con 0x7f33680f64b0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.657+0000 7f33677fe700 1 --2- 172.21.15.113:0/2222607352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33680f64b0 0x7f3368107820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f3358009300 tx=0x7f3358009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.657+0000 7f3364ff9700 1 -- 172.21.15.113:0/2222607352 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3358018070 con 0x7f33680f64b0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.657+0000 7f3364ff9700 1 -- 172.21.15.113:0/2222607352 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f3358009890 con 0x7f33680f64b0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.657+0000 7f336de3f700 1 -- 172.21.15.113:0/2222607352 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3368107f10 con 0x7f33680f64b0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.657+0000 7f3364ff9700 1 -- 172.21.15.113:0/2222607352 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f335801c400 con 0x7f33680f64b0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.657+0000 7f336de3f700 1 -- 172.21.15.113:0/2222607352 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33681083a0 con 0x7f33680f64b0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.658+0000 7f3364ff9700 1 -- 172.21.15.113:0/2222607352 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7f335801c6c0 con 0x7f33680f64b0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.658+0000 7f3364ff9700 1 --2- 172.21.15.113:0/2222607352 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3350032700 0x7f3350034bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.658+0000 7f336de3f700 1 -- 172.21.15.113:0/2222607352 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3368046ce0 con 0x7f33680f64b0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.658+0000 7f3364ff9700 1 -- 172.21.15.113:0/2222607352 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3358044890 con 0x7f33680f64b0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.658+0000 7f3366ffd700 1 --2- 172.21.15.113:0/2222607352 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3350032700 0x7f3350034bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.658+0000 7f3366ffd700 1 --2- 172.21.15.113:0/2222607352 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3350032700 0x7f3350034bc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f335c009780 tx=0x7f335c006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:00:56.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.663+0000 7f3364ff9700 1 -- 172.21.15.113:0/2222607352 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f335803e030 con 0x7f33680f64b0 2024-04-12T05:00:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.833+0000 7f336de3f700 1 -- 172.21.15.113:0/2222607352 --> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3368109450 con 0x7f3350032700 2024-04-12T05:00:56.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.834+0000 7f3364ff9700 1 -- 172.21.15.113:0/2222607352 <== mgr.14154 v2:172.21.15.113:6800/1349703403 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f3368109450 con 0x7f3350032700 2024-04-12T05:00:56.836 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:00:56.836 INFO:teuthology.orchestra.run.smithi113.stdout:[{"addr": "172.21.15.113", "hostname": "smithi113", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2024-04-12T05:00:56.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.836+0000 7f336de3f700 1 -- 172.21.15.113:0/2222607352 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3350032700 msgr2=0x7f3350034bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:56.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.836+0000 7f336de3f700 1 --2- 172.21.15.113:0/2222607352 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3350032700 0x7f3350034bc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f335c009780 tx=0x7f335c006cc0).stop 2024-04-12T05:00:56.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.836+0000 7f336de3f700 1 -- 172.21.15.113:0/2222607352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33680f64b0 msgr2=0x7f3368107820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:00:56.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.836+0000 7f336de3f700 1 --2- 172.21.15.113:0/2222607352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33680f64b0 0x7f3368107820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f3358009300 tx=0x7f3358009cb0).stop 2024-04-12T05:00:56.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.836+0000 7f336de3f700 1 -- 172.21.15.113:0/2222607352 shutdown_connections 2024-04-12T05:00:56.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.836+0000 7f336de3f700 1 --2- 172.21.15.113:0/2222607352 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3350032700 0x7f3350034bc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:56.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.836+0000 7f336de3f700 1 --2- 172.21.15.113:0/2222607352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33680f64b0 0x7f3368107820 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:00:56.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.836+0000 7f336de3f700 1 -- 172.21.15.113:0/2222607352 >> 172.21.15.113:0/2222607352 conn(0x7f33680f1c10 msgr2=0x7f3368060dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:00:56.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.836+0000 7f336de3f700 1 -- 172.21.15.113:0/2222607352 shutdown_connections 2024-04-12T05:00:56.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:00:56.836+0000 7f336de3f700 1 -- 172.21.15.113:0/2222607352 wait complete. 2024-04-12T05:00:57.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:55.840261+0000 mgr.smithi113.ctsazg (mgr.14154) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:00:57.952 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-12T05:00:57.952 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd crush tunables default 2024-04-12T05:00:58.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:56.834609+0000 mgr.smithi113.ctsazg (mgr.14154) 13 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T05:00:58.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:57.560773+0000 mon.smithi113 (mon.0) 99 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:00:58.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:57.562410+0000 mon.smithi113 (mon.0) 100 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:00:58.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:57.563992+0000 mon.smithi113 (mon.0) 101 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-12T05:00:58.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:57.564484+0000 mon.smithi113 (mon.0) 102 : audit [DBG] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:00:59.853 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:00:57.564966+0000 mgr.smithi113.ctsazg (mgr.14154) 14 : cephadm [INF] Deploying daemon crash.smithi113 on smithi113 2024-04-12T05:00:59.853 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:00:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:57.840529+0000 mgr.smithi113.ctsazg (mgr.14154) 15 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:00.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.688+0000 7fab26d67700 1 -- 172.21.15.113:0/1313745170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab200f75c0 msgr2=0x7fab200f79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:00.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.688+0000 7fab26d67700 1 --2- 172.21.15.113:0/1313745170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab200f75c0 0x7fab200f79e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fab14004770 tx=0x7fab14009cb0).stop 2024-04-12T05:01:00.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.688+0000 7fab26d67700 1 -- 172.21.15.113:0/1313745170 shutdown_connections 2024-04-12T05:01:00.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.688+0000 7fab26d67700 1 --2- 172.21.15.113:0/1313745170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab200f75c0 0x7fab200f79e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:00.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.688+0000 7fab26d67700 1 -- 172.21.15.113:0/1313745170 >> 172.21.15.113:0/1313745170 conn(0x7fab200f2d40 msgr2=0x7fab200f5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:00.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.688+0000 7fab26d67700 1 -- 172.21.15.113:0/1313745170 shutdown_connections 2024-04-12T05:01:00.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.688+0000 7fab26d67700 1 -- 172.21.15.113:0/1313745170 wait complete. 2024-04-12T05:01:00.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab26d67700 1 Processor -- start 2024-04-12T05:01:00.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab26d67700 1 -- start start 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab26d67700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab200f75c0 0x7fab20107840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab26d67700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab20107d30 con 0x7fab200f75c0 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab24b03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab200f75c0 0x7fab20107840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab24b03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab200f75c0 0x7fab20107840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33864/0 (socket says 172.21.15.113:33864) 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab24b03700 1 -- 172.21.15.113:0/1941251093 learned_addr learned my addr 172.21.15.113:0/1941251093 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab24b03700 1 -- 172.21.15.113:0/1941251093 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab14005040 con 0x7fab200f75c0 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab24b03700 1 --2- 172.21.15.113:0/1941251093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab200f75c0 0x7fab20107840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fab14009300 tx=0x7fab14009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab1dffb700 1 -- 172.21.15.113:0/1941251093 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab14018070 con 0x7fab200f75c0 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab1dffb700 1 -- 172.21.15.113:0/1941251093 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fab14009890 con 0x7fab200f75c0 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab1dffb700 1 -- 172.21.15.113:0/1941251093 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab1401c400 con 0x7fab200f75c0 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.689+0000 7fab26d67700 1 -- 172.21.15.113:0/1941251093 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab20107f30 con 0x7fab200f75c0 2024-04-12T05:01:00.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.690+0000 7fab26d67700 1 -- 172.21.15.113:0/1941251093 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab201083c0 con 0x7fab200f75c0 2024-04-12T05:01:00.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.690+0000 7fab1dffb700 1 -- 172.21.15.113:0/1941251093 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7fab1401d420 con 0x7fab200f75c0 2024-04-12T05:01:00.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.690+0000 7fab1dffb700 1 --2- 172.21.15.113:0/1941251093 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fab10032750 0x7fab10034c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:00.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.691+0000 7fab1dffb700 1 -- 172.21.15.113:0/1941251093 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fab140445a0 con 0x7fab200f75c0 2024-04-12T05:01:00.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.691+0000 7fab26d67700 1 -- 172.21.15.113:0/1941251093 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab20046ce0 con 0x7fab200f75c0 2024-04-12T05:01:00.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.691+0000 7fab1ffff700 1 --2- 172.21.15.113:0/1941251093 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fab10032750 0x7fab10034c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:00.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.691+0000 7fab1ffff700 1 --2- 172.21.15.113:0/1941251093 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fab10032750 0x7fab10034c10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fab0c009780 tx=0x7fab0c006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:00.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.695+0000 7fab1dffb700 1 -- 172.21.15.113:0/1941251093 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fab1401b760 con 0x7fab200f75c0 2024-04-12T05:01:00.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:00.864+0000 7fab26d67700 1 -- 172.21.15.113:0/1941251093 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fab201090e0 con 0x7fab200f75c0 2024-04-12T05:01:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:00:59.840921+0000 mgr.smithi113.ctsazg (mgr.14154) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:00:59.968775+0000 mon.smithi113 (mon.0) 103 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:00.865338+0000 mon.smithi113 (mon.0) 104 : audit [INF] from='client.? 172.21.15.113:0/1941251093' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-12T05:01:01.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:01.571+0000 7fab1dffb700 1 -- 172.21.15.113:0/1941251093 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fab1401b370 con 0x7fab200f75c0 2024-04-12T05:01:01.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:01.573+0000 7fab26d67700 1 -- 172.21.15.113:0/1941251093 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fab10032750 msgr2=0x7fab10034c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:01.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:01.573+0000 7fab26d67700 1 --2- 172.21.15.113:0/1941251093 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fab10032750 0x7fab10034c10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fab0c009780 tx=0x7fab0c006cc0).stop 2024-04-12T05:01:01.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:01.573+0000 7fab26d67700 1 -- 172.21.15.113:0/1941251093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab200f75c0 msgr2=0x7fab20107840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:01.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:01.573+0000 7fab26d67700 1 --2- 172.21.15.113:0/1941251093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab200f75c0 0x7fab20107840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fab14009300 tx=0x7fab14009cb0).stop 2024-04-12T05:01:01.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:01.573+0000 7fab26d67700 1 -- 172.21.15.113:0/1941251093 shutdown_connections 2024-04-12T05:01:01.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:01.573+0000 7fab26d67700 1 --2- 172.21.15.113:0/1941251093 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fab10032750 0x7fab10034c10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:01.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:01.573+0000 7fab26d67700 1 --2- 172.21.15.113:0/1941251093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab200f75c0 0x7fab20107840 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:01.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:01.573+0000 7fab26d67700 1 -- 172.21.15.113:0/1941251093 >> 172.21.15.113:0/1941251093 conn(0x7fab200f2d40 msgr2=0x7fab200f3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:01.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:01.573+0000 7fab26d67700 1 -- 172.21.15.113:0/1941251093 shutdown_connections 2024-04-12T05:01:01.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:01.573+0000 7fab26d67700 1 -- 172.21.15.113:0/1941251093 wait complete. 2024-04-12T05:01:01.576 INFO:teuthology.orchestra.run.smithi113.stderr:adjusted tunables profile to default 2024-04-12T05:01:02.323 INFO:tasks.cephadm:Adding mon.smithi113 on smithi113 2024-04-12T05:01:02.323 INFO:tasks.cephadm:Adding mon.smithi192 on smithi192 2024-04-12T05:01:02.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph orch apply mon '2;smithi113:172.21.15.113=smithi113;smithi192:172.21.15.192=smithi192' 2024-04-12T05:01:02.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:01.238641+0000 mon.smithi113 (mon.0) 105 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:02.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:01.293065+0000 mon.smithi113 (mon.0) 106 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:02.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:01.297664+0000 mon.smithi113 (mon.0) 107 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:02.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:01.299516+0000 mon.smithi113 (mon.0) 108 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T05:01:02.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:01.299853+0000 mgr.smithi113.ctsazg (mgr.14154) 17 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T05:01:02.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:01.304958+0000 mon.smithi113 (mon.0) 109 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:02.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:01.311831+0000 mgr.smithi113.ctsazg (mgr.14154) 18 : cephadm [INF] Deploying daemon grafana.smithi113 on smithi113 2024-04-12T05:01:02.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:01.572176+0000 mon.smithi113 (mon.0) 110 : audit [INF] from='client.? 172.21.15.113:0/1941251093' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-12T05:01:02.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:01.572248+0000 mon.smithi113 (mon.0) 111 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-12T05:01:03.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:01.841205+0000 mgr.smithi113.ctsazg (mgr.14154) 19 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:04.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.119+0000 7f57a5f41700 1 -- 172.21.15.192:0/2807691034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a00f75c0 msgr2=0x7f57a00f79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:04.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.119+0000 7f57a5f41700 1 --2- 172.21.15.192:0/2807691034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a00f75c0 0x7f57a00f79e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f5794004770 tx=0x7f5794009cb0).stop 2024-04-12T05:01:04.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.119+0000 7f57a5f41700 1 -- 172.21.15.192:0/2807691034 shutdown_connections 2024-04-12T05:01:04.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.119+0000 7f57a5f41700 1 --2- 172.21.15.192:0/2807691034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a00f75c0 0x7f57a00f79e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:04.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.119+0000 7f57a5f41700 1 -- 172.21.15.192:0/2807691034 >> 172.21.15.192:0/2807691034 conn(0x7f57a00f2d40 msgr2=0x7f57a00f5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:04.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.119+0000 7f57a5f41700 1 -- 172.21.15.192:0/2807691034 shutdown_connections 2024-04-12T05:01:04.121 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.119+0000 7f57a5f41700 1 -- 172.21.15.192:0/2807691034 wait complete. 2024-04-12T05:01:04.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.120+0000 7f57a5f41700 1 Processor -- start 2024-04-12T05:01:04.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.120+0000 7f57a5f41700 1 -- start start 2024-04-12T05:01:04.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.120+0000 7f57a5f41700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a00f75c0 0x7f57a0109a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:04.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.120+0000 7f57a5f41700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a0109f50 con 0x7f57a00f75c0 2024-04-12T05:01:04.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.120+0000 7f579f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a00f75c0 0x7f57a0109a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:04.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.121+0000 7f579f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a00f75c0 0x7f57a0109a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.192:50124/0 (socket says 172.21.15.192:50124) 2024-04-12T05:01:04.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.121+0000 7f579f7fe700 1 -- 172.21.15.192:0/1747241606 learned_addr learned my addr 172.21.15.192:0/1747241606 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:01:04.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.121+0000 7f579f7fe700 1 -- 172.21.15.192:0/1747241606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5794005040 con 0x7f57a00f75c0 2024-04-12T05:01:04.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.121+0000 7f579f7fe700 1 --2- 172.21.15.192:0/1747241606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a00f75c0 0x7f57a0109a60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f57940099b0 tx=0x7f5794009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:04.124 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.121+0000 7f579cff9700 1 -- 172.21.15.192:0/1747241606 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5794018070 con 0x7f57a00f75c0 2024-04-12T05:01:04.125 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.121+0000 7f579cff9700 1 -- 172.21.15.192:0/1747241606 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f5794013730 con 0x7f57a00f75c0 2024-04-12T05:01:04.125 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.121+0000 7f579cff9700 1 -- 172.21.15.192:0/1747241606 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f579401d460 con 0x7f57a00f75c0 2024-04-12T05:01:04.125 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.121+0000 7f57a5f41700 1 -- 172.21.15.192:0/1747241606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a010a150 con 0x7f57a00f75c0 2024-04-12T05:01:04.125 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.121+0000 7f57a5f41700 1 -- 172.21.15.192:0/1747241606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a010a5e0 con 0x7f57a00f75c0 2024-04-12T05:01:04.125 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.122+0000 7f579cff9700 1 -- 172.21.15.192:0/1747241606 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7f57940125d0 con 0x7f57a00f75c0 2024-04-12T05:01:04.125 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.122+0000 7f579cff9700 1 --2- 172.21.15.192:0/1747241606 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f57880326d0 0x7f5788034b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:04.125 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.122+0000 7f579cff9700 1 -- 172.21.15.192:0/1747241606 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f57940440f0 con 0x7f57a00f75c0 2024-04-12T05:01:04.125 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.122+0000 7f57a5f41700 1 -- 172.21.15.192:0/1747241606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a0046ce0 con 0x7f57a00f75c0 2024-04-12T05:01:04.125 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.122+0000 7f579effd700 1 --2- 172.21.15.192:0/1747241606 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f57880326d0 0x7f5788034b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:04.126 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.123+0000 7f579effd700 1 --2- 172.21.15.192:0/1747241606 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f57880326d0 0x7f5788034b90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f5790009780 tx=0x7f5790006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:04.131 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.130+0000 7f579cff9700 1 -- 172.21.15.192:0/1747241606 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f57940128d0 con 0x7f57a00f75c0 2024-04-12T05:01:04.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.300+0000 7f57a5f41700 1 -- 172.21.15.192:0/1747241606 --> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi113:172.21.15.113=smithi113;smithi192:172.21.15.192=smithi192", "target": ["mon-mgr", ""]}) v1 -- 0x7f57a01035a0 con 0x7f57880326d0 2024-04-12T05:01:04.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.307+0000 7f579cff9700 1 -- 172.21.15.192:0/1747241606 <== mgr.14154 v2:172.21.15.113:6800/1349703403 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f57a01035a0 con 0x7f57880326d0 2024-04-12T05:01:04.309 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2024-04-12T05:01:04.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.308+0000 7f57a5f41700 1 -- 172.21.15.192:0/1747241606 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f57880326d0 msgr2=0x7f5788034b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:04.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.308+0000 7f57a5f41700 1 --2- 172.21.15.192:0/1747241606 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f57880326d0 0x7f5788034b90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f5790009780 tx=0x7f5790006cc0).stop 2024-04-12T05:01:04.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.308+0000 7f57a5f41700 1 -- 172.21.15.192:0/1747241606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a00f75c0 msgr2=0x7f57a0109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:04.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.308+0000 7f57a5f41700 1 --2- 172.21.15.192:0/1747241606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a00f75c0 0x7f57a0109a60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f57940099b0 tx=0x7f5794009cb0).stop 2024-04-12T05:01:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.308+0000 7f57a5f41700 1 -- 172.21.15.192:0/1747241606 shutdown_connections 2024-04-12T05:01:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.308+0000 7f57a5f41700 1 --2- 172.21.15.192:0/1747241606 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f57880326d0 0x7f5788034b90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.308+0000 7f57a5f41700 1 --2- 172.21.15.192:0/1747241606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a00f75c0 0x7f57a0109a60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.308+0000 7f57a5f41700 1 -- 172.21.15.192:0/1747241606 >> 172.21.15.192:0/1747241606 conn(0x7f57a00f2d40 msgr2=0x7f57a00f5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.309+0000 7f57a5f41700 1 -- 172.21.15.192:0/1747241606 shutdown_connections 2024-04-12T05:01:04.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:04.309+0000 7f57a5f41700 1 -- 172.21.15.192:0/1747241606 wait complete. 2024-04-12T05:01:04.943 DEBUG:teuthology.orchestra.run.smithi192:mon.smithi192> sudo journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi192.service 2024-04-12T05:01:04.954 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:01:04.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:01:04.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:-- Logs begin at Fri 2024-04-12 04:48:43 UTC. -- 2024-04-12T05:01:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:03.841640+0000 mgr.smithi113.ctsazg (mgr.14154) 20 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:04.302032+0000 mgr.smithi113.ctsazg (mgr.14154) 21 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi113:172.21.15.113=smithi113;smithi192:172.21.15.192=smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:01:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:04.304633+0000 mgr.smithi113.ctsazg (mgr.14154) 22 : cephadm [INF] Saving service mon spec with placement smithi113:172.21.15.113=smithi113;smithi192:172.21.15.192=smithi192;count:2 2024-04-12T05:01:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:04.307106+0000 mon.smithi113 (mon.0) 112 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:04.972426+0000 mon.smithi113 (mon.0) 113 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:06.743 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.740+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/2008023139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa99c0f75c0 msgr2=0x7fa99c0f79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:06.743 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.740+0000 7fa9a0cd0700 1 --2- 172.21.15.192:0/2008023139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa99c0f75c0 0x7fa99c0f79e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fa990004770 tx=0x7fa990009cb0).stop 2024-04-12T05:01:06.743 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.740+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/2008023139 shutdown_connections 2024-04-12T05:01:06.743 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.740+0000 7fa9a0cd0700 1 --2- 172.21.15.192:0/2008023139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa99c0f75c0 0x7fa99c0f79e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:06.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.740+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/2008023139 >> 172.21.15.192:0/2008023139 conn(0x7fa99c0f2d40 msgr2=0x7fa99c0f5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:06.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.740+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/2008023139 shutdown_connections 2024-04-12T05:01:06.744 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.741+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/2008023139 wait complete. 2024-04-12T05:01:06.746 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.741+0000 7fa9a0cd0700 1 Processor -- start 2024-04-12T05:01:06.746 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa9a0cd0700 1 -- start start 2024-04-12T05:01:06.746 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa9a0cd0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa99c0f75c0 0x7fa99c107840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:06.746 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa9a0cd0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c107d30 con 0x7fa99c0f75c0 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa99a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa99c0f75c0 0x7fa99c107840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa99a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa99c0f75c0 0x7fa99c107840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.192:50130/0 (socket says 172.21.15.192:50130) 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa99a59c700 1 -- 172.21.15.192:0/489924452 learned_addr learned my addr 172.21.15.192:0/489924452 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa99a59c700 1 -- 172.21.15.192:0/489924452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa990005040 con 0x7fa99c0f75c0 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa99a59c700 1 --2- 172.21.15.192:0/489924452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa99c0f75c0 0x7fa99c107840 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fa990004740 tx=0x7fa990009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa98b7fe700 1 -- 172.21.15.192:0/489924452 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa990018070 con 0x7fa99c0f75c0 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/489924452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c107f30 con 0x7fa99c0f75c0 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa98b7fe700 1 -- 172.21.15.192:0/489924452 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fa9900097f0 con 0x7fa99c0f75c0 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.742+0000 7fa98b7fe700 1 -- 172.21.15.192:0/489924452 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa990012400 con 0x7fa99c0f75c0 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.743+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/489924452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c1083c0 con 0x7fa99c0f75c0 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.743+0000 7fa98b7fe700 1 -- 172.21.15.192:0/489924452 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7fa990012700 con 0x7fa99c0f75c0 2024-04-12T05:01:06.747 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.743+0000 7fa98b7fe700 1 --2- 172.21.15.192:0/489924452 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fa9840326d0 0x7fa984034b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:06.748 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.743+0000 7fa98b7fe700 1 -- 172.21.15.192:0/489924452 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa990043ef0 con 0x7fa99c0f75c0 2024-04-12T05:01:06.748 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.744+0000 7fa999d9b700 1 --2- 172.21.15.192:0/489924452 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fa9840326d0 0x7fa984034b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:06.748 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.744+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/489924452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99c05a310 con 0x7fa99c0f75c0 2024-04-12T05:01:06.748 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.744+0000 7fa999d9b700 1 --2- 172.21.15.192:0/489924452 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fa9840326d0 0x7fa984034b90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa98c009780 tx=0x7fa98c006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:06.750 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.749+0000 7fa98b7fe700 1 -- 172.21.15.192:0/489924452 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa990012a00 con 0x7fa99c0f75c0 2024-04-12T05:01:06.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.974+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/489924452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa99c046ce0 con 0x7fa99c0f75c0 2024-04-12T05:01:06.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.975+0000 7fa98b7fe700 1 -- 172.21.15.192:0/489924452 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fa990022210 con 0x7fa99c0f75c0 2024-04-12T05:01:06.977 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T05:01:06.977 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","modified":"2024-04-12T04:59:27.208775Z","created":"2024-04-12T04:59:27.208775Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi113","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:01:06.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.976+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/489924452 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fa9840326d0 msgr2=0x7fa984034b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:06.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.976+0000 7fa9a0cd0700 1 --2- 172.21.15.192:0/489924452 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fa9840326d0 0x7fa984034b90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa98c009780 tx=0x7fa98c006cc0).stop 2024-04-12T05:01:06.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.976+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/489924452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa99c0f75c0 msgr2=0x7fa99c107840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:06.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.976+0000 7fa9a0cd0700 1 --2- 172.21.15.192:0/489924452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa99c0f75c0 0x7fa99c107840 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fa990004740 tx=0x7fa990009cb0).stop 2024-04-12T05:01:06.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.976+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/489924452 shutdown_connections 2024-04-12T05:01:06.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.976+0000 7fa9a0cd0700 1 --2- 172.21.15.192:0/489924452 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7fa9840326d0 0x7fa984034b90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:06.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.976+0000 7fa9a0cd0700 1 --2- 172.21.15.192:0/489924452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa99c0f75c0 0x7fa99c107840 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:06.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.976+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/489924452 >> 172.21.15.192:0/489924452 conn(0x7fa99c0f2d40 msgr2=0x7fa99c0f3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:06.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.976+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/489924452 shutdown_connections 2024-04-12T05:01:06.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:06.976+0000 7fa9a0cd0700 1 -- 172.21.15.192:0/489924452 wait complete. 2024-04-12T05:01:06.979 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-04-12T05:01:07.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:05.841877+0000 mgr.smithi113.ctsazg (mgr.14154) 23 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:07.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:06.975288+0000 mon.smithi113 (mon.0) 114 : audit [DBG] from='client.? 172.21.15.192:0/489924452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:01:08.607 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:01:08.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:01:09.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:07.842103+0000 mgr.smithi113.ctsazg (mgr.14154) 24 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:10.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.243+0000 7efd758cc700 1 -- 172.21.15.192:0/453034014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f5880 msgr2=0x7efd700f5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:10.245 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.243+0000 7efd758cc700 1 --2- 172.21.15.192:0/453034014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f5880 0x7efd700f5ca0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7efd64004770 tx=0x7efd64009cb0).stop 2024-04-12T05:01:10.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.243+0000 7efd758cc700 1 -- 172.21.15.192:0/453034014 shutdown_connections 2024-04-12T05:01:10.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.243+0000 7efd758cc700 1 --2- 172.21.15.192:0/453034014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f5880 0x7efd700f5ca0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:10.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.243+0000 7efd758cc700 1 -- 172.21.15.192:0/453034014 >> 172.21.15.192:0/453034014 conn(0x7efd700f13a0 msgr2=0x7efd700f37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:10.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.243+0000 7efd758cc700 1 -- 172.21.15.192:0/453034014 shutdown_connections 2024-04-12T05:01:10.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.243+0000 7efd758cc700 1 -- 172.21.15.192:0/453034014 wait complete. 2024-04-12T05:01:10.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.244+0000 7efd758cc700 1 Processor -- start 2024-04-12T05:01:10.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.244+0000 7efd758cc700 1 -- start start 2024-04-12T05:01:10.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.244+0000 7efd758cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f5880 0x7efd70107810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:10.246 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.244+0000 7efd758cc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd70107d00 con 0x7efd700f5880 2024-04-12T05:01:10.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.244+0000 7efd6effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f5880 0x7efd70107810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:10.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.244+0000 7efd6effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f5880 0x7efd70107810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.192:50156/0 (socket says 172.21.15.192:50156) 2024-04-12T05:01:10.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.244+0000 7efd6effd700 1 -- 172.21.15.192:0/2868506832 learned_addr learned my addr 172.21.15.192:0/2868506832 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:01:10.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.245+0000 7efd6effd700 1 -- 172.21.15.192:0/2868506832 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd64005040 con 0x7efd700f5880 2024-04-12T05:01:10.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.245+0000 7efd6effd700 1 --2- 172.21.15.192:0/2868506832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f5880 0x7efd70107810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7efd64009300 tx=0x7efd64009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:10.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.245+0000 7efd57fff700 1 -- 172.21.15.192:0/2868506832 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd64018070 con 0x7efd700f5880 2024-04-12T05:01:10.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.245+0000 7efd57fff700 1 -- 172.21.15.192:0/2868506832 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7efd64009890 con 0x7efd700f5880 2024-04-12T05:01:10.248 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.245+0000 7efd758cc700 1 -- 172.21.15.192:0/2868506832 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd70107f00 con 0x7efd700f5880 2024-04-12T05:01:10.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.245+0000 7efd57fff700 1 -- 172.21.15.192:0/2868506832 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd6401c400 con 0x7efd700f5880 2024-04-12T05:01:10.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.245+0000 7efd758cc700 1 -- 172.21.15.192:0/2868506832 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd70108390 con 0x7efd700f5880 2024-04-12T05:01:10.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.246+0000 7efd57fff700 1 -- 172.21.15.192:0/2868506832 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7efd6401c6c0 con 0x7efd700f5880 2024-04-12T05:01:10.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.246+0000 7efd57fff700 1 --2- 172.21.15.192:0/2868506832 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efd58032700 0x7efd58034bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:10.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.246+0000 7efd57fff700 1 -- 172.21.15.192:0/2868506832 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efd640446a0 con 0x7efd700f5880 2024-04-12T05:01:10.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.246+0000 7efd758cc700 1 -- 172.21.15.192:0/2868506832 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd70046ce0 con 0x7efd700f5880 2024-04-12T05:01:10.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.246+0000 7efd6e7fc700 1 --2- 172.21.15.192:0/2868506832 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efd58032700 0x7efd58034bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:10.249 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.246+0000 7efd6e7fc700 1 --2- 172.21.15.192:0/2868506832 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efd58032700 0x7efd58034bc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7efd60009780 tx=0x7efd60006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:10.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.251+0000 7efd57fff700 1 -- 172.21.15.192:0/2868506832 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7efd64027440 con 0x7efd700f5880 2024-04-12T05:01:10.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.475+0000 7efd758cc700 1 -- 172.21.15.192:0/2868506832 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd701090b0 con 0x7efd700f5880 2024-04-12T05:01:10.477 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.476+0000 7efd57fff700 1 -- 172.21.15.192:0/2868506832 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7efd6401b370 con 0x7efd700f5880 2024-04-12T05:01:10.478 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T05:01:10.478 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","modified":"2024-04-12T04:59:27.208775Z","created":"2024-04-12T04:59:27.208775Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi113","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:01:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.477+0000 7efd758cc700 1 -- 172.21.15.192:0/2868506832 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efd58032700 msgr2=0x7efd58034bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.477+0000 7efd758cc700 1 --2- 172.21.15.192:0/2868506832 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efd58032700 0x7efd58034bc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7efd60009780 tx=0x7efd60006cc0).stop 2024-04-12T05:01:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.477+0000 7efd758cc700 1 -- 172.21.15.192:0/2868506832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f5880 msgr2=0x7efd70107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.477+0000 7efd758cc700 1 --2- 172.21.15.192:0/2868506832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f5880 0x7efd70107810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7efd64009300 tx=0x7efd64009cb0).stop 2024-04-12T05:01:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.477+0000 7efd758cc700 1 -- 172.21.15.192:0/2868506832 shutdown_connections 2024-04-12T05:01:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.477+0000 7efd758cc700 1 --2- 172.21.15.192:0/2868506832 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efd58032700 0x7efd58034bc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.477+0000 7efd758cc700 1 --2- 172.21.15.192:0/2868506832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f5880 0x7efd70107810 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.477+0000 7efd758cc700 1 -- 172.21.15.192:0/2868506832 >> 172.21.15.192:0/2868506832 conn(0x7efd700f13a0 msgr2=0x7efd700f1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.477+0000 7efd758cc700 1 -- 172.21.15.192:0/2868506832 shutdown_connections 2024-04-12T05:01:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:10.478+0000 7efd758cc700 1 -- 172.21.15.192:0/2868506832 wait complete. 2024-04-12T05:01:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-04-12T05:01:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:09.842364+0000 mgr.smithi113.ctsazg (mgr.14154) 25 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:10.476216+0000 mon.smithi113 (mon.0) 115 : audit [DBG] from='client.? 172.21.15.192:0/2868506832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:01:12.139 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:01:12.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:01:13.411 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:11.842766+0000 mgr.smithi113.ctsazg (mgr.14154) 26 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:13.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.987+0000 7efc5dfd0700 1 -- 172.21.15.192:0/1741278860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc580f64b0 msgr2=0x7efc580f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:13.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.987+0000 7efc5dfd0700 1 --2- 172.21.15.192:0/1741278860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc580f64b0 0x7efc580f68d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7efc4c004770 tx=0x7efc4c009cb0).stop 2024-04-12T05:01:13.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.988+0000 7efc5dfd0700 1 -- 172.21.15.192:0/1741278860 shutdown_connections 2024-04-12T05:01:13.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.988+0000 7efc5dfd0700 1 --2- 172.21.15.192:0/1741278860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc580f64b0 0x7efc580f68d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:13.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.988+0000 7efc5dfd0700 1 -- 172.21.15.192:0/1741278860 >> 172.21.15.192:0/1741278860 conn(0x7efc580f1c10 msgr2=0x7efc580f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:13.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.988+0000 7efc5dfd0700 1 -- 172.21.15.192:0/1741278860 shutdown_connections 2024-04-12T05:01:13.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.988+0000 7efc5dfd0700 1 -- 172.21.15.192:0/1741278860 wait complete. 2024-04-12T05:01:13.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.988+0000 7efc5dfd0700 1 Processor -- start 2024-04-12T05:01:13.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.988+0000 7efc5dfd0700 1 -- start start 2024-04-12T05:01:13.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.988+0000 7efc5dfd0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc580f64b0 0x7efc5810c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:13.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.988+0000 7efc5dfd0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc58108330 con 0x7efc580f64b0 2024-04-12T05:01:13.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.989+0000 7efc577fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc580f64b0 0x7efc5810c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:13.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.989+0000 7efc577fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc580f64b0 0x7efc5810c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.192:42190/0 (socket says 172.21.15.192:42190) 2024-04-12T05:01:13.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.989+0000 7efc577fe700 1 -- 172.21.15.192:0/469935937 learned_addr learned my addr 172.21.15.192:0/469935937 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:01:13.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.989+0000 7efc577fe700 1 -- 172.21.15.192:0/469935937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c005040 con 0x7efc580f64b0 2024-04-12T05:01:13.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.989+0000 7efc577fe700 1 --2- 172.21.15.192:0/469935937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc580f64b0 0x7efc5810c940 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7efc4c0099b0 tx=0x7efc4c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:13.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.989+0000 7efc54ff9700 1 -- 172.21.15.192:0/469935937 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc4c018070 con 0x7efc580f64b0 2024-04-12T05:01:13.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.989+0000 7efc54ff9700 1 -- 172.21.15.192:0/469935937 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7efc4c013730 con 0x7efc580f64b0 2024-04-12T05:01:13.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.989+0000 7efc5dfd0700 1 -- 172.21.15.192:0/469935937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc58061190 con 0x7efc580f64b0 2024-04-12T05:01:13.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.989+0000 7efc54ff9700 1 -- 172.21.15.192:0/469935937 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc4c01d460 con 0x7efc580f64b0 2024-04-12T05:01:13.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.989+0000 7efc5dfd0700 1 -- 172.21.15.192:0/469935937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5810ce30 con 0x7efc580f64b0 2024-04-12T05:01:13.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.990+0000 7efc54ff9700 1 -- 172.21.15.192:0/469935937 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7efc4c0125d0 con 0x7efc580f64b0 2024-04-12T05:01:13.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.990+0000 7efc54ff9700 1 --2- 172.21.15.192:0/469935937 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efc400326d0 0x7efc40034b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:13.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.990+0000 7efc54ff9700 1 -- 172.21.15.192:0/469935937 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efc4c0440c0 con 0x7efc580f64b0 2024-04-12T05:01:13.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.990+0000 7efc5dfd0700 1 -- 172.21.15.192:0/469935937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc58046ce0 con 0x7efc580f64b0 2024-04-12T05:01:13.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.990+0000 7efc56ffd700 1 --2- 172.21.15.192:0/469935937 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efc400326d0 0x7efc40034b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:13.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.991+0000 7efc56ffd700 1 --2- 172.21.15.192:0/469935937 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efc400326d0 0x7efc40034b90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7efc48009780 tx=0x7efc48006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:13.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:13.995+0000 7efc54ff9700 1 -- 172.21.15.192:0/469935937 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7efc4c0128d0 con 0x7efc580f64b0 2024-04-12T05:01:14.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.220+0000 7efc5dfd0700 1 -- 172.21.15.192:0/469935937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc5810d5f0 con 0x7efc580f64b0 2024-04-12T05:01:14.222 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.220+0000 7efc54ff9700 1 -- 172.21.15.192:0/469935937 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7efc4c022210 con 0x7efc580f64b0 2024-04-12T05:01:14.222 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T05:01:14.222 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","modified":"2024-04-12T04:59:27.208775Z","created":"2024-04-12T04:59:27.208775Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi113","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:01:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.222+0000 7efc5dfd0700 1 -- 172.21.15.192:0/469935937 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efc400326d0 msgr2=0x7efc40034b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.222+0000 7efc5dfd0700 1 --2- 172.21.15.192:0/469935937 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efc400326d0 0x7efc40034b90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7efc48009780 tx=0x7efc48006cc0).stop 2024-04-12T05:01:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.222+0000 7efc5dfd0700 1 -- 172.21.15.192:0/469935937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc580f64b0 msgr2=0x7efc5810c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.222+0000 7efc5dfd0700 1 --2- 172.21.15.192:0/469935937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc580f64b0 0x7efc5810c940 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7efc4c0099b0 tx=0x7efc4c009cb0).stop 2024-04-12T05:01:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.222+0000 7efc5dfd0700 1 -- 172.21.15.192:0/469935937 shutdown_connections 2024-04-12T05:01:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.222+0000 7efc5dfd0700 1 --2- 172.21.15.192:0/469935937 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7efc400326d0 0x7efc40034b90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.222+0000 7efc5dfd0700 1 --2- 172.21.15.192:0/469935937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc580f64b0 0x7efc5810c940 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.222+0000 7efc5dfd0700 1 -- 172.21.15.192:0/469935937 >> 172.21.15.192:0/469935937 conn(0x7efc580f1c10 msgr2=0x7efc580f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.222+0000 7efc5dfd0700 1 -- 172.21.15.192:0/469935937 shutdown_connections 2024-04-12T05:01:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:14.222+0000 7efc5dfd0700 1 -- 172.21.15.192:0/469935937 wait complete. 2024-04-12T05:01:14.225 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-04-12T05:01:14.487 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:14.220880+0000 mon.smithi113 (mon.0) 116 : audit [DBG] from='client.? 172.21.15.192:0/469935937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:01:15.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:13.843074+0000 mgr.smithi113.ctsazg (mgr.14154) 27 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:15.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:14.362616+0000 mon.smithi113 (mon.0) 117 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:15.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:14.364730+0000 mgr.smithi113.ctsazg (mgr.14154) 28 : cephadm [INF] Deploying daemon node-exporter.smithi113 on smithi113 2024-04-12T05:01:15.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:14.976119+0000 mon.smithi113 (mon.0) 118 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:15.886 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:01:15.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:01:17.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:15.843469+0000 mgr.smithi113.ctsazg (mgr.14154) 29 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:17.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:17.250885+0000 mon.smithi113 (mon.0) 119 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:17.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:17.252927+0000 mon.smithi113 (mon.0) 120 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-12T05:01:17.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.786+0000 7f351eff1700 1 -- 172.21.15.192:0/3894467660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35180f64b0 msgr2=0x7f35180f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:17.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.786+0000 7f351eff1700 1 --2- 172.21.15.192:0/3894467660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35180f64b0 0x7f35180f68d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f350c004770 tx=0x7f350c009cb0).stop 2024-04-12T05:01:17.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.787+0000 7f351eff1700 1 -- 172.21.15.192:0/3894467660 shutdown_connections 2024-04-12T05:01:17.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.787+0000 7f351eff1700 1 --2- 172.21.15.192:0/3894467660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35180f64b0 0x7f35180f68d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:17.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.787+0000 7f351eff1700 1 -- 172.21.15.192:0/3894467660 >> 172.21.15.192:0/3894467660 conn(0x7f35180f1c10 msgr2=0x7f35180f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:17.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.787+0000 7f351eff1700 1 -- 172.21.15.192:0/3894467660 shutdown_connections 2024-04-12T05:01:17.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.787+0000 7f351eff1700 1 -- 172.21.15.192:0/3894467660 wait complete. 2024-04-12T05:01:17.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.787+0000 7f351eff1700 1 Processor -- start 2024-04-12T05:01:17.793 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.787+0000 7f351eff1700 1 -- start start 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.787+0000 7f351eff1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35180f64b0 0x7f3518106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.787+0000 7f351eff1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3518106c10 con 0x7f35180f64b0 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.788+0000 7f351cd8d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35180f64b0 0x7f3518106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.788+0000 7f351cd8d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35180f64b0 0x7f3518106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.192:42200/0 (socket says 172.21.15.192:42200) 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.788+0000 7f351cd8d700 1 -- 172.21.15.192:0/1456761186 learned_addr learned my addr 172.21.15.192:0/1456761186 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.788+0000 7f351cd8d700 1 -- 172.21.15.192:0/1456761186 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c005040 con 0x7f35180f64b0 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.788+0000 7f351cd8d700 1 --2- 172.21.15.192:0/1456761186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35180f64b0 0x7f3518106720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f350c009300 tx=0x7f350c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.788+0000 7f3515ffb700 1 -- 172.21.15.192:0/1456761186 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f350c018070 con 0x7f35180f64b0 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.788+0000 7f351eff1700 1 -- 172.21.15.192:0/1456761186 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3518106e10 con 0x7f35180f64b0 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.788+0000 7f3515ffb700 1 -- 172.21.15.192:0/1456761186 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f350c009890 con 0x7f35180f64b0 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.788+0000 7f3515ffb700 1 -- 172.21.15.192:0/1456761186 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f350c01c400 con 0x7f35180f64b0 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.788+0000 7f351eff1700 1 -- 172.21.15.192:0/1456761186 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35181072a0 con 0x7f35180f64b0 2024-04-12T05:01:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.789+0000 7f3515ffb700 1 -- 172.21.15.192:0/1456761186 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 38689+0+0 (secure 0 0 0) 0x7f350c01d420 con 0x7f35180f64b0 2024-04-12T05:01:17.795 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.789+0000 7f351eff1700 1 -- 172.21.15.192:0/1456761186 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3518046ce0 con 0x7f35180f64b0 2024-04-12T05:01:17.795 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.789+0000 7f3515ffb700 1 --2- 172.21.15.192:0/1456761186 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3508032750 0x7f3508034c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:17.795 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.789+0000 7f3515ffb700 1 -- 172.21.15.192:0/1456761186 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f350c044690 con 0x7f35180f64b0 2024-04-12T05:01:17.795 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.790+0000 7f3517fff700 1 --2- 172.21.15.192:0/1456761186 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3508032750 0x7f3508034c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:17.795 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.790+0000 7f3517fff700 1 --2- 172.21.15.192:0/1456761186 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3508032750 0x7f3508034c10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f3504009780 tx=0x7f3504006cc0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:17.797 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:17.795+0000 7f3515ffb700 1 -- 172.21.15.192:0/1456761186 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f350c01b370 con 0x7f35180f64b0 2024-04-12T05:01:18.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.018+0000 7f351eff1700 1 -- 172.21.15.192:0/1456761186 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3518108150 con 0x7f35180f64b0 2024-04-12T05:01:18.021 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.018+0000 7f3515ffb700 1 -- 172.21.15.192:0/1456761186 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f350c01b370 con 0x7f35180f64b0 2024-04-12T05:01:18.021 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T05:01:18.021 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","modified":"2024-04-12T04:59:27.208775Z","created":"2024-04-12T04:59:27.208775Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi113","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:01:18.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.020+0000 7f351eff1700 1 -- 172.21.15.192:0/1456761186 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3508032750 msgr2=0x7f3508034c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:18.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.020+0000 7f351eff1700 1 --2- 172.21.15.192:0/1456761186 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3508032750 0x7f3508034c10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f3504009780 tx=0x7f3504006cc0).stop 2024-04-12T05:01:18.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.020+0000 7f351eff1700 1 -- 172.21.15.192:0/1456761186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35180f64b0 msgr2=0x7f3518106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:18.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.020+0000 7f351eff1700 1 --2- 172.21.15.192:0/1456761186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35180f64b0 0x7f3518106720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f350c009300 tx=0x7f350c009cb0).stop 2024-04-12T05:01:18.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.020+0000 7f351eff1700 1 -- 172.21.15.192:0/1456761186 shutdown_connections 2024-04-12T05:01:18.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.020+0000 7f351eff1700 1 --2- 172.21.15.192:0/1456761186 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f3508032750 0x7f3508034c10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:18.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.020+0000 7f351eff1700 1 --2- 172.21.15.192:0/1456761186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35180f64b0 0x7f3518106720 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:18.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.020+0000 7f351eff1700 1 -- 172.21.15.192:0/1456761186 >> 172.21.15.192:0/1456761186 conn(0x7f35180f1c10 msgr2=0x7f35180f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:18.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.020+0000 7f351eff1700 1 -- 172.21.15.192:0/1456761186 shutdown_connections 2024-04-12T05:01:18.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:18.020+0000 7f351eff1700 1 -- 172.21.15.192:0/1456761186 wait complete. 2024-04-12T05:01:18.024 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-04-12T05:01:18.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:18.020066+0000 mon.smithi113 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.192:0/1456761186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:01:18.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:18.253155+0000 mon.smithi113 (mon.0) 122 : audit [INF] from='mgr.14154 172.21.15.113:0/71483868' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-12T05:01:18.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:18.253216+0000 mon.smithi113 (mon.0) 123 : cluster [DBG] mgrmap e14: smithi113.ctsazg(active, since 48s) 2024-04-12T05:01:19.645 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:01:19.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:01:21.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.540+0000 7f91dbd1f700 1 -- 172.21.15.192:0/55539875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40f64b0 msgr2=0x7f91d40f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:21.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.540+0000 7f91dbd1f700 1 --2- 172.21.15.192:0/55539875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40f64b0 0x7f91d40f68d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f91c4004770 tx=0x7f91c4009cb0).stop 2024-04-12T05:01:21.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.540+0000 7f91dbd1f700 1 -- 172.21.15.192:0/55539875 shutdown_connections 2024-04-12T05:01:21.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.540+0000 7f91dbd1f700 1 --2- 172.21.15.192:0/55539875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40f64b0 0x7f91d40f68d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:21.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.540+0000 7f91dbd1f700 1 -- 172.21.15.192:0/55539875 >> 172.21.15.192:0/55539875 conn(0x7f91d40f1c10 msgr2=0x7f91d40f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:21.543 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.540+0000 7f91dbd1f700 1 -- 172.21.15.192:0/55539875 shutdown_connections 2024-04-12T05:01:21.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.540+0000 7f91dbd1f700 1 -- 172.21.15.192:0/55539875 wait complete. 2024-04-12T05:01:21.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.540+0000 7f91dbd1f700 1 Processor -- start 2024-04-12T05:01:21.544 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.540+0000 7f91dbd1f700 1 -- start start 2024-04-12T05:01:21.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91dbd1f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40f64b0 0x7f91d410a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91dbd1f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40611b0 con 0x7f91d40f64b0 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91d9abb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40f64b0 0x7f91d410a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91d9abb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40f64b0 0x7f91d410a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.192:42224/0 (socket says 172.21.15.192:42224) 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91d9abb700 1 -- 172.21.15.192:0/689723823 learned_addr learned my addr 172.21.15.192:0/689723823 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91d9abb700 1 -- 172.21.15.192:0/689723823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c4005040 con 0x7f91d40f64b0 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91d9abb700 1 --2- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40f64b0 0x7f91d410a520 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f91c4009300 tx=0x7f91c4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91caffd700 1 -- 172.21.15.192:0/689723823 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91c4018070 con 0x7f91d40f64b0 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91caffd700 1 -- 172.21.15.192:0/689723823 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f91c4009890 con 0x7f91d40f64b0 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91dbd1f700 1 -- 172.21.15.192:0/689723823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d40613b0 con 0x7f91d40f64b0 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91caffd700 1 -- 172.21.15.192:0/689723823 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91c401c400 con 0x7f91d40f64b0 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.541+0000 7f91dbd1f700 1 -- 172.21.15.192:0/689723823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d410abd0 con 0x7f91d40f64b0 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.542+0000 7f91caffd700 1 -- 172.21.15.192:0/689723823 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 14) v1 ==== 38703+0+0 (secure 0 0 0) 0x7f91c401d420 con 0x7f91d40f64b0 2024-04-12T05:01:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.542+0000 7f91dbd1f700 1 -- 172.21.15.192:0/689723823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d4046ce0 con 0x7f91d40f64b0 2024-04-12T05:01:21.547 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.542+0000 7f91caffd700 1 --2- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f91c0032790 0x7f91c0034c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:21.547 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.542+0000 7f91d92ba700 1 -- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f91c0032790 msgr2=0x7f91c0034c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/1349703403 2024-04-12T05:01:21.547 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.542+0000 7f91caffd700 1 -- 172.21.15.192:0/689723823 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f91c40446f0 con 0x7f91d40f64b0 2024-04-12T05:01:21.547 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.542+0000 7f91d92ba700 1 --2- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f91c0032790 0x7f91c0034c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:01:21.554 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.551+0000 7f91caffd700 1 -- 172.21.15.192:0/689723823 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f91c401b370 con 0x7f91d40f64b0 2024-04-12T05:01:21.745 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.743+0000 7f91d92ba700 1 -- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f91c0032790 msgr2=0x7f91c0034c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:6800/1349703403 2024-04-12T05:01:21.746 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.743+0000 7f91d92ba700 1 --2- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f91c0032790 0x7f91c0034c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-04-12T05:01:21.776 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.774+0000 7f91dbd1f700 1 -- 172.21.15.192:0/689723823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91d410b3f0 con 0x7f91d40f64b0 2024-04-12T05:01:21.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.774+0000 7f91caffd700 1 -- 172.21.15.192:0/689723823 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f91c401b370 con 0x7f91d40f64b0 2024-04-12T05:01:21.777 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T05:01:21.777 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","modified":"2024-04-12T04:59:27.208775Z","created":"2024-04-12T04:59:27.208775Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi113","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:01:21.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.776+0000 7f91dbd1f700 1 -- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f91c0032790 msgr2=0x7f91c0034c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:01:21.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.776+0000 7f91dbd1f700 1 --2- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f91c0032790 0x7f91c0034c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:01:21.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.776+0000 7f91dbd1f700 1 -- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40f64b0 msgr2=0x7f91d410a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:21.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.776+0000 7f91dbd1f700 1 --2- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40f64b0 0x7f91d410a520 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f91c4009300 tx=0x7f91c4009cb0).stop 2024-04-12T05:01:21.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.776+0000 7f91dbd1f700 1 -- 172.21.15.192:0/689723823 shutdown_connections 2024-04-12T05:01:21.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.776+0000 7f91dbd1f700 1 --2- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:6800/1349703403,v1:172.21.15.113:6801/1349703403] conn(0x7f91c0032790 0x7f91c0034c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:01:21.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.776+0000 7f91dbd1f700 1 --2- 172.21.15.192:0/689723823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40f64b0 0x7f91d410a520 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:21.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.776+0000 7f91dbd1f700 1 -- 172.21.15.192:0/689723823 >> 172.21.15.192:0/689723823 conn(0x7f91d40f1c10 msgr2=0x7f91d40f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:21.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.776+0000 7f91dbd1f700 1 -- 172.21.15.192:0/689723823 shutdown_connections 2024-04-12T05:01:21.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:21.776+0000 7f91dbd1f700 1 -- 172.21.15.192:0/689723823 wait complete. 2024-04-12T05:01:21.780 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-04-12T05:01:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:21.776096+0000 mon.smithi113 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.192:0/689723823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:01:23.383 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:01:23.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:01:23.797 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:23.491601+0000 mon.smithi113 (mon.0) 125 : cluster [INF] Active manager daemon smithi113.ctsazg restarted 2024-04-12T05:01:23.798 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:23.491824+0000 mon.smithi113 (mon.0) 126 : cluster [INF] Activating manager daemon smithi113.ctsazg 2024-04-12T05:01:23.798 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:23.497072+0000 mon.smithi113 (mon.0) 127 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-04-12T05:01:23.798 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:23.497173+0000 mon.smithi113 (mon.0) 128 : cluster [DBG] mgrmap e15: smithi113.ctsazg(active, starting, since 0.00544577s) 2024-04-12T05:01:23.798 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:23.498039+0000 mon.smithi113 (mon.0) 129 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:01:23.798 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:23.498736+0000 mon.smithi113 (mon.0) 130 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi113.ctsazg", "id": "smithi113.ctsazg"}]: dispatch 2024-04-12T05:01:23.798 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:23.499384+0000 mon.smithi113 (mon.0) 131 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:01:23.798 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:23.499602+0000 mon.smithi113 (mon.0) 132 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:01:23.798 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:23.499829+0000 mon.smithi113 (mon.0) 133 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:01:23.798 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:23.513395+0000 mon.smithi113 (mon.0) 134 : cluster [INF] Manager daemon smithi113.ctsazg is now available 2024-04-12T05:01:24.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:23.605481+0000 mon.smithi113 (mon.0) 135 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:01:24.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:23.606764+0000 mon.smithi113 (mon.0) 136 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:01:24.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:23.637374+0000 mon.smithi113 (mon.0) 137 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:24.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:23.646228+0000 mon.smithi113 (mon.0) 138 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:01:24.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:23.649709+0000 mon.smithi113 (mon.0) 139 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/trash_purge_schedule"}]: dispatch 2024-04-12T05:01:24.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:24.408558+0000 mon.smithi113 (mon.0) 140 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:24.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:24.496661+0000 mon.smithi113 (mon.0) 141 : cluster [DBG] mgrmap e16: smithi113.ctsazg(active, since 1.00493s) 2024-04-12T05:01:25.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.199+0000 7fac294d0700 1 -- 172.21.15.192:0/3969996221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac240f64b0 msgr2=0x7fac240f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:25.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.199+0000 7fac294d0700 1 --2- 172.21.15.192:0/3969996221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac240f64b0 0x7fac240f68d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fac14004770 tx=0x7fac14009cb0).stop 2024-04-12T05:01:25.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.199+0000 7fac294d0700 1 -- 172.21.15.192:0/3969996221 shutdown_connections 2024-04-12T05:01:25.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.199+0000 7fac294d0700 1 --2- 172.21.15.192:0/3969996221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac240f64b0 0x7fac240f68d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:25.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.199+0000 7fac294d0700 1 -- 172.21.15.192:0/3969996221 >> 172.21.15.192:0/3969996221 conn(0x7fac240f1c10 msgr2=0x7fac240f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:25.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.200+0000 7fac294d0700 1 -- 172.21.15.192:0/3969996221 shutdown_connections 2024-04-12T05:01:25.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.200+0000 7fac294d0700 1 -- 172.21.15.192:0/3969996221 wait complete. 2024-04-12T05:01:25.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.200+0000 7fac294d0700 1 Processor -- start 2024-04-12T05:01:25.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.200+0000 7fac294d0700 1 -- start start 2024-04-12T05:01:25.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.200+0000 7fac294d0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac240f64b0 0x7fac2410a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:25.204 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.200+0000 7fac294d0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac240611b0 con 0x7fac240f64b0 2024-04-12T05:01:25.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.201+0000 7fac22ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac240f64b0 0x7fac2410a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:25.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.201+0000 7fac22ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac240f64b0 0x7fac2410a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.192:56466/0 (socket says 172.21.15.192:56466) 2024-04-12T05:01:25.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.201+0000 7fac22ffd700 1 -- 172.21.15.192:0/2942290530 learned_addr learned my addr 172.21.15.192:0/2942290530 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:01:25.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.201+0000 7fac22ffd700 1 -- 172.21.15.192:0/2942290530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac14005040 con 0x7fac240f64b0 2024-04-12T05:01:25.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.201+0000 7fac22ffd700 1 --2- 172.21.15.192:0/2942290530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac240f64b0 0x7fac2410a520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fac14009300 tx=0x7fac14009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:25.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.201+0000 7fac0bfff700 1 -- 172.21.15.192:0/2942290530 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac14018070 con 0x7fac240f64b0 2024-04-12T05:01:25.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.201+0000 7fac0bfff700 1 -- 172.21.15.192:0/2942290530 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fac14009890 con 0x7fac240f64b0 2024-04-12T05:01:25.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.201+0000 7fac294d0700 1 -- 172.21.15.192:0/2942290530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac240613b0 con 0x7fac240f64b0 2024-04-12T05:01:25.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.201+0000 7fac0bfff700 1 -- 172.21.15.192:0/2942290530 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac1401c400 con 0x7fac240f64b0 2024-04-12T05:01:25.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.201+0000 7fac294d0700 1 -- 172.21.15.192:0/2942290530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2410abd0 con 0x7fac240f64b0 2024-04-12T05:01:25.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.202+0000 7fac0bfff700 1 -- 172.21.15.192:0/2942290530 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) v1 ==== 38553+0+0 (secure 0 0 0) 0x7fac1401c670 con 0x7fac240f64b0 2024-04-12T05:01:25.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.202+0000 7fac0bfff700 1 --2- 172.21.15.192:0/2942290530 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fac0c0325e0 0x7fac0c034aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:25.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.202+0000 7fac0bfff700 1 -- 172.21.15.192:0/2942290530 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1237+0+0 (secure 0 0 0) 0x7fac140452b0 con 0x7fac240f64b0 2024-04-12T05:01:25.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.202+0000 7fac227fc700 1 --2- 172.21.15.192:0/2942290530 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fac0c0325e0 0x7fac0c034aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:25.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.203+0000 7fac294d0700 1 -- 172.21.15.192:0/2942290530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2405a310 con 0x7fac240f64b0 2024-04-12T05:01:25.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.203+0000 7fac227fc700 1 --2- 172.21.15.192:0/2942290530 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fac0c0325e0 0x7fac0c034aa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fac18009780 tx=0x7fac18006cc0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:25.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.208+0000 7fac0bfff700 1 -- 172.21.15.192:0/2942290530 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fac1401ca60 con 0x7fac240f64b0 2024-04-12T05:01:25.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.458+0000 7fac294d0700 1 -- 172.21.15.192:0/2942290530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac240fbfa0 con 0x7fac240f64b0 2024-04-12T05:01:25.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.459+0000 7fac0bfff700 1 -- 172.21.15.192:0/2942290530 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fac1401bb60 con 0x7fac240f64b0 2024-04-12T05:01:25.462 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T05:01:25.462 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","modified":"2024-04-12T04:59:27.208775Z","created":"2024-04-12T04:59:27.208775Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi113","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:01:25.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.461+0000 7fac294d0700 1 -- 172.21.15.192:0/2942290530 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fac0c0325e0 msgr2=0x7fac0c034aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:25.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.461+0000 7fac294d0700 1 --2- 172.21.15.192:0/2942290530 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fac0c0325e0 0x7fac0c034aa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fac18009780 tx=0x7fac18006cc0).stop 2024-04-12T05:01:25.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.461+0000 7fac294d0700 1 -- 172.21.15.192:0/2942290530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac240f64b0 msgr2=0x7fac2410a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:25.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.461+0000 7fac294d0700 1 --2- 172.21.15.192:0/2942290530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac240f64b0 0x7fac2410a520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fac14009300 tx=0x7fac14009cb0).stop 2024-04-12T05:01:25.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.461+0000 7fac294d0700 1 -- 172.21.15.192:0/2942290530 shutdown_connections 2024-04-12T05:01:25.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.461+0000 7fac294d0700 1 --2- 172.21.15.192:0/2942290530 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fac0c0325e0 0x7fac0c034aa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:25.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.461+0000 7fac294d0700 1 --2- 172.21.15.192:0/2942290530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac240f64b0 0x7fac2410a520 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:25.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.461+0000 7fac294d0700 1 -- 172.21.15.192:0/2942290530 >> 172.21.15.192:0/2942290530 conn(0x7fac240f1c10 msgr2=0x7fac240f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:25.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.461+0000 7fac294d0700 1 -- 172.21.15.192:0/2942290530 shutdown_connections 2024-04-12T05:01:25.464 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:25.461+0000 7fac294d0700 1 -- 172.21.15.192:0/2942290530 wait complete. 2024-04-12T05:01:25.465 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-04-12T05:01:26.589 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:25.223809+0000 mon.smithi113 (mon.0) 142 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:26.589 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:25.460563+0000 mon.smithi113 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.192:0/2942290530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:01:27.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:26.225213+0000 mon.smithi113 (mon.0) 144 : cluster [DBG] mgrmap e17: smithi113.ctsazg(active, since 2s) 2024-04-12T05:01:28.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:01:28.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:01:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:27.830588+0000 mon.smithi113 (mon.0) 145 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:28.104973+0000 mon.smithi113 (mon.0) 146 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd/host:smithi113", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:01:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:28.435325+0000 mon.smithi113 (mon.0) 147 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:29.018924+0000 mon.smithi113 (mon.0) 148 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:29.019722+0000 mon.smithi113 (mon.0) 149 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:01:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:29.020685+0000 mon.smithi113 (mon.0) 150 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:01:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:29.021530+0000 mon.smithi113 (mon.0) 151 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:01:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:29.022947+0000 mgr.smithi113.ctsazg (mgr.14194) 1 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.conf 2024-04-12T05:01:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:29.023239+0000 mgr.smithi113.ctsazg (mgr.14194) 2 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-04-12T05:01:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:29.046668+0000 mgr.smithi113.ctsazg (mgr.14194) 3 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:01:30.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:29.046994+0000 mgr.smithi113.ctsazg (mgr.14194) 4 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:01:30.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:29.075298+0000 mon.smithi113 (mon.0) 152 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:30.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:29.095960+0000 mon.smithi113 (mon.0) 153 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:30.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:29.098734+0000 mon.smithi113 (mon.0) 154 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:30.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:29.100737+0000 mon.smithi113 (mon.0) 155 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:01:30.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:29.102738+0000 mon.smithi113 (mon.0) 156 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-12T05:01:30.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:29.103516+0000 mon.smithi113 (mon.0) 157 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:01:30.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:29.104251+0000 mgr.smithi113.ctsazg (mgr.14194) 5 : cephadm [INF] Deploying daemon crash.smithi192 on smithi192 2024-04-12T05:01:31.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.199+0000 7fce669fa700 1 -- 172.21.15.192:0/1306296653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce600f8690 msgr2=0x7fce600f8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:31.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.199+0000 7fce669fa700 1 --2- 172.21.15.192:0/1306296653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce600f8690 0x7fce600f8a70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fce50004770 tx=0x7fce50009cb0).stop 2024-04-12T05:01:31.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.199+0000 7fce669fa700 1 -- 172.21.15.192:0/1306296653 shutdown_connections 2024-04-12T05:01:31.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.199+0000 7fce669fa700 1 --2- 172.21.15.192:0/1306296653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce600f8690 0x7fce600f8a70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:31.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.199+0000 7fce669fa700 1 -- 172.21.15.192:0/1306296653 >> 172.21.15.192:0/1306296653 conn(0x7fce6006c440 msgr2=0x7fce6006c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:31.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.199+0000 7fce669fa700 1 -- 172.21.15.192:0/1306296653 shutdown_connections 2024-04-12T05:01:31.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.199+0000 7fce669fa700 1 -- 172.21.15.192:0/1306296653 wait complete. 2024-04-12T05:01:31.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.200+0000 7fce669fa700 1 Processor -- start 2024-04-12T05:01:31.203 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.200+0000 7fce669fa700 1 -- start start 2024-04-12T05:01:31.204 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.200+0000 7fce669fa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce600f8690 0x7fce6010d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:31.204 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.200+0000 7fce669fa700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6010de20 con 0x7fce600f8690 2024-04-12T05:01:31.204 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.201+0000 7fce5ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce600f8690 0x7fce6010d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:31.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.201+0000 7fce5ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce600f8690 0x7fce6010d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.192:56488/0 (socket says 172.21.15.192:56488) 2024-04-12T05:01:31.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.201+0000 7fce5ffff700 1 -- 172.21.15.192:0/599687410 learned_addr learned my addr 172.21.15.192:0/599687410 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:01:31.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.201+0000 7fce5ffff700 1 -- 172.21.15.192:0/599687410 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce50005040 con 0x7fce600f8690 2024-04-12T05:01:31.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.201+0000 7fce5ffff700 1 --2- 172.21.15.192:0/599687410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce600f8690 0x7fce6010d930 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fce50004740 tx=0x7fce50009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:31.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.201+0000 7fce5d7fa700 1 -- 172.21.15.192:0/599687410 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce50018070 con 0x7fce600f8690 2024-04-12T05:01:31.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.201+0000 7fce5d7fa700 1 -- 172.21.15.192:0/599687410 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fce50009860 con 0x7fce600f8690 2024-04-12T05:01:31.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.201+0000 7fce669fa700 1 -- 172.21.15.192:0/599687410 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6010e020 con 0x7fce600f8690 2024-04-12T05:01:31.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.201+0000 7fce5d7fa700 1 -- 172.21.15.192:0/599687410 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce5001c5b0 con 0x7fce600f8690 2024-04-12T05:01:31.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.201+0000 7fce669fa700 1 -- 172.21.15.192:0/599687410 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6010e430 con 0x7fce600f8690 2024-04-12T05:01:31.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.202+0000 7fce5d7fa700 1 -- 172.21.15.192:0/599687410 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 38747+0+0 (secure 0 0 0) 0x7fce5001d470 con 0x7fce600f8690 2024-04-12T05:01:31.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.202+0000 7fce5d7fa700 1 --2- 172.21.15.192:0/599687410 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fce4c032820 0x7fce4c034ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:31.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.202+0000 7fce669fa700 1 -- 172.21.15.192:0/599687410 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6005f1f0 con 0x7fce600f8690 2024-04-12T05:01:31.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.202+0000 7fce5d7fa700 1 -- 172.21.15.192:0/599687410 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1237+0+0 (secure 0 0 0) 0x7fce50043ff0 con 0x7fce600f8690 2024-04-12T05:01:31.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.203+0000 7fce5f7fe700 1 --2- 172.21.15.192:0/599687410 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fce4c032820 0x7fce4c034ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:31.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.203+0000 7fce5f7fe700 1 --2- 172.21.15.192:0/599687410 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fce4c032820 0x7fce4c034ce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fce48009780 tx=0x7fce48006cc0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:31.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.208+0000 7fce5d7fa700 1 -- 172.21.15.192:0/599687410 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fce500248c0 con 0x7fce600f8690 2024-04-12T05:01:31.436 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.434+0000 7fce669fa700 1 -- 172.21.15.192:0/599687410 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce60046ce0 con 0x7fce600f8690 2024-04-12T05:01:31.437 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.435+0000 7fce5d7fa700 1 -- 172.21.15.192:0/599687410 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fce5002b2f0 con 0x7fce600f8690 2024-04-12T05:01:31.437 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T05:01:31.437 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","modified":"2024-04-12T04:59:27.208775Z","created":"2024-04-12T04:59:27.208775Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi113","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:01:31.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.436+0000 7fce669fa700 1 -- 172.21.15.192:0/599687410 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fce4c032820 msgr2=0x7fce4c034ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:31.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.436+0000 7fce669fa700 1 --2- 172.21.15.192:0/599687410 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fce4c032820 0x7fce4c034ce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fce48009780 tx=0x7fce48006cc0).stop 2024-04-12T05:01:31.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.436+0000 7fce669fa700 1 -- 172.21.15.192:0/599687410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce600f8690 msgr2=0x7fce6010d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:31.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.436+0000 7fce669fa700 1 --2- 172.21.15.192:0/599687410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce600f8690 0x7fce6010d930 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fce50004740 tx=0x7fce50009cb0).stop 2024-04-12T05:01:31.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.436+0000 7fce669fa700 1 -- 172.21.15.192:0/599687410 shutdown_connections 2024-04-12T05:01:31.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.436+0000 7fce669fa700 1 --2- 172.21.15.192:0/599687410 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fce4c032820 0x7fce4c034ce0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:31.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.436+0000 7fce669fa700 1 --2- 172.21.15.192:0/599687410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce600f8690 0x7fce6010d930 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:31.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.437+0000 7fce669fa700 1 -- 172.21.15.192:0/599687410 >> 172.21.15.192:0/599687410 conn(0x7fce6006c440 msgr2=0x7fce600f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:31.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.437+0000 7fce669fa700 1 -- 172.21.15.192:0/599687410 shutdown_connections 2024-04-12T05:01:31.440 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:31.437+0000 7fce669fa700 1 -- 172.21.15.192:0/599687410 wait complete. 2024-04-12T05:01:31.440 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-04-12T05:01:31.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:31.436023+0000 mon.smithi113 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.192:0/599687410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:01:33.106 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:01:33.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:01:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:32.748345+0000 mon.smithi113 (mon.0) 159 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:32.750459+0000 mon.smithi113 (mon.0) 160 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.ocppyo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:01:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:32.754864+0000 mon.smithi113 (mon.0) 161 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.ocppyo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T05:01:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:32.755610+0000 mon.smithi113 (mon.0) 162 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:01:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:32.756441+0000 mon.smithi113 (mon.0) 163 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:01:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:32.757077+0000 mgr.smithi113.ctsazg (mgr.14194) 6 : cephadm [INF] Deploying daemon mgr.smithi192.ocppyo on smithi192 2024-04-12T05:01:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:33.639331+0000 mon.smithi113 (mon.0) 164 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:35.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.680+0000 7f98b5279700 1 -- 172.21.15.192:0/1619783233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98b00fbc60 msgr2=0x7f98b00fc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:35.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.680+0000 7f98b5279700 1 --2- 172.21.15.192:0/1619783233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98b00fbc60 0x7f98b00fc040 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f98a0004770 tx=0x7f98a0009cb0).stop 2024-04-12T05:01:35.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.680+0000 7f98b5279700 1 -- 172.21.15.192:0/1619783233 shutdown_connections 2024-04-12T05:01:35.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.680+0000 7f98b5279700 1 --2- 172.21.15.192:0/1619783233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98b00fbc60 0x7f98b00fc040 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:35.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.680+0000 7f98b5279700 1 -- 172.21.15.192:0/1619783233 >> 172.21.15.192:0/1619783233 conn(0x7f98b00f1870 msgr2=0x7f98b00f3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:35.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.680+0000 7f98b5279700 1 -- 172.21.15.192:0/1619783233 shutdown_connections 2024-04-12T05:01:35.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.680+0000 7f98b5279700 1 -- 172.21.15.192:0/1619783233 wait complete. 2024-04-12T05:01:35.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.680+0000 7f98b5279700 1 Processor -- start 2024-04-12T05:01:35.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.681+0000 7f98b5279700 1 -- start start 2024-04-12T05:01:35.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.681+0000 7f98b5279700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98b00fbc60 0x7f98b010b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:35.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.681+0000 7f98b5279700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b010bc30 con 0x7f98b00fbc60 2024-04-12T05:01:35.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.681+0000 7f98aeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98b00fbc60 0x7f98b010b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:35.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.681+0000 7f98aeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98b00fbc60 0x7f98b010b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.192:36284/0 (socket says 172.21.15.192:36284) 2024-04-12T05:01:35.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.681+0000 7f98aeffd700 1 -- 172.21.15.192:0/265097512 learned_addr learned my addr 172.21.15.192:0/265097512 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:01:35.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.681+0000 7f98aeffd700 1 -- 172.21.15.192:0/265097512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a0005040 con 0x7f98b00fbc60 2024-04-12T05:01:35.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.681+0000 7f98aeffd700 1 --2- 172.21.15.192:0/265097512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98b00fbc60 0x7f98b010b740 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f98a0004740 tx=0x7f98a0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:35.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.682+0000 7f9897fff700 1 -- 172.21.15.192:0/265097512 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a0018070 con 0x7f98b00fbc60 2024-04-12T05:01:35.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.682+0000 7f9897fff700 1 -- 172.21.15.192:0/265097512 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f98a0009860 con 0x7f98b00fbc60 2024-04-12T05:01:35.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.682+0000 7f98b5279700 1 -- 172.21.15.192:0/265097512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b010be30 con 0x7f98b00fbc60 2024-04-12T05:01:35.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.682+0000 7f9897fff700 1 -- 172.21.15.192:0/265097512 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a001c5b0 con 0x7f98b00fbc60 2024-04-12T05:01:35.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.682+0000 7f98b5279700 1 -- 172.21.15.192:0/265097512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b010c240 con 0x7f98b00fbc60 2024-04-12T05:01:35.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.682+0000 7f9897fff700 1 -- 172.21.15.192:0/265097512 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 38747+0+0 (secure 0 0 0) 0x7f98a001c820 con 0x7f98b00fbc60 2024-04-12T05:01:35.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.683+0000 7f9897fff700 1 --2- 172.21.15.192:0/265097512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f98980338e0 0x7f9898035da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:35.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.683+0000 7f9897fff700 1 -- 172.21.15.192:0/265097512 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1237+0+0 (secure 0 0 0) 0x7f98a0044060 con 0x7f98b00fbc60 2024-04-12T05:01:35.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.683+0000 7f98ae7fc700 1 --2- 172.21.15.192:0/265097512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f98980338e0 0x7f9898035da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:35.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.683+0000 7f98b5279700 1 -- 172.21.15.192:0/265097512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b005f080 con 0x7f98b00fbc60 2024-04-12T05:01:35.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.683+0000 7f98ae7fc700 1 --2- 172.21.15.192:0/265097512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f98980338e0 0x7f9898035da0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f98a4009780 tx=0x7f98a4006cc0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:35.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.688+0000 7f9897fff700 1 -- 172.21.15.192:0/265097512 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f98a0026910 con 0x7f98b00fbc60 2024-04-12T05:01:35.922 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.919+0000 7f98b5279700 1 -- 172.21.15.192:0/265097512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98b0046ce0 con 0x7f98b00fbc60 2024-04-12T05:01:35.923 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.920+0000 7f9897fff700 1 -- 172.21.15.192:0/265097512 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f98a0022210 con 0x7f98b00fbc60 2024-04-12T05:01:35.923 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T05:01:35.923 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","modified":"2024-04-12T04:59:27.208775Z","created":"2024-04-12T04:59:27.208775Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi113","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:01:35.925 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.922+0000 7f98b5279700 1 -- 172.21.15.192:0/265097512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f98980338e0 msgr2=0x7f9898035da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:35.925 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.922+0000 7f98b5279700 1 --2- 172.21.15.192:0/265097512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f98980338e0 0x7f9898035da0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f98a4009780 tx=0x7f98a4006cc0).stop 2024-04-12T05:01:35.925 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.922+0000 7f98b5279700 1 -- 172.21.15.192:0/265097512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98b00fbc60 msgr2=0x7f98b010b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:35.925 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.922+0000 7f98b5279700 1 --2- 172.21.15.192:0/265097512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98b00fbc60 0x7f98b010b740 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f98a0004740 tx=0x7f98a0009cb0).stop 2024-04-12T05:01:35.925 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.922+0000 7f98b5279700 1 -- 172.21.15.192:0/265097512 shutdown_connections 2024-04-12T05:01:35.925 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.922+0000 7f98b5279700 1 --2- 172.21.15.192:0/265097512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f98980338e0 0x7f9898035da0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:35.925 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.922+0000 7f98b5279700 1 --2- 172.21.15.192:0/265097512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98b00fbc60 0x7f98b010b740 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:35.925 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.922+0000 7f98b5279700 1 -- 172.21.15.192:0/265097512 >> 172.21.15.192:0/265097512 conn(0x7f98b00f1870 msgr2=0x7f98b00f1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:35.925 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.922+0000 7f98b5279700 1 -- 172.21.15.192:0/265097512 shutdown_connections 2024-04-12T05:01:35.925 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:35.922+0000 7f98b5279700 1 -- 172.21.15.192:0/265097512 wait complete. 2024-04-12T05:01:35.926 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-04-12T05:01:36.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:35.921586+0000 mon.smithi113 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.192:0/265097512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:01:37.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:01:37.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:01:38.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:36.781304+0000 mon.smithi113 (mon.0) 166 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:38.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:36.783516+0000 mon.smithi113 (mon.0) 167 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:01:38.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:36.784287+0000 mon.smithi113 (mon.0) 168 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:01:38.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:36.784933+0000 mgr.smithi113.ctsazg (mgr.14194) 7 : cephadm [INF] Deploying daemon mon.smithi192 on smithi192 2024-04-12T05:01:39.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:38.642868+0000 mon.smithi113 (mon.0) 169 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:41.089 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.085+0000 7fb689ac6700 1 -- 172.21.15.192:0/832408303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6840f8690 msgr2=0x7fb6840f8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:41.089 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.085+0000 7fb689ac6700 1 --2- 172.21.15.192:0/832408303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6840f8690 0x7fb6840f8a70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb674004770 tx=0x7fb674009cb0).stop 2024-04-12T05:01:41.089 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.085+0000 7fb689ac6700 1 -- 172.21.15.192:0/832408303 shutdown_connections 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.085+0000 7fb689ac6700 1 --2- 172.21.15.192:0/832408303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6840f8690 0x7fb6840f8a70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.085+0000 7fb689ac6700 1 -- 172.21.15.192:0/832408303 >> 172.21.15.192:0/832408303 conn(0x7fb68406c440 msgr2=0x7fb68406c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.085+0000 7fb689ac6700 1 -- 172.21.15.192:0/832408303 shutdown_connections 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.085+0000 7fb689ac6700 1 -- 172.21.15.192:0/832408303 wait complete. 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.085+0000 7fb689ac6700 1 Processor -- start 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.085+0000 7fb689ac6700 1 -- start start 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb689ac6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6840f8690 0x7fb68410d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb689ac6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68410de20 con 0x7fb6840f8690 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb6837fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6840f8690 0x7fb68410d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb6837fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6840f8690 0x7fb68410d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.192:36316/0 (socket says 172.21.15.192:36316) 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb6837fe700 1 -- 172.21.15.192:0/1400279621 learned_addr learned my addr 172.21.15.192:0/1400279621 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb6837fe700 1 -- 172.21.15.192:0/1400279621 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb674005040 con 0x7fb6840f8690 2024-04-12T05:01:41.090 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb6837fe700 1 --2- 172.21.15.192:0/1400279621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6840f8690 0x7fb68410d930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb674004740 tx=0x7fb674009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:41.091 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb680f74700 1 -- 172.21.15.192:0/1400279621 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb674018070 con 0x7fb6840f8690 2024-04-12T05:01:41.091 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb689ac6700 1 -- 172.21.15.192:0/1400279621 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68410e020 con 0x7fb6840f8690 2024-04-12T05:01:41.091 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb680f74700 1 -- 172.21.15.192:0/1400279621 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fb674009860 con 0x7fb6840f8690 2024-04-12T05:01:41.091 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb689ac6700 1 -- 172.21.15.192:0/1400279621 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68410e430 con 0x7fb6840f8690 2024-04-12T05:01:41.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.086+0000 7fb680f74700 1 -- 172.21.15.192:0/1400279621 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb67401c5b0 con 0x7fb6840f8690 2024-04-12T05:01:41.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.087+0000 7fb680f74700 1 -- 172.21.15.192:0/1400279621 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 38747+0+0 (secure 0 0 0) 0x7fb67401c820 con 0x7fb6840f8690 2024-04-12T05:01:41.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.088+0000 7fb680f74700 1 --2- 172.21.15.192:0/1400279621 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb66c0327d0 0x7fb66c034c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:41.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.088+0000 7fb689ac6700 1 -- 172.21.15.192:0/1400279621 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68405f1f0 con 0x7fb6840f8690 2024-04-12T05:01:41.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.088+0000 7fb680f74700 1 -- 172.21.15.192:0/1400279621 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1237+0+0 (secure 0 0 0) 0x7fb674044080 con 0x7fb6840f8690 2024-04-12T05:01:41.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.088+0000 7fb682ffd700 1 --2- 172.21.15.192:0/1400279621 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb66c0327d0 0x7fb66c034c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:41.092 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.088+0000 7fb682ffd700 1 --2- 172.21.15.192:0/1400279621 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb66c0327d0 0x7fb66c034c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb678009780 tx=0x7fb678006cc0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:41.095 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.093+0000 7fb680f74700 1 -- 172.21.15.192:0/1400279621 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb674026910 con 0x7fb6840f8690 2024-04-12T05:01:41.333 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.331+0000 7fb689ac6700 1 -- 172.21.15.192:0/1400279621 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb684046ce0 con 0x7fb6840f8690 2024-04-12T05:01:41.334 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.331+0000 7fb680f74700 1 -- 172.21.15.192:0/1400279621 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fb67402b2f0 con 0x7fb6840f8690 2024-04-12T05:01:41.334 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T05:01:41.334 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","modified":"2024-04-12T04:59:27.208775Z","created":"2024-04-12T04:59:27.208775Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi113","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T05:01:41.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.333+0000 7fb689ac6700 1 -- 172.21.15.192:0/1400279621 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb66c0327d0 msgr2=0x7fb66c034c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:41.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.333+0000 7fb689ac6700 1 --2- 172.21.15.192:0/1400279621 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb66c0327d0 0x7fb66c034c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb678009780 tx=0x7fb678006cc0).stop 2024-04-12T05:01:41.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.333+0000 7fb689ac6700 1 -- 172.21.15.192:0/1400279621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6840f8690 msgr2=0x7fb68410d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:41.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.333+0000 7fb689ac6700 1 --2- 172.21.15.192:0/1400279621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6840f8690 0x7fb68410d930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb674004740 tx=0x7fb674009cb0).stop 2024-04-12T05:01:41.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.334+0000 7fb689ac6700 1 -- 172.21.15.192:0/1400279621 shutdown_connections 2024-04-12T05:01:41.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.334+0000 7fb689ac6700 1 --2- 172.21.15.192:0/1400279621 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb66c0327d0 0x7fb66c034c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:41.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.334+0000 7fb689ac6700 1 --2- 172.21.15.192:0/1400279621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6840f8690 0x7fb68410d930 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:41.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.334+0000 7fb689ac6700 1 -- 172.21.15.192:0/1400279621 >> 172.21.15.192:0/1400279621 conn(0x7fb68406c440 msgr2=0x7fb6840f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:41.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.334+0000 7fb689ac6700 1 -- 172.21.15.192:0/1400279621 shutdown_connections 2024-04-12T05:01:41.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:41.334+0000 7fb689ac6700 1 -- 172.21.15.192:0/1400279621 wait complete. 2024-04-12T05:01:41.337 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-04-12T05:01:41.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:41.333040+0000 mon.smithi113 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.192:0/1400279621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:01:42.011 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:41 smithi192 bash[42550]: adce07508243c3601c23033a32cd61f9adefc4257c28870198f30ef0e146a9a9 2024-04-12T05:01:42.011 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:41 smithi192 systemd[1]: Started Ceph mon.smithi192 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:01:43.278 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-12T05:01:43.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph mon dump -f json 2024-04-12T05:01:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:41.774795+0000 mon.smithi113 (mon.0) 172 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:01:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:41.774965+0000 mon.smithi113 (mon.0) 173 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:41.775059+0000 mon.smithi113 (mon.0) 174 : cluster [INF] mon.smithi113 calling monitor election 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:42.286692+0000 mon.smithi113 (mon.0) 175 : audit [DBG] from='mgr.? 172.21.15.192:0/1071742738' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/crt"}]: dispatch 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:42.770700+0000 mon.smithi113 (mon.0) 176 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:43.501171+0000 mgr.smithi113.ctsazg (mgr.14194) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:43.770899+0000 mon.smithi113 (mon.0) 177 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:43.772959+0000 mon.smithi192 (mon.1) 1 : cluster [INF] mon.smithi192 calling monitor election 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:44.771235+0000 mon.smithi113 (mon.0) 178 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:45.501474+0000 mgr.smithi113.ctsazg (mgr.14194) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:45.771006+0000 mon.smithi113 (mon.0) 179 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:46.771156+0000 mon.smithi113 (mon.0) 180 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:46.802692+0000 mon.smithi113 (mon.0) 181 : cluster [INF] mon.smithi113 is new leader, mons smithi113,smithi192 in quorum (ranks 0,1) 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:46.805716+0000 mon.smithi113 (mon.0) 182 : cluster [DBG] monmap e2: 2 mons at {smithi113=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:46.818430+0000 mon.smithi113 (mon.0) 183 : cluster [DBG] fsmap 2024-04-12T05:01:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:46.818475+0000 mon.smithi113 (mon.0) 184 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-04-12T05:01:47.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:46.818730+0000 mon.smithi113 (mon.0) 185 : cluster [DBG] mgrmap e17: smithi113.ctsazg(active, since 23s) 2024-04-12T05:01:47.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:46.818894+0000 mon.smithi113 (mon.0) 186 : cluster [INF] overall HEALTH_OK 2024-04-12T05:01:47.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:46.819430+0000 mon.smithi113 (mon.0) 187 : cluster [DBG] Standby manager daemon smithi192.ocppyo started 2024-04-12T05:01:47.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:46.820779+0000 mon.smithi113 (mon.0) 188 : audit [DBG] from='mgr.? 172.21.15.192:0/1071742738' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:01:47.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:46.822575+0000 mon.smithi113 (mon.0) 189 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:47.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:46.822904+0000 mon.smithi113 (mon.0) 190 : audit [DBG] from='mgr.? 172.21.15.192:0/1071742738' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/key"}]: dispatch 2024-04-12T05:01:47.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:46.823490+0000 mon.smithi113 (mon.0) 191 : audit [DBG] from='mgr.? 172.21.15.192:0/1071742738' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:01:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:41.774795+0000 mon.smithi113 (mon.0) 172 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:41.774965+0000 mon.smithi113 (mon.0) 173 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:41.775059+0000 mon.smithi113 (mon.0) 174 : cluster [INF] mon.smithi113 calling monitor election 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:42.286692+0000 mon.smithi113 (mon.0) 175 : audit [DBG] from='mgr.? 172.21.15.192:0/1071742738' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/crt"}]: dispatch 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:42.770700+0000 mon.smithi113 (mon.0) 176 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:43.501171+0000 mgr.smithi113.ctsazg (mgr.14194) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:43.770899+0000 mon.smithi113 (mon.0) 177 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:43.772959+0000 mon.smithi192 (mon.1) 1 : cluster [INF] mon.smithi192 calling monitor election 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:44.771235+0000 mon.smithi113 (mon.0) 178 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:45.501474+0000 mgr.smithi113.ctsazg (mgr.14194) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:45.771006+0000 mon.smithi113 (mon.0) 179 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:46.771156+0000 mon.smithi113 (mon.0) 180 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:46.802692+0000 mon.smithi113 (mon.0) 181 : cluster [INF] mon.smithi113 is new leader, mons smithi113,smithi192 in quorum (ranks 0,1) 2024-04-12T05:01:47.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:46.805716+0000 mon.smithi113 (mon.0) 182 : cluster [DBG] monmap e2: 2 mons at {smithi113=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} 2024-04-12T05:01:47.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:46.818430+0000 mon.smithi113 (mon.0) 183 : cluster [DBG] fsmap 2024-04-12T05:01:47.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:46.818475+0000 mon.smithi113 (mon.0) 184 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-04-12T05:01:47.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:46.818730+0000 mon.smithi113 (mon.0) 185 : cluster [DBG] mgrmap e17: smithi113.ctsazg(active, since 23s) 2024-04-12T05:01:47.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:46.818894+0000 mon.smithi113 (mon.0) 186 : cluster [INF] overall HEALTH_OK 2024-04-12T05:01:47.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:46.819430+0000 mon.smithi113 (mon.0) 187 : cluster [DBG] Standby manager daemon smithi192.ocppyo started 2024-04-12T05:01:47.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:46.820779+0000 mon.smithi113 (mon.0) 188 : audit [DBG] from='mgr.? 172.21.15.192:0/1071742738' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:01:47.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:46.822575+0000 mon.smithi113 (mon.0) 189 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:47.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:46.822904+0000 mon.smithi113 (mon.0) 190 : audit [DBG] from='mgr.? 172.21.15.192:0/1071742738' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/key"}]: dispatch 2024-04-12T05:01:47.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:46.823490+0000 mon.smithi113 (mon.0) 191 : audit [DBG] from='mgr.? 172.21.15.192:0/1071742738' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:01:48.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.150+0000 7f9610a2b700 1 -- 172.21.15.192:0/3429634165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f4003400 msgr2=0x7f95f4005890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:48.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.150+0000 7f9610a2b700 1 --2- 172.21.15.192:0/3429634165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f4003400 0x7f95f4005890 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f960c0f6590 tx=0x7f960000b040).stop 2024-04-12T05:01:48.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.150+0000 7f9610a2b700 1 -- 172.21.15.192:0/3429634165 shutdown_connections 2024-04-12T05:01:48.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.150+0000 7f9610a2b700 1 --2- 172.21.15.192:0/3429634165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f4003400 0x7f95f4005890 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:48.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.150+0000 7f9610a2b700 1 -- 172.21.15.192:0/3429634165 >> 172.21.15.192:0/3429634165 conn(0x7f960c06c440 msgr2=0x7f960c06c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:48.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.150+0000 7f9610a2b700 1 -- 172.21.15.192:0/3429634165 shutdown_connections 2024-04-12T05:01:48.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.150+0000 7f9610a2b700 1 -- 172.21.15.192:0/3429634165 wait complete. 2024-04-12T05:01:48.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.152+0000 7f9610a2b700 1 Processor -- start 2024-04-12T05:01:48.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.152+0000 7f9610a2b700 1 -- start start 2024-04-12T05:01:48.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.152+0000 7f9610a2b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f4003400 0x7f960c10c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:48.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.152+0000 7f9610a2b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f960c10cbf0 0x7f960c106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:48.156 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.152+0000 7f9610a2b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c10d1c0 con 0x7f95f4003400 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.152+0000 7f9610a2b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c10d300 con 0x7f960c10cbf0 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.152+0000 7f9609d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f960c10cbf0 0x7f960c106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.152+0000 7f9609d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f960c10cbf0 0x7f960c106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:48448/0 (socket says 172.21.15.192:48448) 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.152+0000 7f9609d9b700 1 -- 172.21.15.192:0/1979634013 learned_addr learned my addr 172.21.15.192:0/1979634013 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f9609d9b700 1 -- 172.21.15.192:0/1979634013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f960c10cbf0 msgr2=0x7f960c106690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f9609d9b700 1 -- 172.21.15.192:0/1979634013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f960c10cbf0 msgr2=0x7f960c106690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f9609d9b700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f960c10cbf0 0x7f960c106690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f960a59c700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f4003400 0x7f960c10c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f9609d9b700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f960c10cbf0 0x7f960c106690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f960a59c700 1 -- 172.21.15.192:0/1979634013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f960c10cbf0 msgr2=0x7f960c106690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f960a59c700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f960c10cbf0 0x7f960c106690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f960a59c700 1 -- 172.21.15.192:0/1979634013 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9600005040 con 0x7f95f4003400 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f960a59c700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f4003400 0x7f960c10c700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9600019040 tx=0x7f9600002040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f95fb7fe700 1 -- 172.21.15.192:0/1979634013 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9600020440 con 0x7f95f4003400 2024-04-12T05:01:48.158 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f9610a2b700 1 -- 172.21.15.192:0/1979634013 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960c106be0 con 0x7f95f4003400 2024-04-12T05:01:48.158 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f95fb7fe700 1 -- 172.21.15.192:0/1979634013 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f9600020a30 con 0x7f95f4003400 2024-04-12T05:01:48.158 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f95fb7fe700 1 -- 172.21.15.192:0/1979634013 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f960001f840 con 0x7f95f4003400 2024-04-12T05:01:48.158 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.153+0000 7f9610a2b700 1 -- 172.21.15.192:0/1979634013 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960c1070f0 con 0x7f95f4003400 2024-04-12T05:01:48.159 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.154+0000 7f9610a2b700 1 -- 172.21.15.192:0/1979634013 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960c05f1f0 con 0x7f95f4003400 2024-04-12T05:01:48.159 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.155+0000 7f95fb7fe700 1 -- 172.21.15.192:0/1979634013 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f96000205a0 con 0x7f95f4003400 2024-04-12T05:01:48.159 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.155+0000 7f95fb7fe700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f95ec0612d0 0x7f95ec063790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:48.159 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.155+0000 7f95fb7fe700 1 -- 172.21.15.192:0/1979634013 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1237+0+0 (secure 0 0 0) 0x7f960007ce60 con 0x7f95f4003400 2024-04-12T05:01:48.159 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.156+0000 7f9609d9b700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f95ec0612d0 0x7f95ec063790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:48.159 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.156+0000 7f9609d9b700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f95ec0612d0 0x7f95ec063790 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f95fc006fd0 tx=0x7f95fc008040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:48.162 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.160+0000 7f95fb7fe700 1 -- 172.21.15.192:0/1979634013 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9600051e00 con 0x7f95f4003400 2024-04-12T05:01:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:46.824632+0000 mgr.smithi113.ctsazg (mgr.14194) 10 : cephadm [INF] Deploying daemon node-exporter.smithi192 on smithi192 2024-04-12T05:01:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:46.878319+0000 mon.smithi113 (mon.0) 192 : cluster [DBG] mgrmap e18: smithi113.ctsazg(active, since 23s), standbys: smithi192.ocppyo 2024-04-12T05:01:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:46.878479+0000 mon.smithi113 (mon.0) 193 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi192.ocppyo", "id": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:01:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:47.771859+0000 mon.smithi113 (mon.0) 194 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:48.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:01:46.824632+0000 mgr.smithi113.ctsazg (mgr.14194) 10 : cephadm [INF] Deploying daemon node-exporter.smithi192 on smithi192 2024-04-12T05:01:48.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:46.878319+0000 mon.smithi113 (mon.0) 192 : cluster [DBG] mgrmap e18: smithi113.ctsazg(active, since 23s), standbys: smithi192.ocppyo 2024-04-12T05:01:48.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:46.878479+0000 mon.smithi113 (mon.0) 193 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi192.ocppyo", "id": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:01:48.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:47.771859+0000 mon.smithi113 (mon.0) 194 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:01:48.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.403+0000 7f9610a2b700 1 -- 172.21.15.192:0/1979634013 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f960c046ce0 con 0x7f95f4003400 2024-04-12T05:01:48.406 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.404+0000 7f95fb7fe700 1 -- 172.21.15.192:0/1979634013 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1013 (secure 0 0 0) 0x7f960001b020 con 0x7f95f4003400 2024-04-12T05:01:48.407 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-12T05:01:48.407 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","modified":"2024-04-12T05:01:41.772101Z","created":"2024-04-12T04:59:27.208775Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi113","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi192","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-12T05:01:48.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.405+0000 7f9610a2b700 1 -- 172.21.15.192:0/1979634013 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f95ec0612d0 msgr2=0x7f95ec063790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:48.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.405+0000 7f9610a2b700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f95ec0612d0 0x7f95ec063790 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f95fc006fd0 tx=0x7f95fc008040).stop 2024-04-12T05:01:48.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.405+0000 7f9610a2b700 1 -- 172.21.15.192:0/1979634013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f4003400 msgr2=0x7f960c10c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:48.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.405+0000 7f9610a2b700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f4003400 0x7f960c10c700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9600019040 tx=0x7f9600002040).stop 2024-04-12T05:01:48.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.405+0000 7f9610a2b700 1 -- 172.21.15.192:0/1979634013 shutdown_connections 2024-04-12T05:01:48.408 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.405+0000 7f9610a2b700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f95ec0612d0 0x7f95ec063790 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:48.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.405+0000 7f9610a2b700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f4003400 0x7f960c10c700 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:48.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.405+0000 7f9610a2b700 1 --2- 172.21.15.192:0/1979634013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f960c10cbf0 0x7f960c106690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:48.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.405+0000 7f9610a2b700 1 -- 172.21.15.192:0/1979634013 >> 172.21.15.192:0/1979634013 conn(0x7f960c06c440 msgr2=0x7f960c0f80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:48.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.406+0000 7f9610a2b700 1 -- 172.21.15.192:0/1979634013 shutdown_connections 2024-04-12T05:01:48.409 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:01:48.406+0000 7f9610a2b700 1 -- 172.21.15.192:0/1979634013 wait complete. 2024-04-12T05:01:48.409 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-04-12T05:01:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:47.501870+0000 mgr.smithi113.ctsazg (mgr.14194) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:48.405218+0000 mon.smithi113 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.192:0/1979634013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:01:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:48.648738+0000 mon.smithi113 (mon.0) 196 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:47.501870+0000 mgr.smithi113.ctsazg (mgr.14194) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:48.405218+0000 mon.smithi113 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.192:0/1979634013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T05:01:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:48.648738+0000 mon.smithi113 (mon.0) 196 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:49.426 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-12T05:01:49.427 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph config generate-minimal-conf 2024-04-12T05:01:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:49.502127+0000 mgr.smithi113.ctsazg (mgr.14194) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:50.741648+0000 mon.smithi113 (mon.0) 197 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:51.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:49.502127+0000 mgr.smithi113.ctsazg (mgr.14194) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:51.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:50.741648+0000 mon.smithi113 (mon.0) 197 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:51.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.963+0000 7f2ef6f61700 1 -- 172.21.15.113:0/3576794472 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef00fd620 msgr2=0x7f2ef00fda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:51.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.963+0000 7f2ef6f61700 1 --2- 172.21.15.113:0/3576794472 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef00fd620 0x7f2ef00fda00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f2ee8009ae0 tx=0x7f2ee8004d10).stop 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.963+0000 7f2ef6f61700 1 -- 172.21.15.113:0/3576794472 shutdown_connections 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.963+0000 7f2ef6f61700 1 --2- 172.21.15.113:0/3576794472 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef00fd620 0x7f2ef00fda00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.963+0000 7f2ef6f61700 1 -- 172.21.15.113:0/3576794472 >> 172.21.15.113:0/3576794472 conn(0x7f2ef00f3220 msgr2=0x7f2ef00f5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.963+0000 7f2ef6f61700 1 -- 172.21.15.113:0/3576794472 shutdown_connections 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.963+0000 7f2ef6f61700 1 -- 172.21.15.113:0/3576794472 wait complete. 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2ef6f61700 1 Processor -- start 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2ef6f61700 1 -- start start 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2ef6f61700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ef00fd620 0x7f2ef010da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2ef6f61700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef010def0 0x7f2ef01079e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2ef6f61700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef010e4c0 con 0x7f2ef010def0 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2ef6f61700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef010e600 con 0x7f2ef00fd620 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2eeffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef010def0 0x7f2ef01079e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:51.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2eeffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef010def0 0x7f2ef01079e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39098/0 (socket says 172.21.15.113:39098) 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2eeffff700 1 -- 172.21.15.113:0/2533637995 learned_addr learned my addr 172.21.15.113:0/2533637995 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2eeffff700 1 -- 172.21.15.113:0/2533637995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ef00fd620 msgr2=0x7f2ef010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2eeffff700 1 --2- 172.21.15.113:0/2533637995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ef00fd620 0x7f2ef010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2eeffff700 1 -- 172.21.15.113:0/2533637995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee80097c0 con 0x7f2ef010def0 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2eeffff700 1 --2- 172.21.15.113:0/2533637995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef010def0 0x7f2ef01079e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2ee400b550 tx=0x7f2ee40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.964+0000 7f2eedffb700 1 -- 172.21.15.113:0/2533637995 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ee400f6f0 con 0x7f2ef010def0 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.965+0000 7f2eedffb700 1 -- 172.21.15.113:0/2533637995 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f2ee400fce0 con 0x7f2ef010def0 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.965+0000 7f2ef6f61700 1 -- 172.21.15.113:0/2533637995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef0107f90 con 0x7f2ef010def0 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.965+0000 7f2eedffb700 1 -- 172.21.15.113:0/2533637995 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ee4011ce0 con 0x7f2ef010def0 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.965+0000 7f2ef6f61700 1 -- 172.21.15.113:0/2533637995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef0108550 con 0x7f2ef010def0 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.966+0000 7f2ef6f61700 1 -- 172.21.15.113:0/2533637995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef005f150 con 0x7f2ef010def0 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.966+0000 7f2eedffb700 1 -- 172.21.15.113:0/2533637995 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f2ee400f850 con 0x7f2ef010def0 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.966+0000 7f2eedffb700 1 --2- 172.21.15.113:0/2533637995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ed8061280 0x7f2ed8063740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.966+0000 7f2eedffb700 1 -- 172.21.15.113:0/2533637995 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1237+0+0 (secure 0 0 0) 0x7f2ee407b720 con 0x7f2ef010def0 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.966+0000 7f2ef4cfd700 1 --2- 172.21.15.113:0/2533637995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ed8061280 0x7f2ed8063740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:51.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.967+0000 7f2ef4cfd700 1 --2- 172.21.15.113:0/2533637995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ed8061280 0x7f2ed8063740 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f2ee8004750 tx=0x7f2ee8004d10).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:51.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:51.971+0000 7f2eedffb700 1 -- 172.21.15.113:0/2533637995 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2ee4050670 con 0x7f2ef010def0 2024-04-12T05:01:52.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.131+0000 7f2ef6f61700 1 -- 172.21.15.113:0/2533637995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2ef0101630 con 0x7f2ef010def0 2024-04-12T05:01:52.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.131+0000 7f2eedffb700 1 -- 172.21.15.113:0/2533637995 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f2ee4050260 con 0x7f2ef010def0 2024-04-12T05:01:52.134 INFO:teuthology.orchestra.run.smithi113.stdout:# minimal ceph.conf for 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T05:01:52.134 INFO:teuthology.orchestra.run.smithi113.stdout:[global] 2024-04-12T05:01:52.134 INFO:teuthology.orchestra.run.smithi113.stdout: fsid = 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T05:01:52.134 INFO:teuthology.orchestra.run.smithi113.stdout: mon_host = [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2024-04-12T05:01:52.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.133+0000 7f2ef6f61700 1 -- 172.21.15.113:0/2533637995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ed8061280 msgr2=0x7f2ed8063740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:52.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.133+0000 7f2ef6f61700 1 --2- 172.21.15.113:0/2533637995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ed8061280 0x7f2ed8063740 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f2ee8004750 tx=0x7f2ee8004d10).stop 2024-04-12T05:01:52.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.133+0000 7f2ef6f61700 1 -- 172.21.15.113:0/2533637995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef010def0 msgr2=0x7f2ef01079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:52.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.133+0000 7f2ef6f61700 1 --2- 172.21.15.113:0/2533637995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef010def0 0x7f2ef01079e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2ee400b550 tx=0x7f2ee40090f0).stop 2024-04-12T05:01:52.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.133+0000 7f2ef6f61700 1 -- 172.21.15.113:0/2533637995 shutdown_connections 2024-04-12T05:01:52.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.133+0000 7f2ef6f61700 1 --2- 172.21.15.113:0/2533637995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ed8061280 0x7f2ed8063740 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:52.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.133+0000 7f2ef6f61700 1 --2- 172.21.15.113:0/2533637995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ef00fd620 0x7f2ef010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:01:52.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.133+0000 7f2ef6f61700 1 --2- 172.21.15.113:0/2533637995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ef010def0 0x7f2ef01079e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:52.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.133+0000 7f2ef6f61700 1 -- 172.21.15.113:0/2533637995 >> 172.21.15.113:0/2533637995 conn(0x7f2ef00f3220 msgr2=0x7f2ef00f3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:52.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.133+0000 7f2ef6f61700 1 -- 172.21.15.113:0/2533637995 shutdown_connections 2024-04-12T05:01:52.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:52.133+0000 7f2ef6f61700 1 -- 172.21.15.113:0/2533637995 wait complete. 2024-04-12T05:01:52.159 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:01:50.754136+0000 mgr.smithi113.ctsazg (mgr.14194) 13 : cephadm [INF] Deploying daemon prometheus.smithi192 on smithi192 2024-04-12T05:01:52.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:01:50.754136+0000 mgr.smithi113.ctsazg (mgr.14194) 13 : cephadm [INF] Deploying daemon prometheus.smithi192 on smithi192 2024-04-12T05:01:52.858 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-12T05:01:52.858 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:01:52.858 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.conf 2024-04-12T05:01:52.889 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:01:52.889 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:01:52.956 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T05:01:52.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2024-04-12T05:01:52.994 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T05:01:52.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:01:53.065 INFO:tasks.cephadm:Deploying OSDs... 2024-04-12T05:01:53.065 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:01:53.065 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T05:01:53.083 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T05:01:53.083 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_1 2024-04-12T05:01:53.140 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T05:01:53.141 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:01:53.141 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 165287 Links: 1 2024-04-12T05:01:53.141 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:01:53.141 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:01:53.141 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-12 05:01:14.845932920 +0000 2024-04-12T05:01:53.141 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-12 04:54:47.880916077 +0000 2024-04-12T05:01:53.141 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-12 04:54:47.880916077 +0000 2024-04-12T05:01:53.141 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-04-12T05:01:53.141 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T05:01:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:51.502383+0000 mgr.smithi113.ctsazg (mgr.14194) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:52.132638+0000 mon.smithi113 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.113:0/2533637995' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:01:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:51.502383+0000 mgr.smithi113.ctsazg (mgr.14194) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:52.132638+0000 mon.smithi113 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.113:0/2533637995' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:01:53.199 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-12T05:01:53.199 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-12T05:01:53.199 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000213961 s, 2.4 MB/s 2024-04-12T05:01:53.200 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T05:01:53.259 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_2 2024-04-12T05:01:53.317 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T05:01:53.317 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:01:53.317 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 165335 Links: 1 2024-04-12T05:01:53.317 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:01:53.317 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:01:53.317 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-12 05:01:14.845932920 +0000 2024-04-12T05:01:53.317 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-12 04:54:48.219909100 +0000 2024-04-12T05:01:53.317 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-12 04:54:48.219909100 +0000 2024-04-12T05:01:53.317 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-04-12T05:01:53.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T05:01:53.382 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-12T05:01:53.382 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-12T05:01:53.382 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000225487 s, 2.3 MB/s 2024-04-12T05:01:53.383 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T05:01:53.441 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_3 2024-04-12T05:01:53.499 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T05:01:53.499 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:01:53.499 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 165390 Links: 1 2024-04-12T05:01:53.499 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:01:53.499 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:01:53.499 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-12 05:01:14.846932900 +0000 2024-04-12T05:01:53.499 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-12 04:54:48.561902061 +0000 2024-04-12T05:01:53.499 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-12 04:54:48.561902061 +0000 2024-04-12T05:01:53.500 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-04-12T05:01:53.500 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T05:01:53.565 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-12T05:01:53.565 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-12T05:01:53.565 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000189085 s, 2.7 MB/s 2024-04-12T05:01:53.566 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T05:01:53.625 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_4 2024-04-12T05:01:53.684 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T05:01:53.685 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:01:53.685 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 6h/6d Inode: 165439 Links: 1 2024-04-12T05:01:53.685 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:01:53.685 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:01:53.685 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-04-12 05:01:14.846932900 +0000 2024-04-12T05:01:53.685 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-12 04:54:48.900895084 +0000 2024-04-12T05:01:53.685 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-12 04:54:48.900895084 +0000 2024-04-12T05:01:53.685 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-04-12T05:01:53.685 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T05:01:53.750 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-04-12T05:01:53.751 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-04-12T05:01:53.751 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000218219 s, 2.3 MB/s 2024-04-12T05:01:53.752 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T05:01:53.810 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T05:01:53.811 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T05:01:53.832 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T05:01:53.833 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-04-12T05:01:53.894 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T05:01:53.894 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:01:53.894 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 164464 Links: 1 2024-04-12T05:01:53.894 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:01:53.894 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:01:53.894 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-12 05:01:47.270411846 +0000 2024-04-12T05:01:53.894 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-12 04:54:47.872968629 +0000 2024-04-12T05:01:53.894 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-12 04:54:47.872968629 +0000 2024-04-12T05:01:53.894 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-12T05:01:53.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T05:01:53.964 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-12T05:01:53.964 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-12T05:01:53.964 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000242311 s, 2.1 MB/s 2024-04-12T05:01:53.965 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T05:01:54.026 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-04-12T05:01:54.085 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T05:01:54.085 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:01:54.085 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 166743 Links: 1 2024-04-12T05:01:54.085 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:01:54.086 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:01:54.086 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-12 05:01:47.270411846 +0000 2024-04-12T05:01:54.086 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-12 04:54:48.216961969 +0000 2024-04-12T05:01:54.086 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-12 04:54:48.216961969 +0000 2024-04-12T05:01:54.086 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-12T05:01:54.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T05:01:54.152 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-12T05:01:54.153 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-12T05:01:54.153 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000242628 s, 2.1 MB/s 2024-04-12T05:01:54.154 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T05:01:54.214 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-04-12T05:01:54.274 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T05:01:54.274 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:01:54.274 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 167801 Links: 1 2024-04-12T05:01:54.274 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:01:54.274 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:01:54.274 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-12 05:01:47.270411846 +0000 2024-04-12T05:01:54.274 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-12 04:54:48.544955619 +0000 2024-04-12T05:01:54.274 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-12 04:54:48.544955619 +0000 2024-04-12T05:01:54.274 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-12T05:01:54.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T05:01:54.341 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-12T05:01:54.341 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-12T05:01:54.341 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000163199 s, 3.1 MB/s 2024-04-12T05:01:54.342 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T05:01:54.403 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-04-12T05:01:54.461 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T05:01:54.461 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T05:01:54.461 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 169089 Links: 1 2024-04-12T05:01:54.461 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:01:54.461 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T05:01:54.461 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-12 05:01:47.270411846 +0000 2024-04-12T05:01:54.462 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-12 04:54:48.894948842 +0000 2024-04-12T05:01:54.462 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-12 04:54:48.894948842 +0000 2024-04-12T05:01:54.462 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-12T05:01:54.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T05:01:54.527 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-12T05:01:54.527 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-12T05:01:54.527 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000230622 s, 2.2 MB/s 2024-04-12T05:01:54.528 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T05:01:54.589 INFO:tasks.cephadm:Deploying osd.0 on smithi113 with /dev/vg_nvme/lv_4... 2024-04-12T05:01:54.589 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-12T05:01:54.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:53.502709+0000 mgr.smithi113.ctsazg (mgr.14194) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:54.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:53.652782+0000 mon.smithi113 (mon.0) 199 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:54.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:53.502709+0000 mgr.smithi113.ctsazg (mgr.14194) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:54.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:53.652782+0000 mon.smithi113 (mon.0) 199 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:01:56.887 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-12T05:01:56.887 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-12T05:01:56.887 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-04-12T05:01:56.887 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-04-12T05:01:56.887 INFO:teuthology.orchestra.run.smithi113.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00818217 s, 1.3 GB/s 2024-04-12T05:01:56.887 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-04-12T05:01:56.923 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:55.502969+0000 mgr.smithi113.ctsazg (mgr.14194) 16 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:56.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:55.502969+0000 mgr.smithi113.ctsazg (mgr.14194) 16 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:57.572 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi113:vg_nvme/lv_4 2024-04-12T05:01:58.819 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:01:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:57.503466+0000 mgr.smithi113.ctsazg (mgr.14194) 17 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:58.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:01:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:57.503466+0000 mgr.smithi113.ctsazg (mgr.14194) 17 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:01:59.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.527+0000 7f40a7431700 1 -- 172.21.15.113:0/4107520148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a00617d0 msgr2=0x7f40a00fa460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:59.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.527+0000 7f40a7431700 1 --2- 172.21.15.113:0/4107520148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a00617d0 0x7f40a00fa460 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f40900047f0 tx=0x7f4090009cb0).stop 2024-04-12T05:01:59.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.527+0000 7f40a7431700 1 -- 172.21.15.113:0/4107520148 shutdown_connections 2024-04-12T05:01:59.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.527+0000 7f40a7431700 1 --2- 172.21.15.113:0/4107520148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40a00fa950 0x7f40a00fcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:59.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.527+0000 7f40a7431700 1 --2- 172.21.15.113:0/4107520148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a00617d0 0x7f40a00fa460 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:59.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.527+0000 7f40a7431700 1 -- 172.21.15.113:0/4107520148 >> 172.21.15.113:0/4107520148 conn(0x7f40a00f1ec0 msgr2=0x7f40a00f4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:01:59.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.527+0000 7f40a7431700 1 -- 172.21.15.113:0/4107520148 shutdown_connections 2024-04-12T05:01:59.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.527+0000 7f40a7431700 1 -- 172.21.15.113:0/4107520148 wait complete. 2024-04-12T05:01:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.528+0000 7f40a7431700 1 Processor -- start 2024-04-12T05:01:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.528+0000 7f40a7431700 1 -- start start 2024-04-12T05:01:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.528+0000 7f40a7431700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40a00617d0 0x7f40a00f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.528+0000 7f40a7431700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a00fa950 0x7f40a00f6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.528+0000 7f40a7431700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a00f82b0 con 0x7f40a00fa950 2024-04-12T05:01:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.528+0000 7f40a7431700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a00f6850 con 0x7f40a00617d0 2024-04-12T05:01:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.528+0000 7f40a49cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a00fa950 0x7f40a00f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.528+0000 7f40a49cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a00fa950 0x7f40a00f6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39120/0 (socket says 172.21.15.113:39120) 2024-04-12T05:01:59.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.528+0000 7f40a49cc700 1 -- 172.21.15.113:0/3308138490 learned_addr learned my addr 172.21.15.113:0/3308138490 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.528+0000 7f40a51cd700 1 --2- 172.21.15.113:0/3308138490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40a00617d0 0x7f40a00f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.529+0000 7f40a49cc700 1 -- 172.21.15.113:0/3308138490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40a00617d0 msgr2=0x7f40a00f7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.529+0000 7f40a49cc700 1 --2- 172.21.15.113:0/3308138490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40a00617d0 0x7f40a00f7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.529+0000 7f40a49cc700 1 -- 172.21.15.113:0/3308138490 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4090005040 con 0x7f40a00fa950 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.529+0000 7f40a49cc700 1 --2- 172.21.15.113:0/3308138490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a00fa950 0x7f40a00f6360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f409c00c790 tx=0x7f409c00caa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.529+0000 7f40967fc700 1 -- 172.21.15.113:0/3308138490 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f409c00e680 con 0x7f40a00fa950 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.529+0000 7f40967fc700 1 -- 172.21.15.113:0/3308138490 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f409c00ec70 con 0x7f40a00fa950 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.529+0000 7f40a7431700 1 -- 172.21.15.113:0/3308138490 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a00f6ad0 con 0x7f40a00fa950 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.529+0000 7f40967fc700 1 -- 172.21.15.113:0/3308138490 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f409c012d70 con 0x7f40a00fa950 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.529+0000 7f40a7431700 1 -- 172.21.15.113:0/3308138490 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a00f7010 con 0x7f40a00fa950 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.530+0000 7f40a7431700 1 -- 172.21.15.113:0/3308138490 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a00fffe0 con 0x7f40a00fa950 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.530+0000 7f40967fc700 1 -- 172.21.15.113:0/3308138490 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f409c00e7e0 con 0x7f40a00fa950 2024-04-12T05:01:59.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.530+0000 7f40967fc700 1 --2- 172.21.15.113:0/3308138490 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f408c0611d0 0x7f408c063690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:01:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.531+0000 7f40a51cd700 1 --2- 172.21.15.113:0/3308138490 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f408c0611d0 0x7f408c063690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:01:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.531+0000 7f40967fc700 1 -- 172.21.15.113:0/3308138490 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1237+0+0 (secure 0 0 0) 0x7f409c07b0f0 con 0x7f40a00fa950 2024-04-12T05:01:59.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.531+0000 7f40a51cd700 1 --2- 172.21.15.113:0/3308138490 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f408c0611d0 0x7f408c063690 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f4090004f10 tx=0x7f409000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:01:59.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.535+0000 7f40967fc700 1 -- 172.21.15.113:0/3308138490 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f409c047f60 con 0x7f40a00fa950 2024-04-12T05:01:59.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:01:59.707+0000 7f40a7431700 1 -- 172.21.15.113:0/3308138490 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f40a0059120 con 0x7f408c0611d0 2024-04-12T05:02:00.896 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:01:59.503742+0000 mgr.smithi113.ctsazg (mgr.14194) 18 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:00.896 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:59.709255+0000 mgr.smithi113.ctsazg (mgr.14194) 19 : audit [DBG] from='client.14222 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:02:00.897 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:59.711784+0000 mon.smithi113 (mon.0) 200 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:02:00.897 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:59.713651+0000 mon.smithi113 (mon.0) 201 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:02:00.897 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:01:59.714157+0000 mon.smithi113 (mon.0) 202 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:00.947 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:01:59.503742+0000 mgr.smithi113.ctsazg (mgr.14194) 18 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:00.947 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:59.709255+0000 mgr.smithi113.ctsazg (mgr.14194) 19 : audit [DBG] from='client.14222 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:02:00.947 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:59.711784+0000 mon.smithi113 (mon.0) 200 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:02:00.947 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:59.713651+0000 mon.smithi113 (mon.0) 201 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:02:00.947 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:01:59.714157+0000 mon.smithi113 (mon.0) 202 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:03.355 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:01.504074+0000 mgr.smithi113.ctsazg (mgr.14194) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:03.355 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:01.923383+0000 mon.smithi113 (mon.0) 203 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:03.355 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:01.924272+0000 mon.smithi113 (mon.0) 204 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:02:03.355 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:02.217670+0000 mon.smithi113 (mon.0) 205 : audit [INF] from='client.? 172.21.15.113:0/999043887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc942a17-378a-48a6-b1d3-4f5730550810"}]: dispatch 2024-04-12T05:02:03.355 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:02.225045+0000 mon.smithi113 (mon.0) 206 : audit [INF] from='client.? 172.21.15.113:0/999043887' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc942a17-378a-48a6-b1d3-4f5730550810"}]': finished 2024-04-12T05:02:03.355 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:02.225086+0000 mon.smithi113 (mon.0) 207 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-04-12T05:02:03.355 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:02.225185+0000 mon.smithi113 (mon.0) 208 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:03.355 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:02.793514+0000 mon.smithi113 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.113:0/3198172929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:02:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:01.504074+0000 mgr.smithi113.ctsazg (mgr.14194) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:01.923383+0000 mon.smithi113 (mon.0) 203 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:01.924272+0000 mon.smithi113 (mon.0) 204 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:02:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:02.217670+0000 mon.smithi113 (mon.0) 205 : audit [INF] from='client.? 172.21.15.113:0/999043887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc942a17-378a-48a6-b1d3-4f5730550810"}]: dispatch 2024-04-12T05:02:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:02.225045+0000 mon.smithi113 (mon.0) 206 : audit [INF] from='client.? 172.21.15.113:0/999043887' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc942a17-378a-48a6-b1d3-4f5730550810"}]': finished 2024-04-12T05:02:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:02.225086+0000 mon.smithi113 (mon.0) 207 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-04-12T05:02:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:02.225185+0000 mon.smithi113 (mon.0) 208 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:02.793514+0000 mon.smithi113 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.113:0/3198172929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:02:04.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:03.504360+0000 mgr.smithi113.ctsazg (mgr.14194) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:04.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:03.665232+0000 mon.smithi113 (mon.0) 210 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:04.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:03.504360+0000 mgr.smithi113.ctsazg (mgr.14194) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:04.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:03.665232+0000 mon.smithi113 (mon.0) 210 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:06.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:04.888958+0000 mon.smithi113 (mon.0) 211 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:06.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:04.889538+0000 mon.smithi113 (mon.0) 212 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:06.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:04.890098+0000 mon.smithi113 (mon.0) 213 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:02:06.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:04.890839+0000 mgr.smithi113.ctsazg (mgr.14194) 22 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.conf 2024-04-12T05:02:06.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:04.891087+0000 mgr.smithi113.ctsazg (mgr.14194) 23 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-04-12T05:02:06.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:04.912929+0000 mon.smithi113 (mon.0) 214 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:06.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:04.920955+0000 mon.smithi113 (mon.0) 215 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:06.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:04.926294+0000 mon.smithi113 (mon.0) 216 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:06.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:04.928753+0000 mgr.smithi113.ctsazg (mgr.14194) 24 : cephadm [INF] Reconfiguring mon.smithi113 (unknown last config time)... 2024-04-12T05:02:06.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:04.928945+0000 mon.smithi113 (mon.0) 217 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:02:06.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:04.929453+0000 mon.smithi113 (mon.0) 218 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:02:06.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:04.929934+0000 mon.smithi113 (mon.0) 219 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:06.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:04.930385+0000 mgr.smithi113.ctsazg (mgr.14194) 25 : cephadm [INF] Reconfiguring daemon mon.smithi113 on smithi113 2024-04-12T05:02:06.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:04.888958+0000 mon.smithi113 (mon.0) 211 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:06.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:04.889538+0000 mon.smithi113 (mon.0) 212 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:06.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:04.890098+0000 mon.smithi113 (mon.0) 213 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:02:06.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:04.890839+0000 mgr.smithi113.ctsazg (mgr.14194) 22 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.conf 2024-04-12T05:02:06.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:04.891087+0000 mgr.smithi113.ctsazg (mgr.14194) 23 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-04-12T05:02:06.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:04.912929+0000 mon.smithi113 (mon.0) 214 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:06.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:04.920955+0000 mon.smithi113 (mon.0) 215 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:06.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:04.926294+0000 mon.smithi113 (mon.0) 216 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:06.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:04.928753+0000 mgr.smithi113.ctsazg (mgr.14194) 24 : cephadm [INF] Reconfiguring mon.smithi113 (unknown last config time)... 2024-04-12T05:02:06.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:04.928945+0000 mon.smithi113 (mon.0) 217 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:02:06.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:04.929453+0000 mon.smithi113 (mon.0) 218 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:02:06.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:04.929934+0000 mon.smithi113 (mon.0) 219 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:06.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:04.930385+0000 mgr.smithi113.ctsazg (mgr.14194) 25 : cephadm [INF] Reconfiguring daemon mon.smithi113 on smithi113 2024-04-12T05:02:07.009 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:05.504606+0000 mgr.smithi113.ctsazg (mgr.14194) 26 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:05.504606+0000 mgr.smithi113.ctsazg (mgr.14194) 26 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:07.504994+0000 mgr.smithi113.ctsazg (mgr.14194) 27 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:07.971635+0000 mon.smithi113 (mon.0) 220 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:07.972095+0000 mgr.smithi113.ctsazg (mgr.14194) 28 : cephadm [INF] Reconfiguring mgr.smithi113.ctsazg (unknown last config time)... 2024-04-12T05:02:09.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:07.972372+0000 mon.smithi113 (mon.0) 221 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.ctsazg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:02:09.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:07.973043+0000 mon.smithi113 (mon.0) 222 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:02:09.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:07.973550+0000 mon.smithi113 (mon.0) 223 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:09.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:07.974036+0000 mgr.smithi113.ctsazg (mgr.14194) 29 : cephadm [INF] Reconfiguring daemon mgr.smithi113.ctsazg on smithi113 2024-04-12T05:02:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:08 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:07.504994+0000 mgr.smithi113.ctsazg (mgr.14194) 27 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:08 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:07.971635+0000 mon.smithi113 (mon.0) 220 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:08 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:07.972095+0000 mgr.smithi113.ctsazg (mgr.14194) 28 : cephadm [INF] Reconfiguring mgr.smithi113.ctsazg (unknown last config time)... 2024-04-12T05:02:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:08 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:07.972372+0000 mon.smithi113 (mon.0) 221 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.ctsazg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:02:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:08 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:07.973043+0000 mon.smithi113 (mon.0) 222 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:02:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:08 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:07.973550+0000 mon.smithi113 (mon.0) 223 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:09.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:08 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:07.974036+0000 mgr.smithi113.ctsazg (mgr.14194) 29 : cephadm [INF] Reconfiguring daemon mgr.smithi113.ctsazg on smithi113 2024-04-12T05:02:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:09.505240+0000 mgr.smithi113.ctsazg (mgr.14194) 30 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:09.862032+0000 mon.smithi113 (mon.0) 224 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:09.862596+0000 mgr.smithi113.ctsazg (mgr.14194) 31 : cephadm [INF] Reconfiguring alertmanager.smithi113 (dependencies changed)... 2024-04-12T05:02:11.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:09.868387+0000 mgr.smithi113.ctsazg (mgr.14194) 32 : cephadm [INF] Reconfiguring daemon alertmanager.smithi113 on smithi113 2024-04-12T05:02:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:09.505240+0000 mgr.smithi113.ctsazg (mgr.14194) 30 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:09.862032+0000 mon.smithi113 (mon.0) 224 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:09.862596+0000 mgr.smithi113.ctsazg (mgr.14194) 31 : cephadm [INF] Reconfiguring alertmanager.smithi113 (dependencies changed)... 2024-04-12T05:02:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:09.868387+0000 mgr.smithi113.ctsazg (mgr.14194) 32 : cephadm [INF] Reconfiguring daemon alertmanager.smithi113 on smithi113 2024-04-12T05:02:12.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:11.605394+0000 mon.smithi113 (mon.0) 225 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:02:12.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:11.606036+0000 mon.smithi113 (mon.0) 226 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:11.605394+0000 mon.smithi113 (mon.0) 225 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:02:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:11.606036+0000 mon.smithi113 (mon.0) 226 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:13.164 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:11.505475+0000 mgr.smithi113.ctsazg (mgr.14194) 33 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:13.164 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:11.606540+0000 mgr.smithi113.ctsazg (mgr.14194) 34 : cephadm [INF] Deploying daemon osd.0 on smithi113 2024-04-12T05:02:13.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:11.505475+0000 mgr.smithi113.ctsazg (mgr.14194) 33 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:13.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:11.606540+0000 mgr.smithi113.ctsazg (mgr.14194) 34 : cephadm [INF] Deploying daemon osd.0 on smithi113 2024-04-12T05:02:15.032 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:13.505764+0000 mgr.smithi113.ctsazg (mgr.14194) 35 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:15.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:13.505764+0000 mgr.smithi113.ctsazg (mgr.14194) 35 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:17.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:15.506025+0000 mgr.smithi113.ctsazg (mgr.14194) 36 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:17.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:15.506025+0000 mgr.smithi113.ctsazg (mgr.14194) 36 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:18.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:16.938034+0000 mon.smithi113 (mon.0) 227 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:18.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:16.938583+0000 mgr.smithi113.ctsazg (mgr.14194) 37 : cephadm [INF] Reconfiguring crash.smithi113 (monmap changed)... 2024-04-12T05:02:18.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:16.938861+0000 mon.smithi113 (mon.0) 228 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:02:18.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:16.939527+0000 mon.smithi113 (mon.0) 229 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:18.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:16.940018+0000 mgr.smithi113.ctsazg (mgr.14194) 38 : cephadm [INF] Reconfiguring daemon crash.smithi113 on smithi113 2024-04-12T05:02:18.366 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:16.938034+0000 mon.smithi113 (mon.0) 227 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:18.366 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:16.938583+0000 mgr.smithi113.ctsazg (mgr.14194) 37 : cephadm [INF] Reconfiguring crash.smithi113 (monmap changed)... 2024-04-12T05:02:18.366 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:16.938861+0000 mon.smithi113 (mon.0) 228 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:02:18.366 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:16.939527+0000 mon.smithi113 (mon.0) 229 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:18.366 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:16.940018+0000 mgr.smithi113.ctsazg (mgr.14194) 38 : cephadm [INF] Reconfiguring daemon crash.smithi113 on smithi113 2024-04-12T05:02:19.141 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:17.506254+0000 mgr.smithi113.ctsazg (mgr.14194) 39 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:19.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:17.506254+0000 mgr.smithi113.ctsazg (mgr.14194) 39 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:21.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:19.506489+0000 mgr.smithi113.ctsazg (mgr.14194) 40 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:21.400 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:19.506489+0000 mgr.smithi113.ctsazg (mgr.14194) 40 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:23.156 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:21.506728+0000 mgr.smithi113.ctsazg (mgr.14194) 41 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:23.156 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:22.580947+0000 mon.smithi113 (mon.0) 230 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:21.506728+0000 mgr.smithi113.ctsazg (mgr.14194) 41 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:22.580947+0000 mon.smithi113 (mon.0) 230 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:25.147 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:23.506961+0000 mgr.smithi113.ctsazg (mgr.14194) 42 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:25.147 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:24.372651+0000 mon.smithi113 (mon.0) 231 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:25.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:23.506961+0000 mgr.smithi113.ctsazg (mgr.14194) 42 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:25.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:24.372651+0000 mon.smithi113 (mon.0) 231 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:26.172 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:24.373182+0000 mgr.smithi113.ctsazg (mgr.14194) 43 : cephadm [INF] Reconfiguring grafana.smithi113 (dependencies changed)... 2024-04-12T05:02:26.172 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:24.403656+0000 mgr.smithi113.ctsazg (mgr.14194) 44 : cephadm [INF] Reconfiguring daemon grafana.smithi113 on smithi113 2024-04-12T05:02:26.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:24.373182+0000 mgr.smithi113.ctsazg (mgr.14194) 43 : cephadm [INF] Reconfiguring grafana.smithi113 (dependencies changed)... 2024-04-12T05:02:26.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:24.403656+0000 mgr.smithi113.ctsazg (mgr.14194) 44 : cephadm [INF] Reconfiguring daemon grafana.smithi113 on smithi113 2024-04-12T05:02:27.157 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:25.507237+0000 mgr.smithi113.ctsazg (mgr.14194) 45 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:27.158 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:26.516516+0000 mon.smithi113 (mon.0) 232 : audit [INF] from='osd.0 [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T05:02:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:25.507237+0000 mgr.smithi113.ctsazg (mgr.14194) 45 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:26.516516+0000 mon.smithi113 (mon.0) 232 : audit [INF] from='osd.0 [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T05:02:28.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:26.959750+0000 mon.smithi113 (mon.0) 233 : audit [INF] from='osd.0 [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T05:02:28.121 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:26.959834+0000 mon.smithi113 (mon.0) 234 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-04-12T05:02:28.121 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:26.960186+0000 mon.smithi113 (mon.0) 235 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:28.121 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:26.960475+0000 mon.smithi113 (mon.0) 236 : audit [INF] from='osd.0 [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:02:28.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:26.959750+0000 mon.smithi113 (mon.0) 233 : audit [INF] from='osd.0 [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T05:02:28.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:26.959834+0000 mon.smithi113 (mon.0) 234 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-04-12T05:02:28.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:26.960186+0000 mon.smithi113 (mon.0) 235 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:28.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:26.960475+0000 mon.smithi113 (mon.0) 236 : audit [INF] from='osd.0 [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:02:29.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:27.507521+0000 mgr.smithi113.ctsazg (mgr.14194) 46 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:29.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:27.960653+0000 mon.smithi113 (mon.0) 237 : audit [INF] from='osd.0 [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-12T05:02:29.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:27.960698+0000 mon.smithi113 (mon.0) 238 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-04-12T05:02:29.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:27.960800+0000 mon.smithi113 (mon.0) 239 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:29.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:27.964292+0000 mon.smithi113 (mon.0) 240 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:29.237 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:28.964039+0000 mon.smithi113 (mon.0) 241 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:27.507521+0000 mgr.smithi113.ctsazg (mgr.14194) 46 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:27.960653+0000 mon.smithi113 (mon.0) 237 : audit [INF] from='osd.0 [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-12T05:02:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:27.960698+0000 mon.smithi113 (mon.0) 238 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-04-12T05:02:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:27.960800+0000 mon.smithi113 (mon.0) 239 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:27.964292+0000 mon.smithi113 (mon.0) 240 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:29.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:28.964039+0000 mon.smithi113 (mon.0) 241 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:30.406 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:27.514439+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:02:30.407 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:27.514511+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:02:30.407 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:28.967869+0000 mon.smithi113 (mon.0) 242 : cluster [INF] osd.0 [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970] boot 2024-04-12T05:02:30.407 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:28.967893+0000 mon.smithi113 (mon.0) 243 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-04-12T05:02:30.407 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:28.968158+0000 mon.smithi113 (mon.0) 244 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:30.407 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:29.296676+0000 mon.smithi113 (mon.0) 245 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:30.407 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:29.297242+0000 mgr.smithi113.ctsazg (mgr.14194) 47 : cephadm [INF] Reconfiguring crash.smithi192 (monmap changed)... 2024-04-12T05:02:30.407 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:29.297499+0000 mon.smithi113 (mon.0) 246 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:02:30.407 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:29.298180+0000 mon.smithi113 (mon.0) 247 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:30.407 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:29.298669+0000 mgr.smithi113.ctsazg (mgr.14194) 48 : cephadm [INF] Reconfiguring daemon crash.smithi192 on smithi192 2024-04-12T05:02:30.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:27.514439+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:02:30.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:27.514511+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:02:30.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:28.967869+0000 mon.smithi113 (mon.0) 242 : cluster [INF] osd.0 [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970] boot 2024-04-12T05:02:30.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:28.967893+0000 mon.smithi113 (mon.0) 243 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-04-12T05:02:30.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:28.968158+0000 mon.smithi113 (mon.0) 244 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:02:30.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:29.296676+0000 mon.smithi113 (mon.0) 245 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:30.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:29.297242+0000 mgr.smithi113.ctsazg (mgr.14194) 47 : cephadm [INF] Reconfiguring crash.smithi192 (monmap changed)... 2024-04-12T05:02:30.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:29.297499+0000 mon.smithi113 (mon.0) 246 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:02:30.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:29.298180+0000 mon.smithi113 (mon.0) 247 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:30.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:29.298669+0000 mgr.smithi113.ctsazg (mgr.14194) 48 : cephadm [INF] Reconfiguring daemon crash.smithi192 on smithi192 2024-04-12T05:02:30.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.701+0000 7f40967fc700 1 -- 172.21.15.113:0/3308138490 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f40a0059120 con 0x7f408c0611d0 2024-04-12T05:02:30.704 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 0 on host 'smithi113' 2024-04-12T05:02:30.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.703+0000 7f40a7431700 1 -- 172.21.15.113:0/3308138490 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f408c0611d0 msgr2=0x7f408c063690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:02:30.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.703+0000 7f40a7431700 1 --2- 172.21.15.113:0/3308138490 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f408c0611d0 0x7f408c063690 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f4090004f10 tx=0x7f409000b040).stop 2024-04-12T05:02:30.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.703+0000 7f40a7431700 1 -- 172.21.15.113:0/3308138490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a00fa950 msgr2=0x7f40a00f6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:02:30.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.703+0000 7f40a7431700 1 --2- 172.21.15.113:0/3308138490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a00fa950 0x7f40a00f6360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f409c00c790 tx=0x7f409c00caa0).stop 2024-04-12T05:02:30.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.703+0000 7f40a7431700 1 -- 172.21.15.113:0/3308138490 shutdown_connections 2024-04-12T05:02:30.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.703+0000 7f40a7431700 1 --2- 172.21.15.113:0/3308138490 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f408c0611d0 0x7f408c063690 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:02:30.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.703+0000 7f40a7431700 1 --2- 172.21.15.113:0/3308138490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40a00617d0 0x7f40a00f7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:02:30.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.703+0000 7f40a7431700 1 --2- 172.21.15.113:0/3308138490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40a00fa950 0x7f40a00f6360 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:02:30.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.703+0000 7f40a7431700 1 -- 172.21.15.113:0/3308138490 >> 172.21.15.113:0/3308138490 conn(0x7f40a00f1ec0 msgr2=0x7f40a00f4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:02:30.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.703+0000 7f40a7431700 1 -- 172.21.15.113:0/3308138490 shutdown_connections 2024-04-12T05:02:30.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:30.703+0000 7f40a7431700 1 -- 172.21.15.113:0/3308138490 wait complete. 2024-04-12T05:02:31.388 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:29.507867+0000 mgr.smithi113.ctsazg (mgr.14194) 49 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:31.388 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:30.303216+0000 mon.smithi113 (mon.0) 248 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2024-04-12T05:02:31.388 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:30.701530+0000 mon.smithi113 (mon.0) 249 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:31.412 DEBUG:teuthology.orchestra.run.smithi113:osd.0> sudo journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.0.service 2024-04-12T05:02:31.414 INFO:tasks.cephadm:Deploying osd.1 on smithi113 with /dev/vg_nvme/lv_3... 2024-04-12T05:02:31.414 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-12T05:02:31.446 INFO:journalctl@ceph.osd.0.smithi113.stdout:-- Logs begin at Fri 2024-04-12 04:49:01 UTC. -- 2024-04-12T05:02:31.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:29.507867+0000 mgr.smithi113.ctsazg (mgr.14194) 49 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T05:02:31.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:30.303216+0000 mon.smithi113 (mon.0) 248 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2024-04-12T05:02:31.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:30.701530+0000 mon.smithi113 (mon.0) 249 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:33.672 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:31.508217+0000 mgr.smithi113.ctsazg (mgr.14194) 50 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:33.672 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:32.252705+0000 mon.smithi113 (mon.0) 250 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:33.672 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:32.253238+0000 mgr.smithi113.ctsazg (mgr.14194) 51 : cephadm [INF] Reconfiguring mgr.smithi192.ocppyo (monmap changed)... 2024-04-12T05:02:33.672 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:32.253470+0000 mon.smithi113 (mon.0) 251 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.ocppyo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:02:33.672 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:32.254080+0000 mon.smithi113 (mon.0) 252 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:02:33.672 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:32.254529+0000 mon.smithi113 (mon.0) 253 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:33.672 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:32.254967+0000 mgr.smithi113.ctsazg (mgr.14194) 52 : cephadm [INF] Reconfiguring daemon mgr.smithi192.ocppyo on smithi192 2024-04-12T05:02:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:31.508217+0000 mgr.smithi113.ctsazg (mgr.14194) 50 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:32.252705+0000 mon.smithi113 (mon.0) 250 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:32.253238+0000 mgr.smithi113.ctsazg (mgr.14194) 51 : cephadm [INF] Reconfiguring mgr.smithi192.ocppyo (monmap changed)... 2024-04-12T05:02:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:32.253470+0000 mon.smithi113 (mon.0) 251 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.ocppyo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:02:33.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:32.254080+0000 mon.smithi113 (mon.0) 252 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:02:33.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:32.254529+0000 mon.smithi113 (mon.0) 253 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:33.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:32.254967+0000 mgr.smithi113.ctsazg (mgr.14194) 52 : cephadm [INF] Reconfiguring daemon mgr.smithi192.ocppyo on smithi192 2024-04-12T05:02:33.790 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-12T05:02:33.791 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-12T05:02:33.791 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-04-12T05:02:33.791 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-04-12T05:02:33.791 INFO:teuthology.orchestra.run.smithi113.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00901036 s, 1.2 GB/s 2024-04-12T05:02:33.791 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-04-12T05:02:34.476 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi113:vg_nvme/lv_3 2024-04-12T05:02:34.995 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:33.508499+0000 mgr.smithi113.ctsazg (mgr.14194) 53 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:34.995 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:33.728236+0000 mon.smithi113 (mon.0) 254 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:34.996 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:33.728833+0000 mgr.smithi113.ctsazg (mgr.14194) 54 : cephadm [INF] Reconfiguring mon.smithi192 (monmap changed)... 2024-04-12T05:02:34.996 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:33.729047+0000 mon.smithi113 (mon.0) 255 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:02:34.996 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:33.729553+0000 mon.smithi113 (mon.0) 256 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:02:34.996 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:33.730025+0000 mon.smithi113 (mon.0) 257 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:34.996 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:33.730510+0000 mgr.smithi113.ctsazg (mgr.14194) 55 : cephadm [INF] Reconfiguring daemon mon.smithi192 on smithi192 2024-04-12T05:02:35.050 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:33.508499+0000 mgr.smithi113.ctsazg (mgr.14194) 53 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:35.050 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:33.728236+0000 mon.smithi113 (mon.0) 254 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:35.050 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:33.728833+0000 mgr.smithi113.ctsazg (mgr.14194) 54 : cephadm [INF] Reconfiguring mon.smithi192 (monmap changed)... 2024-04-12T05:02:35.050 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:33.729047+0000 mon.smithi113 (mon.0) 255 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:02:35.051 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:33.729553+0000 mon.smithi113 (mon.0) 256 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:02:35.051 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:33.730025+0000 mon.smithi113 (mon.0) 257 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:35.051 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:33.730510+0000 mgr.smithi113.ctsazg (mgr.14194) 55 : cephadm [INF] Reconfiguring daemon mon.smithi192 on smithi192 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.034837+0000 mon.smithi113 (mon.0) 258 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.037756+0000 mon.smithi113 (mon.0) 259 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.038084+0000 mgr.smithi113.ctsazg (mgr.14194) 56 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.038851+0000 mon.smithi113 (mon.0) 260 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi113.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.039071+0000 mgr.smithi113.ctsazg (mgr.14194) 57 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi113.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.042517+0000 mon.smithi113 (mon.0) 261 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.047798+0000 mon.smithi113 (mon.0) 262 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.048277+0000 mgr.smithi113.ctsazg (mgr.14194) 58 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.053124+0000 mon.smithi113 (mon.0) 263 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi113.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.053525+0000 mgr.smithi113.ctsazg (mgr.14194) 59 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi113.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.057273+0000 mon.smithi113 (mon.0) 264 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.061672+0000 mon.smithi113 (mon.0) 265 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.062046+0000 mgr.smithi113.ctsazg (mgr.14194) 60 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:02:36.120 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.065235+0000 mon.smithi113 (mon.0) 266 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T05:02:36.121 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.065531+0000 mgr.smithi113.ctsazg (mgr.14194) 61 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T05:02:36.121 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.069954+0000 mon.smithi113 (mon.0) 267 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:36.121 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:35.072661+0000 mon.smithi113 (mon.0) 268 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:02:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.367+0000 7fa41d3c6700 1 -- 172.21.15.113:0/193137846 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f66d0 msgr2=0x7fa4180f6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:02:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.367+0000 7fa41d3c6700 1 --2- 172.21.15.113:0/193137846 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f66d0 0x7fa4180f6af0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fa40c004730 tx=0x7fa40c009cb0).stop 2024-04-12T05:02:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.367+0000 7fa41d3c6700 1 -- 172.21.15.113:0/193137846 shutdown_connections 2024-04-12T05:02:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.367+0000 7fa41d3c6700 1 --2- 172.21.15.113:0/193137846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f6fe0 0x7fa4180f7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:02:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.367+0000 7fa41d3c6700 1 --2- 172.21.15.113:0/193137846 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f66d0 0x7fa4180f6af0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:02:36.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.367+0000 7fa41d3c6700 1 -- 172.21.15.113:0/193137846 >> 172.21.15.113:0/193137846 conn(0x7fa4180f1eb0 msgr2=0x7fa4180f42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:02:36.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.367+0000 7fa41d3c6700 1 -- 172.21.15.113:0/193137846 shutdown_connections 2024-04-12T05:02:36.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.367+0000 7fa41d3c6700 1 -- 172.21.15.113:0/193137846 wait complete. 2024-04-12T05:02:36.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.367+0000 7fa41d3c6700 1 Processor -- start 2024-04-12T05:02:36.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa41d3c6700 1 -- start start 2024-04-12T05:02:36.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa41d3c6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f66d0 0x7fa418108080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:02:36.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa41d3c6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f6fe0 0x7fa418108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:02:36.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa41d3c6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa418108b20 con 0x7fa4180f66d0 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa41d3c6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa418108c60 con 0x7fa4180f6fe0 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa416ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f66d0 0x7fa418108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa416ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f66d0 0x7fa418108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56220/0 (socket says 172.21.15.113:56220) 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa416ffd700 1 -- 172.21.15.113:0/2846456512 learned_addr learned my addr 172.21.15.113:0/2846456512 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa4167fc700 1 --2- 172.21.15.113:0/2846456512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f6fe0 0x7fa418108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa416ffd700 1 -- 172.21.15.113:0/2846456512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f6fe0 msgr2=0x7fa418108570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa416ffd700 1 --2- 172.21.15.113:0/2846456512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f6fe0 0x7fa418108570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa416ffd700 1 -- 172.21.15.113:0/2846456512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40c005040 con 0x7fa4180f66d0 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa416ffd700 1 --2- 172.21.15.113:0/2846456512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f66d0 0x7fa418108080 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fa40c0099f0 tx=0x7fa40c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.368+0000 7fa3fffff700 1 -- 172.21.15.113:0/2846456512 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa40c019070 con 0x7fa4180f66d0 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.369+0000 7fa41d3c6700 1 -- 172.21.15.113:0/2846456512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41810d6a0 con 0x7fa4180f66d0 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.369+0000 7fa3fffff700 1 -- 172.21.15.113:0/2846456512 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fa40c0097b0 con 0x7fa4180f66d0 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.369+0000 7fa3fffff700 1 -- 172.21.15.113:0/2846456512 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa40c014ca0 con 0x7fa4180f66d0 2024-04-12T05:02:36.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.369+0000 7fa41d3c6700 1 -- 172.21.15.113:0/2846456512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41810dc60 con 0x7fa4180f66d0 2024-04-12T05:02:36.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.370+0000 7fa41d3c6700 1 -- 172.21.15.113:0/2846456512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4180feed0 con 0x7fa4180f66d0 2024-04-12T05:02:36.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.370+0000 7fa3fffff700 1 -- 172.21.15.113:0/2846456512 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fa40c0144b0 con 0x7fa4180f66d0 2024-04-12T05:02:36.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.370+0000 7fa3fffff700 1 --2- 172.21.15.113:0/2846456512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa400061180 0x7fa400063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:02:36.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.370+0000 7fa4167fc700 1 --2- 172.21.15.113:0/2846456512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa400061180 0x7fa400063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:02:36.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.370+0000 7fa3fffff700 1 -- 172.21.15.113:0/2846456512 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1849+0+0 (secure 0 0 0) 0x7fa40c07c8a0 con 0x7fa4180f66d0 2024-04-12T05:02:36.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.371+0000 7fa4167fc700 1 --2- 172.21.15.113:0/2846456512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa400061180 0x7fa400063640 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fa408009a00 tx=0x7fa40800b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:02:36.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.375+0000 7fa3fffff700 1 -- 172.21.15.113:0/2846456512 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa40c051620 con 0x7fa4180f66d0 2024-04-12T05:02:36.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.034837+0000 mon.smithi113 (mon.0) 258 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:36.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.037756+0000 mon.smithi113 (mon.0) 259 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:02:36.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.038084+0000 mgr.smithi113.ctsazg (mgr.14194) 56 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:02:36.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.038851+0000 mon.smithi113 (mon.0) 260 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi113.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T05:02:36.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.039071+0000 mgr.smithi113.ctsazg (mgr.14194) 57 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi113.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T05:02:36.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.042517+0000 mon.smithi113 (mon.0) 261 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:36.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.047798+0000 mon.smithi113 (mon.0) 262 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:02:36.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.048277+0000 mgr.smithi113.ctsazg (mgr.14194) 58 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:02:36.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.053124+0000 mon.smithi113 (mon.0) 263 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi113.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T05:02:36.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.053525+0000 mgr.smithi113.ctsazg (mgr.14194) 59 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi113.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T05:02:36.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.057273+0000 mon.smithi113 (mon.0) 264 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:36.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.061672+0000 mon.smithi113 (mon.0) 265 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:02:36.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.062046+0000 mgr.smithi113.ctsazg (mgr.14194) 60 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:02:36.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.065235+0000 mon.smithi113 (mon.0) 266 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T05:02:36.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.065531+0000 mgr.smithi113.ctsazg (mgr.14194) 61 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T05:02:36.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.069954+0000 mon.smithi113 (mon.0) 267 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:36.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:35.072661+0000 mon.smithi113 (mon.0) 268 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:02:36.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:02:36.549+0000 7fa41d3c6700 1 -- 172.21.15.113:0/2846456512 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa41810e2a0 con 0x7fa400061180 2024-04-12T05:02:37.416 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:35.508762+0000 mgr.smithi113.ctsazg (mgr.14194) 62 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:37.416 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:36.553121+0000 mon.smithi113 (mon.0) 269 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:02:37.416 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:36.554901+0000 mon.smithi113 (mon.0) 270 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:02:37.416 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:36.555381+0000 mon.smithi113 (mon.0) 271 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:35.508762+0000 mgr.smithi113.ctsazg (mgr.14194) 62 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:36.553121+0000 mon.smithi113 (mon.0) 269 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:02:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:36.554901+0000 mon.smithi113 (mon.0) 270 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:02:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:36.555381+0000 mon.smithi113 (mon.0) 271 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:38.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:36.551139+0000 mgr.smithi113.ctsazg (mgr.14194) 63 : audit [DBG] from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:02:38.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:37.894713+0000 mon.smithi113 (mon.0) 272 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:38.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:36.551139+0000 mgr.smithi113.ctsazg (mgr.14194) 63 : audit [DBG] from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:02:38.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:37.894713+0000 mon.smithi113 (mon.0) 272 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:39.320 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:37.509177+0000 mgr.smithi113.ctsazg (mgr.14194) 64 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:37.509177+0000 mgr.smithi113.ctsazg (mgr.14194) 64 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:39.509502+0000 mgr.smithi113.ctsazg (mgr.14194) 65 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:40.295207+0000 mon.smithi113 (mon.0) 273 : audit [INF] from='client.? 172.21.15.113:0/1907854618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39fec899-371b-4d48-99a8-52f90f52a71f"}]: dispatch 2024-04-12T05:02:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:40.298903+0000 mon.smithi113 (mon.0) 274 : audit [INF] from='client.? 172.21.15.113:0/1907854618' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39fec899-371b-4d48-99a8-52f90f52a71f"}]': finished 2024-04-12T05:02:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:40.298942+0000 mon.smithi113 (mon.0) 275 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-04-12T05:02:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:40.299025+0000 mon.smithi113 (mon.0) 276 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:02:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:40.952865+0000 mon.smithi113 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.113:0/618401014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:02:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:39.509502+0000 mgr.smithi113.ctsazg (mgr.14194) 65 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:40.295207+0000 mon.smithi113 (mon.0) 273 : audit [INF] from='client.? 172.21.15.113:0/1907854618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39fec899-371b-4d48-99a8-52f90f52a71f"}]: dispatch 2024-04-12T05:02:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:40.298903+0000 mon.smithi113 (mon.0) 274 : audit [INF] from='client.? 172.21.15.113:0/1907854618' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39fec899-371b-4d48-99a8-52f90f52a71f"}]': finished 2024-04-12T05:02:41.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:40.298942+0000 mon.smithi113 (mon.0) 275 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-04-12T05:02:41.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:40.299025+0000 mon.smithi113 (mon.0) 276 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:02:41.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:40.952865+0000 mon.smithi113 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.113:0/618401014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:02:42.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:42 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:41.105604+0000 mon.smithi113 (mon.0) 278 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:42.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:42 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:41.105604+0000 mon.smithi113 (mon.0) 278 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:41.509721+0000 mgr.smithi113.ctsazg (mgr.14194) 66 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:43.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:41.509721+0000 mgr.smithi113.ctsazg (mgr.14194) 66 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:45.284 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:43.510063+0000 mgr.smithi113.ctsazg (mgr.14194) 67 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:43.510063+0000 mgr.smithi113.ctsazg (mgr.14194) 67 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:45.510328+0000 mgr.smithi113.ctsazg (mgr.14194) 68 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:45.510328+0000 mgr.smithi113.ctsazg (mgr.14194) 68 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:49.068 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:47.510649+0000 mgr.smithi113.ctsazg (mgr.14194) 69 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:49.068 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:47.854893+0000 mgr.smithi113.ctsazg (mgr.14194) 70 : cephadm [INF] Detected new or changed devices on smithi113 2024-04-12T05:02:49.068 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:47.860306+0000 mon.smithi113 (mon.0) 279 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:49.068 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:48.161084+0000 mon.smithi113 (mon.0) 280 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:02:49.069 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:48.161566+0000 mgr.smithi113.ctsazg (mgr.14194) 71 : cephadm [INF] Adjusting osd_memory_target on smithi113 to 13922M 2024-04-12T05:02:49.069 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:48.165391+0000 mon.smithi113 (mon.0) 281 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:49.069 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:48.166236+0000 mon.smithi113 (mon.0) 282 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:49.069 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:48.166756+0000 mon.smithi113 (mon.0) 283 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:02:49.069 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:48.172638+0000 mon.smithi113 (mon.0) 284 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:49.069 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:48.176032+0000 mon.smithi113 (mon.0) 285 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:02:49.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:47.510649+0000 mgr.smithi113.ctsazg (mgr.14194) 69 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:49.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:47.854893+0000 mgr.smithi113.ctsazg (mgr.14194) 70 : cephadm [INF] Detected new or changed devices on smithi113 2024-04-12T05:02:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:47.860306+0000 mon.smithi113 (mon.0) 279 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:48.161084+0000 mon.smithi113 (mon.0) 280 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:02:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:48.161566+0000 mgr.smithi113.ctsazg (mgr.14194) 71 : cephadm [INF] Adjusting osd_memory_target on smithi113 to 13922M 2024-04-12T05:02:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:48.165391+0000 mon.smithi113 (mon.0) 281 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:48.166236+0000 mon.smithi113 (mon.0) 282 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:48.166756+0000 mon.smithi113 (mon.0) 283 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:02:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:48.172638+0000 mon.smithi113 (mon.0) 284 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:48.176032+0000 mon.smithi113 (mon.0) 285 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:02:50.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:49.088134+0000 mon.smithi113 (mon.0) 286 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:02:50.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:49.088847+0000 mon.smithi113 (mon.0) 287 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:50.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:02:49.089363+0000 mgr.smithi113.ctsazg (mgr.14194) 72 : cephadm [INF] Deploying daemon osd.1 on smithi113 2024-04-12T05:02:50.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:49.088134+0000 mon.smithi113 (mon.0) 286 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:02:50.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:49.088847+0000 mon.smithi113 (mon.0) 287 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:50.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:02:49.089363+0000 mgr.smithi113.ctsazg (mgr.14194) 72 : cephadm [INF] Deploying daemon osd.1 on smithi113 2024-04-12T05:02:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:49.510929+0000 mgr.smithi113.ctsazg (mgr.14194) 73 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:51.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:50.697668+0000 mon.smithi113 (mon.0) 288 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:51.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:49.510929+0000 mgr.smithi113.ctsazg (mgr.14194) 73 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:51.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:50.697668+0000 mon.smithi113 (mon.0) 288 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:51.511179+0000 mgr.smithi113.ctsazg (mgr.14194) 74 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:51.511179+0000 mgr.smithi113.ctsazg (mgr.14194) 74 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:55.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:53.511456+0000 mgr.smithi113.ctsazg (mgr.14194) 75 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:55.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:53.511456+0000 mgr.smithi113.ctsazg (mgr.14194) 75 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:57.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:55.511916+0000 mgr.smithi113.ctsazg (mgr.14194) 76 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:57.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:55.889806+0000 mon.smithi113 (mon.0) 289 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:57.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:55.890850+0000 mon.smithi113 (mon.0) 290 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:57.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:55.891536+0000 mon.smithi113 (mon.0) 291 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:02:57.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:55.895804+0000 mon.smithi113 (mon.0) 292 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:57.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:56.453385+0000 mon.smithi113 (mon.0) 293 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:57.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:56.463984+0000 mon.smithi113 (mon.0) 294 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:02:57.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:55.511916+0000 mgr.smithi113.ctsazg (mgr.14194) 76 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:57.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:55.889806+0000 mon.smithi113 (mon.0) 289 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:57.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:55.890850+0000 mon.smithi113 (mon.0) 290 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:02:57.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:55.891536+0000 mon.smithi113 (mon.0) 291 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:02:57.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:55.895804+0000 mon.smithi113 (mon.0) 292 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:57.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:56.453385+0000 mon.smithi113 (mon.0) 293 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:02:57.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:56.463984+0000 mon.smithi113 (mon.0) 294 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:02:59.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:02:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:57.512224+0000 mgr.smithi113.ctsazg (mgr.14194) 77 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:02:59.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:02:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:57.512224+0000 mgr.smithi113.ctsazg (mgr.14194) 77 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:03:00.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:02:59.095118+0000 mon.smithi113 (mon.0) 295 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:00.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:02:59.095118+0000 mon.smithi113 (mon.0) 295 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:00.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.492+0000 7fa3fffff700 1 -- 172.21.15.113:0/2846456512 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa41810e2a0 con 0x7fa400061180 2024-04-12T05:03:00.494 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 1 on host 'smithi113' 2024-04-12T05:03:00.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.494+0000 7fa41d3c6700 1 -- 172.21.15.113:0/2846456512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa400061180 msgr2=0x7fa400063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:03:00.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.494+0000 7fa41d3c6700 1 --2- 172.21.15.113:0/2846456512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa400061180 0x7fa400063640 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fa408009a00 tx=0x7fa40800b040).stop 2024-04-12T05:03:00.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.494+0000 7fa41d3c6700 1 -- 172.21.15.113:0/2846456512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f66d0 msgr2=0x7fa418108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:03:00.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.494+0000 7fa41d3c6700 1 --2- 172.21.15.113:0/2846456512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f66d0 0x7fa418108080 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fa40c0099f0 tx=0x7fa40c00b040).stop 2024-04-12T05:03:00.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.494+0000 7fa41d3c6700 1 -- 172.21.15.113:0/2846456512 shutdown_connections 2024-04-12T05:03:00.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.494+0000 7fa41d3c6700 1 --2- 172.21.15.113:0/2846456512 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa400061180 0x7fa400063640 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:00.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.494+0000 7fa41d3c6700 1 --2- 172.21.15.113:0/2846456512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f66d0 0x7fa418108080 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:00.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.494+0000 7fa41d3c6700 1 --2- 172.21.15.113:0/2846456512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f6fe0 0x7fa418108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:00.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.494+0000 7fa41d3c6700 1 -- 172.21.15.113:0/2846456512 >> 172.21.15.113:0/2846456512 conn(0x7fa4180f1eb0 msgr2=0x7fa418061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:03:00.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.494+0000 7fa41d3c6700 1 -- 172.21.15.113:0/2846456512 shutdown_connections 2024-04-12T05:03:00.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:00.494+0000 7fa41d3c6700 1 -- 172.21.15.113:0/2846456512 wait complete. 2024-04-12T05:03:01.398 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:02:59.512730+0000 mgr.smithi113.ctsazg (mgr.14194) 78 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:03:01.398 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:00.325477+0000 mon.smithi113 (mon.0) 296 : audit [INF] from='osd.1 [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T05:03:01.398 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:00.492675+0000 mon.smithi113 (mon.0) 297 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:01.420 DEBUG:teuthology.orchestra.run.smithi113:osd.1> sudo journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.1.service 2024-04-12T05:03:01.423 INFO:tasks.cephadm:Deploying osd.2 on smithi113 with /dev/vg_nvme/lv_2... 2024-04-12T05:03:01.423 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-12T05:03:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:02:59.512730+0000 mgr.smithi113.ctsazg (mgr.14194) 78 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:03:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:00.325477+0000 mon.smithi113 (mon.0) 296 : audit [INF] from='osd.1 [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T05:03:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:00.492675+0000 mon.smithi113 (mon.0) 297 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:01.452 INFO:journalctl@ceph.osd.1.smithi113.stdout:-- Logs begin at Fri 2024-04-12 04:49:01 UTC. -- 2024-04-12T05:03:02.353 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:01.101804+0000 mon.smithi113 (mon.0) 298 : audit [INF] from='osd.1 [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T05:03:02.353 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:01.101892+0000 mon.smithi113 (mon.0) 299 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-04-12T05:03:02.353 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:01.102196+0000 mon.smithi113 (mon.0) 300 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:03:02.354 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:01.102425+0000 mon.smithi113 (mon.0) 301 : audit [INF] from='osd.1 [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:03:02.354 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:02.102617+0000 mon.smithi113 (mon.0) 302 : audit [INF] from='osd.1 [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-12T05:03:02.354 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:02.102676+0000 mon.smithi113 (mon.0) 303 : cluster [DBG] osdmap e13: 2 total, 1 up, 2 in 2024-04-12T05:03:02.354 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:02.102790+0000 mon.smithi113 (mon.0) 304 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:03:02.354 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:03:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[58952]: debug 2024-04-12T05:03:02.105+0000 7feb4610d700 -1 osd.1 0 waiting for initial osdmap 2024-04-12T05:03:02.354 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:03:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[58952]: debug 2024-04-12T05:03:02.110+0000 7feb42889700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:03:02.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:01.101804+0000 mon.smithi113 (mon.0) 298 : audit [INF] from='osd.1 [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T05:03:02.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:01.101892+0000 mon.smithi113 (mon.0) 299 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-04-12T05:03:02.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:01.102196+0000 mon.smithi113 (mon.0) 300 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:03:02.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:01.102425+0000 mon.smithi113 (mon.0) 301 : audit [INF] from='osd.1 [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:03:02.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:02.102617+0000 mon.smithi113 (mon.0) 302 : audit [INF] from='osd.1 [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-12T05:03:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:02.102676+0000 mon.smithi113 (mon.0) 303 : cluster [DBG] osdmap e13: 2 total, 1 up, 2 in 2024-04-12T05:03:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:02.102790+0000 mon.smithi113 (mon.0) 304 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:03:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:01.513026+0000 mgr.smithi113.ctsazg (mgr.14194) 79 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:03:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:02.106582+0000 mon.smithi113 (mon.0) 305 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:03:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:03.106020+0000 mon.smithi113 (mon.0) 306 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:03:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:01.513026+0000 mgr.smithi113.ctsazg (mgr.14194) 79 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:03:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:02.106582+0000 mon.smithi113 (mon.0) 305 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:03:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:03.106020+0000 mon.smithi113 (mon.0) 306 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:03:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:01.358863+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:03:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:01.358928+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:03:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:03.110188+0000 mon.smithi113 (mon.0) 307 : cluster [INF] osd.1 [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368] boot 2024-04-12T05:03:04.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:03.110212+0000 mon.smithi113 (mon.0) 308 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-04-12T05:03:04.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:03.110689+0000 mon.smithi113 (mon.0) 309 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:03:04.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:03.833122+0000 mon.smithi113 (mon.0) 310 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:04.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:01.358863+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:03:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:01.358928+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:03:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:03.110188+0000 mon.smithi113 (mon.0) 307 : cluster [INF] osd.1 [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368] boot 2024-04-12T05:03:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:03.110212+0000 mon.smithi113 (mon.0) 308 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-04-12T05:03:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:03.110689+0000 mon.smithi113 (mon.0) 309 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:03:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:03.833122+0000 mon.smithi113 (mon.0) 310 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:04.629 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-12T05:03:04.630 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-12T05:03:04.630 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-04-12T05:03:04.630 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-04-12T05:03:04.630 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00969456 s, 1.1 GB/s 2024-04-12T05:03:04.630 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-04-12T05:03:05.387 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:03.513419+0000 mgr.smithi113.ctsazg (mgr.14194) 80 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:03:05.387 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:04.125414+0000 mon.smithi113 (mon.0) 311 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-04-12T05:03:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:03.513419+0000 mgr.smithi113.ctsazg (mgr.14194) 80 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-12T05:03:05.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:04.125414+0000 mon.smithi113 (mon.0) 311 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-04-12T05:03:06.745 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi113:vg_nvme/lv_2 2024-04-12T05:03:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:05.513828+0000 mgr.smithi113.ctsazg (mgr.14194) 81 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:05.513828+0000 mgr.smithi113.ctsazg (mgr.14194) 81 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:09.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.008+0000 7f47c3ea3700 1 -- 172.21.15.113:0/429832989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47bc0611c0 msgr2=0x7f47bc0fa490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:03:09.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.008+0000 7f47c3ea3700 1 --2- 172.21.15.113:0/429832989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47bc0611c0 0x7f47bc0fa490 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f47b80047f0 tx=0x7f47b8009cb0).stop 2024-04-12T05:03:09.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 -- 172.21.15.113:0/429832989 shutdown_connections 2024-04-12T05:03:09.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 --2- 172.21.15.113:0/429832989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47bc0fa980 0x7f47bc0fcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:09.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 --2- 172.21.15.113:0/429832989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47bc0611c0 0x7f47bc0fa490 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:09.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 -- 172.21.15.113:0/429832989 >> 172.21.15.113:0/429832989 conn(0x7f47bc0f1e50 msgr2=0x7f47bc0f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:03:09.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 -- 172.21.15.113:0/429832989 shutdown_connections 2024-04-12T05:03:09.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 -- 172.21.15.113:0/429832989 wait complete. 2024-04-12T05:03:09.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 Processor -- start 2024-04-12T05:03:09.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 -- start start 2024-04-12T05:03:09.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47bc0611c0 0x7f47bc0f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:03:09.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47bc0fa980 0x7f47bc0f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:03:09.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc0f82e0 con 0x7f47bc0fa980 2024-04-12T05:03:09.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.009+0000 7f47c3ea3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc0f6880 con 0x7f47bc0611c0 2024-04-12T05:03:09.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47c143e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47bc0fa980 0x7f47bc0f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:03:09.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47c143e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47bc0fa980 0x7f47bc0f6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41138/0 (socket says 172.21.15.113:41138) 2024-04-12T05:03:09.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47c143e700 1 -- 172.21.15.113:0/3408799480 learned_addr learned my addr 172.21.15.113:0/3408799480 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:03:09.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47c1c3f700 1 --2- 172.21.15.113:0/3408799480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47bc0611c0 0x7f47bc0f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:03:09.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47c143e700 1 -- 172.21.15.113:0/3408799480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47bc0611c0 msgr2=0x7f47bc0f7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:03:09.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47c143e700 1 --2- 172.21.15.113:0/3408799480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47bc0611c0 0x7f47bc0f7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:09.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47c143e700 1 -- 172.21.15.113:0/3408799480 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b8005040 con 0x7f47bc0fa980 2024-04-12T05:03:09.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47c143e700 1 --2- 172.21.15.113:0/3408799480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47bc0fa980 0x7f47bc0f6390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f47ac002750 tx=0x7f47ac009ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:03:09.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47b2ffd700 1 -- 172.21.15.113:0/3408799480 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47ac007b40 con 0x7f47bc0fa980 2024-04-12T05:03:09.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47b2ffd700 1 -- 172.21.15.113:0/3408799480 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f47ac00f040 con 0x7f47bc0fa980 2024-04-12T05:03:09.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47c3ea3700 1 -- 172.21.15.113:0/3408799480 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc0f6b00 con 0x7f47bc0fa980 2024-04-12T05:03:09.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47b2ffd700 1 -- 172.21.15.113:0/3408799480 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47ac014870 con 0x7f47bc0fa980 2024-04-12T05:03:09.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.010+0000 7f47c3ea3700 1 -- 172.21.15.113:0/3408799480 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc0f7040 con 0x7f47bc0fa980 2024-04-12T05:03:09.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.011+0000 7f47c3ea3700 1 -- 172.21.15.113:0/3408799480 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc100000 con 0x7f47bc0fa980 2024-04-12T05:03:09.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.013+0000 7f47b2ffd700 1 -- 172.21.15.113:0/3408799480 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f47ac014af0 con 0x7f47bc0fa980 2024-04-12T05:03:09.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.013+0000 7f47b2ffd700 1 --2- 172.21.15.113:0/3408799480 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f47a8061180 0x7f47a8063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:03:09.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.013+0000 7f47b2ffd700 1 -- 172.21.15.113:0/3408799480 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2281+0+0 (secure 0 0 0) 0x7f47ac07be50 con 0x7f47bc0fa980 2024-04-12T05:03:09.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.013+0000 7f47c1c3f700 1 --2- 172.21.15.113:0/3408799480 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f47a8061180 0x7f47a8063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:03:09.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.013+0000 7f47c1c3f700 1 --2- 172.21.15.113:0/3408799480 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f47a8061180 0x7f47a8063640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f47b80047c0 tx=0x7f47b800b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:03:09.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.017+0000 7f47b2ffd700 1 -- 172.21.15.113:0/3408799480 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f47ac050a70 con 0x7f47bc0fa980 2024-04-12T05:03:09.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:09.201+0000 7f47c3ea3700 1 -- 172.21.15.113:0/3408799480 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f47bc059120 con 0x7f47a8061180 2024-04-12T05:03:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:07.514213+0000 mgr.smithi113.ctsazg (mgr.14194) 82 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:08.069531+0000 mon.smithi113 (mon.0) 312 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:08.070375+0000 mon.smithi113 (mon.0) 313 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:03:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:08.070980+0000 mon.smithi113 (mon.0) 314 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:03:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:03:08.071392+0000 mgr.smithi113.ctsazg (mgr.14194) 83 : cephadm [INF] Adjusting osd_memory_target on smithi113 to 6961M 2024-04-12T05:03:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:08.074880+0000 mon.smithi113 (mon.0) 315 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:08.075596+0000 mon.smithi113 (mon.0) 316 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:08.076155+0000 mon.smithi113 (mon.0) 317 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:03:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:08.080612+0000 mon.smithi113 (mon.0) 318 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:08.084012+0000 mon.smithi113 (mon.0) 319 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:03:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:07.514213+0000 mgr.smithi113.ctsazg (mgr.14194) 82 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:08.069531+0000 mon.smithi113 (mon.0) 312 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:08.070375+0000 mon.smithi113 (mon.0) 313 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:03:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:08.070980+0000 mon.smithi113 (mon.0) 314 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:03:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:03:08.071392+0000 mgr.smithi113.ctsazg (mgr.14194) 83 : cephadm [INF] Adjusting osd_memory_target on smithi113 to 6961M 2024-04-12T05:03:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:08.074880+0000 mon.smithi113 (mon.0) 315 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:09.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:08.075596+0000 mon.smithi113 (mon.0) 316 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:09.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:08.076155+0000 mon.smithi113 (mon.0) 317 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:03:09.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:08.080612+0000 mon.smithi113 (mon.0) 318 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:09.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:08.084012+0000 mon.smithi113 (mon.0) 319 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:03:10.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:09.202778+0000 mgr.smithi113.ctsazg (mgr.14194) 84 : audit [DBG] from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:03:10.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:09.204766+0000 mon.smithi113 (mon.0) 320 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:03:10.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:09.206631+0000 mon.smithi113 (mon.0) 321 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:03:10.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:09.207107+0000 mon.smithi113 (mon.0) 322 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:10.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:09.202778+0000 mgr.smithi113.ctsazg (mgr.14194) 84 : audit [DBG] from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:03:10.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:09.204766+0000 mon.smithi113 (mon.0) 320 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:03:10.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:09.206631+0000 mon.smithi113 (mon.0) 321 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:03:10.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:09.207107+0000 mon.smithi113 (mon.0) 322 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:11.307 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:09.514652+0000 mgr.smithi113.ctsazg (mgr.14194) 85 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:11.308 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:10.713951+0000 mon.smithi113 (mon.0) 323 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:09.514652+0000 mgr.smithi113.ctsazg (mgr.14194) 85 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:10.713951+0000 mon.smithi113 (mon.0) 323 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:11.515073+0000 mgr.smithi113.ctsazg (mgr.14194) 86 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:12.344335+0000 mon.smithi113 (mon.0) 324 : audit [INF] from='client.? 172.21.15.113:0/780463579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b135f3a9-d2f0-4737-8a30-9f6dc3f2571b"}]: dispatch 2024-04-12T05:03:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:12.349049+0000 mon.smithi113 (mon.0) 325 : audit [INF] from='client.? 172.21.15.113:0/780463579' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b135f3a9-d2f0-4737-8a30-9f6dc3f2571b"}]': finished 2024-04-12T05:03:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:12.349110+0000 mon.smithi113 (mon.0) 326 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-04-12T05:03:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:12.349219+0000 mon.smithi113 (mon.0) 327 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:13.059396+0000 mon.smithi113 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.113:0/2257840257' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:03:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:11.515073+0000 mgr.smithi113.ctsazg (mgr.14194) 86 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:12.344335+0000 mon.smithi113 (mon.0) 324 : audit [INF] from='client.? 172.21.15.113:0/780463579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b135f3a9-d2f0-4737-8a30-9f6dc3f2571b"}]: dispatch 2024-04-12T05:03:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:12.349049+0000 mon.smithi113 (mon.0) 325 : audit [INF] from='client.? 172.21.15.113:0/780463579' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b135f3a9-d2f0-4737-8a30-9f6dc3f2571b"}]': finished 2024-04-12T05:03:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:12.349110+0000 mon.smithi113 (mon.0) 326 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-04-12T05:03:13.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:12.349219+0000 mon.smithi113 (mon.0) 327 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:13.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:13.059396+0000 mon.smithi113 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.113:0/2257840257' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:03:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:13.515389+0000 mgr.smithi113.ctsazg (mgr.14194) 87 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:14.076276+0000 mon.smithi113 (mon.0) 329 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:14.076966+0000 mon.smithi113 (mon.0) 330 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:14.079058+0000 mon.smithi113 (mon.0) 331 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:03:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:14.085935+0000 mon.smithi113 (mon.0) 332 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:13.515389+0000 mgr.smithi113.ctsazg (mgr.14194) 87 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:14.076276+0000 mon.smithi113 (mon.0) 329 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:14.076966+0000 mon.smithi113 (mon.0) 330 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:14.079058+0000 mon.smithi113 (mon.0) 331 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:03:15.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:14.085935+0000 mon.smithi113 (mon.0) 332 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:15.515662+0000 mgr.smithi113.ctsazg (mgr.14194) 88 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:17.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:15.515662+0000 mgr.smithi113.ctsazg (mgr.14194) 88 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:17.515926+0000 mgr.smithi113.ctsazg (mgr.14194) 89 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:17.515926+0000 mgr.smithi113.ctsazg (mgr.14194) 89 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:20.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:19.947194+0000 mon.smithi113 (mon.0) 333 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:03:20.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:19.947882+0000 mon.smithi113 (mon.0) 334 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:19.947194+0000 mon.smithi113 (mon.0) 333 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:03:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:19.947882+0000 mon.smithi113 (mon.0) 334 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:21.405 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:19.516156+0000 mgr.smithi113.ctsazg (mgr.14194) 90 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:21.405 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:03:19.948450+0000 mgr.smithi113.ctsazg (mgr.14194) 91 : cephadm [INF] Deploying daemon osd.2 on smithi113 2024-04-12T05:03:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:19.516156+0000 mgr.smithi113.ctsazg (mgr.14194) 90 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:03:19.948450+0000 mgr.smithi113.ctsazg (mgr.14194) 91 : cephadm [INF] Deploying daemon osd.2 on smithi113 2024-04-12T05:03:23.384 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:21.516523+0000 mgr.smithi113.ctsazg (mgr.14194) 92 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:21.516523+0000 mgr.smithi113.ctsazg (mgr.14194) 92 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:25.318 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:23.516772+0000 mgr.smithi113.ctsazg (mgr.14194) 93 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:23.516772+0000 mgr.smithi113.ctsazg (mgr.14194) 93 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:26.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:25.361436+0000 mon.smithi113 (mon.0) 335 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:26.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:25.375345+0000 mon.smithi113 (mon.0) 336 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:03:26.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:25.516997+0000 mgr.smithi113.ctsazg (mgr.14194) 94 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:26.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:25.361436+0000 mon.smithi113 (mon.0) 335 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:26.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:25.375345+0000 mon.smithi113 (mon.0) 336 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:03:26.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:25.516997+0000 mgr.smithi113.ctsazg (mgr.14194) 94 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:28.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:27.517240+0000 mgr.smithi113.ctsazg (mgr.14194) 95 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:28.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:27.517240+0000 mgr.smithi113.ctsazg (mgr.14194) 95 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:29.375 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 2 on host 'smithi113' 2024-04-12T05:03:29.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.373+0000 7f47b2ffd700 1 -- 172.21.15.113:0/3408799480 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f47bc059120 con 0x7f47a8061180 2024-04-12T05:03:29.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.375+0000 7f47c3ea3700 1 -- 172.21.15.113:0/3408799480 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f47a8061180 msgr2=0x7f47a8063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:03:29.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.375+0000 7f47c3ea3700 1 --2- 172.21.15.113:0/3408799480 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f47a8061180 0x7f47a8063640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f47b80047c0 tx=0x7f47b800b040).stop 2024-04-12T05:03:29.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.375+0000 7f47c3ea3700 1 -- 172.21.15.113:0/3408799480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47bc0fa980 msgr2=0x7f47bc0f6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:03:29.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.375+0000 7f47c3ea3700 1 --2- 172.21.15.113:0/3408799480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47bc0fa980 0x7f47bc0f6390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f47ac002750 tx=0x7f47ac009ec0).stop 2024-04-12T05:03:29.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.375+0000 7f47c3ea3700 1 -- 172.21.15.113:0/3408799480 shutdown_connections 2024-04-12T05:03:29.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.375+0000 7f47c3ea3700 1 --2- 172.21.15.113:0/3408799480 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f47a8061180 0x7f47a8063640 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:29.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.375+0000 7f47c3ea3700 1 --2- 172.21.15.113:0/3408799480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47bc0611c0 0x7f47bc0f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:29.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.375+0000 7f47c3ea3700 1 --2- 172.21.15.113:0/3408799480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47bc0fa980 0x7f47bc0f6390 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:29.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.375+0000 7f47c3ea3700 1 -- 172.21.15.113:0/3408799480 >> 172.21.15.113:0/3408799480 conn(0x7f47bc0f1e50 msgr2=0x7f47bc0f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:03:29.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.375+0000 7f47c3ea3700 1 -- 172.21.15.113:0/3408799480 shutdown_connections 2024-04-12T05:03:29.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:03:29.375+0000 7f47c3ea3700 1 -- 172.21.15.113:0/3408799480 wait complete. 2024-04-12T05:03:29.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:29.169672+0000 mon.smithi113 (mon.0) 337 : audit [INF] from='osd.2 [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T05:03:29.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:29.373024+0000 mon.smithi113 (mon.0) 338 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:29.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:29.169672+0000 mon.smithi113 (mon.0) 337 : audit [INF] from='osd.2 [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T05:03:29.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:29.373024+0000 mon.smithi113 (mon.0) 338 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:30.328 DEBUG:teuthology.orchestra.run.smithi113:osd.2> sudo journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.2.service 2024-04-12T05:03:30.330 INFO:tasks.cephadm:Deploying osd.3 on smithi192 with /dev/vg_nvme/lv_4... 2024-04-12T05:03:30.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-12T05:03:30.359 INFO:journalctl@ceph.osd.2.smithi113.stdout:-- Logs begin at Fri 2024-04-12 04:49:01 UTC. -- 2024-04-12T05:03:30.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:29.517512+0000 mgr.smithi113.ctsazg (mgr.14194) 96 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:30.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:29.576108+0000 mon.smithi113 (mon.0) 339 : audit [INF] from='osd.2 [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T05:03:30.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:29.576155+0000 mon.smithi113 (mon.0) 340 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-04-12T05:03:30.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:29.576243+0000 mon.smithi113 (mon.0) 341 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:30.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:29.576548+0000 mon.smithi113 (mon.0) 342 : audit [INF] from='osd.2 [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:03:30.924 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:03:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[64924]: debug 2024-04-12T05:03:30.581+0000 7f67a7fd3700 -1 osd.2 0 waiting for initial osdmap 2024-04-12T05:03:30.924 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:03:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[64924]: debug 2024-04-12T05:03:30.586+0000 7f67a1f4a700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:03:30.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:29.517512+0000 mgr.smithi113.ctsazg (mgr.14194) 96 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:30.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:29.576108+0000 mon.smithi113 (mon.0) 339 : audit [INF] from='osd.2 [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T05:03:30.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:29.576155+0000 mon.smithi113 (mon.0) 340 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-04-12T05:03:30.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:29.576243+0000 mon.smithi113 (mon.0) 341 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:30.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:29.576548+0000 mon.smithi113 (mon.0) 342 : audit [INF] from='osd.2 [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:03:32.039 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:30.577875+0000 mon.smithi113 (mon.0) 343 : audit [INF] from='osd.2 [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-12T05:03:32.040 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:30.577951+0000 mon.smithi113 (mon.0) 344 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-04-12T05:03:32.040 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:30.578330+0000 mon.smithi113 (mon.0) 345 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:32.040 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:30.582147+0000 mon.smithi113 (mon.0) 346 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:32.040 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:31.581842+0000 mon.smithi113 (mon.0) 347 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:30.577875+0000 mon.smithi113 (mon.0) 343 : audit [INF] from='osd.2 [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-04-12T05:03:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:30.577951+0000 mon.smithi113 (mon.0) 344 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-04-12T05:03:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:30.578330+0000 mon.smithi113 (mon.0) 345 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:30.582147+0000 mon.smithi113 (mon.0) 346 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:31.581842+0000 mon.smithi113 (mon.0) 347 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:32.604 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-12T05:03:32.604 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-12T05:03:32.604 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-04-12T05:03:32.604 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-04-12T05:03:32.604 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110296 s, 951 MB/s 2024-04-12T05:03:32.605 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-04-12T05:03:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:30.202936+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:03:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:30.203052+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:03:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:31.517789+0000 mgr.smithi113.ctsazg (mgr.14194) 97 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:31.588994+0000 mon.smithi113 (mon.0) 348 : cluster [INF] osd.2 [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711] boot 2024-04-12T05:03:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:31.589034+0000 mon.smithi113 (mon.0) 349 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-04-12T05:03:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:31.589603+0000 mon.smithi113 (mon.0) 350 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:31.589916+0000 mon.smithi113 (mon.0) 351 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-12T05:03:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:31.846186+0000 mon.smithi113 (mon.0) 352 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:32.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:32.589185+0000 mon.smithi113 (mon.0) 353 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-12T05:03:32.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:32.589232+0000 mon.smithi113 (mon.0) 354 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-04-12T05:03:32.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:32.589876+0000 mon.smithi113 (mon.0) 355 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-04-12T05:03:32.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:30.202936+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:03:32.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:30.203052+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:03:32.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:31.517789+0000 mgr.smithi113.ctsazg (mgr.14194) 97 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-12T05:03:32.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:31.588994+0000 mon.smithi113 (mon.0) 348 : cluster [INF] osd.2 [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711] boot 2024-04-12T05:03:32.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:31.589034+0000 mon.smithi113 (mon.0) 349 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-04-12T05:03:32.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:31.589603+0000 mon.smithi113 (mon.0) 350 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:03:32.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:31.589916+0000 mon.smithi113 (mon.0) 351 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-12T05:03:32.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:31.846186+0000 mon.smithi113 (mon.0) 352 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:32.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:32.589185+0000 mon.smithi113 (mon.0) 353 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-12T05:03:32.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:32.589232+0000 mon.smithi113 (mon.0) 354 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-04-12T05:03:32.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:32.589876+0000 mon.smithi113 (mon.0) 355 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-04-12T05:03:33.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2024-04-12T05:03:34.839 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:33.518105+0000 mgr.smithi113.ctsazg (mgr.14194) 98 : cluster [DBG] pgmap v73: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:34.839 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:33.590760+0000 mon.smithi113 (mon.0) 356 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-04-12T05:03:34.839 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:33.590830+0000 mon.smithi113 (mon.0) 357 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-04-12T05:03:34.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:33.518105+0000 mgr.smithi113.ctsazg (mgr.14194) 98 : cluster [DBG] pgmap v73: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:34.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:33.590760+0000 mon.smithi113 (mon.0) 356 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-04-12T05:03:34.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:33.590830+0000 mon.smithi113 (mon.0) 357 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-04-12T05:03:35.085 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.082+0000 7f2af414e700 1 -- 172.21.15.192:0/897931503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2aec0f7a10 msgr2=0x7f2aec0f7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:03:35.085 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.082+0000 7f2af414e700 1 --2- 172.21.15.192:0/897931503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2aec0f7a10 0x7f2aec0f7e30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f2adc0047f0 tx=0x7f2adc009cb0).stop 2024-04-12T05:03:35.086 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.082+0000 7f2af414e700 1 -- 172.21.15.192:0/897931503 shutdown_connections 2024-04-12T05:03:35.086 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.082+0000 7f2af414e700 1 --2- 172.21.15.192:0/897931503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2aec0f8cb0 0x7f2aec0f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:35.086 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.082+0000 7f2af414e700 1 --2- 172.21.15.192:0/897931503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2aec0f7a10 0x7f2aec0f7e30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:35.086 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.082+0000 7f2af414e700 1 -- 172.21.15.192:0/897931503 >> 172.21.15.192:0/897931503 conn(0x7f2aec0f3150 msgr2=0x7f2aec0f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:03:35.086 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.082+0000 7f2af414e700 1 -- 172.21.15.192:0/897931503 shutdown_connections 2024-04-12T05:03:35.086 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.082+0000 7f2af414e700 1 -- 172.21.15.192:0/897931503 wait complete. 2024-04-12T05:03:35.086 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af414e700 1 Processor -- start 2024-04-12T05:03:35.086 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af414e700 1 -- start start 2024-04-12T05:03:35.086 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af414e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2aec0f7a10 0x7f2aec1081e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:03:35.086 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af414e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2aec0f8cb0 0x7f2aec1086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:03:35.086 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af414e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec108c80 con 0x7f2aec0f7a10 2024-04-12T05:03:35.087 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af414e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec108dc0 con 0x7f2aec0f8cb0 2024-04-12T05:03:35.087 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af16e9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2aec0f8cb0 0x7f2aec1086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:03:35.087 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af16e9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2aec0f8cb0 0x7f2aec1086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:59874/0 (socket says 172.21.15.192:59874) 2024-04-12T05:03:35.087 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af16e9700 1 -- 172.21.15.192:0/2265403987 learned_addr learned my addr 172.21.15.192:0/2265403987 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:03:35.087 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af1eea700 1 --2- 172.21.15.192:0/2265403987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2aec0f7a10 0x7f2aec1081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:03:35.087 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af16e9700 1 -- 172.21.15.192:0/2265403987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2aec0f7a10 msgr2=0x7f2aec1081e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:03:35.087 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af16e9700 1 --2- 172.21.15.192:0/2265403987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2aec0f7a10 0x7f2aec1081e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:35.087 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af16e9700 1 -- 172.21.15.192:0/2265403987 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2adc005040 con 0x7f2aec0f8cb0 2024-04-12T05:03:35.087 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af16e9700 1 --2- 172.21.15.192:0/2265403987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2aec0f8cb0 0x7f2aec1086d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f2ae80041d0 tx=0x7f2ae8009f60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:03:35.087 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2ae2ffd700 1 -- 172.21.15.192:0/2265403987 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ae8007b40 con 0x7f2aec0f8cb0 2024-04-12T05:03:35.087 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2ae2ffd700 1 -- 172.21.15.192:0/2265403987 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f2ae800f040 con 0x7f2aec0f8cb0 2024-04-12T05:03:35.088 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2af414e700 1 -- 172.21.15.192:0/2265403987 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aec10d860 con 0x7f2aec0f8cb0 2024-04-12T05:03:35.088 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.083+0000 7f2ae2ffd700 1 -- 172.21.15.192:0/2265403987 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ae800d760 con 0x7f2aec0f8cb0 2024-04-12T05:03:35.088 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.084+0000 7f2af414e700 1 -- 172.21.15.192:0/2265403987 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aec10dda0 con 0x7f2aec0f8cb0 2024-04-12T05:03:35.089 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.084+0000 7f2af414e700 1 -- 172.21.15.192:0/2265403987 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aec05fb50 con 0x7f2aec0f8cb0 2024-04-12T05:03:35.089 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.085+0000 7f2ae2ffd700 1 -- 172.21.15.192:0/2265403987 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f2ae8009110 con 0x7f2aec0f8cb0 2024-04-12T05:03:35.089 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.085+0000 7f2ae2ffd700 1 --2- 172.21.15.192:0/2265403987 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ad8061180 0x7f2ad8063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:03:35.089 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.085+0000 7f2ae2ffd700 1 -- 172.21.15.192:0/2265403987 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3129+0+0 (secure 0 0 0) 0x7f2ae807bb00 con 0x7f2aec0f8cb0 2024-04-12T05:03:35.089 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.085+0000 7f2af1eea700 1 --2- 172.21.15.192:0/2265403987 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ad8061180 0x7f2ad8063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:03:35.089 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.086+0000 7f2af1eea700 1 --2- 172.21.15.192:0/2265403987 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ad8061180 0x7f2ad8063640 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f2adc013470 tx=0x7f2adc00b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:03:35.091 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.090+0000 7f2ae2ffd700 1 -- 172.21.15.192:0/2265403987 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2ae8050350 con 0x7f2aec0f8cb0 2024-04-12T05:03:35.259 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:35.258+0000 7f2af414e700 1 -- 172.21.15.192:0/2265403987 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2aec1021d0 con 0x7f2ad8061180 2024-04-12T05:03:35.956 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:34.847567+0000 mon.smithi113 (mon.0) 358 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.259315+0000 mgr.smithi113.ctsazg (mgr.14194) 99 : audit [DBG] from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.261584+0000 mon.smithi113 (mon.0) 359 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.263464+0000 mon.smithi113 (mon.0) 360 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.263957+0000 mon.smithi113 (mon.0) 361 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.335772+0000 mon.smithi113 (mon.0) 362 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.336734+0000 mon.smithi113 (mon.0) 363 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.337313+0000 mon.smithi113 (mon.0) 364 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.337807+0000 mon.smithi113 (mon.0) 365 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.341617+0000 mon.smithi113 (mon.0) 366 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.342493+0000 mon.smithi113 (mon.0) 367 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.343116+0000 mon.smithi113 (mon.0) 368 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.347025+0000 mon.smithi113 (mon.0) 369 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.350260+0000 mon.smithi113 (mon.0) 370 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:03:35.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.351454+0000 mon.smithi113 (mon.0) 371 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:35.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.351938+0000 mon.smithi113 (mon.0) 372 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:03:35.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:35.355614+0000 mon.smithi113 (mon.0) 373 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:36.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:34.847567+0000 mon.smithi113 (mon.0) 358 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-04-12T05:03:36.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.259315+0000 mgr.smithi113.ctsazg (mgr.14194) 99 : audit [DBG] from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:03:36.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.261584+0000 mon.smithi113 (mon.0) 359 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:03:36.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.263464+0000 mon.smithi113 (mon.0) 360 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:03:36.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.263957+0000 mon.smithi113 (mon.0) 361 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:36.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.335772+0000 mon.smithi113 (mon.0) 362 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:36.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.336734+0000 mon.smithi113 (mon.0) 363 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:03:36.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.337313+0000 mon.smithi113 (mon.0) 364 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:03:36.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.337807+0000 mon.smithi113 (mon.0) 365 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:03:36.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.341617+0000 mon.smithi113 (mon.0) 366 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:36.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.342493+0000 mon.smithi113 (mon.0) 367 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:36.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.343116+0000 mon.smithi113 (mon.0) 368 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:03:36.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.347025+0000 mon.smithi113 (mon.0) 369 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:36.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.350260+0000 mon.smithi113 (mon.0) 370 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:03:36.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.351454+0000 mon.smithi113 (mon.0) 371 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:36.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.351938+0000 mon.smithi113 (mon.0) 372 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:03:36.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:35.355614+0000 mon.smithi113 (mon.0) 373 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:37.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:03:35.331024+0000 mgr.smithi113.ctsazg (mgr.14194) 100 : cephadm [INF] Detected new or changed devices on smithi113 2024-04-12T05:03:37.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:03:35.338208+0000 mgr.smithi113.ctsazg (mgr.14194) 101 : cephadm [INF] Adjusting osd_memory_target on smithi113 to 4640M 2024-04-12T05:03:37.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:35.518398+0000 mgr.smithi113.ctsazg (mgr.14194) 102 : cluster [DBG] pgmap v76: 1 pgs: 1 creating+activating; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:37.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:03:35.331024+0000 mgr.smithi113.ctsazg (mgr.14194) 100 : cephadm [INF] Detected new or changed devices on smithi113 2024-04-12T05:03:37.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:03:35.338208+0000 mgr.smithi113.ctsazg (mgr.14194) 101 : cephadm [INF] Adjusting osd_memory_target on smithi113 to 4640M 2024-04-12T05:03:37.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:35.518398+0000 mgr.smithi113.ctsazg (mgr.14194) 102 : cluster [DBG] pgmap v76: 1 pgs: 1 creating+activating; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:38.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:37.569785+0000 mon.smithi192 (mon.1) 2 : audit [INF] from='client.? 172.21.15.192:0/1107223992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49fd856c-c351-4e90-9088-ece0024cd2c6"}]: dispatch 2024-04-12T05:03:38.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:37.570273+0000 mon.smithi113 (mon.0) 374 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49fd856c-c351-4e90-9088-ece0024cd2c6"}]: dispatch 2024-04-12T05:03:38.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:37.575152+0000 mon.smithi113 (mon.0) 375 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49fd856c-c351-4e90-9088-ece0024cd2c6"}]': finished 2024-04-12T05:03:38.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:37.575208+0000 mon.smithi113 (mon.0) 376 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-04-12T05:03:38.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:37.575375+0000 mon.smithi113 (mon.0) 377 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:03:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:37.569785+0000 mon.smithi192 (mon.1) 2 : audit [INF] from='client.? 172.21.15.192:0/1107223992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49fd856c-c351-4e90-9088-ece0024cd2c6"}]: dispatch 2024-04-12T05:03:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:37.570273+0000 mon.smithi113 (mon.0) 374 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49fd856c-c351-4e90-9088-ece0024cd2c6"}]: dispatch 2024-04-12T05:03:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:37.575152+0000 mon.smithi113 (mon.0) 375 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49fd856c-c351-4e90-9088-ece0024cd2c6"}]': finished 2024-04-12T05:03:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:37.575208+0000 mon.smithi113 (mon.0) 376 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-04-12T05:03:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:37.575375+0000 mon.smithi113 (mon.0) 377 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:03:39.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:37.518740+0000 mgr.smithi113.ctsazg (mgr.14194) 103 : cluster [DBG] pgmap v77: 1 pgs: 1 creating+activating; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:39.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:38.157987+0000 mon.smithi192 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.192:0/2697749165' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:03:39.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:37.518740+0000 mgr.smithi113.ctsazg (mgr.14194) 103 : cluster [DBG] pgmap v77: 1 pgs: 1 creating+activating; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:39.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:38.157987+0000 mon.smithi192 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.192:0/2697749165' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:03:41.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:39.519205+0000 mgr.smithi113.ctsazg (mgr.14194) 104 : cluster [DBG] pgmap v79: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:41.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:39.519205+0000 mgr.smithi113.ctsazg (mgr.14194) 104 : cluster [DBG] pgmap v79: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:43.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:42 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:41.519674+0000 mgr.smithi113.ctsazg (mgr.14194) 105 : cluster [DBG] pgmap v80: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:43.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:42 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:41.519674+0000 mgr.smithi113.ctsazg (mgr.14194) 105 : cluster [DBG] pgmap v80: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:43.520019+0000 mgr.smithi113.ctsazg (mgr.14194) 106 : cluster [DBG] pgmap v81: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:44.849610+0000 mon.smithi113 (mon.0) 378 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:03:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:44.850413+0000 mon.smithi113 (mon.0) 379 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:45.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:43.520019+0000 mgr.smithi113.ctsazg (mgr.14194) 106 : cluster [DBG] pgmap v81: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:45.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:44.849610+0000 mon.smithi113 (mon.0) 378 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:03:45.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:44.850413+0000 mon.smithi113 (mon.0) 379 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:03:46.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:03:44.851069+0000 mgr.smithi113.ctsazg (mgr.14194) 107 : cephadm [INF] Deploying daemon osd.3 on smithi192 2024-04-12T05:03:46.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:03:44.851069+0000 mgr.smithi113.ctsazg (mgr.14194) 107 : cephadm [INF] Deploying daemon osd.3 on smithi192 2024-04-12T05:03:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:45.520416+0000 mgr.smithi113.ctsazg (mgr.14194) 108 : cluster [DBG] pgmap v82: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:45.520416+0000 mgr.smithi113.ctsazg (mgr.14194) 108 : cluster [DBG] pgmap v82: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:49.146 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:47.520804+0000 mgr.smithi113.ctsazg (mgr.14194) 109 : cluster [DBG] pgmap v83: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:47.520804+0000 mgr.smithi113.ctsazg (mgr.14194) 109 : cluster [DBG] pgmap v83: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:49.521234+0000 mgr.smithi113.ctsazg (mgr.14194) 110 : cluster [DBG] pgmap v84: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:50.006964+0000 mon.smithi113 (mon.0) 380 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:03:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:50.009544+0000 mon.smithi113 (mon.0) 381 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:51.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:49.521234+0000 mgr.smithi113.ctsazg (mgr.14194) 110 : cluster [DBG] pgmap v84: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:51.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:50.006964+0000 mon.smithi113 (mon.0) 380 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:03:51.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:50.009544+0000 mon.smithi113 (mon.0) 381 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:53.116 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:51.521548+0000 mgr.smithi113.ctsazg (mgr.14194) 111 : cluster [DBG] pgmap v85: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:51.521548+0000 mgr.smithi113.ctsazg (mgr.14194) 111 : cluster [DBG] pgmap v85: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:53.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.425+0000 7f2ae2ffd700 1 -- 172.21.15.192:0/2265403987 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2aec1021d0 con 0x7f2ad8061180 2024-04-12T05:03:53.428 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 3 on host 'smithi192' 2024-04-12T05:03:53.429 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.427+0000 7f2af414e700 1 -- 172.21.15.192:0/2265403987 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ad8061180 msgr2=0x7f2ad8063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:03:53.429 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.427+0000 7f2af414e700 1 --2- 172.21.15.192:0/2265403987 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ad8061180 0x7f2ad8063640 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f2adc013470 tx=0x7f2adc00b040).stop 2024-04-12T05:03:53.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.427+0000 7f2af414e700 1 -- 172.21.15.192:0/2265403987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2aec0f8cb0 msgr2=0x7f2aec1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:03:53.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.427+0000 7f2af414e700 1 --2- 172.21.15.192:0/2265403987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2aec0f8cb0 0x7f2aec1086d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f2ae80041d0 tx=0x7f2ae8009f60).stop 2024-04-12T05:03:53.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.427+0000 7f2af414e700 1 -- 172.21.15.192:0/2265403987 shutdown_connections 2024-04-12T05:03:53.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.427+0000 7f2af414e700 1 --2- 172.21.15.192:0/2265403987 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2ad8061180 0x7f2ad8063640 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:53.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.427+0000 7f2af414e700 1 --2- 172.21.15.192:0/2265403987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2aec0f7a10 0x7f2aec1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:53.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.427+0000 7f2af414e700 1 --2- 172.21.15.192:0/2265403987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2aec0f8cb0 0x7f2aec1086d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:03:53.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.427+0000 7f2af414e700 1 -- 172.21.15.192:0/2265403987 >> 172.21.15.192:0/2265403987 conn(0x7f2aec0f3150 msgr2=0x7f2aec0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:03:53.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.427+0000 7f2af414e700 1 -- 172.21.15.192:0/2265403987 shutdown_connections 2024-04-12T05:03:53.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:03:53.427+0000 7f2af414e700 1 -- 172.21.15.192:0/2265403987 wait complete. 2024-04-12T05:03:53.977 DEBUG:teuthology.orchestra.run.smithi192:osd.3> sudo journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.3.service 2024-04-12T05:03:53.979 INFO:tasks.cephadm:Deploying osd.4 on smithi192 with /dev/vg_nvme/lv_3... 2024-04-12T05:03:53.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-12T05:03:54.009 INFO:journalctl@ceph.osd.3.smithi192.stdout:-- Logs begin at Fri 2024-04-12 04:48:43 UTC. -- 2024-04-12T05:03:54.430 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:53.425518+0000 mon.smithi113 (mon.0) 382 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:54.431 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:53.521904+0000 mgr.smithi113.ctsazg (mgr.14194) 112 : cluster [DBG] pgmap v86: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:54.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:53.898643+0000 mon.smithi192 (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:03:54.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:53.899147+0000 mon.smithi113 (mon.0) 383 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:03:54.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:53.425518+0000 mon.smithi113 (mon.0) 382 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:54.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:53.521904+0000 mgr.smithi113.ctsazg (mgr.14194) 112 : cluster [DBG] pgmap v86: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:54.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:53.898643+0000 mon.smithi192 (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:03:54.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:53.899147+0000 mon.smithi113 (mon.0) 383 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:03:55.698 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:03:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[48787]: debug 2024-04-12T05:03:55.434+0000 7f07789da700 -1 osd.3 0 waiting for initial osdmap 2024-04-12T05:03:55.699 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:03:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[48787]: debug 2024-04-12T05:03:55.441+0000 7f0773152700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:03:55.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:54.430403+0000 mon.smithi113 (mon.0) 384 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T05:03:55.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:54.430465+0000 mon.smithi113 (mon.0) 385 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-04-12T05:03:55.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:54.430742+0000 mon.smithi113 (mon.0) 386 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:03:55.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:54.431628+0000 mon.smithi192 (mon.1) 5 : audit [INF] from='osd.3 [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:03:55.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:54.432047+0000 mon.smithi113 (mon.0) 387 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:03:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:54.430403+0000 mon.smithi113 (mon.0) 384 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T05:03:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:54.430465+0000 mon.smithi113 (mon.0) 385 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-04-12T05:03:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:54.430742+0000 mon.smithi113 (mon.0) 386 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:03:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:54.431628+0000 mon.smithi192 (mon.1) 5 : audit [INF] from='osd.3 [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:03:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:54.432047+0000 mon.smithi113 (mon.0) 387 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:03:56.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:55.432770+0000 mon.smithi113 (mon.0) 388 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-12T05:03:56.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:55.432884+0000 mon.smithi113 (mon.0) 389 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-04-12T05:03:56.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:55.433270+0000 mon.smithi113 (mon.0) 390 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:03:56.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:55.437846+0000 mon.smithi113 (mon.0) 391 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:03:56.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:55.478320+0000 mon.smithi113 (mon.0) 392 : cluster [INF] osd.3 [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207] boot 2024-04-12T05:03:56.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:55.478361+0000 mon.smithi113 (mon.0) 393 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-04-12T05:03:56.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:55.478497+0000 mon.smithi113 (mon.0) 394 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:03:56.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:55.522241+0000 mgr.smithi113.ctsazg (mgr.14194) 113 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:56.899 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-12T05:03:56.900 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-12T05:03:56.900 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-04-12T05:03:56.900 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-04-12T05:03:56.900 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0143816 s, 729 MB/s 2024-04-12T05:03:56.900 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-04-12T05:03:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:55.432770+0000 mon.smithi113 (mon.0) 388 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-12T05:03:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:55.432884+0000 mon.smithi113 (mon.0) 389 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-04-12T05:03:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:55.433270+0000 mon.smithi113 (mon.0) 390 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:03:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:55.437846+0000 mon.smithi113 (mon.0) 391 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:03:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:55.478320+0000 mon.smithi113 (mon.0) 392 : cluster [INF] osd.3 [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207] boot 2024-04-12T05:03:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:55.478361+0000 mon.smithi113 (mon.0) 393 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-04-12T05:03:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:55.478497+0000 mon.smithi113 (mon.0) 394 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:03:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:55.522241+0000 mgr.smithi113.ctsazg (mgr.14194) 113 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-04-12T05:03:57.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:54.928521+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:03:57.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:54.928608+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:03:57.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:56.485957+0000 mon.smithi113 (mon.0) 395 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-04-12T05:03:57.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:03:56.638440+0000 mon.smithi113 (mon.0) 396 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:57.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:54.928521+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:03:57.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:54.928608+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:03:57.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:56.485957+0000 mon.smithi113 (mon.0) 395 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-04-12T05:03:57.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:03:56.638440+0000 mon.smithi113 (mon.0) 396 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:03:58.529 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:57.488278+0000 mon.smithi113 (mon.0) 397 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-04-12T05:03:58.529 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:03:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:57.522602+0000 mgr.smithi113.ctsazg (mgr.14194) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:03:58.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:57.488278+0000 mon.smithi113 (mon.0) 397 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-04-12T05:03:58.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:03:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:57.522602+0000 mgr.smithi113.ctsazg (mgr.14194) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:03:59.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2024-04-12T05:04:00.636 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:03:59.522959+0000 mgr.smithi113.ctsazg (mgr.14194) 115 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:00.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:03:59.522959+0000 mgr.smithi113.ctsazg (mgr.14194) 115 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:01.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.702+0000 7f62c4a4e700 1 -- 172.21.15.192:0/938862513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62c00f7a10 msgr2=0x7f62c00f7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:01.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.702+0000 7f62c4a4e700 1 --2- 172.21.15.192:0/938862513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62c00f7a10 0x7f62c00f7e30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f62b00047f0 tx=0x7f62b0009cb0).stop 2024-04-12T05:04:01.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.702+0000 7f62c4a4e700 1 -- 172.21.15.192:0/938862513 shutdown_connections 2024-04-12T05:04:01.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.702+0000 7f62c4a4e700 1 --2- 172.21.15.192:0/938862513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62c00f8cb0 0x7f62c00f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:01.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.702+0000 7f62c4a4e700 1 --2- 172.21.15.192:0/938862513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62c00f7a10 0x7f62c00f7e30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:01.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.702+0000 7f62c4a4e700 1 -- 172.21.15.192:0/938862513 >> 172.21.15.192:0/938862513 conn(0x7f62c00f3150 msgr2=0x7f62c00f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:01.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.702+0000 7f62c4a4e700 1 -- 172.21.15.192:0/938862513 shutdown_connections 2024-04-12T05:04:01.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.702+0000 7f62c4a4e700 1 -- 172.21.15.192:0/938862513 wait complete. 2024-04-12T05:04:01.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62c4a4e700 1 Processor -- start 2024-04-12T05:04:01.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62c4a4e700 1 -- start start 2024-04-12T05:04:01.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62c4a4e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62c00f7a10 0x7f62c0108170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:01.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62c4a4e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62c00f8cb0 0x7f62c0108660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62c4a4e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c0108c10 con 0x7f62c00f7a10 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62c4a4e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c0108d50 con 0x7f62c00f8cb0 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62bdd9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62c00f8cb0 0x7f62c0108660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62bdd9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62c00f8cb0 0x7f62c0108660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:55308/0 (socket says 172.21.15.192:55308) 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62bdd9b700 1 -- 172.21.15.192:0/2932147566 learned_addr learned my addr 172.21.15.192:0/2932147566 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62be59c700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62c00f7a10 0x7f62c0108170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62bdd9b700 1 -- 172.21.15.192:0/2932147566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62c00f7a10 msgr2=0x7f62c0108170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62bdd9b700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62c00f7a10 0x7f62c0108170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62bdd9b700 1 -- 172.21.15.192:0/2932147566 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0005040 con 0x7f62c00f8cb0 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62be59c700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62c00f7a10 0x7f62c0108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62bdd9b700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62c00f8cb0 0x7f62c0108660 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f62b400c740 tx=0x7f62b4007ef0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62af7fe700 1 -- 172.21.15.192:0/2932147566 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62b400e680 con 0x7f62c00f8cb0 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62af7fe700 1 -- 172.21.15.192:0/2932147566 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f62b400ec70 con 0x7f62c00f8cb0 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62c4a4e700 1 -- 172.21.15.192:0/2932147566 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c010d7f0 con 0x7f62c00f8cb0 2024-04-12T05:04:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62af7fe700 1 -- 172.21.15.192:0/2932147566 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62b4013d30 con 0x7f62c00f8cb0 2024-04-12T05:04:01.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.703+0000 7f62c4a4e700 1 -- 172.21.15.192:0/2932147566 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c010dd30 con 0x7f62c00f8cb0 2024-04-12T05:04:01.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.704+0000 7f62c4a4e700 1 -- 172.21.15.192:0/2932147566 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c005fb50 con 0x7f62c00f8cb0 2024-04-12T05:04:01.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.705+0000 7f62af7fe700 1 -- 172.21.15.192:0/2932147566 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f62b400e7e0 con 0x7f62c00f8cb0 2024-04-12T05:04:01.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.705+0000 7f62af7fe700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f62a8061180 0x7f62a8063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:01.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.705+0000 7f62af7fe700 1 -- 172.21.15.192:0/2932147566 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3671+0+0 (secure 0 0 0) 0x7f62b407c4e0 con 0x7f62c00f8cb0 2024-04-12T05:04:01.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.705+0000 7f62be59c700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f62a8061180 0x7f62a8063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:01.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.706+0000 7f62be59c700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f62a8061180 0x7f62a8063640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f62b0013470 tx=0x7f62b000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:01.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.709+0000 7f62af7fe700 1 -- 172.21.15.192:0/2932147566 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f62b4050b10 con 0x7f62c00f8cb0 2024-04-12T05:04:01.881 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:01.879+0000 7f62c4a4e700 1 -- 172.21.15.192:0/2932147566 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f62c0102180 con 0x7f62a8061180 2024-04-12T05:04:02.599 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:04:01.212320+0000 mgr.smithi113.ctsazg (mgr.14194) 116 : cephadm [INF] Detected new or changed devices on smithi192 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.217643+0000 mon.smithi113 (mon.0) 398 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.554507+0000 mon.smithi113 (mon.0) 399 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.561904+0000 mon.smithi113 (mon.0) 400 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.562837+0000 mon.smithi113 (mon.0) 401 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.563518+0000 mon.smithi113 (mon.0) 402 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.569773+0000 mon.smithi113 (mon.0) 403 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.573942+0000 mon.smithi113 (mon.0) 404 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.575408+0000 mon.smithi113 (mon.0) 405 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.576059+0000 mon.smithi113 (mon.0) 406 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.581899+0000 mon.smithi113 (mon.0) 407 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.883646+0000 mon.smithi113 (mon.0) 408 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.885881+0000 mon.smithi113 (mon.0) 409 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:04:02.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.886548+0000 mon.smithi113 (mon.0) 410 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:02.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:04:01.212320+0000 mgr.smithi113.ctsazg (mgr.14194) 116 : cephadm [INF] Detected new or changed devices on smithi192 2024-04-12T05:04:02.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.217643+0000 mon.smithi113 (mon.0) 398 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:02.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.554507+0000 mon.smithi113 (mon.0) 399 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:02.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.561904+0000 mon.smithi113 (mon.0) 400 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:02.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.562837+0000 mon.smithi113 (mon.0) 401 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:02.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.563518+0000 mon.smithi113 (mon.0) 402 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:02.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.569773+0000 mon.smithi113 (mon.0) 403 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:02.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.573942+0000 mon.smithi113 (mon.0) 404 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:04:02.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.575408+0000 mon.smithi113 (mon.0) 405 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:02.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.576059+0000 mon.smithi113 (mon.0) 406 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:02.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.581899+0000 mon.smithi113 (mon.0) 407 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:02.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.883646+0000 mon.smithi113 (mon.0) 408 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:04:02.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.885881+0000 mon.smithi113 (mon.0) 409 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:04:02.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.886548+0000 mon.smithi113 (mon.0) 410 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:03.382 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:01.523311+0000 mgr.smithi113.ctsazg (mgr.14194) 117 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:03.382 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:04:01.555182+0000 mgr.smithi113.ctsazg (mgr.14194) 118 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 14946M 2024-04-12T05:04:03.382 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:01.881539+0000 mgr.smithi113.ctsazg (mgr.14194) 119 : audit [DBG] from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:04:03.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:01.523311+0000 mgr.smithi113.ctsazg (mgr.14194) 117 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:03.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:04:01.555182+0000 mgr.smithi113.ctsazg (mgr.14194) 118 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 14946M 2024-04-12T05:04:03.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:01.881539+0000 mgr.smithi113.ctsazg (mgr.14194) 119 : audit [DBG] from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:04:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:03.523787+0000 mgr.smithi113.ctsazg (mgr.14194) 120 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:04.411643+0000 mon.smithi192 (mon.1) 6 : audit [INF] from='client.? 172.21.15.192:0/3136710312' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b72db8-682a-4a51-8d33-b0addda91fb6"}]: dispatch 2024-04-12T05:04:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:04.412206+0000 mon.smithi113 (mon.0) 411 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b72db8-682a-4a51-8d33-b0addda91fb6"}]: dispatch 2024-04-12T05:04:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:04.418779+0000 mon.smithi113 (mon.0) 412 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80b72db8-682a-4a51-8d33-b0addda91fb6"}]': finished 2024-04-12T05:04:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:04.418846+0000 mon.smithi113 (mon.0) 413 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-04-12T05:04:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:04.418989+0000 mon.smithi113 (mon.0) 414 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:04:05.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:05.044521+0000 mon.smithi192 (mon.1) 7 : audit [DBG] from='client.? 172.21.15.192:0/1505696009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:04:05.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:03.523787+0000 mgr.smithi113.ctsazg (mgr.14194) 120 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:05.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:04.411643+0000 mon.smithi192 (mon.1) 6 : audit [INF] from='client.? 172.21.15.192:0/3136710312' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b72db8-682a-4a51-8d33-b0addda91fb6"}]: dispatch 2024-04-12T05:04:05.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:04.412206+0000 mon.smithi113 (mon.0) 411 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b72db8-682a-4a51-8d33-b0addda91fb6"}]: dispatch 2024-04-12T05:04:05.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:04.418779+0000 mon.smithi113 (mon.0) 412 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80b72db8-682a-4a51-8d33-b0addda91fb6"}]': finished 2024-04-12T05:04:05.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:04.418846+0000 mon.smithi113 (mon.0) 413 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-04-12T05:04:05.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:04.418989+0000 mon.smithi113 (mon.0) 414 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:04:05.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:05.044521+0000 mon.smithi192 (mon.1) 7 : audit [DBG] from='client.? 172.21.15.192:0/1505696009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:04:07.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:05.524224+0000 mgr.smithi113.ctsazg (mgr.14194) 121 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:07.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:05.524224+0000 mgr.smithi113.ctsazg (mgr.14194) 121 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:07.524617+0000 mgr.smithi113.ctsazg (mgr.14194) 122 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:09.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:07.524617+0000 mgr.smithi113.ctsazg (mgr.14194) 122 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:11.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:09.525046+0000 mgr.smithi113.ctsazg (mgr.14194) 123 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:11.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:09.525046+0000 mgr.smithi113.ctsazg (mgr.14194) 123 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:12.521 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:11.812686+0000 mon.smithi113 (mon.0) 415 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:04:12.521 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:11.813467+0000 mon.smithi113 (mon.0) 416 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:12.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:11.812686+0000 mon.smithi113 (mon.0) 415 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:04:12.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:11.813467+0000 mon.smithi113 (mon.0) 416 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:13.502 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:11.525352+0000 mgr.smithi113.ctsazg (mgr.14194) 124 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:13.502 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:04:11.814130+0000 mgr.smithi113.ctsazg (mgr.14194) 125 : cephadm [INF] Deploying daemon osd.4 on smithi192 2024-04-12T05:04:13.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:11.525352+0000 mgr.smithi113.ctsazg (mgr.14194) 124 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:13.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:04:11.814130+0000 mgr.smithi113.ctsazg (mgr.14194) 125 : cephadm [INF] Deploying daemon osd.4 on smithi192 2024-04-12T05:04:15.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:13.525650+0000 mgr.smithi113.ctsazg (mgr.14194) 126 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:15.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:13.525650+0000 mgr.smithi113.ctsazg (mgr.14194) 126 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:17.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:15.526039+0000 mgr.smithi113.ctsazg (mgr.14194) 127 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:17.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:16.938614+0000 mon.smithi113 (mon.0) 417 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:17.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:16.940386+0000 mon.smithi113 (mon.0) 418 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:04:17.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:15.526039+0000 mgr.smithi113.ctsazg (mgr.14194) 127 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:17.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:16.938614+0000 mon.smithi113 (mon.0) 417 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:17.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:16.940386+0000 mon.smithi113 (mon.0) 418 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:04:19.396 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:17.526381+0000 mgr.smithi113.ctsazg (mgr.14194) 128 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:19.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:17.526381+0000 mgr.smithi113.ctsazg (mgr.14194) 128 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:20.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.989+0000 7f62af7fe700 1 -- 172.21.15.192:0/2932147566 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f62c0102180 con 0x7f62a8061180 2024-04-12T05:04:20.992 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 4 on host 'smithi192' 2024-04-12T05:04:20.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.991+0000 7f62c4a4e700 1 -- 172.21.15.192:0/2932147566 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f62a8061180 msgr2=0x7f62a8063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:20.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.991+0000 7f62c4a4e700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f62a8061180 0x7f62a8063640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f62b0013470 tx=0x7f62b000b040).stop 2024-04-12T05:04:20.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.991+0000 7f62c4a4e700 1 -- 172.21.15.192:0/2932147566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62c00f8cb0 msgr2=0x7f62c0108660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:20.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.991+0000 7f62c4a4e700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62c00f8cb0 0x7f62c0108660 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f62b400c740 tx=0x7f62b4007ef0).stop 2024-04-12T05:04:20.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.991+0000 7f62c4a4e700 1 -- 172.21.15.192:0/2932147566 shutdown_connections 2024-04-12T05:04:20.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.991+0000 7f62c4a4e700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f62a8061180 0x7f62a8063640 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:20.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.991+0000 7f62c4a4e700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62c00f7a10 0x7f62c0108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:20.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.991+0000 7f62c4a4e700 1 --2- 172.21.15.192:0/2932147566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62c00f8cb0 0x7f62c0108660 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:20.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.991+0000 7f62c4a4e700 1 -- 172.21.15.192:0/2932147566 >> 172.21.15.192:0/2932147566 conn(0x7f62c00f3150 msgr2=0x7f62c00fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:20.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.991+0000 7f62c4a4e700 1 -- 172.21.15.192:0/2932147566 shutdown_connections 2024-04-12T05:04:20.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:20.991+0000 7f62c4a4e700 1 -- 172.21.15.192:0/2932147566 wait complete. 2024-04-12T05:04:21.282 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:19.526795+0000 mgr.smithi113.ctsazg (mgr.14194) 129 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:21.282 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:20.760019+0000 mon.smithi192 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:04:21.283 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:20.760612+0000 mon.smithi113 (mon.0) 419 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:04:21.283 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:20.990181+0000 mon.smithi113 (mon.0) 420 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:21.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:19.526795+0000 mgr.smithi113.ctsazg (mgr.14194) 129 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:21.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:20.760019+0000 mon.smithi192 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:04:21.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:20.760612+0000 mon.smithi113 (mon.0) 419 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:04:21.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:20.990181+0000 mon.smithi113 (mon.0) 420 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:21.278364+0000 mon.smithi113 (mon.0) 421 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T05:04:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:21.278451+0000 mon.smithi113 (mon.0) 422 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-04-12T05:04:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:21.278843+0000 mon.smithi113 (mon.0) 423 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:04:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:21.279265+0000 mon.smithi192 (mon.1) 9 : audit [INF] from='osd.4 [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:04:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:21.282164+0000 mon.smithi113 (mon.0) 424 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:04:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:22.037515+0000 mon.smithi113 (mon.0) 425 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:22.538 DEBUG:teuthology.orchestra.run.smithi192:osd.4> sudo journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.4.service 2024-04-12T05:04:22.540 INFO:tasks.cephadm:Deploying osd.5 on smithi192 with /dev/vg_nvme/lv_2... 2024-04-12T05:04:22.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-12T05:04:22.571 INFO:journalctl@ceph.osd.4.smithi192.stdout:-- Logs begin at Fri 2024-04-12 04:48:43 UTC. -- 2024-04-12T05:04:22.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:21.278364+0000 mon.smithi113 (mon.0) 421 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T05:04:22.604 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:21.278451+0000 mon.smithi113 (mon.0) 422 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-04-12T05:04:22.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:21.278843+0000 mon.smithi113 (mon.0) 423 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:04:22.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:21.279265+0000 mon.smithi192 (mon.1) 9 : audit [INF] from='osd.4 [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:04:22.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:21.282164+0000 mon.smithi113 (mon.0) 424 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:04:22.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:22.037515+0000 mon.smithi113 (mon.0) 425 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:21.527141+0000 mgr.smithi113.ctsazg (mgr.14194) 130 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:22.289750+0000 mon.smithi113 (mon.0) 426 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-12T05:04:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:22.289859+0000 mon.smithi113 (mon.0) 427 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-04-12T05:04:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:22.290155+0000 mon.smithi113 (mon.0) 428 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:04:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:22.294557+0000 mon.smithi113 (mon.0) 429 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:04:23.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:21.527141+0000 mgr.smithi113.ctsazg (mgr.14194) 130 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-12T05:04:23.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:22.289750+0000 mon.smithi113 (mon.0) 426 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-12T05:04:23.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:22.289859+0000 mon.smithi113 (mon.0) 427 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2024-04-12T05:04:23.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:22.290155+0000 mon.smithi113 (mon.0) 428 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:04:23.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:22.294557+0000 mon.smithi113 (mon.0) 429 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:04:24.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:21.787933+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:04:24.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:21.787998+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:04:24.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:23.293777+0000 mon.smithi113 (mon.0) 430 : cluster [INF] osd.4 [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248] boot 2024-04-12T05:04:24.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:23.293841+0000 mon.smithi113 (mon.0) 431 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-04-12T05:04:24.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:23.294024+0000 mon.smithi113 (mon.0) 432 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:04:24.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:21.787933+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:04:24.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:21.787998+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:04:24.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:23.293777+0000 mon.smithi113 (mon.0) 430 : cluster [INF] osd.4 [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248] boot 2024-04-12T05:04:24.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:23.293841+0000 mon.smithi113 (mon.0) 431 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-04-12T05:04:24.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:23.294024+0000 mon.smithi113 (mon.0) 432 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:04:25.585 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-12T05:04:25.585 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-12T05:04:25.585 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-04-12T05:04:25.585 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-04-12T05:04:25.586 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111076 s, 944 MB/s 2024-04-12T05:04:25.586 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-04-12T05:04:25.622 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:23.527558+0000 mgr.smithi113.ctsazg (mgr.14194) 131 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:25.622 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:24.299306+0000 mon.smithi113 (mon.0) 433 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-04-12T05:04:25.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:23.527558+0000 mgr.smithi113.ctsazg (mgr.14194) 131 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:25.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:24.299306+0000 mon.smithi113 (mon.0) 433 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-04-12T05:04:26.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2024-04-12T05:04:27.411 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:25.528033+0000 mgr.smithi113.ctsazg (mgr.14194) 132 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:27.411 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:27.184038+0000 mon.smithi113 (mon.0) 434 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:27.411 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:27.185017+0000 mon.smithi113 (mon.0) 435 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:27.411 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:27.185860+0000 mon.smithi113 (mon.0) 436 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:27.411 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:27.190317+0000 mon.smithi113 (mon.0) 437 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:27.411 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:27.191206+0000 mon.smithi113 (mon.0) 438 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:27.411 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:27.191907+0000 mon.smithi113 (mon.0) 439 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:27.412 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:27.196040+0000 mon.smithi113 (mon.0) 440 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:27.412 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:27.200229+0000 mon.smithi113 (mon.0) 441 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:04:27.412 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:27.201712+0000 mon.smithi113 (mon.0) 442 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:27.412 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:27.202353+0000 mon.smithi113 (mon.0) 443 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:27.412 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:27.206274+0000 mon.smithi113 (mon.0) 444 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:27.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:25.528033+0000 mgr.smithi113.ctsazg (mgr.14194) 132 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:27.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:27.184038+0000 mon.smithi113 (mon.0) 434 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:27.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:27.185017+0000 mon.smithi113 (mon.0) 435 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:27.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:27.185860+0000 mon.smithi113 (mon.0) 436 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:27.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:27.190317+0000 mon.smithi113 (mon.0) 437 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:27.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:27.191206+0000 mon.smithi113 (mon.0) 438 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:27.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:27.191907+0000 mon.smithi113 (mon.0) 439 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:27.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:27.196040+0000 mon.smithi113 (mon.0) 440 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:27.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:27.200229+0000 mon.smithi113 (mon.0) 441 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:04:27.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:27.201712+0000 mon.smithi113 (mon.0) 442 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:27.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:27.202353+0000 mon.smithi113 (mon.0) 443 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:27.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:27.206274+0000 mon.smithi113 (mon.0) 444 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:28.410 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:04:27.178497+0000 mgr.smithi113.ctsazg (mgr.14194) 133 : cephadm [INF] Detected new or changed devices on smithi192 2024-04-12T05:04:28.410 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:04:27.186427+0000 mgr.smithi113.ctsazg (mgr.14194) 134 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 7473M 2024-04-12T05:04:28.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:04:27.178497+0000 mgr.smithi113.ctsazg (mgr.14194) 133 : cephadm [INF] Detected new or changed devices on smithi192 2024-04-12T05:04:28.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:04:27.186427+0000 mgr.smithi113.ctsazg (mgr.14194) 134 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 7473M 2024-04-12T05:04:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.659+0000 7fbffe9fe700 1 -- 172.21.15.192:0/99939066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbff80f8cb0 msgr2=0x7fbff80f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.659+0000 7fbffe9fe700 1 --2- 172.21.15.192:0/99939066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbff80f8cb0 0x7fbff80f9110 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fbfec0047f0 tx=0x7fbfec009cb0).stop 2024-04-12T05:04:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.659+0000 7fbffe9fe700 1 -- 172.21.15.192:0/99939066 shutdown_connections 2024-04-12T05:04:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.659+0000 7fbffe9fe700 1 --2- 172.21.15.192:0/99939066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbff80f8cb0 0x7fbff80f9110 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.659+0000 7fbffe9fe700 1 --2- 172.21.15.192:0/99939066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff80f7a10 0x7fbff80f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:04:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.659+0000 7fbffe9fe700 1 -- 172.21.15.192:0/99939066 >> 172.21.15.192:0/99939066 conn(0x7fbff80f3150 msgr2=0x7fbff80f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.659+0000 7fbffe9fe700 1 -- 172.21.15.192:0/99939066 shutdown_connections 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.659+0000 7fbffe9fe700 1 -- 172.21.15.192:0/99939066 wait complete. 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.660+0000 7fbffe9fe700 1 Processor -- start 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.660+0000 7fbffe9fe700 1 -- start start 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.660+0000 7fbffe9fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbff80f7a10 0x7fbff806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.660+0000 7fbffe9fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff80f8cb0 0x7fbff806b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.660+0000 7fbffe9fe700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff806b520 con 0x7fbff80f8cb0 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.660+0000 7fbffe9fe700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff806b660 con 0x7fbff80f7a10 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.660+0000 7fbff7fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbff80f7a10 0x7fbff806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.660+0000 7fbff7fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbff80f7a10 0x7fbff806c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:36206/0 (socket says 172.21.15.192:36206) 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.660+0000 7fbff7fff700 1 -- 172.21.15.192:0/3454758889 learned_addr learned my addr 172.21.15.192:0/3454758889 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.661+0000 7fbff7fff700 1 -- 172.21.15.192:0/3454758889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff80f8cb0 msgr2=0x7fbff806b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.661+0000 7fbff7fff700 1 --2- 172.21.15.192:0/3454758889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff80f8cb0 0x7fbff806b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.661+0000 7fbff7fff700 1 -- 172.21.15.192:0/3454758889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec005040 con 0x7fbff80f7a10 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.661+0000 7fbff7fff700 1 --2- 172.21.15.192:0/3454758889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbff80f7a10 0x7fbff806c9d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fbfe800b480 tx=0x7fbfe80090f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.661+0000 7fbff57fa700 1 -- 172.21.15.192:0/3454758889 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfe8002500 con 0x7fbff80f7a10 2024-04-12T05:04:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.661+0000 7fbff57fa700 1 -- 172.21.15.192:0/3454758889 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fbfe8002af0 con 0x7fbff80f7a10 2024-04-12T05:04:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.661+0000 7fbff57fa700 1 -- 172.21.15.192:0/3454758889 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfe8012c30 con 0x7fbff80f7a10 2024-04-12T05:04:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.661+0000 7fbffe9fe700 1 -- 172.21.15.192:0/3454758889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff806b940 con 0x7fbff80f7a10 2024-04-12T05:04:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.661+0000 7fbffe9fe700 1 -- 172.21.15.192:0/3454758889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff806be80 con 0x7fbff80f7a10 2024-04-12T05:04:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.662+0000 7fbffe9fe700 1 -- 172.21.15.192:0/3454758889 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff805fb50 con 0x7fbff80f7a10 2024-04-12T05:04:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.662+0000 7fbff57fa700 1 -- 172.21.15.192:0/3454758889 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fbfe801c440 con 0x7fbff80f7a10 2024-04-12T05:04:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.662+0000 7fbff57fa700 1 --2- 172.21.15.192:0/3454758889 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fbfe0061180 0x7fbfe0063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.663+0000 7fbff57fa700 1 -- 172.21.15.192:0/3454758889 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4103+0+0 (secure 0 0 0) 0x7fbfe807c550 con 0x7fbff80f7a10 2024-04-12T05:04:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.663+0000 7fbff77fe700 1 --2- 172.21.15.192:0/3454758889 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fbfe0061180 0x7fbfe0063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.663+0000 7fbff77fe700 1 --2- 172.21.15.192:0/3454758889 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fbfe0061180 0x7fbfe0063640 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fbfec013510 tx=0x7fbfec009cb0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.667+0000 7fbff57fa700 1 -- 172.21.15.192:0/3454758889 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fbfe8050b20 con 0x7fbff80f7a10 2024-04-12T05:04:28.841 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:28.839+0000 7fbffe9fe700 1 -- 172.21.15.192:0/3454758889 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fbff81022d0 con 0x7fbfe0061180 2024-04-12T05:04:29.568 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:27.528498+0000 mgr.smithi113.ctsazg (mgr.14194) 135 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:29.568 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:28.842953+0000 mon.smithi113 (mon.0) 445 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:04:29.568 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:28.845319+0000 mon.smithi113 (mon.0) 446 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:04:29.568 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:28.846065+0000 mon.smithi113 (mon.0) 447 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:29.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:27.528498+0000 mgr.smithi113.ctsazg (mgr.14194) 135 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:29.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:28.842953+0000 mon.smithi113 (mon.0) 445 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T05:04:29.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:28.845319+0000 mon.smithi113 (mon.0) 446 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T05:04:29.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:28.846065+0000 mon.smithi113 (mon.0) 447 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:30.508 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:28.840674+0000 mgr.smithi113.ctsazg (mgr.14194) 136 : audit [DBG] from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:04:30.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:28.840674+0000 mgr.smithi113.ctsazg (mgr.14194) 136 : audit [DBG] from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:04:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:29.528895+0000 mgr.smithi113.ctsazg (mgr.14194) 137 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:31.247483+0000 mon.smithi192 (mon.1) 10 : audit [INF] from='client.? 172.21.15.192:0/1178216174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8e1e98f-5edb-4fec-bdf6-bcda40562682"}]: dispatch 2024-04-12T05:04:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:31.248072+0000 mon.smithi113 (mon.0) 448 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8e1e98f-5edb-4fec-bdf6-bcda40562682"}]: dispatch 2024-04-12T05:04:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:31.253478+0000 mon.smithi113 (mon.0) 449 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8e1e98f-5edb-4fec-bdf6-bcda40562682"}]': finished 2024-04-12T05:04:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:31.253558+0000 mon.smithi113 (mon.0) 450 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-04-12T05:04:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:31.253680+0000 mon.smithi113 (mon.0) 451 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:04:31.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:29.528895+0000 mgr.smithi113.ctsazg (mgr.14194) 137 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:31.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:31.247483+0000 mon.smithi192 (mon.1) 10 : audit [INF] from='client.? 172.21.15.192:0/1178216174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8e1e98f-5edb-4fec-bdf6-bcda40562682"}]: dispatch 2024-04-12T05:04:31.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:31.248072+0000 mon.smithi113 (mon.0) 448 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8e1e98f-5edb-4fec-bdf6-bcda40562682"}]: dispatch 2024-04-12T05:04:31.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:31.253478+0000 mon.smithi113 (mon.0) 449 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8e1e98f-5edb-4fec-bdf6-bcda40562682"}]': finished 2024-04-12T05:04:31.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:31.253558+0000 mon.smithi113 (mon.0) 450 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-04-12T05:04:31.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:31.253680+0000 mon.smithi113 (mon.0) 451 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:04:32.605 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:31.840494+0000 mon.smithi192 (mon.1) 11 : audit [DBG] from='client.? 172.21.15.192:0/1360097714' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:04:32.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:31.840494+0000 mon.smithi192 (mon.1) 11 : audit [DBG] from='client.? 172.21.15.192:0/1360097714' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T05:04:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:31.529229+0000 mgr.smithi113.ctsazg (mgr.14194) 138 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:33.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:31.529229+0000 mgr.smithi113.ctsazg (mgr.14194) 138 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:35.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:33.529690+0000 mgr.smithi113.ctsazg (mgr.14194) 139 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:35.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:33.529690+0000 mgr.smithi113.ctsazg (mgr.14194) 139 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:36.595 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:35.530107+0000 mgr.smithi113.ctsazg (mgr.14194) 140 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:36.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:35.530107+0000 mgr.smithi113.ctsazg (mgr.14194) 140 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:38.628 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:37.530449+0000 mgr.smithi113.ctsazg (mgr.14194) 141 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:38.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:37.530449+0000 mgr.smithi113.ctsazg (mgr.14194) 141 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:39.894 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:38.651097+0000 mon.smithi113 (mon.0) 452 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:04:39.894 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:38.651987+0000 mon.smithi113 (mon.0) 453 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:39.894 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:04:38.652689+0000 mgr.smithi113.ctsazg (mgr.14194) 142 : cephadm [INF] Deploying daemon osd.5 on smithi192 2024-04-12T05:04:39.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:38.651097+0000 mon.smithi113 (mon.0) 452 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:04:39.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:38.651987+0000 mon.smithi113 (mon.0) 453 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:39.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:04:38.652689+0000 mgr.smithi113.ctsazg (mgr.14194) 142 : cephadm [INF] Deploying daemon osd.5 on smithi192 2024-04-12T05:04:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:39.530896+0000 mgr.smithi113.ctsazg (mgr.14194) 143 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:40.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:39.530896+0000 mgr.smithi113.ctsazg (mgr.14194) 143 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:42.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:42 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:41.531303+0000 mgr.smithi113.ctsazg (mgr.14194) 144 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:42.933 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:42 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:41.531303+0000 mgr.smithi113.ctsazg (mgr.14194) 144 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:43.531631+0000 mgr.smithi113.ctsazg (mgr.14194) 145 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:43.916807+0000 mon.smithi113 (mon.0) 454 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:45.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:43.917076+0000 mon.smithi113 (mon.0) 455 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:04:45.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:43.531631+0000 mgr.smithi113.ctsazg (mgr.14194) 145 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:45.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:43.916807+0000 mon.smithi113 (mon.0) 454 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:45.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:43.917076+0000 mon.smithi113 (mon.0) 455 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:04:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:45.532019+0000 mgr.smithi113.ctsazg (mgr.14194) 146 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:45.532019+0000 mgr.smithi113.ctsazg (mgr.14194) 146 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:47.777 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.774+0000 7fbff57fa700 1 -- 172.21.15.192:0/3454758889 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbfec007e90 con 0x7fbfe0061180 2024-04-12T05:04:47.777 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 5 on host 'smithi192' 2024-04-12T05:04:47.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.776+0000 7fbffe9fe700 1 -- 172.21.15.192:0/3454758889 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fbfe0061180 msgr2=0x7fbfe0063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:47.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.776+0000 7fbffe9fe700 1 --2- 172.21.15.192:0/3454758889 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fbfe0061180 0x7fbfe0063640 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fbfec013510 tx=0x7fbfec009cb0).stop 2024-04-12T05:04:47.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.776+0000 7fbffe9fe700 1 -- 172.21.15.192:0/3454758889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbff80f7a10 msgr2=0x7fbff806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:47.779 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.776+0000 7fbffe9fe700 1 --2- 172.21.15.192:0/3454758889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbff80f7a10 0x7fbff806c9d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fbfe800b480 tx=0x7fbfe80090f0).stop 2024-04-12T05:04:47.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.776+0000 7fbffe9fe700 1 -- 172.21.15.192:0/3454758889 shutdown_connections 2024-04-12T05:04:47.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.776+0000 7fbffe9fe700 1 --2- 172.21.15.192:0/3454758889 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fbfe0061180 0x7fbfe0063640 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:47.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.776+0000 7fbffe9fe700 1 --2- 172.21.15.192:0/3454758889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbff80f7a10 0x7fbff806c9d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:47.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.776+0000 7fbffe9fe700 1 --2- 172.21.15.192:0/3454758889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff80f8cb0 0x7fbff806b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:04:47.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.776+0000 7fbffe9fe700 1 -- 172.21.15.192:0/3454758889 >> 172.21.15.192:0/3454758889 conn(0x7fbff80f3150 msgr2=0x7fbff80fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:47.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.777+0000 7fbffe9fe700 1 -- 172.21.15.192:0/3454758889 shutdown_connections 2024-04-12T05:04:47.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:04:47.777+0000 7fbffe9fe700 1 -- 172.21.15.192:0/3454758889 wait complete. 2024-04-12T05:04:48.684 DEBUG:teuthology.orchestra.run.smithi192:osd.5> sudo journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.5.service 2024-04-12T05:04:48.687 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-12T05:04:48.687 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd stat -f json 2024-04-12T05:04:48.716 INFO:journalctl@ceph.osd.5.smithi192.stdout:-- Logs begin at Fri 2024-04-12 04:48:43 UTC. -- 2024-04-12T05:04:48.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:47.532440+0000 mgr.smithi113.ctsazg (mgr.14194) 147 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:48.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:47.775047+0000 mon.smithi113 (mon.0) 456 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:48.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:47.811230+0000 mon.smithi192 (mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:04:48.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:47.811832+0000 mon.smithi113 (mon.0) 457 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:04:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:47.532440+0000 mgr.smithi113.ctsazg (mgr.14194) 147 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:47.775047+0000 mon.smithi113 (mon.0) 456 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:47.811230+0000 mon.smithi192 (mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:04:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:47.811832+0000 mon.smithi113 (mon.0) 457 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:04:50.108 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:04:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[59730]: debug 2024-04-12T05:04:49.793+0000 7f4408ff9700 -1 osd.5 0 waiting for initial osdmap 2024-04-12T05:04:50.109 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:04:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[59730]: debug 2024-04-12T05:04:49.801+0000 7f4402f70700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:04:50.109 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:48.779027+0000 mon.smithi113 (mon.0) 458 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T05:04:50.109 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:48.779104+0000 mon.smithi113 (mon.0) 459 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-04-12T05:04:50.109 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:48.779782+0000 mon.smithi113 (mon.0) 460 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:04:50.109 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:48.780479+0000 mon.smithi192 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:04:50.109 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:48.785621+0000 mon.smithi113 (mon.0) 461 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:04:50.109 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:49.639668+0000 mon.smithi113 (mon.0) 462 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:50.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:48.779027+0000 mon.smithi113 (mon.0) 458 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T05:04:50.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:48.779104+0000 mon.smithi113 (mon.0) 459 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-04-12T05:04:50.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:48.779782+0000 mon.smithi113 (mon.0) 460 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:04:50.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:48.780479+0000 mon.smithi192 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:04:50.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:48.785621+0000 mon.smithi113 (mon.0) 461 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:04:50.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:49.639668+0000 mon.smithi113 (mon.0) 462 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:50.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.818+0000 7fb75db0b700 1 -- 172.21.15.113:0/913902192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7580f8cb0 msgr2=0x7fb7580f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:50.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.818+0000 7fb75db0b700 1 --2- 172.21.15.113:0/913902192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7580f8cb0 0x7fb7580f9110 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fb7400047f0 tx=0x7fb740009cb0).stop 2024-04-12T05:04:50.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.818+0000 7fb75db0b700 1 -- 172.21.15.113:0/913902192 shutdown_connections 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.818+0000 7fb75db0b700 1 --2- 172.21.15.113:0/913902192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7580f8cb0 0x7fb7580f9110 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.818+0000 7fb75db0b700 1 --2- 172.21.15.113:0/913902192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7580f7a10 0x7fb7580f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.818+0000 7fb75db0b700 1 -- 172.21.15.113:0/913902192 >> 172.21.15.113:0/913902192 conn(0x7fb7580f3150 msgr2=0x7fb7580f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb75db0b700 1 -- 172.21.15.113:0/913902192 shutdown_connections 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb75db0b700 1 -- 172.21.15.113:0/913902192 wait complete. 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb75db0b700 1 Processor -- start 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb75db0b700 1 -- start start 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb75db0b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7580f7a10 0x7fb758108270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb75db0b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7580f8cb0 0x7fb758108760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb75db0b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb758108d10 con 0x7fb7580f7a10 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb75db0b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb758108e50 con 0x7fb7580f8cb0 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb7577fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7580f7a10 0x7fb758108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:50.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb7577fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7580f7a10 0x7fb758108270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40880/0 (socket says 172.21.15.113:40880) 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb7577fe700 1 -- 172.21.15.113:0/3618196201 learned_addr learned my addr 172.21.15.113:0/3618196201 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.819+0000 7fb74ffff700 1 --2- 172.21.15.113:0/3618196201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7580f8cb0 0x7fb758108760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.820+0000 7fb7577fe700 1 -- 172.21.15.113:0/3618196201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7580f8cb0 msgr2=0x7fb758108760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.820+0000 7fb7577fe700 1 --2- 172.21.15.113:0/3618196201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7580f8cb0 0x7fb758108760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.820+0000 7fb7577fe700 1 -- 172.21.15.113:0/3618196201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb740005040 con 0x7fb7580f7a10 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.820+0000 7fb7577fe700 1 --2- 172.21.15.113:0/3618196201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7580f7a10 0x7fb758108270 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb74800e8c0 tx=0x7fb74800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.820+0000 7fb7557fa700 1 -- 172.21.15.113:0/3618196201 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb748007b40 con 0x7fb7580f7a10 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.820+0000 7fb7557fa700 1 -- 172.21.15.113:0/3618196201 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fb74800f040 con 0x7fb7580f7a10 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.820+0000 7fb75db0b700 1 -- 172.21.15.113:0/3618196201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75810d8f0 con 0x7fb7580f7a10 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.820+0000 7fb7557fa700 1 -- 172.21.15.113:0/3618196201 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7480147e0 con 0x7fb7580f7a10 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.820+0000 7fb75db0b700 1 -- 172.21.15.113:0/3618196201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75810deb0 con 0x7fb7580f7a10 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.821+0000 7fb75db0b700 1 -- 172.21.15.113:0/3618196201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75805fb50 con 0x7fb7580f7a10 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.821+0000 7fb7557fa700 1 -- 172.21.15.113:0/3618196201 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fb7480090d0 con 0x7fb7580f7a10 2024-04-12T05:04:50.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.822+0000 7fb7557fa700 1 --2- 172.21.15.113:0/3618196201 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb744061180 0x7fb744063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:50.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.822+0000 7fb7557fa700 1 -- 172.21.15.113:0/3618196201 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4535+0+0 (secure 0 0 0) 0x7fb74807c1a0 con 0x7fb7580f7a10 2024-04-12T05:04:50.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.822+0000 7fb74ffff700 1 --2- 172.21.15.113:0/3618196201 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb744061180 0x7fb744063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:50.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.822+0000 7fb74ffff700 1 --2- 172.21.15.113:0/3618196201 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb744061180 0x7fb744063640 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb7400047c0 tx=0x7fb740009cb0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:50.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.826+0000 7fb7557fa700 1 -- 172.21.15.113:0/3618196201 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb748050470 con 0x7fb7580f7a10 2024-04-12T05:04:50.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:49.532735+0000 mgr.smithi113.ctsazg (mgr.14194) 148 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:50.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:49.789316+0000 mon.smithi113 (mon.0) 463 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-12T05:04:50.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:49.789374+0000 mon.smithi113 (mon.0) 464 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-04-12T05:04:50.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:49.789596+0000 mon.smithi113 (mon.0) 465 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:04:50.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:49.795088+0000 mon.smithi113 (mon.0) 466 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:04:50.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.990+0000 7fb75db0b700 1 -- 172.21.15.113:0/3618196201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fb7581001d0 con 0x7fb7580f7a10 2024-04-12T05:04:50.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.991+0000 7fb7557fa700 1 -- 172.21.15.113:0/3618196201 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v37) v1 ==== 74+0+130 (secure 0 0 0) 0x7fb7480190a0 con 0x7fb7580f7a10 2024-04-12T05:04:50.994 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:04:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.992+0000 7fb75db0b700 1 -- 172.21.15.113:0/3618196201 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb744061180 msgr2=0x7fb744063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.992+0000 7fb75db0b700 1 --2- 172.21.15.113:0/3618196201 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb744061180 0x7fb744063640 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb7400047c0 tx=0x7fb740009cb0).stop 2024-04-12T05:04:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.992+0000 7fb75db0b700 1 -- 172.21.15.113:0/3618196201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7580f7a10 msgr2=0x7fb758108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:50.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.992+0000 7fb75db0b700 1 --2- 172.21.15.113:0/3618196201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7580f7a10 0x7fb758108270 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb74800e8c0 tx=0x7fb74800b040).stop 2024-04-12T05:04:50.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.992+0000 7fb75db0b700 1 -- 172.21.15.113:0/3618196201 shutdown_connections 2024-04-12T05:04:50.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.992+0000 7fb75db0b700 1 --2- 172.21.15.113:0/3618196201 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb744061180 0x7fb744063640 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:50.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.992+0000 7fb75db0b700 1 --2- 172.21.15.113:0/3618196201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7580f7a10 0x7fb758108270 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:50.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.992+0000 7fb75db0b700 1 --2- 172.21.15.113:0/3618196201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7580f8cb0 0x7fb758108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:50.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.992+0000 7fb75db0b700 1 -- 172.21.15.113:0/3618196201 >> 172.21.15.113:0/3618196201 conn(0x7fb7580f3150 msgr2=0x7fb7580fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:50.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.993+0000 7fb75db0b700 1 -- 172.21.15.113:0/3618196201 shutdown_connections 2024-04-12T05:04:50.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:50.993+0000 7fb75db0b700 1 -- 172.21.15.113:0/3618196201 wait complete. 2024-04-12T05:04:51.160 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:49.532735+0000 mgr.smithi113.ctsazg (mgr.14194) 148 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-12T05:04:51.160 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:49.789316+0000 mon.smithi113 (mon.0) 463 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-12T05:04:51.160 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:49.789374+0000 mon.smithi113 (mon.0) 464 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-04-12T05:04:51.160 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:49.789596+0000 mon.smithi113 (mon.0) 465 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:04:51.160 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:49.795088+0000 mon.smithi113 (mon.0) 466 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:04:51.745 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":37,"num_osds":6,"num_up_osds":6,"osd_up_since":1712898290,"num_in_osds":6,"osd_in_since":1712898271,"num_remapped_pgs":0} 2024-04-12T05:04:51.746 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd dump --format=json 2024-04-12T05:04:52.013 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:04:52.100 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:48.830594+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:04:52.101 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:48.830650+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:04:52.101 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:50.792730+0000 mon.smithi113 (mon.0) 467 : cluster [INF] osd.5 [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323] boot 2024-04-12T05:04:52.101 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:50.792771+0000 mon.smithi113 (mon.0) 468 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-04-12T05:04:52.101 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:50.793563+0000 mon.smithi113 (mon.0) 469 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:04:52.101 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:50.991921+0000 mon.smithi113 (mon.0) 470 : audit [DBG] from='client.? 172.21.15.113:0/3618196201' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T05:04:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:48.830594+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-12T05:04:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:48.830650+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-12T05:04:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:50.792730+0000 mon.smithi113 (mon.0) 467 : cluster [INF] osd.5 [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323] boot 2024-04-12T05:04:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:50.792771+0000 mon.smithi113 (mon.0) 468 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-04-12T05:04:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:50.793563+0000 mon.smithi113 (mon.0) 469 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:04:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:50.991921+0000 mon.smithi113 (mon.0) 470 : audit [DBG] from='client.? 172.21.15.113:0/3618196201' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T05:04:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:51.533068+0000 mgr.smithi113.ctsazg (mgr.14194) 149 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:04:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:51.805251+0000 mon.smithi113 (mon.0) 471 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-04-12T05:04:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:51.533068+0000 mgr.smithi113.ctsazg (mgr.14194) 149 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:04:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:51.805251+0000 mon.smithi113 (mon.0) 471 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-04-12T05:04:53.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.965+0000 7f19d48c8700 1 -- 172.21.15.113:0/2179028606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19d00f8c40 msgr2=0x7f19d00f90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:53.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.965+0000 7f19d48c8700 1 --2- 172.21.15.113:0/2179028606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19d00f8c40 0x7f19d00f90a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f19c00047f0 tx=0x7f19c0009cb0).stop 2024-04-12T05:04:53.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.965+0000 7f19d48c8700 1 -- 172.21.15.113:0/2179028606 shutdown_connections 2024-04-12T05:04:53.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.965+0000 7f19d48c8700 1 --2- 172.21.15.113:0/2179028606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19d00f8c40 0x7f19d00f90a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:53.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.965+0000 7f19d48c8700 1 --2- 172.21.15.113:0/2179028606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19d00f79a0 0x7f19d00f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:53.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.965+0000 7f19d48c8700 1 -- 172.21.15.113:0/2179028606 >> 172.21.15.113:0/2179028606 conn(0x7f19d00f3100 msgr2=0x7f19d00f5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:53.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.965+0000 7f19d48c8700 1 -- 172.21.15.113:0/2179028606 shutdown_connections 2024-04-12T05:04:53.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.965+0000 7f19d48c8700 1 -- 172.21.15.113:0/2179028606 wait complete. 2024-04-12T05:04:53.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19d48c8700 1 Processor -- start 2024-04-12T05:04:53.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19d48c8700 1 -- start start 2024-04-12T05:04:53.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19d48c8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19d00f79a0 0x7f19d006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:53.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19d48c8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19d00f8c40 0x7f19d006b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:53.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19d48c8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d006cf80 con 0x7f19d00f8c40 2024-04-12T05:04:53.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19d48c8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d006b520 con 0x7f19d00f79a0 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19cdd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19d00f8c40 0x7f19d006b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19cdd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19d00f8c40 0x7f19d006b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40888/0 (socket says 172.21.15.113:40888) 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19cdd9b700 1 -- 172.21.15.113:0/3381737605 learned_addr learned my addr 172.21.15.113:0/3381737605 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19ce59c700 1 --2- 172.21.15.113:0/3381737605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19d00f79a0 0x7f19d006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19cdd9b700 1 -- 172.21.15.113:0/3381737605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19d00f79a0 msgr2=0x7f19d006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19cdd9b700 1 --2- 172.21.15.113:0/3381737605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19d00f79a0 0x7f19d006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.966+0000 7f19cdd9b700 1 -- 172.21.15.113:0/3381737605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c0005040 con 0x7f19d00f8c40 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.967+0000 7f19cdd9b700 1 --2- 172.21.15.113:0/3381737605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19d00f8c40 0x7f19d006b030 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f19c0013510 tx=0x7f19c0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.967+0000 7f19c77fe700 1 -- 172.21.15.113:0/3381737605 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19c000e030 con 0x7f19d00f8c40 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.967+0000 7f19c77fe700 1 -- 172.21.15.113:0/3381737605 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f19c0007e90 con 0x7f19d00f8c40 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.967+0000 7f19d48c8700 1 -- 172.21.15.113:0/3381737605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d006b740 con 0x7f19d00f8c40 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.967+0000 7f19c77fe700 1 -- 172.21.15.113:0/3381737605 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19c0018070 con 0x7f19d00f8c40 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.967+0000 7f19d48c8700 1 -- 172.21.15.113:0/3381737605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d006bc20 con 0x7f19d00f8c40 2024-04-12T05:04:53.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.968+0000 7f19d48c8700 1 -- 172.21.15.113:0/3381737605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d006be90 con 0x7f19d00f8c40 2024-04-12T05:04:53.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.968+0000 7f19c77fe700 1 -- 172.21.15.113:0/3381737605 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f19c001d940 con 0x7f19d00f8c40 2024-04-12T05:04:53.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.968+0000 7f19c77fe700 1 --2- 172.21.15.113:0/3381737605 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f19bc061180 0x7f19bc063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:53.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.969+0000 7f19c77fe700 1 -- 172.21.15.113:0/3381737605 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f19c007da10 con 0x7f19d00f8c40 2024-04-12T05:04:53.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.969+0000 7f19ce59c700 1 --2- 172.21.15.113:0/3381737605 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f19bc061180 0x7f19bc063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:53.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.969+0000 7f19ce59c700 1 --2- 172.21.15.113:0/3381737605 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f19bc061180 0x7f19bc063640 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f19d00f8aa0 tx=0x7f19b8009040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:53.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:53.973+0000 7f19c77fe700 1 -- 172.21.15.113:0/3381737605 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f19c0051d60 con 0x7f19d00f8c40 2024-04-12T05:04:54.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.137+0000 7f19d48c8700 1 -- 172.21.15.113:0/3381737605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19d00598f0 con 0x7f19d00f8c40 2024-04-12T05:04:54.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.138+0000 7f19c77fe700 1 -- 172.21.15.113:0/3381737605 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+10953 (secure 0 0 0) 0x7f19c0051950 con 0x7f19d00f8c40 2024-04-12T05:04:54.140 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:04:54.140 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":38,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","created":"2024-04-12T04:59:30.005034+0000","modified":"2024-04-12T05:04:51.796458+0000","last_up_change":"2024-04-12T05:04:50.786863+0000","last_in_change":"2024-04-12T05:04:31.248597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-04-12T05:03:31.839630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"fc942a17-378a-48a6-b1d3-4f5730550810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6803","nonce":1323860970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6805","nonce":1323860970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6809","nonce":1323860970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6807","nonce":1323860970}]},"public_addr":"172.21.15.113:6803/1323860970","cluster_addr":"172.21.15.113:6805/1323860970","heartbeat_back_addr":"172.21.15.113:6809/1323860970","heartbeat_front_addr":"172.21.15.113:6807/1323860970","state":["exists","up"]},{"osd":1,"uuid":"39fec899-371b-4d48-99a8-52f90f52a71f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6811","nonce":563293368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6813","nonce":563293368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6817","nonce":563293368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6815","nonce":563293368}]},"public_addr":"172.21.15.113:6811/563293368","cluster_addr":"172.21.15.113:6813/563293368","heartbeat_back_addr":"172.21.15.113:6817/563293368","heartbeat_front_addr":"172.21.15.113:6815/563293368","state":["exists","up"]},{"osd":2,"uuid":"b135f3a9-d2f0-4737-8a30-9f6dc3f2571b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6819","nonce":2998637711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6821","nonce":2998637711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6825","nonce":2998637711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6823","nonce":2998637711}]},"public_addr":"172.21.15.113:6819/2998637711","cluster_addr":"172.21.15.113:6821/2998637711","heartbeat_back_addr":"172.21.15.113:6825/2998637711","heartbeat_front_addr":"172.21.15.113:6823/2998637711","state":["exists","up"]},{"osd":3,"uuid":"49fd856c-c351-4e90-9088-ece0024cd2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6801","nonce":821756207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6803","nonce":821756207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6807","nonce":821756207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6805","nonce":821756207}]},"public_addr":"172.21.15.192:6801/821756207","cluster_addr":"172.21.15.192:6803/821756207","heartbeat_back_addr":"172.21.15.192:6807/821756207","heartbeat_front_addr":"172.21.15.192:6805/821756207","state":["exists","up"]},{"osd":4,"uuid":"80b72db8-682a-4a51-8d33-b0addda91fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6809","nonce":563933248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6811","nonce":563933248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6815","nonce":563933248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6813","nonce":563933248}]},"public_addr":"172.21.15.192:6809/563933248","cluster_addr":"172.21.15.192:6811/563933248","heartbeat_back_addr":"172.21.15.192:6815/563933248","heartbeat_front_addr":"172.21.15.192:6813/563933248","state":["exists","up"]},{"osd":5,"uuid":"c8e1e98f-5edb-4fec-bdf6-bcda40562682","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6817","nonce":2433186323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6819","nonce":2433186323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6823","nonce":2433186323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6821","nonce":2433186323}]},"public_addr":"172.21.15.192:6817/2433186323","cluster_addr":"172.21.15.192:6819/2433186323","heartbeat_back_addr":"172.21.15.192:6823/2433186323","heartbeat_front_addr":"172.21.15.192:6821/2433186323","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:02:27.514514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:01.358930+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:30.203057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:54.928612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:21.788000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:48.830652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.113:6801/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.113:6800/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/516367458":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/3213888755":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/4242055626":"2024-04-13T05:00:29.828900+0000","172.21.15.113:6801/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:6800/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/2902584436":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/1777682971":"2024-04-13T04:59:57.862720+0000","172.21.15.113:6800/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.113:6801/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.113:0/3450096338":"2024-04-13T04:59:57.862720+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T05:04:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.140+0000 7f19d48c8700 1 -- 172.21.15.113:0/3381737605 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f19bc061180 msgr2=0x7f19bc063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.140+0000 7f19d48c8700 1 --2- 172.21.15.113:0/3381737605 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f19bc061180 0x7f19bc063640 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f19d00f8aa0 tx=0x7f19b8009040).stop 2024-04-12T05:04:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.140+0000 7f19d48c8700 1 -- 172.21.15.113:0/3381737605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19d00f8c40 msgr2=0x7f19d006b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.140+0000 7f19d48c8700 1 --2- 172.21.15.113:0/3381737605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19d00f8c40 0x7f19d006b030 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f19c0013510 tx=0x7f19c0009cb0).stop 2024-04-12T05:04:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.140+0000 7f19d48c8700 1 -- 172.21.15.113:0/3381737605 shutdown_connections 2024-04-12T05:04:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.140+0000 7f19d48c8700 1 --2- 172.21.15.113:0/3381737605 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f19bc061180 0x7f19bc063640 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.140+0000 7f19d48c8700 1 --2- 172.21.15.113:0/3381737605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19d00f79a0 0x7f19d006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.140+0000 7f19d48c8700 1 --2- 172.21.15.113:0/3381737605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19d00f8c40 0x7f19d006b030 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.140+0000 7f19d48c8700 1 -- 172.21.15.113:0/3381737605 >> 172.21.15.113:0/3381737605 conn(0x7f19d00f3100 msgr2=0x7f19d00fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.140+0000 7f19d48c8700 1 -- 172.21.15.113:0/3381737605 shutdown_connections 2024-04-12T05:04:54.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:54.140+0000 7f19d48c8700 1 -- 172.21.15.113:0/3381737605 wait complete. 2024-04-12T05:04:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:04:52.978050+0000 mgr.smithi113.ctsazg (mgr.14194) 150 : cephadm [INF] Detected new or changed devices on smithi192 2024-04-12T05:04:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:52.982898+0000 mon.smithi113 (mon.0) 472 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:52.983775+0000 mon.smithi113 (mon.0) 473 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:52.984357+0000 mon.smithi113 (mon.0) 474 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:52.984905+0000 mon.smithi113 (mon.0) 475 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:04:52.985306+0000 mgr.smithi113.ctsazg (mgr.14194) 151 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 4982M 2024-04-12T05:04:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:52.990726+0000 mon.smithi113 (mon.0) 476 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:52.991446+0000 mon.smithi113 (mon.0) 477 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:54.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:52.991978+0000 mon.smithi113 (mon.0) 478 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:54.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:52.997620+0000 mon.smithi113 (mon.0) 479 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:54.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:53.001036+0000 mon.smithi113 (mon.0) 480 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:04:54.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:53.512869+0000 mon.smithi113 (mon.0) 481 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:54.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:53.513462+0000 mon.smithi113 (mon.0) 482 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:54.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:53.517402+0000 mon.smithi113 (mon.0) 483 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:54.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:04:52.978050+0000 mgr.smithi113.ctsazg (mgr.14194) 150 : cephadm [INF] Detected new or changed devices on smithi192 2024-04-12T05:04:54.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:52.982898+0000 mon.smithi113 (mon.0) 472 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:54.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:52.983775+0000 mon.smithi113 (mon.0) 473 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:54.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:52.984357+0000 mon.smithi113 (mon.0) 474 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:54.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:52.984905+0000 mon.smithi113 (mon.0) 475 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:04:54.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:04:52.985306+0000 mgr.smithi113.ctsazg (mgr.14194) 151 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 4982M 2024-04-12T05:04:54.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:52.990726+0000 mon.smithi113 (mon.0) 476 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:54.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:52.991446+0000 mon.smithi113 (mon.0) 477 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:54.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:52.991978+0000 mon.smithi113 (mon.0) 478 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:54.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:52.997620+0000 mon.smithi113 (mon.0) 479 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:54.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:53.001036+0000 mon.smithi113 (mon.0) 480 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:04:54.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:53.512869+0000 mon.smithi113 (mon.0) 481 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:04:54.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:53.513462+0000 mon.smithi113 (mon.0) 482 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:04:54.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:53.517402+0000 mon.smithi113 (mon.0) 483 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:04:54.860 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-04-12T05:03:31.839630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2024-04-12T05:04:54.861 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd pool get device_health_metrics pg_num 2024-04-12T05:04:55.132 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:04:55.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:53.533346+0000 mgr.smithi113.ctsazg (mgr.14194) 152 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:04:55.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:54.138524+0000 mon.smithi113 (mon.0) 484 : audit [DBG] from='client.? 172.21.15.113:0/3381737605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:04:55.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:53.533346+0000 mgr.smithi113.ctsazg (mgr.14194) 152 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:04:55.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:54.138524+0000 mon.smithi113 (mon.0) 484 : audit [DBG] from='client.? 172.21.15.113:0/3381737605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:04:56.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.810+0000 7f96b5d86700 1 -- 172.21.15.113:0/3055863308 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96b00f68a0 msgr2=0x7f96b00f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:56.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.810+0000 7f96b5d86700 1 --2- 172.21.15.113:0/3055863308 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96b00f68a0 0x7f96b00f6cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f96a40047f0 tx=0x7f96a4009cb0).stop 2024-04-12T05:04:56.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.810+0000 7f96b5d86700 1 -- 172.21.15.113:0/3055863308 shutdown_connections 2024-04-12T05:04:56.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.810+0000 7f96b5d86700 1 --2- 172.21.15.113:0/3055863308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96b00f7b40 0x7f96b00f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:04:56.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.810+0000 7f96b5d86700 1 --2- 172.21.15.113:0/3055863308 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96b00f68a0 0x7f96b00f6cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:56.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.810+0000 7f96b5d86700 1 -- 172.21.15.113:0/3055863308 >> 172.21.15.113:0/3055863308 conn(0x7f96b00f2020 msgr2=0x7f96b00f4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:56.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.810+0000 7f96b5d86700 1 -- 172.21.15.113:0/3055863308 shutdown_connections 2024-04-12T05:04:56.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.810+0000 7f96b5d86700 1 -- 172.21.15.113:0/3055863308 wait complete. 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.810+0000 7f96b5d86700 1 Processor -- start 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96b5d86700 1 -- start start 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96b5d86700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96b00f68a0 0x7f96b0107030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96b5d86700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96b00f7b40 0x7f96b0107520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96b5d86700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0107b60 con 0x7f96b00f68a0 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96b5d86700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0107ca0 con 0x7f96b00f7b40 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96af7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96b00f68a0 0x7f96b0107030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96af7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96b00f68a0 0x7f96b0107030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40902/0 (socket says 172.21.15.113:40902) 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96af7fe700 1 -- 172.21.15.113:0/340384960 learned_addr learned my addr 172.21.15.113:0/340384960 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96af7fe700 1 -- 172.21.15.113:0/340384960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96b00f7b40 msgr2=0x7f96b0107520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96af7fe700 1 --2- 172.21.15.113:0/340384960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96b00f7b40 0x7f96b0107520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:04:56.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96af7fe700 1 -- 172.21.15.113:0/340384960 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4005040 con 0x7f96b00f68a0 2024-04-12T05:04:56.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96af7fe700 1 --2- 172.21.15.113:0/340384960 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96b00f68a0 0x7f96b0107030 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f96a4009f20 tx=0x7f96a4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:56.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96acff9700 1 -- 172.21.15.113:0/340384960 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96a400e030 con 0x7f96b00f68a0 2024-04-12T05:04:56.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96b5d86700 1 -- 172.21.15.113:0/340384960 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b0060bb0 con 0x7f96b00f68a0 2024-04-12T05:04:56.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96acff9700 1 -- 172.21.15.113:0/340384960 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f96a4007e90 con 0x7f96b00f68a0 2024-04-12T05:04:56.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96acff9700 1 -- 172.21.15.113:0/340384960 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96a4018070 con 0x7f96b00f68a0 2024-04-12T05:04:56.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.811+0000 7f96b5d86700 1 -- 172.21.15.113:0/340384960 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b0061110 con 0x7f96b00f68a0 2024-04-12T05:04:56.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.812+0000 7f96b5d86700 1 -- 172.21.15.113:0/340384960 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b0061b00 con 0x7f96b00f68a0 2024-04-12T05:04:56.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.813+0000 7f96acff9700 1 -- 172.21.15.113:0/340384960 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f96a401d420 con 0x7f96b00f68a0 2024-04-12T05:04:56.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.813+0000 7f96acff9700 1 --2- 172.21.15.113:0/340384960 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9698061160 0x7f9698063620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:56.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.813+0000 7f96acff9700 1 -- 172.21.15.113:0/340384960 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f96a407da30 con 0x7f96b00f68a0 2024-04-12T05:04:56.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.813+0000 7f96aeffd700 1 --2- 172.21.15.113:0/340384960 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9698061160 0x7f9698063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:56.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.813+0000 7f96aeffd700 1 --2- 172.21.15.113:0/340384960 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9698061160 0x7f9698063620 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f96a0009640 tx=0x7f96a0006b60).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:56.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.817+0000 7f96acff9700 1 -- 172.21.15.113:0/340384960 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f96a4051ca0 con 0x7f96b00f68a0 2024-04-12T05:04:56.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.979+0000 7f96b5d86700 1 -- 172.21.15.113:0/340384960 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f96b0101080 con 0x7f96b00f68a0 2024-04-12T05:04:56.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.980+0000 7f96acff9700 1 -- 172.21.15.113:0/340384960 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v38) v1 ==== 110+0+10 (secure 0 0 0) 0x7f96a4051890 con 0x7f96b00f68a0 2024-04-12T05:04:56.982 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-04-12T05:04:56.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.982+0000 7f96b5d86700 1 -- 172.21.15.113:0/340384960 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9698061160 msgr2=0x7f9698063620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:56.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.982+0000 7f96b5d86700 1 --2- 172.21.15.113:0/340384960 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9698061160 0x7f9698063620 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f96a0009640 tx=0x7f96a0006b60).stop 2024-04-12T05:04:56.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.982+0000 7f96b5d86700 1 -- 172.21.15.113:0/340384960 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96b00f68a0 msgr2=0x7f96b0107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:56.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.982+0000 7f96b5d86700 1 --2- 172.21.15.113:0/340384960 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96b00f68a0 0x7f96b0107030 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f96a4009f20 tx=0x7f96a4009cb0).stop 2024-04-12T05:04:56.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.982+0000 7f96b5d86700 1 -- 172.21.15.113:0/340384960 shutdown_connections 2024-04-12T05:04:56.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.982+0000 7f96b5d86700 1 --2- 172.21.15.113:0/340384960 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9698061160 0x7f9698063620 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:56.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.982+0000 7f96b5d86700 1 --2- 172.21.15.113:0/340384960 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96b00f68a0 0x7f96b0107030 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:56.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.982+0000 7f96b5d86700 1 --2- 172.21.15.113:0/340384960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96b00f7b40 0x7f96b0107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:04:56.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.982+0000 7f96b5d86700 1 -- 172.21.15.113:0/340384960 >> 172.21.15.113:0/340384960 conn(0x7f96b00f2020 msgr2=0x7f96b00fb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:56.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.982+0000 7f96b5d86700 1 -- 172.21.15.113:0/340384960 shutdown_connections 2024-04-12T05:04:56.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:56.982+0000 7f96b5d86700 1 -- 172.21.15.113:0/340384960 wait complete. 2024-04-12T05:04:57.366 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:55.533773+0000 mgr.smithi113.ctsazg (mgr.14194) 153 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:04:57.367 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:56.980963+0000 mon.smithi113 (mon.0) 485 : audit [DBG] from='client.? 172.21.15.113:0/340384960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-04-12T05:04:57.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:55.533773+0000 mgr.smithi113.ctsazg (mgr.14194) 153 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:04:57.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:56.980963+0000 mon.smithi113 (mon.0) 485 : audit [DBG] from='client.? 172.21.15.113:0/340384960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-04-12T05:04:57.652 INFO:tasks.cephadm:Setting up client nodes... 2024-04-12T05:04:57.652 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-12T05:04:59.155 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:57.534467+0000 mgr.smithi113.ctsazg (mgr.14194) 154 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:04:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:57.534467+0000 mgr.smithi113.ctsazg (mgr.14194) 154 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:04:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.683+0000 7f01f1116700 1 -- 172.21.15.113:0/1193148073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec0f7990 msgr2=0x7f01ec0f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.683+0000 7f01f1116700 1 --2- 172.21.15.113:0/1193148073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec0f7990 0x7f01ec0f7df0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f01dc0047f0 tx=0x7f01dc009cb0).stop 2024-04-12T05:04:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.683+0000 7f01f1116700 1 -- 172.21.15.113:0/1193148073 shutdown_connections 2024-04-12T05:04:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.683+0000 7f01f1116700 1 --2- 172.21.15.113:0/1193148073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec0f7990 0x7f01ec0f7df0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.683+0000 7f01f1116700 1 --2- 172.21.15.113:0/1193148073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01ec0f66f0 0x7f01ec0f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:04:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.683+0000 7f01f1116700 1 -- 172.21.15.113:0/1193148073 >> 172.21.15.113:0/1193148073 conn(0x7f01ec0f1e50 msgr2=0x7f01ec0f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.683+0000 7f01f1116700 1 -- 172.21.15.113:0/1193148073 shutdown_connections 2024-04-12T05:04:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.683+0000 7f01f1116700 1 -- 172.21.15.113:0/1193148073 wait complete. 2024-04-12T05:04:59.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.684+0000 7f01f1116700 1 Processor -- start 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.684+0000 7f01f1116700 1 -- start start 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.684+0000 7f01f1116700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01ec0f66f0 0x7f01ec108000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.684+0000 7f01f1116700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec0f7990 0x7f01ec1084f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.684+0000 7f01f1116700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec108b30 con 0x7f01ec0f7990 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.684+0000 7f01f1116700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec108c70 con 0x7f01ec0f66f0 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.684+0000 7f01ea59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec0f7990 0x7f01ec1084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01ea59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec0f7990 0x7f01ec1084f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40930/0 (socket says 172.21.15.113:40930) 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01ea59c700 1 -- 172.21.15.113:0/698147217 learned_addr learned my addr 172.21.15.113:0/698147217 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01ead9d700 1 --2- 172.21.15.113:0/698147217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01ec0f66f0 0x7f01ec108000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01ea59c700 1 -- 172.21.15.113:0/698147217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01ec0f66f0 msgr2=0x7f01ec108000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01ea59c700 1 --2- 172.21.15.113:0/698147217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01ec0f66f0 0x7f01ec108000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01ea59c700 1 -- 172.21.15.113:0/698147217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01dc005040 con 0x7f01ec0f7990 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01ea59c700 1 --2- 172.21.15.113:0/698147217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec0f7990 0x7f01ec1084f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f01dc013510 tx=0x7f01dc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:59.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01d3fff700 1 -- 172.21.15.113:0/698147217 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01dc00e030 con 0x7f01ec0f7990 2024-04-12T05:04:59.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01f1116700 1 -- 172.21.15.113:0/698147217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec10d6b0 con 0x7f01ec0f7990 2024-04-12T05:04:59.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01d3fff700 1 -- 172.21.15.113:0/698147217 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f01dc007e90 con 0x7f01ec0f7990 2024-04-12T05:04:59.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01d3fff700 1 -- 172.21.15.113:0/698147217 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01dc018070 con 0x7f01ec0f7990 2024-04-12T05:04:59.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.685+0000 7f01f1116700 1 -- 172.21.15.113:0/698147217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec10db90 con 0x7f01ec0f7990 2024-04-12T05:04:59.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.686+0000 7f01f1116700 1 -- 172.21.15.113:0/698147217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ec0fef50 con 0x7f01ec0f7990 2024-04-12T05:04:59.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.686+0000 7f01d3fff700 1 -- 172.21.15.113:0/698147217 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f01dc01d940 con 0x7f01ec0f7990 2024-04-12T05:04:59.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.687+0000 7f01d3fff700 1 --2- 172.21.15.113:0/698147217 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f01d4061160 0x7f01d4063620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:04:59.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.687+0000 7f01d3fff700 1 -- 172.21.15.113:0/698147217 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f01dc07dac0 con 0x7f01ec0f7990 2024-04-12T05:04:59.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.687+0000 7f01ead9d700 1 --2- 172.21.15.113:0/698147217 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f01d4061160 0x7f01d4063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:04:59.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.687+0000 7f01ead9d700 1 --2- 172.21.15.113:0/698147217 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f01d4061160 0x7f01d4063620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f01e0006fd0 tx=0x7f01e0009060).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:04:59.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.691+0000 7f01d3fff700 1 -- 172.21.15.113:0/698147217 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f01dc051d90 con 0x7f01ec0f7990 2024-04-12T05:04:59.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.917+0000 7f01f1116700 1 -- 172.21.15.113:0/698147217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f01ec10e480 con 0x7f01ec0f7990 2024-04-12T05:04:59.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.920+0000 7f01d3fff700 1 -- 172.21.15.113:0/698147217 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f01dc051980 con 0x7f01ec0f7990 2024-04-12T05:04:59.922 INFO:teuthology.orchestra.run.smithi113.stdout:[client.0] 2024-04-12T05:04:59.922 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD7wBhmbB67NhAAoCyOydE2fNQ7Z3eGI1T/sg== 2024-04-12T05:04:59.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.922+0000 7f01f1116700 1 -- 172.21.15.113:0/698147217 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f01d4061160 msgr2=0x7f01d4063620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:59.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.922+0000 7f01f1116700 1 --2- 172.21.15.113:0/698147217 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f01d4061160 0x7f01d4063620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f01e0006fd0 tx=0x7f01e0009060).stop 2024-04-12T05:04:59.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.922+0000 7f01f1116700 1 -- 172.21.15.113:0/698147217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec0f7990 msgr2=0x7f01ec1084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:04:59.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.922+0000 7f01f1116700 1 --2- 172.21.15.113:0/698147217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec0f7990 0x7f01ec1084f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f01dc013510 tx=0x7f01dc009cb0).stop 2024-04-12T05:04:59.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.922+0000 7f01f1116700 1 -- 172.21.15.113:0/698147217 shutdown_connections 2024-04-12T05:04:59.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.922+0000 7f01f1116700 1 --2- 172.21.15.113:0/698147217 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f01d4061160 0x7f01d4063620 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:59.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.922+0000 7f01f1116700 1 --2- 172.21.15.113:0/698147217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01ec0f66f0 0x7f01ec108000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:59.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.922+0000 7f01f1116700 1 --2- 172.21.15.113:0/698147217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec0f7990 0x7f01ec1084f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:04:59.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.922+0000 7f01f1116700 1 -- 172.21.15.113:0/698147217 >> 172.21.15.113:0/698147217 conn(0x7f01ec0f1e50 msgr2=0x7f01ec060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:04:59.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.922+0000 7f01f1116700 1 -- 172.21.15.113:0/698147217 shutdown_connections 2024-04-12T05:04:59.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:04:59.922+0000 7f01f1116700 1 -- 172.21.15.113:0/698147217 wait complete. 2024-04-12T05:05:00.321 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:59.918080+0000 mon.smithi113 (mon.0) 486 : audit [INF] from='client.? 172.21.15.113:0/698147217' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:05:00.322 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:04:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:04:59.921429+0000 mon.smithi113 (mon.0) 487 : audit [INF] from='client.? 172.21.15.113:0/698147217' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T05:05:00.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:59.918080+0000 mon.smithi113 (mon.0) 486 : audit [INF] from='client.? 172.21.15.113:0/698147217' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:05:00.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:04:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:04:59.921429+0000 mon.smithi113 (mon.0) 487 : audit [INF] from='client.? 172.21.15.113:0/698147217' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T05:05:00.731 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:05:00.731 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-12T05:05:00.731 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-12T05:05:00.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-12T05:05:01.197 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:04:59.534830+0000 mgr.smithi113.ctsazg (mgr.14194) 155 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:04:59.534830+0000 mgr.smithi113.ctsazg (mgr.14194) 155 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:02.518 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.515+0000 7f2d7ca55700 1 -- 172.21.15.192:0/1875831772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d780f8cb0 msgr2=0x7f2d780f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:02.518 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.515+0000 7f2d7ca55700 1 --2- 172.21.15.192:0/1875831772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d780f8cb0 0x7f2d780f9110 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f2d680047f0 tx=0x7f2d68009cb0).stop 2024-04-12T05:05:02.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.516+0000 7f2d7ca55700 1 -- 172.21.15.192:0/1875831772 shutdown_connections 2024-04-12T05:05:02.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.516+0000 7f2d7ca55700 1 --2- 172.21.15.192:0/1875831772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d780f8cb0 0x7f2d780f9110 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:02.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.516+0000 7f2d7ca55700 1 --2- 172.21.15.192:0/1875831772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d780f7a10 0x7f2d780f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:02.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.516+0000 7f2d7ca55700 1 -- 172.21.15.192:0/1875831772 >> 172.21.15.192:0/1875831772 conn(0x7f2d780f3150 msgr2=0x7f2d780f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:02.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.516+0000 7f2d7ca55700 1 -- 172.21.15.192:0/1875831772 shutdown_connections 2024-04-12T05:05:02.519 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.516+0000 7f2d7ca55700 1 -- 172.21.15.192:0/1875831772 wait complete. 2024-04-12T05:05:02.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.516+0000 7f2d7ca55700 1 Processor -- start 2024-04-12T05:05:02.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.516+0000 7f2d7ca55700 1 -- start start 2024-04-12T05:05:02.520 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.516+0000 7f2d7ca55700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d780f7a10 0x7f2d78108150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d7ca55700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d780f8cb0 0x7f2d78108640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d7ca55700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d78108bf0 con 0x7f2d780f7a10 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d7ca55700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d78108d30 con 0x7f2d780f8cb0 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d75d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d780f8cb0 0x7f2d78108640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d75d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d780f8cb0 0x7f2d78108640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:33682/0 (socket says 172.21.15.192:33682) 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d75d9b700 1 -- 172.21.15.192:0/2526520652 learned_addr learned my addr 172.21.15.192:0/2526520652 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d75d9b700 1 -- 172.21.15.192:0/2526520652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d780f7a10 msgr2=0x7f2d78108150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d75d9b700 1 --2- 172.21.15.192:0/2526520652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d780f7a10 0x7f2d78108150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d75d9b700 1 -- 172.21.15.192:0/2526520652 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d68005040 con 0x7f2d780f8cb0 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d75d9b700 1 --2- 172.21.15.192:0/2526520652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d780f8cb0 0x7f2d78108640 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f2d68013510 tx=0x7f2d68009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d677fe700 1 -- 172.21.15.192:0/2526520652 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d6800e030 con 0x7f2d780f8cb0 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d7ca55700 1 -- 172.21.15.192:0/2526520652 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7810d770 con 0x7f2d780f8cb0 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d677fe700 1 -- 172.21.15.192:0/2526520652 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f2d68007e90 con 0x7f2d780f8cb0 2024-04-12T05:05:02.521 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d677fe700 1 -- 172.21.15.192:0/2526520652 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d68018070 con 0x7f2d780f8cb0 2024-04-12T05:05:02.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.517+0000 7f2d7ca55700 1 -- 172.21.15.192:0/2526520652 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7810dcb0 con 0x7f2d780f8cb0 2024-04-12T05:05:02.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.518+0000 7f2d7ca55700 1 -- 172.21.15.192:0/2526520652 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7805fb50 con 0x7f2d780f8cb0 2024-04-12T05:05:02.522 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.518+0000 7f2d677fe700 1 -- 172.21.15.192:0/2526520652 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f2d6801da20 con 0x7f2d780f8cb0 2024-04-12T05:05:02.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.519+0000 7f2d677fe700 1 --2- 172.21.15.192:0/2526520652 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2d60061180 0x7f2d60063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:02.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.519+0000 7f2d677fe700 1 -- 172.21.15.192:0/2526520652 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f2d6807db70 con 0x7f2d780f8cb0 2024-04-12T05:05:02.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.519+0000 7f2d7659c700 1 --2- 172.21.15.192:0/2526520652 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2d60061180 0x7f2d60063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:02.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.519+0000 7f2d7659c700 1 --2- 172.21.15.192:0/2526520652 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2d60061180 0x7f2d60063640 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f2d780f8b10 tx=0x7f2d6c007b60).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:02.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.523+0000 7f2d677fe700 1 -- 172.21.15.192:0/2526520652 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2d68051e40 con 0x7f2d780f8cb0 2024-04-12T05:05:02.749 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.746+0000 7f2d7ca55700 1 -- 172.21.15.192:0/2526520652 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f2d781001d0 con 0x7f2d780f8cb0 2024-04-12T05:05:02.753 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.751+0000 7f2d677fe700 1 -- 172.21.15.192:0/2526520652 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f2d68051a30 con 0x7f2d780f8cb0 2024-04-12T05:05:02.754 INFO:teuthology.orchestra.run.smithi192.stdout:[client.1] 2024-04-12T05:05:02.754 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD+wBhmeT6nLBAARoZqz4NgESU+NYPJceZSrw== 2024-04-12T05:05:02.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.753+0000 7f2d7ca55700 1 -- 172.21.15.192:0/2526520652 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2d60061180 msgr2=0x7f2d60063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:02.755 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.753+0000 7f2d7ca55700 1 --2- 172.21.15.192:0/2526520652 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2d60061180 0x7f2d60063640 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f2d780f8b10 tx=0x7f2d6c007b60).stop 2024-04-12T05:05:02.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.753+0000 7f2d7ca55700 1 -- 172.21.15.192:0/2526520652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d780f8cb0 msgr2=0x7f2d78108640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:02.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.753+0000 7f2d7ca55700 1 --2- 172.21.15.192:0/2526520652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d780f8cb0 0x7f2d78108640 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f2d68013510 tx=0x7f2d68009cb0).stop 2024-04-12T05:05:02.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.753+0000 7f2d7ca55700 1 -- 172.21.15.192:0/2526520652 shutdown_connections 2024-04-12T05:05:02.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.753+0000 7f2d7ca55700 1 --2- 172.21.15.192:0/2526520652 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2d60061180 0x7f2d60063640 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:02.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.753+0000 7f2d7ca55700 1 --2- 172.21.15.192:0/2526520652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d780f7a10 0x7f2d78108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:05:02.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.753+0000 7f2d7ca55700 1 --2- 172.21.15.192:0/2526520652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d780f8cb0 0x7f2d78108640 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:02.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.753+0000 7f2d7ca55700 1 -- 172.21.15.192:0/2526520652 >> 172.21.15.192:0/2526520652 conn(0x7f2d780f3150 msgr2=0x7f2d780fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:02.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.753+0000 7f2d7ca55700 1 -- 172.21.15.192:0/2526520652 shutdown_connections 2024-04-12T05:05:02.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-12T05:05:02.753+0000 7f2d7ca55700 1 -- 172.21.15.192:0/2526520652 wait complete. 2024-04-12T05:05:03.149 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:01.535247+0000 mgr.smithi113.ctsazg (mgr.14194) 156 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:03.150 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:02.748490+0000 mon.smithi192 (mon.1) 14 : audit [INF] from='client.? 172.21.15.192:0/2526520652' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:05:03.150 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:02.748974+0000 mon.smithi113 (mon.0) 488 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:05:03.150 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:02.752665+0000 mon.smithi113 (mon.0) 489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T05:05:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:01.535247+0000 mgr.smithi113.ctsazg (mgr.14194) 156 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:02.748490+0000 mon.smithi192 (mon.1) 14 : audit [INF] from='client.? 172.21.15.192:0/2526520652' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:05:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:02.748974+0000 mon.smithi113 (mon.0) 488 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T05:05:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:02.752665+0000 mon.smithi113 (mon.0) 489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T05:05:03.433 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T05:05:03.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-12T05:05:03.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-12T05:05:03.473 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-12T05:05:03.473 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-12T05:05:03.473 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph mgr dump --format=json 2024-04-12T05:05:03.749 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:05.368 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:03.535593+0000 mgr.smithi113.ctsazg (mgr.14194) 157 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:03.535593+0000 mgr.smithi113.ctsazg (mgr.14194) 157 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:05.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.619+0000 7f00b5bbb700 1 -- 172.21.15.113:0/1717464191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00b00f5c50 msgr2=0x7f00b00f6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:05.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.619+0000 7f00b5bbb700 1 --2- 172.21.15.113:0/1717464191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00b00f5c50 0x7f00b00f6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f00a40047f0 tx=0x7f00a4009cb0).stop 2024-04-12T05:05:05.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.619+0000 7f00b5bbb700 1 -- 172.21.15.113:0/1717464191 shutdown_connections 2024-04-12T05:05:05.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.619+0000 7f00b5bbb700 1 --2- 172.21.15.113:0/1717464191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00f65f0 0x7f00b00f8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:05.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.619+0000 7f00b5bbb700 1 --2- 172.21.15.113:0/1717464191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00b00f5c50 0x7f00b00f6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.619+0000 7f00b5bbb700 1 -- 172.21.15.113:0/1717464191 >> 172.21.15.113:0/1717464191 conn(0x7f00b00f1770 msgr2=0x7f00b00f3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.619+0000 7f00b5bbb700 1 -- 172.21.15.113:0/1717464191 shutdown_connections 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.619+0000 7f00b5bbb700 1 -- 172.21.15.113:0/1717464191 wait complete. 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.619+0000 7f00b5bbb700 1 Processor -- start 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00b5bbb700 1 -- start start 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00b5bbb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00f5c50 0x7f00b01080f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00b5bbb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00b00f65f0 0x7f00b01085e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00b5bbb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b0108b90 con 0x7f00b00f65f0 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00b5bbb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b0108cd0 con 0x7f00b00f5c50 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00aeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00b00f65f0 0x7f00b01085e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00aeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00b00f65f0 0x7f00b01085e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51912/0 (socket says 172.21.15.113:51912) 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00aeffd700 1 -- 172.21.15.113:0/3909520992 learned_addr learned my addr 172.21.15.113:0/3909520992 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:05.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00af7fe700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00f5c50 0x7f00b01080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00aeffd700 1 -- 172.21.15.113:0/3909520992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00f5c50 msgr2=0x7f00b01080f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00aeffd700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00f5c50 0x7f00b01080f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00aeffd700 1 -- 172.21.15.113:0/3909520992 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a4005040 con 0x7f00b00f65f0 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00af7fe700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00f5c50 0x7f00b01080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00aeffd700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00b00f65f0 0x7f00b01085e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f00a0002780 tx=0x7f00a0009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00acff9700 1 -- 172.21.15.113:0/3909520992 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00a0007b40 con 0x7f00b00f65f0 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00acff9700 1 -- 172.21.15.113:0/3909520992 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f00a000f040 con 0x7f00b00f65f0 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00b5bbb700 1 -- 172.21.15.113:0/3909520992 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b010d770 con 0x7f00b00f65f0 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.620+0000 7f00acff9700 1 -- 172.21.15.113:0/3909520992 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00a00146f0 con 0x7f00b00f65f0 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.621+0000 7f00b5bbb700 1 -- 172.21.15.113:0/3909520992 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b010dcb0 con 0x7f00b00f65f0 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.622+0000 7f00b5bbb700 1 -- 172.21.15.113:0/3909520992 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b0100130 con 0x7f00b00f65f0 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.622+0000 7f00acff9700 1 -- 172.21.15.113:0/3909520992 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f00a0007ca0 con 0x7f00b00f65f0 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.622+0000 7f00acff9700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0098061180 0x7f0098063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:05.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.622+0000 7f00af7fe700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0098061180 0x7f0098063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:05.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.622+0000 7f00acff9700 1 -- 172.21.15.113:0/3909520992 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f00a007c550 con 0x7f00b00f65f0 2024-04-12T05:05:05.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.623+0000 7f00af7fe700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0098061180 0x7f0098063640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f00a4013470 tx=0x7f00a400b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:05.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.627+0000 7f00acff9700 1 -- 172.21.15.113:0/3909520992 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f00a0050820 con 0x7f00b00f65f0 2024-04-12T05:05:05.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.832+0000 7f00b5bbb700 1 -- 172.21.15.113:0/3909520992 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f00b0101fa0 con 0x7f00b00f65f0 2024-04-12T05:05:05.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.834+0000 7f00acff9700 1 -- 172.21.15.113:0/3909520992 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v18) v1 ==== 74+0+152029 (secure 0 0 0) 0x7f00a0050410 con 0x7f00b00f65f0 2024-04-12T05:05:05.838 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:05:05.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.837+0000 7f00b5bbb700 1 -- 172.21.15.113:0/3909520992 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0098061180 msgr2=0x7f0098063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:05.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.837+0000 7f00b5bbb700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0098061180 0x7f0098063640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f00a4013470 tx=0x7f00a400b040).stop 2024-04-12T05:05:05.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.837+0000 7f00b5bbb700 1 -- 172.21.15.113:0/3909520992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00b00f65f0 msgr2=0x7f00b01085e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:05.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.837+0000 7f00b5bbb700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00b00f65f0 0x7f00b01085e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f00a0002780 tx=0x7f00a0009ee0).stop 2024-04-12T05:05:05.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.837+0000 7f00b5bbb700 1 -- 172.21.15.113:0/3909520992 shutdown_connections 2024-04-12T05:05:05.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.837+0000 7f00b5bbb700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0098061180 0x7f0098063640 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:05.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.837+0000 7f00b5bbb700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b00f5c50 0x7f00b01080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:05.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.837+0000 7f00b5bbb700 1 --2- 172.21.15.113:0/3909520992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00b00f65f0 0x7f00b01085e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:05.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.837+0000 7f00b5bbb700 1 -- 172.21.15.113:0/3909520992 >> 172.21.15.113:0/3909520992 conn(0x7f00b00f1770 msgr2=0x7f00b00fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:05.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.837+0000 7f00b5bbb700 1 -- 172.21.15.113:0/3909520992 shutdown_connections 2024-04-12T05:05:05.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:05.837+0000 7f00b5bbb700 1 -- 172.21.15.113:0/3909520992 wait complete. 2024-04-12T05:05:06.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:05.833131+0000 mon.smithi113 (mon.0) 490 : audit [DBG] from='client.? 172.21.15.113:0/3909520992' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-12T05:05:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:05.833131+0000 mon.smithi113 (mon.0) 490 : audit [DBG] from='client.? 172.21.15.113:0/3909520992' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-12T05:05:06.664 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":18,"active_gid":14194,"active_name":"smithi113.ctsazg","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":593576174},{"type":"v1","addr":"172.21.15.113:6801","nonce":593576174}]},"active_addr":"172.21.15.113:6801/593576174","active_change":"2024-04-12T05:01:23.491720+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":14208,"name":"smithi192.ocppyo","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.113:8443/","prometheus":"http://172.21.15.113:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.113:0","nonce":781838919}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.113:0","nonce":4036112881}]}]} 2024-04-12T05:05:06.668 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-12T05:05:06.668 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-12T05:05:06.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd dump --format=json 2024-04-12T05:05:06.937 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:05.535933+0000 mgr.smithi113.ctsazg (mgr.14194) 158 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:05.535933+0000 mgr.smithi113.ctsazg (mgr.14194) 158 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:08.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 -- 172.21.15.113:0/2333427822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88e40f7b40 msgr2=0x7f88e40f7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:08.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 --2- 172.21.15.113:0/2333427822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88e40f7b40 0x7f88e40f7fa0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f88d80047f0 tx=0x7f88d8009cb0).stop 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 -- 172.21.15.113:0/2333427822 shutdown_connections 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 --2- 172.21.15.113:0/2333427822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88e40f7b40 0x7f88e40f7fa0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 --2- 172.21.15.113:0/2333427822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e40f68a0 0x7f88e40f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 -- 172.21.15.113:0/2333427822 >> 172.21.15.113:0/2333427822 conn(0x7f88e40f2020 msgr2=0x7f88e40f4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 -- 172.21.15.113:0/2333427822 shutdown_connections 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 -- 172.21.15.113:0/2333427822 wait complete. 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 Processor -- start 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 -- start start 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88e40f68a0 0x7f88e4108130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.786+0000 7f88ec730700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e40f7b40 0x7f88e4108620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88ec730700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e4108c60 con 0x7f88e40f68a0 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88ec730700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e4108da0 con 0x7f88e40f7b40 2024-04-12T05:05:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88ea4cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88e40f68a0 0x7f88e4108130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88ea4cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88e40f68a0 0x7f88e4108130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51932/0 (socket says 172.21.15.113:51932) 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88ea4cc700 1 -- 172.21.15.113:0/4186532481 learned_addr learned my addr 172.21.15.113:0/4186532481 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88e9ccb700 1 --2- 172.21.15.113:0/4186532481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e40f7b40 0x7f88e4108620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88ea4cc700 1 -- 172.21.15.113:0/4186532481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e40f7b40 msgr2=0x7f88e4108620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88ea4cc700 1 --2- 172.21.15.113:0/4186532481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e40f7b40 0x7f88e4108620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88ea4cc700 1 -- 172.21.15.113:0/4186532481 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d8005040 con 0x7f88e40f68a0 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88ea4cc700 1 --2- 172.21.15.113:0/4186532481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88e40f68a0 0x7f88e4108130 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f88e000b480 tx=0x7f88e00090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88d77fe700 1 -- 172.21.15.113:0/4186532481 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88e0002500 con 0x7f88e40f68a0 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88d77fe700 1 -- 172.21.15.113:0/4186532481 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f88e0002af0 con 0x7f88e40f68a0 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88ec730700 1 -- 172.21.15.113:0/4186532481 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e410d840 con 0x7f88e40f68a0 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88d77fe700 1 -- 172.21.15.113:0/4186532481 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88e0012c40 con 0x7f88e40f68a0 2024-04-12T05:05:08.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.787+0000 7f88ec730700 1 -- 172.21.15.113:0/4186532481 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e410dd80 con 0x7f88e40f68a0 2024-04-12T05:05:08.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.788+0000 7f88ec730700 1 -- 172.21.15.113:0/4186532481 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e40ff090 con 0x7f88e40f68a0 2024-04-12T05:05:08.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.789+0000 7f88d77fe700 1 -- 172.21.15.113:0/4186532481 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f88e0002660 con 0x7f88e40f68a0 2024-04-12T05:05:08.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.789+0000 7f88d77fe700 1 --2- 172.21.15.113:0/4186532481 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f88d0061180 0x7f88d0063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:08.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.789+0000 7f88d77fe700 1 -- 172.21.15.113:0/4186532481 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f88e007c9f0 con 0x7f88e40f68a0 2024-04-12T05:05:08.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.789+0000 7f88e9ccb700 1 --2- 172.21.15.113:0/4186532481 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f88d0061180 0x7f88d0063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:08.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.790+0000 7f88e9ccb700 1 --2- 172.21.15.113:0/4186532481 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f88d0061180 0x7f88d0063640 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f88d8007cf0 tx=0x7f88d800b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:08.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.793+0000 7f88d77fe700 1 -- 172.21.15.113:0/4186532481 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f88e0050d40 con 0x7f88e40f68a0 2024-04-12T05:05:08.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.957+0000 7f88ec730700 1 -- 172.21.15.113:0/4186532481 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88e410e3c0 con 0x7f88e40f68a0 2024-04-12T05:05:08.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.958+0000 7f88d77fe700 1 -- 172.21.15.113:0/4186532481 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+10953 (secure 0 0 0) 0x7f88e0050930 con 0x7f88e40f68a0 2024-04-12T05:05:08.962 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:05:08.962 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":38,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","created":"2024-04-12T04:59:30.005034+0000","modified":"2024-04-12T05:04:51.796458+0000","last_up_change":"2024-04-12T05:04:50.786863+0000","last_in_change":"2024-04-12T05:04:31.248597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-04-12T05:03:31.839630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"fc942a17-378a-48a6-b1d3-4f5730550810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6803","nonce":1323860970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6805","nonce":1323860970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6809","nonce":1323860970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6807","nonce":1323860970}]},"public_addr":"172.21.15.113:6803/1323860970","cluster_addr":"172.21.15.113:6805/1323860970","heartbeat_back_addr":"172.21.15.113:6809/1323860970","heartbeat_front_addr":"172.21.15.113:6807/1323860970","state":["exists","up"]},{"osd":1,"uuid":"39fec899-371b-4d48-99a8-52f90f52a71f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6811","nonce":563293368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6813","nonce":563293368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6817","nonce":563293368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6815","nonce":563293368}]},"public_addr":"172.21.15.113:6811/563293368","cluster_addr":"172.21.15.113:6813/563293368","heartbeat_back_addr":"172.21.15.113:6817/563293368","heartbeat_front_addr":"172.21.15.113:6815/563293368","state":["exists","up"]},{"osd":2,"uuid":"b135f3a9-d2f0-4737-8a30-9f6dc3f2571b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6819","nonce":2998637711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6821","nonce":2998637711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6825","nonce":2998637711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6823","nonce":2998637711}]},"public_addr":"172.21.15.113:6819/2998637711","cluster_addr":"172.21.15.113:6821/2998637711","heartbeat_back_addr":"172.21.15.113:6825/2998637711","heartbeat_front_addr":"172.21.15.113:6823/2998637711","state":["exists","up"]},{"osd":3,"uuid":"49fd856c-c351-4e90-9088-ece0024cd2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6801","nonce":821756207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6803","nonce":821756207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6807","nonce":821756207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6805","nonce":821756207}]},"public_addr":"172.21.15.192:6801/821756207","cluster_addr":"172.21.15.192:6803/821756207","heartbeat_back_addr":"172.21.15.192:6807/821756207","heartbeat_front_addr":"172.21.15.192:6805/821756207","state":["exists","up"]},{"osd":4,"uuid":"80b72db8-682a-4a51-8d33-b0addda91fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6809","nonce":563933248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6811","nonce":563933248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6815","nonce":563933248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6813","nonce":563933248}]},"public_addr":"172.21.15.192:6809/563933248","cluster_addr":"172.21.15.192:6811/563933248","heartbeat_back_addr":"172.21.15.192:6815/563933248","heartbeat_front_addr":"172.21.15.192:6813/563933248","state":["exists","up"]},{"osd":5,"uuid":"c8e1e98f-5edb-4fec-bdf6-bcda40562682","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6817","nonce":2433186323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6819","nonce":2433186323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6823","nonce":2433186323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6821","nonce":2433186323}]},"public_addr":"172.21.15.192:6817/2433186323","cluster_addr":"172.21.15.192:6819/2433186323","heartbeat_back_addr":"172.21.15.192:6823/2433186323","heartbeat_front_addr":"172.21.15.192:6821/2433186323","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:02:27.514514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:01.358930+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:30.203057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:54.928612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:21.788000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:48.830652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.113:6801/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.113:6800/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/516367458":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/3213888755":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/4242055626":"2024-04-13T05:00:29.828900+0000","172.21.15.113:6801/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:6800/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/2902584436":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/1777682971":"2024-04-13T04:59:57.862720+0000","172.21.15.113:6800/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.113:6801/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.113:0/3450096338":"2024-04-13T04:59:57.862720+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T05:05:08.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.960+0000 7f88ec730700 1 -- 172.21.15.113:0/4186532481 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f88d0061180 msgr2=0x7f88d0063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:08.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.960+0000 7f88ec730700 1 --2- 172.21.15.113:0/4186532481 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f88d0061180 0x7f88d0063640 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f88d8007cf0 tx=0x7f88d800b040).stop 2024-04-12T05:05:08.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.960+0000 7f88ec730700 1 -- 172.21.15.113:0/4186532481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88e40f68a0 msgr2=0x7f88e4108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:08.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.960+0000 7f88ec730700 1 --2- 172.21.15.113:0/4186532481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88e40f68a0 0x7f88e4108130 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f88e000b480 tx=0x7f88e00090f0).stop 2024-04-12T05:05:08.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.960+0000 7f88ec730700 1 -- 172.21.15.113:0/4186532481 shutdown_connections 2024-04-12T05:05:08.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.960+0000 7f88ec730700 1 --2- 172.21.15.113:0/4186532481 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f88d0061180 0x7f88d0063640 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:08.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.960+0000 7f88ec730700 1 --2- 172.21.15.113:0/4186532481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88e40f68a0 0x7f88e4108130 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:08.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.960+0000 7f88ec730700 1 --2- 172.21.15.113:0/4186532481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e40f7b40 0x7f88e4108620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:08.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.960+0000 7f88ec730700 1 -- 172.21.15.113:0/4186532481 >> 172.21.15.113:0/4186532481 conn(0x7f88e40f2020 msgr2=0x7f88e4061310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:08.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.960+0000 7f88ec730700 1 -- 172.21.15.113:0/4186532481 shutdown_connections 2024-04-12T05:05:08.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:08.960+0000 7f88ec730700 1 -- 172.21.15.113:0/4186532481 wait complete. 2024-04-12T05:05:09.329 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:07.536216+0000 mgr.smithi113.ctsazg (mgr.14194) 159 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:09.329 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:08.958897+0000 mon.smithi113 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.113:0/4186532481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:05:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:07.536216+0000 mgr.smithi113.ctsazg (mgr.14194) 159 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:08.958897+0000 mon.smithi113 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.113:0/4186532481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:05:09.719 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-12T05:05:09.719 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd dump --format=json 2024-04-12T05:05:09.982 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:11.389 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:09.536512+0000 mgr.smithi113.ctsazg (mgr.14194) 160 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:09.536512+0000 mgr.smithi113.ctsazg (mgr.14194) 160 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:11.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.933+0000 7f71d1ed9700 1 -- 172.21.15.113:0/3743035778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71cc0f79a0 msgr2=0x7f71cc0f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:11.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.933+0000 7f71d1ed9700 1 --2- 172.21.15.113:0/3743035778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71cc0f79a0 0x7f71cc0f7dc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f71c00047f0 tx=0x7f71c0009cb0).stop 2024-04-12T05:05:11.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.933+0000 7f71d1ed9700 1 -- 172.21.15.113:0/3743035778 shutdown_connections 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.933+0000 7f71d1ed9700 1 --2- 172.21.15.113:0/3743035778 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71cc0f8c40 0x7f71cc0f90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.933+0000 7f71d1ed9700 1 --2- 172.21.15.113:0/3743035778 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71cc0f79a0 0x7f71cc0f7dc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.933+0000 7f71d1ed9700 1 -- 172.21.15.113:0/3743035778 >> 172.21.15.113:0/3743035778 conn(0x7f71cc0f3100 msgr2=0x7f71cc0f5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.933+0000 7f71d1ed9700 1 -- 172.21.15.113:0/3743035778 shutdown_connections 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.933+0000 7f71d1ed9700 1 -- 172.21.15.113:0/3743035778 wait complete. 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71d1ed9700 1 Processor -- start 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71d1ed9700 1 -- start start 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71d1ed9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71cc0f79a0 0x7f71cc1080d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71d1ed9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71cc0f8c40 0x7f71cc1085c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71d1ed9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc108b70 con 0x7f71cc0f79a0 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71d1ed9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc108cb0 con 0x7f71cc0f8c40 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71cb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71cc0f79a0 0x7f71cc1080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:11.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71cb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71cc0f79a0 0x7f71cc1080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52638/0 (socket says 172.21.15.113:52638) 2024-04-12T05:05:11.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71cb7fe700 1 -- 172.21.15.113:0/1003247761 learned_addr learned my addr 172.21.15.113:0/1003247761 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:11.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71caffd700 1 --2- 172.21.15.113:0/1003247761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71cc0f8c40 0x7f71cc1085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:11.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71cb7fe700 1 -- 172.21.15.113:0/1003247761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71cc0f8c40 msgr2=0x7f71cc1085c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:11.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71cb7fe700 1 --2- 172.21.15.113:0/1003247761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71cc0f8c40 0x7f71cc1085c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:11.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.934+0000 7f71cb7fe700 1 -- 172.21.15.113:0/1003247761 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c0005040 con 0x7f71cc0f79a0 2024-04-12T05:05:11.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.935+0000 7f71cb7fe700 1 --2- 172.21.15.113:0/1003247761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71cc0f79a0 0x7f71cc1080d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f71c0009eb0 tx=0x7f71c0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:11.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.935+0000 7f71c8ff9700 1 -- 172.21.15.113:0/1003247761 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71c000e030 con 0x7f71cc0f79a0 2024-04-12T05:05:11.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.935+0000 7f71c8ff9700 1 -- 172.21.15.113:0/1003247761 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f71c0007e90 con 0x7f71cc0f79a0 2024-04-12T05:05:11.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.935+0000 7f71d1ed9700 1 -- 172.21.15.113:0/1003247761 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71cc10d6f0 con 0x7f71cc0f79a0 2024-04-12T05:05:11.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.935+0000 7f71c8ff9700 1 -- 172.21.15.113:0/1003247761 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71c0018070 con 0x7f71cc0f79a0 2024-04-12T05:05:11.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.935+0000 7f71d1ed9700 1 -- 172.21.15.113:0/1003247761 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71cc10dc30 con 0x7f71cc0f79a0 2024-04-12T05:05:11.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.936+0000 7f71d1ed9700 1 -- 172.21.15.113:0/1003247761 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71cc100110 con 0x7f71cc0f79a0 2024-04-12T05:05:11.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.936+0000 7f71c8ff9700 1 -- 172.21.15.113:0/1003247761 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f71c001d420 con 0x7f71cc0f79a0 2024-04-12T05:05:11.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.936+0000 7f71c8ff9700 1 --2- 172.21.15.113:0/1003247761 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f71b4061110 0x7f71b40635d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:11.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.937+0000 7f71c8ff9700 1 -- 172.21.15.113:0/1003247761 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f71c007d860 con 0x7f71cc0f79a0 2024-04-12T05:05:11.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.937+0000 7f71caffd700 1 --2- 172.21.15.113:0/1003247761 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f71b4061110 0x7f71b40635d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:11.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.937+0000 7f71caffd700 1 --2- 172.21.15.113:0/1003247761 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f71b4061110 0x7f71b40635d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f71bc004de0 tx=0x7f71bc00b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:11.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:11.941+0000 7f71c8ff9700 1 -- 172.21.15.113:0/1003247761 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f71c0051b30 con 0x7f71cc0f79a0 2024-04-12T05:05:12.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.104+0000 7f71d1ed9700 1 -- 172.21.15.113:0/1003247761 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71cc0598f0 con 0x7f71cc0f79a0 2024-04-12T05:05:12.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.105+0000 7f71c8ff9700 1 -- 172.21.15.113:0/1003247761 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+10953 (secure 0 0 0) 0x7f71c0051720 con 0x7f71cc0f79a0 2024-04-12T05:05:12.107 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:05:12.107 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":38,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","created":"2024-04-12T04:59:30.005034+0000","modified":"2024-04-12T05:04:51.796458+0000","last_up_change":"2024-04-12T05:04:50.786863+0000","last_in_change":"2024-04-12T05:04:31.248597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-04-12T05:03:31.839630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"fc942a17-378a-48a6-b1d3-4f5730550810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6803","nonce":1323860970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6805","nonce":1323860970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6809","nonce":1323860970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6807","nonce":1323860970}]},"public_addr":"172.21.15.113:6803/1323860970","cluster_addr":"172.21.15.113:6805/1323860970","heartbeat_back_addr":"172.21.15.113:6809/1323860970","heartbeat_front_addr":"172.21.15.113:6807/1323860970","state":["exists","up"]},{"osd":1,"uuid":"39fec899-371b-4d48-99a8-52f90f52a71f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6811","nonce":563293368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6813","nonce":563293368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6817","nonce":563293368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6815","nonce":563293368}]},"public_addr":"172.21.15.113:6811/563293368","cluster_addr":"172.21.15.113:6813/563293368","heartbeat_back_addr":"172.21.15.113:6817/563293368","heartbeat_front_addr":"172.21.15.113:6815/563293368","state":["exists","up"]},{"osd":2,"uuid":"b135f3a9-d2f0-4737-8a30-9f6dc3f2571b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6819","nonce":2998637711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6821","nonce":2998637711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6825","nonce":2998637711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6823","nonce":2998637711}]},"public_addr":"172.21.15.113:6819/2998637711","cluster_addr":"172.21.15.113:6821/2998637711","heartbeat_back_addr":"172.21.15.113:6825/2998637711","heartbeat_front_addr":"172.21.15.113:6823/2998637711","state":["exists","up"]},{"osd":3,"uuid":"49fd856c-c351-4e90-9088-ece0024cd2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6801","nonce":821756207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6803","nonce":821756207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6807","nonce":821756207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6805","nonce":821756207}]},"public_addr":"172.21.15.192:6801/821756207","cluster_addr":"172.21.15.192:6803/821756207","heartbeat_back_addr":"172.21.15.192:6807/821756207","heartbeat_front_addr":"172.21.15.192:6805/821756207","state":["exists","up"]},{"osd":4,"uuid":"80b72db8-682a-4a51-8d33-b0addda91fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6809","nonce":563933248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6811","nonce":563933248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6815","nonce":563933248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6813","nonce":563933248}]},"public_addr":"172.21.15.192:6809/563933248","cluster_addr":"172.21.15.192:6811/563933248","heartbeat_back_addr":"172.21.15.192:6815/563933248","heartbeat_front_addr":"172.21.15.192:6813/563933248","state":["exists","up"]},{"osd":5,"uuid":"c8e1e98f-5edb-4fec-bdf6-bcda40562682","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6817","nonce":2433186323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6819","nonce":2433186323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6823","nonce":2433186323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6821","nonce":2433186323}]},"public_addr":"172.21.15.192:6817/2433186323","cluster_addr":"172.21.15.192:6819/2433186323","heartbeat_back_addr":"172.21.15.192:6823/2433186323","heartbeat_front_addr":"172.21.15.192:6821/2433186323","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:02:27.514514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:01.358930+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:30.203057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:54.928612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:21.788000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:48.830652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.113:6801/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.113:6800/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/516367458":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/3213888755":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/4242055626":"2024-04-13T05:00:29.828900+0000","172.21.15.113:6801/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:6800/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/2902584436":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/1777682971":"2024-04-13T04:59:57.862720+0000","172.21.15.113:6800/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.113:6801/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.113:0/3450096338":"2024-04-13T04:59:57.862720+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T05:05:12.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.106+0000 7f71d1ed9700 1 -- 172.21.15.113:0/1003247761 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f71b4061110 msgr2=0x7f71b40635d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:12.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.106+0000 7f71d1ed9700 1 --2- 172.21.15.113:0/1003247761 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f71b4061110 0x7f71b40635d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f71bc004de0 tx=0x7f71bc00b040).stop 2024-04-12T05:05:12.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.106+0000 7f71d1ed9700 1 -- 172.21.15.113:0/1003247761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71cc0f79a0 msgr2=0x7f71cc1080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:12.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.106+0000 7f71d1ed9700 1 --2- 172.21.15.113:0/1003247761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71cc0f79a0 0x7f71cc1080d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f71c0009eb0 tx=0x7f71c0009cb0).stop 2024-04-12T05:05:12.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.106+0000 7f71d1ed9700 1 -- 172.21.15.113:0/1003247761 shutdown_connections 2024-04-12T05:05:12.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.106+0000 7f71d1ed9700 1 --2- 172.21.15.113:0/1003247761 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f71b4061110 0x7f71b40635d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:12.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.106+0000 7f71d1ed9700 1 --2- 172.21.15.113:0/1003247761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71cc0f79a0 0x7f71cc1080d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:12.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.106+0000 7f71d1ed9700 1 --2- 172.21.15.113:0/1003247761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71cc0f8c40 0x7f71cc1085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:12.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.106+0000 7f71d1ed9700 1 -- 172.21.15.113:0/1003247761 >> 172.21.15.113:0/1003247761 conn(0x7f71cc0f3100 msgr2=0x7f71cc0fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:12.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.106+0000 7f71d1ed9700 1 -- 172.21.15.113:0/1003247761 shutdown_connections 2024-04-12T05:05:12.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:12.106+0000 7f71d1ed9700 1 -- 172.21.15.113:0/1003247761 wait complete. 2024-04-12T05:05:12.785 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-12T05:05:12.785 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-12T05:05:12.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-12T05:05:12.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-12T05:05:12.787 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-12T05:05:12.787 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-12T05:05:13.116 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:13.129 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:13.384 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:11.537071+0000 mgr.smithi113.ctsazg (mgr.14194) 161 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:13.384 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:12.105347+0000 mon.smithi113 (mon.0) 492 : audit [DBG] from='client.? 172.21.15.113:0/1003247761' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:05:13.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:11.537071+0000 mgr.smithi113.ctsazg (mgr.14194) 161 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:13.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:12.105347+0000 mon.smithi113 (mon.0) 492 : audit [DBG] from='client.? 172.21.15.113:0/1003247761' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:05:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:13.537403+0000 mgr.smithi113.ctsazg (mgr.14194) 162 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:13.537403+0000 mgr.smithi113.ctsazg (mgr.14194) 162 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:15.541 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:15.542 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:15.544 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:15.546 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:17.321 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:15.537732+0000 mgr.smithi113.ctsazg (mgr.14194) 163 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:17.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:15.537732+0000 mgr.smithi113.ctsazg (mgr.14194) 163 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:19.076 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:17.538041+0000 mgr.smithi113.ctsazg (mgr.14194) 164 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:17.538041+0000 mgr.smithi113.ctsazg (mgr.14194) 164 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:19.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.706+0000 7fa53ca5f700 1 -- 172.21.15.113:0/439549277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5380f8c40 msgr2=0x7fa5380f90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:19.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.706+0000 7fa53ca5f700 1 --2- 172.21.15.113:0/439549277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5380f8c40 0x7fa5380f90a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa52c0047f0 tx=0x7fa52c009cb0).stop 2024-04-12T05:05:19.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.706+0000 7fa53ca5f700 1 -- 172.21.15.113:0/439549277 shutdown_connections 2024-04-12T05:05:19.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.706+0000 7fa53ca5f700 1 --2- 172.21.15.113:0/439549277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5380f8c40 0x7fa5380f90a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:19.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.706+0000 7fa53ca5f700 1 --2- 172.21.15.113:0/439549277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5380f79a0 0x7fa5380f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:19.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.706+0000 7fa53ca5f700 1 -- 172.21.15.113:0/439549277 >> 172.21.15.113:0/439549277 conn(0x7fa5380f3100 msgr2=0x7fa5380f5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:19.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.707+0000 7fa53ca5f700 1 -- 172.21.15.113:0/439549277 shutdown_connections 2024-04-12T05:05:19.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.707+0000 7fa53ca5f700 1 -- 172.21.15.113:0/439549277 wait complete. 2024-04-12T05:05:19.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.707+0000 7fa53ca5f700 1 Processor -- start 2024-04-12T05:05:19.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa53ca5f700 1 -- start start 2024-04-12T05:05:19.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa53ca5f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5380f79a0 0x7fa538108080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:19.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa53ca5f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5380f8c40 0x7fa538108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:19.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa53ca5f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa538108b20 con 0x7fa5380f8c40 2024-04-12T05:05:19.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa53ca5f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa538108c60 con 0x7fa5380f79a0 2024-04-12T05:05:19.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa535d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5380f8c40 0x7fa538108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:19.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa535d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5380f8c40 0x7fa538108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52668/0 (socket says 172.21.15.113:52668) 2024-04-12T05:05:19.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa535d9b700 1 -- 172.21.15.113:0/2770580974 learned_addr learned my addr 172.21.15.113:0/2770580974 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa53659c700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5380f79a0 0x7fa538108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa535d9b700 1 -- 172.21.15.113:0/2770580974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5380f79a0 msgr2=0x7fa538108080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa535d9b700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5380f79a0 0x7fa538108080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa535d9b700 1 -- 172.21.15.113:0/2770580974 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c005040 con 0x7fa5380f8c40 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa53659c700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5380f79a0 0x7fa538108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa535d9b700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5380f8c40 0x7fa538108570 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa52c013470 tx=0x7fa52c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa5277fe700 1 -- 172.21.15.113:0/2770580974 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa52c00e030 con 0x7fa5380f8c40 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa53ca5f700 1 -- 172.21.15.113:0/2770580974 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53810d6a0 con 0x7fa5380f8c40 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa5277fe700 1 -- 172.21.15.113:0/2770580974 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fa52c007e90 con 0x7fa5380f8c40 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.708+0000 7fa5277fe700 1 -- 172.21.15.113:0/2770580974 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa52c018070 con 0x7fa5380f8c40 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.709+0000 7fa53ca5f700 1 -- 172.21.15.113:0/2770580974 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53810dbe0 con 0x7fa5380f8c40 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.709+0000 7fa53ca5f700 1 -- 172.21.15.113:0/2770580974 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa53805f5f0 con 0x7fa5380f8c40 2024-04-12T05:05:19.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.710+0000 7fa5277fe700 1 -- 172.21.15.113:0/2770580974 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fa52c01d420 con 0x7fa5380f8c40 2024-04-12T05:05:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.710+0000 7fa5277fe700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa520061180 0x7fa520063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.710+0000 7fa5277fe700 1 -- 172.21.15.113:0/2770580974 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7fa52c07da20 con 0x7fa5380f8c40 2024-04-12T05:05:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.710+0000 7fa53659c700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa520061180 0x7fa520063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.710+0000 7fa5277fe700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248] conn(0x7fa520066d50 0x7fa520069190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.710+0000 7fa5277fe700 1 -- 172.21.15.113:0/2770580974 --> [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa520069810 con 0x7fa520066d50 2024-04-12T05:05:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.710+0000 7fa5277fe700 1 -- 172.21.15.113:0/2770580974 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa52c082220 con 0x7fa5380f8c40 2024-04-12T05:05:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.711+0000 7fa53659c700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa520061180 0x7fa520063640 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa5380f8aa0 tx=0x7fa528009d60).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.711+0000 7fa536d9d700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248] conn(0x7fa520066d50 0x7fa520069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.711+0000 7fa536d9d700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248] conn(0x7fa520066d50 0x7fa520069190 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.712+0000 7fa5277fe700 1 -- 172.21.15.113:0/2770580974 <== osd.4 v2:172.21.15.192:6808/563933248 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa520069810 con 0x7fa520066d50 2024-04-12T05:05:19.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.726+0000 7fa53ca5f700 1 -- 172.21.15.113:0/2770580974 --> [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa5380593e0 con 0x7fa520066d50 2024-04-12T05:05:19.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.726+0000 7fa5277fe700 1 -- 172.21.15.113:0/2770580974 <== osd.4 v2:172.21.15.192:6808/563933248 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa5380593e0 con 0x7fa520066d50 2024-04-12T05:05:19.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 -- 172.21.15.113:0/2770580974 >> [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248] conn(0x7fa520066d50 msgr2=0x7fa520069190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:19.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248] conn(0x7fa520066d50 0x7fa520069190 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:19.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 -- 172.21.15.113:0/2770580974 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa520061180 msgr2=0x7fa520063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:19.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa520061180 0x7fa520063640 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa5380f8aa0 tx=0x7fa528009d60).stop 2024-04-12T05:05:19.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 -- 172.21.15.113:0/2770580974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5380f8c40 msgr2=0x7fa538108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:19.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5380f8c40 0x7fa538108570 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa52c013470 tx=0x7fa52c009cb0).stop 2024-04-12T05:05:19.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 -- 172.21.15.113:0/2770580974 shutdown_connections 2024-04-12T05:05:19.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.192:6808/563933248,v1:172.21.15.192:6809/563933248] conn(0x7fa520066d50 0x7fa520069190 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:19.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa520061180 0x7fa520063640 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:19.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5380f79a0 0x7fa538108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:19.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 --2- 172.21.15.113:0/2770580974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5380f8c40 0x7fa538108570 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:19.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 -- 172.21.15.113:0/2770580974 >> 172.21.15.113:0/2770580974 conn(0x7fa5380f3100 msgr2=0x7fa5380fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:19.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 -- 172.21.15.113:0/2770580974 shutdown_connections 2024-04-12T05:05:19.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.727+0000 7fa53ca5f700 1 -- 172.21.15.113:0/2770580974 wait complete. 2024-04-12T05:05:20.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.912+0000 7f48c2559700 1 -- 172.21.15.113:0/3589361745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48bc0f8c40 msgr2=0x7f48bc0f90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:20.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.912+0000 7f48c2559700 1 --2- 172.21.15.113:0/3589361745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48bc0f8c40 0x7f48bc0f90a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f48b00047f0 tx=0x7f48b0009cb0).stop 2024-04-12T05:05:20.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.912+0000 7f48c2559700 1 -- 172.21.15.113:0/3589361745 shutdown_connections 2024-04-12T05:05:20.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.912+0000 7f48c2559700 1 --2- 172.21.15.113:0/3589361745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48bc0f8c40 0x7f48bc0f90a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.912+0000 7f48c2559700 1 --2- 172.21.15.113:0/3589361745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48bc0f79a0 0x7f48bc0f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.912+0000 7f48c2559700 1 -- 172.21.15.113:0/3589361745 >> 172.21.15.113:0/3589361745 conn(0x7f48bc0f3100 msgr2=0x7f48bc0f5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:20.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.913+0000 7f48c2559700 1 -- 172.21.15.113:0/3589361745 shutdown_connections 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.913+0000 7f48c2559700 1 -- 172.21.15.113:0/3589361745 wait complete. 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.913+0000 7f48c2559700 1 Processor -- start 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.913+0000 7f48c2559700 1 -- start start 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48c2559700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48bc0f79a0 0x7f48bc108080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48c2559700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48bc0f8c40 0x7f48bc108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48c2559700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc108b20 con 0x7f48bc0f8c40 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48c2559700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc108c60 con 0x7f48bc0f79a0 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48bb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48bc0f8c40 0x7f48bc108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48bb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48bc0f8c40 0x7f48bc108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52674/0 (socket says 172.21.15.113:52674) 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48bb7fe700 1 -- 172.21.15.113:0/91942318 learned_addr learned my addr 172.21.15.113:0/91942318 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48bb7fe700 1 -- 172.21.15.113:0/91942318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48bc0f79a0 msgr2=0x7f48bc108080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48bbfff700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48bc0f79a0 0x7f48bc108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48bb7fe700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48bc0f79a0 0x7f48bc108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48bb7fe700 1 -- 172.21.15.113:0/91942318 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b0005040 con 0x7f48bc0f8c40 2024-04-12T05:05:20.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48bb7fe700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48bc0f8c40 0x7f48bc108570 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f48b0013510 tx=0x7f48b0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48b97fa700 1 -- 172.21.15.113:0/91942318 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48b000e030 con 0x7f48bc0f8c40 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48b97fa700 1 -- 172.21.15.113:0/91942318 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f48b0007e90 con 0x7f48bc0f8c40 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48c2559700 1 -- 172.21.15.113:0/91942318 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc10d6a0 con 0x7f48bc0f8c40 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48b97fa700 1 -- 172.21.15.113:0/91942318 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48b0018070 con 0x7f48bc0f8c40 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.914+0000 7f48c2559700 1 -- 172.21.15.113:0/91942318 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc10dbe0 con 0x7f48bc0f8c40 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.915+0000 7f48c2559700 1 -- 172.21.15.113:0/91942318 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f48bc05f5f0 con 0x7f48bc0f8c40 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.916+0000 7f48b97fa700 1 -- 172.21.15.113:0/91942318 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f48b001da20 con 0x7f48bc0f8c40 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.916+0000 7f48b97fa700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f48a4061180 0x7f48a4063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.916+0000 7f48b97fa700 1 -- 172.21.15.113:0/91942318 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f48b007e1b0 con 0x7f48bc0f8c40 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.916+0000 7f48bbfff700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f48a4061180 0x7f48a4063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.916+0000 7f48b97fa700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711] conn(0x7f48a4066d50 0x7f48a4069190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.916+0000 7f48b97fa700 1 -- 172.21.15.113:0/91942318 --> [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48a4069810 con 0x7f48a4066d50 2024-04-12T05:05:20.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.916+0000 7f48b97fa700 1 -- 172.21.15.113:0/91942318 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48b007e5b0 con 0x7f48bc0f8c40 2024-04-12T05:05:20.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.916+0000 7f48c0af6700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711] conn(0x7f48a4066d50 0x7f48a4069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:20.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.916+0000 7f48bbfff700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f48a4061180 0x7f48a4063640 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f48ac0041d0 tx=0x7f48ac009040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:20.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.917+0000 7f48c0af6700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711] conn(0x7f48a4066d50 0x7f48a4069190 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:20.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.917+0000 7f48b97fa700 1 -- 172.21.15.113:0/91942318 <== osd.2 v2:172.21.15.113:6818/2998637711 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f48a4069810 con 0x7f48a4066d50 2024-04-12T05:05:20.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.932+0000 7f48c2559700 1 -- 172.21.15.113:0/91942318 --> [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f48bc059360 con 0x7f48a4066d50 2024-04-12T05:05:20.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.932+0000 7f48b97fa700 1 -- 172.21.15.113:0/91942318 <== osd.2 v2:172.21.15.113:6818/2998637711 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f48bc059360 con 0x7f48a4066d50 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.932+0000 7f48c2559700 1 -- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711] conn(0x7f48a4066d50 msgr2=0x7f48a4069190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.932+0000 7f48c2559700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711] conn(0x7f48a4066d50 0x7f48a4069190 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 -- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f48a4061180 msgr2=0x7f48a4063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f48a4061180 0x7f48a4063640 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f48ac0041d0 tx=0x7f48ac009040).stop 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 -- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48bc0f8c40 msgr2=0x7f48bc108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48bc0f8c40 0x7f48bc108570 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f48b0013510 tx=0x7f48b0009cb0).stop 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 -- 172.21.15.113:0/91942318 shutdown_connections 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6818/2998637711,v1:172.21.15.113:6819/2998637711] conn(0x7f48a4066d50 0x7f48a4069190 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f48a4061180 0x7f48a4063640 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48bc0f79a0 0x7f48bc108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 --2- 172.21.15.113:0/91942318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48bc0f8c40 0x7f48bc108570 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 -- 172.21.15.113:0/91942318 >> 172.21.15.113:0/91942318 conn(0x7f48bc0f3100 msgr2=0x7f48bc0fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 -- 172.21.15.113:0/91942318 shutdown_connections 2024-04-12T05:05:20.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:19.933+0000 7f48c2559700 1 -- 172.21.15.113:0/91942318 wait complete. 2024-04-12T05:05:20.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.520+0000 7ffab7ef4700 1 -- 172.21.15.113:0/2518992272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffab00f7ec0 msgr2=0x7ffab00f82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:20.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.520+0000 7ffab7ef4700 1 --2- 172.21.15.113:0/2518992272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffab00f7ec0 0x7ffab00f82e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ffaa00047f0 tx=0x7ffaa0009cb0).stop 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.520+0000 7ffab7ef4700 1 -- 172.21.15.113:0/2518992272 shutdown_connections 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.520+0000 7ffab7ef4700 1 --2- 172.21.15.113:0/2518992272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab00f8860 0x7ffab00ff1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.520+0000 7ffab7ef4700 1 --2- 172.21.15.113:0/2518992272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffab00f7ec0 0x7ffab00f82e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.520+0000 7ffab7ef4700 1 -- 172.21.15.113:0/2518992272 >> 172.21.15.113:0/2518992272 conn(0x7ffab00f1770 msgr2=0x7ffab00f3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.520+0000 7ffab7ef4700 1 -- 172.21.15.113:0/2518992272 shutdown_connections 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.520+0000 7ffab7ef4700 1 -- 172.21.15.113:0/2518992272 wait complete. 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.520+0000 7ffab7ef4700 1 Processor -- start 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.520+0000 7ffab7ef4700 1 -- start start 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab7ef4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab00f7ec0 0x7ffab010a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab7ef4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffab00f8860 0x7ffab010a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab7ef4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab010ae30 con 0x7ffab00f8860 2024-04-12T05:05:20.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab7ef4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab010af70 con 0x7ffab00f7ec0 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab548f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffab00f8860 0x7ffab010a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab548f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffab00f8860 0x7ffab010a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52646/0 (socket says 172.21.15.113:52646) 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab548f700 1 -- 172.21.15.113:0/4251373890 learned_addr learned my addr 172.21.15.113:0/4251373890 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab5c90700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab00f7ec0 0x7ffab010a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab548f700 1 -- 172.21.15.113:0/4251373890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab00f7ec0 msgr2=0x7ffab010a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab548f700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab00f7ec0 0x7ffab010a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab548f700 1 -- 172.21.15.113:0/4251373890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa0005040 con 0x7ffab00f8860 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab548f700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffab00f8860 0x7ffab010a7f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7ffaac00b4b0 tx=0x7ffaac0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffaa6ffd700 1 -- 172.21.15.113:0/4251373890 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaac002500 con 0x7ffab00f8860 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffaa6ffd700 1 -- 172.21.15.113:0/4251373890 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7ffaac002af0 con 0x7ffab00f8860 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab7ef4700 1 -- 172.21.15.113:0/4251373890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab010fa10 con 0x7ffab00f8860 2024-04-12T05:05:20.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffaa6ffd700 1 -- 172.21.15.113:0/4251373890 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaac012c80 con 0x7ffab00f8860 2024-04-12T05:05:20.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.521+0000 7ffab7ef4700 1 -- 172.21.15.113:0/4251373890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab010ff50 con 0x7ffab00f8860 2024-04-12T05:05:20.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.522+0000 7ffab7ef4700 1 -- 172.21.15.113:0/4251373890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffab005f5f0 con 0x7ffab00f8860 2024-04-12T05:05:20.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.522+0000 7ffaa6ffd700 1 -- 172.21.15.113:0/4251373890 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7ffaac002660 con 0x7ffab00f8860 2024-04-12T05:05:20.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.523+0000 7ffaa6ffd700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ffa9c061180 0x7ffa9c063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:20.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.523+0000 7ffaa6ffd700 1 -- 172.21.15.113:0/4251373890 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7ffaac07c620 con 0x7ffab00f8860 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.523+0000 7ffab5c90700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ffa9c061180 0x7ffa9c063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.523+0000 7ffaa6ffd700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207] conn(0x7ffa9c066d50 0x7ffa9c069190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.523+0000 7ffaa6ffd700 1 -- 172.21.15.113:0/4251373890 --> [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa9c069810 con 0x7ffa9c066d50 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.523+0000 7ffaa6ffd700 1 -- 172.21.15.113:0/4251373890 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffaac07ec90 con 0x7ffab00f8860 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.523+0000 7ffab5c90700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ffa9c061180 0x7ffa9c063640 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ffaa0004f10 tx=0x7ffaa000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.523+0000 7ffab6491700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207] conn(0x7ffa9c066d50 0x7ffa9c069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.524+0000 7ffab6491700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207] conn(0x7ffa9c066d50 0x7ffa9c069190 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.524+0000 7ffaa6ffd700 1 -- 172.21.15.113:0/4251373890 <== osd.3 v2:172.21.15.192:6800/821756207 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ffa9c069810 con 0x7ffa9c066d50 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.539+0000 7ffab7ef4700 1 -- 172.21.15.113:0/4251373890 --> [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffab00593e0 con 0x7ffa9c066d50 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffaa6ffd700 1 -- 172.21.15.113:0/4251373890 <== osd.3 v2:172.21.15.192:6800/821756207 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ffab00593e0 con 0x7ffa9c066d50 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 -- 172.21.15.113:0/4251373890 >> [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207] conn(0x7ffa9c066d50 msgr2=0x7ffa9c069190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207] conn(0x7ffa9c066d50 0x7ffa9c069190 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 -- 172.21.15.113:0/4251373890 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ffa9c061180 msgr2=0x7ffa9c063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:20.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ffa9c061180 0x7ffa9c063640 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ffaa0004f10 tx=0x7ffaa000b040).stop 2024-04-12T05:05:20.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 -- 172.21.15.113:0/4251373890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffab00f8860 msgr2=0x7ffab010a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:20.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffab00f8860 0x7ffab010a7f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7ffaac00b4b0 tx=0x7ffaac0090f0).stop 2024-04-12T05:05:20.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 -- 172.21.15.113:0/4251373890 shutdown_connections 2024-04-12T05:05:20.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.192:6800/821756207,v1:172.21.15.192:6801/821756207] conn(0x7ffa9c066d50 0x7ffa9c069190 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ffa9c061180 0x7ffa9c063640 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab00f7ec0 0x7ffab010a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 --2- 172.21.15.113:0/4251373890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffab00f8860 0x7ffab010a7f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:20.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 -- 172.21.15.113:0/4251373890 >> 172.21.15.113:0/4251373890 conn(0x7ffab00f1770 msgr2=0x7ffab00f3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:20.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 -- 172.21.15.113:0/4251373890 shutdown_connections 2024-04-12T05:05:20.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:16.540+0000 7ffab7ef4700 1 -- 172.21.15.113:0/4251373890 wait complete. 2024-04-12T05:05:21.399 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:19.538462+0000 mgr.smithi113.ctsazg (mgr.14194) 165 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:19.538462+0000 mgr.smithi113.ctsazg (mgr.14194) 165 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:22.001 INFO:teuthology.orchestra.run.smithi113.stdout:81604378649 2024-04-12T05:05:22.001 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-12T05:05:22.708 INFO:teuthology.orchestra.run.smithi113.stdout:137438953486 2024-04-12T05:05:22.709 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-12T05:05:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:21.538947+0000 mgr.smithi113.ctsazg (mgr.14194) 166 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:21.538947+0000 mgr.smithi113.ctsazg (mgr.14194) 166 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:24.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.338+0000 7f241a817700 1 -- 172.21.15.113:0/3663428798 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140fab60 msgr2=0x7f24140fcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:24.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.338+0000 7f241a817700 1 --2- 172.21.15.113:0/3663428798 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140fab60 0x7f24140fcf50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f24080047f0 tx=0x7f2408009cb0).stop 2024-04-12T05:05:24.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.338+0000 7f241a817700 1 -- 172.21.15.113:0/3663428798 shutdown_connections 2024-04-12T05:05:24.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.338+0000 7f241a817700 1 --2- 172.21.15.113:0/3663428798 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140fab60 0x7f24140fcf50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.338+0000 7f241a817700 1 --2- 172.21.15.113:0/3663428798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2414061780 0x7f24140fa5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.338+0000 7f241a817700 1 -- 172.21.15.113:0/3663428798 >> 172.21.15.113:0/3663428798 conn(0x7f24140f2020 msgr2=0x7f24140f4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:24.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.338+0000 7f241a817700 1 -- 172.21.15.113:0/3663428798 shutdown_connections 2024-04-12T05:05:24.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.338+0000 7f241a817700 1 -- 172.21.15.113:0/3663428798 wait complete. 2024-04-12T05:05:24.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.339+0000 7f241a817700 1 Processor -- start 2024-04-12T05:05:24.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.339+0000 7f241a817700 1 -- start start 2024-04-12T05:05:24.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.339+0000 7f241a817700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2414061780 0x7f241410a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.339+0000 7f241a817700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140fab60 0x7f241410a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.339+0000 7f241a817700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241410ad30 con 0x7f24140fab60 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.339+0000 7f241a817700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241410ae70 con 0x7f2414061780 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.339+0000 7f24137fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140fab60 0x7f241410a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.339+0000 7f24137fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140fab60 0x7f241410a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43138/0 (socket says 172.21.15.113:43138) 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.339+0000 7f24137fe700 1 -- 172.21.15.113:0/1520208830 learned_addr learned my addr 172.21.15.113:0/1520208830 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.340+0000 7f2413fff700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2414061780 0x7f241410a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.340+0000 7f24137fe700 1 -- 172.21.15.113:0/1520208830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2414061780 msgr2=0x7f241410a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.340+0000 7f24137fe700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2414061780 0x7f241410a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.340+0000 7f24137fe700 1 -- 172.21.15.113:0/1520208830 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2408005040 con 0x7f24140fab60 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.340+0000 7f2413fff700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2414061780 0x7f241410a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:05:24.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.340+0000 7f24137fe700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140fab60 0x7f241410a780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f2408007c40 tx=0x7f2408009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.340+0000 7f24117fa700 1 -- 172.21.15.113:0/1520208830 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f240800e030 con 0x7f24140fab60 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.340+0000 7f241a817700 1 -- 172.21.15.113:0/1520208830 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241410f8b0 con 0x7f24140fab60 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.340+0000 7f24117fa700 1 -- 172.21.15.113:0/1520208830 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f2408007e90 con 0x7f24140fab60 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.340+0000 7f24117fa700 1 -- 172.21.15.113:0/1520208830 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2408018070 con 0x7f24140fab60 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.340+0000 7f241a817700 1 -- 172.21.15.113:0/1520208830 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241410fdf0 con 0x7f24140fab60 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.341+0000 7f241a817700 1 -- 172.21.15.113:0/1520208830 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2414061c00 con 0x7f24140fab60 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.341+0000 7f24117fa700 1 -- 172.21.15.113:0/1520208830 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f240801d420 con 0x7f24140fab60 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.342+0000 7f24117fa700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f23fc061180 0x7f23fc063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.342+0000 7f24117fa700 1 -- 172.21.15.113:0/1520208830 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f240807e620 con 0x7f24140fab60 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.342+0000 7f2413fff700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f23fc061180 0x7f23fc063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.342+0000 7f24117fa700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970] conn(0x7f23fc066d50 0x7f23fc069190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:24.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.342+0000 7f24117fa700 1 -- 172.21.15.113:0/1520208830 --> [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23fc069810 con 0x7f23fc066d50 2024-04-12T05:05:24.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.342+0000 7f24117fa700 1 -- 172.21.15.113:0/1520208830 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f240807ea20 con 0x7f24140fab60 2024-04-12T05:05:24.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.342+0000 7f2418db4700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970] conn(0x7f23fc066d50 0x7f23fc069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:24.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.342+0000 7f2413fff700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f23fc061180 0x7f23fc063640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f24040041d0 tx=0x7f2404009040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:24.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.342+0000 7f2418db4700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970] conn(0x7f23fc066d50 0x7f23fc069190 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:24.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.343+0000 7f24117fa700 1 -- 172.21.15.113:0/1520208830 <== osd.0 v2:172.21.15.113:6802/1323860970 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f23fc069810 con 0x7f23fc066d50 2024-04-12T05:05:24.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.357+0000 7f241a817700 1 -- 172.21.15.113:0/1520208830 --> [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f24140fbbc0 con 0x7f23fc066d50 2024-04-12T05:05:24.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.357+0000 7f24117fa700 1 -- 172.21.15.113:0/1520208830 <== osd.0 v2:172.21.15.113:6802/1323860970 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f24140fbbc0 con 0x7f23fc066d50 2024-04-12T05:05:24.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 -- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970] conn(0x7f23fc066d50 msgr2=0x7f23fc069190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:24.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970] conn(0x7f23fc066d50 0x7f23fc069190 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 -- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f23fc061180 msgr2=0x7f23fc063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:24.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f23fc061180 0x7f23fc063640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f24040041d0 tx=0x7f2404009040).stop 2024-04-12T05:05:24.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 -- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140fab60 msgr2=0x7f241410a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:24.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140fab60 0x7f241410a780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f2408007c40 tx=0x7f2408009cb0).stop 2024-04-12T05:05:24.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 -- 172.21.15.113:0/1520208830 shutdown_connections 2024-04-12T05:05:24.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6802/1323860970,v1:172.21.15.113:6803/1323860970] conn(0x7f23fc066d50 0x7f23fc069190 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f23fc061180 0x7f23fc063640 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2414061780 0x7f241410a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 --2- 172.21.15.113:0/1520208830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24140fab60 0x7f241410a780 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 -- 172.21.15.113:0/1520208830 >> 172.21.15.113:0/1520208830 conn(0x7f24140f2020 msgr2=0x7f24140fb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:24.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 -- 172.21.15.113:0/1520208830 shutdown_connections 2024-04-12T05:05:24.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.358+0000 7f241a817700 1 -- 172.21.15.113:0/1520208830 wait complete. 2024-04-12T05:05:24.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.432+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/3601112103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0f65f0 msgr2=0x7fa0cc0f8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:24.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.432+0000 7fa0d0b8a700 1 --2- 172.21.15.113:0/3601112103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0f65f0 0x7fa0cc0f8d70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa0c00047f0 tx=0x7fa0c0009cb0).stop 2024-04-12T05:05:24.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.432+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/3601112103 shutdown_connections 2024-04-12T05:05:24.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.432+0000 7fa0d0b8a700 1 --2- 172.21.15.113:0/3601112103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0f65f0 0x7fa0cc0f8d70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.432+0000 7fa0d0b8a700 1 --2- 172.21.15.113:0/3601112103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0f5c50 0x7fa0cc0f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.432+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/3601112103 >> 172.21.15.113:0/3601112103 conn(0x7fa0cc0f1770 msgr2=0x7fa0cc0f3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.432+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/3601112103 shutdown_connections 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.432+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/3601112103 wait complete. 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0d0b8a700 1 Processor -- start 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0d0b8a700 1 -- start start 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0d0b8a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0f5c50 0x7fa0cc1080e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0d0b8a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0f65f0 0x7fa0cc1085d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0d0b8a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc108b80 con 0x7fa0cc0f5c50 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0d0b8a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc108cc0 con 0x7fa0cc0f65f0 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0ca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0f5c50 0x7fa0cc1080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0ca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0f5c50 0x7fa0cc1080e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43166/0 (socket says 172.21.15.113:43166) 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0ca59c700 1 -- 172.21.15.113:0/2822044233 learned_addr learned my addr 172.21.15.113:0/2822044233 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:24.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0c9d9b700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0f65f0 0x7fa0cc1085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0ca59c700 1 -- 172.21.15.113:0/2822044233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0f65f0 msgr2=0x7fa0cc1085d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0ca59c700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0f65f0 0x7fa0cc1085d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.433+0000 7fa0ca59c700 1 -- 172.21.15.113:0/2822044233 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c0005040 con 0x7fa0cc0f5c50 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.434+0000 7fa0ca59c700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0f5c50 0x7fa0cc1080e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa0bc00c790 tx=0x7fa0bc007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.434+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2822044233 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0bc0106f0 con 0x7fa0cc0f5c50 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.434+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/2822044233 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0cc10d760 con 0x7fa0cc0f5c50 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.434+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/2822044233 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc10dca0 con 0x7fa0cc0f5c50 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.434+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2822044233 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fa0bc010ce0 con 0x7fa0cc0f5c50 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.434+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2822044233 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0bc01bc90 con 0x7fa0cc0f5c50 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.435+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2822044233 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fa0bc01b490 con 0x7fa0cc0f5c50 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.435+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/2822044233 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa0cc05f5f0 con 0x7fa0cc0f5c50 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.436+0000 7fa0bb7fe700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa0b4061180 0x7fa0b4063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.436+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2822044233 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7fa0bc07c100 con 0x7fa0cc0f5c50 2024-04-12T05:05:24.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.436+0000 7fa0c9d9b700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa0b4061180 0x7fa0b4063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:24.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.436+0000 7fa0bb7fe700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368] conn(0x7fa0b4066d50 0x7fa0b4069190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:24.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.436+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2822044233 --> [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa0b4069810 con 0x7fa0b4066d50 2024-04-12T05:05:24.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.436+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2822044233 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0bc07e770 con 0x7fa0cc0f5c50 2024-04-12T05:05:24.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.436+0000 7fa0cad9d700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368] conn(0x7fa0b4066d50 0x7fa0b4069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:24.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.436+0000 7fa0c9d9b700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa0b4061180 0x7fa0b4063640 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fa0c0013470 tx=0x7fa0c000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:24.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.436+0000 7fa0cad9d700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368] conn(0x7fa0b4066d50 0x7fa0b4069190 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:24.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.437+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2822044233 <== osd.1 v2:172.21.15.113:6810/563293368 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa0b4069810 con 0x7fa0b4066d50 2024-04-12T05:05:24.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/2822044233 --> [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa0cc0593e0 con 0x7fa0b4066d50 2024-04-12T05:05:24.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2822044233 <== osd.1 v2:172.21.15.113:6810/563293368 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa0cc0593e0 con 0x7fa0b4066d50 2024-04-12T05:05:24.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368] conn(0x7fa0b4066d50 msgr2=0x7fa0b4069190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:24.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368] conn(0x7fa0b4066d50 0x7fa0b4069190 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa0b4061180 msgr2=0x7fa0b4063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:24.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa0b4061180 0x7fa0b4063640 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fa0c0013470 tx=0x7fa0c000b040).stop 2024-04-12T05:05:24.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0f5c50 msgr2=0x7fa0cc1080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:24.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0f5c50 0x7fa0cc1080e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa0bc00c790 tx=0x7fa0bc007f70).stop 2024-04-12T05:05:24.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/2822044233 shutdown_connections 2024-04-12T05:05:24.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa0b4061180 0x7fa0b4063640 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0f5c50 0x7fa0cc1080e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.113:6810/563293368,v1:172.21.15.113:6811/563293368] conn(0x7fa0b4066d50 0x7fa0b4069190 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 --2- 172.21.15.113:0/2822044233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0f65f0 0x7fa0cc1085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:24.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/2822044233 >> 172.21.15.113:0/2822044233 conn(0x7fa0cc0f1770 msgr2=0x7fa0cc0fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:24.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/2822044233 shutdown_connections 2024-04-12T05:05:24.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:24.453+0000 7fa0d0b8a700 1 -- 172.21.15.113:0/2822044233 wait complete. 2024-04-12T05:05:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:23.539273+0000 mgr.smithi113.ctsazg (mgr.14194) 167 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:23.539273+0000 mgr.smithi113.ctsazg (mgr.14194) 167 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:25.848 INFO:teuthology.orchestra.run.smithi113.stdout:111669149715 2024-04-12T05:05:25.848 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-12T05:05:25.960 INFO:teuthology.orchestra.run.smithi113.stdout:60129542176 2024-04-12T05:05:25.961 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-12T05:05:26.491 INFO:teuthology.orchestra.run.smithi113.stdout:38654705702 2024-04-12T05:05:26.491 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-12T05:05:26.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.951+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1713726741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6c80f79a0 msgr2=0x7fa6c80f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:26.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.951+0000 7fa6ce6c9700 1 --2- 172.21.15.113:0/1713726741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6c80f79a0 0x7fa6c80f7dc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fa6b80047f0 tx=0x7fa6b8009cb0).stop 2024-04-12T05:05:26.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.951+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1713726741 shutdown_connections 2024-04-12T05:05:26.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.951+0000 7fa6ce6c9700 1 --2- 172.21.15.113:0/1713726741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6c80f8c40 0x7fa6c80f90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:05:26.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.951+0000 7fa6ce6c9700 1 --2- 172.21.15.113:0/1713726741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6c80f79a0 0x7fa6c80f7dc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:26.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.951+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1713726741 >> 172.21.15.113:0/1713726741 conn(0x7fa6c80f3100 msgr2=0x7fa6c80f5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:26.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.951+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1713726741 shutdown_connections 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.951+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1713726741 wait complete. 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.952+0000 7fa6ce6c9700 1 Processor -- start 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.952+0000 7fa6ce6c9700 1 -- start start 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.952+0000 7fa6ce6c9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6c80f79a0 0x7fa6c8108110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.952+0000 7fa6ce6c9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6c80f8c40 0x7fa6c8108600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.952+0000 7fa6ce6c9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c8108c40 con 0x7fa6c80f79a0 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.952+0000 7fa6ce6c9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c8108d80 con 0x7fa6c80f8c40 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6c80f79a0 0x7fa6c8108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6c80f79a0 0x7fa6c8108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43110/0 (socket says 172.21.15.113:43110) 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c7fff700 1 -- 172.21.15.113:0/1667174485 learned_addr learned my addr 172.21.15.113:0/1667174485 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c77fe700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6c80f8c40 0x7fa6c8108600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c7fff700 1 -- 172.21.15.113:0/1667174485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6c80f8c40 msgr2=0x7fa6c8108600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:26.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c7fff700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6c80f8c40 0x7fa6c8108600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c7fff700 1 -- 172.21.15.113:0/1667174485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b8005040 con 0x7fa6c80f79a0 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c77fe700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6c80f8c40 0x7fa6c8108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c7fff700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6c80f79a0 0x7fa6c8108110 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fa6b8013470 tx=0x7fa6b8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c57fa700 1 -- 172.21.15.113:0/1667174485 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b800e030 con 0x7fa6c80f79a0 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1667174485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c810d7c0 con 0x7fa6c80f79a0 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c57fa700 1 -- 172.21.15.113:0/1667174485 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fa6b8007e90 con 0x7fa6c80f79a0 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6c57fa700 1 -- 172.21.15.113:0/1667174485 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b8018070 con 0x7fa6c80f79a0 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.953+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1667174485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c810dca0 con 0x7fa6c80f79a0 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.954+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1667174485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa6c805f5f0 con 0x7fa6c80f79a0 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.954+0000 7fa6c57fa700 1 -- 172.21.15.113:0/1667174485 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fa6b801d940 con 0x7fa6c80f79a0 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.955+0000 7fa6c57fa700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa6b00611b0 0x7fa6b0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:26.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.955+0000 7fa6c57fa700 1 -- 172.21.15.113:0/1667174485 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7fa6b807dc70 con 0x7fa6c80f79a0 2024-04-12T05:05:26.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.955+0000 7fa6c77fe700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa6b00611b0 0x7fa6b0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:26.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.955+0000 7fa6c57fa700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323] conn(0x7fa6b0066d80 0x7fa6b00691c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:26.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.955+0000 7fa6c57fa700 1 -- 172.21.15.113:0/1667174485 --> [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6b0069840 con 0x7fa6b0066d80 2024-04-12T05:05:26.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.955+0000 7fa6c57fa700 1 -- 172.21.15.113:0/1667174485 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6b80831c0 con 0x7fa6c80f79a0 2024-04-12T05:05:26.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.955+0000 7fa6c77fe700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa6b00611b0 0x7fa6b0063670 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fa6bc00a900 tx=0x7fa6bc008040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:26.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.955+0000 7fa6ccc66700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323] conn(0x7fa6b0066d80 0x7fa6b00691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:26.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.956+0000 7fa6ccc66700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323] conn(0x7fa6b0066d80 0x7fa6b00691c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:26.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.956+0000 7fa6c57fa700 1 -- 172.21.15.113:0/1667174485 <== osd.5 v2:172.21.15.192:6816/2433186323 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa6b0069840 con 0x7fa6b0066d80 2024-04-12T05:05:26.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.970+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1667174485 --> [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6c8059360 con 0x7fa6b0066d80 2024-04-12T05:05:26.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6c57fa700 1 -- 172.21.15.113:0/1667174485 <== osd.5 v2:172.21.15.192:6816/2433186323 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa6c8059360 con 0x7fa6b0066d80 2024-04-12T05:05:26.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1667174485 >> [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323] conn(0x7fa6b0066d80 msgr2=0x7fa6b00691c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323] conn(0x7fa6b0066d80 0x7fa6b00691c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1667174485 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa6b00611b0 msgr2=0x7fa6b0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa6b00611b0 0x7fa6b0063670 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fa6bc00a900 tx=0x7fa6bc008040).stop 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1667174485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6c80f79a0 msgr2=0x7fa6c8108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6c80f79a0 0x7fa6c8108110 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fa6b8013470 tx=0x7fa6b8009cb0).stop 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1667174485 shutdown_connections 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.192:6816/2433186323,v1:172.21.15.192:6817/2433186323] conn(0x7fa6b0066d80 0x7fa6b00691c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa6b00611b0 0x7fa6b0063670 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6c80f79a0 0x7fa6c8108110 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 --2- 172.21.15.113:0/1667174485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6c80f8c40 0x7fa6c8108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1667174485 >> 172.21.15.113:0/1667174485 conn(0x7fa6c80f3100 msgr2=0x7fa6c80fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1667174485 shutdown_connections 2024-04-12T05:05:26.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:23.971+0000 7fa6ce6c9700 1 -- 172.21.15.113:0/1667174485 wait complete. 2024-04-12T05:05:26.803 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:25.539775+0000 mgr.smithi113.ctsazg (mgr.14194) 168 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:25.539775+0000 mgr.smithi113.ctsazg (mgr.14194) 168 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:27.529 INFO:teuthology.orchestra.run.smithi113.stdout:158913789961 2024-04-12T05:05:27.529 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-12T05:05:27.712 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:28.086 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:28.088 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:27.540133+0000 mgr.smithi113.ctsazg (mgr.14194) 169 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:27.540133+0000 mgr.smithi113.ctsazg (mgr.14194) 169 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:31.147 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:29.540441+0000 mgr.smithi113.ctsazg (mgr.14194) 170 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:31.148 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:30.805534+0000 mon.smithi113 (mon.0) 493 : audit [DBG] from='client.? 172.21.15.113:0/434829524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-12T05:05:31.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.633+0000 7f9c1be77700 1 -- 172.21.15.113:0/1330165104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c140fab00 msgr2=0x7f9c140fcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:31.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.633+0000 7f9c1be77700 1 --2- 172.21.15.113:0/1330165104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c140fab00 0x7f9c140fcf50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f9c080047f0 tx=0x7f9c08009cb0).stop 2024-04-12T05:05:31.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.633+0000 7f9c1be77700 1 -- 172.21.15.113:0/1330165104 shutdown_connections 2024-04-12T05:05:31.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.633+0000 7f9c1be77700 1 --2- 172.21.15.113:0/1330165104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c140fab00 0x7f9c140fcf50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:31.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.633+0000 7f9c1be77700 1 --2- 172.21.15.113:0/1330165104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c140617e0 0x7f9c140fa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:31.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.633+0000 7f9c1be77700 1 -- 172.21.15.113:0/1330165104 >> 172.21.15.113:0/1330165104 conn(0x7f9c140f2020 msgr2=0x7f9c140f4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.634+0000 7f9c1be77700 1 -- 172.21.15.113:0/1330165104 shutdown_connections 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.634+0000 7f9c1be77700 1 -- 172.21.15.113:0/1330165104 wait complete. 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.634+0000 7f9c1be77700 1 Processor -- start 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.634+0000 7f9c1be77700 1 -- start start 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.634+0000 7f9c1be77700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c140617e0 0x7f9c14108070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.634+0000 7f9c1be77700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c140fab00 0x7f9c14108560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.634+0000 7f9c1be77700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14108b10 con 0x7f9c140fab00 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.634+0000 7f9c1be77700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14108c50 con 0x7f9c140617e0 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.634+0000 7f9c19412700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c140fab00 0x7f9c14108560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.634+0000 7f9c19412700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c140fab00 0x7f9c14108560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52834/0 (socket says 172.21.15.113:52834) 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.634+0000 7f9c19412700 1 -- 172.21.15.113:0/434829524 learned_addr learned my addr 172.21.15.113:0/434829524 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.635+0000 7f9c19c13700 1 --2- 172.21.15.113:0/434829524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c140617e0 0x7f9c14108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:31.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.635+0000 7f9c19412700 1 -- 172.21.15.113:0/434829524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c140617e0 msgr2=0x7f9c14108070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.635+0000 7f9c19412700 1 --2- 172.21.15.113:0/434829524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c140617e0 0x7f9c14108070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.635+0000 7f9c19412700 1 -- 172.21.15.113:0/434829524 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c08005040 con 0x7f9c140fab00 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.635+0000 7f9c19412700 1 --2- 172.21.15.113:0/434829524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c140fab00 0x7f9c14108560 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f9c08013470 tx=0x7f9c08009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.635+0000 7f9c06ffd700 1 -- 172.21.15.113:0/434829524 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c0800e030 con 0x7f9c140fab00 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.635+0000 7f9c06ffd700 1 -- 172.21.15.113:0/434829524 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f9c08007e90 con 0x7f9c140fab00 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.635+0000 7f9c1be77700 1 -- 172.21.15.113:0/434829524 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1410d690 con 0x7f9c140fab00 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.635+0000 7f9c06ffd700 1 -- 172.21.15.113:0/434829524 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c08018070 con 0x7f9c140fab00 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.635+0000 7f9c1be77700 1 -- 172.21.15.113:0/434829524 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1410dbd0 con 0x7f9c140fab00 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.636+0000 7f9c06ffd700 1 -- 172.21.15.113:0/434829524 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f9c0801d940 con 0x7f9c140fab00 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.636+0000 7f9c1be77700 1 -- 172.21.15.113:0/434829524 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c141000f0 con 0x7f9c140fab00 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.636+0000 7f9c06ffd700 1 --2- 172.21.15.113:0/434829524 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9c00061180 0x7f9c00063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.636+0000 7f9c06ffd700 1 -- 172.21.15.113:0/434829524 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f9c0807db20 con 0x7f9c140fab00 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.637+0000 7f9c19c13700 1 --2- 172.21.15.113:0/434829524 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9c00061180 0x7f9c00063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:31.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.637+0000 7f9c19c13700 1 --2- 172.21.15.113:0/434829524 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9c00061180 0x7f9c00063640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f9c10004b60 tx=0x7f9c10009040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:31.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.641+0000 7f9c06ffd700 1 -- 172.21.15.113:0/434829524 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9c08051e70 con 0x7f9c140fab00 2024-04-12T05:05:31.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.804+0000 7f9c1be77700 1 -- 172.21.15.113:0/434829524 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9c140598f0 con 0x7f9c140fab00 2024-04-12T05:05:31.284 INFO:teuthology.orchestra.run.smithi113.stdout:60129542177 2024-04-12T05:05:31.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.804+0000 7f9c06ffd700 1 -- 172.21.15.113:0/434829524 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9c08051a60 con 0x7f9c140fab00 2024-04-12T05:05:31.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.806+0000 7f9c1be77700 1 -- 172.21.15.113:0/434829524 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9c00061180 msgr2=0x7f9c00063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:31.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.806+0000 7f9c1be77700 1 --2- 172.21.15.113:0/434829524 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9c00061180 0x7f9c00063640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f9c10004b60 tx=0x7f9c10009040).stop 2024-04-12T05:05:31.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.806+0000 7f9c1be77700 1 -- 172.21.15.113:0/434829524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c140fab00 msgr2=0x7f9c14108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:31.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.806+0000 7f9c1be77700 1 --2- 172.21.15.113:0/434829524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c140fab00 0x7f9c14108560 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f9c08013470 tx=0x7f9c08009cb0).stop 2024-04-12T05:05:31.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.806+0000 7f9c1be77700 1 -- 172.21.15.113:0/434829524 shutdown_connections 2024-04-12T05:05:31.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.806+0000 7f9c1be77700 1 --2- 172.21.15.113:0/434829524 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9c00061180 0x7f9c00063640 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:31.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.806+0000 7f9c1be77700 1 --2- 172.21.15.113:0/434829524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c140617e0 0x7f9c14108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:31.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.806+0000 7f9c1be77700 1 --2- 172.21.15.113:0/434829524 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c140fab00 0x7f9c14108560 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:31.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.806+0000 7f9c1be77700 1 -- 172.21.15.113:0/434829524 >> 172.21.15.113:0/434829524 conn(0x7f9c140f2020 msgr2=0x7f9c140f4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:31.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.806+0000 7f9c1be77700 1 -- 172.21.15.113:0/434829524 shutdown_connections 2024-04-12T05:05:31.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:30.806+0000 7f9c1be77700 1 -- 172.21.15.113:0/434829524 wait complete. 2024-04-12T05:05:31.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:29.540441+0000 mgr.smithi113.ctsazg (mgr.14194) 170 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:31.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:30.805534+0000 mon.smithi113 (mon.0) 493 : audit [DBG] from='client.? 172.21.15.113:0/434829524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-12T05:05:31.997 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542176 got 60129542177 for osd.1 2024-04-12T05:05:31.997 DEBUG:teuthology.parallel:result is None 2024-04-12T05:05:32.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.751+0000 7efd79516700 1 -- 172.21.15.113:0/1226317399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd740f79b0 msgr2=0x7efd740f7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:32.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.751+0000 7efd79516700 1 --2- 172.21.15.113:0/1226317399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd740f79b0 0x7efd740f7dd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7efd680047f0 tx=0x7efd68009cb0).stop 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.751+0000 7efd79516700 1 -- 172.21.15.113:0/1226317399 shutdown_connections 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.751+0000 7efd79516700 1 --2- 172.21.15.113:0/1226317399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd740f8c50 0x7efd740f90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.751+0000 7efd79516700 1 --2- 172.21.15.113:0/1226317399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd740f79b0 0x7efd740f7dd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.751+0000 7efd79516700 1 -- 172.21.15.113:0/1226317399 >> 172.21.15.113:0/1226317399 conn(0x7efd740f30f0 msgr2=0x7efd740f5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.751+0000 7efd79516700 1 -- 172.21.15.113:0/1226317399 shutdown_connections 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.751+0000 7efd79516700 1 -- 172.21.15.113:0/1226317399 wait complete. 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.752+0000 7efd79516700 1 Processor -- start 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.752+0000 7efd79516700 1 -- start start 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.752+0000 7efd79516700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd740f79b0 0x7efd74108080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.752+0000 7efd79516700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd740f8c50 0x7efd74108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.752+0000 7efd79516700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74108b20 con 0x7efd740f8c50 2024-04-12T05:05:32.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.752+0000 7efd79516700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74108c60 con 0x7efd740f79b0 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.752+0000 7efd727fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd740f8c50 0x7efd74108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.752+0000 7efd727fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd740f8c50 0x7efd74108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52864/0 (socket says 172.21.15.113:52864) 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.752+0000 7efd727fc700 1 -- 172.21.15.113:0/2137320120 learned_addr learned my addr 172.21.15.113:0/2137320120 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.753+0000 7efd72ffd700 1 --2- 172.21.15.113:0/2137320120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd740f79b0 0x7efd74108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.753+0000 7efd727fc700 1 -- 172.21.15.113:0/2137320120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd740f79b0 msgr2=0x7efd74108080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.753+0000 7efd727fc700 1 --2- 172.21.15.113:0/2137320120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd740f79b0 0x7efd74108080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.753+0000 7efd727fc700 1 -- 172.21.15.113:0/2137320120 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd68005040 con 0x7efd740f8c50 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.753+0000 7efd727fc700 1 --2- 172.21.15.113:0/2137320120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd740f8c50 0x7efd74108570 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7efd6400c740 tx=0x7efd64007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.753+0000 7efd5bfff700 1 -- 172.21.15.113:0/2137320120 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd6400e680 con 0x7efd740f8c50 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.753+0000 7efd5bfff700 1 -- 172.21.15.113:0/2137320120 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7efd6400ec70 con 0x7efd740f8c50 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.753+0000 7efd79516700 1 -- 172.21.15.113:0/2137320120 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7410d700 con 0x7efd740f8c50 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.753+0000 7efd5bfff700 1 -- 172.21.15.113:0/2137320120 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd64013d00 con 0x7efd740f8c50 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.753+0000 7efd79516700 1 -- 172.21.15.113:0/2137320120 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7410dc40 con 0x7efd740f8c50 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.754+0000 7efd79516700 1 -- 172.21.15.113:0/2137320120 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd74100100 con 0x7efd740f8c50 2024-04-12T05:05:32.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.754+0000 7efd5bfff700 1 -- 172.21.15.113:0/2137320120 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7efd6400e7e0 con 0x7efd740f8c50 2024-04-12T05:05:32.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.754+0000 7efd5bfff700 1 --2- 172.21.15.113:0/2137320120 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7efd5c061180 0x7efd5c063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:32.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.755+0000 7efd5bfff700 1 -- 172.21.15.113:0/2137320120 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7efd6407c7f0 con 0x7efd740f8c50 2024-04-12T05:05:32.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.755+0000 7efd72ffd700 1 --2- 172.21.15.113:0/2137320120 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7efd5c061180 0x7efd5c063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:32.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.755+0000 7efd72ffd700 1 --2- 172.21.15.113:0/2137320120 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7efd5c061180 0x7efd5c063640 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7efd68004f10 tx=0x7efd6800b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:32.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.759+0000 7efd5bfff700 1 -- 172.21.15.113:0/2137320120 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7efd64050ac0 con 0x7efd740f8c50 2024-04-12T05:05:32.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.924+0000 7efd79516700 1 -- 172.21.15.113:0/2137320120 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7efd74101f70 con 0x7efd740f8c50 2024-04-12T05:05:32.080 INFO:teuthology.orchestra.run.smithi113.stdout:81604378651 2024-04-12T05:05:32.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.924+0000 7efd5bfff700 1 -- 172.21.15.113:0/2137320120 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd640171b0 con 0x7efd740f8c50 2024-04-12T05:05:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.926+0000 7efd79516700 1 -- 172.21.15.113:0/2137320120 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7efd5c061180 msgr2=0x7efd5c063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.926+0000 7efd79516700 1 --2- 172.21.15.113:0/2137320120 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7efd5c061180 0x7efd5c063640 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7efd68004f10 tx=0x7efd6800b040).stop 2024-04-12T05:05:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.926+0000 7efd79516700 1 -- 172.21.15.113:0/2137320120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd740f8c50 msgr2=0x7efd74108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.926+0000 7efd79516700 1 --2- 172.21.15.113:0/2137320120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd740f8c50 0x7efd74108570 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7efd6400c740 tx=0x7efd64007ef0).stop 2024-04-12T05:05:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.926+0000 7efd79516700 1 -- 172.21.15.113:0/2137320120 shutdown_connections 2024-04-12T05:05:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.926+0000 7efd79516700 1 --2- 172.21.15.113:0/2137320120 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7efd5c061180 0x7efd5c063640 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.926+0000 7efd79516700 1 --2- 172.21.15.113:0/2137320120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd740f79b0 0x7efd74108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.926+0000 7efd79516700 1 --2- 172.21.15.113:0/2137320120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd740f8c50 0x7efd74108570 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.926+0000 7efd79516700 1 -- 172.21.15.113:0/2137320120 >> 172.21.15.113:0/2137320120 conn(0x7efd740f30f0 msgr2=0x7efd740fc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.926+0000 7efd79516700 1 -- 172.21.15.113:0/2137320120 shutdown_connections 2024-04-12T05:05:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:31.926+0000 7efd79516700 1 -- 172.21.15.113:0/2137320120 wait complete. 2024-04-12T05:05:32.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:31.925054+0000 mon.smithi113 (mon.0) 494 : audit [DBG] from='client.? 172.21.15.113:0/2137320120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-12T05:05:32.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:31.925054+0000 mon.smithi113 (mon.0) 494 : audit [DBG] from='client.? 172.21.15.113:0/2137320120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-12T05:05:33.276 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378651 for osd.2 2024-04-12T05:05:33.276 DEBUG:teuthology.parallel:result is None 2024-04-12T05:05:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:31.540760+0000 mgr.smithi113.ctsazg (mgr.14194) 171 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:33.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:31.540760+0000 mgr.smithi113.ctsazg (mgr.14194) 171 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:33.525 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:33.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.889+0000 7f945b0f6700 1 -- 172.21.15.113:0/4291009763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94540f7b40 msgr2=0x7f94540f7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:33.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.889+0000 7f945b0f6700 1 --2- 172.21.15.113:0/4291009763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94540f7b40 0x7f94540f7fa0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f94400047f0 tx=0x7f9440009cb0).stop 2024-04-12T05:05:33.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.889+0000 7f945b0f6700 1 -- 172.21.15.113:0/4291009763 shutdown_connections 2024-04-12T05:05:33.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.889+0000 7f945b0f6700 1 --2- 172.21.15.113:0/4291009763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94540f7b40 0x7f94540f7fa0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:33.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.889+0000 7f945b0f6700 1 --2- 172.21.15.113:0/4291009763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94540f68a0 0x7f94540f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:33.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.889+0000 7f945b0f6700 1 -- 172.21.15.113:0/4291009763 >> 172.21.15.113:0/4291009763 conn(0x7f94540f2020 msgr2=0x7f94540f4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:33.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.889+0000 7f945b0f6700 1 -- 172.21.15.113:0/4291009763 shutdown_connections 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.889+0000 7f945b0f6700 1 -- 172.21.15.113:0/4291009763 wait complete. 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.890+0000 7f945b0f6700 1 Processor -- start 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.890+0000 7f945b0f6700 1 -- start start 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.890+0000 7f945b0f6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94540f68a0 0x7f94540fa820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.890+0000 7f945b0f6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94540f7b40 0x7f94540f8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.890+0000 7f945b0f6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94540fadd0 con 0x7f94540f7b40 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.890+0000 7f945b0f6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94540f9370 con 0x7f94540f68a0 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.890+0000 7f9453fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94540f7b40 0x7f94540f8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.890+0000 7f9453fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94540f7b40 0x7f94540f8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52874/0 (socket says 172.21.15.113:52874) 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.890+0000 7f9453fff700 1 -- 172.21.15.113:0/433769427 learned_addr learned my addr 172.21.15.113:0/433769427 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.890+0000 7f9458e92700 1 --2- 172.21.15.113:0/433769427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94540f68a0 0x7f94540fa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.891+0000 7f9453fff700 1 -- 172.21.15.113:0/433769427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94540f68a0 msgr2=0x7f94540fa820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.891+0000 7f9453fff700 1 --2- 172.21.15.113:0/433769427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94540f68a0 0x7f94540fa820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:33.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.891+0000 7f9453fff700 1 -- 172.21.15.113:0/433769427 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9440005040 con 0x7f94540f7b40 2024-04-12T05:05:33.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.891+0000 7f9453fff700 1 --2- 172.21.15.113:0/433769427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94540f7b40 0x7f94540f8e80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f9440013510 tx=0x7f9440009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:33.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.891+0000 7f9451ffb700 1 -- 172.21.15.113:0/433769427 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f944000e030 con 0x7f94540f7b40 2024-04-12T05:05:33.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.891+0000 7f945b0f6700 1 -- 172.21.15.113:0/433769427 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94540f9590 con 0x7f94540f7b40 2024-04-12T05:05:33.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.891+0000 7f9451ffb700 1 -- 172.21.15.113:0/433769427 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f9440007e90 con 0x7f94540f7b40 2024-04-12T05:05:33.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.891+0000 7f9451ffb700 1 -- 172.21.15.113:0/433769427 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9440018070 con 0x7f94540f7b40 2024-04-12T05:05:33.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.891+0000 7f945b0f6700 1 -- 172.21.15.113:0/433769427 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94540f9a70 con 0x7f94540f7b40 2024-04-12T05:05:33.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.892+0000 7f945b0f6700 1 -- 172.21.15.113:0/433769427 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94540f9c90 con 0x7f94540f7b40 2024-04-12T05:05:33.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.892+0000 7f9451ffb700 1 -- 172.21.15.113:0/433769427 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f944001d940 con 0x7f94540f7b40 2024-04-12T05:05:33.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.893+0000 7f9451ffb700 1 --2- 172.21.15.113:0/433769427 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9444061180 0x7f9444063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:33.648 INFO:teuthology.orchestra.run.smithi113.stdout:38654705704 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.893+0000 7f9451ffb700 1 -- 172.21.15.113:0/433769427 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f944007dab0 con 0x7f94540f7b40 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.893+0000 7f9458e92700 1 --2- 172.21.15.113:0/433769427 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9444061180 0x7f9444063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.893+0000 7f9458e92700 1 --2- 172.21.15.113:0/433769427 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9444061180 0x7f9444063640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f94540f79a0 tx=0x7f9448009040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:32.897+0000 7f9451ffb700 1 -- 172.21.15.113:0/433769427 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9440051d80 con 0x7f94540f7b40 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.061+0000 7f945b0f6700 1 -- 172.21.15.113:0/433769427 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f94541106f0 con 0x7f94540f7b40 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.062+0000 7f9451ffb700 1 -- 172.21.15.113:0/433769427 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9440051970 con 0x7f94540f7b40 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.064+0000 7f945b0f6700 1 -- 172.21.15.113:0/433769427 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9444061180 msgr2=0x7f9444063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.064+0000 7f945b0f6700 1 --2- 172.21.15.113:0/433769427 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9444061180 0x7f9444063640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f94540f79a0 tx=0x7f9448009040).stop 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.064+0000 7f945b0f6700 1 -- 172.21.15.113:0/433769427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94540f7b40 msgr2=0x7f94540f8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.064+0000 7f945b0f6700 1 --2- 172.21.15.113:0/433769427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94540f7b40 0x7f94540f8e80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f9440013510 tx=0x7f9440009cb0).stop 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.064+0000 7f945b0f6700 1 -- 172.21.15.113:0/433769427 shutdown_connections 2024-04-12T05:05:33.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.064+0000 7f945b0f6700 1 --2- 172.21.15.113:0/433769427 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f9444061180 0x7f9444063640 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:33.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.064+0000 7f945b0f6700 1 --2- 172.21.15.113:0/433769427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94540f68a0 0x7f94540fa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:33.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.064+0000 7f945b0f6700 1 --2- 172.21.15.113:0/433769427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94540f7b40 0x7f94540f8e80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:33.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.064+0000 7f945b0f6700 1 -- 172.21.15.113:0/433769427 >> 172.21.15.113:0/433769427 conn(0x7f94540f2020 msgr2=0x7f9454061310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:33.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.064+0000 7f945b0f6700 1 -- 172.21.15.113:0/433769427 shutdown_connections 2024-04-12T05:05:33.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:33.064+0000 7f945b0f6700 1 -- 172.21.15.113:0/433769427 wait complete. 2024-04-12T05:05:34.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:33.062975+0000 mon.smithi113 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.113:0/433769427' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-12T05:05:34.306 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705702 got 38654705704 for osd.0 2024-04-12T05:05:34.307 DEBUG:teuthology.parallel:result is None 2024-04-12T05:05:34.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:33.062975+0000 mon.smithi113 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.113:0/433769427' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-12T05:05:35.169 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:33.541019+0000 mgr.smithi113.ctsazg (mgr.14194) 172 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:35.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:33.541019+0000 mgr.smithi113.ctsazg (mgr.14194) 172 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:37.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.168+0000 7f2a39e6b700 1 -- 172.21.15.113:0/2707151441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a340617e0 msgr2=0x7f2a340fa580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:37.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.168+0000 7f2a39e6b700 1 --2- 172.21.15.113:0/2707151441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a340617e0 0x7f2a340fa580 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2a1c0047f0 tx=0x7f2a1c009cb0).stop 2024-04-12T05:05:37.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.168+0000 7f2a39e6b700 1 -- 172.21.15.113:0/2707151441 shutdown_connections 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.168+0000 7f2a39e6b700 1 --2- 172.21.15.113:0/2707151441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a340fab00 0x7f2a340fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.168+0000 7f2a39e6b700 1 --2- 172.21.15.113:0/2707151441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a340617e0 0x7f2a340fa580 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.168+0000 7f2a39e6b700 1 -- 172.21.15.113:0/2707151441 >> 172.21.15.113:0/2707151441 conn(0x7f2a340f2020 msgr2=0x7f2a340f4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.168+0000 7f2a39e6b700 1 -- 172.21.15.113:0/2707151441 shutdown_connections 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.168+0000 7f2a39e6b700 1 -- 172.21.15.113:0/2707151441 wait complete. 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.168+0000 7f2a39e6b700 1 Processor -- start 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a39e6b700 1 -- start start 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a39e6b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a340617e0 0x7f2a34108080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a39e6b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a340fab00 0x7f2a34108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a39e6b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a34108b20 con 0x7f2a340617e0 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a39e6b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a34108c60 con 0x7f2a340fab00 2024-04-12T05:05:37.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a337fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a340617e0 0x7f2a34108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a337fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a340617e0 0x7f2a34108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52914/0 (socket says 172.21.15.113:52914) 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a337fe700 1 -- 172.21.15.113:0/1326078479 learned_addr learned my addr 172.21.15.113:0/1326078479 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a32ffd700 1 --2- 172.21.15.113:0/1326078479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a340fab00 0x7f2a34108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a337fe700 1 -- 172.21.15.113:0/1326078479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a340fab00 msgr2=0x7f2a34108570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a337fe700 1 --2- 172.21.15.113:0/1326078479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a340fab00 0x7f2a34108570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a337fe700 1 -- 172.21.15.113:0/1326078479 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c005040 con 0x7f2a340617e0 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a337fe700 1 --2- 172.21.15.113:0/1326078479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a340617e0 0x7f2a34108080 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2a1c013510 tx=0x7f2a1c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a30ff9700 1 -- 172.21.15.113:0/1326078479 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a1c00e030 con 0x7f2a340617e0 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a39e6b700 1 -- 172.21.15.113:0/1326078479 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3410d6a0 con 0x7f2a340617e0 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a30ff9700 1 -- 172.21.15.113:0/1326078479 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f2a1c007e90 con 0x7f2a340617e0 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a30ff9700 1 -- 172.21.15.113:0/1326078479 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a1c018070 con 0x7f2a340617e0 2024-04-12T05:05:37.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.169+0000 7f2a39e6b700 1 -- 172.21.15.113:0/1326078479 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3410dbe0 con 0x7f2a340617e0 2024-04-12T05:05:37.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.170+0000 7f2a39e6b700 1 -- 172.21.15.113:0/1326078479 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a34100140 con 0x7f2a340617e0 2024-04-12T05:05:37.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.171+0000 7f2a30ff9700 1 -- 172.21.15.113:0/1326078479 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f2a1c01d420 con 0x7f2a340617e0 2024-04-12T05:05:37.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.171+0000 7f2a30ff9700 1 --2- 172.21.15.113:0/1326078479 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2a20061180 0x7f2a20063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:37.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.171+0000 7f2a30ff9700 1 -- 172.21.15.113:0/1326078479 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f2a1c07daf0 con 0x7f2a340617e0 2024-04-12T05:05:37.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.171+0000 7f2a32ffd700 1 --2- 172.21.15.113:0/1326078479 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2a20061180 0x7f2a20063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:37.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.171+0000 7f2a32ffd700 1 --2- 172.21.15.113:0/1326078479 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2a20061180 0x7f2a20063640 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2a240041d0 tx=0x7f2a24009040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:37.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.175+0000 7f2a30ff9700 1 -- 172.21.15.113:0/1326078479 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2a1c051dc0 con 0x7f2a340617e0 2024-04-12T05:05:37.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.370+0000 7f7e35bbe700 1 -- 172.21.15.113:0/2689614184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e300f79a0 msgr2=0x7f7e300f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.370+0000 7f7e35bbe700 1 --2- 172.21.15.113:0/2689614184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e300f79a0 0x7f7e300f7dc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f7e240047f0 tx=0x7f7e24009cb0).stop 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.370+0000 7f7e35bbe700 1 -- 172.21.15.113:0/2689614184 shutdown_connections 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.370+0000 7f7e35bbe700 1 --2- 172.21.15.113:0/2689614184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e300f8c40 0x7f7e300f90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.370+0000 7f7e35bbe700 1 --2- 172.21.15.113:0/2689614184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e300f79a0 0x7f7e300f7dc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.370+0000 7f7e35bbe700 1 -- 172.21.15.113:0/2689614184 >> 172.21.15.113:0/2689614184 conn(0x7f7e300f3100 msgr2=0x7f7e300f5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.370+0000 7f7e35bbe700 1 -- 172.21.15.113:0/2689614184 shutdown_connections 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.370+0000 7f7e35bbe700 1 -- 172.21.15.113:0/2689614184 wait complete. 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.370+0000 7f7e35bbe700 1 Processor -- start 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.370+0000 7f7e35bbe700 1 -- start start 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e35bbe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e300f79a0 0x7f7e3006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e35bbe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e300f8c40 0x7f7e3006b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e35bbe700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3006cf80 con 0x7f7e300f8c40 2024-04-12T05:05:37.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e35bbe700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3006b520 con 0x7f7e300f79a0 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e2effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e300f8c40 0x7f7e3006b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e2effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e300f8c40 0x7f7e3006b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52894/0 (socket says 172.21.15.113:52894) 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e2effd700 1 -- 172.21.15.113:0/1653547033 learned_addr learned my addr 172.21.15.113:0/1653547033 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e2f7fe700 1 --2- 172.21.15.113:0/1653547033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e300f79a0 0x7f7e3006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e2effd700 1 -- 172.21.15.113:0/1653547033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e300f79a0 msgr2=0x7f7e3006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e2effd700 1 --2- 172.21.15.113:0/1653547033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e300f79a0 0x7f7e3006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e2effd700 1 -- 172.21.15.113:0/1653547033 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e24005040 con 0x7f7e300f8c40 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e2effd700 1 --2- 172.21.15.113:0/1653547033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e300f8c40 0x7f7e3006b030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7e2000c740 tx=0x7f7e20007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e2cff9700 1 -- 172.21.15.113:0/1653547033 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e2000e680 con 0x7f7e300f8c40 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e2cff9700 1 -- 172.21.15.113:0/1653547033 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f7e2000ec70 con 0x7f7e300f8c40 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e35bbe700 1 -- 172.21.15.113:0/1653547033 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3006b7a0 con 0x7f7e300f8c40 2024-04-12T05:05:37.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e2cff9700 1 -- 172.21.15.113:0/1653547033 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e20013d00 con 0x7f7e300f8c40 2024-04-12T05:05:37.338 INFO:teuthology.orchestra.run.smithi113.stdout:137438953489 2024-04-12T05:05:37.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.371+0000 7f7e35bbe700 1 -- 172.21.15.113:0/1653547033 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3006bce0 con 0x7f7e300f8c40 2024-04-12T05:05:37.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.372+0000 7f7e35bbe700 1 -- 172.21.15.113:0/1653547033 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e30047540 con 0x7f7e300f8c40 2024-04-12T05:05:37.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.372+0000 7f7e2cff9700 1 -- 172.21.15.113:0/1653547033 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f7e2000e7e0 con 0x7f7e300f8c40 2024-04-12T05:05:37.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.373+0000 7f7e2cff9700 1 --2- 172.21.15.113:0/1653547033 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f7e18061180 0x7f7e18063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:37.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.373+0000 7f7e2cff9700 1 -- 172.21.15.113:0/1653547033 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f7e2007c700 con 0x7f7e300f8c40 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.373+0000 7f7e2f7fe700 1 --2- 172.21.15.113:0/1653547033 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f7e18061180 0x7f7e18063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.373+0000 7f7e2f7fe700 1 --2- 172.21.15.113:0/1653547033 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f7e18061180 0x7f7e18063640 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f7e24004f10 tx=0x7f7e2400b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.377+0000 7f7e2cff9700 1 -- 172.21.15.113:0/1653547033 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7e20050a50 con 0x7f7e300f8c40 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.542+0000 7f7e35bbe700 1 -- 172.21.15.113:0/1653547033 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f7e300598f0 con 0x7f7e300f8c40 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.542+0000 7f7e2cff9700 1 -- 172.21.15.113:0/1653547033 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7e20013e60 con 0x7f7e300f8c40 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.544+0000 7f7e35bbe700 1 -- 172.21.15.113:0/1653547033 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f7e18061180 msgr2=0x7f7e18063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.544+0000 7f7e35bbe700 1 --2- 172.21.15.113:0/1653547033 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f7e18061180 0x7f7e18063640 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f7e24004f10 tx=0x7f7e2400b040).stop 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.544+0000 7f7e35bbe700 1 -- 172.21.15.113:0/1653547033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e300f8c40 msgr2=0x7f7e3006b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.544+0000 7f7e35bbe700 1 --2- 172.21.15.113:0/1653547033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e300f8c40 0x7f7e3006b030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7e2000c740 tx=0x7f7e20007ef0).stop 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.544+0000 7f7e35bbe700 1 -- 172.21.15.113:0/1653547033 shutdown_connections 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.544+0000 7f7e35bbe700 1 --2- 172.21.15.113:0/1653547033 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f7e18061180 0x7f7e18063640 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.544+0000 7f7e35bbe700 1 --2- 172.21.15.113:0/1653547033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e300f79a0 0x7f7e3006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.544+0000 7f7e35bbe700 1 --2- 172.21.15.113:0/1653547033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e300f8c40 0x7f7e3006b030 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.544+0000 7f7e35bbe700 1 -- 172.21.15.113:0/1653547033 >> 172.21.15.113:0/1653547033 conn(0x7f7e300f3100 msgr2=0x7f7e300fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:37.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.544+0000 7f7e35bbe700 1 -- 172.21.15.113:0/1653547033 shutdown_connections 2024-04-12T05:05:37.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:36.544+0000 7f7e35bbe700 1 -- 172.21.15.113:0/1653547033 wait complete. 2024-04-12T05:05:37.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.341+0000 7f2a39e6b700 1 -- 172.21.15.113:0/1326078479 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2a340598f0 con 0x7f2a340617e0 2024-04-12T05:05:37.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.341+0000 7f2a30ff9700 1 -- 172.21.15.113:0/1326078479 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2a1c0519b0 con 0x7f2a340617e0 2024-04-12T05:05:37.343 INFO:teuthology.orchestra.run.smithi113.stdout:111669149718 2024-04-12T05:05:37.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.343+0000 7f2a39e6b700 1 -- 172.21.15.113:0/1326078479 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2a20061180 msgr2=0x7f2a20063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:37.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.343+0000 7f2a39e6b700 1 --2- 172.21.15.113:0/1326078479 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2a20061180 0x7f2a20063640 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2a240041d0 tx=0x7f2a24009040).stop 2024-04-12T05:05:37.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.343+0000 7f2a39e6b700 1 -- 172.21.15.113:0/1326078479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a340617e0 msgr2=0x7f2a34108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:37.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.343+0000 7f2a39e6b700 1 --2- 172.21.15.113:0/1326078479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a340617e0 0x7f2a34108080 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2a1c013510 tx=0x7f2a1c009cb0).stop 2024-04-12T05:05:37.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.343+0000 7f2a39e6b700 1 -- 172.21.15.113:0/1326078479 shutdown_connections 2024-04-12T05:05:37.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.343+0000 7f2a39e6b700 1 --2- 172.21.15.113:0/1326078479 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2a20061180 0x7f2a20063640 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.343+0000 7f2a39e6b700 1 --2- 172.21.15.113:0/1326078479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a340617e0 0x7f2a34108080 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.343+0000 7f2a39e6b700 1 --2- 172.21.15.113:0/1326078479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a340fab00 0x7f2a34108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:37.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.343+0000 7f2a39e6b700 1 -- 172.21.15.113:0/1326078479 >> 172.21.15.113:0/1326078479 conn(0x7f2a340f2020 msgr2=0x7f2a340f4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:37.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.343+0000 7f2a39e6b700 1 -- 172.21.15.113:0/1326078479 shutdown_connections 2024-04-12T05:05:37.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:37.343+0000 7f2a39e6b700 1 -- 172.21.15.113:0/1326078479 wait complete. 2024-04-12T05:05:37.366 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:35.541289+0000 mgr.smithi113.ctsazg (mgr.14194) 173 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:37.366 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:36.543501+0000 mon.smithi113 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.113:0/1653547033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-12T05:05:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:35.541289+0000 mgr.smithi113.ctsazg (mgr.14194) 173 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:36.543501+0000 mon.smithi113 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.113:0/1653547033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-12T05:05:37.677 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:38.082 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149715 got 111669149718 for osd.3 2024-04-12T05:05:38.083 DEBUG:teuthology.parallel:result is None 2024-04-12T05:05:38.316 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:37.342157+0000 mon.smithi113 (mon.0) 497 : audit [DBG] from='client.? 172.21.15.113:0/1326078479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-12T05:05:38.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:37.342157+0000 mon.smithi113 (mon.0) 497 : audit [DBG] from='client.? 172.21.15.113:0/1326078479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-12T05:05:38.869 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953486 got 137438953489 for osd.4 2024-04-12T05:05:38.869 DEBUG:teuthology.parallel:result is None 2024-04-12T05:05:39.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:37.541778+0000 mgr.smithi113.ctsazg (mgr.14194) 174 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:37.541778+0000 mgr.smithi113.ctsazg (mgr.14194) 174 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:40.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.337+0000 7fc37805f700 1 -- 172.21.15.113:0/2092451020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3700f8c50 msgr2=0x7fc3700f90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:40.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.337+0000 7fc37805f700 1 --2- 172.21.15.113:0/2092451020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3700f8c50 0x7fc3700f90b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fc3600047f0 tx=0x7fc360009cb0).stop 2024-04-12T05:05:40.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.337+0000 7fc37805f700 1 -- 172.21.15.113:0/2092451020 shutdown_connections 2024-04-12T05:05:40.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.337+0000 7fc37805f700 1 --2- 172.21.15.113:0/2092451020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3700f8c50 0x7fc3700f90b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:40.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.337+0000 7fc37805f700 1 --2- 172.21.15.113:0/2092451020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3700f79b0 0x7fc3700f7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:40.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.337+0000 7fc37805f700 1 -- 172.21.15.113:0/2092451020 >> 172.21.15.113:0/2092451020 conn(0x7fc3700f30f0 msgr2=0x7fc3700f5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:40.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.337+0000 7fc37805f700 1 -- 172.21.15.113:0/2092451020 shutdown_connections 2024-04-12T05:05:40.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.337+0000 7fc37805f700 1 -- 172.21.15.113:0/2092451020 wait complete. 2024-04-12T05:05:40.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc37805f700 1 Processor -- start 2024-04-12T05:05:40.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc37805f700 1 -- start start 2024-04-12T05:05:40.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc37805f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3700f79b0 0x7fc370108090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:40.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc37805f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3700f8c50 0x7fc370108580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc37805f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc370108b30 con 0x7fc3700f8c50 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc37805f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc370108c70 con 0x7fc3700f79b0 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc3755fa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3700f8c50 0x7fc370108580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc3755fa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3700f8c50 0x7fc370108580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60518/0 (socket says 172.21.15.113:60518) 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc3755fa700 1 -- 172.21.15.113:0/3507303100 learned_addr learned my addr 172.21.15.113:0/3507303100 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc375dfb700 1 --2- 172.21.15.113:0/3507303100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3700f79b0 0x7fc370108090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc3755fa700 1 -- 172.21.15.113:0/3507303100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3700f79b0 msgr2=0x7fc370108090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc3755fa700 1 --2- 172.21.15.113:0/3507303100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3700f79b0 0x7fc370108090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.338+0000 7fc3755fa700 1 -- 172.21.15.113:0/3507303100 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc360005040 con 0x7fc3700f8c50 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.339+0000 7fc3755fa700 1 --2- 172.21.15.113:0/3507303100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3700f8c50 0x7fc370108580 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fc360013470 tx=0x7fc360009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.339+0000 7fc366ffd700 1 -- 172.21.15.113:0/3507303100 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc36000e030 con 0x7fc3700f8c50 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.339+0000 7fc37805f700 1 -- 172.21.15.113:0/3507303100 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37010d6b0 con 0x7fc3700f8c50 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.339+0000 7fc366ffd700 1 -- 172.21.15.113:0/3507303100 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fc360007e90 con 0x7fc3700f8c50 2024-04-12T05:05:40.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.339+0000 7fc366ffd700 1 -- 172.21.15.113:0/3507303100 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc360018070 con 0x7fc3700f8c50 2024-04-12T05:05:40.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.339+0000 7fc37805f700 1 -- 172.21.15.113:0/3507303100 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37010dbf0 con 0x7fc3700f8c50 2024-04-12T05:05:40.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.340+0000 7fc37805f700 1 -- 172.21.15.113:0/3507303100 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc370100100 con 0x7fc3700f8c50 2024-04-12T05:05:40.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.341+0000 7fc366ffd700 1 -- 172.21.15.113:0/3507303100 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fc36001d420 con 0x7fc3700f8c50 2024-04-12T05:05:40.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.341+0000 7fc366ffd700 1 --2- 172.21.15.113:0/3507303100 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc35c061110 0x7fc35c0635d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:40.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.341+0000 7fc366ffd700 1 -- 172.21.15.113:0/3507303100 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7fc36007df20 con 0x7fc3700f8c50 2024-04-12T05:05:40.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.341+0000 7fc375dfb700 1 --2- 172.21.15.113:0/3507303100 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc35c061110 0x7fc35c0635d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:40.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.341+0000 7fc375dfb700 1 --2- 172.21.15.113:0/3507303100 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc35c061110 0x7fc35c0635d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fc3700f8ab0 tx=0x7fc36800b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:40.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.345+0000 7fc366ffd700 1 -- 172.21.15.113:0/3507303100 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc360052270 con 0x7fc3700f8c50 2024-04-12T05:05:40.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.508+0000 7fc37805f700 1 -- 172.21.15.113:0/3507303100 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc370101f70 con 0x7fc3700f8c50 2024-04-12T05:05:40.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.508+0000 7fc366ffd700 1 -- 172.21.15.113:0/3507303100 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc360051e60 con 0x7fc3700f8c50 2024-04-12T05:05:40.510 INFO:teuthology.orchestra.run.smithi113.stdout:158913789964 2024-04-12T05:05:40.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.510+0000 7fc37805f700 1 -- 172.21.15.113:0/3507303100 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc35c061110 msgr2=0x7fc35c0635d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:40.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.510+0000 7fc37805f700 1 --2- 172.21.15.113:0/3507303100 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc35c061110 0x7fc35c0635d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fc3700f8ab0 tx=0x7fc36800b040).stop 2024-04-12T05:05:40.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.510+0000 7fc37805f700 1 -- 172.21.15.113:0/3507303100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3700f8c50 msgr2=0x7fc370108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:40.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.510+0000 7fc37805f700 1 --2- 172.21.15.113:0/3507303100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3700f8c50 0x7fc370108580 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fc360013470 tx=0x7fc360009cb0).stop 2024-04-12T05:05:40.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.510+0000 7fc37805f700 1 -- 172.21.15.113:0/3507303100 shutdown_connections 2024-04-12T05:05:40.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.510+0000 7fc37805f700 1 --2- 172.21.15.113:0/3507303100 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc35c061110 0x7fc35c0635d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:40.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.510+0000 7fc37805f700 1 --2- 172.21.15.113:0/3507303100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3700f79b0 0x7fc370108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:40.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.510+0000 7fc37805f700 1 --2- 172.21.15.113:0/3507303100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3700f8c50 0x7fc370108580 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:40.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.510+0000 7fc37805f700 1 -- 172.21.15.113:0/3507303100 >> 172.21.15.113:0/3507303100 conn(0x7fc3700f30f0 msgr2=0x7fc3700fc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:40.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.510+0000 7fc37805f700 1 -- 172.21.15.113:0/3507303100 shutdown_connections 2024-04-12T05:05:40.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:40.510+0000 7fc37805f700 1 -- 172.21.15.113:0/3507303100 wait complete. 2024-04-12T05:05:41.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:39.542067+0000 mgr.smithi113.ctsazg (mgr.14194) 175 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:41.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:40.509217+0000 mon.smithi113 (mon.0) 498 : audit [DBG] from='client.? 172.21.15.113:0/3507303100' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-12T05:05:41.223 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789961 got 158913789964 for osd.5 2024-04-12T05:05:41.223 DEBUG:teuthology.parallel:result is None 2024-04-12T05:05:41.224 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-12T05:05:41.224 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph pg dump --format=json 2024-04-12T05:05:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:39.542067+0000 mgr.smithi113.ctsazg (mgr.14194) 175 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:40.509217+0000 mon.smithi113 (mon.0) 498 : audit [DBG] from='client.? 172.21.15.113:0/3507303100' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-12T05:05:41.491 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:43.119 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:41.542510+0000 mgr.smithi113.ctsazg (mgr.14194) 176 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:43.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.375+0000 7fba61826700 1 -- 172.21.15.113:0/3351297295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba5c0f7b40 msgr2=0x7fba5c0f7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:43.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.375+0000 7fba61826700 1 --2- 172.21.15.113:0/3351297295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba5c0f7b40 0x7fba5c0f7fa0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fba500047f0 tx=0x7fba50009cb0).stop 2024-04-12T05:05:43.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.375+0000 7fba61826700 1 -- 172.21.15.113:0/3351297295 shutdown_connections 2024-04-12T05:05:43.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.375+0000 7fba61826700 1 --2- 172.21.15.113:0/3351297295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba5c0f7b40 0x7fba5c0f7fa0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:43.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.375+0000 7fba61826700 1 --2- 172.21.15.113:0/3351297295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba5c0f68a0 0x7fba5c0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:05:43.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.375+0000 7fba61826700 1 -- 172.21.15.113:0/3351297295 >> 172.21.15.113:0/3351297295 conn(0x7fba5c0f2020 msgr2=0x7fba5c0f4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:43.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.375+0000 7fba61826700 1 -- 172.21.15.113:0/3351297295 shutdown_connections 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.375+0000 7fba61826700 1 -- 172.21.15.113:0/3351297295 wait complete. 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.376+0000 7fba61826700 1 Processor -- start 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.376+0000 7fba61826700 1 -- start start 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.376+0000 7fba61826700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba5c0f68a0 0x7fba5c106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.376+0000 7fba61826700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba5c0f7b40 0x7fba5c1074d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.376+0000 7fba61826700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c107b10 con 0x7fba5c0f68a0 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.376+0000 7fba61826700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c107c50 con 0x7fba5c0f7b40 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.376+0000 7fba5affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba5c0f68a0 0x7fba5c106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba5affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba5c0f68a0 0x7fba5c106fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60532/0 (socket says 172.21.15.113:60532) 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba5affd700 1 -- 172.21.15.113:0/2202756623 learned_addr learned my addr 172.21.15.113:0/2202756623 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba5a7fc700 1 --2- 172.21.15.113:0/2202756623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba5c0f7b40 0x7fba5c1074d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba5affd700 1 -- 172.21.15.113:0/2202756623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba5c0f7b40 msgr2=0x7fba5c1074d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba5affd700 1 --2- 172.21.15.113:0/2202756623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba5c0f7b40 0x7fba5c1074d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:43.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba5affd700 1 -- 172.21.15.113:0/2202756623 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba50005040 con 0x7fba5c0f68a0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba5affd700 1 --2- 172.21.15.113:0/2202756623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba5c0f68a0 0x7fba5c106fe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fba4c00b480 tx=0x7fba4c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba43fff700 1 -- 172.21.15.113:0/2202756623 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba4c002500 con 0x7fba5c0f68a0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba61826700 1 -- 172.21.15.113:0/2202756623 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c10c6f0 con 0x7fba5c0f68a0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba43fff700 1 -- 172.21.15.113:0/2202756623 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fba4c002af0 con 0x7fba5c0f68a0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba43fff700 1 -- 172.21.15.113:0/2202756623 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba4c012c40 con 0x7fba5c0f68a0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.377+0000 7fba61826700 1 -- 172.21.15.113:0/2202756623 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c10cc30 con 0x7fba5c0f68a0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.378+0000 7fba61826700 1 -- 172.21.15.113:0/2202756623 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5c0ff090 con 0x7fba5c0f68a0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.378+0000 7fba43fff700 1 -- 172.21.15.113:0/2202756623 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fba4c002660 con 0x7fba5c0f68a0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.379+0000 7fba43fff700 1 --2- 172.21.15.113:0/2202756623 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fba44061180 0x7fba44063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.379+0000 7fba43fff700 1 -- 172.21.15.113:0/2202756623 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7fba4c07c5e0 con 0x7fba5c0f68a0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.379+0000 7fba5a7fc700 1 --2- 172.21.15.113:0/2202756623 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fba44061180 0x7fba44063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.379+0000 7fba5a7fc700 1 --2- 172.21.15.113:0/2202756623 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fba44061180 0x7fba44063640 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fba50007cf0 tx=0x7fba5000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:43.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.383+0000 7fba43fff700 1 -- 172.21.15.113:0/2202756623 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fba4c0508b0 con 0x7fba5c0f68a0 2024-04-12T05:05:43.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:41.542510+0000 mgr.smithi113.ctsazg (mgr.14194) 176 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:43.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.543+0000 7fba61826700 1 -- 172.21.15.113:0/2202756623 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fba5c059120 con 0x7fba44061180 2024-04-12T05:05:43.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.545+0000 7fba43fff700 1 -- 172.21.15.113:0/2202756623 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27687 (secure 0 0 0) 0x7fba5c059120 con 0x7fba44061180 2024-04-12T05:05:43.547 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:05:43.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.546+0000 7fba61826700 1 -- 172.21.15.113:0/2202756623 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fba44061180 msgr2=0x7fba44063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:43.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.546+0000 7fba61826700 1 --2- 172.21.15.113:0/2202756623 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fba44061180 0x7fba44063640 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fba50007cf0 tx=0x7fba5000b040).stop 2024-04-12T05:05:43.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.546+0000 7fba61826700 1 -- 172.21.15.113:0/2202756623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba5c0f68a0 msgr2=0x7fba5c106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:43.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.546+0000 7fba61826700 1 --2- 172.21.15.113:0/2202756623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba5c0f68a0 0x7fba5c106fe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fba4c00b480 tx=0x7fba4c0090f0).stop 2024-04-12T05:05:43.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.547+0000 7fba61826700 1 -- 172.21.15.113:0/2202756623 shutdown_connections 2024-04-12T05:05:43.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.547+0000 7fba61826700 1 --2- 172.21.15.113:0/2202756623 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fba44061180 0x7fba44063640 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:43.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.547+0000 7fba61826700 1 --2- 172.21.15.113:0/2202756623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba5c0f68a0 0x7fba5c106fe0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:43.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.547+0000 7fba61826700 1 --2- 172.21.15.113:0/2202756623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba5c0f7b40 0x7fba5c1074d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:43.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.547+0000 7fba61826700 1 -- 172.21.15.113:0/2202756623 >> 172.21.15.113:0/2202756623 conn(0x7fba5c0f2020 msgr2=0x7fba5c0fb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:43.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.547+0000 7fba61826700 1 -- 172.21.15.113:0/2202756623 shutdown_connections 2024-04-12T05:05:43.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:43.547+0000 7fba61826700 1 -- 172.21.15.113:0/2202756623 wait complete. 2024-04-12T05:05:43.550 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-04-12T05:05:44.300 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-04-12T05:05:43.542675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1784564,"kb_used_data":1988,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":560686348,"statfs":{"total":575970213888,"available":574142820352,"internally_reserved":0,"allocated":2035712,"data_stored":761538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":36,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-12T05:04:50.805411+0000","last_change":"2024-04-12T05:04:23.474044+0000","last_active":"2024-04-12T05:04:50.805411+0000","last_peered":"2024-04-12T05:04:50.805411+0000","last_clean":"2024-04-12T05:04:50.805411+0000","last_became_active":"2024-04-12T05:04:23.473719+0000","last_became_peered":"2024-04-12T05:04:23.473719+0000","last_unstale":"2024-04-12T05:04:50.805411+0000","last_undegraded":"2024-04-12T05:04:50.805411+0000","last_fullsized":"2024-04-12T05:04:50.805411+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T05:03:32.585539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T05:03:32.585539+0000","last_clean_scrub_stamp":"2024-04-12T05:03:32.585539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":37,"seq":158913789965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297428,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447724,"statfs":{"total":95995035648,"available":95690469376,"internally_reserved":0,"allocated":339968,"data_stored":127036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953491,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297428,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447724,"statfs":{"total":95995035648,"available":95690469376,"internally_reserved":0,"allocated":339968,"data_stored":127036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.622}]},{"osd":1,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.50900000000000001}]},{"osd":2,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.65400000000000003}]},{"osd":3,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.495},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297428,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447724,"statfs":{"total":95995035648,"available":95690469376,"internally_reserved":0,"allocated":339968,"data_stored":127036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri Apr 12 05:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.495}]},{"osd":2,"last update":"Fri Apr 12 05:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53400000000000003}]},{"osd":4,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.443},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.57599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]},{"osd":2,"up_from":19,"seq":81604378654,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297424,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447728,"statfs":{"total":95995035648,"available":95690473472,"internally_reserved":0,"allocated":335872,"data_stored":126358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:05:34 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.35499999999999998,"15min":0.35699999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.29899999999999999}]},{"osd":1,"last update":"Fri Apr 12 05:05:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54800000000000004,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.378,"15min":0.38},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.28000000000000003}]},{"osd":3,"last update":"Fri Apr 12 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.57899999999999996}]},{"osd":4,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.46999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705706,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297428,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447724,"statfs":{"total":95995035648,"available":95690469376,"internally_reserved":0,"allocated":339968,"data_stored":127036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 12 05:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.34399999999999997},"min":{"1min":0.24099999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.34399999999999997},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55100000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.32600000000000001}]},{"osd":2,"last update":"Fri Apr 12 05:05:38 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.36799999999999999,"15min":0.372},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.38500000000000001,"15min":0.39200000000000002},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.28100000000000003}]},{"osd":3,"last update":"Fri Apr 12 05:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.63900000000000001}]},{"osd":4,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.68999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297428,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447724,"statfs":{"total":95995035648,"available":95690469376,"internally_reserved":0,"allocated":339968,"data_stored":127036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:05:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.33300000000000002,"15min":0.33100000000000002},"min":{"1min":0.218,"5min":0.186,"15min":0.186},"max":{"1min":0.47899999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.24299999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48799999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.23999999999999999}]},{"osd":2,"last update":"Fri Apr 12 05:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.35599999999999998,"15min":0.36399999999999999},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.438,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.38500000000000001,"15min":0.39600000000000002},"min":{"1min":0.25,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.48899999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.28100000000000003}]},{"osd":3,"last update":"Fri Apr 12 05:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.626}]},{"osd":4,"last update":"Fri Apr 12 05:05:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T05:05:44.301 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph pg dump --format=json 2024-04-12T05:05:44.565 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:45.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:43.542783+0000 mgr.smithi113.ctsazg (mgr.14194) 177 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:43.544844+0000 mgr.smithi113.ctsazg (mgr.14194) 178 : audit [DBG] from='client.14382 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T05:05:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:43.542783+0000 mgr.smithi113.ctsazg (mgr.14194) 177 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:43.544844+0000 mgr.smithi113.ctsazg (mgr.14194) 178 : audit [DBG] from='client.14382 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T05:05:46.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.475+0000 7f6515236700 1 -- 172.21.15.113:0/2632458533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65100f79a0 msgr2=0x7f65100f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:46.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.475+0000 7f6515236700 1 --2- 172.21.15.113:0/2632458533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65100f79a0 0x7f65100f7dc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f65040047f0 tx=0x7f6504009cb0).stop 2024-04-12T05:05:46.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.475+0000 7f6515236700 1 -- 172.21.15.113:0/2632458533 shutdown_connections 2024-04-12T05:05:46.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.475+0000 7f6515236700 1 --2- 172.21.15.113:0/2632458533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65100f8c40 0x7f65100f90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:05:46.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.475+0000 7f6515236700 1 --2- 172.21.15.113:0/2632458533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65100f79a0 0x7f65100f7dc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:46.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.475+0000 7f6515236700 1 -- 172.21.15.113:0/2632458533 >> 172.21.15.113:0/2632458533 conn(0x7f65100f3100 msgr2=0x7f65100f5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:46.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.475+0000 7f6515236700 1 -- 172.21.15.113:0/2632458533 shutdown_connections 2024-04-12T05:05:46.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.475+0000 7f6515236700 1 -- 172.21.15.113:0/2632458533 wait complete. 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.476+0000 7f6515236700 1 Processor -- start 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.476+0000 7f6515236700 1 -- start start 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.476+0000 7f6515236700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65100f79a0 0x7f6510108110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.476+0000 7f6515236700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65100f8c40 0x7f6510108600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.476+0000 7f6515236700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6510108c40 con 0x7f65100f79a0 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.476+0000 7f6515236700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6510108d80 con 0x7f65100f8c40 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.476+0000 7f650ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65100f79a0 0x7f6510108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.476+0000 7f650ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65100f79a0 0x7f6510108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60544/0 (socket says 172.21.15.113:60544) 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.476+0000 7f650ed9d700 1 -- 172.21.15.113:0/3500827836 learned_addr learned my addr 172.21.15.113:0/3500827836 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.477+0000 7f650e59c700 1 --2- 172.21.15.113:0/3500827836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65100f8c40 0x7f6510108600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.477+0000 7f650ed9d700 1 -- 172.21.15.113:0/3500827836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65100f8c40 msgr2=0x7f6510108600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:46.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.477+0000 7f650ed9d700 1 --2- 172.21.15.113:0/3500827836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65100f8c40 0x7f6510108600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:46.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.477+0000 7f650ed9d700 1 -- 172.21.15.113:0/3500827836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6504005040 con 0x7f65100f79a0 2024-04-12T05:05:46.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.477+0000 7f650ed9d700 1 --2- 172.21.15.113:0/3500827836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65100f79a0 0x7f6510108110 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6504013470 tx=0x7f6504009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:46.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.477+0000 7f64f7fff700 1 -- 172.21.15.113:0/3500827836 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f650400e030 con 0x7f65100f79a0 2024-04-12T05:05:46.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.477+0000 7f6515236700 1 -- 172.21.15.113:0/3500827836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651010d7c0 con 0x7f65100f79a0 2024-04-12T05:05:46.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.477+0000 7f64f7fff700 1 -- 172.21.15.113:0/3500827836 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f6504007e90 con 0x7f65100f79a0 2024-04-12T05:05:46.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.477+0000 7f64f7fff700 1 -- 172.21.15.113:0/3500827836 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6504018070 con 0x7f65100f79a0 2024-04-12T05:05:46.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.477+0000 7f6515236700 1 -- 172.21.15.113:0/3500827836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651010dca0 con 0x7f65100f79a0 2024-04-12T05:05:46.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.478+0000 7f6515236700 1 -- 172.21.15.113:0/3500827836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6510100170 con 0x7f65100f79a0 2024-04-12T05:05:46.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.478+0000 7f64f7fff700 1 -- 172.21.15.113:0/3500827836 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f650401d940 con 0x7f65100f79a0 2024-04-12T05:05:46.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.479+0000 7f64f7fff700 1 --2- 172.21.15.113:0/3500827836 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f64f8061160 0x7f64f8063620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:46.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.479+0000 7f64f7fff700 1 -- 172.21.15.113:0/3500827836 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f650407dc30 con 0x7f65100f79a0 2024-04-12T05:05:46.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.479+0000 7f650e59c700 1 --2- 172.21.15.113:0/3500827836 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f64f8061160 0x7f64f8063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:46.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.479+0000 7f650e59c700 1 --2- 172.21.15.113:0/3500827836 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f64f8061160 0x7f64f8063620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f650000b910 tx=0x7f6500008040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:46.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.483+0000 7f64f7fff700 1 -- 172.21.15.113:0/3500827836 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6504084080 con 0x7f65100f79a0 2024-04-12T05:05:46.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.641+0000 7f6515236700 1 -- 172.21.15.113:0/3500827836 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6510059120 con 0x7f64f8061160 2024-04-12T05:05:46.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.642+0000 7f64f7fff700 1 -- 172.21.15.113:0/3500827836 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27673 (secure 0 0 0) 0x7f6510059120 con 0x7f64f8061160 2024-04-12T05:05:46.644 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:05:46.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.644+0000 7f6515236700 1 -- 172.21.15.113:0/3500827836 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f64f8061160 msgr2=0x7f64f8063620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:46.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.644+0000 7f6515236700 1 --2- 172.21.15.113:0/3500827836 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f64f8061160 0x7f64f8063620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f650000b910 tx=0x7f6500008040).stop 2024-04-12T05:05:46.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.644+0000 7f6515236700 1 -- 172.21.15.113:0/3500827836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65100f79a0 msgr2=0x7f6510108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:46.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.644+0000 7f6515236700 1 --2- 172.21.15.113:0/3500827836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65100f79a0 0x7f6510108110 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6504013470 tx=0x7f6504009cb0).stop 2024-04-12T05:05:46.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.644+0000 7f6515236700 1 -- 172.21.15.113:0/3500827836 shutdown_connections 2024-04-12T05:05:46.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.644+0000 7f6515236700 1 --2- 172.21.15.113:0/3500827836 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f64f8061160 0x7f64f8063620 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:46.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.644+0000 7f6515236700 1 --2- 172.21.15.113:0/3500827836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65100f79a0 0x7f6510108110 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:46.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.644+0000 7f6515236700 1 --2- 172.21.15.113:0/3500827836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65100f8c40 0x7f6510108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:46.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.644+0000 7f6515236700 1 -- 172.21.15.113:0/3500827836 >> 172.21.15.113:0/3500827836 conn(0x7f65100f3100 msgr2=0x7f65100fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:46.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.644+0000 7f6515236700 1 -- 172.21.15.113:0/3500827836 shutdown_connections 2024-04-12T05:05:46.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:46.644+0000 7f6515236700 1 -- 172.21.15.113:0/3500827836 wait complete. 2024-04-12T05:05:46.647 INFO:teuthology.orchestra.run.smithi113.stderr:dumped all 2024-04-12T05:05:47.319 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:45.543010+0000 mgr.smithi113.ctsazg (mgr.14194) 179 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:47.342 INFO:teuthology.orchestra.run.smithi113.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-04-12T05:05:45.542892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1784564,"kb_used_data":1988,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":560686348,"statfs":{"total":575970213888,"available":574142820352,"internally_reserved":0,"allocated":2035712,"data_stored":761538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":36,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-12T05:04:50.805411+0000","last_change":"2024-04-12T05:04:23.474044+0000","last_active":"2024-04-12T05:04:50.805411+0000","last_peered":"2024-04-12T05:04:50.805411+0000","last_clean":"2024-04-12T05:04:50.805411+0000","last_became_active":"2024-04-12T05:04:23.473719+0000","last_became_peered":"2024-04-12T05:04:23.473719+0000","last_unstale":"2024-04-12T05:04:50.805411+0000","last_undegraded":"2024-04-12T05:04:50.805411+0000","last_fullsized":"2024-04-12T05:04:50.805411+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T05:03:32.585539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T05:03:32.585539+0000","last_clean_scrub_stamp":"2024-04-12T05:03:32.585539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":37,"seq":158913789965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297428,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447724,"statfs":{"total":95995035648,"available":95690469376,"internally_reserved":0,"allocated":339968,"data_stored":127036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953491,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297428,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447724,"statfs":{"total":95995035648,"available":95690469376,"internally_reserved":0,"allocated":339968,"data_stored":127036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.622}]},{"osd":1,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.50900000000000001}]},{"osd":2,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.65400000000000003}]},{"osd":3,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.495},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297428,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447724,"statfs":{"total":95995035648,"available":95690469376,"internally_reserved":0,"allocated":339968,"data_stored":127036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri Apr 12 05:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.495}]},{"osd":2,"last update":"Fri Apr 12 05:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53400000000000003}]},{"osd":4,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.443},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.57599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]}]},{"osd":2,"up_from":19,"seq":81604378654,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297424,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447728,"statfs":{"total":95995035648,"available":95690473472,"internally_reserved":0,"allocated":335872,"data_stored":126358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:05:34 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.35499999999999998,"15min":0.35699999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.29899999999999999}]},{"osd":1,"last update":"Fri Apr 12 05:05:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54800000000000004,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.378,"15min":0.38},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.28000000000000003}]},{"osd":3,"last update":"Fri Apr 12 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.57899999999999996}]},{"osd":4,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.46999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705707,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297428,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447724,"statfs":{"total":95995035648,"available":95690469376,"internally_reserved":0,"allocated":339968,"data_stored":127036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 12 05:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.34399999999999997},"min":{"1min":0.24099999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.34399999999999997},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55100000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.41499999999999998}]},{"osd":2,"last update":"Fri Apr 12 05:05:38 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.36799999999999999,"15min":0.372},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.38500000000000001,"15min":0.39200000000000002},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Apr 12 05:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.70899999999999996}]},{"osd":4,"last update":"Fri Apr 12 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.624},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.68500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542180,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297428,"kb_used_data":332,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447724,"statfs":{"total":95995035648,"available":95690469376,"internally_reserved":0,"allocated":339968,"data_stored":127036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 05:05:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.33300000000000002,"15min":0.33100000000000002},"min":{"1min":0.218,"5min":0.186,"15min":0.186},"max":{"1min":0.47899999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.24299999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48799999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.24099999999999999}]},{"osd":2,"last update":"Fri Apr 12 05:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.35599999999999998,"15min":0.36399999999999999},"min":{"1min":0.224,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.438,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.38500000000000001,"15min":0.39600000000000002},"min":{"1min":0.25,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.48899999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.28199999999999997}]},{"osd":3,"last update":"Fri Apr 12 05:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47799999999999998}]},{"osd":4,"last update":"Fri Apr 12 05:05:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T05:05:47.344 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-12T05:05:47.344 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-12T05:05:47.344 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-12T05:05:47.345 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph health --format=json 2024-04-12T05:05:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:45.543010+0000 mgr.smithi113.ctsazg (mgr.14194) 179 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:47.611 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:05:48.158 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:46.642201+0000 mgr.smithi113.ctsazg (mgr.14194) 180 : audit [DBG] from='client.14386 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T05:05:48.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:46.642201+0000 mgr.smithi113.ctsazg (mgr.14194) 180 : audit [DBG] from='client.14386 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T05:05:49.210 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:47.543294+0000 mgr.smithi113.ctsazg (mgr.14194) 181 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:49.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:47.543294+0000 mgr.smithi113.ctsazg (mgr.14194) 181 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:49.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 -- 172.21.15.113:0/2502113513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea80f7b40 msgr2=0x7f3ea80f7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:49.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 --2- 172.21.15.113:0/2502113513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea80f7b40 0x7f3ea80f7fa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f3e940047f0 tx=0x7f3e94009cb0).stop 2024-04-12T05:05:49.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 -- 172.21.15.113:0/2502113513 shutdown_connections 2024-04-12T05:05:49.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 --2- 172.21.15.113:0/2502113513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea80f7b40 0x7f3ea80f7fa0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:49.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 --2- 172.21.15.113:0/2502113513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea80f68a0 0x7f3ea80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:49.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 -- 172.21.15.113:0/2502113513 >> 172.21.15.113:0/2502113513 conn(0x7f3ea80f2020 msgr2=0x7f3ea80f4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:49.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 -- 172.21.15.113:0/2502113513 shutdown_connections 2024-04-12T05:05:49.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 -- 172.21.15.113:0/2502113513 wait complete. 2024-04-12T05:05:49.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 Processor -- start 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 -- start start 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea80f68a0 0x7f3ea8108060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea80f7b40 0x7f3ea8108550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8108b00 con 0x7f3ea80f68a0 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.458+0000 7f3eaef02700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8108c40 con 0x7f3ea80f7b40 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3eacc9e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea80f68a0 0x7f3ea8108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3eacc9e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea80f68a0 0x7f3ea8108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60570/0 (socket says 172.21.15.113:60570) 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3eacc9e700 1 -- 172.21.15.113:0/1342777564 learned_addr learned my addr 172.21.15.113:0/1342777564 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3ea7fff700 1 --2- 172.21.15.113:0/1342777564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea80f7b40 0x7f3ea8108550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3eacc9e700 1 -- 172.21.15.113:0/1342777564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea80f7b40 msgr2=0x7f3ea8108550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3eacc9e700 1 --2- 172.21.15.113:0/1342777564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea80f7b40 0x7f3ea8108550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3eacc9e700 1 -- 172.21.15.113:0/1342777564 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e94005040 con 0x7f3ea80f68a0 2024-04-12T05:05:49.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3eacc9e700 1 --2- 172.21.15.113:0/1342777564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea80f68a0 0x7f3ea8108060 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f3e9c002820 tx=0x7f3e9c009f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:49.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3ea5ffb700 1 -- 172.21.15.113:0/1342777564 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e9c002d60 con 0x7f3ea80f68a0 2024-04-12T05:05:49.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3eaef02700 1 -- 172.21.15.113:0/1342777564 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea810d6e0 con 0x7f3ea80f68a0 2024-04-12T05:05:49.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3ea5ffb700 1 -- 172.21.15.113:0/1342777564 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f3e9c00f040 con 0x7f3ea80f68a0 2024-04-12T05:05:49.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3ea5ffb700 1 -- 172.21.15.113:0/1342777564 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e9c00d820 con 0x7f3ea80f68a0 2024-04-12T05:05:49.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.459+0000 7f3eaef02700 1 -- 172.21.15.113:0/1342777564 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea810dc20 con 0x7f3ea80f68a0 2024-04-12T05:05:49.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.460+0000 7f3eaef02700 1 -- 172.21.15.113:0/1342777564 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea80feff0 con 0x7f3ea80f68a0 2024-04-12T05:05:49.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.461+0000 7f3ea5ffb700 1 -- 172.21.15.113:0/1342777564 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f3e9c0092f0 con 0x7f3ea80f68a0 2024-04-12T05:05:49.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.461+0000 7f3ea5ffb700 1 --2- 172.21.15.113:0/1342777564 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3e98061180 0x7f3e98063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:49.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.461+0000 7f3ea5ffb700 1 -- 172.21.15.113:0/1342777564 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f3e9c07c1d0 con 0x7f3ea80f68a0 2024-04-12T05:05:49.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.461+0000 7f3ea7fff700 1 --2- 172.21.15.113:0/1342777564 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3e98061180 0x7f3e98063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:49.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.461+0000 7f3ea7fff700 1 --2- 172.21.15.113:0/1342777564 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3e98061180 0x7f3e98063640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3e94013470 tx=0x7f3e9400b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.465+0000 7f3ea5ffb700 1 -- 172.21.15.113:0/1342777564 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3e9c050550 con 0x7f3ea80f68a0 2024-04-12T05:05:49.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.666+0000 7f3eaef02700 1 -- 172.21.15.113:0/1342777564 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3ea810e320 con 0x7f3ea80f68a0 2024-04-12T05:05:49.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.666+0000 7f3ea5ffb700 1 -- 172.21.15.113:0/1342777564 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f3e9c050140 con 0x7f3ea80f68a0 2024-04-12T05:05:49.669 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:05:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.668+0000 7f3eaef02700 1 -- 172.21.15.113:0/1342777564 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3e98061180 msgr2=0x7f3e98063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.668+0000 7f3eaef02700 1 --2- 172.21.15.113:0/1342777564 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3e98061180 0x7f3e98063640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3e94013470 tx=0x7f3e9400b040).stop 2024-04-12T05:05:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.668+0000 7f3eaef02700 1 -- 172.21.15.113:0/1342777564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea80f68a0 msgr2=0x7f3ea8108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.668+0000 7f3eaef02700 1 --2- 172.21.15.113:0/1342777564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea80f68a0 0x7f3ea8108060 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f3e9c002820 tx=0x7f3e9c009f00).stop 2024-04-12T05:05:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.668+0000 7f3eaef02700 1 -- 172.21.15.113:0/1342777564 shutdown_connections 2024-04-12T05:05:49.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.668+0000 7f3eaef02700 1 --2- 172.21.15.113:0/1342777564 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3e98061180 0x7f3e98063640 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.668+0000 7f3eaef02700 1 --2- 172.21.15.113:0/1342777564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea80f68a0 0x7f3ea8108060 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.668+0000 7f3eaef02700 1 --2- 172.21.15.113:0/1342777564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea80f7b40 0x7f3ea8108550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.668+0000 7f3eaef02700 1 -- 172.21.15.113:0/1342777564 >> 172.21.15.113:0/1342777564 conn(0x7f3ea80f2020 msgr2=0x7f3ea8061310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.668+0000 7f3eaef02700 1 -- 172.21.15.113:0/1342777564 shutdown_connections 2024-04-12T05:05:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:49.668+0000 7f3eaef02700 1 -- 172.21.15.113:0/1342777564 wait complete. 2024-04-12T05:05:50.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:49.667449+0000 mon.smithi113 (mon.0) 499 : audit [DBG] from='client.? 172.21.15.113:0/1342777564' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-12T05:05:50.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:49.667449+0000 mon.smithi113 (mon.0) 499 : audit [DBG] from='client.? 172.21.15.113:0/1342777564' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-12T05:05:50.470 INFO:teuthology.orchestra.run.smithi113.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-12T05:05:50.471 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-12T05:05:50.471 INFO:tasks.cephadm:Setup complete, yielding 2024-04-12T05:05:50.471 INFO:teuthology.run_tasks:Running task print... 2024-04-12T05:05:50.480 INFO:teuthology.task.print:**** done end installing pacific cephadm ... 2024-04-12T05:05:50.480 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:05:50.496 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:05:50.496 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-12T05:05:51.227 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:49.543590+0000 mgr.smithi113.ctsazg (mgr.14194) 182 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:49.543590+0000 mgr.smithi113.ctsazg (mgr.14194) 182 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:52.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.459+0000 7f4acbbe8700 1 -- 172.21.15.113:0/2365267376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac40f66d0 msgr2=0x7f4ac40f6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:52.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.459+0000 7f4acbbe8700 1 --2- 172.21.15.113:0/2365267376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac40f66d0 0x7f4ac40f6af0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4ac00047f0 tx=0x7f4ac0009cb0).stop 2024-04-12T05:05:52.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.459+0000 7f4acbbe8700 1 -- 172.21.15.113:0/2365267376 shutdown_connections 2024-04-12T05:05:52.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.459+0000 7f4acbbe8700 1 --2- 172.21.15.113:0/2365267376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac40f6fe0 0x7f4ac40f7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:52.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.459+0000 7f4acbbe8700 1 --2- 172.21.15.113:0/2365267376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac40f66d0 0x7f4ac40f6af0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:52.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.459+0000 7f4acbbe8700 1 -- 172.21.15.113:0/2365267376 >> 172.21.15.113:0/2365267376 conn(0x7f4ac40f1eb0 msgr2=0x7f4ac40f42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:52.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.459+0000 7f4acbbe8700 1 -- 172.21.15.113:0/2365267376 shutdown_connections 2024-04-12T05:05:52.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.459+0000 7f4acbbe8700 1 -- 172.21.15.113:0/2365267376 wait complete. 2024-04-12T05:05:52.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4acbbe8700 1 Processor -- start 2024-04-12T05:05:52.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4acbbe8700 1 -- start start 2024-04-12T05:05:52.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4acbbe8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac40f66d0 0x7f4ac40615f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:52.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4acbbe8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac40f6fe0 0x7f4ac410cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:52.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4acbbe8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac410d2e0 con 0x7f4ac40f66d0 2024-04-12T05:05:52.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4acbbe8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac410d420 con 0x7f4ac40f6fe0 2024-04-12T05:05:52.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4ac9984700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac40f66d0 0x7f4ac40615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:52.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4ac9984700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac40f66d0 0x7f4ac40615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57446/0 (socket says 172.21.15.113:57446) 2024-04-12T05:05:52.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4ac9984700 1 -- 172.21.15.113:0/399966401 learned_addr learned my addr 172.21.15.113:0/399966401 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:52.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4ac9183700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac40f6fe0 0x7f4ac410cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:52.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4ac9984700 1 -- 172.21.15.113:0/399966401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac40f6fe0 msgr2=0x7f4ac410cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:52.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4ac9984700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac40f6fe0 0x7f4ac410cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:52.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4ac9984700 1 -- 172.21.15.113:0/399966401 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac0005040 con 0x7f4ac40f66d0 2024-04-12T05:05:52.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4ac9183700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac40f6fe0 0x7f4ac410cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:05:52.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4ac9984700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac40f66d0 0x7f4ac40615f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f4ac0005010 tx=0x7f4ac0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:52.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4abaffd700 1 -- 172.21.15.113:0/399966401 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ac000e030 con 0x7f4ac40f66d0 2024-04-12T05:05:52.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4abaffd700 1 -- 172.21.15.113:0/399966401 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f4ac0007e90 con 0x7f4ac40f66d0 2024-04-12T05:05:52.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4abaffd700 1 -- 172.21.15.113:0/399966401 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ac0018070 con 0x7f4ac40f66d0 2024-04-12T05:05:52.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4acbbe8700 1 -- 172.21.15.113:0/399966401 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac410d620 con 0x7f4ac40f66d0 2024-04-12T05:05:52.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.460+0000 7f4acbbe8700 1 -- 172.21.15.113:0/399966401 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac410dad0 con 0x7f4ac40f66d0 2024-04-12T05:05:52.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.462+0000 7f4acbbe8700 1 -- 172.21.15.113:0/399966401 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac40feed0 con 0x7f4ac40f66d0 2024-04-12T05:05:52.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.462+0000 7f4abaffd700 1 -- 172.21.15.113:0/399966401 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f4ac001d420 con 0x7f4ac40f66d0 2024-04-12T05:05:52.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.462+0000 7f4abaffd700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f4ab0061180 0x7f4ab0063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:52.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.462+0000 7f4abaffd700 1 -- 172.21.15.113:0/399966401 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f4ac007da20 con 0x7f4ac40f66d0 2024-04-12T05:05:52.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.462+0000 7f4ac9183700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f4ab0061180 0x7f4ab0063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:52.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.462+0000 7f4ac9183700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f4ab0061180 0x7f4ab0063640 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4ab4007fd0 tx=0x7f4ab4009040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:52.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.466+0000 7f4abaffd700 1 -- 172.21.15.113:0/399966401 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4ac0051d10 con 0x7f4ac40f66d0 2024-04-12T05:05:52.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.628+0000 7f4acbbe8700 1 -- 172.21.15.113:0/399966401 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f4ac4100d40 con 0x7f4ac40f66d0 2024-04-12T05:05:52.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.633+0000 7f4abaffd700 1 -- 172.21.15.113:0/399966401 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f4ac0051900 con 0x7f4ac40f66d0 2024-04-12T05:05:52.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.635+0000 7f4acbbe8700 1 -- 172.21.15.113:0/399966401 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f4ab0061180 msgr2=0x7f4ab0063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:52.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.635+0000 7f4acbbe8700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f4ab0061180 0x7f4ab0063640 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4ab4007fd0 tx=0x7f4ab4009040).stop 2024-04-12T05:05:52.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.635+0000 7f4acbbe8700 1 -- 172.21.15.113:0/399966401 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac40f66d0 msgr2=0x7f4ac40615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:52.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.635+0000 7f4acbbe8700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac40f66d0 0x7f4ac40615f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f4ac0005010 tx=0x7f4ac0009cb0).stop 2024-04-12T05:05:52.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.635+0000 7f4acbbe8700 1 -- 172.21.15.113:0/399966401 shutdown_connections 2024-04-12T05:05:52.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.635+0000 7f4acbbe8700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f4ab0061180 0x7f4ab0063640 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:52.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.635+0000 7f4acbbe8700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac40f66d0 0x7f4ac40615f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:52.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.635+0000 7f4acbbe8700 1 --2- 172.21.15.113:0/399966401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac40f6fe0 0x7f4ac410cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:52.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.635+0000 7f4acbbe8700 1 -- 172.21.15.113:0/399966401 >> 172.21.15.113:0/399966401 conn(0x7f4ac40f1eb0 msgr2=0x7f4ac40fb460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:52.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.635+0000 7f4acbbe8700 1 -- 172.21.15.113:0/399966401 shutdown_connections 2024-04-12T05:05:52.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:52.635+0000 7f4acbbe8700 1 -- 172.21.15.113:0/399966401 wait complete. 2024-04-12T05:05:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:51.543943+0000 mgr.smithi113.ctsazg (mgr.14194) 183 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:52.633927+0000 mon.smithi113 (mon.0) 500 : audit [INF] from='client.? 172.21.15.113:0/399966401' entity='client.admin' 2024-04-12T05:05:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:52.657056+0000 mon.smithi113 (mon.0) 501 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:05:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:52.950997+0000 mon.smithi113 (mon.0) 502 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:05:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:52.951600+0000 mon.smithi113 (mon.0) 503 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:05:53.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:52.955885+0000 mon.smithi113 (mon.0) 504 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:05:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:51.543943+0000 mgr.smithi113.ctsazg (mgr.14194) 183 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:52.633927+0000 mon.smithi113 (mon.0) 500 : audit [INF] from='client.? 172.21.15.113:0/399966401' entity='client.admin' 2024-04-12T05:05:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:52.657056+0000 mon.smithi113 (mon.0) 501 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:05:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:52.950997+0000 mon.smithi113 (mon.0) 502 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:05:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:52.951600+0000 mon.smithi113 (mon.0) 503 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:05:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:52.955885+0000 mon.smithi113 (mon.0) 504 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:05:53.527 INFO:teuthology.run_tasks:Running task print... 2024-04-12T05:05:53.537 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-12T05:05:53.537 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:05:53.559 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:05:53.560 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph orch status' 2024-04-12T05:05:55.258 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:53.544235+0000 mgr.smithi113.ctsazg (mgr.14194) 184 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:55.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:53.544235+0000 mgr.smithi113.ctsazg (mgr.14194) 184 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:55.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.511+0000 7f879c8d0700 1 -- 172.21.15.113:0/1702887922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87980f7990 msgr2=0x7f87980f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:55.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.511+0000 7f879c8d0700 1 --2- 172.21.15.113:0/1702887922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87980f7990 0x7f87980f7df0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f87880047f0 tx=0x7f8788009cb0).stop 2024-04-12T05:05:55.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.511+0000 7f879c8d0700 1 -- 172.21.15.113:0/1702887922 shutdown_connections 2024-04-12T05:05:55.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.511+0000 7f879c8d0700 1 --2- 172.21.15.113:0/1702887922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87980f7990 0x7f87980f7df0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:55.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.511+0000 7f879c8d0700 1 --2- 172.21.15.113:0/1702887922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87980f66f0 0x7f87980f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:55.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.511+0000 7f879c8d0700 1 -- 172.21.15.113:0/1702887922 >> 172.21.15.113:0/1702887922 conn(0x7f87980f1e50 msgr2=0x7f87980f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:55.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.511+0000 7f879c8d0700 1 -- 172.21.15.113:0/1702887922 shutdown_connections 2024-04-12T05:05:55.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.511+0000 7f879c8d0700 1 -- 172.21.15.113:0/1702887922 wait complete. 2024-04-12T05:05:55.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.512+0000 7f879c8d0700 1 Processor -- start 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.512+0000 7f879c8d0700 1 -- start start 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879c8d0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87980f66f0 0x7f8798061010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879c8d0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87980f7990 0x7f879810cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879c8d0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879810d290 con 0x7f87980f66f0 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879c8d0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879810d3d0 con 0x7f87980f7990 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87980f66f0 0x7f8798061010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87980f66f0 0x7f8798061010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57470/0 (socket says 172.21.15.113:57470) 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879659c700 1 -- 172.21.15.113:0/2977198543 learned_addr learned my addr 172.21.15.113:0/2977198543 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f8795d9b700 1 --2- 172.21.15.113:0/2977198543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87980f7990 0x7f879810cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879659c700 1 -- 172.21.15.113:0/2977198543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87980f7990 msgr2=0x7f879810cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879659c700 1 --2- 172.21.15.113:0/2977198543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87980f7990 0x7f879810cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879659c700 1 -- 172.21.15.113:0/2977198543 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8788005040 con 0x7f87980f66f0 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879659c700 1 --2- 172.21.15.113:0/2977198543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87980f66f0 0x7f8798061010 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f8780002780 tx=0x7f8780009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:55.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f878f7fe700 1 -- 172.21.15.113:0/2977198543 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8780007b40 con 0x7f87980f66f0 2024-04-12T05:05:55.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f878f7fe700 1 -- 172.21.15.113:0/2977198543 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f878000f040 con 0x7f87980f66f0 2024-04-12T05:05:55.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f878f7fe700 1 -- 172.21.15.113:0/2977198543 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87800146f0 con 0x7f87980f66f0 2024-04-12T05:05:55.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879c8d0700 1 -- 172.21.15.113:0/2977198543 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879810d5d0 con 0x7f87980f66f0 2024-04-12T05:05:55.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.513+0000 7f879c8d0700 1 -- 172.21.15.113:0/2977198543 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879810db10 con 0x7f87980f66f0 2024-04-12T05:05:55.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.514+0000 7f879c8d0700 1 -- 172.21.15.113:0/2977198543 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87980feed0 con 0x7f87980f66f0 2024-04-12T05:05:55.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.515+0000 7f878f7fe700 1 -- 172.21.15.113:0/2977198543 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f8780007ca0 con 0x7f87980f66f0 2024-04-12T05:05:55.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.515+0000 7f878f7fe700 1 --2- 172.21.15.113:0/2977198543 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8784061180 0x7f8784063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:55.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.515+0000 7f878f7fe700 1 -- 172.21.15.113:0/2977198543 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f878007c420 con 0x7f87980f66f0 2024-04-12T05:05:55.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.515+0000 7f8795d9b700 1 --2- 172.21.15.113:0/2977198543 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8784061180 0x7f8784063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:55.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.515+0000 7f8795d9b700 1 --2- 172.21.15.113:0/2977198543 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8784061180 0x7f8784063640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8788013470 tx=0x7f878800b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:55.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.519+0000 7f878f7fe700 1 -- 172.21.15.113:0/2977198543 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8780050770 con 0x7f87980f66f0 2024-04-12T05:05:55.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.694+0000 7f879c8d0700 1 -- 172.21.15.113:0/2977198543 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8798059120 con 0x7f8784061180 2024-04-12T05:05:55.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.695+0000 7f878f7fe700 1 -- 172.21.15.113:0/2977198543 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8798059120 con 0x7f8784061180 2024-04-12T05:05:55.697 INFO:teuthology.orchestra.run.smithi113.stdout:Backend: cephadm 2024-04-12T05:05:55.697 INFO:teuthology.orchestra.run.smithi113.stdout:Available: Yes 2024-04-12T05:05:55.697 INFO:teuthology.orchestra.run.smithi113.stdout:Paused: No 2024-04-12T05:05:55.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.697+0000 7f879c8d0700 1 -- 172.21.15.113:0/2977198543 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8784061180 msgr2=0x7f8784063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:55.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.697+0000 7f879c8d0700 1 --2- 172.21.15.113:0/2977198543 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8784061180 0x7f8784063640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8788013470 tx=0x7f878800b040).stop 2024-04-12T05:05:55.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.697+0000 7f879c8d0700 1 -- 172.21.15.113:0/2977198543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87980f66f0 msgr2=0x7f8798061010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:55.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.697+0000 7f879c8d0700 1 --2- 172.21.15.113:0/2977198543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87980f66f0 0x7f8798061010 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f8780002780 tx=0x7f8780009ee0).stop 2024-04-12T05:05:55.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.697+0000 7f879c8d0700 1 -- 172.21.15.113:0/2977198543 shutdown_connections 2024-04-12T05:05:55.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.697+0000 7f879c8d0700 1 --2- 172.21.15.113:0/2977198543 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8784061180 0x7f8784063640 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:55.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.697+0000 7f879c8d0700 1 --2- 172.21.15.113:0/2977198543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87980f66f0 0x7f8798061010 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:55.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.697+0000 7f879c8d0700 1 --2- 172.21.15.113:0/2977198543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87980f7990 0x7f879810cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:55.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.697+0000 7f879c8d0700 1 -- 172.21.15.113:0/2977198543 >> 172.21.15.113:0/2977198543 conn(0x7f87980f1e50 msgr2=0x7f87980fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:55.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.697+0000 7f879c8d0700 1 -- 172.21.15.113:0/2977198543 shutdown_connections 2024-04-12T05:05:55.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:55.697+0000 7f879c8d0700 1 -- 172.21.15.113:0/2977198543 wait complete. 2024-04-12T05:05:56.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-12T05:05:57.228 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:55.544517+0000 mgr.smithi113.ctsazg (mgr.14194) 185 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:57.228 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:55.696271+0000 mgr.smithi113.ctsazg (mgr.14194) 186 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:05:57.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:55.544517+0000 mgr.smithi113.ctsazg (mgr.14194) 185 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:57.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:55.696271+0000 mgr.smithi113.ctsazg (mgr.14194) 186 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:05:58.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.646+0000 7f8468e91700 1 -- 172.21.15.113:0/3016988112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84640f66f0 msgr2=0x7f84640f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:58.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.646+0000 7f8468e91700 1 --2- 172.21.15.113:0/3016988112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84640f66f0 0x7f84640f6b10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f84540047f0 tx=0x7f8454009cb0).stop 2024-04-12T05:05:58.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.646+0000 7f8468e91700 1 -- 172.21.15.113:0/3016988112 shutdown_connections 2024-04-12T05:05:58.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.646+0000 7f8468e91700 1 --2- 172.21.15.113:0/3016988112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84640f7990 0x7f84640f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:58.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.646+0000 7f8468e91700 1 --2- 172.21.15.113:0/3016988112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84640f66f0 0x7f84640f6b10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:58.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.646+0000 7f8468e91700 1 -- 172.21.15.113:0/3016988112 >> 172.21.15.113:0/3016988112 conn(0x7f84640f1e50 msgr2=0x7f84640f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:58.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.646+0000 7f8468e91700 1 -- 172.21.15.113:0/3016988112 shutdown_connections 2024-04-12T05:05:58.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.646+0000 7f8468e91700 1 -- 172.21.15.113:0/3016988112 wait complete. 2024-04-12T05:05:58.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.647+0000 7f8468e91700 1 Processor -- start 2024-04-12T05:05:58.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.647+0000 7f8468e91700 1 -- start start 2024-04-12T05:05:58.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.647+0000 7f8468e91700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84640f66f0 0x7f8464106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:58.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.647+0000 7f8468e91700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84640f7990 0x7f84641073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:58.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.647+0000 7f8468e91700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8464107950 con 0x7f84640f66f0 2024-04-12T05:05:58.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.647+0000 7f8468e91700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8464107a90 con 0x7f84640f7990 2024-04-12T05:05:58.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.647+0000 7f846259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84640f66f0 0x7f8464106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:58.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.647+0000 7f846259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84640f66f0 0x7f8464106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57488/0 (socket says 172.21.15.113:57488) 2024-04-12T05:05:58.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.647+0000 7f846259c700 1 -- 172.21.15.113:0/2867355194 learned_addr learned my addr 172.21.15.113:0/2867355194 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:05:58.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.648+0000 7f846259c700 1 -- 172.21.15.113:0/2867355194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84640f7990 msgr2=0x7f84641073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:58.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.648+0000 7f846259c700 1 --2- 172.21.15.113:0/2867355194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84640f7990 0x7f84641073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:05:58.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.648+0000 7f846259c700 1 -- 172.21.15.113:0/2867355194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8454005040 con 0x7f84640f66f0 2024-04-12T05:05:58.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.648+0000 7f846259c700 1 --2- 172.21.15.113:0/2867355194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84640f66f0 0x7f8464106eb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f8454004f10 tx=0x7f8454009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:58.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.648+0000 7f84537fe700 1 -- 172.21.15.113:0/2867355194 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f845400e070 con 0x7f84640f66f0 2024-04-12T05:05:58.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.648+0000 7f8468e91700 1 -- 172.21.15.113:0/2867355194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846410c4d0 con 0x7f84640f66f0 2024-04-12T05:05:58.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.648+0000 7f84537fe700 1 -- 172.21.15.113:0/2867355194 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f8454007e90 con 0x7f84640f66f0 2024-04-12T05:05:58.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.648+0000 7f84537fe700 1 -- 172.21.15.113:0/2867355194 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8454018070 con 0x7f84640f66f0 2024-04-12T05:05:58.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.648+0000 7f8468e91700 1 -- 172.21.15.113:0/2867355194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846410ca10 con 0x7f84640f66f0 2024-04-12T05:05:58.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.649+0000 7f8468e91700 1 -- 172.21.15.113:0/2867355194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84640feed0 con 0x7f84640f66f0 2024-04-12T05:05:58.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.649+0000 7f84537fe700 1 -- 172.21.15.113:0/2867355194 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f84540123c0 con 0x7f84640f66f0 2024-04-12T05:05:58.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.650+0000 7f84537fe700 1 --2- 172.21.15.113:0/2867355194 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f844c061180 0x7f844c063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:05:58.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.650+0000 7f84537fe700 1 -- 172.21.15.113:0/2867355194 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f845407d430 con 0x7f84640f66f0 2024-04-12T05:05:58.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.650+0000 7f8461d9b700 1 --2- 172.21.15.113:0/2867355194 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f844c061180 0x7f844c063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:05:58.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.650+0000 7f8461d9b700 1 --2- 172.21.15.113:0/2867355194 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f844c061180 0x7f844c063640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f8458004b90 tx=0x7f8458009b60).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:05:58.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.654+0000 7f84537fe700 1 -- 172.21.15.113:0/2867355194 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8454051700 con 0x7f84640f66f0 2024-04-12T05:05:58.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.826+0000 7f8468e91700 1 -- 172.21.15.113:0/2867355194 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8464059120 con 0x7f844c061180 2024-04-12T05:05:58.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.834+0000 7f84537fe700 1 -- 172.21.15.113:0/2867355194 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2700 (secure 0 0 0) 0x7f8464059120 con 0x7f844c061180 2024-04-12T05:05:58.836 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:05:58.836 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (3m) 2m ago 5m 15.4M - 0.23.0 ba2b418f427c aae81b8048e6 2024-04-12T05:05:58.836 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (4m) 2m ago 4m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:05:58.836 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (4m) 69s ago 4m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:05:58.836 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (3m) 2m ago 4m 48.4M - 8.3.5 dad864ee21e9 7d83fd029dea 2024-04-12T05:05:58.836 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:9283 running (6m) 2m ago 6m 425M - 16.2.15-1-g8963e9de 727f386679cb 3fab7f291e6b 2024-04-12T05:05:58.836 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283 running (4m) 69s ago 4m 386M - 16.2.15-1-g8963e9de 727f386679cb 58d65802163f 2024-04-12T05:05:58.836 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (6m) 2m ago 6m 46.7M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:05:58.836 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (4m) 69s ago 4m 50.7M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:05:58.836 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (4m) 2m ago 4m 21.1M - 1.3.1 1dbe0e931976 046bcfb16579 2024-04-12T05:05:58.837 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (4m) 69s ago 4m 26.0M - 1.3.1 1dbe0e931976 bc604318825c 2024-04-12T05:05:58.837 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (3m) 2m ago 3m 32.5M 4640M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:05:58.837 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (3m) 2m ago 3m 33.4M 4640M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:05:58.837 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (2m) 2m ago 2m 12.8M 4640M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:05:58.837 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (2m) 69s ago 2m 39.8M 4982M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:05:58.837 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (102s) 69s ago 101s 34.6M 4982M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:05:58.837 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (75s) 69s ago 74s 12.8M 4982M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:05:58.837 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (3m) 69s ago 3m 56.2M - 2.33.4 514e6a882f6e b4259537bd55 2024-04-12T05:05:58.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.836+0000 7f8468e91700 1 -- 172.21.15.113:0/2867355194 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f844c061180 msgr2=0x7f844c063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:58.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.836+0000 7f8468e91700 1 --2- 172.21.15.113:0/2867355194 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f844c061180 0x7f844c063640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f8458004b90 tx=0x7f8458009b60).stop 2024-04-12T05:05:58.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.836+0000 7f8468e91700 1 -- 172.21.15.113:0/2867355194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84640f66f0 msgr2=0x7f8464106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:05:58.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.836+0000 7f8468e91700 1 --2- 172.21.15.113:0/2867355194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84640f66f0 0x7f8464106eb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f8454004f10 tx=0x7f8454009cb0).stop 2024-04-12T05:05:58.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.836+0000 7f8468e91700 1 -- 172.21.15.113:0/2867355194 shutdown_connections 2024-04-12T05:05:58.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.836+0000 7f8468e91700 1 --2- 172.21.15.113:0/2867355194 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f844c061180 0x7f844c063640 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:58.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.836+0000 7f8468e91700 1 --2- 172.21.15.113:0/2867355194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84640f66f0 0x7f8464106eb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:05:58.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.836+0000 7f8468e91700 1 --2- 172.21.15.113:0/2867355194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84640f7990 0x7f84641073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:05:58.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.836+0000 7f8468e91700 1 -- 172.21.15.113:0/2867355194 >> 172.21.15.113:0/2867355194 conn(0x7f84640f1e50 msgr2=0x7f84640fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:05:58.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.836+0000 7f8468e91700 1 -- 172.21.15.113:0/2867355194 shutdown_connections 2024-04-12T05:05:58.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:05:58.836+0000 7f8468e91700 1 -- 172.21.15.113:0/2867355194 wait complete. 2024-04-12T05:05:59.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:05:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:57.544973+0000 mgr.smithi113.ctsazg (mgr.14194) 187 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:05:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:57.544973+0000 mgr.smithi113.ctsazg (mgr.14194) 187 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:05:59.652 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-12T05:06:00.400 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:05:58.828061+0000 mgr.smithi113.ctsazg (mgr.14194) 188 : audit [DBG] from='client.14402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:06:00.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:05:58.828061+0000 mgr.smithi113.ctsazg (mgr.14194) 188 : audit [DBG] from='client.14402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:06:01.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:05:59.545445+0000 mgr.smithi113.ctsazg (mgr.14194) 189 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:05:59.545445+0000 mgr.smithi113.ctsazg (mgr.14194) 189 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:01.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.883+0000 7f3b61f49700 1 -- 172.21.15.113:0/1688734964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b5c0f8cb0 msgr2=0x7f3b5c0f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:01.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.883+0000 7f3b61f49700 1 --2- 172.21.15.113:0/1688734964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b5c0f8cb0 0x7f3b5c0f9110 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f3b500047f0 tx=0x7f3b50009cb0).stop 2024-04-12T05:06:01.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.883+0000 7f3b61f49700 1 -- 172.21.15.113:0/1688734964 shutdown_connections 2024-04-12T05:06:01.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.883+0000 7f3b61f49700 1 --2- 172.21.15.113:0/1688734964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b5c0f8cb0 0x7f3b5c0f9110 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:01.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.883+0000 7f3b61f49700 1 --2- 172.21.15.113:0/1688734964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b5c0f7a10 0x7f3b5c0f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:06:01.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.883+0000 7f3b61f49700 1 -- 172.21.15.113:0/1688734964 >> 172.21.15.113:0/1688734964 conn(0x7f3b5c0f3150 msgr2=0x7f3b5c0f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:01.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.883+0000 7f3b61f49700 1 -- 172.21.15.113:0/1688734964 shutdown_connections 2024-04-12T05:06:01.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.883+0000 7f3b61f49700 1 -- 172.21.15.113:0/1688734964 wait complete. 2024-04-12T05:06:01.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.884+0000 7f3b61f49700 1 Processor -- start 2024-04-12T05:06:01.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.884+0000 7f3b61f49700 1 -- start start 2024-04-12T05:06:01.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b61f49700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b5c0f7a10 0x7f3b5c108210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:01.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b61f49700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b5c0f8cb0 0x7f3b5c108700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:01.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b61f49700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c108d40 con 0x7f3b5c0f7a10 2024-04-12T05:06:01.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b61f49700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c108e80 con 0x7f3b5c0f8cb0 2024-04-12T05:06:01.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b5b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b5c0f7a10 0x7f3b5c108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b5b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b5c0f7a10 0x7f3b5c108210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44398/0 (socket says 172.21.15.113:44398) 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b5b7fe700 1 -- 172.21.15.113:0/2140532947 learned_addr learned my addr 172.21.15.113:0/2140532947 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b5affd700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b5c0f8cb0 0x7f3b5c108700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b5b7fe700 1 -- 172.21.15.113:0/2140532947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b5c0f8cb0 msgr2=0x7f3b5c108700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b5b7fe700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b5c0f8cb0 0x7f3b5c108700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b5b7fe700 1 -- 172.21.15.113:0/2140532947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b50005040 con 0x7f3b5c0f7a10 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b5affd700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b5c0f8cb0 0x7f3b5c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b5b7fe700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b5c0f7a10 0x7f3b5c108210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f3b4c00b480 tx=0x7f3b4c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b58ff9700 1 -- 172.21.15.113:0/2140532947 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b4c002500 con 0x7f3b5c0f7a10 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b58ff9700 1 -- 172.21.15.113:0/2140532947 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f3b4c002af0 con 0x7f3b5c0f7a10 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b61f49700 1 -- 172.21.15.113:0/2140532947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5c10d920 con 0x7f3b5c0f7a10 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b58ff9700 1 -- 172.21.15.113:0/2140532947 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b4c012c40 con 0x7f3b5c0f7a10 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.885+0000 7f3b61f49700 1 -- 172.21.15.113:0/2140532947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c10de60 con 0x7f3b5c0f7a10 2024-04-12T05:06:01.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.886+0000 7f3b61f49700 1 -- 172.21.15.113:0/2140532947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5c05fb50 con 0x7f3b5c0f7a10 2024-04-12T05:06:01.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.887+0000 7f3b58ff9700 1 -- 172.21.15.113:0/2140532947 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f3b4c002660 con 0x7f3b5c0f7a10 2024-04-12T05:06:01.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.887+0000 7f3b58ff9700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3b44061180 0x7f3b44063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:01.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.887+0000 7f3b58ff9700 1 -- 172.21.15.113:0/2140532947 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f3b4c07c600 con 0x7f3b5c0f7a10 2024-04-12T05:06:01.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.887+0000 7f3b5affd700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3b44061180 0x7f3b44063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:01.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.887+0000 7f3b5affd700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3b44061180 0x7f3b44063640 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f3b50007cf0 tx=0x7f3b5000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:01.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:01.891+0000 7f3b58ff9700 1 -- 172.21.15.113:0/2140532947 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3b4c0508d0 con 0x7f3b5c0f7a10 2024-04-12T05:06:02.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.060+0000 7f3b61f49700 1 -- 172.21.15.113:0/2140532947 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b5c102260 con 0x7f3b44061180 2024-04-12T05:06:02.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.064+0000 7f3b58ff9700 1 -- 172.21.15.113:0/2140532947 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1179 (secure 0 0 0) 0x7f3b5c102260 con 0x7f3b44061180 2024-04-12T05:06:02.066 INFO:teuthology.orchestra.run.smithi113.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-12T05:06:02.066 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-04-12T05:06:02.066 INFO:teuthology.orchestra.run.smithi113.stdout:crash 2/2 2m ago 5m * 2024-04-12T05:06:02.066 INFO:teuthology.orchestra.run.smithi113.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-04-12T05:06:02.066 INFO:teuthology.orchestra.run.smithi113.stdout:mgr 2/2 2m ago 5m count:2 2024-04-12T05:06:02.066 INFO:teuthology.orchestra.run.smithi113.stdout:mon 2/2 2m ago 4m smithi113:172.21.15.113=smithi113;smithi192:172.21.15.192=smithi192;count:2 2024-04-12T05:06:02.067 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-04-12T05:06:02.067 INFO:teuthology.orchestra.run.smithi113.stdout:osd 6 2m ago - 2024-04-12T05:06:02.067 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus ?:9095 1/1 72s ago 5m count:1 2024-04-12T05:06:02.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.066+0000 7f3b61f49700 1 -- 172.21.15.113:0/2140532947 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3b44061180 msgr2=0x7f3b44063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:02.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.066+0000 7f3b61f49700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3b44061180 0x7f3b44063640 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f3b50007cf0 tx=0x7f3b5000b040).stop 2024-04-12T05:06:02.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.066+0000 7f3b61f49700 1 -- 172.21.15.113:0/2140532947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b5c0f7a10 msgr2=0x7f3b5c108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:02.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.066+0000 7f3b61f49700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b5c0f7a10 0x7f3b5c108210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f3b4c00b480 tx=0x7f3b4c0090f0).stop 2024-04-12T05:06:02.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.066+0000 7f3b61f49700 1 -- 172.21.15.113:0/2140532947 shutdown_connections 2024-04-12T05:06:02.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.066+0000 7f3b61f49700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3b44061180 0x7f3b44063640 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:02.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.066+0000 7f3b61f49700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b5c0f7a10 0x7f3b5c108210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:02.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.066+0000 7f3b61f49700 1 --2- 172.21.15.113:0/2140532947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b5c0f8cb0 0x7f3b5c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:02.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.066+0000 7f3b61f49700 1 -- 172.21.15.113:0/2140532947 >> 172.21.15.113:0/2140532947 conn(0x7f3b5c0f3150 msgr2=0x7f3b5c0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:02.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.066+0000 7f3b61f49700 1 -- 172.21.15.113:0/2140532947 shutdown_connections 2024-04-12T05:06:02.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:02.066+0000 7f3b61f49700 1 -- 172.21.15.113:0/2140532947 wait complete. 2024-04-12T05:06:02.774 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-12T05:06:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:01.545980+0000 mgr.smithi113.ctsazg (mgr.14194) 190 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:02.061920+0000 mgr.smithi113.ctsazg (mgr.14194) 191 : audit [DBG] from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:06:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:01.545980+0000 mgr.smithi113.ctsazg (mgr.14194) 190 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:02.061920+0000 mgr.smithi113.ctsazg (mgr.14194) 191 : audit [DBG] from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:06:04.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.805+0000 7f8eebea0700 1 -- 172.21.15.113:0/960534310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee40f7000 msgr2=0x7f8ee40f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:04.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.805+0000 7f8eebea0700 1 --2- 172.21.15.113:0/960534310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee40f7000 0x7f8ee40f7a30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f8ee00047f0 tx=0x7f8ee0009cb0).stop 2024-04-12T05:06:04.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.805+0000 7f8eebea0700 1 -- 172.21.15.113:0/960534310 shutdown_connections 2024-04-12T05:06:04.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.805+0000 7f8eebea0700 1 --2- 172.21.15.113:0/960534310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee40f7000 0x7f8ee40f7a30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:04.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.805+0000 7f8eebea0700 1 --2- 172.21.15.113:0/960534310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee40f66f0 0x7f8ee40f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:04.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.805+0000 7f8eebea0700 1 -- 172.21.15.113:0/960534310 >> 172.21.15.113:0/960534310 conn(0x7f8ee40f1ef0 msgr2=0x7f8ee40f4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:04.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.805+0000 7f8eebea0700 1 -- 172.21.15.113:0/960534310 shutdown_connections 2024-04-12T05:06:04.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.805+0000 7f8eebea0700 1 -- 172.21.15.113:0/960534310 wait complete. 2024-04-12T05:06:04.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.806+0000 7f8eebea0700 1 Processor -- start 2024-04-12T05:06:04.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.806+0000 7f8eebea0700 1 -- start start 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.806+0000 7f8eebea0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee40f66f0 0x7f8ee40615f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.806+0000 7f8eebea0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee40f7000 0x7f8ee410cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.806+0000 7f8eebea0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee410d290 con 0x7f8ee40f7000 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.806+0000 7f8eebea0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee410d3d0 con 0x7f8ee40f66f0 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.806+0000 7f8ee943b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee40f7000 0x7f8ee410cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.806+0000 7f8ee943b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee40f7000 0x7f8ee410cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44410/0 (socket says 172.21.15.113:44410) 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.806+0000 7f8ee943b700 1 -- 172.21.15.113:0/3264428476 learned_addr learned my addr 172.21.15.113:0/3264428476 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.806+0000 7f8ee9c3c700 1 --2- 172.21.15.113:0/3264428476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee40f66f0 0x7f8ee40615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.807+0000 7f8ee943b700 1 -- 172.21.15.113:0/3264428476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee40f66f0 msgr2=0x7f8ee40615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.807+0000 7f8ee943b700 1 --2- 172.21.15.113:0/3264428476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee40f66f0 0x7f8ee40615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.807+0000 7f8ee943b700 1 -- 172.21.15.113:0/3264428476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee0005040 con 0x7f8ee40f7000 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.807+0000 7f8ee943b700 1 --2- 172.21.15.113:0/3264428476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee40f7000 0x7f8ee410cda0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f8ee0009380 tx=0x7f8ee0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:04.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.807+0000 7f8edaffd700 1 -- 172.21.15.113:0/3264428476 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ee000e030 con 0x7f8ee40f7000 2024-04-12T05:06:04.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.807+0000 7f8edaffd700 1 -- 172.21.15.113:0/3264428476 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f8ee0007e90 con 0x7f8ee40f7000 2024-04-12T05:06:04.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.807+0000 7f8edaffd700 1 -- 172.21.15.113:0/3264428476 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ee0018070 con 0x7f8ee40f7000 2024-04-12T05:06:04.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.807+0000 7f8eebea0700 1 -- 172.21.15.113:0/3264428476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee410d5d0 con 0x7f8ee40f7000 2024-04-12T05:06:04.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.807+0000 7f8eebea0700 1 -- 172.21.15.113:0/3264428476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee410dab0 con 0x7f8ee40f7000 2024-04-12T05:06:04.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.808+0000 7f8eebea0700 1 -- 172.21.15.113:0/3264428476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee40feef0 con 0x7f8ee40f7000 2024-04-12T05:06:04.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.808+0000 7f8edaffd700 1 -- 172.21.15.113:0/3264428476 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f8ee001d420 con 0x7f8ee40f7000 2024-04-12T05:06:04.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.809+0000 7f8edaffd700 1 --2- 172.21.15.113:0/3264428476 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8ed00611d0 0x7f8ed0063690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:04.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.809+0000 7f8edaffd700 1 -- 172.21.15.113:0/3264428476 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f8ee007dac0 con 0x7f8ee40f7000 2024-04-12T05:06:04.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.809+0000 7f8ee9c3c700 1 --2- 172.21.15.113:0/3264428476 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8ed00611d0 0x7f8ed0063690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:04.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.809+0000 7f8ee9c3c700 1 --2- 172.21.15.113:0/3264428476 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8ed00611d0 0x7f8ed0063690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8ed4004d20 tx=0x7f8ed4009040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:04.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.813+0000 7f8edaffd700 1 -- 172.21.15.113:0/3264428476 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8ee007fe60 con 0x7f8ee40f7000 2024-04-12T05:06:04.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.986+0000 7f8eebea0700 1 -- 172.21.15.113:0/3264428476 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ee4059120 con 0x7f8ed00611d0 2024-04-12T05:06:04.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.987+0000 7f8edaffd700 1 -- 172.21.15.113:0/3264428476 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f8ee4059120 con 0x7f8ed00611d0 2024-04-12T05:06:04.989 INFO:teuthology.orchestra.run.smithi113.stdout:HOST ADDR LABELS STATUS 2024-04-12T05:06:04.989 INFO:teuthology.orchestra.run.smithi113.stdout:smithi113 172.21.15.113 2024-04-12T05:06:04.989 INFO:teuthology.orchestra.run.smithi113.stdout:smithi192 172.21.15.192 2024-04-12T05:06:04.989 INFO:teuthology.orchestra.run.smithi113.stdout:2 hosts in cluster 2024-04-12T05:06:04.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.989+0000 7f8eebea0700 1 -- 172.21.15.113:0/3264428476 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8ed00611d0 msgr2=0x7f8ed0063690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:04.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.989+0000 7f8eebea0700 1 --2- 172.21.15.113:0/3264428476 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8ed00611d0 0x7f8ed0063690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f8ed4004d20 tx=0x7f8ed4009040).stop 2024-04-12T05:06:04.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.989+0000 7f8eebea0700 1 -- 172.21.15.113:0/3264428476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee40f7000 msgr2=0x7f8ee410cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:04.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.989+0000 7f8eebea0700 1 --2- 172.21.15.113:0/3264428476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee40f7000 0x7f8ee410cda0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f8ee0009380 tx=0x7f8ee0009cb0).stop 2024-04-12T05:06:04.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.989+0000 7f8eebea0700 1 -- 172.21.15.113:0/3264428476 shutdown_connections 2024-04-12T05:06:04.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.989+0000 7f8eebea0700 1 --2- 172.21.15.113:0/3264428476 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8ed00611d0 0x7f8ed0063690 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:04.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.989+0000 7f8eebea0700 1 --2- 172.21.15.113:0/3264428476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee40f66f0 0x7f8ee40615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:04.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.989+0000 7f8eebea0700 1 --2- 172.21.15.113:0/3264428476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ee40f7000 0x7f8ee410cda0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:04.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.989+0000 7f8eebea0700 1 -- 172.21.15.113:0/3264428476 >> 172.21.15.113:0/3264428476 conn(0x7f8ee40f1ef0 msgr2=0x7f8ee40fb480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:04.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.989+0000 7f8eebea0700 1 -- 172.21.15.113:0/3264428476 shutdown_connections 2024-04-12T05:06:04.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:04.989+0000 7f8eebea0700 1 -- 172.21.15.113:0/3264428476 wait complete. 2024-04-12T05:06:05.375 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:03.546617+0000 mgr.smithi113.ctsazg (mgr.14194) 192 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:03.546617+0000 mgr.smithi113.ctsazg (mgr.14194) 192 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:05.717 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-12T05:06:06.411 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:04.987728+0000 mgr.smithi113.ctsazg (mgr.14194) 193 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:06:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:04.987728+0000 mgr.smithi113.ctsazg (mgr.14194) 193 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:06:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:05.546944+0000 mgr.smithi113.ctsazg (mgr.14194) 194 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:05.546944+0000 mgr.smithi113.ctsazg (mgr.14194) 194 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.792+0000 7f1d9218e700 1 -- 172.21.15.113:0/3746697742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 msgr2=0x7f1d8c0f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.792+0000 7f1d9218e700 1 --2- 172.21.15.113:0/3746697742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c0f9110 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f1d7c0047f0 tx=0x7f1d7c009cb0).stop 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.792+0000 7f1d9218e700 1 -- 172.21.15.113:0/3746697742 shutdown_connections 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.792+0000 7f1d9218e700 1 --2- 172.21.15.113:0/3746697742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c0f9110 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.792+0000 7f1d9218e700 1 --2- 172.21.15.113:0/3746697742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c0f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.792+0000 7f1d9218e700 1 -- 172.21.15.113:0/3746697742 >> 172.21.15.113:0/3746697742 conn(0x7f1d8c0f3150 msgr2=0x7f1d8c0f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.792+0000 7f1d9218e700 1 -- 172.21.15.113:0/3746697742 shutdown_connections 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.792+0000 7f1d9218e700 1 -- 172.21.15.113:0/3746697742 wait complete. 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.792+0000 7f1d9218e700 1 Processor -- start 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d9218e700 1 -- start start 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d9218e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c108190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d9218e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c108680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:07.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d9218e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c108c30 con 0x7f1d8c0f8cb0 2024-04-12T05:06:07.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d9218e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c108d70 con 0x7f1d8c0f7a10 2024-04-12T05:06:07.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d8affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c108680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:07.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d8affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c108680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44422/0 (socket says 172.21.15.113:44422) 2024-04-12T05:06:07.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d8affd700 1 -- 172.21.15.113:0/2528090011 learned_addr learned my addr 172.21.15.113:0/2528090011 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:07.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d8b7fe700 1 --2- 172.21.15.113:0/2528090011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c108190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:07.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d8affd700 1 -- 172.21.15.113:0/2528090011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c0f7a10 msgr2=0x7f1d8c108190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:07.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d8affd700 1 --2- 172.21.15.113:0/2528090011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c108190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:07.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d8affd700 1 -- 172.21.15.113:0/2528090011 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c005040 con 0x7f1d8c0f8cb0 2024-04-12T05:06:07.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d8affd700 1 --2- 172.21.15.113:0/2528090011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c108680 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f1d7c013470 tx=0x7f1d7c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:07.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d88ff9700 1 -- 172.21.15.113:0/2528090011 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d7c00e030 con 0x7f1d8c0f8cb0 2024-04-12T05:06:07.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d88ff9700 1 -- 172.21.15.113:0/2528090011 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f1d7c007e90 con 0x7f1d8c0f8cb0 2024-04-12T05:06:07.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d9218e700 1 -- 172.21.15.113:0/2528090011 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8c10d7b0 con 0x7f1d8c0f8cb0 2024-04-12T05:06:07.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.793+0000 7f1d88ff9700 1 -- 172.21.15.113:0/2528090011 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d7c018070 con 0x7f1d8c0f8cb0 2024-04-12T05:06:07.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.794+0000 7f1d9218e700 1 -- 172.21.15.113:0/2528090011 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8c10dcf0 con 0x7f1d8c0f8cb0 2024-04-12T05:06:07.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.794+0000 7f1d9218e700 1 -- 172.21.15.113:0/2528090011 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d8c05fb50 con 0x7f1d8c0f8cb0 2024-04-12T05:06:07.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.795+0000 7f1d88ff9700 1 -- 172.21.15.113:0/2528090011 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f1d7c01d420 con 0x7f1d8c0f8cb0 2024-04-12T05:06:07.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.796+0000 7f1d88ff9700 1 --2- 172.21.15.113:0/2528090011 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f1d74061110 0x7f1d740635d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:07.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.796+0000 7f1d88ff9700 1 -- 172.21.15.113:0/2528090011 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f1d7c07df20 con 0x7f1d8c0f8cb0 2024-04-12T05:06:07.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.796+0000 7f1d8b7fe700 1 --2- 172.21.15.113:0/2528090011 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f1d74061110 0x7f1d740635d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:07.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.796+0000 7f1d8b7fe700 1 --2- 172.21.15.113:0/2528090011 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f1d74061110 0x7f1d740635d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f1d8c0f8b10 tx=0x7f1d8000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:07.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.799+0000 7f1d88ff9700 1 -- 172.21.15.113:0/2528090011 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1d7c052270 con 0x7f1d8c0f8cb0 2024-04-12T05:06:07.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.970+0000 7f1d9218e700 1 -- 172.21.15.113:0/2528090011 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d8c102230 con 0x7f1d74061110 2024-04-12T05:06:07.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.971+0000 7f1d88ff9700 1 -- 172.21.15.113:0/2528090011 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f1d8000f040 con 0x7f1d74061110 2024-04-12T05:06:07.974 INFO:teuthology.orchestra.run.smithi113.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-12T05:06:07.975 INFO:teuthology.orchestra.run.smithi113.stdout:smithi113 /dev/nvme0n1 ssd INTEL SSDPEDMX020T4_CVPD5171000M2P0SGN 1863G 2m ago Has a FileSystem, LVM detected 2024-04-12T05:06:07.975 INFO:teuthology.orchestra.run.smithi113.stdout:smithi192 /dev/nvme0n1 ssd INTEL SSDPEDMD400G4_PHFT620500NN400BGN 372G 74s ago Has a FileSystem, LVM detected 2024-04-12T05:06:07.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.973+0000 7f1d9218e700 1 -- 172.21.15.113:0/2528090011 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f1d74061110 msgr2=0x7f1d740635d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:07.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.973+0000 7f1d9218e700 1 --2- 172.21.15.113:0/2528090011 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f1d74061110 0x7f1d740635d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f1d8c0f8b10 tx=0x7f1d8000b040).stop 2024-04-12T05:06:07.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.973+0000 7f1d9218e700 1 -- 172.21.15.113:0/2528090011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 msgr2=0x7f1d8c108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:07.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.973+0000 7f1d9218e700 1 --2- 172.21.15.113:0/2528090011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c108680 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f1d7c013470 tx=0x7f1d7c009cb0).stop 2024-04-12T05:06:07.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.973+0000 7f1d9218e700 1 -- 172.21.15.113:0/2528090011 shutdown_connections 2024-04-12T05:06:07.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.973+0000 7f1d9218e700 1 --2- 172.21.15.113:0/2528090011 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f1d74061110 0x7f1d740635d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:07.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.973+0000 7f1d9218e700 1 --2- 172.21.15.113:0/2528090011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:07.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.973+0000 7f1d9218e700 1 --2- 172.21.15.113:0/2528090011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c108680 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:07.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.973+0000 7f1d9218e700 1 -- 172.21.15.113:0/2528090011 >> 172.21.15.113:0/2528090011 conn(0x7f1d8c0f3150 msgr2=0x7f1d8c0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:07.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.973+0000 7f1d9218e700 1 -- 172.21.15.113:0/2528090011 shutdown_connections 2024-04-12T05:06:07.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:07.973+0000 7f1d9218e700 1 -- 172.21.15.113:0/2528090011 wait complete. 2024-04-12T05:06:08.624 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:06:08.637 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:06:08.637 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-12T05:06:09.386 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:07.547296+0000 mgr.smithi113.ctsazg (mgr.14194) 195 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:09.386 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:07.972811+0000 mgr.smithi113.ctsazg (mgr.14194) 196 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:06:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:07.547296+0000 mgr.smithi113.ctsazg (mgr.14194) 195 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:07.972811+0000 mgr.smithi113.ctsazg (mgr.14194) 196 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:06:10.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.565+0000 7f447bc2a700 1 -- 172.21.15.113:0/4191818003 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44740f66e0 msgr2=0x7f44740f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:10.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.565+0000 7f447bc2a700 1 --2- 172.21.15.113:0/4191818003 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44740f66e0 0x7f44740f6b00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f4464009b60 tx=0x7f4464004d10).stop 2024-04-12T05:06:10.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.565+0000 7f447bc2a700 1 -- 172.21.15.113:0/4191818003 shutdown_connections 2024-04-12T05:06:10.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.565+0000 7f447bc2a700 1 --2- 172.21.15.113:0/4191818003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44740f6ff0 0x7f44740f7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:06:10.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.565+0000 7f447bc2a700 1 --2- 172.21.15.113:0/4191818003 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44740f66e0 0x7f44740f6b00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:10.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.565+0000 7f447bc2a700 1 -- 172.21.15.113:0/4191818003 >> 172.21.15.113:0/4191818003 conn(0x7f44740f1ee0 msgr2=0x7f44740f4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.565+0000 7f447bc2a700 1 -- 172.21.15.113:0/4191818003 shutdown_connections 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.565+0000 7f447bc2a700 1 -- 172.21.15.113:0/4191818003 wait complete. 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f447bc2a700 1 Processor -- start 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f447bc2a700 1 -- start start 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f447bc2a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44740f66e0 0x7f44740615f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f447bc2a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44740f6ff0 0x7f447410ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f447bc2a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447410d340 con 0x7f44740f6ff0 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f447bc2a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447410d480 con 0x7f44740f66e0 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f44791c5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44740f6ff0 0x7f447410ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f44791c5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44740f6ff0 0x7f447410ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46428/0 (socket says 172.21.15.113:46428) 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f44791c5700 1 -- 172.21.15.113:0/713966284 learned_addr learned my addr 172.21.15.113:0/713966284 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f44799c6700 1 --2- 172.21.15.113:0/713966284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44740f66e0 0x7f44740615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f44791c5700 1 -- 172.21.15.113:0/713966284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44740f66e0 msgr2=0x7f44740615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f44791c5700 1 --2- 172.21.15.113:0/713966284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44740f66e0 0x7f44740615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f44791c5700 1 -- 172.21.15.113:0/713966284 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44640097c0 con 0x7f44740f6ff0 2024-04-12T05:06:10.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.566+0000 7f44791c5700 1 --2- 172.21.15.113:0/713966284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44740f6ff0 0x7f447410ce50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f446800b4b0 tx=0x7f44680090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:10.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.567+0000 7f4472ffd700 1 -- 172.21.15.113:0/713966284 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4468002500 con 0x7f44740f6ff0 2024-04-12T05:06:10.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.567+0000 7f4472ffd700 1 -- 172.21.15.113:0/713966284 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f4468002af0 con 0x7f44740f6ff0 2024-04-12T05:06:10.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.567+0000 7f447bc2a700 1 -- 172.21.15.113:0/713966284 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447410d760 con 0x7f44740f6ff0 2024-04-12T05:06:10.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.567+0000 7f4472ffd700 1 -- 172.21.15.113:0/713966284 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4468012c90 con 0x7f44740f6ff0 2024-04-12T05:06:10.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.567+0000 7f447bc2a700 1 -- 172.21.15.113:0/713966284 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447410dca0 con 0x7f44740f6ff0 2024-04-12T05:06:10.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.567+0000 7f447bc2a700 1 -- 172.21.15.113:0/713966284 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44740fef60 con 0x7f44740f6ff0 2024-04-12T05:06:10.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.568+0000 7f4472ffd700 1 -- 172.21.15.113:0/713966284 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f4468002660 con 0x7f44740f6ff0 2024-04-12T05:06:10.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.568+0000 7f4472ffd700 1 --2- 172.21.15.113:0/713966284 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f445c061180 0x7f445c063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:10.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.568+0000 7f4472ffd700 1 -- 172.21.15.113:0/713966284 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4535+0+0 (secure 0 0 0) 0x7f446807c660 con 0x7f44740f6ff0 2024-04-12T05:06:10.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.568+0000 7f44799c6700 1 --2- 172.21.15.113:0/713966284 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f445c061180 0x7f445c063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:10.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.569+0000 7f44799c6700 1 --2- 172.21.15.113:0/713966284 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f445c061180 0x7f445c063640 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f446400b500 tx=0x7f4464004d10).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:10.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.573+0000 7f4472ffd700 1 -- 172.21.15.113:0/713966284 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4468050930 con 0x7f44740f6ff0 2024-04-12T05:06:10.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:10.763+0000 7f447bc2a700 1 -- 172.21.15.113:0/713966284 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4474059120 con 0x7f445c061180 2024-04-12T05:06:11.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:09.547767+0000 mgr.smithi113.ctsazg (mgr.14194) 197 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:11.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:10.766235+0000 mon.smithi113 (mon.0) 505 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-12T05:06:11.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:09.547767+0000 mgr.smithi113.ctsazg (mgr.14194) 197 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:11.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:10.766235+0000 mon.smithi113 (mon.0) 505 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-12T05:06:12.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.164+0000 7f4472ffd700 1 -- 172.21.15.113:0/713966284 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4474059120 con 0x7f445c061180 2024-04-12T05:06:12.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.166+0000 7f447bc2a700 1 -- 172.21.15.113:0/713966284 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f445c061180 msgr2=0x7f445c063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:12.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.166+0000 7f447bc2a700 1 --2- 172.21.15.113:0/713966284 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f445c061180 0x7f445c063640 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f446400b500 tx=0x7f4464004d10).stop 2024-04-12T05:06:12.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.166+0000 7f447bc2a700 1 -- 172.21.15.113:0/713966284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44740f6ff0 msgr2=0x7f447410ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:12.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.166+0000 7f447bc2a700 1 --2- 172.21.15.113:0/713966284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44740f6ff0 0x7f447410ce50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f446800b4b0 tx=0x7f44680090f0).stop 2024-04-12T05:06:12.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.166+0000 7f447bc2a700 1 -- 172.21.15.113:0/713966284 shutdown_connections 2024-04-12T05:06:12.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.166+0000 7f447bc2a700 1 --2- 172.21.15.113:0/713966284 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f445c061180 0x7f445c063640 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:12.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.166+0000 7f447bc2a700 1 --2- 172.21.15.113:0/713966284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44740f66e0 0x7f44740615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:12.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.166+0000 7f447bc2a700 1 --2- 172.21.15.113:0/713966284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44740f6ff0 0x7f447410ce50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:12.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.166+0000 7f447bc2a700 1 -- 172.21.15.113:0/713966284 >> 172.21.15.113:0/713966284 conn(0x7f44740f1ee0 msgr2=0x7f44740fb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:12.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.166+0000 7f447bc2a700 1 -- 172.21.15.113:0/713966284 shutdown_connections 2024-04-12T05:06:12.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:12.166+0000 7f447bc2a700 1 -- 172.21.15.113:0/713966284 wait complete. 2024-04-12T05:06:12.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T05:06:12.141+0000 7fc484821700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:06:12.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:10.765719+0000 mgr.smithi113.ctsazg (mgr.14194) 198 : audit [DBG] from='client.14418 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:06:12.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:11.139871+0000 mon.smithi113 (mon.0) 506 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:11.139915+0000 mon.smithi113 (mon.0) 507 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:11.141147+0000 mon.smithi113 (mon.0) 508 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:11.548212+0000 mgr.smithi113.ctsazg (mgr.14194) 199 : cluster [DBG] pgmap v171: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.139316+0000 mon.smithi113 (mon.0) 509 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:12.139450+0000 mon.smithi113 (mon.0) 510 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.142794+0000 mon.smithi113 (mon.0) 511 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:12.143608+0000 mon.smithi113 (mon.0) 512 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:12.143646+0000 mon.smithi113 (mon.0) 513 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:12.152098+0000 mon.smithi113 (mon.0) 514 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.153724+0000 mon.smithi113 (mon.0) 515 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:12.153812+0000 mon.smithi113 (mon.0) 516 : cluster [DBG] fsmap cephfs:0 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:06:12.159716+0000 mgr.smithi113.ctsazg (mgr.14194) 200 : cephadm [INF] Saving service mds.cephfs spec with placement count:4 2024-04-12T05:06:12.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.165706+0000 mon.smithi113 (mon.0) 517 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:12.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.169777+0000 mon.smithi113 (mon.0) 518 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:06:12.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:10.765719+0000 mgr.smithi113.ctsazg (mgr.14194) 198 : audit [DBG] from='client.14418 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:11.139871+0000 mon.smithi113 (mon.0) 506 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:11.139915+0000 mon.smithi113 (mon.0) 507 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:11.141147+0000 mon.smithi113 (mon.0) 508 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:11.548212+0000 mgr.smithi113.ctsazg (mgr.14194) 199 : cluster [DBG] pgmap v171: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.139316+0000 mon.smithi113 (mon.0) 509 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:12.139450+0000 mon.smithi113 (mon.0) 510 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.142794+0000 mon.smithi113 (mon.0) 511 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:12.143608+0000 mon.smithi113 (mon.0) 512 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:12.143646+0000 mon.smithi113 (mon.0) 513 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:12.152098+0000 mon.smithi113 (mon.0) 514 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.153724+0000 mon.smithi113 (mon.0) 515 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-12T05:06:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:12.153812+0000 mon.smithi113 (mon.0) 516 : cluster [DBG] fsmap cephfs:0 2024-04-12T05:06:12.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:06:12.159716+0000 mgr.smithi113.ctsazg (mgr.14194) 200 : cephadm [INF] Saving service mds.cephfs spec with placement count:4 2024-04-12T05:06:12.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.165706+0000 mon.smithi113 (mon.0) 517 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:12.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.169777+0000 mon.smithi113 (mon.0) 518 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:06:12.870 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-12T05:06:13.483 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.465243+0000 mon.smithi113 (mon.0) 519 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:13.483 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.465893+0000 mon.smithi113 (mon.0) 520 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:06:13.484 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.471840+0000 mon.smithi113 (mon.0) 521 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:13.484 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.480454+0000 mon.smithi113 (mon.0) 522 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:13.484 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.481152+0000 mon.smithi113 (mon.0) 523 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ovmadh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:06:13.484 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.485497+0000 mon.smithi113 (mon.0) 524 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ovmadh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:06:13.484 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:12.486085+0000 mon.smithi113 (mon.0) 525 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:13.484 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:06:12.486611+0000 mgr.smithi113.ctsazg (mgr.14194) 201 : cephadm [INF] Deploying daemon mds.cephfs.smithi113.ovmadh on smithi113 2024-04-12T05:06:13.484 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:13.153661+0000 mon.smithi113 (mon.0) 526 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2024-04-12T05:06:13.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.465243+0000 mon.smithi113 (mon.0) 519 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:13.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.465893+0000 mon.smithi113 (mon.0) 520 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:06:13.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.471840+0000 mon.smithi113 (mon.0) 521 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:13.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.480454+0000 mon.smithi113 (mon.0) 522 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:13.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.481152+0000 mon.smithi113 (mon.0) 523 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ovmadh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:06:13.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.485497+0000 mon.smithi113 (mon.0) 524 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ovmadh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:06:13.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:12.486085+0000 mon.smithi113 (mon.0) 525 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:13.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:06:12.486611+0000 mgr.smithi113.ctsazg (mgr.14194) 201 : cephadm [INF] Deploying daemon mds.cephfs.smithi113.ovmadh on smithi113 2024-04-12T05:06:13.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:13.153661+0000 mon.smithi113 (mon.0) 526 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2024-04-12T05:06:15.392 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:13.548590+0000 mgr.smithi113.ctsazg (mgr.14194) 202 : cluster [DBG] pgmap v175: 65 pgs: 13 creating+peering, 40 unknown, 12 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:15.392 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:14.153961+0000 mon.smithi113 (mon.0) 527 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2024-04-12T05:06:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:13.548590+0000 mgr.smithi113.ctsazg (mgr.14194) 202 : cluster [DBG] pgmap v175: 65 pgs: 13 creating+peering, 40 unknown, 12 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:14.153961+0000 mon.smithi113 (mon.0) 527 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2024-04-12T05:06:15.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.661+0000 7fa71a065700 1 -- 172.21.15.113:0/502577596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7140fa9b0 msgr2=0x7fa7140fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:15.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.661+0000 7fa71a065700 1 --2- 172.21.15.113:0/502577596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7140fa9b0 0x7fa7140fcda0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fa6fc0047f0 tx=0x7fa6fc009cb0).stop 2024-04-12T05:06:15.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.661+0000 7fa71a065700 1 -- 172.21.15.113:0/502577596 shutdown_connections 2024-04-12T05:06:15.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.661+0000 7fa71a065700 1 --2- 172.21.15.113:0/502577596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7140fa9b0 0x7fa7140fcda0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:15.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.661+0000 7fa71a065700 1 --2- 172.21.15.113:0/502577596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa714061220 0x7fa7140fa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:15.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.661+0000 7fa71a065700 1 -- 172.21.15.113:0/502577596 >> 172.21.15.113:0/502577596 conn(0x7fa7140f1e50 msgr2=0x7fa7140f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:15.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.661+0000 7fa71a065700 1 -- 172.21.15.113:0/502577596 shutdown_connections 2024-04-12T05:06:15.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.661+0000 7fa71a065700 1 -- 172.21.15.113:0/502577596 wait complete. 2024-04-12T05:06:15.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa71a065700 1 Processor -- start 2024-04-12T05:06:15.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa71a065700 1 -- start start 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa71a065700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa714061220 0x7fa7140f7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa71a065700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7140fa9b0 0x7fa7140f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa71a065700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7140f8280 con 0x7fa7140fa9b0 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa71a065700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7140f6820 con 0x7fa714061220 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa70affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7140fa9b0 0x7fa7140f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa70affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7140fa9b0 0x7fa7140f6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46458/0 (socket says 172.21.15.113:46458) 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa70affd700 1 -- 172.21.15.113:0/3908319978 learned_addr learned my addr 172.21.15.113:0/3908319978 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa7137fe700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa714061220 0x7fa7140f7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa70affd700 1 -- 172.21.15.113:0/3908319978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa714061220 msgr2=0x7fa7140f7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa70affd700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa714061220 0x7fa7140f7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.662+0000 7fa70affd700 1 -- 172.21.15.113:0/3908319978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6fc005040 con 0x7fa7140fa9b0 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.663+0000 7fa7137fe700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa714061220 0x7fa7140f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.663+0000 7fa70affd700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7140fa9b0 0x7fa7140f6330 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fa6fc013580 tx=0x7fa6fc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:15.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.663+0000 7fa7117fa700 1 -- 172.21.15.113:0/3908319978 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6fc018070 con 0x7fa7140fa9b0 2024-04-12T05:06:15.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.663+0000 7fa71a065700 1 -- 172.21.15.113:0/3908319978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7140f6a40 con 0x7fa7140fa9b0 2024-04-12T05:06:15.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.663+0000 7fa7117fa700 1 -- 172.21.15.113:0/3908319978 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fa6fc007e90 con 0x7fa7140fa9b0 2024-04-12T05:06:15.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.663+0000 7fa7117fa700 1 -- 172.21.15.113:0/3908319978 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6fc0123c0 con 0x7fa7140fa9b0 2024-04-12T05:06:15.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.663+0000 7fa71a065700 1 -- 172.21.15.113:0/3908319978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7140f6fa0 con 0x7fa7140fa9b0 2024-04-12T05:06:15.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.664+0000 7fa7117fa700 1 -- 172.21.15.113:0/3908319978 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fa6fc01d420 con 0x7fa7140fa9b0 2024-04-12T05:06:15.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.664+0000 7fa71a065700 1 -- 172.21.15.113:0/3908319978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa714047540 con 0x7fa7140fa9b0 2024-04-12T05:06:15.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.664+0000 7fa7117fa700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa700061180 0x7fa700063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:15.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.665+0000 7fa7117fa700 1 -- 172.21.15.113:0/3908319978 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7fa6fc07d500 con 0x7fa7140fa9b0 2024-04-12T05:06:15.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.665+0000 7fa7137fe700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa700061180 0x7fa700063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:15.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.665+0000 7fa7137fe700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa700061180 0x7fa700063640 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa704007fd0 tx=0x7fa70400b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:15.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.669+0000 7fa7117fa700 1 -- 172.21.15.113:0/3908319978 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa6fc051490 con 0x7fa7140fa9b0 2024-04-12T05:06:15.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.898+0000 7fa71a065700 1 -- 172.21.15.113:0/3908319978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa714101fc0 con 0x7fa7140fa9b0 2024-04-12T05:06:15.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.898+0000 7fa7117fa700 1 -- 172.21.15.113:0/3908319978 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1034 (secure 0 0 0) 0x7fa6fc051080 con 0x7fa7140fa9b0 2024-04-12T05:06:15.901 INFO:teuthology.orchestra.run.smithi113.stdout:e2 2024-04-12T05:06:15.901 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:06:15.901 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:06:15.901 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:06:15.901 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:06:15.901 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:06:15.901 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:06:15.901 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 2 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:06:12.143515+0000 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:06:15.902 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 1 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout:in 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 0 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:06:15.903 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:06:15.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.900+0000 7fa71a065700 1 -- 172.21.15.113:0/3908319978 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa700061180 msgr2=0x7fa700063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:15.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.900+0000 7fa71a065700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa700061180 0x7fa700063640 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa704007fd0 tx=0x7fa70400b040).stop 2024-04-12T05:06:15.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.900+0000 7fa71a065700 1 -- 172.21.15.113:0/3908319978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7140fa9b0 msgr2=0x7fa7140f6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:15.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.900+0000 7fa71a065700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7140fa9b0 0x7fa7140f6330 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fa6fc013580 tx=0x7fa6fc009cb0).stop 2024-04-12T05:06:15.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.900+0000 7fa71a065700 1 -- 172.21.15.113:0/3908319978 shutdown_connections 2024-04-12T05:06:15.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.900+0000 7fa71a065700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa700061180 0x7fa700063640 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:15.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.900+0000 7fa71a065700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa714061220 0x7fa7140f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:15.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.900+0000 7fa71a065700 1 --2- 172.21.15.113:0/3908319978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7140fa9b0 0x7fa7140f6330 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:15.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.900+0000 7fa71a065700 1 -- 172.21.15.113:0/3908319978 >> 172.21.15.113:0/3908319978 conn(0x7fa7140f1e50 msgr2=0x7fa7140f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:15.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.900+0000 7fa71a065700 1 -- 172.21.15.113:0/3908319978 shutdown_connections 2024-04-12T05:06:15.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:15.900+0000 7fa71a065700 1 -- 172.21.15.113:0/3908319978 wait complete. 2024-04-12T05:06:15.905 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 2 2024-04-12T05:06:16.415 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:15.899511+0000 mon.smithi113 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.113:0/3908319978' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:06:16.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:15.899511+0000 mon.smithi113 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.113:0/3908319978' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:06:16.862 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:06:16.872 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:06:16.872 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-12T05:06:17.387 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:15.548964+0000 mgr.smithi113.ctsazg (mgr.14194) 203 : cluster [DBG] pgmap v177: 65 pgs: 13 creating+peering, 19 unknown, 33 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:17.387 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:16.496760+0000 mon.smithi113 (mon.0) 529 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:17.387 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:16.497659+0000 mon.smithi113 (mon.0) 530 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.ammjap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:06:17.388 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:16.500711+0000 mon.smithi113 (mon.0) 531 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.ammjap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:06:17.388 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:16.501297+0000 mon.smithi113 (mon.0) 532 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:17.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:15.548964+0000 mgr.smithi113.ctsazg (mgr.14194) 203 : cluster [DBG] pgmap v177: 65 pgs: 13 creating+peering, 19 unknown, 33 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:16.496760+0000 mon.smithi113 (mon.0) 529 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:16.497659+0000 mon.smithi113 (mon.0) 530 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.ammjap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:06:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:16.500711+0000 mon.smithi113 (mon.0) 531 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.ammjap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:06:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:16.501297+0000 mon.smithi113 (mon.0) 532 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:18.300 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:06:16.501765+0000 mgr.smithi113.ctsazg (mgr.14194) 204 : cephadm [INF] Deploying daemon mds.cephfs.smithi192.ammjap on smithi192 2024-04-12T05:06:18.301 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:17.159890+0000 mon.smithi113 (mon.0) 533 : cluster [DBG] mds.? [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] up:boot 2024-04-12T05:06:18.301 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:17.159924+0000 mon.smithi113 (mon.0) 534 : cluster [INF] daemon mds.cephfs.smithi113.ovmadh assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-12T05:06:18.301 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:17.159967+0000 mon.smithi113 (mon.0) 535 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T05:06:18.301 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:17.159980+0000 mon.smithi113 (mon.0) 536 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-12T05:06:18.301 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:17.159988+0000 mon.smithi113 (mon.0) 537 : cluster [INF] Cluster is now healthy 2024-04-12T05:06:18.301 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:17.160200+0000 mon.smithi113 (mon.0) 538 : cluster [DBG] fsmap cephfs:0 1 up:standby 2024-04-12T05:06:18.301 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:17.160285+0000 mon.smithi113 (mon.0) 539 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:06:18.301 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:17.165125+0000 mon.smithi113 (mon.0) 540 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi113.ovmadh=up:creating} 2024-04-12T05:06:18.301 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:17.175570+0000 mon.smithi113 (mon.0) 541 : cluster [INF] daemon mds.cephfs.smithi113.ovmadh is now active in filesystem cephfs as rank 0 2024-04-12T05:06:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:06:16.501765+0000 mgr.smithi113.ctsazg (mgr.14194) 204 : cephadm [INF] Deploying daemon mds.cephfs.smithi192.ammjap on smithi192 2024-04-12T05:06:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:17.159890+0000 mon.smithi113 (mon.0) 533 : cluster [DBG] mds.? [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] up:boot 2024-04-12T05:06:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:17.159924+0000 mon.smithi113 (mon.0) 534 : cluster [INF] daemon mds.cephfs.smithi113.ovmadh assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-12T05:06:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:17.159967+0000 mon.smithi113 (mon.0) 535 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T05:06:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:17.159980+0000 mon.smithi113 (mon.0) 536 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-12T05:06:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:17.159988+0000 mon.smithi113 (mon.0) 537 : cluster [INF] Cluster is now healthy 2024-04-12T05:06:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:17.160200+0000 mon.smithi113 (mon.0) 538 : cluster [DBG] fsmap cephfs:0 1 up:standby 2024-04-12T05:06:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:17.160285+0000 mon.smithi113 (mon.0) 539 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:06:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:17.165125+0000 mon.smithi113 (mon.0) 540 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi113.ovmadh=up:creating} 2024-04-12T05:06:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:17.175570+0000 mon.smithi113 (mon.0) 541 : cluster [INF] daemon mds.cephfs.smithi113.ovmadh is now active in filesystem cephfs as rank 0 2024-04-12T05:06:19.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.991+0000 7f8f5d32a700 1 -- 172.21.15.113:0/1719122233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fa9b0 msgr2=0x7f8f580fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:19.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.991+0000 7f8f5d32a700 1 --2- 172.21.15.113:0/1719122233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fa9b0 0x7f8f580fcda0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f8f4c0047f0 tx=0x7f8f4c009cb0).stop 2024-04-12T05:06:19.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.991+0000 7f8f5d32a700 1 -- 172.21.15.113:0/1719122233 shutdown_connections 2024-04-12T05:06:19.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.991+0000 7f8f5d32a700 1 --2- 172.21.15.113:0/1719122233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fa9b0 0x7f8f580fcda0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:19.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.991+0000 7f8f5d32a700 1 --2- 172.21.15.113:0/1719122233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f58061730 0x7f8f580fa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:19.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.991+0000 7f8f5d32a700 1 -- 172.21.15.113:0/1719122233 >> 172.21.15.113:0/1719122233 conn(0x7f8f580f1ee0 msgr2=0x7f8f580f4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:19.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.991+0000 7f8f5d32a700 1 -- 172.21.15.113:0/1719122233 shutdown_connections 2024-04-12T05:06:19.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.991+0000 7f8f5d32a700 1 -- 172.21.15.113:0/1719122233 wait complete. 2024-04-12T05:06:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.992+0000 7f8f5d32a700 1 Processor -- start 2024-04-12T05:06:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.992+0000 7f8f5d32a700 1 -- start start 2024-04-12T05:06:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f5d32a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f58061730 0x7f8f581080d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f5d32a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f580fa9b0 0x7f8f581085c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f5d32a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f58108b70 con 0x7f8f58061730 2024-04-12T05:06:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f5d32a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f58108cb0 con 0x7f8f580fa9b0 2024-04-12T05:06:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f56ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f58061730 0x7f8f581080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:19.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f56ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f58061730 0x7f8f581080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46484/0 (socket says 172.21.15.113:46484) 2024-04-12T05:06:19.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f56ffd700 1 -- 172.21.15.113:0/386837995 learned_addr learned my addr 172.21.15.113:0/386837995 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:19.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f567fc700 1 --2- 172.21.15.113:0/386837995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f580fa9b0 0x7f8f581085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:19.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f56ffd700 1 -- 172.21.15.113:0/386837995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f580fa9b0 msgr2=0x7f8f581085c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:19.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f56ffd700 1 --2- 172.21.15.113:0/386837995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f580fa9b0 0x7f8f581085c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:19.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f56ffd700 1 -- 172.21.15.113:0/386837995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4c005040 con 0x7f8f58061730 2024-04-12T05:06:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f56ffd700 1 --2- 172.21.15.113:0/386837995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f58061730 0x7f8f581080d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f8f4800e8c0 tx=0x7f8f4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f3ffff700 1 -- 172.21.15.113:0/386837995 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f48007b40 con 0x7f8f58061730 2024-04-12T05:06:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f5d32a700 1 -- 172.21.15.113:0/386837995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5810d750 con 0x7f8f58061730 2024-04-12T05:06:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f3ffff700 1 -- 172.21.15.113:0/386837995 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f8f4800f040 con 0x7f8f58061730 2024-04-12T05:06:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f3ffff700 1 -- 172.21.15.113:0/386837995 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f480147e0 con 0x7f8f58061730 2024-04-12T05:06:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.993+0000 7f8f5d32a700 1 -- 172.21.15.113:0/386837995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5810dd10 con 0x7f8f58061730 2024-04-12T05:06:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.994+0000 7f8f5d32a700 1 -- 172.21.15.113:0/386837995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f580fffb0 con 0x7f8f58061730 2024-04-12T05:06:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.995+0000 7f8f3ffff700 1 -- 172.21.15.113:0/386837995 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f8f480090d0 con 0x7f8f58061730 2024-04-12T05:06:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.995+0000 7f8f3ffff700 1 --2- 172.21.15.113:0/386837995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8f40061180 0x7f8f40063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.995+0000 7f8f3ffff700 1 -- 172.21.15.113:0/386837995 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f8f4807c2c0 con 0x7f8f58061730 2024-04-12T05:06:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.995+0000 7f8f567fc700 1 --2- 172.21.15.113:0/386837995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8f40061180 0x7f8f40063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.995+0000 7f8f567fc700 1 --2- 172.21.15.113:0/386837995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8f40061180 0x7f8f40063640 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f8f4c0047c0 tx=0x7f8f4c009cb0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:19.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:18.999+0000 7f8f3ffff700 1 -- 172.21.15.113:0/386837995 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8f4804af60 con 0x7f8f58061730 2024-04-12T05:06:19.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:19.209+0000 7f8f5d32a700 1 -- 172.21.15.113:0/386837995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f8f580598f0 con 0x7f8f58061730 2024-04-12T05:06:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:17.549349+0000 mgr.smithi113.ctsazg (mgr.14194) 205 : cluster [DBG] pgmap v178: 65 pgs: 4 creating+peering, 61 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:18.167305+0000 mon.smithi113 (mon.0) 542 : cluster [DBG] mds.? [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] up:active 2024-04-12T05:06:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:18.167348+0000 mon.smithi113 (mon.0) 543 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi113.ovmadh=up:active} 2024-04-12T05:06:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:19.115796+0000 mon.smithi113 (mon.0) 544 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:19.116667+0000 mon.smithi113 (mon.0) 545 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ydsbei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:06:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:19.119293+0000 mon.smithi113 (mon.0) 546 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ydsbei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:06:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:19.119890+0000 mon.smithi113 (mon.0) 547 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:17.549349+0000 mgr.smithi113.ctsazg (mgr.14194) 205 : cluster [DBG] pgmap v178: 65 pgs: 4 creating+peering, 61 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:18.167305+0000 mon.smithi113 (mon.0) 542 : cluster [DBG] mds.? [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] up:active 2024-04-12T05:06:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:18.167348+0000 mon.smithi113 (mon.0) 543 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi113.ovmadh=up:active} 2024-04-12T05:06:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:19.115796+0000 mon.smithi113 (mon.0) 544 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:19.116667+0000 mon.smithi113 (mon.0) 545 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ydsbei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:06:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:19.119293+0000 mon.smithi113 (mon.0) 546 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ydsbei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:06:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:19.119890+0000 mon.smithi113 (mon.0) 547 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:20.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.180+0000 7f8f3ffff700 1 -- 172.21.15.113:0/386837995 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7f8f48050300 con 0x7f8f58061730 2024-04-12T05:06:20.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.182+0000 7f8f5d32a700 1 -- 172.21.15.113:0/386837995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8f40061180 msgr2=0x7f8f40063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.182+0000 7f8f5d32a700 1 --2- 172.21.15.113:0/386837995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8f40061180 0x7f8f40063640 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f8f4c0047c0 tx=0x7f8f4c009cb0).stop 2024-04-12T05:06:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.183+0000 7f8f5d32a700 1 -- 172.21.15.113:0/386837995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f58061730 msgr2=0x7f8f581080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.183+0000 7f8f5d32a700 1 --2- 172.21.15.113:0/386837995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f58061730 0x7f8f581080d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f8f4800e8c0 tx=0x7f8f4800b040).stop 2024-04-12T05:06:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.183+0000 7f8f5d32a700 1 -- 172.21.15.113:0/386837995 shutdown_connections 2024-04-12T05:06:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.183+0000 7f8f5d32a700 1 --2- 172.21.15.113:0/386837995 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8f40061180 0x7f8f40063640 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.183+0000 7f8f5d32a700 1 --2- 172.21.15.113:0/386837995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f58061730 0x7f8f581080d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.183+0000 7f8f5d32a700 1 --2- 172.21.15.113:0/386837995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f580fa9b0 0x7f8f581085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.183+0000 7f8f5d32a700 1 -- 172.21.15.113:0/386837995 >> 172.21.15.113:0/386837995 conn(0x7f8f580f1ee0 msgr2=0x7f8f580f42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.183+0000 7f8f5d32a700 1 -- 172.21.15.113:0/386837995 shutdown_connections 2024-04-12T05:06:20.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:20.183+0000 7f8f5d32a700 1 -- 172.21.15.113:0/386837995 wait complete. 2024-04-12T05:06:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:06:19.120440+0000 mgr.smithi113.ctsazg (mgr.14194) 206 : cephadm [INF] Deploying daemon mds.cephfs.smithi113.ydsbei on smithi113 2024-04-12T05:06:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:19.172333+0000 mon.smithi113 (mon.0) 548 : cluster [DBG] mds.? [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] up:boot 2024-04-12T05:06:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:19.172598+0000 mon.smithi113 (mon.0) 549 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi113.ovmadh=up:active} 1 up:standby 2024-04-12T05:06:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:19.172718+0000 mon.smithi113 (mon.0) 550 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:06:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:19.175987+0000 mon.smithi113 (mon.0) 551 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi113.ovmadh=up:active} 1 up:standby 2024-04-12T05:06:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:19.210432+0000 mon.smithi113 (mon.0) 552 : audit [INF] from='client.? 172.21.15.113:0/386837995' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-12T05:06:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:06:19.120440+0000 mgr.smithi113.ctsazg (mgr.14194) 206 : cephadm [INF] Deploying daemon mds.cephfs.smithi113.ydsbei on smithi113 2024-04-12T05:06:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:19.172333+0000 mon.smithi113 (mon.0) 548 : cluster [DBG] mds.? [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] up:boot 2024-04-12T05:06:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:19.172598+0000 mon.smithi113 (mon.0) 549 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi113.ovmadh=up:active} 1 up:standby 2024-04-12T05:06:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:19.172718+0000 mon.smithi113 (mon.0) 550 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:06:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:19.175987+0000 mon.smithi113 (mon.0) 551 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi113.ovmadh=up:active} 1 up:standby 2024-04-12T05:06:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:19.210432+0000 mon.smithi113 (mon.0) 552 : audit [INF] from='client.? 172.21.15.113:0/386837995' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-12T05:06:21.177 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:06:21.189 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:06:21.189 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-04-12T05:06:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:19.549780+0000 mgr.smithi113.ctsazg (mgr.14194) 207 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 182 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:20.174995+0000 mon.smithi113 (mon.0) 553 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-12T05:06:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:20.181484+0000 mon.smithi113 (mon.0) 554 : audit [INF] from='client.? 172.21.15.113:0/386837995' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-12T05:06:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:20.181540+0000 mon.smithi113 (mon.0) 555 : cluster [INF] daemon mds.cephfs.smithi192.ammjap assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-12T05:06:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:20.181609+0000 mon.smithi113 (mon.0) 556 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-12T05:06:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:20.181627+0000 mon.smithi113 (mon.0) 557 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-12T05:06:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:20.181833+0000 mon.smithi113 (mon.0) 558 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi113.ovmadh=up:active} 1 up:standby 2024-04-12T05:06:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:20.184407+0000 mon.smithi113 (mon.0) 559 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:creating} 2024-04-12T05:06:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:20.194537+0000 mon.smithi113 (mon.0) 560 : cluster [INF] daemon mds.cephfs.smithi192.ammjap is now active in filesystem cephfs as rank 1 2024-04-12T05:06:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:19.549780+0000 mgr.smithi113.ctsazg (mgr.14194) 207 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 182 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:20.174995+0000 mon.smithi113 (mon.0) 553 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-12T05:06:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:20.181484+0000 mon.smithi113 (mon.0) 554 : audit [INF] from='client.? 172.21.15.113:0/386837995' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-12T05:06:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:20.181540+0000 mon.smithi113 (mon.0) 555 : cluster [INF] daemon mds.cephfs.smithi192.ammjap assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-12T05:06:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:20.181609+0000 mon.smithi113 (mon.0) 556 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-12T05:06:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:20.181627+0000 mon.smithi113 (mon.0) 557 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-12T05:06:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:20.181833+0000 mon.smithi113 (mon.0) 558 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi113.ovmadh=up:active} 1 up:standby 2024-04-12T05:06:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:20.184407+0000 mon.smithi113 (mon.0) 559 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:creating} 2024-04-12T05:06:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:20.194537+0000 mon.smithi113 (mon.0) 560 : cluster [INF] daemon mds.cephfs.smithi192.ammjap is now active in filesystem cephfs as rank 1 2024-04-12T05:06:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:21.185964+0000 mon.smithi113 (mon.0) 561 : cluster [DBG] mds.? [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] up:active 2024-04-12T05:06:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:21.186006+0000 mon.smithi113 (mon.0) 562 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2024-04-12T05:06:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:22.171076+0000 mon.smithi113 (mon.0) 563 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:22.172038+0000 mon.smithi113 (mon.0) 564 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.dtnaoi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:06:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:22.174794+0000 mon.smithi113 (mon.0) 565 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.dtnaoi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:06:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:22.175373+0000 mon.smithi113 (mon.0) 566 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:21.185964+0000 mon.smithi113 (mon.0) 561 : cluster [DBG] mds.? [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] up:active 2024-04-12T05:06:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:21.186006+0000 mon.smithi113 (mon.0) 562 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2024-04-12T05:06:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:22.171076+0000 mon.smithi113 (mon.0) 563 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:22.172038+0000 mon.smithi113 (mon.0) 564 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.dtnaoi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:06:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:22.174794+0000 mon.smithi113 (mon.0) 565 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.dtnaoi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T05:06:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:22.175373+0000 mon.smithi113 (mon.0) 566 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:23.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.343+0000 7fe01dd26700 1 -- 172.21.15.113:0/1619335066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0180f66d0 msgr2=0x7fe0180f6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:23.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.343+0000 7fe01dd26700 1 --2- 172.21.15.113:0/1619335066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0180f66d0 0x7fe0180f6af0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fe0000047f0 tx=0x7fe000009cb0).stop 2024-04-12T05:06:23.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.343+0000 7fe01dd26700 1 -- 172.21.15.113:0/1619335066 shutdown_connections 2024-04-12T05:06:23.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.343+0000 7fe01dd26700 1 --2- 172.21.15.113:0/1619335066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0180f6fe0 0x7fe0180f7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:23.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.343+0000 7fe01dd26700 1 --2- 172.21.15.113:0/1619335066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0180f66d0 0x7fe0180f6af0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:23.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.343+0000 7fe01dd26700 1 -- 172.21.15.113:0/1619335066 >> 172.21.15.113:0/1619335066 conn(0x7fe0180f1eb0 msgr2=0x7fe0180f42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:23.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.343+0000 7fe01dd26700 1 -- 172.21.15.113:0/1619335066 shutdown_connections 2024-04-12T05:06:23.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.343+0000 7fe01dd26700 1 -- 172.21.15.113:0/1619335066 wait complete. 2024-04-12T05:06:23.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.344+0000 7fe01dd26700 1 Processor -- start 2024-04-12T05:06:23.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.344+0000 7fe01dd26700 1 -- start start 2024-04-12T05:06:23.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.344+0000 7fe01dd26700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0180f66d0 0x7fe018108020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:23.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.344+0000 7fe01dd26700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0180f6fe0 0x7fe018108510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:23.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.344+0000 7fe01dd26700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe018108ac0 con 0x7fe0180f66d0 2024-04-12T05:06:23.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.344+0000 7fe01dd26700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe018108c00 con 0x7fe0180f6fe0 2024-04-12T05:06:23.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.344+0000 7fe0177fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0180f66d0 0x7fe018108020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:23.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.344+0000 7fe0177fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0180f66d0 0x7fe018108020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36116/0 (socket says 172.21.15.113:36116) 2024-04-12T05:06:23.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.344+0000 7fe0177fe700 1 -- 172.21.15.113:0/1520785409 learned_addr learned my addr 172.21.15.113:0/1520785409 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:23.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.344+0000 7fe016ffd700 1 --2- 172.21.15.113:0/1520785409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0180f6fe0 0x7fe018108510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:23.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.345+0000 7fe0177fe700 1 -- 172.21.15.113:0/1520785409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0180f6fe0 msgr2=0x7fe018108510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:23.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.345+0000 7fe0177fe700 1 --2- 172.21.15.113:0/1520785409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0180f6fe0 0x7fe018108510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:23.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.345+0000 7fe0177fe700 1 -- 172.21.15.113:0/1520785409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe000005040 con 0x7fe0180f66d0 2024-04-12T05:06:23.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.345+0000 7fe0177fe700 1 --2- 172.21.15.113:0/1520785409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0180f66d0 0x7fe018108020 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fe000005010 tx=0x7fe000009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:23.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.345+0000 7fe014ff9700 1 -- 172.21.15.113:0/1520785409 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe00000e030 con 0x7fe0180f66d0 2024-04-12T05:06:23.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.345+0000 7fe014ff9700 1 -- 172.21.15.113:0/1520785409 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fe000007e90 con 0x7fe0180f66d0 2024-04-12T05:06:23.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.345+0000 7fe01dd26700 1 -- 172.21.15.113:0/1520785409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01810d640 con 0x7fe0180f66d0 2024-04-12T05:06:23.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.345+0000 7fe014ff9700 1 -- 172.21.15.113:0/1520785409 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe000018070 con 0x7fe0180f66d0 2024-04-12T05:06:23.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.345+0000 7fe01dd26700 1 -- 172.21.15.113:0/1520785409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01810db80 con 0x7fe0180f66d0 2024-04-12T05:06:23.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.346+0000 7fe01dd26700 1 -- 172.21.15.113:0/1520785409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0180fef00 con 0x7fe0180f66d0 2024-04-12T05:06:23.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.346+0000 7fe014ff9700 1 -- 172.21.15.113:0/1520785409 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fe00001d420 con 0x7fe0180f66d0 2024-04-12T05:06:23.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.346+0000 7fe014ff9700 1 --2- 172.21.15.113:0/1520785409 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe004061180 0x7fe004063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:23.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.347+0000 7fe014ff9700 1 -- 172.21.15.113:0/1520785409 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7fe00007ddc0 con 0x7fe0180f66d0 2024-04-12T05:06:23.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.347+0000 7fe016ffd700 1 --2- 172.21.15.113:0/1520785409 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe004061180 0x7fe004063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:23.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.347+0000 7fe016ffd700 1 --2- 172.21.15.113:0/1520785409 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe004061180 0x7fe004063640 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe0080041d0 tx=0x7fe008009040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:23.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.351+0000 7fe014ff9700 1 -- 172.21.15.113:0/1520785409 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe000051d50 con 0x7fe0180f66d0 2024-04-12T05:06:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:21.550404+0000 mgr.smithi113.ctsazg (mgr.14194) 208 : cluster [DBG] pgmap v180: 65 pgs: 65 active+clean; 2.4 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 8.5 KiB/s wr, 22 op/s 2024-04-12T05:06:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:06:22.175886+0000 mgr.smithi113.ctsazg (mgr.14194) 209 : cephadm [INF] Deploying daemon mds.cephfs.smithi192.dtnaoi on smithi192 2024-04-12T05:06:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:22.184290+0000 mon.smithi113 (mon.0) 567 : cluster [INF] Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-12T05:06:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:22.184307+0000 mon.smithi113 (mon.0) 568 : cluster [INF] Cluster is now healthy 2024-04-12T05:06:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:22.192789+0000 mon.smithi113 (mon.0) 569 : cluster [DBG] mds.? [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] up:boot 2024-04-12T05:06:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:22.192846+0000 mon.smithi113 (mon.0) 570 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 1 up:standby 2024-04-12T05:06:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:22.193101+0000 mon.smithi113 (mon.0) 571 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:06:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:21.550404+0000 mgr.smithi113.ctsazg (mgr.14194) 208 : cluster [DBG] pgmap v180: 65 pgs: 65 active+clean; 2.4 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 8.5 KiB/s wr, 22 op/s 2024-04-12T05:06:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:06:22.175886+0000 mgr.smithi113.ctsazg (mgr.14194) 209 : cephadm [INF] Deploying daemon mds.cephfs.smithi192.dtnaoi on smithi192 2024-04-12T05:06:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:22.184290+0000 mon.smithi113 (mon.0) 567 : cluster [INF] Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-12T05:06:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:22.184307+0000 mon.smithi113 (mon.0) 568 : cluster [INF] Cluster is now healthy 2024-04-12T05:06:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:22.192789+0000 mon.smithi113 (mon.0) 569 : cluster [DBG] mds.? [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] up:boot 2024-04-12T05:06:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:22.192846+0000 mon.smithi113 (mon.0) 570 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 1 up:standby 2024-04-12T05:06:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:22.193101+0000 mon.smithi113 (mon.0) 571 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:06:23.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:23.555+0000 7fe01dd26700 1 -- 172.21.15.113:0/1520785409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe01810e1c0 con 0x7fe0180f66d0 2024-04-12T05:06:24.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.199+0000 7fe014ff9700 1 -- 172.21.15.113:0/1520785409 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe000051940 con 0x7fe0180f66d0 2024-04-12T05:06:24.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.201+0000 7fe01dd26700 1 -- 172.21.15.113:0/1520785409 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe004061180 msgr2=0x7fe004063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:24.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.201+0000 7fe01dd26700 1 --2- 172.21.15.113:0/1520785409 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe004061180 0x7fe004063640 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe0080041d0 tx=0x7fe008009040).stop 2024-04-12T05:06:24.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.201+0000 7fe01dd26700 1 -- 172.21.15.113:0/1520785409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0180f66d0 msgr2=0x7fe018108020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:24.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.201+0000 7fe01dd26700 1 --2- 172.21.15.113:0/1520785409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0180f66d0 0x7fe018108020 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fe000005010 tx=0x7fe000009cb0).stop 2024-04-12T05:06:24.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.201+0000 7fe01dd26700 1 -- 172.21.15.113:0/1520785409 shutdown_connections 2024-04-12T05:06:24.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.201+0000 7fe01dd26700 1 --2- 172.21.15.113:0/1520785409 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe004061180 0x7fe004063640 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:24.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.201+0000 7fe01dd26700 1 --2- 172.21.15.113:0/1520785409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0180f66d0 0x7fe018108020 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:24.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.201+0000 7fe01dd26700 1 --2- 172.21.15.113:0/1520785409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0180f6fe0 0x7fe018108510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:24.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.201+0000 7fe01dd26700 1 -- 172.21.15.113:0/1520785409 >> 172.21.15.113:0/1520785409 conn(0x7fe0180f1eb0 msgr2=0x7fe018061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:24.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.201+0000 7fe01dd26700 1 -- 172.21.15.113:0/1520785409 shutdown_connections 2024-04-12T05:06:24.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:24.201+0000 7fe01dd26700 1 -- 172.21.15.113:0/1520785409 wait complete. 2024-04-12T05:06:24.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:23.556741+0000 mon.smithi113 (mon.0) 572 : audit [INF] from='client.? 172.21.15.113:0/1520785409' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-12T05:06:24.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:23.870282+0000 mon.smithi113 (mon.0) 573 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:24.571 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:23.556741+0000 mon.smithi113 (mon.0) 572 : audit [INF] from='client.? 172.21.15.113:0/1520785409' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-12T05:06:24.571 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:23.870282+0000 mon.smithi113 (mon.0) 573 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:24.880 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:06:24.890 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:06:24.891 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data false' 2024-04-12T05:06:25.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:23.550737+0000 mgr.smithi113.ctsazg (mgr.14194) 210 : cluster [DBG] pgmap v181: 65 pgs: 65 active+clean; 2.4 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s wr, 12 op/s 2024-04-12T05:06:25.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:24.200482+0000 mon.smithi113 (mon.0) 574 : audit [INF] from='client.? 172.21.15.113:0/1520785409' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-12T05:06:25.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:24.200535+0000 mon.smithi113 (mon.0) 575 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 1 up:standby 2024-04-12T05:06:25.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:24.675265+0000 mon.smithi113 (mon.0) 576 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:25.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:24.676307+0000 mon.smithi113 (mon.0) 577 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:06:25.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:23.550737+0000 mgr.smithi113.ctsazg (mgr.14194) 210 : cluster [DBG] pgmap v181: 65 pgs: 65 active+clean; 2.4 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s wr, 12 op/s 2024-04-12T05:06:25.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:24.200482+0000 mon.smithi113 (mon.0) 574 : audit [INF] from='client.? 172.21.15.113:0/1520785409' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-12T05:06:25.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:24.200535+0000 mon.smithi113 (mon.0) 575 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 1 up:standby 2024-04-12T05:06:25.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:24.675265+0000 mon.smithi113 (mon.0) 576 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:25.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:24.676307+0000 mon.smithi113 (mon.0) 577 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:06:26.337 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:25.205055+0000 mon.smithi113 (mon.0) 578 : cluster [DBG] mds.? [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] up:boot 2024-04-12T05:06:26.337 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:25.205102+0000 mon.smithi113 (mon.0) 579 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:06:26.337 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:25.205224+0000 mon.smithi113 (mon.0) 580 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:06:26.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:25.205055+0000 mon.smithi113 (mon.0) 578 : cluster [DBG] mds.? [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] up:boot 2024-04-12T05:06:26.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:25.205102+0000 mon.smithi113 (mon.0) 579 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:06:26.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:25.205224+0000 mon.smithi113 (mon.0) 580 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:06:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:25.551141+0000 mgr.smithi113.ctsazg (mgr.14194) 211 : cluster [DBG] pgmap v182: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.5 KiB/s wr, 10 op/s 2024-04-12T05:06:27.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:25.551141+0000 mgr.smithi113.ctsazg (mgr.14194) 211 : cluster [DBG] pgmap v182: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.5 KiB/s wr, 10 op/s 2024-04-12T05:06:28.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:27.453013+0000 mon.smithi113 (mon.0) 581 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:28.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:27.551749+0000 mgr.smithi113.ctsazg (mgr.14194) 212 : cluster [DBG] pgmap v183: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s wr, 7 op/s 2024-04-12T05:06:28.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:27.453013+0000 mon.smithi113 (mon.0) 581 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:28.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:27.551749+0000 mgr.smithi113.ctsazg (mgr.14194) 212 : cluster [DBG] pgmap v183: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s wr, 7 op/s 2024-04-12T05:06:28.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.994+0000 7f664eda7700 1 -- 172.21.15.113:0/2079537896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66480fa950 msgr2=0x7f66480fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:28.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.994+0000 7f664eda7700 1 --2- 172.21.15.113:0/2079537896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66480fa950 0x7f66480fcda0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f66340047f0 tx=0x7f6634009cb0).stop 2024-04-12T05:06:28.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.994+0000 7f664eda7700 1 -- 172.21.15.113:0/2079537896 shutdown_connections 2024-04-12T05:06:28.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.994+0000 7f664eda7700 1 --2- 172.21.15.113:0/2079537896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f66480fa950 0x7f66480fcda0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:28.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.994+0000 7f664eda7700 1 --2- 172.21.15.113:0/2079537896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6648061280 0x7f66480fa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:28.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.994+0000 7f664eda7700 1 -- 172.21.15.113:0/2079537896 >> 172.21.15.113:0/2079537896 conn(0x7f66480f1e50 msgr2=0x7f66480f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:28.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.994+0000 7f664eda7700 1 -- 172.21.15.113:0/2079537896 shutdown_connections 2024-04-12T05:06:28.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.994+0000 7f664eda7700 1 -- 172.21.15.113:0/2079537896 wait complete. 2024-04-12T05:06:28.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664eda7700 1 Processor -- start 2024-04-12T05:06:28.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664eda7700 1 -- start start 2024-04-12T05:06:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664eda7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6648061280 0x7f66480f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664eda7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66480fa950 0x7f66480f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664eda7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66480f81d0 con 0x7f6648061280 2024-04-12T05:06:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664eda7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66480f67c0 con 0x7f66480fa950 2024-04-12T05:06:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664cb43700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6648061280 0x7f66480f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664cb43700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6648061280 0x7f66480f7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36138/0 (socket says 172.21.15.113:36138) 2024-04-12T05:06:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664cb43700 1 -- 172.21.15.113:0/313178044 learned_addr learned my addr 172.21.15.113:0/313178044 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f6647fff700 1 --2- 172.21.15.113:0/313178044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66480fa950 0x7f66480f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664cb43700 1 -- 172.21.15.113:0/313178044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66480fa950 msgr2=0x7f66480f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664cb43700 1 --2- 172.21.15.113:0/313178044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66480fa950 0x7f66480f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:28.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664cb43700 1 -- 172.21.15.113:0/313178044 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6634005040 con 0x7f6648061280 2024-04-12T05:06:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.995+0000 7f664cb43700 1 --2- 172.21.15.113:0/313178044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6648061280 0x7f66480f7c20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f663c0041d0 tx=0x7f663c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.996+0000 7f6645ffb700 1 -- 172.21.15.113:0/313178044 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663c007b40 con 0x7f6648061280 2024-04-12T05:06:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.996+0000 7f6645ffb700 1 -- 172.21.15.113:0/313178044 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f663c00f040 con 0x7f6648061280 2024-04-12T05:06:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.996+0000 7f664eda7700 1 -- 172.21.15.113:0/313178044 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66480f6aa0 con 0x7f6648061280 2024-04-12T05:06:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.996+0000 7f6645ffb700 1 -- 172.21.15.113:0/313178044 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663c00d760 con 0x7f6648061280 2024-04-12T05:06:29.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.996+0000 7f664eda7700 1 -- 172.21.15.113:0/313178044 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66480f6fe0 con 0x7f6648061280 2024-04-12T05:06:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.997+0000 7f664eda7700 1 -- 172.21.15.113:0/313178044 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66480fffd0 con 0x7f6648061280 2024-04-12T05:06:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.998+0000 7f6645ffb700 1 -- 172.21.15.113:0/313178044 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f663c009110 con 0x7f6648061280 2024-04-12T05:06:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.998+0000 7f6645ffb700 1 --2- 172.21.15.113:0/313178044 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f6638061180 0x7f6638063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.998+0000 7f6645ffb700 1 -- 172.21.15.113:0/313178044 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f663c07c350 con 0x7f6648061280 2024-04-12T05:06:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.998+0000 7f6647fff700 1 --2- 172.21.15.113:0/313178044 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f6638061180 0x7f6638063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:29.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:28.999+0000 7f6647fff700 1 --2- 172.21.15.113:0/313178044 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f6638061180 0x7f6638063640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6634013470 tx=0x7f663400b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:29.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.003+0000 7f6645ffb700 1 -- 172.21.15.113:0/313178044 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f663c0502e0 con 0x7f6648061280 2024-04-12T05:06:29.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.215+0000 7f664eda7700 1 -- 172.21.15.113:0/313178044 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7f6648102000 con 0x7f6648061280 2024-04-12T05:06:29.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.881+0000 7f6645ffb700 1 -- 172.21.15.113:0/313178044 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v14) v1 ==== 133+0+0 (secure 0 0 0) 0x7f663c0093c0 con 0x7f6648061280 2024-04-12T05:06:29.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.883+0000 7f664eda7700 1 -- 172.21.15.113:0/313178044 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f6638061180 msgr2=0x7f6638063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:29.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.883+0000 7f664eda7700 1 --2- 172.21.15.113:0/313178044 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f6638061180 0x7f6638063640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6634013470 tx=0x7f663400b040).stop 2024-04-12T05:06:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.883+0000 7f664eda7700 1 -- 172.21.15.113:0/313178044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6648061280 msgr2=0x7f66480f7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.883+0000 7f664eda7700 1 --2- 172.21.15.113:0/313178044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6648061280 0x7f66480f7c20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f663c0041d0 tx=0x7f663c009f60).stop 2024-04-12T05:06:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.884+0000 7f664eda7700 1 -- 172.21.15.113:0/313178044 shutdown_connections 2024-04-12T05:06:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.884+0000 7f664eda7700 1 --2- 172.21.15.113:0/313178044 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f6638061180 0x7f6638063640 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.884+0000 7f664eda7700 1 --2- 172.21.15.113:0/313178044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6648061280 0x7f66480f7c20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.884+0000 7f664eda7700 1 --2- 172.21.15.113:0/313178044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f66480fa950 0x7f66480f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.884+0000 7f664eda7700 1 -- 172.21.15.113:0/313178044 >> 172.21.15.113:0/313178044 conn(0x7f66480f1e50 msgr2=0x7f66480f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.884+0000 7f664eda7700 1 -- 172.21.15.113:0/313178044 shutdown_connections 2024-04-12T05:06:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:29.884+0000 7f664eda7700 1 -- 172.21.15.113:0/313178044 wait complete. 2024-04-12T05:06:29.887 INFO:teuthology.orchestra.run.smithi113.stderr:inline data disabled 2024-04-12T05:06:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:28.877317+0000 mon.smithi113 (mon.0) 582 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:29.216146+0000 mon.smithi113 (mon.0) 583 : audit [INF] from='client.? 172.21.15.113:0/313178044' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-12T05:06:30.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:28.877317+0000 mon.smithi113 (mon.0) 582 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:30.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:29.216146+0000 mon.smithi113 (mon.0) 583 : audit [INF] from='client.? 172.21.15.113:0/313178044' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-12T05:06:30.624 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:06:30.634 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:06:30.634 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-12T05:06:31.151 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:29.552121+0000 mgr.smithi113.ctsazg (mgr.14194) 213 : cluster [DBG] pgmap v184: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s wr, 6 op/s 2024-04-12T05:06:31.151 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:29.882328+0000 mon.smithi113 (mon.0) 584 : audit [INF] from='client.? 172.21.15.113:0/313178044' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-12T05:06:31.151 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:29.882397+0000 mon.smithi113 (mon.0) 585 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:06:31.151 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:30.689648+0000 mon.smithi113 (mon.0) 586 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:31.151 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:30.690483+0000 mon.smithi113 (mon.0) 587 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:31.152 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:30.691133+0000 mon.smithi113 (mon.0) 588 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:06:31.152 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:30.695026+0000 mon.smithi113 (mon.0) 589 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:31.152 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:30.699283+0000 mon.smithi113 (mon.0) 590 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:06:31.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:29.552121+0000 mgr.smithi113.ctsazg (mgr.14194) 213 : cluster [DBG] pgmap v184: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s wr, 6 op/s 2024-04-12T05:06:31.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:29.882328+0000 mon.smithi113 (mon.0) 584 : audit [INF] from='client.? 172.21.15.113:0/313178044' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-12T05:06:31.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:29.882397+0000 mon.smithi113 (mon.0) 585 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:06:31.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:30.689648+0000 mon.smithi113 (mon.0) 586 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:31.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:30.690483+0000 mon.smithi113 (mon.0) 587 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:30.691133+0000 mon.smithi113 (mon.0) 588 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:06:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:30.695026+0000 mon.smithi113 (mon.0) 589 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:30.699283+0000 mon.smithi113 (mon.0) 590 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:06:32.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.909+0000 7f8156c35700 1 -- 172.21.15.113:0/80244044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81500f7990 msgr2=0x7f81500f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:32.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.909+0000 7f8156c35700 1 --2- 172.21.15.113:0/80244044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81500f7990 0x7f81500f7df0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f813c0047f0 tx=0x7f813c009cb0).stop 2024-04-12T05:06:32.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.909+0000 7f8156c35700 1 -- 172.21.15.113:0/80244044 shutdown_connections 2024-04-12T05:06:32.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.909+0000 7f8156c35700 1 --2- 172.21.15.113:0/80244044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81500f7990 0x7f81500f7df0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:32.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.909+0000 7f8156c35700 1 --2- 172.21.15.113:0/80244044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81500f66f0 0x7f81500f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:32.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.909+0000 7f8156c35700 1 -- 172.21.15.113:0/80244044 >> 172.21.15.113:0/80244044 conn(0x7f81500f1e50 msgr2=0x7f81500f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:32.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.909+0000 7f8156c35700 1 -- 172.21.15.113:0/80244044 shutdown_connections 2024-04-12T05:06:32.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.909+0000 7f8156c35700 1 -- 172.21.15.113:0/80244044 wait complete. 2024-04-12T05:06:32.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f8156c35700 1 Processor -- start 2024-04-12T05:06:32.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f8156c35700 1 -- start start 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f8156c35700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81500f66f0 0x7f8150107f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f8156c35700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81500f7990 0x7f8150108440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f8156c35700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81501089f0 con 0x7f81500f66f0 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f8156c35700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8150108b30 con 0x7f81500f7990 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f81549d1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81500f66f0 0x7f8150107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f81549d1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81500f66f0 0x7f8150107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60272/0 (socket says 172.21.15.113:60272) 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f81549d1700 1 -- 172.21.15.113:0/4220336014 learned_addr learned my addr 172.21.15.113:0/4220336014 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f814ffff700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81500f7990 0x7f8150108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f81549d1700 1 -- 172.21.15.113:0/4220336014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81500f7990 msgr2=0x7f8150108440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f81549d1700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81500f7990 0x7f8150108440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f81549d1700 1 -- 172.21.15.113:0/4220336014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c005040 con 0x7f81500f66f0 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f814ffff700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81500f7990 0x7f8150108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.910+0000 7f81549d1700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81500f66f0 0x7f8150107f50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f8144002820 tx=0x7f8144009f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:32.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.911+0000 7f814dffb700 1 -- 172.21.15.113:0/4220336014 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8144002d60 con 0x7f81500f66f0 2024-04-12T05:06:32.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.911+0000 7f814dffb700 1 -- 172.21.15.113:0/4220336014 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f814400f040 con 0x7f81500f66f0 2024-04-12T05:06:32.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.911+0000 7f8156c35700 1 -- 172.21.15.113:0/4220336014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815010d5d0 con 0x7f81500f66f0 2024-04-12T05:06:32.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.911+0000 7f814dffb700 1 -- 172.21.15.113:0/4220336014 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f814400d820 con 0x7f81500f66f0 2024-04-12T05:06:32.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.911+0000 7f8156c35700 1 -- 172.21.15.113:0/4220336014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815010db10 con 0x7f81500f66f0 2024-04-12T05:06:32.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.912+0000 7f8156c35700 1 -- 172.21.15.113:0/4220336014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81500feeb0 con 0x7f81500f66f0 2024-04-12T05:06:32.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.912+0000 7f814dffb700 1 -- 172.21.15.113:0/4220336014 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f81440092f0 con 0x7f81500f66f0 2024-04-12T05:06:32.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.912+0000 7f814dffb700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8140061180 0x7f8140063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:32.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.912+0000 7f814dffb700 1 -- 172.21.15.113:0/4220336014 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f814407c510 con 0x7f81500f66f0 2024-04-12T05:06:32.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.912+0000 7f814ffff700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8140061180 0x7f8140063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:32.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.913+0000 7f814ffff700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8140061180 0x7f8140063640 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f813c013470 tx=0x7f813c00b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:32.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:32.917+0000 7f814dffb700 1 -- 172.21.15.113:0/4220336014 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8144050550 con 0x7f81500f66f0 2024-04-12T05:06:33.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.122+0000 7f8156c35700 1 -- 172.21.15.113:0/4220336014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f815010e4d0 con 0x7f81500f66f0 2024-04-12T05:06:33.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.122+0000 7f814dffb700 1 -- 172.21.15.113:0/4220336014 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1779 (secure 0 0 0) 0x7f8144050140 con 0x7f81500f66f0 2024-04-12T05:06:33.124 INFO:teuthology.orchestra.run.smithi113.stdout:e14 2024-04-12T05:06:33.124 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 14 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:06:29.875846+0000 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:06:33.125 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=14426,1=24261} 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:06:33.126 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:06:33.127 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:06:33.127 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:06:33.127 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:06:33.127 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:06:33.127 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:06:33.127 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:06:33.127 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:06:33.127 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:06:33.127 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:06:33.127 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:06:33.127 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:06:33.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.124+0000 7f8156c35700 1 -- 172.21.15.113:0/4220336014 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8140061180 msgr2=0x7f8140063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:33.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.124+0000 7f8156c35700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8140061180 0x7f8140063640 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f813c013470 tx=0x7f813c00b040).stop 2024-04-12T05:06:33.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.124+0000 7f8156c35700 1 -- 172.21.15.113:0/4220336014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81500f66f0 msgr2=0x7f8150107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:33.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.124+0000 7f8156c35700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81500f66f0 0x7f8150107f50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f8144002820 tx=0x7f8144009f00).stop 2024-04-12T05:06:33.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.124+0000 7f8156c35700 1 -- 172.21.15.113:0/4220336014 shutdown_connections 2024-04-12T05:06:33.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.124+0000 7f8156c35700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8140061180 0x7f8140063640 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:33.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.124+0000 7f8156c35700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81500f66f0 0x7f8150107f50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:33.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.124+0000 7f8156c35700 1 --2- 172.21.15.113:0/4220336014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81500f7990 0x7f8150108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:33.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.124+0000 7f8156c35700 1 -- 172.21.15.113:0/4220336014 >> 172.21.15.113:0/4220336014 conn(0x7f81500f1e50 msgr2=0x7f8150060d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:33.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.124+0000 7f8156c35700 1 -- 172.21.15.113:0/4220336014 shutdown_connections 2024-04-12T05:06:33.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:33.124+0000 7f8156c35700 1 -- 172.21.15.113:0/4220336014 wait complete. 2024-04-12T05:06:33.129 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 14 2024-04-12T05:06:33.145 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:31.552501+0000 mgr.smithi113.ctsazg (mgr.14194) 214 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-12T05:06:33.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:31.552501+0000 mgr.smithi113.ctsazg (mgr.14194) 214 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-12T05:06:34.015 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:33.123240+0000 mon.smithi113 (mon.0) 591 : audit [DBG] from='client.? 172.21.15.113:0/4220336014' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:06:34.015 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:33.624350+0000 mon.smithi113 (mon.0) 592 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:34.034 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-12T05:06:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:33.123240+0000 mon.smithi113 (mon.0) 591 : audit [DBG] from='client.? 172.21.15.113:0/4220336014' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:06:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:33.624350+0000 mon.smithi113 (mon.0) 592 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:33.552970+0000 mgr.smithi113.ctsazg (mgr.14194) 215 : cluster [DBG] pgmap v186: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 1 op/s 2024-04-12T05:06:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:33.552970+0000 mgr.smithi113.ctsazg (mgr.14194) 215 : cluster [DBG] pgmap v186: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 1 op/s 2024-04-12T05:06:36.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.683+0000 7f8483f19700 1 -- 172.21.15.113:0/3181265927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f847c0f6ce0 msgr2=0x7f847c0f7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:36.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.683+0000 7f8483f19700 1 --2- 172.21.15.113:0/3181265927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f847c0f6ce0 0x7f847c0f7100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f84780047f0 tx=0x7f8478009cb0).stop 2024-04-12T05:06:36.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.683+0000 7f8483f19700 1 -- 172.21.15.113:0/3181265927 shutdown_connections 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.683+0000 7f8483f19700 1 --2- 172.21.15.113:0/3181265927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f847c0f7680 0x7f847c0f7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.683+0000 7f8483f19700 1 --2- 172.21.15.113:0/3181265927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f847c0f6ce0 0x7f847c0f7100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.683+0000 7f8483f19700 1 -- 172.21.15.113:0/3181265927 >> 172.21.15.113:0/3181265927 conn(0x7f847c0f1610 msgr2=0x7f847c0f3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.684+0000 7f8483f19700 1 -- 172.21.15.113:0/3181265927 shutdown_connections 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.684+0000 7f8483f19700 1 -- 172.21.15.113:0/3181265927 wait complete. 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.684+0000 7f8483f19700 1 Processor -- start 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.684+0000 7f8483f19700 1 -- start start 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.684+0000 7f8483f19700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f847c0f6ce0 0x7f847c10a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.684+0000 7f8483f19700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f847c0f7680 0x7f847c10a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.684+0000 7f8483f19700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c10ac40 con 0x7f847c0f7680 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.684+0000 7f8483f19700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c10ad80 con 0x7f847c0f6ce0 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.684+0000 7f84814b4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f847c0f7680 0x7f847c10a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.684+0000 7f84814b4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f847c0f7680 0x7f847c10a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60294/0 (socket says 172.21.15.113:60294) 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.684+0000 7f84814b4700 1 -- 172.21.15.113:0/3632795420 learned_addr learned my addr 172.21.15.113:0/3632795420 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.685+0000 7f8481cb5700 1 --2- 172.21.15.113:0/3632795420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f847c0f6ce0 0x7f847c10a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.685+0000 7f84814b4700 1 -- 172.21.15.113:0/3632795420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f847c0f6ce0 msgr2=0x7f847c10a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.685+0000 7f84814b4700 1 --2- 172.21.15.113:0/3632795420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f847c0f6ce0 0x7f847c10a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.685+0000 7f84814b4700 1 -- 172.21.15.113:0/3632795420 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8478005040 con 0x7f847c0f7680 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.685+0000 7f84814b4700 1 --2- 172.21.15.113:0/3632795420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f847c0f7680 0x7f847c10a690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f846c00c740 tx=0x7f846c007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.685+0000 7f8472ffd700 1 -- 172.21.15.113:0/3632795420 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f846c00e680 con 0x7f847c0f7680 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.685+0000 7f8483f19700 1 -- 172.21.15.113:0/3632795420 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c10f820 con 0x7f847c0f7680 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.685+0000 7f8472ffd700 1 -- 172.21.15.113:0/3632795420 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f846c00ec70 con 0x7f847c0f7680 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.685+0000 7f8472ffd700 1 -- 172.21.15.113:0/3632795420 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f846c013d00 con 0x7f847c0f7680 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.685+0000 7f8483f19700 1 -- 172.21.15.113:0/3632795420 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c10fd60 con 0x7f847c0f7680 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.686+0000 7f8483f19700 1 -- 172.21.15.113:0/3632795420 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c0feee0 con 0x7f847c0f7680 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.686+0000 7f8472ffd700 1 -- 172.21.15.113:0/3632795420 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f846c00e7e0 con 0x7f847c0f7680 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.687+0000 7f8472ffd700 1 --2- 172.21.15.113:0/3632795420 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8468061180 0x7f8468063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.687+0000 7f8472ffd700 1 -- 172.21.15.113:0/3632795420 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f846c07cb90 con 0x7f847c0f7680 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.687+0000 7f8481cb5700 1 --2- 172.21.15.113:0/3632795420 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8468061180 0x7f8468063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:36.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.687+0000 7f8481cb5700 1 --2- 172.21.15.113:0/3632795420 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8468061180 0x7f8468063640 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f8478004f10 tx=0x7f847800b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:36.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.691+0000 7f8472ffd700 1 -- 172.21.15.113:0/3632795420 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f846c050b20 con 0x7f847c0f7680 2024-04-12T05:06:36.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.896+0000 7f8483f19700 1 -- 172.21.15.113:0/3632795420 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f847c110400 con 0x7f847c0f7680 2024-04-12T05:06:36.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.898+0000 7f8472ffd700 1 -- 172.21.15.113:0/3632795420 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4602 (secure 0 0 0) 0x7f846c050710 con 0x7f847c0f7680 2024-04-12T05:06:36.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.900+0000 7f8483f19700 1 -- 172.21.15.113:0/3632795420 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8468061180 msgr2=0x7f8468063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:36.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.900+0000 7f8483f19700 1 --2- 172.21.15.113:0/3632795420 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8468061180 0x7f8468063640 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f8478004f10 tx=0x7f847800b040).stop 2024-04-12T05:06:36.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.900+0000 7f8483f19700 1 -- 172.21.15.113:0/3632795420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f847c0f7680 msgr2=0x7f847c10a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:36.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.900+0000 7f8483f19700 1 --2- 172.21.15.113:0/3632795420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f847c0f7680 0x7f847c10a690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f846c00c740 tx=0x7f846c007ef0).stop 2024-04-12T05:06:36.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.900+0000 7f8483f19700 1 -- 172.21.15.113:0/3632795420 shutdown_connections 2024-04-12T05:06:36.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.900+0000 7f8483f19700 1 --2- 172.21.15.113:0/3632795420 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f8468061180 0x7f8468063640 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:36.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.900+0000 7f8483f19700 1 --2- 172.21.15.113:0/3632795420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f847c0f6ce0 0x7f847c10a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:36.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.900+0000 7f8483f19700 1 --2- 172.21.15.113:0/3632795420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f847c0f7680 0x7f847c10a690 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:36.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.900+0000 7f8483f19700 1 -- 172.21.15.113:0/3632795420 >> 172.21.15.113:0/3632795420 conn(0x7f847c0f1610 msgr2=0x7f847c0f6230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:36.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.900+0000 7f8483f19700 1 -- 172.21.15.113:0/3632795420 shutdown_connections 2024-04-12T05:06:36.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:36.900+0000 7f8483f19700 1 -- 172.21.15.113:0/3632795420 wait complete. 2024-04-12T05:06:36.903 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 14 2024-04-12T05:06:36.912 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:06:37.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:35.553361+0000 mgr.smithi113.ctsazg (mgr.14194) 216 : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-12T05:06:37.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:35.553361+0000 mgr.smithi113.ctsazg (mgr.14194) 216 : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-12T05:06:37.721 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-12T05:06:38.168 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:36.899440+0000 mon.smithi113 (mon.0) 593 : audit [DBG] from='client.? 172.21.15.113:0/3632795420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T05:06:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:36.899440+0000 mon.smithi113 (mon.0) 593 : audit [DBG] from='client.? 172.21.15.113:0/3632795420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T05:06:39.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:37.553688+0000 mgr.smithi113.ctsazg (mgr.14194) 217 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:39.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:37.553688+0000 mgr.smithi113.ctsazg (mgr.14194) 217 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:40.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.169+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/270582134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c0f66f0 msgr2=0x7f3f9c0f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:40.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.169+0000 7f3fa3f8d700 1 --2- 172.21.15.113:0/270582134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c0f66f0 0x7f3f9c0f6b10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f3f980047f0 tx=0x7f3f98009cb0).stop 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.169+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/270582134 shutdown_connections 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.169+0000 7f3fa3f8d700 1 --2- 172.21.15.113:0/270582134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f9c0f7990 0x7f3f9c0f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.169+0000 7f3fa3f8d700 1 --2- 172.21.15.113:0/270582134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c0f66f0 0x7f3f9c0f6b10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.169+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/270582134 >> 172.21.15.113:0/270582134 conn(0x7f3f9c0f1e50 msgr2=0x7f3f9c0f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.169+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/270582134 shutdown_connections 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.170+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/270582134 wait complete. 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.170+0000 7f3fa3f8d700 1 Processor -- start 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.170+0000 7f3fa3f8d700 1 -- start start 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.170+0000 7f3fa3f8d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c0f66f0 0x7f3f9c106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.170+0000 7f3fa3f8d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f9c0f7990 0x7f3f9c107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.170+0000 7f3fa3f8d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c107930 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.170+0000 7f3fa3f8d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c107a70 con 0x7f3f9c0f7990 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3fa1d29700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c0f66f0 0x7f3f9c106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:40.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3fa1d29700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c0f66f0 0x7f3f9c106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60300/0 (socket says 172.21.15.113:60300) 2024-04-12T05:06:40.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3fa1d29700 1 -- 172.21.15.113:0/3834727911 learned_addr learned my addr 172.21.15.113:0/3834727911 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:40.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3fa1528700 1 --2- 172.21.15.113:0/3834727911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f9c0f7990 0x7f3f9c107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:40.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3fa1d29700 1 -- 172.21.15.113:0/3834727911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f9c0f7990 msgr2=0x7f3f9c107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:40.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3fa1d29700 1 --2- 172.21.15.113:0/3834727911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f9c0f7990 0x7f3f9c107380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:40.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3fa1d29700 1 -- 172.21.15.113:0/3834727911 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f98005040 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3fa1d29700 1 --2- 172.21.15.113:0/3834727911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c0f66f0 0x7f3f9c106e90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f3f98004f10 tx=0x7f3f98009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:40.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3f92ffd700 1 -- 172.21.15.113:0/3834727911 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f9800e070 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/3834727911 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9c10c4b0 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3f92ffd700 1 -- 172.21.15.113:0/3834727911 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f3f98007e90 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3f92ffd700 1 -- 172.21.15.113:0/3834727911 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f98018070 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.171+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/3834727911 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9c10c9f0 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.172+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/3834727911 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9c0feeb0 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.173+0000 7f3f92ffd700 1 -- 172.21.15.113:0/3834727911 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f3f980123c0 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.173+0000 7f3f92ffd700 1 --2- 172.21.15.113:0/3834727911 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3f88061160 0x7f3f88063620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:40.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.173+0000 7f3f92ffd700 1 -- 172.21.15.113:0/3834727911 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f3f9807d740 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.173+0000 7f3fa1528700 1 --2- 172.21.15.113:0/3834727911 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3f88061160 0x7f3f88063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:40.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.174+0000 7f3fa1528700 1 --2- 172.21.15.113:0/3834727911 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3f88061160 0x7f3f88063620 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f3f8c004dc0 tx=0x7f3f8c00b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:40.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.178+0000 7f3f92ffd700 1 -- 172.21.15.113:0/3834727911 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3f98051610 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.412+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/3834727911 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f3f9c0598f0 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.413+0000 7f3f92ffd700 1 -- 172.21.15.113:0/3834727911 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v14) v1 ==== 78+0+99 (secure 0 0 0) 0x7f3f98051200 con 0x7f3f9c0f66f0 2024-04-12T05:06:40.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.414+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/3834727911 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3f88061160 msgr2=0x7f3f88063620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:40.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.414+0000 7f3fa3f8d700 1 --2- 172.21.15.113:0/3834727911 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3f88061160 0x7f3f88063620 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f3f8c004dc0 tx=0x7f3f8c00b040).stop 2024-04-12T05:06:40.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.414+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/3834727911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c0f66f0 msgr2=0x7f3f9c106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:40.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.414+0000 7f3fa3f8d700 1 --2- 172.21.15.113:0/3834727911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c0f66f0 0x7f3f9c106e90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f3f98004f10 tx=0x7f3f98009cb0).stop 2024-04-12T05:06:40.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.415+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/3834727911 shutdown_connections 2024-04-12T05:06:40.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.415+0000 7f3fa3f8d700 1 --2- 172.21.15.113:0/3834727911 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f3f88061160 0x7f3f88063620 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:40.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.415+0000 7f3fa3f8d700 1 --2- 172.21.15.113:0/3834727911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c0f66f0 0x7f3f9c106e90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:40.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.415+0000 7f3fa3f8d700 1 --2- 172.21.15.113:0/3834727911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f9c0f7990 0x7f3f9c107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:40.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.415+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/3834727911 >> 172.21.15.113:0/3834727911 conn(0x7f3f9c0f1e50 msgr2=0x7f3f9c0fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:40.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.415+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/3834727911 shutdown_connections 2024-04-12T05:06:40.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:40.415+0000 7f3fa3f8d700 1 -- 172.21.15.113:0/3834727911 wait complete. 2024-04-12T05:06:40.426 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:06:41.167 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-12T05:06:41.193 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T05:06:41.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.372+0000 7f081b4f1700 1 -- 172.21.15.113:0/3316418454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08140f77e0 msgr2=0x7f08140f7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:41.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.372+0000 7f081b4f1700 1 --2- 172.21.15.113:0/3316418454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08140f77e0 0x7f08140f7c00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f08040047f0 tx=0x7f0804009cb0).stop 2024-04-12T05:06:41.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.372+0000 7f081b4f1700 1 -- 172.21.15.113:0/3316418454 shutdown_connections 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.372+0000 7f081b4f1700 1 --2- 172.21.15.113:0/3316418454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140f8a80 0x7f08140f8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.372+0000 7f081b4f1700 1 --2- 172.21.15.113:0/3316418454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08140f77e0 0x7f08140f7c00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.372+0000 7f081b4f1700 1 -- 172.21.15.113:0/3316418454 >> 172.21.15.113:0/3316418454 conn(0x7f08140f2f20 msgr2=0x7f08140f5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.372+0000 7f081b4f1700 1 -- 172.21.15.113:0/3316418454 shutdown_connections 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.372+0000 7f081b4f1700 1 -- 172.21.15.113:0/3316418454 wait complete. 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f081b4f1700 1 Processor -- start 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f081b4f1700 1 -- start start 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f081b4f1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140f77e0 0x7f0814107f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f081b4f1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08140f8a80 0x7f0814108410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f081b4f1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08141089c0 con 0x7f08140f8a80 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f081b4f1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814108b00 con 0x7f08140f77e0 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f0818a8c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08140f8a80 0x7f0814108410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f0818a8c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08140f8a80 0x7f0814108410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55924/0 (socket says 172.21.15.113:55924) 2024-04-12T05:06:41.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f0818a8c700 1 -- 172.21.15.113:0/861055300 learned_addr learned my addr 172.21.15.113:0/861055300 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f081928d700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140f77e0 0x7f0814107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f0818a8c700 1 -- 172.21.15.113:0/861055300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140f77e0 msgr2=0x7f0814107f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f0818a8c700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140f77e0 0x7f0814107f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f0818a8c700 1 -- 172.21.15.113:0/861055300 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0804005040 con 0x7f08140f8a80 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f081928d700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140f77e0 0x7f0814107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f0818a8c700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08140f8a80 0x7f0814108410 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f081000c740 tx=0x7f0810007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.373+0000 7f080a7fc700 1 -- 172.21.15.113:0/861055300 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f081000e680 con 0x7f08140f8a80 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.374+0000 7f080a7fc700 1 -- 172.21.15.113:0/861055300 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f081000ec70 con 0x7f08140f8a80 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.374+0000 7f081b4f1700 1 -- 172.21.15.113:0/861055300 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081410d5a0 con 0x7f08140f8a80 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.374+0000 7f080a7fc700 1 -- 172.21.15.113:0/861055300 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0810013d30 con 0x7f08140f8a80 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.374+0000 7f081b4f1700 1 -- 172.21.15.113:0/861055300 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081410dae0 con 0x7f08140f8a80 2024-04-12T05:06:41.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.375+0000 7f081b4f1700 1 -- 172.21.15.113:0/861055300 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08140fffa0 con 0x7f08140f8a80 2024-04-12T05:06:41.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.375+0000 7f080a7fc700 1 -- 172.21.15.113:0/861055300 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f081000e7e0 con 0x7f08140f8a80 2024-04-12T05:06:41.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.375+0000 7f080a7fc700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f08000611d0 0x7f0800063690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:41.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.375+0000 7f080a7fc700 1 -- 172.21.15.113:0/861055300 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f081007cba0 con 0x7f08140f8a80 2024-04-12T05:06:41.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.375+0000 7f081928d700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f08000611d0 0x7f0800063690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:41.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.376+0000 7f081928d700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f08000611d0 0x7f0800063690 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f0804013470 tx=0x7f080400b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:41.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.379+0000 7f080a7fc700 1 -- 172.21.15.113:0/861055300 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0810050af0 con 0x7f08140f8a80 2024-04-12T05:06:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:39.554093+0000 mgr.smithi113.ctsazg (mgr.14194) 218 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:40.239343+0000 mon.smithi113 (mon.0) 594 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:40.240162+0000 mon.smithi113 (mon.0) 595 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:40.240749+0000 mon.smithi113 (mon.0) 596 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:06:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:40.246645+0000 mon.smithi113 (mon.0) 597 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:40.413855+0000 mon.smithi113 (mon.0) 598 : audit [DBG] from='client.? 172.21.15.113:0/3834727911' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-12T05:06:41.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.542+0000 7f081b4f1700 1 -- 172.21.15.113:0/861055300 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0814101e10 con 0x7f08140f8a80 2024-04-12T05:06:41.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.543+0000 7f080a7fc700 1 -- 172.21.15.113:0/861055300 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+14138 (secure 0 0 0) 0x7f08100506e0 con 0x7f08140f8a80 2024-04-12T05:06:41.545 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:06:41.545 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":43,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","created":"2024-04-12T04:59:30.005034+0000","modified":"2024-04-12T05:06:14.149022+0000","last_up_change":"2024-04-12T05:04:50.786863+0000","last_in_change":"2024-04-12T05:04:31.248597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-04-12T05:03:31.839630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-12T05:06:11.000491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-12T05:06:11.375495+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fc942a17-378a-48a6-b1d3-4f5730550810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6803","nonce":1323860970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6805","nonce":1323860970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6809","nonce":1323860970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1323860970},{"type":"v1","addr":"172.21.15.113:6807","nonce":1323860970}]},"public_addr":"172.21.15.113:6803/1323860970","cluster_addr":"172.21.15.113:6805/1323860970","heartbeat_back_addr":"172.21.15.113:6809/1323860970","heartbeat_front_addr":"172.21.15.113:6807/1323860970","state":["exists","up"]},{"osd":1,"uuid":"39fec899-371b-4d48-99a8-52f90f52a71f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6811","nonce":563293368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6813","nonce":563293368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6817","nonce":563293368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":563293368},{"type":"v1","addr":"172.21.15.113:6815","nonce":563293368}]},"public_addr":"172.21.15.113:6811/563293368","cluster_addr":"172.21.15.113:6813/563293368","heartbeat_back_addr":"172.21.15.113:6817/563293368","heartbeat_front_addr":"172.21.15.113:6815/563293368","state":["exists","up"]},{"osd":2,"uuid":"b135f3a9-d2f0-4737-8a30-9f6dc3f2571b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6819","nonce":2998637711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6821","nonce":2998637711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6825","nonce":2998637711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2998637711},{"type":"v1","addr":"172.21.15.113:6823","nonce":2998637711}]},"public_addr":"172.21.15.113:6819/2998637711","cluster_addr":"172.21.15.113:6821/2998637711","heartbeat_back_addr":"172.21.15.113:6825/2998637711","heartbeat_front_addr":"172.21.15.113:6823/2998637711","state":["exists","up"]},{"osd":3,"uuid":"49fd856c-c351-4e90-9088-ece0024cd2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6801","nonce":821756207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6803","nonce":821756207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6807","nonce":821756207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":821756207},{"type":"v1","addr":"172.21.15.192:6805","nonce":821756207}]},"public_addr":"172.21.15.192:6801/821756207","cluster_addr":"172.21.15.192:6803/821756207","heartbeat_back_addr":"172.21.15.192:6807/821756207","heartbeat_front_addr":"172.21.15.192:6805/821756207","state":["exists","up"]},{"osd":4,"uuid":"80b72db8-682a-4a51-8d33-b0addda91fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6809","nonce":563933248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6811","nonce":563933248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6815","nonce":563933248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":563933248},{"type":"v1","addr":"172.21.15.192:6813","nonce":563933248}]},"public_addr":"172.21.15.192:6809/563933248","cluster_addr":"172.21.15.192:6811/563933248","heartbeat_back_addr":"172.21.15.192:6815/563933248","heartbeat_front_addr":"172.21.15.192:6813/563933248","state":["exists","up"]},{"osd":5,"uuid":"c8e1e98f-5edb-4fec-bdf6-bcda40562682","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6817","nonce":2433186323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6819","nonce":2433186323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6823","nonce":2433186323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2433186323},{"type":"v1","addr":"172.21.15.192:6821","nonce":2433186323}]},"public_addr":"172.21.15.192:6817/2433186323","cluster_addr":"172.21.15.192:6819/2433186323","heartbeat_back_addr":"172.21.15.192:6823/2433186323","heartbeat_front_addr":"172.21.15.192:6821/2433186323","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:02:27.514514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:01.358930+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:30.203057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:54.928612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:21.788000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:48.830652+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.113:6801/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.113:6800/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/516367458":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/3213888755":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/4242055626":"2024-04-13T05:00:29.828900+0000","172.21.15.113:6801/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:6800/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/2902584436":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/1777682971":"2024-04-13T04:59:57.862720+0000","172.21.15.113:6800/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.113:6801/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.113:0/3450096338":"2024-04-13T04:59:57.862720+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T05:06:41.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.545+0000 7f081b4f1700 1 -- 172.21.15.113:0/861055300 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f08000611d0 msgr2=0x7f0800063690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:41.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.545+0000 7f081b4f1700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f08000611d0 0x7f0800063690 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f0804013470 tx=0x7f080400b040).stop 2024-04-12T05:06:41.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.545+0000 7f081b4f1700 1 -- 172.21.15.113:0/861055300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08140f8a80 msgr2=0x7f0814108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:41.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.545+0000 7f081b4f1700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08140f8a80 0x7f0814108410 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f081000c740 tx=0x7f0810007ef0).stop 2024-04-12T05:06:41.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.545+0000 7f081b4f1700 1 -- 172.21.15.113:0/861055300 shutdown_connections 2024-04-12T05:06:41.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.545+0000 7f081b4f1700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f08000611d0 0x7f0800063690 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.545+0000 7f081b4f1700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08140f77e0 0x7f0814107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.545+0000 7f081b4f1700 1 --2- 172.21.15.113:0/861055300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08140f8a80 0x7f0814108410 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.545+0000 7f081b4f1700 1 -- 172.21.15.113:0/861055300 >> 172.21.15.113:0/861055300 conn(0x7f08140f2f20 msgr2=0x7f08140fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:41.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.545+0000 7f081b4f1700 1 -- 172.21.15.113:0/861055300 shutdown_connections 2024-04-12T05:06:41.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.545+0000 7f081b4f1700 1 -- 172.21.15.113:0/861055300 wait complete. 2024-04-12T05:06:41.559 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-04-12T05:03:31.839630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-12T05:06:11.000491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-12T05:06:11.375495+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-12T05:06:41.559 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2024-04-12T05:06:41.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.676+0000 7f00fe06d700 1 -- 172.21.15.113:0/3614370090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00f80612a0 msgr2=0x7f00f80fa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:41.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.676+0000 7f00fe06d700 1 --2- 172.21.15.113:0/3614370090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00f80612a0 0x7f00f80fa3b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f00ec0047f0 tx=0x7f00ec009cb0).stop 2024-04-12T05:06:41.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.676+0000 7f00fe06d700 1 -- 172.21.15.113:0/3614370090 shutdown_connections 2024-04-12T05:06:41.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.676+0000 7f00fe06d700 1 --2- 172.21.15.113:0/3614370090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f80fa930 0x7f00f80fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.676+0000 7f00fe06d700 1 --2- 172.21.15.113:0/3614370090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00f80612a0 0x7f00f80fa3b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.676+0000 7f00fe06d700 1 -- 172.21.15.113:0/3614370090 >> 172.21.15.113:0/3614370090 conn(0x7f00f80f1df0 msgr2=0x7f00f80f4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:41.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.676+0000 7f00fe06d700 1 -- 172.21.15.113:0/3614370090 shutdown_connections 2024-04-12T05:06:41.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.676+0000 7f00fe06d700 1 -- 172.21.15.113:0/3614370090 wait complete. 2024-04-12T05:06:41.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00fe06d700 1 Processor -- start 2024-04-12T05:06:41.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00fe06d700 1 -- start start 2024-04-12T05:06:41.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00fe06d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f80612a0 0x7f00f8107f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:41.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00fe06d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00f80fa930 0x7f00f8108430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00fe06d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f81089e0 con 0x7f00f80fa930 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00fe06d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8108b20 con 0x7f00f80612a0 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00f6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00f80fa930 0x7f00f8108430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00f6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00f80fa930 0x7f00f8108430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55942/0 (socket says 172.21.15.113:55942) 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00f6ffd700 1 -- 172.21.15.113:0/257108083 learned_addr learned my addr 172.21.15.113:0/257108083 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00f77fe700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f80612a0 0x7f00f8107f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00f6ffd700 1 -- 172.21.15.113:0/257108083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f80612a0 msgr2=0x7f00f8107f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00f6ffd700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f80612a0 0x7f00f8107f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00f6ffd700 1 -- 172.21.15.113:0/257108083 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec005040 con 0x7f00f80fa930 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00f77fe700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f80612a0 0x7f00f8107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.677+0000 7f00f6ffd700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00f80fa930 0x7f00f8108430 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f00e800c710 tx=0x7f00e8007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.678+0000 7f00f4ff9700 1 -- 172.21.15.113:0/257108083 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00e800e680 con 0x7f00f80fa930 2024-04-12T05:06:41.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.678+0000 7f00f4ff9700 1 -- 172.21.15.113:0/257108083 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f00e800ec70 con 0x7f00f80fa930 2024-04-12T05:06:41.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.678+0000 7f00fe06d700 1 -- 172.21.15.113:0/257108083 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f810d5c0 con 0x7f00f80fa930 2024-04-12T05:06:41.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.678+0000 7f00f4ff9700 1 -- 172.21.15.113:0/257108083 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00e8013ca0 con 0x7f00f80fa930 2024-04-12T05:06:41.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.678+0000 7f00fe06d700 1 -- 172.21.15.113:0/257108083 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f810db00 con 0x7f00f80fa930 2024-04-12T05:06:41.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.679+0000 7f00fe06d700 1 -- 172.21.15.113:0/257108083 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f80fffc0 con 0x7f00f80fa930 2024-04-12T05:06:41.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.679+0000 7f00f4ff9700 1 -- 172.21.15.113:0/257108083 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f00e800e7e0 con 0x7f00f80fa930 2024-04-12T05:06:41.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.679+0000 7f00f4ff9700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f00e0061180 0x7f00e0063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:41.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.679+0000 7f00f4ff9700 1 -- 172.21.15.113:0/257108083 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f00e807cbc0 con 0x7f00f80fa930 2024-04-12T05:06:41.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.679+0000 7f00f77fe700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f00e0061180 0x7f00e0063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:41.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.680+0000 7f00f77fe700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f00e0061180 0x7f00e0063640 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f00ec013470 tx=0x7f00ec00b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:41.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.684+0000 7f00f4ff9700 1 -- 172.21.15.113:0/257108083 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f00e8050b50 con 0x7f00f80fa930 2024-04-12T05:06:41.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:39.554093+0000 mgr.smithi113.ctsazg (mgr.14194) 218 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:41.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:40.239343+0000 mon.smithi113 (mon.0) 594 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:41.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:40.240162+0000 mon.smithi113 (mon.0) 595 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:06:41.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:40.240749+0000 mon.smithi113 (mon.0) 596 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:06:41.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:40.246645+0000 mon.smithi113 (mon.0) 597 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:06:41.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:40.413855+0000 mon.smithi113 (mon.0) 598 : audit [DBG] from='client.? 172.21.15.113:0/3834727911' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-12T05:06:41.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.845+0000 7f00fe06d700 1 -- 172.21.15.113:0/257108083 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f00f810d750 con 0x7f00f80fa930 2024-04-12T05:06:41.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.845+0000 7f00f4ff9700 1 -- 172.21.15.113:0/257108083 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v43) v1 ==== 110+0+10 (secure 0 0 0) 0x7f00f810d750 con 0x7f00f80fa930 2024-04-12T05:06:41.847 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-04-12T05:06:41.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.847+0000 7f00fe06d700 1 -- 172.21.15.113:0/257108083 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f00e0061180 msgr2=0x7f00e0063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:41.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.847+0000 7f00fe06d700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f00e0061180 0x7f00e0063640 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f00ec013470 tx=0x7f00ec00b040).stop 2024-04-12T05:06:41.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.847+0000 7f00fe06d700 1 -- 172.21.15.113:0/257108083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00f80fa930 msgr2=0x7f00f8108430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:41.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.847+0000 7f00fe06d700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00f80fa930 0x7f00f8108430 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f00e800c710 tx=0x7f00e8007ec0).stop 2024-04-12T05:06:41.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.847+0000 7f00fe06d700 1 -- 172.21.15.113:0/257108083 shutdown_connections 2024-04-12T05:06:41.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.847+0000 7f00fe06d700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f00e0061180 0x7f00e0063640 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.847+0000 7f00fe06d700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00f80612a0 0x7f00f8107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.847+0000 7f00fe06d700 1 --2- 172.21.15.113:0/257108083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00f80fa930 0x7f00f8108430 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.847+0000 7f00fe06d700 1 -- 172.21.15.113:0/257108083 >> 172.21.15.113:0/257108083 conn(0x7f00f80f1df0 msgr2=0x7f00f80f4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:41.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.847+0000 7f00fe06d700 1 -- 172.21.15.113:0/257108083 shutdown_connections 2024-04-12T05:06:41.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.847+0000 7f00fe06d700 1 -- 172.21.15.113:0/257108083 wait complete. 2024-04-12T05:06:41.862 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-12T05:06:41.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.978+0000 7fc515edf700 1 -- 172.21.15.113:0/2167086049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5100fa940 msgr2=0x7fc5100fcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:41.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.978+0000 7fc515edf700 1 --2- 172.21.15.113:0/2167086049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5100fa940 0x7fc5100fcd90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fc5000047f0 tx=0x7fc500009cb0).stop 2024-04-12T05:06:41.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.978+0000 7fc515edf700 1 -- 172.21.15.113:0/2167086049 shutdown_connections 2024-04-12T05:06:41.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.978+0000 7fc515edf700 1 --2- 172.21.15.113:0/2167086049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5100fa940 0x7fc5100fcd90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:41.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.978+0000 7fc515edf700 1 --2- 172.21.15.113:0/2167086049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc510061200 0x7fc5100fa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:06:41.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.978+0000 7fc515edf700 1 -- 172.21.15.113:0/2167086049 >> 172.21.15.113:0/2167086049 conn(0x7fc5100f1df0 msgr2=0x7fc5100f4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:41.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.978+0000 7fc515edf700 1 -- 172.21.15.113:0/2167086049 shutdown_connections 2024-04-12T05:06:41.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.978+0000 7fc515edf700 1 -- 172.21.15.113:0/2167086049 wait complete. 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.978+0000 7fc515edf700 1 Processor -- start 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc515edf700 1 -- start start 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc515edf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc510061200 0x7fc510107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc515edf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5100fa940 0x7fc5101084d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc515edf700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc510108b10 con 0x7fc5100fa940 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc515edf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc510108c50 con 0x7fc510061200 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc50effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5100fa940 0x7fc5101084d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc50effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5100fa940 0x7fc5101084d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55954/0 (socket says 172.21.15.113:55954) 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc50effd700 1 -- 172.21.15.113:0/2077364055 learned_addr learned my addr 172.21.15.113:0/2077364055 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc50effd700 1 -- 172.21.15.113:0/2077364055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc510061200 msgr2=0x7fc510107fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc50effd700 1 --2- 172.21.15.113:0/2077364055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc510061200 0x7fc510107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:06:41.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc50effd700 1 -- 172.21.15.113:0/2077364055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc500005040 con 0x7fc5100fa940 2024-04-12T05:06:41.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc50effd700 1 --2- 172.21.15.113:0/2077364055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5100fa940 0x7fc5101084d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fc500013470 tx=0x7fc500009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:41.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc50cff9700 1 -- 172.21.15.113:0/2077364055 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc50000e030 con 0x7fc5100fa940 2024-04-12T05:06:41.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc515edf700 1 -- 172.21.15.113:0/2077364055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51010d690 con 0x7fc5100fa940 2024-04-12T05:06:41.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc50cff9700 1 -- 172.21.15.113:0/2077364055 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fc500007e90 con 0x7fc5100fa940 2024-04-12T05:06:41.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc50cff9700 1 -- 172.21.15.113:0/2077364055 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc500018070 con 0x7fc5100fa940 2024-04-12T05:06:41.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.979+0000 7fc515edf700 1 -- 172.21.15.113:0/2077364055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51010db70 con 0x7fc5100fa940 2024-04-12T05:06:41.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.980+0000 7fc515edf700 1 -- 172.21.15.113:0/2077364055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc510100020 con 0x7fc5100fa940 2024-04-12T05:06:41.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.981+0000 7fc50cff9700 1 -- 172.21.15.113:0/2077364055 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fc50001d9b0 con 0x7fc5100fa940 2024-04-12T05:06:41.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.981+0000 7fc50cff9700 1 --2- 172.21.15.113:0/2077364055 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc4f8061160 0x7fc4f8063620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:41.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.981+0000 7fc50cff9700 1 -- 172.21.15.113:0/2077364055 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7fc50007e010 con 0x7fc5100fa940 2024-04-12T05:06:41.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.981+0000 7fc50f7fe700 1 --2- 172.21.15.113:0/2077364055 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc4f8061160 0x7fc4f8063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:41.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.981+0000 7fc50f7fe700 1 --2- 172.21.15.113:0/2077364055 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc4f8061160 0x7fc4f8063620 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fc504009700 tx=0x7fc504006c30).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:41.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:41.985+0000 7fc50cff9700 1 -- 172.21.15.113:0/2077364055 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc500051fa0 con 0x7fc5100fa940 2024-04-12T05:06:42.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.148+0000 7fc515edf700 1 -- 172.21.15.113:0/2077364055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fc5100598f0 con 0x7fc5100fa940 2024-04-12T05:06:42.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.148+0000 7fc50cff9700 1 -- 172.21.15.113:0/2077364055 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc500051b90 con 0x7fc5100fa940 2024-04-12T05:06:42.151 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-04-12T05:06:42.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.150+0000 7fc515edf700 1 -- 172.21.15.113:0/2077364055 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc4f8061160 msgr2=0x7fc4f8063620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:42.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.150+0000 7fc515edf700 1 --2- 172.21.15.113:0/2077364055 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc4f8061160 0x7fc4f8063620 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fc504009700 tx=0x7fc504006c30).stop 2024-04-12T05:06:42.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.150+0000 7fc515edf700 1 -- 172.21.15.113:0/2077364055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5100fa940 msgr2=0x7fc5101084d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:42.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.150+0000 7fc515edf700 1 --2- 172.21.15.113:0/2077364055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5100fa940 0x7fc5101084d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fc500013470 tx=0x7fc500009cb0).stop 2024-04-12T05:06:42.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.151+0000 7fc515edf700 1 -- 172.21.15.113:0/2077364055 shutdown_connections 2024-04-12T05:06:42.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.151+0000 7fc515edf700 1 --2- 172.21.15.113:0/2077364055 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fc4f8061160 0x7fc4f8063620 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.151+0000 7fc515edf700 1 --2- 172.21.15.113:0/2077364055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc510061200 0x7fc510107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:06:42.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.151+0000 7fc515edf700 1 --2- 172.21.15.113:0/2077364055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5100fa940 0x7fc5101084d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.151+0000 7fc515edf700 1 -- 172.21.15.113:0/2077364055 >> 172.21.15.113:0/2077364055 conn(0x7fc5100f1df0 msgr2=0x7fc5100f4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:42.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.151+0000 7fc515edf700 1 -- 172.21.15.113:0/2077364055 shutdown_connections 2024-04-12T05:06:42.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.151+0000 7fc515edf700 1 -- 172.21.15.113:0/2077364055 wait complete. 2024-04-12T05:06:42.166 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-12T05:06:42.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.284+0000 7fa5531c0700 1 -- 172.21.15.113:0/3356933096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa54c0f7970 msgr2=0x7fa54c0f7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:42.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.284+0000 7fa5531c0700 1 --2- 172.21.15.113:0/3356933096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa54c0f7970 0x7fa54c0f7dd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fa5380047f0 tx=0x7fa538009cb0).stop 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.284+0000 7fa5531c0700 1 -- 172.21.15.113:0/3356933096 shutdown_connections 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.284+0000 7fa5531c0700 1 --2- 172.21.15.113:0/3356933096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa54c0f7970 0x7fa54c0f7dd0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.284+0000 7fa5531c0700 1 --2- 172.21.15.113:0/3356933096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa54c0f66d0 0x7fa54c0f6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.284+0000 7fa5531c0700 1 -- 172.21.15.113:0/3356933096 >> 172.21.15.113:0/3356933096 conn(0x7fa54c0f1df0 msgr2=0x7fa54c0f4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.284+0000 7fa5531c0700 1 -- 172.21.15.113:0/3356933096 shutdown_connections 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.284+0000 7fa5531c0700 1 -- 172.21.15.113:0/3356933096 wait complete. 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.284+0000 7fa5531c0700 1 Processor -- start 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.284+0000 7fa5531c0700 1 -- start start 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa5531c0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa54c0f66d0 0x7fa54c061060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa5531c0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa54c0f7970 0x7fa54c10cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa5531c0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c10d210 con 0x7fa54c0f66d0 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa5531c0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c10d350 con 0x7fa54c0f7970 2024-04-12T05:06:42.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa550f5c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa54c0f66d0 0x7fa54c061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa550f5c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa54c0f66d0 0x7fa54c061060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55968/0 (socket says 172.21.15.113:55968) 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa550f5c700 1 -- 172.21.15.113:0/694570173 learned_addr learned my addr 172.21.15.113:0/694570173 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa54bfff700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa54c0f7970 0x7fa54c10cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa550f5c700 1 -- 172.21.15.113:0/694570173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa54c0f7970 msgr2=0x7fa54c10cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa550f5c700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa54c0f7970 0x7fa54c10cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa550f5c700 1 -- 172.21.15.113:0/694570173 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa538005040 con 0x7fa54c0f66d0 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa54bfff700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa54c0f7970 0x7fa54c10cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa550f5c700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa54c0f66d0 0x7fa54c061060 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fa54000c790 tx=0x7fa540007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa549ffb700 1 -- 172.21.15.113:0/694570173 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa54000e680 con 0x7fa54c0f66d0 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa549ffb700 1 -- 172.21.15.113:0/694570173 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fa54000ec70 con 0x7fa54c0f66d0 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa549ffb700 1 -- 172.21.15.113:0/694570173 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa540013db0 con 0x7fa54c0f66d0 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa5531c0700 1 -- 172.21.15.113:0/694570173 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54c10d5b0 con 0x7fa54c0f66d0 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.285+0000 7fa5531c0700 1 -- 172.21.15.113:0/694570173 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c10dac0 con 0x7fa54c0f66d0 2024-04-12T05:06:42.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.286+0000 7fa5531c0700 1 -- 172.21.15.113:0/694570173 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54c0fee90 con 0x7fa54c0f66d0 2024-04-12T05:06:42.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.287+0000 7fa549ffb700 1 -- 172.21.15.113:0/694570173 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fa54000e7e0 con 0x7fa54c0f66d0 2024-04-12T05:06:42.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.287+0000 7fa549ffb700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa53c061180 0x7fa53c063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:42.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.287+0000 7fa549ffb700 1 -- 172.21.15.113:0/694570173 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7fa54007caf0 con 0x7fa54c0f66d0 2024-04-12T05:06:42.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.287+0000 7fa54bfff700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa53c061180 0x7fa53c063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:42.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.287+0000 7fa54bfff700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa53c061180 0x7fa53c063640 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa538013470 tx=0x7fa53800b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:42.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.291+0000 7fa549ffb700 1 -- 172.21.15.113:0/694570173 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa540050a80 con 0x7fa54c0f66d0 2024-04-12T05:06:42.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:42 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:41.543956+0000 mon.smithi113 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.113:0/861055300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:06:42.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:42 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:41.846513+0000 mon.smithi113 (mon.0) 600 : audit [DBG] from='client.? 172.21.15.113:0/257108083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-04-12T05:06:42.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:42 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:42.149587+0000 mon.smithi113 (mon.0) 601 : audit [DBG] from='client.? 172.21.15.113:0/2077364055' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T05:06:42.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.453+0000 7fa5531c0700 1 -- 172.21.15.113:0/694570173 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fa54c100d00 con 0x7fa54c0f66d0 2024-04-12T05:06:42.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.454+0000 7fa549ffb700 1 -- 172.21.15.113:0/694570173 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7fa540050670 con 0x7fa54c0f66d0 2024-04-12T05:06:42.456 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-04-12T05:06:42.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.455+0000 7fa5531c0700 1 -- 172.21.15.113:0/694570173 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa53c061180 msgr2=0x7fa53c063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:42.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.455+0000 7fa5531c0700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa53c061180 0x7fa53c063640 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa538013470 tx=0x7fa53800b040).stop 2024-04-12T05:06:42.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.455+0000 7fa5531c0700 1 -- 172.21.15.113:0/694570173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa54c0f66d0 msgr2=0x7fa54c061060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:42.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.455+0000 7fa5531c0700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa54c0f66d0 0x7fa54c061060 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fa54000c790 tx=0x7fa540007f10).stop 2024-04-12T05:06:42.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.455+0000 7fa5531c0700 1 -- 172.21.15.113:0/694570173 shutdown_connections 2024-04-12T05:06:42.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.455+0000 7fa5531c0700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fa53c061180 0x7fa53c063640 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.455+0000 7fa5531c0700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa54c0f66d0 0x7fa54c061060 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.455+0000 7fa5531c0700 1 --2- 172.21.15.113:0/694570173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa54c0f7970 0x7fa54c10cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.455+0000 7fa5531c0700 1 -- 172.21.15.113:0/694570173 >> 172.21.15.113:0/694570173 conn(0x7fa54c0f1df0 msgr2=0x7fa54c0fb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:42.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.455+0000 7fa5531c0700 1 -- 172.21.15.113:0/694570173 shutdown_connections 2024-04-12T05:06:42.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.455+0000 7fa5531c0700 1 -- 172.21.15.113:0/694570173 wait complete. 2024-04-12T05:06:42.469 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T05:06:42.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.582+0000 7fcc39260700 1 -- 172.21.15.113:0/3577907080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc340f6710 msgr2=0x7fcc340f6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:42.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.582+0000 7fcc39260700 1 --2- 172.21.15.113:0/3577907080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc340f6710 0x7fcc340f6b30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fcc240047f0 tx=0x7fcc24009cb0).stop 2024-04-12T05:06:42.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.582+0000 7fcc39260700 1 -- 172.21.15.113:0/3577907080 shutdown_connections 2024-04-12T05:06:42.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.582+0000 7fcc39260700 1 --2- 172.21.15.113:0/3577907080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc340f79b0 0x7fcc340f7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.582+0000 7fcc39260700 1 --2- 172.21.15.113:0/3577907080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc340f6710 0x7fcc340f6b30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.582+0000 7fcc39260700 1 -- 172.21.15.113:0/3577907080 >> 172.21.15.113:0/3577907080 conn(0x7fcc340f1e50 msgr2=0x7fcc340f4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:42.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.582+0000 7fcc39260700 1 -- 172.21.15.113:0/3577907080 shutdown_connections 2024-04-12T05:06:42.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.582+0000 7fcc39260700 1 -- 172.21.15.113:0/3577907080 wait complete. 2024-04-12T05:06:42.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.582+0000 7fcc39260700 1 Processor -- start 2024-04-12T05:06:42.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc39260700 1 -- start start 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc39260700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc340f6710 0x7fcc34061140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc39260700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc340f79b0 0x7fcc3410ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc39260700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3410d310 con 0x7fcc340f6710 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc39260700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3410d450 con 0x7fcc340f79b0 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc32d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc340f6710 0x7fcc34061140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc32d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc340f6710 0x7fcc34061140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55986/0 (socket says 172.21.15.113:55986) 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc32d9d700 1 -- 172.21.15.113:0/1504257667 learned_addr learned my addr 172.21.15.113:0/1504257667 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc3259c700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc340f79b0 0x7fcc3410ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc32d9d700 1 -- 172.21.15.113:0/1504257667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc340f79b0 msgr2=0x7fcc3410ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc32d9d700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc340f79b0 0x7fcc3410ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc32d9d700 1 -- 172.21.15.113:0/1504257667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc24005040 con 0x7fcc340f6710 2024-04-12T05:06:42.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc3259c700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc340f79b0 0x7fcc3410ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:06:42.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc32d9d700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc340f6710 0x7fcc34061140 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fcc24013580 tx=0x7fcc24009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:42.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc1bfff700 1 -- 172.21.15.113:0/1504257667 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc24018070 con 0x7fcc340f6710 2024-04-12T05:06:42.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc1bfff700 1 -- 172.21.15.113:0/1504257667 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fcc24007e90 con 0x7fcc340f6710 2024-04-12T05:06:42.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc1bfff700 1 -- 172.21.15.113:0/1504257667 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc240123c0 con 0x7fcc340f6710 2024-04-12T05:06:42.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc39260700 1 -- 172.21.15.113:0/1504257667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3410d680 con 0x7fcc340f6710 2024-04-12T05:06:42.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.583+0000 7fcc39260700 1 -- 172.21.15.113:0/1504257667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3410dbe0 con 0x7fcc340f6710 2024-04-12T05:06:42.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.584+0000 7fcc39260700 1 -- 172.21.15.113:0/1504257667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc340feef0 con 0x7fcc340f6710 2024-04-12T05:06:42.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.585+0000 7fcc1bfff700 1 -- 172.21.15.113:0/1504257667 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fcc2401d420 con 0x7fcc340f6710 2024-04-12T05:06:42.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.585+0000 7fcc1bfff700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fcc1c061180 0x7fcc1c063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:42.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.585+0000 7fcc1bfff700 1 -- 172.21.15.113:0/1504257667 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7fcc2407d470 con 0x7fcc340f6710 2024-04-12T05:06:42.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.585+0000 7fcc3259c700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fcc1c061180 0x7fcc1c063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:42.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.585+0000 7fcc3259c700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fcc1c061180 0x7fcc1c063640 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fcc28009f20 tx=0x7fcc2800b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:42.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.589+0000 7fcc1bfff700 1 -- 172.21.15.113:0/1504257667 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fcc24051480 con 0x7fcc340f6710 2024-04-12T05:06:42.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:42 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:41.543956+0000 mon.smithi113 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.113:0/861055300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:06:42.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:42 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:41.846513+0000 mon.smithi113 (mon.0) 600 : audit [DBG] from='client.? 172.21.15.113:0/257108083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-04-12T05:06:42.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:42 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:42.149587+0000 mon.smithi113 (mon.0) 601 : audit [DBG] from='client.? 172.21.15.113:0/2077364055' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T05:06:42.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.791+0000 7fcc39260700 1 -- 172.21.15.113:0/1504257667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc340598f0 con 0x7fcc340f6710 2024-04-12T05:06:42.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.791+0000 7fcc1bfff700 1 -- 172.21.15.113:0/1504257667 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4602 (secure 0 0 0) 0x7fcc24051070 con 0x7fcc340f6710 2024-04-12T05:06:42.793 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:06:42.794 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14438,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/2511637182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2511637182},{"type":"v1","addr":"172.21.15.113:6829","nonce":2511637182}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:06:29.875846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14426,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14426":{"gid":14426,"name":"cephfs.smithi113.ovmadh","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.113:6827/1262609849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1262609849},{"type":"v1","addr":"172.21.15.113:6827","nonce":1262609849}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi192.ammjap","rank":1,"incarnation":9,"state":"up:active","state_seq":2,"addr":"172.21.15.192:6825/616350866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":616350866},{"type":"v1","addr":"172.21.15.192:6825","nonce":616350866}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-04-12T05:06:42.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.793+0000 7fcc39260700 1 -- 172.21.15.113:0/1504257667 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fcc1c061180 msgr2=0x7fcc1c063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:42.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.793+0000 7fcc39260700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fcc1c061180 0x7fcc1c063640 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fcc28009f20 tx=0x7fcc2800b040).stop 2024-04-12T05:06:42.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.793+0000 7fcc39260700 1 -- 172.21.15.113:0/1504257667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc340f6710 msgr2=0x7fcc34061140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:42.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.793+0000 7fcc39260700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc340f6710 0x7fcc34061140 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fcc24013580 tx=0x7fcc24009cb0).stop 2024-04-12T05:06:42.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.793+0000 7fcc39260700 1 -- 172.21.15.113:0/1504257667 shutdown_connections 2024-04-12T05:06:42.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.793+0000 7fcc39260700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fcc1c061180 0x7fcc1c063640 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.793+0000 7fcc39260700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc340f6710 0x7fcc34061140 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.793+0000 7fcc39260700 1 --2- 172.21.15.113:0/1504257667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc340f79b0 0x7fcc3410ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:42.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.793+0000 7fcc39260700 1 -- 172.21.15.113:0/1504257667 >> 172.21.15.113:0/1504257667 conn(0x7fcc340f1e50 msgr2=0x7fcc340fb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:42.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.793+0000 7fcc39260700 1 -- 172.21.15.113:0/1504257667 shutdown_connections 2024-04-12T05:06:42.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:42.793+0000 7fcc39260700 1 -- 172.21.15.113:0/1504257667 wait complete. 2024-04-12T05:06:42.796 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 14 2024-04-12T05:06:42.807 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 14, 'max_mds': 2, 'flags': 18} 2024-04-12T05:06:42.807 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-12T05:06:42.891 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-12T05:06:42.891 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-12T05:06:42.891 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-12T05:06:42.891 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-12T05:06:42.891 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-12T05:06:42.891 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-12T05:06:42.891 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-12T05:06:42.891 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-12T05:06:42.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:42.892 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2024-04-12T05:06:42.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:42.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link delete ceph-brx 2024-04-12T05:06:42.995 INFO:teuthology.orchestra.run.smithi192.stderr:Cannot find device "ceph-brx" 2024-04-12T05:06:42.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:06:42.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:42.997 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-04-12T05:06:43.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:43.015 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link delete ceph-brx 2024-04-12T05:06:43.085 INFO:teuthology.orchestra.run.smithi113.stderr:Cannot find device "ceph-brx" 2024-04-12T05:06:43.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:06:43.086 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-12T05:06:43.087 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph fs ls 2024-04-12T05:06:43.392 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:06:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:41.554534+0000 mgr.smithi113.ctsazg (mgr.14194) 219 : cluster [DBG] pgmap v190: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:42.454724+0000 mon.smithi113 (mon.0) 602 : audit [DBG] from='client.? 172.21.15.113:0/694570173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T05:06:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:42.792492+0000 mon.smithi113 (mon.0) 603 : audit [DBG] from='client.? 172.21.15.113:0/1504257667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T05:06:43.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:41.554534+0000 mgr.smithi113.ctsazg (mgr.14194) 219 : cluster [DBG] pgmap v190: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:43.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:42.454724+0000 mon.smithi113 (mon.0) 602 : audit [DBG] from='client.? 172.21.15.113:0/694570173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T05:06:43.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:42.792492+0000 mon.smithi113 (mon.0) 603 : audit [DBG] from='client.? 172.21.15.113:0/1504257667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T05:06:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.048+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1448786122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5c00f68a0 msgr2=0x7fe5c00f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.048+0000 7fe5c58e0700 1 --2- 172.21.15.113:0/1448786122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5c00f68a0 0x7fe5c00f6cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe5b00047f0 tx=0x7fe5b0009cb0).stop 2024-04-12T05:06:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.048+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1448786122 shutdown_connections 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.048+0000 7fe5c58e0700 1 --2- 172.21.15.113:0/1448786122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c00f7b40 0x7fe5c00f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.048+0000 7fe5c58e0700 1 --2- 172.21.15.113:0/1448786122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5c00f68a0 0x7fe5c00f6cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.048+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1448786122 >> 172.21.15.113:0/1448786122 conn(0x7fe5c00f2020 msgr2=0x7fe5c00f4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.048+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1448786122 shutdown_connections 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.048+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1448786122 wait complete. 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.049+0000 7fe5c58e0700 1 Processor -- start 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.049+0000 7fe5c58e0700 1 -- start start 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.049+0000 7fe5c58e0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5c00f68a0 0x7fe5c01080f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.049+0000 7fe5c58e0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c00f7b40 0x7fe5c01085e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.049+0000 7fe5c58e0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c0108b90 con 0x7fe5c00f68a0 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.049+0000 7fe5c58e0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c0108cd0 con 0x7fe5c00f7b40 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.049+0000 7fe5beffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5c00f68a0 0x7fe5c01080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.049+0000 7fe5beffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5c00f68a0 0x7fe5c01080f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55992/0 (socket says 172.21.15.113:55992) 2024-04-12T05:06:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.049+0000 7fe5beffd700 1 -- 172.21.15.113:0/1465944349 learned_addr learned my addr 172.21.15.113:0/1465944349 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.049+0000 7fe5be7fc700 1 --2- 172.21.15.113:0/1465944349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c00f7b40 0x7fe5c01085e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.050+0000 7fe5beffd700 1 -- 172.21.15.113:0/1465944349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c00f7b40 msgr2=0x7fe5c01085e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.050+0000 7fe5beffd700 1 --2- 172.21.15.113:0/1465944349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c00f7b40 0x7fe5c01085e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.050+0000 7fe5beffd700 1 -- 172.21.15.113:0/1465944349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b0005040 con 0x7fe5c00f68a0 2024-04-12T05:06:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.050+0000 7fe5beffd700 1 --2- 172.21.15.113:0/1465944349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5c00f68a0 0x7fe5c01080f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fe5b0004f10 tx=0x7fe5b0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.050+0000 7fe5a7fff700 1 -- 172.21.15.113:0/1465944349 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5b000e070 con 0x7fe5c00f68a0 2024-04-12T05:06:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.050+0000 7fe5a7fff700 1 -- 172.21.15.113:0/1465944349 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fe5b0007e90 con 0x7fe5c00f68a0 2024-04-12T05:06:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.050+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1465944349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c010d710 con 0x7fe5c00f68a0 2024-04-12T05:06:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.050+0000 7fe5a7fff700 1 -- 172.21.15.113:0/1465944349 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5b0018070 con 0x7fe5c00f68a0 2024-04-12T05:06:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.050+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1465944349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c010dc50 con 0x7fe5c00f68a0 2024-04-12T05:06:45.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.051+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1465944349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c00ff010 con 0x7fe5c00f68a0 2024-04-12T05:06:45.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.051+0000 7fe5a7fff700 1 -- 172.21.15.113:0/1465944349 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fe5b00123c0 con 0x7fe5c00f68a0 2024-04-12T05:06:45.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.051+0000 7fe5a7fff700 1 --2- 172.21.15.113:0/1465944349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe5a8061110 0x7fe5a80635d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:45.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.052+0000 7fe5a7fff700 1 -- 172.21.15.113:0/1465944349 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7fe5b007d680 con 0x7fe5c00f68a0 2024-04-12T05:06:45.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.052+0000 7fe5be7fc700 1 --2- 172.21.15.113:0/1465944349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe5a8061110 0x7fe5a80635d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:45.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.052+0000 7fe5be7fc700 1 --2- 172.21.15.113:0/1465944349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe5a8061110 0x7fe5a80635d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe5b4004de0 tx=0x7fe5b400b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:45.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.056+0000 7fe5a7fff700 1 -- 172.21.15.113:0/1465944349 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe5b0051610 con 0x7fe5c00f68a0 2024-04-12T05:06:45.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.261+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1465944349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe5c010e2e0 con 0x7fe5c00f68a0 2024-04-12T05:06:45.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.261+0000 7fe5a7fff700 1 -- 172.21.15.113:0/1465944349 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7fe5b0051200 con 0x7fe5c00f68a0 2024-04-12T05:06:45.263 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-12T05:06:45.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.263+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1465944349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe5a8061110 msgr2=0x7fe5a80635d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:45.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.263+0000 7fe5c58e0700 1 --2- 172.21.15.113:0/1465944349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe5a8061110 0x7fe5a80635d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe5b4004de0 tx=0x7fe5b400b040).stop 2024-04-12T05:06:45.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.263+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1465944349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5c00f68a0 msgr2=0x7fe5c01080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:45.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.263+0000 7fe5c58e0700 1 --2- 172.21.15.113:0/1465944349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5c00f68a0 0x7fe5c01080f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fe5b0004f10 tx=0x7fe5b0009cb0).stop 2024-04-12T05:06:45.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.263+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1465944349 shutdown_connections 2024-04-12T05:06:45.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.263+0000 7fe5c58e0700 1 --2- 172.21.15.113:0/1465944349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe5a8061110 0x7fe5a80635d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:45.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.263+0000 7fe5c58e0700 1 --2- 172.21.15.113:0/1465944349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5c00f68a0 0x7fe5c01080f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:45.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.263+0000 7fe5c58e0700 1 --2- 172.21.15.113:0/1465944349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c00f7b40 0x7fe5c01085e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:45.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.263+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1465944349 >> 172.21.15.113:0/1465944349 conn(0x7fe5c00f2020 msgr2=0x7fe5c0060af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:45.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.263+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1465944349 shutdown_connections 2024-04-12T05:06:45.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:45.263+0000 7fe5c58e0700 1 -- 172.21.15.113:0/1465944349 wait complete. 2024-04-12T05:06:45.589 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:43.554973+0000 mgr.smithi113.ctsazg (mgr.14194) 220 : cluster [DBG] pgmap v191: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:45.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:43.554973+0000 mgr.smithi113.ctsazg (mgr.14194) 220 : cluster [DBG] pgmap v191: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:45.872 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-12T05:06:45.872 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-12T05:06:45.872 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi113.front.sepia.ceph.com 2024-04-12T05:06:45.872 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-12T05:06:45.872 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-12T05:06:45.872 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-12T05:06:45.872 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-12T05:06:45.872 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-12T05:06:45.872 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-12T05:06:45.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:45.872 DEBUG:teuthology.orchestra.run.smithi113:> ip addr 2024-04-12T05:06:45.892 INFO:teuthology.orchestra.run.smithi113.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-12T05:06:45.892 INFO:teuthology.orchestra.run.smithi113.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-12T05:06:45.892 INFO:teuthology.orchestra.run.smithi113.stdout: inet 127.0.0.1/8 scope host lo 2024-04-12T05:06:45.892 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:06:45.892 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 ::1/128 scope host 2024-04-12T05:06:45.892 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:06:45.892 INFO:teuthology.orchestra.run.smithi113.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:06:45.892 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:44 brd ff:ff:ff:ff:ff:ff 2024-04-12T05:06:45.892 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f0 2024-04-12T05:06:45.893 INFO:teuthology.orchestra.run.smithi113.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:06:45.893 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:d8:db:45 brd ff:ff:ff:ff:ff:ff 2024-04-12T05:06:45.893 INFO:teuthology.orchestra.run.smithi113.stdout: altname enp7s0f1 2024-04-12T05:06:45.893 INFO:teuthology.orchestra.run.smithi113.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:06:45.893 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c0 brd ff:ff:ff:ff:ff:ff 2024-04-12T05:06:45.893 INFO:teuthology.orchestra.run.smithi113.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-12T05:06:45.893 INFO:teuthology.orchestra.run.smithi113.stdout: link/ether 0c:c4:7a:8f:d7:c1 brd ff:ff:ff:ff:ff:ff 2024-04-12T05:06:45.893 INFO:teuthology.orchestra.run.smithi113.stdout: inet 172.21.15.113/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-12T05:06:45.893 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft 42149sec preferred_lft 42149sec 2024-04-12T05:06:45.893 INFO:teuthology.orchestra.run.smithi113.stdout: inet6 fe80::ec4:7aff:fe8f:d7c1/64 scope link 2024-04-12T05:06:45.893 INFO:teuthology.orchestra.run.smithi113.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:06:45.893 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-12T05:06:45.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:45.894 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-12T05:06:45.894 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-12T05:06:45.894 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link add name ceph-brx type bridge 2024-04-12T05:06:45.894 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip addr flush dev ceph-brx 2024-04-12T05:06:45.894 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set ceph-brx up 2024-04-12T05:06:45.894 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-12T05:06:45.894 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-12T05:06:46.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:46.060 DEBUG:teuthology.orchestra.run.smithi113:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-12T05:06:46.087 INFO:teuthology.orchestra.run.smithi113.stdout:1 2024-04-12T05:06:46.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:46.089 DEBUG:teuthology.orchestra.run.smithi113:> route 2024-04-12T05:06:46.163 INFO:teuthology.orchestra.run.smithi113.stdout:Kernel IP routing table 2024-04-12T05:06:46.163 INFO:teuthology.orchestra.run.smithi113.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-12T05:06:46.163 INFO:teuthology.orchestra.run.smithi113.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-12T05:06:46.163 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-12T05:06:46.163 INFO:teuthology.orchestra.run.smithi113.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-12T05:06:46.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:46.163 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-12T05:06:46.164 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-12T05:06:46.164 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-12T05:06:46.164 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-12T05:06:46.164 DEBUG:teuthology.orchestra.run.smithi113:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-12T05:06:46.164 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-12T05:06:46.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:46.444 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list 2024-04-12T05:06:46.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:46.462 DEBUG:teuthology.orchestra.run.smithi113:> ip netns list-id 2024-04-12T05:06:46.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:46.520 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-12T05:06:46.520 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-12T05:06:46.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-12T05:06:46.520 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-12T05:06:46.520 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-12T05:06:46.599 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:45.262501+0000 mon.smithi113 (mon.0) 604 : audit [DBG] from='client.? 172.21.15.113:0/1465944349' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T05:06:46.615 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-12T05:06:46.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:46.616 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-12T05:06:46.616 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-12T05:06:46.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-12T05:06:46.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-12T05:06:46.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-12T05:06:46.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-12T05:06:46.616 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-12T05:06:46.616 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-12T05:06:46.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:45.262501+0000 mon.smithi113 (mon.0) 604 : audit [DBG] from='client.? 172.21.15.113:0/1465944349' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T05:06:46.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:46.866 DEBUG:teuthology.orchestra.run.smithi113:> (cd / && exec bash -c ' 2024-04-12T05:06:46.866 DEBUG:teuthology.orchestra.run.smithi113:> set -e 2024-04-12T05:06:46.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set brx.0 up 2024-04-12T05:06:46.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo ip link set dev brx.0 master ceph-brx 2024-04-12T05:06:46.867 DEBUG:teuthology.orchestra.run.smithi113:> ') 2024-04-12T05:06:46.963 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-12T05:06:46.964 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T05:06:46.964 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-12T05:06:46.982 INFO:teuthology.orchestra.run.smithi113.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-12T05:06:46.982 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T05:06:46.982 DEBUG:teuthology.orchestra.run.smithi113:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-12T05:06:47.039 DEBUG:teuthology.orchestra.run.smithi113:> sudo modprobe fuse 2024-04-12T05:06:47.106 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-12T05:06:47.106 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-12T05:06:47.173 INFO:teuthology.orchestra.run.smithi113.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-12T05:06:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-12T05:06:47.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:47.175 DEBUG:teuthology.orchestra.run.smithi113:> ls /sys/fs/fuse/connections 2024-04-12T05:06:47.232 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-12T05:06:47.232 DEBUG:teuthology.orchestra.run.smithi113:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-12T05:06:47.275 DEBUG:teuthology.orchestra.run.smithi113:> sudo modprobe fuse 2024-04-12T05:06:47.301 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-12T05:06:47.301 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-12T05:06:47.368 INFO:teuthology.orchestra.run.smithi113.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-12T05:06:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-12T05:06:47.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:47.369 DEBUG:teuthology.orchestra.run.smithi113:> ls /sys/fs/fuse/connections 2024-04-12T05:06:47.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi113.stderr:2024-04-12T05:06:47.375+0000 7f4f7a1fc3c0 -1 init, newargv = 0x56255abc6630 newargc=15 2024-04-12T05:06:47.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi113.stderr:ceph-fuse[77892]: starting ceph client 2024-04-12T05:06:47.385 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi113.stderr:ceph-fuse[77892]: starting fuse 2024-04-12T05:06:47.396 INFO:teuthology.orchestra.run.smithi113.stdout:149 2024-04-12T05:06:47.397 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [149] 2024-04-12T05:06:47.397 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-04-12T05:06:47.397 DEBUG:teuthology.orchestra.run.smithi113:> import glob 2024-04-12T05:06:47.397 DEBUG:teuthology.orchestra.run.smithi113:> import re 2024-04-12T05:06:47.397 DEBUG:teuthology.orchestra.run.smithi113:> import os 2024-04-12T05:06:47.397 DEBUG:teuthology.orchestra.run.smithi113:> import subprocess 2024-04-12T05:06:47.397 DEBUG:teuthology.orchestra.run.smithi113:> 2024-04-12T05:06:47.397 DEBUG:teuthology.orchestra.run.smithi113:> def _find_admin_socket(client_name): 2024-04-12T05:06:47.397 DEBUG:teuthology.orchestra.run.smithi113:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-12T05:06:47.397 DEBUG:teuthology.orchestra.run.smithi113:> files = glob.glob(asok_path) 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> # Given a non-glob path, it better be there 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> if "*" not in asok_path: 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> assert(len(files) == 1) 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> return files[0] 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> for f in files: 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> contents = proc_f.read() 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> if mountpoint in contents: 2024-04-12T05:06:47.398 DEBUG:teuthology.orchestra.run.smithi113:> return f 2024-04-12T05:06:47.399 DEBUG:teuthology.orchestra.run.smithi113:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-12T05:06:47.399 DEBUG:teuthology.orchestra.run.smithi113:> 2024-04-12T05:06:47.399 DEBUG:teuthology.orchestra.run.smithi113:> print(_find_admin_socket("client.0")) 2024-04-12T05:06:47.399 DEBUG:teuthology.orchestra.run.smithi113:> ' 2024-04-12T05:06:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:45.555547+0000 mgr.smithi113.ctsazg (mgr.14194) 221 : cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:47.505 INFO:teuthology.orchestra.run.smithi113.stdout:/var/run/ceph/ceph-client.0.77892.asok 2024-04-12T05:06:47.688 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.77892.asok 2024-04-12T05:06:47.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:47.688 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.77892.asok status 2024-04-12T05:06:47.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:45.555547+0000 mgr.smithi113.ctsazg (mgr.14194) 221 : cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:47.838 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:06:47.838 INFO:teuthology.orchestra.run.smithi113.stdout: "metadata": { 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_sha1": "618f440892089921c3e944a991122ddc44e60516", 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_version": "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)", 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "entity_id": "0", 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "hostname": "smithi113", 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "pid": "77892", 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "root": "/" 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "dentry_count": 0, 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "dentry_pinned_count": 0, 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "id": 14486, 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "inst": { 2024-04-12T05:06:47.839 INFO:teuthology.orchestra.run.smithi113.stdout: "name": { 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: "type": "client", 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: "num": 14486 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": { 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: "type": "v1", 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": "192.168.0.1:0", 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: "nonce": 3583258718 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": { 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: "type": "v1", 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: "addr": "192.168.0.1:0", 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: "nonce": 3583258718 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:06:47.840 INFO:teuthology.orchestra.run.smithi113.stdout: "inst_str": "client.14486 192.168.0.1:0/3583258718", 2024-04-12T05:06:47.841 INFO:teuthology.orchestra.run.smithi113.stdout: "addr_str": "192.168.0.1:0/3583258718", 2024-04-12T05:06:47.841 INFO:teuthology.orchestra.run.smithi113.stdout: "inode_count": 1, 2024-04-12T05:06:47.841 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_epoch": 14, 2024-04-12T05:06:47.841 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_epoch": 43, 2024-04-12T05:06:47.841 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_epoch_barrier": 0, 2024-04-12T05:06:47.841 INFO:teuthology.orchestra.run.smithi113.stdout: "blocklisted": false, 2024-04-12T05:06:47.841 INFO:teuthology.orchestra.run.smithi113.stdout: "fs_name": "cephfs" 2024-04-12T05:06:47.841 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:06:47.848 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- ceph fs ls 2024-04-12T05:06:48.115 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi113/config 2024-04-12T05:06:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:47.556201+0000 mgr.smithi113.ctsazg (mgr.14194) 222 : cluster [DBG] pgmap v193: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:49.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:47.556201+0000 mgr.smithi113.ctsazg (mgr.14194) 222 : cluster [DBG] pgmap v193: 65 pgs: 65 active+clean; 3.6 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:06:50.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.027+0000 7f6400e92700 1 -- 172.21.15.113:0/4047795948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63fc0f68a0 msgr2=0x7f63fc0f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:50.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.027+0000 7f6400e92700 1 --2- 172.21.15.113:0/4047795948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63fc0f68a0 0x7f63fc0f6cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f63e40047f0 tx=0x7f63e4009cb0).stop 2024-04-12T05:06:50.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.027+0000 7f6400e92700 1 -- 172.21.15.113:0/4047795948 shutdown_connections 2024-04-12T05:06:50.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.027+0000 7f6400e92700 1 --2- 172.21.15.113:0/4047795948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc0f7b40 0x7f63fc0f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:50.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.027+0000 7f6400e92700 1 --2- 172.21.15.113:0/4047795948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63fc0f68a0 0x7f63fc0f6cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:50.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.027+0000 7f6400e92700 1 -- 172.21.15.113:0/4047795948 >> 172.21.15.113:0/4047795948 conn(0x7f63fc0f2020 msgr2=0x7f63fc0f4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:50.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.027+0000 7f6400e92700 1 -- 172.21.15.113:0/4047795948 shutdown_connections 2024-04-12T05:06:50.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.027+0000 7f6400e92700 1 -- 172.21.15.113:0/4047795948 wait complete. 2024-04-12T05:06:50.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.028+0000 7f6400e92700 1 Processor -- start 2024-04-12T05:06:50.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.028+0000 7f6400e92700 1 -- start start 2024-04-12T05:06:50.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.028+0000 7f6400e92700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc0f68a0 0x7f63fc1080d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:50.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.028+0000 7f6400e92700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63fc0f7b40 0x7f63fc1085c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.028+0000 7f6400e92700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc108b70 con 0x7f63fc0f7b40 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.028+0000 7f6400e92700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc108cb0 con 0x7f63fc0f68a0 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f63f9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63fc0f7b40 0x7f63fc1085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f63f9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63fc0f7b40 0x7f63fc1085c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56014/0 (socket says 172.21.15.113:56014) 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f63f9d9b700 1 -- 172.21.15.113:0/909346349 learned_addr learned my addr 172.21.15.113:0/909346349 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f63fa59c700 1 --2- 172.21.15.113:0/909346349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc0f68a0 0x7f63fc1080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f63f9d9b700 1 -- 172.21.15.113:0/909346349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc0f68a0 msgr2=0x7f63fc1080d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f63f9d9b700 1 --2- 172.21.15.113:0/909346349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc0f68a0 0x7f63fc1080d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f63f9d9b700 1 -- 172.21.15.113:0/909346349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e4005040 con 0x7f63fc0f7b40 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f63f9d9b700 1 --2- 172.21.15.113:0/909346349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63fc0f7b40 0x7f63fc1085c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f63ec0041d0 tx=0x7f63ec009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f63f37fe700 1 -- 172.21.15.113:0/909346349 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63ec007b40 con 0x7f63fc0f7b40 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f6400e92700 1 -- 172.21.15.113:0/909346349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63fc10d750 con 0x7f63fc0f7b40 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f63f37fe700 1 -- 172.21.15.113:0/909346349 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f63ec00f040 con 0x7f63fc0f7b40 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f63f37fe700 1 -- 172.21.15.113:0/909346349 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63ec013730 con 0x7f63fc0f7b40 2024-04-12T05:06:50.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.029+0000 7f6400e92700 1 -- 172.21.15.113:0/909346349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63fc10dc90 con 0x7f63fc0f7b40 2024-04-12T05:06:50.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.030+0000 7f6400e92700 1 -- 172.21.15.113:0/909346349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63fc0ff010 con 0x7f63fc0f7b40 2024-04-12T05:06:50.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.031+0000 7f63f37fe700 1 -- 172.21.15.113:0/909346349 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f63ec009110 con 0x7f63fc0f7b40 2024-04-12T05:06:50.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.031+0000 7f63f37fe700 1 --2- 172.21.15.113:0/909346349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f63e8061180 0x7f63e8063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:50.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.031+0000 7f63f37fe700 1 -- 172.21.15.113:0/909346349 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f63ec07c2e0 con 0x7f63fc0f7b40 2024-04-12T05:06:50.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.031+0000 7f63fa59c700 1 --2- 172.21.15.113:0/909346349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f63e8061180 0x7f63e8063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:50.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.031+0000 7f63fa59c700 1 --2- 172.21.15.113:0/909346349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f63e8061180 0x7f63e8063640 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f63e4004f10 tx=0x7f63e400b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:50.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.035+0000 7f63f37fe700 1 -- 172.21.15.113:0/909346349 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f63ec050270 con 0x7f63fc0f7b40 2024-04-12T05:06:50.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.238+0000 7f6400e92700 1 -- 172.21.15.113:0/909346349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f63fc10e390 con 0x7f63fc0f7b40 2024-04-12T05:06:50.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.239+0000 7f63f37fe700 1 -- 172.21.15.113:0/909346349 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7f63ec04fe60 con 0x7f63fc0f7b40 2024-04-12T05:06:50.241 INFO:teuthology.orchestra.run.smithi113.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-12T05:06:50.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.240+0000 7f6400e92700 1 -- 172.21.15.113:0/909346349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f63e8061180 msgr2=0x7f63e8063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:50.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.240+0000 7f6400e92700 1 --2- 172.21.15.113:0/909346349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f63e8061180 0x7f63e8063640 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f63e4004f10 tx=0x7f63e400b040).stop 2024-04-12T05:06:50.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.240+0000 7f6400e92700 1 -- 172.21.15.113:0/909346349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63fc0f7b40 msgr2=0x7f63fc1085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:50.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.240+0000 7f6400e92700 1 --2- 172.21.15.113:0/909346349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63fc0f7b40 0x7f63fc1085c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f63ec0041d0 tx=0x7f63ec009f60).stop 2024-04-12T05:06:50.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.241+0000 7f6400e92700 1 -- 172.21.15.113:0/909346349 shutdown_connections 2024-04-12T05:06:50.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.241+0000 7f6400e92700 1 --2- 172.21.15.113:0/909346349 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f63e8061180 0x7f63e8063640 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:50.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.241+0000 7f6400e92700 1 --2- 172.21.15.113:0/909346349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63fc0f68a0 0x7f63fc1080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:50.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.241+0000 7f6400e92700 1 --2- 172.21.15.113:0/909346349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63fc0f7b40 0x7f63fc1085c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:50.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.241+0000 7f6400e92700 1 -- 172.21.15.113:0/909346349 >> 172.21.15.113:0/909346349 conn(0x7f63fc0f2020 msgr2=0x7f63fc061310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:50.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.241+0000 7f6400e92700 1 -- 172.21.15.113:0/909346349 shutdown_connections 2024-04-12T05:06:50.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:50.241+0000 7f6400e92700 1 -- 172.21.15.113:0/909346349 wait complete. 2024-04-12T05:06:50.597 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:06:50.239935+0000 mon.smithi113 (mon.0) 605 : audit [DBG] from='client.? 172.21.15.113:0/909346349' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T05:06:50.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:06:50.239935+0000 mon.smithi113 (mon.0) 605 : audit [DBG] from='client.? 172.21.15.113:0/909346349' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T05:06:50.969 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-12T05:06:50.969 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-12T05:06:50.969 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi192.front.sepia.ceph.com 2024-04-12T05:06:50.969 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-12T05:06:50.969 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-12T05:06:50.969 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-12T05:06:50.969 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-12T05:06:50.969 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-12T05:06:50.969 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-12T05:06:50.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:50.970 DEBUG:teuthology.orchestra.run.smithi192:> ip addr 2024-04-12T05:06:50.993 INFO:teuthology.orchestra.run.smithi192.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-12T05:06:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-12T05:06:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: inet 127.0.0.1/8 scope host lo 2024-04-12T05:06:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:06:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 ::1/128 scope host 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:d9:38:4e brd ff:ff:ff:ff:ff:ff 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: altname enp7s0f0 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:d9:38:4f brd ff:ff:ff:ff:ff:ff 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: altname enp7s0f1 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:88:80:48 brd ff:ff:ff:ff:ff:ff 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:88:80:49 brd ff:ff:ff:ff:ff:ff 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: inet 172.21.15.192/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-12T05:06:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft 42126sec preferred_lft 42126sec 2024-04-12T05:06:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::ec4:7aff:fe88:8049/64 scope link 2024-04-12T05:06:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2024-04-12T05:06:50.995 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-12T05:06:50.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:50.995 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-12T05:06:50.995 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-12T05:06:50.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link add name ceph-brx type bridge 2024-04-12T05:06:50.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip addr flush dev ceph-brx 2024-04-12T05:06:50.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set ceph-brx up 2024-04-12T05:06:50.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-12T05:06:50.995 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-12T05:06:51.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:51.169 DEBUG:teuthology.orchestra.run.smithi192:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-12T05:06:51.198 INFO:teuthology.orchestra.run.smithi192.stdout:1 2024-04-12T05:06:51.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:51.200 DEBUG:teuthology.orchestra.run.smithi192:> route 2024-04-12T05:06:51.272 INFO:teuthology.orchestra.run.smithi192.stdout:Kernel IP routing table 2024-04-12T05:06:51.272 INFO:teuthology.orchestra.run.smithi192.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-12T05:06:51.272 INFO:teuthology.orchestra.run.smithi192.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-12T05:06:51.272 INFO:teuthology.orchestra.run.smithi192.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-12T05:06:51.272 INFO:teuthology.orchestra.run.smithi192.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-12T05:06:51.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:51.273 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-12T05:06:51.273 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-12T05:06:51.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-12T05:06:51.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-12T05:06:51.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-12T05:06:51.273 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-12T05:06:51.527 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:49.556611+0000 mgr.smithi113.ctsazg (mgr.14194) 223 : cluster [DBG] pgmap v194: 65 pgs: 65 active+clean; 4.0 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-12T05:06:51.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:51.628 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2024-04-12T05:06:51.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:51.647 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list-id 2024-04-12T05:06:51.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:49.556611+0000 mgr.smithi113.ctsazg (mgr.14194) 223 : cluster [DBG] pgmap v194: 65 pgs: 65 active+clean; 4.0 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-12T05:06:51.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:51.705 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-12T05:06:51.705 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-12T05:06:51.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-12T05:06:51.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-12T05:06:51.705 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-12T05:06:51.800 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-12T05:06:51.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:51.800 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-12T05:06:51.800 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-12T05:06:51.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-12T05:06:51.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-12T05:06:51.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-12T05:06:51.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-12T05:06:51.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-12T05:06:51.801 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-12T05:06:52.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:52.010 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2024-04-12T05:06:52.010 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2024-04-12T05:06:52.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set brx.0 up 2024-04-12T05:06:52.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set dev brx.0 master ceph-brx 2024-04-12T05:06:52.010 DEBUG:teuthology.orchestra.run.smithi192:> ') 2024-04-12T05:06:52.107 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-12T05:06:52.108 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T05:06:52.108 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-12T05:06:52.125 INFO:teuthology.orchestra.run.smithi192.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-12T05:06:52.125 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T05:06:52.125 DEBUG:teuthology.orchestra.run.smithi192:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-12T05:06:52.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo modprobe fuse 2024-04-12T05:06:52.251 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-12T05:06:52.251 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-12T05:06:52.318 INFO:teuthology.orchestra.run.smithi192.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-12T05:06:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-12T05:06:52.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:52.319 DEBUG:teuthology.orchestra.run.smithi192:> ls /sys/fs/fuse/connections 2024-04-12T05:06:52.378 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-12T05:06:52.378 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-12T05:06:52.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo modprobe fuse 2024-04-12T05:06:52.447 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-12T05:06:52.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-12T05:06:52.515 INFO:teuthology.orchestra.run.smithi192.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-12T05:06:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-12T05:06:52.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:52.516 DEBUG:teuthology.orchestra.run.smithi192:> ls /sys/fs/fuse/connections 2024-04-12T05:06:52.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi192.stderr:2024-04-12T05:06:52.559+0000 7f363b8ea3c0 -1 init, newargv = 0x55574488a700 newargc=15 2024-04-12T05:06:52.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi192.stderr:ceph-fuse[64900]: starting ceph client 2024-04-12T05:06:52.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi192.stderr:ceph-fuse[64900]: starting fuse 2024-04-12T05:06:52.576 INFO:teuthology.orchestra.run.smithi192.stdout:137 2024-04-12T05:06:52.576 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [137] 2024-04-12T05:06:52.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-04-12T05:06:52.576 DEBUG:teuthology.orchestra.run.smithi192:> import glob 2024-04-12T05:06:52.576 DEBUG:teuthology.orchestra.run.smithi192:> import re 2024-04-12T05:06:52.576 DEBUG:teuthology.orchestra.run.smithi192:> import os 2024-04-12T05:06:52.576 DEBUG:teuthology.orchestra.run.smithi192:> import subprocess 2024-04-12T05:06:52.576 DEBUG:teuthology.orchestra.run.smithi192:> 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> def _find_admin_socket(client_name): 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> files = glob.glob(asok_path) 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> # Given a non-glob path, it better be there 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> if "*" not in asok_path: 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> assert(len(files) == 1) 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> return files[0] 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> for f in files: 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> contents = proc_f.read() 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> if mountpoint in contents: 2024-04-12T05:06:52.577 DEBUG:teuthology.orchestra.run.smithi192:> return f 2024-04-12T05:06:52.578 DEBUG:teuthology.orchestra.run.smithi192:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-12T05:06:52.578 DEBUG:teuthology.orchestra.run.smithi192:> 2024-04-12T05:06:52.578 DEBUG:teuthology.orchestra.run.smithi192:> print(_find_admin_socket("client.1")) 2024-04-12T05:06:52.578 DEBUG:teuthology.orchestra.run.smithi192:> ' 2024-04-12T05:06:52.699 INFO:teuthology.orchestra.run.smithi192.stdout:/var/run/ceph/ceph-client.1.64900.asok 2024-04-12T05:06:52.882 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.64900.asok 2024-04-12T05:06:52.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:52.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.64900.asok status 2024-04-12T05:06:53.033 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-12T05:06:53.033 INFO:teuthology.orchestra.run.smithi192.stdout: "metadata": { 2024-04-12T05:06:53.033 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_sha1": "618f440892089921c3e944a991122ddc44e60516", 2024-04-12T05:06:53.033 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)", 2024-04-12T05:06:53.033 INFO:teuthology.orchestra.run.smithi192.stdout: "entity_id": "1", 2024-04-12T05:06:53.033 INFO:teuthology.orchestra.run.smithi192.stdout: "hostname": "smithi192", 2024-04-12T05:06:53.033 INFO:teuthology.orchestra.run.smithi192.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-12T05:06:53.033 INFO:teuthology.orchestra.run.smithi192.stdout: "pid": "64900", 2024-04-12T05:06:53.033 INFO:teuthology.orchestra.run.smithi192.stdout: "root": "/" 2024-04-12T05:06:53.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-12T05:06:53.033 INFO:teuthology.orchestra.run.smithi192.stdout: "dentry_count": 0, 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: "dentry_pinned_count": 0, 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: "id": 24303, 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: "inst": { 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: "name": { 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: "type": "client", 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: "num": 24303 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": { 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: "type": "v1", 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": "192.168.0.1:0", 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: "nonce": 1598147928 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-12T05:06:53.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": { 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "type": "v1", 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": "192.168.0.1:0", 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "nonce": 1598147928 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "inst_str": "client.24303 192.168.0.1:0/1598147928", 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "addr_str": "192.168.0.1:0/1598147928", 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "inode_count": 1, 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_epoch": 14, 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_epoch": 43, 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_epoch_barrier": 0, 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "blocklisted": false, 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout: "fs_name": "cephfs" 2024-04-12T05:06:53.035 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-12T05:06:53.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:53.043 DEBUG:teuthology.orchestra.run.smithi113:> stat --file-system '--printf=%T 2024-04-12T05:06:53.043 DEBUG:teuthology.orchestra.run.smithi113:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-12T05:06:53.067 INFO:teuthology.orchestra.run.smithi113.stdout:fuseblk 2024-04-12T05:06:53.068 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-12T05:06:53.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:53.068 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-12T05:06:53.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:53.147 DEBUG:teuthology.orchestra.run.smithi192:> stat --file-system '--printf=%T 2024-04-12T05:06:53.147 DEBUG:teuthology.orchestra.run.smithi192:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-12T05:06:53.165 INFO:teuthology.orchestra.run.smithi192.stdout:fuseblk 2024-04-12T05:06:53.166 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-12T05:06:53.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:06:53.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-12T05:06:53.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:51.557114+0000 mgr.smithi113.ctsazg (mgr.14194) 224 : cluster [DBG] pgmap v195: 65 pgs: 65 active+clean; 4.4 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-12T05:06:53.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:51.557114+0000 mgr.smithi113.ctsazg (mgr.14194) 224 : cluster [DBG] pgmap v195: 65 pgs: 65 active+clean; 4.4 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-12T05:06:55.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:53.557529+0000 mgr.smithi113.ctsazg (mgr.14194) 225 : cluster [DBG] pgmap v196: 65 pgs: 65 active+clean; 4.4 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-12T05:06:55.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:53.557529+0000 mgr.smithi113.ctsazg (mgr.14194) 225 : cluster [DBG] pgmap v196: 65 pgs: 65 active+clean; 4.4 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-12T05:06:56.424 INFO:teuthology.run_tasks:Running task print... 2024-04-12T05:06:56.437 INFO:teuthology.task.print:**** done client 2024-04-12T05:06:56.437 INFO:teuthology.run_tasks:Running task parallel... 2024-04-12T05:06:56.472 INFO:teuthology.task.parallel:starting parallel... 2024-04-12T05:06:56.472 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-12T05:06:56.472 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-12T05:06:56.473 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:06:56.473 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-12T05:06:56.474 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-12T05:06:56.474 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-12T05:06:56.477 INFO:tasks.workunit:Pulling workunits from ref d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:06:56.477 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-12T05:06:56.477 INFO:tasks.workunit:timeout=3h 2024-04-12T05:06:56.478 INFO:tasks.workunit:cleanup=True 2024-04-12T05:06:56.478 DEBUG:teuthology.orchestra.run.smithi113:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-12T05:06:56.501 INFO:teuthology.orchestra.run.smithi113.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-12T05:06:56.502 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-12T05:06:56.502 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 95h/149d Inode: 1 Links: 2 2024-04-12T05:06:56.502 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:06:56.502 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-12T05:06:56.502 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-12T05:06:56.502 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-04-12 05:06:17.164647482 +0000 2024-04-12T05:06:56.502 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-04-12 05:06:53.142403626 +0000 2024-04-12T05:06:56.502 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: - 2024-04-12T05:06:56.502 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-12T05:06:56.502 DEBUG:teuthology.orchestra.run.smithi113:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-12T05:06:56.576 DEBUG:teuthology.orchestra.run.smithi192:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-12T05:06:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-12T05:06:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-12T05:06:56.601 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 89h/137d Inode: 1 Links: 3 2024-04-12T05:06:56.601 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T05:06:56.601 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-12T05:06:56.602 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-12T05:06:56.602 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-12 05:06:56.569583883 +0000 2024-04-12T05:06:56.602 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-12 05:06:56.569583883 +0000 2024-04-12T05:06:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2024-04-12T05:06:56.602 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-12T05:06:56.602 DEBUG:teuthology.orchestra.run.smithi192:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-12T05:06:56.679 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:06:56.680 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:06:56.700 INFO:tasks.workunit.client.0.smithi113.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-12T05:06:56.740 INFO:tasks.workunit.client.1.smithi192.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-12T05:06:57.636 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:55.558029+0000 mgr.smithi113.ctsazg (mgr.14194) 226 : cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 4.8 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:06:57.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:55.558029+0000 mgr.smithi113.ctsazg (mgr.14194) 226 : cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 4.8 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:06:58.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.550+0000 7fb924c71700 1 -- 172.21.15.113:0/579947898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9200fa940 msgr2=0x7fb9200fcd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:58.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.550+0000 7fb924c71700 1 --2- 172.21.15.113:0/579947898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9200fa940 0x7fb9200fcd30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fb9100047f0 tx=0x7fb910009cb0).stop 2024-04-12T05:06:58.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.550+0000 7fb924c71700 1 -- 172.21.15.113:0/579947898 shutdown_connections 2024-04-12T05:06:58.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.550+0000 7fb924c71700 1 --2- 172.21.15.113:0/579947898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9200fa940 0x7fb9200fcd30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:58.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.550+0000 7fb924c71700 1 --2- 172.21.15.113:0/579947898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb920061790 0x7fb9200fa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:58.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.550+0000 7fb924c71700 1 -- 172.21.15.113:0/579947898 >> 172.21.15.113:0/579947898 conn(0x7fb9200f1eb0 msgr2=0x7fb9200f42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:58.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.550+0000 7fb924c71700 1 -- 172.21.15.113:0/579947898 shutdown_connections 2024-04-12T05:06:58.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.550+0000 7fb924c71700 1 -- 172.21.15.113:0/579947898 wait complete. 2024-04-12T05:06:58.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb924c71700 1 Processor -- start 2024-04-12T05:06:58.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb924c71700 1 -- start start 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb924c71700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb920061790 0x7fb9200f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb924c71700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9200fa940 0x7fb9200f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb924c71700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9200f8250 con 0x7fb9200fa940 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb924c71700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9200f6840 con 0x7fb920061790 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb91dd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9200fa940 0x7fb9200f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb91dd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9200fa940 0x7fb9200f6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33040/0 (socket says 172.21.15.113:33040) 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb91dd9b700 1 -- 172.21.15.113:0/1746788169 learned_addr learned my addr 172.21.15.113:0/1746788169 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb91dd9b700 1 -- 172.21.15.113:0/1746788169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb920061790 msgr2=0x7fb9200f7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb91dd9b700 1 --2- 172.21.15.113:0/1746788169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb920061790 0x7fb9200f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb91dd9b700 1 -- 172.21.15.113:0/1746788169 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb910005040 con 0x7fb9200fa940 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb91dd9b700 1 --2- 172.21.15.113:0/1746788169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9200fa940 0x7fb9200f6350 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fb910005010 tx=0x7fb910009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:58.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb90f7fe700 1 -- 172.21.15.113:0/1746788169 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb91000e030 con 0x7fb9200fa940 2024-04-12T05:06:58.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb90f7fe700 1 -- 172.21.15.113:0/1746788169 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fb910007e90 con 0x7fb9200fa940 2024-04-12T05:06:58.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb924c71700 1 -- 172.21.15.113:0/1746788169 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9200f6ac0 con 0x7fb9200fa940 2024-04-12T05:06:58.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.551+0000 7fb90f7fe700 1 -- 172.21.15.113:0/1746788169 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb910018070 con 0x7fb9200fa940 2024-04-12T05:06:58.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.552+0000 7fb924c71700 1 -- 172.21.15.113:0/1746788169 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9200f6fa0 con 0x7fb9200fa940 2024-04-12T05:06:58.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.552+0000 7fb924c71700 1 -- 172.21.15.113:0/1746788169 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb920047540 con 0x7fb9200fa940 2024-04-12T05:06:58.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.553+0000 7fb90f7fe700 1 -- 172.21.15.113:0/1746788169 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fb91001d420 con 0x7fb9200fa940 2024-04-12T05:06:58.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.554+0000 7fb90f7fe700 1 --2- 172.21.15.113:0/1746788169 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb908061180 0x7fb908063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:06:58.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.554+0000 7fb90f7fe700 1 -- 172.21.15.113:0/1746788169 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7fb91007e170 con 0x7fb9200fa940 2024-04-12T05:06:58.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.554+0000 7fb91e59c700 1 --2- 172.21.15.113:0/1746788169 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb908061180 0x7fb908063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:06:58.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.554+0000 7fb91e59c700 1 --2- 172.21.15.113:0/1746788169 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb908061180 0x7fb908063640 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fb920061560 tx=0x7fb914007be0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:06:58.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.557+0000 7fb90f7fe700 1 -- 172.21.15.113:0/1746788169 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb910052120 con 0x7fb9200fa940 2024-04-12T05:06:58.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.721+0000 7fb924c71700 1 -- 172.21.15.113:0/1746788169 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fb9200fffc0 con 0x7fb9200fa940 2024-04-12T05:06:58.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.722+0000 7fb90f7fe700 1 -- 172.21.15.113:0/1746788169 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fb910051d10 con 0x7fb9200fa940 2024-04-12T05:06:58.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.723+0000 7fb924c71700 1 -- 172.21.15.113:0/1746788169 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb908061180 msgr2=0x7fb908063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:58.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.723+0000 7fb924c71700 1 --2- 172.21.15.113:0/1746788169 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb908061180 0x7fb908063640 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fb920061560 tx=0x7fb914007be0).stop 2024-04-12T05:06:58.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.723+0000 7fb924c71700 1 -- 172.21.15.113:0/1746788169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9200fa940 msgr2=0x7fb9200f6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:06:58.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.723+0000 7fb924c71700 1 --2- 172.21.15.113:0/1746788169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9200fa940 0x7fb9200f6350 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fb910005010 tx=0x7fb910009cb0).stop 2024-04-12T05:06:58.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.723+0000 7fb924c71700 1 -- 172.21.15.113:0/1746788169 shutdown_connections 2024-04-12T05:06:58.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.723+0000 7fb924c71700 1 --2- 172.21.15.113:0/1746788169 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb908061180 0x7fb908063640 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:58.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.723+0000 7fb924c71700 1 --2- 172.21.15.113:0/1746788169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb920061790 0x7fb9200f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:06:58.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.723+0000 7fb924c71700 1 --2- 172.21.15.113:0/1746788169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9200fa940 0x7fb9200f6350 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:06:58.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.723+0000 7fb924c71700 1 -- 172.21.15.113:0/1746788169 >> 172.21.15.113:0/1746788169 conn(0x7fb9200f1eb0 msgr2=0x7fb9200f42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:06:58.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.723+0000 7fb924c71700 1 -- 172.21.15.113:0/1746788169 shutdown_connections 2024-04-12T05:06:58.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:06:58.724+0000 7fb924c71700 1 -- 172.21.15.113:0/1746788169 wait complete. 2024-04-12T05:06:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:06:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:57.558458+0000 mgr.smithi113.ctsazg (mgr.14194) 227 : cluster [DBG] pgmap v198: 65 pgs: 65 active+clean; 6.0 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-04-12T05:06:59.554 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-12T05:06:59.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:06:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:57.558458+0000 mgr.smithi113.ctsazg (mgr.14194) 227 : cluster [DBG] pgmap v198: 65 pgs: 65 active+clean; 6.0 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-04-12T05:07:01.482 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:06:59.558882+0000 mgr.smithi113.ctsazg (mgr.14194) 228 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 6.0 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 1 op/s 2024-04-12T05:07:01.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:06:59.558882+0000 mgr.smithi113.ctsazg (mgr.14194) 228 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 6.0 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 1 op/s 2024-04-12T05:07:01.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.734+0000 7f45c32fb700 1 -- 172.21.15.113:0/1623386976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45bc0f7990 msgr2=0x7f45bc0f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.734+0000 7f45c32fb700 1 --2- 172.21.15.113:0/1623386976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45bc0f7990 0x7f45bc0f7df0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f45ac0047f0 tx=0x7f45ac009cb0).stop 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.734+0000 7f45c32fb700 1 -- 172.21.15.113:0/1623386976 shutdown_connections 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.734+0000 7f45c32fb700 1 --2- 172.21.15.113:0/1623386976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45bc0f7990 0x7f45bc0f7df0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.734+0000 7f45c32fb700 1 --2- 172.21.15.113:0/1623386976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45bc0f66f0 0x7f45bc0f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.734+0000 7f45c32fb700 1 -- 172.21.15.113:0/1623386976 >> 172.21.15.113:0/1623386976 conn(0x7f45bc0f1e50 msgr2=0x7f45bc0f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.734+0000 7f45c32fb700 1 -- 172.21.15.113:0/1623386976 shutdown_connections 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.734+0000 7f45c32fb700 1 -- 172.21.15.113:0/1623386976 wait complete. 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c32fb700 1 Processor -- start 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c32fb700 1 -- start start 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c32fb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45bc0f66f0 0x7f45bc0fa670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c32fb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45bc0f7990 0x7f45bc0f8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:01.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c32fb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0fac20 con 0x7f45bc0f7990 2024-04-12T05:07:01.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c32fb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0f91c0 con 0x7f45bc0f66f0 2024-04-12T05:07:01.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c0896700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45bc0f7990 0x7f45bc0f8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:01.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c1097700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45bc0f66f0 0x7f45bc0fa670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:01.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c0896700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45bc0f7990 0x7f45bc0f8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41138/0 (socket says 172.21.15.113:41138) 2024-04-12T05:07:01.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c0896700 1 -- 172.21.15.113:0/835115552 learned_addr learned my addr 172.21.15.113:0/835115552 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:07:01.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c1097700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45bc0f66f0 0x7f45bc0fa670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.113:38324/0 (socket says 172.21.15.113:38324) 2024-04-12T05:07:01.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c0896700 1 -- 172.21.15.113:0/835115552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45bc0f66f0 msgr2=0x7f45bc0fa670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:01.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c0896700 1 --2- 172.21.15.113:0/835115552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45bc0f66f0 0x7f45bc0fa670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:01.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.735+0000 7f45c0896700 1 -- 172.21.15.113:0/835115552 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac005040 con 0x7f45bc0f7990 2024-04-12T05:07:01.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.736+0000 7f45c0896700 1 --2- 172.21.15.113:0/835115552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45bc0f7990 0x7f45bc0f8cd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f45ac013470 tx=0x7f45ac009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:01.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.736+0000 7f45b27fc700 1 -- 172.21.15.113:0/835115552 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45ac00e030 con 0x7f45bc0f7990 2024-04-12T05:07:01.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.736+0000 7f45b27fc700 1 -- 172.21.15.113:0/835115552 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7f45ac007e90 con 0x7f45bc0f7990 2024-04-12T05:07:01.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.736+0000 7f45c32fb700 1 -- 172.21.15.113:0/835115552 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45bc0f93e0 con 0x7f45bc0f7990 2024-04-12T05:07:01.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.736+0000 7f45b27fc700 1 -- 172.21.15.113:0/835115552 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45ac018070 con 0x7f45bc0f7990 2024-04-12T05:07:01.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.736+0000 7f45c32fb700 1 -- 172.21.15.113:0/835115552 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45bc0f98c0 con 0x7f45bc0f7990 2024-04-12T05:07:01.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.737+0000 7f45c32fb700 1 -- 172.21.15.113:0/835115552 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45bc0f9b30 con 0x7f45bc0f7990 2024-04-12T05:07:01.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.738+0000 7f45b27fc700 1 -- 172.21.15.113:0/835115552 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f45ac01d420 con 0x7f45bc0f7990 2024-04-12T05:07:01.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.738+0000 7f45b27fc700 1 --2- 172.21.15.113:0/835115552 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f45a8061180 0x7f45a8063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:01.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.738+0000 7f45b27fc700 1 -- 172.21.15.113:0/835115552 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f45ac07e1c0 con 0x7f45bc0f7990 2024-04-12T05:07:01.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.738+0000 7f45c1097700 1 --2- 172.21.15.113:0/835115552 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f45a8061180 0x7f45a8063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:01.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.738+0000 7f45c1097700 1 --2- 172.21.15.113:0/835115552 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f45a8061180 0x7f45a8063640 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f45bc0f77f0 tx=0x7f45b800b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:01.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.742+0000 7f45b27fc700 1 -- 172.21.15.113:0/835115552 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f45ac0521d0 con 0x7f45bc0f7990 2024-04-12T05:07:01.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.931+0000 7f45c32fb700 1 -- 172.21.15.113:0/835115552 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f45bc0feeb0 con 0x7f45bc0f7990 2024-04-12T05:07:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.931+0000 7f45b27fc700 1 -- 172.21.15.113:0/835115552 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f45ac051dc0 con 0x7f45bc0f7990 2024-04-12T05:07:01.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.933+0000 7f45c32fb700 1 -- 172.21.15.113:0/835115552 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f45a8061180 msgr2=0x7f45a8063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:01.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.933+0000 7f45c32fb700 1 --2- 172.21.15.113:0/835115552 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f45a8061180 0x7f45a8063640 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f45bc0f77f0 tx=0x7f45b800b040).stop 2024-04-12T05:07:01.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.933+0000 7f45c32fb700 1 -- 172.21.15.113:0/835115552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45bc0f7990 msgr2=0x7f45bc0f8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:01.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.933+0000 7f45c32fb700 1 --2- 172.21.15.113:0/835115552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45bc0f7990 0x7f45bc0f8cd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f45ac013470 tx=0x7f45ac009cb0).stop 2024-04-12T05:07:01.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.933+0000 7f45c32fb700 1 -- 172.21.15.113:0/835115552 shutdown_connections 2024-04-12T05:07:01.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.933+0000 7f45c32fb700 1 --2- 172.21.15.113:0/835115552 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f45a8061180 0x7f45a8063640 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:01.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.933+0000 7f45c32fb700 1 --2- 172.21.15.113:0/835115552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45bc0f66f0 0x7f45bc0fa670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:01.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.933+0000 7f45c32fb700 1 --2- 172.21.15.113:0/835115552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45bc0f7990 0x7f45bc0f8cd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:01.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.933+0000 7f45c32fb700 1 -- 172.21.15.113:0/835115552 >> 172.21.15.113:0/835115552 conn(0x7f45bc0f1e50 msgr2=0x7f45bc060d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:01.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.933+0000 7f45c32fb700 1 -- 172.21.15.113:0/835115552 shutdown_connections 2024-04-12T05:07:01.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:01.933+0000 7f45c32fb700 1 -- 172.21.15.113:0/835115552 wait complete. 2024-04-12T05:07:02.613 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-12T05:07:03.349 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:01.559248+0000 mgr.smithi113.ctsazg (mgr.14194) 229 : cluster [DBG] pgmap v200: 65 pgs: 65 active+clean; 9.9 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 1 op/s 2024-04-12T05:07:03.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:01.559248+0000 mgr.smithi113.ctsazg (mgr.14194) 229 : cluster [DBG] pgmap v200: 65 pgs: 65 active+clean; 9.9 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 1 op/s 2024-04-12T05:07:04.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.749+0000 7fabd6700700 1 -- 172.21.15.113:0/4087605740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0061280 msgr2=0x7fabd00fa3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:04.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.749+0000 7fabd6700700 1 --2- 172.21.15.113:0/4087605740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0061280 0x7fabd00fa3d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fabc00047f0 tx=0x7fabc0009cb0).stop 2024-04-12T05:07:04.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.749+0000 7fabd6700700 1 -- 172.21.15.113:0/4087605740 shutdown_connections 2024-04-12T05:07:04.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.749+0000 7fabd6700700 1 --2- 172.21.15.113:0/4087605740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd00fa950 0x7fabd00fcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:04.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.749+0000 7fabd6700700 1 --2- 172.21.15.113:0/4087605740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0061280 0x7fabd00fa3d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:04.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.749+0000 7fabd6700700 1 -- 172.21.15.113:0/4087605740 >> 172.21.15.113:0/4087605740 conn(0x7fabd00f1e50 msgr2=0x7fabd00f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:04.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.749+0000 7fabd6700700 1 -- 172.21.15.113:0/4087605740 shutdown_connections 2024-04-12T05:07:04.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.749+0000 7fabd6700700 1 -- 172.21.15.113:0/4087605740 wait complete. 2024-04-12T05:07:04.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.749+0000 7fabd6700700 1 Processor -- start 2024-04-12T05:07:04.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.749+0000 7fabd6700700 1 -- start start 2024-04-12T05:07:04.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabd6700700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0061280 0x7fabd00f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:04.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabd6700700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd00fa950 0x7fabd00f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:04.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabd6700700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00f8220 con 0x7fabd0061280 2024-04-12T05:07:04.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabd6700700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00f67c0 con 0x7fabd00fa950 2024-04-12T05:07:04.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabcffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0061280 0x7fabd00f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:04.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabcffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0061280 0x7fabd00f7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41144/0 (socket says 172.21.15.113:41144) 2024-04-12T05:07:04.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabcffff700 1 -- 172.21.15.113:0/3961865495 learned_addr learned my addr 172.21.15.113:0/3961865495 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:07:04.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabcf7fe700 1 --2- 172.21.15.113:0/3961865495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd00fa950 0x7fabd00f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:04.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabcffff700 1 -- 172.21.15.113:0/3961865495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd00fa950 msgr2=0x7fabd00f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:04.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabcffff700 1 --2- 172.21.15.113:0/3961865495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd00fa950 0x7fabd00f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:04.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabcffff700 1 -- 172.21.15.113:0/3961865495 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc0005040 con 0x7fabd0061280 2024-04-12T05:07:04.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.750+0000 7fabcffff700 1 --2- 172.21.15.113:0/3961865495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0061280 0x7fabd00f7c70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fabc0013580 tx=0x7fabc0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:04.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.751+0000 7fabcd7fa700 1 -- 172.21.15.113:0/3961865495 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabc0018070 con 0x7fabd0061280 2024-04-12T05:07:04.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.751+0000 7fabd6700700 1 -- 172.21.15.113:0/3961865495 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd00f69e0 con 0x7fabd0061280 2024-04-12T05:07:04.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.751+0000 7fabcd7fa700 1 -- 172.21.15.113:0/3961865495 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(25 keys) v1 ==== 1046+0+0 (secure 0 0 0) 0x7fabc0007e90 con 0x7fabd0061280 2024-04-12T05:07:04.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.751+0000 7fabcd7fa700 1 -- 172.21.15.113:0/3961865495 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabc00123c0 con 0x7fabd0061280 2024-04-12T05:07:04.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.751+0000 7fabd6700700 1 -- 172.21.15.113:0/3961865495 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd00f6f40 con 0x7fabd0061280 2024-04-12T05:07:04.755 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.752+0000 7fabd6700700 1 -- 172.21.15.113:0/3961865495 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd0047540 con 0x7fabd0061280 2024-04-12T05:07:04.755 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.753+0000 7fabcd7fa700 1 -- 172.21.15.113:0/3961865495 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fabc001d420 con 0x7fabd0061280 2024-04-12T05:07:04.755 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.753+0000 7fabcd7fa700 1 --2- 172.21.15.113:0/3961865495 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fabb8061180 0x7fabb8063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:04.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.753+0000 7fabcd7fa700 1 -- 172.21.15.113:0/3961865495 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7fabc007d4a0 con 0x7fabd0061280 2024-04-12T05:07:04.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.753+0000 7fabcf7fe700 1 --2- 172.21.15.113:0/3961865495 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fabb8061180 0x7fabb8063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:04.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.754+0000 7fabcf7fe700 1 --2- 172.21.15.113:0/3961865495 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fabb8061180 0x7fabb8063640 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fabd0061050 tx=0x7fabc400b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:04.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.758+0000 7fabcd7fa700 1 -- 172.21.15.113:0/3961865495 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fabc0051430 con 0x7fabd0061280 2024-04-12T05:07:04.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.938+0000 7fabd6700700 1 -- 172.21.15.113:0/3961865495 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fabd00fffe0 con 0x7fabd0061280 2024-04-12T05:07:04.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.943+0000 7fabcd7fa700 1 -- 172.21.15.113:0/3961865495 <== mon.0 v2:172.21.15.113:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabc0051430 con 0x7fabd0061280 2024-04-12T05:07:04.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.943+0000 7fabcd7fa700 1 -- 172.21.15.113:0/3961865495 <== mon.0 v2:172.21.15.113:3300/0 8 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7fabc0051020 con 0x7fabd0061280 2024-04-12T05:07:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.946+0000 7fabd6700700 1 -- 172.21.15.113:0/3961865495 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fabb8061180 msgr2=0x7fabb8063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.946+0000 7fabd6700700 1 --2- 172.21.15.113:0/3961865495 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fabb8061180 0x7fabb8063640 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fabd0061050 tx=0x7fabc400b040).stop 2024-04-12T05:07:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.946+0000 7fabd6700700 1 -- 172.21.15.113:0/3961865495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0061280 msgr2=0x7fabd00f7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.946+0000 7fabd6700700 1 --2- 172.21.15.113:0/3961865495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0061280 0x7fabd00f7c70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fabc0013580 tx=0x7fabc0009cb0).stop 2024-04-12T05:07:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.946+0000 7fabd6700700 1 -- 172.21.15.113:0/3961865495 shutdown_connections 2024-04-12T05:07:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.946+0000 7fabd6700700 1 --2- 172.21.15.113:0/3961865495 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fabb8061180 0x7fabb8063640 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.946+0000 7fabd6700700 1 --2- 172.21.15.113:0/3961865495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd0061280 0x7fabd00f7c70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.946+0000 7fabd6700700 1 --2- 172.21.15.113:0/3961865495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd00fa950 0x7fabd00f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.946+0000 7fabd6700700 1 -- 172.21.15.113:0/3961865495 >> 172.21.15.113:0/3961865495 conn(0x7fabd00f1e50 msgr2=0x7fabd00f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.946+0000 7fabd6700700 1 -- 172.21.15.113:0/3961865495 shutdown_connections 2024-04-12T05:07:04.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:04.946+0000 7fabd6700700 1 -- 172.21.15.113:0/3961865495 wait complete. 2024-04-12T05:07:05.659 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:03.559584+0000 mgr.smithi113.ctsazg (mgr.14194) 230 : cluster [DBG] pgmap v201: 65 pgs: 65 active+clean; 9.9 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-12T05:07:05.659 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:04.943488+0000 mon.smithi113 (mon.0) 606 : audit [INF] from='client.? 172.21.15.113:0/3961865495' entity='client.admin' 2024-04-12T05:07:05.660 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:04.964085+0000 mon.smithi113 (mon.0) 607 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:07:05.660 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:05.272153+0000 mon.smithi113 (mon.0) 608 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:07:05.660 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:05.272865+0000 mon.smithi113 (mon.0) 609 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:07:05.660 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:05.277131+0000 mon.smithi113 (mon.0) 610 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:07:05.679 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-12T05:07:05.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:03.559584+0000 mgr.smithi113.ctsazg (mgr.14194) 230 : cluster [DBG] pgmap v201: 65 pgs: 65 active+clean; 9.9 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-04-12T05:07:05.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:04.943488+0000 mon.smithi113 (mon.0) 606 : audit [INF] from='client.? 172.21.15.113:0/3961865495' entity='client.admin' 2024-04-12T05:07:05.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:04.964085+0000 mon.smithi113 (mon.0) 607 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:07:05.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:05.272153+0000 mon.smithi113 (mon.0) 608 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:07:05.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:05.272865+0000 mon.smithi113 (mon.0) 609 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:07:05.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:05.277131+0000 mon.smithi113 (mon.0) 610 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:07:07.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:05.559941+0000 mgr.smithi113.ctsazg (mgr.14194) 231 : cluster [DBG] pgmap v202: 65 pgs: 65 active+clean; 9.9 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s wr, 1 op/s 2024-04-12T05:07:07.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:05.559941+0000 mgr.smithi113.ctsazg (mgr.14194) 231 : cluster [DBG] pgmap v202: 65 pgs: 65 active+clean; 9.9 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s wr, 1 op/s 2024-04-12T05:07:07.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.977+0000 7f0d95619700 1 -- 172.21.15.113:0/2545895353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d900f8aa0 msgr2=0x7f0d900f8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:07.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.977+0000 7f0d95619700 1 --2- 172.21.15.113:0/2545895353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d900f8aa0 0x7f0d900f8f00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f0d840047f0 tx=0x7f0d84009cb0).stop 2024-04-12T05:07:07.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.977+0000 7f0d95619700 1 -- 172.21.15.113:0/2545895353 shutdown_connections 2024-04-12T05:07:07.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.977+0000 7f0d95619700 1 --2- 172.21.15.113:0/2545895353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d900f8aa0 0x7f0d900f8f00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:07.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.977+0000 7f0d95619700 1 --2- 172.21.15.113:0/2545895353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d900f7800 0x7f0d900f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:07.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.977+0000 7f0d95619700 1 -- 172.21.15.113:0/2545895353 >> 172.21.15.113:0/2545895353 conn(0x7f0d900f2f80 msgr2=0x7f0d900f53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:07.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.977+0000 7f0d95619700 1 -- 172.21.15.113:0/2545895353 shutdown_connections 2024-04-12T05:07:07.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.977+0000 7f0d95619700 1 -- 172.21.15.113:0/2545895353 wait complete. 2024-04-12T05:07:07.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.978+0000 7f0d95619700 1 Processor -- start 2024-04-12T05:07:07.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.978+0000 7f0d95619700 1 -- start start 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.978+0000 7f0d95619700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d900f7800 0x7f0d9006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.978+0000 7f0d95619700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d900f8aa0 0x7f0d9006b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.978+0000 7f0d95619700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9006cf80 con 0x7f0d900f8aa0 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.978+0000 7f0d95619700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9006b520 con 0x7f0d900f7800 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.978+0000 7f0d8e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d900f8aa0 0x7f0d9006b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.978+0000 7f0d8e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d900f8aa0 0x7f0d9006b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41154/0 (socket says 172.21.15.113:41154) 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.978+0000 7f0d8e7fc700 1 -- 172.21.15.113:0/4176363749 learned_addr learned my addr 172.21.15.113:0/4176363749 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.978+0000 7f0d8effd700 1 --2- 172.21.15.113:0/4176363749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d900f7800 0x7f0d9006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.979+0000 7f0d8e7fc700 1 -- 172.21.15.113:0/4176363749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d900f7800 msgr2=0x7f0d9006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.979+0000 7f0d8e7fc700 1 --2- 172.21.15.113:0/4176363749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d900f7800 0x7f0d9006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.979+0000 7f0d8e7fc700 1 -- 172.21.15.113:0/4176363749 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d84005040 con 0x7f0d900f8aa0 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.979+0000 7f0d8e7fc700 1 --2- 172.21.15.113:0/4176363749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d900f8aa0 0x7f0d9006b030 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f0d84013620 tx=0x7f0d84009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.979+0000 7f0d77fff700 1 -- 172.21.15.113:0/4176363749 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d8400e070 con 0x7f0d900f8aa0 2024-04-12T05:07:07.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.979+0000 7f0d77fff700 1 -- 172.21.15.113:0/4176363749 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d84007e90 con 0x7f0d900f8aa0 2024-04-12T05:07:07.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.979+0000 7f0d95619700 1 -- 172.21.15.113:0/4176363749 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9006b740 con 0x7f0d900f8aa0 2024-04-12T05:07:07.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.979+0000 7f0d77fff700 1 -- 172.21.15.113:0/4176363749 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d84018070 con 0x7f0d900f8aa0 2024-04-12T05:07:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.979+0000 7f0d95619700 1 -- 172.21.15.113:0/4176363749 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9006bca0 con 0x7f0d900f8aa0 2024-04-12T05:07:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.980+0000 7f0d95619700 1 -- 172.21.15.113:0/4176363749 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d900fffe0 con 0x7f0d900f8aa0 2024-04-12T05:07:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.980+0000 7f0d77fff700 1 -- 172.21.15.113:0/4176363749 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f0d8401d420 con 0x7f0d900f8aa0 2024-04-12T05:07:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.981+0000 7f0d77fff700 1 --2- 172.21.15.113:0/4176363749 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0d78061180 0x7f0d78063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.981+0000 7f0d77fff700 1 -- 172.21.15.113:0/4176363749 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f0d8407d560 con 0x7f0d900f8aa0 2024-04-12T05:07:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.981+0000 7f0d8effd700 1 --2- 172.21.15.113:0/4176363749 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0d78061180 0x7f0d78063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.981+0000 7f0d8effd700 1 --2- 172.21.15.113:0/4176363749 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0d78061180 0x7f0d78063640 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0d900f8900 tx=0x7f0d8000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:07.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:07.985+0000 7f0d77fff700 1 -- 172.21.15.113:0/4176363749 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0d84051570 con 0x7f0d900f8aa0 2024-04-12T05:07:08.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:08.176+0000 7f0d95619700 1 -- 172.21.15.113:0/4176363749 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d900591d0 con 0x7f0d78061180 2024-04-12T05:07:09.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:07.560344+0000 mgr.smithi113.ctsazg (mgr.14194) 232 : cluster [DBG] pgmap v203: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-04-12T05:07:09.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:08.177551+0000 mgr.smithi113.ctsazg (mgr.14194) 233 : audit [DBG] from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:07:09.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:07.560344+0000 mgr.smithi113.ctsazg (mgr.14194) 232 : cluster [DBG] pgmap v203: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 0 op/s 2024-04-12T05:07:09.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:08.177551+0000 mgr.smithi113.ctsazg (mgr.14194) 233 : audit [DBG] from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:07:11.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:09.560735+0000 mgr.smithi113.ctsazg (mgr.14194) 234 : cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-04-12T05:07:11.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:09.560735+0000 mgr.smithi113.ctsazg (mgr.14194) 234 : cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 0 op/s 2024-04-12T05:07:13.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:11.561206+0000 mgr.smithi113.ctsazg (mgr.14194) 235 : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-12T05:07:13.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:11.561206+0000 mgr.smithi113.ctsazg (mgr.14194) 235 : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-04-12T05:07:15.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:13.566126+0000 mgr.smithi113.ctsazg (mgr.14194) 236 : cluster [DBG] pgmap v206: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 596 B/s wr, 0 op/s 2024-04-12T05:07:15.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:13.566126+0000 mgr.smithi113.ctsazg (mgr.14194) 236 : cluster [DBG] pgmap v206: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 596 B/s wr, 0 op/s 2024-04-12T05:07:17.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:15.566560+0000 mgr.smithi113.ctsazg (mgr.14194) 237 : cluster [DBG] pgmap v207: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 596 B/s wr, 0 op/s 2024-04-12T05:07:17.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:15.566560+0000 mgr.smithi113.ctsazg (mgr.14194) 237 : cluster [DBG] pgmap v207: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 596 B/s wr, 0 op/s 2024-04-12T05:07:19.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:17.567017+0000 mgr.smithi113.ctsazg (mgr.14194) 238 : cluster [DBG] pgmap v208: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:07:19.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:17.567017+0000 mgr.smithi113.ctsazg (mgr.14194) 238 : cluster [DBG] pgmap v208: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:07:21.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:19.567379+0000 mgr.smithi113.ctsazg (mgr.14194) 239 : cluster [DBG] pgmap v209: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-12T05:07:21.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:19.567379+0000 mgr.smithi113.ctsazg (mgr.14194) 239 : cluster [DBG] pgmap v209: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-12T05:07:23.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:21.567846+0000 mgr.smithi113.ctsazg (mgr.14194) 240 : cluster [DBG] pgmap v210: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:23.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:21.567846+0000 mgr.smithi113.ctsazg (mgr.14194) 240 : cluster [DBG] pgmap v210: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:25.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:23.568273+0000 mgr.smithi113.ctsazg (mgr.14194) 241 : cluster [DBG] pgmap v211: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:25.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:23.568273+0000 mgr.smithi113.ctsazg (mgr.14194) 241 : cluster [DBG] pgmap v211: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:27.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:25.568709+0000 mgr.smithi113.ctsazg (mgr.14194) 242 : cluster [DBG] pgmap v212: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:27.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:25.568709+0000 mgr.smithi113.ctsazg (mgr.14194) 242 : cluster [DBG] pgmap v212: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:29.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:27.569489+0000 mgr.smithi113.ctsazg (mgr.14194) 243 : cluster [DBG] pgmap v213: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:29.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:27.569489+0000 mgr.smithi113.ctsazg (mgr.14194) 243 : cluster [DBG] pgmap v213: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:31.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:29.569961+0000 mgr.smithi113.ctsazg (mgr.14194) 244 : cluster [DBG] pgmap v214: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:29.569961+0000 mgr.smithi113.ctsazg (mgr.14194) 244 : cluster [DBG] pgmap v214: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:33.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:31.570396+0000 mgr.smithi113.ctsazg (mgr.14194) 245 : cluster [DBG] pgmap v215: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:33.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:31.570396+0000 mgr.smithi113.ctsazg (mgr.14194) 245 : cluster [DBG] pgmap v215: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:35.682 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:33.570805+0000 mgr.smithi113.ctsazg (mgr.14194) 246 : cluster [DBG] pgmap v216: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:35.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:33.570805+0000 mgr.smithi113.ctsazg (mgr.14194) 246 : cluster [DBG] pgmap v216: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:37.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:35.571237+0000 mgr.smithi113.ctsazg (mgr.14194) 247 : cluster [DBG] pgmap v217: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:37.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:35.571237+0000 mgr.smithi113.ctsazg (mgr.14194) 247 : cluster [DBG] pgmap v217: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:38.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.598+0000 7f0d77fff700 1 -- 172.21.15.113:0/4176363749 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f0d900591d0 con 0x7f0d78061180 2024-04-12T05:07:38.600 INFO:teuthology.orchestra.run.smithi113.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:07:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.600+0000 7f0d95619700 1 -- 172.21.15.113:0/4176363749 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0d78061180 msgr2=0x7f0d78063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.600+0000 7f0d95619700 1 --2- 172.21.15.113:0/4176363749 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0d78061180 0x7f0d78063640 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0d900f8900 tx=0x7f0d8000b040).stop 2024-04-12T05:07:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.600+0000 7f0d95619700 1 -- 172.21.15.113:0/4176363749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d900f8aa0 msgr2=0x7f0d9006b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.600+0000 7f0d95619700 1 --2- 172.21.15.113:0/4176363749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d900f8aa0 0x7f0d9006b030 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f0d84013620 tx=0x7f0d84009cb0).stop 2024-04-12T05:07:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.600+0000 7f0d95619700 1 -- 172.21.15.113:0/4176363749 shutdown_connections 2024-04-12T05:07:38.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.600+0000 7f0d95619700 1 --2- 172.21.15.113:0/4176363749 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0d78061180 0x7f0d78063640 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:38.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.600+0000 7f0d95619700 1 --2- 172.21.15.113:0/4176363749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d900f7800 0x7f0d9006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:38.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.600+0000 7f0d95619700 1 --2- 172.21.15.113:0/4176363749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d900f8aa0 0x7f0d9006b030 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:38.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.600+0000 7f0d95619700 1 -- 172.21.15.113:0/4176363749 >> 172.21.15.113:0/4176363749 conn(0x7f0d900f2f80 msgr2=0x7f0d900fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:38.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.600+0000 7f0d95619700 1 -- 172.21.15.113:0/4176363749 shutdown_connections 2024-04-12T05:07:38.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:38.600+0000 7f0d95619700 1 -- 172.21.15.113:0/4176363749 wait complete. 2024-04-12T05:07:39.321 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-12T05:07:39.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:37.571683+0000 mgr.smithi113.ctsazg (mgr.14194) 248 : cluster [DBG] pgmap v218: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:39.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:38.598934+0000 mon.smithi113 (mon.0) 611 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:07:39.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:38.649164+0000 mon.smithi113 (mon.0) 612 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:07:39.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:37.571683+0000 mgr.smithi113.ctsazg (mgr.14194) 248 : cluster [DBG] pgmap v218: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:39.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:38.598934+0000 mon.smithi113 (mon.0) 611 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:07:39.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:38.649164+0000 mon.smithi113 (mon.0) 612 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:07:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:07:38.594460+0000 mgr.smithi113.ctsazg (mgr.14194) 249 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:07:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:39.472850+0000 mon.smithi113 (mon.0) 613 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:07:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:39.473796+0000 mon.smithi113 (mon.0) 614 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:07:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:39.481273+0000 mon.smithi113 (mon.0) 615 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:07:40.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:07:38.594460+0000 mgr.smithi113.ctsazg (mgr.14194) 249 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:07:40.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:39.472850+0000 mon.smithi113 (mon.0) 613 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:07:40.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:39.473796+0000 mon.smithi113 (mon.0) 614 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:07:40.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:39.481273+0000 mon.smithi113 (mon.0) 615 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:07:41.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:07:39.487516+0000 mgr.smithi113.ctsazg (mgr.14194) 250 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:07:41.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:39.572147+0000 mgr.smithi113.ctsazg (mgr.14194) 251 : cluster [DBG] pgmap v219: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:41.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:07:39.487516+0000 mgr.smithi113.ctsazg (mgr.14194) 250 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:07:41.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:39.572147+0000 mgr.smithi113.ctsazg (mgr.14194) 251 : cluster [DBG] pgmap v219: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.517+0000 7f371a51e700 1 -- 172.21.15.113:0/4008411150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f6450 msgr2=0x7f37140f8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.517+0000 7f371a51e700 1 --2- 172.21.15.113:0/4008411150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f6450 0x7f37140f8bd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f37080047f0 tx=0x7f3708009cb0).stop 2024-04-12T05:07:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.517+0000 7f371a51e700 1 -- 172.21.15.113:0/4008411150 shutdown_connections 2024-04-12T05:07:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.517+0000 7f371a51e700 1 --2- 172.21.15.113:0/4008411150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f6450 0x7f37140f8bd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.517+0000 7f371a51e700 1 --2- 172.21.15.113:0/4008411150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f5ab0 0x7f37140f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.517+0000 7f371a51e700 1 -- 172.21.15.113:0/4008411150 >> 172.21.15.113:0/4008411150 conn(0x7f37140f15d0 msgr2=0x7f37140f39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.518+0000 7f371a51e700 1 -- 172.21.15.113:0/4008411150 shutdown_connections 2024-04-12T05:07:42.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.518+0000 7f371a51e700 1 -- 172.21.15.113:0/4008411150 wait complete. 2024-04-12T05:07:42.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.518+0000 7f371a51e700 1 Processor -- start 2024-04-12T05:07:42.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.518+0000 7f371a51e700 1 -- start start 2024-04-12T05:07:42.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.519+0000 7f371a51e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f5ab0 0x7f3714069f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:42.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.519+0000 7f371a51e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f6450 0x7f371406a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:42.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.519+0000 7f371a51e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371406aa30 con 0x7f37140f6450 2024-04-12T05:07:42.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.519+0000 7f371a51e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371406ab70 con 0x7f37140f5ab0 2024-04-12T05:07:42.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.519+0000 7f3713fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f5ab0 0x7f3714069f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:42.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.519+0000 7f3713fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f5ab0 0x7f3714069f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.113:55482/0 (socket says 172.21.15.113:55482) 2024-04-12T05:07:42.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.519+0000 7f3713fff700 1 -- 172.21.15.113:0/2998377898 learned_addr learned my addr 172.21.15.113:0/2998377898 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:07:42.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.519+0000 7f37137fe700 1 --2- 172.21.15.113:0/2998377898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f6450 0x7f371406a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:42.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.519+0000 7f3713fff700 1 -- 172.21.15.113:0/2998377898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f6450 msgr2=0x7f371406a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:42.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.519+0000 7f3713fff700 1 --2- 172.21.15.113:0/2998377898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f6450 0x7f371406a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:42.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.519+0000 7f3713fff700 1 -- 172.21.15.113:0/2998377898 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3708005040 con 0x7f37140f5ab0 2024-04-12T05:07:42.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.520+0000 7f3713fff700 1 --2- 172.21.15.113:0/2998377898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f5ab0 0x7f3714069f90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f370400e8c0 tx=0x7f370400b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:42.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.520+0000 7f37117fa700 1 -- 172.21.15.113:0/2998377898 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3704007b40 con 0x7f37140f5ab0 2024-04-12T05:07:42.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.520+0000 7f371a51e700 1 -- 172.21.15.113:0/2998377898 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3714115f70 con 0x7f37140f5ab0 2024-04-12T05:07:42.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.520+0000 7f37117fa700 1 -- 172.21.15.113:0/2998377898 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f370400f040 con 0x7f37140f5ab0 2024-04-12T05:07:42.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.520+0000 7f37117fa700 1 -- 172.21.15.113:0/2998377898 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3704014880 con 0x7f37140f5ab0 2024-04-12T05:07:42.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.520+0000 7f371a51e700 1 -- 172.21.15.113:0/2998377898 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37140f6a40 con 0x7f37140f5ab0 2024-04-12T05:07:42.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.521+0000 7f371a51e700 1 -- 172.21.15.113:0/2998377898 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37140fffd0 con 0x7f37140f5ab0 2024-04-12T05:07:42.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.522+0000 7f37117fa700 1 -- 172.21.15.113:0/2998377898 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f3704007ca0 con 0x7f37140f5ab0 2024-04-12T05:07:42.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.522+0000 7f37117fa700 1 --2- 172.21.15.113:0/2998377898 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f36fc065590 0x7f36fc067a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:42.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.522+0000 7f37117fa700 1 -- 172.21.15.113:0/2998377898 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f370407c510 con 0x7f37140f5ab0 2024-04-12T05:07:42.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.522+0000 7f37137fe700 1 --2- 172.21.15.113:0/2998377898 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f36fc065590 0x7f36fc067a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:42.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.523+0000 7f37137fe700 1 --2- 172.21.15.113:0/2998377898 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f36fc065590 0x7f36fc067a50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f37080047c0 tx=0x7f3708009cb0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:42.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.526+0000 7f37117fa700 1 -- 172.21.15.113:0/2998377898 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3704050420 con 0x7f37140f5ab0 2024-04-12T05:07:42.542 INFO:tasks.workunit.client.1.smithi192.stderr:Note: switching to 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'. 2024-04-12T05:07:42.542 INFO:tasks.workunit.client.1.smithi192.stderr: 2024-04-12T05:07:42.542 INFO:tasks.workunit.client.1.smithi192.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-12T05:07:42.542 INFO:tasks.workunit.client.1.smithi192.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr:state without impacting any branches by switching back to a branch. 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr: 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr: 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr: git switch -c 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr: 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr:Or undo this operation with: 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr: 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr: git switch - 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr: 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr: 2024-04-12T05:07:42.543 INFO:tasks.workunit.client.1.smithi192.stderr:HEAD is now at d540ebaca6b Merge pull request #56672 from batrick/wip-65294-reef 2024-04-12T05:07:42.548 DEBUG:teuthology.orchestra.run.smithi192:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-12T05:07:42.610 INFO:tasks.workunit.client.1.smithi192.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-12T05:07:42.612 INFO:tasks.workunit.client.1.smithi192.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-12T05:07:42.612 INFO:tasks.workunit.client.1.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-12T05:07:42.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.816+0000 7f371a51e700 1 -- 172.21.15.113:0/2998377898 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3714059120 con 0x7f36fc065590 2024-04-12T05:07:42.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.817+0000 7f37117fa700 1 -- 172.21.15.113:0/2998377898 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f3714059120 con 0x7f36fc065590 2024-04-12T05:07:42.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.820+0000 7f371a51e700 1 -- 172.21.15.113:0/2998377898 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f36fc065590 msgr2=0x7f36fc067a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:42.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.820+0000 7f371a51e700 1 --2- 172.21.15.113:0/2998377898 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f36fc065590 0x7f36fc067a50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f37080047c0 tx=0x7f3708009cb0).stop 2024-04-12T05:07:42.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.820+0000 7f371a51e700 1 -- 172.21.15.113:0/2998377898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f5ab0 msgr2=0x7f3714069f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:42.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.820+0000 7f371a51e700 1 --2- 172.21.15.113:0/2998377898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f5ab0 0x7f3714069f90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f370400e8c0 tx=0x7f370400b040).stop 2024-04-12T05:07:42.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.820+0000 7f371a51e700 1 -- 172.21.15.113:0/2998377898 shutdown_connections 2024-04-12T05:07:42.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.820+0000 7f371a51e700 1 --2- 172.21.15.113:0/2998377898 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f36fc065590 0x7f36fc067a50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:42.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.820+0000 7f371a51e700 1 --2- 172.21.15.113:0/2998377898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f5ab0 0x7f3714069f90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:42.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.820+0000 7f371a51e700 1 --2- 172.21.15.113:0/2998377898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f6450 0x7f371406a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:42.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.820+0000 7f371a51e700 1 -- 172.21.15.113:0/2998377898 >> 172.21.15.113:0/2998377898 conn(0x7f37140f15d0 msgr2=0x7f37140fc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:42.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.820+0000 7f371a51e700 1 -- 172.21.15.113:0/2998377898 shutdown_connections 2024-04-12T05:07:42.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.820+0000 7f371a51e700 1 -- 172.21.15.113:0/2998377898 wait complete. 2024-04-12T05:07:42.833 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:07:42.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.963+0000 7f27ba079700 1 -- 172.21.15.113:0/1030474522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27b40f66f0 msgr2=0x7f27b40f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:42.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.963+0000 7f27ba079700 1 --2- 172.21.15.113:0/1030474522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27b40f66f0 0x7f27b40f6b10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f27a40047f0 tx=0x7f27a4009cb0).stop 2024-04-12T05:07:42.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.963+0000 7f27ba079700 1 -- 172.21.15.113:0/1030474522 shutdown_connections 2024-04-12T05:07:42.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.963+0000 7f27ba079700 1 --2- 172.21.15.113:0/1030474522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b40f7990 0x7f27b40f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:42.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.963+0000 7f27ba079700 1 --2- 172.21.15.113:0/1030474522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27b40f66f0 0x7f27b40f6b10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:42.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.963+0000 7f27ba079700 1 -- 172.21.15.113:0/1030474522 >> 172.21.15.113:0/1030474522 conn(0x7f27b40f1e50 msgr2=0x7f27b40f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:42.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.963+0000 7f27ba079700 1 -- 172.21.15.113:0/1030474522 shutdown_connections 2024-04-12T05:07:42.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.964+0000 7f27ba079700 1 -- 172.21.15.113:0/1030474522 wait complete. 2024-04-12T05:07:42.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.964+0000 7f27ba079700 1 Processor -- start 2024-04-12T05:07:42.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.964+0000 7f27ba079700 1 -- start start 2024-04-12T05:07:42.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27ba079700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b40f66f0 0x7f27b40fae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:42.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27ba079700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27b40f7990 0x7f27b40f94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:42.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27ba079700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b40fb440 con 0x7f27b40f7990 2024-04-12T05:07:42.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27ba079700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b40f99e0 con 0x7f27b40f66f0 2024-04-12T05:07:42.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27b2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27b40f7990 0x7f27b40f94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:42.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27b2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27b40f7990 0x7f27b40f94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52634/0 (socket says 172.21.15.113:52634) 2024-04-12T05:07:42.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27b2ffd700 1 -- 172.21.15.113:0/2977944937 learned_addr learned my addr 172.21.15.113:0/2977944937 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:07:42.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27b37fe700 1 --2- 172.21.15.113:0/2977944937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b40f66f0 0x7f27b40fae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:42.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27b2ffd700 1 -- 172.21.15.113:0/2977944937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b40f66f0 msgr2=0x7f27b40fae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:42.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27b2ffd700 1 --2- 172.21.15.113:0/2977944937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b40f66f0 0x7f27b40fae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:42.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27b2ffd700 1 -- 172.21.15.113:0/2977944937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a4005040 con 0x7f27b40f7990 2024-04-12T05:07:42.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.965+0000 7f27b2ffd700 1 --2- 172.21.15.113:0/2977944937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27b40f7990 0x7f27b40f94f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f27a800e8f0 tx=0x7f27a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:42.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.966+0000 7f27b0ff9700 1 -- 172.21.15.113:0/2977944937 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27a8007b40 con 0x7f27b40f7990 2024-04-12T05:07:42.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.966+0000 7f27b0ff9700 1 -- 172.21.15.113:0/2977944937 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27a800f040 con 0x7f27b40f7990 2024-04-12T05:07:42.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.966+0000 7f27b0ff9700 1 -- 172.21.15.113:0/2977944937 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27a8014880 con 0x7f27b40f7990 2024-04-12T05:07:42.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.966+0000 7f27ba079700 1 -- 172.21.15.113:0/2977944937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b40f9c60 con 0x7f27b40f7990 2024-04-12T05:07:42.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.966+0000 7f27ba079700 1 -- 172.21.15.113:0/2977944937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b40fa220 con 0x7f27b40f7990 2024-04-12T05:07:42.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.967+0000 7f27b0ff9700 1 -- 172.21.15.113:0/2977944937 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f27a8014ab0 con 0x7f27b40f7990 2024-04-12T05:07:42.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.968+0000 7f27ba079700 1 -- 172.21.15.113:0/2977944937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b40feeb0 con 0x7f27b40f7990 2024-04-12T05:07:42.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.968+0000 7f27b0ff9700 1 --2- 172.21.15.113:0/2977944937 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f279c061180 0x7f279c063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:42.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.968+0000 7f27b0ff9700 1 -- 172.21.15.113:0/2977944937 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f27a807c130 con 0x7f27b40f7990 2024-04-12T05:07:42.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.968+0000 7f27b37fe700 1 --2- 172.21.15.113:0/2977944937 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f279c061180 0x7f279c063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:42.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.969+0000 7f27b37fe700 1 --2- 172.21.15.113:0/2977944937 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f279c061180 0x7f279c063640 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f27a40047c0 tx=0x7f27a4009cb0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:42.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:42.973+0000 7f27b0ff9700 1 -- 172.21.15.113:0/2977944937 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f27a8081060 con 0x7f27b40f7990 2024-04-12T05:07:43.159 INFO:tasks.workunit.client.1.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-12T05:07:43.217 INFO:tasks.workunit.client.1.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-12T05:07:43.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.233+0000 7f27ba079700 1 -- 172.21.15.113:0/2977944937 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f27b4060ca0 con 0x7f279c061180 2024-04-12T05:07:43.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.234+0000 7f27b0ff9700 1 -- 172.21.15.113:0/2977944937 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f27b4060ca0 con 0x7f279c061180 2024-04-12T05:07:43.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.237+0000 7f27ba079700 1 -- 172.21.15.113:0/2977944937 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f279c061180 msgr2=0x7f279c063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:43.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.237+0000 7f27ba079700 1 --2- 172.21.15.113:0/2977944937 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f279c061180 0x7f279c063640 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f27a40047c0 tx=0x7f27a4009cb0).stop 2024-04-12T05:07:43.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.237+0000 7f27ba079700 1 -- 172.21.15.113:0/2977944937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27b40f7990 msgr2=0x7f27b40f94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:43.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.237+0000 7f27ba079700 1 --2- 172.21.15.113:0/2977944937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27b40f7990 0x7f27b40f94f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f27a800e8f0 tx=0x7f27a800b040).stop 2024-04-12T05:07:43.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.237+0000 7f27ba079700 1 -- 172.21.15.113:0/2977944937 shutdown_connections 2024-04-12T05:07:43.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.237+0000 7f27ba079700 1 --2- 172.21.15.113:0/2977944937 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f279c061180 0x7f279c063640 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.237+0000 7f27ba079700 1 --2- 172.21.15.113:0/2977944937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27b40f66f0 0x7f27b40fae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.237+0000 7f27ba079700 1 --2- 172.21.15.113:0/2977944937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27b40f7990 0x7f27b40f94f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.237+0000 7f27ba079700 1 -- 172.21.15.113:0/2977944937 >> 172.21.15.113:0/2977944937 conn(0x7f27b40f1e50 msgr2=0x7f27b4060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:43.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.237+0000 7f27ba079700 1 -- 172.21.15.113:0/2977944937 shutdown_connections 2024-04-12T05:07:43.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.237+0000 7f27ba079700 1 -- 172.21.15.113:0/2977944937 wait complete. 2024-04-12T05:07:43.261 INFO:tasks.workunit.client.1.smithi192.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-12T05:07:43.263 INFO:tasks.workunit.client.1.smithi192.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-12T05:07:43.263 INFO:tasks.workunit.client.1.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-12T05:07:43.300 INFO:tasks.workunit.client.1.smithi192.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-12T05:07:43.304 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-12T05:07:43.304 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-12T05:07:43.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.347+0000 7f016664c700 1 -- 172.21.15.113:0/4141775166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01600f7990 msgr2=0x7f01600f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:43.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.347+0000 7f016664c700 1 --2- 172.21.15.113:0/4141775166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01600f7990 0x7f01600f7df0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f01540047f0 tx=0x7f0154009cb0).stop 2024-04-12T05:07:43.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.347+0000 7f016664c700 1 -- 172.21.15.113:0/4141775166 shutdown_connections 2024-04-12T05:07:43.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.347+0000 7f016664c700 1 --2- 172.21.15.113:0/4141775166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01600f7990 0x7f01600f7df0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.347+0000 7f016664c700 1 --2- 172.21.15.113:0/4141775166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01600f66f0 0x7f01600f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.347+0000 7f016664c700 1 -- 172.21.15.113:0/4141775166 >> 172.21.15.113:0/4141775166 conn(0x7f01600f1e50 msgr2=0x7f01600f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:43.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.347+0000 7f016664c700 1 -- 172.21.15.113:0/4141775166 shutdown_connections 2024-04-12T05:07:43.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.347+0000 7f016664c700 1 -- 172.21.15.113:0/4141775166 wait complete. 2024-04-12T05:07:43.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.348+0000 7f016664c700 1 Processor -- start 2024-04-12T05:07:43.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.348+0000 7f016664c700 1 -- start start 2024-04-12T05:07:43.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.348+0000 7f016664c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01600f66f0 0x7f01600fe690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:43.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.348+0000 7f016664c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01600f7990 0x7f01600fccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:43.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.348+0000 7f016664c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01600fec40 con 0x7f01600f66f0 2024-04-12T05:07:43.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.348+0000 7f016664c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016010bd90 con 0x7f01600f7990 2024-04-12T05:07:43.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.348+0000 7f015f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01600f7990 0x7f01600fccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:43.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.348+0000 7f015f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01600f7990 0x7f01600fccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.113:55514/0 (socket says 172.21.15.113:55514) 2024-04-12T05:07:43.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.348+0000 7f015f7fe700 1 -- 172.21.15.113:0/973150875 learned_addr learned my addr 172.21.15.113:0/973150875 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:07:43.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.348+0000 7f015ffff700 1 --2- 172.21.15.113:0/973150875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01600f66f0 0x7f01600fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:43.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.349+0000 7f015f7fe700 1 -- 172.21.15.113:0/973150875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01600f66f0 msgr2=0x7f01600fe690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:43.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.349+0000 7f015f7fe700 1 --2- 172.21.15.113:0/973150875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01600f66f0 0x7f01600fe690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.349+0000 7f015f7fe700 1 -- 172.21.15.113:0/973150875 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154005040 con 0x7f01600f7990 2024-04-12T05:07:43.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.349+0000 7f015f7fe700 1 --2- 172.21.15.113:0/973150875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01600f7990 0x7f01600fccf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f0154013560 tx=0x7f0154009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:43.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.349+0000 7f015d7fa700 1 -- 172.21.15.113:0/973150875 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0154018070 con 0x7f01600f7990 2024-04-12T05:07:43.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.349+0000 7f015d7fa700 1 -- 172.21.15.113:0/973150875 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0154007e90 con 0x7f01600f7990 2024-04-12T05:07:43.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.349+0000 7f015d7fa700 1 -- 172.21.15.113:0/973150875 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f015401c380 con 0x7f01600f7990 2024-04-12T05:07:43.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.349+0000 7f016664c700 1 -- 172.21.15.113:0/973150875 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016010bfb0 con 0x7f01600f7990 2024-04-12T05:07:43.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.350+0000 7f016664c700 1 -- 172.21.15.113:0/973150875 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016010c490 con 0x7f01600f7990 2024-04-12T05:07:43.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.351+0000 7f015d7fa700 1 -- 172.21.15.113:0/973150875 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f01540123c0 con 0x7f01600f7990 2024-04-12T05:07:43.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.351+0000 7f015d7fa700 1 --2- 172.21.15.113:0/973150875 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0148061180 0x7f0148063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:43.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.351+0000 7f015d7fa700 1 -- 172.21.15.113:0/973150875 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f015407d390 con 0x7f01600f7990 2024-04-12T05:07:43.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.352+0000 7f016664c700 1 -- 172.21.15.113:0/973150875 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01600fffe0 con 0x7f01600f7990 2024-04-12T05:07:43.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.356+0000 7f015ffff700 1 --2- 172.21.15.113:0/973150875 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0148061180 0x7f0148063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:43.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.356+0000 7f015ffff700 1 --2- 172.21.15.113:0/973150875 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0148061180 0x7f0148063640 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f01600f77f0 tx=0x7f015000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:43.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.358+0000 7f015d7fa700 1 -- 172.21.15.113:0/973150875 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0154051260 con 0x7f01600f7990 2024-04-12T05:07:43.362 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:41.572612+0000 mgr.smithi113.ctsazg (mgr.14194) 252 : cluster [DBG] pgmap v220: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:43.363 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:43.010319+0000 mon.smithi113 (mon.0) 616 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:07:43.363 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:43.011755+0000 mon.smithi113 (mon.0) 617 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:07:43.363 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:43.016337+0000 mon.smithi113 (mon.0) 618 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:07:43.364 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-12T05:07:43.366 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-12T05:07:43.366 DEBUG:teuthology.orchestra.run.smithi192:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d540ebaca6b131a1dd560e7f69e024b133bbaa42 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-12T05:07:43.435 INFO:tasks.workunit.client.1.smithi192.stderr:+ mkdir -p fsstress 2024-04-12T05:07:43.438 INFO:tasks.workunit.client.1.smithi192.stderr:+ pushd fsstress 2024-04-12T05:07:43.439 INFO:tasks.workunit.client.1.smithi192.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-12T05:07:43.440 INFO:tasks.workunit.client.1.smithi192.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-12T05:07:43.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.581+0000 7f016664c700 1 -- 172.21.15.113:0/973150875 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0160101e80 con 0x7f0148061180 2024-04-12T05:07:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.591+0000 7f015d7fa700 1 -- 172.21.15.113:0/973150875 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3344 (secure 0 0 0) 0x7f0160101e80 con 0x7f0148061180 2024-04-12T05:07:43.594 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (5m) 63s ago 6m 26.2M - 0.23.0 ba2b418f427c aae81b8048e6 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (6m) 63s ago 6m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (6m) 69s ago 6m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (5m) 63s ago 6m 59.0M - 8.3.5 dad864ee21e9 7d83fd029dea 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (87s) 63s ago 87s 17.0M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (82s) 63s ago 81s 14.4M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (84s) 69s ago 84s 14.4M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (79s) 69s ago 78s 11.9M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:9283 running (8m) 63s ago 8m 433M - 16.2.15-1-g8963e9de 727f386679cb 3fab7f291e6b 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283 running (6m) 69s ago 6m 387M - 16.2.15-1-g8963e9de 727f386679cb 58d65802163f 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (8m) 63s ago 8m 61.2M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (6m) 69s ago 6m 60.1M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (6m) 63s ago 6m 19.1M - 1.3.1 1dbe0e931976 046bcfb16579 2024-04-12T05:07:43.595 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (5m) 69s ago 5m 26.2M - 1.3.1 1dbe0e931976 bc604318825c 2024-04-12T05:07:43.596 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (5m) 63s ago 5m 39.9M 4640M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:07:43.596 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (4m) 63s ago 4m 42.0M 4640M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:07:43.596 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (4m) 63s ago 4m 40.0M 4640M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:07:43.596 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (3m) 69s ago 3m 45.4M 4982M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:07:43.596 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (3m) 69s ago 3m 39.6M 4982M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:07:43.596 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (3m) 69s ago 2m 39.1M 4982M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:07:43.596 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (5m) 69s ago 5m 60.6M - 2.33.4 514e6a882f6e b4259537bd55 2024-04-12T05:07:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.594+0000 7f016664c700 1 -- 172.21.15.113:0/973150875 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0148061180 msgr2=0x7f0148063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.594+0000 7f016664c700 1 --2- 172.21.15.113:0/973150875 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0148061180 0x7f0148063640 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f01600f77f0 tx=0x7f015000b040).stop 2024-04-12T05:07:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.594+0000 7f016664c700 1 -- 172.21.15.113:0/973150875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01600f7990 msgr2=0x7f01600fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.594+0000 7f016664c700 1 --2- 172.21.15.113:0/973150875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01600f7990 0x7f01600fccf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f0154013560 tx=0x7f0154009cb0).stop 2024-04-12T05:07:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.594+0000 7f016664c700 1 -- 172.21.15.113:0/973150875 shutdown_connections 2024-04-12T05:07:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.594+0000 7f016664c700 1 --2- 172.21.15.113:0/973150875 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f0148061180 0x7f0148063640 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.594+0000 7f016664c700 1 --2- 172.21.15.113:0/973150875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01600f66f0 0x7f01600fe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.594+0000 7f016664c700 1 --2- 172.21.15.113:0/973150875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01600f7990 0x7f01600fccf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.594+0000 7f016664c700 1 -- 172.21.15.113:0/973150875 >> 172.21.15.113:0/973150875 conn(0x7f01600f1e50 msgr2=0x7f01600fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.594+0000 7f016664c700 1 -- 172.21.15.113:0/973150875 shutdown_connections 2024-04-12T05:07:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.594+0000 7f016664c700 1 -- 172.21.15.113:0/973150875 wait complete. 2024-04-12T05:07:43.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:41.572612+0000 mgr.smithi113.ctsazg (mgr.14194) 252 : cluster [DBG] pgmap v220: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:43.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:43.010319+0000 mon.smithi113 (mon.0) 616 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:07:43.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:43.011755+0000 mon.smithi113 (mon.0) 617 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:07:43.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:43.016337+0000 mon.smithi113 (mon.0) 618 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:07:43.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.702+0000 7ff94ad73700 1 -- 172.21.15.113:0/1350630557 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9440f66f0 msgr2=0x7ff9440f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:43.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.702+0000 7ff94ad73700 1 --2- 172.21.15.113:0/1350630557 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9440f66f0 0x7ff9440f6b10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7ff9380047f0 tx=0x7ff938009cb0).stop 2024-04-12T05:07:43.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.702+0000 7ff94ad73700 1 -- 172.21.15.113:0/1350630557 shutdown_connections 2024-04-12T05:07:43.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.702+0000 7ff94ad73700 1 --2- 172.21.15.113:0/1350630557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9440f7990 0x7ff9440f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.702+0000 7ff94ad73700 1 --2- 172.21.15.113:0/1350630557 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9440f66f0 0x7ff9440f6b10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.702+0000 7ff94ad73700 1 -- 172.21.15.113:0/1350630557 >> 172.21.15.113:0/1350630557 conn(0x7ff9440f1e50 msgr2=0x7ff9440f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:43.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.702+0000 7ff94ad73700 1 -- 172.21.15.113:0/1350630557 shutdown_connections 2024-04-12T05:07:43.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.702+0000 7ff94ad73700 1 -- 172.21.15.113:0/1350630557 wait complete. 2024-04-12T05:07:43.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.702+0000 7ff94ad73700 1 Processor -- start 2024-04-12T05:07:43.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.703+0000 7ff94ad73700 1 -- start start 2024-04-12T05:07:43.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.703+0000 7ff94ad73700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9440f66f0 0x7ff9440fe690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.703+0000 7ff94ad73700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9440f7990 0x7ff9440fccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.703+0000 7ff94ad73700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440fec40 con 0x7ff9440f66f0 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.703+0000 7ff94ad73700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94410bd90 con 0x7ff9440f7990 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.703+0000 7ff948b0f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9440f66f0 0x7ff9440fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.703+0000 7ff948b0f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9440f66f0 0x7ff9440fe690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52670/0 (socket says 172.21.15.113:52670) 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.703+0000 7ff948b0f700 1 -- 172.21.15.113:0/1882878556 learned_addr learned my addr 172.21.15.113:0/1882878556 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.703+0000 7ff943fff700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9440f7990 0x7ff9440fccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.704+0000 7ff948b0f700 1 -- 172.21.15.113:0/1882878556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9440f7990 msgr2=0x7ff9440fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.704+0000 7ff948b0f700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9440f7990 0x7ff9440fccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.704+0000 7ff948b0f700 1 -- 172.21.15.113:0/1882878556 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff938005040 con 0x7ff9440f66f0 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.704+0000 7ff943fff700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9440f7990 0x7ff9440fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:07:43.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.704+0000 7ff948b0f700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9440f66f0 0x7ff9440fe690 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff938013560 tx=0x7ff938009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:43.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.704+0000 7ff941ffb700 1 -- 172.21.15.113:0/1882878556 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff938018070 con 0x7ff9440f66f0 2024-04-12T05:07:43.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.704+0000 7ff94ad73700 1 -- 172.21.15.113:0/1882878556 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94410bfb0 con 0x7ff9440f66f0 2024-04-12T05:07:43.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.704+0000 7ff941ffb700 1 -- 172.21.15.113:0/1882878556 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff938007e90 con 0x7ff9440f66f0 2024-04-12T05:07:43.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.704+0000 7ff941ffb700 1 -- 172.21.15.113:0/1882878556 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff93801c5e0 con 0x7ff9440f66f0 2024-04-12T05:07:43.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.704+0000 7ff94ad73700 1 -- 172.21.15.113:0/1882878556 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94410c510 con 0x7ff9440f66f0 2024-04-12T05:07:43.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.706+0000 7ff94ad73700 1 -- 172.21.15.113:0/1882878556 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9440fffe0 con 0x7ff9440f66f0 2024-04-12T05:07:43.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.706+0000 7ff941ffb700 1 -- 172.21.15.113:0/1882878556 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7ff9380123c0 con 0x7ff9440f66f0 2024-04-12T05:07:43.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.706+0000 7ff941ffb700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ff934061180 0x7ff934063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:43.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.707+0000 7ff941ffb700 1 -- 172.21.15.113:0/1882878556 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7ff93807d5d0 con 0x7ff9440f66f0 2024-04-12T05:07:43.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.710+0000 7ff943fff700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ff934061180 0x7ff934063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:43.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.710+0000 7ff943fff700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ff934061180 0x7ff934063640 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff930007fd0 tx=0x7ff93000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:43.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.711+0000 7ff941ffb700 1 -- 172.21.15.113:0/1882878556 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff938051690 con 0x7ff9440f66f0 2024-04-12T05:07:43.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.951+0000 7ff94ad73700 1 -- 172.21.15.113:0/1882878556 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff944101eb0 con 0x7ff9440f66f0 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.952+0000 7ff941ffb700 1 -- 172.21.15.113:0/1882878556 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7ff938051280 con 0x7ff9440f66f0 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:07:43.955 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:07:43.956 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:07:43.956 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:07:43.956 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:07:43.956 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 14 2024-04-12T05:07:43.956 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:07:43.956 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:07:43.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.954+0000 7ff94ad73700 1 -- 172.21.15.113:0/1882878556 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ff934061180 msgr2=0x7ff934063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:43.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.954+0000 7ff94ad73700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ff934061180 0x7ff934063640 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff930007fd0 tx=0x7ff93000b040).stop 2024-04-12T05:07:43.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.954+0000 7ff94ad73700 1 -- 172.21.15.113:0/1882878556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9440f66f0 msgr2=0x7ff9440fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:43.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.954+0000 7ff94ad73700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9440f66f0 0x7ff9440fe690 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff938013560 tx=0x7ff938009cb0).stop 2024-04-12T05:07:43.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.954+0000 7ff94ad73700 1 -- 172.21.15.113:0/1882878556 shutdown_connections 2024-04-12T05:07:43.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.955+0000 7ff94ad73700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7ff934061180 0x7ff934063640 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.955+0000 7ff94ad73700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9440f66f0 0x7ff9440fe690 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.955+0000 7ff94ad73700 1 --2- 172.21.15.113:0/1882878556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9440f7990 0x7ff9440fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:43.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.955+0000 7ff94ad73700 1 -- 172.21.15.113:0/1882878556 >> 172.21.15.113:0/1882878556 conn(0x7ff9440f1e50 msgr2=0x7ff9440fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:43.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.955+0000 7ff94ad73700 1 -- 172.21.15.113:0/1882878556 shutdown_connections 2024-04-12T05:07:43.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:43.955+0000 7ff94ad73700 1 -- 172.21.15.113:0/1882878556 wait complete. 2024-04-12T05:07:44.103 INFO:tasks.workunit.client.1.smithi192.stderr:+ tar xzf ltp-full.tgz 2024-04-12T05:07:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.105+0000 7f67e4c55700 1 -- 172.21.15.113:0/3543994298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67e00f7990 msgr2=0x7f67e00f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.106+0000 7f67e4c55700 1 --2- 172.21.15.113:0/3543994298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67e00f7990 0x7f67e00f7df0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f67d40047f0 tx=0x7f67d4009cb0).stop 2024-04-12T05:07:44.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.107+0000 7f67e4c55700 1 -- 172.21.15.113:0/3543994298 shutdown_connections 2024-04-12T05:07:44.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.107+0000 7f67e4c55700 1 --2- 172.21.15.113:0/3543994298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67e00f7990 0x7f67e00f7df0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:44.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.107+0000 7f67e4c55700 1 --2- 172.21.15.113:0/3543994298 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67e00f66f0 0x7f67e00f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:44.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.107+0000 7f67e4c55700 1 -- 172.21.15.113:0/3543994298 >> 172.21.15.113:0/3543994298 conn(0x7f67e00f1e50 msgr2=0x7f67e00f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:44.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.107+0000 7f67e4c55700 1 -- 172.21.15.113:0/3543994298 shutdown_connections 2024-04-12T05:07:44.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.107+0000 7f67e4c55700 1 -- 172.21.15.113:0/3543994298 wait complete. 2024-04-12T05:07:44.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.108+0000 7f67e4c55700 1 Processor -- start 2024-04-12T05:07:44.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.109+0000 7f67e4c55700 1 -- start start 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.109+0000 7f67e4c55700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67e00f66f0 0x7f67e0106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.109+0000 7f67e4c55700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67e00f7990 0x7f67e0107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.109+0000 7f67e4c55700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e0107930 con 0x7f67e00f7990 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.109+0000 7f67e4c55700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e0107a70 con 0x7f67e00f66f0 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.109+0000 7f67de59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67e00f66f0 0x7f67e0106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.109+0000 7f67de59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67e00f66f0 0x7f67e0106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.113:55560/0 (socket says 172.21.15.113:55560) 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.109+0000 7f67de59c700 1 -- 172.21.15.113:0/805798823 learned_addr learned my addr 172.21.15.113:0/805798823 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.110+0000 7f67de59c700 1 -- 172.21.15.113:0/805798823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67e00f7990 msgr2=0x7f67e0107380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.110+0000 7f67de59c700 1 --2- 172.21.15.113:0/805798823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67e00f7990 0x7f67e0107380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.110+0000 7f67de59c700 1 -- 172.21.15.113:0/805798823 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d4005040 con 0x7f67e00f66f0 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.110+0000 7f67de59c700 1 --2- 172.21.15.113:0/805798823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67e00f66f0 0x7f67e0106e90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f67d0002820 tx=0x7f67d0009f00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.110+0000 7f67cf7fe700 1 -- 172.21.15.113:0/805798823 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67d0002d60 con 0x7f67e00f66f0 2024-04-12T05:07:44.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.110+0000 7f67cf7fe700 1 -- 172.21.15.113:0/805798823 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67d000f040 con 0x7f67e00f66f0 2024-04-12T05:07:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.110+0000 7f67cf7fe700 1 -- 172.21.15.113:0/805798823 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67d000d800 con 0x7f67e00f66f0 2024-04-12T05:07:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.110+0000 7f67e4c55700 1 -- 172.21.15.113:0/805798823 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e00605f0 con 0x7f67e00f66f0 2024-04-12T05:07:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.110+0000 7f67e4c55700 1 -- 172.21.15.113:0/805798823 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e0060bb0 con 0x7f67e00f66f0 2024-04-12T05:07:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.113+0000 7f67cf7fe700 1 -- 172.21.15.113:0/805798823 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f67d0009360 con 0x7f67e00f66f0 2024-04-12T05:07:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.113+0000 7f67cf7fe700 1 --2- 172.21.15.113:0/805798823 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f67c80655f0 0x7f67c8067ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:07:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.113+0000 7f67cf7fe700 1 -- 172.21.15.113:0/805798823 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f67d007c550 con 0x7f67e00f66f0 2024-04-12T05:07:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.114+0000 7f67e4c55700 1 -- 172.21.15.113:0/805798823 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c0005150 con 0x7f67e00f66f0 2024-04-12T05:07:44.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.114+0000 7f67ddd9b700 1 --2- 172.21.15.113:0/805798823 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f67c80655f0 0x7f67c8067ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:07:44.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.114+0000 7f67ddd9b700 1 --2- 172.21.15.113:0/805798823 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f67c80655f0 0x7f67c8067ab0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f67d4004f10 tx=0x7f67d4009cb0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:07:44.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.121+0000 7f67cf7fe700 1 -- 172.21.15.113:0/805798823 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f67d00504a0 con 0x7f67e00f66f0 2024-04-12T05:07:44.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.427+0000 7f67e4c55700 1 -- 172.21.15.113:0/805798823 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f67c0000bf0 con 0x7f67c80655f0 2024-04-12T05:07:44.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.430+0000 7f67cf7fe700 1 -- 172.21.15.113:0/805798823 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f67c0000bf0 con 0x7f67c80655f0 2024-04-12T05:07:44.433 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:07:44.433 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:07:44.433 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:07:44.433 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:07:44.433 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [], 2024-04-12T05:07:44.433 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "0/2 daemons upgraded", 2024-04-12T05:07:44.433 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 image on host smithi192", 2024-04-12T05:07:44.433 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:07:44.433 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:07:44.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.432+0000 7f67e4c55700 1 -- 172.21.15.113:0/805798823 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f67c80655f0 msgr2=0x7f67c8067ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:44.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.432+0000 7f67e4c55700 1 --2- 172.21.15.113:0/805798823 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f67c80655f0 0x7f67c8067ab0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f67d4004f10 tx=0x7f67d4009cb0).stop 2024-04-12T05:07:44.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.432+0000 7f67e4c55700 1 -- 172.21.15.113:0/805798823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67e00f66f0 msgr2=0x7f67e0106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:07:44.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.432+0000 7f67e4c55700 1 --2- 172.21.15.113:0/805798823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67e00f66f0 0x7f67e0106e90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f67d0002820 tx=0x7f67d0009f00).stop 2024-04-12T05:07:44.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.433+0000 7f67e4c55700 1 -- 172.21.15.113:0/805798823 shutdown_connections 2024-04-12T05:07:44.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.433+0000 7f67e4c55700 1 --2- 172.21.15.113:0/805798823 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f67c80655f0 0x7f67c8067ab0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:44.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.433+0000 7f67e4c55700 1 --2- 172.21.15.113:0/805798823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67e00f66f0 0x7f67e0106e90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:07:44.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.433+0000 7f67e4c55700 1 --2- 172.21.15.113:0/805798823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67e00f7990 0x7f67e0107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:07:44.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.433+0000 7f67e4c55700 1 -- 172.21.15.113:0/805798823 >> 172.21.15.113:0/805798823 conn(0x7f67e00f1e50 msgr2=0x7f67e00fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:07:44.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.433+0000 7f67e4c55700 1 -- 172.21.15.113:0/805798823 shutdown_connections 2024-04-12T05:07:44.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:07:44.433+0000 7f67e4c55700 1 -- 172.21.15.113:0/805798823 wait complete. 2024-04-12T05:07:44.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:42.818304+0000 mgr.smithi113.ctsazg (mgr.14194) 253 : audit [DBG] from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:07:44.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:07:43.010704+0000 mgr.smithi113.ctsazg (mgr.14194) 254 : cephadm [INF] Upgrade: Target is version 18.2.2-1187-gd540ebac (unknown) 2024-04-12T05:07:44.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:07:43.010796+0000 mgr.smithi113.ctsazg (mgr.14194) 255 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3'] 2024-04-12T05:07:44.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:07:43.017039+0000 mgr.smithi113.ctsazg (mgr.14194) 256 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi113.ctsazg) 2024-04-12T05:07:44.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:43.235890+0000 mgr.smithi113.ctsazg (mgr.14194) 257 : audit [DBG] from='client.14514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:07:44.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:07:43.268298+0000 mgr.smithi113.ctsazg (mgr.14194) 258 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 on smithi192 2024-04-12T05:07:44.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:43.953740+0000 mon.smithi113 (mon.0) 619 : audit [DBG] from='client.? 172.21.15.113:0/1882878556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:07:44.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:42.818304+0000 mgr.smithi113.ctsazg (mgr.14194) 253 : audit [DBG] from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:07:44.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:07:43.010704+0000 mgr.smithi113.ctsazg (mgr.14194) 254 : cephadm [INF] Upgrade: Target is version 18.2.2-1187-gd540ebac (unknown) 2024-04-12T05:07:44.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:07:43.010796+0000 mgr.smithi113.ctsazg (mgr.14194) 255 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3'] 2024-04-12T05:07:44.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:07:43.017039+0000 mgr.smithi113.ctsazg (mgr.14194) 256 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi113.ctsazg) 2024-04-12T05:07:44.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:43.235890+0000 mgr.smithi113.ctsazg (mgr.14194) 257 : audit [DBG] from='client.14514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:07:44.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:07:43.268298+0000 mgr.smithi113.ctsazg (mgr.14194) 258 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 on smithi192 2024-04-12T05:07:44.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:43.953740+0000 mon.smithi113 (mon.0) 619 : audit [DBG] from='client.? 172.21.15.113:0/1882878556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:07:45.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:43.573038+0000 mgr.smithi113.ctsazg (mgr.14194) 259 : cluster [DBG] pgmap v221: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:45.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:43.584235+0000 mgr.smithi113.ctsazg (mgr.14194) 260 : audit [DBG] from='client.24327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:07:45.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:07:44.430647+0000 mgr.smithi113.ctsazg (mgr.14194) 261 : audit [DBG] from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:07:45.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:43.573038+0000 mgr.smithi113.ctsazg (mgr.14194) 259 : cluster [DBG] pgmap v221: 65 pgs: 65 active+clean; 13 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:07:45.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:43.584235+0000 mgr.smithi113.ctsazg (mgr.14194) 260 : audit [DBG] from='client.24327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:07:45.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:07:44.430647+0000 mgr.smithi113.ctsazg (mgr.14194) 261 : audit [DBG] from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:07:46.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:45.573472+0000 mgr.smithi113.ctsazg (mgr.14194) 262 : cluster [DBG] pgmap v222: 65 pgs: 65 active+clean; 4.3 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 370 KiB/s wr, 4 op/s 2024-04-12T05:07:46.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:45.573472+0000 mgr.smithi113.ctsazg (mgr.14194) 262 : cluster [DBG] pgmap v222: 65 pgs: 65 active+clean; 4.3 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 370 KiB/s wr, 4 op/s 2024-04-12T05:07:48.305 INFO:tasks.workunit.client.0.smithi113.stderr:Updating files: 93% (11342/12088) Updating files: 94% (11363/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-04-12T05:07:48.919 INFO:tasks.workunit.client.0.smithi113.stderr:Note: switching to 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'. 2024-04-12T05:07:48.919 INFO:tasks.workunit.client.0.smithi113.stderr: 2024-04-12T05:07:48.919 INFO:tasks.workunit.client.0.smithi113.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-12T05:07:48.919 INFO:tasks.workunit.client.0.smithi113.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-12T05:07:48.919 INFO:tasks.workunit.client.0.smithi113.stderr:state without impacting any branches by switching back to a branch. 2024-04-12T05:07:48.919 INFO:tasks.workunit.client.0.smithi113.stderr: 2024-04-12T05:07:48.919 INFO:tasks.workunit.client.0.smithi113.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-12T05:07:48.919 INFO:tasks.workunit.client.0.smithi113.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-12T05:07:48.919 INFO:tasks.workunit.client.0.smithi113.stderr: 2024-04-12T05:07:48.920 INFO:tasks.workunit.client.0.smithi113.stderr: git switch -c 2024-04-12T05:07:48.920 INFO:tasks.workunit.client.0.smithi113.stderr: 2024-04-12T05:07:48.920 INFO:tasks.workunit.client.0.smithi113.stderr:Or undo this operation with: 2024-04-12T05:07:48.920 INFO:tasks.workunit.client.0.smithi113.stderr: 2024-04-12T05:07:48.920 INFO:tasks.workunit.client.0.smithi113.stderr: git switch - 2024-04-12T05:07:48.920 INFO:tasks.workunit.client.0.smithi113.stderr: 2024-04-12T05:07:48.920 INFO:tasks.workunit.client.0.smithi113.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-12T05:07:48.920 INFO:tasks.workunit.client.0.smithi113.stderr: 2024-04-12T05:07:48.920 INFO:tasks.workunit.client.0.smithi113.stderr:HEAD is now at d540ebaca6b Merge pull request #56672 from batrick/wip-65294-reef 2024-04-12T05:07:48.923 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:47.573920+0000 mgr.smithi113.ctsazg (mgr.14194) 263 : cluster [DBG] pgmap v223: 65 pgs: 65 active+clean; 11 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 917 KiB/s wr, 24 op/s 2024-04-12T05:07:48.925 DEBUG:teuthology.orchestra.run.smithi113:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-12T05:07:48.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:47.573920+0000 mgr.smithi113.ctsazg (mgr.14194) 263 : cluster [DBG] pgmap v223: 65 pgs: 65 active+clean; 11 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 917 KiB/s wr, 24 op/s 2024-04-12T05:07:48.960 INFO:tasks.workunit.client.0.smithi113.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-12T05:07:48.962 INFO:tasks.workunit.client.0.smithi113.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-12T05:07:48.963 INFO:tasks.workunit.client.0.smithi113.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-12T05:07:49.505 INFO:tasks.workunit.client.0.smithi113.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-12T05:07:49.567 INFO:tasks.workunit.client.0.smithi113.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-12T05:07:49.607 INFO:tasks.workunit.client.0.smithi113.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-12T05:07:49.608 INFO:tasks.workunit.client.0.smithi113.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-12T05:07:49.608 INFO:tasks.workunit.client.0.smithi113.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-12T05:07:49.645 INFO:tasks.workunit.client.0.smithi113.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-12T05:07:49.648 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:07:49.648 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-12T05:07:49.707 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-12T05:07:49.708 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-12T05:07:49.709 DEBUG:teuthology.orchestra.run.smithi113:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d540ebaca6b131a1dd560e7f69e024b133bbaa42 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-12T05:07:49.780 INFO:tasks.workunit.client.0.smithi113.stderr:+ mkdir -p fsstress 2024-04-12T05:07:49.783 INFO:tasks.workunit.client.0.smithi113.stderr:+ pushd fsstress 2024-04-12T05:07:49.784 INFO:tasks.workunit.client.0.smithi113.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-12T05:07:49.785 INFO:tasks.workunit.client.0.smithi113.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-12T05:07:50.378 INFO:tasks.workunit.client.0.smithi113.stderr:+ tar xzf ltp-full.tgz 2024-04-12T05:07:50.923 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:49.574310+0000 mgr.smithi113.ctsazg (mgr.14194) 264 : cluster [DBG] pgmap v224: 65 pgs: 65 active+clean; 15 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s wr, 65 op/s 2024-04-12T05:07:50.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:49.574310+0000 mgr.smithi113.ctsazg (mgr.14194) 264 : cluster [DBG] pgmap v224: 65 pgs: 65 active+clean; 15 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s wr, 65 op/s 2024-04-12T05:07:52.923 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:51.574796+0000 mgr.smithi113.ctsazg (mgr.14194) 265 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 27 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s wr, 148 op/s 2024-04-12T05:07:52.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:51.574796+0000 mgr.smithi113.ctsazg (mgr.14194) 265 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 27 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s wr, 148 op/s 2024-04-12T05:07:54.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:53.575138+0000 mgr.smithi113.ctsazg (mgr.14194) 266 : cluster [DBG] pgmap v226: 65 pgs: 65 active+clean; 28 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s wr, 182 op/s 2024-04-12T05:07:54.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:53.575138+0000 mgr.smithi113.ctsazg (mgr.14194) 266 : cluster [DBG] pgmap v226: 65 pgs: 65 active+clean; 28 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s wr, 182 op/s 2024-04-12T05:07:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:55.575547+0000 mgr.smithi113.ctsazg (mgr.14194) 267 : cluster [DBG] pgmap v227: 65 pgs: 65 active+clean; 39 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 3.4 MiB/s wr, 250 op/s 2024-04-12T05:07:56.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:55.575547+0000 mgr.smithi113.ctsazg (mgr.14194) 267 : cluster [DBG] pgmap v227: 65 pgs: 65 active+clean; 39 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 3.4 MiB/s wr, 250 op/s 2024-04-12T05:07:57.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:56.648489+0000 mon.smithi113 (mon.0) 620 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:07:57.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:56.648489+0000 mon.smithi113 (mon.0) 620 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:07:58.923 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:07:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:57.575910+0000 mgr.smithi113.ctsazg (mgr.14194) 268 : cluster [DBG] pgmap v228: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 3.7 MiB/s wr, 319 op/s 2024-04-12T05:07:58.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:07:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:57.575910+0000 mgr.smithi113.ctsazg (mgr.14194) 268 : cluster [DBG] pgmap v228: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 3.7 MiB/s wr, 319 op/s 2024-04-12T05:08:00.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:07:59.576265+0000 mgr.smithi113.ctsazg (mgr.14194) 269 : cluster [DBG] pgmap v229: 65 pgs: 65 active+clean; 52 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 3.7 MiB/s wr, 366 op/s 2024-04-12T05:08:00.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:07:59.576265+0000 mgr.smithi113.ctsazg (mgr.14194) 269 : cluster [DBG] pgmap v229: 65 pgs: 65 active+clean; 52 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 3.7 MiB/s wr, 366 op/s 2024-04-12T05:08:02.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:01.576728+0000 mgr.smithi113.ctsazg (mgr.14194) 270 : cluster [DBG] pgmap v230: 65 pgs: 65 active+clean; 69 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.8 MiB/s wr, 440 op/s 2024-04-12T05:08:02.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:01.576728+0000 mgr.smithi113.ctsazg (mgr.14194) 270 : cluster [DBG] pgmap v230: 65 pgs: 65 active+clean; 69 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.8 MiB/s wr, 440 op/s 2024-04-12T05:08:04.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:03.577146+0000 mgr.smithi113.ctsazg (mgr.14194) 271 : cluster [DBG] pgmap v231: 65 pgs: 65 active+clean; 76 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.3 MiB/s wr, 403 op/s 2024-04-12T05:08:04.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:03.577146+0000 mgr.smithi113.ctsazg (mgr.14194) 271 : cluster [DBG] pgmap v231: 65 pgs: 65 active+clean; 76 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.3 MiB/s wr, 403 op/s 2024-04-12T05:08:06.923 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:05.577570+0000 mgr.smithi113.ctsazg (mgr.14194) 272 : cluster [DBG] pgmap v232: 65 pgs: 65 active+clean; 79 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.5 MiB/s wr, 427 op/s 2024-04-12T05:08:06.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:05.577570+0000 mgr.smithi113.ctsazg (mgr.14194) 272 : cluster [DBG] pgmap v232: 65 pgs: 65 active+clean; 79 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.5 MiB/s wr, 427 op/s 2024-04-12T05:08:08.923 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:07.577960+0000 mgr.smithi113.ctsazg (mgr.14194) 273 : cluster [DBG] pgmap v233: 65 pgs: 65 active+clean; 88 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.3 MiB/s wr, 434 op/s 2024-04-12T05:08:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:08 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:07.577960+0000 mgr.smithi113.ctsazg (mgr.14194) 273 : cluster [DBG] pgmap v233: 65 pgs: 65 active+clean; 88 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.3 MiB/s wr, 434 op/s 2024-04-12T05:08:10.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:09.578308+0000 mgr.smithi113.ctsazg (mgr.14194) 274 : cluster [DBG] pgmap v234: 65 pgs: 65 active+clean; 94 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 4.2 MiB/s wr, 420 op/s 2024-04-12T05:08:10.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:09.578308+0000 mgr.smithi113.ctsazg (mgr.14194) 274 : cluster [DBG] pgmap v234: 65 pgs: 65 active+clean; 94 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 4.2 MiB/s wr, 420 op/s 2024-04-12T05:08:12.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:11.578724+0000 mgr.smithi113.ctsazg (mgr.14194) 275 : cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 4.9 MiB/s wr, 465 op/s 2024-04-12T05:08:12.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:11.578724+0000 mgr.smithi113.ctsazg (mgr.14194) 275 : cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 4.9 MiB/s wr, 465 op/s 2024-04-12T05:08:14.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.566+0000 7f06c395f700 1 -- 172.21.15.113:0/1929309571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06bc0f7990 msgr2=0x7f06bc0f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:14.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.566+0000 7f06c395f700 1 --2- 172.21.15.113:0/1929309571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06bc0f7990 0x7f06bc0f7df0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f06ac0047f0 tx=0x7f06ac009cb0).stop 2024-04-12T05:08:14.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.566+0000 7f06c395f700 1 -- 172.21.15.113:0/1929309571 shutdown_connections 2024-04-12T05:08:14.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.566+0000 7f06c395f700 1 --2- 172.21.15.113:0/1929309571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06bc0f7990 0x7f06bc0f7df0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:14.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.566+0000 7f06c395f700 1 --2- 172.21.15.113:0/1929309571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06bc0f66f0 0x7f06bc0f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:14.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.566+0000 7f06c395f700 1 -- 172.21.15.113:0/1929309571 >> 172.21.15.113:0/1929309571 conn(0x7f06bc0f1e50 msgr2=0x7f06bc0f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:14.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.567+0000 7f06c395f700 1 -- 172.21.15.113:0/1929309571 shutdown_connections 2024-04-12T05:08:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.567+0000 7f06c395f700 1 -- 172.21.15.113:0/1929309571 wait complete. 2024-04-12T05:08:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.567+0000 7f06c395f700 1 Processor -- start 2024-04-12T05:08:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.567+0000 7f06c395f700 1 -- start start 2024-04-12T05:08:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.567+0000 7f06c395f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06bc0f66f0 0x7f06bc061040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.567+0000 7f06c395f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06bc0f7990 0x7f06bc10f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.567+0000 7f06c395f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc108c50 con 0x7f06bc0f66f0 2024-04-12T05:08:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.567+0000 7f06c395f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc10f760 con 0x7f06bc0f7990 2024-04-12T05:08:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06c16fb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06bc0f66f0 0x7f06bc061040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06c16fb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06bc0f66f0 0x7f06bc061040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46634/0 (socket says 172.21.15.113:46634) 2024-04-12T05:08:14.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06c16fb700 1 -- 172.21.15.113:0/3121073526 learned_addr learned my addr 172.21.15.113:0/3121073526 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:08:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06c0efa700 1 --2- 172.21.15.113:0/3121073526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06bc0f7990 0x7f06bc10f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06c16fb700 1 -- 172.21.15.113:0/3121073526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06bc0f7990 msgr2=0x7f06bc10f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06c16fb700 1 --2- 172.21.15.113:0/3121073526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06bc0f7990 0x7f06bc10f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06c16fb700 1 -- 172.21.15.113:0/3121073526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac005040 con 0x7f06bc0f66f0 2024-04-12T05:08:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06c16fb700 1 --2- 172.21.15.113:0/3121073526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06bc0f66f0 0x7f06bc061040 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f06b800e8c0 tx=0x7f06b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06b27fc700 1 -- 172.21.15.113:0/3121073526 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06b8007b40 con 0x7f06bc0f66f0 2024-04-12T05:08:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06b27fc700 1 -- 172.21.15.113:0/3121073526 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06b800f040 con 0x7f06bc0f66f0 2024-04-12T05:08:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06c395f700 1 -- 172.21.15.113:0/3121073526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06bc10f900 con 0x7f06bc0f66f0 2024-04-12T05:08:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06b27fc700 1 -- 172.21.15.113:0/3121073526 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06b8014880 con 0x7f06bc0f66f0 2024-04-12T05:08:14.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.568+0000 7f06c395f700 1 -- 172.21.15.113:0/3121073526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06bc10fe90 con 0x7f06bc0f66f0 2024-04-12T05:08:14.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.569+0000 7f06c395f700 1 -- 172.21.15.113:0/3121073526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06bc0feeb0 con 0x7f06bc0f66f0 2024-04-12T05:08:14.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.570+0000 7f06b27fc700 1 -- 172.21.15.113:0/3121073526 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f06b8007ca0 con 0x7f06bc0f66f0 2024-04-12T05:08:14.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.570+0000 7f06b27fc700 1 --2- 172.21.15.113:0/3121073526 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f06a8061180 0x7f06a8063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:14.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.570+0000 7f06b27fc700 1 -- 172.21.15.113:0/3121073526 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f06b807c4d0 con 0x7f06bc0f66f0 2024-04-12T05:08:14.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.570+0000 7f06c0efa700 1 --2- 172.21.15.113:0/3121073526 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f06a8061180 0x7f06a8063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:14.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.571+0000 7f06c0efa700 1 --2- 172.21.15.113:0/3121073526 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f06a8061180 0x7f06a8063640 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f06ac0047c0 tx=0x7f06ac009cb0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.576+0000 7f06b27fc700 1 -- 172.21.15.113:0/3121073526 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f06b8050460 con 0x7f06bc0f66f0 2024-04-12T05:08:14.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.842+0000 7f06c395f700 1 -- 172.21.15.113:0/3121073526 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f06bc059120 con 0x7f06a8061180 2024-04-12T05:08:14.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.843+0000 7f06b27fc700 1 -- 172.21.15.113:0/3121073526 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f06bc059120 con 0x7f06a8061180 2024-04-12T05:08:14.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.845+0000 7f06c395f700 1 -- 172.21.15.113:0/3121073526 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f06a8061180 msgr2=0x7f06a8063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:14.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.845+0000 7f06c395f700 1 --2- 172.21.15.113:0/3121073526 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f06a8061180 0x7f06a8063640 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f06ac0047c0 tx=0x7f06ac009cb0).stop 2024-04-12T05:08:14.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.845+0000 7f06c395f700 1 -- 172.21.15.113:0/3121073526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06bc0f66f0 msgr2=0x7f06bc061040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:14.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.845+0000 7f06c395f700 1 --2- 172.21.15.113:0/3121073526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06bc0f66f0 0x7f06bc061040 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f06b800e8c0 tx=0x7f06b800b040).stop 2024-04-12T05:08:14.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.845+0000 7f06c395f700 1 -- 172.21.15.113:0/3121073526 shutdown_connections 2024-04-12T05:08:14.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.845+0000 7f06c395f700 1 --2- 172.21.15.113:0/3121073526 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f06a8061180 0x7f06a8063640 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:14.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.845+0000 7f06c395f700 1 --2- 172.21.15.113:0/3121073526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06bc0f66f0 0x7f06bc061040 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:14.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.845+0000 7f06c395f700 1 --2- 172.21.15.113:0/3121073526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06bc0f7990 0x7f06bc10f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:14.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.845+0000 7f06c395f700 1 -- 172.21.15.113:0/3121073526 >> 172.21.15.113:0/3121073526 conn(0x7f06bc0f1e50 msgr2=0x7f06bc0fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:14.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.845+0000 7f06c395f700 1 -- 172.21.15.113:0/3121073526 shutdown_connections 2024-04-12T05:08:14.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.845+0000 7f06c395f700 1 -- 172.21.15.113:0/3121073526 wait complete. 2024-04-12T05:08:14.860 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:08:14.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:13.579075+0000 mgr.smithi113.ctsazg (mgr.14194) 276 : cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 109 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 3.6 MiB/s wr, 390 op/s 2024-04-12T05:08:14.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.975+0000 7f21f16ba700 1 -- 172.21.15.113:0/924257198 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21ec0fa960 msgr2=0x7f21ec0fcdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:14.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.975+0000 7f21f16ba700 1 --2- 172.21.15.113:0/924257198 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21ec0fa960 0x7f21ec0fcdb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f21e00047f0 tx=0x7f21e0009cb0).stop 2024-04-12T05:08:14.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.975+0000 7f21f16ba700 1 -- 172.21.15.113:0/924257198 shutdown_connections 2024-04-12T05:08:14.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.975+0000 7f21f16ba700 1 --2- 172.21.15.113:0/924257198 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21ec0fa960 0x7f21ec0fcdb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:14.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.975+0000 7f21f16ba700 1 --2- 172.21.15.113:0/924257198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21ec0611e0 0x7f21ec0fa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:14.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.975+0000 7f21f16ba700 1 -- 172.21.15.113:0/924257198 >> 172.21.15.113:0/924257198 conn(0x7f21ec0f1e50 msgr2=0x7f21ec0f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:14.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.975+0000 7f21f16ba700 1 -- 172.21.15.113:0/924257198 shutdown_connections 2024-04-12T05:08:14.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.975+0000 7f21f16ba700 1 -- 172.21.15.113:0/924257198 wait complete. 2024-04-12T05:08:14.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21f16ba700 1 Processor -- start 2024-04-12T05:08:14.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21f16ba700 1 -- start start 2024-04-12T05:08:14.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21f16ba700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21ec0611e0 0x7f21ec107f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:14.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21f16ba700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21ec0fa960 0x7f21ec108440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:14.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21f16ba700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec1089f0 con 0x7f21ec0fa960 2024-04-12T05:08:14.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21f16ba700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec108b30 con 0x7f21ec0611e0 2024-04-12T05:08:14.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21ea7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21ec0fa960 0x7f21ec108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:14.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21ea7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21ec0fa960 0x7f21ec108440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46650/0 (socket says 172.21.15.113:46650) 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21ea7fc700 1 -- 172.21.15.113:0/3951732560 learned_addr learned my addr 172.21.15.113:0/3951732560 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21eaffd700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21ec0611e0 0x7f21ec107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21ea7fc700 1 -- 172.21.15.113:0/3951732560 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21ec0611e0 msgr2=0x7f21ec107f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21ea7fc700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21ec0611e0 0x7f21ec107f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21ea7fc700 1 -- 172.21.15.113:0/3951732560 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e0005040 con 0x7f21ec0fa960 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.976+0000 7f21eaffd700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21ec0611e0 0x7f21ec107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.977+0000 7f21ea7fc700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21ec0fa960 0x7f21ec108440 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f21e0013620 tx=0x7f21e0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.977+0000 7f21d3fff700 1 -- 172.21.15.113:0/3951732560 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21e000e070 con 0x7f21ec0fa960 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.977+0000 7f21d3fff700 1 -- 172.21.15.113:0/3951732560 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21e0007e90 con 0x7f21ec0fa960 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.977+0000 7f21f16ba700 1 -- 172.21.15.113:0/3951732560 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ec10d570 con 0x7f21ec0fa960 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.977+0000 7f21d3fff700 1 -- 172.21.15.113:0/3951732560 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21e0018070 con 0x7f21ec0fa960 2024-04-12T05:08:14.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.977+0000 7f21f16ba700 1 -- 172.21.15.113:0/3951732560 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ec10db30 con 0x7f21ec0fa960 2024-04-12T05:08:14.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.978+0000 7f21f16ba700 1 -- 172.21.15.113:0/3951732560 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ec0fffe0 con 0x7f21ec0fa960 2024-04-12T05:08:14.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.980+0000 7f21d3fff700 1 -- 172.21.15.113:0/3951732560 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f21e001d960 con 0x7f21ec0fa960 2024-04-12T05:08:14.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.980+0000 7f21d3fff700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f21d4061180 0x7f21d4063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:14.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.980+0000 7f21d3fff700 1 -- 172.21.15.113:0/3951732560 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f21e007da60 con 0x7f21ec0fa960 2024-04-12T05:08:14.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.980+0000 7f21eaffd700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f21d4061180 0x7f21d4063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:14.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.981+0000 7f21eaffd700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f21d4061180 0x7f21d4063640 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f21dc004ca0 tx=0x7f21dc00b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:14.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:14.985+0000 7f21d3fff700 1 -- 172.21.15.113:0/3951732560 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f21e0051a70 con 0x7f21ec0fa960 2024-04-12T05:08:15.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:13.579075+0000 mgr.smithi113.ctsazg (mgr.14194) 276 : cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 109 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 3.6 MiB/s wr, 390 op/s 2024-04-12T05:08:15.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.244+0000 7f21f16ba700 1 -- 172.21.15.113:0/3951732560 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21ec059120 con 0x7f21d4061180 2024-04-12T05:08:15.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.246+0000 7f21d3fff700 1 -- 172.21.15.113:0/3951732560 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f21ec059120 con 0x7f21d4061180 2024-04-12T05:08:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.248+0000 7f21f16ba700 1 -- 172.21.15.113:0/3951732560 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f21d4061180 msgr2=0x7f21d4063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.248+0000 7f21f16ba700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f21d4061180 0x7f21d4063640 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f21dc004ca0 tx=0x7f21dc00b040).stop 2024-04-12T05:08:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.248+0000 7f21f16ba700 1 -- 172.21.15.113:0/3951732560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21ec0fa960 msgr2=0x7f21ec108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.248+0000 7f21f16ba700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21ec0fa960 0x7f21ec108440 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f21e0013620 tx=0x7f21e0009cb0).stop 2024-04-12T05:08:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.248+0000 7f21f16ba700 1 -- 172.21.15.113:0/3951732560 shutdown_connections 2024-04-12T05:08:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.248+0000 7f21f16ba700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f21d4061180 0x7f21d4063640 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.248+0000 7f21f16ba700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21ec0611e0 0x7f21ec107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.248+0000 7f21f16ba700 1 --2- 172.21.15.113:0/3951732560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21ec0fa960 0x7f21ec108440 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.248+0000 7f21f16ba700 1 -- 172.21.15.113:0/3951732560 >> 172.21.15.113:0/3951732560 conn(0x7f21ec0f1e50 msgr2=0x7f21ec0f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.248+0000 7f21f16ba700 1 -- 172.21.15.113:0/3951732560 shutdown_connections 2024-04-12T05:08:15.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.248+0000 7f21f16ba700 1 -- 172.21.15.113:0/3951732560 wait complete. 2024-04-12T05:08:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.379+0000 7fb879e0b700 1 -- 172.21.15.113:0/1603268733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8740f6ad0 msgr2=0x7fb8740f6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.379+0000 7fb879e0b700 1 --2- 172.21.15.113:0/1603268733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8740f6ad0 0x7fb8740f6ef0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb8680047f0 tx=0x7fb868009cb0).stop 2024-04-12T05:08:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.379+0000 7fb879e0b700 1 -- 172.21.15.113:0/1603268733 shutdown_connections 2024-04-12T05:08:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.379+0000 7fb879e0b700 1 --2- 172.21.15.113:0/1603268733 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8740f7470 0x7fb8740f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.379+0000 7fb879e0b700 1 --2- 172.21.15.113:0/1603268733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8740f6ad0 0x7fb8740f6ef0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.379+0000 7fb879e0b700 1 -- 172.21.15.113:0/1603268733 >> 172.21.15.113:0/1603268733 conn(0x7fb8740f1220 msgr2=0x7fb8740f3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:15.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.379+0000 7fb879e0b700 1 -- 172.21.15.113:0/1603268733 shutdown_connections 2024-04-12T05:08:15.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.380+0000 7fb879e0b700 1 -- 172.21.15.113:0/1603268733 wait complete. 2024-04-12T05:08:15.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.380+0000 7fb879e0b700 1 Processor -- start 2024-04-12T05:08:15.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.380+0000 7fb879e0b700 1 -- start start 2024-04-12T05:08:15.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.381+0000 7fb879e0b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8740f6ad0 0x7fb87410e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:15.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.381+0000 7fb879e0b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8740f7470 0x7fb87410ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:15.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.381+0000 7fb879e0b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87410f040 con 0x7fb8740f7470 2024-04-12T05:08:15.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.381+0000 7fb879e0b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87410f180 con 0x7fb8740f6ad0 2024-04-12T05:08:15.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.381+0000 7fb8737fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8740f6ad0 0x7fb87410e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:15.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.381+0000 7fb872ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8740f7470 0x7fb87410ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:15.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.381+0000 7fb872ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8740f7470 0x7fb87410ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46680/0 (socket says 172.21.15.113:46680) 2024-04-12T05:08:15.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.381+0000 7fb872ffd700 1 -- 172.21.15.113:0/2736861493 learned_addr learned my addr 172.21.15.113:0/2736861493 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:08:15.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.381+0000 7fb872ffd700 1 -- 172.21.15.113:0/2736861493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8740f6ad0 msgr2=0x7fb87410e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:15.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.381+0000 7fb872ffd700 1 --2- 172.21.15.113:0/2736861493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8740f6ad0 0x7fb87410e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.381+0000 7fb872ffd700 1 -- 172.21.15.113:0/2736861493 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb868005040 con 0x7fb8740f7470 2024-04-12T05:08:15.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.382+0000 7fb872ffd700 1 --2- 172.21.15.113:0/2736861493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8740f7470 0x7fb87410ea90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb8640041d0 tx=0x7fb864009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:15.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.382+0000 7fb870ff9700 1 -- 172.21.15.113:0/2736861493 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb864007b40 con 0x7fb8740f7470 2024-04-12T05:08:15.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.382+0000 7fb870ff9700 1 -- 172.21.15.113:0/2736861493 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb86400f040 con 0x7fb8740f7470 2024-04-12T05:08:15.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.382+0000 7fb870ff9700 1 -- 172.21.15.113:0/2736861493 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8640147b0 con 0x7fb8740f7470 2024-04-12T05:08:15.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.383+0000 7fb879e0b700 1 -- 172.21.15.113:0/2736861493 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb874113c20 con 0x7fb8740f7470 2024-04-12T05:08:15.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.383+0000 7fb879e0b700 1 -- 172.21.15.113:0/2736861493 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8741141b0 con 0x7fb8740f7470 2024-04-12T05:08:15.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.384+0000 7fb879e0b700 1 -- 172.21.15.113:0/2736861493 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb874106590 con 0x7fb8740f7470 2024-04-12T05:08:15.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.385+0000 7fb870ff9700 1 -- 172.21.15.113:0/2736861493 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fb864009160 con 0x7fb8740f7470 2024-04-12T05:08:15.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.385+0000 7fb870ff9700 1 --2- 172.21.15.113:0/2736861493 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb85c0611d0 0x7fb85c063690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:15.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.386+0000 7fb870ff9700 1 -- 172.21.15.113:0/2736861493 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7fb86407c780 con 0x7fb8740f7470 2024-04-12T05:08:15.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.386+0000 7fb8737fe700 1 --2- 172.21.15.113:0/2736861493 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb85c0611d0 0x7fb85c063690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:15.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.386+0000 7fb8737fe700 1 --2- 172.21.15.113:0/2736861493 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb85c0611d0 0x7fb85c063690 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb8680047c0 tx=0x7fb868009cb0).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:15.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.391+0000 7fb870ff9700 1 -- 172.21.15.113:0/2736861493 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb8640506d0 con 0x7fb8740f7470 2024-04-12T05:08:15.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.636+0000 7fb879e0b700 1 -- 172.21.15.113:0/2736861493 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb874059120 con 0x7fb85c0611d0 2024-04-12T05:08:15.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.650+0000 7fb870ff9700 1 -- 172.21.15.113:0/2736861493 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7fb874059120 con 0x7fb85c0611d0 2024-04-12T05:08:15.652 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:08:15.652 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (5m) 95s ago 7m 26.2M - 0.23.0 ba2b418f427c aae81b8048e6 2024-04-12T05:08:15.652 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (7m) 95s ago 7m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:08:15.652 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (6m) 102s ago 6m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:08:15.652 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (5m) 95s ago 7m 59.0M - 8.3.5 dad864ee21e9 7d83fd029dea 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (119s) 95s ago 119s 17.0M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (114s) 95s ago 113s 14.4M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (117s) 102s ago 116s 14.4M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (111s) 102s ago 111s 11.9M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:9283 running (8m) 95s ago 8m 433M - 16.2.15-1-g8963e9de 727f386679cb 3fab7f291e6b 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283 running (6m) 102s ago 6m 387M - 16.2.15-1-g8963e9de 727f386679cb 58d65802163f 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (8m) 95s ago 8m 61.2M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (6m) 102s ago 6m 60.1M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (6m) 95s ago 6m 19.1M - 1.3.1 1dbe0e931976 046bcfb16579 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (6m) 102s ago 6m 26.2M - 1.3.1 1dbe0e931976 bc604318825c 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (5m) 95s ago 5m 39.9M 4640M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (5m) 95s ago 5m 42.0M 4640M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (4m) 95s ago 4m 40.0M 4640M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (4m) 102s ago 4m 45.4M 4982M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (3m) 102s ago 3m 39.6M 4982M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:08:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (3m) 102s ago 3m 39.1M 4982M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:08:15.654 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (6m) 102s ago 6m 60.6M - 2.33.4 514e6a882f6e b4259537bd55 2024-04-12T05:08:15.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.652+0000 7fb879e0b700 1 -- 172.21.15.113:0/2736861493 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb85c0611d0 msgr2=0x7fb85c063690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:15.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.652+0000 7fb879e0b700 1 --2- 172.21.15.113:0/2736861493 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb85c0611d0 0x7fb85c063690 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb8680047c0 tx=0x7fb868009cb0).stop 2024-04-12T05:08:15.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.652+0000 7fb879e0b700 1 -- 172.21.15.113:0/2736861493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8740f7470 msgr2=0x7fb87410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:15.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.652+0000 7fb879e0b700 1 --2- 172.21.15.113:0/2736861493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8740f7470 0x7fb87410ea90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb8640041d0 tx=0x7fb864009f60).stop 2024-04-12T05:08:15.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.652+0000 7fb879e0b700 1 -- 172.21.15.113:0/2736861493 shutdown_connections 2024-04-12T05:08:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.652+0000 7fb879e0b700 1 --2- 172.21.15.113:0/2736861493 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fb85c0611d0 0x7fb85c063690 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.652+0000 7fb879e0b700 1 --2- 172.21.15.113:0/2736861493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8740f6ad0 0x7fb87410e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.652+0000 7fb879e0b700 1 --2- 172.21.15.113:0/2736861493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8740f7470 0x7fb87410ea90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.652+0000 7fb879e0b700 1 -- 172.21.15.113:0/2736861493 >> 172.21.15.113:0/2736861493 conn(0x7fb8740f1220 msgr2=0x7fb8740fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.652+0000 7fb879e0b700 1 -- 172.21.15.113:0/2736861493 shutdown_connections 2024-04-12T05:08:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.652+0000 7fb879e0b700 1 -- 172.21.15.113:0/2736861493 wait complete. 2024-04-12T05:08:15.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.782+0000 7fe09bc68700 1 -- 172.21.15.113:0/2216662126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0940f8aa0 msgr2=0x7fe0940f8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:15.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.782+0000 7fe09bc68700 1 --2- 172.21.15.113:0/2216662126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0940f8aa0 0x7fe0940f8f00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fe084004730 tx=0x7fe084009cb0).stop 2024-04-12T05:08:15.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.782+0000 7fe09bc68700 1 -- 172.21.15.113:0/2216662126 shutdown_connections 2024-04-12T05:08:15.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.782+0000 7fe09bc68700 1 --2- 172.21.15.113:0/2216662126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0940f8aa0 0x7fe0940f8f00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.782+0000 7fe09bc68700 1 --2- 172.21.15.113:0/2216662126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940f7800 0x7fe0940f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.782+0000 7fe09bc68700 1 -- 172.21.15.113:0/2216662126 >> 172.21.15.113:0/2216662126 conn(0x7fe0940f2f80 msgr2=0x7fe0940f53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:15.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.782+0000 7fe09bc68700 1 -- 172.21.15.113:0/2216662126 shutdown_connections 2024-04-12T05:08:15.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.782+0000 7fe09bc68700 1 -- 172.21.15.113:0/2216662126 wait complete. 2024-04-12T05:08:15.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.783+0000 7fe09bc68700 1 Processor -- start 2024-04-12T05:08:15.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.783+0000 7fe09bc68700 1 -- start start 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.783+0000 7fe09bc68700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940f7800 0x7fe094107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.783+0000 7fe09bc68700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0940f8aa0 0x7fe0941084e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.783+0000 7fe09bc68700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe094108a90 con 0x7fe0940f8aa0 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.783+0000 7fe09bc68700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe094108bd0 con 0x7fe0940f7800 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.783+0000 7fe099203700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0940f8aa0 0x7fe0941084e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.783+0000 7fe099203700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0940f8aa0 0x7fe0941084e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46700/0 (socket says 172.21.15.113:46700) 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.783+0000 7fe099203700 1 -- 172.21.15.113:0/3392331629 learned_addr learned my addr 172.21.15.113:0/3392331629 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.783+0000 7fe099a04700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940f7800 0x7fe094107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.784+0000 7fe099203700 1 -- 172.21.15.113:0/3392331629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940f7800 msgr2=0x7fe094107ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.784+0000 7fe099203700 1 --2- 172.21.15.113:0/3392331629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940f7800 0x7fe094107ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.784+0000 7fe099203700 1 -- 172.21.15.113:0/3392331629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe084005040 con 0x7fe0940f8aa0 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.784+0000 7fe099a04700 1 --2- 172.21.15.113:0/3392331629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940f7800 0x7fe094107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.784+0000 7fe099203700 1 --2- 172.21.15.113:0/3392331629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0940f8aa0 0x7fe0941084e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fe084000c00 tx=0x7fe08400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:15.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.784+0000 7fe08affd700 1 -- 172.21.15.113:0/3392331629 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe084019070 con 0x7fe0940f8aa0 2024-04-12T05:08:15.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.784+0000 7fe09bc68700 1 -- 172.21.15.113:0/3392331629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09406af70 con 0x7fe0940f8aa0 2024-04-12T05:08:15.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.784+0000 7fe08affd700 1 -- 172.21.15.113:0/3392331629 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0840097d0 con 0x7fe0940f8aa0 2024-04-12T05:08:15.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.784+0000 7fe09bc68700 1 -- 172.21.15.113:0/3392331629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09406b530 con 0x7fe0940f8aa0 2024-04-12T05:08:15.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.785+0000 7fe09bc68700 1 -- 172.21.15.113:0/3392331629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0940fffc0 con 0x7fe0940f8aa0 2024-04-12T05:08:15.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.785+0000 7fe08affd700 1 -- 172.21.15.113:0/3392331629 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe08401eda0 con 0x7fe0940f8aa0 2024-04-12T05:08:15.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.786+0000 7fe08affd700 1 -- 172.21.15.113:0/3392331629 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7fe084014420 con 0x7fe0940f8aa0 2024-04-12T05:08:15.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.786+0000 7fe08affd700 1 --2- 172.21.15.113:0/3392331629 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe0800611d0 0x7fe080063690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:15.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.787+0000 7fe099a04700 1 --2- 172.21.15.113:0/3392331629 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe0800611d0 0x7fe080063690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:15.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.787+0000 7fe099a04700 1 --2- 172.21.15.113:0/3392331629 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe0800611d0 0x7fe080063690 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe090009a00 tx=0x7fe09000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:15.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.790+0000 7fe08affd700 1 -- 172.21.15.113:0/3392331629 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7fe084051db0 con 0x7fe0940f8aa0 2024-04-12T05:08:15.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:15.791+0000 7fe08affd700 1 -- 172.21.15.113:0/3392331629 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe084083060 con 0x7fe0940f8aa0 2024-04-12T05:08:15.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:14.843778+0000 mgr.smithi113.ctsazg (mgr.14194) 277 : audit [DBG] from='client.14528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:15.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:15.246590+0000 mgr.smithi113.ctsazg (mgr.14194) 278 : audit [DBG] from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:16.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.133+0000 7fe09bc68700 1 -- 172.21.15.113:0/3392331629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe094101e30 con 0x7fe0940f8aa0 2024-04-12T05:08:16.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.134+0000 7fe08affd700 1 -- 172.21.15.113:0/3392331629 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fe084083290 con 0x7fe0940f8aa0 2024-04-12T05:08:16.136 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:08:16.136 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:08:16.136 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:08:16.136 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:08:16.136 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:08:16.136 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:08:16.137 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:08:16.137 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:08:16.137 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:08:16.137 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:08:16.137 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:08:16.137 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:08:16.137 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:08:16.137 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:08:16.137 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 14 2024-04-12T05:08:16.137 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:08:16.137 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:08:16.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.136+0000 7fe09bc68700 1 -- 172.21.15.113:0/3392331629 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe0800611d0 msgr2=0x7fe080063690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:16.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.136+0000 7fe09bc68700 1 --2- 172.21.15.113:0/3392331629 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe0800611d0 0x7fe080063690 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe090009a00 tx=0x7fe09000b040).stop 2024-04-12T05:08:16.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.136+0000 7fe09bc68700 1 -- 172.21.15.113:0/3392331629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0940f8aa0 msgr2=0x7fe0941084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:16.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.136+0000 7fe09bc68700 1 --2- 172.21.15.113:0/3392331629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0940f8aa0 0x7fe0941084e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fe084000c00 tx=0x7fe08400b040).stop 2024-04-12T05:08:16.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.136+0000 7fe09bc68700 1 -- 172.21.15.113:0/3392331629 shutdown_connections 2024-04-12T05:08:16.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.136+0000 7fe09bc68700 1 --2- 172.21.15.113:0/3392331629 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7fe0800611d0 0x7fe080063690 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:16.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.136+0000 7fe09bc68700 1 --2- 172.21.15.113:0/3392331629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0940f7800 0x7fe094107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:16.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.137+0000 7fe09bc68700 1 --2- 172.21.15.113:0/3392331629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0940f8aa0 0x7fe0941084e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:16.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.137+0000 7fe09bc68700 1 -- 172.21.15.113:0/3392331629 >> 172.21.15.113:0/3392331629 conn(0x7fe0940f2f80 msgr2=0x7fe0940fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:16.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.137+0000 7fe09bc68700 1 -- 172.21.15.113:0/3392331629 shutdown_connections 2024-04-12T05:08:16.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.137+0000 7fe09bc68700 1 -- 172.21.15.113:0/3392331629 wait complete. 2024-04-12T05:08:16.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:14.843778+0000 mgr.smithi113.ctsazg (mgr.14194) 277 : audit [DBG] from='client.14528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:16.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:15.246590+0000 mgr.smithi113.ctsazg (mgr.14194) 278 : audit [DBG] from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:16.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.276+0000 7f2f769cc700 1 -- 172.21.15.113:0/3559169847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700f5e70 msgr2=0x7f2f700f6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:16.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.276+0000 7f2f769cc700 1 --2- 172.21.15.113:0/3559169847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700f5e70 0x7f2f700f6290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f2f58004730 tx=0x7f2f58009cb0).stop 2024-04-12T05:08:16.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.276+0000 7f2f769cc700 1 -- 172.21.15.113:0/3559169847 shutdown_connections 2024-04-12T05:08:16.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.276+0000 7f2f769cc700 1 --2- 172.21.15.113:0/3559169847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700f7070 0x7f2f700f74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:16.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.276+0000 7f2f769cc700 1 --2- 172.21.15.113:0/3559169847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700f5e70 0x7f2f700f6290 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0x7f2f58004730 tx=0x7f2f58009cb0).stop 2024-04-12T05:08:16.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.276+0000 7f2f769cc700 1 -- 172.21.15.113:0/3559169847 >> 172.21.15.113:0/3559169847 conn(0x7f2f700f15d0 msgr2=0x7f2f700f39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:16.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.276+0000 7f2f769cc700 1 -- 172.21.15.113:0/3559169847 shutdown_connections 2024-04-12T05:08:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.276+0000 7f2f769cc700 1 -- 172.21.15.113:0/3559169847 wait complete. 2024-04-12T05:08:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.277+0000 7f2f769cc700 1 Processor -- start 2024-04-12T05:08:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.277+0000 7f2f769cc700 1 -- start start 2024-04-12T05:08:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.277+0000 7f2f769cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700f7070 0x7f2f700fef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.277+0000 7f2f769cc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700fd570 0x7f2f700fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.277+0000 7f2f769cc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700ff450 con 0x7f2f700f7070 2024-04-12T05:08:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.277+0000 7f2f769cc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7010be70 con 0x7f2f700fd570 2024-04-12T05:08:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.277+0000 7f2f6ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700f7070 0x7f2f700fef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.277+0000 7f2f6ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700f7070 0x7f2f700fef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46722/0 (socket says 172.21.15.113:46722) 2024-04-12T05:08:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.277+0000 7f2f6ffff700 1 -- 172.21.15.113:0/1455185544 learned_addr learned my addr 172.21.15.113:0/1455185544 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:08:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.277+0000 7f2f6f7fe700 1 --2- 172.21.15.113:0/1455185544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700fd570 0x7f2f700fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.278+0000 7f2f6ffff700 1 -- 172.21.15.113:0/1455185544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700fd570 msgr2=0x7f2f700fd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.278+0000 7f2f6ffff700 1 --2- 172.21.15.113:0/1455185544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700fd570 0x7f2f700fd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.278+0000 7f2f6ffff700 1 -- 172.21.15.113:0/1455185544 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f58005040 con 0x7f2f700f7070 2024-04-12T05:08:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.278+0000 7f2f6ffff700 1 --2- 172.21.15.113:0/1455185544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700f7070 0x7f2f700fef10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f2f58005010 tx=0x7f2f58009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:16.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.278+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/1455185544 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f58009320 con 0x7f2f700f7070 2024-04-12T05:08:16.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.278+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/1455185544 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f58007e90 con 0x7f2f700f7070 2024-04-12T05:08:16.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.278+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/1455185544 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f5801edd0 con 0x7f2f700f7070 2024-04-12T05:08:16.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.279+0000 7f2f769cc700 1 -- 172.21.15.113:0/1455185544 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7010c0c0 con 0x7f2f700f7070 2024-04-12T05:08:16.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.279+0000 7f2f769cc700 1 -- 172.21.15.113:0/1455185544 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7010c5d0 con 0x7f2f700f7070 2024-04-12T05:08:16.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.279+0000 7f2f769cc700 1 -- 172.21.15.113:0/1455185544 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f700ffff0 con 0x7f2f700f7070 2024-04-12T05:08:16.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.280+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/1455185544 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 77153+0+0 (secure 0 0 0) 0x7f2f5801e640 con 0x7f2f700f7070 2024-04-12T05:08:16.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.281+0000 7f2f6d7fa700 1 --2- 172.21.15.113:0/1455185544 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2f5c061180 0x7f2f5c063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:16.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.281+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/1455185544 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5365+0+0 (secure 0 0 0) 0x7f2f5807db20 con 0x7f2f700f7070 2024-04-12T05:08:16.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.281+0000 7f2f6f7fe700 1 --2- 172.21.15.113:0/1455185544 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2f5c061180 0x7f2f5c063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:16.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.282+0000 7f2f6f7fe700 1 --2- 172.21.15.113:0/1455185544 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2f5c061180 0x7f2f5c063640 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2f60009a00 tx=0x7f2f6000b040).ready entity=mgr.14194 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:16.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.285+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/1455185544 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2f58051b30 con 0x7f2f700f7070 2024-04-12T05:08:16.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.582+0000 7f2f769cc700 1 -- 172.21.15.113:0/1455185544 --> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f700fdee0 con 0x7f2f5c061180 2024-04-12T05:08:16.586 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:08:16.586 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:08:16.586 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:08:16.586 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:08:16.586 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [], 2024-04-12T05:08:16.586 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "0/2 daemons upgraded", 2024-04-12T05:08:16.586 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 image on host smithi192", 2024-04-12T05:08:16.586 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:08:16.586 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:08:16.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.584+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/1455185544 <== mgr.14194 v2:172.21.15.113:6800/593576174 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f2f700fdee0 con 0x7f2f5c061180 2024-04-12T05:08:16.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.586+0000 7f2f769cc700 1 -- 172.21.15.113:0/1455185544 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2f5c061180 msgr2=0x7f2f5c063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:16.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.586+0000 7f2f769cc700 1 --2- 172.21.15.113:0/1455185544 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2f5c061180 0x7f2f5c063640 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2f60009a00 tx=0x7f2f6000b040).stop 2024-04-12T05:08:16.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.586+0000 7f2f769cc700 1 -- 172.21.15.113:0/1455185544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700f7070 msgr2=0x7f2f700fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:16.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.586+0000 7f2f769cc700 1 --2- 172.21.15.113:0/1455185544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700f7070 0x7f2f700fef10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f2f58005010 tx=0x7f2f58009cb0).stop 2024-04-12T05:08:16.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.587+0000 7f2f769cc700 1 -- 172.21.15.113:0/1455185544 shutdown_connections 2024-04-12T05:08:16.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.587+0000 7f2f769cc700 1 --2- 172.21.15.113:0/1455185544 >> [v2:172.21.15.113:6800/593576174,v1:172.21.15.113:6801/593576174] conn(0x7f2f5c061180 0x7f2f5c063640 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:16.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.587+0000 7f2f769cc700 1 --2- 172.21.15.113:0/1455185544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700f7070 0x7f2f700fef10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:16.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.587+0000 7f2f769cc700 1 --2- 172.21.15.113:0/1455185544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700fd570 0x7f2f700fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:16.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.587+0000 7f2f769cc700 1 -- 172.21.15.113:0/1455185544 >> 172.21.15.113:0/1455185544 conn(0x7f2f700f15d0 msgr2=0x7f2f700fab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:16.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.587+0000 7f2f769cc700 1 -- 172.21.15.113:0/1455185544 shutdown_connections 2024-04-12T05:08:16.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:16.587+0000 7f2f769cc700 1 -- 172.21.15.113:0/1455185544 wait complete. 2024-04-12T05:08:16.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:15.579509+0000 mgr.smithi113.ctsazg (mgr.14194) 279 : cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 118 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 3.8 MiB/s wr, 417 op/s 2024-04-12T05:08:16.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:15.638367+0000 mgr.smithi113.ctsazg (mgr.14194) 280 : audit [DBG] from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:16.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:16.134685+0000 mon.smithi113 (mon.0) 621 : audit [DBG] from='client.? 172.21.15.113:0/3392331629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:08:17.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:15.579509+0000 mgr.smithi113.ctsazg (mgr.14194) 279 : cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 118 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 3.8 MiB/s wr, 417 op/s 2024-04-12T05:08:17.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:15.638367+0000 mgr.smithi113.ctsazg (mgr.14194) 280 : audit [DBG] from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:17.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:16.134685+0000 mon.smithi113 (mon.0) 621 : audit [DBG] from='client.? 172.21.15.113:0/3392331629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:08:17.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:16.584569+0000 mgr.smithi113.ctsazg (mgr.14194) 281 : audit [DBG] from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:17.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:17.514402+0000 mon.smithi113 (mon.0) 622 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:08:17.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:17.515149+0000 mon.smithi113 (mon.0) 623 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.ocppyo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:08:17.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:17.515805+0000 mon.smithi113 (mon.0) 624 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:08:17.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:17.516370+0000 mon.smithi113 (mon.0) 625 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:08:18.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:16.584569+0000 mgr.smithi113.ctsazg (mgr.14194) 281 : audit [DBG] from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:18.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:17.514402+0000 mon.smithi113 (mon.0) 622 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:08:18.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:17.515149+0000 mon.smithi113 (mon.0) 623 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.ocppyo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:08:18.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:17.515805+0000 mon.smithi113 (mon.0) 624 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:08:18.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:17.516370+0000 mon.smithi113 (mon.0) 625 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:08:18.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:08:17.510185+0000 mgr.smithi113.ctsazg (mgr.14194) 282 : cephadm [INF] Upgrade: Updating mgr.smithi192.ocppyo 2024-04-12T05:08:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:08:17.516983+0000 mgr.smithi113.ctsazg (mgr.14194) 283 : cephadm [INF] Deploying daemon mgr.smithi192.ocppyo on smithi192 2024-04-12T05:08:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:17.579996+0000 mgr.smithi113.ctsazg (mgr.14194) 284 : cluster [DBG] pgmap v238: 65 pgs: 65 active+clean; 122 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 3.9 MiB/s wr, 448 op/s 2024-04-12T05:08:19.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:08:17.510185+0000 mgr.smithi113.ctsazg (mgr.14194) 282 : cephadm [INF] Upgrade: Updating mgr.smithi192.ocppyo 2024-04-12T05:08:19.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:08:17.516983+0000 mgr.smithi113.ctsazg (mgr.14194) 283 : cephadm [INF] Deploying daemon mgr.smithi192.ocppyo on smithi192 2024-04-12T05:08:19.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:17.579996+0000 mgr.smithi113.ctsazg (mgr.14194) 284 : cluster [DBG] pgmap v238: 65 pgs: 65 active+clean; 122 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 3.9 MiB/s wr, 448 op/s 2024-04-12T05:08:20.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:19.580387+0000 mgr.smithi113.ctsazg (mgr.14194) 285 : cluster [DBG] pgmap v239: 65 pgs: 65 active+clean; 127 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 3.5 MiB/s wr, 439 op/s 2024-04-12T05:08:21.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:19.580387+0000 mgr.smithi113.ctsazg (mgr.14194) 285 : cluster [DBG] pgmap v239: 65 pgs: 65 active+clean; 127 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 3.5 MiB/s wr, 439 op/s 2024-04-12T05:08:22.884 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:21.444750+0000 mon.smithi113 (mon.0) 626 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:08:22.884 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:21.445696+0000 mon.smithi113 (mon.0) 627 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:08:22.884 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:21.580864+0000 mgr.smithi113.ctsazg (mgr.14194) 286 : cluster [DBG] pgmap v240: 65 pgs: 65 active+clean; 144 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 4.4 MiB/s wr, 493 op/s 2024-04-12T05:08:22.923 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:21.444750+0000 mon.smithi113 (mon.0) 626 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:08:22.923 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:21.445696+0000 mon.smithi113 (mon.0) 627 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:08:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:21.580864+0000 mgr.smithi113.ctsazg (mgr.14194) 286 : cluster [DBG] pgmap v240: 65 pgs: 65 active+clean; 144 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 4.4 MiB/s wr, 493 op/s 2024-04-12T05:08:24.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:23.581270+0000 mgr.smithi113.ctsazg (mgr.14194) 287 : cluster [DBG] pgmap v241: 65 pgs: 65 active+clean; 148 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 3.6 MiB/s wr, 421 op/s 2024-04-12T05:08:24.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:23.581270+0000 mgr.smithi113.ctsazg (mgr.14194) 287 : cluster [DBG] pgmap v241: 65 pgs: 65 active+clean; 148 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 3.6 MiB/s wr, 421 op/s 2024-04-12T05:08:26.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:25.773655+0000 mon.smithi113 (mon.0) 628 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:08:26.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:26.196131+0000 mon.smithi113 (mon.0) 629 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:08:26.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:26.196906+0000 mon.smithi113 (mon.0) 630 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:08:26.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:26.202494+0000 mon.smithi113 (mon.0) 631 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:08:26.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:26.208655+0000 mon.smithi113 (mon.0) 632 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:08:26.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:26.213986+0000 mon.smithi113 (mon.0) 633 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr fail", "who": "smithi113.ctsazg"}]: dispatch 2024-04-12T05:08:26.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:26.220920+0000 mon.smithi113 (mon.0) 634 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2024-04-12T05:08:26.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:26.221103+0000 mon.smithi113 (mon.0) 635 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "mgr fail", "who": "smithi113.ctsazg"}]': finished 2024-04-12T05:08:26.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:26.221145+0000 mon.smithi113 (mon.0) 636 : cluster [DBG] mgrmap e19: smithi192.ocppyo(active, starting, since 0.00672258s) 2024-04-12T05:08:26.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:25.773655+0000 mon.smithi113 (mon.0) 628 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:08:26.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:26.196131+0000 mon.smithi113 (mon.0) 629 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:08:26.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:26.196906+0000 mon.smithi113 (mon.0) 630 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:08:26.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:26.202494+0000 mon.smithi113 (mon.0) 631 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' 2024-04-12T05:08:26.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:26.208655+0000 mon.smithi113 (mon.0) 632 : audit [DBG] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:08:26.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:26.213986+0000 mon.smithi113 (mon.0) 633 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr fail", "who": "smithi113.ctsazg"}]: dispatch 2024-04-12T05:08:26.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:26.220920+0000 mon.smithi113 (mon.0) 634 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2024-04-12T05:08:26.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:26.221103+0000 mon.smithi113 (mon.0) 635 : audit [INF] from='mgr.14194 172.21.15.113:0/1613537956' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "mgr fail", "who": "smithi113.ctsazg"}]': finished 2024-04-12T05:08:26.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:26.221145+0000 mon.smithi113 (mon.0) 636 : cluster [DBG] mgrmap e19: smithi192.ocppyo(active, starting, since 0.00672258s) 2024-04-12T05:08:30.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:30.649310+0000 mon.smithi113 (mon.0) 637 : cluster [INF] Active manager daemon smithi192.ocppyo restarted 2024-04-12T05:08:30.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:30.649773+0000 mon.smithi113 (mon.0) 638 : cluster [INF] Activating manager daemon smithi192.ocppyo 2024-04-12T05:08:30.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.658729+0000 mon.smithi192 (mon.1) 15 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/crt"}]: dispatch 2024-04-12T05:08:30.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.658892+0000 mon.smithi192 (mon.1) 16 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:08:30.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:30.659058+0000 mon.smithi113 (mon.0) 639 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-04-12T05:08:30.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.659084+0000 mon.smithi192 (mon.1) 17 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:08:30.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:30.659885+0000 mon.smithi113 (mon.0) 640 : cluster [DBG] mgrmap e20: smithi192.ocppyo(active, starting, since 0.0103584s) 2024-04-12T05:08:30.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.661072+0000 mon.smithi192 (mon.1) 18 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:08:30.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.661481+0000 mon.smithi192 (mon.1) 19 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.661799+0000 mon.smithi192 (mon.1) 20 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.662086+0000 mon.smithi192 (mon.1) 21 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.662432+0000 mon.smithi192 (mon.1) 22 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mgr metadata", "who": "smithi192.ocppyo", "id": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.662609+0000 mon.smithi192 (mon.1) 23 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.662932+0000 mon.smithi192 (mon.1) 24 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.663224+0000 mon.smithi192 (mon.1) 25 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.663631+0000 mon.smithi192 (mon.1) 26 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.663857+0000 mon.smithi192 (mon.1) 27 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.664088+0000 mon.smithi192 (mon.1) 28 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.664405+0000 mon.smithi192 (mon.1) 29 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.665231+0000 mon.smithi192 (mon.1) 30 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:08:30.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.665624+0000 mon.smithi192 (mon.1) 31 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:08:30.914 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.666056+0000 mon.smithi192 (mon.1) 32 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:08:30.914 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.666279+0000 mon.smithi192 (mon.1) 33 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/key"}]: dispatch 2024-04-12T05:08:30.914 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.667499+0000 mon.smithi192 (mon.1) 34 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:08:30.976 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:30.649310+0000 mon.smithi113 (mon.0) 637 : cluster [INF] Active manager daemon smithi192.ocppyo restarted 2024-04-12T05:08:30.976 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:30.649773+0000 mon.smithi113 (mon.0) 638 : cluster [INF] Activating manager daemon smithi192.ocppyo 2024-04-12T05:08:30.976 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.658729+0000 mon.smithi192 (mon.1) 15 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/crt"}]: dispatch 2024-04-12T05:08:30.976 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.658892+0000 mon.smithi192 (mon.1) 16 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:30.659058+0000 mon.smithi113 (mon.0) 639 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.659084+0000 mon.smithi192 (mon.1) 17 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:30.659885+0000 mon.smithi113 (mon.0) 640 : cluster [DBG] mgrmap e20: smithi192.ocppyo(active, starting, since 0.0103584s) 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.661072+0000 mon.smithi192 (mon.1) 18 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.661481+0000 mon.smithi192 (mon.1) 19 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.661799+0000 mon.smithi192 (mon.1) 20 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.662086+0000 mon.smithi192 (mon.1) 21 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.662432+0000 mon.smithi192 (mon.1) 22 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mgr metadata", "who": "smithi192.ocppyo", "id": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.662609+0000 mon.smithi192 (mon.1) 23 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.662932+0000 mon.smithi192 (mon.1) 24 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.663224+0000 mon.smithi192 (mon.1) 25 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.663631+0000 mon.smithi192 (mon.1) 26 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.663857+0000 mon.smithi192 (mon.1) 27 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:08:30.977 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.664088+0000 mon.smithi192 (mon.1) 28 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:08:30.978 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.664405+0000 mon.smithi192 (mon.1) 29 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:08:30.978 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.665231+0000 mon.smithi192 (mon.1) 30 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:08:30.978 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.665624+0000 mon.smithi192 (mon.1) 31 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:08:30.978 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.666056+0000 mon.smithi192 (mon.1) 32 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:08:30.978 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.666279+0000 mon.smithi192 (mon.1) 33 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/key"}]: dispatch 2024-04-12T05:08:30.978 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.667499+0000 mon.smithi192 (mon.1) 34 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:08:31.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:30.849651+0000 mon.smithi113 (mon.0) 641 : cluster [INF] Manager daemon smithi192.ocppyo is now available 2024-04-12T05:08:31.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.876020+0000 mon.smithi113 (mon.0) 642 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:31.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.884443+0000 mon.smithi192 (mon.1) 35 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:08:31.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.908388+0000 mon.smithi192 (mon.1) 36 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:08:31.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.915737+0000 mon.smithi192 (mon.1) 37 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.ocppyo/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:08:31.996 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.916202+0000 mon.smithi113 (mon.0) 643 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.ocppyo/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:08:31.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.931495+0000 mon.smithi192 (mon.1) 38 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.ocppyo/trash_purge_schedule"}]: dispatch 2024-04-12T05:08:31.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.931908+0000 mon.smithi113 (mon.0) 644 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.ocppyo/trash_purge_schedule"}]: dispatch 2024-04-12T05:08:31.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:30.935067+0000 mon.smithi113 (mon.0) 645 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:31.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:31.459241+0000 mon.smithi113 (mon.0) 646 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:31.997 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:31.469989+0000 mon.smithi113 (mon.0) 647 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:32.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:30.849651+0000 mon.smithi113 (mon.0) 641 : cluster [INF] Manager daemon smithi192.ocppyo is now available 2024-04-12T05:08:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.876020+0000 mon.smithi113 (mon.0) 642 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.884443+0000 mon.smithi192 (mon.1) 35 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:08:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.908388+0000 mon.smithi192 (mon.1) 36 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:08:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.915737+0000 mon.smithi192 (mon.1) 37 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.ocppyo/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:08:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.916202+0000 mon.smithi113 (mon.0) 643 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.ocppyo/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:08:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.931495+0000 mon.smithi192 (mon.1) 38 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.ocppyo/trash_purge_schedule"}]: dispatch 2024-04-12T05:08:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.931908+0000 mon.smithi113 (mon.0) 644 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.ocppyo/trash_purge_schedule"}]: dispatch 2024-04-12T05:08:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:30.935067+0000 mon.smithi113 (mon.0) 645 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:31.459241+0000 mon.smithi113 (mon.0) 646 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:31.469989+0000 mon.smithi113 (mon.0) 647 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:32.857 INFO:tasks.workunit.client.1.smithi192.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-12T05:08:32.864 INFO:tasks.workunit.client.1.smithi192.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-12T05:08:32.865 INFO:tasks.workunit.client.1.smithi192.stderr:+ make 2024-04-12T05:08:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:08:31.422976+0000 mgr.smithi192.ocppyo (mgr.24357) 1 : cephadm [INF] Deploying cephadm binary to smithi113 2024-04-12T05:08:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:08:31.462863+0000 mgr.smithi192.ocppyo (mgr.24357) 2 : cephadm [INF] Deploying cephadm binary to smithi192 2024-04-12T05:08:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:31.709071+0000 mon.smithi113 (mon.0) 648 : cluster [DBG] mgrmap e21: smithi192.ocppyo(active, since 1.05955s) 2024-04-12T05:08:32.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:31.721886+0000 mgr.smithi192.ocppyo (mgr.24357) 3 : cluster [DBG] pgmap v3: 65 pgs: 65 active+clean; 195 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:08:33.022 INFO:tasks.workunit.client.1.smithi192.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-12T05:08:33.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:08:31.422976+0000 mgr.smithi192.ocppyo (mgr.24357) 1 : cephadm [INF] Deploying cephadm binary to smithi113 2024-04-12T05:08:33.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:08:31.462863+0000 mgr.smithi192.ocppyo (mgr.24357) 2 : cephadm [INF] Deploying cephadm binary to smithi192 2024-04-12T05:08:33.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:31.709071+0000 mon.smithi113 (mon.0) 648 : cluster [DBG] mgrmap e21: smithi192.ocppyo(active, since 1.05955s) 2024-04-12T05:08:33.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:31.721886+0000 mgr.smithi192.ocppyo (mgr.24357) 3 : cluster [DBG] pgmap v3: 65 pgs: 65 active+clean; 195 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:08:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:32.664954+0000 mgr.smithi192.ocppyo (mgr.24357) 4 : cluster [DBG] pgmap v4: 65 pgs: 65 active+clean; 195 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:08:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:32.722295+0000 mon.smithi113 (mon.0) 649 : cluster [DBG] mgrmap e22: smithi192.ocppyo(active, since 2s) 2024-04-12T05:08:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:08:32.728249+0000 mgr.smithi192.ocppyo (mgr.24357) 5 : cephadm [INF] [12/Apr/2024:05:08:32] ENGINE Bus STARTING 2024-04-12T05:08:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:08:32.871157+0000 mgr.smithi192.ocppyo (mgr.24357) 6 : cephadm [INF] [12/Apr/2024:05:08:32] ENGINE Serving on http://172.21.15.192:8765 2024-04-12T05:08:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:32.902637+0000 mon.smithi192 (mon.1) 39 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T05:08:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:32.903088+0000 mon.smithi113 (mon.0) 650 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T05:08:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:08:33.031681+0000 mgr.smithi192.ocppyo (mgr.24357) 7 : cephadm [INF] [12/Apr/2024:05:08:33] ENGINE Serving on https://172.21.15.192:7150 2024-04-12T05:08:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:08:33.031871+0000 mgr.smithi192.ocppyo (mgr.24357) 8 : cephadm [INF] [12/Apr/2024:05:08:33] ENGINE Bus STARTED 2024-04-12T05:08:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:33.304677+0000 mon.smithi113 (mon.0) 651 : cluster [DBG] Standby manager daemon smithi113.ctsazg started 2024-04-12T05:08:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:33.307483+0000 mon.smithi113 (mon.0) 652 : audit [DBG] from='mgr.? 172.21.15.113:0/2540594912' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.ctsazg/crt"}]: dispatch 2024-04-12T05:08:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:33.308520+0000 mon.smithi113 (mon.0) 653 : audit [DBG] from='mgr.? 172.21.15.113:0/2540594912' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:08:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:33.311159+0000 mon.smithi113 (mon.0) 654 : audit [DBG] from='mgr.? 172.21.15.113:0/2540594912' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.ctsazg/key"}]: dispatch 2024-04-12T05:08:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:33.311518+0000 mon.smithi113 (mon.0) 655 : audit [DBG] from='mgr.? 172.21.15.113:0/2540594912' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:08:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:32.664954+0000 mgr.smithi192.ocppyo (mgr.24357) 4 : cluster [DBG] pgmap v4: 65 pgs: 65 active+clean; 195 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:08:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:32.722295+0000 mon.smithi113 (mon.0) 649 : cluster [DBG] mgrmap e22: smithi192.ocppyo(active, since 2s) 2024-04-12T05:08:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:08:32.728249+0000 mgr.smithi192.ocppyo (mgr.24357) 5 : cephadm [INF] [12/Apr/2024:05:08:32] ENGINE Bus STARTING 2024-04-12T05:08:34.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:08:32.871157+0000 mgr.smithi192.ocppyo (mgr.24357) 6 : cephadm [INF] [12/Apr/2024:05:08:32] ENGINE Serving on http://172.21.15.192:8765 2024-04-12T05:08:34.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:32.902637+0000 mon.smithi192 (mon.1) 39 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T05:08:34.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:32.903088+0000 mon.smithi113 (mon.0) 650 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T05:08:34.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:08:33.031681+0000 mgr.smithi192.ocppyo (mgr.24357) 7 : cephadm [INF] [12/Apr/2024:05:08:33] ENGINE Serving on https://172.21.15.192:7150 2024-04-12T05:08:34.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:08:33.031871+0000 mgr.smithi192.ocppyo (mgr.24357) 8 : cephadm [INF] [12/Apr/2024:05:08:33] ENGINE Bus STARTED 2024-04-12T05:08:34.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:33.304677+0000 mon.smithi113 (mon.0) 651 : cluster [DBG] Standby manager daemon smithi113.ctsazg started 2024-04-12T05:08:34.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:33.307483+0000 mon.smithi113 (mon.0) 652 : audit [DBG] from='mgr.? 172.21.15.113:0/2540594912' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.ctsazg/crt"}]: dispatch 2024-04-12T05:08:34.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:33.308520+0000 mon.smithi113 (mon.0) 653 : audit [DBG] from='mgr.? 172.21.15.113:0/2540594912' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:08:34.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:33.311159+0000 mon.smithi113 (mon.0) 654 : audit [DBG] from='mgr.? 172.21.15.113:0/2540594912' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.ctsazg/key"}]: dispatch 2024-04-12T05:08:34.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:33.311518+0000 mon.smithi113 (mon.0) 655 : audit [DBG] from='mgr.? 172.21.15.113:0/2540594912' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:08:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:33.730833+0000 mon.smithi113 (mon.0) 656 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T05:08:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:33.730912+0000 mon.smithi113 (mon.0) 657 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-04-12T05:08:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:33.731393+0000 mon.smithi113 (mon.0) 658 : cluster [DBG] mgrmap e23: smithi192.ocppyo(active, since 3s), standbys: smithi113.ctsazg 2024-04-12T05:08:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:33.735344+0000 mon.smithi192 (mon.1) 40 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mgr metadata", "who": "smithi113.ctsazg", "id": "smithi113.ctsazg"}]: dispatch 2024-04-12T05:08:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:33.736562+0000 mon.smithi192 (mon.1) 41 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T05:08:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:33.737016+0000 mon.smithi113 (mon.0) 659 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T05:08:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:33.730833+0000 mon.smithi113 (mon.0) 656 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T05:08:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:33.730912+0000 mon.smithi113 (mon.0) 657 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-04-12T05:08:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:33.731393+0000 mon.smithi113 (mon.0) 658 : cluster [DBG] mgrmap e23: smithi192.ocppyo(active, since 3s), standbys: smithi113.ctsazg 2024-04-12T05:08:35.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:33.735344+0000 mon.smithi192 (mon.1) 40 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mgr metadata", "who": "smithi113.ctsazg", "id": "smithi113.ctsazg"}]: dispatch 2024-04-12T05:08:35.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:33.736562+0000 mon.smithi192 (mon.1) 41 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T05:08:35.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:33.737016+0000 mon.smithi113 (mon.0) 659 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T05:08:35.837 INFO:tasks.workunit.client.1.smithi192.stderr:++ readlink -f fsstress 2024-04-12T05:08:35.839 INFO:tasks.workunit.client.1.smithi192.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-12T05:08:35.839 INFO:tasks.workunit.client.1.smithi192.stderr:+ popd 2024-04-12T05:08:35.840 INFO:tasks.workunit.client.1.smithi192.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-12T05:08:35.840 INFO:tasks.workunit.client.1.smithi192.stderr:+ popd 2024-04-12T05:08:35.840 INFO:tasks.workunit.client.1.smithi192.stdout:~/cephtest/mnt.1/client.1/tmp 2024-04-12T05:08:35.841 INFO:tasks.workunit.client.1.smithi192.stderr:++ mktemp -d -p . 2024-04-12T05:08:35.844 INFO:tasks.workunit.client.1.smithi192.stderr:+ T=./tmp.Eu8tZstG0i 2024-04-12T05:08:35.844 INFO:tasks.workunit.client.1.smithi192.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.Eu8tZstG0i -l 1 -n 1000 -p 10 -v 2024-04-12T05:08:35.848 INFO:tasks.workunit.client.1.smithi192.stdout:seed = 1712599252 2024-04-12T05:08:35.854 INFO:tasks.workunit.client.1.smithi192.stdout:0/0: chown . 654 1 2024-04-12T05:08:35.857 INFO:tasks.workunit.client.1.smithi192.stdout:0/1: mkdir d0 0 2024-04-12T05:08:35.857 INFO:tasks.workunit.client.1.smithi192.stdout:0/2: read - no filename 2024-04-12T05:08:35.859 INFO:tasks.workunit.client.1.smithi192.stdout:2/0: rename - no filename 2024-04-12T05:08:35.859 INFO:tasks.workunit.client.1.smithi192.stdout:2/1: write - no filename 2024-04-12T05:08:35.859 INFO:tasks.workunit.client.1.smithi192.stdout:2/2: fdatasync - no filename 2024-04-12T05:08:35.860 INFO:tasks.workunit.client.1.smithi192.stdout:1/0: mkdir d0 0 2024-04-12T05:08:35.860 INFO:tasks.workunit.client.1.smithi192.stdout:1/1: rename d0 to d0/d1 22 2024-04-12T05:08:35.860 INFO:tasks.workunit.client.1.smithi192.stdout:1/2: write - no filename 2024-04-12T05:08:35.860 INFO:tasks.workunit.client.1.smithi192.stdout:1/3: dread - no filename 2024-04-12T05:08:35.861 INFO:tasks.workunit.client.1.smithi192.stdout:0/3: mknod d0/c1 0 2024-04-12T05:08:35.864 INFO:tasks.workunit.client.1.smithi192.stdout:0/4: rename d0/c1 to d0/c2 0 2024-04-12T05:08:35.864 INFO:tasks.workunit.client.1.smithi192.stdout:2/3: symlink l0 0 2024-04-12T05:08:35.864 INFO:tasks.workunit.client.1.smithi192.stdout:2/4: dread - no filename 2024-04-12T05:08:35.864 INFO:tasks.workunit.client.1.smithi192.stdout:2/5: write - no filename 2024-04-12T05:08:35.865 INFO:tasks.workunit.client.1.smithi192.stdout:1/4: mkdir d0/d2 0 2024-04-12T05:08:35.865 INFO:tasks.workunit.client.1.smithi192.stdout:1/5: dwrite - no filename 2024-04-12T05:08:35.866 INFO:tasks.workunit.client.1.smithi192.stdout:4/0: dread - no filename 2024-04-12T05:08:35.866 INFO:tasks.workunit.client.1.smithi192.stdout:4/1: dread - no filename 2024-04-12T05:08:35.867 INFO:tasks.workunit.client.1.smithi192.stdout:4/2: dwrite - no filename 2024-04-12T05:08:35.867 INFO:tasks.workunit.client.1.smithi192.stdout:4/3: fsync - no filename 2024-04-12T05:08:35.867 INFO:tasks.workunit.client.1.smithi192.stdout:4/4: dwrite - no filename 2024-04-12T05:08:35.867 INFO:tasks.workunit.client.1.smithi192.stdout:4/5: dwrite - no filename 2024-04-12T05:08:35.867 INFO:tasks.workunit.client.1.smithi192.stdout:4/6: stat - no entries 2024-04-12T05:08:35.867 INFO:tasks.workunit.client.1.smithi192.stdout:4/7: dread - no filename 2024-04-12T05:08:35.867 INFO:tasks.workunit.client.1.smithi192.stdout:0/5: creat d0/f3 x:0 0 0 2024-04-12T05:08:35.868 INFO:tasks.workunit.client.1.smithi192.stdout:2/6: mkdir d1 0 2024-04-12T05:08:35.868 INFO:tasks.workunit.client.1.smithi192.stdout:2/7: dread - no filename 2024-04-12T05:08:35.868 INFO:tasks.workunit.client.1.smithi192.stdout:2/8: fdatasync - no filename 2024-04-12T05:08:35.868 INFO:tasks.workunit.client.1.smithi192.stdout:2/9: chown d1 2934910 1 2024-04-12T05:08:35.868 INFO:tasks.workunit.client.1.smithi192.stdout:2/10: read - no filename 2024-04-12T05:08:35.869 INFO:tasks.workunit.client.1.smithi192.stdout:2/11: fdatasync - no filename 2024-04-12T05:08:35.869 INFO:tasks.workunit.client.1.smithi192.stdout:2/12: dwrite - no filename 2024-04-12T05:08:35.869 INFO:tasks.workunit.client.1.smithi192.stdout:1/6: creat d0/f3 x:0 0 0 2024-04-12T05:08:35.870 INFO:tasks.workunit.client.1.smithi192.stdout:3/0: creat f0 x:0 0 0 2024-04-12T05:08:35.871 INFO:tasks.workunit.client.1.smithi192.stdout:5/0: dread - no filename 2024-04-12T05:08:35.871 INFO:tasks.workunit.client.1.smithi192.stdout:0/6: symlink d0/l4 0 2024-04-12T05:08:35.872 INFO:tasks.workunit.client.1.smithi192.stdout:0/7: write d0/f3 [818308,123210] 0 2024-04-12T05:08:35.874 INFO:tasks.workunit.client.1.smithi192.stdout:3/1: symlink l1 0 2024-04-12T05:08:35.874 INFO:tasks.workunit.client.1.smithi192.stdout:0/8: write d0/f3 [263244,45602] 0 2024-04-12T05:08:35.874 INFO:tasks.workunit.client.1.smithi192.stdout:4/8: creat f0 x:0 0 0 2024-04-12T05:08:35.875 INFO:tasks.workunit.client.1.smithi192.stdout:6/0: chown . 1 1 2024-04-12T05:08:35.876 INFO:tasks.workunit.client.1.smithi192.stdout:4/9: write f0 [630348,10984] 0 2024-04-12T05:08:35.876 INFO:tasks.workunit.client.1.smithi192.stdout:2/13: creat d1/f2 x:0 0 0 2024-04-12T05:08:35.877 INFO:tasks.workunit.client.1.smithi192.stdout:1/7: creat d0/d2/f4 x:0 0 0 2024-04-12T05:08:35.877 INFO:tasks.workunit.client.1.smithi192.stdout:3/2: creat f2 x:0 0 0 2024-04-12T05:08:35.878 INFO:tasks.workunit.client.1.smithi192.stdout:3/3: readlink l1 0 2024-04-12T05:08:35.878 INFO:tasks.workunit.client.1.smithi192.stdout:5/1: mkdir d0 0 2024-04-12T05:08:35.878 INFO:tasks.workunit.client.1.smithi192.stdout:5/2: write - no filename 2024-04-12T05:08:35.879 INFO:tasks.workunit.client.1.smithi192.stdout:3/4: dread - f0 zero size 2024-04-12T05:08:35.879 INFO:tasks.workunit.client.1.smithi192.stdout:5/3: chown d0 4616127 1 2024-04-12T05:08:35.880 INFO:tasks.workunit.client.1.smithi192.stdout:7/0: dread - no filename 2024-04-12T05:08:35.880 INFO:tasks.workunit.client.1.smithi192.stdout:7/1: truncate - no filename 2024-04-12T05:08:35.880 INFO:tasks.workunit.client.1.smithi192.stdout:0/9: mknod d0/c5 0 2024-04-12T05:08:35.881 INFO:tasks.workunit.client.1.smithi192.stdout:4/10: mkdir d1 0 2024-04-12T05:08:35.883 INFO:tasks.workunit.client.1.smithi192.stdout:2/14: rename l0 to d1/l3 0 2024-04-12T05:08:35.883 INFO:tasks.workunit.client.1.smithi192.stdout:5/4: rename d0 to d0/d1 22 2024-04-12T05:08:35.883 INFO:tasks.workunit.client.1.smithi192.stdout:5/5: chown d0 1263 1 2024-04-12T05:08:35.884 INFO:tasks.workunit.client.1.smithi192.stdout:0/10: dread d0/f3 [0,4194304] 0 2024-04-12T05:08:35.884 INFO:tasks.workunit.client.1.smithi192.stdout:1/8: rename d0/d2/f4 to d0/d2/f5 0 2024-04-12T05:08:35.885 INFO:tasks.workunit.client.1.smithi192.stdout:3/5: mkdir d3 0 2024-04-12T05:08:35.885 INFO:tasks.workunit.client.1.smithi192.stdout:0/11: readlink d0/l4 0 2024-04-12T05:08:35.885 INFO:tasks.workunit.client.1.smithi192.stdout:1/9: stat d0/f3 0 2024-04-12T05:08:35.886 INFO:tasks.workunit.client.1.smithi192.stdout:6/1: getdents . 0 2024-04-12T05:08:35.886 INFO:tasks.workunit.client.1.smithi192.stdout:6/2: write - no filename 2024-04-12T05:08:35.886 INFO:tasks.workunit.client.1.smithi192.stdout:6/3: truncate - no filename 2024-04-12T05:08:35.886 INFO:tasks.workunit.client.1.smithi192.stdout:6/4: fsync - no filename 2024-04-12T05:08:35.886 INFO:tasks.workunit.client.1.smithi192.stdout:8/0: link - no file 2024-04-12T05:08:35.886 INFO:tasks.workunit.client.1.smithi192.stdout:3/6: chown l1 7 1 2024-04-12T05:08:35.887 INFO:tasks.workunit.client.1.smithi192.stdout:3/7: write f0 [201968,59075] 0 2024-04-12T05:08:35.887 INFO:tasks.workunit.client.1.smithi192.stdout:2/15: creat d1/f4 x:0 0 0 2024-04-12T05:08:35.888 INFO:tasks.workunit.client.1.smithi192.stdout:4/11: rename f0 to d1/f2 0 2024-04-12T05:08:35.889 INFO:tasks.workunit.client.1.smithi192.stdout:7/2: creat f0 x:0 0 0 2024-04-12T05:08:35.891 INFO:tasks.workunit.client.1.smithi192.stdout:0/12: mkdir d0/d6 0 2024-04-12T05:08:35.892 INFO:tasks.workunit.client.1.smithi192.stdout:3/8: dread f0 [0,4194304] 0 2024-04-12T05:08:35.892 INFO:tasks.workunit.client.1.smithi192.stdout:7/3: write f0 [579137,71493] 0 2024-04-12T05:08:35.893 INFO:tasks.workunit.client.1.smithi192.stdout:7/4: rmdir - no directory 2024-04-12T05:08:35.893 INFO:tasks.workunit.client.1.smithi192.stdout:0/13: chown d0/d6 705088 1 2024-04-12T05:08:35.895 INFO:tasks.workunit.client.1.smithi192.stdout:1/10: write d0/d2/f5 [68600,4388] 0 2024-04-12T05:08:35.896 INFO:tasks.workunit.client.1.smithi192.stdout:5/6: mknod d0/c2 0 2024-04-12T05:08:35.897 INFO:tasks.workunit.client.1.smithi192.stdout:6/5: creat f0 x:0 0 0 2024-04-12T05:08:35.898 INFO:tasks.workunit.client.1.smithi192.stdout:8/1: symlink l0 0 2024-04-12T05:08:35.898 INFO:tasks.workunit.client.1.smithi192.stdout:8/2: write - no filename 2024-04-12T05:08:35.898 INFO:tasks.workunit.client.1.smithi192.stdout:4/12: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:35.898 INFO:tasks.workunit.client.1.smithi192.stdout:7/5: write f0 [536805,67905] 0 2024-04-12T05:08:35.899 INFO:tasks.workunit.client.1.smithi192.stdout:8/3: stat l0 0 2024-04-12T05:08:35.901 INFO:tasks.workunit.client.1.smithi192.stdout:3/9: unlink f0 0 2024-04-12T05:08:35.903 INFO:tasks.workunit.client.1.smithi192.stdout:6/6: chown f0 34696 1 2024-04-12T05:08:35.903 INFO:tasks.workunit.client.1.smithi192.stdout:9/0: getdents . 0 2024-04-12T05:08:35.903 INFO:tasks.workunit.client.1.smithi192.stdout:9/1: truncate - no filename 2024-04-12T05:08:35.903 INFO:tasks.workunit.client.1.smithi192.stdout:5/7: chown d0/c2 3 1 2024-04-12T05:08:35.904 INFO:tasks.workunit.client.1.smithi192.stdout:7/6: fdatasync f0 0 2024-04-12T05:08:35.904 INFO:tasks.workunit.client.1.smithi192.stdout:9/2: chown . 17669968 1 2024-04-12T05:08:35.904 INFO:tasks.workunit.client.1.smithi192.stdout:4/13: symlink d1/l3 0 2024-04-12T05:08:35.904 INFO:tasks.workunit.client.1.smithi192.stdout:5/8: stat d0 0 2024-04-12T05:08:35.905 INFO:tasks.workunit.client.1.smithi192.stdout:5/9: write - no filename 2024-04-12T05:08:35.905 INFO:tasks.workunit.client.1.smithi192.stdout:1/11: dread d0/d2/f5 [0,4194304] 0 2024-04-12T05:08:35.905 INFO:tasks.workunit.client.1.smithi192.stdout:6/7: chown f0 56 1 2024-04-12T05:08:35.906 INFO:tasks.workunit.client.1.smithi192.stdout:2/16: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:35.906 INFO:tasks.workunit.client.1.smithi192.stdout:4/14: rename d1 to d1/d4 22 2024-04-12T05:08:35.910 INFO:tasks.workunit.client.1.smithi192.stdout:5/10: chown d0/c2 821176934 1 2024-04-12T05:08:35.910 INFO:tasks.workunit.client.1.smithi192.stdout:2/17: rename d1 to d1/d5 22 2024-04-12T05:08:35.910 INFO:tasks.workunit.client.1.smithi192.stdout:1/12: dread d0/d2/f5 [0,4194304] 0 2024-04-12T05:08:35.911 INFO:tasks.workunit.client.1.smithi192.stdout:6/8: rename f0 to f1 0 2024-04-12T05:08:35.911 INFO:tasks.workunit.client.1.smithi192.stdout:2/18: dread - d1/f4 zero size 2024-04-12T05:08:35.911 INFO:tasks.workunit.client.1.smithi192.stdout:0/14: link d0/c5 d0/d6/c7 0 2024-04-12T05:08:35.911 INFO:tasks.workunit.client.1.smithi192.stdout:6/9: stat f1 0 2024-04-12T05:08:35.915 INFO:tasks.workunit.client.1.smithi192.stdout:3/10: getdents d3 0 2024-04-12T05:08:35.915 INFO:tasks.workunit.client.1.smithi192.stdout:4/15: write d1/f2 [624297,55436] 0 2024-04-12T05:08:35.915 INFO:tasks.workunit.client.1.smithi192.stdout:5/11: mknod d0/c3 0 2024-04-12T05:08:35.916 INFO:tasks.workunit.client.1.smithi192.stdout:9/3: mknod c0 0 2024-04-12T05:08:35.918 INFO:tasks.workunit.client.1.smithi192.stdout:5/12: rename d0 to d0/d4 22 2024-04-12T05:08:35.919 INFO:tasks.workunit.client.1.smithi192.stdout:7/7: dwrite f0 [0,4194304] 0 2024-04-12T05:08:35.922 INFO:tasks.workunit.client.1.smithi192.stdout:7/8: chown f0 1 1 2024-04-12T05:08:35.922 INFO:tasks.workunit.client.1.smithi192.stdout:0/15: write d0/f3 [732996,61829] 0 2024-04-12T05:08:35.924 INFO:tasks.workunit.client.1.smithi192.stdout:9/4: creat f1 x:0 0 0 2024-04-12T05:08:35.924 INFO:tasks.workunit.client.1.smithi192.stdout:6/10: write f1 [452236,89139] 0 2024-04-12T05:08:35.929 INFO:tasks.workunit.client.1.smithi192.stdout:9/5: write f1 [11120,62701] 0 2024-04-12T05:08:35.934 INFO:tasks.workunit.client.1.smithi192.stdout:3/11: dwrite f2 [0,4194304] 0 2024-04-12T05:08:35.935 INFO:tasks.workunit.client.1.smithi192.stdout:5/13: mknod d0/c5 0 2024-04-12T05:08:35.935 INFO:tasks.workunit.client.1.smithi192.stdout:5/14: fsync - no filename 2024-04-12T05:08:35.935 INFO:tasks.workunit.client.1.smithi192.stdout:4/16: link d1/f2 d1/f5 0 2024-04-12T05:08:35.935 INFO:tasks.workunit.client.1.smithi192.stdout:3/12: write f2 [1493046,77979] 0 2024-04-12T05:08:35.936 INFO:tasks.workunit.client.1.smithi192.stdout:7/9: rename f0 to f1 0 2024-04-12T05:08:35.937 INFO:tasks.workunit.client.1.smithi192.stdout:6/11: mknod c2 0 2024-04-12T05:08:35.938 INFO:tasks.workunit.client.1.smithi192.stdout:9/6: creat f2 x:0 0 0 2024-04-12T05:08:35.940 INFO:tasks.workunit.client.1.smithi192.stdout:5/15: creat d0/f6 x:0 0 0 2024-04-12T05:08:35.944 INFO:tasks.workunit.client.1.smithi192.stdout:9/7: dread f1 [0,4194304] 0 2024-04-12T05:08:35.945 INFO:tasks.workunit.client.1.smithi192.stdout:9/8: chown f1 0 1 2024-04-12T05:08:35.945 INFO:tasks.workunit.client.1.smithi192.stdout:4/17: mknod d1/c6 0 2024-04-12T05:08:35.946 INFO:tasks.workunit.client.1.smithi192.stdout:4/18: read d1/f5 [281177,28299] 0 2024-04-12T05:08:35.946 INFO:tasks.workunit.client.1.smithi192.stdout:6/12: mkdir d3 0 2024-04-12T05:08:35.947 INFO:tasks.workunit.client.1.smithi192.stdout:4/19: stat d1/f5 0 2024-04-12T05:08:35.947 INFO:tasks.workunit.client.1.smithi192.stdout:5/16: creat d0/f7 x:0 0 0 2024-04-12T05:08:35.948 INFO:tasks.workunit.client.1.smithi192.stdout:9/9: creat f3 x:0 0 0 2024-04-12T05:08:35.949 INFO:tasks.workunit.client.1.smithi192.stdout:5/17: truncate d0/f6 827253 0 2024-04-12T05:08:35.949 INFO:tasks.workunit.client.1.smithi192.stdout:7/10: link f1 f2 0 2024-04-12T05:08:35.953 INFO:tasks.workunit.client.1.smithi192.stdout:3/13: rmdir d3 0 2024-04-12T05:08:35.954 INFO:tasks.workunit.client.1.smithi192.stdout:6/13: rename c2 to d3/c4 0 2024-04-12T05:08:35.956 INFO:tasks.workunit.client.1.smithi192.stdout:9/10: mknod c4 0 2024-04-12T05:08:35.958 INFO:tasks.workunit.client.1.smithi192.stdout:5/18: creat d0/f8 x:0 0 0 2024-04-12T05:08:35.961 INFO:tasks.workunit.client.1.smithi192.stdout:3/14: mknod c4 0 2024-04-12T05:08:35.961 INFO:tasks.workunit.client.1.smithi192.stdout:5/19: write d0/f7 [612958,52560] 0 2024-04-12T05:08:35.962 INFO:tasks.workunit.client.1.smithi192.stdout:6/14: dread f1 [0,4194304] 0 2024-04-12T05:08:35.963 INFO:tasks.workunit.client.1.smithi192.stdout:9/11: rename f1 to f5 0 2024-04-12T05:08:35.964 INFO:tasks.workunit.client.1.smithi192.stdout:6/15: chown f1 2 1 2024-04-12T05:08:35.965 INFO:tasks.workunit.client.1.smithi192.stdout:3/15: mkdir d5 0 2024-04-12T05:08:35.966 INFO:tasks.workunit.client.1.smithi192.stdout:4/20: dwrite d1/f5 [0,4194304] 0 2024-04-12T05:08:35.967 INFO:tasks.workunit.client.1.smithi192.stdout:5/20: unlink d0/f8 0 2024-04-12T05:08:35.967 INFO:tasks.workunit.client.1.smithi192.stdout:3/16: write f2 [3593606,62685] 0 2024-04-12T05:08:35.967 INFO:tasks.workunit.client.1.smithi192.stdout:6/16: dread f1 [0,4194304] 0 2024-04-12T05:08:35.969 INFO:tasks.workunit.client.1.smithi192.stdout:5/21: write d0/f6 [1364364,38881] 0 2024-04-12T05:08:35.969 INFO:tasks.workunit.client.1.smithi192.stdout:5/22: chown d0/c2 363 1 2024-04-12T05:08:35.972 INFO:tasks.workunit.client.1.smithi192.stdout:4/21: symlink d1/l7 0 2024-04-12T05:08:35.975 INFO:tasks.workunit.client.1.smithi192.stdout:6/17: mkdir d3/d5 0 2024-04-12T05:08:35.976 INFO:tasks.workunit.client.1.smithi192.stdout:5/23: mknod d0/c9 0 2024-04-12T05:08:35.977 INFO:tasks.workunit.client.1.smithi192.stdout:3/17: mknod d5/c6 0 2024-04-12T05:08:35.978 INFO:tasks.workunit.client.1.smithi192.stdout:4/22: mkdir d1/d8 0 2024-04-12T05:08:35.978 INFO:tasks.workunit.client.1.smithi192.stdout:4/23: chown d1/d8 33845565 1 2024-04-12T05:08:35.979 INFO:tasks.workunit.client.1.smithi192.stdout:5/24: creat d0/fa x:0 0 0 2024-04-12T05:08:35.981 INFO:tasks.workunit.client.1.smithi192.stdout:3/18: mknod d5/c7 0 2024-04-12T05:08:35.982 INFO:tasks.workunit.client.1.smithi192.stdout:6/18: creat d3/d5/f6 x:0 0 0 2024-04-12T05:08:35.983 INFO:tasks.workunit.client.1.smithi192.stdout:3/19: write f2 [1888130,104423] 0 2024-04-12T05:08:35.990 INFO:tasks.workunit.client.1.smithi192.stdout:5/25: dwrite d0/fa [0,4194304] 0 2024-04-12T05:08:35.991 INFO:tasks.workunit.client.1.smithi192.stdout:5/26: fdatasync d0/f6 0 2024-04-12T05:08:35.994 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:34.665253+0000 mgr.smithi192.ocppyo (mgr.24357) 9 : cluster [DBG] pgmap v6: 65 pgs: 65 active+clean; 195 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:08:35.994 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:34.733228+0000 mon.smithi113 (mon.0) 660 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T05:08:35.994 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:34.733274+0000 mon.smithi113 (mon.0) 661 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-04-12T05:08:35.994 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:35.391419+0000 mon.smithi113 (mon.0) 662 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:36.007 INFO:tasks.workunit.client.1.smithi192.stdout:5/27: dread d0/f6 [0,4194304] 0 2024-04-12T05:08:36.036 INFO:tasks.workunit.client.1.smithi192.stdout:5/28: link d0/c9 d0/cb 0 2024-04-12T05:08:36.036 INFO:tasks.workunit.client.1.smithi192.stdout:5/29: symlink d0/lc 0 2024-04-12T05:08:36.036 INFO:tasks.workunit.client.1.smithi192.stdout:5/30: symlink d0/ld 0 2024-04-12T05:08:36.036 INFO:tasks.workunit.client.1.smithi192.stdout:5/31: write d0/fa [2533742,93580] 0 2024-04-12T05:08:36.036 INFO:tasks.workunit.client.1.smithi192.stdout:5/32: fdatasync d0/f7 0 2024-04-12T05:08:36.036 INFO:tasks.workunit.client.1.smithi192.stdout:5/33: symlink d0/le 0 2024-04-12T05:08:36.047 INFO:tasks.workunit.client.1.smithi192.stdout:1/13: rename d0/d2/f5 to d0/d2/f6 0 2024-04-12T05:08:36.048 INFO:tasks.workunit.client.1.smithi192.stdout:1/14: creat d0/d2/f7 x:0 0 0 2024-04-12T05:08:36.048 INFO:tasks.workunit.client.1.smithi192.stdout:1/15: read - d0/d2/f7 zero size 2024-04-12T05:08:36.048 INFO:tasks.workunit.client.1.smithi192.stdout:2/19: getdents d1 0 2024-04-12T05:08:36.050 INFO:tasks.workunit.client.1.smithi192.stdout:2/20: mknod d1/c6 0 2024-04-12T05:08:36.052 INFO:tasks.workunit.client.1.smithi192.stdout:2/21: symlink d1/l7 0 2024-04-12T05:08:36.054 INFO:tasks.workunit.client.1.smithi192.stdout:2/22: mkdir d1/d8 0 2024-04-12T05:08:36.055 INFO:tasks.workunit.client.1.smithi192.stdout:8/4: getdents . 0 2024-04-12T05:08:36.057 INFO:tasks.workunit.client.1.smithi192.stdout:8/5: creat f1 x:0 0 0 2024-04-12T05:08:36.058 INFO:tasks.workunit.client.1.smithi192.stdout:8/6: write f1 [6918,102193] 0 2024-04-12T05:08:36.061 INFO:tasks.workunit.client.1.smithi192.stdout:8/7: rename l0 to l2 0 2024-04-12T05:08:36.062 INFO:tasks.workunit.client.1.smithi192.stdout:8/8: write f1 [401693,3912] 0 2024-04-12T05:08:36.063 INFO:tasks.workunit.client.1.smithi192.stdout:8/9: write f1 [835090,55361] 0 2024-04-12T05:08:36.065 INFO:tasks.workunit.client.1.smithi192.stdout:6/19: rename f1 to d3/d5/f7 0 2024-04-12T05:08:36.066 INFO:tasks.workunit.client.1.smithi192.stdout:8/10: creat f3 x:0 0 0 2024-04-12T05:08:36.068 INFO:tasks.workunit.client.1.smithi192.stdout:8/11: write f1 [1007584,87691] 0 2024-04-12T05:08:36.068 INFO:tasks.workunit.client.1.smithi192.stdout:8/12: chown f1 147449 1 2024-04-12T05:08:36.068 INFO:tasks.workunit.client.1.smithi192.stdout:6/20: mkdir d3/d8 0 2024-04-12T05:08:36.069 INFO:tasks.workunit.client.1.smithi192.stdout:8/13: write f3 [499736,42962] 0 2024-04-12T05:08:36.073 INFO:tasks.workunit.client.1.smithi192.stdout:6/21: creat d3/f9 x:0 0 0 2024-04-12T05:08:36.073 INFO:tasks.workunit.client.1.smithi192.stdout:6/22: chown d3/d8 235907281 1 2024-04-12T05:08:36.074 INFO:tasks.workunit.client.1.smithi192.stdout:6/23: stat d3/d5/f6 0 2024-04-12T05:08:36.075 INFO:tasks.workunit.client.1.smithi192.stdout:6/24: mknod d3/d5/ca 0 2024-04-12T05:08:36.076 INFO:tasks.workunit.client.1.smithi192.stdout:3/20: write f2 [4906769,119505] 0 2024-04-12T05:08:36.078 INFO:tasks.workunit.client.1.smithi192.stdout:3/21: read f2 [1316977,126776] 0 2024-04-12T05:08:36.079 INFO:tasks.workunit.client.1.smithi192.stdout:3/22: write f2 [2287573,79217] 0 2024-04-12T05:08:36.082 INFO:tasks.workunit.client.1.smithi192.stdout:3/23: rmdir d5 39 2024-04-12T05:08:36.083 INFO:tasks.workunit.client.1.smithi192.stdout:6/25: link d3/d5/ca d3/cb 0 2024-04-12T05:08:36.084 INFO:tasks.workunit.client.1.smithi192.stdout:3/24: write f2 [1886107,126012] 0 2024-04-12T05:08:36.086 INFO:tasks.workunit.client.1.smithi192.stdout:7/11: dwrite f1 [4194304,4194304] 0 2024-04-12T05:08:36.087 INFO:tasks.workunit.client.1.smithi192.stdout:6/26: mkdir d3/dc 0 2024-04-12T05:08:36.091 INFO:tasks.workunit.client.1.smithi192.stdout:7/12: mkdir d3 0 2024-04-12T05:08:36.091 INFO:tasks.workunit.client.1.smithi192.stdout:7/13: stat f1 0 2024-04-12T05:08:36.092 INFO:tasks.workunit.client.1.smithi192.stdout:6/27: write d3/d5/f7 [1368742,79307] 0 2024-04-12T05:08:36.093 INFO:tasks.workunit.client.1.smithi192.stdout:7/14: symlink d3/l4 0 2024-04-12T05:08:36.095 INFO:tasks.workunit.client.1.smithi192.stdout:7/15: creat d3/f5 x:0 0 0 2024-04-12T05:08:36.097 INFO:tasks.workunit.client.1.smithi192.stdout:9/12: rename f5 to f6 0 2024-04-12T05:08:36.097 INFO:tasks.workunit.client.1.smithi192.stdout:9/13: dread - f2 zero size 2024-04-12T05:08:36.098 INFO:tasks.workunit.client.1.smithi192.stdout:6/28: chown d3/cb 22040827 1 2024-04-12T05:08:36.098 INFO:tasks.workunit.client.1.smithi192.stdout:9/14: dread f6 [0,4194304] 0 2024-04-12T05:08:36.098 INFO:tasks.workunit.client.1.smithi192.stdout:6/29: chown d3/f9 8141 1 2024-04-12T05:08:36.105 INFO:tasks.workunit.client.1.smithi192.stdout:9/15: creat f7 x:0 0 0 2024-04-12T05:08:36.106 INFO:tasks.workunit.client.1.smithi192.stdout:9/16: dread f6 [0,4194304] 0 2024-04-12T05:08:36.107 INFO:tasks.workunit.client.1.smithi192.stdout:9/17: write f2 [212633,96000] 0 2024-04-12T05:08:36.107 INFO:tasks.workunit.client.1.smithi192.stdout:6/30: creat d3/fd x:0 0 0 2024-04-12T05:08:36.107 INFO:tasks.workunit.client.1.smithi192.stdout:6/31: fdatasync d3/fd 0 2024-04-12T05:08:36.108 INFO:tasks.workunit.client.1.smithi192.stdout:9/18: chown f6 125964999 1 2024-04-12T05:08:36.108 INFO:tasks.workunit.client.1.smithi192.stdout:3/25: mknod d5/c8 0 2024-04-12T05:08:36.109 INFO:tasks.workunit.client.1.smithi192.stdout:3/26: write f2 [914001,77558] 0 2024-04-12T05:08:36.113 INFO:tasks.workunit.client.1.smithi192.stdout:9/19: mknod c8 0 2024-04-12T05:08:36.115 INFO:tasks.workunit.client.1.smithi192.stdout:3/27: creat d5/f9 x:0 0 0 2024-04-12T05:08:36.171 INFO:tasks.workunit.client.1.smithi192.stdout:4/24: truncate d1/f5 2434434 0 2024-04-12T05:08:36.171 INFO:tasks.workunit.client.1.smithi192.stdout:9/20: creat f9 x:0 0 0 2024-04-12T05:08:36.172 INFO:tasks.workunit.client.1.smithi192.stdout:5/34: truncate d0/f6 1028256 0 2024-04-12T05:08:36.172 INFO:tasks.workunit.client.1.smithi192.stdout:4/25: stat d1/f5 0 2024-04-12T05:08:36.172 INFO:tasks.workunit.client.1.smithi192.stdout:9/21: dread - f9 zero size 2024-04-12T05:08:36.173 INFO:tasks.workunit.client.1.smithi192.stdout:1/16: rename d0/d2/f6 to d0/f8 0 2024-04-12T05:08:36.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:34.665253+0000 mgr.smithi192.ocppyo (mgr.24357) 9 : cluster [DBG] pgmap v6: 65 pgs: 65 active+clean; 195 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:08:36.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:34.733228+0000 mon.smithi113 (mon.0) 660 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T05:08:36.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:34.733274+0000 mon.smithi113 (mon.0) 661 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-04-12T05:08:36.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:35.391419+0000 mon.smithi113 (mon.0) 662 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:36.174 INFO:tasks.workunit.client.1.smithi192.stdout:2/23: truncate d1/f2 2801705 0 2024-04-12T05:08:36.176 INFO:tasks.workunit.client.1.smithi192.stdout:8/14: chown l2 4211030 1 2024-04-12T05:08:36.177 INFO:tasks.workunit.client.1.smithi192.stdout:8/15: write f1 [794887,13706] 0 2024-04-12T05:08:36.178 INFO:tasks.workunit.client.1.smithi192.stdout:7/16: rmdir d3 39 2024-04-12T05:08:36.181 INFO:tasks.workunit.client.1.smithi192.stdout:3/28: rmdir d5 39 2024-04-12T05:08:36.182 INFO:tasks.workunit.client.1.smithi192.stdout:6/32: link d3/cb d3/ce 0 2024-04-12T05:08:36.182 INFO:tasks.workunit.client.1.smithi192.stdout:5/35: rename d0/le to d0/lf 0 2024-04-12T05:08:36.184 INFO:tasks.workunit.client.1.smithi192.stdout:8/16: symlink l4 0 2024-04-12T05:08:36.184 INFO:tasks.workunit.client.1.smithi192.stdout:9/22: dwrite f7 [0,4194304] 0 2024-04-12T05:08:36.184 INFO:tasks.workunit.client.1.smithi192.stdout:9/23: readlink - no filename 2024-04-12T05:08:36.186 INFO:tasks.workunit.client.1.smithi192.stdout:9/24: dread - f3 zero size 2024-04-12T05:08:36.187 INFO:tasks.workunit.client.1.smithi192.stdout:6/33: write d3/d5/f7 [1198878,87397] 0 2024-04-12T05:08:36.188 INFO:tasks.workunit.client.1.smithi192.stdout:1/17: link d0/f3 d0/d2/f9 0 2024-04-12T05:08:36.192 INFO:tasks.workunit.client.1.smithi192.stdout:5/36: rename d0/f6 to d0/f10 0 2024-04-12T05:08:36.195 INFO:tasks.workunit.client.1.smithi192.stdout:8/17: dwrite f3 [0,4194304] 0 2024-04-12T05:08:36.195 INFO:tasks.workunit.client.1.smithi192.stdout:4/26: truncate d1/f2 328620 0 2024-04-12T05:08:36.196 INFO:tasks.workunit.client.1.smithi192.stdout:3/29: mknod d5/ca 0 2024-04-12T05:08:36.196 INFO:tasks.workunit.client.1.smithi192.stdout:8/18: truncate f1 2085767 0 2024-04-12T05:08:36.196 INFO:tasks.workunit.client.1.smithi192.stdout:4/27: chown d1/l7 31122106 1 2024-04-12T05:08:36.197 INFO:tasks.workunit.client.1.smithi192.stdout:2/24: write d1/f2 [2844512,18066] 0 2024-04-12T05:08:36.200 INFO:tasks.workunit.client.1.smithi192.stdout:6/34: unlink d3/f9 0 2024-04-12T05:08:36.200 INFO:tasks.workunit.client.1.smithi192.stdout:6/35: write d3/fd [178361,104701] 0 2024-04-12T05:08:36.203 INFO:tasks.workunit.client.1.smithi192.stdout:1/18: rmdir d0 39 2024-04-12T05:08:36.205 INFO:tasks.workunit.client.1.smithi192.stdout:7/17: link f2 d3/f6 0 2024-04-12T05:08:36.206 INFO:tasks.workunit.client.1.smithi192.stdout:5/37: creat d0/f11 x:0 0 0 2024-04-12T05:08:36.206 INFO:tasks.workunit.client.1.smithi192.stdout:8/19: creat f5 x:0 0 0 2024-04-12T05:08:36.207 INFO:tasks.workunit.client.1.smithi192.stdout:8/20: dread - f5 zero size 2024-04-12T05:08:36.209 INFO:tasks.workunit.client.1.smithi192.stdout:8/21: write f1 [888953,5658] 0 2024-04-12T05:08:36.209 INFO:tasks.workunit.client.1.smithi192.stdout:6/36: dread d3/fd [0,4194304] 0 2024-04-12T05:08:36.209 INFO:tasks.workunit.client.1.smithi192.stdout:2/25: creat d1/f9 x:0 0 0 2024-04-12T05:08:36.213 INFO:tasks.workunit.client.1.smithi192.stdout:8/22: write f3 [1464256,86545] 0 2024-04-12T05:08:36.214 INFO:tasks.workunit.client.1.smithi192.stdout:7/18: write d3/f6 [2296003,7889] 0 2024-04-12T05:08:36.215 INFO:tasks.workunit.client.1.smithi192.stdout:1/19: dread - d0/d2/f9 zero size 2024-04-12T05:08:36.216 INFO:tasks.workunit.client.1.smithi192.stdout:8/23: chown l2 12098134 1 2024-04-12T05:08:36.217 INFO:tasks.workunit.client.1.smithi192.stdout:5/38: mknod d0/c12 0 2024-04-12T05:08:36.217 INFO:tasks.workunit.client.1.smithi192.stdout:7/19: write d3/f6 [1935255,21230] 0 2024-04-12T05:08:36.219 INFO:tasks.workunit.client.1.smithi192.stdout:4/28: unlink d1/f5 0 2024-04-12T05:08:36.219 INFO:tasks.workunit.client.1.smithi192.stdout:9/25: write f6 [719501,35474] 0 2024-04-12T05:08:36.220 INFO:tasks.workunit.client.1.smithi192.stdout:9/26: fsync f3 0 2024-04-12T05:08:36.226 INFO:tasks.workunit.client.1.smithi192.stdout:2/26: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:36.227 INFO:tasks.workunit.client.1.smithi192.stdout:8/24: mkdir d6 0 2024-04-12T05:08:36.231 INFO:tasks.workunit.client.1.smithi192.stdout:4/29: mknod d1/c9 0 2024-04-12T05:08:36.232 INFO:tasks.workunit.client.1.smithi192.stdout:9/27: creat fa x:0 0 0 2024-04-12T05:08:36.233 INFO:tasks.workunit.client.1.smithi192.stdout:6/37: link d3/fd d3/d5/ff 0 2024-04-12T05:08:36.235 INFO:tasks.workunit.client.1.smithi192.stdout:3/30: getdents d5 0 2024-04-12T05:08:36.235 INFO:tasks.workunit.client.1.smithi192.stdout:6/38: dread - d3/d5/f6 zero size 2024-04-12T05:08:36.237 INFO:tasks.workunit.client.1.smithi192.stdout:8/25: dwrite f3 [0,4194304] 0 2024-04-12T05:08:36.238 INFO:tasks.workunit.client.1.smithi192.stdout:9/28: dread f2 [0,4194304] 0 2024-04-12T05:08:36.239 INFO:tasks.workunit.client.1.smithi192.stdout:8/26: write f5 [1018774,66121] 0 2024-04-12T05:08:36.240 INFO:tasks.workunit.client.1.smithi192.stdout:4/30: mkdir d1/da 0 2024-04-12T05:08:36.240 INFO:tasks.workunit.client.1.smithi192.stdout:2/27: mknod d1/d8/ca 0 2024-04-12T05:08:36.243 INFO:tasks.workunit.client.1.smithi192.stdout:3/31: mknod d5/cb 0 2024-04-12T05:08:36.245 INFO:tasks.workunit.client.1.smithi192.stdout:6/39: fsync d3/d5/ff 0 2024-04-12T05:08:36.250 INFO:tasks.workunit.client.1.smithi192.stdout:6/40: dread d3/fd [0,4194304] 0 2024-04-12T05:08:36.252 INFO:tasks.workunit.client.1.smithi192.stdout:9/29: dwrite f3 [0,4194304] 0 2024-04-12T05:08:36.253 INFO:tasks.workunit.client.1.smithi192.stdout:2/28: mkdir d1/d8/db 0 2024-04-12T05:08:36.253 INFO:tasks.workunit.client.1.smithi192.stdout:3/32: symlink d5/lc 0 2024-04-12T05:08:36.254 INFO:tasks.workunit.client.1.smithi192.stdout:7/20: truncate f1 7710350 0 2024-04-12T05:08:36.254 INFO:tasks.workunit.client.1.smithi192.stdout:2/29: dread - d1/f9 zero size 2024-04-12T05:08:36.257 INFO:tasks.workunit.client.1.smithi192.stdout:4/31: fsync d1/f2 0 2024-04-12T05:08:36.260 INFO:tasks.workunit.client.1.smithi192.stdout:3/33: creat d5/fd x:0 0 0 2024-04-12T05:08:36.262 INFO:tasks.workunit.client.1.smithi192.stdout:3/34: dread - d5/f9 zero size 2024-04-12T05:08:36.264 INFO:tasks.workunit.client.1.smithi192.stdout:5/39: dwrite d0/f10 [0,4194304] 0 2024-04-12T05:08:36.264 INFO:tasks.workunit.client.1.smithi192.stdout:3/35: stat d5 0 2024-04-12T05:08:36.266 INFO:tasks.workunit.client.1.smithi192.stdout:2/30: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:36.268 INFO:tasks.workunit.client.1.smithi192.stdout:2/31: write d1/f4 [205049,83366] 0 2024-04-12T05:08:36.268 INFO:tasks.workunit.client.1.smithi192.stdout:6/41: mkdir d3/d8/d10 0 2024-04-12T05:08:36.271 INFO:tasks.workunit.client.1.smithi192.stdout:4/32: rename d1/c6 to d1/cb 0 2024-04-12T05:08:36.271 INFO:tasks.workunit.client.1.smithi192.stdout:7/21: fsync d3/f6 0 2024-04-12T05:08:36.272 INFO:tasks.workunit.client.1.smithi192.stdout:2/32: truncate d1/f9 814674 0 2024-04-12T05:08:36.272 INFO:tasks.workunit.client.1.smithi192.stdout:5/40: creat d0/f13 x:0 0 0 2024-04-12T05:08:36.272 INFO:tasks.workunit.client.1.smithi192.stdout:9/30: getdents . 0 2024-04-12T05:08:36.273 INFO:tasks.workunit.client.1.smithi192.stdout:9/31: chown f2 1286667 1 2024-04-12T05:08:36.275 INFO:tasks.workunit.client.1.smithi192.stdout:4/33: write d1/f2 [1124536,58778] 0 2024-04-12T05:08:36.276 INFO:tasks.workunit.client.1.smithi192.stdout:2/33: creat d1/d8/fc x:0 0 0 2024-04-12T05:08:36.278 INFO:tasks.workunit.client.1.smithi192.stdout:9/32: creat fb x:0 0 0 2024-04-12T05:08:36.279 INFO:tasks.workunit.client.1.smithi192.stdout:6/42: dread d3/d5/f7 [0,4194304] 0 2024-04-12T05:08:36.279 INFO:tasks.workunit.client.1.smithi192.stdout:5/41: dread d0/f10 [0,4194304] 0 2024-04-12T05:08:36.279 INFO:tasks.workunit.client.1.smithi192.stdout:9/33: write fb [749174,17303] 0 2024-04-12T05:08:36.280 INFO:tasks.workunit.client.1.smithi192.stdout:4/34: symlink d1/lc 0 2024-04-12T05:08:36.281 INFO:tasks.workunit.client.1.smithi192.stdout:7/22: fsync f2 0 2024-04-12T05:08:36.283 INFO:tasks.workunit.client.1.smithi192.stdout:2/34: symlink d1/d8/ld 0 2024-04-12T05:08:36.284 INFO:tasks.workunit.client.1.smithi192.stdout:5/42: creat d0/f14 x:0 0 0 2024-04-12T05:08:36.285 INFO:tasks.workunit.client.1.smithi192.stdout:2/35: dread d1/f4 [0,4194304] 0 2024-04-12T05:08:36.287 INFO:tasks.workunit.client.1.smithi192.stdout:2/36: read d1/f9 [267935,130926] 0 2024-04-12T05:08:36.287 INFO:tasks.workunit.client.1.smithi192.stdout:9/34: unlink c4 0 2024-04-12T05:08:36.287 INFO:tasks.workunit.client.1.smithi192.stdout:2/37: dread - d1/d8/fc zero size 2024-04-12T05:08:36.288 INFO:tasks.workunit.client.1.smithi192.stdout:9/35: write f6 [1490028,6813] 0 2024-04-12T05:08:36.290 INFO:tasks.workunit.client.1.smithi192.stdout:2/38: dread d1/f4 [0,4194304] 0 2024-04-12T05:08:36.290 INFO:tasks.workunit.client.1.smithi192.stdout:7/23: mkdir d3/d7 0 2024-04-12T05:08:36.290 INFO:tasks.workunit.client.1.smithi192.stdout:9/36: write fa [473569,76771] 0 2024-04-12T05:08:36.292 INFO:tasks.workunit.client.1.smithi192.stdout:5/43: mkdir d0/d15 0 2024-04-12T05:08:36.292 INFO:tasks.workunit.client.1.smithi192.stdout:7/24: chown f1 478976641 1 2024-04-12T05:08:36.294 INFO:tasks.workunit.client.1.smithi192.stdout:4/35: mknod d1/da/cd 0 2024-04-12T05:08:36.296 INFO:tasks.workunit.client.1.smithi192.stdout:9/37: creat fc x:0 0 0 2024-04-12T05:08:36.298 INFO:tasks.workunit.client.1.smithi192.stdout:9/38: write fb [1109023,67386] 0 2024-04-12T05:08:36.298 INFO:tasks.workunit.client.1.smithi192.stdout:4/36: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:36.300 INFO:tasks.workunit.client.1.smithi192.stdout:6/43: getdents d3/d5 0 2024-04-12T05:08:36.300 INFO:tasks.workunit.client.1.smithi192.stdout:6/44: readlink - no filename 2024-04-12T05:08:36.301 INFO:tasks.workunit.client.1.smithi192.stdout:2/39: mkdir d1/d8/db/de 0 2024-04-12T05:08:36.301 INFO:tasks.workunit.client.1.smithi192.stdout:6/45: chown d3/fd 1 1 2024-04-12T05:08:36.302 INFO:tasks.workunit.client.1.smithi192.stdout:2/40: dread - d1/d8/fc zero size 2024-04-12T05:08:36.303 INFO:tasks.workunit.client.1.smithi192.stdout:7/25: creat d3/d7/f8 x:0 0 0 2024-04-12T05:08:36.303 INFO:tasks.workunit.client.1.smithi192.stdout:7/26: chown d3/d7 100 1 2024-04-12T05:08:36.305 INFO:tasks.workunit.client.1.smithi192.stdout:9/39: creat fd x:0 0 0 2024-04-12T05:08:36.306 INFO:tasks.workunit.client.1.smithi192.stdout:4/37: mkdir d1/de 0 2024-04-12T05:08:36.309 INFO:tasks.workunit.client.1.smithi192.stdout:6/46: write d3/d5/f7 [950893,118882] 0 2024-04-12T05:08:36.311 INFO:tasks.workunit.client.1.smithi192.stdout:7/27: unlink f2 0 2024-04-12T05:08:36.314 INFO:tasks.workunit.client.1.smithi192.stdout:7/28: fsync d3/d7/f8 0 2024-04-12T05:08:36.318 INFO:tasks.workunit.client.1.smithi192.stdout:5/44: truncate d0/fa 3320848 0 2024-04-12T05:08:36.319 INFO:tasks.workunit.client.1.smithi192.stdout:9/40: unlink fc 0 2024-04-12T05:08:36.319 INFO:tasks.workunit.client.1.smithi192.stdout:4/38: truncate d1/f2 2055569 0 2024-04-12T05:08:36.319 INFO:tasks.workunit.client.1.smithi192.stdout:6/47: creat d3/dc/f11 x:0 0 0 2024-04-12T05:08:36.322 INFO:tasks.workunit.client.1.smithi192.stdout:9/41: creat fe x:0 0 0 2024-04-12T05:08:36.323 INFO:tasks.workunit.client.1.smithi192.stdout:2/41: getdents d1/d8/db 0 2024-04-12T05:08:36.323 INFO:tasks.workunit.client.1.smithi192.stdout:9/42: write fa [34903,18156] 0 2024-04-12T05:08:36.324 INFO:tasks.workunit.client.1.smithi192.stdout:6/48: truncate d3/fd 319002 0 2024-04-12T05:08:36.326 INFO:tasks.workunit.client.1.smithi192.stdout:5/45: creat d0/d15/f16 x:0 0 0 2024-04-12T05:08:36.327 INFO:tasks.workunit.client.1.smithi192.stdout:7/29: truncate d3/f6 829387 0 2024-04-12T05:08:36.327 INFO:tasks.workunit.client.1.smithi192.stdout:7/30: fdatasync d3/f5 0 2024-04-12T05:08:36.327 INFO:tasks.workunit.client.1.smithi192.stdout:5/46: truncate d0/d15/f16 867324 0 2024-04-12T05:08:36.328 INFO:tasks.workunit.client.1.smithi192.stdout:5/47: stat d0/d15/f16 0 2024-04-12T05:08:36.328 INFO:tasks.workunit.client.1.smithi192.stdout:6/49: mknod d3/dc/c12 0 2024-04-12T05:08:36.329 INFO:tasks.workunit.client.1.smithi192.stdout:5/48: fsync d0/f14 0 2024-04-12T05:08:36.330 INFO:tasks.workunit.client.1.smithi192.stdout:4/39: rmdir d1/d8 0 2024-04-12T05:08:36.331 INFO:tasks.workunit.client.1.smithi192.stdout:7/31: creat d3/d7/f9 x:0 0 0 2024-04-12T05:08:36.331 INFO:tasks.workunit.client.1.smithi192.stdout:5/49: symlink d0/d15/l17 0 2024-04-12T05:08:36.333 INFO:tasks.workunit.client.1.smithi192.stdout:7/32: dread - d3/f5 zero size 2024-04-12T05:08:36.334 INFO:tasks.workunit.client.1.smithi192.stdout:7/33: chown d3/l4 8050 1 2024-04-12T05:08:36.335 INFO:tasks.workunit.client.1.smithi192.stdout:5/50: truncate d0/f10 5081879 0 2024-04-12T05:08:36.335 INFO:tasks.workunit.client.1.smithi192.stdout:4/40: rename d1/c9 to d1/da/cf 0 2024-04-12T05:08:36.336 INFO:tasks.workunit.client.1.smithi192.stdout:2/42: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:36.340 INFO:tasks.workunit.client.1.smithi192.stdout:4/41: creat d1/da/f10 x:0 0 0 2024-04-12T05:08:36.343 INFO:tasks.workunit.client.1.smithi192.stdout:2/43: rename d1/f9 to d1/d8/db/ff 0 2024-04-12T05:08:36.345 INFO:tasks.workunit.client.1.smithi192.stdout:2/44: read d1/f2 [2296046,120683] 0 2024-04-12T05:08:36.347 INFO:tasks.workunit.client.1.smithi192.stdout:7/34: dwrite d3/f5 [0,4194304] 0 2024-04-12T05:08:36.348 INFO:tasks.workunit.client.1.smithi192.stdout:6/50: write d3/d5/ff [722428,39194] 0 2024-04-12T05:08:36.349 INFO:tasks.workunit.client.1.smithi192.stdout:7/35: fdatasync d3/d7/f8 0 2024-04-12T05:08:36.350 INFO:tasks.workunit.client.1.smithi192.stdout:5/51: getdents d0 0 2024-04-12T05:08:36.358 INFO:tasks.workunit.client.1.smithi192.stdout:7/36: mknod d3/ca 0 2024-04-12T05:08:36.358 INFO:tasks.workunit.client.1.smithi192.stdout:5/52: symlink d0/l18 0 2024-04-12T05:08:36.359 INFO:tasks.workunit.client.1.smithi192.stdout:6/51: creat d3/d8/f13 x:0 0 0 2024-04-12T05:08:36.360 INFO:tasks.workunit.client.1.smithi192.stdout:4/42: write d1/f2 [2788246,45845] 0 2024-04-12T05:08:36.361 INFO:tasks.workunit.client.1.smithi192.stdout:6/52: write d3/dc/f11 [943118,109867] 0 2024-04-12T05:08:36.363 INFO:tasks.workunit.client.1.smithi192.stdout:6/53: stat d3/dc/c12 0 2024-04-12T05:08:36.364 INFO:tasks.workunit.client.1.smithi192.stdout:7/37: symlink d3/lb 0 2024-04-12T05:08:36.365 INFO:tasks.workunit.client.1.smithi192.stdout:6/54: fsync d3/d5/f7 0 2024-04-12T05:08:36.365 INFO:tasks.workunit.client.1.smithi192.stdout:2/45: write d1/f4 [819187,107487] 0 2024-04-12T05:08:36.368 INFO:tasks.workunit.client.1.smithi192.stdout:4/43: rename d1/da/f10 to d1/f11 0 2024-04-12T05:08:36.368 INFO:tasks.workunit.client.1.smithi192.stdout:7/38: mknod d3/cc 0 2024-04-12T05:08:36.368 INFO:tasks.workunit.client.1.smithi192.stdout:6/55: symlink d3/l14 0 2024-04-12T05:08:36.368 INFO:tasks.workunit.client.1.smithi192.stdout:2/46: write d1/d8/fc [1042899,8559] 0 2024-04-12T05:08:36.371 INFO:tasks.workunit.client.1.smithi192.stdout:2/47: write d1/d8/fc [1491177,64236] 0 2024-04-12T05:08:36.371 INFO:tasks.workunit.client.1.smithi192.stdout:6/56: read d3/dc/f11 [701855,48557] 0 2024-04-12T05:08:36.372 INFO:tasks.workunit.client.1.smithi192.stdout:5/53: link d0/c9 d0/d15/c19 0 2024-04-12T05:08:36.375 INFO:tasks.workunit.client.1.smithi192.stdout:6/57: mknod d3/d5/c15 0 2024-04-12T05:08:36.376 INFO:tasks.workunit.client.1.smithi192.stdout:2/48: creat d1/d8/f10 x:0 0 0 2024-04-12T05:08:36.377 INFO:tasks.workunit.client.1.smithi192.stdout:5/54: rename d0/c5 to d0/d15/c1a 0 2024-04-12T05:08:36.378 INFO:tasks.workunit.client.1.smithi192.stdout:5/55: stat d0/f11 0 2024-04-12T05:08:36.381 INFO:tasks.workunit.client.1.smithi192.stdout:2/49: creat d1/d8/db/de/f11 x:0 0 0 2024-04-12T05:08:36.383 INFO:tasks.workunit.client.1.smithi192.stdout:5/56: creat d0/f1b x:0 0 0 2024-04-12T05:08:36.384 INFO:tasks.workunit.client.1.smithi192.stdout:4/44: fsync d1/f11 0 2024-04-12T05:08:36.384 INFO:tasks.workunit.client.1.smithi192.stdout:7/39: truncate f1 287237 0 2024-04-12T05:08:36.385 INFO:tasks.workunit.client.1.smithi192.stdout:6/58: dwrite d3/fd [0,4194304] 0 2024-04-12T05:08:36.388 INFO:tasks.workunit.client.1.smithi192.stdout:5/57: unlink d0/d15/f16 0 2024-04-12T05:08:36.388 INFO:tasks.workunit.client.1.smithi192.stdout:7/40: dread - d3/d7/f8 zero size 2024-04-12T05:08:36.391 INFO:tasks.workunit.client.1.smithi192.stdout:6/59: mknod d3/d8/c16 0 2024-04-12T05:08:36.392 INFO:tasks.workunit.client.1.smithi192.stdout:4/45: creat d1/de/f12 x:0 0 0 2024-04-12T05:08:36.392 INFO:tasks.workunit.client.1.smithi192.stdout:6/60: write d3/d5/f7 [1663858,103798] 0 2024-04-12T05:08:36.393 INFO:tasks.workunit.client.1.smithi192.stdout:2/50: dwrite d1/d8/db/de/f11 [0,4194304] 0 2024-04-12T05:08:36.394 INFO:tasks.workunit.client.1.smithi192.stdout:4/46: write d1/de/f12 [260156,81205] 0 2024-04-12T05:08:36.395 INFO:tasks.workunit.client.1.smithi192.stdout:6/61: write d3/d8/f13 [535903,49952] 0 2024-04-12T05:08:36.395 INFO:tasks.workunit.client.1.smithi192.stdout:6/62: chown d3/fd 10682 1 2024-04-12T05:08:36.397 INFO:tasks.workunit.client.1.smithi192.stdout:5/58: creat d0/d15/f1c x:0 0 0 2024-04-12T05:08:36.397 INFO:tasks.workunit.client.1.smithi192.stdout:4/47: truncate d1/f11 112790 0 2024-04-12T05:08:36.406 INFO:tasks.workunit.client.1.smithi192.stdout:2/51: dread d1/d8/db/ff [0,4194304] 0 2024-04-12T05:08:36.410 INFO:tasks.workunit.client.1.smithi192.stdout:5/59: dwrite d0/f1b [0,4194304] 0 2024-04-12T05:08:36.410 INFO:tasks.workunit.client.1.smithi192.stdout:7/41: getdents d3/d7 0 2024-04-12T05:08:36.412 INFO:tasks.workunit.client.1.smithi192.stdout:2/52: mknod d1/d8/c12 0 2024-04-12T05:08:36.419 INFO:tasks.workunit.client.1.smithi192.stdout:5/60: mknod d0/d15/c1d 0 2024-04-12T05:08:36.419 INFO:tasks.workunit.client.1.smithi192.stdout:7/42: rmdir d3 39 2024-04-12T05:08:36.419 INFO:tasks.workunit.client.1.smithi192.stdout:4/48: truncate d1/f2 1376132 0 2024-04-12T05:08:36.422 INFO:tasks.workunit.client.1.smithi192.stdout:4/49: readlink d1/lc 0 2024-04-12T05:08:36.425 INFO:tasks.workunit.client.1.smithi192.stdout:2/53: dread d1/d8/fc [0,4194304] 0 2024-04-12T05:08:36.426 INFO:tasks.workunit.client.1.smithi192.stdout:7/43: write d3/d7/f9 [194629,37067] 0 2024-04-12T05:08:36.433 INFO:tasks.workunit.client.1.smithi192.stdout:6/63: dwrite d3/dc/f11 [0,4194304] 0 2024-04-12T05:08:36.434 INFO:tasks.workunit.client.1.smithi192.stdout:5/61: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:08:36.447 INFO:tasks.workunit.client.1.smithi192.stdout:4/50: symlink d1/l13 0 2024-04-12T05:08:36.447 INFO:tasks.workunit.client.1.smithi192.stdout:6/64: creat d3/d8/f17 x:0 0 0 2024-04-12T05:08:36.448 INFO:tasks.workunit.client.1.smithi192.stdout:5/62: creat d0/f1e x:0 0 0 2024-04-12T05:08:36.450 INFO:tasks.workunit.client.1.smithi192.stdout:4/51: creat d1/f14 x:0 0 0 2024-04-12T05:08:36.452 INFO:tasks.workunit.client.1.smithi192.stdout:6/65: dread d3/d8/f13 [0,4194304] 0 2024-04-12T05:08:36.471 INFO:tasks.workunit.client.1.smithi192.stdout:5/63: mknod d0/c1f 0 2024-04-12T05:08:36.474 INFO:tasks.workunit.client.1.smithi192.stdout:2/54: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:36.475 INFO:tasks.workunit.client.1.smithi192.stdout:2/55: read - d1/d8/f10 zero size 2024-04-12T05:08:36.654 INFO:tasks.workunit.client.1.smithi192.stdout:6/66: mknod d3/d5/c18 0 2024-04-12T05:08:36.663 INFO:tasks.workunit.client.1.smithi192.stdout:6/67: mknod d3/d8/d10/c19 0 2024-04-12T05:08:36.665 INFO:tasks.workunit.client.1.smithi192.stdout:6/68: mknod d3/d8/c1a 0 2024-04-12T05:08:36.666 INFO:tasks.workunit.client.1.smithi192.stdout:6/69: write d3/d5/f6 [800846,46440] 0 2024-04-12T05:08:36.676 INFO:tasks.workunit.client.1.smithi192.stdout:6/70: dwrite d3/d8/f17 [0,4194304] 0 2024-04-12T05:08:36.677 INFO:tasks.workunit.client.1.smithi192.stdout:6/71: write d3/d8/f17 [711025,9524] 0 2024-04-12T05:08:36.680 INFO:tasks.workunit.client.1.smithi192.stdout:6/72: stat d3/d5/c15 0 2024-04-12T05:08:36.691 INFO:tasks.workunit.client.1.smithi192.stdout:6/73: dwrite d3/d5/f7 [0,4194304] 0 2024-04-12T05:08:36.692 INFO:tasks.workunit.client.1.smithi192.stdout:6/74: fdatasync d3/d8/f17 0 2024-04-12T05:08:36.720 INFO:tasks.workunit.client.1.smithi192.stdout:5/64: truncate d0/f10 3557428 0 2024-04-12T05:08:36.722 INFO:tasks.workunit.client.1.smithi192.stdout:4/52: truncate d1/f2 1680731 0 2024-04-12T05:08:36.726 INFO:tasks.workunit.client.1.smithi192.stdout:6/75: write d3/fd [4525548,46656] 0 2024-04-12T05:08:36.726 INFO:tasks.workunit.client.1.smithi192.stdout:6/76: fsync d3/dc/f11 0 2024-04-12T05:08:36.728 INFO:tasks.workunit.client.1.smithi192.stdout:2/56: dwrite d1/d8/db/ff [0,4194304] 0 2024-04-12T05:08:36.728 INFO:tasks.workunit.client.1.smithi192.stdout:6/77: fsync d3/fd 0 2024-04-12T05:08:36.731 INFO:tasks.workunit.client.1.smithi192.stdout:5/65: getdents d0/d15 0 2024-04-12T05:08:36.734 INFO:tasks.workunit.client.1.smithi192.stdout:5/66: write d0/f13 [383122,78792] 0 2024-04-12T05:08:36.739 INFO:tasks.workunit.client.1.smithi192.stdout:6/78: symlink d3/l1b 0 2024-04-12T05:08:36.739 INFO:tasks.workunit.client.1.smithi192.stdout:6/79: chown d3/dc/f11 0 1 2024-04-12T05:08:36.739 INFO:tasks.workunit.client.1.smithi192.stdout:5/67: creat d0/f20 x:0 0 0 2024-04-12T05:08:36.739 INFO:tasks.workunit.client.1.smithi192.stdout:2/57: dread d1/d8/db/de/f11 [0,4194304] 0 2024-04-12T05:08:36.740 INFO:tasks.workunit.client.1.smithi192.stdout:6/80: creat d3/d8/f1c x:0 0 0 2024-04-12T05:08:36.740 INFO:tasks.workunit.client.1.smithi192.stdout:2/58: fsync d1/f4 0 2024-04-12T05:08:36.741 INFO:tasks.workunit.client.1.smithi192.stdout:5/68: mkdir d0/d21 0 2024-04-12T05:08:36.743 INFO:tasks.workunit.client.1.smithi192.stdout:2/59: creat d1/f13 x:0 0 0 2024-04-12T05:08:36.743 INFO:tasks.workunit.client.1.smithi192.stdout:2/60: chown d1/l7 0 1 2024-04-12T05:08:36.743 INFO:tasks.workunit.client.1.smithi192.stdout:6/81: mknod d3/d8/d10/c1d 0 2024-04-12T05:08:36.745 INFO:tasks.workunit.client.1.smithi192.stdout:5/69: symlink d0/d15/l22 0 2024-04-12T05:08:36.747 INFO:tasks.workunit.client.1.smithi192.stdout:5/70: dread - d0/f1e zero size 2024-04-12T05:08:36.749 INFO:tasks.workunit.client.1.smithi192.stdout:2/61: write d1/d8/db/ff [686276,83793] 0 2024-04-12T05:08:36.751 INFO:tasks.workunit.client.1.smithi192.stdout:2/62: readlink d1/l7 0 2024-04-12T05:08:36.755 INFO:tasks.workunit.client.1.smithi192.stdout:5/71: creat d0/d15/f23 x:0 0 0 2024-04-12T05:08:36.756 INFO:tasks.workunit.client.1.smithi192.stdout:6/82: dwrite d3/d8/f1c [0,4194304] 0 2024-04-12T05:08:36.757 INFO:tasks.workunit.client.1.smithi192.stdout:6/83: chown d3/l1b 345 1 2024-04-12T05:08:36.758 INFO:tasks.workunit.client.1.smithi192.stdout:5/72: write d0/f14 [398062,111828] 0 2024-04-12T05:08:36.758 INFO:tasks.workunit.client.1.smithi192.stdout:5/73: fsync d0/f7 0 2024-04-12T05:08:36.759 INFO:tasks.workunit.client.1.smithi192.stdout:5/74: stat d0/f1b 0 2024-04-12T05:08:36.760 INFO:tasks.workunit.client.1.smithi192.stdout:5/75: dread - d0/f20 zero size 2024-04-12T05:08:36.762 INFO:tasks.workunit.client.1.smithi192.stdout:5/76: write d0/f1e [635469,8585] 0 2024-04-12T05:08:36.765 INFO:tasks.workunit.client.1.smithi192.stdout:5/77: mkdir d0/d15/d24 0 2024-04-12T05:08:36.765 INFO:tasks.workunit.client.1.smithi192.stdout:5/78: fdatasync d0/f11 0 2024-04-12T05:08:36.768 INFO:tasks.workunit.client.1.smithi192.stdout:5/79: creat d0/d15/d24/f25 x:0 0 0 2024-04-12T05:08:36.771 INFO:tasks.workunit.client.1.smithi192.stdout:5/80: creat d0/d21/f26 x:0 0 0 2024-04-12T05:08:36.772 INFO:tasks.workunit.client.1.smithi192.stdout:5/81: truncate d0/f13 1499825 0 2024-04-12T05:08:36.773 INFO:tasks.workunit.client.1.smithi192.stdout:5/82: chown d0/f14 0 1 2024-04-12T05:08:36.774 INFO:tasks.workunit.client.1.smithi192.stdout:5/83: truncate d0/d21/f26 525959 0 2024-04-12T05:08:36.774 INFO:tasks.workunit.client.1.smithi192.stdout:5/84: stat d0/d15/l17 0 2024-04-12T05:08:36.775 INFO:tasks.workunit.client.1.smithi192.stdout:5/85: chown d0/d15/c1d 264 1 2024-04-12T05:08:36.777 INFO:tasks.workunit.client.1.smithi192.stdout:5/86: symlink d0/d15/d24/l27 0 2024-04-12T05:08:36.783 INFO:tasks.workunit.client.1.smithi192.stdout:5/87: getdents d0/d15/d24 0 2024-04-12T05:08:36.785 INFO:tasks.workunit.client.1.smithi192.stdout:5/88: symlink d0/l28 0 2024-04-12T05:08:36.792 INFO:tasks.workunit.client.1.smithi192.stdout:5/89: dwrite d0/f1e [0,4194304] 0 2024-04-12T05:08:36.802 INFO:tasks.workunit.client.1.smithi192.stdout:5/90: dwrite d0/d15/d24/f25 [0,4194304] 0 2024-04-12T05:08:36.803 INFO:tasks.workunit.client.1.smithi192.stdout:5/91: readlink d0/l28 0 2024-04-12T05:08:36.803 INFO:tasks.workunit.client.1.smithi192.stdout:5/92: fsync d0/f14 0 2024-04-12T05:08:36.808 INFO:tasks.workunit.client.1.smithi192.stdout:5/93: dread d0/f1e [0,4194304] 0 2024-04-12T05:08:36.808 INFO:tasks.workunit.client.1.smithi192.stdout:5/94: dread - d0/f11 zero size 2024-04-12T05:08:36.810 INFO:tasks.workunit.client.1.smithi192.stdout:5/95: dread - d0/f11 zero size 2024-04-12T05:08:36.812 INFO:tasks.workunit.client.1.smithi192.stdout:5/96: mkdir d0/d15/d24/d29 0 2024-04-12T05:08:36.813 INFO:tasks.workunit.client.1.smithi192.stdout:5/97: readlink d0/d15/l22 0 2024-04-12T05:08:36.816 INFO:tasks.workunit.client.1.smithi192.stdout:5/98: creat d0/d15/d24/d29/f2a x:0 0 0 2024-04-12T05:08:36.819 INFO:tasks.workunit.client.1.smithi192.stdout:5/99: creat d0/d15/d24/f2b x:0 0 0 2024-04-12T05:08:36.823 INFO:tasks.workunit.client.1.smithi192.stdout:5/100: link d0/c12 d0/c2c 0 2024-04-12T05:08:36.825 INFO:tasks.workunit.client.1.smithi192.stdout:5/101: write d0/f1e [1555487,101560] 0 2024-04-12T05:08:36.828 INFO:tasks.workunit.client.1.smithi192.stdout:5/102: dread d0/fa [0,4194304] 0 2024-04-12T05:08:36.835 INFO:tasks.workunit.client.1.smithi192.stdout:5/103: dwrite d0/f1b [0,4194304] 0 2024-04-12T05:08:36.838 INFO:tasks.workunit.client.1.smithi192.stdout:5/104: truncate d0/f11 539101 0 2024-04-12T05:08:36.846 INFO:tasks.workunit.client.1.smithi192.stdout:5/105: link d0/l28 d0/d15/d24/d29/l2d 0 2024-04-12T05:08:36.853 INFO:tasks.workunit.client.1.smithi192.stdout:5/106: truncate d0/d15/d24/f25 581893 0 2024-04-12T05:08:36.859 INFO:tasks.workunit.client.1.smithi192.stdout:5/107: write d0/d15/d24/f25 [336611,6466] 0 2024-04-12T05:08:36.864 INFO:tasks.workunit.client.1.smithi192.stdout:5/108: chown d0/d15/d24/d29/f2a 1799747065 1 2024-04-12T05:08:36.865 INFO:tasks.workunit.client.1.smithi192.stdout:5/109: write d0/d15/d24/f2b [878709,45935] 0 2024-04-12T05:08:36.867 INFO:tasks.workunit.client.1.smithi192.stdout:5/110: dread d0/f11 [0,4194304] 0 2024-04-12T05:08:36.869 INFO:tasks.workunit.client.1.smithi192.stdout:5/111: symlink d0/d15/d24/l2e 0 2024-04-12T05:08:36.871 INFO:tasks.workunit.client.1.smithi192.stdout:5/112: creat d0/d21/f2f x:0 0 0 2024-04-12T05:08:36.877 INFO:tasks.workunit.client.1.smithi192.stdout:5/113: write d0/f10 [1533265,5400] 0 2024-04-12T05:08:36.880 INFO:tasks.workunit.client.1.smithi192.stdout:5/114: creat d0/d15/d24/d29/f30 x:0 0 0 2024-04-12T05:08:36.884 INFO:tasks.workunit.client.1.smithi192.stdout:5/115: getdents d0/d15/d24 0 2024-04-12T05:08:36.886 INFO:tasks.workunit.client.1.smithi192.stdout:5/116: symlink d0/d15/d24/l31 0 2024-04-12T05:08:36.888 INFO:tasks.workunit.client.1.smithi192.stdout:5/117: mkdir d0/d15/d32 0 2024-04-12T05:08:36.888 INFO:tasks.workunit.client.1.smithi192.stdout:5/118: chown d0/d15/f1c 3600 1 2024-04-12T05:08:36.889 INFO:tasks.workunit.client.1.smithi192.stdout:5/119: chown d0/fa 27 1 2024-04-12T05:08:36.892 INFO:tasks.workunit.client.1.smithi192.stdout:5/120: rename d0/d15/f1c to d0/d21/f33 0 2024-04-12T05:08:36.893 INFO:tasks.workunit.client.1.smithi192.stdout:5/121: symlink d0/d15/d32/l34 0 2024-04-12T05:08:36.901 INFO:tasks.workunit.client.1.smithi192.stdout:5/122: write d0/f11 [1360574,30442] 0 2024-04-12T05:08:36.901 INFO:tasks.workunit.client.1.smithi192.stdout:5/123: dread - d0/f20 zero size 2024-04-12T05:08:36.902 INFO:tasks.workunit.client.1.smithi192.stdout:5/124: stat d0/d15/d24/d29/f2a 0 2024-04-12T05:08:36.903 INFO:tasks.workunit.client.1.smithi192.stdout:5/125: fdatasync d0/f20 0 2024-04-12T05:08:36.906 INFO:tasks.workunit.client.1.smithi192.stdout:5/126: getdents d0/d15 0 2024-04-12T05:08:36.908 INFO:tasks.workunit.client.1.smithi192.stdout:5/127: rmdir d0/d15/d24 39 2024-04-12T05:08:36.908 INFO:tasks.workunit.client.1.smithi192.stdout:5/128: fdatasync d0/f10 0 2024-04-12T05:08:36.909 INFO:tasks.workunit.client.1.smithi192.stdout:5/129: truncate d0/d21/f33 1021131 0 2024-04-12T05:08:36.913 INFO:tasks.workunit.client.1.smithi192.stdout:5/130: dread d0/f13 [0,4194304] 0 2024-04-12T05:08:36.917 INFO:tasks.workunit.client.1.smithi192.stdout:5/131: rename d0/d15/d24/f2b to d0/d21/f35 0 2024-04-12T05:08:36.918 INFO:tasks.workunit.client.1.smithi192.stdout:5/132: chown d0/d21/f2f 138631 1 2024-04-12T05:08:36.925 INFO:tasks.workunit.client.1.smithi192.stdout:5/133: dwrite d0/d15/f23 [0,4194304] 0 2024-04-12T05:08:36.926 INFO:tasks.workunit.client.1.smithi192.stdout:5/134: readlink d0/d15/l17 0 2024-04-12T05:08:37.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:36.665632+0000 mgr.smithi192.ocppyo (mgr.24357) 10 : cluster [DBG] pgmap v8: 65 pgs: 65 active+clean; 195 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:08:37.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:36.750021+0000 mon.smithi113 (mon.0) 663 : cluster [DBG] mgrmap e24: smithi192.ocppyo(active, since 6s), standbys: smithi113.ctsazg 2024-04-12T05:08:37.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:37.169733+0000 mon.smithi113 (mon.0) 664 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:36.665632+0000 mgr.smithi192.ocppyo (mgr.24357) 10 : cluster [DBG] pgmap v8: 65 pgs: 65 active+clean; 195 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:08:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:36.750021+0000 mon.smithi113 (mon.0) 663 : cluster [DBG] mgrmap e24: smithi192.ocppyo(active, since 6s), standbys: smithi113.ctsazg 2024-04-12T05:08:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:37.169733+0000 mon.smithi113 (mon.0) 664 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:39.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:38.261581+0000 mon.smithi113 (mon.0) 665 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T05:08:39.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:38.261765+0000 mon.smithi192 (mon.1) 42 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:08:39.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:38.261971+0000 mon.smithi192 (mon.1) 43 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:08:39.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:38.795508+0000 mon.smithi113 (mon.0) 666 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T05:08:39.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:38.261581+0000 mon.smithi113 (mon.0) 665 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T05:08:39.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:38.261765+0000 mon.smithi192 (mon.1) 42 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:08:39.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:38.261971+0000 mon.smithi192 (mon.1) 43 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:08:39.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:38.795508+0000 mon.smithi113 (mon.0) 666 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T05:08:40.429 INFO:tasks.workunit.client.0.smithi113.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-12T05:08:40.437 INFO:tasks.workunit.client.0.smithi113.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-12T05:08:40.437 INFO:tasks.workunit.client.0.smithi113.stderr:+ make 2024-04-12T05:08:40.642 INFO:tasks.workunit.client.0.smithi113.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-12T05:08:40.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:38.666290+0000 mgr.smithi192.ocppyo (mgr.24357) 11 : cluster [DBG] pgmap v9: 65 pgs: 65 active+clean; 283 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 15 MiB/s wr, 227 op/s 2024-04-12T05:08:40.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:38.797655+0000 mon.smithi192 (mon.1) 44 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T05:08:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:39.338256+0000 mon.smithi192 (mon.1) 45 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T05:08:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:39.338861+0000 mon.smithi192 (mon.1) 46 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:08:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:39.338984+0000 mon.smithi192 (mon.1) 47 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:08:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:40.165994+0000 mon.smithi113 (mon.0) 667 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:40.171101+0000 mon.smithi113 (mon.0) 668 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:38.666290+0000 mgr.smithi192.ocppyo (mgr.24357) 11 : cluster [DBG] pgmap v9: 65 pgs: 65 active+clean; 283 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 15 MiB/s wr, 227 op/s 2024-04-12T05:08:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:38.797655+0000 mon.smithi192 (mon.1) 44 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T05:08:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:39.338256+0000 mon.smithi192 (mon.1) 45 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T05:08:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:39.338861+0000 mon.smithi192 (mon.1) 46 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:08:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:39.338984+0000 mon.smithi192 (mon.1) 47 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:08:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:40.165994+0000 mon.smithi113 (mon.0) 667 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:40.171101+0000 mon.smithi113 (mon.0) 668 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:40.993 INFO:tasks.workunit.client.0.smithi113.stderr:++ readlink -f fsstress 2024-04-12T05:08:40.995 INFO:tasks.workunit.client.0.smithi113.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-12T05:08:40.995 INFO:tasks.workunit.client.0.smithi113.stderr:+ popd 2024-04-12T05:08:40.996 INFO:tasks.workunit.client.0.smithi113.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-12T05:08:40.996 INFO:tasks.workunit.client.0.smithi113.stderr:+ popd 2024-04-12T05:08:40.997 INFO:tasks.workunit.client.0.smithi113.stdout:~/cephtest/mnt.0/client.0/tmp 2024-04-12T05:08:40.997 INFO:tasks.workunit.client.0.smithi113.stderr:++ mktemp -d -p . 2024-04-12T05:08:41.000 INFO:tasks.workunit.client.0.smithi113.stderr:+ T=./tmp.blyVfylW1H 2024-04-12T05:08:41.000 INFO:tasks.workunit.client.0.smithi113.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.blyVfylW1H -l 1 -n 1000 -p 10 -v 2024-04-12T05:08:41.003 INFO:tasks.workunit.client.0.smithi113.stdout:seed = 1712901516 2024-04-12T05:08:41.009 INFO:tasks.workunit.client.0.smithi113.stdout:0/0: creat f0 x:0 0 0 2024-04-12T05:08:41.010 INFO:tasks.workunit.client.0.smithi113.stdout:0/1: chown f0 0 1 2024-04-12T05:08:41.014 INFO:tasks.workunit.client.0.smithi113.stdout:0/2: read - f0 zero size 2024-04-12T05:08:41.014 INFO:tasks.workunit.client.0.smithi113.stdout:1/0: mknod c0 0 2024-04-12T05:08:41.014 INFO:tasks.workunit.client.0.smithi113.stdout:1/1: chown c0 12999 1 2024-04-12T05:08:41.015 INFO:tasks.workunit.client.0.smithi113.stdout:1/2: truncate - no filename 2024-04-12T05:08:41.015 INFO:tasks.workunit.client.0.smithi113.stdout:0/3: mkdir d1 0 2024-04-12T05:08:41.015 INFO:tasks.workunit.client.0.smithi113.stdout:2/0: getdents . 0 2024-04-12T05:08:41.015 INFO:tasks.workunit.client.0.smithi113.stdout:2/1: write - no filename 2024-04-12T05:08:41.015 INFO:tasks.workunit.client.0.smithi113.stdout:2/2: fdatasync - no filename 2024-04-12T05:08:41.015 INFO:tasks.workunit.client.0.smithi113.stdout:2/3: write - no filename 2024-04-12T05:08:41.015 INFO:tasks.workunit.client.0.smithi113.stdout:0/4: truncate f0 1029421 0 2024-04-12T05:08:41.015 INFO:tasks.workunit.client.0.smithi113.stdout:4/0: unlink - no file 2024-04-12T05:08:41.015 INFO:tasks.workunit.client.0.smithi113.stdout:1/3: creat f1 x:0 0 0 2024-04-12T05:08:41.015 INFO:tasks.workunit.client.0.smithi113.stdout:1/4: dread - f1 zero size 2024-04-12T05:08:41.016 INFO:tasks.workunit.client.0.smithi113.stdout:5/0: dread - no filename 2024-04-12T05:08:41.016 INFO:tasks.workunit.client.0.smithi113.stdout:2/4: mknod c0 0 2024-04-12T05:08:41.016 INFO:tasks.workunit.client.0.smithi113.stdout:2/5: write - no filename 2024-04-12T05:08:41.016 INFO:tasks.workunit.client.0.smithi113.stdout:2/6: write - no filename 2024-04-12T05:08:41.016 INFO:tasks.workunit.client.0.smithi113.stdout:2/7: truncate - no filename 2024-04-12T05:08:41.016 INFO:tasks.workunit.client.0.smithi113.stdout:2/8: dread - no filename 2024-04-12T05:08:41.016 INFO:tasks.workunit.client.0.smithi113.stdout:2/9: readlink - no filename 2024-04-12T05:08:41.016 INFO:tasks.workunit.client.0.smithi113.stdout:2/10: read - no filename 2024-04-12T05:08:41.016 INFO:tasks.workunit.client.0.smithi113.stdout:4/1: getdents . 0 2024-04-12T05:08:41.018 INFO:tasks.workunit.client.0.smithi113.stdout:2/11: creat f1 x:0 0 0 2024-04-12T05:08:41.020 INFO:tasks.workunit.client.0.smithi113.stdout:5/1: mkdir d0 0 2024-04-12T05:08:41.020 INFO:tasks.workunit.client.0.smithi113.stdout:7/0: dwrite - no filename 2024-04-12T05:08:41.021 INFO:tasks.workunit.client.0.smithi113.stdout:4/2: symlink l0 0 2024-04-12T05:08:41.021 INFO:tasks.workunit.client.0.smithi113.stdout:5/2: stat d0 0 2024-04-12T05:08:41.021 INFO:tasks.workunit.client.0.smithi113.stdout:5/3: dread - no filename 2024-04-12T05:08:41.023 INFO:tasks.workunit.client.0.smithi113.stdout:6/0: getdents . 0 2024-04-12T05:08:41.023 INFO:tasks.workunit.client.0.smithi113.stdout:6/1: rename - no filename 2024-04-12T05:08:41.024 INFO:tasks.workunit.client.0.smithi113.stdout:8/0: truncate - no filename 2024-04-12T05:08:41.025 INFO:tasks.workunit.client.0.smithi113.stdout:8/1: chown . 1002535 1 2024-04-12T05:08:41.025 INFO:tasks.workunit.client.0.smithi113.stdout:8/2: dwrite - no filename 2024-04-12T05:08:41.025 INFO:tasks.workunit.client.0.smithi113.stdout:8/3: dread - no filename 2024-04-12T05:08:41.025 INFO:tasks.workunit.client.0.smithi113.stdout:4/3: mknod c1 0 2024-04-12T05:08:41.025 INFO:tasks.workunit.client.0.smithi113.stdout:4/4: read - no filename 2024-04-12T05:08:41.025 INFO:tasks.workunit.client.0.smithi113.stdout:4/5: fsync - no filename 2024-04-12T05:08:41.025 INFO:tasks.workunit.client.0.smithi113.stdout:4/6: dwrite - no filename 2024-04-12T05:08:41.025 INFO:tasks.workunit.client.0.smithi113.stdout:4/7: dread - no filename 2024-04-12T05:08:41.026 INFO:tasks.workunit.client.0.smithi113.stdout:8/4: chown . 1 1 2024-04-12T05:08:41.026 INFO:tasks.workunit.client.0.smithi113.stdout:8/5: chown . 780 1 2024-04-12T05:08:41.026 INFO:tasks.workunit.client.0.smithi113.stdout:8/6: truncate - no filename 2024-04-12T05:08:41.026 INFO:tasks.workunit.client.0.smithi113.stdout:8/7: dwrite - no filename 2024-04-12T05:08:41.026 INFO:tasks.workunit.client.0.smithi113.stdout:8/8: dread - no filename 2024-04-12T05:08:41.027 INFO:tasks.workunit.client.0.smithi113.stdout:8/9: chown . 1 1 2024-04-12T05:08:41.028 INFO:tasks.workunit.client.0.smithi113.stdout:7/1: mkdir d0 0 2024-04-12T05:08:41.028 INFO:tasks.workunit.client.0.smithi113.stdout:9/0: dread - no filename 2024-04-12T05:08:41.030 INFO:tasks.workunit.client.0.smithi113.stdout:0/5: dwrite f0 [0,4194304] 0 2024-04-12T05:08:41.031 INFO:tasks.workunit.client.0.smithi113.stdout:4/8: creat f2 x:0 0 0 2024-04-12T05:08:41.031 INFO:tasks.workunit.client.0.smithi113.stdout:1/5: dwrite f1 [0,4194304] 0 2024-04-12T05:08:41.033 INFO:tasks.workunit.client.0.smithi113.stdout:5/4: creat d0/f1 x:0 0 0 2024-04-12T05:08:41.033 INFO:tasks.workunit.client.0.smithi113.stdout:6/2: mknod c0 0 2024-04-12T05:08:41.034 INFO:tasks.workunit.client.0.smithi113.stdout:5/5: chown d0/f1 919 1 2024-04-12T05:08:41.035 INFO:tasks.workunit.client.0.smithi113.stdout:5/6: rename d0 to d0/d2 22 2024-04-12T05:08:41.035 INFO:tasks.workunit.client.0.smithi113.stdout:8/10: mknod c0 0 2024-04-12T05:08:41.036 INFO:tasks.workunit.client.0.smithi113.stdout:1/6: dread f1 [0,4194304] 0 2024-04-12T05:08:41.037 INFO:tasks.workunit.client.0.smithi113.stdout:4/9: mknod c3 0 2024-04-12T05:08:41.038 INFO:tasks.workunit.client.0.smithi113.stdout:4/10: chown l0 1620 1 2024-04-12T05:08:41.041 INFO:tasks.workunit.client.0.smithi113.stdout:9/1: mknod c0 0 2024-04-12T05:08:41.042 INFO:tasks.workunit.client.0.smithi113.stdout:6/3: symlink l1 0 2024-04-12T05:08:41.043 INFO:tasks.workunit.client.0.smithi113.stdout:7/2: mkdir d0/d1 0 2024-04-12T05:08:41.045 INFO:tasks.workunit.client.0.smithi113.stdout:8/11: mknod c1 0 2024-04-12T05:08:41.045 INFO:tasks.workunit.client.0.smithi113.stdout:8/12: fdatasync - no filename 2024-04-12T05:08:41.045 INFO:tasks.workunit.client.0.smithi113.stdout:8/13: readlink - no filename 2024-04-12T05:08:41.047 INFO:tasks.workunit.client.0.smithi113.stdout:0/6: creat d1/f2 x:0 0 0 2024-04-12T05:08:41.048 INFO:tasks.workunit.client.0.smithi113.stdout:1/7: creat f2 x:0 0 0 2024-04-12T05:08:41.049 INFO:tasks.workunit.client.0.smithi113.stdout:1/8: chown f2 81 1 2024-04-12T05:08:41.050 INFO:tasks.workunit.client.0.smithi113.stdout:1/9: write f2 [317728,126552] 0 2024-04-12T05:08:41.052 INFO:tasks.workunit.client.0.smithi113.stdout:9/2: mkdir d1 0 2024-04-12T05:08:41.052 INFO:tasks.workunit.client.0.smithi113.stdout:9/3: rename d1 to d1/d2 22 2024-04-12T05:08:41.052 INFO:tasks.workunit.client.0.smithi113.stdout:9/4: write - no filename 2024-04-12T05:08:41.053 INFO:tasks.workunit.client.0.smithi113.stdout:6/4: symlink l2 0 2024-04-12T05:08:41.053 INFO:tasks.workunit.client.0.smithi113.stdout:6/5: readlink l2 0 2024-04-12T05:08:41.054 INFO:tasks.workunit.client.0.smithi113.stdout:6/6: truncate - no filename 2024-04-12T05:08:41.054 INFO:tasks.workunit.client.0.smithi113.stdout:6/7: rmdir - no directory 2024-04-12T05:08:41.054 INFO:tasks.workunit.client.0.smithi113.stdout:6/8: write - no filename 2024-04-12T05:08:41.054 INFO:tasks.workunit.client.0.smithi113.stdout:6/9: write - no filename 2024-04-12T05:08:41.054 INFO:tasks.workunit.client.0.smithi113.stdout:6/10: dwrite - no filename 2024-04-12T05:08:41.054 INFO:tasks.workunit.client.0.smithi113.stdout:6/11: chown l2 221799999 1 2024-04-12T05:08:41.055 INFO:tasks.workunit.client.0.smithi113.stdout:7/3: rename d0/d1 to d0/d2 0 2024-04-12T05:08:41.056 INFO:tasks.workunit.client.0.smithi113.stdout:8/14: creat f2 x:0 0 0 2024-04-12T05:08:41.057 INFO:tasks.workunit.client.0.smithi113.stdout:8/15: truncate f2 93404 0 2024-04-12T05:08:41.059 INFO:tasks.workunit.client.0.smithi113.stdout:1/10: rename c0 to c3 0 2024-04-12T05:08:41.062 INFO:tasks.workunit.client.0.smithi113.stdout:6/12: symlink l3 0 2024-04-12T05:08:41.062 INFO:tasks.workunit.client.0.smithi113.stdout:6/13: write - no filename 2024-04-12T05:08:41.062 INFO:tasks.workunit.client.0.smithi113.stdout:6/14: write - no filename 2024-04-12T05:08:41.063 INFO:tasks.workunit.client.0.smithi113.stdout:7/4: symlink d0/d2/l3 0 2024-04-12T05:08:41.063 INFO:tasks.workunit.client.0.smithi113.stdout:7/5: dread - no filename 2024-04-12T05:08:41.063 INFO:tasks.workunit.client.0.smithi113.stdout:7/6: fdatasync - no filename 2024-04-12T05:08:41.063 INFO:tasks.workunit.client.0.smithi113.stdout:0/7: fsync f0 0 2024-04-12T05:08:41.064 INFO:tasks.workunit.client.0.smithi113.stdout:5/7: fsync d0/f1 0 2024-04-12T05:08:41.064 INFO:tasks.workunit.client.0.smithi113.stdout:4/11: link l0 l4 0 2024-04-12T05:08:41.065 INFO:tasks.workunit.client.0.smithi113.stdout:9/5: mkdir d1/d3 0 2024-04-12T05:08:41.067 INFO:tasks.workunit.client.0.smithi113.stdout:8/16: dwrite f2 [0,4194304] 0 2024-04-12T05:08:41.067 INFO:tasks.workunit.client.0.smithi113.stdout:4/12: write f2 [724438,68971] 0 2024-04-12T05:08:41.067 INFO:tasks.workunit.client.0.smithi113.stdout:7/7: chown d0/d2/l3 66509 1 2024-04-12T05:08:41.067 INFO:tasks.workunit.client.0.smithi113.stdout:6/15: symlink l4 0 2024-04-12T05:08:41.069 INFO:tasks.workunit.client.0.smithi113.stdout:0/8: rename f0 to d1/f3 0 2024-04-12T05:08:41.070 INFO:tasks.workunit.client.0.smithi113.stdout:5/8: creat d0/f3 x:0 0 0 2024-04-12T05:08:41.071 INFO:tasks.workunit.client.0.smithi113.stdout:5/9: dread - d0/f3 zero size 2024-04-12T05:08:41.072 INFO:tasks.workunit.client.0.smithi113.stdout:8/17: symlink l3 0 2024-04-12T05:08:41.072 INFO:tasks.workunit.client.0.smithi113.stdout:0/9: fsync d1/f2 0 2024-04-12T05:08:41.073 INFO:tasks.workunit.client.0.smithi113.stdout:4/13: mknod c5 0 2024-04-12T05:08:41.074 INFO:tasks.workunit.client.0.smithi113.stdout:4/14: fsync f2 0 2024-04-12T05:08:41.074 INFO:tasks.workunit.client.0.smithi113.stdout:6/16: creat f5 x:0 0 0 2024-04-12T05:08:41.077 INFO:tasks.workunit.client.0.smithi113.stdout:9/6: rename d1/d3 to d1/d4 0 2024-04-12T05:08:41.077 INFO:tasks.workunit.client.0.smithi113.stdout:9/7: dwrite - no filename 2024-04-12T05:08:41.077 INFO:tasks.workunit.client.0.smithi113.stdout:9/8: truncate - no filename 2024-04-12T05:08:41.077 INFO:tasks.workunit.client.0.smithi113.stdout:9/9: read - no filename 2024-04-12T05:08:41.078 INFO:tasks.workunit.client.0.smithi113.stdout:8/18: dread f2 [0,4194304] 0 2024-04-12T05:08:41.080 INFO:tasks.workunit.client.0.smithi113.stdout:5/10: rename d0/f3 to d0/f4 0 2024-04-12T05:08:41.081 INFO:tasks.workunit.client.0.smithi113.stdout:4/15: dread f2 [0,4194304] 0 2024-04-12T05:08:41.084 INFO:tasks.workunit.client.0.smithi113.stdout:1/11: stat c3 0 2024-04-12T05:08:41.084 INFO:tasks.workunit.client.0.smithi113.stdout:1/12: rmdir - no directory 2024-04-12T05:08:41.088 INFO:tasks.workunit.client.0.smithi113.stdout:9/10: mkdir d1/d4/d5 0 2024-04-12T05:08:41.088 INFO:tasks.workunit.client.0.smithi113.stdout:9/11: dread - no filename 2024-04-12T05:08:41.088 INFO:tasks.workunit.client.0.smithi113.stdout:9/12: write - no filename 2024-04-12T05:08:41.088 INFO:tasks.workunit.client.0.smithi113.stdout:6/17: creat f6 x:0 0 0 2024-04-12T05:08:41.089 INFO:tasks.workunit.client.0.smithi113.stdout:5/11: creat d0/f5 x:0 0 0 2024-04-12T05:08:41.092 INFO:tasks.workunit.client.0.smithi113.stdout:4/16: unlink c3 0 2024-04-12T05:08:41.092 INFO:tasks.workunit.client.0.smithi113.stdout:8/19: fsync f2 0 2024-04-12T05:08:41.098 INFO:tasks.workunit.client.0.smithi113.stdout:5/12: rename d0/f1 to d0/f6 0 2024-04-12T05:08:41.099 INFO:tasks.workunit.client.0.smithi113.stdout:4/17: chown l4 19878 1 2024-04-12T05:08:41.100 INFO:tasks.workunit.client.0.smithi113.stdout:7/8: getdents d0/d2 0 2024-04-12T05:08:41.100 INFO:tasks.workunit.client.0.smithi113.stdout:7/9: fsync - no filename 2024-04-12T05:08:41.100 INFO:tasks.workunit.client.0.smithi113.stdout:7/10: fsync - no filename 2024-04-12T05:08:41.103 INFO:tasks.workunit.client.0.smithi113.stdout:6/18: rename f5 to f7 0 2024-04-12T05:08:41.104 INFO:tasks.workunit.client.0.smithi113.stdout:0/10: truncate d1/f3 1189950 0 2024-04-12T05:08:41.104 INFO:tasks.workunit.client.0.smithi113.stdout:9/13: mkdir d1/d4/d5/d6 0 2024-04-12T05:08:41.105 INFO:tasks.workunit.client.0.smithi113.stdout:5/13: mknod d0/c7 0 2024-04-12T05:08:41.106 INFO:tasks.workunit.client.0.smithi113.stdout:4/18: mknod c6 0 2024-04-12T05:08:41.106 INFO:tasks.workunit.client.0.smithi113.stdout:6/19: mkdir d8 0 2024-04-12T05:08:41.107 INFO:tasks.workunit.client.0.smithi113.stdout:8/20: dwrite f2 [4194304,4194304] 0 2024-04-12T05:08:41.107 INFO:tasks.workunit.client.0.smithi113.stdout:0/11: creat d1/f4 x:0 0 0 2024-04-12T05:08:41.108 INFO:tasks.workunit.client.0.smithi113.stdout:9/14: creat d1/d4/d5/f7 x:0 0 0 2024-04-12T05:08:41.109 INFO:tasks.workunit.client.0.smithi113.stdout:5/14: truncate d0/f6 523795 0 2024-04-12T05:08:41.109 INFO:tasks.workunit.client.0.smithi113.stdout:0/12: chown d1 25926 1 2024-04-12T05:08:41.109 INFO:tasks.workunit.client.0.smithi113.stdout:4/19: unlink c1 0 2024-04-12T05:08:41.110 INFO:tasks.workunit.client.0.smithi113.stdout:4/20: chown c6 1463168 1 2024-04-12T05:08:41.110 INFO:tasks.workunit.client.0.smithi113.stdout:8/21: mknod c4 0 2024-04-12T05:08:41.111 INFO:tasks.workunit.client.0.smithi113.stdout:6/20: mkdir d8/d9 0 2024-04-12T05:08:41.112 INFO:tasks.workunit.client.0.smithi113.stdout:9/15: write d1/d4/d5/f7 [256237,9315] 0 2024-04-12T05:08:41.112 INFO:tasks.workunit.client.0.smithi113.stdout:0/13: symlink d1/l5 0 2024-04-12T05:08:41.113 INFO:tasks.workunit.client.0.smithi113.stdout:5/15: symlink d0/l8 0 2024-04-12T05:08:41.114 INFO:tasks.workunit.client.0.smithi113.stdout:0/14: read - d1/f4 zero size 2024-04-12T05:08:41.116 INFO:tasks.workunit.client.0.smithi113.stdout:4/21: rename c5 to c7 0 2024-04-12T05:08:41.120 INFO:tasks.workunit.client.0.smithi113.stdout:9/16: mknod d1/c8 0 2024-04-12T05:08:41.123 INFO:tasks.workunit.client.0.smithi113.stdout:8/22: dwrite f2 [4194304,4194304] 0 2024-04-12T05:08:41.128 INFO:tasks.workunit.client.0.smithi113.stdout:9/17: write d1/d4/d5/f7 [709823,77328] 0 2024-04-12T05:08:41.128 INFO:tasks.workunit.client.0.smithi113.stdout:6/21: creat d8/d9/fa x:0 0 0 2024-04-12T05:08:41.135 INFO:tasks.workunit.client.0.smithi113.stdout:6/22: creat d8/d9/fb x:0 0 0 2024-04-12T05:08:41.136 INFO:tasks.workunit.client.0.smithi113.stdout:0/15: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:41.136 INFO:tasks.workunit.client.0.smithi113.stdout:9/18: symlink d1/d4/d5/l9 0 2024-04-12T05:08:41.140 INFO:tasks.workunit.client.0.smithi113.stdout:4/22: dwrite f2 [0,4194304] 0 2024-04-12T05:08:41.144 INFO:tasks.workunit.client.0.smithi113.stdout:8/23: dwrite f2 [0,4194304] 0 2024-04-12T05:08:41.144 INFO:tasks.workunit.client.0.smithi113.stdout:8/24: rmdir - no directory 2024-04-12T05:08:41.145 INFO:tasks.workunit.client.0.smithi113.stdout:4/23: truncate f2 4966851 0 2024-04-12T05:08:41.146 INFO:tasks.workunit.client.0.smithi113.stdout:0/16: mkdir d1/d6 0 2024-04-12T05:08:41.147 INFO:tasks.workunit.client.0.smithi113.stdout:5/16: dwrite d0/f6 [0,4194304] 0 2024-04-12T05:08:41.149 INFO:tasks.workunit.client.0.smithi113.stdout:9/19: symlink d1/d4/d5/la 0 2024-04-12T05:08:41.151 INFO:tasks.workunit.client.0.smithi113.stdout:8/25: mknod c5 0 2024-04-12T05:08:41.151 INFO:tasks.workunit.client.0.smithi113.stdout:5/17: symlink d0/l9 0 2024-04-12T05:08:41.153 INFO:tasks.workunit.client.0.smithi113.stdout:5/18: write d0/f4 [330284,71765] 0 2024-04-12T05:08:41.160 INFO:tasks.workunit.client.0.smithi113.stdout:5/19: truncate d0/f4 879198 0 2024-04-12T05:08:41.166 INFO:tasks.workunit.client.0.smithi113.stdout:8/26: getdents . 0 2024-04-12T05:08:41.168 INFO:tasks.workunit.client.0.smithi113.stdout:6/23: dwrite f7 [0,4194304] 0 2024-04-12T05:08:41.168 INFO:tasks.workunit.client.0.smithi113.stdout:8/27: chown l3 12097 1 2024-04-12T05:08:41.169 INFO:tasks.workunit.client.0.smithi113.stdout:6/24: dread - d8/d9/fb zero size 2024-04-12T05:08:41.171 INFO:tasks.workunit.client.0.smithi113.stdout:5/20: unlink d0/c7 0 2024-04-12T05:08:41.174 INFO:tasks.workunit.client.0.smithi113.stdout:8/28: rename c1 to c6 0 2024-04-12T05:08:41.176 INFO:tasks.workunit.client.0.smithi113.stdout:8/29: chown c0 255591665 1 2024-04-12T05:08:41.178 INFO:tasks.workunit.client.0.smithi113.stdout:6/25: unlink f7 0 2024-04-12T05:08:41.179 INFO:tasks.workunit.client.0.smithi113.stdout:8/30: symlink l7 0 2024-04-12T05:08:41.179 INFO:tasks.workunit.client.0.smithi113.stdout:6/26: rmdir d8 39 2024-04-12T05:08:41.201 INFO:tasks.workunit.client.0.smithi113.stdout:6/27: dread - d8/d9/fa zero size 2024-04-12T05:08:41.201 INFO:tasks.workunit.client.0.smithi113.stdout:5/21: dwrite d0/f4 [0,4194304] 0 2024-04-12T05:08:41.202 INFO:tasks.workunit.client.0.smithi113.stdout:5/22: stat d0 0 2024-04-12T05:08:41.205 INFO:tasks.workunit.client.0.smithi113.stdout:8/31: dwrite f2 [0,4194304] 0 2024-04-12T05:08:41.214 INFO:tasks.workunit.client.0.smithi113.stdout:8/32: dwrite f2 [4194304,4194304] 0 2024-04-12T05:08:41.325 INFO:tasks.workunit.client.0.smithi113.stdout:5/23: creat d0/fa x:0 0 0 2024-04-12T05:08:41.328 INFO:tasks.workunit.client.0.smithi113.stdout:5/24: creat d0/fb x:0 0 0 2024-04-12T05:08:41.328 INFO:tasks.workunit.client.0.smithi113.stdout:5/25: chown d0/f4 775128 1 2024-04-12T05:08:41.331 INFO:tasks.workunit.client.0.smithi113.stdout:5/26: rename d0/fa to d0/fc 0 2024-04-12T05:08:41.334 INFO:tasks.workunit.client.0.smithi113.stdout:5/27: fsync d0/f5 0 2024-04-12T05:08:41.334 INFO:tasks.workunit.client.0.smithi113.stdout:5/28: dread - d0/f5 zero size 2024-04-12T05:08:41.336 INFO:tasks.workunit.client.0.smithi113.stdout:5/29: mknod d0/cd 0 2024-04-12T05:08:41.338 INFO:tasks.workunit.client.0.smithi113.stdout:5/30: mkdir d0/de 0 2024-04-12T05:08:41.340 INFO:tasks.workunit.client.0.smithi113.stdout:5/31: rename d0/l8 to d0/de/lf 0 2024-04-12T05:08:41.341 INFO:tasks.workunit.client.0.smithi113.stdout:5/32: dread - d0/fc zero size 2024-04-12T05:08:41.348 INFO:tasks.workunit.client.0.smithi113.stdout:5/33: dwrite d0/fc [0,4194304] 0 2024-04-12T05:08:41.355 INFO:tasks.workunit.client.0.smithi113.stdout:5/34: dwrite d0/fc [0,4194304] 0 2024-04-12T05:08:41.355 INFO:tasks.workunit.client.0.smithi113.stdout:5/35: read - d0/f5 zero size 2024-04-12T05:08:41.356 INFO:tasks.workunit.client.0.smithi113.stdout:5/36: chown d0/f4 123456950 1 2024-04-12T05:08:41.359 INFO:tasks.workunit.client.0.smithi113.stdout:5/37: dread d0/fc [0,4194304] 0 2024-04-12T05:08:41.360 INFO:tasks.workunit.client.0.smithi113.stdout:5/38: read d0/f4 [3535607,130888] 0 2024-04-12T05:08:41.363 INFO:tasks.workunit.client.0.smithi113.stdout:5/39: mkdir d0/d10 0 2024-04-12T05:08:41.371 INFO:tasks.workunit.client.0.smithi113.stdout:5/40: dwrite d0/f4 [0,4194304] 0 2024-04-12T05:08:41.374 INFO:tasks.workunit.client.0.smithi113.stdout:5/41: getdents d0/d10 0 2024-04-12T05:08:41.378 INFO:tasks.workunit.client.0.smithi113.stdout:5/42: link d0/l9 d0/de/l11 0 2024-04-12T05:08:41.380 INFO:tasks.workunit.client.0.smithi113.stdout:5/43: creat d0/d10/f12 x:0 0 0 2024-04-12T05:08:41.409 INFO:tasks.workunit.client.0.smithi113.stdout:8/33: truncate f2 3096642 0 2024-04-12T05:08:41.411 INFO:tasks.workunit.client.0.smithi113.stdout:8/34: fdatasync f2 0 2024-04-12T05:08:41.413 INFO:tasks.workunit.client.0.smithi113.stdout:5/44: write d0/fc [4200801,101348] 0 2024-04-12T05:08:41.414 INFO:tasks.workunit.client.0.smithi113.stdout:8/35: unlink c6 0 2024-04-12T05:08:41.416 INFO:tasks.workunit.client.0.smithi113.stdout:5/45: mkdir d0/de/d13 0 2024-04-12T05:08:41.417 INFO:tasks.workunit.client.0.smithi113.stdout:8/36: creat f8 x:0 0 0 2024-04-12T05:08:41.422 INFO:tasks.workunit.client.0.smithi113.stdout:5/46: rename d0/f4 to d0/f14 0 2024-04-12T05:08:41.422 INFO:tasks.workunit.client.0.smithi113.stdout:5/47: chown d0/cd 11 1 2024-04-12T05:08:41.422 INFO:tasks.workunit.client.0.smithi113.stdout:5/48: write d0/f6 [167178,115398] 0 2024-04-12T05:08:41.422 INFO:tasks.workunit.client.0.smithi113.stdout:5/49: mknod d0/de/c15 0 2024-04-12T05:08:41.425 INFO:tasks.workunit.client.0.smithi113.stdout:5/50: creat d0/d10/f16 x:0 0 0 2024-04-12T05:08:41.426 INFO:tasks.workunit.client.0.smithi113.stdout:5/51: fdatasync d0/d10/f16 0 2024-04-12T05:08:41.427 INFO:tasks.workunit.client.0.smithi113.stdout:5/52: chown d0/de/c15 2 1 2024-04-12T05:08:41.427 INFO:tasks.workunit.client.0.smithi113.stdout:5/53: write d0/fc [2572876,129952] 0 2024-04-12T05:08:41.429 INFO:tasks.workunit.client.0.smithi113.stdout:5/54: write d0/d10/f16 [865513,54246] 0 2024-04-12T05:08:41.430 INFO:tasks.workunit.client.0.smithi113.stdout:5/55: read d0/f14 [2249352,73638] 0 2024-04-12T05:08:41.434 INFO:tasks.workunit.client.0.smithi113.stdout:5/56: dread d0/fc [0,4194304] 0 2024-04-12T05:08:41.482 INFO:tasks.workunit.client.0.smithi113.stdout:5/57: truncate d0/fc 625436 0 2024-04-12T05:08:41.493 INFO:tasks.workunit.client.0.smithi113.stdout:5/58: dwrite d0/d10/f12 [0,4194304] 0 2024-04-12T05:08:41.496 INFO:tasks.workunit.client.0.smithi113.stdout:5/59: creat d0/de/f17 x:0 0 0 2024-04-12T05:08:41.501 INFO:tasks.workunit.client.0.smithi113.stdout:5/60: dread d0/f6 [0,4194304] 0 2024-04-12T05:08:41.503 INFO:tasks.workunit.client.0.smithi113.stdout:5/61: creat d0/de/f18 x:0 0 0 2024-04-12T05:08:41.508 INFO:tasks.workunit.client.0.smithi113.stdout:5/62: dread d0/fc [0,4194304] 0 2024-04-12T05:08:41.510 INFO:tasks.workunit.client.0.smithi113.stdout:5/63: readlink d0/de/l11 0 2024-04-12T05:08:41.510 INFO:tasks.workunit.client.0.smithi113.stdout:5/64: dread - d0/f5 zero size 2024-04-12T05:08:41.513 INFO:tasks.workunit.client.0.smithi113.stdout:5/65: getdents d0/de 0 2024-04-12T05:08:41.516 INFO:tasks.workunit.client.0.smithi113.stdout:5/66: rename d0/fb to d0/d10/f19 0 2024-04-12T05:08:41.521 INFO:tasks.workunit.client.0.smithi113.stdout:5/67: dwrite d0/d10/f16 [0,4194304] 0 2024-04-12T05:08:41.524 INFO:tasks.workunit.client.0.smithi113.stdout:5/68: symlink d0/de/d13/l1a 0 2024-04-12T05:08:41.527 INFO:tasks.workunit.client.0.smithi113.stdout:5/69: truncate d0/fc 1026691 0 2024-04-12T05:08:42.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:40.666651+0000 mgr.smithi192.ocppyo (mgr.24357) 12 : cluster [DBG] pgmap v10: 65 pgs: 65 active+clean; 283 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 952 KiB/s rd, 13 MiB/s wr, 197 op/s 2024-04-12T05:08:42.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:40.860000+0000 mon.smithi113 (mon.0) 669 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:42.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:40.865185+0000 mon.smithi113 (mon.0) 670 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:42.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:40.666651+0000 mgr.smithi192.ocppyo (mgr.24357) 12 : cluster [DBG] pgmap v10: 65 pgs: 65 active+clean; 283 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 952 KiB/s rd, 13 MiB/s wr, 197 op/s 2024-04-12T05:08:42.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:40.860000+0000 mon.smithi113 (mon.0) 669 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:42.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:40.865185+0000 mon.smithi113 (mon.0) 670 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:08:44.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:42.667101+0000 mgr.smithi192.ocppyo (mgr.24357) 13 : cluster [DBG] pgmap v11: 65 pgs: 65 active+clean; 359 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 977 KiB/s rd, 25 MiB/s wr, 320 op/s 2024-04-12T05:08:44.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:42.667101+0000 mgr.smithi192.ocppyo (mgr.24357) 13 : cluster [DBG] pgmap v11: 65 pgs: 65 active+clean; 359 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 977 KiB/s rd, 25 MiB/s wr, 320 op/s 2024-04-12T05:08:46.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:44.667651+0000 mgr.smithi192.ocppyo (mgr.24357) 14 : cluster [DBG] pgmap v12: 65 pgs: 65 active+clean; 359 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 873 KiB/s rd, 23 MiB/s wr, 286 op/s 2024-04-12T05:08:46.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:44.667651+0000 mgr.smithi192.ocppyo (mgr.24357) 14 : cluster [DBG] pgmap v12: 65 pgs: 65 active+clean; 359 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 873 KiB/s rd, 23 MiB/s wr, 286 op/s 2024-04-12T05:08:46.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.703+0000 7f371c244700 1 -- 172.21.15.113:0/4050655703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f7990 msgr2=0x7f37140f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:46.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.703+0000 7f371c244700 1 --2- 172.21.15.113:0/4050655703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f7990 0x7f37140f7df0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f37100047f0 tx=0x7f3710009cb0).stop 2024-04-12T05:08:46.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.703+0000 7f371c244700 1 -- 172.21.15.113:0/4050655703 shutdown_connections 2024-04-12T05:08:46.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.703+0000 7f371c244700 1 --2- 172.21.15.113:0/4050655703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f7990 0x7f37140f7df0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:46.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.703+0000 7f371c244700 1 --2- 172.21.15.113:0/4050655703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f66f0 0x7f37140f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:08:46.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.703+0000 7f371c244700 1 -- 172.21.15.113:0/4050655703 >> 172.21.15.113:0/4050655703 conn(0x7f37140f1e50 msgr2=0x7f37140f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:46.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.704+0000 7f371c244700 1 -- 172.21.15.113:0/4050655703 shutdown_connections 2024-04-12T05:08:46.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.704+0000 7f371c244700 1 -- 172.21.15.113:0/4050655703 wait complete. 2024-04-12T05:08:46.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.704+0000 7f371c244700 1 Processor -- start 2024-04-12T05:08:46.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.704+0000 7f371c244700 1 -- start start 2024-04-12T05:08:46.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.704+0000 7f371c244700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f66f0 0x7f37141006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:46.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.704+0000 7f371c244700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f7990 0x7f37140fed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:46.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.704+0000 7f371c244700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37140ff240 con 0x7f37140f7990 2024-04-12T05:08:46.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.704+0000 7f371c244700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37140ff380 con 0x7f37140f66f0 2024-04-12T05:08:46.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.704+0000 7f37197df700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f7990 0x7f37140fed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:46.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.704+0000 7f37197df700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f7990 0x7f37140fed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37788/0 (socket says 172.21.15.113:37788) 2024-04-12T05:08:46.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.704+0000 7f37197df700 1 -- 172.21.15.113:0/2790972528 learned_addr learned my addr 172.21.15.113:0/2790972528 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:08:46.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.705+0000 7f37197df700 1 -- 172.21.15.113:0/2790972528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f66f0 msgr2=0x7f37141006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:46.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.705+0000 7f3719fe0700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f66f0 0x7f37141006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:46.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.705+0000 7f37197df700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f66f0 0x7f37141006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:46.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.705+0000 7f37197df700 1 -- 172.21.15.113:0/2790972528 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3710005040 con 0x7f37140f7990 2024-04-12T05:08:46.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.705+0000 7f3719fe0700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f66f0 0x7f37141006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:08:46.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.705+0000 7f37197df700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f7990 0x7f37140fed50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f371000b120 tx=0x7f3710009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:46.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.705+0000 7f370affd700 1 -- 172.21.15.113:0/2790972528 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f371000e030 con 0x7f37140f7990 2024-04-12T05:08:46.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.705+0000 7f371c244700 1 -- 172.21.15.113:0/2790972528 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37140ff600 con 0x7f37140f7990 2024-04-12T05:08:46.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.705+0000 7f370affd700 1 -- 172.21.15.113:0/2790972528 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3710007e90 con 0x7f37140f7990 2024-04-12T05:08:46.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.705+0000 7f370affd700 1 -- 172.21.15.113:0/2790972528 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3710018070 con 0x7f37140f7990 2024-04-12T05:08:46.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.705+0000 7f371c244700 1 -- 172.21.15.113:0/2790972528 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37140ffae0 con 0x7f37140f7990 2024-04-12T05:08:46.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.706+0000 7f371c244700 1 -- 172.21.15.113:0/2790972528 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37140608b0 con 0x7f37140f7990 2024-04-12T05:08:46.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.707+0000 7f370affd700 1 -- 172.21.15.113:0/2790972528 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7f371001d960 con 0x7f37140f7990 2024-04-12T05:08:46.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.707+0000 7f370affd700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f3700066dc0 0x7f3700069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:46.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.707+0000 7f370affd700 1 -- 172.21.15.113:0/2790972528 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f3710085990 con 0x7f37140f7990 2024-04-12T05:08:46.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.708+0000 7f3719fe0700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f3700066dc0 0x7f3700069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:08:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.709+0000 7f3719fe0700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f3700066dc0 0x7f3700069280 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f37140f77f0 tx=0x7f3704008040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:46.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.712+0000 7f370affd700 1 -- 172.21.15.113:0/2790972528 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f3710053a30 con 0x7f37140f7990 2024-04-12T05:08:46.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.916+0000 7f371c244700 1 -- 172.21.15.113:0/2790972528 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f37140fc2e0 con 0x7f3700066dc0 2024-04-12T05:08:46.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.919+0000 7f370affd700 1 -- 172.21.15.113:0/2790972528 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f37140fc2e0 con 0x7f3700066dc0 2024-04-12T05:08:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.921+0000 7f371c244700 1 -- 172.21.15.113:0/2790972528 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f3700066dc0 msgr2=0x7f3700069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.921+0000 7f371c244700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f3700066dc0 0x7f3700069280 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f37140f77f0 tx=0x7f3704008040).stop 2024-04-12T05:08:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.921+0000 7f371c244700 1 -- 172.21.15.113:0/2790972528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f7990 msgr2=0x7f37140fed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.921+0000 7f371c244700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f7990 0x7f37140fed50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f371000b120 tx=0x7f3710009cb0).stop 2024-04-12T05:08:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.921+0000 7f371c244700 1 -- 172.21.15.113:0/2790972528 shutdown_connections 2024-04-12T05:08:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.921+0000 7f371c244700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f3700066dc0 0x7f3700069280 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.921+0000 7f371c244700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37140f66f0 0x7f37141006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.921+0000 7f371c244700 1 --2- 172.21.15.113:0/2790972528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37140f7990 0x7f37140fed50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.921+0000 7f371c244700 1 -- 172.21.15.113:0/2790972528 >> 172.21.15.113:0/2790972528 conn(0x7f37140f1e50 msgr2=0x7f37140fabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.921+0000 7f371c244700 1 -- 172.21.15.113:0/2790972528 shutdown_connections 2024-04-12T05:08:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:46.921+0000 7f371c244700 1 -- 172.21.15.113:0/2790972528 wait complete. 2024-04-12T05:08:46.932 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:08:47.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.019+0000 7f053f8c7700 1 -- 172.21.15.113:0/1328416447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05380f7990 msgr2=0x7f05380f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.019+0000 7f053f8c7700 1 --2- 172.21.15.113:0/1328416447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05380f7990 0x7f05380f7df0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f05340047f0 tx=0x7f0534009cb0).stop 2024-04-12T05:08:47.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.019+0000 7f053f8c7700 1 -- 172.21.15.113:0/1328416447 shutdown_connections 2024-04-12T05:08:47.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.019+0000 7f053f8c7700 1 --2- 172.21.15.113:0/1328416447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05380f7990 0x7f05380f7df0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.019+0000 7f053f8c7700 1 --2- 172.21.15.113:0/1328416447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05380f66f0 0x7f05380f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:08:47.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.019+0000 7f053f8c7700 1 -- 172.21.15.113:0/1328416447 >> 172.21.15.113:0/1328416447 conn(0x7f05380f1e50 msgr2=0x7f05380f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:47.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.019+0000 7f053f8c7700 1 -- 172.21.15.113:0/1328416447 shutdown_connections 2024-04-12T05:08:47.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.019+0000 7f053f8c7700 1 -- 172.21.15.113:0/1328416447 wait complete. 2024-04-12T05:08:47.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.019+0000 7f053f8c7700 1 Processor -- start 2024-04-12T05:08:47.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.019+0000 7f053f8c7700 1 -- start start 2024-04-12T05:08:47.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053f8c7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05380f66f0 0x7f0538061060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053f8c7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05380f7990 0x7f053810cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053f8c7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053810d2e0 con 0x7f05380f66f0 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053f8c7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053810d420 con 0x7f05380f7990 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053d663700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05380f66f0 0x7f0538061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053d663700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05380f66f0 0x7f0538061060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37806/0 (socket says 172.21.15.113:37806) 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053d663700 1 -- 172.21.15.113:0/4007508942 learned_addr learned my addr 172.21.15.113:0/4007508942 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053ce62700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05380f7990 0x7f053810cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053d663700 1 -- 172.21.15.113:0/4007508942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05380f7990 msgr2=0x7f053810cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053d663700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05380f7990 0x7f053810cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053d663700 1 -- 172.21.15.113:0/4007508942 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0534005040 con 0x7f05380f66f0 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053ce62700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05380f7990 0x7f053810cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053d663700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05380f66f0 0x7f0538061060 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f052800b480 tx=0x7f05280090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:47.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f052e7fc700 1 -- 172.21.15.113:0/4007508942 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0528002500 con 0x7f05380f66f0 2024-04-12T05:08:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f052e7fc700 1 -- 172.21.15.113:0/4007508942 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0528002b10 con 0x7f05380f66f0 2024-04-12T05:08:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053f8c7700 1 -- 172.21.15.113:0/4007508942 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053810d700 con 0x7f05380f66f0 2024-04-12T05:08:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f052e7fc700 1 -- 172.21.15.113:0/4007508942 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0528012c90 con 0x7f05380f66f0 2024-04-12T05:08:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.020+0000 7f053f8c7700 1 -- 172.21.15.113:0/4007508942 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053810dc40 con 0x7f05380f66f0 2024-04-12T05:08:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.021+0000 7f053f8c7700 1 -- 172.21.15.113:0/4007508942 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05380fef50 con 0x7f05380f66f0 2024-04-12T05:08:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.022+0000 7f052e7fc700 1 -- 172.21.15.113:0/4007508942 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7f0528002660 con 0x7f05380f66f0 2024-04-12T05:08:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.022+0000 7f052e7fc700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0524066dc0 0x7f0524069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.022+0000 7f052e7fc700 1 -- 172.21.15.113:0/4007508942 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f0528084230 con 0x7f05380f66f0 2024-04-12T05:08:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.022+0000 7f053ce62700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0524066dc0 0x7f0524069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:08:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.023+0000 7f053ce62700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0524066dc0 0x7f0524069280 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f05340043d0 tx=0x7f053400b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:47.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.027+0000 7f052e7fc700 1 -- 172.21.15.113:0/4007508942 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f0528052460 con 0x7f05380f66f0 2024-04-12T05:08:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:45.908693+0000 mon.smithi192 (mon.1) 48 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:08:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:45.908693+0000 mon.smithi192 (mon.1) 48 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:08:47.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.232+0000 7f053f8c7700 1 -- 172.21.15.113:0/4007508942 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0538059120 con 0x7f0524066dc0 2024-04-12T05:08:47.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.234+0000 7f052e7fc700 1 -- 172.21.15.113:0/4007508942 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f0538059120 con 0x7f0524066dc0 2024-04-12T05:08:47.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.236+0000 7f053f8c7700 1 -- 172.21.15.113:0/4007508942 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0524066dc0 msgr2=0x7f0524069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.236+0000 7f053f8c7700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0524066dc0 0x7f0524069280 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f05340043d0 tx=0x7f053400b040).stop 2024-04-12T05:08:47.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.236+0000 7f053f8c7700 1 -- 172.21.15.113:0/4007508942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05380f66f0 msgr2=0x7f0538061060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.236+0000 7f053f8c7700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05380f66f0 0x7f0538061060 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f052800b480 tx=0x7f05280090f0).stop 2024-04-12T05:08:47.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.236+0000 7f053f8c7700 1 -- 172.21.15.113:0/4007508942 shutdown_connections 2024-04-12T05:08:47.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.236+0000 7f053f8c7700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0524066dc0 0x7f0524069280 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.236+0000 7f053f8c7700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05380f66f0 0x7f0538061060 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.239 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.236+0000 7f053f8c7700 1 --2- 172.21.15.113:0/4007508942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05380f7990 0x7f053810cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.239 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.236+0000 7f053f8c7700 1 -- 172.21.15.113:0/4007508942 >> 172.21.15.113:0/4007508942 conn(0x7f05380f1e50 msgr2=0x7f05380fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:47.239 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.236+0000 7f053f8c7700 1 -- 172.21.15.113:0/4007508942 shutdown_connections 2024-04-12T05:08:47.239 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.236+0000 7f053f8c7700 1 -- 172.21.15.113:0/4007508942 wait complete. 2024-04-12T05:08:47.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.335+0000 7fe645a2b700 1 -- 172.21.15.113:0/2212013898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6400fa950 msgr2=0x7fe6400fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.335+0000 7fe645a2b700 1 --2- 172.21.15.113:0/2212013898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6400fa950 0x7fe6400fcda0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fe6340047f0 tx=0x7fe634009cb0).stop 2024-04-12T05:08:47.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.335+0000 7fe645a2b700 1 -- 172.21.15.113:0/2212013898 shutdown_connections 2024-04-12T05:08:47.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.335+0000 7fe645a2b700 1 --2- 172.21.15.113:0/2212013898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6400fa950 0x7fe6400fcda0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.335+0000 7fe645a2b700 1 --2- 172.21.15.113:0/2212013898 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe640061280 0x7fe6400fa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.335+0000 7fe645a2b700 1 -- 172.21.15.113:0/2212013898 >> 172.21.15.113:0/2212013898 conn(0x7fe6400f1e50 msgr2=0x7fe6400f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:47.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.335+0000 7fe645a2b700 1 -- 172.21.15.113:0/2212013898 shutdown_connections 2024-04-12T05:08:47.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.335+0000 7fe645a2b700 1 -- 172.21.15.113:0/2212013898 wait complete. 2024-04-12T05:08:47.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.335+0000 7fe645a2b700 1 Processor -- start 2024-04-12T05:08:47.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe645a2b700 1 -- start start 2024-04-12T05:08:47.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe645a2b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe640061280 0x7fe640107f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:47.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe645a2b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6400fa950 0x7fe640108400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:47.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe645a2b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6401089b0 con 0x7fe6400fa950 2024-04-12T05:08:47.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe645a2b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe640108af0 con 0x7fe640061280 2024-04-12T05:08:47.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe63e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6400fa950 0x7fe640108400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:47.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe63e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6400fa950 0x7fe640108400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37814/0 (socket says 172.21.15.113:37814) 2024-04-12T05:08:47.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe63e7fc700 1 -- 172.21.15.113:0/1212360678 learned_addr learned my addr 172.21.15.113:0/1212360678 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:08:47.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe63effd700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe640061280 0x7fe640107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:47.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe63e7fc700 1 -- 172.21.15.113:0/1212360678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe640061280 msgr2=0x7fe640107f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe63e7fc700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe640061280 0x7fe640107f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe63e7fc700 1 -- 172.21.15.113:0/1212360678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe634005040 con 0x7fe6400fa950 2024-04-12T05:08:47.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe63effd700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe640061280 0x7fe640107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:08:47.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.336+0000 7fe63e7fc700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6400fa950 0x7fe640108400 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fe634013620 tx=0x7fe634009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:47.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.337+0000 7fe627fff700 1 -- 172.21.15.113:0/1212360678 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe63400e070 con 0x7fe6400fa950 2024-04-12T05:08:47.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.337+0000 7fe645a2b700 1 -- 172.21.15.113:0/1212360678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64010d530 con 0x7fe6400fa950 2024-04-12T05:08:47.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.337+0000 7fe627fff700 1 -- 172.21.15.113:0/1212360678 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe634007e90 con 0x7fe6400fa950 2024-04-12T05:08:47.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.337+0000 7fe627fff700 1 -- 172.21.15.113:0/1212360678 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe634018070 con 0x7fe6400fa950 2024-04-12T05:08:47.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.337+0000 7fe645a2b700 1 -- 172.21.15.113:0/1212360678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64010da70 con 0x7fe6400fa950 2024-04-12T05:08:47.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.338+0000 7fe645a2b700 1 -- 172.21.15.113:0/1212360678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6400fffb0 con 0x7fe6400fa950 2024-04-12T05:08:47.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.338+0000 7fe627fff700 1 -- 172.21.15.113:0/1212360678 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7fe63401d420 con 0x7fe6400fa950 2024-04-12T05:08:47.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.338+0000 7fe627fff700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe628066dc0 0x7fe628069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:47.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.339+0000 7fe627fff700 1 -- 172.21.15.113:0/1212360678 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7fe634085020 con 0x7fe6400fa950 2024-04-12T05:08:47.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.339+0000 7fe63effd700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe628066dc0 0x7fe628069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:08:47.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.339+0000 7fe63effd700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe628066dc0 0x7fe628069280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fe630009250 tx=0x7fe630009de0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:47.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.343+0000 7fe627fff700 1 -- 172.21.15.113:0/1212360678 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fe6340530c0 con 0x7fe6400fa950 2024-04-12T05:08:47.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.537+0000 7fe645a2b700 1 -- 172.21.15.113:0/1212360678 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe640059120 con 0x7fe628066dc0 2024-04-12T05:08:47.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.548+0000 7fe627fff700 1 -- 172.21.15.113:0/1212360678 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7fe640059120 con 0x7fe628066dc0 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (6m) 7s ago 7m 26.2M - 0.23.0 ba2b418f427c aae81b8048e6 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (7m) 7s ago 7m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (7m) 21s ago 7m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (6m) 7s ago 7m 60.7M - 8.3.5 dad864ee21e9 7d83fd029dea 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (2m) 7s ago 2m 97.7M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (2m) 7s ago 2m 15.4M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (2m) 21s ago 2m 92.7M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (2m) 21s ago 2m 12.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:9283 running (9m) 7s ago 9m 382M - 16.2.15-1-g8963e9de 727f386679cb 3fab7f291e6b 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283 running (26s) 21s ago 7m 45.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 550d8cc53ae9 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (9m) 7s ago 9m 49.3M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (7m) 21s ago 7m 36.5M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (7m) 7s ago 7m 22.5M - 1.3.1 1dbe0e931976 046bcfb16579 2024-04-12T05:08:47.551 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (6m) 21s ago 6m 28.8M - 1.3.1 1dbe0e931976 bc604318825c 2024-04-12T05:08:47.552 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (6m) 7s ago 6m 109M 4640M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:08:47.552 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (5m) 7s ago 5m 124M 4640M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:08:47.552 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (5m) 7s ago 5m 102M 4640M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:08:47.552 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (4m) 21s ago 4m 111M 4982M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:08:47.552 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (4m) 21s ago 4m 102M 4982M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:08:47.552 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (4m) 21s ago 4m 96.6M 4982M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:08:47.552 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (6m) 21s ago 6m 62.7M - 2.33.4 514e6a882f6e b4259537bd55 2024-04-12T05:08:47.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.550+0000 7fe645a2b700 1 -- 172.21.15.113:0/1212360678 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe628066dc0 msgr2=0x7fe628069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.551+0000 7fe645a2b700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe628066dc0 0x7fe628069280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fe630009250 tx=0x7fe630009de0).stop 2024-04-12T05:08:47.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.551+0000 7fe645a2b700 1 -- 172.21.15.113:0/1212360678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6400fa950 msgr2=0x7fe640108400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.551+0000 7fe645a2b700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6400fa950 0x7fe640108400 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fe634013620 tx=0x7fe634009cb0).stop 2024-04-12T05:08:47.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.551+0000 7fe645a2b700 1 -- 172.21.15.113:0/1212360678 shutdown_connections 2024-04-12T05:08:47.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.551+0000 7fe645a2b700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe628066dc0 0x7fe628069280 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.551+0000 7fe645a2b700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe640061280 0x7fe640107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.551+0000 7fe645a2b700 1 --2- 172.21.15.113:0/1212360678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6400fa950 0x7fe640108400 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.551+0000 7fe645a2b700 1 -- 172.21.15.113:0/1212360678 >> 172.21.15.113:0/1212360678 conn(0x7fe6400f1e50 msgr2=0x7fe6400f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:47.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.551+0000 7fe645a2b700 1 -- 172.21.15.113:0/1212360678 shutdown_connections 2024-04-12T05:08:47.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.551+0000 7fe645a2b700 1 -- 172.21.15.113:0/1212360678 wait complete. 2024-04-12T05:08:47.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.647+0000 7f8c64229700 1 -- 172.21.15.113:0/3333171504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c5c0f7990 msgr2=0x7f8c5c0f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.647+0000 7f8c64229700 1 --2- 172.21.15.113:0/3333171504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c5c0f7990 0x7f8c5c0f7df0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f8c500047f0 tx=0x7f8c50009cb0).stop 2024-04-12T05:08:47.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.647+0000 7f8c64229700 1 -- 172.21.15.113:0/3333171504 shutdown_connections 2024-04-12T05:08:47.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.647+0000 7f8c64229700 1 --2- 172.21.15.113:0/3333171504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c5c0f7990 0x7f8c5c0f7df0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.647+0000 7f8c64229700 1 --2- 172.21.15.113:0/3333171504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c5c0f66f0 0x7f8c5c0f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.647+0000 7f8c64229700 1 -- 172.21.15.113:0/3333171504 >> 172.21.15.113:0/3333171504 conn(0x7f8c5c0f1e50 msgr2=0x7f8c5c0f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:47.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.647+0000 7f8c64229700 1 -- 172.21.15.113:0/3333171504 shutdown_connections 2024-04-12T05:08:47.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.647+0000 7f8c64229700 1 -- 172.21.15.113:0/3333171504 wait complete. 2024-04-12T05:08:47.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.647+0000 7f8c64229700 1 Processor -- start 2024-04-12T05:08:47.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.647+0000 7f8c64229700 1 -- start start 2024-04-12T05:08:47.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c64229700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c5c0f66f0 0x7f8c5c107f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:47.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c64229700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c5c0f7990 0x7f8c5c108440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:47.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c64229700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c1089f0 con 0x7f8c5c0f7990 2024-04-12T05:08:47.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c64229700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c108b30 con 0x7f8c5c0f66f0 2024-04-12T05:08:47.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c617c4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c5c0f7990 0x7f8c5c108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c617c4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c5c0f7990 0x7f8c5c108440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37834/0 (socket says 172.21.15.113:37834) 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c617c4700 1 -- 172.21.15.113:0/1931801007 learned_addr learned my addr 172.21.15.113:0/1931801007 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c61fc5700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c5c0f66f0 0x7f8c5c107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c617c4700 1 -- 172.21.15.113:0/1931801007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c5c0f66f0 msgr2=0x7f8c5c107f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c617c4700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c5c0f66f0 0x7f8c5c107f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c617c4700 1 -- 172.21.15.113:0/1931801007 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c50005040 con 0x7f8c5c0f7990 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c61fc5700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c5c0f66f0 0x7f8c5c107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c617c4700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c5c0f7990 0x7f8c5c108440 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f8c5000b120 tx=0x7f8c50009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c5affd700 1 -- 172.21.15.113:0/1931801007 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c5000e030 con 0x7f8c5c0f7990 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c5affd700 1 -- 172.21.15.113:0/1931801007 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c50007e90 con 0x7f8c5c0f7990 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c5affd700 1 -- 172.21.15.113:0/1931801007 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c50018070 con 0x7f8c5c0f7990 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c64229700 1 -- 172.21.15.113:0/1931801007 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5c10d570 con 0x7f8c5c0f7990 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.648+0000 7f8c64229700 1 -- 172.21.15.113:0/1931801007 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5c10db30 con 0x7f8c5c0f7990 2024-04-12T05:08:47.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.649+0000 7f8c64229700 1 -- 172.21.15.113:0/1931801007 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5c0feeb0 con 0x7f8c5c0f7990 2024-04-12T05:08:47.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.650+0000 7f8c5affd700 1 -- 172.21.15.113:0/1931801007 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7f8c5001d960 con 0x7f8c5c0f7990 2024-04-12T05:08:47.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.650+0000 7f8c5affd700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8c44066dc0 0x7f8c44069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:47.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.650+0000 7f8c5affd700 1 -- 172.21.15.113:0/1931801007 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f8c50085990 con 0x7f8c5c0f7990 2024-04-12T05:08:47.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.651+0000 7f8c61fc5700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8c44066dc0 0x7f8c44069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:08:47.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.651+0000 7f8c61fc5700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8c44066dc0 0x7f8c44069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f8c5c0f77f0 tx=0x7f8c4c009040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:47.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.655+0000 7f8c5affd700 1 -- 172.21.15.113:0/1931801007 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f8c50053a30 con 0x7f8c5c0f7990 2024-04-12T05:08:47.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.924+0000 7f8c64229700 1 -- 172.21.15.113:0/1931801007 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8c5c10e480 con 0x7f8c5c0f7990 2024-04-12T05:08:47.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.925+0000 7f8c5affd700 1 -- 172.21.15.113:0/1931801007 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+850 (secure 0 0 0) 0x7f8c50053620 con 0x7f8c5c0f7990 2024-04-12T05:08:47.927 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:08:47.927 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:08:47.927 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:08:47.927 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:08:47.927 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:08:47.927 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:08:47.927 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:08:47.927 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:08:47.927 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:08:47.927 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:08:47.927 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:08:47.928 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:08:47.928 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:08:47.928 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:08:47.928 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:08:47.928 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 13, 2024-04-12T05:08:47.928 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:08:47.928 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:08:47.928 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:08:47.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.927+0000 7f8c64229700 1 -- 172.21.15.113:0/1931801007 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8c44066dc0 msgr2=0x7f8c44069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.927+0000 7f8c64229700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8c44066dc0 0x7f8c44069280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f8c5c0f77f0 tx=0x7f8c4c009040).stop 2024-04-12T05:08:47.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.927+0000 7f8c64229700 1 -- 172.21.15.113:0/1931801007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c5c0f7990 msgr2=0x7f8c5c108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:47.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.927+0000 7f8c64229700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c5c0f7990 0x7f8c5c108440 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f8c5000b120 tx=0x7f8c50009cb0).stop 2024-04-12T05:08:47.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.927+0000 7f8c64229700 1 -- 172.21.15.113:0/1931801007 shutdown_connections 2024-04-12T05:08:47.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.927+0000 7f8c64229700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8c44066dc0 0x7f8c44069280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.927+0000 7f8c64229700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c5c0f66f0 0x7f8c5c107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.927+0000 7f8c64229700 1 --2- 172.21.15.113:0/1931801007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c5c0f7990 0x7f8c5c108440 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:47.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.927+0000 7f8c64229700 1 -- 172.21.15.113:0/1931801007 >> 172.21.15.113:0/1931801007 conn(0x7f8c5c0f1e50 msgr2=0x7f8c5c060d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:47.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.927+0000 7f8c64229700 1 -- 172.21.15.113:0/1931801007 shutdown_connections 2024-04-12T05:08:47.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:47.927+0000 7f8c64229700 1 -- 172.21.15.113:0/1931801007 wait complete. 2024-04-12T05:08:48.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.024+0000 7f8fff623700 1 -- 172.21.15.113:0/562234370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ff80f7a10 msgr2=0x7f8ff80f7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:48.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.024+0000 7f8fff623700 1 --2- 172.21.15.113:0/562234370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ff80f7a10 0x7f8ff80f7e30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f8ff40047f0 tx=0x7f8ff4009cb0).stop 2024-04-12T05:08:48.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.024+0000 7f8fff623700 1 -- 172.21.15.113:0/562234370 shutdown_connections 2024-04-12T05:08:48.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.024+0000 7f8fff623700 1 --2- 172.21.15.113:0/562234370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff80f8cb0 0x7f8ff80f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:08:48.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.024+0000 7f8fff623700 1 --2- 172.21.15.113:0/562234370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ff80f7a10 0x7f8ff80f7e30 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:48.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.024+0000 7f8fff623700 1 -- 172.21.15.113:0/562234370 >> 172.21.15.113:0/562234370 conn(0x7f8ff80f3150 msgr2=0x7f8ff80f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:48.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.024+0000 7f8fff623700 1 -- 172.21.15.113:0/562234370 shutdown_connections 2024-04-12T05:08:48.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.024+0000 7f8fff623700 1 -- 172.21.15.113:0/562234370 wait complete. 2024-04-12T05:08:48.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8fff623700 1 Processor -- start 2024-04-12T05:08:48.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8fff623700 1 -- start start 2024-04-12T05:08:48.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8fff623700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff80f7a10 0x7f8ff8108230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8fff623700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ff80f8cb0 0x7f8ff8108720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8fff623700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff8108d60 con 0x7f8ff80f8cb0 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8fff623700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff8108ea0 con 0x7f8ff80f7a10 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8ffcbbe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ff80f8cb0 0x7f8ff8108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8ffcbbe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ff80f8cb0 0x7f8ff8108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37858/0 (socket says 172.21.15.113:37858) 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8ffcbbe700 1 -- 172.21.15.113:0/3157020735 learned_addr learned my addr 172.21.15.113:0/3157020735 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8ffd3bf700 1 --2- 172.21.15.113:0/3157020735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff80f7a10 0x7f8ff8108230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8ffcbbe700 1 -- 172.21.15.113:0/3157020735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff80f7a10 msgr2=0x7f8ff8108230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8ffcbbe700 1 --2- 172.21.15.113:0/3157020735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff80f7a10 0x7f8ff8108230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.025+0000 7f8ffcbbe700 1 -- 172.21.15.113:0/3157020735 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff4005040 con 0x7f8ff80f8cb0 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.026+0000 7f8ffcbbe700 1 --2- 172.21.15.113:0/3157020735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ff80f8cb0 0x7f8ff8108720 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f8fe800b4b0 tx=0x7f8fe80090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.026+0000 7f8fee7fc700 1 -- 172.21.15.113:0/3157020735 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fe8002500 con 0x7f8ff80f8cb0 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.026+0000 7f8fff623700 1 -- 172.21.15.113:0/3157020735 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff810d940 con 0x7f8ff80f8cb0 2024-04-12T05:08:48.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.026+0000 7f8fee7fc700 1 -- 172.21.15.113:0/3157020735 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8fe8002b10 con 0x7f8ff80f8cb0 2024-04-12T05:08:48.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.026+0000 7f8fee7fc700 1 -- 172.21.15.113:0/3157020735 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fe8012bf0 con 0x7f8ff80f8cb0 2024-04-12T05:08:48.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.026+0000 7f8fff623700 1 -- 172.21.15.113:0/3157020735 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff810de80 con 0x7f8ff80f8cb0 2024-04-12T05:08:48.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.027+0000 7f8fff623700 1 -- 172.21.15.113:0/3157020735 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff805fb50 con 0x7f8ff80f8cb0 2024-04-12T05:08:48.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.027+0000 7f8fee7fc700 1 -- 172.21.15.113:0/3157020735 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7f8fe8002660 con 0x7f8ff80f8cb0 2024-04-12T05:08:48.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.027+0000 7f8fee7fc700 1 --2- 172.21.15.113:0/3157020735 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8fe4066dc0 0x7f8fe4069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:08:48.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.028+0000 7f8fee7fc700 1 -- 172.21.15.113:0/3157020735 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f8fe8084250 con 0x7f8ff80f8cb0 2024-04-12T05:08:48.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.028+0000 7f8ffd3bf700 1 --2- 172.21.15.113:0/3157020735 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8fe4066dc0 0x7f8fe4069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:08:48.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.028+0000 7f8ffd3bf700 1 --2- 172.21.15.113:0/3157020735 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8fe4066dc0 0x7f8fe4069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f8ff4004f10 tx=0x7f8ff400b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:08:48.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.032+0000 7f8fee7fc700 1 -- 172.21.15.113:0/3157020735 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f8fe8052400 con 0x7f8ff80f8cb0 2024-04-12T05:08:48.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:46.668261+0000 mgr.smithi192.ocppyo (mgr.24357) 15 : cluster [DBG] pgmap v13: 65 pgs: 65 active+clean; 359 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 731 KiB/s rd, 19 MiB/s wr, 240 op/s 2024-04-12T05:08:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:46.918959+0000 mgr.smithi192.ocppyo (mgr.24357) 16 : audit [DBG] from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:47.234404+0000 mgr.smithi192.ocppyo (mgr.24357) 17 : audit [DBG] from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:48.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:46.668261+0000 mgr.smithi192.ocppyo (mgr.24357) 15 : cluster [DBG] pgmap v13: 65 pgs: 65 active+clean; 359 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 731 KiB/s rd, 19 MiB/s wr, 240 op/s 2024-04-12T05:08:48.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:46.918959+0000 mgr.smithi192.ocppyo (mgr.24357) 16 : audit [DBG] from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:48.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:47.234404+0000 mgr.smithi192.ocppyo (mgr.24357) 17 : audit [DBG] from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:48.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.238+0000 7f8fff623700 1 -- 172.21.15.113:0/3157020735 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ff81003e0 con 0x7f8fe4066dc0 2024-04-12T05:08:48.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.240+0000 7f8fee7fc700 1 -- 172.21.15.113:0/3157020735 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f8ff81003e0 con 0x7f8fe4066dc0 2024-04-12T05:08:48.242 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:08:48.242 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:08:48.242 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:08:48.242 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:08:48.242 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [], 2024-04-12T05:08:48.242 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "1/2 daemons upgraded", 2024-04-12T05:08:48.242 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "", 2024-04-12T05:08:48.242 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:08:48.242 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:08:48.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.242+0000 7f8fff623700 1 -- 172.21.15.113:0/3157020735 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8fe4066dc0 msgr2=0x7f8fe4069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:48.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.242+0000 7f8fff623700 1 --2- 172.21.15.113:0/3157020735 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8fe4066dc0 0x7f8fe4069280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f8ff4004f10 tx=0x7f8ff400b040).stop 2024-04-12T05:08:48.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.242+0000 7f8fff623700 1 -- 172.21.15.113:0/3157020735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ff80f8cb0 msgr2=0x7f8ff8108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:08:48.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.242+0000 7f8fff623700 1 --2- 172.21.15.113:0/3157020735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ff80f8cb0 0x7f8ff8108720 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f8fe800b4b0 tx=0x7f8fe80090f0).stop 2024-04-12T05:08:48.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.242+0000 7f8fff623700 1 -- 172.21.15.113:0/3157020735 shutdown_connections 2024-04-12T05:08:48.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.242+0000 7f8fff623700 1 --2- 172.21.15.113:0/3157020735 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8fe4066dc0 0x7f8fe4069280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:48.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.242+0000 7f8fff623700 1 --2- 172.21.15.113:0/3157020735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ff80f7a10 0x7f8ff8108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:48.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.242+0000 7f8fff623700 1 --2- 172.21.15.113:0/3157020735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ff80f8cb0 0x7f8ff8108720 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:08:48.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.242+0000 7f8fff623700 1 -- 172.21.15.113:0/3157020735 >> 172.21.15.113:0/3157020735 conn(0x7f8ff80f3150 msgr2=0x7f8ff80fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:08:48.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.242+0000 7f8fff623700 1 -- 172.21.15.113:0/3157020735 shutdown_connections 2024-04-12T05:08:48.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:08:48.242+0000 7f8fff623700 1 -- 172.21.15.113:0/3157020735 wait complete. 2024-04-12T05:08:48.555 INFO:tasks.workunit.client.0.smithi113.stdout:7/11: sync 2024-04-12T05:08:48.555 INFO:tasks.workunit.client.0.smithi113.stdout:7/12: dwrite - no filename 2024-04-12T05:08:48.555 INFO:tasks.workunit.client.0.smithi113.stdout:1/13: sync 2024-04-12T05:08:48.556 INFO:tasks.workunit.client.0.smithi113.stdout:8/37: sync 2024-04-12T05:08:48.556 INFO:tasks.workunit.client.0.smithi113.stdout:9/20: sync 2024-04-12T05:08:48.556 INFO:tasks.workunit.client.0.smithi113.stdout:2/12: sync 2024-04-12T05:08:48.556 INFO:tasks.workunit.client.0.smithi113.stdout:3/0: sync 2024-04-12T05:08:48.556 INFO:tasks.workunit.client.0.smithi113.stdout:6/28: sync 2024-04-12T05:08:48.556 INFO:tasks.workunit.client.0.smithi113.stdout:5/70: sync 2024-04-12T05:08:48.556 INFO:tasks.workunit.client.0.smithi113.stdout:4/24: sync 2024-04-12T05:08:48.556 INFO:tasks.workunit.client.0.smithi113.stdout:0/17: sync 2024-04-12T05:08:48.559 INFO:tasks.workunit.client.0.smithi113.stdout:1/14: unlink f1 0 2024-04-12T05:08:48.560 INFO:tasks.workunit.client.0.smithi113.stdout:8/38: mkdir d9 0 2024-04-12T05:08:48.560 INFO:tasks.workunit.client.0.smithi113.stdout:8/39: stat f8 0 2024-04-12T05:08:48.560 INFO:tasks.workunit.client.0.smithi113.stdout:2/13: fsync f1 0 2024-04-12T05:08:48.563 INFO:tasks.workunit.client.0.smithi113.stdout:7/13: symlink d0/d2/l4 0 2024-04-12T05:08:48.563 INFO:tasks.workunit.client.0.smithi113.stdout:7/14: write - no filename 2024-04-12T05:08:48.563 INFO:tasks.workunit.client.0.smithi113.stdout:7/15: write - no filename 2024-04-12T05:08:48.563 INFO:tasks.workunit.client.0.smithi113.stdout:5/71: rmdir d0/de 39 2024-04-12T05:08:48.565 INFO:tasks.workunit.client.0.smithi113.stdout:6/29: fdatasync d8/d9/fa 0 2024-04-12T05:08:48.567 INFO:tasks.workunit.client.0.smithi113.stdout:6/30: readlink l3 0 2024-04-12T05:08:48.568 INFO:tasks.workunit.client.0.smithi113.stdout:1/15: fsync f2 0 2024-04-12T05:08:48.571 INFO:tasks.workunit.client.0.smithi113.stdout:5/72: stat d0/fc 0 2024-04-12T05:08:48.572 INFO:tasks.workunit.client.0.smithi113.stdout:4/25: dwrite f2 [0,4194304] 0 2024-04-12T05:08:48.574 INFO:tasks.workunit.client.0.smithi113.stdout:6/31: creat d8/d9/fc x:0 0 0 2024-04-12T05:08:48.575 INFO:tasks.workunit.client.0.smithi113.stdout:1/16: creat f4 x:0 0 0 2024-04-12T05:08:48.576 INFO:tasks.workunit.client.0.smithi113.stdout:0/18: dwrite d1/f4 [0,4194304] 0 2024-04-12T05:08:48.578 INFO:tasks.workunit.client.0.smithi113.stdout:4/26: mknod c8 0 2024-04-12T05:08:48.579 INFO:tasks.workunit.client.0.smithi113.stdout:2/14: link c0 c2 0 2024-04-12T05:08:48.579 INFO:tasks.workunit.client.0.smithi113.stdout:2/15: readlink - no filename 2024-04-12T05:08:48.580 INFO:tasks.workunit.client.0.smithi113.stdout:0/19: write d1/f4 [196372,90333] 0 2024-04-12T05:08:48.580 INFO:tasks.workunit.client.0.smithi113.stdout:5/73: creat d0/d10/f1b x:0 0 0 2024-04-12T05:08:48.580 INFO:tasks.workunit.client.0.smithi113.stdout:7/16: link d0/d2/l3 d0/d2/l5 0 2024-04-12T05:08:48.581 INFO:tasks.workunit.client.0.smithi113.stdout:7/17: dread - no filename 2024-04-12T05:08:48.583 INFO:tasks.workunit.client.0.smithi113.stdout:1/17: creat f5 x:0 0 0 2024-04-12T05:08:48.585 INFO:tasks.workunit.client.0.smithi113.stdout:6/32: symlink d8/ld 0 2024-04-12T05:08:48.587 INFO:tasks.workunit.client.0.smithi113.stdout:1/18: write f5 [770429,88218] 0 2024-04-12T05:08:48.587 INFO:tasks.workunit.client.0.smithi113.stdout:1/19: rmdir - no directory 2024-04-12T05:08:48.588 INFO:tasks.workunit.client.0.smithi113.stdout:9/21: dwrite d1/d4/d5/f7 [0,4194304] 0 2024-04-12T05:08:48.590 INFO:tasks.workunit.client.0.smithi113.stdout:7/18: mkdir d0/d2/d6 0 2024-04-12T05:08:48.590 INFO:tasks.workunit.client.0.smithi113.stdout:7/19: dread - no filename 2024-04-12T05:08:48.591 INFO:tasks.workunit.client.0.smithi113.stdout:7/20: chown d0/d2/l4 0 1 2024-04-12T05:08:48.591 INFO:tasks.workunit.client.0.smithi113.stdout:7/21: dread - no filename 2024-04-12T05:08:48.591 INFO:tasks.workunit.client.0.smithi113.stdout:7/22: fsync - no filename 2024-04-12T05:08:48.591 INFO:tasks.workunit.client.0.smithi113.stdout:7/23: write - no filename 2024-04-12T05:08:48.591 INFO:tasks.workunit.client.0.smithi113.stdout:7/24: dread - no filename 2024-04-12T05:08:48.592 INFO:tasks.workunit.client.0.smithi113.stdout:4/27: dread f2 [0,4194304] 0 2024-04-12T05:08:48.593 INFO:tasks.workunit.client.0.smithi113.stdout:6/33: write d8/d9/fb [156290,29507] 0 2024-04-12T05:08:48.594 INFO:tasks.workunit.client.0.smithi113.stdout:1/20: mknod c6 0 2024-04-12T05:08:48.594 INFO:tasks.workunit.client.0.smithi113.stdout:6/34: truncate d8/d9/fb 911035 0 2024-04-12T05:08:48.595 INFO:tasks.workunit.client.0.smithi113.stdout:6/35: chown l2 7 1 2024-04-12T05:08:48.595 INFO:tasks.workunit.client.0.smithi113.stdout:1/21: truncate f2 1233469 0 2024-04-12T05:08:48.596 INFO:tasks.workunit.client.0.smithi113.stdout:2/16: chown c2 2 1 2024-04-12T05:08:48.596 INFO:tasks.workunit.client.0.smithi113.stdout:2/17: rmdir - no directory 2024-04-12T05:08:48.597 INFO:tasks.workunit.client.0.smithi113.stdout:7/25: readlink d0/d2/l3 0 2024-04-12T05:08:48.598 INFO:tasks.workunit.client.0.smithi113.stdout:7/26: chown d0/d2/d6 495226296 1 2024-04-12T05:08:48.598 INFO:tasks.workunit.client.0.smithi113.stdout:7/27: dwrite - no filename 2024-04-12T05:08:48.598 INFO:tasks.workunit.client.0.smithi113.stdout:7/28: dwrite - no filename 2024-04-12T05:08:48.598 INFO:tasks.workunit.client.0.smithi113.stdout:7/29: dwrite - no filename 2024-04-12T05:08:48.600 INFO:tasks.workunit.client.0.smithi113.stdout:4/28: write f2 [2360072,10870] 0 2024-04-12T05:08:48.601 INFO:tasks.workunit.client.0.smithi113.stdout:6/36: mkdir d8/de 0 2024-04-12T05:08:48.603 INFO:tasks.workunit.client.0.smithi113.stdout:7/30: creat d0/f7 x:0 0 0 2024-04-12T05:08:48.604 INFO:tasks.workunit.client.0.smithi113.stdout:0/20: link d1/l5 d1/l7 0 2024-04-12T05:08:48.604 INFO:tasks.workunit.client.0.smithi113.stdout:7/31: chown d0/d2 22 1 2024-04-12T05:08:48.605 INFO:tasks.workunit.client.0.smithi113.stdout:7/32: dread - d0/f7 zero size 2024-04-12T05:08:48.607 INFO:tasks.workunit.client.0.smithi113.stdout:6/37: read d8/d9/fb [447497,81560] 0 2024-04-12T05:08:48.608 INFO:tasks.workunit.client.0.smithi113.stdout:4/29: dwrite f2 [4194304,4194304] 0 2024-04-12T05:08:48.609 INFO:tasks.workunit.client.0.smithi113.stdout:5/74: getdents d0/de/d13 0 2024-04-12T05:08:48.609 INFO:tasks.workunit.client.0.smithi113.stdout:4/30: chown c8 105633133 1 2024-04-12T05:08:48.609 INFO:tasks.workunit.client.0.smithi113.stdout:4/31: rmdir - no directory 2024-04-12T05:08:48.614 INFO:tasks.workunit.client.0.smithi113.stdout:0/21: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:48.614 INFO:tasks.workunit.client.0.smithi113.stdout:7/33: unlink d0/d2/l5 0 2024-04-12T05:08:48.620 INFO:tasks.workunit.client.0.smithi113.stdout:5/75: dwrite d0/d10/f12 [0,4194304] 0 2024-04-12T05:08:48.621 INFO:tasks.workunit.client.0.smithi113.stdout:0/22: unlink d1/f4 0 2024-04-12T05:08:48.622 INFO:tasks.workunit.client.0.smithi113.stdout:4/32: link l4 l9 0 2024-04-12T05:08:48.626 INFO:tasks.workunit.client.0.smithi113.stdout:5/76: fdatasync d0/f6 0 2024-04-12T05:08:48.629 INFO:tasks.workunit.client.0.smithi113.stdout:0/23: dread d1/f3 [0,4194304] 0 2024-04-12T05:08:48.629 INFO:tasks.workunit.client.0.smithi113.stdout:4/33: mkdir da 0 2024-04-12T05:08:48.631 INFO:tasks.workunit.client.0.smithi113.stdout:5/77: fdatasync d0/f14 0 2024-04-12T05:08:48.634 INFO:tasks.workunit.client.0.smithi113.stdout:5/78: unlink d0/de/l11 0 2024-04-12T05:08:48.634 INFO:tasks.workunit.client.0.smithi113.stdout:4/34: getdents da 0 2024-04-12T05:08:48.636 INFO:tasks.workunit.client.0.smithi113.stdout:0/24: creat d1/d6/f8 x:0 0 0 2024-04-12T05:08:48.636 INFO:tasks.workunit.client.0.smithi113.stdout:5/79: dread d0/fc [0,4194304] 0 2024-04-12T05:08:48.637 INFO:tasks.workunit.client.0.smithi113.stdout:5/80: chown d0/cd 1105499633 1 2024-04-12T05:08:48.637 INFO:tasks.workunit.client.0.smithi113.stdout:4/35: creat da/fb x:0 0 0 2024-04-12T05:08:48.638 INFO:tasks.workunit.client.0.smithi113.stdout:5/81: readlink d0/l9 0 2024-04-12T05:08:48.642 INFO:tasks.workunit.client.0.smithi113.stdout:5/82: dread d0/d10/f16 [0,4194304] 0 2024-04-12T05:08:48.644 INFO:tasks.workunit.client.0.smithi113.stdout:5/83: symlink d0/d10/l1c 0 2024-04-12T05:08:48.645 INFO:tasks.workunit.client.0.smithi113.stdout:5/84: fsync d0/de/f17 0 2024-04-12T05:08:48.648 INFO:tasks.workunit.client.0.smithi113.stdout:5/85: dread d0/d10/f16 [0,4194304] 0 2024-04-12T05:08:48.649 INFO:tasks.workunit.client.0.smithi113.stdout:5/86: write d0/d10/f12 [1079364,6648] 0 2024-04-12T05:08:48.651 INFO:tasks.workunit.client.0.smithi113.stdout:5/87: fdatasync d0/fc 0 2024-04-12T05:08:48.654 INFO:tasks.workunit.client.0.smithi113.stdout:5/88: dread d0/d10/f16 [0,4194304] 0 2024-04-12T05:08:48.656 INFO:tasks.workunit.client.0.smithi113.stdout:5/89: mkdir d0/de/d1d 0 2024-04-12T05:08:48.656 INFO:tasks.workunit.client.0.smithi113.stdout:5/90: readlink d0/d10/l1c 0 2024-04-12T05:08:48.746 INFO:tasks.workunit.client.0.smithi113.stdout:3/1: sync 2024-04-12T05:08:48.746 INFO:tasks.workunit.client.0.smithi113.stdout:3/2: dread - no filename 2024-04-12T05:08:48.749 INFO:tasks.workunit.client.0.smithi113.stdout:3/3: mkdir d0 0 2024-04-12T05:08:48.750 INFO:tasks.workunit.client.0.smithi113.stdout:8/40: truncate f2 668411 0 2024-04-12T05:08:48.751 INFO:tasks.workunit.client.0.smithi113.stdout:3/4: getdents d0 0 2024-04-12T05:08:48.751 INFO:tasks.workunit.client.0.smithi113.stdout:3/5: rename d0 to d0/d1 22 2024-04-12T05:08:48.751 INFO:tasks.workunit.client.0.smithi113.stdout:3/6: write - no filename 2024-04-12T05:08:48.751 INFO:tasks.workunit.client.0.smithi113.stdout:3/7: dread - no filename 2024-04-12T05:08:48.752 INFO:tasks.workunit.client.0.smithi113.stdout:8/41: symlink d9/la 0 2024-04-12T05:08:48.753 INFO:tasks.workunit.client.0.smithi113.stdout:3/8: getdents d0 0 2024-04-12T05:08:48.753 INFO:tasks.workunit.client.0.smithi113.stdout:3/9: dwrite - no filename 2024-04-12T05:08:48.753 INFO:tasks.workunit.client.0.smithi113.stdout:3/10: dread - no filename 2024-04-12T05:08:48.754 INFO:tasks.workunit.client.0.smithi113.stdout:8/42: truncate f8 748667 0 2024-04-12T05:08:48.756 INFO:tasks.workunit.client.0.smithi113.stdout:3/11: creat d0/f2 x:0 0 0 2024-04-12T05:08:48.757 INFO:tasks.workunit.client.0.smithi113.stdout:3/12: chown d0/f2 14011 1 2024-04-12T05:08:48.760 INFO:tasks.workunit.client.0.smithi113.stdout:3/13: link d0/f2 d0/f3 0 2024-04-12T05:08:48.761 INFO:tasks.workunit.client.0.smithi113.stdout:3/14: creat d0/f4 x:0 0 0 2024-04-12T05:08:48.762 INFO:tasks.workunit.client.0.smithi113.stdout:6/38: getdents d8/d9 0 2024-04-12T05:08:48.763 INFO:tasks.workunit.client.0.smithi113.stdout:6/39: chown f6 2 1 2024-04-12T05:08:48.763 INFO:tasks.workunit.client.0.smithi113.stdout:3/15: unlink d0/f3 0 2024-04-12T05:08:48.765 INFO:tasks.workunit.client.0.smithi113.stdout:6/40: unlink f6 0 2024-04-12T05:08:48.765 INFO:tasks.workunit.client.0.smithi113.stdout:6/41: truncate d8/d9/fb 1464559 0 2024-04-12T05:08:48.767 INFO:tasks.workunit.client.0.smithi113.stdout:6/42: mknod d8/cf 0 2024-04-12T05:08:48.768 INFO:tasks.workunit.client.0.smithi113.stdout:6/43: dread - d8/d9/fa zero size 2024-04-12T05:08:48.770 INFO:tasks.workunit.client.0.smithi113.stdout:7/34: fsync d0/f7 0 2024-04-12T05:08:48.771 INFO:tasks.workunit.client.0.smithi113.stdout:7/35: truncate d0/f7 62447 0 2024-04-12T05:08:48.837 INFO:tasks.workunit.client.0.smithi113.stdout:7/36: fdatasync d0/f7 0 2024-04-12T05:08:48.839 INFO:tasks.workunit.client.0.smithi113.stdout:6/44: dwrite d8/d9/fa [0,4194304] 0 2024-04-12T05:08:48.839 INFO:tasks.workunit.client.0.smithi113.stdout:4/36: rmdir da 39 2024-04-12T05:08:48.839 INFO:tasks.workunit.client.0.smithi113.stdout:5/91: rmdir d0 39 2024-04-12T05:08:48.840 INFO:tasks.workunit.client.0.smithi113.stdout:4/37: read f2 [5167307,72227] 0 2024-04-12T05:08:48.840 INFO:tasks.workunit.client.0.smithi113.stdout:8/43: dread f2 [0,4194304] 0 2024-04-12T05:08:48.842 INFO:tasks.workunit.client.0.smithi113.stdout:4/38: chown l9 1 1 2024-04-12T05:08:48.844 INFO:tasks.workunit.client.0.smithi113.stdout:6/45: creat d8/de/f10 x:0 0 0 2024-04-12T05:08:48.844 INFO:tasks.workunit.client.0.smithi113.stdout:7/37: dread d0/f7 [0,4194304] 0 2024-04-12T05:08:48.844 INFO:tasks.workunit.client.0.smithi113.stdout:6/46: chown d8/d9/fb 15 1 2024-04-12T05:08:48.845 INFO:tasks.workunit.client.0.smithi113.stdout:8/44: creat d9/fb x:0 0 0 2024-04-12T05:08:48.845 INFO:tasks.workunit.client.0.smithi113.stdout:7/38: dread d0/f7 [0,4194304] 0 2024-04-12T05:08:48.847 INFO:tasks.workunit.client.0.smithi113.stdout:7/39: write d0/f7 [804381,129175] 0 2024-04-12T05:08:48.849 INFO:tasks.workunit.client.0.smithi113.stdout:4/39: chown c7 27 1 2024-04-12T05:08:48.851 INFO:tasks.workunit.client.0.smithi113.stdout:8/45: fsync f8 0 2024-04-12T05:08:48.854 INFO:tasks.workunit.client.0.smithi113.stdout:8/46: rmdir d9 39 2024-04-12T05:08:48.856 INFO:tasks.workunit.client.0.smithi113.stdout:4/40: mknod da/cc 0 2024-04-12T05:08:48.856 INFO:tasks.workunit.client.0.smithi113.stdout:6/47: dwrite d8/d9/fc [0,4194304] 0 2024-04-12T05:08:48.860 INFO:tasks.workunit.client.0.smithi113.stdout:8/47: unlink f8 0 2024-04-12T05:08:48.860 INFO:tasks.workunit.client.0.smithi113.stdout:6/48: truncate d8/d9/fa 4720123 0 2024-04-12T05:08:48.861 INFO:tasks.workunit.client.0.smithi113.stdout:7/40: link d0/d2/l3 d0/l8 0 2024-04-12T05:08:48.862 INFO:tasks.workunit.client.0.smithi113.stdout:8/48: read f2 [229442,112493] 0 2024-04-12T05:08:48.863 INFO:tasks.workunit.client.0.smithi113.stdout:5/92: dwrite d0/f5 [0,4194304] 0 2024-04-12T05:08:48.864 INFO:tasks.workunit.client.0.smithi113.stdout:7/41: fdatasync d0/f7 0 2024-04-12T05:08:48.867 INFO:tasks.workunit.client.0.smithi113.stdout:9/22: sync 2024-04-12T05:08:48.869 INFO:tasks.workunit.client.0.smithi113.stdout:5/93: fdatasync d0/d10/f16 0 2024-04-12T05:08:48.869 INFO:tasks.workunit.client.0.smithi113.stdout:8/49: rename l7 to d9/lc 0 2024-04-12T05:08:48.872 INFO:tasks.workunit.client.0.smithi113.stdout:9/23: write d1/d4/d5/f7 [1869216,55531] 0 2024-04-12T05:08:48.872 INFO:tasks.workunit.client.0.smithi113.stdout:4/41: dwrite da/fb [0,4194304] 0 2024-04-12T05:08:48.872 INFO:tasks.workunit.client.0.smithi113.stdout:8/50: chown c0 217 1 2024-04-12T05:08:48.873 INFO:tasks.workunit.client.0.smithi113.stdout:5/94: write d0/f6 [3832994,23927] 0 2024-04-12T05:08:48.875 INFO:tasks.workunit.client.0.smithi113.stdout:7/42: truncate d0/f7 849879 0 2024-04-12T05:08:48.876 INFO:tasks.workunit.client.0.smithi113.stdout:4/42: stat da/cc 0 2024-04-12T05:08:48.876 INFO:tasks.workunit.client.0.smithi113.stdout:9/24: chown d1/d4/d5 442839 1 2024-04-12T05:08:48.878 INFO:tasks.workunit.client.0.smithi113.stdout:8/51: creat d9/fd x:0 0 0 2024-04-12T05:08:48.878 INFO:tasks.workunit.client.0.smithi113.stdout:5/95: write d0/f5 [3193488,21631] 0 2024-04-12T05:08:48.879 INFO:tasks.workunit.client.0.smithi113.stdout:6/49: dwrite d8/d9/fa [4194304,4194304] 0 2024-04-12T05:08:48.880 INFO:tasks.workunit.client.0.smithi113.stdout:4/43: mkdir da/dd 0 2024-04-12T05:08:48.881 INFO:tasks.workunit.client.0.smithi113.stdout:7/43: symlink d0/d2/d6/l9 0 2024-04-12T05:08:48.888 INFO:tasks.workunit.client.0.smithi113.stdout:9/25: link d1/d4/d5/la d1/lb 0 2024-04-12T05:08:48.891 INFO:tasks.workunit.client.0.smithi113.stdout:9/26: stat d1/d4 0 2024-04-12T05:08:48.894 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:47.539243+0000 mgr.smithi192.ocppyo (mgr.24357) 18 : audit [DBG] from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:48.894 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:47.925703+0000 mon.smithi113 (mon.0) 671 : audit [DBG] from='client.? 172.21.15.113:0/1931801007' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:08:48.894 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:08:48.240398+0000 mgr.smithi192.ocppyo (mgr.24357) 19 : audit [DBG] from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:48.900 INFO:tasks.workunit.client.0.smithi113.stdout:7/44: read d0/f7 [29801,56159] 0 2024-04-12T05:08:48.901 INFO:tasks.workunit.client.0.smithi113.stdout:6/50: dwrite d8/de/f10 [0,4194304] 0 2024-04-12T05:08:48.902 INFO:tasks.workunit.client.0.smithi113.stdout:8/52: dwrite d9/fd [0,4194304] 0 2024-04-12T05:08:48.902 INFO:tasks.workunit.client.0.smithi113.stdout:9/27: unlink d1/lb 0 2024-04-12T05:08:48.904 INFO:tasks.workunit.client.0.smithi113.stdout:4/44: dread f2 [0,4194304] 0 2024-04-12T05:08:48.904 INFO:tasks.workunit.client.0.smithi113.stdout:6/51: chown d8/d9 7 1 2024-04-12T05:08:48.906 INFO:tasks.workunit.client.0.smithi113.stdout:5/96: dwrite d0/d10/f19 [0,4194304] 0 2024-04-12T05:08:48.906 INFO:tasks.workunit.client.0.smithi113.stdout:4/45: chown c8 6 1 2024-04-12T05:08:48.907 INFO:tasks.workunit.client.0.smithi113.stdout:7/45: write d0/f7 [768357,40059] 0 2024-04-12T05:08:48.910 INFO:tasks.workunit.client.0.smithi113.stdout:9/28: chown d1/d4/d5/la 4 1 2024-04-12T05:08:48.910 INFO:tasks.workunit.client.0.smithi113.stdout:8/53: write f2 [95594,51555] 0 2024-04-12T05:08:48.910 INFO:tasks.workunit.client.0.smithi113.stdout:4/46: mkdir da/de 0 2024-04-12T05:08:48.910 INFO:tasks.workunit.client.0.smithi113.stdout:5/97: write d0/f5 [4470669,86541] 0 2024-04-12T05:08:48.912 INFO:tasks.workunit.client.0.smithi113.stdout:4/47: chown f2 89 1 2024-04-12T05:08:48.912 INFO:tasks.workunit.client.0.smithi113.stdout:5/98: readlink d0/de/lf 0 2024-04-12T05:08:48.914 INFO:tasks.workunit.client.0.smithi113.stdout:4/48: write da/fb [685448,66530] 0 2024-04-12T05:08:48.914 INFO:tasks.workunit.client.0.smithi113.stdout:9/29: creat d1/fc x:0 0 0 2024-04-12T05:08:48.916 INFO:tasks.workunit.client.0.smithi113.stdout:1/22: sync 2024-04-12T05:08:48.916 INFO:tasks.workunit.client.0.smithi113.stdout:0/25: sync 2024-04-12T05:08:48.916 INFO:tasks.workunit.client.0.smithi113.stdout:2/18: sync 2024-04-12T05:08:48.918 INFO:tasks.workunit.client.0.smithi113.stdout:1/23: chown f5 2209976 1 2024-04-12T05:08:48.919 INFO:tasks.workunit.client.0.smithi113.stdout:8/54: dread d9/fd [0,4194304] 0 2024-04-12T05:08:48.921 INFO:tasks.workunit.client.0.smithi113.stdout:1/24: chown c6 0 1 2024-04-12T05:08:48.921 INFO:tasks.workunit.client.0.smithi113.stdout:2/19: unlink f1 0 2024-04-12T05:08:48.921 INFO:tasks.workunit.client.0.smithi113.stdout:5/99: dread d0/d10/f19 [0,4194304] 0 2024-04-12T05:08:48.922 INFO:tasks.workunit.client.0.smithi113.stdout:4/49: mkdir da/dd/df 0 2024-04-12T05:08:48.926 INFO:tasks.workunit.client.0.smithi113.stdout:8/55: unlink c5 0 2024-04-12T05:08:48.926 INFO:tasks.workunit.client.0.smithi113.stdout:9/30: dwrite d1/d4/d5/f7 [0,4194304] 0 2024-04-12T05:08:48.927 INFO:tasks.workunit.client.0.smithi113.stdout:2/20: stat c2 0 2024-04-12T05:08:48.929 INFO:tasks.workunit.client.0.smithi113.stdout:9/31: dread - d1/fc zero size 2024-04-12T05:08:48.929 INFO:tasks.workunit.client.0.smithi113.stdout:2/21: chown c0 547149938 1 2024-04-12T05:08:48.929 INFO:tasks.workunit.client.0.smithi113.stdout:5/100: truncate d0/de/f18 877781 0 2024-04-12T05:08:48.930 INFO:tasks.workunit.client.0.smithi113.stdout:8/56: symlink d9/le 0 2024-04-12T05:08:48.931 INFO:tasks.workunit.client.0.smithi113.stdout:4/50: creat da/dd/df/f10 x:0 0 0 2024-04-12T05:08:48.932 INFO:tasks.workunit.client.0.smithi113.stdout:4/51: read f2 [7381347,125410] 0 2024-04-12T05:08:48.933 INFO:tasks.workunit.client.0.smithi113.stdout:2/22: creat f3 x:0 0 0 2024-04-12T05:08:48.933 INFO:tasks.workunit.client.0.smithi113.stdout:1/25: dwrite f5 [0,4194304] 0 2024-04-12T05:08:48.935 INFO:tasks.workunit.client.0.smithi113.stdout:2/23: truncate f3 1038586 0 2024-04-12T05:08:48.935 INFO:tasks.workunit.client.0.smithi113.stdout:8/57: mknod d9/cf 0 2024-04-12T05:08:48.937 INFO:tasks.workunit.client.0.smithi113.stdout:6/52: truncate d8/d9/fb 683536 0 2024-04-12T05:08:48.939 INFO:tasks.workunit.client.0.smithi113.stdout:6/53: readlink l3 0 2024-04-12T05:08:48.939 INFO:tasks.workunit.client.0.smithi113.stdout:4/52: fsync f2 0 2024-04-12T05:08:48.942 INFO:tasks.workunit.client.0.smithi113.stdout:6/54: chown d8/cf 143 1 2024-04-12T05:08:48.942 INFO:tasks.workunit.client.0.smithi113.stdout:5/101: creat d0/de/d1d/f1e x:0 0 0 2024-04-12T05:08:48.943 INFO:tasks.workunit.client.0.smithi113.stdout:2/24: symlink l4 0 2024-04-12T05:08:48.943 INFO:tasks.workunit.client.0.smithi113.stdout:1/26: dread f2 [0,4194304] 0 2024-04-12T05:08:48.944 INFO:tasks.workunit.client.0.smithi113.stdout:7/46: read d0/f7 [765162,33558] 0 2024-04-12T05:08:48.945 INFO:tasks.workunit.client.0.smithi113.stdout:8/58: creat d9/f10 x:0 0 0 2024-04-12T05:08:48.945 INFO:tasks.workunit.client.0.smithi113.stdout:9/32: dwrite d1/d4/d5/f7 [4194304,4194304] 0 2024-04-12T05:08:48.947 INFO:tasks.workunit.client.0.smithi113.stdout:0/26: write d1/f3 [1926089,41110] 0 2024-04-12T05:08:48.948 INFO:tasks.workunit.client.0.smithi113.stdout:6/55: write d8/d9/fc [368147,112492] 0 2024-04-12T05:08:48.948 INFO:tasks.workunit.client.0.smithi113.stdout:5/102: chown d0/d10/f19 3127 1 2024-04-12T05:08:48.950 INFO:tasks.workunit.client.0.smithi113.stdout:1/27: mknod c7 0 2024-04-12T05:08:48.954 INFO:tasks.workunit.client.0.smithi113.stdout:2/25: dread f3 [0,4194304] 0 2024-04-12T05:08:48.955 INFO:tasks.workunit.client.0.smithi113.stdout:8/59: write d9/fd [959627,93439] 0 2024-04-12T05:08:48.956 INFO:tasks.workunit.client.0.smithi113.stdout:2/26: chown f3 183 1 2024-04-12T05:08:48.957 INFO:tasks.workunit.client.0.smithi113.stdout:5/103: dread d0/fc [0,4194304] 0 2024-04-12T05:08:48.958 INFO:tasks.workunit.client.0.smithi113.stdout:4/53: dread f2 [4194304,4194304] 0 2024-04-12T05:08:48.960 INFO:tasks.workunit.client.0.smithi113.stdout:3/16: sync 2024-04-12T05:08:48.961 INFO:tasks.workunit.client.0.smithi113.stdout:8/60: mknod d9/c11 0 2024-04-12T05:08:48.962 INFO:tasks.workunit.client.0.smithi113.stdout:4/54: fsync da/dd/df/f10 0 2024-04-12T05:08:48.962 INFO:tasks.workunit.client.0.smithi113.stdout:3/17: read - d0/f4 zero size 2024-04-12T05:08:48.962 INFO:tasks.workunit.client.0.smithi113.stdout:0/27: write d1/f2 [4404180,104516] 0 2024-04-12T05:08:48.963 INFO:tasks.workunit.client.0.smithi113.stdout:5/104: creat d0/de/d13/f1f x:0 0 0 2024-04-12T05:08:48.963 INFO:tasks.workunit.client.0.smithi113.stdout:8/61: rename d9 to d9/d12 22 2024-04-12T05:08:48.965 INFO:tasks.workunit.client.0.smithi113.stdout:6/56: truncate d8/d9/fa 6942561 0 2024-04-12T05:08:48.967 INFO:tasks.workunit.client.0.smithi113.stdout:9/33: fsync d1/d4/d5/f7 0 2024-04-12T05:08:48.968 INFO:tasks.workunit.client.0.smithi113.stdout:4/55: rename c6 to da/c11 0 2024-04-12T05:08:48.969 INFO:tasks.workunit.client.0.smithi113.stdout:3/18: dread - d0/f2 zero size 2024-04-12T05:08:48.969 INFO:tasks.workunit.client.0.smithi113.stdout:5/105: read - d0/de/d1d/f1e zero size 2024-04-12T05:08:48.969 INFO:tasks.workunit.client.0.smithi113.stdout:8/62: mknod d9/c13 0 2024-04-12T05:08:48.970 INFO:tasks.workunit.client.0.smithi113.stdout:7/47: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:08:48.972 INFO:tasks.workunit.client.0.smithi113.stdout:3/19: read - d0/f4 zero size 2024-04-12T05:08:48.974 INFO:tasks.workunit.client.0.smithi113.stdout:5/106: dread - d0/d10/f1b zero size 2024-04-12T05:08:48.975 INFO:tasks.workunit.client.0.smithi113.stdout:2/27: dwrite f3 [0,4194304] 0 2024-04-12T05:08:48.976 INFO:tasks.workunit.client.0.smithi113.stdout:3/20: dread - d0/f2 zero size 2024-04-12T05:08:48.977 INFO:tasks.workunit.client.0.smithi113.stdout:2/28: stat c2 0 2024-04-12T05:08:48.979 INFO:tasks.workunit.client.0.smithi113.stdout:3/21: stat d0 0 2024-04-12T05:08:48.980 INFO:tasks.workunit.client.0.smithi113.stdout:9/34: mknod d1/d4/d5/d6/cd 0 2024-04-12T05:08:48.984 INFO:tasks.workunit.client.0.smithi113.stdout:3/22: mknod d0/c5 0 2024-04-12T05:08:48.986 INFO:tasks.workunit.client.0.smithi113.stdout:0/28: dwrite d1/f3 [0,4194304] 0 2024-04-12T05:08:48.986 INFO:tasks.workunit.client.0.smithi113.stdout:5/107: truncate d0/d10/f1b 515424 0 2024-04-12T05:08:48.988 INFO:tasks.workunit.client.0.smithi113.stdout:9/35: symlink d1/d4/le 0 2024-04-12T05:08:48.990 INFO:tasks.workunit.client.0.smithi113.stdout:3/23: creat d0/f6 x:0 0 0 2024-04-12T05:08:48.990 INFO:tasks.workunit.client.0.smithi113.stdout:9/36: dread - d1/fc zero size 2024-04-12T05:08:48.991 INFO:tasks.workunit.client.0.smithi113.stdout:4/56: link da/c11 da/dd/df/c12 0 2024-04-12T05:08:48.991 INFO:tasks.workunit.client.0.smithi113.stdout:7/48: dread d0/f7 [0,4194304] 0 2024-04-12T05:08:48.993 INFO:tasks.workunit.client.0.smithi113.stdout:8/63: dwrite d9/f10 [0,4194304] 0 2024-04-12T05:08:48.994 INFO:tasks.workunit.client.0.smithi113.stdout:0/29: mkdir d1/d9 0 2024-04-12T05:08:48.995 INFO:tasks.workunit.client.0.smithi113.stdout:9/37: dread - d1/fc zero size 2024-04-12T05:08:48.997 INFO:tasks.workunit.client.0.smithi113.stdout:3/24: creat d0/f7 x:0 0 0 2024-04-12T05:08:49.001 INFO:tasks.workunit.client.0.smithi113.stdout:7/49: creat d0/d2/d6/fa x:0 0 0 2024-04-12T05:08:49.001 INFO:tasks.workunit.client.0.smithi113.stdout:2/29: truncate f3 2386253 0 2024-04-12T05:08:49.002 INFO:tasks.workunit.client.0.smithi113.stdout:0/30: mkdir d1/d6/da 0 2024-04-12T05:08:49.002 INFO:tasks.workunit.client.0.smithi113.stdout:2/30: stat c0 0 2024-04-12T05:08:49.004 INFO:tasks.workunit.client.0.smithi113.stdout:0/31: fdatasync d1/f2 0 2024-04-12T05:08:49.005 INFO:tasks.workunit.client.0.smithi113.stdout:5/108: dwrite d0/de/d13/f1f [0,4194304] 0 2024-04-12T05:08:49.006 INFO:tasks.workunit.client.0.smithi113.stdout:5/109: stat d0/d10 0 2024-04-12T05:08:49.008 INFO:tasks.workunit.client.0.smithi113.stdout:4/57: dread da/fb [0,4194304] 0 2024-04-12T05:08:49.009 INFO:tasks.workunit.client.0.smithi113.stdout:9/38: dread d1/d4/d5/f7 [4194304,4194304] 0 2024-04-12T05:08:49.010 INFO:tasks.workunit.client.0.smithi113.stdout:9/39: chown d1/d4/d5 661 1 2024-04-12T05:08:49.012 INFO:tasks.workunit.client.0.smithi113.stdout:5/110: dread d0/f14 [0,4194304] 0 2024-04-12T05:08:49.013 INFO:tasks.workunit.client.0.smithi113.stdout:4/58: dread da/fb [0,4194304] 0 2024-04-12T05:08:49.016 INFO:tasks.workunit.client.0.smithi113.stdout:5/111: dread d0/f6 [0,4194304] 0 2024-04-12T05:08:49.018 INFO:tasks.workunit.client.0.smithi113.stdout:3/25: creat d0/f8 x:0 0 0 2024-04-12T05:08:49.023 INFO:tasks.workunit.client.0.smithi113.stdout:7/50: mknod d0/d2/cb 0 2024-04-12T05:08:49.025 INFO:tasks.workunit.client.0.smithi113.stdout:2/31: creat f5 x:0 0 0 2024-04-12T05:08:49.026 INFO:tasks.workunit.client.0.smithi113.stdout:0/32: creat d1/fb x:0 0 0 2024-04-12T05:08:49.027 INFO:tasks.workunit.client.0.smithi113.stdout:0/33: truncate d1/fb 424804 0 2024-04-12T05:08:49.031 INFO:tasks.workunit.client.0.smithi113.stdout:4/59: creat da/dd/df/f13 x:0 0 0 2024-04-12T05:08:49.035 INFO:tasks.workunit.client.0.smithi113.stdout:3/26: creat d0/f9 x:0 0 0 2024-04-12T05:08:49.035 INFO:tasks.workunit.client.0.smithi113.stdout:0/34: dwrite d1/d6/f8 [0,4194304] 0 2024-04-12T05:08:49.041 INFO:tasks.workunit.client.0.smithi113.stdout:8/64: getdents d9 0 2024-04-12T05:08:49.046 INFO:tasks.workunit.client.0.smithi113.stdout:4/60: dwrite da/dd/df/f10 [0,4194304] 0 2024-04-12T05:08:49.050 INFO:tasks.workunit.client.0.smithi113.stdout:8/65: symlink d9/l14 0 2024-04-12T05:08:49.052 INFO:tasks.workunit.client.0.smithi113.stdout:4/61: write da/dd/df/f10 [3033556,106232] 0 2024-04-12T05:08:49.053 INFO:tasks.workunit.client.0.smithi113.stdout:5/112: getdents d0 0 2024-04-12T05:08:49.054 INFO:tasks.workunit.client.0.smithi113.stdout:0/35: dwrite d1/d6/f8 [0,4194304] 0 2024-04-12T05:08:49.056 INFO:tasks.workunit.client.0.smithi113.stdout:2/32: dwrite f3 [0,4194304] 0 2024-04-12T05:08:49.059 INFO:tasks.workunit.client.0.smithi113.stdout:5/113: rmdir d0 39 2024-04-12T05:08:49.064 INFO:tasks.workunit.client.0.smithi113.stdout:4/62: mkdir da/dd/df/d14 0 2024-04-12T05:08:49.064 INFO:tasks.workunit.client.0.smithi113.stdout:3/27: write d0/f2 [515004,88247] 0 2024-04-12T05:08:49.065 INFO:tasks.workunit.client.0.smithi113.stdout:4/63: readlink l0 0 2024-04-12T05:08:49.065 INFO:tasks.workunit.client.0.smithi113.stdout:7/51: rename d0/d2 to d0/dc 0 2024-04-12T05:08:49.067 INFO:tasks.workunit.client.0.smithi113.stdout:4/64: readlink l0 0 2024-04-12T05:08:49.068 INFO:tasks.workunit.client.0.smithi113.stdout:9/40: mkdir d1/df 0 2024-04-12T05:08:49.068 INFO:tasks.workunit.client.0.smithi113.stdout:0/36: symlink d1/d6/da/lc 0 2024-04-12T05:08:49.070 INFO:tasks.workunit.client.0.smithi113.stdout:7/52: readlink d0/dc/l4 0 2024-04-12T05:08:49.070 INFO:tasks.workunit.client.0.smithi113.stdout:3/28: write d0/f9 [657601,63134] 0 2024-04-12T05:08:49.070 INFO:tasks.workunit.client.0.smithi113.stdout:5/114: mknod d0/c20 0 2024-04-12T05:08:49.071 INFO:tasks.workunit.client.0.smithi113.stdout:9/41: fdatasync d1/fc 0 2024-04-12T05:08:49.072 INFO:tasks.workunit.client.0.smithi113.stdout:8/66: dwrite d9/fb [0,4194304] 0 2024-04-12T05:08:49.076 INFO:tasks.workunit.client.0.smithi113.stdout:3/29: truncate d0/f4 507046 0 2024-04-12T05:08:49.076 INFO:tasks.workunit.client.0.smithi113.stdout:0/37: unlink d1/l5 0 2024-04-12T05:08:49.078 INFO:tasks.workunit.client.0.smithi113.stdout:2/33: dwrite f5 [0,4194304] 0 2024-04-12T05:08:49.079 INFO:tasks.workunit.client.0.smithi113.stdout:9/42: mknod d1/c10 0 2024-04-12T05:08:49.081 INFO:tasks.workunit.client.0.smithi113.stdout:0/38: fdatasync d1/fb 0 2024-04-12T05:08:49.081 INFO:tasks.workunit.client.0.smithi113.stdout:3/30: write d0/f9 [75255,81340] 0 2024-04-12T05:08:49.088 INFO:tasks.workunit.client.0.smithi113.stdout:4/65: dwrite da/fb [0,4194304] 0 2024-04-12T05:08:49.089 INFO:tasks.workunit.client.0.smithi113.stdout:0/39: write d1/f2 [4233734,31128] 0 2024-04-12T05:08:49.092 INFO:tasks.workunit.client.0.smithi113.stdout:3/31: rename d0/f4 to d0/fa 0 2024-04-12T05:08:49.094 INFO:tasks.workunit.client.0.smithi113.stdout:8/67: dwrite d9/f10 [0,4194304] 0 2024-04-12T05:08:49.094 INFO:tasks.workunit.client.0.smithi113.stdout:5/115: creat d0/de/d1d/f21 x:0 0 0 2024-04-12T05:08:49.095 INFO:tasks.workunit.client.0.smithi113.stdout:4/66: mknod da/dd/c15 0 2024-04-12T05:08:49.095 INFO:tasks.workunit.client.0.smithi113.stdout:8/68: chown c0 7213065 1 2024-04-12T05:08:49.096 INFO:tasks.workunit.client.0.smithi113.stdout:2/34: fsync f5 0 2024-04-12T05:08:49.097 INFO:tasks.workunit.client.0.smithi113.stdout:7/53: truncate d0/f7 2395450 0 2024-04-12T05:08:49.099 INFO:tasks.workunit.client.0.smithi113.stdout:3/32: mknod d0/cb 0 2024-04-12T05:08:49.100 INFO:tasks.workunit.client.0.smithi113.stdout:9/43: dwrite d1/fc [0,4194304] 0 2024-04-12T05:08:49.101 INFO:tasks.workunit.client.0.smithi113.stdout:3/33: read - d0/f7 zero size 2024-04-12T05:08:49.101 INFO:tasks.workunit.client.0.smithi113.stdout:7/54: write d0/dc/d6/fa [841666,39471] 0 2024-04-12T05:08:49.103 INFO:tasks.workunit.client.0.smithi113.stdout:5/116: creat d0/de/f22 x:0 0 0 2024-04-12T05:08:49.103 INFO:tasks.workunit.client.0.smithi113.stdout:3/34: write d0/f7 [640605,112837] 0 2024-04-12T05:08:49.104 INFO:tasks.workunit.client.0.smithi113.stdout:5/117: read - d0/de/f22 zero size 2024-04-12T05:08:49.105 INFO:tasks.workunit.client.0.smithi113.stdout:5/118: dread - d0/de/f17 zero size 2024-04-12T05:08:49.106 INFO:tasks.workunit.client.0.smithi113.stdout:2/35: mkdir d6 0 2024-04-12T05:08:49.109 INFO:tasks.workunit.client.0.smithi113.stdout:0/40: fdatasync d1/d6/f8 0 2024-04-12T05:08:49.110 INFO:tasks.workunit.client.0.smithi113.stdout:3/35: read d0/f2 [448890,21567] 0 2024-04-12T05:08:49.110 INFO:tasks.workunit.client.0.smithi113.stdout:7/55: mkdir d0/dd 0 2024-04-12T05:08:49.111 INFO:tasks.workunit.client.0.smithi113.stdout:2/36: dread f3 [0,4194304] 0 2024-04-12T05:08:49.113 INFO:tasks.workunit.client.0.smithi113.stdout:9/44: creat d1/df/f11 x:0 0 0 2024-04-12T05:08:49.116 INFO:tasks.workunit.client.0.smithi113.stdout:7/56: mknod d0/dc/ce 0 2024-04-12T05:08:49.118 INFO:tasks.workunit.client.0.smithi113.stdout:4/67: fdatasync da/fb 0 2024-04-12T05:08:49.119 INFO:tasks.workunit.client.0.smithi113.stdout:3/36: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:08:49.120 INFO:tasks.workunit.client.0.smithi113.stdout:0/41: getdents d1/d9 0 2024-04-12T05:08:49.120 INFO:tasks.workunit.client.0.smithi113.stdout:4/68: chown da/fb 2930720 1 2024-04-12T05:08:49.122 INFO:tasks.workunit.client.0.smithi113.stdout:7/57: rename d0/dc/cb to d0/dd/cf 0 2024-04-12T05:08:49.123 INFO:tasks.workunit.client.0.smithi113.stdout:9/45: dread d1/d4/d5/f7 [4194304,4194304] 0 2024-04-12T05:08:49.123 INFO:tasks.workunit.client.0.smithi113.stdout:3/37: symlink d0/lc 0 2024-04-12T05:08:49.124 INFO:tasks.workunit.client.0.smithi113.stdout:2/37: dread f3 [0,4194304] 0 2024-04-12T05:08:49.125 INFO:tasks.workunit.client.0.smithi113.stdout:0/42: unlink d1/d6/f8 0 2024-04-12T05:08:49.126 INFO:tasks.workunit.client.0.smithi113.stdout:4/69: mknod da/dd/df/c16 0 2024-04-12T05:08:49.127 INFO:tasks.workunit.client.0.smithi113.stdout:8/69: truncate d9/fb 3895727 0 2024-04-12T05:08:49.127 INFO:tasks.workunit.client.0.smithi113.stdout:1/28: sync 2024-04-12T05:08:49.127 INFO:tasks.workunit.client.0.smithi113.stdout:6/57: sync 2024-04-12T05:08:49.129 INFO:tasks.workunit.client.0.smithi113.stdout:7/58: rename d0/dc/d6/l9 to d0/l10 0 2024-04-12T05:08:49.132 INFO:tasks.workunit.client.0.smithi113.stdout:3/38: truncate d0/f2 132240 0 2024-04-12T05:08:49.133 INFO:tasks.workunit.client.0.smithi113.stdout:9/46: dwrite d1/d4/d5/f7 [0,4194304] 0 2024-04-12T05:08:49.133 INFO:tasks.workunit.client.0.smithi113.stdout:2/38: dwrite f5 [0,4194304] 0 2024-04-12T05:08:49.135 INFO:tasks.workunit.client.0.smithi113.stdout:6/58: mkdir d8/d9/d11 0 2024-04-12T05:08:49.135 INFO:tasks.workunit.client.0.smithi113.stdout:9/47: write d1/fc [2069504,21572] 0 2024-04-12T05:08:49.135 INFO:tasks.workunit.client.0.smithi113.stdout:5/119: write d0/de/f18 [1386665,76871] 0 2024-04-12T05:08:49.138 INFO:tasks.workunit.client.0.smithi113.stdout:4/70: mkdir da/de/d17 0 2024-04-12T05:08:49.140 INFO:tasks.workunit.client.0.smithi113.stdout:9/48: write d1/d4/d5/f7 [7921147,53007] 0 2024-04-12T05:08:49.140 INFO:tasks.workunit.client.0.smithi113.stdout:3/39: creat d0/fd x:0 0 0 2024-04-12T05:08:49.149 INFO:tasks.workunit.client.0.smithi113.stdout:6/59: creat d8/f12 x:0 0 0 2024-04-12T05:08:49.151 INFO:tasks.workunit.client.0.smithi113.stdout:8/70: dread d9/fb [0,4194304] 0 2024-04-12T05:08:49.154 INFO:tasks.workunit.client.0.smithi113.stdout:9/49: creat d1/d4/d5/f12 x:0 0 0 2024-04-12T05:08:49.155 INFO:tasks.workunit.client.0.smithi113.stdout:8/71: dread d9/fb [0,4194304] 0 2024-04-12T05:08:49.155 INFO:tasks.workunit.client.0.smithi113.stdout:3/40: unlink d0/cb 0 2024-04-12T05:08:49.156 INFO:tasks.workunit.client.0.smithi113.stdout:8/72: chown l3 14268183 1 2024-04-12T05:08:49.159 INFO:tasks.workunit.client.0.smithi113.stdout:1/29: truncate f5 57356 0 2024-04-12T05:08:49.160 INFO:tasks.workunit.client.0.smithi113.stdout:1/30: chown c7 1 1 2024-04-12T05:08:49.161 INFO:tasks.workunit.client.0.smithi113.stdout:4/71: creat da/dd/df/d14/f18 x:0 0 0 2024-04-12T05:08:49.161 INFO:tasks.workunit.client.0.smithi113.stdout:9/50: dwrite d1/fc [0,4194304] 0 2024-04-12T05:08:49.162 INFO:tasks.workunit.client.0.smithi113.stdout:3/41: symlink d0/le 0 2024-04-12T05:08:49.163 INFO:tasks.workunit.client.0.smithi113.stdout:2/39: rmdir d6 0 2024-04-12T05:08:49.164 INFO:tasks.workunit.client.0.smithi113.stdout:9/51: write d1/fc [1031980,20604] 0 2024-04-12T05:08:49.167 INFO:tasks.workunit.client.0.smithi113.stdout:1/31: fdatasync f2 0 2024-04-12T05:08:49.168 INFO:tasks.workunit.client.0.smithi113.stdout:9/52: chown d1/d4/d5/f7 1993622 1 2024-04-12T05:08:49.171 INFO:tasks.workunit.client.0.smithi113.stdout:4/72: creat da/dd/df/d14/f19 x:0 0 0 2024-04-12T05:08:49.172 INFO:tasks.workunit.client.0.smithi113.stdout:2/40: rename f5 to f7 0 2024-04-12T05:08:49.172 INFO:tasks.workunit.client.0.smithi113.stdout:5/120: dwrite d0/d10/f19 [0,4194304] 0 2024-04-12T05:08:49.175 INFO:tasks.workunit.client.0.smithi113.stdout:9/53: write d1/d4/d5/f12 [232310,31533] 0 2024-04-12T05:08:49.176 INFO:tasks.workunit.client.0.smithi113.stdout:6/60: write d8/d9/fb [822075,104854] 0 2024-04-12T05:08:49.179 INFO:tasks.workunit.client.0.smithi113.stdout:3/42: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:08:49.182 INFO:tasks.workunit.client.0.smithi113.stdout:8/73: getdents d9 0 2024-04-12T05:08:49.182 INFO:tasks.workunit.client.0.smithi113.stdout:9/54: dread d1/d4/d5/f12 [0,4194304] 0 2024-04-12T05:08:49.183 INFO:tasks.workunit.client.0.smithi113.stdout:8/74: fdatasync d9/fd 0 2024-04-12T05:08:49.184 INFO:tasks.workunit.client.0.smithi113.stdout:8/75: fdatasync f2 0 2024-04-12T05:08:49.184 INFO:tasks.workunit.client.0.smithi113.stdout:6/61: rename d8/d9 to d8/de/d13 0 2024-04-12T05:08:49.185 INFO:tasks.workunit.client.0.smithi113.stdout:9/55: dread d1/d4/d5/f12 [0,4194304] 0 2024-04-12T05:08:49.185 INFO:tasks.workunit.client.0.smithi113.stdout:3/43: creat d0/ff x:0 0 0 2024-04-12T05:08:49.190 INFO:tasks.workunit.client.0.smithi113.stdout:6/62: read d8/de/d13/fa [117750,63834] 0 2024-04-12T05:08:49.191 INFO:tasks.workunit.client.0.smithi113.stdout:3/44: creat d0/f10 x:0 0 0 2024-04-12T05:08:49.192 INFO:tasks.workunit.client.0.smithi113.stdout:5/121: dwrite d0/d10/f16 [0,4194304] 0 2024-04-12T05:08:49.192 INFO:tasks.workunit.client.0.smithi113.stdout:9/56: creat d1/d4/f13 x:0 0 0 2024-04-12T05:08:49.193 INFO:tasks.workunit.client.0.smithi113.stdout:2/41: getdents . 0 2024-04-12T05:08:49.195 INFO:tasks.workunit.client.0.smithi113.stdout:3/45: mknod d0/c11 0 2024-04-12T05:08:49.196 INFO:tasks.workunit.client.0.smithi113.stdout:5/122: read d0/f14 [2167774,97754] 0 2024-04-12T05:08:49.197 INFO:tasks.workunit.client.0.smithi113.stdout:2/42: mkdir d8 0 2024-04-12T05:08:49.197 INFO:tasks.workunit.client.0.smithi113.stdout:6/63: dread d8/de/f10 [0,4194304] 0 2024-04-12T05:08:49.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:47.539243+0000 mgr.smithi192.ocppyo (mgr.24357) 18 : audit [DBG] from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:49.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:47.925703+0000 mon.smithi113 (mon.0) 671 : audit [DBG] from='client.? 172.21.15.113:0/1931801007' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:08:49.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:08:48.240398+0000 mgr.smithi192.ocppyo (mgr.24357) 19 : audit [DBG] from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:08:49.198 INFO:tasks.workunit.client.0.smithi113.stdout:3/46: creat d0/f12 x:0 0 0 2024-04-12T05:08:49.198 INFO:tasks.workunit.client.0.smithi113.stdout:6/64: readlink l2 0 2024-04-12T05:08:49.198 INFO:tasks.workunit.client.0.smithi113.stdout:9/57: creat d1/f14 x:0 0 0 2024-04-12T05:08:49.199 INFO:tasks.workunit.client.0.smithi113.stdout:2/43: write f7 [1853594,118377] 0 2024-04-12T05:08:49.200 INFO:tasks.workunit.client.0.smithi113.stdout:3/47: truncate d0/ff 950873 0 2024-04-12T05:08:49.200 INFO:tasks.workunit.client.0.smithi113.stdout:3/48: stat d0 0 2024-04-12T05:08:49.201 INFO:tasks.workunit.client.0.smithi113.stdout:6/65: mknod d8/c14 0 2024-04-12T05:08:49.201 INFO:tasks.workunit.client.0.smithi113.stdout:3/49: chown d0 4150868 1 2024-04-12T05:08:49.204 INFO:tasks.workunit.client.0.smithi113.stdout:5/123: link d0/de/d1d/f21 d0/de/d13/f23 0 2024-04-12T05:08:49.205 INFO:tasks.workunit.client.0.smithi113.stdout:3/50: mkdir d0/d13 0 2024-04-12T05:08:49.206 INFO:tasks.workunit.client.0.smithi113.stdout:2/44: mkdir d8/d9 0 2024-04-12T05:08:49.207 INFO:tasks.workunit.client.0.smithi113.stdout:6/66: creat d8/de/f15 x:0 0 0 2024-04-12T05:08:49.208 INFO:tasks.workunit.client.0.smithi113.stdout:9/58: dwrite d1/d4/f13 [0,4194304] 0 2024-04-12T05:08:49.208 INFO:tasks.workunit.client.0.smithi113.stdout:6/67: dread - d8/f12 zero size 2024-04-12T05:08:49.209 INFO:tasks.workunit.client.0.smithi113.stdout:9/59: dread - d1/f14 zero size 2024-04-12T05:08:49.212 INFO:tasks.workunit.client.0.smithi113.stdout:2/45: unlink c0 0 2024-04-12T05:08:49.212 INFO:tasks.workunit.client.0.smithi113.stdout:5/124: dread d0/d10/f12 [0,4194304] 0 2024-04-12T05:08:49.213 INFO:tasks.workunit.client.0.smithi113.stdout:8/76: write d9/fb [1288400,53082] 0 2024-04-12T05:08:49.213 INFO:tasks.workunit.client.0.smithi113.stdout:3/51: symlink d0/d13/l14 0 2024-04-12T05:08:49.215 INFO:tasks.workunit.client.0.smithi113.stdout:1/32: truncate f5 593142 0 2024-04-12T05:08:49.216 INFO:tasks.workunit.client.0.smithi113.stdout:8/77: chown l3 239024935 1 2024-04-12T05:08:49.218 INFO:tasks.workunit.client.0.smithi113.stdout:8/78: write d9/f10 [4423559,120742] 0 2024-04-12T05:08:49.219 INFO:tasks.workunit.client.0.smithi113.stdout:8/79: write f2 [304769,21711] 0 2024-04-12T05:08:49.220 INFO:tasks.workunit.client.0.smithi113.stdout:5/125: unlink d0/de/d13/f1f 0 2024-04-12T05:08:49.221 INFO:tasks.workunit.client.0.smithi113.stdout:2/46: dwrite f7 [0,4194304] 0 2024-04-12T05:08:49.223 INFO:tasks.workunit.client.0.smithi113.stdout:1/33: mknod c8 0 2024-04-12T05:08:49.225 INFO:tasks.workunit.client.0.smithi113.stdout:3/52: mkdir d0/d13/d15 0 2024-04-12T05:08:49.226 INFO:tasks.workunit.client.0.smithi113.stdout:5/126: dread d0/d10/f16 [0,4194304] 0 2024-04-12T05:08:49.229 INFO:tasks.workunit.client.0.smithi113.stdout:8/80: fdatasync d9/fb 0 2024-04-12T05:08:49.230 INFO:tasks.workunit.client.0.smithi113.stdout:3/53: dread d0/f7 [0,4194304] 0 2024-04-12T05:08:49.230 INFO:tasks.workunit.client.0.smithi113.stdout:8/81: stat d9/f10 0 2024-04-12T05:08:49.231 INFO:tasks.workunit.client.0.smithi113.stdout:2/47: rename f3 to d8/d9/fa 0 2024-04-12T05:08:49.231 INFO:tasks.workunit.client.0.smithi113.stdout:3/54: chown d0/d13 469340623 1 2024-04-12T05:08:49.233 INFO:tasks.workunit.client.0.smithi113.stdout:2/48: chown d8/d9 28167 1 2024-04-12T05:08:49.233 INFO:tasks.workunit.client.0.smithi113.stdout:3/55: chown d0/f12 40580 1 2024-04-12T05:08:49.234 INFO:tasks.workunit.client.0.smithi113.stdout:5/127: symlink d0/d10/l24 0 2024-04-12T05:08:49.235 INFO:tasks.workunit.client.0.smithi113.stdout:3/56: truncate d0/fd 123107 0 2024-04-12T05:08:49.237 INFO:tasks.workunit.client.0.smithi113.stdout:9/60: dwrite d1/d4/f13 [4194304,4194304] 0 2024-04-12T05:08:49.239 INFO:tasks.workunit.client.0.smithi113.stdout:6/68: truncate d8/de/f10 1728328 0 2024-04-12T05:08:49.240 INFO:tasks.workunit.client.0.smithi113.stdout:6/69: chown d8 77 1 2024-04-12T05:08:49.243 INFO:tasks.workunit.client.0.smithi113.stdout:2/49: rmdir d8/d9 39 2024-04-12T05:08:49.254 INFO:tasks.workunit.client.0.smithi113.stdout:9/61: mkdir d1/d4/d5/d6/d15 0 2024-04-12T05:08:49.255 INFO:tasks.workunit.client.0.smithi113.stdout:9/62: dread d1/d4/d5/f12 [0,4194304] 0 2024-04-12T05:08:49.258 INFO:tasks.workunit.client.0.smithi113.stdout:6/70: creat d8/de/f16 x:0 0 0 2024-04-12T05:08:49.260 INFO:tasks.workunit.client.0.smithi113.stdout:6/71: write d8/de/d13/fb [1674313,5801] 0 2024-04-12T05:08:49.260 INFO:tasks.workunit.client.0.smithi113.stdout:3/57: truncate d0/f2 644719 0 2024-04-12T05:08:49.263 INFO:tasks.workunit.client.0.smithi113.stdout:9/63: rmdir d1/d4/d5/d6 39 2024-04-12T05:08:49.264 INFO:tasks.workunit.client.0.smithi113.stdout:3/58: truncate d0/f9 1422698 0 2024-04-12T05:08:49.265 INFO:tasks.workunit.client.0.smithi113.stdout:8/82: truncate f2 349043 0 2024-04-12T05:08:49.266 INFO:tasks.workunit.client.0.smithi113.stdout:1/34: dwrite f5 [0,4194304] 0 2024-04-12T05:08:49.266 INFO:tasks.workunit.client.0.smithi113.stdout:6/72: mkdir d8/de/d17 0 2024-04-12T05:08:49.276 INFO:tasks.workunit.client.0.smithi113.stdout:5/128: truncate d0/d10/f19 1998105 0 2024-04-12T05:08:49.276 INFO:tasks.workunit.client.0.smithi113.stdout:2/50: dwrite f7 [0,4194304] 0 2024-04-12T05:08:49.277 INFO:tasks.workunit.client.0.smithi113.stdout:9/64: mkdir d1/d4/d16 0 2024-04-12T05:08:49.279 INFO:tasks.workunit.client.0.smithi113.stdout:5/129: chown d0/de/d1d/f21 19495913 1 2024-04-12T05:08:49.283 INFO:tasks.workunit.client.0.smithi113.stdout:6/73: unlink d8/de/f10 0 2024-04-12T05:08:49.284 INFO:tasks.workunit.client.0.smithi113.stdout:9/65: mkdir d1/d4/d5/d17 0 2024-04-12T05:08:49.286 INFO:tasks.workunit.client.0.smithi113.stdout:2/51: chown d8/d9 576851550 1 2024-04-12T05:08:49.286 INFO:tasks.workunit.client.0.smithi113.stdout:6/74: dread - d8/de/f16 zero size 2024-04-12T05:08:49.286 INFO:tasks.workunit.client.0.smithi113.stdout:8/83: dread d9/fb [0,4194304] 0 2024-04-12T05:08:49.287 INFO:tasks.workunit.client.0.smithi113.stdout:3/59: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:08:49.289 INFO:tasks.workunit.client.0.smithi113.stdout:9/66: creat d1/d4/f18 x:0 0 0 2024-04-12T05:08:49.289 INFO:tasks.workunit.client.0.smithi113.stdout:8/84: chown c4 3 1 2024-04-12T05:08:49.291 INFO:tasks.workunit.client.0.smithi113.stdout:3/60: symlink d0/l16 0 2024-04-12T05:08:49.292 INFO:tasks.workunit.client.0.smithi113.stdout:6/75: mknod d8/de/c18 0 2024-04-12T05:08:49.292 INFO:tasks.workunit.client.0.smithi113.stdout:1/35: dwrite f2 [0,4194304] 0 2024-04-12T05:08:49.292 INFO:tasks.workunit.client.0.smithi113.stdout:9/67: mknod d1/c19 0 2024-04-12T05:08:49.294 INFO:tasks.workunit.client.0.smithi113.stdout:5/130: dread d0/d10/f16 [0,4194304] 0 2024-04-12T05:08:49.294 INFO:tasks.workunit.client.0.smithi113.stdout:6/76: chown d8/de/c18 1805292 1 2024-04-12T05:08:49.295 INFO:tasks.workunit.client.0.smithi113.stdout:1/36: mknod c9 0 2024-04-12T05:08:49.297 INFO:tasks.workunit.client.0.smithi113.stdout:1/37: write f2 [2015014,105993] 0 2024-04-12T05:08:49.299 INFO:tasks.workunit.client.0.smithi113.stdout:1/38: fdatasync f5 0 2024-04-12T05:08:49.300 INFO:tasks.workunit.client.0.smithi113.stdout:5/131: creat d0/de/f25 x:0 0 0 2024-04-12T05:08:49.300 INFO:tasks.workunit.client.0.smithi113.stdout:8/85: dread d9/fd [0,4194304] 0 2024-04-12T05:08:49.303 INFO:tasks.workunit.client.0.smithi113.stdout:1/39: creat fa x:0 0 0 2024-04-12T05:08:49.304 INFO:tasks.workunit.client.0.smithi113.stdout:5/132: dread - d0/de/f17 zero size 2024-04-12T05:08:49.305 INFO:tasks.workunit.client.0.smithi113.stdout:8/86: mknod d9/c15 0 2024-04-12T05:08:49.306 INFO:tasks.workunit.client.0.smithi113.stdout:3/61: dwrite d0/f10 [0,4194304] 0 2024-04-12T05:08:49.308 INFO:tasks.workunit.client.0.smithi113.stdout:8/87: write d9/fb [1389635,105125] 0 2024-04-12T05:08:49.308 INFO:tasks.workunit.client.0.smithi113.stdout:1/40: write fa [258729,108913] 0 2024-04-12T05:08:49.309 INFO:tasks.workunit.client.0.smithi113.stdout:1/41: rmdir - no directory 2024-04-12T05:08:49.309 INFO:tasks.workunit.client.0.smithi113.stdout:5/133: fsync d0/de/d1d/f1e 0 2024-04-12T05:08:49.312 INFO:tasks.workunit.client.0.smithi113.stdout:3/62: mknod d0/c17 0 2024-04-12T05:08:49.312 INFO:tasks.workunit.client.0.smithi113.stdout:9/68: dwrite d1/d4/d5/f12 [0,4194304] 0 2024-04-12T05:08:49.315 INFO:tasks.workunit.client.0.smithi113.stdout:6/77: dwrite d8/de/f15 [0,4194304] 0 2024-04-12T05:08:49.319 INFO:tasks.workunit.client.0.smithi113.stdout:6/78: readlink d8/ld 0 2024-04-12T05:08:49.319 INFO:tasks.workunit.client.0.smithi113.stdout:3/63: fsync d0/f8 0 2024-04-12T05:08:49.320 INFO:tasks.workunit.client.0.smithi113.stdout:6/79: write d8/f12 [850656,29910] 0 2024-04-12T05:08:49.322 INFO:tasks.workunit.client.0.smithi113.stdout:1/42: creat fb x:0 0 0 2024-04-12T05:08:49.322 INFO:tasks.workunit.client.0.smithi113.stdout:9/69: mknod d1/d4/d5/d6/c1a 0 2024-04-12T05:08:49.322 INFO:tasks.workunit.client.0.smithi113.stdout:6/80: chown d8/de 1499298436 1 2024-04-12T05:08:49.325 INFO:tasks.workunit.client.0.smithi113.stdout:8/88: dwrite d9/fd [0,4194304] 0 2024-04-12T05:08:49.327 INFO:tasks.workunit.client.0.smithi113.stdout:9/70: write d1/fc [82743,120989] 0 2024-04-12T05:08:49.327 INFO:tasks.workunit.client.0.smithi113.stdout:1/43: unlink c7 0 2024-04-12T05:08:49.330 INFO:tasks.workunit.client.0.smithi113.stdout:6/81: unlink d8/cf 0 2024-04-12T05:08:49.331 INFO:tasks.workunit.client.0.smithi113.stdout:9/71: read - d1/d4/f18 zero size 2024-04-12T05:08:49.332 INFO:tasks.workunit.client.0.smithi113.stdout:6/82: creat d8/de/f19 x:0 0 0 2024-04-12T05:08:49.335 INFO:tasks.workunit.client.0.smithi113.stdout:9/72: mknod d1/d4/d16/c1b 0 2024-04-12T05:08:49.337 INFO:tasks.workunit.client.0.smithi113.stdout:9/73: stat d1/c8 0 2024-04-12T05:08:49.337 INFO:tasks.workunit.client.0.smithi113.stdout:6/83: creat d8/f1a x:0 0 0 2024-04-12T05:08:49.338 INFO:tasks.workunit.client.0.smithi113.stdout:1/44: truncate f5 2290241 0 2024-04-12T05:08:49.340 INFO:tasks.workunit.client.0.smithi113.stdout:9/74: chown d1/c8 9539896 1 2024-04-12T05:08:49.340 INFO:tasks.workunit.client.0.smithi113.stdout:1/45: mknod cc 0 2024-04-12T05:08:49.341 INFO:tasks.workunit.client.0.smithi113.stdout:8/89: dwrite d9/fb [0,4194304] 0 2024-04-12T05:08:49.342 INFO:tasks.workunit.client.0.smithi113.stdout:3/64: dwrite d0/f2 [0,4194304] 0 2024-04-12T05:08:49.344 INFO:tasks.workunit.client.0.smithi113.stdout:9/75: write d1/df/f11 [559959,91631] 0 2024-04-12T05:08:49.347 INFO:tasks.workunit.client.0.smithi113.stdout:9/76: mknod d1/c1c 0 2024-04-12T05:08:49.350 INFO:tasks.workunit.client.0.smithi113.stdout:3/65: dread d0/fa [0,4194304] 0 2024-04-12T05:08:49.350 INFO:tasks.workunit.client.0.smithi113.stdout:9/77: write d1/d4/f18 [389752,117803] 0 2024-04-12T05:08:49.352 INFO:tasks.workunit.client.0.smithi113.stdout:3/66: creat d0/f18 x:0 0 0 2024-04-12T05:08:49.353 INFO:tasks.workunit.client.0.smithi113.stdout:8/90: dread d9/f10 [0,4194304] 0 2024-04-12T05:08:49.355 INFO:tasks.workunit.client.0.smithi113.stdout:3/67: creat d0/d13/f19 x:0 0 0 2024-04-12T05:08:49.356 INFO:tasks.workunit.client.0.smithi113.stdout:8/91: rmdir d9 39 2024-04-12T05:08:49.358 INFO:tasks.workunit.client.0.smithi113.stdout:9/78: rmdir d1 39 2024-04-12T05:08:49.360 INFO:tasks.workunit.client.0.smithi113.stdout:8/92: mkdir d9/d16 0 2024-04-12T05:08:49.362 INFO:tasks.workunit.client.0.smithi113.stdout:3/68: link d0/fd d0/d13/f1a 0 2024-04-12T05:08:49.364 INFO:tasks.workunit.client.0.smithi113.stdout:9/79: write d1/d4/f18 [1310459,18816] 0 2024-04-12T05:08:49.366 INFO:tasks.workunit.client.0.smithi113.stdout:8/93: truncate f2 1122920 0 2024-04-12T05:08:49.366 INFO:tasks.workunit.client.0.smithi113.stdout:9/80: write d1/fc [951010,39344] 0 2024-04-12T05:08:49.368 INFO:tasks.workunit.client.0.smithi113.stdout:8/94: creat d9/f17 x:0 0 0 2024-04-12T05:08:49.370 INFO:tasks.workunit.client.0.smithi113.stdout:9/81: unlink d1/d4/d5/d6/c1a 0 2024-04-12T05:08:49.370 INFO:tasks.workunit.client.0.smithi113.stdout:8/95: symlink d9/l18 0 2024-04-12T05:08:49.371 INFO:tasks.workunit.client.0.smithi113.stdout:3/69: dwrite d0/f18 [0,4194304] 0 2024-04-12T05:08:49.372 INFO:tasks.workunit.client.0.smithi113.stdout:8/96: write d9/fb [682253,5359] 0 2024-04-12T05:08:49.375 INFO:tasks.workunit.client.0.smithi113.stdout:8/97: write d9/f17 [143618,120268] 0 2024-04-12T05:08:49.376 INFO:tasks.workunit.client.0.smithi113.stdout:3/70: fsync d0/fd 0 2024-04-12T05:08:49.377 INFO:tasks.workunit.client.0.smithi113.stdout:9/82: dread d1/df/f11 [0,4194304] 0 2024-04-12T05:08:49.378 INFO:tasks.workunit.client.0.smithi113.stdout:3/71: write d0/f8 [3779187,80157] 0 2024-04-12T05:08:49.379 INFO:tasks.workunit.client.0.smithi113.stdout:8/98: creat d9/f19 x:0 0 0 2024-04-12T05:08:49.379 INFO:tasks.workunit.client.0.smithi113.stdout:9/83: dread - d1/f14 zero size 2024-04-12T05:08:49.381 INFO:tasks.workunit.client.0.smithi113.stdout:3/72: fdatasync d0/f7 0 2024-04-12T05:08:49.383 INFO:tasks.workunit.client.0.smithi113.stdout:9/84: symlink d1/d4/d16/l1d 0 2024-04-12T05:08:49.386 INFO:tasks.workunit.client.0.smithi113.stdout:3/73: creat d0/f1b x:0 0 0 2024-04-12T05:08:49.392 INFO:tasks.workunit.client.0.smithi113.stdout:9/85: dwrite d1/d4/f18 [0,4194304] 0 2024-04-12T05:08:49.395 INFO:tasks.workunit.client.0.smithi113.stdout:8/99: dwrite f2 [0,4194304] 0 2024-04-12T05:08:49.397 INFO:tasks.workunit.client.0.smithi113.stdout:8/100: stat d9/l18 0 2024-04-12T05:08:49.400 INFO:tasks.workunit.client.0.smithi113.stdout:9/86: dread d1/d4/d5/f7 [0,4194304] 0 2024-04-12T05:08:49.401 INFO:tasks.workunit.client.0.smithi113.stdout:8/101: dread d9/f10 [0,4194304] 0 2024-04-12T05:08:49.401 INFO:tasks.workunit.client.0.smithi113.stdout:9/87: mknod d1/c1e 0 2024-04-12T05:08:49.402 INFO:tasks.workunit.client.0.smithi113.stdout:8/102: dread - d9/f19 zero size 2024-04-12T05:08:49.403 INFO:tasks.workunit.client.0.smithi113.stdout:8/103: rmdir d9 39 2024-04-12T05:08:49.408 INFO:tasks.workunit.client.0.smithi113.stdout:0/43: sync 2024-04-12T05:08:49.410 INFO:tasks.workunit.client.0.smithi113.stdout:9/88: link d1/c10 d1/d4/d5/c1f 0 2024-04-12T05:08:49.410 INFO:tasks.workunit.client.0.smithi113.stdout:8/104: dread d9/fd [0,4194304] 0 2024-04-12T05:08:49.411 INFO:tasks.workunit.client.0.smithi113.stdout:8/105: read f2 [3207446,98271] 0 2024-04-12T05:08:49.412 INFO:tasks.workunit.client.0.smithi113.stdout:8/106: truncate d9/f17 884370 0 2024-04-12T05:08:49.416 INFO:tasks.workunit.client.0.smithi113.stdout:0/44: truncate d1/f2 3854616 0 2024-04-12T05:08:49.417 INFO:tasks.workunit.client.0.smithi113.stdout:9/89: getdents d1/d4/d16 0 2024-04-12T05:08:49.419 INFO:tasks.workunit.client.0.smithi113.stdout:8/107: dread f2 [0,4194304] 0 2024-04-12T05:08:49.427 INFO:tasks.workunit.client.0.smithi113.stdout:8/108: dread d9/fd [0,4194304] 0 2024-04-12T05:08:49.427 INFO:tasks.workunit.client.0.smithi113.stdout:9/90: dwrite d1/d4/d5/f12 [0,4194304] 0 2024-04-12T05:08:49.430 INFO:tasks.workunit.client.0.smithi113.stdout:0/45: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:49.430 INFO:tasks.workunit.client.0.smithi113.stdout:8/109: creat d9/d16/f1a x:0 0 0 2024-04-12T05:08:49.431 INFO:tasks.workunit.client.0.smithi113.stdout:8/110: readlink d9/la 0 2024-04-12T05:08:49.433 INFO:tasks.workunit.client.0.smithi113.stdout:9/91: link d1/c1c d1/c20 0 2024-04-12T05:08:49.443 INFO:tasks.workunit.client.0.smithi113.stdout:8/111: dwrite d9/d16/f1a [0,4194304] 0 2024-04-12T05:08:49.444 INFO:tasks.workunit.client.0.smithi113.stdout:0/46: dwrite d1/fb [0,4194304] 0 2024-04-12T05:08:49.444 INFO:tasks.workunit.client.0.smithi113.stdout:8/112: readlink d9/lc 0 2024-04-12T05:08:49.448 INFO:tasks.workunit.client.0.smithi113.stdout:9/92: truncate d1/d4/d5/f7 1385336 0 2024-04-12T05:08:49.451 INFO:tasks.workunit.client.0.smithi113.stdout:9/93: rmdir d1/d4 39 2024-04-12T05:08:49.452 INFO:tasks.workunit.client.0.smithi113.stdout:0/47: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:49.453 INFO:tasks.workunit.client.0.smithi113.stdout:8/113: dwrite d9/fb [0,4194304] 0 2024-04-12T05:08:49.455 INFO:tasks.workunit.client.0.smithi113.stdout:0/48: symlink d1/d9/ld 0 2024-04-12T05:08:49.463 INFO:tasks.workunit.client.0.smithi113.stdout:0/49: creat d1/fe x:0 0 0 2024-04-12T05:08:49.463 INFO:tasks.workunit.client.0.smithi113.stdout:9/94: dwrite d1/d4/f18 [0,4194304] 0 2024-04-12T05:08:49.468 INFO:tasks.workunit.client.0.smithi113.stdout:9/95: dread d1/df/f11 [0,4194304] 0 2024-04-12T05:08:49.468 INFO:tasks.workunit.client.0.smithi113.stdout:0/50: truncate d1/f3 4915302 0 2024-04-12T05:08:49.471 INFO:tasks.workunit.client.0.smithi113.stdout:9/96: symlink d1/d4/d5/d6/l21 0 2024-04-12T05:08:49.471 INFO:tasks.workunit.client.0.smithi113.stdout:9/97: stat d1/f14 0 2024-04-12T05:08:49.474 INFO:tasks.workunit.client.0.smithi113.stdout:0/51: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:49.474 INFO:tasks.workunit.client.0.smithi113.stdout:9/98: stat d1/d4/d5/f7 0 2024-04-12T05:08:49.474 INFO:tasks.workunit.client.0.smithi113.stdout:9/99: stat d1/c1e 0 2024-04-12T05:08:49.475 INFO:tasks.workunit.client.0.smithi113.stdout:0/52: rename d1/d6 to d1/d6/da/df 22 2024-04-12T05:08:49.476 INFO:tasks.workunit.client.0.smithi113.stdout:9/100: mkdir d1/d4/d5/d22 0 2024-04-12T05:08:49.479 INFO:tasks.workunit.client.0.smithi113.stdout:9/101: rename d1/c19 to d1/df/c23 0 2024-04-12T05:08:49.480 INFO:tasks.workunit.client.0.smithi113.stdout:9/102: read d1/df/f11 [579501,106670] 0 2024-04-12T05:08:49.484 INFO:tasks.workunit.client.0.smithi113.stdout:0/53: fdatasync d1/fb 0 2024-04-12T05:08:49.487 INFO:tasks.workunit.client.0.smithi113.stdout:9/103: dwrite d1/f14 [0,4194304] 0 2024-04-12T05:08:49.489 INFO:tasks.workunit.client.0.smithi113.stdout:8/114: truncate d9/fb 561307 0 2024-04-12T05:08:49.494 INFO:tasks.workunit.client.0.smithi113.stdout:8/115: dread d9/fb [0,4194304] 0 2024-04-12T05:08:49.495 INFO:tasks.workunit.client.0.smithi113.stdout:0/54: truncate d1/f2 257055 0 2024-04-12T05:08:49.496 INFO:tasks.workunit.client.0.smithi113.stdout:8/116: write d9/f19 [114484,104583] 0 2024-04-12T05:08:49.501 INFO:tasks.workunit.client.0.smithi113.stdout:0/55: rename d1/fb to d1/d6/f10 0 2024-04-12T05:08:49.502 INFO:tasks.workunit.client.0.smithi113.stdout:0/56: write d1/fe [328641,93346] 0 2024-04-12T05:08:49.505 INFO:tasks.workunit.client.0.smithi113.stdout:8/117: dwrite f2 [0,4194304] 0 2024-04-12T05:08:49.506 INFO:tasks.workunit.client.0.smithi113.stdout:0/57: creat d1/d6/da/f11 x:0 0 0 2024-04-12T05:08:49.507 INFO:tasks.workunit.client.0.smithi113.stdout:9/104: dread d1/d4/d5/f7 [0,4194304] 0 2024-04-12T05:08:49.508 INFO:tasks.workunit.client.0.smithi113.stdout:8/118: mkdir d9/d1b 0 2024-04-12T05:08:49.509 INFO:tasks.workunit.client.0.smithi113.stdout:8/119: stat d9/d16/f1a 0 2024-04-12T05:08:49.510 INFO:tasks.workunit.client.0.smithi113.stdout:0/58: rename d1/d6/da/lc to d1/d6/da/l12 0 2024-04-12T05:08:49.514 INFO:tasks.workunit.client.0.smithi113.stdout:8/120: mkdir d9/d16/d1c 0 2024-04-12T05:08:49.515 INFO:tasks.workunit.client.0.smithi113.stdout:9/105: dwrite d1/f14 [0,4194304] 0 2024-04-12T05:08:49.520 INFO:tasks.workunit.client.0.smithi113.stdout:8/121: mknod d9/d1b/c1d 0 2024-04-12T05:08:49.522 INFO:tasks.workunit.client.0.smithi113.stdout:0/59: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:49.522 INFO:tasks.workunit.client.0.smithi113.stdout:9/106: dwrite d1/fc [0,4194304] 0 2024-04-12T05:08:49.524 INFO:tasks.workunit.client.0.smithi113.stdout:9/107: readlink d1/d4/d16/l1d 0 2024-04-12T05:08:49.526 INFO:tasks.workunit.client.0.smithi113.stdout:8/122: dread d9/fd [0,4194304] 0 2024-04-12T05:08:49.531 INFO:tasks.workunit.client.0.smithi113.stdout:8/123: truncate d9/f17 1054335 0 2024-04-12T05:08:49.533 INFO:tasks.workunit.client.0.smithi113.stdout:9/108: dwrite d1/d4/f13 [0,4194304] 0 2024-04-12T05:08:49.533 INFO:tasks.workunit.client.0.smithi113.stdout:0/60: truncate d1/d6/f10 57655 0 2024-04-12T05:08:49.537 INFO:tasks.workunit.client.0.smithi113.stdout:9/109: mkdir d1/d24 0 2024-04-12T05:08:49.538 INFO:tasks.workunit.client.0.smithi113.stdout:8/124: dwrite d9/fd [0,4194304] 0 2024-04-12T05:08:49.540 INFO:tasks.workunit.client.0.smithi113.stdout:8/125: chown d9/fb 3 1 2024-04-12T05:08:49.540 INFO:tasks.workunit.client.0.smithi113.stdout:0/61: creat d1/f13 x:0 0 0 2024-04-12T05:08:49.543 INFO:tasks.workunit.client.0.smithi113.stdout:8/126: unlink d9/f10 0 2024-04-12T05:08:49.545 INFO:tasks.workunit.client.0.smithi113.stdout:7/59: sync 2024-04-12T05:08:49.547 INFO:tasks.workunit.client.0.smithi113.stdout:9/110: dwrite d1/fc [0,4194304] 0 2024-04-12T05:08:49.547 INFO:tasks.workunit.client.0.smithi113.stdout:8/127: mkdir d9/d16/d1e 0 2024-04-12T05:08:49.548 INFO:tasks.workunit.client.0.smithi113.stdout:0/62: getdents d1/d6/da 0 2024-04-12T05:08:49.549 INFO:tasks.workunit.client.0.smithi113.stdout:7/60: mkdir d0/dd/d11 0 2024-04-12T05:08:49.551 INFO:tasks.workunit.client.0.smithi113.stdout:0/63: chown d1/l7 0 1 2024-04-12T05:08:49.554 INFO:tasks.workunit.client.0.smithi113.stdout:7/61: creat d0/f12 x:0 0 0 2024-04-12T05:08:49.556 INFO:tasks.workunit.client.0.smithi113.stdout:9/111: dread d1/d4/d5/f7 [0,4194304] 0 2024-04-12T05:08:49.558 INFO:tasks.workunit.client.0.smithi113.stdout:0/64: symlink d1/d9/l14 0 2024-04-12T05:08:49.559 INFO:tasks.workunit.client.0.smithi113.stdout:9/112: write d1/fc [4262090,67634] 0 2024-04-12T05:08:49.561 INFO:tasks.workunit.client.0.smithi113.stdout:9/113: stat d1/c1e 0 2024-04-12T05:08:49.561 INFO:tasks.workunit.client.0.smithi113.stdout:8/128: dwrite d9/fb [0,4194304] 0 2024-04-12T05:08:49.562 INFO:tasks.workunit.client.0.smithi113.stdout:0/65: symlink d1/d6/da/l15 0 2024-04-12T05:08:49.563 INFO:tasks.workunit.client.0.smithi113.stdout:8/129: readlink d9/l18 0 2024-04-12T05:08:49.564 INFO:tasks.workunit.client.0.smithi113.stdout:0/66: chown d1/d9 2462 1 2024-04-12T05:08:49.565 INFO:tasks.workunit.client.0.smithi113.stdout:0/67: dread - d1/d6/da/f11 zero size 2024-04-12T05:08:49.567 INFO:tasks.workunit.client.0.smithi113.stdout:0/68: chown d1/d6/da/l15 0 1 2024-04-12T05:08:49.568 INFO:tasks.workunit.client.0.smithi113.stdout:7/62: dwrite d0/f12 [0,4194304] 0 2024-04-12T05:08:49.568 INFO:tasks.workunit.client.0.smithi113.stdout:0/69: chown d1/d6/da/l12 23751 1 2024-04-12T05:08:49.570 INFO:tasks.workunit.client.0.smithi113.stdout:0/70: readlink d1/d9/l14 0 2024-04-12T05:08:49.573 INFO:tasks.workunit.client.0.smithi113.stdout:7/63: mknod d0/dc/d6/c13 0 2024-04-12T05:08:49.575 INFO:tasks.workunit.client.0.smithi113.stdout:9/114: getdents d1/d4/d5/d6/d15 0 2024-04-12T05:08:49.576 INFO:tasks.workunit.client.0.smithi113.stdout:8/130: mknod d9/d16/d1c/c1f 0 2024-04-12T05:08:49.577 INFO:tasks.workunit.client.0.smithi113.stdout:8/131: fsync f2 0 2024-04-12T05:08:49.577 INFO:tasks.workunit.client.0.smithi113.stdout:8/132: fdatasync d9/fd 0 2024-04-12T05:08:49.580 INFO:tasks.workunit.client.0.smithi113.stdout:8/133: dread f2 [0,4194304] 0 2024-04-12T05:08:49.582 INFO:tasks.workunit.client.0.smithi113.stdout:7/64: unlink d0/l10 0 2024-04-12T05:08:49.583 INFO:tasks.workunit.client.0.smithi113.stdout:8/134: mknod d9/d16/d1e/c20 0 2024-04-12T05:08:49.584 INFO:tasks.workunit.client.0.smithi113.stdout:7/65: mknod d0/dc/d6/c14 0 2024-04-12T05:08:49.637 INFO:tasks.workunit.client.0.smithi113.stdout:8/135: dread d9/f17 [0,4194304] 0 2024-04-12T05:08:49.637 INFO:tasks.workunit.client.0.smithi113.stdout:0/71: write d1/f2 [977240,124432] 0 2024-04-12T05:08:49.640 INFO:tasks.workunit.client.0.smithi113.stdout:0/72: write d1/f3 [10111,95797] 0 2024-04-12T05:08:49.640 INFO:tasks.workunit.client.0.smithi113.stdout:8/136: mknod d9/d1b/c21 0 2024-04-12T05:08:49.644 INFO:tasks.workunit.client.0.smithi113.stdout:7/66: dwrite d0/dc/d6/fa [0,4194304] 0 2024-04-12T05:08:49.647 INFO:tasks.workunit.client.0.smithi113.stdout:7/67: readlink d0/l8 0 2024-04-12T05:08:49.650 INFO:tasks.workunit.client.0.smithi113.stdout:0/73: dwrite d1/fe [0,4194304] 0 2024-04-12T05:08:49.653 INFO:tasks.workunit.client.0.smithi113.stdout:8/137: dwrite d9/f17 [0,4194304] 0 2024-04-12T05:08:49.653 INFO:tasks.workunit.client.0.smithi113.stdout:0/74: write d1/f3 [4664745,73286] 0 2024-04-12T05:08:49.655 INFO:tasks.workunit.client.0.smithi113.stdout:8/138: dread d9/f19 [0,4194304] 0 2024-04-12T05:08:49.657 INFO:tasks.workunit.client.0.smithi113.stdout:0/75: unlink d1/d9/ld 0 2024-04-12T05:08:49.661 INFO:tasks.workunit.client.0.smithi113.stdout:0/76: mkdir d1/d6/d16 0 2024-04-12T05:08:49.666 INFO:tasks.workunit.client.0.smithi113.stdout:8/139: dwrite d9/d16/f1a [4194304,4194304] 0 2024-04-12T05:08:49.668 INFO:tasks.workunit.client.0.smithi113.stdout:8/140: mknod d9/d16/c22 0 2024-04-12T05:08:49.671 INFO:tasks.workunit.client.0.smithi113.stdout:8/141: write d9/f19 [787795,48280] 0 2024-04-12T05:08:49.671 INFO:tasks.workunit.client.0.smithi113.stdout:7/68: truncate d0/dc/d6/fa 3209911 0 2024-04-12T05:08:49.675 INFO:tasks.workunit.client.0.smithi113.stdout:2/52: sync 2024-04-12T05:08:49.675 INFO:tasks.workunit.client.0.smithi113.stdout:3/74: sync 2024-04-12T05:08:49.675 INFO:tasks.workunit.client.0.smithi113.stdout:4/73: sync 2024-04-12T05:08:49.675 INFO:tasks.workunit.client.0.smithi113.stdout:6/84: sync 2024-04-12T05:08:49.675 INFO:tasks.workunit.client.0.smithi113.stdout:5/134: sync 2024-04-12T05:08:49.675 INFO:tasks.workunit.client.0.smithi113.stdout:1/46: sync 2024-04-12T05:08:49.676 INFO:tasks.workunit.client.0.smithi113.stdout:4/74: chown l0 56563 1 2024-04-12T05:08:49.676 INFO:tasks.workunit.client.0.smithi113.stdout:7/69: rename d0/dc/d6/c14 to d0/dc/d6/c15 0 2024-04-12T05:08:49.681 INFO:tasks.workunit.client.0.smithi113.stdout:1/47: chown f5 766 1 2024-04-12T05:08:49.683 INFO:tasks.workunit.client.0.smithi113.stdout:5/135: creat d0/de/f26 x:0 0 0 2024-04-12T05:08:49.683 INFO:tasks.workunit.client.0.smithi113.stdout:2/53: chown c2 469914257 1 2024-04-12T05:08:49.683 INFO:tasks.workunit.client.0.smithi113.stdout:7/70: creat d0/dc/d6/f16 x:0 0 0 2024-04-12T05:08:49.683 INFO:tasks.workunit.client.0.smithi113.stdout:0/77: truncate d1/fe 1965731 0 2024-04-12T05:08:49.684 INFO:tasks.workunit.client.0.smithi113.stdout:6/85: truncate d8/de/d13/fa 5796662 0 2024-04-12T05:08:49.686 INFO:tasks.workunit.client.0.smithi113.stdout:2/54: chown f7 578 1 2024-04-12T05:08:49.686 INFO:tasks.workunit.client.0.smithi113.stdout:6/86: dread - d8/f1a zero size 2024-04-12T05:08:49.686 INFO:tasks.workunit.client.0.smithi113.stdout:5/136: chown d0/f5 72 1 2024-04-12T05:08:49.687 INFO:tasks.workunit.client.0.smithi113.stdout:7/71: symlink d0/dd/l17 0 2024-04-12T05:08:49.688 INFO:tasks.workunit.client.0.smithi113.stdout:1/48: dread f5 [0,4194304] 0 2024-04-12T05:08:49.688 INFO:tasks.workunit.client.0.smithi113.stdout:2/55: rename d8/d9 to d8/d9/db 22 2024-04-12T05:08:49.689 INFO:tasks.workunit.client.0.smithi113.stdout:6/87: chown d8/de/d17 67254 1 2024-04-12T05:08:49.690 INFO:tasks.workunit.client.0.smithi113.stdout:4/75: dread da/fb [0,4194304] 0 2024-04-12T05:08:49.690 INFO:tasks.workunit.client.0.smithi113.stdout:7/72: dread - d0/dc/d6/f16 zero size 2024-04-12T05:08:49.690 INFO:tasks.workunit.client.0.smithi113.stdout:5/137: write d0/f14 [4578084,34154] 0 2024-04-12T05:08:49.690 INFO:tasks.workunit.client.0.smithi113.stdout:0/78: truncate d1/d6/f10 701184 0 2024-04-12T05:08:49.692 INFO:tasks.workunit.client.0.smithi113.stdout:4/76: stat da/cc 0 2024-04-12T05:08:49.692 INFO:tasks.workunit.client.0.smithi113.stdout:7/73: chown d0/dd/d11 0 1 2024-04-12T05:08:49.692 INFO:tasks.workunit.client.0.smithi113.stdout:6/88: mknod d8/de/c1b 0 2024-04-12T05:08:49.693 INFO:tasks.workunit.client.0.smithi113.stdout:0/79: write d1/f2 [2036371,9764] 0 2024-04-12T05:08:49.693 INFO:tasks.workunit.client.0.smithi113.stdout:7/74: readlink d0/dd/l17 0 2024-04-12T05:08:49.694 INFO:tasks.workunit.client.0.smithi113.stdout:2/56: creat d8/d9/fc x:0 0 0 2024-04-12T05:08:49.695 INFO:tasks.workunit.client.0.smithi113.stdout:4/77: chown da/dd/df/f13 1105175183 1 2024-04-12T05:08:49.696 INFO:tasks.workunit.client.0.smithi113.stdout:1/49: link f2 fd 0 2024-04-12T05:08:49.697 INFO:tasks.workunit.client.0.smithi113.stdout:6/89: truncate d8/de/f19 823564 0 2024-04-12T05:08:49.700 INFO:tasks.workunit.client.0.smithi113.stdout:6/90: dread - d8/de/f16 zero size 2024-04-12T05:08:49.700 INFO:tasks.workunit.client.0.smithi113.stdout:7/75: chown d0/dc/d6/c15 721 1 2024-04-12T05:08:49.701 INFO:tasks.workunit.client.0.smithi113.stdout:3/75: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:08:49.702 INFO:tasks.workunit.client.0.smithi113.stdout:5/138: getdents d0 0 2024-04-12T05:08:49.704 INFO:tasks.workunit.client.0.smithi113.stdout:0/80: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:49.704 INFO:tasks.workunit.client.0.smithi113.stdout:6/91: unlink l3 0 2024-04-12T05:08:49.705 INFO:tasks.workunit.client.0.smithi113.stdout:6/92: chown d8/ld 31573805 1 2024-04-12T05:08:49.706 INFO:tasks.workunit.client.0.smithi113.stdout:4/78: dread da/dd/df/f10 [0,4194304] 0 2024-04-12T05:08:49.706 INFO:tasks.workunit.client.0.smithi113.stdout:2/57: truncate d8/d9/fa 24671 0 2024-04-12T05:08:49.706 INFO:tasks.workunit.client.0.smithi113.stdout:0/81: truncate d1/d6/da/f11 643959 0 2024-04-12T05:08:49.726 INFO:tasks.workunit.client.0.smithi113.stdout:7/76: unlink d0/dc/d6/c13 0 2024-04-12T05:08:49.727 INFO:tasks.workunit.client.0.smithi113.stdout:0/82: write d1/f3 [4758810,127501] 0 2024-04-12T05:08:49.729 INFO:tasks.workunit.client.0.smithi113.stdout:1/50: getdents . 0 2024-04-12T05:08:49.730 INFO:tasks.workunit.client.0.smithi113.stdout:6/93: dread d8/f12 [0,4194304] 0 2024-04-12T05:08:49.731 INFO:tasks.workunit.client.0.smithi113.stdout:4/79: write da/dd/df/f10 [3739719,1886] 0 2024-04-12T05:08:49.731 INFO:tasks.workunit.client.0.smithi113.stdout:7/77: creat d0/dc/f18 x:0 0 0 2024-04-12T05:08:49.733 INFO:tasks.workunit.client.0.smithi113.stdout:2/58: dread d8/d9/fa [0,4194304] 0 2024-04-12T05:08:49.734 INFO:tasks.workunit.client.0.smithi113.stdout:1/51: getdents . 0 2024-04-12T05:08:49.736 INFO:tasks.workunit.client.0.smithi113.stdout:6/94: dread d8/de/f15 [0,4194304] 0 2024-04-12T05:08:49.737 INFO:tasks.workunit.client.0.smithi113.stdout:4/80: rename da/dd/df/f13 to da/f1a 0 2024-04-12T05:08:49.737 INFO:tasks.workunit.client.0.smithi113.stdout:4/81: fdatasync da/dd/df/f10 0 2024-04-12T05:08:49.739 INFO:tasks.workunit.client.0.smithi113.stdout:1/52: creat fe x:0 0 0 2024-04-12T05:08:49.739 INFO:tasks.workunit.client.0.smithi113.stdout:1/53: fdatasync fb 0 2024-04-12T05:08:49.740 INFO:tasks.workunit.client.0.smithi113.stdout:0/83: truncate d1/d6/da/f11 316565 0 2024-04-12T05:08:49.741 INFO:tasks.workunit.client.0.smithi113.stdout:6/95: mkdir d8/de/d1c 0 2024-04-12T05:08:49.743 INFO:tasks.workunit.client.0.smithi113.stdout:4/82: write da/f1a [290829,92568] 0 2024-04-12T05:08:49.744 INFO:tasks.workunit.client.0.smithi113.stdout:0/84: write d1/fe [2838146,114167] 0 2024-04-12T05:08:49.745 INFO:tasks.workunit.client.0.smithi113.stdout:4/83: write da/dd/df/d14/f19 [803990,101503] 0 2024-04-12T05:08:49.745 INFO:tasks.workunit.client.0.smithi113.stdout:7/78: rename d0/f7 to d0/dd/d11/f19 0 2024-04-12T05:08:49.747 INFO:tasks.workunit.client.0.smithi113.stdout:2/59: link d8/d9/fa d8/fd 0 2024-04-12T05:08:49.749 INFO:tasks.workunit.client.0.smithi113.stdout:4/84: mkdir da/dd/d1b 0 2024-04-12T05:08:49.750 INFO:tasks.workunit.client.0.smithi113.stdout:7/79: mkdir d0/dc/d6/d1a 0 2024-04-12T05:08:49.750 INFO:tasks.workunit.client.0.smithi113.stdout:6/96: dread d8/de/d13/fa [0,4194304] 0 2024-04-12T05:08:49.751 INFO:tasks.workunit.client.0.smithi113.stdout:2/60: mkdir d8/d9/de 0 2024-04-12T05:08:49.752 INFO:tasks.workunit.client.0.smithi113.stdout:2/61: stat c2 0 2024-04-12T05:08:49.752 INFO:tasks.workunit.client.0.smithi113.stdout:0/85: dwrite d1/f13 [0,4194304] 0 2024-04-12T05:08:49.753 INFO:tasks.workunit.client.0.smithi113.stdout:4/85: creat da/dd/df/f1c x:0 0 0 2024-04-12T05:08:49.754 INFO:tasks.workunit.client.0.smithi113.stdout:4/86: stat da/cc 0 2024-04-12T05:08:49.754 INFO:tasks.workunit.client.0.smithi113.stdout:7/80: mkdir d0/dd/d1b 0 2024-04-12T05:08:49.754 INFO:tasks.workunit.client.0.smithi113.stdout:6/97: fsync d8/de/f15 0 2024-04-12T05:08:49.755 INFO:tasks.workunit.client.0.smithi113.stdout:7/81: chown d0/dc/ce 26315750 1 2024-04-12T05:08:49.757 INFO:tasks.workunit.client.0.smithi113.stdout:2/62: write d8/fd [971926,51294] 0 2024-04-12T05:08:49.759 INFO:tasks.workunit.client.0.smithi113.stdout:2/63: write d8/d9/fa [1949040,89147] 0 2024-04-12T05:08:49.759 INFO:tasks.workunit.client.0.smithi113.stdout:4/87: mkdir da/dd/df/d1d 0 2024-04-12T05:08:49.761 INFO:tasks.workunit.client.0.smithi113.stdout:7/82: creat d0/dd/f1c x:0 0 0 2024-04-12T05:08:49.762 INFO:tasks.workunit.client.0.smithi113.stdout:0/86: mknod d1/c17 0 2024-04-12T05:08:49.763 INFO:tasks.workunit.client.0.smithi113.stdout:4/88: dread da/f1a [0,4194304] 0 2024-04-12T05:08:49.764 INFO:tasks.workunit.client.0.smithi113.stdout:2/64: readlink l4 0 2024-04-12T05:08:49.767 INFO:tasks.workunit.client.0.smithi113.stdout:7/83: symlink d0/dc/d6/l1d 0 2024-04-12T05:08:49.769 INFO:tasks.workunit.client.0.smithi113.stdout:6/98: dwrite d8/f12 [0,4194304] 0 2024-04-12T05:08:49.772 INFO:tasks.workunit.client.0.smithi113.stdout:0/87: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:49.773 INFO:tasks.workunit.client.0.smithi113.stdout:4/89: rename da/cc to da/dd/d1b/c1e 0 2024-04-12T05:08:49.774 INFO:tasks.workunit.client.0.smithi113.stdout:4/90: chown l0 99143125 1 2024-04-12T05:08:49.774 INFO:tasks.workunit.client.0.smithi113.stdout:6/99: mkdir d8/d1d 0 2024-04-12T05:08:49.777 INFO:tasks.workunit.client.0.smithi113.stdout:4/91: write da/fb [5192680,119037] 0 2024-04-12T05:08:49.777 INFO:tasks.workunit.client.0.smithi113.stdout:0/88: mknod d1/d6/c18 0 2024-04-12T05:08:49.778 INFO:tasks.workunit.client.0.smithi113.stdout:7/84: dwrite d0/f12 [0,4194304] 0 2024-04-12T05:08:49.779 INFO:tasks.workunit.client.0.smithi113.stdout:7/85: read d0/dc/d6/fa [2389433,48853] 0 2024-04-12T05:08:49.780 INFO:tasks.workunit.client.0.smithi113.stdout:6/100: rename d8/de/c1b to d8/de/d17/c1e 0 2024-04-12T05:08:49.780 INFO:tasks.workunit.client.0.smithi113.stdout:2/65: getdents d8 0 2024-04-12T05:08:49.781 INFO:tasks.workunit.client.0.smithi113.stdout:7/86: write d0/dc/d6/f16 [441551,128689] 0 2024-04-12T05:08:49.783 INFO:tasks.workunit.client.0.smithi113.stdout:4/92: mknod da/de/c1f 0 2024-04-12T05:08:49.784 INFO:tasks.workunit.client.0.smithi113.stdout:2/66: symlink d8/d9/lf 0 2024-04-12T05:08:49.785 INFO:tasks.workunit.client.0.smithi113.stdout:2/67: write f7 [935378,78876] 0 2024-04-12T05:08:49.791 INFO:tasks.workunit.client.0.smithi113.stdout:4/93: dwrite da/dd/df/f10 [0,4194304] 0 2024-04-12T05:08:49.800 INFO:tasks.workunit.client.0.smithi113.stdout:7/87: write d0/dd/d11/f19 [2851615,130120] 0 2024-04-12T05:08:49.800 INFO:tasks.workunit.client.0.smithi113.stdout:2/68: truncate d8/fd 624390 0 2024-04-12T05:08:49.802 INFO:tasks.workunit.client.0.smithi113.stdout:0/89: dwrite d1/d6/f10 [0,4194304] 0 2024-04-12T05:08:49.802 INFO:tasks.workunit.client.0.smithi113.stdout:2/69: write f7 [2058673,13759] 0 2024-04-12T05:08:49.803 INFO:tasks.workunit.client.0.smithi113.stdout:4/94: truncate da/dd/df/d14/f19 450783 0 2024-04-12T05:08:49.803 INFO:tasks.workunit.client.0.smithi113.stdout:7/88: truncate d0/dc/f18 527041 0 2024-04-12T05:08:49.807 INFO:tasks.workunit.client.0.smithi113.stdout:4/95: rename l0 to da/dd/df/d14/l20 0 2024-04-12T05:08:49.810 INFO:tasks.workunit.client.0.smithi113.stdout:4/96: unlink f2 0 2024-04-12T05:08:49.894 INFO:tasks.workunit.client.0.smithi113.stdout:4/97: dread - da/dd/df/f1c zero size 2024-04-12T05:08:49.897 INFO:tasks.workunit.client.0.smithi113.stdout:0/90: write d1/d6/da/f11 [20425,59531] 0 2024-04-12T05:08:49.898 INFO:tasks.workunit.client.0.smithi113.stdout:7/89: link d0/dd/l17 d0/dc/d6/d1a/l1e 0 2024-04-12T05:08:49.900 INFO:tasks.workunit.client.0.smithi113.stdout:4/98: creat da/dd/f21 x:0 0 0 2024-04-12T05:08:49.901 INFO:tasks.workunit.client.0.smithi113.stdout:0/91: unlink d1/d9/l14 0 2024-04-12T05:08:49.902 INFO:tasks.workunit.client.0.smithi113.stdout:4/99: write da/fb [1559941,107065] 0 2024-04-12T05:08:49.903 INFO:tasks.workunit.client.0.smithi113.stdout:7/90: rename d0/dc/ce to d0/dd/d11/c1f 0 2024-04-12T05:08:49.905 INFO:tasks.workunit.client.0.smithi113.stdout:2/70: dwrite d8/d9/fa [0,4194304] 0 2024-04-12T05:08:49.907 INFO:tasks.workunit.client.0.smithi113.stdout:0/92: rename d1/l7 to d1/d6/da/l19 0 2024-04-12T05:08:49.911 INFO:tasks.workunit.client.0.smithi113.stdout:4/100: mkdir da/dd/df/d1d/d22 0 2024-04-12T05:08:49.914 INFO:tasks.workunit.client.0.smithi113.stdout:2/71: creat d8/d9/de/f10 x:0 0 0 2024-04-12T05:08:49.914 INFO:tasks.workunit.client.0.smithi113.stdout:0/93: dwrite d1/f13 [0,4194304] 0 2024-04-12T05:08:49.917 INFO:tasks.workunit.client.0.smithi113.stdout:4/101: chown da/c11 15 1 2024-04-12T05:08:49.918 INFO:tasks.workunit.client.0.smithi113.stdout:0/94: read d1/d6/da/f11 [169339,121898] 0 2024-04-12T05:08:49.919 INFO:tasks.workunit.client.0.smithi113.stdout:7/91: truncate d0/dc/d6/f16 364291 0 2024-04-12T05:08:49.922 INFO:tasks.workunit.client.0.smithi113.stdout:4/102: creat da/dd/d1b/f23 x:0 0 0 2024-04-12T05:08:49.924 INFO:tasks.workunit.client.0.smithi113.stdout:2/72: dwrite f7 [0,4194304] 0 2024-04-12T05:08:49.925 INFO:tasks.workunit.client.0.smithi113.stdout:7/92: creat d0/dd/d11/f20 x:0 0 0 2024-04-12T05:08:49.926 INFO:tasks.workunit.client.0.smithi113.stdout:7/93: fdatasync d0/dc/f18 0 2024-04-12T05:08:49.926 INFO:tasks.workunit.client.0.smithi113.stdout:4/103: truncate da/dd/df/d14/f18 323557 0 2024-04-12T05:08:49.927 INFO:tasks.workunit.client.0.smithi113.stdout:0/95: dread d1/d6/f10 [0,4194304] 0 2024-04-12T05:08:49.927 INFO:tasks.workunit.client.0.smithi113.stdout:9/115: sync 2024-04-12T05:08:49.927 INFO:tasks.workunit.client.0.smithi113.stdout:8/142: sync 2024-04-12T05:08:49.931 INFO:tasks.workunit.client.0.smithi113.stdout:8/143: dread d9/f19 [0,4194304] 0 2024-04-12T05:08:49.932 INFO:tasks.workunit.client.0.smithi113.stdout:7/94: creat d0/dc/d6/d1a/f21 x:0 0 0 2024-04-12T05:08:49.933 INFO:tasks.workunit.client.0.smithi113.stdout:7/95: dread - d0/dd/f1c zero size 2024-04-12T05:08:49.934 INFO:tasks.workunit.client.0.smithi113.stdout:7/96: dread - d0/dd/d11/f20 zero size 2024-04-12T05:08:49.935 INFO:tasks.workunit.client.0.smithi113.stdout:8/144: symlink d9/l23 0 2024-04-12T05:08:49.937 INFO:tasks.workunit.client.0.smithi113.stdout:4/104: getdents da/dd/df/d14 0 2024-04-12T05:08:49.938 INFO:tasks.workunit.client.0.smithi113.stdout:7/97: mknod d0/c22 0 2024-04-12T05:08:49.939 INFO:tasks.workunit.client.0.smithi113.stdout:8/145: mkdir d9/d24 0 2024-04-12T05:08:49.939 INFO:tasks.workunit.client.0.smithi113.stdout:8/146: stat d9/d24 0 2024-04-12T05:08:49.940 INFO:tasks.workunit.client.0.smithi113.stdout:0/96: truncate d1/f2 3564949 0 2024-04-12T05:08:49.942 INFO:tasks.workunit.client.0.smithi113.stdout:9/116: truncate d1/d4/f18 2571261 0 2024-04-12T05:08:49.943 INFO:tasks.workunit.client.0.smithi113.stdout:7/98: creat d0/dd/d1b/f23 x:0 0 0 2024-04-12T05:08:49.943 INFO:tasks.workunit.client.0.smithi113.stdout:8/147: creat d9/d16/f25 x:0 0 0 2024-04-12T05:08:49.945 INFO:tasks.workunit.client.0.smithi113.stdout:9/117: stat d1/d4/d5/d6/cd 0 2024-04-12T05:08:49.947 INFO:tasks.workunit.client.0.smithi113.stdout:0/97: link d1/fe d1/d9/f1a 0 2024-04-12T05:08:49.947 INFO:tasks.workunit.client.0.smithi113.stdout:4/105: dwrite da/dd/df/d14/f18 [0,4194304] 0 2024-04-12T05:08:49.948 INFO:tasks.workunit.client.0.smithi113.stdout:7/99: mkdir d0/dc/d6/d24 0 2024-04-12T05:08:49.949 INFO:tasks.workunit.client.0.smithi113.stdout:7/100: dread - d0/dd/f1c zero size 2024-04-12T05:08:49.950 INFO:tasks.workunit.client.0.smithi113.stdout:8/148: mkdir d9/d16/d1e/d26 0 2024-04-12T05:08:49.951 INFO:tasks.workunit.client.0.smithi113.stdout:9/118: mknod d1/df/c25 0 2024-04-12T05:08:49.954 INFO:tasks.workunit.client.0.smithi113.stdout:0/98: write d1/d6/f10 [4983104,1776] 0 2024-04-12T05:08:49.955 INFO:tasks.workunit.client.0.smithi113.stdout:4/106: mkdir da/d24 0 2024-04-12T05:08:49.958 INFO:tasks.workunit.client.0.smithi113.stdout:7/101: dwrite d0/f12 [0,4194304] 0 2024-04-12T05:08:49.958 INFO:tasks.workunit.client.0.smithi113.stdout:4/107: write da/dd/d1b/f23 [223744,69618] 0 2024-04-12T05:08:49.958 INFO:tasks.workunit.client.0.smithi113.stdout:0/99: mknod d1/d6/da/c1b 0 2024-04-12T05:08:49.961 INFO:tasks.workunit.client.0.smithi113.stdout:7/102: creat d0/dd/d1b/f25 x:0 0 0 2024-04-12T05:08:49.963 INFO:tasks.workunit.client.0.smithi113.stdout:7/103: read d0/dc/d6/fa [2744988,69922] 0 2024-04-12T05:08:49.964 INFO:tasks.workunit.client.0.smithi113.stdout:4/108: dread da/dd/df/d14/f18 [0,4194304] 0 2024-04-12T05:08:49.965 INFO:tasks.workunit.client.0.smithi113.stdout:0/100: rename d1/f3 to d1/d6/d16/f1c 0 2024-04-12T05:08:49.966 INFO:tasks.workunit.client.0.smithi113.stdout:9/119: creat d1/f26 x:0 0 0 2024-04-12T05:08:49.967 INFO:tasks.workunit.client.0.smithi113.stdout:9/120: fdatasync d1/d4/f13 0 2024-04-12T05:08:49.968 INFO:tasks.workunit.client.0.smithi113.stdout:8/149: getdents d9/d16/d1e 0 2024-04-12T05:08:49.968 INFO:tasks.workunit.client.0.smithi113.stdout:9/121: read d1/d4/d5/f12 [3654934,125696] 0 2024-04-12T05:08:49.969 INFO:tasks.workunit.client.0.smithi113.stdout:9/122: write d1/d4/d5/f12 [5075275,30322] 0 2024-04-12T05:08:49.970 INFO:tasks.workunit.client.0.smithi113.stdout:7/104: mkdir d0/d26 0 2024-04-12T05:08:49.970 INFO:tasks.workunit.client.0.smithi113.stdout:7/105: dread - d0/dd/f1c zero size 2024-04-12T05:08:49.971 INFO:tasks.workunit.client.0.smithi113.stdout:4/109: mknod da/c25 0 2024-04-12T05:08:49.972 INFO:tasks.workunit.client.0.smithi113.stdout:0/101: rmdir d1/d9 39 2024-04-12T05:08:49.982 INFO:tasks.workunit.client.0.smithi113.stdout:8/150: rename d9/d1b/c21 to d9/d16/c27 0 2024-04-12T05:08:49.985 INFO:tasks.workunit.client.0.smithi113.stdout:8/151: fdatasync d9/f19 0 2024-04-12T05:08:49.986 INFO:tasks.workunit.client.0.smithi113.stdout:8/152: fdatasync d9/fd 0 2024-04-12T05:08:49.989 INFO:tasks.workunit.client.0.smithi113.stdout:0/102: getdents d1/d9 0 2024-04-12T05:08:49.989 INFO:tasks.workunit.client.0.smithi113.stdout:8/153: creat d9/d16/d1e/d26/f28 x:0 0 0 2024-04-12T05:08:49.990 INFO:tasks.workunit.client.0.smithi113.stdout:0/103: creat d1/d6/f1d x:0 0 0 2024-04-12T05:08:49.991 INFO:tasks.workunit.client.0.smithi113.stdout:8/154: symlink d9/d16/d1e/l29 0 2024-04-12T05:08:49.992 INFO:tasks.workunit.client.0.smithi113.stdout:0/104: fsync d1/fe 0 2024-04-12T05:08:49.994 INFO:tasks.workunit.client.0.smithi113.stdout:0/105: read d1/fe [1281936,10747] 0 2024-04-12T05:08:49.994 INFO:tasks.workunit.client.0.smithi113.stdout:8/155: rename d9/f19 to d9/d24/f2a 0 2024-04-12T05:08:49.996 INFO:tasks.workunit.client.0.smithi113.stdout:7/106: write d0/dc/d6/f16 [444009,86662] 0 2024-04-12T05:08:49.997 INFO:tasks.workunit.client.0.smithi113.stdout:7/107: truncate d0/dc/d6/d1a/f21 644657 0 2024-04-12T05:08:49.998 INFO:tasks.workunit.client.0.smithi113.stdout:8/156: unlink f2 0 2024-04-12T05:08:49.999 INFO:tasks.workunit.client.0.smithi113.stdout:0/106: creat d1/d6/f1e x:0 0 0 2024-04-12T05:08:50.000 INFO:tasks.workunit.client.0.smithi113.stdout:7/108: fsync d0/dc/d6/fa 0 2024-04-12T05:08:50.002 INFO:tasks.workunit.client.0.smithi113.stdout:7/109: write d0/dd/f1c [784308,88158] 0 2024-04-12T05:08:50.005 INFO:tasks.workunit.client.0.smithi113.stdout:8/157: dwrite d9/d16/f1a [0,4194304] 0 2024-04-12T05:08:50.006 INFO:tasks.workunit.client.0.smithi113.stdout:0/107: write d1/f2 [43734,40500] 0 2024-04-12T05:08:50.007 INFO:tasks.workunit.client.0.smithi113.stdout:7/110: link d0/dd/d11/f19 d0/dc/f27 0 2024-04-12T05:08:50.010 INFO:tasks.workunit.client.0.smithi113.stdout:8/158: dread d9/fb [0,4194304] 0 2024-04-12T05:08:50.011 INFO:tasks.workunit.client.0.smithi113.stdout:8/159: fdatasync d9/f17 0 2024-04-12T05:08:50.011 INFO:tasks.workunit.client.0.smithi113.stdout:7/111: getdents d0/dc/d6 0 2024-04-12T05:08:50.055 INFO:tasks.workunit.client.0.smithi113.stdout:3/76: sync 2024-04-12T05:08:50.055 INFO:tasks.workunit.client.0.smithi113.stdout:5/139: sync 2024-04-12T05:08:50.057 INFO:tasks.workunit.client.0.smithi113.stdout:0/108: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:50.059 INFO:tasks.workunit.client.0.smithi113.stdout:7/112: truncate d0/dd/d11/f20 470592 0 2024-04-12T05:08:50.059 INFO:tasks.workunit.client.0.smithi113.stdout:3/77: creat d0/f1c x:0 0 0 2024-04-12T05:08:50.059 INFO:tasks.workunit.client.0.smithi113.stdout:3/78: readlink d0/d13/l14 0 2024-04-12T05:08:50.060 INFO:tasks.workunit.client.0.smithi113.stdout:8/160: symlink d9/d16/d1e/l2b 0 2024-04-12T05:08:50.062 INFO:tasks.workunit.client.0.smithi113.stdout:5/140: rename d0/de to d0/d10/d27 0 2024-04-12T05:08:50.066 INFO:tasks.workunit.client.0.smithi113.stdout:7/113: dwrite d0/dc/d6/fa [0,4194304] 0 2024-04-12T05:08:50.066 INFO:tasks.workunit.client.0.smithi113.stdout:8/161: creat d9/d16/d1c/f2c x:0 0 0 2024-04-12T05:08:50.070 INFO:tasks.workunit.client.0.smithi113.stdout:7/114: creat d0/dc/d6/f28 x:0 0 0 2024-04-12T05:08:50.078 INFO:tasks.workunit.client.0.smithi113.stdout:8/162: dwrite d9/d16/f25 [0,4194304] 0 2024-04-12T05:08:50.078 INFO:tasks.workunit.client.0.smithi113.stdout:5/141: write d0/d10/f16 [3478933,42235] 0 2024-04-12T05:08:50.079 INFO:tasks.workunit.client.0.smithi113.stdout:3/79: dwrite d0/fd [0,4194304] 0 2024-04-12T05:08:50.080 INFO:tasks.workunit.client.0.smithi113.stdout:5/142: chown d0/d10/l1c 69355 1 2024-04-12T05:08:50.081 INFO:tasks.workunit.client.0.smithi113.stdout:7/115: write d0/dd/d11/f19 [1699909,66094] 0 2024-04-12T05:08:50.081 INFO:tasks.workunit.client.0.smithi113.stdout:3/80: truncate d0/ff 1408418 0 2024-04-12T05:08:50.082 INFO:tasks.workunit.client.0.smithi113.stdout:7/116: readlink d0/dc/d6/l1d 0 2024-04-12T05:08:50.083 INFO:tasks.workunit.client.0.smithi113.stdout:3/81: creat d0/f1d x:0 0 0 2024-04-12T05:08:50.084 INFO:tasks.workunit.client.0.smithi113.stdout:7/117: write d0/dd/d11/f20 [688693,93414] 0 2024-04-12T05:08:50.085 INFO:tasks.workunit.client.0.smithi113.stdout:3/82: readlink d0/lc 0 2024-04-12T05:08:50.088 INFO:tasks.workunit.client.0.smithi113.stdout:3/83: symlink d0/l1e 0 2024-04-12T05:08:50.090 INFO:tasks.workunit.client.0.smithi113.stdout:3/84: truncate d0/f12 1023621 0 2024-04-12T05:08:50.091 INFO:tasks.workunit.client.0.smithi113.stdout:5/143: dwrite d0/d10/d27/f25 [0,4194304] 0 2024-04-12T05:08:50.091 INFO:tasks.workunit.client.0.smithi113.stdout:3/85: fsync d0/f18 0 2024-04-12T05:08:50.091 INFO:tasks.workunit.client.0.smithi113.stdout:7/118: link d0/dd/d11/f20 d0/dc/d6/d1a/f29 0 2024-04-12T05:08:50.095 INFO:tasks.workunit.client.0.smithi113.stdout:5/144: creat d0/d10/d27/d13/f28 x:0 0 0 2024-04-12T05:08:50.099 INFO:tasks.workunit.client.0.smithi113.stdout:7/119: rename d0/dd/d11/c1f to d0/dc/d6/d24/c2a 0 2024-04-12T05:08:50.099 INFO:tasks.workunit.client.0.smithi113.stdout:3/86: dwrite d0/f7 [4194304,4194304] 0 2024-04-12T05:08:50.100 INFO:tasks.workunit.client.0.smithi113.stdout:3/87: dread - d0/f1c zero size 2024-04-12T05:08:50.103 INFO:tasks.workunit.client.0.smithi113.stdout:3/88: creat d0/d13/f1f x:0 0 0 2024-04-12T05:08:50.105 INFO:tasks.workunit.client.0.smithi113.stdout:3/89: creat d0/d13/d15/f20 x:0 0 0 2024-04-12T05:08:50.109 INFO:tasks.workunit.client.0.smithi113.stdout:3/90: rename d0/f1d to d0/d13/f21 0 2024-04-12T05:08:50.109 INFO:tasks.workunit.client.0.smithi113.stdout:5/145: dwrite d0/fc [0,4194304] 0 2024-04-12T05:08:50.113 INFO:tasks.workunit.client.0.smithi113.stdout:5/146: symlink d0/d10/d27/d1d/l29 0 2024-04-12T05:08:50.113 INFO:tasks.workunit.client.0.smithi113.stdout:3/91: dread d0/ff [0,4194304] 0 2024-04-12T05:08:50.117 INFO:tasks.workunit.client.0.smithi113.stdout:3/92: rename d0/f7 to d0/f22 0 2024-04-12T05:08:50.119 INFO:tasks.workunit.client.0.smithi113.stdout:5/147: write d0/d10/d27/d13/f23 [867223,2644] 0 2024-04-12T05:08:50.121 INFO:tasks.workunit.client.0.smithi113.stdout:3/93: fdatasync d0/d13/f21 0 2024-04-12T05:08:50.122 INFO:tasks.workunit.client.0.smithi113.stdout:5/148: write d0/f14 [965603,10077] 0 2024-04-12T05:08:50.123 INFO:tasks.workunit.client.0.smithi113.stdout:5/149: chown d0/d10/d27/c15 214476 1 2024-04-12T05:08:50.124 INFO:tasks.workunit.client.0.smithi113.stdout:5/150: read - d0/d10/d27/f17 zero size 2024-04-12T05:08:50.126 INFO:tasks.workunit.client.0.smithi113.stdout:5/151: chown d0/d10/d27/f18 166236 1 2024-04-12T05:08:50.128 INFO:tasks.workunit.client.0.smithi113.stdout:5/152: write d0/d10/f16 [3998072,22534] 0 2024-04-12T05:08:50.129 INFO:tasks.workunit.client.0.smithi113.stdout:3/94: dwrite d0/fa [0,4194304] 0 2024-04-12T05:08:50.130 INFO:tasks.workunit.client.0.smithi113.stdout:5/153: write d0/d10/d27/f22 [39256,59030] 0 2024-04-12T05:08:50.136 INFO:tasks.workunit.client.0.smithi113.stdout:5/154: creat d0/d10/d27/d1d/f2a x:0 0 0 2024-04-12T05:08:50.137 INFO:tasks.workunit.client.0.smithi113.stdout:3/95: dwrite d0/fa [0,4194304] 0 2024-04-12T05:08:50.140 INFO:tasks.workunit.client.0.smithi113.stdout:5/155: dread d0/d10/d27/f22 [0,4194304] 0 2024-04-12T05:08:50.143 INFO:tasks.workunit.client.0.smithi113.stdout:5/156: unlink d0/c20 0 2024-04-12T05:08:50.146 INFO:tasks.workunit.client.0.smithi113.stdout:3/96: dwrite d0/fa [0,4194304] 0 2024-04-12T05:08:50.148 INFO:tasks.workunit.client.0.smithi113.stdout:5/157: mknod d0/d10/c2b 0 2024-04-12T05:08:50.150 INFO:tasks.workunit.client.0.smithi113.stdout:5/158: mknod d0/d10/d27/c2c 0 2024-04-12T05:08:50.151 INFO:tasks.workunit.client.0.smithi113.stdout:5/159: chown d0/d10/f1b 498711843 1 2024-04-12T05:08:50.152 INFO:tasks.workunit.client.0.smithi113.stdout:3/97: getdents d0/d13/d15 0 2024-04-12T05:08:50.152 INFO:tasks.workunit.client.0.smithi113.stdout:3/98: dread - d0/d13/f21 zero size 2024-04-12T05:08:50.153 INFO:tasks.workunit.client.0.smithi113.stdout:5/160: write d0/d10/d27/d1d/f1e [626959,121394] 0 2024-04-12T05:08:50.154 INFO:tasks.workunit.client.0.smithi113.stdout:5/161: readlink d0/d10/d27/d1d/l29 0 2024-04-12T05:08:50.154 INFO:tasks.workunit.client.0.smithi113.stdout:1/54: sync 2024-04-12T05:08:50.154 INFO:tasks.workunit.client.0.smithi113.stdout:6/101: sync 2024-04-12T05:08:50.154 INFO:tasks.workunit.client.0.smithi113.stdout:2/73: sync 2024-04-12T05:08:50.155 INFO:tasks.workunit.client.0.smithi113.stdout:1/55: chown c3 304518672 1 2024-04-12T05:08:50.156 INFO:tasks.workunit.client.0.smithi113.stdout:1/56: fdatasync fa 0 2024-04-12T05:08:50.156 INFO:tasks.workunit.client.0.smithi113.stdout:1/57: rmdir - no directory 2024-04-12T05:08:50.156 INFO:tasks.workunit.client.0.smithi113.stdout:3/99: rename d0/ff to d0/d13/d15/f23 0 2024-04-12T05:08:50.156 INFO:tasks.workunit.client.0.smithi113.stdout:1/58: chown c8 5363325 1 2024-04-12T05:08:50.157 INFO:tasks.workunit.client.0.smithi113.stdout:3/100: write d0/d13/f21 [287381,102077] 0 2024-04-12T05:08:50.158 INFO:tasks.workunit.client.0.smithi113.stdout:5/162: symlink d0/d10/l2d 0 2024-04-12T05:08:50.161 INFO:tasks.workunit.client.0.smithi113.stdout:6/102: truncate d8/de/f15 2482324 0 2024-04-12T05:08:50.167 INFO:tasks.workunit.client.0.smithi113.stdout:6/103: unlink d8/de/d13/fc 0 2024-04-12T05:08:50.167 INFO:tasks.workunit.client.0.smithi113.stdout:6/104: read - d8/de/f16 zero size 2024-04-12T05:08:50.168 INFO:tasks.workunit.client.0.smithi113.stdout:1/59: getdents . 0 2024-04-12T05:08:50.169 INFO:tasks.workunit.client.0.smithi113.stdout:5/163: truncate d0/d10/f1b 102707 0 2024-04-12T05:08:50.170 INFO:tasks.workunit.client.0.smithi113.stdout:3/101: getdents d0/d13 0 2024-04-12T05:08:50.170 INFO:tasks.workunit.client.0.smithi113.stdout:3/102: read d0/fa [1833699,7162] 0 2024-04-12T05:08:50.171 INFO:tasks.workunit.client.0.smithi113.stdout:6/105: mknod d8/de/c1f 0 2024-04-12T05:08:50.172 INFO:tasks.workunit.client.0.smithi113.stdout:1/60: dread fa [0,4194304] 0 2024-04-12T05:08:50.172 INFO:tasks.workunit.client.0.smithi113.stdout:1/61: dread - fe zero size 2024-04-12T05:08:50.173 INFO:tasks.workunit.client.0.smithi113.stdout:1/62: chown fa 47692534 1 2024-04-12T05:08:50.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:48.668726+0000 mgr.smithi192.ocppyo (mgr.24357) 20 : cluster [DBG] pgmap v14: 65 pgs: 65 active+clean; 356 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 727 KiB/s rd, 19 MiB/s wr, 245 op/s 2024-04-12T05:08:50.174 INFO:tasks.workunit.client.0.smithi113.stdout:6/106: symlink d8/de/d1c/l20 0 2024-04-12T05:08:50.174 INFO:tasks.workunit.client.0.smithi113.stdout:3/103: dread d0/f22 [0,4194304] 0 2024-04-12T05:08:50.174 INFO:tasks.workunit.client.0.smithi113.stdout:6/107: chown d8/f12 12 1 2024-04-12T05:08:50.178 INFO:tasks.workunit.client.0.smithi113.stdout:1/63: dread fd [0,4194304] 0 2024-04-12T05:08:50.181 INFO:tasks.workunit.client.0.smithi113.stdout:1/64: creat ff x:0 0 0 2024-04-12T05:08:50.182 INFO:tasks.workunit.client.0.smithi113.stdout:6/108: dwrite d8/de/f16 [0,4194304] 0 2024-04-12T05:08:50.183 INFO:tasks.workunit.client.0.smithi113.stdout:1/65: mkdir d10 0 2024-04-12T05:08:50.183 INFO:tasks.workunit.client.0.smithi113.stdout:3/104: write d0/d13/d15/f23 [740122,26231] 0 2024-04-12T05:08:50.184 INFO:tasks.workunit.client.0.smithi113.stdout:1/66: dread - fb zero size 2024-04-12T05:08:50.184 INFO:tasks.workunit.client.0.smithi113.stdout:6/109: symlink d8/de/l21 0 2024-04-12T05:08:50.186 INFO:tasks.workunit.client.0.smithi113.stdout:3/105: unlink d0/f1b 0 2024-04-12T05:08:50.188 INFO:tasks.workunit.client.0.smithi113.stdout:6/110: creat d8/f22 x:0 0 0 2024-04-12T05:08:50.189 INFO:tasks.workunit.client.0.smithi113.stdout:6/111: stat d8/f1a 0 2024-04-12T05:08:50.189 INFO:tasks.workunit.client.0.smithi113.stdout:1/67: dread f2 [0,4194304] 0 2024-04-12T05:08:50.190 INFO:tasks.workunit.client.0.smithi113.stdout:3/106: link d0/f10 d0/d13/d15/f24 0 2024-04-12T05:08:50.192 INFO:tasks.workunit.client.0.smithi113.stdout:1/68: symlink d10/l11 0 2024-04-12T05:08:50.196 INFO:tasks.workunit.client.0.smithi113.stdout:1/69: rename cc to d10/c12 0 2024-04-12T05:08:50.196 INFO:tasks.workunit.client.0.smithi113.stdout:6/112: dwrite d8/de/f19 [0,4194304] 0 2024-04-12T05:08:50.197 INFO:tasks.workunit.client.0.smithi113.stdout:3/107: dread d0/f9 [0,4194304] 0 2024-04-12T05:08:50.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:48.668726+0000 mgr.smithi192.ocppyo (mgr.24357) 20 : cluster [DBG] pgmap v14: 65 pgs: 65 active+clean; 356 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 727 KiB/s rd, 19 MiB/s wr, 245 op/s 2024-04-12T05:08:50.198 INFO:tasks.workunit.client.0.smithi113.stdout:6/113: write d8/f22 [985930,57606] 0 2024-04-12T05:08:50.199 INFO:tasks.workunit.client.0.smithi113.stdout:1/70: unlink f4 0 2024-04-12T05:08:50.200 INFO:tasks.workunit.client.0.smithi113.stdout:3/108: creat d0/d13/f25 x:0 0 0 2024-04-12T05:08:50.200 INFO:tasks.workunit.client.0.smithi113.stdout:3/109: chown d0/l1e 2 1 2024-04-12T05:08:50.201 INFO:tasks.workunit.client.0.smithi113.stdout:6/114: creat d8/de/d1c/f23 x:0 0 0 2024-04-12T05:08:50.201 INFO:tasks.workunit.client.0.smithi113.stdout:3/110: write d0/d13/f1a [120086,62862] 0 2024-04-12T05:08:50.203 INFO:tasks.workunit.client.0.smithi113.stdout:1/71: link f5 d10/f13 0 2024-04-12T05:08:50.205 INFO:tasks.workunit.client.0.smithi113.stdout:6/115: symlink d8/de/d13/l24 0 2024-04-12T05:08:50.206 INFO:tasks.workunit.client.0.smithi113.stdout:3/111: write d0/f9 [2367024,49271] 0 2024-04-12T05:08:50.206 INFO:tasks.workunit.client.0.smithi113.stdout:1/72: mkdir d10/d14 0 2024-04-12T05:08:50.206 INFO:tasks.workunit.client.0.smithi113.stdout:1/73: fdatasync fb 0 2024-04-12T05:08:50.207 INFO:tasks.workunit.client.0.smithi113.stdout:6/116: mkdir d8/de/d17/d25 0 2024-04-12T05:08:50.208 INFO:tasks.workunit.client.0.smithi113.stdout:1/74: truncate fa 1230763 0 2024-04-12T05:08:50.209 INFO:tasks.workunit.client.0.smithi113.stdout:1/75: read fa [869282,20448] 0 2024-04-12T05:08:50.209 INFO:tasks.workunit.client.0.smithi113.stdout:3/112: rename d0/f2 to d0/d13/f26 0 2024-04-12T05:08:50.212 INFO:tasks.workunit.client.0.smithi113.stdout:1/76: mknod d10/c15 0 2024-04-12T05:08:50.212 INFO:tasks.workunit.client.0.smithi113.stdout:1/77: chown c8 41123915 1 2024-04-12T05:08:50.213 INFO:tasks.workunit.client.0.smithi113.stdout:3/113: unlink d0/d13/f1a 0 2024-04-12T05:08:50.214 INFO:tasks.workunit.client.0.smithi113.stdout:6/117: mkdir d8/de/d17/d25/d26 0 2024-04-12T05:08:50.215 INFO:tasks.workunit.client.0.smithi113.stdout:3/114: symlink d0/d13/d15/l27 0 2024-04-12T05:08:50.216 INFO:tasks.workunit.client.0.smithi113.stdout:3/115: dread - d0/f6 zero size 2024-04-12T05:08:50.216 INFO:tasks.workunit.client.0.smithi113.stdout:6/118: creat d8/de/d17/f27 x:0 0 0 2024-04-12T05:08:50.217 INFO:tasks.workunit.client.0.smithi113.stdout:6/119: dread - d8/de/d1c/f23 zero size 2024-04-12T05:08:50.218 INFO:tasks.workunit.client.0.smithi113.stdout:3/116: mkdir d0/d13/d15/d28 0 2024-04-12T05:08:50.220 INFO:tasks.workunit.client.0.smithi113.stdout:3/117: write d0/d13/f21 [116768,80448] 0 2024-04-12T05:08:50.220 INFO:tasks.workunit.client.0.smithi113.stdout:6/120: creat d8/de/d17/d25/f28 x:0 0 0 2024-04-12T05:08:50.222 INFO:tasks.workunit.client.0.smithi113.stdout:6/121: mknod d8/c29 0 2024-04-12T05:08:50.223 INFO:tasks.workunit.client.0.smithi113.stdout:3/118: creat d0/f29 x:0 0 0 2024-04-12T05:08:50.225 INFO:tasks.workunit.client.0.smithi113.stdout:3/119: write d0/d13/f21 [1095609,34765] 0 2024-04-12T05:08:50.229 INFO:tasks.workunit.client.0.smithi113.stdout:3/120: mkdir d0/d13/d15/d28/d2a 0 2024-04-12T05:08:50.232 INFO:tasks.workunit.client.0.smithi113.stdout:3/121: creat d0/d13/d15/d28/d2a/f2b x:0 0 0 2024-04-12T05:08:50.233 INFO:tasks.workunit.client.0.smithi113.stdout:3/122: readlink d0/d13/l14 0 2024-04-12T05:08:50.235 INFO:tasks.workunit.client.0.smithi113.stdout:3/123: write d0/f18 [3625877,89077] 0 2024-04-12T05:08:50.240 INFO:tasks.workunit.client.0.smithi113.stdout:3/124: rename d0/c11 to d0/d13/d15/d28/c2c 0 2024-04-12T05:08:50.242 INFO:tasks.workunit.client.0.smithi113.stdout:3/125: readlink d0/d13/d15/l27 0 2024-04-12T05:08:50.243 INFO:tasks.workunit.client.0.smithi113.stdout:3/126: mkdir d0/d13/d15/d28/d2a/d2d 0 2024-04-12T05:08:50.250 INFO:tasks.workunit.client.0.smithi113.stdout:1/78: getdents d10 0 2024-04-12T05:08:50.255 INFO:tasks.workunit.client.0.smithi113.stdout:3/127: getdents d0/d13/d15 0 2024-04-12T05:08:50.258 INFO:tasks.workunit.client.0.smithi113.stdout:3/128: rename d0/d13/f26 to d0/d13/d15/f2e 0 2024-04-12T05:08:50.258 INFO:tasks.workunit.client.0.smithi113.stdout:3/129: rename d0 to d0/d13/d15/d2f 22 2024-04-12T05:08:50.259 INFO:tasks.workunit.client.0.smithi113.stdout:6/122: truncate d8/de/f19 2795964 0 2024-04-12T05:08:50.260 INFO:tasks.workunit.client.0.smithi113.stdout:3/130: read d0/d13/d15/f23 [364530,122194] 0 2024-04-12T05:08:50.261 INFO:tasks.workunit.client.0.smithi113.stdout:6/123: write d8/de/d13/fb [2422277,6482] 0 2024-04-12T05:08:50.261 INFO:tasks.workunit.client.0.smithi113.stdout:6/124: chown l4 0 1 2024-04-12T05:08:50.262 INFO:tasks.workunit.client.0.smithi113.stdout:6/125: readlink l1 0 2024-04-12T05:08:50.264 INFO:tasks.workunit.client.0.smithi113.stdout:1/79: write fd [3859652,70558] 0 2024-04-12T05:08:50.265 INFO:tasks.workunit.client.0.smithi113.stdout:1/80: write ff [176581,43903] 0 2024-04-12T05:08:50.266 INFO:tasks.workunit.client.0.smithi113.stdout:1/81: write fb [846348,81941] 0 2024-04-12T05:08:50.270 INFO:tasks.workunit.client.0.smithi113.stdout:1/82: dread fa [0,4194304] 0 2024-04-12T05:08:50.271 INFO:tasks.workunit.client.0.smithi113.stdout:6/126: write d8/de/f15 [470436,102763] 0 2024-04-12T05:08:50.275 INFO:tasks.workunit.client.0.smithi113.stdout:6/127: symlink d8/de/d17/d25/d26/l2a 0 2024-04-12T05:08:50.279 INFO:tasks.workunit.client.0.smithi113.stdout:3/131: dwrite d0/d13/d15/f24 [0,4194304] 0 2024-04-12T05:08:50.279 INFO:tasks.workunit.client.0.smithi113.stdout:1/83: dread fb [0,4194304] 0 2024-04-12T05:08:50.280 INFO:tasks.workunit.client.0.smithi113.stdout:3/132: fsync d0/d13/d15/d28/d2a/f2b 0 2024-04-12T05:08:50.283 INFO:tasks.workunit.client.0.smithi113.stdout:3/133: dread d0/d13/d15/f2e [0,4194304] 0 2024-04-12T05:08:50.284 INFO:tasks.workunit.client.0.smithi113.stdout:1/84: link f5 d10/f16 0 2024-04-12T05:08:50.287 INFO:tasks.workunit.client.0.smithi113.stdout:6/128: write d8/de/d13/fa [5337208,73279] 0 2024-04-12T05:08:50.290 INFO:tasks.workunit.client.0.smithi113.stdout:3/134: dwrite d0/d13/f25 [0,4194304] 0 2024-04-12T05:08:50.291 INFO:tasks.workunit.client.0.smithi113.stdout:6/129: mkdir d8/de/d13/d2b 0 2024-04-12T05:08:50.291 INFO:tasks.workunit.client.0.smithi113.stdout:3/135: truncate d0/d13/f21 1704595 0 2024-04-12T05:08:50.291 INFO:tasks.workunit.client.0.smithi113.stdout:1/85: fsync d10/f16 0 2024-04-12T05:08:50.293 INFO:tasks.workunit.client.0.smithi113.stdout:3/136: creat d0/f30 x:0 0 0 2024-04-12T05:08:50.295 INFO:tasks.workunit.client.0.smithi113.stdout:6/130: mkdir d8/de/d1c/d2c 0 2024-04-12T05:08:50.297 INFO:tasks.workunit.client.0.smithi113.stdout:1/86: write d10/f13 [30696,65559] 0 2024-04-12T05:08:50.298 INFO:tasks.workunit.client.0.smithi113.stdout:1/87: write ff [144346,61260] 0 2024-04-12T05:08:50.299 INFO:tasks.workunit.client.0.smithi113.stdout:3/137: creat d0/d13/d15/d28/f31 x:0 0 0 2024-04-12T05:08:50.300 INFO:tasks.workunit.client.0.smithi113.stdout:3/138: stat d0/c5 0 2024-04-12T05:08:50.302 INFO:tasks.workunit.client.0.smithi113.stdout:6/131: fdatasync d8/de/f19 0 2024-04-12T05:08:50.304 INFO:tasks.workunit.client.0.smithi113.stdout:3/139: symlink d0/d13/d15/d28/l32 0 2024-04-12T05:08:50.305 INFO:tasks.workunit.client.0.smithi113.stdout:1/88: creat d10/d14/f17 x:0 0 0 2024-04-12T05:08:50.305 INFO:tasks.workunit.client.0.smithi113.stdout:1/89: dread - fe zero size 2024-04-12T05:08:50.305 INFO:tasks.workunit.client.0.smithi113.stdout:9/123: sync 2024-04-12T05:08:50.306 INFO:tasks.workunit.client.0.smithi113.stdout:4/110: sync 2024-04-12T05:08:50.312 INFO:tasks.workunit.client.0.smithi113.stdout:4/111: rename da/dd/df/d1d/d22 to da/d26 0 2024-04-12T05:08:50.312 INFO:tasks.workunit.client.0.smithi113.stdout:3/140: dwrite d0/f12 [0,4194304] 0 2024-04-12T05:08:50.313 INFO:tasks.workunit.client.0.smithi113.stdout:9/124: dread d1/d4/f13 [4194304,4194304] 0 2024-04-12T05:08:50.314 INFO:tasks.workunit.client.0.smithi113.stdout:1/90: getdents d10/d14 0 2024-04-12T05:08:50.316 INFO:tasks.workunit.client.0.smithi113.stdout:1/91: read d10/f13 [841752,27256] 0 2024-04-12T05:08:50.316 INFO:tasks.workunit.client.0.smithi113.stdout:4/112: dread da/f1a [0,4194304] 0 2024-04-12T05:08:50.320 INFO:tasks.workunit.client.0.smithi113.stdout:6/132: dwrite d8/de/f19 [0,4194304] 0 2024-04-12T05:08:50.341 INFO:tasks.workunit.client.0.smithi113.stdout:9/125: mkdir d1/d27 0 2024-04-12T05:08:50.342 INFO:tasks.workunit.client.0.smithi113.stdout:4/113: mknod da/c27 0 2024-04-12T05:08:50.343 INFO:tasks.workunit.client.0.smithi113.stdout:6/133: creat d8/f2d x:0 0 0 2024-04-12T05:08:50.344 INFO:tasks.workunit.client.0.smithi113.stdout:4/114: creat da/dd/d1b/f28 x:0 0 0 2024-04-12T05:08:50.345 INFO:tasks.workunit.client.0.smithi113.stdout:9/126: mknod d1/d27/c28 0 2024-04-12T05:08:50.345 INFO:tasks.workunit.client.0.smithi113.stdout:6/134: getdents d8/de/d13/d2b 0 2024-04-12T05:08:50.347 INFO:tasks.workunit.client.0.smithi113.stdout:9/127: mkdir d1/df/d29 0 2024-04-12T05:08:50.348 INFO:tasks.workunit.client.0.smithi113.stdout:6/135: mkdir d8/de/d13/d2b/d2e 0 2024-04-12T05:08:50.349 INFO:tasks.workunit.client.0.smithi113.stdout:6/136: dread - d8/de/d1c/f23 zero size 2024-04-12T05:08:50.350 INFO:tasks.workunit.client.0.smithi113.stdout:1/92: truncate fb 795054 0 2024-04-12T05:08:50.352 INFO:tasks.workunit.client.0.smithi113.stdout:4/115: dwrite da/dd/df/f10 [0,4194304] 0 2024-04-12T05:08:50.353 INFO:tasks.workunit.client.0.smithi113.stdout:6/137: write d8/de/f16 [365448,90564] 0 2024-04-12T05:08:50.354 INFO:tasks.workunit.client.0.smithi113.stdout:4/116: readlink l4 0 2024-04-12T05:08:50.357 INFO:tasks.workunit.client.0.smithi113.stdout:4/117: write da/dd/f21 [480552,59660] 0 2024-04-12T05:08:50.357 INFO:tasks.workunit.client.0.smithi113.stdout:4/118: chown da/dd/d1b 40 1 2024-04-12T05:08:50.359 INFO:tasks.workunit.client.0.smithi113.stdout:9/128: dwrite d1/df/f11 [0,4194304] 0 2024-04-12T05:08:50.359 INFO:tasks.workunit.client.0.smithi113.stdout:4/119: chown da/dd/df/f10 530545381 1 2024-04-12T05:08:50.364 INFO:tasks.workunit.client.0.smithi113.stdout:9/129: mknod d1/c2a 0 2024-04-12T05:08:50.364 INFO:tasks.workunit.client.0.smithi113.stdout:4/120: dread da/dd/f21 [0,4194304] 0 2024-04-12T05:08:50.365 INFO:tasks.workunit.client.0.smithi113.stdout:4/121: fdatasync da/dd/df/f1c 0 2024-04-12T05:08:50.367 INFO:tasks.workunit.client.0.smithi113.stdout:1/93: dread fb [0,4194304] 0 2024-04-12T05:08:50.369 INFO:tasks.workunit.client.0.smithi113.stdout:1/94: read fa [162811,126216] 0 2024-04-12T05:08:50.369 INFO:tasks.workunit.client.0.smithi113.stdout:1/95: fdatasync fd 0 2024-04-12T05:08:50.370 INFO:tasks.workunit.client.0.smithi113.stdout:9/130: dread d1/d4/f13 [4194304,4194304] 0 2024-04-12T05:08:50.372 INFO:tasks.workunit.client.0.smithi113.stdout:4/122: dwrite da/dd/df/f1c [0,4194304] 0 2024-04-12T05:08:50.375 INFO:tasks.workunit.client.0.smithi113.stdout:4/123: symlink da/dd/l29 0 2024-04-12T05:08:50.376 INFO:tasks.workunit.client.0.smithi113.stdout:4/124: fdatasync da/dd/df/f10 0 2024-04-12T05:08:50.435 INFO:tasks.workunit.client.0.smithi113.stdout:8/163: sync 2024-04-12T05:08:50.435 INFO:tasks.workunit.client.0.smithi113.stdout:0/109: sync 2024-04-12T05:08:50.435 INFO:tasks.workunit.client.0.smithi113.stdout:9/131: getdents d1/d27 0 2024-04-12T05:08:50.436 INFO:tasks.workunit.client.0.smithi113.stdout:1/96: write d10/f16 [3162853,96817] 0 2024-04-12T05:08:50.437 INFO:tasks.workunit.client.0.smithi113.stdout:8/164: write d9/d16/f1a [8116227,114485] 0 2024-04-12T05:08:50.438 INFO:tasks.workunit.client.0.smithi113.stdout:0/110: unlink d1/f13 0 2024-04-12T05:08:50.440 INFO:tasks.workunit.client.0.smithi113.stdout:0/111: dread - d1/d6/f1e zero size 2024-04-12T05:08:50.442 INFO:tasks.workunit.client.0.smithi113.stdout:8/165: creat d9/d16/d1c/f2d x:0 0 0 2024-04-12T05:08:50.443 INFO:tasks.workunit.client.0.smithi113.stdout:9/132: dread d1/df/f11 [0,4194304] 0 2024-04-12T05:08:50.443 INFO:tasks.workunit.client.0.smithi113.stdout:8/166: fdatasync d9/f17 0 2024-04-12T05:08:50.443 INFO:tasks.workunit.client.0.smithi113.stdout:0/112: unlink d1/d6/f10 0 2024-04-12T05:08:50.444 INFO:tasks.workunit.client.0.smithi113.stdout:4/125: dread da/dd/df/f1c [0,4194304] 0 2024-04-12T05:08:50.445 INFO:tasks.workunit.client.0.smithi113.stdout:4/126: write da/dd/d1b/f23 [313399,104629] 0 2024-04-12T05:08:50.447 INFO:tasks.workunit.client.0.smithi113.stdout:9/133: rmdir d1/d4 39 2024-04-12T05:08:50.447 INFO:tasks.workunit.client.0.smithi113.stdout:8/167: dread d9/fb [0,4194304] 0 2024-04-12T05:08:50.448 INFO:tasks.workunit.client.0.smithi113.stdout:9/134: write d1/fc [4826446,55062] 0 2024-04-12T05:08:50.449 INFO:tasks.workunit.client.0.smithi113.stdout:4/127: write da/dd/df/d14/f18 [5128865,21884] 0 2024-04-12T05:08:50.452 INFO:tasks.workunit.client.0.smithi113.stdout:1/97: truncate d10/f13 1516608 0 2024-04-12T05:08:50.453 INFO:tasks.workunit.client.0.smithi113.stdout:8/168: mknod d9/d16/c2e 0 2024-04-12T05:08:50.454 INFO:tasks.workunit.client.0.smithi113.stdout:8/169: truncate d9/f17 4339461 0 2024-04-12T05:08:50.457 INFO:tasks.workunit.client.0.smithi113.stdout:1/98: mknod d10/d14/c18 0 2024-04-12T05:08:50.460 INFO:tasks.workunit.client.0.smithi113.stdout:8/170: rename d9/d16/c2e to d9/d24/c2f 0 2024-04-12T05:08:50.460 INFO:tasks.workunit.client.0.smithi113.stdout:8/171: fsync d9/fd 0 2024-04-12T05:08:50.462 INFO:tasks.workunit.client.0.smithi113.stdout:0/113: getdents d1/d6 0 2024-04-12T05:08:50.463 INFO:tasks.workunit.client.0.smithi113.stdout:4/128: write da/dd/df/d14/f19 [1492817,95853] 0 2024-04-12T05:08:50.464 INFO:tasks.workunit.client.0.smithi113.stdout:4/129: write da/dd/df/d14/f19 [1507876,27166] 0 2024-04-12T05:08:50.464 INFO:tasks.workunit.client.0.smithi113.stdout:1/99: rename d10/c15 to d10/d14/c19 0 2024-04-12T05:08:50.465 INFO:tasks.workunit.client.0.smithi113.stdout:1/100: readlink d10/l11 0 2024-04-12T05:08:50.465 INFO:tasks.workunit.client.0.smithi113.stdout:1/101: readlink d10/l11 0 2024-04-12T05:08:50.466 INFO:tasks.workunit.client.0.smithi113.stdout:8/172: creat d9/d24/f30 x:0 0 0 2024-04-12T05:08:50.467 INFO:tasks.workunit.client.0.smithi113.stdout:9/135: write d1/d4/d5/f12 [3087285,71483] 0 2024-04-12T05:08:50.468 INFO:tasks.workunit.client.0.smithi113.stdout:4/130: rmdir da/dd/d1b 39 2024-04-12T05:08:50.471 INFO:tasks.workunit.client.0.smithi113.stdout:8/173: unlink l3 0 2024-04-12T05:08:50.472 INFO:tasks.workunit.client.0.smithi113.stdout:9/136: mknod d1/d27/c2b 0 2024-04-12T05:08:50.472 INFO:tasks.workunit.client.0.smithi113.stdout:9/137: chown c0 995015 1 2024-04-12T05:08:50.473 INFO:tasks.workunit.client.0.smithi113.stdout:4/131: symlink da/l2a 0 2024-04-12T05:08:50.474 INFO:tasks.workunit.client.0.smithi113.stdout:4/132: read da/dd/df/d14/f18 [2590601,13204] 0 2024-04-12T05:08:50.474 INFO:tasks.workunit.client.0.smithi113.stdout:8/174: symlink d9/d16/d1c/l31 0 2024-04-12T05:08:50.474 INFO:tasks.workunit.client.0.smithi113.stdout:8/175: chown d9/f17 0 1 2024-04-12T05:08:50.476 INFO:tasks.workunit.client.0.smithi113.stdout:9/138: getdents d1/d4/d5/d6/d15 0 2024-04-12T05:08:50.480 INFO:tasks.workunit.client.0.smithi113.stdout:9/139: fsync d1/d4/f18 0 2024-04-12T05:08:50.481 INFO:tasks.workunit.client.0.smithi113.stdout:4/133: dwrite da/dd/df/d14/f18 [4194304,4194304] 0 2024-04-12T05:08:50.481 INFO:tasks.workunit.client.0.smithi113.stdout:9/140: read d1/f14 [214697,117243] 0 2024-04-12T05:08:50.485 INFO:tasks.workunit.client.0.smithi113.stdout:4/134: write da/dd/f21 [263723,124033] 0 2024-04-12T05:08:50.486 INFO:tasks.workunit.client.0.smithi113.stdout:4/135: readlink l4 0 2024-04-12T05:08:50.486 INFO:tasks.workunit.client.0.smithi113.stdout:9/141: write d1/df/f11 [258340,98754] 0 2024-04-12T05:08:50.487 INFO:tasks.workunit.client.0.smithi113.stdout:9/142: write d1/d4/f13 [669505,36599] 0 2024-04-12T05:08:50.487 INFO:tasks.workunit.client.0.smithi113.stdout:4/136: truncate da/dd/f21 1285288 0 2024-04-12T05:08:50.488 INFO:tasks.workunit.client.0.smithi113.stdout:4/137: chown c7 13916 1 2024-04-12T05:08:50.488 INFO:tasks.workunit.client.0.smithi113.stdout:4/138: readlink l4 0 2024-04-12T05:08:50.489 INFO:tasks.workunit.client.0.smithi113.stdout:4/139: fdatasync da/fb 0 2024-04-12T05:08:50.493 INFO:tasks.workunit.client.0.smithi113.stdout:4/140: mkdir da/d2b 0 2024-04-12T05:08:50.494 INFO:tasks.workunit.client.0.smithi113.stdout:4/141: fsync da/dd/df/d14/f19 0 2024-04-12T05:08:50.495 INFO:tasks.workunit.client.0.smithi113.stdout:9/143: dwrite d1/d4/d5/f12 [0,4194304] 0 2024-04-12T05:08:50.496 INFO:tasks.workunit.client.0.smithi113.stdout:4/142: dread da/f1a [0,4194304] 0 2024-04-12T05:08:50.500 INFO:tasks.workunit.client.0.smithi113.stdout:4/143: creat da/dd/d1b/f2c x:0 0 0 2024-04-12T05:08:50.501 INFO:tasks.workunit.client.0.smithi113.stdout:9/144: dwrite d1/fc [0,4194304] 0 2024-04-12T05:08:50.502 INFO:tasks.workunit.client.0.smithi113.stdout:4/144: rename da/c25 to da/de/c2d 0 2024-04-12T05:08:50.503 INFO:tasks.workunit.client.0.smithi113.stdout:9/145: mknod d1/df/d29/c2c 0 2024-04-12T05:08:50.506 INFO:tasks.workunit.client.0.smithi113.stdout:9/146: mknod d1/d27/c2d 0 2024-04-12T05:08:50.506 INFO:tasks.workunit.client.0.smithi113.stdout:9/147: stat d1 0 2024-04-12T05:08:50.509 INFO:tasks.workunit.client.0.smithi113.stdout:8/176: truncate d9/fb 922346 0 2024-04-12T05:08:50.510 INFO:tasks.workunit.client.0.smithi113.stdout:9/148: symlink d1/d4/d5/l2e 0 2024-04-12T05:08:50.512 INFO:tasks.workunit.client.0.smithi113.stdout:4/145: dwrite da/dd/d1b/f23 [0,4194304] 0 2024-04-12T05:08:50.516 INFO:tasks.workunit.client.0.smithi113.stdout:8/177: mkdir d9/d16/d1e/d26/d32 0 2024-04-12T05:08:50.523 INFO:tasks.workunit.client.0.smithi113.stdout:1/102: dwrite d10/f13 [0,4194304] 0 2024-04-12T05:08:50.525 INFO:tasks.workunit.client.0.smithi113.stdout:4/146: dwrite da/dd/df/f10 [0,4194304] 0 2024-04-12T05:08:50.532 INFO:tasks.workunit.client.0.smithi113.stdout:8/178: link d9/d16/d1e/l29 d9/d16/d1e/d26/d32/l33 0 2024-04-12T05:08:50.533 INFO:tasks.workunit.client.0.smithi113.stdout:4/147: creat da/de/d17/f2e x:0 0 0 2024-04-12T05:08:50.536 INFO:tasks.workunit.client.0.smithi113.stdout:8/179: chown d9/d24/c2f 506 1 2024-04-12T05:08:50.539 INFO:tasks.workunit.client.0.smithi113.stdout:1/103: dwrite d10/f16 [0,4194304] 0 2024-04-12T05:08:50.540 INFO:tasks.workunit.client.0.smithi113.stdout:9/149: dwrite d1/f14 [0,4194304] 0 2024-04-12T05:08:50.541 INFO:tasks.workunit.client.0.smithi113.stdout:1/104: write fe [533198,65389] 0 2024-04-12T05:08:50.546 INFO:tasks.workunit.client.0.smithi113.stdout:1/105: rename c6 to d10/d14/c1a 0 2024-04-12T05:08:50.547 INFO:tasks.workunit.client.0.smithi113.stdout:1/106: dread fb [0,4194304] 0 2024-04-12T05:08:50.548 INFO:tasks.workunit.client.0.smithi113.stdout:9/150: link d1/f26 d1/d4/d5/d6/d15/f2f 0 2024-04-12T05:08:50.548 INFO:tasks.workunit.client.0.smithi113.stdout:9/151: stat d1/d4/f18 0 2024-04-12T05:08:50.550 INFO:tasks.workunit.client.0.smithi113.stdout:7/120: sync 2024-04-12T05:08:50.550 INFO:tasks.workunit.client.0.smithi113.stdout:3/141: sync 2024-04-12T05:08:50.550 INFO:tasks.workunit.client.0.smithi113.stdout:2/74: sync 2024-04-12T05:08:50.550 INFO:tasks.workunit.client.0.smithi113.stdout:5/164: sync 2024-04-12T05:08:50.551 INFO:tasks.workunit.client.0.smithi113.stdout:1/107: rmdir d10 39 2024-04-12T05:08:50.552 INFO:tasks.workunit.client.0.smithi113.stdout:7/121: fdatasync d0/dd/d1b/f25 0 2024-04-12T05:08:50.552 INFO:tasks.workunit.client.0.smithi113.stdout:2/75: write d8/fd [2172183,70596] 0 2024-04-12T05:08:50.554 INFO:tasks.workunit.client.0.smithi113.stdout:9/152: unlink d1/df/f11 0 2024-04-12T05:08:50.558 INFO:tasks.workunit.client.0.smithi113.stdout:3/142: rmdir d0/d13/d15/d28/d2a 39 2024-04-12T05:08:50.560 INFO:tasks.workunit.client.0.smithi113.stdout:2/76: mknod d8/c11 0 2024-04-12T05:08:50.561 INFO:tasks.workunit.client.0.smithi113.stdout:7/122: creat d0/dc/d6/d1a/f2b x:0 0 0 2024-04-12T05:08:50.561 INFO:tasks.workunit.client.0.smithi113.stdout:9/153: mkdir d1/d4/d30 0 2024-04-12T05:08:50.563 INFO:tasks.workunit.client.0.smithi113.stdout:1/108: rmdir d10 39 2024-04-12T05:08:50.565 INFO:tasks.workunit.client.0.smithi113.stdout:9/154: dread d1/d4/f18 [0,4194304] 0 2024-04-12T05:08:50.565 INFO:tasks.workunit.client.0.smithi113.stdout:3/143: symlink d0/d13/d15/l33 0 2024-04-12T05:08:50.566 INFO:tasks.workunit.client.0.smithi113.stdout:7/123: mknod d0/dc/d6/d24/c2c 0 2024-04-12T05:08:50.568 INFO:tasks.workunit.client.0.smithi113.stdout:5/165: creat d0/f2e x:0 0 0 2024-04-12T05:08:50.569 INFO:tasks.workunit.client.0.smithi113.stdout:7/124: write d0/dd/f1c [102123,87542] 0 2024-04-12T05:08:50.569 INFO:tasks.workunit.client.0.smithi113.stdout:8/180: dwrite d9/fb [0,4194304] 0 2024-04-12T05:08:50.572 INFO:tasks.workunit.client.0.smithi113.stdout:9/155: mknod d1/d24/c31 0 2024-04-12T05:08:50.576 INFO:tasks.workunit.client.0.smithi113.stdout:8/181: mkdir d9/d34 0 2024-04-12T05:08:50.578 INFO:tasks.workunit.client.0.smithi113.stdout:1/109: dread d10/f16 [0,4194304] 0 2024-04-12T05:08:50.580 INFO:tasks.workunit.client.0.smithi113.stdout:3/144: truncate d0/f22 4657669 0 2024-04-12T05:08:50.582 INFO:tasks.workunit.client.0.smithi113.stdout:7/125: dread d0/dc/d6/fa [0,4194304] 0 2024-04-12T05:08:50.582 INFO:tasks.workunit.client.0.smithi113.stdout:2/77: dwrite d8/d9/de/f10 [0,4194304] 0 2024-04-12T05:08:50.582 INFO:tasks.workunit.client.0.smithi113.stdout:9/156: rename d1/d27/c28 to d1/d4/d16/c32 0 2024-04-12T05:08:50.584 INFO:tasks.workunit.client.0.smithi113.stdout:5/166: getdents d0/d10/d27 0 2024-04-12T05:08:50.584 INFO:tasks.workunit.client.0.smithi113.stdout:5/167: read - d0/d10/d27/f26 zero size 2024-04-12T05:08:50.585 INFO:tasks.workunit.client.0.smithi113.stdout:1/110: symlink d10/d14/l1b 0 2024-04-12T05:08:50.586 INFO:tasks.workunit.client.0.smithi113.stdout:9/157: dread d1/d4/d5/f7 [0,4194304] 0 2024-04-12T05:08:50.587 INFO:tasks.workunit.client.0.smithi113.stdout:2/78: chown d8/fd 6953 1 2024-04-12T05:08:50.591 INFO:tasks.workunit.client.0.smithi113.stdout:7/126: unlink d0/dd/l17 0 2024-04-12T05:08:50.591 INFO:tasks.workunit.client.0.smithi113.stdout:7/127: chown d0/dc/d6/d1a 1555093 1 2024-04-12T05:08:50.592 INFO:tasks.workunit.client.0.smithi113.stdout:7/128: chown d0/dc/d6/d1a/f2b 167611237 1 2024-04-12T05:08:50.592 INFO:tasks.workunit.client.0.smithi113.stdout:5/168: creat d0/d10/d27/d13/f2f x:0 0 0 2024-04-12T05:08:50.593 INFO:tasks.workunit.client.0.smithi113.stdout:7/129: readlink d0/dc/l3 0 2024-04-12T05:08:50.593 INFO:tasks.workunit.client.0.smithi113.stdout:3/145: mkdir d0/d13/d15/d28/d2a/d34 0 2024-04-12T05:08:50.594 INFO:tasks.workunit.client.0.smithi113.stdout:9/158: creat d1/d4/f33 x:0 0 0 2024-04-12T05:08:50.595 INFO:tasks.workunit.client.0.smithi113.stdout:2/79: mkdir d8/d9/de/d12 0 2024-04-12T05:08:50.595 INFO:tasks.workunit.client.0.smithi113.stdout:9/159: rename d1/d4/d16 to d1/d4/d16/d34 22 2024-04-12T05:08:50.595 INFO:tasks.workunit.client.0.smithi113.stdout:5/169: symlink d0/d10/l30 0 2024-04-12T05:08:50.600 INFO:tasks.workunit.client.0.smithi113.stdout:9/160: dread d1/d4/f18 [0,4194304] 0 2024-04-12T05:08:50.601 INFO:tasks.workunit.client.0.smithi113.stdout:7/130: dwrite d0/dc/d6/d1a/f21 [0,4194304] 0 2024-04-12T05:08:50.604 INFO:tasks.workunit.client.0.smithi113.stdout:9/161: fsync d1/f26 0 2024-04-12T05:08:50.606 INFO:tasks.workunit.client.0.smithi113.stdout:7/131: mkdir d0/dd/d2d 0 2024-04-12T05:08:50.606 INFO:tasks.workunit.client.0.smithi113.stdout:7/132: dread - d0/dc/d6/f28 zero size 2024-04-12T05:08:50.613 INFO:tasks.workunit.client.0.smithi113.stdout:9/162: mkdir d1/d4/d16/d35 0 2024-04-12T05:08:50.614 INFO:tasks.workunit.client.0.smithi113.stdout:7/133: symlink d0/dc/d6/l2e 0 2024-04-12T05:08:50.617 INFO:tasks.workunit.client.0.smithi113.stdout:9/163: link d1/d4/f33 d1/df/f36 0 2024-04-12T05:08:50.617 INFO:tasks.workunit.client.0.smithi113.stdout:7/134: mknod d0/d26/c2f 0 2024-04-12T05:08:50.621 INFO:tasks.workunit.client.0.smithi113.stdout:9/164: dread d1/d4/d5/f12 [0,4194304] 0 2024-04-12T05:08:50.625 INFO:tasks.workunit.client.0.smithi113.stdout:3/146: dwrite d0/f22 [0,4194304] 0 2024-04-12T05:08:50.629 INFO:tasks.workunit.client.0.smithi113.stdout:3/147: rename d0/d13/d15/f23 to d0/d13/d15/d28/d2a/d2d/f35 0 2024-04-12T05:08:50.634 INFO:tasks.workunit.client.0.smithi113.stdout:9/165: dwrite d1/d4/f33 [0,4194304] 0 2024-04-12T05:08:50.637 INFO:tasks.workunit.client.0.smithi113.stdout:3/148: dwrite d0/f12 [0,4194304] 0 2024-04-12T05:08:50.640 INFO:tasks.workunit.client.0.smithi113.stdout:9/166: rename d1/d4/d16 to d1/d4/d30/d37 0 2024-04-12T05:08:50.643 INFO:tasks.workunit.client.0.smithi113.stdout:3/149: creat d0/d13/d15/d28/d2a/f36 x:0 0 0 2024-04-12T05:08:50.644 INFO:tasks.workunit.client.0.smithi113.stdout:9/167: dread d1/d4/d5/f7 [0,4194304] 0 2024-04-12T05:08:50.647 INFO:tasks.workunit.client.0.smithi113.stdout:9/168: unlink d1/d4/f13 0 2024-04-12T05:08:50.648 INFO:tasks.workunit.client.0.smithi113.stdout:9/169: read d1/fc [3129314,58894] 0 2024-04-12T05:08:50.652 INFO:tasks.workunit.client.0.smithi113.stdout:9/170: dread d1/f14 [0,4194304] 0 2024-04-12T05:08:50.652 INFO:tasks.workunit.client.0.smithi113.stdout:9/171: chown d1/f14 30 1 2024-04-12T05:08:50.653 INFO:tasks.workunit.client.0.smithi113.stdout:9/172: stat d1/f14 0 2024-04-12T05:08:50.654 INFO:tasks.workunit.client.0.smithi113.stdout:9/173: getdents d1/d4/d5/d22 0 2024-04-12T05:08:50.656 INFO:tasks.workunit.client.0.smithi113.stdout:9/174: write d1/d4/d5/f12 [5795014,122906] 0 2024-04-12T05:08:50.658 INFO:tasks.workunit.client.0.smithi113.stdout:9/175: mkdir d1/d4/d38 0 2024-04-12T05:08:50.660 INFO:tasks.workunit.client.0.smithi113.stdout:9/176: symlink d1/d4/l39 0 2024-04-12T05:08:50.664 INFO:tasks.workunit.client.0.smithi113.stdout:9/177: dwrite d1/d4/d5/f12 [0,4194304] 0 2024-04-12T05:08:50.665 INFO:tasks.workunit.client.0.smithi113.stdout:9/178: truncate d1/f14 4608128 0 2024-04-12T05:08:50.672 INFO:tasks.workunit.client.0.smithi113.stdout:9/179: dwrite d1/d4/f33 [0,4194304] 0 2024-04-12T05:08:50.676 INFO:tasks.workunit.client.0.smithi113.stdout:9/180: symlink d1/d4/d30/d37/d35/l3a 0 2024-04-12T05:08:50.677 INFO:tasks.workunit.client.0.smithi113.stdout:9/181: dread - d1/d4/d5/d6/d15/f2f zero size 2024-04-12T05:08:50.677 INFO:tasks.workunit.client.0.smithi113.stdout:9/182: stat d1/d27/c2b 0 2024-04-12T05:08:50.678 INFO:tasks.workunit.client.0.smithi113.stdout:6/138: sync 2024-04-12T05:08:50.680 INFO:tasks.workunit.client.0.smithi113.stdout:6/139: mknod d8/c2f 0 2024-04-12T05:08:50.680 INFO:tasks.workunit.client.0.smithi113.stdout:6/140: fdatasync d8/f2d 0 2024-04-12T05:08:50.681 INFO:tasks.workunit.client.0.smithi113.stdout:9/183: truncate d1/d4/d5/d6/d15/f2f 162511 0 2024-04-12T05:08:50.681 INFO:tasks.workunit.client.0.smithi113.stdout:6/141: chown d8/c14 913907257 1 2024-04-12T05:08:50.682 INFO:tasks.workunit.client.0.smithi113.stdout:6/142: write d8/de/f16 [3328671,76455] 0 2024-04-12T05:08:50.682 INFO:tasks.workunit.client.0.smithi113.stdout:6/143: dread - d8/f2d zero size 2024-04-12T05:08:50.683 INFO:tasks.workunit.client.0.smithi113.stdout:9/184: creat d1/d4/d5/d6/f3b x:0 0 0 2024-04-12T05:08:50.685 INFO:tasks.workunit.client.0.smithi113.stdout:6/144: symlink d8/de/d1c/l30 0 2024-04-12T05:08:50.686 INFO:tasks.workunit.client.0.smithi113.stdout:6/145: write d8/de/d17/d25/f28 [610200,115019] 0 2024-04-12T05:08:50.686 INFO:tasks.workunit.client.0.smithi113.stdout:9/185: creat d1/d27/f3c x:0 0 0 2024-04-12T05:08:50.688 INFO:tasks.workunit.client.0.smithi113.stdout:6/146: mkdir d8/de/d17/d25/d31 0 2024-04-12T05:08:50.689 INFO:tasks.workunit.client.0.smithi113.stdout:6/147: chown d8/ld 30991679 1 2024-04-12T05:08:50.691 INFO:tasks.workunit.client.0.smithi113.stdout:6/148: dread d8/de/f15 [0,4194304] 0 2024-04-12T05:08:50.692 INFO:tasks.workunit.client.0.smithi113.stdout:9/186: getdents d1/d4/d30 0 2024-04-12T05:08:50.695 INFO:tasks.workunit.client.0.smithi113.stdout:9/187: rename d1/d24 to d1/d4/d30/d37/d35/d3d 0 2024-04-12T05:08:50.696 INFO:tasks.workunit.client.0.smithi113.stdout:9/188: write d1/d27/f3c [558681,322] 0 2024-04-12T05:08:50.697 INFO:tasks.workunit.client.0.smithi113.stdout:6/149: link d8/c14 d8/de/d17/d25/d31/c32 0 2024-04-12T05:08:50.698 INFO:tasks.workunit.client.0.smithi113.stdout:6/150: dread - d8/de/d17/f27 zero size 2024-04-12T05:08:50.699 INFO:tasks.workunit.client.0.smithi113.stdout:9/189: write d1/f14 [4919020,50520] 0 2024-04-12T05:08:50.699 INFO:tasks.workunit.client.0.smithi113.stdout:6/151: readlink d8/de/d17/d25/d26/l2a 0 2024-04-12T05:08:50.701 INFO:tasks.workunit.client.0.smithi113.stdout:6/152: write d8/de/d17/f27 [653556,105493] 0 2024-04-12T05:08:50.703 INFO:tasks.workunit.client.0.smithi113.stdout:9/190: rename d1/d4/d5/l9 to d1/d4/d5/d22/l3e 0 2024-04-12T05:08:50.706 INFO:tasks.workunit.client.0.smithi113.stdout:6/153: link d8/de/d1c/l30 d8/l33 0 2024-04-12T05:08:50.708 INFO:tasks.workunit.client.0.smithi113.stdout:9/191: rename d1/d4/d5/d6/d15/f2f to d1/d4/d5/d6/d15/f3f 0 2024-04-12T05:08:50.712 INFO:tasks.workunit.client.0.smithi113.stdout:6/154: dwrite d8/f1a [0,4194304] 0 2024-04-12T05:08:50.714 INFO:tasks.workunit.client.0.smithi113.stdout:9/192: dwrite d1/f14 [0,4194304] 0 2024-04-12T05:08:50.716 INFO:tasks.workunit.client.0.smithi113.stdout:6/155: creat d8/de/d1c/f34 x:0 0 0 2024-04-12T05:08:50.717 INFO:tasks.workunit.client.0.smithi113.stdout:9/193: stat d1/d4/d5/d22/l3e 0 2024-04-12T05:08:50.722 INFO:tasks.workunit.client.0.smithi113.stdout:6/156: dwrite d8/de/d1c/f34 [0,4194304] 0 2024-04-12T05:08:50.723 INFO:tasks.workunit.client.0.smithi113.stdout:6/157: fsync d8/f2d 0 2024-04-12T05:08:50.724 INFO:tasks.workunit.client.0.smithi113.stdout:9/194: dread d1/d4/d5/d6/d15/f3f [0,4194304] 0 2024-04-12T05:08:50.725 INFO:tasks.workunit.client.0.smithi113.stdout:9/195: chown d1/d4/d5/d6 504968238 1 2024-04-12T05:08:50.730 INFO:tasks.workunit.client.0.smithi113.stdout:9/196: rmdir d1/d4/d30/d37 39 2024-04-12T05:08:50.731 INFO:tasks.workunit.client.0.smithi113.stdout:6/158: dwrite d8/de/f19 [0,4194304] 0 2024-04-12T05:08:50.735 INFO:tasks.workunit.client.0.smithi113.stdout:9/197: dread d1/d4/d5/f7 [0,4194304] 0 2024-04-12T05:08:50.738 INFO:tasks.workunit.client.0.smithi113.stdout:9/198: rmdir d1/d4/d5/d6 39 2024-04-12T05:08:50.740 INFO:tasks.workunit.client.0.smithi113.stdout:6/159: dwrite d8/f2d [0,4194304] 0 2024-04-12T05:08:50.743 INFO:tasks.workunit.client.0.smithi113.stdout:9/199: truncate d1/d4/f18 139649 0 2024-04-12T05:08:50.744 INFO:tasks.workunit.client.0.smithi113.stdout:6/160: creat d8/de/d1c/f35 x:0 0 0 2024-04-12T05:08:50.745 INFO:tasks.workunit.client.0.smithi113.stdout:6/161: readlink d8/de/d1c/l20 0 2024-04-12T05:08:50.756 INFO:tasks.workunit.client.0.smithi113.stdout:6/162: rename d8/de/d1c to d8/de/d13/d11/d36 0 2024-04-12T05:08:50.757 INFO:tasks.workunit.client.0.smithi113.stdout:6/163: chown d8/c14 34646471 1 2024-04-12T05:08:50.760 INFO:tasks.workunit.client.0.smithi113.stdout:9/200: dwrite d1/f26 [0,4194304] 0 2024-04-12T05:08:50.764 INFO:tasks.workunit.client.0.smithi113.stdout:9/201: rename d1/df/d29/c2c to d1/d4/d30/d37/d35/c40 0 2024-04-12T05:08:50.768 INFO:tasks.workunit.client.0.smithi113.stdout:9/202: rename d1/c1c to d1/df/d29/c41 0 2024-04-12T05:08:50.768 INFO:tasks.workunit.client.0.smithi113.stdout:9/203: chown d1/d4/d5/d22 7 1 2024-04-12T05:08:50.769 INFO:tasks.workunit.client.0.smithi113.stdout:9/204: write d1/d4/f33 [554156,67620] 0 2024-04-12T05:08:50.774 INFO:tasks.workunit.client.0.smithi113.stdout:9/205: rename d1/d27/c2d to d1/d4/d5/d6/d15/c42 0 2024-04-12T05:08:50.776 INFO:tasks.workunit.client.0.smithi113.stdout:9/206: fsync d1/d4/f18 0 2024-04-12T05:08:50.780 INFO:tasks.workunit.client.0.smithi113.stdout:9/207: link d1/c2a d1/df/c43 0 2024-04-12T05:08:50.781 INFO:tasks.workunit.client.0.smithi113.stdout:9/208: mknod d1/df/c44 0 2024-04-12T05:08:50.783 INFO:tasks.workunit.client.0.smithi113.stdout:9/209: chown d1/df/c23 0 1 2024-04-12T05:08:50.784 INFO:tasks.workunit.client.0.smithi113.stdout:9/210: unlink d1/df/c25 0 2024-04-12T05:08:50.787 INFO:tasks.workunit.client.0.smithi113.stdout:9/211: dread d1/f26 [0,4194304] 0 2024-04-12T05:08:50.788 INFO:tasks.workunit.client.0.smithi113.stdout:9/212: chown d1/d4/d30/d37/d35/l3a 106 1 2024-04-12T05:08:50.792 INFO:tasks.workunit.client.0.smithi113.stdout:9/213: link d1/df/d29/c41 d1/d4/d30/d37/c45 0 2024-04-12T05:08:50.795 INFO:tasks.workunit.client.0.smithi113.stdout:9/214: rename d1/f14 to d1/f46 0 2024-04-12T05:08:50.803 INFO:tasks.workunit.client.0.smithi113.stdout:9/215: dwrite d1/d4/d5/f7 [0,4194304] 0 2024-04-12T05:08:50.805 INFO:tasks.workunit.client.0.smithi113.stdout:9/216: fsync d1/d4/f18 0 2024-04-12T05:08:50.808 INFO:tasks.workunit.client.0.smithi113.stdout:9/217: mknod d1/d4/d38/c47 0 2024-04-12T05:08:50.811 INFO:tasks.workunit.client.0.smithi113.stdout:9/218: dread d1/fc [0,4194304] 0 2024-04-12T05:08:50.814 INFO:tasks.workunit.client.0.smithi113.stdout:9/219: creat d1/d4/d5/d6/f48 x:0 0 0 2024-04-12T05:08:50.815 INFO:tasks.workunit.client.0.smithi113.stdout:4/148: sync 2024-04-12T05:08:50.815 INFO:tasks.workunit.client.0.smithi113.stdout:0/114: sync 2024-04-12T05:08:50.815 INFO:tasks.workunit.client.0.smithi113.stdout:4/149: readlink l9 0 2024-04-12T05:08:50.816 INFO:tasks.workunit.client.0.smithi113.stdout:0/115: fsync d1/d6/f1e 0 2024-04-12T05:08:50.816 INFO:tasks.workunit.client.0.smithi113.stdout:9/220: creat d1/d4/d30/d37/d35/d3d/f49 x:0 0 0 2024-04-12T05:08:50.817 INFO:tasks.workunit.client.0.smithi113.stdout:0/116: read d1/fe [393037,89452] 0 2024-04-12T05:08:50.817 INFO:tasks.workunit.client.0.smithi113.stdout:4/150: mkdir da/de/d17/d2f 0 2024-04-12T05:08:50.818 INFO:tasks.workunit.client.0.smithi113.stdout:9/221: write d1/d4/d5/f12 [6346758,126405] 0 2024-04-12T05:08:50.818 INFO:tasks.workunit.client.0.smithi113.stdout:0/117: chown d1/d6/da/f11 335 1 2024-04-12T05:08:50.820 INFO:tasks.workunit.client.0.smithi113.stdout:4/151: mknod da/dd/df/d14/c30 0 2024-04-12T05:08:50.822 INFO:tasks.workunit.client.0.smithi113.stdout:9/222: mkdir d1/d4a 0 2024-04-12T05:08:50.822 INFO:tasks.workunit.client.0.smithi113.stdout:0/118: getdents d1/d9 0 2024-04-12T05:08:50.823 INFO:tasks.workunit.client.0.smithi113.stdout:9/223: read d1/d4/d5/f12 [1159255,4066] 0 2024-04-12T05:08:50.823 INFO:tasks.workunit.client.0.smithi113.stdout:4/152: mknod da/c31 0 2024-04-12T05:08:50.824 INFO:tasks.workunit.client.0.smithi113.stdout:9/224: fdatasync d1/d4/d30/d37/d35/d3d/f49 0 2024-04-12T05:08:50.829 INFO:tasks.workunit.client.0.smithi113.stdout:4/153: dread da/dd/df/d14/f18 [0,4194304] 0 2024-04-12T05:08:50.829 INFO:tasks.workunit.client.0.smithi113.stdout:0/119: rename d1/d6/da/l19 to d1/d6/l1f 0 2024-04-12T05:08:50.830 INFO:tasks.workunit.client.0.smithi113.stdout:0/120: stat d1/fe 0 2024-04-12T05:08:50.831 INFO:tasks.workunit.client.0.smithi113.stdout:9/225: dwrite d1/d4/d5/f7 [0,4194304] 0 2024-04-12T05:08:50.832 INFO:tasks.workunit.client.0.smithi113.stdout:9/226: chown c0 472362840 1 2024-04-12T05:08:50.833 INFO:tasks.workunit.client.0.smithi113.stdout:0/121: fsync d1/fe 0 2024-04-12T05:08:50.834 INFO:tasks.workunit.client.0.smithi113.stdout:0/122: write d1/d6/f1e [663927,2895] 0 2024-04-12T05:08:50.834 INFO:tasks.workunit.client.0.smithi113.stdout:4/154: creat da/dd/f32 x:0 0 0 2024-04-12T05:08:50.836 INFO:tasks.workunit.client.0.smithi113.stdout:9/227: dread d1/d4/d5/d6/d15/f3f [0,4194304] 0 2024-04-12T05:08:50.837 INFO:tasks.workunit.client.0.smithi113.stdout:4/155: mknod da/dd/df/d1d/c33 0 2024-04-12T05:08:50.838 INFO:tasks.workunit.client.0.smithi113.stdout:0/123: mknod d1/c20 0 2024-04-12T05:08:50.840 INFO:tasks.workunit.client.0.smithi113.stdout:4/156: symlink da/dd/df/d1d/l34 0 2024-04-12T05:08:50.841 INFO:tasks.workunit.client.0.smithi113.stdout:0/124: rmdir d1/d6/d16 39 2024-04-12T05:08:50.843 INFO:tasks.workunit.client.0.smithi113.stdout:9/228: dwrite d1/d4/d5/d6/f48 [0,4194304] 0 2024-04-12T05:08:50.844 INFO:tasks.workunit.client.0.smithi113.stdout:4/157: creat da/dd/df/d1d/f35 x:0 0 0 2024-04-12T05:08:50.844 INFO:tasks.workunit.client.0.smithi113.stdout:0/125: fdatasync d1/d6/da/f11 0 2024-04-12T05:08:50.845 INFO:tasks.workunit.client.0.smithi113.stdout:4/158: fdatasync da/dd/d1b/f23 0 2024-04-12T05:08:50.845 INFO:tasks.workunit.client.0.smithi113.stdout:0/126: fdatasync d1/d6/f1d 0 2024-04-12T05:08:50.845 INFO:tasks.workunit.client.0.smithi113.stdout:9/229: chown d1/d4/d5/l2e 221521 1 2024-04-12T05:08:50.846 INFO:tasks.workunit.client.0.smithi113.stdout:0/127: chown d1/d6/da/l12 17875564 1 2024-04-12T05:08:50.846 INFO:tasks.workunit.client.0.smithi113.stdout:4/159: write da/dd/d1b/f2c [565954,912] 0 2024-04-12T05:08:50.847 INFO:tasks.workunit.client.0.smithi113.stdout:0/128: chown d1/d6 8154970 1 2024-04-12T05:08:50.851 INFO:tasks.workunit.client.0.smithi113.stdout:4/160: mknod da/dd/df/d1d/c36 0 2024-04-12T05:08:50.852 INFO:tasks.workunit.client.0.smithi113.stdout:0/129: mknod d1/d6/c21 0 2024-04-12T05:08:50.856 INFO:tasks.workunit.client.0.smithi113.stdout:9/230: truncate d1/d4/f33 2434560 0 2024-04-12T05:08:50.858 INFO:tasks.workunit.client.0.smithi113.stdout:9/231: truncate d1/d4/f18 851125 0 2024-04-12T05:08:50.859 INFO:tasks.workunit.client.0.smithi113.stdout:0/130: link d1/d6/d16/f1c d1/f22 0 2024-04-12T05:08:50.860 INFO:tasks.workunit.client.0.smithi113.stdout:9/232: mkdir d1/df/d4b 0 2024-04-12T05:08:50.861 INFO:tasks.workunit.client.0.smithi113.stdout:0/131: symlink d1/d9/l23 0 2024-04-12T05:08:50.862 INFO:tasks.workunit.client.0.smithi113.stdout:0/132: write d1/d6/f1d [260995,127117] 0 2024-04-12T05:08:50.864 INFO:tasks.workunit.client.0.smithi113.stdout:9/233: rename d1/d4/d5/f7 to d1/d4/d5/f4c 0 2024-04-12T05:08:50.867 INFO:tasks.workunit.client.0.smithi113.stdout:9/234: rmdir d1/d4/d30/d37 39 2024-04-12T05:08:50.867 INFO:tasks.workunit.client.0.smithi113.stdout:4/161: dwrite da/f1a [0,4194304] 0 2024-04-12T05:08:50.868 INFO:tasks.workunit.client.0.smithi113.stdout:9/235: chown d1/f46 1423 1 2024-04-12T05:08:50.869 INFO:tasks.workunit.client.0.smithi113.stdout:4/162: write da/dd/df/d14/f19 [451812,28383] 0 2024-04-12T05:08:50.870 INFO:tasks.workunit.client.0.smithi113.stdout:9/236: stat d1/c2a 0 2024-04-12T05:08:50.870 INFO:tasks.workunit.client.0.smithi113.stdout:4/163: chown da/dd/d1b 61 1 2024-04-12T05:08:50.872 INFO:tasks.workunit.client.0.smithi113.stdout:9/237: creat d1/df/f4d x:0 0 0 2024-04-12T05:08:50.876 INFO:tasks.workunit.client.0.smithi113.stdout:9/238: rmdir d1/d4/d5/d6 39 2024-04-12T05:08:50.879 INFO:tasks.workunit.client.0.smithi113.stdout:9/239: symlink d1/d4/d5/d22/l4e 0 2024-04-12T05:08:50.884 INFO:tasks.workunit.client.0.smithi113.stdout:0/133: dwrite d1/fe [0,4194304] 0 2024-04-12T05:08:50.885 INFO:tasks.workunit.client.0.smithi113.stdout:4/164: dwrite da/f1a [4194304,4194304] 0 2024-04-12T05:08:50.888 INFO:tasks.workunit.client.0.smithi113.stdout:0/134: creat d1/d6/d16/f24 x:0 0 0 2024-04-12T05:08:50.889 INFO:tasks.workunit.client.0.smithi113.stdout:9/240: dread d1/d4/f33 [0,4194304] 0 2024-04-12T05:08:50.889 INFO:tasks.workunit.client.0.smithi113.stdout:0/135: chown d1/d6/da 3719731 1 2024-04-12T05:08:50.890 INFO:tasks.workunit.client.0.smithi113.stdout:1/111: sync 2024-04-12T05:08:50.890 INFO:tasks.workunit.client.0.smithi113.stdout:5/170: sync 2024-04-12T05:08:50.891 INFO:tasks.workunit.client.0.smithi113.stdout:8/182: sync 2024-04-12T05:08:50.891 INFO:tasks.workunit.client.0.smithi113.stdout:2/80: sync 2024-04-12T05:08:50.891 INFO:tasks.workunit.client.0.smithi113.stdout:7/135: sync 2024-04-12T05:08:50.891 INFO:tasks.workunit.client.0.smithi113.stdout:3/150: sync 2024-04-12T05:08:50.894 INFO:tasks.workunit.client.0.smithi113.stdout:4/165: dwrite da/dd/f32 [0,4194304] 0 2024-04-12T05:08:50.900 INFO:tasks.workunit.client.0.smithi113.stdout:3/151: fdatasync d0/f1c 0 2024-04-12T05:08:50.900 INFO:tasks.workunit.client.0.smithi113.stdout:4/166: fdatasync da/fb 0 2024-04-12T05:08:50.901 INFO:tasks.workunit.client.0.smithi113.stdout:7/136: readlink d0/dc/l4 0 2024-04-12T05:08:50.905 INFO:tasks.workunit.client.0.smithi113.stdout:2/81: truncate d8/d9/fc 825946 0 2024-04-12T05:08:50.907 INFO:tasks.workunit.client.0.smithi113.stdout:8/183: rename d9/d16/d1e/c20 to d9/d16/c35 0 2024-04-12T05:08:50.909 INFO:tasks.workunit.client.0.smithi113.stdout:1/112: truncate f2 3949193 0 2024-04-12T05:08:50.909 INFO:tasks.workunit.client.0.smithi113.stdout:4/167: unlink da/dd/df/c12 0 2024-04-12T05:08:50.910 INFO:tasks.workunit.client.0.smithi113.stdout:7/137: creat d0/dd/f30 x:0 0 0 2024-04-12T05:08:50.913 INFO:tasks.workunit.client.0.smithi113.stdout:2/82: symlink d8/d9/l13 0 2024-04-12T05:08:50.915 INFO:tasks.workunit.client.0.smithi113.stdout:8/184: creat d9/d34/f36 x:0 0 0 2024-04-12T05:08:50.915 INFO:tasks.workunit.client.0.smithi113.stdout:9/241: dwrite d1/d27/f3c [0,4194304] 0 2024-04-12T05:08:50.917 INFO:tasks.workunit.client.0.smithi113.stdout:7/138: creat d0/d26/f31 x:0 0 0 2024-04-12T05:08:50.918 INFO:tasks.workunit.client.0.smithi113.stdout:8/185: chown d9/d24 0 1 2024-04-12T05:08:50.918 INFO:tasks.workunit.client.0.smithi113.stdout:0/136: dwrite d1/d6/d16/f24 [0,4194304] 0 2024-04-12T05:08:50.919 INFO:tasks.workunit.client.0.smithi113.stdout:4/168: fdatasync da/dd/df/d14/f18 0 2024-04-12T05:08:50.921 INFO:tasks.workunit.client.0.smithi113.stdout:9/242: chown d1/d27/f3c 26 1 2024-04-12T05:08:50.922 INFO:tasks.workunit.client.0.smithi113.stdout:2/83: mknod d8/c14 0 2024-04-12T05:08:50.924 INFO:tasks.workunit.client.0.smithi113.stdout:8/186: rename d9 to d9/d16/d1e/d26/d37 22 2024-04-12T05:08:50.925 INFO:tasks.workunit.client.0.smithi113.stdout:3/152: dwrite d0/d13/f19 [0,4194304] 0 2024-04-12T05:08:50.927 INFO:tasks.workunit.client.0.smithi113.stdout:4/169: write da/dd/df/f10 [4371245,25916] 0 2024-04-12T05:08:50.927 INFO:tasks.workunit.client.0.smithi113.stdout:9/243: chown d1/d27/f3c 5501 1 2024-04-12T05:08:50.928 INFO:tasks.workunit.client.0.smithi113.stdout:8/187: fsync d9/d16/f1a 0 2024-04-12T05:08:50.929 INFO:tasks.workunit.client.0.smithi113.stdout:1/113: dwrite ff [0,4194304] 0 2024-04-12T05:08:50.931 INFO:tasks.workunit.client.0.smithi113.stdout:5/171: truncate d0/d10/d27/f25 481388 0 2024-04-12T05:08:50.931 INFO:tasks.workunit.client.0.smithi113.stdout:4/170: chown da/dd/df/d14 2164 1 2024-04-12T05:08:50.931 INFO:tasks.workunit.client.0.smithi113.stdout:8/188: chown d9/f17 4921 1 2024-04-12T05:08:50.932 INFO:tasks.workunit.client.0.smithi113.stdout:3/153: write d0/d13/d15/d28/f31 [874790,127477] 0 2024-04-12T05:08:50.934 INFO:tasks.workunit.client.0.smithi113.stdout:0/137: mkdir d1/d6/d16/d25 0 2024-04-12T05:08:50.934 INFO:tasks.workunit.client.0.smithi113.stdout:8/189: rename d9 to d9/d38 22 2024-04-12T05:08:50.934 INFO:tasks.workunit.client.0.smithi113.stdout:5/172: stat d0/d10/d27/c15 0 2024-04-12T05:08:50.937 INFO:tasks.workunit.client.0.smithi113.stdout:1/114: symlink d10/d14/l1c 0 2024-04-12T05:08:50.937 INFO:tasks.workunit.client.0.smithi113.stdout:4/171: mknod da/dd/df/c37 0 2024-04-12T05:08:50.939 INFO:tasks.workunit.client.0.smithi113.stdout:7/139: dwrite d0/dc/f27 [0,4194304] 0 2024-04-12T05:08:50.939 INFO:tasks.workunit.client.0.smithi113.stdout:5/173: chown d0/d10/f12 32 1 2024-04-12T05:08:50.941 INFO:tasks.workunit.client.0.smithi113.stdout:4/172: chown da/c31 144 1 2024-04-12T05:08:50.941 INFO:tasks.workunit.client.0.smithi113.stdout:2/84: mkdir d8/d9/de/d15 0 2024-04-12T05:08:50.941 INFO:tasks.workunit.client.0.smithi113.stdout:9/244: write d1/d4/d30/d37/d35/d3d/f49 [711866,110622] 0 2024-04-12T05:08:50.942 INFO:tasks.workunit.client.0.smithi113.stdout:0/138: mkdir d1/d6/d26 0 2024-04-12T05:08:50.946 INFO:tasks.workunit.client.0.smithi113.stdout:7/140: truncate d0/dc/f27 4222273 0 2024-04-12T05:08:50.946 INFO:tasks.workunit.client.0.smithi113.stdout:3/154: symlink d0/d13/d15/d28/d2a/d2d/l37 0 2024-04-12T05:08:50.947 INFO:tasks.workunit.client.0.smithi113.stdout:1/115: rename c3 to d10/c1d 0 2024-04-12T05:08:50.948 INFO:tasks.workunit.client.0.smithi113.stdout:0/139: fsync d1/d6/d16/f24 0 2024-04-12T05:08:50.948 INFO:tasks.workunit.client.0.smithi113.stdout:7/141: readlink d0/l8 0 2024-04-12T05:08:50.951 INFO:tasks.workunit.client.0.smithi113.stdout:8/190: dwrite d9/fb [4194304,4194304] 0 2024-04-12T05:08:50.952 INFO:tasks.workunit.client.0.smithi113.stdout:4/173: mknod da/dd/df/c38 0 2024-04-12T05:08:50.954 INFO:tasks.workunit.client.0.smithi113.stdout:6/164: sync 2024-04-12T05:08:50.955 INFO:tasks.workunit.client.0.smithi113.stdout:9/245: mknod d1/d4/d30/d37/c4f 0 2024-04-12T05:08:50.955 INFO:tasks.workunit.client.0.smithi113.stdout:5/174: dread d0/d10/f1b [0,4194304] 0 2024-04-12T05:08:50.956 INFO:tasks.workunit.client.0.smithi113.stdout:3/155: unlink d0/d13/d15/f24 0 2024-04-12T05:08:50.957 INFO:tasks.workunit.client.0.smithi113.stdout:6/165: fdatasync d8/f12 0 2024-04-12T05:08:50.957 INFO:tasks.workunit.client.0.smithi113.stdout:2/85: mkdir d8/d16 0 2024-04-12T05:08:50.959 INFO:tasks.workunit.client.0.smithi113.stdout:3/156: dread - d0/f6 zero size 2024-04-12T05:08:50.960 INFO:tasks.workunit.client.0.smithi113.stdout:8/191: mknod d9/d16/d1e/d26/c39 0 2024-04-12T05:08:50.964 INFO:tasks.workunit.client.0.smithi113.stdout:9/246: mknod d1/d4a/c50 0 2024-04-12T05:08:50.964 INFO:tasks.workunit.client.0.smithi113.stdout:6/166: write d8/de/d13/d11/d36/f23 [730741,7982] 0 2024-04-12T05:08:50.966 INFO:tasks.workunit.client.0.smithi113.stdout:2/86: rename d8/d9/fa to d8/d16/f17 0 2024-04-12T05:08:50.967 INFO:tasks.workunit.client.0.smithi113.stdout:0/140: dwrite d1/d6/f1d [0,4194304] 0 2024-04-12T05:08:50.973 INFO:tasks.workunit.client.0.smithi113.stdout:7/142: dwrite d0/dc/d6/fa [0,4194304] 0 2024-04-12T05:08:50.973 INFO:tasks.workunit.client.0.smithi113.stdout:8/192: rename d9/d24 to d9/d16/d1e/d3a 0 2024-04-12T05:08:50.975 INFO:tasks.workunit.client.0.smithi113.stdout:6/167: mknod d8/de/d17/d25/c37 0 2024-04-12T05:08:50.979 INFO:tasks.workunit.client.0.smithi113.stdout:3/157: getdents d0/d13/d15/d28 0 2024-04-12T05:08:50.980 INFO:tasks.workunit.client.0.smithi113.stdout:9/247: rename d1/d4/d5/la to d1/d4/d5/d17/l51 0 2024-04-12T05:08:50.980 INFO:tasks.workunit.client.0.smithi113.stdout:6/168: chown d8/de/d17/d25/d26/l2a 454478 1 2024-04-12T05:08:50.980 INFO:tasks.workunit.client.0.smithi113.stdout:2/87: dread f7 [0,4194304] 0 2024-04-12T05:08:50.984 INFO:tasks.workunit.client.0.smithi113.stdout:6/169: chown d8/de/d17/d25/d31 345 1 2024-04-12T05:08:50.985 INFO:tasks.workunit.client.0.smithi113.stdout:9/248: chown d1/df/c23 97 1 2024-04-12T05:08:50.987 INFO:tasks.workunit.client.0.smithi113.stdout:5/175: dread d0/d10/f12 [0,4194304] 0 2024-04-12T05:08:50.988 INFO:tasks.workunit.client.0.smithi113.stdout:0/141: rename d1/d6/da/f11 to d1/d9/f27 0 2024-04-12T05:08:50.988 INFO:tasks.workunit.client.0.smithi113.stdout:8/193: dwrite d9/d16/f1a [4194304,4194304] 0 2024-04-12T05:08:50.988 INFO:tasks.workunit.client.0.smithi113.stdout:7/143: dwrite d0/dd/f30 [0,4194304] 0 2024-04-12T05:08:50.990 INFO:tasks.workunit.client.0.smithi113.stdout:7/144: stat d0/dd/d2d 0 2024-04-12T05:08:51.029 INFO:tasks.workunit.client.0.smithi113.stdout:3/158: creat d0/d13/d15/d28/f38 x:0 0 0 2024-04-12T05:08:51.030 INFO:tasks.workunit.client.0.smithi113.stdout:7/145: chown d0/d26/c2f 117 1 2024-04-12T05:08:51.031 INFO:tasks.workunit.client.0.smithi113.stdout:5/176: unlink d0/d10/d27/c15 0 2024-04-12T05:08:51.031 INFO:tasks.workunit.client.0.smithi113.stdout:2/88: dread d8/d9/de/f10 [0,4194304] 0 2024-04-12T05:08:51.032 INFO:tasks.workunit.client.0.smithi113.stdout:8/194: unlink d9/d16/d1e/d3a/f30 0 2024-04-12T05:08:51.032 INFO:tasks.workunit.client.0.smithi113.stdout:7/146: readlink d0/dc/d6/l1d 0 2024-04-12T05:08:51.034 INFO:tasks.workunit.client.0.smithi113.stdout:6/170: symlink d8/d1d/l38 0 2024-04-12T05:08:51.034 INFO:tasks.workunit.client.0.smithi113.stdout:3/159: creat d0/d13/d15/f39 x:0 0 0 2024-04-12T05:08:51.034 INFO:tasks.workunit.client.0.smithi113.stdout:0/142: dread d1/fe [0,4194304] 0 2024-04-12T05:08:51.036 INFO:tasks.workunit.client.0.smithi113.stdout:6/171: fdatasync d8/de/d13/d11/d36/f35 0 2024-04-12T05:08:51.037 INFO:tasks.workunit.client.0.smithi113.stdout:7/147: mknod d0/d26/c32 0 2024-04-12T05:08:51.038 INFO:tasks.workunit.client.0.smithi113.stdout:8/195: chown d9/d16/d1e/d26/d32/l33 82 1 2024-04-12T05:08:51.045 INFO:tasks.workunit.client.0.smithi113.stdout:7/148: fdatasync d0/dc/d6/d1a/f21 0 2024-04-12T05:08:51.045 INFO:tasks.workunit.client.0.smithi113.stdout:8/196: rename d9/c15 to d9/d16/d1c/c3b 0 2024-04-12T05:08:51.047 INFO:tasks.workunit.client.0.smithi113.stdout:9/249: dwrite d1/d4/d5/d6/d15/f3f [0,4194304] 0 2024-04-12T05:08:51.049 INFO:tasks.workunit.client.0.smithi113.stdout:3/160: getdents d0/d13/d15/d28/d2a/d34 0 2024-04-12T05:08:51.052 INFO:tasks.workunit.client.0.smithi113.stdout:7/149: fsync d0/dd/d11/f20 0 2024-04-12T05:08:51.053 INFO:tasks.workunit.client.0.smithi113.stdout:6/172: dwrite d8/de/f16 [0,4194304] 0 2024-04-12T05:08:51.054 INFO:tasks.workunit.client.0.smithi113.stdout:2/89: dwrite d8/fd [0,4194304] 0 2024-04-12T05:08:51.055 INFO:tasks.workunit.client.0.smithi113.stdout:6/173: fdatasync d8/f2d 0 2024-04-12T05:08:51.055 INFO:tasks.workunit.client.0.smithi113.stdout:6/174: readlink d8/de/d13/l24 0 2024-04-12T05:08:51.060 INFO:tasks.workunit.client.0.smithi113.stdout:2/90: dread d8/d9/de/f10 [0,4194304] 0 2024-04-12T05:08:51.061 INFO:tasks.workunit.client.0.smithi113.stdout:2/91: readlink d8/d9/l13 0 2024-04-12T05:08:51.063 INFO:tasks.workunit.client.0.smithi113.stdout:8/197: rmdir d9 39 2024-04-12T05:08:51.063 INFO:tasks.workunit.client.0.smithi113.stdout:3/161: symlink d0/d13/d15/d28/d2a/d34/l3a 0 2024-04-12T05:08:51.063 INFO:tasks.workunit.client.0.smithi113.stdout:7/150: dread - d0/dc/d6/d1a/f2b zero size 2024-04-12T05:08:51.064 INFO:tasks.workunit.client.0.smithi113.stdout:6/175: symlink d8/de/d17/d25/d31/l39 0 2024-04-12T05:08:51.064 INFO:tasks.workunit.client.0.smithi113.stdout:0/143: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:51.065 INFO:tasks.workunit.client.0.smithi113.stdout:3/162: dread - d0/f1c zero size 2024-04-12T05:08:51.067 INFO:tasks.workunit.client.0.smithi113.stdout:0/144: fsync d1/f2 0 2024-04-12T05:08:51.068 INFO:tasks.workunit.client.0.smithi113.stdout:2/92: creat d8/d9/de/d15/f18 x:0 0 0 2024-04-12T05:08:51.069 INFO:tasks.workunit.client.0.smithi113.stdout:6/176: mknod d8/de/d13/d11/c3a 0 2024-04-12T05:08:51.070 INFO:tasks.workunit.client.0.smithi113.stdout:8/198: mkdir d9/d16/d3c 0 2024-04-12T05:08:51.072 INFO:tasks.workunit.client.0.smithi113.stdout:7/151: rename d0/dc/d6/d1a/f21 to d0/f33 0 2024-04-12T05:08:51.073 INFO:tasks.workunit.client.0.smithi113.stdout:6/177: mkdir d8/de/d17/d3b 0 2024-04-12T05:08:51.077 INFO:tasks.workunit.client.0.smithi113.stdout:8/199: chown d9/d16/d1c/c3b 9 1 2024-04-12T05:08:51.078 INFO:tasks.workunit.client.0.smithi113.stdout:9/250: dwrite d1/d4/f18 [0,4194304] 0 2024-04-12T05:08:51.082 INFO:tasks.workunit.client.0.smithi113.stdout:6/178: mknod d8/de/d17/d3b/c3c 0 2024-04-12T05:08:51.085 INFO:tasks.workunit.client.0.smithi113.stdout:3/163: dread d0/d13/d15/d28/f31 [0,4194304] 0 2024-04-12T05:08:51.087 INFO:tasks.workunit.client.0.smithi113.stdout:8/200: mknod d9/d16/d1e/c3d 0 2024-04-12T05:08:51.089 INFO:tasks.workunit.client.0.smithi113.stdout:9/251: write d1/d4/f18 [4500998,98085] 0 2024-04-12T05:08:51.089 INFO:tasks.workunit.client.0.smithi113.stdout:3/164: fsync d0/f29 0 2024-04-12T05:08:51.091 INFO:tasks.workunit.client.0.smithi113.stdout:6/179: mkdir d8/de/d13/d11/d36/d3d 0 2024-04-12T05:08:51.092 INFO:tasks.workunit.client.0.smithi113.stdout:3/165: fdatasync d0/f12 0 2024-04-12T05:08:51.093 INFO:tasks.workunit.client.0.smithi113.stdout:2/93: dwrite d8/d9/de/d15/f18 [0,4194304] 0 2024-04-12T05:08:51.095 INFO:tasks.workunit.client.0.smithi113.stdout:7/152: dwrite d0/dc/f18 [0,4194304] 0 2024-04-12T05:08:51.097 INFO:tasks.workunit.client.0.smithi113.stdout:2/94: chown d8/c11 9 1 2024-04-12T05:08:51.097 INFO:tasks.workunit.client.0.smithi113.stdout:9/252: mkdir d1/d4/d30/d37/d35/d3d/d52 0 2024-04-12T05:08:51.099 INFO:tasks.workunit.client.0.smithi113.stdout:7/153: chown d0/dc/d6/l1d 770865 1 2024-04-12T05:08:51.104 INFO:tasks.workunit.client.0.smithi113.stdout:8/201: dwrite d9/d16/f25 [0,4194304] 0 2024-04-12T05:08:51.106 INFO:tasks.workunit.client.0.smithi113.stdout:8/202: fsync d9/fd 0 2024-04-12T05:08:51.106 INFO:tasks.workunit.client.0.smithi113.stdout:1/116: sync 2024-04-12T05:08:51.107 INFO:tasks.workunit.client.0.smithi113.stdout:3/166: dread d0/f10 [0,4194304] 0 2024-04-12T05:08:51.109 INFO:tasks.workunit.client.0.smithi113.stdout:0/145: dread d1/d6/d16/f1c [0,4194304] 0 2024-04-12T05:08:51.109 INFO:tasks.workunit.client.0.smithi113.stdout:1/117: write fe [1140857,86208] 0 2024-04-12T05:08:51.110 INFO:tasks.workunit.client.0.smithi113.stdout:2/95: dread d8/d9/de/d15/f18 [0,4194304] 0 2024-04-12T05:08:51.110 INFO:tasks.workunit.client.0.smithi113.stdout:9/253: rename d1/d4a to d1/d4/d30/d37/d35/d3d/d53 0 2024-04-12T05:08:51.112 INFO:tasks.workunit.client.0.smithi113.stdout:7/154: write d0/dd/d11/f20 [1571429,126480] 0 2024-04-12T05:08:51.113 INFO:tasks.workunit.client.0.smithi113.stdout:6/180: getdents d8/de/d13/d11 0 2024-04-12T05:08:51.114 INFO:tasks.workunit.client.0.smithi113.stdout:8/203: mknod d9/d16/d1e/d3a/c3e 0 2024-04-12T05:08:51.115 INFO:tasks.workunit.client.0.smithi113.stdout:6/181: chown d8/c29 1274546 1 2024-04-12T05:08:51.174 INFO:tasks.workunit.client.0.smithi113.stdout:1/118: creat d10/f1e x:0 0 0 2024-04-12T05:08:51.175 INFO:tasks.workunit.client.0.smithi113.stdout:0/146: unlink d1/d6/f1e 0 2024-04-12T05:08:51.176 INFO:tasks.workunit.client.0.smithi113.stdout:2/96: creat d8/d16/f19 x:0 0 0 2024-04-12T05:08:51.176 INFO:tasks.workunit.client.0.smithi113.stdout:9/254: dread d1/d4/f18 [0,4194304] 0 2024-04-12T05:08:51.179 INFO:tasks.workunit.client.0.smithi113.stdout:0/147: truncate d1/f22 5644396 0 2024-04-12T05:08:51.180 INFO:tasks.workunit.client.0.smithi113.stdout:8/204: creat d9/d16/d1e/d26/d32/f3f x:0 0 0 2024-04-12T05:08:51.181 INFO:tasks.workunit.client.0.smithi113.stdout:7/155: dread d0/dc/d6/fa [0,4194304] 0 2024-04-12T05:08:51.181 INFO:tasks.workunit.client.0.smithi113.stdout:9/255: mkdir d1/d4/d5/d6/d54 0 2024-04-12T05:08:51.182 INFO:tasks.workunit.client.0.smithi113.stdout:1/119: dread fe [0,4194304] 0 2024-04-12T05:08:51.184 INFO:tasks.workunit.client.0.smithi113.stdout:7/156: read d0/dc/f18 [895394,56392] 0 2024-04-12T05:08:51.186 INFO:tasks.workunit.client.0.smithi113.stdout:0/148: write d1/d6/d16/f1c [2524645,102732] 0 2024-04-12T05:08:51.188 INFO:tasks.workunit.client.0.smithi113.stdout:6/182: dwrite d8/f2d [0,4194304] 0 2024-04-12T05:08:51.191 INFO:tasks.workunit.client.0.smithi113.stdout:7/157: fsync d0/dd/d1b/f23 0 2024-04-12T05:08:51.194 INFO:tasks.workunit.client.0.smithi113.stdout:1/120: read f2 [2521769,90178] 0 2024-04-12T05:08:51.197 INFO:tasks.workunit.client.0.smithi113.stdout:2/97: dwrite d8/d16/f17 [0,4194304] 0 2024-04-12T05:08:51.199 INFO:tasks.workunit.client.0.smithi113.stdout:7/158: chown d0/dd/f1c 952345401 1 2024-04-12T05:08:51.200 INFO:tasks.workunit.client.0.smithi113.stdout:9/256: truncate d1/f46 1080966 0 2024-04-12T05:08:51.202 INFO:tasks.workunit.client.0.smithi113.stdout:8/205: dwrite d9/fb [0,4194304] 0 2024-04-12T05:08:51.204 INFO:tasks.workunit.client.0.smithi113.stdout:0/149: dread d1/fe [0,4194304] 0 2024-04-12T05:08:51.207 INFO:tasks.workunit.client.0.smithi113.stdout:9/257: fsync d1/d27/f3c 0 2024-04-12T05:08:51.208 INFO:tasks.workunit.client.0.smithi113.stdout:4/174: sync 2024-04-12T05:08:51.208 INFO:tasks.workunit.client.0.smithi113.stdout:5/177: sync 2024-04-12T05:08:51.210 INFO:tasks.workunit.client.0.smithi113.stdout:8/206: symlink d9/l40 0 2024-04-12T05:08:51.211 INFO:tasks.workunit.client.0.smithi113.stdout:6/183: dwrite d8/f1a [4194304,4194304] 0 2024-04-12T05:08:51.212 INFO:tasks.workunit.client.0.smithi113.stdout:5/178: stat d0/d10/d27/lf 0 2024-04-12T05:08:51.214 INFO:tasks.workunit.client.0.smithi113.stdout:6/184: write d8/de/d13/fb [1715273,24864] 0 2024-04-12T05:08:51.215 INFO:tasks.workunit.client.0.smithi113.stdout:2/98: dread d8/d9/de/f10 [0,4194304] 0 2024-04-12T05:08:51.215 INFO:tasks.workunit.client.0.smithi113.stdout:9/258: mknod d1/d4/c55 0 2024-04-12T05:08:51.218 INFO:tasks.workunit.client.0.smithi113.stdout:8/207: mknod d9/d16/c41 0 2024-04-12T05:08:51.221 INFO:tasks.workunit.client.0.smithi113.stdout:7/159: dwrite d0/dc/f18 [0,4194304] 0 2024-04-12T05:08:51.224 INFO:tasks.workunit.client.0.smithi113.stdout:0/150: rmdir d1/d6/d26 0 2024-04-12T05:08:51.228 INFO:tasks.workunit.client.0.smithi113.stdout:4/175: creat da/de/d17/d2f/f39 x:0 0 0 2024-04-12T05:08:51.228 INFO:tasks.workunit.client.0.smithi113.stdout:7/160: chown d0/f12 126745807 1 2024-04-12T05:08:51.229 INFO:tasks.workunit.client.0.smithi113.stdout:5/179: fsync d0/f6 0 2024-04-12T05:08:51.230 INFO:tasks.workunit.client.0.smithi113.stdout:9/259: fsync d1/f46 0 2024-04-12T05:08:51.231 INFO:tasks.workunit.client.0.smithi113.stdout:7/161: chown d0/dc/d6 97 1 2024-04-12T05:08:51.232 INFO:tasks.workunit.client.0.smithi113.stdout:2/99: dwrite d8/d16/f19 [0,4194304] 0 2024-04-12T05:08:51.232 INFO:tasks.workunit.client.0.smithi113.stdout:4/176: chown da/fb 459 1 2024-04-12T05:08:51.232 INFO:tasks.workunit.client.0.smithi113.stdout:0/151: write d1/d9/f27 [783687,81917] 0 2024-04-12T05:08:51.236 INFO:tasks.workunit.client.0.smithi113.stdout:8/208: dwrite d9/d16/f1a [0,4194304] 0 2024-04-12T05:08:51.237 INFO:tasks.workunit.client.0.smithi113.stdout:0/152: fdatasync d1/d6/f1d 0 2024-04-12T05:08:51.238 INFO:tasks.workunit.client.0.smithi113.stdout:2/100: write d8/fd [753094,38068] 0 2024-04-12T05:08:51.238 INFO:tasks.workunit.client.0.smithi113.stdout:4/177: unlink da/f1a 0 2024-04-12T05:08:51.240 INFO:tasks.workunit.client.0.smithi113.stdout:1/121: dread fd [0,4194304] 0 2024-04-12T05:08:51.241 INFO:tasks.workunit.client.0.smithi113.stdout:9/260: symlink d1/d4/d5/d17/l56 0 2024-04-12T05:08:51.242 INFO:tasks.workunit.client.0.smithi113.stdout:0/153: truncate d1/d6/f1d 4216008 0 2024-04-12T05:08:51.243 INFO:tasks.workunit.client.0.smithi113.stdout:9/261: chown d1/d4/f18 60166438 1 2024-04-12T05:08:51.244 INFO:tasks.workunit.client.0.smithi113.stdout:7/162: fsync d0/dc/f18 0 2024-04-12T05:08:51.248 INFO:tasks.workunit.client.0.smithi113.stdout:1/122: dread ff [0,4194304] 0 2024-04-12T05:08:51.249 INFO:tasks.workunit.client.0.smithi113.stdout:4/178: mkdir da/dd/d3a 0 2024-04-12T05:08:51.253 INFO:tasks.workunit.client.0.smithi113.stdout:8/209: link d9/le d9/d16/d3c/l42 0 2024-04-12T05:08:51.253 INFO:tasks.workunit.client.0.smithi113.stdout:2/101: fdatasync d8/fd 0 2024-04-12T05:08:51.254 INFO:tasks.workunit.client.0.smithi113.stdout:5/180: getdents d0/d10/d27/d13 0 2024-04-12T05:08:51.255 INFO:tasks.workunit.client.0.smithi113.stdout:4/179: fdatasync da/dd/df/d1d/f35 0 2024-04-12T05:08:51.257 INFO:tasks.workunit.client.0.smithi113.stdout:1/123: link d10/f1e d10/d14/f1f 0 2024-04-12T05:08:51.258 INFO:tasks.workunit.client.0.smithi113.stdout:8/210: symlink d9/l43 0 2024-04-12T05:08:51.258 INFO:tasks.workunit.client.0.smithi113.stdout:9/262: dwrite d1/d4/d30/d37/d35/d3d/f49 [0,4194304] 0 2024-04-12T05:08:51.259 INFO:tasks.workunit.client.0.smithi113.stdout:2/102: mkdir d8/d9/d1a 0 2024-04-12T05:08:51.260 INFO:tasks.workunit.client.0.smithi113.stdout:6/185: dread d8/de/d13/fa [0,4194304] 0 2024-04-12T05:08:51.260 INFO:tasks.workunit.client.0.smithi113.stdout:0/154: dwrite d1/fe [0,4194304] 0 2024-04-12T05:08:51.260 INFO:tasks.workunit.client.0.smithi113.stdout:4/180: write da/dd/f21 [2269759,69210] 0 2024-04-12T05:08:51.262 INFO:tasks.workunit.client.0.smithi113.stdout:7/163: truncate d0/f12 2577157 0 2024-04-12T05:08:51.264 INFO:tasks.workunit.client.0.smithi113.stdout:6/186: fdatasync d8/f2d 0 2024-04-12T05:08:51.264 INFO:tasks.workunit.client.0.smithi113.stdout:9/263: dread - d1/d4/d5/d6/f3b zero size 2024-04-12T05:08:51.265 INFO:tasks.workunit.client.0.smithi113.stdout:4/181: chown da/dd/df/d1d 267981 1 2024-04-12T05:08:51.266 INFO:tasks.workunit.client.0.smithi113.stdout:8/211: write d9/d16/d1e/d3a/f2a [650437,82700] 0 2024-04-12T05:08:51.267 INFO:tasks.workunit.client.0.smithi113.stdout:6/187: chown d8/de/d13/d11/d36/f23 1251 1 2024-04-12T05:08:51.267 INFO:tasks.workunit.client.0.smithi113.stdout:9/264: write d1/d4/d30/d37/d35/d3d/f49 [1404048,14629] 0 2024-04-12T05:08:51.268 INFO:tasks.workunit.client.0.smithi113.stdout:5/181: rename d0/d10/d27/d13/f2f to d0/f31 0 2024-04-12T05:08:51.270 INFO:tasks.workunit.client.0.smithi113.stdout:7/164: fsync d0/f33 0 2024-04-12T05:08:51.270 INFO:tasks.workunit.client.0.smithi113.stdout:4/182: truncate da/de/d17/d2f/f39 82645 0 2024-04-12T05:08:51.271 INFO:tasks.workunit.client.0.smithi113.stdout:2/103: dread d8/d16/f19 [0,4194304] 0 2024-04-12T05:08:51.271 INFO:tasks.workunit.client.0.smithi113.stdout:6/188: chown d8/de/d17/d3b/c3c 11151 1 2024-04-12T05:08:51.274 INFO:tasks.workunit.client.0.smithi113.stdout:2/104: write d8/fd [4583336,64286] 0 2024-04-12T05:08:51.275 INFO:tasks.workunit.client.0.smithi113.stdout:4/183: write da/dd/df/d1d/f35 [817772,10007] 0 2024-04-12T05:08:51.277 INFO:tasks.workunit.client.0.smithi113.stdout:5/182: rename d0/d10/d27/f17 to d0/d10/d27/d1d/f32 0 2024-04-12T05:08:51.278 INFO:tasks.workunit.client.0.smithi113.stdout:4/184: rename da/dd to da/dd/d3a/d3b 22 2024-04-12T05:08:51.281 INFO:tasks.workunit.client.0.smithi113.stdout:7/165: dread d0/dc/d6/f16 [0,4194304] 0 2024-04-12T05:08:51.281 INFO:tasks.workunit.client.0.smithi113.stdout:0/155: rmdir d1/d6/d16/d25 0 2024-04-12T05:08:51.281 INFO:tasks.workunit.client.0.smithi113.stdout:6/189: mknod d8/de/d17/c3e 0 2024-04-12T05:08:51.282 INFO:tasks.workunit.client.0.smithi113.stdout:4/185: fsync da/dd/d1b/f2c 0 2024-04-12T05:08:51.282 INFO:tasks.workunit.client.0.smithi113.stdout:6/190: fsync d8/de/f16 0 2024-04-12T05:08:51.283 INFO:tasks.workunit.client.0.smithi113.stdout:0/156: rename d1/d6/d16 to d1/d6/d16/d28 22 2024-04-12T05:08:51.283 INFO:tasks.workunit.client.0.smithi113.stdout:2/105: unlink d8/c14 0 2024-04-12T05:08:51.283 INFO:tasks.workunit.client.0.smithi113.stdout:4/186: readlink da/l2a 0 2024-04-12T05:08:51.284 INFO:tasks.workunit.client.0.smithi113.stdout:6/191: chown d8/de/d17/d3b/c3c 48 1 2024-04-12T05:08:51.284 INFO:tasks.workunit.client.0.smithi113.stdout:5/183: mkdir d0/d10/d27/d33 0 2024-04-12T05:08:51.285 INFO:tasks.workunit.client.0.smithi113.stdout:6/192: read d8/de/f15 [1346192,73945] 0 2024-04-12T05:08:51.286 INFO:tasks.workunit.client.0.smithi113.stdout:8/212: link d9/d16/c22 d9/d16/d1e/d3a/c44 0 2024-04-12T05:08:51.287 INFO:tasks.workunit.client.0.smithi113.stdout:8/213: read - d9/d16/d1e/d26/f28 zero size 2024-04-12T05:08:51.287 INFO:tasks.workunit.client.0.smithi113.stdout:2/106: dread d8/d16/f17 [0,4194304] 0 2024-04-12T05:08:51.288 INFO:tasks.workunit.client.0.smithi113.stdout:0/157: creat d1/d9/f29 x:0 0 0 2024-04-12T05:08:51.289 INFO:tasks.workunit.client.0.smithi113.stdout:2/107: write d8/d9/fc [444112,29840] 0 2024-04-12T05:08:51.289 INFO:tasks.workunit.client.0.smithi113.stdout:4/187: mkdir da/d3c 0 2024-04-12T05:08:51.291 INFO:tasks.workunit.client.0.smithi113.stdout:9/265: link c0 d1/d4/d5/d6/d15/c57 0 2024-04-12T05:08:51.291 INFO:tasks.workunit.client.0.smithi113.stdout:5/184: chown d0/f5 374530 1 2024-04-12T05:08:51.293 INFO:tasks.workunit.client.0.smithi113.stdout:9/266: dread d1/fc [4194304,4194304] 0 2024-04-12T05:08:51.293 INFO:tasks.workunit.client.0.smithi113.stdout:5/185: dread - d0/f31 zero size 2024-04-12T05:08:51.293 INFO:tasks.workunit.client.0.smithi113.stdout:8/214: dread d9/fb [4194304,4194304] 0 2024-04-12T05:08:51.293 INFO:tasks.workunit.client.0.smithi113.stdout:6/193: mknod d8/de/d13/d11/d36/c3f 0 2024-04-12T05:08:51.294 INFO:tasks.workunit.client.0.smithi113.stdout:4/188: dread da/dd/d1b/f2c [0,4194304] 0 2024-04-12T05:08:51.338 INFO:tasks.workunit.client.0.smithi113.stdout:3/167: sync 2024-04-12T05:08:51.353 INFO:tasks.workunit.client.0.smithi113.stdout:6/194: stat d8/de/d17/c3e 0 2024-04-12T05:08:51.354 INFO:tasks.workunit.client.0.smithi113.stdout:4/189: chown da/de/c1f 60206286 1 2024-04-12T05:08:51.354 INFO:tasks.workunit.client.0.smithi113.stdout:5/186: write d0/d10/d27/d1d/f2a [867931,81933] 0 2024-04-12T05:08:51.355 INFO:tasks.workunit.client.0.smithi113.stdout:6/195: creat d8/f40 x:0 0 0 2024-04-12T05:08:51.357 INFO:tasks.workunit.client.0.smithi113.stdout:4/190: write da/dd/d1b/f28 [680032,47410] 0 2024-04-12T05:08:51.357 INFO:tasks.workunit.client.0.smithi113.stdout:9/267: truncate d1/f46 932072 0 2024-04-12T05:08:51.357 INFO:tasks.workunit.client.0.smithi113.stdout:2/108: truncate d8/fd 1549705 0 2024-04-12T05:08:51.362 INFO:tasks.workunit.client.0.smithi113.stdout:8/215: dwrite d9/d16/f25 [0,4194304] 0 2024-04-12T05:08:51.364 INFO:tasks.workunit.client.0.smithi113.stdout:7/166: dwrite d0/f12 [0,4194304] 0 2024-04-12T05:08:51.364 INFO:tasks.workunit.client.0.smithi113.stdout:9/268: unlink d1/d4/d5/d22/l3e 0 2024-04-12T05:08:51.365 INFO:tasks.workunit.client.0.smithi113.stdout:4/191: rename da/dd/df/c16 to da/dd/df/d14/c3d 0 2024-04-12T05:08:51.365 INFO:tasks.workunit.client.0.smithi113.stdout:4/192: readlink da/dd/df/d14/l20 0 2024-04-12T05:08:51.366 INFO:tasks.workunit.client.0.smithi113.stdout:4/193: chown da/dd/d1b 11514912 1 2024-04-12T05:08:51.366 INFO:tasks.workunit.client.0.smithi113.stdout:8/216: creat d9/d1b/f45 x:0 0 0 2024-04-12T05:08:51.367 INFO:tasks.workunit.client.0.smithi113.stdout:4/194: chown da/dd/d1b 280458640 1 2024-04-12T05:08:51.368 INFO:tasks.workunit.client.0.smithi113.stdout:7/167: write d0/dc/d6/f16 [716605,124236] 0 2024-04-12T05:08:51.368 INFO:tasks.workunit.client.0.smithi113.stdout:8/217: chown d9/d16/c41 658811375 1 2024-04-12T05:08:51.369 INFO:tasks.workunit.client.0.smithi113.stdout:9/269: creat d1/d4/d5/f58 x:0 0 0 2024-04-12T05:08:51.371 INFO:tasks.workunit.client.0.smithi113.stdout:7/168: write d0/dc/d6/fa [3885588,40325] 0 2024-04-12T05:08:51.372 INFO:tasks.workunit.client.0.smithi113.stdout:4/195: creat da/d2b/f3e x:0 0 0 2024-04-12T05:08:51.373 INFO:tasks.workunit.client.0.smithi113.stdout:7/169: write d0/dd/f1c [952527,114662] 0 2024-04-12T05:08:51.375 INFO:tasks.workunit.client.0.smithi113.stdout:2/109: dread d8/d9/fc [0,4194304] 0 2024-04-12T05:08:51.375 INFO:tasks.workunit.client.0.smithi113.stdout:8/218: dread d9/d16/f25 [0,4194304] 0 2024-04-12T05:08:51.376 INFO:tasks.workunit.client.0.smithi113.stdout:2/110: read d8/d9/fc [517940,30174] 0 2024-04-12T05:08:51.378 INFO:tasks.workunit.client.0.smithi113.stdout:2/111: truncate d8/d9/fc 1431523 0 2024-04-12T05:08:51.378 INFO:tasks.workunit.client.0.smithi113.stdout:8/219: unlink c0 0 2024-04-12T05:08:51.380 INFO:tasks.workunit.client.0.smithi113.stdout:2/112: symlink d8/d9/de/l1b 0 2024-04-12T05:08:51.382 INFO:tasks.workunit.client.0.smithi113.stdout:7/170: link d0/d26/f31 d0/dc/d6/d1a/f34 0 2024-04-12T05:08:51.384 INFO:tasks.workunit.client.0.smithi113.stdout:2/113: write d8/d9/de/d15/f18 [4049425,60037] 0 2024-04-12T05:08:51.385 INFO:tasks.workunit.client.0.smithi113.stdout:7/171: stat d0/dd/cf 0 2024-04-12T05:08:51.385 INFO:tasks.workunit.client.0.smithi113.stdout:8/220: dwrite d9/f17 [0,4194304] 0 2024-04-12T05:08:51.392 INFO:tasks.workunit.client.0.smithi113.stdout:7/172: dread d0/dd/d11/f19 [0,4194304] 0 2024-04-12T05:08:51.393 INFO:tasks.workunit.client.0.smithi113.stdout:8/221: truncate d9/fb 5329079 0 2024-04-12T05:08:51.396 INFO:tasks.workunit.client.0.smithi113.stdout:2/114: creat d8/d9/de/d12/f1c x:0 0 0 2024-04-12T05:08:51.396 INFO:tasks.workunit.client.0.smithi113.stdout:2/115: chown d8/d9/d1a 28 1 2024-04-12T05:08:51.397 INFO:tasks.workunit.client.0.smithi113.stdout:2/116: readlink d8/d9/de/l1b 0 2024-04-12T05:08:51.397 INFO:tasks.workunit.client.0.smithi113.stdout:7/173: mknod d0/dd/d11/c35 0 2024-04-12T05:08:51.398 INFO:tasks.workunit.client.0.smithi113.stdout:2/117: chown l4 668791569 1 2024-04-12T05:08:51.398 INFO:tasks.workunit.client.0.smithi113.stdout:2/118: chown d8/d9/fc 463 1 2024-04-12T05:08:51.399 INFO:tasks.workunit.client.0.smithi113.stdout:8/222: creat d9/d1b/f46 x:0 0 0 2024-04-12T05:08:51.404 INFO:tasks.workunit.client.0.smithi113.stdout:9/270: dwrite d1/d4/d5/f12 [4194304,4194304] 0 2024-04-12T05:08:51.404 INFO:tasks.workunit.client.0.smithi113.stdout:8/223: creat d9/f47 x:0 0 0 2024-04-12T05:08:51.405 INFO:tasks.workunit.client.0.smithi113.stdout:7/174: link d0/l8 d0/dc/d6/d24/l36 0 2024-04-12T05:08:51.406 INFO:tasks.workunit.client.0.smithi113.stdout:7/175: stat d0/dc/l4 0 2024-04-12T05:08:51.407 INFO:tasks.workunit.client.0.smithi113.stdout:4/196: dread da/dd/d1b/f23 [0,4194304] 0 2024-04-12T05:08:51.408 INFO:tasks.workunit.client.0.smithi113.stdout:9/271: creat d1/d4/d30/d37/d35/d3d/d53/f59 x:0 0 0 2024-04-12T05:08:51.415 INFO:tasks.workunit.client.0.smithi113.stdout:4/197: rmdir da 39 2024-04-12T05:08:51.419 INFO:tasks.workunit.client.0.smithi113.stdout:9/272: creat d1/d4/d5/d22/f5a x:0 0 0 2024-04-12T05:08:51.420 INFO:tasks.workunit.client.0.smithi113.stdout:7/176: symlink d0/dd/d2d/l37 0 2024-04-12T05:08:51.421 INFO:tasks.workunit.client.0.smithi113.stdout:7/177: read - d0/dc/d6/d1a/f2b zero size 2024-04-12T05:08:51.422 INFO:tasks.workunit.client.0.smithi113.stdout:4/198: fdatasync da/d2b/f3e 0 2024-04-12T05:08:51.422 INFO:tasks.workunit.client.0.smithi113.stdout:4/199: write da/de/d17/d2f/f39 [1060005,105080] 0 2024-04-12T05:08:51.431 INFO:tasks.workunit.client.0.smithi113.stdout:2/119: dread d8/d16/f17 [0,4194304] 0 2024-04-12T05:08:51.432 INFO:tasks.workunit.client.0.smithi113.stdout:4/200: dread da/de/d17/d2f/f39 [0,4194304] 0 2024-04-12T05:08:51.435 INFO:tasks.workunit.client.0.smithi113.stdout:4/201: chown da/dd/df 1 1 2024-04-12T05:08:51.435 INFO:tasks.workunit.client.0.smithi113.stdout:2/120: rmdir d8/d16 39 2024-04-12T05:08:51.435 INFO:tasks.workunit.client.0.smithi113.stdout:2/121: read d8/fd [1059607,15398] 0 2024-04-12T05:08:51.435 INFO:tasks.workunit.client.0.smithi113.stdout:4/202: symlink da/dd/d3a/l3f 0 2024-04-12T05:08:51.438 INFO:tasks.workunit.client.0.smithi113.stdout:2/122: creat d8/d16/f1d x:0 0 0 2024-04-12T05:08:51.439 INFO:tasks.workunit.client.0.smithi113.stdout:2/123: chown d8 2 1 2024-04-12T05:08:51.442 INFO:tasks.workunit.client.0.smithi113.stdout:4/203: dwrite da/dd/f21 [0,4194304] 0 2024-04-12T05:08:51.444 INFO:tasks.workunit.client.0.smithi113.stdout:2/124: dread d8/d9/de/f10 [0,4194304] 0 2024-04-12T05:08:51.447 INFO:tasks.workunit.client.0.smithi113.stdout:2/125: creat d8/f1e x:0 0 0 2024-04-12T05:08:51.448 INFO:tasks.workunit.client.0.smithi113.stdout:2/126: read - d8/d16/f1d zero size 2024-04-12T05:08:51.448 INFO:tasks.workunit.client.0.smithi113.stdout:4/204: dwrite da/dd/df/d14/f19 [0,4194304] 0 2024-04-12T05:08:51.452 INFO:tasks.workunit.client.0.smithi113.stdout:4/205: creat da/dd/df/d1d/f40 x:0 0 0 2024-04-12T05:08:51.529 INFO:tasks.workunit.client.0.smithi113.stdout:4/206: dread da/dd/df/d14/f19 [0,4194304] 0 2024-04-12T05:08:51.533 INFO:tasks.workunit.client.0.smithi113.stdout:4/207: link da/l2a da/dd/df/d1d/l41 0 2024-04-12T05:08:51.535 INFO:tasks.workunit.client.0.smithi113.stdout:4/208: creat da/dd/d3a/f42 x:0 0 0 2024-04-12T05:08:51.537 INFO:tasks.workunit.client.0.smithi113.stdout:4/209: mkdir da/d3c/d43 0 2024-04-12T05:08:51.538 INFO:tasks.workunit.client.0.smithi113.stdout:4/210: write da/d2b/f3e [337153,111116] 0 2024-04-12T05:08:51.541 INFO:tasks.workunit.client.0.smithi113.stdout:4/211: unlink da/de/d17/d2f/f39 0 2024-04-12T05:08:51.543 INFO:tasks.workunit.client.0.smithi113.stdout:4/212: unlink da/de/d17/f2e 0 2024-04-12T05:08:51.548 INFO:tasks.workunit.client.0.smithi113.stdout:4/213: dwrite da/dd/df/f1c [0,4194304] 0 2024-04-12T05:08:51.550 INFO:tasks.workunit.client.0.smithi113.stdout:4/214: creat da/dd/d3a/f44 x:0 0 0 2024-04-12T05:08:51.551 INFO:tasks.workunit.client.0.smithi113.stdout:4/215: chown da/dd/df/c37 267 1 2024-04-12T05:08:51.551 INFO:tasks.workunit.client.0.smithi113.stdout:4/216: stat c7 0 2024-04-12T05:08:51.553 INFO:tasks.workunit.client.0.smithi113.stdout:4/217: fsync da/dd/df/d14/f19 0 2024-04-12T05:08:51.555 INFO:tasks.workunit.client.0.smithi113.stdout:4/218: write da/dd/df/f10 [1978682,116136] 0 2024-04-12T05:08:51.560 INFO:tasks.workunit.client.0.smithi113.stdout:4/219: dread da/dd/f32 [0,4194304] 0 2024-04-12T05:08:51.563 INFO:tasks.workunit.client.0.smithi113.stdout:4/220: dwrite da/dd/df/d14/f18 [4194304,4194304] 0 2024-04-12T05:08:51.567 INFO:tasks.workunit.client.0.smithi113.stdout:1/124: sync 2024-04-12T05:08:51.581 INFO:tasks.workunit.client.0.smithi113.stdout:1/125: dwrite fa [0,4194304] 0 2024-04-12T05:08:51.584 INFO:tasks.workunit.client.0.smithi113.stdout:1/126: mknod d10/c20 0 2024-04-12T05:08:51.585 INFO:tasks.workunit.client.0.smithi113.stdout:1/127: chown d10/d14/c19 0 1 2024-04-12T05:08:51.586 INFO:tasks.workunit.client.0.smithi113.stdout:1/128: dread - d10/f1e zero size 2024-04-12T05:08:51.587 INFO:tasks.workunit.client.0.smithi113.stdout:1/129: read f5 [512343,77709] 0 2024-04-12T05:08:51.594 INFO:tasks.workunit.client.0.smithi113.stdout:1/130: dwrite fe [0,4194304] 0 2024-04-12T05:08:51.596 INFO:tasks.workunit.client.0.smithi113.stdout:1/131: mkdir d10/d21 0 2024-04-12T05:08:51.601 INFO:tasks.workunit.client.0.smithi113.stdout:1/132: getdents d10/d14 0 2024-04-12T05:08:51.603 INFO:tasks.workunit.client.0.smithi113.stdout:1/133: mknod d10/d21/c22 0 2024-04-12T05:08:51.604 INFO:tasks.workunit.client.0.smithi113.stdout:1/134: mkdir d10/d21/d23 0 2024-04-12T05:08:51.606 INFO:tasks.workunit.client.0.smithi113.stdout:1/135: fdatasync d10/f16 0 2024-04-12T05:08:51.671 INFO:tasks.workunit.client.0.smithi113.stdout:5/187: sync 2024-04-12T05:08:51.671 INFO:tasks.workunit.client.0.smithi113.stdout:0/158: sync 2024-04-12T05:08:51.671 INFO:tasks.workunit.client.0.smithi113.stdout:3/168: sync 2024-04-12T05:08:51.672 INFO:tasks.workunit.client.0.smithi113.stdout:0/159: write d1/d9/f27 [1714789,66817] 0 2024-04-12T05:08:51.674 INFO:tasks.workunit.client.0.smithi113.stdout:3/169: symlink d0/d13/d15/l3b 0 2024-04-12T05:08:51.677 INFO:tasks.workunit.client.0.smithi113.stdout:0/160: truncate d1/f2 2552646 0 2024-04-12T05:08:51.678 INFO:tasks.workunit.client.0.smithi113.stdout:5/188: getdents d0/d10/d27/d13 0 2024-04-12T05:08:51.679 INFO:tasks.workunit.client.0.smithi113.stdout:5/189: write d0/d10/d27/d13/f28 [102459,83583] 0 2024-04-12T05:08:51.683 INFO:tasks.workunit.client.0.smithi113.stdout:5/190: rmdir d0/d10/d27 39 2024-04-12T05:08:51.684 INFO:tasks.workunit.client.0.smithi113.stdout:5/191: stat d0/d10/l2d 0 2024-04-12T05:08:51.686 INFO:tasks.workunit.client.0.smithi113.stdout:0/161: rmdir d1 39 2024-04-12T05:08:51.687 INFO:tasks.workunit.client.0.smithi113.stdout:5/192: symlink d0/d10/d27/l34 0 2024-04-12T05:08:51.687 INFO:tasks.workunit.client.0.smithi113.stdout:3/170: dread d0/f18 [0,4194304] 0 2024-04-12T05:08:51.690 INFO:tasks.workunit.client.0.smithi113.stdout:3/171: unlink d0/d13/f25 0 2024-04-12T05:08:51.691 INFO:tasks.workunit.client.0.smithi113.stdout:0/162: mkdir d1/d6/d16/d2a 0 2024-04-12T05:08:51.691 INFO:tasks.workunit.client.0.smithi113.stdout:5/193: creat d0/d10/d27/d33/f35 x:0 0 0 2024-04-12T05:08:51.693 INFO:tasks.workunit.client.0.smithi113.stdout:0/163: creat d1/d9/f2b x:0 0 0 2024-04-12T05:08:51.694 INFO:tasks.workunit.client.0.smithi113.stdout:0/164: mkdir d1/d6/d16/d2a/d2c 0 2024-04-12T05:08:51.697 INFO:tasks.workunit.client.0.smithi113.stdout:5/194: dread d0/d10/d27/f18 [0,4194304] 0 2024-04-12T05:08:51.697 INFO:tasks.workunit.client.0.smithi113.stdout:0/165: truncate d1/f2 412084 0 2024-04-12T05:08:51.697 INFO:tasks.workunit.client.0.smithi113.stdout:0/166: dread - d1/d9/f29 zero size 2024-04-12T05:08:51.698 INFO:tasks.workunit.client.0.smithi113.stdout:5/195: rename d0/d10 to d0/d10/d27/d33/d36 22 2024-04-12T05:08:51.699 INFO:tasks.workunit.client.0.smithi113.stdout:0/167: mknod d1/c2d 0 2024-04-12T05:08:51.703 INFO:tasks.workunit.client.0.smithi113.stdout:0/168: link d1/d6/da/c1b d1/d6/d16/d2a/c2e 0 2024-04-12T05:08:51.704 INFO:tasks.workunit.client.0.smithi113.stdout:5/196: dwrite d0/d10/d27/d1d/f1e [0,4194304] 0 2024-04-12T05:08:51.705 INFO:tasks.workunit.client.0.smithi113.stdout:5/197: chown d0/d10/d27/d1d/f32 29616746 1 2024-04-12T05:08:51.706 INFO:tasks.workunit.client.0.smithi113.stdout:5/198: dread - d0/d10/d27/f26 zero size 2024-04-12T05:08:51.708 INFO:tasks.workunit.client.0.smithi113.stdout:0/169: dread d1/f22 [0,4194304] 0 2024-04-12T05:08:51.708 INFO:tasks.workunit.client.0.smithi113.stdout:0/170: chown d1/d6/c21 16542242 1 2024-04-12T05:08:51.712 INFO:tasks.workunit.client.0.smithi113.stdout:0/171: creat d1/f2f x:0 0 0 2024-04-12T05:08:51.712 INFO:tasks.workunit.client.0.smithi113.stdout:0/172: write d1/d9/f2b [530649,89105] 0 2024-04-12T05:08:51.716 INFO:tasks.workunit.client.0.smithi113.stdout:0/173: creat d1/d6/f30 x:0 0 0 2024-04-12T05:08:51.718 INFO:tasks.workunit.client.0.smithi113.stdout:0/174: mknod d1/c31 0 2024-04-12T05:08:51.720 INFO:tasks.workunit.client.0.smithi113.stdout:0/175: mknod d1/d9/c32 0 2024-04-12T05:08:51.727 INFO:tasks.workunit.client.0.smithi113.stdout:0/176: read d1/f2 [145032,120740] 0 2024-04-12T05:08:51.728 INFO:tasks.workunit.client.0.smithi113.stdout:0/177: chown d1/f22 1015656 1 2024-04-12T05:08:51.731 INFO:tasks.workunit.client.0.smithi113.stdout:5/199: truncate d0/fc 475215 0 2024-04-12T05:08:51.732 INFO:tasks.workunit.client.0.smithi113.stdout:5/200: chown d0/d10/d27/c2c 2767129 1 2024-04-12T05:08:51.732 INFO:tasks.workunit.client.0.smithi113.stdout:5/201: readlink d0/d10/d27/d1d/l29 0 2024-04-12T05:08:51.733 INFO:tasks.workunit.client.0.smithi113.stdout:0/178: dread d1/d9/f27 [0,4194304] 0 2024-04-12T05:08:51.735 INFO:tasks.workunit.client.0.smithi113.stdout:0/179: rename d1/c2d to d1/d6/d16/d2a/c33 0 2024-04-12T05:08:51.739 INFO:tasks.workunit.client.0.smithi113.stdout:5/202: dwrite d0/f2e [0,4194304] 0 2024-04-12T05:08:51.739 INFO:tasks.workunit.client.0.smithi113.stdout:0/180: mkdir d1/d6/d34 0 2024-04-12T05:08:51.749 INFO:tasks.workunit.client.0.smithi113.stdout:5/203: fsync d0/fc 0 2024-04-12T05:08:51.750 INFO:tasks.workunit.client.0.smithi113.stdout:0/181: dwrite d1/d6/d16/f1c [0,4194304] 0 2024-04-12T05:08:51.756 INFO:tasks.workunit.client.0.smithi113.stdout:0/182: dwrite d1/d6/d16/f1c [0,4194304] 0 2024-04-12T05:08:51.758 INFO:tasks.workunit.client.0.smithi113.stdout:0/183: rmdir d1/d6/da 39 2024-04-12T05:08:51.761 INFO:tasks.workunit.client.0.smithi113.stdout:0/184: mknod d1/d6/d16/d2a/c35 0 2024-04-12T05:08:51.765 INFO:tasks.workunit.client.0.smithi113.stdout:5/204: dwrite d0/d10/d27/f25 [0,4194304] 0 2024-04-12T05:08:51.766 INFO:tasks.workunit.client.1.smithi192.stdout:3/36: sync 2024-04-12T05:08:51.766 INFO:tasks.workunit.client.1.smithi192.stdout:9/43: sync 2024-04-12T05:08:51.766 INFO:tasks.workunit.client.0.smithi113.stdout:0/185: mkdir d1/d6/da/d36 0 2024-04-12T05:08:51.766 INFO:tasks.workunit.client.0.smithi113.stdout:1/136: sync 2024-04-12T05:08:51.767 INFO:tasks.workunit.client.0.smithi113.stdout:9/273: sync 2024-04-12T05:08:51.767 INFO:tasks.workunit.client.0.smithi113.stdout:2/127: sync 2024-04-12T05:08:51.767 INFO:tasks.workunit.client.0.smithi113.stdout:7/178: sync 2024-04-12T05:08:51.767 INFO:tasks.workunit.client.0.smithi113.stdout:6/196: sync 2024-04-12T05:08:51.767 INFO:tasks.workunit.client.0.smithi113.stdout:8/224: sync 2024-04-12T05:08:51.767 INFO:tasks.workunit.client.0.smithi113.stdout:4/221: sync 2024-04-12T05:08:51.769 INFO:tasks.workunit.client.1.smithi192.stdout:0/16: sync 2024-04-12T05:08:51.769 INFO:tasks.workunit.client.1.smithi192.stdout:5/135: sync 2024-04-12T05:08:51.769 INFO:tasks.workunit.client.1.smithi192.stdout:2/63: sync 2024-04-12T05:08:51.769 INFO:tasks.workunit.client.1.smithi192.stdout:1/20: sync 2024-04-12T05:08:51.769 INFO:tasks.workunit.client.1.smithi192.stdout:4/53: sync 2024-04-12T05:08:51.769 INFO:tasks.workunit.client.1.smithi192.stdout:6/84: sync 2024-04-12T05:08:51.769 INFO:tasks.workunit.client.1.smithi192.stdout:8/27: sync 2024-04-12T05:08:51.770 INFO:tasks.workunit.client.1.smithi192.stdout:7/44: sync 2024-04-12T05:08:51.770 INFO:tasks.workunit.client.0.smithi113.stdout:2/128: fdatasync d8/d9/de/d15/f18 0 2024-04-12T05:08:51.770 INFO:tasks.workunit.client.1.smithi192.stdout:9/44: chown fd 0 1 2024-04-12T05:08:51.771 INFO:tasks.workunit.client.0.smithi113.stdout:6/197: fsync d8/de/d13/d11/d36/f35 0 2024-04-12T05:08:51.772 INFO:tasks.workunit.client.0.smithi113.stdout:7/179: creat d0/d26/f38 x:0 0 0 2024-04-12T05:08:51.772 INFO:tasks.workunit.client.1.smithi192.stdout:5/136: chown d0/ld 0 1 2024-04-12T05:08:51.773 INFO:tasks.workunit.client.0.smithi113.stdout:1/137: symlink d10/d14/l24 0 2024-04-12T05:08:51.773 INFO:tasks.workunit.client.0.smithi113.stdout:5/205: dread d0/d10/d27/d13/f28 [0,4194304] 0 2024-04-12T05:08:51.774 INFO:tasks.workunit.client.0.smithi113.stdout:8/225: dread - d9/d16/d1e/d26/d32/f3f zero size 2024-04-12T05:08:51.775 INFO:tasks.workunit.client.1.smithi192.stdout:3/37: symlink d5/le 0 2024-04-12T05:08:51.776 INFO:tasks.workunit.client.1.smithi192.stdout:2/64: symlink d1/d8/db/l14 0 2024-04-12T05:08:51.777 INFO:tasks.workunit.client.0.smithi113.stdout:4/222: rename da/dd/d1b/f23 to da/dd/df/f45 0 2024-04-12T05:08:51.777 INFO:tasks.workunit.client.0.smithi113.stdout:9/274: symlink d1/d4/d30/d37/d35/l5b 0 2024-04-12T05:08:51.778 INFO:tasks.workunit.client.0.smithi113.stdout:0/186: creat d1/d6/d16/d2a/d2c/f37 x:0 0 0 2024-04-12T05:08:51.779 INFO:tasks.workunit.client.1.smithi192.stdout:1/21: creat d0/d2/fa x:0 0 0 2024-04-12T05:08:51.780 INFO:tasks.workunit.client.0.smithi113.stdout:2/129: dread d8/d9/fc [0,4194304] 0 2024-04-12T05:08:51.780 INFO:tasks.workunit.client.0.smithi113.stdout:7/180: symlink d0/dd/l39 0 2024-04-12T05:08:51.781 INFO:tasks.workunit.client.1.smithi192.stdout:4/54: truncate d1/f14 613787 0 2024-04-12T05:08:51.781 INFO:tasks.workunit.client.1.smithi192.stdout:9/45: mknod cf 0 2024-04-12T05:08:51.781 INFO:tasks.workunit.client.1.smithi192.stdout:8/28: dread f5 [0,4194304] 0 2024-04-12T05:08:51.781 INFO:tasks.workunit.client.0.smithi113.stdout:0/187: write d1/d6/d16/d2a/d2c/f37 [647968,59611] 0 2024-04-12T05:08:51.782 INFO:tasks.workunit.client.1.smithi192.stdout:7/45: rename d3/f5 to d3/fd 0 2024-04-12T05:08:51.783 INFO:tasks.workunit.client.0.smithi113.stdout:8/226: mkdir d9/d16/d1e/d26/d32/d48 0 2024-04-12T05:08:51.783 INFO:tasks.workunit.client.0.smithi113.stdout:1/138: truncate d10/d14/f1f 167217 0 2024-04-12T05:08:51.785 INFO:tasks.workunit.client.1.smithi192.stdout:6/85: write d3/d8/f17 [4269087,48999] 0 2024-04-12T05:08:51.786 INFO:tasks.workunit.client.0.smithi113.stdout:8/227: fdatasync d9/d1b/f45 0 2024-04-12T05:08:51.786 INFO:tasks.workunit.client.1.smithi192.stdout:2/65: symlink d1/l15 0 2024-04-12T05:08:51.786 INFO:tasks.workunit.client.1.smithi192.stdout:7/46: stat d3/d7/f9 0 2024-04-12T05:08:51.787 INFO:tasks.workunit.client.0.smithi113.stdout:4/223: readlink da/dd/df/d1d/l41 0 2024-04-12T05:08:51.787 INFO:tasks.workunit.client.0.smithi113.stdout:8/228: chown d9/d16/d1e/c3d 15653 1 2024-04-12T05:08:51.787 INFO:tasks.workunit.client.1.smithi192.stdout:4/55: unlink d1/l3 0 2024-04-12T05:08:51.788 INFO:tasks.workunit.client.0.smithi113.stdout:2/130: creat d8/d9/f1f x:0 0 0 2024-04-12T05:08:51.788 INFO:tasks.workunit.client.1.smithi192.stdout:0/17: dwrite d0/f3 [0,4194304] 0 2024-04-12T05:08:51.788 INFO:tasks.workunit.client.1.smithi192.stdout:4/56: chown d1/de 0 1 2024-04-12T05:08:51.789 INFO:tasks.workunit.client.0.smithi113.stdout:6/198: creat d8/de/d13/d11/d36/d2c/f41 x:0 0 0 2024-04-12T05:08:51.789 INFO:tasks.workunit.client.1.smithi192.stdout:9/46: creat f10 x:0 0 0 2024-04-12T05:08:51.790 INFO:tasks.workunit.client.1.smithi192.stdout:0/18: write d0/f3 [1529479,78534] 0 2024-04-12T05:08:51.791 INFO:tasks.workunit.client.0.smithi113.stdout:4/224: creat da/dd/df/d1d/f46 x:0 0 0 2024-04-12T05:08:51.797 INFO:tasks.workunit.client.1.smithi192.stdout:6/86: creat d3/f1e x:0 0 0 2024-04-12T05:08:51.798 INFO:tasks.workunit.client.1.smithi192.stdout:7/47: creat d3/d7/fe x:0 0 0 2024-04-12T05:08:51.798 INFO:tasks.workunit.client.0.smithi113.stdout:2/131: mknod d8/d9/c20 0 2024-04-12T05:08:51.799 INFO:tasks.workunit.client.0.smithi113.stdout:7/181: creat d0/dc/d6/d1a/f3a x:0 0 0 2024-04-12T05:08:51.800 INFO:tasks.workunit.client.0.smithi113.stdout:4/225: mkdir da/dd/df/d1d/d47 0 2024-04-12T05:08:51.800 INFO:tasks.workunit.client.1.smithi192.stdout:9/47: mkdir d11 0 2024-04-12T05:08:51.801 INFO:tasks.workunit.client.1.smithi192.stdout:1/22: write d0/f8 [870354,56908] 0 2024-04-12T05:08:51.803 INFO:tasks.workunit.client.0.smithi113.stdout:4/226: fdatasync da/dd/df/d1d/f46 0 2024-04-12T05:08:51.803 INFO:tasks.workunit.client.0.smithi113.stdout:9/275: rename d1/df/d29/c41 to d1/d4/d5/c5c 0 2024-04-12T05:08:51.804 INFO:tasks.workunit.client.1.smithi192.stdout:0/19: dwrite d0/f3 [0,4194304] 0 2024-04-12T05:08:51.807 INFO:tasks.workunit.client.0.smithi113.stdout:0/188: rename d1/c17 to d1/d6/d16/d2a/d2c/c38 0 2024-04-12T05:08:51.807 INFO:tasks.workunit.client.1.smithi192.stdout:7/48: chown d3/f6 922167 1 2024-04-12T05:08:51.808 INFO:tasks.workunit.client.0.smithi113.stdout:2/132: dwrite d8/d16/f1d [0,4194304] 0 2024-04-12T05:08:51.809 INFO:tasks.workunit.client.0.smithi113.stdout:7/182: mknod d0/dc/d6/d1a/c3b 0 2024-04-12T05:08:51.810 INFO:tasks.workunit.client.1.smithi192.stdout:6/87: fsync d3/d8/f13 0 2024-04-12T05:08:51.811 INFO:tasks.workunit.client.0.smithi113.stdout:4/227: fdatasync da/dd/f32 0 2024-04-12T05:08:51.815 INFO:tasks.workunit.client.0.smithi113.stdout:6/199: rename c0 to d8/de/d17/c42 0 2024-04-12T05:08:51.815 INFO:tasks.workunit.client.0.smithi113.stdout:0/189: dwrite d1/f22 [4194304,4194304] 0 2024-04-12T05:08:51.816 INFO:tasks.workunit.client.1.smithi192.stdout:2/66: write d1/d8/db/ff [48991,67900] 0 2024-04-12T05:08:51.816 INFO:tasks.workunit.client.1.smithi192.stdout:7/49: dread d3/f6 [0,4194304] 0 2024-04-12T05:08:51.816 INFO:tasks.workunit.client.1.smithi192.stdout:4/57: dread d1/f14 [0,4194304] 0 2024-04-12T05:08:51.818 INFO:tasks.workunit.client.1.smithi192.stdout:1/23: mknod d0/d2/cb 0 2024-04-12T05:08:51.831 INFO:tasks.workunit.client.1.smithi192.stdout:3/38: dwrite d5/fd [0,4194304] 0 2024-04-12T05:08:51.835 INFO:tasks.workunit.client.1.smithi192.stdout:4/58: unlink d1/l7 0 2024-04-12T05:08:51.835 INFO:tasks.workunit.client.1.smithi192.stdout:8/29: dwrite f1 [0,4194304] 0 2024-04-12T05:08:51.837 INFO:tasks.workunit.client.1.smithi192.stdout:9/48: dwrite f10 [0,4194304] 0 2024-04-12T05:08:51.840 INFO:tasks.workunit.client.1.smithi192.stdout:9/49: chown f10 0 1 2024-04-12T05:08:51.848 INFO:tasks.workunit.client.1.smithi192.stdout:1/24: dwrite d0/d2/fa [0,4194304] 0 2024-04-12T05:08:51.850 INFO:tasks.workunit.client.1.smithi192.stdout:0/20: dread d0/f3 [0,4194304] 0 2024-04-12T05:08:51.851 INFO:tasks.workunit.client.1.smithi192.stdout:3/39: dread - d5/f9 zero size 2024-04-12T05:08:51.851 INFO:tasks.workunit.client.1.smithi192.stdout:6/88: link d3/d5/ff d3/d8/f1f 0 2024-04-12T05:08:51.851 INFO:tasks.workunit.client.1.smithi192.stdout:2/67: dread d1/d8/fc [0,4194304] 0 2024-04-12T05:08:51.852 INFO:tasks.workunit.client.1.smithi192.stdout:7/50: rename d3/d7/fe to d3/ff 0 2024-04-12T05:08:51.858 INFO:tasks.workunit.client.1.smithi192.stdout:6/89: mknod d3/d5/c20 0 2024-04-12T05:08:51.859 INFO:tasks.workunit.client.1.smithi192.stdout:1/25: rename d0/f8 to d0/fc 0 2024-04-12T05:08:51.859 INFO:tasks.workunit.client.1.smithi192.stdout:4/59: truncate d1/de/f12 291782 0 2024-04-12T05:08:51.860 INFO:tasks.workunit.client.1.smithi192.stdout:7/51: creat d3/d7/f10 x:0 0 0 2024-04-12T05:08:51.862 INFO:tasks.workunit.client.1.smithi192.stdout:0/21: creat d0/f8 x:0 0 0 2024-04-12T05:08:51.862 INFO:tasks.workunit.client.1.smithi192.stdout:2/68: dwrite d1/d8/db/ff [0,4194304] 0 2024-04-12T05:08:51.863 INFO:tasks.workunit.client.1.smithi192.stdout:3/40: mknod d5/cf 0 2024-04-12T05:08:51.864 INFO:tasks.workunit.client.1.smithi192.stdout:6/90: mkdir d3/d21 0 2024-04-12T05:08:51.866 INFO:tasks.workunit.client.1.smithi192.stdout:7/52: fdatasync d3/ff 0 2024-04-12T05:08:51.867 INFO:tasks.workunit.client.1.smithi192.stdout:4/60: truncate d1/f14 680986 0 2024-04-12T05:08:51.870 INFO:tasks.workunit.client.1.smithi192.stdout:2/69: unlink d1/d8/db/ff 0 2024-04-12T05:08:51.871 INFO:tasks.workunit.client.0.smithi113.stdout:7/183: mknod d0/dc/d6/d24/c3c 0 2024-04-12T05:08:51.871 INFO:tasks.workunit.client.1.smithi192.stdout:3/41: symlink d5/l10 0 2024-04-12T05:08:51.872 INFO:tasks.workunit.client.0.smithi113.stdout:1/139: read d10/d14/f1f [64612,72935] 0 2024-04-12T05:08:51.872 INFO:tasks.workunit.client.0.smithi113.stdout:3/172: sync 2024-04-12T05:08:51.873 INFO:tasks.workunit.client.0.smithi113.stdout:8/229: rename d9/d34/f36 to d9/d16/d1e/d26/d32/d48/f49 0 2024-04-12T05:08:51.873 INFO:tasks.workunit.client.0.smithi113.stdout:6/200: rmdir d8/d1d 39 2024-04-12T05:08:51.875 INFO:tasks.workunit.client.1.smithi192.stdout:3/42: write d5/fd [1914561,42259] 0 2024-04-12T05:08:51.875 INFO:tasks.workunit.client.1.smithi192.stdout:6/91: mkdir d3/d8/d22 0 2024-04-12T05:08:51.875 INFO:tasks.workunit.client.1.smithi192.stdout:3/43: dread - d5/f9 zero size 2024-04-12T05:08:51.875 INFO:tasks.workunit.client.1.smithi192.stdout:6/92: dread - d3/f1e zero size 2024-04-12T05:08:51.877 INFO:tasks.workunit.client.0.smithi113.stdout:0/190: dread d1/d6/d16/f24 [0,4194304] 0 2024-04-12T05:08:51.878 INFO:tasks.workunit.client.0.smithi113.stdout:1/140: symlink d10/d21/l25 0 2024-04-12T05:08:51.878 INFO:tasks.workunit.client.1.smithi192.stdout:7/53: unlink d3/lb 0 2024-04-12T05:08:51.879 INFO:tasks.workunit.client.1.smithi192.stdout:9/50: write fe [51740,106539] 0 2024-04-12T05:08:51.881 INFO:tasks.workunit.client.0.smithi113.stdout:9/276: rename d1/d4/d5/f4c to d1/d27/f5d 0 2024-04-12T05:08:51.882 INFO:tasks.workunit.client.0.smithi113.stdout:8/230: mkdir d9/d34/d4a 0 2024-04-12T05:08:51.882 INFO:tasks.workunit.client.1.smithi192.stdout:4/61: creat d1/f15 x:0 0 0 2024-04-12T05:08:51.882 INFO:tasks.workunit.client.0.smithi113.stdout:6/201: dread d8/f22 [0,4194304] 0 2024-04-12T05:08:51.883 INFO:tasks.workunit.client.1.smithi192.stdout:4/62: dread - d1/f15 zero size 2024-04-12T05:08:51.883 INFO:tasks.workunit.client.0.smithi113.stdout:4/228: creat da/dd/f48 x:0 0 0 2024-04-12T05:08:51.883 INFO:tasks.workunit.client.1.smithi192.stdout:4/63: rename d1/de to d1/de/d16 22 2024-04-12T05:08:51.884 INFO:tasks.workunit.client.0.smithi113.stdout:0/191: mknod d1/d9/c39 0 2024-04-12T05:08:51.885 INFO:tasks.workunit.client.0.smithi113.stdout:8/231: rename d9/d34/d4a to d9/d34/d4a/d4b 22 2024-04-12T05:08:51.888 INFO:tasks.workunit.client.1.smithi192.stdout:6/93: mkdir d3/d23 0 2024-04-12T05:08:51.890 INFO:tasks.workunit.client.0.smithi113.stdout:4/229: write da/d2b/f3e [860418,53547] 0 2024-04-12T05:08:51.892 INFO:tasks.workunit.client.0.smithi113.stdout:0/192: creat d1/d6/da/f3a x:0 0 0 2024-04-12T05:08:51.892 INFO:tasks.workunit.client.0.smithi113.stdout:7/184: rmdir d0/dc 39 2024-04-12T05:08:51.892 INFO:tasks.workunit.client.1.smithi192.stdout:7/54: mkdir d3/d11 0 2024-04-12T05:08:51.895 INFO:tasks.workunit.client.0.smithi113.stdout:6/202: rename d8/de/d13/d11/d36/f34 to d8/de/d13/d11/f43 0 2024-04-12T05:08:51.895 INFO:tasks.workunit.client.1.smithi192.stdout:4/64: mknod d1/c17 0 2024-04-12T05:08:51.895 INFO:tasks.workunit.client.0.smithi113.stdout:4/230: readlink da/dd/df/d14/l20 0 2024-04-12T05:08:51.896 INFO:tasks.workunit.client.1.smithi192.stdout:1/26: write d0/f3 [1006845,52966] 0 2024-04-12T05:08:51.897 INFO:tasks.workunit.client.1.smithi192.stdout:1/27: chown d0/d2/cb 130704821 1 2024-04-12T05:08:51.898 INFO:tasks.workunit.client.0.smithi113.stdout:3/173: dread d0/d13/d15/f2e [0,4194304] 0 2024-04-12T05:08:51.900 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:50.669048+0000 mgr.smithi192.ocppyo (mgr.24357) 21 : cluster [DBG] pgmap v15: 65 pgs: 65 active+clean; 356 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 93 KiB/s rd, 10 MiB/s wr, 114 op/s 2024-04-12T05:08:51.901 INFO:tasks.workunit.client.1.smithi192.stdout:0/22: getdents d0 0 2024-04-12T05:08:51.902 INFO:tasks.workunit.client.0.smithi113.stdout:6/203: fdatasync d8/de/d13/d11/f43 0 2024-04-12T05:08:51.903 INFO:tasks.workunit.client.0.smithi113.stdout:8/232: dwrite d9/d16/d1e/d3a/f2a [0,4194304] 0 2024-04-12T05:08:51.905 INFO:tasks.workunit.client.1.smithi192.stdout:7/55: creat d3/f12 x:0 0 0 2024-04-12T05:08:51.905 INFO:tasks.workunit.client.1.smithi192.stdout:9/51: creat d11/f12 x:0 0 0 2024-04-12T05:08:51.906 INFO:tasks.workunit.client.0.smithi113.stdout:3/174: truncate d0/d13/f21 2333396 0 2024-04-12T05:08:51.906 INFO:tasks.workunit.client.1.smithi192.stdout:9/52: chown f9 36563323 1 2024-04-12T05:08:51.908 INFO:tasks.workunit.client.0.smithi113.stdout:7/185: rename d0/d26/c32 to d0/dc/d6/c3d 0 2024-04-12T05:08:51.909 INFO:tasks.workunit.client.1.smithi192.stdout:4/65: creat d1/f18 x:0 0 0 2024-04-12T05:08:51.912 INFO:tasks.workunit.client.0.smithi113.stdout:0/193: write d1/f2 [306950,101148] 0 2024-04-12T05:08:51.915 INFO:tasks.workunit.client.1.smithi192.stdout:3/44: link d5/cf d5/c11 0 2024-04-12T05:08:51.917 INFO:tasks.workunit.client.0.smithi113.stdout:8/233: symlink d9/l4c 0 2024-04-12T05:08:51.917 INFO:tasks.workunit.client.0.smithi113.stdout:4/231: dwrite da/dd/d1b/f28 [0,4194304] 0 2024-04-12T05:08:51.918 INFO:tasks.workunit.client.0.smithi113.stdout:3/175: read d0/f12 [129078,19139] 0 2024-04-12T05:08:51.919 INFO:tasks.workunit.client.0.smithi113.stdout:9/277: dread d1/d4/d5/f12 [0,4194304] 0 2024-04-12T05:08:51.920 INFO:tasks.workunit.client.1.smithi192.stdout:9/53: unlink f2 0 2024-04-12T05:08:51.921 INFO:tasks.workunit.client.0.smithi113.stdout:6/204: read d8/f12 [3895398,130991] 0 2024-04-12T05:08:51.921 INFO:tasks.workunit.client.0.smithi113.stdout:0/194: rmdir d1/d6/da 39 2024-04-12T05:08:51.922 INFO:tasks.workunit.client.0.smithi113.stdout:1/141: dwrite d10/d14/f1f [0,4194304] 0 2024-04-12T05:08:51.922 INFO:tasks.workunit.client.0.smithi113.stdout:8/234: dread - d9/d16/d1c/f2d zero size 2024-04-12T05:08:51.923 INFO:tasks.workunit.client.1.smithi192.stdout:2/70: dwrite d1/f4 [0,4194304] 0 2024-04-12T05:08:51.925 INFO:tasks.workunit.client.1.smithi192.stdout:9/54: chown fd 60078685 1 2024-04-12T05:08:51.925 INFO:tasks.workunit.client.0.smithi113.stdout:1/142: chown d10 4575183 1 2024-04-12T05:08:51.926 INFO:tasks.workunit.client.0.smithi113.stdout:3/176: mknod d0/d13/c3c 0 2024-04-12T05:08:51.926 INFO:tasks.workunit.client.0.smithi113.stdout:0/195: chown d1/c31 523974 1 2024-04-12T05:08:51.927 INFO:tasks.workunit.client.1.smithi192.stdout:1/28: dwrite d0/fc [0,4194304] 0 2024-04-12T05:08:51.928 INFO:tasks.workunit.client.1.smithi192.stdout:3/45: rmdir d5 39 2024-04-12T05:08:51.929 INFO:tasks.workunit.client.0.smithi113.stdout:6/205: chown d8/de/d17/c1e 1922 1 2024-04-12T05:08:51.930 INFO:tasks.workunit.client.0.smithi113.stdout:8/235: creat d9/d16/d1e/d26/f4d x:0 0 0 2024-04-12T05:08:51.930 INFO:tasks.workunit.client.0.smithi113.stdout:0/196: write d1/fe [3188576,32210] 0 2024-04-12T05:08:51.933 INFO:tasks.workunit.client.0.smithi113.stdout:1/143: unlink d10/f16 0 2024-04-12T05:08:51.934 INFO:tasks.workunit.client.1.smithi192.stdout:2/71: symlink d1/d8/db/l16 0 2024-04-12T05:08:51.935 INFO:tasks.workunit.client.1.smithi192.stdout:7/56: dread d3/fd [0,4194304] 0 2024-04-12T05:08:51.936 INFO:tasks.workunit.client.0.smithi113.stdout:0/197: write d1/f2 [658883,94420] 0 2024-04-12T05:08:51.936 INFO:tasks.workunit.client.0.smithi113.stdout:6/206: chown d8/c29 472503432 1 2024-04-12T05:08:51.937 INFO:tasks.workunit.client.0.smithi113.stdout:8/236: dread - d9/d16/d1c/f2d zero size 2024-04-12T05:08:51.939 INFO:tasks.workunit.client.0.smithi113.stdout:1/144: write fe [2559283,82281] 0 2024-04-12T05:08:51.940 INFO:tasks.workunit.client.1.smithi192.stdout:0/23: getdents d0/d6 0 2024-04-12T05:08:51.941 INFO:tasks.workunit.client.0.smithi113.stdout:3/177: rename d0/f8 to d0/d13/d15/f3d 0 2024-04-12T05:08:51.945 INFO:tasks.workunit.client.1.smithi192.stdout:9/55: link f10 d11/f13 0 2024-04-12T05:08:51.945 INFO:tasks.workunit.client.0.smithi113.stdout:1/145: chown d10/d14/c1a 1618233 1 2024-04-12T05:08:51.945 INFO:tasks.workunit.client.0.smithi113.stdout:4/232: dwrite da/fb [0,4194304] 0 2024-04-12T05:08:51.946 INFO:tasks.workunit.client.1.smithi192.stdout:6/94: dwrite d3/d5/ff [0,4194304] 0 2024-04-12T05:08:51.948 INFO:tasks.workunit.client.0.smithi113.stdout:7/186: dwrite d0/dc/d6/d1a/f34 [0,4194304] 0 2024-04-12T05:08:51.949 INFO:tasks.workunit.client.1.smithi192.stdout:2/72: mkdir d1/d8/d17 0 2024-04-12T05:08:51.950 INFO:tasks.workunit.client.0.smithi113.stdout:8/237: unlink d9/d16/c41 0 2024-04-12T05:08:51.951 INFO:tasks.workunit.client.0.smithi113.stdout:7/187: fsync d0/dd/f1c 0 2024-04-12T05:08:51.952 INFO:tasks.workunit.client.1.smithi192.stdout:1/29: dwrite d0/fc [0,4194304] 0 2024-04-12T05:08:51.952 INFO:tasks.workunit.client.1.smithi192.stdout:3/46: symlink d5/l12 0 2024-04-12T05:08:51.953 INFO:tasks.workunit.client.0.smithi113.stdout:3/178: creat d0/d13/d15/f3e x:0 0 0 2024-04-12T05:08:51.955 INFO:tasks.workunit.client.0.smithi113.stdout:7/188: stat d0/dc/d6/d1a 0 2024-04-12T05:08:51.958 INFO:tasks.workunit.client.1.smithi192.stdout:2/73: symlink d1/d8/db/de/l18 0 2024-04-12T05:08:51.961 INFO:tasks.workunit.client.1.smithi192.stdout:1/30: mkdir d0/d2/dd 0 2024-04-12T05:08:51.961 INFO:tasks.workunit.client.1.smithi192.stdout:3/47: mkdir d5/d13 0 2024-04-12T05:08:51.961 INFO:tasks.workunit.client.0.smithi113.stdout:7/189: write d0/dc/d6/fa [4327598,114672] 0 2024-04-12T05:08:51.966 INFO:tasks.workunit.client.1.smithi192.stdout:0/24: link d0/d6/c7 d0/c9 0 2024-04-12T05:08:51.966 INFO:tasks.workunit.client.1.smithi192.stdout:6/95: dwrite d3/d8/f1f [4194304,4194304] 0 2024-04-12T05:08:51.966 INFO:tasks.workunit.client.1.smithi192.stdout:0/25: write d0/f8 [622998,26840] 0 2024-04-12T05:08:51.967 INFO:tasks.workunit.client.0.smithi113.stdout:9/278: link d1/d4/d30/d37/d35/l3a d1/d4/d30/l5e 0 2024-04-12T05:08:51.968 INFO:tasks.workunit.client.1.smithi192.stdout:6/96: dread - d3/f1e zero size 2024-04-12T05:08:51.969 INFO:tasks.workunit.client.0.smithi113.stdout:1/146: dread d10/d14/f1f [0,4194304] 0 2024-04-12T05:08:51.969 INFO:tasks.workunit.client.0.smithi113.stdout:0/198: dread d1/f22 [0,4194304] 0 2024-04-12T05:08:51.972 INFO:tasks.workunit.client.0.smithi113.stdout:4/233: dwrite da/dd/df/d1d/f46 [0,4194304] 0 2024-04-12T05:08:51.973 INFO:tasks.workunit.client.1.smithi192.stdout:9/56: rmdir d11 39 2024-04-12T05:08:51.976 INFO:tasks.workunit.client.1.smithi192.stdout:4/66: write d1/de/f12 [708337,38422] 0 2024-04-12T05:08:51.977 INFO:tasks.workunit.client.0.smithi113.stdout:8/238: dwrite d9/fd [0,4194304] 0 2024-04-12T05:08:51.979 INFO:tasks.workunit.client.1.smithi192.stdout:6/97: dwrite d3/fd [4194304,4194304] 0 2024-04-12T05:08:51.980 INFO:tasks.workunit.client.0.smithi113.stdout:3/179: creat d0/d13/d15/d28/f3f x:0 0 0 2024-04-12T05:08:51.980 INFO:tasks.workunit.client.1.smithi192.stdout:1/31: dread d0/f3 [0,4194304] 0 2024-04-12T05:08:51.982 INFO:tasks.workunit.client.0.smithi113.stdout:5/206: sync 2024-04-12T05:08:51.982 INFO:tasks.workunit.client.0.smithi113.stdout:2/133: sync 2024-04-12T05:08:51.983 INFO:tasks.workunit.client.1.smithi192.stdout:9/57: mkdir d11/d14 0 2024-04-12T05:08:51.983 INFO:tasks.workunit.client.1.smithi192.stdout:9/58: stat c0 0 2024-04-12T05:08:51.984 INFO:tasks.workunit.client.0.smithi113.stdout:1/147: truncate f2 2155432 0 2024-04-12T05:08:51.984 INFO:tasks.workunit.client.1.smithi192.stdout:3/48: dread f2 [0,4194304] 0 2024-04-12T05:08:51.985 INFO:tasks.workunit.client.1.smithi192.stdout:4/67: chown d1/da/cf 4 1 2024-04-12T05:08:51.986 INFO:tasks.workunit.client.0.smithi113.stdout:1/148: stat d10/d14 0 2024-04-12T05:08:51.988 INFO:tasks.workunit.client.1.smithi192.stdout:3/49: readlink d5/l12 0 2024-04-12T05:08:51.988 INFO:tasks.workunit.client.0.smithi113.stdout:5/207: read - d0/d10/d27/d33/f35 zero size 2024-04-12T05:08:51.989 INFO:tasks.workunit.client.1.smithi192.stdout:1/32: write d0/d2/f9 [674554,87335] 0 2024-04-12T05:08:51.990 INFO:tasks.workunit.client.1.smithi192.stdout:6/98: mkdir d3/d21/d24 0 2024-04-12T05:08:51.990 INFO:tasks.workunit.client.0.smithi113.stdout:3/180: mknod d0/d13/d15/d28/d2a/c40 0 2024-04-12T05:08:51.991 INFO:tasks.workunit.client.1.smithi192.stdout:9/59: getdents d11/d14 0 2024-04-12T05:08:51.993 INFO:tasks.workunit.client.0.smithi113.stdout:5/208: chown d0/d10/l2d 57 1 2024-04-12T05:08:51.994 INFO:tasks.workunit.client.1.smithi192.stdout:0/26: fsync d0/f8 0 2024-04-12T05:08:51.994 INFO:tasks.workunit.client.1.smithi192.stdout:7/57: write f1 [1054365,122538] 0 2024-04-12T05:08:51.995 INFO:tasks.workunit.client.0.smithi113.stdout:1/149: rename d10/c12 to d10/d21/c26 0 2024-04-12T05:08:51.996 INFO:tasks.workunit.client.1.smithi192.stdout:1/33: mkdir d0/de 0 2024-04-12T05:08:51.998 INFO:tasks.workunit.client.1.smithi192.stdout:0/27: write d0/f8 [824524,111140] 0 2024-04-12T05:08:51.998 INFO:tasks.workunit.client.1.smithi192.stdout:7/58: truncate d3/ff 307099 0 2024-04-12T05:08:51.999 INFO:tasks.workunit.client.0.smithi113.stdout:1/150: chown d10/d21 4668 1 2024-04-12T05:08:52.003 INFO:tasks.workunit.client.1.smithi192.stdout:6/99: getdents d3/d23 0 2024-04-12T05:08:52.003 INFO:tasks.workunit.client.1.smithi192.stdout:4/68: truncate d1/f14 277544 0 2024-04-12T05:08:52.004 INFO:tasks.workunit.client.1.smithi192.stdout:6/100: stat d3/dc/f11 0 2024-04-12T05:08:52.005 INFO:tasks.workunit.client.1.smithi192.stdout:3/50: dwrite d5/fd [0,4194304] 0 2024-04-12T05:08:52.005 INFO:tasks.workunit.client.1.smithi192.stdout:7/59: creat d3/d7/f13 x:0 0 0 2024-04-12T05:08:52.005 INFO:tasks.workunit.client.1.smithi192.stdout:3/51: dread - d5/f9 zero size 2024-04-12T05:08:52.006 INFO:tasks.workunit.client.0.smithi113.stdout:4/234: dread da/dd/df/d14/f19 [0,4194304] 0 2024-04-12T05:08:52.006 INFO:tasks.workunit.client.0.smithi113.stdout:5/209: mknod d0/d10/d27/d1d/c37 0 2024-04-12T05:08:52.006 INFO:tasks.workunit.client.0.smithi113.stdout:0/199: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:52.006 INFO:tasks.workunit.client.0.smithi113.stdout:7/190: dwrite d0/dc/d6/d1a/f34 [0,4194304] 0 2024-04-12T05:08:52.007 INFO:tasks.workunit.client.0.smithi113.stdout:2/134: dread d8/d16/f19 [0,4194304] 0 2024-04-12T05:08:52.008 INFO:tasks.workunit.client.1.smithi192.stdout:4/69: chown d1/f2 175410857 1 2024-04-12T05:08:52.008 INFO:tasks.workunit.client.1.smithi192.stdout:0/28: write d0/f3 [12742,51367] 0 2024-04-12T05:08:52.009 INFO:tasks.workunit.client.1.smithi192.stdout:3/52: chown d5/l10 59576 1 2024-04-12T05:08:52.009 INFO:tasks.workunit.client.1.smithi192.stdout:1/34: symlink d0/d2/dd/lf 0 2024-04-12T05:08:52.009 INFO:tasks.workunit.client.1.smithi192.stdout:3/53: chown d5/d13 394552 1 2024-04-12T05:08:52.012 INFO:tasks.workunit.client.1.smithi192.stdout:6/101: unlink d3/dc/c12 0 2024-04-12T05:08:52.012 INFO:tasks.workunit.client.1.smithi192.stdout:6/102: readlink d3/l1b 0 2024-04-12T05:08:52.013 INFO:tasks.workunit.client.0.smithi113.stdout:5/210: write d0/d10/d27/f26 [191575,80689] 0 2024-04-12T05:08:52.014 INFO:tasks.workunit.client.0.smithi113.stdout:1/151: symlink d10/d21/d23/l27 0 2024-04-12T05:08:52.015 INFO:tasks.workunit.client.1.smithi192.stdout:2/74: dwrite d1/d8/fc [0,4194304] 0 2024-04-12T05:08:52.016 INFO:tasks.workunit.client.1.smithi192.stdout:7/60: mkdir d3/d7/d14 0 2024-04-12T05:08:52.016 INFO:tasks.workunit.client.0.smithi113.stdout:4/235: read da/dd/df/d14/f19 [4050124,56879] 0 2024-04-12T05:08:52.016 INFO:tasks.workunit.client.1.smithi192.stdout:7/61: dread - d3/f12 zero size 2024-04-12T05:08:52.017 INFO:tasks.workunit.client.1.smithi192.stdout:2/75: chown d1/d8/fc 76 1 2024-04-12T05:08:52.017 INFO:tasks.workunit.client.1.smithi192.stdout:1/35: mkdir d0/d2/dd/d10 0 2024-04-12T05:08:52.019 INFO:tasks.workunit.client.1.smithi192.stdout:9/60: write f10 [434524,32825] 0 2024-04-12T05:08:52.021 INFO:tasks.workunit.client.0.smithi113.stdout:6/207: dread d8/de/f19 [0,4194304] 0 2024-04-12T05:08:52.022 INFO:tasks.workunit.client.1.smithi192.stdout:0/29: rename d0/f3 to d0/fa 0 2024-04-12T05:08:52.023 INFO:tasks.workunit.client.1.smithi192.stdout:3/54: creat d5/f14 x:0 0 0 2024-04-12T05:08:52.023 INFO:tasks.workunit.client.0.smithi113.stdout:9/279: dwrite d1/f46 [0,4194304] 0 2024-04-12T05:08:52.024 INFO:tasks.workunit.client.1.smithi192.stdout:1/36: write d0/f3 [541166,39875] 0 2024-04-12T05:08:52.025 INFO:tasks.workunit.client.1.smithi192.stdout:6/103: rename d3/d5/f7 to d3/d8/f25 0 2024-04-12T05:08:52.026 INFO:tasks.workunit.client.0.smithi113.stdout:7/191: creat d0/dc/d6/d1a/f3e x:0 0 0 2024-04-12T05:08:52.026 INFO:tasks.workunit.client.0.smithi113.stdout:1/152: write ff [247630,48343] 0 2024-04-12T05:08:52.029 INFO:tasks.workunit.client.0.smithi113.stdout:0/200: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:52.029 INFO:tasks.workunit.client.0.smithi113.stdout:4/236: creat da/d3c/d43/f49 x:0 0 0 2024-04-12T05:08:52.032 INFO:tasks.workunit.client.1.smithi192.stdout:7/62: dwrite d3/ff [0,4194304] 0 2024-04-12T05:08:52.034 INFO:tasks.workunit.client.1.smithi192.stdout:7/63: fsync d3/f6 0 2024-04-12T05:08:52.034 INFO:tasks.workunit.client.0.smithi113.stdout:0/201: mkdir d1/d6/d16/d3b 0 2024-04-12T05:08:52.035 INFO:tasks.workunit.client.0.smithi113.stdout:4/237: chown da/dd/df/f1c 131531482 1 2024-04-12T05:08:52.037 INFO:tasks.workunit.client.0.smithi113.stdout:7/192: symlink d0/d26/l3f 0 2024-04-12T05:08:52.038 INFO:tasks.workunit.client.0.smithi113.stdout:9/280: chown d1/d4/d38/c47 46 1 2024-04-12T05:08:52.038 INFO:tasks.workunit.client.0.smithi113.stdout:4/238: stat da/dd/c15 0 2024-04-12T05:08:52.039 INFO:tasks.workunit.client.0.smithi113.stdout:7/193: chown d0/dd/d11/c35 323 1 2024-04-12T05:08:52.042 INFO:tasks.workunit.client.0.smithi113.stdout:5/211: dwrite d0/f14 [4194304,4194304] 0 2024-04-12T05:08:52.042 INFO:tasks.workunit.client.0.smithi113.stdout:6/208: dread d8/de/d17/d25/f28 [0,4194304] 0 2024-04-12T05:08:52.044 INFO:tasks.workunit.client.0.smithi113.stdout:7/194: chown d0/dc/d6/d1a/f3e 14312 1 2024-04-12T05:08:52.045 INFO:tasks.workunit.client.1.smithi192.stdout:9/61: mkdir d11/d15 0 2024-04-12T05:08:52.046 INFO:tasks.workunit.client.0.smithi113.stdout:4/239: mkdir da/d4a 0 2024-04-12T05:08:52.046 INFO:tasks.workunit.client.1.smithi192.stdout:3/55: rename d5/cb to d5/c15 0 2024-04-12T05:08:52.046 INFO:tasks.workunit.client.0.smithi113.stdout:0/202: read - d1/d6/da/f3a zero size 2024-04-12T05:08:52.048 INFO:tasks.workunit.client.0.smithi113.stdout:9/281: creat d1/d4/d30/d37/d35/d3d/f5f x:0 0 0 2024-04-12T05:08:52.048 INFO:tasks.workunit.client.0.smithi113.stdout:1/153: dread d10/f1e [0,4194304] 0 2024-04-12T05:08:52.049 INFO:tasks.workunit.client.1.smithi192.stdout:0/30: chown d0/c9 1243666 1 2024-04-12T05:08:52.049 INFO:tasks.workunit.client.1.smithi192.stdout:2/76: mkdir d1/d8/d17/d19 0 2024-04-12T05:08:52.049 INFO:tasks.workunit.client.1.smithi192.stdout:0/31: fsync d0/f8 0 2024-04-12T05:08:52.049 INFO:tasks.workunit.client.1.smithi192.stdout:6/104: mkdir d3/d26 0 2024-04-12T05:08:52.050 INFO:tasks.workunit.client.0.smithi113.stdout:7/195: mknod d0/dd/c40 0 2024-04-12T05:08:52.050 INFO:tasks.workunit.client.1.smithi192.stdout:1/37: dread - d0/d2/f7 zero size 2024-04-12T05:08:52.050 INFO:tasks.workunit.client.1.smithi192.stdout:4/70: link d1/da/cd d1/da/c19 0 2024-04-12T05:08:52.051 INFO:tasks.workunit.client.0.smithi113.stdout:9/282: dread - d1/d4/d5/d22/f5a zero size 2024-04-12T05:08:52.051 INFO:tasks.workunit.client.0.smithi113.stdout:4/240: write da/dd/f21 [3771316,93360] 0 2024-04-12T05:08:52.051 INFO:tasks.workunit.client.0.smithi113.stdout:0/203: chown d1/d6/da/l15 488902093 1 2024-04-12T05:08:52.055 INFO:tasks.workunit.client.1.smithi192.stdout:9/62: dread fe [0,4194304] 0 2024-04-12T05:08:52.056 INFO:tasks.workunit.client.1.smithi192.stdout:1/38: fsync d0/f3 0 2024-04-12T05:08:52.059 INFO:tasks.workunit.client.1.smithi192.stdout:7/64: dwrite d3/d7/f13 [0,4194304] 0 2024-04-12T05:08:52.059 INFO:tasks.workunit.client.1.smithi192.stdout:1/39: fsync d0/fc 0 2024-04-12T05:08:52.066 INFO:tasks.workunit.client.1.smithi192.stdout:2/77: dwrite d1/d8/fc [0,4194304] 0 2024-04-12T05:08:52.068 INFO:tasks.workunit.client.1.smithi192.stdout:9/63: dread d11/f13 [0,4194304] 0 2024-04-12T05:08:52.072 INFO:tasks.workunit.client.1.smithi192.stdout:7/65: dwrite d3/d7/f10 [0,4194304] 0 2024-04-12T05:08:52.119 INFO:tasks.workunit.client.1.smithi192.stdout:5/137: sync 2024-04-12T05:08:52.120 INFO:tasks.workunit.client.1.smithi192.stdout:0/32: write d0/fa [5135185,69192] 0 2024-04-12T05:08:52.121 INFO:tasks.workunit.client.0.smithi113.stdout:0/204: dread d1/d9/f27 [0,4194304] 0 2024-04-12T05:08:52.121 INFO:tasks.workunit.client.0.smithi113.stdout:4/241: symlink da/d24/l4b 0 2024-04-12T05:08:52.121 INFO:tasks.workunit.client.0.smithi113.stdout:6/209: read d8/de/d13/d11/d36/f23 [654832,111420] 0 2024-04-12T05:08:52.121 INFO:tasks.workunit.client.1.smithi192.stdout:0/33: readlink d0/l4 0 2024-04-12T05:08:52.121 INFO:tasks.workunit.client.1.smithi192.stdout:0/34: chown d0/d6 1806 1 2024-04-12T05:08:52.122 INFO:tasks.workunit.client.0.smithi113.stdout:7/196: unlink d0/dc/d6/d24/l36 0 2024-04-12T05:08:52.123 INFO:tasks.workunit.client.1.smithi192.stdout:1/40: symlink d0/d2/dd/l11 0 2024-04-12T05:08:52.123 INFO:tasks.workunit.client.0.smithi113.stdout:0/205: read d1/fe [2636097,26272] 0 2024-04-12T05:08:52.125 INFO:tasks.workunit.client.0.smithi113.stdout:6/210: rmdir d8 39 2024-04-12T05:08:52.127 INFO:tasks.workunit.client.0.smithi113.stdout:3/181: dread d0/d13/d15/f3d [0,4194304] 0 2024-04-12T05:08:52.127 INFO:tasks.workunit.client.1.smithi192.stdout:6/105: creat d3/d8/d22/f27 x:0 0 0 2024-04-12T05:08:52.129 INFO:tasks.workunit.client.1.smithi192.stdout:5/138: fsync d0/f1b 0 2024-04-12T05:08:52.129 INFO:tasks.workunit.client.1.smithi192.stdout:2/78: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:52.130 INFO:tasks.workunit.client.1.smithi192.stdout:6/106: chown d3/d5/ca 2447 1 2024-04-12T05:08:52.130 INFO:tasks.workunit.client.0.smithi113.stdout:3/182: fdatasync d0/d13/d15/d28/f3f 0 2024-04-12T05:08:52.131 INFO:tasks.workunit.client.1.smithi192.stdout:2/79: chown d1/d8/ca 745 1 2024-04-12T05:08:52.131 INFO:tasks.workunit.client.1.smithi192.stdout:0/35: rename d0/fa to d0/fb 0 2024-04-12T05:08:52.131 INFO:tasks.workunit.client.0.smithi113.stdout:0/206: mknod d1/d6/d16/c3c 0 2024-04-12T05:08:52.132 INFO:tasks.workunit.client.0.smithi113.stdout:4/242: dwrite da/dd/df/d14/f18 [0,4194304] 0 2024-04-12T05:08:52.134 INFO:tasks.workunit.client.1.smithi192.stdout:9/64: symlink d11/d14/l16 0 2024-04-12T05:08:52.134 INFO:tasks.workunit.client.0.smithi113.stdout:5/212: fdatasync d0/f14 0 2024-04-12T05:08:52.135 INFO:tasks.workunit.client.0.smithi113.stdout:9/283: link d1/c10 d1/d4/d5/d6/d15/c60 0 2024-04-12T05:08:52.135 INFO:tasks.workunit.client.1.smithi192.stdout:2/80: dread d1/f4 [0,4194304] 0 2024-04-12T05:08:52.136 INFO:tasks.workunit.client.0.smithi113.stdout:1/154: write d10/f13 [2308686,30185] 0 2024-04-12T05:08:52.136 INFO:tasks.workunit.client.0.smithi113.stdout:5/213: chown d0/f2e 6 1 2024-04-12T05:08:52.141 INFO:tasks.workunit.client.0.smithi113.stdout:3/183: rmdir d0 39 2024-04-12T05:08:52.141 INFO:tasks.workunit.client.0.smithi113.stdout:0/207: fdatasync d1/d9/f1a 0 2024-04-12T05:08:52.141 INFO:tasks.workunit.client.0.smithi113.stdout:4/243: chown da/dd/df/d1d/f40 259 1 2024-04-12T05:08:52.142 INFO:tasks.workunit.client.0.smithi113.stdout:9/284: fdatasync d1/d4/d5/f12 0 2024-04-12T05:08:52.143 INFO:tasks.workunit.client.1.smithi192.stdout:9/65: dread - f9 zero size 2024-04-12T05:08:52.143 INFO:tasks.workunit.client.1.smithi192.stdout:6/107: creat d3/d26/f28 x:0 0 0 2024-04-12T05:08:52.144 INFO:tasks.workunit.client.1.smithi192.stdout:2/81: rmdir d1/d8/db 39 2024-04-12T05:08:52.145 INFO:tasks.workunit.client.0.smithi113.stdout:7/197: dread d0/dc/f27 [0,4194304] 0 2024-04-12T05:08:52.145 INFO:tasks.workunit.client.0.smithi113.stdout:5/214: dread d0/d10/d27/d13/f28 [0,4194304] 0 2024-04-12T05:08:52.145 INFO:tasks.workunit.client.0.smithi113.stdout:1/155: mkdir d10/d21/d23/d28 0 2024-04-12T05:08:52.145 INFO:tasks.workunit.client.1.smithi192.stdout:6/108: dread - d3/d26/f28 zero size 2024-04-12T05:08:52.146 INFO:tasks.workunit.client.1.smithi192.stdout:6/109: read d3/fd [2623138,23551] 0 2024-04-12T05:08:52.146 INFO:tasks.workunit.client.1.smithi192.stdout:1/41: rmdir d0/d2/dd/d10 0 2024-04-12T05:08:52.146 INFO:tasks.workunit.client.0.smithi113.stdout:4/244: write da/dd/f48 [536648,123950] 0 2024-04-12T05:08:52.146 INFO:tasks.workunit.client.1.smithi192.stdout:1/42: read d0/d2/f9 [232280,6700] 0 2024-04-12T05:08:52.147 INFO:tasks.workunit.client.1.smithi192.stdout:5/139: link d0/d15/d24/f25 d0/d21/f36 0 2024-04-12T05:08:52.149 INFO:tasks.workunit.client.0.smithi113.stdout:0/208: read d1/d6/d16/d2a/d2c/f37 [317948,58894] 0 2024-04-12T05:08:52.149 INFO:tasks.workunit.client.0.smithi113.stdout:3/184: write d0/f9 [1888487,12017] 0 2024-04-12T05:08:52.150 INFO:tasks.workunit.client.1.smithi192.stdout:9/66: mknod d11/d14/c17 0 2024-04-12T05:08:52.152 INFO:tasks.workunit.client.1.smithi192.stdout:1/43: dread d0/f3 [0,4194304] 0 2024-04-12T05:08:52.153 INFO:tasks.workunit.client.0.smithi113.stdout:7/198: truncate d0/dc/d6/f28 899048 0 2024-04-12T05:08:52.153 INFO:tasks.workunit.client.0.smithi113.stdout:3/185: chown d0/d13/d15 11 1 2024-04-12T05:08:52.153 INFO:tasks.workunit.client.0.smithi113.stdout:1/156: creat d10/d21/f29 x:0 0 0 2024-04-12T05:08:52.154 INFO:tasks.workunit.client.1.smithi192.stdout:6/110: symlink d3/d26/l29 0 2024-04-12T05:08:52.155 INFO:tasks.workunit.client.1.smithi192.stdout:0/36: dread d0/fb [4194304,4194304] 0 2024-04-12T05:08:52.156 INFO:tasks.workunit.client.0.smithi113.stdout:5/215: mkdir d0/d10/d27/d1d/d38 0 2024-04-12T05:08:52.156 INFO:tasks.workunit.client.0.smithi113.stdout:0/209: chown d1/d9/l23 3602 1 2024-04-12T05:08:52.156 INFO:tasks.workunit.client.1.smithi192.stdout:6/111: dread - d3/d26/f28 zero size 2024-04-12T05:08:52.157 INFO:tasks.workunit.client.0.smithi113.stdout:7/199: rmdir d0 39 2024-04-12T05:08:52.157 INFO:tasks.workunit.client.1.smithi192.stdout:5/140: dread d0/d21/f26 [0,4194304] 0 2024-04-12T05:08:52.158 INFO:tasks.workunit.client.1.smithi192.stdout:2/82: dread d1/f4 [0,4194304] 0 2024-04-12T05:08:52.159 INFO:tasks.workunit.client.0.smithi113.stdout:9/285: mknod d1/d4/d30/d37/d35/d3d/d52/c61 0 2024-04-12T05:08:52.162 INFO:tasks.workunit.client.1.smithi192.stdout:9/67: mknod d11/d15/c18 0 2024-04-12T05:08:52.166 INFO:tasks.workunit.client.1.smithi192.stdout:2/83: fdatasync d1/d8/f10 0 2024-04-12T05:08:52.167 INFO:tasks.workunit.client.0.smithi113.stdout:9/286: creat d1/d4/d38/f62 x:0 0 0 2024-04-12T05:08:52.167 INFO:tasks.workunit.client.1.smithi192.stdout:1/44: getdents d0/de 0 2024-04-12T05:08:52.169 INFO:tasks.workunit.client.1.smithi192.stdout:9/68: symlink d11/d15/l19 0 2024-04-12T05:08:52.171 INFO:tasks.workunit.client.1.smithi192.stdout:5/141: symlink d0/d15/l37 0 2024-04-12T05:08:52.172 INFO:tasks.workunit.client.0.smithi113.stdout:4/245: dwrite da/dd/d1b/f2c [0,4194304] 0 2024-04-12T05:08:52.172 INFO:tasks.workunit.client.1.smithi192.stdout:0/37: dwrite d0/fb [0,4194304] 0 2024-04-12T05:08:52.173 INFO:tasks.workunit.client.0.smithi113.stdout:7/200: read d0/dc/d6/f16 [734907,56485] 0 2024-04-12T05:08:52.173 INFO:tasks.workunit.client.0.smithi113.stdout:5/216: dread d0/d10/d27/f18 [0,4194304] 0 2024-04-12T05:08:52.174 INFO:tasks.workunit.client.0.smithi113.stdout:1/157: dread fe [0,4194304] 0 2024-04-12T05:08:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:50.669048+0000 mgr.smithi192.ocppyo (mgr.24357) 21 : cluster [DBG] pgmap v15: 65 pgs: 65 active+clean; 356 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 93 KiB/s rd, 10 MiB/s wr, 114 op/s 2024-04-12T05:08:52.174 INFO:tasks.workunit.client.1.smithi192.stdout:2/84: symlink d1/d8/d17/l1a 0 2024-04-12T05:08:52.175 INFO:tasks.workunit.client.0.smithi113.stdout:0/210: getdents d1/d6/d16 0 2024-04-12T05:08:52.176 INFO:tasks.workunit.client.1.smithi192.stdout:7/66: write d3/fd [1398388,15130] 0 2024-04-12T05:08:52.176 INFO:tasks.workunit.client.1.smithi192.stdout:9/69: dread - fd zero size 2024-04-12T05:08:52.177 INFO:tasks.workunit.client.1.smithi192.stdout:3/56: truncate d5/fd 2346370 0 2024-04-12T05:08:52.178 INFO:tasks.workunit.client.1.smithi192.stdout:5/142: stat d0/l28 0 2024-04-12T05:08:52.180 INFO:tasks.workunit.client.0.smithi113.stdout:0/211: unlink d1/d6/c18 0 2024-04-12T05:08:52.180 INFO:tasks.workunit.client.0.smithi113.stdout:3/186: dwrite d0/f29 [0,4194304] 0 2024-04-12T05:08:52.181 INFO:tasks.workunit.client.1.smithi192.stdout:0/38: rename d0/c2 to d0/cc 0 2024-04-12T05:08:52.181 INFO:tasks.workunit.client.1.smithi192.stdout:4/71: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:52.181 INFO:tasks.workunit.client.1.smithi192.stdout:6/112: getdents d3 0 2024-04-12T05:08:52.182 INFO:tasks.workunit.client.1.smithi192.stdout:5/143: chown d0/f7 1 1 2024-04-12T05:08:52.182 INFO:tasks.workunit.client.1.smithi192.stdout:1/45: getdents d0/de 0 2024-04-12T05:08:52.183 INFO:tasks.workunit.client.0.smithi113.stdout:6/211: dread d8/de/f16 [0,4194304] 0 2024-04-12T05:08:52.185 INFO:tasks.workunit.client.1.smithi192.stdout:1/46: write d0/d2/f9 [1532358,113840] 0 2024-04-12T05:08:52.186 INFO:tasks.workunit.client.1.smithi192.stdout:2/85: symlink d1/d8/d17/l1b 0 2024-04-12T05:08:52.187 INFO:tasks.workunit.client.0.smithi113.stdout:7/201: dread d0/dc/d6/f28 [0,4194304] 0 2024-04-12T05:08:52.188 INFO:tasks.workunit.client.0.smithi113.stdout:9/287: rename d1/d4/d30/d37/d35 to d1/df/d4b/d63 0 2024-04-12T05:08:52.189 INFO:tasks.workunit.client.0.smithi113.stdout:7/202: fdatasync d0/dd/f1c 0 2024-04-12T05:08:52.191 INFO:tasks.workunit.client.0.smithi113.stdout:5/217: mkdir d0/d10/d27/d13/d39 0 2024-04-12T05:08:52.191 INFO:tasks.workunit.client.1.smithi192.stdout:7/67: mkdir d3/d7/d15 0 2024-04-12T05:08:52.191 INFO:tasks.workunit.client.1.smithi192.stdout:8/30: sync 2024-04-12T05:08:52.192 INFO:tasks.workunit.client.1.smithi192.stdout:0/39: dread d0/f8 [0,4194304] 0 2024-04-12T05:08:52.193 INFO:tasks.workunit.client.0.smithi113.stdout:1/158: write fe [174252,52928] 0 2024-04-12T05:08:52.195 INFO:tasks.workunit.client.0.smithi113.stdout:0/212: creat d1/f3d x:0 0 0 2024-04-12T05:08:52.195 INFO:tasks.workunit.client.1.smithi192.stdout:4/72: mknod d1/de/c1a 0 2024-04-12T05:08:52.196 INFO:tasks.workunit.client.1.smithi192.stdout:8/31: dread f5 [0,4194304] 0 2024-04-12T05:08:52.196 INFO:tasks.workunit.client.1.smithi192.stdout:4/73: dread - d1/f15 zero size 2024-04-12T05:08:52.196 INFO:tasks.workunit.client.0.smithi113.stdout:5/218: write d0/f5 [3113538,32952] 0 2024-04-12T05:08:52.198 INFO:tasks.workunit.client.1.smithi192.stdout:8/32: write f1 [2765695,104335] 0 2024-04-12T05:08:52.199 INFO:tasks.workunit.client.0.smithi113.stdout:5/219: readlink d0/d10/l2d 0 2024-04-12T05:08:52.201 INFO:tasks.workunit.client.0.smithi113.stdout:1/159: creat d10/d14/f2a x:0 0 0 2024-04-12T05:08:52.201 INFO:tasks.workunit.client.0.smithi113.stdout:3/187: dwrite d0/d13/d15/f3e [0,4194304] 0 2024-04-12T05:08:52.202 INFO:tasks.workunit.client.0.smithi113.stdout:1/160: chown d10 7402868 1 2024-04-12T05:08:52.202 INFO:tasks.workunit.client.0.smithi113.stdout:5/220: dread - d0/f31 zero size 2024-04-12T05:08:52.203 INFO:tasks.workunit.client.1.smithi192.stdout:3/57: creat d5/d13/f16 x:0 0 0 2024-04-12T05:08:52.205 INFO:tasks.workunit.client.0.smithi113.stdout:9/288: getdents d1/d4/d38 0 2024-04-12T05:08:52.209 INFO:tasks.workunit.client.1.smithi192.stdout:7/68: unlink d3/cc 0 2024-04-12T05:08:52.210 INFO:tasks.workunit.client.0.smithi113.stdout:4/246: dwrite da/dd/df/d14/f18 [0,4194304] 0 2024-04-12T05:08:52.213 INFO:tasks.workunit.client.1.smithi192.stdout:0/40: mknod d0/d6/cd 0 2024-04-12T05:08:52.214 INFO:tasks.workunit.client.0.smithi113.stdout:3/188: dread - d0/d13/d15/f20 zero size 2024-04-12T05:08:52.216 INFO:tasks.workunit.client.1.smithi192.stdout:4/74: mkdir d1/de/d1b 0 2024-04-12T05:08:52.216 INFO:tasks.workunit.client.0.smithi113.stdout:0/213: rename d1/d6/l1f to d1/d6/l3e 0 2024-04-12T05:08:52.217 INFO:tasks.workunit.client.0.smithi113.stdout:9/289: mknod d1/c64 0 2024-04-12T05:08:52.218 INFO:tasks.workunit.client.0.smithi113.stdout:1/161: dwrite ff [0,4194304] 0 2024-04-12T05:08:52.220 INFO:tasks.workunit.client.0.smithi113.stdout:4/247: creat da/dd/df/d1d/f4c x:0 0 0 2024-04-12T05:08:52.221 INFO:tasks.workunit.client.1.smithi192.stdout:6/113: creat d3/d21/d24/f2a x:0 0 0 2024-04-12T05:08:52.223 INFO:tasks.workunit.client.0.smithi113.stdout:0/214: creat d1/d6/d16/d2a/f3f x:0 0 0 2024-04-12T05:08:52.224 INFO:tasks.workunit.client.1.smithi192.stdout:3/58: creat d5/d13/f17 x:0 0 0 2024-04-12T05:08:52.224 INFO:tasks.workunit.client.1.smithi192.stdout:3/59: chown d5 806581 1 2024-04-12T05:08:52.224 INFO:tasks.workunit.client.0.smithi113.stdout:1/162: creat d10/d21/f2b x:0 0 0 2024-04-12T05:08:52.226 INFO:tasks.workunit.client.1.smithi192.stdout:3/60: truncate d5/d13/f17 285600 0 2024-04-12T05:08:52.226 INFO:tasks.workunit.client.0.smithi113.stdout:0/215: read - d1/f2f zero size 2024-04-12T05:08:52.226 INFO:tasks.workunit.client.1.smithi192.stdout:2/86: unlink d1/d8/db/l16 0 2024-04-12T05:08:52.227 INFO:tasks.workunit.client.0.smithi113.stdout:1/163: readlink d10/d14/l1b 0 2024-04-12T05:08:52.228 INFO:tasks.workunit.client.0.smithi113.stdout:5/221: dwrite d0/d10/f1b [0,4194304] 0 2024-04-12T05:08:52.229 INFO:tasks.workunit.client.0.smithi113.stdout:1/164: dread - d10/d14/f17 zero size 2024-04-12T05:08:52.230 INFO:tasks.workunit.client.0.smithi113.stdout:0/216: read d1/d9/f1a [114100,89073] 0 2024-04-12T05:08:52.230 INFO:tasks.workunit.client.0.smithi113.stdout:3/189: creat d0/d13/d15/d28/f41 x:0 0 0 2024-04-12T05:08:52.230 INFO:tasks.workunit.client.0.smithi113.stdout:4/248: rename da/d3c/d43/f49 to da/d3c/d43/f4d 0 2024-04-12T05:08:52.231 INFO:tasks.workunit.client.0.smithi113.stdout:5/222: chown d0/d10/d27/d13/l1a 1 1 2024-04-12T05:08:52.231 INFO:tasks.workunit.client.1.smithi192.stdout:0/41: symlink d0/le 0 2024-04-12T05:08:52.232 INFO:tasks.workunit.client.1.smithi192.stdout:4/75: mkdir d1/da/d1c 0 2024-04-12T05:08:52.233 INFO:tasks.workunit.client.1.smithi192.stdout:0/42: chown d0/d6 748296 1 2024-04-12T05:08:52.234 INFO:tasks.workunit.client.0.smithi113.stdout:9/290: rename d1/d4/d30/d37 to d1/d4/d38/d65 0 2024-04-12T05:08:52.235 INFO:tasks.workunit.client.1.smithi192.stdout:6/114: rename d3/ce to d3/d26/c2b 0 2024-04-12T05:08:52.236 INFO:tasks.workunit.client.0.smithi113.stdout:4/249: truncate da/dd/df/d1d/f35 1596640 0 2024-04-12T05:08:52.236 INFO:tasks.workunit.client.1.smithi192.stdout:2/87: mknod d1/c1c 0 2024-04-12T05:08:52.238 INFO:tasks.workunit.client.1.smithi192.stdout:0/43: write d0/f8 [1807979,16769] 0 2024-04-12T05:08:52.242 INFO:tasks.workunit.client.1.smithi192.stdout:5/144: dwrite d0/f13 [0,4194304] 0 2024-04-12T05:08:52.242 INFO:tasks.workunit.client.1.smithi192.stdout:3/61: creat d5/d13/f18 x:0 0 0 2024-04-12T05:08:52.243 INFO:tasks.workunit.client.0.smithi113.stdout:5/223: creat d0/f3a x:0 0 0 2024-04-12T05:08:52.243 INFO:tasks.workunit.client.1.smithi192.stdout:6/115: dread - d3/d21/d24/f2a zero size 2024-04-12T05:08:52.243 INFO:tasks.workunit.client.1.smithi192.stdout:7/69: creat d3/d11/f16 x:0 0 0 2024-04-12T05:08:52.244 INFO:tasks.workunit.client.1.smithi192.stdout:3/62: chown d5/d13/f16 0 1 2024-04-12T05:08:52.244 INFO:tasks.workunit.client.0.smithi113.stdout:1/165: dread fd [0,4194304] 0 2024-04-12T05:08:52.245 INFO:tasks.workunit.client.1.smithi192.stdout:0/44: write d0/f8 [867586,76369] 0 2024-04-12T05:08:52.245 INFO:tasks.workunit.client.1.smithi192.stdout:6/116: stat d3/d26/f28 0 2024-04-12T05:08:52.245 INFO:tasks.workunit.client.1.smithi192.stdout:9/70: getdents d11/d15 0 2024-04-12T05:08:52.246 INFO:tasks.workunit.client.0.smithi113.stdout:0/217: dwrite d1/fe [0,4194304] 0 2024-04-12T05:08:52.246 INFO:tasks.workunit.client.0.smithi113.stdout:9/291: getdents d1/d4/d5/d6/d15 0 2024-04-12T05:08:52.246 INFO:tasks.workunit.client.0.smithi113.stdout:5/224: dread d0/d10/d27/f22 [0,4194304] 0 2024-04-12T05:08:52.248 INFO:tasks.workunit.client.1.smithi192.stdout:7/70: mknod d3/d11/c17 0 2024-04-12T05:08:52.249 INFO:tasks.workunit.client.1.smithi192.stdout:3/63: creat d5/f19 x:0 0 0 2024-04-12T05:08:52.250 INFO:tasks.workunit.client.0.smithi113.stdout:4/250: dwrite da/dd/d3a/f44 [0,4194304] 0 2024-04-12T05:08:52.250 INFO:tasks.workunit.client.0.smithi113.stdout:0/218: read d1/d6/d16/f1c [1732781,19864] 0 2024-04-12T05:08:52.250 INFO:tasks.workunit.client.1.smithi192.stdout:1/47: dwrite d0/d2/f7 [0,4194304] 0 2024-04-12T05:08:52.250 INFO:tasks.workunit.client.1.smithi192.stdout:5/145: chown d0/d15/c19 15 1 2024-04-12T05:08:52.257 INFO:tasks.workunit.client.1.smithi192.stdout:2/88: mknod d1/d8/c1d 0 2024-04-12T05:08:52.257 INFO:tasks.workunit.client.0.smithi113.stdout:1/166: creat d10/d21/d23/d28/f2c x:0 0 0 2024-04-12T05:08:52.257 INFO:tasks.workunit.client.0.smithi113.stdout:9/292: mknod d1/df/d4b/d63/c66 0 2024-04-12T05:08:52.259 INFO:tasks.workunit.client.1.smithi192.stdout:9/71: truncate f3 1798794 0 2024-04-12T05:08:52.259 INFO:tasks.workunit.client.1.smithi192.stdout:0/45: chown d0/cc 61 1 2024-04-12T05:08:52.259 INFO:tasks.workunit.client.1.smithi192.stdout:4/76: dread d1/f11 [0,4194304] 0 2024-04-12T05:08:52.259 INFO:tasks.workunit.client.0.smithi113.stdout:5/225: write d0/d10/d27/f22 [1013751,110446] 0 2024-04-12T05:08:52.259 INFO:tasks.workunit.client.0.smithi113.stdout:9/293: dread - d1/df/d4b/d63/d3d/f5f zero size 2024-04-12T05:08:52.259 INFO:tasks.workunit.client.0.smithi113.stdout:1/167: creat d10/d21/d23/d28/f2d x:0 0 0 2024-04-12T05:08:52.259 INFO:tasks.workunit.client.0.smithi113.stdout:9/294: write d1/d4/d38/f62 [546082,77566] 0 2024-04-12T05:08:52.264 INFO:tasks.workunit.client.0.smithi113.stdout:1/168: write d10/d21/d23/d28/f2d [657724,21760] 0 2024-04-12T05:08:52.265 INFO:tasks.workunit.client.0.smithi113.stdout:4/251: dwrite da/dd/df/f10 [4194304,4194304] 0 2024-04-12T05:08:52.265 INFO:tasks.workunit.client.0.smithi113.stdout:0/219: rmdir d1/d6/d16/d3b 0 2024-04-12T05:08:52.268 INFO:tasks.workunit.client.1.smithi192.stdout:6/117: link d3/d5/ff d3/f2c 0 2024-04-12T05:08:52.268 INFO:tasks.workunit.client.1.smithi192.stdout:8/33: dwrite f3 [0,4194304] 0 2024-04-12T05:08:52.268 INFO:tasks.workunit.client.1.smithi192.stdout:9/72: chown d11/d14/l16 15 1 2024-04-12T05:08:52.268 INFO:tasks.workunit.client.0.smithi113.stdout:5/226: mkdir d0/d10/d27/d1d/d38/d3b 0 2024-04-12T05:08:52.268 INFO:tasks.workunit.client.0.smithi113.stdout:9/295: chown d1/d4/d5/d6/d54 116588 1 2024-04-12T05:08:52.269 INFO:tasks.workunit.client.1.smithi192.stdout:8/34: readlink l4 0 2024-04-12T05:08:52.270 INFO:tasks.workunit.client.1.smithi192.stdout:8/35: readlink l2 0 2024-04-12T05:08:52.270 INFO:tasks.workunit.client.1.smithi192.stdout:0/46: stat d0/cc 0 2024-04-12T05:08:52.272 INFO:tasks.workunit.client.0.smithi113.stdout:4/252: write da/dd/df/d14/f18 [1486663,36570] 0 2024-04-12T05:08:52.275 INFO:tasks.workunit.client.1.smithi192.stdout:2/89: creat d1/d8/db/de/f1e x:0 0 0 2024-04-12T05:08:52.276 INFO:tasks.workunit.client.0.smithi113.stdout:4/253: chown da/dd/df/d14 25 1 2024-04-12T05:08:52.279 INFO:tasks.workunit.client.0.smithi113.stdout:5/227: creat d0/d10/d27/d33/f3c x:0 0 0 2024-04-12T05:08:52.280 INFO:tasks.workunit.client.1.smithi192.stdout:4/77: chown d1/f15 185903 1 2024-04-12T05:08:52.284 INFO:tasks.workunit.client.1.smithi192.stdout:7/71: dwrite d3/f12 [0,4194304] 0 2024-04-12T05:08:52.285 INFO:tasks.workunit.client.0.smithi113.stdout:4/254: rename da/dd/df/d1d/f35 to da/de/d17/f4e 0 2024-04-12T05:08:52.288 INFO:tasks.workunit.client.1.smithi192.stdout:6/118: mknod d3/d8/c2d 0 2024-04-12T05:08:52.289 INFO:tasks.workunit.client.0.smithi113.stdout:1/169: dwrite ff [0,4194304] 0 2024-04-12T05:08:52.291 INFO:tasks.workunit.client.1.smithi192.stdout:1/48: dwrite d0/d2/f7 [0,4194304] 0 2024-04-12T05:08:52.292 INFO:tasks.workunit.client.1.smithi192.stdout:9/73: creat d11/d14/f1a x:0 0 0 2024-04-12T05:08:52.292 INFO:tasks.workunit.client.0.smithi113.stdout:0/220: dwrite d1/d6/d16/d2a/f3f [0,4194304] 0 2024-04-12T05:08:52.292 INFO:tasks.workunit.client.1.smithi192.stdout:2/90: fsync d1/f4 0 2024-04-12T05:08:52.294 INFO:tasks.workunit.client.0.smithi113.stdout:5/228: mknod d0/d10/d27/d1d/d38/d3b/c3d 0 2024-04-12T05:08:52.295 INFO:tasks.workunit.client.1.smithi192.stdout:2/91: readlink d1/d8/db/l14 0 2024-04-12T05:08:52.295 INFO:tasks.workunit.client.0.smithi113.stdout:9/296: dwrite d1/df/d4b/d63/d3d/f49 [0,4194304] 0 2024-04-12T05:08:52.295 INFO:tasks.workunit.client.0.smithi113.stdout:0/221: readlink d1/d9/l23 0 2024-04-12T05:08:52.295 INFO:tasks.workunit.client.0.smithi113.stdout:5/229: fdatasync d0/f2e 0 2024-04-12T05:08:52.295 INFO:tasks.workunit.client.1.smithi192.stdout:8/36: rename f1 to d6/f7 0 2024-04-12T05:08:52.301 INFO:tasks.workunit.client.1.smithi192.stdout:7/72: creat d3/d11/f18 x:0 0 0 2024-04-12T05:08:52.301 INFO:tasks.workunit.client.0.smithi113.stdout:5/230: unlink d0/d10/d27/l34 0 2024-04-12T05:08:52.304 INFO:tasks.workunit.client.0.smithi113.stdout:0/222: creat d1/d6/d34/f40 x:0 0 0 2024-04-12T05:08:52.305 INFO:tasks.workunit.client.1.smithi192.stdout:6/119: chown d3/d26/l29 0 1 2024-04-12T05:08:52.305 INFO:tasks.workunit.client.1.smithi192.stdout:2/92: chown d1/d8/ca 0 1 2024-04-12T05:08:52.307 INFO:tasks.workunit.client.0.smithi113.stdout:9/297: link d1/d4/d5/c5c d1/d4/d38/c67 0 2024-04-12T05:08:52.307 INFO:tasks.workunit.client.1.smithi192.stdout:8/37: mknod d6/c8 0 2024-04-12T05:08:52.311 INFO:tasks.workunit.client.1.smithi192.stdout:7/73: write d3/f12 [1816834,6769] 0 2024-04-12T05:08:52.311 INFO:tasks.workunit.client.1.smithi192.stdout:8/38: read f3 [1158074,101195] 0 2024-04-12T05:08:52.312 INFO:tasks.workunit.client.0.smithi113.stdout:5/231: dwrite d0/d10/d27/f26 [0,4194304] 0 2024-04-12T05:08:52.313 INFO:tasks.workunit.client.1.smithi192.stdout:3/64: dwrite f2 [0,4194304] 0 2024-04-12T05:08:52.315 INFO:tasks.workunit.client.0.smithi113.stdout:5/232: chown d0/d10/d27/d1d/f21 658 1 2024-04-12T05:08:52.319 INFO:tasks.workunit.client.1.smithi192.stdout:2/93: read - d1/f13 zero size 2024-04-12T05:08:52.320 INFO:tasks.workunit.client.1.smithi192.stdout:1/49: dread d0/f3 [0,4194304] 0 2024-04-12T05:08:52.320 INFO:tasks.workunit.client.1.smithi192.stdout:0/47: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:08:52.321 INFO:tasks.workunit.client.1.smithi192.stdout:3/65: mkdir d5/d13/d1a 0 2024-04-12T05:08:52.323 INFO:tasks.workunit.client.1.smithi192.stdout:7/74: unlink d3/d11/c17 0 2024-04-12T05:08:52.324 INFO:tasks.workunit.client.0.smithi113.stdout:5/233: dwrite d0/fc [0,4194304] 0 2024-04-12T05:08:52.325 INFO:tasks.workunit.client.1.smithi192.stdout:7/75: chown d3/f6 509270779 1 2024-04-12T05:08:52.328 INFO:tasks.workunit.client.1.smithi192.stdout:9/74: link d11/d15/l19 d11/d14/l1b 0 2024-04-12T05:08:52.331 INFO:tasks.workunit.client.1.smithi192.stdout:7/76: fdatasync d3/d7/f9 0 2024-04-12T05:08:52.331 INFO:tasks.workunit.client.1.smithi192.stdout:0/48: write d0/f8 [2009550,130617] 0 2024-04-12T05:08:52.332 INFO:tasks.workunit.client.1.smithi192.stdout:2/94: mknod d1/d8/c1f 0 2024-04-12T05:08:52.349 INFO:tasks.workunit.client.1.smithi192.stdout:7/77: stat d3/ff 0 2024-04-12T05:08:52.349 INFO:tasks.workunit.client.1.smithi192.stdout:6/120: getdents d3/d5 0 2024-04-12T05:08:52.349 INFO:tasks.workunit.client.1.smithi192.stdout:5/146: dwrite d0/f1e [4194304,4194304] 0 2024-04-12T05:08:52.349 INFO:tasks.workunit.client.1.smithi192.stdout:3/66: unlink d5/c7 0 2024-04-12T05:08:52.349 INFO:tasks.workunit.client.1.smithi192.stdout:2/95: fdatasync d1/d8/f10 0 2024-04-12T05:08:52.349 INFO:tasks.workunit.client.1.smithi192.stdout:3/67: stat l1 0 2024-04-12T05:08:52.350 INFO:tasks.workunit.client.0.smithi113.stdout:1/170: truncate fe 2459756 0 2024-04-12T05:08:52.350 INFO:tasks.workunit.client.0.smithi113.stdout:5/234: link d0/d10/l24 d0/l3e 0 2024-04-12T05:08:52.350 INFO:tasks.workunit.client.0.smithi113.stdout:1/171: chown d10/d14/c19 27492 1 2024-04-12T05:08:52.350 INFO:tasks.workunit.client.0.smithi113.stdout:5/235: rename d0/d10/d27/d13/d39 to d0/d10/d27/d1d/d3f 0 2024-04-12T05:08:52.350 INFO:tasks.workunit.client.0.smithi113.stdout:1/172: dwrite d10/d21/f2b [0,4194304] 0 2024-04-12T05:08:52.351 INFO:tasks.workunit.client.1.smithi192.stdout:5/147: creat d0/d15/d32/f38 x:0 0 0 2024-04-12T05:08:52.351 INFO:tasks.workunit.client.1.smithi192.stdout:0/49: dread d0/fb [0,4194304] 0 2024-04-12T05:08:52.352 INFO:tasks.workunit.client.1.smithi192.stdout:6/121: getdents d3/d23 0 2024-04-12T05:08:52.353 INFO:tasks.workunit.client.1.smithi192.stdout:1/50: dwrite d0/f3 [0,4194304] 0 2024-04-12T05:08:52.354 INFO:tasks.workunit.client.0.smithi113.stdout:5/236: creat d0/d10/d27/d13/f40 x:0 0 0 2024-04-12T05:08:52.354 INFO:tasks.workunit.client.1.smithi192.stdout:6/122: chown d3/d26/l29 45 1 2024-04-12T05:08:52.355 INFO:tasks.workunit.client.0.smithi113.stdout:5/237: fdatasync d0/d10/d27/d33/f3c 0 2024-04-12T05:08:52.356 INFO:tasks.workunit.client.1.smithi192.stdout:8/39: rename d6/f7 to d6/f9 0 2024-04-12T05:08:52.356 INFO:tasks.workunit.client.0.smithi113.stdout:1/173: mkdir d10/d2e 0 2024-04-12T05:08:52.360 INFO:tasks.workunit.client.0.smithi113.stdout:5/238: mkdir d0/d10/d27/d1d/d41 0 2024-04-12T05:08:52.361 INFO:tasks.workunit.client.0.smithi113.stdout:1/174: mkdir d10/d2f 0 2024-04-12T05:08:52.363 INFO:tasks.workunit.client.1.smithi192.stdout:2/96: creat d1/d8/d17/f20 x:0 0 0 2024-04-12T05:08:52.363 INFO:tasks.workunit.client.1.smithi192.stdout:7/78: link d3/d11/f18 d3/d7/f19 0 2024-04-12T05:08:52.363 INFO:tasks.workunit.client.1.smithi192.stdout:1/51: mknod d0/d2/dd/c12 0 2024-04-12T05:08:52.365 INFO:tasks.workunit.client.1.smithi192.stdout:6/123: creat d3/d8/d22/f2e x:0 0 0 2024-04-12T05:08:52.366 INFO:tasks.workunit.client.0.smithi113.stdout:0/223: truncate d1/d6/d16/d2a/f3f 338602 0 2024-04-12T05:08:52.368 INFO:tasks.workunit.client.0.smithi113.stdout:5/239: rename d0/d10/d27/d1d/l29 to d0/d10/d27/d1d/d38/l42 0 2024-04-12T05:08:52.370 INFO:tasks.workunit.client.0.smithi113.stdout:1/175: creat d10/d21/d23/f30 x:0 0 0 2024-04-12T05:08:52.371 INFO:tasks.workunit.client.1.smithi192.stdout:8/40: write d6/f9 [4866603,106251] 0 2024-04-12T05:08:52.371 INFO:tasks.workunit.client.1.smithi192.stdout:7/79: chown d3/d11/f16 16995931 1 2024-04-12T05:08:52.371 INFO:tasks.workunit.client.1.smithi192.stdout:9/75: write f6 [246098,91433] 0 2024-04-12T05:08:52.375 INFO:tasks.workunit.client.1.smithi192.stdout:5/148: dread d0/f10 [0,4194304] 0 2024-04-12T05:08:52.376 INFO:tasks.workunit.client.0.smithi113.stdout:1/176: dwrite d10/d21/f2b [0,4194304] 0 2024-04-12T05:08:52.377 INFO:tasks.workunit.client.1.smithi192.stdout:8/41: mkdir d6/da 0 2024-04-12T05:08:52.378 INFO:tasks.workunit.client.1.smithi192.stdout:8/42: chown d6 1542346 1 2024-04-12T05:08:52.378 INFO:tasks.workunit.client.1.smithi192.stdout:1/52: unlink d0/d2/dd/l11 0 2024-04-12T05:08:52.378 INFO:tasks.workunit.client.1.smithi192.stdout:1/53: fdatasync d0/fc 0 2024-04-12T05:08:52.381 INFO:tasks.workunit.client.1.smithi192.stdout:8/43: dread f3 [0,4194304] 0 2024-04-12T05:08:52.381 INFO:tasks.workunit.client.0.smithi113.stdout:0/224: dwrite d1/f22 [0,4194304] 0 2024-04-12T05:08:52.382 INFO:tasks.workunit.client.1.smithi192.stdout:7/80: mkdir d3/d7/d15/d1a 0 2024-04-12T05:08:52.386 INFO:tasks.workunit.client.1.smithi192.stdout:8/44: rename l2 to d6/da/lb 0 2024-04-12T05:08:52.388 INFO:tasks.workunit.client.1.smithi192.stdout:6/124: getdents d3/d21 0 2024-04-12T05:08:52.389 INFO:tasks.workunit.client.0.smithi113.stdout:1/177: getdents d10 0 2024-04-12T05:08:52.390 INFO:tasks.workunit.client.1.smithi192.stdout:7/81: mkdir d3/d7/d15/d1a/d1b 0 2024-04-12T05:08:52.391 INFO:tasks.workunit.client.0.smithi113.stdout:0/225: dread d1/d6/d16/d2a/f3f [0,4194304] 0 2024-04-12T05:08:52.391 INFO:tasks.workunit.client.0.smithi113.stdout:0/226: readlink d1/d9/l23 0 2024-04-12T05:08:52.392 INFO:tasks.workunit.client.1.smithi192.stdout:8/45: creat d6/da/fc x:0 0 0 2024-04-12T05:08:52.392 INFO:tasks.workunit.client.1.smithi192.stdout:5/149: rmdir d0/d15/d32 39 2024-04-12T05:08:52.393 INFO:tasks.workunit.client.1.smithi192.stdout:7/82: rename d3/d7 to d3/d7/d15/d1a/d1c 22 2024-04-12T05:08:52.393 INFO:tasks.workunit.client.0.smithi113.stdout:1/178: mkdir d10/d2e/d31 0 2024-04-12T05:08:52.393 INFO:tasks.workunit.client.0.smithi113.stdout:1/179: fdatasync d10/d21/f29 0 2024-04-12T05:08:52.393 INFO:tasks.workunit.client.0.smithi113.stdout:0/227: symlink d1/d6/da/l41 0 2024-04-12T05:08:52.396 INFO:tasks.workunit.client.1.smithi192.stdout:2/97: getdents d1/d8/d17 0 2024-04-12T05:08:52.397 INFO:tasks.workunit.client.1.smithi192.stdout:8/46: write d6/f9 [4169276,68455] 0 2024-04-12T05:08:52.397 INFO:tasks.workunit.client.0.smithi113.stdout:1/180: rename d10/d21/d23/f30 to d10/f32 0 2024-04-12T05:08:52.399 INFO:tasks.workunit.client.1.smithi192.stdout:5/150: symlink d0/l39 0 2024-04-12T05:08:52.400 INFO:tasks.workunit.client.1.smithi192.stdout:7/83: mkdir d3/d1d 0 2024-04-12T05:08:52.402 INFO:tasks.workunit.client.1.smithi192.stdout:2/98: creat d1/d8/db/f21 x:0 0 0 2024-04-12T05:08:52.403 INFO:tasks.workunit.client.1.smithi192.stdout:6/125: dread d3/d8/f1c [0,4194304] 0 2024-04-12T05:08:52.404 INFO:tasks.workunit.client.0.smithi113.stdout:1/181: rename d10/d21/d23/d28 to d10/d2e/d31/d33 0 2024-04-12T05:08:52.404 INFO:tasks.workunit.client.0.smithi113.stdout:0/228: getdents d1/d9 0 2024-04-12T05:08:52.405 INFO:tasks.workunit.client.1.smithi192.stdout:5/151: mknod d0/d15/d24/c3a 0 2024-04-12T05:08:52.405 INFO:tasks.workunit.client.1.smithi192.stdout:6/126: read d3/d8/f1c [1995109,90809] 0 2024-04-12T05:08:52.406 INFO:tasks.workunit.client.1.smithi192.stdout:7/84: dread f1 [0,4194304] 0 2024-04-12T05:08:52.406 INFO:tasks.workunit.client.1.smithi192.stdout:1/54: truncate d0/d2/fa 2120706 0 2024-04-12T05:08:52.407 INFO:tasks.workunit.client.0.smithi113.stdout:1/182: creat d10/f34 x:0 0 0 2024-04-12T05:08:52.408 INFO:tasks.workunit.client.1.smithi192.stdout:6/127: chown d3/d8/c2d 11071261 1 2024-04-12T05:08:52.408 INFO:tasks.workunit.client.0.smithi113.stdout:0/229: unlink d1/d9/f27 0 2024-04-12T05:08:52.409 INFO:tasks.workunit.client.1.smithi192.stdout:5/152: creat d0/d15/d24/d29/f3b x:0 0 0 2024-04-12T05:08:52.411 INFO:tasks.workunit.client.1.smithi192.stdout:1/55: creat d0/de/f13 x:0 0 0 2024-04-12T05:08:52.412 INFO:tasks.workunit.client.0.smithi113.stdout:5/240: dread d0/f5 [0,4194304] 0 2024-04-12T05:08:52.412 INFO:tasks.workunit.client.1.smithi192.stdout:8/47: rmdir d6/da 39 2024-04-12T05:08:52.413 INFO:tasks.workunit.client.1.smithi192.stdout:1/56: fsync d0/fc 0 2024-04-12T05:08:52.415 INFO:tasks.workunit.client.0.smithi113.stdout:5/241: write d0/d10/d27/d33/f35 [429810,74991] 0 2024-04-12T05:08:52.415 INFO:tasks.workunit.client.1.smithi192.stdout:2/99: dwrite d1/d8/db/f21 [0,4194304] 0 2024-04-12T05:08:52.416 INFO:tasks.workunit.client.1.smithi192.stdout:5/153: symlink d0/d15/d24/l3c 0 2024-04-12T05:08:52.417 INFO:tasks.workunit.client.0.smithi113.stdout:0/230: dwrite d1/d6/f1d [0,4194304] 0 2024-04-12T05:08:52.418 INFO:tasks.workunit.client.1.smithi192.stdout:6/128: fsync d3/d8/f25 0 2024-04-12T05:08:52.419 INFO:tasks.workunit.client.0.smithi113.stdout:5/242: write d0/d10/d27/d33/f35 [1109920,106120] 0 2024-04-12T05:08:52.420 INFO:tasks.workunit.client.0.smithi113.stdout:1/183: write fe [252278,40910] 0 2024-04-12T05:08:52.420 INFO:tasks.workunit.client.1.smithi192.stdout:7/85: symlink d3/d7/d15/d1a/d1b/l1e 0 2024-04-12T05:08:52.421 INFO:tasks.workunit.client.0.smithi113.stdout:1/184: chown d10/d14 762 1 2024-04-12T05:08:52.422 INFO:tasks.workunit.client.1.smithi192.stdout:6/129: chown d3/l14 143 1 2024-04-12T05:08:52.422 INFO:tasks.workunit.client.0.smithi113.stdout:5/243: mknod d0/d10/d27/d1d/c43 0 2024-04-12T05:08:52.424 INFO:tasks.workunit.client.0.smithi113.stdout:5/244: stat d0/d10/d27/d1d/d38/d3b 0 2024-04-12T05:08:52.426 INFO:tasks.workunit.client.0.smithi113.stdout:5/245: fdatasync d0/d10/d27/f22 0 2024-04-12T05:08:52.427 INFO:tasks.workunit.client.1.smithi192.stdout:6/130: read d3/d5/f6 [480467,119274] 0 2024-04-12T05:08:52.429 INFO:tasks.workunit.client.0.smithi113.stdout:1/185: creat d10/d2e/d31/f35 x:0 0 0 2024-04-12T05:08:52.429 INFO:tasks.workunit.client.0.smithi113.stdout:0/231: getdents d1/d6/d16 0 2024-04-12T05:08:52.429 INFO:tasks.workunit.client.1.smithi192.stdout:6/131: dread - d3/f1e zero size 2024-04-12T05:08:52.429 INFO:tasks.workunit.client.1.smithi192.stdout:8/48: dread f3 [0,4194304] 0 2024-04-12T05:08:52.431 INFO:tasks.workunit.client.0.smithi113.stdout:5/246: mkdir d0/d10/d27/d1d/d38/d44 0 2024-04-12T05:08:52.432 INFO:tasks.workunit.client.1.smithi192.stdout:2/100: dwrite d1/d8/db/de/f1e [0,4194304] 0 2024-04-12T05:08:52.433 INFO:tasks.workunit.client.0.smithi113.stdout:0/232: rmdir d1/d6/d16/d2a 39 2024-04-12T05:08:52.433 INFO:tasks.workunit.client.1.smithi192.stdout:1/57: fsync d0/d2/fa 0 2024-04-12T05:08:52.436 INFO:tasks.workunit.client.1.smithi192.stdout:7/86: write d3/d7/f8 [639436,72391] 0 2024-04-12T05:08:52.438 INFO:tasks.workunit.client.0.smithi113.stdout:5/247: creat d0/d10/f45 x:0 0 0 2024-04-12T05:08:52.438 INFO:tasks.workunit.client.1.smithi192.stdout:1/58: rmdir d0/d2 39 2024-04-12T05:08:52.441 INFO:tasks.workunit.client.1.smithi192.stdout:6/132: mknod d3/d23/c2f 0 2024-04-12T05:08:52.442 INFO:tasks.workunit.client.1.smithi192.stdout:8/49: creat d6/da/fd x:0 0 0 2024-04-12T05:08:52.443 INFO:tasks.workunit.client.0.smithi113.stdout:0/233: dwrite d1/d6/d34/f40 [0,4194304] 0 2024-04-12T05:08:52.443 INFO:tasks.workunit.client.1.smithi192.stdout:7/87: rename d3/d7/d15/d1a/d1b to d3/d7/d14/d1f 0 2024-04-12T05:08:52.444 INFO:tasks.workunit.client.0.smithi113.stdout:1/186: truncate fe 1409418 0 2024-04-12T05:08:52.446 INFO:tasks.workunit.client.0.smithi113.stdout:5/248: symlink d0/d10/d27/d1d/d38/d3b/l46 0 2024-04-12T05:08:52.446 INFO:tasks.workunit.client.1.smithi192.stdout:6/133: stat d3/d26/c2b 0 2024-04-12T05:08:52.446 INFO:tasks.workunit.client.1.smithi192.stdout:1/59: symlink d0/d2/l14 0 2024-04-12T05:08:52.447 INFO:tasks.workunit.client.0.smithi113.stdout:0/234: readlink d1/d6/l3e 0 2024-04-12T05:08:52.447 INFO:tasks.workunit.client.1.smithi192.stdout:6/134: chown d3/d26/l29 1 1 2024-04-12T05:08:52.448 INFO:tasks.workunit.client.0.smithi113.stdout:1/187: dread f2 [0,4194304] 0 2024-04-12T05:08:52.449 INFO:tasks.workunit.client.1.smithi192.stdout:7/88: mknod d3/d7/d14/d1f/c20 0 2024-04-12T05:08:52.449 INFO:tasks.workunit.client.1.smithi192.stdout:1/60: chown d0/d2/f9 15435088 1 2024-04-12T05:08:52.450 INFO:tasks.workunit.client.1.smithi192.stdout:5/154: dwrite d0/d21/f36 [0,4194304] 0 2024-04-12T05:08:52.451 INFO:tasks.workunit.client.0.smithi113.stdout:5/249: creat d0/d10/d27/d1d/d3f/f47 x:0 0 0 2024-04-12T05:08:52.451 INFO:tasks.workunit.client.1.smithi192.stdout:8/50: mknod d6/ce 0 2024-04-12T05:08:52.451 INFO:tasks.workunit.client.0.smithi113.stdout:0/235: mknod d1/c42 0 2024-04-12T05:08:52.451 INFO:tasks.workunit.client.1.smithi192.stdout:7/89: read d3/f12 [2622350,25241] 0 2024-04-12T05:08:52.452 INFO:tasks.workunit.client.1.smithi192.stdout:5/155: fsync d0/f1e 0 2024-04-12T05:08:52.452 INFO:tasks.workunit.client.0.smithi113.stdout:0/236: dread - d1/d6/da/f3a zero size 2024-04-12T05:08:52.453 INFO:tasks.workunit.client.0.smithi113.stdout:5/250: readlink d0/d10/d27/d1d/d38/d3b/l46 0 2024-04-12T05:08:52.453 INFO:tasks.workunit.client.0.smithi113.stdout:0/237: readlink d1/d6/da/l12 0 2024-04-12T05:08:52.454 INFO:tasks.workunit.client.1.smithi192.stdout:6/135: mknod d3/d8/d22/c30 0 2024-04-12T05:08:52.454 INFO:tasks.workunit.client.1.smithi192.stdout:8/51: write d6/da/fc [173063,75022] 0 2024-04-12T05:08:52.454 INFO:tasks.workunit.client.0.smithi113.stdout:5/251: dread - d0/d10/f45 zero size 2024-04-12T05:08:52.454 INFO:tasks.workunit.client.0.smithi113.stdout:0/238: readlink d1/d6/da/l41 0 2024-04-12T05:08:52.454 INFO:tasks.workunit.client.0.smithi113.stdout:1/188: unlink d10/d21/d23/l27 0 2024-04-12T05:08:52.455 INFO:tasks.workunit.client.1.smithi192.stdout:5/156: chown d0/f20 97 1 2024-04-12T05:08:52.455 INFO:tasks.workunit.client.0.smithi113.stdout:1/189: write fa [789556,19431] 0 2024-04-12T05:08:52.457 INFO:tasks.workunit.client.0.smithi113.stdout:5/252: write d0/f2e [1147521,40597] 0 2024-04-12T05:08:52.457 INFO:tasks.workunit.client.0.smithi113.stdout:1/190: chown d10/d21/c22 0 1 2024-04-12T05:08:52.459 INFO:tasks.workunit.client.0.smithi113.stdout:1/191: dread - d10/d14/f2a zero size 2024-04-12T05:08:52.459 INFO:tasks.workunit.client.0.smithi113.stdout:0/239: rmdir d1/d6/d16 39 2024-04-12T05:08:52.460 INFO:tasks.workunit.client.0.smithi113.stdout:1/192: read - d10/d14/f17 zero size 2024-04-12T05:08:52.461 INFO:tasks.workunit.client.0.smithi113.stdout:0/240: write d1/f2f [496033,72091] 0 2024-04-12T05:08:52.462 INFO:tasks.workunit.client.0.smithi113.stdout:1/193: truncate d10/d2e/d31/f35 85843 0 2024-04-12T05:08:52.462 INFO:tasks.workunit.client.1.smithi192.stdout:6/136: creat d3/d26/f31 x:0 0 0 2024-04-12T05:08:52.464 INFO:tasks.workunit.client.0.smithi113.stdout:5/253: rename d0/f14 to d0/d10/d27/d1d/d38/d44/f48 0 2024-04-12T05:08:52.466 INFO:tasks.workunit.client.1.smithi192.stdout:6/137: dread - d3/d26/f31 zero size 2024-04-12T05:08:52.468 INFO:tasks.workunit.client.0.smithi113.stdout:1/194: write d10/d14/f1f [4730088,72335] 0 2024-04-12T05:08:52.468 INFO:tasks.workunit.client.1.smithi192.stdout:5/157: rename d0/d21/f35 to d0/f3d 0 2024-04-12T05:08:52.472 INFO:tasks.workunit.client.0.smithi113.stdout:0/241: dwrite d1/d6/d16/f24 [0,4194304] 0 2024-04-12T05:08:52.473 INFO:tasks.workunit.client.1.smithi192.stdout:7/90: dwrite d3/ff [0,4194304] 0 2024-04-12T05:08:52.475 INFO:tasks.workunit.client.1.smithi192.stdout:1/61: dwrite d0/d2/f9 [0,4194304] 0 2024-04-12T05:08:52.479 INFO:tasks.workunit.client.1.smithi192.stdout:8/52: dwrite d6/da/fd [0,4194304] 0 2024-04-12T05:08:52.481 INFO:tasks.workunit.client.1.smithi192.stdout:1/62: write d0/de/f13 [439555,47903] 0 2024-04-12T05:08:52.481 INFO:tasks.workunit.client.1.smithi192.stdout:7/91: creat d3/f21 x:0 0 0 2024-04-12T05:08:52.484 INFO:tasks.workunit.client.0.smithi113.stdout:8/239: sync 2024-04-12T05:08:52.485 INFO:tasks.workunit.client.0.smithi113.stdout:8/240: chown d9/d1b/c1d 5443282 1 2024-04-12T05:08:52.485 INFO:tasks.workunit.client.1.smithi192.stdout:1/63: write d0/f3 [3231225,124765] 0 2024-04-12T05:08:52.485 INFO:tasks.workunit.client.1.smithi192.stdout:1/64: chown d0 1 1 2024-04-12T05:08:52.485 INFO:tasks.workunit.client.1.smithi192.stdout:7/92: write f1 [1463361,22665] 0 2024-04-12T05:08:52.486 INFO:tasks.workunit.client.0.smithi113.stdout:1/195: dread d10/f13 [0,4194304] 0 2024-04-12T05:08:52.487 INFO:tasks.workunit.client.1.smithi192.stdout:6/138: dread d3/d8/f1f [0,4194304] 0 2024-04-12T05:08:52.487 INFO:tasks.workunit.client.0.smithi113.stdout:1/196: write d10/d14/f17 [414853,106146] 0 2024-04-12T05:08:52.489 INFO:tasks.workunit.client.1.smithi192.stdout:1/65: write d0/fc [5134052,96120] 0 2024-04-12T05:08:52.489 INFO:tasks.workunit.client.1.smithi192.stdout:7/93: dread - d3/d7/f19 zero size 2024-04-12T05:08:52.490 INFO:tasks.workunit.client.1.smithi192.stdout:7/94: chown d3/f6 36488 1 2024-04-12T05:08:52.495 INFO:tasks.workunit.client.0.smithi113.stdout:1/197: dwrite d10/d14/f2a [0,4194304] 0 2024-04-12T05:08:52.497 INFO:tasks.workunit.client.0.smithi113.stdout:8/241: unlink d9/d16/c22 0 2024-04-12T05:08:52.497 INFO:tasks.workunit.client.0.smithi113.stdout:0/242: truncate d1/d6/d16/d2a/d2c/f37 277170 0 2024-04-12T05:08:52.497 INFO:tasks.workunit.client.0.smithi113.stdout:1/198: read - d10/d2e/d31/d33/f2c zero size 2024-04-12T05:08:52.499 INFO:tasks.workunit.client.1.smithi192.stdout:6/139: mknod d3/d21/c32 0 2024-04-12T05:08:52.503 INFO:tasks.workunit.client.0.smithi113.stdout:0/243: mkdir d1/d43 0 2024-04-12T05:08:52.505 INFO:tasks.workunit.client.0.smithi113.stdout:8/242: creat d9/d16/d3c/f4e x:0 0 0 2024-04-12T05:08:52.505 INFO:tasks.workunit.client.1.smithi192.stdout:5/158: write d0/d15/d24/d29/f2a [859906,60713] 0 2024-04-12T05:08:52.505 INFO:tasks.workunit.client.1.smithi192.stdout:1/66: creat d0/d2/dd/f15 x:0 0 0 2024-04-12T05:08:52.505 INFO:tasks.workunit.client.1.smithi192.stdout:8/53: dread f3 [0,4194304] 0 2024-04-12T05:08:52.505 INFO:tasks.workunit.client.1.smithi192.stdout:7/95: mkdir d3/d7/d15/d22 0 2024-04-12T05:08:52.506 INFO:tasks.workunit.client.0.smithi113.stdout:8/243: write d9/d16/d1e/d26/f4d [608746,38225] 0 2024-04-12T05:08:52.507 INFO:tasks.workunit.client.0.smithi113.stdout:0/244: write d1/d6/d16/d2a/f3f [705507,72977] 0 2024-04-12T05:08:52.507 INFO:tasks.workunit.client.1.smithi192.stdout:1/67: symlink d0/d2/l16 0 2024-04-12T05:08:52.507 INFO:tasks.workunit.client.1.smithi192.stdout:8/54: write d6/da/fc [722034,118070] 0 2024-04-12T05:08:52.507 INFO:tasks.workunit.client.0.smithi113.stdout:1/199: rmdir d10/d2f 0 2024-04-12T05:08:52.508 INFO:tasks.workunit.client.0.smithi113.stdout:1/200: readlink d10/d14/l1c 0 2024-04-12T05:08:52.509 INFO:tasks.workunit.client.0.smithi113.stdout:1/201: fdatasync d10/d21/f29 0 2024-04-12T05:08:52.511 INFO:tasks.workunit.client.0.smithi113.stdout:8/244: unlink d9/l40 0 2024-04-12T05:08:52.512 INFO:tasks.workunit.client.0.smithi113.stdout:0/245: dread d1/fe [0,4194304] 0 2024-04-12T05:08:52.512 INFO:tasks.workunit.client.1.smithi192.stdout:8/55: fsync f3 0 2024-04-12T05:08:52.513 INFO:tasks.workunit.client.0.smithi113.stdout:8/245: write d9/d16/d1e/d26/d32/f3f [156907,21401] 0 2024-04-12T05:08:52.513 INFO:tasks.workunit.client.1.smithi192.stdout:1/68: creat d0/d2/dd/f17 x:0 0 0 2024-04-12T05:08:52.514 INFO:tasks.workunit.client.1.smithi192.stdout:8/56: stat d6/da/fd 0 2024-04-12T05:08:52.514 INFO:tasks.workunit.client.0.smithi113.stdout:1/202: rename d10/f34 to d10/d21/d23/f36 0 2024-04-12T05:08:52.515 INFO:tasks.workunit.client.1.smithi192.stdout:6/140: dwrite d3/d8/d22/f27 [0,4194304] 0 2024-04-12T05:08:52.515 INFO:tasks.workunit.client.0.smithi113.stdout:8/246: truncate d9/d16/d1c/f2c 918245 0 2024-04-12T05:08:52.516 INFO:tasks.workunit.client.0.smithi113.stdout:1/203: dread - d10/d2e/d31/d33/f2c zero size 2024-04-12T05:08:52.520 INFO:tasks.workunit.client.1.smithi192.stdout:8/57: mkdir d6/df 0 2024-04-12T05:08:52.521 INFO:tasks.workunit.client.0.smithi113.stdout:8/247: creat d9/d1b/f4f x:0 0 0 2024-04-12T05:08:52.522 INFO:tasks.workunit.client.0.smithi113.stdout:8/248: chown d9/l4c 162022220 1 2024-04-12T05:08:52.522 INFO:tasks.workunit.client.1.smithi192.stdout:8/58: write d6/f9 [1123413,22760] 0 2024-04-12T05:08:52.523 INFO:tasks.workunit.client.0.smithi113.stdout:1/204: symlink d10/d21/l37 0 2024-04-12T05:08:52.523 INFO:tasks.workunit.client.1.smithi192.stdout:6/141: symlink d3/d21/d24/l33 0 2024-04-12T05:08:52.528 INFO:tasks.workunit.client.1.smithi192.stdout:5/159: write d0/d21/f26 [321516,83552] 0 2024-04-12T05:08:52.531 INFO:tasks.workunit.client.1.smithi192.stdout:6/142: symlink d3/d5/l34 0 2024-04-12T05:08:52.536 INFO:tasks.workunit.client.1.smithi192.stdout:8/59: mkdir d6/d10 0 2024-04-12T05:08:52.536 INFO:tasks.workunit.client.1.smithi192.stdout:5/160: chown d0/d15/d24/d29/f30 16550704 1 2024-04-12T05:08:52.538 INFO:tasks.workunit.client.1.smithi192.stdout:6/143: write d3/d8/f17 [5049305,55122] 0 2024-04-12T05:08:52.540 INFO:tasks.workunit.client.1.smithi192.stdout:8/60: chown d6/da/fc 183754 1 2024-04-12T05:08:52.541 INFO:tasks.workunit.client.1.smithi192.stdout:1/69: dwrite d0/d2/fa [0,4194304] 0 2024-04-12T05:08:52.542 INFO:tasks.workunit.client.0.smithi113.stdout:1/205: rmdir d10/d21/d23 39 2024-04-12T05:08:52.544 INFO:tasks.workunit.client.0.smithi113.stdout:0/246: truncate d1/d6/d34/f40 2190272 0 2024-04-12T05:08:52.544 INFO:tasks.workunit.client.1.smithi192.stdout:5/161: mkdir d0/d15/d3e 0 2024-04-12T05:08:52.544 INFO:tasks.workunit.client.0.smithi113.stdout:0/247: readlink d1/d9/l23 0 2024-04-12T05:08:52.547 INFO:tasks.workunit.client.0.smithi113.stdout:1/206: dread d10/d14/f2a [0,4194304] 0 2024-04-12T05:08:52.549 INFO:tasks.workunit.client.1.smithi192.stdout:7/96: dwrite d3/f12 [4194304,4194304] 0 2024-04-12T05:08:52.549 INFO:tasks.workunit.client.1.smithi192.stdout:6/144: fdatasync d3/d8/f25 0 2024-04-12T05:08:52.549 INFO:tasks.workunit.client.1.smithi192.stdout:8/61: creat d6/d10/f11 x:0 0 0 2024-04-12T05:08:52.551 INFO:tasks.workunit.client.1.smithi192.stdout:5/162: fsync d0/d15/d24/f25 0 2024-04-12T05:08:52.551 INFO:tasks.workunit.client.1.smithi192.stdout:8/62: write d6/da/fd [3489666,12409] 0 2024-04-12T05:08:52.552 INFO:tasks.workunit.client.1.smithi192.stdout:5/163: chown d0/f7 74546 1 2024-04-12T05:08:52.552 INFO:tasks.workunit.client.1.smithi192.stdout:8/63: fdatasync d6/f9 0 2024-04-12T05:08:52.552 INFO:tasks.workunit.client.0.smithi113.stdout:0/248: creat d1/d6/d16/d2a/f44 x:0 0 0 2024-04-12T05:08:52.553 INFO:tasks.workunit.client.1.smithi192.stdout:1/70: creat d0/f18 x:0 0 0 2024-04-12T05:08:52.555 INFO:tasks.workunit.client.0.smithi113.stdout:1/207: symlink d10/d21/l38 0 2024-04-12T05:08:52.556 INFO:tasks.workunit.client.1.smithi192.stdout:7/97: symlink d3/d7/d14/l23 0 2024-04-12T05:08:52.556 INFO:tasks.workunit.client.1.smithi192.stdout:1/71: write d0/d2/fa [940421,89243] 0 2024-04-12T05:08:52.557 INFO:tasks.workunit.client.0.smithi113.stdout:0/249: getdents d1/d43 0 2024-04-12T05:08:52.558 INFO:tasks.workunit.client.1.smithi192.stdout:1/72: dread - d0/d2/dd/f15 zero size 2024-04-12T05:08:52.558 INFO:tasks.workunit.client.0.smithi113.stdout:8/249: dwrite d9/fb [0,4194304] 0 2024-04-12T05:08:52.558 INFO:tasks.workunit.client.1.smithi192.stdout:1/73: chown d0/d2/f9 2304885 1 2024-04-12T05:08:52.559 INFO:tasks.workunit.client.0.smithi113.stdout:1/208: truncate f2 2578217 0 2024-04-12T05:08:52.560 INFO:tasks.workunit.client.0.smithi113.stdout:0/250: write d1/d6/d16/d2a/f44 [744733,46900] 0 2024-04-12T05:08:52.561 INFO:tasks.workunit.client.1.smithi192.stdout:5/164: read d0/d15/f23 [4091917,83980] 0 2024-04-12T05:08:52.563 INFO:tasks.workunit.client.1.smithi192.stdout:1/74: creat d0/de/f19 x:0 0 0 2024-04-12T05:08:52.565 INFO:tasks.workunit.client.0.smithi113.stdout:0/251: chown d1/f22 1609 1 2024-04-12T05:08:52.565 INFO:tasks.workunit.client.0.smithi113.stdout:8/250: rename d9/d16/f25 to d9/d16/d3c/f50 0 2024-04-12T05:08:52.566 INFO:tasks.workunit.client.0.smithi113.stdout:8/251: read - d9/d16/d3c/f4e zero size 2024-04-12T05:08:52.566 INFO:tasks.workunit.client.1.smithi192.stdout:5/165: dread d0/f13 [0,4194304] 0 2024-04-12T05:08:52.567 INFO:tasks.workunit.client.0.smithi113.stdout:1/209: dread d10/f13 [0,4194304] 0 2024-04-12T05:08:52.568 INFO:tasks.workunit.client.0.smithi113.stdout:0/252: link d1/d9/f2b d1/d9/f45 0 2024-04-12T05:08:52.569 INFO:tasks.workunit.client.0.smithi113.stdout:0/253: chown d1/d6/d16/d2a/d2c 193187 1 2024-04-12T05:08:52.571 INFO:tasks.workunit.client.1.smithi192.stdout:7/98: link f1 d3/d7/d15/f24 0 2024-04-12T05:08:52.571 INFO:tasks.workunit.client.0.smithi113.stdout:0/254: read d1/f2f [209778,114025] 0 2024-04-12T05:08:52.572 INFO:tasks.workunit.client.0.smithi113.stdout:1/210: mknod d10/d21/c39 0 2024-04-12T05:08:52.572 INFO:tasks.workunit.client.0.smithi113.stdout:0/255: chown d1/d9/f29 4716 1 2024-04-12T05:08:52.574 INFO:tasks.workunit.client.0.smithi113.stdout:0/256: fsync d1/d6/d16/f1c 0 2024-04-12T05:08:52.574 INFO:tasks.workunit.client.0.smithi113.stdout:1/211: write d10/f13 [241051,98399] 0 2024-04-12T05:08:52.574 INFO:tasks.workunit.client.0.smithi113.stdout:8/252: getdents d9/d16/d1c 0 2024-04-12T05:08:52.575 INFO:tasks.workunit.client.1.smithi192.stdout:7/99: mknod d3/d1d/c25 0 2024-04-12T05:08:52.579 INFO:tasks.workunit.client.1.smithi192.stdout:1/75: creat d0/f1a x:0 0 0 2024-04-12T05:08:52.579 INFO:tasks.workunit.client.0.smithi113.stdout:8/253: rename d9/f17 to d9/d16/d1c/f51 0 2024-04-12T05:08:52.582 INFO:tasks.workunit.client.0.smithi113.stdout:8/254: creat d9/d16/d3c/f52 x:0 0 0 2024-04-12T05:08:52.583 INFO:tasks.workunit.client.0.smithi113.stdout:1/212: dwrite d10/d21/f29 [0,4194304] 0 2024-04-12T05:08:52.583 INFO:tasks.workunit.client.1.smithi192.stdout:8/64: truncate d6/da/fd 4028970 0 2024-04-12T05:08:52.585 INFO:tasks.workunit.client.1.smithi192.stdout:0/50: sync 2024-04-12T05:08:52.585 INFO:tasks.workunit.client.1.smithi192.stdout:4/78: sync 2024-04-12T05:08:52.585 INFO:tasks.workunit.client.1.smithi192.stdout:2/101: sync 2024-04-12T05:08:52.585 INFO:tasks.workunit.client.1.smithi192.stdout:9/76: sync 2024-04-12T05:08:52.585 INFO:tasks.workunit.client.1.smithi192.stdout:3/68: sync 2024-04-12T05:08:52.586 INFO:tasks.workunit.client.0.smithi113.stdout:8/255: creat d9/f53 x:0 0 0 2024-04-12T05:08:52.587 INFO:tasks.workunit.client.1.smithi192.stdout:5/166: truncate d0/d15/d24/d29/f2a 912703 0 2024-04-12T05:08:52.587 INFO:tasks.workunit.client.0.smithi113.stdout:8/256: truncate d9/f53 386904 0 2024-04-12T05:08:52.587 INFO:tasks.workunit.client.0.smithi113.stdout:1/213: symlink d10/l3a 0 2024-04-12T05:08:52.588 INFO:tasks.workunit.client.0.smithi113.stdout:0/257: fsync d1/d9/f2b 0 2024-04-12T05:08:52.588 INFO:tasks.workunit.client.1.smithi192.stdout:7/100: dread d3/f12 [4194304,4194304] 0 2024-04-12T05:08:52.588 INFO:tasks.workunit.client.1.smithi192.stdout:0/51: read d0/f8 [1171311,127953] 0 2024-04-12T05:08:52.589 INFO:tasks.workunit.client.1.smithi192.stdout:0/52: chown d0 0 1 2024-04-12T05:08:52.590 INFO:tasks.workunit.client.1.smithi192.stdout:6/145: dwrite d3/f2c [4194304,4194304] 0 2024-04-12T05:08:52.591 INFO:tasks.workunit.client.0.smithi113.stdout:8/257: symlink d9/d34/l54 0 2024-04-12T05:08:52.591 INFO:tasks.workunit.client.1.smithi192.stdout:3/69: mknod d5/d13/c1b 0 2024-04-12T05:08:52.591 INFO:tasks.workunit.client.0.smithi113.stdout:8/258: dread - d9/d1b/f46 zero size 2024-04-12T05:08:52.592 INFO:tasks.workunit.client.1.smithi192.stdout:9/77: truncate f10 650274 0 2024-04-12T05:08:52.593 INFO:tasks.workunit.client.1.smithi192.stdout:4/79: write d1/f2 [370813,33264] 0 2024-04-12T05:08:52.593 INFO:tasks.workunit.client.0.smithi113.stdout:1/214: symlink d10/d2e/l3b 0 2024-04-12T05:08:52.594 INFO:tasks.workunit.client.1.smithi192.stdout:1/76: rename d0/de to d0/d1b 0 2024-04-12T05:08:52.597 INFO:tasks.workunit.client.1.smithi192.stdout:8/65: rename d6/c8 to d6/df/c12 0 2024-04-12T05:08:52.597 INFO:tasks.workunit.client.0.smithi113.stdout:0/258: dread d1/d6/d34/f40 [0,4194304] 0 2024-04-12T05:08:52.598 INFO:tasks.workunit.client.1.smithi192.stdout:7/101: readlink d3/d7/d14/l23 0 2024-04-12T05:08:52.603 INFO:tasks.workunit.client.1.smithi192.stdout:5/167: rmdir d0/d15/d24/d29 39 2024-04-12T05:08:52.603 INFO:tasks.workunit.client.1.smithi192.stdout:0/53: write d0/f8 [2430447,120275] 0 2024-04-12T05:08:52.603 INFO:tasks.workunit.client.0.smithi113.stdout:1/215: mkdir d10/d21/d3c 0 2024-04-12T05:08:52.603 INFO:tasks.workunit.client.0.smithi113.stdout:8/259: chown d9/d16/d1e/d3a/c44 2210 1 2024-04-12T05:08:52.603 INFO:tasks.workunit.client.0.smithi113.stdout:0/259: rename d1/d9/c32 to d1/d6/d16/d2a/c46 0 2024-04-12T05:08:52.605 INFO:tasks.workunit.client.0.smithi113.stdout:1/216: chown f2 37180 1 2024-04-12T05:08:52.607 INFO:tasks.workunit.client.0.smithi113.stdout:0/260: read d1/d6/d16/f24 [1475276,90808] 0 2024-04-12T05:08:52.607 INFO:tasks.workunit.client.0.smithi113.stdout:8/260: truncate d9/d16/d1c/f2d 730172 0 2024-04-12T05:08:52.608 INFO:tasks.workunit.client.1.smithi192.stdout:3/70: dwrite d5/f19 [0,4194304] 0 2024-04-12T05:08:52.609 INFO:tasks.workunit.client.1.smithi192.stdout:1/77: mkdir d0/d2/dd/d1c 0 2024-04-12T05:08:52.610 INFO:tasks.workunit.client.1.smithi192.stdout:4/80: mkdir d1/d1d 0 2024-04-12T05:08:52.610 INFO:tasks.workunit.client.0.smithi113.stdout:8/261: creat d9/f55 x:0 0 0 2024-04-12T05:08:52.610 INFO:tasks.workunit.client.1.smithi192.stdout:7/102: dread d3/d7/d15/f24 [0,4194304] 0 2024-04-12T05:08:52.611 INFO:tasks.workunit.client.1.smithi192.stdout:0/54: fdatasync d0/f8 0 2024-04-12T05:08:52.612 INFO:tasks.workunit.client.0.smithi113.stdout:0/261: dread d1/d6/d16/d2a/f3f [0,4194304] 0 2024-04-12T05:08:52.613 INFO:tasks.workunit.client.1.smithi192.stdout:3/71: write f2 [49210,102250] 0 2024-04-12T05:08:52.613 INFO:tasks.workunit.client.1.smithi192.stdout:1/78: chown d0/d2/f9 5902449 1 2024-04-12T05:08:52.614 INFO:tasks.workunit.client.0.smithi113.stdout:8/262: creat d9/d1b/f56 x:0 0 0 2024-04-12T05:08:52.615 INFO:tasks.workunit.client.1.smithi192.stdout:1/79: readlink d0/d2/l14 0 2024-04-12T05:08:52.615 INFO:tasks.workunit.client.0.smithi113.stdout:1/217: link d10/d21/l38 d10/d2e/l3d 0 2024-04-12T05:08:52.616 INFO:tasks.workunit.client.1.smithi192.stdout:8/66: creat d6/d10/f13 x:0 0 0 2024-04-12T05:08:52.616 INFO:tasks.workunit.client.1.smithi192.stdout:0/55: write d0/f8 [1776272,58903] 0 2024-04-12T05:08:52.618 INFO:tasks.workunit.client.0.smithi113.stdout:1/218: dread d10/d2e/d31/f35 [0,4194304] 0 2024-04-12T05:08:52.618 INFO:tasks.workunit.client.0.smithi113.stdout:8/263: creat d9/d34/f57 x:0 0 0 2024-04-12T05:08:52.620 INFO:tasks.workunit.client.0.smithi113.stdout:1/219: fdatasync d10/d21/f29 0 2024-04-12T05:08:52.621 INFO:tasks.workunit.client.0.smithi113.stdout:0/262: rename d1/d6/d16/d2a/f44 to d1/d6/d16/f47 0 2024-04-12T05:08:52.627 INFO:tasks.workunit.client.1.smithi192.stdout:2/102: dwrite d1/d8/db/de/f11 [4194304,4194304] 0 2024-04-12T05:08:52.627 INFO:tasks.workunit.client.1.smithi192.stdout:4/81: dread d1/f14 [0,4194304] 0 2024-04-12T05:08:52.627 INFO:tasks.workunit.client.1.smithi192.stdout:3/72: rename d5/f9 to d5/f1c 0 2024-04-12T05:08:52.628 INFO:tasks.workunit.client.1.smithi192.stdout:7/103: dwrite d3/ff [0,4194304] 0 2024-04-12T05:08:52.629 INFO:tasks.workunit.client.1.smithi192.stdout:9/78: write f10 [997872,67395] 0 2024-04-12T05:08:52.632 INFO:tasks.workunit.client.1.smithi192.stdout:3/73: write d5/f19 [300601,76826] 0 2024-04-12T05:08:52.632 INFO:tasks.workunit.client.1.smithi192.stdout:0/56: fsync d0/fb 0 2024-04-12T05:08:52.632 INFO:tasks.workunit.client.1.smithi192.stdout:5/168: symlink d0/d15/d24/l3f 0 2024-04-12T05:08:52.633 INFO:tasks.workunit.client.1.smithi192.stdout:6/146: getdents d3/d21 0 2024-04-12T05:08:52.640 INFO:tasks.workunit.client.1.smithi192.stdout:6/147: dread - d3/d21/d24/f2a zero size 2024-04-12T05:08:52.641 INFO:tasks.workunit.client.1.smithi192.stdout:3/74: getdents d5/d13 0 2024-04-12T05:08:52.644 INFO:tasks.workunit.client.1.smithi192.stdout:2/103: dread d1/d8/db/de/f1e [0,4194304] 0 2024-04-12T05:08:52.645 INFO:tasks.workunit.client.1.smithi192.stdout:9/79: dwrite f6 [0,4194304] 0 2024-04-12T05:08:52.648 INFO:tasks.workunit.client.1.smithi192.stdout:7/104: dread d3/fd [0,4194304] 0 2024-04-12T05:08:52.648 INFO:tasks.workunit.client.1.smithi192.stdout:0/57: dwrite d0/f8 [0,4194304] 0 2024-04-12T05:08:52.651 INFO:tasks.workunit.client.1.smithi192.stdout:2/104: chown d1/d8/d17/l1a 9756 1 2024-04-12T05:08:52.651 INFO:tasks.workunit.client.1.smithi192.stdout:1/80: truncate d0/d2/f9 2874476 0 2024-04-12T05:08:52.655 INFO:tasks.workunit.client.1.smithi192.stdout:9/80: symlink d11/d14/l1c 0 2024-04-12T05:08:52.655 INFO:tasks.workunit.client.1.smithi192.stdout:0/58: chown d0/f8 829247 1 2024-04-12T05:08:52.657 INFO:tasks.workunit.client.1.smithi192.stdout:1/81: dread - d0/d1b/f19 zero size 2024-04-12T05:08:52.658 INFO:tasks.workunit.client.1.smithi192.stdout:4/82: dwrite d1/f14 [0,4194304] 0 2024-04-12T05:08:52.661 INFO:tasks.workunit.client.1.smithi192.stdout:1/82: rmdir d0/d1b 39 2024-04-12T05:08:52.665 INFO:tasks.workunit.client.1.smithi192.stdout:0/59: truncate d0/fb 3682383 0 2024-04-12T05:08:52.665 INFO:tasks.workunit.client.1.smithi192.stdout:2/105: dread d1/d8/db/de/f1e [0,4194304] 0 2024-04-12T05:08:52.667 INFO:tasks.workunit.client.1.smithi192.stdout:2/106: dread - d1/d8/d17/f20 zero size 2024-04-12T05:08:52.668 INFO:tasks.workunit.client.1.smithi192.stdout:4/83: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:52.669 INFO:tasks.workunit.client.0.smithi113.stdout:2/135: sync 2024-04-12T05:08:52.669 INFO:tasks.workunit.client.1.smithi192.stdout:9/81: getdents d11 0 2024-04-12T05:08:52.670 INFO:tasks.workunit.client.1.smithi192.stdout:4/84: write d1/f2 [171971,35998] 0 2024-04-12T05:08:52.670 INFO:tasks.workunit.client.0.smithi113.stdout:1/220: mknod d10/d2e/d31/d33/c3e 0 2024-04-12T05:08:52.671 INFO:tasks.workunit.client.1.smithi192.stdout:4/85: read d1/f14 [732764,48950] 0 2024-04-12T05:08:52.672 INFO:tasks.workunit.client.1.smithi192.stdout:0/60: mknod d0/d6/cf 0 2024-04-12T05:08:52.673 INFO:tasks.workunit.client.1.smithi192.stdout:0/61: readlink d0/le 0 2024-04-12T05:08:52.673 INFO:tasks.workunit.client.1.smithi192.stdout:6/148: truncate d3/fd 7259563 0 2024-04-12T05:08:52.674 INFO:tasks.workunit.client.0.smithi113.stdout:2/136: symlink d8/d9/de/l21 0 2024-04-12T05:08:52.676 INFO:tasks.workunit.client.1.smithi192.stdout:0/62: chown d0/fb 11150487 1 2024-04-12T05:08:52.680 INFO:tasks.workunit.client.1.smithi192.stdout:4/86: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:52.681 INFO:tasks.workunit.client.1.smithi192.stdout:1/83: rmdir d0/d2/dd/d1c 0 2024-04-12T05:08:52.683 INFO:tasks.workunit.client.1.smithi192.stdout:6/149: mknod d3/d8/c35 0 2024-04-12T05:08:52.684 INFO:tasks.workunit.client.1.smithi192.stdout:7/105: truncate d3/fd 2578594 0 2024-04-12T05:08:52.684 INFO:tasks.workunit.client.1.smithi192.stdout:7/106: readlink d3/d7/d14/d1f/l1e 0 2024-04-12T05:08:52.684 INFO:tasks.workunit.client.1.smithi192.stdout:0/63: symlink d0/l10 0 2024-04-12T05:08:52.687 INFO:tasks.workunit.client.1.smithi192.stdout:2/107: creat d1/d8/f22 x:0 0 0 2024-04-12T05:08:52.692 INFO:tasks.workunit.client.1.smithi192.stdout:6/150: mknod d3/d5/c36 0 2024-04-12T05:08:52.693 INFO:tasks.workunit.client.1.smithi192.stdout:6/151: stat d3/dc 0 2024-04-12T05:08:52.693 INFO:tasks.workunit.client.1.smithi192.stdout:0/64: rmdir d0 39 2024-04-12T05:08:52.695 INFO:tasks.workunit.client.1.smithi192.stdout:7/107: mknod d3/d7/c26 0 2024-04-12T05:08:52.695 INFO:tasks.workunit.client.1.smithi192.stdout:7/108: chown f1 1 1 2024-04-12T05:08:52.698 INFO:tasks.workunit.client.1.smithi192.stdout:2/108: rename d1/d8/d17/l1b to d1/d8/d17/d19/l23 0 2024-04-12T05:08:52.700 INFO:tasks.workunit.client.1.smithi192.stdout:6/152: creat d3/f37 x:0 0 0 2024-04-12T05:08:52.702 INFO:tasks.workunit.client.1.smithi192.stdout:0/65: rename d0 to d0/d11 22 2024-04-12T05:08:52.704 INFO:tasks.workunit.client.1.smithi192.stdout:7/109: mkdir d3/d7/d27 0 2024-04-12T05:08:52.704 INFO:tasks.workunit.client.1.smithi192.stdout:7/110: chown d3/d7/d14/l23 95469 1 2024-04-12T05:08:52.705 INFO:tasks.workunit.client.1.smithi192.stdout:4/87: rmdir d1/de/d1b 0 2024-04-12T05:08:52.706 INFO:tasks.workunit.client.1.smithi192.stdout:2/109: creat d1/d8/d17/f24 x:0 0 0 2024-04-12T05:08:52.709 INFO:tasks.workunit.client.1.smithi192.stdout:6/153: creat d3/d5/f38 x:0 0 0 2024-04-12T05:08:52.711 INFO:tasks.workunit.client.1.smithi192.stdout:6/154: read d3/d8/f17 [3651868,95930] 0 2024-04-12T05:08:52.711 INFO:tasks.workunit.client.1.smithi192.stdout:7/111: unlink d3/f6 0 2024-04-12T05:08:52.713 INFO:tasks.workunit.client.1.smithi192.stdout:2/110: creat d1/f25 x:0 0 0 2024-04-12T05:08:52.717 INFO:tasks.workunit.client.1.smithi192.stdout:7/112: creat d3/d1d/f28 x:0 0 0 2024-04-12T05:08:52.717 INFO:tasks.workunit.client.1.smithi192.stdout:4/88: dwrite d1/f14 [0,4194304] 0 2024-04-12T05:08:52.718 INFO:tasks.workunit.client.1.smithi192.stdout:7/113: write d3/f21 [954172,74379] 0 2024-04-12T05:08:52.718 INFO:tasks.workunit.client.1.smithi192.stdout:6/155: mkdir d3/dc/d39 0 2024-04-12T05:08:52.719 INFO:tasks.workunit.client.1.smithi192.stdout:2/111: creat d1/d8/db/f26 x:0 0 0 2024-04-12T05:08:52.720 INFO:tasks.workunit.client.1.smithi192.stdout:2/112: chown d1/d8/db/de/l18 1300891774 1 2024-04-12T05:08:52.722 INFO:tasks.workunit.client.1.smithi192.stdout:4/89: symlink d1/d1d/l1e 0 2024-04-12T05:08:52.722 INFO:tasks.workunit.client.1.smithi192.stdout:1/84: write d0/f3 [117193,101929] 0 2024-04-12T05:08:52.723 INFO:tasks.workunit.client.1.smithi192.stdout:6/156: creat d3/dc/d39/f3a x:0 0 0 2024-04-12T05:08:52.725 INFO:tasks.workunit.client.1.smithi192.stdout:2/113: rename d1/d8/db/f21 to d1/d8/d17/d19/f27 0 2024-04-12T05:08:52.728 INFO:tasks.workunit.client.1.smithi192.stdout:1/85: creat d0/d2/dd/f1d x:0 0 0 2024-04-12T05:08:52.730 INFO:tasks.workunit.client.1.smithi192.stdout:4/90: dwrite d1/f14 [0,4194304] 0 2024-04-12T05:08:52.730 INFO:tasks.workunit.client.1.smithi192.stdout:1/86: read - d0/d2/dd/f17 zero size 2024-04-12T05:08:52.731 INFO:tasks.workunit.client.1.smithi192.stdout:4/91: rename d1 to d1/d1f 22 2024-04-12T05:08:52.731 INFO:tasks.workunit.client.0.smithi113.stdout:0/263: mknod d1/d43/c48 0 2024-04-12T05:08:52.732 INFO:tasks.workunit.client.1.smithi192.stdout:6/157: truncate d3/d8/f1f 5865932 0 2024-04-12T05:08:52.732 INFO:tasks.workunit.client.1.smithi192.stdout:4/92: fsync d1/f18 0 2024-04-12T05:08:52.732 INFO:tasks.workunit.client.0.smithi113.stdout:8/264: link d9/d16/d1c/l31 d9/d16/l58 0 2024-04-12T05:08:52.732 INFO:tasks.workunit.client.0.smithi113.stdout:0/264: chown d1/d9/f29 439 1 2024-04-12T05:08:52.734 INFO:tasks.workunit.client.0.smithi113.stdout:2/137: creat d8/d9/de/f22 x:0 0 0 2024-04-12T05:08:52.734 INFO:tasks.workunit.client.1.smithi192.stdout:6/158: rmdir d3/d8/d10 39 2024-04-12T05:08:52.735 INFO:tasks.workunit.client.1.smithi192.stdout:6/159: dread - d3/d26/f31 zero size 2024-04-12T05:08:52.737 INFO:tasks.workunit.client.1.smithi192.stdout:4/93: rename d1/de to d1/d20 0 2024-04-12T05:08:52.737 INFO:tasks.workunit.client.0.smithi113.stdout:8/265: mkdir d9/d16/d1c/d59 0 2024-04-12T05:08:52.739 INFO:tasks.workunit.client.1.smithi192.stdout:4/94: chown d1/d1d/l1e 20422 1 2024-04-12T05:08:52.739 INFO:tasks.workunit.client.1.smithi192.stdout:1/87: dwrite d0/fc [0,4194304] 0 2024-04-12T05:08:52.739 INFO:tasks.workunit.client.0.smithi113.stdout:0/265: symlink d1/d6/d16/d2a/l49 0 2024-04-12T05:08:52.740 INFO:tasks.workunit.client.0.smithi113.stdout:8/266: dread d9/d16/d1c/f2d [0,4194304] 0 2024-04-12T05:08:52.741 INFO:tasks.workunit.client.1.smithi192.stdout:2/114: write d1/f2 [4651094,9951] 0 2024-04-12T05:08:52.742 INFO:tasks.workunit.client.0.smithi113.stdout:2/138: creat d8/d9/f23 x:0 0 0 2024-04-12T05:08:52.742 INFO:tasks.workunit.client.1.smithi192.stdout:1/88: truncate d0/d1b/f19 258849 0 2024-04-12T05:08:52.743 INFO:tasks.workunit.client.1.smithi192.stdout:4/95: unlink d1/f14 0 2024-04-12T05:08:52.749 INFO:tasks.workunit.client.1.smithi192.stdout:1/89: unlink d0/d1b/f19 0 2024-04-12T05:08:52.749 INFO:tasks.workunit.client.0.smithi113.stdout:8/267: mkdir d9/d16/d5a 0 2024-04-12T05:08:52.750 INFO:tasks.workunit.client.1.smithi192.stdout:1/90: fdatasync d0/d2/fa 0 2024-04-12T05:08:52.753 INFO:tasks.workunit.client.1.smithi192.stdout:1/91: write d0/d2/f7 [1307600,62289] 0 2024-04-12T05:08:52.754 INFO:tasks.workunit.client.0.smithi113.stdout:0/266: symlink d1/l4a 0 2024-04-12T05:08:52.757 INFO:tasks.workunit.client.1.smithi192.stdout:1/92: unlink d0/d2/l16 0 2024-04-12T05:08:52.757 INFO:tasks.workunit.client.1.smithi192.stdout:1/93: dread - d0/f1a zero size 2024-04-12T05:08:52.758 INFO:tasks.workunit.client.0.smithi113.stdout:0/267: fdatasync d1/fe 0 2024-04-12T05:08:52.758 INFO:tasks.workunit.client.0.smithi113.stdout:8/268: dwrite d9/d1b/f56 [0,4194304] 0 2024-04-12T05:08:52.759 INFO:tasks.workunit.client.1.smithi192.stdout:1/94: truncate d0/d2/dd/f1d 989020 0 2024-04-12T05:08:52.761 INFO:tasks.workunit.client.1.smithi192.stdout:1/95: read d0/d1b/f13 [242767,18105] 0 2024-04-12T05:08:52.762 INFO:tasks.workunit.client.0.smithi113.stdout:8/269: dread d9/d16/d1c/f2d [0,4194304] 0 2024-04-12T05:08:52.763 INFO:tasks.workunit.client.0.smithi113.stdout:0/268: symlink d1/l4b 0 2024-04-12T05:08:52.764 INFO:tasks.workunit.client.0.smithi113.stdout:1/221: truncate d10/d14/f1f 4433686 0 2024-04-12T05:08:52.765 INFO:tasks.workunit.client.1.smithi192.stdout:1/96: mkdir d0/d2/d1e 0 2024-04-12T05:08:52.766 INFO:tasks.workunit.client.0.smithi113.stdout:8/270: mkdir d9/d16/d1e/d26/d32/d5b 0 2024-04-12T05:08:52.767 INFO:tasks.workunit.client.1.smithi192.stdout:6/160: truncate d3/f2c 50878 0 2024-04-12T05:08:52.769 INFO:tasks.workunit.client.0.smithi113.stdout:8/271: unlink d9/d16/d3c/f50 0 2024-04-12T05:08:52.771 INFO:tasks.workunit.client.0.smithi113.stdout:8/272: write d9/d16/d1c/f2c [39872,28952] 0 2024-04-12T05:08:52.771 INFO:tasks.workunit.client.0.smithi113.stdout:1/222: write d10/d21/d23/f36 [994393,27456] 0 2024-04-12T05:08:52.771 INFO:tasks.workunit.client.1.smithi192.stdout:6/161: truncate d3/d21/d24/f2a 934850 0 2024-04-12T05:08:52.773 INFO:tasks.workunit.client.0.smithi113.stdout:2/139: truncate f7 2168716 0 2024-04-12T05:08:52.775 INFO:tasks.workunit.client.0.smithi113.stdout:1/223: symlink d10/d2e/l3f 0 2024-04-12T05:08:52.775 INFO:tasks.workunit.client.0.smithi113.stdout:8/273: read d9/d16/d1c/f2c [328063,107904] 0 2024-04-12T05:08:52.775 INFO:tasks.workunit.client.1.smithi192.stdout:6/162: creat d3/dc/f3b x:0 0 0 2024-04-12T05:08:52.776 INFO:tasks.workunit.client.1.smithi192.stdout:1/97: dwrite d0/d2/fa [4194304,4194304] 0 2024-04-12T05:08:52.776 INFO:tasks.workunit.client.0.smithi113.stdout:8/274: fsync d9/fd 0 2024-04-12T05:08:52.777 INFO:tasks.workunit.client.1.smithi192.stdout:4/96: dwrite d1/f11 [0,4194304] 0 2024-04-12T05:08:52.778 INFO:tasks.workunit.client.1.smithi192.stdout:4/97: rename d1/da to d1/da/d21 22 2024-04-12T05:08:52.779 INFO:tasks.workunit.client.1.smithi192.stdout:4/98: dread - d1/f15 zero size 2024-04-12T05:08:52.780 INFO:tasks.workunit.client.0.smithi113.stdout:1/224: creat d10/d21/f40 x:0 0 0 2024-04-12T05:08:52.781 INFO:tasks.workunit.client.1.smithi192.stdout:6/163: creat d3/d26/f3c x:0 0 0 2024-04-12T05:08:52.781 INFO:tasks.workunit.client.0.smithi113.stdout:8/275: fsync d9/d16/d1c/f2c 0 2024-04-12T05:08:52.783 INFO:tasks.workunit.client.0.smithi113.stdout:8/276: chown d9/d16/d1e/c3d 0 1 2024-04-12T05:08:52.784 INFO:tasks.workunit.client.0.smithi113.stdout:2/140: dwrite d8/d9/de/f22 [0,4194304] 0 2024-04-12T05:08:52.786 INFO:tasks.workunit.client.1.smithi192.stdout:4/99: symlink d1/da/l22 0 2024-04-12T05:08:52.786 INFO:tasks.workunit.client.1.smithi192.stdout:4/100: readlink d1/l13 0 2024-04-12T05:08:52.787 INFO:tasks.workunit.client.0.smithi113.stdout:1/225: dread fd [0,4194304] 0 2024-04-12T05:08:52.790 INFO:tasks.workunit.client.1.smithi192.stdout:4/101: rename d1/d20/c1a to d1/da/c23 0 2024-04-12T05:08:52.795 INFO:tasks.workunit.client.0.smithi113.stdout:1/226: mkdir d10/d41 0 2024-04-12T05:08:52.797 INFO:tasks.workunit.client.0.smithi113.stdout:1/227: stat d10/d2e/l3f 0 2024-04-12T05:08:52.800 INFO:tasks.workunit.client.0.smithi113.stdout:1/228: write ff [5181389,87056] 0 2024-04-12T05:08:52.800 INFO:tasks.workunit.client.0.smithi113.stdout:2/141: dread d8/d16/f1d [0,4194304] 0 2024-04-12T05:08:52.801 INFO:tasks.workunit.client.1.smithi192.stdout:1/98: dwrite d0/d1b/f13 [0,4194304] 0 2024-04-12T05:08:52.802 INFO:tasks.workunit.client.1.smithi192.stdout:4/102: dwrite d1/f15 [0,4194304] 0 2024-04-12T05:08:52.802 INFO:tasks.workunit.client.0.smithi113.stdout:8/277: dwrite d9/d16/d3c/f4e [0,4194304] 0 2024-04-12T05:08:52.804 INFO:tasks.workunit.client.0.smithi113.stdout:8/278: write d9/d1b/f46 [845234,13113] 0 2024-04-12T05:08:52.807 INFO:tasks.workunit.client.0.smithi113.stdout:2/142: mkdir d8/d9/de/d15/d24 0 2024-04-12T05:08:52.809 INFO:tasks.workunit.client.0.smithi113.stdout:1/229: dread d10/d14/f2a [0,4194304] 0 2024-04-12T05:08:52.811 INFO:tasks.workunit.client.1.smithi192.stdout:4/103: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:52.811 INFO:tasks.workunit.client.1.smithi192.stdout:1/99: getdents d0/d1b 0 2024-04-12T05:08:52.813 INFO:tasks.workunit.client.1.smithi192.stdout:4/104: stat d1/f2 0 2024-04-12T05:08:52.815 INFO:tasks.workunit.client.1.smithi192.stdout:1/100: creat d0/d2/f1f x:0 0 0 2024-04-12T05:08:52.816 INFO:tasks.workunit.client.1.smithi192.stdout:4/105: mknod d1/c24 0 2024-04-12T05:08:52.817 INFO:tasks.workunit.client.0.smithi113.stdout:2/143: symlink d8/l25 0 2024-04-12T05:08:52.817 INFO:tasks.workunit.client.0.smithi113.stdout:1/230: rename d10/d14/l1c to d10/d41/l42 0 2024-04-12T05:08:52.819 INFO:tasks.workunit.client.0.smithi113.stdout:8/279: truncate d9/d16/d1c/f2c 89671 0 2024-04-12T05:08:52.820 INFO:tasks.workunit.client.0.smithi113.stdout:8/280: readlink d9/l4c 0 2024-04-12T05:08:52.820 INFO:tasks.workunit.client.1.smithi192.stdout:4/106: dread - d1/f18 zero size 2024-04-12T05:08:52.821 INFO:tasks.workunit.client.0.smithi113.stdout:2/144: chown d8/d9/de/l21 32126406 1 2024-04-12T05:08:52.821 INFO:tasks.workunit.client.0.smithi113.stdout:1/231: truncate fd 1750162 0 2024-04-12T05:08:52.822 INFO:tasks.workunit.client.1.smithi192.stdout:6/164: dwrite d3/f2c [0,4194304] 0 2024-04-12T05:08:52.822 INFO:tasks.workunit.client.0.smithi113.stdout:8/281: dread - d9/d16/d3c/f52 zero size 2024-04-12T05:08:52.822 INFO:tasks.workunit.client.1.smithi192.stdout:4/107: write d1/d20/f12 [725502,33471] 0 2024-04-12T05:08:52.824 INFO:tasks.workunit.client.1.smithi192.stdout:4/108: write d1/f11 [583766,112441] 0 2024-04-12T05:08:52.827 INFO:tasks.workunit.client.0.smithi113.stdout:8/282: readlink d9/d16/l58 0 2024-04-12T05:08:52.827 INFO:tasks.workunit.client.0.smithi113.stdout:2/145: creat d8/d9/d1a/f26 x:0 0 0 2024-04-12T05:08:52.827 INFO:tasks.workunit.client.1.smithi192.stdout:1/101: unlink d0/d2/dd/f17 0 2024-04-12T05:08:52.830 INFO:tasks.workunit.client.0.smithi113.stdout:1/232: dread d10/d14/f1f [0,4194304] 0 2024-04-12T05:08:52.831 INFO:tasks.workunit.client.0.smithi113.stdout:8/283: mknod d9/d16/c5c 0 2024-04-12T05:08:52.832 INFO:tasks.workunit.client.0.smithi113.stdout:1/233: chown d10/d2e/d31 5869 1 2024-04-12T05:08:52.835 INFO:tasks.workunit.client.1.smithi192.stdout:4/109: rename d1/da/c19 to d1/d20/c25 0 2024-04-12T05:08:52.835 INFO:tasks.workunit.client.1.smithi192.stdout:1/102: dread d0/d2/f9 [0,4194304] 0 2024-04-12T05:08:52.836 INFO:tasks.workunit.client.0.smithi113.stdout:1/234: symlink d10/d2e/d31/d33/l43 0 2024-04-12T05:08:52.837 INFO:tasks.workunit.client.0.smithi113.stdout:2/146: getdents d8/d9/de/d12 0 2024-04-12T05:08:52.838 INFO:tasks.workunit.client.0.smithi113.stdout:1/235: dread - d10/d2e/d31/d33/f2c zero size 2024-04-12T05:08:52.838 INFO:tasks.workunit.client.1.smithi192.stdout:4/110: creat d1/f26 x:0 0 0 2024-04-12T05:08:52.841 INFO:tasks.workunit.client.1.smithi192.stdout:1/103: dread d0/d2/fa [4194304,4194304] 0 2024-04-12T05:08:52.842 INFO:tasks.workunit.client.1.smithi192.stdout:4/111: getdents d1/da/d1c 0 2024-04-12T05:08:52.847 INFO:tasks.workunit.client.1.smithi192.stdout:4/112: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:52.847 INFO:tasks.workunit.client.1.smithi192.stdout:1/104: dread d0/d2/f7 [0,4194304] 0 2024-04-12T05:08:52.850 INFO:tasks.workunit.client.1.smithi192.stdout:6/165: fsync d3/d8/f1f 0 2024-04-12T05:08:52.854 INFO:tasks.workunit.client.1.smithi192.stdout:1/105: rename d0/d2/f9 to d0/f20 0 2024-04-12T05:08:52.854 INFO:tasks.workunit.client.0.smithi113.stdout:8/284: truncate d9/d16/d1c/f2c 801719 0 2024-04-12T05:08:52.854 INFO:tasks.workunit.client.0.smithi113.stdout:1/236: link fd d10/d21/f44 0 2024-04-12T05:08:52.855 INFO:tasks.workunit.client.1.smithi192.stdout:6/166: mknod d3/d8/c3d 0 2024-04-12T05:08:52.856 INFO:tasks.workunit.client.0.smithi113.stdout:1/237: write d10/d21/f2b [3161492,12894] 0 2024-04-12T05:08:52.857 INFO:tasks.workunit.client.0.smithi113.stdout:1/238: read d10/d21/f29 [1735903,114568] 0 2024-04-12T05:08:52.858 INFO:tasks.workunit.client.0.smithi113.stdout:8/285: getdents d9/d16 0 2024-04-12T05:08:52.859 INFO:tasks.workunit.client.1.smithi192.stdout:4/113: dwrite d1/f26 [0,4194304] 0 2024-04-12T05:08:52.861 INFO:tasks.workunit.client.1.smithi192.stdout:1/106: dread d0/f3 [0,4194304] 0 2024-04-12T05:08:52.861 INFO:tasks.workunit.client.0.smithi113.stdout:1/239: creat d10/d41/f45 x:0 0 0 2024-04-12T05:08:52.862 INFO:tasks.workunit.client.1.smithi192.stdout:4/114: write d1/f11 [3132041,56951] 0 2024-04-12T05:08:52.863 INFO:tasks.workunit.client.0.smithi113.stdout:8/286: unlink d9/d16/d1e/d26/d32/l33 0 2024-04-12T05:08:52.865 INFO:tasks.workunit.client.1.smithi192.stdout:6/167: dread d3/d8/f25 [0,4194304] 0 2024-04-12T05:08:52.866 INFO:tasks.workunit.client.1.smithi192.stdout:4/115: rename d1/f15 to d1/da/d1c/f27 0 2024-04-12T05:08:52.866 INFO:tasks.workunit.client.1.smithi192.stdout:1/107: write d0/f20 [3272415,92190] 0 2024-04-12T05:08:52.866 INFO:tasks.workunit.client.0.smithi113.stdout:8/287: symlink d9/d16/d1e/d26/d32/l5d 0 2024-04-12T05:08:52.868 INFO:tasks.workunit.client.1.smithi192.stdout:6/168: dread - d3/d8/d22/f2e zero size 2024-04-12T05:08:52.869 INFO:tasks.workunit.client.1.smithi192.stdout:6/169: chown d3/l1b 466417 1 2024-04-12T05:08:52.869 INFO:tasks.workunit.client.1.smithi192.stdout:4/116: write d1/f2 [3612016,40120] 0 2024-04-12T05:08:52.870 INFO:tasks.workunit.client.1.smithi192.stdout:4/117: write d1/d20/f12 [1236941,64875] 0 2024-04-12T05:08:52.871 INFO:tasks.workunit.client.1.smithi192.stdout:1/108: mkdir d0/d2/dd/d21 0 2024-04-12T05:08:52.872 INFO:tasks.workunit.client.1.smithi192.stdout:1/109: stat d0/d2/cb 0 2024-04-12T05:08:52.873 INFO:tasks.workunit.client.0.smithi113.stdout:8/288: link d9/fd d9/d16/d1e/d3a/f5e 0 2024-04-12T05:08:52.873 INFO:tasks.workunit.client.1.smithi192.stdout:6/170: symlink d3/d8/l3e 0 2024-04-12T05:08:52.876 INFO:tasks.workunit.client.0.smithi113.stdout:8/289: fdatasync d9/d16/d1e/d3a/f5e 0 2024-04-12T05:08:52.876 INFO:tasks.workunit.client.1.smithi192.stdout:4/118: write d1/da/d1c/f27 [1493584,10248] 0 2024-04-12T05:08:52.878 INFO:tasks.workunit.client.1.smithi192.stdout:6/171: read d3/d5/f6 [38776,56600] 0 2024-04-12T05:08:52.881 INFO:tasks.workunit.client.0.smithi113.stdout:1/240: write d10/f1e [563106,24315] 0 2024-04-12T05:08:52.881 INFO:tasks.workunit.client.1.smithi192.stdout:6/172: symlink d3/d5/l3f 0 2024-04-12T05:08:52.883 INFO:tasks.workunit.client.1.smithi192.stdout:1/110: dwrite d0/f20 [0,4194304] 0 2024-04-12T05:08:52.884 INFO:tasks.workunit.client.0.smithi113.stdout:1/241: unlink d10/d21/l37 0 2024-04-12T05:08:52.885 INFO:tasks.workunit.client.0.smithi113.stdout:1/242: read d10/d21/f2b [916155,128603] 0 2024-04-12T05:08:52.885 INFO:tasks.workunit.client.1.smithi192.stdout:6/173: creat d3/d21/f40 x:0 0 0 2024-04-12T05:08:52.885 INFO:tasks.workunit.client.1.smithi192.stdout:4/119: dread d1/f26 [0,4194304] 0 2024-04-12T05:08:52.885 INFO:tasks.workunit.client.1.smithi192.stdout:1/111: truncate d0/d2/dd/f1d 1104515 0 2024-04-12T05:08:52.888 INFO:tasks.workunit.client.1.smithi192.stdout:6/174: creat d3/d23/f41 x:0 0 0 2024-04-12T05:08:52.890 INFO:tasks.workunit.client.1.smithi192.stdout:1/112: dread d0/d2/dd/f1d [0,4194304] 0 2024-04-12T05:08:52.890 INFO:tasks.workunit.client.0.smithi113.stdout:8/290: dwrite d9/d16/d1c/f2d [0,4194304] 0 2024-04-12T05:08:52.890 INFO:tasks.workunit.client.0.smithi113.stdout:1/243: unlink d10/d21/f44 0 2024-04-12T05:08:52.891 INFO:tasks.workunit.client.0.smithi113.stdout:1/244: stat d10/l11 0 2024-04-12T05:08:52.891 INFO:tasks.workunit.client.1.smithi192.stdout:1/113: chown d0/d1b/f13 1725 1 2024-04-12T05:08:52.894 INFO:tasks.workunit.client.1.smithi192.stdout:4/120: link d1/f26 d1/da/d1c/f28 0 2024-04-12T05:08:52.896 INFO:tasks.workunit.client.1.smithi192.stdout:1/114: symlink d0/l22 0 2024-04-12T05:08:52.896 INFO:tasks.workunit.client.0.smithi113.stdout:7/203: sync 2024-04-12T05:08:52.896 INFO:tasks.workunit.client.0.smithi113.stdout:6/212: sync 2024-04-12T05:08:52.896 INFO:tasks.workunit.client.0.smithi113.stdout:3/190: sync 2024-04-12T05:08:52.897 INFO:tasks.workunit.client.1.smithi192.stdout:1/115: dread - d0/f18 zero size 2024-04-12T05:08:52.897 INFO:tasks.workunit.client.1.smithi192.stdout:4/121: creat d1/f29 x:0 0 0 2024-04-12T05:08:52.897 INFO:tasks.workunit.client.0.smithi113.stdout:1/245: dread d10/d21/f2b [0,4194304] 0 2024-04-12T05:08:52.897 INFO:tasks.workunit.client.0.smithi113.stdout:3/191: fsync d0/d13/f21 0 2024-04-12T05:08:52.899 INFO:tasks.workunit.client.1.smithi192.stdout:6/175: dread d3/d8/f17 [4194304,4194304] 0 2024-04-12T05:08:52.901 INFO:tasks.workunit.client.1.smithi192.stdout:4/122: fdatasync d1/da/d1c/f28 0 2024-04-12T05:08:52.901 INFO:tasks.workunit.client.1.smithi192.stdout:6/176: read - d3/dc/f3b zero size 2024-04-12T05:08:52.901 INFO:tasks.workunit.client.0.smithi113.stdout:6/213: symlink d8/de/d17/d25/d31/l44 0 2024-04-12T05:08:52.902 INFO:tasks.workunit.client.0.smithi113.stdout:1/246: mknod d10/c46 0 2024-04-12T05:08:52.902 INFO:tasks.workunit.client.1.smithi192.stdout:6/177: truncate d3/d23/f41 523473 0 2024-04-12T05:08:52.904 INFO:tasks.workunit.client.0.smithi113.stdout:1/247: fsync d10/d21/f29 0 2024-04-12T05:08:52.904 INFO:tasks.workunit.client.0.smithi113.stdout:6/214: mkdir d8/d45 0 2024-04-12T05:08:52.905 INFO:tasks.workunit.client.1.smithi192.stdout:1/116: link d0/f3 d0/d2/d1e/f23 0 2024-04-12T05:08:52.908 INFO:tasks.workunit.client.1.smithi192.stdout:1/117: dread - d0/f18 zero size 2024-04-12T05:08:52.909 INFO:tasks.workunit.client.0.smithi113.stdout:6/215: rmdir d8/de/d13 39 2024-04-12T05:08:52.912 INFO:tasks.workunit.client.0.smithi113.stdout:7/204: dwrite d0/dc/d6/d1a/f2b [0,4194304] 0 2024-04-12T05:08:52.913 INFO:tasks.workunit.client.1.smithi192.stdout:4/123: dwrite d1/f29 [0,4194304] 0 2024-04-12T05:08:52.915 INFO:tasks.workunit.client.0.smithi113.stdout:3/192: dwrite d0/d13/d15/f39 [0,4194304] 0 2024-04-12T05:08:52.918 INFO:tasks.workunit.client.1.smithi192.stdout:1/118: write d0/d2/dd/f1d [1684264,53079] 0 2024-04-12T05:08:52.918 INFO:tasks.workunit.client.1.smithi192.stdout:6/178: dwrite d3/fd [4194304,4194304] 0 2024-04-12T05:08:52.919 INFO:tasks.workunit.client.0.smithi113.stdout:6/216: mknod d8/de/c46 0 2024-04-12T05:08:52.920 INFO:tasks.workunit.client.1.smithi192.stdout:4/124: symlink d1/da/l2a 0 2024-04-12T05:08:52.925 INFO:tasks.workunit.client.0.smithi113.stdout:1/248: dwrite fa [0,4194304] 0 2024-04-12T05:08:52.926 INFO:tasks.workunit.client.1.smithi192.stdout:6/179: truncate d3/dc/f11 1261538 0 2024-04-12T05:08:52.926 INFO:tasks.workunit.client.0.smithi113.stdout:7/205: dwrite d0/f33 [4194304,4194304] 0 2024-04-12T05:08:52.928 INFO:tasks.workunit.client.1.smithi192.stdout:6/180: chown d3/d8 14 1 2024-04-12T05:08:52.928 INFO:tasks.workunit.client.0.smithi113.stdout:6/217: unlink d8/de/d17/d25/d26/l2a 0 2024-04-12T05:08:52.929 INFO:tasks.workunit.client.0.smithi113.stdout:3/193: write d0/d13/d15/d28/d2a/d2d/f35 [496583,126817] 0 2024-04-12T05:08:52.933 INFO:tasks.workunit.client.1.smithi192.stdout:1/119: dwrite d0/f1a [0,4194304] 0 2024-04-12T05:08:52.933 INFO:tasks.workunit.client.1.smithi192.stdout:6/181: symlink d3/d21/d24/l42 0 2024-04-12T05:08:52.935 INFO:tasks.workunit.client.1.smithi192.stdout:4/125: dwrite d1/d20/f12 [0,4194304] 0 2024-04-12T05:08:52.935 INFO:tasks.workunit.client.0.smithi113.stdout:3/194: rename d0/f29 to d0/d13/f42 0 2024-04-12T05:08:52.936 INFO:tasks.workunit.client.0.smithi113.stdout:3/195: write d0/f30 [918843,119073] 0 2024-04-12T05:08:52.944 INFO:tasks.workunit.client.0.smithi113.stdout:3/196: dwrite d0/f30 [0,4194304] 0 2024-04-12T05:08:52.946 INFO:tasks.workunit.client.0.smithi113.stdout:6/218: creat d8/de/d17/d25/d31/f47 x:0 0 0 2024-04-12T05:08:52.946 INFO:tasks.workunit.client.0.smithi113.stdout:7/206: creat d0/dc/f41 x:0 0 0 2024-04-12T05:08:52.947 INFO:tasks.workunit.client.1.smithi192.stdout:6/182: unlink d3/dc/f11 0 2024-04-12T05:08:52.951 INFO:tasks.workunit.client.1.smithi192.stdout:6/183: truncate d3/d23/f41 1367267 0 2024-04-12T05:08:52.954 INFO:tasks.workunit.client.1.smithi192.stdout:4/126: dread d1/da/d1c/f28 [0,4194304] 0 2024-04-12T05:08:52.954 INFO:tasks.workunit.client.0.smithi113.stdout:7/207: rename d0/dc/f27 to d0/dc/f42 0 2024-04-12T05:08:52.955 INFO:tasks.workunit.client.1.smithi192.stdout:6/184: creat d3/f43 x:0 0 0 2024-04-12T05:08:52.958 INFO:tasks.workunit.client.1.smithi192.stdout:1/120: dwrite d0/d1b/f13 [0,4194304] 0 2024-04-12T05:08:52.959 INFO:tasks.workunit.client.1.smithi192.stdout:6/185: symlink d3/d21/d24/l44 0 2024-04-12T05:08:52.960 INFO:tasks.workunit.client.1.smithi192.stdout:6/186: readlink d3/d26/l29 0 2024-04-12T05:08:52.961 INFO:tasks.workunit.client.0.smithi113.stdout:6/219: dwrite d8/de/f19 [4194304,4194304] 0 2024-04-12T05:08:52.963 INFO:tasks.workunit.client.0.smithi113.stdout:6/220: stat d8/de/c1f 0 2024-04-12T05:08:52.965 INFO:tasks.workunit.client.1.smithi192.stdout:6/187: fdatasync d3/d8/f13 0 2024-04-12T05:08:52.967 INFO:tasks.workunit.client.1.smithi192.stdout:1/121: creat d0/f24 x:0 0 0 2024-04-12T05:08:52.968 INFO:tasks.workunit.client.0.smithi113.stdout:6/221: rmdir d8/de/d17/d25 39 2024-04-12T05:08:52.968 INFO:tasks.workunit.client.1.smithi192.stdout:4/127: dwrite d1/f11 [0,4194304] 0 2024-04-12T05:08:52.968 INFO:tasks.workunit.client.0.smithi113.stdout:7/208: dwrite d0/d26/f31 [0,4194304] 0 2024-04-12T05:08:52.969 INFO:tasks.workunit.client.0.smithi113.stdout:6/222: readlink l4 0 2024-04-12T05:08:52.975 INFO:tasks.workunit.client.0.smithi113.stdout:7/209: symlink d0/dc/d6/l43 0 2024-04-12T05:08:52.979 INFO:tasks.workunit.client.1.smithi192.stdout:1/122: rename d0/d2/d1e/f23 to d0/d2/dd/f25 0 2024-04-12T05:08:52.979 INFO:tasks.workunit.client.1.smithi192.stdout:4/128: rename d1/d20/f12 to d1/d20/f2b 0 2024-04-12T05:08:52.980 INFO:tasks.workunit.client.0.smithi113.stdout:1/249: dread fb [0,4194304] 0 2024-04-12T05:08:52.980 INFO:tasks.workunit.client.0.smithi113.stdout:3/197: dwrite d0/fa [0,4194304] 0 2024-04-12T05:08:52.987 INFO:tasks.workunit.client.1.smithi192.stdout:1/123: rename d0/d2/fa to d0/d2/dd/f26 0 2024-04-12T05:08:52.987 INFO:tasks.workunit.client.1.smithi192.stdout:1/124: chown d0/d2/dd/lf 30689 1 2024-04-12T05:08:52.989 INFO:tasks.workunit.client.0.smithi113.stdout:7/210: dread d0/dc/d6/f28 [0,4194304] 0 2024-04-12T05:08:52.989 INFO:tasks.workunit.client.0.smithi113.stdout:1/250: fsync d10/f32 0 2024-04-12T05:08:52.989 INFO:tasks.workunit.client.0.smithi113.stdout:6/223: getdents d8/de/d13/d11 0 2024-04-12T05:08:52.990 INFO:tasks.workunit.client.0.smithi113.stdout:3/198: rmdir d0/d13/d15/d28 39 2024-04-12T05:08:52.991 INFO:tasks.workunit.client.0.smithi113.stdout:3/199: fsync d0/d13/d15/f20 0 2024-04-12T05:08:52.993 INFO:tasks.workunit.client.1.smithi192.stdout:1/125: unlink d0/d2/f1f 0 2024-04-12T05:08:52.994 INFO:tasks.workunit.client.0.smithi113.stdout:6/224: symlink d8/de/d17/d25/d31/l48 0 2024-04-12T05:08:52.994 INFO:tasks.workunit.client.1.smithi192.stdout:1/126: fdatasync d0/fc 0 2024-04-12T05:08:53.000 INFO:tasks.workunit.client.0.smithi113.stdout:3/200: unlink d0/d13/f21 0 2024-04-12T05:08:53.001 INFO:tasks.workunit.client.0.smithi113.stdout:6/225: read d8/de/f19 [5546099,434] 0 2024-04-12T05:08:53.001 INFO:tasks.workunit.client.0.smithi113.stdout:3/201: write d0/f30 [3013064,62090] 0 2024-04-12T05:08:53.003 INFO:tasks.workunit.client.1.smithi192.stdout:1/127: dwrite d0/f24 [0,4194304] 0 2024-04-12T05:08:53.008 INFO:tasks.workunit.client.1.smithi192.stdout:1/128: symlink d0/d1b/l27 0 2024-04-12T05:08:53.009 INFO:tasks.workunit.client.1.smithi192.stdout:6/188: dwrite d3/d8/f17 [0,4194304] 0 2024-04-12T05:08:53.010 INFO:tasks.workunit.client.1.smithi192.stdout:1/129: read d0/d2/f7 [1250350,96199] 0 2024-04-12T05:08:53.010 INFO:tasks.workunit.client.0.smithi113.stdout:3/202: rename d0/d13/d15/d28/f41 to d0/d13/d15/d28/d2a/d34/f43 0 2024-04-12T05:08:53.011 INFO:tasks.workunit.client.1.smithi192.stdout:6/189: chown d3/d8/d22/f27 1303507 1 2024-04-12T05:08:53.012 INFO:tasks.workunit.client.1.smithi192.stdout:1/130: rmdir d0/d1b 39 2024-04-12T05:08:53.012 INFO:tasks.workunit.client.0.smithi113.stdout:7/211: truncate d0/dc/d6/d1a/f34 3162966 0 2024-04-12T05:08:53.014 INFO:tasks.workunit.client.1.smithi192.stdout:6/190: creat d3/d5/f45 x:0 0 0 2024-04-12T05:08:53.014 INFO:tasks.workunit.client.1.smithi192.stdout:4/129: fsync d1/f11 0 2024-04-12T05:08:53.016 INFO:tasks.workunit.client.0.smithi113.stdout:7/212: rmdir d0/dd 39 2024-04-12T05:08:53.016 INFO:tasks.workunit.client.0.smithi113.stdout:1/251: dwrite f2 [0,4194304] 0 2024-04-12T05:08:53.017 INFO:tasks.workunit.client.1.smithi192.stdout:4/130: symlink d1/l2c 0 2024-04-12T05:08:53.017 INFO:tasks.workunit.client.1.smithi192.stdout:1/131: dread d0/f20 [0,4194304] 0 2024-04-12T05:08:53.018 INFO:tasks.workunit.client.0.smithi113.stdout:6/226: dwrite d8/de/f19 [0,4194304] 0 2024-04-12T05:08:53.019 INFO:tasks.workunit.client.0.smithi113.stdout:3/203: truncate d0/d13/d15/f20 823445 0 2024-04-12T05:08:53.019 INFO:tasks.workunit.client.0.smithi113.stdout:7/213: chown d0/d26/c2f 1043967985 1 2024-04-12T05:08:53.019 INFO:tasks.workunit.client.1.smithi192.stdout:6/191: rename d3/f2c to d3/d26/f46 0 2024-04-12T05:08:53.020 INFO:tasks.workunit.client.1.smithi192.stdout:6/192: chown d3/dc/d39/f3a 26285 1 2024-04-12T05:08:53.021 INFO:tasks.workunit.client.0.smithi113.stdout:3/204: chown d0/d13/d15/l33 437 1 2024-04-12T05:08:53.021 INFO:tasks.workunit.client.1.smithi192.stdout:4/131: creat d1/d1d/f2d x:0 0 0 2024-04-12T05:08:53.023 INFO:tasks.workunit.client.0.smithi113.stdout:6/227: creat d8/d45/f49 x:0 0 0 2024-04-12T05:08:53.023 INFO:tasks.workunit.client.1.smithi192.stdout:4/132: read d1/f2 [159430,112572] 0 2024-04-12T05:08:53.023 INFO:tasks.workunit.client.1.smithi192.stdout:1/132: write d0/f20 [4100788,24486] 0 2024-04-12T05:08:53.024 INFO:tasks.workunit.client.0.smithi113.stdout:6/228: read - d8/f40 zero size 2024-04-12T05:08:53.024 INFO:tasks.workunit.client.1.smithi192.stdout:6/193: mknod d3/dc/d39/c47 0 2024-04-12T05:08:53.024 INFO:tasks.workunit.client.0.smithi113.stdout:7/214: mkdir d0/dd/d11/d44 0 2024-04-12T05:08:53.025 INFO:tasks.workunit.client.0.smithi113.stdout:7/215: fsync d0/dc/d6/d1a/f3a 0 2024-04-12T05:08:53.026 INFO:tasks.workunit.client.1.smithi192.stdout:1/133: fdatasync d0/fc 0 2024-04-12T05:08:53.026 INFO:tasks.workunit.client.1.smithi192.stdout:6/194: stat d3/d5/f6 0 2024-04-12T05:08:53.028 INFO:tasks.workunit.client.1.smithi192.stdout:4/133: truncate d1/d1d/f2d 693405 0 2024-04-12T05:08:53.028 INFO:tasks.workunit.client.1.smithi192.stdout:6/195: write d3/d5/f38 [896992,64206] 0 2024-04-12T05:08:53.031 INFO:tasks.workunit.client.0.smithi113.stdout:1/252: rename fa to d10/d21/f47 0 2024-04-12T05:08:53.031 INFO:tasks.workunit.client.0.smithi113.stdout:3/205: mknod d0/d13/d15/d28/c44 0 2024-04-12T05:08:53.032 INFO:tasks.workunit.client.0.smithi113.stdout:6/229: symlink d8/de/d13/l4a 0 2024-04-12T05:08:53.035 INFO:tasks.workunit.client.0.smithi113.stdout:1/253: creat d10/d21/d3c/f48 x:0 0 0 2024-04-12T05:08:53.037 INFO:tasks.workunit.client.1.smithi192.stdout:4/134: dwrite d1/d1d/f2d [0,4194304] 0 2024-04-12T05:08:53.037 INFO:tasks.workunit.client.0.smithi113.stdout:6/230: symlink d8/de/d17/d25/d26/l4b 0 2024-04-12T05:08:53.038 INFO:tasks.workunit.client.1.smithi192.stdout:4/135: write d1/f29 [4197264,102386] 0 2024-04-12T05:08:53.042 INFO:tasks.workunit.client.0.smithi113.stdout:6/231: dread d8/f22 [0,4194304] 0 2024-04-12T05:08:53.044 INFO:tasks.workunit.client.1.smithi192.stdout:4/136: rename d1/d20/f2b to d1/f2e 0 2024-04-12T05:08:53.046 INFO:tasks.workunit.client.1.smithi192.stdout:4/137: write d1/f18 [647252,126467] 0 2024-04-12T05:08:53.046 INFO:tasks.workunit.client.1.smithi192.stdout:4/138: stat d1/da 0 2024-04-12T05:08:53.047 INFO:tasks.workunit.client.1.smithi192.stdout:4/139: read d1/d1d/f2d [1686433,117949] 0 2024-04-12T05:08:53.047 INFO:tasks.workunit.client.0.smithi113.stdout:7/216: truncate d0/dc/d6/d1a/f29 1134720 0 2024-04-12T05:08:53.050 INFO:tasks.workunit.client.0.smithi113.stdout:1/254: truncate ff 2890922 0 2024-04-12T05:08:53.051 INFO:tasks.workunit.client.0.smithi113.stdout:7/217: creat d0/d26/f45 x:0 0 0 2024-04-12T05:08:53.052 INFO:tasks.workunit.client.0.smithi113.stdout:7/218: dread - d0/dd/d1b/f23 zero size 2024-04-12T05:08:53.052 INFO:tasks.workunit.client.0.smithi113.stdout:1/255: rmdir d10/d41 39 2024-04-12T05:08:53.054 INFO:tasks.workunit.client.1.smithi192.stdout:4/140: getdents d1/d1d 0 2024-04-12T05:08:53.057 INFO:tasks.workunit.client.1.smithi192.stdout:4/141: write d1/f11 [3011321,98178] 0 2024-04-12T05:08:53.062 INFO:tasks.workunit.client.0.smithi113.stdout:7/219: dwrite d0/dd/f30 [0,4194304] 0 2024-04-12T05:08:53.062 INFO:tasks.workunit.client.0.smithi113.stdout:6/232: dread d8/f12 [0,4194304] 0 2024-04-12T05:08:53.062 INFO:tasks.workunit.client.1.smithi192.stdout:4/142: dread d1/da/d1c/f28 [0,4194304] 0 2024-04-12T05:08:53.063 INFO:tasks.workunit.client.1.smithi192.stdout:4/143: stat d1/da/d1c/f28 0 2024-04-12T05:08:53.064 INFO:tasks.workunit.client.1.smithi192.stdout:4/144: write d1/da/d1c/f28 [4496469,11960] 0 2024-04-12T05:08:53.064 INFO:tasks.workunit.client.0.smithi113.stdout:1/256: dwrite d10/d14/f2a [0,4194304] 0 2024-04-12T05:08:53.128 INFO:tasks.workunit.client.1.smithi192.stdout:8/67: sync 2024-04-12T05:08:53.131 INFO:tasks.workunit.client.1.smithi192.stdout:8/68: rename d6/df/c12 to d6/df/c14 0 2024-04-12T05:08:53.135 INFO:tasks.workunit.client.1.smithi192.stdout:8/69: unlink d6/df/c14 0 2024-04-12T05:08:53.136 INFO:tasks.workunit.client.0.smithi113.stdout:7/220: write d0/dc/f42 [4777642,37692] 0 2024-04-12T05:08:53.136 INFO:tasks.workunit.client.0.smithi113.stdout:1/257: dread - d10/d41/f45 zero size 2024-04-12T05:08:53.136 INFO:tasks.workunit.client.0.smithi113.stdout:5/254: sync 2024-04-12T05:08:53.137 INFO:tasks.workunit.client.0.smithi113.stdout:9/298: sync 2024-04-12T05:08:53.137 INFO:tasks.workunit.client.0.smithi113.stdout:4/255: sync 2024-04-12T05:08:53.137 INFO:tasks.workunit.client.1.smithi192.stdout:8/70: fsync f5 0 2024-04-12T05:08:53.138 INFO:tasks.workunit.client.1.smithi192.stdout:8/71: truncate d6/d10/f13 931051 0 2024-04-12T05:08:53.138 INFO:tasks.workunit.client.0.smithi113.stdout:6/233: rename d8/de/d17/d25 to d8/de/d13/d11/d36/d3d/d4c 0 2024-04-12T05:08:53.141 INFO:tasks.workunit.client.0.smithi113.stdout:5/255: dread d0/d10/d27/d13/f28 [0,4194304] 0 2024-04-12T05:08:53.143 INFO:tasks.workunit.client.1.smithi192.stdout:8/72: truncate d6/da/fd 4458906 0 2024-04-12T05:08:53.143 INFO:tasks.workunit.client.0.smithi113.stdout:4/256: rename da/de/d17 to da/d4f 0 2024-04-12T05:08:53.146 INFO:tasks.workunit.client.1.smithi192.stdout:8/73: dread d6/f9 [4194304,4194304] 0 2024-04-12T05:08:53.147 INFO:tasks.workunit.client.0.smithi113.stdout:5/256: creat d0/d10/d27/d1d/f49 x:0 0 0 2024-04-12T05:08:53.147 INFO:tasks.workunit.client.0.smithi113.stdout:1/258: dwrite f2 [0,4194304] 0 2024-04-12T05:08:53.148 INFO:tasks.workunit.client.0.smithi113.stdout:9/299: dwrite d1/d4/d5/d6/d15/f3f [0,4194304] 0 2024-04-12T05:08:53.148 INFO:tasks.workunit.client.0.smithi113.stdout:7/221: symlink d0/dc/d6/d1a/l46 0 2024-04-12T05:08:53.149 INFO:tasks.workunit.client.1.smithi192.stdout:8/74: creat d6/df/f15 x:0 0 0 2024-04-12T05:08:53.151 INFO:tasks.workunit.client.0.smithi113.stdout:5/257: creat d0/d10/d27/d33/f4a x:0 0 0 2024-04-12T05:08:53.152 INFO:tasks.workunit.client.0.smithi113.stdout:5/258: chown d0/d10/d27/d1d/d38/d44/f48 3615529 1 2024-04-12T05:08:53.153 INFO:tasks.workunit.client.0.smithi113.stdout:5/259: write d0/d10/d27/d13/f23 [1874836,12330] 0 2024-04-12T05:08:53.153 INFO:tasks.workunit.client.1.smithi192.stdout:8/75: dread f3 [0,4194304] 0 2024-04-12T05:08:53.155 INFO:tasks.workunit.client.0.smithi113.stdout:9/300: rename d1/d4/l39 to d1/d4/l68 0 2024-04-12T05:08:53.156 INFO:tasks.workunit.client.1.smithi192.stdout:8/76: creat d6/df/f16 x:0 0 0 2024-04-12T05:08:53.156 INFO:tasks.workunit.client.0.smithi113.stdout:9/301: dread - d1/d4/d5/d6/f3b zero size 2024-04-12T05:08:53.158 INFO:tasks.workunit.client.0.smithi113.stdout:9/302: write d1/d4/d5/d6/d15/f3f [3492185,94134] 0 2024-04-12T05:08:53.159 INFO:tasks.workunit.client.0.smithi113.stdout:7/222: creat d0/dd/d11/d44/f47 x:0 0 0 2024-04-12T05:08:53.160 INFO:tasks.workunit.client.1.smithi192.stdout:8/77: symlink d6/l17 0 2024-04-12T05:08:53.160 INFO:tasks.workunit.client.0.smithi113.stdout:9/303: read d1/fc [847025,16323] 0 2024-04-12T05:08:53.161 INFO:tasks.workunit.client.1.smithi192.stdout:8/78: write f3 [2493714,41152] 0 2024-04-12T05:08:53.162 INFO:tasks.workunit.client.0.smithi113.stdout:9/304: fsync d1/d4/d5/f12 0 2024-04-12T05:08:53.165 INFO:tasks.workunit.client.0.smithi113.stdout:7/223: dread d0/dc/f42 [0,4194304] 0 2024-04-12T05:08:53.165 INFO:tasks.workunit.client.0.smithi113.stdout:9/305: mknod d1/d4/d5/d17/c69 0 2024-04-12T05:08:53.166 INFO:tasks.workunit.client.1.smithi192.stdout:8/79: creat d6/f18 x:0 0 0 2024-04-12T05:08:53.166 INFO:tasks.workunit.client.1.smithi192.stdout:8/80: chown d6/l17 0 1 2024-04-12T05:08:53.169 INFO:tasks.workunit.client.0.smithi113.stdout:9/306: mknod d1/d4/d5/d6/d15/c6a 0 2024-04-12T05:08:53.173 INFO:tasks.workunit.client.0.smithi113.stdout:7/224: creat d0/f48 x:0 0 0 2024-04-12T05:08:53.175 INFO:tasks.workunit.client.1.smithi192.stdout:8/81: write d6/da/fd [5280789,128321] 0 2024-04-12T05:08:53.177 INFO:tasks.workunit.client.0.smithi113.stdout:1/259: dwrite ff [0,4194304] 0 2024-04-12T05:08:53.177 INFO:tasks.workunit.client.0.smithi113.stdout:7/225: write d0/dd/d1b/f25 [498397,63451] 0 2024-04-12T05:08:53.180 INFO:tasks.workunit.client.1.smithi192.stdout:8/82: rename d6/d10/f13 to d6/df/f19 0 2024-04-12T05:08:53.180 INFO:tasks.workunit.client.0.smithi113.stdout:1/260: dread - d10/d2e/d31/d33/f2c zero size 2024-04-12T05:08:53.180 INFO:tasks.workunit.client.0.smithi113.stdout:9/307: dwrite d1/f26 [0,4194304] 0 2024-04-12T05:08:53.182 INFO:tasks.workunit.client.1.smithi192.stdout:8/83: mknod d6/df/c1a 0 2024-04-12T05:08:53.183 INFO:tasks.workunit.client.0.smithi113.stdout:4/257: dread da/dd/d1b/f28 [0,4194304] 0 2024-04-12T05:08:53.184 INFO:tasks.workunit.client.0.smithi113.stdout:1/261: chown d10/d21/c39 0 1 2024-04-12T05:08:53.185 INFO:tasks.workunit.client.0.smithi113.stdout:9/308: creat d1/df/d4b/d63/d3d/d53/f6b x:0 0 0 2024-04-12T05:08:53.187 INFO:tasks.workunit.client.0.smithi113.stdout:7/226: link d0/d26/f45 d0/f49 0 2024-04-12T05:08:53.188 INFO:tasks.workunit.client.0.smithi113.stdout:4/258: rmdir da/dd/df/d1d 39 2024-04-12T05:08:53.189 INFO:tasks.workunit.client.0.smithi113.stdout:4/259: stat da/c27 0 2024-04-12T05:08:53.190 INFO:tasks.workunit.client.0.smithi113.stdout:1/262: rmdir d10/d2e 39 2024-04-12T05:08:53.193 INFO:tasks.workunit.client.1.smithi192.stdout:8/84: dwrite d6/da/fc [0,4194304] 0 2024-04-12T05:08:53.194 INFO:tasks.workunit.client.0.smithi113.stdout:1/263: dread - d10/d2e/d31/d33/f2c zero size 2024-04-12T05:08:53.194 INFO:tasks.workunit.client.1.smithi192.stdout:8/85: write d6/df/f16 [1011631,52341] 0 2024-04-12T05:08:53.195 INFO:tasks.workunit.client.0.smithi113.stdout:1/264: chown d10/d2e/l3f 14337 1 2024-04-12T05:08:53.195 INFO:tasks.workunit.client.0.smithi113.stdout:4/260: mknod da/dd/df/d1d/c50 0 2024-04-12T05:08:53.195 INFO:tasks.workunit.client.0.smithi113.stdout:1/265: chown f2 2379 1 2024-04-12T05:08:53.197 INFO:tasks.workunit.client.0.smithi113.stdout:7/227: dread d0/dc/d6/f16 [0,4194304] 0 2024-04-12T05:08:53.202 INFO:tasks.workunit.client.0.smithi113.stdout:1/266: dread d10/d14/f2a [0,4194304] 0 2024-04-12T05:08:53.202 INFO:tasks.workunit.client.0.smithi113.stdout:7/228: getdents d0/dd/d1b 0 2024-04-12T05:08:53.203 INFO:tasks.workunit.client.0.smithi113.stdout:1/267: chown f5 19 1 2024-04-12T05:08:53.204 INFO:tasks.workunit.client.0.smithi113.stdout:4/261: dwrite da/d4f/f4e [0,4194304] 0 2024-04-12T05:08:53.206 INFO:tasks.workunit.client.1.smithi192.stdout:8/86: dread d6/df/f16 [0,4194304] 0 2024-04-12T05:08:53.209 INFO:tasks.workunit.client.1.smithi192.stdout:3/75: sync 2024-04-12T05:08:53.209 INFO:tasks.workunit.client.1.smithi192.stdout:9/82: sync 2024-04-12T05:08:53.209 INFO:tasks.workunit.client.1.smithi192.stdout:5/169: sync 2024-04-12T05:08:53.209 INFO:tasks.workunit.client.1.smithi192.stdout:8/87: unlink d6/df/c1a 0 2024-04-12T05:08:53.211 INFO:tasks.workunit.client.1.smithi192.stdout:8/88: truncate d6/f18 248917 0 2024-04-12T05:08:53.211 INFO:tasks.workunit.client.0.smithi113.stdout:7/229: link d0/dc/d6/d1a/f3e d0/dd/d11/f4a 0 2024-04-12T05:08:53.212 INFO:tasks.workunit.client.1.smithi192.stdout:9/83: mkdir d11/d1d 0 2024-04-12T05:08:53.213 INFO:tasks.workunit.client.1.smithi192.stdout:9/84: dread - d11/f12 zero size 2024-04-12T05:08:53.213 INFO:tasks.workunit.client.1.smithi192.stdout:8/89: truncate d6/df/f19 1449396 0 2024-04-12T05:08:53.214 INFO:tasks.workunit.client.0.smithi113.stdout:4/262: mknod da/d26/c51 0 2024-04-12T05:08:53.215 INFO:tasks.workunit.client.1.smithi192.stdout:5/170: mknod d0/d15/d24/d29/c40 0 2024-04-12T05:08:53.217 INFO:tasks.workunit.client.0.smithi113.stdout:7/230: symlink d0/dd/d11/d44/l4b 0 2024-04-12T05:08:53.218 INFO:tasks.workunit.client.1.smithi192.stdout:9/85: mkdir d11/d15/d1e 0 2024-04-12T05:08:53.218 INFO:tasks.workunit.client.0.smithi113.stdout:9/309: dwrite d1/fc [0,4194304] 0 2024-04-12T05:08:53.219 INFO:tasks.workunit.client.1.smithi192.stdout:8/90: symlink d6/df/l1b 0 2024-04-12T05:08:53.219 INFO:tasks.workunit.client.0.smithi113.stdout:4/263: creat da/d24/f52 x:0 0 0 2024-04-12T05:08:53.222 INFO:tasks.workunit.client.1.smithi192.stdout:8/91: creat d6/df/f1c x:0 0 0 2024-04-12T05:08:53.222 INFO:tasks.workunit.client.0.smithi113.stdout:1/268: dwrite d10/d14/f1f [4194304,4194304] 0 2024-04-12T05:08:53.223 INFO:tasks.workunit.client.1.smithi192.stdout:8/92: chown d6/df/f15 230011 1 2024-04-12T05:08:53.225 INFO:tasks.workunit.client.1.smithi192.stdout:9/86: dread fb [0,4194304] 0 2024-04-12T05:08:53.228 INFO:tasks.workunit.client.0.smithi113.stdout:7/231: rename d0/dd/cf to d0/dd/d1b/c4c 0 2024-04-12T05:08:53.228 INFO:tasks.workunit.client.0.smithi113.stdout:4/264: getdents da/d4a 0 2024-04-12T05:08:53.228 INFO:tasks.workunit.client.0.smithi113.stdout:7/232: read d0/dd/f30 [4155566,48542] 0 2024-04-12T05:08:53.228 INFO:tasks.workunit.client.0.smithi113.stdout:1/269: creat d10/d21/d23/f49 x:0 0 0 2024-04-12T05:08:53.230 INFO:tasks.workunit.client.1.smithi192.stdout:9/87: mkdir d11/d14/d1f 0 2024-04-12T05:08:53.230 INFO:tasks.workunit.client.0.smithi113.stdout:9/310: rename d1/df/d4b/d63/d3d/d53 to d1/df/d4b/d6c 0 2024-04-12T05:08:53.230 INFO:tasks.workunit.client.1.smithi192.stdout:3/76: truncate d5/f19 995256 0 2024-04-12T05:08:53.231 INFO:tasks.workunit.client.1.smithi192.stdout:9/88: write d11/d14/f1a [728387,45908] 0 2024-04-12T05:08:53.233 INFO:tasks.workunit.client.1.smithi192.stdout:3/77: truncate d5/d13/f18 639457 0 2024-04-12T05:08:53.234 INFO:tasks.workunit.client.1.smithi192.stdout:9/89: fdatasync fb 0 2024-04-12T05:08:53.238 INFO:tasks.workunit.client.0.smithi113.stdout:9/311: dwrite d1/d4/d5/d6/d15/f3f [0,4194304] 0 2024-04-12T05:08:53.239 INFO:tasks.workunit.client.1.smithi192.stdout:3/78: link d5/lc d5/d13/d1a/l1d 0 2024-04-12T05:08:53.240 INFO:tasks.workunit.client.0.smithi113.stdout:7/233: rename d0/dc/l3 to d0/d26/l4d 0 2024-04-12T05:08:53.240 INFO:tasks.workunit.client.1.smithi192.stdout:5/171: dwrite d0/f13 [0,4194304] 0 2024-04-12T05:08:53.240 INFO:tasks.workunit.client.1.smithi192.stdout:3/79: dread - d5/f14 zero size 2024-04-12T05:08:53.241 INFO:tasks.workunit.client.0.smithi113.stdout:7/234: chown d0/dc/d6/c15 9900913 1 2024-04-12T05:08:53.242 INFO:tasks.workunit.client.1.smithi192.stdout:9/90: rename d11/d15/l19 to d11/d1d/l20 0 2024-04-12T05:08:53.244 INFO:tasks.workunit.client.1.smithi192.stdout:5/172: unlink d0/l39 0 2024-04-12T05:08:53.246 INFO:tasks.workunit.client.0.smithi113.stdout:9/312: dwrite d1/d4/d38/f62 [0,4194304] 0 2024-04-12T05:08:53.248 INFO:tasks.workunit.client.0.smithi113.stdout:1/270: truncate f2 2349928 0 2024-04-12T05:08:53.250 INFO:tasks.workunit.client.0.smithi113.stdout:1/271: chown d10/d21/d3c 658848639 1 2024-04-12T05:08:53.252 INFO:tasks.workunit.client.0.smithi113.stdout:9/313: mkdir d1/d4/d6d 0 2024-04-12T05:08:53.252 INFO:tasks.workunit.client.0.smithi113.stdout:7/235: write d0/dd/d11/f4a [598371,4133] 0 2024-04-12T05:08:53.256 INFO:tasks.workunit.client.0.smithi113.stdout:1/272: creat d10/d41/f4a x:0 0 0 2024-04-12T05:08:53.257 INFO:tasks.workunit.client.0.smithi113.stdout:7/236: write d0/dd/d11/f19 [227186,7222] 0 2024-04-12T05:08:53.258 INFO:tasks.workunit.client.0.smithi113.stdout:9/314: symlink d1/d4/d30/l6e 0 2024-04-12T05:08:53.259 INFO:tasks.workunit.client.0.smithi113.stdout:7/237: fsync d0/dd/f1c 0 2024-04-12T05:08:53.261 INFO:tasks.workunit.client.0.smithi113.stdout:7/238: symlink d0/dd/d1b/l4e 0 2024-04-12T05:08:53.261 INFO:tasks.workunit.client.1.smithi192.stdout:3/80: getdents d5/d13/d1a 0 2024-04-12T05:08:53.264 INFO:tasks.workunit.client.1.smithi192.stdout:9/91: dread f3 [0,4194304] 0 2024-04-12T05:08:53.269 INFO:tasks.workunit.client.0.smithi113.stdout:1/273: write d10/d21/f2b [24890,43747] 0 2024-04-12T05:08:53.270 INFO:tasks.workunit.client.0.smithi113.stdout:9/315: truncate d1/d27/f5d 3716069 0 2024-04-12T05:08:53.270 INFO:tasks.workunit.client.1.smithi192.stdout:3/81: dwrite d5/f14 [0,4194304] 0 2024-04-12T05:08:53.271 INFO:tasks.workunit.client.0.smithi113.stdout:7/239: write d0/d26/f45 [625024,123878] 0 2024-04-12T05:08:53.273 INFO:tasks.workunit.client.1.smithi192.stdout:3/82: mkdir d5/d1e 0 2024-04-12T05:08:53.274 INFO:tasks.workunit.client.0.smithi113.stdout:1/274: link d10/l3a d10/d2e/l4b 0 2024-04-12T05:08:53.279 INFO:tasks.workunit.client.1.smithi192.stdout:9/92: dwrite f7 [0,4194304] 0 2024-04-12T05:08:53.279 INFO:tasks.workunit.client.0.smithi113.stdout:9/316: dwrite d1/d4/d5/f58 [0,4194304] 0 2024-04-12T05:08:53.280 INFO:tasks.workunit.client.0.smithi113.stdout:1/275: symlink d10/l4c 0 2024-04-12T05:08:53.280 INFO:tasks.workunit.client.1.smithi192.stdout:3/83: rename d5/c15 to d5/d13/d1a/c1f 0 2024-04-12T05:08:53.282 INFO:tasks.workunit.client.0.smithi113.stdout:9/317: fdatasync d1/fc 0 2024-04-12T05:08:53.282 INFO:tasks.workunit.client.0.smithi113.stdout:7/240: dwrite d0/dc/f18 [4194304,4194304] 0 2024-04-12T05:08:53.283 INFO:tasks.workunit.client.0.smithi113.stdout:1/276: write d10/d21/f29 [586684,68709] 0 2024-04-12T05:08:53.283 INFO:tasks.workunit.client.0.smithi113.stdout:2/147: sync 2024-04-12T05:08:53.283 INFO:tasks.workunit.client.0.smithi113.stdout:8/291: sync 2024-04-12T05:08:53.283 INFO:tasks.workunit.client.0.smithi113.stdout:0/269: sync 2024-04-12T05:08:53.285 INFO:tasks.workunit.client.1.smithi192.stdout:3/84: getdents d5/d13 0 2024-04-12T05:08:53.288 INFO:tasks.workunit.client.0.smithi113.stdout:1/277: dread fe [0,4194304] 0 2024-04-12T05:08:53.289 INFO:tasks.workunit.client.1.smithi192.stdout:3/85: mkdir d5/d13/d20 0 2024-04-12T05:08:53.291 INFO:tasks.workunit.client.1.smithi192.stdout:3/86: symlink d5/d13/d1a/l21 0 2024-04-12T05:08:53.294 INFO:tasks.workunit.client.1.smithi192.stdout:3/87: dread d5/f19 [0,4194304] 0 2024-04-12T05:08:53.295 INFO:tasks.workunit.client.0.smithi113.stdout:1/278: creat d10/d14/f4d x:0 0 0 2024-04-12T05:08:53.295 INFO:tasks.workunit.client.0.smithi113.stdout:8/292: chown d9/d1b 574354 1 2024-04-12T05:08:53.296 INFO:tasks.workunit.client.1.smithi192.stdout:3/88: stat d5/f1c 0 2024-04-12T05:08:53.296 INFO:tasks.workunit.client.0.smithi113.stdout:1/279: readlink d10/d21/l25 0 2024-04-12T05:08:53.297 INFO:tasks.workunit.client.0.smithi113.stdout:7/241: chown d0/f12 13685 1 2024-04-12T05:08:53.297 INFO:tasks.workunit.client.0.smithi113.stdout:8/293: dread - d9/d1b/f45 zero size 2024-04-12T05:08:53.300 INFO:tasks.workunit.client.0.smithi113.stdout:8/294: write d9/fd [4420553,80279] 0 2024-04-12T05:08:53.301 INFO:tasks.workunit.client.0.smithi113.stdout:2/148: mkdir d8/d9/d27 0 2024-04-12T05:08:53.302 INFO:tasks.workunit.client.0.smithi113.stdout:2/149: fdatasync d8/d9/de/f22 0 2024-04-12T05:08:53.303 INFO:tasks.workunit.client.0.smithi113.stdout:8/295: chown d9/d16/d1e/c3d 213739992 1 2024-04-12T05:08:53.305 INFO:tasks.workunit.client.1.smithi192.stdout:3/89: write d5/f19 [1093538,27949] 0 2024-04-12T05:08:53.305 INFO:tasks.workunit.client.0.smithi113.stdout:0/270: creat d1/d6/d16/f4c x:0 0 0 2024-04-12T05:08:53.306 INFO:tasks.workunit.client.1.smithi192.stdout:3/90: rmdir d5/d13 39 2024-04-12T05:08:53.307 INFO:tasks.workunit.client.0.smithi113.stdout:9/318: dwrite d1/f46 [4194304,4194304] 0 2024-04-12T05:08:53.309 INFO:tasks.workunit.client.0.smithi113.stdout:9/319: write d1/df/f4d [1044216,101812] 0 2024-04-12T05:08:53.309 INFO:tasks.workunit.client.1.smithi192.stdout:2/115: sync 2024-04-12T05:08:53.309 INFO:tasks.workunit.client.1.smithi192.stdout:0/66: sync 2024-04-12T05:08:53.310 INFO:tasks.workunit.client.1.smithi192.stdout:4/145: sync 2024-04-12T05:08:53.310 INFO:tasks.workunit.client.1.smithi192.stdout:6/196: sync 2024-04-12T05:08:53.310 INFO:tasks.workunit.client.1.smithi192.stdout:1/134: sync 2024-04-12T05:08:53.310 INFO:tasks.workunit.client.1.smithi192.stdout:7/114: sync 2024-04-12T05:08:53.310 INFO:tasks.workunit.client.0.smithi113.stdout:0/271: creat d1/d6/da/f4d x:0 0 0 2024-04-12T05:08:53.312 INFO:tasks.workunit.client.1.smithi192.stdout:2/116: write d1/f25 [417575,17577] 0 2024-04-12T05:08:53.314 INFO:tasks.workunit.client.0.smithi113.stdout:8/296: rmdir d9/d16/d5a 0 2024-04-12T05:08:53.314 INFO:tasks.workunit.client.1.smithi192.stdout:3/91: rename d5/lc to d5/d13/d20/l22 0 2024-04-12T05:08:53.314 INFO:tasks.workunit.client.0.smithi113.stdout:9/320: mkdir d1/df/d4b/d63/d3d/d6f 0 2024-04-12T05:08:53.316 INFO:tasks.workunit.client.0.smithi113.stdout:0/272: rename d1/c42 to d1/d6/d16/d2a/d2c/c4e 0 2024-04-12T05:08:53.317 INFO:tasks.workunit.client.1.smithi192.stdout:6/197: mkdir d3/d8/d48 0 2024-04-12T05:08:53.317 INFO:tasks.workunit.client.1.smithi192.stdout:1/135: dread d0/d2/dd/f1d [0,4194304] 0 2024-04-12T05:08:53.318 INFO:tasks.workunit.client.0.smithi113.stdout:7/242: fsync d0/dc/f18 0 2024-04-12T05:08:53.318 INFO:tasks.workunit.client.1.smithi192.stdout:2/117: symlink d1/d8/db/de/l28 0 2024-04-12T05:08:53.319 INFO:tasks.workunit.client.0.smithi113.stdout:8/297: mknod d9/d16/d1e/d26/d32/d5b/c5f 0 2024-04-12T05:08:53.319 INFO:tasks.workunit.client.0.smithi113.stdout:7/243: write d0/f12 [243340,33970] 0 2024-04-12T05:08:53.319 INFO:tasks.workunit.client.1.smithi192.stdout:4/146: write d1/f2 [1713584,17796] 0 2024-04-12T05:08:53.320 INFO:tasks.workunit.client.0.smithi113.stdout:7/244: fsync d0/dd/d1b/f23 0 2024-04-12T05:08:53.321 INFO:tasks.workunit.client.0.smithi113.stdout:9/321: mknod d1/df/d4b/d63/d3d/d6f/c70 0 2024-04-12T05:08:53.322 INFO:tasks.workunit.client.0.smithi113.stdout:8/298: dread d9/d16/d1e/d26/d32/f3f [0,4194304] 0 2024-04-12T05:08:53.322 INFO:tasks.workunit.client.1.smithi192.stdout:7/115: truncate d3/d11/f18 471208 0 2024-04-12T05:08:53.322 INFO:tasks.workunit.client.1.smithi192.stdout:1/136: readlink d0/l22 0 2024-04-12T05:08:53.323 INFO:tasks.workunit.client.0.smithi113.stdout:9/322: readlink d1/d4/d38/d65/l1d 0 2024-04-12T05:08:53.324 INFO:tasks.workunit.client.0.smithi113.stdout:0/273: symlink d1/d6/l4f 0 2024-04-12T05:08:53.324 INFO:tasks.workunit.client.1.smithi192.stdout:1/137: fsync d0/fc 0 2024-04-12T05:08:53.325 INFO:tasks.workunit.client.1.smithi192.stdout:6/198: write d3/dc/d39/f3a [984444,93298] 0 2024-04-12T05:08:53.325 INFO:tasks.workunit.client.1.smithi192.stdout:7/116: creat d3/d7/d15/f29 x:0 0 0 2024-04-12T05:08:53.326 INFO:tasks.workunit.client.0.smithi113.stdout:8/299: write d9/d1b/f45 [325840,126605] 0 2024-04-12T05:08:53.327 INFO:tasks.workunit.client.0.smithi113.stdout:9/323: write d1/fc [5420761,82607] 0 2024-04-12T05:08:53.328 INFO:tasks.workunit.client.1.smithi192.stdout:2/118: creat d1/d8/f29 x:0 0 0 2024-04-12T05:08:53.329 INFO:tasks.workunit.client.1.smithi192.stdout:2/119: stat d1/d8/c12 0 2024-04-12T05:08:53.331 INFO:tasks.workunit.client.0.smithi113.stdout:0/274: read d1/d6/d16/f47 [500033,44693] 0 2024-04-12T05:08:53.332 INFO:tasks.workunit.client.0.smithi113.stdout:9/324: readlink d1/d4/d30/l5e 0 2024-04-12T05:08:53.332 INFO:tasks.workunit.client.0.smithi113.stdout:8/300: readlink d9/d16/d1e/l2b 0 2024-04-12T05:08:53.333 INFO:tasks.workunit.client.1.smithi192.stdout:2/120: creat d1/d8/d17/f2a x:0 0 0 2024-04-12T05:08:53.333 INFO:tasks.workunit.client.1.smithi192.stdout:4/147: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:53.334 INFO:tasks.workunit.client.1.smithi192.stdout:6/199: dread d3/d8/f13 [0,4194304] 0 2024-04-12T05:08:53.335 INFO:tasks.workunit.client.1.smithi192.stdout:3/92: unlink d5/d13/d20/l22 0 2024-04-12T05:08:53.337 INFO:tasks.workunit.client.0.smithi113.stdout:0/275: mkdir d1/d6/d16/d2a/d50 0 2024-04-12T05:08:53.337 INFO:tasks.workunit.client.0.smithi113.stdout:7/245: dwrite d0/dc/f42 [4194304,4194304] 0 2024-04-12T05:08:53.338 INFO:tasks.workunit.client.1.smithi192.stdout:1/138: dwrite d0/d2/dd/f26 [4194304,4194304] 0 2024-04-12T05:08:53.338 INFO:tasks.workunit.client.1.smithi192.stdout:0/67: truncate d0/f8 3059891 0 2024-04-12T05:08:53.339 INFO:tasks.workunit.client.1.smithi192.stdout:1/139: write d0/d2/dd/f26 [3036755,115298] 0 2024-04-12T05:08:53.340 INFO:tasks.workunit.client.1.smithi192.stdout:4/148: creat d1/da/f2f x:0 0 0 2024-04-12T05:08:53.341 INFO:tasks.workunit.client.1.smithi192.stdout:6/200: symlink d3/d8/d22/l49 0 2024-04-12T05:08:53.342 INFO:tasks.workunit.client.0.smithi113.stdout:8/301: creat d9/d16/d1e/d26/d32/f60 x:0 0 0 2024-04-12T05:08:53.342 INFO:tasks.workunit.client.1.smithi192.stdout:6/201: read - d3/d26/f3c zero size 2024-04-12T05:08:53.345 INFO:tasks.workunit.client.1.smithi192.stdout:2/121: mknod d1/d8/c2b 0 2024-04-12T05:08:53.347 INFO:tasks.workunit.client.0.smithi113.stdout:7/246: read - d0/dd/d11/d44/f47 zero size 2024-04-12T05:08:53.347 INFO:tasks.workunit.client.0.smithi113.stdout:2/150: dwrite d8/d16/f17 [0,4194304] 0 2024-04-12T05:08:53.347 INFO:tasks.workunit.client.1.smithi192.stdout:2/122: truncate d1/f25 1029281 0 2024-04-12T05:08:53.348 INFO:tasks.workunit.client.1.smithi192.stdout:1/140: write d0/d1b/f13 [3376575,35712] 0 2024-04-12T05:08:53.351 INFO:tasks.workunit.client.1.smithi192.stdout:7/117: dwrite d3/d7/d15/f24 [0,4194304] 0 2024-04-12T05:08:53.353 INFO:tasks.workunit.client.1.smithi192.stdout:6/202: rmdir d3/d23 39 2024-04-12T05:08:53.356 INFO:tasks.workunit.client.1.smithi192.stdout:1/141: mkdir d0/d1b/d28 0 2024-04-12T05:08:53.356 INFO:tasks.workunit.client.1.smithi192.stdout:2/123: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:53.357 INFO:tasks.workunit.client.1.smithi192.stdout:1/142: fdatasync d0/d2/dd/f25 0 2024-04-12T05:08:53.357 INFO:tasks.workunit.client.0.smithi113.stdout:1/280: dwrite f2 [0,4194304] 0 2024-04-12T05:08:53.358 INFO:tasks.workunit.client.0.smithi113.stdout:7/247: dread d0/dc/d6/d1a/f3e [0,4194304] 0 2024-04-12T05:08:53.358 INFO:tasks.workunit.client.0.smithi113.stdout:7/248: write d0/d26/f45 [556240,97989] 0 2024-04-12T05:08:53.358 INFO:tasks.workunit.client.1.smithi192.stdout:2/124: fdatasync d1/d8/fc 0 2024-04-12T05:08:53.361 INFO:tasks.workunit.client.1.smithi192.stdout:7/118: rmdir d3/d7/d14/d1f 39 2024-04-12T05:08:53.362 INFO:tasks.workunit.client.0.smithi113.stdout:0/276: truncate d1/f22 5950406 0 2024-04-12T05:08:53.362 INFO:tasks.workunit.client.1.smithi192.stdout:3/93: rename f2 to d5/f23 0 2024-04-12T05:08:53.362 INFO:tasks.workunit.client.1.smithi192.stdout:6/203: symlink d3/d5/l4a 0 2024-04-12T05:08:53.363 INFO:tasks.workunit.client.0.smithi113.stdout:9/325: dwrite d1/df/d4b/d6c/f6b [0,4194304] 0 2024-04-12T05:08:53.363 INFO:tasks.workunit.client.0.smithi113.stdout:2/151: creat d8/d9/d27/f28 x:0 0 0 2024-04-12T05:08:53.364 INFO:tasks.workunit.client.0.smithi113.stdout:1/281: creat d10/d2e/f4e x:0 0 0 2024-04-12T05:08:53.365 INFO:tasks.workunit.client.1.smithi192.stdout:6/204: dread - d3/f43 zero size 2024-04-12T05:08:53.367 INFO:tasks.workunit.client.1.smithi192.stdout:3/94: truncate d5/d13/f17 902901 0 2024-04-12T05:08:53.369 INFO:tasks.workunit.client.1.smithi192.stdout:1/143: dwrite d0/d2/dd/f15 [0,4194304] 0 2024-04-12T05:08:53.370 INFO:tasks.workunit.client.1.smithi192.stdout:2/125: mknod d1/d8/db/c2c 0 2024-04-12T05:08:53.370 INFO:tasks.workunit.client.0.smithi113.stdout:2/152: write d8/fd [1009477,121847] 0 2024-04-12T05:08:53.371 INFO:tasks.workunit.client.1.smithi192.stdout:1/144: write d0/f1a [4131845,78710] 0 2024-04-12T05:08:53.372 INFO:tasks.workunit.client.0.smithi113.stdout:1/282: readlink d10/d14/l1b 0 2024-04-12T05:08:53.374 INFO:tasks.workunit.client.0.smithi113.stdout:2/153: chown d8/d16/f1d 0 1 2024-04-12T05:08:53.374 INFO:tasks.workunit.client.1.smithi192.stdout:0/68: rename d0/l4 to d0/d6/l12 0 2024-04-12T05:08:53.376 INFO:tasks.workunit.client.0.smithi113.stdout:0/277: symlink d1/d6/d16/d2a/l51 0 2024-04-12T05:08:53.376 INFO:tasks.workunit.client.1.smithi192.stdout:6/205: creat d3/d26/f4b x:0 0 0 2024-04-12T05:08:53.380 INFO:tasks.workunit.client.1.smithi192.stdout:3/95: truncate d5/f1c 348005 0 2024-04-12T05:08:53.381 INFO:tasks.workunit.client.1.smithi192.stdout:2/126: mkdir d1/d2d 0 2024-04-12T05:08:53.383 INFO:tasks.workunit.client.1.smithi192.stdout:2/127: chown d1/d2d 6 1 2024-04-12T05:08:53.384 INFO:tasks.workunit.client.1.smithi192.stdout:2/128: chown d1/f25 19839 1 2024-04-12T05:08:53.384 INFO:tasks.workunit.client.1.smithi192.stdout:0/69: creat d0/d6/f13 x:0 0 0 2024-04-12T05:08:53.385 INFO:tasks.workunit.client.1.smithi192.stdout:2/129: read d1/f4 [2543578,50900] 0 2024-04-12T05:08:53.386 INFO:tasks.workunit.client.0.smithi113.stdout:0/278: unlink d1/d43/c48 0 2024-04-12T05:08:53.387 INFO:tasks.workunit.client.1.smithi192.stdout:1/145: mkdir d0/d29 0 2024-04-12T05:08:53.387 INFO:tasks.workunit.client.1.smithi192.stdout:2/130: write d1/d8/db/de/f11 [4256191,41675] 0 2024-04-12T05:08:53.388 INFO:tasks.workunit.client.0.smithi113.stdout:0/279: chown d1/c31 32 1 2024-04-12T05:08:53.389 INFO:tasks.workunit.client.0.smithi113.stdout:0/280: write d1/d6/f30 [585167,117247] 0 2024-04-12T05:08:53.389 INFO:tasks.workunit.client.0.smithi113.stdout:7/249: dwrite d0/dc/d6/d1a/f3e [0,4194304] 0 2024-04-12T05:08:53.390 INFO:tasks.workunit.client.0.smithi113.stdout:2/154: getdents d8/d9/de/d12 0 2024-04-12T05:08:53.390 INFO:tasks.workunit.client.1.smithi192.stdout:4/149: truncate d1/da/d1c/f28 3357124 0 2024-04-12T05:08:53.391 INFO:tasks.workunit.client.0.smithi113.stdout:9/326: link d1/d4/d30/l6e d1/d4/d5/d17/l71 0 2024-04-12T05:08:53.391 INFO:tasks.workunit.client.0.smithi113.stdout:7/250: chown d0/d26 704 1 2024-04-12T05:08:53.391 INFO:tasks.workunit.client.0.smithi113.stdout:2/155: readlink d8/d9/l13 0 2024-04-12T05:08:53.391 INFO:tasks.workunit.client.0.smithi113.stdout:0/281: readlink d1/d6/da/l12 0 2024-04-12T05:08:53.393 INFO:tasks.workunit.client.1.smithi192.stdout:3/96: link d5/d13/f18 d5/d13/d20/f24 0 2024-04-12T05:08:53.393 INFO:tasks.workunit.client.0.smithi113.stdout:2/156: dread - d8/d9/d1a/f26 zero size 2024-04-12T05:08:53.395 INFO:tasks.workunit.client.1.smithi192.stdout:0/70: rename d0/d6/cf to d0/d6/c14 0 2024-04-12T05:08:53.395 INFO:tasks.workunit.client.0.smithi113.stdout:1/283: rename d10/d21/c26 to d10/c4f 0 2024-04-12T05:08:53.395 INFO:tasks.workunit.client.1.smithi192.stdout:3/97: dread d5/d13/f17 [0,4194304] 0 2024-04-12T05:08:53.396 INFO:tasks.workunit.client.0.smithi113.stdout:1/284: chown d10/d2e/d31/d33/l43 233392508 1 2024-04-12T05:08:53.397 INFO:tasks.workunit.client.0.smithi113.stdout:0/282: chown d1/d6/d16/d2a/c46 168330 1 2024-04-12T05:08:53.398 INFO:tasks.workunit.client.0.smithi113.stdout:7/251: mknod d0/c4f 0 2024-04-12T05:08:53.398 INFO:tasks.workunit.client.0.smithi113.stdout:0/283: fsync d1/d6/f30 0 2024-04-12T05:08:53.398 INFO:tasks.workunit.client.1.smithi192.stdout:2/131: creat d1/d8/db/f2e x:0 0 0 2024-04-12T05:08:53.399 INFO:tasks.workunit.client.0.smithi113.stdout:9/327: symlink d1/d4/d5/l72 0 2024-04-12T05:08:53.399 INFO:tasks.workunit.client.1.smithi192.stdout:4/150: creat d1/d20/f30 x:0 0 0 2024-04-12T05:08:53.401 INFO:tasks.workunit.client.0.smithi113.stdout:1/285: dread d10/d14/f17 [0,4194304] 0 2024-04-12T05:08:53.401 INFO:tasks.workunit.client.0.smithi113.stdout:2/157: rename d8/d9/de/d12/f1c to d8/d9/de/d15/f29 0 2024-04-12T05:08:53.402 INFO:tasks.workunit.client.0.smithi113.stdout:1/286: chown d10/d21/f2b 0 1 2024-04-12T05:08:53.402 INFO:tasks.workunit.client.1.smithi192.stdout:0/71: mkdir d0/d6/d15 0 2024-04-12T05:08:53.402 INFO:tasks.workunit.client.1.smithi192.stdout:4/151: write d1/f29 [2165578,113396] 0 2024-04-12T05:08:53.403 INFO:tasks.workunit.client.0.smithi113.stdout:7/252: mkdir d0/dc/d50 0 2024-04-12T05:08:53.406 INFO:tasks.workunit.client.1.smithi192.stdout:1/146: dwrite d0/d2/dd/f1d [0,4194304] 0 2024-04-12T05:08:53.409 INFO:tasks.workunit.client.1.smithi192.stdout:3/98: chown d5/d13/d1a 1 1 2024-04-12T05:08:53.409 INFO:tasks.workunit.client.0.smithi113.stdout:1/287: creat d10/d2e/d31/d33/f50 x:0 0 0 2024-04-12T05:08:53.410 INFO:tasks.workunit.client.0.smithi113.stdout:0/284: mkdir d1/d6/d16/d2a/d50/d52 0 2024-04-12T05:08:53.410 INFO:tasks.workunit.client.1.smithi192.stdout:2/132: creat d1/d8/db/de/f2f x:0 0 0 2024-04-12T05:08:53.412 INFO:tasks.workunit.client.1.smithi192.stdout:1/147: write d0/fc [1765141,102794] 0 2024-04-12T05:08:53.412 INFO:tasks.workunit.client.0.smithi113.stdout:0/285: dread - d1/d9/f29 zero size 2024-04-12T05:08:53.413 INFO:tasks.workunit.client.0.smithi113.stdout:7/253: dwrite d0/d26/f38 [0,4194304] 0 2024-04-12T05:08:53.413 INFO:tasks.workunit.client.0.smithi113.stdout:9/328: rmdir d1/d4/d5/d6/d54 0 2024-04-12T05:08:53.415 INFO:tasks.workunit.client.1.smithi192.stdout:6/206: truncate d3/d8/d22/f27 2322638 0 2024-04-12T05:08:53.417 INFO:tasks.workunit.client.1.smithi192.stdout:4/152: dwrite d1/da/d1c/f27 [0,4194304] 0 2024-04-12T05:08:53.417 INFO:tasks.workunit.client.1.smithi192.stdout:1/148: write d0/d2/dd/f25 [4224348,100082] 0 2024-04-12T05:08:53.419 INFO:tasks.workunit.client.1.smithi192.stdout:4/153: stat d1/da/cf 0 2024-04-12T05:08:53.421 INFO:tasks.workunit.client.0.smithi113.stdout:1/288: dread d10/d21/d23/f36 [0,4194304] 0 2024-04-12T05:08:53.422 INFO:tasks.workunit.client.1.smithi192.stdout:2/133: creat d1/d8/db/de/f30 x:0 0 0 2024-04-12T05:08:53.424 INFO:tasks.workunit.client.0.smithi113.stdout:2/158: getdents d8/d9/d1a 0 2024-04-12T05:08:53.425 INFO:tasks.workunit.client.0.smithi113.stdout:7/254: dread d0/dc/f42 [4194304,4194304] 0 2024-04-12T05:08:53.425 INFO:tasks.workunit.client.0.smithi113.stdout:9/329: symlink d1/d4/d30/l73 0 2024-04-12T05:08:53.428 INFO:tasks.workunit.client.1.smithi192.stdout:6/207: dread d3/d5/f6 [0,4194304] 0 2024-04-12T05:08:53.429 INFO:tasks.workunit.client.1.smithi192.stdout:4/154: creat d1/da/f31 x:0 0 0 2024-04-12T05:08:53.429 INFO:tasks.workunit.client.0.smithi113.stdout:9/330: read d1/df/d4b/d6c/f6b [3468809,55454] 0 2024-04-12T05:08:53.430 INFO:tasks.workunit.client.0.smithi113.stdout:1/289: symlink d10/d21/l51 0 2024-04-12T05:08:53.432 INFO:tasks.workunit.client.1.smithi192.stdout:3/99: rmdir d5/d1e 0 2024-04-12T05:08:53.435 INFO:tasks.workunit.client.1.smithi192.stdout:1/149: dwrite d0/d2/dd/f26 [8388608,4194304] 0 2024-04-12T05:08:53.438 INFO:tasks.workunit.client.0.smithi113.stdout:9/331: creat d1/d4/d5/d17/f74 x:0 0 0 2024-04-12T05:08:53.440 INFO:tasks.workunit.client.1.smithi192.stdout:2/134: dwrite d1/d8/db/f2e [0,4194304] 0 2024-04-12T05:08:53.440 INFO:tasks.workunit.client.1.smithi192.stdout:4/155: mknod d1/da/d1c/c32 0 2024-04-12T05:08:53.441 INFO:tasks.workunit.client.1.smithi192.stdout:1/150: readlink d0/d2/dd/lf 0 2024-04-12T05:08:53.442 INFO:tasks.workunit.client.1.smithi192.stdout:0/72: truncate d0/f8 2965909 0 2024-04-12T05:08:53.443 INFO:tasks.workunit.client.0.smithi113.stdout:0/286: dwrite d1/d6/d16/d2a/f3f [0,4194304] 0 2024-04-12T05:08:53.446 INFO:tasks.workunit.client.0.smithi113.stdout:7/255: dwrite d0/dc/d6/d1a/f2b [0,4194304] 0 2024-04-12T05:08:53.447 INFO:tasks.workunit.client.0.smithi113.stdout:2/159: dwrite d8/d16/f1d [4194304,4194304] 0 2024-04-12T05:08:53.447 INFO:tasks.workunit.client.0.smithi113.stdout:9/332: read - d1/df/d4b/d6c/f59 zero size 2024-04-12T05:08:53.448 INFO:tasks.workunit.client.1.smithi192.stdout:6/208: mkdir d3/d8/d10/d4c 0 2024-04-12T05:08:53.448 INFO:tasks.workunit.client.1.smithi192.stdout:3/100: rename d5/d13/f16 to d5/d13/d20/f25 0 2024-04-12T05:08:53.450 INFO:tasks.workunit.client.0.smithi113.stdout:2/160: readlink d8/l25 0 2024-04-12T05:08:53.451 INFO:tasks.workunit.client.0.smithi113.stdout:1/290: link d10/d2e/d31/d33/c3e d10/d2e/c52 0 2024-04-12T05:08:53.452 INFO:tasks.workunit.client.1.smithi192.stdout:4/156: creat d1/f33 x:0 0 0 2024-04-12T05:08:53.454 INFO:tasks.workunit.client.1.smithi192.stdout:1/151: getdents d0/d29 0 2024-04-12T05:08:53.454 INFO:tasks.workunit.client.0.smithi113.stdout:2/161: chown f7 58 1 2024-04-12T05:08:53.455 INFO:tasks.workunit.client.1.smithi192.stdout:1/152: read - d0/f18 zero size 2024-04-12T05:08:53.458 INFO:tasks.workunit.client.0.smithi113.stdout:1/291: creat d10/d2e/d31/f53 x:0 0 0 2024-04-12T05:08:53.458 INFO:tasks.workunit.client.0.smithi113.stdout:7/256: creat d0/f51 x:0 0 0 2024-04-12T05:08:53.460 INFO:tasks.workunit.client.1.smithi192.stdout:6/209: rename d3/d5/c36 to d3/d8/c4d 0 2024-04-12T05:08:53.460 INFO:tasks.workunit.client.0.smithi113.stdout:2/162: symlink d8/d9/de/l2a 0 2024-04-12T05:08:53.461 INFO:tasks.workunit.client.0.smithi113.stdout:0/287: dwrite d1/d6/f30 [0,4194304] 0 2024-04-12T05:08:53.465 INFO:tasks.workunit.client.0.smithi113.stdout:1/292: unlink d10/d14/c1a 0 2024-04-12T05:08:53.465 INFO:tasks.workunit.client.0.smithi113.stdout:9/333: link d1/d4/f33 d1/d4/d30/f75 0 2024-04-12T05:08:53.465 INFO:tasks.workunit.client.1.smithi192.stdout:4/157: dwrite d1/da/f31 [0,4194304] 0 2024-04-12T05:08:53.467 INFO:tasks.workunit.client.0.smithi113.stdout:1/293: stat d10/d21/d23/f49 0 2024-04-12T05:08:53.467 INFO:tasks.workunit.client.0.smithi113.stdout:1/294: write d10/d14/f4d [410828,122499] 0 2024-04-12T05:08:53.470 INFO:tasks.workunit.client.0.smithi113.stdout:0/288: read d1/d6/d16/f47 [32322,65601] 0 2024-04-12T05:08:53.471 INFO:tasks.workunit.client.1.smithi192.stdout:2/135: getdents d1/d8/db/de 0 2024-04-12T05:08:53.472 INFO:tasks.workunit.client.0.smithi113.stdout:2/163: mkdir d8/d9/de/d12/d2b 0 2024-04-12T05:08:53.472 INFO:tasks.workunit.client.1.smithi192.stdout:6/210: creat d3/dc/d39/f4e x:0 0 0 2024-04-12T05:08:53.473 INFO:tasks.workunit.client.0.smithi113.stdout:2/164: stat d8/d9/de/d12/d2b 0 2024-04-12T05:08:53.473 INFO:tasks.workunit.client.0.smithi113.stdout:0/289: dread d1/fe [0,4194304] 0 2024-04-12T05:08:53.474 INFO:tasks.workunit.client.0.smithi113.stdout:9/334: creat d1/d4/d5/d6/f76 x:0 0 0 2024-04-12T05:08:53.474 INFO:tasks.workunit.client.1.smithi192.stdout:6/211: fdatasync d3/f1e 0 2024-04-12T05:08:53.476 INFO:tasks.workunit.client.0.smithi113.stdout:9/335: write d1/d4/d5/d6/f3b [591819,83752] 0 2024-04-12T05:08:53.478 INFO:tasks.workunit.client.0.smithi113.stdout:9/336: creat d1/d4/f77 x:0 0 0 2024-04-12T05:08:53.479 INFO:tasks.workunit.client.0.smithi113.stdout:0/290: dread d1/d9/f1a [0,4194304] 0 2024-04-12T05:08:53.479 INFO:tasks.workunit.client.0.smithi113.stdout:9/337: stat d1/df/d4b/d63/d3d/c31 0 2024-04-12T05:08:53.479 INFO:tasks.workunit.client.0.smithi113.stdout:0/291: dread - d1/f3d zero size 2024-04-12T05:08:53.483 INFO:tasks.workunit.client.1.smithi192.stdout:6/212: fdatasync d3/d26/f31 0 2024-04-12T05:08:53.486 INFO:tasks.workunit.client.0.smithi113.stdout:0/292: mknod d1/d6/da/c53 0 2024-04-12T05:08:53.487 INFO:tasks.workunit.client.0.smithi113.stdout:9/338: truncate d1/d27/f5d 415202 0 2024-04-12T05:08:53.489 INFO:tasks.workunit.client.1.smithi192.stdout:4/158: dwrite d1/f11 [0,4194304] 0 2024-04-12T05:08:53.492 INFO:tasks.workunit.client.1.smithi192.stdout:4/159: chown d1/da/cd 3470 1 2024-04-12T05:08:53.492 INFO:tasks.workunit.client.1.smithi192.stdout:2/136: dwrite d1/d8/db/de/f1e [0,4194304] 0 2024-04-12T05:08:53.492 INFO:tasks.workunit.client.1.smithi192.stdout:3/101: truncate d5/f19 258837 0 2024-04-12T05:08:53.493 INFO:tasks.workunit.client.1.smithi192.stdout:3/102: chown d5/le 33 1 2024-04-12T05:08:53.493 INFO:tasks.workunit.client.1.smithi192.stdout:4/160: truncate d1/d20/f30 1030962 0 2024-04-12T05:08:53.541 INFO:tasks.workunit.client.1.smithi192.stdout:2/137: rename d1/f25 to d1/d8/d17/d19/f31 0 2024-04-12T05:08:53.541 INFO:tasks.workunit.client.0.smithi113.stdout:9/339: getdents d1/df 0 2024-04-12T05:08:53.541 INFO:tasks.workunit.client.0.smithi113.stdout:3/206: sync 2024-04-12T05:08:53.541 INFO:tasks.workunit.client.0.smithi113.stdout:6/234: sync 2024-04-12T05:08:53.541 INFO:tasks.workunit.client.0.smithi113.stdout:9/340: chown d1 8747 1 2024-04-12T05:08:53.541 INFO:tasks.workunit.client.1.smithi192.stdout:2/138: dread - d1/d8/db/de/f2f zero size 2024-04-12T05:08:53.543 INFO:tasks.workunit.client.0.smithi113.stdout:7/257: truncate d0/f49 203295 0 2024-04-12T05:08:53.544 INFO:tasks.workunit.client.1.smithi192.stdout:4/161: mknod d1/da/c34 0 2024-04-12T05:08:53.549 INFO:tasks.workunit.client.1.smithi192.stdout:3/103: dwrite d5/fd [0,4194304] 0 2024-04-12T05:08:53.549 INFO:tasks.workunit.client.0.smithi113.stdout:7/258: read - d0/dd/d1b/f23 zero size 2024-04-12T05:08:53.549 INFO:tasks.workunit.client.0.smithi113.stdout:3/207: dread d0/d13/d15/f20 [0,4194304] 0 2024-04-12T05:08:53.549 INFO:tasks.workunit.client.0.smithi113.stdout:6/235: mknod d8/de/d17/c4d 0 2024-04-12T05:08:53.550 INFO:tasks.workunit.client.1.smithi192.stdout:2/139: creat d1/d2d/f32 x:0 0 0 2024-04-12T05:08:53.552 INFO:tasks.workunit.client.1.smithi192.stdout:6/213: truncate d3/d26/f46 904033 0 2024-04-12T05:08:53.553 INFO:tasks.workunit.client.0.smithi113.stdout:7/259: mkdir d0/d52 0 2024-04-12T05:08:53.555 INFO:tasks.workunit.client.1.smithi192.stdout:3/104: fsync d5/d13/d20/f25 0 2024-04-12T05:08:53.556 INFO:tasks.workunit.client.1.smithi192.stdout:4/162: dwrite d1/f18 [0,4194304] 0 2024-04-12T05:08:53.557 INFO:tasks.workunit.client.1.smithi192.stdout:2/140: creat d1/d2d/f33 x:0 0 0 2024-04-12T05:08:53.558 INFO:tasks.workunit.client.0.smithi113.stdout:1/295: dwrite d10/d14/f17 [0,4194304] 0 2024-04-12T05:08:53.559 INFO:tasks.workunit.client.0.smithi113.stdout:7/260: chown d0/dc/d6/d1a/f2b 146593 1 2024-04-12T05:08:53.562 INFO:tasks.workunit.client.0.smithi113.stdout:6/236: creat d8/de/d13/d11/d36/f4e x:0 0 0 2024-04-12T05:08:53.564 INFO:tasks.workunit.client.0.smithi113.stdout:0/293: dwrite d1/d9/f2b [0,4194304] 0 2024-04-12T05:08:53.567 INFO:tasks.workunit.client.1.smithi192.stdout:6/214: fdatasync d3/fd 0 2024-04-12T05:08:53.568 INFO:tasks.workunit.client.1.smithi192.stdout:6/215: chown d3/d21/c32 229 1 2024-04-12T05:08:53.568 INFO:tasks.workunit.client.1.smithi192.stdout:2/141: creat d1/d2d/f34 x:0 0 0 2024-04-12T05:08:53.570 INFO:tasks.workunit.client.0.smithi113.stdout:6/237: write d8/de/d13/fb [2081848,25889] 0 2024-04-12T05:08:53.570 INFO:tasks.workunit.client.0.smithi113.stdout:1/296: dread d10/d21/d23/f36 [0,4194304] 0 2024-04-12T05:08:53.570 INFO:tasks.workunit.client.0.smithi113.stdout:7/261: dread d0/dd/d11/f4a [0,4194304] 0 2024-04-12T05:08:53.570 INFO:tasks.workunit.client.0.smithi113.stdout:7/262: fdatasync d0/dd/f1c 0 2024-04-12T05:08:53.572 INFO:tasks.workunit.client.1.smithi192.stdout:2/142: read - d1/d8/d17/f24 zero size 2024-04-12T05:08:53.573 INFO:tasks.workunit.client.0.smithi113.stdout:3/208: link d0/f1c d0/d13/d15/d28/f45 0 2024-04-12T05:08:53.575 INFO:tasks.workunit.client.0.smithi113.stdout:0/294: rmdir d1/d6/d16/d2a/d50 39 2024-04-12T05:08:53.578 INFO:tasks.workunit.client.0.smithi113.stdout:7/263: getdents d0/d52 0 2024-04-12T05:08:53.580 INFO:tasks.workunit.client.0.smithi113.stdout:1/297: dwrite d10/d41/f4a [0,4194304] 0 2024-04-12T05:08:53.581 INFO:tasks.workunit.client.1.smithi192.stdout:6/216: dread d3/d26/f46 [0,4194304] 0 2024-04-12T05:08:53.581 INFO:tasks.workunit.client.0.smithi113.stdout:6/238: symlink d8/de/l4f 0 2024-04-12T05:08:53.582 INFO:tasks.workunit.client.1.smithi192.stdout:6/217: dread - d3/d26/f28 zero size 2024-04-12T05:08:53.583 INFO:tasks.workunit.client.0.smithi113.stdout:7/264: creat d0/dd/d2d/f53 x:0 0 0 2024-04-12T05:08:53.583 INFO:tasks.workunit.client.0.smithi113.stdout:3/209: dread d0/f30 [0,4194304] 0 2024-04-12T05:08:53.585 INFO:tasks.workunit.client.0.smithi113.stdout:7/265: fdatasync d0/dd/f1c 0 2024-04-12T05:08:53.587 INFO:tasks.workunit.client.0.smithi113.stdout:7/266: readlink d0/dc/d6/l2e 0 2024-04-12T05:08:53.590 INFO:tasks.workunit.client.1.smithi192.stdout:2/143: creat d1/d8/f35 x:0 0 0 2024-04-12T05:08:53.590 INFO:tasks.workunit.client.0.smithi113.stdout:3/210: mknod d0/d13/d15/c46 0 2024-04-12T05:08:53.591 INFO:tasks.workunit.client.1.smithi192.stdout:3/105: dwrite d5/d13/f18 [0,4194304] 0 2024-04-12T05:08:53.592 INFO:tasks.workunit.client.0.smithi113.stdout:7/267: fdatasync d0/dc/d6/f28 0 2024-04-12T05:08:53.592 INFO:tasks.workunit.client.1.smithi192.stdout:6/218: rename d3/fd to d3/d21/d24/f4f 0 2024-04-12T05:08:53.592 INFO:tasks.workunit.client.0.smithi113.stdout:1/298: dwrite d10/d21/f29 [0,4194304] 0 2024-04-12T05:08:53.594 INFO:tasks.workunit.client.0.smithi113.stdout:1/299: write d10/f13 [832615,76172] 0 2024-04-12T05:08:53.594 INFO:tasks.workunit.client.1.smithi192.stdout:2/144: fdatasync d1/f4 0 2024-04-12T05:08:53.597 INFO:tasks.workunit.client.0.smithi113.stdout:1/300: write d10/d21/d23/f36 [398841,49399] 0 2024-04-12T05:08:53.597 INFO:tasks.workunit.client.1.smithi192.stdout:4/163: write d1/f26 [4193941,2484] 0 2024-04-12T05:08:53.597 INFO:tasks.workunit.client.1.smithi192.stdout:6/219: write d3/d8/f1f [858604,80469] 0 2024-04-12T05:08:53.598 INFO:tasks.workunit.client.0.smithi113.stdout:1/301: chown d10/d21/d23/f36 3360 1 2024-04-12T05:08:53.598 INFO:tasks.workunit.client.1.smithi192.stdout:3/106: mkdir d5/d13/d20/d26 0 2024-04-12T05:08:53.601 INFO:tasks.workunit.client.0.smithi113.stdout:1/302: creat d10/d2e/f54 x:0 0 0 2024-04-12T05:08:53.605 INFO:tasks.workunit.client.1.smithi192.stdout:6/220: chown d3/d5/ca 0 1 2024-04-12T05:08:53.610 INFO:tasks.workunit.client.0.smithi113.stdout:1/303: mkdir d10/d55 0 2024-04-12T05:08:53.613 INFO:tasks.workunit.client.1.smithi192.stdout:2/145: dwrite d1/d2d/f33 [0,4194304] 0 2024-04-12T05:08:53.613 INFO:tasks.workunit.client.1.smithi192.stdout:4/164: dread d1/f18 [0,4194304] 0 2024-04-12T05:08:53.614 INFO:tasks.workunit.client.1.smithi192.stdout:6/221: creat d3/d8/d48/f50 x:0 0 0 2024-04-12T05:08:53.614 INFO:tasks.workunit.client.0.smithi113.stdout:7/268: unlink d0/f49 0 2024-04-12T05:08:53.615 INFO:tasks.workunit.client.0.smithi113.stdout:1/304: dwrite d10/d21/f2b [0,4194304] 0 2024-04-12T05:08:53.616 INFO:tasks.workunit.client.0.smithi113.stdout:7/269: chown d0/c22 10 1 2024-04-12T05:08:53.616 INFO:tasks.workunit.client.1.smithi192.stdout:2/146: write d1/d8/d17/f2a [773115,82667] 0 2024-04-12T05:08:53.617 INFO:tasks.workunit.client.1.smithi192.stdout:3/107: dread d5/d13/f18 [0,4194304] 0 2024-04-12T05:08:53.617 INFO:tasks.workunit.client.0.smithi113.stdout:1/305: read d10/d41/f4a [3747911,121587] 0 2024-04-12T05:08:53.618 INFO:tasks.workunit.client.0.smithi113.stdout:6/239: truncate d8/de/d13/fa 1246161 0 2024-04-12T05:08:53.618 INFO:tasks.workunit.client.1.smithi192.stdout:6/222: creat d3/d5/f51 x:0 0 0 2024-04-12T05:08:53.618 INFO:tasks.workunit.client.1.smithi192.stdout:2/147: chown d1/c1c 2253301 1 2024-04-12T05:08:53.618 INFO:tasks.workunit.client.1.smithi192.stdout:4/165: symlink d1/d1d/l35 0 2024-04-12T05:08:53.620 INFO:tasks.workunit.client.1.smithi192.stdout:3/108: write d5/d13/f18 [4572261,56393] 0 2024-04-12T05:08:53.620 INFO:tasks.workunit.client.0.smithi113.stdout:6/240: chown d8/de/d13/d2b/d2e 4 1 2024-04-12T05:08:53.623 INFO:tasks.workunit.client.1.smithi192.stdout:6/223: getdents d3/d8/d10/d4c 0 2024-04-12T05:08:53.626 INFO:tasks.workunit.client.1.smithi192.stdout:2/148: rename d1/l3 to d1/d8/l36 0 2024-04-12T05:08:53.627 INFO:tasks.workunit.client.1.smithi192.stdout:6/224: creat d3/d8/d22/f52 x:0 0 0 2024-04-12T05:08:53.627 INFO:tasks.workunit.client.0.smithi113.stdout:6/241: rename l1 to d8/de/l50 0 2024-04-12T05:08:53.629 INFO:tasks.workunit.client.0.smithi113.stdout:6/242: write d8/d45/f49 [907946,50775] 0 2024-04-12T05:08:53.630 INFO:tasks.workunit.client.1.smithi192.stdout:3/109: rename l1 to d5/d13/l27 0 2024-04-12T05:08:53.630 INFO:tasks.workunit.client.0.smithi113.stdout:7/270: dwrite d0/dd/d11/f19 [4194304,4194304] 0 2024-04-12T05:08:53.632 INFO:tasks.workunit.client.1.smithi192.stdout:6/225: symlink d3/d8/d10/l53 0 2024-04-12T05:08:53.633 INFO:tasks.workunit.client.1.smithi192.stdout:6/226: fsync d3/d26/f46 0 2024-04-12T05:08:53.634 INFO:tasks.workunit.client.1.smithi192.stdout:6/227: write d3/d26/f46 [894116,49182] 0 2024-04-12T05:08:53.634 INFO:tasks.workunit.client.0.smithi113.stdout:6/243: creat d8/de/d13/d11/d36/d3d/d4c/d26/f51 x:0 0 0 2024-04-12T05:08:53.635 INFO:tasks.workunit.client.1.smithi192.stdout:5/173: sync 2024-04-12T05:08:53.635 INFO:tasks.workunit.client.1.smithi192.stdout:8/93: sync 2024-04-12T05:08:53.635 INFO:tasks.workunit.client.1.smithi192.stdout:2/149: rename d1/l15 to d1/d8/db/de/l37 0 2024-04-12T05:08:53.636 INFO:tasks.workunit.client.1.smithi192.stdout:3/110: rmdir d5/d13/d20 39 2024-04-12T05:08:53.640 INFO:tasks.workunit.client.1.smithi192.stdout:8/94: truncate d6/df/f15 468807 0 2024-04-12T05:08:53.641 INFO:tasks.workunit.client.0.smithi113.stdout:6/244: rename d8/de/d13 to d8/de/d13/d11/d36/d52 22 2024-04-12T05:08:53.641 INFO:tasks.workunit.client.1.smithi192.stdout:8/95: write d6/df/f15 [380436,53073] 0 2024-04-12T05:08:53.642 INFO:tasks.workunit.client.1.smithi192.stdout:2/150: unlink d1/d2d/f33 0 2024-04-12T05:08:53.644 INFO:tasks.workunit.client.1.smithi192.stdout:6/228: dwrite d3/d5/ff [0,4194304] 0 2024-04-12T05:08:53.645 INFO:tasks.workunit.client.0.smithi113.stdout:7/271: fsync d0/dc/f42 0 2024-04-12T05:08:53.645 INFO:tasks.workunit.client.0.smithi113.stdout:3/211: dread d0/d13/f19 [0,4194304] 0 2024-04-12T05:08:53.646 INFO:tasks.workunit.client.0.smithi113.stdout:6/245: mkdir d8/de/d13/d11/d36/d53 0 2024-04-12T05:08:53.648 INFO:tasks.workunit.client.0.smithi113.stdout:1/306: dwrite d10/d2e/d31/f35 [0,4194304] 0 2024-04-12T05:08:53.649 INFO:tasks.workunit.client.0.smithi113.stdout:3/212: write d0/d13/d15/d28/d2a/d2d/f35 [608213,51270] 0 2024-04-12T05:08:53.650 INFO:tasks.workunit.client.1.smithi192.stdout:3/111: symlink d5/d13/d20/l28 0 2024-04-12T05:08:53.650 INFO:tasks.workunit.client.0.smithi113.stdout:7/272: creat d0/d52/f54 x:0 0 0 2024-04-12T05:08:53.652 INFO:tasks.workunit.client.1.smithi192.stdout:5/174: creat d0/d15/f41 x:0 0 0 2024-04-12T05:08:53.653 INFO:tasks.workunit.client.1.smithi192.stdout:2/151: dwrite d1/d8/db/de/f2f [0,4194304] 0 2024-04-12T05:08:53.654 INFO:tasks.workunit.client.0.smithi113.stdout:1/307: symlink d10/d2e/l56 0 2024-04-12T05:08:53.654 INFO:tasks.workunit.client.1.smithi192.stdout:6/229: creat d3/d21/f54 x:0 0 0 2024-04-12T05:08:53.656 INFO:tasks.workunit.client.0.smithi113.stdout:3/213: stat d0/d13/d15/d28/d2a/d2d/f35 0 2024-04-12T05:08:53.657 INFO:tasks.workunit.client.1.smithi192.stdout:6/230: write d3/dc/d39/f4e [787255,118478] 0 2024-04-12T05:08:53.657 INFO:tasks.workunit.client.0.smithi113.stdout:6/246: write d8/de/d13/fa [1508708,44927] 0 2024-04-12T05:08:53.658 INFO:tasks.workunit.client.1.smithi192.stdout:8/96: creat d6/f1d x:0 0 0 2024-04-12T05:08:53.659 INFO:tasks.workunit.client.1.smithi192.stdout:8/97: stat d6/d10/f11 0 2024-04-12T05:08:53.659 INFO:tasks.workunit.client.0.smithi113.stdout:3/214: mkdir d0/d13/d47 0 2024-04-12T05:08:53.660 INFO:tasks.workunit.client.1.smithi192.stdout:8/98: write d6/f18 [730897,83130] 0 2024-04-12T05:08:53.662 INFO:tasks.workunit.client.0.smithi113.stdout:6/247: truncate d8/de/d13/d11/d36/d3d/d4c/d26/f51 291138 0 2024-04-12T05:08:53.664 INFO:tasks.workunit.client.1.smithi192.stdout:2/152: mknod d1/d8/db/de/c38 0 2024-04-12T05:08:53.666 INFO:tasks.workunit.client.1.smithi192.stdout:2/153: chown d1/d8/db/de/f1e 838 1 2024-04-12T05:08:53.666 INFO:tasks.workunit.client.1.smithi192.stdout:5/175: truncate d0/d15/d24/d29/f2a 1442510 0 2024-04-12T05:08:53.669 INFO:tasks.workunit.client.1.smithi192.stdout:5/176: rename d0/d15 to d0/d15/d24/d29/d42 22 2024-04-12T05:08:53.669 INFO:tasks.workunit.client.0.smithi113.stdout:3/215: mknod d0/d13/c48 0 2024-04-12T05:08:53.671 INFO:tasks.workunit.client.0.smithi113.stdout:1/308: getdents d10/d21/d3c 0 2024-04-12T05:08:53.671 INFO:tasks.workunit.client.0.smithi113.stdout:6/248: dwrite d8/de/d13/d11/d36/f35 [0,4194304] 0 2024-04-12T05:08:53.673 INFO:tasks.workunit.client.0.smithi113.stdout:6/249: write d8/f1a [1390303,91755] 0 2024-04-12T05:08:53.674 INFO:tasks.workunit.client.0.smithi113.stdout:3/216: fdatasync d0/fd 0 2024-04-12T05:08:53.675 INFO:tasks.workunit.client.1.smithi192.stdout:8/99: dwrite d6/da/fd [0,4194304] 0 2024-04-12T05:08:53.677 INFO:tasks.workunit.client.0.smithi113.stdout:1/309: dread fd [0,4194304] 0 2024-04-12T05:08:53.680 INFO:tasks.workunit.client.0.smithi113.stdout:3/217: mknod d0/d13/d15/d28/d2a/d34/c49 0 2024-04-12T05:08:53.681 INFO:tasks.workunit.client.0.smithi113.stdout:1/310: dread d10/d21/f2b [0,4194304] 0 2024-04-12T05:08:53.684 INFO:tasks.workunit.client.1.smithi192.stdout:5/177: link d0/c12 d0/d15/c43 0 2024-04-12T05:08:53.685 INFO:tasks.workunit.client.0.smithi113.stdout:3/218: creat d0/d13/d47/f4a x:0 0 0 2024-04-12T05:08:53.686 INFO:tasks.workunit.client.0.smithi113.stdout:3/219: fdatasync d0/f9 0 2024-04-12T05:08:53.688 INFO:tasks.workunit.client.0.smithi113.stdout:3/220: write d0/f30 [1035998,51906] 0 2024-04-12T05:08:53.689 INFO:tasks.workunit.client.1.smithi192.stdout:5/178: symlink d0/d15/l44 0 2024-04-12T05:08:53.691 INFO:tasks.workunit.client.1.smithi192.stdout:8/100: dwrite d6/df/f19 [0,4194304] 0 2024-04-12T05:08:53.691 INFO:tasks.workunit.client.0.smithi113.stdout:7/273: dread d0/dc/d6/fa [0,4194304] 0 2024-04-12T05:08:53.694 INFO:tasks.workunit.client.1.smithi192.stdout:8/101: creat d6/da/f1e x:0 0 0 2024-04-12T05:08:53.696 INFO:tasks.workunit.client.1.smithi192.stdout:5/179: mkdir d0/d15/d32/d45 0 2024-04-12T05:08:53.699 INFO:tasks.workunit.client.0.smithi113.stdout:7/274: dwrite d0/dc/d6/d1a/f3a [0,4194304] 0 2024-04-12T05:08:53.700 INFO:tasks.workunit.client.1.smithi192.stdout:5/180: fdatasync d0/d15/d24/d29/f2a 0 2024-04-12T05:08:53.702 INFO:tasks.workunit.client.0.smithi113.stdout:7/275: creat d0/dd/d1b/f55 x:0 0 0 2024-04-12T05:08:53.704 INFO:tasks.workunit.client.1.smithi192.stdout:3/112: truncate d5/f23 1174609 0 2024-04-12T05:08:53.704 INFO:tasks.workunit.client.1.smithi192.stdout:3/113: readlink d5/l12 0 2024-04-12T05:08:53.705 INFO:tasks.workunit.client.0.smithi113.stdout:6/250: dread d8/de/d13/fb [0,4194304] 0 2024-04-12T05:08:53.706 INFO:tasks.workunit.client.0.smithi113.stdout:7/276: fsync d0/f48 0 2024-04-12T05:08:53.708 INFO:tasks.workunit.client.0.smithi113.stdout:7/277: read d0/dc/d6/d1a/f3e [542325,25384] 0 2024-04-12T05:08:53.709 INFO:tasks.workunit.client.1.smithi192.stdout:3/114: dread d5/d13/d20/f24 [0,4194304] 0 2024-04-12T05:08:53.709 INFO:tasks.workunit.client.1.smithi192.stdout:2/154: write d1/f4 [1181911,33248] 0 2024-04-12T05:08:53.711 INFO:tasks.workunit.client.1.smithi192.stdout:2/155: chown d1/d8/db/de/f1e 1033417 1 2024-04-12T05:08:53.712 INFO:tasks.workunit.client.1.smithi192.stdout:3/115: symlink d5/d13/d20/l29 0 2024-04-12T05:08:53.712 INFO:tasks.workunit.client.0.smithi113.stdout:6/251: dwrite d8/f12 [0,4194304] 0 2024-04-12T05:08:53.715 INFO:tasks.workunit.client.1.smithi192.stdout:2/156: creat d1/f39 x:0 0 0 2024-04-12T05:08:53.715 INFO:tasks.workunit.client.0.smithi113.stdout:6/252: dread - d8/de/d13/d11/d36/d2c/f41 zero size 2024-04-12T05:08:53.715 INFO:tasks.workunit.client.0.smithi113.stdout:1/311: truncate d10/d21/f29 1813354 0 2024-04-12T05:08:53.718 INFO:tasks.workunit.client.1.smithi192.stdout:8/102: write f5 [154,48464] 0 2024-04-12T05:08:53.723 INFO:tasks.workunit.client.1.smithi192.stdout:3/116: creat d5/d13/f2a x:0 0 0 2024-04-12T05:08:53.725 INFO:tasks.workunit.client.1.smithi192.stdout:3/117: write d5/f14 [1051162,40776] 0 2024-04-12T05:08:53.725 INFO:tasks.workunit.client.0.smithi113.stdout:1/312: dread d10/f1e [0,4194304] 0 2024-04-12T05:08:53.726 INFO:tasks.workunit.client.1.smithi192.stdout:2/157: rename d1/d8/db to d1/d8/d3a 0 2024-04-12T05:08:53.729 INFO:tasks.workunit.client.1.smithi192.stdout:8/103: dwrite d6/df/f19 [4194304,4194304] 0 2024-04-12T05:08:53.731 INFO:tasks.workunit.client.0.smithi113.stdout:7/278: dwrite d0/d26/f45 [0,4194304] 0 2024-04-12T05:08:53.732 INFO:tasks.workunit.client.0.smithi113.stdout:6/253: dwrite d8/de/d13/d11/d36/f35 [0,4194304] 0 2024-04-12T05:08:53.733 INFO:tasks.workunit.client.1.smithi192.stdout:2/158: creat d1/d8/d3a/f3b x:0 0 0 2024-04-12T05:08:53.734 INFO:tasks.workunit.client.1.smithi192.stdout:2/159: fdatasync d1/d8/f35 0 2024-04-12T05:08:53.735 INFO:tasks.workunit.client.1.smithi192.stdout:3/118: fdatasync d5/d13/d20/f24 0 2024-04-12T05:08:53.735 INFO:tasks.workunit.client.1.smithi192.stdout:2/160: stat d1/d2d 0 2024-04-12T05:08:53.735 INFO:tasks.workunit.client.0.smithi113.stdout:7/279: chown d0/dc/d6/l1d 7269234 1 2024-04-12T05:08:53.736 INFO:tasks.workunit.client.0.smithi113.stdout:3/221: dread d0/f22 [0,4194304] 0 2024-04-12T05:08:53.736 INFO:tasks.workunit.client.0.smithi113.stdout:4/265: sync 2024-04-12T05:08:53.736 INFO:tasks.workunit.client.0.smithi113.stdout:8/302: sync 2024-04-12T05:08:53.738 INFO:tasks.workunit.client.0.smithi113.stdout:5/260: sync 2024-04-12T05:08:53.740 INFO:tasks.workunit.client.1.smithi192.stdout:2/161: mkdir d1/d8/d17/d3c 0 2024-04-12T05:08:53.741 INFO:tasks.workunit.client.1.smithi192.stdout:2/162: chown d1/d8/c12 1 1 2024-04-12T05:08:53.743 INFO:tasks.workunit.client.0.smithi113.stdout:6/254: creat d8/de/d13/d11/d36/d3d/d4c/d31/f54 x:0 0 0 2024-04-12T05:08:53.743 INFO:tasks.workunit.client.0.smithi113.stdout:1/313: dwrite d10/d21/f40 [0,4194304] 0 2024-04-12T05:08:53.744 INFO:tasks.workunit.client.1.smithi192.stdout:2/163: creat d1/d2d/f3d x:0 0 0 2024-04-12T05:08:53.744 INFO:tasks.workunit.client.1.smithi192.stdout:2/164: dread - d1/d8/f10 zero size 2024-04-12T05:08:53.745 INFO:tasks.workunit.client.0.smithi113.stdout:3/222: mknod d0/d13/c4b 0 2024-04-12T05:08:53.745 INFO:tasks.workunit.client.0.smithi113.stdout:5/261: dread - d0/d10/f45 zero size 2024-04-12T05:08:53.753 INFO:tasks.workunit.client.0.smithi113.stdout:7/280: dwrite d0/dd/d11/d44/f47 [0,4194304] 0 2024-04-12T05:08:53.754 INFO:tasks.workunit.client.1.smithi192.stdout:3/119: link d5/le d5/d13/d1a/l2b 0 2024-04-12T05:08:53.754 INFO:tasks.workunit.client.1.smithi192.stdout:8/104: truncate d6/df/f16 401782 0 2024-04-12T05:08:53.754 INFO:tasks.workunit.client.0.smithi113.stdout:5/262: creat d0/d10/d27/d13/f4b x:0 0 0 2024-04-12T05:08:53.757 INFO:tasks.workunit.client.1.smithi192.stdout:3/120: unlink d5/l12 0 2024-04-12T05:08:53.758 INFO:tasks.workunit.client.1.smithi192.stdout:3/121: chown d5/f14 49493901 1 2024-04-12T05:08:53.758 INFO:tasks.workunit.client.1.smithi192.stdout:8/105: mkdir d6/d1f 0 2024-04-12T05:08:53.759 INFO:tasks.workunit.client.0.smithi113.stdout:6/255: truncate d8/de/d13/d11/d36/d3d/d4c/d31/f54 960926 0 2024-04-12T05:08:53.759 INFO:tasks.workunit.client.1.smithi192.stdout:8/106: fsync f5 0 2024-04-12T05:08:53.760 INFO:tasks.workunit.client.1.smithi192.stdout:8/107: dread - d6/d10/f11 zero size 2024-04-12T05:08:53.760 INFO:tasks.workunit.client.1.smithi192.stdout:3/122: write d5/d13/f18 [290368,75527] 0 2024-04-12T05:08:53.763 INFO:tasks.workunit.client.0.smithi113.stdout:7/281: write d0/f12 [4494869,34856] 0 2024-04-12T05:08:53.764 INFO:tasks.workunit.client.1.smithi192.stdout:8/108: creat d6/d10/f20 x:0 0 0 2024-04-12T05:08:53.764 INFO:tasks.workunit.client.1.smithi192.stdout:8/109: chown f5 34 1 2024-04-12T05:08:53.765 INFO:tasks.workunit.client.1.smithi192.stdout:3/123: rename c4 to d5/d13/d20/d26/c2c 0 2024-04-12T05:08:53.765 INFO:tasks.workunit.client.1.smithi192.stdout:3/124: stat d5/d13 0 2024-04-12T05:08:53.769 INFO:tasks.workunit.client.1.smithi192.stdout:8/110: mkdir d6/d21 0 2024-04-12T05:08:53.770 INFO:tasks.workunit.client.1.smithi192.stdout:3/125: unlink d5/c11 0 2024-04-12T05:08:53.772 INFO:tasks.workunit.client.1.smithi192.stdout:8/111: mkdir d6/d10/d22 0 2024-04-12T05:08:53.772 INFO:tasks.workunit.client.0.smithi113.stdout:1/314: dwrite d10/d21/d23/f36 [0,4194304] 0 2024-04-12T05:08:53.772 INFO:tasks.workunit.client.0.smithi113.stdout:3/223: dwrite d0/f6 [0,4194304] 0 2024-04-12T05:08:53.773 INFO:tasks.workunit.client.0.smithi113.stdout:8/303: dwrite d9/d16/d1e/d26/d32/d48/f49 [0,4194304] 0 2024-04-12T05:08:53.775 INFO:tasks.workunit.client.0.smithi113.stdout:6/256: write d8/de/d13/d11/d36/d3d/d4c/d26/f51 [500301,32068] 0 2024-04-12T05:08:53.780 INFO:tasks.workunit.client.1.smithi192.stdout:8/112: getdents d6/da 0 2024-04-12T05:08:53.781 INFO:tasks.workunit.client.1.smithi192.stdout:8/113: chown d6/da/lb 0 1 2024-04-12T05:08:53.783 INFO:tasks.workunit.client.0.smithi113.stdout:5/263: rename d0/d10/d27/d1d/d38/l42 to d0/d10/l4c 0 2024-04-12T05:08:53.783 INFO:tasks.workunit.client.0.smithi113.stdout:1/315: getdents d10/d55 0 2024-04-12T05:08:53.784 INFO:tasks.workunit.client.0.smithi113.stdout:8/304: creat d9/d1b/f61 x:0 0 0 2024-04-12T05:08:53.786 INFO:tasks.workunit.client.0.smithi113.stdout:6/257: dread - d8/de/d13/d11/d36/d2c/f41 zero size 2024-04-12T05:08:53.788 INFO:tasks.workunit.client.1.smithi192.stdout:8/114: rename d6/d10/f20 to d6/d21/f23 0 2024-04-12T05:08:53.788 INFO:tasks.workunit.client.1.smithi192.stdout:3/126: link d5/f19 d5/f2d 0 2024-04-12T05:08:53.788 INFO:tasks.workunit.client.0.smithi113.stdout:5/264: symlink d0/d10/d27/l4d 0 2024-04-12T05:08:53.789 INFO:tasks.workunit.client.0.smithi113.stdout:5/265: fdatasync d0/f3a 0 2024-04-12T05:08:53.789 INFO:tasks.workunit.client.0.smithi113.stdout:8/305: write d9/d16/d3c/f52 [525502,116646] 0 2024-04-12T05:08:53.791 INFO:tasks.workunit.client.0.smithi113.stdout:1/316: rename d10/d21/c39 to d10/d2e/d31/d33/c57 0 2024-04-12T05:08:53.792 INFO:tasks.workunit.client.1.smithi192.stdout:8/115: dread f3 [0,4194304] 0 2024-04-12T05:08:53.792 INFO:tasks.workunit.client.0.smithi113.stdout:7/282: dread d0/dc/d6/d1a/f34 [0,4194304] 0 2024-04-12T05:08:53.793 INFO:tasks.workunit.client.1.smithi192.stdout:2/165: truncate d1/d8/d3a/de/f1e 3452900 0 2024-04-12T05:08:53.793 INFO:tasks.workunit.client.0.smithi113.stdout:6/258: mknod d8/de/d13/d11/d36/d3d/d4c/d31/c55 0 2024-04-12T05:08:53.793 INFO:tasks.workunit.client.0.smithi113.stdout:3/224: dread d0/d13/f19 [0,4194304] 0 2024-04-12T05:08:53.793 INFO:tasks.workunit.client.0.smithi113.stdout:8/306: chown d9/d16/d1c/l31 6886799 1 2024-04-12T05:08:53.795 INFO:tasks.workunit.client.1.smithi192.stdout:8/116: symlink d6/df/l24 0 2024-04-12T05:08:53.796 INFO:tasks.workunit.client.0.smithi113.stdout:8/307: chown d9/d16/d1c/c1f 42233 1 2024-04-12T05:08:53.797 INFO:tasks.workunit.client.1.smithi192.stdout:3/127: creat d5/d13/f2e x:0 0 0 2024-04-12T05:08:53.798 INFO:tasks.workunit.client.0.smithi113.stdout:1/317: write d10/f32 [677996,10714] 0 2024-04-12T05:08:53.798 INFO:tasks.workunit.client.0.smithi113.stdout:5/266: mkdir d0/d10/d27/d1d/d38/d44/d4e 0 2024-04-12T05:08:53.799 INFO:tasks.workunit.client.0.smithi113.stdout:8/308: readlink d9/d16/d1c/l31 0 2024-04-12T05:08:53.799 INFO:tasks.workunit.client.1.smithi192.stdout:8/117: dread d6/df/f19 [4194304,4194304] 0 2024-04-12T05:08:53.799 INFO:tasks.workunit.client.1.smithi192.stdout:2/166: unlink d1/d8/d3a/de/l28 0 2024-04-12T05:08:53.800 INFO:tasks.workunit.client.0.smithi113.stdout:8/309: chown d9/fd 676 1 2024-04-12T05:08:53.803 INFO:tasks.workunit.client.0.smithi113.stdout:4/266: dread da/dd/d3a/f44 [0,4194304] 0 2024-04-12T05:08:53.803 INFO:tasks.workunit.client.0.smithi113.stdout:1/318: write d10/d14/f2a [4762539,51553] 0 2024-04-12T05:08:53.804 INFO:tasks.workunit.client.0.smithi113.stdout:4/267: fsync da/d24/f52 0 2024-04-12T05:08:53.806 INFO:tasks.workunit.client.1.smithi192.stdout:2/167: readlink d1/d8/d3a/de/l37 0 2024-04-12T05:08:53.806 INFO:tasks.workunit.client.0.smithi113.stdout:4/268: fdatasync da/dd/df/d1d/f40 0 2024-04-12T05:08:53.806 INFO:tasks.workunit.client.0.smithi113.stdout:5/267: rename d0/d10/d27/d1d/f32 to d0/d10/d27/d33/f4f 0 2024-04-12T05:08:53.807 INFO:tasks.workunit.client.0.smithi113.stdout:5/268: dread - d0/d10/d27/d13/f4b zero size 2024-04-12T05:08:53.808 INFO:tasks.workunit.client.1.smithi192.stdout:8/118: mknod d6/c25 0 2024-04-12T05:08:53.809 INFO:tasks.workunit.client.0.smithi113.stdout:5/269: chown d0/d10/d27/d1d/f2a 0 1 2024-04-12T05:08:53.809 INFO:tasks.workunit.client.0.smithi113.stdout:1/319: rename fe to d10/d21/d3c/f58 0 2024-04-12T05:08:53.811 INFO:tasks.workunit.client.0.smithi113.stdout:8/310: link d9/la d9/d34/l62 0 2024-04-12T05:08:53.813 INFO:tasks.workunit.client.1.smithi192.stdout:3/128: link d5/f14 d5/d13/f2f 0 2024-04-12T05:08:53.813 INFO:tasks.workunit.client.0.smithi113.stdout:7/283: truncate d0/dc/d6/d1a/f3a 3769070 0 2024-04-12T05:08:53.814 INFO:tasks.workunit.client.0.smithi113.stdout:1/320: creat d10/d2e/d31/f59 x:0 0 0 2024-04-12T05:08:53.814 INFO:tasks.workunit.client.0.smithi113.stdout:8/311: dread - d9/d1b/f4f zero size 2024-04-12T05:08:53.814 INFO:tasks.workunit.client.0.smithi113.stdout:5/270: mknod d0/d10/c50 0 2024-04-12T05:08:53.816 INFO:tasks.workunit.client.1.smithi192.stdout:8/119: creat d6/df/f26 x:0 0 0 2024-04-12T05:08:53.816 INFO:tasks.workunit.client.1.smithi192.stdout:1/153: sync 2024-04-12T05:08:53.816 INFO:tasks.workunit.client.1.smithi192.stdout:0/73: sync 2024-04-12T05:08:53.816 INFO:tasks.workunit.client.1.smithi192.stdout:9/93: sync 2024-04-12T05:08:53.816 INFO:tasks.workunit.client.1.smithi192.stdout:7/119: sync 2024-04-12T05:08:53.816 INFO:tasks.workunit.client.0.smithi113.stdout:4/269: dwrite da/dd/f48 [0,4194304] 0 2024-04-12T05:08:53.816 INFO:tasks.workunit.client.1.smithi192.stdout:9/94: read f6 [3531936,44244] 0 2024-04-12T05:08:53.818 INFO:tasks.workunit.client.1.smithi192.stdout:1/154: write d0/f18 [796208,123646] 0 2024-04-12T05:08:53.821 INFO:tasks.workunit.client.0.smithi113.stdout:8/312: write d9/d16/d1e/d26/d32/d48/f49 [4500617,34829] 0 2024-04-12T05:08:53.821 INFO:tasks.workunit.client.0.smithi113.stdout:4/270: creat da/d24/f53 x:0 0 0 2024-04-12T05:08:53.821 INFO:tasks.workunit.client.0.smithi113.stdout:3/225: dread d0/fd [0,4194304] 0 2024-04-12T05:08:53.826 INFO:tasks.workunit.client.1.smithi192.stdout:7/120: symlink d3/d7/d15/d1a/l2a 0 2024-04-12T05:08:53.826 INFO:tasks.workunit.client.0.smithi113.stdout:4/271: symlink da/dd/df/d1d/l54 0 2024-04-12T05:08:53.826 INFO:tasks.workunit.client.0.smithi113.stdout:3/226: rmdir d0/d13/d15/d28/d2a/d34 39 2024-04-12T05:08:53.833 INFO:tasks.workunit.client.0.smithi113.stdout:7/284: dread d0/dd/f1c [0,4194304] 0 2024-04-12T05:08:53.833 INFO:tasks.workunit.client.0.smithi113.stdout:5/271: rmdir d0 39 2024-04-12T05:08:53.834 INFO:tasks.workunit.client.1.smithi192.stdout:3/129: mknod d5/d13/d1a/c30 0 2024-04-12T05:08:53.834 INFO:tasks.workunit.client.1.smithi192.stdout:3/130: chown d5 2536758 1 2024-04-12T05:08:53.835 INFO:tasks.workunit.client.0.smithi113.stdout:1/321: truncate d10/d14/f1f 571316 0 2024-04-12T05:08:53.835 INFO:tasks.workunit.client.1.smithi192.stdout:2/168: write d1/d8/d3a/de/f1e [2243448,5148] 0 2024-04-12T05:08:53.835 INFO:tasks.workunit.client.1.smithi192.stdout:2/169: chown d1 3 1 2024-04-12T05:08:53.836 INFO:tasks.workunit.client.0.smithi113.stdout:3/227: write d0/f10 [1519842,70258] 0 2024-04-12T05:08:53.836 INFO:tasks.workunit.client.0.smithi113.stdout:8/313: dwrite d9/d16/d3c/f4e [4194304,4194304] 0 2024-04-12T05:08:53.838 INFO:tasks.workunit.client.1.smithi192.stdout:7/121: rename d3/l4 to d3/d7/l2b 0 2024-04-12T05:08:53.840 INFO:tasks.workunit.client.1.smithi192.stdout:9/95: write fa [540856,91726] 0 2024-04-12T05:08:53.840 INFO:tasks.workunit.client.0.smithi113.stdout:7/285: mknod d0/dd/d2d/c56 0 2024-04-12T05:08:53.841 INFO:tasks.workunit.client.1.smithi192.stdout:1/155: symlink d0/d2/dd/d21/l2a 0 2024-04-12T05:08:53.844 INFO:tasks.workunit.client.1.smithi192.stdout:0/74: write d0/fb [577483,10627] 0 2024-04-12T05:08:53.846 INFO:tasks.workunit.client.1.smithi192.stdout:2/170: symlink d1/d8/d17/d19/l3e 0 2024-04-12T05:08:53.847 INFO:tasks.workunit.client.1.smithi192.stdout:0/75: write d0/d6/f13 [968423,27301] 0 2024-04-12T05:08:53.847 INFO:tasks.workunit.client.0.smithi113.stdout:8/314: creat d9/d16/d1e/d26/d32/d5b/f63 x:0 0 0 2024-04-12T05:08:53.847 INFO:tasks.workunit.client.0.smithi113.stdout:4/272: dwrite da/d3c/d43/f4d [0,4194304] 0 2024-04-12T05:08:53.847 INFO:tasks.workunit.client.1.smithi192.stdout:9/96: fsync f9 0 2024-04-12T05:08:53.852 INFO:tasks.workunit.client.1.smithi192.stdout:1/156: rename d0/d2/dd/f25 to d0/d1b/f2b 0 2024-04-12T05:08:53.853 INFO:tasks.workunit.client.0.smithi113.stdout:7/286: creat d0/d52/f57 x:0 0 0 2024-04-12T05:08:53.854 INFO:tasks.workunit.client.1.smithi192.stdout:7/122: unlink d3/fd 0 2024-04-12T05:08:53.854 INFO:tasks.workunit.client.0.smithi113.stdout:8/315: creat d9/d16/d1e/d26/d32/f64 x:0 0 0 2024-04-12T05:08:53.855 INFO:tasks.workunit.client.0.smithi113.stdout:1/322: dwrite fd [0,4194304] 0 2024-04-12T05:08:53.856 INFO:tasks.workunit.client.0.smithi113.stdout:8/316: dread - d9/d34/f57 zero size 2024-04-12T05:08:53.856 INFO:tasks.workunit.client.0.smithi113.stdout:3/228: dwrite d0/f30 [0,4194304] 0 2024-04-12T05:08:53.857 INFO:tasks.workunit.client.0.smithi113.stdout:4/273: unlink da/dd/df/d1d/l34 0 2024-04-12T05:08:53.857 INFO:tasks.workunit.client.1.smithi192.stdout:2/171: creat d1/d8/d3a/f3f x:0 0 0 2024-04-12T05:08:53.858 INFO:tasks.workunit.client.1.smithi192.stdout:9/97: dread fe [0,4194304] 0 2024-04-12T05:08:53.859 INFO:tasks.workunit.client.1.smithi192.stdout:8/120: truncate d6/da/fd 5313219 0 2024-04-12T05:08:53.860 INFO:tasks.workunit.client.1.smithi192.stdout:0/76: creat d0/f16 x:0 0 0 2024-04-12T05:08:53.860 INFO:tasks.workunit.client.0.smithi113.stdout:1/323: symlink d10/d21/d23/l5a 0 2024-04-12T05:08:53.862 INFO:tasks.workunit.client.1.smithi192.stdout:1/157: dread d0/f24 [0,4194304] 0 2024-04-12T05:08:53.862 INFO:tasks.workunit.client.1.smithi192.stdout:3/131: truncate d5/d13/d20/f24 4154524 0 2024-04-12T05:08:53.866 INFO:tasks.workunit.client.0.smithi113.stdout:3/229: rename d0/f22 to d0/d13/d15/d28/d2a/d2d/f4c 0 2024-04-12T05:08:53.867 INFO:tasks.workunit.client.0.smithi113.stdout:4/274: dwrite da/dd/df/d1d/f40 [0,4194304] 0 2024-04-12T05:08:53.867 INFO:tasks.workunit.client.0.smithi113.stdout:7/287: creat d0/dc/d6/d1a/f58 x:0 0 0 2024-04-12T05:08:53.868 INFO:tasks.workunit.client.1.smithi192.stdout:9/98: truncate fd 342287 0 2024-04-12T05:08:53.868 INFO:tasks.workunit.client.1.smithi192.stdout:0/77: creat d0/f17 x:0 0 0 2024-04-12T05:08:53.870 INFO:tasks.workunit.client.0.smithi113.stdout:1/324: mkdir d10/d41/d5b 0 2024-04-12T05:08:53.871 INFO:tasks.workunit.client.0.smithi113.stdout:8/317: rmdir d9/d16/d1e/d26 39 2024-04-12T05:08:53.874 INFO:tasks.workunit.client.1.smithi192.stdout:1/158: mknod d0/d1b/c2c 0 2024-04-12T05:08:53.874 INFO:tasks.workunit.client.1.smithi192.stdout:1/159: readlink d0/d2/l14 0 2024-04-12T05:08:53.875 INFO:tasks.workunit.client.1.smithi192.stdout:1/160: fdatasync d0/d1b/f13 0 2024-04-12T05:08:53.875 INFO:tasks.workunit.client.1.smithi192.stdout:9/99: rmdir d11/d15 39 2024-04-12T05:08:53.876 INFO:tasks.workunit.client.1.smithi192.stdout:2/172: rmdir d1/d8/d3a 39 2024-04-12T05:08:53.878 INFO:tasks.workunit.client.0.smithi113.stdout:1/325: rmdir d10/d2e 39 2024-04-12T05:08:53.880 INFO:tasks.workunit.client.0.smithi113.stdout:8/318: fsync d9/d16/d1e/d26/d32/f64 0 2024-04-12T05:08:53.881 INFO:tasks.workunit.client.0.smithi113.stdout:3/230: dwrite d0/d13/d15/f2e [0,4194304] 0 2024-04-12T05:08:53.881 INFO:tasks.workunit.client.0.smithi113.stdout:4/275: rename da/de/c2d to da/c55 0 2024-04-12T05:08:53.884 INFO:tasks.workunit.client.0.smithi113.stdout:8/319: mkdir d9/d16/d65 0 2024-04-12T05:08:53.885 INFO:tasks.workunit.client.1.smithi192.stdout:8/121: rename d6/da/lb to d6/l27 0 2024-04-12T05:08:53.885 INFO:tasks.workunit.client.0.smithi113.stdout:3/231: unlink d0/lc 0 2024-04-12T05:08:53.886 INFO:tasks.workunit.client.1.smithi192.stdout:2/173: unlink d1/d8/c1f 0 2024-04-12T05:08:53.887 INFO:tasks.workunit.client.0.smithi113.stdout:3/232: fdatasync d0/d13/f1f 0 2024-04-12T05:08:53.889 INFO:tasks.workunit.client.0.smithi113.stdout:7/288: link d0/d26/l3f d0/dd/d11/d44/l59 0 2024-04-12T05:08:53.891 INFO:tasks.workunit.client.0.smithi113.stdout:8/320: rmdir d9/d16/d1e/d26 39 2024-04-12T05:08:53.892 INFO:tasks.workunit.client.0.smithi113.stdout:8/321: chown d9/lc 343334 1 2024-04-12T05:08:53.893 INFO:tasks.workunit.client.0.smithi113.stdout:3/233: unlink d0/f30 0 2024-04-12T05:08:53.894 INFO:tasks.workunit.client.1.smithi192.stdout:2/174: mkdir d1/d8/d17/d19/d40 0 2024-04-12T05:08:53.896 INFO:tasks.workunit.client.0.smithi113.stdout:4/276: rename da/c11 to da/de/c56 0 2024-04-12T05:08:53.896 INFO:tasks.workunit.client.0.smithi113.stdout:8/322: write d9/d16/d3c/f52 [399357,62284] 0 2024-04-12T05:08:53.898 INFO:tasks.workunit.client.1.smithi192.stdout:7/123: dwrite d3/d7/f9 [0,4194304] 0 2024-04-12T05:08:53.898 INFO:tasks.workunit.client.0.smithi113.stdout:4/277: write da/dd/df/d14/f19 [3074907,120229] 0 2024-04-12T05:08:53.900 INFO:tasks.workunit.client.1.smithi192.stdout:7/124: write d3/d7/f13 [462691,55449] 0 2024-04-12T05:08:53.900 INFO:tasks.workunit.client.1.smithi192.stdout:1/161: link d0/f18 d0/d2/dd/d21/f2d 0 2024-04-12T05:08:53.904 INFO:tasks.workunit.client.1.smithi192.stdout:1/162: write d0/d1b/f13 [2763321,71595] 0 2024-04-12T05:08:53.904 INFO:tasks.workunit.client.1.smithi192.stdout:2/175: unlink d1/d8/d17/d19/f27 0 2024-04-12T05:08:53.907 INFO:tasks.workunit.client.1.smithi192.stdout:2/176: fsync d1/f39 0 2024-04-12T05:08:53.910 INFO:tasks.workunit.client.0.smithi113.stdout:7/289: rename d0/dc/d6/d1a/f3e to d0/f5a 0 2024-04-12T05:08:53.910 INFO:tasks.workunit.client.0.smithi113.stdout:1/326: dwrite d10/d21/f29 [0,4194304] 0 2024-04-12T05:08:53.910 INFO:tasks.workunit.client.0.smithi113.stdout:4/278: link da/d26/c51 da/dd/d1b/c57 0 2024-04-12T05:08:53.911 INFO:tasks.workunit.client.1.smithi192.stdout:7/125: mknod d3/d7/d27/c2c 0 2024-04-12T05:08:53.912 INFO:tasks.workunit.client.1.smithi192.stdout:3/132: dwrite d5/f2d [0,4194304] 0 2024-04-12T05:08:53.912 INFO:tasks.workunit.client.0.smithi113.stdout:3/234: getdents d0/d13/d15/d28 0 2024-04-12T05:08:53.912 INFO:tasks.workunit.client.1.smithi192.stdout:2/177: fsync d1/d2d/f32 0 2024-04-12T05:08:53.913 INFO:tasks.workunit.client.0.smithi113.stdout:1/327: readlink d10/d2e/l3f 0 2024-04-12T05:08:53.915 INFO:tasks.workunit.client.1.smithi192.stdout:2/178: dread - d1/d8/f10 zero size 2024-04-12T05:08:53.916 INFO:tasks.workunit.client.1.smithi192.stdout:7/126: write d3/d11/f16 [28597,79703] 0 2024-04-12T05:08:53.917 INFO:tasks.workunit.client.0.smithi113.stdout:4/279: mknod da/dd/d1b/c58 0 2024-04-12T05:08:53.918 INFO:tasks.workunit.client.0.smithi113.stdout:7/290: mknod d0/dd/c5b 0 2024-04-12T05:08:53.920 INFO:tasks.workunit.client.1.smithi192.stdout:9/100: truncate f7 3673799 0 2024-04-12T05:08:53.921 INFO:tasks.workunit.client.0.smithi113.stdout:7/291: dread - d0/dd/d1b/f23 zero size 2024-04-12T05:08:53.921 INFO:tasks.workunit.client.0.smithi113.stdout:3/235: write d0/d13/f19 [3447138,63086] 0 2024-04-12T05:08:53.922 INFO:tasks.workunit.client.1.smithi192.stdout:1/163: dwrite d0/fc [4194304,4194304] 0 2024-04-12T05:08:53.923 INFO:tasks.workunit.client.0.smithi113.stdout:1/328: mkdir d10/d2e/d31/d5c 0 2024-04-12T05:08:53.926 INFO:tasks.workunit.client.0.smithi113.stdout:8/323: link d9/d16/d1c/f2c d9/d16/d1e/d26/f66 0 2024-04-12T05:08:53.930 INFO:tasks.workunit.client.1.smithi192.stdout:7/127: rename d3/ff to d3/d11/f2d 0 2024-04-12T05:08:53.930 INFO:tasks.workunit.client.1.smithi192.stdout:2/179: creat d1/d8/f41 x:0 0 0 2024-04-12T05:08:53.930 INFO:tasks.workunit.client.0.smithi113.stdout:4/280: dwrite da/dd/df/f10 [0,4194304] 0 2024-04-12T05:08:53.931 INFO:tasks.workunit.client.0.smithi113.stdout:8/324: truncate d9/d34/f57 219400 0 2024-04-12T05:08:53.933 INFO:tasks.workunit.client.1.smithi192.stdout:8/122: dwrite d6/df/f19 [4194304,4194304] 0 2024-04-12T05:08:53.933 INFO:tasks.workunit.client.1.smithi192.stdout:1/164: mknod d0/d2/d1e/c2e 0 2024-04-12T05:08:53.937 INFO:tasks.workunit.client.0.smithi113.stdout:8/325: write d9/d1b/f4f [749825,6320] 0 2024-04-12T05:08:53.938 INFO:tasks.workunit.client.0.smithi113.stdout:4/281: symlink da/de/l59 0 2024-04-12T05:08:53.940 INFO:tasks.workunit.client.0.smithi113.stdout:7/292: dwrite d0/dd/f1c [0,4194304] 0 2024-04-12T05:08:53.940 INFO:tasks.workunit.client.1.smithi192.stdout:9/101: rename d11/d1d to d11/d15/d21 0 2024-04-12T05:08:53.942 INFO:tasks.workunit.client.0.smithi113.stdout:1/329: dread d10/d14/f17 [0,4194304] 0 2024-04-12T05:08:53.944 INFO:tasks.workunit.client.1.smithi192.stdout:1/165: write d0/f3 [3623673,56696] 0 2024-04-12T05:08:53.946 INFO:tasks.workunit.client.0.smithi113.stdout:7/293: mkdir d0/dd/d11/d44/d5c 0 2024-04-12T05:08:53.947 INFO:tasks.workunit.client.1.smithi192.stdout:2/180: symlink d1/d8/d3a/de/l42 0 2024-04-12T05:08:53.947 INFO:tasks.workunit.client.1.smithi192.stdout:6/231: sync 2024-04-12T05:08:53.947 INFO:tasks.workunit.client.1.smithi192.stdout:4/166: sync 2024-04-12T05:08:53.948 INFO:tasks.workunit.client.1.smithi192.stdout:1/166: chown d0/d2/d1e 36401943 1 2024-04-12T05:08:53.949 INFO:tasks.workunit.client.1.smithi192.stdout:8/123: creat d6/f28 x:0 0 0 2024-04-12T05:08:53.950 INFO:tasks.workunit.client.1.smithi192.stdout:9/102: symlink d11/l22 0 2024-04-12T05:08:53.950 INFO:tasks.workunit.client.0.smithi113.stdout:7/294: creat d0/dd/d1b/f5d x:0 0 0 2024-04-12T05:08:53.951 INFO:tasks.workunit.client.0.smithi113.stdout:4/282: dwrite da/d2b/f3e [0,4194304] 0 2024-04-12T05:08:53.952 INFO:tasks.workunit.client.1.smithi192.stdout:4/167: creat d1/da/d1c/f36 x:0 0 0 2024-04-12T05:08:53.953 INFO:tasks.workunit.client.1.smithi192.stdout:9/103: chown d11/l22 119281399 1 2024-04-12T05:08:53.953 INFO:tasks.workunit.client.1.smithi192.stdout:3/133: dwrite d5/d13/f17 [0,4194304] 0 2024-04-12T05:08:53.955 INFO:tasks.workunit.client.1.smithi192.stdout:4/168: fsync d1/da/d1c/f28 0 2024-04-12T05:08:53.962 INFO:tasks.workunit.client.1.smithi192.stdout:1/167: dwrite d0/d1b/f13 [0,4194304] 0 2024-04-12T05:08:53.964 INFO:tasks.workunit.client.1.smithi192.stdout:1/168: read d0/d2/dd/f26 [7812683,95234] 0 2024-04-12T05:08:53.964 INFO:tasks.workunit.client.1.smithi192.stdout:1/169: chown d0/d2 474732 1 2024-04-12T05:08:53.965 INFO:tasks.workunit.client.1.smithi192.stdout:4/169: dwrite d1/f11 [0,4194304] 0 2024-04-12T05:08:53.969 INFO:tasks.workunit.client.1.smithi192.stdout:7/128: dread d3/f12 [0,4194304] 0 2024-04-12T05:08:53.978 INFO:tasks.workunit.client.0.smithi113.stdout:2/165: sync 2024-04-12T05:08:53.978 INFO:tasks.workunit.client.0.smithi113.stdout:9/341: sync 2024-04-12T05:08:53.979 INFO:tasks.workunit.client.0.smithi113.stdout:4/283: fsync da/dd/f32 0 2024-04-12T05:08:53.980 INFO:tasks.workunit.client.1.smithi192.stdout:6/232: mknod d3/dc/d39/c55 0 2024-04-12T05:08:53.981 INFO:tasks.workunit.client.1.smithi192.stdout:2/181: read d1/d8/d3a/de/f11 [1849205,76716] 0 2024-04-12T05:08:53.981 INFO:tasks.workunit.client.0.smithi113.stdout:2/166: chown d8/d9 1292185224 1 2024-04-12T05:08:53.981 INFO:tasks.workunit.client.1.smithi192.stdout:6/233: truncate d3/d26/f31 664362 0 2024-04-12T05:08:53.982 INFO:tasks.workunit.client.1.smithi192.stdout:8/124: symlink d6/d10/d22/l29 0 2024-04-12T05:08:53.982 INFO:tasks.workunit.client.1.smithi192.stdout:2/182: rename d1/d8/d17/d3c to d1/d8/d17/d3c/d43 22 2024-04-12T05:08:53.983 INFO:tasks.workunit.client.0.smithi113.stdout:2/167: chown d8/d9/de/d15 1 1 2024-04-12T05:08:53.983 INFO:tasks.workunit.client.1.smithi192.stdout:8/125: dread - d6/df/f26 zero size 2024-04-12T05:08:53.983 INFO:tasks.workunit.client.1.smithi192.stdout:9/104: mknod d11/d14/c23 0 2024-04-12T05:08:53.986 INFO:tasks.workunit.client.1.smithi192.stdout:6/234: dread d3/d5/ff [0,4194304] 0 2024-04-12T05:08:53.989 INFO:tasks.workunit.client.1.smithi192.stdout:3/134: creat d5/d13/d20/f31 x:0 0 0 2024-04-12T05:08:53.989 INFO:tasks.workunit.client.1.smithi192.stdout:6/235: truncate d3/d26/f28 208491 0 2024-04-12T05:08:53.991 INFO:tasks.workunit.client.0.smithi113.stdout:2/168: rmdir d8/d16 39 2024-04-12T05:08:53.993 INFO:tasks.workunit.client.0.smithi113.stdout:4/284: rmdir da/d3c 39 2024-04-12T05:08:53.993 INFO:tasks.workunit.client.0.smithi113.stdout:7/295: link d0/f12 d0/dd/d11/f5e 0 2024-04-12T05:08:53.994 INFO:tasks.workunit.client.1.smithi192.stdout:7/129: creat d3/d1d/f2e x:0 0 0 2024-04-12T05:08:53.996 INFO:tasks.workunit.client.0.smithi113.stdout:8/326: dread d9/d16/d1c/f2c [0,4194304] 0 2024-04-12T05:08:53.997 INFO:tasks.workunit.client.1.smithi192.stdout:9/105: creat d11/d15/d21/f24 x:0 0 0 2024-04-12T05:08:53.999 INFO:tasks.workunit.client.0.smithi113.stdout:8/327: readlink d9/lc 0 2024-04-12T05:08:54.003 INFO:tasks.workunit.client.1.smithi192.stdout:7/130: mknod d3/d7/c2f 0 2024-04-12T05:08:54.003 INFO:tasks.workunit.client.0.smithi113.stdout:8/328: dread - d9/f55 zero size 2024-04-12T05:08:54.003 INFO:tasks.workunit.client.0.smithi113.stdout:3/236: dread d0/d13/d15/f3e [0,4194304] 0 2024-04-12T05:08:54.004 INFO:tasks.workunit.client.1.smithi192.stdout:8/126: creat d6/d1f/f2a x:0 0 0 2024-04-12T05:08:54.005 INFO:tasks.workunit.client.0.smithi113.stdout:2/169: dwrite d8/d9/f23 [0,4194304] 0 2024-04-12T05:08:54.006 INFO:tasks.workunit.client.1.smithi192.stdout:6/236: mkdir d3/d8/d56 0 2024-04-12T05:08:54.009 INFO:tasks.workunit.client.0.smithi113.stdout:4/285: dread da/dd/d3a/f44 [0,4194304] 0 2024-04-12T05:08:54.009 INFO:tasks.workunit.client.1.smithi192.stdout:3/135: truncate d5/d13/d20/f24 3988539 0 2024-04-12T05:08:54.011 INFO:tasks.workunit.client.0.smithi113.stdout:9/342: link d1/d4/f33 d1/f78 0 2024-04-12T05:08:54.011 INFO:tasks.workunit.client.1.smithi192.stdout:7/131: write d3/d11/f2d [145837,128379] 0 2024-04-12T05:08:54.014 INFO:tasks.workunit.client.1.smithi192.stdout:9/106: fdatasync fd 0 2024-04-12T05:08:54.016 INFO:tasks.workunit.client.1.smithi192.stdout:6/237: mknod d3/d8/d10/c57 0 2024-04-12T05:08:54.017 INFO:tasks.workunit.client.0.smithi113.stdout:7/296: getdents d0/dc/d50 0 2024-04-12T05:08:54.020 INFO:tasks.workunit.client.0.smithi113.stdout:8/329: dwrite d9/d16/d1e/d26/d32/f60 [0,4194304] 0 2024-04-12T05:08:54.021 INFO:tasks.workunit.client.0.smithi113.stdout:2/170: creat d8/d9/f2c x:0 0 0 2024-04-12T05:08:54.021 INFO:tasks.workunit.client.1.smithi192.stdout:8/127: dread d6/da/fd [0,4194304] 0 2024-04-12T05:08:54.022 INFO:tasks.workunit.client.0.smithi113.stdout:4/286: dread da/dd/d3a/f44 [0,4194304] 0 2024-04-12T05:08:54.023 INFO:tasks.workunit.client.1.smithi192.stdout:8/128: readlink l4 0 2024-04-12T05:08:54.024 INFO:tasks.workunit.client.0.smithi113.stdout:4/287: stat da/dd/f48 0 2024-04-12T05:08:54.024 INFO:tasks.workunit.client.0.smithi113.stdout:8/330: truncate d9/d1b/f45 615056 0 2024-04-12T05:08:54.024 INFO:tasks.workunit.client.0.smithi113.stdout:3/237: dwrite d0/d13/d15/d28/d2a/d2d/f35 [0,4194304] 0 2024-04-12T05:08:54.025 INFO:tasks.workunit.client.1.smithi192.stdout:7/132: dwrite d3/f21 [0,4194304] 0 2024-04-12T05:08:54.026 INFO:tasks.workunit.client.1.smithi192.stdout:6/238: creat d3/d8/d10/f58 x:0 0 0 2024-04-12T05:08:54.031 INFO:tasks.workunit.client.0.smithi113.stdout:7/297: rmdir d0/dd/d1b 39 2024-04-12T05:08:54.031 INFO:tasks.workunit.client.1.smithi192.stdout:8/129: symlink d6/da/l2b 0 2024-04-12T05:08:54.036 INFO:tasks.workunit.client.0.smithi113.stdout:2/171: mknod d8/d9/d27/c2d 0 2024-04-12T05:08:54.038 INFO:tasks.workunit.client.1.smithi192.stdout:6/239: mkdir d3/d8/d56/d59 0 2024-04-12T05:08:54.038 INFO:tasks.workunit.client.1.smithi192.stdout:4/170: truncate d1/f29 1739401 0 2024-04-12T05:08:54.040 INFO:tasks.workunit.client.0.smithi113.stdout:3/238: rmdir d0/d13/d15/d28 39 2024-04-12T05:08:54.043 INFO:tasks.workunit.client.1.smithi192.stdout:8/130: dwrite d6/df/f19 [4194304,4194304] 0 2024-04-12T05:08:54.045 INFO:tasks.workunit.client.0.smithi113.stdout:7/298: unlink d0/dc/d6/l2e 0 2024-04-12T05:08:54.045 INFO:tasks.workunit.client.0.smithi113.stdout:7/299: chown d0/dc/d6 14 1 2024-04-12T05:08:54.047 INFO:tasks.workunit.client.1.smithi192.stdout:4/171: mkdir d1/d20/d37 0 2024-04-12T05:08:54.047 INFO:tasks.workunit.client.1.smithi192.stdout:9/107: write fd [268487,58382] 0 2024-04-12T05:08:54.048 INFO:tasks.workunit.client.0.smithi113.stdout:8/331: mknod d9/d16/d65/c67 0 2024-04-12T05:08:54.050 INFO:tasks.workunit.client.0.smithi113.stdout:8/332: write d9/fb [173755,86368] 0 2024-04-12T05:08:54.050 INFO:tasks.workunit.client.0.smithi113.stdout:9/343: creat d1/f79 x:0 0 0 2024-04-12T05:08:54.052 INFO:tasks.workunit.client.0.smithi113.stdout:2/172: rmdir d8/d16 39 2024-04-12T05:08:54.052 INFO:tasks.workunit.client.1.smithi192.stdout:6/240: rename d3/c4 to d3/d8/d22/c5a 0 2024-04-12T05:08:54.052 INFO:tasks.workunit.client.1.smithi192.stdout:6/241: dread - d3/d8/d22/f2e zero size 2024-04-12T05:08:54.052 INFO:tasks.workunit.client.1.smithi192.stdout:3/136: dread d5/d13/f18 [0,4194304] 0 2024-04-12T05:08:54.053 INFO:tasks.workunit.client.1.smithi192.stdout:4/172: symlink d1/da/l38 0 2024-04-12T05:08:54.055 INFO:tasks.workunit.client.1.smithi192.stdout:9/108: mkdir d11/d25 0 2024-04-12T05:08:54.055 INFO:tasks.workunit.client.0.smithi113.stdout:4/288: rmdir da/d4f/d2f 0 2024-04-12T05:08:54.056 INFO:tasks.workunit.client.1.smithi192.stdout:9/109: chown d11/d15 2367 1 2024-04-12T05:08:54.056 INFO:tasks.workunit.client.1.smithi192.stdout:6/242: creat d3/d8/d48/f5b x:0 0 0 2024-04-12T05:08:54.059 INFO:tasks.workunit.client.0.smithi113.stdout:7/300: dread d0/dc/d6/f16 [0,4194304] 0 2024-04-12T05:08:54.060 INFO:tasks.workunit.client.1.smithi192.stdout:6/243: rmdir d3/d21/d24 39 2024-04-12T05:08:54.061 INFO:tasks.workunit.client.1.smithi192.stdout:6/244: fdatasync d3/d8/d48/f50 0 2024-04-12T05:08:54.062 INFO:tasks.workunit.client.0.smithi113.stdout:9/344: rmdir d1/df 39 2024-04-12T05:08:54.062 INFO:tasks.workunit.client.1.smithi192.stdout:3/137: dwrite d5/d13/f2a [0,4194304] 0 2024-04-12T05:08:54.065 INFO:tasks.workunit.client.0.smithi113.stdout:3/239: dwrite d0/d13/d15/d28/f38 [0,4194304] 0 2024-04-12T05:08:54.067 INFO:tasks.workunit.client.0.smithi113.stdout:8/333: mkdir d9/d16/d1c/d59/d68 0 2024-04-12T05:08:54.069 INFO:tasks.workunit.client.0.smithi113.stdout:2/173: dwrite d8/d9/de/f22 [0,4194304] 0 2024-04-12T05:08:54.074 INFO:tasks.workunit.client.0.smithi113.stdout:7/301: creat d0/dc/d50/f5f x:0 0 0 2024-04-12T05:08:54.075 INFO:tasks.workunit.client.1.smithi192.stdout:5/181: sync 2024-04-12T05:08:54.076 INFO:tasks.workunit.client.1.smithi192.stdout:3/138: dwrite d5/fd [0,4194304] 0 2024-04-12T05:08:54.079 INFO:tasks.workunit.client.1.smithi192.stdout:9/110: link f6 d11/d15/f26 0 2024-04-12T05:08:54.079 INFO:tasks.workunit.client.0.smithi113.stdout:3/240: dread d0/d13/d15/f20 [0,4194304] 0 2024-04-12T05:08:54.079 INFO:tasks.workunit.client.0.smithi113.stdout:2/174: write d8/d16/f1d [4785300,38945] 0 2024-04-12T05:08:54.080 INFO:tasks.workunit.client.1.smithi192.stdout:4/173: getdents d1/d20 0 2024-04-12T05:08:54.081 INFO:tasks.workunit.client.0.smithi113.stdout:3/241: chown d0/d13/d47 219982 1 2024-04-12T05:08:54.082 INFO:tasks.workunit.client.0.smithi113.stdout:3/242: fsync d0/fa 0 2024-04-12T05:08:54.083 INFO:tasks.workunit.client.1.smithi192.stdout:7/133: dwrite d3/d11/f18 [0,4194304] 0 2024-04-12T05:08:54.083 INFO:tasks.workunit.client.0.smithi113.stdout:3/243: dread - d0/d13/f1f zero size 2024-04-12T05:08:54.084 INFO:tasks.workunit.client.0.smithi113.stdout:8/334: creat d9/d16/d1e/d26/d32/d48/f69 x:0 0 0 2024-04-12T05:08:54.085 INFO:tasks.workunit.client.0.smithi113.stdout:3/244: dread - d0/d13/d47/f4a zero size 2024-04-12T05:08:54.085 INFO:tasks.workunit.client.0.smithi113.stdout:7/302: write d0/dc/d6/d1a/f34 [355525,28724] 0 2024-04-12T05:08:54.087 INFO:tasks.workunit.client.0.smithi113.stdout:9/345: dread d1/df/d4b/d6c/f6b [0,4194304] 0 2024-04-12T05:08:54.087 INFO:tasks.workunit.client.0.smithi113.stdout:4/289: mkdir da/d5a 0 2024-04-12T05:08:54.088 INFO:tasks.workunit.client.0.smithi113.stdout:2/175: mknod d8/d9/d1a/c2e 0 2024-04-12T05:08:54.090 INFO:tasks.workunit.client.0.smithi113.stdout:3/245: chown d0/d13/d15/d28/c44 3 1 2024-04-12T05:08:54.090 INFO:tasks.workunit.client.1.smithi192.stdout:9/111: unlink f9 0 2024-04-12T05:08:54.090 INFO:tasks.workunit.client.1.smithi192.stdout:3/139: unlink d5/f23 0 2024-04-12T05:08:54.091 INFO:tasks.workunit.client.0.smithi113.stdout:8/335: creat d9/d16/d65/f6a x:0 0 0 2024-04-12T05:08:54.092 INFO:tasks.workunit.client.0.smithi113.stdout:9/346: mkdir d1/d27/d7a 0 2024-04-12T05:08:54.094 INFO:tasks.workunit.client.1.smithi192.stdout:6/245: link d3/d8/d10/c19 d3/d21/c5c 0 2024-04-12T05:08:54.094 INFO:tasks.workunit.client.0.smithi113.stdout:4/290: creat da/dd/d3a/f5b x:0 0 0 2024-04-12T05:08:54.094 INFO:tasks.workunit.client.0.smithi113.stdout:9/347: stat d1/df/d4b/d63/d3d/f49 0 2024-04-12T05:08:54.095 INFO:tasks.workunit.client.1.smithi192.stdout:7/134: creat d3/d7/d27/f30 x:0 0 0 2024-04-12T05:08:54.095 INFO:tasks.workunit.client.0.smithi113.stdout:7/303: mknod d0/dd/d11/d44/c60 0 2024-04-12T05:08:54.097 INFO:tasks.workunit.client.0.smithi113.stdout:7/304: readlink d0/dc/d6/l43 0 2024-04-12T05:08:54.099 INFO:tasks.workunit.client.1.smithi192.stdout:5/182: dwrite d0/f3d [0,4194304] 0 2024-04-12T05:08:54.100 INFO:tasks.workunit.client.1.smithi192.stdout:7/135: write d3/d7/d27/f30 [972017,103448] 0 2024-04-12T05:08:54.101 INFO:tasks.workunit.client.1.smithi192.stdout:5/183: stat d0/d15/l44 0 2024-04-12T05:08:54.101 INFO:tasks.workunit.client.0.smithi113.stdout:9/348: creat d1/df/d29/f7b x:0 0 0 2024-04-12T05:08:54.102 INFO:tasks.workunit.client.0.smithi113.stdout:3/246: mknod d0/d13/d15/d28/d2a/d34/c4d 0 2024-04-12T05:08:54.102 INFO:tasks.workunit.client.0.smithi113.stdout:4/291: rename da/dd/d3a/f44 to da/d4a/f5c 0 2024-04-12T05:08:54.102 INFO:tasks.workunit.client.1.smithi192.stdout:4/174: link d1/da/l38 d1/d1d/l39 0 2024-04-12T05:08:54.102 INFO:tasks.workunit.client.1.smithi192.stdout:5/184: fdatasync d0/d21/f36 0 2024-04-12T05:08:54.103 INFO:tasks.workunit.client.0.smithi113.stdout:7/305: creat d0/dc/d6/f61 x:0 0 0 2024-04-12T05:08:54.104 INFO:tasks.workunit.client.1.smithi192.stdout:6/246: rmdir d3 39 2024-04-12T05:08:54.104 INFO:tasks.workunit.client.0.smithi113.stdout:8/336: link d9/l43 d9/d16/d1e/d26/d32/d48/l6b 0 2024-04-12T05:08:54.106 INFO:tasks.workunit.client.1.smithi192.stdout:7/136: creat d3/d7/d27/f31 x:0 0 0 2024-04-12T05:08:54.106 INFO:tasks.workunit.client.1.smithi192.stdout:3/140: dwrite d5/f19 [0,4194304] 0 2024-04-12T05:08:54.109 INFO:tasks.workunit.client.1.smithi192.stdout:4/175: symlink d1/d1d/l3a 0 2024-04-12T05:08:54.109 INFO:tasks.workunit.client.1.smithi192.stdout:4/176: dread - d1/da/d1c/f36 zero size 2024-04-12T05:08:54.111 INFO:tasks.workunit.client.0.smithi113.stdout:9/349: mknod d1/df/d4b/d63/d3d/c7c 0 2024-04-12T05:08:54.112 INFO:tasks.workunit.client.1.smithi192.stdout:5/185: unlink d0/d15/c1a 0 2024-04-12T05:08:54.112 INFO:tasks.workunit.client.0.smithi113.stdout:4/292: creat da/d4f/f5d x:0 0 0 2024-04-12T05:08:54.115 INFO:tasks.workunit.client.0.smithi113.stdout:7/306: creat d0/dd/d1b/f62 x:0 0 0 2024-04-12T05:08:54.119 INFO:tasks.workunit.client.1.smithi192.stdout:9/112: write f7 [656292,102742] 0 2024-04-12T05:08:54.120 INFO:tasks.workunit.client.1.smithi192.stdout:7/137: symlink d3/d7/d15/d22/l32 0 2024-04-12T05:08:54.121 INFO:tasks.workunit.client.1.smithi192.stdout:7/138: write d3/f21 [1856721,39565] 0 2024-04-12T05:08:54.121 INFO:tasks.workunit.client.0.smithi113.stdout:4/293: creat da/d3c/d43/f5e x:0 0 0 2024-04-12T05:08:54.121 INFO:tasks.workunit.client.1.smithi192.stdout:4/177: mknod d1/d20/d37/c3b 0 2024-04-12T05:08:54.122 INFO:tasks.workunit.client.0.smithi113.stdout:7/307: read - d0/dd/d1b/f62 zero size 2024-04-12T05:08:54.122 INFO:tasks.workunit.client.0.smithi113.stdout:8/337: dwrite d9/f47 [0,4194304] 0 2024-04-12T05:08:54.122 INFO:tasks.workunit.client.0.smithi113.stdout:3/247: dwrite d0/f9 [0,4194304] 0 2024-04-12T05:08:54.125 INFO:tasks.workunit.client.1.smithi192.stdout:3/141: write d5/d13/d20/f24 [3737589,34050] 0 2024-04-12T05:08:54.127 INFO:tasks.workunit.client.1.smithi192.stdout:6/247: creat d3/d8/f5d x:0 0 0 2024-04-12T05:08:54.129 INFO:tasks.workunit.client.1.smithi192.stdout:4/178: symlink d1/d20/l3c 0 2024-04-12T05:08:54.133 INFO:tasks.workunit.client.1.smithi192.stdout:3/142: chown d5/f1c 27085755 1 2024-04-12T05:08:54.134 INFO:tasks.workunit.client.1.smithi192.stdout:9/113: dread d11/d15/f26 [0,4194304] 0 2024-04-12T05:08:54.134 INFO:tasks.workunit.client.1.smithi192.stdout:5/186: link d0/d15/f23 d0/d15/d24/f46 0 2024-04-12T05:08:54.136 INFO:tasks.workunit.client.1.smithi192.stdout:5/187: readlink d0/d15/d24/d29/l2d 0 2024-04-12T05:08:54.136 INFO:tasks.workunit.client.0.smithi113.stdout:0/295: sync 2024-04-12T05:08:54.136 INFO:tasks.workunit.client.0.smithi113.stdout:3/248: rename d0/d13/d15/f20 to d0/d13/d15/d28/d2a/d2d/f4e 0 2024-04-12T05:08:54.136 INFO:tasks.workunit.client.0.smithi113.stdout:2/176: dread d8/d9/de/f10 [0,4194304] 0 2024-04-12T05:08:54.138 INFO:tasks.workunit.client.1.smithi192.stdout:4/179: dread d1/da/f31 [0,4194304] 0 2024-04-12T05:08:54.138 INFO:tasks.workunit.client.0.smithi113.stdout:9/350: dread d1/d27/f5d [0,4194304] 0 2024-04-12T05:08:54.138 INFO:tasks.workunit.client.1.smithi192.stdout:5/188: read d0/d15/d24/f25 [3631159,53036] 0 2024-04-12T05:08:54.139 INFO:tasks.workunit.client.0.smithi113.stdout:4/294: dwrite da/fb [4194304,4194304] 0 2024-04-12T05:08:54.139 INFO:tasks.workunit.client.1.smithi192.stdout:9/114: rename d11/d14/l1c to d11/l27 0 2024-04-12T05:08:54.140 INFO:tasks.workunit.client.1.smithi192.stdout:4/180: truncate d1/f33 749611 0 2024-04-12T05:08:54.140 INFO:tasks.workunit.client.0.smithi113.stdout:7/308: link d0/dc/d6/l1d d0/dc/l63 0 2024-04-12T05:08:54.142 INFO:tasks.workunit.client.0.smithi113.stdout:0/296: mknod d1/d6/d16/d2a/d2c/c54 0 2024-04-12T05:08:54.144 INFO:tasks.workunit.client.0.smithi113.stdout:2/177: mknod d8/d9/de/c2f 0 2024-04-12T05:08:54.145 INFO:tasks.workunit.client.1.smithi192.stdout:3/143: symlink d5/d13/d1a/l32 0 2024-04-12T05:08:54.146 INFO:tasks.workunit.client.1.smithi192.stdout:3/144: write d5/f2d [304815,106958] 0 2024-04-12T05:08:54.146 INFO:tasks.workunit.client.1.smithi192.stdout:9/115: symlink d11/l28 0 2024-04-12T05:08:54.147 INFO:tasks.workunit.client.1.smithi192.stdout:4/181: dread d1/f11 [0,4194304] 0 2024-04-12T05:08:54.148 INFO:tasks.workunit.client.0.smithi113.stdout:4/295: write da/dd/d1b/f28 [4976346,47676] 0 2024-04-12T05:08:54.149 INFO:tasks.workunit.client.1.smithi192.stdout:5/189: truncate d0/f1e 3115056 0 2024-04-12T05:08:54.150 INFO:tasks.workunit.client.1.smithi192.stdout:6/248: write d3/d8/f25 [3846818,83885] 0 2024-04-12T05:08:54.152 INFO:tasks.workunit.client.1.smithi192.stdout:5/190: chown d0/c9 10479600 1 2024-04-12T05:08:54.154 INFO:tasks.workunit.client.0.smithi113.stdout:3/249: dread d0/d13/d15/d28/d2a/d2d/f35 [0,4194304] 0 2024-04-12T05:08:54.155 INFO:tasks.workunit.client.0.smithi113.stdout:4/296: write da/dd/df/d1d/f46 [3746615,40826] 0 2024-04-12T05:08:54.155 INFO:tasks.workunit.client.1.smithi192.stdout:4/182: mknod d1/d20/d37/c3d 0 2024-04-12T05:08:54.157 INFO:tasks.workunit.client.0.smithi113.stdout:2/178: link d8/fd d8/d9/f30 0 2024-04-12T05:08:54.158 INFO:tasks.workunit.client.1.smithi192.stdout:6/249: readlink d3/d21/d24/l44 0 2024-04-12T05:08:54.159 INFO:tasks.workunit.client.1.smithi192.stdout:9/116: dwrite d11/d14/f1a [0,4194304] 0 2024-04-12T05:08:54.159 INFO:tasks.workunit.client.0.smithi113.stdout:4/297: dread - da/d4f/f5d zero size 2024-04-12T05:08:54.159 INFO:tasks.workunit.client.1.smithi192.stdout:5/191: mkdir d0/d15/d24/d47 0 2024-04-12T05:08:54.161 INFO:tasks.workunit.client.0.smithi113.stdout:7/309: dwrite d0/d26/f45 [4194304,4194304] 0 2024-04-12T05:08:54.161 INFO:tasks.workunit.client.0.smithi113.stdout:0/297: rename d1/d6/d16/d2a/l51 to d1/d6/d16/d2a/d50/d52/l55 0 2024-04-12T05:08:54.162 INFO:tasks.workunit.client.0.smithi113.stdout:8/338: dread d9/fb [0,4194304] 0 2024-04-12T05:08:54.163 INFO:tasks.workunit.client.0.smithi113.stdout:0/298: fsync d1/d9/f2b 0 2024-04-12T05:08:54.164 INFO:tasks.workunit.client.0.smithi113.stdout:2/179: creat d8/d9/de/f31 x:0 0 0 2024-04-12T05:08:54.166 INFO:tasks.workunit.client.1.smithi192.stdout:4/183: dread d1/f11 [0,4194304] 0 2024-04-12T05:08:54.167 INFO:tasks.workunit.client.0.smithi113.stdout:2/180: write d8/d16/f1d [8630379,55333] 0 2024-04-12T05:08:54.168 INFO:tasks.workunit.client.0.smithi113.stdout:3/250: rename d0/d13/d15/d28/d2a/d2d to d0/d13/d47/d4f 0 2024-04-12T05:08:54.169 INFO:tasks.workunit.client.1.smithi192.stdout:6/250: dread d3/d26/f46 [0,4194304] 0 2024-04-12T05:08:54.170 INFO:tasks.workunit.client.1.smithi192.stdout:9/117: dwrite d11/d15/d21/f24 [0,4194304] 0 2024-04-12T05:08:54.171 INFO:tasks.workunit.client.0.smithi113.stdout:4/298: chown da/dd/d1b/c57 4723 1 2024-04-12T05:08:54.172 INFO:tasks.workunit.client.1.smithi192.stdout:9/118: write f10 [2073385,97596] 0 2024-04-12T05:08:54.172 INFO:tasks.workunit.client.0.smithi113.stdout:7/310: creat d0/dd/d2d/f64 x:0 0 0 2024-04-12T05:08:54.173 INFO:tasks.workunit.client.0.smithi113.stdout:9/351: dread d1/df/f36 [0,4194304] 0 2024-04-12T05:08:54.174 INFO:tasks.workunit.client.1.smithi192.stdout:5/192: link d0/c1f d0/d15/d32/d45/c48 0 2024-04-12T05:08:54.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:52.669672+0000 mgr.smithi192.ocppyo (mgr.24357) 22 : cluster [DBG] pgmap v16: 65 pgs: 65 active+clean; 668 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 3.7 MiB/s rd, 68 MiB/s wr, 196 op/s 2024-04-12T05:08:54.175 INFO:tasks.workunit.client.0.smithi113.stdout:0/299: rename d1/l4b to d1/d6/da/l56 0 2024-04-12T05:08:54.175 INFO:tasks.workunit.client.1.smithi192.stdout:9/119: truncate f10 2485411 0 2024-04-12T05:08:54.176 INFO:tasks.workunit.client.0.smithi113.stdout:2/181: mkdir d8/d9/de/d15/d32 0 2024-04-12T05:08:54.177 INFO:tasks.workunit.client.1.smithi192.stdout:4/184: link d1/f33 d1/f3e 0 2024-04-12T05:08:54.180 INFO:tasks.workunit.client.0.smithi113.stdout:0/300: write d1/d6/d16/d2a/f3f [3664880,125161] 0 2024-04-12T05:08:54.180 INFO:tasks.workunit.client.0.smithi113.stdout:3/251: creat d0/d13/d47/f50 x:0 0 0 2024-04-12T05:08:54.180 INFO:tasks.workunit.client.1.smithi192.stdout:3/145: write d5/f14 [3921837,114404] 0 2024-04-12T05:08:54.184 INFO:tasks.workunit.client.1.smithi192.stdout:6/251: dwrite d3/d5/f51 [0,4194304] 0 2024-04-12T05:08:54.184 INFO:tasks.workunit.client.0.smithi113.stdout:4/299: unlink da/dd/df/d14/f18 0 2024-04-12T05:08:54.185 INFO:tasks.workunit.client.0.smithi113.stdout:7/311: dread d0/f5a [0,4194304] 0 2024-04-12T05:08:54.187 INFO:tasks.workunit.client.1.smithi192.stdout:3/146: symlink d5/d13/d20/d26/l33 0 2024-04-12T05:08:54.188 INFO:tasks.workunit.client.0.smithi113.stdout:3/252: creat d0/d13/f51 x:0 0 0 2024-04-12T05:08:54.188 INFO:tasks.workunit.client.0.smithi113.stdout:4/300: write da/dd/df/d1d/f4c [882315,35163] 0 2024-04-12T05:08:54.188 INFO:tasks.workunit.client.1.smithi192.stdout:6/252: truncate d3/d8/d10/f58 125605 0 2024-04-12T05:08:54.188 INFO:tasks.workunit.client.0.smithi113.stdout:7/312: readlink d0/dd/d1b/l4e 0 2024-04-12T05:08:54.190 INFO:tasks.workunit.client.1.smithi192.stdout:6/253: truncate d3/f37 682382 0 2024-04-12T05:08:54.190 INFO:tasks.workunit.client.0.smithi113.stdout:7/313: write d0/dd/d1b/f62 [518211,82161] 0 2024-04-12T05:08:54.191 INFO:tasks.workunit.client.1.smithi192.stdout:9/120: dread d11/d14/f1a [0,4194304] 0 2024-04-12T05:08:54.191 INFO:tasks.workunit.client.1.smithi192.stdout:9/121: chown fa 14803 1 2024-04-12T05:08:54.192 INFO:tasks.workunit.client.0.smithi113.stdout:7/314: chown d0/dc/d6/l43 12734455 1 2024-04-12T05:08:54.197 INFO:tasks.workunit.client.1.smithi192.stdout:4/185: fsync d1/f33 0 2024-04-12T05:08:54.197 INFO:tasks.workunit.client.1.smithi192.stdout:6/254: mknod d3/d23/c5e 0 2024-04-12T05:08:54.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:52.669672+0000 mgr.smithi192.ocppyo (mgr.24357) 22 : cluster [DBG] pgmap v16: 65 pgs: 65 active+clean; 668 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 3.7 MiB/s rd, 68 MiB/s wr, 196 op/s 2024-04-12T05:08:54.198 INFO:tasks.workunit.client.1.smithi192.stdout:9/122: symlink d11/d25/l29 0 2024-04-12T05:08:54.199 INFO:tasks.workunit.client.0.smithi113.stdout:2/182: getdents d8/d9 0 2024-04-12T05:08:54.199 INFO:tasks.workunit.client.0.smithi113.stdout:9/352: dread d1/d4/d38/f62 [0,4194304] 0 2024-04-12T05:08:54.199 INFO:tasks.workunit.client.1.smithi192.stdout:3/147: symlink d5/d13/d1a/l34 0 2024-04-12T05:08:54.200 INFO:tasks.workunit.client.0.smithi113.stdout:0/301: rename d1/d6/d16/d2a/d2c/c4e to d1/c57 0 2024-04-12T05:08:54.200 INFO:tasks.workunit.client.0.smithi113.stdout:4/301: link c8 da/d24/c5f 0 2024-04-12T05:08:54.200 INFO:tasks.workunit.client.1.smithi192.stdout:3/148: write d5/fd [429884,105434] 0 2024-04-12T05:08:54.201 INFO:tasks.workunit.client.1.smithi192.stdout:4/186: symlink d1/da/d1c/l3f 0 2024-04-12T05:08:54.202 INFO:tasks.workunit.client.1.smithi192.stdout:6/255: symlink d3/dc/l5f 0 2024-04-12T05:08:54.204 INFO:tasks.workunit.client.0.smithi113.stdout:4/302: mkdir da/d24/d60 0 2024-04-12T05:08:54.204 INFO:tasks.workunit.client.1.smithi192.stdout:9/123: symlink d11/l2a 0 2024-04-12T05:08:54.205 INFO:tasks.workunit.client.0.smithi113.stdout:3/253: dwrite d0/d13/d47/d4f/f35 [4194304,4194304] 0 2024-04-12T05:08:54.206 INFO:tasks.workunit.client.0.smithi113.stdout:2/183: mknod d8/d9/d1a/c33 0 2024-04-12T05:08:54.206 INFO:tasks.workunit.client.1.smithi192.stdout:3/149: mknod d5/c35 0 2024-04-12T05:08:54.206 INFO:tasks.workunit.client.1.smithi192.stdout:5/193: dread d0/fa [0,4194304] 0 2024-04-12T05:08:54.207 INFO:tasks.workunit.client.0.smithi113.stdout:7/315: rename d0/dd/d2d/l37 to d0/dd/d11/d44/l65 0 2024-04-12T05:08:54.208 INFO:tasks.workunit.client.0.smithi113.stdout:9/353: creat d1/d4/d38/d65/f7d x:0 0 0 2024-04-12T05:08:54.209 INFO:tasks.workunit.client.1.smithi192.stdout:5/194: rmdir d0/d15/d24 39 2024-04-12T05:08:54.210 INFO:tasks.workunit.client.0.smithi113.stdout:7/316: write d0/dd/d1b/f25 [1281414,76756] 0 2024-04-12T05:08:54.212 INFO:tasks.workunit.client.1.smithi192.stdout:6/256: dwrite d3/d26/f31 [0,4194304] 0 2024-04-12T05:08:54.214 INFO:tasks.workunit.client.0.smithi113.stdout:4/303: creat da/d24/d60/f61 x:0 0 0 2024-04-12T05:08:54.214 INFO:tasks.workunit.client.0.smithi113.stdout:9/354: mkdir d1/df/d4b/d63/d3d/d6f/d7e 0 2024-04-12T05:08:54.215 INFO:tasks.workunit.client.1.smithi192.stdout:5/195: write d0/d21/f26 [1191460,33708] 0 2024-04-12T05:08:54.216 INFO:tasks.workunit.client.0.smithi113.stdout:2/184: creat d8/d9/de/d15/d24/f34 x:0 0 0 2024-04-12T05:08:54.218 INFO:tasks.workunit.client.0.smithi113.stdout:9/355: write d1/df/f4d [951450,89556] 0 2024-04-12T05:08:54.220 INFO:tasks.workunit.client.0.smithi113.stdout:3/254: dwrite d0/d13/d15/d28/d2a/d34/f43 [0,4194304] 0 2024-04-12T05:08:54.220 INFO:tasks.workunit.client.1.smithi192.stdout:5/196: rmdir d0 39 2024-04-12T05:08:54.221 INFO:tasks.workunit.client.0.smithi113.stdout:9/356: chown d1/d4/d30/l5e 22418864 1 2024-04-12T05:08:54.221 INFO:tasks.workunit.client.1.smithi192.stdout:3/150: dwrite d5/d13/f2e [0,4194304] 0 2024-04-12T05:08:54.224 INFO:tasks.workunit.client.0.smithi113.stdout:3/255: write d0/f12 [3598996,90305] 0 2024-04-12T05:08:54.225 INFO:tasks.workunit.client.0.smithi113.stdout:4/304: dread da/dd/f48 [0,4194304] 0 2024-04-12T05:08:54.226 INFO:tasks.workunit.client.1.smithi192.stdout:6/257: dwrite d3/f37 [0,4194304] 0 2024-04-12T05:08:54.227 INFO:tasks.workunit.client.0.smithi113.stdout:2/185: creat d8/d9/de/d12/f35 x:0 0 0 2024-04-12T05:08:54.228 INFO:tasks.workunit.client.0.smithi113.stdout:4/305: write da/d4f/f5d [343031,119798] 0 2024-04-12T05:08:54.228 INFO:tasks.workunit.client.0.smithi113.stdout:9/357: creat d1/df/d4b/f7f x:0 0 0 2024-04-12T05:08:54.230 INFO:tasks.workunit.client.1.smithi192.stdout:4/187: truncate d1/d1d/f2d 3965167 0 2024-04-12T05:08:54.231 INFO:tasks.workunit.client.0.smithi113.stdout:0/302: rename d1/d6/da/f3a to d1/d6/f58 0 2024-04-12T05:08:54.231 INFO:tasks.workunit.client.0.smithi113.stdout:9/358: write d1/d4/d5/f12 [3166725,25392] 0 2024-04-12T05:08:54.231 INFO:tasks.workunit.client.1.smithi192.stdout:3/151: mkdir d5/d13/d20/d36 0 2024-04-12T05:08:54.232 INFO:tasks.workunit.client.0.smithi113.stdout:7/317: truncate d0/dc/d6/d1a/f3a 2508474 0 2024-04-12T05:08:54.233 INFO:tasks.workunit.client.1.smithi192.stdout:3/152: stat d5/d13/f2f 0 2024-04-12T05:08:54.233 INFO:tasks.workunit.client.1.smithi192.stdout:6/258: rmdir d3/d23 39 2024-04-12T05:08:54.233 INFO:tasks.workunit.client.1.smithi192.stdout:4/188: truncate d1/f3e 802319 0 2024-04-12T05:08:54.234 INFO:tasks.workunit.client.0.smithi113.stdout:4/306: creat da/de/f62 x:0 0 0 2024-04-12T05:08:54.235 INFO:tasks.workunit.client.0.smithi113.stdout:9/359: fsync d1/df/d4b/d6c/f59 0 2024-04-12T05:08:54.235 INFO:tasks.workunit.client.1.smithi192.stdout:5/197: symlink d0/d15/d32/d45/l49 0 2024-04-12T05:08:54.235 INFO:tasks.workunit.client.1.smithi192.stdout:6/259: chown d3/d5/l3f 1681 1 2024-04-12T05:08:54.236 INFO:tasks.workunit.client.1.smithi192.stdout:6/260: dread - d3/f1e zero size 2024-04-12T05:08:54.236 INFO:tasks.workunit.client.0.smithi113.stdout:9/360: dread - d1/d4/d5/d17/f74 zero size 2024-04-12T05:08:54.237 INFO:tasks.workunit.client.1.smithi192.stdout:3/153: symlink d5/l37 0 2024-04-12T05:08:54.238 INFO:tasks.workunit.client.0.smithi113.stdout:2/186: symlink d8/d9/de/d15/d24/l36 0 2024-04-12T05:08:54.241 INFO:tasks.workunit.client.1.smithi192.stdout:5/198: getdents d0/d15/d24/d47 0 2024-04-12T05:08:54.242 INFO:tasks.workunit.client.1.smithi192.stdout:5/199: dread - d0/d15/f41 zero size 2024-04-12T05:08:54.244 INFO:tasks.workunit.client.1.smithi192.stdout:5/200: symlink d0/d15/d32/l4a 0 2024-04-12T05:08:54.245 INFO:tasks.workunit.client.0.smithi113.stdout:3/256: rename d0/d13/c48 to d0/d13/d15/d28/d2a/d34/c52 0 2024-04-12T05:08:54.245 INFO:tasks.workunit.client.0.smithi113.stdout:4/307: mknod da/d3c/d43/c63 0 2024-04-12T05:08:54.245 INFO:tasks.workunit.client.0.smithi113.stdout:2/187: creat d8/d16/f37 x:0 0 0 2024-04-12T05:08:54.246 INFO:tasks.workunit.client.0.smithi113.stdout:9/361: symlink d1/df/d4b/d63/l80 0 2024-04-12T05:08:54.246 INFO:tasks.workunit.client.0.smithi113.stdout:0/303: stat d1/f22 0 2024-04-12T05:08:54.248 INFO:tasks.workunit.client.0.smithi113.stdout:3/257: stat d0/d13/c4b 0 2024-04-12T05:08:54.249 INFO:tasks.workunit.client.0.smithi113.stdout:7/318: link d0/dc/d6/f28 d0/dd/f66 0 2024-04-12T05:08:54.249 INFO:tasks.workunit.client.0.smithi113.stdout:3/258: fsync d0/f10 0 2024-04-12T05:08:54.250 INFO:tasks.workunit.client.0.smithi113.stdout:0/304: mkdir d1/d43/d59 0 2024-04-12T05:08:54.251 INFO:tasks.workunit.client.1.smithi192.stdout:3/154: truncate d5/d13/f17 921782 0 2024-04-12T05:08:54.252 INFO:tasks.workunit.client.0.smithi113.stdout:9/362: creat d1/d4/d5/d6/d15/f81 x:0 0 0 2024-04-12T05:08:54.254 INFO:tasks.workunit.client.0.smithi113.stdout:2/188: rename d8/d9/de/d15/d32 to d8/d9/de/d15/d38 0 2024-04-12T05:08:54.255 INFO:tasks.workunit.client.0.smithi113.stdout:5/272: sync 2024-04-12T05:08:54.255 INFO:tasks.workunit.client.0.smithi113.stdout:1/330: sync 2024-04-12T05:08:54.255 INFO:tasks.workunit.client.0.smithi113.stdout:6/259: sync 2024-04-12T05:08:54.256 INFO:tasks.workunit.client.0.smithi113.stdout:3/259: unlink d0/d13/c4b 0 2024-04-12T05:08:54.258 INFO:tasks.workunit.client.1.smithi192.stdout:3/155: dread d5/d13/f17 [0,4194304] 0 2024-04-12T05:08:54.258 INFO:tasks.workunit.client.1.smithi192.stdout:4/189: dwrite d1/d1d/f2d [0,4194304] 0 2024-04-12T05:08:54.259 INFO:tasks.workunit.client.0.smithi113.stdout:0/305: creat d1/d9/f5a x:0 0 0 2024-04-12T05:08:54.261 INFO:tasks.workunit.client.0.smithi113.stdout:3/260: fdatasync d0/d13/d47/f50 0 2024-04-12T05:08:54.262 INFO:tasks.workunit.client.0.smithi113.stdout:5/273: chown d0/d10/d27/d33/f4f 86 1 2024-04-12T05:08:54.263 INFO:tasks.workunit.client.1.smithi192.stdout:4/190: write d1/da/f31 [2345062,60735] 0 2024-04-12T05:08:54.264 INFO:tasks.workunit.client.0.smithi113.stdout:2/189: rename d8/d9/f1f to d8/d9/de/d15/d38/f39 0 2024-04-12T05:08:54.264 INFO:tasks.workunit.client.0.smithi113.stdout:7/319: getdents d0/dd/d11/d44/d5c 0 2024-04-12T05:08:54.266 INFO:tasks.workunit.client.0.smithi113.stdout:1/331: dread d10/d21/d23/f36 [0,4194304] 0 2024-04-12T05:08:54.266 INFO:tasks.workunit.client.0.smithi113.stdout:6/260: symlink d8/de/d13/d11/d36/d3d/d4c/d31/l56 0 2024-04-12T05:08:54.266 INFO:tasks.workunit.client.1.smithi192.stdout:3/156: getdents d5/d13 0 2024-04-12T05:08:54.268 INFO:tasks.workunit.client.1.smithi192.stdout:4/191: symlink d1/d20/d37/l40 0 2024-04-12T05:08:54.269 INFO:tasks.workunit.client.0.smithi113.stdout:7/320: chown d0/d26/l4d 31250 1 2024-04-12T05:08:54.270 INFO:tasks.workunit.client.0.smithi113.stdout:3/261: unlink d0/d13/f51 0 2024-04-12T05:08:54.271 INFO:tasks.workunit.client.0.smithi113.stdout:3/262: fsync d0/d13/d15/d28/f3f 0 2024-04-12T05:08:54.272 INFO:tasks.workunit.client.0.smithi113.stdout:2/190: creat d8/d9/d1a/f3a x:0 0 0 2024-04-12T05:08:54.272 INFO:tasks.workunit.client.1.smithi192.stdout:5/201: dread d0/f1e [0,4194304] 0 2024-04-12T05:08:54.273 INFO:tasks.workunit.client.0.smithi113.stdout:9/363: dread d1/d4/d5/f58 [0,4194304] 0 2024-04-12T05:08:54.274 INFO:tasks.workunit.client.0.smithi113.stdout:6/261: creat d8/de/d13/d2b/f57 x:0 0 0 2024-04-12T05:08:54.275 INFO:tasks.workunit.client.1.smithi192.stdout:4/192: getdents d1/d20/d37 0 2024-04-12T05:08:54.276 INFO:tasks.workunit.client.1.smithi192.stdout:5/202: dread d0/fa [0,4194304] 0 2024-04-12T05:08:54.277 INFO:tasks.workunit.client.1.smithi192.stdout:3/157: truncate d5/fd 3820071 0 2024-04-12T05:08:54.277 INFO:tasks.workunit.client.1.smithi192.stdout:4/193: read d1/f2 [771275,28416] 0 2024-04-12T05:08:54.278 INFO:tasks.workunit.client.1.smithi192.stdout:4/194: dread - d1/da/f2f zero size 2024-04-12T05:08:54.279 INFO:tasks.workunit.client.1.smithi192.stdout:3/158: symlink d5/d13/d1a/l38 0 2024-04-12T05:08:54.280 INFO:tasks.workunit.client.1.smithi192.stdout:3/159: write d5/f19 [1699082,75690] 0 2024-04-12T05:08:54.281 INFO:tasks.workunit.client.0.smithi113.stdout:2/191: creat d8/d9/de/d15/d24/f3b x:0 0 0 2024-04-12T05:08:54.282 INFO:tasks.workunit.client.1.smithi192.stdout:5/203: rename d0/d15/d24/d29/f30 to d0/d15/d24/f4b 0 2024-04-12T05:08:54.282 INFO:tasks.workunit.client.0.smithi113.stdout:3/263: rename d0/d13/d15/d28/d2a/c40 to d0/d13/c53 0 2024-04-12T05:08:54.283 INFO:tasks.workunit.client.0.smithi113.stdout:2/192: write d8/d9/de/f31 [151915,69944] 0 2024-04-12T05:08:54.283 INFO:tasks.workunit.client.0.smithi113.stdout:1/332: getdents d10/d55 0 2024-04-12T05:08:54.285 INFO:tasks.workunit.client.0.smithi113.stdout:3/264: write d0/d13/d15/d28/f3f [416656,32820] 0 2024-04-12T05:08:54.286 INFO:tasks.workunit.client.0.smithi113.stdout:2/193: dread - d8/d9/de/d15/d38/f39 zero size 2024-04-12T05:08:54.288 INFO:tasks.workunit.client.0.smithi113.stdout:1/333: truncate d10/d2e/d31/d33/f50 843849 0 2024-04-12T05:08:54.288 INFO:tasks.workunit.client.0.smithi113.stdout:2/194: stat d8/d9/de/d15/d24 0 2024-04-12T05:08:54.289 INFO:tasks.workunit.client.0.smithi113.stdout:2/195: chown d8/d9/de/d15/d24/l36 391445 1 2024-04-12T05:08:54.290 INFO:tasks.workunit.client.0.smithi113.stdout:0/306: dread d1/fe [0,4194304] 0 2024-04-12T05:08:54.291 INFO:tasks.workunit.client.0.smithi113.stdout:3/265: fsync d0/f18 0 2024-04-12T05:08:54.291 INFO:tasks.workunit.client.0.smithi113.stdout:0/307: dread - d1/d9/f29 zero size 2024-04-12T05:08:54.292 INFO:tasks.workunit.client.0.smithi113.stdout:4/308: truncate da/dd/df/f10 4915136 0 2024-04-12T05:08:54.292 INFO:tasks.workunit.client.0.smithi113.stdout:0/308: chown d1/d9 2 1 2024-04-12T05:08:54.293 INFO:tasks.workunit.client.1.smithi192.stdout:5/204: write d0/d15/d24/f46 [3667426,95508] 0 2024-04-12T05:08:54.294 INFO:tasks.workunit.client.0.smithi113.stdout:1/334: symlink d10/d41/l5d 0 2024-04-12T05:08:54.295 INFO:tasks.workunit.client.1.smithi192.stdout:4/195: dread d1/da/d1c/f28 [0,4194304] 0 2024-04-12T05:08:54.298 INFO:tasks.workunit.client.0.smithi113.stdout:7/321: link d0/dd/d2d/c56 d0/dd/d2d/c67 0 2024-04-12T05:08:54.298 INFO:tasks.workunit.client.0.smithi113.stdout:2/196: mkdir d8/d9/de/d12/d3c 0 2024-04-12T05:08:54.299 INFO:tasks.workunit.client.0.smithi113.stdout:1/335: stat d10/d2e/d31/f59 0 2024-04-12T05:08:54.300 INFO:tasks.workunit.client.1.smithi192.stdout:4/196: dread d1/f18 [0,4194304] 0 2024-04-12T05:08:54.301 INFO:tasks.workunit.client.0.smithi113.stdout:2/197: read d8/d16/f1d [8121370,83891] 0 2024-04-12T05:08:54.302 INFO:tasks.workunit.client.0.smithi113.stdout:4/309: mknod da/dd/c64 0 2024-04-12T05:08:54.302 INFO:tasks.workunit.client.1.smithi192.stdout:5/205: dwrite d0/f1b [0,4194304] 0 2024-04-12T05:08:54.303 INFO:tasks.workunit.client.0.smithi113.stdout:7/322: dread d0/d26/f45 [4194304,4194304] 0 2024-04-12T05:08:54.303 INFO:tasks.workunit.client.0.smithi113.stdout:0/309: creat d1/d43/d59/f5b x:0 0 0 2024-04-12T05:08:54.303 INFO:tasks.workunit.client.0.smithi113.stdout:4/310: write da/dd/d1b/f2c [2890258,111374] 0 2024-04-12T05:08:54.305 INFO:tasks.workunit.client.1.smithi192.stdout:4/197: dread d1/f26 [0,4194304] 0 2024-04-12T05:08:54.305 INFO:tasks.workunit.client.0.smithi113.stdout:4/311: write da/d24/d60/f61 [317853,60120] 0 2024-04-12T05:08:54.306 INFO:tasks.workunit.client.1.smithi192.stdout:4/198: readlink d1/l13 0 2024-04-12T05:08:54.306 INFO:tasks.workunit.client.0.smithi113.stdout:7/323: truncate d0/dd/d1b/f23 195561 0 2024-04-12T05:08:54.306 INFO:tasks.workunit.client.0.smithi113.stdout:4/312: chown da/d2b 15 1 2024-04-12T05:08:54.307 INFO:tasks.workunit.client.0.smithi113.stdout:9/364: write d1/df/d4b/d6c/f6b [499146,46613] 0 2024-04-12T05:08:54.307 INFO:tasks.workunit.client.1.smithi192.stdout:5/206: link d0/d21/f2f d0/d15/d32/f4c 0 2024-04-12T05:08:54.308 INFO:tasks.workunit.client.0.smithi113.stdout:4/313: fsync da/d24/f52 0 2024-04-12T05:08:54.308 INFO:tasks.workunit.client.0.smithi113.stdout:5/274: truncate d0/d10/d27/f25 2554822 0 2024-04-12T05:08:54.309 INFO:tasks.workunit.client.1.smithi192.stdout:4/199: creat d1/da/f41 x:0 0 0 2024-04-12T05:08:54.310 INFO:tasks.workunit.client.0.smithi113.stdout:1/336: chown d10/d2e/l4b 834198 1 2024-04-12T05:08:54.311 INFO:tasks.workunit.client.1.smithi192.stdout:4/200: symlink d1/l42 0 2024-04-12T05:08:54.315 INFO:tasks.workunit.client.0.smithi113.stdout:6/262: truncate d8/de/f15 17916 0 2024-04-12T05:08:54.316 INFO:tasks.workunit.client.1.smithi192.stdout:5/207: dwrite d0/d15/d24/f25 [0,4194304] 0 2024-04-12T05:08:54.317 INFO:tasks.workunit.client.0.smithi113.stdout:0/310: creat d1/d6/da/f5c x:0 0 0 2024-04-12T05:08:54.318 INFO:tasks.workunit.client.0.smithi113.stdout:0/311: dread - d1/d6/da/f4d zero size 2024-04-12T05:08:54.319 INFO:tasks.workunit.client.0.smithi113.stdout:0/312: write d1/d9/f1a [5197939,52717] 0 2024-04-12T05:08:54.320 INFO:tasks.workunit.client.1.smithi192.stdout:5/208: creat d0/d15/d24/d47/f4d x:0 0 0 2024-04-12T05:08:54.320 INFO:tasks.workunit.client.1.smithi192.stdout:4/201: write d1/f33 [1612240,98552] 0 2024-04-12T05:08:54.320 INFO:tasks.workunit.client.1.smithi192.stdout:5/209: readlink d0/d15/d32/d45/l49 0 2024-04-12T05:08:54.321 INFO:tasks.workunit.client.1.smithi192.stdout:5/210: read d0/fa [3040518,106192] 0 2024-04-12T05:08:54.367 INFO:tasks.workunit.client.1.smithi192.stdout:0/78: sync 2024-04-12T05:08:54.369 INFO:tasks.workunit.client.1.smithi192.stdout:5/211: rmdir d0/d15/d24/d29 39 2024-04-12T05:08:54.369 INFO:tasks.workunit.client.1.smithi192.stdout:4/202: mknod d1/d20/d37/c43 0 2024-04-12T05:08:54.371 INFO:tasks.workunit.client.1.smithi192.stdout:5/212: mknod d0/c4e 0 2024-04-12T05:08:54.373 INFO:tasks.workunit.client.1.smithi192.stdout:0/79: rename d0/c5 to d0/c18 0 2024-04-12T05:08:54.373 INFO:tasks.workunit.client.0.smithi113.stdout:9/365: rename d1/c8 to d1/df/d4b/d6c/c82 0 2024-04-12T05:08:54.373 INFO:tasks.workunit.client.1.smithi192.stdout:5/213: truncate d0/f11 2206960 0 2024-04-12T05:08:54.374 INFO:tasks.workunit.client.1.smithi192.stdout:0/80: truncate d0/f16 613643 0 2024-04-12T05:08:54.374 INFO:tasks.workunit.client.0.smithi113.stdout:5/275: creat d0/d10/d27/d1d/d38/f51 x:0 0 0 2024-04-12T05:08:54.375 INFO:tasks.workunit.client.0.smithi113.stdout:1/337: symlink d10/d2e/d31/l5e 0 2024-04-12T05:08:54.376 INFO:tasks.workunit.client.0.smithi113.stdout:5/276: dread - d0/d10/d27/d33/f4a zero size 2024-04-12T05:08:54.377 INFO:tasks.workunit.client.1.smithi192.stdout:4/203: getdents d1 0 2024-04-12T05:08:54.378 INFO:tasks.workunit.client.1.smithi192.stdout:5/214: mknod d0/d15/d24/c4f 0 2024-04-12T05:08:54.379 INFO:tasks.workunit.client.0.smithi113.stdout:3/266: getdents d0/d13/d47/d4f 0 2024-04-12T05:08:54.379 INFO:tasks.workunit.client.1.smithi192.stdout:4/204: write d1/d1d/f2d [416324,84853] 0 2024-04-12T05:08:54.380 INFO:tasks.workunit.client.1.smithi192.stdout:0/81: mkdir d0/d6/d15/d19 0 2024-04-12T05:08:54.380 INFO:tasks.workunit.client.0.smithi113.stdout:2/198: write d8/d9/f30 [1382188,27277] 0 2024-04-12T05:08:54.381 INFO:tasks.workunit.client.1.smithi192.stdout:4/205: truncate d1/da/f2f 427921 0 2024-04-12T05:08:54.384 INFO:tasks.workunit.client.1.smithi192.stdout:4/206: creat d1/d1d/f44 x:0 0 0 2024-04-12T05:08:54.385 INFO:tasks.workunit.client.0.smithi113.stdout:1/338: read d10/d41/f4a [3366786,64010] 0 2024-04-12T05:08:54.385 INFO:tasks.workunit.client.0.smithi113.stdout:5/277: creat d0/d10/d27/d1d/d38/f52 x:0 0 0 2024-04-12T05:08:54.387 INFO:tasks.workunit.client.0.smithi113.stdout:5/278: chown d0/d10/d27/d33/f4a 3042 1 2024-04-12T05:08:54.388 INFO:tasks.workunit.client.0.smithi113.stdout:9/366: dread d1/d4/f18 [0,4194304] 0 2024-04-12T05:08:54.389 INFO:tasks.workunit.client.0.smithi113.stdout:2/199: unlink d8/d9/fc 0 2024-04-12T05:08:54.389 INFO:tasks.workunit.client.1.smithi192.stdout:5/215: getdents d0/d15/d24/d29 0 2024-04-12T05:08:54.391 INFO:tasks.workunit.client.0.smithi113.stdout:9/367: write d1/d4/d5/d6/f3b [1459748,98808] 0 2024-04-12T05:08:54.391 INFO:tasks.workunit.client.1.smithi192.stdout:5/216: mknod d0/d15/d24/d29/c50 0 2024-04-12T05:08:54.392 INFO:tasks.workunit.client.1.smithi192.stdout:4/207: dwrite d1/f11 [0,4194304] 0 2024-04-12T05:08:54.395 INFO:tasks.workunit.client.1.smithi192.stdout:4/208: creat d1/da/d1c/f45 x:0 0 0 2024-04-12T05:08:54.395 INFO:tasks.workunit.client.0.smithi113.stdout:5/279: dwrite d0/d10/d27/d13/f23 [0,4194304] 0 2024-04-12T05:08:54.398 INFO:tasks.workunit.client.1.smithi192.stdout:4/209: unlink d1/da/d1c/f27 0 2024-04-12T05:08:54.404 INFO:tasks.workunit.client.1.smithi192.stdout:5/217: dwrite d0/f7 [0,4194304] 0 2024-04-12T05:08:54.405 INFO:tasks.workunit.client.1.smithi192.stdout:4/210: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:54.406 INFO:tasks.workunit.client.1.smithi192.stdout:5/218: read - d0/d15/f41 zero size 2024-04-12T05:08:54.407 INFO:tasks.workunit.client.1.smithi192.stdout:5/219: write d0/d15/f23 [2185696,46687] 0 2024-04-12T05:08:54.409 INFO:tasks.workunit.client.1.smithi192.stdout:4/211: write d1/f2 [3246236,26071] 0 2024-04-12T05:08:54.411 INFO:tasks.workunit.client.1.smithi192.stdout:4/212: read - d1/da/f41 zero size 2024-04-12T05:08:54.415 INFO:tasks.workunit.client.1.smithi192.stdout:4/213: creat d1/da/d1c/f46 x:0 0 0 2024-04-12T05:08:54.421 INFO:tasks.workunit.client.1.smithi192.stdout:5/220: dwrite d0/fa [0,4194304] 0 2024-04-12T05:08:54.423 INFO:tasks.workunit.client.1.smithi192.stdout:4/214: dwrite d1/f3e [0,4194304] 0 2024-04-12T05:08:54.424 INFO:tasks.workunit.client.1.smithi192.stdout:5/221: rmdir d0/d15/d24/d47 39 2024-04-12T05:08:54.425 INFO:tasks.workunit.client.1.smithi192.stdout:4/215: write d1/f33 [1636767,122929] 0 2024-04-12T05:08:54.427 INFO:tasks.workunit.client.1.smithi192.stdout:5/222: write d0/f1b [1729107,6371] 0 2024-04-12T05:08:54.430 INFO:tasks.workunit.client.0.smithi113.stdout:8/339: sync 2024-04-12T05:08:54.432 INFO:tasks.workunit.client.1.smithi192.stdout:5/223: chown d0/c12 44 1 2024-04-12T05:08:54.433 INFO:tasks.workunit.client.1.smithi192.stdout:5/224: chown d0/d15/d24/c4f 0 1 2024-04-12T05:08:54.435 INFO:tasks.workunit.client.1.smithi192.stdout:4/216: getdents d1/d20 0 2024-04-12T05:08:54.439 INFO:tasks.workunit.client.1.smithi192.stdout:4/217: symlink d1/da/l47 0 2024-04-12T05:08:54.441 INFO:tasks.workunit.client.1.smithi192.stdout:5/225: creat d0/d15/d24/f51 x:0 0 0 2024-04-12T05:08:54.443 INFO:tasks.workunit.client.1.smithi192.stdout:5/226: rename d0/f1e to d0/f52 0 2024-04-12T05:08:54.445 INFO:tasks.workunit.client.1.smithi192.stdout:4/218: dwrite d1/f11 [0,4194304] 0 2024-04-12T05:08:54.446 INFO:tasks.workunit.client.1.smithi192.stdout:5/227: truncate d0/f20 1040713 0 2024-04-12T05:08:54.448 INFO:tasks.workunit.client.1.smithi192.stdout:4/219: unlink d1/da/d1c/f46 0 2024-04-12T05:08:54.466 INFO:tasks.workunit.client.1.smithi192.stdout:5/228: dwrite d0/d15/d24/f4b [0,4194304] 0 2024-04-12T05:08:54.467 INFO:tasks.workunit.client.1.smithi192.stdout:5/229: dread - d0/d15/d24/f51 zero size 2024-04-12T05:08:54.470 INFO:tasks.workunit.client.1.smithi192.stdout:5/230: symlink d0/l53 0 2024-04-12T05:08:54.473 INFO:tasks.workunit.client.1.smithi192.stdout:5/231: creat d0/d15/d24/d47/f54 x:0 0 0 2024-04-12T05:08:54.475 INFO:tasks.workunit.client.0.smithi113.stdout:4/314: rename da/d24/d60/f61 to da/f65 0 2024-04-12T05:08:54.476 INFO:tasks.workunit.client.0.smithi113.stdout:6/263: rename d8/de/d13/d11/d36/d3d to d8/de/d13/d11/d36/d3d/d4c/d26/d58 22 2024-04-12T05:08:54.476 INFO:tasks.workunit.client.0.smithi113.stdout:1/339: creat d10/d2e/d31/d5c/f5f x:0 0 0 2024-04-12T05:08:54.477 INFO:tasks.workunit.client.0.smithi113.stdout:7/324: truncate d0/f5a 2264792 0 2024-04-12T05:08:54.477 INFO:tasks.workunit.client.1.smithi192.stdout:5/232: dread d0/f20 [0,4194304] 0 2024-04-12T05:08:54.478 INFO:tasks.workunit.client.0.smithi113.stdout:6/264: dread - d8/f40 zero size 2024-04-12T05:08:54.480 INFO:tasks.workunit.client.0.smithi113.stdout:0/313: dwrite d1/d6/d34/f40 [0,4194304] 0 2024-04-12T05:08:54.481 INFO:tasks.workunit.client.0.smithi113.stdout:5/280: symlink d0/d10/d27/d1d/d38/d44/l53 0 2024-04-12T05:08:54.481 INFO:tasks.workunit.client.1.smithi192.stdout:5/233: rename d0/c2c to d0/d15/c55 0 2024-04-12T05:08:54.483 INFO:tasks.workunit.client.0.smithi113.stdout:3/267: write d0/d13/d15/d28/f45 [675524,45531] 0 2024-04-12T05:08:54.483 INFO:tasks.workunit.client.0.smithi113.stdout:0/314: write d1/d9/f1a [1071143,120725] 0 2024-04-12T05:08:54.484 INFO:tasks.workunit.client.1.smithi192.stdout:5/234: symlink d0/l56 0 2024-04-12T05:08:54.485 INFO:tasks.workunit.client.0.smithi113.stdout:7/325: creat d0/dc/f68 x:0 0 0 2024-04-12T05:08:54.487 INFO:tasks.workunit.client.0.smithi113.stdout:9/368: getdents d1/df/d4b/d63/d3d/d6f/d7e 0 2024-04-12T05:08:54.487 INFO:tasks.workunit.client.0.smithi113.stdout:8/340: mkdir d9/d16/d1e/d3a/d6c 0 2024-04-12T05:08:54.487 INFO:tasks.workunit.client.0.smithi113.stdout:1/340: fsync d10/d21/f47 0 2024-04-12T05:08:54.487 INFO:tasks.workunit.client.1.smithi192.stdout:5/235: mknod d0/d15/c57 0 2024-04-12T05:08:54.488 INFO:tasks.workunit.client.1.smithi192.stdout:5/236: fdatasync d0/d15/f23 0 2024-04-12T05:08:54.489 INFO:tasks.workunit.client.0.smithi113.stdout:5/281: mknod d0/d10/d27/d1d/d3f/c54 0 2024-04-12T05:08:54.489 INFO:tasks.workunit.client.0.smithi113.stdout:6/265: dread d8/de/d13/d11/d36/f23 [0,4194304] 0 2024-04-12T05:08:54.490 INFO:tasks.workunit.client.0.smithi113.stdout:2/200: creat d8/f3d x:0 0 0 2024-04-12T05:08:54.491 INFO:tasks.workunit.client.1.smithi192.stdout:5/237: fdatasync d0/f20 0 2024-04-12T05:08:54.492 INFO:tasks.workunit.client.0.smithi113.stdout:4/315: creat da/f66 x:0 0 0 2024-04-12T05:08:54.493 INFO:tasks.workunit.client.0.smithi113.stdout:2/201: read - d8/d9/d1a/f3a zero size 2024-04-12T05:08:54.493 INFO:tasks.workunit.client.1.smithi192.stdout:5/238: symlink d0/d15/d32/l58 0 2024-04-12T05:08:54.493 INFO:tasks.workunit.client.0.smithi113.stdout:6/266: chown d8/de/d13/d11/d36/d3d/d4c/c37 670767557 1 2024-04-12T05:08:54.494 INFO:tasks.workunit.client.0.smithi113.stdout:7/326: rename d0/dd/d1b/l4e to d0/dd/d2d/l69 0 2024-04-12T05:08:54.495 INFO:tasks.workunit.client.0.smithi113.stdout:6/267: chown d8/de/d13/d11/c3a 0 1 2024-04-12T05:08:54.496 INFO:tasks.workunit.client.0.smithi113.stdout:3/268: symlink d0/d13/d15/l54 0 2024-04-12T05:08:54.497 INFO:tasks.workunit.client.0.smithi113.stdout:6/268: write d8/de/d13/d11/d36/d3d/d4c/d26/f51 [1269833,58185] 0 2024-04-12T05:08:54.499 INFO:tasks.workunit.client.0.smithi113.stdout:4/316: rmdir da/dd/d1b 39 2024-04-12T05:08:54.504 INFO:tasks.workunit.client.0.smithi113.stdout:1/341: link d10/d41/f45 d10/d41/f60 0 2024-04-12T05:08:54.505 INFO:tasks.workunit.client.0.smithi113.stdout:1/342: chown d10/d2e/d31/d33 237885777 1 2024-04-12T05:08:54.505 INFO:tasks.workunit.client.0.smithi113.stdout:3/269: dwrite d0/d13/d15/d28/d2a/d34/f43 [0,4194304] 0 2024-04-12T05:08:54.507 INFO:tasks.workunit.client.0.smithi113.stdout:6/269: fdatasync d8/de/d13/d11/d36/d3d/d4c/f28 0 2024-04-12T05:08:54.507 INFO:tasks.workunit.client.0.smithi113.stdout:1/343: chown d10/d14/f17 696997 1 2024-04-12T05:08:54.507 INFO:tasks.workunit.client.0.smithi113.stdout:3/270: chown d0/d13/d47/d4f/l37 65914 1 2024-04-12T05:08:54.509 INFO:tasks.workunit.client.0.smithi113.stdout:9/369: rename d1/d4/d38/d65/c32 to d1/df/d4b/d63/c83 0 2024-04-12T05:08:54.510 INFO:tasks.workunit.client.0.smithi113.stdout:5/282: creat d0/f55 x:0 0 0 2024-04-12T05:08:54.511 INFO:tasks.workunit.client.0.smithi113.stdout:7/327: dread d0/d26/f45 [4194304,4194304] 0 2024-04-12T05:08:54.512 INFO:tasks.workunit.client.0.smithi113.stdout:2/202: dread d8/d16/f19 [0,4194304] 0 2024-04-12T05:08:54.514 INFO:tasks.workunit.client.0.smithi113.stdout:4/317: unlink da/dd/df/f10 0 2024-04-12T05:08:54.516 INFO:tasks.workunit.client.0.smithi113.stdout:6/270: write d8/de/d13/fb [1650228,3647] 0 2024-04-12T05:08:54.516 INFO:tasks.workunit.client.0.smithi113.stdout:0/315: write d1/d6/d16/f24 [1626776,10889] 0 2024-04-12T05:08:54.519 INFO:tasks.workunit.client.1.smithi192.stdout:2/183: sync 2024-04-12T05:08:54.519 INFO:tasks.workunit.client.1.smithi192.stdout:1/170: sync 2024-04-12T05:08:54.519 INFO:tasks.workunit.client.1.smithi192.stdout:8/131: sync 2024-04-12T05:08:54.520 INFO:tasks.workunit.client.0.smithi113.stdout:1/344: creat d10/d2e/d31/d33/f61 x:0 0 0 2024-04-12T05:08:54.521 INFO:tasks.workunit.client.0.smithi113.stdout:3/271: write d0/d13/d47/d4f/f4c [2081520,16128] 0 2024-04-12T05:08:54.523 INFO:tasks.workunit.client.1.smithi192.stdout:8/132: mknod d6/c2c 0 2024-04-12T05:08:54.524 INFO:tasks.workunit.client.1.smithi192.stdout:1/171: symlink d0/d29/l2f 0 2024-04-12T05:08:54.524 INFO:tasks.workunit.client.0.smithi113.stdout:9/370: creat d1/df/d4b/d6c/f84 x:0 0 0 2024-04-12T05:08:54.525 INFO:tasks.workunit.client.0.smithi113.stdout:7/328: creat d0/d26/f6a x:0 0 0 2024-04-12T05:08:54.526 INFO:tasks.workunit.client.1.smithi192.stdout:1/172: fsync d0/f3 0 2024-04-12T05:08:54.526 INFO:tasks.workunit.client.0.smithi113.stdout:8/341: write d9/d16/d1e/d3a/f2a [5209415,21772] 0 2024-04-12T05:08:54.527 INFO:tasks.workunit.client.1.smithi192.stdout:1/173: readlink d0/l22 0 2024-04-12T05:08:54.527 INFO:tasks.workunit.client.1.smithi192.stdout:2/184: dwrite d1/d2d/f32 [0,4194304] 0 2024-04-12T05:08:54.528 INFO:tasks.workunit.client.0.smithi113.stdout:4/318: mknod da/d4a/c67 0 2024-04-12T05:08:54.529 INFO:tasks.workunit.client.1.smithi192.stdout:2/185: write d1/d8/d3a/f3b [489088,82133] 0 2024-04-12T05:08:54.530 INFO:tasks.workunit.client.0.smithi113.stdout:0/316: fdatasync d1/f22 0 2024-04-12T05:08:54.531 INFO:tasks.workunit.client.1.smithi192.stdout:8/133: mknod d6/da/c2d 0 2024-04-12T05:08:54.531 INFO:tasks.workunit.client.1.smithi192.stdout:8/134: chown d6/f1d 5 1 2024-04-12T05:08:54.532 INFO:tasks.workunit.client.0.smithi113.stdout:1/345: dread fb [0,4194304] 0 2024-04-12T05:08:54.533 INFO:tasks.workunit.client.0.smithi113.stdout:7/329: write d0/dd/d2d/f53 [977920,60563] 0 2024-04-12T05:08:54.534 INFO:tasks.workunit.client.0.smithi113.stdout:6/271: dread d8/de/d13/fa [0,4194304] 0 2024-04-12T05:08:54.534 INFO:tasks.workunit.client.0.smithi113.stdout:3/272: mknod d0/d13/d47/d4f/c55 0 2024-04-12T05:08:54.534 INFO:tasks.workunit.client.1.smithi192.stdout:2/186: truncate d1/d8/d17/f20 238410 0 2024-04-12T05:08:54.535 INFO:tasks.workunit.client.0.smithi113.stdout:8/342: chown d9/d16/d1e/d3a/f5e 108139 1 2024-04-12T05:08:54.535 INFO:tasks.workunit.client.0.smithi113.stdout:0/317: chown d1/d6/f58 25 1 2024-04-12T05:08:54.536 INFO:tasks.workunit.client.1.smithi192.stdout:8/135: chown d6/l27 29268 1 2024-04-12T05:08:54.539 INFO:tasks.workunit.client.0.smithi113.stdout:5/283: dread d0/d10/f16 [0,4194304] 0 2024-04-12T05:08:54.539 INFO:tasks.workunit.client.0.smithi113.stdout:4/319: mknod da/d24/d60/c68 0 2024-04-12T05:08:54.539 INFO:tasks.workunit.client.0.smithi113.stdout:6/272: dread - d8/de/d13/d11/d36/d2c/f41 zero size 2024-04-12T05:08:54.540 INFO:tasks.workunit.client.1.smithi192.stdout:1/174: dwrite d0/fc [4194304,4194304] 0 2024-04-12T05:08:54.540 INFO:tasks.workunit.client.0.smithi113.stdout:2/203: dread d8/fd [0,4194304] 0 2024-04-12T05:08:54.542 INFO:tasks.workunit.client.0.smithi113.stdout:6/273: write d8/f2d [4638717,28858] 0 2024-04-12T05:08:54.543 INFO:tasks.workunit.client.1.smithi192.stdout:2/187: link d1/d8/d17/f2a d1/f44 0 2024-04-12T05:08:54.545 INFO:tasks.workunit.client.0.smithi113.stdout:1/346: rename d10/d41/l5d to d10/d21/d3c/l62 0 2024-04-12T05:08:54.545 INFO:tasks.workunit.client.0.smithi113.stdout:7/330: dwrite d0/dc/f42 [4194304,4194304] 0 2024-04-12T05:08:54.545 INFO:tasks.workunit.client.0.smithi113.stdout:6/274: write d8/de/d13/d11/f43 [2508319,96461] 0 2024-04-12T05:08:54.546 INFO:tasks.workunit.client.1.smithi192.stdout:2/188: creat d1/d8/d17/d19/f45 x:0 0 0 2024-04-12T05:08:54.547 INFO:tasks.workunit.client.0.smithi113.stdout:3/273: fdatasync d0/d13/d47/d4f/f4e 0 2024-04-12T05:08:54.548 INFO:tasks.workunit.client.0.smithi113.stdout:7/331: stat d0/dd/d11/d44 0 2024-04-12T05:08:54.549 INFO:tasks.workunit.client.0.smithi113.stdout:3/274: fdatasync d0/f6 0 2024-04-12T05:08:54.553 INFO:tasks.workunit.client.1.smithi192.stdout:2/189: creat d1/d8/f46 x:0 0 0 2024-04-12T05:08:54.553 INFO:tasks.workunit.client.1.smithi192.stdout:8/136: dwrite d6/df/f15 [0,4194304] 0 2024-04-12T05:08:54.553 INFO:tasks.workunit.client.1.smithi192.stdout:1/175: dwrite d0/fc [0,4194304] 0 2024-04-12T05:08:54.554 INFO:tasks.workunit.client.0.smithi113.stdout:9/371: link d1/d4/d38/d65/c1b d1/d27/c85 0 2024-04-12T05:08:54.554 INFO:tasks.workunit.client.0.smithi113.stdout:4/320: creat da/d24/f69 x:0 0 0 2024-04-12T05:08:54.556 INFO:tasks.workunit.client.1.smithi192.stdout:2/190: write d1/d8/d3a/de/f2f [619045,22524] 0 2024-04-12T05:08:54.556 INFO:tasks.workunit.client.0.smithi113.stdout:4/321: write da/de/f62 [947807,83291] 0 2024-04-12T05:08:54.556 INFO:tasks.workunit.client.0.smithi113.stdout:2/204: rmdir d8/d9/de 39 2024-04-12T05:08:54.560 INFO:tasks.workunit.client.1.smithi192.stdout:1/176: creat d0/f30 x:0 0 0 2024-04-12T05:08:54.565 INFO:tasks.workunit.client.0.smithi113.stdout:7/332: creat d0/dd/d11/d44/f6b x:0 0 0 2024-04-12T05:08:54.566 INFO:tasks.workunit.client.1.smithi192.stdout:2/191: getdents d1/d8/d17 0 2024-04-12T05:08:54.567 INFO:tasks.workunit.client.0.smithi113.stdout:9/372: dwrite d1/d4/f77 [0,4194304] 0 2024-04-12T05:08:54.569 INFO:tasks.workunit.client.0.smithi113.stdout:4/322: dwrite da/dd/f21 [0,4194304] 0 2024-04-12T05:08:54.569 INFO:tasks.workunit.client.0.smithi113.stdout:7/333: chown d0/dd/d11/d44/l4b 1900 1 2024-04-12T05:08:54.570 INFO:tasks.workunit.client.0.smithi113.stdout:4/323: stat da/d26/c51 0 2024-04-12T05:08:54.570 INFO:tasks.workunit.client.1.smithi192.stdout:2/192: dread - d1/d8/d3a/f26 zero size 2024-04-12T05:08:54.570 INFO:tasks.workunit.client.1.smithi192.stdout:8/137: dwrite d6/d10/f11 [0,4194304] 0 2024-04-12T05:08:54.571 INFO:tasks.workunit.client.0.smithi113.stdout:1/347: mknod d10/d41/d5b/c63 0 2024-04-12T05:08:54.581 INFO:tasks.workunit.client.1.smithi192.stdout:8/138: symlink d6/d21/l2e 0 2024-04-12T05:08:54.581 INFO:tasks.workunit.client.1.smithi192.stdout:2/193: write d1/d8/f41 [214264,2771] 0 2024-04-12T05:08:54.582 INFO:tasks.workunit.client.0.smithi113.stdout:3/275: mknod d0/d13/d15/c56 0 2024-04-12T05:08:54.583 INFO:tasks.workunit.client.0.smithi113.stdout:3/276: read d0/d13/d15/f2e [2138867,105848] 0 2024-04-12T05:08:54.585 INFO:tasks.workunit.client.0.smithi113.stdout:9/373: unlink d1/df/c43 0 2024-04-12T05:08:54.585 INFO:tasks.workunit.client.1.smithi192.stdout:2/194: truncate d1/d8/d3a/de/f30 918909 0 2024-04-12T05:08:54.587 INFO:tasks.workunit.client.0.smithi113.stdout:9/374: chown d1/d4/d38/f62 375 1 2024-04-12T05:08:54.588 INFO:tasks.workunit.client.1.smithi192.stdout:1/177: dwrite d0/d1b/f2b [0,4194304] 0 2024-04-12T05:08:54.588 INFO:tasks.workunit.client.0.smithi113.stdout:5/284: rename d0/d10/d27/d13/f4b to d0/d10/f56 0 2024-04-12T05:08:54.588 INFO:tasks.workunit.client.0.smithi113.stdout:2/205: write d8/d9/de/f10 [4666100,108018] 0 2024-04-12T05:08:54.590 INFO:tasks.workunit.client.0.smithi113.stdout:2/206: dread - d8/d9/d1a/f3a zero size 2024-04-12T05:08:54.591 INFO:tasks.workunit.client.1.smithi192.stdout:1/178: fsync d0/f24 0 2024-04-12T05:08:54.591 INFO:tasks.workunit.client.0.smithi113.stdout:1/348: creat d10/d2e/d31/d33/f64 x:0 0 0 2024-04-12T05:08:54.591 INFO:tasks.workunit.client.0.smithi113.stdout:6/275: creat d8/de/d17/f59 x:0 0 0 2024-04-12T05:08:54.593 INFO:tasks.workunit.client.1.smithi192.stdout:8/139: dwrite d6/da/fc [0,4194304] 0 2024-04-12T05:08:54.594 INFO:tasks.workunit.client.0.smithi113.stdout:8/343: getdents d9/d34 0 2024-04-12T05:08:54.597 INFO:tasks.workunit.client.0.smithi113.stdout:0/318: getdents d1 0 2024-04-12T05:08:54.597 INFO:tasks.workunit.client.0.smithi113.stdout:7/334: mkdir d0/d6c 0 2024-04-12T05:08:54.598 INFO:tasks.workunit.client.1.smithi192.stdout:1/179: mknod d0/d1b/c31 0 2024-04-12T05:08:54.598 INFO:tasks.workunit.client.0.smithi113.stdout:3/277: dwrite d0/d13/d47/d4f/f35 [4194304,4194304] 0 2024-04-12T05:08:54.599 INFO:tasks.workunit.client.0.smithi113.stdout:7/335: fsync d0/f48 0 2024-04-12T05:08:54.603 INFO:tasks.workunit.client.1.smithi192.stdout:8/140: dwrite d6/da/f1e [0,4194304] 0 2024-04-12T05:08:54.603 INFO:tasks.workunit.client.0.smithi113.stdout:5/285: dwrite d0/d10/d27/f26 [0,4194304] 0 2024-04-12T05:08:54.606 INFO:tasks.workunit.client.0.smithi113.stdout:0/319: write d1/d6/d34/f40 [4225228,71682] 0 2024-04-12T05:08:54.609 INFO:tasks.workunit.client.0.smithi113.stdout:5/286: write d0/fc [2260386,61256] 0 2024-04-12T05:08:54.614 INFO:tasks.workunit.client.1.smithi192.stdout:2/195: dread d1/f44 [0,4194304] 0 2024-04-12T05:08:54.614 INFO:tasks.workunit.client.1.smithi192.stdout:2/196: write d1/d8/f35 [506197,51600] 0 2024-04-12T05:08:54.614 INFO:tasks.workunit.client.0.smithi113.stdout:6/276: symlink d8/de/d13/d2b/l5a 0 2024-04-12T05:08:54.618 INFO:tasks.workunit.client.0.smithi113.stdout:3/278: creat d0/f57 x:0 0 0 2024-04-12T05:08:54.620 INFO:tasks.workunit.client.1.smithi192.stdout:8/141: dwrite d6/f18 [0,4194304] 0 2024-04-12T05:08:54.625 INFO:tasks.workunit.client.0.smithi113.stdout:8/344: symlink d9/d16/d1c/d59/d68/l6d 0 2024-04-12T05:08:54.625 INFO:tasks.workunit.client.0.smithi113.stdout:3/279: stat d0/d13/f1f 0 2024-04-12T05:08:54.625 INFO:tasks.workunit.client.0.smithi113.stdout:9/375: link d1/d4/d38/d65/c4f d1/d4/d38/c86 0 2024-04-12T05:08:54.625 INFO:tasks.workunit.client.0.smithi113.stdout:2/207: dwrite d8/d16/f19 [0,4194304] 0 2024-04-12T05:08:54.625 INFO:tasks.workunit.client.0.smithi113.stdout:4/324: fdatasync da/dd/f21 0 2024-04-12T05:08:54.627 INFO:tasks.workunit.client.1.smithi192.stdout:2/197: symlink d1/d8/d17/d3c/l47 0 2024-04-12T05:08:54.631 INFO:tasks.workunit.client.1.smithi192.stdout:8/142: dwrite d6/f1d [0,4194304] 0 2024-04-12T05:08:54.634 INFO:tasks.workunit.client.0.smithi113.stdout:6/277: mkdir d8/de/d13/d11/d36/d3d/d4c/d26/d5b 0 2024-04-12T05:08:54.634 INFO:tasks.workunit.client.0.smithi113.stdout:4/325: chown da/dd/df 117389 1 2024-04-12T05:08:54.634 INFO:tasks.workunit.client.0.smithi113.stdout:8/345: write d9/d16/d1e/d3a/f5e [2023803,125444] 0 2024-04-12T05:08:54.637 INFO:tasks.workunit.client.1.smithi192.stdout:8/143: creat d6/d1f/f2f x:0 0 0 2024-04-12T05:08:54.638 INFO:tasks.workunit.client.0.smithi113.stdout:9/376: mkdir d1/d4/d87 0 2024-04-12T05:08:54.638 INFO:tasks.workunit.client.0.smithi113.stdout:5/287: dread d0/d10/d27/d13/f23 [0,4194304] 0 2024-04-12T05:08:54.638 INFO:tasks.workunit.client.0.smithi113.stdout:3/280: symlink d0/d13/d15/d28/l58 0 2024-04-12T05:08:54.642 INFO:tasks.workunit.client.1.smithi192.stdout:2/198: dwrite d1/d8/d3a/f26 [0,4194304] 0 2024-04-12T05:08:54.643 INFO:tasks.workunit.client.0.smithi113.stdout:2/208: rename d8/d9/f2c to d8/d9/de/d15/d24/f3e 0 2024-04-12T05:08:54.644 INFO:tasks.workunit.client.1.smithi192.stdout:8/144: creat d6/da/f30 x:0 0 0 2024-04-12T05:08:54.644 INFO:tasks.workunit.client.0.smithi113.stdout:8/346: write d9/d16/d1e/d26/f4d [784856,104543] 0 2024-04-12T05:08:54.645 INFO:tasks.workunit.client.0.smithi113.stdout:3/281: write d0/f6 [4094196,68751] 0 2024-04-12T05:08:54.645 INFO:tasks.workunit.client.0.smithi113.stdout:0/320: dwrite d1/d6/d16/f24 [0,4194304] 0 2024-04-12T05:08:54.650 INFO:tasks.workunit.client.0.smithi113.stdout:5/288: chown d0/d10/d27/f25 21 1 2024-04-12T05:08:54.651 INFO:tasks.workunit.client.0.smithi113.stdout:2/209: mkdir d8/d9/d27/d3f 0 2024-04-12T05:08:54.653 INFO:tasks.workunit.client.0.smithi113.stdout:4/326: dwrite da/d3c/d43/f4d [4194304,4194304] 0 2024-04-12T05:08:54.653 INFO:tasks.workunit.client.0.smithi113.stdout:8/347: readlink d9/la 0 2024-04-12T05:08:54.655 INFO:tasks.workunit.client.1.smithi192.stdout:2/199: dwrite d1/d8/d3a/de/f30 [0,4194304] 0 2024-04-12T05:08:54.655 INFO:tasks.workunit.client.0.smithi113.stdout:9/377: dread d1/f78 [0,4194304] 0 2024-04-12T05:08:54.656 INFO:tasks.workunit.client.1.smithi192.stdout:2/200: fdatasync d1/d2d/f32 0 2024-04-12T05:08:54.659 INFO:tasks.workunit.client.0.smithi113.stdout:2/210: dread - d8/d9/d1a/f26 zero size 2024-04-12T05:08:54.659 INFO:tasks.workunit.client.1.smithi192.stdout:2/201: write d1/d8/f35 [123969,96602] 0 2024-04-12T05:08:54.659 INFO:tasks.workunit.client.1.smithi192.stdout:7/139: sync 2024-04-12T05:08:54.659 INFO:tasks.workunit.client.1.smithi192.stdout:9/124: sync 2024-04-12T05:08:54.659 INFO:tasks.workunit.client.0.smithi113.stdout:5/289: unlink d0/d10/c2b 0 2024-04-12T05:08:54.660 INFO:tasks.workunit.client.1.smithi192.stdout:2/202: dread - d1/d8/f46 zero size 2024-04-12T05:08:54.661 INFO:tasks.workunit.client.1.smithi192.stdout:7/140: chown d3/d7/f9 63941 1 2024-04-12T05:08:54.664 INFO:tasks.workunit.client.0.smithi113.stdout:7/336: truncate d0/d26/f38 595819 0 2024-04-12T05:08:54.664 INFO:tasks.workunit.client.0.smithi113.stdout:2/211: dread - d8/d16/f37 zero size 2024-04-12T05:08:54.664 INFO:tasks.workunit.client.0.smithi113.stdout:1/349: dwrite d10/d41/f4a [4194304,4194304] 0 2024-04-12T05:08:54.664 INFO:tasks.workunit.client.1.smithi192.stdout:9/125: fdatasync fb 0 2024-04-12T05:08:54.666 INFO:tasks.workunit.client.0.smithi113.stdout:9/378: fdatasync d1/f78 0 2024-04-12T05:08:54.666 INFO:tasks.workunit.client.0.smithi113.stdout:6/278: getdents d8/de 0 2024-04-12T05:08:54.666 INFO:tasks.workunit.client.0.smithi113.stdout:5/290: creat d0/d10/d27/f57 x:0 0 0 2024-04-12T05:08:54.673 INFO:tasks.workunit.client.0.smithi113.stdout:9/379: read d1/df/d4b/d6c/f6b [2155550,9090] 0 2024-04-12T05:08:54.673 INFO:tasks.workunit.client.0.smithi113.stdout:7/337: fsync d0/f33 0 2024-04-12T05:08:54.673 INFO:tasks.workunit.client.0.smithi113.stdout:6/279: write d8/de/d13/d11/d36/f4e [588655,58362] 0 2024-04-12T05:08:54.675 INFO:tasks.workunit.client.0.smithi113.stdout:4/327: link da/d4f/f4e da/dd/df/d1d/f6a 0 2024-04-12T05:08:54.676 INFO:tasks.workunit.client.1.smithi192.stdout:9/126: rmdir d11/d14 39 2024-04-12T05:08:54.677 INFO:tasks.workunit.client.1.smithi192.stdout:7/141: creat d3/d7/d14/d1f/f33 x:0 0 0 2024-04-12T05:08:54.679 INFO:tasks.workunit.client.1.smithi192.stdout:9/127: symlink d11/d15/l2b 0 2024-04-12T05:08:54.680 INFO:tasks.workunit.client.1.smithi192.stdout:9/128: write f7 [1056214,64985] 0 2024-04-12T05:08:54.681 INFO:tasks.workunit.client.0.smithi113.stdout:2/212: dwrite d8/d16/f37 [0,4194304] 0 2024-04-12T05:08:54.683 INFO:tasks.workunit.client.0.smithi113.stdout:8/348: link d9/l4c d9/d16/d3c/l6e 0 2024-04-12T05:08:54.685 INFO:tasks.workunit.client.0.smithi113.stdout:0/321: getdents d1/d6/d16/d2a 0 2024-04-12T05:08:54.685 INFO:tasks.workunit.client.0.smithi113.stdout:8/349: stat d9/l18 0 2024-04-12T05:08:54.685 INFO:tasks.workunit.client.1.smithi192.stdout:7/142: link d3/d7/f13 d3/d7/d14/d1f/f34 0 2024-04-12T05:08:54.689 INFO:tasks.workunit.client.0.smithi113.stdout:4/328: chown da/dd/d1b/c57 366 1 2024-04-12T05:08:54.692 INFO:tasks.workunit.client.1.smithi192.stdout:7/143: symlink d3/d11/l35 0 2024-04-12T05:08:54.693 INFO:tasks.workunit.client.0.smithi113.stdout:6/280: dwrite d8/de/d13/d11/d36/f4e [0,4194304] 0 2024-04-12T05:08:54.694 INFO:tasks.workunit.client.1.smithi192.stdout:8/145: dread d6/f9 [0,4194304] 0 2024-04-12T05:08:54.694 INFO:tasks.workunit.client.0.smithi113.stdout:7/338: symlink d0/l6d 0 2024-04-12T05:08:54.698 INFO:tasks.workunit.client.0.smithi113.stdout:8/350: mknod d9/d16/d1e/d26/d32/d48/c6f 0 2024-04-12T05:08:54.698 INFO:tasks.workunit.client.0.smithi113.stdout:9/380: truncate d1/d4/f33 3390883 0 2024-04-12T05:08:54.699 INFO:tasks.workunit.client.1.smithi192.stdout:2/203: dwrite d1/d8/f10 [0,4194304] 0 2024-04-12T05:08:54.701 INFO:tasks.workunit.client.0.smithi113.stdout:4/329: dwrite da/d4f/f5d [0,4194304] 0 2024-04-12T05:08:54.701 INFO:tasks.workunit.client.1.smithi192.stdout:8/146: dread d6/df/f15 [0,4194304] 0 2024-04-12T05:08:54.704 INFO:tasks.workunit.client.0.smithi113.stdout:5/291: creat d0/d10/f58 x:0 0 0 2024-04-12T05:08:54.704 INFO:tasks.workunit.client.0.smithi113.stdout:6/281: chown d8/de/d17/d3b/c3c 1935 1 2024-04-12T05:08:54.705 INFO:tasks.workunit.client.0.smithi113.stdout:8/351: dread d9/d16/d3c/f52 [0,4194304] 0 2024-04-12T05:08:54.705 INFO:tasks.workunit.client.0.smithi113.stdout:4/330: write da/d24/f53 [35925,26342] 0 2024-04-12T05:08:54.706 INFO:tasks.workunit.client.1.smithi192.stdout:8/147: chown d6/df/f16 37097122 1 2024-04-12T05:08:54.706 INFO:tasks.workunit.client.0.smithi113.stdout:9/381: creat d1/d4/f88 x:0 0 0 2024-04-12T05:08:54.707 INFO:tasks.workunit.client.0.smithi113.stdout:4/331: readlink da/l2a 0 2024-04-12T05:08:54.707 INFO:tasks.workunit.client.1.smithi192.stdout:7/144: truncate d3/d7/f13 3609158 0 2024-04-12T05:08:54.708 INFO:tasks.workunit.client.1.smithi192.stdout:9/129: truncate d11/d15/d21/f24 2574419 0 2024-04-12T05:08:54.709 INFO:tasks.workunit.client.1.smithi192.stdout:9/130: write d11/f13 [430620,37815] 0 2024-04-12T05:08:54.711 INFO:tasks.workunit.client.1.smithi192.stdout:7/145: symlink d3/d1d/l36 0 2024-04-12T05:08:54.711 INFO:tasks.workunit.client.0.smithi113.stdout:1/350: getdents d10/d41 0 2024-04-12T05:08:54.711 INFO:tasks.workunit.client.0.smithi113.stdout:3/282: rename d0/d13/d15/d28/c44 to d0/d13/d15/d28/d2a/d34/c59 0 2024-04-12T05:08:54.712 INFO:tasks.workunit.client.0.smithi113.stdout:0/322: creat d1/d6/f5d x:0 0 0 2024-04-12T05:08:54.713 INFO:tasks.workunit.client.1.smithi192.stdout:7/146: read d3/d11/f2d [1368086,124372] 0 2024-04-12T05:08:54.713 INFO:tasks.workunit.client.0.smithi113.stdout:6/282: creat d8/de/d13/d11/d36/d2c/f5c x:0 0 0 2024-04-12T05:08:54.714 INFO:tasks.workunit.client.0.smithi113.stdout:6/283: readlink d8/de/d13/l24 0 2024-04-12T05:08:54.714 INFO:tasks.workunit.client.0.smithi113.stdout:6/284: fdatasync d8/f40 0 2024-04-12T05:08:54.716 INFO:tasks.workunit.client.1.smithi192.stdout:7/147: write d3/d7/d27/f30 [1059658,4944] 0 2024-04-12T05:08:54.720 INFO:tasks.workunit.client.0.smithi113.stdout:8/352: mknod d9/d16/d1e/d3a/c70 0 2024-04-12T05:08:54.720 INFO:tasks.workunit.client.0.smithi113.stdout:0/323: symlink d1/d43/l5e 0 2024-04-12T05:08:54.721 INFO:tasks.workunit.client.1.smithi192.stdout:7/148: chown d3/d7/d14/d1f/c20 1928 1 2024-04-12T05:08:54.721 INFO:tasks.workunit.client.1.smithi192.stdout:9/131: dread d11/d15/f26 [0,4194304] 0 2024-04-12T05:08:54.722 INFO:tasks.workunit.client.0.smithi113.stdout:8/353: read - d9/d16/d65/f6a zero size 2024-04-12T05:08:54.723 INFO:tasks.workunit.client.0.smithi113.stdout:8/354: stat d9/d16/d65 0 2024-04-12T05:08:54.724 INFO:tasks.workunit.client.0.smithi113.stdout:7/339: rename d0/dc/d6/l1d to d0/dd/d11/d44/d5c/l6e 0 2024-04-12T05:08:54.724 INFO:tasks.workunit.client.1.smithi192.stdout:8/148: dwrite d6/d21/f23 [0,4194304] 0 2024-04-12T05:08:54.725 INFO:tasks.workunit.client.0.smithi113.stdout:6/285: creat d8/d45/f5d x:0 0 0 2024-04-12T05:08:54.725 INFO:tasks.workunit.client.1.smithi192.stdout:8/149: readlink d6/d10/d22/l29 0 2024-04-12T05:08:54.726 INFO:tasks.workunit.client.0.smithi113.stdout:8/355: chown d9/d1b/c1d 35282361 1 2024-04-12T05:08:54.726 INFO:tasks.workunit.client.0.smithi113.stdout:0/324: creat d1/d6/d16/d2a/d2c/f5f x:0 0 0 2024-04-12T05:08:54.728 INFO:tasks.workunit.client.0.smithi113.stdout:2/213: dwrite f7 [0,4194304] 0 2024-04-12T05:08:54.729 INFO:tasks.workunit.client.0.smithi113.stdout:0/325: chown d1/d6/d34/f40 0 1 2024-04-12T05:08:54.731 INFO:tasks.workunit.client.1.smithi192.stdout:8/150: dread - d6/d1f/f2a zero size 2024-04-12T05:08:54.731 INFO:tasks.workunit.client.1.smithi192.stdout:9/132: dread fd [0,4194304] 0 2024-04-12T05:08:54.733 INFO:tasks.workunit.client.0.smithi113.stdout:4/332: getdents da/dd/df/d1d 0 2024-04-12T05:08:54.734 INFO:tasks.workunit.client.1.smithi192.stdout:9/133: truncate fe 242353 0 2024-04-12T05:08:54.734 INFO:tasks.workunit.client.1.smithi192.stdout:8/151: write d6/df/f19 [5600783,31370] 0 2024-04-12T05:08:54.735 INFO:tasks.workunit.client.0.smithi113.stdout:9/382: rename d1/fc to d1/df/d4b/f89 0 2024-04-12T05:08:54.736 INFO:tasks.workunit.client.1.smithi192.stdout:7/149: dwrite d3/d11/f18 [4194304,4194304] 0 2024-04-12T05:08:54.738 INFO:tasks.workunit.client.0.smithi113.stdout:6/286: symlink d8/de/d17/d3b/l5e 0 2024-04-12T05:08:54.739 INFO:tasks.workunit.client.0.smithi113.stdout:7/340: symlink d0/d6c/l6f 0 2024-04-12T05:08:54.739 INFO:tasks.workunit.client.0.smithi113.stdout:4/333: readlink da/d24/l4b 0 2024-04-12T05:08:54.742 INFO:tasks.workunit.client.1.smithi192.stdout:9/134: rename d11/d14 to d11/d15/d21/d2c 0 2024-04-12T05:08:54.743 INFO:tasks.workunit.client.0.smithi113.stdout:8/356: dwrite d9/d16/d1e/d26/d32/d5b/f63 [0,4194304] 0 2024-04-12T05:08:54.745 INFO:tasks.workunit.client.0.smithi113.stdout:1/351: rename d10/d14/f17 to d10/d41/f65 0 2024-04-12T05:08:54.748 INFO:tasks.workunit.client.0.smithi113.stdout:8/357: dread - d9/f55 zero size 2024-04-12T05:08:54.752 INFO:tasks.workunit.client.0.smithi113.stdout:9/383: mknod d1/d4/d38/d65/c8a 0 2024-04-12T05:08:54.752 INFO:tasks.workunit.client.0.smithi113.stdout:6/287: write d8/de/d13/d11/d36/d3d/d4c/d31/f54 [876584,90972] 0 2024-04-12T05:08:54.752 INFO:tasks.workunit.client.0.smithi113.stdout:0/326: dwrite d1/d6/d34/f40 [4194304,4194304] 0 2024-04-12T05:08:54.753 INFO:tasks.workunit.client.1.smithi192.stdout:8/152: dread f5 [0,4194304] 0 2024-04-12T05:08:54.753 INFO:tasks.workunit.client.1.smithi192.stdout:9/135: dwrite f10 [0,4194304] 0 2024-04-12T05:08:54.754 INFO:tasks.workunit.client.0.smithi113.stdout:7/341: creat d0/d6c/f70 x:0 0 0 2024-04-12T05:08:54.754 INFO:tasks.workunit.client.0.smithi113.stdout:4/334: mkdir da/d6b 0 2024-04-12T05:08:54.756 INFO:tasks.workunit.client.1.smithi192.stdout:7/150: dread d3/d11/f2d [0,4194304] 0 2024-04-12T05:08:54.758 INFO:tasks.workunit.client.0.smithi113.stdout:2/214: rename d8/d9/de/d15/d38/f39 to d8/d9/de/d15/f40 0 2024-04-12T05:08:54.759 INFO:tasks.workunit.client.1.smithi192.stdout:9/136: rename d11/d15/f26 to d11/d15/d21/f2d 0 2024-04-12T05:08:54.760 INFO:tasks.workunit.client.0.smithi113.stdout:2/215: read d8/d16/f19 [3483163,121518] 0 2024-04-12T05:08:54.760 INFO:tasks.workunit.client.1.smithi192.stdout:7/151: creat d3/d11/f37 x:0 0 0 2024-04-12T05:08:54.762 INFO:tasks.workunit.client.0.smithi113.stdout:2/216: dread - d8/d9/de/d15/f40 zero size 2024-04-12T05:08:54.762 INFO:tasks.workunit.client.0.smithi113.stdout:7/342: dwrite d0/dc/f41 [0,4194304] 0 2024-04-12T05:08:54.762 INFO:tasks.workunit.client.0.smithi113.stdout:9/384: creat d1/d4/d38/f8b x:0 0 0 2024-04-12T05:08:54.763 INFO:tasks.workunit.client.1.smithi192.stdout:8/153: getdents d6/d21 0 2024-04-12T05:08:54.763 INFO:tasks.workunit.client.1.smithi192.stdout:7/152: chown d3/d7/d15/d22/l32 14843 1 2024-04-12T05:08:54.764 INFO:tasks.workunit.client.1.smithi192.stdout:9/137: mkdir d11/d15/d1e/d2e 0 2024-04-12T05:08:54.764 INFO:tasks.workunit.client.0.smithi113.stdout:8/358: creat d9/d16/d1e/d26/f71 x:0 0 0 2024-04-12T05:08:54.767 INFO:tasks.workunit.client.0.smithi113.stdout:0/327: creat d1/d6/d34/f60 x:0 0 0 2024-04-12T05:08:54.768 INFO:tasks.workunit.client.1.smithi192.stdout:7/153: truncate d3/d7/d14/d1f/f33 193607 0 2024-04-12T05:08:54.768 INFO:tasks.workunit.client.1.smithi192.stdout:8/154: rename d6/d10/d22 to d6/df/d31 0 2024-04-12T05:08:54.770 INFO:tasks.workunit.client.0.smithi113.stdout:6/288: mknod d8/de/d13/d11/d36/d3d/d4c/d31/c5f 0 2024-04-12T05:08:54.770 INFO:tasks.workunit.client.1.smithi192.stdout:7/154: fdatasync d3/f12 0 2024-04-12T05:08:54.770 INFO:tasks.workunit.client.1.smithi192.stdout:8/155: mknod d6/df/c32 0 2024-04-12T05:08:54.771 INFO:tasks.workunit.client.1.smithi192.stdout:9/138: dread d11/d15/d21/d2c/f1a [0,4194304] 0 2024-04-12T05:08:54.772 INFO:tasks.workunit.client.1.smithi192.stdout:7/155: truncate d3/d1d/f2e 269535 0 2024-04-12T05:08:54.773 INFO:tasks.workunit.client.0.smithi113.stdout:3/283: symlink d0/d13/d15/d28/d2a/d34/l5a 0 2024-04-12T05:08:54.776 INFO:tasks.workunit.client.1.smithi192.stdout:7/156: mkdir d3/d38 0 2024-04-12T05:08:54.778 INFO:tasks.workunit.client.0.smithi113.stdout:2/217: creat d8/d9/de/d12/f41 x:0 0 0 2024-04-12T05:08:54.779 INFO:tasks.workunit.client.0.smithi113.stdout:9/385: creat d1/d4/d38/f8c x:0 0 0 2024-04-12T05:08:54.780 INFO:tasks.workunit.client.0.smithi113.stdout:3/284: dwrite d0/fd [0,4194304] 0 2024-04-12T05:08:54.781 INFO:tasks.workunit.client.0.smithi113.stdout:3/285: write d0/f9 [1198127,86851] 0 2024-04-12T05:08:54.781 INFO:tasks.workunit.client.0.smithi113.stdout:7/343: creat d0/dc/d6/d24/f71 x:0 0 0 2024-04-12T05:08:54.782 INFO:tasks.workunit.client.0.smithi113.stdout:8/359: rename d9/d16/c5c to d9/d16/d1c/d59/d68/c72 0 2024-04-12T05:08:54.784 INFO:tasks.workunit.client.0.smithi113.stdout:6/289: creat d8/de/d17/d3b/f60 x:0 0 0 2024-04-12T05:08:54.785 INFO:tasks.workunit.client.1.smithi192.stdout:7/157: rename d3/d11/f18 to d3/d7/d15/d1a/f39 0 2024-04-12T05:08:54.785 INFO:tasks.workunit.client.0.smithi113.stdout:6/290: fdatasync d8/de/d17/f59 0 2024-04-12T05:08:54.785 INFO:tasks.workunit.client.0.smithi113.stdout:1/352: link d10/l3a d10/d2e/d31/l66 0 2024-04-12T05:08:54.789 INFO:tasks.workunit.client.0.smithi113.stdout:0/328: rename d1/d9/f29 to d1/d6/d16/d2a/d50/f61 0 2024-04-12T05:08:54.790 INFO:tasks.workunit.client.0.smithi113.stdout:8/360: creat d9/d34/f73 x:0 0 0 2024-04-12T05:08:54.791 INFO:tasks.workunit.client.0.smithi113.stdout:1/353: mkdir d10/d14/d67 0 2024-04-12T05:08:54.791 INFO:tasks.workunit.client.0.smithi113.stdout:8/361: chown d9/d16/d65/c67 1501722 1 2024-04-12T05:08:54.791 INFO:tasks.workunit.client.1.smithi192.stdout:7/158: symlink d3/d7/d27/l3a 0 2024-04-12T05:08:54.792 INFO:tasks.workunit.client.0.smithi113.stdout:3/286: symlink d0/d13/d15/d28/d2a/l5b 0 2024-04-12T05:08:54.793 INFO:tasks.workunit.client.0.smithi113.stdout:4/335: getdents da/d24/d60 0 2024-04-12T05:08:54.795 INFO:tasks.workunit.client.0.smithi113.stdout:7/344: rename d0/dc/d6/f61 to d0/dc/d50/f72 0 2024-04-12T05:08:54.796 INFO:tasks.workunit.client.1.smithi192.stdout:7/159: link d3/d7/d15/f29 d3/d11/f3b 0 2024-04-12T05:08:54.796 INFO:tasks.workunit.client.0.smithi113.stdout:3/287: write d0/fa [64633,60932] 0 2024-04-12T05:08:54.798 INFO:tasks.workunit.client.1.smithi192.stdout:7/160: creat d3/d7/d15/f3c x:0 0 0 2024-04-12T05:08:54.799 INFO:tasks.workunit.client.0.smithi113.stdout:1/354: symlink d10/d14/l68 0 2024-04-12T05:08:54.799 INFO:tasks.workunit.client.1.smithi192.stdout:7/161: read - d3/d7/d15/f3c zero size 2024-04-12T05:08:54.800 INFO:tasks.workunit.client.0.smithi113.stdout:7/345: write d0/f51 [259497,61503] 0 2024-04-12T05:08:54.803 INFO:tasks.workunit.client.0.smithi113.stdout:0/329: dwrite d1/d6/d16/f24 [0,4194304] 0 2024-04-12T05:08:54.803 INFO:tasks.workunit.client.0.smithi113.stdout:6/291: rename d8/de/d13/d11/d36/d3d/d4c/d31/f47 to d8/de/d13/d11/d36/f61 0 2024-04-12T05:08:54.806 INFO:tasks.workunit.client.1.smithi192.stdout:7/162: creat d3/d7/d14/f3d x:0 0 0 2024-04-12T05:08:54.807 INFO:tasks.workunit.client.0.smithi113.stdout:0/330: readlink d1/d6/d16/d2a/l49 0 2024-04-12T05:08:54.807 INFO:tasks.workunit.client.1.smithi192.stdout:7/163: chown d3/d1d 274 1 2024-04-12T05:08:54.811 INFO:tasks.workunit.client.0.smithi113.stdout:6/292: chown d8/de/d17/d3b/l5e 7919223 1 2024-04-12T05:08:54.811 INFO:tasks.workunit.client.0.smithi113.stdout:9/386: link d1/c10 d1/d4/d30/c8d 0 2024-04-12T05:08:54.811 INFO:tasks.workunit.client.1.smithi192.stdout:7/164: readlink d3/d7/l2b 0 2024-04-12T05:08:54.814 INFO:tasks.workunit.client.0.smithi113.stdout:3/288: dwrite d0/f57 [0,4194304] 0 2024-04-12T05:08:54.815 INFO:tasks.workunit.client.1.smithi192.stdout:8/156: dwrite d6/df/f16 [0,4194304] 0 2024-04-12T05:08:54.817 INFO:tasks.workunit.client.1.smithi192.stdout:9/139: dwrite f3 [0,4194304] 0 2024-04-12T05:08:54.822 INFO:tasks.workunit.client.0.smithi113.stdout:8/362: dwrite d9/d16/d1c/f51 [0,4194304] 0 2024-04-12T05:08:54.824 INFO:tasks.workunit.client.1.smithi192.stdout:9/140: dread d11/d15/d21/f24 [0,4194304] 0 2024-04-12T05:08:54.825 INFO:tasks.workunit.client.0.smithi113.stdout:4/336: dread da/dd/df/d1d/f40 [0,4194304] 0 2024-04-12T05:08:54.825 INFO:tasks.workunit.client.0.smithi113.stdout:9/387: creat d1/df/d4b/d63/d3d/f8e x:0 0 0 2024-04-12T05:08:54.841 INFO:tasks.workunit.client.1.smithi192.stdout:0/82: sync 2024-04-12T05:08:54.841 INFO:tasks.workunit.client.1.smithi192.stdout:6/261: sync 2024-04-12T05:08:54.841 INFO:tasks.workunit.client.1.smithi192.stdout:3/160: sync 2024-04-12T05:08:54.879 INFO:tasks.workunit.client.1.smithi192.stdout:9/141: unlink fd 0 2024-04-12T05:08:54.880 INFO:tasks.workunit.client.0.smithi113.stdout:7/346: getdents d0/dc/d50 0 2024-04-12T05:08:54.881 INFO:tasks.workunit.client.1.smithi192.stdout:0/83: mknod d0/d6/c1a 0 2024-04-12T05:08:54.881 INFO:tasks.workunit.client.0.smithi113.stdout:8/363: write d9/d16/d1c/f2d [4188663,22049] 0 2024-04-12T05:08:54.882 INFO:tasks.workunit.client.0.smithi113.stdout:4/337: chown da 459529 1 2024-04-12T05:08:54.883 INFO:tasks.workunit.client.1.smithi192.stdout:9/142: creat d11/d15/d1e/f2f x:0 0 0 2024-04-12T05:08:54.883 INFO:tasks.workunit.client.1.smithi192.stdout:6/262: symlink d3/d8/d56/d59/l60 0 2024-04-12T05:08:54.884 INFO:tasks.workunit.client.1.smithi192.stdout:3/161: getdents d5/d13/d20/d36 0 2024-04-12T05:08:54.885 INFO:tasks.workunit.client.1.smithi192.stdout:7/165: write d3/d11/f3b [892947,76628] 0 2024-04-12T05:08:54.885 INFO:tasks.workunit.client.0.smithi113.stdout:6/293: creat d8/de/d13/d11/d36/d3d/d4c/d31/f62 x:0 0 0 2024-04-12T05:08:54.888 INFO:tasks.workunit.client.1.smithi192.stdout:3/162: rename d5/d13 to d5/d13/d20/d36/d39 22 2024-04-12T05:08:54.889 INFO:tasks.workunit.client.0.smithi113.stdout:0/331: dwrite d1/d6/d16/d2a/f3f [0,4194304] 0 2024-04-12T05:08:54.892 INFO:tasks.workunit.client.1.smithi192.stdout:7/166: creat d3/d7/d15/d1a/f3e x:0 0 0 2024-04-12T05:08:54.894 INFO:tasks.workunit.client.1.smithi192.stdout:8/157: write d6/f18 [4194995,5262] 0 2024-04-12T05:08:54.894 INFO:tasks.workunit.client.1.smithi192.stdout:9/143: dwrite fa [0,4194304] 0 2024-04-12T05:08:54.896 INFO:tasks.workunit.client.1.smithi192.stdout:8/158: dread - d6/f28 zero size 2024-04-12T05:08:54.900 INFO:tasks.workunit.client.1.smithi192.stdout:7/167: symlink d3/d7/d15/d22/l3f 0 2024-04-12T05:08:54.903 INFO:tasks.workunit.client.1.smithi192.stdout:9/144: creat d11/d15/d1e/d2e/f30 x:0 0 0 2024-04-12T05:08:54.903 INFO:tasks.workunit.client.1.smithi192.stdout:8/159: rename d6/d1f/f2f to d6/d21/f33 0 2024-04-12T05:08:54.904 INFO:tasks.workunit.client.1.smithi192.stdout:7/168: creat d3/d7/d15/d1a/f40 x:0 0 0 2024-04-12T05:08:54.905 INFO:tasks.workunit.client.1.smithi192.stdout:0/84: dread d0/fb [0,4194304] 0 2024-04-12T05:08:54.905 INFO:tasks.workunit.client.1.smithi192.stdout:7/169: readlink d3/d7/d15/d22/l32 0 2024-04-12T05:08:54.908 INFO:tasks.workunit.client.1.smithi192.stdout:8/160: symlink d6/d21/l34 0 2024-04-12T05:08:54.911 INFO:tasks.workunit.client.1.smithi192.stdout:0/85: truncate d0/f8 2904434 0 2024-04-12T05:08:54.912 INFO:tasks.workunit.client.1.smithi192.stdout:8/161: rename d6/f1d to d6/d21/f35 0 2024-04-12T05:08:54.913 INFO:tasks.workunit.client.1.smithi192.stdout:7/170: rmdir d3/d38 0 2024-04-12T05:08:54.915 INFO:tasks.workunit.client.1.smithi192.stdout:0/86: mkdir d0/d6/d15/d1b 0 2024-04-12T05:08:54.916 INFO:tasks.workunit.client.1.smithi192.stdout:8/162: rmdir d6/da 39 2024-04-12T05:08:54.916 INFO:tasks.workunit.client.1.smithi192.stdout:0/87: dread - d0/f17 zero size 2024-04-12T05:08:54.918 INFO:tasks.workunit.client.1.smithi192.stdout:8/163: creat d6/df/d31/f36 x:0 0 0 2024-04-12T05:08:54.920 INFO:tasks.workunit.client.1.smithi192.stdout:7/171: dwrite d3/d1d/f2e [0,4194304] 0 2024-04-12T05:08:54.924 INFO:tasks.workunit.client.1.smithi192.stdout:8/164: dread d6/f18 [0,4194304] 0 2024-04-12T05:08:54.925 INFO:tasks.workunit.client.0.smithi113.stdout:3/289: creat d0/d13/d15/f5c x:0 0 0 2024-04-12T05:08:54.927 INFO:tasks.workunit.client.0.smithi113.stdout:1/355: dread d10/d21/f47 [0,4194304] 0 2024-04-12T05:08:54.929 INFO:tasks.workunit.client.0.smithi113.stdout:0/332: rename d1/d6/d34/f60 to d1/d43/f62 0 2024-04-12T05:08:54.929 INFO:tasks.workunit.client.0.smithi113.stdout:6/294: dwrite d8/de/f19 [0,4194304] 0 2024-04-12T05:08:54.930 INFO:tasks.workunit.client.1.smithi192.stdout:3/163: truncate d5/f19 4131758 0 2024-04-12T05:08:54.932 INFO:tasks.workunit.client.0.smithi113.stdout:8/364: dread d9/f47 [0,4194304] 0 2024-04-12T05:08:54.932 INFO:tasks.workunit.client.0.smithi113.stdout:9/388: truncate d1/f78 472228 0 2024-04-12T05:08:54.933 INFO:tasks.workunit.client.0.smithi113.stdout:4/338: mknod da/dd/df/c6c 0 2024-04-12T05:08:54.933 INFO:tasks.workunit.client.0.smithi113.stdout:6/295: read d8/de/d13/fb [1539574,80064] 0 2024-04-12T05:08:54.936 INFO:tasks.workunit.client.0.smithi113.stdout:7/347: link d0/dd/d2d/f53 d0/dd/f73 0 2024-04-12T05:08:54.936 INFO:tasks.workunit.client.0.smithi113.stdout:4/339: chown da/f66 1329 1 2024-04-12T05:08:54.936 INFO:tasks.workunit.client.0.smithi113.stdout:0/333: symlink d1/d6/d16/d2a/d50/l63 0 2024-04-12T05:08:54.937 INFO:tasks.workunit.client.0.smithi113.stdout:7/348: dread - d0/dc/d50/f5f zero size 2024-04-12T05:08:54.938 INFO:tasks.workunit.client.0.smithi113.stdout:9/389: fsync d1/d4/f18 0 2024-04-12T05:08:54.939 INFO:tasks.workunit.client.0.smithi113.stdout:4/340: write da/dd/d1b/f2c [1634767,65820] 0 2024-04-12T05:08:54.942 INFO:tasks.workunit.client.0.smithi113.stdout:8/365: dread d9/d16/d3c/f52 [0,4194304] 0 2024-04-12T05:08:54.948 INFO:tasks.workunit.client.0.smithi113.stdout:1/356: dread d10/f13 [0,4194304] 0 2024-04-12T05:08:54.950 INFO:tasks.workunit.client.0.smithi113.stdout:6/296: getdents d8/de/d13/d11/d36/d3d/d4c 0 2024-04-12T05:08:54.952 INFO:tasks.workunit.client.0.smithi113.stdout:8/366: unlink d9/lc 0 2024-04-12T05:08:54.953 INFO:tasks.workunit.client.0.smithi113.stdout:4/341: symlink da/dd/df/l6d 0 2024-04-12T05:08:54.955 INFO:tasks.workunit.client.0.smithi113.stdout:9/390: dwrite d1/d4/d5/d22/f5a [0,4194304] 0 2024-04-12T05:08:54.955 INFO:tasks.workunit.client.0.smithi113.stdout:7/349: dwrite d0/d6c/f70 [0,4194304] 0 2024-04-12T05:08:54.956 INFO:tasks.workunit.client.0.smithi113.stdout:8/367: chown d9/d16/d1e/d26/d32/d5b 655 1 2024-04-12T05:08:54.957 INFO:tasks.workunit.client.0.smithi113.stdout:6/297: mkdir d8/de/d17/d3b/d63 0 2024-04-12T05:08:54.957 INFO:tasks.workunit.client.0.smithi113.stdout:0/334: write d1/f2f [1174712,15921] 0 2024-04-12T05:08:54.959 INFO:tasks.workunit.client.0.smithi113.stdout:8/368: chown d9/d16/d1e/d26/f28 30844916 1 2024-04-12T05:08:54.959 INFO:tasks.workunit.client.0.smithi113.stdout:9/391: readlink d1/d4/l68 0 2024-04-12T05:08:54.961 INFO:tasks.workunit.client.0.smithi113.stdout:0/335: fsync d1/d6/d34/f40 0 2024-04-12T05:08:54.962 INFO:tasks.workunit.client.0.smithi113.stdout:9/392: write d1/df/d4b/f89 [638544,91083] 0 2024-04-12T05:08:54.965 INFO:tasks.workunit.client.0.smithi113.stdout:0/336: stat d1/d6/d16/d2a/d50/f61 0 2024-04-12T05:08:54.966 INFO:tasks.workunit.client.0.smithi113.stdout:0/337: fdatasync d1/d9/f45 0 2024-04-12T05:08:54.968 INFO:tasks.workunit.client.0.smithi113.stdout:1/357: dwrite d10/d41/f65 [0,4194304] 0 2024-04-12T05:08:54.968 INFO:tasks.workunit.client.0.smithi113.stdout:8/369: creat d9/d16/d3c/f74 x:0 0 0 2024-04-12T05:08:54.969 INFO:tasks.workunit.client.0.smithi113.stdout:9/393: chown d1/c2a 18794 1 2024-04-12T05:08:54.970 INFO:tasks.workunit.client.0.smithi113.stdout:1/358: write d10/d21/f40 [3664286,72336] 0 2024-04-12T05:08:54.972 INFO:tasks.workunit.client.0.smithi113.stdout:1/359: fsync d10/d14/f2a 0 2024-04-12T05:08:54.974 INFO:tasks.workunit.client.0.smithi113.stdout:1/360: fsync d10/d14/f4d 0 2024-04-12T05:08:54.977 INFO:tasks.workunit.client.0.smithi113.stdout:1/361: read - d10/d2e/d31/d33/f61 zero size 2024-04-12T05:08:54.978 INFO:tasks.workunit.client.0.smithi113.stdout:0/338: dwrite d1/d43/f62 [0,4194304] 0 2024-04-12T05:08:54.980 INFO:tasks.workunit.client.0.smithi113.stdout:4/342: dread da/d4f/f4e [0,4194304] 0 2024-04-12T05:08:54.982 INFO:tasks.workunit.client.0.smithi113.stdout:1/362: creat d10/d14/d67/f69 x:0 0 0 2024-04-12T05:08:54.983 INFO:tasks.workunit.client.0.smithi113.stdout:0/339: creat d1/d6/f64 x:0 0 0 2024-04-12T05:08:54.984 INFO:tasks.workunit.client.0.smithi113.stdout:0/340: dread - d1/d6/f64 zero size 2024-04-12T05:08:54.984 INFO:tasks.workunit.client.0.smithi113.stdout:1/363: symlink d10/d41/l6a 0 2024-04-12T05:08:54.984 INFO:tasks.workunit.client.1.smithi192.stdout:3/164: write d5/f14 [2426274,58125] 0 2024-04-12T05:08:54.986 INFO:tasks.workunit.client.1.smithi192.stdout:8/165: dwrite d6/da/fc [4194304,4194304] 0 2024-04-12T05:08:54.987 INFO:tasks.workunit.client.1.smithi192.stdout:9/145: write f6 [3776169,838] 0 2024-04-12T05:08:54.987 INFO:tasks.workunit.client.0.smithi113.stdout:1/364: symlink d10/d41/d5b/l6b 0 2024-04-12T05:08:54.989 INFO:tasks.workunit.client.1.smithi192.stdout:0/88: rename d0/f8 to d0/d6/d15/d19/f1c 0 2024-04-12T05:08:54.989 INFO:tasks.workunit.client.0.smithi113.stdout:0/341: creat d1/d6/d16/d2a/d50/f65 x:0 0 0 2024-04-12T05:08:54.991 INFO:tasks.workunit.client.0.smithi113.stdout:1/365: rename d10/d2e/l3f to d10/d21/d3c/l6c 0 2024-04-12T05:08:54.992 INFO:tasks.workunit.client.1.smithi192.stdout:6/263: dwrite d3/d8/d22/f27 [0,4194304] 0 2024-04-12T05:08:54.992 INFO:tasks.workunit.client.0.smithi113.stdout:0/342: creat d1/d43/d59/f66 x:0 0 0 2024-04-12T05:08:54.994 INFO:tasks.workunit.client.0.smithi113.stdout:7/350: write d0/dd/d11/f4a [993729,114631] 0 2024-04-12T05:08:54.994 INFO:tasks.workunit.client.1.smithi192.stdout:9/146: write d11/d15/d1e/f2f [71506,107137] 0 2024-04-12T05:08:54.995 INFO:tasks.workunit.client.0.smithi113.stdout:7/351: read - d0/dd/d11/d44/f6b zero size 2024-04-12T05:08:54.998 INFO:tasks.workunit.client.1.smithi192.stdout:6/264: stat d3/d8/d48/f50 0 2024-04-12T05:08:54.999 INFO:tasks.workunit.client.1.smithi192.stdout:3/165: mkdir d5/d13/d3a 0 2024-04-12T05:08:54.999 INFO:tasks.workunit.client.1.smithi192.stdout:8/166: mkdir d6/df/d37 0 2024-04-12T05:08:54.999 INFO:tasks.workunit.client.0.smithi113.stdout:7/352: creat d0/d52/f74 x:0 0 0 2024-04-12T05:08:55.000 INFO:tasks.workunit.client.1.smithi192.stdout:0/89: creat d0/d6/d15/d19/f1d x:0 0 0 2024-04-12T05:08:55.001 INFO:tasks.workunit.client.0.smithi113.stdout:1/366: link d10/d2e/l3b d10/d55/l6d 0 2024-04-12T05:08:55.001 INFO:tasks.workunit.client.1.smithi192.stdout:0/90: read d0/f16 [336940,110406] 0 2024-04-12T05:08:55.002 INFO:tasks.workunit.client.1.smithi192.stdout:9/147: symlink d11/d15/d21/l31 0 2024-04-12T05:08:55.002 INFO:tasks.workunit.client.0.smithi113.stdout:4/343: dread da/d4a/f5c [0,4194304] 0 2024-04-12T05:08:55.002 INFO:tasks.workunit.client.0.smithi113.stdout:0/343: creat d1/d6/d16/f67 x:0 0 0 2024-04-12T05:08:55.004 INFO:tasks.workunit.client.0.smithi113.stdout:7/353: symlink d0/dd/d11/l75 0 2024-04-12T05:08:55.004 INFO:tasks.workunit.client.1.smithi192.stdout:8/167: mkdir d6/df/d38 0 2024-04-12T05:08:55.006 INFO:tasks.workunit.client.0.smithi113.stdout:1/367: unlink d10/d2e/d31/d5c/f5f 0 2024-04-12T05:08:55.008 INFO:tasks.workunit.client.1.smithi192.stdout:9/148: creat d11/d25/f32 x:0 0 0 2024-04-12T05:08:55.010 INFO:tasks.workunit.client.1.smithi192.stdout:0/91: rename d0/le to d0/d6/l1e 0 2024-04-12T05:08:55.010 INFO:tasks.workunit.client.0.smithi113.stdout:7/354: mknod d0/dc/d6/c76 0 2024-04-12T05:08:55.012 INFO:tasks.workunit.client.0.smithi113.stdout:1/368: symlink d10/d14/l6e 0 2024-04-12T05:08:55.012 INFO:tasks.workunit.client.1.smithi192.stdout:6/265: getdents d3/dc 0 2024-04-12T05:08:55.012 INFO:tasks.workunit.client.0.smithi113.stdout:7/355: truncate d0/dd/d11/d44/f47 5228237 0 2024-04-12T05:08:55.014 INFO:tasks.workunit.client.1.smithi192.stdout:0/92: mknod d0/d6/d15/c1f 0 2024-04-12T05:08:55.015 INFO:tasks.workunit.client.0.smithi113.stdout:0/344: link d1/fe d1/d6/d34/f68 0 2024-04-12T05:08:55.017 INFO:tasks.workunit.client.1.smithi192.stdout:8/168: dwrite d6/df/f19 [8388608,4194304] 0 2024-04-12T05:08:55.018 INFO:tasks.workunit.client.0.smithi113.stdout:4/344: creat da/dd/f6e x:0 0 0 2024-04-12T05:08:55.019 INFO:tasks.workunit.client.1.smithi192.stdout:0/93: fdatasync d0/fb 0 2024-04-12T05:08:55.020 INFO:tasks.workunit.client.0.smithi113.stdout:9/394: write d1/d4/f33 [1106678,100028] 0 2024-04-12T05:08:55.021 INFO:tasks.workunit.client.0.smithi113.stdout:1/369: mknod d10/c6f 0 2024-04-12T05:08:55.022 INFO:tasks.workunit.client.1.smithi192.stdout:6/266: rename d3/d5/l4a to d3/d23/l61 0 2024-04-12T05:08:55.022 INFO:tasks.workunit.client.0.smithi113.stdout:9/395: stat d1/c64 0 2024-04-12T05:08:55.022 INFO:tasks.workunit.client.1.smithi192.stdout:9/149: dwrite d11/f12 [0,4194304] 0 2024-04-12T05:08:55.023 INFO:tasks.workunit.client.0.smithi113.stdout:4/345: symlink da/dd/df/d1d/l6f 0 2024-04-12T05:08:55.023 INFO:tasks.workunit.client.0.smithi113.stdout:0/345: chown d1/d6/d16/d2a/d50/d52/l55 885179 1 2024-04-12T05:08:55.024 INFO:tasks.workunit.client.1.smithi192.stdout:9/150: chown d11/f13 87612 1 2024-04-12T05:08:55.026 INFO:tasks.workunit.client.0.smithi113.stdout:4/346: stat da/dd/c64 0 2024-04-12T05:08:55.026 INFO:tasks.workunit.client.0.smithi113.stdout:7/356: dread d0/dd/d11/f20 [0,4194304] 0 2024-04-12T05:08:55.030 INFO:tasks.workunit.client.1.smithi192.stdout:9/151: mkdir d11/d15/d1e/d2e/d33 0 2024-04-12T05:08:55.030 INFO:tasks.workunit.client.0.smithi113.stdout:9/396: creat d1/d4/f8f x:0 0 0 2024-04-12T05:08:55.030 INFO:tasks.workunit.client.0.smithi113.stdout:0/346: mknod d1/d6/d16/d2a/d50/d52/c69 0 2024-04-12T05:08:55.031 INFO:tasks.workunit.client.0.smithi113.stdout:4/347: rmdir da/dd/df/d1d 39 2024-04-12T05:08:55.033 INFO:tasks.workunit.client.1.smithi192.stdout:6/267: link d3/d8/d48/f50 d3/d8/d10/d4c/f62 0 2024-04-12T05:08:55.033 INFO:tasks.workunit.client.0.smithi113.stdout:7/357: creat d0/dd/d11/f77 x:0 0 0 2024-04-12T05:08:55.036 INFO:tasks.workunit.client.0.smithi113.stdout:0/347: symlink d1/l6a 0 2024-04-12T05:08:55.036 INFO:tasks.workunit.client.0.smithi113.stdout:7/358: stat d0/dd/d11/f77 0 2024-04-12T05:08:55.036 INFO:tasks.workunit.client.0.smithi113.stdout:1/370: dwrite d10/d2e/d31/d33/f50 [0,4194304] 0 2024-04-12T05:08:55.036 INFO:tasks.workunit.client.1.smithi192.stdout:6/268: creat d3/d8/d48/f63 x:0 0 0 2024-04-12T05:08:55.037 INFO:tasks.workunit.client.0.smithi113.stdout:4/348: mkdir da/d70 0 2024-04-12T05:08:55.037 INFO:tasks.workunit.client.0.smithi113.stdout:1/371: chown d10/d41/d5b 110548692 1 2024-04-12T05:08:55.038 INFO:tasks.workunit.client.1.smithi192.stdout:3/166: rename d5/d13/f17 to d5/d13/d1a/f3b 0 2024-04-12T05:08:55.039 INFO:tasks.workunit.client.1.smithi192.stdout:4/220: sync 2024-04-12T05:08:55.039 INFO:tasks.workunit.client.1.smithi192.stdout:3/167: fsync d5/f14 0 2024-04-12T05:08:55.040 INFO:tasks.workunit.client.0.smithi113.stdout:4/349: write da/dd/df/d1d/f46 [733998,382] 0 2024-04-12T05:08:55.042 INFO:tasks.workunit.client.1.smithi192.stdout:6/269: truncate d3/d8/f1c 4539737 0 2024-04-12T05:08:55.042 INFO:tasks.workunit.client.1.smithi192.stdout:9/152: dwrite d11/d15/d21/d2c/f1a [0,4194304] 0 2024-04-12T05:08:55.043 INFO:tasks.workunit.client.0.smithi113.stdout:9/397: rename d1/d4/d38/c67 to d1/d4/d5/c90 0 2024-04-12T05:08:55.045 INFO:tasks.workunit.client.0.smithi113.stdout:4/350: write da/dd/df/f45 [1429639,102331] 0 2024-04-12T05:08:55.045 INFO:tasks.workunit.client.0.smithi113.stdout:9/398: fsync d1/d4/d38/d65/f7d 0 2024-04-12T05:08:55.047 INFO:tasks.workunit.client.0.smithi113.stdout:9/399: write d1/d4/f33 [1554583,88548] 0 2024-04-12T05:08:55.047 INFO:tasks.workunit.client.1.smithi192.stdout:8/169: rename d6/d21/f35 to d6/df/d31/f39 0 2024-04-12T05:08:55.047 INFO:tasks.workunit.client.1.smithi192.stdout:4/221: unlink d1/da/d1c/f45 0 2024-04-12T05:08:55.048 INFO:tasks.workunit.client.0.smithi113.stdout:7/359: rename d0/d26 to d0/dc/d6/d78 0 2024-04-12T05:08:55.051 INFO:tasks.workunit.client.0.smithi113.stdout:7/360: chown d0/dc/d6/d78/c2f 6342535 1 2024-04-12T05:08:55.051 INFO:tasks.workunit.client.1.smithi192.stdout:6/270: mknod d3/dc/c64 0 2024-04-12T05:08:55.051 INFO:tasks.workunit.client.0.smithi113.stdout:9/400: creat d1/d4/d38/d65/f91 x:0 0 0 2024-04-12T05:08:55.054 INFO:tasks.workunit.client.1.smithi192.stdout:8/170: creat d6/da/f3a x:0 0 0 2024-04-12T05:08:55.055 INFO:tasks.workunit.client.0.smithi113.stdout:1/372: rename d10/d21/d3c to d10/d14/d67/d70 0 2024-04-12T05:08:55.056 INFO:tasks.workunit.client.1.smithi192.stdout:4/222: unlink d1/da/f2f 0 2024-04-12T05:08:55.059 INFO:tasks.workunit.client.0.smithi113.stdout:1/373: creat d10/d2e/f71 x:0 0 0 2024-04-12T05:08:55.059 INFO:tasks.workunit.client.1.smithi192.stdout:4/223: fdatasync d1/da/d1c/f28 0 2024-04-12T05:08:55.060 INFO:tasks.workunit.client.1.smithi192.stdout:0/94: dwrite d0/d6/d15/d19/f1c [0,4194304] 0 2024-04-12T05:08:55.061 INFO:tasks.workunit.client.1.smithi192.stdout:8/171: mkdir d6/df/d38/d3b 0 2024-04-12T05:08:55.062 INFO:tasks.workunit.client.0.smithi113.stdout:4/351: rename da/dd/f21 to da/dd/df/f71 0 2024-04-12T05:08:55.066 INFO:tasks.workunit.client.0.smithi113.stdout:1/374: dread d10/d14/d67/d70/f58 [0,4194304] 0 2024-04-12T05:08:55.067 INFO:tasks.workunit.client.1.smithi192.stdout:0/95: read d0/d6/d15/d19/f1c [1801934,36695] 0 2024-04-12T05:08:55.067 INFO:tasks.workunit.client.1.smithi192.stdout:6/271: dwrite d3/d8/f5d [0,4194304] 0 2024-04-12T05:08:55.067 INFO:tasks.workunit.client.1.smithi192.stdout:8/172: write d6/df/f26 [68588,71309] 0 2024-04-12T05:08:55.070 INFO:tasks.workunit.client.1.smithi192.stdout:4/224: creat d1/d20/d37/f48 x:0 0 0 2024-04-12T05:08:55.070 INFO:tasks.workunit.client.0.smithi113.stdout:0/348: mkdir d1/d43/d6b 0 2024-04-12T05:08:55.072 INFO:tasks.workunit.client.0.smithi113.stdout:1/375: symlink d10/d14/d67/l72 0 2024-04-12T05:08:55.074 INFO:tasks.workunit.client.0.smithi113.stdout:1/376: symlink d10/d2e/l73 0 2024-04-12T05:08:55.075 INFO:tasks.workunit.client.1.smithi192.stdout:4/225: chown d1/d1d 8213210 1 2024-04-12T05:08:55.075 INFO:tasks.workunit.client.1.smithi192.stdout:3/168: rename d5/d13/d20 to d5/d13/d3c 0 2024-04-12T05:08:55.076 INFO:tasks.workunit.client.1.smithi192.stdout:4/226: mkdir d1/d49 0 2024-04-12T05:08:55.076 INFO:tasks.workunit.client.0.smithi113.stdout:1/377: creat d10/d2e/d31/d5c/f74 x:0 0 0 2024-04-12T05:08:55.077 INFO:tasks.workunit.client.1.smithi192.stdout:0/96: getdents d0/d6/d15/d1b 0 2024-04-12T05:08:55.078 INFO:tasks.workunit.client.1.smithi192.stdout:3/169: read d5/fd [51836,92188] 0 2024-04-12T05:08:55.080 INFO:tasks.workunit.client.1.smithi192.stdout:0/97: creat d0/f20 x:0 0 0 2024-04-12T05:08:55.080 INFO:tasks.workunit.client.1.smithi192.stdout:4/227: mknod d1/d49/c4a 0 2024-04-12T05:08:55.088 INFO:tasks.workunit.client.1.smithi192.stdout:6/272: dwrite d3/d5/ff [4194304,4194304] 0 2024-04-12T05:08:55.089 INFO:tasks.workunit.client.0.smithi113.stdout:1/378: rmdir d10/d14 39 2024-04-12T05:08:55.090 INFO:tasks.workunit.client.1.smithi192.stdout:6/273: stat d3/f43 0 2024-04-12T05:08:55.091 INFO:tasks.workunit.client.1.smithi192.stdout:3/170: rename d5/d13/d1a/l1d to d5/l3d 0 2024-04-12T05:08:55.094 INFO:tasks.workunit.client.1.smithi192.stdout:6/274: write d3/dc/d39/f4e [1675806,96655] 0 2024-04-12T05:08:55.095 INFO:tasks.workunit.client.0.smithi113.stdout:1/379: unlink d10/d2e/d31/d33/c57 0 2024-04-12T05:08:55.095 INFO:tasks.workunit.client.1.smithi192.stdout:0/98: dwrite d0/f20 [0,4194304] 0 2024-04-12T05:08:55.096 INFO:tasks.workunit.client.0.smithi113.stdout:1/380: chown ff 62068 1 2024-04-12T05:08:55.097 INFO:tasks.workunit.client.0.smithi113.stdout:1/381: fsync d10/d2e/f4e 0 2024-04-12T05:08:55.098 INFO:tasks.workunit.client.1.smithi192.stdout:0/99: creat d0/d6/f21 x:0 0 0 2024-04-12T05:08:55.101 INFO:tasks.workunit.client.1.smithi192.stdout:4/228: dread d1/da/f31 [0,4194304] 0 2024-04-12T05:08:55.102 INFO:tasks.workunit.client.1.smithi192.stdout:6/275: rename d3/d5/f38 to d3/d8/d22/f65 0 2024-04-12T05:08:55.102 INFO:tasks.workunit.client.1.smithi192.stdout:4/229: readlink d1/lc 0 2024-04-12T05:08:55.104 INFO:tasks.workunit.client.0.smithi113.stdout:0/349: dread d1/d6/d16/f1c [0,4194304] 0 2024-04-12T05:08:55.107 INFO:tasks.workunit.client.1.smithi192.stdout:6/276: creat d3/d5/f66 x:0 0 0 2024-04-12T05:08:55.107 INFO:tasks.workunit.client.0.smithi113.stdout:0/350: creat d1/d9/f6c x:0 0 0 2024-04-12T05:08:55.108 INFO:tasks.workunit.client.0.smithi113.stdout:0/351: dread - d1/d6/da/f4d zero size 2024-04-12T05:08:55.109 INFO:tasks.workunit.client.1.smithi192.stdout:6/277: stat d3/d21/d24/l33 0 2024-04-12T05:08:55.110 INFO:tasks.workunit.client.1.smithi192.stdout:6/278: stat d3/d8/d22/f27 0 2024-04-12T05:08:55.111 INFO:tasks.workunit.client.0.smithi113.stdout:1/382: dread d10/d14/f1f [0,4194304] 0 2024-04-12T05:08:55.112 INFO:tasks.workunit.client.0.smithi113.stdout:1/383: readlink d10/d14/l1b 0 2024-04-12T05:08:55.112 INFO:tasks.workunit.client.1.smithi192.stdout:4/230: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:55.114 INFO:tasks.workunit.client.0.smithi113.stdout:1/384: write d10/d2e/d31/f59 [486540,43348] 0 2024-04-12T05:08:55.114 INFO:tasks.workunit.client.1.smithi192.stdout:6/279: mkdir d3/d23/d67 0 2024-04-12T05:08:55.115 INFO:tasks.workunit.client.1.smithi192.stdout:4/231: chown d1/l2c 352818605 1 2024-04-12T05:08:55.116 INFO:tasks.workunit.client.0.smithi113.stdout:0/352: dwrite d1/d6/d16/d2a/d2c/f5f [0,4194304] 0 2024-04-12T05:08:55.119 INFO:tasks.workunit.client.1.smithi192.stdout:6/280: write d3/d21/d24/f4f [6508885,89626] 0 2024-04-12T05:08:55.121 INFO:tasks.workunit.client.1.smithi192.stdout:0/100: dwrite d0/f16 [0,4194304] 0 2024-04-12T05:08:55.121 INFO:tasks.workunit.client.1.smithi192.stdout:3/171: dread d5/d13/f2a [0,4194304] 0 2024-04-12T05:08:55.121 INFO:tasks.workunit.client.1.smithi192.stdout:6/281: write d3/d26/f31 [318729,125617] 0 2024-04-12T05:08:55.122 INFO:tasks.workunit.client.0.smithi113.stdout:1/385: mknod d10/d41/c75 0 2024-04-12T05:08:55.128 INFO:tasks.workunit.client.1.smithi192.stdout:4/232: creat d1/f4b x:0 0 0 2024-04-12T05:08:55.129 INFO:tasks.workunit.client.1.smithi192.stdout:0/101: mkdir d0/d6/d22 0 2024-04-12T05:08:55.131 INFO:tasks.workunit.client.1.smithi192.stdout:4/233: truncate d1/d1d/f44 398297 0 2024-04-12T05:08:55.133 INFO:tasks.workunit.client.1.smithi192.stdout:3/172: symlink d5/d13/d3c/d36/l3e 0 2024-04-12T05:08:55.134 INFO:tasks.workunit.client.0.smithi113.stdout:0/353: write d1/fe [3031628,53797] 0 2024-04-12T05:08:55.135 INFO:tasks.workunit.client.1.smithi192.stdout:4/234: unlink d1/d1d/f2d 0 2024-04-12T05:08:55.135 INFO:tasks.workunit.client.1.smithi192.stdout:4/235: dread - d1/d20/d37/f48 zero size 2024-04-12T05:08:55.137 INFO:tasks.workunit.client.1.smithi192.stdout:4/236: fdatasync d1/d1d/f44 0 2024-04-12T05:08:55.141 INFO:tasks.workunit.client.1.smithi192.stdout:4/237: mkdir d1/d1d/d4c 0 2024-04-12T05:08:55.141 INFO:tasks.workunit.client.1.smithi192.stdout:4/238: fsync d1/f33 0 2024-04-12T05:08:55.143 INFO:tasks.workunit.client.1.smithi192.stdout:3/173: getdents d5/d13/d3c/d26 0 2024-04-12T05:08:55.146 INFO:tasks.workunit.client.1.smithi192.stdout:6/282: truncate d3/d8/f17 4385774 0 2024-04-12T05:08:55.148 INFO:tasks.workunit.client.1.smithi192.stdout:0/102: truncate d0/f16 826507 0 2024-04-12T05:08:55.149 INFO:tasks.workunit.client.0.smithi113.stdout:0/354: dwrite d1/f22 [0,4194304] 0 2024-04-12T05:08:55.151 INFO:tasks.workunit.client.1.smithi192.stdout:4/239: dwrite d1/f2e [0,4194304] 0 2024-04-12T05:08:55.151 INFO:tasks.workunit.client.1.smithi192.stdout:6/283: mkdir d3/d8/d10/d4c/d68 0 2024-04-12T05:08:55.152 INFO:tasks.workunit.client.1.smithi192.stdout:0/103: creat d0/d6/d15/f23 x:0 0 0 2024-04-12T05:08:55.153 INFO:tasks.workunit.client.1.smithi192.stdout:5/239: sync 2024-04-12T05:08:55.153 INFO:tasks.workunit.client.1.smithi192.stdout:2/204: sync 2024-04-12T05:08:55.153 INFO:tasks.workunit.client.1.smithi192.stdout:1/180: sync 2024-04-12T05:08:55.153 INFO:tasks.workunit.client.1.smithi192.stdout:3/174: dwrite d5/d13/f2f [4194304,4194304] 0 2024-04-12T05:08:55.154 INFO:tasks.workunit.client.1.smithi192.stdout:6/284: stat d3/d5/ff 0 2024-04-12T05:08:55.156 INFO:tasks.workunit.client.1.smithi192.stdout:2/205: chown d1/d8/ld 534766918 1 2024-04-12T05:08:55.159 INFO:tasks.workunit.client.1.smithi192.stdout:0/104: dread - d0/d6/d15/f23 zero size 2024-04-12T05:08:55.166 INFO:tasks.workunit.client.1.smithi192.stdout:4/240: dwrite d1/f4b [0,4194304] 0 2024-04-12T05:08:55.167 INFO:tasks.workunit.client.1.smithi192.stdout:0/105: creat d0/d6/d22/f24 x:0 0 0 2024-04-12T05:08:55.168 INFO:tasks.workunit.client.1.smithi192.stdout:1/181: getdents d0/d29 0 2024-04-12T05:08:55.169 INFO:tasks.workunit.client.0.smithi113.stdout:0/355: truncate d1/d6/d16/f47 429902 0 2024-04-12T05:08:55.170 INFO:tasks.workunit.client.1.smithi192.stdout:1/182: fsync d0/f3 0 2024-04-12T05:08:55.170 INFO:tasks.workunit.client.1.smithi192.stdout:2/206: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:55.171 INFO:tasks.workunit.client.0.smithi113.stdout:0/356: fsync d1/f3d 0 2024-04-12T05:08:55.171 INFO:tasks.workunit.client.1.smithi192.stdout:0/106: creat d0/d6/f25 x:0 0 0 2024-04-12T05:08:55.173 INFO:tasks.workunit.client.1.smithi192.stdout:5/240: getdents d0/d15/d24/d29 0 2024-04-12T05:08:55.180 INFO:tasks.workunit.client.1.smithi192.stdout:1/183: rename d0/f30 to d0/d2/dd/f32 0 2024-04-12T05:08:55.181 INFO:tasks.workunit.client.0.smithi113.stdout:0/357: link d1/d6/d16/d2a/d2c/c54 d1/d43/d59/c6d 0 2024-04-12T05:08:55.183 INFO:tasks.workunit.client.1.smithi192.stdout:2/207: dwrite d1/d8/f22 [0,4194304] 0 2024-04-12T05:08:55.185 INFO:tasks.workunit.client.1.smithi192.stdout:0/107: dwrite d0/d6/f25 [0,4194304] 0 2024-04-12T05:08:55.185 INFO:tasks.workunit.client.0.smithi113.stdout:0/358: symlink d1/d6/d16/d2a/d2c/l6e 0 2024-04-12T05:08:55.186 INFO:tasks.workunit.client.1.smithi192.stdout:5/241: truncate d0/d15/d24/d29/f2a 1244945 0 2024-04-12T05:08:55.186 INFO:tasks.workunit.client.1.smithi192.stdout:2/208: dread - d1/d8/f46 zero size 2024-04-12T05:08:55.187 INFO:tasks.workunit.client.1.smithi192.stdout:1/184: mkdir d0/d2/dd/d21/d33 0 2024-04-12T05:08:55.188 INFO:tasks.workunit.client.1.smithi192.stdout:0/108: mkdir d0/d26 0 2024-04-12T05:08:55.189 INFO:tasks.workunit.client.0.smithi113.stdout:0/359: read d1/f2f [982384,44783] 0 2024-04-12T05:08:55.189 INFO:tasks.workunit.client.1.smithi192.stdout:2/209: write d1/d8/f46 [461592,13780] 0 2024-04-12T05:08:55.190 INFO:tasks.workunit.client.1.smithi192.stdout:5/242: mknod d0/d15/d24/d29/c59 0 2024-04-12T05:08:55.190 INFO:tasks.workunit.client.0.smithi113.stdout:0/360: write d1/d43/f62 [3494876,8327] 0 2024-04-12T05:08:55.193 INFO:tasks.workunit.client.1.smithi192.stdout:1/185: creat d0/d2/f34 x:0 0 0 2024-04-12T05:08:55.196 INFO:tasks.workunit.client.1.smithi192.stdout:0/109: write d0/fb [1190990,3346] 0 2024-04-12T05:08:55.197 INFO:tasks.workunit.client.1.smithi192.stdout:2/210: creat d1/d2d/f48 x:0 0 0 2024-04-12T05:08:55.200 INFO:tasks.workunit.client.0.smithi113.stdout:0/361: getdents d1/d43/d6b 0 2024-04-12T05:08:55.204 INFO:tasks.workunit.client.1.smithi192.stdout:6/285: dwrite d3/d5/f6 [0,4194304] 0 2024-04-12T05:08:55.205 INFO:tasks.workunit.client.0.smithi113.stdout:0/362: dread d1/d6/d16/d2a/d2c/f5f [0,4194304] 0 2024-04-12T05:08:55.208 INFO:tasks.workunit.client.1.smithi192.stdout:0/110: creat d0/d6/d15/f27 x:0 0 0 2024-04-12T05:08:55.213 INFO:tasks.workunit.client.1.smithi192.stdout:3/175: dwrite d5/d13/d1a/f3b [0,4194304] 0 2024-04-12T05:08:55.217 INFO:tasks.workunit.client.1.smithi192.stdout:0/111: dread d0/f16 [0,4194304] 0 2024-04-12T05:08:55.219 INFO:tasks.workunit.client.1.smithi192.stdout:3/176: creat d5/d13/d3c/d36/f3f x:0 0 0 2024-04-12T05:08:55.220 INFO:tasks.workunit.client.0.smithi113.stdout:0/363: rename d1/d6/d16/d2a/d2c/c54 to d1/d6/d16/c6f 0 2024-04-12T05:08:55.221 INFO:tasks.workunit.client.0.smithi113.stdout:0/364: write d1/d9/f1a [5180586,83536] 0 2024-04-12T05:08:55.222 INFO:tasks.workunit.client.1.smithi192.stdout:3/177: chown d5/d13/d3c/l29 11345 1 2024-04-12T05:08:55.227 INFO:tasks.workunit.client.0.smithi113.stdout:0/365: rename d1/fe to d1/d6/d16/d2a/f70 0 2024-04-12T05:08:55.227 INFO:tasks.workunit.client.1.smithi192.stdout:0/112: getdents d0/d6 0 2024-04-12T05:08:55.227 INFO:tasks.workunit.client.1.smithi192.stdout:3/178: symlink d5/d13/d3a/l40 0 2024-04-12T05:08:55.230 INFO:tasks.workunit.client.1.smithi192.stdout:0/113: creat d0/f28 x:0 0 0 2024-04-12T05:08:55.230 INFO:tasks.workunit.client.1.smithi192.stdout:0/114: dread - d0/f28 zero size 2024-04-12T05:08:55.234 INFO:tasks.workunit.client.1.smithi192.stdout:5/243: dread d0/f3d [0,4194304] 0 2024-04-12T05:08:55.237 INFO:tasks.workunit.client.1.smithi192.stdout:0/115: mkdir d0/d26/d29 0 2024-04-12T05:08:55.237 INFO:tasks.workunit.client.1.smithi192.stdout:5/244: mkdir d0/d15/d32/d5a 0 2024-04-12T05:08:55.239 INFO:tasks.workunit.client.1.smithi192.stdout:0/116: mkdir d0/d6/d2a 0 2024-04-12T05:08:55.240 INFO:tasks.workunit.client.1.smithi192.stdout:3/179: dwrite d5/d13/f2e [0,4194304] 0 2024-04-12T05:08:55.240 INFO:tasks.workunit.client.1.smithi192.stdout:5/245: dread - d0/d15/d32/f4c zero size 2024-04-12T05:08:55.240 INFO:tasks.workunit.client.1.smithi192.stdout:3/180: chown d5/f1c 583884 1 2024-04-12T05:08:55.243 INFO:tasks.workunit.client.1.smithi192.stdout:5/246: mknod d0/d21/c5b 0 2024-04-12T05:08:55.243 INFO:tasks.workunit.client.1.smithi192.stdout:3/181: mkdir d5/d41 0 2024-04-12T05:08:55.244 INFO:tasks.workunit.client.1.smithi192.stdout:3/182: stat d5/d41 0 2024-04-12T05:08:55.248 INFO:tasks.workunit.client.1.smithi192.stdout:2/211: truncate d1/d8/d3a/f26 3699243 0 2024-04-12T05:08:55.250 INFO:tasks.workunit.client.1.smithi192.stdout:0/117: dwrite d0/f28 [0,4194304] 0 2024-04-12T05:08:55.252 INFO:tasks.workunit.client.1.smithi192.stdout:6/286: write d3/d8/f17 [883296,30723] 0 2024-04-12T05:08:55.253 INFO:tasks.workunit.client.1.smithi192.stdout:3/183: truncate d5/f2d 439740 0 2024-04-12T05:08:55.254 INFO:tasks.workunit.client.1.smithi192.stdout:2/212: symlink d1/d8/d17/l49 0 2024-04-12T05:08:55.255 INFO:tasks.workunit.client.1.smithi192.stdout:3/184: readlink d5/d13/d1a/l32 0 2024-04-12T05:08:55.258 INFO:tasks.workunit.client.0.smithi113.stdout:0/366: link d1/d6/d16/f4c d1/d6/d16/d2a/f71 0 2024-04-12T05:08:55.259 INFO:tasks.workunit.client.1.smithi192.stdout:5/247: dwrite d0/d21/f36 [0,4194304] 0 2024-04-12T05:08:55.259 INFO:tasks.workunit.client.1.smithi192.stdout:2/213: mknod d1/d8/d3a/c4a 0 2024-04-12T05:08:55.259 INFO:tasks.workunit.client.1.smithi192.stdout:0/118: read d0/d6/f13 [868571,21186] 0 2024-04-12T05:08:55.260 INFO:tasks.workunit.client.1.smithi192.stdout:6/287: write d3/d8/d48/f63 [362103,65875] 0 2024-04-12T05:08:55.262 INFO:tasks.workunit.client.1.smithi192.stdout:6/288: chown d3/dc/d39/f4e 11034475 1 2024-04-12T05:08:55.262 INFO:tasks.workunit.client.1.smithi192.stdout:5/248: symlink d0/d21/l5c 0 2024-04-12T05:08:55.263 INFO:tasks.workunit.client.0.smithi113.stdout:0/367: symlink d1/d43/d59/l72 0 2024-04-12T05:08:55.263 INFO:tasks.workunit.client.1.smithi192.stdout:5/249: stat d0/l18 0 2024-04-12T05:08:55.268 INFO:tasks.workunit.client.1.smithi192.stdout:0/119: creat d0/d26/d29/f2b x:0 0 0 2024-04-12T05:08:55.273 INFO:tasks.workunit.client.1.smithi192.stdout:6/289: dwrite d3/d5/f66 [0,4194304] 0 2024-04-12T05:08:55.275 INFO:tasks.workunit.client.1.smithi192.stdout:0/120: symlink d0/l2c 0 2024-04-12T05:08:55.275 INFO:tasks.workunit.client.1.smithi192.stdout:2/214: getdents d1/d2d 0 2024-04-12T05:08:55.276 INFO:tasks.workunit.client.1.smithi192.stdout:5/250: dwrite d0/d21/f26 [0,4194304] 0 2024-04-12T05:08:55.276 INFO:tasks.workunit.client.1.smithi192.stdout:0/121: stat d0/l2c 0 2024-04-12T05:08:55.277 INFO:tasks.workunit.client.1.smithi192.stdout:2/215: stat d1/d8/d3a/de/f11 0 2024-04-12T05:08:55.278 INFO:tasks.workunit.client.1.smithi192.stdout:6/290: creat d3/dc/f69 x:0 0 0 2024-04-12T05:08:55.278 INFO:tasks.workunit.client.0.smithi113.stdout:0/368: dread d1/d6/d16/d2a/f70 [0,4194304] 0 2024-04-12T05:08:55.282 INFO:tasks.workunit.client.1.smithi192.stdout:0/122: truncate d0/f17 183028 0 2024-04-12T05:08:55.282 INFO:tasks.workunit.client.0.smithi113.stdout:0/369: rmdir d1/d6/d16/d2a 39 2024-04-12T05:08:55.282 INFO:tasks.workunit.client.0.smithi113.stdout:0/370: truncate d1/d6/f30 5148908 0 2024-04-12T05:08:55.285 INFO:tasks.workunit.client.1.smithi192.stdout:6/291: mknod d3/d8/d10/d4c/c6a 0 2024-04-12T05:08:55.285 INFO:tasks.workunit.client.1.smithi192.stdout:5/251: getdents d0/d15/d32/d5a 0 2024-04-12T05:08:55.285 INFO:tasks.workunit.client.1.smithi192.stdout:3/185: write d5/f19 [936640,21568] 0 2024-04-12T05:08:55.285 INFO:tasks.workunit.client.0.smithi113.stdout:0/371: creat d1/d43/f73 x:0 0 0 2024-04-12T05:08:55.286 INFO:tasks.workunit.client.1.smithi192.stdout:5/252: chown d0/d15/d3e 14967652 1 2024-04-12T05:08:55.287 INFO:tasks.workunit.client.1.smithi192.stdout:3/186: dread - d5/d13/d3c/d36/f3f zero size 2024-04-12T05:08:55.287 INFO:tasks.workunit.client.1.smithi192.stdout:0/123: link d0/d6/d15/f27 d0/d6/d15/d1b/f2d 0 2024-04-12T05:08:55.291 INFO:tasks.workunit.client.1.smithi192.stdout:0/124: mkdir d0/d26/d2e 0 2024-04-12T05:08:55.293 INFO:tasks.workunit.client.1.smithi192.stdout:0/125: fdatasync d0/d26/d29/f2b 0 2024-04-12T05:08:55.295 INFO:tasks.workunit.client.1.smithi192.stdout:0/126: write d0/d6/d15/d19/f1d [557497,82339] 0 2024-04-12T05:08:55.296 INFO:tasks.workunit.client.0.smithi113.stdout:0/372: dwrite d1/d6/da/f4d [0,4194304] 0 2024-04-12T05:08:55.296 INFO:tasks.workunit.client.1.smithi192.stdout:3/187: dwrite d5/d13/f18 [0,4194304] 0 2024-04-12T05:08:55.296 INFO:tasks.workunit.client.1.smithi192.stdout:6/292: rename d3/d8/c16 to d3/d21/c6b 0 2024-04-12T05:08:55.298 INFO:tasks.workunit.client.1.smithi192.stdout:3/188: fdatasync d5/d13/d1a/f3b 0 2024-04-12T05:08:55.299 INFO:tasks.workunit.client.1.smithi192.stdout:6/293: chown d3/d8/d22/f65 156 1 2024-04-12T05:08:55.299 INFO:tasks.workunit.client.1.smithi192.stdout:7/172: sync 2024-04-12T05:08:55.300 INFO:tasks.workunit.client.1.smithi192.stdout:0/127: symlink d0/d6/d22/l2f 0 2024-04-12T05:08:55.301 INFO:tasks.workunit.client.1.smithi192.stdout:0/128: chown d0/d6/d15/d1b 10575148 1 2024-04-12T05:08:55.305 INFO:tasks.workunit.client.0.smithi113.stdout:0/373: creat d1/d6/da/d36/f74 x:0 0 0 2024-04-12T05:08:55.305 INFO:tasks.workunit.client.1.smithi192.stdout:7/173: read d3/d7/d27/f30 [77389,100350] 0 2024-04-12T05:08:55.305 INFO:tasks.workunit.client.1.smithi192.stdout:5/253: write d0/d21/f2f [563206,11017] 0 2024-04-12T05:08:55.307 INFO:tasks.workunit.client.1.smithi192.stdout:0/129: creat d0/d6/d15/d19/f30 x:0 0 0 2024-04-12T05:08:55.309 INFO:tasks.workunit.client.1.smithi192.stdout:2/216: dread d1/d8/d3a/de/f2f [0,4194304] 0 2024-04-12T05:08:55.310 INFO:tasks.workunit.client.1.smithi192.stdout:3/189: creat d5/d13/f42 x:0 0 0 2024-04-12T05:08:55.313 INFO:tasks.workunit.client.1.smithi192.stdout:0/130: creat d0/d6/d15/d19/f31 x:0 0 0 2024-04-12T05:08:55.314 INFO:tasks.workunit.client.1.smithi192.stdout:2/217: write d1/d8/f35 [826875,39727] 0 2024-04-12T05:08:55.314 INFO:tasks.workunit.client.1.smithi192.stdout:3/190: chown d5/d13/d1a/c30 3121 1 2024-04-12T05:08:55.316 INFO:tasks.workunit.client.0.smithi113.stdout:0/374: truncate d1/d6/d16/d2a/f70 1286673 0 2024-04-12T05:08:55.316 INFO:tasks.workunit.client.0.smithi113.stdout:0/375: readlink d1/d6/da/l41 0 2024-04-12T05:08:55.318 INFO:tasks.workunit.client.1.smithi192.stdout:3/191: chown d5/c6 9201 1 2024-04-12T05:08:55.318 INFO:tasks.workunit.client.1.smithi192.stdout:2/218: write d1/d2d/f3d [604800,43321] 0 2024-04-12T05:08:55.319 INFO:tasks.workunit.client.0.smithi113.stdout:5/292: sync 2024-04-12T05:08:55.319 INFO:tasks.workunit.client.1.smithi192.stdout:5/254: dwrite d0/f1b [0,4194304] 0 2024-04-12T05:08:55.320 INFO:tasks.workunit.client.0.smithi113.stdout:0/376: chown d1/d6/d34/f68 4050397 1 2024-04-12T05:08:55.320 INFO:tasks.workunit.client.0.smithi113.stdout:5/293: dread - d0/f3a zero size 2024-04-12T05:08:55.322 INFO:tasks.workunit.client.1.smithi192.stdout:2/219: stat d1/d8/d3a/f3f 0 2024-04-12T05:08:55.322 INFO:tasks.workunit.client.0.smithi113.stdout:0/377: dread - d1/d6/d16/d2a/d50/f65 zero size 2024-04-12T05:08:55.323 INFO:tasks.workunit.client.0.smithi113.stdout:0/378: fdatasync d1/d6/da/d36/f74 0 2024-04-12T05:08:55.326 INFO:tasks.workunit.client.1.smithi192.stdout:2/220: stat d1/f44 0 2024-04-12T05:08:55.326 INFO:tasks.workunit.client.1.smithi192.stdout:3/192: rmdir d5/d41 0 2024-04-12T05:08:55.329 INFO:tasks.workunit.client.0.smithi113.stdout:0/379: creat d1/f75 x:0 0 0 2024-04-12T05:08:55.329 INFO:tasks.workunit.client.1.smithi192.stdout:5/255: unlink d0/d15/d32/f38 0 2024-04-12T05:08:55.329 INFO:tasks.workunit.client.1.smithi192.stdout:6/294: dwrite d3/d8/d10/d4c/f62 [0,4194304] 0 2024-04-12T05:08:55.330 INFO:tasks.workunit.client.1.smithi192.stdout:2/221: creat d1/f4b x:0 0 0 2024-04-12T05:08:55.334 INFO:tasks.workunit.client.1.smithi192.stdout:0/131: dwrite d0/d6/d15/d19/f31 [0,4194304] 0 2024-04-12T05:08:55.338 INFO:tasks.workunit.client.1.smithi192.stdout:6/295: symlink d3/d26/l6c 0 2024-04-12T05:08:55.339 INFO:tasks.workunit.client.1.smithi192.stdout:3/193: symlink d5/d13/l43 0 2024-04-12T05:08:55.339 INFO:tasks.workunit.client.1.smithi192.stdout:2/222: dread d1/d8/d17/f2a [0,4194304] 0 2024-04-12T05:08:55.342 INFO:tasks.workunit.client.1.smithi192.stdout:3/194: fdatasync d5/d13/f18 0 2024-04-12T05:08:55.342 INFO:tasks.workunit.client.1.smithi192.stdout:7/174: dread d3/f12 [4194304,4194304] 0 2024-04-12T05:08:55.345 INFO:tasks.workunit.client.0.smithi113.stdout:0/380: dwrite d1/d6/d16/d2a/d2c/f5f [0,4194304] 0 2024-04-12T05:08:55.346 INFO:tasks.workunit.client.1.smithi192.stdout:2/223: truncate d1/d8/f46 1342065 0 2024-04-12T05:08:55.349 INFO:tasks.workunit.client.1.smithi192.stdout:5/256: dwrite d0/d21/f36 [4194304,4194304] 0 2024-04-12T05:08:55.353 INFO:tasks.workunit.client.0.smithi113.stdout:0/381: write d1/d9/f45 [1433102,88966] 0 2024-04-12T05:08:55.354 INFO:tasks.workunit.client.1.smithi192.stdout:3/195: creat d5/d13/d3c/d26/f44 x:0 0 0 2024-04-12T05:08:55.354 INFO:tasks.workunit.client.0.smithi113.stdout:0/382: chown d1/d6/da/f5c 0 1 2024-04-12T05:08:55.356 INFO:tasks.workunit.client.1.smithi192.stdout:6/296: dwrite d3/d8/d22/f2e [0,4194304] 0 2024-04-12T05:08:55.357 INFO:tasks.workunit.client.0.smithi113.stdout:0/383: stat d1/d43/d59/f5b 0 2024-04-12T05:08:55.357 INFO:tasks.workunit.client.1.smithi192.stdout:5/257: rmdir d0/d15 39 2024-04-12T05:08:55.357 INFO:tasks.workunit.client.0.smithi113.stdout:0/384: write d1/d9/f5a [125584,112334] 0 2024-04-12T05:08:55.359 INFO:tasks.workunit.client.1.smithi192.stdout:0/132: dwrite d0/d6/f21 [0,4194304] 0 2024-04-12T05:08:55.362 INFO:tasks.workunit.client.0.smithi113.stdout:0/385: symlink d1/d6/da/d36/l76 0 2024-04-12T05:08:55.363 INFO:tasks.workunit.client.1.smithi192.stdout:2/224: rename d1/d8/d3a/de/f1e to d1/d8/d17/d19/d40/f4c 0 2024-04-12T05:08:55.364 INFO:tasks.workunit.client.1.smithi192.stdout:3/196: symlink d5/d13/d3c/l45 0 2024-04-12T05:08:55.364 INFO:tasks.workunit.client.0.smithi113.stdout:0/386: chown d1/d43/f73 897909955 1 2024-04-12T05:08:55.365 INFO:tasks.workunit.client.0.smithi113.stdout:0/387: dread - d1/d6/d16/d2a/f71 zero size 2024-04-12T05:08:55.367 INFO:tasks.workunit.client.0.smithi113.stdout:0/388: write d1/d43/f62 [1859707,125371] 0 2024-04-12T05:08:55.368 INFO:tasks.workunit.client.1.smithi192.stdout:5/258: symlink d0/d15/d32/l5d 0 2024-04-12T05:08:55.369 INFO:tasks.workunit.client.1.smithi192.stdout:6/297: write d3/d26/f4b [39701,62533] 0 2024-04-12T05:08:55.370 INFO:tasks.workunit.client.1.smithi192.stdout:2/225: chown d1/d8/d17/l49 0 1 2024-04-12T05:08:55.370 INFO:tasks.workunit.client.0.smithi113.stdout:0/389: write d1/d6/d16/d2a/d2c/f5f [833162,75119] 0 2024-04-12T05:08:55.370 INFO:tasks.workunit.client.1.smithi192.stdout:3/197: write d5/d13/f42 [654849,124330] 0 2024-04-12T05:08:55.374 INFO:tasks.workunit.client.1.smithi192.stdout:7/175: dread d3/f21 [0,4194304] 0 2024-04-12T05:08:55.376 INFO:tasks.workunit.client.1.smithi192.stdout:0/133: dwrite d0/f28 [0,4194304] 0 2024-04-12T05:08:55.376 INFO:tasks.workunit.client.1.smithi192.stdout:2/226: dread - d1/d2d/f34 zero size 2024-04-12T05:08:55.376 INFO:tasks.workunit.client.1.smithi192.stdout:3/198: write d5/d13/f2e [1497834,25566] 0 2024-04-12T05:08:55.379 INFO:tasks.workunit.client.1.smithi192.stdout:6/298: unlink d3/d8/d10/l53 0 2024-04-12T05:08:55.380 INFO:tasks.workunit.client.0.smithi113.stdout:0/390: truncate d1/d6/d16/d2a/d2c/f37 1089678 0 2024-04-12T05:08:55.381 INFO:tasks.workunit.client.1.smithi192.stdout:0/134: dread - d0/d6/d15/f27 zero size 2024-04-12T05:08:55.381 INFO:tasks.workunit.client.1.smithi192.stdout:7/176: chown d3/d7/d15/d1a/l2a 22 1 2024-04-12T05:08:55.382 INFO:tasks.workunit.client.1.smithi192.stdout:3/199: chown d5/d13/d3c/d36/l3e 0 1 2024-04-12T05:08:55.383 INFO:tasks.workunit.client.1.smithi192.stdout:3/200: chown d5/d13/d1a/l38 992 1 2024-04-12T05:08:55.385 INFO:tasks.workunit.client.1.smithi192.stdout:2/227: symlink d1/d8/l4d 0 2024-04-12T05:08:55.385 INFO:tasks.workunit.client.1.smithi192.stdout:5/259: dread d0/d21/f26 [0,4194304] 0 2024-04-12T05:08:55.385 INFO:tasks.workunit.client.0.smithi113.stdout:0/391: creat d1/d6/d16/d2a/d50/f77 x:0 0 0 2024-04-12T05:08:55.385 INFO:tasks.workunit.client.1.smithi192.stdout:7/177: creat d3/d1d/f41 x:0 0 0 2024-04-12T05:08:55.386 INFO:tasks.workunit.client.0.smithi113.stdout:0/392: readlink d1/d6/d16/d2a/d50/l63 0 2024-04-12T05:08:55.386 INFO:tasks.workunit.client.1.smithi192.stdout:0/135: symlink d0/l32 0 2024-04-12T05:08:55.389 INFO:tasks.workunit.client.1.smithi192.stdout:0/136: dread - d0/d6/d15/d19/f30 zero size 2024-04-12T05:08:55.389 INFO:tasks.workunit.client.0.smithi113.stdout:0/393: unlink d1/d43/d59/f66 0 2024-04-12T05:08:55.390 INFO:tasks.workunit.client.1.smithi192.stdout:6/299: creat d3/d8/f6d x:0 0 0 2024-04-12T05:08:55.392 INFO:tasks.workunit.client.1.smithi192.stdout:0/137: mkdir d0/d6/d22/d33 0 2024-04-12T05:08:55.393 INFO:tasks.workunit.client.1.smithi192.stdout:0/138: truncate d0/f16 1309384 0 2024-04-12T05:08:55.394 INFO:tasks.workunit.client.1.smithi192.stdout:7/178: symlink d3/d7/d14/l42 0 2024-04-12T05:08:55.396 INFO:tasks.workunit.client.0.smithi113.stdout:0/394: dread d1/d6/d16/f47 [0,4194304] 0 2024-04-12T05:08:55.397 INFO:tasks.workunit.client.0.smithi113.stdout:0/395: readlink d1/d6/d16/d2a/d50/d52/l55 0 2024-04-12T05:08:55.397 INFO:tasks.workunit.client.1.smithi192.stdout:5/260: getdents d0 0 2024-04-12T05:08:55.399 INFO:tasks.workunit.client.1.smithi192.stdout:2/228: getdents d1/d8/d3a 0 2024-04-12T05:08:55.399 INFO:tasks.workunit.client.1.smithi192.stdout:7/179: truncate d3/d7/d15/d1a/f3e 836268 0 2024-04-12T05:08:55.401 INFO:tasks.workunit.client.1.smithi192.stdout:5/261: read d0/d15/f23 [2953390,21802] 0 2024-04-12T05:08:55.401 INFO:tasks.workunit.client.1.smithi192.stdout:2/229: write d1/d2d/f3d [1117725,93064] 0 2024-04-12T05:08:55.402 INFO:tasks.workunit.client.1.smithi192.stdout:7/180: symlink d3/d1d/l43 0 2024-04-12T05:08:55.404 INFO:tasks.workunit.client.1.smithi192.stdout:5/262: mkdir d0/d21/d5e 0 2024-04-12T05:08:55.405 INFO:tasks.workunit.client.1.smithi192.stdout:0/139: dread d0/d6/d15/d19/f1c [0,4194304] 0 2024-04-12T05:08:55.408 INFO:tasks.workunit.client.1.smithi192.stdout:0/140: fsync d0/d6/f13 0 2024-04-12T05:08:55.409 INFO:tasks.workunit.client.1.smithi192.stdout:3/201: dwrite d5/fd [0,4194304] 0 2024-04-12T05:08:55.411 INFO:tasks.workunit.client.1.smithi192.stdout:5/263: rename d0/d15/c43 to d0/d15/d32/d5a/c5f 0 2024-04-12T05:08:55.411 INFO:tasks.workunit.client.1.smithi192.stdout:9/153: sync 2024-04-12T05:08:55.411 INFO:tasks.workunit.client.1.smithi192.stdout:8/173: sync 2024-04-12T05:08:55.413 INFO:tasks.workunit.client.1.smithi192.stdout:0/141: mkdir d0/d6/d15/d34 0 2024-04-12T05:08:55.417 INFO:tasks.workunit.client.1.smithi192.stdout:7/181: dwrite d3/d7/d15/d1a/f40 [0,4194304] 0 2024-04-12T05:08:55.419 INFO:tasks.workunit.client.1.smithi192.stdout:0/142: stat d0/d6/d15/d19/f1d 0 2024-04-12T05:08:55.425 INFO:tasks.workunit.client.1.smithi192.stdout:6/300: dread d3/d8/f1c [0,4194304] 0 2024-04-12T05:08:55.425 INFO:tasks.workunit.client.1.smithi192.stdout:9/154: rename d11/d15/d21/d2c/c23 to d11/d15/c34 0 2024-04-12T05:08:55.425 INFO:tasks.workunit.client.1.smithi192.stdout:8/174: truncate d6/da/fd 6049828 0 2024-04-12T05:08:55.427 INFO:tasks.workunit.client.1.smithi192.stdout:3/202: dwrite d5/f14 [8388608,4194304] 0 2024-04-12T05:08:55.431 INFO:tasks.workunit.client.1.smithi192.stdout:6/301: rmdir d3/d8/d22 39 2024-04-12T05:08:55.432 INFO:tasks.workunit.client.1.smithi192.stdout:2/230: dread d1/d8/d17/d19/d40/f4c [0,4194304] 0 2024-04-12T05:08:55.433 INFO:tasks.workunit.client.1.smithi192.stdout:8/175: unlink d6/da/fc 0 2024-04-12T05:08:55.434 INFO:tasks.workunit.client.1.smithi192.stdout:5/264: dread d0/d21/f36 [4194304,4194304] 0 2024-04-12T05:08:55.437 INFO:tasks.workunit.client.1.smithi192.stdout:9/155: dwrite f10 [0,4194304] 0 2024-04-12T05:08:55.437 INFO:tasks.workunit.client.1.smithi192.stdout:7/182: rename d3/d7/d15/d22/l3f to d3/l44 0 2024-04-12T05:08:55.439 INFO:tasks.workunit.client.1.smithi192.stdout:8/176: write f3 [61648,112498] 0 2024-04-12T05:08:55.442 INFO:tasks.workunit.client.1.smithi192.stdout:9/156: symlink d11/d15/d21/d2c/l35 0 2024-04-12T05:08:55.445 INFO:tasks.workunit.client.1.smithi192.stdout:7/183: mknod d3/d7/d15/d22/c45 0 2024-04-12T05:08:55.445 INFO:tasks.workunit.client.1.smithi192.stdout:7/184: stat d3/d7/f9 0 2024-04-12T05:08:55.446 INFO:tasks.workunit.client.1.smithi192.stdout:6/302: creat d3/d21/f6e x:0 0 0 2024-04-12T05:08:55.446 INFO:tasks.workunit.client.1.smithi192.stdout:2/231: symlink d1/d8/l4e 0 2024-04-12T05:08:55.447 INFO:tasks.workunit.client.1.smithi192.stdout:8/177: mkdir d6/d21/d3c 0 2024-04-12T05:08:55.450 INFO:tasks.workunit.client.1.smithi192.stdout:2/232: mknod d1/d8/d17/d19/d40/c4f 0 2024-04-12T05:08:55.451 INFO:tasks.workunit.client.1.smithi192.stdout:0/143: write d0/d6/d15/f27 [406406,96094] 0 2024-04-12T05:08:55.454 INFO:tasks.workunit.client.1.smithi192.stdout:2/233: write d1/d8/d3a/f3f [7203,26785] 0 2024-04-12T05:08:55.454 INFO:tasks.workunit.client.1.smithi192.stdout:6/303: dwrite d3/d5/f45 [0,4194304] 0 2024-04-12T05:08:55.458 INFO:tasks.workunit.client.1.smithi192.stdout:9/157: write fb [351516,39444] 0 2024-04-12T05:08:55.461 INFO:tasks.workunit.client.1.smithi192.stdout:8/178: dwrite d6/f18 [0,4194304] 0 2024-04-12T05:08:55.462 INFO:tasks.workunit.client.1.smithi192.stdout:8/179: chown d6/df 4632999 1 2024-04-12T05:08:55.463 INFO:tasks.workunit.client.1.smithi192.stdout:9/158: mkdir d11/d25/d36 0 2024-04-12T05:08:55.464 INFO:tasks.workunit.client.1.smithi192.stdout:2/234: creat d1/d8/f50 x:0 0 0 2024-04-12T05:08:55.465 INFO:tasks.workunit.client.1.smithi192.stdout:2/235: chown d1/d8/d3a/f3f 499335 1 2024-04-12T05:08:55.468 INFO:tasks.workunit.client.1.smithi192.stdout:9/159: rename d11/d15/d1e/d2e to d11/d37 0 2024-04-12T05:08:55.469 INFO:tasks.workunit.client.1.smithi192.stdout:2/236: symlink d1/d8/d3a/l51 0 2024-04-12T05:08:55.470 INFO:tasks.workunit.client.1.smithi192.stdout:6/304: dread d3/d8/d10/f58 [0,4194304] 0 2024-04-12T05:08:55.471 INFO:tasks.workunit.client.1.smithi192.stdout:2/237: write d1/d8/f10 [1411100,94563] 0 2024-04-12T05:08:55.474 INFO:tasks.workunit.client.1.smithi192.stdout:9/160: readlink d11/l27 0 2024-04-12T05:08:55.475 INFO:tasks.workunit.client.1.smithi192.stdout:9/161: chown f3 6 1 2024-04-12T05:08:55.475 INFO:tasks.workunit.client.1.smithi192.stdout:7/185: dread d3/d7/d14/d1f/f34 [0,4194304] 0 2024-04-12T05:08:55.476 INFO:tasks.workunit.client.1.smithi192.stdout:2/238: write d1/d2d/f34 [644178,65977] 0 2024-04-12T05:08:55.478 INFO:tasks.workunit.client.1.smithi192.stdout:0/144: truncate d0/d6/f25 2814805 0 2024-04-12T05:08:55.481 INFO:tasks.workunit.client.1.smithi192.stdout:7/186: creat d3/d11/f46 x:0 0 0 2024-04-12T05:08:55.483 INFO:tasks.workunit.client.1.smithi192.stdout:2/239: creat d1/d8/d17/d19/d40/f52 x:0 0 0 2024-04-12T05:08:55.483 INFO:tasks.workunit.client.1.smithi192.stdout:6/305: getdents d3/d23/d67 0 2024-04-12T05:08:55.485 INFO:tasks.workunit.client.1.smithi192.stdout:0/145: chown d0/d6/f25 26878749 1 2024-04-12T05:08:55.485 INFO:tasks.workunit.client.1.smithi192.stdout:6/306: write d3/f37 [418862,81928] 0 2024-04-12T05:08:55.486 INFO:tasks.workunit.client.1.smithi192.stdout:2/240: creat d1/d8/d3a/f53 x:0 0 0 2024-04-12T05:08:55.488 INFO:tasks.workunit.client.1.smithi192.stdout:6/307: chown d3/d8/d10 2642825 1 2024-04-12T05:08:55.489 INFO:tasks.workunit.client.1.smithi192.stdout:2/241: rmdir d1/d2d 39 2024-04-12T05:08:55.491 INFO:tasks.workunit.client.1.smithi192.stdout:7/187: dwrite d3/d11/f3b [0,4194304] 0 2024-04-12T05:08:55.494 INFO:tasks.workunit.client.1.smithi192.stdout:9/162: truncate d11/f13 975614 0 2024-04-12T05:08:55.494 INFO:tasks.workunit.client.1.smithi192.stdout:2/242: read d1/d8/d3a/f3f [10755,83931] 0 2024-04-12T05:08:55.495 INFO:tasks.workunit.client.1.smithi192.stdout:7/188: dread - d3/d7/d15/f3c zero size 2024-04-12T05:08:55.497 INFO:tasks.workunit.client.1.smithi192.stdout:0/146: dread d0/fb [0,4194304] 0 2024-04-12T05:08:55.500 INFO:tasks.workunit.client.1.smithi192.stdout:2/243: chown d1/d8/d17/d19/d40/f52 270545643 1 2024-04-12T05:08:55.502 INFO:tasks.workunit.client.1.smithi192.stdout:6/308: dwrite d3/f43 [0,4194304] 0 2024-04-12T05:08:55.513 INFO:tasks.workunit.client.1.smithi192.stdout:2/244: dread d1/d8/d3a/de/f2f [0,4194304] 0 2024-04-12T05:08:55.514 INFO:tasks.workunit.client.1.smithi192.stdout:7/189: dwrite d3/d1d/f28 [0,4194304] 0 2024-04-12T05:08:55.516 INFO:tasks.workunit.client.1.smithi192.stdout:7/190: write d3/d1d/f2e [1099161,87997] 0 2024-04-12T05:08:55.518 INFO:tasks.workunit.client.1.smithi192.stdout:0/147: dread d0/d6/f25 [0,4194304] 0 2024-04-12T05:08:55.524 INFO:tasks.workunit.client.1.smithi192.stdout:6/309: truncate d3/d5/ff 356672 0 2024-04-12T05:08:55.524 INFO:tasks.workunit.client.0.smithi113.stdout:2/218: sync 2024-04-12T05:08:55.525 INFO:tasks.workunit.client.0.smithi113.stdout:2/219: stat d8/d9/de/d12/d3c 0 2024-04-12T05:08:55.526 INFO:tasks.workunit.client.1.smithi192.stdout:9/163: dread f6 [0,4194304] 0 2024-04-12T05:08:55.527 INFO:tasks.workunit.client.1.smithi192.stdout:7/191: creat d3/d7/d14/f47 x:0 0 0 2024-04-12T05:08:55.528 INFO:tasks.workunit.client.1.smithi192.stdout:0/148: write d0/d6/d15/d19/f1c [1093998,62581] 0 2024-04-12T05:08:55.528 INFO:tasks.workunit.client.0.smithi113.stdout:2/220: unlink d8/d9/d1a/c33 0 2024-04-12T05:08:55.528 INFO:tasks.workunit.client.1.smithi192.stdout:2/245: dwrite d1/f4 [4194304,4194304] 0 2024-04-12T05:08:55.529 INFO:tasks.workunit.client.0.smithi113.stdout:2/221: read d8/fd [2392383,31883] 0 2024-04-12T05:08:55.530 INFO:tasks.workunit.client.1.smithi192.stdout:0/149: readlink d0/l10 0 2024-04-12T05:08:55.532 INFO:tasks.workunit.client.0.smithi113.stdout:2/222: write d8/d9/de/d15/d24/f3e [681924,40135] 0 2024-04-12T05:08:55.533 INFO:tasks.workunit.client.1.smithi192.stdout:7/192: mkdir d3/d1d/d48 0 2024-04-12T05:08:55.535 INFO:tasks.workunit.client.1.smithi192.stdout:9/164: dread d11/d15/d21/d2c/f1a [0,4194304] 0 2024-04-12T05:08:55.536 INFO:tasks.workunit.client.1.smithi192.stdout:0/150: unlink d0/d6/l1e 0 2024-04-12T05:08:55.536 INFO:tasks.workunit.client.0.smithi113.stdout:2/223: creat d8/d9/de/f42 x:0 0 0 2024-04-12T05:08:55.537 INFO:tasks.workunit.client.1.smithi192.stdout:2/246: rename d1/d8/d17/d3c/l47 to d1/d8/d17/d19/l54 0 2024-04-12T05:08:55.538 INFO:tasks.workunit.client.1.smithi192.stdout:0/151: truncate d0/f16 1948822 0 2024-04-12T05:08:55.539 INFO:tasks.workunit.client.1.smithi192.stdout:2/247: chown d1/d8/f46 5 1 2024-04-12T05:08:55.543 INFO:tasks.workunit.client.1.smithi192.stdout:6/310: truncate d3/f43 844404 0 2024-04-12T05:08:55.543 INFO:tasks.workunit.client.1.smithi192.stdout:0/152: truncate d0/d6/d15/f27 875022 0 2024-04-12T05:08:55.543 INFO:tasks.workunit.client.1.smithi192.stdout:2/248: truncate d1/d8/f41 500908 0 2024-04-12T05:08:55.543 INFO:tasks.workunit.client.1.smithi192.stdout:9/165: dwrite f3 [0,4194304] 0 2024-04-12T05:08:55.544 INFO:tasks.workunit.client.1.smithi192.stdout:7/193: creat d3/d7/d14/f49 x:0 0 0 2024-04-12T05:08:55.546 INFO:tasks.workunit.client.1.smithi192.stdout:2/249: read - d1/d8/d3a/f53 zero size 2024-04-12T05:08:55.547 INFO:tasks.workunit.client.1.smithi192.stdout:2/250: readlink d1/d8/d17/d19/l3e 0 2024-04-12T05:08:55.548 INFO:tasks.workunit.client.1.smithi192.stdout:7/194: mkdir d3/d1d/d4a 0 2024-04-12T05:08:55.548 INFO:tasks.workunit.client.0.smithi113.stdout:2/224: rmdir d8/d9/de 39 2024-04-12T05:08:55.549 INFO:tasks.workunit.client.0.smithi113.stdout:2/225: fdatasync d8/d9/d27/f28 0 2024-04-12T05:08:55.554 INFO:tasks.workunit.client.1.smithi192.stdout:6/311: rename d3/d26/f31 to d3/d8/d48/f6f 0 2024-04-12T05:08:55.555 INFO:tasks.workunit.client.1.smithi192.stdout:7/195: symlink d3/d1d/l4b 0 2024-04-12T05:08:55.556 INFO:tasks.workunit.client.1.smithi192.stdout:9/166: creat d11/d25/d36/f38 x:0 0 0 2024-04-12T05:08:55.557 INFO:tasks.workunit.client.1.smithi192.stdout:7/196: write d3/d7/d15/d1a/f40 [4776436,111645] 0 2024-04-12T05:08:55.557 INFO:tasks.workunit.client.0.smithi113.stdout:2/226: dwrite d8/d9/f30 [0,4194304] 0 2024-04-12T05:08:55.557 INFO:tasks.workunit.client.1.smithi192.stdout:0/153: mkdir d0/d6/d15/d34/d35 0 2024-04-12T05:08:55.558 INFO:tasks.workunit.client.1.smithi192.stdout:4/241: sync 2024-04-12T05:08:55.558 INFO:tasks.workunit.client.1.smithi192.stdout:1/186: sync 2024-04-12T05:08:55.560 INFO:tasks.workunit.client.1.smithi192.stdout:0/154: write d0/f17 [585414,127046] 0 2024-04-12T05:08:55.561 INFO:tasks.workunit.client.1.smithi192.stdout:6/312: creat d3/dc/d39/f70 x:0 0 0 2024-04-12T05:08:55.561 INFO:tasks.workunit.client.1.smithi192.stdout:4/242: dread - d1/da/d1c/f36 zero size 2024-04-12T05:08:55.565 INFO:tasks.workunit.client.0.smithi113.stdout:2/227: mkdir d8/d9/de/d43 0 2024-04-12T05:08:55.567 INFO:tasks.workunit.client.0.smithi113.stdout:2/228: fdatasync d8/d16/f1d 0 2024-04-12T05:08:55.570 INFO:tasks.workunit.client.0.smithi113.stdout:2/229: mkdir d8/d9/de/d12/d44 0 2024-04-12T05:08:55.570 INFO:tasks.workunit.client.1.smithi192.stdout:9/167: dwrite f7 [0,4194304] 0 2024-04-12T05:08:55.573 INFO:tasks.workunit.client.1.smithi192.stdout:4/243: rename d1/d20/d37/c3d to d1/d20/d37/c4d 0 2024-04-12T05:08:55.574 INFO:tasks.workunit.client.1.smithi192.stdout:0/155: link d0/d6/f25 d0/d6/d15/d19/f36 0 2024-04-12T05:08:55.575 INFO:tasks.workunit.client.1.smithi192.stdout:9/168: creat d11/d15/d1e/f39 x:0 0 0 2024-04-12T05:08:55.577 INFO:tasks.workunit.client.1.smithi192.stdout:6/313: getdents d3 0 2024-04-12T05:08:55.580 INFO:tasks.workunit.client.1.smithi192.stdout:0/156: creat d0/d6/d15/f37 x:0 0 0 2024-04-12T05:08:55.580 INFO:tasks.workunit.client.0.smithi113.stdout:2/230: write d8/d16/f19 [2441508,23649] 0 2024-04-12T05:08:55.585 INFO:tasks.workunit.client.1.smithi192.stdout:4/244: dread d1/da/f31 [0,4194304] 0 2024-04-12T05:08:55.586 INFO:tasks.workunit.client.0.smithi113.stdout:2/231: creat d8/d9/de/d12/d2b/f45 x:0 0 0 2024-04-12T05:08:55.587 INFO:tasks.workunit.client.0.smithi113.stdout:2/232: dread - d8/d9/d27/f28 zero size 2024-04-12T05:08:55.587 INFO:tasks.workunit.client.1.smithi192.stdout:9/169: getdents d11/d15/d21 0 2024-04-12T05:08:55.590 INFO:tasks.workunit.client.1.smithi192.stdout:9/170: write d11/f12 [1929789,128744] 0 2024-04-12T05:08:55.591 INFO:tasks.workunit.client.0.smithi113.stdout:2/233: rename d8/d9/f23 to d8/d16/f46 0 2024-04-12T05:08:55.593 INFO:tasks.workunit.client.1.smithi192.stdout:0/157: dwrite d0/d6/d15/d19/f1d [0,4194304] 0 2024-04-12T05:08:55.598 INFO:tasks.workunit.client.0.smithi113.stdout:2/234: dwrite d8/d9/de/f31 [0,4194304] 0 2024-04-12T05:08:55.599 INFO:tasks.workunit.client.1.smithi192.stdout:2/251: dread d1/d8/d3a/f2e [0,4194304] 0 2024-04-12T05:08:55.599 INFO:tasks.workunit.client.1.smithi192.stdout:0/158: chown d0/d6/c1a 0 1 2024-04-12T05:08:55.600 INFO:tasks.workunit.client.1.smithi192.stdout:6/314: write d3/d26/f46 [1155775,31311] 0 2024-04-12T05:08:55.601 INFO:tasks.workunit.client.0.smithi113.stdout:2/235: symlink d8/d9/de/d43/l47 0 2024-04-12T05:08:55.602 INFO:tasks.workunit.client.1.smithi192.stdout:2/252: chown d1/d8/c1d 315 1 2024-04-12T05:08:55.604 INFO:tasks.workunit.client.1.smithi192.stdout:4/245: dwrite d1/f3e [4194304,4194304] 0 2024-04-12T05:08:55.607 INFO:tasks.workunit.client.1.smithi192.stdout:1/187: dread d0/d2/dd/f15 [0,4194304] 0 2024-04-12T05:08:55.608 INFO:tasks.workunit.client.1.smithi192.stdout:4/246: chown d1/f26 124916 1 2024-04-12T05:08:55.608 INFO:tasks.workunit.client.1.smithi192.stdout:6/315: rename d3/d21/f40 to d3/d8/d10/f71 0 2024-04-12T05:08:55.619 INFO:tasks.workunit.client.1.smithi192.stdout:2/253: mknod d1/c55 0 2024-04-12T05:08:55.619 INFO:tasks.workunit.client.1.smithi192.stdout:0/159: dwrite d0/f28 [0,4194304] 0 2024-04-12T05:08:55.622 INFO:tasks.workunit.client.1.smithi192.stdout:1/188: getdents d0/d29 0 2024-04-12T05:08:55.622 INFO:tasks.workunit.client.1.smithi192.stdout:4/247: fdatasync d1/da/d1c/f36 0 2024-04-12T05:08:55.626 INFO:tasks.workunit.client.1.smithi192.stdout:2/254: write d1/f44 [155248,95166] 0 2024-04-12T05:08:55.627 INFO:tasks.workunit.client.1.smithi192.stdout:4/248: write d1/f4b [410944,6447] 0 2024-04-12T05:08:55.629 INFO:tasks.workunit.client.1.smithi192.stdout:4/249: dread - d1/d20/d37/f48 zero size 2024-04-12T05:08:55.629 INFO:tasks.workunit.client.1.smithi192.stdout:6/316: rename d3/d8/d48/f5b to d3/d5/f72 0 2024-04-12T05:08:55.632 INFO:tasks.workunit.client.1.smithi192.stdout:0/160: link d0/d26/d29/f2b d0/d6/d15/d34/f38 0 2024-04-12T05:08:55.632 INFO:tasks.workunit.client.1.smithi192.stdout:2/255: write d1/d2d/f3d [1984021,9350] 0 2024-04-12T05:08:55.632 INFO:tasks.workunit.client.1.smithi192.stdout:9/171: dread d11/d15/d21/f24 [0,4194304] 0 2024-04-12T05:08:55.634 INFO:tasks.workunit.client.1.smithi192.stdout:6/317: rename d3/d8/d10/f58 to d3/d8/d48/f73 0 2024-04-12T05:08:55.636 INFO:tasks.workunit.client.1.smithi192.stdout:4/250: link d1/f33 d1/da/d1c/f4e 0 2024-04-12T05:08:55.637 INFO:tasks.workunit.client.1.smithi192.stdout:9/172: rmdir d11/d37 39 2024-04-12T05:08:55.640 INFO:tasks.workunit.client.1.smithi192.stdout:0/161: mknod d0/d6/d2a/c39 0 2024-04-12T05:08:55.641 INFO:tasks.workunit.client.1.smithi192.stdout:1/189: truncate d0/d2/f7 238470 0 2024-04-12T05:08:55.643 INFO:tasks.workunit.client.1.smithi192.stdout:6/318: dwrite d3/dc/d39/f70 [0,4194304] 0 2024-04-12T05:08:55.643 INFO:tasks.workunit.client.1.smithi192.stdout:2/256: creat d1/d8/f56 x:0 0 0 2024-04-12T05:08:55.644 INFO:tasks.workunit.client.1.smithi192.stdout:9/173: dread fb [0,4194304] 0 2024-04-12T05:08:55.646 INFO:tasks.workunit.client.1.smithi192.stdout:0/162: mkdir d0/d6/d15/d1b/d3a 0 2024-04-12T05:08:55.651 INFO:tasks.workunit.client.1.smithi192.stdout:0/163: unlink d0/f17 0 2024-04-12T05:08:55.655 INFO:tasks.workunit.client.1.smithi192.stdout:4/251: dwrite d1/da/f31 [0,4194304] 0 2024-04-12T05:08:55.656 INFO:tasks.workunit.client.1.smithi192.stdout:2/257: rmdir d1/d8 39 2024-04-12T05:08:55.658 INFO:tasks.workunit.client.1.smithi192.stdout:1/190: write d0/f18 [1527570,32214] 0 2024-04-12T05:08:55.660 INFO:tasks.workunit.client.1.smithi192.stdout:9/174: link d11/l27 d11/d15/d21/d2c/d1f/l3a 0 2024-04-12T05:08:55.661 INFO:tasks.workunit.client.1.smithi192.stdout:1/191: stat d0/d2/dd/f26 0 2024-04-12T05:08:55.662 INFO:tasks.workunit.client.1.smithi192.stdout:0/164: dread d0/d6/d15/d1b/f2d [0,4194304] 0 2024-04-12T05:08:55.663 INFO:tasks.workunit.client.1.smithi192.stdout:2/258: creat d1/d8/d3a/de/f57 x:0 0 0 2024-04-12T05:08:55.664 INFO:tasks.workunit.client.1.smithi192.stdout:9/175: mknod d11/d15/d21/d2c/c3b 0 2024-04-12T05:08:55.664 INFO:tasks.workunit.client.1.smithi192.stdout:6/319: getdents d3/d8/d56 0 2024-04-12T05:08:55.665 INFO:tasks.workunit.client.1.smithi192.stdout:4/252: getdents d1/d20/d37 0 2024-04-12T05:08:55.666 INFO:tasks.workunit.client.1.smithi192.stdout:4/253: chown d1/d1d 17689046 1 2024-04-12T05:08:55.667 INFO:tasks.workunit.client.1.smithi192.stdout:9/176: mknod d11/d15/d21/c3c 0 2024-04-12T05:08:55.668 INFO:tasks.workunit.client.1.smithi192.stdout:9/177: chown d11/d15/d21/d2c/l35 639667 1 2024-04-12T05:08:55.669 INFO:tasks.workunit.client.1.smithi192.stdout:1/192: link d0/d2/dd/d21/f2d d0/d1b/f35 0 2024-04-12T05:08:55.669 INFO:tasks.workunit.client.1.smithi192.stdout:1/193: dread - d0/d2/dd/f32 zero size 2024-04-12T05:08:55.670 INFO:tasks.workunit.client.1.smithi192.stdout:6/320: rename d3/d21/f6e to d3/d23/f74 0 2024-04-12T05:08:55.671 INFO:tasks.workunit.client.1.smithi192.stdout:4/254: symlink d1/d49/l4f 0 2024-04-12T05:08:55.671 INFO:tasks.workunit.client.1.smithi192.stdout:9/178: mkdir d11/d25/d36/d3d 0 2024-04-12T05:08:55.674 INFO:tasks.workunit.client.1.smithi192.stdout:1/194: creat d0/d2/d1e/f36 x:0 0 0 2024-04-12T05:08:55.674 INFO:tasks.workunit.client.1.smithi192.stdout:4/255: rmdir d1/d20 39 2024-04-12T05:08:55.677 INFO:tasks.workunit.client.1.smithi192.stdout:1/195: write d0/d1b/f35 [603017,16086] 0 2024-04-12T05:08:55.680 INFO:tasks.workunit.client.1.smithi192.stdout:4/256: symlink d1/d1d/d4c/l50 0 2024-04-12T05:08:55.681 INFO:tasks.workunit.client.1.smithi192.stdout:2/259: write d1/d8/d3a/f3f [799619,94262] 0 2024-04-12T05:08:55.683 INFO:tasks.workunit.client.1.smithi192.stdout:1/196: mknod d0/d2/dd/d21/d33/c37 0 2024-04-12T05:08:55.686 INFO:tasks.workunit.client.1.smithi192.stdout:6/321: dwrite d3/f43 [0,4194304] 0 2024-04-12T05:08:55.688 INFO:tasks.workunit.client.1.smithi192.stdout:2/260: unlink d1/d8/d3a/f53 0 2024-04-12T05:08:55.689 INFO:tasks.workunit.client.1.smithi192.stdout:9/179: dwrite fb [0,4194304] 0 2024-04-12T05:08:55.690 INFO:tasks.workunit.client.1.smithi192.stdout:9/180: stat d11/l2a 0 2024-04-12T05:08:55.691 INFO:tasks.workunit.client.1.smithi192.stdout:2/261: symlink d1/d2d/l58 0 2024-04-12T05:08:55.693 INFO:tasks.workunit.client.1.smithi192.stdout:9/181: readlink d11/l27 0 2024-04-12T05:08:55.694 INFO:tasks.workunit.client.1.smithi192.stdout:1/197: fsync d0/d1b/f35 0 2024-04-12T05:08:55.698 INFO:tasks.workunit.client.1.smithi192.stdout:9/182: unlink fa 0 2024-04-12T05:08:55.700 INFO:tasks.workunit.client.1.smithi192.stdout:6/322: dwrite d3/dc/f3b [0,4194304] 0 2024-04-12T05:08:55.702 INFO:tasks.workunit.client.1.smithi192.stdout:1/198: creat d0/d2/dd/d21/f38 x:0 0 0 2024-04-12T05:08:55.704 INFO:tasks.workunit.client.1.smithi192.stdout:1/199: chown d0/d2/d1e/c2e 43377756 1 2024-04-12T05:08:55.704 INFO:tasks.workunit.client.1.smithi192.stdout:9/183: unlink d11/l28 0 2024-04-12T05:08:55.704 INFO:tasks.workunit.client.1.smithi192.stdout:2/262: dwrite d1/d8/d17/d19/d40/f52 [0,4194304] 0 2024-04-12T05:08:55.705 INFO:tasks.workunit.client.1.smithi192.stdout:9/184: read fb [2318016,21442] 0 2024-04-12T05:08:55.710 INFO:tasks.workunit.client.1.smithi192.stdout:1/200: chown d0/d2/dd/f32 1548 1 2024-04-12T05:08:55.710 INFO:tasks.workunit.client.1.smithi192.stdout:9/185: write d11/d25/d36/f38 [257774,24878] 0 2024-04-12T05:08:55.710 INFO:tasks.workunit.client.1.smithi192.stdout:4/257: truncate d1/f2e 2169613 0 2024-04-12T05:08:55.710 INFO:tasks.workunit.client.1.smithi192.stdout:2/263: mkdir d1/d8/d17/d3c/d59 0 2024-04-12T05:08:55.710 INFO:tasks.workunit.client.1.smithi192.stdout:3/203: sync 2024-04-12T05:08:55.711 INFO:tasks.workunit.client.1.smithi192.stdout:4/258: chown d1/f33 996411 1 2024-04-12T05:08:55.714 INFO:tasks.workunit.client.1.smithi192.stdout:4/259: creat d1/da/d1c/f51 x:0 0 0 2024-04-12T05:08:55.716 INFO:tasks.workunit.client.1.smithi192.stdout:3/204: rename d5/cf to d5/d13/d3c/c46 0 2024-04-12T05:08:55.716 INFO:tasks.workunit.client.1.smithi192.stdout:2/264: rename d1/d8/d17/d3c to d1/d8/d17/d3c/d59/d5a 22 2024-04-12T05:08:55.717 INFO:tasks.workunit.client.1.smithi192.stdout:9/186: unlink d11/d15/d21/d2c/l1b 0 2024-04-12T05:08:55.717 INFO:tasks.workunit.client.1.smithi192.stdout:2/265: readlink d1/d8/d3a/de/l42 0 2024-04-12T05:08:55.720 INFO:tasks.workunit.client.1.smithi192.stdout:2/266: write d1/d8/d3a/de/f30 [2416317,14296] 0 2024-04-12T05:08:55.723 INFO:tasks.workunit.client.1.smithi192.stdout:2/267: truncate d1/d8/d17/f20 525057 0 2024-04-12T05:08:55.723 INFO:tasks.workunit.client.1.smithi192.stdout:2/268: chown d1/d8/f35 167 1 2024-04-12T05:08:55.724 INFO:tasks.workunit.client.1.smithi192.stdout:3/205: rename d5/d13/l27 to d5/d13/d1a/l47 0 2024-04-12T05:08:55.725 INFO:tasks.workunit.client.1.smithi192.stdout:3/206: read d5/f1c [49605,104462] 0 2024-04-12T05:08:55.725 INFO:tasks.workunit.client.1.smithi192.stdout:2/269: creat d1/d8/d17/d19/d40/f5b x:0 0 0 2024-04-12T05:08:55.727 INFO:tasks.workunit.client.1.smithi192.stdout:9/187: rename d11/d15/d1e/f2f to d11/d37/f3e 0 2024-04-12T05:08:55.728 INFO:tasks.workunit.client.1.smithi192.stdout:2/270: dread d1/d8/d17/f2a [0,4194304] 0 2024-04-12T05:08:55.752 INFO:tasks.workunit.client.0.smithi113.stdout:3/290: sync 2024-04-12T05:08:55.752 INFO:tasks.workunit.client.0.smithi113.stdout:6/298: sync 2024-04-12T05:08:55.752 INFO:tasks.workunit.client.0.smithi113.stdout:8/370: sync 2024-04-12T05:08:55.753 INFO:tasks.workunit.client.0.smithi113.stdout:6/299: readlink d8/de/d13/l4a 0 2024-04-12T05:08:55.758 INFO:tasks.workunit.client.0.smithi113.stdout:8/371: link d9/d16/d1e/d26/d32/d5b/c5f d9/d16/d1e/d26/d32/d48/c75 0 2024-04-12T05:08:55.760 INFO:tasks.workunit.client.0.smithi113.stdout:8/372: chown d9/d34/f57 47 1 2024-04-12T05:08:55.761 INFO:tasks.workunit.client.0.smithi113.stdout:8/373: fdatasync d9/d16/d3c/f74 0 2024-04-12T05:08:55.763 INFO:tasks.workunit.client.0.smithi113.stdout:6/300: dwrite d8/de/d13/fa [0,4194304] 0 2024-04-12T05:08:55.765 INFO:tasks.workunit.client.0.smithi113.stdout:8/374: creat d9/d16/d1e/d26/d32/d48/f76 x:0 0 0 2024-04-12T05:08:55.766 INFO:tasks.workunit.client.0.smithi113.stdout:6/301: chown d8/l33 6 1 2024-04-12T05:08:55.767 INFO:tasks.workunit.client.0.smithi113.stdout:6/302: readlink d8/de/l21 0 2024-04-12T05:08:55.768 INFO:tasks.workunit.client.0.smithi113.stdout:8/375: symlink d9/d16/d1c/l77 0 2024-04-12T05:08:55.770 INFO:tasks.workunit.client.0.smithi113.stdout:6/303: mkdir d8/de/d13/d11/d36/d3d/d4c/d26/d64 0 2024-04-12T05:08:55.771 INFO:tasks.workunit.client.0.smithi113.stdout:8/376: mkdir d9/d16/d1c/d59/d78 0 2024-04-12T05:08:55.772 INFO:tasks.workunit.client.0.smithi113.stdout:6/304: dread d8/de/f15 [0,4194304] 0 2024-04-12T05:08:55.773 INFO:tasks.workunit.client.0.smithi113.stdout:8/377: mknod d9/d16/d1e/d26/d32/d48/c79 0 2024-04-12T05:08:55.781 INFO:tasks.workunit.client.0.smithi113.stdout:3/291: dread d0/d13/f42 [0,4194304] 0 2024-04-12T05:08:55.781 INFO:tasks.workunit.client.0.smithi113.stdout:3/292: stat d0/d13/f42 0 2024-04-12T05:08:55.782 INFO:tasks.workunit.client.0.smithi113.stdout:3/293: dread - d0/d13/d47/f4a zero size 2024-04-12T05:08:55.784 INFO:tasks.workunit.client.0.smithi113.stdout:3/294: fdatasync d0/d13/d15/f2e 0 2024-04-12T05:08:55.786 INFO:tasks.workunit.client.0.smithi113.stdout:3/295: mknod d0/d13/d15/d28/d2a/d34/c5d 0 2024-04-12T05:08:55.786 INFO:tasks.workunit.client.0.smithi113.stdout:8/378: truncate d9/f53 227283 0 2024-04-12T05:08:55.788 INFO:tasks.workunit.client.0.smithi113.stdout:3/296: stat d0/d13/l14 0 2024-04-12T05:08:55.791 INFO:tasks.workunit.client.0.smithi113.stdout:3/297: rename d0/f10 to d0/f5e 0 2024-04-12T05:08:55.791 INFO:tasks.workunit.client.0.smithi113.stdout:6/305: dwrite d8/de/d13/fb [0,4194304] 0 2024-04-12T05:08:55.792 INFO:tasks.workunit.client.0.smithi113.stdout:5/294: sync 2024-04-12T05:08:55.793 INFO:tasks.workunit.client.0.smithi113.stdout:9/401: sync 2024-04-12T05:08:55.793 INFO:tasks.workunit.client.0.smithi113.stdout:7/361: sync 2024-04-12T05:08:55.793 INFO:tasks.workunit.client.0.smithi113.stdout:4/352: sync 2024-04-12T05:08:55.793 INFO:tasks.workunit.client.0.smithi113.stdout:0/396: sync 2024-04-12T05:08:55.793 INFO:tasks.workunit.client.0.smithi113.stdout:1/386: sync 2024-04-12T05:08:55.795 INFO:tasks.workunit.client.0.smithi113.stdout:9/402: read d1/d4/f77 [1455955,88648] 0 2024-04-12T05:08:55.796 INFO:tasks.workunit.client.0.smithi113.stdout:3/298: mkdir d0/d13/d15/d28/d2a/d34/d5f 0 2024-04-12T05:08:55.798 INFO:tasks.workunit.client.0.smithi113.stdout:3/299: readlink d0/d13/d15/l27 0 2024-04-12T05:08:55.798 INFO:tasks.workunit.client.0.smithi113.stdout:7/362: truncate d0/dc/d6/d1a/f58 676587 0 2024-04-12T05:08:55.798 INFO:tasks.workunit.client.0.smithi113.stdout:8/379: link d9/d16/d3c/l42 d9/d16/d1c/l7a 0 2024-04-12T05:08:55.799 INFO:tasks.workunit.client.0.smithi113.stdout:6/306: truncate d8/de/d13/d11/d36/d3d/d4c/d26/f51 2351578 0 2024-04-12T05:08:55.801 INFO:tasks.workunit.client.0.smithi113.stdout:9/403: read d1/d4/d5/d6/f48 [2327044,102719] 0 2024-04-12T05:08:55.803 INFO:tasks.workunit.client.0.smithi113.stdout:8/380: write d9/d16/d1e/d26/d32/f64 [661398,57032] 0 2024-04-12T05:08:55.803 INFO:tasks.workunit.client.0.smithi113.stdout:5/295: symlink d0/d10/d27/d1d/d38/d44/l59 0 2024-04-12T05:08:55.805 INFO:tasks.workunit.client.0.smithi113.stdout:1/387: dwrite d10/d2e/d31/f35 [0,4194304] 0 2024-04-12T05:08:55.806 INFO:tasks.workunit.client.0.smithi113.stdout:5/296: chown d0/f6 1897 1 2024-04-12T05:08:55.807 INFO:tasks.workunit.client.0.smithi113.stdout:8/381: dread - d9/d16/d1e/d26/d32/d48/f69 zero size 2024-04-12T05:08:55.808 INFO:tasks.workunit.client.0.smithi113.stdout:5/297: stat d0/d10/l30 0 2024-04-12T05:08:55.808 INFO:tasks.workunit.client.1.smithi192.stdout:2/271: fsync d1/d8/d17/d19/f45 0 2024-04-12T05:08:55.808 INFO:tasks.workunit.client.1.smithi192.stdout:9/188: mknod d11/d15/d21/d2c/d1f/c3f 0 2024-04-12T05:08:55.810 INFO:tasks.workunit.client.0.smithi113.stdout:3/300: unlink d0/f5e 0 2024-04-12T05:08:55.813 INFO:tasks.workunit.client.1.smithi192.stdout:2/272: readlink d1/d8/d17/d19/l54 0 2024-04-12T05:08:55.814 INFO:tasks.workunit.client.1.smithi192.stdout:3/207: dwrite d5/d13/d3c/f24 [4194304,4194304] 0 2024-04-12T05:08:55.814 INFO:tasks.workunit.client.0.smithi113.stdout:3/301: write d0/d13/f19 [2438243,95374] 0 2024-04-12T05:08:55.814 INFO:tasks.workunit.client.0.smithi113.stdout:9/404: dwrite d1/df/f36 [0,4194304] 0 2024-04-12T05:08:55.816 INFO:tasks.workunit.client.0.smithi113.stdout:9/405: chown d1/f46 4064304 1 2024-04-12T05:08:55.816 INFO:tasks.workunit.client.0.smithi113.stdout:3/302: write d0/d13/d47/f4a [256010,643] 0 2024-04-12T05:08:55.823 INFO:tasks.workunit.client.0.smithi113.stdout:0/397: write d1/f2f [291157,113235] 0 2024-04-12T05:08:55.823 INFO:tasks.workunit.client.0.smithi113.stdout:1/388: creat d10/d2e/f76 x:0 0 0 2024-04-12T05:08:55.823 INFO:tasks.workunit.client.0.smithi113.stdout:3/303: write d0/d13/d47/d4f/f35 [262625,61586] 0 2024-04-12T05:08:55.824 INFO:tasks.workunit.client.0.smithi113.stdout:8/382: fdatasync d9/fb 0 2024-04-12T05:08:55.825 INFO:tasks.workunit.client.0.smithi113.stdout:5/298: creat d0/d10/d27/d13/f5a x:0 0 0 2024-04-12T05:08:55.825 INFO:tasks.workunit.client.0.smithi113.stdout:7/363: unlink d0/dd/d2d/c56 0 2024-04-12T05:08:55.826 INFO:tasks.workunit.client.0.smithi113.stdout:3/304: dread - d0/d13/d15/d28/d2a/f36 zero size 2024-04-12T05:08:55.831 INFO:tasks.workunit.client.0.smithi113.stdout:0/398: creat d1/d6/d34/f78 x:0 0 0 2024-04-12T05:08:55.831 INFO:tasks.workunit.client.0.smithi113.stdout:1/389: mkdir d10/d41/d77 0 2024-04-12T05:08:55.831 INFO:tasks.workunit.client.0.smithi113.stdout:4/353: getdents da/dd/df/d14 0 2024-04-12T05:08:55.831 INFO:tasks.workunit.client.0.smithi113.stdout:0/399: dread - d1/d6/d16/f67 zero size 2024-04-12T05:08:55.832 INFO:tasks.workunit.client.0.smithi113.stdout:6/307: rmdir d8/de/d13/d2b/d2e 0 2024-04-12T05:08:55.833 INFO:tasks.workunit.client.0.smithi113.stdout:0/400: fsync d1/f75 0 2024-04-12T05:08:55.835 INFO:tasks.workunit.client.0.smithi113.stdout:9/406: rename d1/d4/d5/d17/l51 to d1/d4/d38/d65/l92 0 2024-04-12T05:08:55.837 INFO:tasks.workunit.client.0.smithi113.stdout:7/364: creat d0/dd/d11/d44/d5c/f79 x:0 0 0 2024-04-12T05:08:55.837 INFO:tasks.workunit.client.0.smithi113.stdout:3/305: creat d0/d13/f60 x:0 0 0 2024-04-12T05:08:55.839 INFO:tasks.workunit.client.0.smithi113.stdout:4/354: creat da/d4f/f72 x:0 0 0 2024-04-12T05:08:55.839 INFO:tasks.workunit.client.0.smithi113.stdout:1/390: dread d10/d2e/d31/d33/f50 [0,4194304] 0 2024-04-12T05:08:55.840 INFO:tasks.workunit.client.0.smithi113.stdout:9/407: readlink d1/df/d4b/d63/l3a 0 2024-04-12T05:08:55.840 INFO:tasks.workunit.client.0.smithi113.stdout:6/308: mknod d8/de/d13/d2b/c65 0 2024-04-12T05:08:55.841 INFO:tasks.workunit.client.0.smithi113.stdout:5/299: rename d0/d10/d27/d1d/d38/d3b/l46 to d0/d10/d27/d1d/l5b 0 2024-04-12T05:08:55.844 INFO:tasks.workunit.client.0.smithi113.stdout:3/306: creat d0/d13/d47/f61 x:0 0 0 2024-04-12T05:08:55.844 INFO:tasks.workunit.client.0.smithi113.stdout:8/383: getdents d9/d16/d1c/d59/d68 0 2024-04-12T05:08:55.845 INFO:tasks.workunit.client.0.smithi113.stdout:9/408: fsync d1/d4/d5/d22/f5a 0 2024-04-12T05:08:55.848 INFO:tasks.workunit.client.0.smithi113.stdout:0/401: dwrite d1/d43/f62 [4194304,4194304] 0 2024-04-12T05:08:55.855 INFO:tasks.workunit.client.0.smithi113.stdout:1/391: truncate d10/d21/f47 2083847 0 2024-04-12T05:08:55.855 INFO:tasks.workunit.client.0.smithi113.stdout:6/309: dread d8/d45/f49 [0,4194304] 0 2024-04-12T05:08:55.855 INFO:tasks.workunit.client.0.smithi113.stdout:7/365: dwrite d0/dd/d1b/f62 [0,4194304] 0 2024-04-12T05:08:55.855 INFO:tasks.workunit.client.0.smithi113.stdout:1/392: write d10/d2e/f54 [133171,61431] 0 2024-04-12T05:08:55.857 INFO:tasks.workunit.client.0.smithi113.stdout:4/355: rename da/dd/df to da/d73 0 2024-04-12T05:08:55.862 INFO:tasks.workunit.client.0.smithi113.stdout:9/409: mknod d1/d4/d6d/c93 0 2024-04-12T05:08:55.863 INFO:tasks.workunit.client.0.smithi113.stdout:9/410: write d1/d4/d38/f8c [446054,74557] 0 2024-04-12T05:08:55.865 INFO:tasks.workunit.client.0.smithi113.stdout:7/366: chown d0/dd/d11/d44/l65 8 1 2024-04-12T05:08:55.868 INFO:tasks.workunit.client.0.smithi113.stdout:4/356: creat da/dd/d1b/f74 x:0 0 0 2024-04-12T05:08:55.869 INFO:tasks.workunit.client.0.smithi113.stdout:6/310: mkdir d8/de/d17/d3b/d63/d66 0 2024-04-12T05:08:55.873 INFO:tasks.workunit.client.0.smithi113.stdout:9/411: symlink d1/df/d4b/d63/d3d/d6f/l94 0 2024-04-12T05:08:55.875 INFO:tasks.workunit.client.0.smithi113.stdout:1/393: symlink d10/d21/l78 0 2024-04-12T05:08:55.876 INFO:tasks.workunit.client.0.smithi113.stdout:6/311: dwrite d8/de/d13/d11/d36/d3d/d4c/d31/f54 [0,4194304] 0 2024-04-12T05:08:55.878 INFO:tasks.workunit.client.0.smithi113.stdout:8/384: getdents d9/d16/d1e 0 2024-04-12T05:08:55.879 INFO:tasks.workunit.client.0.smithi113.stdout:4/357: write da/d4a/f5c [558843,53499] 0 2024-04-12T05:08:55.880 INFO:tasks.workunit.client.0.smithi113.stdout:8/385: write d9/d16/f1a [7792683,22535] 0 2024-04-12T05:08:55.881 INFO:tasks.workunit.client.0.smithi113.stdout:3/307: link d0/d13/c53 d0/d13/d15/d28/d2a/d34/d5f/c62 0 2024-04-12T05:08:55.884 INFO:tasks.workunit.client.1.smithi192.stdout:7/197: sync 2024-04-12T05:08:55.884 INFO:tasks.workunit.client.1.smithi192.stdout:8/180: sync 2024-04-12T05:08:55.884 INFO:tasks.workunit.client.1.smithi192.stdout:5/265: sync 2024-04-12T05:08:55.884 INFO:tasks.workunit.client.0.smithi113.stdout:1/394: rmdir d10/d2e/d31 39 2024-04-12T05:08:55.885 INFO:tasks.workunit.client.0.smithi113.stdout:9/412: dwrite d1/d4/d5/d6/d15/f81 [0,4194304] 0 2024-04-12T05:08:55.886 INFO:tasks.workunit.client.1.smithi192.stdout:9/189: rename d11/d15/d21/d2c/l35 to d11/d15/d1e/l40 0 2024-04-12T05:08:55.886 INFO:tasks.workunit.client.0.smithi113.stdout:6/312: creat d8/de/d13/f67 x:0 0 0 2024-04-12T05:08:55.887 INFO:tasks.workunit.client.1.smithi192.stdout:2/273: write d1/d2d/f32 [234950,78724] 0 2024-04-12T05:08:55.888 INFO:tasks.workunit.client.0.smithi113.stdout:7/367: link d0/dc/d6/d78/c2f d0/dc/d50/c7a 0 2024-04-12T05:08:55.890 INFO:tasks.workunit.client.1.smithi192.stdout:7/198: readlink d3/d11/l35 0 2024-04-12T05:08:55.890 INFO:tasks.workunit.client.1.smithi192.stdout:4/260: dwrite d1/f3e [4194304,4194304] 0 2024-04-12T05:08:55.890 INFO:tasks.workunit.client.0.smithi113.stdout:3/308: creat d0/d13/d15/d28/d2a/d34/f63 x:0 0 0 2024-04-12T05:08:55.892 INFO:tasks.workunit.client.0.smithi113.stdout:8/386: symlink d9/d16/d1e/d26/d32/l7b 0 2024-04-12T05:08:55.895 INFO:tasks.workunit.client.0.smithi113.stdout:9/413: creat d1/d27/f95 x:0 0 0 2024-04-12T05:08:55.895 INFO:tasks.workunit.client.1.smithi192.stdout:2/274: symlink d1/d8/d17/d19/l5c 0 2024-04-12T05:08:55.898 INFO:tasks.workunit.client.1.smithi192.stdout:9/190: rename c8 to d11/d25/d36/c41 0 2024-04-12T05:08:55.899 INFO:tasks.workunit.client.1.smithi192.stdout:7/199: mknod d3/d7/d27/c4c 0 2024-04-12T05:08:55.900 INFO:tasks.workunit.client.0.smithi113.stdout:9/414: dread d1/df/f4d [0,4194304] 0 2024-04-12T05:08:55.900 INFO:tasks.workunit.client.1.smithi192.stdout:2/275: creat d1/d2d/f5d x:0 0 0 2024-04-12T05:08:55.900 INFO:tasks.workunit.client.1.smithi192.stdout:3/208: dread d5/d13/d1a/f3b [0,4194304] 0 2024-04-12T05:08:55.901 INFO:tasks.workunit.client.1.smithi192.stdout:5/266: rename d0/d21/f26 to d0/d15/d32/d45/f60 0 2024-04-12T05:08:55.902 INFO:tasks.workunit.client.0.smithi113.stdout:3/309: creat d0/d13/f64 x:0 0 0 2024-04-12T05:08:55.903 INFO:tasks.workunit.client.1.smithi192.stdout:9/191: creat d11/d15/f42 x:0 0 0 2024-04-12T05:08:55.903 INFO:tasks.workunit.client.0.smithi113.stdout:1/395: creat d10/d41/d77/f79 x:0 0 0 2024-04-12T05:08:55.904 INFO:tasks.workunit.client.0.smithi113.stdout:8/387: chown d9/d16/d1e/d26/d32/d48/l6b 4078683 1 2024-04-12T05:08:55.906 INFO:tasks.workunit.client.1.smithi192.stdout:7/200: creat d3/d7/d15/d22/f4d x:0 0 0 2024-04-12T05:08:55.907 INFO:tasks.workunit.client.1.smithi192.stdout:2/276: unlink d1/c6 0 2024-04-12T05:08:55.908 INFO:tasks.workunit.client.1.smithi192.stdout:3/209: dread d5/d13/f18 [0,4194304] 0 2024-04-12T05:08:55.908 INFO:tasks.workunit.client.0.smithi113.stdout:6/313: symlink d8/de/d17/l68 0 2024-04-12T05:08:55.909 INFO:tasks.workunit.client.0.smithi113.stdout:2/236: sync 2024-04-12T05:08:55.909 INFO:tasks.workunit.client.1.smithi192.stdout:5/267: creat d0/f61 x:0 0 0 2024-04-12T05:08:55.910 INFO:tasks.workunit.client.0.smithi113.stdout:6/314: stat d8/ld 0 2024-04-12T05:08:55.910 INFO:tasks.workunit.client.0.smithi113.stdout:2/237: dread - d8/d9/d1a/f26 zero size 2024-04-12T05:08:55.910 INFO:tasks.workunit.client.1.smithi192.stdout:3/210: fdatasync d5/d13/f42 0 2024-04-12T05:08:55.912 INFO:tasks.workunit.client.0.smithi113.stdout:2/238: fsync d8/d9/de/d15/d24/f34 0 2024-04-12T05:08:55.912 INFO:tasks.workunit.client.1.smithi192.stdout:7/201: mknod d3/d1d/d48/c4e 0 2024-04-12T05:08:55.913 INFO:tasks.workunit.client.1.smithi192.stdout:3/211: write d5/f14 [9308002,76360] 0 2024-04-12T05:08:55.913 INFO:tasks.workunit.client.1.smithi192.stdout:9/192: dwrite d11/f12 [0,4194304] 0 2024-04-12T05:08:55.914 INFO:tasks.workunit.client.0.smithi113.stdout:6/315: truncate d8/f40 582535 0 2024-04-12T05:08:55.915 INFO:tasks.workunit.client.1.smithi192.stdout:2/277: read d1/d8/d3a/de/f11 [5489108,123605] 0 2024-04-12T05:08:55.915 INFO:tasks.workunit.client.0.smithi113.stdout:6/316: stat d8/de/d13/d2b/l5a 0 2024-04-12T05:08:55.916 INFO:tasks.workunit.client.0.smithi113.stdout:3/310: creat d0/d13/d15/d28/d2a/d34/d5f/f65 x:0 0 0 2024-04-12T05:08:55.916 INFO:tasks.workunit.client.1.smithi192.stdout:7/202: creat d3/f4f x:0 0 0 2024-04-12T05:08:55.917 INFO:tasks.workunit.client.0.smithi113.stdout:1/396: dread d10/d2e/d31/f35 [0,4194304] 0 2024-04-12T05:08:55.918 INFO:tasks.workunit.client.1.smithi192.stdout:2/278: write d1/d8/d3a/de/f30 [4709557,89366] 0 2024-04-12T05:08:55.919 INFO:tasks.workunit.client.1.smithi192.stdout:9/193: creat d11/d25/d36/f43 x:0 0 0 2024-04-12T05:08:55.920 INFO:tasks.workunit.client.1.smithi192.stdout:2/279: chown d1/d8/l4e 6403435 1 2024-04-12T05:08:55.921 INFO:tasks.workunit.client.0.smithi113.stdout:3/311: symlink d0/l66 0 2024-04-12T05:08:55.922 INFO:tasks.workunit.client.1.smithi192.stdout:7/203: truncate d3/d7/d15/d22/f4d 282965 0 2024-04-12T05:08:55.924 INFO:tasks.workunit.client.0.smithi113.stdout:2/239: creat d8/f48 x:0 0 0 2024-04-12T05:08:55.924 INFO:tasks.workunit.client.0.smithi113.stdout:2/240: stat d8/d9/d1a/c2e 0 2024-04-12T05:08:55.927 INFO:tasks.workunit.client.1.smithi192.stdout:9/194: unlink d11/d15/d21/d2c/c3b 0 2024-04-12T05:08:55.929 INFO:tasks.workunit.client.1.smithi192.stdout:4/261: truncate d1/f2 4144932 0 2024-04-12T05:08:55.930 INFO:tasks.workunit.client.1.smithi192.stdout:7/204: rename d3/d7/d14/d1f/f33 to d3/d7/d15/f50 0 2024-04-12T05:08:55.931 INFO:tasks.workunit.client.1.smithi192.stdout:9/195: mknod d11/d15/c44 0 2024-04-12T05:08:55.931 INFO:tasks.workunit.client.1.smithi192.stdout:2/280: dread d1/d8/f46 [0,4194304] 0 2024-04-12T05:08:55.936 INFO:tasks.workunit.client.1.smithi192.stdout:3/212: link d5/le d5/d13/d3c/l48 0 2024-04-12T05:08:55.938 INFO:tasks.workunit.client.1.smithi192.stdout:2/281: dread d1/d8/f41 [0,4194304] 0 2024-04-12T05:08:55.938 INFO:tasks.workunit.client.1.smithi192.stdout:3/213: chown d5/c8 33011132 1 2024-04-12T05:08:55.939 INFO:tasks.workunit.client.1.smithi192.stdout:4/262: fsync d1/f2e 0 2024-04-12T05:08:55.942 INFO:tasks.workunit.client.1.smithi192.stdout:5/268: chown d0/d15/d24/d29/f3b 0 1 2024-04-12T05:08:55.943 INFO:tasks.workunit.client.1.smithi192.stdout:9/196: dwrite f10 [0,4194304] 0 2024-04-12T05:08:55.943 INFO:tasks.workunit.client.1.smithi192.stdout:3/214: creat d5/d13/d3c/d36/f49 x:0 0 0 2024-04-12T05:08:55.944 INFO:tasks.workunit.client.1.smithi192.stdout:4/263: creat d1/da/f52 x:0 0 0 2024-04-12T05:08:55.945 INFO:tasks.workunit.client.1.smithi192.stdout:2/282: unlink d1/d8/d17/d19/l23 0 2024-04-12T05:08:55.950 INFO:tasks.workunit.client.1.smithi192.stdout:5/269: symlink d0/d15/l62 0 2024-04-12T05:08:55.952 INFO:tasks.workunit.client.1.smithi192.stdout:2/283: creat d1/d8/d17/d19/f5e x:0 0 0 2024-04-12T05:08:55.953 INFO:tasks.workunit.client.1.smithi192.stdout:9/197: truncate d11/d15/d21/f24 280630 0 2024-04-12T05:08:55.954 INFO:tasks.workunit.client.1.smithi192.stdout:5/270: dread - d0/d15/d24/d47/f4d zero size 2024-04-12T05:08:55.955 INFO:tasks.workunit.client.1.smithi192.stdout:2/284: chown d1/c55 53999722 1 2024-04-12T05:08:55.956 INFO:tasks.workunit.client.1.smithi192.stdout:3/215: dwrite d5/d13/d3c/f25 [0,4194304] 0 2024-04-12T05:08:55.957 INFO:tasks.workunit.client.1.smithi192.stdout:4/264: dwrite d1/da/f41 [0,4194304] 0 2024-04-12T05:08:55.958 INFO:tasks.workunit.client.1.smithi192.stdout:3/216: dread - d5/d13/d3c/f31 zero size 2024-04-12T05:08:55.959 INFO:tasks.workunit.client.1.smithi192.stdout:9/198: creat d11/d15/f45 x:0 0 0 2024-04-12T05:08:55.961 INFO:tasks.workunit.client.1.smithi192.stdout:5/271: unlink d0/l56 0 2024-04-12T05:08:55.962 INFO:tasks.workunit.client.1.smithi192.stdout:3/217: stat d5/le 0 2024-04-12T05:08:55.967 INFO:tasks.workunit.client.1.smithi192.stdout:4/265: rename d1/da to d1/d1d/d53 0 2024-04-12T05:08:55.969 INFO:tasks.workunit.client.1.smithi192.stdout:9/199: mknod d11/d15/c46 0 2024-04-12T05:08:55.975 INFO:tasks.workunit.client.1.smithi192.stdout:9/200: fsync d11/d15/d21/f2d 0 2024-04-12T05:08:55.979 INFO:tasks.workunit.client.1.smithi192.stdout:5/272: symlink d0/l63 0 2024-04-12T05:08:55.980 INFO:tasks.workunit.client.1.smithi192.stdout:4/266: mkdir d1/d54 0 2024-04-12T05:08:55.980 INFO:tasks.workunit.client.1.smithi192.stdout:4/267: chown d1/d1d/d4c 11665 1 2024-04-12T05:08:55.982 INFO:tasks.workunit.client.1.smithi192.stdout:2/285: getdents d1 0 2024-04-12T05:08:55.982 INFO:tasks.workunit.client.1.smithi192.stdout:9/201: write fe [286206,11834] 0 2024-04-12T05:08:55.983 INFO:tasks.workunit.client.1.smithi192.stdout:4/268: write d1/f4b [3970365,119146] 0 2024-04-12T05:08:55.986 INFO:tasks.workunit.client.1.smithi192.stdout:2/286: creat d1/d8/d17/f5f x:0 0 0 2024-04-12T05:08:55.988 INFO:tasks.workunit.client.1.smithi192.stdout:4/269: write d1/d20/f30 [1393144,34563] 0 2024-04-12T05:08:55.988 INFO:tasks.workunit.client.1.smithi192.stdout:9/202: creat d11/f47 x:0 0 0 2024-04-12T05:08:55.991 INFO:tasks.workunit.client.1.smithi192.stdout:5/273: dread d0/d15/d32/d45/f60 [0,4194304] 0 2024-04-12T05:08:55.992 INFO:tasks.workunit.client.1.smithi192.stdout:2/287: rename d1/d8/d17/d3c to d1/d8/d17/d19/d60 0 2024-04-12T05:08:55.994 INFO:tasks.workunit.client.1.smithi192.stdout:4/270: rename d1/d20 to d1/d54/d55 0 2024-04-12T05:08:55.997 INFO:tasks.workunit.client.1.smithi192.stdout:9/203: link d11/d15/d21/d2c/f1a d11/d25/d36/d3d/f48 0 2024-04-12T05:08:55.998 INFO:tasks.workunit.client.1.smithi192.stdout:4/271: symlink d1/d1d/d4c/l56 0 2024-04-12T05:08:55.999 INFO:tasks.workunit.client.1.smithi192.stdout:2/288: link d1/d8/d17/d19/d40/c4f d1/d8/d17/d19/c61 0 2024-04-12T05:08:56.001 INFO:tasks.workunit.client.1.smithi192.stdout:4/272: write d1/d54/d55/d37/f48 [470820,18479] 0 2024-04-12T05:08:56.001 INFO:tasks.workunit.client.0.smithi113.stdout:1/397: getdents d10/d21/d23 0 2024-04-12T05:08:56.004 INFO:tasks.workunit.client.0.smithi113.stdout:3/312: rename d0/d13/d15/d28/d2a/d34/c5d to d0/d13/d15/c67 0 2024-04-12T05:08:56.004 INFO:tasks.workunit.client.1.smithi192.stdout:9/204: dwrite d11/d25/d36/f43 [0,4194304] 0 2024-04-12T05:08:56.005 INFO:tasks.workunit.client.1.smithi192.stdout:4/273: rmdir d1/d1d/d4c 39 2024-04-12T05:08:56.006 INFO:tasks.workunit.client.0.smithi113.stdout:9/415: dread d1/d27/f3c [0,4194304] 0 2024-04-12T05:08:56.007 INFO:tasks.workunit.client.0.smithi113.stdout:4/358: dread da/d73/f45 [0,4194304] 0 2024-04-12T05:08:56.008 INFO:tasks.workunit.client.1.smithi192.stdout:2/289: rename d1/c1c to d1/d8/d17/d19/d60/c62 0 2024-04-12T05:08:56.010 INFO:tasks.workunit.client.0.smithi113.stdout:9/416: dread - d1/d4/f8f zero size 2024-04-12T05:08:56.010 INFO:tasks.workunit.client.1.smithi192.stdout:9/205: fdatasync d11/d37/f3e 0 2024-04-12T05:08:56.011 INFO:tasks.workunit.client.1.smithi192.stdout:9/206: write f10 [938582,120713] 0 2024-04-12T05:08:56.012 INFO:tasks.workunit.client.0.smithi113.stdout:1/398: rename d10/f32 to d10/d2e/d31/d33/f7a 0 2024-04-12T05:08:56.012 INFO:tasks.workunit.client.0.smithi113.stdout:2/241: dwrite d8/d16/f46 [0,4194304] 0 2024-04-12T05:08:56.012 INFO:tasks.workunit.client.1.smithi192.stdout:4/274: creat d1/d1d/d53/f57 x:0 0 0 2024-04-12T05:08:56.015 INFO:tasks.workunit.client.0.smithi113.stdout:8/388: dwrite d9/d16/d1e/d26/d32/f3f [0,4194304] 0 2024-04-12T05:08:56.017 INFO:tasks.workunit.client.1.smithi192.stdout:5/274: dwrite d0/d21/f33 [0,4194304] 0 2024-04-12T05:08:56.020 INFO:tasks.workunit.client.0.smithi113.stdout:9/417: fdatasync d1/d4/d38/f62 0 2024-04-12T05:08:56.022 INFO:tasks.workunit.client.1.smithi192.stdout:4/275: symlink d1/d54/d55/l58 0 2024-04-12T05:08:56.023 INFO:tasks.workunit.client.1.smithi192.stdout:9/207: rename d11/l2a to d11/d15/d1e/l49 0 2024-04-12T05:08:56.025 INFO:tasks.workunit.client.1.smithi192.stdout:4/276: rmdir d1/d49 39 2024-04-12T05:08:56.026 INFO:tasks.workunit.client.0.smithi113.stdout:1/399: truncate d10/f1e 1182599 0 2024-04-12T05:08:56.027 INFO:tasks.workunit.client.1.smithi192.stdout:9/208: mkdir d11/d15/d21/d2c/d1f/d4a 0 2024-04-12T05:08:56.027 INFO:tasks.workunit.client.0.smithi113.stdout:1/400: readlink d10/d21/l25 0 2024-04-12T05:08:56.028 INFO:tasks.workunit.client.0.smithi113.stdout:9/418: rename d1/d4/d38/d65/f91 to d1/d4/d5/d17/f96 0 2024-04-12T05:08:56.029 INFO:tasks.workunit.client.0.smithi113.stdout:9/419: stat d1/d27/d7a 0 2024-04-12T05:08:56.030 INFO:tasks.workunit.client.0.smithi113.stdout:7/368: dread d0/f12 [0,4194304] 0 2024-04-12T05:08:56.033 INFO:tasks.workunit.client.0.smithi113.stdout:3/313: rename d0/d13/d15/f3e to d0/d13/d15/f68 0 2024-04-12T05:08:56.034 INFO:tasks.workunit.client.1.smithi192.stdout:5/275: getdents d0 0 2024-04-12T05:08:56.034 INFO:tasks.workunit.client.0.smithi113.stdout:8/389: dread d9/d1b/f4f [0,4194304] 0 2024-04-12T05:08:56.035 INFO:tasks.workunit.client.0.smithi113.stdout:2/242: link d8/d9/de/d15/d24/f3b d8/d9/d1a/f49 0 2024-04-12T05:08:56.035 INFO:tasks.workunit.client.1.smithi192.stdout:5/276: stat d0/d15/d24/f4b 0 2024-04-12T05:08:56.036 INFO:tasks.workunit.client.0.smithi113.stdout:9/420: rmdir d1/d27 39 2024-04-12T05:08:56.040 INFO:tasks.workunit.client.0.smithi113.stdout:8/390: write d9/f47 [964045,53445] 0 2024-04-12T05:08:56.040 INFO:tasks.workunit.client.1.smithi192.stdout:9/209: getdents d11/d37/d33 0 2024-04-12T05:08:56.040 INFO:tasks.workunit.client.0.smithi113.stdout:9/421: symlink d1/df/l97 0 2024-04-12T05:08:56.041 INFO:tasks.workunit.client.1.smithi192.stdout:4/277: dread d1/f2e [0,4194304] 0 2024-04-12T05:08:56.042 INFO:tasks.workunit.client.0.smithi113.stdout:1/401: dwrite fd [4194304,4194304] 0 2024-04-12T05:08:56.042 INFO:tasks.workunit.client.1.smithi192.stdout:5/277: mkdir d0/d15/d64 0 2024-04-12T05:08:56.043 INFO:tasks.workunit.client.1.smithi192.stdout:2/290: truncate d1/d8/d3a/de/f30 4945854 0 2024-04-12T05:08:56.043 INFO:tasks.workunit.client.1.smithi192.stdout:9/210: mknod d11/d37/d33/c4b 0 2024-04-12T05:08:56.044 INFO:tasks.workunit.client.0.smithi113.stdout:9/422: truncate d1/df/d4b/f7f 45765 0 2024-04-12T05:08:56.045 INFO:tasks.workunit.client.1.smithi192.stdout:2/291: chown d1/d8/d3a/de/l42 8 1 2024-04-12T05:08:56.045 INFO:tasks.workunit.client.0.smithi113.stdout:2/243: rename d8/d9/de/l1b to d8/d9/d27/d3f/l4a 0 2024-04-12T05:08:56.046 INFO:tasks.workunit.client.1.smithi192.stdout:5/278: creat d0/d15/d32/d45/f65 x:0 0 0 2024-04-12T05:08:56.046 INFO:tasks.workunit.client.0.smithi113.stdout:8/391: dread d9/f47 [0,4194304] 0 2024-04-12T05:08:56.061 INFO:tasks.workunit.client.0.smithi113.stdout:1/402: symlink d10/d2e/d31/d5c/l7b 0 2024-04-12T05:08:56.061 INFO:tasks.workunit.client.0.smithi113.stdout:5/300: sync 2024-04-12T05:08:56.061 INFO:tasks.workunit.client.0.smithi113.stdout:9/423: truncate d1/d4/d5/d6/f48 431631 0 2024-04-12T05:08:56.062 INFO:tasks.workunit.client.1.smithi192.stdout:9/211: rename d11/d25/d36/d3d/f48 to d11/d37/d33/f4c 0 2024-04-12T05:08:56.062 INFO:tasks.workunit.client.1.smithi192.stdout:9/212: fsync d11/f47 0 2024-04-12T05:08:56.062 INFO:tasks.workunit.client.0.smithi113.stdout:8/392: stat d9/d16/d1e/d3a/c70 0 2024-04-12T05:08:56.062 INFO:tasks.workunit.client.0.smithi113.stdout:2/244: fdatasync d8/d9/d1a/f49 0 2024-04-12T05:08:56.064 INFO:tasks.workunit.client.1.smithi192.stdout:4/278: getdents d1/d1d/d4c 0 2024-04-12T05:08:56.065 INFO:tasks.workunit.client.0.smithi113.stdout:8/393: fsync d9/d16/d1e/d26/d32/d5b/f63 0 2024-04-12T05:08:56.065 INFO:tasks.workunit.client.0.smithi113.stdout:4/359: truncate da/d73/f71 2160347 0 2024-04-12T05:08:56.066 INFO:tasks.workunit.client.0.smithi113.stdout:8/394: readlink d9/d34/l62 0 2024-04-12T05:08:56.066 INFO:tasks.workunit.client.0.smithi113.stdout:2/245: truncate d8/d9/de/d12/f41 781387 0 2024-04-12T05:08:56.067 INFO:tasks.workunit.client.0.smithi113.stdout:2/246: fdatasync d8/d16/f46 0 2024-04-12T05:08:56.067 INFO:tasks.workunit.client.0.smithi113.stdout:8/395: chown d9/d16/d1e/d26/d32 29426 1 2024-04-12T05:08:56.068 INFO:tasks.workunit.client.0.smithi113.stdout:3/314: rename d0/d13/d15/d28/d2a/d34/d5f/c62 to d0/d13/d15/d28/d2a/c69 0 2024-04-12T05:08:56.068 INFO:tasks.workunit.client.0.smithi113.stdout:8/396: write d9/d16/f1a [7591343,65230] 0 2024-04-12T05:08:56.069 INFO:tasks.workunit.client.0.smithi113.stdout:3/315: dread - d0/d13/d15/f5c zero size 2024-04-12T05:08:56.071 INFO:tasks.workunit.client.0.smithi113.stdout:3/316: readlink d0/d13/d15/d28/l58 0 2024-04-12T05:08:56.071 INFO:tasks.workunit.client.0.smithi113.stdout:1/403: dread d10/d14/f4d [0,4194304] 0 2024-04-12T05:08:56.071 INFO:tasks.workunit.client.0.smithi113.stdout:3/317: fdatasync d0/d13/d15/f39 0 2024-04-12T05:08:56.072 INFO:tasks.workunit.client.0.smithi113.stdout:9/424: link d1/f79 d1/d4/f98 0 2024-04-12T05:08:56.073 INFO:tasks.workunit.client.0.smithi113.stdout:9/425: fdatasync d1/d4/d5/f12 0 2024-04-12T05:08:56.073 INFO:tasks.workunit.client.0.smithi113.stdout:3/318: truncate d0/d13/d15/f68 4916653 0 2024-04-12T05:08:56.074 INFO:tasks.workunit.client.0.smithi113.stdout:4/360: creat da/d3c/d43/f75 x:0 0 0 2024-04-12T05:08:56.075 INFO:tasks.workunit.client.0.smithi113.stdout:7/369: write d0/dd/f66 [645073,14239] 0 2024-04-12T05:08:56.076 INFO:tasks.workunit.client.0.smithi113.stdout:5/301: mknod d0/d10/d27/d1d/d38/d44/d4e/c5c 0 2024-04-12T05:08:56.078 INFO:tasks.workunit.client.0.smithi113.stdout:8/397: mknod d9/d16/d1e/c7c 0 2024-04-12T05:08:56.081 INFO:tasks.workunit.client.0.smithi113.stdout:3/319: dwrite d0/d13/d15/f5c [0,4194304] 0 2024-04-12T05:08:56.083 INFO:tasks.workunit.client.0.smithi113.stdout:9/426: mknod d1/d4/d5/d6/d15/c99 0 2024-04-12T05:08:56.089 INFO:tasks.workunit.client.0.smithi113.stdout:5/302: mknod d0/d10/d27/d1d/d38/d44/d4e/c5d 0 2024-04-12T05:08:56.090 INFO:tasks.workunit.client.0.smithi113.stdout:8/398: mknod d9/d34/c7d 0 2024-04-12T05:08:56.091 INFO:tasks.workunit.client.0.smithi113.stdout:1/404: mkdir d10/d2e/d7c 0 2024-04-12T05:08:56.092 INFO:tasks.workunit.client.0.smithi113.stdout:3/320: creat d0/d13/f6a x:0 0 0 2024-04-12T05:08:56.093 INFO:tasks.workunit.client.0.smithi113.stdout:4/361: getdents da/d73/d14 0 2024-04-12T05:08:56.095 INFO:tasks.workunit.client.0.smithi113.stdout:9/427: unlink d1/df/d4b/d63/d3d/f8e 0 2024-04-12T05:08:56.095 INFO:tasks.workunit.client.0.smithi113.stdout:4/362: write da/d2b/f3e [4047773,114251] 0 2024-04-12T05:08:56.098 INFO:tasks.workunit.client.0.smithi113.stdout:5/303: rename d0/d10/d27/f26 to d0/d10/d27/d1d/d38/d3b/f5e 0 2024-04-12T05:08:56.098 INFO:tasks.workunit.client.0.smithi113.stdout:8/399: fdatasync d9/f47 0 2024-04-12T05:08:56.099 INFO:tasks.workunit.client.0.smithi113.stdout:9/428: dread - d1/df/d4b/d63/d3d/f5f zero size 2024-04-12T05:08:56.100 INFO:tasks.workunit.client.0.smithi113.stdout:1/405: dwrite d10/d2e/d31/d5c/f74 [0,4194304] 0 2024-04-12T05:08:56.101 INFO:tasks.workunit.client.0.smithi113.stdout:9/429: write d1/df/d4b/f7f [482591,50204] 0 2024-04-12T05:08:56.102 INFO:tasks.workunit.client.0.smithi113.stdout:2/247: truncate d8/d9/de/f22 137696 0 2024-04-12T05:08:56.105 INFO:tasks.workunit.client.0.smithi113.stdout:4/363: chown da/de/c56 158 1 2024-04-12T05:08:56.107 INFO:tasks.workunit.client.0.smithi113.stdout:3/321: rename d0/d13/d47/d4f/f4e to d0/d13/f6b 0 2024-04-12T05:08:56.107 INFO:tasks.workunit.client.0.smithi113.stdout:8/400: dwrite d9/d34/f57 [0,4194304] 0 2024-04-12T05:08:56.108 INFO:tasks.workunit.client.0.smithi113.stdout:5/304: mkdir d0/d10/d27/d5f 0 2024-04-12T05:08:56.109 INFO:tasks.workunit.client.0.smithi113.stdout:5/305: readlink d0/d10/l30 0 2024-04-12T05:08:56.111 INFO:tasks.workunit.client.0.smithi113.stdout:5/306: stat d0/d10/d27/d1d/d38/d44/d4e/c5c 0 2024-04-12T05:08:56.113 INFO:tasks.workunit.client.0.smithi113.stdout:7/370: getdents d0/dc/d6/d78 0 2024-04-12T05:08:56.113 INFO:tasks.workunit.client.0.smithi113.stdout:5/307: chown d0/d10/d27/d1d/d38/d44 0 1 2024-04-12T05:08:56.114 INFO:tasks.workunit.client.0.smithi113.stdout:2/248: creat d8/d9/de/d15/d24/f4b x:0 0 0 2024-04-12T05:08:56.116 INFO:tasks.workunit.client.0.smithi113.stdout:3/322: dwrite d0/d13/d15/d28/f38 [0,4194304] 0 2024-04-12T05:08:56.118 INFO:tasks.workunit.client.0.smithi113.stdout:3/323: fsync d0/f9 0 2024-04-12T05:08:56.119 INFO:tasks.workunit.client.0.smithi113.stdout:4/364: mknod da/dd/d1b/c76 0 2024-04-12T05:08:56.119 INFO:tasks.workunit.client.0.smithi113.stdout:8/401: mkdir d9/d34/d7e 0 2024-04-12T05:08:56.141 INFO:tasks.workunit.client.1.smithi192.stdout:0/165: sync 2024-04-12T05:08:56.141 INFO:tasks.workunit.client.0.smithi113.stdout:5/308: creat d0/d10/d27/d1d/f60 x:0 0 0 2024-04-12T05:08:56.143 INFO:tasks.workunit.client.0.smithi113.stdout:2/249: creat d8/d9/de/d12/f4c x:0 0 0 2024-04-12T05:08:56.144 INFO:tasks.workunit.client.0.smithi113.stdout:2/250: fdatasync d8/d16/f17 0 2024-04-12T05:08:56.145 INFO:tasks.workunit.client.1.smithi192.stdout:4/279: fsync d1/d1d/d53/d1c/f36 0 2024-04-12T05:08:56.147 INFO:tasks.workunit.client.0.smithi113.stdout:8/402: dread d9/f53 [0,4194304] 0 2024-04-12T05:08:56.148 INFO:tasks.workunit.client.0.smithi113.stdout:4/365: mknod da/d3c/d43/c77 0 2024-04-12T05:08:56.148 INFO:tasks.workunit.client.0.smithi113.stdout:8/403: fsync d9/d16/d1c/f51 0 2024-04-12T05:08:56.148 INFO:tasks.workunit.client.1.smithi192.stdout:9/213: dwrite d11/d25/d36/f43 [4194304,4194304] 0 2024-04-12T05:08:56.148 INFO:tasks.workunit.client.1.smithi192.stdout:4/280: read d1/f18 [1536389,94602] 0 2024-04-12T05:08:56.149 INFO:tasks.workunit.client.0.smithi113.stdout:4/366: fdatasync da/d3c/d43/f4d 0 2024-04-12T05:08:56.149 INFO:tasks.workunit.client.1.smithi192.stdout:9/214: stat d11/d15/d21/d2c/d1f/d4a 0 2024-04-12T05:08:56.150 INFO:tasks.workunit.client.0.smithi113.stdout:4/367: dread - da/d3c/d43/f5e zero size 2024-04-12T05:08:56.150 INFO:tasks.workunit.client.1.smithi192.stdout:9/215: dread - d11/d15/f45 zero size 2024-04-12T05:08:56.150 INFO:tasks.workunit.client.0.smithi113.stdout:3/324: rename d0/d13/d15/f68 to d0/d13/d47/f6c 0 2024-04-12T05:08:56.152 INFO:tasks.workunit.client.1.smithi192.stdout:9/216: truncate d11/d37/f3e 601440 0 2024-04-12T05:08:56.152 INFO:tasks.workunit.client.0.smithi113.stdout:5/309: fdatasync d0/d10/d27/f18 0 2024-04-12T05:08:56.153 INFO:tasks.workunit.client.1.smithi192.stdout:4/281: mknod d1/d54/c59 0 2024-04-12T05:08:56.154 INFO:tasks.workunit.client.1.smithi192.stdout:5/279: dread d0/f13 [0,4194304] 0 2024-04-12T05:08:56.154 INFO:tasks.workunit.client.1.smithi192.stdout:4/282: fsync d1/d1d/d53/d1c/f51 0 2024-04-12T05:08:56.155 INFO:tasks.workunit.client.1.smithi192.stdout:0/166: write d0/d6/d15/d1b/f2d [1619557,100531] 0 2024-04-12T05:08:56.156 INFO:tasks.workunit.client.0.smithi113.stdout:1/406: truncate fd 3809103 0 2024-04-12T05:08:56.158 INFO:tasks.workunit.client.1.smithi192.stdout:5/280: write d0/f13 [1908451,84638] 0 2024-04-12T05:08:56.162 INFO:tasks.workunit.client.0.smithi113.stdout:5/310: creat d0/d10/d27/d1d/d38/d3b/f61 x:0 0 0 2024-04-12T05:08:56.163 INFO:tasks.workunit.client.1.smithi192.stdout:0/167: mknod d0/d6/d15/d34/c3b 0 2024-04-12T05:08:56.163 INFO:tasks.workunit.client.1.smithi192.stdout:4/283: creat d1/d1d/d4c/f5a x:0 0 0 2024-04-12T05:08:56.164 INFO:tasks.workunit.client.1.smithi192.stdout:5/281: mknod d0/d15/d24/d47/c66 0 2024-04-12T05:08:56.164 INFO:tasks.workunit.client.0.smithi113.stdout:7/371: getdents d0/dd/d1b 0 2024-04-12T05:08:56.165 INFO:tasks.workunit.client.0.smithi113.stdout:5/311: dread - d0/f55 zero size 2024-04-12T05:08:56.166 INFO:tasks.workunit.client.0.smithi113.stdout:4/368: link da/d73/d1d/c33 da/d24/c78 0 2024-04-12T05:08:56.166 INFO:tasks.workunit.client.0.smithi113.stdout:7/372: dread - d0/d52/f54 zero size 2024-04-12T05:08:56.167 INFO:tasks.workunit.client.1.smithi192.stdout:0/168: creat d0/d6/d15/f3c x:0 0 0 2024-04-12T05:08:56.168 INFO:tasks.workunit.client.1.smithi192.stdout:4/284: creat d1/d49/f5b x:0 0 0 2024-04-12T05:08:56.168 INFO:tasks.workunit.client.0.smithi113.stdout:8/404: getdents d9/d16/d1c 0 2024-04-12T05:08:56.169 INFO:tasks.workunit.client.1.smithi192.stdout:4/285: readlink d1/d1d/d4c/l56 0 2024-04-12T05:08:56.169 INFO:tasks.workunit.client.1.smithi192.stdout:5/282: symlink d0/d15/l67 0 2024-04-12T05:08:56.170 INFO:tasks.workunit.client.0.smithi113.stdout:4/369: mkdir da/d73/d14/d79 0 2024-04-12T05:08:56.170 INFO:tasks.workunit.client.1.smithi192.stdout:4/286: write d1/d1d/d53/d1c/f4e [1362620,44119] 0 2024-04-12T05:08:56.171 INFO:tasks.workunit.client.0.smithi113.stdout:8/405: chown d9/d34/d7e 12518618 1 2024-04-12T05:08:56.173 INFO:tasks.workunit.client.0.smithi113.stdout:1/407: dwrite d10/d2e/d31/d33/f64 [0,4194304] 0 2024-04-12T05:08:56.173 INFO:tasks.workunit.client.0.smithi113.stdout:4/370: chown da/dd/d1b/c58 29953 1 2024-04-12T05:08:56.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:54.670030+0000 mgr.smithi192.ocppyo (mgr.24357) 23 : cluster [DBG] pgmap v17: 65 pgs: 65 active+clean; 668 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 3.6 MiB/s rd, 58 MiB/s wr, 89 op/s 2024-04-12T05:08:56.174 INFO:tasks.workunit.client.1.smithi192.stdout:0/169: link d0/d6/d15/f37 d0/d6/d15/f3d 0 2024-04-12T05:08:56.176 INFO:tasks.workunit.client.1.smithi192.stdout:5/283: unlink d0/d15/d24/l27 0 2024-04-12T05:08:56.176 INFO:tasks.workunit.client.0.smithi113.stdout:2/251: write d8/d9/de/f31 [4980182,92075] 0 2024-04-12T05:08:56.177 INFO:tasks.workunit.client.1.smithi192.stdout:0/170: stat d0/d6/f21 0 2024-04-12T05:08:56.178 INFO:tasks.workunit.client.1.smithi192.stdout:5/284: readlink d0/d15/d24/l3c 0 2024-04-12T05:08:56.183 INFO:tasks.workunit.client.0.smithi113.stdout:1/408: unlink d10/d21/l78 0 2024-04-12T05:08:56.183 INFO:tasks.workunit.client.1.smithi192.stdout:9/217: dwrite d11/d15/d21/f24 [0,4194304] 0 2024-04-12T05:08:56.184 INFO:tasks.workunit.client.1.smithi192.stdout:4/287: mkdir d1/d1d/d53/d1c/d5c 0 2024-04-12T05:08:56.184 INFO:tasks.workunit.client.1.smithi192.stdout:0/171: read d0/d6/f13 [638622,20788] 0 2024-04-12T05:08:56.184 INFO:tasks.workunit.client.1.smithi192.stdout:9/218: stat f7 0 2024-04-12T05:08:56.184 INFO:tasks.workunit.client.0.smithi113.stdout:1/409: fdatasync d10/d41/f65 0 2024-04-12T05:08:56.186 INFO:tasks.workunit.client.1.smithi192.stdout:4/288: dread d1/d54/d55/d37/f48 [0,4194304] 0 2024-04-12T05:08:56.186 INFO:tasks.workunit.client.1.smithi192.stdout:0/172: mknod d0/d6/d15/d1b/c3e 0 2024-04-12T05:08:56.189 INFO:tasks.workunit.client.1.smithi192.stdout:5/285: rename d0/d15/l17 to d0/d15/d24/d47/l68 0 2024-04-12T05:08:56.189 INFO:tasks.workunit.client.0.smithi113.stdout:5/312: dread d0/d10/d27/d1d/d38/d3b/f5e [0,4194304] 0 2024-04-12T05:08:56.190 INFO:tasks.workunit.client.0.smithi113.stdout:8/406: rename d9/d16/d1e to d9/d16/d7f 0 2024-04-12T05:08:56.190 INFO:tasks.workunit.client.1.smithi192.stdout:9/219: dwrite f3 [0,4194304] 0 2024-04-12T05:08:56.191 INFO:tasks.workunit.client.1.smithi192.stdout:5/286: write d0/fa [747116,115357] 0 2024-04-12T05:08:56.192 INFO:tasks.workunit.client.0.smithi113.stdout:2/252: symlink d8/d9/de/d15/l4d 0 2024-04-12T05:08:56.193 INFO:tasks.workunit.client.1.smithi192.stdout:4/289: truncate d1/f29 1166029 0 2024-04-12T05:08:56.196 INFO:tasks.workunit.client.0.smithi113.stdout:3/325: truncate d0/d13/d15/d28/f38 3713972 0 2024-04-12T05:08:56.197 INFO:tasks.workunit.client.0.smithi113.stdout:5/313: creat d0/d10/d27/d13/f62 x:0 0 0 2024-04-12T05:08:56.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:54.670030+0000 mgr.smithi192.ocppyo (mgr.24357) 23 : cluster [DBG] pgmap v17: 65 pgs: 65 active+clean; 668 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 3.6 MiB/s rd, 58 MiB/s wr, 89 op/s 2024-04-12T05:08:56.198 INFO:tasks.workunit.client.0.smithi113.stdout:4/371: symlink da/d73/d1d/d47/l7a 0 2024-04-12T05:08:56.199 INFO:tasks.workunit.client.1.smithi192.stdout:0/173: dread d0/d6/d15/d1b/f2d [0,4194304] 0 2024-04-12T05:08:56.200 INFO:tasks.workunit.client.1.smithi192.stdout:9/220: rmdir d11/d15/d21/d2c/d1f/d4a 0 2024-04-12T05:08:56.200 INFO:tasks.workunit.client.0.smithi113.stdout:4/372: dread - da/d24/f69 zero size 2024-04-12T05:08:56.200 INFO:tasks.workunit.client.0.smithi113.stdout:8/407: unlink d9/d16/d1c/d59/d68/l6d 0 2024-04-12T05:08:56.201 INFO:tasks.workunit.client.0.smithi113.stdout:2/253: dwrite d8/fd [0,4194304] 0 2024-04-12T05:08:56.201 INFO:tasks.workunit.client.0.smithi113.stdout:4/373: dread - da/d24/f52 zero size 2024-04-12T05:08:56.203 INFO:tasks.workunit.client.0.smithi113.stdout:3/326: symlink d0/d13/d47/l6d 0 2024-04-12T05:08:56.204 INFO:tasks.workunit.client.0.smithi113.stdout:8/408: write d9/d16/d1c/f51 [4802482,118883] 0 2024-04-12T05:08:56.206 INFO:tasks.workunit.client.1.smithi192.stdout:0/174: rename d0/d6/d15/d1b/c3e to d0/d6/d22/d33/c3f 0 2024-04-12T05:08:56.207 INFO:tasks.workunit.client.0.smithi113.stdout:3/327: readlink d0/d13/d15/l54 0 2024-04-12T05:08:56.213 INFO:tasks.workunit.client.0.smithi113.stdout:4/374: write da/d3c/d43/f5e [345521,38670] 0 2024-04-12T05:08:56.215 INFO:tasks.workunit.client.1.smithi192.stdout:5/287: truncate d0/d21/f33 1766243 0 2024-04-12T05:08:56.215 INFO:tasks.workunit.client.0.smithi113.stdout:1/410: truncate d10/d2e/d31/d33/f50 2871962 0 2024-04-12T05:08:56.216 INFO:tasks.workunit.client.1.smithi192.stdout:5/288: write d0/f1b [1354715,10883] 0 2024-04-12T05:08:56.220 INFO:tasks.workunit.client.0.smithi113.stdout:3/328: dwrite d0/f12 [4194304,4194304] 0 2024-04-12T05:08:56.221 INFO:tasks.workunit.client.1.smithi192.stdout:5/289: rename d0/d15/d32/d5a to d0/d15/d32/d69 0 2024-04-12T05:08:56.221 INFO:tasks.workunit.client.1.smithi192.stdout:0/175: link d0/d6/c7 d0/d6/d15/c40 0 2024-04-12T05:08:56.224 INFO:tasks.workunit.client.1.smithi192.stdout:4/290: dread d1/f18 [0,4194304] 0 2024-04-12T05:08:56.224 INFO:tasks.workunit.client.1.smithi192.stdout:0/176: mknod d0/c41 0 2024-04-12T05:08:56.224 INFO:tasks.workunit.client.0.smithi113.stdout:1/411: rename d10/d41/d77 to d10/d21/d23/d7d 0 2024-04-12T05:08:56.227 INFO:tasks.workunit.client.0.smithi113.stdout:7/373: dwrite d0/dc/d6/d1a/f29 [0,4194304] 0 2024-04-12T05:08:56.228 INFO:tasks.workunit.client.0.smithi113.stdout:2/254: dwrite d8/d9/de/f31 [0,4194304] 0 2024-04-12T05:08:56.229 INFO:tasks.workunit.client.0.smithi113.stdout:1/412: unlink d10/l11 0 2024-04-12T05:08:56.230 INFO:tasks.workunit.client.0.smithi113.stdout:2/255: write d8/d9/f30 [2784008,90785] 0 2024-04-12T05:08:56.233 INFO:tasks.workunit.client.1.smithi192.stdout:4/291: dwrite d1/f33 [0,4194304] 0 2024-04-12T05:08:56.234 INFO:tasks.workunit.client.0.smithi113.stdout:1/413: write d10/d14/d67/f69 [430482,99022] 0 2024-04-12T05:08:56.234 INFO:tasks.workunit.client.0.smithi113.stdout:7/374: chown d0/dc/d6/d78/l3f 99510321 1 2024-04-12T05:08:56.234 INFO:tasks.workunit.client.0.smithi113.stdout:1/414: stat f5 0 2024-04-12T05:08:56.235 INFO:tasks.workunit.client.1.smithi192.stdout:0/177: dwrite d0/d6/d15/d1b/f2d [0,4194304] 0 2024-04-12T05:08:56.236 INFO:tasks.workunit.client.0.smithi113.stdout:3/329: getdents d0/d13/d47 0 2024-04-12T05:08:56.239 INFO:tasks.workunit.client.1.smithi192.stdout:0/178: dread - d0/d6/d15/d34/f38 zero size 2024-04-12T05:08:56.239 INFO:tasks.workunit.client.0.smithi113.stdout:7/375: symlink d0/dd/l7b 0 2024-04-12T05:08:56.239 INFO:tasks.workunit.client.1.smithi192.stdout:4/292: creat d1/d54/d55/d37/f5d x:0 0 0 2024-04-12T05:08:56.240 INFO:tasks.workunit.client.1.smithi192.stdout:0/179: read - d0/d6/d15/f37 zero size 2024-04-12T05:08:56.241 INFO:tasks.workunit.client.1.smithi192.stdout:0/180: read - d0/d6/d15/f3d zero size 2024-04-12T05:08:56.242 INFO:tasks.workunit.client.1.smithi192.stdout:4/293: rmdir d1/d54 39 2024-04-12T05:08:56.245 INFO:tasks.workunit.client.1.smithi192.stdout:0/181: dread d0/fb [0,4194304] 0 2024-04-12T05:08:56.246 INFO:tasks.workunit.client.1.smithi192.stdout:4/294: write d1/d54/d55/f30 [670517,61179] 0 2024-04-12T05:08:56.247 INFO:tasks.workunit.client.1.smithi192.stdout:4/295: fdatasync d1/d1d/f44 0 2024-04-12T05:08:56.248 INFO:tasks.workunit.client.1.smithi192.stdout:5/290: dread d0/fa [0,4194304] 0 2024-04-12T05:08:56.248 INFO:tasks.workunit.client.1.smithi192.stdout:4/296: chown d1/d1d/d53/f31 143514815 1 2024-04-12T05:08:56.254 INFO:tasks.workunit.client.1.smithi192.stdout:0/182: rename d0/d6/d15/d1b/f2d to d0/d26/d2e/f42 0 2024-04-12T05:08:56.255 INFO:tasks.workunit.client.0.smithi113.stdout:2/256: dwrite d8/d9/d27/f28 [0,4194304] 0 2024-04-12T05:08:56.255 INFO:tasks.workunit.client.0.smithi113.stdout:3/330: creat d0/d13/d15/d28/f6e x:0 0 0 2024-04-12T05:08:56.255 INFO:tasks.workunit.client.0.smithi113.stdout:0/402: sync 2024-04-12T05:08:56.255 INFO:tasks.workunit.client.0.smithi113.stdout:6/317: sync 2024-04-12T05:08:56.257 INFO:tasks.workunit.client.0.smithi113.stdout:7/376: dread d0/f12 [0,4194304] 0 2024-04-12T05:08:56.258 INFO:tasks.workunit.client.1.smithi192.stdout:4/297: symlink d1/l5e 0 2024-04-12T05:08:56.258 INFO:tasks.workunit.client.0.smithi113.stdout:2/257: fsync d8/d9/de/d15/f29 0 2024-04-12T05:08:56.259 INFO:tasks.workunit.client.0.smithi113.stdout:7/377: chown d0/c4f 1 1 2024-04-12T05:08:56.259 INFO:tasks.workunit.client.1.smithi192.stdout:0/183: symlink d0/d26/d29/l43 0 2024-04-12T05:08:56.260 INFO:tasks.workunit.client.0.smithi113.stdout:1/415: dwrite d10/d21/d23/d7d/f79 [0,4194304] 0 2024-04-12T05:08:56.260 INFO:tasks.workunit.client.1.smithi192.stdout:0/184: dread - d0/d6/d15/f3d zero size 2024-04-12T05:08:56.261 INFO:tasks.workunit.client.1.smithi192.stdout:4/298: fdatasync d1/f18 0 2024-04-12T05:08:56.262 INFO:tasks.workunit.client.1.smithi192.stdout:4/299: dread - d1/d1d/d53/f52 zero size 2024-04-12T05:08:56.262 INFO:tasks.workunit.client.0.smithi113.stdout:3/331: creat d0/d13/d15/d28/f6f x:0 0 0 2024-04-12T05:08:56.263 INFO:tasks.workunit.client.0.smithi113.stdout:2/258: write d8/d9/d27/f28 [2712300,6430] 0 2024-04-12T05:08:56.265 INFO:tasks.workunit.client.0.smithi113.stdout:0/403: symlink d1/d6/d34/l79 0 2024-04-12T05:08:56.266 INFO:tasks.workunit.client.1.smithi192.stdout:4/300: chown d1/d1d/d53/d1c/f4e 49 1 2024-04-12T05:08:56.266 INFO:tasks.workunit.client.0.smithi113.stdout:7/378: creat d0/dc/d6/d78/f7c x:0 0 0 2024-04-12T05:08:56.266 INFO:tasks.workunit.client.0.smithi113.stdout:4/375: dread da/d4f/f5d [0,4194304] 0 2024-04-12T05:08:56.269 INFO:tasks.workunit.client.1.smithi192.stdout:0/185: dwrite d0/d6/d22/f24 [0,4194304] 0 2024-04-12T05:08:56.270 INFO:tasks.workunit.client.0.smithi113.stdout:2/259: creat d8/d16/f4e x:0 0 0 2024-04-12T05:08:56.271 INFO:tasks.workunit.client.0.smithi113.stdout:2/260: fdatasync d8/d9/de/d15/d24/f3b 0 2024-04-12T05:08:56.271 INFO:tasks.workunit.client.0.smithi113.stdout:0/404: unlink d1/d43/d59/f5b 0 2024-04-12T05:08:56.274 INFO:tasks.workunit.client.1.smithi192.stdout:0/186: unlink d0/d6/f13 0 2024-04-12T05:08:56.274 INFO:tasks.workunit.client.0.smithi113.stdout:1/416: rename d10/f1e to d10/d21/d23/d7d/f7e 0 2024-04-12T05:08:56.276 INFO:tasks.workunit.client.0.smithi113.stdout:4/376: write da/d73/f45 [5125007,92787] 0 2024-04-12T05:08:56.276 INFO:tasks.workunit.client.0.smithi113.stdout:3/332: rmdir d0/d13/d15/d28/d2a/d34/d5f 39 2024-04-12T05:08:56.276 INFO:tasks.workunit.client.1.smithi192.stdout:4/301: dwrite d1/d1d/d53/d1c/f36 [0,4194304] 0 2024-04-12T05:08:56.277 INFO:tasks.workunit.client.1.smithi192.stdout:4/302: write d1/d1d/f44 [828977,95715] 0 2024-04-12T05:08:56.278 INFO:tasks.workunit.client.0.smithi113.stdout:4/377: dread - da/dd/d3a/f5b zero size 2024-04-12T05:08:56.280 INFO:tasks.workunit.client.0.smithi113.stdout:2/261: unlink d8/d9/c20 0 2024-04-12T05:08:56.280 INFO:tasks.workunit.client.0.smithi113.stdout:0/405: mknod d1/d6/da/c7a 0 2024-04-12T05:08:56.284 INFO:tasks.workunit.client.1.smithi192.stdout:0/187: rename d0/d6/f25 to d0/d6/d15/d19/f44 0 2024-04-12T05:08:56.284 INFO:tasks.workunit.client.0.smithi113.stdout:7/379: rename d0/l6d to d0/d52/l7d 0 2024-04-12T05:08:56.285 INFO:tasks.workunit.client.0.smithi113.stdout:2/262: truncate d8/d16/f1d 5294991 0 2024-04-12T05:08:56.285 INFO:tasks.workunit.client.0.smithi113.stdout:1/417: write d10/d14/d67/d70/f58 [1123730,10441] 0 2024-04-12T05:08:56.285 INFO:tasks.workunit.client.1.smithi192.stdout:4/303: dwrite d1/d1d/d53/d1c/f4e [0,4194304] 0 2024-04-12T05:08:56.287 INFO:tasks.workunit.client.1.smithi192.stdout:4/304: fdatasync d1/d54/d55/d37/f5d 0 2024-04-12T05:08:56.287 INFO:tasks.workunit.client.0.smithi113.stdout:0/406: chown d1/d6/d16/d2a/d2c/f37 82021 1 2024-04-12T05:08:56.287 INFO:tasks.workunit.client.1.smithi192.stdout:0/188: symlink d0/d6/d15/l45 0 2024-04-12T05:08:56.288 INFO:tasks.workunit.client.1.smithi192.stdout:0/189: readlink d0/l10 0 2024-04-12T05:08:56.288 INFO:tasks.workunit.client.0.smithi113.stdout:6/318: dread d8/f2d [0,4194304] 0 2024-04-12T05:08:56.288 INFO:tasks.workunit.client.0.smithi113.stdout:2/263: stat d8/d9/de/d15/d38 0 2024-04-12T05:08:56.291 INFO:tasks.workunit.client.0.smithi113.stdout:7/380: rmdir d0/dd/d2d 39 2024-04-12T05:08:56.293 INFO:tasks.workunit.client.0.smithi113.stdout:2/264: stat d8/d9/de/d12/f41 0 2024-04-12T05:08:56.298 INFO:tasks.workunit.client.0.smithi113.stdout:4/378: dwrite da/f66 [0,4194304] 0 2024-04-12T05:08:56.298 INFO:tasks.workunit.client.1.smithi192.stdout:0/190: write d0/f16 [2634589,110545] 0 2024-04-12T05:08:56.299 INFO:tasks.workunit.client.1.smithi192.stdout:4/305: rmdir d1/d1d/d53/d1c/d5c 0 2024-04-12T05:08:56.300 INFO:tasks.workunit.client.0.smithi113.stdout:1/418: creat d10/d21/d23/d7d/f7f x:0 0 0 2024-04-12T05:08:56.301 INFO:tasks.workunit.client.0.smithi113.stdout:7/381: fsync d0/dc/d6/d78/f45 0 2024-04-12T05:08:56.302 INFO:tasks.workunit.client.0.smithi113.stdout:4/379: rmdir da/d3c/d43 39 2024-04-12T05:08:56.309 INFO:tasks.workunit.client.0.smithi113.stdout:0/407: dwrite d1/d6/d16/d2a/d50/f77 [0,4194304] 0 2024-04-12T05:08:56.309 INFO:tasks.workunit.client.1.smithi192.stdout:0/191: mkdir d0/d6/d15/d1b/d46 0 2024-04-12T05:08:56.311 INFO:tasks.workunit.client.1.smithi192.stdout:4/306: dwrite d1/f3e [4194304,4194304] 0 2024-04-12T05:08:56.318 INFO:tasks.workunit.client.1.smithi192.stdout:0/192: fsync d0/d6/d15/d34/f38 0 2024-04-12T05:08:56.320 INFO:tasks.workunit.client.0.smithi113.stdout:4/380: dwrite da/d4f/f72 [0,4194304] 0 2024-04-12T05:08:56.320 INFO:tasks.workunit.client.0.smithi113.stdout:6/319: getdents d8/de/d13/d2b 0 2024-04-12T05:08:56.320 INFO:tasks.workunit.client.0.smithi113.stdout:8/409: truncate d9/d34/f57 2337861 0 2024-04-12T05:08:56.328 INFO:tasks.workunit.client.1.smithi192.stdout:0/193: rename d0/d6/d2a to d0/d26/d2e/d47 0 2024-04-12T05:08:56.329 INFO:tasks.workunit.client.0.smithi113.stdout:1/419: link d10/c46 d10/d2e/d31/d33/c80 0 2024-04-12T05:08:56.329 INFO:tasks.workunit.client.0.smithi113.stdout:0/408: creat d1/d6/d16/d2a/d2c/f7b x:0 0 0 2024-04-12T05:08:56.334 INFO:tasks.workunit.client.0.smithi113.stdout:2/265: dread d8/d9/de/d15/d24/f3e [0,4194304] 0 2024-04-12T05:08:56.334 INFO:tasks.workunit.client.1.smithi192.stdout:0/194: creat d0/d26/f48 x:0 0 0 2024-04-12T05:08:56.335 INFO:tasks.workunit.client.1.smithi192.stdout:5/291: write d0/d15/d24/d29/f2a [2083714,53684] 0 2024-04-12T05:08:56.335 INFO:tasks.workunit.client.0.smithi113.stdout:6/320: creat d8/de/d17/d3b/d63/f69 x:0 0 0 2024-04-12T05:08:56.335 INFO:tasks.workunit.client.0.smithi113.stdout:7/382: dwrite d0/dd/d1b/f5d [0,4194304] 0 2024-04-12T05:08:56.336 INFO:tasks.workunit.client.1.smithi192.stdout:1/201: sync 2024-04-12T05:08:56.336 INFO:tasks.workunit.client.1.smithi192.stdout:7/205: sync 2024-04-12T05:08:56.336 INFO:tasks.workunit.client.1.smithi192.stdout:6/323: sync 2024-04-12T05:08:56.336 INFO:tasks.workunit.client.1.smithi192.stdout:8/181: sync 2024-04-12T05:08:56.336 INFO:tasks.workunit.client.1.smithi192.stdout:5/292: fdatasync d0/d15/d24/f51 0 2024-04-12T05:08:56.337 INFO:tasks.workunit.client.0.smithi113.stdout:2/266: read - d8/d9/de/d12/f4c zero size 2024-04-12T05:08:56.339 INFO:tasks.workunit.client.1.smithi192.stdout:6/324: write d3/d8/f5d [1551997,116113] 0 2024-04-12T05:08:56.340 INFO:tasks.workunit.client.0.smithi113.stdout:7/383: write d0/dd/f66 [1897970,107762] 0 2024-04-12T05:08:56.341 INFO:tasks.workunit.client.0.smithi113.stdout:2/267: chown d8/d16/f4e 3 1 2024-04-12T05:08:56.342 INFO:tasks.workunit.client.0.smithi113.stdout:2/268: readlink d8/d9/de/d15/l4d 0 2024-04-12T05:08:56.344 INFO:tasks.workunit.client.0.smithi113.stdout:3/333: truncate d0/d13/d15/f5c 3105030 0 2024-04-12T05:08:56.345 INFO:tasks.workunit.client.1.smithi192.stdout:6/325: chown d3/d5/c15 444457663 1 2024-04-12T05:08:56.346 INFO:tasks.workunit.client.0.smithi113.stdout:0/409: symlink d1/d9/l7c 0 2024-04-12T05:08:56.348 INFO:tasks.workunit.client.1.smithi192.stdout:0/195: unlink d0/f28 0 2024-04-12T05:08:56.349 INFO:tasks.workunit.client.0.smithi113.stdout:6/321: unlink d8/ld 0 2024-04-12T05:08:56.351 INFO:tasks.workunit.client.0.smithi113.stdout:1/420: symlink d10/d21/d23/l81 0 2024-04-12T05:08:56.351 INFO:tasks.workunit.client.0.smithi113.stdout:4/381: dwrite da/d3c/d43/f4d [4194304,4194304] 0 2024-04-12T05:08:56.352 INFO:tasks.workunit.client.1.smithi192.stdout:1/202: dwrite d0/f24 [0,4194304] 0 2024-04-12T05:08:56.352 INFO:tasks.workunit.client.1.smithi192.stdout:7/206: unlink d3/d7/d15/d1a/f40 0 2024-04-12T05:08:56.354 INFO:tasks.workunit.client.0.smithi113.stdout:8/410: dread d9/d16/d7f/d26/d32/f64 [0,4194304] 0 2024-04-12T05:08:56.355 INFO:tasks.workunit.client.0.smithi113.stdout:2/269: mknod d8/d9/d27/c4f 0 2024-04-12T05:08:56.356 INFO:tasks.workunit.client.1.smithi192.stdout:6/326: creat d3/d8/d56/f75 x:0 0 0 2024-04-12T05:08:56.357 INFO:tasks.workunit.client.1.smithi192.stdout:4/307: truncate d1/d54/d55/f30 705827 0 2024-04-12T05:08:56.358 INFO:tasks.workunit.client.0.smithi113.stdout:3/334: creat d0/d13/f70 x:0 0 0 2024-04-12T05:08:56.358 INFO:tasks.workunit.client.1.smithi192.stdout:8/182: dwrite d6/da/f3a [0,4194304] 0 2024-04-12T05:08:56.361 INFO:tasks.workunit.client.0.smithi113.stdout:3/335: write d0/f9 [4392329,26322] 0 2024-04-12T05:08:56.363 INFO:tasks.workunit.client.0.smithi113.stdout:4/382: mkdir da/d24/d7b 0 2024-04-12T05:08:56.365 INFO:tasks.workunit.client.0.smithi113.stdout:1/421: dwrite d10/d14/d67/d70/f58 [0,4194304] 0 2024-04-12T05:08:56.367 INFO:tasks.workunit.client.1.smithi192.stdout:0/196: creat d0/d6/d15/d1b/d46/f49 x:0 0 0 2024-04-12T05:08:56.367 INFO:tasks.workunit.client.1.smithi192.stdout:8/183: unlink d6/f28 0 2024-04-12T05:08:56.368 INFO:tasks.workunit.client.1.smithi192.stdout:7/207: symlink d3/d7/d14/l51 0 2024-04-12T05:08:56.369 INFO:tasks.workunit.client.0.smithi113.stdout:7/384: creat d0/dc/d6/d1a/f7e x:0 0 0 2024-04-12T05:08:56.370 INFO:tasks.workunit.client.0.smithi113.stdout:3/336: unlink d0/d13/d15/c56 0 2024-04-12T05:08:56.371 INFO:tasks.workunit.client.1.smithi192.stdout:4/308: dwrite d1/f11 [0,4194304] 0 2024-04-12T05:08:56.372 INFO:tasks.workunit.client.0.smithi113.stdout:4/383: unlink l9 0 2024-04-12T05:08:56.373 INFO:tasks.workunit.client.1.smithi192.stdout:4/309: dread - d1/d49/f5b zero size 2024-04-12T05:08:56.373 INFO:tasks.workunit.client.1.smithi192.stdout:0/197: mknod d0/d26/d2e/c4a 0 2024-04-12T05:08:56.374 INFO:tasks.workunit.client.0.smithi113.stdout:6/322: dwrite d8/f2d [0,4194304] 0 2024-04-12T05:08:56.375 INFO:tasks.workunit.client.0.smithi113.stdout:4/384: read - da/d3c/d43/f75 zero size 2024-04-12T05:08:56.378 INFO:tasks.workunit.client.1.smithi192.stdout:7/208: symlink d3/d7/l52 0 2024-04-12T05:08:56.379 INFO:tasks.workunit.client.0.smithi113.stdout:3/337: symlink d0/l71 0 2024-04-12T05:08:56.381 INFO:tasks.workunit.client.0.smithi113.stdout:0/410: link d1/d6/d16/d2a/f70 d1/d6/d16/d2a/d50/f7d 0 2024-04-12T05:08:56.386 INFO:tasks.workunit.client.1.smithi192.stdout:0/198: mkdir d0/d6/d15/d34/d4b 0 2024-04-12T05:08:56.386 INFO:tasks.workunit.client.1.smithi192.stdout:4/310: mknod d1/d1d/d53/c5f 0 2024-04-12T05:08:56.386 INFO:tasks.workunit.client.1.smithi192.stdout:8/184: rmdir d6/d21/d3c 0 2024-04-12T05:08:56.387 INFO:tasks.workunit.client.0.smithi113.stdout:4/385: mknod da/d73/d1d/d47/c7c 0 2024-04-12T05:08:56.387 INFO:tasks.workunit.client.1.smithi192.stdout:8/185: readlink d6/l17 0 2024-04-12T05:08:56.387 INFO:tasks.workunit.client.0.smithi113.stdout:6/323: creat d8/de/d13/d2b/f6a x:0 0 0 2024-04-12T05:08:56.388 INFO:tasks.workunit.client.1.smithi192.stdout:8/186: stat l4 0 2024-04-12T05:08:56.392 INFO:tasks.workunit.client.0.smithi113.stdout:3/338: dwrite d0/f6 [4194304,4194304] 0 2024-04-12T05:08:56.393 INFO:tasks.workunit.client.0.smithi113.stdout:3/339: write d0/d13/f70 [36554,86131] 0 2024-04-12T05:08:56.393 INFO:tasks.workunit.client.1.smithi192.stdout:8/187: dwrite d6/d21/f23 [0,4194304] 0 2024-04-12T05:08:56.394 INFO:tasks.workunit.client.0.smithi113.stdout:0/411: mkdir d1/d6/d16/d2a/d50/d52/d7e 0 2024-04-12T05:08:56.396 INFO:tasks.workunit.client.0.smithi113.stdout:0/412: write d1/d9/f45 [1468255,66753] 0 2024-04-12T05:08:56.397 INFO:tasks.workunit.client.0.smithi113.stdout:7/385: symlink d0/dd/d2d/l7f 0 2024-04-12T05:08:56.398 INFO:tasks.workunit.client.0.smithi113.stdout:7/386: write d0/dd/d11/f4a [359782,30074] 0 2024-04-12T05:08:56.403 INFO:tasks.workunit.client.0.smithi113.stdout:6/324: rename d8/de/d17/c4d to d8/de/d13/d11/d36/d3d/c6b 0 2024-04-12T05:08:56.405 INFO:tasks.workunit.client.0.smithi113.stdout:0/413: unlink d1/d6/da/c53 0 2024-04-12T05:08:56.405 INFO:tasks.workunit.client.1.smithi192.stdout:0/199: creat d0/d6/d15/d19/f4c x:0 0 0 2024-04-12T05:08:56.406 INFO:tasks.workunit.client.1.smithi192.stdout:6/327: write d3/d8/f1c [4756950,14896] 0 2024-04-12T05:08:56.406 INFO:tasks.workunit.client.1.smithi192.stdout:0/200: stat d0/c41 0 2024-04-12T05:08:56.407 INFO:tasks.workunit.client.1.smithi192.stdout:4/311: symlink d1/l60 0 2024-04-12T05:08:56.409 INFO:tasks.workunit.client.1.smithi192.stdout:4/312: write d1/f18 [1895688,68291] 0 2024-04-12T05:08:56.409 INFO:tasks.workunit.client.1.smithi192.stdout:6/328: creat d3/d23/f76 x:0 0 0 2024-04-12T05:08:56.411 INFO:tasks.workunit.client.0.smithi113.stdout:7/387: mkdir d0/dd/d1b/d80 0 2024-04-12T05:08:56.413 INFO:tasks.workunit.client.0.smithi113.stdout:7/388: write d0/dd/d11/d44/f6b [906133,110184] 0 2024-04-12T05:08:56.413 INFO:tasks.workunit.client.1.smithi192.stdout:6/329: write d3/d5/f6 [4318108,98836] 0 2024-04-12T05:08:56.414 INFO:tasks.workunit.client.1.smithi192.stdout:8/188: dwrite d6/f9 [4194304,4194304] 0 2024-04-12T05:08:56.414 INFO:tasks.workunit.client.0.smithi113.stdout:6/325: creat d8/de/d13/f6c x:0 0 0 2024-04-12T05:08:56.415 INFO:tasks.workunit.client.1.smithi192.stdout:1/203: dread d0/fc [4194304,4194304] 0 2024-04-12T05:08:56.416 INFO:tasks.workunit.client.0.smithi113.stdout:0/414: mknod d1/d6/d34/c7f 0 2024-04-12T05:08:56.418 INFO:tasks.workunit.client.0.smithi113.stdout:6/326: dread d8/d45/f49 [0,4194304] 0 2024-04-12T05:08:56.418 INFO:tasks.workunit.client.1.smithi192.stdout:0/201: rename d0/d6/d15/f37 to d0/d6/d15/d1b/d46/f4d 0 2024-04-12T05:08:56.419 INFO:tasks.workunit.client.0.smithi113.stdout:3/340: creat d0/d13/d15/d28/d2a/d34/f72 x:0 0 0 2024-04-12T05:08:56.421 INFO:tasks.workunit.client.1.smithi192.stdout:6/330: write d3/d8/d22/f2e [3686739,129185] 0 2024-04-12T05:08:56.425 INFO:tasks.workunit.client.0.smithi113.stdout:7/389: mkdir d0/dc/d6/d1a/d81 0 2024-04-12T05:08:56.425 INFO:tasks.workunit.client.0.smithi113.stdout:3/341: read d0/fa [731702,51648] 0 2024-04-12T05:08:56.427 INFO:tasks.workunit.client.1.smithi192.stdout:1/204: mkdir d0/d2/dd/d21/d39 0 2024-04-12T05:08:56.427 INFO:tasks.workunit.client.1.smithi192.stdout:0/202: dwrite d0/d6/d22/f24 [0,4194304] 0 2024-04-12T05:08:56.428 INFO:tasks.workunit.client.0.smithi113.stdout:3/342: fdatasync d0/f18 0 2024-04-12T05:08:56.431 INFO:tasks.workunit.client.1.smithi192.stdout:8/189: rename d6/df/f19 to d6/d10/f3d 0 2024-04-12T05:08:56.432 INFO:tasks.workunit.client.0.smithi113.stdout:0/415: dwrite d1/d6/f58 [0,4194304] 0 2024-04-12T05:08:56.434 INFO:tasks.workunit.client.0.smithi113.stdout:6/327: mknod d8/de/d13/d11/d36/d3d/d4c/d26/d64/c6d 0 2024-04-12T05:08:56.435 INFO:tasks.workunit.client.1.smithi192.stdout:6/331: dwrite d3/d8/f17 [0,4194304] 0 2024-04-12T05:08:56.436 INFO:tasks.workunit.client.0.smithi113.stdout:7/390: dread d0/d6c/f70 [0,4194304] 0 2024-04-12T05:08:56.438 INFO:tasks.workunit.client.0.smithi113.stdout:6/328: creat d8/d45/f6e x:0 0 0 2024-04-12T05:08:56.439 INFO:tasks.workunit.client.0.smithi113.stdout:1/422: truncate d10/d14/d67/d70/f58 1561224 0 2024-04-12T05:08:56.444 INFO:tasks.workunit.client.0.smithi113.stdout:3/343: getdents d0/d13/d47 0 2024-04-12T05:08:56.444 INFO:tasks.workunit.client.1.smithi192.stdout:0/203: mkdir d0/d4e 0 2024-04-12T05:08:56.446 INFO:tasks.workunit.client.0.smithi113.stdout:3/344: chown d0/d13/d47 213382704 1 2024-04-12T05:08:56.449 INFO:tasks.workunit.client.0.smithi113.stdout:6/329: symlink d8/de/d13/d11/l6f 0 2024-04-12T05:08:56.449 INFO:tasks.workunit.client.0.smithi113.stdout:0/416: mkdir d1/d80 0 2024-04-12T05:08:56.451 INFO:tasks.workunit.client.1.smithi192.stdout:8/190: mkdir d6/d3e 0 2024-04-12T05:08:56.452 INFO:tasks.workunit.client.0.smithi113.stdout:1/423: dwrite d10/d2e/f71 [0,4194304] 0 2024-04-12T05:08:56.455 INFO:tasks.workunit.client.1.smithi192.stdout:4/313: truncate d1/f3e 264388 0 2024-04-12T05:08:56.455 INFO:tasks.workunit.client.0.smithi113.stdout:4/386: truncate da/f66 1924702 0 2024-04-12T05:08:56.455 INFO:tasks.workunit.client.1.smithi192.stdout:0/204: rename d0/d6/d22 to d0/d6/d15/d1b/d3a/d4f 0 2024-04-12T05:08:56.456 INFO:tasks.workunit.client.0.smithi113.stdout:7/391: mknod d0/dd/d2d/c82 0 2024-04-12T05:08:56.457 INFO:tasks.workunit.client.1.smithi192.stdout:1/205: link d0/d2/dd/f26 d0/d2/dd/d21/d33/f3a 0 2024-04-12T05:08:56.457 INFO:tasks.workunit.client.0.smithi113.stdout:4/387: readlink da/de/l59 0 2024-04-12T05:08:56.457 INFO:tasks.workunit.client.0.smithi113.stdout:3/345: chown d0/d13/d15/d28/f31 448539 1 2024-04-12T05:08:56.458 INFO:tasks.workunit.client.0.smithi113.stdout:7/392: fsync d0/dd/d11/d44/f47 0 2024-04-12T05:08:56.459 INFO:tasks.workunit.client.0.smithi113.stdout:0/417: creat d1/d6/d16/d2a/d50/d52/f81 x:0 0 0 2024-04-12T05:08:56.460 INFO:tasks.workunit.client.1.smithi192.stdout:1/206: stat d0/f24 0 2024-04-12T05:08:56.460 INFO:tasks.workunit.client.0.smithi113.stdout:4/388: chown da/d26/c51 742 1 2024-04-12T05:08:56.460 INFO:tasks.workunit.client.1.smithi192.stdout:6/332: dwrite d3/d23/f74 [0,4194304] 0 2024-04-12T05:08:56.461 INFO:tasks.workunit.client.0.smithi113.stdout:6/330: rename d8/de/c46 to d8/de/d13/d11/d36/d3d/d4c/d26/d5b/c70 0 2024-04-12T05:08:56.462 INFO:tasks.workunit.client.1.smithi192.stdout:8/191: creat d6/d10/f3f x:0 0 0 2024-04-12T05:08:56.464 INFO:tasks.workunit.client.1.smithi192.stdout:4/314: rename d1/f2e to d1/d54/f61 0 2024-04-12T05:08:56.464 INFO:tasks.workunit.client.1.smithi192.stdout:8/192: write f3 [1295666,73884] 0 2024-04-12T05:08:56.465 INFO:tasks.workunit.client.0.smithi113.stdout:1/424: symlink d10/d2e/l82 0 2024-04-12T05:08:56.467 INFO:tasks.workunit.client.1.smithi192.stdout:1/207: creat d0/d2/dd/f3b x:0 0 0 2024-04-12T05:08:56.467 INFO:tasks.workunit.client.1.smithi192.stdout:8/193: dread - d6/d10/f3f zero size 2024-04-12T05:08:56.468 INFO:tasks.workunit.client.1.smithi192.stdout:1/208: readlink d0/d2/l14 0 2024-04-12T05:08:56.468 INFO:tasks.workunit.client.0.smithi113.stdout:7/393: creat d0/dc/f83 x:0 0 0 2024-04-12T05:08:56.469 INFO:tasks.workunit.client.1.smithi192.stdout:4/315: creat d1/d54/d55/d37/f62 x:0 0 0 2024-04-12T05:08:56.469 INFO:tasks.workunit.client.1.smithi192.stdout:0/205: mkdir d0/d6/d15/d34/d4b/d50 0 2024-04-12T05:08:56.470 INFO:tasks.workunit.client.1.smithi192.stdout:1/209: fdatasync d0/d2/d1e/f36 0 2024-04-12T05:08:56.470 INFO:tasks.workunit.client.0.smithi113.stdout:0/418: creat d1/d43/d59/f82 x:0 0 0 2024-04-12T05:08:56.473 INFO:tasks.workunit.client.0.smithi113.stdout:4/389: rename da/d73/d1d/f46 to da/d3c/d43/f7d 0 2024-04-12T05:08:56.475 INFO:tasks.workunit.client.0.smithi113.stdout:1/425: rmdir d10/d41/d5b 39 2024-04-12T05:08:56.475 INFO:tasks.workunit.client.1.smithi192.stdout:6/333: rename d3/d8/d22/f27 to d3/d8/d10/d4c/d68/f77 0 2024-04-12T05:08:56.475 INFO:tasks.workunit.client.1.smithi192.stdout:0/206: write d0/d6/d15/d19/f1d [1318063,121916] 0 2024-04-12T05:08:56.477 INFO:tasks.workunit.client.0.smithi113.stdout:3/346: symlink d0/d13/d15/d28/l73 0 2024-04-12T05:08:56.480 INFO:tasks.workunit.client.1.smithi192.stdout:6/334: chown d3/d8/c1a 14688 1 2024-04-12T05:08:56.481 INFO:tasks.workunit.client.0.smithi113.stdout:7/394: creat d0/dd/d1b/f84 x:0 0 0 2024-04-12T05:08:56.483 INFO:tasks.workunit.client.1.smithi192.stdout:8/194: mknod d6/c40 0 2024-04-12T05:08:56.484 INFO:tasks.workunit.client.0.smithi113.stdout:4/390: rmdir da/dd 39 2024-04-12T05:08:56.485 INFO:tasks.workunit.client.1.smithi192.stdout:6/335: write d3/d8/d10/d4c/f62 [401889,99745] 0 2024-04-12T05:08:56.485 INFO:tasks.workunit.client.0.smithi113.stdout:4/391: write da/d73/d1d/f4c [172656,68595] 0 2024-04-12T05:08:56.485 INFO:tasks.workunit.client.1.smithi192.stdout:4/316: rename d1/d1d/d53/d1c/f28 to d1/d1d/d4c/f63 0 2024-04-12T05:08:56.486 INFO:tasks.workunit.client.0.smithi113.stdout:1/426: mknod d10/d2e/d31/d5c/c83 0 2024-04-12T05:08:56.486 INFO:tasks.workunit.client.1.smithi192.stdout:1/210: dwrite d0/d2/dd/f3b [0,4194304] 0 2024-04-12T05:08:56.486 INFO:tasks.workunit.client.0.smithi113.stdout:4/392: stat da/d73/d1d/c36 0 2024-04-12T05:08:56.487 INFO:tasks.workunit.client.0.smithi113.stdout:1/427: stat d10/d21/d23/f49 0 2024-04-12T05:08:56.488 INFO:tasks.workunit.client.0.smithi113.stdout:4/393: fsync da/d3c/d43/f5e 0 2024-04-12T05:08:56.490 INFO:tasks.workunit.client.0.smithi113.stdout:4/394: write da/d4f/f72 [200635,5963] 0 2024-04-12T05:08:56.491 INFO:tasks.workunit.client.1.smithi192.stdout:0/207: rmdir d0/d26 39 2024-04-12T05:08:56.495 INFO:tasks.workunit.client.1.smithi192.stdout:8/195: creat d6/df/d38/f41 x:0 0 0 2024-04-12T05:08:56.495 INFO:tasks.workunit.client.1.smithi192.stdout:0/208: write d0/d6/f21 [2768957,104327] 0 2024-04-12T05:08:56.501 INFO:tasks.workunit.client.1.smithi192.stdout:0/209: unlink d0/d6/f21 0 2024-04-12T05:08:56.501 INFO:tasks.workunit.client.0.smithi113.stdout:1/428: symlink d10/d2e/d31/l84 0 2024-04-12T05:08:56.503 INFO:tasks.workunit.client.1.smithi192.stdout:1/211: dwrite d0/d2/d1e/f36 [0,4194304] 0 2024-04-12T05:08:56.503 INFO:tasks.workunit.client.1.smithi192.stdout:6/336: dwrite d3/d23/f74 [0,4194304] 0 2024-04-12T05:08:56.506 INFO:tasks.workunit.client.1.smithi192.stdout:6/337: write d3/d21/f54 [23847,78104] 0 2024-04-12T05:08:56.507 INFO:tasks.workunit.client.1.smithi192.stdout:8/196: mknod d6/d21/c42 0 2024-04-12T05:08:56.510 INFO:tasks.workunit.client.0.smithi113.stdout:1/429: write d10/d2e/d31/f35 [1690283,68333] 0 2024-04-12T05:08:56.511 INFO:tasks.workunit.client.0.smithi113.stdout:3/347: creat d0/d13/d15/d28/f74 x:0 0 0 2024-04-12T05:08:56.512 INFO:tasks.workunit.client.1.smithi192.stdout:8/197: read d6/f18 [2530053,78519] 0 2024-04-12T05:08:56.512 INFO:tasks.workunit.client.1.smithi192.stdout:6/338: creat d3/d23/f78 x:0 0 0 2024-04-12T05:08:56.512 INFO:tasks.workunit.client.1.smithi192.stdout:0/210: mknod d0/d6/d15/d1b/d3a/c51 0 2024-04-12T05:08:56.513 INFO:tasks.workunit.client.0.smithi113.stdout:4/395: dwrite da/d73/d1d/f40 [0,4194304] 0 2024-04-12T05:08:56.514 INFO:tasks.workunit.client.1.smithi192.stdout:0/211: chown d0/fb 82185 1 2024-04-12T05:08:56.515 INFO:tasks.workunit.client.1.smithi192.stdout:1/212: unlink d0/d2/dd/f26 0 2024-04-12T05:08:56.519 INFO:tasks.workunit.client.1.smithi192.stdout:1/213: mkdir d0/d2/dd/d3c 0 2024-04-12T05:08:56.519 INFO:tasks.workunit.client.0.smithi113.stdout:6/331: symlink d8/de/l71 0 2024-04-12T05:08:56.519 INFO:tasks.workunit.client.0.smithi113.stdout:0/419: getdents d1/d6/d16/d2a/d2c 0 2024-04-12T05:08:56.520 INFO:tasks.workunit.client.1.smithi192.stdout:6/339: dwrite d3/dc/d39/f4e [0,4194304] 0 2024-04-12T05:08:56.520 INFO:tasks.workunit.client.0.smithi113.stdout:4/396: chown da/d73/d1d/f40 13453451 1 2024-04-12T05:08:56.521 INFO:tasks.workunit.client.1.smithi192.stdout:4/317: dread d1/d1d/d53/d1c/f4e [0,4194304] 0 2024-04-12T05:08:56.522 INFO:tasks.workunit.client.1.smithi192.stdout:6/340: chown d3/d8/d10/f71 9352 1 2024-04-12T05:08:56.522 INFO:tasks.workunit.client.0.smithi113.stdout:3/348: write d0/d13/d15/d28/d2a/f36 [928146,84366] 0 2024-04-12T05:08:56.527 INFO:tasks.workunit.client.1.smithi192.stdout:4/318: dread - d1/d1d/d53/d1c/f51 zero size 2024-04-12T05:08:56.527 INFO:tasks.workunit.client.0.smithi113.stdout:0/420: unlink d1/d43/f73 0 2024-04-12T05:08:56.530 INFO:tasks.workunit.client.1.smithi192.stdout:6/341: mkdir d3/d21/d24/d79 0 2024-04-12T05:08:56.530 INFO:tasks.workunit.client.1.smithi192.stdout:4/319: write d1/d1d/d53/d1c/f51 [836538,45173] 0 2024-04-12T05:08:56.532 INFO:tasks.workunit.client.0.smithi113.stdout:4/397: unlink da/dd/d1b/c1e 0 2024-04-12T05:08:56.533 INFO:tasks.workunit.client.1.smithi192.stdout:4/320: write d1/d1d/d4c/f5a [314848,23599] 0 2024-04-12T05:08:56.533 INFO:tasks.workunit.client.1.smithi192.stdout:6/342: truncate d3/d5/f66 4667892 0 2024-04-12T05:08:56.534 INFO:tasks.workunit.client.0.smithi113.stdout:7/395: dwrite d0/dc/d6/d1a/f3a [0,4194304] 0 2024-04-12T05:08:56.536 INFO:tasks.workunit.client.1.smithi192.stdout:6/343: creat d3/dc/d39/f7a x:0 0 0 2024-04-12T05:08:56.537 INFO:tasks.workunit.client.1.smithi192.stdout:6/344: chown d3/dc/f69 0 1 2024-04-12T05:08:56.539 INFO:tasks.workunit.client.0.smithi113.stdout:6/332: dwrite d8/d45/f49 [0,4194304] 0 2024-04-12T05:08:56.539 INFO:tasks.workunit.client.0.smithi113.stdout:3/349: creat d0/d13/d15/f75 x:0 0 0 2024-04-12T05:08:56.540 INFO:tasks.workunit.client.0.smithi113.stdout:1/430: link d10/d21/d23/d7d/f7e d10/d2e/d7c/f85 0 2024-04-12T05:08:56.542 INFO:tasks.workunit.client.0.smithi113.stdout:1/431: fsync d10/d21/f40 0 2024-04-12T05:08:56.542 INFO:tasks.workunit.client.1.smithi192.stdout:4/321: rename d1/d1d/d4c to d1/d54/d55/d64 0 2024-04-12T05:08:56.543 INFO:tasks.workunit.client.0.smithi113.stdout:4/398: rename da/d73/d14 to da/d24/d7e 0 2024-04-12T05:08:56.543 INFO:tasks.workunit.client.0.smithi113.stdout:7/396: dread d0/dd/d1b/f23 [0,4194304] 0 2024-04-12T05:08:56.545 INFO:tasks.workunit.client.1.smithi192.stdout:1/214: link d0/d1b/l27 d0/d2/dd/d21/l3d 0 2024-04-12T05:08:56.545 INFO:tasks.workunit.client.1.smithi192.stdout:3/218: sync 2024-04-12T05:08:56.545 INFO:tasks.workunit.client.0.smithi113.stdout:4/399: stat da/dd/f6e 0 2024-04-12T05:08:56.545 INFO:tasks.workunit.client.1.smithi192.stdout:6/345: rename d3/d8/l3e to d3/d8/d10/d4c/d68/l7b 0 2024-04-12T05:08:56.548 INFO:tasks.workunit.client.1.smithi192.stdout:6/346: read - d3/f1e zero size 2024-04-12T05:08:56.548 INFO:tasks.workunit.client.1.smithi192.stdout:4/322: unlink d1/d1d/d53/l22 0 2024-04-12T05:08:56.549 INFO:tasks.workunit.client.1.smithi192.stdout:2/292: sync 2024-04-12T05:08:56.550 INFO:tasks.workunit.client.0.smithi113.stdout:3/350: dwrite d0/d13/d47/f50 [0,4194304] 0 2024-04-12T05:08:56.550 INFO:tasks.workunit.client.0.smithi113.stdout:1/432: symlink d10/d14/d67/l86 0 2024-04-12T05:08:56.551 INFO:tasks.workunit.client.1.smithi192.stdout:4/323: chown d1/d54/d55/d37/f62 366393384 1 2024-04-12T05:08:56.552 INFO:tasks.workunit.client.0.smithi113.stdout:7/397: mkdir d0/dd/d11/d44/d5c/d85 0 2024-04-12T05:08:56.553 INFO:tasks.workunit.client.1.smithi192.stdout:3/219: mkdir d5/d13/d3a/d4a 0 2024-04-12T05:08:56.555 INFO:tasks.workunit.client.1.smithi192.stdout:3/220: write d5/f14 [8032246,44550] 0 2024-04-12T05:08:56.556 INFO:tasks.workunit.client.1.smithi192.stdout:2/293: fdatasync d1/d2d/f5d 0 2024-04-12T05:08:56.557 INFO:tasks.workunit.client.1.smithi192.stdout:4/324: creat d1/d1d/d53/d1c/f65 x:0 0 0 2024-04-12T05:08:56.557 INFO:tasks.workunit.client.1.smithi192.stdout:8/198: dread d6/d10/f3d [0,4194304] 0 2024-04-12T05:08:56.561 INFO:tasks.workunit.client.1.smithi192.stdout:2/294: symlink d1/d8/d3a/l63 0 2024-04-12T05:08:56.562 INFO:tasks.workunit.client.0.smithi113.stdout:3/351: mknod d0/d13/d47/d4f/c76 0 2024-04-12T05:08:56.562 INFO:tasks.workunit.client.0.smithi113.stdout:3/352: stat d0/l66 0 2024-04-12T05:08:56.562 INFO:tasks.workunit.client.0.smithi113.stdout:4/400: truncate da/dd/f32 1645798 0 2024-04-12T05:08:56.564 INFO:tasks.workunit.client.0.smithi113.stdout:1/433: symlink d10/d2e/d7c/l87 0 2024-04-12T05:08:56.565 INFO:tasks.workunit.client.0.smithi113.stdout:6/333: link d8/d45/f49 d8/de/d17/f72 0 2024-04-12T05:08:56.565 INFO:tasks.workunit.client.1.smithi192.stdout:8/199: link d6/d10/f3f d6/df/f43 0 2024-04-12T05:08:56.565 INFO:tasks.workunit.client.0.smithi113.stdout:7/398: dwrite d0/dc/d6/d1a/f3a [0,4194304] 0 2024-04-12T05:08:56.567 INFO:tasks.workunit.client.0.smithi113.stdout:3/353: mknod d0/d13/d47/d4f/c77 0 2024-04-12T05:08:56.568 INFO:tasks.workunit.client.1.smithi192.stdout:8/200: write d6/df/d31/f39 [2979257,129042] 0 2024-04-12T05:08:56.572 INFO:tasks.workunit.client.0.smithi113.stdout:4/401: mknod da/dd/c7f 0 2024-04-12T05:08:56.573 INFO:tasks.workunit.client.0.smithi113.stdout:1/434: rmdir d10/d2e/d31/d33 39 2024-04-12T05:08:56.574 INFO:tasks.workunit.client.1.smithi192.stdout:2/295: link d1/d8/d3a/de/l37 d1/l64 0 2024-04-12T05:08:56.576 INFO:tasks.workunit.client.0.smithi113.stdout:0/421: truncate d1/d43/f62 5444341 0 2024-04-12T05:08:56.580 INFO:tasks.workunit.client.0.smithi113.stdout:6/334: unlink d8/de/d13/f6c 0 2024-04-12T05:08:56.580 INFO:tasks.workunit.client.1.smithi192.stdout:4/325: truncate d1/f4b 3119304 0 2024-04-12T05:08:56.583 INFO:tasks.workunit.client.1.smithi192.stdout:8/201: getdents d6/df/d38 0 2024-04-12T05:08:56.583 INFO:tasks.workunit.client.1.smithi192.stdout:4/326: creat d1/d1d/d53/d1c/f66 x:0 0 0 2024-04-12T05:08:56.585 INFO:tasks.workunit.client.1.smithi192.stdout:6/347: dread d3/d5/f51 [0,4194304] 0 2024-04-12T05:08:56.585 INFO:tasks.workunit.client.1.smithi192.stdout:4/327: write d1/d54/d55/d37/f62 [104496,12479] 0 2024-04-12T05:08:56.587 INFO:tasks.workunit.client.0.smithi113.stdout:3/354: link d0/d13/d15/d28/d2a/f36 d0/d13/d47/f78 0 2024-04-12T05:08:56.587 INFO:tasks.workunit.client.1.smithi192.stdout:2/296: write d1/d8/f46 [1443742,96589] 0 2024-04-12T05:08:56.588 INFO:tasks.workunit.client.0.smithi113.stdout:0/422: mknod d1/d6/d34/c83 0 2024-04-12T05:08:56.589 INFO:tasks.workunit.client.1.smithi192.stdout:8/202: write d6/f18 [2095496,37748] 0 2024-04-12T05:08:56.590 INFO:tasks.workunit.client.0.smithi113.stdout:0/423: chown d1/d6/d16/f67 8557 1 2024-04-12T05:08:56.590 INFO:tasks.workunit.client.0.smithi113.stdout:0/424: dread - d1/d6/d16/f4c zero size 2024-04-12T05:08:56.590 INFO:tasks.workunit.client.0.smithi113.stdout:1/435: unlink d10/d2e/l3b 0 2024-04-12T05:08:56.592 INFO:tasks.workunit.client.0.smithi113.stdout:1/436: chown d10/d21/d23/d7d 6987 1 2024-04-12T05:08:56.592 INFO:tasks.workunit.client.0.smithi113.stdout:1/437: readlink d10/d41/l42 0 2024-04-12T05:08:56.593 INFO:tasks.workunit.client.1.smithi192.stdout:3/221: dread d5/fd [0,4194304] 0 2024-04-12T05:08:56.593 INFO:tasks.workunit.client.1.smithi192.stdout:4/328: symlink d1/d1d/l67 0 2024-04-12T05:08:56.594 INFO:tasks.workunit.client.0.smithi113.stdout:7/399: dread d0/dc/d6/d1a/f34 [0,4194304] 0 2024-04-12T05:08:56.594 INFO:tasks.workunit.client.0.smithi113.stdout:3/355: rmdir d0/d13/d47 39 2024-04-12T05:08:56.595 INFO:tasks.workunit.client.1.smithi192.stdout:8/203: creat d6/d21/f44 x:0 0 0 2024-04-12T05:08:56.596 INFO:tasks.workunit.client.0.smithi113.stdout:0/425: chown d1/d6/d16/d2a/c33 1 1 2024-04-12T05:08:56.597 INFO:tasks.workunit.client.0.smithi113.stdout:6/335: mknod d8/de/d13/d11/d36/d53/c73 0 2024-04-12T05:08:56.597 INFO:tasks.workunit.client.0.smithi113.stdout:0/426: stat d1/d6/d16/d2a/f71 0 2024-04-12T05:08:56.597 INFO:tasks.workunit.client.1.smithi192.stdout:2/297: rename d1/d8/f56 to d1/d8/d17/d19/d60/f65 0 2024-04-12T05:08:56.598 INFO:tasks.workunit.client.0.smithi113.stdout:0/427: dread - d1/d6/d16/f4c zero size 2024-04-12T05:08:56.598 INFO:tasks.workunit.client.0.smithi113.stdout:7/400: creat d0/dc/d6/f86 x:0 0 0 2024-04-12T05:08:56.599 INFO:tasks.workunit.client.1.smithi192.stdout:2/298: truncate d1/d8/f46 1869427 0 2024-04-12T05:08:56.599 INFO:tasks.workunit.client.0.smithi113.stdout:0/428: dread - d1/d6/f64 zero size 2024-04-12T05:08:56.599 INFO:tasks.workunit.client.0.smithi113.stdout:3/356: mkdir d0/d79 0 2024-04-12T05:08:56.600 INFO:tasks.workunit.client.1.smithi192.stdout:6/348: truncate d3/d8/f1c 3453905 0 2024-04-12T05:08:56.602 INFO:tasks.workunit.client.1.smithi192.stdout:8/204: rename d6/df/f26 to d6/d10/f45 0 2024-04-12T05:08:56.603 INFO:tasks.workunit.client.1.smithi192.stdout:4/329: fsync d1/d54/f61 0 2024-04-12T05:08:56.604 INFO:tasks.workunit.client.1.smithi192.stdout:2/299: creat d1/d8/d17/d19/d60/f66 x:0 0 0 2024-04-12T05:08:56.609 INFO:tasks.workunit.client.1.smithi192.stdout:8/205: mkdir d6/d10/d46 0 2024-04-12T05:08:56.610 INFO:tasks.workunit.client.1.smithi192.stdout:3/222: link d5/d13/d3c/l29 d5/d13/d3c/d36/l4b 0 2024-04-12T05:08:56.611 INFO:tasks.workunit.client.0.smithi113.stdout:7/401: symlink d0/dc/d6/d1a/l87 0 2024-04-12T05:08:56.611 INFO:tasks.workunit.client.1.smithi192.stdout:6/349: dwrite d3/d26/f4b [0,4194304] 0 2024-04-12T05:08:56.612 INFO:tasks.workunit.client.1.smithi192.stdout:2/300: symlink d1/d8/d17/d19/l67 0 2024-04-12T05:08:56.612 INFO:tasks.workunit.client.0.smithi113.stdout:1/438: rename d10/d2e/d31/d33/f2c to d10/d41/f88 0 2024-04-12T05:08:56.613 INFO:tasks.workunit.client.1.smithi192.stdout:6/350: stat d3/d26/c2b 0 2024-04-12T05:08:56.614 INFO:tasks.workunit.client.1.smithi192.stdout:3/223: symlink d5/d13/d3c/d26/l4c 0 2024-04-12T05:08:56.614 INFO:tasks.workunit.client.0.smithi113.stdout:3/357: mknod d0/d13/c7a 0 2024-04-12T05:08:56.615 INFO:tasks.workunit.client.0.smithi113.stdout:0/429: rmdir d1 39 2024-04-12T05:08:56.618 INFO:tasks.workunit.client.0.smithi113.stdout:1/439: readlink d10/d21/l38 0 2024-04-12T05:08:56.619 INFO:tasks.workunit.client.0.smithi113.stdout:1/440: stat d10/d2e/f54 0 2024-04-12T05:08:56.621 INFO:tasks.workunit.client.1.smithi192.stdout:3/224: readlink d5/d13/d3c/d26/l33 0 2024-04-12T05:08:56.622 INFO:tasks.workunit.client.1.smithi192.stdout:4/330: dwrite d1/d1d/d53/d1c/f66 [0,4194304] 0 2024-04-12T05:08:56.623 INFO:tasks.workunit.client.1.smithi192.stdout:6/351: creat d3/d8/d22/f7c x:0 0 0 2024-04-12T05:08:56.624 INFO:tasks.workunit.client.1.smithi192.stdout:3/225: write d5/d13/d1a/f3b [4686660,56039] 0 2024-04-12T05:08:56.624 INFO:tasks.workunit.client.1.smithi192.stdout:2/301: mkdir d1/d8/d17/d19/d60/d59/d68 0 2024-04-12T05:08:56.627 INFO:tasks.workunit.client.0.smithi113.stdout:1/441: dwrite d10/d21/f29 [0,4194304] 0 2024-04-12T05:08:56.628 INFO:tasks.workunit.client.0.smithi113.stdout:1/442: chown d10/d41/f65 7 1 2024-04-12T05:08:56.630 INFO:tasks.workunit.client.1.smithi192.stdout:2/302: read d1/f4 [101412,50843] 0 2024-04-12T05:08:56.630 INFO:tasks.workunit.client.1.smithi192.stdout:6/352: creat d3/d21/f7d x:0 0 0 2024-04-12T05:08:56.632 INFO:tasks.workunit.client.1.smithi192.stdout:2/303: stat d1/d8/d3a/f3f 0 2024-04-12T05:08:56.634 INFO:tasks.workunit.client.0.smithi113.stdout:3/358: rename d0/d79 to d0/d13/d15/d28/d2a/d34/d5f/d7b 0 2024-04-12T05:08:56.635 INFO:tasks.workunit.client.0.smithi113.stdout:0/430: creat d1/d80/f84 x:0 0 0 2024-04-12T05:08:56.638 INFO:tasks.workunit.client.0.smithi113.stdout:1/443: mkdir d10/d41/d89 0 2024-04-12T05:08:56.640 INFO:tasks.workunit.client.0.smithi113.stdout:3/359: stat d0/d13/d47/l6d 0 2024-04-12T05:08:56.641 INFO:tasks.workunit.client.0.smithi113.stdout:7/402: truncate d0/dd/d11/f19 423677 0 2024-04-12T05:08:56.641 INFO:tasks.workunit.client.0.smithi113.stdout:0/431: rmdir d1/d6/d16/d2a 39 2024-04-12T05:08:56.642 INFO:tasks.workunit.client.1.smithi192.stdout:4/331: dwrite d1/d1d/d53/d1c/f65 [0,4194304] 0 2024-04-12T05:08:56.644 INFO:tasks.workunit.client.1.smithi192.stdout:6/353: dread d3/d26/f4b [0,4194304] 0 2024-04-12T05:08:56.646 INFO:tasks.workunit.client.0.smithi113.stdout:3/360: dread - d0/d13/d15/d28/d2a/f2b zero size 2024-04-12T05:08:56.646 INFO:tasks.workunit.client.0.smithi113.stdout:1/444: dwrite d10/d21/d23/d7d/f79 [4194304,4194304] 0 2024-04-12T05:08:56.647 INFO:tasks.workunit.client.1.smithi192.stdout:6/354: truncate d3/d21/d24/f4f 1708479 0 2024-04-12T05:08:56.647 INFO:tasks.workunit.client.1.smithi192.stdout:6/355: chown d3 13 1 2024-04-12T05:08:56.647 INFO:tasks.workunit.client.1.smithi192.stdout:4/332: mkdir d1/d54/d55/d64/d68 0 2024-04-12T05:08:56.647 INFO:tasks.workunit.client.0.smithi113.stdout:7/403: creat d0/dc/d6/d78/f88 x:0 0 0 2024-04-12T05:08:56.649 INFO:tasks.workunit.client.0.smithi113.stdout:7/404: chown d0/dc/d6/d78 152144860 1 2024-04-12T05:08:56.649 INFO:tasks.workunit.client.1.smithi192.stdout:6/356: write d3/d5/f6 [573819,54556] 0 2024-04-12T05:08:56.649 INFO:tasks.workunit.client.0.smithi113.stdout:1/445: read d10/d2e/f54 [6402,51946] 0 2024-04-12T05:08:56.651 INFO:tasks.workunit.client.0.smithi113.stdout:1/446: readlink d10/d41/l6a 0 2024-04-12T05:08:56.651 INFO:tasks.workunit.client.0.smithi113.stdout:7/405: unlink d0/c22 0 2024-04-12T05:08:56.653 INFO:tasks.workunit.client.1.smithi192.stdout:3/226: dread d5/d13/f2a [0,4194304] 0 2024-04-12T05:08:56.653 INFO:tasks.workunit.client.1.smithi192.stdout:4/333: unlink d1/d1d/d53/f31 0 2024-04-12T05:08:56.654 INFO:tasks.workunit.client.1.smithi192.stdout:6/357: rmdir d3/d8/d56 39 2024-04-12T05:08:56.657 INFO:tasks.workunit.client.1.smithi192.stdout:4/334: write d1/d1d/d53/d1c/f36 [2897575,124175] 0 2024-04-12T05:08:56.658 INFO:tasks.workunit.client.0.smithi113.stdout:3/361: dwrite d0/fa [0,4194304] 0 2024-04-12T05:08:56.658 INFO:tasks.workunit.client.0.smithi113.stdout:3/362: read - d0/d13/f64 zero size 2024-04-12T05:08:56.660 INFO:tasks.workunit.client.0.smithi113.stdout:7/406: mknod d0/dd/d2d/c89 0 2024-04-12T05:08:56.662 INFO:tasks.workunit.client.1.smithi192.stdout:4/335: write d1/d1d/d53/d1c/f65 [233466,78875] 0 2024-04-12T05:08:56.662 INFO:tasks.workunit.client.0.smithi113.stdout:3/363: rmdir d0/d13/d47/d4f 39 2024-04-12T05:08:56.664 INFO:tasks.workunit.client.1.smithi192.stdout:6/358: dwrite d3/d21/d24/f2a [0,4194304] 0 2024-04-12T05:08:56.664 INFO:tasks.workunit.client.1.smithi192.stdout:3/227: link d5/d13/d3c/d26/f44 d5/d13/d3c/d26/f4d 0 2024-04-12T05:08:56.666 INFO:tasks.workunit.client.1.smithi192.stdout:3/228: fsync d5/d13/d3c/d36/f49 0 2024-04-12T05:08:56.667 INFO:tasks.workunit.client.1.smithi192.stdout:4/336: creat d1/d49/f69 x:0 0 0 2024-04-12T05:08:56.673 INFO:tasks.workunit.client.1.smithi192.stdout:3/229: mkdir d5/d13/d3a/d4e 0 2024-04-12T05:08:56.676 INFO:tasks.workunit.client.0.smithi113.stdout:7/407: getdents d0/dc/d6 0 2024-04-12T05:08:56.678 INFO:tasks.workunit.client.1.smithi192.stdout:6/359: rename d3/d23/f41 to d3/d8/f7e 0 2024-04-12T05:08:56.679 INFO:tasks.workunit.client.1.smithi192.stdout:4/337: creat d1/f6a x:0 0 0 2024-04-12T05:08:56.680 INFO:tasks.workunit.client.0.smithi113.stdout:7/408: creat d0/dd/d1b/d80/f8a x:0 0 0 2024-04-12T05:08:56.682 INFO:tasks.workunit.client.1.smithi192.stdout:3/230: mkdir d5/d13/d3a/d4e/d4f 0 2024-04-12T05:08:56.684 INFO:tasks.workunit.client.0.smithi113.stdout:1/447: dwrite d10/d2e/d31/d33/f50 [0,4194304] 0 2024-04-12T05:08:56.684 INFO:tasks.workunit.client.1.smithi192.stdout:3/231: write d5/d13/d3c/d36/f3f [882328,108134] 0 2024-04-12T05:08:56.684 INFO:tasks.workunit.client.1.smithi192.stdout:4/338: rmdir d1/d1d 39 2024-04-12T05:08:56.686 INFO:tasks.workunit.client.0.smithi113.stdout:1/448: fdatasync d10/d2e/d31/d33/f61 0 2024-04-12T05:08:56.688 INFO:tasks.workunit.client.1.smithi192.stdout:3/232: mknod d5/d13/c50 0 2024-04-12T05:08:56.690 INFO:tasks.workunit.client.1.smithi192.stdout:6/360: link d3/d5/ca d3/d21/c7f 0 2024-04-12T05:08:56.691 INFO:tasks.workunit.client.1.smithi192.stdout:6/361: fsync d3/d5/ff 0 2024-04-12T05:08:56.692 INFO:tasks.workunit.client.1.smithi192.stdout:6/362: chown d3/d8/d10/d4c/c6a 2787 1 2024-04-12T05:08:56.693 INFO:tasks.workunit.client.1.smithi192.stdout:6/363: write d3/d5/f45 [4093474,37592] 0 2024-04-12T05:08:56.694 INFO:tasks.workunit.client.0.smithi113.stdout:3/364: dwrite d0/d13/d15/d28/f3f [0,4194304] 0 2024-04-12T05:08:56.695 INFO:tasks.workunit.client.1.smithi192.stdout:6/364: chown d3/dc/d39/f70 533178 1 2024-04-12T05:08:56.696 INFO:tasks.workunit.client.1.smithi192.stdout:2/304: dread d1/d8/f22 [0,4194304] 0 2024-04-12T05:08:56.696 INFO:tasks.workunit.client.1.smithi192.stdout:6/365: dread - d3/dc/d39/f7a zero size 2024-04-12T05:08:56.697 INFO:tasks.workunit.client.1.smithi192.stdout:6/366: dread - d3/d5/f72 zero size 2024-04-12T05:08:56.698 INFO:tasks.workunit.client.1.smithi192.stdout:6/367: readlink d3/d21/d24/l33 0 2024-04-12T05:08:56.699 INFO:tasks.workunit.client.0.smithi113.stdout:3/365: mknod d0/d13/d15/d28/d2a/d34/c7c 0 2024-04-12T05:08:56.700 INFO:tasks.workunit.client.1.smithi192.stdout:6/368: write d3/d8/d10/d4c/f62 [1630095,20195] 0 2024-04-12T05:08:56.704 INFO:tasks.workunit.client.1.smithi192.stdout:2/305: symlink d1/d8/l69 0 2024-04-12T05:08:56.706 INFO:tasks.workunit.client.1.smithi192.stdout:2/306: mknod d1/d8/d17/d19/d60/d59/d68/c6a 0 2024-04-12T05:08:56.708 INFO:tasks.workunit.client.1.smithi192.stdout:3/233: dread d5/f19 [0,4194304] 0 2024-04-12T05:08:56.708 INFO:tasks.workunit.client.0.smithi113.stdout:1/449: dwrite fb [0,4194304] 0 2024-04-12T05:08:56.709 INFO:tasks.workunit.client.0.smithi113.stdout:1/450: chown d10/d21/c22 76978875 1 2024-04-12T05:08:56.711 INFO:tasks.workunit.client.0.smithi113.stdout:1/451: truncate d10/d41/f45 300452 0 2024-04-12T05:08:56.711 INFO:tasks.workunit.client.1.smithi192.stdout:2/307: write d1/d8/d3a/de/f57 [626324,3072] 0 2024-04-12T05:08:56.713 INFO:tasks.workunit.client.1.smithi192.stdout:6/369: dwrite d3/d8/f7e [0,4194304] 0 2024-04-12T05:08:56.714 INFO:tasks.workunit.client.0.smithi113.stdout:7/409: dread d0/dd/f1c [0,4194304] 0 2024-04-12T05:08:56.716 INFO:tasks.workunit.client.0.smithi113.stdout:1/452: symlink d10/d2e/d31/d5c/l8a 0 2024-04-12T05:08:56.717 INFO:tasks.workunit.client.0.smithi113.stdout:1/453: stat d10/d14/d67/d70/f58 0 2024-04-12T05:08:56.717 INFO:tasks.workunit.client.1.smithi192.stdout:4/339: dwrite d1/f33 [0,4194304] 0 2024-04-12T05:08:56.718 INFO:tasks.workunit.client.0.smithi113.stdout:7/410: symlink d0/dc/d6/d24/l8b 0 2024-04-12T05:08:56.719 INFO:tasks.workunit.client.1.smithi192.stdout:3/234: creat d5/d13/d3a/d4e/d4f/f51 x:0 0 0 2024-04-12T05:08:56.721 INFO:tasks.workunit.client.0.smithi113.stdout:7/411: write d0/dc/d6/d78/f31 [4169952,105562] 0 2024-04-12T05:08:56.721 INFO:tasks.workunit.client.1.smithi192.stdout:4/340: write d1/d54/d55/d37/f5d [587457,110827] 0 2024-04-12T05:08:56.722 INFO:tasks.workunit.client.1.smithi192.stdout:6/370: truncate d3/d5/ff 1879035 0 2024-04-12T05:08:56.724 INFO:tasks.workunit.client.1.smithi192.stdout:2/308: unlink d1/f13 0 2024-04-12T05:08:56.727 INFO:tasks.workunit.client.1.smithi192.stdout:4/341: unlink d1/f11 0 2024-04-12T05:08:56.727 INFO:tasks.workunit.client.0.smithi113.stdout:1/454: rename d10/d21/d23/f36 to d10/d55/f8b 0 2024-04-12T05:08:56.729 INFO:tasks.workunit.client.1.smithi192.stdout:6/371: unlink d3/d26/f4b 0 2024-04-12T05:08:56.730 INFO:tasks.workunit.client.0.smithi113.stdout:7/412: link d0/dd/d1b/f5d d0/dd/d11/f8c 0 2024-04-12T05:08:56.730 INFO:tasks.workunit.client.1.smithi192.stdout:6/372: chown d3/l1b 2 1 2024-04-12T05:08:56.735 INFO:tasks.workunit.client.1.smithi192.stdout:3/235: write d5/d13/d3c/d26/f4d [575380,111077] 0 2024-04-12T05:08:56.736 INFO:tasks.workunit.client.0.smithi113.stdout:9/430: sync 2024-04-12T05:08:56.738 INFO:tasks.workunit.client.1.smithi192.stdout:6/373: rename d3/d5/f6 to d3/d5/f80 0 2024-04-12T05:08:56.739 INFO:tasks.workunit.client.1.smithi192.stdout:4/342: mkdir d1/d6b 0 2024-04-12T05:08:56.740 INFO:tasks.workunit.client.1.smithi192.stdout:3/236: symlink d5/d13/d3c/l52 0 2024-04-12T05:08:56.744 INFO:tasks.workunit.client.0.smithi113.stdout:7/413: dread d0/dd/f66 [0,4194304] 0 2024-04-12T05:08:56.749 INFO:tasks.workunit.client.0.smithi113.stdout:7/414: dread d0/dc/d6/f16 [0,4194304] 0 2024-04-12T05:08:56.762 INFO:tasks.workunit.client.1.smithi192.stdout:7/209: sync 2024-04-12T05:08:56.762 INFO:tasks.workunit.client.1.smithi192.stdout:9/221: sync 2024-04-12T05:08:56.762 INFO:tasks.workunit.client.1.smithi192.stdout:5/293: sync 2024-04-12T05:08:56.762 INFO:tasks.workunit.client.1.smithi192.stdout:6/374: creat d3/d5/f81 x:0 0 0 2024-04-12T05:08:56.765 INFO:tasks.workunit.client.1.smithi192.stdout:2/309: truncate d1/f4 6115403 0 2024-04-12T05:08:56.766 INFO:tasks.workunit.client.1.smithi192.stdout:3/237: dread d5/d13/d3c/d26/f44 [0,4194304] 0 2024-04-12T05:08:56.766 INFO:tasks.workunit.client.0.smithi113.stdout:1/455: link d10/d2e/d31/d33/f2d d10/d55/f8c 0 2024-04-12T05:08:56.767 INFO:tasks.workunit.client.1.smithi192.stdout:9/222: unlink d11/d25/d36/f43 0 2024-04-12T05:08:56.768 INFO:tasks.workunit.client.0.smithi113.stdout:7/415: creat d0/dd/d11/d44/d5c/d85/f8d x:0 0 0 2024-04-12T05:08:56.768 INFO:tasks.workunit.client.1.smithi192.stdout:9/223: dread - d11/f47 zero size 2024-04-12T05:08:56.768 INFO:tasks.workunit.client.1.smithi192.stdout:3/238: readlink d5/d13/d1a/l38 0 2024-04-12T05:08:56.769 INFO:tasks.workunit.client.1.smithi192.stdout:5/294: write d0/d15/d24/f25 [5498066,34277] 0 2024-04-12T05:08:56.769 INFO:tasks.workunit.client.1.smithi192.stdout:7/210: mknod d3/d7/d15/c53 0 2024-04-12T05:08:56.769 INFO:tasks.workunit.client.0.smithi113.stdout:1/456: mknod d10/d2e/d7c/c8d 0 2024-04-12T05:08:56.769 INFO:tasks.workunit.client.1.smithi192.stdout:3/239: stat d5/d13/d3c/d26/l4c 0 2024-04-12T05:08:56.770 INFO:tasks.workunit.client.1.smithi192.stdout:7/211: fdatasync d3/d7/f8 0 2024-04-12T05:08:56.772 INFO:tasks.workunit.client.0.smithi113.stdout:7/416: getdents d0/dc/d50 0 2024-04-12T05:08:56.772 INFO:tasks.workunit.client.0.smithi113.stdout:1/457: fsync d10/d21/d23/d7d/f79 0 2024-04-12T05:08:56.772 INFO:tasks.workunit.client.1.smithi192.stdout:6/375: rename d3/d8/f5d to d3/d8/d10/d4c/f82 0 2024-04-12T05:08:56.773 INFO:tasks.workunit.client.1.smithi192.stdout:7/212: dread - d3/d7/d15/f3c zero size 2024-04-12T05:08:56.775 INFO:tasks.workunit.client.1.smithi192.stdout:4/343: link d1/d54/d55/d64/f63 d1/d1d/f6c 0 2024-04-12T05:08:56.775 INFO:tasks.workunit.client.0.smithi113.stdout:9/431: dwrite d1/d4/f98 [0,4194304] 0 2024-04-12T05:08:56.776 INFO:tasks.workunit.client.1.smithi192.stdout:5/295: truncate d0/f14 1097893 0 2024-04-12T05:08:56.776 INFO:tasks.workunit.client.0.smithi113.stdout:7/417: truncate d0/dc/d6/d1a/f34 4357140 0 2024-04-12T05:08:56.778 INFO:tasks.workunit.client.0.smithi113.stdout:1/458: symlink d10/d2e/d31/d5c/l8e 0 2024-04-12T05:08:56.778 INFO:tasks.workunit.client.1.smithi192.stdout:2/310: mknod d1/d2d/c6b 0 2024-04-12T05:08:56.779 INFO:tasks.workunit.client.1.smithi192.stdout:9/224: dwrite fe [0,4194304] 0 2024-04-12T05:08:56.780 INFO:tasks.workunit.client.1.smithi192.stdout:3/240: mkdir d5/d13/d1a/d53 0 2024-04-12T05:08:56.781 INFO:tasks.workunit.client.0.smithi113.stdout:1/459: truncate d10/d55/f8b 4532643 0 2024-04-12T05:08:56.782 INFO:tasks.workunit.client.0.smithi113.stdout:9/432: mknod d1/d27/c9a 0 2024-04-12T05:08:56.785 INFO:tasks.workunit.client.0.smithi113.stdout:1/460: unlink d10/d14/d67/l72 0 2024-04-12T05:08:56.786 INFO:tasks.workunit.client.1.smithi192.stdout:3/241: chown d5/d13/f2f 11431 1 2024-04-12T05:08:56.787 INFO:tasks.workunit.client.0.smithi113.stdout:9/433: fdatasync d1/d27/f5d 0 2024-04-12T05:08:56.788 INFO:tasks.workunit.client.0.smithi113.stdout:9/434: dread - d1/d4/d5/d17/f74 zero size 2024-04-12T05:08:56.788 INFO:tasks.workunit.client.0.smithi113.stdout:1/461: fsync d10/d55/f8c 0 2024-04-12T05:08:56.791 INFO:tasks.workunit.client.1.smithi192.stdout:6/376: dwrite d3/dc/f69 [0,4194304] 0 2024-04-12T05:08:56.793 INFO:tasks.workunit.client.1.smithi192.stdout:4/344: dwrite d1/d1d/f44 [0,4194304] 0 2024-04-12T05:08:56.796 INFO:tasks.workunit.client.0.smithi113.stdout:9/435: dwrite d1/d27/f95 [0,4194304] 0 2024-04-12T05:08:56.800 INFO:tasks.workunit.client.0.smithi113.stdout:9/436: mknod d1/df/d4b/d63/d3d/d52/c9b 0 2024-04-12T05:08:56.803 INFO:tasks.workunit.client.0.smithi113.stdout:9/437: creat d1/d4/d5/d22/f9c x:0 0 0 2024-04-12T05:08:56.805 INFO:tasks.workunit.client.0.smithi113.stdout:1/462: getdents d10/d2e/d31/d5c 0 2024-04-12T05:08:56.805 INFO:tasks.workunit.client.0.smithi113.stdout:1/463: stat d10/d2e/d7c/c8d 0 2024-04-12T05:08:56.807 INFO:tasks.workunit.client.0.smithi113.stdout:9/438: mknod d1/d4/d5/d6/d15/c9d 0 2024-04-12T05:08:56.807 INFO:tasks.workunit.client.1.smithi192.stdout:6/377: dwrite d3/d8/f7e [0,4194304] 0 2024-04-12T05:08:56.811 INFO:tasks.workunit.client.0.smithi113.stdout:1/464: symlink d10/d14/d67/l8f 0 2024-04-12T05:08:56.814 INFO:tasks.workunit.client.0.smithi113.stdout:9/439: mkdir d1/d4/d5/d22/d9e 0 2024-04-12T05:08:56.815 INFO:tasks.workunit.client.1.smithi192.stdout:6/378: read d3/dc/f69 [1633295,90247] 0 2024-04-12T05:08:56.816 INFO:tasks.workunit.client.0.smithi113.stdout:1/465: mknod d10/d2e/d31/d5c/c90 0 2024-04-12T05:08:56.817 INFO:tasks.workunit.client.1.smithi192.stdout:5/296: dread d0/d15/d24/f4b [0,4194304] 0 2024-04-12T05:08:56.817 INFO:tasks.workunit.client.0.smithi113.stdout:1/466: chown d10/d2e/d31/d5c/f74 48 1 2024-04-12T05:08:56.820 INFO:tasks.workunit.client.0.smithi113.stdout:1/467: creat d10/d21/d23/f91 x:0 0 0 2024-04-12T05:08:56.820 INFO:tasks.workunit.client.0.smithi113.stdout:9/440: dwrite d1/f46 [0,4194304] 0 2024-04-12T05:08:56.823 INFO:tasks.workunit.client.0.smithi113.stdout:1/468: creat d10/d2e/d7c/f92 x:0 0 0 2024-04-12T05:08:56.824 INFO:tasks.workunit.client.0.smithi113.stdout:1/469: fsync d10/d41/f65 0 2024-04-12T05:08:56.826 INFO:tasks.workunit.client.0.smithi113.stdout:9/441: getdents d1/d4/d5/d17 0 2024-04-12T05:08:56.828 INFO:tasks.workunit.client.0.smithi113.stdout:1/470: creat d10/d2e/d31/d5c/f93 x:0 0 0 2024-04-12T05:08:56.829 INFO:tasks.workunit.client.0.smithi113.stdout:9/442: mknod d1/d4/d87/c9f 0 2024-04-12T05:08:56.830 INFO:tasks.workunit.client.0.smithi113.stdout:9/443: chown d1/d4/f98 26472884 1 2024-04-12T05:08:56.834 INFO:tasks.workunit.client.0.smithi113.stdout:9/444: fsync d1/df/f4d 0 2024-04-12T05:08:56.837 INFO:tasks.workunit.client.0.smithi113.stdout:9/445: link d1/d4/d38/d65/l1d d1/d4/d6d/la0 0 2024-04-12T05:08:56.839 INFO:tasks.workunit.client.0.smithi113.stdout:9/446: truncate d1/d4/d5/d6/f3b 1818424 0 2024-04-12T05:08:56.841 INFO:tasks.workunit.client.0.smithi113.stdout:9/447: truncate d1/df/d4b/d6c/f6b 2551791 0 2024-04-12T05:08:56.843 INFO:tasks.workunit.client.0.smithi113.stdout:9/448: symlink d1/d4/d5/d17/la1 0 2024-04-12T05:08:56.850 INFO:tasks.workunit.client.1.smithi192.stdout:5/297: dread d0/f1b [0,4194304] 0 2024-04-12T05:08:56.850 INFO:tasks.workunit.client.0.smithi113.stdout:1/471: dwrite f2 [0,4194304] 0 2024-04-12T05:08:56.851 INFO:tasks.workunit.client.1.smithi192.stdout:5/298: write d0/d15/d24/d47/f4d [931709,16976] 0 2024-04-12T05:08:56.854 INFO:tasks.workunit.client.0.smithi113.stdout:1/472: mknod d10/d41/d89/c94 0 2024-04-12T05:08:56.860 INFO:tasks.workunit.client.0.smithi113.stdout:1/473: dwrite d10/d21/d23/d7d/f7f [0,4194304] 0 2024-04-12T05:08:56.861 INFO:tasks.workunit.client.0.smithi113.stdout:9/449: dread d1/df/d4b/d63/d3d/f49 [0,4194304] 0 2024-04-12T05:08:56.865 INFO:tasks.workunit.client.0.smithi113.stdout:1/474: mkdir d10/d14/d95 0 2024-04-12T05:08:56.868 INFO:tasks.workunit.client.0.smithi113.stdout:1/475: fdatasync d10/d2e/f54 0 2024-04-12T05:08:56.871 INFO:tasks.workunit.client.0.smithi113.stdout:9/450: dwrite d1/df/d4b/d63/d3d/f5f [0,4194304] 0 2024-04-12T05:08:56.874 INFO:tasks.workunit.client.0.smithi113.stdout:1/476: rename d10/d21/d23 to d10/d41/d89/d96 0 2024-04-12T05:08:56.876 INFO:tasks.workunit.client.0.smithi113.stdout:1/477: rmdir d10/d2e/d31 39 2024-04-12T05:08:56.879 INFO:tasks.workunit.client.0.smithi113.stdout:9/451: getdents d1/d4/d87 0 2024-04-12T05:08:56.880 INFO:tasks.workunit.client.0.smithi113.stdout:1/478: creat d10/d2e/d31/d5c/f97 x:0 0 0 2024-04-12T05:08:56.882 INFO:tasks.workunit.client.0.smithi113.stdout:1/479: mkdir d10/d14/d67/d70/d98 0 2024-04-12T05:08:56.883 INFO:tasks.workunit.client.0.smithi113.stdout:9/452: getdents d1/d4/d5/d6 0 2024-04-12T05:08:56.884 INFO:tasks.workunit.client.0.smithi113.stdout:9/453: chown d1/d4/d5/d22 2718 1 2024-04-12T05:08:56.888 INFO:tasks.workunit.client.0.smithi113.stdout:1/480: truncate d10/d2e/d7c/f85 399702 0 2024-04-12T05:08:56.893 INFO:tasks.workunit.client.0.smithi113.stdout:1/481: rmdir d10/d14/d67 39 2024-04-12T05:08:56.894 INFO:tasks.workunit.client.0.smithi113.stdout:9/454: write d1/df/d4b/d6c/f6b [1028673,51724] 0 2024-04-12T05:08:56.898 INFO:tasks.workunit.client.0.smithi113.stdout:1/482: mkdir d10/d14/d67/d99 0 2024-04-12T05:08:56.898 INFO:tasks.workunit.client.0.smithi113.stdout:1/483: dread - d10/d2e/d31/d5c/f97 zero size 2024-04-12T05:08:56.899 INFO:tasks.workunit.client.0.smithi113.stdout:1/484: dread - d10/d2e/d31/d5c/f97 zero size 2024-04-12T05:08:56.902 INFO:tasks.workunit.client.0.smithi113.stdout:1/485: getdents d10/d55 0 2024-04-12T05:08:56.906 INFO:tasks.workunit.client.0.smithi113.stdout:1/486: write d10/d2e/d31/d33/f2d [827597,62575] 0 2024-04-12T05:08:56.908 INFO:tasks.workunit.client.1.smithi192.stdout:1/215: sync 2024-04-12T05:08:56.908 INFO:tasks.workunit.client.1.smithi192.stdout:0/212: sync 2024-04-12T05:08:56.909 INFO:tasks.workunit.client.1.smithi192.stdout:7/213: link d3/d7/c2f d3/d7/d27/c54 0 2024-04-12T05:08:56.909 INFO:tasks.workunit.client.1.smithi192.stdout:1/216: fdatasync d0/f20 0 2024-04-12T05:08:56.910 INFO:tasks.workunit.client.1.smithi192.stdout:0/213: chown d0/d6/l12 0 1 2024-04-12T05:08:56.911 INFO:tasks.workunit.client.1.smithi192.stdout:7/214: read d3/d7/f8 [180785,120497] 0 2024-04-12T05:08:56.911 INFO:tasks.workunit.client.0.smithi113.stdout:1/487: rename d10/d14/d67/d70/f58 to d10/f9a 0 2024-04-12T05:08:56.911 INFO:tasks.workunit.client.1.smithi192.stdout:3/242: fdatasync d5/f19 0 2024-04-12T05:08:56.912 INFO:tasks.workunit.client.1.smithi192.stdout:7/215: chown d3/f21 2 1 2024-04-12T05:08:56.917 INFO:tasks.workunit.client.0.smithi113.stdout:1/488: dwrite d10/d14/d67/d70/f48 [0,4194304] 0 2024-04-12T05:08:56.917 INFO:tasks.workunit.client.1.smithi192.stdout:0/214: dwrite d0/d6/d15/d1b/d46/f49 [0,4194304] 0 2024-04-12T05:08:56.928 INFO:tasks.workunit.client.0.smithi113.stdout:1/489: dwrite d10/d41/f45 [0,4194304] 0 2024-04-12T05:08:56.932 INFO:tasks.workunit.client.0.smithi113.stdout:1/490: dread d10/d21/f29 [0,4194304] 0 2024-04-12T05:08:56.937 INFO:tasks.workunit.client.0.smithi113.stdout:1/491: rename c8 to d10/d14/c9b 0 2024-04-12T05:08:56.940 INFO:tasks.workunit.client.0.smithi113.stdout:1/492: dread d10/d2e/f54 [0,4194304] 0 2024-04-12T05:08:56.942 INFO:tasks.workunit.client.1.smithi192.stdout:8/206: sync 2024-04-12T05:08:56.944 INFO:tasks.workunit.client.1.smithi192.stdout:8/207: write d6/df/f16 [226974,127828] 0 2024-04-12T05:08:56.944 INFO:tasks.workunit.client.1.smithi192.stdout:8/208: readlink d6/df/l24 0 2024-04-12T05:08:56.944 INFO:tasks.workunit.client.1.smithi192.stdout:8/209: readlink d6/l17 0 2024-04-12T05:08:56.947 INFO:tasks.workunit.client.0.smithi113.stdout:1/493: dwrite d10/d2e/f76 [0,4194304] 0 2024-04-12T05:08:56.951 INFO:tasks.workunit.client.0.smithi113.stdout:1/494: dread f2 [0,4194304] 0 2024-04-12T05:08:56.953 INFO:tasks.workunit.client.0.smithi113.stdout:1/495: mkdir d10/d55/d9c 0 2024-04-12T05:08:56.953 INFO:tasks.workunit.client.0.smithi113.stdout:1/496: readlink d10/d21/l51 0 2024-04-12T05:08:56.956 INFO:tasks.workunit.client.0.smithi113.stdout:1/497: unlink d10/d2e/d7c/f85 0 2024-04-12T05:08:56.957 INFO:tasks.workunit.client.0.smithi113.stdout:1/498: chown d10/d2e/d31/l84 10060427 1 2024-04-12T05:08:56.967 INFO:tasks.workunit.client.0.smithi113.stdout:5/314: sync 2024-04-12T05:08:56.968 INFO:tasks.workunit.client.0.smithi113.stdout:1/499: symlink d10/d55/l9d 0 2024-04-12T05:08:56.968 INFO:tasks.workunit.client.0.smithi113.stdout:5/315: dread - d0/d10/d27/d1d/f60 zero size 2024-04-12T05:08:56.971 INFO:tasks.workunit.client.0.smithi113.stdout:1/500: fdatasync d10/d21/f47 0 2024-04-12T05:08:56.972 INFO:tasks.workunit.client.1.smithi192.stdout:4/345: creat d1/d1d/d53/d1c/f6d x:0 0 0 2024-04-12T05:08:56.972 INFO:tasks.workunit.client.1.smithi192.stdout:9/225: rmdir d11/d25 39 2024-04-12T05:08:56.977 INFO:tasks.workunit.client.0.smithi113.stdout:5/316: dwrite d0/d10/d27/d13/f62 [0,4194304] 0 2024-04-12T05:08:56.978 INFO:tasks.workunit.client.0.smithi113.stdout:1/501: link d10/d2e/d31/d33/f64 d10/d14/d67/d70/d98/f9e 0 2024-04-12T05:08:56.979 INFO:tasks.workunit.client.0.smithi113.stdout:5/317: write d0/f5 [5168953,97086] 0 2024-04-12T05:08:56.983 INFO:tasks.workunit.client.0.smithi113.stdout:5/318: symlink d0/d10/d27/d1d/d38/l63 0 2024-04-12T05:08:56.983 INFO:tasks.workunit.client.1.smithi192.stdout:7/216: symlink d3/d7/d14/d1f/l55 0 2024-04-12T05:08:56.984 INFO:tasks.workunit.client.0.smithi113.stdout:1/502: truncate d10/d21/f40 4159539 0 2024-04-12T05:08:56.985 INFO:tasks.workunit.client.1.smithi192.stdout:2/311: rmdir d1 39 2024-04-12T05:08:56.987 INFO:tasks.workunit.client.0.smithi113.stdout:5/319: creat d0/d10/d27/d1d/d38/d44/d4e/f64 x:0 0 0 2024-04-12T05:08:56.988 INFO:tasks.workunit.client.0.smithi113.stdout:1/503: dread d10/d21/f29 [0,4194304] 0 2024-04-12T05:08:56.989 INFO:tasks.workunit.client.1.smithi192.stdout:8/210: creat d6/df/d38/f47 x:0 0 0 2024-04-12T05:08:56.990 INFO:tasks.workunit.client.0.smithi113.stdout:5/320: creat d0/d10/d27/d5f/f65 x:0 0 0 2024-04-12T05:08:56.990 INFO:tasks.workunit.client.1.smithi192.stdout:7/217: dwrite d3/d1d/f28 [0,4194304] 0 2024-04-12T05:08:56.990 INFO:tasks.workunit.client.0.smithi113.stdout:1/504: write fd [3069573,125255] 0 2024-04-12T05:08:56.992 INFO:tasks.workunit.client.1.smithi192.stdout:4/346: fdatasync d1/d54/f61 0 2024-04-12T05:08:56.993 INFO:tasks.workunit.client.0.smithi113.stdout:5/321: mkdir d0/d10/d27/d1d/d38/d44/d66 0 2024-04-12T05:08:56.994 INFO:tasks.workunit.client.1.smithi192.stdout:6/379: truncate d3/d8/d48/f73 339129 0 2024-04-12T05:08:56.995 INFO:tasks.workunit.client.0.smithi113.stdout:1/505: mkdir d10/d41/d89/d96/d7d/d9f 0 2024-04-12T05:08:56.996 INFO:tasks.workunit.client.1.smithi192.stdout:5/299: fsync d0/d21/f33 0 2024-04-12T05:08:56.997 INFO:tasks.workunit.client.1.smithi192.stdout:4/347: dread d1/d54/f61 [0,4194304] 0 2024-04-12T05:08:56.997 INFO:tasks.workunit.client.0.smithi113.stdout:1/506: write d10/d2e/d31/f53 [8300,123346] 0 2024-04-12T05:08:56.997 INFO:tasks.workunit.client.1.smithi192.stdout:1/217: creat d0/d2/dd/d3c/f3e x:0 0 0 2024-04-12T05:08:56.998 INFO:tasks.workunit.client.0.smithi113.stdout:1/507: readlink d10/d2e/d31/l5e 0 2024-04-12T05:08:56.998 INFO:tasks.workunit.client.1.smithi192.stdout:3/243: mknod d5/d13/d3a/d4a/c54 0 2024-04-12T05:08:56.999 INFO:tasks.workunit.client.0.smithi113.stdout:5/322: rename d0/d10/d27/d1d/d38/d3b/c3d to d0/d10/d27/d1d/d41/c67 0 2024-04-12T05:08:56.999 INFO:tasks.workunit.client.1.smithi192.stdout:4/348: fdatasync d1/d1d/d53/d1c/f6d 0 2024-04-12T05:08:56.999 INFO:tasks.workunit.client.0.smithi113.stdout:5/323: dread - d0/d10/d27/d1d/d38/f51 zero size 2024-04-12T05:08:57.000 INFO:tasks.workunit.client.1.smithi192.stdout:3/244: write d5/f14 [3305930,20215] 0 2024-04-12T05:08:57.001 INFO:tasks.workunit.client.1.smithi192.stdout:4/349: chown d1/d54/d55/d37/c4d 1004 1 2024-04-12T05:08:57.002 INFO:tasks.workunit.client.0.smithi113.stdout:5/324: truncate d0/d10/d27/d1d/f49 402883 0 2024-04-12T05:08:57.002 INFO:tasks.workunit.client.0.smithi113.stdout:1/508: link d10/l4c d10/d14/d67/la0 0 2024-04-12T05:08:57.004 INFO:tasks.workunit.client.1.smithi192.stdout:2/312: write d1/d8/d17/d19/d60/f66 [97642,42926] 0 2024-04-12T05:08:57.004 INFO:tasks.workunit.client.1.smithi192.stdout:0/215: mknod d0/c52 0 2024-04-12T05:08:57.006 INFO:tasks.workunit.client.1.smithi192.stdout:0/216: chown d0/d6/d15/d19/f4c 0 1 2024-04-12T05:08:57.006 INFO:tasks.workunit.client.0.smithi113.stdout:5/325: rename d0/d10/d27/lf to d0/d10/d27/d5f/l68 0 2024-04-12T05:08:57.007 INFO:tasks.workunit.client.1.smithi192.stdout:7/218: creat d3/d7/f56 x:0 0 0 2024-04-12T05:08:57.008 INFO:tasks.workunit.client.0.smithi113.stdout:1/509: truncate d10/d14/f2a 1067483 0 2024-04-12T05:08:57.009 INFO:tasks.workunit.client.1.smithi192.stdout:6/380: mkdir d3/d8/d10/d4c/d68/d83 0 2024-04-12T05:08:57.009 INFO:tasks.workunit.client.0.smithi113.stdout:5/326: symlink d0/d10/d27/d1d/d41/l69 0 2024-04-12T05:08:57.012 INFO:tasks.workunit.client.1.smithi192.stdout:5/300: rename d0/d15/d24/d29/f3b to d0/d15/d32/f6a 0 2024-04-12T05:08:57.014 INFO:tasks.workunit.client.1.smithi192.stdout:7/219: dwrite f1 [0,4194304] 0 2024-04-12T05:08:57.016 INFO:tasks.workunit.client.0.smithi113.stdout:5/327: dwrite d0/d10/d27/d5f/f65 [0,4194304] 0 2024-04-12T05:08:57.018 INFO:tasks.workunit.client.1.smithi192.stdout:4/350: creat d1/d54/f6e x:0 0 0 2024-04-12T05:08:57.019 INFO:tasks.workunit.client.1.smithi192.stdout:2/313: rmdir d1/d8/d17/d19/d40 39 2024-04-12T05:08:57.019 INFO:tasks.workunit.client.1.smithi192.stdout:8/211: creat d6/df/d37/f48 x:0 0 0 2024-04-12T05:08:57.020 INFO:tasks.workunit.client.0.smithi113.stdout:5/328: creat d0/d10/d27/d1d/d3f/f6a x:0 0 0 2024-04-12T05:08:57.022 INFO:tasks.workunit.client.1.smithi192.stdout:8/212: chown d6/df/f15 3205 1 2024-04-12T05:08:57.022 INFO:tasks.workunit.client.1.smithi192.stdout:2/314: chown d1/d2d/f3d 18632223 1 2024-04-12T05:08:57.028 INFO:tasks.workunit.client.1.smithi192.stdout:5/301: creat d0/d15/d24/d47/f6b x:0 0 0 2024-04-12T05:08:57.030 INFO:tasks.workunit.client.1.smithi192.stdout:7/220: mknod d3/d11/c57 0 2024-04-12T05:08:57.032 INFO:tasks.workunit.client.1.smithi192.stdout:0/217: fsync d0/d6/d15/f3d 0 2024-04-12T05:08:57.032 INFO:tasks.workunit.client.1.smithi192.stdout:2/315: mknod d1/d8/d17/d19/d60/d59/d68/c6c 0 2024-04-12T05:08:57.033 INFO:tasks.workunit.client.1.smithi192.stdout:6/381: dwrite d3/d5/f51 [0,4194304] 0 2024-04-12T05:08:57.033 INFO:tasks.workunit.client.1.smithi192.stdout:3/245: creat d5/f55 x:0 0 0 2024-04-12T05:08:57.034 INFO:tasks.workunit.client.1.smithi192.stdout:9/226: getdents d11/d25/d36/d3d 0 2024-04-12T05:08:57.035 INFO:tasks.workunit.client.1.smithi192.stdout:8/213: dread d6/d10/f3d [4194304,4194304] 0 2024-04-12T05:08:57.035 INFO:tasks.workunit.client.1.smithi192.stdout:4/351: symlink d1/l6f 0 2024-04-12T05:08:57.038 INFO:tasks.workunit.client.1.smithi192.stdout:4/352: readlink d1/d49/l4f 0 2024-04-12T05:08:57.038 INFO:tasks.workunit.client.1.smithi192.stdout:2/316: fdatasync d1/d8/d3a/de/f30 0 2024-04-12T05:08:57.038 INFO:tasks.workunit.client.1.smithi192.stdout:3/246: fsync d5/fd 0 2024-04-12T05:08:57.042 INFO:tasks.workunit.client.1.smithi192.stdout:9/227: unlink d11/d15/d21/c3c 0 2024-04-12T05:08:57.043 INFO:tasks.workunit.client.1.smithi192.stdout:8/214: creat d6/da/f49 x:0 0 0 2024-04-12T05:08:57.043 INFO:tasks.workunit.client.1.smithi192.stdout:6/382: creat d3/d8/d48/f84 x:0 0 0 2024-04-12T05:08:57.044 INFO:tasks.workunit.client.1.smithi192.stdout:0/218: getdents d0/d6/d15/d34/d4b/d50 0 2024-04-12T05:08:57.044 INFO:tasks.workunit.client.1.smithi192.stdout:4/353: readlink d1/d1d/d53/l38 0 2024-04-12T05:08:57.046 INFO:tasks.workunit.client.1.smithi192.stdout:0/219: chown d0/d6/d15/d1b/d3a/d4f/d33 130 1 2024-04-12T05:08:57.047 INFO:tasks.workunit.client.0.smithi113.stdout:5/329: dwrite d0/d10/f19 [0,4194304] 0 2024-04-12T05:08:57.049 INFO:tasks.workunit.client.1.smithi192.stdout:6/383: stat d3/d8/d10/d4c/d68/l7b 0 2024-04-12T05:08:57.050 INFO:tasks.workunit.client.1.smithi192.stdout:4/354: mknod d1/d54/c70 0 2024-04-12T05:08:57.051 INFO:tasks.workunit.client.0.smithi113.stdout:5/330: rmdir d0/d10/d27/d1d/d38/d3b 39 2024-04-12T05:08:57.051 INFO:tasks.workunit.client.1.smithi192.stdout:9/228: creat d11/d15/d21/d2c/d1f/f4d x:0 0 0 2024-04-12T05:08:57.053 INFO:tasks.workunit.client.1.smithi192.stdout:3/247: dwrite d5/d13/d3c/d26/f44 [0,4194304] 0 2024-04-12T05:08:57.053 INFO:tasks.workunit.client.0.smithi113.stdout:5/331: truncate d0/d10/d27/d1d/d38/d44/d4e/f64 374117 0 2024-04-12T05:08:57.055 INFO:tasks.workunit.client.1.smithi192.stdout:2/317: rename d1/d8/f50 to d1/f6d 0 2024-04-12T05:08:57.059 INFO:tasks.workunit.client.1.smithi192.stdout:2/318: write d1/d8/d3a/f3b [196815,86280] 0 2024-04-12T05:08:57.059 INFO:tasks.workunit.client.0.smithi113.stdout:5/332: mkdir d0/d10/d27/d1d/d38/d3b/d6b 0 2024-04-12T05:08:57.061 INFO:tasks.workunit.client.1.smithi192.stdout:1/218: write d0/d2/dd/d21/d33/f3a [6493232,41715] 0 2024-04-12T05:08:57.061 INFO:tasks.workunit.client.0.smithi113.stdout:5/333: read d0/d10/d27/d1d/f21 [164416,75222] 0 2024-04-12T05:08:57.061 INFO:tasks.workunit.client.1.smithi192.stdout:0/220: mkdir d0/d4e/d53 0 2024-04-12T05:08:57.066 INFO:tasks.workunit.client.1.smithi192.stdout:6/384: unlink d3/d21/c5c 0 2024-04-12T05:08:57.067 INFO:tasks.workunit.client.1.smithi192.stdout:9/229: dwrite d11/f13 [0,4194304] 0 2024-04-12T05:08:57.068 INFO:tasks.workunit.client.1.smithi192.stdout:1/219: write d0/d2/dd/f3b [1552756,61771] 0 2024-04-12T05:08:57.071 INFO:tasks.workunit.client.1.smithi192.stdout:8/215: getdents d6/d10 0 2024-04-12T05:08:57.072 INFO:tasks.workunit.client.1.smithi192.stdout:3/248: dwrite d5/d13/d3c/f25 [4194304,4194304] 0 2024-04-12T05:08:57.079 INFO:tasks.workunit.client.1.smithi192.stdout:6/385: chown d3/d8/d10/d4c/c6a 78 1 2024-04-12T05:08:57.082 INFO:tasks.workunit.client.0.smithi113.stdout:5/334: dread d0/d10/d27/f25 [0,4194304] 0 2024-04-12T05:08:57.084 INFO:tasks.workunit.client.1.smithi192.stdout:1/220: mkdir d0/d2/d1e/d3f 0 2024-04-12T05:08:57.084 INFO:tasks.workunit.client.1.smithi192.stdout:4/355: getdents d1/d54/d55 0 2024-04-12T05:08:57.084 INFO:tasks.workunit.client.1.smithi192.stdout:2/319: dwrite d1/d8/f46 [0,4194304] 0 2024-04-12T05:08:57.085 INFO:tasks.workunit.client.0.smithi113.stdout:5/335: creat d0/d10/d27/d1d/d3f/f6c x:0 0 0 2024-04-12T05:08:57.087 INFO:tasks.workunit.client.1.smithi192.stdout:3/249: mknod d5/d13/c56 0 2024-04-12T05:08:57.088 INFO:tasks.workunit.client.1.smithi192.stdout:4/356: fdatasync d1/f6a 0 2024-04-12T05:08:57.088 INFO:tasks.workunit.client.1.smithi192.stdout:6/386: mkdir d3/d8/d48/d85 0 2024-04-12T05:08:57.089 INFO:tasks.workunit.client.0.smithi113.stdout:5/336: creat d0/d10/d27/d13/f6d x:0 0 0 2024-04-12T05:08:57.090 INFO:tasks.workunit.client.1.smithi192.stdout:3/250: fdatasync d5/f14 0 2024-04-12T05:08:57.090 INFO:tasks.workunit.client.1.smithi192.stdout:5/302: dread d0/d15/f23 [0,4194304] 0 2024-04-12T05:08:57.092 INFO:tasks.workunit.client.1.smithi192.stdout:4/357: fdatasync d1/d1d/d53/d1c/f66 0 2024-04-12T05:08:57.092 INFO:tasks.workunit.client.1.smithi192.stdout:8/216: dread d6/d21/f23 [0,4194304] 0 2024-04-12T05:08:57.093 INFO:tasks.workunit.client.1.smithi192.stdout:9/230: dwrite d11/d15/f45 [0,4194304] 0 2024-04-12T05:08:57.095 INFO:tasks.workunit.client.0.smithi113.stdout:5/337: rename d0/d10/d27/d33 to d0/d10/d6e 0 2024-04-12T05:08:57.096 INFO:tasks.workunit.client.0.smithi113.stdout:5/338: chown d0/d10/d27/d1d/d38/d44/d4e 70 1 2024-04-12T05:08:57.098 INFO:tasks.workunit.client.0.smithi113.stdout:5/339: rename d0 to d0/d10/d6f 22 2024-04-12T05:08:57.100 INFO:tasks.workunit.client.1.smithi192.stdout:6/387: fsync d3/d8/d48/f73 0 2024-04-12T05:08:57.100 INFO:tasks.workunit.client.1.smithi192.stdout:4/358: rmdir d1/d54/d55/d37 39 2024-04-12T05:08:57.101 INFO:tasks.workunit.client.1.smithi192.stdout:0/221: write d0/d6/d15/d19/f44 [2202809,47286] 0 2024-04-12T05:08:57.101 INFO:tasks.workunit.client.1.smithi192.stdout:1/221: dwrite d0/d2/dd/f15 [0,4194304] 0 2024-04-12T05:08:57.101 INFO:tasks.workunit.client.1.smithi192.stdout:9/231: symlink d11/d37/l4e 0 2024-04-12T05:08:57.102 INFO:tasks.workunit.client.1.smithi192.stdout:6/388: chown d3/dc/d39/c55 2 1 2024-04-12T05:08:57.102 INFO:tasks.workunit.client.1.smithi192.stdout:8/217: creat d6/df/d37/f4a x:0 0 0 2024-04-12T05:08:57.105 INFO:tasks.workunit.client.1.smithi192.stdout:3/251: rename d5/d13/d3a/d4e/d4f to d5/d13/d3a/d4e/d57 0 2024-04-12T05:08:57.106 INFO:tasks.workunit.client.1.smithi192.stdout:2/320: write d1/f4 [4501338,113469] 0 2024-04-12T05:08:57.107 INFO:tasks.workunit.client.1.smithi192.stdout:2/321: fsync d1/d8/d3a/f3b 0 2024-04-12T05:08:57.108 INFO:tasks.workunit.client.1.smithi192.stdout:2/322: chown d1/d2d/f5d 1377 1 2024-04-12T05:08:57.109 INFO:tasks.workunit.client.1.smithi192.stdout:9/232: rmdir d11/d15/d21/d2c/d1f 39 2024-04-12T05:08:57.110 INFO:tasks.workunit.client.1.smithi192.stdout:6/389: fsync d3/d8/d22/f65 0 2024-04-12T05:08:57.110 INFO:tasks.workunit.client.1.smithi192.stdout:6/390: chown d3/d8/d10/d4c/f62 510 1 2024-04-12T05:08:57.112 INFO:tasks.workunit.client.1.smithi192.stdout:0/222: rename d0/l32 to d0/d6/d15/d34/d4b/l54 0 2024-04-12T05:08:57.114 INFO:tasks.workunit.client.1.smithi192.stdout:1/222: rename d0/d2/dd/d21/l2a to d0/d2/dd/d21/l40 0 2024-04-12T05:08:57.117 INFO:tasks.workunit.client.1.smithi192.stdout:6/391: creat d3/d8/d10/d4c/f86 x:0 0 0 2024-04-12T05:08:57.118 INFO:tasks.workunit.client.0.smithi113.stdout:8/411: sync 2024-04-12T05:08:57.118 INFO:tasks.workunit.client.0.smithi113.stdout:2/270: sync 2024-04-12T05:08:57.118 INFO:tasks.workunit.client.0.smithi113.stdout:6/336: sync 2024-04-12T05:08:57.118 INFO:tasks.workunit.client.0.smithi113.stdout:4/402: sync 2024-04-12T05:08:57.118 INFO:tasks.workunit.client.1.smithi192.stdout:2/323: rename d1/d8/ca to d1/d8/d17/d19/d60/d59/c6e 0 2024-04-12T05:08:57.119 INFO:tasks.workunit.client.1.smithi192.stdout:2/324: write d1/d8/d3a/f3b [940528,75671] 0 2024-04-12T05:08:57.120 INFO:tasks.workunit.client.0.smithi113.stdout:6/337: chown d8/de/d13/d11/d36/f35 60 1 2024-04-12T05:08:57.120 INFO:tasks.workunit.client.1.smithi192.stdout:9/233: mknod d11/d15/d21/d2c/d1f/c4f 0 2024-04-12T05:08:57.124 INFO:tasks.workunit.client.1.smithi192.stdout:9/234: mkdir d11/d15/d21/d2c/d1f/d50 0 2024-04-12T05:08:57.127 INFO:tasks.workunit.client.1.smithi192.stdout:0/223: rename d0/d26 to d0/d6/d15/d55 0 2024-04-12T05:08:57.129 INFO:tasks.workunit.client.1.smithi192.stdout:2/325: dwrite d1/d8/d17/d19/d40/f5b [0,4194304] 0 2024-04-12T05:08:57.131 INFO:tasks.workunit.client.1.smithi192.stdout:3/252: dread d5/d13/f2e [0,4194304] 0 2024-04-12T05:08:57.131 INFO:tasks.workunit.client.1.smithi192.stdout:2/326: write d1/d2d/f32 [2636099,130750] 0 2024-04-12T05:08:57.133 INFO:tasks.workunit.client.1.smithi192.stdout:0/224: rename d0/d6/d15/d19/f1c to d0/d4e/f56 0 2024-04-12T05:08:57.133 INFO:tasks.workunit.client.1.smithi192.stdout:6/392: dread d3/d21/d24/f4f [0,4194304] 0 2024-04-12T05:08:57.134 INFO:tasks.workunit.client.1.smithi192.stdout:9/235: link d11/d15/l2b d11/l51 0 2024-04-12T05:08:57.134 INFO:tasks.workunit.client.1.smithi192.stdout:6/393: write d3/d8/d48/f63 [610963,68218] 0 2024-04-12T05:08:57.135 INFO:tasks.workunit.client.1.smithi192.stdout:9/236: chown d11/d15/d21/l31 0 1 2024-04-12T05:08:57.137 INFO:tasks.workunit.client.1.smithi192.stdout:9/237: truncate d11/f47 60598 0 2024-04-12T05:08:57.137 INFO:tasks.workunit.client.1.smithi192.stdout:3/253: unlink d5/d13/d3c/l52 0 2024-04-12T05:08:57.138 INFO:tasks.workunit.client.1.smithi192.stdout:9/238: chown d11/f47 848883048 1 2024-04-12T05:08:57.139 INFO:tasks.workunit.client.1.smithi192.stdout:2/327: creat d1/d2d/f6f x:0 0 0 2024-04-12T05:08:57.139 INFO:tasks.workunit.client.1.smithi192.stdout:0/225: dwrite d0/d6/d15/d19/f36 [0,4194304] 0 2024-04-12T05:08:57.140 INFO:tasks.workunit.client.1.smithi192.stdout:9/239: readlink d11/d15/d21/l31 0 2024-04-12T05:08:57.141 INFO:tasks.workunit.client.0.smithi113.stdout:5/340: mknod d0/d10/d27/d1d/d38/d44/d66/c70 0 2024-04-12T05:08:57.150 INFO:tasks.workunit.client.0.smithi113.stdout:2/271: creat d8/d9/d1a/f50 x:0 0 0 2024-04-12T05:08:57.150 INFO:tasks.workunit.client.0.smithi113.stdout:8/412: creat d9/d16/d1c/d59/d68/f80 x:0 0 0 2024-04-12T05:08:57.151 INFO:tasks.workunit.client.0.smithi113.stdout:4/403: mkdir da/d70/d80 0 2024-04-12T05:08:57.152 INFO:tasks.workunit.client.0.smithi113.stdout:2/272: write d8/d9/de/d12/f4c [624674,105708] 0 2024-04-12T05:08:57.152 INFO:tasks.workunit.client.0.smithi113.stdout:8/413: chown d9/d16/d7f/d26/d32/l7b 11896348 1 2024-04-12T05:08:57.154 INFO:tasks.workunit.client.0.smithi113.stdout:8/414: truncate d9/d16/d7f/d26/f4d 1299444 0 2024-04-12T05:08:57.155 INFO:tasks.workunit.client.0.smithi113.stdout:5/341: mkdir d0/d10/d27/d1d/d38/d3b/d6b/d71 0 2024-04-12T05:08:57.157 INFO:tasks.workunit.client.1.smithi192.stdout:6/394: write d3/d5/f80 [1229588,11095] 0 2024-04-12T05:08:57.157 INFO:tasks.workunit.client.1.smithi192.stdout:2/328: chown d1/d8/d3a/l14 4833 1 2024-04-12T05:08:57.157 INFO:tasks.workunit.client.1.smithi192.stdout:9/240: creat d11/d25/f52 x:0 0 0 2024-04-12T05:08:57.157 INFO:tasks.workunit.client.1.smithi192.stdout:3/254: rename d5/d13/d3c/d36/f3f to d5/d13/d1a/d53/f58 0 2024-04-12T05:08:57.157 INFO:tasks.workunit.client.1.smithi192.stdout:2/329: rmdir d1/d8/d17/d19/d60/d59/d68 39 2024-04-12T05:08:57.157 INFO:tasks.workunit.client.1.smithi192.stdout:9/241: chown d11/d15/d21/d2c 527597618 1 2024-04-12T05:08:57.157 INFO:tasks.workunit.client.1.smithi192.stdout:2/330: fsync d1/d2d/f48 0 2024-04-12T05:08:57.157 INFO:tasks.workunit.client.1.smithi192.stdout:0/226: mknod d0/d4e/d53/c57 0 2024-04-12T05:08:57.157 INFO:tasks.workunit.client.1.smithi192.stdout:9/242: write d11/d15/d21/d2c/d1f/f4d [262123,104306] 0 2024-04-12T05:08:57.158 INFO:tasks.workunit.client.1.smithi192.stdout:3/255: link d5/f2d d5/d13/d3c/f59 0 2024-04-12T05:08:57.159 INFO:tasks.workunit.client.0.smithi113.stdout:2/273: unlink d8/d16/f46 0 2024-04-12T05:08:57.160 INFO:tasks.workunit.client.0.smithi113.stdout:2/274: dread - d8/d9/de/d15/d24/f3b zero size 2024-04-12T05:08:57.163 INFO:tasks.workunit.client.0.smithi113.stdout:8/415: mkdir d9/d16/d7f/d26/d81 0 2024-04-12T05:08:57.165 INFO:tasks.workunit.client.0.smithi113.stdout:4/404: rmdir da/d6b 0 2024-04-12T05:08:57.166 INFO:tasks.workunit.client.1.smithi192.stdout:2/331: creat d1/f70 x:0 0 0 2024-04-12T05:08:57.168 INFO:tasks.workunit.client.1.smithi192.stdout:9/243: unlink d11/d15/d21/l20 0 2024-04-12T05:08:57.168 INFO:tasks.workunit.client.1.smithi192.stdout:4/359: write d1/d54/d55/d64/f63 [2253648,87479] 0 2024-04-12T05:08:57.168 INFO:tasks.workunit.client.0.smithi113.stdout:2/275: dwrite d8/d9/de/d15/d24/f4b [0,4194304] 0 2024-04-12T05:08:57.169 INFO:tasks.workunit.client.1.smithi192.stdout:5/303: dwrite d0/f14 [0,4194304] 0 2024-04-12T05:08:57.169 INFO:tasks.workunit.client.0.smithi113.stdout:4/405: mknod da/d70/d80/c81 0 2024-04-12T05:08:57.170 INFO:tasks.workunit.client.0.smithi113.stdout:4/406: chown da/d24/f53 13 1 2024-04-12T05:08:57.173 INFO:tasks.workunit.client.1.smithi192.stdout:5/304: fdatasync d0/f1b 0 2024-04-12T05:08:57.174 INFO:tasks.workunit.client.0.smithi113.stdout:2/276: rmdir d8/d9/de/d12/d2b 39 2024-04-12T05:08:57.175 INFO:tasks.workunit.client.0.smithi113.stdout:2/277: dread - d8/f1e zero size 2024-04-12T05:08:57.176 INFO:tasks.workunit.client.1.smithi192.stdout:9/244: unlink d11/d37/l4e 0 2024-04-12T05:08:57.178 INFO:tasks.workunit.client.1.smithi192.stdout:2/332: mkdir d1/d8/d17/d19/d71 0 2024-04-12T05:08:57.179 INFO:tasks.workunit.client.0.smithi113.stdout:4/407: dwrite da/d3c/d43/f5e [0,4194304] 0 2024-04-12T05:08:57.181 INFO:tasks.workunit.client.1.smithi192.stdout:4/360: mkdir d1/d54/d55/d37/d71 0 2024-04-12T05:08:57.181 INFO:tasks.workunit.client.0.smithi113.stdout:4/408: readlink da/dd/d3a/l3f 0 2024-04-12T05:08:57.184 INFO:tasks.workunit.client.0.smithi113.stdout:2/278: creat d8/d9/de/d15/d38/f51 x:0 0 0 2024-04-12T05:08:57.184 INFO:tasks.workunit.client.1.smithi192.stdout:1/223: truncate d0/d2/dd/f1d 2605077 0 2024-04-12T05:08:57.185 INFO:tasks.workunit.client.1.smithi192.stdout:9/245: mknod d11/d37/d33/c53 0 2024-04-12T05:08:57.187 INFO:tasks.workunit.client.0.smithi113.stdout:2/279: write d8/d9/de/d15/d38/f51 [306831,26793] 0 2024-04-12T05:08:57.187 INFO:tasks.workunit.client.1.smithi192.stdout:2/333: mkdir d1/d8/d3a/d72 0 2024-04-12T05:08:57.189 INFO:tasks.workunit.client.1.smithi192.stdout:2/334: fdatasync d1/f4 0 2024-04-12T05:08:57.191 INFO:tasks.workunit.client.0.smithi113.stdout:2/280: creat d8/d9/de/d15/d38/f52 x:0 0 0 2024-04-12T05:08:57.191 INFO:tasks.workunit.client.0.smithi113.stdout:4/409: dwrite da/d4f/f4e [0,4194304] 0 2024-04-12T05:08:57.192 INFO:tasks.workunit.client.1.smithi192.stdout:5/305: creat d0/d21/d5e/f6c x:0 0 0 2024-04-12T05:08:57.193 INFO:tasks.workunit.client.1.smithi192.stdout:0/227: rmdir d0/d6/d15 39 2024-04-12T05:08:57.196 INFO:tasks.workunit.client.0.smithi113.stdout:2/281: getdents d8/d9/de/d12/d3c 0 2024-04-12T05:08:57.197 INFO:tasks.workunit.client.1.smithi192.stdout:9/246: creat d11/d37/d33/f54 x:0 0 0 2024-04-12T05:08:57.198 INFO:tasks.workunit.client.1.smithi192.stdout:1/224: dwrite d0/d2/dd/d21/f38 [0,4194304] 0 2024-04-12T05:08:57.199 INFO:tasks.workunit.client.1.smithi192.stdout:9/247: dread - d11/d25/f52 zero size 2024-04-12T05:08:57.199 INFO:tasks.workunit.client.0.smithi113.stdout:4/410: dread da/d73/d1d/f6a [0,4194304] 0 2024-04-12T05:08:57.199 INFO:tasks.workunit.client.1.smithi192.stdout:2/335: dread d1/d8/d3a/de/f57 [0,4194304] 0 2024-04-12T05:08:57.203 INFO:tasks.workunit.client.0.smithi113.stdout:4/411: rename da/d3c/d43/c63 to da/d4f/c82 0 2024-04-12T05:08:57.204 INFO:tasks.workunit.client.1.smithi192.stdout:0/228: write d0/d6/d15/f23 [110097,51268] 0 2024-04-12T05:08:57.204 INFO:tasks.workunit.client.0.smithi113.stdout:4/412: fsync da/fb 0 2024-04-12T05:08:57.209 INFO:tasks.workunit.client.1.smithi192.stdout:2/336: mkdir d1/d2d/d73 0 2024-04-12T05:08:57.209 INFO:tasks.workunit.client.0.smithi113.stdout:4/413: rename da/d73/d1d to da/d3c/d83 0 2024-04-12T05:08:57.210 INFO:tasks.workunit.client.1.smithi192.stdout:5/306: mknod d0/d15/d64/c6d 0 2024-04-12T05:08:57.211 INFO:tasks.workunit.client.0.smithi113.stdout:5/342: dread d0/d10/d27/d1d/f1e [0,4194304] 0 2024-04-12T05:08:57.213 INFO:tasks.workunit.client.1.smithi192.stdout:0/229: chown d0/c9 482637142 1 2024-04-12T05:08:57.213 INFO:tasks.workunit.client.0.smithi113.stdout:5/343: fdatasync d0/f6 0 2024-04-12T05:08:57.213 INFO:tasks.workunit.client.1.smithi192.stdout:6/395: dwrite d3/d8/f1c [0,4194304] 0 2024-04-12T05:08:57.214 INFO:tasks.workunit.client.1.smithi192.stdout:4/361: getdents d1/d54/d55/d64 0 2024-04-12T05:08:57.215 INFO:tasks.workunit.client.0.smithi113.stdout:4/414: dread da/de/f62 [0,4194304] 0 2024-04-12T05:08:57.215 INFO:tasks.workunit.client.1.smithi192.stdout:1/225: truncate d0/d2/f7 287796 0 2024-04-12T05:08:57.223 INFO:tasks.workunit.client.0.smithi113.stdout:4/415: creat da/dd/f84 x:0 0 0 2024-04-12T05:08:57.229 INFO:tasks.workunit.client.0.smithi113.stdout:5/344: truncate d0/d10/d27/d5f/f65 396746 0 2024-04-12T05:08:57.234 INFO:tasks.workunit.client.0.smithi113.stdout:5/345: dread d0/d10/d27/f22 [0,4194304] 0 2024-04-12T05:08:57.235 INFO:tasks.workunit.client.0.smithi113.stdout:5/346: dread - d0/d10/d27/f57 zero size 2024-04-12T05:08:57.236 INFO:tasks.workunit.client.0.smithi113.stdout:5/347: write d0/d10/d27/d1d/d3f/f6a [333929,6959] 0 2024-04-12T05:08:57.242 INFO:tasks.workunit.client.0.smithi113.stdout:4/416: dread da/d3c/d43/f7d [0,4194304] 0 2024-04-12T05:08:57.244 INFO:tasks.workunit.client.0.smithi113.stdout:4/417: symlink da/d4a/l85 0 2024-04-12T05:08:57.244 INFO:tasks.workunit.client.0.smithi113.stdout:4/418: stat da/d3c/d83/l41 0 2024-04-12T05:08:57.247 INFO:tasks.workunit.client.0.smithi113.stdout:4/419: rename da/c31 to da/d24/d7e/d79/c86 0 2024-04-12T05:08:57.250 INFO:tasks.workunit.client.0.smithi113.stdout:4/420: symlink da/d24/d7b/l87 0 2024-04-12T05:08:57.252 INFO:tasks.workunit.client.0.smithi113.stdout:4/421: creat da/dd/d3a/f88 x:0 0 0 2024-04-12T05:08:57.255 INFO:tasks.workunit.client.0.smithi113.stdout:4/422: dread da/d4f/f4e [0,4194304] 0 2024-04-12T05:08:57.257 INFO:tasks.workunit.client.0.smithi113.stdout:4/423: rmdir da/d3c/d83 39 2024-04-12T05:08:57.262 INFO:tasks.workunit.client.0.smithi113.stdout:4/424: dwrite da/d3c/d43/f75 [0,4194304] 0 2024-04-12T05:08:57.267 INFO:tasks.workunit.client.0.smithi113.stdout:4/425: link da/dd/c15 da/d70/d80/c89 0 2024-04-12T05:08:57.270 INFO:tasks.workunit.client.0.smithi113.stdout:4/426: dread da/d3c/d43/f7d [0,4194304] 0 2024-04-12T05:08:57.277 INFO:tasks.workunit.client.0.smithi113.stdout:4/427: unlink da/d3c/d83/f4c 0 2024-04-12T05:08:57.280 INFO:tasks.workunit.client.0.smithi113.stdout:4/428: creat da/d3c/d83/f8a x:0 0 0 2024-04-12T05:08:57.282 INFO:tasks.workunit.client.0.smithi113.stdout:4/429: creat da/d70/d80/f8b x:0 0 0 2024-04-12T05:08:57.283 INFO:tasks.workunit.client.0.smithi113.stdout:4/430: chown da/d3c/d83/f8a 6876622 1 2024-04-12T05:08:57.332 INFO:tasks.workunit.client.1.smithi192.stdout:1/226: write d0/d2/dd/d21/f2d [2139629,84221] 0 2024-04-12T05:08:57.334 INFO:tasks.workunit.client.1.smithi192.stdout:9/248: dread d11/d25/d36/f38 [0,4194304] 0 2024-04-12T05:08:57.337 INFO:tasks.workunit.client.0.smithi113.stdout:0/432: sync 2024-04-12T05:08:57.337 INFO:tasks.workunit.client.0.smithi113.stdout:7/418: sync 2024-04-12T05:08:57.337 INFO:tasks.workunit.client.0.smithi113.stdout:3/366: sync 2024-04-12T05:08:57.337 INFO:tasks.workunit.client.1.smithi192.stdout:9/249: chown d11/f47 7 1 2024-04-12T05:08:57.339 INFO:tasks.workunit.client.1.smithi192.stdout:0/230: getdents d0 0 2024-04-12T05:08:57.341 INFO:tasks.workunit.client.1.smithi192.stdout:6/396: dread d3/d8/f17 [0,4194304] 0 2024-04-12T05:08:57.343 INFO:tasks.workunit.client.0.smithi113.stdout:3/367: write d0/d13/d15/d28/d2a/d34/f72 [733640,22753] 0 2024-04-12T05:08:57.344 INFO:tasks.workunit.client.1.smithi192.stdout:4/362: dwrite d1/d1d/d53/d1c/f51 [0,4194304] 0 2024-04-12T05:08:57.346 INFO:tasks.workunit.client.1.smithi192.stdout:9/250: creat d11/d37/d33/f55 x:0 0 0 2024-04-12T05:08:57.346 INFO:tasks.workunit.client.1.smithi192.stdout:0/231: rmdir d0/d6/d15/d55/d2e 39 2024-04-12T05:08:57.347 INFO:tasks.workunit.client.1.smithi192.stdout:0/232: chown d0/l2c 2419802 1 2024-04-12T05:08:57.347 INFO:tasks.workunit.client.1.smithi192.stdout:5/307: dwrite d0/f1b [0,4194304] 0 2024-04-12T05:08:57.348 INFO:tasks.workunit.client.0.smithi113.stdout:0/433: dread d1/d6/f58 [0,4194304] 0 2024-04-12T05:08:57.348 INFO:tasks.workunit.client.0.smithi113.stdout:7/419: dwrite d0/d52/f57 [0,4194304] 0 2024-04-12T05:08:57.348 INFO:tasks.workunit.client.1.smithi192.stdout:1/227: dread d0/d2/dd/f15 [0,4194304] 0 2024-04-12T05:08:57.348 INFO:tasks.workunit.client.0.smithi113.stdout:0/434: dread - d1/d6/f64 zero size 2024-04-12T05:08:57.350 INFO:tasks.workunit.client.0.smithi113.stdout:7/420: write d0/dc/f41 [357468,4400] 0 2024-04-12T05:08:57.350 INFO:tasks.workunit.client.1.smithi192.stdout:0/233: mkdir d0/d4e/d58 0 2024-04-12T05:08:57.353 INFO:tasks.workunit.client.1.smithi192.stdout:5/308: creat d0/f6e x:0 0 0 2024-04-12T05:08:57.355 INFO:tasks.workunit.client.0.smithi113.stdout:3/368: creat d0/d13/f7d x:0 0 0 2024-04-12T05:08:57.361 INFO:tasks.workunit.client.0.smithi113.stdout:7/421: dwrite d0/d52/f54 [0,4194304] 0 2024-04-12T05:08:57.361 INFO:tasks.workunit.client.1.smithi192.stdout:9/251: dwrite d11/f47 [0,4194304] 0 2024-04-12T05:08:57.361 INFO:tasks.workunit.client.1.smithi192.stdout:0/234: fdatasync d0/d6/d15/f3d 0 2024-04-12T05:08:57.361 INFO:tasks.workunit.client.1.smithi192.stdout:5/309: symlink d0/d15/d32/d45/l6f 0 2024-04-12T05:08:57.361 INFO:tasks.workunit.client.1.smithi192.stdout:1/228: creat d0/d2/dd/d3c/f41 x:0 0 0 2024-04-12T05:08:57.363 INFO:tasks.workunit.client.0.smithi113.stdout:0/435: creat d1/d6/d16/d2a/d50/f85 x:0 0 0 2024-04-12T05:08:57.364 INFO:tasks.workunit.client.0.smithi113.stdout:0/436: read - d1/d43/d59/f82 zero size 2024-04-12T05:08:57.365 INFO:tasks.workunit.client.0.smithi113.stdout:7/422: rmdir d0/dc/d6/d1a 39 2024-04-12T05:08:57.365 INFO:tasks.workunit.client.1.smithi192.stdout:9/252: read - d11/d25/f52 zero size 2024-04-12T05:08:57.365 INFO:tasks.workunit.client.0.smithi113.stdout:0/437: readlink d1/d43/d59/l72 0 2024-04-12T05:08:57.366 INFO:tasks.workunit.client.1.smithi192.stdout:4/363: rename d1/d1d/d53/d1c/f51 to d1/d54/d55/f72 0 2024-04-12T05:08:57.367 INFO:tasks.workunit.client.0.smithi113.stdout:7/423: fsync d0/dd/d11/f8c 0 2024-04-12T05:08:57.373 INFO:tasks.workunit.client.0.smithi113.stdout:7/424: dread - d0/dc/d6/d1a/f7e zero size 2024-04-12T05:08:57.374 INFO:tasks.workunit.client.0.smithi113.stdout:0/438: dwrite d1/d6/d16/d2a/d50/f65 [0,4194304] 0 2024-04-12T05:08:57.374 INFO:tasks.workunit.client.1.smithi192.stdout:5/310: creat d0/d15/d64/f70 x:0 0 0 2024-04-12T05:08:57.377 INFO:tasks.workunit.client.1.smithi192.stdout:0/235: dwrite d0/d6/d15/d1b/d3a/d4f/f24 [4194304,4194304] 0 2024-04-12T05:08:57.379 INFO:tasks.workunit.client.0.smithi113.stdout:7/425: write d0/dc/d50/f72 [310658,43495] 0 2024-04-12T05:08:57.379 INFO:tasks.workunit.client.0.smithi113.stdout:0/439: stat d1/d6/d16/d2a/c2e 0 2024-04-12T05:08:57.380 INFO:tasks.workunit.client.1.smithi192.stdout:1/229: link d0/d2/f34 d0/d2/dd/d21/d33/f42 0 2024-04-12T05:08:57.385 INFO:tasks.workunit.client.0.smithi113.stdout:7/426: dread d0/dd/f1c [0,4194304] 0 2024-04-12T05:08:57.386 INFO:tasks.workunit.client.1.smithi192.stdout:1/230: creat d0/d2/dd/d21/f43 x:0 0 0 2024-04-12T05:08:57.387 INFO:tasks.workunit.client.1.smithi192.stdout:5/311: dwrite d0/d15/d24/d47/f6b [0,4194304] 0 2024-04-12T05:08:57.391 INFO:tasks.workunit.client.0.smithi113.stdout:0/440: rename d1/d6/d16/d2a/c35 to d1/d80/c86 0 2024-04-12T05:08:57.391 INFO:tasks.workunit.client.1.smithi192.stdout:1/231: mkdir d0/d1b/d44 0 2024-04-12T05:08:57.392 INFO:tasks.workunit.client.0.smithi113.stdout:0/441: fdatasync d1/d6/d34/f40 0 2024-04-12T05:08:57.393 INFO:tasks.workunit.client.1.smithi192.stdout:1/232: fdatasync d0/f18 0 2024-04-12T05:08:57.393 INFO:tasks.workunit.client.0.smithi113.stdout:0/442: dread - d1/d6/d16/d2a/f71 zero size 2024-04-12T05:08:57.393 INFO:tasks.workunit.client.0.smithi113.stdout:7/427: symlink d0/dd/l8e 0 2024-04-12T05:08:57.394 INFO:tasks.workunit.client.1.smithi192.stdout:0/236: dwrite d0/d6/d15/d1b/d46/f49 [0,4194304] 0 2024-04-12T05:08:57.394 INFO:tasks.workunit.client.1.smithi192.stdout:1/233: fdatasync d0/d1b/f35 0 2024-04-12T05:08:57.397 INFO:tasks.workunit.client.0.smithi113.stdout:0/443: mkdir d1/d9/d87 0 2024-04-12T05:08:57.398 INFO:tasks.workunit.client.1.smithi192.stdout:0/237: rmdir d0/d6/d15/d55 39 2024-04-12T05:08:57.398 INFO:tasks.workunit.client.1.smithi192.stdout:1/234: write d0/d2/dd/d21/f43 [218228,67240] 0 2024-04-12T05:08:57.398 INFO:tasks.workunit.client.1.smithi192.stdout:5/312: getdents d0/d15/d24 0 2024-04-12T05:08:57.400 INFO:tasks.workunit.client.0.smithi113.stdout:0/444: symlink d1/d43/l88 0 2024-04-12T05:08:57.403 INFO:tasks.workunit.client.0.smithi113.stdout:3/369: write d0/d13/d15/f2e [1383409,33302] 0 2024-04-12T05:08:57.407 INFO:tasks.workunit.client.0.smithi113.stdout:0/445: dwrite d1/d6/f30 [0,4194304] 0 2024-04-12T05:08:57.409 INFO:tasks.workunit.client.0.smithi113.stdout:7/428: getdents d0/dd/d11/d44/d5c 0 2024-04-12T05:08:57.410 INFO:tasks.workunit.client.1.smithi192.stdout:1/235: unlink d0/f3 0 2024-04-12T05:08:57.410 INFO:tasks.workunit.client.0.smithi113.stdout:3/370: mkdir d0/d7e 0 2024-04-12T05:08:57.412 INFO:tasks.workunit.client.0.smithi113.stdout:7/429: mknod d0/dd/d11/d44/c8f 0 2024-04-12T05:08:57.413 INFO:tasks.workunit.client.1.smithi192.stdout:5/313: creat d0/d15/d3e/f71 x:0 0 0 2024-04-12T05:08:57.415 INFO:tasks.workunit.client.0.smithi113.stdout:7/430: dread d0/dc/d6/f28 [0,4194304] 0 2024-04-12T05:08:57.415 INFO:tasks.workunit.client.1.smithi192.stdout:1/236: creat d0/d2/d1e/f45 x:0 0 0 2024-04-12T05:08:57.416 INFO:tasks.workunit.client.0.smithi113.stdout:3/371: rename d0/l71 to d0/d13/d47/l7f 0 2024-04-12T05:08:57.417 INFO:tasks.workunit.client.0.smithi113.stdout:3/372: dread - d0/d13/f60 zero size 2024-04-12T05:08:57.417 INFO:tasks.workunit.client.1.smithi192.stdout:1/237: symlink d0/d29/l46 0 2024-04-12T05:08:57.418 INFO:tasks.workunit.client.0.smithi113.stdout:3/373: chown d0/d13/d15/d28/d2a/d34/l5a 30649 1 2024-04-12T05:08:57.419 INFO:tasks.workunit.client.0.smithi113.stdout:7/431: mknod d0/dc/d6/d78/c90 0 2024-04-12T05:08:57.421 INFO:tasks.workunit.client.1.smithi192.stdout:1/238: truncate d0/d2/dd/d3c/f41 823935 0 2024-04-12T05:08:57.422 INFO:tasks.workunit.client.1.smithi192.stdout:1/239: readlink d0/d29/l46 0 2024-04-12T05:08:57.422 INFO:tasks.workunit.client.1.smithi192.stdout:0/238: rename d0/d6/d15/d55/d2e/d47 to d0/d6/d15/d55/d2e/d59 0 2024-04-12T05:08:57.423 INFO:tasks.workunit.client.1.smithi192.stdout:4/364: dread d1/d1d/d53/d1c/f36 [0,4194304] 0 2024-04-12T05:08:57.425 INFO:tasks.workunit.client.0.smithi113.stdout:7/432: fdatasync d0/dc/d6/d78/f38 0 2024-04-12T05:08:57.425 INFO:tasks.workunit.client.1.smithi192.stdout:5/314: write d0/f52 [1742385,65722] 0 2024-04-12T05:08:57.425 INFO:tasks.workunit.client.0.smithi113.stdout:7/433: read - d0/dc/d6/d78/f7c zero size 2024-04-12T05:08:57.427 INFO:tasks.workunit.client.0.smithi113.stdout:7/434: chown d0/dd/d11/d44/c60 1 1 2024-04-12T05:08:57.428 INFO:tasks.workunit.client.0.smithi113.stdout:3/374: getdents d0/d13/d15/d28/d2a 0 2024-04-12T05:08:57.428 INFO:tasks.workunit.client.1.smithi192.stdout:0/239: rename d0/fb to d0/d6/d15/d1b/d3a/d4f/f5a 0 2024-04-12T05:08:57.432 INFO:tasks.workunit.client.0.smithi113.stdout:3/375: mkdir d0/d80 0 2024-04-12T05:08:57.432 INFO:tasks.workunit.client.0.smithi113.stdout:7/435: creat d0/dd/d11/f91 x:0 0 0 2024-04-12T05:08:57.434 INFO:tasks.workunit.client.1.smithi192.stdout:5/315: rename d0/ld to d0/d15/d32/d45/l72 0 2024-04-12T05:08:57.436 INFO:tasks.workunit.client.0.smithi113.stdout:7/436: mknod d0/dc/c92 0 2024-04-12T05:08:57.438 INFO:tasks.workunit.client.0.smithi113.stdout:3/376: creat d0/d13/f81 x:0 0 0 2024-04-12T05:08:57.438 INFO:tasks.workunit.client.0.smithi113.stdout:7/437: truncate d0/dc/d6/d1a/f58 1426834 0 2024-04-12T05:08:57.439 INFO:tasks.workunit.client.0.smithi113.stdout:3/377: fdatasync d0/d13/d15/d28/f6e 0 2024-04-12T05:08:57.441 INFO:tasks.workunit.client.0.smithi113.stdout:7/438: mknod d0/dd/c93 0 2024-04-12T05:08:57.443 INFO:tasks.workunit.client.1.smithi192.stdout:0/240: readlink d0/d6/d15/d55/d29/l43 0 2024-04-12T05:08:57.444 INFO:tasks.workunit.client.0.smithi113.stdout:7/439: truncate d0/dd/d1b/d80/f8a 956006 0 2024-04-12T05:08:57.444 INFO:tasks.workunit.client.1.smithi192.stdout:0/241: chown d0/d6/d15/d1b/d3a/d4f/f5a 14 1 2024-04-12T05:08:57.444 INFO:tasks.workunit.client.1.smithi192.stdout:5/316: write d0/d15/d24/f4b [3732247,53912] 0 2024-04-12T05:08:57.447 INFO:tasks.workunit.client.0.smithi113.stdout:3/378: dwrite d0/d13/f81 [0,4194304] 0 2024-04-12T05:08:57.449 INFO:tasks.workunit.client.1.smithi192.stdout:5/317: mknod d0/d15/d64/c73 0 2024-04-12T05:08:57.449 INFO:tasks.workunit.client.1.smithi192.stdout:4/365: dwrite d1/f4b [0,4194304] 0 2024-04-12T05:08:57.451 INFO:tasks.workunit.client.1.smithi192.stdout:0/242: write d0/d6/d15/d34/f38 [668968,55057] 0 2024-04-12T05:08:57.452 INFO:tasks.workunit.client.0.smithi113.stdout:3/379: symlink d0/d7e/l82 0 2024-04-12T05:08:57.453 INFO:tasks.workunit.client.0.smithi113.stdout:7/440: link d0/dc/l4 d0/dc/d6/d24/l94 0 2024-04-12T05:08:57.454 INFO:tasks.workunit.client.1.smithi192.stdout:5/318: chown d0/d15/d24/d29/c40 27 1 2024-04-12T05:08:57.455 INFO:tasks.workunit.client.0.smithi113.stdout:7/441: truncate d0/dc/d6/d24/f71 526480 0 2024-04-12T05:08:57.455 INFO:tasks.workunit.client.1.smithi192.stdout:0/243: write d0/d6/d15/d19/f44 [2010010,124796] 0 2024-04-12T05:08:57.457 INFO:tasks.workunit.client.0.smithi113.stdout:3/380: rmdir d0/d13/d15 39 2024-04-12T05:08:57.458 INFO:tasks.workunit.client.1.smithi192.stdout:5/319: fdatasync d0/d15/d24/d47/f4d 0 2024-04-12T05:08:57.458 INFO:tasks.workunit.client.1.smithi192.stdout:4/366: dread d1/d1d/f44 [0,4194304] 0 2024-04-12T05:08:57.460 INFO:tasks.workunit.client.1.smithi192.stdout:5/320: write d0/f7 [1911292,123628] 0 2024-04-12T05:08:57.460 INFO:tasks.workunit.client.1.smithi192.stdout:0/244: symlink d0/d4e/d53/l5b 0 2024-04-12T05:08:57.464 INFO:tasks.workunit.client.0.smithi113.stdout:3/381: symlink d0/d7e/l83 0 2024-04-12T05:08:57.464 INFO:tasks.workunit.client.0.smithi113.stdout:7/442: mkdir d0/dd/d95 0 2024-04-12T05:08:57.464 INFO:tasks.workunit.client.0.smithi113.stdout:3/382: readlink d0/l16 0 2024-04-12T05:08:57.466 INFO:tasks.workunit.client.1.smithi192.stdout:0/245: creat d0/d6/d15/d55/d2e/f5c x:0 0 0 2024-04-12T05:08:57.469 INFO:tasks.workunit.client.1.smithi192.stdout:0/246: readlink d0/d6/d15/d34/d4b/l54 0 2024-04-12T05:08:57.471 INFO:tasks.workunit.client.1.smithi192.stdout:4/367: dwrite d1/d1d/d53/d1c/f36 [0,4194304] 0 2024-04-12T05:08:57.473 INFO:tasks.workunit.client.0.smithi113.stdout:3/383: dwrite d0/d13/f7d [0,4194304] 0 2024-04-12T05:08:57.474 INFO:tasks.workunit.client.0.smithi113.stdout:7/443: dwrite d0/dc/d6/f28 [0,4194304] 0 2024-04-12T05:08:57.477 INFO:tasks.workunit.client.0.smithi113.stdout:1/510: sync 2024-04-12T05:08:57.477 INFO:tasks.workunit.client.0.smithi113.stdout:9/455: sync 2024-04-12T05:08:57.479 INFO:tasks.workunit.client.1.smithi192.stdout:3/256: sync 2024-04-12T05:08:57.479 INFO:tasks.workunit.client.1.smithi192.stdout:8/218: sync 2024-04-12T05:08:57.479 INFO:tasks.workunit.client.1.smithi192.stdout:7/221: sync 2024-04-12T05:08:57.479 INFO:tasks.workunit.client.0.smithi113.stdout:9/456: dread - d1/df/d29/f7b zero size 2024-04-12T05:08:57.481 INFO:tasks.workunit.client.1.smithi192.stdout:8/219: unlink d6/l27 0 2024-04-12T05:08:57.481 INFO:tasks.workunit.client.1.smithi192.stdout:0/247: dread d0/d6/d15/f23 [0,4194304] 0 2024-04-12T05:08:57.481 INFO:tasks.workunit.client.0.smithi113.stdout:7/444: rename d0/dd/d11/f91 to d0/dd/d1b/d80/f96 0 2024-04-12T05:08:57.482 INFO:tasks.workunit.client.0.smithi113.stdout:9/457: stat d1/d27/c85 0 2024-04-12T05:08:57.482 INFO:tasks.workunit.client.1.smithi192.stdout:4/368: rename d1/d1d/l67 to d1/d6b/l73 0 2024-04-12T05:08:57.485 INFO:tasks.workunit.client.0.smithi113.stdout:1/511: mknod d10/d21/ca1 0 2024-04-12T05:08:57.485 INFO:tasks.workunit.client.1.smithi192.stdout:5/321: dwrite d0/d21/f33 [0,4194304] 0 2024-04-12T05:08:57.486 INFO:tasks.workunit.client.1.smithi192.stdout:0/248: write d0/d6/d15/d19/f30 [553024,52134] 0 2024-04-12T05:08:57.486 INFO:tasks.workunit.client.0.smithi113.stdout:9/458: fdatasync d1/d27/f5d 0 2024-04-12T05:08:57.487 INFO:tasks.workunit.client.1.smithi192.stdout:3/257: write d5/d13/f2a [3368705,75522] 0 2024-04-12T05:08:57.488 INFO:tasks.workunit.client.1.smithi192.stdout:7/222: creat d3/d7/d14/f58 x:0 0 0 2024-04-12T05:08:57.488 INFO:tasks.workunit.client.1.smithi192.stdout:4/369: chown d1/d54/d55/d37/c4d 859149447 1 2024-04-12T05:08:57.489 INFO:tasks.workunit.client.0.smithi113.stdout:7/445: symlink d0/dc/d6/l97 0 2024-04-12T05:08:57.489 INFO:tasks.workunit.client.0.smithi113.stdout:9/459: write d1/df/d4b/d6c/f59 [687123,85494] 0 2024-04-12T05:08:57.490 INFO:tasks.workunit.client.1.smithi192.stdout:5/322: chown d0/d15/d64/c73 2 1 2024-04-12T05:08:57.491 INFO:tasks.workunit.client.1.smithi192.stdout:0/249: truncate d0/f16 2774164 0 2024-04-12T05:08:57.491 INFO:tasks.workunit.client.0.smithi113.stdout:3/384: creat d0/d13/d15/d28/d2a/d34/f84 x:0 0 0 2024-04-12T05:08:57.493 INFO:tasks.workunit.client.1.smithi192.stdout:3/258: creat d5/d13/d3a/d4a/f5a x:0 0 0 2024-04-12T05:08:57.493 INFO:tasks.workunit.client.0.smithi113.stdout:1/512: unlink d10/d2e/d31/d33/c80 0 2024-04-12T05:08:57.494 INFO:tasks.workunit.client.1.smithi192.stdout:5/323: readlink d0/d15/d24/l3c 0 2024-04-12T05:08:57.495 INFO:tasks.workunit.client.0.smithi113.stdout:1/513: write d10/d2e/d7c/f92 [833204,11779] 0 2024-04-12T05:08:57.496 INFO:tasks.workunit.client.0.smithi113.stdout:9/460: creat d1/df/d29/fa2 x:0 0 0 2024-04-12T05:08:57.497 INFO:tasks.workunit.client.1.smithi192.stdout:0/250: write d0/d6/d15/d1b/d46/f49 [1454197,66893] 0 2024-04-12T05:08:57.499 INFO:tasks.workunit.client.1.smithi192.stdout:6/397: sync 2024-04-12T05:08:57.499 INFO:tasks.workunit.client.1.smithi192.stdout:9/253: sync 2024-04-12T05:08:57.499 INFO:tasks.workunit.client.1.smithi192.stdout:2/337: sync 2024-04-12T05:08:57.503 INFO:tasks.workunit.client.1.smithi192.stdout:4/370: read d1/d1d/f6c [334912,38548] 0 2024-04-12T05:08:57.504 INFO:tasks.workunit.client.0.smithi113.stdout:9/461: mknod d1/d4/d5/d22/ca3 0 2024-04-12T05:08:57.505 INFO:tasks.workunit.client.0.smithi113.stdout:1/514: mknod d10/d41/d89/d96/d7d/d9f/ca2 0 2024-04-12T05:08:57.505 INFO:tasks.workunit.client.1.smithi192.stdout:8/220: dwrite d6/df/f16 [0,4194304] 0 2024-04-12T05:08:57.506 INFO:tasks.workunit.client.0.smithi113.stdout:1/515: chown d10/d2e/d31/d5c/l8e 16163 1 2024-04-12T05:08:57.507 INFO:tasks.workunit.client.0.smithi113.stdout:1/516: dread - d10/d41/f88 zero size 2024-04-12T05:08:57.508 INFO:tasks.workunit.client.0.smithi113.stdout:9/462: symlink d1/df/d4b/d63/d3d/d6f/d7e/la4 0 2024-04-12T05:08:57.509 INFO:tasks.workunit.client.0.smithi113.stdout:9/463: write d1/d4/d38/f8c [1456380,62712] 0 2024-04-12T05:08:57.513 INFO:tasks.workunit.client.1.smithi192.stdout:2/338: creat d1/d2d/d73/f74 x:0 0 0 2024-04-12T05:08:57.513 INFO:tasks.workunit.client.0.smithi113.stdout:1/517: creat d10/d41/d5b/fa3 x:0 0 0 2024-04-12T05:08:57.513 INFO:tasks.workunit.client.1.smithi192.stdout:0/251: mknod d0/d6/d15/d1b/d3a/d4f/d33/c5d 0 2024-04-12T05:08:57.514 INFO:tasks.workunit.client.0.smithi113.stdout:9/464: symlink d1/df/d4b/d6c/la5 0 2024-04-12T05:08:57.516 INFO:tasks.workunit.client.1.smithi192.stdout:5/324: dread d0/d15/d24/f46 [0,4194304] 0 2024-04-12T05:08:57.517 INFO:tasks.workunit.client.0.smithi113.stdout:1/518: mknod d10/d2e/d7c/ca4 0 2024-04-12T05:08:57.518 INFO:tasks.workunit.client.0.smithi113.stdout:3/385: getdents d0/d13/d15/d28/d2a/d34 0 2024-04-12T05:08:57.520 INFO:tasks.workunit.client.1.smithi192.stdout:8/221: truncate d6/f18 1463631 0 2024-04-12T05:08:57.522 INFO:tasks.workunit.client.0.smithi113.stdout:1/519: write d10/d41/f88 [34351,80743] 0 2024-04-12T05:08:57.523 INFO:tasks.workunit.client.1.smithi192.stdout:4/371: truncate d1/d54/d55/d37/f48 736130 0 2024-04-12T05:08:57.523 INFO:tasks.workunit.client.1.smithi192.stdout:0/252: write d0/d6/d15/d19/f4c [952151,85216] 0 2024-04-12T05:08:57.527 INFO:tasks.workunit.client.1.smithi192.stdout:6/398: dwrite d3/d8/d48/f6f [0,4194304] 0 2024-04-12T05:08:57.528 INFO:tasks.workunit.client.0.smithi113.stdout:7/446: dwrite d0/dd/d2d/f53 [0,4194304] 0 2024-04-12T05:08:57.530 INFO:tasks.workunit.client.0.smithi113.stdout:9/465: truncate d1/d4/f18 2389991 0 2024-04-12T05:08:57.533 INFO:tasks.workunit.client.1.smithi192.stdout:5/325: write d0/f11 [2276581,115227] 0 2024-04-12T05:08:57.536 INFO:tasks.workunit.client.0.smithi113.stdout:1/520: unlink d10/d2e/d7c/c8d 0 2024-04-12T05:08:57.536 INFO:tasks.workunit.client.1.smithi192.stdout:3/259: dwrite d5/d13/d1a/d53/f58 [0,4194304] 0 2024-04-12T05:08:57.538 INFO:tasks.workunit.client.1.smithi192.stdout:0/253: chown d0/c52 1146884 1 2024-04-12T05:08:57.538 INFO:tasks.workunit.client.1.smithi192.stdout:3/260: stat d5/fd 0 2024-04-12T05:08:57.538 INFO:tasks.workunit.client.1.smithi192.stdout:5/326: truncate d0/d15/d32/d45/f65 1041098 0 2024-04-12T05:08:57.541 INFO:tasks.workunit.client.0.smithi113.stdout:3/386: link d0/d13/d15/d28/f45 d0/d7e/f85 0 2024-04-12T05:08:57.542 INFO:tasks.workunit.client.0.smithi113.stdout:3/387: read - d0/d13/d15/d28/f6e zero size 2024-04-12T05:08:57.544 INFO:tasks.workunit.client.1.smithi192.stdout:7/223: dread d3/d11/f3b [0,4194304] 0 2024-04-12T05:08:57.545 INFO:tasks.workunit.client.1.smithi192.stdout:2/339: getdents d1/d8/d17/d19 0 2024-04-12T05:08:57.545 INFO:tasks.workunit.client.0.smithi113.stdout:1/521: mkdir d10/da5 0 2024-04-12T05:08:57.548 INFO:tasks.workunit.client.0.smithi113.stdout:3/388: dread d0/d13/d15/d28/d2a/f36 [0,4194304] 0 2024-04-12T05:08:57.548 INFO:tasks.workunit.client.1.smithi192.stdout:4/372: creat d1/d54/d55/d64/f74 x:0 0 0 2024-04-12T05:08:57.549 INFO:tasks.workunit.client.0.smithi113.stdout:1/522: creat d10/d14/d67/d70/d98/fa6 x:0 0 0 2024-04-12T05:08:57.549 INFO:tasks.workunit.client.1.smithi192.stdout:5/327: symlink d0/d21/l74 0 2024-04-12T05:08:57.550 INFO:tasks.workunit.client.0.smithi113.stdout:3/389: stat d0/d13/d15/f5c 0 2024-04-12T05:08:57.550 INFO:tasks.workunit.client.1.smithi192.stdout:7/224: unlink d3/d7/f9 0 2024-04-12T05:08:57.550 INFO:tasks.workunit.client.0.smithi113.stdout:1/523: write d10/d2e/f71 [3059161,58758] 0 2024-04-12T05:08:57.552 INFO:tasks.workunit.client.1.smithi192.stdout:0/254: mkdir d0/d5e 0 2024-04-12T05:08:57.553 INFO:tasks.workunit.client.1.smithi192.stdout:7/225: chown d3/d7/d15/d22/l32 0 1 2024-04-12T05:08:57.553 INFO:tasks.workunit.client.1.smithi192.stdout:4/373: mknod d1/d49/c75 0 2024-04-12T05:08:57.554 INFO:tasks.workunit.client.1.smithi192.stdout:2/340: rmdir d1/d8/d17/d19/d60/d59/d68 39 2024-04-12T05:08:57.554 INFO:tasks.workunit.client.0.smithi113.stdout:3/390: creat d0/d13/d15/d28/d2a/d34/d5f/f86 x:0 0 0 2024-04-12T05:08:57.557 INFO:tasks.workunit.client.1.smithi192.stdout:6/399: creat d3/dc/d39/f87 x:0 0 0 2024-04-12T05:08:57.558 INFO:tasks.workunit.client.0.smithi113.stdout:1/524: dwrite d10/d41/f4a [0,4194304] 0 2024-04-12T05:08:57.558 INFO:tasks.workunit.client.1.smithi192.stdout:7/226: fdatasync d3/d11/f2d 0 2024-04-12T05:08:57.559 INFO:tasks.workunit.client.0.smithi113.stdout:1/525: dread - d10/d41/d89/d96/f91 zero size 2024-04-12T05:08:57.560 INFO:tasks.workunit.client.1.smithi192.stdout:5/328: rename d0/d15/d32/d45/c48 to d0/d21/d5e/c75 0 2024-04-12T05:08:57.561 INFO:tasks.workunit.client.0.smithi113.stdout:3/391: rename d0/d13/d47/d4f/c77 to d0/d7e/c87 0 2024-04-12T05:08:57.562 INFO:tasks.workunit.client.0.smithi113.stdout:1/526: stat d10/d55/f8b 0 2024-04-12T05:08:57.562 INFO:tasks.workunit.client.0.smithi113.stdout:3/392: fsync d0/f18 0 2024-04-12T05:08:57.563 INFO:tasks.workunit.client.0.smithi113.stdout:3/393: fsync d0/d13/d15/f39 0 2024-04-12T05:08:57.566 INFO:tasks.workunit.client.1.smithi192.stdout:8/222: write d6/f18 [1935032,124444] 0 2024-04-12T05:08:57.568 INFO:tasks.workunit.client.1.smithi192.stdout:7/227: truncate d3/d7/d15/f3c 785293 0 2024-04-12T05:08:57.571 INFO:tasks.workunit.client.1.smithi192.stdout:0/255: write d0/d6/d15/f3c [10780,128821] 0 2024-04-12T05:08:57.573 INFO:tasks.workunit.client.0.smithi113.stdout:3/394: rename d0/d13/d15/d28/c2c to d0/d13/d15/d28/c88 0 2024-04-12T05:08:57.575 INFO:tasks.workunit.client.1.smithi192.stdout:7/228: rmdir d3/d7/d15/d22 39 2024-04-12T05:08:57.577 INFO:tasks.workunit.client.1.smithi192.stdout:9/254: dread fb [0,4194304] 0 2024-04-12T05:08:57.579 INFO:tasks.workunit.client.0.smithi113.stdout:3/395: creat d0/d13/d47/f89 x:0 0 0 2024-04-12T05:08:57.579 INFO:tasks.workunit.client.1.smithi192.stdout:6/400: dwrite d3/d23/f74 [0,4194304] 0 2024-04-12T05:08:57.580 INFO:tasks.workunit.client.1.smithi192.stdout:8/223: truncate d6/df/d37/f4a 535924 0 2024-04-12T05:08:57.583 INFO:tasks.workunit.client.0.smithi113.stdout:3/396: symlink d0/d13/d47/d4f/l8a 0 2024-04-12T05:08:57.584 INFO:tasks.workunit.client.1.smithi192.stdout:5/329: rename d0/d15/d32/d69/c5f to d0/d15/d24/d47/c76 0 2024-04-12T05:08:57.585 INFO:tasks.workunit.client.1.smithi192.stdout:0/256: read d0/d6/d15/d19/f4c [1000383,101952] 0 2024-04-12T05:08:57.586 INFO:tasks.workunit.client.1.smithi192.stdout:2/341: dwrite d1/f6d [0,4194304] 0 2024-04-12T05:08:57.587 INFO:tasks.workunit.client.1.smithi192.stdout:4/374: truncate d1/d54/d55/f72 3021608 0 2024-04-12T05:08:57.587 INFO:tasks.workunit.client.1.smithi192.stdout:7/229: dread d3/d11/f16 [0,4194304] 0 2024-04-12T05:08:57.589 INFO:tasks.workunit.client.1.smithi192.stdout:6/401: symlink d3/l88 0 2024-04-12T05:08:57.590 INFO:tasks.workunit.client.0.smithi113.stdout:7/447: dwrite d0/dd/d1b/f5d [0,4194304] 0 2024-04-12T05:08:57.590 INFO:tasks.workunit.client.0.smithi113.stdout:1/527: truncate d10/d55/f8c 428816 0 2024-04-12T05:08:57.591 INFO:tasks.workunit.client.1.smithi192.stdout:3/261: dwrite d5/f1c [0,4194304] 0 2024-04-12T05:08:57.592 INFO:tasks.workunit.client.0.smithi113.stdout:9/466: dread d1/d4/f77 [0,4194304] 0 2024-04-12T05:08:57.592 INFO:tasks.workunit.client.1.smithi192.stdout:3/262: chown d5/d13/f2f 24232053 1 2024-04-12T05:08:57.594 INFO:tasks.workunit.client.0.smithi113.stdout:1/528: truncate d10/d2e/f4e 808989 0 2024-04-12T05:08:57.594 INFO:tasks.workunit.client.1.smithi192.stdout:7/230: unlink d3/ca 0 2024-04-12T05:08:57.594 INFO:tasks.workunit.client.0.smithi113.stdout:3/397: write d0/d13/f42 [714386,122153] 0 2024-04-12T05:08:57.601 INFO:tasks.workunit.client.1.smithi192.stdout:3/263: symlink d5/d13/d3c/l5b 0 2024-04-12T05:08:57.602 INFO:tasks.workunit.client.0.smithi113.stdout:6/338: sync 2024-04-12T05:08:57.602 INFO:tasks.workunit.client.0.smithi113.stdout:8/416: sync 2024-04-12T05:08:57.602 INFO:tasks.workunit.client.0.smithi113.stdout:2/282: sync 2024-04-12T05:08:57.604 INFO:tasks.workunit.client.1.smithi192.stdout:9/255: truncate d11/d15/d21/f2d 2495092 0 2024-04-12T05:08:57.607 INFO:tasks.workunit.client.0.smithi113.stdout:1/529: link d10/f13 d10/d2e/d31/d5c/fa7 0 2024-04-12T05:08:57.607 INFO:tasks.workunit.client.1.smithi192.stdout:2/342: dwrite d1/d8/d17/d19/f5e [0,4194304] 0 2024-04-12T05:08:57.607 INFO:tasks.workunit.client.1.smithi192.stdout:0/257: rename d0/d6/d15/d1b/d3a/d4f/l2f to d0/d6/d15/d1b/d3a/d4f/l5f 0 2024-04-12T05:08:57.608 INFO:tasks.workunit.client.0.smithi113.stdout:9/467: getdents d1/d4/d6d 0 2024-04-12T05:08:57.608 INFO:tasks.workunit.client.0.smithi113.stdout:3/398: dwrite d0/d13/f6a [0,4194304] 0 2024-04-12T05:08:57.609 INFO:tasks.workunit.client.1.smithi192.stdout:9/256: creat d11/d37/f56 x:0 0 0 2024-04-12T05:08:57.609 INFO:tasks.workunit.client.0.smithi113.stdout:1/530: chown d10/d14/d67/l8f 489 1 2024-04-12T05:08:57.610 INFO:tasks.workunit.client.1.smithi192.stdout:8/224: dread d6/d10/f3d [4194304,4194304] 0 2024-04-12T05:08:57.610 INFO:tasks.workunit.client.0.smithi113.stdout:6/339: mknod d8/de/d13/c74 0 2024-04-12T05:08:57.611 INFO:tasks.workunit.client.0.smithi113.stdout:2/283: dread - d8/d9/de/d12/f35 zero size 2024-04-12T05:08:57.612 INFO:tasks.workunit.client.1.smithi192.stdout:9/257: fdatasync fe 0 2024-04-12T05:08:57.612 INFO:tasks.workunit.client.1.smithi192.stdout:4/375: dwrite d1/d54/d55/d37/f5d [0,4194304] 0 2024-04-12T05:08:57.614 INFO:tasks.workunit.client.1.smithi192.stdout:8/225: dread - d6/df/f1c zero size 2024-04-12T05:08:57.614 INFO:tasks.workunit.client.1.smithi192.stdout:9/258: stat d11/f13 0 2024-04-12T05:08:57.615 INFO:tasks.workunit.client.0.smithi113.stdout:9/468: rename d1/df/d4b/d63/d3d/d6f/d7e to d1/df/d29/da6 0 2024-04-12T05:08:57.616 INFO:tasks.workunit.client.1.smithi192.stdout:0/258: readlink d0/l10 0 2024-04-12T05:08:57.616 INFO:tasks.workunit.client.1.smithi192.stdout:3/264: dread d5/d13/d1a/d53/f58 [0,4194304] 0 2024-04-12T05:08:57.617 INFO:tasks.workunit.client.1.smithi192.stdout:9/259: chown d11/d25/l29 1792 1 2024-04-12T05:08:57.618 INFO:tasks.workunit.client.0.smithi113.stdout:6/340: mknod d8/d45/c75 0 2024-04-12T05:08:57.618 INFO:tasks.workunit.client.0.smithi113.stdout:2/284: write d8/d9/de/d15/f40 [834900,47109] 0 2024-04-12T05:08:57.620 INFO:tasks.workunit.client.0.smithi113.stdout:1/531: rename d10/d2e/d7c/ca4 to d10/d14/d67/ca8 0 2024-04-12T05:08:57.621 INFO:tasks.workunit.client.0.smithi113.stdout:6/341: fsync d8/f12 0 2024-04-12T05:08:57.621 INFO:tasks.workunit.client.0.smithi113.stdout:9/469: mknod d1/df/d4b/d6c/ca7 0 2024-04-12T05:08:57.622 INFO:tasks.workunit.client.0.smithi113.stdout:8/417: dread d9/d16/d7f/d26/f66 [0,4194304] 0 2024-04-12T05:08:57.622 INFO:tasks.workunit.client.1.smithi192.stdout:2/343: mknod d1/d2d/d73/c75 0 2024-04-12T05:08:57.623 INFO:tasks.workunit.client.0.smithi113.stdout:2/285: rename d8/d9/de/d12/d44 to d8/d9/de/d12/d44/d53 22 2024-04-12T05:08:57.624 INFO:tasks.workunit.client.1.smithi192.stdout:3/265: rename d5/d13/c50 to d5/d13/d3a/d4e/c5c 0 2024-04-12T05:08:57.625 INFO:tasks.workunit.client.0.smithi113.stdout:1/532: chown d10/l4c 950178246 1 2024-04-12T05:08:57.625 INFO:tasks.workunit.client.0.smithi113.stdout:6/342: mkdir d8/de/d13/d2b/d76 0 2024-04-12T05:08:57.626 INFO:tasks.workunit.client.1.smithi192.stdout:9/260: symlink d11/d15/l57 0 2024-04-12T05:08:57.627 INFO:tasks.workunit.client.0.smithi113.stdout:9/470: creat d1/df/d4b/d63/d3d/d6f/fa8 x:0 0 0 2024-04-12T05:08:57.628 INFO:tasks.workunit.client.0.smithi113.stdout:8/418: mknod d9/d16/d1c/d59/c82 0 2024-04-12T05:08:57.628 INFO:tasks.workunit.client.1.smithi192.stdout:8/226: mkdir d6/d4b 0 2024-04-12T05:08:57.629 INFO:tasks.workunit.client.1.smithi192.stdout:8/227: fdatasync d6/df/d31/f36 0 2024-04-12T05:08:57.629 INFO:tasks.workunit.client.1.smithi192.stdout:3/266: dread d5/d13/d3c/d26/f44 [0,4194304] 0 2024-04-12T05:08:57.630 INFO:tasks.workunit.client.1.smithi192.stdout:4/376: truncate d1/d1d/d53/d1c/f66 1898925 0 2024-04-12T05:08:57.630 INFO:tasks.workunit.client.1.smithi192.stdout:3/267: dread - d5/d13/d3a/d4e/d57/f51 zero size 2024-04-12T05:08:57.631 INFO:tasks.workunit.client.0.smithi113.stdout:9/471: mknod d1/ca9 0 2024-04-12T05:08:57.633 INFO:tasks.workunit.client.0.smithi113.stdout:1/533: rename d10/d2e/d31/f59 to d10/d41/d89/d96/d7d/fa9 0 2024-04-12T05:08:57.635 INFO:tasks.workunit.client.1.smithi192.stdout:8/228: mkdir d6/d3e/d4c 0 2024-04-12T05:08:57.636 INFO:tasks.workunit.client.1.smithi192.stdout:8/229: dread - d6/d21/f33 zero size 2024-04-12T05:08:57.637 INFO:tasks.workunit.client.0.smithi113.stdout:2/286: dwrite d8/d9/de/d12/f4c [0,4194304] 0 2024-04-12T05:08:57.637 INFO:tasks.workunit.client.0.smithi113.stdout:1/534: dread - d10/d2e/d31/d33/f61 zero size 2024-04-12T05:08:57.638 INFO:tasks.workunit.client.1.smithi192.stdout:3/268: dwrite d5/d13/d3c/d36/f49 [0,4194304] 0 2024-04-12T05:08:57.639 INFO:tasks.workunit.client.0.smithi113.stdout:1/535: chown d10/d2e/f54 91952 1 2024-04-12T05:08:57.639 INFO:tasks.workunit.client.0.smithi113.stdout:8/419: rename d9/fd to d9/d16/d7f/d26/d32/d5b/f83 0 2024-04-12T05:08:57.640 INFO:tasks.workunit.client.0.smithi113.stdout:6/343: dwrite d8/de/d17/f72 [0,4194304] 0 2024-04-12T05:08:57.641 INFO:tasks.workunit.client.0.smithi113.stdout:8/420: truncate d9/d1b/f56 4839233 0 2024-04-12T05:08:57.642 INFO:tasks.workunit.client.0.smithi113.stdout:1/536: write d10/d14/d67/f69 [1128923,124636] 0 2024-04-12T05:08:57.645 INFO:tasks.workunit.client.1.smithi192.stdout:3/269: getdents d5/d13/d1a/d53 0 2024-04-12T05:08:57.646 INFO:tasks.workunit.client.0.smithi113.stdout:9/472: creat d1/d4/d5/d22/d9e/faa x:0 0 0 2024-04-12T05:08:57.648 INFO:tasks.workunit.client.1.smithi192.stdout:4/377: dread d1/f18 [0,4194304] 0 2024-04-12T05:08:57.649 INFO:tasks.workunit.client.1.smithi192.stdout:3/270: truncate d5/f2d 701736 0 2024-04-12T05:08:57.649 INFO:tasks.workunit.client.0.smithi113.stdout:2/287: creat d8/d9/de/d12/d2b/f54 x:0 0 0 2024-04-12T05:08:57.650 INFO:tasks.workunit.client.0.smithi113.stdout:2/288: chown d8/d9/lf 653354 1 2024-04-12T05:08:57.651 INFO:tasks.workunit.client.0.smithi113.stdout:6/344: mknod d8/de/d13/d11/d36/d3d/d4c/d26/c77 0 2024-04-12T05:08:57.653 INFO:tasks.workunit.client.1.smithi192.stdout:4/378: mkdir d1/d76 0 2024-04-12T05:08:57.654 INFO:tasks.workunit.client.0.smithi113.stdout:8/421: creat d9/d16/d7f/d26/f84 x:0 0 0 2024-04-12T05:08:57.654 INFO:tasks.workunit.client.0.smithi113.stdout:9/473: mkdir d1/df/d4b/d63/d3d/dab 0 2024-04-12T05:08:57.656 INFO:tasks.workunit.client.1.smithi192.stdout:4/379: truncate d1/d54/d55/d37/f48 1231153 0 2024-04-12T05:08:57.656 INFO:tasks.workunit.client.0.smithi113.stdout:9/474: chown d1/d4/f33 13 1 2024-04-12T05:08:57.657 INFO:tasks.workunit.client.0.smithi113.stdout:9/475: dread - d1/d4/d5/d17/f96 zero size 2024-04-12T05:08:57.659 INFO:tasks.workunit.client.1.smithi192.stdout:4/380: chown d1/d54/d55/f72 5597971 1 2024-04-12T05:08:57.660 INFO:tasks.workunit.client.0.smithi113.stdout:1/537: creat d10/d41/faa x:0 0 0 2024-04-12T05:08:57.663 INFO:tasks.workunit.client.0.smithi113.stdout:8/422: truncate d9/d16/d7f/d26/f66 583844 0 2024-04-12T05:08:57.666 INFO:tasks.workunit.client.0.smithi113.stdout:6/345: creat d8/de/d17/f78 x:0 0 0 2024-04-12T05:08:57.667 INFO:tasks.workunit.client.0.smithi113.stdout:1/538: rmdir d10/d55 39 2024-04-12T05:08:57.668 INFO:tasks.workunit.client.1.smithi192.stdout:8/230: dread f5 [0,4194304] 0 2024-04-12T05:08:57.668 INFO:tasks.workunit.client.0.smithi113.stdout:8/423: chown d9/c13 282954 1 2024-04-12T05:08:57.668 INFO:tasks.workunit.client.1.smithi192.stdout:8/231: chown d6/c25 4 1 2024-04-12T05:08:57.669 INFO:tasks.workunit.client.1.smithi192.stdout:8/232: read - d6/da/f49 zero size 2024-04-12T05:08:57.669 INFO:tasks.workunit.client.1.smithi192.stdout:4/381: symlink d1/l77 0 2024-04-12T05:08:57.671 INFO:tasks.workunit.client.0.smithi113.stdout:7/448: truncate d0/f5a 1572295 0 2024-04-12T05:08:57.672 INFO:tasks.workunit.client.1.smithi192.stdout:4/382: mknod d1/d1d/d53/c78 0 2024-04-12T05:08:57.673 INFO:tasks.workunit.client.0.smithi113.stdout:8/424: readlink d9/d16/l58 0 2024-04-12T05:08:57.673 INFO:tasks.workunit.client.0.smithi113.stdout:6/346: creat d8/de/d13/d11/d36/d3d/d4c/d26/d5b/f79 x:0 0 0 2024-04-12T05:08:57.673 INFO:tasks.workunit.client.1.smithi192.stdout:6/402: write d3/d8/f7e [3026336,69386] 0 2024-04-12T05:08:57.673 INFO:tasks.workunit.client.0.smithi113.stdout:9/476: dwrite d1/d4/f8f [0,4194304] 0 2024-04-12T05:08:57.675 INFO:tasks.workunit.client.1.smithi192.stdout:6/403: fsync d3/d8/d22/f65 0 2024-04-12T05:08:57.675 INFO:tasks.workunit.client.0.smithi113.stdout:9/477: chown d1/d4/d6d 2898 1 2024-04-12T05:08:57.676 INFO:tasks.workunit.client.1.smithi192.stdout:4/383: unlink d1/l5e 0 2024-04-12T05:08:57.679 INFO:tasks.workunit.client.0.smithi113.stdout:9/478: readlink d1/df/d4b/d6c/la5 0 2024-04-12T05:08:57.679 INFO:tasks.workunit.client.1.smithi192.stdout:6/404: dread d3/d8/f7e [0,4194304] 0 2024-04-12T05:08:57.682 INFO:tasks.workunit.client.0.smithi113.stdout:7/449: dwrite d0/f51 [0,4194304] 0 2024-04-12T05:08:57.682 INFO:tasks.workunit.client.1.smithi192.stdout:4/384: creat d1/d76/f79 x:0 0 0 2024-04-12T05:08:57.683 INFO:tasks.workunit.client.0.smithi113.stdout:9/479: read d1/d4/d5/d6/f3b [1631139,95109] 0 2024-04-12T05:08:57.683 INFO:tasks.workunit.client.0.smithi113.stdout:8/425: rmdir d9/d16/d1c/d59/d68 39 2024-04-12T05:08:57.686 INFO:tasks.workunit.client.0.smithi113.stdout:3/399: truncate d0/fa 3755981 0 2024-04-12T05:08:57.686 INFO:tasks.workunit.client.0.smithi113.stdout:1/539: truncate d10/d14/f2a 2098398 0 2024-04-12T05:08:57.687 INFO:tasks.workunit.client.0.smithi113.stdout:9/480: write d1/d4/d5/d6/d15/f81 [2427892,76749] 0 2024-04-12T05:08:57.687 INFO:tasks.workunit.client.1.smithi192.stdout:6/405: dread - d3/d21/f7d zero size 2024-04-12T05:08:57.687 INFO:tasks.workunit.client.1.smithi192.stdout:7/231: dwrite d3/d7/d14/d1f/f34 [0,4194304] 0 2024-04-12T05:08:57.691 INFO:tasks.workunit.client.0.smithi113.stdout:8/426: creat d9/d16/d7f/d26/d32/d48/f85 x:0 0 0 2024-04-12T05:08:57.691 INFO:tasks.workunit.client.0.smithi113.stdout:1/540: chown d10/d41/d89/d96/d7d/f7f 6708 1 2024-04-12T05:08:57.692 INFO:tasks.workunit.client.1.smithi192.stdout:2/344: truncate d1/d8/d3a/f3f 225136 0 2024-04-12T05:08:57.692 INFO:tasks.workunit.client.0.smithi113.stdout:7/450: rename d0/dd/d1b/d80/f8a to d0/dd/d11/d44/d5c/d85/f98 0 2024-04-12T05:08:57.693 INFO:tasks.workunit.client.1.smithi192.stdout:4/385: mknod d1/d54/d55/d64/d68/c7a 0 2024-04-12T05:08:57.696 INFO:tasks.workunit.client.0.smithi113.stdout:0/446: sync 2024-04-12T05:08:57.696 INFO:tasks.workunit.client.0.smithi113.stdout:4/431: sync 2024-04-12T05:08:57.696 INFO:tasks.workunit.client.0.smithi113.stdout:5/348: sync 2024-04-12T05:08:57.697 INFO:tasks.workunit.client.0.smithi113.stdout:4/432: fsync da/dd/d1b/f74 0 2024-04-12T05:08:57.698 INFO:tasks.workunit.client.0.smithi113.stdout:4/433: dread - da/dd/f84 zero size 2024-04-12T05:08:57.698 INFO:tasks.workunit.client.1.smithi192.stdout:7/232: creat d3/d7/f59 x:0 0 0 2024-04-12T05:08:57.699 INFO:tasks.workunit.client.1.smithi192.stdout:2/345: read - d1/d8/d17/f5f zero size 2024-04-12T05:08:57.699 INFO:tasks.workunit.client.1.smithi192.stdout:0/259: dwrite d0/d6/d15/f3d [0,4194304] 0 2024-04-12T05:08:57.700 INFO:tasks.workunit.client.0.smithi113.stdout:9/481: rmdir d1/d4/d38 39 2024-04-12T05:08:57.701 INFO:tasks.workunit.client.1.smithi192.stdout:6/406: write d3/d8/f17 [4680689,89483] 0 2024-04-12T05:08:57.701 INFO:tasks.workunit.client.0.smithi113.stdout:9/482: chown d1/d4/d6d 1078456 1 2024-04-12T05:08:57.702 INFO:tasks.workunit.client.1.smithi192.stdout:4/386: chown d1/d1d/d53 1 1 2024-04-12T05:08:57.703 INFO:tasks.workunit.client.1.smithi192.stdout:2/346: chown d1/d8/f35 422 1 2024-04-12T05:08:57.704 INFO:tasks.workunit.client.0.smithi113.stdout:9/483: read d1/df/d4b/f7f [126281,41494] 0 2024-04-12T05:08:57.704 INFO:tasks.workunit.client.1.smithi192.stdout:4/387: chown d1/d1d/d53/f52 1 1 2024-04-12T05:08:57.706 INFO:tasks.workunit.client.0.smithi113.stdout:4/434: dread da/d24/f53 [0,4194304] 0 2024-04-12T05:08:57.706 INFO:tasks.workunit.client.0.smithi113.stdout:4/435: fdatasync da/d73/f45 0 2024-04-12T05:08:57.708 INFO:tasks.workunit.client.1.smithi192.stdout:7/233: mkdir d3/d1d/d5a 0 2024-04-12T05:08:57.710 INFO:tasks.workunit.client.0.smithi113.stdout:7/451: mkdir d0/dd/d11/d99 0 2024-04-12T05:08:57.710 INFO:tasks.workunit.client.0.smithi113.stdout:0/447: symlink d1/d43/d59/l89 0 2024-04-12T05:08:57.710 INFO:tasks.workunit.client.1.smithi192.stdout:2/347: dread - d1/f4b zero size 2024-04-12T05:08:57.710 INFO:tasks.workunit.client.1.smithi192.stdout:4/388: write d1/f3e [3804104,25931] 0 2024-04-12T05:08:57.711 INFO:tasks.workunit.client.1.smithi192.stdout:6/407: write d3/d23/f74 [136611,91477] 0 2024-04-12T05:08:57.711 INFO:tasks.workunit.client.1.smithi192.stdout:2/348: chown d1/d2d/c6b 2329415 1 2024-04-12T05:08:57.713 INFO:tasks.workunit.client.0.smithi113.stdout:5/349: symlink d0/d10/d27/d1d/d38/d3b/l72 0 2024-04-12T05:08:57.714 INFO:tasks.workunit.client.0.smithi113.stdout:3/400: unlink d0/d13/d15/d28/c88 0 2024-04-12T05:08:57.715 INFO:tasks.workunit.client.1.smithi192.stdout:7/234: unlink d3/d1d/l43 0 2024-04-12T05:08:57.716 INFO:tasks.workunit.client.0.smithi113.stdout:3/401: chown d0/d13/d15/d28/d2a/d34/f43 0 1 2024-04-12T05:08:57.716 INFO:tasks.workunit.client.0.smithi113.stdout:2/289: truncate d8/d9/de/f10 4741116 0 2024-04-12T05:08:57.722 INFO:tasks.workunit.client.1.smithi192.stdout:2/349: dwrite d1/d8/d3a/f3b [0,4194304] 0 2024-04-12T05:08:57.722 INFO:tasks.workunit.client.1.smithi192.stdout:7/235: fsync d3/d7/d15/f50 0 2024-04-12T05:08:57.722 INFO:tasks.workunit.client.0.smithi113.stdout:1/541: write d10/d55/f8b [3848925,76526] 0 2024-04-12T05:08:57.723 INFO:tasks.workunit.client.1.smithi192.stdout:7/236: stat d3/d7/d27/c2c 0 2024-04-12T05:08:57.726 INFO:tasks.workunit.client.0.smithi113.stdout:0/448: creat d1/d9/f8a x:0 0 0 2024-04-12T05:08:57.727 INFO:tasks.workunit.client.0.smithi113.stdout:7/452: mkdir d0/dd/d11/d44/d5c/d9a 0 2024-04-12T05:08:57.728 INFO:tasks.workunit.client.0.smithi113.stdout:7/453: dread - d0/dd/d11/f77 zero size 2024-04-12T05:08:57.734 INFO:tasks.workunit.client.0.smithi113.stdout:3/402: mkdir d0/d13/d15/d28/d2a/d8b 0 2024-04-12T05:08:57.735 INFO:tasks.workunit.client.1.smithi192.stdout:7/237: rename d3/d7/d27/c2c to d3/d7/d15/d22/c5b 0 2024-04-12T05:08:57.735 INFO:tasks.workunit.client.0.smithi113.stdout:6/347: write d8/de/f16 [742250,64552] 0 2024-04-12T05:08:57.736 INFO:tasks.workunit.client.1.smithi192.stdout:0/260: truncate d0/d6/d15/d55/d29/f2b 671507 0 2024-04-12T05:08:57.737 INFO:tasks.workunit.client.0.smithi113.stdout:9/484: unlink d1/df/d4b/d6c/c82 0 2024-04-12T05:08:57.742 INFO:tasks.workunit.client.0.smithi113.stdout:9/485: dread d1/d4/f98 [0,4194304] 0 2024-04-12T05:08:57.742 INFO:tasks.workunit.client.1.smithi192.stdout:4/389: dwrite d1/f29 [0,4194304] 0 2024-04-12T05:08:57.744 INFO:tasks.workunit.client.1.smithi192.stdout:2/350: dread d1/d8/f22 [0,4194304] 0 2024-04-12T05:08:57.745 INFO:tasks.workunit.client.0.smithi113.stdout:8/427: write d9/d16/d3c/f52 [523225,80425] 0 2024-04-12T05:08:57.746 INFO:tasks.workunit.client.1.smithi192.stdout:2/351: chown d1/d8/d17/l1a 20649357 1 2024-04-12T05:08:57.746 INFO:tasks.workunit.client.1.smithi192.stdout:4/390: creat d1/d54/d55/d37/d71/f7b x:0 0 0 2024-04-12T05:08:57.750 INFO:tasks.workunit.client.1.smithi192.stdout:2/352: stat d1/d8/d3a/de 0 2024-04-12T05:08:57.754 INFO:tasks.workunit.client.1.smithi192.stdout:2/353: write d1/d8/d3a/de/f30 [2307153,3426] 0 2024-04-12T05:08:57.754 INFO:tasks.workunit.client.1.smithi192.stdout:4/391: mknod d1/d54/d55/d37/c7c 0 2024-04-12T05:08:57.754 INFO:tasks.workunit.client.0.smithi113.stdout:6/348: symlink d8/de/d17/d3b/l7a 0 2024-04-12T05:08:57.754 INFO:tasks.workunit.client.0.smithi113.stdout:6/349: dread - d8/de/d13/d2b/f57 zero size 2024-04-12T05:08:57.754 INFO:tasks.workunit.client.0.smithi113.stdout:6/350: readlink d8/de/d17/d3b/l5e 0 2024-04-12T05:08:57.754 INFO:tasks.workunit.client.1.smithi192.stdout:2/354: truncate d1/d8/d17/d19/f45 1043943 0 2024-04-12T05:08:57.755 INFO:tasks.workunit.client.0.smithi113.stdout:6/351: chown d8/d45/f49 204 1 2024-04-12T05:08:57.755 INFO:tasks.workunit.client.0.smithi113.stdout:1/542: read d10/d14/f2a [1450827,63316] 0 2024-04-12T05:08:57.756 INFO:tasks.workunit.client.0.smithi113.stdout:6/352: chown d8/de/d13/l24 3 1 2024-04-12T05:08:57.758 INFO:tasks.workunit.client.0.smithi113.stdout:6/353: write d8/de/d13/d11/d36/d3d/d4c/f28 [696492,81563] 0 2024-04-12T05:08:57.760 INFO:tasks.workunit.client.1.smithi192.stdout:4/392: write d1/f18 [2544113,52741] 0 2024-04-12T05:08:57.762 INFO:tasks.workunit.client.1.smithi192.stdout:2/355: creat d1/d8/d17/d19/d60/f76 x:0 0 0 2024-04-12T05:08:57.764 INFO:tasks.workunit.client.1.smithi192.stdout:6/408: dread d3/d5/f45 [0,4194304] 0 2024-04-12T05:08:57.766 INFO:tasks.workunit.client.1.smithi192.stdout:7/238: dread d3/d7/f10 [0,4194304] 0 2024-04-12T05:08:57.767 INFO:tasks.workunit.client.1.smithi192.stdout:4/393: creat d1/d54/d55/d64/f7d x:0 0 0 2024-04-12T05:08:57.768 INFO:tasks.workunit.client.1.smithi192.stdout:2/356: creat d1/d8/d3a/f77 x:0 0 0 2024-04-12T05:08:57.768 INFO:tasks.workunit.client.1.smithi192.stdout:0/261: dwrite d0/d6/d15/d55/d29/f2b [0,4194304] 0 2024-04-12T05:08:57.771 INFO:tasks.workunit.client.1.smithi192.stdout:2/357: stat d1/d8/d17/f5f 0 2024-04-12T05:08:57.771 INFO:tasks.workunit.client.1.smithi192.stdout:6/409: write d3/d8/d56/f75 [118477,34843] 0 2024-04-12T05:08:57.774 INFO:tasks.workunit.client.1.smithi192.stdout:4/394: rename d1/d54/d55/d64/f5a to d1/d6b/f7e 0 2024-04-12T05:08:57.776 INFO:tasks.workunit.client.1.smithi192.stdout:7/239: dread d3/d11/f3b [0,4194304] 0 2024-04-12T05:08:57.776 INFO:tasks.workunit.client.1.smithi192.stdout:6/410: fdatasync d3/dc/d39/f87 0 2024-04-12T05:08:57.778 INFO:tasks.workunit.client.1.smithi192.stdout:0/262: rename d0/d4e/f56 to d0/d6/d15/d34/f60 0 2024-04-12T05:08:57.779 INFO:tasks.workunit.client.1.smithi192.stdout:4/395: symlink d1/d54/l7f 0 2024-04-12T05:08:57.780 INFO:tasks.workunit.client.1.smithi192.stdout:7/240: creat d3/d11/f5c x:0 0 0 2024-04-12T05:08:57.782 INFO:tasks.workunit.client.1.smithi192.stdout:0/263: fsync d0/d6/d15/d55/d2e/f5c 0 2024-04-12T05:08:57.783 INFO:tasks.workunit.client.1.smithi192.stdout:2/358: dwrite d1/f39 [0,4194304] 0 2024-04-12T05:08:57.787 INFO:tasks.workunit.client.1.smithi192.stdout:0/264: write d0/f16 [1809444,107671] 0 2024-04-12T05:08:57.790 INFO:tasks.workunit.client.1.smithi192.stdout:6/411: dwrite d3/d8/f1c [4194304,4194304] 0 2024-04-12T05:08:57.792 INFO:tasks.workunit.client.1.smithi192.stdout:4/396: link d1/d1d/d53/l47 d1/d54/d55/d64/d68/l80 0 2024-04-12T05:08:57.793 INFO:tasks.workunit.client.1.smithi192.stdout:7/241: link d3/d7/d27/l3a d3/d7/d15/d1a/l5d 0 2024-04-12T05:08:57.794 INFO:tasks.workunit.client.1.smithi192.stdout:6/412: creat d3/d8/d10/d4c/d68/f89 x:0 0 0 2024-04-12T05:08:57.796 INFO:tasks.workunit.client.1.smithi192.stdout:4/397: creat d1/d54/f81 x:0 0 0 2024-04-12T05:08:57.797 INFO:tasks.workunit.client.1.smithi192.stdout:6/413: mkdir d3/d8/d56/d59/d8a 0 2024-04-12T05:08:57.798 INFO:tasks.workunit.client.1.smithi192.stdout:0/265: link d0/d6/d15/d55/d2e/c4a d0/d6/d15/d34/d4b/c61 0 2024-04-12T05:08:57.799 INFO:tasks.workunit.client.1.smithi192.stdout:0/266: write d0/d6/d15/d1b/d3a/d4f/f24 [431035,90796] 0 2024-04-12T05:08:57.802 INFO:tasks.workunit.client.1.smithi192.stdout:7/242: getdents d3 0 2024-04-12T05:08:57.803 INFO:tasks.workunit.client.1.smithi192.stdout:0/267: write d0/d6/d15/f27 [765196,14398] 0 2024-04-12T05:08:57.804 INFO:tasks.workunit.client.1.smithi192.stdout:7/243: stat d3/d7/d15/d22/l32 0 2024-04-12T05:08:57.804 INFO:tasks.workunit.client.1.smithi192.stdout:0/268: chown d0/d6/d15/d55/d29 88795 1 2024-04-12T05:08:57.805 INFO:tasks.workunit.client.1.smithi192.stdout:7/244: write d3/d11/f16 [22124,66337] 0 2024-04-12T05:08:57.806 INFO:tasks.workunit.client.1.smithi192.stdout:7/245: write d3/d7/d14/f3d [338963,55985] 0 2024-04-12T05:08:57.807 INFO:tasks.workunit.client.1.smithi192.stdout:0/269: creat d0/d6/d15/d34/d4b/d50/f62 x:0 0 0 2024-04-12T05:08:57.808 INFO:tasks.workunit.client.1.smithi192.stdout:7/246: write d3/d1d/f28 [1336694,88613] 0 2024-04-12T05:08:57.808 INFO:tasks.workunit.client.1.smithi192.stdout:1/240: sync 2024-04-12T05:08:57.810 INFO:tasks.workunit.client.0.smithi113.stdout:8/428: symlink d9/d16/d1c/l86 0 2024-04-12T05:08:57.810 INFO:tasks.workunit.client.0.smithi113.stdout:2/290: creat d8/d9/f55 x:0 0 0 2024-04-12T05:08:57.811 INFO:tasks.workunit.client.0.smithi113.stdout:4/436: getdents da/dd 0 2024-04-12T05:08:57.811 INFO:tasks.workunit.client.0.smithi113.stdout:1/543: creat d10/d14/d67/d70/d98/fab x:0 0 0 2024-04-12T05:08:57.812 INFO:tasks.workunit.client.0.smithi113.stdout:3/403: write d0/d13/d15/f5c [699677,8935] 0 2024-04-12T05:08:57.814 INFO:tasks.workunit.client.1.smithi192.stdout:0/270: unlink d0/d6/d15/d55/d2e/f5c 0 2024-04-12T05:08:57.816 INFO:tasks.workunit.client.0.smithi113.stdout:7/454: rename d0/dd/d1b to d0/dc/d6/d9b 0 2024-04-12T05:08:57.819 INFO:tasks.workunit.client.1.smithi192.stdout:1/241: getdents d0/d1b/d44 0 2024-04-12T05:08:57.819 INFO:tasks.workunit.client.0.smithi113.stdout:4/437: symlink da/de/l8c 0 2024-04-12T05:08:57.820 INFO:tasks.workunit.client.0.smithi113.stdout:5/350: dwrite d0/d10/f1b [0,4194304] 0 2024-04-12T05:08:57.820 INFO:tasks.workunit.client.0.smithi113.stdout:3/404: readlink d0/d13/d47/l7f 0 2024-04-12T05:08:57.822 INFO:tasks.workunit.client.0.smithi113.stdout:8/429: creat d9/d16/d1c/d59/d68/f87 x:0 0 0 2024-04-12T05:08:57.824 INFO:tasks.workunit.client.0.smithi113.stdout:4/438: rmdir da/d24/d7e 39 2024-04-12T05:08:57.825 INFO:tasks.workunit.client.0.smithi113.stdout:7/455: chown d0/dc/d6/d1a/l1e 24210387 1 2024-04-12T05:08:57.825 INFO:tasks.workunit.client.0.smithi113.stdout:6/354: dread d8/de/f15 [0,4194304] 0 2024-04-12T05:08:57.825 INFO:tasks.workunit.client.0.smithi113.stdout:0/449: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:08:57.825 INFO:tasks.workunit.client.0.smithi113.stdout:3/405: chown d0/d13/d47/l6d 7537986 1 2024-04-12T05:08:57.828 INFO:tasks.workunit.client.0.smithi113.stdout:2/291: rename d8/f3d to d8/d9/f56 0 2024-04-12T05:08:57.829 INFO:tasks.workunit.client.0.smithi113.stdout:0/450: fsync d1/d9/f8a 0 2024-04-12T05:08:57.830 INFO:tasks.workunit.client.0.smithi113.stdout:6/355: unlink l4 0 2024-04-12T05:08:57.832 INFO:tasks.workunit.client.0.smithi113.stdout:6/356: readlink d8/de/l71 0 2024-04-12T05:08:57.833 INFO:tasks.workunit.client.0.smithi113.stdout:4/439: mknod da/d24/d7e/c8d 0 2024-04-12T05:08:57.834 INFO:tasks.workunit.client.0.smithi113.stdout:5/351: dwrite d0/d10/d27/d13/f28 [0,4194304] 0 2024-04-12T05:08:57.837 INFO:tasks.workunit.client.0.smithi113.stdout:7/456: unlink d0/dc/l4 0 2024-04-12T05:08:57.837 INFO:tasks.workunit.client.0.smithi113.stdout:0/451: symlink d1/d6/d34/l8b 0 2024-04-12T05:08:57.839 INFO:tasks.workunit.client.1.smithi192.stdout:4/398: getdents d1/d54/d55/d64/d68 0 2024-04-12T05:08:57.842 INFO:tasks.workunit.client.0.smithi113.stdout:5/352: mkdir d0/d10/d27/d1d/d3f/d73 0 2024-04-12T05:08:57.843 INFO:tasks.workunit.client.0.smithi113.stdout:7/457: dread - d0/dd/d2d/f64 zero size 2024-04-12T05:08:57.848 INFO:tasks.workunit.client.1.smithi192.stdout:0/271: truncate d0/d6/d15/d19/f4c 424239 0 2024-04-12T05:08:57.848 INFO:tasks.workunit.client.0.smithi113.stdout:4/440: mkdir da/d73/d8e 0 2024-04-12T05:08:57.848 INFO:tasks.workunit.client.1.smithi192.stdout:7/247: write d3/d7/d27/f30 [841880,73479] 0 2024-04-12T05:08:57.850 INFO:tasks.workunit.client.0.smithi113.stdout:2/292: link d8/fd d8/d9/de/d12/f57 0 2024-04-12T05:08:57.851 INFO:tasks.workunit.client.0.smithi113.stdout:6/357: dread d8/de/d17/f27 [0,4194304] 0 2024-04-12T05:08:57.851 INFO:tasks.workunit.client.1.smithi192.stdout:7/248: truncate d3/d7/d15/f50 749771 0 2024-04-12T05:08:57.853 INFO:tasks.workunit.client.0.smithi113.stdout:5/353: mknod d0/d10/d27/d1d/d38/d44/c74 0 2024-04-12T05:08:57.853 INFO:tasks.workunit.client.0.smithi113.stdout:6/358: read d8/de/d17/f27 [421816,1290] 0 2024-04-12T05:08:57.854 INFO:tasks.workunit.client.0.smithi113.stdout:7/458: creat d0/dd/d11/d44/d5c/d85/f9c x:0 0 0 2024-04-12T05:08:57.855 INFO:tasks.workunit.client.1.smithi192.stdout:0/272: dwrite d0/d6/d15/d1b/d46/f4d [0,4194304] 0 2024-04-12T05:08:57.855 INFO:tasks.workunit.client.0.smithi113.stdout:5/354: chown d0/d10/d27/d13/f23 426 1 2024-04-12T05:08:57.855 INFO:tasks.workunit.client.1.smithi192.stdout:1/242: dread d0/d2/dd/d21/d33/f3a [4194304,4194304] 0 2024-04-12T05:08:57.857 INFO:tasks.workunit.client.0.smithi113.stdout:6/359: fdatasync d8/de/d13/d11/f43 0 2024-04-12T05:08:57.857 INFO:tasks.workunit.client.1.smithi192.stdout:7/249: mkdir d3/d1d/d4a/d5e 0 2024-04-12T05:08:57.858 INFO:tasks.workunit.client.0.smithi113.stdout:4/441: mknod da/d26/c8f 0 2024-04-12T05:08:57.859 INFO:tasks.workunit.client.0.smithi113.stdout:6/360: fsync d8/de/d13/d2b/f6a 0 2024-04-12T05:08:57.859 INFO:tasks.workunit.client.0.smithi113.stdout:4/442: readlink da/de/l8c 0 2024-04-12T05:08:57.861 INFO:tasks.workunit.client.1.smithi192.stdout:7/250: fdatasync f1 0 2024-04-12T05:08:57.862 INFO:tasks.workunit.client.0.smithi113.stdout:3/406: write d0/d13/d15/d28/d2a/f2b [600067,59509] 0 2024-04-12T05:08:57.862 INFO:tasks.workunit.client.1.smithi192.stdout:0/273: rename d0/d6/d15/f27 to d0/d6/d15/d55/d2e/f63 0 2024-04-12T05:08:57.865 INFO:tasks.workunit.client.0.smithi113.stdout:3/407: truncate d0/d13/d15/d28/f6f 1030509 0 2024-04-12T05:08:57.868 INFO:tasks.workunit.client.1.smithi192.stdout:7/251: rename d3/d7/d27 to d3/d1d/d5a/d5f 0 2024-04-12T05:08:57.871 INFO:tasks.workunit.client.0.smithi113.stdout:7/459: mknod d0/dd/d11/d44/d5c/d9a/c9d 0 2024-04-12T05:08:57.871 INFO:tasks.workunit.client.1.smithi192.stdout:0/274: mkdir d0/d6/d15/d55/d2e/d59/d64 0 2024-04-12T05:08:57.872 INFO:tasks.workunit.client.0.smithi113.stdout:5/355: mkdir d0/d10/d27/d1d/d38/d75 0 2024-04-12T05:08:57.873 INFO:tasks.workunit.client.1.smithi192.stdout:1/243: getdents d0/d2/d1e 0 2024-04-12T05:08:57.874 INFO:tasks.workunit.client.0.smithi113.stdout:5/356: write d0/d10/d27/d1d/d38/d44/d4e/f64 [754533,491] 0 2024-04-12T05:08:57.875 INFO:tasks.workunit.client.1.smithi192.stdout:0/275: fsync d0/d6/d15/d55/d2e/f63 0 2024-04-12T05:08:57.876 INFO:tasks.workunit.client.0.smithi113.stdout:3/408: mknod d0/d13/d15/d28/d2a/c8c 0 2024-04-12T05:08:57.878 INFO:tasks.workunit.client.1.smithi192.stdout:1/244: creat d0/d2/dd/d21/f47 x:0 0 0 2024-04-12T05:08:57.880 INFO:tasks.workunit.client.0.smithi113.stdout:0/452: truncate d1/d6/f30 310371 0 2024-04-12T05:08:57.881 INFO:tasks.workunit.client.0.smithi113.stdout:8/430: dread d9/d16/d7f/d26/d32/f60 [0,4194304] 0 2024-04-12T05:08:57.881 INFO:tasks.workunit.client.1.smithi192.stdout:0/276: creat d0/d4e/f65 x:0 0 0 2024-04-12T05:08:57.882 INFO:tasks.workunit.client.0.smithi113.stdout:4/443: write da/dd/d3a/f42 [796399,35357] 0 2024-04-12T05:08:57.883 INFO:tasks.workunit.client.0.smithi113.stdout:4/444: readlink da/l2a 0 2024-04-12T05:08:57.883 INFO:tasks.workunit.client.0.smithi113.stdout:8/431: write d9/d16/d7f/d26/d32/d48/f69 [163677,12009] 0 2024-04-12T05:08:57.883 INFO:tasks.workunit.client.1.smithi192.stdout:7/252: dwrite d3/d11/f3b [4194304,4194304] 0 2024-04-12T05:08:57.884 INFO:tasks.workunit.client.0.smithi113.stdout:4/445: dread - da/dd/f6e zero size 2024-04-12T05:08:57.884 INFO:tasks.workunit.client.0.smithi113.stdout:2/293: getdents d8/d9/de/d43 0 2024-04-12T05:08:57.886 INFO:tasks.workunit.client.1.smithi192.stdout:1/245: symlink d0/d2/d1e/d3f/l48 0 2024-04-12T05:08:57.887 INFO:tasks.workunit.client.0.smithi113.stdout:6/361: creat d8/de/d17/f7b x:0 0 0 2024-04-12T05:08:57.888 INFO:tasks.workunit.client.0.smithi113.stdout:0/453: rmdir d1/d43/d59 39 2024-04-12T05:08:57.896 INFO:tasks.workunit.client.0.smithi113.stdout:2/294: unlink d8/d9/de/l21 0 2024-04-12T05:08:57.898 INFO:tasks.workunit.client.0.smithi113.stdout:2/295: dread - d8/d9/f55 zero size 2024-04-12T05:08:57.898 INFO:tasks.workunit.client.1.smithi192.stdout:0/277: dwrite d0/d6/d15/d34/d4b/d50/f62 [0,4194304] 0 2024-04-12T05:08:57.900 INFO:tasks.workunit.client.1.smithi192.stdout:7/253: dwrite d3/d1d/f2e [0,4194304] 0 2024-04-12T05:08:57.900 INFO:tasks.workunit.client.0.smithi113.stdout:8/432: dwrite d9/d16/d7f/d26/d32/f60 [4194304,4194304] 0 2024-04-12T05:08:57.901 INFO:tasks.workunit.client.0.smithi113.stdout:6/362: creat d8/de/d17/f7c x:0 0 0 2024-04-12T05:08:57.902 INFO:tasks.workunit.client.1.smithi192.stdout:1/246: dwrite d0/d1b/f35 [0,4194304] 0 2024-04-12T05:08:57.906 INFO:tasks.workunit.client.1.smithi192.stdout:0/278: mkdir d0/d6/d15/d55/d2e/d66 0 2024-04-12T05:08:57.907 INFO:tasks.workunit.client.0.smithi113.stdout:4/446: getdents da/d24/d7e 0 2024-04-12T05:08:57.909 INFO:tasks.workunit.client.1.smithi192.stdout:1/247: rename d0/d2/d1e/f45 to d0/d2/dd/d21/d39/f49 0 2024-04-12T05:08:57.910 INFO:tasks.workunit.client.0.smithi113.stdout:8/433: symlink d9/d16/d1c/l88 0 2024-04-12T05:08:57.911 INFO:tasks.workunit.client.0.smithi113.stdout:6/363: mkdir d8/de/d13/d11/d36/d3d/d4c/d26/d7d 0 2024-04-12T05:08:57.912 INFO:tasks.workunit.client.0.smithi113.stdout:7/460: dwrite d0/f5a [0,4194304] 0 2024-04-12T05:08:57.913 INFO:tasks.workunit.client.0.smithi113.stdout:4/447: mkdir da/d4f/d90 0 2024-04-12T05:08:57.914 INFO:tasks.workunit.client.1.smithi192.stdout:7/254: dwrite d3/d7/d15/f3c [0,4194304] 0 2024-04-12T05:08:57.915 INFO:tasks.workunit.client.1.smithi192.stdout:0/279: mkdir d0/d6/d67 0 2024-04-12T05:08:57.916 INFO:tasks.workunit.client.0.smithi113.stdout:6/364: dread - d8/de/d17/d3b/d63/f69 zero size 2024-04-12T05:08:57.918 INFO:tasks.workunit.client.1.smithi192.stdout:1/248: write d0/d2/dd/d3c/f41 [1432672,127573] 0 2024-04-12T05:08:57.922 INFO:tasks.workunit.client.0.smithi113.stdout:7/461: mkdir d0/dd/d11/d44/d5c/d9a/d9e 0 2024-04-12T05:08:57.925 INFO:tasks.workunit.client.0.smithi113.stdout:4/448: creat da/d5a/f91 x:0 0 0 2024-04-12T05:08:57.925 INFO:tasks.workunit.client.0.smithi113.stdout:8/434: mknod d9/d34/d4a/c89 0 2024-04-12T05:08:57.926 INFO:tasks.workunit.client.1.smithi192.stdout:1/249: chown d0/d1b/f13 1595 1 2024-04-12T05:08:57.926 INFO:tasks.workunit.client.0.smithi113.stdout:6/365: getdents d8/de/d13/d11/d36/d3d/d4c/d26/d7d 0 2024-04-12T05:08:57.927 INFO:tasks.workunit.client.0.smithi113.stdout:4/449: write da/d3c/d43/f75 [686741,6504] 0 2024-04-12T05:08:57.929 INFO:tasks.workunit.client.1.smithi192.stdout:1/250: read d0/d2/dd/d21/f38 [3316683,47909] 0 2024-04-12T05:08:57.929 INFO:tasks.workunit.client.0.smithi113.stdout:4/450: write da/d3c/d43/f5e [1264787,27987] 0 2024-04-12T05:08:57.932 INFO:tasks.workunit.client.0.smithi113.stdout:8/435: symlink d9/d34/l8a 0 2024-04-12T05:08:57.935 INFO:tasks.workunit.client.0.smithi113.stdout:4/451: mknod da/d24/d7e/d79/c92 0 2024-04-12T05:08:57.936 INFO:tasks.workunit.client.0.smithi113.stdout:4/452: write da/d4f/f72 [4941022,2365] 0 2024-04-12T05:08:57.938 INFO:tasks.workunit.client.1.smithi192.stdout:1/251: dwrite d0/f24 [0,4194304] 0 2024-04-12T05:08:57.941 INFO:tasks.workunit.client.1.smithi192.stdout:1/252: readlink d0/d2/dd/d21/l40 0 2024-04-12T05:08:57.943 INFO:tasks.workunit.client.0.smithi113.stdout:4/453: unlink da/dd/d1b/c57 0 2024-04-12T05:08:57.945 INFO:tasks.workunit.client.0.smithi113.stdout:6/366: getdents d8/de/d17/d3b/d63 0 2024-04-12T05:08:57.947 INFO:tasks.workunit.client.0.smithi113.stdout:8/436: getdents d9/d16/d7f/d26/d32/d48 0 2024-04-12T05:08:57.948 INFO:tasks.workunit.client.0.smithi113.stdout:8/437: readlink d9/l23 0 2024-04-12T05:08:57.948 INFO:tasks.workunit.client.0.smithi113.stdout:4/454: rename da/de/c56 to da/d73/d8e/c93 0 2024-04-12T05:08:57.949 INFO:tasks.workunit.client.1.smithi192.stdout:1/253: dwrite d0/d2/dd/d3c/f41 [0,4194304] 0 2024-04-12T05:08:57.950 INFO:tasks.workunit.client.0.smithi113.stdout:4/455: write da/dd/d1b/f28 [3306036,64288] 0 2024-04-12T05:08:57.953 INFO:tasks.workunit.client.1.smithi192.stdout:1/254: creat d0/d2/d1e/f4a x:0 0 0 2024-04-12T05:08:57.954 INFO:tasks.workunit.client.0.smithi113.stdout:6/367: creat d8/de/d17/d3b/d63/d66/f7e x:0 0 0 2024-04-12T05:08:57.954 INFO:tasks.workunit.client.0.smithi113.stdout:6/368: dread - d8/d45/f6e zero size 2024-04-12T05:08:57.955 INFO:tasks.workunit.client.0.smithi113.stdout:6/369: write d8/de/d13/d11/d36/f4e [670731,13275] 0 2024-04-12T05:08:57.957 INFO:tasks.workunit.client.1.smithi192.stdout:1/255: creat d0/d2/dd/f4b x:0 0 0 2024-04-12T05:08:57.958 INFO:tasks.workunit.client.0.smithi113.stdout:2/296: write d8/d9/de/f22 [1026142,34729] 0 2024-04-12T05:08:57.959 INFO:tasks.workunit.client.1.smithi192.stdout:1/256: stat d0/d2/dd/f1d 0 2024-04-12T05:08:57.960 INFO:tasks.workunit.client.0.smithi113.stdout:6/370: creat d8/de/d13/d11/d36/d3d/f7f x:0 0 0 2024-04-12T05:08:57.962 INFO:tasks.workunit.client.0.smithi113.stdout:4/456: link da/d73/c38 da/d24/d7b/c94 0 2024-04-12T05:08:57.963 INFO:tasks.workunit.client.0.smithi113.stdout:4/457: chown da/d5a 76073518 1 2024-04-12T05:08:57.965 INFO:tasks.workunit.client.1.smithi192.stdout:0/280: dread d0/f20 [0,4194304] 0 2024-04-12T05:08:57.966 INFO:tasks.workunit.client.0.smithi113.stdout:2/297: dwrite d8/d9/d1a/f3a [0,4194304] 0 2024-04-12T05:08:57.968 INFO:tasks.workunit.client.1.smithi192.stdout:1/257: mknod d0/d2/dd/d21/c4c 0 2024-04-12T05:08:57.969 INFO:tasks.workunit.client.0.smithi113.stdout:7/462: write d0/dd/f30 [663678,96640] 0 2024-04-12T05:08:57.972 INFO:tasks.workunit.client.0.smithi113.stdout:7/463: fdatasync d0/dc/d6/d78/f7c 0 2024-04-12T05:08:57.972 INFO:tasks.workunit.client.0.smithi113.stdout:6/371: rename d8/d1d/l38 to d8/de/d13/d11/d36/d3d/d4c/d31/l80 0 2024-04-12T05:08:57.973 INFO:tasks.workunit.client.0.smithi113.stdout:8/438: write d9/d16/d7f/d26/f28 [947913,30508] 0 2024-04-12T05:08:57.975 INFO:tasks.workunit.client.0.smithi113.stdout:8/439: stat d9/d34 0 2024-04-12T05:08:57.976 INFO:tasks.workunit.client.0.smithi113.stdout:2/298: dwrite d8/d9/de/d15/d24/f4b [0,4194304] 0 2024-04-12T05:08:57.978 INFO:tasks.workunit.client.0.smithi113.stdout:4/458: creat da/d3c/f95 x:0 0 0 2024-04-12T05:08:57.982 INFO:tasks.workunit.client.0.smithi113.stdout:8/440: rename d9/d16/d1c/d59/d68/f80 to d9/d16/d1c/f8b 0 2024-04-12T05:08:57.983 INFO:tasks.workunit.client.1.smithi192.stdout:1/258: dwrite d0/f1a [0,4194304] 0 2024-04-12T05:08:57.983 INFO:tasks.workunit.client.0.smithi113.stdout:4/459: stat da/d73/c38 0 2024-04-12T05:08:57.983 INFO:tasks.workunit.client.1.smithi192.stdout:0/281: truncate d0/d6/d15/d34/f60 530126 0 2024-04-12T05:08:57.986 INFO:tasks.workunit.client.0.smithi113.stdout:2/299: mkdir d8/d9/de/d15/d24/d58 0 2024-04-12T05:08:57.987 INFO:tasks.workunit.client.0.smithi113.stdout:2/300: dread - d8/f1e zero size 2024-04-12T05:08:57.987 INFO:tasks.workunit.client.1.smithi192.stdout:1/259: unlink d0/fc 0 2024-04-12T05:08:57.988 INFO:tasks.workunit.client.0.smithi113.stdout:2/301: truncate d8/d9/f56 144460 0 2024-04-12T05:08:57.989 INFO:tasks.workunit.client.1.smithi192.stdout:0/282: chown d0/d6/d15/d34/d4b/d50/f62 9 1 2024-04-12T05:08:57.990 INFO:tasks.workunit.client.1.smithi192.stdout:0/283: chown d0/cc 852363 1 2024-04-12T05:08:57.991 INFO:tasks.workunit.client.0.smithi113.stdout:8/441: mkdir d9/d16/d65/d8c 0 2024-04-12T05:08:57.994 INFO:tasks.workunit.client.0.smithi113.stdout:4/460: rmdir da/dd/d3a 39 2024-04-12T05:08:57.994 INFO:tasks.workunit.client.1.smithi192.stdout:0/284: mkdir d0/d6/d15/d55/d29/d68 0 2024-04-12T05:08:57.996 INFO:tasks.workunit.client.1.smithi192.stdout:0/285: write d0/d6/d15/d55/d2e/f42 [3459855,1682] 0 2024-04-12T05:08:57.997 INFO:tasks.workunit.client.0.smithi113.stdout:8/442: rename d9/d34 to d9/d16/d7f/d26/d32/d48/d8d 0 2024-04-12T05:08:57.997 INFO:tasks.workunit.client.0.smithi113.stdout:6/372: getdents d8/de/d13/d11/d36/d3d/d4c/d26/d5b 0 2024-04-12T05:08:57.998 INFO:tasks.workunit.client.1.smithi192.stdout:1/260: creat d0/d1b/d28/f4d x:0 0 0 2024-04-12T05:08:57.998 INFO:tasks.workunit.client.1.smithi192.stdout:1/261: readlink d0/d2/dd/lf 0 2024-04-12T05:08:58.000 INFO:tasks.workunit.client.0.smithi113.stdout:8/443: write d9/d16/d1c/f51 [3025355,55189] 0 2024-04-12T05:08:58.002 INFO:tasks.workunit.client.0.smithi113.stdout:2/302: dwrite d8/d9/de/d12/f41 [0,4194304] 0 2024-04-12T05:08:58.003 INFO:tasks.workunit.client.1.smithi192.stdout:0/286: creat d0/d6/d15/d55/d29/d68/f69 x:0 0 0 2024-04-12T05:08:58.004 INFO:tasks.workunit.client.1.smithi192.stdout:0/287: truncate d0/f16 2847566 0 2024-04-12T05:08:58.004 INFO:tasks.workunit.client.0.smithi113.stdout:8/444: mkdir d9/d1b/d8e 0 2024-04-12T05:08:58.004 INFO:tasks.workunit.client.0.smithi113.stdout:2/303: dread - d8/d9/de/d15/d24/f34 zero size 2024-04-12T05:08:58.005 INFO:tasks.workunit.client.1.smithi192.stdout:1/262: truncate d0/f20 5015373 0 2024-04-12T05:08:58.006 INFO:tasks.workunit.client.0.smithi113.stdout:4/461: rename da/d4f/c82 to da/d24/c96 0 2024-04-12T05:08:58.009 INFO:tasks.workunit.client.0.smithi113.stdout:7/464: truncate d0/dc/d6/fa 3726258 0 2024-04-12T05:08:58.012 INFO:tasks.workunit.client.0.smithi113.stdout:2/304: truncate d8/d9/de/d15/d24/f34 365863 0 2024-04-12T05:08:58.012 INFO:tasks.workunit.client.1.smithi192.stdout:1/263: dwrite d0/d2/dd/d21/f2d [0,4194304] 0 2024-04-12T05:08:58.015 INFO:tasks.workunit.client.0.smithi113.stdout:2/305: fdatasync d8/d9/de/f31 0 2024-04-12T05:08:58.017 INFO:tasks.workunit.client.0.smithi113.stdout:8/445: dwrite d9/d16/d3c/f74 [0,4194304] 0 2024-04-12T05:08:58.017 INFO:tasks.workunit.client.0.smithi113.stdout:7/465: truncate d0/dc/f18 8234141 0 2024-04-12T05:08:58.021 INFO:tasks.workunit.client.0.smithi113.stdout:2/306: symlink d8/d9/l59 0 2024-04-12T05:08:58.021 INFO:tasks.workunit.client.0.smithi113.stdout:7/466: mkdir d0/dc/d6/d9b/d9f 0 2024-04-12T05:08:58.022 INFO:tasks.workunit.client.0.smithi113.stdout:4/462: dwrite da/d3c/d83/f6a [0,4194304] 0 2024-04-12T05:08:58.025 INFO:tasks.workunit.client.0.smithi113.stdout:8/446: mknod d9/d16/d7f/d26/d32/d48/d8d/d4a/c8f 0 2024-04-12T05:08:58.026 INFO:tasks.workunit.client.0.smithi113.stdout:7/467: fsync d0/dd/d11/f77 0 2024-04-12T05:08:58.032 INFO:tasks.workunit.client.0.smithi113.stdout:4/463: mknod da/d24/d7e/d79/c97 0 2024-04-12T05:08:58.033 INFO:tasks.workunit.client.1.smithi192.stdout:0/288: dread d0/d6/d15/d19/f1d [0,4194304] 0 2024-04-12T05:08:58.034 INFO:tasks.workunit.client.0.smithi113.stdout:7/468: truncate d0/dc/d6/f28 4465967 0 2024-04-12T05:08:58.034 INFO:tasks.workunit.client.0.smithi113.stdout:4/464: symlink da/d2b/l98 0 2024-04-12T05:08:58.035 INFO:tasks.workunit.client.0.smithi113.stdout:2/307: dread d8/d9/de/d12/f41 [0,4194304] 0 2024-04-12T05:08:58.036 INFO:tasks.workunit.client.1.smithi192.stdout:0/289: truncate d0/d6/d15/f23 1167149 0 2024-04-12T05:08:58.038 INFO:tasks.workunit.client.1.smithi192.stdout:1/264: fdatasync d0/d1b/f35 0 2024-04-12T05:08:58.039 INFO:tasks.workunit.client.1.smithi192.stdout:0/290: creat d0/d6/d15/d1b/d3a/d4f/f6a x:0 0 0 2024-04-12T05:08:58.040 INFO:tasks.workunit.client.1.smithi192.stdout:0/291: stat d0/d6/d15/l45 0 2024-04-12T05:08:58.041 INFO:tasks.workunit.client.0.smithi113.stdout:6/373: dread d8/f1a [0,4194304] 0 2024-04-12T05:08:58.041 INFO:tasks.workunit.client.0.smithi113.stdout:8/447: dwrite d9/d16/d1c/d59/d68/f87 [0,4194304] 0 2024-04-12T05:08:58.042 INFO:tasks.workunit.client.0.smithi113.stdout:7/469: mkdir d0/dc/d6/d1a/da0 0 2024-04-12T05:08:58.042 INFO:tasks.workunit.client.1.smithi192.stdout:1/265: symlink d0/d2/dd/d21/d33/l4e 0 2024-04-12T05:08:58.045 INFO:tasks.workunit.client.1.smithi192.stdout:0/292: unlink d0/d6/d15/d1b/d3a/d4f/d33/c3f 0 2024-04-12T05:08:58.047 INFO:tasks.workunit.client.1.smithi192.stdout:1/266: write d0/d2/dd/d21/d39/f49 [475743,15691] 0 2024-04-12T05:08:58.047 INFO:tasks.workunit.client.0.smithi113.stdout:2/308: write d8/d9/de/d15/d24/f3e [1412767,6327] 0 2024-04-12T05:08:58.047 INFO:tasks.workunit.client.0.smithi113.stdout:7/470: write d0/dd/f1c [811202,16164] 0 2024-04-12T05:08:58.050 INFO:tasks.workunit.client.1.smithi192.stdout:1/267: symlink d0/d2/dd/l4f 0 2024-04-12T05:08:58.051 INFO:tasks.workunit.client.0.smithi113.stdout:6/374: mknod d8/de/d13/d11/d36/d53/c81 0 2024-04-12T05:08:58.051 INFO:tasks.workunit.client.1.smithi192.stdout:0/293: dread d0/d6/d15/d1b/d46/f4d [0,4194304] 0 2024-04-12T05:08:58.054 INFO:tasks.workunit.client.1.smithi192.stdout:1/268: rename d0/d1b/f13 to d0/d2/f50 0 2024-04-12T05:08:58.054 INFO:tasks.workunit.client.0.smithi113.stdout:4/465: dwrite da/dd/d3a/f42 [0,4194304] 0 2024-04-12T05:08:58.055 INFO:tasks.workunit.client.0.smithi113.stdout:7/471: write d0/dd/d11/d44/d5c/d85/f98 [814194,82843] 0 2024-04-12T05:08:58.057 INFO:tasks.workunit.client.0.smithi113.stdout:6/375: symlink d8/de/d13/l82 0 2024-04-12T05:08:58.058 INFO:tasks.workunit.client.1.smithi192.stdout:1/269: dread d0/f18 [0,4194304] 0 2024-04-12T05:08:58.059 INFO:tasks.workunit.client.1.smithi192.stdout:0/294: rename d0/d6/d15/d34 to d0/d6b 0 2024-04-12T05:08:58.060 INFO:tasks.workunit.client.0.smithi113.stdout:4/466: creat da/d24/d60/f99 x:0 0 0 2024-04-12T05:08:58.061 INFO:tasks.workunit.client.1.smithi192.stdout:0/295: chown d0/c52 210458388 1 2024-04-12T05:08:58.063 INFO:tasks.workunit.client.0.smithi113.stdout:2/309: link d8/d9/de/d12/f4c d8/d9/de/d12/d2b/f5a 0 2024-04-12T05:08:58.063 INFO:tasks.workunit.client.0.smithi113.stdout:8/448: dwrite d9/d16/d7f/d3a/f2a [4194304,4194304] 0 2024-04-12T05:08:58.067 INFO:tasks.workunit.client.0.smithi113.stdout:4/467: chown da/d2b/l98 6158274 1 2024-04-12T05:08:58.069 INFO:tasks.workunit.client.1.smithi192.stdout:1/270: rename d0/d2/f7 to d0/d2/dd/d21/d33/f51 0 2024-04-12T05:08:58.069 INFO:tasks.workunit.client.1.smithi192.stdout:1/271: chown d0/d2/dd/f1d 229767970 1 2024-04-12T05:08:58.070 INFO:tasks.workunit.client.0.smithi113.stdout:8/449: chown d9/d16/d1c/d59/d78 66615106 1 2024-04-12T05:08:58.072 INFO:tasks.workunit.client.0.smithi113.stdout:3/409: sync 2024-04-12T05:08:58.072 INFO:tasks.workunit.client.0.smithi113.stdout:9/486: sync 2024-04-12T05:08:58.072 INFO:tasks.workunit.client.1.smithi192.stdout:1/272: creat d0/d2/d1e/d3f/f52 x:0 0 0 2024-04-12T05:08:58.075 INFO:tasks.workunit.client.1.smithi192.stdout:0/296: rename d0/d6/d15/f3c to d0/d6/d15/d55/f6c 0 2024-04-12T05:08:58.076 INFO:tasks.workunit.client.0.smithi113.stdout:4/468: write da/d70/d80/f8b [129985,34331] 0 2024-04-12T05:08:58.076 INFO:tasks.workunit.client.0.smithi113.stdout:7/472: dwrite d0/dc/d6/d9b/f25 [0,4194304] 0 2024-04-12T05:08:58.076 INFO:tasks.workunit.client.0.smithi113.stdout:8/450: mkdir d9/d16/d7f/d3a/d6c/d90 0 2024-04-12T05:08:58.078 INFO:tasks.workunit.client.0.smithi113.stdout:6/376: dwrite d8/d45/f5d [0,4194304] 0 2024-04-12T05:08:58.078 INFO:tasks.workunit.client.0.smithi113.stdout:3/410: chown d0/d13/d15/l54 125 1 2024-04-12T05:08:58.079 INFO:tasks.workunit.client.1.smithi192.stdout:1/273: dwrite d0/d2/dd/d21/f2d [0,4194304] 0 2024-04-12T05:08:58.079 INFO:tasks.workunit.client.0.smithi113.stdout:3/411: stat d0/d13/f19 0 2024-04-12T05:08:58.080 INFO:tasks.workunit.client.0.smithi113.stdout:7/473: dread d0/dd/f66 [4194304,4194304] 0 2024-04-12T05:08:58.080 INFO:tasks.workunit.client.0.smithi113.stdout:9/487: creat d1/d4/d5/fac x:0 0 0 2024-04-12T05:08:58.083 INFO:tasks.workunit.client.0.smithi113.stdout:7/474: dread - d0/f48 zero size 2024-04-12T05:08:58.083 INFO:tasks.workunit.client.0.smithi113.stdout:4/469: mkdir da/d3c/d9a 0 2024-04-12T05:08:58.083 INFO:tasks.workunit.client.0.smithi113.stdout:7/475: dread - d0/dc/f83 zero size 2024-04-12T05:08:58.084 INFO:tasks.workunit.client.1.smithi192.stdout:0/297: rmdir d0/d6/d15/d55/d2e/d59/d64 0 2024-04-12T05:08:58.084 INFO:tasks.workunit.client.0.smithi113.stdout:6/377: mkdir d8/de/d13/d11/d36/d3d/d4c/d31/d83 0 2024-04-12T05:08:58.087 INFO:tasks.workunit.client.0.smithi113.stdout:9/488: unlink d1/d27/c9a 0 2024-04-12T05:08:58.088 INFO:tasks.workunit.client.1.smithi192.stdout:0/298: truncate d0/d6/d15/d55/f48 700864 0 2024-04-12T05:08:58.088 INFO:tasks.workunit.client.0.smithi113.stdout:8/451: mknod d9/d16/d7f/d3a/d6c/c91 0 2024-04-12T05:08:58.090 INFO:tasks.workunit.client.1.smithi192.stdout:0/299: write d0/d6b/f38 [4478652,117695] 0 2024-04-12T05:08:58.091 INFO:tasks.workunit.client.0.smithi113.stdout:4/470: dwrite da/d4f/f4e [0,4194304] 0 2024-04-12T05:08:58.091 INFO:tasks.workunit.client.1.smithi192.stdout:1/274: dwrite d0/f24 [0,4194304] 0 2024-04-12T05:08:58.093 INFO:tasks.workunit.client.0.smithi113.stdout:7/476: rename d0/dd/d11/d44/f6b to d0/dd/d11/d44/fa1 0 2024-04-12T05:08:58.094 INFO:tasks.workunit.client.0.smithi113.stdout:8/452: chown d9/d16/d1c/l7a 59817653 1 2024-04-12T05:08:58.096 INFO:tasks.workunit.client.0.smithi113.stdout:6/378: mknod d8/de/d17/c84 0 2024-04-12T05:08:58.098 INFO:tasks.workunit.client.0.smithi113.stdout:4/471: symlink da/d70/d80/l9b 0 2024-04-12T05:08:58.098 INFO:tasks.workunit.client.0.smithi113.stdout:4/472: chown da/d3c/d83/d47 435 1 2024-04-12T05:08:58.099 INFO:tasks.workunit.client.0.smithi113.stdout:8/453: rename d9/d16/d7f/d26/d81 to d9/d16/d7f/d26/d92 0 2024-04-12T05:08:58.099 INFO:tasks.workunit.client.0.smithi113.stdout:3/412: mknod d0/d13/d15/c8d 0 2024-04-12T05:08:58.100 INFO:tasks.workunit.client.0.smithi113.stdout:6/379: symlink d8/de/d13/d11/l85 0 2024-04-12T05:08:58.102 INFO:tasks.workunit.client.0.smithi113.stdout:3/413: rmdir d0/d7e 39 2024-04-12T05:08:58.102 INFO:tasks.workunit.client.1.smithi192.stdout:0/300: rename d0/d6b/d4b/d50 to d0/d4e/d53/d6d 0 2024-04-12T05:08:58.103 INFO:tasks.workunit.client.0.smithi113.stdout:7/477: creat d0/fa2 x:0 0 0 2024-04-12T05:08:58.105 INFO:tasks.workunit.client.1.smithi192.stdout:0/301: mknod d0/d6/d15/d1b/d46/c6e 0 2024-04-12T05:08:58.105 INFO:tasks.workunit.client.0.smithi113.stdout:6/380: truncate d8/de/f15 341376 0 2024-04-12T05:08:58.106 INFO:tasks.workunit.client.0.smithi113.stdout:6/381: fsync d8/de/d13/d11/d36/f35 0 2024-04-12T05:08:58.111 INFO:tasks.workunit.client.0.smithi113.stdout:6/382: symlink d8/de/d13/d11/d36/d3d/d4c/d31/l86 0 2024-04-12T05:08:58.111 INFO:tasks.workunit.client.0.smithi113.stdout:3/414: creat d0/d13/d15/d28/d2a/f8e x:0 0 0 2024-04-12T05:08:58.112 INFO:tasks.workunit.client.0.smithi113.stdout:6/383: chown d8/de/d17/d3b/l5e 170753 1 2024-04-12T05:08:58.113 INFO:tasks.workunit.client.1.smithi192.stdout:0/302: creat d0/d6/d15/f6f x:0 0 0 2024-04-12T05:08:58.114 INFO:tasks.workunit.client.0.smithi113.stdout:6/384: write d8/de/d13/d11/d36/d3d/d4c/d26/f51 [1678681,46043] 0 2024-04-12T05:08:58.114 INFO:tasks.workunit.client.1.smithi192.stdout:1/275: write d0/d2/dd/d21/d33/f42 [814189,66439] 0 2024-04-12T05:08:58.115 INFO:tasks.workunit.client.1.smithi192.stdout:1/276: chown d0/d2/f34 829880609 1 2024-04-12T05:08:58.116 INFO:tasks.workunit.client.1.smithi192.stdout:0/303: write d0/d6/d15/d1b/d3a/d4f/f6a [368268,38232] 0 2024-04-12T05:08:58.116 INFO:tasks.workunit.client.0.smithi113.stdout:6/385: creat d8/de/d13/d2b/f87 x:0 0 0 2024-04-12T05:08:58.117 INFO:tasks.workunit.client.0.smithi113.stdout:3/415: read d0/d13/d47/d4f/f35 [962139,128808] 0 2024-04-12T05:08:58.119 INFO:tasks.workunit.client.1.smithi192.stdout:1/277: creat d0/d1b/f53 x:0 0 0 2024-04-12T05:08:58.120 INFO:tasks.workunit.client.0.smithi113.stdout:6/386: unlink d8/de/d13/d11/d36/d3d/f7f 0 2024-04-12T05:08:58.120 INFO:tasks.workunit.client.1.smithi192.stdout:0/304: dread d0/d6/d15/d1b/d3a/d4f/f24 [4194304,4194304] 0 2024-04-12T05:08:58.122 INFO:tasks.workunit.client.0.smithi113.stdout:3/416: mkdir d0/d13/d15/d28/d2a/d8f 0 2024-04-12T05:08:58.122 INFO:tasks.workunit.client.1.smithi192.stdout:1/278: mknod d0/d29/c54 0 2024-04-12T05:08:58.123 INFO:tasks.workunit.client.1.smithi192.stdout:1/279: stat d0/f1a 0 2024-04-12T05:08:58.123 INFO:tasks.workunit.client.1.smithi192.stdout:0/305: write d0/d6/d15/d19/f1d [4094022,100220] 0 2024-04-12T05:08:58.127 INFO:tasks.workunit.client.0.smithi113.stdout:6/387: dwrite d8/de/d17/d3b/d63/d66/f7e [0,4194304] 0 2024-04-12T05:08:58.128 INFO:tasks.workunit.client.0.smithi113.stdout:1/544: sync 2024-04-12T05:08:58.128 INFO:tasks.workunit.client.0.smithi113.stdout:5/357: sync 2024-04-12T05:08:58.130 INFO:tasks.workunit.client.0.smithi113.stdout:6/388: chown d8/d45/f6e 1476 1 2024-04-12T05:08:58.132 INFO:tasks.workunit.client.1.smithi192.stdout:1/280: mkdir d0/d2/dd/d21/d55 0 2024-04-12T05:08:58.134 INFO:tasks.workunit.client.1.smithi192.stdout:0/306: dwrite d0/d6/d15/d19/f31 [4194304,4194304] 0 2024-04-12T05:08:58.134 INFO:tasks.workunit.client.0.smithi113.stdout:3/417: dwrite d0/f12 [8388608,4194304] 0 2024-04-12T05:08:58.136 INFO:tasks.workunit.client.0.smithi113.stdout:2/310: rmdir d8/d9 39 2024-04-12T05:08:58.140 INFO:tasks.workunit.client.1.smithi192.stdout:0/307: dread d0/d6b/f38 [0,4194304] 0 2024-04-12T05:08:58.144 INFO:tasks.workunit.client.0.smithi113.stdout:1/545: dwrite d10/d2e/f4e [0,4194304] 0 2024-04-12T05:08:58.144 INFO:tasks.workunit.client.0.smithi113.stdout:2/311: truncate d8/d9/de/d12/f41 4869598 0 2024-04-12T05:08:58.145 INFO:tasks.workunit.client.0.smithi113.stdout:3/418: creat d0/d13/d47/d4f/f90 x:0 0 0 2024-04-12T05:08:58.145 INFO:tasks.workunit.client.0.smithi113.stdout:9/489: rmdir d1/d27 39 2024-04-12T05:08:58.147 INFO:tasks.workunit.client.0.smithi113.stdout:6/389: creat d8/de/d13/d11/d36/d3d/d4c/d26/d5b/f88 x:0 0 0 2024-04-12T05:08:58.147 INFO:tasks.workunit.client.0.smithi113.stdout:9/490: chown d1/c2a 5 1 2024-04-12T05:08:58.148 INFO:tasks.workunit.client.1.smithi192.stdout:0/308: getdents d0/d6b/d4b 0 2024-04-12T05:08:58.149 INFO:tasks.workunit.client.0.smithi113.stdout:3/419: truncate d0/d13/d15/d28/d2a/f2b 883139 0 2024-04-12T05:08:58.149 INFO:tasks.workunit.client.0.smithi113.stdout:9/491: readlink d1/df/d29/da6/la4 0 2024-04-12T05:08:58.155 INFO:tasks.workunit.client.0.smithi113.stdout:6/390: mkdir d8/de/d13/d11/d36/d89 0 2024-04-12T05:08:58.156 INFO:tasks.workunit.client.0.smithi113.stdout:1/546: dwrite d10/d41/d89/d96/d7d/fa9 [0,4194304] 0 2024-04-12T05:08:58.157 INFO:tasks.workunit.client.0.smithi113.stdout:9/492: mkdir d1/df/d4b/d63/dad 0 2024-04-12T05:08:58.161 INFO:tasks.workunit.client.1.smithi192.stdout:1/281: dwrite d0/d2/dd/d21/f38 [4194304,4194304] 0 2024-04-12T05:08:58.163 INFO:tasks.workunit.client.0.smithi113.stdout:1/547: write d10/d41/d89/d96/f91 [418053,86944] 0 2024-04-12T05:08:58.163 INFO:tasks.workunit.client.0.smithi113.stdout:8/454: rename d9/d16/d7f/d26 to d9/d1b/d8e/d93 0 2024-04-12T05:08:58.165 INFO:tasks.workunit.client.0.smithi113.stdout:3/420: mkdir d0/d13/d15/d28/d2a/d8b/d91 0 2024-04-12T05:08:58.165 INFO:tasks.workunit.client.1.smithi192.stdout:0/309: dwrite d0/d6/d15/f6f [0,4194304] 0 2024-04-12T05:08:58.167 INFO:tasks.workunit.client.0.smithi113.stdout:9/493: mknod d1/d4/d5/d6/cae 0 2024-04-12T05:08:58.168 INFO:tasks.workunit.client.1.smithi192.stdout:0/310: read d0/d6/d15/f23 [172309,110296] 0 2024-04-12T05:08:58.168 INFO:tasks.workunit.client.1.smithi192.stdout:1/282: unlink d0/d2/dd/d21/d33/f51 0 2024-04-12T05:08:58.169 INFO:tasks.workunit.client.0.smithi113.stdout:6/391: rename d8/de/d13/d11/d36/d3d/d4c/d26/d64/c6d to d8/de/d13/d11/d36/d53/c8a 0 2024-04-12T05:08:58.169 INFO:tasks.workunit.client.1.smithi192.stdout:1/283: chown d0/d2/f50 23042 1 2024-04-12T05:08:58.171 INFO:tasks.workunit.client.1.smithi192.stdout:0/311: fdatasync d0/d6/d15/f3d 0 2024-04-12T05:08:58.172 INFO:tasks.workunit.client.0.smithi113.stdout:1/548: dwrite d10/d2e/f71 [0,4194304] 0 2024-04-12T05:08:58.172 INFO:tasks.workunit.client.0.smithi113.stdout:6/392: dread - d8/de/d13/f67 zero size 2024-04-12T05:08:58.172 INFO:tasks.workunit.client.0.smithi113.stdout:1/549: readlink d10/d55/l9d 0 2024-04-12T05:08:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:56.670324+0000 mgr.smithi192.ocppyo (mgr.24357) 24 : cluster [DBG] pgmap v18: 65 pgs: 65 active+clean; 668 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 3.6 MiB/s rd, 58 MiB/s wr, 89 op/s 2024-04-12T05:08:58.175 INFO:tasks.workunit.client.0.smithi113.stdout:6/393: truncate d8/de/d13/d11/d36/f61 448115 0 2024-04-12T05:08:58.176 INFO:tasks.workunit.client.0.smithi113.stdout:3/421: unlink d0/d13/d47/d4f/l37 0 2024-04-12T05:08:58.177 INFO:tasks.workunit.client.1.smithi192.stdout:1/284: rmdir d0/d2/dd/d21/d39 39 2024-04-12T05:08:58.177 INFO:tasks.workunit.client.0.smithi113.stdout:9/494: creat d1/df/d4b/d63/d3d/d6f/faf x:0 0 0 2024-04-12T05:08:58.180 INFO:tasks.workunit.client.1.smithi192.stdout:1/285: mkdir d0/d2/d1e/d3f/d56 0 2024-04-12T05:08:58.182 INFO:tasks.workunit.client.1.smithi192.stdout:0/312: dwrite d0/d4e/f65 [0,4194304] 0 2024-04-12T05:08:58.182 INFO:tasks.workunit.client.0.smithi113.stdout:1/550: dwrite d10/d2e/d7c/f92 [0,4194304] 0 2024-04-12T05:08:58.185 INFO:tasks.workunit.client.1.smithi192.stdout:1/286: symlink d0/d2/l57 0 2024-04-12T05:08:58.191 INFO:tasks.workunit.client.1.smithi192.stdout:1/287: mkdir d0/d2/d1e/d3f/d56/d58 0 2024-04-12T05:08:58.192 INFO:tasks.workunit.client.0.smithi113.stdout:5/358: dread d0/f5 [0,4194304] 0 2024-04-12T05:08:58.194 INFO:tasks.workunit.client.1.smithi192.stdout:1/288: truncate d0/d2/dd/f32 541641 0 2024-04-12T05:08:58.195 INFO:tasks.workunit.client.1.smithi192.stdout:1/289: readlink d0/d2/l14 0 2024-04-12T05:08:58.196 INFO:tasks.workunit.client.0.smithi113.stdout:6/394: dwrite d8/de/d17/f78 [0,4194304] 0 2024-04-12T05:08:58.196 INFO:tasks.workunit.client.0.smithi113.stdout:3/422: rename d0/d13/d15/d28/d2a/d34/d5f/f86 to d0/d7e/f92 0 2024-04-12T05:08:58.197 INFO:tasks.workunit.client.0.smithi113.stdout:3/423: dread - d0/d7e/f92 zero size 2024-04-12T05:08:58.197 INFO:tasks.workunit.client.0.smithi113.stdout:9/495: dwrite d1/f78 [0,4194304] 0 2024-04-12T05:08:58.199 INFO:tasks.workunit.client.1.smithi192.stdout:5/330: sync 2024-04-12T05:08:58.201 INFO:tasks.workunit.client.1.smithi192.stdout:5/331: readlink d0/d15/d32/l58 0 2024-04-12T05:08:58.201 INFO:tasks.workunit.client.1.smithi192.stdout:0/313: dwrite d0/d6/d15/d19/f31 [0,4194304] 0 2024-04-12T05:08:58.202 INFO:tasks.workunit.client.0.smithi113.stdout:1/551: fsync d10/d2e/d31/d33/f7a 0 2024-04-12T05:08:58.202 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:56.670324+0000 mgr.smithi192.ocppyo (mgr.24357) 24 : cluster [DBG] pgmap v18: 65 pgs: 65 active+clean; 668 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail; 3.6 MiB/s rd, 58 MiB/s wr, 89 op/s 2024-04-12T05:08:58.204 INFO:tasks.workunit.client.1.smithi192.stdout:1/290: dread d0/d2/dd/d21/d39/f49 [0,4194304] 0 2024-04-12T05:08:58.207 INFO:tasks.workunit.client.0.smithi113.stdout:3/424: write d0/d13/f1f [629939,120120] 0 2024-04-12T05:08:58.209 INFO:tasks.workunit.client.0.smithi113.stdout:9/496: chown d1/d27/d7a 14712 1 2024-04-12T05:08:58.211 INFO:tasks.workunit.client.0.smithi113.stdout:6/395: truncate d8/de/d13/d11/d36/d3d/d4c/f28 1683469 0 2024-04-12T05:08:58.212 INFO:tasks.workunit.client.0.smithi113.stdout:2/312: dwrite d8/fd [0,4194304] 0 2024-04-12T05:08:58.213 INFO:tasks.workunit.client.0.smithi113.stdout:3/425: read - d0/d13/f60 zero size 2024-04-12T05:08:58.215 INFO:tasks.workunit.client.0.smithi113.stdout:2/313: chown d8/l25 28088 1 2024-04-12T05:08:58.216 INFO:tasks.workunit.client.0.smithi113.stdout:2/314: fsync d8/f48 0 2024-04-12T05:08:58.217 INFO:tasks.workunit.client.1.smithi192.stdout:1/291: fsync d0/d2/dd/f15 0 2024-04-12T05:08:58.217 INFO:tasks.workunit.client.0.smithi113.stdout:3/426: chown d0/d13/d47/f6c 241 1 2024-04-12T05:08:58.217 INFO:tasks.workunit.client.0.smithi113.stdout:2/315: fdatasync d8/d9/d1a/f3a 0 2024-04-12T05:08:58.218 INFO:tasks.workunit.client.0.smithi113.stdout:1/552: write d10/d41/f88 [974445,71230] 0 2024-04-12T05:08:58.218 INFO:tasks.workunit.client.0.smithi113.stdout:9/497: chown d1/df/c44 30628 1 2024-04-12T05:08:58.219 INFO:tasks.workunit.client.1.smithi192.stdout:1/292: read - d0/d1b/d28/f4d zero size 2024-04-12T05:08:58.219 INFO:tasks.workunit.client.1.smithi192.stdout:5/332: dwrite d0/d15/d32/f4c [0,4194304] 0 2024-04-12T05:08:58.220 INFO:tasks.workunit.client.0.smithi113.stdout:8/455: rename d9/d16/d7f/d3a/c2f to d9/d1b/d8e/d93/d32/d48/d8d/c94 0 2024-04-12T05:08:58.221 INFO:tasks.workunit.client.0.smithi113.stdout:5/359: dwrite d0/f3a [0,4194304] 0 2024-04-12T05:08:58.221 INFO:tasks.workunit.client.1.smithi192.stdout:0/314: truncate d0/d6/d15/d1b/d46/f49 201498 0 2024-04-12T05:08:58.224 INFO:tasks.workunit.client.0.smithi113.stdout:8/456: write d9/f47 [2869580,31869] 0 2024-04-12T05:08:58.226 INFO:tasks.workunit.client.0.smithi113.stdout:6/396: mknod d8/de/d13/d11/d36/d3d/c8b 0 2024-04-12T05:08:58.229 INFO:tasks.workunit.client.1.smithi192.stdout:1/293: rename d0/d2/d1e/d3f/l48 to d0/d2/dd/d21/d39/l59 0 2024-04-12T05:08:58.229 INFO:tasks.workunit.client.0.smithi113.stdout:9/498: truncate d1/d4/d5/d22/d9e/faa 210439 0 2024-04-12T05:08:58.231 INFO:tasks.workunit.client.1.smithi192.stdout:0/315: creat d0/d6/d15/d55/d2e/d66/f70 x:0 0 0 2024-04-12T05:08:58.232 INFO:tasks.workunit.client.0.smithi113.stdout:3/427: dread d0/d13/d15/d28/f45 [0,4194304] 0 2024-04-12T05:08:58.233 INFO:tasks.workunit.client.0.smithi113.stdout:5/360: creat d0/d10/d27/d1d/d3f/d73/f76 x:0 0 0 2024-04-12T05:08:58.235 INFO:tasks.workunit.client.0.smithi113.stdout:2/316: dwrite d8/d9/d1a/f49 [0,4194304] 0 2024-04-12T05:08:58.235 INFO:tasks.workunit.client.1.smithi192.stdout:0/316: chown d0/d6/d15/d55/d2e/d66/f70 19433078 1 2024-04-12T05:08:58.236 INFO:tasks.workunit.client.0.smithi113.stdout:9/499: creat d1/df/d29/fb0 x:0 0 0 2024-04-12T05:08:58.237 INFO:tasks.workunit.client.0.smithi113.stdout:8/457: dread d9/d16/d1c/f2c [0,4194304] 0 2024-04-12T05:08:58.237 INFO:tasks.workunit.client.0.smithi113.stdout:5/361: write d0/d10/f19 [645965,105728] 0 2024-04-12T05:08:58.238 INFO:tasks.workunit.client.0.smithi113.stdout:6/397: creat d8/de/d13/d11/d36/d3d/d4c/d26/d64/f8c x:0 0 0 2024-04-12T05:08:58.239 INFO:tasks.workunit.client.0.smithi113.stdout:9/500: fsync d1/d4/f77 0 2024-04-12T05:08:58.241 INFO:tasks.workunit.client.1.smithi192.stdout:0/317: write d0/d6/d15/d19/f31 [833144,73147] 0 2024-04-12T05:08:58.241 INFO:tasks.workunit.client.1.smithi192.stdout:1/294: dwrite d0/d2/dd/f32 [0,4194304] 0 2024-04-12T05:08:58.245 INFO:tasks.workunit.client.0.smithi113.stdout:5/362: symlink d0/d10/d27/d5f/l77 0 2024-04-12T05:08:58.248 INFO:tasks.workunit.client.1.smithi192.stdout:0/318: rename d0/d6/d15/d19/f30 to d0/d6/d15/d19/f71 0 2024-04-12T05:08:58.249 INFO:tasks.workunit.client.0.smithi113.stdout:2/317: dwrite d8/d9/de/d12/d2b/f54 [0,4194304] 0 2024-04-12T05:08:58.251 INFO:tasks.workunit.client.0.smithi113.stdout:6/398: link d8/f12 d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8d 0 2024-04-12T05:08:58.252 INFO:tasks.workunit.client.1.smithi192.stdout:0/319: mkdir d0/d6b/d4b/d72 0 2024-04-12T05:08:58.253 INFO:tasks.workunit.client.1.smithi192.stdout:0/320: readlink d0/d6/d15/l45 0 2024-04-12T05:08:58.257 INFO:tasks.workunit.client.0.smithi113.stdout:2/318: mkdir d8/d9/d5b 0 2024-04-12T05:08:58.258 INFO:tasks.workunit.client.0.smithi113.stdout:6/399: rename d8/de/d17/d3b to d8/de/d13/d11/d36/d3d/d4c/d31/d8e 0 2024-04-12T05:08:58.259 INFO:tasks.workunit.client.0.smithi113.stdout:9/501: getdents d1/df/d4b 0 2024-04-12T05:08:58.260 INFO:tasks.workunit.client.0.smithi113.stdout:6/400: read d8/d45/f49 [4112070,127370] 0 2024-04-12T05:08:58.260 INFO:tasks.workunit.client.0.smithi113.stdout:9/502: readlink d1/d4/d30/l5e 0 2024-04-12T05:08:58.262 INFO:tasks.workunit.client.0.smithi113.stdout:9/503: write d1/f46 [6805428,127991] 0 2024-04-12T05:08:58.262 INFO:tasks.workunit.client.0.smithi113.stdout:6/401: chown d8/de/l50 3 1 2024-04-12T05:08:58.265 INFO:tasks.workunit.client.0.smithi113.stdout:6/402: truncate d8/de/d13/d11/d36/d3d/d4c/d26/d5b/f88 639004 0 2024-04-12T05:08:58.267 INFO:tasks.workunit.client.0.smithi113.stdout:6/403: dread d8/de/d13/d11/d36/d3d/d4c/d26/d5b/f88 [0,4194304] 0 2024-04-12T05:08:58.270 INFO:tasks.workunit.client.0.smithi113.stdout:6/404: write d8/de/d13/d11/d36/f61 [68735,80856] 0 2024-04-12T05:08:58.273 INFO:tasks.workunit.client.0.smithi113.stdout:5/363: fdatasync d0/f3a 0 2024-04-12T05:08:58.273 INFO:tasks.workunit.client.0.smithi113.stdout:6/405: unlink d8/de/d13/d11/c3a 0 2024-04-12T05:08:58.275 INFO:tasks.workunit.client.0.smithi113.stdout:5/364: unlink d0/d10/f19 0 2024-04-12T05:08:58.277 INFO:tasks.workunit.client.1.smithi192.stdout:1/295: getdents d0/d2/dd/d21/d39 0 2024-04-12T05:08:58.278 INFO:tasks.workunit.client.0.smithi113.stdout:5/365: fdatasync d0/d10/f16 0 2024-04-12T05:08:58.279 INFO:tasks.workunit.client.0.smithi113.stdout:5/366: chown d0/d10/d27/d1d/c37 0 1 2024-04-12T05:08:58.279 INFO:tasks.workunit.client.0.smithi113.stdout:6/406: creat d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8f x:0 0 0 2024-04-12T05:08:58.280 INFO:tasks.workunit.client.0.smithi113.stdout:6/407: readlink d8/de/l71 0 2024-04-12T05:08:58.280 INFO:tasks.workunit.client.1.smithi192.stdout:0/321: rename d0/d6b to d0/d6/d15/d1b/d46/d73 0 2024-04-12T05:08:58.281 INFO:tasks.workunit.client.0.smithi113.stdout:5/367: mknod d0/d10/d27/d1d/d38/d44/c78 0 2024-04-12T05:08:58.284 INFO:tasks.workunit.client.0.smithi113.stdout:6/408: fsync d8/de/d13/d11/d36/f61 0 2024-04-12T05:08:58.286 INFO:tasks.workunit.client.1.smithi192.stdout:0/322: rename d0/l10 to d0/l74 0 2024-04-12T05:08:58.287 INFO:tasks.workunit.client.1.smithi192.stdout:1/296: truncate d0/d2/dd/d21/f2d 1228404 0 2024-04-12T05:08:58.288 INFO:tasks.workunit.client.0.smithi113.stdout:6/409: creat d8/de/d13/d11/d36/d3d/f90 x:0 0 0 2024-04-12T05:08:58.288 INFO:tasks.workunit.client.1.smithi192.stdout:0/323: symlink d0/d6/l75 0 2024-04-12T05:08:58.289 INFO:tasks.workunit.client.0.smithi113.stdout:8/458: dwrite d9/d1b/d8e/d93/d32/d48/d8d/f57 [0,4194304] 0 2024-04-12T05:08:58.289 INFO:tasks.workunit.client.1.smithi192.stdout:0/324: readlink d0/d6/d15/l45 0 2024-04-12T05:08:58.290 INFO:tasks.workunit.client.1.smithi192.stdout:1/297: creat d0/d2/d1e/d3f/d56/f5a x:0 0 0 2024-04-12T05:08:58.290 INFO:tasks.workunit.client.1.smithi192.stdout:0/325: fdatasync d0/d6/d15/d55/d2e/f63 0 2024-04-12T05:08:58.292 INFO:tasks.workunit.client.0.smithi113.stdout:8/459: fsync d9/d1b/d8e/d93/f84 0 2024-04-12T05:08:58.292 INFO:tasks.workunit.client.0.smithi113.stdout:5/368: dwrite d0/d10/d6e/f4f [0,4194304] 0 2024-04-12T05:08:58.294 INFO:tasks.workunit.client.0.smithi113.stdout:5/369: dread - d0/d10/d27/d1d/d38/d3b/f61 zero size 2024-04-12T05:08:58.295 INFO:tasks.workunit.client.1.smithi192.stdout:0/326: getdents d0/d6/d15/d1b/d3a 0 2024-04-12T05:08:58.297 INFO:tasks.workunit.client.0.smithi113.stdout:6/410: symlink d8/de/d13/d11/d36/l91 0 2024-04-12T05:08:58.297 INFO:tasks.workunit.client.1.smithi192.stdout:0/327: read d0/d6/d15/d19/f1d [3378266,33951] 0 2024-04-12T05:08:58.298 INFO:tasks.workunit.client.0.smithi113.stdout:8/460: unlink d9/d16/c27 0 2024-04-12T05:08:58.298 INFO:tasks.workunit.client.1.smithi192.stdout:1/298: dwrite d0/d2/f50 [0,4194304] 0 2024-04-12T05:08:58.299 INFO:tasks.workunit.client.1.smithi192.stdout:0/328: fsync d0/d6/d15/d55/f48 0 2024-04-12T05:08:58.301 INFO:tasks.workunit.client.0.smithi113.stdout:6/411: symlink d8/de/d13/d11/d36/d3d/d4c/d26/d5b/l92 0 2024-04-12T05:08:58.303 INFO:tasks.workunit.client.0.smithi113.stdout:8/461: mknod d9/d16/d1c/d59/d68/c95 0 2024-04-12T05:08:58.305 INFO:tasks.workunit.client.0.smithi113.stdout:8/462: chown d9/d16/d1c/d59/d68 27756 1 2024-04-12T05:08:58.305 INFO:tasks.workunit.client.0.smithi113.stdout:5/370: mknod d0/d10/d27/d1d/d38/c79 0 2024-04-12T05:08:58.305 INFO:tasks.workunit.client.1.smithi192.stdout:0/329: dread d0/d6/d15/d19/f71 [0,4194304] 0 2024-04-12T05:08:58.308 INFO:tasks.workunit.client.0.smithi113.stdout:5/371: dread - d0/d10/d27/d13/f6d zero size 2024-04-12T05:08:58.309 INFO:tasks.workunit.client.0.smithi113.stdout:6/412: dwrite d8/de/d13/d11/d36/d3d/d4c/f28 [0,4194304] 0 2024-04-12T05:08:58.318 INFO:tasks.workunit.client.1.smithi192.stdout:1/299: truncate d0/d2/dd/f32 3495084 0 2024-04-12T05:08:58.318 INFO:tasks.workunit.client.1.smithi192.stdout:1/300: fsync d0/d1b/f53 0 2024-04-12T05:08:58.319 INFO:tasks.workunit.client.0.smithi113.stdout:8/463: read d9/d1b/d8e/d93/d32/d5b/f63 [3427445,101439] 0 2024-04-12T05:08:58.319 INFO:tasks.workunit.client.1.smithi192.stdout:1/301: stat d0/d2/dd/d21/f47 0 2024-04-12T05:08:58.320 INFO:tasks.workunit.client.1.smithi192.stdout:1/302: truncate d0/d2/dd/f4b 877797 0 2024-04-12T05:08:58.323 INFO:tasks.workunit.client.1.smithi192.stdout:1/303: mkdir d0/d2/dd/d3c/d5b 0 2024-04-12T05:08:58.324 INFO:tasks.workunit.client.0.smithi113.stdout:5/372: rename d0/d10/d27/d13 to d0/d10/d6e/d7a 0 2024-04-12T05:08:58.326 INFO:tasks.workunit.client.0.smithi113.stdout:8/464: dwrite d9/fb [0,4194304] 0 2024-04-12T05:08:58.326 INFO:tasks.workunit.client.0.smithi113.stdout:6/413: mkdir d8/de/d13/d11/d36/d3d/d93 0 2024-04-12T05:08:58.327 INFO:tasks.workunit.client.0.smithi113.stdout:5/373: fdatasync d0/d10/d27/d1d/f21 0 2024-04-12T05:08:58.329 INFO:tasks.workunit.client.0.smithi113.stdout:8/465: mkdir d9/d1b/d96 0 2024-04-12T05:08:58.330 INFO:tasks.workunit.client.0.smithi113.stdout:0/454: sync 2024-04-12T05:08:58.331 INFO:tasks.workunit.client.0.smithi113.stdout:5/374: chown d0/d10/d27/d1d/d38/d44/c74 0 1 2024-04-12T05:08:58.332 INFO:tasks.workunit.client.0.smithi113.stdout:6/414: creat d8/de/f94 x:0 0 0 2024-04-12T05:08:58.334 INFO:tasks.workunit.client.1.smithi192.stdout:0/330: dwrite d0/d6/d15/d1b/d46/f49 [0,4194304] 0 2024-04-12T05:08:58.336 INFO:tasks.workunit.client.0.smithi113.stdout:8/466: creat d9/d16/d65/f97 x:0 0 0 2024-04-12T05:08:58.336 INFO:tasks.workunit.client.1.smithi192.stdout:9/261: sync 2024-04-12T05:08:58.336 INFO:tasks.workunit.client.1.smithi192.stdout:3/271: sync 2024-04-12T05:08:58.336 INFO:tasks.workunit.client.1.smithi192.stdout:0/331: chown d0/d4e/d53/d6d 127 1 2024-04-12T05:08:58.337 INFO:tasks.workunit.client.1.smithi192.stdout:1/304: truncate d0/d2/dd/d3c/f41 601362 0 2024-04-12T05:08:58.340 INFO:tasks.workunit.client.0.smithi113.stdout:6/415: fdatasync d8/f12 0 2024-04-12T05:08:58.341 INFO:tasks.workunit.client.0.smithi113.stdout:6/416: readlink d8/de/d13/d11/l6f 0 2024-04-12T05:08:58.342 INFO:tasks.workunit.client.1.smithi192.stdout:3/272: symlink d5/d13/d3a/d4a/l5d 0 2024-04-12T05:08:58.342 INFO:tasks.workunit.client.1.smithi192.stdout:9/262: mknod d11/d25/d36/c58 0 2024-04-12T05:08:58.343 INFO:tasks.workunit.client.1.smithi192.stdout:0/332: creat d0/d6/d15/d1b/d3a/f76 x:0 0 0 2024-04-12T05:08:58.343 INFO:tasks.workunit.client.0.smithi113.stdout:6/417: write d8/de/f94 [382442,31202] 0 2024-04-12T05:08:58.345 INFO:tasks.workunit.client.1.smithi192.stdout:1/305: symlink d0/l5c 0 2024-04-12T05:08:58.345 INFO:tasks.workunit.client.1.smithi192.stdout:0/333: dread - d0/d6/d15/d55/d29/d68/f69 zero size 2024-04-12T05:08:58.346 INFO:tasks.workunit.client.0.smithi113.stdout:5/375: dwrite d0/f2e [0,4194304] 0 2024-04-12T05:08:58.346 INFO:tasks.workunit.client.1.smithi192.stdout:1/306: fdatasync d0/f1a 0 2024-04-12T05:08:58.348 INFO:tasks.workunit.client.0.smithi113.stdout:8/467: rename d9/l4c to d9/d16/d1c/l98 0 2024-04-12T05:08:58.348 INFO:tasks.workunit.client.1.smithi192.stdout:3/273: symlink d5/d13/d1a/l5e 0 2024-04-12T05:08:58.348 INFO:tasks.workunit.client.0.smithi113.stdout:6/418: mknod d8/de/d13/d11/d36/d3d/d4c/d26/c95 0 2024-04-12T05:08:58.349 INFO:tasks.workunit.client.1.smithi192.stdout:0/334: symlink d0/d6/d15/d1b/l77 0 2024-04-12T05:08:58.353 INFO:tasks.workunit.client.1.smithi192.stdout:3/274: mkdir d5/d13/d1a/d5f 0 2024-04-12T05:08:58.354 INFO:tasks.workunit.client.1.smithi192.stdout:9/263: dwrite d11/d25/f52 [0,4194304] 0 2024-04-12T05:08:58.354 INFO:tasks.workunit.client.0.smithi113.stdout:6/419: mknod d8/de/d13/d11/d36/d3d/d4c/d31/d8e/d63/c96 0 2024-04-12T05:08:58.357 INFO:tasks.workunit.client.0.smithi113.stdout:5/376: dwrite d0/d10/f58 [0,4194304] 0 2024-04-12T05:08:58.359 INFO:tasks.workunit.client.0.smithi113.stdout:0/455: write d1/d6/d16/f47 [638025,77368] 0 2024-04-12T05:08:58.363 INFO:tasks.workunit.client.0.smithi113.stdout:5/377: unlink d0/d10/d27/l4d 0 2024-04-12T05:08:58.364 INFO:tasks.workunit.client.1.smithi192.stdout:9/264: truncate d11/d15/f42 418722 0 2024-04-12T05:08:58.365 INFO:tasks.workunit.client.0.smithi113.stdout:5/378: stat d0/d10/d27/c2c 0 2024-04-12T05:08:58.366 INFO:tasks.workunit.client.0.smithi113.stdout:6/420: dwrite d8/de/d13/d11/d36/d3d/f90 [0,4194304] 0 2024-04-12T05:08:58.368 INFO:tasks.workunit.client.0.smithi113.stdout:5/379: dread - d0/d10/d27/f57 zero size 2024-04-12T05:08:58.368 INFO:tasks.workunit.client.1.smithi192.stdout:1/307: fdatasync d0/d1b/f35 0 2024-04-12T05:08:58.370 INFO:tasks.workunit.client.1.smithi192.stdout:3/275: dread d5/f19 [0,4194304] 0 2024-04-12T05:08:58.370 INFO:tasks.workunit.client.0.smithi113.stdout:8/468: write d9/d1b/f46 [1242981,102212] 0 2024-04-12T05:08:58.371 INFO:tasks.workunit.client.0.smithi113.stdout:5/380: chown d0/d10/d27/d1d/c43 303750982 1 2024-04-12T05:08:58.371 INFO:tasks.workunit.client.1.smithi192.stdout:9/265: creat d11/d15/d1e/f59 x:0 0 0 2024-04-12T05:08:58.372 INFO:tasks.workunit.client.0.smithi113.stdout:0/456: rename d1/d6/d16/d2a/d50/l63 to d1/d6/da/l8c 0 2024-04-12T05:08:58.372 INFO:tasks.workunit.client.1.smithi192.stdout:3/276: write d5/f14 [7418816,86024] 0 2024-04-12T05:08:58.373 INFO:tasks.workunit.client.0.smithi113.stdout:6/421: rmdir d8/de/d17 39 2024-04-12T05:08:58.374 INFO:tasks.workunit.client.1.smithi192.stdout:0/335: dwrite d0/d6/d15/d19/f31 [4194304,4194304] 0 2024-04-12T05:08:58.376 INFO:tasks.workunit.client.1.smithi192.stdout:0/336: chown d0/d6/d15/d1b/d46/c6e 8729 1 2024-04-12T05:08:58.377 INFO:tasks.workunit.client.1.smithi192.stdout:9/266: mknod d11/d15/d21/d2c/d1f/c5a 0 2024-04-12T05:08:58.377 INFO:tasks.workunit.client.0.smithi113.stdout:8/469: creat d9/d1b/d8e/d93/d92/f99 x:0 0 0 2024-04-12T05:08:58.379 INFO:tasks.workunit.client.1.smithi192.stdout:9/267: dread - d11/d37/f56 zero size 2024-04-12T05:08:58.380 INFO:tasks.workunit.client.0.smithi113.stdout:5/381: unlink d0/d10/l4c 0 2024-04-12T05:08:58.380 INFO:tasks.workunit.client.1.smithi192.stdout:3/277: dread d5/f1c [0,4194304] 0 2024-04-12T05:08:58.381 INFO:tasks.workunit.client.0.smithi113.stdout:5/382: fsync d0/d10/f1b 0 2024-04-12T05:08:58.382 INFO:tasks.workunit.client.0.smithi113.stdout:0/457: creat d1/d6/d16/d2a/d50/d52/d7e/f8d x:0 0 0 2024-04-12T05:08:58.383 INFO:tasks.workunit.client.0.smithi113.stdout:6/422: creat d8/de/d13/d11/d36/d89/f97 x:0 0 0 2024-04-12T05:08:58.385 INFO:tasks.workunit.client.0.smithi113.stdout:6/423: write d8/de/d13/fb [2166707,75815] 0 2024-04-12T05:08:58.386 INFO:tasks.workunit.client.0.smithi113.stdout:8/470: rename d9/d16/d1c/d59/d68/c95 to d9/d1b/d8e/d93/d32/d48/d8d/d4a/c9a 0 2024-04-12T05:08:58.388 INFO:tasks.workunit.client.1.smithi192.stdout:0/337: creat d0/d6/d15/d55/d2e/d59/f78 x:0 0 0 2024-04-12T05:08:58.388 INFO:tasks.workunit.client.0.smithi113.stdout:5/383: mknod d0/d10/d6e/d7a/c7b 0 2024-04-12T05:08:58.390 INFO:tasks.workunit.client.1.smithi192.stdout:0/338: write d0/d6/d15/d1b/d46/f4d [1058937,71967] 0 2024-04-12T05:08:58.391 INFO:tasks.workunit.client.0.smithi113.stdout:0/458: mkdir d1/d6/d8e 0 2024-04-12T05:08:58.393 INFO:tasks.workunit.client.0.smithi113.stdout:8/471: rename d9/d1b/d8e/d93/d32/d48/d8d/f57 to d9/d1b/d96/f9b 0 2024-04-12T05:08:58.394 INFO:tasks.workunit.client.1.smithi192.stdout:9/268: rename d11/d25/d36/c41 to d11/d15/d21/d2c/d1f/d50/c5b 0 2024-04-12T05:08:58.398 INFO:tasks.workunit.client.0.smithi113.stdout:6/424: rename d8/de/d13/d11/d36/d3d/d4c/d31/l44 to d8/d1d/l98 0 2024-04-12T05:08:58.400 INFO:tasks.workunit.client.1.smithi192.stdout:0/339: dwrite d0/d6/d15/d1b/d46/f4d [0,4194304] 0 2024-04-12T05:08:58.400 INFO:tasks.workunit.client.0.smithi113.stdout:0/459: mkdir d1/d6/d16/d8f 0 2024-04-12T05:08:58.401 INFO:tasks.workunit.client.1.smithi192.stdout:3/278: truncate d5/d13/d3c/d36/f49 1732253 0 2024-04-12T05:08:58.402 INFO:tasks.workunit.client.0.smithi113.stdout:0/460: truncate d1/f75 277854 0 2024-04-12T05:08:58.404 INFO:tasks.workunit.client.0.smithi113.stdout:6/425: dwrite d8/d45/f5d [0,4194304] 0 2024-04-12T05:08:58.405 INFO:tasks.workunit.client.0.smithi113.stdout:5/384: link d0/d10/d27/d1d/d41/l69 d0/d10/d27/l7c 0 2024-04-12T05:08:58.405 INFO:tasks.workunit.client.1.smithi192.stdout:3/279: creat d5/d13/d1a/d53/f60 x:0 0 0 2024-04-12T05:08:58.412 INFO:tasks.workunit.client.1.smithi192.stdout:9/269: dwrite d11/d15/d21/d2c/f1a [4194304,4194304] 0 2024-04-12T05:08:58.412 INFO:tasks.workunit.client.1.smithi192.stdout:9/270: read - d11/d37/f56 zero size 2024-04-12T05:08:58.412 INFO:tasks.workunit.client.0.smithi113.stdout:0/461: dwrite d1/d6/da/f5c [0,4194304] 0 2024-04-12T05:08:58.413 INFO:tasks.workunit.client.1.smithi192.stdout:9/271: fdatasync d11/d15/f45 0 2024-04-12T05:08:58.415 INFO:tasks.workunit.client.0.smithi113.stdout:5/385: fdatasync d0/d10/d27/d1d/d3f/f6c 0 2024-04-12T05:08:58.415 INFO:tasks.workunit.client.0.smithi113.stdout:8/472: link d9/d1b/d8e/d93/d32/d48/d8d/d4a/c8f d9/d16/d65/c9c 0 2024-04-12T05:08:58.415 INFO:tasks.workunit.client.0.smithi113.stdout:0/462: dread - d1/d6/f64 zero size 2024-04-12T05:08:58.415 INFO:tasks.workunit.client.0.smithi113.stdout:6/426: rename d8/de/d13/d11/d36 to d8/de/d13/d11/d36/d3d/d93/d99 22 2024-04-12T05:08:58.417 INFO:tasks.workunit.client.0.smithi113.stdout:0/463: fsync d1/d9/f8a 0 2024-04-12T05:08:58.417 INFO:tasks.workunit.client.0.smithi113.stdout:5/386: dread - d0/d10/d27/d1d/d38/f52 zero size 2024-04-12T05:08:58.418 INFO:tasks.workunit.client.1.smithi192.stdout:0/340: rename d0/d6/d15/d1b/d3a/d4f/d33 to d0/d6/d15/d1b/d3a/d79 0 2024-04-12T05:08:58.421 INFO:tasks.workunit.client.1.smithi192.stdout:9/272: creat d11/d15/f5c x:0 0 0 2024-04-12T05:08:58.422 INFO:tasks.workunit.client.0.smithi113.stdout:6/427: mknod d8/de/d13/d11/d36/d3d/d4c/d31/d8e/c9a 0 2024-04-12T05:08:58.424 INFO:tasks.workunit.client.0.smithi113.stdout:5/387: mknod d0/c7d 0 2024-04-12T05:08:58.424 INFO:tasks.workunit.client.0.smithi113.stdout:8/473: dwrite d9/d1b/d8e/d93/d32/f60 [0,4194304] 0 2024-04-12T05:08:58.425 INFO:tasks.workunit.client.1.smithi192.stdout:3/280: write d5/f2d [192810,106921] 0 2024-04-12T05:08:58.427 INFO:tasks.workunit.client.1.smithi192.stdout:9/273: symlink d11/d15/d21/d2c/d1f/l5d 0 2024-04-12T05:08:58.429 INFO:tasks.workunit.client.1.smithi192.stdout:0/341: mknod d0/d6/d15/d1b/d46/d73/d35/c7a 0 2024-04-12T05:08:58.429 INFO:tasks.workunit.client.0.smithi113.stdout:8/474: creat d9/f9d x:0 0 0 2024-04-12T05:08:58.429 INFO:tasks.workunit.client.0.smithi113.stdout:5/388: mkdir d0/d10/d27/d1d/d3f/d7e 0 2024-04-12T05:08:58.430 INFO:tasks.workunit.client.0.smithi113.stdout:8/475: dread - d9/d1b/d8e/d93/d32/d48/f85 zero size 2024-04-12T05:08:58.430 INFO:tasks.workunit.client.0.smithi113.stdout:6/428: creat d8/de/d13/d11/d36/d3d/d4c/d31/d8e/f9b x:0 0 0 2024-04-12T05:08:58.432 INFO:tasks.workunit.client.1.smithi192.stdout:3/281: mkdir d5/d13/d3a/d4a/d61 0 2024-04-12T05:08:58.432 INFO:tasks.workunit.client.0.smithi113.stdout:5/389: symlink d0/l7f 0 2024-04-12T05:08:58.433 INFO:tasks.workunit.client.1.smithi192.stdout:9/274: rename d11/f13 to d11/d25/d36/d3d/f5e 0 2024-04-12T05:08:58.434 INFO:tasks.workunit.client.0.smithi113.stdout:8/476: creat d9/f9e x:0 0 0 2024-04-12T05:08:58.434 INFO:tasks.workunit.client.0.smithi113.stdout:6/429: read d8/d45/f49 [3554101,4182] 0 2024-04-12T05:08:58.437 INFO:tasks.workunit.client.0.smithi113.stdout:5/390: mkdir d0/d10/d27/d5f/d80 0 2024-04-12T05:08:58.437 INFO:tasks.workunit.client.1.smithi192.stdout:0/342: mkdir d0/d7b 0 2024-04-12T05:08:58.442 INFO:tasks.workunit.client.0.smithi113.stdout:8/477: creat d9/d16/d1c/d59/d78/f9f x:0 0 0 2024-04-12T05:08:58.442 INFO:tasks.workunit.client.0.smithi113.stdout:0/464: truncate d1/d6/da/f5c 4150784 0 2024-04-12T05:08:58.443 INFO:tasks.workunit.client.0.smithi113.stdout:6/430: mknod d8/de/d13/d11/d36/d3d/d4c/d31/d8e/d63/d66/c9c 0 2024-04-12T05:08:58.443 INFO:tasks.workunit.client.1.smithi192.stdout:0/343: write d0/d6/d15/d55/f6c [344148,121922] 0 2024-04-12T05:08:58.446 INFO:tasks.workunit.client.1.smithi192.stdout:0/344: mkdir d0/d7b/d7c 0 2024-04-12T05:08:58.447 INFO:tasks.workunit.client.0.smithi113.stdout:5/391: symlink d0/d10/d27/d1d/d38/d75/l81 0 2024-04-12T05:08:58.450 INFO:tasks.workunit.client.0.smithi113.stdout:8/478: creat d9/d1b/d8e/d93/d32/d48/d8d/fa0 x:0 0 0 2024-04-12T05:08:58.450 INFO:tasks.workunit.client.0.smithi113.stdout:6/431: creat d8/de/d13/d11/d36/d3d/d4c/d31/d8e/f9d x:0 0 0 2024-04-12T05:08:58.451 INFO:tasks.workunit.client.0.smithi113.stdout:0/465: dread d1/d6/d16/f47 [0,4194304] 0 2024-04-12T05:08:58.453 INFO:tasks.workunit.client.0.smithi113.stdout:5/392: rename d0/d10/d27/d1d/d38/d3b to d0/d10/d27/d1d/d38/d3b/d6b/d71/d82 22 2024-04-12T05:08:58.453 INFO:tasks.workunit.client.0.smithi113.stdout:6/432: symlink d8/d1d/l9e 0 2024-04-12T05:08:58.455 INFO:tasks.workunit.client.0.smithi113.stdout:5/393: readlink d0/d10/l30 0 2024-04-12T05:08:58.455 INFO:tasks.workunit.client.0.smithi113.stdout:8/479: creat d9/d1b/d8e/d93/d32/d5b/fa1 x:0 0 0 2024-04-12T05:08:58.456 INFO:tasks.workunit.client.0.smithi113.stdout:5/394: chown d0/d10/d6e/f35 0 1 2024-04-12T05:08:58.457 INFO:tasks.workunit.client.0.smithi113.stdout:0/466: getdents d1/d6/d34 0 2024-04-12T05:08:58.457 INFO:tasks.workunit.client.0.smithi113.stdout:8/480: chown d9/d1b/d8e/d93/d32/d48/f49 0 1 2024-04-12T05:08:58.462 INFO:tasks.workunit.client.1.smithi192.stdout:3/282: dread d5/d13/f2f [4194304,4194304] 0 2024-04-12T05:08:58.462 INFO:tasks.workunit.client.0.smithi113.stdout:0/467: symlink d1/d6/d16/d2a/d2c/l90 0 2024-04-12T05:08:58.463 INFO:tasks.workunit.client.1.smithi192.stdout:3/283: chown d5/d13/d1a/l38 0 1 2024-04-12T05:08:58.463 INFO:tasks.workunit.client.0.smithi113.stdout:5/395: truncate d0/d10/d27/d1d/d3f/f47 454085 0 2024-04-12T05:08:58.464 INFO:tasks.workunit.client.1.smithi192.stdout:3/284: write d5/d13/d1a/f3b [5620158,15196] 0 2024-04-12T05:08:58.466 INFO:tasks.workunit.client.0.smithi113.stdout:6/433: link d8/de/d13/d11/d36/d3d/c8b d8/de/d13/d11/d36/d3d/d4c/d26/d64/c9f 0 2024-04-12T05:08:58.467 INFO:tasks.workunit.client.1.smithi192.stdout:9/275: dread f10 [0,4194304] 0 2024-04-12T05:08:58.467 INFO:tasks.workunit.client.1.smithi192.stdout:3/285: symlink d5/d13/d1a/d53/l62 0 2024-04-12T05:08:58.468 INFO:tasks.workunit.client.1.smithi192.stdout:3/286: write d5/d13/d1a/d53/f60 [187675,75951] 0 2024-04-12T05:08:58.469 INFO:tasks.workunit.client.1.smithi192.stdout:9/276: unlink cf 0 2024-04-12T05:08:58.469 INFO:tasks.workunit.client.0.smithi113.stdout:0/468: dwrite d1/d6/d16/d2a/d2c/f7b [0,4194304] 0 2024-04-12T05:08:58.470 INFO:tasks.workunit.client.1.smithi192.stdout:9/277: chown d11/d15/f5c 1731 1 2024-04-12T05:08:58.471 INFO:tasks.workunit.client.1.smithi192.stdout:3/287: chown d5/d13/d1a/l47 5 1 2024-04-12T05:08:58.474 INFO:tasks.workunit.client.0.smithi113.stdout:6/434: rename d8/de/d13/d11/d36/d3d/d4c/d26/d5b/f88 to d8/d1d/fa0 0 2024-04-12T05:08:58.478 INFO:tasks.workunit.client.0.smithi113.stdout:0/469: dwrite d1/d9/f45 [0,4194304] 0 2024-04-12T05:08:58.479 INFO:tasks.workunit.client.1.smithi192.stdout:9/278: rename d11/d15/d21/f24 to d11/d25/d36/f5f 0 2024-04-12T05:08:58.479 INFO:tasks.workunit.client.1.smithi192.stdout:9/279: stat d11/d15 0 2024-04-12T05:08:58.480 INFO:tasks.workunit.client.0.smithi113.stdout:8/481: rmdir d9 39 2024-04-12T05:08:58.480 INFO:tasks.workunit.client.1.smithi192.stdout:3/288: symlink d5/d13/d3a/d4a/d61/l63 0 2024-04-12T05:08:58.481 INFO:tasks.workunit.client.1.smithi192.stdout:3/289: chown d5/d13/d3c/f24 122985 1 2024-04-12T05:08:58.486 INFO:tasks.workunit.client.1.smithi192.stdout:3/290: unlink d5/l3d 0 2024-04-12T05:08:58.486 INFO:tasks.workunit.client.0.smithi113.stdout:5/396: write d0/f6 [4236707,8214] 0 2024-04-12T05:08:58.487 INFO:tasks.workunit.client.0.smithi113.stdout:8/482: dread d9/d1b/d8e/d93/f28 [0,4194304] 0 2024-04-12T05:08:58.489 INFO:tasks.workunit.client.1.smithi192.stdout:3/291: write d5/d13/d3c/f59 [1713705,59592] 0 2024-04-12T05:08:58.490 INFO:tasks.workunit.client.0.smithi113.stdout:8/483: write d9/d1b/d8e/d93/d32/d48/f49 [4809071,94627] 0 2024-04-12T05:08:58.490 INFO:tasks.workunit.client.0.smithi113.stdout:0/470: creat d1/d6/f91 x:0 0 0 2024-04-12T05:08:58.494 INFO:tasks.workunit.client.1.smithi192.stdout:3/292: dread d5/d13/f42 [0,4194304] 0 2024-04-12T05:08:58.496 INFO:tasks.workunit.client.0.smithi113.stdout:8/484: write d9/d16/d3c/f52 [1401709,72866] 0 2024-04-12T05:08:58.499 INFO:tasks.workunit.client.1.smithi192.stdout:3/293: getdents d5/d13 0 2024-04-12T05:08:58.499 INFO:tasks.workunit.client.0.smithi113.stdout:5/397: dwrite d0/d10/d27/d1d/d38/d3b/f61 [0,4194304] 0 2024-04-12T05:08:58.500 INFO:tasks.workunit.client.0.smithi113.stdout:0/471: rename d1/d6/c21 to d1/d6/c92 0 2024-04-12T05:08:58.500 INFO:tasks.workunit.client.1.smithi192.stdout:3/294: write d5/d13/d1a/f3b [1880439,44690] 0 2024-04-12T05:08:58.503 INFO:tasks.workunit.client.0.smithi113.stdout:6/435: dwrite d8/de/d17/f27 [0,4194304] 0 2024-04-12T05:08:58.504 INFO:tasks.workunit.client.1.smithi192.stdout:3/295: creat d5/d13/d3a/d4a/d61/f64 x:0 0 0 2024-04-12T05:08:58.505 INFO:tasks.workunit.client.0.smithi113.stdout:0/472: symlink d1/d6/d16/d2a/l93 0 2024-04-12T05:08:58.506 INFO:tasks.workunit.client.0.smithi113.stdout:0/473: chown d1/d6/da/c7a 425163614 1 2024-04-12T05:08:58.511 INFO:tasks.workunit.client.1.smithi192.stdout:8/233: sync 2024-04-12T05:08:58.511 INFO:tasks.workunit.client.0.smithi113.stdout:6/436: getdents d8 0 2024-04-12T05:08:58.512 INFO:tasks.workunit.client.0.smithi113.stdout:6/437: write d8/de/d13/d11/d36/d3d/d4c/d31/f62 [100899,108647] 0 2024-04-12T05:08:58.513 INFO:tasks.workunit.client.1.smithi192.stdout:3/296: dwrite d5/f19 [0,4194304] 0 2024-04-12T05:08:58.515 INFO:tasks.workunit.client.1.smithi192.stdout:3/297: write d5/d13/d1a/f3b [5938572,51400] 0 2024-04-12T05:08:58.515 INFO:tasks.workunit.client.1.smithi192.stdout:8/234: fsync d6/df/d37/f4a 0 2024-04-12T05:08:58.517 INFO:tasks.workunit.client.0.smithi113.stdout:0/474: rename d1/c57 to d1/d6/da/d36/c94 0 2024-04-12T05:08:58.517 INFO:tasks.workunit.client.0.smithi113.stdout:8/485: truncate d9/d16/d1c/f2d 1502112 0 2024-04-12T05:08:58.518 INFO:tasks.workunit.client.0.smithi113.stdout:0/475: write d1/d9/f2b [1509543,14491] 0 2024-04-12T05:08:58.519 INFO:tasks.workunit.client.1.smithi192.stdout:3/298: symlink d5/d13/d3a/d4e/l65 0 2024-04-12T05:08:58.520 INFO:tasks.workunit.client.1.smithi192.stdout:8/235: getdents d6/df/d38/d3b 0 2024-04-12T05:08:58.521 INFO:tasks.workunit.client.0.smithi113.stdout:0/476: creat d1/d9/f95 x:0 0 0 2024-04-12T05:08:58.521 INFO:tasks.workunit.client.0.smithi113.stdout:8/486: dread d9/d1b/f4f [0,4194304] 0 2024-04-12T05:08:58.522 INFO:tasks.workunit.client.1.smithi192.stdout:8/236: truncate d6/da/f30 611541 0 2024-04-12T05:08:58.522 INFO:tasks.workunit.client.1.smithi192.stdout:3/299: creat d5/d13/d3a/d4a/f66 x:0 0 0 2024-04-12T05:08:58.522 INFO:tasks.workunit.client.0.smithi113.stdout:8/487: read - d9/f9d zero size 2024-04-12T05:08:58.523 INFO:tasks.workunit.client.0.smithi113.stdout:8/488: read - d9/d1b/d8e/d93/d32/d48/d8d/f73 zero size 2024-04-12T05:08:58.524 INFO:tasks.workunit.client.0.smithi113.stdout:0/477: mknod d1/d6/c96 0 2024-04-12T05:08:58.524 INFO:tasks.workunit.client.0.smithi113.stdout:0/478: dread - d1/d9/f95 zero size 2024-04-12T05:08:58.528 INFO:tasks.workunit.client.0.smithi113.stdout:0/479: readlink d1/d6/d34/l8b 0 2024-04-12T05:08:58.530 INFO:tasks.workunit.client.0.smithi113.stdout:0/480: rmdir d1 39 2024-04-12T05:08:58.532 INFO:tasks.workunit.client.1.smithi192.stdout:3/300: dwrite d5/d13/f18 [4194304,4194304] 0 2024-04-12T05:08:58.532 INFO:tasks.workunit.client.0.smithi113.stdout:7/478: sync 2024-04-12T05:08:58.534 INFO:tasks.workunit.client.0.smithi113.stdout:7/479: write d0/dd/f30 [1287434,77332] 0 2024-04-12T05:08:58.536 INFO:tasks.workunit.client.1.smithi192.stdout:3/301: creat d5/d13/d3a/d4a/f67 x:0 0 0 2024-04-12T05:08:58.536 INFO:tasks.workunit.client.0.smithi113.stdout:0/481: symlink d1/d6/da/d36/l97 0 2024-04-12T05:08:58.538 INFO:tasks.workunit.client.0.smithi113.stdout:0/482: fsync d1/d6/d16/d2a/d50/f61 0 2024-04-12T05:08:58.539 INFO:tasks.workunit.client.0.smithi113.stdout:7/480: dread d0/dc/d6/d1a/f58 [0,4194304] 0 2024-04-12T05:08:58.539 INFO:tasks.workunit.client.1.smithi192.stdout:8/237: truncate d6/da/f1e 282040 0 2024-04-12T05:08:58.540 INFO:tasks.workunit.client.0.smithi113.stdout:0/483: stat d1/d6/da/l15 0 2024-04-12T05:08:58.542 INFO:tasks.workunit.client.0.smithi113.stdout:7/481: symlink d0/dd/d11/d99/la3 0 2024-04-12T05:08:58.543 INFO:tasks.workunit.client.1.smithi192.stdout:3/302: dwrite d5/d13/d3a/d4a/f5a [0,4194304] 0 2024-04-12T05:08:58.544 INFO:tasks.workunit.client.0.smithi113.stdout:0/484: truncate d1/d6/d16/d2a/d50/d52/f81 37362 0 2024-04-12T05:08:58.545 INFO:tasks.workunit.client.0.smithi113.stdout:0/485: write d1/d6/d16/d2a/d2c/f7b [1122638,126775] 0 2024-04-12T05:08:58.549 INFO:tasks.workunit.client.0.smithi113.stdout:7/482: rename d0/dd/d11/d44/d5c/d85/f98 to d0/dd/d11/d44/d5c/d85/fa4 0 2024-04-12T05:08:58.552 INFO:tasks.workunit.client.1.smithi192.stdout:3/303: creat d5/d13/d3c/f68 x:0 0 0 2024-04-12T05:08:58.552 INFO:tasks.workunit.client.1.smithi192.stdout:3/304: read d5/d13/d3c/f59 [757886,113882] 0 2024-04-12T05:08:58.557 INFO:tasks.workunit.client.0.smithi113.stdout:0/486: rename d1/f3d to d1/d6/d16/d8f/f98 0 2024-04-12T05:08:58.558 INFO:tasks.workunit.client.0.smithi113.stdout:0/487: chown d1/d6/f30 8124309 1 2024-04-12T05:08:58.560 INFO:tasks.workunit.client.0.smithi113.stdout:0/488: creat d1/d6/d16/d2a/d50/f99 x:0 0 0 2024-04-12T05:08:58.561 INFO:tasks.workunit.client.0.smithi113.stdout:0/489: write d1/d9/f45 [3336129,51161] 0 2024-04-12T05:08:58.561 INFO:tasks.workunit.client.1.smithi192.stdout:3/305: rmdir d5 39 2024-04-12T05:08:58.562 INFO:tasks.workunit.client.0.smithi113.stdout:0/490: fsync d1/f2f 0 2024-04-12T05:08:58.562 INFO:tasks.workunit.client.1.smithi192.stdout:8/238: write d6/da/f1e [495809,78663] 0 2024-04-12T05:08:58.563 INFO:tasks.workunit.client.1.smithi192.stdout:8/239: fsync d6/df/d37/f48 0 2024-04-12T05:08:58.565 INFO:tasks.workunit.client.1.smithi192.stdout:3/306: read d5/d13/d3c/f25 [6540505,10466] 0 2024-04-12T05:08:58.566 INFO:tasks.workunit.client.1.smithi192.stdout:3/307: readlink d5/d13/d3c/d26/l4c 0 2024-04-12T05:08:58.568 INFO:tasks.workunit.client.0.smithi113.stdout:0/491: stat d1/d6/da/d36/c94 0 2024-04-12T05:08:58.569 INFO:tasks.workunit.client.1.smithi192.stdout:3/308: write d5/d13/d1a/d53/f58 [149080,50888] 0 2024-04-12T05:08:58.569 INFO:tasks.workunit.client.0.smithi113.stdout:0/492: read - d1/d6/d16/d2a/d50/f99 zero size 2024-04-12T05:08:58.576 INFO:tasks.workunit.client.1.smithi192.stdout:8/240: dwrite d6/d10/f3f [0,4194304] 0 2024-04-12T05:08:58.578 INFO:tasks.workunit.client.0.smithi113.stdout:0/493: dwrite d1/d6/d16/d8f/f98 [0,4194304] 0 2024-04-12T05:08:58.579 INFO:tasks.workunit.client.0.smithi113.stdout:0/494: dread - d1/d6/d16/f67 zero size 2024-04-12T05:08:58.580 INFO:tasks.workunit.client.1.smithi192.stdout:8/241: read d6/d10/f3d [1021363,70532] 0 2024-04-12T05:08:58.582 INFO:tasks.workunit.client.0.smithi113.stdout:0/495: stat d1/d6/d16/c6f 0 2024-04-12T05:08:58.586 INFO:tasks.workunit.client.0.smithi113.stdout:0/496: chown d1/l4a 3795 1 2024-04-12T05:08:58.586 INFO:tasks.workunit.client.0.smithi113.stdout:0/497: readlink d1/d43/l88 0 2024-04-12T05:08:58.590 INFO:tasks.workunit.client.0.smithi113.stdout:0/498: creat d1/d43/f9a x:0 0 0 2024-04-12T05:08:58.593 INFO:tasks.workunit.client.0.smithi113.stdout:0/499: creat d1/d6/d8e/f9b x:0 0 0 2024-04-12T05:08:58.593 INFO:tasks.workunit.client.0.smithi113.stdout:0/500: readlink d1/d6/d16/d2a/l93 0 2024-04-12T05:08:58.597 INFO:tasks.workunit.client.0.smithi113.stdout:0/501: getdents d1/d80 0 2024-04-12T05:08:58.601 INFO:tasks.workunit.client.0.smithi113.stdout:0/502: link d1/d6/c96 d1/d6/da/d36/c9c 0 2024-04-12T05:08:58.601 INFO:tasks.workunit.client.1.smithi192.stdout:8/242: fsync d6/df/f43 0 2024-04-12T05:08:58.603 INFO:tasks.workunit.client.1.smithi192.stdout:8/243: truncate d6/df/d38/f41 254932 0 2024-04-12T05:08:58.603 INFO:tasks.workunit.client.1.smithi192.stdout:8/244: write d6/f9 [9303388,3204] 0 2024-04-12T05:08:58.604 INFO:tasks.workunit.client.0.smithi113.stdout:0/503: rename d1/d6/f64 to d1/d9/f9d 0 2024-04-12T05:08:58.604 INFO:tasks.workunit.client.1.smithi192.stdout:8/245: fdatasync d6/df/f16 0 2024-04-12T05:08:58.605 INFO:tasks.workunit.client.0.smithi113.stdout:0/504: write d1/d43/f9a [111164,88641] 0 2024-04-12T05:08:58.605 INFO:tasks.workunit.client.1.smithi192.stdout:8/246: fsync d6/d10/f11 0 2024-04-12T05:08:58.606 INFO:tasks.workunit.client.0.smithi113.stdout:0/505: readlink d1/d6/d16/d2a/l93 0 2024-04-12T05:08:58.609 INFO:tasks.workunit.client.0.smithi113.stdout:0/506: creat d1/d6/d16/d8f/f9e x:0 0 0 2024-04-12T05:08:58.611 INFO:tasks.workunit.client.1.smithi192.stdout:8/247: dwrite d6/d21/f23 [0,4194304] 0 2024-04-12T05:08:58.611 INFO:tasks.workunit.client.0.smithi113.stdout:0/507: rename d1/d6/da/c7a to d1/d6/d8e/c9f 0 2024-04-12T05:08:58.612 INFO:tasks.workunit.client.0.smithi113.stdout:0/508: read - d1/d9/f6c zero size 2024-04-12T05:08:58.613 INFO:tasks.workunit.client.1.smithi192.stdout:8/248: creat d6/df/f4d x:0 0 0 2024-04-12T05:08:58.614 INFO:tasks.workunit.client.1.smithi192.stdout:8/249: read - d6/df/d37/f48 zero size 2024-04-12T05:08:58.615 INFO:tasks.workunit.client.0.smithi113.stdout:0/509: creat d1/d6/d16/d2a/d50/d52/fa0 x:0 0 0 2024-04-12T05:08:58.616 INFO:tasks.workunit.client.0.smithi113.stdout:0/510: dread - d1/d6/d16/d8f/f9e zero size 2024-04-12T05:08:58.619 INFO:tasks.workunit.client.0.smithi113.stdout:0/511: symlink d1/d6/d16/la1 0 2024-04-12T05:08:58.621 INFO:tasks.workunit.client.0.smithi113.stdout:0/512: creat d1/d6/d16/d2a/d50/d52/d7e/fa2 x:0 0 0 2024-04-12T05:08:58.625 INFO:tasks.workunit.client.1.smithi192.stdout:8/250: rmdir d6/df 39 2024-04-12T05:08:58.627 INFO:tasks.workunit.client.1.smithi192.stdout:8/251: stat d6/df/d31/f36 0 2024-04-12T05:08:58.629 INFO:tasks.workunit.client.0.smithi113.stdout:0/513: getdents d1/d6/d16/d2a/d50/d52 0 2024-04-12T05:08:58.631 INFO:tasks.workunit.client.1.smithi192.stdout:8/252: truncate d6/d10/f11 2255150 0 2024-04-12T05:08:58.631 INFO:tasks.workunit.client.1.smithi192.stdout:8/253: stat d6/f9 0 2024-04-12T05:08:58.631 INFO:tasks.workunit.client.0.smithi113.stdout:0/514: chown d1/d6/d16/d2a/l49 857161 1 2024-04-12T05:08:58.633 INFO:tasks.workunit.client.1.smithi192.stdout:8/254: symlink d6/df/l4e 0 2024-04-12T05:08:58.633 INFO:tasks.workunit.client.0.smithi113.stdout:0/515: unlink d1/d9/f6c 0 2024-04-12T05:08:58.636 INFO:tasks.workunit.client.1.smithi192.stdout:8/255: rename d6/d3e to d6/df/d37/d4f 0 2024-04-12T05:08:58.639 INFO:tasks.workunit.client.1.smithi192.stdout:8/256: mknod d6/df/d37/c50 0 2024-04-12T05:08:58.639 INFO:tasks.workunit.client.1.smithi192.stdout:8/257: chown d6/df/f15 1292835 1 2024-04-12T05:08:58.640 INFO:tasks.workunit.client.0.smithi113.stdout:0/516: dwrite d1/d6/f1d [0,4194304] 0 2024-04-12T05:08:58.641 INFO:tasks.workunit.client.1.smithi192.stdout:8/258: rmdir d6/da 39 2024-04-12T05:08:58.642 INFO:tasks.workunit.client.1.smithi192.stdout:8/259: chown d6/ce 491 1 2024-04-12T05:08:58.642 INFO:tasks.workunit.client.0.smithi113.stdout:0/517: chown d1/d6/d16/d2a/d50/f7d 2565187 1 2024-04-12T05:08:58.644 INFO:tasks.workunit.client.0.smithi113.stdout:0/518: read - d1/d6/d16/f67 zero size 2024-04-12T05:08:58.645 INFO:tasks.workunit.client.1.smithi192.stdout:8/260: mkdir d6/df/d37/d4f/d51 0 2024-04-12T05:08:58.647 INFO:tasks.workunit.client.0.smithi113.stdout:0/519: symlink d1/d6/d16/d8f/la3 0 2024-04-12T05:08:58.648 INFO:tasks.workunit.client.1.smithi192.stdout:8/261: chown d6/da 2375 1 2024-04-12T05:08:58.648 INFO:tasks.workunit.client.1.smithi192.stdout:8/262: readlink l4 0 2024-04-12T05:08:58.649 INFO:tasks.workunit.client.0.smithi113.stdout:0/520: readlink d1/d6/da/l8c 0 2024-04-12T05:08:58.650 INFO:tasks.workunit.client.0.smithi113.stdout:0/521: write d1/d6/f1d [664764,114107] 0 2024-04-12T05:08:58.652 INFO:tasks.workunit.client.1.smithi192.stdout:8/263: creat d6/d4b/f52 x:0 0 0 2024-04-12T05:08:58.652 INFO:tasks.workunit.client.0.smithi113.stdout:0/522: symlink d1/d6/d16/la4 0 2024-04-12T05:08:58.654 INFO:tasks.workunit.client.1.smithi192.stdout:8/264: symlink d6/df/l53 0 2024-04-12T05:08:58.657 INFO:tasks.workunit.client.1.smithi192.stdout:8/265: mkdir d6/df/d37/d4f/d4c/d54 0 2024-04-12T05:08:58.658 INFO:tasks.workunit.client.0.smithi113.stdout:0/523: dwrite d1/d9/f8a [0,4194304] 0 2024-04-12T05:08:58.659 INFO:tasks.workunit.client.0.smithi113.stdout:0/524: readlink d1/d6/da/l12 0 2024-04-12T05:08:58.661 INFO:tasks.workunit.client.1.smithi192.stdout:8/266: rename d6/df/f15 to d6/df/d37/d4f/d4c/f55 0 2024-04-12T05:08:58.661 INFO:tasks.workunit.client.1.smithi192.stdout:8/267: chown d6/ce 1553 1 2024-04-12T05:08:58.663 INFO:tasks.workunit.client.0.smithi113.stdout:0/525: getdents d1/d6/d16/d2a/d2c 0 2024-04-12T05:08:58.664 INFO:tasks.workunit.client.1.smithi192.stdout:8/268: unlink l4 0 2024-04-12T05:08:58.664 INFO:tasks.workunit.client.1.smithi192.stdout:8/269: read d6/d10/f3d [6087043,82560] 0 2024-04-12T05:08:58.667 INFO:tasks.workunit.client.1.smithi192.stdout:8/270: symlink d6/df/d37/d4f/l56 0 2024-04-12T05:08:58.668 INFO:tasks.workunit.client.1.smithi192.stdout:8/271: chown d6/df/d37/f48 241578 1 2024-04-12T05:08:58.669 INFO:tasks.workunit.client.1.smithi192.stdout:8/272: creat d6/df/f57 x:0 0 0 2024-04-12T05:08:58.670 INFO:tasks.workunit.client.0.smithi113.stdout:0/526: truncate d1/d6/f58 302418 0 2024-04-12T05:08:58.670 INFO:tasks.workunit.client.0.smithi113.stdout:0/527: write d1/d9/f95 [661320,79925] 0 2024-04-12T05:08:58.671 INFO:tasks.workunit.client.1.smithi192.stdout:8/273: rmdir d6/df/d31 39 2024-04-12T05:08:58.676 INFO:tasks.workunit.client.1.smithi192.stdout:8/274: dwrite d6/da/f3a [0,4194304] 0 2024-04-12T05:08:58.680 INFO:tasks.workunit.client.1.smithi192.stdout:8/275: creat d6/df/d37/d4f/d4c/f58 x:0 0 0 2024-04-12T05:08:58.686 INFO:tasks.workunit.client.1.smithi192.stdout:8/276: dwrite d6/f9 [4194304,4194304] 0 2024-04-12T05:08:58.694 INFO:tasks.workunit.client.1.smithi192.stdout:8/277: dwrite d6/df/d37/f4a [0,4194304] 0 2024-04-12T05:08:58.700 INFO:tasks.workunit.client.1.smithi192.stdout:8/278: read d6/df/d37/d4f/d4c/f55 [1714949,104195] 0 2024-04-12T05:08:58.701 INFO:tasks.workunit.client.1.smithi192.stdout:8/279: chown d6/df/d37/d4f/d4c/d54 3425753 1 2024-04-12T05:08:58.702 INFO:tasks.workunit.client.1.smithi192.stdout:8/280: write f3 [2532977,46692] 0 2024-04-12T05:08:58.702 INFO:tasks.workunit.client.1.smithi192.stdout:8/281: readlink d6/l17 0 2024-04-12T05:08:58.709 INFO:tasks.workunit.client.1.smithi192.stdout:8/282: dwrite d6/d21/f33 [0,4194304] 0 2024-04-12T05:08:58.721 INFO:tasks.workunit.client.1.smithi192.stdout:8/283: dread d6/f18 [0,4194304] 0 2024-04-12T05:08:58.722 INFO:tasks.workunit.client.1.smithi192.stdout:8/284: dread - d6/df/d37/d4f/d4c/f58 zero size 2024-04-12T05:08:58.724 INFO:tasks.workunit.client.1.smithi192.stdout:8/285: symlink d6/df/d37/d4f/d4c/l59 0 2024-04-12T05:08:58.729 INFO:tasks.workunit.client.1.smithi192.stdout:8/286: dwrite d6/df/d37/f4a [0,4194304] 0 2024-04-12T05:08:58.730 INFO:tasks.workunit.client.1.smithi192.stdout:8/287: chown d6/d21/f23 2 1 2024-04-12T05:08:58.732 INFO:tasks.workunit.client.1.smithi192.stdout:8/288: creat d6/df/d38/f5a x:0 0 0 2024-04-12T05:08:58.735 INFO:tasks.workunit.client.1.smithi192.stdout:8/289: mknod d6/df/d31/c5b 0 2024-04-12T05:08:58.737 INFO:tasks.workunit.client.1.smithi192.stdout:8/290: dread d6/da/f30 [0,4194304] 0 2024-04-12T05:08:58.738 INFO:tasks.workunit.client.1.smithi192.stdout:8/291: stat d6/d10/d46 0 2024-04-12T05:08:58.739 INFO:tasks.workunit.client.1.smithi192.stdout:8/292: creat d6/d1f/f5c x:0 0 0 2024-04-12T05:08:58.740 INFO:tasks.workunit.client.1.smithi192.stdout:8/293: truncate d6/d21/f44 99360 0 2024-04-12T05:08:58.741 INFO:tasks.workunit.client.1.smithi192.stdout:8/294: dread - d6/df/f1c zero size 2024-04-12T05:08:58.742 INFO:tasks.workunit.client.1.smithi192.stdout:6/414: sync 2024-04-12T05:08:58.742 INFO:tasks.workunit.client.1.smithi192.stdout:7/255: sync 2024-04-12T05:08:58.742 INFO:tasks.workunit.client.1.smithi192.stdout:5/333: sync 2024-04-12T05:08:58.742 INFO:tasks.workunit.client.1.smithi192.stdout:2/359: sync 2024-04-12T05:08:58.743 INFO:tasks.workunit.client.1.smithi192.stdout:4/399: sync 2024-04-12T05:08:58.744 INFO:tasks.workunit.client.1.smithi192.stdout:8/295: dread d6/d10/f45 [0,4194304] 0 2024-04-12T05:08:58.744 INFO:tasks.workunit.client.1.smithi192.stdout:7/256: mkdir d3/d7/d60 0 2024-04-12T05:08:58.745 INFO:tasks.workunit.client.1.smithi192.stdout:8/296: write d6/df/f16 [1576615,89120] 0 2024-04-12T05:08:58.745 INFO:tasks.workunit.client.1.smithi192.stdout:2/360: rmdir d1/d2d/d73 39 2024-04-12T05:08:58.746 INFO:tasks.workunit.client.1.smithi192.stdout:7/257: chown d3/d11/f2d 119 1 2024-04-12T05:08:58.747 INFO:tasks.workunit.client.1.smithi192.stdout:5/334: mkdir d0/d15/d3e/d77 0 2024-04-12T05:08:58.748 INFO:tasks.workunit.client.1.smithi192.stdout:7/258: chown d3/d7/d14 0 1 2024-04-12T05:08:58.748 INFO:tasks.workunit.client.1.smithi192.stdout:6/415: mknod d3/d8/d48/c8b 0 2024-04-12T05:08:58.750 INFO:tasks.workunit.client.1.smithi192.stdout:2/361: write d1/d8/d17/d19/d60/f66 [1132486,7670] 0 2024-04-12T05:08:58.750 INFO:tasks.workunit.client.1.smithi192.stdout:5/335: read - d0/d21/d5e/f6c zero size 2024-04-12T05:08:58.751 INFO:tasks.workunit.client.0.smithi113.stdout:1/553: sync 2024-04-12T05:08:58.751 INFO:tasks.workunit.client.0.smithi113.stdout:4/473: sync 2024-04-12T05:08:58.751 INFO:tasks.workunit.client.0.smithi113.stdout:3/428: sync 2024-04-12T05:08:58.752 INFO:tasks.workunit.client.0.smithi113.stdout:9/504: sync 2024-04-12T05:08:58.752 INFO:tasks.workunit.client.0.smithi113.stdout:2/319: sync 2024-04-12T05:08:58.753 INFO:tasks.workunit.client.0.smithi113.stdout:4/474: fdatasync da/d73/f45 0 2024-04-12T05:08:58.754 INFO:tasks.workunit.client.1.smithi192.stdout:8/297: fsync d6/da/fd 0 2024-04-12T05:08:58.755 INFO:tasks.workunit.client.0.smithi113.stdout:2/320: read d8/d9/de/d12/f4c [3471928,58748] 0 2024-04-12T05:08:58.755 INFO:tasks.workunit.client.1.smithi192.stdout:4/400: dwrite d1/d1d/d53/d1c/f4e [4194304,4194304] 0 2024-04-12T05:08:58.756 INFO:tasks.workunit.client.1.smithi192.stdout:6/416: mknod d3/d8/d10/d4c/c8c 0 2024-04-12T05:08:58.756 INFO:tasks.workunit.client.0.smithi113.stdout:3/429: dread d0/d13/f6b [0,4194304] 0 2024-04-12T05:08:58.757 INFO:tasks.workunit.client.0.smithi113.stdout:2/321: dread - d8/d9/d1a/f26 zero size 2024-04-12T05:08:58.759 INFO:tasks.workunit.client.0.smithi113.stdout:3/430: stat d0/d13/d15/d28/d2a/d34/c52 0 2024-04-12T05:08:58.759 INFO:tasks.workunit.client.1.smithi192.stdout:2/362: read d1/d8/d17/d19/f5e [230068,83034] 0 2024-04-12T05:08:58.760 INFO:tasks.workunit.client.0.smithi113.stdout:3/431: dread - d0/d13/d15/d28/f74 zero size 2024-04-12T05:08:58.763 INFO:tasks.workunit.client.1.smithi192.stdout:8/298: write d6/f18 [2466810,32781] 0 2024-04-12T05:08:58.764 INFO:tasks.workunit.client.0.smithi113.stdout:9/505: rename d1/d4/d5/d17/l71 to d1/lb1 0 2024-04-12T05:08:58.765 INFO:tasks.workunit.client.0.smithi113.stdout:3/432: chown d0/d13/c53 0 1 2024-04-12T05:08:58.765 INFO:tasks.workunit.client.1.smithi192.stdout:7/259: dwrite d3/d1d/f2e [4194304,4194304] 0 2024-04-12T05:08:58.767 INFO:tasks.workunit.client.0.smithi113.stdout:3/433: fsync d0/d13/d15/d28/d2a/f8e 0 2024-04-12T05:08:58.770 INFO:tasks.workunit.client.0.smithi113.stdout:9/506: unlink d1/d4/d5/d17/f96 0 2024-04-12T05:08:58.772 INFO:tasks.workunit.client.1.smithi192.stdout:8/299: chown d6/da/f1e 47711 1 2024-04-12T05:08:58.773 INFO:tasks.workunit.client.1.smithi192.stdout:1/308: sync 2024-04-12T05:08:58.773 INFO:tasks.workunit.client.1.smithi192.stdout:0/345: sync 2024-04-12T05:08:58.774 INFO:tasks.workunit.client.1.smithi192.stdout:7/260: stat d3/d1d/d5a/d5f/c54 0 2024-04-12T05:08:58.774 INFO:tasks.workunit.client.1.smithi192.stdout:4/401: rename d1/d54/d55/d37 to d1/d54/d82 0 2024-04-12T05:08:58.775 INFO:tasks.workunit.client.0.smithi113.stdout:1/554: dwrite d10/d21/f47 [0,4194304] 0 2024-04-12T05:08:58.775 INFO:tasks.workunit.client.1.smithi192.stdout:1/309: read - d0/d2/d1e/f4a zero size 2024-04-12T05:08:58.775 INFO:tasks.workunit.client.1.smithi192.stdout:5/336: truncate d0/d15/d32/d45/f60 1363882 0 2024-04-12T05:08:58.777 INFO:tasks.workunit.client.1.smithi192.stdout:6/417: dwrite d3/d8/d10/d4c/d68/f77 [0,4194304] 0 2024-04-12T05:08:58.777 INFO:tasks.workunit.client.0.smithi113.stdout:3/434: dwrite d0/d13/f6a [0,4194304] 0 2024-04-12T05:08:58.778 INFO:tasks.workunit.client.0.smithi113.stdout:2/322: write d8/d9/de/f10 [4039227,3212] 0 2024-04-12T05:08:58.780 INFO:tasks.workunit.client.0.smithi113.stdout:1/555: creat d10/d21/fac x:0 0 0 2024-04-12T05:08:58.780 INFO:tasks.workunit.client.1.smithi192.stdout:8/300: rmdir d6/df 39 2024-04-12T05:08:58.785 INFO:tasks.workunit.client.0.smithi113.stdout:2/323: creat d8/d9/de/d43/f5c x:0 0 0 2024-04-12T05:08:58.788 INFO:tasks.workunit.client.1.smithi192.stdout:2/363: rename d1/d2d/c6b to d1/d8/d17/d19/c78 0 2024-04-12T05:08:58.789 INFO:tasks.workunit.client.1.smithi192.stdout:4/402: symlink d1/d54/l83 0 2024-04-12T05:08:58.794 INFO:tasks.workunit.client.0.smithi113.stdout:1/556: dread d10/d55/f8c [0,4194304] 0 2024-04-12T05:08:58.795 INFO:tasks.workunit.client.1.smithi192.stdout:0/346: read d0/d6/d15/d1b/d3a/d4f/f6a [325099,47365] 0 2024-04-12T05:08:58.795 INFO:tasks.workunit.client.1.smithi192.stdout:1/310: dread d0/d2/f34 [0,4194304] 0 2024-04-12T05:08:58.795 INFO:tasks.workunit.client.1.smithi192.stdout:5/337: rename d0/d15/d24/d47/f6b to d0/d15/d3e/d77/f78 0 2024-04-12T05:08:58.795 INFO:tasks.workunit.client.1.smithi192.stdout:0/347: chown d0/d6/d15/d55/d2e 35838350 1 2024-04-12T05:08:58.796 INFO:tasks.workunit.client.0.smithi113.stdout:9/507: dread d1/d4/f18 [0,4194304] 0 2024-04-12T05:08:58.798 INFO:tasks.workunit.client.1.smithi192.stdout:4/403: mkdir d1/d1d/d53/d1c/d84 0 2024-04-12T05:08:58.801 INFO:tasks.workunit.client.1.smithi192.stdout:1/311: creat d0/d2/dd/f5d x:0 0 0 2024-04-12T05:08:58.801 INFO:tasks.workunit.client.1.smithi192.stdout:8/301: dread d6/df/f43 [0,4194304] 0 2024-04-12T05:08:58.802 INFO:tasks.workunit.client.0.smithi113.stdout:3/435: write d0/d13/d15/d28/d2a/f36 [1496894,45123] 0 2024-04-12T05:08:58.804 INFO:tasks.workunit.client.1.smithi192.stdout:1/312: dread d0/f18 [0,4194304] 0 2024-04-12T05:08:58.804 INFO:tasks.workunit.client.1.smithi192.stdout:0/348: creat d0/f7d x:0 0 0 2024-04-12T05:08:58.805 INFO:tasks.workunit.client.0.smithi113.stdout:1/557: dwrite d10/d14/d67/d70/d98/fa6 [0,4194304] 0 2024-04-12T05:08:58.806 INFO:tasks.workunit.client.1.smithi192.stdout:5/338: creat d0/d15/f79 x:0 0 0 2024-04-12T05:08:58.806 INFO:tasks.workunit.client.0.smithi113.stdout:3/436: fsync d0/d13/d47/f61 0 2024-04-12T05:08:58.808 INFO:tasks.workunit.client.1.smithi192.stdout:4/404: getdents d1/d49 0 2024-04-12T05:08:58.809 INFO:tasks.workunit.client.1.smithi192.stdout:0/349: mknod d0/d4e/d53/c7e 0 2024-04-12T05:08:58.811 INFO:tasks.workunit.client.0.smithi113.stdout:1/558: write d10/d41/d5b/fa3 [381577,66561] 0 2024-04-12T05:08:58.813 INFO:tasks.workunit.client.0.smithi113.stdout:9/508: dwrite d1/d4/d5/d17/f74 [0,4194304] 0 2024-04-12T05:08:58.813 INFO:tasks.workunit.client.0.smithi113.stdout:5/398: sync 2024-04-12T05:08:58.814 INFO:tasks.workunit.client.1.smithi192.stdout:3/309: sync 2024-04-12T05:08:58.814 INFO:tasks.workunit.client.1.smithi192.stdout:8/302: dwrite d6/d1f/f5c [0,4194304] 0 2024-04-12T05:08:58.814 INFO:tasks.workunit.client.1.smithi192.stdout:9/280: sync 2024-04-12T05:08:58.815 INFO:tasks.workunit.client.1.smithi192.stdout:4/405: mknod d1/d54/d82/c85 0 2024-04-12T05:08:58.817 INFO:tasks.workunit.client.0.smithi113.stdout:9/509: chown d1/df/f4d 71 1 2024-04-12T05:08:58.817 INFO:tasks.workunit.client.0.smithi113.stdout:2/324: dread d8/d9/de/f31 [0,4194304] 0 2024-04-12T05:08:58.818 INFO:tasks.workunit.client.0.smithi113.stdout:9/510: dread - d1/df/d29/fb0 zero size 2024-04-12T05:08:58.822 INFO:tasks.workunit.client.0.smithi113.stdout:3/437: rename d0/d13/d47/d4f/c55 to d0/d13/d15/d28/d2a/d34/d5f/c93 0 2024-04-12T05:08:58.823 INFO:tasks.workunit.client.0.smithi113.stdout:5/399: creat d0/d10/d6e/f83 x:0 0 0 2024-04-12T05:08:58.825 INFO:tasks.workunit.client.0.smithi113.stdout:1/559: dread d10/d14/f4d [0,4194304] 0 2024-04-12T05:08:58.825 INFO:tasks.workunit.client.0.smithi113.stdout:3/438: rmdir d0/d13/d15 39 2024-04-12T05:08:58.831 INFO:tasks.workunit.client.0.smithi113.stdout:9/511: read d1/d4/d5/d6/d15/f3f [2273971,71641] 0 2024-04-12T05:08:58.831 INFO:tasks.workunit.client.1.smithi192.stdout:5/339: rmdir d0/d15/d24/d47 39 2024-04-12T05:08:58.831 INFO:tasks.workunit.client.1.smithi192.stdout:8/303: mkdir d6/d1f/d5d 0 2024-04-12T05:08:58.832 INFO:tasks.workunit.client.1.smithi192.stdout:4/406: creat d1/d54/d55/d64/f86 x:0 0 0 2024-04-12T05:08:58.832 INFO:tasks.workunit.client.0.smithi113.stdout:2/325: truncate d8/d16/f1d 2269542 0 2024-04-12T05:08:58.833 INFO:tasks.workunit.client.0.smithi113.stdout:1/560: mkdir d10/d14/d67/d70/d98/dad 0 2024-04-12T05:08:58.833 INFO:tasks.workunit.client.1.smithi192.stdout:9/281: mkdir d11/d25/d60 0 2024-04-12T05:08:58.834 INFO:tasks.workunit.client.0.smithi113.stdout:2/326: stat d8/d9/de/d15/d24 0 2024-04-12T05:08:58.834 INFO:tasks.workunit.client.1.smithi192.stdout:8/304: read d6/d10/f3f [1459755,3937] 0 2024-04-12T05:08:58.835 INFO:tasks.workunit.client.0.smithi113.stdout:1/561: readlink d10/d2e/d31/d5c/l8e 0 2024-04-12T05:08:58.835 INFO:tasks.workunit.client.0.smithi113.stdout:5/400: rmdir d0/d10/d27/d5f/d80 0 2024-04-12T05:08:58.835 INFO:tasks.workunit.client.1.smithi192.stdout:2/364: dread d1/d8/f10 [0,4194304] 0 2024-04-12T05:08:58.836 INFO:tasks.workunit.client.1.smithi192.stdout:3/310: getdents d5/d13/d3a/d4a/d61 0 2024-04-12T05:08:58.836 INFO:tasks.workunit.client.0.smithi113.stdout:2/327: fsync d8/d9/de/d15/d24/f3b 0 2024-04-12T05:08:58.837 INFO:tasks.workunit.client.1.smithi192.stdout:5/340: read d0/f7 [3580798,85958] 0 2024-04-12T05:08:58.837 INFO:tasks.workunit.client.0.smithi113.stdout:3/439: getdents d0/d13/d15/d28/d2a/d8b/d91 0 2024-04-12T05:08:58.839 INFO:tasks.workunit.client.0.smithi113.stdout:2/328: write d8/d9/de/d12/d2b/f54 [2439195,123618] 0 2024-04-12T05:08:58.840 INFO:tasks.workunit.client.1.smithi192.stdout:0/350: dread d0/d6/d15/d19/f1d [0,4194304] 0 2024-04-12T05:08:58.840 INFO:tasks.workunit.client.1.smithi192.stdout:6/418: dread d3/d8/d10/d4c/f62 [0,4194304] 0 2024-04-12T05:08:58.840 INFO:tasks.workunit.client.1.smithi192.stdout:4/407: mknod d1/d1d/d53/d1c/c87 0 2024-04-12T05:08:58.840 INFO:tasks.workunit.client.1.smithi192.stdout:5/341: chown d0/l53 0 1 2024-04-12T05:08:58.842 INFO:tasks.workunit.client.1.smithi192.stdout:9/282: mknod d11/d25/d36/d3d/c61 0 2024-04-12T05:08:58.842 INFO:tasks.workunit.client.1.smithi192.stdout:3/311: write d5/fd [4453696,70059] 0 2024-04-12T05:08:58.842 INFO:tasks.workunit.client.1.smithi192.stdout:8/305: chown d6/df/d37/c50 9 1 2024-04-12T05:08:58.842 INFO:tasks.workunit.client.1.smithi192.stdout:6/419: readlink d3/d5/l3f 0 2024-04-12T05:08:58.843 INFO:tasks.workunit.client.0.smithi113.stdout:3/440: mkdir d0/d13/d47/d94 0 2024-04-12T05:08:58.844 INFO:tasks.workunit.client.1.smithi192.stdout:7/261: dread d3/d7/f19 [4194304,4194304] 0 2024-04-12T05:08:58.845 INFO:tasks.workunit.client.0.smithi113.stdout:1/562: dwrite d10/d2e/f54 [0,4194304] 0 2024-04-12T05:08:58.846 INFO:tasks.workunit.client.1.smithi192.stdout:2/365: chown d1/f44 377266662 1 2024-04-12T05:08:58.846 INFO:tasks.workunit.client.1.smithi192.stdout:8/306: stat d6/d21/c42 0 2024-04-12T05:08:58.846 INFO:tasks.workunit.client.1.smithi192.stdout:4/408: mknod d1/d6b/c88 0 2024-04-12T05:08:58.847 INFO:tasks.workunit.client.1.smithi192.stdout:5/342: dread - d0/d15/d32/f6a zero size 2024-04-12T05:08:58.847 INFO:tasks.workunit.client.0.smithi113.stdout:3/441: mkdir d0/d13/d15/d28/d2a/d8b/d95 0 2024-04-12T05:08:58.847 INFO:tasks.workunit.client.1.smithi192.stdout:0/351: symlink d0/d6/l7f 0 2024-04-12T05:08:58.848 INFO:tasks.workunit.client.0.smithi113.stdout:5/401: dread d0/d10/d27/f25 [0,4194304] 0 2024-04-12T05:08:58.852 INFO:tasks.workunit.client.1.smithi192.stdout:2/366: mkdir d1/d8/d79 0 2024-04-12T05:08:58.852 INFO:tasks.workunit.client.0.smithi113.stdout:1/563: symlink d10/da5/lae 0 2024-04-12T05:08:58.853 INFO:tasks.workunit.client.0.smithi113.stdout:3/442: creat d0/d13/d15/d28/d2a/d34/d5f/f96 x:0 0 0 2024-04-12T05:08:58.853 INFO:tasks.workunit.client.1.smithi192.stdout:0/352: chown d0/d6/d15/d55/d2e/f42 12847085 1 2024-04-12T05:08:58.853 INFO:tasks.workunit.client.1.smithi192.stdout:9/283: read d11/d15/d21/d2c/f1a [6755349,103814] 0 2024-04-12T05:08:58.854 INFO:tasks.workunit.client.1.smithi192.stdout:6/420: fsync d3/f37 0 2024-04-12T05:08:58.856 INFO:tasks.workunit.client.1.smithi192.stdout:7/262: write d3/d11/f46 [78053,91207] 0 2024-04-12T05:08:58.856 INFO:tasks.workunit.client.1.smithi192.stdout:1/313: dwrite d0/f20 [0,4194304] 0 2024-04-12T05:08:58.856 INFO:tasks.workunit.client.1.smithi192.stdout:5/343: symlink d0/l7a 0 2024-04-12T05:08:58.857 INFO:tasks.workunit.client.1.smithi192.stdout:2/367: read - d1/f70 zero size 2024-04-12T05:08:58.857 INFO:tasks.workunit.client.1.smithi192.stdout:0/353: readlink d0/d6/d15/l45 0 2024-04-12T05:08:58.858 INFO:tasks.workunit.client.1.smithi192.stdout:8/307: read d6/df/f16 [397391,106434] 0 2024-04-12T05:08:58.858 INFO:tasks.workunit.client.0.smithi113.stdout:9/512: dwrite d1/d4/f77 [0,4194304] 0 2024-04-12T05:08:58.860 INFO:tasks.workunit.client.1.smithi192.stdout:9/284: readlink d11/d15/l2b 0 2024-04-12T05:08:58.869 INFO:tasks.workunit.client.1.smithi192.stdout:2/368: creat d1/d2d/d73/f7a x:0 0 0 2024-04-12T05:08:58.869 INFO:tasks.workunit.client.1.smithi192.stdout:3/312: dwrite d5/d13/d3a/d4a/f5a [0,4194304] 0 2024-04-12T05:08:58.872 INFO:tasks.workunit.client.1.smithi192.stdout:5/344: read d0/d15/d24/f46 [3197986,115749] 0 2024-04-12T05:08:58.872 INFO:tasks.workunit.client.0.smithi113.stdout:9/513: dread d1/df/d4b/d63/d3d/f5f [0,4194304] 0 2024-04-12T05:08:58.872 INFO:tasks.workunit.client.1.smithi192.stdout:9/285: chown d11/d15/d21/d2c/d1f/c3f 563807 1 2024-04-12T05:08:58.873 INFO:tasks.workunit.client.1.smithi192.stdout:3/313: write d5/d13/d3a/d4a/f67 [524480,85598] 0 2024-04-12T05:08:58.875 INFO:tasks.workunit.client.1.smithi192.stdout:9/286: fdatasync d11/d15/f45 0 2024-04-12T05:08:58.879 INFO:tasks.workunit.client.1.smithi192.stdout:5/345: mkdir d0/d15/d24/d47/d7b 0 2024-04-12T05:08:58.880 INFO:tasks.workunit.client.1.smithi192.stdout:9/287: write d11/d15/d1e/f59 [551416,117251] 0 2024-04-12T05:08:58.880 INFO:tasks.workunit.client.1.smithi192.stdout:8/308: getdents d6/df/d37/d4f/d4c 0 2024-04-12T05:08:58.881 INFO:tasks.workunit.client.1.smithi192.stdout:2/369: mknod d1/d8/d17/d19/d60/d59/d68/c7b 0 2024-04-12T05:08:58.883 INFO:tasks.workunit.client.0.smithi113.stdout:1/564: mkdir d10/d2e/d31/d5c/daf 0 2024-04-12T05:08:58.884 INFO:tasks.workunit.client.0.smithi113.stdout:2/329: write d8/d9/de/d15/f18 [320673,69949] 0 2024-04-12T05:08:58.884 INFO:tasks.workunit.client.0.smithi113.stdout:1/565: read d10/f9a [1539583,92268] 0 2024-04-12T05:08:58.886 INFO:tasks.workunit.client.0.smithi113.stdout:2/330: write d8/d9/de/f42 [464303,49868] 0 2024-04-12T05:08:58.887 INFO:tasks.workunit.client.0.smithi113.stdout:9/514: write d1/d4/d38/f8c [2552754,65667] 0 2024-04-12T05:08:58.889 INFO:tasks.workunit.client.1.smithi192.stdout:5/346: mkdir d0/d15/d64/d7c 0 2024-04-12T05:08:58.890 INFO:tasks.workunit.client.1.smithi192.stdout:2/370: unlink d1/d8/d3a/de/f2f 0 2024-04-12T05:08:58.890 INFO:tasks.workunit.client.0.smithi113.stdout:3/443: rename d0/d13/d47/l7f to d0/d13/d15/l97 0 2024-04-12T05:08:58.892 INFO:tasks.workunit.client.0.smithi113.stdout:6/438: sync 2024-04-12T05:08:58.892 INFO:tasks.workunit.client.0.smithi113.stdout:8/489: sync 2024-04-12T05:08:58.893 INFO:tasks.workunit.client.0.smithi113.stdout:9/515: unlink d1/d4/d38/d65/c8a 0 2024-04-12T05:08:58.893 INFO:tasks.workunit.client.1.smithi192.stdout:0/354: truncate d0/d6/d15/d19/f36 1548829 0 2024-04-12T05:08:58.894 INFO:tasks.workunit.client.0.smithi113.stdout:1/566: dwrite d10/d2e/f4e [0,4194304] 0 2024-04-12T05:08:58.896 INFO:tasks.workunit.client.0.smithi113.stdout:8/490: stat d9/d16/d1c/d59/d68/c72 0 2024-04-12T05:08:58.896 INFO:tasks.workunit.client.0.smithi113.stdout:9/516: chown d1/df/l97 2085348 1 2024-04-12T05:08:58.896 INFO:tasks.workunit.client.1.smithi192.stdout:8/309: dwrite d6/da/f3a [4194304,4194304] 0 2024-04-12T05:08:58.896 INFO:tasks.workunit.client.0.smithi113.stdout:3/444: creat d0/d13/d15/d28/d2a/f98 x:0 0 0 2024-04-12T05:08:58.897 INFO:tasks.workunit.client.0.smithi113.stdout:2/331: creat d8/d9/de/d12/f5d x:0 0 0 2024-04-12T05:08:58.898 INFO:tasks.workunit.client.0.smithi113.stdout:2/332: chown d8/d9/de/d15/d24 861 1 2024-04-12T05:08:58.899 INFO:tasks.workunit.client.0.smithi113.stdout:6/439: mknod d8/de/d13/d11/ca1 0 2024-04-12T05:08:58.900 INFO:tasks.workunit.client.0.smithi113.stdout:2/333: read d8/d9/de/d12/d2b/f54 [1552833,42664] 0 2024-04-12T05:08:58.901 INFO:tasks.workunit.client.0.smithi113.stdout:6/440: stat d8/de/d13/d11/f43 0 2024-04-12T05:08:58.902 INFO:tasks.workunit.client.1.smithi192.stdout:7/263: dread f1 [0,4194304] 0 2024-04-12T05:08:58.909 INFO:tasks.workunit.client.1.smithi192.stdout:9/288: dwrite d11/d15/d21/f2d [0,4194304] 0 2024-04-12T05:08:58.909 INFO:tasks.workunit.client.0.smithi113.stdout:8/491: creat d9/d1b/d8e/d93/fa2 x:0 0 0 2024-04-12T05:08:58.910 INFO:tasks.workunit.client.0.smithi113.stdout:1/567: truncate d10/d41/d89/d96/f49 193915 0 2024-04-12T05:08:58.910 INFO:tasks.workunit.client.0.smithi113.stdout:8/492: chown d9/d16 1157 1 2024-04-12T05:08:58.910 INFO:tasks.workunit.client.0.smithi113.stdout:5/402: dwrite d0/d10/d27/f25 [0,4194304] 0 2024-04-12T05:08:58.912 INFO:tasks.workunit.client.1.smithi192.stdout:2/371: rmdir d1/d8/d3a/d72 0 2024-04-12T05:08:58.913 INFO:tasks.workunit.client.0.smithi113.stdout:5/403: read - d0/f31 zero size 2024-04-12T05:08:58.914 INFO:tasks.workunit.client.0.smithi113.stdout:2/334: mknod d8/d9/d5b/c5e 0 2024-04-12T05:08:58.915 INFO:tasks.workunit.client.0.smithi113.stdout:8/493: dwrite d9/d16/d3c/f74 [0,4194304] 0 2024-04-12T05:08:58.917 INFO:tasks.workunit.client.0.smithi113.stdout:8/494: fdatasync d9/d16/d3c/f4e 0 2024-04-12T05:08:58.918 INFO:tasks.workunit.client.1.smithi192.stdout:0/355: write d0/d6/d15/d55/d29/f2b [211664,94916] 0 2024-04-12T05:08:58.919 INFO:tasks.workunit.client.0.smithi113.stdout:1/568: link d10/d41/d89/d96/d7d/f7f d10/d41/d5b/fb0 0 2024-04-12T05:08:58.919 INFO:tasks.workunit.client.1.smithi192.stdout:1/314: dwrite d0/d2/dd/d3c/f41 [0,4194304] 0 2024-04-12T05:08:58.922 INFO:tasks.workunit.client.1.smithi192.stdout:7/264: write d3/d7/d14/d1f/f34 [208227,49346] 0 2024-04-12T05:08:58.922 INFO:tasks.workunit.client.0.smithi113.stdout:2/335: mknod d8/d9/de/d15/d38/c5f 0 2024-04-12T05:08:58.926 INFO:tasks.workunit.client.1.smithi192.stdout:7/265: readlink d3/d7/d14/l42 0 2024-04-12T05:08:58.927 INFO:tasks.workunit.client.0.smithi113.stdout:5/404: creat d0/d10/d27/d1d/d38/d3b/f84 x:0 0 0 2024-04-12T05:08:58.927 INFO:tasks.workunit.client.0.smithi113.stdout:8/495: dread d9/d1b/d96/f9b [0,4194304] 0 2024-04-12T05:08:58.930 INFO:tasks.workunit.client.1.smithi192.stdout:2/372: mkdir d1/d8/d3a/de/d7c 0 2024-04-12T05:08:58.932 INFO:tasks.workunit.client.0.smithi113.stdout:5/405: read d0/d10/d6e/d7a/f28 [4002773,53300] 0 2024-04-12T05:08:58.932 INFO:tasks.workunit.client.1.smithi192.stdout:0/356: chown d0/c9 150609 1 2024-04-12T05:08:58.933 INFO:tasks.workunit.client.1.smithi192.stdout:9/289: dwrite fe [0,4194304] 0 2024-04-12T05:08:58.936 INFO:tasks.workunit.client.1.smithi192.stdout:0/357: read d0/f20 [3305465,59490] 0 2024-04-12T05:08:58.938 INFO:tasks.workunit.client.1.smithi192.stdout:1/315: creat d0/d2/dd/d3c/d5b/f5e x:0 0 0 2024-04-12T05:08:58.938 INFO:tasks.workunit.client.0.smithi113.stdout:6/441: link d8/de/d13/d11/d36/d3d/d4c/d31/l56 d8/de/d13/la2 0 2024-04-12T05:08:58.940 INFO:tasks.workunit.client.0.smithi113.stdout:5/406: mknod d0/d10/d27/d5f/c85 0 2024-04-12T05:08:58.943 INFO:tasks.workunit.client.1.smithi192.stdout:8/310: dwrite d6/df/d37/f48 [0,4194304] 0 2024-04-12T05:08:58.945 INFO:tasks.workunit.client.1.smithi192.stdout:0/358: creat d0/d6/d15/d1b/f80 x:0 0 0 2024-04-12T05:08:58.945 INFO:tasks.workunit.client.1.smithi192.stdout:9/290: creat d11/d15/d21/d2c/f62 x:0 0 0 2024-04-12T05:08:58.945 INFO:tasks.workunit.client.1.smithi192.stdout:1/316: mknod d0/d2/dd/d3c/c5f 0 2024-04-12T05:08:58.946 INFO:tasks.workunit.client.0.smithi113.stdout:2/336: fsync d8/d16/f1d 0 2024-04-12T05:08:58.947 INFO:tasks.workunit.client.0.smithi113.stdout:5/407: symlink d0/d10/d27/d1d/d38/d44/d4e/l86 0 2024-04-12T05:08:58.949 INFO:tasks.workunit.client.0.smithi113.stdout:5/408: chown d0/f6 21154445 1 2024-04-12T05:08:58.950 INFO:tasks.workunit.client.1.smithi192.stdout:0/359: readlink d0/l2c 0 2024-04-12T05:08:58.950 INFO:tasks.workunit.client.1.smithi192.stdout:5/347: dread d0/d21/f36 [4194304,4194304] 0 2024-04-12T05:08:58.950 INFO:tasks.workunit.client.0.smithi113.stdout:3/445: dwrite d0/f1c [0,4194304] 0 2024-04-12T05:08:58.951 INFO:tasks.workunit.client.0.smithi113.stdout:8/496: rename d9/d1b/d8e/d93/d32/d48/d8d/d7e to d9/d16/d1c/d59/da3 0 2024-04-12T05:08:58.951 INFO:tasks.workunit.client.1.smithi192.stdout:9/291: rename f7 to d11/d25/d36/f63 0 2024-04-12T05:08:58.952 INFO:tasks.workunit.client.1.smithi192.stdout:0/360: read d0/d6/d15/d55/d2e/f63 [2356458,21491] 0 2024-04-12T05:08:58.953 INFO:tasks.workunit.client.0.smithi113.stdout:2/337: mknod d8/d9/d1a/c60 0 2024-04-12T05:08:58.955 INFO:tasks.workunit.client.1.smithi192.stdout:0/361: read - d0/d6/d15/d55/d2e/d59/f78 zero size 2024-04-12T05:08:58.956 INFO:tasks.workunit.client.1.smithi192.stdout:2/373: dwrite d1/d8/f22 [0,4194304] 0 2024-04-12T05:08:58.958 INFO:tasks.workunit.client.1.smithi192.stdout:5/348: truncate d0/d15/d24/f51 624135 0 2024-04-12T05:08:58.958 INFO:tasks.workunit.client.0.smithi113.stdout:2/338: truncate d8/d16/f4e 618441 0 2024-04-12T05:08:58.958 INFO:tasks.workunit.client.0.smithi113.stdout:6/442: dwrite d8/de/f19 [4194304,4194304] 0 2024-04-12T05:08:58.959 INFO:tasks.workunit.client.1.smithi192.stdout:1/317: creat d0/d1b/f60 x:0 0 0 2024-04-12T05:08:58.961 INFO:tasks.workunit.client.0.smithi113.stdout:2/339: chown d8/d16/f4e 10256646 1 2024-04-12T05:08:58.962 INFO:tasks.workunit.client.1.smithi192.stdout:0/362: symlink d0/d6/d15/d19/l81 0 2024-04-12T05:08:58.964 INFO:tasks.workunit.client.1.smithi192.stdout:0/363: chown d0/d6/d15/d19/f31 36311 1 2024-04-12T05:08:58.964 INFO:tasks.workunit.client.1.smithi192.stdout:1/318: readlink d0/d1b/l27 0 2024-04-12T05:08:58.965 INFO:tasks.workunit.client.1.smithi192.stdout:5/349: dread d0/d15/d24/d47/f4d [0,4194304] 0 2024-04-12T05:08:58.966 INFO:tasks.workunit.client.0.smithi113.stdout:5/409: creat d0/d10/d6e/f87 x:0 0 0 2024-04-12T05:08:58.966 INFO:tasks.workunit.client.0.smithi113.stdout:1/569: write d10/d2e/d31/d33/f64 [1842958,46722] 0 2024-04-12T05:08:58.969 INFO:tasks.workunit.client.0.smithi113.stdout:2/340: write d8/d9/d1a/f50 [921246,126385] 0 2024-04-12T05:08:58.969 INFO:tasks.workunit.client.1.smithi192.stdout:2/374: creat d1/d8/d3a/de/d7c/f7d x:0 0 0 2024-04-12T05:08:58.969 INFO:tasks.workunit.client.1.smithi192.stdout:2/375: write d1/d8/d17/f5f [314407,54633] 0 2024-04-12T05:08:58.969 INFO:tasks.workunit.client.1.smithi192.stdout:1/319: creat d0/d1b/d28/f61 x:0 0 0 2024-04-12T05:08:58.970 INFO:tasks.workunit.client.0.smithi113.stdout:8/497: unlink d9/d16/d7f/l29 0 2024-04-12T05:08:58.971 INFO:tasks.workunit.client.1.smithi192.stdout:5/350: mkdir d0/d15/d32/d69/d7d 0 2024-04-12T05:08:58.973 INFO:tasks.workunit.client.1.smithi192.stdout:0/364: creat d0/d6/d15/d55/d2e/d66/f82 x:0 0 0 2024-04-12T05:08:58.974 INFO:tasks.workunit.client.1.smithi192.stdout:5/351: truncate d0/d15/f79 921500 0 2024-04-12T05:08:58.976 INFO:tasks.workunit.client.0.smithi113.stdout:5/410: creat d0/d10/d27/d5f/f88 x:0 0 0 2024-04-12T05:08:58.976 INFO:tasks.workunit.client.0.smithi113.stdout:3/446: dwrite d0/d13/d15/d28/f74 [0,4194304] 0 2024-04-12T05:08:58.977 INFO:tasks.workunit.client.1.smithi192.stdout:5/352: dread - d0/d15/d3e/f71 zero size 2024-04-12T05:08:58.978 INFO:tasks.workunit.client.0.smithi113.stdout:5/411: fsync d0/d10/f45 0 2024-04-12T05:08:58.979 INFO:tasks.workunit.client.0.smithi113.stdout:8/498: creat d9/d16/d65/d8c/fa4 x:0 0 0 2024-04-12T05:08:58.979 INFO:tasks.workunit.client.1.smithi192.stdout:1/320: chown d0/d1b/f2b 0 1 2024-04-12T05:08:58.979 INFO:tasks.workunit.client.1.smithi192.stdout:2/376: chown d1/d8/d17/d19/d71 1091 1 2024-04-12T05:08:58.980 INFO:tasks.workunit.client.1.smithi192.stdout:5/353: write d0/d15/d3e/f71 [359126,40036] 0 2024-04-12T05:08:58.981 INFO:tasks.workunit.client.0.smithi113.stdout:2/341: getdents d8/d9/de/d15/d24/d58 0 2024-04-12T05:08:58.983 INFO:tasks.workunit.client.1.smithi192.stdout:2/377: mknod d1/d8/d3a/c7e 0 2024-04-12T05:08:58.985 INFO:tasks.workunit.client.1.smithi192.stdout:5/354: readlink d0/d15/d24/l2e 0 2024-04-12T05:08:58.985 INFO:tasks.workunit.client.0.smithi113.stdout:8/499: fdatasync d9/d1b/d8e/d93/d92/f99 0 2024-04-12T05:08:58.985 INFO:tasks.workunit.client.0.smithi113.stdout:1/570: dwrite d10/d41/f60 [4194304,4194304] 0 2024-04-12T05:08:58.986 INFO:tasks.workunit.client.0.smithi113.stdout:5/412: write d0/d10/d6e/d7a/f62 [2820533,41850] 0 2024-04-12T05:08:58.986 INFO:tasks.workunit.client.0.smithi113.stdout:3/447: creat d0/d13/d15/d28/d2a/d8f/f99 x:0 0 0 2024-04-12T05:08:58.987 INFO:tasks.workunit.client.1.smithi192.stdout:2/378: write d1/d8/d17/f5f [931754,63999] 0 2024-04-12T05:08:58.987 INFO:tasks.workunit.client.1.smithi192.stdout:1/321: mkdir d0/d2/dd/d21/d62 0 2024-04-12T05:08:58.988 INFO:tasks.workunit.client.1.smithi192.stdout:0/365: dwrite d0/f16 [0,4194304] 0 2024-04-12T05:08:58.990 INFO:tasks.workunit.client.1.smithi192.stdout:2/379: stat d1/d8/d3a/de/d7c/f7d 0 2024-04-12T05:08:58.994 INFO:tasks.workunit.client.0.smithi113.stdout:3/448: write d0/d13/d15/d28/d2a/d34/f63 [543109,9777] 0 2024-04-12T05:08:58.996 INFO:tasks.workunit.client.1.smithi192.stdout:0/366: write d0/d6/d15/d19/f31 [1961940,128202] 0 2024-04-12T05:08:58.996 INFO:tasks.workunit.client.1.smithi192.stdout:5/355: chown d0/c12 11 1 2024-04-12T05:08:58.997 INFO:tasks.workunit.client.1.smithi192.stdout:1/322: write d0/d2/d1e/f36 [3759684,66809] 0 2024-04-12T05:08:58.999 INFO:tasks.workunit.client.0.smithi113.stdout:5/413: write d0/d10/d27/d1d/f1e [1562482,107567] 0 2024-04-12T05:08:58.999 INFO:tasks.workunit.client.0.smithi113.stdout:8/500: dwrite d9/d1b/f45 [0,4194304] 0 2024-04-12T05:08:59.000 INFO:tasks.workunit.client.0.smithi113.stdout:1/571: dread d10/d2e/f54 [0,4194304] 0 2024-04-12T05:08:59.001 INFO:tasks.workunit.client.1.smithi192.stdout:2/380: dwrite d1/d8/d17/d19/d60/f66 [0,4194304] 0 2024-04-12T05:08:59.003 INFO:tasks.workunit.client.0.smithi113.stdout:2/342: truncate d8/d16/f19 1171703 0 2024-04-12T05:08:59.003 INFO:tasks.workunit.client.1.smithi192.stdout:5/356: mknod d0/d15/d32/d45/c7e 0 2024-04-12T05:08:59.005 INFO:tasks.workunit.client.0.smithi113.stdout:8/501: chown d9/d16/d3c/f52 49997 1 2024-04-12T05:08:59.006 INFO:tasks.workunit.client.0.smithi113.stdout:1/572: write d10/d14/d67/d70/d98/fab [952033,3880] 0 2024-04-12T05:08:59.006 INFO:tasks.workunit.client.0.smithi113.stdout:2/343: chown d8/d9/d27/c2d 4792 1 2024-04-12T05:08:59.007 INFO:tasks.workunit.client.1.smithi192.stdout:0/367: getdents d0/d7b/d7c 0 2024-04-12T05:08:59.008 INFO:tasks.workunit.client.0.smithi113.stdout:2/344: chown d8/d9/de/d15 17 1 2024-04-12T05:08:59.009 INFO:tasks.workunit.client.0.smithi113.stdout:1/573: chown d10/d14/f2a 252254 1 2024-04-12T05:08:59.010 INFO:tasks.workunit.client.0.smithi113.stdout:3/449: dwrite d0/d13/f6a [0,4194304] 0 2024-04-12T05:08:59.011 INFO:tasks.workunit.client.0.smithi113.stdout:2/345: chown d8/d9/de/d12/d2b/f5a 165179702 1 2024-04-12T05:08:59.013 INFO:tasks.workunit.client.1.smithi192.stdout:5/357: fdatasync d0/d15/d24/f4b 0 2024-04-12T05:08:59.013 INFO:tasks.workunit.client.1.smithi192.stdout:1/323: creat d0/f63 x:0 0 0 2024-04-12T05:08:59.013 INFO:tasks.workunit.client.0.smithi113.stdout:1/574: write d10/d2e/d31/d33/f64 [1363681,89409] 0 2024-04-12T05:08:59.016 INFO:tasks.workunit.client.0.smithi113.stdout:6/443: dread d8/de/d13/d11/d36/f4e [0,4194304] 0 2024-04-12T05:08:59.022 INFO:tasks.workunit.client.0.smithi113.stdout:4/475: sync 2024-04-12T05:08:59.022 INFO:tasks.workunit.client.0.smithi113.stdout:7/483: sync 2024-04-12T05:08:59.023 INFO:tasks.workunit.client.0.smithi113.stdout:0/528: sync 2024-04-12T05:08:59.024 INFO:tasks.workunit.client.1.smithi192.stdout:1/324: dwrite d0/d2/dd/d3c/f41 [0,4194304] 0 2024-04-12T05:08:59.024 INFO:tasks.workunit.client.0.smithi113.stdout:7/484: write d0/dc/f68 [981859,24915] 0 2024-04-12T05:08:59.027 INFO:tasks.workunit.client.1.smithi192.stdout:2/381: mkdir d1/d8/d17/d19/d71/d7f 0 2024-04-12T05:08:59.028 INFO:tasks.workunit.client.1.smithi192.stdout:5/358: creat d0/d15/d32/d45/f7f x:0 0 0 2024-04-12T05:08:59.036 INFO:tasks.workunit.client.1.smithi192.stdout:2/382: mknod d1/d8/c80 0 2024-04-12T05:08:59.041 INFO:tasks.workunit.client.1.smithi192.stdout:1/325: getdents d0/d2/d1e/d3f/d56 0 2024-04-12T05:08:59.041 INFO:tasks.workunit.client.1.smithi192.stdout:5/359: mkdir d0/d15/d24/d80 0 2024-04-12T05:08:59.042 INFO:tasks.workunit.client.1.smithi192.stdout:5/360: write d0/d15/d32/d45/f7f [71002,90733] 0 2024-04-12T05:08:59.043 INFO:tasks.workunit.client.1.smithi192.stdout:5/361: chown d0/f11 21 1 2024-04-12T05:08:59.048 INFO:tasks.workunit.client.1.smithi192.stdout:1/326: creat d0/d2/d1e/d3f/d56/d58/f64 x:0 0 0 2024-04-12T05:08:59.051 INFO:tasks.workunit.client.1.smithi192.stdout:5/362: getdents d0/d15/d32/d45 0 2024-04-12T05:08:59.056 INFO:tasks.workunit.client.1.smithi192.stdout:5/363: truncate d0/f11 2650696 0 2024-04-12T05:08:59.058 INFO:tasks.workunit.client.1.smithi192.stdout:0/368: dwrite d0/d6/d15/d19/f44 [0,4194304] 0 2024-04-12T05:08:59.058 INFO:tasks.workunit.client.0.smithi113.stdout:5/414: rename d0/d10/d6e/d7a to d0/d10/d27/d1d/d3f/d73/d89 0 2024-04-12T05:08:59.059 INFO:tasks.workunit.client.1.smithi192.stdout:5/364: symlink d0/d15/d24/d47/l81 0 2024-04-12T05:08:59.060 INFO:tasks.workunit.client.1.smithi192.stdout:0/369: stat d0/d4e/f65 0 2024-04-12T05:08:59.062 INFO:tasks.workunit.client.1.smithi192.stdout:5/365: chown d0/d15/d32/f6a 3 1 2024-04-12T05:08:59.064 INFO:tasks.workunit.client.1.smithi192.stdout:5/366: dread d0/d15/d32/d45/f7f [0,4194304] 0 2024-04-12T05:08:59.065 INFO:tasks.workunit.client.1.smithi192.stdout:0/370: creat d0/d6/d15/d1b/d46/d73/d4b/f83 x:0 0 0 2024-04-12T05:08:59.066 INFO:tasks.workunit.client.1.smithi192.stdout:0/371: dread - d0/f7d zero size 2024-04-12T05:08:59.071 INFO:tasks.workunit.client.1.smithi192.stdout:0/372: creat d0/d6/d15/d55/d2e/d59/f84 x:0 0 0 2024-04-12T05:08:59.072 INFO:tasks.workunit.client.1.smithi192.stdout:0/373: write d0/d6/d15/d55/f48 [893123,90116] 0 2024-04-12T05:08:59.077 INFO:tasks.workunit.client.0.smithi113.stdout:4/476: mknod da/d2b/c9c 0 2024-04-12T05:08:59.078 INFO:tasks.workunit.client.1.smithi192.stdout:5/367: truncate d0/f52 3058303 0 2024-04-12T05:08:59.081 INFO:tasks.workunit.client.1.smithi192.stdout:5/368: write d0/d15/d24/d29/f2a [727191,60162] 0 2024-04-12T05:08:59.083 INFO:tasks.workunit.client.0.smithi113.stdout:5/415: mknod d0/d10/d27/d1d/d38/d44/d66/c8a 0 2024-04-12T05:08:59.085 INFO:tasks.workunit.client.1.smithi192.stdout:5/369: symlink d0/d15/d24/d47/l82 0 2024-04-12T05:08:59.085 INFO:tasks.workunit.client.1.smithi192.stdout:2/383: dread d1/f2 [0,4194304] 0 2024-04-12T05:08:59.085 INFO:tasks.workunit.client.1.smithi192.stdout:0/374: dread d0/d6/d15/d55/f48 [0,4194304] 0 2024-04-12T05:08:59.086 INFO:tasks.workunit.client.1.smithi192.stdout:2/384: readlink d1/d8/d3a/l63 0 2024-04-12T05:08:59.087 INFO:tasks.workunit.client.0.smithi113.stdout:0/529: fdatasync d1/d43/f62 0 2024-04-12T05:08:59.097 INFO:tasks.workunit.client.1.smithi192.stdout:2/385: rename d1/d8/d17/d19/d60/d59/c6e to d1/d8/d17/d19/d60/d59/c81 0 2024-04-12T05:08:59.098 INFO:tasks.workunit.client.0.smithi113.stdout:8/502: getdents d9/d16/d7f 0 2024-04-12T05:08:59.100 INFO:tasks.workunit.client.0.smithi113.stdout:8/503: write d9/d1b/f46 [1775875,31806] 0 2024-04-12T05:08:59.105 INFO:tasks.workunit.client.1.smithi192.stdout:2/386: write d1/d8/f29 [495366,18172] 0 2024-04-12T05:08:59.107 INFO:tasks.workunit.client.0.smithi113.stdout:8/504: dwrite d9/d1b/d8e/d93/d92/f99 [0,4194304] 0 2024-04-12T05:08:59.107 INFO:tasks.workunit.client.0.smithi113.stdout:0/530: dread d1/d43/f62 [0,4194304] 0 2024-04-12T05:08:59.108 INFO:tasks.workunit.client.1.smithi192.stdout:2/387: fsync d1/d8/f10 0 2024-04-12T05:08:59.108 INFO:tasks.workunit.client.1.smithi192.stdout:0/375: dread d0/d6/d15/d1b/d3a/d4f/f24 [4194304,4194304] 0 2024-04-12T05:08:59.111 INFO:tasks.workunit.client.0.smithi113.stdout:8/505: mknod d9/d16/d1c/d59/d78/ca5 0 2024-04-12T05:08:59.115 INFO:tasks.workunit.client.1.smithi192.stdout:2/388: link d1/d8/d3a/l14 d1/d8/d3a/l82 0 2024-04-12T05:08:59.116 INFO:tasks.workunit.client.1.smithi192.stdout:0/376: write d0/d6/d15/d1b/d46/d73/f60 [1263786,65234] 0 2024-04-12T05:08:59.119 INFO:tasks.workunit.client.0.smithi113.stdout:0/531: creat d1/d43/d59/fa5 x:0 0 0 2024-04-12T05:08:59.122 INFO:tasks.workunit.client.0.smithi113.stdout:8/506: symlink d9/d16/d7f/d3a/la6 0 2024-04-12T05:08:59.123 INFO:tasks.workunit.client.1.smithi192.stdout:2/389: dread d1/d8/d17/d19/d60/f66 [0,4194304] 0 2024-04-12T05:08:59.125 INFO:tasks.workunit.client.0.smithi113.stdout:8/507: mknod d9/d16/d7f/ca7 0 2024-04-12T05:08:59.126 INFO:tasks.workunit.client.0.smithi113.stdout:0/532: dwrite d1/d6/d16/d2a/d50/f65 [4194304,4194304] 0 2024-04-12T05:08:59.127 INFO:tasks.workunit.client.1.smithi192.stdout:0/377: dwrite d0/f7d [0,4194304] 0 2024-04-12T05:08:59.128 INFO:tasks.workunit.client.0.smithi113.stdout:0/533: write d1/d6/d8e/f9b [57139,84647] 0 2024-04-12T05:08:59.128 INFO:tasks.workunit.client.0.smithi113.stdout:0/534: read d1/d9/f45 [2311411,5604] 0 2024-04-12T05:08:59.136 INFO:tasks.workunit.client.0.smithi113.stdout:2/346: write d8/d9/de/d12/f4c [1163809,54964] 0 2024-04-12T05:08:59.138 INFO:tasks.workunit.client.1.smithi192.stdout:0/378: link d0/d6/d15/d1b/d46/d73/f38 d0/d6/d15/d55/d29/d68/f85 0 2024-04-12T05:08:59.141 INFO:tasks.workunit.client.1.smithi192.stdout:2/390: truncate d1/d8/d3a/f3b 1589271 0 2024-04-12T05:08:59.142 INFO:tasks.workunit.client.0.smithi113.stdout:6/444: write d8/d45/f49 [4057249,28085] 0 2024-04-12T05:08:59.143 INFO:tasks.workunit.client.1.smithi192.stdout:0/379: creat d0/d6/d15/d55/d2e/d66/f86 x:0 0 0 2024-04-12T05:08:59.144 INFO:tasks.workunit.client.0.smithi113.stdout:2/347: dwrite d8/f48 [0,4194304] 0 2024-04-12T05:08:59.145 INFO:tasks.workunit.client.1.smithi192.stdout:0/380: chown d0/d6/d15/d19/f1d 1995804 1 2024-04-12T05:08:59.146 INFO:tasks.workunit.client.0.smithi113.stdout:7/485: unlink d0/dd/c40 0 2024-04-12T05:08:59.148 INFO:tasks.workunit.client.0.smithi113.stdout:7/486: readlink d0/dd/d11/d44/l65 0 2024-04-12T05:08:59.149 INFO:tasks.workunit.client.1.smithi192.stdout:0/381: rmdir d0/d6/d15/d1b/d46/d73/d35 39 2024-04-12T05:08:59.151 INFO:tasks.workunit.client.1.smithi192.stdout:2/391: dwrite d1/d8/f10 [4194304,4194304] 0 2024-04-12T05:08:59.152 INFO:tasks.workunit.client.1.smithi192.stdout:0/382: read - d0/d6/d15/d1b/d3a/f76 zero size 2024-04-12T05:08:59.153 INFO:tasks.workunit.client.1.smithi192.stdout:2/392: chown d1/d8/d17/f20 85958 1 2024-04-12T05:08:59.154 INFO:tasks.workunit.client.1.smithi192.stdout:2/393: fdatasync d1/d8/d3a/de/d7c/f7d 0 2024-04-12T05:08:59.155 INFO:tasks.workunit.client.1.smithi192.stdout:0/383: chown d0/d6/d15/d1b/d3a/d4f/l5f 1 1 2024-04-12T05:08:59.155 INFO:tasks.workunit.client.1.smithi192.stdout:2/394: rename d1/d8 to d1/d8/d79/d83 22 2024-04-12T05:08:59.156 INFO:tasks.workunit.client.0.smithi113.stdout:7/487: readlink d0/dc/d6/d1a/l1e 0 2024-04-12T05:08:59.157 INFO:tasks.workunit.client.1.smithi192.stdout:0/384: chown d0/d6/d15/d55/d2e/c4a 1 1 2024-04-12T05:08:59.159 INFO:tasks.workunit.client.1.smithi192.stdout:0/385: readlink d0/d6/d15/d1b/d46/d73/d4b/l54 0 2024-04-12T05:08:59.163 INFO:tasks.workunit.client.1.smithi192.stdout:2/395: creat d1/d8/f84 x:0 0 0 2024-04-12T05:08:59.164 INFO:tasks.workunit.client.1.smithi192.stdout:2/396: truncate d1/d2d/f48 977038 0 2024-04-12T05:08:59.165 INFO:tasks.workunit.client.1.smithi192.stdout:0/386: creat d0/d4e/f87 x:0 0 0 2024-04-12T05:08:59.167 INFO:tasks.workunit.client.0.smithi113.stdout:1/575: rename d10/d21/f47 to d10/d14/d67/d70/fb1 0 2024-04-12T05:08:59.167 INFO:tasks.workunit.client.0.smithi113.stdout:5/416: dwrite d0/d10/d27/d1d/d3f/d73/d89/f40 [0,4194304] 0 2024-04-12T05:08:59.168 INFO:tasks.workunit.client.0.smithi113.stdout:3/450: symlink d0/d13/l9a 0 2024-04-12T05:08:59.168 INFO:tasks.workunit.client.0.smithi113.stdout:6/445: dwrite d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8f [0,4194304] 0 2024-04-12T05:08:59.168 INFO:tasks.workunit.client.0.smithi113.stdout:4/477: dwrite da/f66 [0,4194304] 0 2024-04-12T05:08:59.169 INFO:tasks.workunit.client.0.smithi113.stdout:4/478: stat da/d4f 0 2024-04-12T05:08:59.170 INFO:tasks.workunit.client.0.smithi113.stdout:8/508: unlink d9/d16/d7f/c7c 0 2024-04-12T05:08:59.171 INFO:tasks.workunit.client.0.smithi113.stdout:3/451: fdatasync d0/d13/d47/f89 0 2024-04-12T05:08:59.171 INFO:tasks.workunit.client.1.smithi192.stdout:2/397: truncate d1/d8/fc 606449 0 2024-04-12T05:08:59.174 INFO:tasks.workunit.client.0.smithi113.stdout:2/348: rename d8/d9/de/d15/d24/f34 to d8/d16/f61 0 2024-04-12T05:08:59.174 INFO:tasks.workunit.client.0.smithi113.stdout:3/452: fsync d0/d13/f1f 0 2024-04-12T05:08:59.174 INFO:tasks.workunit.client.0.smithi113.stdout:1/576: write d10/d2e/d31/d5c/f93 [752476,29567] 0 2024-04-12T05:08:59.176 INFO:tasks.workunit.client.0.smithi113.stdout:8/509: read d9/d1b/d8e/d93/f4d [920549,79666] 0 2024-04-12T05:08:59.177 INFO:tasks.workunit.client.0.smithi113.stdout:1/577: write f2 [272583,48937] 0 2024-04-12T05:08:59.179 INFO:tasks.workunit.client.0.smithi113.stdout:8/510: read d9/f53 [206024,116600] 0 2024-04-12T05:08:59.180 INFO:tasks.workunit.client.0.smithi113.stdout:2/349: truncate d8/d9/de/d12/f41 5496070 0 2024-04-12T05:08:59.182 INFO:tasks.workunit.client.0.smithi113.stdout:6/446: dread d8/d45/f5d [0,4194304] 0 2024-04-12T05:08:59.186 INFO:tasks.workunit.client.1.smithi192.stdout:2/398: dwrite d1/d8/d17/d19/f45 [0,4194304] 0 2024-04-12T05:08:59.188 INFO:tasks.workunit.client.0.smithi113.stdout:3/453: dwrite d0/d13/d47/d4f/f4c [4194304,4194304] 0 2024-04-12T05:08:59.191 INFO:tasks.workunit.client.0.smithi113.stdout:5/417: creat d0/d10/d27/d1d/d3f/d7e/f8b x:0 0 0 2024-04-12T05:08:59.193 INFO:tasks.workunit.client.0.smithi113.stdout:1/578: creat d10/d41/d89/d96/d7d/fb2 x:0 0 0 2024-04-12T05:08:59.195 INFO:tasks.workunit.client.0.smithi113.stdout:2/350: fsync d8/d9/de/d12/d2b/f54 0 2024-04-12T05:08:59.196 INFO:tasks.workunit.client.0.smithi113.stdout:6/447: creat d8/de/d13/d11/d36/d2c/fa3 x:0 0 0 2024-04-12T05:08:59.196 INFO:tasks.workunit.client.1.smithi192.stdout:2/399: rmdir d1/d8/d79 0 2024-04-12T05:08:59.197 INFO:tasks.workunit.client.0.smithi113.stdout:3/454: dread d0/d13/d47/f78 [0,4194304] 0 2024-04-12T05:08:59.198 INFO:tasks.workunit.client.0.smithi113.stdout:5/418: mkdir d0/d10/d27/d1d/d8c 0 2024-04-12T05:08:59.199 INFO:tasks.workunit.client.0.smithi113.stdout:3/455: chown d0/d13/d15/d28/d2a/d34/f84 35 1 2024-04-12T05:08:59.200 INFO:tasks.workunit.client.0.smithi113.stdout:3/456: readlink d0/d13/d15/l3b 0 2024-04-12T05:08:59.201 INFO:tasks.workunit.client.0.smithi113.stdout:1/579: creat d10/da5/fb3 x:0 0 0 2024-04-12T05:08:59.203 INFO:tasks.workunit.client.0.smithi113.stdout:7/488: rename d0/dc/d6/d1a/f3a to d0/dd/fa5 0 2024-04-12T05:08:59.205 INFO:tasks.workunit.client.0.smithi113.stdout:6/448: fsync d8/de/d13/d11/d36/f4e 0 2024-04-12T05:08:59.206 INFO:tasks.workunit.client.0.smithi113.stdout:6/449: stat d8/de/f94 0 2024-04-12T05:08:59.207 INFO:tasks.workunit.client.1.smithi192.stdout:0/387: dread d0/d6/d15/d55/d29/f2b [0,4194304] 0 2024-04-12T05:08:59.209 INFO:tasks.workunit.client.1.smithi192.stdout:0/388: truncate d0/d6/d15/d55/d2e/d59/f84 341716 0 2024-04-12T05:08:59.210 INFO:tasks.workunit.client.0.smithi113.stdout:8/511: rename d9/d16/d1c/f51 to d9/d16/d1c/d59/d68/fa8 0 2024-04-12T05:08:59.212 INFO:tasks.workunit.client.1.smithi192.stdout:0/389: symlink d0/d6/d15/d55/l88 0 2024-04-12T05:08:59.214 INFO:tasks.workunit.client.0.smithi113.stdout:4/479: dread da/dd/d1b/f2c [0,4194304] 0 2024-04-12T05:08:59.216 INFO:tasks.workunit.client.0.smithi113.stdout:6/450: dwrite d8/de/d13/d11/d36/f61 [0,4194304] 0 2024-04-12T05:08:59.217 INFO:tasks.workunit.client.0.smithi113.stdout:4/480: chown da/d3c/d83/c50 6574299 1 2024-04-12T05:08:59.218 INFO:tasks.workunit.client.0.smithi113.stdout:5/419: link d0/d10/d27/f25 d0/d10/d27/d1d/d38/f8d 0 2024-04-12T05:08:59.220 INFO:tasks.workunit.client.0.smithi113.stdout:3/457: dread d0/d13/d15/f5c [0,4194304] 0 2024-04-12T05:08:59.221 INFO:tasks.workunit.client.0.smithi113.stdout:8/512: truncate d9/d1b/d8e/d93/d32/d5b/fa1 531381 0 2024-04-12T05:08:59.221 INFO:tasks.workunit.client.0.smithi113.stdout:4/481: creat da/d4a/f9d x:0 0 0 2024-04-12T05:08:59.221 INFO:tasks.workunit.client.0.smithi113.stdout:6/451: stat d8/de/d13/d11/d36/d3d/d4c/d31/c32 0 2024-04-12T05:08:59.224 INFO:tasks.workunit.client.1.smithi192.stdout:0/390: truncate d0/d6/d15/d19/f31 2786876 0 2024-04-12T05:08:59.225 INFO:tasks.workunit.client.0.smithi113.stdout:5/420: rename d0/c7d to d0/d10/d27/d1d/d8c/c8e 0 2024-04-12T05:08:59.226 INFO:tasks.workunit.client.0.smithi113.stdout:4/482: rmdir da/d24/d7e 39 2024-04-12T05:08:59.228 INFO:tasks.workunit.client.1.smithi192.stdout:0/391: unlink d0/d6/d15/d19/f44 0 2024-04-12T05:08:59.229 INFO:tasks.workunit.client.0.smithi113.stdout:3/458: unlink d0/d13/d15/d28/f6e 0 2024-04-12T05:08:59.230 INFO:tasks.workunit.client.1.smithi192.stdout:0/392: read d0/d6/d15/d19/f4c [322961,67529] 0 2024-04-12T05:08:59.231 INFO:tasks.workunit.client.0.smithi113.stdout:6/452: dwrite d8/de/d13/d2b/f6a [0,4194304] 0 2024-04-12T05:08:59.231 INFO:tasks.workunit.client.0.smithi113.stdout:8/513: creat d9/d16/d65/fa9 x:0 0 0 2024-04-12T05:08:59.232 INFO:tasks.workunit.client.0.smithi113.stdout:2/351: fsync d8/d9/f56 0 2024-04-12T05:08:59.234 INFO:tasks.workunit.client.1.smithi192.stdout:0/393: getdents d0 0 2024-04-12T05:08:59.234 INFO:tasks.workunit.client.0.smithi113.stdout:5/421: symlink d0/d10/d27/d1d/d41/l8f 0 2024-04-12T05:08:59.234 INFO:tasks.workunit.client.1.smithi192.stdout:0/394: stat d0/d6/d15/d55/d29/d68/f85 0 2024-04-12T05:08:59.235 INFO:tasks.workunit.client.1.smithi192.stdout:4/409: sync 2024-04-12T05:08:59.236 INFO:tasks.workunit.client.1.smithi192.stdout:4/410: truncate d1/d49/f5b 19430 0 2024-04-12T05:08:59.241 INFO:tasks.workunit.client.0.smithi113.stdout:4/483: dwrite da/d3c/d83/f40 [4194304,4194304] 0 2024-04-12T05:08:59.241 INFO:tasks.workunit.client.0.smithi113.stdout:6/453: write d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8f [4890275,15033] 0 2024-04-12T05:08:59.242 INFO:tasks.workunit.client.1.smithi192.stdout:0/395: rename d0/d6/d15/d55/d29/d68 to d0/d6/d15/d55/d29/d89 0 2024-04-12T05:08:59.242 INFO:tasks.workunit.client.1.smithi192.stdout:4/411: creat d1/d1d/d53/f89 x:0 0 0 2024-04-12T05:08:59.242 INFO:tasks.workunit.client.1.smithi192.stdout:0/396: write d0/d6/d15/d55/d2e/d66/f86 [572498,106962] 0 2024-04-12T05:08:59.244 INFO:tasks.workunit.client.0.smithi113.stdout:9/517: sync 2024-04-12T05:08:59.245 INFO:tasks.workunit.client.0.smithi113.stdout:7/489: dwrite d0/dc/f42 [0,4194304] 0 2024-04-12T05:08:59.245 INFO:tasks.workunit.client.0.smithi113.stdout:4/484: stat da/d24/d7b/c94 0 2024-04-12T05:08:59.246 INFO:tasks.workunit.client.0.smithi113.stdout:3/459: link d0/le d0/d7e/l9b 0 2024-04-12T05:08:59.246 INFO:tasks.workunit.client.1.smithi192.stdout:0/397: chown d0/cc 87325 1 2024-04-12T05:08:59.247 INFO:tasks.workunit.client.0.smithi113.stdout:9/518: stat d1/df/d4b/d6c/f6b 0 2024-04-12T05:08:59.247 INFO:tasks.workunit.client.0.smithi113.stdout:7/490: readlink d0/dc/d6/d1a/l1e 0 2024-04-12T05:08:59.249 INFO:tasks.workunit.client.0.smithi113.stdout:4/485: chown da/fb 11 1 2024-04-12T05:08:59.249 INFO:tasks.workunit.client.0.smithi113.stdout:2/352: truncate d8/d16/f61 1195349 0 2024-04-12T05:08:59.250 INFO:tasks.workunit.client.1.smithi192.stdout:4/412: creat d1/f8a x:0 0 0 2024-04-12T05:08:59.251 INFO:tasks.workunit.client.1.smithi192.stdout:4/413: stat d1/cb 0 2024-04-12T05:08:59.252 INFO:tasks.workunit.client.0.smithi113.stdout:6/454: chown d8/de/d13/d11/d36/d3d/d4c/d26 370210 1 2024-04-12T05:08:59.256 INFO:tasks.workunit.client.0.smithi113.stdout:5/422: write d0/d10/d27/d1d/d38/f8d [610338,7508] 0 2024-04-12T05:08:59.257 INFO:tasks.workunit.client.0.smithi113.stdout:6/455: dwrite d8/f2d [0,4194304] 0 2024-04-12T05:08:59.258 INFO:tasks.workunit.client.0.smithi113.stdout:5/423: chown d0/d10/d27/d1d/d3f/d7e 129 1 2024-04-12T05:08:59.260 INFO:tasks.workunit.client.1.smithi192.stdout:4/414: rmdir d1/d54/d55/d64 39 2024-04-12T05:08:59.260 INFO:tasks.workunit.client.1.smithi192.stdout:0/398: dwrite d0/d6/d15/d1b/d3a/d4f/f6a [0,4194304] 0 2024-04-12T05:08:59.262 INFO:tasks.workunit.client.1.smithi192.stdout:4/415: fsync d1/d1d/d53/d1c/f65 0 2024-04-12T05:08:59.266 INFO:tasks.workunit.client.1.smithi192.stdout:4/416: read - d1/d54/d55/d64/f74 zero size 2024-04-12T05:08:59.266 INFO:tasks.workunit.client.1.smithi192.stdout:0/399: readlink d0/d6/d15/d55/d29/l43 0 2024-04-12T05:08:59.267 INFO:tasks.workunit.client.0.smithi113.stdout:3/460: mknod d0/d13/d15/d28/c9c 0 2024-04-12T05:08:59.267 INFO:tasks.workunit.client.0.smithi113.stdout:9/519: mknod d1/df/d4b/d63/d3d/d52/cb2 0 2024-04-12T05:08:59.269 INFO:tasks.workunit.client.0.smithi113.stdout:3/461: readlink d0/d13/d15/l27 0 2024-04-12T05:08:59.269 INFO:tasks.workunit.client.1.smithi192.stdout:4/417: mkdir d1/d54/d55/d8b 0 2024-04-12T05:08:59.270 INFO:tasks.workunit.client.1.smithi192.stdout:4/418: write d1/d1d/d53/d1c/f36 [2675823,105768] 0 2024-04-12T05:08:59.271 INFO:tasks.workunit.client.1.smithi192.stdout:4/419: chown d1/d54/f61 1067 1 2024-04-12T05:08:59.272 INFO:tasks.workunit.client.0.smithi113.stdout:5/424: rename d0/d10/d27/d1d/d41/l69 to d0/d10/d27/l90 0 2024-04-12T05:08:59.272 INFO:tasks.workunit.client.0.smithi113.stdout:6/456: dwrite d8/de/d17/f78 [0,4194304] 0 2024-04-12T05:08:59.275 INFO:tasks.workunit.client.1.smithi192.stdout:4/420: symlink d1/d54/d55/d8b/l8c 0 2024-04-12T05:08:59.278 INFO:tasks.workunit.client.0.smithi113.stdout:5/425: write d0/d10/d27/f22 [2081084,100832] 0 2024-04-12T05:08:59.280 INFO:tasks.workunit.client.0.smithi113.stdout:7/491: truncate d0/dc/d6/f16 754687 0 2024-04-12T05:08:59.281 INFO:tasks.workunit.client.0.smithi113.stdout:3/462: mknod d0/d13/d15/d28/d2a/d34/d5f/c9d 0 2024-04-12T05:08:59.283 INFO:tasks.workunit.client.0.smithi113.stdout:8/514: dread d9/d1b/d8e/d93/d32/d5b/f63 [0,4194304] 0 2024-04-12T05:08:59.283 INFO:tasks.workunit.client.1.smithi192.stdout:4/421: rename d1/d6b/c88 to d1/d54/d82/c8d 0 2024-04-12T05:08:59.284 INFO:tasks.workunit.client.1.smithi192.stdout:0/400: fdatasync d0/d6/d15/d1b/d3a/d4f/f6a 0 2024-04-12T05:08:59.284 INFO:tasks.workunit.client.0.smithi113.stdout:6/457: unlink d8/de/d13/d2b/f87 0 2024-04-12T05:08:59.285 INFO:tasks.workunit.client.1.smithi192.stdout:4/422: chown d1/d54 159 1 2024-04-12T05:08:59.287 INFO:tasks.workunit.client.1.smithi192.stdout:4/423: mkdir d1/d1d/d53/d8e 0 2024-04-12T05:08:59.287 INFO:tasks.workunit.client.1.smithi192.stdout:4/424: fdatasync d1/f29 0 2024-04-12T05:08:59.288 INFO:tasks.workunit.client.0.smithi113.stdout:4/486: dwrite da/d3c/d43/f7d [0,4194304] 0 2024-04-12T05:08:59.290 INFO:tasks.workunit.client.1.smithi192.stdout:0/401: rmdir d0/d4e/d58 0 2024-04-12T05:08:59.291 INFO:tasks.workunit.client.1.smithi192.stdout:4/425: creat d1/d54/d55/d64/f8f x:0 0 0 2024-04-12T05:08:59.293 INFO:tasks.workunit.client.0.smithi113.stdout:3/463: truncate d0/fa 1439058 0 2024-04-12T05:08:59.294 INFO:tasks.workunit.client.0.smithi113.stdout:9/520: dread d1/d4/d38/f8c [0,4194304] 0 2024-04-12T05:08:59.297 INFO:tasks.workunit.client.1.smithi192.stdout:0/402: creat d0/d6/d67/f8a x:0 0 0 2024-04-12T05:08:59.298 INFO:tasks.workunit.client.0.smithi113.stdout:5/426: dread d0/d10/d27/d1d/d3f/f47 [0,4194304] 0 2024-04-12T05:08:59.298 INFO:tasks.workunit.client.0.smithi113.stdout:7/492: dwrite d0/fa2 [0,4194304] 0 2024-04-12T05:08:59.298 INFO:tasks.workunit.client.1.smithi192.stdout:0/403: read d0/d6/d15/d1b/d46/d73/f38 [2954848,103546] 0 2024-04-12T05:08:59.303 INFO:tasks.workunit.client.0.smithi113.stdout:9/521: read d1/d4/d5/f58 [912382,51117] 0 2024-04-12T05:08:59.305 INFO:tasks.workunit.client.0.smithi113.stdout:8/515: dwrite d9/d1b/d96/f9b [0,4194304] 0 2024-04-12T05:08:59.306 INFO:tasks.workunit.client.1.smithi192.stdout:0/404: dwrite d0/d4e/d53/d6d/f62 [0,4194304] 0 2024-04-12T05:08:59.308 INFO:tasks.workunit.client.0.smithi113.stdout:7/493: truncate d0/dc/d6/d78/f7c 286688 0 2024-04-12T05:08:59.316 INFO:tasks.workunit.client.0.smithi113.stdout:6/458: dwrite d8/de/d13/d11/d36/d3d/d4c/d26/d64/f8c [0,4194304] 0 2024-04-12T05:08:59.318 INFO:tasks.workunit.client.1.smithi192.stdout:4/426: dread d1/d1d/d53/d1c/f36 [0,4194304] 0 2024-04-12T05:08:59.318 INFO:tasks.workunit.client.0.smithi113.stdout:9/522: unlink d1/d4/d5/d22/d9e/faa 0 2024-04-12T05:08:59.325 INFO:tasks.workunit.client.1.smithi192.stdout:4/427: read d1/d1d/d53/d1c/f66 [455453,101633] 0 2024-04-12T05:08:59.331 INFO:tasks.workunit.client.0.smithi113.stdout:4/487: dwrite da/d73/f1c [0,4194304] 0 2024-04-12T05:08:59.331 INFO:tasks.workunit.client.0.smithi113.stdout:7/494: creat d0/dd/d11/d99/fa6 x:0 0 0 2024-04-12T05:08:59.332 INFO:tasks.workunit.client.0.smithi113.stdout:5/427: dwrite d0/fc [0,4194304] 0 2024-04-12T05:08:59.333 INFO:tasks.workunit.client.0.smithi113.stdout:3/464: truncate d0/d13/d47/d4f/f4c 6650437 0 2024-04-12T05:08:59.335 INFO:tasks.workunit.client.1.smithi192.stdout:0/405: truncate d0/d6/d15/d1b/d3a/d4f/f24 3266213 0 2024-04-12T05:08:59.337 INFO:tasks.workunit.client.0.smithi113.stdout:9/523: creat d1/d4/d5/d6/d15/fb3 x:0 0 0 2024-04-12T05:08:59.338 INFO:tasks.workunit.client.1.smithi192.stdout:0/406: symlink d0/d6/d15/l8b 0 2024-04-12T05:08:59.339 INFO:tasks.workunit.client.0.smithi113.stdout:8/516: dwrite d9/d16/d65/d8c/fa4 [0,4194304] 0 2024-04-12T05:08:59.340 INFO:tasks.workunit.client.0.smithi113.stdout:9/524: readlink d1/d4/d5/d17/l56 0 2024-04-12T05:08:59.340 INFO:tasks.workunit.client.0.smithi113.stdout:7/495: write d0/f48 [944279,35432] 0 2024-04-12T05:08:59.342 INFO:tasks.workunit.client.0.smithi113.stdout:4/488: mknod da/dd/d3a/c9e 0 2024-04-12T05:08:59.346 INFO:tasks.workunit.client.0.smithi113.stdout:6/459: dwrite d8/de/d13/d11/d36/f4e [4194304,4194304] 0 2024-04-12T05:08:59.346 INFO:tasks.workunit.client.0.smithi113.stdout:8/517: mknod d9/d16/d1c/d59/d68/caa 0 2024-04-12T05:08:59.346 INFO:tasks.workunit.client.0.smithi113.stdout:9/525: chown d1/df/d4b/d63/l80 2441091 1 2024-04-12T05:08:59.350 INFO:tasks.workunit.client.0.smithi113.stdout:7/496: read d0/dc/d6/d78/f31 [3414616,126120] 0 2024-04-12T05:08:59.350 INFO:tasks.workunit.client.1.smithi192.stdout:0/407: write d0/d6/d15/d55/f48 [260017,28920] 0 2024-04-12T05:08:59.354 INFO:tasks.workunit.client.0.smithi113.stdout:8/518: write d9/d1b/d8e/d93/d32/f60 [798478,96038] 0 2024-04-12T05:08:59.354 INFO:tasks.workunit.client.1.smithi192.stdout:0/408: creat d0/d6/d15/d1b/d46/f8c x:0 0 0 2024-04-12T05:08:59.356 INFO:tasks.workunit.client.0.smithi113.stdout:3/465: dwrite d0/f12 [4194304,4194304] 0 2024-04-12T05:08:59.356 INFO:tasks.workunit.client.0.smithi113.stdout:8/519: write d9/d1b/f45 [1109328,39746] 0 2024-04-12T05:08:59.357 INFO:tasks.workunit.client.0.smithi113.stdout:9/526: rename d1/c1e to d1/df/d29/da6/cb4 0 2024-04-12T05:08:59.358 INFO:tasks.workunit.client.0.smithi113.stdout:6/460: mknod d8/de/d13/d11/d36/d3d/ca4 0 2024-04-12T05:08:59.358 INFO:tasks.workunit.client.1.smithi192.stdout:0/409: creat d0/d6/d15/d55/d2e/d59/f8d x:0 0 0 2024-04-12T05:08:59.359 INFO:tasks.workunit.client.0.smithi113.stdout:9/527: write d1/d4/d5/d22/f9c [473683,576] 0 2024-04-12T05:08:59.359 INFO:tasks.workunit.client.1.smithi192.stdout:0/410: chown d0/c41 237869 1 2024-04-12T05:08:59.360 INFO:tasks.workunit.client.1.smithi192.stdout:0/411: read - d0/d4e/f87 zero size 2024-04-12T05:08:59.361 INFO:tasks.workunit.client.0.smithi113.stdout:7/497: write d0/dd/d11/d44/d5c/d85/fa4 [1254743,118759] 0 2024-04-12T05:08:59.363 INFO:tasks.workunit.client.0.smithi113.stdout:5/428: link d0/d10/l2d d0/l91 0 2024-04-12T05:08:59.364 INFO:tasks.workunit.client.0.smithi113.stdout:5/429: stat d0/d10/d27/d1d/d3f/c54 0 2024-04-12T05:08:59.364 INFO:tasks.workunit.client.1.smithi192.stdout:0/412: rmdir d0/d5e 0 2024-04-12T05:08:59.365 INFO:tasks.workunit.client.0.smithi113.stdout:5/430: readlink d0/d10/d27/d1d/d38/d3b/l72 0 2024-04-12T05:08:59.365 INFO:tasks.workunit.client.0.smithi113.stdout:5/431: write d0/f55 [313146,50585] 0 2024-04-12T05:08:59.367 INFO:tasks.workunit.client.0.smithi113.stdout:6/461: symlink d8/de/d13/d11/la5 0 2024-04-12T05:08:59.368 INFO:tasks.workunit.client.0.smithi113.stdout:8/520: creat d9/d1b/d8e/fab x:0 0 0 2024-04-12T05:08:59.368 INFO:tasks.workunit.client.0.smithi113.stdout:8/521: read - d9/f9e zero size 2024-04-12T05:08:59.369 INFO:tasks.workunit.client.0.smithi113.stdout:9/528: mkdir d1/df/d29/db5 0 2024-04-12T05:08:59.370 INFO:tasks.workunit.client.0.smithi113.stdout:7/498: symlink d0/dd/d11/d44/d5c/la7 0 2024-04-12T05:08:59.372 INFO:tasks.workunit.client.0.smithi113.stdout:6/462: symlink d8/la6 0 2024-04-12T05:08:59.373 INFO:tasks.workunit.client.1.smithi192.stdout:4/428: dread d1/d1d/d53/f41 [0,4194304] 0 2024-04-12T05:08:59.374 INFO:tasks.workunit.client.0.smithi113.stdout:5/432: dwrite d0/f6 [0,4194304] 0 2024-04-12T05:08:59.375 INFO:tasks.workunit.client.1.smithi192.stdout:4/429: rmdir d1/d6b 39 2024-04-12T05:08:59.376 INFO:tasks.workunit.client.0.smithi113.stdout:9/529: rename d1/df/d4b/d6c to d1/df/db6 0 2024-04-12T05:08:59.376 INFO:tasks.workunit.client.0.smithi113.stdout:8/522: dread d9/d1b/d8e/d93/d32/f64 [0,4194304] 0 2024-04-12T05:08:59.379 INFO:tasks.workunit.client.1.smithi192.stdout:4/430: read d1/d1d/d53/d1c/f4e [1076263,105915] 0 2024-04-12T05:08:59.380 INFO:tasks.workunit.client.0.smithi113.stdout:7/499: mknod d0/dd/d2d/ca8 0 2024-04-12T05:08:59.383 INFO:tasks.workunit.client.0.smithi113.stdout:8/523: readlink d9/d16/d7f/l2b 0 2024-04-12T05:08:59.383 INFO:tasks.workunit.client.0.smithi113.stdout:9/530: rename d1/df/d4b/d63/d3d/d52 to d1/df/d4b/d63/d3d/dab/db7 0 2024-04-12T05:08:59.385 INFO:tasks.workunit.client.0.smithi113.stdout:8/524: write d9/d1b/d96/f9b [4981133,94210] 0 2024-04-12T05:08:59.385 INFO:tasks.workunit.client.0.smithi113.stdout:9/531: chown d1/d4/d5/d22/l4e 317958 1 2024-04-12T05:08:59.398 INFO:tasks.workunit.client.1.smithi192.stdout:6/421: sync 2024-04-12T05:08:59.400 INFO:tasks.workunit.client.1.smithi192.stdout:6/422: write d3/d5/f72 [618617,102510] 0 2024-04-12T05:08:59.401 INFO:tasks.workunit.client.1.smithi192.stdout:0/413: truncate d0/d6/d67/f8a 303306 0 2024-04-12T05:08:59.402 INFO:tasks.workunit.client.1.smithi192.stdout:6/423: chown d3/d8/d56/f75 1135772 1 2024-04-12T05:08:59.403 INFO:tasks.workunit.client.1.smithi192.stdout:6/424: read d3/d8/f13 [449624,44382] 0 2024-04-12T05:08:59.404 INFO:tasks.workunit.client.1.smithi192.stdout:6/425: read d3/d5/f51 [252366,102922] 0 2024-04-12T05:08:59.415 INFO:tasks.workunit.client.0.smithi113.stdout:9/532: creat d1/d4/d6d/fb8 x:0 0 0 2024-04-12T05:08:59.415 INFO:tasks.workunit.client.0.smithi113.stdout:7/500: rename d0/dd/d11/d44 to d0/dc/da9 0 2024-04-12T05:08:59.416 INFO:tasks.workunit.client.1.smithi192.stdout:0/414: mknod d0/d6/d15/d1b/d3a/d4f/c8e 0 2024-04-12T05:08:59.417 INFO:tasks.workunit.client.0.smithi113.stdout:8/525: mkdir d9/d16/d7f/d3a/d6c/d90/dac 0 2024-04-12T05:08:59.418 INFO:tasks.workunit.client.0.smithi113.stdout:6/463: stat d8/de/d17/f7b 0 2024-04-12T05:08:59.419 INFO:tasks.workunit.client.1.smithi192.stdout:0/415: symlink d0/d6/d67/l8f 0 2024-04-12T05:08:59.420 INFO:tasks.workunit.client.0.smithi113.stdout:8/526: fsync d9/d16/d1c/f2c 0 2024-04-12T05:08:59.421 INFO:tasks.workunit.client.1.smithi192.stdout:0/416: chown d0/d6/d15/d55/d2e/d59/f78 6927 1 2024-04-12T05:08:59.424 INFO:tasks.workunit.client.1.smithi192.stdout:6/426: rename d3/d26/f3c to d3/d21/f8d 0 2024-04-12T05:08:59.426 INFO:tasks.workunit.client.1.smithi192.stdout:6/427: fdatasync d3/d8/d10/d4c/d68/f77 0 2024-04-12T05:08:59.426 INFO:tasks.workunit.client.0.smithi113.stdout:7/501: rename d0/dd/d11/d99 to d0/dc/d6/d78/daa 0 2024-04-12T05:08:59.427 INFO:tasks.workunit.client.0.smithi113.stdout:9/533: dwrite d1/d4/d38/f62 [0,4194304] 0 2024-04-12T05:08:59.427 INFO:tasks.workunit.client.1.smithi192.stdout:6/428: fdatasync d3/d8/f17 0 2024-04-12T05:08:59.428 INFO:tasks.workunit.client.0.smithi113.stdout:7/502: readlink d0/dc/d6/d78/l4d 0 2024-04-12T05:08:59.428 INFO:tasks.workunit.client.1.smithi192.stdout:0/417: creat d0/d6/d15/d1b/d46/d73/d4b/d72/f90 x:0 0 0 2024-04-12T05:08:59.429 INFO:tasks.workunit.client.0.smithi113.stdout:8/527: read d9/d16/d1c/f2d [399683,20647] 0 2024-04-12T05:08:59.429 INFO:tasks.workunit.client.1.smithi192.stdout:4/431: dwrite d1/d1d/d53/d1c/f66 [0,4194304] 0 2024-04-12T05:08:59.430 INFO:tasks.workunit.client.0.smithi113.stdout:6/464: symlink d8/de/d13/d11/d36/d3d/d4c/d26/la7 0 2024-04-12T05:08:59.434 INFO:tasks.workunit.client.0.smithi113.stdout:5/433: truncate d0/d10/d27/d1d/d3f/d73/d89/f62 4067046 0 2024-04-12T05:08:59.434 INFO:tasks.workunit.client.0.smithi113.stdout:6/465: dread - d8/de/d13/d11/d36/d2c/fa3 zero size 2024-04-12T05:08:59.435 INFO:tasks.workunit.client.0.smithi113.stdout:7/503: truncate d0/dc/d6/d78/daa/fa6 757537 0 2024-04-12T05:08:59.435 INFO:tasks.workunit.client.1.smithi192.stdout:6/429: creat d3/d8/d56/f8e x:0 0 0 2024-04-12T05:08:59.436 INFO:tasks.workunit.client.0.smithi113.stdout:9/534: rmdir d1/df/d4b/d63/d3d 39 2024-04-12T05:08:59.438 INFO:tasks.workunit.client.0.smithi113.stdout:7/504: creat d0/dc/da9/d5c/d9a/fab x:0 0 0 2024-04-12T05:08:59.440 INFO:tasks.workunit.client.1.smithi192.stdout:4/432: creat d1/d76/f90 x:0 0 0 2024-04-12T05:08:59.442 INFO:tasks.workunit.client.0.smithi113.stdout:5/434: creat d0/d10/d27/d1d/d38/d44/f92 x:0 0 0 2024-04-12T05:08:59.443 INFO:tasks.workunit.client.1.smithi192.stdout:6/430: dwrite d3/d26/f28 [0,4194304] 0 2024-04-12T05:08:59.444 INFO:tasks.workunit.client.0.smithi113.stdout:8/528: getdents d9/d1b/d8e 0 2024-04-12T05:08:59.444 INFO:tasks.workunit.client.0.smithi113.stdout:4/489: dread da/d4f/f72 [0,4194304] 0 2024-04-12T05:08:59.445 INFO:tasks.workunit.client.0.smithi113.stdout:6/466: dread - d8/de/d17/f59 zero size 2024-04-12T05:08:59.448 INFO:tasks.workunit.client.1.smithi192.stdout:4/433: chown d1/d54/d55/d64/d68/l80 40618641 1 2024-04-12T05:08:59.450 INFO:tasks.workunit.client.1.smithi192.stdout:4/434: truncate d1/d76/f79 513600 0 2024-04-12T05:08:59.451 INFO:tasks.workunit.client.1.smithi192.stdout:4/435: readlink d1/d54/d55/d8b/l8c 0 2024-04-12T05:08:59.452 INFO:tasks.workunit.client.1.smithi192.stdout:0/418: truncate d0/d6/d15/d19/f1d 2958618 0 2024-04-12T05:08:59.454 INFO:tasks.workunit.client.1.smithi192.stdout:4/436: write d1/d54/d82/f48 [464715,127633] 0 2024-04-12T05:08:59.456 INFO:tasks.workunit.client.0.smithi113.stdout:9/535: getdents d1/df/d4b/d63 0 2024-04-12T05:08:59.457 INFO:tasks.workunit.client.1.smithi192.stdout:6/431: dwrite d3/d8/d10/d4c/f82 [0,4194304] 0 2024-04-12T05:08:59.458 INFO:tasks.workunit.client.0.smithi113.stdout:6/467: symlink d8/de/d13/d2b/d76/la8 0 2024-04-12T05:08:59.460 INFO:tasks.workunit.client.1.smithi192.stdout:6/432: symlink d3/d8/d10/l8f 0 2024-04-12T05:08:59.461 INFO:tasks.workunit.client.0.smithi113.stdout:7/505: dwrite d0/dc/d50/f5f [0,4194304] 0 2024-04-12T05:08:59.462 INFO:tasks.workunit.client.1.smithi192.stdout:0/419: mknod d0/d7b/d7c/c91 0 2024-04-12T05:08:59.466 INFO:tasks.workunit.client.1.smithi192.stdout:0/420: write d0/d6/d15/d55/d2e/f42 [3014122,15433] 0 2024-04-12T05:08:59.467 INFO:tasks.workunit.client.0.smithi113.stdout:8/529: truncate d9/d16/d3c/f74 1894787 0 2024-04-12T05:08:59.467 INFO:tasks.workunit.client.1.smithi192.stdout:6/433: mknod d3/d8/d10/d4c/c90 0 2024-04-12T05:08:59.468 INFO:tasks.workunit.client.1.smithi192.stdout:0/421: chown d0/d6/d15/d19/f31 3605 1 2024-04-12T05:08:59.469 INFO:tasks.workunit.client.1.smithi192.stdout:0/422: readlink d0/d6/l7f 0 2024-04-12T05:08:59.469 INFO:tasks.workunit.client.0.smithi113.stdout:5/435: dwrite d0/d10/d27/d1d/d3f/d73/f76 [0,4194304] 0 2024-04-12T05:08:59.469 INFO:tasks.workunit.client.1.smithi192.stdout:0/423: fsync d0/d6/d15/d55/d2e/d66/f70 0 2024-04-12T05:08:59.470 INFO:tasks.workunit.client.0.smithi113.stdout:4/490: dwrite da/d3c/d43/f5e [0,4194304] 0 2024-04-12T05:08:59.470 INFO:tasks.workunit.client.1.smithi192.stdout:6/434: rename d3/d8/f7e to d3/d26/f91 0 2024-04-12T05:08:59.473 INFO:tasks.workunit.client.0.smithi113.stdout:6/468: chown d8/de/d13/d11/d36/d3d/d4c/d31/d8e/d63/d66/f7e 144 1 2024-04-12T05:08:59.473 INFO:tasks.workunit.client.0.smithi113.stdout:8/530: rmdir d9/d16/d1c 39 2024-04-12T05:08:59.478 INFO:tasks.workunit.client.1.smithi192.stdout:0/424: unlink d0/d6/d15/d55/d2e/d59/f8d 0 2024-04-12T05:08:59.481 INFO:tasks.workunit.client.1.smithi192.stdout:0/425: creat d0/d6/d67/f92 x:0 0 0 2024-04-12T05:08:59.482 INFO:tasks.workunit.client.1.smithi192.stdout:6/435: creat d3/d8/f92 x:0 0 0 2024-04-12T05:08:59.483 INFO:tasks.workunit.client.0.smithi113.stdout:5/436: dwrite d0/d10/f1b [0,4194304] 0 2024-04-12T05:08:59.485 INFO:tasks.workunit.client.0.smithi113.stdout:7/506: rename d0/dc/d6/d78/c90 to d0/dc/d6/d9b/cac 0 2024-04-12T05:08:59.490 INFO:tasks.workunit.client.1.smithi192.stdout:0/426: creat d0/d6/d15/d55/d29/d89/f93 x:0 0 0 2024-04-12T05:08:59.492 INFO:tasks.workunit.client.1.smithi192.stdout:6/436: dwrite d3/d8/d22/f65 [0,4194304] 0 2024-04-12T05:08:59.495 INFO:tasks.workunit.client.0.smithi113.stdout:5/437: write d0/d10/f45 [793659,81704] 0 2024-04-12T05:08:59.495 INFO:tasks.workunit.client.0.smithi113.stdout:9/536: dwrite d1/df/d4b/d63/d3d/d6f/faf [0,4194304] 0 2024-04-12T05:08:59.496 INFO:tasks.workunit.client.0.smithi113.stdout:8/531: truncate d9/f53 256195 0 2024-04-12T05:08:59.496 INFO:tasks.workunit.client.0.smithi113.stdout:4/491: rename da/d73/d8e to da/d3c/d9f 0 2024-04-12T05:08:59.498 INFO:tasks.workunit.client.1.smithi192.stdout:4/437: dread d1/f4b [0,4194304] 0 2024-04-12T05:08:59.498 INFO:tasks.workunit.client.1.smithi192.stdout:4/438: fdatasync d1/d54/f81 0 2024-04-12T05:08:59.499 INFO:tasks.workunit.client.0.smithi113.stdout:5/438: chown d0/d10/d6e 27 1 2024-04-12T05:08:59.500 INFO:tasks.workunit.client.0.smithi113.stdout:7/507: mkdir d0/dc/d6/d1a/da0/dad 0 2024-04-12T05:08:59.502 INFO:tasks.workunit.client.0.smithi113.stdout:4/492: fdatasync da/dd/f6e 0 2024-04-12T05:08:59.502 INFO:tasks.workunit.client.1.smithi192.stdout:6/437: symlink d3/d8/d56/d59/l93 0 2024-04-12T05:08:59.504 INFO:tasks.workunit.client.1.smithi192.stdout:6/438: dread - d3/d8/d10/d4c/d68/f89 zero size 2024-04-12T05:08:59.504 INFO:tasks.workunit.client.1.smithi192.stdout:4/439: unlink d1/d1d/d53/c5f 0 2024-04-12T05:08:59.505 INFO:tasks.workunit.client.0.smithi113.stdout:6/469: rename d8/de/d13/d11/d36/d3d/d4c/d31/c32 to d8/de/d13/d11/d36/ca9 0 2024-04-12T05:08:59.506 INFO:tasks.workunit.client.1.smithi192.stdout:4/440: write d1/d54/f6e [622997,65152] 0 2024-04-12T05:08:59.508 INFO:tasks.workunit.client.0.smithi113.stdout:0/535: sync 2024-04-12T05:08:59.508 INFO:tasks.workunit.client.0.smithi113.stdout:2/353: sync 2024-04-12T05:08:59.511 INFO:tasks.workunit.client.0.smithi113.stdout:1/580: sync 2024-04-12T05:08:59.512 INFO:tasks.workunit.client.0.smithi113.stdout:5/439: mkdir d0/d10/d6e/d93 0 2024-04-12T05:08:59.514 INFO:tasks.workunit.client.0.smithi113.stdout:4/493: creat da/d2b/fa0 x:0 0 0 2024-04-12T05:08:59.515 INFO:tasks.workunit.client.0.smithi113.stdout:9/537: dwrite d1/df/d29/fa2 [0,4194304] 0 2024-04-12T05:08:59.516 INFO:tasks.workunit.client.1.smithi192.stdout:4/441: getdents d1/d6b 0 2024-04-12T05:08:59.517 INFO:tasks.workunit.client.1.smithi192.stdout:6/439: dread d3/dc/d39/f3a [0,4194304] 0 2024-04-12T05:08:59.519 INFO:tasks.workunit.client.1.smithi192.stdout:4/442: truncate d1/d1d/d53/f57 267433 0 2024-04-12T05:08:59.519 INFO:tasks.workunit.client.1.smithi192.stdout:6/440: stat d3/d8/d10/d4c/f82 0 2024-04-12T05:08:59.519 INFO:tasks.workunit.client.0.smithi113.stdout:6/470: fdatasync d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8d 0 2024-04-12T05:08:59.520 INFO:tasks.workunit.client.0.smithi113.stdout:7/508: unlink d0/dc/d6/d78/c2f 0 2024-04-12T05:08:59.521 INFO:tasks.workunit.client.1.smithi192.stdout:6/441: dread - d3/d21/f7d zero size 2024-04-12T05:08:59.522 INFO:tasks.workunit.client.0.smithi113.stdout:1/581: symlink d10/da5/lb4 0 2024-04-12T05:08:59.523 INFO:tasks.workunit.client.1.smithi192.stdout:0/427: dwrite d0/d6/d15/d19/f4c [0,4194304] 0 2024-04-12T05:08:59.527 INFO:tasks.workunit.client.0.smithi113.stdout:3/466: sync 2024-04-12T05:08:59.527 INFO:tasks.workunit.client.0.smithi113.stdout:4/494: symlink da/d3c/d83/d47/la1 0 2024-04-12T05:08:59.528 INFO:tasks.workunit.client.0.smithi113.stdout:5/440: unlink d0/d10/f12 0 2024-04-12T05:08:59.529 INFO:tasks.workunit.client.1.smithi192.stdout:0/428: stat d0/d6/d15/d19 0 2024-04-12T05:08:59.531 INFO:tasks.workunit.client.1.smithi192.stdout:6/442: creat d3/f94 x:0 0 0 2024-04-12T05:08:59.533 INFO:tasks.workunit.client.0.smithi113.stdout:4/495: fdatasync da/d4a/f9d 0 2024-04-12T05:08:59.533 INFO:tasks.workunit.client.0.smithi113.stdout:6/471: readlink d8/l33 0 2024-04-12T05:08:59.533 INFO:tasks.workunit.client.0.smithi113.stdout:2/354: dread d8/d9/de/d15/f40 [0,4194304] 0 2024-04-12T05:08:59.535 INFO:tasks.workunit.client.1.smithi192.stdout:4/443: dwrite d1/d76/f90 [0,4194304] 0 2024-04-12T05:08:59.536 INFO:tasks.workunit.client.0.smithi113.stdout:0/536: creat d1/fa6 x:0 0 0 2024-04-12T05:08:59.536 INFO:tasks.workunit.client.1.smithi192.stdout:4/444: readlink d1/d1d/d53/l47 0 2024-04-12T05:08:59.539 INFO:tasks.workunit.client.1.smithi192.stdout:0/429: symlink d0/d6/d15/d1b/d3a/d4f/l94 0 2024-04-12T05:08:59.539 INFO:tasks.workunit.client.1.smithi192.stdout:0/430: chown d0/d6/d15/d1b/d46/f4d 5 1 2024-04-12T05:08:59.540 INFO:tasks.workunit.client.0.smithi113.stdout:4/496: dwrite da/d2b/f3e [4194304,4194304] 0 2024-04-12T05:08:59.540 INFO:tasks.workunit.client.0.smithi113.stdout:4/497: chown da 120181 1 2024-04-12T05:08:59.542 INFO:tasks.workunit.client.0.smithi113.stdout:9/538: creat d1/d4/d5/fb9 x:0 0 0 2024-04-12T05:08:59.544 INFO:tasks.workunit.client.1.smithi192.stdout:4/445: rmdir d1/d1d/d53/d8e 0 2024-04-12T05:08:59.545 INFO:tasks.workunit.client.0.smithi113.stdout:3/467: rename d0/d13/d47/f6c to d0/d13/d15/d28/d2a/f9e 0 2024-04-12T05:08:59.549 INFO:tasks.workunit.client.0.smithi113.stdout:2/355: symlink d8/d9/de/d43/l62 0 2024-04-12T05:08:59.549 INFO:tasks.workunit.client.1.smithi192.stdout:4/446: creat d1/d54/d55/d64/f91 x:0 0 0 2024-04-12T05:08:59.551 INFO:tasks.workunit.client.0.smithi113.stdout:7/509: link d0/dc/d6/d78/f31 d0/dc/da9/d5c/d9a/fae 0 2024-04-12T05:08:59.552 INFO:tasks.workunit.client.0.smithi113.stdout:3/468: chown d0/f12 190319392 1 2024-04-12T05:08:59.552 INFO:tasks.workunit.client.0.smithi113.stdout:1/582: link d10/f9a d10/d14/d67/fb5 0 2024-04-12T05:08:59.554 INFO:tasks.workunit.client.0.smithi113.stdout:9/539: mkdir d1/df/d4b/d63/d3d/d6f/dba 0 2024-04-12T05:08:59.556 INFO:tasks.workunit.client.1.smithi192.stdout:4/447: link d1/d54/d55/f72 d1/d76/f92 0 2024-04-12T05:08:59.556 INFO:tasks.workunit.client.0.smithi113.stdout:0/537: getdents d1/d6 0 2024-04-12T05:08:59.557 INFO:tasks.workunit.client.0.smithi113.stdout:7/510: stat d0/dc/d6/d9b/f5d 0 2024-04-12T05:08:59.558 INFO:tasks.workunit.client.0.smithi113.stdout:1/583: dread - d10/d2e/d31/d33/f61 zero size 2024-04-12T05:08:59.562 INFO:tasks.workunit.client.1.smithi192.stdout:3/314: sync 2024-04-12T05:08:59.562 INFO:tasks.workunit.client.1.smithi192.stdout:4/448: fdatasync d1/d76/f90 0 2024-04-12T05:08:59.563 INFO:tasks.workunit.client.0.smithi113.stdout:8/532: dwrite d9/d16/d1c/f2c [0,4194304] 0 2024-04-12T05:08:59.563 INFO:tasks.workunit.client.0.smithi113.stdout:0/538: creat d1/fa7 x:0 0 0 2024-04-12T05:08:59.564 INFO:tasks.workunit.client.0.smithi113.stdout:7/511: chown d0/dd/d11/f8c 1594204834 1 2024-04-12T05:08:59.564 INFO:tasks.workunit.client.0.smithi113.stdout:0/539: chown d1/d6/d16 8594 1 2024-04-12T05:08:59.565 INFO:tasks.workunit.client.1.smithi192.stdout:6/443: truncate d3/d8/d48/f50 1064170 0 2024-04-12T05:08:59.568 INFO:tasks.workunit.client.0.smithi113.stdout:7/512: readlink d0/dc/d6/d78/daa/la3 0 2024-04-12T05:08:59.568 INFO:tasks.workunit.client.0.smithi113.stdout:9/540: link d1/df/db6/f6b d1/d4/d87/fbb 0 2024-04-12T05:08:59.569 INFO:tasks.workunit.client.1.smithi192.stdout:3/315: symlink d5/d13/d3a/l69 0 2024-04-12T05:08:59.570 INFO:tasks.workunit.client.1.smithi192.stdout:4/449: symlink d1/d1d/d53/d1c/l93 0 2024-04-12T05:08:59.570 INFO:tasks.workunit.client.1.smithi192.stdout:3/316: fdatasync d5/d13/d3c/f24 0 2024-04-12T05:08:59.571 INFO:tasks.workunit.client.1.smithi192.stdout:4/450: chown d1/l77 16394568 1 2024-04-12T05:08:59.572 INFO:tasks.workunit.client.1.smithi192.stdout:0/431: dread d0/d6/d15/d19/f31 [0,4194304] 0 2024-04-12T05:08:59.576 INFO:tasks.workunit.client.0.smithi113.stdout:9/541: dread - d1/d4/d5/d6/f76 zero size 2024-04-12T05:08:59.576 INFO:tasks.workunit.client.1.smithi192.stdout:4/451: rename d1/d1d/d53/f41 to d1/d54/f94 0 2024-04-12T05:08:59.578 INFO:tasks.workunit.client.1.smithi192.stdout:0/432: stat d0/d6/d15/d19/f71 0 2024-04-12T05:08:59.579 INFO:tasks.workunit.client.0.smithi113.stdout:0/540: rename d1/d9/f9d to d1/d6/da/fa8 0 2024-04-12T05:08:59.579 INFO:tasks.workunit.client.0.smithi113.stdout:6/472: dread d8/de/d13/fa [0,4194304] 0 2024-04-12T05:08:59.581 INFO:tasks.workunit.client.0.smithi113.stdout:0/541: stat d1/d6/da/d36/l76 0 2024-04-12T05:08:59.582 INFO:tasks.workunit.client.0.smithi113.stdout:5/441: write d0/f31 [260442,101830] 0 2024-04-12T05:08:59.582 INFO:tasks.workunit.client.0.smithi113.stdout:0/542: fsync d1/d6/f91 0 2024-04-12T05:08:59.582 INFO:tasks.workunit.client.1.smithi192.stdout:6/444: write d3/dc/f69 [1864689,71037] 0 2024-04-12T05:08:59.585 INFO:tasks.workunit.client.0.smithi113.stdout:3/469: getdents d0/d13/d15/d28/d2a 0 2024-04-12T05:08:59.586 INFO:tasks.workunit.client.1.smithi192.stdout:0/433: mkdir d0/d4e/d95 0 2024-04-12T05:08:59.586 INFO:tasks.workunit.client.1.smithi192.stdout:0/434: write d0/d6/d15/f6f [3073844,15945] 0 2024-04-12T05:08:59.587 INFO:tasks.workunit.client.1.smithi192.stdout:0/435: stat d0/d6/d15 0 2024-04-12T05:08:59.590 INFO:tasks.workunit.client.0.smithi113.stdout:3/470: rmdir d0/d7e 39 2024-04-12T05:08:59.592 INFO:tasks.workunit.client.0.smithi113.stdout:1/584: truncate d10/d55/f8b 1112696 0 2024-04-12T05:08:59.595 INFO:tasks.workunit.client.1.smithi192.stdout:0/436: dwrite d0/d4e/d53/d6d/f62 [0,4194304] 0 2024-04-12T05:08:59.596 INFO:tasks.workunit.client.0.smithi113.stdout:6/473: rename d8/de/d13/d11/d36/d3d/d4c/d31/d8e/c3c to d8/de/d13/d11/d36/d3d/d93/caa 0 2024-04-12T05:08:59.597 INFO:tasks.workunit.client.0.smithi113.stdout:3/471: mknod d0/d13/c9f 0 2024-04-12T05:08:59.599 INFO:tasks.workunit.client.1.smithi192.stdout:3/317: dread d5/d13/d3c/d26/f4d [0,4194304] 0 2024-04-12T05:08:59.601 INFO:tasks.workunit.client.1.smithi192.stdout:0/437: unlink d0/d6/d15/d1b/d3a/d4f/l5f 0 2024-04-12T05:08:59.601 INFO:tasks.workunit.client.0.smithi113.stdout:3/472: rmdir d0 39 2024-04-12T05:08:59.602 INFO:tasks.workunit.client.0.smithi113.stdout:1/585: mknod d10/d41/d89/d96/cb6 0 2024-04-12T05:08:59.603 INFO:tasks.workunit.client.0.smithi113.stdout:2/356: dread d8/d9/d1a/f3a [0,4194304] 0 2024-04-12T05:08:59.606 INFO:tasks.workunit.client.1.smithi192.stdout:0/438: getdents d0/d6/d15/d55/d29/d89 0 2024-04-12T05:08:59.610 INFO:tasks.workunit.client.0.smithi113.stdout:0/543: dwrite d1/d6/d16/f67 [0,4194304] 0 2024-04-12T05:08:59.610 INFO:tasks.workunit.client.0.smithi113.stdout:6/474: creat d8/de/d13/d2b/d76/fab x:0 0 0 2024-04-12T05:08:59.611 INFO:tasks.workunit.client.1.smithi192.stdout:0/439: creat d0/d6/f96 x:0 0 0 2024-04-12T05:08:59.611 INFO:tasks.workunit.client.0.smithi113.stdout:9/542: dread d1/df/f36 [0,4194304] 0 2024-04-12T05:08:59.611 INFO:tasks.workunit.client.0.smithi113.stdout:8/533: dwrite d9/d1b/f61 [0,4194304] 0 2024-04-12T05:08:59.613 INFO:tasks.workunit.client.0.smithi113.stdout:7/513: dwrite d0/dc/d6/d78/f45 [0,4194304] 0 2024-04-12T05:08:59.614 INFO:tasks.workunit.client.0.smithi113.stdout:3/473: write d0/d13/d15/d28/f74 [2559778,42265] 0 2024-04-12T05:08:59.615 INFO:tasks.workunit.client.1.smithi192.stdout:0/440: fsync d0/f20 0 2024-04-12T05:08:59.615 INFO:tasks.workunit.client.0.smithi113.stdout:6/475: fsync d8/de/d13/d11/d36/f61 0 2024-04-12T05:08:59.616 INFO:tasks.workunit.client.0.smithi113.stdout:8/534: stat d9/l43 0 2024-04-12T05:08:59.624 INFO:tasks.workunit.client.0.smithi113.stdout:1/586: dwrite fd [0,4194304] 0 2024-04-12T05:08:59.625 INFO:tasks.workunit.client.0.smithi113.stdout:9/543: creat d1/df/d29/fbc x:0 0 0 2024-04-12T05:08:59.629 INFO:tasks.workunit.client.0.smithi113.stdout:0/544: getdents d1/d6/da 0 2024-04-12T05:08:59.634 INFO:tasks.workunit.client.0.smithi113.stdout:3/474: mknod d0/d13/d15/d28/ca0 0 2024-04-12T05:08:59.634 INFO:tasks.workunit.client.0.smithi113.stdout:9/544: creat d1/d4/d5/d6/fbd x:0 0 0 2024-04-12T05:08:59.637 INFO:tasks.workunit.client.0.smithi113.stdout:8/535: getdents d9/d1b/d8e/d93/d32 0 2024-04-12T05:08:59.637 INFO:tasks.workunit.client.0.smithi113.stdout:1/587: symlink d10/d14/d67/d70/lb7 0 2024-04-12T05:08:59.640 INFO:tasks.workunit.client.0.smithi113.stdout:0/545: write d1/f2 [2873060,13347] 0 2024-04-12T05:08:59.640 INFO:tasks.workunit.client.0.smithi113.stdout:5/442: dread d0/d10/d27/d1d/d38/d44/f48 [4194304,4194304] 0 2024-04-12T05:08:59.642 INFO:tasks.workunit.client.0.smithi113.stdout:6/476: dwrite d8/de/d13/d11/d36/d2c/f41 [0,4194304] 0 2024-04-12T05:08:59.642 INFO:tasks.workunit.client.0.smithi113.stdout:2/357: dwrite d8/d9/de/d15/d24/f3b [0,4194304] 0 2024-04-12T05:08:59.644 INFO:tasks.workunit.client.1.smithi192.stdout:0/441: dwrite d0/d6/d15/d19/f1d [0,4194304] 0 2024-04-12T05:08:59.646 INFO:tasks.workunit.client.0.smithi113.stdout:0/546: stat d1/d6/d16/d2a/d50/d52/fa0 0 2024-04-12T05:08:59.648 INFO:tasks.workunit.client.1.smithi192.stdout:0/442: creat d0/d6/d15/d55/d29/d89/f97 x:0 0 0 2024-04-12T05:08:59.651 INFO:tasks.workunit.client.1.smithi192.stdout:0/443: rmdir d0/d6/d15/d1b/d46/d73 39 2024-04-12T05:08:59.653 INFO:tasks.workunit.client.0.smithi113.stdout:6/477: write d8/de/d13/d11/d36/d2c/f41 [2729517,69058] 0 2024-04-12T05:08:59.654 INFO:tasks.workunit.client.1.smithi192.stdout:0/444: dread d0/d6/d15/d55/f48 [0,4194304] 0 2024-04-12T05:08:59.656 INFO:tasks.workunit.client.0.smithi113.stdout:3/475: getdents d0/d13/d15/d28/d2a/d34 0 2024-04-12T05:08:59.659 INFO:tasks.workunit.client.1.smithi192.stdout:0/445: fdatasync d0/d6/d15/d1b/d46/d73/d4b/d72/f90 0 2024-04-12T05:08:59.660 INFO:tasks.workunit.client.0.smithi113.stdout:1/588: dwrite f2 [4194304,4194304] 0 2024-04-12T05:08:59.665 INFO:tasks.workunit.client.0.smithi113.stdout:5/443: rename d0/d10/d27/d1d/d38/d44/d66/c8a to d0/d10/d27/d1d/d38/d44/d66/c94 0 2024-04-12T05:08:59.666 INFO:tasks.workunit.client.1.smithi192.stdout:0/446: rmdir d0/d6/d15/d19 39 2024-04-12T05:08:59.666 INFO:tasks.workunit.client.0.smithi113.stdout:8/536: rename d9/d16/d65/c67 to d9/d16/d7f/d3a/d6c/d90/cad 0 2024-04-12T05:08:59.667 INFO:tasks.workunit.client.1.smithi192.stdout:0/447: fsync d0/d6/d67/f8a 0 2024-04-12T05:08:59.668 INFO:tasks.workunit.client.0.smithi113.stdout:5/444: write d0/d10/d6e/f4f [1385845,2639] 0 2024-04-12T05:08:59.668 INFO:tasks.workunit.client.0.smithi113.stdout:2/358: dread d8/d9/de/d15/d38/f51 [0,4194304] 0 2024-04-12T05:08:59.669 INFO:tasks.workunit.client.1.smithi192.stdout:0/448: write d0/d6/d15/f6f [3779452,66957] 0 2024-04-12T05:08:59.670 INFO:tasks.workunit.client.0.smithi113.stdout:2/359: chown d8/d9/de/d15/d24/f3b 856417 1 2024-04-12T05:08:59.671 INFO:tasks.workunit.client.0.smithi113.stdout:2/360: truncate d8/d9/f55 664512 0 2024-04-12T05:08:59.672 INFO:tasks.workunit.client.0.smithi113.stdout:6/478: creat d8/d1d/fac x:0 0 0 2024-04-12T05:08:59.673 INFO:tasks.workunit.client.0.smithi113.stdout:6/479: truncate d8/d1d/fac 835925 0 2024-04-12T05:08:59.675 INFO:tasks.workunit.client.0.smithi113.stdout:3/476: symlink d0/d13/d47/la1 0 2024-04-12T05:08:59.677 INFO:tasks.workunit.client.0.smithi113.stdout:8/537: mkdir d9/d1b/d8e/d93/d32/d48/d8d/dae 0 2024-04-12T05:08:59.678 INFO:tasks.workunit.client.0.smithi113.stdout:3/477: write d0/d13/d15/d28/d2a/d34/f84 [614852,68134] 0 2024-04-12T05:08:59.678 INFO:tasks.workunit.client.1.smithi192.stdout:0/449: rename d0/d6/d15/d19/l81 to d0/d6/d15/d1b/l98 0 2024-04-12T05:08:59.679 INFO:tasks.workunit.client.0.smithi113.stdout:0/547: link d1/d80/f84 d1/d6/d16/fa9 0 2024-04-12T05:08:59.679 INFO:tasks.workunit.client.0.smithi113.stdout:9/545: write d1/d4/d87/fbb [1986261,51016] 0 2024-04-12T05:08:59.679 INFO:tasks.workunit.client.1.smithi192.stdout:0/450: chown d0/d6/d15/d1b/d46/d73/c3b 1071624 1 2024-04-12T05:08:59.680 INFO:tasks.workunit.client.0.smithi113.stdout:0/548: chown d1/d6/d16/fa9 17873129 1 2024-04-12T05:08:59.680 INFO:tasks.workunit.client.0.smithi113.stdout:2/361: mknod d8/d9/de/c63 0 2024-04-12T05:08:59.681 INFO:tasks.workunit.client.1.smithi192.stdout:0/451: chown d0/d6/d15/d1b/d46/d73/f60 358 1 2024-04-12T05:08:59.681 INFO:tasks.workunit.client.0.smithi113.stdout:0/549: readlink d1/d43/l88 0 2024-04-12T05:08:59.681 INFO:tasks.workunit.client.0.smithi113.stdout:1/589: truncate d10/d2e/d31/d5c/fa7 1783166 0 2024-04-12T05:08:59.682 INFO:tasks.workunit.client.0.smithi113.stdout:6/480: mknod d8/de/d13/d11/d36/cad 0 2024-04-12T05:08:59.683 INFO:tasks.workunit.client.1.smithi192.stdout:0/452: mknod d0/d6/d15/d55/d2e/d66/c99 0 2024-04-12T05:08:59.683 INFO:tasks.workunit.client.0.smithi113.stdout:5/445: mknod d0/c95 0 2024-04-12T05:08:59.685 INFO:tasks.workunit.client.0.smithi113.stdout:8/538: creat d9/d1b/d8e/d93/faf x:0 0 0 2024-04-12T05:08:59.685 INFO:tasks.workunit.client.1.smithi192.stdout:0/453: rmdir d0/d6/d15/d55/d29 39 2024-04-12T05:08:59.687 INFO:tasks.workunit.client.0.smithi113.stdout:3/478: unlink d0/d13/c3c 0 2024-04-12T05:08:59.687 INFO:tasks.workunit.client.0.smithi113.stdout:5/446: read d0/d10/d27/d1d/d38/d3b/f61 [937215,66803] 0 2024-04-12T05:08:59.688 INFO:tasks.workunit.client.1.smithi192.stdout:0/454: creat d0/d4e/d53/d6d/f9a x:0 0 0 2024-04-12T05:08:59.688 INFO:tasks.workunit.client.0.smithi113.stdout:9/546: creat d1/d27/fbe x:0 0 0 2024-04-12T05:08:59.689 INFO:tasks.workunit.client.0.smithi113.stdout:3/479: chown d0/d13/d47/l6d 835470189 1 2024-04-12T05:08:59.690 INFO:tasks.workunit.client.0.smithi113.stdout:2/362: unlink d8/d9/de/d12/f41 0 2024-04-12T05:08:59.690 INFO:tasks.workunit.client.0.smithi113.stdout:6/481: dwrite d8/de/d13/d2b/f57 [0,4194304] 0 2024-04-12T05:08:59.691 INFO:tasks.workunit.client.0.smithi113.stdout:9/547: write d1/d27/f3c [2431891,32902] 0 2024-04-12T05:08:59.691 INFO:tasks.workunit.client.1.smithi192.stdout:0/455: rename d0/d6/d15/d1b/d46/d73 to d0/d7b/d9b 0 2024-04-12T05:08:59.692 INFO:tasks.workunit.client.0.smithi113.stdout:9/548: write d1/d4/f8f [531296,12188] 0 2024-04-12T05:08:59.693 INFO:tasks.workunit.client.0.smithi113.stdout:0/550: rename d1/d6/c92 to d1/d80/caa 0 2024-04-12T05:08:59.693 INFO:tasks.workunit.client.0.smithi113.stdout:3/480: chown d0/d13/d15/d28/d2a/d34/f43 8792 1 2024-04-12T05:08:59.694 INFO:tasks.workunit.client.0.smithi113.stdout:2/363: chown d8/f1e 24952577 1 2024-04-12T05:08:59.696 INFO:tasks.workunit.client.0.smithi113.stdout:8/539: mknod d9/d16/d1c/d59/d68/cb0 0 2024-04-12T05:08:59.697 INFO:tasks.workunit.client.1.smithi192.stdout:0/456: symlink d0/d4e/d95/l9c 0 2024-04-12T05:08:59.698 INFO:tasks.workunit.client.1.smithi192.stdout:0/457: read d0/d6/d15/d55/f6c [55326,21644] 0 2024-04-12T05:08:59.698 INFO:tasks.workunit.client.0.smithi113.stdout:1/590: creat d10/d14/d67/d99/fb8 x:0 0 0 2024-04-12T05:08:59.699 INFO:tasks.workunit.client.0.smithi113.stdout:1/591: readlink d10/da5/lb4 0 2024-04-12T05:08:59.701 INFO:tasks.workunit.client.1.smithi192.stdout:0/458: chown d0/d6/d15/d19/f36 28138007 1 2024-04-12T05:08:59.701 INFO:tasks.workunit.client.0.smithi113.stdout:2/364: mknod d8/d9/de/c64 0 2024-04-12T05:08:59.702 INFO:tasks.workunit.client.1.smithi192.stdout:0/459: stat d0/d6/d15/f6f 0 2024-04-12T05:08:59.702 INFO:tasks.workunit.client.0.smithi113.stdout:8/540: dread d9/d1b/f4f [0,4194304] 0 2024-04-12T05:08:59.702 INFO:tasks.workunit.client.0.smithi113.stdout:9/549: mknod d1/d4/d5/d17/cbf 0 2024-04-12T05:08:59.704 INFO:tasks.workunit.client.0.smithi113.stdout:9/550: fsync d1/d4/d5/fb9 0 2024-04-12T05:08:59.708 INFO:tasks.workunit.client.0.smithi113.stdout:6/482: rmdir d8/de/d13/d11/d36/d3d 39 2024-04-12T05:08:59.709 INFO:tasks.workunit.client.1.smithi192.stdout:0/460: dwrite d0/d4e/d53/d6d/f9a [0,4194304] 0 2024-04-12T05:08:59.710 INFO:tasks.workunit.client.0.smithi113.stdout:3/481: mknod d0/d13/d15/ca2 0 2024-04-12T05:08:59.710 INFO:tasks.workunit.client.0.smithi113.stdout:1/592: dread d10/f9a [0,4194304] 0 2024-04-12T05:08:59.715 INFO:tasks.workunit.client.0.smithi113.stdout:1/593: fsync d10/d41/d89/d96/d7d/fb2 0 2024-04-12T05:08:59.717 INFO:tasks.workunit.client.0.smithi113.stdout:3/482: fsync d0/d13/d15/d28/d2a/f98 0 2024-04-12T05:08:59.717 INFO:tasks.workunit.client.0.smithi113.stdout:2/365: dwrite d8/d9/de/d43/f5c [0,4194304] 0 2024-04-12T05:08:59.719 INFO:tasks.workunit.client.0.smithi113.stdout:1/594: chown d10/c1d 558646260 1 2024-04-12T05:08:59.721 INFO:tasks.workunit.client.1.smithi192.stdout:9/292: sync 2024-04-12T05:08:59.721 INFO:tasks.workunit.client.1.smithi192.stdout:8/311: sync 2024-04-12T05:08:59.721 INFO:tasks.workunit.client.1.smithi192.stdout:7/266: sync 2024-04-12T05:08:59.721 INFO:tasks.workunit.client.1.smithi192.stdout:1/327: sync 2024-04-12T05:08:59.721 INFO:tasks.workunit.client.1.smithi192.stdout:5/370: sync 2024-04-12T05:08:59.721 INFO:tasks.workunit.client.1.smithi192.stdout:2/400: sync 2024-04-12T05:08:59.722 INFO:tasks.workunit.client.0.smithi113.stdout:9/551: dwrite d1/d4/f8f [0,4194304] 0 2024-04-12T05:08:59.723 INFO:tasks.workunit.client.0.smithi113.stdout:2/366: read d8/d9/f55 [514799,41948] 0 2024-04-12T05:08:59.723 INFO:tasks.workunit.client.1.smithi192.stdout:1/328: fsync d0/d2/d1e/f36 0 2024-04-12T05:08:59.724 INFO:tasks.workunit.client.1.smithi192.stdout:8/312: dread - d6/df/f4d zero size 2024-04-12T05:08:59.724 INFO:tasks.workunit.client.1.smithi192.stdout:9/293: readlink d11/d15/d21/l31 0 2024-04-12T05:08:59.725 INFO:tasks.workunit.client.0.smithi113.stdout:0/551: rename d1/c31 to d1/d6/da/cab 0 2024-04-12T05:08:59.725 INFO:tasks.workunit.client.0.smithi113.stdout:1/595: write d10/d2e/d31/d5c/f74 [2447346,50306] 0 2024-04-12T05:08:59.728 INFO:tasks.workunit.client.1.smithi192.stdout:0/461: getdents d0/d7b/d9b/d35 0 2024-04-12T05:08:59.729 INFO:tasks.workunit.client.1.smithi192.stdout:2/401: unlink d1/d8/d17/f20 0 2024-04-12T05:08:59.733 INFO:tasks.workunit.client.1.smithi192.stdout:5/371: dread d0/d15/d32/d45/f7f [0,4194304] 0 2024-04-12T05:08:59.733 INFO:tasks.workunit.client.1.smithi192.stdout:9/294: write fb [1865327,85683] 0 2024-04-12T05:08:59.734 INFO:tasks.workunit.client.1.smithi192.stdout:9/295: dread - d11/d15/f5c zero size 2024-04-12T05:08:59.736 INFO:tasks.workunit.client.1.smithi192.stdout:2/402: creat d1/d8/d3a/de/f85 x:0 0 0 2024-04-12T05:08:59.737 INFO:tasks.workunit.client.1.smithi192.stdout:1/329: link d0/d1b/f2b d0/d2/dd/d21/d39/f65 0 2024-04-12T05:08:59.738 INFO:tasks.workunit.client.1.smithi192.stdout:9/296: symlink d11/d37/d33/l64 0 2024-04-12T05:08:59.740 INFO:tasks.workunit.client.1.smithi192.stdout:5/372: symlink d0/d15/d24/d47/d7b/l83 0 2024-04-12T05:08:59.741 INFO:tasks.workunit.client.1.smithi192.stdout:9/297: fsync d11/d15/d1e/f39 0 2024-04-12T05:08:59.741 INFO:tasks.workunit.client.1.smithi192.stdout:2/403: write d1/d8/d17/d19/d60/f76 [713861,86439] 0 2024-04-12T05:08:59.741 INFO:tasks.workunit.client.0.smithi113.stdout:1/596: symlink d10/d14/d67/d70/d98/dad/lb9 0 2024-04-12T05:08:59.742 INFO:tasks.workunit.client.1.smithi192.stdout:1/330: mknod d0/d2/d1e/d3f/d56/c66 0 2024-04-12T05:08:59.744 INFO:tasks.workunit.client.0.smithi113.stdout:6/483: getdents d8/de/d13/d11/d36/d3d/d4c 0 2024-04-12T05:08:59.744 INFO:tasks.workunit.client.1.smithi192.stdout:2/404: chown d1/d8/c2b 115 1 2024-04-12T05:08:59.745 INFO:tasks.workunit.client.0.smithi113.stdout:1/597: truncate d10/d14/d67/d70/fb1 303609 0 2024-04-12T05:08:59.746 INFO:tasks.workunit.client.0.smithi113.stdout:2/367: link d8/d16/f19 d8/d9/d27/f65 0 2024-04-12T05:08:59.746 INFO:tasks.workunit.client.0.smithi113.stdout:1/598: fsync d10/d2e/f71 0 2024-04-12T05:08:59.748 INFO:tasks.workunit.client.1.smithi192.stdout:7/267: dwrite d3/d11/f2d [0,4194304] 0 2024-04-12T05:08:59.749 INFO:tasks.workunit.client.0.smithi113.stdout:8/541: write d9/d16/d3c/f74 [287247,122624] 0 2024-04-12T05:08:59.750 INFO:tasks.workunit.client.1.smithi192.stdout:1/331: write d0/d2/d1e/d3f/f52 [260139,123655] 0 2024-04-12T05:08:59.750 INFO:tasks.workunit.client.1.smithi192.stdout:9/298: creat d11/d15/d21/f65 x:0 0 0 2024-04-12T05:08:59.752 INFO:tasks.workunit.client.0.smithi113.stdout:8/542: truncate d9/d1b/d8e/d93/d32/d48/f85 640112 0 2024-04-12T05:08:59.753 INFO:tasks.workunit.client.1.smithi192.stdout:0/462: dread d0/d6/d15/d55/d29/f2b [0,4194304] 0 2024-04-12T05:08:59.753 INFO:tasks.workunit.client.1.smithi192.stdout:1/332: readlink d0/d29/l46 0 2024-04-12T05:08:59.753 INFO:tasks.workunit.client.1.smithi192.stdout:7/268: truncate d3/f4f 510744 0 2024-04-12T05:08:59.755 INFO:tasks.workunit.client.0.smithi113.stdout:8/543: dread - d9/d1b/d8e/d93/f84 zero size 2024-04-12T05:08:59.755 INFO:tasks.workunit.client.1.smithi192.stdout:2/405: rename d1/d8/c1d to d1/d8/d17/d19/d60/d59/d68/c86 0 2024-04-12T05:08:59.755 INFO:tasks.workunit.client.1.smithi192.stdout:0/463: chown d0/d6/c14 1164877 1 2024-04-12T05:08:59.756 INFO:tasks.workunit.client.1.smithi192.stdout:1/333: dread - d0/d2/d1e/d3f/d56/d58/f64 zero size 2024-04-12T05:08:59.756 INFO:tasks.workunit.client.1.smithi192.stdout:9/299: creat d11/d37/f66 x:0 0 0 2024-04-12T05:08:59.757 INFO:tasks.workunit.client.0.smithi113.stdout:6/484: rename d8/de/d13/d11/d36/d2c/f41 to d8/d45/fae 0 2024-04-12T05:08:59.757 INFO:tasks.workunit.client.0.smithi113.stdout:2/368: dread d8/d9/f55 [0,4194304] 0 2024-04-12T05:08:59.758 INFO:tasks.workunit.client.0.smithi113.stdout:8/544: chown d9/d16/d7f/d3a/c44 240179 1 2024-04-12T05:08:59.758 INFO:tasks.workunit.client.0.smithi113.stdout:2/369: fdatasync d8/d9/f30 0 2024-04-12T05:08:59.759 INFO:tasks.workunit.client.1.smithi192.stdout:2/406: chown d1/d8/d17/d19/d71 118327878 1 2024-04-12T05:08:59.761 INFO:tasks.workunit.client.1.smithi192.stdout:0/464: creat d0/d4e/d53/f9d x:0 0 0 2024-04-12T05:08:59.762 INFO:tasks.workunit.client.1.smithi192.stdout:8/313: dread d6/da/fd [0,4194304] 0 2024-04-12T05:08:59.762 INFO:tasks.workunit.client.1.smithi192.stdout:1/334: rename d0/d2/cb to d0/d2/c67 0 2024-04-12T05:08:59.763 INFO:tasks.workunit.client.0.smithi113.stdout:2/370: mknod d8/d9/de/d12/d2b/c66 0 2024-04-12T05:08:59.764 INFO:tasks.workunit.client.1.smithi192.stdout:2/407: mknod d1/c87 0 2024-04-12T05:08:59.764 INFO:tasks.workunit.client.1.smithi192.stdout:0/465: read d0/d6/d15/d19/f4c [2153024,43607] 0 2024-04-12T05:08:59.765 INFO:tasks.workunit.client.0.smithi113.stdout:8/545: rename d9/d16/d1c to d9/d16/d65/db1 0 2024-04-12T05:08:59.765 INFO:tasks.workunit.client.0.smithi113.stdout:3/483: dread d0/f6 [4194304,4194304] 0 2024-04-12T05:08:59.766 INFO:tasks.workunit.client.0.smithi113.stdout:3/484: fdatasync d0/f1c 0 2024-04-12T05:08:59.767 INFO:tasks.workunit.client.0.smithi113.stdout:6/485: creat d8/faf x:0 0 0 2024-04-12T05:08:59.767 INFO:tasks.workunit.client.0.smithi113.stdout:1/599: dread d10/d55/f8b [0,4194304] 0 2024-04-12T05:08:59.768 INFO:tasks.workunit.client.1.smithi192.stdout:2/408: write d1/d2d/f32 [4325411,69171] 0 2024-04-12T05:08:59.769 INFO:tasks.workunit.client.1.smithi192.stdout:7/269: dread d3/d1d/f2e [4194304,4194304] 0 2024-04-12T05:08:59.770 INFO:tasks.workunit.client.1.smithi192.stdout:8/314: unlink d6/d10/f3d 0 2024-04-12T05:08:59.775 INFO:tasks.workunit.client.1.smithi192.stdout:2/409: mknod d1/c88 0 2024-04-12T05:08:59.776 INFO:tasks.workunit.client.1.smithi192.stdout:8/315: readlink d6/da/l2b 0 2024-04-12T05:08:59.778 INFO:tasks.workunit.client.0.smithi113.stdout:6/486: write d8/de/d13/d11/d36/d2c/fa3 [608412,37610] 0 2024-04-12T05:08:59.779 INFO:tasks.workunit.client.1.smithi192.stdout:9/300: dwrite d11/d15/f42 [0,4194304] 0 2024-04-12T05:08:59.779 INFO:tasks.workunit.client.0.smithi113.stdout:1/600: symlink d10/d2e/d31/d33/lba 0 2024-04-12T05:08:59.779 INFO:tasks.workunit.client.1.smithi192.stdout:7/270: dread d3/d11/f16 [0,4194304] 0 2024-04-12T05:08:59.782 INFO:tasks.workunit.client.0.smithi113.stdout:3/485: mkdir d0/d7e/da3 0 2024-04-12T05:08:59.783 INFO:tasks.workunit.client.1.smithi192.stdout:7/271: chown d3/d1d/d5a 106943 1 2024-04-12T05:08:59.783 INFO:tasks.workunit.client.0.smithi113.stdout:2/371: dread d8/d9/de/d15/f18 [0,4194304] 0 2024-04-12T05:08:59.784 INFO:tasks.workunit.client.1.smithi192.stdout:1/335: dwrite d0/d2/dd/f5d [0,4194304] 0 2024-04-12T05:08:59.784 INFO:tasks.workunit.client.1.smithi192.stdout:0/466: rename d0/d6/d15/d1b/d3a/d4f/f6a to d0/d6/d15/d55/d2e/f9e 0 2024-04-12T05:08:59.786 INFO:tasks.workunit.client.0.smithi113.stdout:3/486: read d0/d13/d47/f4a [46544,87248] 0 2024-04-12T05:08:59.787 INFO:tasks.workunit.client.0.smithi113.stdout:6/487: creat d8/de/d13/d11/d36/d3d/d4c/d26/d64/fb0 x:0 0 0 2024-04-12T05:08:59.790 INFO:tasks.workunit.client.1.smithi192.stdout:8/316: write d6/df/d37/d4f/d4c/f55 [3109665,49388] 0 2024-04-12T05:08:59.790 INFO:tasks.workunit.client.0.smithi113.stdout:8/546: getdents d9/d16/d65/db1/d59/d68 0 2024-04-12T05:08:59.790 INFO:tasks.workunit.client.0.smithi113.stdout:8/547: dread - d9/d1b/d8e/d93/faf zero size 2024-04-12T05:08:59.792 INFO:tasks.workunit.client.1.smithi192.stdout:8/317: chown d6/df/d37/d4f/d4c/d54 668 1 2024-04-12T05:08:59.792 INFO:tasks.workunit.client.1.smithi192.stdout:0/467: mkdir d0/d4e/d9f 0 2024-04-12T05:08:59.792 INFO:tasks.workunit.client.0.smithi113.stdout:8/548: dread - d9/d1b/d8e/d93/d32/d48/d8d/f73 zero size 2024-04-12T05:08:59.793 INFO:tasks.workunit.client.0.smithi113.stdout:2/372: creat d8/d9/de/f67 x:0 0 0 2024-04-12T05:08:59.794 INFO:tasks.workunit.client.1.smithi192.stdout:1/336: fsync d0/d2/dd/d21/d33/f3a 0 2024-04-12T05:08:59.801 INFO:tasks.workunit.client.1.smithi192.stdout:0/468: mknod d0/d6/d15/d1b/d3a/d4f/ca0 0 2024-04-12T05:08:59.804 INFO:tasks.workunit.client.1.smithi192.stdout:7/272: dread d3/d7/d15/f3c [0,4194304] 0 2024-04-12T05:08:59.805 INFO:tasks.workunit.client.1.smithi192.stdout:0/469: unlink d0/f7d 0 2024-04-12T05:08:59.806 INFO:tasks.workunit.client.0.smithi113.stdout:1/601: write d10/d41/d89/d96/d7d/f7f [2798772,111892] 0 2024-04-12T05:08:59.807 INFO:tasks.workunit.client.1.smithi192.stdout:1/337: rename d0/d2/d1e/d3f/d56/d58/f64 to d0/d2/dd/d21/d33/f68 0 2024-04-12T05:08:59.807 INFO:tasks.workunit.client.0.smithi113.stdout:6/488: rename d8/de/d13/d11/d36/d3d/ca4 to d8/d1d/cb1 0 2024-04-12T05:08:59.809 INFO:tasks.workunit.client.0.smithi113.stdout:2/373: symlink d8/d9/l68 0 2024-04-12T05:08:59.809 INFO:tasks.workunit.client.0.smithi113.stdout:8/549: symlink d9/d16/d7f/d3a/d6c/d90/dac/lb2 0 2024-04-12T05:08:59.810 INFO:tasks.workunit.client.0.smithi113.stdout:3/487: link d0/d7e/c87 d0/d80/ca4 0 2024-04-12T05:08:59.812 INFO:tasks.workunit.client.1.smithi192.stdout:1/338: dread d0/d1b/f2b [0,4194304] 0 2024-04-12T05:08:59.812 INFO:tasks.workunit.client.0.smithi113.stdout:1/602: mknod d10/d14/cbb 0 2024-04-12T05:08:59.813 INFO:tasks.workunit.client.1.smithi192.stdout:1/339: read d0/d2/dd/f1d [1019809,61366] 0 2024-04-12T05:08:59.815 INFO:tasks.workunit.client.0.smithi113.stdout:1/603: fsync d10/d14/d67/d70/d98/f9e 0 2024-04-12T05:08:59.817 INFO:tasks.workunit.client.1.smithi192.stdout:9/301: truncate d11/d15/d21/f2d 3763279 0 2024-04-12T05:08:59.818 INFO:tasks.workunit.client.0.smithi113.stdout:1/604: stat d10/d14/d67/d70/d98/dad/lb9 0 2024-04-12T05:08:59.819 INFO:tasks.workunit.client.0.smithi113.stdout:8/550: dwrite d9/d16/d3c/f52 [0,4194304] 0 2024-04-12T05:08:59.820 INFO:tasks.workunit.client.1.smithi192.stdout:2/410: dwrite d1/d8/d17/f24 [0,4194304] 0 2024-04-12T05:08:59.822 INFO:tasks.workunit.client.1.smithi192.stdout:7/273: getdents d3/d7/d15/d22 0 2024-04-12T05:08:59.823 INFO:tasks.workunit.client.1.smithi192.stdout:7/274: dread - d3/d11/f37 zero size 2024-04-12T05:08:59.824 INFO:tasks.workunit.client.0.smithi113.stdout:2/374: rename d8/d9/d1a/f49 to d8/d9/f69 0 2024-04-12T05:08:59.827 INFO:tasks.workunit.client.0.smithi113.stdout:6/489: truncate d8/de/d13/d11/d36/f4e 6639921 0 2024-04-12T05:08:59.830 INFO:tasks.workunit.client.1.smithi192.stdout:9/302: creat d11/d15/d21/d2c/d1f/f67 x:0 0 0 2024-04-12T05:08:59.830 INFO:tasks.workunit.client.1.smithi192.stdout:0/470: getdents d0/d7b/d9b 0 2024-04-12T05:08:59.830 INFO:tasks.workunit.client.1.smithi192.stdout:7/275: rmdir d3/d1d/d5a 39 2024-04-12T05:08:59.831 INFO:tasks.workunit.client.1.smithi192.stdout:9/303: chown f10 326 1 2024-04-12T05:08:59.832 INFO:tasks.workunit.client.0.smithi113.stdout:2/375: rename d8/d9/de/c2f to d8/d9/de/d43/c6a 0 2024-04-12T05:08:59.832 INFO:tasks.workunit.client.1.smithi192.stdout:9/304: dread - d11/d37/f56 zero size 2024-04-12T05:08:59.833 INFO:tasks.workunit.client.0.smithi113.stdout:3/488: creat d0/d13/d15/d28/fa5 x:0 0 0 2024-04-12T05:08:59.835 INFO:tasks.workunit.client.0.smithi113.stdout:1/605: creat d10/d14/fbc x:0 0 0 2024-04-12T05:08:59.836 INFO:tasks.workunit.client.0.smithi113.stdout:2/376: mknod d8/d9/d5b/c6b 0 2024-04-12T05:08:59.838 INFO:tasks.workunit.client.0.smithi113.stdout:2/377: rmdir d8 39 2024-04-12T05:08:59.840 INFO:tasks.workunit.client.1.smithi192.stdout:1/340: dwrite d0/d2/dd/f32 [0,4194304] 0 2024-04-12T05:08:59.841 INFO:tasks.workunit.client.0.smithi113.stdout:1/606: fsync d10/d2e/d31/d5c/fa7 0 2024-04-12T05:08:59.842 INFO:tasks.workunit.client.1.smithi192.stdout:7/276: dwrite d3/d7/d15/f3c [0,4194304] 0 2024-04-12T05:08:59.844 INFO:tasks.workunit.client.0.smithi113.stdout:1/607: write d10/d14/fbc [839331,71299] 0 2024-04-12T05:08:59.844 INFO:tasks.workunit.client.1.smithi192.stdout:2/411: write d1/d8/d3a/de/f57 [924857,41412] 0 2024-04-12T05:08:59.844 INFO:tasks.workunit.client.1.smithi192.stdout:0/471: link d0/c18 d0/d6/d15/d55/d29/ca1 0 2024-04-12T05:08:59.848 INFO:tasks.workunit.client.1.smithi192.stdout:7/277: creat d3/d1d/d4a/f61 x:0 0 0 2024-04-12T05:08:59.848 INFO:tasks.workunit.client.1.smithi192.stdout:9/305: write f6 [4007294,35917] 0 2024-04-12T05:08:59.849 INFO:tasks.workunit.client.1.smithi192.stdout:1/341: mkdir d0/d1b/d44/d69 0 2024-04-12T05:08:59.850 INFO:tasks.workunit.client.1.smithi192.stdout:2/412: mknod d1/d8/d3a/de/d7c/c89 0 2024-04-12T05:08:59.851 INFO:tasks.workunit.client.1.smithi192.stdout:1/342: read d0/d2/dd/d21/f2d [889664,100403] 0 2024-04-12T05:08:59.853 INFO:tasks.workunit.client.1.smithi192.stdout:9/306: write d11/d25/d36/d3d/f5e [4835974,32572] 0 2024-04-12T05:08:59.854 INFO:tasks.workunit.client.1.smithi192.stdout:9/307: write fb [3534778,127826] 0 2024-04-12T05:08:59.856 INFO:tasks.workunit.client.0.smithi113.stdout:2/378: mknod d8/d9/de/d12/d2b/c6c 0 2024-04-12T05:08:59.856 INFO:tasks.workunit.client.0.smithi113.stdout:2/379: readlink d8/d9/l68 0 2024-04-12T05:08:59.856 INFO:tasks.workunit.client.0.smithi113.stdout:1/608: write d10/d2e/f54 [3170937,16371] 0 2024-04-12T05:08:59.856 INFO:tasks.workunit.client.1.smithi192.stdout:7/278: mknod d3/d7/d15/c62 0 2024-04-12T05:08:59.856 INFO:tasks.workunit.client.1.smithi192.stdout:2/413: write d1/d8/d17/d19/d40/f52 [3896950,29575] 0 2024-04-12T05:08:59.857 INFO:tasks.workunit.client.0.smithi113.stdout:6/490: truncate d8/f1a 3219083 0 2024-04-12T05:08:59.858 INFO:tasks.workunit.client.1.smithi192.stdout:2/414: write d1/d8/d3a/de/f57 [1077152,111501] 0 2024-04-12T05:08:59.863 INFO:tasks.workunit.client.1.smithi192.stdout:2/415: symlink d1/d8/d17/d19/d60/d59/d68/l8a 0 2024-04-12T05:08:59.863 INFO:tasks.workunit.client.0.smithi113.stdout:8/551: dwrite d9/d16/d65/db1/f2d [0,4194304] 0 2024-04-12T05:08:59.863 INFO:tasks.workunit.client.1.smithi192.stdout:0/472: link d0/d6/d15/d1b/l77 d0/d6/la2 0 2024-04-12T05:08:59.864 INFO:tasks.workunit.client.0.smithi113.stdout:8/552: stat d9/d16/d7f/l2b 0 2024-04-12T05:08:59.865 INFO:tasks.workunit.client.1.smithi192.stdout:1/343: creat d0/d1b/f6a x:0 0 0 2024-04-12T05:08:59.866 INFO:tasks.workunit.client.0.smithi113.stdout:1/609: fsync d10/d41/d5b/fa3 0 2024-04-12T05:08:59.866 INFO:tasks.workunit.client.0.smithi113.stdout:2/380: link d8/d9/de/f22 d8/d9/d27/f6d 0 2024-04-12T05:08:59.867 INFO:tasks.workunit.client.0.smithi113.stdout:8/553: write d9/d1b/d8e/d93/d32/d48/d8d/fa0 [369932,8606] 0 2024-04-12T05:08:59.868 INFO:tasks.workunit.client.0.smithi113.stdout:3/489: dread d0/d13/f6a [0,4194304] 0 2024-04-12T05:08:59.870 INFO:tasks.workunit.client.0.smithi113.stdout:6/491: dread d8/de/d13/d11/d36/f23 [0,4194304] 0 2024-04-12T05:08:59.871 INFO:tasks.workunit.client.1.smithi192.stdout:0/473: dwrite d0/d6/d67/f92 [0,4194304] 0 2024-04-12T05:08:59.872 INFO:tasks.workunit.client.0.smithi113.stdout:6/492: fsync d8/de/d13/d11/d36/d3d/d4c/d26/f51 0 2024-04-12T05:08:59.875 INFO:tasks.workunit.client.1.smithi192.stdout:1/344: dread d0/d2/dd/d21/d33/f3a [4194304,4194304] 0 2024-04-12T05:08:59.875 INFO:tasks.workunit.client.0.smithi113.stdout:2/381: unlink d8/d9/de/f42 0 2024-04-12T05:08:59.876 INFO:tasks.workunit.client.0.smithi113.stdout:2/382: read - d8/d9/d1a/f26 zero size 2024-04-12T05:08:59.877 INFO:tasks.workunit.client.0.smithi113.stdout:1/610: creat d10/d41/d89/d96/d7d/d9f/fbd x:0 0 0 2024-04-12T05:08:59.877 INFO:tasks.workunit.client.1.smithi192.stdout:0/474: creat d0/d6/fa3 x:0 0 0 2024-04-12T05:08:59.880 INFO:tasks.workunit.client.1.smithi192.stdout:0/475: chown d0/d6/d15/d1b/d3a/d4f/c8e 628595 1 2024-04-12T05:08:59.881 INFO:tasks.workunit.client.0.smithi113.stdout:6/493: creat d8/de/d13/d11/d36/d3d/d4c/d26/d5b/fb2 x:0 0 0 2024-04-12T05:08:59.882 INFO:tasks.workunit.client.0.smithi113.stdout:2/383: creat d8/d16/f6e x:0 0 0 2024-04-12T05:08:59.883 INFO:tasks.workunit.client.1.smithi192.stdout:0/476: fdatasync d0/d6/d15/d1b/d46/f4d 0 2024-04-12T05:08:59.883 INFO:tasks.workunit.client.1.smithi192.stdout:1/345: dwrite d0/d1b/f53 [0,4194304] 0 2024-04-12T05:08:59.884 INFO:tasks.workunit.client.1.smithi192.stdout:1/346: truncate d0/f63 558615 0 2024-04-12T05:08:59.885 INFO:tasks.workunit.client.1.smithi192.stdout:1/347: dread - d0/d1b/d28/f61 zero size 2024-04-12T05:08:59.887 INFO:tasks.workunit.client.0.smithi113.stdout:1/611: symlink d10/d41/d89/d96/d7d/lbe 0 2024-04-12T05:08:59.887 INFO:tasks.workunit.client.0.smithi113.stdout:6/494: write d8/de/d13/d11/d36/d2c/fa3 [409242,57975] 0 2024-04-12T05:08:59.888 INFO:tasks.workunit.client.0.smithi113.stdout:8/554: dwrite d9/d16/d65/db1/f8b [0,4194304] 0 2024-04-12T05:08:59.890 INFO:tasks.workunit.client.0.smithi113.stdout:2/384: truncate d8/d9/de/d15/f40 1765292 0 2024-04-12T05:08:59.890 INFO:tasks.workunit.client.0.smithi113.stdout:8/555: dread - d9/d16/d65/f97 zero size 2024-04-12T05:08:59.891 INFO:tasks.workunit.client.1.smithi192.stdout:1/348: dread d0/d1b/f35 [0,4194304] 0 2024-04-12T05:08:59.892 INFO:tasks.workunit.client.0.smithi113.stdout:8/556: fdatasync d9/d1b/d8e/d93/f66 0 2024-04-12T05:08:59.892 INFO:tasks.workunit.client.0.smithi113.stdout:6/495: mkdir d8/de/d13/d2b/db3 0 2024-04-12T05:08:59.892 INFO:tasks.workunit.client.0.smithi113.stdout:1/612: write d10/f13 [1536233,121428] 0 2024-04-12T05:08:59.895 INFO:tasks.workunit.client.0.smithi113.stdout:6/496: chown d8/de/d13/d11/d36/l20 52097101 1 2024-04-12T05:08:59.897 INFO:tasks.workunit.client.1.smithi192.stdout:0/477: dwrite d0/d6/d15/f6f [4194304,4194304] 0 2024-04-12T05:08:59.898 INFO:tasks.workunit.client.0.smithi113.stdout:6/497: truncate d8/de/d13/d11/d36/d2c/f5c 17263 0 2024-04-12T05:08:59.899 INFO:tasks.workunit.client.0.smithi113.stdout:2/385: rename d8/d9/de/d12/d2b/c66 to d8/d9/de/d15/d24/c6f 0 2024-04-12T05:08:59.899 INFO:tasks.workunit.client.0.smithi113.stdout:3/490: read d0/f9 [4057270,102617] 0 2024-04-12T05:08:59.900 INFO:tasks.workunit.client.0.smithi113.stdout:8/557: rename d9/d1b/d8e/fab to d9/d1b/d8e/fb3 0 2024-04-12T05:08:59.903 INFO:tasks.workunit.client.0.smithi113.stdout:1/613: symlink d10/d41/lbf 0 2024-04-12T05:08:59.904 INFO:tasks.workunit.client.1.smithi192.stdout:0/478: mkdir d0/d6/d15/d1b/da4 0 2024-04-12T05:08:59.905 INFO:tasks.workunit.client.0.smithi113.stdout:2/386: dread d8/d9/d27/f6d [0,4194304] 0 2024-04-12T05:08:59.907 INFO:tasks.workunit.client.0.smithi113.stdout:6/498: symlink d8/de/d13/d11/d36/d3d/d4c/d31/d8e/d63/d66/lb4 0 2024-04-12T05:08:59.909 INFO:tasks.workunit.client.1.smithi192.stdout:1/349: creat d0/d2/dd/d21/d33/f6b x:0 0 0 2024-04-12T05:08:59.911 INFO:tasks.workunit.client.0.smithi113.stdout:8/558: dwrite d9/d1b/d96/f9b [0,4194304] 0 2024-04-12T05:08:59.912 INFO:tasks.workunit.client.1.smithi192.stdout:0/479: write d0/d6/d15/d55/d2e/f63 [3142677,105909] 0 2024-04-12T05:08:59.912 INFO:tasks.workunit.client.1.smithi192.stdout:7/279: dread d3/d7/f13 [0,4194304] 0 2024-04-12T05:08:59.914 INFO:tasks.workunit.client.1.smithi192.stdout:1/350: mknod d0/d2/c6c 0 2024-04-12T05:08:59.916 INFO:tasks.workunit.client.0.smithi113.stdout:8/559: dread - d9/d16/d65/f97 zero size 2024-04-12T05:08:59.916 INFO:tasks.workunit.client.0.smithi113.stdout:3/491: truncate d0/f18 2345300 0 2024-04-12T05:08:59.921 INFO:tasks.workunit.client.0.smithi113.stdout:6/499: dwrite d8/faf [0,4194304] 0 2024-04-12T05:08:59.924 INFO:tasks.workunit.client.0.smithi113.stdout:2/387: truncate d8/d16/f17 3104683 0 2024-04-12T05:08:59.927 INFO:tasks.workunit.client.0.smithi113.stdout:2/388: write d8/f1e [274947,25033] 0 2024-04-12T05:08:59.927 INFO:tasks.workunit.client.1.smithi192.stdout:0/480: dwrite d0/d6/d15/d19/f71 [0,4194304] 0 2024-04-12T05:08:59.927 INFO:tasks.workunit.client.1.smithi192.stdout:1/351: rename d0/l5c to d0/d2/dd/d21/l6d 0 2024-04-12T05:08:59.928 INFO:tasks.workunit.client.0.smithi113.stdout:1/614: dread d10/f13 [0,4194304] 0 2024-04-12T05:08:59.928 INFO:tasks.workunit.client.1.smithi192.stdout:7/280: dwrite d3/f21 [0,4194304] 0 2024-04-12T05:08:59.931 INFO:tasks.workunit.client.0.smithi113.stdout:8/560: unlink d9/d1b/d8e/d93/fa2 0 2024-04-12T05:08:59.931 INFO:tasks.workunit.client.1.smithi192.stdout:7/281: fdatasync d3/d7/d15/f24 0 2024-04-12T05:08:59.931 INFO:tasks.workunit.client.0.smithi113.stdout:8/561: chown d9/d1b/d8e/d93/f66 223 1 2024-04-12T05:08:59.934 INFO:tasks.workunit.client.0.smithi113.stdout:8/562: write d9/d1b/d8e/d93/d32/d48/d8d/fa0 [276473,37799] 0 2024-04-12T05:08:59.935 INFO:tasks.workunit.client.0.smithi113.stdout:6/500: mknod d8/cb5 0 2024-04-12T05:08:59.935 INFO:tasks.workunit.client.1.smithi192.stdout:0/481: unlink d0/d6/l75 0 2024-04-12T05:08:59.936 INFO:tasks.workunit.client.1.smithi192.stdout:7/282: mknod d3/d1d/d4a/c63 0 2024-04-12T05:08:59.940 INFO:tasks.workunit.client.0.smithi113.stdout:2/389: write d8/d9/de/d15/f18 [3427243,64741] 0 2024-04-12T05:08:59.942 INFO:tasks.workunit.client.1.smithi192.stdout:0/482: fsync d0/d6/d15/d55/f6c 0 2024-04-12T05:08:59.942 INFO:tasks.workunit.client.1.smithi192.stdout:7/283: dread d3/d7/d15/f50 [0,4194304] 0 2024-04-12T05:08:59.949 INFO:tasks.workunit.client.1.smithi192.stdout:7/284: link d3/d11/f3b d3/d1d/d5a/d5f/f64 0 2024-04-12T05:08:59.949 INFO:tasks.workunit.client.0.smithi113.stdout:1/615: mknod d10/d41/d89/d96/d7d/d9f/cc0 0 2024-04-12T05:08:59.950 INFO:tasks.workunit.client.0.smithi113.stdout:8/563: mkdir d9/d16/d65/db4 0 2024-04-12T05:08:59.950 INFO:tasks.workunit.client.0.smithi113.stdout:3/492: creat d0/fa6 x:0 0 0 2024-04-12T05:08:59.950 INFO:tasks.workunit.client.0.smithi113.stdout:1/616: chown d10/d14/fbc 15151 1 2024-04-12T05:08:59.951 INFO:tasks.workunit.client.0.smithi113.stdout:1/617: stat d10/d41/d89/d96/d7d/f79 0 2024-04-12T05:08:59.954 INFO:tasks.workunit.client.0.smithi113.stdout:2/390: rename d8/d9/de/d15/l4d to d8/d9/de/d15/d38/l70 0 2024-04-12T05:08:59.955 INFO:tasks.workunit.client.1.smithi192.stdout:7/285: mkdir d3/d7/d60/d65 0 2024-04-12T05:08:59.956 INFO:tasks.workunit.client.1.smithi192.stdout:7/286: fdatasync d3/d7/d14/f3d 0 2024-04-12T05:08:59.957 INFO:tasks.workunit.client.0.smithi113.stdout:6/501: creat d8/de/d13/d2b/db3/fb6 x:0 0 0 2024-04-12T05:08:59.958 INFO:tasks.workunit.client.0.smithi113.stdout:3/493: mknod d0/ca7 0 2024-04-12T05:08:59.959 INFO:tasks.workunit.client.1.smithi192.stdout:7/287: creat d3/d7/d60/f66 x:0 0 0 2024-04-12T05:08:59.959 INFO:tasks.workunit.client.0.smithi113.stdout:1/618: mknod d10/d2e/d31/d5c/daf/cc1 0 2024-04-12T05:08:59.960 INFO:tasks.workunit.client.1.smithi192.stdout:7/288: write d3/d1d/d5a/d5f/f30 [1486690,124556] 0 2024-04-12T05:08:59.960 INFO:tasks.workunit.client.0.smithi113.stdout:3/494: fdatasync d0/d13/d47/f4a 0 2024-04-12T05:08:59.961 INFO:tasks.workunit.client.0.smithi113.stdout:8/564: getdents d9/d16/d7f/d3a/d6c/d90 0 2024-04-12T05:08:59.962 INFO:tasks.workunit.client.1.smithi192.stdout:7/289: readlink d3/d7/l52 0 2024-04-12T05:08:59.962 INFO:tasks.workunit.client.1.smithi192.stdout:7/290: stat d3/d1d/d4a/c63 0 2024-04-12T05:08:59.963 INFO:tasks.workunit.client.0.smithi113.stdout:8/565: fdatasync d9/d1b/d8e/d93/d32/f60 0 2024-04-12T05:08:59.966 INFO:tasks.workunit.client.1.smithi192.stdout:7/291: creat d3/d7/d15/f67 x:0 0 0 2024-04-12T05:08:59.966 INFO:tasks.workunit.client.0.smithi113.stdout:8/566: creat d9/d16/d65/d8c/fb5 x:0 0 0 2024-04-12T05:08:59.967 INFO:tasks.workunit.client.0.smithi113.stdout:3/495: creat d0/fa8 x:0 0 0 2024-04-12T05:08:59.968 INFO:tasks.workunit.client.1.smithi192.stdout:1/352: truncate d0/d1b/f53 808572 0 2024-04-12T05:08:59.968 INFO:tasks.workunit.client.0.smithi113.stdout:6/502: getdents d8/de/d13/d11/d36/d3d/d4c/d31/d8e/d63 0 2024-04-12T05:08:59.969 INFO:tasks.workunit.client.0.smithi113.stdout:8/567: chown d9/d16/d65/db1/d59/d68/caa 2 1 2024-04-12T05:08:59.970 INFO:tasks.workunit.client.1.smithi192.stdout:0/483: write d0/d6/d15/d19/f4c [1154935,49135] 0 2024-04-12T05:08:59.973 INFO:tasks.workunit.client.0.smithi113.stdout:3/496: dread d0/d13/d15/d28/d2a/f36 [0,4194304] 0 2024-04-12T05:08:59.974 INFO:tasks.workunit.client.0.smithi113.stdout:6/503: getdents d8/de/d13/d11/d36/d3d/d4c/d31/d83 0 2024-04-12T05:08:59.975 INFO:tasks.workunit.client.1.smithi192.stdout:0/484: unlink d0/c18 0 2024-04-12T05:08:59.975 INFO:tasks.workunit.client.0.smithi113.stdout:3/497: read d0/d13/d15/d28/f6f [431097,122667] 0 2024-04-12T05:08:59.976 INFO:tasks.workunit.client.1.smithi192.stdout:7/292: dwrite d3/d1d/f2e [0,4194304] 0 2024-04-12T05:08:59.978 INFO:tasks.workunit.client.1.smithi192.stdout:7/293: write d3/d11/f16 [158742,27144] 0 2024-04-12T05:08:59.981 INFO:tasks.workunit.client.1.smithi192.stdout:0/485: creat d0/d4e/d9f/fa5 x:0 0 0 2024-04-12T05:08:59.982 INFO:tasks.workunit.client.1.smithi192.stdout:0/486: write d0/d6/d15/d55/d2e/d66/f82 [870685,28830] 0 2024-04-12T05:08:59.984 INFO:tasks.workunit.client.1.smithi192.stdout:1/353: truncate d0/d2/f34 465110 0 2024-04-12T05:08:59.985 INFO:tasks.workunit.client.1.smithi192.stdout:7/294: write d3/d7/d14/d1f/f34 [881998,23244] 0 2024-04-12T05:08:59.987 INFO:tasks.workunit.client.0.smithi113.stdout:6/504: getdents d8/de/d13/d11/d36/d3d/d4c/d26/d5b 0 2024-04-12T05:08:59.988 INFO:tasks.workunit.client.1.smithi192.stdout:1/354: creat d0/d1b/d44/f6e x:0 0 0 2024-04-12T05:08:59.988 INFO:tasks.workunit.client.1.smithi192.stdout:7/295: mkdir d3/d11/d68 0 2024-04-12T05:08:59.990 INFO:tasks.workunit.client.0.smithi113.stdout:3/498: write d0/fa [1610145,48346] 0 2024-04-12T05:08:59.991 INFO:tasks.workunit.client.1.smithi192.stdout:1/355: creat d0/d1b/f6f x:0 0 0 2024-04-12T05:08:59.992 INFO:tasks.workunit.client.1.smithi192.stdout:1/356: write d0/d2/dd/d21/d33/f6b [605262,23388] 0 2024-04-12T05:08:59.992 INFO:tasks.workunit.client.1.smithi192.stdout:7/296: creat d3/d1d/f69 x:0 0 0 2024-04-12T05:08:59.993 INFO:tasks.workunit.client.1.smithi192.stdout:7/297: chown d3/d11/l35 129331 1 2024-04-12T05:08:59.994 INFO:tasks.workunit.client.1.smithi192.stdout:1/357: write d0/d2/dd/d3c/f3e [892363,94444] 0 2024-04-12T05:08:59.997 INFO:tasks.workunit.client.1.smithi192.stdout:1/358: chown d0/d1b/d44 16319932 1 2024-04-12T05:08:59.998 INFO:tasks.workunit.client.0.smithi113.stdout:3/499: dwrite d0/d13/d15/d28/d2a/d34/d5f/f65 [0,4194304] 0 2024-04-12T05:08:59.999 INFO:tasks.workunit.client.0.smithi113.stdout:3/500: chown d0/d13/f6a 2116011 1 2024-04-12T05:08:59.999 INFO:tasks.workunit.client.1.smithi192.stdout:1/359: truncate d0/d2/dd/d21/f2d 1623002 0 2024-04-12T05:09:00.002 INFO:tasks.workunit.client.1.smithi192.stdout:7/298: dwrite d3/d7/f19 [4194304,4194304] 0 2024-04-12T05:09:00.003 INFO:tasks.workunit.client.1.smithi192.stdout:1/360: mknod d0/c70 0 2024-04-12T05:09:00.017 INFO:tasks.workunit.client.0.smithi113.stdout:6/505: getdents d8/de/d13/d11/d36/d3d/d4c 0 2024-04-12T05:09:00.019 INFO:tasks.workunit.client.0.smithi113.stdout:6/506: chown d8/de/d13/d11/d36/d3d/d4c/d31/d8e/f9b 31452762 1 2024-04-12T05:09:00.020 INFO:tasks.workunit.client.0.smithi113.stdout:3/501: rename d0/d13/d15/f5c to d0/d13/fa9 0 2024-04-12T05:09:00.027 INFO:tasks.workunit.client.1.smithi192.stdout:7/299: mkdir d3/d1d/d5a/d5f/d6a 0 2024-04-12T05:09:00.027 INFO:tasks.workunit.client.1.smithi192.stdout:7/300: chown d3/d11/f2d 627933212 1 2024-04-12T05:09:00.051 INFO:tasks.workunit.client.1.smithi192.stdout:7/301: mknod d3/d1d/d5a/d5f/c6b 0 2024-04-12T05:09:00.051 INFO:tasks.workunit.client.1.smithi192.stdout:7/302: dread - d3/d1d/d4a/f61 zero size 2024-04-12T05:09:00.052 INFO:tasks.workunit.client.1.smithi192.stdout:7/303: read - d3/d7/f56 zero size 2024-04-12T05:09:00.059 INFO:tasks.workunit.client.1.smithi192.stdout:7/304: fdatasync d3/d1d/d5a/d5f/f64 0 2024-04-12T05:09:00.061 INFO:tasks.workunit.client.1.smithi192.stdout:1/361: truncate d0/f18 1289179 0 2024-04-12T05:09:00.063 INFO:tasks.workunit.client.1.smithi192.stdout:1/362: write d0/d2/dd/d3c/f3e [889838,69903] 0 2024-04-12T05:09:00.064 INFO:tasks.workunit.client.1.smithi192.stdout:1/363: fdatasync d0/d2/dd/f5d 0 2024-04-12T05:09:00.091 INFO:tasks.workunit.client.0.smithi113.stdout:7/514: sync 2024-04-12T05:09:00.091 INFO:tasks.workunit.client.0.smithi113.stdout:4/498: sync 2024-04-12T05:09:00.097 INFO:tasks.workunit.client.0.smithi113.stdout:4/499: dwrite da/dd/d3a/f5b [0,4194304] 0 2024-04-12T05:09:00.108 INFO:tasks.workunit.client.0.smithi113.stdout:7/515: chown d0/dc/d6/d78/f31 28927 1 2024-04-12T05:09:00.109 INFO:tasks.workunit.client.0.smithi113.stdout:7/516: write d0/dc/d6/d78/f6a [379809,60166] 0 2024-04-12T05:09:00.114 INFO:tasks.workunit.client.0.smithi113.stdout:4/500: getdents da/dd/d3a 0 2024-04-12T05:09:00.115 INFO:tasks.workunit.client.0.smithi113.stdout:4/501: write da/d3c/d83/f8a [1037134,30694] 0 2024-04-12T05:09:00.118 INFO:tasks.workunit.client.1.smithi192.stdout:3/318: sync 2024-04-12T05:09:00.118 INFO:tasks.workunit.client.1.smithi192.stdout:6/445: sync 2024-04-12T05:09:00.118 INFO:tasks.workunit.client.1.smithi192.stdout:4/452: sync 2024-04-12T05:09:00.120 INFO:tasks.workunit.client.1.smithi192.stdout:4/453: dread - d1/d54/d55/d64/f7d zero size 2024-04-12T05:09:00.157 INFO:tasks.workunit.client.0.smithi113.stdout:7/517: creat d0/faf x:0 0 0 2024-04-12T05:09:00.158 INFO:tasks.workunit.client.0.smithi113.stdout:4/502: mkdir da/d4a/da2 0 2024-04-12T05:09:00.163 INFO:tasks.workunit.client.0.smithi113.stdout:7/518: creat d0/d6c/fb0 x:0 0 0 2024-04-12T05:09:00.164 INFO:tasks.workunit.client.0.smithi113.stdout:4/503: creat da/dd/d3a/fa3 x:0 0 0 2024-04-12T05:09:00.165 INFO:tasks.workunit.client.0.smithi113.stdout:7/519: creat d0/dc/da9/d5c/d85/fb1 x:0 0 0 2024-04-12T05:09:00.166 INFO:tasks.workunit.client.0.smithi113.stdout:7/520: readlink d0/dc/d6/d1a/l46 0 2024-04-12T05:09:00.166 INFO:tasks.workunit.client.0.smithi113.stdout:4/504: read da/d3c/d83/f40 [7435215,76659] 0 2024-04-12T05:09:00.168 INFO:tasks.workunit.client.0.smithi113.stdout:7/521: symlink d0/lb2 0 2024-04-12T05:09:00.172 INFO:tasks.workunit.client.0.smithi113.stdout:4/505: link da/d26/c8f da/d3c/d9f/ca4 0 2024-04-12T05:09:00.173 INFO:tasks.workunit.client.0.smithi113.stdout:7/522: mkdir d0/dc/d6/d9b/d9f/db3 0 2024-04-12T05:09:00.178 INFO:tasks.workunit.client.0.smithi113.stdout:4/506: getdents da 0 2024-04-12T05:09:00.180 INFO:tasks.workunit.client.0.smithi113.stdout:7/523: dwrite d0/dd/f30 [0,4194304] 0 2024-04-12T05:09:00.182 INFO:tasks.workunit.client.0.smithi113.stdout:4/507: unlink da/d24/f69 0 2024-04-12T05:09:00.183 INFO:tasks.workunit.client.0.smithi113.stdout:7/524: creat d0/dc/d6/d9b/fb4 x:0 0 0 2024-04-12T05:09:00.185 INFO:tasks.workunit.client.0.smithi113.stdout:4/508: mkdir da/d3c/da5 0 2024-04-12T05:09:00.186 INFO:tasks.workunit.client.0.smithi113.stdout:4/509: chown da/d70/d80/l9b 595 1 2024-04-12T05:09:00.187 INFO:tasks.workunit.client.0.smithi113.stdout:7/525: creat d0/dd/fb5 x:0 0 0 2024-04-12T05:09:00.189 INFO:tasks.workunit.client.0.smithi113.stdout:7/526: write d0/dc/da9/d5c/d85/fa4 [1885610,74895] 0 2024-04-12T05:09:00.196 INFO:tasks.workunit.client.1.smithi192.stdout:7/305: dwrite d3/d7/d15/f29 [0,4194304] 0 2024-04-12T05:09:00.197 INFO:tasks.workunit.client.1.smithi192.stdout:6/446: fsync d3/d26/f28 0 2024-04-12T05:09:00.197 INFO:tasks.workunit.client.1.smithi192.stdout:7/306: chown d3/d1d/d4a/c63 108872447 1 2024-04-12T05:09:00.198 INFO:tasks.workunit.client.1.smithi192.stdout:3/319: unlink d5/d13/d3c/f24 0 2024-04-12T05:09:00.210 INFO:tasks.workunit.client.0.smithi113.stdout:4/510: dread da/d4f/f5d [0,4194304] 0 2024-04-12T05:09:00.212 INFO:tasks.workunit.client.0.smithi113.stdout:4/511: creat da/d3c/d83/fa6 x:0 0 0 2024-04-12T05:09:00.215 INFO:tasks.workunit.client.0.smithi113.stdout:4/512: mknod da/dd/d3a/ca7 0 2024-04-12T05:09:00.217 INFO:tasks.workunit.client.0.smithi113.stdout:4/513: unlink da/dd/d1b/c76 0 2024-04-12T05:09:00.219 INFO:tasks.workunit.client.0.smithi113.stdout:9/552: sync 2024-04-12T05:09:00.219 INFO:tasks.workunit.client.0.smithi113.stdout:5/447: sync 2024-04-12T05:09:00.220 INFO:tasks.workunit.client.0.smithi113.stdout:9/553: chown d1/c64 17 1 2024-04-12T05:09:00.220 INFO:tasks.workunit.client.0.smithi113.stdout:4/514: creat da/d24/d7e/d79/fa8 x:0 0 0 2024-04-12T05:09:00.221 INFO:tasks.workunit.client.0.smithi113.stdout:9/554: chown d1/df/d29/db5 106605860 1 2024-04-12T05:09:00.223 INFO:tasks.workunit.client.0.smithi113.stdout:5/448: creat d0/d10/d27/d1d/d38/d44/d66/f96 x:0 0 0 2024-04-12T05:09:00.224 INFO:tasks.workunit.client.0.smithi113.stdout:4/515: chown da/d3c/d83/c33 182595 1 2024-04-12T05:09:00.225 INFO:tasks.workunit.client.0.smithi113.stdout:4/516: chown da/d24/l4b 1775453 1 2024-04-12T05:09:00.227 INFO:tasks.workunit.client.0.smithi113.stdout:4/517: write da/dd/d3a/fa3 [315526,126121] 0 2024-04-12T05:09:00.228 INFO:tasks.workunit.client.0.smithi113.stdout:9/555: dwrite d1/d27/f3c [0,4194304] 0 2024-04-12T05:09:00.229 INFO:tasks.workunit.client.0.smithi113.stdout:9/556: write d1/d4/f77 [1568537,14880] 0 2024-04-12T05:09:00.230 INFO:tasks.workunit.client.0.smithi113.stdout:5/449: rename d0/l91 to d0/d10/d27/d1d/d3f/d73/d89/l97 0 2024-04-12T05:09:00.231 INFO:tasks.workunit.client.0.smithi113.stdout:9/557: chown d1/d4/d5/f12 1197862408 1 2024-04-12T05:09:00.233 INFO:tasks.workunit.client.0.smithi113.stdout:5/450: write d0/d10/d27/f22 [2811672,46867] 0 2024-04-12T05:09:00.240 INFO:tasks.workunit.client.0.smithi113.stdout:5/451: rename d0/d10/d27/d1d/f60 to d0/f98 0 2024-04-12T05:09:00.241 INFO:tasks.workunit.client.0.smithi113.stdout:5/452: read d0/d10/d27/d1d/d3f/f47 [388846,42203] 0 2024-04-12T05:09:00.248 INFO:tasks.workunit.client.0.smithi113.stdout:5/453: rename d0/d10/d27/d1d/l5b to d0/d10/l99 0 2024-04-12T05:09:00.249 INFO:tasks.workunit.client.0.smithi113.stdout:4/518: dwrite da/d24/f53 [0,4194304] 0 2024-04-12T05:09:00.252 INFO:tasks.workunit.client.1.smithi192.stdout:4/454: dwrite d1/f26 [0,4194304] 0 2024-04-12T05:09:00.253 INFO:tasks.workunit.client.1.smithi192.stdout:4/455: chown d1/d54/c70 0 1 2024-04-12T05:09:00.257 INFO:tasks.workunit.client.0.smithi113.stdout:4/519: dwrite da/d24/d60/f99 [0,4194304] 0 2024-04-12T05:09:00.262 INFO:tasks.workunit.client.0.smithi113.stdout:4/520: rename da/d3c/d43/f4d to da/d70/fa9 0 2024-04-12T05:09:00.263 INFO:tasks.workunit.client.0.smithi113.stdout:4/521: stat da/d3c/d83/fa6 0 2024-04-12T05:09:00.265 INFO:tasks.workunit.client.0.smithi113.stdout:4/522: chown da/d70/d80/l9b 1 1 2024-04-12T05:09:00.270 INFO:tasks.workunit.client.1.smithi192.stdout:3/320: write d5/f1c [1332683,3927] 0 2024-04-12T05:09:00.271 INFO:tasks.workunit.client.1.smithi192.stdout:4/456: mkdir d1/d1d/d53/d1c/d95 0 2024-04-12T05:09:00.271 INFO:tasks.workunit.client.1.smithi192.stdout:3/321: stat d5/d13/d3a/d4e/d57 0 2024-04-12T05:09:00.274 INFO:tasks.workunit.client.0.smithi113.stdout:5/454: dread d0/d10/d27/d1d/f1e [0,4194304] 0 2024-04-12T05:09:00.276 INFO:tasks.workunit.client.0.smithi113.stdout:5/455: rmdir d0/d10/d27/d5f 39 2024-04-12T05:09:00.277 INFO:tasks.workunit.client.1.smithi192.stdout:4/457: link d1/d54/d82/f62 d1/d54/d55/d8b/f96 0 2024-04-12T05:09:00.278 INFO:tasks.workunit.client.0.smithi113.stdout:5/456: write d0/d10/f45 [850271,120522] 0 2024-04-12T05:09:00.280 INFO:tasks.workunit.client.0.smithi113.stdout:5/457: rmdir d0 39 2024-04-12T05:09:00.281 INFO:tasks.workunit.client.1.smithi192.stdout:7/307: truncate d3/d11/f16 27614 0 2024-04-12T05:09:00.283 INFO:tasks.workunit.client.1.smithi192.stdout:4/458: mknod d1/d54/d55/d64/d68/c97 0 2024-04-12T05:09:00.284 INFO:tasks.workunit.client.1.smithi192.stdout:7/308: dread - d3/d7/d60/f66 zero size 2024-04-12T05:09:00.287 INFO:tasks.workunit.client.1.smithi192.stdout:1/364: dwrite d0/d2/dd/d21/f2d [0,4194304] 0 2024-04-12T05:09:00.293 INFO:tasks.workunit.client.1.smithi192.stdout:4/459: symlink d1/d54/d55/d64/d68/l98 0 2024-04-12T05:09:00.293 INFO:tasks.workunit.client.1.smithi192.stdout:7/309: read - d3/d7/d14/f58 zero size 2024-04-12T05:09:00.293 INFO:tasks.workunit.client.1.smithi192.stdout:1/365: chown d0/d29/l2f 501546 1 2024-04-12T05:09:00.294 INFO:tasks.workunit.client.1.smithi192.stdout:6/447: dwrite d3/d8/f25 [4194304,4194304] 0 2024-04-12T05:09:00.296 INFO:tasks.workunit.client.1.smithi192.stdout:3/322: dwrite d5/f19 [0,4194304] 0 2024-04-12T05:09:00.299 INFO:tasks.workunit.client.1.smithi192.stdout:1/366: mkdir d0/d2/dd/d71 0 2024-04-12T05:09:00.299 INFO:tasks.workunit.client.1.smithi192.stdout:4/460: creat d1/d54/f99 x:0 0 0 2024-04-12T05:09:00.301 INFO:tasks.workunit.client.1.smithi192.stdout:1/367: fsync d0/d2/d1e/f36 0 2024-04-12T05:09:00.303 INFO:tasks.workunit.client.1.smithi192.stdout:4/461: dread d1/d76/f79 [0,4194304] 0 2024-04-12T05:09:00.303 INFO:tasks.workunit.client.1.smithi192.stdout:3/323: fsync d5/d13/f2e 0 2024-04-12T05:09:00.303 INFO:tasks.workunit.client.1.smithi192.stdout:4/462: chown d1/f33 4 1 2024-04-12T05:09:00.304 INFO:tasks.workunit.client.1.smithi192.stdout:7/310: creat d3/d1d/d5a/d5f/d6a/f6c x:0 0 0 2024-04-12T05:09:00.305 INFO:tasks.workunit.client.1.smithi192.stdout:7/311: chown d3/d1d/f28 334350289 1 2024-04-12T05:09:00.306 INFO:tasks.workunit.client.1.smithi192.stdout:6/448: truncate d3/d8/d48/f50 23509 0 2024-04-12T05:09:00.308 INFO:tasks.workunit.client.1.smithi192.stdout:1/368: mknod d0/d2/c72 0 2024-04-12T05:09:00.308 INFO:tasks.workunit.client.1.smithi192.stdout:3/324: fdatasync d5/d13/d3c/f25 0 2024-04-12T05:09:00.309 INFO:tasks.workunit.client.1.smithi192.stdout:1/369: stat d0/d2/dd/f5d 0 2024-04-12T05:09:00.309 INFO:tasks.workunit.client.1.smithi192.stdout:4/463: creat d1/d54/f9a x:0 0 0 2024-04-12T05:09:00.311 INFO:tasks.workunit.client.1.smithi192.stdout:8/318: sync 2024-04-12T05:09:00.311 INFO:tasks.workunit.client.1.smithi192.stdout:5/373: sync 2024-04-12T05:09:00.311 INFO:tasks.workunit.client.1.smithi192.stdout:3/325: dread - d5/d13/d3a/d4e/d57/f51 zero size 2024-04-12T05:09:00.312 INFO:tasks.workunit.client.1.smithi192.stdout:6/449: rmdir d3/d8/d10 39 2024-04-12T05:09:00.313 INFO:tasks.workunit.client.1.smithi192.stdout:1/370: unlink d0/d2/d1e/d3f/f52 0 2024-04-12T05:09:00.314 INFO:tasks.workunit.client.1.smithi192.stdout:5/374: write d0/d15/f23 [639501,103846] 0 2024-04-12T05:09:00.316 INFO:tasks.workunit.client.1.smithi192.stdout:4/464: dread d1/d54/d55/f30 [0,4194304] 0 2024-04-12T05:09:00.318 INFO:tasks.workunit.client.1.smithi192.stdout:5/375: read d0/f14 [3944417,47726] 0 2024-04-12T05:09:00.318 INFO:tasks.workunit.client.1.smithi192.stdout:3/326: write d5/d13/d3a/d4a/d61/f64 [875681,32120] 0 2024-04-12T05:09:00.319 INFO:tasks.workunit.client.1.smithi192.stdout:1/371: symlink d0/d2/dd/d71/l73 0 2024-04-12T05:09:00.321 INFO:tasks.workunit.client.1.smithi192.stdout:7/312: write d3/d7/d15/f50 [1403457,30410] 0 2024-04-12T05:09:00.322 INFO:tasks.workunit.client.1.smithi192.stdout:3/327: symlink d5/d13/d1a/d53/l6a 0 2024-04-12T05:09:00.323 INFO:tasks.workunit.client.1.smithi192.stdout:5/376: chown d0/d15/d24/c3a 22954754 1 2024-04-12T05:09:00.324 INFO:tasks.workunit.client.1.smithi192.stdout:3/328: write d5/d13/d3c/f68 [765552,57518] 0 2024-04-12T05:09:00.324 INFO:tasks.workunit.client.1.smithi192.stdout:4/465: symlink d1/d1d/d53/l9b 0 2024-04-12T05:09:00.326 INFO:tasks.workunit.client.1.smithi192.stdout:6/450: getdents d3/d8/d56/d59 0 2024-04-12T05:09:00.326 INFO:tasks.workunit.client.1.smithi192.stdout:4/466: chown d1/d54/f81 2049260 1 2024-04-12T05:09:00.326 INFO:tasks.workunit.client.1.smithi192.stdout:8/319: mkdir d6/d10/d46/d5e 0 2024-04-12T05:09:00.328 INFO:tasks.workunit.client.1.smithi192.stdout:1/372: unlink d0/d2/dd/d3c/f3e 0 2024-04-12T05:09:00.328 INFO:tasks.workunit.client.1.smithi192.stdout:5/377: dread d0/f20 [0,4194304] 0 2024-04-12T05:09:00.330 INFO:tasks.workunit.client.1.smithi192.stdout:1/373: write d0/d2/dd/f32 [3551561,40147] 0 2024-04-12T05:09:00.332 INFO:tasks.workunit.client.1.smithi192.stdout:3/329: dread d5/d13/d3c/f59 [0,4194304] 0 2024-04-12T05:09:00.353 INFO:tasks.workunit.client.0.smithi113.stdout:5/458: symlink d0/d10/d27/d1d/d38/l9a 0 2024-04-12T05:09:00.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:08:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:58.670869+0000 mgr.smithi192.ocppyo (mgr.24357) 25 : cluster [DBG] pgmap v19: 65 pgs: 65 active+clean; 1.5 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 31 MiB/s rd, 202 MiB/s wr, 311 op/s 2024-04-12T05:09:00.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:08:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:58.670869+0000 mgr.smithi192.ocppyo (mgr.24357) 25 : cluster [DBG] pgmap v19: 65 pgs: 65 active+clean; 1.5 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 31 MiB/s rd, 202 MiB/s wr, 311 op/s 2024-04-12T05:09:00.453 INFO:tasks.workunit.client.0.smithi113.stdout:0/552: sync 2024-04-12T05:09:00.483 INFO:tasks.workunit.client.1.smithi192.stdout:0/487: sync 2024-04-12T05:09:00.483 INFO:tasks.workunit.client.1.smithi192.stdout:9/308: sync 2024-04-12T05:09:00.483 INFO:tasks.workunit.client.1.smithi192.stdout:2/416: sync 2024-04-12T05:09:00.484 INFO:tasks.workunit.client.1.smithi192.stdout:2/417: chown d1/d8/d17/d19/d40 1198 1 2024-04-12T05:09:00.501 INFO:tasks.workunit.client.1.smithi192.stdout:1/374: mkdir d0/d2/dd/d3c/d5b/d74 0 2024-04-12T05:09:00.508 INFO:tasks.workunit.client.1.smithi192.stdout:6/451: mknod d3/d8/d10/d4c/c95 0 2024-04-12T05:09:00.511 INFO:tasks.workunit.client.1.smithi192.stdout:6/452: write d3/d8/d10/d4c/d68/f89 [373489,114131] 0 2024-04-12T05:09:00.512 INFO:tasks.workunit.client.1.smithi192.stdout:6/453: readlink d3/d8/d10/d4c/d68/l7b 0 2024-04-12T05:09:00.551 INFO:tasks.workunit.client.1.smithi192.stdout:8/320: sync 2024-04-12T05:09:00.591 INFO:tasks.workunit.client.1.smithi192.stdout:5/378: sync 2024-04-12T05:09:00.601 INFO:tasks.workunit.client.0.smithi113.stdout:2/391: sync 2024-04-12T05:09:00.719 INFO:tasks.workunit.client.0.smithi113.stdout:2/392: link d8/d9/de/d43/l62 d8/d9/d1a/l71 0 2024-04-12T05:09:00.721 INFO:tasks.workunit.client.0.smithi113.stdout:2/393: creat d8/d9/d27/f72 x:0 0 0 2024-04-12T05:09:00.722 INFO:tasks.workunit.client.0.smithi113.stdout:2/394: truncate d8/d16/f6e 64485 0 2024-04-12T05:09:00.728 INFO:tasks.workunit.client.0.smithi113.stdout:2/395: dwrite d8/d9/de/f67 [0,4194304] 0 2024-04-12T05:09:00.732 INFO:tasks.workunit.client.0.smithi113.stdout:2/396: mknod d8/d9/de/d12/d3c/c73 0 2024-04-12T05:09:00.733 INFO:tasks.workunit.client.0.smithi113.stdout:2/397: mknod d8/d9/de/c74 0 2024-04-12T05:09:00.734 INFO:tasks.workunit.client.0.smithi113.stdout:2/398: fsync d8/d9/de/d12/f4c 0 2024-04-12T05:09:00.737 INFO:tasks.workunit.client.0.smithi113.stdout:2/399: read d8/d9/de/d12/f4c [317415,98551] 0 2024-04-12T05:09:00.739 INFO:tasks.workunit.client.0.smithi113.stdout:2/400: fsync d8/d9/de/d12/d2b/f54 0 2024-04-12T05:09:00.740 INFO:tasks.workunit.client.0.smithi113.stdout:2/401: chown d8/d16/f61 26241 1 2024-04-12T05:09:00.741 INFO:tasks.workunit.client.0.smithi113.stdout:2/402: write d8/d9/de/f10 [2508987,75742] 0 2024-04-12T05:09:00.744 INFO:tasks.workunit.client.0.smithi113.stdout:2/403: creat d8/d9/de/d15/d24/d58/f75 x:0 0 0 2024-04-12T05:09:00.747 INFO:tasks.workunit.client.0.smithi113.stdout:2/404: creat d8/d9/de/d12/d44/f76 x:0 0 0 2024-04-12T05:09:00.748 INFO:tasks.workunit.client.0.smithi113.stdout:2/405: dread d8/d16/f6e [0,4194304] 0 2024-04-12T05:09:00.750 INFO:tasks.workunit.client.0.smithi113.stdout:2/406: unlink d8/d9/de/d12/d2b/c6c 0 2024-04-12T05:09:00.751 INFO:tasks.workunit.client.0.smithi113.stdout:2/407: write d8/d9/de/f31 [2138962,80006] 0 2024-04-12T05:09:00.767 INFO:tasks.workunit.client.0.smithi113.stdout:2/408: dread d8/d16/f37 [0,4194304] 0 2024-04-12T05:09:00.769 INFO:tasks.workunit.client.0.smithi113.stdout:2/409: mknod d8/d9/de/d15/c77 0 2024-04-12T05:09:00.772 INFO:tasks.workunit.client.0.smithi113.stdout:2/410: link d8/d9/d1a/f3a d8/d9/de/d12/f78 0 2024-04-12T05:09:00.775 INFO:tasks.workunit.client.0.smithi113.stdout:2/411: getdents d8/d9/de/d43 0 2024-04-12T05:09:00.777 INFO:tasks.workunit.client.0.smithi113.stdout:2/412: mkdir d8/d9/de/d12/d2b/d79 0 2024-04-12T05:09:00.782 INFO:tasks.workunit.client.0.smithi113.stdout:1/619: sync 2024-04-12T05:09:00.783 INFO:tasks.workunit.client.0.smithi113.stdout:2/413: write f7 [2838977,96209] 0 2024-04-12T05:09:00.786 INFO:tasks.workunit.client.0.smithi113.stdout:2/414: getdents d8/d9/de/d12/d44 0 2024-04-12T05:09:00.789 INFO:tasks.workunit.client.0.smithi113.stdout:2/415: symlink d8/d9/l7a 0 2024-04-12T05:09:00.796 INFO:tasks.workunit.client.0.smithi113.stdout:2/416: dwrite d8/d16/f6e [0,4194304] 0 2024-04-12T05:09:00.798 INFO:tasks.workunit.client.0.smithi113.stdout:2/417: read - d8/d9/de/d12/d44/f76 zero size 2024-04-12T05:09:00.800 INFO:tasks.workunit.client.0.smithi113.stdout:2/418: unlink d8/d9/f55 0 2024-04-12T05:09:00.800 INFO:tasks.workunit.client.0.smithi113.stdout:2/419: stat d8/d9/lf 0 2024-04-12T05:09:00.803 INFO:tasks.workunit.client.0.smithi113.stdout:2/420: mknod d8/d9/de/d15/d24/d58/c7b 0 2024-04-12T05:09:00.805 INFO:tasks.workunit.client.0.smithi113.stdout:2/421: fsync d8/d9/de/d12/d2b/f54 0 2024-04-12T05:09:00.806 INFO:tasks.workunit.client.1.smithi192.stdout:9/309: unlink d11/d15/d21/d2c/f62 0 2024-04-12T05:09:00.806 INFO:tasks.workunit.client.0.smithi113.stdout:2/422: chown d8/d9/de/d15/f29 236 1 2024-04-12T05:09:00.809 INFO:tasks.workunit.client.0.smithi113.stdout:2/423: unlink d8/d9/l59 0 2024-04-12T05:09:00.810 INFO:tasks.workunit.client.0.smithi113.stdout:2/424: chown d8/d9/de/l2a 177933 1 2024-04-12T05:09:00.817 INFO:tasks.workunit.client.0.smithi113.stdout:2/425: dwrite d8/d9/de/d15/f18 [0,4194304] 0 2024-04-12T05:09:00.837 INFO:tasks.workunit.client.1.smithi192.stdout:9/310: dread d11/d25/f52 [0,4194304] 0 2024-04-12T05:09:00.843 INFO:tasks.workunit.client.0.smithi113.stdout:2/426: getdents d8/d9/d27/d3f 0 2024-04-12T05:09:00.844 INFO:tasks.workunit.client.0.smithi113.stdout:2/427: write d8/d9/d1a/f50 [683147,70668] 0 2024-04-12T05:09:00.845 INFO:tasks.workunit.client.0.smithi113.stdout:2/428: chown d8/d9/d27/d3f/l4a 41003 1 2024-04-12T05:09:00.847 INFO:tasks.workunit.client.0.smithi113.stdout:2/429: truncate d8/d9/de/d12/f5d 826487 0 2024-04-12T05:09:00.850 INFO:tasks.workunit.client.0.smithi113.stdout:2/430: mknod d8/d9/d27/d3f/c7c 0 2024-04-12T05:09:00.850 INFO:tasks.workunit.client.0.smithi113.stdout:2/431: readlink d8/l25 0 2024-04-12T05:09:00.857 INFO:tasks.workunit.client.0.smithi113.stdout:2/432: dwrite d8/d9/de/d12/f5d [0,4194304] 0 2024-04-12T05:09:00.859 INFO:tasks.workunit.client.0.smithi113.stdout:2/433: write d8/d16/f6e [1556763,23777] 0 2024-04-12T05:09:00.870 INFO:tasks.workunit.client.0.smithi113.stdout:2/434: mknod d8/d9/de/d12/d44/c7d 0 2024-04-12T05:09:00.873 INFO:tasks.workunit.client.0.smithi113.stdout:2/435: creat d8/d9/de/d15/d24/f7e x:0 0 0 2024-04-12T05:09:00.876 INFO:tasks.workunit.client.0.smithi113.stdout:2/436: dread d8/d16/f4e [0,4194304] 0 2024-04-12T05:09:00.877 INFO:tasks.workunit.client.0.smithi113.stdout:2/437: getdents d8/d9/de/d43 0 2024-04-12T05:09:00.879 INFO:tasks.workunit.client.0.smithi113.stdout:6/507: sync 2024-04-12T05:09:00.879 INFO:tasks.workunit.client.0.smithi113.stdout:3/502: sync 2024-04-12T05:09:00.879 INFO:tasks.workunit.client.0.smithi113.stdout:8/568: sync 2024-04-12T05:09:00.879 INFO:tasks.workunit.client.0.smithi113.stdout:7/527: sync 2024-04-12T05:09:00.880 INFO:tasks.workunit.client.0.smithi113.stdout:2/438: dread d8/d9/de/d15/f40 [0,4194304] 0 2024-04-12T05:09:00.881 INFO:tasks.workunit.client.0.smithi113.stdout:6/508: dread - d8/de/d17/f59 zero size 2024-04-12T05:09:00.883 INFO:tasks.workunit.client.0.smithi113.stdout:6/509: write d8/de/d17/f72 [1773788,98022] 0 2024-04-12T05:09:00.884 INFO:tasks.workunit.client.0.smithi113.stdout:3/503: creat d0/d13/d15/d28/d2a/d8b/d95/faa x:0 0 0 2024-04-12T05:09:00.887 INFO:tasks.workunit.client.0.smithi113.stdout:8/569: rename d9/d16/d7f/d3a/c3e to d9/d1b/d8e/d93/d32/d48/d8d/dae/cb6 0 2024-04-12T05:09:00.888 INFO:tasks.workunit.client.0.smithi113.stdout:6/510: mkdir d8/de/d13/d11/d36/d53/db7 0 2024-04-12T05:09:00.889 INFO:tasks.workunit.client.0.smithi113.stdout:8/570: dread - d9/d16/d65/d8c/fb5 zero size 2024-04-12T05:09:00.891 INFO:tasks.workunit.client.1.smithi192.stdout:3/330: symlink d5/d13/d1a/d5f/l6b 0 2024-04-12T05:09:00.892 INFO:tasks.workunit.client.0.smithi113.stdout:8/571: read - d9/d1b/d8e/d93/f71 zero size 2024-04-12T05:09:00.892 INFO:tasks.workunit.client.0.smithi113.stdout:3/504: link d0/d13/d47/la1 d0/d13/d47/lab 0 2024-04-12T05:09:00.894 INFO:tasks.workunit.client.0.smithi113.stdout:8/572: truncate d9/d16/d65/db1/d59/d78/f9f 851794 0 2024-04-12T05:09:00.895 INFO:tasks.workunit.client.0.smithi113.stdout:2/439: dread d8/d9/de/d12/d2b/f54 [0,4194304] 0 2024-04-12T05:09:00.896 INFO:tasks.workunit.client.0.smithi113.stdout:6/511: rename d8/de/d13/d11/d36/d3d/d4c/d31/d8e to d8/de/d13/d11/d36/d53/db8 0 2024-04-12T05:09:00.896 INFO:tasks.workunit.client.0.smithi113.stdout:3/505: mkdir d0/d13/d47/d94/dac 0 2024-04-12T05:09:00.897 INFO:tasks.workunit.client.0.smithi113.stdout:8/573: creat d9/d1b/d8e/d93/fb7 x:0 0 0 2024-04-12T05:09:00.897 INFO:tasks.workunit.client.0.smithi113.stdout:2/440: stat d8/d9/d27/f65 0 2024-04-12T05:09:00.899 INFO:tasks.workunit.client.0.smithi113.stdout:6/512: mknod d8/de/cb9 0 2024-04-12T05:09:00.899 INFO:tasks.workunit.client.0.smithi113.stdout:2/441: fdatasync d8/d9/de/d12/f4c 0 2024-04-12T05:09:00.899 INFO:tasks.workunit.client.0.smithi113.stdout:8/574: rmdir d9/d1b/d8e/d93/d92 39 2024-04-12T05:09:00.900 INFO:tasks.workunit.client.1.smithi192.stdout:7/313: getdents d3/d7/d60 0 2024-04-12T05:09:00.901 INFO:tasks.workunit.client.0.smithi113.stdout:2/442: symlink d8/d9/de/d15/d24/l7f 0 2024-04-12T05:09:00.903 INFO:tasks.workunit.client.1.smithi192.stdout:4/467: dwrite d1/d54/d55/d8b/f96 [0,4194304] 0 2024-04-12T05:09:00.903 INFO:tasks.workunit.client.1.smithi192.stdout:4/468: readlink d1/d1d/l1e 0 2024-04-12T05:09:00.903 INFO:tasks.workunit.client.0.smithi113.stdout:8/575: unlink d9/d1b/d8e/d93/d32/d48/f76 0 2024-04-12T05:09:00.906 INFO:tasks.workunit.client.0.smithi113.stdout:6/513: rename d8/de/f15 to d8/de/d13/d11/d36/d53/db7/fba 0 2024-04-12T05:09:00.907 INFO:tasks.workunit.client.0.smithi113.stdout:2/443: mknod d8/d9/de/d15/c80 0 2024-04-12T05:09:00.910 INFO:tasks.workunit.client.0.smithi113.stdout:8/576: dwrite d9/d16/d3c/f74 [0,4194304] 0 2024-04-12T05:09:00.916 INFO:tasks.workunit.client.0.smithi113.stdout:8/577: creat d9/d1b/d8e/d93/d32/d5b/fb8 x:0 0 0 2024-04-12T05:09:00.917 INFO:tasks.workunit.client.0.smithi113.stdout:2/444: dwrite d8/d9/de/d15/d24/f3b [0,4194304] 0 2024-04-12T05:09:00.917 INFO:tasks.workunit.client.1.smithi192.stdout:6/454: rmdir d3/dc 39 2024-04-12T05:09:00.918 INFO:tasks.workunit.client.0.smithi113.stdout:8/578: write d9/d16/d3c/f4e [4499046,10904] 0 2024-04-12T05:09:00.921 INFO:tasks.workunit.client.0.smithi113.stdout:2/445: symlink d8/l81 0 2024-04-12T05:09:00.922 INFO:tasks.workunit.client.0.smithi113.stdout:8/579: write d9/d1b/d8e/fb3 [597111,62296] 0 2024-04-12T05:09:00.924 INFO:tasks.workunit.client.0.smithi113.stdout:8/580: chown d9/d16/d7f/d3a/d6c/d90/cad 16154 1 2024-04-12T05:09:00.924 INFO:tasks.workunit.client.1.smithi192.stdout:8/321: rename d6/df to d6/d10/d46/d5e/d5f 0 2024-04-12T05:09:00.925 INFO:tasks.workunit.client.1.smithi192.stdout:8/322: dread - d6/d10/d46/d5e/d5f/f4d zero size 2024-04-12T05:09:00.926 INFO:tasks.workunit.client.0.smithi113.stdout:2/446: creat d8/d9/de/d15/d38/f82 x:0 0 0 2024-04-12T05:09:00.932 INFO:tasks.workunit.client.0.smithi113.stdout:8/581: link d9/d1b/d8e/d93/d32/f60 d9/d16/d7f/d3a/d6c/d90/fb9 0 2024-04-12T05:09:00.947 INFO:tasks.workunit.client.0.smithi113.stdout:1/620: write d10/d14/d67/f69 [2092328,32837] 0 2024-04-12T05:09:00.949 INFO:tasks.workunit.client.0.smithi113.stdout:1/621: dread - d10/d41/d89/d96/d7d/fb2 zero size 2024-04-12T05:09:00.952 INFO:tasks.workunit.client.0.smithi113.stdout:8/582: dread d9/d1b/d8e/d93/d32/d5b/f63 [0,4194304] 0 2024-04-12T05:09:00.954 INFO:tasks.workunit.client.0.smithi113.stdout:0/553: mknod d1/d6/d16/d2a/d50/cac 0 2024-04-12T05:09:00.957 INFO:tasks.workunit.client.0.smithi113.stdout:1/622: rename d10/c6f to d10/d2e/d31/d5c/daf/cc2 0 2024-04-12T05:09:00.959 INFO:tasks.workunit.client.1.smithi192.stdout:9/311: mknod d11/d15/d21/c68 0 2024-04-12T05:09:00.960 INFO:tasks.workunit.client.1.smithi192.stdout:0/488: dwrite d0/d6/d15/d1b/d46/f49 [0,4194304] 0 2024-04-12T05:09:00.961 INFO:tasks.workunit.client.0.smithi113.stdout:0/554: rename d1/d6/f1d to d1/d6/da/d36/fad 0 2024-04-12T05:09:00.963 INFO:tasks.workunit.client.0.smithi113.stdout:8/583: dread d9/fb [0,4194304] 0 2024-04-12T05:09:00.964 INFO:tasks.workunit.client.0.smithi113.stdout:8/584: truncate d9/f9e 413744 0 2024-04-12T05:09:00.965 INFO:tasks.workunit.client.0.smithi113.stdout:8/585: write d9/d1b/f46 [224837,39233] 0 2024-04-12T05:09:00.967 INFO:tasks.workunit.client.1.smithi192.stdout:3/331: creat d5/d13/d3a/d4e/f6c x:0 0 0 2024-04-12T05:09:00.968 INFO:tasks.workunit.client.1.smithi192.stdout:3/332: chown d5/d13/d1a/d53 7188004 1 2024-04-12T05:09:00.968 INFO:tasks.workunit.client.0.smithi113.stdout:0/555: creat d1/d6/d16/d2a/d50/fae x:0 0 0 2024-04-12T05:09:00.969 INFO:tasks.workunit.client.1.smithi192.stdout:1/375: symlink d0/d2/dd/d21/d55/l75 0 2024-04-12T05:09:00.970 INFO:tasks.workunit.client.1.smithi192.stdout:1/376: write d0/d1b/d28/f61 [588244,91976] 0 2024-04-12T05:09:00.973 INFO:tasks.workunit.client.1.smithi192.stdout:4/469: truncate d1/d54/f61 1532583 0 2024-04-12T05:09:00.974 INFO:tasks.workunit.client.0.smithi113.stdout:8/586: dwrite d9/d16/d65/f97 [0,4194304] 0 2024-04-12T05:09:00.979 INFO:tasks.workunit.client.0.smithi113.stdout:0/556: mknod d1/d43/caf 0 2024-04-12T05:09:00.979 INFO:tasks.workunit.client.1.smithi192.stdout:5/379: mknod d0/d15/d24/c84 0 2024-04-12T05:09:00.981 INFO:tasks.workunit.client.1.smithi192.stdout:8/323: creat d6/d21/f60 x:0 0 0 2024-04-12T05:09:00.981 INFO:tasks.workunit.client.1.smithi192.stdout:5/380: truncate d0/d15/d24/f46 4573265 0 2024-04-12T05:09:00.982 INFO:tasks.workunit.client.0.smithi113.stdout:8/587: link d9/d1b/f4f d9/d1b/d96/fba 0 2024-04-12T05:09:00.983 INFO:tasks.workunit.client.0.smithi113.stdout:0/557: unlink d1/d6/d34/f68 0 2024-04-12T05:09:00.986 INFO:tasks.workunit.client.0.smithi113.stdout:0/558: unlink d1/d9/c39 0 2024-04-12T05:09:00.988 INFO:tasks.workunit.client.1.smithi192.stdout:0/489: mknod d0/d4e/d95/ca6 0 2024-04-12T05:09:00.989 INFO:tasks.workunit.client.1.smithi192.stdout:2/418: rename d1/f6d to d1/f8b 0 2024-04-12T05:09:00.990 INFO:tasks.workunit.client.1.smithi192.stdout:2/419: dread - d1/f4b zero size 2024-04-12T05:09:00.992 INFO:tasks.workunit.client.0.smithi113.stdout:0/559: unlink d1/d43/d59/fa5 0 2024-04-12T05:09:00.993 INFO:tasks.workunit.client.1.smithi192.stdout:3/333: symlink d5/d13/d3c/l6d 0 2024-04-12T05:09:00.994 INFO:tasks.workunit.client.0.smithi113.stdout:0/560: chown d1/d9/f2b 27 1 2024-04-12T05:09:00.994 INFO:tasks.workunit.client.0.smithi113.stdout:8/588: dwrite d9/d1b/f4f [0,4194304] 0 2024-04-12T05:09:00.994 INFO:tasks.workunit.client.1.smithi192.stdout:3/334: write d5/d13/d3a/d4a/f67 [115386,76720] 0 2024-04-12T05:09:00.996 INFO:tasks.workunit.client.1.smithi192.stdout:7/314: mkdir d3/d1d/d4a/d5e/d6d 0 2024-04-12T05:09:00.998 INFO:tasks.workunit.client.0.smithi113.stdout:8/589: symlink d9/d1b/d8e/d93/d32/d48/d8d/lbb 0 2024-04-12T05:09:01.000 INFO:tasks.workunit.client.1.smithi192.stdout:1/377: truncate d0/d2/dd/f1d 2981373 0 2024-04-12T05:09:01.001 INFO:tasks.workunit.client.1.smithi192.stdout:1/378: write d0/d2/dd/d21/d33/f6b [695176,12456] 0 2024-04-12T05:09:01.002 INFO:tasks.workunit.client.1.smithi192.stdout:1/379: truncate d0/f1a 4612650 0 2024-04-12T05:09:01.003 INFO:tasks.workunit.client.1.smithi192.stdout:4/470: mkdir d1/d54/d82/d9c 0 2024-04-12T05:09:01.004 INFO:tasks.workunit.client.0.smithi113.stdout:8/590: creat d9/d16/d7f/d3a/d6c/d90/fbc x:0 0 0 2024-04-12T05:09:01.004 INFO:tasks.workunit.client.1.smithi192.stdout:4/471: write d1/d1d/d53/d1c/f65 [1163517,21690] 0 2024-04-12T05:09:01.005 INFO:tasks.workunit.client.0.smithi113.stdout:0/561: rename d1/f75 to d1/d6/fb0 0 2024-04-12T05:09:01.008 INFO:tasks.workunit.client.1.smithi192.stdout:6/455: mknod d3/d21/d24/c96 0 2024-04-12T05:09:01.009 INFO:tasks.workunit.client.1.smithi192.stdout:5/381: creat d0/d15/d24/d29/f85 x:0 0 0 2024-04-12T05:09:01.013 INFO:tasks.workunit.client.1.smithi192.stdout:8/324: rename d6/d10/d46/d5e/d5f/d37/d4f/d4c/d54 to d6/d10/d46/d5e/d5f/d31/d61 0 2024-04-12T05:09:01.014 INFO:tasks.workunit.client.1.smithi192.stdout:8/325: chown d6/d1f/f2a 186 1 2024-04-12T05:09:01.017 INFO:tasks.workunit.client.1.smithi192.stdout:2/420: rmdir d1/d8/d3a/de/d7c 39 2024-04-12T05:09:01.017 INFO:tasks.workunit.client.1.smithi192.stdout:2/421: chown d1/d8/d17/d19/d40/f5b 23 1 2024-04-12T05:09:01.019 INFO:tasks.workunit.client.1.smithi192.stdout:3/335: creat d5/d13/d3a/d4e/d57/f6e x:0 0 0 2024-04-12T05:09:01.022 INFO:tasks.workunit.client.1.smithi192.stdout:2/422: dwrite d1/d8/d17/d19/d40/f52 [0,4194304] 0 2024-04-12T05:09:01.023 INFO:tasks.workunit.client.1.smithi192.stdout:1/380: readlink d0/d2/dd/d21/l6d 0 2024-04-12T05:09:01.024 INFO:tasks.workunit.client.1.smithi192.stdout:4/472: mknod d1/d6b/c9d 0 2024-04-12T05:09:01.026 INFO:tasks.workunit.client.1.smithi192.stdout:6/456: mkdir d3/d8/d56/d59/d97 0 2024-04-12T05:09:01.027 INFO:tasks.workunit.client.1.smithi192.stdout:9/312: creat d11/f69 x:0 0 0 2024-04-12T05:09:01.028 INFO:tasks.workunit.client.1.smithi192.stdout:2/423: dwrite d1/f39 [0,4194304] 0 2024-04-12T05:09:01.028 INFO:tasks.workunit.client.1.smithi192.stdout:5/382: creat d0/d15/d32/d45/f86 x:0 0 0 2024-04-12T05:09:01.028 INFO:tasks.workunit.client.1.smithi192.stdout:9/313: chown d11/d15/l2b 58244055 1 2024-04-12T05:09:01.035 INFO:tasks.workunit.client.1.smithi192.stdout:7/315: link d3/d7/d15/f24 d3/d7/d15/d1a/f6e 0 2024-04-12T05:09:01.042 INFO:tasks.workunit.client.0.smithi113.stdout:0/562: symlink d1/d43/d59/lb1 0 2024-04-12T05:09:01.044 INFO:tasks.workunit.client.0.smithi113.stdout:0/563: readlink d1/d6/da/d36/l76 0 2024-04-12T05:09:01.045 INFO:tasks.workunit.client.1.smithi192.stdout:0/490: creat d0/fa7 x:0 0 0 2024-04-12T05:09:01.046 INFO:tasks.workunit.client.0.smithi113.stdout:0/564: rmdir d1/d6/d34 39 2024-04-12T05:09:01.047 INFO:tasks.workunit.client.1.smithi192.stdout:8/326: mknod d6/d10/d46/d5e/d5f/c62 0 2024-04-12T05:09:01.048 INFO:tasks.workunit.client.0.smithi113.stdout:0/565: mkdir d1/d43/d59/db2 0 2024-04-12T05:09:01.049 INFO:tasks.workunit.client.0.smithi113.stdout:0/566: fdatasync d1/d6/d16/f67 0 2024-04-12T05:09:01.050 INFO:tasks.workunit.client.1.smithi192.stdout:7/316: rename d3/d1d/d5a/d5f/d6a/f6c to d3/f6f 0 2024-04-12T05:09:01.052 INFO:tasks.workunit.client.0.smithi113.stdout:9/558: sync 2024-04-12T05:09:01.053 INFO:tasks.workunit.client.0.smithi113.stdout:4/523: sync 2024-04-12T05:09:01.053 INFO:tasks.workunit.client.0.smithi113.stdout:5/459: sync 2024-04-12T05:09:01.053 INFO:tasks.workunit.client.1.smithi192.stdout:9/314: dread f6 [0,4194304] 0 2024-04-12T05:09:01.053 INFO:tasks.workunit.client.0.smithi113.stdout:0/567: dread d1/d43/f62 [4194304,4194304] 0 2024-04-12T05:09:01.055 INFO:tasks.workunit.client.0.smithi113.stdout:4/524: truncate da/d3c/d83/f40 8435379 0 2024-04-12T05:09:01.055 INFO:tasks.workunit.client.0.smithi113.stdout:0/568: dread - d1/d43/d59/f82 zero size 2024-04-12T05:09:01.055 INFO:tasks.workunit.client.1.smithi192.stdout:0/491: rmdir d0/d6/d15/d55/d29 39 2024-04-12T05:09:01.056 INFO:tasks.workunit.client.0.smithi113.stdout:5/460: symlink d0/d10/d27/d5f/l9b 0 2024-04-12T05:09:01.057 INFO:tasks.workunit.client.0.smithi113.stdout:0/569: chown d1/fa6 10079 1 2024-04-12T05:09:01.059 INFO:tasks.workunit.client.1.smithi192.stdout:9/315: dwrite d11/d15/d1e/f39 [0,4194304] 0 2024-04-12T05:09:01.059 INFO:tasks.workunit.client.0.smithi113.stdout:4/525: rename da/d26 to da/d24/d60/daa 0 2024-04-12T05:09:01.061 INFO:tasks.workunit.client.0.smithi113.stdout:5/461: mknod d0/d10/d27/d1d/d38/c9c 0 2024-04-12T05:09:01.063 INFO:tasks.workunit.client.0.smithi113.stdout:5/462: dread - d0/d10/d27/d1d/d38/f52 zero size 2024-04-12T05:09:01.066 INFO:tasks.workunit.client.1.smithi192.stdout:5/383: write d0/d15/d32/d45/f60 [1451703,126190] 0 2024-04-12T05:09:01.066 INFO:tasks.workunit.client.0.smithi113.stdout:5/463: chown d0/d10/d27/d1d/d8c/c8e 1041777 1 2024-04-12T05:09:01.066 INFO:tasks.workunit.client.1.smithi192.stdout:7/317: unlink d3/d7/d14/d1f/l1e 0 2024-04-12T05:09:01.069 INFO:tasks.workunit.client.0.smithi113.stdout:5/464: rmdir d0/d10/d27/d1d/d3f/d7e 39 2024-04-12T05:09:01.071 INFO:tasks.workunit.client.1.smithi192.stdout:0/492: mknod d0/d7b/ca8 0 2024-04-12T05:09:01.072 INFO:tasks.workunit.client.1.smithi192.stdout:6/457: write d3/dc/d39/f4e [4875520,88966] 0 2024-04-12T05:09:01.073 INFO:tasks.workunit.client.1.smithi192.stdout:8/327: creat d6/d10/d46/d5e/d5f/d37/d4f/d51/f63 x:0 0 0 2024-04-12T05:09:01.073 INFO:tasks.workunit.client.0.smithi113.stdout:5/465: read d0/d10/d27/d1d/d3f/d73/d89/f62 [509657,25072] 0 2024-04-12T05:09:01.074 INFO:tasks.workunit.client.1.smithi192.stdout:2/424: truncate d1/d8/d17/d19/d40/f5b 1129017 0 2024-04-12T05:09:01.075 INFO:tasks.workunit.client.0.smithi113.stdout:4/526: dwrite da/d3c/d83/f8a [0,4194304] 0 2024-04-12T05:09:01.076 INFO:tasks.workunit.client.1.smithi192.stdout:8/328: chown d6/d10/d46/d5e/d5f/d37/f4a 589663 1 2024-04-12T05:09:01.076 INFO:tasks.workunit.client.1.smithi192.stdout:3/336: getdents d5/d13/d3a/d4a 0 2024-04-12T05:09:01.079 INFO:tasks.workunit.client.1.smithi192.stdout:8/329: dread - d6/d10/d46/d5e/d5f/f57 zero size 2024-04-12T05:09:01.080 INFO:tasks.workunit.client.1.smithi192.stdout:7/318: creat d3/d1d/d4a/f70 x:0 0 0 2024-04-12T05:09:01.080 INFO:tasks.workunit.client.0.smithi113.stdout:0/570: dwrite d1/d9/f2b [0,4194304] 0 2024-04-12T05:09:01.081 INFO:tasks.workunit.client.1.smithi192.stdout:6/458: readlink d3/l1b 0 2024-04-12T05:09:01.083 INFO:tasks.workunit.client.0.smithi113.stdout:5/466: rmdir d0/d10/d27/d1d/d38/d3b/d6b/d71 0 2024-04-12T05:09:01.085 INFO:tasks.workunit.client.0.smithi113.stdout:0/571: rmdir d1/d43/d6b 0 2024-04-12T05:09:01.086 INFO:tasks.workunit.client.1.smithi192.stdout:6/459: dread - d3/d21/f7d zero size 2024-04-12T05:09:01.086 INFO:tasks.workunit.client.1.smithi192.stdout:0/493: creat d0/d6/d15/d55/d2e/fa9 x:0 0 0 2024-04-12T05:09:01.086 INFO:tasks.workunit.client.0.smithi113.stdout:0/572: stat d1/d43/l5e 0 2024-04-12T05:09:01.087 INFO:tasks.workunit.client.1.smithi192.stdout:3/337: mknod d5/d13/d3a/d4e/c6f 0 2024-04-12T05:09:01.088 INFO:tasks.workunit.client.0.smithi113.stdout:5/467: rmdir d0/d10/d27/d1d/d3f/d7e 39 2024-04-12T05:09:01.088 INFO:tasks.workunit.client.1.smithi192.stdout:5/384: getdents d0/d15/d24/d80 0 2024-04-12T05:09:01.090 INFO:tasks.workunit.client.0.smithi113.stdout:0/573: rename d1/d6/d16/la4 to d1/d43/d59/lb3 0 2024-04-12T05:09:01.090 INFO:tasks.workunit.client.0.smithi113.stdout:9/559: dread d1/d4/d5/f12 [4194304,4194304] 0 2024-04-12T05:09:01.091 INFO:tasks.workunit.client.0.smithi113.stdout:9/560: chown d1/d4/d30/l5e 31 1 2024-04-12T05:09:01.092 INFO:tasks.workunit.client.1.smithi192.stdout:9/316: dwrite d11/d15/d21/f2d [4194304,4194304] 0 2024-04-12T05:09:01.094 INFO:tasks.workunit.client.1.smithi192.stdout:7/319: truncate d3/d7/d15/d1a/f39 9172485 0 2024-04-12T05:09:01.095 INFO:tasks.workunit.client.1.smithi192.stdout:8/330: creat d6/d10/d46/d5e/d5f/d37/d4f/d4c/f64 x:0 0 0 2024-04-12T05:09:01.098 INFO:tasks.workunit.client.0.smithi113.stdout:9/561: truncate d1/df/d4b/d63/d3d/f5f 4622717 0 2024-04-12T05:09:01.102 INFO:tasks.workunit.client.1.smithi192.stdout:3/338: mkdir d5/d13/d3a/d70 0 2024-04-12T05:09:01.102 INFO:tasks.workunit.client.1.smithi192.stdout:4/473: dwrite d1/d54/f61 [0,4194304] 0 2024-04-12T05:09:01.104 INFO:tasks.workunit.client.1.smithi192.stdout:6/460: mknod d3/d8/c98 0 2024-04-12T05:09:01.104 INFO:tasks.workunit.client.1.smithi192.stdout:0/494: write d0/d6/d15/d55/d2e/f9e [823048,126252] 0 2024-04-12T05:09:01.104 INFO:tasks.workunit.client.1.smithi192.stdout:7/320: mknod d3/d1d/d4a/c71 0 2024-04-12T05:09:01.105 INFO:tasks.workunit.client.1.smithi192.stdout:8/331: unlink d6/f9 0 2024-04-12T05:09:01.105 INFO:tasks.workunit.client.0.smithi113.stdout:9/562: dwrite d1/d4/d5/d17/f74 [4194304,4194304] 0 2024-04-12T05:09:01.110 INFO:tasks.workunit.client.1.smithi192.stdout:9/317: dread d11/d15/d21/d2c/d1f/f4d [0,4194304] 0 2024-04-12T05:09:01.111 INFO:tasks.workunit.client.1.smithi192.stdout:9/318: stat d11/d25/d36 0 2024-04-12T05:09:01.114 INFO:tasks.workunit.client.1.smithi192.stdout:8/332: mkdir d6/d10/d46/d5e/d5f/d37/d4f/d51/d65 0 2024-04-12T05:09:01.118 INFO:tasks.workunit.client.1.smithi192.stdout:7/321: dread d3/d11/f46 [0,4194304] 0 2024-04-12T05:09:01.118 INFO:tasks.workunit.client.1.smithi192.stdout:8/333: chown d6/da/f49 58 1 2024-04-12T05:09:01.118 INFO:tasks.workunit.client.1.smithi192.stdout:4/474: mkdir d1/d54/d82/d71/d9e 0 2024-04-12T05:09:01.120 INFO:tasks.workunit.client.1.smithi192.stdout:6/461: mkdir d3/d8/d56/d59/d8a/d99 0 2024-04-12T05:09:01.120 INFO:tasks.workunit.client.1.smithi192.stdout:0/495: dwrite d0/d6/d15/d55/d2e/f42 [0,4194304] 0 2024-04-12T05:09:01.121 INFO:tasks.workunit.client.1.smithi192.stdout:7/322: dread - d3/f6f zero size 2024-04-12T05:09:01.121 INFO:tasks.workunit.client.1.smithi192.stdout:8/334: chown f5 11 1 2024-04-12T05:09:01.122 INFO:tasks.workunit.client.1.smithi192.stdout:4/475: readlink d1/d1d/d53/d1c/l3f 0 2024-04-12T05:09:01.123 INFO:tasks.workunit.client.0.smithi113.stdout:9/563: dread d1/d27/f5d [0,4194304] 0 2024-04-12T05:09:01.125 INFO:tasks.workunit.client.0.smithi113.stdout:0/574: dwrite d1/d6/f30 [0,4194304] 0 2024-04-12T05:09:01.128 INFO:tasks.workunit.client.1.smithi192.stdout:8/335: read d6/d10/f45 [8803,65082] 0 2024-04-12T05:09:01.130 INFO:tasks.workunit.client.0.smithi113.stdout:5/468: dread d0/f3a [0,4194304] 0 2024-04-12T05:09:01.132 INFO:tasks.workunit.client.0.smithi113.stdout:0/575: mknod d1/d6/d16/d2a/d50/d52/cb4 0 2024-04-12T05:09:01.134 INFO:tasks.workunit.client.1.smithi192.stdout:7/323: rename d3/d11/l35 to d3/d7/d15/d22/l72 0 2024-04-12T05:09:01.135 INFO:tasks.workunit.client.0.smithi113.stdout:5/469: unlink d0/d10/d27/d1d/d38/f51 0 2024-04-12T05:09:01.136 INFO:tasks.workunit.client.1.smithi192.stdout:4/476: dwrite d1/d49/f69 [0,4194304] 0 2024-04-12T05:09:01.137 INFO:tasks.workunit.client.0.smithi113.stdout:4/527: dread da/d73/f1c [0,4194304] 0 2024-04-12T05:09:01.138 INFO:tasks.workunit.client.0.smithi113.stdout:9/564: truncate d1/d27/f95 1143247 0 2024-04-12T05:09:01.141 INFO:tasks.workunit.client.1.smithi192.stdout:8/336: creat d6/d10/d46/d5e/d5f/d37/d4f/d51/f66 x:0 0 0 2024-04-12T05:09:01.141 INFO:tasks.workunit.client.0.smithi113.stdout:5/470: creat d0/d10/d27/d1d/d8c/f9d x:0 0 0 2024-04-12T05:09:01.144 INFO:tasks.workunit.client.0.smithi113.stdout:0/576: unlink d1/d80/caa 0 2024-04-12T05:09:01.145 INFO:tasks.workunit.client.1.smithi192.stdout:2/425: truncate d1/d8/d17/d19/d40/f5b 1942810 0 2024-04-12T05:09:01.145 INFO:tasks.workunit.client.1.smithi192.stdout:5/385: dread d0/d15/d3e/d77/f78 [0,4194304] 0 2024-04-12T05:09:01.147 INFO:tasks.workunit.client.0.smithi113.stdout:9/565: getdents d1/df/d4b/d63/d3d/d6f 0 2024-04-12T05:09:01.148 INFO:tasks.workunit.client.0.smithi113.stdout:4/528: dwrite da/d3c/f95 [0,4194304] 0 2024-04-12T05:09:01.149 INFO:tasks.workunit.client.1.smithi192.stdout:7/324: read d3/d7/d15/d1a/f3e [699420,68258] 0 2024-04-12T05:09:01.151 INFO:tasks.workunit.client.1.smithi192.stdout:7/325: write d3/d7/d15/f67 [678374,68521] 0 2024-04-12T05:09:01.151 INFO:tasks.workunit.client.1.smithi192.stdout:4/477: link d1/d54/d55/d8b/l8c d1/d54/d82/l9f 0 2024-04-12T05:09:01.151 INFO:tasks.workunit.client.1.smithi192.stdout:5/386: write d0/d15/d3e/d77/f78 [2894736,92467] 0 2024-04-12T05:09:01.152 INFO:tasks.workunit.client.1.smithi192.stdout:0/496: write d0/d6/d15/d1b/d3a/d4f/f24 [3701759,9603] 0 2024-04-12T05:09:01.154 INFO:tasks.workunit.client.1.smithi192.stdout:7/326: readlink d3/d7/l52 0 2024-04-12T05:09:01.154 INFO:tasks.workunit.client.0.smithi113.stdout:9/566: symlink d1/d4/d5/d6/d15/lc0 0 2024-04-12T05:09:01.154 INFO:tasks.workunit.client.0.smithi113.stdout:9/567: read - d1/d4/d38/f8b zero size 2024-04-12T05:09:01.155 INFO:tasks.workunit.client.1.smithi192.stdout:7/327: chown d3/d7/d15/f24 343844327 1 2024-04-12T05:09:01.158 INFO:tasks.workunit.client.0.smithi113.stdout:9/568: fdatasync d1/df/d4b/d63/d3d/f49 0 2024-04-12T05:09:01.158 INFO:tasks.workunit.client.0.smithi113.stdout:0/577: rmdir d1/d6/d16/d2a 39 2024-04-12T05:09:01.160 INFO:tasks.workunit.client.0.smithi113.stdout:9/569: read d1/d4/f77 [1634179,20242] 0 2024-04-12T05:09:01.160 INFO:tasks.workunit.client.1.smithi192.stdout:0/497: write d0/d6/d67/f8a [725685,43489] 0 2024-04-12T05:09:01.161 INFO:tasks.workunit.client.1.smithi192.stdout:6/462: dread d3/d8/f1c [0,4194304] 0 2024-04-12T05:09:01.162 INFO:tasks.workunit.client.1.smithi192.stdout:7/328: fsync d3/d7/d14/f47 0 2024-04-12T05:09:01.163 INFO:tasks.workunit.client.1.smithi192.stdout:7/329: chown d3/d1d/d48 6 1 2024-04-12T05:09:01.163 INFO:tasks.workunit.client.1.smithi192.stdout:4/478: fdatasync d1/f2 0 2024-04-12T05:09:01.165 INFO:tasks.workunit.client.1.smithi192.stdout:5/387: mknod d0/d15/d32/d69/d7d/c87 0 2024-04-12T05:09:01.167 INFO:tasks.workunit.client.0.smithi113.stdout:0/578: rename d1/d6/d16/d2a/d2c to d1/d6/d16/d2a/d50/d52/d7e/db5 0 2024-04-12T05:09:01.167 INFO:tasks.workunit.client.1.smithi192.stdout:2/426: getdents d1/d2d/d73 0 2024-04-12T05:09:01.168 INFO:tasks.workunit.client.0.smithi113.stdout:0/579: write d1/d6/da/d36/f74 [125953,38286] 0 2024-04-12T05:09:01.169 INFO:tasks.workunit.client.1.smithi192.stdout:0/498: dread - d0/d6/d15/d55/d29/d89/f93 zero size 2024-04-12T05:09:01.170 INFO:tasks.workunit.client.0.smithi113.stdout:9/570: rename d1/df/d4b/d63/l80 to d1/df/d4b/d63/d3d/d6f/lc1 0 2024-04-12T05:09:01.171 INFO:tasks.workunit.client.0.smithi113.stdout:0/580: read - d1/d6/d16/d2a/d50/f61 zero size 2024-04-12T05:09:01.172 INFO:tasks.workunit.client.1.smithi192.stdout:2/427: mknod d1/d8/d17/c8c 0 2024-04-12T05:09:01.176 INFO:tasks.workunit.client.1.smithi192.stdout:4/479: creat d1/d54/d82/d71/d9e/fa0 x:0 0 0 2024-04-12T05:09:01.178 INFO:tasks.workunit.client.0.smithi113.stdout:0/581: rename d1/d6/da to d1/d6/d16/d2a/d50/db6 0 2024-04-12T05:09:01.252 INFO:tasks.workunit.client.1.smithi192.stdout:7/330: link d3/d7/d15/c53 d3/c73 0 2024-04-12T05:09:01.253 INFO:tasks.workunit.client.1.smithi192.stdout:4/480: mkdir d1/d54/d82/da1 0 2024-04-12T05:09:01.254 INFO:tasks.workunit.client.1.smithi192.stdout:0/499: creat d0/d6/faa x:0 0 0 2024-04-12T05:09:01.254 INFO:tasks.workunit.client.1.smithi192.stdout:4/481: write d1/f26 [2464064,102554] 0 2024-04-12T05:09:01.255 INFO:tasks.workunit.client.1.smithi192.stdout:8/337: getdents d6/d10/d46/d5e/d5f/d37/d4f/d51 0 2024-04-12T05:09:01.255 INFO:tasks.workunit.client.1.smithi192.stdout:7/331: dread d3/d7/d15/d1a/f3e [0,4194304] 0 2024-04-12T05:09:01.257 INFO:tasks.workunit.client.1.smithi192.stdout:8/338: chown d6/d21/f23 115 1 2024-04-12T05:09:01.258 INFO:tasks.workunit.client.0.smithi113.stdout:9/571: write d1/d4/d5/d6/f3b [2052152,32923] 0 2024-04-12T05:09:01.259 INFO:tasks.workunit.client.1.smithi192.stdout:0/500: symlink d0/d7b/d9b/d4b/d72/lab 0 2024-04-12T05:09:01.261 INFO:tasks.workunit.client.0.smithi113.stdout:9/572: symlink d1/df/db6/lc2 0 2024-04-12T05:09:01.262 INFO:tasks.workunit.client.0.smithi113.stdout:5/471: dread d0/d10/d27/d1d/f21 [0,4194304] 0 2024-04-12T05:09:01.263 INFO:tasks.workunit.client.1.smithi192.stdout:8/339: truncate d6/d10/d46/d5e/d5f/d31/f39 4549600 0 2024-04-12T05:09:01.263 INFO:tasks.workunit.client.1.smithi192.stdout:8/340: chown d6 237117 1 2024-04-12T05:09:01.263 INFO:tasks.workunit.client.1.smithi192.stdout:5/388: write d0/f3d [2042727,117653] 0 2024-04-12T05:09:01.264 INFO:tasks.workunit.client.0.smithi113.stdout:5/472: fsync d0/d10/d27/d1d/d3f/d73/d89/f5a 0 2024-04-12T05:09:01.267 INFO:tasks.workunit.client.1.smithi192.stdout:4/482: getdents d1/d54/d55/d64/d68 0 2024-04-12T05:09:01.267 INFO:tasks.workunit.client.1.smithi192.stdout:7/332: dwrite d3/d1d/f2e [4194304,4194304] 0 2024-04-12T05:09:01.270 INFO:tasks.workunit.client.1.smithi192.stdout:5/389: unlink d0/d15/d32/d45/c7e 0 2024-04-12T05:09:01.271 INFO:tasks.workunit.client.0.smithi113.stdout:5/473: write d0/d10/d27/d5f/f65 [1140150,128132] 0 2024-04-12T05:09:01.273 INFO:tasks.workunit.client.1.smithi192.stdout:2/428: truncate d1/d8/f10 6645177 0 2024-04-12T05:09:01.275 INFO:tasks.workunit.client.0.smithi113.stdout:9/573: dwrite d1/f79 [0,4194304] 0 2024-04-12T05:09:01.276 INFO:tasks.workunit.client.0.smithi113.stdout:0/582: dwrite d1/d6/d16/d2a/d50/db6/f4d [4194304,4194304] 0 2024-04-12T05:09:01.276 INFO:tasks.workunit.client.0.smithi113.stdout:5/474: rmdir d0/d10/d27/d1d/d3f/d73 39 2024-04-12T05:09:01.279 INFO:tasks.workunit.client.0.smithi113.stdout:5/475: read d0/d10/d27/d1d/f49 [45604,17152] 0 2024-04-12T05:09:01.279 INFO:tasks.workunit.client.0.smithi113.stdout:9/574: creat d1/df/fc3 x:0 0 0 2024-04-12T05:09:01.279 INFO:tasks.workunit.client.0.smithi113.stdout:0/583: readlink d1/d43/d59/lb3 0 2024-04-12T05:09:01.282 INFO:tasks.workunit.client.0.smithi113.stdout:9/575: creat d1/d4/d6d/fc4 x:0 0 0 2024-04-12T05:09:01.283 INFO:tasks.workunit.client.0.smithi113.stdout:5/476: creat d0/d10/d27/d1d/d41/f9e x:0 0 0 2024-04-12T05:09:01.283 INFO:tasks.workunit.client.0.smithi113.stdout:9/576: read - d1/d4/d38/f8b zero size 2024-04-12T05:09:01.283 INFO:tasks.workunit.client.0.smithi113.stdout:0/584: rmdir d1/d6/d16/d2a/d50/d52/d7e 39 2024-04-12T05:09:01.284 INFO:tasks.workunit.client.0.smithi113.stdout:5/477: chown d0/l3e 28 1 2024-04-12T05:09:01.289 INFO:tasks.workunit.client.0.smithi113.stdout:9/577: getdents d1/d4/d38 0 2024-04-12T05:09:01.294 INFO:tasks.workunit.client.0.smithi113.stdout:5/478: dwrite d0/d10/d27/d1d/d38/d44/d66/f96 [0,4194304] 0 2024-04-12T05:09:01.294 INFO:tasks.workunit.client.0.smithi113.stdout:9/578: write d1/df/d4b/d63/d3d/f49 [2695545,44365] 0 2024-04-12T05:09:01.297 INFO:tasks.workunit.client.0.smithi113.stdout:5/479: readlink d0/d10/d27/d1d/d3f/d73/d89/l1a 0 2024-04-12T05:09:01.302 INFO:tasks.workunit.client.0.smithi113.stdout:5/480: dwrite d0/f2e [0,4194304] 0 2024-04-12T05:09:01.309 INFO:tasks.workunit.client.0.smithi113.stdout:0/585: dwrite d1/d6/d16/f24 [0,4194304] 0 2024-04-12T05:09:01.311 INFO:tasks.workunit.client.0.smithi113.stdout:0/586: truncate d1/d6/f91 373964 0 2024-04-12T05:09:01.343 INFO:tasks.workunit.client.1.smithi192.stdout:0/501: dwrite d0/d6/d15/d1b/d46/f49 [0,4194304] 0 2024-04-12T05:09:01.343 INFO:tasks.workunit.client.1.smithi192.stdout:4/483: getdents d1/d49 0 2024-04-12T05:09:01.352 INFO:tasks.workunit.client.1.smithi192.stdout:2/429: write d1/d8/d17/f5f [1681642,83831] 0 2024-04-12T05:09:01.353 INFO:tasks.workunit.client.1.smithi192.stdout:4/484: rename d1/l6f to d1/d6b/la2 0 2024-04-12T05:09:01.355 INFO:tasks.workunit.client.1.smithi192.stdout:8/341: dwrite d6/d10/d46/d5e/d5f/d37/d4f/d51/f66 [0,4194304] 0 2024-04-12T05:09:01.356 INFO:tasks.workunit.client.1.smithi192.stdout:7/333: dread d3/d7/f10 [0,4194304] 0 2024-04-12T05:09:01.358 INFO:tasks.workunit.client.1.smithi192.stdout:7/334: stat d3/d7/d14/l42 0 2024-04-12T05:09:01.358 INFO:tasks.workunit.client.1.smithi192.stdout:8/342: write f3 [4020844,87929] 0 2024-04-12T05:09:01.361 INFO:tasks.workunit.client.1.smithi192.stdout:7/335: truncate d3/d11/f5c 164566 0 2024-04-12T05:09:01.363 INFO:tasks.workunit.client.1.smithi192.stdout:4/485: dread d1/d54/f61 [0,4194304] 0 2024-04-12T05:09:01.363 INFO:tasks.workunit.client.1.smithi192.stdout:7/336: fsync d3/d7/f59 0 2024-04-12T05:09:01.364 INFO:tasks.workunit.client.1.smithi192.stdout:8/343: symlink d6/d10/l67 0 2024-04-12T05:09:01.364 INFO:tasks.workunit.client.1.smithi192.stdout:5/390: dread d0/f7 [0,4194304] 0 2024-04-12T05:09:01.366 INFO:tasks.workunit.client.1.smithi192.stdout:7/337: chown d3/d11/f46 1325 1 2024-04-12T05:09:01.366 INFO:tasks.workunit.client.1.smithi192.stdout:5/391: stat d0/f7 0 2024-04-12T05:09:01.369 INFO:tasks.workunit.client.1.smithi192.stdout:5/392: write d0/d15/d24/d29/f2a [2013845,41320] 0 2024-04-12T05:09:01.370 INFO:tasks.workunit.client.1.smithi192.stdout:7/338: mkdir d3/d1d/d74 0 2024-04-12T05:09:01.370 INFO:tasks.workunit.client.1.smithi192.stdout:5/393: chown d0/d15/d24/d29/c50 89074 1 2024-04-12T05:09:01.372 INFO:tasks.workunit.client.1.smithi192.stdout:0/502: dread d0/f16 [0,4194304] 0 2024-04-12T05:09:01.373 INFO:tasks.workunit.client.1.smithi192.stdout:7/339: dread d3/d7/d15/d22/f4d [0,4194304] 0 2024-04-12T05:09:01.375 INFO:tasks.workunit.client.1.smithi192.stdout:5/394: creat d0/d15/d24/d47/d7b/f88 x:0 0 0 2024-04-12T05:09:01.376 INFO:tasks.workunit.client.1.smithi192.stdout:8/344: truncate d6/f18 990925 0 2024-04-12T05:09:01.378 INFO:tasks.workunit.client.1.smithi192.stdout:7/340: rename d3/d7/f59 to d3/d1d/d74/f75 0 2024-04-12T05:09:01.379 INFO:tasks.workunit.client.1.smithi192.stdout:5/395: fdatasync d0/d15/f41 0 2024-04-12T05:09:01.379 INFO:tasks.workunit.client.1.smithi192.stdout:0/503: symlink d0/d6/d15/d55/lac 0 2024-04-12T05:09:01.381 INFO:tasks.workunit.client.1.smithi192.stdout:5/396: chown d0/d15/f41 12182333 1 2024-04-12T05:09:01.382 INFO:tasks.workunit.client.1.smithi192.stdout:2/430: dread d1/d8/f46 [0,4194304] 0 2024-04-12T05:09:01.382 INFO:tasks.workunit.client.1.smithi192.stdout:7/341: mkdir d3/d7/d14/d1f/d76 0 2024-04-12T05:09:01.383 INFO:tasks.workunit.client.1.smithi192.stdout:0/504: read d0/d6/d15/d55/d29/f2b [504528,61528] 0 2024-04-12T05:09:01.383 INFO:tasks.workunit.client.1.smithi192.stdout:5/397: stat d0/d15/d64/c6d 0 2024-04-12T05:09:01.385 INFO:tasks.workunit.client.1.smithi192.stdout:0/505: readlink d0/d6/d15/l8b 0 2024-04-12T05:09:01.386 INFO:tasks.workunit.client.1.smithi192.stdout:2/431: creat d1/d2d/d73/f8d x:0 0 0 2024-04-12T05:09:01.387 INFO:tasks.workunit.client.1.smithi192.stdout:4/486: dread d1/d54/f94 [0,4194304] 0 2024-04-12T05:09:01.389 INFO:tasks.workunit.client.1.smithi192.stdout:4/487: write d1/d54/d82/f62 [4808259,130761] 0 2024-04-12T05:09:01.393 INFO:tasks.workunit.client.1.smithi192.stdout:8/345: truncate d6/d10/d46/d5e/d5f/d37/d4f/d51/f66 3222101 0 2024-04-12T05:09:01.394 INFO:tasks.workunit.client.1.smithi192.stdout:5/398: rmdir d0/d15/d24/d80 0 2024-04-12T05:09:01.395 INFO:tasks.workunit.client.1.smithi192.stdout:8/346: chown f5 62521 1 2024-04-12T05:09:01.395 INFO:tasks.workunit.client.1.smithi192.stdout:2/432: dwrite d1/d8/d3a/de/f85 [0,4194304] 0 2024-04-12T05:09:01.398 INFO:tasks.workunit.client.1.smithi192.stdout:5/399: write d0/f61 [727854,62411] 0 2024-04-12T05:09:01.399 INFO:tasks.workunit.client.1.smithi192.stdout:7/342: link d3/d7/f8 d3/d7/d15/f77 0 2024-04-12T05:09:01.399 INFO:tasks.workunit.client.1.smithi192.stdout:8/347: chown d6/d1f/f5c 4697 1 2024-04-12T05:09:01.401 INFO:tasks.workunit.client.1.smithi192.stdout:0/506: creat d0/fad x:0 0 0 2024-04-12T05:09:01.404 INFO:tasks.workunit.client.1.smithi192.stdout:4/488: dwrite d1/d1d/d53/f57 [0,4194304] 0 2024-04-12T05:09:01.405 INFO:tasks.workunit.client.1.smithi192.stdout:5/400: mkdir d0/d15/d32/d69/d7d/d89 0 2024-04-12T05:09:01.406 INFO:tasks.workunit.client.1.smithi192.stdout:0/507: write d0/d4e/d53/d6d/f62 [4132354,68418] 0 2024-04-12T05:09:01.406 INFO:tasks.workunit.client.1.smithi192.stdout:7/343: symlink d3/d1d/d4a/l78 0 2024-04-12T05:09:01.408 INFO:tasks.workunit.client.1.smithi192.stdout:5/401: chown d0/d15/d24/f51 152 1 2024-04-12T05:09:01.412 INFO:tasks.workunit.client.1.smithi192.stdout:0/508: symlink d0/d7b/d7c/lae 0 2024-04-12T05:09:01.412 INFO:tasks.workunit.client.1.smithi192.stdout:2/433: dwrite d1/d8/d3a/de/f11 [0,4194304] 0 2024-04-12T05:09:01.414 INFO:tasks.workunit.client.1.smithi192.stdout:7/344: rename d3/d1d/d5a/d5f/c4c to d3/d7/c79 0 2024-04-12T05:09:01.414 INFO:tasks.workunit.client.1.smithi192.stdout:5/402: write d0/d21/f36 [1589689,25396] 0 2024-04-12T05:09:01.415 INFO:tasks.workunit.client.1.smithi192.stdout:4/489: dread d1/d54/f94 [0,4194304] 0 2024-04-12T05:09:01.419 INFO:tasks.workunit.client.1.smithi192.stdout:8/348: truncate d6/d21/f33 984431 0 2024-04-12T05:09:01.420 INFO:tasks.workunit.client.1.smithi192.stdout:7/345: mkdir d3/d1d/d5a/d5f/d7a 0 2024-04-12T05:09:01.420 INFO:tasks.workunit.client.1.smithi192.stdout:0/509: symlink d0/d6/d15/d19/laf 0 2024-04-12T05:09:01.422 INFO:tasks.workunit.client.1.smithi192.stdout:2/434: creat d1/d8/f8e x:0 0 0 2024-04-12T05:09:01.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:08:59.972361+0000 mon.smithi113 (mon.0) 672 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:09:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:00.909304+0000 mon.smithi192 (mon.1) 49 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:09:01.427 INFO:tasks.workunit.client.1.smithi192.stdout:8/349: chown d6/d10/d46/d5e/d5f/f43 2 1 2024-04-12T05:09:01.429 INFO:tasks.workunit.client.1.smithi192.stdout:5/403: dwrite d0/d15/d32/d45/f65 [0,4194304] 0 2024-04-12T05:09:01.432 INFO:tasks.workunit.client.1.smithi192.stdout:0/510: write d0/d4e/f87 [326913,38207] 0 2024-04-12T05:09:01.433 INFO:tasks.workunit.client.1.smithi192.stdout:4/490: dwrite d1/d54/d82/d71/f7b [0,4194304] 0 2024-04-12T05:09:01.438 INFO:tasks.workunit.client.1.smithi192.stdout:4/491: read - d1/d54/f99 zero size 2024-04-12T05:09:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:08:59.972361+0000 mon.smithi113 (mon.0) 672 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:09:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:00.909304+0000 mon.smithi192 (mon.1) 49 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:09:01.467 INFO:tasks.workunit.client.1.smithi192.stdout:5/404: mknod d0/d15/d64/c8a 0 2024-04-12T05:09:01.470 INFO:tasks.workunit.client.0.smithi113.stdout:7/528: sync 2024-04-12T05:09:01.470 INFO:tasks.workunit.client.0.smithi113.stdout:3/506: sync 2024-04-12T05:09:01.475 INFO:tasks.workunit.client.0.smithi113.stdout:3/507: mkdir d0/d13/d15/d28/d2a/d8b/d91/dad 0 2024-04-12T05:09:01.477 INFO:tasks.workunit.client.0.smithi113.stdout:3/508: chown d0/d13/d15/d28/d2a/c8c 1 1 2024-04-12T05:09:01.477 INFO:tasks.workunit.client.1.smithi192.stdout:2/435: rename d1/d8/d17/d19/d60/c62 to d1/d8/d17/d19/d71/d7f/c8f 0 2024-04-12T05:09:01.477 INFO:tasks.workunit.client.1.smithi192.stdout:8/350: dwrite d6/d10/f45 [0,4194304] 0 2024-04-12T05:09:01.479 INFO:tasks.workunit.client.1.smithi192.stdout:4/492: creat d1/d1d/d53/d1c/d84/fa3 x:0 0 0 2024-04-12T05:09:01.480 INFO:tasks.workunit.client.0.smithi113.stdout:3/509: mknod d0/d7e/cae 0 2024-04-12T05:09:01.481 INFO:tasks.workunit.client.1.smithi192.stdout:4/493: dread - d1/d1d/d53/d1c/d84/fa3 zero size 2024-04-12T05:09:01.483 INFO:tasks.workunit.client.0.smithi113.stdout:7/529: read d0/dc/d6/d78/daa/fa6 [693340,70912] 0 2024-04-12T05:09:01.484 INFO:tasks.workunit.client.1.smithi192.stdout:0/511: getdents d0/d6/d15/d55/d29 0 2024-04-12T05:09:01.485 INFO:tasks.workunit.client.0.smithi113.stdout:3/510: creat d0/d13/d15/faf x:0 0 0 2024-04-12T05:09:01.485 INFO:tasks.workunit.client.1.smithi192.stdout:4/494: stat d1/d1d/d53/d1c/f66 0 2024-04-12T05:09:01.486 INFO:tasks.workunit.client.0.smithi113.stdout:3/511: truncate d0/fa 2348093 0 2024-04-12T05:09:01.486 INFO:tasks.workunit.client.1.smithi192.stdout:8/351: creat d6/d10/d46/d5e/d5f/d37/f68 x:0 0 0 2024-04-12T05:09:01.488 INFO:tasks.workunit.client.0.smithi113.stdout:7/530: mkdir d0/dc/da9/d5c/d9a/db6 0 2024-04-12T05:09:01.489 INFO:tasks.workunit.client.1.smithi192.stdout:0/512: mkdir d0/d4e/db0 0 2024-04-12T05:09:01.489 INFO:tasks.workunit.client.1.smithi192.stdout:8/352: symlink d6/d4b/l69 0 2024-04-12T05:09:01.491 INFO:tasks.workunit.client.0.smithi113.stdout:7/531: symlink d0/dc/da9/d5c/d9a/lb7 0 2024-04-12T05:09:01.492 INFO:tasks.workunit.client.1.smithi192.stdout:5/405: dread d0/f1b [0,4194304] 0 2024-04-12T05:09:01.493 INFO:tasks.workunit.client.0.smithi113.stdout:3/512: mknod d0/d13/d47/cb0 0 2024-04-12T05:09:01.493 INFO:tasks.workunit.client.1.smithi192.stdout:0/513: mkdir d0/d6/d67/db1 0 2024-04-12T05:09:01.496 INFO:tasks.workunit.client.1.smithi192.stdout:2/436: link d1/d8/f8e d1/d8/d17/d19/d60/d59/f90 0 2024-04-12T05:09:01.497 INFO:tasks.workunit.client.0.smithi113.stdout:7/532: creat d0/dc/d6/d1a/fb8 x:0 0 0 2024-04-12T05:09:01.499 INFO:tasks.workunit.client.0.smithi113.stdout:6/514: sync 2024-04-12T05:09:01.499 INFO:tasks.workunit.client.0.smithi113.stdout:8/591: sync 2024-04-12T05:09:01.499 INFO:tasks.workunit.client.0.smithi113.stdout:0/587: sync 2024-04-12T05:09:01.499 INFO:tasks.workunit.client.0.smithi113.stdout:2/447: sync 2024-04-12T05:09:01.499 INFO:tasks.workunit.client.0.smithi113.stdout:9/579: sync 2024-04-12T05:09:01.499 INFO:tasks.workunit.client.0.smithi113.stdout:4/529: sync 2024-04-12T05:09:01.499 INFO:tasks.workunit.client.0.smithi113.stdout:1/623: sync 2024-04-12T05:09:01.500 INFO:tasks.workunit.client.1.smithi192.stdout:2/437: stat d1/d2d/l58 0 2024-04-12T05:09:01.500 INFO:tasks.workunit.client.0.smithi113.stdout:5/481: sync 2024-04-12T05:09:01.500 INFO:tasks.workunit.client.0.smithi113.stdout:7/533: read d0/dd/fa5 [1719941,66850] 0 2024-04-12T05:09:01.502 INFO:tasks.workunit.client.0.smithi113.stdout:8/592: write d9/d1b/f56 [5019974,24744] 0 2024-04-12T05:09:01.503 INFO:tasks.workunit.client.0.smithi113.stdout:6/515: write d8/d1d/fac [1117606,96398] 0 2024-04-12T05:09:01.505 INFO:tasks.workunit.client.0.smithi113.stdout:7/534: readlink d0/dc/d6/d78/l4d 0 2024-04-12T05:09:01.505 INFO:tasks.workunit.client.0.smithi113.stdout:2/448: mknod d8/d9/c83 0 2024-04-12T05:09:01.507 INFO:tasks.workunit.client.0.smithi113.stdout:2/449: read - d8/d9/de/d15/d24/f7e zero size 2024-04-12T05:09:01.509 INFO:tasks.workunit.client.0.smithi113.stdout:6/516: write d8/de/d13/d11/f43 [3744780,92222] 0 2024-04-12T05:09:01.509 INFO:tasks.workunit.client.0.smithi113.stdout:4/530: unlink da/d24/d7e/d79/c97 0 2024-04-12T05:09:01.510 INFO:tasks.workunit.client.1.smithi192.stdout:2/438: symlink d1/d8/d17/d19/d71/d7f/l91 0 2024-04-12T05:09:01.513 INFO:tasks.workunit.client.1.smithi192.stdout:2/439: fsync d1/d8/d17/d19/d60/f76 0 2024-04-12T05:09:01.517 INFO:tasks.workunit.client.0.smithi113.stdout:7/535: creat d0/dc/da9/d5c/fb9 x:0 0 0 2024-04-12T05:09:01.519 INFO:tasks.workunit.client.0.smithi113.stdout:4/531: chown da/d70/d80/l9b 76 1 2024-04-12T05:09:01.520 INFO:tasks.workunit.client.0.smithi113.stdout:9/580: rename d1/df/d4b/d63/l5b to d1/d4/d38/d65/lc5 0 2024-04-12T05:09:01.520 INFO:tasks.workunit.client.0.smithi113.stdout:5/482: dwrite d0/d10/d27/d1d/d3f/d73/d89/f40 [4194304,4194304] 0 2024-04-12T05:09:01.522 INFO:tasks.workunit.client.0.smithi113.stdout:1/624: getdents d10/da5 0 2024-04-12T05:09:01.522 INFO:tasks.workunit.client.1.smithi192.stdout:2/440: mknod d1/d8/d3a/de/c92 0 2024-04-12T05:09:01.523 INFO:tasks.workunit.client.0.smithi113.stdout:7/536: mknod d0/dc/da9/d5c/cba 0 2024-04-12T05:09:01.523 INFO:tasks.workunit.client.0.smithi113.stdout:5/483: chown d0/d10/d27/d1d/d38/d3b/l72 513626 1 2024-04-12T05:09:01.523 INFO:tasks.workunit.client.0.smithi113.stdout:0/588: truncate d1/d6/d16/d2a/d50/d52/d7e/db5/f37 480317 0 2024-04-12T05:09:01.524 INFO:tasks.workunit.client.0.smithi113.stdout:0/589: chown d1/d43/d59 1526881 1 2024-04-12T05:09:01.525 INFO:tasks.workunit.client.0.smithi113.stdout:5/484: dread - d0/d10/d27/d1d/d3f/d73/d89/f5a zero size 2024-04-12T05:09:01.527 INFO:tasks.workunit.client.1.smithi192.stdout:2/441: creat d1/d8/d17/d19/d60/f93 x:0 0 0 2024-04-12T05:09:01.529 INFO:tasks.workunit.client.0.smithi113.stdout:6/517: rename d8/de/d13/d11/d36/d3d/d4c/d26/d5b/l92 to d8/de/d13/d11/d36/d3d/d4c/d31/lbb 0 2024-04-12T05:09:01.530 INFO:tasks.workunit.client.0.smithi113.stdout:6/518: chown d8/de/d13/d11/d36/d89 399233771 1 2024-04-12T05:09:01.531 INFO:tasks.workunit.client.0.smithi113.stdout:0/590: symlink d1/d6/d16/d2a/d50/d52/lb7 0 2024-04-12T05:09:01.531 INFO:tasks.workunit.client.0.smithi113.stdout:8/593: link d9/d1b/d8e/d93/d32/d5b/fb8 d9/d1b/d8e/d93/d32/d48/d8d/d4a/fbd 0 2024-04-12T05:09:01.533 INFO:tasks.workunit.client.1.smithi192.stdout:7/346: write d3/d7/d15/f24 [1245167,20384] 0 2024-04-12T05:09:01.534 INFO:tasks.workunit.client.0.smithi113.stdout:0/591: truncate d1/d6/d16/d2a/d50/f61 726450 0 2024-04-12T05:09:01.535 INFO:tasks.workunit.client.0.smithi113.stdout:5/485: unlink d0/d10/d27/d1d/d3f/d73/d89/f6d 0 2024-04-12T05:09:01.535 INFO:tasks.workunit.client.0.smithi113.stdout:0/592: write d1/d6/d16/d2a/d50/f61 [24003,17874] 0 2024-04-12T05:09:01.537 INFO:tasks.workunit.client.0.smithi113.stdout:3/513: getdents d0/d7e 0 2024-04-12T05:09:01.537 INFO:tasks.workunit.client.1.smithi192.stdout:7/347: mkdir d3/d1d/d48/d7b 0 2024-04-12T05:09:01.539 INFO:tasks.workunit.client.0.smithi113.stdout:2/450: rename d8/d9/de/d15/d24/f3e to d8/d9/de/d15/d24/d58/f84 0 2024-04-12T05:09:01.541 INFO:tasks.workunit.client.0.smithi113.stdout:7/537: mkdir d0/dd/d2d/dbb 0 2024-04-12T05:09:01.542 INFO:tasks.workunit.client.0.smithi113.stdout:5/486: dread d0/d10/d27/d1d/d38/d44/d66/f96 [0,4194304] 0 2024-04-12T05:09:01.543 INFO:tasks.workunit.client.1.smithi192.stdout:7/348: mknod d3/d7/d60/c7c 0 2024-04-12T05:09:01.544 INFO:tasks.workunit.client.1.smithi192.stdout:7/349: truncate d3/d7/d15/f77 1384503 0 2024-04-12T05:09:01.544 INFO:tasks.workunit.client.0.smithi113.stdout:0/593: creat d1/d6/d16/fb8 x:0 0 0 2024-04-12T05:09:01.545 INFO:tasks.workunit.client.0.smithi113.stdout:0/594: stat d1/fa6 0 2024-04-12T05:09:01.545 INFO:tasks.workunit.client.0.smithi113.stdout:8/594: mknod d9/d1b/d8e/d93/d32/cbe 0 2024-04-12T05:09:01.546 INFO:tasks.workunit.client.1.smithi192.stdout:7/350: fdatasync d3/d7/d15/f29 0 2024-04-12T05:09:01.549 INFO:tasks.workunit.client.1.smithi192.stdout:7/351: chown d3/c73 232224889 1 2024-04-12T05:09:01.551 INFO:tasks.workunit.client.1.smithi192.stdout:8/353: dread d6/d21/f33 [0,4194304] 0 2024-04-12T05:09:01.552 INFO:tasks.workunit.client.0.smithi113.stdout:4/532: rename da/d24/d7e/d79/fa8 to da/d5a/fab 0 2024-04-12T05:09:01.554 INFO:tasks.workunit.client.0.smithi113.stdout:2/451: creat d8/d9/d5b/f85 x:0 0 0 2024-04-12T05:09:01.555 INFO:tasks.workunit.client.0.smithi113.stdout:5/487: chown d0/d10/d27/d1d/d3f/d7e 3 1 2024-04-12T05:09:01.555 INFO:tasks.workunit.client.1.smithi192.stdout:8/354: getdents d6/d1f 0 2024-04-12T05:09:01.556 INFO:tasks.workunit.client.0.smithi113.stdout:8/595: symlink d9/d16/d65/db1/lbf 0 2024-04-12T05:09:01.559 INFO:tasks.workunit.client.0.smithi113.stdout:3/514: symlink d0/d13/lb1 0 2024-04-12T05:09:01.560 INFO:tasks.workunit.client.0.smithi113.stdout:9/581: dwrite d1/d27/f5d [0,4194304] 0 2024-04-12T05:09:01.561 INFO:tasks.workunit.client.1.smithi192.stdout:4/495: rename d1/d76 to d1/d54/d82/da4 0 2024-04-12T05:09:01.561 INFO:tasks.workunit.client.1.smithi192.stdout:8/355: dwrite d6/d21/f60 [0,4194304] 0 2024-04-12T05:09:01.562 INFO:tasks.workunit.client.0.smithi113.stdout:3/515: chown d0/d13/d15/d28/d2a/d34/d5f/d7b 19593854 1 2024-04-12T05:09:01.562 INFO:tasks.workunit.client.0.smithi113.stdout:6/519: rename d8/de/d13/d2b/l5a to d8/de/d13/d11/d36/d3d/d4c/d26/lbc 0 2024-04-12T05:09:01.565 INFO:tasks.workunit.client.0.smithi113.stdout:5/488: truncate d0/d10/d6e/f3c 994762 0 2024-04-12T05:09:01.565 INFO:tasks.workunit.client.1.smithi192.stdout:5/406: mkdir d0/d8b 0 2024-04-12T05:09:01.566 INFO:tasks.workunit.client.1.smithi192.stdout:5/407: chown d0/l7a 1 1 2024-04-12T05:09:01.566 INFO:tasks.workunit.client.1.smithi192.stdout:4/496: mknod d1/d1d/d53/d1c/d95/ca5 0 2024-04-12T05:09:01.568 INFO:tasks.workunit.client.1.smithi192.stdout:7/352: creat d3/d7/d15/f7d x:0 0 0 2024-04-12T05:09:01.570 INFO:tasks.workunit.client.1.smithi192.stdout:7/353: stat d3/c73 0 2024-04-12T05:09:01.570 INFO:tasks.workunit.client.0.smithi113.stdout:3/516: unlink d0/ca7 0 2024-04-12T05:09:01.572 INFO:tasks.workunit.client.0.smithi113.stdout:8/596: dread d9/d16/d65/d8c/fa4 [0,4194304] 0 2024-04-12T05:09:01.573 INFO:tasks.workunit.client.0.smithi113.stdout:7/538: rename d0/dc/da9/d5c/f79 to d0/d52/fbc 0 2024-04-12T05:09:01.576 INFO:tasks.workunit.client.1.smithi192.stdout:0/514: mkdir d0/d6/db2 0 2024-04-12T05:09:01.577 INFO:tasks.workunit.client.0.smithi113.stdout:0/595: rename d1/d6/d8e/f9b to d1/d6/d16/d8f/fb9 0 2024-04-12T05:09:01.579 INFO:tasks.workunit.client.0.smithi113.stdout:6/520: fdatasync d8/f1a 0 2024-04-12T05:09:01.580 INFO:tasks.workunit.client.0.smithi113.stdout:3/517: rmdir d0/d13/d15/d28/d2a/d34 39 2024-04-12T05:09:01.581 INFO:tasks.workunit.client.1.smithi192.stdout:7/354: dwrite d3/d7/f56 [0,4194304] 0 2024-04-12T05:09:01.582 INFO:tasks.workunit.client.1.smithi192.stdout:5/408: creat d0/d15/f8c x:0 0 0 2024-04-12T05:09:01.583 INFO:tasks.workunit.client.0.smithi113.stdout:8/597: write d9/d16/d7f/d3a/f5e [564839,38254] 0 2024-04-12T05:09:01.585 INFO:tasks.workunit.client.0.smithi113.stdout:9/582: creat d1/df/d4b/d63/dad/fc6 x:0 0 0 2024-04-12T05:09:01.588 INFO:tasks.workunit.client.0.smithi113.stdout:3/518: write d0/d13/d15/d28/d2a/f98 [398514,53856] 0 2024-04-12T05:09:01.588 INFO:tasks.workunit.client.0.smithi113.stdout:7/539: creat d0/dc/da9/d5c/d9a/d9e/fbd x:0 0 0 2024-04-12T05:09:01.591 INFO:tasks.workunit.client.1.smithi192.stdout:7/355: creat d3/d7/f7e x:0 0 0 2024-04-12T05:09:01.591 INFO:tasks.workunit.client.0.smithi113.stdout:1/625: dwrite d10/d21/f2b [0,4194304] 0 2024-04-12T05:09:01.591 INFO:tasks.workunit.client.1.smithi192.stdout:2/442: dwrite d1/d8/d3a/f26 [0,4194304] 0 2024-04-12T05:09:01.593 INFO:tasks.workunit.client.1.smithi192.stdout:2/443: chown d1/d8/d17/f2a 129 1 2024-04-12T05:09:01.593 INFO:tasks.workunit.client.0.smithi113.stdout:6/521: dwrite d8/de/f94 [0,4194304] 0 2024-04-12T05:09:01.594 INFO:tasks.workunit.client.0.smithi113.stdout:8/598: mknod d9/d1b/d8e/d93/d92/cc0 0 2024-04-12T05:09:01.596 INFO:tasks.workunit.client.1.smithi192.stdout:9/319: sync 2024-04-12T05:09:01.599 INFO:tasks.workunit.client.1.smithi192.stdout:9/320: fdatasync d11/d37/d33/f55 0 2024-04-12T05:09:01.599 INFO:tasks.workunit.client.1.smithi192.stdout:9/321: readlink d11/l27 0 2024-04-12T05:09:01.600 INFO:tasks.workunit.client.0.smithi113.stdout:9/583: mkdir d1/df/d4b/d63/d3d/d6f/dba/dc7 0 2024-04-12T05:09:01.601 INFO:tasks.workunit.client.1.smithi192.stdout:9/322: write f10 [1940162,43843] 0 2024-04-12T05:09:01.602 INFO:tasks.workunit.client.1.smithi192.stdout:9/323: dread - d11/d15/d21/f65 zero size 2024-04-12T05:09:01.603 INFO:tasks.workunit.client.0.smithi113.stdout:8/599: unlink d9/d16/c35 0 2024-04-12T05:09:01.608 INFO:tasks.workunit.client.0.smithi113.stdout:8/600: write d9/fb [1137558,51413] 0 2024-04-12T05:09:01.609 INFO:tasks.workunit.client.0.smithi113.stdout:6/522: dwrite d8/de/d13/d11/d36/d3d/d4c/d26/f51 [0,4194304] 0 2024-04-12T05:09:01.614 INFO:tasks.workunit.client.0.smithi113.stdout:5/489: dwrite d0/d10/d6e/f35 [0,4194304] 0 2024-04-12T05:09:01.622 INFO:tasks.workunit.client.0.smithi113.stdout:7/540: dwrite d0/dc/d6/d9b/f62 [0,4194304] 0 2024-04-12T05:09:01.624 INFO:tasks.workunit.client.0.smithi113.stdout:8/601: dwrite d9/d1b/d8e/d93/d32/d48/f49 [0,4194304] 0 2024-04-12T05:09:01.626 INFO:tasks.workunit.client.0.smithi113.stdout:5/490: dread d0/d10/d27/d1d/f49 [0,4194304] 0 2024-04-12T05:09:01.655 INFO:tasks.workunit.client.1.smithi192.stdout:0/515: mknod d0/d6/cb3 0 2024-04-12T05:09:01.657 INFO:tasks.workunit.client.1.smithi192.stdout:0/516: chown d0/d7b/d9b/d4b/d72/lab 6474 1 2024-04-12T05:09:01.658 INFO:tasks.workunit.client.1.smithi192.stdout:0/517: fdatasync d0/d6/d15/d55/d2e/fa9 0 2024-04-12T05:09:01.658 INFO:tasks.workunit.client.0.smithi113.stdout:7/541: truncate d0/dc/d50/f72 782059 0 2024-04-12T05:09:01.659 INFO:tasks.workunit.client.0.smithi113.stdout:3/519: dread d0/f57 [0,4194304] 0 2024-04-12T05:09:01.659 INFO:tasks.workunit.client.1.smithi192.stdout:5/409: symlink d0/d21/l8d 0 2024-04-12T05:09:01.662 INFO:tasks.workunit.client.0.smithi113.stdout:5/491: dread d0/d10/d6e/f3c [0,4194304] 0 2024-04-12T05:09:01.662 INFO:tasks.workunit.client.0.smithi113.stdout:1/626: fsync d10/d41/d89/d96/d7d/f7f 0 2024-04-12T05:09:01.665 INFO:tasks.workunit.client.0.smithi113.stdout:9/584: dread d1/f26 [0,4194304] 0 2024-04-12T05:09:01.667 INFO:tasks.workunit.client.0.smithi113.stdout:8/602: getdents d9/d16/d65/db1/d59 0 2024-04-12T05:09:01.668 INFO:tasks.workunit.client.1.smithi192.stdout:4/497: rename d1/d54/d55/f72 to d1/d54/d55/fa6 0 2024-04-12T05:09:01.668 INFO:tasks.workunit.client.0.smithi113.stdout:9/585: fsync d1/d4/d5/d6/f76 0 2024-04-12T05:09:01.669 INFO:tasks.workunit.client.1.smithi192.stdout:4/498: chown d1/f29 189 1 2024-04-12T05:09:01.669 INFO:tasks.workunit.client.0.smithi113.stdout:3/520: creat d0/d13/d47/fb2 x:0 0 0 2024-04-12T05:09:01.670 INFO:tasks.workunit.client.1.smithi192.stdout:9/324: write d11/d25/d36/f38 [43876,16334] 0 2024-04-12T05:09:01.673 INFO:tasks.workunit.client.0.smithi113.stdout:3/521: creat d0/d7e/fb3 x:0 0 0 2024-04-12T05:09:01.674 INFO:tasks.workunit.client.0.smithi113.stdout:1/627: unlink d10/d2e/d31/d33/l43 0 2024-04-12T05:09:01.676 INFO:tasks.workunit.client.0.smithi113.stdout:8/603: mkdir d9/d1b/d8e/d93/d32/d48/dc1 0 2024-04-12T05:09:01.680 INFO:tasks.workunit.client.0.smithi113.stdout:8/604: mknod d9/d1b/d8e/d93/d92/cc2 0 2024-04-12T05:09:01.682 INFO:tasks.workunit.client.0.smithi113.stdout:8/605: write d9/d1b/f61 [145162,103365] 0 2024-04-12T05:09:01.698 INFO:tasks.workunit.client.1.smithi192.stdout:0/518: mkdir d0/d4e/d95/db4 0 2024-04-12T05:09:01.698 INFO:tasks.workunit.client.1.smithi192.stdout:7/356: dwrite d3/d7/f8 [0,4194304] 0 2024-04-12T05:09:01.700 INFO:tasks.workunit.client.1.smithi192.stdout:2/444: mknod d1/d2d/c94 0 2024-04-12T05:09:01.700 INFO:tasks.workunit.client.1.smithi192.stdout:4/499: symlink d1/d1d/d53/la7 0 2024-04-12T05:09:01.701 INFO:tasks.workunit.client.1.smithi192.stdout:2/445: chown d1/f4 16274269 1 2024-04-12T05:09:01.703 INFO:tasks.workunit.client.1.smithi192.stdout:0/519: dread d0/d4e/f87 [0,4194304] 0 2024-04-12T05:09:01.706 INFO:tasks.workunit.client.1.smithi192.stdout:4/500: mkdir d1/d54/d82/da4/da8 0 2024-04-12T05:09:01.706 INFO:tasks.workunit.client.1.smithi192.stdout:5/410: rmdir d0/d15/d32/d69/d7d/d89 0 2024-04-12T05:09:01.707 INFO:tasks.workunit.client.1.smithi192.stdout:9/325: unlink d11/d15/d1e/l49 0 2024-04-12T05:09:01.707 INFO:tasks.workunit.client.1.smithi192.stdout:2/446: rmdir d1/d8/d17/d19/d60/d59 39 2024-04-12T05:09:01.710 INFO:tasks.workunit.client.1.smithi192.stdout:9/326: chown d11/d25/d36/c58 115842625 1 2024-04-12T05:09:01.712 INFO:tasks.workunit.client.1.smithi192.stdout:5/411: rename d0/f20 to d0/d15/d24/f8e 0 2024-04-12T05:09:01.713 INFO:tasks.workunit.client.1.smithi192.stdout:0/520: dwrite d0/d6/d15/d55/d2e/d59/f78 [0,4194304] 0 2024-04-12T05:09:01.720 INFO:tasks.workunit.client.1.smithi192.stdout:0/521: dwrite d0/d4e/d53/d6d/f9a [0,4194304] 0 2024-04-12T05:09:01.721 INFO:tasks.workunit.client.1.smithi192.stdout:5/412: link d0/d15/d24/d29/c40 d0/d15/d3e/c8f 0 2024-04-12T05:09:01.722 INFO:tasks.workunit.client.1.smithi192.stdout:0/522: fsync d0/d6/d15/d1b/d46/f4d 0 2024-04-12T05:09:01.724 INFO:tasks.workunit.client.1.smithi192.stdout:5/413: mknod d0/d15/d64/c90 0 2024-04-12T05:09:01.726 INFO:tasks.workunit.client.1.smithi192.stdout:0/523: rename d0/d6/d15/d1b/da4 to d0/d6/d15/d55/d29/db5 0 2024-04-12T05:09:01.728 INFO:tasks.workunit.client.1.smithi192.stdout:0/524: write d0/d6/fa3 [369318,30853] 0 2024-04-12T05:09:01.729 INFO:tasks.workunit.client.1.smithi192.stdout:0/525: chown d0/d6/d67/f92 126 1 2024-04-12T05:09:01.730 INFO:tasks.workunit.client.1.smithi192.stdout:0/526: chown d0/d4e/d9f 15 1 2024-04-12T05:09:01.731 INFO:tasks.workunit.client.1.smithi192.stdout:0/527: dread - d0/d6/f96 zero size 2024-04-12T05:09:01.732 INFO:tasks.workunit.client.1.smithi192.stdout:7/357: write d3/d7/f8 [4502603,7728] 0 2024-04-12T05:09:01.734 INFO:tasks.workunit.client.1.smithi192.stdout:7/358: readlink d3/d7/d15/d1a/l2a 0 2024-04-12T05:09:01.734 INFO:tasks.workunit.client.1.smithi192.stdout:7/359: chown d3/d1d/d4a 312546664 1 2024-04-12T05:09:01.737 INFO:tasks.workunit.client.1.smithi192.stdout:2/447: truncate d1/d2d/f32 1833721 0 2024-04-12T05:09:01.738 INFO:tasks.workunit.client.1.smithi192.stdout:7/360: write d3/d7/f19 [5378152,8471] 0 2024-04-12T05:09:01.744 INFO:tasks.workunit.client.1.smithi192.stdout:2/448: chown d1/d8/d17/d19/d60/d59/d68/l8a 357 1 2024-04-12T05:09:01.744 INFO:tasks.workunit.client.1.smithi192.stdout:0/528: dwrite d0/d6/d15/f6f [4194304,4194304] 0 2024-04-12T05:09:01.745 INFO:tasks.workunit.client.1.smithi192.stdout:7/361: truncate d3/d1d/d74/f75 26012 0 2024-04-12T05:09:01.745 INFO:tasks.workunit.client.1.smithi192.stdout:9/327: dwrite d11/d25/f32 [0,4194304] 0 2024-04-12T05:09:01.750 INFO:tasks.workunit.client.1.smithi192.stdout:2/449: write d1/d8/d3a/de/f57 [1229405,112046] 0 2024-04-12T05:09:01.755 INFO:tasks.workunit.client.1.smithi192.stdout:9/328: mkdir d11/d15/d21/d2c/d1f/d6a 0 2024-04-12T05:09:01.755 INFO:tasks.workunit.client.1.smithi192.stdout:4/501: dwrite d1/d54/d55/fa6 [0,4194304] 0 2024-04-12T05:09:01.757 INFO:tasks.workunit.client.1.smithi192.stdout:2/450: chown d1/d8/d3a/de/l18 2114234784 1 2024-04-12T05:09:01.758 INFO:tasks.workunit.client.1.smithi192.stdout:4/502: chown d1/d1d/l35 58587 1 2024-04-12T05:09:01.759 INFO:tasks.workunit.client.1.smithi192.stdout:9/329: creat d11/d15/f6b x:0 0 0 2024-04-12T05:09:01.760 INFO:tasks.workunit.client.1.smithi192.stdout:2/451: write d1/f4 [1586117,16518] 0 2024-04-12T05:09:01.762 INFO:tasks.workunit.client.1.smithi192.stdout:0/529: dread d0/d4e/d53/d6d/f9a [0,4194304] 0 2024-04-12T05:09:01.764 INFO:tasks.workunit.client.1.smithi192.stdout:7/362: dwrite d3/d7/d60/f66 [0,4194304] 0 2024-04-12T05:09:01.764 INFO:tasks.workunit.client.1.smithi192.stdout:9/330: unlink d11/d37/f66 0 2024-04-12T05:09:01.765 INFO:tasks.workunit.client.1.smithi192.stdout:0/530: stat d0/d6/d15/c1f 0 2024-04-12T05:09:01.770 INFO:tasks.workunit.client.1.smithi192.stdout:9/331: creat d11/d37/d33/f6c x:0 0 0 2024-04-12T05:09:01.772 INFO:tasks.workunit.client.1.smithi192.stdout:4/503: dwrite d1/d54/f61 [0,4194304] 0 2024-04-12T05:09:01.772 INFO:tasks.workunit.client.1.smithi192.stdout:0/531: unlink d0/d6/d15/d19/f1d 0 2024-04-12T05:09:01.772 INFO:tasks.workunit.client.0.smithi113.stdout:9/586: dwrite d1/df/db6/f84 [0,4194304] 0 2024-04-12T05:09:01.773 INFO:tasks.workunit.client.0.smithi113.stdout:3/522: dread d0/d13/d15/d28/d2a/d34/f63 [0,4194304] 0 2024-04-12T05:09:01.773 INFO:tasks.workunit.client.0.smithi113.stdout:0/596: truncate d1/d6/d16/f24 574184 0 2024-04-12T05:09:01.775 INFO:tasks.workunit.client.0.smithi113.stdout:1/628: mkdir d10/d2e/d31/dc3 0 2024-04-12T05:09:01.775 INFO:tasks.workunit.client.0.smithi113.stdout:9/587: dread - d1/d4/d6d/fb8 zero size 2024-04-12T05:09:01.776 INFO:tasks.workunit.client.0.smithi113.stdout:6/523: dread d8/de/d13/d11/d36/d3d/d4c/d31/f54 [0,4194304] 0 2024-04-12T05:09:01.777 INFO:tasks.workunit.client.0.smithi113.stdout:8/606: write d9/d16/f1a [2686257,75265] 0 2024-04-12T05:09:01.777 INFO:tasks.workunit.client.1.smithi192.stdout:7/363: dwrite d3/d7/d15/f77 [0,4194304] 0 2024-04-12T05:09:01.778 INFO:tasks.workunit.client.1.smithi192.stdout:9/332: rmdir d11/d25/d36/d3d 39 2024-04-12T05:09:01.780 INFO:tasks.workunit.client.0.smithi113.stdout:8/607: write d9/f9d [789237,48696] 0 2024-04-12T05:09:01.781 INFO:tasks.workunit.client.0.smithi113.stdout:6/524: chown d8/de/d13/d11/d36/d3d/d4c/d26 2 1 2024-04-12T05:09:01.782 INFO:tasks.workunit.client.0.smithi113.stdout:1/629: unlink d10/d41/l42 0 2024-04-12T05:09:01.782 INFO:tasks.workunit.client.0.smithi113.stdout:8/608: fsync d9/f9e 0 2024-04-12T05:09:01.783 INFO:tasks.workunit.client.0.smithi113.stdout:9/588: fdatasync d1/d4/d38/f8c 0 2024-04-12T05:09:01.783 INFO:tasks.workunit.client.1.smithi192.stdout:7/364: dread - d3/d7/d14/f47 zero size 2024-04-12T05:09:01.785 INFO:tasks.workunit.client.0.smithi113.stdout:9/589: truncate d1/d4/d5/fb9 531867 0 2024-04-12T05:09:01.786 INFO:tasks.workunit.client.0.smithi113.stdout:0/597: truncate d1/d6/d16/d2a/d50/d52/d7e/db5/f5f 2704617 0 2024-04-12T05:09:01.787 INFO:tasks.workunit.client.0.smithi113.stdout:7/542: dread d0/dd/d11/f4a [0,4194304] 0 2024-04-12T05:09:01.788 INFO:tasks.workunit.client.0.smithi113.stdout:3/523: creat d0/d13/d15/fb4 x:0 0 0 2024-04-12T05:09:01.791 INFO:tasks.workunit.client.0.smithi113.stdout:9/590: mknod d1/d4/d5/d22/cc8 0 2024-04-12T05:09:01.791 INFO:tasks.workunit.client.0.smithi113.stdout:6/525: symlink d8/de/d13/d11/d36/d3d/d4c/d31/d83/lbd 0 2024-04-12T05:09:01.792 INFO:tasks.workunit.client.0.smithi113.stdout:0/598: write d1/d6/d16/d2a/d50/db6/f5c [2769492,61914] 0 2024-04-12T05:09:01.792 INFO:tasks.workunit.client.1.smithi192.stdout:0/532: mknod d0/d6/cb6 0 2024-04-12T05:09:01.793 INFO:tasks.workunit.client.1.smithi192.stdout:2/452: dwrite d1/d8/d17/d19/f5e [0,4194304] 0 2024-04-12T05:09:01.795 INFO:tasks.workunit.client.1.smithi192.stdout:0/533: chown d0/d7b/d7c/c91 1 1 2024-04-12T05:09:01.795 INFO:tasks.workunit.client.1.smithi192.stdout:4/504: getdents d1/d54/d82/d71 0 2024-04-12T05:09:01.796 INFO:tasks.workunit.client.1.smithi192.stdout:7/365: dwrite d3/d7/f56 [4194304,4194304] 0 2024-04-12T05:09:01.797 INFO:tasks.workunit.client.0.smithi113.stdout:7/543: rename d0/dc/d6/d9b/c4c to d0/dc/d6/d78/daa/cbe 0 2024-04-12T05:09:01.797 INFO:tasks.workunit.client.0.smithi113.stdout:6/526: symlink d8/de/d13/d11/d36/d3d/d4c/d26/d7d/lbe 0 2024-04-12T05:09:01.803 INFO:tasks.workunit.client.0.smithi113.stdout:3/524: rename d0/d13/d15/d28/d2a/c69 to d0/d7e/da3/cb5 0 2024-04-12T05:09:01.804 INFO:tasks.workunit.client.0.smithi113.stdout:5/492: dread d0/d10/d27/f25 [0,4194304] 0 2024-04-12T05:09:01.806 INFO:tasks.workunit.client.1.smithi192.stdout:4/505: rename d1/d54/f81 to d1/d1d/d53/d1c/d84/fa9 0 2024-04-12T05:09:01.806 INFO:tasks.workunit.client.0.smithi113.stdout:7/544: mknod d0/dc/d6/d78/daa/cbf 0 2024-04-12T05:09:01.808 INFO:tasks.workunit.client.0.smithi113.stdout:5/493: chown d0/d10/f56 655401 1 2024-04-12T05:09:01.809 INFO:tasks.workunit.client.0.smithi113.stdout:0/599: rename d1/d6/f58 to d1/d6/d8e/fba 0 2024-04-12T05:09:01.809 INFO:tasks.workunit.client.0.smithi113.stdout:9/591: dwrite d1/f79 [0,4194304] 0 2024-04-12T05:09:01.809 INFO:tasks.workunit.client.0.smithi113.stdout:6/527: mknod d8/de/d17/cbf 0 2024-04-12T05:09:01.813 INFO:tasks.workunit.client.1.smithi192.stdout:0/534: rename d0/d6/d15/d55/d29/d89 to d0/d6/d15/d55/d2e/d66/db7 0 2024-04-12T05:09:01.814 INFO:tasks.workunit.client.1.smithi192.stdout:3/339: sync 2024-04-12T05:09:01.814 INFO:tasks.workunit.client.1.smithi192.stdout:6/463: sync 2024-04-12T05:09:01.814 INFO:tasks.workunit.client.1.smithi192.stdout:1/381: sync 2024-04-12T05:09:01.814 INFO:tasks.workunit.client.0.smithi113.stdout:9/592: creat d1/d4/d87/fc9 x:0 0 0 2024-04-12T05:09:01.816 INFO:tasks.workunit.client.0.smithi113.stdout:7/545: dread d0/dc/d6/d78/f6a [0,4194304] 0 2024-04-12T05:09:01.816 INFO:tasks.workunit.client.0.smithi113.stdout:5/494: truncate d0/d10/d27/d5f/f88 230754 0 2024-04-12T05:09:01.818 INFO:tasks.workunit.client.1.smithi192.stdout:6/464: fdatasync d3/d26/f28 0 2024-04-12T05:09:01.819 INFO:tasks.workunit.client.0.smithi113.stdout:9/593: chown d1/d4/le 1361324 1 2024-04-12T05:09:01.819 INFO:tasks.workunit.client.1.smithi192.stdout:4/506: dwrite d1/d54/d82/f62 [4194304,4194304] 0 2024-04-12T05:09:01.819 INFO:tasks.workunit.client.1.smithi192.stdout:3/340: write d5/d13/d3a/d4a/f5a [3390388,104447] 0 2024-04-12T05:09:01.820 INFO:tasks.workunit.client.1.smithi192.stdout:1/382: unlink d0/d2/c72 0 2024-04-12T05:09:01.821 INFO:tasks.workunit.client.0.smithi113.stdout:7/546: chown d0/dd/d11/f8c 66430705 1 2024-04-12T05:09:01.821 INFO:tasks.workunit.client.0.smithi113.stdout:5/495: dread - d0/d10/d27/d1d/d3f/f6c zero size 2024-04-12T05:09:01.823 INFO:tasks.workunit.client.1.smithi192.stdout:3/341: chown d5/d13/c56 2 1 2024-04-12T05:09:01.825 INFO:tasks.workunit.client.1.smithi192.stdout:6/465: creat d3/dc/f9a x:0 0 0 2024-04-12T05:09:01.826 INFO:tasks.workunit.client.0.smithi113.stdout:8/609: dread d9/d1b/d8e/d93/d32/f3f [0,4194304] 0 2024-04-12T05:09:01.826 INFO:tasks.workunit.client.0.smithi113.stdout:6/528: dwrite d8/de/d17/f59 [0,4194304] 0 2024-04-12T05:09:01.826 INFO:tasks.workunit.client.0.smithi113.stdout:5/496: stat d0/d10/d27/d1d/d38/d44/c74 0 2024-04-12T05:09:01.826 INFO:tasks.workunit.client.0.smithi113.stdout:9/594: unlink d1/d4/d38/c86 0 2024-04-12T05:09:01.827 INFO:tasks.workunit.client.1.smithi192.stdout:0/535: dwrite d0/d4e/d53/f9d [0,4194304] 0 2024-04-12T05:09:01.828 INFO:tasks.workunit.client.1.smithi192.stdout:6/466: chown d3/d8/d48/c8b 4 1 2024-04-12T05:09:01.829 INFO:tasks.workunit.client.0.smithi113.stdout:7/547: write d0/dc/d6/d1a/fb8 [847527,30473] 0 2024-04-12T05:09:01.830 INFO:tasks.workunit.client.1.smithi192.stdout:0/536: write d0/d6/d15/d1b/d46/f8c [821138,121626] 0 2024-04-12T05:09:01.833 INFO:tasks.workunit.client.1.smithi192.stdout:1/383: dwrite d0/d1b/d28/f61 [0,4194304] 0 2024-04-12T05:09:01.836 INFO:tasks.workunit.client.1.smithi192.stdout:7/366: dread d3/d1d/f28 [0,4194304] 0 2024-04-12T05:09:01.836 INFO:tasks.workunit.client.0.smithi113.stdout:9/595: creat d1/d4/d5/d22/fca x:0 0 0 2024-04-12T05:09:01.838 INFO:tasks.workunit.client.0.smithi113.stdout:9/596: chown d1/df/db6/lc2 326925381 1 2024-04-12T05:09:01.839 INFO:tasks.workunit.client.1.smithi192.stdout:7/367: chown d3/d1d/d74 702517 1 2024-04-12T05:09:01.839 INFO:tasks.workunit.client.1.smithi192.stdout:6/467: write d3/d5/f80 [3799100,14408] 0 2024-04-12T05:09:01.839 INFO:tasks.workunit.client.1.smithi192.stdout:1/384: symlink d0/d1b/d28/l76 0 2024-04-12T05:09:01.842 INFO:tasks.workunit.client.1.smithi192.stdout:4/507: dread d1/d6b/f7e [0,4194304] 0 2024-04-12T05:09:01.843 INFO:tasks.workunit.client.1.smithi192.stdout:7/368: fsync d3/f12 0 2024-04-12T05:09:01.844 INFO:tasks.workunit.client.0.smithi113.stdout:5/497: creat d0/d10/f9f x:0 0 0 2024-04-12T05:09:01.844 INFO:tasks.workunit.client.0.smithi113.stdout:6/529: unlink d8/f1a 0 2024-04-12T05:09:01.845 INFO:tasks.workunit.client.1.smithi192.stdout:6/468: chown d3/d23/c2f 889804 1 2024-04-12T05:09:01.846 INFO:tasks.workunit.client.1.smithi192.stdout:7/369: write d3/d11/f37 [380877,105346] 0 2024-04-12T05:09:01.850 INFO:tasks.workunit.client.0.smithi113.stdout:5/498: stat d0/d10/d27/d1d/d38 0 2024-04-12T05:09:01.851 INFO:tasks.workunit.client.1.smithi192.stdout:4/508: rmdir d1/d54/d82 39 2024-04-12T05:09:01.851 INFO:tasks.workunit.client.1.smithi192.stdout:0/537: dread d0/d6/d15/d55/f6c [0,4194304] 0 2024-04-12T05:09:01.851 INFO:tasks.workunit.client.0.smithi113.stdout:6/530: creat d8/de/d13/d11/d36/d3d/d4c/d31/fc0 x:0 0 0 2024-04-12T05:09:01.853 INFO:tasks.workunit.client.0.smithi113.stdout:0/600: write d1/d43/f62 [4475940,109051] 0 2024-04-12T05:09:01.855 INFO:tasks.workunit.client.1.smithi192.stdout:3/342: dread d5/f14 [4194304,4194304] 0 2024-04-12T05:09:01.856 INFO:tasks.workunit.client.1.smithi192.stdout:0/538: rmdir d0/d7b/d9b/d4b 39 2024-04-12T05:09:01.857 INFO:tasks.workunit.client.0.smithi113.stdout:5/499: rename d0/d10/d27/d1d/d3f/d73/d89/f28 to d0/d10/d27/d1d/d41/fa0 0 2024-04-12T05:09:01.858 INFO:tasks.workunit.client.1.smithi192.stdout:0/539: chown d0/d6/d67/f8a 8176 1 2024-04-12T05:09:01.858 INFO:tasks.workunit.client.1.smithi192.stdout:4/509: chown d1/d54/d82/d71/d9e/fa0 140441998 1 2024-04-12T05:09:01.860 INFO:tasks.workunit.client.0.smithi113.stdout:5/500: stat d0/d10/d27/d1d/d38/d44/l59 0 2024-04-12T05:09:01.861 INFO:tasks.workunit.client.0.smithi113.stdout:6/531: dwrite d8/de/d13/d2b/d76/fab [0,4194304] 0 2024-04-12T05:09:01.863 INFO:tasks.workunit.client.0.smithi113.stdout:0/601: mkdir d1/d6/d16/d2a/d50/d52/d7e/db5/dbb 0 2024-04-12T05:09:01.863 INFO:tasks.workunit.client.1.smithi192.stdout:4/510: dread d1/d49/f69 [0,4194304] 0 2024-04-12T05:09:01.863 INFO:tasks.workunit.client.1.smithi192.stdout:7/370: symlink d3/d1d/l7f 0 2024-04-12T05:09:01.864 INFO:tasks.workunit.client.1.smithi192.stdout:4/511: dread - d1/d54/d55/d64/f91 zero size 2024-04-12T05:09:01.865 INFO:tasks.workunit.client.1.smithi192.stdout:4/512: chown d1/d1d/l1e 8675 1 2024-04-12T05:09:01.866 INFO:tasks.workunit.client.1.smithi192.stdout:4/513: write d1/d54/d55/d64/f8f [786263,38576] 0 2024-04-12T05:09:01.868 INFO:tasks.workunit.client.0.smithi113.stdout:8/610: dread d9/d16/d3c/f52 [0,4194304] 0 2024-04-12T05:09:01.869 INFO:tasks.workunit.client.0.smithi113.stdout:7/548: truncate d0/dc/d6/d9b/f62 2019585 0 2024-04-12T05:09:01.869 INFO:tasks.workunit.client.1.smithi192.stdout:7/371: creat d3/d1d/d5a/d5f/f80 x:0 0 0 2024-04-12T05:09:01.870 INFO:tasks.workunit.client.0.smithi113.stdout:3/525: dwrite d0/d13/d47/d4f/f4c [0,4194304] 0 2024-04-12T05:09:01.871 INFO:tasks.workunit.client.1.smithi192.stdout:4/514: write d1/d54/f94 [3459669,15555] 0 2024-04-12T05:09:01.871 INFO:tasks.workunit.client.1.smithi192.stdout:1/385: truncate d0/f24 3082997 0 2024-04-12T05:09:01.872 INFO:tasks.workunit.client.1.smithi192.stdout:1/386: fdatasync d0/d1b/f6a 0 2024-04-12T05:09:01.872 INFO:tasks.workunit.client.1.smithi192.stdout:3/343: creat d5/d13/d3a/d70/f71 x:0 0 0 2024-04-12T05:09:01.873 INFO:tasks.workunit.client.1.smithi192.stdout:6/469: write d3/d21/f8d [729663,2602] 0 2024-04-12T05:09:01.875 INFO:tasks.workunit.client.1.smithi192.stdout:0/540: mkdir d0/d4e/d95/db4/db8 0 2024-04-12T05:09:01.876 INFO:tasks.workunit.client.1.smithi192.stdout:4/515: rmdir d1/d49 39 2024-04-12T05:09:01.877 INFO:tasks.workunit.client.1.smithi192.stdout:1/387: symlink d0/d2/dd/d71/l77 0 2024-04-12T05:09:01.878 INFO:tasks.workunit.client.1.smithi192.stdout:7/372: creat d3/d7/d60/d65/f81 x:0 0 0 2024-04-12T05:09:01.879 INFO:tasks.workunit.client.1.smithi192.stdout:3/344: mknod d5/d13/d1a/c72 0 2024-04-12T05:09:01.880 INFO:tasks.workunit.client.1.smithi192.stdout:7/373: truncate d3/d1d/d74/f75 688978 0 2024-04-12T05:09:01.880 INFO:tasks.workunit.client.1.smithi192.stdout:4/516: creat d1/d54/d55/d64/d68/faa x:0 0 0 2024-04-12T05:09:01.880 INFO:tasks.workunit.client.1.smithi192.stdout:6/470: symlink d3/d21/d24/d79/l9b 0 2024-04-12T05:09:01.881 INFO:tasks.workunit.client.1.smithi192.stdout:1/388: mkdir d0/d2/d1e/d3f/d78 0 2024-04-12T05:09:01.882 INFO:tasks.workunit.client.1.smithi192.stdout:3/345: mknod d5/d13/c73 0 2024-04-12T05:09:01.882 INFO:tasks.workunit.client.1.smithi192.stdout:1/389: fdatasync d0/d2/dd/d21/d33/f6b 0 2024-04-12T05:09:01.883 INFO:tasks.workunit.client.1.smithi192.stdout:4/517: write d1/f4b [1885588,125157] 0 2024-04-12T05:09:01.883 INFO:tasks.workunit.client.1.smithi192.stdout:6/471: stat d3/d5/ca 0 2024-04-12T05:09:01.883 INFO:tasks.workunit.client.1.smithi192.stdout:0/541: dread d0/d6/d15/d55/f48 [0,4194304] 0 2024-04-12T05:09:01.885 INFO:tasks.workunit.client.1.smithi192.stdout:1/390: readlink d0/d2/dd/d21/d55/l75 0 2024-04-12T05:09:01.885 INFO:tasks.workunit.client.1.smithi192.stdout:7/374: creat d3/d1d/f82 x:0 0 0 2024-04-12T05:09:01.886 INFO:tasks.workunit.client.1.smithi192.stdout:6/472: stat d3/dc/d39/c47 0 2024-04-12T05:09:01.887 INFO:tasks.workunit.client.1.smithi192.stdout:7/375: dread - d3/d1d/f69 zero size 2024-04-12T05:09:01.888 INFO:tasks.workunit.client.1.smithi192.stdout:4/518: symlink d1/d54/d55/lab 0 2024-04-12T05:09:01.889 INFO:tasks.workunit.client.1.smithi192.stdout:0/542: write d0/d6/d15/d55/d2e/d66/db7/f93 [865396,77225] 0 2024-04-12T05:09:01.889 INFO:tasks.workunit.client.1.smithi192.stdout:7/376: chown d3/d7/f7e 183397238 1 2024-04-12T05:09:01.890 INFO:tasks.workunit.client.1.smithi192.stdout:6/473: mkdir d3/d5/d9c 0 2024-04-12T05:09:01.890 INFO:tasks.workunit.client.1.smithi192.stdout:4/519: write d1/d54/d82/da4/f90 [743452,128635] 0 2024-04-12T05:09:01.891 INFO:tasks.workunit.client.1.smithi192.stdout:7/377: chown d3/d1d/l7f 3396 1 2024-04-12T05:09:01.894 INFO:tasks.workunit.client.1.smithi192.stdout:0/543: symlink d0/d6/d15/d1b/d3a/lb9 0 2024-04-12T05:09:01.895 INFO:tasks.workunit.client.1.smithi192.stdout:6/474: creat d3/d8/d56/d59/d97/f9d x:0 0 0 2024-04-12T05:09:01.896 INFO:tasks.workunit.client.1.smithi192.stdout:4/520: truncate d1/d54/d55/f30 83706 0 2024-04-12T05:09:01.896 INFO:tasks.workunit.client.1.smithi192.stdout:3/346: mknod d5/d13/d3a/d70/c74 0 2024-04-12T05:09:01.898 INFO:tasks.workunit.client.1.smithi192.stdout:0/544: rename d0/d6/d15/d19/f31 to d0/d6/d15/d1b/d3a/d79/fba 0 2024-04-12T05:09:01.898 INFO:tasks.workunit.client.1.smithi192.stdout:0/545: read - d0/fad zero size 2024-04-12T05:09:01.899 INFO:tasks.workunit.client.1.smithi192.stdout:6/475: mkdir d3/d5/d9e 0 2024-04-12T05:09:01.900 INFO:tasks.workunit.client.1.smithi192.stdout:4/521: mknod d1/d1d/d53/d1c/d95/cac 0 2024-04-12T05:09:01.900 INFO:tasks.workunit.client.1.smithi192.stdout:0/546: write d0/d6/d15/d55/d2e/f9e [4250349,85940] 0 2024-04-12T05:09:01.901 INFO:tasks.workunit.client.1.smithi192.stdout:3/347: creat d5/d13/d1a/d5f/f75 x:0 0 0 2024-04-12T05:09:01.903 INFO:tasks.workunit.client.1.smithi192.stdout:0/547: readlink d0/d6/d15/d1b/d3a/d4f/l94 0 2024-04-12T05:09:01.904 INFO:tasks.workunit.client.1.smithi192.stdout:3/348: mknod d5/d13/d3a/d4e/c76 0 2024-04-12T05:09:01.907 INFO:tasks.workunit.client.1.smithi192.stdout:0/548: write d0/d6/d15/d1b/d46/f8c [545528,75206] 0 2024-04-12T05:09:01.908 INFO:tasks.workunit.client.1.smithi192.stdout:3/349: mknod d5/d13/d3a/c77 0 2024-04-12T05:09:01.909 INFO:tasks.workunit.client.1.smithi192.stdout:4/522: dread d1/d54/d82/d71/f7b [0,4194304] 0 2024-04-12T05:09:01.910 INFO:tasks.workunit.client.1.smithi192.stdout:6/476: dwrite d3/d8/d10/f71 [0,4194304] 0 2024-04-12T05:09:01.915 INFO:tasks.workunit.client.1.smithi192.stdout:0/549: dread d0/d6/d15/d1b/d46/f49 [0,4194304] 0 2024-04-12T05:09:01.915 INFO:tasks.workunit.client.1.smithi192.stdout:3/350: symlink d5/d13/d3a/d4a/l78 0 2024-04-12T05:09:01.916 INFO:tasks.workunit.client.1.smithi192.stdout:6/477: creat d3/d8/d56/d59/d8a/d99/f9f x:0 0 0 2024-04-12T05:09:01.918 INFO:tasks.workunit.client.1.smithi192.stdout:4/523: rename d1/d54/d55/d64/f63 to d1/d1d/fad 0 2024-04-12T05:09:01.919 INFO:tasks.workunit.client.1.smithi192.stdout:4/524: fdatasync d1/f4b 0 2024-04-12T05:09:01.919 INFO:tasks.workunit.client.1.smithi192.stdout:3/351: fsync d5/d13/d3c/f68 0 2024-04-12T05:09:01.920 INFO:tasks.workunit.client.1.smithi192.stdout:0/550: mkdir d0/d7b/d7c/dbb 0 2024-04-12T05:09:01.925 INFO:tasks.workunit.client.1.smithi192.stdout:4/525: dread d1/d1d/d53/f57 [0,4194304] 0 2024-04-12T05:09:01.927 INFO:tasks.workunit.client.1.smithi192.stdout:0/551: mkdir d0/d7b/d9b/d4b/dbc 0 2024-04-12T05:09:01.927 INFO:tasks.workunit.client.1.smithi192.stdout:4/526: creat d1/d54/d55/d64/fae x:0 0 0 2024-04-12T05:09:01.927 INFO:tasks.workunit.client.1.smithi192.stdout:6/478: dread d3/d21/d24/f4f [0,4194304] 0 2024-04-12T05:09:01.929 INFO:tasks.workunit.client.1.smithi192.stdout:6/479: read d3/d26/f28 [2829386,19036] 0 2024-04-12T05:09:01.929 INFO:tasks.workunit.client.1.smithi192.stdout:4/527: symlink d1/d54/d82/laf 0 2024-04-12T05:09:01.931 INFO:tasks.workunit.client.1.smithi192.stdout:4/528: fdatasync d1/d1d/d53/d1c/f66 0 2024-04-12T05:09:01.935 INFO:tasks.workunit.client.1.smithi192.stdout:0/552: dwrite d0/d7b/d9b/f38 [4194304,4194304] 0 2024-04-12T05:09:01.948 INFO:tasks.workunit.client.1.smithi192.stdout:6/480: dread d3/d5/f66 [0,4194304] 0 2024-04-12T05:09:01.952 INFO:tasks.workunit.client.0.smithi113.stdout:5/501: mkdir d0/d10/da1 0 2024-04-12T05:09:01.953 INFO:tasks.workunit.client.0.smithi113.stdout:0/602: creat d1/d80/fbc x:0 0 0 2024-04-12T05:09:01.955 INFO:tasks.workunit.client.0.smithi113.stdout:8/611: chown d9/d16/d65/db1/f8b 5 1 2024-04-12T05:09:01.955 INFO:tasks.workunit.client.0.smithi113.stdout:6/532: creat d8/de/d13/d11/d36/d3d/d4c/d26/fc1 x:0 0 0 2024-04-12T05:09:01.957 INFO:tasks.workunit.client.0.smithi113.stdout:5/502: symlink d0/d10/d27/d1d/d8c/la2 0 2024-04-12T05:09:01.958 INFO:tasks.workunit.client.0.smithi113.stdout:6/533: write d8/de/d17/f7c [760474,80278] 0 2024-04-12T05:09:01.958 INFO:tasks.workunit.client.0.smithi113.stdout:7/549: truncate d0/dd/d11/f5e 2168318 0 2024-04-12T05:09:01.959 INFO:tasks.workunit.client.0.smithi113.stdout:3/526: mknod d0/d13/d15/d28/d2a/d8b/d95/cb6 0 2024-04-12T05:09:01.960 INFO:tasks.workunit.client.0.smithi113.stdout:0/603: rmdir d1/d6/d16/d2a/d50/d52/d7e 39 2024-04-12T05:09:01.960 INFO:tasks.workunit.client.0.smithi113.stdout:7/550: chown d0/dc/d6/d9b 114 1 2024-04-12T05:09:01.960 INFO:tasks.workunit.client.0.smithi113.stdout:5/503: chown d0/d10/d27/d1d/d38/f8d 43524475 1 2024-04-12T05:09:01.961 INFO:tasks.workunit.client.0.smithi113.stdout:3/527: mknod d0/d7e/cb7 0 2024-04-12T05:09:01.962 INFO:tasks.workunit.client.1.smithi192.stdout:0/553: dread d0/d6/d67/f92 [0,4194304] 0 2024-04-12T05:09:01.964 INFO:tasks.workunit.client.0.smithi113.stdout:9/597: dread d1/d4/f77 [0,4194304] 0 2024-04-12T05:09:01.965 INFO:tasks.workunit.client.1.smithi192.stdout:0/554: mknod d0/d7b/d9b/d4b/cbd 0 2024-04-12T05:09:01.965 INFO:tasks.workunit.client.1.smithi192.stdout:0/555: stat d0/f16 0 2024-04-12T05:09:01.966 INFO:tasks.workunit.client.0.smithi113.stdout:3/528: rmdir d0/d13/d15/d28/d2a/d8b 39 2024-04-12T05:09:01.966 INFO:tasks.workunit.client.1.smithi192.stdout:0/556: fsync d0/d6/d15/d55/d2e/d66/f86 0 2024-04-12T05:09:01.967 INFO:tasks.workunit.client.0.smithi113.stdout:0/604: creat d1/d6/d16/d2a/fbd x:0 0 0 2024-04-12T05:09:01.968 INFO:tasks.workunit.client.0.smithi113.stdout:0/605: fdatasync d1/d80/fbc 0 2024-04-12T05:09:01.969 INFO:tasks.workunit.client.1.smithi192.stdout:0/557: creat d0/d6/d15/d1b/d46/fbe x:0 0 0 2024-04-12T05:09:01.970 INFO:tasks.workunit.client.0.smithi113.stdout:6/534: dwrite d8/de/d13/d11/d36/f35 [4194304,4194304] 0 2024-04-12T05:09:01.970 INFO:tasks.workunit.client.0.smithi113.stdout:3/529: write d0/d13/d15/d28/d2a/d34/f72 [388160,30339] 0 2024-04-12T05:09:01.971 INFO:tasks.workunit.client.0.smithi113.stdout:0/606: readlink d1/d6/d16/d2a/d50/db6/l12 0 2024-04-12T05:09:01.973 INFO:tasks.workunit.client.0.smithi113.stdout:9/598: rename d1/d4/d5/d6/d15 to d1/d4/d30/dcb 0 2024-04-12T05:09:01.973 INFO:tasks.workunit.client.0.smithi113.stdout:0/607: read d1/d6/d16/d2a/d50/db6/f4d [7595202,76485] 0 2024-04-12T05:09:01.975 INFO:tasks.workunit.client.0.smithi113.stdout:3/530: unlink d0/d13/d15/l3b 0 2024-04-12T05:09:01.976 INFO:tasks.workunit.client.0.smithi113.stdout:0/608: mknod d1/d6/d8e/cbe 0 2024-04-12T05:09:01.977 INFO:tasks.workunit.client.0.smithi113.stdout:6/535: creat d8/de/d13/d11/d36/d53/db8/fc2 x:0 0 0 2024-04-12T05:09:01.979 INFO:tasks.workunit.client.0.smithi113.stdout:6/536: chown d8/de/d13/d11/d36/d53/db8/f9b 336 1 2024-04-12T05:09:01.980 INFO:tasks.workunit.client.0.smithi113.stdout:3/531: mknod d0/d13/d15/d28/d2a/d34/d5f/d7b/cb8 0 2024-04-12T05:09:01.990 INFO:tasks.workunit.client.0.smithi113.stdout:8/612: truncate d9/d1b/d8e/d93/d32/d48/d8d/fa0 107576 0 2024-04-12T05:09:01.990 INFO:tasks.workunit.client.1.smithi192.stdout:7/378: truncate d3/d7/d15/f24 2273325 0 2024-04-12T05:09:01.990 INFO:tasks.workunit.client.0.smithi113.stdout:5/504: write d0/d10/d27/d1d/d38/d3b/f61 [923013,81597] 0 2024-04-12T05:09:01.991 INFO:tasks.workunit.client.1.smithi192.stdout:7/379: write d3/d7/d15/f3c [4743894,83218] 0 2024-04-12T05:09:01.994 INFO:tasks.workunit.client.0.smithi113.stdout:6/537: dread d8/de/d17/f7c [0,4194304] 0 2024-04-12T05:09:01.999 INFO:tasks.workunit.client.1.smithi192.stdout:3/352: write d5/f2d [4822845,109629] 0 2024-04-12T05:09:02.005 INFO:tasks.workunit.client.1.smithi192.stdout:0/558: truncate d0/d6/d15/d55/d2e/d66/db7/f85 1849043 0 2024-04-12T05:09:02.005 INFO:tasks.workunit.client.0.smithi113.stdout:0/609: write d1/d6/d16/d2a/d50/f7d [1522873,110632] 0 2024-04-12T05:09:02.010 INFO:tasks.workunit.client.0.smithi113.stdout:0/610: chown d1/d6/d16/d2a/d50/db6/cab 7426599 1 2024-04-12T05:09:02.011 INFO:tasks.workunit.client.1.smithi192.stdout:7/380: write d3/d1d/f41 [839065,69613] 0 2024-04-12T05:09:02.012 INFO:tasks.workunit.client.0.smithi113.stdout:0/611: fsync d1/d9/f2b 0 2024-04-12T05:09:02.012 INFO:tasks.workunit.client.1.smithi192.stdout:3/353: truncate d5/fd 1954673 0 2024-04-12T05:09:02.013 INFO:tasks.workunit.client.1.smithi192.stdout:6/481: dwrite d3/d8/f1f [0,4194304] 0 2024-04-12T05:09:02.013 INFO:tasks.workunit.client.0.smithi113.stdout:5/505: dwrite d0/d10/d6e/f3c [0,4194304] 0 2024-04-12T05:09:02.015 INFO:tasks.workunit.client.1.smithi192.stdout:1/391: dread d0/f24 [0,4194304] 0 2024-04-12T05:09:02.015 INFO:tasks.workunit.client.1.smithi192.stdout:7/381: fsync d3/d1d/f69 0 2024-04-12T05:09:02.016 INFO:tasks.workunit.client.0.smithi113.stdout:5/506: fdatasync d0/d10/d27/d1d/d8c/f9d 0 2024-04-12T05:09:02.017 INFO:tasks.workunit.client.1.smithi192.stdout:4/529: dwrite d1/d1d/d53/f52 [0,4194304] 0 2024-04-12T05:09:02.024 INFO:tasks.workunit.client.1.smithi192.stdout:0/559: mkdir d0/d4e/db0/dbf 0 2024-04-12T05:09:02.026 INFO:tasks.workunit.client.1.smithi192.stdout:6/482: dwrite d3/dc/d39/f87 [0,4194304] 0 2024-04-12T05:09:02.027 INFO:tasks.workunit.client.1.smithi192.stdout:4/530: creat d1/d54/d82/d71/fb0 x:0 0 0 2024-04-12T05:09:02.028 INFO:tasks.workunit.client.1.smithi192.stdout:3/354: write d5/d13/f2a [2443448,94607] 0 2024-04-12T05:09:02.029 INFO:tasks.workunit.client.0.smithi113.stdout:0/612: dwrite d1/d6/d16/d2a/d50/db6/f5c [0,4194304] 0 2024-04-12T05:09:02.030 INFO:tasks.workunit.client.1.smithi192.stdout:1/392: rename d0/f63 to d0/d2/d1e/d3f/d56/f79 0 2024-04-12T05:09:02.030 INFO:tasks.workunit.client.1.smithi192.stdout:0/560: fsync d0/d6/d15/d1b/d46/f4d 0 2024-04-12T05:09:02.030 INFO:tasks.workunit.client.0.smithi113.stdout:5/507: fsync d0/d10/d27/d1d/d38/d3b/f5e 0 2024-04-12T05:09:02.031 INFO:tasks.workunit.client.1.smithi192.stdout:7/382: symlink d3/d1d/d5a/d5f/d6a/l83 0 2024-04-12T05:09:02.033 INFO:tasks.workunit.client.1.smithi192.stdout:3/355: fsync d5/d13/d3c/f68 0 2024-04-12T05:09:02.033 INFO:tasks.workunit.client.1.smithi192.stdout:4/531: fdatasync d1/d54/d55/d64/f91 0 2024-04-12T05:09:02.034 INFO:tasks.workunit.client.0.smithi113.stdout:5/508: read d0/d10/f16 [3410108,33348] 0 2024-04-12T05:09:02.035 INFO:tasks.workunit.client.0.smithi113.stdout:5/509: readlink d0/d10/l24 0 2024-04-12T05:09:02.035 INFO:tasks.workunit.client.1.smithi192.stdout:7/383: mknod d3/c84 0 2024-04-12T05:09:02.035 INFO:tasks.workunit.client.1.smithi192.stdout:1/393: rename d0/d2 to d0/d2/dd/d21/d62/d7a 22 2024-04-12T05:09:02.036 INFO:tasks.workunit.client.0.smithi113.stdout:5/510: fsync d0/d10/d27/d1d/d38/d3b/f84 0 2024-04-12T05:09:02.041 INFO:tasks.workunit.client.0.smithi113.stdout:5/511: creat d0/d10/d27/d1d/d38/d44/fa3 x:0 0 0 2024-04-12T05:09:02.043 INFO:tasks.workunit.client.0.smithi113.stdout:5/512: chown d0/d10/d27/d1d/d38/c79 260980958 1 2024-04-12T05:09:02.047 INFO:tasks.workunit.client.1.smithi192.stdout:4/532: creat d1/d1d/d53/d1c/fb1 x:0 0 0 2024-04-12T05:09:02.047 INFO:tasks.workunit.client.1.smithi192.stdout:3/356: mkdir d5/d13/d3a/d4a/d61/d79 0 2024-04-12T05:09:02.047 INFO:tasks.workunit.client.1.smithi192.stdout:1/394: write d0/d2/d1e/d3f/d56/f5a [915651,119333] 0 2024-04-12T05:09:02.047 INFO:tasks.workunit.client.1.smithi192.stdout:6/483: dread d3/d8/d10/d4c/d68/f89 [0,4194304] 0 2024-04-12T05:09:02.047 INFO:tasks.workunit.client.0.smithi113.stdout:5/513: rename d0/d10/l1c to d0/d10/d27/d1d/d8c/la4 0 2024-04-12T05:09:02.050 INFO:tasks.workunit.client.0.smithi113.stdout:5/514: rmdir d0/d10/d27/d1d/d3f/d73/d89 39 2024-04-12T05:09:02.051 INFO:tasks.workunit.client.1.smithi192.stdout:3/357: dwrite d5/d13/f2a [0,4194304] 0 2024-04-12T05:09:02.054 INFO:tasks.workunit.client.0.smithi113.stdout:5/515: creat d0/d10/da1/fa5 x:0 0 0 2024-04-12T05:09:02.056 INFO:tasks.workunit.client.1.smithi192.stdout:4/533: mkdir d1/d1d/d53/db2 0 2024-04-12T05:09:02.057 INFO:tasks.workunit.client.1.smithi192.stdout:7/384: link d3/d1d/d5a/d5f/c6b d3/d1d/d5a/d5f/c85 0 2024-04-12T05:09:02.058 INFO:tasks.workunit.client.1.smithi192.stdout:6/484: link d3/d21/d24/f4f d3/d21/d24/d79/fa0 0 2024-04-12T05:09:02.061 INFO:tasks.workunit.client.1.smithi192.stdout:4/534: creat d1/d54/d82/fb3 x:0 0 0 2024-04-12T05:09:02.062 INFO:tasks.workunit.client.0.smithi113.stdout:5/516: rename d0/d10/d27/d1d/d8c/la2 to d0/d10/d27/d1d/d38/d3b/d6b/la6 0 2024-04-12T05:09:02.063 INFO:tasks.workunit.client.1.smithi192.stdout:3/358: creat d5/f7a x:0 0 0 2024-04-12T05:09:02.064 INFO:tasks.workunit.client.1.smithi192.stdout:6/485: rename d3/d21/d24/l44 to d3/d23/la1 0 2024-04-12T05:09:02.065 INFO:tasks.workunit.client.0.smithi113.stdout:5/517: creat d0/d10/d27/d1d/d8c/fa7 x:0 0 0 2024-04-12T05:09:02.070 INFO:tasks.workunit.client.0.smithi113.stdout:5/518: rename d0/f2e to d0/fa8 0 2024-04-12T05:09:02.070 INFO:tasks.workunit.client.1.smithi192.stdout:0/561: write d0/d6/d15/d55/f6c [1390384,115738] 0 2024-04-12T05:09:02.072 INFO:tasks.workunit.client.1.smithi192.stdout:0/562: chown d0/d6/d15/d55/d2e/fa9 740367 1 2024-04-12T05:09:02.073 INFO:tasks.workunit.client.0.smithi113.stdout:0/613: dread d1/d6/d16/f67 [0,4194304] 0 2024-04-12T05:09:02.073 INFO:tasks.workunit.client.0.smithi113.stdout:5/519: unlink d0/d10/d27/d1d/d38/l9a 0 2024-04-12T05:09:02.074 INFO:tasks.workunit.client.1.smithi192.stdout:3/359: link d5/d13/d1a/l21 d5/d13/d3a/d4e/l7b 0 2024-04-12T05:09:02.075 INFO:tasks.workunit.client.1.smithi192.stdout:4/535: dwrite d1/d1d/fad [0,4194304] 0 2024-04-12T05:09:02.076 INFO:tasks.workunit.client.0.smithi113.stdout:5/520: symlink d0/d10/d27/d1d/d8c/la9 0 2024-04-12T05:09:02.076 INFO:tasks.workunit.client.0.smithi113.stdout:5/521: dread - d0/d10/da1/fa5 zero size 2024-04-12T05:09:02.081 INFO:tasks.workunit.client.0.smithi113.stdout:5/522: mkdir d0/d10/d27/d1d/d38/d44/d4e/daa 0 2024-04-12T05:09:02.083 INFO:tasks.workunit.client.1.smithi192.stdout:1/395: dread d0/d2/dd/f15 [0,4194304] 0 2024-04-12T05:09:02.084 INFO:tasks.workunit.client.1.smithi192.stdout:1/396: stat d0/d1b/f6a 0 2024-04-12T05:09:02.087 INFO:tasks.workunit.client.0.smithi113.stdout:5/523: rename d0/d10/l99 to d0/d10/d27/d1d/d38/d44/d66/lab 0 2024-04-12T05:09:02.088 INFO:tasks.workunit.client.1.smithi192.stdout:3/360: dwrite d5/d13/f42 [0,4194304] 0 2024-04-12T05:09:02.095 INFO:tasks.workunit.client.1.smithi192.stdout:3/361: getdents d5/d13/d1a 0 2024-04-12T05:09:02.097 INFO:tasks.workunit.client.1.smithi192.stdout:6/486: dread d3/f37 [0,4194304] 0 2024-04-12T05:09:02.099 INFO:tasks.workunit.client.1.smithi192.stdout:3/362: chown d5/d13/f2e 7991 1 2024-04-12T05:09:02.099 INFO:tasks.workunit.client.1.smithi192.stdout:0/563: dread d0/d6/d15/d19/f4c [0,4194304] 0 2024-04-12T05:09:02.099 INFO:tasks.workunit.client.0.smithi113.stdout:5/524: rmdir d0/d10/d27/d1d/d38/d44/d66 39 2024-04-12T05:09:02.101 INFO:tasks.workunit.client.1.smithi192.stdout:4/536: truncate d1/d54/d55/d64/f8f 274010 0 2024-04-12T05:09:02.102 INFO:tasks.workunit.client.0.smithi113.stdout:5/525: unlink d0/d10/d27/d1d/d38/d44/l59 0 2024-04-12T05:09:02.102 INFO:tasks.workunit.client.0.smithi113.stdout:5/526: dread - d0/f98 zero size 2024-04-12T05:09:02.104 INFO:tasks.workunit.client.1.smithi192.stdout:6/487: unlink d3/d8/d48/f50 0 2024-04-12T05:09:02.105 INFO:tasks.workunit.client.1.smithi192.stdout:1/397: truncate d0/d1b/f35 2857682 0 2024-04-12T05:09:02.108 INFO:tasks.workunit.client.1.smithi192.stdout:6/488: mknod d3/d8/d22/ca2 0 2024-04-12T05:09:02.109 INFO:tasks.workunit.client.1.smithi192.stdout:3/363: getdents d5/d13 0 2024-04-12T05:09:02.109 INFO:tasks.workunit.client.1.smithi192.stdout:1/398: creat d0/d2/f7b x:0 0 0 2024-04-12T05:09:02.110 INFO:tasks.workunit.client.0.smithi113.stdout:5/527: dwrite d0/d10/f56 [0,4194304] 0 2024-04-12T05:09:02.111 INFO:tasks.workunit.client.1.smithi192.stdout:3/364: symlink d5/d13/l7c 0 2024-04-12T05:09:02.111 INFO:tasks.workunit.client.0.smithi113.stdout:5/528: fdatasync d0/f5 0 2024-04-12T05:09:02.112 INFO:tasks.workunit.client.1.smithi192.stdout:4/537: getdents d1/d54/d55 0 2024-04-12T05:09:02.114 INFO:tasks.workunit.client.1.smithi192.stdout:1/399: mkdir d0/d2/dd/d3c/d5b/d7c 0 2024-04-12T05:09:02.116 INFO:tasks.workunit.client.0.smithi113.stdout:5/529: rename d0/d10/d27/f18 to d0/d10/d27/d1d/fac 0 2024-04-12T05:09:02.118 INFO:tasks.workunit.client.1.smithi192.stdout:6/489: creat d3/d8/fa3 x:0 0 0 2024-04-12T05:09:02.122 INFO:tasks.workunit.client.1.smithi192.stdout:3/365: symlink d5/l7d 0 2024-04-12T05:09:02.122 INFO:tasks.workunit.client.0.smithi113.stdout:5/530: dwrite d0/d10/d27/d1d/d38/d3b/f61 [0,4194304] 0 2024-04-12T05:09:02.123 INFO:tasks.workunit.client.1.smithi192.stdout:1/400: mkdir d0/d2/dd/d3c/d5b/d74/d7d 0 2024-04-12T05:09:02.124 INFO:tasks.workunit.client.0.smithi113.stdout:5/531: chown d0/d10/d27/d5f/f65 766069 1 2024-04-12T05:09:02.125 INFO:tasks.workunit.client.0.smithi113.stdout:5/532: write d0/d10/d6e/f87 [1035160,113468] 0 2024-04-12T05:09:02.129 INFO:tasks.workunit.client.0.smithi113.stdout:5/533: readlink d0/d10/d27/d1d/d3f/d73/d89/l1a 0 2024-04-12T05:09:02.129 INFO:tasks.workunit.client.1.smithi192.stdout:1/401: chown d0/d2/dd/d21/d55/l75 116 1 2024-04-12T05:09:02.130 INFO:tasks.workunit.client.1.smithi192.stdout:6/490: dread d3/f37 [0,4194304] 0 2024-04-12T05:09:02.130 INFO:tasks.workunit.client.1.smithi192.stdout:0/564: dwrite d0/d6/d15/d55/d29/f2b [0,4194304] 0 2024-04-12T05:09:02.130 INFO:tasks.workunit.client.1.smithi192.stdout:4/538: dwrite d1/f3e [0,4194304] 0 2024-04-12T05:09:02.131 INFO:tasks.workunit.client.0.smithi113.stdout:5/534: write d0/d10/d27/d1d/d38/d44/f92 [836814,30026] 0 2024-04-12T05:09:02.137 INFO:tasks.workunit.client.0.smithi113.stdout:5/535: creat d0/d10/d27/d1d/d3f/d73/fad x:0 0 0 2024-04-12T05:09:02.137 INFO:tasks.workunit.client.1.smithi192.stdout:6/491: mkdir d3/d26/da4 0 2024-04-12T05:09:02.139 INFO:tasks.workunit.client.1.smithi192.stdout:1/402: write d0/f24 [1644407,103246] 0 2024-04-12T05:09:02.139 INFO:tasks.workunit.client.1.smithi192.stdout:6/492: chown d3/d21/f7d 31 1 2024-04-12T05:09:02.143 INFO:tasks.workunit.client.1.smithi192.stdout:0/565: getdents d0/d6/d15/d19 0 2024-04-12T05:09:02.143 INFO:tasks.workunit.client.1.smithi192.stdout:4/539: dwrite d1/f29 [0,4194304] 0 2024-04-12T05:09:02.144 INFO:tasks.workunit.client.1.smithi192.stdout:6/493: rmdir d3/d8/d22 39 2024-04-12T05:09:02.145 INFO:tasks.workunit.client.1.smithi192.stdout:4/540: write d1/d54/d82/d71/d9e/fa0 [604917,25548] 0 2024-04-12T05:09:02.147 INFO:tasks.workunit.client.1.smithi192.stdout:1/403: rename d0/d2/dd/d21/l40 to d0/d2/dd/d3c/d5b/l7e 0 2024-04-12T05:09:02.151 INFO:tasks.workunit.client.0.smithi113.stdout:5/536: write d0/d10/d27/d1d/d38/d44/d66/f96 [1318746,86634] 0 2024-04-12T05:09:02.152 INFO:tasks.workunit.client.1.smithi192.stdout:6/494: getdents d3/d8/d48/d85 0 2024-04-12T05:09:02.155 INFO:tasks.workunit.client.1.smithi192.stdout:1/404: getdents d0/d2/d1e/d3f 0 2024-04-12T05:09:02.155 INFO:tasks.workunit.client.0.smithi113.stdout:5/537: creat d0/d10/d27/d1d/d3f/d7e/fae x:0 0 0 2024-04-12T05:09:02.156 INFO:tasks.workunit.client.1.smithi192.stdout:6/495: readlink d3/d23/la1 0 2024-04-12T05:09:02.157 INFO:tasks.workunit.client.1.smithi192.stdout:4/541: dwrite d1/d54/d82/f48 [0,4194304] 0 2024-04-12T05:09:02.157 INFO:tasks.workunit.client.0.smithi113.stdout:5/538: write d0/d10/d27/d1d/d38/d3b/f84 [42417,79901] 0 2024-04-12T05:09:02.158 INFO:tasks.workunit.client.0.smithi113.stdout:5/539: fdatasync d0/d10/da1/fa5 0 2024-04-12T05:09:02.161 INFO:tasks.workunit.client.1.smithi192.stdout:0/566: dwrite d0/f20 [0,4194304] 0 2024-04-12T05:09:02.163 INFO:tasks.workunit.client.1.smithi192.stdout:1/405: read d0/d2/dd/d21/f43 [147364,29698] 0 2024-04-12T05:09:02.165 INFO:tasks.workunit.client.1.smithi192.stdout:6/496: mknod d3/d8/d56/d59/ca5 0 2024-04-12T05:09:02.165 INFO:tasks.workunit.client.1.smithi192.stdout:0/567: dread - d0/d6/faa zero size 2024-04-12T05:09:02.166 INFO:tasks.workunit.client.1.smithi192.stdout:6/497: chown d3/f37 260990 1 2024-04-12T05:09:02.166 INFO:tasks.workunit.client.0.smithi113.stdout:5/540: dread d0/f31 [0,4194304] 0 2024-04-12T05:09:02.167 INFO:tasks.workunit.client.1.smithi192.stdout:1/406: creat d0/d2/dd/d3c/f7f x:0 0 0 2024-04-12T05:09:02.169 INFO:tasks.workunit.client.1.smithi192.stdout:4/542: rename d1/d1d/d53/cf to d1/d1d/d53/cb4 0 2024-04-12T05:09:02.170 INFO:tasks.workunit.client.1.smithi192.stdout:0/568: readlink d0/l74 0 2024-04-12T05:09:02.170 INFO:tasks.workunit.client.0.smithi113.stdout:5/541: mknod d0/d10/d27/d1d/d38/d44/d4e/caf 0 2024-04-12T05:09:02.171 INFO:tasks.workunit.client.1.smithi192.stdout:6/498: rmdir d3/d8/d56/d59/d97 39 2024-04-12T05:09:02.172 INFO:tasks.workunit.client.1.smithi192.stdout:3/366: dread d5/f2d [0,4194304] 0 2024-04-12T05:09:02.172 INFO:tasks.workunit.client.0.smithi113.stdout:5/542: chown d0/d10/d27/d1d/d3f/f6a 0 1 2024-04-12T05:09:02.174 INFO:tasks.workunit.client.1.smithi192.stdout:6/499: dread - d3/dc/d39/f7a zero size 2024-04-12T05:09:02.175 INFO:tasks.workunit.client.0.smithi113.stdout:5/543: rmdir d0/d10/d27/d1d/d38/d3b 39 2024-04-12T05:09:02.177 INFO:tasks.workunit.client.1.smithi192.stdout:0/569: mknod d0/cc0 0 2024-04-12T05:09:02.178 INFO:tasks.workunit.client.1.smithi192.stdout:4/543: mkdir d1/d49/db5 0 2024-04-12T05:09:02.179 INFO:tasks.workunit.client.1.smithi192.stdout:0/570: chown d0/l74 176139550 1 2024-04-12T05:09:02.181 INFO:tasks.workunit.client.0.smithi113.stdout:5/544: getdents d0/d10/d27/d1d/d8c 0 2024-04-12T05:09:02.181 INFO:tasks.workunit.client.1.smithi192.stdout:0/571: fdatasync d0/d6/d15/d19/f71 0 2024-04-12T05:09:02.182 INFO:tasks.workunit.client.0.smithi113.stdout:5/545: stat d0/d10/d27/d1d/d38/d44/d4e 0 2024-04-12T05:09:02.184 INFO:tasks.workunit.client.1.smithi192.stdout:3/367: rename d5/d13/d3c/f59 to d5/d13/d1a/f7e 0 2024-04-12T05:09:02.185 INFO:tasks.workunit.client.1.smithi192.stdout:0/572: truncate d0/d6/d15/d55/d2e/d59/f78 4457502 0 2024-04-12T05:09:02.185 INFO:tasks.workunit.client.1.smithi192.stdout:6/500: creat d3/d8/d10/d4c/fa6 x:0 0 0 2024-04-12T05:09:02.185 INFO:tasks.workunit.client.0.smithi113.stdout:5/546: mkdir d0/d10/d27/d1d/d3f/db0 0 2024-04-12T05:09:02.187 INFO:tasks.workunit.client.0.smithi113.stdout:4/533: sync 2024-04-12T05:09:02.187 INFO:tasks.workunit.client.1.smithi192.stdout:3/368: chown d5/d13/d1a/f3b 1 1 2024-04-12T05:09:02.188 INFO:tasks.workunit.client.1.smithi192.stdout:5/414: sync 2024-04-12T05:09:02.188 INFO:tasks.workunit.client.1.smithi192.stdout:8/356: sync 2024-04-12T05:09:02.189 INFO:tasks.workunit.client.1.smithi192.stdout:1/407: dwrite d0/d2/dd/d21/d33/f42 [0,4194304] 0 2024-04-12T05:09:02.191 INFO:tasks.workunit.client.1.smithi192.stdout:3/369: chown d5/d13/d3a/d4e/d57 10114615 1 2024-04-12T05:09:02.192 INFO:tasks.workunit.client.0.smithi113.stdout:5/547: dwrite d0/d10/d27/d1d/d38/d3b/f5e [0,4194304] 0 2024-04-12T05:09:02.193 INFO:tasks.workunit.client.1.smithi192.stdout:0/573: read d0/d7b/d9b/f60 [596874,51097] 0 2024-04-12T05:09:02.193 INFO:tasks.workunit.client.0.smithi113.stdout:4/534: link da/d3c/d83/c36 da/d3c/d9f/cac 0 2024-04-12T05:09:02.194 INFO:tasks.workunit.client.1.smithi192.stdout:3/370: chown d5/d13/d3a/d4e/f6c 270 1 2024-04-12T05:09:02.197 INFO:tasks.workunit.client.1.smithi192.stdout:5/415: dread d0/d15/d24/f8e [0,4194304] 0 2024-04-12T05:09:02.198 INFO:tasks.workunit.client.1.smithi192.stdout:8/357: rename d6/d4b to d6/d1f/d6a 0 2024-04-12T05:09:02.199 INFO:tasks.workunit.client.0.smithi113.stdout:4/535: mkdir da/d3c/da5/dad 0 2024-04-12T05:09:02.200 INFO:tasks.workunit.client.1.smithi192.stdout:1/408: symlink d0/d2/dd/d21/d55/l80 0 2024-04-12T05:09:02.200 INFO:tasks.workunit.client.0.smithi113.stdout:4/536: truncate da/d24/d60/f99 5127052 0 2024-04-12T05:09:02.201 INFO:tasks.workunit.client.1.smithi192.stdout:6/501: dwrite d3/d8/d56/f75 [0,4194304] 0 2024-04-12T05:09:02.202 INFO:tasks.workunit.client.1.smithi192.stdout:1/409: chown d0/d1b/l27 45262 1 2024-04-12T05:09:02.203 INFO:tasks.workunit.client.1.smithi192.stdout:8/358: fsync d6/d21/f44 0 2024-04-12T05:09:02.204 INFO:tasks.workunit.client.0.smithi113.stdout:4/537: chown da/dd/d3a/ca7 2173812 1 2024-04-12T05:09:02.204 INFO:tasks.workunit.client.1.smithi192.stdout:5/416: readlink d0/d15/d24/l3c 0 2024-04-12T05:09:02.205 INFO:tasks.workunit.client.0.smithi113.stdout:5/548: truncate d0/d10/f56 2109009 0 2024-04-12T05:09:02.207 INFO:tasks.workunit.client.1.smithi192.stdout:3/371: symlink d5/d13/d3a/d4a/d61/d79/l7f 0 2024-04-12T05:09:02.207 INFO:tasks.workunit.client.1.smithi192.stdout:1/410: creat d0/d1b/d28/f81 x:0 0 0 2024-04-12T05:09:02.208 INFO:tasks.workunit.client.0.smithi113.stdout:5/549: mkdir d0/d10/d27/d1d/d41/db1 0 2024-04-12T05:09:02.209 INFO:tasks.workunit.client.1.smithi192.stdout:6/502: stat d3/d8/c2d 0 2024-04-12T05:09:02.210 INFO:tasks.workunit.client.0.smithi113.stdout:5/550: write d0/d10/d27/d1d/d3f/f6c [746158,82612] 0 2024-04-12T05:09:02.210 INFO:tasks.workunit.client.1.smithi192.stdout:8/359: read f3 [3101122,85545] 0 2024-04-12T05:09:02.213 INFO:tasks.workunit.client.1.smithi192.stdout:3/372: write d5/d13/d3a/d70/f71 [496963,73796] 0 2024-04-12T05:09:02.214 INFO:tasks.workunit.client.1.smithi192.stdout:5/417: rename d0/d15/l44 to d0/d15/d3e/d77/l91 0 2024-04-12T05:09:02.214 INFO:tasks.workunit.client.1.smithi192.stdout:8/360: fsync d6/da/f3a 0 2024-04-12T05:09:02.215 INFO:tasks.workunit.client.0.smithi113.stdout:5/551: creat d0/d10/d27/d1d/d38/fb2 x:0 0 0 2024-04-12T05:09:02.216 INFO:tasks.workunit.client.1.smithi192.stdout:0/574: dwrite d0/d6/d15/d1b/d3a/d4f/f24 [0,4194304] 0 2024-04-12T05:09:02.216 INFO:tasks.workunit.client.1.smithi192.stdout:1/411: write d0/f20 [708543,1672] 0 2024-04-12T05:09:02.217 INFO:tasks.workunit.client.1.smithi192.stdout:8/361: chown d6/d10/f3f 11007378 1 2024-04-12T05:09:02.219 INFO:tasks.workunit.client.0.smithi113.stdout:4/538: read da/dd/d3a/fa3 [312193,116828] 0 2024-04-12T05:09:02.219 INFO:tasks.workunit.client.1.smithi192.stdout:0/575: chown d0/d7b/ca8 83691 1 2024-04-12T05:09:02.221 INFO:tasks.workunit.client.1.smithi192.stdout:6/503: getdents d3/d8/d10/d4c/d68/d83 0 2024-04-12T05:09:02.222 INFO:tasks.workunit.client.1.smithi192.stdout:0/576: chown d0/d6/fa3 238850635 1 2024-04-12T05:09:02.222 INFO:tasks.workunit.client.0.smithi113.stdout:5/552: creat d0/d10/d27/d1d/d38/d44/fb3 x:0 0 0 2024-04-12T05:09:02.223 INFO:tasks.workunit.client.1.smithi192.stdout:5/418: rmdir d0/d15/d3e/d77 39 2024-04-12T05:09:02.225 INFO:tasks.workunit.client.0.smithi113.stdout:5/553: read - d0/d10/d6e/f4a zero size 2024-04-12T05:09:02.225 INFO:tasks.workunit.client.1.smithi192.stdout:6/504: dread d3/d8/d10/d4c/d68/f89 [0,4194304] 0 2024-04-12T05:09:02.226 INFO:tasks.workunit.client.1.smithi192.stdout:6/505: readlink d3/d5/l34 0 2024-04-12T05:09:02.226 INFO:tasks.workunit.client.0.smithi113.stdout:5/554: fsync d0/d10/d27/d5f/f65 0 2024-04-12T05:09:02.226 INFO:tasks.workunit.client.1.smithi192.stdout:0/577: unlink d0/d6/d15/d55/d2e/fa9 0 2024-04-12T05:09:02.227 INFO:tasks.workunit.client.1.smithi192.stdout:0/578: write d0/fad [262764,48017] 0 2024-04-12T05:09:02.228 INFO:tasks.workunit.client.1.smithi192.stdout:8/362: mknod d6/d10/d46/d5e/d5f/d38/d3b/c6b 0 2024-04-12T05:09:02.230 INFO:tasks.workunit.client.1.smithi192.stdout:6/506: write d3/d8/d56/d59/d8a/d99/f9f [204890,652] 0 2024-04-12T05:09:02.238 INFO:tasks.workunit.client.1.smithi192.stdout:3/373: dread d5/d13/d3a/d70/f71 [0,4194304] 0 2024-04-12T05:09:02.239 INFO:tasks.workunit.client.0.smithi113.stdout:4/539: dwrite da/dd/f32 [0,4194304] 0 2024-04-12T05:09:02.239 INFO:tasks.workunit.client.1.smithi192.stdout:5/419: truncate d0/d21/f33 1775108 0 2024-04-12T05:09:02.241 INFO:tasks.workunit.client.1.smithi192.stdout:0/579: dread d0/d6/d15/d55/d2e/d59/f78 [0,4194304] 0 2024-04-12T05:09:02.241 INFO:tasks.workunit.client.1.smithi192.stdout:6/507: dread d3/d5/ff [0,4194304] 0 2024-04-12T05:09:02.241 INFO:tasks.workunit.client.1.smithi192.stdout:8/363: truncate d6/d10/d46/d5e/d5f/d37/d4f/d4c/f58 995516 0 2024-04-12T05:09:02.245 INFO:tasks.workunit.client.1.smithi192.stdout:1/412: dread d0/d1b/f2b [0,4194304] 0 2024-04-12T05:09:02.246 INFO:tasks.workunit.client.0.smithi113.stdout:5/555: write d0/d10/d27/d1d/d3f/f47 [1101321,92946] 0 2024-04-12T05:09:02.248 INFO:tasks.workunit.client.1.smithi192.stdout:6/508: creat d3/d5/d9c/fa7 x:0 0 0 2024-04-12T05:09:02.249 INFO:tasks.workunit.client.1.smithi192.stdout:8/364: truncate d6/d10/d46/d5e/d5f/d37/d4f/d4c/f64 645301 0 2024-04-12T05:09:02.251 INFO:tasks.workunit.client.1.smithi192.stdout:1/413: fdatasync d0/d2/dd/f32 0 2024-04-12T05:09:02.251 INFO:tasks.workunit.client.1.smithi192.stdout:3/374: dwrite d5/d13/f42 [0,4194304] 0 2024-04-12T05:09:02.252 INFO:tasks.workunit.client.1.smithi192.stdout:5/420: write d0/d15/d24/f8e [966465,68558] 0 2024-04-12T05:09:02.253 INFO:tasks.workunit.client.1.smithi192.stdout:6/509: rmdir d3/d8/d56 39 2024-04-12T05:09:02.253 INFO:tasks.workunit.client.1.smithi192.stdout:3/375: chown d5/d13/c73 0 1 2024-04-12T05:09:02.255 INFO:tasks.workunit.client.1.smithi192.stdout:6/510: write d3/d21/f7d [432899,77239] 0 2024-04-12T05:09:02.256 INFO:tasks.workunit.client.1.smithi192.stdout:1/414: creat d0/d2/dd/d21/d39/f82 x:0 0 0 2024-04-12T05:09:02.256 INFO:tasks.workunit.client.1.smithi192.stdout:5/421: rename d0/d15/d24 to d0/d15/d24/d92 22 2024-04-12T05:09:02.257 INFO:tasks.workunit.client.1.smithi192.stdout:6/511: rmdir d3/d26 39 2024-04-12T05:09:02.257 INFO:tasks.workunit.client.1.smithi192.stdout:8/365: read d6/da/f30 [200453,82856] 0 2024-04-12T05:09:02.258 INFO:tasks.workunit.client.1.smithi192.stdout:1/415: write d0/d2/dd/d21/d33/f3a [13351680,23086] 0 2024-04-12T05:09:02.258 INFO:tasks.workunit.client.1.smithi192.stdout:3/376: creat d5/d13/d3a/d4a/d61/d79/f80 x:0 0 0 2024-04-12T05:09:02.260 INFO:tasks.workunit.client.1.smithi192.stdout:5/422: truncate d0/d21/d5e/f6c 146638 0 2024-04-12T05:09:02.261 INFO:tasks.workunit.client.1.smithi192.stdout:1/416: dread - d0/d1b/d28/f4d zero size 2024-04-12T05:09:02.261 INFO:tasks.workunit.client.1.smithi192.stdout:8/366: write d6/d10/d46/d5e/d5f/f4d [8108,77750] 0 2024-04-12T05:09:02.264 INFO:tasks.workunit.client.1.smithi192.stdout:1/417: write d0/d2/dd/d21/d33/f3a [4471651,7811] 0 2024-04-12T05:09:02.264 INFO:tasks.workunit.client.1.smithi192.stdout:8/367: dread - d6/d10/d46/d5e/d5f/f57 zero size 2024-04-12T05:09:02.349 INFO:tasks.workunit.client.1.smithi192.stdout:5/423: mknod d0/d15/d24/d29/c93 0 2024-04-12T05:09:02.350 INFO:tasks.workunit.client.1.smithi192.stdout:6/512: truncate d3/d8/d56/d59/d97/f9d 89611 0 2024-04-12T05:09:02.351 INFO:tasks.workunit.client.1.smithi192.stdout:5/424: dread d0/d15/d24/f51 [0,4194304] 0 2024-04-12T05:09:02.353 INFO:tasks.workunit.client.1.smithi192.stdout:8/368: mkdir d6/d10/d6c 0 2024-04-12T05:09:02.353 INFO:tasks.workunit.client.1.smithi192.stdout:1/418: mknod d0/d2/dd/d21/d55/c83 0 2024-04-12T05:09:02.353 INFO:tasks.workunit.client.0.smithi113.stdout:2/452: sync 2024-04-12T05:09:02.354 INFO:tasks.workunit.client.1.smithi192.stdout:1/419: chown d0/d2/l14 86286291 1 2024-04-12T05:09:02.355 INFO:tasks.workunit.client.1.smithi192.stdout:6/513: dread d3/f37 [0,4194304] 0 2024-04-12T05:09:02.362 INFO:tasks.workunit.client.1.smithi192.stdout:5/425: write d0/d15/d3e/d77/f78 [4342176,76785] 0 2024-04-12T05:09:02.363 INFO:tasks.workunit.client.1.smithi192.stdout:8/369: creat d6/d21/f6d x:0 0 0 2024-04-12T05:09:02.365 INFO:tasks.workunit.client.1.smithi192.stdout:8/370: write d6/d10/d46/d5e/d5f/d38/f5a [526181,16451] 0 2024-04-12T05:09:02.366 INFO:tasks.workunit.client.1.smithi192.stdout:3/377: link d5/d13/d3a/c77 d5/d13/d3a/d4a/c81 0 2024-04-12T05:09:02.367 INFO:tasks.workunit.client.1.smithi192.stdout:5/426: mknod d0/d15/d24/d29/c94 0 2024-04-12T05:09:02.368 INFO:tasks.workunit.client.1.smithi192.stdout:1/420: mkdir d0/d2/dd/d21/d62/d84 0 2024-04-12T05:09:02.369 INFO:tasks.workunit.client.1.smithi192.stdout:8/371: dread f5 [0,4194304] 0 2024-04-12T05:09:02.369 INFO:tasks.workunit.client.1.smithi192.stdout:1/421: readlink d0/d29/l46 0 2024-04-12T05:09:02.370 INFO:tasks.workunit.client.1.smithi192.stdout:6/514: link d3/d5/f45 d3/d21/fa8 0 2024-04-12T05:09:02.371 INFO:tasks.workunit.client.1.smithi192.stdout:5/427: dread d0/f61 [0,4194304] 0 2024-04-12T05:09:02.372 INFO:tasks.workunit.client.1.smithi192.stdout:8/372: mkdir d6/d10/d46/d5e/d6e 0 2024-04-12T05:09:02.372 INFO:tasks.workunit.client.1.smithi192.stdout:9/333: sync 2024-04-12T05:09:02.372 INFO:tasks.workunit.client.1.smithi192.stdout:2/453: sync 2024-04-12T05:09:02.373 INFO:tasks.workunit.client.1.smithi192.stdout:6/515: stat d3/l14 0 2024-04-12T05:09:02.374 INFO:tasks.workunit.client.1.smithi192.stdout:8/373: write d6/d1f/d6a/f52 [387506,80498] 0 2024-04-12T05:09:02.377 INFO:tasks.workunit.client.1.smithi192.stdout:3/378: dread d5/d13/d3a/d4a/d61/f64 [0,4194304] 0 2024-04-12T05:09:02.377 INFO:tasks.workunit.client.1.smithi192.stdout:1/422: getdents d0/d2/dd/d21/d39 0 2024-04-12T05:09:02.378 INFO:tasks.workunit.client.1.smithi192.stdout:6/516: dread d3/d8/d48/f73 [0,4194304] 0 2024-04-12T05:09:02.380 INFO:tasks.workunit.client.1.smithi192.stdout:8/374: creat d6/d10/d46/d5e/d5f/d37/f6f x:0 0 0 2024-04-12T05:09:02.382 INFO:tasks.workunit.client.1.smithi192.stdout:3/379: symlink d5/d13/d1a/l82 0 2024-04-12T05:09:02.384 INFO:tasks.workunit.client.1.smithi192.stdout:8/375: creat d6/da/f70 x:0 0 0 2024-04-12T05:09:02.385 INFO:tasks.workunit.client.1.smithi192.stdout:2/454: dwrite d1/d8/d17/d19/f45 [4194304,4194304] 0 2024-04-12T05:09:02.387 INFO:tasks.workunit.client.1.smithi192.stdout:9/334: dwrite d11/d37/d33/f4c [4194304,4194304] 0 2024-04-12T05:09:02.388 INFO:tasks.workunit.client.1.smithi192.stdout:3/380: write d5/d13/d3a/d70/f71 [1379702,43874] 0 2024-04-12T05:09:02.390 INFO:tasks.workunit.client.1.smithi192.stdout:6/517: rmdir d3/d26/da4 0 2024-04-12T05:09:02.392 INFO:tasks.workunit.client.1.smithi192.stdout:2/455: readlink d1/d8/d17/d19/l54 0 2024-04-12T05:09:02.393 INFO:tasks.workunit.client.1.smithi192.stdout:1/423: dwrite d0/d2/dd/d3c/f41 [0,4194304] 0 2024-04-12T05:09:02.395 INFO:tasks.workunit.client.1.smithi192.stdout:8/376: mknod d6/d10/d46/d5e/d5f/d37/d4f/d4c/c71 0 2024-04-12T05:09:02.396 INFO:tasks.workunit.client.1.smithi192.stdout:9/335: symlink d11/d15/d1e/l6d 0 2024-04-12T05:09:02.397 INFO:tasks.workunit.client.1.smithi192.stdout:3/381: rename d5/d13/l7c to d5/d13/d1a/d5f/l83 0 2024-04-12T05:09:02.399 INFO:tasks.workunit.client.1.smithi192.stdout:2/456: creat d1/d8/d3a/f95 x:0 0 0 2024-04-12T05:09:02.399 INFO:tasks.workunit.client.1.smithi192.stdout:9/336: readlink d11/d15/d1e/l6d 0 2024-04-12T05:09:02.401 INFO:tasks.workunit.client.1.smithi192.stdout:5/428: write d0/fa [2603138,31808] 0 2024-04-12T05:09:02.401 INFO:tasks.workunit.client.1.smithi192.stdout:3/382: mkdir d5/d13/d3c/d84 0 2024-04-12T05:09:02.408 INFO:tasks.workunit.client.1.smithi192.stdout:3/383: chown d5/d13/d1a/d53/l6a 0 1 2024-04-12T05:09:02.411 INFO:tasks.workunit.client.1.smithi192.stdout:5/429: dread - d0/d15/d24/d47/f54 zero size 2024-04-12T05:09:02.413 INFO:tasks.workunit.client.1.smithi192.stdout:6/518: dwrite d3/f94 [0,4194304] 0 2024-04-12T05:09:02.422 INFO:tasks.workunit.client.1.smithi192.stdout:9/337: rename d11/d37 to d11/d25/d36/d6e 0 2024-04-12T05:09:02.422 INFO:tasks.workunit.client.1.smithi192.stdout:3/384: chown d5/d13/f2a 0 1 2024-04-12T05:09:02.422 INFO:tasks.workunit.client.1.smithi192.stdout:1/424: dwrite d0/d2/dd/d3c/d5b/f5e [0,4194304] 0 2024-04-12T05:09:02.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:00.671211+0000 mgr.smithi192.ocppyo (mgr.24357) 26 : cluster [DBG] pgmap v20: 65 pgs: 65 active+clean; 1.5 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 31 MiB/s rd, 202 MiB/s wr, 304 op/s 2024-04-12T05:09:02.425 INFO:tasks.workunit.client.1.smithi192.stdout:2/457: dwrite d1/d8/d3a/f95 [0,4194304] 0 2024-04-12T05:09:02.428 INFO:tasks.workunit.client.1.smithi192.stdout:9/338: creat d11/d15/d21/f6f x:0 0 0 2024-04-12T05:09:02.428 INFO:tasks.workunit.client.1.smithi192.stdout:5/430: creat d0/d15/d24/d29/f95 x:0 0 0 2024-04-12T05:09:02.428 INFO:tasks.workunit.client.1.smithi192.stdout:3/385: mkdir d5/d13/d3a/d85 0 2024-04-12T05:09:02.429 INFO:tasks.workunit.client.1.smithi192.stdout:8/377: dread d6/d10/f11 [0,4194304] 0 2024-04-12T05:09:02.431 INFO:tasks.workunit.client.1.smithi192.stdout:3/386: write d5/d13/f42 [868372,96455] 0 2024-04-12T05:09:02.432 INFO:tasks.workunit.client.1.smithi192.stdout:3/387: read d5/f2d [4548283,113406] 0 2024-04-12T05:09:02.432 INFO:tasks.workunit.client.1.smithi192.stdout:1/425: symlink d0/d2/dd/d21/d33/l85 0 2024-04-12T05:09:02.435 INFO:tasks.workunit.client.1.smithi192.stdout:8/378: rename d6/d10/d46/d5e/d5f/d38/f47 to d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/f72 0 2024-04-12T05:09:02.436 INFO:tasks.workunit.client.1.smithi192.stdout:1/426: read d0/d2/dd/d21/d33/f3a [10875221,58841] 0 2024-04-12T05:09:02.436 INFO:tasks.workunit.client.1.smithi192.stdout:3/388: symlink d5/d13/d3a/d4a/d61/l86 0 2024-04-12T05:09:02.438 INFO:tasks.workunit.client.1.smithi192.stdout:1/427: chown d0/d2/dd/f4b 5 1 2024-04-12T05:09:02.440 INFO:tasks.workunit.client.1.smithi192.stdout:9/339: getdents d11/d15/d1e 0 2024-04-12T05:09:02.441 INFO:tasks.workunit.client.1.smithi192.stdout:9/340: chown d11/f47 3118 1 2024-04-12T05:09:02.441 INFO:tasks.workunit.client.1.smithi192.stdout:3/389: mknod d5/d13/d3a/d4e/c87 0 2024-04-12T05:09:02.442 INFO:tasks.workunit.client.1.smithi192.stdout:9/341: readlink d11/l51 0 2024-04-12T05:09:02.443 INFO:tasks.workunit.client.1.smithi192.stdout:1/428: dread d0/d1b/f2b [0,4194304] 0 2024-04-12T05:09:02.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:00.671211+0000 mgr.smithi192.ocppyo (mgr.24357) 26 : cluster [DBG] pgmap v20: 65 pgs: 65 active+clean; 1.5 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 31 MiB/s rd, 202 MiB/s wr, 304 op/s 2024-04-12T05:09:02.449 INFO:tasks.workunit.client.1.smithi192.stdout:9/342: creat d11/d25/f70 x:0 0 0 2024-04-12T05:09:02.451 INFO:tasks.workunit.client.1.smithi192.stdout:1/429: symlink d0/d2/dd/d21/d62/d84/l86 0 2024-04-12T05:09:02.453 INFO:tasks.workunit.client.1.smithi192.stdout:9/343: write d11/d25/d36/d6e/d33/f55 [290079,78431] 0 2024-04-12T05:09:02.455 INFO:tasks.workunit.client.1.smithi192.stdout:3/390: dwrite d5/d13/d3a/d4e/f6c [0,4194304] 0 2024-04-12T05:09:02.455 INFO:tasks.workunit.client.1.smithi192.stdout:9/344: fsync d11/d25/d36/f38 0 2024-04-12T05:09:02.459 INFO:tasks.workunit.client.1.smithi192.stdout:3/391: creat d5/d13/d1a/d53/f88 x:0 0 0 2024-04-12T05:09:02.462 INFO:tasks.workunit.client.1.smithi192.stdout:1/430: link d0/d1b/f6f d0/d2/d1e/d3f/f87 0 2024-04-12T05:09:02.464 INFO:tasks.workunit.client.1.smithi192.stdout:2/458: dread d1/d8/d17/d19/f45 [0,4194304] 0 2024-04-12T05:09:02.466 INFO:tasks.workunit.client.1.smithi192.stdout:9/345: dwrite d11/d15/d1e/f39 [0,4194304] 0 2024-04-12T05:09:02.469 INFO:tasks.workunit.client.1.smithi192.stdout:3/392: dread d5/d13/d3c/f68 [0,4194304] 0 2024-04-12T05:09:02.469 INFO:tasks.workunit.client.1.smithi192.stdout:2/459: rmdir d1/d8/d17 39 2024-04-12T05:09:02.471 INFO:tasks.workunit.client.1.smithi192.stdout:6/519: truncate d3/d8/d48/f6f 3890861 0 2024-04-12T05:09:02.472 INFO:tasks.workunit.client.1.smithi192.stdout:2/460: stat d1/d2d/d73/f8d 0 2024-04-12T05:09:02.473 INFO:tasks.workunit.client.1.smithi192.stdout:8/379: dread d6/da/f3a [0,4194304] 0 2024-04-12T05:09:02.473 INFO:tasks.workunit.client.1.smithi192.stdout:6/520: chown d3/dc 1671387291 1 2024-04-12T05:09:02.476 INFO:tasks.workunit.client.1.smithi192.stdout:2/461: fdatasync d1/d8/d3a/de/f11 0 2024-04-12T05:09:02.480 INFO:tasks.workunit.client.1.smithi192.stdout:8/380: creat d6/d10/d46/d5e/d5f/d38/d3b/f73 x:0 0 0 2024-04-12T05:09:02.482 INFO:tasks.workunit.client.1.smithi192.stdout:6/521: truncate d3/d21/f54 169714 0 2024-04-12T05:09:02.483 INFO:tasks.workunit.client.1.smithi192.stdout:9/346: dwrite d11/d25/d36/f63 [0,4194304] 0 2024-04-12T05:09:02.485 INFO:tasks.workunit.client.1.smithi192.stdout:3/393: getdents d5/d13/d3a/d4e/d57 0 2024-04-12T05:09:02.486 INFO:tasks.workunit.client.1.smithi192.stdout:6/522: creat d3/dc/fa9 x:0 0 0 2024-04-12T05:09:02.487 INFO:tasks.workunit.client.1.smithi192.stdout:3/394: chown d5/d13/d1a/d5f/l6b 21309 1 2024-04-12T05:09:02.487 INFO:tasks.workunit.client.1.smithi192.stdout:9/347: mkdir d11/d25/d36/d6e/d71 0 2024-04-12T05:09:02.489 INFO:tasks.workunit.client.1.smithi192.stdout:2/462: write d1/d8/d17/d19/f31 [1714656,13112] 0 2024-04-12T05:09:02.491 INFO:tasks.workunit.client.1.smithi192.stdout:3/395: read - d5/d13/d3a/d4a/f66 zero size 2024-04-12T05:09:02.492 INFO:tasks.workunit.client.1.smithi192.stdout:6/523: write d3/d5/f80 [4222948,130275] 0 2024-04-12T05:09:02.492 INFO:tasks.workunit.client.1.smithi192.stdout:1/431: write d0/d2/dd/d21/f2d [1213722,10669] 0 2024-04-12T05:09:02.494 INFO:tasks.workunit.client.1.smithi192.stdout:6/524: readlink d3/d5/l34 0 2024-04-12T05:09:02.494 INFO:tasks.workunit.client.1.smithi192.stdout:9/348: dread d11/d25/d36/f38 [0,4194304] 0 2024-04-12T05:09:02.495 INFO:tasks.workunit.client.1.smithi192.stdout:2/463: chown d1/d8/d17/d19/d60/f93 3018106 1 2024-04-12T05:09:02.496 INFO:tasks.workunit.client.1.smithi192.stdout:6/525: chown d3/d8/f13 35675823 1 2024-04-12T05:09:02.497 INFO:tasks.workunit.client.1.smithi192.stdout:1/432: symlink d0/d2/d1e/d3f/d56/l88 0 2024-04-12T05:09:02.498 INFO:tasks.workunit.client.1.smithi192.stdout:9/349: creat d11/d25/d36/d3d/f72 x:0 0 0 2024-04-12T05:09:02.499 INFO:tasks.workunit.client.1.smithi192.stdout:2/464: creat d1/d8/d17/d19/d71/d7f/f96 x:0 0 0 2024-04-12T05:09:02.501 INFO:tasks.workunit.client.1.smithi192.stdout:6/526: creat d3/d8/d56/d59/d8a/faa x:0 0 0 2024-04-12T05:09:02.502 INFO:tasks.workunit.client.1.smithi192.stdout:2/465: rmdir d1/d2d/d73 39 2024-04-12T05:09:02.503 INFO:tasks.workunit.client.1.smithi192.stdout:1/433: mknod d0/d2/dd/d3c/d5b/d7c/c89 0 2024-04-12T05:09:02.503 INFO:tasks.workunit.client.1.smithi192.stdout:7/385: sync 2024-04-12T05:09:02.503 INFO:tasks.workunit.client.1.smithi192.stdout:4/544: sync 2024-04-12T05:09:02.504 INFO:tasks.workunit.client.1.smithi192.stdout:6/527: mknod d3/d8/d56/d59/d8a/cab 0 2024-04-12T05:09:02.509 INFO:tasks.workunit.client.1.smithi192.stdout:2/466: creat d1/d8/d3a/de/f97 x:0 0 0 2024-04-12T05:09:02.510 INFO:tasks.workunit.client.1.smithi192.stdout:9/350: dwrite d11/d15/f45 [4194304,4194304] 0 2024-04-12T05:09:02.511 INFO:tasks.workunit.client.1.smithi192.stdout:1/434: rename d0/d1b/f6f to d0/d2/dd/d21/d62/d84/f8a 0 2024-04-12T05:09:02.513 INFO:tasks.workunit.client.1.smithi192.stdout:4/545: getdents d1/d1d/d53/db2 0 2024-04-12T05:09:02.517 INFO:tasks.workunit.client.1.smithi192.stdout:9/351: mkdir d11/d15/d21/d2c/d1f/d50/d73 0 2024-04-12T05:09:02.519 INFO:tasks.workunit.client.1.smithi192.stdout:3/396: dread d5/d13/d3c/f25 [4194304,4194304] 0 2024-04-12T05:09:02.520 INFO:tasks.workunit.client.1.smithi192.stdout:2/467: mkdir d1/d8/d3a/de/d7c/d98 0 2024-04-12T05:09:02.523 INFO:tasks.workunit.client.1.smithi192.stdout:1/435: rename d0/f1a to d0/d2/d1e/d3f/d56/d58/f8b 0 2024-04-12T05:09:02.525 INFO:tasks.workunit.client.1.smithi192.stdout:7/386: dwrite d3/d7/d14/f47 [0,4194304] 0 2024-04-12T05:09:02.528 INFO:tasks.workunit.client.1.smithi192.stdout:7/387: stat d3/f6f 0 2024-04-12T05:09:02.530 INFO:tasks.workunit.client.1.smithi192.stdout:1/436: fsync d0/d1b/f35 0 2024-04-12T05:09:02.532 INFO:tasks.workunit.client.1.smithi192.stdout:3/397: rename d5/d13/d1a/l21 to d5/d13/d3c/l89 0 2024-04-12T05:09:02.532 INFO:tasks.workunit.client.1.smithi192.stdout:2/468: read d1/d8/d3a/f2e [850314,111238] 0 2024-04-12T05:09:02.534 INFO:tasks.workunit.client.1.smithi192.stdout:4/546: dwrite d1/d54/d82/fb3 [0,4194304] 0 2024-04-12T05:09:02.537 INFO:tasks.workunit.client.1.smithi192.stdout:9/352: dwrite fb [0,4194304] 0 2024-04-12T05:09:02.537 INFO:tasks.workunit.client.1.smithi192.stdout:7/388: mkdir d3/d1d/d74/d86 0 2024-04-12T05:09:02.537 INFO:tasks.workunit.client.1.smithi192.stdout:6/528: dread d3/d23/f74 [0,4194304] 0 2024-04-12T05:09:02.540 INFO:tasks.workunit.client.1.smithi192.stdout:1/437: mknod d0/d2/dd/d3c/d5b/d74/c8c 0 2024-04-12T05:09:02.543 INFO:tasks.workunit.client.1.smithi192.stdout:3/398: link d5/f19 d5/d13/d3c/d36/f8a 0 2024-04-12T05:09:02.544 INFO:tasks.workunit.client.1.smithi192.stdout:1/438: mkdir d0/d29/d8d 0 2024-04-12T05:09:02.547 INFO:tasks.workunit.client.1.smithi192.stdout:4/547: link d1/d54/f61 d1/fb6 0 2024-04-12T05:09:02.547 INFO:tasks.workunit.client.1.smithi192.stdout:7/389: getdents d3/d1d/d5a/d5f/d7a 0 2024-04-12T05:09:02.548 INFO:tasks.workunit.client.1.smithi192.stdout:3/399: mkdir d5/d13/d3a/d8b 0 2024-04-12T05:09:02.551 INFO:tasks.workunit.client.1.smithi192.stdout:6/529: dread d3/d5/ff [0,4194304] 0 2024-04-12T05:09:02.552 INFO:tasks.workunit.client.1.smithi192.stdout:9/353: dwrite d11/d15/f5c [0,4194304] 0 2024-04-12T05:09:02.553 INFO:tasks.workunit.client.0.smithi113.stdout:1/630: sync 2024-04-12T05:09:02.553 INFO:tasks.workunit.client.0.smithi113.stdout:7/551: sync 2024-04-12T05:09:02.556 INFO:tasks.workunit.client.1.smithi192.stdout:3/400: mkdir d5/d13/d3a/d70/d8c 0 2024-04-12T05:09:02.557 INFO:tasks.workunit.client.1.smithi192.stdout:6/530: chown d3/d8/c2d 360 1 2024-04-12T05:09:02.558 INFO:tasks.workunit.client.0.smithi113.stdout:7/552: creat d0/dc/d6/fc0 x:0 0 0 2024-04-12T05:09:02.559 INFO:tasks.workunit.client.1.smithi192.stdout:6/531: truncate d3/d5/d9c/fa7 308929 0 2024-04-12T05:09:02.561 INFO:tasks.workunit.client.0.smithi113.stdout:7/553: mknod d0/dc/d6/d78/cc1 0 2024-04-12T05:09:02.562 INFO:tasks.workunit.client.1.smithi192.stdout:7/390: rename d3/d1d/d5a/d5f/c54 to d3/d1d/d48/d7b/c87 0 2024-04-12T05:09:02.564 INFO:tasks.workunit.client.0.smithi113.stdout:7/554: fsync d0/dc/da9/f47 0 2024-04-12T05:09:02.564 INFO:tasks.workunit.client.1.smithi192.stdout:2/469: dread d1/d8/f22 [0,4194304] 0 2024-04-12T05:09:02.567 INFO:tasks.workunit.client.1.smithi192.stdout:4/548: getdents d1/d54/d82/da4 0 2024-04-12T05:09:02.568 INFO:tasks.workunit.client.1.smithi192.stdout:3/401: mkdir d5/d13/d3a/d70/d8c/d8d 0 2024-04-12T05:09:02.570 INFO:tasks.workunit.client.1.smithi192.stdout:4/549: stat d1/d6b/f7e 0 2024-04-12T05:09:02.570 INFO:tasks.workunit.client.1.smithi192.stdout:9/354: creat d11/d15/d21/d2c/d1f/d50/d73/f74 x:0 0 0 2024-04-12T05:09:02.570 INFO:tasks.workunit.client.1.smithi192.stdout:6/532: mkdir d3/dac 0 2024-04-12T05:09:02.571 INFO:tasks.workunit.client.1.smithi192.stdout:7/391: mknod d3/d7/d14/d1f/d76/c88 0 2024-04-12T05:09:02.575 INFO:tasks.workunit.client.1.smithi192.stdout:4/550: chown d1/d54/f6e 75 1 2024-04-12T05:09:02.577 INFO:tasks.workunit.client.0.smithi113.stdout:7/555: truncate d0/dc/d6/d9b/f25 3590180 0 2024-04-12T05:09:02.577 INFO:tasks.workunit.client.1.smithi192.stdout:6/533: rmdir d3/d23 39 2024-04-12T05:09:02.577 INFO:tasks.workunit.client.1.smithi192.stdout:7/392: unlink d3/d1d/d5a/d5f/f80 0 2024-04-12T05:09:02.578 INFO:tasks.workunit.client.0.smithi113.stdout:7/556: dread - d0/dc/d6/d1a/f7e zero size 2024-04-12T05:09:02.579 INFO:tasks.workunit.client.1.smithi192.stdout:1/439: dwrite d0/d1b/f53 [0,4194304] 0 2024-04-12T05:09:02.580 INFO:tasks.workunit.client.1.smithi192.stdout:9/355: creat d11/d15/d21/d2c/d1f/d50/f75 x:0 0 0 2024-04-12T05:09:02.581 INFO:tasks.workunit.client.0.smithi113.stdout:7/557: chown d0/dc/d6/d9b/f62 283606463 1 2024-04-12T05:09:02.585 INFO:tasks.workunit.client.0.smithi113.stdout:7/558: creat d0/dd/d11/fc2 x:0 0 0 2024-04-12T05:09:02.586 INFO:tasks.workunit.client.1.smithi192.stdout:6/534: dread - d3/d8/d22/f7c zero size 2024-04-12T05:09:02.588 INFO:tasks.workunit.client.1.smithi192.stdout:3/402: dwrite d5/f55 [0,4194304] 0 2024-04-12T05:09:02.591 INFO:tasks.workunit.client.1.smithi192.stdout:6/535: readlink d3/d8/d10/d4c/d68/l7b 0 2024-04-12T05:09:02.591 INFO:tasks.workunit.client.0.smithi113.stdout:7/559: dwrite d0/dc/d50/f72 [0,4194304] 0 2024-04-12T05:09:02.592 INFO:tasks.workunit.client.1.smithi192.stdout:1/440: truncate d0/d1b/d28/f81 160373 0 2024-04-12T05:09:02.592 INFO:tasks.workunit.client.0.smithi113.stdout:7/560: dread - d0/dc/f83 zero size 2024-04-12T05:09:02.593 INFO:tasks.workunit.client.0.smithi113.stdout:7/561: read - d0/dd/d11/f77 zero size 2024-04-12T05:09:02.594 INFO:tasks.workunit.client.1.smithi192.stdout:2/470: dread d1/d8/d3a/de/f85 [0,4194304] 0 2024-04-12T05:09:02.596 INFO:tasks.workunit.client.1.smithi192.stdout:4/551: truncate d1/d54/f6e 352053 0 2024-04-12T05:09:02.596 INFO:tasks.workunit.client.1.smithi192.stdout:6/536: dread - d3/d8/d10/d4c/fa6 zero size 2024-04-12T05:09:02.597 INFO:tasks.workunit.client.1.smithi192.stdout:0/580: sync 2024-04-12T05:09:02.597 INFO:tasks.workunit.client.1.smithi192.stdout:1/441: write d0/d1b/f53 [1845938,23812] 0 2024-04-12T05:09:02.597 INFO:tasks.workunit.client.0.smithi113.stdout:7/562: mknod d0/cc3 0 2024-04-12T05:09:02.598 INFO:tasks.workunit.client.1.smithi192.stdout:7/393: dwrite d3/f21 [0,4194304] 0 2024-04-12T05:09:02.600 INFO:tasks.workunit.client.1.smithi192.stdout:9/356: dread d11/d15/d1e/f39 [0,4194304] 0 2024-04-12T05:09:02.602 INFO:tasks.workunit.client.0.smithi113.stdout:7/563: read d0/dc/d6/d24/f71 [29201,97156] 0 2024-04-12T05:09:02.604 INFO:tasks.workunit.client.1.smithi192.stdout:6/537: chown d3/d8/c98 85 1 2024-04-12T05:09:02.607 INFO:tasks.workunit.client.0.smithi113.stdout:7/564: dread - d0/dc/d6/d9b/f84 zero size 2024-04-12T05:09:02.608 INFO:tasks.workunit.client.0.smithi113.stdout:7/565: write d0/dc/f83 [724343,87862] 0 2024-04-12T05:09:02.608 INFO:tasks.workunit.client.1.smithi192.stdout:7/394: write d3/f21 [2832162,28248] 0 2024-04-12T05:09:02.611 INFO:tasks.workunit.client.1.smithi192.stdout:0/581: write d0/d7b/d9b/d4b/d72/f90 [88627,73495] 0 2024-04-12T05:09:02.611 INFO:tasks.workunit.client.1.smithi192.stdout:3/403: dwrite d5/d13/d1a/d53/f60 [0,4194304] 0 2024-04-12T05:09:02.612 INFO:tasks.workunit.client.1.smithi192.stdout:1/442: creat d0/d2/dd/d21/d62/d84/f8e x:0 0 0 2024-04-12T05:09:02.613 INFO:tasks.workunit.client.1.smithi192.stdout:4/552: unlink d1/d54/d55/d64/d68/l98 0 2024-04-12T05:09:02.615 INFO:tasks.workunit.client.1.smithi192.stdout:0/582: chown d0/d6/d15/f6f 7202 1 2024-04-12T05:09:02.615 INFO:tasks.workunit.client.1.smithi192.stdout:3/404: stat d5/d13/f2a 0 2024-04-12T05:09:02.617 INFO:tasks.workunit.client.1.smithi192.stdout:7/395: creat d3/d7/f89 x:0 0 0 2024-04-12T05:09:02.618 INFO:tasks.workunit.client.1.smithi192.stdout:4/553: unlink d1/d54/d55/d64/l50 0 2024-04-12T05:09:02.618 INFO:tasks.workunit.client.1.smithi192.stdout:9/357: link d11/l51 d11/d25/d60/l76 0 2024-04-12T05:09:02.619 INFO:tasks.workunit.client.1.smithi192.stdout:2/471: creat d1/d8/d17/d19/d60/f99 x:0 0 0 2024-04-12T05:09:02.621 INFO:tasks.workunit.client.1.smithi192.stdout:6/538: rename d3/l1b to d3/d8/d56/d59/lad 0 2024-04-12T05:09:02.622 INFO:tasks.workunit.client.1.smithi192.stdout:0/583: mknod d0/d6/d15/d19/cc1 0 2024-04-12T05:09:02.622 INFO:tasks.workunit.client.1.smithi192.stdout:6/539: chown d3/d8/fa3 729063098 1 2024-04-12T05:09:02.622 INFO:tasks.workunit.client.1.smithi192.stdout:3/405: mknod d5/d13/d3a/c8e 0 2024-04-12T05:09:02.623 INFO:tasks.workunit.client.1.smithi192.stdout:7/396: creat d3/d7/d60/d65/f8a x:0 0 0 2024-04-12T05:09:02.625 INFO:tasks.workunit.client.1.smithi192.stdout:7/397: dread - d3/d7/d14/f58 zero size 2024-04-12T05:09:02.627 INFO:tasks.workunit.client.1.smithi192.stdout:6/540: chown d3/d8/d10/d4c/f62 55 1 2024-04-12T05:09:02.628 INFO:tasks.workunit.client.1.smithi192.stdout:3/406: symlink d5/d13/d3a/d4a/d61/l8f 0 2024-04-12T05:09:02.628 INFO:tasks.workunit.client.1.smithi192.stdout:2/472: dread d1/d8/d3a/de/f57 [0,4194304] 0 2024-04-12T05:09:02.629 INFO:tasks.workunit.client.1.smithi192.stdout:7/398: chown d3/d7/d14/d1f/c20 35720 1 2024-04-12T05:09:02.631 INFO:tasks.workunit.client.1.smithi192.stdout:0/584: mknod d0/d7b/d9b/d4b/dbc/cc2 0 2024-04-12T05:09:02.631 INFO:tasks.workunit.client.1.smithi192.stdout:5/431: sync 2024-04-12T05:09:02.631 INFO:tasks.workunit.client.1.smithi192.stdout:8/381: sync 2024-04-12T05:09:02.636 INFO:tasks.workunit.client.1.smithi192.stdout:4/554: read d1/d54/f6e [203190,55264] 0 2024-04-12T05:09:02.637 INFO:tasks.workunit.client.1.smithi192.stdout:7/399: symlink d3/d1d/d5a/l8b 0 2024-04-12T05:09:02.638 INFO:tasks.workunit.client.1.smithi192.stdout:5/432: creat d0/f96 x:0 0 0 2024-04-12T05:09:02.638 INFO:tasks.workunit.client.1.smithi192.stdout:3/407: dread d5/d13/d3a/d4e/f6c [0,4194304] 0 2024-04-12T05:09:02.639 INFO:tasks.workunit.client.1.smithi192.stdout:6/541: symlink d3/d8/d10/d4c/lae 0 2024-04-12T05:09:02.647 INFO:tasks.workunit.client.1.smithi192.stdout:0/585: creat d0/d4e/d53/fc3 x:0 0 0 2024-04-12T05:09:02.648 INFO:tasks.workunit.client.1.smithi192.stdout:1/443: dread d0/d1b/f35 [0,4194304] 0 2024-04-12T05:09:02.649 INFO:tasks.workunit.client.1.smithi192.stdout:8/382: rename d6/d10/d46/d5e/d5f/d37/d4f/d51/f63 to d6/d10/d46/d5e/d5f/d31/d61/f74 0 2024-04-12T05:09:02.650 INFO:tasks.workunit.client.1.smithi192.stdout:4/555: fdatasync d1/d1d/d53/d1c/d84/fa9 0 2024-04-12T05:09:02.653 INFO:tasks.workunit.client.1.smithi192.stdout:0/586: dread - d0/d6/d15/d55/d2e/d66/db7/f97 zero size 2024-04-12T05:09:02.656 INFO:tasks.workunit.client.1.smithi192.stdout:7/400: dwrite d3/d7/f7e [0,4194304] 0 2024-04-12T05:09:02.658 INFO:tasks.workunit.client.1.smithi192.stdout:9/358: dwrite d11/d25/d36/d6e/f3e [0,4194304] 0 2024-04-12T05:09:02.660 INFO:tasks.workunit.client.1.smithi192.stdout:9/359: fsync d11/d25/f70 0 2024-04-12T05:09:02.660 INFO:tasks.workunit.client.1.smithi192.stdout:7/401: truncate d3/d11/f37 539969 0 2024-04-12T05:09:02.663 INFO:tasks.workunit.client.1.smithi192.stdout:9/360: write d11/d15/d21/d2c/d1f/d50/f75 [80179,123453] 0 2024-04-12T05:09:02.663 INFO:tasks.workunit.client.1.smithi192.stdout:0/587: dwrite d0/d7b/d9b/f38 [0,4194304] 0 2024-04-12T05:09:02.668 INFO:tasks.workunit.client.1.smithi192.stdout:7/402: dwrite d3/d1d/d5a/d5f/f30 [0,4194304] 0 2024-04-12T05:09:02.669 INFO:tasks.workunit.client.1.smithi192.stdout:7/403: dread - d3/d7/d60/d65/f8a zero size 2024-04-12T05:09:02.671 INFO:tasks.workunit.client.1.smithi192.stdout:7/404: dread d3/d11/f5c [0,4194304] 0 2024-04-12T05:09:02.698 INFO:tasks.workunit.client.1.smithi192.stdout:2/473: dwrite d1/d8/d17/d19/d40/f5b [0,4194304] 0 2024-04-12T05:09:02.703 INFO:tasks.workunit.client.1.smithi192.stdout:3/408: creat d5/d13/d3a/d4a/f90 x:0 0 0 2024-04-12T05:09:02.705 INFO:tasks.workunit.client.1.smithi192.stdout:4/556: symlink d1/d1d/d53/lb7 0 2024-04-12T05:09:02.708 INFO:tasks.workunit.client.1.smithi192.stdout:1/444: rename d0/d2/dd/d21/d33/f3a to d0/d29/d8d/f8f 0 2024-04-12T05:09:02.709 INFO:tasks.workunit.client.1.smithi192.stdout:1/445: dread - d0/d1b/d44/f6e zero size 2024-04-12T05:09:02.710 INFO:tasks.workunit.client.1.smithi192.stdout:9/361: creat d11/d25/d36/d6e/f77 x:0 0 0 2024-04-12T05:09:02.711 INFO:tasks.workunit.client.0.smithi113.stdout:5/556: sync 2024-04-12T05:09:02.711 INFO:tasks.workunit.client.0.smithi113.stdout:3/532: sync 2024-04-12T05:09:02.711 INFO:tasks.workunit.client.0.smithi113.stdout:4/540: sync 2024-04-12T05:09:02.712 INFO:tasks.workunit.client.0.smithi113.stdout:9/599: sync 2024-04-12T05:09:02.712 INFO:tasks.workunit.client.0.smithi113.stdout:0/614: sync 2024-04-12T05:09:02.712 INFO:tasks.workunit.client.0.smithi113.stdout:2/453: sync 2024-04-12T05:09:02.712 INFO:tasks.workunit.client.0.smithi113.stdout:8/613: sync 2024-04-12T05:09:02.712 INFO:tasks.workunit.client.0.smithi113.stdout:6/538: sync 2024-04-12T05:09:02.713 INFO:tasks.workunit.client.1.smithi192.stdout:2/474: fdatasync d1/d8/d3a/f2e 0 2024-04-12T05:09:02.714 INFO:tasks.workunit.client.0.smithi113.stdout:8/614: chown d9/d1b/d96 38186322 1 2024-04-12T05:09:02.714 INFO:tasks.workunit.client.1.smithi192.stdout:6/542: truncate d3/d23/f78 693464 0 2024-04-12T05:09:02.715 INFO:tasks.workunit.client.0.smithi113.stdout:6/539: chown d8/de/d17/f27 89562 1 2024-04-12T05:09:02.715 INFO:tasks.workunit.client.0.smithi113.stdout:9/600: write d1/d4/d30/f75 [461253,24917] 0 2024-04-12T05:09:02.716 INFO:tasks.workunit.client.1.smithi192.stdout:5/433: symlink d0/d15/d3e/l97 0 2024-04-12T05:09:02.716 INFO:tasks.workunit.client.0.smithi113.stdout:2/454: chown d8/d9/de/d15/d24/d58/f84 1 1 2024-04-12T05:09:02.718 INFO:tasks.workunit.client.0.smithi113.stdout:5/557: link d0/f98 d0/d10/d6e/fb4 0 2024-04-12T05:09:02.718 INFO:tasks.workunit.client.1.smithi192.stdout:3/409: stat d5/d13/f2f 0 2024-04-12T05:09:02.720 INFO:tasks.workunit.client.0.smithi113.stdout:4/541: rename da/d24/d7e/c8d to da/d2b/cae 0 2024-04-12T05:09:02.721 INFO:tasks.workunit.client.1.smithi192.stdout:8/383: rename d6/d10/f3f to d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/f75 0 2024-04-12T05:09:02.723 INFO:tasks.workunit.client.0.smithi113.stdout:6/540: mknod d8/de/d13/d2b/db3/cc3 0 2024-04-12T05:09:02.726 INFO:tasks.workunit.client.0.smithi113.stdout:2/455: mknod d8/d9/de/d12/d44/c86 0 2024-04-12T05:09:02.726 INFO:tasks.workunit.client.1.smithi192.stdout:5/434: dwrite d0/d15/d3e/f71 [0,4194304] 0 2024-04-12T05:09:02.726 INFO:tasks.workunit.client.1.smithi192.stdout:8/384: write d6/d1f/d6a/f52 [636368,59976] 0 2024-04-12T05:09:02.727 INFO:tasks.workunit.client.1.smithi192.stdout:8/385: readlink d6/l17 0 2024-04-12T05:09:02.729 INFO:tasks.workunit.client.0.smithi113.stdout:3/533: rename d0/d13/d15/f39 to d0/d7e/da3/fb9 0 2024-04-12T05:09:02.729 INFO:tasks.workunit.client.1.smithi192.stdout:5/435: dread - d0/d15/d32/f6a zero size 2024-04-12T05:09:02.730 INFO:tasks.workunit.client.0.smithi113.stdout:0/615: link d1/d6/d16/d2a/d50/d52/d7e/db5/f7b d1/d6/d16/d2a/d50/d52/fbf 0 2024-04-12T05:09:02.732 INFO:tasks.workunit.client.0.smithi113.stdout:9/601: symlink d1/d4/d5/lcc 0 2024-04-12T05:09:02.733 INFO:tasks.workunit.client.1.smithi192.stdout:6/543: mknod d3/d21/d24/d79/caf 0 2024-04-12T05:09:02.733 INFO:tasks.workunit.client.1.smithi192.stdout:5/436: dread d0/f61 [0,4194304] 0 2024-04-12T05:09:02.733 INFO:tasks.workunit.client.1.smithi192.stdout:5/437: read d0/f61 [441663,20508] 0 2024-04-12T05:09:02.737 INFO:tasks.workunit.client.0.smithi113.stdout:0/616: dwrite d1/d9/f8a [0,4194304] 0 2024-04-12T05:09:02.738 INFO:tasks.workunit.client.0.smithi113.stdout:5/558: rename d0/d10/d27/d1d/d38/d44/f92 to d0/d10/d6e/fb5 0 2024-04-12T05:09:02.739 INFO:tasks.workunit.client.0.smithi113.stdout:3/534: mknod d0/d7e/da3/cba 0 2024-04-12T05:09:02.740 INFO:tasks.workunit.client.1.smithi192.stdout:2/475: fsync d1/d8/d3a/f3b 0 2024-04-12T05:09:02.740 INFO:tasks.workunit.client.0.smithi113.stdout:5/559: chown d0/d10/d6e/f87 93840 1 2024-04-12T05:09:02.741 INFO:tasks.workunit.client.0.smithi113.stdout:4/542: link da/dd/f32 da/d24/d7e/faf 0 2024-04-12T05:09:02.742 INFO:tasks.workunit.client.0.smithi113.stdout:3/535: fsync d0/d13/d15/d28/d2a/d34/f84 0 2024-04-12T05:09:02.743 INFO:tasks.workunit.client.1.smithi192.stdout:8/386: symlink d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/l76 0 2024-04-12T05:09:02.744 INFO:tasks.workunit.client.0.smithi113.stdout:0/617: write d1/d6/d16/d2a/d50/d52/d7e/db5/f7b [4618243,37191] 0 2024-04-12T05:09:02.745 INFO:tasks.workunit.client.0.smithi113.stdout:2/456: rename d8/d9/l68 to d8/d9/de/d43/l87 0 2024-04-12T05:09:02.748 INFO:tasks.workunit.client.0.smithi113.stdout:0/618: dread d1/d6/d16/d8f/fb9 [0,4194304] 0 2024-04-12T05:09:02.749 INFO:tasks.workunit.client.0.smithi113.stdout:0/619: truncate d1/d6/fb0 1252421 0 2024-04-12T05:09:02.750 INFO:tasks.workunit.client.0.smithi113.stdout:8/615: write d9/d1b/d8e/d93/d32/f3f [1665962,11772] 0 2024-04-12T05:09:02.752 INFO:tasks.workunit.client.1.smithi192.stdout:8/387: creat d6/d10/d46/f77 x:0 0 0 2024-04-12T05:09:02.752 INFO:tasks.workunit.client.1.smithi192.stdout:2/476: read d1/d8/f10 [1678425,6566] 0 2024-04-12T05:09:02.752 INFO:tasks.workunit.client.1.smithi192.stdout:9/362: write d11/d25/d36/d6e/d33/f54 [907290,46705] 0 2024-04-12T05:09:02.753 INFO:tasks.workunit.client.1.smithi192.stdout:9/363: stat f3 0 2024-04-12T05:09:02.753 INFO:tasks.workunit.client.0.smithi113.stdout:5/560: mknod d0/d10/d6e/cb6 0 2024-04-12T05:09:02.753 INFO:tasks.workunit.client.0.smithi113.stdout:3/536: dread - d0/d13/d15/f75 zero size 2024-04-12T05:09:02.755 INFO:tasks.workunit.client.0.smithi113.stdout:4/543: write da/d4f/f5d [5049963,88721] 0 2024-04-12T05:09:02.756 INFO:tasks.workunit.client.0.smithi113.stdout:8/616: mknod d9/d16/d65/d8c/cc3 0 2024-04-12T05:09:02.756 INFO:tasks.workunit.client.1.smithi192.stdout:0/588: rename d0/d6/d15/d1b/d3a/d79 to d0/d6/dc4 0 2024-04-12T05:09:02.758 INFO:tasks.workunit.client.1.smithi192.stdout:5/438: creat d0/d15/d64/d7c/f98 x:0 0 0 2024-04-12T05:09:02.760 INFO:tasks.workunit.client.0.smithi113.stdout:0/620: creat d1/d6/d16/d2a/d50/d52/d7e/db5/dbb/fc0 x:0 0 0 2024-04-12T05:09:02.760 INFO:tasks.workunit.client.1.smithi192.stdout:9/364: dwrite d11/f69 [0,4194304] 0 2024-04-12T05:09:02.762 INFO:tasks.workunit.client.1.smithi192.stdout:8/388: creat d6/d10/f78 x:0 0 0 2024-04-12T05:09:02.763 INFO:tasks.workunit.client.1.smithi192.stdout:8/389: readlink d6/d10/d46/d5e/d5f/d31/l29 0 2024-04-12T05:09:02.767 INFO:tasks.workunit.client.0.smithi113.stdout:6/541: truncate d8/d45/f49 3210642 0 2024-04-12T05:09:02.767 INFO:tasks.workunit.client.0.smithi113.stdout:0/621: rename d1/d6/d16/fa9 to d1/d43/d59/db2/fc1 0 2024-04-12T05:09:02.768 INFO:tasks.workunit.client.0.smithi113.stdout:3/537: dread d0/d7e/f85 [0,4194304] 0 2024-04-12T05:09:02.769 INFO:tasks.workunit.client.1.smithi192.stdout:1/446: dwrite d0/d2/dd/f1d [0,4194304] 0 2024-04-12T05:09:02.769 INFO:tasks.workunit.client.0.smithi113.stdout:3/538: write d0/fa6 [606908,9511] 0 2024-04-12T05:09:02.770 INFO:tasks.workunit.client.1.smithi192.stdout:1/447: read d0/d1b/f53 [2402505,79351] 0 2024-04-12T05:09:02.770 INFO:tasks.workunit.client.0.smithi113.stdout:5/561: write d0/d10/f56 [1327400,23228] 0 2024-04-12T05:09:02.770 INFO:tasks.workunit.client.1.smithi192.stdout:7/405: rename d3/d7/d15/f67 to d3/d1d/d74/f8c 0 2024-04-12T05:09:02.775 INFO:tasks.workunit.client.1.smithi192.stdout:5/439: fsync d0/d15/d32/d45/f7f 0 2024-04-12T05:09:02.775 INFO:tasks.workunit.client.1.smithi192.stdout:6/544: write d3/d8/d48/f63 [237849,52710] 0 2024-04-12T05:09:02.776 INFO:tasks.workunit.client.0.smithi113.stdout:6/542: mkdir d8/de/d13/d11/d36/d53/db8/d63/d66/dc4 0 2024-04-12T05:09:02.776 INFO:tasks.workunit.client.0.smithi113.stdout:5/562: symlink d0/d10/da1/lb7 0 2024-04-12T05:09:02.777 INFO:tasks.workunit.client.0.smithi113.stdout:6/543: chown d8/de/d17/c84 178042 1 2024-04-12T05:09:02.777 INFO:tasks.workunit.client.1.smithi192.stdout:0/589: mkdir d0/dc5 0 2024-04-12T05:09:02.779 INFO:tasks.workunit.client.0.smithi113.stdout:5/563: chown d0/d10/d27/d1d/d8c 1280969265 1 2024-04-12T05:09:02.779 INFO:tasks.workunit.client.0.smithi113.stdout:6/544: chown d8/de/d13/d11/d36/l20 975 1 2024-04-12T05:09:02.781 INFO:tasks.workunit.client.1.smithi192.stdout:1/448: fsync d0/f20 0 2024-04-12T05:09:02.784 INFO:tasks.workunit.client.1.smithi192.stdout:4/557: rename d1/d1d/d53/d1c/l93 to d1/d54/d82/da4/da8/lb8 0 2024-04-12T05:09:02.785 INFO:tasks.workunit.client.1.smithi192.stdout:4/558: write d1/d1d/d53/d1c/f4e [701567,96635] 0 2024-04-12T05:09:02.786 INFO:tasks.workunit.client.0.smithi113.stdout:0/622: getdents d1/d6/d16/d2a/d50/d52 0 2024-04-12T05:09:02.791 INFO:tasks.workunit.client.0.smithi113.stdout:5/564: dwrite d0/d10/d27/d1d/d3f/f47 [0,4194304] 0 2024-04-12T05:09:02.791 INFO:tasks.workunit.client.1.smithi192.stdout:9/365: creat d11/d25/d36/d6e/d71/f78 x:0 0 0 2024-04-12T05:09:02.792 INFO:tasks.workunit.client.0.smithi113.stdout:3/539: dwrite d0/d13/fa9 [0,4194304] 0 2024-04-12T05:09:02.797 INFO:tasks.workunit.client.0.smithi113.stdout:5/565: unlink d0/d10/d27/f25 0 2024-04-12T05:09:02.797 INFO:tasks.workunit.client.0.smithi113.stdout:5/566: fdatasync d0/f55 0 2024-04-12T05:09:02.798 INFO:tasks.workunit.client.0.smithi113.stdout:5/567: write d0/f55 [790250,32392] 0 2024-04-12T05:09:02.802 INFO:tasks.workunit.client.0.smithi113.stdout:3/540: creat d0/d13/d15/d28/d2a/fbb x:0 0 0 2024-04-12T05:09:02.803 INFO:tasks.workunit.client.0.smithi113.stdout:5/568: mkdir d0/d10/d27/d1d/d8c/db8 0 2024-04-12T05:09:02.804 INFO:tasks.workunit.client.1.smithi192.stdout:8/390: symlink d6/d10/d46/d5e/d6e/l79 0 2024-04-12T05:09:02.804 INFO:tasks.workunit.client.1.smithi192.stdout:5/440: chown d0/d15/d24/f25 15760211 1 2024-04-12T05:09:02.805 INFO:tasks.workunit.client.0.smithi113.stdout:3/541: write d0/d13/d47/fb2 [607656,105271] 0 2024-04-12T05:09:02.805 INFO:tasks.workunit.client.1.smithi192.stdout:0/590: write d0/d6/d15/d55/d2e/d59/f78 [110369,129393] 0 2024-04-12T05:09:02.805 INFO:tasks.workunit.client.0.smithi113.stdout:5/569: fdatasync d0/d10/d27/d1d/d38/d44/fa3 0 2024-04-12T05:09:02.809 INFO:tasks.workunit.client.1.smithi192.stdout:4/559: mknod d1/d54/d55/d64/cb9 0 2024-04-12T05:09:02.810 INFO:tasks.workunit.client.0.smithi113.stdout:3/542: write d0/d13/d15/d28/d2a/d34/d5f/f96 [136463,73184] 0 2024-04-12T05:09:02.810 INFO:tasks.workunit.client.0.smithi113.stdout:0/623: unlink d1/d6/d16/d2a/f71 0 2024-04-12T05:09:02.811 INFO:tasks.workunit.client.1.smithi192.stdout:2/477: getdents d1/d8/d17/d19/d71/d7f 0 2024-04-12T05:09:02.812 INFO:tasks.workunit.client.0.smithi113.stdout:6/545: dread d8/d45/f49 [0,4194304] 0 2024-04-12T05:09:02.812 INFO:tasks.workunit.client.0.smithi113.stdout:0/624: truncate d1/d6/fb0 1656229 0 2024-04-12T05:09:02.813 INFO:tasks.workunit.client.1.smithi192.stdout:2/478: write d1/d8/d17/d19/f31 [1954275,1770] 0 2024-04-12T05:09:02.813 INFO:tasks.workunit.client.1.smithi192.stdout:1/449: dread d0/d2/f50 [0,4194304] 0 2024-04-12T05:09:02.814 INFO:tasks.workunit.client.0.smithi113.stdout:0/625: chown d1/d6/d16/d2a/d50/db6/d36/f74 74755065 1 2024-04-12T05:09:02.814 INFO:tasks.workunit.client.1.smithi192.stdout:2/479: stat d1/d8/d3a/c7e 0 2024-04-12T05:09:02.815 INFO:tasks.workunit.client.1.smithi192.stdout:8/391: symlink d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/l7a 0 2024-04-12T05:09:02.815 INFO:tasks.workunit.client.1.smithi192.stdout:7/406: link d3/d7/d15/d22/c5b d3/d1d/d4a/c8d 0 2024-04-12T05:09:02.816 INFO:tasks.workunit.client.1.smithi192.stdout:2/480: stat d1/d8/d17/d19/d60/d59/d68 0 2024-04-12T05:09:02.816 INFO:tasks.workunit.client.1.smithi192.stdout:7/407: chown d3/d7/d15 16 1 2024-04-12T05:09:02.817 INFO:tasks.workunit.client.0.smithi113.stdout:6/546: dread d8/d45/f49 [0,4194304] 0 2024-04-12T05:09:02.818 INFO:tasks.workunit.client.1.smithi192.stdout:7/408: write d3/d7/d14/f47 [4498037,48918] 0 2024-04-12T05:09:02.818 INFO:tasks.workunit.client.1.smithi192.stdout:9/366: link d11/d15/d1e/f39 d11/d25/d36/d3d/f79 0 2024-04-12T05:09:02.820 INFO:tasks.workunit.client.0.smithi113.stdout:0/626: mknod d1/d43/d59/cc2 0 2024-04-12T05:09:02.821 INFO:tasks.workunit.client.1.smithi192.stdout:6/545: creat d3/d8/d10/d4c/fb0 x:0 0 0 2024-04-12T05:09:02.821 INFO:tasks.workunit.client.0.smithi113.stdout:6/547: truncate d8/de/d13/d11/d36/d53/db8/f9d 733593 0 2024-04-12T05:09:02.823 INFO:tasks.workunit.client.1.smithi192.stdout:1/450: unlink d0/d2/dd/d71/l73 0 2024-04-12T05:09:02.824 INFO:tasks.workunit.client.0.smithi113.stdout:6/548: write d8/de/d13/d11/d36/d3d/d4c/d31/f62 [414484,7901] 0 2024-04-12T05:09:02.824 INFO:tasks.workunit.client.1.smithi192.stdout:1/451: stat d0/d2/c6c 0 2024-04-12T05:09:02.826 INFO:tasks.workunit.client.0.smithi113.stdout:5/570: dread d0/d10/d27/d1d/d38/d44/f48 [0,4194304] 0 2024-04-12T05:09:02.826 INFO:tasks.workunit.client.1.smithi192.stdout:4/560: fdatasync d1/d54/d55/d64/f8f 0 2024-04-12T05:09:02.827 INFO:tasks.workunit.client.1.smithi192.stdout:1/452: dread - d0/d2/dd/d21/d62/d84/f8a zero size 2024-04-12T05:09:02.827 INFO:tasks.workunit.client.1.smithi192.stdout:5/441: dread d0/d21/f33 [0,4194304] 0 2024-04-12T05:09:02.827 INFO:tasks.workunit.client.1.smithi192.stdout:8/392: mkdir d6/d10/d46/d5e/d5f/d37/d7b 0 2024-04-12T05:09:02.827 INFO:tasks.workunit.client.0.smithi113.stdout:5/571: fdatasync d0/d10/d6e/f3c 0 2024-04-12T05:09:02.828 INFO:tasks.workunit.client.1.smithi192.stdout:1/453: write d0/d2/dd/f4b [667595,55765] 0 2024-04-12T05:09:02.829 INFO:tasks.workunit.client.0.smithi113.stdout:5/572: fsync d0/d10/d27/d1d/d38/d3b/f61 0 2024-04-12T05:09:02.830 INFO:tasks.workunit.client.1.smithi192.stdout:8/393: chown d6/d10/d46/d5e/d5f/d38 5 1 2024-04-12T05:09:02.832 INFO:tasks.workunit.client.1.smithi192.stdout:9/367: mkdir d11/d25/d36/d3d/d7a 0 2024-04-12T05:09:02.832 INFO:tasks.workunit.client.1.smithi192.stdout:8/394: dread - d6/d10/d46/d5e/d5f/d38/d3b/f73 zero size 2024-04-12T05:09:02.835 INFO:tasks.workunit.client.1.smithi192.stdout:0/591: symlink d0/d6/d15/d19/lc6 0 2024-04-12T05:09:02.836 INFO:tasks.workunit.client.0.smithi113.stdout:0/627: dread d1/d6/d16/d2a/d50/d52/d7e/db5/f7b [0,4194304] 0 2024-04-12T05:09:02.840 INFO:tasks.workunit.client.0.smithi113.stdout:0/628: rmdir d1/d6/d16/d2a/d50/d52/d7e/db5 39 2024-04-12T05:09:02.841 INFO:tasks.workunit.client.1.smithi192.stdout:5/442: dwrite d0/d15/d24/d47/d7b/f88 [0,4194304] 0 2024-04-12T05:09:02.844 INFO:tasks.workunit.client.1.smithi192.stdout:2/481: symlink d1/d8/d17/d19/d60/d59/l9a 0 2024-04-12T05:09:02.847 INFO:tasks.workunit.client.1.smithi192.stdout:7/409: creat d3/d11/d68/f8e x:0 0 0 2024-04-12T05:09:02.849 INFO:tasks.workunit.client.1.smithi192.stdout:9/368: dread d11/d15/f45 [4194304,4194304] 0 2024-04-12T05:09:02.852 INFO:tasks.workunit.client.1.smithi192.stdout:7/410: fdatasync d3/d7/d15/f7d 0 2024-04-12T05:09:02.854 INFO:tasks.workunit.client.1.smithi192.stdout:8/395: dwrite d6/d10/d46/d5e/d5f/d31/f39 [0,4194304] 0 2024-04-12T05:09:02.855 INFO:tasks.workunit.client.1.smithi192.stdout:5/443: mknod d0/d15/d32/c99 0 2024-04-12T05:09:02.855 INFO:tasks.workunit.client.1.smithi192.stdout:0/592: dwrite d0/d6/d67/f8a [0,4194304] 0 2024-04-12T05:09:02.858 INFO:tasks.workunit.client.1.smithi192.stdout:6/546: rmdir d3/dac 0 2024-04-12T05:09:02.858 INFO:tasks.workunit.client.1.smithi192.stdout:7/411: dread - d3/d11/d68/f8e zero size 2024-04-12T05:09:02.858 INFO:tasks.workunit.client.1.smithi192.stdout:4/561: link d1/l77 d1/d49/lba 0 2024-04-12T05:09:02.858 INFO:tasks.workunit.client.1.smithi192.stdout:8/396: chown d6/da/fd 3 1 2024-04-12T05:09:02.859 INFO:tasks.workunit.client.0.smithi113.stdout:0/629: truncate d1/d9/f1a 1250694 0 2024-04-12T05:09:02.860 INFO:tasks.workunit.client.1.smithi192.stdout:2/482: creat d1/d2d/d73/f9b x:0 0 0 2024-04-12T05:09:02.862 INFO:tasks.workunit.client.1.smithi192.stdout:1/454: dread d0/d2/dd/d21/f38 [4194304,4194304] 0 2024-04-12T05:09:02.863 INFO:tasks.workunit.client.1.smithi192.stdout:7/412: write d3/d7/d14/f47 [2042046,71377] 0 2024-04-12T05:09:02.864 INFO:tasks.workunit.client.1.smithi192.stdout:0/593: write d0/d6/d15/d55/d2e/d66/db7/f85 [2319803,71999] 0 2024-04-12T05:09:02.865 INFO:tasks.workunit.client.1.smithi192.stdout:9/369: rename d11/d25/d36/d3d/f72 to d11/d15/d1e/f7b 0 2024-04-12T05:09:02.865 INFO:tasks.workunit.client.1.smithi192.stdout:8/397: chown d6/d10/d46/d5e/d5f/d37/d4f/d4c/f55 134158 1 2024-04-12T05:09:02.867 INFO:tasks.workunit.client.1.smithi192.stdout:4/562: mkdir d1/d1d/d53/d1c/d84/dbb 0 2024-04-12T05:09:02.868 INFO:tasks.workunit.client.1.smithi192.stdout:6/547: truncate d3/d23/f76 913150 0 2024-04-12T05:09:02.871 INFO:tasks.workunit.client.1.smithi192.stdout:4/563: write d1/d54/d82/da4/f92 [3937667,34797] 0 2024-04-12T05:09:02.873 INFO:tasks.workunit.client.1.smithi192.stdout:1/455: unlink d0/d2/dd/d21/c4c 0 2024-04-12T05:09:02.874 INFO:tasks.workunit.client.1.smithi192.stdout:5/444: dwrite d0/f61 [0,4194304] 0 2024-04-12T05:09:02.876 INFO:tasks.workunit.client.1.smithi192.stdout:7/413: mknod d3/d1d/d4a/c8f 0 2024-04-12T05:09:02.876 INFO:tasks.workunit.client.1.smithi192.stdout:8/398: mkdir d6/d10/d46/d7c 0 2024-04-12T05:09:02.882 INFO:tasks.workunit.client.0.smithi113.stdout:0/630: rename d1/d9 to d1/d6/d34/dc3 0 2024-04-12T05:09:02.884 INFO:tasks.workunit.client.1.smithi192.stdout:5/445: dwrite d0/d15/d32/d45/f65 [0,4194304] 0 2024-04-12T05:09:02.889 INFO:tasks.workunit.client.1.smithi192.stdout:1/456: mkdir d0/d2/dd/d3c/d5b/d74/d90 0 2024-04-12T05:09:02.890 INFO:tasks.workunit.client.1.smithi192.stdout:5/446: dread d0/d15/d24/d47/d7b/f88 [0,4194304] 0 2024-04-12T05:09:02.893 INFO:tasks.workunit.client.1.smithi192.stdout:9/370: creat d11/d15/d21/d2c/d1f/d6a/f7c x:0 0 0 2024-04-12T05:09:02.894 INFO:tasks.workunit.client.1.smithi192.stdout:9/371: write d11/d15/f42 [3483280,22482] 0 2024-04-12T05:09:02.894 INFO:tasks.workunit.client.1.smithi192.stdout:8/399: creat d6/d10/d46/d5e/d5f/d38/f7d x:0 0 0 2024-04-12T05:09:02.895 INFO:tasks.workunit.client.1.smithi192.stdout:9/372: write fb [2523116,52561] 0 2024-04-12T05:09:02.896 INFO:tasks.workunit.client.1.smithi192.stdout:7/414: truncate d3/d1d/d5a/d5f/f31 844998 0 2024-04-12T05:09:02.897 INFO:tasks.workunit.client.1.smithi192.stdout:8/400: readlink d6/d10/d46/d5e/d5f/l24 0 2024-04-12T05:09:02.898 INFO:tasks.workunit.client.1.smithi192.stdout:7/415: chown d3/d1d/f69 2175 1 2024-04-12T05:09:02.899 INFO:tasks.workunit.client.1.smithi192.stdout:7/416: readlink d3/d1d/d5a/l8b 0 2024-04-12T05:09:02.900 INFO:tasks.workunit.client.1.smithi192.stdout:8/401: truncate d6/d10/d46/d5e/d5f/f16 5076875 0 2024-04-12T05:09:02.901 INFO:tasks.workunit.client.1.smithi192.stdout:8/402: readlink d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/l76 0 2024-04-12T05:09:02.903 INFO:tasks.workunit.client.1.smithi192.stdout:0/594: link d0/d6/f96 d0/d4e/db0/dbf/fc7 0 2024-04-12T05:09:02.907 INFO:tasks.workunit.client.1.smithi192.stdout:5/447: unlink d0/d15/d24/f8e 0 2024-04-12T05:09:02.907 INFO:tasks.workunit.client.1.smithi192.stdout:5/448: dread - d0/d15/d32/d45/f86 zero size 2024-04-12T05:09:02.908 INFO:tasks.workunit.client.1.smithi192.stdout:3/410: sync 2024-04-12T05:09:02.910 INFO:tasks.workunit.client.0.smithi113.stdout:0/631: dwrite d1/d6/d16/d2a/d50/db6/fa8 [0,4194304] 0 2024-04-12T05:09:02.910 INFO:tasks.workunit.client.1.smithi192.stdout:2/483: getdents d1/d8/d3a 0 2024-04-12T05:09:02.910 INFO:tasks.workunit.client.1.smithi192.stdout:5/449: truncate d0/d15/d24/f4b 4684974 0 2024-04-12T05:09:02.912 INFO:tasks.workunit.client.1.smithi192.stdout:3/411: write d5/d13/d3a/d4e/d57/f51 [740804,63993] 0 2024-04-12T05:09:02.912 INFO:tasks.workunit.client.1.smithi192.stdout:7/417: symlink d3/d1d/d74/l90 0 2024-04-12T05:09:02.913 INFO:tasks.workunit.client.1.smithi192.stdout:8/403: unlink d6/d10/f45 0 2024-04-12T05:09:02.915 INFO:tasks.workunit.client.0.smithi113.stdout:0/632: dread d1/d6/d16/d2a/d50/db6/fa8 [0,4194304] 0 2024-04-12T05:09:02.917 INFO:tasks.workunit.client.1.smithi192.stdout:0/595: mknod d0/d6/dc4/cc8 0 2024-04-12T05:09:02.918 INFO:tasks.workunit.client.1.smithi192.stdout:4/564: truncate d1/d1d/d53/d1c/f36 4103794 0 2024-04-12T05:09:02.921 INFO:tasks.workunit.client.1.smithi192.stdout:9/373: symlink d11/d25/d36/d3d/d7a/l7d 0 2024-04-12T05:09:02.923 INFO:tasks.workunit.client.0.smithi113.stdout:0/633: symlink d1/d6/lc4 0 2024-04-12T05:09:02.923 INFO:tasks.workunit.client.1.smithi192.stdout:6/548: getdents d3/d8/d56/d59/d8a/d99 0 2024-04-12T05:09:02.926 INFO:tasks.workunit.client.1.smithi192.stdout:2/484: unlink d1/d8/d3a/de/f57 0 2024-04-12T05:09:02.929 INFO:tasks.workunit.client.0.smithi113.stdout:0/634: dwrite d1/d6/d16/f67 [0,4194304] 0 2024-04-12T05:09:02.930 INFO:tasks.workunit.client.1.smithi192.stdout:3/412: creat d5/d13/d3c/f91 x:0 0 0 2024-04-12T05:09:02.930 INFO:tasks.workunit.client.1.smithi192.stdout:1/457: write d0/d29/d8d/f8f [689949,96450] 0 2024-04-12T05:09:02.933 INFO:tasks.workunit.client.1.smithi192.stdout:0/596: write d0/d6/d67/f92 [687641,67938] 0 2024-04-12T05:09:02.939 INFO:tasks.workunit.client.1.smithi192.stdout:9/374: mkdir d11/d15/d21/d2c/d1f/d6a/d7e 0 2024-04-12T05:09:02.940 INFO:tasks.workunit.client.1.smithi192.stdout:6/549: symlink d3/d8/d56/lb1 0 2024-04-12T05:09:02.941 INFO:tasks.workunit.client.1.smithi192.stdout:1/458: dwrite d0/d2/f7b [0,4194304] 0 2024-04-12T05:09:02.942 INFO:tasks.workunit.client.1.smithi192.stdout:1/459: stat d0/f18 0 2024-04-12T05:09:02.942 INFO:tasks.workunit.client.1.smithi192.stdout:9/375: read d11/d15/d21/d2c/f1a [5121397,88031] 0 2024-04-12T05:09:02.942 INFO:tasks.workunit.client.1.smithi192.stdout:7/418: rename d3/d7/d15/d22/l72 to d3/l91 0 2024-04-12T05:09:02.943 INFO:tasks.workunit.client.1.smithi192.stdout:3/413: mknod d5/d13/d3a/d4a/d61/d79/c92 0 2024-04-12T05:09:02.943 INFO:tasks.workunit.client.1.smithi192.stdout:9/376: readlink d11/l22 0 2024-04-12T05:09:02.944 INFO:tasks.workunit.client.1.smithi192.stdout:4/565: symlink d1/d1d/d53/d1c/d84/dbb/lbc 0 2024-04-12T05:09:02.944 INFO:tasks.workunit.client.1.smithi192.stdout:7/419: chown d3/d7/d14/d1f/d76 243019 1 2024-04-12T05:09:02.944 INFO:tasks.workunit.client.1.smithi192.stdout:1/460: write d0/d2/dd/f32 [920121,99884] 0 2024-04-12T05:09:02.945 INFO:tasks.workunit.client.0.smithi113.stdout:0/635: truncate d1/d6/f30 1492424 0 2024-04-12T05:09:02.946 INFO:tasks.workunit.client.1.smithi192.stdout:1/461: stat d0/f18 0 2024-04-12T05:09:02.946 INFO:tasks.workunit.client.1.smithi192.stdout:3/414: chown d5/d13/d1a/d53/l62 1 1 2024-04-12T05:09:02.947 INFO:tasks.workunit.client.1.smithi192.stdout:3/415: dread - d5/d13/d1a/d53/f88 zero size 2024-04-12T05:09:02.949 INFO:tasks.workunit.client.1.smithi192.stdout:5/450: rmdir d0/d8b 0 2024-04-12T05:09:02.953 INFO:tasks.workunit.client.1.smithi192.stdout:2/485: mkdir d1/d8/d17/d19/d60/d9c 0 2024-04-12T05:09:02.955 INFO:tasks.workunit.client.1.smithi192.stdout:6/550: mkdir d3/d23/db2 0 2024-04-12T05:09:02.956 INFO:tasks.workunit.client.1.smithi192.stdout:3/416: dwrite d5/d13/d1a/f3b [4194304,4194304] 0 2024-04-12T05:09:02.959 INFO:tasks.workunit.client.1.smithi192.stdout:5/451: dwrite d0/f61 [4194304,4194304] 0 2024-04-12T05:09:02.961 INFO:tasks.workunit.client.1.smithi192.stdout:9/377: creat d11/d15/d21/d2c/d1f/f7f x:0 0 0 2024-04-12T05:09:02.962 INFO:tasks.workunit.client.1.smithi192.stdout:5/452: read d0/d15/d32/f4c [2284472,11686] 0 2024-04-12T05:09:02.967 INFO:tasks.workunit.client.1.smithi192.stdout:2/486: chown d1/c55 0 1 2024-04-12T05:09:02.970 INFO:tasks.workunit.client.0.smithi113.stdout:0/636: dread d1/d6/d16/d2a/d50/f77 [0,4194304] 0 2024-04-12T05:09:02.973 INFO:tasks.workunit.client.1.smithi192.stdout:0/597: getdents d0/d7b/d9b/d35 0 2024-04-12T05:09:02.973 INFO:tasks.workunit.client.1.smithi192.stdout:5/453: dwrite d0/f96 [0,4194304] 0 2024-04-12T05:09:02.974 INFO:tasks.workunit.client.1.smithi192.stdout:4/566: symlink d1/d1d/d53/db2/lbd 0 2024-04-12T05:09:02.975 INFO:tasks.workunit.client.1.smithi192.stdout:0/598: truncate d0/d6/d15/f6f 8979011 0 2024-04-12T05:09:02.975 INFO:tasks.workunit.client.1.smithi192.stdout:6/551: creat d3/d8/d56/d59/d8a/d99/fb3 x:0 0 0 2024-04-12T05:09:02.977 INFO:tasks.workunit.client.1.smithi192.stdout:1/462: link d0/d2/dd/d21/f38 d0/d2/dd/d21/d62/d84/f91 0 2024-04-12T05:09:02.981 INFO:tasks.workunit.client.1.smithi192.stdout:9/378: truncate d11/f12 3632726 0 2024-04-12T05:09:02.984 INFO:tasks.workunit.client.1.smithi192.stdout:9/379: chown d11/d15/d21/d2c/d1f/d6a/f7c 821026 1 2024-04-12T05:09:02.985 INFO:tasks.workunit.client.1.smithi192.stdout:7/420: dwrite f1 [0,4194304] 0 2024-04-12T05:09:02.985 INFO:tasks.workunit.client.0.smithi113.stdout:0/637: dread d1/d6/d16/d2a/d50/d52/d7e/db5/f7b [0,4194304] 0 2024-04-12T05:09:02.988 INFO:tasks.workunit.client.1.smithi192.stdout:4/567: rename d1/d1d/d53/d1c/d84/dbb/lbc to d1/d54/d82/d9c/lbe 0 2024-04-12T05:09:02.990 INFO:tasks.workunit.client.1.smithi192.stdout:6/552: dread d3/d8/f1c [4194304,4194304] 0 2024-04-12T05:09:02.991 INFO:tasks.workunit.client.1.smithi192.stdout:0/599: mknod d0/d7b/d9b/d35/cc9 0 2024-04-12T05:09:02.991 INFO:tasks.workunit.client.1.smithi192.stdout:3/417: dread d5/f19 [0,4194304] 0 2024-04-12T05:09:02.994 INFO:tasks.workunit.client.1.smithi192.stdout:3/418: chown d5/ca 11517683 1 2024-04-12T05:09:02.994 INFO:tasks.workunit.client.1.smithi192.stdout:0/600: readlink d0/d4e/d53/l5b 0 2024-04-12T05:09:02.994 INFO:tasks.workunit.client.1.smithi192.stdout:3/419: read d5/d13/d1a/f7e [1974403,112499] 0 2024-04-12T05:09:02.995 INFO:tasks.workunit.client.1.smithi192.stdout:5/454: mknod d0/d15/c9a 0 2024-04-12T05:09:02.995 INFO:tasks.workunit.client.1.smithi192.stdout:2/487: creat d1/d8/d17/d19/d60/d9c/f9d x:0 0 0 2024-04-12T05:09:02.996 INFO:tasks.workunit.client.1.smithi192.stdout:9/380: creat d11/d15/d21/d2c/d1f/d50/d73/f80 x:0 0 0 2024-04-12T05:09:02.997 INFO:tasks.workunit.client.1.smithi192.stdout:4/568: creat d1/d54/d82/da4/fbf x:0 0 0 2024-04-12T05:09:02.999 INFO:tasks.workunit.client.1.smithi192.stdout:4/569: fsync d1/d54/d55/d8b/f96 0 2024-04-12T05:09:03.000 INFO:tasks.workunit.client.1.smithi192.stdout:9/381: write d11/d15/d21/d2c/d1f/d50/d73/f80 [718982,19443] 0 2024-04-12T05:09:03.000 INFO:tasks.workunit.client.1.smithi192.stdout:3/420: dread - d5/d13/d3c/f31 zero size 2024-04-12T05:09:03.001 INFO:tasks.workunit.client.1.smithi192.stdout:2/488: fdatasync d1/d8/d17/d19/d60/f66 0 2024-04-12T05:09:03.003 INFO:tasks.workunit.client.1.smithi192.stdout:9/382: read d11/d25/d36/d3d/f79 [524958,65859] 0 2024-04-12T05:09:03.003 INFO:tasks.workunit.client.1.smithi192.stdout:0/601: mknod d0/d7b/d7c/dbb/cca 0 2024-04-12T05:09:03.003 INFO:tasks.workunit.client.1.smithi192.stdout:2/489: read - d1/d2d/d73/f9b zero size 2024-04-12T05:09:03.004 INFO:tasks.workunit.client.1.smithi192.stdout:1/463: getdents d0/d2/dd/d21/d62 0 2024-04-12T05:09:03.005 INFO:tasks.workunit.client.1.smithi192.stdout:3/421: chown d5/d13/d3a/c8e 121 1 2024-04-12T05:09:03.006 INFO:tasks.workunit.client.1.smithi192.stdout:4/570: write d1/d54/d82/d71/f7b [4873175,28841] 0 2024-04-12T05:09:03.007 INFO:tasks.workunit.client.1.smithi192.stdout:9/383: readlink d11/l51 0 2024-04-12T05:09:03.009 INFO:tasks.workunit.client.1.smithi192.stdout:6/553: write d3/d5/f45 [3670182,109373] 0 2024-04-12T05:09:03.011 INFO:tasks.workunit.client.1.smithi192.stdout:0/602: dread d0/d6/fa3 [0,4194304] 0 2024-04-12T05:09:03.011 INFO:tasks.workunit.client.1.smithi192.stdout:5/455: getdents d0/d15/d3e/d77 0 2024-04-12T05:09:03.013 INFO:tasks.workunit.client.1.smithi192.stdout:1/464: symlink d0/d2/dd/d21/d39/l92 0 2024-04-12T05:09:03.016 INFO:tasks.workunit.client.1.smithi192.stdout:3/422: dread d5/d13/d3a/d70/f71 [0,4194304] 0 2024-04-12T05:09:03.019 INFO:tasks.workunit.client.1.smithi192.stdout:6/554: chown d3/d8/d56/d59/lad 95 1 2024-04-12T05:09:03.021 INFO:tasks.workunit.client.0.smithi113.stdout:0/638: dread d1/f2 [0,4194304] 0 2024-04-12T05:09:03.022 INFO:tasks.workunit.client.1.smithi192.stdout:0/603: read d0/d6/d15/d19/f71 [2534210,26563] 0 2024-04-12T05:09:03.022 INFO:tasks.workunit.client.1.smithi192.stdout:9/384: rename fb to d11/d25/d36/f81 0 2024-04-12T05:09:03.023 INFO:tasks.workunit.client.1.smithi192.stdout:5/456: creat d0/d15/f9b x:0 0 0 2024-04-12T05:09:03.024 INFO:tasks.workunit.client.1.smithi192.stdout:1/465: dwrite d0/d2/dd/d3c/d5b/f5e [0,4194304] 0 2024-04-12T05:09:03.025 INFO:tasks.workunit.client.0.smithi113.stdout:0/639: chown d1/fa6 149898750 1 2024-04-12T05:09:03.025 INFO:tasks.workunit.client.1.smithi192.stdout:9/385: write d11/d25/d36/d6e/d33/f55 [383646,114027] 0 2024-04-12T05:09:03.027 INFO:tasks.workunit.client.1.smithi192.stdout:9/386: chown fe 47971466 1 2024-04-12T05:09:03.027 INFO:tasks.workunit.client.1.smithi192.stdout:2/490: dread d1/f39 [0,4194304] 0 2024-04-12T05:09:03.028 INFO:tasks.workunit.client.1.smithi192.stdout:0/604: unlink d0/d6/d15/d55/lac 0 2024-04-12T05:09:03.029 INFO:tasks.workunit.client.1.smithi192.stdout:2/491: readlink d1/d8/l4e 0 2024-04-12T05:09:03.029 INFO:tasks.workunit.client.1.smithi192.stdout:5/457: symlink d0/d15/d32/l9c 0 2024-04-12T05:09:03.030 INFO:tasks.workunit.client.1.smithi192.stdout:3/423: creat d5/d13/d3a/d70/d8c/d8d/f93 x:0 0 0 2024-04-12T05:09:03.031 INFO:tasks.workunit.client.0.smithi113.stdout:0/640: dwrite d1/d6/d16/d2a/fbd [0,4194304] 0 2024-04-12T05:09:03.031 INFO:tasks.workunit.client.1.smithi192.stdout:2/492: readlink d1/d8/d17/d19/l54 0 2024-04-12T05:09:03.032 INFO:tasks.workunit.client.1.smithi192.stdout:1/466: rmdir d0/d2/dd/d3c/d5b/d7c 39 2024-04-12T05:09:03.032 INFO:tasks.workunit.client.1.smithi192.stdout:9/387: fdatasync d11/d25/d36/d3d/f79 0 2024-04-12T05:09:03.035 INFO:tasks.workunit.client.1.smithi192.stdout:4/571: dwrite d1/d54/d82/da4/f79 [0,4194304] 0 2024-04-12T05:09:03.036 INFO:tasks.workunit.client.0.smithi113.stdout:0/641: unlink d1/d80/fbc 0 2024-04-12T05:09:03.036 INFO:tasks.workunit.client.1.smithi192.stdout:1/467: write d0/d2/dd/d21/f2d [2301899,67180] 0 2024-04-12T05:09:03.036 INFO:tasks.workunit.client.1.smithi192.stdout:0/605: rmdir d0/d7b/d9b/d4b 39 2024-04-12T05:09:03.042 INFO:tasks.workunit.client.0.smithi113.stdout:0/642: dread - d1/d43/d59/db2/fc1 zero size 2024-04-12T05:09:03.043 INFO:tasks.workunit.client.0.smithi113.stdout:0/643: chown d1/d6/d16/d2a/d50/d52/d7e 89409136 1 2024-04-12T05:09:03.045 INFO:tasks.workunit.client.1.smithi192.stdout:4/572: dread d1/d54/d82/d71/d9e/fa0 [0,4194304] 0 2024-04-12T05:09:03.045 INFO:tasks.workunit.client.1.smithi192.stdout:3/424: mkdir d5/d13/d94 0 2024-04-12T05:09:03.046 INFO:tasks.workunit.client.1.smithi192.stdout:5/458: dread d0/f61 [4194304,4194304] 0 2024-04-12T05:09:03.047 INFO:tasks.workunit.client.1.smithi192.stdout:9/388: fsync d11/d15/d21/d2c/f1a 0 2024-04-12T05:09:03.048 INFO:tasks.workunit.client.0.smithi113.stdout:0/644: rename d1/d6/d16/d2a/d50/db6/l41 to d1/d43/d59/db2/lc5 0 2024-04-12T05:09:03.054 INFO:tasks.workunit.client.1.smithi192.stdout:1/468: write d0/d2/f50 [2669779,86092] 0 2024-04-12T05:09:03.054 INFO:tasks.workunit.client.1.smithi192.stdout:2/493: mknod d1/d8/c9e 0 2024-04-12T05:09:03.056 INFO:tasks.workunit.client.1.smithi192.stdout:3/425: rmdir d5/d13/d3c 39 2024-04-12T05:09:03.056 INFO:tasks.workunit.client.1.smithi192.stdout:9/389: write d11/d25/d36/d6e/d33/f55 [1199466,105680] 0 2024-04-12T05:09:03.056 INFO:tasks.workunit.client.1.smithi192.stdout:4/573: truncate d1/d1d/d53/d1c/f6d 22298 0 2024-04-12T05:09:03.057 INFO:tasks.workunit.client.1.smithi192.stdout:2/494: chown d1/c88 21532653 1 2024-04-12T05:09:03.058 INFO:tasks.workunit.client.1.smithi192.stdout:4/574: readlink d1/lc 0 2024-04-12T05:09:03.058 INFO:tasks.workunit.client.1.smithi192.stdout:1/469: readlink d0/d2/dd/d21/d55/l75 0 2024-04-12T05:09:03.060 INFO:tasks.workunit.client.1.smithi192.stdout:9/390: write d11/d25/d36/d6e/d33/f55 [1858598,96030] 0 2024-04-12T05:09:03.061 INFO:tasks.workunit.client.1.smithi192.stdout:5/459: dwrite d0/d15/d24/f25 [4194304,4194304] 0 2024-04-12T05:09:03.062 INFO:tasks.workunit.client.1.smithi192.stdout:2/495: chown d1/d8/d17/d19/d71/d7f/f96 39773208 1 2024-04-12T05:09:03.063 INFO:tasks.workunit.client.1.smithi192.stdout:4/575: creat d1/d1d/d53/d1c/d95/fc0 x:0 0 0 2024-04-12T05:09:03.064 INFO:tasks.workunit.client.1.smithi192.stdout:5/460: stat d0/d15/d24/d47/l82 0 2024-04-12T05:09:03.065 INFO:tasks.workunit.client.1.smithi192.stdout:1/470: creat d0/d2/d1e/d3f/d56/f93 x:0 0 0 2024-04-12T05:09:03.066 INFO:tasks.workunit.client.1.smithi192.stdout:2/496: unlink d1/d8/d17/l49 0 2024-04-12T05:09:03.070 INFO:tasks.workunit.client.1.smithi192.stdout:3/426: dwrite d5/f14 [4194304,4194304] 0 2024-04-12T05:09:03.073 INFO:tasks.workunit.client.1.smithi192.stdout:5/461: creat d0/d15/d3e/f9d x:0 0 0 2024-04-12T05:09:03.073 INFO:tasks.workunit.client.1.smithi192.stdout:3/427: stat d5/l10 0 2024-04-12T05:09:03.075 INFO:tasks.workunit.client.1.smithi192.stdout:1/471: creat d0/d2/dd/d3c/d5b/d74/d90/f94 x:0 0 0 2024-04-12T05:09:03.075 INFO:tasks.workunit.client.1.smithi192.stdout:9/391: dwrite d11/d25/d36/f38 [0,4194304] 0 2024-04-12T05:09:03.078 INFO:tasks.workunit.client.1.smithi192.stdout:3/428: creat d5/d13/d3a/d4e/f95 x:0 0 0 2024-04-12T05:09:03.079 INFO:tasks.workunit.client.1.smithi192.stdout:4/576: dwrite d1/d54/f6e [0,4194304] 0 2024-04-12T05:09:03.085 INFO:tasks.workunit.client.1.smithi192.stdout:3/429: creat d5/d13/d3a/d70/d8c/f96 x:0 0 0 2024-04-12T05:09:03.089 INFO:tasks.workunit.client.1.smithi192.stdout:5/462: link d0/d15/d32/d45/f60 d0/d15/d32/d69/f9e 0 2024-04-12T05:09:03.089 INFO:tasks.workunit.client.1.smithi192.stdout:3/430: chown d5/f7a 71162615 1 2024-04-12T05:09:03.091 INFO:tasks.workunit.client.1.smithi192.stdout:1/472: dwrite d0/d2/dd/d21/d33/f6b [0,4194304] 0 2024-04-12T05:09:03.093 INFO:tasks.workunit.client.1.smithi192.stdout:4/577: dwrite d1/d54/d82/f62 [0,4194304] 0 2024-04-12T05:09:03.095 INFO:tasks.workunit.client.1.smithi192.stdout:3/431: readlink d5/d13/d3c/d26/l33 0 2024-04-12T05:09:03.095 INFO:tasks.workunit.client.1.smithi192.stdout:5/463: symlink d0/d21/l9f 0 2024-04-12T05:09:03.097 INFO:tasks.workunit.client.1.smithi192.stdout:3/432: fsync d5/f1c 0 2024-04-12T05:09:03.100 INFO:tasks.workunit.client.1.smithi192.stdout:5/464: chown d0/d15/d3e/d77/l91 6652 1 2024-04-12T05:09:03.100 INFO:tasks.workunit.client.1.smithi192.stdout:4/578: mkdir d1/d1d/d53/d1c/dc1 0 2024-04-12T05:09:03.101 INFO:tasks.workunit.client.1.smithi192.stdout:3/433: mknod d5/d13/d3c/c97 0 2024-04-12T05:09:03.102 INFO:tasks.workunit.client.1.smithi192.stdout:9/392: dread f10 [0,4194304] 0 2024-04-12T05:09:03.102 INFO:tasks.workunit.client.1.smithi192.stdout:1/473: dwrite d0/d1b/d28/f4d [0,4194304] 0 2024-04-12T05:09:03.102 INFO:tasks.workunit.client.1.smithi192.stdout:5/465: rmdir d0/d15/d64 39 2024-04-12T05:09:03.104 INFO:tasks.workunit.client.1.smithi192.stdout:3/434: write d5/d13/d1a/d53/f88 [62259,69924] 0 2024-04-12T05:09:03.105 INFO:tasks.workunit.client.1.smithi192.stdout:5/466: fdatasync d0/f1b 0 2024-04-12T05:09:03.106 INFO:tasks.workunit.client.1.smithi192.stdout:4/579: getdents d1/d54/d82/da1 0 2024-04-12T05:09:03.107 INFO:tasks.workunit.client.1.smithi192.stdout:4/580: fdatasync d1/d49/f5b 0 2024-04-12T05:09:03.107 INFO:tasks.workunit.client.1.smithi192.stdout:1/474: mknod d0/d2/dd/d21/d62/c95 0 2024-04-12T05:09:03.108 INFO:tasks.workunit.client.1.smithi192.stdout:9/393: rename d11/d15/d21/l31 to d11/d15/d1e/l82 0 2024-04-12T05:09:03.110 INFO:tasks.workunit.client.1.smithi192.stdout:3/435: creat d5/d13/d3a/d4a/f98 x:0 0 0 2024-04-12T05:09:03.112 INFO:tasks.workunit.client.1.smithi192.stdout:1/475: mkdir d0/d29/d8d/d96 0 2024-04-12T05:09:03.113 INFO:tasks.workunit.client.1.smithi192.stdout:1/476: readlink d0/d2/dd/d21/d39/l92 0 2024-04-12T05:09:03.114 INFO:tasks.workunit.client.1.smithi192.stdout:9/394: rmdir d11/d25/d36/d3d 39 2024-04-12T05:09:03.115 INFO:tasks.workunit.client.1.smithi192.stdout:1/477: write d0/d2/dd/d21/d39/f65 [2899808,121622] 0 2024-04-12T05:09:03.116 INFO:tasks.workunit.client.1.smithi192.stdout:3/436: symlink d5/d13/d94/l99 0 2024-04-12T05:09:03.116 INFO:tasks.workunit.client.1.smithi192.stdout:9/395: read - d11/d25/d36/d6e/d71/f78 zero size 2024-04-12T05:09:03.118 INFO:tasks.workunit.client.1.smithi192.stdout:5/467: getdents d0/d15/d24 0 2024-04-12T05:09:03.118 INFO:tasks.workunit.client.1.smithi192.stdout:1/478: symlink d0/d2/l97 0 2024-04-12T05:09:03.119 INFO:tasks.workunit.client.1.smithi192.stdout:3/437: creat d5/d13/d3a/d4e/f9a x:0 0 0 2024-04-12T05:09:03.119 INFO:tasks.workunit.client.1.smithi192.stdout:9/396: chown d11/d15/d21/d2c/d1f/f67 13768 1 2024-04-12T05:09:03.120 INFO:tasks.workunit.client.1.smithi192.stdout:3/438: chown d5/d13/d3a/d70 136 1 2024-04-12T05:09:03.122 INFO:tasks.workunit.client.1.smithi192.stdout:3/439: truncate d5/d13/d3a/d4a/d61/f64 973953 0 2024-04-12T05:09:03.123 INFO:tasks.workunit.client.1.smithi192.stdout:1/479: unlink d0/d2/dd/d3c/d5b/f5e 0 2024-04-12T05:09:03.124 INFO:tasks.workunit.client.1.smithi192.stdout:9/397: mknod d11/d15/d1e/c83 0 2024-04-12T05:09:03.124 INFO:tasks.workunit.client.1.smithi192.stdout:3/440: read - d5/d13/d3a/d70/d8c/d8d/f93 zero size 2024-04-12T05:09:03.124 INFO:tasks.workunit.client.1.smithi192.stdout:5/468: mkdir d0/d15/d64/d7c/da0 0 2024-04-12T05:09:03.125 INFO:tasks.workunit.client.1.smithi192.stdout:5/469: chown d0/l18 1309581 1 2024-04-12T05:09:03.125 INFO:tasks.workunit.client.1.smithi192.stdout:2/497: dread d1/d8/d17/d19/d40/f52 [0,4194304] 0 2024-04-12T05:09:03.125 INFO:tasks.workunit.client.1.smithi192.stdout:1/480: unlink d0/d2/dd/d71/l77 0 2024-04-12T05:09:03.126 INFO:tasks.workunit.client.1.smithi192.stdout:2/498: chown d1/d8/d17/d19 19 1 2024-04-12T05:09:03.127 INFO:tasks.workunit.client.1.smithi192.stdout:3/441: mknod d5/d13/d3a/d85/c9b 0 2024-04-12T05:09:03.131 INFO:tasks.workunit.client.1.smithi192.stdout:1/481: mknod d0/d2/d1e/d3f/d78/c98 0 2024-04-12T05:09:03.140 INFO:tasks.workunit.client.0.smithi113.stdout:0/645: write d1/f22 [5835090,82992] 0 2024-04-12T05:09:03.141 INFO:tasks.workunit.client.1.smithi192.stdout:0/606: dread d0/f16 [0,4194304] 0 2024-04-12T05:09:03.143 INFO:tasks.workunit.client.1.smithi192.stdout:1/482: creat d0/d2/dd/d21/f99 x:0 0 0 2024-04-12T05:09:03.143 INFO:tasks.workunit.client.1.smithi192.stdout:3/442: dwrite d5/d13/d3a/d70/f71 [0,4194304] 0 2024-04-12T05:09:03.144 INFO:tasks.workunit.client.0.smithi113.stdout:0/646: truncate d1/d6/d16/d2a/d50/d52/d7e/db5/f37 882512 0 2024-04-12T05:09:03.145 INFO:tasks.workunit.client.1.smithi192.stdout:3/443: rename d5/d13/d3a to d5/d13/d3a/d70/d9c 22 2024-04-12T05:09:03.145 INFO:tasks.workunit.client.0.smithi113.stdout:0/647: chown d1/d6/d16/d2a/d50/db6/l15 56367271 1 2024-04-12T05:09:03.148 INFO:tasks.workunit.client.0.smithi113.stdout:0/648: creat d1/d6/d34/fc6 x:0 0 0 2024-04-12T05:09:03.148 INFO:tasks.workunit.client.1.smithi192.stdout:3/444: dread d5/d13/d1a/d53/f88 [0,4194304] 0 2024-04-12T05:09:03.152 INFO:tasks.workunit.client.1.smithi192.stdout:3/445: creat d5/d13/d3c/d84/f9d x:0 0 0 2024-04-12T05:09:03.157 INFO:tasks.workunit.client.1.smithi192.stdout:3/446: rename d5/d13/d1a/f7e to d5/d13/d3c/d26/f9e 0 2024-04-12T05:09:03.160 INFO:tasks.workunit.client.1.smithi192.stdout:3/447: mkdir d5/d13/d3a/d4a/d61/d9f 0 2024-04-12T05:09:03.162 INFO:tasks.workunit.client.1.smithi192.stdout:5/470: dread d0/f52 [0,4194304] 0 2024-04-12T05:09:03.165 INFO:tasks.workunit.client.1.smithi192.stdout:5/471: read d0/d15/d24/f25 [2171576,128174] 0 2024-04-12T05:09:03.167 INFO:tasks.workunit.client.1.smithi192.stdout:5/472: fdatasync d0/f13 0 2024-04-12T05:09:03.167 INFO:tasks.workunit.client.1.smithi192.stdout:3/448: fsync d5/d13/d3a/d4a/f98 0 2024-04-12T05:09:03.170 INFO:tasks.workunit.client.1.smithi192.stdout:3/449: mkdir d5/d13/d3c/d36/da0 0 2024-04-12T05:09:03.173 INFO:tasks.workunit.client.1.smithi192.stdout:3/450: creat d5/d13/d1a/d5f/fa1 x:0 0 0 2024-04-12T05:09:03.174 INFO:tasks.workunit.client.1.smithi192.stdout:2/499: write d1/d8/f35 [911685,30972] 0 2024-04-12T05:09:03.174 INFO:tasks.workunit.client.1.smithi192.stdout:3/451: chown d5/f7a 16733144 1 2024-04-12T05:09:03.174 INFO:tasks.workunit.client.1.smithi192.stdout:2/500: chown d1/d8/d3a/f95 8 1 2024-04-12T05:09:03.175 INFO:tasks.workunit.client.0.smithi113.stdout:0/649: truncate d1/d6/d16/d2a/d50/d52/d7e/db5/f37 294361 0 2024-04-12T05:09:03.177 INFO:tasks.workunit.client.1.smithi192.stdout:2/501: write d1/d8/d17/d19/d40/f52 [2524835,111599] 0 2024-04-12T05:09:03.178 INFO:tasks.workunit.client.1.smithi192.stdout:3/452: mkdir d5/d13/d3a/d8b/da2 0 2024-04-12T05:09:03.179 INFO:tasks.workunit.client.1.smithi192.stdout:2/502: dread - d1/d8/d3a/de/d7c/f7d zero size 2024-04-12T05:09:03.181 INFO:tasks.workunit.client.1.smithi192.stdout:1/483: dwrite d0/d1b/f53 [0,4194304] 0 2024-04-12T05:09:03.183 INFO:tasks.workunit.client.1.smithi192.stdout:1/484: write d0/d1b/f60 [350147,100796] 0 2024-04-12T05:09:03.184 INFO:tasks.workunit.client.0.smithi113.stdout:0/650: dwrite d1/fa7 [0,4194304] 0 2024-04-12T05:09:03.187 INFO:tasks.workunit.client.1.smithi192.stdout:1/485: fsync d0/d2/dd/d21/d33/f6b 0 2024-04-12T05:09:03.187 INFO:tasks.workunit.client.1.smithi192.stdout:3/453: dread d5/d13/d3a/d70/f71 [0,4194304] 0 2024-04-12T05:09:03.188 INFO:tasks.workunit.client.0.smithi113.stdout:0/651: dread d1/d6/d16/d2a/d50/f61 [0,4194304] 0 2024-04-12T05:09:03.188 INFO:tasks.workunit.client.1.smithi192.stdout:2/503: creat d1/d8/d17/d19/d60/d9c/f9f x:0 0 0 2024-04-12T05:09:03.194 INFO:tasks.workunit.client.1.smithi192.stdout:3/454: write d5/d13/d3c/d26/f9e [3214806,4558] 0 2024-04-12T05:09:03.195 INFO:tasks.workunit.client.0.smithi113.stdout:0/652: rmdir d1/d6/d16 39 2024-04-12T05:09:03.200 INFO:tasks.workunit.client.1.smithi192.stdout:5/473: dwrite d0/d15/d32/d45/f60 [0,4194304] 0 2024-04-12T05:09:03.201 INFO:tasks.workunit.client.0.smithi113.stdout:0/653: getdents d1/d6 0 2024-04-12T05:09:03.201 INFO:tasks.workunit.client.1.smithi192.stdout:1/486: dwrite d0/d2/dd/d21/d39/f49 [0,4194304] 0 2024-04-12T05:09:03.203 INFO:tasks.workunit.client.0.smithi113.stdout:0/654: creat d1/d6/d16/d2a/d50/d52/d7e/db5/fc7 x:0 0 0 2024-04-12T05:09:03.207 INFO:tasks.workunit.client.0.smithi113.stdout:0/655: link d1/d6/d16/d2a/d50/d52/d7e/db5/l6e d1/d80/lc8 0 2024-04-12T05:09:03.207 INFO:tasks.workunit.client.1.smithi192.stdout:3/455: rename d5/ca to d5/d13/d3a/d70/ca3 0 2024-04-12T05:09:03.208 INFO:tasks.workunit.client.1.smithi192.stdout:1/487: symlink d0/d2/dd/d21/d55/l9a 0 2024-04-12T05:09:03.209 INFO:tasks.workunit.client.0.smithi113.stdout:0/656: truncate d1/d6/d16/d2a/d50/d52/fa0 39089 0 2024-04-12T05:09:03.212 INFO:tasks.workunit.client.1.smithi192.stdout:3/456: rmdir d5/d13/d3a/d4a 39 2024-04-12T05:09:03.215 INFO:tasks.workunit.client.0.smithi113.stdout:0/657: dread d1/d6/d16/d2a/d50/d52/d7e/db5/f37 [0,4194304] 0 2024-04-12T05:09:03.215 INFO:tasks.workunit.client.1.smithi192.stdout:3/457: read d5/d13/f2e [152814,31310] 0 2024-04-12T05:09:03.216 INFO:tasks.workunit.client.0.smithi113.stdout:0/658: dread - d1/d6/d16/d2a/d50/d52/d7e/db5/fc7 zero size 2024-04-12T05:09:03.218 INFO:tasks.workunit.client.1.smithi192.stdout:3/458: creat d5/d13/d3a/d85/fa4 x:0 0 0 2024-04-12T05:09:03.222 INFO:tasks.workunit.client.0.smithi113.stdout:0/659: link d1/d6/d16/d2a/d50/d52/d7e/db5/f7b d1/fc9 0 2024-04-12T05:09:03.224 INFO:tasks.workunit.client.1.smithi192.stdout:3/459: rename d5/d13/d3a/d4a/c81 to d5/d13/d1a/ca5 0 2024-04-12T05:09:03.230 INFO:tasks.workunit.client.1.smithi192.stdout:3/460: dwrite d5/f7a [0,4194304] 0 2024-04-12T05:09:03.233 INFO:tasks.workunit.client.1.smithi192.stdout:3/461: mknod d5/d13/d94/ca6 0 2024-04-12T05:09:03.234 INFO:tasks.workunit.client.1.smithi192.stdout:1/488: dread d0/d2/dd/f15 [0,4194304] 0 2024-04-12T05:09:03.236 INFO:tasks.workunit.client.1.smithi192.stdout:3/462: rmdir d5/d13/d3a/d85 39 2024-04-12T05:09:03.236 INFO:tasks.workunit.client.0.smithi113.stdout:0/660: write d1/d6/d34/dc3/f5a [939750,29226] 0 2024-04-12T05:09:03.237 INFO:tasks.workunit.client.1.smithi192.stdout:3/463: readlink d5/d13/d3c/l48 0 2024-04-12T05:09:03.240 INFO:tasks.workunit.client.0.smithi113.stdout:0/661: mknod d1/d80/cca 0 2024-04-12T05:09:03.240 INFO:tasks.workunit.client.1.smithi192.stdout:1/489: dwrite d0/d2/d1e/f36 [0,4194304] 0 2024-04-12T05:09:03.241 INFO:tasks.workunit.client.1.smithi192.stdout:3/464: mkdir d5/d13/d3a/d8b/da2/da7 0 2024-04-12T05:09:03.242 INFO:tasks.workunit.client.1.smithi192.stdout:1/490: chown d0/d1b/c2c 65833324 1 2024-04-12T05:09:03.244 INFO:tasks.workunit.client.0.smithi113.stdout:0/662: fsync d1/d6/d16/f24 0 2024-04-12T05:09:03.245 INFO:tasks.workunit.client.1.smithi192.stdout:1/491: read d0/d2/d1e/d3f/d56/d58/f8b [3249243,114434] 0 2024-04-12T05:09:03.246 INFO:tasks.workunit.client.1.smithi192.stdout:1/492: stat d0/d29/d8d/f8f 0 2024-04-12T05:09:03.250 INFO:tasks.workunit.client.1.smithi192.stdout:1/493: getdents d0/d2/dd/d71 0 2024-04-12T05:09:03.252 INFO:tasks.workunit.client.1.smithi192.stdout:1/494: mkdir d0/d2/dd/d3c/d5b/d74/d7d/d9b 0 2024-04-12T05:09:03.258 INFO:tasks.workunit.client.1.smithi192.stdout:1/495: dwrite d0/d1b/d28/f4d [0,4194304] 0 2024-04-12T05:09:03.262 INFO:tasks.workunit.client.1.smithi192.stdout:1/496: link d0/f18 d0/d2/dd/d3c/f9c 0 2024-04-12T05:09:03.266 INFO:tasks.workunit.client.1.smithi192.stdout:1/497: rename d0/d2/d1e/d3f/d56/d58 to d0/d1b/d28/d9d 0 2024-04-12T05:09:03.271 INFO:tasks.workunit.client.1.smithi192.stdout:1/498: rename d0/d2/dd/d21/d55 to d0/d2/d1e/d3f/d56/d9e 0 2024-04-12T05:09:03.272 INFO:tasks.workunit.client.1.smithi192.stdout:1/499: fsync d0/d1b/d44/f6e 0 2024-04-12T05:09:03.273 INFO:tasks.workunit.client.0.smithi113.stdout:0/663: dread d1/f22 [0,4194304] 0 2024-04-12T05:09:03.277 INFO:tasks.workunit.client.1.smithi192.stdout:1/500: dread d0/d2/dd/d21/d62/d84/f91 [4194304,4194304] 0 2024-04-12T05:09:03.281 INFO:tasks.workunit.client.0.smithi113.stdout:0/664: dwrite d1/d43/f9a [0,4194304] 0 2024-04-12T05:09:03.282 INFO:tasks.workunit.client.0.smithi113.stdout:0/665: chown d1/d43/l5e 4502385 1 2024-04-12T05:09:03.285 INFO:tasks.workunit.client.0.smithi113.stdout:0/666: creat d1/d6/d8e/fcb x:0 0 0 2024-04-12T05:09:03.287 INFO:tasks.workunit.client.0.smithi113.stdout:0/667: mknod d1/d6/d34/dc3/ccc 0 2024-04-12T05:09:03.290 INFO:tasks.workunit.client.0.smithi113.stdout:0/668: rename d1/d6/d16/d2a/d50/d52/d7e/db5/l90 to d1/d6/d16/d2a/d50/d52/d7e/db5/lcd 0 2024-04-12T05:09:03.292 INFO:tasks.workunit.client.0.smithi113.stdout:0/669: creat d1/d6/d16/d8f/fce x:0 0 0 2024-04-12T05:09:03.299 INFO:tasks.workunit.client.0.smithi113.stdout:0/670: dwrite d1/d6/d16/d2a/d50/d52/d7e/f8d [0,4194304] 0 2024-04-12T05:09:03.300 INFO:tasks.workunit.client.0.smithi113.stdout:0/671: write d1/d6/d34/f40 [7276226,41560] 0 2024-04-12T05:09:03.301 INFO:tasks.workunit.client.0.smithi113.stdout:0/672: mkdir d1/d80/dcf 0 2024-04-12T05:09:03.303 INFO:tasks.workunit.client.1.smithi192.stdout:1/501: dread d0/d2/f34 [0,4194304] 0 2024-04-12T05:09:03.303 INFO:tasks.workunit.client.0.smithi113.stdout:0/673: mknod d1/d6/d16/d2a/d50/cd0 0 2024-04-12T05:09:03.304 INFO:tasks.workunit.client.1.smithi192.stdout:1/502: chown d0/d29/d8d/f8f 3336 1 2024-04-12T05:09:03.307 INFO:tasks.workunit.client.0.smithi113.stdout:0/674: rmdir d1/d6/d16/d2a/d50 39 2024-04-12T05:09:03.307 INFO:tasks.workunit.client.1.smithi192.stdout:1/503: unlink d0/d2/dd/lf 0 2024-04-12T05:09:03.309 INFO:tasks.workunit.client.1.smithi192.stdout:1/504: mkdir d0/d2/d9f 0 2024-04-12T05:09:03.310 INFO:tasks.workunit.client.1.smithi192.stdout:1/505: stat d0/d2/d1e/d3f/d56/f93 0 2024-04-12T05:09:03.310 INFO:tasks.workunit.client.1.smithi192.stdout:1/506: read - d0/d1b/f6a zero size 2024-04-12T05:09:03.312 INFO:tasks.workunit.client.1.smithi192.stdout:1/507: creat d0/d29/d8d/d96/fa0 x:0 0 0 2024-04-12T05:09:03.317 INFO:tasks.workunit.client.1.smithi192.stdout:1/508: read d0/d2/dd/f32 [1603184,120417] 0 2024-04-12T05:09:03.318 INFO:tasks.workunit.client.1.smithi192.stdout:1/509: chown d0/d2/l57 574 1 2024-04-12T05:09:03.320 INFO:tasks.workunit.client.1.smithi192.stdout:1/510: symlink d0/d2/dd/d21/d62/la1 0 2024-04-12T05:09:03.323 INFO:tasks.workunit.client.1.smithi192.stdout:1/511: rename d0/d2/dd/f15 to d0/d1b/d44/d69/fa2 0 2024-04-12T05:09:03.326 INFO:tasks.workunit.client.1.smithi192.stdout:1/512: unlink d0/d1b/f2b 0 2024-04-12T05:09:03.335 INFO:tasks.workunit.client.1.smithi192.stdout:8/404: sync 2024-04-12T05:09:03.335 INFO:tasks.workunit.client.1.smithi192.stdout:7/421: sync 2024-04-12T05:09:03.336 INFO:tasks.workunit.client.1.smithi192.stdout:7/422: truncate d3/d11/f2d 4631993 0 2024-04-12T05:09:03.336 INFO:tasks.workunit.client.1.smithi192.stdout:8/405: dread - d6/d10/d46/d5e/d5f/d37/f68 zero size 2024-04-12T05:09:03.337 INFO:tasks.workunit.client.1.smithi192.stdout:8/406: readlink d6/l17 0 2024-04-12T05:09:03.337 INFO:tasks.workunit.client.1.smithi192.stdout:7/423: truncate d3/d1d/d4a/f61 265979 0 2024-04-12T05:09:03.341 INFO:tasks.workunit.client.1.smithi192.stdout:7/424: readlink d3/l91 0 2024-04-12T05:09:03.341 INFO:tasks.workunit.client.1.smithi192.stdout:7/425: stat d3/d1d/f41 0 2024-04-12T05:09:03.350 INFO:tasks.workunit.client.1.smithi192.stdout:7/426: creat d3/d7/d14/f92 x:0 0 0 2024-04-12T05:09:03.352 INFO:tasks.workunit.client.1.smithi192.stdout:1/513: dread d0/d2/dd/d21/d39/f65 [0,4194304] 0 2024-04-12T05:09:03.354 INFO:tasks.workunit.client.1.smithi192.stdout:1/514: dread - d0/d2/dd/d21/d62/d84/f8a zero size 2024-04-12T05:09:03.356 INFO:tasks.workunit.client.1.smithi192.stdout:1/515: write d0/d1b/f6a [915530,107934] 0 2024-04-12T05:09:03.357 INFO:tasks.workunit.client.1.smithi192.stdout:7/427: dwrite d3/d7/f19 [0,4194304] 0 2024-04-12T05:09:03.360 INFO:tasks.workunit.client.1.smithi192.stdout:7/428: read d3/f12 [5961901,20588] 0 2024-04-12T05:09:03.362 INFO:tasks.workunit.client.1.smithi192.stdout:1/516: getdents d0/d2/dd/d21/d62 0 2024-04-12T05:09:03.365 INFO:tasks.workunit.client.1.smithi192.stdout:1/517: creat d0/d2/dd/d21/d62/d84/fa3 x:0 0 0 2024-04-12T05:09:03.367 INFO:tasks.workunit.client.1.smithi192.stdout:7/429: dwrite d3/d7/d14/f3d [0,4194304] 0 2024-04-12T05:09:03.369 INFO:tasks.workunit.client.1.smithi192.stdout:1/518: creat d0/d1b/d28/d9d/fa4 x:0 0 0 2024-04-12T05:09:03.370 INFO:tasks.workunit.client.1.smithi192.stdout:7/430: mkdir d3/d1d/d74/d86/d93 0 2024-04-12T05:09:03.371 INFO:tasks.workunit.client.1.smithi192.stdout:7/431: dread - d3/f6f zero size 2024-04-12T05:09:03.371 INFO:tasks.workunit.client.1.smithi192.stdout:1/519: fdatasync d0/d2/dd/d21/d33/f68 0 2024-04-12T05:09:03.376 INFO:tasks.workunit.client.1.smithi192.stdout:1/520: unlink d0/d29/d8d/d96/fa0 0 2024-04-12T05:09:03.377 INFO:tasks.workunit.client.1.smithi192.stdout:1/521: chown d0/c70 18 1 2024-04-12T05:09:03.380 INFO:tasks.workunit.client.1.smithi192.stdout:7/432: dwrite d3/d7/d15/f7d [0,4194304] 0 2024-04-12T05:09:03.382 INFO:tasks.workunit.client.1.smithi192.stdout:7/433: fdatasync d3/d7/d14/f47 0 2024-04-12T05:09:03.385 INFO:tasks.workunit.client.1.smithi192.stdout:7/434: rmdir d3/d1d/d74 39 2024-04-12T05:09:03.385 INFO:tasks.workunit.client.1.smithi192.stdout:1/522: dwrite d0/f24 [0,4194304] 0 2024-04-12T05:09:03.387 INFO:tasks.workunit.client.1.smithi192.stdout:7/435: chown d3/d7/d15/f24 1185932 1 2024-04-12T05:09:03.388 INFO:tasks.workunit.client.1.smithi192.stdout:1/523: fdatasync d0/d1b/f35 0 2024-04-12T05:09:03.390 INFO:tasks.workunit.client.1.smithi192.stdout:7/436: unlink d3/d1d/d4a/c71 0 2024-04-12T05:09:03.394 INFO:tasks.workunit.client.1.smithi192.stdout:7/437: symlink d3/d7/d14/d1f/d76/l94 0 2024-04-12T05:09:03.395 INFO:tasks.workunit.client.0.smithi113.stdout:1/631: sync 2024-04-12T05:09:03.396 INFO:tasks.workunit.client.1.smithi192.stdout:1/524: link d0/d1b/d28/d9d/f8b d0/d2/dd/d3c/d5b/fa5 0 2024-04-12T05:09:03.399 INFO:tasks.workunit.client.1.smithi192.stdout:7/438: link d3/d1d/d5a/d5f/f30 d3/d1d/d48/d7b/f95 0 2024-04-12T05:09:03.406 INFO:tasks.workunit.client.1.smithi192.stdout:7/439: creat d3/d1d/d4a/d5e/d6d/f96 x:0 0 0 2024-04-12T05:09:03.407 INFO:tasks.workunit.client.0.smithi113.stdout:1/632: link d10/d2e/d31/f53 d10/d14/d67/d70/d98/dad/fc4 0 2024-04-12T05:09:03.407 INFO:tasks.workunit.client.0.smithi113.stdout:1/633: readlink d10/d2e/d7c/l87 0 2024-04-12T05:09:03.409 INFO:tasks.workunit.client.0.smithi113.stdout:1/634: mkdir d10/d2e/d7c/dc5 0 2024-04-12T05:09:03.411 INFO:tasks.workunit.client.0.smithi113.stdout:1/635: mknod d10/d21/cc6 0 2024-04-12T05:09:03.412 INFO:tasks.workunit.client.1.smithi192.stdout:7/440: dwrite d3/d7/f56 [0,4194304] 0 2024-04-12T05:09:03.414 INFO:tasks.workunit.client.1.smithi192.stdout:1/525: rmdir d0/d2/dd/d3c 39 2024-04-12T05:09:03.421 INFO:tasks.workunit.client.1.smithi192.stdout:1/526: creat d0/d2/dd/d21/d33/fa6 x:0 0 0 2024-04-12T05:09:03.427 INFO:tasks.workunit.client.0.smithi113.stdout:1/636: dwrite d10/d14/d67/d70/fb1 [0,4194304] 0 2024-04-12T05:09:03.429 INFO:tasks.workunit.client.0.smithi113.stdout:1/637: truncate d10/d2e/d31/d33/f61 869513 0 2024-04-12T05:09:03.432 INFO:tasks.workunit.client.0.smithi113.stdout:1/638: truncate d10/d2e/d31/d33/f2d 971807 0 2024-04-12T05:09:03.432 INFO:tasks.workunit.client.1.smithi192.stdout:1/527: dwrite d0/d2/dd/d21/d62/d84/f8a [0,4194304] 0 2024-04-12T05:09:03.434 INFO:tasks.workunit.client.0.smithi113.stdout:1/639: creat d10/d2e/d7c/dc5/fc7 x:0 0 0 2024-04-12T05:09:03.434 INFO:tasks.workunit.client.1.smithi192.stdout:1/528: chown d0/d2/dd/d21/d62/d84/f8a 122384 1 2024-04-12T05:09:03.436 INFO:tasks.workunit.client.1.smithi192.stdout:7/441: dread d3/d1d/d48/d7b/f95 [0,4194304] 0 2024-04-12T05:09:03.438 INFO:tasks.workunit.client.0.smithi113.stdout:1/640: rename d10/d14/d67/d70/d98 to d10/d2e/d7c/dc5/dc8 0 2024-04-12T05:09:03.440 INFO:tasks.workunit.client.1.smithi192.stdout:7/442: mknod d3/d7/d60/c97 0 2024-04-12T05:09:03.441 INFO:tasks.workunit.client.1.smithi192.stdout:7/443: fsync d3/d1d/f2e 0 2024-04-12T05:09:03.441 INFO:tasks.workunit.client.0.smithi113.stdout:1/641: getdents d10/d2e/d7c/dc5/dc8/dad 0 2024-04-12T05:09:03.442 INFO:tasks.workunit.client.1.smithi192.stdout:1/529: dread d0/d2/d1e/d3f/d56/f79 [0,4194304] 0 2024-04-12T05:09:03.446 INFO:tasks.workunit.client.0.smithi113.stdout:1/642: getdents d10/d2e/d7c 0 2024-04-12T05:09:03.447 INFO:tasks.workunit.client.1.smithi192.stdout:7/444: getdents d3/d7/d15 0 2024-04-12T05:09:03.448 INFO:tasks.workunit.client.1.smithi192.stdout:1/530: dwrite d0/d2/dd/d21/d62/d84/fa3 [0,4194304] 0 2024-04-12T05:09:03.448 INFO:tasks.workunit.client.0.smithi113.stdout:1/643: fsync d10/f13 0 2024-04-12T05:09:03.449 INFO:tasks.workunit.client.0.smithi113.stdout:1/644: write d10/d2e/d7c/dc5/fc7 [861902,14515] 0 2024-04-12T05:09:03.452 INFO:tasks.workunit.client.1.smithi192.stdout:7/445: mknod d3/d7/d60/c98 0 2024-04-12T05:09:03.453 INFO:tasks.workunit.client.1.smithi192.stdout:7/446: write d3/d7/d60/f66 [384266,21482] 0 2024-04-12T05:09:03.454 INFO:tasks.workunit.client.0.smithi113.stdout:1/645: rename d10/d41/d89/d96/d7d/lbe to d10/d41/d89/d96/d7d/d9f/lc9 0 2024-04-12T05:09:03.454 INFO:tasks.workunit.client.1.smithi192.stdout:4/581: sync 2024-04-12T05:09:03.454 INFO:tasks.workunit.client.1.smithi192.stdout:9/398: sync 2024-04-12T05:09:03.454 INFO:tasks.workunit.client.1.smithi192.stdout:6/555: sync 2024-04-12T05:09:03.454 INFO:tasks.workunit.client.1.smithi192.stdout:0/607: sync 2024-04-12T05:09:03.457 INFO:tasks.workunit.client.1.smithi192.stdout:7/447: chown d3/d1d/d5a/d5f/l3a 15081 1 2024-04-12T05:09:03.457 INFO:tasks.workunit.client.0.smithi113.stdout:1/646: symlink d10/d41/d89/lca 0 2024-04-12T05:09:03.458 INFO:tasks.workunit.client.1.smithi192.stdout:9/399: dread - d11/d25/d36/d6e/d71/f78 zero size 2024-04-12T05:09:03.462 INFO:tasks.workunit.client.1.smithi192.stdout:6/556: creat d3/d21/d24/d79/fb4 x:0 0 0 2024-04-12T05:09:03.464 INFO:tasks.workunit.client.0.smithi113.stdout:1/647: creat d10/da5/fcb x:0 0 0 2024-04-12T05:09:03.466 INFO:tasks.workunit.client.1.smithi192.stdout:7/448: chown d3/d1d/d4a/c8d 26 1 2024-04-12T05:09:03.469 INFO:tasks.workunit.client.0.smithi113.stdout:1/648: fsync d10/d2e/d31/f35 0 2024-04-12T05:09:03.470 INFO:tasks.workunit.client.1.smithi192.stdout:9/400: creat d11/d15/d1e/f84 x:0 0 0 2024-04-12T05:09:03.470 INFO:tasks.workunit.client.1.smithi192.stdout:9/401: stat f10 0 2024-04-12T05:09:03.470 INFO:tasks.workunit.client.1.smithi192.stdout:1/531: truncate d0/d1b/d44/d69/fa2 404680 0 2024-04-12T05:09:03.470 INFO:tasks.workunit.client.1.smithi192.stdout:9/402: fsync d11/d25/f32 0 2024-04-12T05:09:03.472 INFO:tasks.workunit.client.1.smithi192.stdout:6/557: mkdir d3/d23/db5 0 2024-04-12T05:09:03.473 INFO:tasks.workunit.client.1.smithi192.stdout:0/608: creat d0/d4e/d95/db4/db8/fcb x:0 0 0 2024-04-12T05:09:03.475 INFO:tasks.workunit.client.0.smithi113.stdout:1/649: symlink d10/d41/d5b/lcc 0 2024-04-12T05:09:03.476 INFO:tasks.workunit.client.1.smithi192.stdout:7/449: mknod d3/d1d/d48/c99 0 2024-04-12T05:09:03.477 INFO:tasks.workunit.client.1.smithi192.stdout:4/582: dwrite d1/fb6 [0,4194304] 0 2024-04-12T05:09:03.488 INFO:tasks.workunit.client.0.smithi113.stdout:1/650: write d10/d14/f2a [131313,107199] 0 2024-04-12T05:09:03.491 INFO:tasks.workunit.client.0.smithi113.stdout:1/651: symlink d10/d2e/d7c/dc5/dc8/dad/lcd 0 2024-04-12T05:09:03.493 INFO:tasks.workunit.client.0.smithi113.stdout:1/652: rmdir d10 39 2024-04-12T05:09:03.495 INFO:tasks.workunit.client.0.smithi113.stdout:1/653: write d10/d14/f4d [707462,64351] 0 2024-04-12T05:09:03.496 INFO:tasks.workunit.client.0.smithi113.stdout:1/654: stat d10/d14/d67/d70/f48 0 2024-04-12T05:09:03.497 INFO:tasks.workunit.client.0.smithi113.stdout:1/655: symlink d10/d14/d95/lce 0 2024-04-12T05:09:03.502 INFO:tasks.workunit.client.0.smithi113.stdout:1/656: dread d10/d14/d67/d70/fb1 [0,4194304] 0 2024-04-12T05:09:03.504 INFO:tasks.workunit.client.0.smithi113.stdout:1/657: mkdir d10/d14/d95/dcf 0 2024-04-12T05:09:03.505 INFO:tasks.workunit.client.0.smithi113.stdout:1/658: write d10/d41/f60 [1272654,126374] 0 2024-04-12T05:09:03.509 INFO:tasks.workunit.client.1.smithi192.stdout:0/609: rename d0/d6/d15/d55/d2e/d66/db7 to d0/d7b/d7c/dcc 0 2024-04-12T05:09:03.510 INFO:tasks.workunit.client.1.smithi192.stdout:9/403: truncate d11/d15/f5c 1036263 0 2024-04-12T05:09:03.510 INFO:tasks.workunit.client.1.smithi192.stdout:0/610: fsync d0/d6/d15/f6f 0 2024-04-12T05:09:03.511 INFO:tasks.workunit.client.1.smithi192.stdout:4/583: unlink d1/d1d/d53/f57 0 2024-04-12T05:09:03.511 INFO:tasks.workunit.client.1.smithi192.stdout:6/558: getdents d3/d5/d9e 0 2024-04-12T05:09:03.512 INFO:tasks.workunit.client.1.smithi192.stdout:9/404: write d11/d25/d36/d6e/f77 [666228,112573] 0 2024-04-12T05:09:03.513 INFO:tasks.workunit.client.1.smithi192.stdout:0/611: truncate d0/d4e/d95/db4/db8/fcb 1044115 0 2024-04-12T05:09:03.514 INFO:tasks.workunit.client.1.smithi192.stdout:4/584: symlink d1/d49/lc2 0 2024-04-12T05:09:03.516 INFO:tasks.workunit.client.1.smithi192.stdout:4/585: fsync d1/d54/f6e 0 2024-04-12T05:09:03.516 INFO:tasks.workunit.client.1.smithi192.stdout:7/450: getdents d3/d7/d14 0 2024-04-12T05:09:03.516 INFO:tasks.workunit.client.1.smithi192.stdout:0/612: write d0/d7b/d7c/dcc/f69 [890221,121540] 0 2024-04-12T05:09:03.517 INFO:tasks.workunit.client.1.smithi192.stdout:9/405: mknod d11/d15/c85 0 2024-04-12T05:09:03.518 INFO:tasks.workunit.client.1.smithi192.stdout:4/586: read d1/f4b [2258146,107906] 0 2024-04-12T05:09:03.519 INFO:tasks.workunit.client.1.smithi192.stdout:6/559: rename d3/d21/c6b to d3/d8/d22/cb6 0 2024-04-12T05:09:03.519 INFO:tasks.workunit.client.1.smithi192.stdout:4/587: dread - d1/d54/d55/d64/d68/faa zero size 2024-04-12T05:09:03.526 INFO:tasks.workunit.client.1.smithi192.stdout:0/613: write d0/d6/d15/d1b/d3a/d4f/f5a [948351,60084] 0 2024-04-12T05:09:03.527 INFO:tasks.workunit.client.1.smithi192.stdout:4/588: write d1/d49/f69 [3471615,22316] 0 2024-04-12T05:09:03.530 INFO:tasks.workunit.client.1.smithi192.stdout:6/560: getdents d3/d8 0 2024-04-12T05:09:03.531 INFO:tasks.workunit.client.1.smithi192.stdout:4/589: mkdir d1/d54/d55/dc3 0 2024-04-12T05:09:03.532 INFO:tasks.workunit.client.1.smithi192.stdout:0/614: unlink d0/d6/d15/d19/f4c 0 2024-04-12T05:09:03.534 INFO:tasks.workunit.client.1.smithi192.stdout:9/406: dwrite d11/d25/d36/d3d/f79 [0,4194304] 0 2024-04-12T05:09:03.535 INFO:tasks.workunit.client.1.smithi192.stdout:0/615: mknod d0/d6/d67/ccd 0 2024-04-12T05:09:03.536 INFO:tasks.workunit.client.1.smithi192.stdout:4/590: getdents d1/d1d/d53/d1c/d84 0 2024-04-12T05:09:03.538 INFO:tasks.workunit.client.1.smithi192.stdout:9/407: creat d11/d15/d21/d2c/d1f/f86 x:0 0 0 2024-04-12T05:09:03.543 INFO:tasks.workunit.client.1.smithi192.stdout:4/591: write d1/f2 [1073038,57232] 0 2024-04-12T05:09:03.543 INFO:tasks.workunit.client.1.smithi192.stdout:9/408: link d11/d15/d1e/l40 d11/d15/d21/l87 0 2024-04-12T05:09:03.545 INFO:tasks.workunit.client.1.smithi192.stdout:4/592: mkdir d1/d54/d82/d71/d9e/dc4 0 2024-04-12T05:09:03.547 INFO:tasks.workunit.client.1.smithi192.stdout:4/593: fsync d1/d54/d82/f5d 0 2024-04-12T05:09:03.557 INFO:tasks.workunit.client.1.smithi192.stdout:9/409: dread d11/f12 [0,4194304] 0 2024-04-12T05:09:03.558 INFO:tasks.workunit.client.1.smithi192.stdout:9/410: write d11/d25/f70 [323856,61338] 0 2024-04-12T05:09:03.560 INFO:tasks.workunit.client.1.smithi192.stdout:9/411: mknod d11/d15/c88 0 2024-04-12T05:09:03.562 INFO:tasks.workunit.client.1.smithi192.stdout:4/594: dread d1/d1d/d53/d1c/f66 [0,4194304] 0 2024-04-12T05:09:03.565 INFO:tasks.workunit.client.1.smithi192.stdout:4/595: creat d1/d54/fc5 x:0 0 0 2024-04-12T05:09:03.567 INFO:tasks.workunit.client.1.smithi192.stdout:9/412: dwrite d11/d25/d36/d3d/f79 [4194304,4194304] 0 2024-04-12T05:09:03.569 INFO:tasks.workunit.client.1.smithi192.stdout:4/596: mkdir d1/d1d/d53/d1c/d95/dc6 0 2024-04-12T05:09:03.570 INFO:tasks.workunit.client.1.smithi192.stdout:9/413: mkdir d11/d15/d21/d2c/d1f/d6a/d7e/d89 0 2024-04-12T05:09:03.571 INFO:tasks.workunit.client.1.smithi192.stdout:9/414: readlink d11/d15/d21/d2c/l16 0 2024-04-12T05:09:03.575 INFO:tasks.workunit.client.1.smithi192.stdout:4/597: dwrite d1/d54/d82/f48 [0,4194304] 0 2024-04-12T05:09:03.575 INFO:tasks.workunit.client.1.smithi192.stdout:4/598: chown d1/d1d/l35 7 1 2024-04-12T05:09:03.576 INFO:tasks.workunit.client.1.smithi192.stdout:2/504: sync 2024-04-12T05:09:03.576 INFO:tasks.workunit.client.1.smithi192.stdout:3/465: sync 2024-04-12T05:09:03.576 INFO:tasks.workunit.client.1.smithi192.stdout:5/474: sync 2024-04-12T05:09:03.577 INFO:tasks.workunit.client.1.smithi192.stdout:9/415: creat d11/d15/d21/d2c/f8a x:0 0 0 2024-04-12T05:09:03.584 INFO:tasks.workunit.client.1.smithi192.stdout:3/466: stat d5/d13/d1a/d5f/fa1 0 2024-04-12T05:09:03.584 INFO:tasks.workunit.client.1.smithi192.stdout:9/416: dread - d11/d15/d21/f6f zero size 2024-04-12T05:09:03.584 INFO:tasks.workunit.client.1.smithi192.stdout:4/599: fdatasync d1/d54/d82/d71/d9e/fa0 0 2024-04-12T05:09:03.584 INFO:tasks.workunit.client.1.smithi192.stdout:2/505: mkdir d1/d8/d17/d19/d71/da0 0 2024-04-12T05:09:03.584 INFO:tasks.workunit.client.1.smithi192.stdout:3/467: fsync d5/d13/d3c/d26/f4d 0 2024-04-12T05:09:03.584 INFO:tasks.workunit.client.1.smithi192.stdout:4/600: chown d1/d1d/d53/lb7 1564136 1 2024-04-12T05:09:03.584 INFO:tasks.workunit.client.1.smithi192.stdout:5/475: mkdir d0/d15/da1 0 2024-04-12T05:09:03.586 INFO:tasks.workunit.client.1.smithi192.stdout:5/476: chown d0/d15/d64/f70 487 1 2024-04-12T05:09:03.586 INFO:tasks.workunit.client.1.smithi192.stdout:3/468: read d5/d13/d3c/d26/f44 [787797,18178] 0 2024-04-12T05:09:03.589 INFO:tasks.workunit.client.1.smithi192.stdout:4/601: link d1/f2 d1/d54/d82/d71/d9e/fc7 0 2024-04-12T05:09:03.590 INFO:tasks.workunit.client.1.smithi192.stdout:5/477: mknod d0/d15/d64/ca2 0 2024-04-12T05:09:03.592 INFO:tasks.workunit.client.1.smithi192.stdout:4/602: write d1/d1d/d53/d1c/f65 [4914561,13307] 0 2024-04-12T05:09:03.593 INFO:tasks.workunit.client.1.smithi192.stdout:2/506: dwrite d1/d8/f22 [0,4194304] 0 2024-04-12T05:09:03.595 INFO:tasks.workunit.client.1.smithi192.stdout:5/478: mknod d0/d15/d24/d47/d7b/ca3 0 2024-04-12T05:09:03.597 INFO:tasks.workunit.client.1.smithi192.stdout:4/603: chown d1/d54/d82/d71/fb0 59 1 2024-04-12T05:09:03.600 INFO:tasks.workunit.client.1.smithi192.stdout:3/469: dwrite d5/d13/d3a/d4e/f6c [0,4194304] 0 2024-04-12T05:09:03.601 INFO:tasks.workunit.client.1.smithi192.stdout:4/604: write d1/d1d/d53/d1c/fb1 [639916,9986] 0 2024-04-12T05:09:03.602 INFO:tasks.workunit.client.1.smithi192.stdout:3/470: fsync d5/d13/d3a/d4e/d57/f6e 0 2024-04-12T05:09:03.603 INFO:tasks.workunit.client.1.smithi192.stdout:2/507: write d1/d2d/f6f [821664,96609] 0 2024-04-12T05:09:03.604 INFO:tasks.workunit.client.1.smithi192.stdout:3/471: dread - d5/d13/d3c/f31 zero size 2024-04-12T05:09:03.606 INFO:tasks.workunit.client.1.smithi192.stdout:5/479: link d0/d15/l62 d0/d15/d24/d47/d7b/la4 0 2024-04-12T05:09:03.608 INFO:tasks.workunit.client.1.smithi192.stdout:4/605: unlink d1/d1d/d53/l9b 0 2024-04-12T05:09:03.608 INFO:tasks.workunit.client.1.smithi192.stdout:9/417: dread d11/d15/d21/d2c/f1a [0,4194304] 0 2024-04-12T05:09:03.608 INFO:tasks.workunit.client.1.smithi192.stdout:5/480: write d0/d15/d32/d45/f86 [372645,32976] 0 2024-04-12T05:09:03.610 INFO:tasks.workunit.client.1.smithi192.stdout:3/472: unlink d5/d13/d3a/d85/c9b 0 2024-04-12T05:09:03.611 INFO:tasks.workunit.client.1.smithi192.stdout:2/508: symlink d1/d8/d17/d19/d60/d59/la1 0 2024-04-12T05:09:03.612 INFO:tasks.workunit.client.1.smithi192.stdout:4/606: write d1/d54/d82/da4/f92 [3004751,3313] 0 2024-04-12T05:09:03.613 INFO:tasks.workunit.client.1.smithi192.stdout:5/481: unlink d0/d15/d64/c6d 0 2024-04-12T05:09:03.614 INFO:tasks.workunit.client.1.smithi192.stdout:3/473: creat d5/d13/d3a/d4e/fa8 x:0 0 0 2024-04-12T05:09:03.616 INFO:tasks.workunit.client.1.smithi192.stdout:9/418: rename d11/d15/d21/f2d to d11/d15/d21/d2c/d1f/d50/d73/f8b 0 2024-04-12T05:09:03.620 INFO:tasks.workunit.client.1.smithi192.stdout:3/474: creat d5/d13/d1a/fa9 x:0 0 0 2024-04-12T05:09:03.620 INFO:tasks.workunit.client.1.smithi192.stdout:5/482: truncate d0/d15/d32/d45/f7f 541667 0 2024-04-12T05:09:03.620 INFO:tasks.workunit.client.1.smithi192.stdout:3/475: write d5/f7a [3896166,30178] 0 2024-04-12T05:09:03.626 INFO:tasks.workunit.client.1.smithi192.stdout:5/483: mknod d0/d21/ca5 0 2024-04-12T05:09:03.627 INFO:tasks.workunit.client.1.smithi192.stdout:9/419: mkdir d11/d25/d60/d8c 0 2024-04-12T05:09:03.629 INFO:tasks.workunit.client.1.smithi192.stdout:3/476: rename d5/d13/d1a/d53/f88 to d5/d13/d3a/d4e/d57/faa 0 2024-04-12T05:09:03.632 INFO:tasks.workunit.client.1.smithi192.stdout:4/607: creat d1/d54/d55/dc3/fc8 x:0 0 0 2024-04-12T05:09:03.632 INFO:tasks.workunit.client.1.smithi192.stdout:2/509: write d1/d2d/f5d [948721,110864] 0 2024-04-12T05:09:03.633 INFO:tasks.workunit.client.1.smithi192.stdout:9/420: truncate f3 2213670 0 2024-04-12T05:09:03.634 INFO:tasks.workunit.client.1.smithi192.stdout:2/510: chown d1/d8/d3a/f2e 418801 1 2024-04-12T05:09:03.635 INFO:tasks.workunit.client.1.smithi192.stdout:5/484: creat d0/d15/d64/d7c/da0/fa6 x:0 0 0 2024-04-12T05:09:03.636 INFO:tasks.workunit.client.1.smithi192.stdout:3/477: mknod d5/d13/d3a/cab 0 2024-04-12T05:09:03.637 INFO:tasks.workunit.client.1.smithi192.stdout:9/421: mknod d11/d25/d36/d6e/d33/c8d 0 2024-04-12T05:09:03.637 INFO:tasks.workunit.client.1.smithi192.stdout:4/608: mknod d1/d54/d82/da1/cc9 0 2024-04-12T05:09:03.639 INFO:tasks.workunit.client.1.smithi192.stdout:2/511: truncate d1/d8/d17/d19/d60/f66 1348013 0 2024-04-12T05:09:03.640 INFO:tasks.workunit.client.1.smithi192.stdout:9/422: symlink d11/d15/d1e/l8e 0 2024-04-12T05:09:03.640 INFO:tasks.workunit.client.1.smithi192.stdout:3/478: truncate d5/d13/d3a/d85/fa4 201829 0 2024-04-12T05:09:03.642 INFO:tasks.workunit.client.1.smithi192.stdout:2/512: write d1/d8/d3a/f95 [2411235,58942] 0 2024-04-12T05:09:03.645 INFO:tasks.workunit.client.1.smithi192.stdout:4/609: mknod d1/d1d/d53/d1c/dc1/cca 0 2024-04-12T05:09:03.645 INFO:tasks.workunit.client.1.smithi192.stdout:5/485: dwrite d0/d15/d24/f46 [4194304,4194304] 0 2024-04-12T05:09:03.655 INFO:tasks.workunit.client.1.smithi192.stdout:5/486: fsync d0/fa 0 2024-04-12T05:09:03.656 INFO:tasks.workunit.client.1.smithi192.stdout:4/610: write d1/d54/d55/dc3/fc8 [731282,54630] 0 2024-04-12T05:09:03.658 INFO:tasks.workunit.client.1.smithi192.stdout:3/479: symlink d5/d13/d3c/d36/da0/lac 0 2024-04-12T05:09:03.660 INFO:tasks.workunit.client.1.smithi192.stdout:3/480: write d5/d13/d1a/d53/f60 [4238010,26768] 0 2024-04-12T05:09:03.665 INFO:tasks.workunit.client.1.smithi192.stdout:4/611: truncate d1/d54/f6e 376786 0 2024-04-12T05:09:03.665 INFO:tasks.workunit.client.1.smithi192.stdout:2/513: link d1/d8/d17/d19/d60/d59/c81 d1/ca2 0 2024-04-12T05:09:03.667 INFO:tasks.workunit.client.1.smithi192.stdout:5/487: dwrite d0/d15/d32/d69/f9e [0,4194304] 0 2024-04-12T05:09:03.667 INFO:tasks.workunit.client.1.smithi192.stdout:2/514: read - d1/d2d/d73/f9b zero size 2024-04-12T05:09:03.668 INFO:tasks.workunit.client.1.smithi192.stdout:4/612: dread - d1/d54/d55/d64/f86 zero size 2024-04-12T05:09:03.669 INFO:tasks.workunit.client.1.smithi192.stdout:2/515: read d1/d2d/f3d [1726334,14101] 0 2024-04-12T05:09:03.669 INFO:tasks.workunit.client.1.smithi192.stdout:5/488: mkdir d0/d15/d64/d7c/da7 0 2024-04-12T05:09:03.673 INFO:tasks.workunit.client.1.smithi192.stdout:4/613: creat d1/d1d/d53/d1c/fcb x:0 0 0 2024-04-12T05:09:03.673 INFO:tasks.workunit.client.1.smithi192.stdout:3/481: getdents d5/d13/d3a/d8b/da2 0 2024-04-12T05:09:03.674 INFO:tasks.workunit.client.1.smithi192.stdout:2/516: mkdir d1/d8/d17/d19/d60/d59/da3 0 2024-04-12T05:09:03.675 INFO:tasks.workunit.client.1.smithi192.stdout:4/614: write d1/d54/d55/d64/d68/faa [647204,70844] 0 2024-04-12T05:09:03.676 INFO:tasks.workunit.client.1.smithi192.stdout:2/517: truncate d1/f8b 4597306 0 2024-04-12T05:09:03.676 INFO:tasks.workunit.client.1.smithi192.stdout:3/482: truncate d5/d13/d3a/d4a/d61/f64 1387817 0 2024-04-12T05:09:03.677 INFO:tasks.workunit.client.1.smithi192.stdout:4/615: fsync d1/d54/d55/d64/fae 0 2024-04-12T05:09:03.677 INFO:tasks.workunit.client.1.smithi192.stdout:8/407: sync 2024-04-12T05:09:03.679 INFO:tasks.workunit.client.1.smithi192.stdout:2/518: getdents d1/d8/d3a 0 2024-04-12T05:09:03.680 INFO:tasks.workunit.client.1.smithi192.stdout:2/519: stat d1/d2d/f6f 0 2024-04-12T05:09:03.682 INFO:tasks.workunit.client.1.smithi192.stdout:5/489: fsync d0/d15/d32/d69/f9e 0 2024-04-12T05:09:03.686 INFO:tasks.workunit.client.1.smithi192.stdout:5/490: creat d0/d15/da1/fa8 x:0 0 0 2024-04-12T05:09:03.686 INFO:tasks.workunit.client.1.smithi192.stdout:8/408: write d6/f18 [993259,131016] 0 2024-04-12T05:09:03.688 INFO:tasks.workunit.client.1.smithi192.stdout:5/491: chown d0/f14 15829133 1 2024-04-12T05:09:03.689 INFO:tasks.workunit.client.1.smithi192.stdout:2/520: dwrite d1/d8/d17/d19/d40/f52 [0,4194304] 0 2024-04-12T05:09:03.696 INFO:tasks.workunit.client.1.smithi192.stdout:3/483: dread d5/f14 [0,4194304] 0 2024-04-12T05:09:03.698 INFO:tasks.workunit.client.1.smithi192.stdout:3/484: chown d5/d13/d1a/d53/f58 1541869 1 2024-04-12T05:09:03.699 INFO:tasks.workunit.client.1.smithi192.stdout:8/409: truncate d6/d10/d46/d5e/d5f/d38/f5a 38852 0 2024-04-12T05:09:03.700 INFO:tasks.workunit.client.1.smithi192.stdout:5/492: getdents d0/d15/d64/d7c/da0 0 2024-04-12T05:09:03.700 INFO:tasks.workunit.client.1.smithi192.stdout:8/410: readlink d6/d1f/d6a/l69 0 2024-04-12T05:09:03.701 INFO:tasks.workunit.client.1.smithi192.stdout:3/485: truncate d5/d13/d3a/d85/fa4 620158 0 2024-04-12T05:09:03.703 INFO:tasks.workunit.client.1.smithi192.stdout:8/411: chown d6/c40 0 1 2024-04-12T05:09:03.703 INFO:tasks.workunit.client.1.smithi192.stdout:5/493: creat d0/d21/fa9 x:0 0 0 2024-04-12T05:09:03.703 INFO:tasks.workunit.client.1.smithi192.stdout:3/486: chown d5/d13/d3c/d26/c2c 8356645 1 2024-04-12T05:09:03.704 INFO:tasks.workunit.client.1.smithi192.stdout:2/521: dwrite d1/d2d/f3d [0,4194304] 0 2024-04-12T05:09:03.708 INFO:tasks.workunit.client.1.smithi192.stdout:2/522: readlink d1/d8/l36 0 2024-04-12T05:09:03.711 INFO:tasks.workunit.client.1.smithi192.stdout:8/412: rename d6/d10/d46/d5e/d5f/d37/d4f/d4c/f58 to d6/d10/d46/d5e/d5f/f7e 0 2024-04-12T05:09:03.712 INFO:tasks.workunit.client.1.smithi192.stdout:3/487: dwrite d5/d13/f2e [0,4194304] 0 2024-04-12T05:09:03.714 INFO:tasks.workunit.client.1.smithi192.stdout:8/413: mkdir d6/d10/d7f 0 2024-04-12T05:09:03.718 INFO:tasks.workunit.client.1.smithi192.stdout:3/488: creat d5/d13/d3a/d70/d8c/d8d/fad x:0 0 0 2024-04-12T05:09:03.719 INFO:tasks.workunit.client.1.smithi192.stdout:8/414: mkdir d6/d10/d46/d5e/d5f/d31/d80 0 2024-04-12T05:09:03.721 INFO:tasks.workunit.client.1.smithi192.stdout:3/489: rename d5/d13/f2e to d5/d13/d3a/d8b/fae 0 2024-04-12T05:09:03.723 INFO:tasks.workunit.client.1.smithi192.stdout:2/523: dread d1/d8/f41 [0,4194304] 0 2024-04-12T05:09:03.728 INFO:tasks.workunit.client.1.smithi192.stdout:3/490: mkdir d5/d13/d3a/d70/d8c/d8d/daf 0 2024-04-12T05:09:03.729 INFO:tasks.workunit.client.1.smithi192.stdout:2/524: chown d1/d8/d17/d19/c78 355 1 2024-04-12T05:09:03.729 INFO:tasks.workunit.client.1.smithi192.stdout:8/415: truncate d6/d10/d46/d5e/d5f/d37/d4f/d4c/f55 3359955 0 2024-04-12T05:09:03.731 INFO:tasks.workunit.client.1.smithi192.stdout:2/525: write d1/d8/d3a/de/f97 [630180,128145] 0 2024-04-12T05:09:03.733 INFO:tasks.workunit.client.1.smithi192.stdout:3/491: creat d5/d13/d3a/d70/d8c/fb0 x:0 0 0 2024-04-12T05:09:03.733 INFO:tasks.workunit.client.1.smithi192.stdout:8/416: creat d6/d10/d46/d5e/d5f/d38/d3b/f81 x:0 0 0 2024-04-12T05:09:03.734 INFO:tasks.workunit.client.1.smithi192.stdout:2/526: write d1/f8b [826904,72259] 0 2024-04-12T05:09:03.736 INFO:tasks.workunit.client.1.smithi192.stdout:3/492: mknod d5/d13/d3a/d85/cb1 0 2024-04-12T05:09:03.737 INFO:tasks.workunit.client.1.smithi192.stdout:3/493: chown d5/d13/f2a 64228856 1 2024-04-12T05:09:03.740 INFO:tasks.workunit.client.1.smithi192.stdout:8/417: write d6/da/f30 [639302,10787] 0 2024-04-12T05:09:03.743 INFO:tasks.workunit.client.1.smithi192.stdout:8/418: write d6/d10/d46/d5e/d5f/f16 [2396450,49084] 0 2024-04-12T05:09:03.743 INFO:tasks.workunit.client.1.smithi192.stdout:3/494: write d5/d13/f18 [8627145,8170] 0 2024-04-12T05:09:03.744 INFO:tasks.workunit.client.1.smithi192.stdout:8/419: chown d6/c25 1987 1 2024-04-12T05:09:03.744 INFO:tasks.workunit.client.1.smithi192.stdout:3/495: fdatasync d5/d13/d1a/d5f/f75 0 2024-04-12T05:09:03.747 INFO:tasks.workunit.client.1.smithi192.stdout:8/420: mknod d6/d10/d46/d5e/d5f/d38/c82 0 2024-04-12T05:09:03.748 INFO:tasks.workunit.client.1.smithi192.stdout:3/496: mknod d5/d13/d3a/d4e/d57/cb2 0 2024-04-12T05:09:03.749 INFO:tasks.workunit.client.1.smithi192.stdout:8/421: symlink d6/d10/d46/l83 0 2024-04-12T05:09:03.750 INFO:tasks.workunit.client.1.smithi192.stdout:3/497: creat d5/d13/d3a/d70/d8c/fb3 x:0 0 0 2024-04-12T05:09:03.751 INFO:tasks.workunit.client.1.smithi192.stdout:3/498: read - d5/d13/d3a/d70/d8c/f96 zero size 2024-04-12T05:09:03.751 INFO:tasks.workunit.client.1.smithi192.stdout:8/422: symlink d6/d10/d46/d5e/d5f/d38/l84 0 2024-04-12T05:09:03.759 INFO:tasks.workunit.client.1.smithi192.stdout:3/499: dwrite d5/d13/d3a/d4e/f95 [0,4194304] 0 2024-04-12T05:09:03.760 INFO:tasks.workunit.client.1.smithi192.stdout:8/423: dwrite d6/d10/d46/d5e/d5f/f57 [0,4194304] 0 2024-04-12T05:09:03.763 INFO:tasks.workunit.client.1.smithi192.stdout:8/424: fsync d6/da/f49 0 2024-04-12T05:09:03.768 INFO:tasks.workunit.client.1.smithi192.stdout:3/500: getdents d5/d13/d1a 0 2024-04-12T05:09:03.769 INFO:tasks.workunit.client.1.smithi192.stdout:8/425: link d6/d21/f6d d6/d10/d46/f85 0 2024-04-12T05:09:03.772 INFO:tasks.workunit.client.1.smithi192.stdout:8/426: symlink d6/d10/d46/d5e/d5f/d37/d4f/d51/l86 0 2024-04-12T05:09:03.772 INFO:tasks.workunit.client.1.smithi192.stdout:8/427: readlink d6/d21/l2e 0 2024-04-12T05:09:03.774 INFO:tasks.workunit.client.0.smithi113.stdout:7/566: sync 2024-04-12T05:09:03.774 INFO:tasks.workunit.client.0.smithi113.stdout:9/602: sync 2024-04-12T05:09:03.776 INFO:tasks.workunit.client.0.smithi113.stdout:9/603: mkdir d1/d4/d38/d65/dcd 0 2024-04-12T05:09:03.780 INFO:tasks.workunit.client.0.smithi113.stdout:7/567: rename d0/dc/da9/d5c/l6e to d0/dc/d6/d1a/lc4 0 2024-04-12T05:09:03.782 INFO:tasks.workunit.client.0.smithi113.stdout:9/604: rename d1/d27/c85 to d1/d4/d5/d22/cce 0 2024-04-12T05:09:03.783 INFO:tasks.workunit.client.0.smithi113.stdout:7/568: fdatasync d0/d52/f54 0 2024-04-12T05:09:03.785 INFO:tasks.workunit.client.0.smithi113.stdout:7/569: mkdir d0/d52/dc5 0 2024-04-12T05:09:03.816 INFO:tasks.workunit.client.1.smithi192.stdout:8/428: dread d6/d21/f33 [0,4194304] 0 2024-04-12T05:09:03.818 INFO:tasks.workunit.client.1.smithi192.stdout:3/501: write d5/fd [960958,24184] 0 2024-04-12T05:09:03.822 INFO:tasks.workunit.client.1.smithi192.stdout:3/502: rename d5/d13/d3a/d70/d8c/d8d/fad to d5/d13/d3a/d70/d8c/fb4 0 2024-04-12T05:09:03.825 INFO:tasks.workunit.client.1.smithi192.stdout:3/503: creat d5/d13/d3a/d8b/fb5 x:0 0 0 2024-04-12T05:09:03.828 INFO:tasks.workunit.client.1.smithi192.stdout:3/504: mknod d5/d13/d3a/d8b/cb6 0 2024-04-12T05:09:03.829 INFO:tasks.workunit.client.1.smithi192.stdout:3/505: fsync d5/d13/d1a/f3b 0 2024-04-12T05:09:03.830 INFO:tasks.workunit.client.1.smithi192.stdout:3/506: readlink d5/d13/d3a/l69 0 2024-04-12T05:09:03.834 INFO:tasks.workunit.client.0.smithi113.stdout:7/570: rmdir d0 39 2024-04-12T05:09:03.835 INFO:tasks.workunit.client.0.smithi113.stdout:9/605: truncate d1/d4/d5/d17/f74 1870736 0 2024-04-12T05:09:03.837 INFO:tasks.workunit.client.1.smithi192.stdout:3/507: dwrite d5/d13/d3a/d70/d8c/fb0 [0,4194304] 0 2024-04-12T05:09:03.839 INFO:tasks.workunit.client.1.smithi192.stdout:3/508: fsync d5/d13/d3a/d4e/f6c 0 2024-04-12T05:09:03.840 INFO:tasks.workunit.client.0.smithi113.stdout:7/571: rename d0/f5a to d0/dc/d6/d1a/da0/fc6 0 2024-04-12T05:09:03.841 INFO:tasks.workunit.client.0.smithi113.stdout:7/572: chown d0/dc/d6/d78/daa/la3 23 1 2024-04-12T05:09:03.841 INFO:tasks.workunit.client.0.smithi113.stdout:9/606: truncate d1/df/d4b/d63/d3d/d6f/fa8 768020 0 2024-04-12T05:09:03.847 INFO:tasks.workunit.client.0.smithi113.stdout:9/607: rename d1/d4/f18 to d1/d4/d38/d65/fcf 0 2024-04-12T05:09:03.847 INFO:tasks.workunit.client.0.smithi113.stdout:9/608: chown d1/df/d29/f7b 1 1 2024-04-12T05:09:03.850 INFO:tasks.workunit.client.1.smithi192.stdout:3/509: dwrite d5/d13/d3a/d4a/f66 [0,4194304] 0 2024-04-12T05:09:03.851 INFO:tasks.workunit.client.0.smithi113.stdout:9/609: creat d1/d4/d38/fd0 x:0 0 0 2024-04-12T05:09:03.852 INFO:tasks.workunit.client.1.smithi192.stdout:8/429: dread d6/d10/d46/d5e/d5f/d37/f48 [0,4194304] 0 2024-04-12T05:09:03.856 INFO:tasks.workunit.client.1.smithi192.stdout:3/510: creat d5/d13/d3c/d36/da0/fb7 x:0 0 0 2024-04-12T05:09:03.860 INFO:tasks.workunit.client.1.smithi192.stdout:3/511: mknod d5/d13/d3c/d84/cb8 0 2024-04-12T05:09:03.860 INFO:tasks.workunit.client.0.smithi113.stdout:7/573: dwrite d0/dc/d6/d9b/f62 [0,4194304] 0 2024-04-12T05:09:03.862 INFO:tasks.workunit.client.1.smithi192.stdout:1/532: sync 2024-04-12T05:09:03.864 INFO:tasks.workunit.client.1.smithi192.stdout:8/430: dwrite f3 [0,4194304] 0 2024-04-12T05:09:03.865 INFO:tasks.workunit.client.0.smithi113.stdout:9/610: dwrite d1/d4/d5/d22/f5a [0,4194304] 0 2024-04-12T05:09:03.867 INFO:tasks.workunit.client.0.smithi113.stdout:7/574: symlink d0/lc7 0 2024-04-12T05:09:03.869 INFO:tasks.workunit.client.0.smithi113.stdout:9/611: mknod d1/d4/d5/d22/d9e/cd1 0 2024-04-12T05:09:03.870 INFO:tasks.workunit.client.0.smithi113.stdout:7/575: rmdir d0/dc/d6/d78 39 2024-04-12T05:09:03.871 INFO:tasks.workunit.client.1.smithi192.stdout:1/533: dwrite d0/d2/dd/f4b [0,4194304] 0 2024-04-12T05:09:03.874 INFO:tasks.workunit.client.0.smithi113.stdout:9/612: rmdir d1/df/d4b/d63/d3d/dab 39 2024-04-12T05:09:03.875 INFO:tasks.workunit.client.1.smithi192.stdout:1/534: dread d0/d2/d1e/d3f/d56/f79 [0,4194304] 0 2024-04-12T05:09:03.875 INFO:tasks.workunit.client.0.smithi113.stdout:7/576: fdatasync d0/dc/d6/d1a/f2b 0 2024-04-12T05:09:03.876 INFO:tasks.workunit.client.1.smithi192.stdout:1/535: read d0/d1b/f53 [3280895,27062] 0 2024-04-12T05:09:03.876 INFO:tasks.workunit.client.0.smithi113.stdout:9/613: chown d1/d4/f98 22774 1 2024-04-12T05:09:03.877 INFO:tasks.workunit.client.1.smithi192.stdout:8/431: getdents d6/d10/d46/d5e/d5f/d37/d4f/d4c 0 2024-04-12T05:09:03.877 INFO:tasks.workunit.client.0.smithi113.stdout:7/577: write d0/dc/d6/fc0 [535213,48652] 0 2024-04-12T05:09:03.878 INFO:tasks.workunit.client.0.smithi113.stdout:7/578: chown d0/dd/d11/f19 7902 1 2024-04-12T05:09:03.881 INFO:tasks.workunit.client.0.smithi113.stdout:7/579: dread - d0/dc/d6/f86 zero size 2024-04-12T05:09:03.881 INFO:tasks.workunit.client.0.smithi113.stdout:7/580: chown d0/f48 1216527260 1 2024-04-12T05:09:03.887 INFO:tasks.workunit.client.1.smithi192.stdout:1/536: chown d0/d2/dd/d3c/d5b/d74/d7d/d9b 137 1 2024-04-12T05:09:03.887 INFO:tasks.workunit.client.1.smithi192.stdout:8/432: mknod d6/d10/d46/d5e/d5f/c87 0 2024-04-12T05:09:03.890 INFO:tasks.workunit.client.1.smithi192.stdout:1/537: write d0/d2/d1e/d3f/d56/f79 [511253,73916] 0 2024-04-12T05:09:03.890 INFO:tasks.workunit.client.0.smithi113.stdout:7/581: truncate d0/dc/d6/d78/f88 148166 0 2024-04-12T05:09:03.891 INFO:tasks.workunit.client.1.smithi192.stdout:1/538: write d0/d2/dd/f4b [3416645,36735] 0 2024-04-12T05:09:03.893 INFO:tasks.workunit.client.0.smithi113.stdout:7/582: chown d0/dd/d2d/c67 28264397 1 2024-04-12T05:09:03.894 INFO:tasks.workunit.client.1.smithi192.stdout:3/512: dread d5/d13/d3a/d4a/d61/f64 [0,4194304] 0 2024-04-12T05:09:03.895 INFO:tasks.workunit.client.1.smithi192.stdout:3/513: chown d5/d13/d1a/f3b 654 1 2024-04-12T05:09:03.897 INFO:tasks.workunit.client.1.smithi192.stdout:8/433: creat d6/d10/d46/d5e/d5f/f88 x:0 0 0 2024-04-12T05:09:03.899 INFO:tasks.workunit.client.1.smithi192.stdout:3/514: symlink d5/d13/lb9 0 2024-04-12T05:09:03.900 INFO:tasks.workunit.client.1.smithi192.stdout:3/515: stat d5/d13/d3c/d26/f9e 0 2024-04-12T05:09:03.902 INFO:tasks.workunit.client.1.smithi192.stdout:3/516: write d5/d13/d3a/d70/d8c/fb4 [945301,112841] 0 2024-04-12T05:09:03.906 INFO:tasks.workunit.client.1.smithi192.stdout:8/434: dwrite d6/d10/d46/d5e/d5f/d37/d4f/d4c/f64 [0,4194304] 0 2024-04-12T05:09:03.911 INFO:tasks.workunit.client.1.smithi192.stdout:3/517: dwrite d5/d13/f18 [0,4194304] 0 2024-04-12T05:09:03.914 INFO:tasks.workunit.client.1.smithi192.stdout:3/518: write d5/d13/d3a/d4a/d61/f64 [19775,115988] 0 2024-04-12T05:09:03.918 INFO:tasks.workunit.client.1.smithi192.stdout:3/519: creat d5/d13/d3a/d8b/fba x:0 0 0 2024-04-12T05:09:03.918 INFO:tasks.workunit.client.0.smithi113.stdout:9/614: write d1/d4/d38/f8b [1021377,129121] 0 2024-04-12T05:09:03.921 INFO:tasks.workunit.client.1.smithi192.stdout:1/539: truncate d0/d2/dd/f32 1296741 0 2024-04-12T05:09:03.921 INFO:tasks.workunit.client.1.smithi192.stdout:8/435: write d6/da/f1e [1497769,62728] 0 2024-04-12T05:09:03.923 INFO:tasks.workunit.client.1.smithi192.stdout:3/520: mkdir d5/d13/d1a/d53/dbb 0 2024-04-12T05:09:03.924 INFO:tasks.workunit.client.1.smithi192.stdout:3/521: readlink d5/d13/d3a/l69 0 2024-04-12T05:09:03.925 INFO:tasks.workunit.client.1.smithi192.stdout:9/423: sync 2024-04-12T05:09:03.925 INFO:tasks.workunit.client.1.smithi192.stdout:7/451: sync 2024-04-12T05:09:03.925 INFO:tasks.workunit.client.1.smithi192.stdout:0/616: sync 2024-04-12T05:09:03.925 INFO:tasks.workunit.client.1.smithi192.stdout:6/561: sync 2024-04-12T05:09:03.926 INFO:tasks.workunit.client.1.smithi192.stdout:3/522: write d5/f55 [1956719,67492] 0 2024-04-12T05:09:03.927 INFO:tasks.workunit.client.1.smithi192.stdout:9/424: chown d11/d15/c18 2668 1 2024-04-12T05:09:03.928 INFO:tasks.workunit.client.1.smithi192.stdout:1/540: mkdir d0/d2/dd/d3c/d5b/da7 0 2024-04-12T05:09:03.931 INFO:tasks.workunit.client.1.smithi192.stdout:7/452: creat d3/d1d/d4a/d5e/f9a x:0 0 0 2024-04-12T05:09:03.932 INFO:tasks.workunit.client.0.smithi113.stdout:7/583: dread d0/dc/f18 [4194304,4194304] 0 2024-04-12T05:09:03.934 INFO:tasks.workunit.client.0.smithi113.stdout:9/615: dwrite d1/d4/f77 [0,4194304] 0 2024-04-12T05:09:03.934 INFO:tasks.workunit.client.1.smithi192.stdout:9/425: mkdir d11/d25/d36/d6e/d8f 0 2024-04-12T05:09:03.935 INFO:tasks.workunit.client.0.smithi113.stdout:9/616: readlink d1/d4/d5/d17/l56 0 2024-04-12T05:09:03.935 INFO:tasks.workunit.client.1.smithi192.stdout:6/562: link d3/d21/d24/d79/l9b d3/d5/lb7 0 2024-04-12T05:09:03.935 INFO:tasks.workunit.client.1.smithi192.stdout:9/426: stat d11/d15/d1e 0 2024-04-12T05:09:03.936 INFO:tasks.workunit.client.0.smithi113.stdout:9/617: fsync d1/d4/d5/d22/f9c 0 2024-04-12T05:09:03.937 INFO:tasks.workunit.client.1.smithi192.stdout:7/453: symlink d3/d1d/l9b 0 2024-04-12T05:09:03.937 INFO:tasks.workunit.client.0.smithi113.stdout:9/618: chown d1/df/d4b 54 1 2024-04-12T05:09:03.940 INFO:tasks.workunit.client.1.smithi192.stdout:3/523: dread d5/d13/d3a/d8b/fae [0,4194304] 0 2024-04-12T05:09:03.940 INFO:tasks.workunit.client.1.smithi192.stdout:8/436: write d6/d1f/f2a [220273,9950] 0 2024-04-12T05:09:03.942 INFO:tasks.workunit.client.1.smithi192.stdout:0/617: dwrite d0/d7b/d9b/d4b/f83 [0,4194304] 0 2024-04-12T05:09:03.943 INFO:tasks.workunit.client.1.smithi192.stdout:7/454: write d3/d1d/d4a/d5e/f9a [129354,28689] 0 2024-04-12T05:09:03.945 INFO:tasks.workunit.client.0.smithi113.stdout:9/619: dwrite d1/d4/d38/f62 [0,4194304] 0 2024-04-12T05:09:03.946 INFO:tasks.workunit.client.1.smithi192.stdout:3/524: dread - d5/d13/d1a/d5f/fa1 zero size 2024-04-12T05:09:03.947 INFO:tasks.workunit.client.1.smithi192.stdout:9/427: rmdir d11/d25/d36/d6e 39 2024-04-12T05:09:03.948 INFO:tasks.workunit.client.1.smithi192.stdout:8/437: dread - d6/d10/d46/f77 zero size 2024-04-12T05:09:03.950 INFO:tasks.workunit.client.0.smithi113.stdout:9/620: mkdir d1/d27/d7a/dd2 0 2024-04-12T05:09:03.950 INFO:tasks.workunit.client.1.smithi192.stdout:0/618: read d0/d6/d15/d1b/d46/f49 [2774499,46910] 0 2024-04-12T05:09:03.954 INFO:tasks.workunit.client.1.smithi192.stdout:6/563: dwrite d3/d8/f92 [0,4194304] 0 2024-04-12T05:09:03.954 INFO:tasks.workunit.client.0.smithi113.stdout:7/584: dwrite d0/dc/d6/f16 [0,4194304] 0 2024-04-12T05:09:03.956 INFO:tasks.workunit.client.1.smithi192.stdout:8/438: rmdir d6/d10/d46/d5e/d5f 39 2024-04-12T05:09:03.956 INFO:tasks.workunit.client.1.smithi192.stdout:7/455: mknod d3/d7/d14/d1f/c9c 0 2024-04-12T05:09:03.958 INFO:tasks.workunit.client.0.smithi113.stdout:9/621: creat d1/df/d29/fd3 x:0 0 0 2024-04-12T05:09:03.961 INFO:tasks.workunit.client.1.smithi192.stdout:0/619: rename d0/d4e to d0/d7b/d9b/d4b/dbc/dce 0 2024-04-12T05:09:03.963 INFO:tasks.workunit.client.0.smithi113.stdout:7/585: mknod d0/dd/d11/cc8 0 2024-04-12T05:09:03.963 INFO:tasks.workunit.client.1.smithi192.stdout:9/428: dread d11/d25/d36/d6e/d33/f4c [4194304,4194304] 0 2024-04-12T05:09:03.964 INFO:tasks.workunit.client.0.smithi113.stdout:9/622: truncate d1/d27/f95 560818 0 2024-04-12T05:09:03.966 INFO:tasks.workunit.client.0.smithi113.stdout:9/623: chown d1/d4/d6d 0 1 2024-04-12T05:09:03.967 INFO:tasks.workunit.client.0.smithi113.stdout:7/586: unlink d0/dd/l39 0 2024-04-12T05:09:03.967 INFO:tasks.workunit.client.1.smithi192.stdout:6/564: creat d3/d23/d67/fb8 x:0 0 0 2024-04-12T05:09:03.967 INFO:tasks.workunit.client.0.smithi113.stdout:7/587: stat d0/dd/d11/l75 0 2024-04-12T05:09:03.969 INFO:tasks.workunit.client.0.smithi113.stdout:9/624: creat d1/d4/d6d/fd4 x:0 0 0 2024-04-12T05:09:03.970 INFO:tasks.workunit.client.1.smithi192.stdout:6/565: chown d3/d8/f1c 609568510 1 2024-04-12T05:09:03.971 INFO:tasks.workunit.client.0.smithi113.stdout:7/588: mknod d0/dc/cc9 0 2024-04-12T05:09:03.971 INFO:tasks.workunit.client.0.smithi113.stdout:2/457: sync 2024-04-12T05:09:03.971 INFO:tasks.workunit.client.0.smithi113.stdout:6/549: sync 2024-04-12T05:09:03.971 INFO:tasks.workunit.client.1.smithi192.stdout:9/429: mknod d11/d25/d36/d6e/d71/c90 0 2024-04-12T05:09:03.973 INFO:tasks.workunit.client.1.smithi192.stdout:9/430: readlink d11/d15/d21/d2c/d1f/l3a 0 2024-04-12T05:09:03.973 INFO:tasks.workunit.client.0.smithi113.stdout:2/458: chown d8/d9/de/f22 55928 1 2024-04-12T05:09:03.973 INFO:tasks.workunit.client.0.smithi113.stdout:6/550: dread - d8/de/d13/d11/d36/d3d/d4c/d31/fc0 zero size 2024-04-12T05:09:03.973 INFO:tasks.workunit.client.0.smithi113.stdout:4/544: sync 2024-04-12T05:09:03.973 INFO:tasks.workunit.client.0.smithi113.stdout:8/617: sync 2024-04-12T05:09:03.973 INFO:tasks.workunit.client.0.smithi113.stdout:3/543: sync 2024-04-12T05:09:03.974 INFO:tasks.workunit.client.1.smithi192.stdout:0/620: rename d0/d7b/d9b/d4b/dbc/dce/d53/l5b to d0/d7b/d9b/d4b/dbc/dce/d53/lcf 0 2024-04-12T05:09:03.974 INFO:tasks.workunit.client.0.smithi113.stdout:8/618: fdatasync d9/f9d 0 2024-04-12T05:09:03.976 INFO:tasks.workunit.client.1.smithi192.stdout:6/566: dread d3/d5/d9c/fa7 [0,4194304] 0 2024-04-12T05:09:03.976 INFO:tasks.workunit.client.1.smithi192.stdout:8/439: rename d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/l7a to d6/d10/d46/d5e/d5f/d37/d7b/l89 0 2024-04-12T05:09:03.976 INFO:tasks.workunit.client.1.smithi192.stdout:9/431: stat d11/d15/d21/d2c/d1f/c3f 0 2024-04-12T05:09:03.977 INFO:tasks.workunit.client.0.smithi113.stdout:3/544: chown d0/d13/f1f 13733835 1 2024-04-12T05:09:03.977 INFO:tasks.workunit.client.0.smithi113.stdout:4/545: truncate da/d3c/d83/fa6 530847 0 2024-04-12T05:09:03.979 INFO:tasks.workunit.client.1.smithi192.stdout:9/432: dread - d11/d15/d21/f6f zero size 2024-04-12T05:09:03.979 INFO:tasks.workunit.client.0.smithi113.stdout:3/545: chown d0/l66 258865 1 2024-04-12T05:09:03.979 INFO:tasks.workunit.client.0.smithi113.stdout:6/551: truncate d8/d45/f6e 617328 0 2024-04-12T05:09:03.980 INFO:tasks.workunit.client.0.smithi113.stdout:8/619: truncate d9/d16/d65/d8c/fb5 779244 0 2024-04-12T05:09:03.981 INFO:tasks.workunit.client.1.smithi192.stdout:7/456: dread d3/d7/d15/f50 [0,4194304] 0 2024-04-12T05:09:03.981 INFO:tasks.workunit.client.1.smithi192.stdout:0/621: rename d0/d7b/d9b/d4b/d72/f90 to d0/fd0 0 2024-04-12T05:09:03.984 INFO:tasks.workunit.client.0.smithi113.stdout:7/589: dwrite d0/dc/da9/d5c/d85/fa4 [0,4194304] 0 2024-04-12T05:09:03.986 INFO:tasks.workunit.client.0.smithi113.stdout:8/620: write d9/d1b/d8e/d93/d32/f3f [64896,45315] 0 2024-04-12T05:09:03.994 INFO:tasks.workunit.client.0.smithi113.stdout:6/552: mkdir d8/de/d13/d2b/db3/dc5 0 2024-04-12T05:09:03.995 INFO:tasks.workunit.client.1.smithi192.stdout:9/433: rmdir d11/d15/d21 39 2024-04-12T05:09:03.995 INFO:tasks.workunit.client.1.smithi192.stdout:8/440: getdents d6/d1f 0 2024-04-12T05:09:03.995 INFO:tasks.workunit.client.1.smithi192.stdout:6/567: dwrite d3/d8/f17 [0,4194304] 0 2024-04-12T05:09:03.995 INFO:tasks.workunit.client.1.smithi192.stdout:7/457: rename d3/d7/d14/l23 to d3/d1d/d4a/d5e/l9d 0 2024-04-12T05:09:03.996 INFO:tasks.workunit.client.1.smithi192.stdout:9/434: creat d11/d25/f91 x:0 0 0 2024-04-12T05:09:04.000 INFO:tasks.workunit.client.1.smithi192.stdout:7/458: unlink d3/d7/d15/d1a/f39 0 2024-04-12T05:09:04.004 INFO:tasks.workunit.client.0.smithi113.stdout:3/546: dwrite d0/d13/f6a [0,4194304] 0 2024-04-12T05:09:04.004 INFO:tasks.workunit.client.1.smithi192.stdout:0/622: dread d0/d7b/d7c/dcc/f93 [0,4194304] 0 2024-04-12T05:09:04.005 INFO:tasks.workunit.client.0.smithi113.stdout:7/590: dwrite d0/dc/d6/fc0 [0,4194304] 0 2024-04-12T05:09:04.008 INFO:tasks.workunit.client.0.smithi113.stdout:4/546: dwrite da/dd/d1b/f28 [4194304,4194304] 0 2024-04-12T05:09:04.011 INFO:tasks.workunit.client.0.smithi113.stdout:2/459: dwrite d8/fd [0,4194304] 0 2024-04-12T05:09:04.014 INFO:tasks.workunit.client.1.smithi192.stdout:8/441: link d6/d10/d46/d5e/d5f/d38/l84 d6/d10/d46/d5e/d5f/d37/d4f/d51/l8a 0 2024-04-12T05:09:04.015 INFO:tasks.workunit.client.1.smithi192.stdout:0/623: creat d0/d7b/d9b/d4b/d72/fd1 x:0 0 0 2024-04-12T05:09:04.021 INFO:tasks.workunit.client.1.smithi192.stdout:9/435: getdents d11/d25/d36/d6e/d71 0 2024-04-12T05:09:04.021 INFO:tasks.workunit.client.1.smithi192.stdout:7/459: readlink d3/d7/d14/d1f/d76/l94 0 2024-04-12T05:09:04.025 INFO:tasks.workunit.client.1.smithi192.stdout:5/494: sync 2024-04-12T05:09:04.025 INFO:tasks.workunit.client.1.smithi192.stdout:4/616: sync 2024-04-12T05:09:04.025 INFO:tasks.workunit.client.1.smithi192.stdout:2/527: sync 2024-04-12T05:09:04.032 INFO:tasks.workunit.client.1.smithi192.stdout:9/436: rmdir d11/d15/d21 39 2024-04-12T05:09:04.032 INFO:tasks.workunit.client.1.smithi192.stdout:2/528: stat d1/l7 0 2024-04-12T05:09:04.032 INFO:tasks.workunit.client.1.smithi192.stdout:8/442: symlink d6/d10/d46/d7c/l8b 0 2024-04-12T05:09:04.033 INFO:tasks.workunit.client.1.smithi192.stdout:2/529: write d1/f4b [11082,74346] 0 2024-04-12T05:09:04.041 INFO:tasks.workunit.client.1.smithi192.stdout:9/437: mkdir d11/d15/d21/d2c/d1f/d6a/d92 0 2024-04-12T05:09:04.042 INFO:tasks.workunit.client.1.smithi192.stdout:0/624: dwrite d0/d6/d15/d1b/d46/f49 [0,4194304] 0 2024-04-12T05:09:04.045 INFO:tasks.workunit.client.1.smithi192.stdout:5/495: dwrite d0/d15/d24/f46 [0,4194304] 0 2024-04-12T05:09:04.049 INFO:tasks.workunit.client.1.smithi192.stdout:6/568: dread d3/d8/d10/f71 [0,4194304] 0 2024-04-12T05:09:04.051 INFO:tasks.workunit.client.1.smithi192.stdout:9/438: read - d11/d15/d21/f65 zero size 2024-04-12T05:09:04.052 INFO:tasks.workunit.client.1.smithi192.stdout:6/569: dread - d3/d23/d67/fb8 zero size 2024-04-12T05:09:04.053 INFO:tasks.workunit.client.1.smithi192.stdout:8/443: getdents d6/d10/d46/d5e/d6e 0 2024-04-12T05:09:04.054 INFO:tasks.workunit.client.1.smithi192.stdout:0/625: mkdir d0/d6/d67/dd2 0 2024-04-12T05:09:04.056 INFO:tasks.workunit.client.0.smithi113.stdout:6/553: rename d8/cb5 to d8/de/d13/d2b/db3/dc5/cc6 0 2024-04-12T05:09:04.056 INFO:tasks.workunit.client.1.smithi192.stdout:5/496: rmdir d0/d15/da1 39 2024-04-12T05:09:04.056 INFO:tasks.workunit.client.1.smithi192.stdout:9/439: chown d11/d25/d36 10 1 2024-04-12T05:09:04.057 INFO:tasks.workunit.client.0.smithi113.stdout:7/591: creat d0/dc/d6/d9b/d80/fca x:0 0 0 2024-04-12T05:09:04.058 INFO:tasks.workunit.client.0.smithi113.stdout:8/621: write d9/d1b/d8e/d93/d32/d48/d8d/d4a/fbd [471199,103352] 0 2024-04-12T05:09:04.061 INFO:tasks.workunit.client.0.smithi113.stdout:7/592: write d0/dc/d6/d1a/fb8 [156965,86555] 0 2024-04-12T05:09:04.061 INFO:tasks.workunit.client.1.smithi192.stdout:5/497: read - d0/d15/d3e/f9d zero size 2024-04-12T05:09:04.061 INFO:tasks.workunit.client.1.smithi192.stdout:6/570: creat d3/d23/fb9 x:0 0 0 2024-04-12T05:09:04.062 INFO:tasks.workunit.client.0.smithi113.stdout:7/593: chown d0/dc/da9/d5c/d85/f9c 2916638 1 2024-04-12T05:09:04.062 INFO:tasks.workunit.client.1.smithi192.stdout:8/444: creat d6/d10/d46/d7c/f8c x:0 0 0 2024-04-12T05:09:04.063 INFO:tasks.workunit.client.0.smithi113.stdout:7/594: write d0/dc/da9/d5c/d85/fb1 [40690,103821] 0 2024-04-12T05:09:04.063 INFO:tasks.workunit.client.1.smithi192.stdout:4/617: dwrite d1/d54/d55/d64/f8f [0,4194304] 0 2024-04-12T05:09:04.065 INFO:tasks.workunit.client.1.smithi192.stdout:8/445: chown d6/c40 903150751 1 2024-04-12T05:09:04.067 INFO:tasks.workunit.client.1.smithi192.stdout:6/571: symlink d3/d8/lba 0 2024-04-12T05:09:04.067 INFO:tasks.workunit.client.1.smithi192.stdout:9/440: symlink d11/d25/d36/d6e/d8f/l93 0 2024-04-12T05:09:04.068 INFO:tasks.workunit.client.0.smithi113.stdout:6/554: symlink d8/de/d13/d11/d36/d3d/d4c/d26/d64/lc7 0 2024-04-12T05:09:04.069 INFO:tasks.workunit.client.0.smithi113.stdout:6/555: chown d8/de/d17/f7c 314902 1 2024-04-12T05:09:04.071 INFO:tasks.workunit.client.1.smithi192.stdout:6/572: chown d3/d26/f46 6176376 1 2024-04-12T05:09:04.072 INFO:tasks.workunit.client.1.smithi192.stdout:7/460: dwrite d3/d11/f16 [0,4194304] 0 2024-04-12T05:09:04.074 INFO:tasks.workunit.client.0.smithi113.stdout:2/460: rmdir d8/d9/de/d15/d24 39 2024-04-12T05:09:04.075 INFO:tasks.workunit.client.1.smithi192.stdout:5/498: symlink d0/d15/d64/d7c/da7/laa 0 2024-04-12T05:09:04.080 INFO:tasks.workunit.client.1.smithi192.stdout:8/446: dwrite d6/d10/d46/d5e/d5f/d37/f48 [0,4194304] 0 2024-04-12T05:09:04.081 INFO:tasks.workunit.client.1.smithi192.stdout:8/447: truncate d6/d10/f78 881102 0 2024-04-12T05:09:04.090 INFO:tasks.workunit.client.1.smithi192.stdout:2/530: dread d1/f4 [0,4194304] 0 2024-04-12T05:09:04.095 INFO:tasks.workunit.client.0.smithi113.stdout:2/461: write d8/d9/d1a/f26 [199850,129202] 0 2024-04-12T05:09:04.095 INFO:tasks.workunit.client.0.smithi113.stdout:6/556: symlink d8/de/d13/d11/d36/d2c/lc8 0 2024-04-12T05:09:04.095 INFO:tasks.workunit.client.0.smithi113.stdout:5/573: sync 2024-04-12T05:09:04.095 INFO:tasks.workunit.client.0.smithi113.stdout:1/659: sync 2024-04-12T05:09:04.095 INFO:tasks.workunit.client.0.smithi113.stdout:0/675: sync 2024-04-12T05:09:04.096 INFO:tasks.workunit.client.0.smithi113.stdout:3/547: mknod d0/d13/d15/d28/d2a/d8b/d91/dad/cbc 0 2024-04-12T05:09:04.097 INFO:tasks.workunit.client.0.smithi113.stdout:4/547: getdents da/d24/d7e/d79 0 2024-04-12T05:09:04.099 INFO:tasks.workunit.client.1.smithi192.stdout:4/618: rename d1/l13 to d1/d54/d55/d64/lcc 0 2024-04-12T05:09:04.099 INFO:tasks.workunit.client.0.smithi113.stdout:1/660: write d10/d2e/d31/d33/f64 [527570,115829] 0 2024-04-12T05:09:04.100 INFO:tasks.workunit.client.1.smithi192.stdout:4/619: chown d1/d54/d82/d71/f7b 3725933 1 2024-04-12T05:09:04.100 INFO:tasks.workunit.client.1.smithi192.stdout:9/441: mkdir d11/d15/d21/d2c/d1f/d94 0 2024-04-12T05:09:04.101 INFO:tasks.workunit.client.1.smithi192.stdout:2/531: dwrite d1/d8/f29 [0,4194304] 0 2024-04-12T05:09:04.105 INFO:tasks.workunit.client.0.smithi113.stdout:2/462: rmdir d8/d9/de/d15 39 2024-04-12T05:09:04.107 INFO:tasks.workunit.client.1.smithi192.stdout:5/499: mknod d0/d15/d64/d7c/da7/cab 0 2024-04-12T05:09:04.108 INFO:tasks.workunit.client.1.smithi192.stdout:2/532: chown d1/d8/d3a/de/d7c/d98 519072304 1 2024-04-12T05:09:04.110 INFO:tasks.workunit.client.1.smithi192.stdout:9/442: rmdir d11/d15/d21/d2c 39 2024-04-12T05:09:04.112 INFO:tasks.workunit.client.1.smithi192.stdout:5/500: creat d0/d15/d32/d69/d7d/fac x:0 0 0 2024-04-12T05:09:04.114 INFO:tasks.workunit.client.1.smithi192.stdout:5/501: dread - d0/d15/d64/d7c/f98 zero size 2024-04-12T05:09:04.118 INFO:tasks.workunit.client.1.smithi192.stdout:5/502: readlink d0/d21/l9f 0 2024-04-12T05:09:04.119 INFO:tasks.workunit.client.1.smithi192.stdout:8/448: dwrite d6/d21/f33 [0,4194304] 0 2024-04-12T05:09:04.120 INFO:tasks.workunit.client.1.smithi192.stdout:9/443: creat d11/d25/d36/d6e/d71/f95 x:0 0 0 2024-04-12T05:09:04.122 INFO:tasks.workunit.client.1.smithi192.stdout:8/449: stat d6/d21/l34 0 2024-04-12T05:09:04.123 INFO:tasks.workunit.client.1.smithi192.stdout:5/503: creat d0/d15/d64/fad x:0 0 0 2024-04-12T05:09:04.124 INFO:tasks.workunit.client.1.smithi192.stdout:4/620: getdents d1/d54/d55/d8b 0 2024-04-12T05:09:04.124 INFO:tasks.workunit.client.1.smithi192.stdout:6/573: truncate d3/d8/fa3 1011242 0 2024-04-12T05:09:04.126 INFO:tasks.workunit.client.1.smithi192.stdout:9/444: read - d11/d15/d21/d2c/d1f/f7f zero size 2024-04-12T05:09:04.127 INFO:tasks.workunit.client.1.smithi192.stdout:4/621: write d1/d54/f99 [382816,36742] 0 2024-04-12T05:09:04.129 INFO:tasks.workunit.client.1.smithi192.stdout:2/533: dwrite d1/f4 [0,4194304] 0 2024-04-12T05:09:04.132 INFO:tasks.workunit.client.1.smithi192.stdout:8/450: truncate d6/d10/d46/d5e/d5f/d38/f41 840000 0 2024-04-12T05:09:04.135 INFO:tasks.workunit.client.1.smithi192.stdout:9/445: link d11/d25/d36/d6e/d33/c53 d11/d25/d60/c96 0 2024-04-12T05:09:04.136 INFO:tasks.workunit.client.1.smithi192.stdout:8/451: write d6/d1f/f2a [193103,125387] 0 2024-04-12T05:09:04.136 INFO:tasks.workunit.client.1.smithi192.stdout:5/504: write d0/d15/d24/d47/f54 [118358,46743] 0 2024-04-12T05:09:04.137 INFO:tasks.workunit.client.1.smithi192.stdout:6/574: rename d3/l88 to d3/d8/d10/d4c/lbb 0 2024-04-12T05:09:04.138 INFO:tasks.workunit.client.1.smithi192.stdout:9/446: dread - d11/d15/d21/d2c/d1f/d6a/f7c zero size 2024-04-12T05:09:04.139 INFO:tasks.workunit.client.1.smithi192.stdout:8/452: write d6/d10/d46/d5e/d5f/d37/f4a [4845498,90510] 0 2024-04-12T05:09:04.141 INFO:tasks.workunit.client.1.smithi192.stdout:2/534: getdents d1/d2d/d73 0 2024-04-12T05:09:04.141 INFO:tasks.workunit.client.1.smithi192.stdout:9/447: dread - d11/d25/d36/d6e/f30 zero size 2024-04-12T05:09:04.144 INFO:tasks.workunit.client.1.smithi192.stdout:6/575: truncate d3/f43 503006 0 2024-04-12T05:09:04.145 INFO:tasks.workunit.client.1.smithi192.stdout:5/505: creat d0/d15/d3e/fae x:0 0 0 2024-04-12T05:09:04.145 INFO:tasks.workunit.client.1.smithi192.stdout:2/535: rmdir d1/d8/d3a/de 39 2024-04-12T05:09:04.148 INFO:tasks.workunit.client.1.smithi192.stdout:5/506: write d0/f11 [2371131,124131] 0 2024-04-12T05:09:04.149 INFO:tasks.workunit.client.1.smithi192.stdout:2/536: dread - d1/d8/d17/d19/d60/f93 zero size 2024-04-12T05:09:04.150 INFO:tasks.workunit.client.1.smithi192.stdout:8/453: write d6/da/f49 [31274,43245] 0 2024-04-12T05:09:04.152 INFO:tasks.workunit.client.1.smithi192.stdout:2/537: fsync d1/d8/f35 0 2024-04-12T05:09:04.153 INFO:tasks.workunit.client.1.smithi192.stdout:5/507: chown d0/d15/da1 696332854 1 2024-04-12T05:09:04.154 INFO:tasks.workunit.client.1.smithi192.stdout:9/448: dwrite d11/d15/d21/d2c/d1f/f86 [0,4194304] 0 2024-04-12T05:09:04.155 INFO:tasks.workunit.client.1.smithi192.stdout:6/576: write d3/d8/d10/d4c/f62 [965433,73649] 0 2024-04-12T05:09:04.156 INFO:tasks.workunit.client.1.smithi192.stdout:4/622: dread d1/d54/d82/fb3 [0,4194304] 0 2024-04-12T05:09:04.162 INFO:tasks.workunit.client.1.smithi192.stdout:2/538: readlink d1/d8/d17/d19/d71/d7f/l91 0 2024-04-12T05:09:04.162 INFO:tasks.workunit.client.1.smithi192.stdout:9/449: chown d11/d25/d36/d6e/d33/l64 24 1 2024-04-12T05:09:04.162 INFO:tasks.workunit.client.1.smithi192.stdout:5/508: rename d0/d15/d24/d47/l82 to d0/d21/laf 0 2024-04-12T05:09:04.163 INFO:tasks.workunit.client.1.smithi192.stdout:6/577: dread d3/d5/f72 [0,4194304] 0 2024-04-12T05:09:04.165 INFO:tasks.workunit.client.1.smithi192.stdout:9/450: creat d11/d25/d60/f97 x:0 0 0 2024-04-12T05:09:04.168 INFO:tasks.workunit.client.1.smithi192.stdout:8/454: link d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/f75 d6/d1f/d5d/f8d 0 2024-04-12T05:09:04.172 INFO:tasks.workunit.client.1.smithi192.stdout:9/451: rename d11/d15/d21/d2c/d1f/f86 to d11/d15/d21/d2c/d1f/d50/f98 0 2024-04-12T05:09:04.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:02.671736+0000 mgr.smithi192.ocppyo (mgr.24357) 27 : cluster [DBG] pgmap v21: 65 pgs: 65 active+clean; 2.1 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 64 MiB/s rd, 311 MiB/s wr, 513 op/s 2024-04-12T05:09:04.174 INFO:tasks.workunit.client.1.smithi192.stdout:2/539: write d1/d8/f10 [2999224,50679] 0 2024-04-12T05:09:04.175 INFO:tasks.workunit.client.1.smithi192.stdout:6/578: fsync d3/f43 0 2024-04-12T05:09:04.175 INFO:tasks.workunit.client.1.smithi192.stdout:2/540: chown d1/d8/f22 1244 1 2024-04-12T05:09:04.177 INFO:tasks.workunit.client.1.smithi192.stdout:6/579: write d3/d23/fb9 [831214,111516] 0 2024-04-12T05:09:04.179 INFO:tasks.workunit.client.1.smithi192.stdout:5/509: dread d0/d21/f2f [0,4194304] 0 2024-04-12T05:09:04.204 INFO:tasks.workunit.client.0.smithi113.stdout:3/548: creat d0/d13/d15/d28/d2a/d34/fbd x:0 0 0 2024-04-12T05:09:04.206 INFO:tasks.workunit.client.0.smithi113.stdout:7/595: creat d0/dc/d6/d9b/fcb x:0 0 0 2024-04-12T05:09:04.206 INFO:tasks.workunit.client.0.smithi113.stdout:1/661: creat d10/d14/d95/dcf/fd0 x:0 0 0 2024-04-12T05:09:04.206 INFO:tasks.workunit.client.0.smithi113.stdout:5/574: mkdir d0/d10/d27/d1d/d3f/d73/db9 0 2024-04-12T05:09:04.206 INFO:tasks.workunit.client.0.smithi113.stdout:2/463: mkdir d8/d9/d27/d88 0 2024-04-12T05:09:04.207 INFO:tasks.workunit.client.0.smithi113.stdout:0/676: creat d1/d6/d16/d2a/d50/db6/d36/fd1 x:0 0 0 2024-04-12T05:09:04.207 INFO:tasks.workunit.client.0.smithi113.stdout:1/662: stat d10/d14/d67/l8f 0 2024-04-12T05:09:04.207 INFO:tasks.workunit.client.0.smithi113.stdout:4/548: link da/d3c/d83/f8a da/d3c/da5/dad/fb0 0 2024-04-12T05:09:04.208 INFO:tasks.workunit.client.0.smithi113.stdout:0/677: fsync d1/d6/f5d 0 2024-04-12T05:09:04.208 INFO:tasks.workunit.client.0.smithi113.stdout:3/549: stat d0/f18 0 2024-04-12T05:09:04.209 INFO:tasks.workunit.client.0.smithi113.stdout:7/596: mknod d0/dc/da9/ccc 0 2024-04-12T05:09:04.209 INFO:tasks.workunit.client.0.smithi113.stdout:1/663: chown d10/d14/c18 70 1 2024-04-12T05:09:04.210 INFO:tasks.workunit.client.0.smithi113.stdout:4/549: creat da/dd/d1b/fb1 x:0 0 0 2024-04-12T05:09:04.211 INFO:tasks.workunit.client.0.smithi113.stdout:7/597: mkdir d0/dc/d6/d9b/dcd 0 2024-04-12T05:09:04.213 INFO:tasks.workunit.client.0.smithi113.stdout:1/664: creat d10/d21/fd1 x:0 0 0 2024-04-12T05:09:04.214 INFO:tasks.workunit.client.0.smithi113.stdout:4/550: chown da/d4a/f9d 1965387355 1 2024-04-12T05:09:04.216 INFO:tasks.workunit.client.0.smithi113.stdout:1/665: truncate d10/d14/fbc 956415 0 2024-04-12T05:09:04.216 INFO:tasks.workunit.client.0.smithi113.stdout:0/678: link d1/c20 d1/d6/d16/d2a/d50/db6/cd2 0 2024-04-12T05:09:04.217 INFO:tasks.workunit.client.0.smithi113.stdout:3/550: dread d0/d13/d15/d28/f45 [0,4194304] 0 2024-04-12T05:09:04.217 INFO:tasks.workunit.client.0.smithi113.stdout:7/598: symlink d0/dc/d6/d9b/d9f/db3/lce 0 2024-04-12T05:09:04.218 INFO:tasks.workunit.client.0.smithi113.stdout:1/666: chown d10/d14/d67/d99/fb8 27 1 2024-04-12T05:09:04.218 INFO:tasks.workunit.client.0.smithi113.stdout:4/551: creat da/d24/d60/fb2 x:0 0 0 2024-04-12T05:09:04.220 INFO:tasks.workunit.client.0.smithi113.stdout:0/679: unlink d1/d6/d34/dc3/l7c 0 2024-04-12T05:09:04.220 INFO:tasks.workunit.client.0.smithi113.stdout:3/551: mknod d0/d7e/cbe 0 2024-04-12T05:09:04.222 INFO:tasks.workunit.client.0.smithi113.stdout:1/667: creat d10/d41/d5b/fd2 x:0 0 0 2024-04-12T05:09:04.223 INFO:tasks.workunit.client.0.smithi113.stdout:0/680: creat d1/d43/d59/db2/fd3 x:0 0 0 2024-04-12T05:09:04.224 INFO:tasks.workunit.client.0.smithi113.stdout:7/599: creat d0/dc/da9/d5c/d9a/db6/fcf x:0 0 0 2024-04-12T05:09:04.233 INFO:tasks.workunit.client.0.smithi113.stdout:2/464: dwrite d8/d9/de/d15/d24/f4b [0,4194304] 0 2024-04-12T05:09:04.233 INFO:tasks.workunit.client.0.smithi113.stdout:3/552: dread d0/f18 [0,4194304] 0 2024-04-12T05:09:04.235 INFO:tasks.workunit.client.0.smithi113.stdout:7/600: dread d0/dc/d6/d9b/f62 [0,4194304] 0 2024-04-12T05:09:04.237 INFO:tasks.workunit.client.0.smithi113.stdout:3/553: mkdir d0/d13/dbf 0 2024-04-12T05:09:04.240 INFO:tasks.workunit.client.0.smithi113.stdout:4/552: dread da/d3c/d83/f8a [0,4194304] 0 2024-04-12T05:09:04.241 INFO:tasks.workunit.client.0.smithi113.stdout:7/601: creat d0/dc/d6/fd0 x:0 0 0 2024-04-12T05:09:04.248 INFO:tasks.workunit.client.0.smithi113.stdout:4/553: read da/d4f/f72 [4233536,109233] 0 2024-04-12T05:09:04.251 INFO:tasks.workunit.client.0.smithi113.stdout:1/668: dwrite d10/d2e/d31/f53 [0,4194304] 0 2024-04-12T05:09:04.251 INFO:tasks.workunit.client.0.smithi113.stdout:3/554: stat d0/d13/d15/d28/d2a/d34/d5f/c93 0 2024-04-12T05:09:04.253 INFO:tasks.workunit.client.0.smithi113.stdout:0/681: dwrite d1/d6/f30 [0,4194304] 0 2024-04-12T05:09:04.254 INFO:tasks.workunit.client.0.smithi113.stdout:2/465: dwrite d8/d16/f1d [0,4194304] 0 2024-04-12T05:09:04.257 INFO:tasks.workunit.client.1.smithi192.stdout:8/455: write d6/d10/d46/f85 [981838,5223] 0 2024-04-12T05:09:04.257 INFO:tasks.workunit.client.1.smithi192.stdout:2/541: truncate d1/d8/d17/f5f 136580 0 2024-04-12T05:09:04.258 INFO:tasks.workunit.client.1.smithi192.stdout:5/510: write d0/fa [4230207,60529] 0 2024-04-12T05:09:04.259 INFO:tasks.workunit.client.1.smithi192.stdout:6/580: write d3/dc/d39/f70 [4826787,25711] 0 2024-04-12T05:09:04.262 INFO:tasks.workunit.client.1.smithi192.stdout:4/623: dwrite d1/d54/d55/f30 [0,4194304] 0 2024-04-12T05:09:04.263 INFO:tasks.workunit.client.0.smithi113.stdout:4/554: symlink da/d3c/d43/lb3 0 2024-04-12T05:09:04.263 INFO:tasks.workunit.client.0.smithi113.stdout:3/555: fdatasync d0/d13/d47/f78 0 2024-04-12T05:09:04.263 INFO:tasks.workunit.client.0.smithi113.stdout:0/682: read - d1/d6/d16/d2a/d50/f85 zero size 2024-04-12T05:09:04.263 INFO:tasks.workunit.client.0.smithi113.stdout:0/683: read d1/fc9 [4329436,73787] 0 2024-04-12T05:09:04.264 INFO:tasks.workunit.client.0.smithi113.stdout:2/466: creat d8/d9/de/d12/d44/f89 x:0 0 0 2024-04-12T05:09:04.264 INFO:tasks.workunit.client.1.smithi192.stdout:2/542: dread - d1/d2d/d73/f8d zero size 2024-04-12T05:09:04.265 INFO:tasks.workunit.client.0.smithi113.stdout:7/602: dwrite d0/dc/d6/d1a/fb8 [0,4194304] 0 2024-04-12T05:09:04.265 INFO:tasks.workunit.client.0.smithi113.stdout:1/669: rename d10/d2e/d31/d33 to d10/d41/d89/d96/dd3 0 2024-04-12T05:09:04.271 INFO:tasks.workunit.client.1.smithi192.stdout:8/456: mknod d6/d10/d46/c8e 0 2024-04-12T05:09:04.272 INFO:tasks.workunit.client.0.smithi113.stdout:3/556: dwrite d0/d13/d15/d28/f74 [0,4194304] 0 2024-04-12T05:09:04.274 INFO:tasks.workunit.client.1.smithi192.stdout:2/543: fdatasync d1/d8/d3a/f95 0 2024-04-12T05:09:04.276 INFO:tasks.workunit.client.1.smithi192.stdout:8/457: truncate d6/d1f/f2a 1126618 0 2024-04-12T05:09:04.276 INFO:tasks.workunit.client.1.smithi192.stdout:6/581: mknod d3/dc/cbc 0 2024-04-12T05:09:04.278 INFO:tasks.workunit.client.1.smithi192.stdout:6/582: dread - d3/d8/d10/d4c/fb0 zero size 2024-04-12T05:09:04.278 INFO:tasks.workunit.client.1.smithi192.stdout:4/624: creat d1/d49/fcd x:0 0 0 2024-04-12T05:09:04.281 INFO:tasks.workunit.client.1.smithi192.stdout:4/625: fdatasync d1/d1d/d53/d1c/fcb 0 2024-04-12T05:09:04.283 INFO:tasks.workunit.client.0.smithi113.stdout:0/684: write d1/f2 [4632127,62901] 0 2024-04-12T05:09:04.283 INFO:tasks.workunit.client.1.smithi192.stdout:8/458: creat d6/d10/d46/d5e/d5f/d38/f8f x:0 0 0 2024-04-12T05:09:04.283 INFO:tasks.workunit.client.0.smithi113.stdout:2/467: mkdir d8/d9/de/d12/d3c/d8a 0 2024-04-12T05:09:04.285 INFO:tasks.workunit.client.0.smithi113.stdout:1/670: truncate d10/d14/f1f 1443636 0 2024-04-12T05:09:04.285 INFO:tasks.workunit.client.1.smithi192.stdout:6/583: rmdir d3/d23/d67 39 2024-04-12T05:09:04.285 INFO:tasks.workunit.client.1.smithi192.stdout:8/459: rename d6/d10/d46 to d6/d10/d46/d7c/d90 22 2024-04-12T05:09:04.285 INFO:tasks.workunit.client.1.smithi192.stdout:9/452: dwrite d11/d25/d36/d6e/d33/f4c [4194304,4194304] 0 2024-04-12T05:09:04.286 INFO:tasks.workunit.client.0.smithi113.stdout:7/603: mkdir d0/dc/d6/d9b/d9f/dd1 0 2024-04-12T05:09:04.287 INFO:tasks.workunit.client.1.smithi192.stdout:4/626: creat d1/d1d/d53/d1c/dc1/fce x:0 0 0 2024-04-12T05:09:04.287 INFO:tasks.workunit.client.0.smithi113.stdout:7/604: chown d0/dc/f83 1082211 1 2024-04-12T05:09:04.288 INFO:tasks.workunit.client.1.smithi192.stdout:5/511: getdents d0/d15/d24 0 2024-04-12T05:09:04.289 INFO:tasks.workunit.client.1.smithi192.stdout:6/584: mkdir d3/d5/d9c/dbd 0 2024-04-12T05:09:04.289 INFO:tasks.workunit.client.0.smithi113.stdout:2/468: creat d8/d9/d27/d88/f8b x:0 0 0 2024-04-12T05:09:04.289 INFO:tasks.workunit.client.1.smithi192.stdout:4/627: chown d1/d54/d55/d64/l56 46836953 1 2024-04-12T05:09:04.291 INFO:tasks.workunit.client.0.smithi113.stdout:7/605: read d0/dc/d6/d78/f7c [74969,30359] 0 2024-04-12T05:09:04.293 INFO:tasks.workunit.client.1.smithi192.stdout:8/460: symlink d6/d10/d46/d5e/d5f/d38/l91 0 2024-04-12T05:09:04.293 INFO:tasks.workunit.client.0.smithi113.stdout:0/685: dwrite d1/d6/d16/fb8 [0,4194304] 0 2024-04-12T05:09:04.295 INFO:tasks.workunit.client.1.smithi192.stdout:9/453: dwrite f10 [0,4194304] 0 2024-04-12T05:09:04.295 INFO:tasks.workunit.client.1.smithi192.stdout:5/512: creat d0/d15/d32/d45/fb0 x:0 0 0 2024-04-12T05:09:04.297 INFO:tasks.workunit.client.0.smithi113.stdout:7/606: mkdir d0/dc/d6/d24/dd2 0 2024-04-12T05:09:04.298 INFO:tasks.workunit.client.0.smithi113.stdout:7/607: stat d0/dd/d2d/dbb 0 2024-04-12T05:09:04.298 INFO:tasks.workunit.client.1.smithi192.stdout:8/461: truncate d6/d10/d46/d7c/f8c 772497 0 2024-04-12T05:09:04.299 INFO:tasks.workunit.client.0.smithi113.stdout:0/686: fdatasync d1/d6/d16/f4c 0 2024-04-12T05:09:04.299 INFO:tasks.workunit.client.0.smithi113.stdout:7/608: stat d0/dc/cc9 0 2024-04-12T05:09:04.300 INFO:tasks.workunit.client.0.smithi113.stdout:1/671: link d10/d41/d89/d96/cb6 d10/d2e/d31/cd4 0 2024-04-12T05:09:04.303 INFO:tasks.workunit.client.1.smithi192.stdout:6/585: rename d3/dc/fa9 to d3/d8/d48/fbe 0 2024-04-12T05:09:04.304 INFO:tasks.workunit.client.1.smithi192.stdout:9/454: mknod d11/d25/d36/d3d/d7a/c99 0 2024-04-12T05:09:04.305 INFO:tasks.workunit.client.1.smithi192.stdout:9/455: write d11/d15/d1e/f59 [572682,22280] 0 2024-04-12T05:09:04.307 INFO:tasks.workunit.client.1.smithi192.stdout:9/456: chown d11/d25/d36/d6e/f56 1986766345 1 2024-04-12T05:09:04.309 INFO:tasks.workunit.client.0.smithi113.stdout:7/609: dwrite d0/dd/d11/f77 [0,4194304] 0 2024-04-12T05:09:04.313 INFO:tasks.workunit.client.1.smithi192.stdout:9/457: mknod d11/d15/c9a 0 2024-04-12T05:09:04.314 INFO:tasks.workunit.client.0.smithi113.stdout:7/610: symlink d0/dd/d2d/ld3 0 2024-04-12T05:09:04.315 INFO:tasks.workunit.client.0.smithi113.stdout:7/611: chown d0/d6c/f70 3 1 2024-04-12T05:09:04.320 INFO:tasks.workunit.client.0.smithi113.stdout:0/687: dread d1/d6/d16/f24 [0,4194304] 0 2024-04-12T05:09:04.326 INFO:tasks.workunit.client.0.smithi113.stdout:0/688: mkdir d1/d6/d16/dd4 0 2024-04-12T05:09:04.328 INFO:tasks.workunit.client.0.smithi113.stdout:7/612: dwrite d0/dc/d6/d78/f6a [0,4194304] 0 2024-04-12T05:09:04.328 INFO:tasks.workunit.client.1.smithi192.stdout:4/628: getdents d1/d49 0 2024-04-12T05:09:04.328 INFO:tasks.workunit.client.0.smithi113.stdout:4/555: truncate da/d2b/f3e 1682762 0 2024-04-12T05:09:04.329 INFO:tasks.workunit.client.1.smithi192.stdout:4/629: write d1/f8a [222363,127314] 0 2024-04-12T05:09:04.330 INFO:tasks.workunit.client.0.smithi113.stdout:1/672: dread d10/d41/f60 [4194304,4194304] 0 2024-04-12T05:09:04.331 INFO:tasks.workunit.client.1.smithi192.stdout:5/513: dread d0/d15/d32/d45/f65 [0,4194304] 0 2024-04-12T05:09:04.331 INFO:tasks.workunit.client.1.smithi192.stdout:4/630: rename d1/d1d/d53/db2 to d1/d1d/d53/db2/dcf 22 2024-04-12T05:09:04.332 INFO:tasks.workunit.client.0.smithi113.stdout:4/556: chown da/d3c/d9f/c93 0 1 2024-04-12T05:09:04.332 INFO:tasks.workunit.client.0.smithi113.stdout:7/613: mkdir d0/dd/d95/dd4 0 2024-04-12T05:09:04.333 INFO:tasks.workunit.client.0.smithi113.stdout:0/689: fdatasync d1/d6/d8e/fba 0 2024-04-12T05:09:04.334 INFO:tasks.workunit.client.1.smithi192.stdout:5/514: truncate d0/d15/d24/d29/f85 750528 0 2024-04-12T05:09:04.336 INFO:tasks.workunit.client.0.smithi113.stdout:4/557: chown da/c55 796676 1 2024-04-12T05:09:04.337 INFO:tasks.workunit.client.0.smithi113.stdout:0/690: write d1/d6/d8e/fcb [392238,116411] 0 2024-04-12T05:09:04.337 INFO:tasks.workunit.client.1.smithi192.stdout:2/544: truncate d1/d8/d3a/de/f97 99129 0 2024-04-12T05:09:04.338 INFO:tasks.workunit.client.0.smithi113.stdout:3/557: write d0/d13/f60 [155727,89996] 0 2024-04-12T05:09:04.339 INFO:tasks.workunit.client.0.smithi113.stdout:2/469: dread d8/d9/de/d12/f78 [0,4194304] 0 2024-04-12T05:09:04.339 INFO:tasks.workunit.client.1.smithi192.stdout:5/515: creat d0/d15/d32/d45/fb1 x:0 0 0 2024-04-12T05:09:04.340 INFO:tasks.workunit.client.0.smithi113.stdout:4/558: mkdir da/d24/d60/db4 0 2024-04-12T05:09:04.346 INFO:tasks.workunit.client.1.smithi192.stdout:4/631: creat d1/d54/d55/fd0 x:0 0 0 2024-04-12T05:09:04.348 INFO:tasks.workunit.client.1.smithi192.stdout:1/541: sync 2024-04-12T05:09:04.348 INFO:tasks.workunit.client.1.smithi192.stdout:3/525: sync 2024-04-12T05:09:04.349 INFO:tasks.workunit.client.0.smithi113.stdout:2/470: write d8/d9/d1a/f50 [72002,8923] 0 2024-04-12T05:09:04.351 INFO:tasks.workunit.client.1.smithi192.stdout:4/632: mkdir d1/d1d/d53/d1c/dd1 0 2024-04-12T05:09:04.354 INFO:tasks.workunit.client.1.smithi192.stdout:8/462: write d6/d10/d46/d5e/d5f/d37/d4f/d51/f66 [3349372,108959] 0 2024-04-12T05:09:04.354 INFO:tasks.workunit.client.0.smithi113.stdout:2/471: dread - d8/d9/de/d15/d24/d58/f75 zero size 2024-04-12T05:09:04.356 INFO:tasks.workunit.client.1.smithi192.stdout:9/458: truncate d11/d15/d21/d2c/d1f/d50/f98 665801 0 2024-04-12T05:09:04.357 INFO:tasks.workunit.client.0.smithi113.stdout:7/614: dwrite d0/dc/d6/d1a/f2b [0,4194304] 0 2024-04-12T05:09:04.358 INFO:tasks.workunit.client.1.smithi192.stdout:6/586: dwrite d3/dc/f3b [0,4194304] 0 2024-04-12T05:09:04.358 INFO:tasks.workunit.client.1.smithi192.stdout:3/526: mkdir d5/d13/d3a/d70/d8c/d8d/dbc 0 2024-04-12T05:09:04.359 INFO:tasks.workunit.client.1.smithi192.stdout:8/463: truncate d6/d10/d46/f85 1570109 0 2024-04-12T05:09:04.361 INFO:tasks.workunit.client.0.smithi113.stdout:0/691: write d1/d6/d8e/fba [426808,105036] 0 2024-04-12T05:09:04.370 INFO:tasks.workunit.client.1.smithi192.stdout:9/459: mknod d11/d25/d36/d6e/d71/c9b 0 2024-04-12T05:09:04.371 INFO:tasks.workunit.client.1.smithi192.stdout:5/516: dread d0/f14 [0,4194304] 0 2024-04-12T05:09:04.371 INFO:tasks.workunit.client.0.smithi113.stdout:1/673: dread d10/d41/f65 [0,4194304] 0 2024-04-12T05:09:04.371 INFO:tasks.workunit.client.0.smithi113.stdout:2/472: dwrite d8/d9/de/f31 [0,4194304] 0 2024-04-12T05:09:04.373 INFO:tasks.workunit.client.0.smithi113.stdout:4/559: dwrite da/d5a/f91 [0,4194304] 0 2024-04-12T05:09:04.376 INFO:tasks.workunit.client.1.smithi192.stdout:8/464: chown d6/d10/d46/d5e/d5f/d38/f41 12372 1 2024-04-12T05:09:04.376 INFO:tasks.workunit.client.0.smithi113.stdout:4/560: chown da/d24/d7e/c30 627425938 1 2024-04-12T05:09:04.382 INFO:tasks.workunit.client.0.smithi113.stdout:7/615: mkdir d0/dd/d11/dd5 0 2024-04-12T05:09:04.383 INFO:tasks.workunit.client.1.smithi192.stdout:6/587: mkdir d3/d8/d56/d59/d97/dbf 0 2024-04-12T05:09:04.385 INFO:tasks.workunit.client.0.smithi113.stdout:4/561: symlink da/d24/d7b/lb5 0 2024-04-12T05:09:04.387 INFO:tasks.workunit.client.1.smithi192.stdout:5/517: mkdir d0/d15/d32/db2 0 2024-04-12T05:09:04.388 INFO:tasks.workunit.client.0.smithi113.stdout:3/558: dread d0/d13/d47/d4f/f4c [0,4194304] 0 2024-04-12T05:09:04.388 INFO:tasks.workunit.client.0.smithi113.stdout:7/616: dread d0/dc/d6/fc0 [0,4194304] 0 2024-04-12T05:09:04.389 INFO:tasks.workunit.client.0.smithi113.stdout:4/562: readlink da/dd/l29 0 2024-04-12T05:09:04.390 INFO:tasks.workunit.client.0.smithi113.stdout:3/559: fsync d0/d13/f6a 0 2024-04-12T05:09:04.391 INFO:tasks.workunit.client.1.smithi192.stdout:8/465: creat d6/d10/d46/f92 x:0 0 0 2024-04-12T05:09:04.391 INFO:tasks.workunit.client.0.smithi113.stdout:0/692: rmdir d1/d6/d16/d2a/d50/db6 39 2024-04-12T05:09:04.393 INFO:tasks.workunit.client.0.smithi113.stdout:3/560: write d0/d13/d15/faf [504889,95928] 0 2024-04-12T05:09:04.393 INFO:tasks.workunit.client.1.smithi192.stdout:4/633: dread d1/d54/d55/fa6 [0,4194304] 0 2024-04-12T05:09:04.393 INFO:tasks.workunit.client.0.smithi113.stdout:1/674: chown d10/d2e/l4b 5201 1 2024-04-12T05:09:04.396 INFO:tasks.workunit.client.0.smithi113.stdout:2/473: rename d8/d9/de/d15/d38/l70 to d8/d9/de/d12/l8c 0 2024-04-12T05:09:04.396 INFO:tasks.workunit.client.0.smithi113.stdout:4/563: unlink da/d3c/d83/l41 0 2024-04-12T05:09:04.398 INFO:tasks.workunit.client.0.smithi113.stdout:1/675: creat d10/d2e/d31/dc3/fd5 x:0 0 0 2024-04-12T05:09:04.398 INFO:tasks.workunit.client.0.smithi113.stdout:3/561: mkdir d0/d13/d15/d28/dc0 0 2024-04-12T05:09:04.399 INFO:tasks.workunit.client.1.smithi192.stdout:5/518: creat d0/d15/d3e/fb3 x:0 0 0 2024-04-12T05:09:04.400 INFO:tasks.workunit.client.1.smithi192.stdout:5/519: readlink d0/d15/d64/d7c/da7/laa 0 2024-04-12T05:09:04.401 INFO:tasks.workunit.client.1.smithi192.stdout:8/466: rmdir d6/d10/d7f 0 2024-04-12T05:09:04.402 INFO:tasks.workunit.client.1.smithi192.stdout:4/634: creat d1/d1d/fd2 x:0 0 0 2024-04-12T05:09:04.405 INFO:tasks.workunit.client.1.smithi192.stdout:5/520: symlink d0/d15/d24/d29/lb4 0 2024-04-12T05:09:04.405 INFO:tasks.workunit.client.0.smithi113.stdout:3/562: write d0/d13/d15/d28/d2a/d34/d5f/f65 [2218012,60504] 0 2024-04-12T05:09:04.407 INFO:tasks.workunit.client.1.smithi192.stdout:5/521: chown d0/d21/f2f 799 1 2024-04-12T05:09:04.408 INFO:tasks.workunit.client.1.smithi192.stdout:5/522: stat d0/d15/d32/f4c 0 2024-04-12T05:09:04.410 INFO:tasks.workunit.client.1.smithi192.stdout:8/467: dread d6/d10/d46/d5e/d5f/d38/f5a [0,4194304] 0 2024-04-12T05:09:04.411 INFO:tasks.workunit.client.0.smithi113.stdout:4/564: dwrite da/d24/d60/fb2 [0,4194304] 0 2024-04-12T05:09:04.411 INFO:tasks.workunit.client.0.smithi113.stdout:3/563: stat d0/d13/d15/d28/l32 0 2024-04-12T05:09:04.412 INFO:tasks.workunit.client.0.smithi113.stdout:0/693: rename d1/d6/d16/d2a/d50/d52/d7e/db5 to d1/d6/d16/dd5 0 2024-04-12T05:09:04.414 INFO:tasks.workunit.client.1.smithi192.stdout:1/542: write d0/d2/dd/d21/d39/f65 [865550,80983] 0 2024-04-12T05:09:04.415 INFO:tasks.workunit.client.0.smithi113.stdout:1/676: dwrite d10/d2e/d31/f53 [4194304,4194304] 0 2024-04-12T05:09:04.415 INFO:tasks.workunit.client.0.smithi113.stdout:7/617: dread d0/dd/d11/f20 [0,4194304] 0 2024-04-12T05:09:04.416 INFO:tasks.workunit.client.0.smithi113.stdout:4/565: symlink da/d4f/lb6 0 2024-04-12T05:09:04.417 INFO:tasks.workunit.client.0.smithi113.stdout:4/566: chown da/d73 40876 1 2024-04-12T05:09:04.417 INFO:tasks.workunit.client.1.smithi192.stdout:5/523: dwrite d0/d15/f23 [4194304,4194304] 0 2024-04-12T05:09:04.417 INFO:tasks.workunit.client.1.smithi192.stdout:6/588: dread d3/d5/f45 [0,4194304] 0 2024-04-12T05:09:04.418 INFO:tasks.workunit.client.0.smithi113.stdout:2/474: rename d8/d9/d1a/f3a to d8/d9/de/d15/d24/f8d 0 2024-04-12T05:09:04.421 INFO:tasks.workunit.client.1.smithi192.stdout:5/524: write d0/d15/d64/fad [1002142,87849] 0 2024-04-12T05:09:04.428 INFO:tasks.workunit.client.1.smithi192.stdout:6/589: unlink d3/d8/d48/f73 0 2024-04-12T05:09:04.428 INFO:tasks.workunit.client.1.smithi192.stdout:1/543: symlink d0/d2/dd/d3c/d5b/d7c/la8 0 2024-04-12T05:09:04.428 INFO:tasks.workunit.client.0.smithi113.stdout:4/567: dwrite da/d3c/d83/fa6 [0,4194304] 0 2024-04-12T05:09:04.429 INFO:tasks.workunit.client.1.smithi192.stdout:5/525: mkdir d0/d15/d64/d7c/db5 0 2024-04-12T05:09:04.430 INFO:tasks.workunit.client.0.smithi113.stdout:7/618: mknod d0/dc/da9/d5c/d9a/d9e/cd6 0 2024-04-12T05:09:04.432 INFO:tasks.workunit.client.1.smithi192.stdout:6/590: chown d3/d8/c98 15856638 1 2024-04-12T05:09:04.436 INFO:tasks.workunit.client.1.smithi192.stdout:6/591: chown d3/d5/f80 75249494 1 2024-04-12T05:09:04.440 INFO:tasks.workunit.client.0.smithi113.stdout:0/694: rmdir d1/d6/d34 39 2024-04-12T05:09:04.440 INFO:tasks.workunit.client.1.smithi192.stdout:8/468: write d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/f75 [1752016,59164] 0 2024-04-12T05:09:04.441 INFO:tasks.workunit.client.0.smithi113.stdout:4/568: link da/d2b/f3e da/d24/d60/db4/fb7 0 2024-04-12T05:09:04.442 INFO:tasks.workunit.client.1.smithi192.stdout:1/544: creat d0/d2/dd/fa9 x:0 0 0 2024-04-12T05:09:04.444 INFO:tasks.workunit.client.0.smithi113.stdout:4/569: mkdir da/d24/d60/db8 0 2024-04-12T05:09:04.445 INFO:tasks.workunit.client.1.smithi192.stdout:5/526: dwrite d0/d15/d32/f6a [0,4194304] 0 2024-04-12T05:09:04.446 INFO:tasks.workunit.client.1.smithi192.stdout:5/527: fsync d0/d15/d24/d29/f85 0 2024-04-12T05:09:04.446 INFO:tasks.workunit.client.1.smithi192.stdout:5/528: readlink d0/l28 0 2024-04-12T05:09:04.447 INFO:tasks.workunit.client.0.smithi113.stdout:4/570: rmdir da/d24/d7b 39 2024-04-12T05:09:04.447 INFO:tasks.workunit.client.1.smithi192.stdout:1/545: creat d0/d1b/d44/d69/faa x:0 0 0 2024-04-12T05:09:04.447 INFO:tasks.workunit.client.0.smithi113.stdout:0/695: mkdir d1/d6/d16/dd5/dd6 0 2024-04-12T05:09:04.449 INFO:tasks.workunit.client.0.smithi113.stdout:0/696: fdatasync d1/d6/d16/dd5/fc7 0 2024-04-12T05:09:04.449 INFO:tasks.workunit.client.1.smithi192.stdout:5/529: fdatasync d0/f11 0 2024-04-12T05:09:04.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:02.671736+0000 mgr.smithi192.ocppyo (mgr.24357) 27 : cluster [DBG] pgmap v21: 65 pgs: 65 active+clean; 2.1 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 64 MiB/s rd, 311 MiB/s wr, 513 op/s 2024-04-12T05:09:04.451 INFO:tasks.workunit.client.1.smithi192.stdout:8/469: getdents d6/d10/d46/d7c 0 2024-04-12T05:09:04.452 INFO:tasks.workunit.client.0.smithi113.stdout:0/697: readlink d1/d6/d16/d2a/d50/db6/l12 0 2024-04-12T05:09:04.453 INFO:tasks.workunit.client.0.smithi113.stdout:4/571: symlink da/d24/d60/db8/lb9 0 2024-04-12T05:09:04.454 INFO:tasks.workunit.client.1.smithi192.stdout:5/530: creat d0/d15/d64/fb6 x:0 0 0 2024-04-12T05:09:04.454 INFO:tasks.workunit.client.0.smithi113.stdout:3/564: write d0/d13/d15/d28/f31 [2013801,56882] 0 2024-04-12T05:09:04.456 INFO:tasks.workunit.client.0.smithi113.stdout:4/572: creat da/d3c/d9f/fba x:0 0 0 2024-04-12T05:09:04.456 INFO:tasks.workunit.client.0.smithi113.stdout:0/698: rename d1/d6/d16/d2a/fbd to d1/fd7 0 2024-04-12T05:09:04.457 INFO:tasks.workunit.client.0.smithi113.stdout:3/565: mkdir d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1 0 2024-04-12T05:09:04.459 INFO:tasks.workunit.client.1.smithi192.stdout:1/546: dwrite d0/d2/f34 [4194304,4194304] 0 2024-04-12T05:09:04.460 INFO:tasks.workunit.client.0.smithi113.stdout:2/475: dread d8/d9/f69 [0,4194304] 0 2024-04-12T05:09:04.460 INFO:tasks.workunit.client.1.smithi192.stdout:5/531: creat d0/d15/d24/d47/d7b/fb7 x:0 0 0 2024-04-12T05:09:04.462 INFO:tasks.workunit.client.0.smithi113.stdout:1/677: write d10/d2e/d7c/dc5/dc8/fa6 [4893533,83103] 0 2024-04-12T05:09:04.465 INFO:tasks.workunit.client.0.smithi113.stdout:4/573: mknod da/d73/cbb 0 2024-04-12T05:09:04.465 INFO:tasks.workunit.client.1.smithi192.stdout:5/532: mknod d0/d15/d64/d7c/da0/cb8 0 2024-04-12T05:09:04.468 INFO:tasks.workunit.client.0.smithi113.stdout:2/476: creat d8/d9/de/d12/d2b/f8e x:0 0 0 2024-04-12T05:09:04.470 INFO:tasks.workunit.client.0.smithi113.stdout:4/574: mkdir da/d3c/d43/dbc 0 2024-04-12T05:09:04.472 INFO:tasks.workunit.client.1.smithi192.stdout:6/592: dwrite d3/dc/d39/f3a [0,4194304] 0 2024-04-12T05:09:04.472 INFO:tasks.workunit.client.0.smithi113.stdout:2/477: fdatasync d8/d9/de/d43/f5c 0 2024-04-12T05:09:04.472 INFO:tasks.workunit.client.0.smithi113.stdout:0/699: rename d1/d6/d16/d2a/d50/db6/l12 to d1/d6/d34/dc3/d87/ld8 0 2024-04-12T05:09:04.473 INFO:tasks.workunit.client.1.smithi192.stdout:6/593: readlink d3/d5/l3f 0 2024-04-12T05:09:04.474 INFO:tasks.workunit.client.1.smithi192.stdout:5/533: getdents d0/d15 0 2024-04-12T05:09:04.475 INFO:tasks.workunit.client.0.smithi113.stdout:7/619: dwrite d0/dc/f41 [0,4194304] 0 2024-04-12T05:09:04.477 INFO:tasks.workunit.client.0.smithi113.stdout:3/566: dwrite d0/d13/d15/d28/d2a/f98 [0,4194304] 0 2024-04-12T05:09:04.477 INFO:tasks.workunit.client.0.smithi113.stdout:2/478: fsync d8/d9/f30 0 2024-04-12T05:09:04.478 INFO:tasks.workunit.client.1.smithi192.stdout:6/594: truncate d3/d5/d9c/fa7 1115285 0 2024-04-12T05:09:04.478 INFO:tasks.workunit.client.0.smithi113.stdout:4/575: mkdir da/d3c/d83/d47/dbd 0 2024-04-12T05:09:04.480 INFO:tasks.workunit.client.1.smithi192.stdout:0/626: sync 2024-04-12T05:09:04.484 INFO:tasks.workunit.client.0.smithi113.stdout:1/678: creat d10/d14/fd6 x:0 0 0 2024-04-12T05:09:04.484 INFO:tasks.workunit.client.0.smithi113.stdout:7/620: unlink d0/dc/d6/d9b/f62 0 2024-04-12T05:09:04.489 INFO:tasks.workunit.client.0.smithi113.stdout:7/621: symlink d0/dd/d11/ld7 0 2024-04-12T05:09:04.489 INFO:tasks.workunit.client.1.smithi192.stdout:5/534: mkdir d0/d15/d24/d47/d7b/db9 0 2024-04-12T05:09:04.490 INFO:tasks.workunit.client.0.smithi113.stdout:3/567: truncate d0/d13/d15/d28/d2a/d34/d5f/f96 1026775 0 2024-04-12T05:09:04.490 INFO:tasks.workunit.client.0.smithi113.stdout:2/479: mknod d8/d9/de/c8f 0 2024-04-12T05:09:04.490 INFO:tasks.workunit.client.0.smithi113.stdout:7/622: chown d0/dc/da9/l4b 6 1 2024-04-12T05:09:04.490 INFO:tasks.workunit.client.0.smithi113.stdout:3/568: dread - d0/fa8 zero size 2024-04-12T05:09:04.490 INFO:tasks.workunit.client.1.smithi192.stdout:8/470: write d6/d10/d46/d5e/d5f/d31/f36 [972190,79188] 0 2024-04-12T05:09:04.491 INFO:tasks.workunit.client.0.smithi113.stdout:4/576: getdents da/d3c/d83/d47/dbd 0 2024-04-12T05:09:04.491 INFO:tasks.workunit.client.1.smithi192.stdout:0/627: dread d0/fad [0,4194304] 0 2024-04-12T05:09:04.492 INFO:tasks.workunit.client.0.smithi113.stdout:0/700: chown d1/d6/d16/d2a/c46 120462325 1 2024-04-12T05:09:04.494 INFO:tasks.workunit.client.1.smithi192.stdout:0/628: stat d0/d7b/d9b/f38 0 2024-04-12T05:09:04.497 INFO:tasks.workunit.client.1.smithi192.stdout:8/471: unlink d6/d21/f6d 0 2024-04-12T05:09:04.497 INFO:tasks.workunit.client.0.smithi113.stdout:7/623: unlink d0/dc/da9/d5c/d85/f9c 0 2024-04-12T05:09:04.499 INFO:tasks.workunit.client.1.smithi192.stdout:5/535: creat d0/d15/d24/d47/d7b/db9/fba x:0 0 0 2024-04-12T05:09:04.500 INFO:tasks.workunit.client.0.smithi113.stdout:2/480: dwrite d8/d9/f69 [0,4194304] 0 2024-04-12T05:09:04.500 INFO:tasks.workunit.client.1.smithi192.stdout:5/536: write d0/d15/d3e/f71 [1020212,45864] 0 2024-04-12T05:09:04.503 INFO:tasks.workunit.client.0.smithi113.stdout:1/679: symlink d10/d55/d9c/ld7 0 2024-04-12T05:09:04.505 INFO:tasks.workunit.client.1.smithi192.stdout:0/629: symlink d0/d7b/d9b/d4b/dbc/dce/d95/db4/ld3 0 2024-04-12T05:09:04.506 INFO:tasks.workunit.client.0.smithi113.stdout:1/680: chown d10/d14/c18 0 1 2024-04-12T05:09:04.507 INFO:tasks.workunit.client.1.smithi192.stdout:5/537: symlink d0/d21/d5e/lbb 0 2024-04-12T05:09:04.508 INFO:tasks.workunit.client.0.smithi113.stdout:3/569: write d0/f18 [492304,107374] 0 2024-04-12T05:09:04.513 INFO:tasks.workunit.client.0.smithi113.stdout:3/570: chown d0/d13/d15/faf 5431 1 2024-04-12T05:09:04.514 INFO:tasks.workunit.client.0.smithi113.stdout:7/624: dwrite d0/dc/da9/fa1 [0,4194304] 0 2024-04-12T05:09:04.514 INFO:tasks.workunit.client.0.smithi113.stdout:3/571: fdatasync d0/d13/d15/d28/d2a/f36 0 2024-04-12T05:09:04.515 INFO:tasks.workunit.client.1.smithi192.stdout:0/630: dwrite d0/d7b/d9b/f38 [0,4194304] 0 2024-04-12T05:09:04.516 INFO:tasks.workunit.client.0.smithi113.stdout:2/481: dwrite d8/d9/de/f10 [0,4194304] 0 2024-04-12T05:09:04.517 INFO:tasks.workunit.client.0.smithi113.stdout:3/572: chown d0/d13/f81 1 1 2024-04-12T05:09:04.523 INFO:tasks.workunit.client.0.smithi113.stdout:3/573: truncate d0/fa6 1093759 0 2024-04-12T05:09:04.523 INFO:tasks.workunit.client.0.smithi113.stdout:7/625: mknod d0/dc/d6/d1a/d81/cd8 0 2024-04-12T05:09:04.527 INFO:tasks.workunit.client.1.smithi192.stdout:0/631: write d0/d7b/d9b/d4b/dbc/dce/f87 [1399473,110981] 0 2024-04-12T05:09:04.527 INFO:tasks.workunit.client.0.smithi113.stdout:7/626: write d0/dc/d50/f5f [3134781,25402] 0 2024-04-12T05:09:04.529 INFO:tasks.workunit.client.0.smithi113.stdout:3/574: dread - d0/d13/d15/d28/d2a/fbb zero size 2024-04-12T05:09:04.531 INFO:tasks.workunit.client.0.smithi113.stdout:7/627: mkdir d0/dc/d6/d9b/d9f/db3/dd9 0 2024-04-12T05:09:04.533 INFO:tasks.workunit.client.0.smithi113.stdout:3/575: creat d0/d13/d15/d28/d2a/d8f/fc2 x:0 0 0 2024-04-12T05:09:04.535 INFO:tasks.workunit.client.0.smithi113.stdout:7/628: write d0/dc/d6/f16 [1876657,68386] 0 2024-04-12T05:09:04.535 INFO:tasks.workunit.client.0.smithi113.stdout:0/701: dread d1/d6/d16/d2a/d50/f7d [0,4194304] 0 2024-04-12T05:09:04.537 INFO:tasks.workunit.client.0.smithi113.stdout:3/576: write d0/d13/d15/d28/d2a/d34/f72 [536415,36590] 0 2024-04-12T05:09:04.538 INFO:tasks.workunit.client.0.smithi113.stdout:7/629: readlink d0/dc/da9/d5c/la7 0 2024-04-12T05:09:04.540 INFO:tasks.workunit.client.0.smithi113.stdout:3/577: creat d0/d13/d47/d94/fc3 x:0 0 0 2024-04-12T05:09:04.541 INFO:tasks.workunit.client.0.smithi113.stdout:7/630: symlink d0/d52/dc5/lda 0 2024-04-12T05:09:04.541 INFO:tasks.workunit.client.0.smithi113.stdout:4/577: dread da/d3c/d83/f40 [4194304,4194304] 0 2024-04-12T05:09:04.543 INFO:tasks.workunit.client.0.smithi113.stdout:3/578: truncate d0/d13/f6b 1166092 0 2024-04-12T05:09:04.544 INFO:tasks.workunit.client.0.smithi113.stdout:3/579: chown d0/d13/d15/d28/d2a/d34/d5f/c93 1126 1 2024-04-12T05:09:04.546 INFO:tasks.workunit.client.0.smithi113.stdout:7/631: link d0/d52/fbc d0/dc/da9/d5c/fdb 0 2024-04-12T05:09:04.547 INFO:tasks.workunit.client.0.smithi113.stdout:7/632: dread - d0/d52/f74 zero size 2024-04-12T05:09:04.550 INFO:tasks.workunit.client.0.smithi113.stdout:7/633: symlink d0/dd/d11/dd5/ldc 0 2024-04-12T05:09:04.552 INFO:tasks.workunit.client.0.smithi113.stdout:7/634: truncate d0/dc/d6/d9b/d80/f96 1013383 0 2024-04-12T05:09:04.554 INFO:tasks.workunit.client.0.smithi113.stdout:1/681: dread fb [0,4194304] 0 2024-04-12T05:09:04.558 INFO:tasks.workunit.client.0.smithi113.stdout:0/702: symlink d1/d6/d16/dd5/dbb/ld9 0 2024-04-12T05:09:04.559 INFO:tasks.workunit.client.0.smithi113.stdout:1/682: dread - d10/d2e/d31/d5c/f97 zero size 2024-04-12T05:09:04.560 INFO:tasks.workunit.client.0.smithi113.stdout:3/580: dwrite d0/d13/d15/d28/d2a/f9e [0,4194304] 0 2024-04-12T05:09:04.561 INFO:tasks.workunit.client.0.smithi113.stdout:1/683: readlink d10/d2e/d31/l84 0 2024-04-12T05:09:04.562 INFO:tasks.workunit.client.0.smithi113.stdout:7/635: dwrite d0/dc/f18 [0,4194304] 0 2024-04-12T05:09:04.564 INFO:tasks.workunit.client.0.smithi113.stdout:3/581: symlink d0/d13/d47/d4f/lc4 0 2024-04-12T05:09:04.564 INFO:tasks.workunit.client.0.smithi113.stdout:0/703: dread d1/d6/d16/d2a/d50/f7d [0,4194304] 0 2024-04-12T05:09:04.566 INFO:tasks.workunit.client.0.smithi113.stdout:7/636: unlink d0/dd/fa5 0 2024-04-12T05:09:04.568 INFO:tasks.workunit.client.0.smithi113.stdout:3/582: read d0/d13/d47/f78 [1089600,59525] 0 2024-04-12T05:09:04.570 INFO:tasks.workunit.client.0.smithi113.stdout:7/637: creat d0/dd/d11/dd5/fdd x:0 0 0 2024-04-12T05:09:04.574 INFO:tasks.workunit.client.0.smithi113.stdout:7/638: write d0/dc/d6/d1a/da0/fc6 [5195429,42526] 0 2024-04-12T05:09:04.576 INFO:tasks.workunit.client.0.smithi113.stdout:3/583: dwrite d0/d13/d15/d28/d2a/d34/d5f/f65 [0,4194304] 0 2024-04-12T05:09:04.577 INFO:tasks.workunit.client.0.smithi113.stdout:7/639: unlink d0/f51 0 2024-04-12T05:09:04.581 INFO:tasks.workunit.client.0.smithi113.stdout:3/584: unlink d0/d13/d47/d4f/c76 0 2024-04-12T05:09:04.589 INFO:tasks.workunit.client.0.smithi113.stdout:0/704: dwrite d1/d6/d34/f78 [0,4194304] 0 2024-04-12T05:09:04.589 INFO:tasks.workunit.client.0.smithi113.stdout:1/684: dwrite d10/d21/f40 [4194304,4194304] 0 2024-04-12T05:09:04.590 INFO:tasks.workunit.client.0.smithi113.stdout:7/640: dwrite d0/dc/d6/d1a/da0/fc6 [0,4194304] 0 2024-04-12T05:09:04.591 INFO:tasks.workunit.client.0.smithi113.stdout:7/641: dread - d0/dc/d6/f86 zero size 2024-04-12T05:09:04.593 INFO:tasks.workunit.client.0.smithi113.stdout:7/642: creat d0/dc/fde x:0 0 0 2024-04-12T05:09:04.603 INFO:tasks.workunit.client.0.smithi113.stdout:1/685: rename d10/d14/d67 to d10/d2e/dd8 0 2024-04-12T05:09:04.603 INFO:tasks.workunit.client.0.smithi113.stdout:5/575: sync 2024-04-12T05:09:04.603 INFO:tasks.workunit.client.0.smithi113.stdout:9/625: sync 2024-04-12T05:09:04.603 INFO:tasks.workunit.client.0.smithi113.stdout:6/557: sync 2024-04-12T05:09:04.603 INFO:tasks.workunit.client.0.smithi113.stdout:8/622: sync 2024-04-12T05:09:04.606 INFO:tasks.workunit.client.0.smithi113.stdout:8/623: write d9/d16/f1a [9389292,14661] 0 2024-04-12T05:09:04.608 INFO:tasks.workunit.client.0.smithi113.stdout:8/624: read d9/f9e [40785,54315] 0 2024-04-12T05:09:04.608 INFO:tasks.workunit.client.0.smithi113.stdout:9/626: symlink d1/d4/d38/ld5 0 2024-04-12T05:09:04.694 INFO:tasks.workunit.client.0.smithi113.stdout:1/686: rename d10/d41/f45 to d10/d41/fd9 0 2024-04-12T05:09:04.694 INFO:tasks.workunit.client.0.smithi113.stdout:6/558: creat d8/de/d13/d11/d36/d53/db7/fc9 x:0 0 0 2024-04-12T05:09:04.697 INFO:tasks.workunit.client.0.smithi113.stdout:8/625: read d9/d1b/d8e/d93/f28 [757053,2106] 0 2024-04-12T05:09:04.697 INFO:tasks.workunit.client.0.smithi113.stdout:9/627: chown d1/d4/d38/d65/lc5 6 1 2024-04-12T05:09:04.697 INFO:tasks.workunit.client.0.smithi113.stdout:7/643: rmdir d0/dc/d6/d9b/d9f/db3/dd9 0 2024-04-12T05:09:04.699 INFO:tasks.workunit.client.0.smithi113.stdout:5/576: rename d0/f6 to d0/d10/d27/d1d/d38/fba 0 2024-04-12T05:09:04.700 INFO:tasks.workunit.client.0.smithi113.stdout:6/559: mkdir d8/de/d13/d11/d36/d2c/dca 0 2024-04-12T05:09:04.700 INFO:tasks.workunit.client.0.smithi113.stdout:6/560: chown d8/de/d17/f72 8794 1 2024-04-12T05:09:04.701 INFO:tasks.workunit.client.0.smithi113.stdout:9/628: unlink d1/d4/d5/fac 0 2024-04-12T05:09:04.702 INFO:tasks.workunit.client.0.smithi113.stdout:7/644: mkdir d0/dc/da9/d5c/ddf 0 2024-04-12T05:09:04.706 INFO:tasks.workunit.client.0.smithi113.stdout:8/626: truncate d9/d16/d65/db1/f2c 1128479 0 2024-04-12T05:09:04.709 INFO:tasks.workunit.client.0.smithi113.stdout:6/561: rename d8/de/c1f to d8/d1d/ccb 0 2024-04-12T05:09:04.712 INFO:tasks.workunit.client.0.smithi113.stdout:6/562: creat d8/d1d/fcc x:0 0 0 2024-04-12T05:09:04.714 INFO:tasks.workunit.client.0.smithi113.stdout:1/687: dread ff [0,4194304] 0 2024-04-12T05:09:04.716 INFO:tasks.workunit.client.0.smithi113.stdout:9/629: dread d1/df/db6/f6b [0,4194304] 0 2024-04-12T05:09:04.717 INFO:tasks.workunit.client.0.smithi113.stdout:8/627: dwrite d9/d16/d65/db1/f8b [0,4194304] 0 2024-04-12T05:09:04.718 INFO:tasks.workunit.client.1.smithi192.stdout:2/545: sync 2024-04-12T05:09:04.718 INFO:tasks.workunit.client.1.smithi192.stdout:7/461: sync 2024-04-12T05:09:04.718 INFO:tasks.workunit.client.1.smithi192.stdout:7/462: stat d3/d11/f46 0 2024-04-12T05:09:04.719 INFO:tasks.workunit.client.1.smithi192.stdout:2/546: write d1/d8/d3a/de/d7c/f7d [916050,230] 0 2024-04-12T05:09:04.721 INFO:tasks.workunit.client.0.smithi113.stdout:1/688: creat d10/d41/d89/d96/dd3/fda x:0 0 0 2024-04-12T05:09:04.722 INFO:tasks.workunit.client.0.smithi113.stdout:5/577: mkdir d0/d10/d27/d1d/d38/d44/d4e/daa/dbb 0 2024-04-12T05:09:04.726 INFO:tasks.workunit.client.1.smithi192.stdout:7/463: write d3/d7/d15/f29 [5834314,84371] 0 2024-04-12T05:09:04.726 INFO:tasks.workunit.client.0.smithi113.stdout:8/628: mknod d9/d16/d65/db4/cc4 0 2024-04-12T05:09:04.726 INFO:tasks.workunit.client.0.smithi113.stdout:6/563: dwrite d8/d45/f49 [0,4194304] 0 2024-04-12T05:09:04.730 INFO:tasks.workunit.client.1.smithi192.stdout:2/547: dwrite d1/d8/d3a/f26 [0,4194304] 0 2024-04-12T05:09:04.731 INFO:tasks.workunit.client.0.smithi113.stdout:5/578: unlink d0/d10/d27/d1d/d38/d3b/f84 0 2024-04-12T05:09:04.731 INFO:tasks.workunit.client.1.smithi192.stdout:2/548: readlink d1/d8/ld 0 2024-04-12T05:09:04.732 INFO:tasks.workunit.client.0.smithi113.stdout:7/645: dread d0/fa2 [0,4194304] 0 2024-04-12T05:09:04.737 INFO:tasks.workunit.client.1.smithi192.stdout:2/549: unlink d1/d8/d17/d19/d60/d59/d68/c6a 0 2024-04-12T05:09:04.737 INFO:tasks.workunit.client.0.smithi113.stdout:6/564: creat d8/d1d/fcd x:0 0 0 2024-04-12T05:09:04.737 INFO:tasks.workunit.client.0.smithi113.stdout:5/579: read - d0/d10/d27/f57 zero size 2024-04-12T05:09:04.740 INFO:tasks.workunit.client.0.smithi113.stdout:8/629: dwrite d9/d1b/f61 [4194304,4194304] 0 2024-04-12T05:09:04.744 INFO:tasks.workunit.client.1.smithi192.stdout:2/550: link d1/d8/d17/d19/d71/d7f/l91 d1/d8/d3a/de/d7c/d98/la4 0 2024-04-12T05:09:04.745 INFO:tasks.workunit.client.1.smithi192.stdout:2/551: chown d1/d8/d17/d19/d60/d59/d68/l8a 23083569 1 2024-04-12T05:09:04.746 INFO:tasks.workunit.client.1.smithi192.stdout:2/552: stat d1/d2d/f3d 0 2024-04-12T05:09:04.749 INFO:tasks.workunit.client.0.smithi113.stdout:6/565: dread d8/de/d13/d11/d36/d53/db7/fba [0,4194304] 0 2024-04-12T05:09:04.749 INFO:tasks.workunit.client.0.smithi113.stdout:8/630: dread d9/f9d [0,4194304] 0 2024-04-12T05:09:04.750 INFO:tasks.workunit.client.1.smithi192.stdout:2/553: symlink d1/d8/d3a/de/d7c/d98/la5 0 2024-04-12T05:09:04.755 INFO:tasks.workunit.client.1.smithi192.stdout:7/464: write d3/d7/d14/f49 [240952,117582] 0 2024-04-12T05:09:04.756 INFO:tasks.workunit.client.0.smithi113.stdout:7/646: dread d0/dc/da9/d5c/d9a/fae [0,4194304] 0 2024-04-12T05:09:04.756 INFO:tasks.workunit.client.1.smithi192.stdout:2/554: creat d1/d8/d17/d19/d60/fa6 x:0 0 0 2024-04-12T05:09:04.757 INFO:tasks.workunit.client.1.smithi192.stdout:7/465: creat d3/f9e x:0 0 0 2024-04-12T05:09:04.758 INFO:tasks.workunit.client.0.smithi113.stdout:7/647: chown d0/dc/d6/d78 187 1 2024-04-12T05:09:04.758 INFO:tasks.workunit.client.1.smithi192.stdout:2/555: fdatasync d1/f70 0 2024-04-12T05:09:04.759 INFO:tasks.workunit.client.1.smithi192.stdout:7/466: write d3/d1d/f69 [19733,79321] 0 2024-04-12T05:09:04.761 INFO:tasks.workunit.client.0.smithi113.stdout:1/689: dwrite d10/d41/d89/d96/d7d/f7e [0,4194304] 0 2024-04-12T05:09:04.761 INFO:tasks.workunit.client.0.smithi113.stdout:7/648: dread - d0/dc/d6/d9b/fcb zero size 2024-04-12T05:09:04.766 INFO:tasks.workunit.client.0.smithi113.stdout:1/690: mkdir d10/d2e/d7c/dc5/ddb 0 2024-04-12T05:09:04.766 INFO:tasks.workunit.client.0.smithi113.stdout:7/649: creat d0/fe0 x:0 0 0 2024-04-12T05:09:04.766 INFO:tasks.workunit.client.0.smithi113.stdout:5/580: write d0/d10/d27/d1d/f49 [311910,9931] 0 2024-04-12T05:09:04.770 INFO:tasks.workunit.client.1.smithi192.stdout:7/467: getdents d3/d1d/d74/d86 0 2024-04-12T05:09:04.771 INFO:tasks.workunit.client.0.smithi113.stdout:1/691: dread - d10/da5/fb3 zero size 2024-04-12T05:09:04.775 INFO:tasks.workunit.client.1.smithi192.stdout:7/468: rename d3/d11/f2d to d3/d1d/f9f 0 2024-04-12T05:09:04.776 INFO:tasks.workunit.client.1.smithi192.stdout:7/469: chown d3/d1d/d74/d86/d93 1 1 2024-04-12T05:09:04.776 INFO:tasks.workunit.client.0.smithi113.stdout:5/581: unlink d0/d10/d27/d1d/d38/d44/c74 0 2024-04-12T05:09:04.776 INFO:tasks.workunit.client.0.smithi113.stdout:1/692: stat d10/d2e/d7c/dc5/ddb 0 2024-04-12T05:09:04.779 INFO:tasks.workunit.client.0.smithi113.stdout:6/566: dwrite d8/de/d13/d11/d36/d53/db8/d63/f69 [0,4194304] 0 2024-04-12T05:09:04.780 INFO:tasks.workunit.client.0.smithi113.stdout:8/631: dwrite d9/f55 [0,4194304] 0 2024-04-12T05:09:04.781 INFO:tasks.workunit.client.1.smithi192.stdout:2/556: dread d1/d8/d17/d19/d40/f5b [0,4194304] 0 2024-04-12T05:09:04.783 INFO:tasks.workunit.client.1.smithi192.stdout:7/470: dwrite d3/d1d/f2e [8388608,4194304] 0 2024-04-12T05:09:04.784 INFO:tasks.workunit.client.1.smithi192.stdout:7/471: fsync d3/d11/f3b 0 2024-04-12T05:09:04.785 INFO:tasks.workunit.client.1.smithi192.stdout:7/472: chown d3/d7/d15/f29 2 1 2024-04-12T05:09:04.785 INFO:tasks.workunit.client.0.smithi113.stdout:1/693: mkdir d10/ddc 0 2024-04-12T05:09:04.787 INFO:tasks.workunit.client.0.smithi113.stdout:5/582: creat d0/d10/d27/d1d/d38/d44/fbc x:0 0 0 2024-04-12T05:09:04.790 INFO:tasks.workunit.client.0.smithi113.stdout:8/632: mkdir d9/d16/dc5 0 2024-04-12T05:09:04.792 INFO:tasks.workunit.client.1.smithi192.stdout:7/473: truncate d3/d7/d14/d1f/f34 5092574 0 2024-04-12T05:09:04.792 INFO:tasks.workunit.client.0.smithi113.stdout:5/583: truncate d0/d10/d27/d1d/d38/d3b/f5e 4959755 0 2024-04-12T05:09:04.792 INFO:tasks.workunit.client.0.smithi113.stdout:1/694: dread - d10/d21/fac zero size 2024-04-12T05:09:04.794 INFO:tasks.workunit.client.0.smithi113.stdout:8/633: mknod d9/d16/d65/db1/d59/da3/cc6 0 2024-04-12T05:09:04.794 INFO:tasks.workunit.client.1.smithi192.stdout:7/474: fsync d3/d7/f8 0 2024-04-12T05:09:04.796 INFO:tasks.workunit.client.0.smithi113.stdout:5/584: creat d0/d10/d27/d1d/d3f/db0/fbd x:0 0 0 2024-04-12T05:09:04.796 INFO:tasks.workunit.client.1.smithi192.stdout:2/557: write d1/d8/d17/d19/d40/f4c [614292,122047] 0 2024-04-12T05:09:04.797 INFO:tasks.workunit.client.0.smithi113.stdout:5/585: write d0/f3a [3467321,22410] 0 2024-04-12T05:09:04.797 INFO:tasks.workunit.client.0.smithi113.stdout:8/634: rename d9/d16/d7f/d3a/f2a to d9/d16/d65/db1/fc7 0 2024-04-12T05:09:04.799 INFO:tasks.workunit.client.0.smithi113.stdout:1/695: mkdir d10/d2e/d31/ddd 0 2024-04-12T05:09:04.800 INFO:tasks.workunit.client.0.smithi113.stdout:8/635: symlink d9/d16/d3c/lc8 0 2024-04-12T05:09:04.801 INFO:tasks.workunit.client.1.smithi192.stdout:2/558: mkdir d1/da7 0 2024-04-12T05:09:04.801 INFO:tasks.workunit.client.0.smithi113.stdout:5/586: symlink d0/d10/d27/d1d/d3f/d7e/lbe 0 2024-04-12T05:09:04.802 INFO:tasks.workunit.client.0.smithi113.stdout:1/696: fsync d10/d14/fbc 0 2024-04-12T05:09:04.803 INFO:tasks.workunit.client.0.smithi113.stdout:8/636: mknod d9/d16/dc5/cc9 0 2024-04-12T05:09:04.803 INFO:tasks.workunit.client.1.smithi192.stdout:2/559: mkdir d1/d2d/d73/da8 0 2024-04-12T05:09:04.805 INFO:tasks.workunit.client.1.smithi192.stdout:2/560: write d1/d8/d17/d19/d60/f65 [656625,112815] 0 2024-04-12T05:09:04.806 INFO:tasks.workunit.client.0.smithi113.stdout:6/567: dread d8/de/d17/f59 [0,4194304] 0 2024-04-12T05:09:04.806 INFO:tasks.workunit.client.0.smithi113.stdout:1/697: mkdir d10/ddc/dde 0 2024-04-12T05:09:04.806 INFO:tasks.workunit.client.0.smithi113.stdout:8/637: mkdir d9/d16/d65/db1/d59/d68/dca 0 2024-04-12T05:09:04.808 INFO:tasks.workunit.client.0.smithi113.stdout:5/587: rename d0/d10/d27/f22 to d0/d10/d27/d1d/fbf 0 2024-04-12T05:09:04.809 INFO:tasks.workunit.client.0.smithi113.stdout:1/698: mkdir d10/da5/ddf 0 2024-04-12T05:09:04.809 INFO:tasks.workunit.client.0.smithi113.stdout:6/568: read - d8/de/d17/f7b zero size 2024-04-12T05:09:04.810 INFO:tasks.workunit.client.0.smithi113.stdout:7/650: dread d0/dc/d6/d9b/f25 [0,4194304] 0 2024-04-12T05:09:04.810 INFO:tasks.workunit.client.1.smithi192.stdout:2/561: mknod d1/d8/d17/d19/d60/d9c/ca9 0 2024-04-12T05:09:04.810 INFO:tasks.workunit.client.0.smithi113.stdout:8/638: mkdir d9/d16/d65/db4/dcb 0 2024-04-12T05:09:04.811 INFO:tasks.workunit.client.1.smithi192.stdout:4/635: sync 2024-04-12T05:09:04.811 INFO:tasks.workunit.client.1.smithi192.stdout:9/460: sync 2024-04-12T05:09:04.811 INFO:tasks.workunit.client.1.smithi192.stdout:3/527: sync 2024-04-12T05:09:04.811 INFO:tasks.workunit.client.1.smithi192.stdout:1/547: sync 2024-04-12T05:09:04.811 INFO:tasks.workunit.client.1.smithi192.stdout:7/475: write d3/d11/f5c [1154972,32035] 0 2024-04-12T05:09:04.811 INFO:tasks.workunit.client.1.smithi192.stdout:8/472: sync 2024-04-12T05:09:04.812 INFO:tasks.workunit.client.1.smithi192.stdout:6/595: sync 2024-04-12T05:09:04.812 INFO:tasks.workunit.client.1.smithi192.stdout:0/632: sync 2024-04-12T05:09:04.812 INFO:tasks.workunit.client.1.smithi192.stdout:5/538: sync 2024-04-12T05:09:04.812 INFO:tasks.workunit.client.0.smithi113.stdout:1/699: write d10/d2e/d7c/dc5/fc7 [83947,58488] 0 2024-04-12T05:09:04.814 INFO:tasks.workunit.client.0.smithi113.stdout:5/588: mkdir d0/d10/d27/d1d/d38/d44/d4e/daa/dc0 0 2024-04-12T05:09:04.814 INFO:tasks.workunit.client.0.smithi113.stdout:8/639: rmdir d9/d1b/d8e/d93/d92 39 2024-04-12T05:09:04.815 INFO:tasks.workunit.client.0.smithi113.stdout:5/589: write d0/d10/d27/d1d/d3f/db0/fbd [85628,61330] 0 2024-04-12T05:09:04.815 INFO:tasks.workunit.client.1.smithi192.stdout:9/461: fsync d11/d15/d21/d2c/f1a 0 2024-04-12T05:09:04.817 INFO:tasks.workunit.client.0.smithi113.stdout:5/590: stat d0/d10/d6e/f35 0 2024-04-12T05:09:04.818 INFO:tasks.workunit.client.1.smithi192.stdout:3/528: fsync d5/d13/d3a/d8b/fae 0 2024-04-12T05:09:04.820 INFO:tasks.workunit.client.0.smithi113.stdout:5/591: creat d0/d10/d27/d1d/d3f/d73/d89/fc1 x:0 0 0 2024-04-12T05:09:04.820 INFO:tasks.workunit.client.1.smithi192.stdout:1/548: read d0/d2/dd/d21/d62/d84/f91 [2095890,68657] 0 2024-04-12T05:09:04.821 INFO:tasks.workunit.client.1.smithi192.stdout:4/636: truncate d1/d54/d55/d64/fae 306659 0 2024-04-12T05:09:04.821 INFO:tasks.workunit.client.1.smithi192.stdout:6/596: mkdir d3/d23/db2/dc0 0 2024-04-12T05:09:04.822 INFO:tasks.workunit.client.0.smithi113.stdout:5/592: write d0/d10/d27/d1d/fbf [808264,25856] 0 2024-04-12T05:09:04.823 INFO:tasks.workunit.client.1.smithi192.stdout:9/462: fdatasync d11/d25/d36/d6e/d71/f95 0 2024-04-12T05:09:04.826 INFO:tasks.workunit.client.1.smithi192.stdout:0/633: mknod d0/d7b/d9b/d4b/dbc/dce/d95/db4/cd4 0 2024-04-12T05:09:04.826 INFO:tasks.workunit.client.1.smithi192.stdout:2/562: truncate d1/d8/d3a/de/f11 275034 0 2024-04-12T05:09:04.826 INFO:tasks.workunit.client.0.smithi113.stdout:5/593: dread d0/f31 [0,4194304] 0 2024-04-12T05:09:04.827 INFO:tasks.workunit.client.0.smithi113.stdout:1/700: truncate d10/d2e/d31/f53 483422 0 2024-04-12T05:09:04.828 INFO:tasks.workunit.client.0.smithi113.stdout:6/569: write d8/de/d13/d11/d36/d53/db7/fba [706032,120023] 0 2024-04-12T05:09:04.829 INFO:tasks.workunit.client.0.smithi113.stdout:5/594: readlink d0/d10/d27/d1d/d38/d75/l81 0 2024-04-12T05:09:04.830 INFO:tasks.workunit.client.0.smithi113.stdout:6/570: mkdir d8/de/d13/d11/d36/d53/db8/dce 0 2024-04-12T05:09:04.831 INFO:tasks.workunit.client.1.smithi192.stdout:3/529: mkdir d5/d13/d94/dbd 0 2024-04-12T05:09:04.831 INFO:tasks.workunit.client.1.smithi192.stdout:7/476: dwrite d3/d1d/d4a/d5e/f9a [0,4194304] 0 2024-04-12T05:09:04.836 INFO:tasks.workunit.client.1.smithi192.stdout:6/597: rmdir d3/d8/d56 39 2024-04-12T05:09:04.837 INFO:tasks.workunit.client.0.smithi113.stdout:6/571: write d8/de/d17/f7b [352330,125548] 0 2024-04-12T05:09:04.837 INFO:tasks.workunit.client.1.smithi192.stdout:1/549: mkdir d0/d2/dd/d3c/dab 0 2024-04-12T05:09:04.840 INFO:tasks.workunit.client.0.smithi113.stdout:5/595: dwrite d0/d10/d27/d1d/d8c/fa7 [0,4194304] 0 2024-04-12T05:09:04.844 INFO:tasks.workunit.client.1.smithi192.stdout:9/463: mkdir d11/d25/d36/d6e/d8f/d9c 0 2024-04-12T05:09:04.844 INFO:tasks.workunit.client.0.smithi113.stdout:6/572: fdatasync d8/d1d/fa0 0 2024-04-12T05:09:04.845 INFO:tasks.workunit.client.1.smithi192.stdout:0/634: symlink d0/d7b/d9b/d35/ld5 0 2024-04-12T05:09:04.848 INFO:tasks.workunit.client.0.smithi113.stdout:5/596: rmdir d0/d10/d27/d1d/d38/d75 39 2024-04-12T05:09:04.850 INFO:tasks.workunit.client.1.smithi192.stdout:5/539: link d0/d15/l37 d0/d15/d64/d7c/da7/lbc 0 2024-04-12T05:09:04.851 INFO:tasks.workunit.client.0.smithi113.stdout:6/573: rename d8/de/d13/l4a to d8/de/d13/d11/d36/d3d/d4c/d31/lcf 0 2024-04-12T05:09:04.853 INFO:tasks.workunit.client.0.smithi113.stdout:1/701: dwrite d10/d41/d89/d96/d7d/f79 [4194304,4194304] 0 2024-04-12T05:09:04.855 INFO:tasks.workunit.client.1.smithi192.stdout:0/635: dread d0/fd0 [0,4194304] 0 2024-04-12T05:09:04.856 INFO:tasks.workunit.client.1.smithi192.stdout:7/477: read d3/d7/d60/f66 [580607,99327] 0 2024-04-12T05:09:04.857 INFO:tasks.workunit.client.0.smithi113.stdout:5/597: mknod d0/d10/d6e/d93/cc2 0 2024-04-12T05:09:04.857 INFO:tasks.workunit.client.0.smithi113.stdout:1/702: creat d10/d14/d95/fe0 x:0 0 0 2024-04-12T05:09:04.857 INFO:tasks.workunit.client.1.smithi192.stdout:1/550: symlink d0/d2/dd/d3c/d5b/d7c/lac 0 2024-04-12T05:09:04.858 INFO:tasks.workunit.client.0.smithi113.stdout:6/574: rename d8/de/cb9 to d8/de/d13/d11/d36/d3d/d4c/d26/d5b/cd0 0 2024-04-12T05:09:04.858 INFO:tasks.workunit.client.1.smithi192.stdout:0/636: write d0/d6/d15/d1b/d46/f49 [141924,16170] 0 2024-04-12T05:09:04.858 INFO:tasks.workunit.client.0.smithi113.stdout:1/703: readlink d10/d2e/l3d 0 2024-04-12T05:09:04.859 INFO:tasks.workunit.client.0.smithi113.stdout:5/598: write d0/d10/f45 [624735,18341] 0 2024-04-12T05:09:04.859 INFO:tasks.workunit.client.1.smithi192.stdout:5/540: read d0/d15/d3e/d77/f78 [651718,35557] 0 2024-04-12T05:09:04.860 INFO:tasks.workunit.client.1.smithi192.stdout:4/637: rename d1/d1d/l39 to d1/d54/ld3 0 2024-04-12T05:09:04.861 INFO:tasks.workunit.client.0.smithi113.stdout:6/575: dread - d8/de/d13/d11/d36/d3d/d4c/d31/fc0 zero size 2024-04-12T05:09:04.862 INFO:tasks.workunit.client.1.smithi192.stdout:5/541: fsync d0/d15/d24/d29/f2a 0 2024-04-12T05:09:04.864 INFO:tasks.workunit.client.0.smithi113.stdout:1/704: mkdir d10/d2e/d7c/dc5/dc8/de1 0 2024-04-12T05:09:04.865 INFO:tasks.workunit.client.0.smithi113.stdout:5/599: chown d0/d10/d27/d1d/d38/d44/d4e/daa 0 1 2024-04-12T05:09:04.867 INFO:tasks.workunit.client.0.smithi113.stdout:6/576: mknod d8/de/d13/d11/d36/d3d/d4c/d31/d83/cd1 0 2024-04-12T05:09:04.869 INFO:tasks.workunit.client.1.smithi192.stdout:1/551: dread d0/d1b/f60 [0,4194304] 0 2024-04-12T05:09:04.871 INFO:tasks.workunit.client.1.smithi192.stdout:9/464: rename d11/d15/d21/d2c/c17 to d11/d15/d21/d2c/d1f/d6a/d7e/c9d 0 2024-04-12T05:09:04.872 INFO:tasks.workunit.client.0.smithi113.stdout:5/600: dread d0/d10/d27/d1d/f2a [0,4194304] 0 2024-04-12T05:09:04.873 INFO:tasks.workunit.client.1.smithi192.stdout:7/478: dwrite d3/d11/f46 [0,4194304] 0 2024-04-12T05:09:04.875 INFO:tasks.workunit.client.0.smithi113.stdout:6/577: dread d8/de/d13/d11/d36/f23 [0,4194304] 0 2024-04-12T05:09:04.875 INFO:tasks.workunit.client.1.smithi192.stdout:4/638: creat d1/d54/d55/d64/fd4 x:0 0 0 2024-04-12T05:09:04.878 INFO:tasks.workunit.client.1.smithi192.stdout:2/563: write d1/d8/d3a/de/f11 [852414,122013] 0 2024-04-12T05:09:04.879 INFO:tasks.workunit.client.1.smithi192.stdout:8/473: getdents d6/d10/d46/d5e/d5f/d37/d4f/d51/d65 0 2024-04-12T05:09:04.880 INFO:tasks.workunit.client.0.smithi113.stdout:6/578: chown d8/de/d13/d11/d36/d53/db8/d63/c96 569840599 1 2024-04-12T05:09:04.881 INFO:tasks.workunit.client.1.smithi192.stdout:1/552: mknod d0/d2/d1e/d3f/d56/cad 0 2024-04-12T05:09:04.881 INFO:tasks.workunit.client.0.smithi113.stdout:5/601: dwrite d0/d10/d27/d1d/fbf [0,4194304] 0 2024-04-12T05:09:04.883 INFO:tasks.workunit.client.1.smithi192.stdout:0/637: mkdir d0/d6/d67/db1/dd6 0 2024-04-12T05:09:04.887 INFO:tasks.workunit.client.0.smithi113.stdout:5/602: getdents d0/d10/d27/d1d/d38/d44/d4e/daa/dbb 0 2024-04-12T05:09:04.887 INFO:tasks.workunit.client.1.smithi192.stdout:9/465: unlink d11/f47 0 2024-04-12T05:09:04.888 INFO:tasks.workunit.client.1.smithi192.stdout:6/598: write d3/d23/f76 [567911,14095] 0 2024-04-12T05:09:04.891 INFO:tasks.workunit.client.0.smithi113.stdout:6/579: dwrite d8/de/d13/d11/d36/f61 [0,4194304] 0 2024-04-12T05:09:04.892 INFO:tasks.workunit.client.1.smithi192.stdout:2/564: mknod d1/d8/d17/d19/d40/caa 0 2024-04-12T05:09:04.893 INFO:tasks.workunit.client.0.smithi113.stdout:1/705: write d10/d2e/d31/d5c/f93 [643387,77404] 0 2024-04-12T05:09:04.893 INFO:tasks.workunit.client.0.smithi113.stdout:5/603: rename d0/d10/d6e/f83 to d0/d10/da1/fc3 0 2024-04-12T05:09:04.894 INFO:tasks.workunit.client.1.smithi192.stdout:1/553: symlink d0/d2/d1e/d3f/d56/lae 0 2024-04-12T05:09:04.896 INFO:tasks.workunit.client.0.smithi113.stdout:6/580: mkdir d8/de/dd2 0 2024-04-12T05:09:04.898 INFO:tasks.workunit.client.0.smithi113.stdout:5/604: stat d0/d10/d27/d1d/d38/d44/d4e 0 2024-04-12T05:09:04.898 INFO:tasks.workunit.client.1.smithi192.stdout:0/638: creat d0/d6/d15/d55/d2e/d66/fd7 x:0 0 0 2024-04-12T05:09:04.901 INFO:tasks.workunit.client.1.smithi192.stdout:4/639: mkdir d1/d1d/d53/d1c/d95/dc6/dd5 0 2024-04-12T05:09:04.903 INFO:tasks.workunit.client.0.smithi113.stdout:5/605: rename d0/d10/f9f to d0/d10/d27/d1d/d38/d44/d4e/daa/dc0/fc4 0 2024-04-12T05:09:04.904 INFO:tasks.workunit.client.1.smithi192.stdout:2/565: dwrite d1/d8/f22 [4194304,4194304] 0 2024-04-12T05:09:04.904 INFO:tasks.workunit.client.1.smithi192.stdout:0/639: write d0/d7b/d9b/d4b/d72/fd1 [701279,55807] 0 2024-04-12T05:09:04.904 INFO:tasks.workunit.client.1.smithi192.stdout:8/474: symlink d6/d10/d46/d5e/d5f/d31/d80/l93 0 2024-04-12T05:09:04.906 INFO:tasks.workunit.client.1.smithi192.stdout:5/542: truncate d0/d21/d5e/f6c 102046 0 2024-04-12T05:09:04.906 INFO:tasks.workunit.client.1.smithi192.stdout:2/566: dread - d1/f70 zero size 2024-04-12T05:09:04.908 INFO:tasks.workunit.client.1.smithi192.stdout:1/554: mknod d0/d29/d8d/d96/caf 0 2024-04-12T05:09:04.908 INFO:tasks.workunit.client.1.smithi192.stdout:8/475: readlink d6/d10/d46/d5e/d6e/l79 0 2024-04-12T05:09:04.913 INFO:tasks.workunit.client.1.smithi192.stdout:0/640: write d0/d7b/d9b/d4b/d72/fd1 [1553764,64695] 0 2024-04-12T05:09:04.913 INFO:tasks.workunit.client.1.smithi192.stdout:9/466: symlink d11/d15/d21/d2c/d1f/d6a/d7e/d89/l9e 0 2024-04-12T05:09:04.913 INFO:tasks.workunit.client.1.smithi192.stdout:4/640: rmdir d1/d54/d82/da4/da8 39 2024-04-12T05:09:04.913 INFO:tasks.workunit.client.1.smithi192.stdout:4/641: stat d1/d54/d55/d64 0 2024-04-12T05:09:04.914 INFO:tasks.workunit.client.1.smithi192.stdout:2/567: unlink d1/d8/c12 0 2024-04-12T05:09:04.915 INFO:tasks.workunit.client.0.smithi113.stdout:1/706: write d10/d2e/d7c/dc5/dc8/fab [692043,31912] 0 2024-04-12T05:09:04.916 INFO:tasks.workunit.client.1.smithi192.stdout:5/543: dread d0/d15/f79 [0,4194304] 0 2024-04-12T05:09:04.918 INFO:tasks.workunit.client.0.smithi113.stdout:1/707: write d10/d41/d89/d96/dd3/f64 [732594,925] 0 2024-04-12T05:09:04.918 INFO:tasks.workunit.client.1.smithi192.stdout:6/599: mknod d3/d8/d56/cc1 0 2024-04-12T05:09:04.918 INFO:tasks.workunit.client.1.smithi192.stdout:1/555: mknod d0/d2/dd/d3c/d5b/d74/d7d/cb0 0 2024-04-12T05:09:04.919 INFO:tasks.workunit.client.1.smithi192.stdout:0/641: rmdir d0/d7b/d9b/d4b/dbc/dce/d9f 39 2024-04-12T05:09:04.920 INFO:tasks.workunit.client.0.smithi113.stdout:6/581: dwrite d8/de/d13/fb [0,4194304] 0 2024-04-12T05:09:04.926 INFO:tasks.workunit.client.0.smithi113.stdout:1/708: rename d10/d21/l25 to d10/d14/le2 0 2024-04-12T05:09:04.927 INFO:tasks.workunit.client.0.smithi113.stdout:1/709: dread - d10/d14/d95/fe0 zero size 2024-04-12T05:09:04.928 INFO:tasks.workunit.client.0.smithi113.stdout:6/582: write d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8f [2020193,39002] 0 2024-04-12T05:09:04.929 INFO:tasks.workunit.client.1.smithi192.stdout:9/467: symlink d11/d15/d21/d2c/d1f/d6a/d92/l9f 0 2024-04-12T05:09:04.930 INFO:tasks.workunit.client.1.smithi192.stdout:7/479: dread d3/d7/d60/f66 [0,4194304] 0 2024-04-12T05:09:04.931 INFO:tasks.workunit.client.1.smithi192.stdout:2/568: symlink d1/da7/lab 0 2024-04-12T05:09:04.932 INFO:tasks.workunit.client.0.smithi113.stdout:6/583: creat d8/de/d13/d11/fd3 x:0 0 0 2024-04-12T05:09:04.935 INFO:tasks.workunit.client.1.smithi192.stdout:1/556: dwrite d0/d2/dd/fa9 [0,4194304] 0 2024-04-12T05:09:04.935 INFO:tasks.workunit.client.0.smithi113.stdout:6/584: creat d8/de/d13/d11/d36/d53/db7/fd4 x:0 0 0 2024-04-12T05:09:04.936 INFO:tasks.workunit.client.0.smithi113.stdout:6/585: stat d8/de/d13/c74 0 2024-04-12T05:09:04.937 INFO:tasks.workunit.client.1.smithi192.stdout:0/642: dwrite d0/f20 [0,4194304] 0 2024-04-12T05:09:04.940 INFO:tasks.workunit.client.0.smithi113.stdout:6/586: mkdir d8/de/dd5 0 2024-04-12T05:09:04.940 INFO:tasks.workunit.client.1.smithi192.stdout:7/480: creat d3/d7/d60/fa0 x:0 0 0 2024-04-12T05:09:04.942 INFO:tasks.workunit.client.1.smithi192.stdout:6/600: symlink d3/d8/d10/d4c/d68/lc2 0 2024-04-12T05:09:04.945 INFO:tasks.workunit.client.1.smithi192.stdout:8/476: dread d6/d21/f60 [0,4194304] 0 2024-04-12T05:09:04.947 INFO:tasks.workunit.client.1.smithi192.stdout:1/557: unlink d0/d2/dd/d21/d62/d84/fa3 0 2024-04-12T05:09:04.947 INFO:tasks.workunit.client.1.smithi192.stdout:1/558: chown d0 55571 1 2024-04-12T05:09:04.949 INFO:tasks.workunit.client.0.smithi113.stdout:6/587: getdents d8/de/d13/d11/d36 0 2024-04-12T05:09:04.951 INFO:tasks.workunit.client.1.smithi192.stdout:7/481: rename d3/d11/d68/f8e to d3/d1d/d4a/d5e/d6d/fa1 0 2024-04-12T05:09:04.952 INFO:tasks.workunit.client.0.smithi113.stdout:6/588: dread - d8/de/d13/d11/d36/d3d/d4c/d26/fc1 zero size 2024-04-12T05:09:04.955 INFO:tasks.workunit.client.0.smithi113.stdout:5/606: dread d0/d10/d27/d1d/d41/fa0 [0,4194304] 0 2024-04-12T05:09:04.956 INFO:tasks.workunit.client.1.smithi192.stdout:8/477: unlink d6/d10/d46/d5e/d5f/d37/f4a 0 2024-04-12T05:09:04.956 INFO:tasks.workunit.client.1.smithi192.stdout:4/642: write d1/d1d/d53/d1c/f36 [323671,96529] 0 2024-04-12T05:09:04.958 INFO:tasks.workunit.client.1.smithi192.stdout:1/559: fdatasync d0/d1b/f35 0 2024-04-12T05:09:04.958 INFO:tasks.workunit.client.0.smithi113.stdout:6/589: getdents d8/de/d13/d11/d36/d53/db8/d63/d66 0 2024-04-12T05:09:04.959 INFO:tasks.workunit.client.0.smithi113.stdout:5/607: mknod d0/d10/d27/d1d/d38/d44/d4e/daa/dbb/cc5 0 2024-04-12T05:09:04.960 INFO:tasks.workunit.client.1.smithi192.stdout:5/544: write d0/d15/d32/d45/f7f [1117693,121452] 0 2024-04-12T05:09:04.960 INFO:tasks.workunit.client.1.smithi192.stdout:6/601: mknod d3/d23/db5/cc3 0 2024-04-12T05:09:04.961 INFO:tasks.workunit.client.0.smithi113.stdout:6/590: dread - d8/de/d13/d11/d36/d89/f97 zero size 2024-04-12T05:09:04.962 INFO:tasks.workunit.client.0.smithi113.stdout:5/608: read d0/d10/d27/d1d/d38/d44/f48 [3302722,95961] 0 2024-04-12T05:09:04.964 INFO:tasks.workunit.client.1.smithi192.stdout:7/482: truncate d3/d7/d14/f92 487890 0 2024-04-12T05:09:04.965 INFO:tasks.workunit.client.0.smithi113.stdout:5/609: readlink d0/d10/d27/d1d/d38/d75/l81 0 2024-04-12T05:09:04.967 INFO:tasks.workunit.client.1.smithi192.stdout:7/483: write f1 [2923109,102820] 0 2024-04-12T05:09:04.967 INFO:tasks.workunit.client.1.smithi192.stdout:2/569: truncate d1/d2d/f6f 616247 0 2024-04-12T05:09:04.967 INFO:tasks.workunit.client.1.smithi192.stdout:4/643: dwrite d1/d49/f5b [0,4194304] 0 2024-04-12T05:09:04.968 INFO:tasks.workunit.client.0.smithi113.stdout:1/710: dread d10/d41/d89/d96/dd3/f64 [0,4194304] 0 2024-04-12T05:09:04.968 INFO:tasks.workunit.client.0.smithi113.stdout:6/591: link d8/de/d13/d11/d36/d53/db8/l7a d8/de/dd2/ld6 0 2024-04-12T05:09:04.970 INFO:tasks.workunit.client.0.smithi113.stdout:5/610: rename d0/d10/d27/d1d/d38/d44/d4e/daa/dc0 to d0/d10/d27/d1d/d8c/db8/dc6 0 2024-04-12T05:09:04.972 INFO:tasks.workunit.client.1.smithi192.stdout:0/643: getdents d0/d7b/d9b/d4b 0 2024-04-12T05:09:04.972 INFO:tasks.workunit.client.1.smithi192.stdout:5/545: readlink d0/d15/d24/d47/l68 0 2024-04-12T05:09:04.974 INFO:tasks.workunit.client.1.smithi192.stdout:7/484: write d3/d7/d15/f3c [4652800,42259] 0 2024-04-12T05:09:04.976 INFO:tasks.workunit.client.0.smithi113.stdout:6/592: rmdir d8/de/d13/d11/d36/d2c 39 2024-04-12T05:09:04.977 INFO:tasks.workunit.client.0.smithi113.stdout:5/611: creat d0/d10/d27/d1d/fc7 x:0 0 0 2024-04-12T05:09:04.978 INFO:tasks.workunit.client.1.smithi192.stdout:7/485: truncate d3/d1d/f82 895570 0 2024-04-12T05:09:04.978 INFO:tasks.workunit.client.1.smithi192.stdout:6/602: rename d3/d26/f91 to d3/dc/fc4 0 2024-04-12T05:09:04.980 INFO:tasks.workunit.client.1.smithi192.stdout:4/644: mknod d1/d1d/d53/d1c/d95/dc6/cd6 0 2024-04-12T05:09:04.981 INFO:tasks.workunit.client.0.smithi113.stdout:5/612: mkdir d0/d10/d27/d1d/dc8 0 2024-04-12T05:09:04.982 INFO:tasks.workunit.client.1.smithi192.stdout:5/546: mknod d0/d21/d5e/cbd 0 2024-04-12T05:09:04.983 INFO:tasks.workunit.client.1.smithi192.stdout:1/560: creat d0/d1b/fb1 x:0 0 0 2024-04-12T05:09:04.984 INFO:tasks.workunit.client.0.smithi113.stdout:5/613: mkdir d0/d10/d6e/d93/dc9 0 2024-04-12T05:09:04.984 INFO:tasks.workunit.client.1.smithi192.stdout:4/645: truncate d1/f6a 204735 0 2024-04-12T05:09:04.985 INFO:tasks.workunit.client.1.smithi192.stdout:6/603: mknod d3/d23/d67/cc5 0 2024-04-12T05:09:04.986 INFO:tasks.workunit.client.0.smithi113.stdout:6/593: mknod d8/de/d13/d11/d36/d3d/d4c/d26/d64/cd7 0 2024-04-12T05:09:04.987 INFO:tasks.workunit.client.1.smithi192.stdout:4/646: write d1/d54/d82/d71/fb0 [340453,25323] 0 2024-04-12T05:09:04.987 INFO:tasks.workunit.client.1.smithi192.stdout:6/604: chown d3/d21/d24/d79/fa0 14096 1 2024-04-12T05:09:04.989 INFO:tasks.workunit.client.1.smithi192.stdout:5/547: mkdir d0/d15/d64/d7c/db5/dbe 0 2024-04-12T05:09:04.990 INFO:tasks.workunit.client.1.smithi192.stdout:5/548: dread - d0/d21/fa9 zero size 2024-04-12T05:09:04.990 INFO:tasks.workunit.client.1.smithi192.stdout:2/570: getdents d1/da7 0 2024-04-12T05:09:04.991 INFO:tasks.workunit.client.0.smithi113.stdout:6/594: mkdir d8/de/d13/d11/d36/d3d/d4c/dd8 0 2024-04-12T05:09:04.992 INFO:tasks.workunit.client.1.smithi192.stdout:4/647: rmdir d1/d1d/d53/db2 39 2024-04-12T05:09:04.993 INFO:tasks.workunit.client.1.smithi192.stdout:1/561: mknod d0/d1b/cb2 0 2024-04-12T05:09:04.993 INFO:tasks.workunit.client.0.smithi113.stdout:5/614: unlink d0/d10/d27/d5f/l68 0 2024-04-12T05:09:04.994 INFO:tasks.workunit.client.1.smithi192.stdout:5/549: write d0/d21/f36 [2356784,20349] 0 2024-04-12T05:09:04.995 INFO:tasks.workunit.client.0.smithi113.stdout:6/595: chown d8/de/d13/d11/d36/d2c 27135534 1 2024-04-12T05:09:04.998 INFO:tasks.workunit.client.0.smithi113.stdout:5/615: truncate d0/d10/d27/d1d/d38/d44/d66/f96 5085596 0 2024-04-12T05:09:04.999 INFO:tasks.workunit.client.1.smithi192.stdout:2/571: creat d1/d2d/d73/fac x:0 0 0 2024-04-12T05:09:04.999 INFO:tasks.workunit.client.1.smithi192.stdout:1/562: creat d0/d1b/d44/fb3 x:0 0 0 2024-04-12T05:09:05.000 INFO:tasks.workunit.client.1.smithi192.stdout:5/550: readlink d0/d15/d64/d7c/da7/lbc 0 2024-04-12T05:09:05.001 INFO:tasks.workunit.client.1.smithi192.stdout:4/648: rename d1/d1d/f44 to d1/d6b/fd7 0 2024-04-12T05:09:05.002 INFO:tasks.workunit.client.0.smithi113.stdout:1/711: dwrite d10/d2e/d7c/dc5/dc8/dad/fc4 [0,4194304] 0 2024-04-12T05:09:05.004 INFO:tasks.workunit.client.0.smithi113.stdout:1/712: fdatasync d10/d14/d95/dcf/fd0 0 2024-04-12T05:09:05.007 INFO:tasks.workunit.client.1.smithi192.stdout:1/563: fdatasync d0/d1b/f35 0 2024-04-12T05:09:05.008 INFO:tasks.workunit.client.1.smithi192.stdout:8/478: dwrite d6/d10/d46/f85 [0,4194304] 0 2024-04-12T05:09:05.009 INFO:tasks.workunit.client.0.smithi113.stdout:1/713: creat d10/d2e/dd8/d70/fe3 x:0 0 0 2024-04-12T05:09:05.010 INFO:tasks.workunit.client.1.smithi192.stdout:8/479: fdatasync d6/d10/d46/d5e/d5f/d38/f8f 0 2024-04-12T05:09:05.010 INFO:tasks.workunit.client.0.smithi113.stdout:5/616: symlink d0/d10/d27/d1d/d8c/db8/lca 0 2024-04-12T05:09:05.011 INFO:tasks.workunit.client.0.smithi113.stdout:6/596: mkdir d8/de/d13/d11/d36/d53/db8/d63/d66/dc4/dd9 0 2024-04-12T05:09:05.011 INFO:tasks.workunit.client.0.smithi113.stdout:6/597: stat d8/de/l71 0 2024-04-12T05:09:05.014 INFO:tasks.workunit.client.1.smithi192.stdout:1/564: fsync d0/d2/dd/d21/d39/f82 0 2024-04-12T05:09:05.015 INFO:tasks.workunit.client.1.smithi192.stdout:5/551: dwrite d0/f96 [0,4194304] 0 2024-04-12T05:09:05.015 INFO:tasks.workunit.client.1.smithi192.stdout:4/649: mknod d1/d54/d82/da4/cd8 0 2024-04-12T05:09:05.015 INFO:tasks.workunit.client.0.smithi113.stdout:1/714: symlink d10/d2e/d31/d5c/daf/le4 0 2024-04-12T05:09:05.020 INFO:tasks.workunit.client.1.smithi192.stdout:1/565: dread d0/d2/dd/f4b [0,4194304] 0 2024-04-12T05:09:05.022 INFO:tasks.workunit.client.1.smithi192.stdout:1/566: write d0/d2/dd/d21/d33/fa6 [311109,77206] 0 2024-04-12T05:09:05.022 INFO:tasks.workunit.client.1.smithi192.stdout:8/480: fdatasync d6/d1f/f5c 0 2024-04-12T05:09:05.023 INFO:tasks.workunit.client.0.smithi113.stdout:1/715: truncate d10/d41/d5b/fb0 1415687 0 2024-04-12T05:09:05.024 INFO:tasks.workunit.client.1.smithi192.stdout:5/552: read d0/f11 [14679,120466] 0 2024-04-12T05:09:05.026 INFO:tasks.workunit.client.0.smithi113.stdout:6/598: rename d8/f12 to d8/fda 0 2024-04-12T05:09:05.026 INFO:tasks.workunit.client.1.smithi192.stdout:5/553: read - d0/d15/d3e/fb3 zero size 2024-04-12T05:09:05.029 INFO:tasks.workunit.client.1.smithi192.stdout:8/481: creat d6/d10/d46/d5e/d5f/d37/d4f/d4c/f94 x:0 0 0 2024-04-12T05:09:05.031 INFO:tasks.workunit.client.1.smithi192.stdout:4/650: mkdir d1/d54/d82/d71/d9e/dc4/dd9 0 2024-04-12T05:09:05.032 INFO:tasks.workunit.client.0.smithi113.stdout:6/599: creat d8/de/d13/d11/d36/d3d/d4c/fdb x:0 0 0 2024-04-12T05:09:05.035 INFO:tasks.workunit.client.1.smithi192.stdout:6/605: dread d3/d23/f74 [0,4194304] 0 2024-04-12T05:09:05.035 INFO:tasks.workunit.client.1.smithi192.stdout:4/651: write d1/d54/d82/f62 [6794181,90418] 0 2024-04-12T05:09:05.038 INFO:tasks.workunit.client.0.smithi113.stdout:5/617: truncate d0/f3a 3036666 0 2024-04-12T05:09:05.044 INFO:tasks.workunit.client.1.smithi192.stdout:4/652: symlink d1/d1d/d53/db2/lda 0 2024-04-12T05:09:05.044 INFO:tasks.workunit.client.1.smithi192.stdout:5/554: link d0/d15/d24/d29/c40 d0/d15/d32/cbf 0 2024-04-12T05:09:05.045 INFO:tasks.workunit.client.1.smithi192.stdout:8/482: getdents d6/d21 0 2024-04-12T05:09:05.046 INFO:tasks.workunit.client.1.smithi192.stdout:4/653: read - d1/d1d/fd2 zero size 2024-04-12T05:09:05.046 INFO:tasks.workunit.client.0.smithi113.stdout:6/600: dwrite d8/de/d13/d11/d36/d3d/d4c/d26/fc1 [0,4194304] 0 2024-04-12T05:09:05.047 INFO:tasks.workunit.client.1.smithi192.stdout:8/483: write d6/da/f3a [6146689,36983] 0 2024-04-12T05:09:05.049 INFO:tasks.workunit.client.1.smithi192.stdout:4/654: rmdir d1/d54/d82/d71/d9e/dc4 39 2024-04-12T05:09:05.051 INFO:tasks.workunit.client.0.smithi113.stdout:1/716: dwrite d10/d21/fac [0,4194304] 0 2024-04-12T05:09:05.053 INFO:tasks.workunit.client.1.smithi192.stdout:4/655: mkdir d1/d54/d82/d9c/ddb 0 2024-04-12T05:09:05.054 INFO:tasks.workunit.client.0.smithi113.stdout:6/601: creat d8/de/d13/d11/d36/d3d/d4c/d26/d5b/fdc x:0 0 0 2024-04-12T05:09:05.059 INFO:tasks.workunit.client.0.smithi113.stdout:1/717: read d10/d41/f4a [6022691,127803] 0 2024-04-12T05:09:05.059 INFO:tasks.workunit.client.0.smithi113.stdout:1/718: write d10/d2e/d31/d5c/fa7 [1636419,16687] 0 2024-04-12T05:09:05.059 INFO:tasks.workunit.client.0.smithi113.stdout:5/618: truncate d0/d10/d27/d1d/d3f/f47 1992988 0 2024-04-12T05:09:05.060 INFO:tasks.workunit.client.0.smithi113.stdout:1/719: write d10/d41/d5b/fa3 [282032,17222] 0 2024-04-12T05:09:05.060 INFO:tasks.workunit.client.1.smithi192.stdout:6/606: write d3/d8/d22/f2e [3591867,6084] 0 2024-04-12T05:09:05.065 INFO:tasks.workunit.client.1.smithi192.stdout:6/607: mknod d3/d8/d56/d59/d97/cc6 0 2024-04-12T05:09:05.065 INFO:tasks.workunit.client.0.smithi113.stdout:5/619: mkdir d0/d10/d27/d1d/d38/d44/dcb 0 2024-04-12T05:09:05.068 INFO:tasks.workunit.client.0.smithi113.stdout:6/602: dwrite d8/de/d13/d11/d36/d3d/d4c/d26/d5b/fb2 [0,4194304] 0 2024-04-12T05:09:05.069 INFO:tasks.workunit.client.0.smithi113.stdout:5/620: unlink d0/d10/d6e/fb4 0 2024-04-12T05:09:05.075 INFO:tasks.workunit.client.1.smithi192.stdout:5/555: dread d0/fa [0,4194304] 0 2024-04-12T05:09:05.076 INFO:tasks.workunit.client.0.smithi113.stdout:5/621: dwrite d0/d10/d27/d1d/d38/d44/fa3 [0,4194304] 0 2024-04-12T05:09:05.076 INFO:tasks.workunit.client.1.smithi192.stdout:1/567: dread d0/d2/dd/d21/d62/d84/f91 [0,4194304] 0 2024-04-12T05:09:05.078 INFO:tasks.workunit.client.1.smithi192.stdout:1/568: rename d0/d29 to d0/d29/d8d/db4 22 2024-04-12T05:09:05.078 INFO:tasks.workunit.client.1.smithi192.stdout:5/556: write d0/fa [3254794,6296] 0 2024-04-12T05:09:05.084 INFO:tasks.workunit.client.0.smithi113.stdout:5/622: dwrite d0/d10/d27/d1d/f49 [0,4194304] 0 2024-04-12T05:09:05.084 INFO:tasks.workunit.client.1.smithi192.stdout:1/569: creat d0/d2/dd/d3c/dab/fb5 x:0 0 0 2024-04-12T05:09:05.088 INFO:tasks.workunit.client.1.smithi192.stdout:4/656: dread d1/f26 [0,4194304] 0 2024-04-12T05:09:05.090 INFO:tasks.workunit.client.1.smithi192.stdout:4/657: chown d1/d54/d82 13299 1 2024-04-12T05:09:05.090 INFO:tasks.workunit.client.1.smithi192.stdout:5/557: dread d0/d15/d32/d45/f7f [0,4194304] 0 2024-04-12T05:09:05.092 INFO:tasks.workunit.client.1.smithi192.stdout:1/570: unlink d0/d2/dd/d21/d39/l59 0 2024-04-12T05:09:05.093 INFO:tasks.workunit.client.1.smithi192.stdout:5/558: mknod d0/d15/d32/d45/cc0 0 2024-04-12T05:09:05.095 INFO:tasks.workunit.client.0.smithi113.stdout:5/623: link d0/d10/d27/d1d/d38/d44/fbc d0/d10/d27/d1d/d38/d3b/fcc 0 2024-04-12T05:09:05.095 INFO:tasks.workunit.client.1.smithi192.stdout:8/484: dread d6/d1f/f5c [0,4194304] 0 2024-04-12T05:09:05.096 INFO:tasks.workunit.client.1.smithi192.stdout:5/559: symlink d0/lc1 0 2024-04-12T05:09:05.097 INFO:tasks.workunit.client.1.smithi192.stdout:1/571: mkdir d0/d2/dd/d3c/d5b/db6 0 2024-04-12T05:09:05.098 INFO:tasks.workunit.client.0.smithi113.stdout:5/624: mkdir d0/d10/d27/d1d/d3f/d73/dcd 0 2024-04-12T05:09:05.099 INFO:tasks.workunit.client.0.smithi113.stdout:5/625: readlink d0/d10/d27/d1d/d8c/la9 0 2024-04-12T05:09:05.100 INFO:tasks.workunit.client.0.smithi113.stdout:5/626: dread - d0/d10/d27/d1d/d3f/d73/fad zero size 2024-04-12T05:09:05.100 INFO:tasks.workunit.client.1.smithi192.stdout:5/560: mknod d0/d15/d24/d47/d7b/db9/cc2 0 2024-04-12T05:09:05.101 INFO:tasks.workunit.client.1.smithi192.stdout:5/561: fsync d0/d15/d24/d47/d7b/db9/fba 0 2024-04-12T05:09:05.102 INFO:tasks.workunit.client.1.smithi192.stdout:1/572: rename d0/f18 to d0/d1b/d44/d69/fb7 0 2024-04-12T05:09:05.102 INFO:tasks.workunit.client.0.smithi113.stdout:5/627: truncate d0/d10/d27/d1d/d38/d44/fbc 240549 0 2024-04-12T05:09:05.103 INFO:tasks.workunit.client.1.smithi192.stdout:1/573: chown d0/d2/d1e/d3f/d56/cad 47815 1 2024-04-12T05:09:05.103 INFO:tasks.workunit.client.0.smithi113.stdout:5/628: chown d0/d10/d27/d1d/d38/d44/d66/c70 5951 1 2024-04-12T05:09:05.103 INFO:tasks.workunit.client.1.smithi192.stdout:8/485: unlink d6/d10/d46/f92 0 2024-04-12T05:09:05.106 INFO:tasks.workunit.client.1.smithi192.stdout:5/562: creat d0/d15/fc3 x:0 0 0 2024-04-12T05:09:05.106 INFO:tasks.workunit.client.0.smithi113.stdout:5/629: rename d0/d10/d27/d1d/dc8 to d0/d10/d27/d1d/dce 0 2024-04-12T05:09:05.108 INFO:tasks.workunit.client.0.smithi113.stdout:5/630: symlink d0/d10/d27/d5f/lcf 0 2024-04-12T05:09:05.109 INFO:tasks.workunit.client.0.smithi113.stdout:5/631: stat d0/d10/d27/d1d/d38/d3b/fcc 0 2024-04-12T05:09:05.111 INFO:tasks.workunit.client.0.smithi113.stdout:5/632: mknod d0/d10/d27/d1d/d38/d44/d66/cd0 0 2024-04-12T05:09:05.114 INFO:tasks.workunit.client.0.smithi113.stdout:5/633: mknod d0/d10/d27/d1d/d38/cd1 0 2024-04-12T05:09:05.119 INFO:tasks.workunit.client.0.smithi113.stdout:5/634: unlink d0/d10/d27/d1d/fac 0 2024-04-12T05:09:05.125 INFO:tasks.workunit.client.1.smithi192.stdout:4/658: truncate d1/d54/d55/d64/fae 116985 0 2024-04-12T05:09:05.126 INFO:tasks.workunit.client.1.smithi192.stdout:4/659: chown d1/d1d/d53/d1c/f66 43714 1 2024-04-12T05:09:05.127 INFO:tasks.workunit.client.0.smithi113.stdout:1/720: dwrite d10/d41/d89/d96/f91 [0,4194304] 0 2024-04-12T05:09:05.130 INFO:tasks.workunit.client.1.smithi192.stdout:4/660: mkdir d1/d54/d82/d9c/ddb/ddc 0 2024-04-12T05:09:05.135 INFO:tasks.workunit.client.0.smithi113.stdout:1/721: creat d10/d55/d9c/fe5 x:0 0 0 2024-04-12T05:09:05.137 INFO:tasks.workunit.client.1.smithi192.stdout:4/661: mkdir d1/d54/d55/d64/d68/ddd 0 2024-04-12T05:09:05.138 INFO:tasks.workunit.client.1.smithi192.stdout:5/563: rmdir d0/d15/d32 39 2024-04-12T05:09:05.139 INFO:tasks.workunit.client.0.smithi113.stdout:1/722: symlink d10/d55/le6 0 2024-04-12T05:09:05.142 INFO:tasks.workunit.client.0.smithi113.stdout:5/635: dwrite d0/d10/d27/d1d/f2a [0,4194304] 0 2024-04-12T05:09:05.143 INFO:tasks.workunit.client.0.smithi113.stdout:1/723: rename d10/d21/fac to d10/ddc/fe7 0 2024-04-12T05:09:05.145 INFO:tasks.workunit.client.0.smithi113.stdout:5/636: mknod d0/d10/da1/cd2 0 2024-04-12T05:09:05.145 INFO:tasks.workunit.client.1.smithi192.stdout:4/662: dwrite d1/d1d/fd2 [0,4194304] 0 2024-04-12T05:09:05.149 INFO:tasks.workunit.client.1.smithi192.stdout:5/564: truncate d0/d15/d32/d69/f9e 2905924 0 2024-04-12T05:09:05.150 INFO:tasks.workunit.client.1.smithi192.stdout:4/663: creat d1/d1d/d53/d1c/fde x:0 0 0 2024-04-12T05:09:05.150 INFO:tasks.workunit.client.0.smithi113.stdout:1/724: mknod d10/d41/ce8 0 2024-04-12T05:09:05.151 INFO:tasks.workunit.client.0.smithi113.stdout:5/637: link d0/d10/d6e/d93/cc2 d0/d10/d27/d1d/dce/cd3 0 2024-04-12T05:09:05.151 INFO:tasks.workunit.client.0.smithi113.stdout:1/725: stat d10/d2e/dd8/l86 0 2024-04-12T05:09:05.157 INFO:tasks.workunit.client.1.smithi192.stdout:5/565: creat d0/d15/d24/d47/d7b/fc4 x:0 0 0 2024-04-12T05:09:05.157 INFO:tasks.workunit.client.1.smithi192.stdout:4/664: symlink d1/d54/d82/d71/d9e/dc4/ldf 0 2024-04-12T05:09:05.158 INFO:tasks.workunit.client.0.smithi113.stdout:1/726: dread d10/d2e/d31/f53 [0,4194304] 0 2024-04-12T05:09:05.158 INFO:tasks.workunit.client.0.smithi113.stdout:5/638: rename d0/f98 to d0/d10/d27/d1d/d38/d3b/d6b/fd4 0 2024-04-12T05:09:05.161 INFO:tasks.workunit.client.0.smithi113.stdout:5/639: stat d0/d10/d27/d1d/d38/c9c 0 2024-04-12T05:09:05.162 INFO:tasks.workunit.client.1.smithi192.stdout:4/665: rmdir d1/d1d/d53/d1c/d95 39 2024-04-12T05:09:05.162 INFO:tasks.workunit.client.1.smithi192.stdout:4/666: chown d1/d1d/d53/f52 115 1 2024-04-12T05:09:05.163 INFO:tasks.workunit.client.1.smithi192.stdout:5/566: mknod d0/d15/d32/db2/cc5 0 2024-04-12T05:09:05.164 INFO:tasks.workunit.client.1.smithi192.stdout:5/567: fdatasync d0/d15/d24/d29/f95 0 2024-04-12T05:09:05.168 INFO:tasks.workunit.client.1.smithi192.stdout:5/568: rename d0/d15/d32/d45/f86 to d0/d15/d32/d45/fc6 0 2024-04-12T05:09:05.170 INFO:tasks.workunit.client.1.smithi192.stdout:5/569: write d0/d15/d24/d47/f54 [724891,41880] 0 2024-04-12T05:09:05.170 INFO:tasks.workunit.client.0.smithi113.stdout:1/727: dwrite d10/d41/fd9 [0,4194304] 0 2024-04-12T05:09:05.170 INFO:tasks.workunit.client.1.smithi192.stdout:4/667: rename d1/d54/c59 to d1/d1d/ce0 0 2024-04-12T05:09:05.171 INFO:tasks.workunit.client.1.smithi192.stdout:3/530: sync 2024-04-12T05:09:05.179 INFO:tasks.workunit.client.1.smithi192.stdout:4/668: mkdir d1/d1d/d53/d1c/de1 0 2024-04-12T05:09:05.181 INFO:tasks.workunit.client.1.smithi192.stdout:5/570: dread d0/d15/d64/fad [0,4194304] 0 2024-04-12T05:09:05.182 INFO:tasks.workunit.client.1.smithi192.stdout:3/531: truncate d5/d13/d3a/d4e/f6c 525818 0 2024-04-12T05:09:05.183 INFO:tasks.workunit.client.0.smithi113.stdout:1/728: rename d10/d2e/dd8/d70/l6c to d10/d21/le9 0 2024-04-12T05:09:05.185 INFO:tasks.workunit.client.0.smithi113.stdout:1/729: truncate d10/d55/d9c/fe5 540736 0 2024-04-12T05:09:05.185 INFO:tasks.workunit.client.1.smithi192.stdout:3/532: write d5/d13/d3a/d4e/f9a [175896,92390] 0 2024-04-12T05:09:05.186 INFO:tasks.workunit.client.0.smithi113.stdout:5/640: dread d0/d10/d27/d1d/d3f/d73/d89/f23 [0,4194304] 0 2024-04-12T05:09:05.187 INFO:tasks.workunit.client.0.smithi113.stdout:5/641: stat d0/d10/d27/d1d/d38/d75/l81 0 2024-04-12T05:09:05.189 INFO:tasks.workunit.client.1.smithi192.stdout:4/669: dwrite d1/d54/d55/d64/fd4 [0,4194304] 0 2024-04-12T05:09:05.189 INFO:tasks.workunit.client.1.smithi192.stdout:3/533: fdatasync d5/d13/d3a/d4a/f98 0 2024-04-12T05:09:05.194 INFO:tasks.workunit.client.0.smithi113.stdout:1/730: dwrite d10/d41/d89/d96/d7d/f79 [4194304,4194304] 0 2024-04-12T05:09:05.194 INFO:tasks.workunit.client.0.smithi113.stdout:5/642: creat d0/d10/d27/d1d/d8c/db8/fd5 x:0 0 0 2024-04-12T05:09:05.194 INFO:tasks.workunit.client.0.smithi113.stdout:2/482: sync 2024-04-12T05:09:05.195 INFO:tasks.workunit.client.1.smithi192.stdout:4/670: unlink d1/d1d/d53/la7 0 2024-04-12T05:09:05.196 INFO:tasks.workunit.client.0.smithi113.stdout:2/483: write d8/fd [4814682,92391] 0 2024-04-12T05:09:05.197 INFO:tasks.workunit.client.0.smithi113.stdout:2/484: stat d8/d9/de/d15 0 2024-04-12T05:09:05.198 INFO:tasks.workunit.client.0.smithi113.stdout:1/731: stat d10/c46 0 2024-04-12T05:09:05.199 INFO:tasks.workunit.client.1.smithi192.stdout:5/571: dwrite d0/d15/d3e/f9d [0,4194304] 0 2024-04-12T05:09:05.202 INFO:tasks.workunit.client.1.smithi192.stdout:3/534: rename d5/d13/d3a/d4e/c76 to d5/d13/d3a/d8b/da2/da7/cbe 0 2024-04-12T05:09:05.203 INFO:tasks.workunit.client.0.smithi113.stdout:2/485: write d8/d9/de/d12/d2b/f8e [243284,122469] 0 2024-04-12T05:09:05.206 INFO:tasks.workunit.client.0.smithi113.stdout:1/732: link d10/da5/lb4 d10/d2e/d7c/dc5/dc8/de1/lea 0 2024-04-12T05:09:05.207 INFO:tasks.workunit.client.1.smithi192.stdout:4/671: dwrite d1/d54/d82/f48 [0,4194304] 0 2024-04-12T05:09:05.208 INFO:tasks.workunit.client.0.smithi113.stdout:2/486: dread d8/d9/d1a/f26 [0,4194304] 0 2024-04-12T05:09:05.208 INFO:tasks.workunit.client.0.smithi113.stdout:1/733: symlink d10/d14/d95/leb 0 2024-04-12T05:09:05.210 INFO:tasks.workunit.client.0.smithi113.stdout:1/734: readlink d10/d55/l9d 0 2024-04-12T05:09:05.212 INFO:tasks.workunit.client.1.smithi192.stdout:4/672: rmdir d1/d54/d82/d9c/ddb 39 2024-04-12T05:09:05.213 INFO:tasks.workunit.client.0.smithi113.stdout:1/735: mknod d10/da5/cec 0 2024-04-12T05:09:05.216 INFO:tasks.workunit.client.0.smithi113.stdout:2/487: rename d8/d9/de/l2a to d8/d9/d27/d88/l90 0 2024-04-12T05:09:05.216 INFO:tasks.workunit.client.0.smithi113.stdout:1/736: readlink d10/d41/d5b/l6b 0 2024-04-12T05:09:05.218 INFO:tasks.workunit.client.1.smithi192.stdout:4/673: rename d1/f18 to d1/d1d/d53/d1c/d84/fe2 0 2024-04-12T05:09:05.219 INFO:tasks.workunit.client.0.smithi113.stdout:5/643: dwrite d0/d10/d27/d1d/d38/d3b/d6b/fd4 [0,4194304] 0 2024-04-12T05:09:05.222 INFO:tasks.workunit.client.0.smithi113.stdout:1/737: creat d10/d2e/d31/dc3/fed x:0 0 0 2024-04-12T05:09:05.224 INFO:tasks.workunit.client.0.smithi113.stdout:2/488: getdents d8/d9/de/d12/d3c 0 2024-04-12T05:09:05.225 INFO:tasks.workunit.client.1.smithi192.stdout:4/674: creat d1/d54/d82/da4/fe3 x:0 0 0 2024-04-12T05:09:05.226 INFO:tasks.workunit.client.0.smithi113.stdout:5/644: creat d0/d10/d27/d1d/d3f/db0/fd6 x:0 0 0 2024-04-12T05:09:05.228 INFO:tasks.workunit.client.0.smithi113.stdout:2/489: mkdir d8/d9/d27/d88/d91 0 2024-04-12T05:09:05.228 INFO:tasks.workunit.client.0.smithi113.stdout:1/738: creat d10/d55/fee x:0 0 0 2024-04-12T05:09:05.230 INFO:tasks.workunit.client.0.smithi113.stdout:5/645: read d0/d10/d27/d1d/d3f/d73/d89/f62 [2637544,49592] 0 2024-04-12T05:09:05.230 INFO:tasks.workunit.client.0.smithi113.stdout:1/739: rmdir d10/d2e/d7c 39 2024-04-12T05:09:05.231 INFO:tasks.workunit.client.0.smithi113.stdout:1/740: readlink d10/d2e/l4b 0 2024-04-12T05:09:05.233 INFO:tasks.workunit.client.1.smithi192.stdout:5/572: dwrite d0/f6e [0,4194304] 0 2024-04-12T05:09:05.234 INFO:tasks.workunit.client.0.smithi113.stdout:5/646: rmdir d0/d10/d27/d1d/dce 39 2024-04-12T05:09:05.235 INFO:tasks.workunit.client.0.smithi113.stdout:5/647: stat d0/d10/da1/fa5 0 2024-04-12T05:09:05.235 INFO:tasks.workunit.client.1.smithi192.stdout:3/535: dread d5/d13/f2a [0,4194304] 0 2024-04-12T05:09:05.236 INFO:tasks.workunit.client.0.smithi113.stdout:1/741: rename d10/d2e/l73 to d10/d2e/d31/dc3/lef 0 2024-04-12T05:09:05.237 INFO:tasks.workunit.client.1.smithi192.stdout:3/536: write d5/d13/d3c/f91 [319379,96359] 0 2024-04-12T05:09:05.237 INFO:tasks.workunit.client.1.smithi192.stdout:5/573: read d0/f61 [783510,31875] 0 2024-04-12T05:09:05.237 INFO:tasks.workunit.client.0.smithi113.stdout:2/490: dwrite d8/d9/f69 [0,4194304] 0 2024-04-12T05:09:05.240 INFO:tasks.workunit.client.1.smithi192.stdout:3/537: symlink d5/d13/d3a/d4a/lbf 0 2024-04-12T05:09:05.242 INFO:tasks.workunit.client.1.smithi192.stdout:5/574: rmdir d0/d15/d3e/d77 39 2024-04-12T05:09:05.243 INFO:tasks.workunit.client.0.smithi113.stdout:2/491: fsync d8/d16/f19 0 2024-04-12T05:09:05.243 INFO:tasks.workunit.client.0.smithi113.stdout:2/492: stat d8/d9/d1a/c60 0 2024-04-12T05:09:05.246 INFO:tasks.workunit.client.1.smithi192.stdout:4/675: truncate d1/d1d/d53/d1c/fb1 123108 0 2024-04-12T05:09:05.247 INFO:tasks.workunit.client.1.smithi192.stdout:5/575: read - d0/d15/f41 zero size 2024-04-12T05:09:05.251 INFO:tasks.workunit.client.0.smithi113.stdout:2/493: link d8/d9/de/d12/d2b/f54 d8/d9/f92 0 2024-04-12T05:09:05.254 INFO:tasks.workunit.client.0.smithi113.stdout:1/742: dread d10/d41/d89/d96/dd3/f7a [0,4194304] 0 2024-04-12T05:09:05.255 INFO:tasks.workunit.client.1.smithi192.stdout:3/538: dwrite d5/f14 [0,4194304] 0 2024-04-12T05:09:05.257 INFO:tasks.workunit.client.0.smithi113.stdout:1/743: creat d10/d41/d89/d96/d7d/ff0 x:0 0 0 2024-04-12T05:09:05.260 INFO:tasks.workunit.client.0.smithi113.stdout:1/744: mkdir d10/d2e/dd8/df1 0 2024-04-12T05:09:05.262 INFO:tasks.workunit.client.0.smithi113.stdout:1/745: mkdir d10/d41/d89/df2 0 2024-04-12T05:09:05.263 INFO:tasks.workunit.client.0.smithi113.stdout:1/746: chown d10/d41/f4a 1467671820 1 2024-04-12T05:09:05.264 INFO:tasks.workunit.client.1.smithi192.stdout:5/576: link d0/d15/d3e/c8f d0/d15/d32/cc7 0 2024-04-12T05:09:05.265 INFO:tasks.workunit.client.1.smithi192.stdout:3/539: link d5/d13/d3c/d26/c2c d5/d13/d3c/d36/da0/cc0 0 2024-04-12T05:09:05.268 INFO:tasks.workunit.client.1.smithi192.stdout:4/676: truncate d1/d54/f99 141635 0 2024-04-12T05:09:05.269 INFO:tasks.workunit.client.1.smithi192.stdout:5/577: mkdir d0/d15/d32/d69/dc8 0 2024-04-12T05:09:05.272 INFO:tasks.workunit.client.1.smithi192.stdout:5/578: creat d0/d15/d32/d69/d7d/fc9 x:0 0 0 2024-04-12T05:09:05.273 INFO:tasks.workunit.client.1.smithi192.stdout:3/540: dwrite d5/f1c [0,4194304] 0 2024-04-12T05:09:05.275 INFO:tasks.workunit.client.0.smithi113.stdout:2/494: dread d8/d9/f92 [0,4194304] 0 2024-04-12T05:09:05.275 INFO:tasks.workunit.client.1.smithi192.stdout:3/541: write d5/d13/d3a/d4a/f66 [3686794,49939] 0 2024-04-12T05:09:05.278 INFO:tasks.workunit.client.0.smithi113.stdout:2/495: mknod d8/d9/de/c93 0 2024-04-12T05:09:05.278 INFO:tasks.workunit.client.0.smithi113.stdout:1/747: write d10/d55/f8b [1923680,129009] 0 2024-04-12T05:09:05.283 INFO:tasks.workunit.client.1.smithi192.stdout:5/579: rename d0/d15/d24/d47/d7b to d0/d15/d3e/dca 0 2024-04-12T05:09:05.283 INFO:tasks.workunit.client.0.smithi113.stdout:2/496: creat d8/d9/d27/d3f/f94 x:0 0 0 2024-04-12T05:09:05.285 INFO:tasks.workunit.client.0.smithi113.stdout:1/748: unlink c9 0 2024-04-12T05:09:05.285 INFO:tasks.workunit.client.0.smithi113.stdout:2/497: creat d8/d9/d27/d88/f95 x:0 0 0 2024-04-12T05:09:05.289 INFO:tasks.workunit.client.1.smithi192.stdout:3/542: rename d5/d13/d3a/l40 to d5/d13/d3c/d36/lc1 0 2024-04-12T05:09:05.289 INFO:tasks.workunit.client.0.smithi113.stdout:2/498: unlink d8/d9/de/d12/d2b/f5a 0 2024-04-12T05:09:05.290 INFO:tasks.workunit.client.0.smithi113.stdout:1/749: fdatasync d10/d41/d89/d96/dd3/f64 0 2024-04-12T05:09:05.292 INFO:tasks.workunit.client.1.smithi192.stdout:3/543: creat d5/d13/d3a/d8b/fc2 x:0 0 0 2024-04-12T05:09:05.292 INFO:tasks.workunit.client.1.smithi192.stdout:3/544: fdatasync d5/f2d 0 2024-04-12T05:09:05.293 INFO:tasks.workunit.client.1.smithi192.stdout:3/545: fsync d5/f55 0 2024-04-12T05:09:05.297 INFO:tasks.workunit.client.0.smithi113.stdout:1/750: mknod d10/d2e/dd8/df1/cf3 0 2024-04-12T05:09:05.300 INFO:tasks.workunit.client.1.smithi192.stdout:5/580: write d0/f13 [4506714,105986] 0 2024-04-12T05:09:05.302 INFO:tasks.workunit.client.1.smithi192.stdout:7/486: sync 2024-04-12T05:09:05.302 INFO:tasks.workunit.client.1.smithi192.stdout:9/468: sync 2024-04-12T05:09:05.302 INFO:tasks.workunit.client.1.smithi192.stdout:2/572: sync 2024-04-12T05:09:05.302 INFO:tasks.workunit.client.1.smithi192.stdout:0/644: sync 2024-04-12T05:09:05.302 INFO:tasks.workunit.client.1.smithi192.stdout:5/581: mknod d0/d15/d3e/dca/ccb 0 2024-04-12T05:09:05.304 INFO:tasks.workunit.client.0.smithi113.stdout:1/751: dwrite d10/d2e/d31/dc3/fed [0,4194304] 0 2024-04-12T05:09:05.305 INFO:tasks.workunit.client.1.smithi192.stdout:9/469: truncate d11/d25/f91 398670 0 2024-04-12T05:09:05.305 INFO:tasks.workunit.client.1.smithi192.stdout:0/645: write d0/d7b/d7c/dcc/f93 [582220,94952] 0 2024-04-12T05:09:05.307 INFO:tasks.workunit.client.1.smithi192.stdout:5/582: creat d0/d15/d32/d69/d7d/fcc x:0 0 0 2024-04-12T05:09:05.308 INFO:tasks.workunit.client.1.smithi192.stdout:0/646: write d0/d6/d15/d1b/d46/f4d [2665954,63022] 0 2024-04-12T05:09:05.308 INFO:tasks.workunit.client.1.smithi192.stdout:5/583: stat d0/d21/d5e/lbb 0 2024-04-12T05:09:05.311 INFO:tasks.workunit.client.0.smithi113.stdout:2/499: dread d8/d9/d27/f28 [0,4194304] 0 2024-04-12T05:09:05.314 INFO:tasks.workunit.client.1.smithi192.stdout:7/487: rename d3/d7/f13 to d3/d7/d60/d65/fa2 0 2024-04-12T05:09:05.316 INFO:tasks.workunit.client.1.smithi192.stdout:7/488: write d3/d11/f5c [682215,53723] 0 2024-04-12T05:09:05.317 INFO:tasks.workunit.client.1.smithi192.stdout:7/489: write d3/d7/f8 [5421451,60127] 0 2024-04-12T05:09:05.319 INFO:tasks.workunit.client.1.smithi192.stdout:9/470: dread d11/d25/d36/f81 [0,4194304] 0 2024-04-12T05:09:05.320 INFO:tasks.workunit.client.1.smithi192.stdout:2/573: link d1/d8/l69 d1/d8/d17/d19/d60/d9c/lad 0 2024-04-12T05:09:05.321 INFO:tasks.workunit.client.1.smithi192.stdout:0/647: creat d0/d6/d15/d55/fd8 x:0 0 0 2024-04-12T05:09:05.322 INFO:tasks.workunit.client.1.smithi192.stdout:7/490: mkdir d3/d1d/d4a/d5e/d6d/da3 0 2024-04-12T05:09:05.324 INFO:tasks.workunit.client.0.smithi113.stdout:2/500: dread d8/d9/d1a/f50 [0,4194304] 0 2024-04-12T05:09:05.327 INFO:tasks.workunit.client.0.smithi113.stdout:2/501: creat d8/d9/de/d15/d24/d58/f96 x:0 0 0 2024-04-12T05:09:05.327 INFO:tasks.workunit.client.1.smithi192.stdout:9/471: creat d11/d15/d21/d2c/d1f/d94/fa0 x:0 0 0 2024-04-12T05:09:05.327 INFO:tasks.workunit.client.1.smithi192.stdout:5/584: truncate d0/d15/d24/f4b 148733 0 2024-04-12T05:09:05.328 INFO:tasks.workunit.client.1.smithi192.stdout:9/472: stat d11/d15/c44 0 2024-04-12T05:09:05.328 INFO:tasks.workunit.client.1.smithi192.stdout:2/574: dwrite d1/d8/d17/d19/d60/f76 [0,4194304] 0 2024-04-12T05:09:05.329 INFO:tasks.workunit.client.1.smithi192.stdout:7/491: creat d3/d1d/d5a/d5f/fa4 x:0 0 0 2024-04-12T05:09:05.331 INFO:tasks.workunit.client.0.smithi113.stdout:2/502: creat d8/d9/d27/d88/d91/f97 x:0 0 0 2024-04-12T05:09:05.331 INFO:tasks.workunit.client.1.smithi192.stdout:2/575: write d1/d8/d17/d19/d60/fa6 [629732,10416] 0 2024-04-12T05:09:05.332 INFO:tasks.workunit.client.1.smithi192.stdout:5/585: fdatasync d0/d15/d24/d47/f4d 0 2024-04-12T05:09:05.333 INFO:tasks.workunit.client.1.smithi192.stdout:0/648: write d0/d6/d15/f23 [2002204,16233] 0 2024-04-12T05:09:05.333 INFO:tasks.workunit.client.1.smithi192.stdout:5/586: write d0/f1b [1076171,4494] 0 2024-04-12T05:09:05.335 INFO:tasks.workunit.client.1.smithi192.stdout:9/473: creat d11/d25/d36/d3d/fa1 x:0 0 0 2024-04-12T05:09:05.336 INFO:tasks.workunit.client.1.smithi192.stdout:7/492: creat d3/d1d/d4a/d5e/fa5 x:0 0 0 2024-04-12T05:09:05.336 INFO:tasks.workunit.client.1.smithi192.stdout:9/474: read f10 [3839060,38037] 0 2024-04-12T05:09:05.338 INFO:tasks.workunit.client.1.smithi192.stdout:9/475: dread - d11/d15/d21/d2c/d1f/f67 zero size 2024-04-12T05:09:05.338 INFO:tasks.workunit.client.1.smithi192.stdout:0/649: creat d0/d6/dc4/fd9 x:0 0 0 2024-04-12T05:09:05.340 INFO:tasks.workunit.client.1.smithi192.stdout:5/587: dread d0/d15/d3e/f9d [0,4194304] 0 2024-04-12T05:09:05.344 INFO:tasks.workunit.client.0.smithi113.stdout:2/503: dwrite d8/f1e [0,4194304] 0 2024-04-12T05:09:05.345 INFO:tasks.workunit.client.1.smithi192.stdout:0/650: chown d0/d7b/d9b/d4b/dbc/dce/d9f/fa5 1617645610 1 2024-04-12T05:09:05.349 INFO:tasks.workunit.client.1.smithi192.stdout:7/493: mkdir d3/d1d/d5a/d5f/d7a/da6 0 2024-04-12T05:09:05.352 INFO:tasks.workunit.client.1.smithi192.stdout:7/494: write d3/d7/d15/f7d [3097123,102818] 0 2024-04-12T05:09:05.354 INFO:tasks.workunit.client.1.smithi192.stdout:5/588: dwrite d0/d21/f2f [0,4194304] 0 2024-04-12T05:09:05.356 INFO:tasks.workunit.client.0.smithi113.stdout:2/504: rename d8/d9/de/c74 to d8/d9/de/d12/c98 0 2024-04-12T05:09:05.357 INFO:tasks.workunit.client.0.smithi113.stdout:2/505: dread - d8/d9/de/d15/d38/f82 zero size 2024-04-12T05:09:05.358 INFO:tasks.workunit.client.1.smithi192.stdout:7/495: write d3/d7/f56 [7676558,122772] 0 2024-04-12T05:09:05.358 INFO:tasks.workunit.client.0.smithi113.stdout:2/506: write d8/d9/d27/d88/f8b [939147,16356] 0 2024-04-12T05:09:05.362 INFO:tasks.workunit.client.1.smithi192.stdout:0/651: getdents d0/d6/d67/db1 0 2024-04-12T05:09:05.365 INFO:tasks.workunit.client.1.smithi192.stdout:7/496: truncate d3/d7/d14/f58 641385 0 2024-04-12T05:09:05.365 INFO:tasks.workunit.client.1.smithi192.stdout:0/652: creat d0/d6/d15/d1b/d3a/d4f/fda x:0 0 0 2024-04-12T05:09:05.366 INFO:tasks.workunit.client.1.smithi192.stdout:7/497: read - d3/d7/d60/fa0 zero size 2024-04-12T05:09:05.372 INFO:tasks.workunit.client.1.smithi192.stdout:2/576: dread d1/d8/d3a/f95 [0,4194304] 0 2024-04-12T05:09:05.379 INFO:tasks.workunit.client.1.smithi192.stdout:2/577: truncate d1/d8/d17/d19/d40/f4c 172011 0 2024-04-12T05:09:05.384 INFO:tasks.workunit.client.1.smithi192.stdout:0/653: dread d0/d7b/d9b/d4b/dbc/dce/f65 [0,4194304] 0 2024-04-12T05:09:05.385 INFO:tasks.workunit.client.0.smithi113.stdout:2/507: creat d8/d9/de/d15/d38/f99 x:0 0 0 2024-04-12T05:09:05.389 INFO:tasks.workunit.client.1.smithi192.stdout:0/654: creat d0/d7b/d7c/fdb x:0 0 0 2024-04-12T05:09:05.389 INFO:tasks.workunit.client.1.smithi192.stdout:2/578: truncate d1/d8/d17/d19/f45 4759073 0 2024-04-12T05:09:05.391 INFO:tasks.workunit.client.1.smithi192.stdout:7/498: dread d3/d11/f16 [0,4194304] 0 2024-04-12T05:09:05.392 INFO:tasks.workunit.client.0.smithi113.stdout:2/508: link d8/d9/de/d15/d24/c6f d8/d9/d1a/c9a 0 2024-04-12T05:09:05.392 INFO:tasks.workunit.client.0.smithi113.stdout:2/509: stat d8/d9/d27/d3f/f94 0 2024-04-12T05:09:05.395 INFO:tasks.workunit.client.1.smithi192.stdout:7/499: creat d3/d7/d14/d1f/d76/fa7 x:0 0 0 2024-04-12T05:09:05.397 INFO:tasks.workunit.client.0.smithi113.stdout:2/510: dread d8/d9/de/f22 [0,4194304] 0 2024-04-12T05:09:05.397 INFO:tasks.workunit.client.1.smithi192.stdout:7/500: fdatasync d3/d7/d14/f58 0 2024-04-12T05:09:05.399 INFO:tasks.workunit.client.1.smithi192.stdout:0/655: dwrite d0/d7b/d7c/dcc/f97 [0,4194304] 0 2024-04-12T05:09:05.401 INFO:tasks.workunit.client.1.smithi192.stdout:2/579: dwrite d1/d8/f10 [0,4194304] 0 2024-04-12T05:09:05.402 INFO:tasks.workunit.client.0.smithi113.stdout:2/511: link d8/d9/de/c63 d8/d9/de/d12/d2b/c9b 0 2024-04-12T05:09:05.403 INFO:tasks.workunit.client.0.smithi113.stdout:2/512: write d8/d9/de/f67 [3164276,48678] 0 2024-04-12T05:09:05.404 INFO:tasks.workunit.client.1.smithi192.stdout:0/656: write d0/d7b/d7c/fdb [942987,2535] 0 2024-04-12T05:09:05.406 INFO:tasks.workunit.client.1.smithi192.stdout:7/501: creat d3/d1d/d4a/d5e/d6d/fa8 x:0 0 0 2024-04-12T05:09:05.414 INFO:tasks.workunit.client.0.smithi113.stdout:2/513: creat d8/d9/de/d12/f9c x:0 0 0 2024-04-12T05:09:05.415 INFO:tasks.workunit.client.1.smithi192.stdout:0/657: getdents d0/d7b/d9b/d4b/dbc/dce/d53/d6d 0 2024-04-12T05:09:05.418 INFO:tasks.workunit.client.1.smithi192.stdout:2/580: dwrite d1/d8/d3a/f26 [0,4194304] 0 2024-04-12T05:09:05.420 INFO:tasks.workunit.client.1.smithi192.stdout:7/502: dwrite d3/d7/d15/f3c [4194304,4194304] 0 2024-04-12T05:09:05.424 INFO:tasks.workunit.client.1.smithi192.stdout:0/658: symlink d0/d7b/d9b/d4b/dbc/dce/d95/db4/ldc 0 2024-04-12T05:09:05.428 INFO:tasks.workunit.client.1.smithi192.stdout:7/503: rename d3/f12 to d3/d11/d68/fa9 0 2024-04-12T05:09:05.431 INFO:tasks.workunit.client.1.smithi192.stdout:2/581: dwrite d1/d8/d17/d19/d60/f76 [0,4194304] 0 2024-04-12T05:09:05.431 INFO:tasks.workunit.client.1.smithi192.stdout:0/659: rename d0/d6/d15/d1b/d46/fbe to d0/d6/d15/d55/d29/fdd 0 2024-04-12T05:09:05.433 INFO:tasks.workunit.client.1.smithi192.stdout:2/582: chown d1/c88 121 1 2024-04-12T05:09:05.433 INFO:tasks.workunit.client.1.smithi192.stdout:0/660: write d0/d6/dc4/fd9 [465247,16638] 0 2024-04-12T05:09:05.434 INFO:tasks.workunit.client.1.smithi192.stdout:7/504: getdents d3/d7/d60/d65 0 2024-04-12T05:09:05.442 INFO:tasks.workunit.client.1.smithi192.stdout:0/661: mkdir d0/d7b/d9b/d4b/dbc/dce/db0/dbf/dde 0 2024-04-12T05:09:05.443 INFO:tasks.workunit.client.1.smithi192.stdout:0/662: read - d0/d7b/d9b/d4b/dbc/dce/d53/fc3 zero size 2024-04-12T05:09:05.448 INFO:tasks.workunit.client.1.smithi192.stdout:7/505: dread d3/d1d/d5a/d5f/f31 [0,4194304] 0 2024-04-12T05:09:05.449 INFO:tasks.workunit.client.1.smithi192.stdout:0/663: mknod d0/d7b/d9b/d4b/dbc/dce/db0/dbf/dde/cdf 0 2024-04-12T05:09:05.451 INFO:tasks.workunit.client.1.smithi192.stdout:7/506: creat d3/d1d/d4a/faa x:0 0 0 2024-04-12T05:09:05.457 INFO:tasks.workunit.client.0.smithi113.stdout:2/514: dwrite d8/d9/de/d15/d38/f52 [0,4194304] 0 2024-04-12T05:09:05.458 INFO:tasks.workunit.client.1.smithi192.stdout:2/583: dwrite d1/f70 [0,4194304] 0 2024-04-12T05:09:05.460 INFO:tasks.workunit.client.1.smithi192.stdout:2/584: fsync d1/d8/d17/d19/f5e 0 2024-04-12T05:09:05.465 INFO:tasks.workunit.client.1.smithi192.stdout:7/507: dwrite d3/d1d/d5a/d5f/fa4 [0,4194304] 0 2024-04-12T05:09:05.468 INFO:tasks.workunit.client.1.smithi192.stdout:7/508: write d3/d1d/d4a/f70 [342146,115797] 0 2024-04-12T05:09:05.475 INFO:tasks.workunit.client.1.smithi192.stdout:0/664: dread d0/d6/d15/d55/d2e/f9e [0,4194304] 0 2024-04-12T05:09:05.477 INFO:tasks.workunit.client.1.smithi192.stdout:0/665: readlink d0/d6/l7f 0 2024-04-12T05:09:05.480 INFO:tasks.workunit.client.1.smithi192.stdout:6/608: sync 2024-04-12T05:09:05.480 INFO:tasks.workunit.client.1.smithi192.stdout:1/574: sync 2024-04-12T05:09:05.480 INFO:tasks.workunit.client.1.smithi192.stdout:8/486: sync 2024-04-12T05:09:05.482 INFO:tasks.workunit.client.1.smithi192.stdout:6/609: write d3/dc/d39/f87 [3601246,68005] 0 2024-04-12T05:09:05.483 INFO:tasks.workunit.client.1.smithi192.stdout:0/666: rename d0/l2c to d0/d6/d15/d19/le0 0 2024-04-12T05:09:05.486 INFO:tasks.workunit.client.0.smithi113.stdout:2/515: write d8/d9/de/d12/d2b/f45 [352243,106789] 0 2024-04-12T05:09:05.486 INFO:tasks.workunit.client.1.smithi192.stdout:6/610: write d3/d8/d22/f2e [3536519,64134] 0 2024-04-12T05:09:05.487 INFO:tasks.workunit.client.1.smithi192.stdout:6/611: readlink d3/d8/lba 0 2024-04-12T05:09:05.488 INFO:tasks.workunit.client.0.smithi113.stdout:2/516: fsync d8/d9/d27/f72 0 2024-04-12T05:09:05.490 INFO:tasks.workunit.client.1.smithi192.stdout:1/575: read d0/f24 [753764,82890] 0 2024-04-12T05:09:05.491 INFO:tasks.workunit.client.1.smithi192.stdout:8/487: link d6/d10/d46/d5e/d5f/d37/d4f/d4c/f55 d6/f95 0 2024-04-12T05:09:05.492 INFO:tasks.workunit.client.1.smithi192.stdout:2/585: dread d1/f4 [0,4194304] 0 2024-04-12T05:09:05.492 INFO:tasks.workunit.client.1.smithi192.stdout:1/576: stat d0/d2/dd/c12 0 2024-04-12T05:09:05.493 INFO:tasks.workunit.client.1.smithi192.stdout:6/612: rename d3/d8/d10/d4c/d68/d83 to d3/d23/db2/dc7 0 2024-04-12T05:09:05.497 INFO:tasks.workunit.client.1.smithi192.stdout:7/509: dread d3/d7/d15/f24 [0,4194304] 0 2024-04-12T05:09:05.499 INFO:tasks.workunit.client.1.smithi192.stdout:2/586: creat d1/d8/d3a/fae x:0 0 0 2024-04-12T05:09:05.500 INFO:tasks.workunit.client.1.smithi192.stdout:1/577: creat d0/d2/dd/d3c/d5b/fb8 x:0 0 0 2024-04-12T05:09:05.500 INFO:tasks.workunit.client.1.smithi192.stdout:8/488: dwrite d6/da/f3a [0,4194304] 0 2024-04-12T05:09:05.505 INFO:tasks.workunit.client.1.smithi192.stdout:6/613: getdents d3/d5 0 2024-04-12T05:09:05.505 INFO:tasks.workunit.client.1.smithi192.stdout:8/489: fdatasync d6/d10/d46/d5e/d5f/f7e 0 2024-04-12T05:09:05.507 INFO:tasks.workunit.client.0.smithi113.stdout:0/705: sync 2024-04-12T05:09:05.507 INFO:tasks.workunit.client.0.smithi113.stdout:4/578: sync 2024-04-12T05:09:05.507 INFO:tasks.workunit.client.0.smithi113.stdout:3/585: sync 2024-04-12T05:09:05.507 INFO:tasks.workunit.client.0.smithi113.stdout:9/630: sync 2024-04-12T05:09:05.508 INFO:tasks.workunit.client.1.smithi192.stdout:7/510: dread d3/d7/d60/f66 [0,4194304] 0 2024-04-12T05:09:05.509 INFO:tasks.workunit.client.0.smithi113.stdout:2/517: truncate d8/d9/de/d12/f78 2524832 0 2024-04-12T05:09:05.510 INFO:tasks.workunit.client.1.smithi192.stdout:1/578: dread - d0/d2/dd/d3c/dab/fb5 zero size 2024-04-12T05:09:05.514 INFO:tasks.workunit.client.0.smithi113.stdout:3/586: rmdir d0/d13 39 2024-04-12T05:09:05.515 INFO:tasks.workunit.client.1.smithi192.stdout:6/614: dread d3/d5/f72 [0,4194304] 0 2024-04-12T05:09:05.517 INFO:tasks.workunit.client.0.smithi113.stdout:4/579: read - da/d2b/fa0 zero size 2024-04-12T05:09:05.517 INFO:tasks.workunit.client.0.smithi113.stdout:9/631: chown d1/df/d4b/d63/l3a 123543 1 2024-04-12T05:09:05.519 INFO:tasks.workunit.client.1.smithi192.stdout:8/490: rename d6/d10/d46/d5e/d5f/d31/f39 to d6/d10/d46/f96 0 2024-04-12T05:09:05.519 INFO:tasks.workunit.client.0.smithi113.stdout:4/580: write da/dd/d1b/f28 [4516964,92589] 0 2024-04-12T05:09:05.522 INFO:tasks.workunit.client.1.smithi192.stdout:6/615: symlink d3/d26/lc8 0 2024-04-12T05:09:05.523 INFO:tasks.workunit.client.0.smithi113.stdout:2/518: rename d8/d9/de/d12/d2b/f45 to d8/d9/de/d15/f9d 0 2024-04-12T05:09:05.525 INFO:tasks.workunit.client.1.smithi192.stdout:7/511: rename d3/d1d/d74/f75 to d3/d1d/d4a/d5e/d6d/da3/fab 0 2024-04-12T05:09:05.526 INFO:tasks.workunit.client.0.smithi113.stdout:9/632: symlink d1/d4/d6d/ld6 0 2024-04-12T05:09:05.527 INFO:tasks.workunit.client.1.smithi192.stdout:8/491: symlink d6/d10/d46/d5e/d6e/l97 0 2024-04-12T05:09:05.528 INFO:tasks.workunit.client.1.smithi192.stdout:1/579: getdents d0/d29/d8d 0 2024-04-12T05:09:05.531 INFO:tasks.workunit.client.1.smithi192.stdout:6/616: creat d3/dc/d39/fc9 x:0 0 0 2024-04-12T05:09:05.531 INFO:tasks.workunit.client.1.smithi192.stdout:7/512: creat d3/d7/d15/d1a/fac x:0 0 0 2024-04-12T05:09:05.532 INFO:tasks.workunit.client.0.smithi113.stdout:2/519: fdatasync d8/d9/de/d12/f78 0 2024-04-12T05:09:05.533 INFO:tasks.workunit.client.0.smithi113.stdout:9/633: mkdir d1/d4/d38/d65/dcd/dd7 0 2024-04-12T05:09:05.534 INFO:tasks.workunit.client.1.smithi192.stdout:7/513: truncate d3/d1d/f41 1639554 0 2024-04-12T05:09:05.535 INFO:tasks.workunit.client.1.smithi192.stdout:1/580: creat d0/d1b/d44/fb9 x:0 0 0 2024-04-12T05:09:05.535 INFO:tasks.workunit.client.1.smithi192.stdout:8/492: mkdir d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d98 0 2024-04-12T05:09:05.536 INFO:tasks.workunit.client.0.smithi113.stdout:2/520: chown d8/d9/d1a/l71 26 1 2024-04-12T05:09:05.538 INFO:tasks.workunit.client.0.smithi113.stdout:2/521: mknod d8/d9/d27/d88/c9e 0 2024-04-12T05:09:05.540 INFO:tasks.workunit.client.1.smithi192.stdout:6/617: dwrite d3/d21/f8d [0,4194304] 0 2024-04-12T05:09:05.542 INFO:tasks.workunit.client.0.smithi113.stdout:9/634: link d1/d27/f5d d1/df/d4b/d63/d3d/dab/db7/fd8 0 2024-04-12T05:09:05.542 INFO:tasks.workunit.client.1.smithi192.stdout:1/581: mknod d0/d2/d1e/cba 0 2024-04-12T05:09:05.543 INFO:tasks.workunit.client.1.smithi192.stdout:6/618: write d3/dc/d39/f70 [2135352,128232] 0 2024-04-12T05:09:05.544 INFO:tasks.workunit.client.1.smithi192.stdout:8/493: unlink d6/d10/d46/d5e/d5f/d38/f8f 0 2024-04-12T05:09:05.544 INFO:tasks.workunit.client.0.smithi113.stdout:2/522: truncate d8/d9/de/d12/f9c 144838 0 2024-04-12T05:09:05.545 INFO:tasks.workunit.client.0.smithi113.stdout:3/587: dwrite d0/d13/f6b [0,4194304] 0 2024-04-12T05:09:05.548 INFO:tasks.workunit.client.0.smithi113.stdout:9/635: write d1/d4/d5/d22/fca [959767,23253] 0 2024-04-12T05:09:05.548 INFO:tasks.workunit.client.0.smithi113.stdout:4/581: dwrite da/f66 [0,4194304] 0 2024-04-12T05:09:05.550 INFO:tasks.workunit.client.0.smithi113.stdout:9/636: stat d1/df/d4b/d63/d3d/f49 0 2024-04-12T05:09:05.552 INFO:tasks.workunit.client.1.smithi192.stdout:1/582: dwrite d0/d29/d8d/f8f [4194304,4194304] 0 2024-04-12T05:09:05.553 INFO:tasks.workunit.client.1.smithi192.stdout:8/494: truncate d6/d10/d46/d5e/d5f/f57 4652063 0 2024-04-12T05:09:05.554 INFO:tasks.workunit.client.0.smithi113.stdout:3/588: read d0/d13/d15/d28/d2a/d34/d5f/f65 [606963,129800] 0 2024-04-12T05:09:05.555 INFO:tasks.workunit.client.1.smithi192.stdout:7/514: rmdir d3 39 2024-04-12T05:09:05.557 INFO:tasks.workunit.client.1.smithi192.stdout:6/619: dwrite d3/dc/d39/f70 [0,4194304] 0 2024-04-12T05:09:05.558 INFO:tasks.workunit.client.1.smithi192.stdout:8/495: symlink d6/d10/l99 0 2024-04-12T05:09:05.559 INFO:tasks.workunit.client.0.smithi113.stdout:9/637: fdatasync d1/d4/d87/fbb 0 2024-04-12T05:09:05.560 INFO:tasks.workunit.client.0.smithi113.stdout:9/638: readlink d1/d4/d6d/ld6 0 2024-04-12T05:09:05.564 INFO:tasks.workunit.client.0.smithi113.stdout:2/523: dwrite d8/d9/de/d15/d38/f51 [0,4194304] 0 2024-04-12T05:09:05.574 INFO:tasks.workunit.client.0.smithi113.stdout:2/524: mknod d8/d9/de/c9f 0 2024-04-12T05:09:05.575 INFO:tasks.workunit.client.0.smithi113.stdout:9/639: getdents d1/df/d4b 0 2024-04-12T05:09:05.576 INFO:tasks.workunit.client.0.smithi113.stdout:2/525: fdatasync d8/d9/de/d15/d38/f52 0 2024-04-12T05:09:05.576 INFO:tasks.workunit.client.1.smithi192.stdout:6/620: creat d3/dc/d39/fca x:0 0 0 2024-04-12T05:09:05.578 INFO:tasks.workunit.client.1.smithi192.stdout:1/583: rename d0/d1b/d28/f61 to d0/d2/d1e/d3f/d56/d9e/fbb 0 2024-04-12T05:09:05.579 INFO:tasks.workunit.client.1.smithi192.stdout:6/621: dread - d3/dc/d39/fc9 zero size 2024-04-12T05:09:05.583 INFO:tasks.workunit.client.1.smithi192.stdout:8/496: rename d6/d10/d46/d5e/d5f/d38/f5a to d6/d10/d46/d5e/d5f/d37/d4f/f9a 0 2024-04-12T05:09:05.585 INFO:tasks.workunit.client.0.smithi113.stdout:9/640: truncate d1/df/d29/f7b 875059 0 2024-04-12T05:09:05.589 INFO:tasks.workunit.client.1.smithi192.stdout:1/584: getdents d0/d2/dd/d3c 0 2024-04-12T05:09:05.589 INFO:tasks.workunit.client.0.smithi113.stdout:2/526: read d8/fd [3584155,130659] 0 2024-04-12T05:09:05.590 INFO:tasks.workunit.client.1.smithi192.stdout:8/497: mknod d6/d10/d46/d5e/d5f/d38/d3b/c9b 0 2024-04-12T05:09:05.590 INFO:tasks.workunit.client.1.smithi192.stdout:7/515: dwrite d3/d7/f7e [0,4194304] 0 2024-04-12T05:09:05.592 INFO:tasks.workunit.client.0.smithi113.stdout:9/641: dwrite d1/d4/f33 [0,4194304] 0 2024-04-12T05:09:05.594 INFO:tasks.workunit.client.0.smithi113.stdout:2/527: mknod d8/d16/ca0 0 2024-04-12T05:09:05.596 INFO:tasks.workunit.client.1.smithi192.stdout:1/585: mkdir d0/d2/dd/d21/d62/dbc 0 2024-04-12T05:09:05.597 INFO:tasks.workunit.client.0.smithi113.stdout:4/582: write da/dd/d3a/f88 [165309,62954] 0 2024-04-12T05:09:05.597 INFO:tasks.workunit.client.0.smithi113.stdout:2/528: write d8/d9/d27/f6d [1829235,34120] 0 2024-04-12T05:09:05.598 INFO:tasks.workunit.client.0.smithi113.stdout:3/589: write d0/d13/d15/d28/fa5 [152845,60397] 0 2024-04-12T05:09:05.599 INFO:tasks.workunit.client.1.smithi192.stdout:7/516: unlink d3/d7/c26 0 2024-04-12T05:09:05.604 INFO:tasks.workunit.client.1.smithi192.stdout:7/517: mknod d3/d1d/d4a/d5e/d6d/da3/cad 0 2024-04-12T05:09:05.608 INFO:tasks.workunit.client.0.smithi113.stdout:2/529: chown d8/d9/de/d12/f4c 3522305 1 2024-04-12T05:09:05.608 INFO:tasks.workunit.client.0.smithi113.stdout:4/583: creat da/d24/d7e/fbe x:0 0 0 2024-04-12T05:09:05.608 INFO:tasks.workunit.client.0.smithi113.stdout:3/590: write d0/fa6 [1856216,120723] 0 2024-04-12T05:09:05.610 INFO:tasks.workunit.client.0.smithi113.stdout:9/642: dwrite d1/df/d4b/d63/dad/fc6 [0,4194304] 0 2024-04-12T05:09:05.611 INFO:tasks.workunit.client.0.smithi113.stdout:3/591: chown d0/d13/d15/d28/f3f 11408 1 2024-04-12T05:09:05.611 INFO:tasks.workunit.client.0.smithi113.stdout:9/643: write d1/d4/f98 [1471688,78804] 0 2024-04-12T05:09:05.613 INFO:tasks.workunit.client.0.smithi113.stdout:3/592: read - d0/d13/d15/fb4 zero size 2024-04-12T05:09:05.613 INFO:tasks.workunit.client.0.smithi113.stdout:9/644: chown d1/df/d4b/d63 383210 1 2024-04-12T05:09:05.613 INFO:tasks.workunit.client.0.smithi113.stdout:4/584: symlink da/d3c/d83/d47/lbf 0 2024-04-12T05:09:05.617 INFO:tasks.workunit.client.1.smithi192.stdout:7/518: getdents d3/d1d/d5a/d5f/d7a 0 2024-04-12T05:09:05.617 INFO:tasks.workunit.client.0.smithi113.stdout:2/530: rename d8/d16/f37 to d8/d9/de/d12/d2b/fa1 0 2024-04-12T05:09:05.618 INFO:tasks.workunit.client.0.smithi113.stdout:4/585: dread - da/dd/f84 zero size 2024-04-12T05:09:05.620 INFO:tasks.workunit.client.1.smithi192.stdout:1/586: dread d0/d2/dd/d21/d62/d84/f8a [0,4194304] 0 2024-04-12T05:09:05.620 INFO:tasks.workunit.client.1.smithi192.stdout:6/622: dread d3/d8/d10/d4c/f82 [0,4194304] 0 2024-04-12T05:09:05.622 INFO:tasks.workunit.client.0.smithi113.stdout:4/586: rmdir da/d4a 39 2024-04-12T05:09:05.625 INFO:tasks.workunit.client.1.smithi192.stdout:6/623: creat d3/d23/db5/fcb x:0 0 0 2024-04-12T05:09:05.625 INFO:tasks.workunit.client.0.smithi113.stdout:2/531: dwrite d8/d9/d27/d3f/f94 [0,4194304] 0 2024-04-12T05:09:05.626 INFO:tasks.workunit.client.1.smithi192.stdout:7/519: dread d3/d7/d14/f49 [0,4194304] 0 2024-04-12T05:09:05.627 INFO:tasks.workunit.client.0.smithi113.stdout:4/587: rmdir da/d24/d60 39 2024-04-12T05:09:05.628 INFO:tasks.workunit.client.1.smithi192.stdout:1/587: link d0/d2/dd/d3c/c5f d0/d1b/d44/cbd 0 2024-04-12T05:09:05.629 INFO:tasks.workunit.client.0.smithi113.stdout:2/532: write d8/fd [5679918,90935] 0 2024-04-12T05:09:05.631 INFO:tasks.workunit.client.1.smithi192.stdout:6/624: truncate d3/d23/f78 559534 0 2024-04-12T05:09:05.634 INFO:tasks.workunit.client.1.smithi192.stdout:6/625: creat d3/dc/fcc x:0 0 0 2024-04-12T05:09:05.635 INFO:tasks.workunit.client.0.smithi113.stdout:2/533: creat d8/d9/de/d12/d3c/d8a/fa2 x:0 0 0 2024-04-12T05:09:05.636 INFO:tasks.workunit.client.0.smithi113.stdout:2/534: fdatasync d8/d16/f4e 0 2024-04-12T05:09:05.637 INFO:tasks.workunit.client.1.smithi192.stdout:7/520: dwrite d3/d1d/d4a/f61 [0,4194304] 0 2024-04-12T05:09:05.638 INFO:tasks.workunit.client.1.smithi192.stdout:1/588: link d0/d2/dd/d21/d33/fa6 d0/d2/d1e/d3f/d56/fbe 0 2024-04-12T05:09:05.639 INFO:tasks.workunit.client.1.smithi192.stdout:1/589: rmdir d0 39 2024-04-12T05:09:05.640 INFO:tasks.workunit.client.1.smithi192.stdout:7/521: mknod d3/d1d/d74/cae 0 2024-04-12T05:09:05.640 INFO:tasks.workunit.client.0.smithi113.stdout:2/535: creat d8/d9/d27/d88/fa3 x:0 0 0 2024-04-12T05:09:05.641 INFO:tasks.workunit.client.0.smithi113.stdout:2/536: write d8/d9/de/d12/f9c [838934,5691] 0 2024-04-12T05:09:05.642 INFO:tasks.workunit.client.1.smithi192.stdout:7/522: readlink d3/d7/l52 0 2024-04-12T05:09:05.642 INFO:tasks.workunit.client.0.smithi113.stdout:2/537: read d8/d9/d1a/f26 [48301,11803] 0 2024-04-12T05:09:05.643 INFO:tasks.workunit.client.1.smithi192.stdout:1/590: mkdir d0/d29/dbf 0 2024-04-12T05:09:05.645 INFO:tasks.workunit.client.0.smithi113.stdout:2/538: creat d8/d9/d27/d88/fa4 x:0 0 0 2024-04-12T05:09:05.646 INFO:tasks.workunit.client.1.smithi192.stdout:1/591: creat d0/d2/dd/d21/d62/dbc/fc0 x:0 0 0 2024-04-12T05:09:05.649 INFO:tasks.workunit.client.1.smithi192.stdout:1/592: mknod d0/d2/dd/d21/d62/dbc/cc1 0 2024-04-12T05:09:05.653 INFO:tasks.workunit.client.0.smithi113.stdout:2/539: dwrite d8/d9/de/d15/d24/d58/f75 [0,4194304] 0 2024-04-12T05:09:05.653 INFO:tasks.workunit.client.0.smithi113.stdout:7/651: sync 2024-04-12T05:09:05.653 INFO:tasks.workunit.client.0.smithi113.stdout:8/640: sync 2024-04-12T05:09:05.653 INFO:tasks.workunit.client.0.smithi113.stdout:1/752: sync 2024-04-12T05:09:05.654 INFO:tasks.workunit.client.1.smithi192.stdout:6/626: dread d3/d8/d22/f65 [0,4194304] 0 2024-04-12T05:09:05.655 INFO:tasks.workunit.client.0.smithi113.stdout:2/540: chown d8/d9/de/d15/f29 126221 1 2024-04-12T05:09:05.657 INFO:tasks.workunit.client.0.smithi113.stdout:6/603: sync 2024-04-12T05:09:05.657 INFO:tasks.workunit.client.0.smithi113.stdout:5/648: sync 2024-04-12T05:09:05.659 INFO:tasks.workunit.client.0.smithi113.stdout:8/641: symlink d9/d16/d7f/d3a/lcc 0 2024-04-12T05:09:05.660 INFO:tasks.workunit.client.0.smithi113.stdout:7/652: creat d0/dc/d6/d24/fe1 x:0 0 0 2024-04-12T05:09:05.661 INFO:tasks.workunit.client.0.smithi113.stdout:2/541: mknod d8/d9/de/d15/d38/ca5 0 2024-04-12T05:09:05.662 INFO:tasks.workunit.client.0.smithi113.stdout:5/649: creat d0/d10/d27/d1d/d38/d75/fd7 x:0 0 0 2024-04-12T05:09:05.664 INFO:tasks.workunit.client.0.smithi113.stdout:6/604: creat d8/de/d13/d11/d36/d3d/d4c/d26/fdd x:0 0 0 2024-04-12T05:09:05.666 INFO:tasks.workunit.client.0.smithi113.stdout:7/653: creat d0/d6c/fe2 x:0 0 0 2024-04-12T05:09:05.667 INFO:tasks.workunit.client.0.smithi113.stdout:7/654: stat d0/dc/d6/l43 0 2024-04-12T05:09:05.668 INFO:tasks.workunit.client.0.smithi113.stdout:5/650: creat d0/d10/d6e/fd8 x:0 0 0 2024-04-12T05:09:05.669 INFO:tasks.workunit.client.0.smithi113.stdout:7/655: write d0/dc/da9/d5c/d9a/d9e/fbd [745259,73965] 0 2024-04-12T05:09:05.671 INFO:tasks.workunit.client.0.smithi113.stdout:8/642: link d9/d1b/d8e/d93/d32/d5b/fb8 d9/d1b/fcd 0 2024-04-12T05:09:05.672 INFO:tasks.workunit.client.1.smithi192.stdout:6/627: getdents d3/d8/d56/d59 0 2024-04-12T05:09:05.673 INFO:tasks.workunit.client.0.smithi113.stdout:2/542: dread d8/d9/de/d12/f78 [0,4194304] 0 2024-04-12T05:09:05.673 INFO:tasks.workunit.client.0.smithi113.stdout:7/656: chown d0/dc/d6/d1a/l46 3718347 1 2024-04-12T05:09:05.674 INFO:tasks.workunit.client.0.smithi113.stdout:6/605: read d8/d45/f6e [613046,66426] 0 2024-04-12T05:09:05.676 INFO:tasks.workunit.client.1.smithi192.stdout:6/628: write d3/d8/f1c [7871694,84365] 0 2024-04-12T05:09:05.676 INFO:tasks.workunit.client.0.smithi113.stdout:3/593: write d0/d13/d47/f4a [663699,124641] 0 2024-04-12T05:09:05.677 INFO:tasks.workunit.client.1.smithi192.stdout:4/677: sync 2024-04-12T05:09:05.677 INFO:tasks.workunit.client.1.smithi192.stdout:8/498: dread f3 [0,4194304] 0 2024-04-12T05:09:05.678 INFO:tasks.workunit.client.0.smithi113.stdout:5/651: creat d0/d10/d27/d1d/d3f/d73/fd9 x:0 0 0 2024-04-12T05:09:05.681 INFO:tasks.workunit.client.0.smithi113.stdout:3/594: truncate d0/fa8 536294 0 2024-04-12T05:09:05.682 INFO:tasks.workunit.client.0.smithi113.stdout:8/643: symlink d9/d16/d7f/d3a/d6c/d90/lce 0 2024-04-12T05:09:05.686 INFO:tasks.workunit.client.1.smithi192.stdout:4/678: symlink d1/d54/d82/d9c/le4 0 2024-04-12T05:09:05.686 INFO:tasks.workunit.client.0.smithi113.stdout:7/657: symlink d0/dc/d6/d9b/dcd/le3 0 2024-04-12T05:09:05.689 INFO:tasks.workunit.client.0.smithi113.stdout:8/644: write d9/d16/d65/d8c/fb5 [1540958,106279] 0 2024-04-12T05:09:05.689 INFO:tasks.workunit.client.0.smithi113.stdout:5/652: rmdir d0/d10/d27/d1d/d3f 39 2024-04-12T05:09:05.689 INFO:tasks.workunit.client.1.smithi192.stdout:8/499: dwrite d6/d10/d46/d5e/d5f/d38/d3b/f81 [0,4194304] 0 2024-04-12T05:09:05.694 INFO:tasks.workunit.client.0.smithi113.stdout:9/645: dwrite d1/df/d29/fb0 [0,4194304] 0 2024-04-12T05:09:05.694 INFO:tasks.workunit.client.0.smithi113.stdout:3/595: mknod d0/d13/d15/cc5 0 2024-04-12T05:09:05.695 INFO:tasks.workunit.client.1.smithi192.stdout:8/500: dread d6/d10/f78 [0,4194304] 0 2024-04-12T05:09:05.697 INFO:tasks.workunit.client.0.smithi113.stdout:4/588: dwrite da/d4a/f9d [0,4194304] 0 2024-04-12T05:09:05.700 INFO:tasks.workunit.client.1.smithi192.stdout:6/629: dread d3/d8/d48/f6f [0,4194304] 0 2024-04-12T05:09:05.701 INFO:tasks.workunit.client.1.smithi192.stdout:8/501: mkdir d6/d1f/d9c 0 2024-04-12T05:09:05.704 INFO:tasks.workunit.client.0.smithi113.stdout:7/658: creat d0/dc/d6/d9b/d9f/db3/fe4 x:0 0 0 2024-04-12T05:09:05.704 INFO:tasks.workunit.client.0.smithi113.stdout:1/753: dread d10/d41/d89/d96/d7d/fa9 [0,4194304] 0 2024-04-12T05:09:05.704 INFO:tasks.workunit.client.0.smithi113.stdout:2/543: write d8/d9/de/d12/f35 [896792,13992] 0 2024-04-12T05:09:05.705 INFO:tasks.workunit.client.0.smithi113.stdout:1/754: chown d10/d2e 5254 1 2024-04-12T05:09:05.708 INFO:tasks.workunit.client.1.smithi192.stdout:6/630: unlink d3/d23/d67/cc5 0 2024-04-12T05:09:05.710 INFO:tasks.workunit.client.0.smithi113.stdout:1/755: creat d10/d21/ff4 x:0 0 0 2024-04-12T05:09:05.711 INFO:tasks.workunit.client.0.smithi113.stdout:5/653: mkdir d0/d10/d27/d1d/d3f/db0/dda 0 2024-04-12T05:09:05.713 INFO:tasks.workunit.client.1.smithi192.stdout:8/502: dwrite d6/d10/d46/d5e/d5f/d38/d3b/f81 [0,4194304] 0 2024-04-12T05:09:05.714 INFO:tasks.workunit.client.0.smithi113.stdout:6/606: write d8/de/d13/d11/d36/d53/db8/d63/d66/f7e [1307462,19680] 0 2024-04-12T05:09:05.714 INFO:tasks.workunit.client.0.smithi113.stdout:4/589: dread da/d4a/f9d [0,4194304] 0 2024-04-12T05:09:05.714 INFO:tasks.workunit.client.0.smithi113.stdout:3/596: mkdir d0/dc6 0 2024-04-12T05:09:05.715 INFO:tasks.workunit.client.1.smithi192.stdout:1/593: dread d0/d2/dd/d21/d33/f42 [0,4194304] 0 2024-04-12T05:09:05.715 INFO:tasks.workunit.client.1.smithi192.stdout:6/631: read d3/d8/f17 [2659761,107196] 0 2024-04-12T05:09:05.715 INFO:tasks.workunit.client.0.smithi113.stdout:1/756: symlink d10/d2e/lf5 0 2024-04-12T05:09:05.715 INFO:tasks.workunit.client.0.smithi113.stdout:7/659: mknod d0/d52/dc5/ce5 0 2024-04-12T05:09:05.716 INFO:tasks.workunit.client.0.smithi113.stdout:2/544: dread d8/d9/de/d15/d24/d58/f75 [0,4194304] 0 2024-04-12T05:09:05.716 INFO:tasks.workunit.client.1.smithi192.stdout:1/594: chown d0/d1b/d44/d69/faa 464 1 2024-04-12T05:09:05.718 INFO:tasks.workunit.client.1.smithi192.stdout:8/503: stat d6/d10/d46/d5e/d5f/d37/d4f/d4c/f94 0 2024-04-12T05:09:05.718 INFO:tasks.workunit.client.0.smithi113.stdout:7/660: write d0/dc/da9/d5c/fb9 [700375,13679] 0 2024-04-12T05:09:05.720 INFO:tasks.workunit.client.0.smithi113.stdout:7/661: write d0/dd/d11/fc2 [239971,106965] 0 2024-04-12T05:09:05.721 INFO:tasks.workunit.client.0.smithi113.stdout:3/597: symlink d0/d13/d15/d28/d2a/d8b/lc7 0 2024-04-12T05:09:05.722 INFO:tasks.workunit.client.1.smithi192.stdout:8/504: chown d6/d21/c42 168 1 2024-04-12T05:09:05.723 INFO:tasks.workunit.client.0.smithi113.stdout:5/654: unlink d0/d10/d27/d1d/d3f/db0/fbd 0 2024-04-12T05:09:05.723 INFO:tasks.workunit.client.1.smithi192.stdout:6/632: rmdir d3/d8/d10/d4c/d68 39 2024-04-12T05:09:05.725 INFO:tasks.workunit.client.0.smithi113.stdout:2/545: rmdir d8/d9/de/d15 39 2024-04-12T05:09:05.725 INFO:tasks.workunit.client.0.smithi113.stdout:4/590: fdatasync da/d3c/d83/f8a 0 2024-04-12T05:09:05.726 INFO:tasks.workunit.client.1.smithi192.stdout:1/595: rename d0/d2/dd/d21/d62/d84/f8a to d0/d2/dd/d21/d62/dbc/fc2 0 2024-04-12T05:09:05.727 INFO:tasks.workunit.client.1.smithi192.stdout:6/633: mkdir d3/d21/d24/dcd 0 2024-04-12T05:09:05.729 INFO:tasks.workunit.client.0.smithi113.stdout:3/598: mknod d0/d13/d15/d28/d2a/d34/d5f/d7b/cc8 0 2024-04-12T05:09:05.730 INFO:tasks.workunit.client.0.smithi113.stdout:7/662: rename d0/dd/d2d to d0/dd/d11/de6 0 2024-04-12T05:09:05.732 INFO:tasks.workunit.client.1.smithi192.stdout:8/505: dwrite d6/d10/d46/d5e/d5f/d37/d4f/d51/f66 [0,4194304] 0 2024-04-12T05:09:05.732 INFO:tasks.workunit.client.0.smithi113.stdout:5/655: mknod d0/d10/d27/d1d/d41/cdb 0 2024-04-12T05:09:05.732 INFO:tasks.workunit.client.0.smithi113.stdout:7/663: fdatasync d0/dc/da9/fa1 0 2024-04-12T05:09:05.732 INFO:tasks.workunit.client.0.smithi113.stdout:6/607: symlink d8/de/d13/d11/d36/d53/db8/dce/lde 0 2024-04-12T05:09:05.733 INFO:tasks.workunit.client.1.smithi192.stdout:6/634: rename d3/d8/d56/d59 to d3/d8/d56/d59/d97/dce 22 2024-04-12T05:09:05.734 INFO:tasks.workunit.client.0.smithi113.stdout:2/546: readlink d8/d9/de/d15/d24/l7f 0 2024-04-12T05:09:05.736 INFO:tasks.workunit.client.1.smithi192.stdout:8/506: mkdir d6/d10/d46/d5e/d5f/d31/d61/d9d 0 2024-04-12T05:09:05.736 INFO:tasks.workunit.client.0.smithi113.stdout:4/591: write da/dd/d1b/fb1 [84876,104650] 0 2024-04-12T05:09:05.737 INFO:tasks.workunit.client.0.smithi113.stdout:3/599: symlink d0/d13/d15/d28/d2a/d8b/d91/dad/lc9 0 2024-04-12T05:09:05.738 INFO:tasks.workunit.client.1.smithi192.stdout:6/635: unlink d3/d21/d24/d79/caf 0 2024-04-12T05:09:05.739 INFO:tasks.workunit.client.0.smithi113.stdout:7/664: mkdir d0/dc/de7 0 2024-04-12T05:09:05.740 INFO:tasks.workunit.client.1.smithi192.stdout:1/596: rename d0/d2/dd/d3c/d5b/l7e to d0/d29/d8d/d96/lc3 0 2024-04-12T05:09:05.740 INFO:tasks.workunit.client.0.smithi113.stdout:6/608: mknod d8/de/d13/d11/d36/d3d/d4c/d26/d64/cdf 0 2024-04-12T05:09:05.742 INFO:tasks.workunit.client.0.smithi113.stdout:2/547: mknod d8/d9/de/d15/d24/d58/ca6 0 2024-04-12T05:09:05.742 INFO:tasks.workunit.client.1.smithi192.stdout:8/507: read d6/d10/d46/d7c/f8c [265753,71191] 0 2024-04-12T05:09:05.746 INFO:tasks.workunit.client.1.smithi192.stdout:6/636: symlink d3/d5/d9c/dbd/lcf 0 2024-04-12T05:09:05.747 INFO:tasks.workunit.client.0.smithi113.stdout:4/592: chown da/d24/d7e/l20 178 1 2024-04-12T05:09:05.747 INFO:tasks.workunit.client.0.smithi113.stdout:7/665: unlink d0/lb2 0 2024-04-12T05:09:05.747 INFO:tasks.workunit.client.0.smithi113.stdout:6/609: stat d8/de/d13/d11/d36/d3d/c6b 0 2024-04-12T05:09:05.748 INFO:tasks.workunit.client.1.smithi192.stdout:6/637: dread - d3/d23/db5/fcb zero size 2024-04-12T05:09:05.749 INFO:tasks.workunit.client.0.smithi113.stdout:2/548: symlink d8/d9/de/d15/d24/la7 0 2024-04-12T05:09:05.749 INFO:tasks.workunit.client.0.smithi113.stdout:8/645: dread d9/d1b/d8e/d93/d32/d5b/f63 [0,4194304] 0 2024-04-12T05:09:05.753 INFO:tasks.workunit.client.0.smithi113.stdout:6/610: creat d8/de/d13/fe0 x:0 0 0 2024-04-12T05:09:05.754 INFO:tasks.workunit.client.0.smithi113.stdout:4/593: symlink da/d24/d7b/lc0 0 2024-04-12T05:09:05.755 INFO:tasks.workunit.client.0.smithi113.stdout:2/549: creat d8/d9/de/d43/fa8 x:0 0 0 2024-04-12T05:09:05.756 INFO:tasks.workunit.client.0.smithi113.stdout:5/656: link d0/d10/d27/l7c d0/d10/d27/d1d/d38/ldc 0 2024-04-12T05:09:05.756 INFO:tasks.workunit.client.0.smithi113.stdout:2/550: dread - d8/d9/d27/d88/fa4 zero size 2024-04-12T05:09:05.757 INFO:tasks.workunit.client.0.smithi113.stdout:5/657: stat d0/d10/d27/d1d/d38/d3b/f5e 0 2024-04-12T05:09:05.757 INFO:tasks.workunit.client.0.smithi113.stdout:8/646: creat d9/d16/d65/db1/d59/d78/fcf x:0 0 0 2024-04-12T05:09:05.758 INFO:tasks.workunit.client.0.smithi113.stdout:6/611: truncate d8/de/d13/d11/d36/d3d/f90 215977 0 2024-04-12T05:09:05.760 INFO:tasks.workunit.client.0.smithi113.stdout:4/594: mkdir da/d3c/d9a/dc1 0 2024-04-12T05:09:05.760 INFO:tasks.workunit.client.0.smithi113.stdout:5/658: rmdir d0/d10/d27/d1d/d3f 39 2024-04-12T05:09:05.763 INFO:tasks.workunit.client.0.smithi113.stdout:4/595: dread - da/dd/d1b/f74 zero size 2024-04-12T05:09:05.766 INFO:tasks.workunit.client.0.smithi113.stdout:2/551: dwrite d8/d9/de/d15/d24/f3b [0,4194304] 0 2024-04-12T05:09:05.768 INFO:tasks.workunit.client.0.smithi113.stdout:2/552: stat d8/d9/de/d43/l62 0 2024-04-12T05:09:05.768 INFO:tasks.workunit.client.1.smithi192.stdout:8/508: mkdir d6/d10/d6c/d9e 0 2024-04-12T05:09:05.769 INFO:tasks.workunit.client.0.smithi113.stdout:2/553: write d8/d9/de/d12/f9c [519166,8953] 0 2024-04-12T05:09:05.770 INFO:tasks.workunit.client.1.smithi192.stdout:6/638: fdatasync d3/d8/d10/d4c/f82 0 2024-04-12T05:09:05.770 INFO:tasks.workunit.client.0.smithi113.stdout:9/646: dread d1/d4/d38/f62 [0,4194304] 0 2024-04-12T05:09:05.774 INFO:tasks.workunit.client.0.smithi113.stdout:5/659: creat d0/d10/d27/d1d/d3f/db0/dda/fdd x:0 0 0 2024-04-12T05:09:05.776 INFO:tasks.workunit.client.0.smithi113.stdout:5/660: dread - d0/d10/d27/d1d/d3f/d73/d89/f5a zero size 2024-04-12T05:09:05.777 INFO:tasks.workunit.client.0.smithi113.stdout:9/647: unlink d1/d4/d5/d22/f5a 0 2024-04-12T05:09:05.780 INFO:tasks.workunit.client.0.smithi113.stdout:5/661: dread d0/d10/d27/d1d/f21 [0,4194304] 0 2024-04-12T05:09:05.783 INFO:tasks.workunit.client.0.smithi113.stdout:2/554: getdents d8/d16 0 2024-04-12T05:09:05.783 INFO:tasks.workunit.client.0.smithi113.stdout:7/666: write d0/dc/d6/f28 [4342545,24150] 0 2024-04-12T05:09:05.784 INFO:tasks.workunit.client.0.smithi113.stdout:5/662: mkdir d0/d10/d27/d1d/d8c/dde 0 2024-04-12T05:09:05.785 INFO:tasks.workunit.client.0.smithi113.stdout:5/663: chown d0/d10/d27/d1d/d3f 27300993 1 2024-04-12T05:09:05.787 INFO:tasks.workunit.client.0.smithi113.stdout:7/667: symlink d0/d52/dc5/le8 0 2024-04-12T05:09:05.789 INFO:tasks.workunit.client.0.smithi113.stdout:7/668: write d0/dd/d11/dd5/fdd [648558,62250] 0 2024-04-12T05:09:05.791 INFO:tasks.workunit.client.0.smithi113.stdout:2/555: dwrite d8/d9/de/d15/f40 [0,4194304] 0 2024-04-12T05:09:05.797 INFO:tasks.workunit.client.0.smithi113.stdout:2/556: dwrite d8/d16/f17 [0,4194304] 0 2024-04-12T05:09:05.801 INFO:tasks.workunit.client.0.smithi113.stdout:7/669: unlink d0/dc/da9/d5c/d9a/fae 0 2024-04-12T05:09:05.803 INFO:tasks.workunit.client.0.smithi113.stdout:8/647: write d9/d1b/d8e/d93/f66 [214120,97470] 0 2024-04-12T05:09:05.808 INFO:tasks.workunit.client.0.smithi113.stdout:5/664: rmdir d0/d10/d27 39 2024-04-12T05:09:05.811 INFO:tasks.workunit.client.0.smithi113.stdout:2/557: dwrite d8/d9/de/d12/f9c [0,4194304] 0 2024-04-12T05:09:05.811 INFO:tasks.workunit.client.0.smithi113.stdout:7/670: truncate d0/dc/d6/d9b/f25 2786202 0 2024-04-12T05:09:05.814 INFO:tasks.workunit.client.0.smithi113.stdout:4/596: dread da/d4f/f4e [0,4194304] 0 2024-04-12T05:09:05.814 INFO:tasks.workunit.client.0.smithi113.stdout:6/612: dwrite d8/f40 [0,4194304] 0 2024-04-12T05:09:05.815 INFO:tasks.workunit.client.0.smithi113.stdout:8/648: mkdir d9/d1b/d8e/d93/d32/d48/dc1/dd0 0 2024-04-12T05:09:05.819 INFO:tasks.workunit.client.0.smithi113.stdout:9/648: dwrite d1/d4/d30/dcb/f3f [0,4194304] 0 2024-04-12T05:09:05.821 INFO:tasks.workunit.client.0.smithi113.stdout:7/671: fsync d0/dc/d6/d78/f38 0 2024-04-12T05:09:05.821 INFO:tasks.workunit.client.0.smithi113.stdout:2/558: symlink d8/d9/de/d12/d3c/d8a/la9 0 2024-04-12T05:09:05.822 INFO:tasks.workunit.client.0.smithi113.stdout:4/597: symlink da/dd/d3a/lc2 0 2024-04-12T05:09:05.823 INFO:tasks.workunit.client.0.smithi113.stdout:5/665: dwrite d0/d10/f45 [0,4194304] 0 2024-04-12T05:09:05.825 INFO:tasks.workunit.client.0.smithi113.stdout:4/598: chown da/d3c/d9f/fba 6173517 1 2024-04-12T05:09:05.825 INFO:tasks.workunit.client.0.smithi113.stdout:8/649: creat d9/d1b/d96/fd1 x:0 0 0 2024-04-12T05:09:05.826 INFO:tasks.workunit.client.0.smithi113.stdout:9/649: mknod d1/d4/d87/cd9 0 2024-04-12T05:09:05.826 INFO:tasks.workunit.client.0.smithi113.stdout:7/672: write d0/dc/d6/f16 [3401957,26359] 0 2024-04-12T05:09:05.826 INFO:tasks.workunit.client.0.smithi113.stdout:6/613: chown d8/de/d13/d2b/db3/dc5/cc6 128 1 2024-04-12T05:09:05.828 INFO:tasks.workunit.client.0.smithi113.stdout:2/559: mkdir d8/d9/de/d12/d2b/daa 0 2024-04-12T05:09:05.828 INFO:tasks.workunit.client.0.smithi113.stdout:7/673: dread - d0/dc/da9/d5c/d9a/db6/fcf zero size 2024-04-12T05:09:05.856 INFO:tasks.workunit.client.0.smithi113.stdout:2/560: creat d8/d9/d27/d88/fab x:0 0 0 2024-04-12T05:09:05.857 INFO:tasks.workunit.client.1.smithi192.stdout:8/509: unlink d6/d10/d46/d5e/d5f/d37/d4f/d4c/l59 0 2024-04-12T05:09:05.858 INFO:tasks.workunit.client.0.smithi113.stdout:8/650: mknod d9/d16/cd2 0 2024-04-12T05:09:05.858 INFO:tasks.workunit.client.0.smithi113.stdout:4/599: mknod da/d4a/da2/cc3 0 2024-04-12T05:09:05.858 INFO:tasks.workunit.client.0.smithi113.stdout:6/614: creat d8/de/d13/d11/d36/d89/fe1 x:0 0 0 2024-04-12T05:09:05.859 INFO:tasks.workunit.client.0.smithi113.stdout:7/674: rename d0/dc/d6/d24/c2c to d0/dc/de7/ce9 0 2024-04-12T05:09:05.860 INFO:tasks.workunit.client.0.smithi113.stdout:1/757: truncate d10/d2e/d31/f53 3590924 0 2024-04-12T05:09:05.861 INFO:tasks.workunit.client.1.smithi192.stdout:6/639: creat d3/d8/d56/d59/fd0 x:0 0 0 2024-04-12T05:09:05.861 INFO:tasks.workunit.client.0.smithi113.stdout:9/650: truncate d1/df/d29/f7b 1360810 0 2024-04-12T05:09:05.861 INFO:tasks.workunit.client.1.smithi192.stdout:8/510: mknod d6/d21/c9f 0 2024-04-12T05:09:05.863 INFO:tasks.workunit.client.1.smithi192.stdout:6/640: truncate d3/d23/db5/fcb 842984 0 2024-04-12T05:09:05.863 INFO:tasks.workunit.client.0.smithi113.stdout:5/666: getdents d0/d10/d27/d1d/d38/d75 0 2024-04-12T05:09:05.864 INFO:tasks.workunit.client.0.smithi113.stdout:1/758: creat d10/d41/d89/ff6 x:0 0 0 2024-04-12T05:09:05.865 INFO:tasks.workunit.client.1.smithi192.stdout:8/511: rmdir d6/d10/d46/d5e/d5f/d37/d7b 39 2024-04-12T05:09:05.865 INFO:tasks.workunit.client.0.smithi113.stdout:8/651: write d9/d16/d65/db1/d59/d68/fa8 [2833416,81532] 0 2024-04-12T05:09:05.866 INFO:tasks.workunit.client.1.smithi192.stdout:6/641: creat d3/d8/d48/fd1 x:0 0 0 2024-04-12T05:09:05.867 INFO:tasks.workunit.client.0.smithi113.stdout:7/675: dread d0/dc/da9/d5c/d85/fb1 [0,4194304] 0 2024-04-12T05:09:05.867 INFO:tasks.workunit.client.1.smithi192.stdout:8/512: rmdir d6/d10/d46/d5e/d5f/d31/d61 39 2024-04-12T05:09:05.868 INFO:tasks.workunit.client.0.smithi113.stdout:6/615: creat d8/de/d13/d11/d36/d53/fe2 x:0 0 0 2024-04-12T05:09:05.869 INFO:tasks.workunit.client.1.smithi192.stdout:8/513: chown d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/f72 1 1 2024-04-12T05:09:05.869 INFO:tasks.workunit.client.1.smithi192.stdout:8/514: chown d6/d10/d46/d5e/d5f/d31/d80 184295 1 2024-04-12T05:09:05.872 INFO:tasks.workunit.client.0.smithi113.stdout:8/652: read d9/d16/d65/f97 [1097864,52059] 0 2024-04-12T05:09:05.872 INFO:tasks.workunit.client.0.smithi113.stdout:7/676: write d0/dd/d11/dd5/fdd [311621,92258] 0 2024-04-12T05:09:05.873 INFO:tasks.workunit.client.0.smithi113.stdout:6/616: dread - d8/de/d13/d11/d36/d3d/d4c/d31/fc0 zero size 2024-04-12T05:09:05.874 INFO:tasks.workunit.client.1.smithi192.stdout:9/476: sync 2024-04-12T05:09:05.874 INFO:tasks.workunit.client.1.smithi192.stdout:3/546: sync 2024-04-12T05:09:05.875 INFO:tasks.workunit.client.0.smithi113.stdout:1/759: link d10/d21/fd1 d10/d55/d9c/ff7 0 2024-04-12T05:09:05.878 INFO:tasks.workunit.client.1.smithi192.stdout:9/477: fsync d11/d15/d21/d2c/d1f/d50/d73/f80 0 2024-04-12T05:09:05.879 INFO:tasks.workunit.client.1.smithi192.stdout:1/597: truncate d0/d2/d1e/d3f/f87 313481 0 2024-04-12T05:09:05.880 INFO:tasks.workunit.client.0.smithi113.stdout:6/617: chown d8/de/dd2/ld6 729367 1 2024-04-12T05:09:05.881 INFO:tasks.workunit.client.0.smithi113.stdout:7/677: dwrite d0/dc/f83 [0,4194304] 0 2024-04-12T05:09:05.882 INFO:tasks.workunit.client.1.smithi192.stdout:3/547: dwrite d5/d13/d3a/d70/d8c/fb3 [0,4194304] 0 2024-04-12T05:09:05.883 INFO:tasks.workunit.client.0.smithi113.stdout:7/678: chown d0/dc/d6/d9b/dcd 2854 1 2024-04-12T05:09:05.884 INFO:tasks.workunit.client.0.smithi113.stdout:1/760: rename d10/d2e/f71 to d10/d2e/d7c/dc5/ddb/ff8 0 2024-04-12T05:09:05.884 INFO:tasks.workunit.client.0.smithi113.stdout:2/561: dread d8/d9/de/f10 [0,4194304] 0 2024-04-12T05:09:05.885 INFO:tasks.workunit.client.1.smithi192.stdout:9/478: fsync d11/d25/d36/d6e/f56 0 2024-04-12T05:09:05.888 INFO:tasks.workunit.client.1.smithi192.stdout:3/548: creat d5/d13/d3a/d4a/d61/d9f/fc3 x:0 0 0 2024-04-12T05:09:05.889 INFO:tasks.workunit.client.1.smithi192.stdout:3/549: chown d5/d13/d1a/l82 1 1 2024-04-12T05:09:05.891 INFO:tasks.workunit.client.0.smithi113.stdout:6/618: mkdir d8/de/d13/d11/d36/d53/db8/d63/d66/dc4/de3 0 2024-04-12T05:09:05.892 INFO:tasks.workunit.client.0.smithi113.stdout:5/667: link d0/d10/d27/d1d/d38/d44/d66/lab d0/d10/d27/d1d/d3f/d7e/ldf 0 2024-04-12T05:09:05.895 INFO:tasks.workunit.client.0.smithi113.stdout:7/679: read d0/dc/d6/d78/f6a [3511119,26153] 0 2024-04-12T05:09:05.896 INFO:tasks.workunit.client.0.smithi113.stdout:8/653: write d9/d16/d3c/f74 [4442219,77072] 0 2024-04-12T05:09:05.899 INFO:tasks.workunit.client.0.smithi113.stdout:2/562: mknod d8/cac 0 2024-04-12T05:09:05.901 INFO:tasks.workunit.client.0.smithi113.stdout:6/619: symlink d8/le4 0 2024-04-12T05:09:05.902 INFO:tasks.workunit.client.1.smithi192.stdout:3/550: creat d5/d13/d94/dbd/fc4 x:0 0 0 2024-04-12T05:09:05.904 INFO:tasks.workunit.client.1.smithi192.stdout:9/479: getdents d11 0 2024-04-12T05:09:05.904 INFO:tasks.workunit.client.0.smithi113.stdout:9/651: dwrite d1/df/db6/f59 [0,4194304] 0 2024-04-12T05:09:05.904 INFO:tasks.workunit.client.0.smithi113.stdout:1/761: dread d10/d41/d89/d96/dd3/f64 [0,4194304] 0 2024-04-12T05:09:05.907 INFO:tasks.workunit.client.0.smithi113.stdout:4/600: dwrite da/d4f/f4e [4194304,4194304] 0 2024-04-12T05:09:05.908 INFO:tasks.workunit.client.1.smithi192.stdout:6/642: mkdir d3/d8/d56/d59/d97/dbf/dd2 0 2024-04-12T05:09:05.911 INFO:tasks.workunit.client.0.smithi113.stdout:1/762: chown d10/d2e/l3d 148855033 1 2024-04-12T05:09:05.911 INFO:tasks.workunit.client.1.smithi192.stdout:9/480: truncate d11/d25/d36/d3d/f5e 2986243 0 2024-04-12T05:09:05.911 INFO:tasks.workunit.client.1.smithi192.stdout:3/551: dwrite d5/d13/d3a/d70/d8c/fb3 [0,4194304] 0 2024-04-12T05:09:05.913 INFO:tasks.workunit.client.0.smithi113.stdout:4/601: chown da/d4f/f4e 20409 1 2024-04-12T05:09:05.918 INFO:tasks.workunit.client.0.smithi113.stdout:7/680: dread d0/dd/d11/f20 [0,4194304] 0 2024-04-12T05:09:05.921 INFO:tasks.workunit.client.0.smithi113.stdout:9/652: dwrite d1/d4/d87/fc9 [0,4194304] 0 2024-04-12T05:09:05.922 INFO:tasks.workunit.client.0.smithi113.stdout:7/681: chown d0/d52/f57 52 1 2024-04-12T05:09:05.924 INFO:tasks.workunit.client.1.smithi192.stdout:3/552: creat d5/d13/d3c/d26/fc5 x:0 0 0 2024-04-12T05:09:05.924 INFO:tasks.workunit.client.1.smithi192.stdout:9/481: rename d11/d15/d1e to d11/d25/d36/d6e/d33/da2 0 2024-04-12T05:09:05.925 INFO:tasks.workunit.client.0.smithi113.stdout:1/763: rmdir d10/d2e/d31/dc3 39 2024-04-12T05:09:05.926 INFO:tasks.workunit.client.0.smithi113.stdout:1/764: chown d10/ddc/fe7 1779 1 2024-04-12T05:09:05.929 INFO:tasks.workunit.client.1.smithi192.stdout:3/553: creat d5/d13/d94/fc6 x:0 0 0 2024-04-12T05:09:05.930 INFO:tasks.workunit.client.0.smithi113.stdout:1/765: creat d10/d2e/d7c/ff9 x:0 0 0 2024-04-12T05:09:05.931 INFO:tasks.workunit.client.0.smithi113.stdout:9/653: mknod d1/d4/cda 0 2024-04-12T05:09:05.932 INFO:tasks.workunit.client.1.smithi192.stdout:9/482: symlink d11/d15/d21/d2c/d1f/la3 0 2024-04-12T05:09:05.933 INFO:tasks.workunit.client.0.smithi113.stdout:4/602: link da/dd/d1b/c58 da/dd/d1b/cc4 0 2024-04-12T05:09:05.934 INFO:tasks.workunit.client.0.smithi113.stdout:9/654: write d1/d4/d38/fd0 [69857,97552] 0 2024-04-12T05:09:05.936 INFO:tasks.workunit.client.0.smithi113.stdout:5/668: write d0/d10/d27/d1d/d3f/f6a [471568,16843] 0 2024-04-12T05:09:05.938 INFO:tasks.workunit.client.0.smithi113.stdout:4/603: chown da/d3c/f95 90059147 1 2024-04-12T05:09:05.938 INFO:tasks.workunit.client.0.smithi113.stdout:9/655: chown d1/df/db6/la5 2027 1 2024-04-12T05:09:05.939 INFO:tasks.workunit.client.1.smithi192.stdout:9/483: readlink d11/d25/d36/d6e/d33/da2/l40 0 2024-04-12T05:09:05.939 INFO:tasks.workunit.client.1.smithi192.stdout:3/554: dwrite d5/d13/d3c/d36/f8a [0,4194304] 0 2024-04-12T05:09:05.941 INFO:tasks.workunit.client.1.smithi192.stdout:3/555: write d5/d13/d3a/d4a/f98 [708212,91852] 0 2024-04-12T05:09:05.943 INFO:tasks.workunit.client.0.smithi113.stdout:9/656: fdatasync d1/d4/d6d/fd4 0 2024-04-12T05:09:05.946 INFO:tasks.workunit.client.0.smithi113.stdout:1/766: dwrite d10/d55/d9c/fe5 [0,4194304] 0 2024-04-12T05:09:05.949 INFO:tasks.workunit.client.0.smithi113.stdout:8/654: dwrite d9/d1b/d8e/d93/d32/d48/d8d/d4a/fbd [0,4194304] 0 2024-04-12T05:09:05.950 INFO:tasks.workunit.client.0.smithi113.stdout:8/655: fdatasync d9/d1b/d8e/d93/d32/f3f 0 2024-04-12T05:09:05.951 INFO:tasks.workunit.client.1.smithi192.stdout:6/643: write d3/d8/f6d [386061,127665] 0 2024-04-12T05:09:05.951 INFO:tasks.workunit.client.0.smithi113.stdout:5/669: read d0/d10/d27/d1d/d38/f8d [647679,121297] 0 2024-04-12T05:09:05.952 INFO:tasks.workunit.client.0.smithi113.stdout:9/657: creat d1/df/d29/da6/fdb x:0 0 0 2024-04-12T05:09:05.953 INFO:tasks.workunit.client.1.smithi192.stdout:3/556: dwrite d5/d13/d3a/d8b/fba [0,4194304] 0 2024-04-12T05:09:05.956 INFO:tasks.workunit.client.0.smithi113.stdout:5/670: fdatasync d0/d10/d6e/f35 0 2024-04-12T05:09:05.958 INFO:tasks.workunit.client.1.smithi192.stdout:3/557: dread - d5/d13/d1a/d5f/f75 zero size 2024-04-12T05:09:05.958 INFO:tasks.workunit.client.0.smithi113.stdout:9/658: creat d1/d27/d7a/dd2/fdc x:0 0 0 2024-04-12T05:09:05.960 INFO:tasks.workunit.client.1.smithi192.stdout:3/558: fsync d5/d13/d3a/d4a/f67 0 2024-04-12T05:09:05.960 INFO:tasks.workunit.client.1.smithi192.stdout:6/644: creat d3/d5/d9c/dbd/fd3 x:0 0 0 2024-04-12T05:09:05.962 INFO:tasks.workunit.client.1.smithi192.stdout:3/559: chown d5/d13/d3c/d36/da0/lac 15316 1 2024-04-12T05:09:05.964 INFO:tasks.workunit.client.0.smithi113.stdout:9/659: rename d1/d27/fbe to d1/d4/d5/d22/fdd 0 2024-04-12T05:09:05.965 INFO:tasks.workunit.client.1.smithi192.stdout:6/645: mkdir d3/d8/d22/dd4 0 2024-04-12T05:09:05.966 INFO:tasks.workunit.client.0.smithi113.stdout:5/671: dread d0/d10/d27/d1d/d38/d44/d4e/f64 [0,4194304] 0 2024-04-12T05:09:05.967 INFO:tasks.workunit.client.1.smithi192.stdout:6/646: chown d3/d21/d24/f4f 1222886 1 2024-04-12T05:09:05.968 INFO:tasks.workunit.client.0.smithi113.stdout:9/660: creat d1/d27/d7a/fde x:0 0 0 2024-04-12T05:09:05.969 INFO:tasks.workunit.client.1.smithi192.stdout:3/560: mknod d5/d13/d94/cc7 0 2024-04-12T05:09:05.971 INFO:tasks.workunit.client.1.smithi192.stdout:3/561: symlink d5/d13/d1a/lc8 0 2024-04-12T05:09:05.973 INFO:tasks.workunit.client.0.smithi113.stdout:5/672: getdents d0/d10/d27/d1d/d38/d44/d4e/daa 0 2024-04-12T05:09:05.978 INFO:tasks.workunit.client.0.smithi113.stdout:5/673: rmdir d0/d10/d27/d1d/d3f 39 2024-04-12T05:09:05.979 INFO:tasks.workunit.client.1.smithi192.stdout:6/647: dwrite d3/d26/f46 [4194304,4194304] 0 2024-04-12T05:09:05.980 INFO:tasks.workunit.client.0.smithi113.stdout:6/620: dread d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8d [0,4194304] 0 2024-04-12T05:09:05.984 INFO:tasks.workunit.client.0.smithi113.stdout:7/682: dwrite d0/dd/d11/de6/f64 [0,4194304] 0 2024-04-12T05:09:05.986 INFO:tasks.workunit.client.1.smithi192.stdout:3/562: getdents d5/d13/d1a/d53/dbb 0 2024-04-12T05:09:05.986 INFO:tasks.workunit.client.0.smithi113.stdout:4/604: dwrite da/d2b/fa0 [0,4194304] 0 2024-04-12T05:09:05.986 INFO:tasks.workunit.client.1.smithi192.stdout:9/484: dread d11/d25/f32 [0,4194304] 0 2024-04-12T05:09:05.988 INFO:tasks.workunit.client.1.smithi192.stdout:3/563: read - d5/d13/d3a/d4a/f90 zero size 2024-04-12T05:09:05.991 INFO:tasks.workunit.client.1.smithi192.stdout:6/648: dwrite d3/d23/db5/fcb [0,4194304] 0 2024-04-12T05:09:05.992 INFO:tasks.workunit.client.0.smithi113.stdout:7/683: creat d0/dc/da9/d5c/d9a/d9e/fea x:0 0 0 2024-04-12T05:09:05.992 INFO:tasks.workunit.client.0.smithi113.stdout:5/674: chown d0/d10/d27/d1d/d38/d44/d66/c94 2 1 2024-04-12T05:09:05.993 INFO:tasks.workunit.client.1.smithi192.stdout:6/649: chown d3/d8/d56/lb1 36260 1 2024-04-12T05:09:05.994 INFO:tasks.workunit.client.1.smithi192.stdout:6/650: chown d3/d8/d22/l49 165 1 2024-04-12T05:09:05.995 INFO:tasks.workunit.client.0.smithi113.stdout:1/767: dread fb [0,4194304] 0 2024-04-12T05:09:05.996 INFO:tasks.workunit.client.1.smithi192.stdout:6/651: write d3/d21/d24/d79/fb4 [132982,36683] 0 2024-04-12T05:09:06.000 INFO:tasks.workunit.client.0.smithi113.stdout:6/621: mkdir d8/de/d13/d11/d36/d53/db8/d63/d66/dc4/dd9/de5 0 2024-04-12T05:09:06.000 INFO:tasks.workunit.client.1.smithi192.stdout:3/564: getdents d5/d13/d3a/d4e/d57 0 2024-04-12T05:09:06.001 INFO:tasks.workunit.client.0.smithi113.stdout:1/768: creat d10/d14/d95/dcf/ffa x:0 0 0 2024-04-12T05:09:06.004 INFO:tasks.workunit.client.0.smithi113.stdout:5/675: fdatasync d0/d10/d27/d1d/d38/f52 0 2024-04-12T05:09:06.004 INFO:tasks.workunit.client.1.smithi192.stdout:6/652: rename d3/d8/c35 to d3/d5/d9e/cd5 0 2024-04-12T05:09:06.004 INFO:tasks.workunit.client.0.smithi113.stdout:9/661: write d1/df/f4d [607610,88294] 0 2024-04-12T05:09:06.006 INFO:tasks.workunit.client.0.smithi113.stdout:9/662: fsync d1/d4/f98 0 2024-04-12T05:09:06.006 INFO:tasks.workunit.client.0.smithi113.stdout:4/605: dwrite da/dd/d1b/fb1 [0,4194304] 0 2024-04-12T05:09:06.009 INFO:tasks.workunit.client.0.smithi113.stdout:1/769: creat d10/d41/d89/df2/ffb x:0 0 0 2024-04-12T05:09:06.009 INFO:tasks.workunit.client.0.smithi113.stdout:6/622: read d8/de/d13/d11/d36/f4e [3564803,123645] 0 2024-04-12T05:09:06.010 INFO:tasks.workunit.client.0.smithi113.stdout:9/663: rmdir d1/d4 39 2024-04-12T05:09:06.011 INFO:tasks.workunit.client.0.smithi113.stdout:1/770: dread - d10/d2e/dd8/d70/fe3 zero size 2024-04-12T05:09:06.011 INFO:tasks.workunit.client.0.smithi113.stdout:9/664: chown d1/df/db6/f6b 6178 1 2024-04-12T05:09:06.012 INFO:tasks.workunit.client.0.smithi113.stdout:5/676: rmdir d0/d10/d27/d1d/d3f/d7e 39 2024-04-12T05:09:06.013 INFO:tasks.workunit.client.0.smithi113.stdout:6/623: fdatasync d8/de/d17/f7c 0 2024-04-12T05:09:06.013 INFO:tasks.workunit.client.1.smithi192.stdout:3/565: dwrite d5/d13/d3a/d4a/f5a [0,4194304] 0 2024-04-12T05:09:06.015 INFO:tasks.workunit.client.0.smithi113.stdout:1/771: unlink d10/d2e/d31/d5c/f97 0 2024-04-12T05:09:06.015 INFO:tasks.workunit.client.0.smithi113.stdout:6/624: dread - d8/de/d13/d11/d36/d3d/d4c/d31/fc0 zero size 2024-04-12T05:09:06.016 INFO:tasks.workunit.client.1.smithi192.stdout:6/653: rename d3/d23/db2/dc0 to d3/d8/dd6 0 2024-04-12T05:09:06.016 INFO:tasks.workunit.client.1.smithi192.stdout:3/566: stat d5/d13/d1a/c30 0 2024-04-12T05:09:06.017 INFO:tasks.workunit.client.0.smithi113.stdout:6/625: write d8/de/d13/d11/d36/d53/db7/fd4 [291463,14707] 0 2024-04-12T05:09:06.018 INFO:tasks.workunit.client.1.smithi192.stdout:6/654: mkdir d3/d21/d24/dd7 0 2024-04-12T05:09:06.018 INFO:tasks.workunit.client.1.smithi192.stdout:3/567: write d5/d13/d3a/d4e/fa8 [1008849,81819] 0 2024-04-12T05:09:06.019 INFO:tasks.workunit.client.0.smithi113.stdout:1/772: creat d10/d55/ffc x:0 0 0 2024-04-12T05:09:06.020 INFO:tasks.workunit.client.0.smithi113.stdout:9/665: creat d1/d4/d5/d6/fdf x:0 0 0 2024-04-12T05:09:06.023 INFO:tasks.workunit.client.1.smithi192.stdout:3/568: readlink d5/d13/d1a/d5f/l6b 0 2024-04-12T05:09:06.025 INFO:tasks.workunit.client.0.smithi113.stdout:0/706: sync 2024-04-12T05:09:06.027 INFO:tasks.workunit.client.0.smithi113.stdout:6/626: dwrite d8/de/d13/fb [0,4194304] 0 2024-04-12T05:09:06.028 INFO:tasks.workunit.client.0.smithi113.stdout:1/773: rmdir d10/d2e/d31/ddd 0 2024-04-12T05:09:06.029 INFO:tasks.workunit.client.0.smithi113.stdout:1/774: readlink d10/d41/d5b/lcc 0 2024-04-12T05:09:06.030 INFO:tasks.workunit.client.0.smithi113.stdout:9/666: symlink d1/d4/le0 0 2024-04-12T05:09:06.031 INFO:tasks.workunit.client.0.smithi113.stdout:0/707: mknod d1/d6/d16/d2a/d50/db6/d36/cda 0 2024-04-12T05:09:06.032 INFO:tasks.workunit.client.0.smithi113.stdout:1/775: dread d10/d41/d89/d96/dd3/f7a [0,4194304] 0 2024-04-12T05:09:06.032 INFO:tasks.workunit.client.0.smithi113.stdout:6/627: truncate d8/de/d13/d11/d36/d2c/f5c 534367 0 2024-04-12T05:09:06.034 INFO:tasks.workunit.client.0.smithi113.stdout:0/708: mknod d1/d6/d34/dc3/cdb 0 2024-04-12T05:09:06.034 INFO:tasks.workunit.client.0.smithi113.stdout:6/628: stat d8/de/d13/d11/d36/d3d/d4c/fdb 0 2024-04-12T05:09:06.036 INFO:tasks.workunit.client.0.smithi113.stdout:0/709: truncate d1/d6/d16/d2a/d50/d52/fa0 726215 0 2024-04-12T05:09:06.037 INFO:tasks.workunit.client.0.smithi113.stdout:0/710: fsync d1/d6/d16/d2a/d50/db6/d36/fd1 0 2024-04-12T05:09:06.038 INFO:tasks.workunit.client.0.smithi113.stdout:0/711: chown d1/d6/d16/d2a/d50/db6/d36/cda 96675345 1 2024-04-12T05:09:06.039 INFO:tasks.workunit.client.0.smithi113.stdout:0/712: write d1/d6/d34/dc3/f45 [3363570,49853] 0 2024-04-12T05:09:06.041 INFO:tasks.workunit.client.0.smithi113.stdout:6/629: rename d8/f2d to d8/de/d13/d11/d36/d53/db8/d63/d66/dc4/fe6 0 2024-04-12T05:09:06.042 INFO:tasks.workunit.client.0.smithi113.stdout:0/713: mkdir d1/d6/d34/dc3/ddc 0 2024-04-12T05:09:06.044 INFO:tasks.workunit.client.0.smithi113.stdout:0/714: creat d1/d6/d16/d2a/d50/db6/fdd x:0 0 0 2024-04-12T05:09:06.047 INFO:tasks.workunit.client.0.smithi113.stdout:6/630: rename d8/de/d13/d2b to d8/de/dd5/de7 0 2024-04-12T05:09:06.048 INFO:tasks.workunit.client.0.smithi113.stdout:6/631: write d8/de/d13/d11/d36/d53/db8/fc2 [400262,39005] 0 2024-04-12T05:09:06.050 INFO:tasks.workunit.client.0.smithi113.stdout:6/632: creat d8/d45/fe8 x:0 0 0 2024-04-12T05:09:06.051 INFO:tasks.workunit.client.0.smithi113.stdout:6/633: rmdir d8/d1d 39 2024-04-12T05:09:06.072 INFO:tasks.workunit.client.1.smithi192.stdout:9/485: truncate d11/d25/d36/d6e/f3e 3147444 0 2024-04-12T05:09:06.073 INFO:tasks.workunit.client.1.smithi192.stdout:5/589: sync 2024-04-12T05:09:06.074 INFO:tasks.workunit.client.1.smithi192.stdout:5/590: dread - d0/d15/d32/d69/d7d/fac zero size 2024-04-12T05:09:06.074 INFO:tasks.workunit.client.1.smithi192.stdout:9/486: symlink d11/d15/d21/d2c/d1f/d50/d73/la4 0 2024-04-12T05:09:06.075 INFO:tasks.workunit.client.1.smithi192.stdout:5/591: chown d0/d15/d24/d29/f95 473448405 1 2024-04-12T05:09:06.077 INFO:tasks.workunit.client.1.smithi192.stdout:9/487: fsync d11/d15/d21/d2c/d1f/d50/d73/f8b 0 2024-04-12T05:09:06.079 INFO:tasks.workunit.client.1.smithi192.stdout:5/592: creat d0/d15/d64/fcd x:0 0 0 2024-04-12T05:09:06.079 INFO:tasks.workunit.client.1.smithi192.stdout:9/488: fsync d11/d15/d21/d2c/d1f/d6a/f7c 0 2024-04-12T05:09:06.080 INFO:tasks.workunit.client.1.smithi192.stdout:5/593: dread - d0/d15/d3e/dca/fb7 zero size 2024-04-12T05:09:06.083 INFO:tasks.workunit.client.1.smithi192.stdout:9/489: rename d11/d25/d36/d3d/c61 to d11/d25/d36/d3d/d7a/ca5 0 2024-04-12T05:09:06.088 INFO:tasks.workunit.client.1.smithi192.stdout:5/594: dwrite d0/d15/d24/d29/f2a [0,4194304] 0 2024-04-12T05:09:06.088 INFO:tasks.workunit.client.1.smithi192.stdout:9/490: mkdir d11/d15/d21/d2c/d1f/d6a/d92/da6 0 2024-04-12T05:09:06.089 INFO:tasks.workunit.client.1.smithi192.stdout:5/595: fsync d0/f11 0 2024-04-12T05:09:06.092 INFO:tasks.workunit.client.1.smithi192.stdout:9/491: symlink d11/d15/d21/d2c/d1f/d94/la7 0 2024-04-12T05:09:06.094 INFO:tasks.workunit.client.1.smithi192.stdout:3/569: dread d5/f7a [0,4194304] 0 2024-04-12T05:09:06.095 INFO:tasks.workunit.client.1.smithi192.stdout:9/492: write d11/d25/d36/d6e/d33/da2/f84 [791046,1548] 0 2024-04-12T05:09:06.098 INFO:tasks.workunit.client.1.smithi192.stdout:9/493: write d11/d25/d36/f63 [4109555,48393] 0 2024-04-12T05:09:06.102 INFO:tasks.workunit.client.1.smithi192.stdout:6/655: dread d3/d8/f1f [0,4194304] 0 2024-04-12T05:09:06.102 INFO:tasks.workunit.client.1.smithi192.stdout:5/596: dwrite d0/d15/d32/d69/d7d/fc9 [0,4194304] 0 2024-04-12T05:09:06.103 INFO:tasks.workunit.client.1.smithi192.stdout:9/494: write d11/d15/d21/d2c/f8a [16014,53496] 0 2024-04-12T05:09:06.108 INFO:tasks.workunit.client.1.smithi192.stdout:3/570: dwrite d5/d13/d3c/d26/fc5 [0,4194304] 0 2024-04-12T05:09:06.109 INFO:tasks.workunit.client.1.smithi192.stdout:3/571: dread - d5/d13/d94/fc6 zero size 2024-04-12T05:09:06.112 INFO:tasks.workunit.client.1.smithi192.stdout:3/572: write d5/d13/d3a/d4a/d61/d79/f80 [244725,43961] 0 2024-04-12T05:09:06.119 INFO:tasks.workunit.client.1.smithi192.stdout:9/495: link d11/d15/f5c d11/d25/d36/d6e/fa8 0 2024-04-12T05:09:06.119 INFO:tasks.workunit.client.1.smithi192.stdout:6/656: link d3/d8/d10/d4c/d68/f77 d3/d21/d24/fd8 0 2024-04-12T05:09:06.121 INFO:tasks.workunit.client.1.smithi192.stdout:3/573: dwrite d5/d13/d3a/d8b/fc2 [0,4194304] 0 2024-04-12T05:09:06.122 INFO:tasks.workunit.client.0.smithi113.stdout:5/677: write d0/d10/d27/d1d/d38/d3b/fcc [51660,98414] 0 2024-04-12T05:09:06.125 INFO:tasks.workunit.client.0.smithi113.stdout:0/715: write d1/d43/d59/db2/fc1 [708405,97166] 0 2024-04-12T05:09:06.125 INFO:tasks.workunit.client.1.smithi192.stdout:3/574: chown d5/d13/d3a/d4e/d57/f51 973529801 1 2024-04-12T05:09:06.125 INFO:tasks.workunit.client.0.smithi113.stdout:6/634: write d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8f [518937,91123] 0 2024-04-12T05:09:06.129 INFO:tasks.workunit.client.0.smithi113.stdout:6/635: fdatasync d8/de/d17/f72 0 2024-04-12T05:09:06.129 INFO:tasks.workunit.client.0.smithi113.stdout:0/716: chown d1/d6/d16/d2a/d50/db6/l15 6 1 2024-04-12T05:09:06.130 INFO:tasks.workunit.client.1.smithi192.stdout:9/496: symlink d11/d25/d36/d6e/d33/la9 0 2024-04-12T05:09:06.130 INFO:tasks.workunit.client.0.smithi113.stdout:4/606: dwrite da/d24/d60/db4/fb7 [0,4194304] 0 2024-04-12T05:09:06.132 INFO:tasks.workunit.client.1.smithi192.stdout:3/575: dread d5/d13/d3c/f68 [0,4194304] 0 2024-04-12T05:09:06.133 INFO:tasks.workunit.client.1.smithi192.stdout:9/497: write d11/d25/d36/f38 [2257879,100690] 0 2024-04-12T05:09:06.133 INFO:tasks.workunit.client.0.smithi113.stdout:6/636: chown d8/de/d13/d11/d36/d3d/d4c/d31/d83 58256 1 2024-04-12T05:09:06.134 INFO:tasks.workunit.client.1.smithi192.stdout:9/498: readlink d11/l27 0 2024-04-12T05:09:06.135 INFO:tasks.workunit.client.1.smithi192.stdout:3/576: chown d5/d13/d1a/d5f/fa1 397082947 1 2024-04-12T05:09:06.138 INFO:tasks.workunit.client.0.smithi113.stdout:6/637: creat d8/d45/fe9 x:0 0 0 2024-04-12T05:09:06.138 INFO:tasks.workunit.client.1.smithi192.stdout:6/657: rename d3/d8/d10/d4c/c6a to d3/d23/cd9 0 2024-04-12T05:09:06.138 INFO:tasks.workunit.client.1.smithi192.stdout:9/499: write d11/d15/d21/d2c/d1f/d50/d73/f74 [395243,57473] 0 2024-04-12T05:09:06.139 INFO:tasks.workunit.client.1.smithi192.stdout:3/577: symlink d5/d13/d3a/d8b/da2/da7/lc9 0 2024-04-12T05:09:06.139 INFO:tasks.workunit.client.0.smithi113.stdout:4/607: link da/d3c/d83/f8a da/d3c/d43/fc5 0 2024-04-12T05:09:06.140 INFO:tasks.workunit.client.1.smithi192.stdout:6/658: mknod d3/d8/d22/cda 0 2024-04-12T05:09:06.141 INFO:tasks.workunit.client.0.smithi113.stdout:0/717: mkdir d1/d6/d16/dd5/dd6/dde 0 2024-04-12T05:09:06.141 INFO:tasks.workunit.client.0.smithi113.stdout:6/638: write d8/de/d13/d11/d36/d53/db7/fba [301556,108287] 0 2024-04-12T05:09:06.146 INFO:tasks.workunit.client.0.smithi113.stdout:4/608: dread da/d2b/fa0 [0,4194304] 0 2024-04-12T05:09:06.149 INFO:tasks.workunit.client.0.smithi113.stdout:6/639: rename d8/de/d13/la2 to d8/de/d13/d11/d36/d53/db8/dce/lea 0 2024-04-12T05:09:06.149 INFO:tasks.workunit.client.0.smithi113.stdout:0/718: symlink d1/d43/d59/db2/ldf 0 2024-04-12T05:09:06.151 INFO:tasks.workunit.client.0.smithi113.stdout:4/609: mknod da/d24/d60/db4/cc6 0 2024-04-12T05:09:06.151 INFO:tasks.workunit.client.0.smithi113.stdout:0/719: dread - d1/d43/d59/db2/fd3 zero size 2024-04-12T05:09:06.151 INFO:tasks.workunit.client.1.smithi192.stdout:6/659: truncate d3/d21/d24/f2a 412480 0 2024-04-12T05:09:06.154 INFO:tasks.workunit.client.0.smithi113.stdout:4/610: creat da/d24/d60/db4/fc7 x:0 0 0 2024-04-12T05:09:06.155 INFO:tasks.workunit.client.1.smithi192.stdout:9/500: link d11/d15/d21/d2c/d1f/d6a/d7e/d89/l9e d11/laa 0 2024-04-12T05:09:06.156 INFO:tasks.workunit.client.0.smithi113.stdout:4/611: chown da/d24/d60/fb2 895 1 2024-04-12T05:09:06.160 INFO:tasks.workunit.client.0.smithi113.stdout:4/612: mkdir da/de/dc8 0 2024-04-12T05:09:06.160 INFO:tasks.workunit.client.0.smithi113.stdout:0/720: dwrite d1/d6/d16/fb8 [4194304,4194304] 0 2024-04-12T05:09:06.161 INFO:tasks.workunit.client.1.smithi192.stdout:6/660: link d3/d8/f92 d3/d8/d10/d4c/fdb 0 2024-04-12T05:09:06.161 INFO:tasks.workunit.client.0.smithi113.stdout:5/678: dread d0/d10/d27/d1d/d3f/d73/f76 [0,4194304] 0 2024-04-12T05:09:06.161 INFO:tasks.workunit.client.1.smithi192.stdout:6/661: chown d3/d21/d24/c96 1828 1 2024-04-12T05:09:06.162 INFO:tasks.workunit.client.0.smithi113.stdout:0/721: readlink d1/d6/d34/l79 0 2024-04-12T05:09:06.164 INFO:tasks.workunit.client.1.smithi192.stdout:6/662: mknod d3/dc/cdc 0 2024-04-12T05:09:06.164 INFO:tasks.workunit.client.0.smithi113.stdout:4/613: creat da/d24/d60/db4/fc9 x:0 0 0 2024-04-12T05:09:06.165 INFO:tasks.workunit.client.1.smithi192.stdout:2/587: sync 2024-04-12T05:09:06.166 INFO:tasks.workunit.client.1.smithi192.stdout:4/679: sync 2024-04-12T05:09:06.166 INFO:tasks.workunit.client.1.smithi192.stdout:7/523: sync 2024-04-12T05:09:06.166 INFO:tasks.workunit.client.1.smithi192.stdout:0/667: sync 2024-04-12T05:09:06.168 INFO:tasks.workunit.client.0.smithi113.stdout:1/776: dread d10/d2e/d31/d5c/fa7 [0,4194304] 0 2024-04-12T05:09:06.170 INFO:tasks.workunit.client.0.smithi113.stdout:0/722: symlink d1/d6/d16/d2a/d50/d52/le0 0 2024-04-12T05:09:06.171 INFO:tasks.workunit.client.1.smithi192.stdout:6/663: symlink d3/d23/db5/ldd 0 2024-04-12T05:09:06.171 INFO:tasks.workunit.client.1.smithi192.stdout:6/664: chown d3/d5/f66 2924014 1 2024-04-12T05:09:06.172 INFO:tasks.workunit.client.0.smithi113.stdout:1/777: truncate d10/d2e/dd8/d99/fb8 808536 0 2024-04-12T05:09:06.173 INFO:tasks.workunit.client.1.smithi192.stdout:0/668: write d0/d7b/d9b/d4b/dbc/dce/d53/d6d/f9a [2310065,25199] 0 2024-04-12T05:09:06.174 INFO:tasks.workunit.client.0.smithi113.stdout:6/640: dread d8/de/dd5/de7/f57 [0,4194304] 0 2024-04-12T05:09:06.175 INFO:tasks.workunit.client.1.smithi192.stdout:7/524: rename d3/d1d/d4a/d5e/f9a to d3/d1d/d74/d86/faf 0 2024-04-12T05:09:06.176 INFO:tasks.workunit.client.1.smithi192.stdout:7/525: fsync d3/d7/d15/f77 0 2024-04-12T05:09:06.177 INFO:tasks.workunit.client.1.smithi192.stdout:7/526: chown d3/d1d/d5a/d5f/l3a 63824770 1 2024-04-12T05:09:06.178 INFO:tasks.workunit.client.0.smithi113.stdout:6/641: creat d8/de/dd5/de7/feb x:0 0 0 2024-04-12T05:09:06.179 INFO:tasks.workunit.client.0.smithi113.stdout:0/723: rename d1/d6/d16/d2a/d50/db6/d36/c94 to d1/d80/ce1 0 2024-04-12T05:09:06.181 INFO:tasks.workunit.client.0.smithi113.stdout:6/642: mknod d8/de/d13/d11/d36/d3d/d4c/cec 0 2024-04-12T05:09:06.182 INFO:tasks.workunit.client.0.smithi113.stdout:6/643: truncate d8/de/dd5/de7/feb 777233 0 2024-04-12T05:09:06.183 INFO:tasks.workunit.client.1.smithi192.stdout:3/578: rmdir d5 39 2024-04-12T05:09:06.185 INFO:tasks.workunit.client.1.smithi192.stdout:5/597: dread d0/f96 [0,4194304] 0 2024-04-12T05:09:06.185 INFO:tasks.workunit.client.0.smithi113.stdout:4/614: write da/d4a/f5c [171690,104944] 0 2024-04-12T05:09:06.186 INFO:tasks.workunit.client.1.smithi192.stdout:7/527: dread d3/d1d/d5a/d5f/f30 [0,4194304] 0 2024-04-12T05:09:06.189 INFO:tasks.workunit.client.1.smithi192.stdout:4/680: mkdir d1/d1d/d53/d1c/d95/dc6/dd5/de5 0 2024-04-12T05:09:06.191 INFO:tasks.workunit.client.0.smithi113.stdout:5/679: dwrite d0/d10/d27/f57 [0,4194304] 0 2024-04-12T05:09:06.194 INFO:tasks.workunit.client.0.smithi113.stdout:5/680: dread - d0/d10/d27/d1d/d3f/d73/fd9 zero size 2024-04-12T05:09:06.197 INFO:tasks.workunit.client.0.smithi113.stdout:5/681: write d0/d10/d27/d1d/d38/d44/d66/f96 [1236082,112582] 0 2024-04-12T05:09:06.201 INFO:tasks.workunit.client.1.smithi192.stdout:4/681: symlink d1/d54/d82/da1/le6 0 2024-04-12T05:09:06.202 INFO:tasks.workunit.client.1.smithi192.stdout:4/682: write d1/d54/fc5 [659168,90344] 0 2024-04-12T05:09:06.204 INFO:tasks.workunit.client.1.smithi192.stdout:3/579: mkdir d5/d13/d3c/d26/dca 0 2024-04-12T05:09:06.205 INFO:tasks.workunit.client.0.smithi113.stdout:5/682: dwrite d0/f5 [0,4194304] 0 2024-04-12T05:09:06.206 INFO:tasks.workunit.client.1.smithi192.stdout:8/515: sync 2024-04-12T05:09:06.206 INFO:tasks.workunit.client.1.smithi192.stdout:1/598: sync 2024-04-12T05:09:06.206 INFO:tasks.workunit.client.1.smithi192.stdout:2/588: write d1/d8/d17/d19/f45 [1137537,7038] 0 2024-04-12T05:09:06.207 INFO:tasks.workunit.client.0.smithi113.stdout:4/615: dwrite da/dd/f84 [0,4194304] 0 2024-04-12T05:09:06.211 INFO:tasks.workunit.client.0.smithi113.stdout:4/616: creat da/d24/d60/db4/fca x:0 0 0 2024-04-12T05:09:06.212 INFO:tasks.workunit.client.0.smithi113.stdout:4/617: chown da/d24/d60/daa 4703 1 2024-04-12T05:09:06.212 INFO:tasks.workunit.client.1.smithi192.stdout:9/501: creat d11/d25/d36/d6e/d33/fab x:0 0 0 2024-04-12T05:09:06.216 INFO:tasks.workunit.client.1.smithi192.stdout:6/665: write d3/d5/d9c/fa7 [337476,125351] 0 2024-04-12T05:09:06.217 INFO:tasks.workunit.client.1.smithi192.stdout:6/666: fsync d3/d8/d10/d4c/fa6 0 2024-04-12T05:09:06.218 INFO:tasks.workunit.client.1.smithi192.stdout:1/599: creat d0/d2/dd/d3c/d5b/d74/d90/fc4 x:0 0 0 2024-04-12T05:09:06.219 INFO:tasks.workunit.client.0.smithi113.stdout:5/683: creat d0/d10/d27/d1d/d38/fe0 x:0 0 0 2024-04-12T05:09:06.220 INFO:tasks.workunit.client.0.smithi113.stdout:4/618: fsync da/dd/d3a/f42 0 2024-04-12T05:09:06.222 INFO:tasks.workunit.client.1.smithi192.stdout:0/669: link d0/d6/d15/d19/le0 d0/d6/d67/le1 0 2024-04-12T05:09:06.222 INFO:tasks.workunit.client.1.smithi192.stdout:7/528: link d3/d1d/d48/c4e d3/d7/d14/d1f/d76/cb0 0 2024-04-12T05:09:06.226 INFO:tasks.workunit.client.0.smithi113.stdout:5/684: rename d0/d10/d27/d1d/d3f/db0/fd6 to d0/d10/d27/d1d/d38/d44/dcb/fe1 0 2024-04-12T05:09:06.226 INFO:tasks.workunit.client.1.smithi192.stdout:0/670: fsync d0/d6/d67/f92 0 2024-04-12T05:09:06.226 INFO:tasks.workunit.client.1.smithi192.stdout:7/529: fdatasync d3/d11/f5c 0 2024-04-12T05:09:06.226 INFO:tasks.workunit.client.1.smithi192.stdout:3/580: link d5/c8 d5/d13/d1a/d5f/ccb 0 2024-04-12T05:09:06.227 INFO:tasks.workunit.client.1.smithi192.stdout:2/589: creat d1/d2d/faf x:0 0 0 2024-04-12T05:09:06.227 INFO:tasks.workunit.client.0.smithi113.stdout:4/619: truncate da/fb 316492 0 2024-04-12T05:09:06.227 INFO:tasks.workunit.client.0.smithi113.stdout:5/685: fdatasync d0/d10/d27/d1d/d3f/d73/d89/fc1 0 2024-04-12T05:09:06.229 INFO:tasks.workunit.client.1.smithi192.stdout:4/683: rename d1/d54/d55/d8b/l8c to d1/d54/d82/d71/d9e/le7 0 2024-04-12T05:09:06.229 INFO:tasks.workunit.client.0.smithi113.stdout:5/686: readlink d0/d10/d27/l7c 0 2024-04-12T05:09:06.231 INFO:tasks.workunit.client.1.smithi192.stdout:7/530: fdatasync d3/d7/d15/d1a/fac 0 2024-04-12T05:09:06.231 INFO:tasks.workunit.client.1.smithi192.stdout:9/502: symlink d11/d15/d21/d2c/d1f/d6a/d7e/lac 0 2024-04-12T05:09:06.231 INFO:tasks.workunit.client.1.smithi192.stdout:3/581: dread - d5/d13/d3a/d70/d8c/f96 zero size 2024-04-12T05:09:06.232 INFO:tasks.workunit.client.0.smithi113.stdout:5/687: symlink d0/d10/d27/d1d/d3f/db0/le2 0 2024-04-12T05:09:06.232 INFO:tasks.workunit.client.1.smithi192.stdout:5/598: dwrite d0/f10 [0,4194304] 0 2024-04-12T05:09:06.232 INFO:tasks.workunit.client.0.smithi113.stdout:4/620: read da/d3c/d43/f5e [126831,75335] 0 2024-04-12T05:09:06.234 INFO:tasks.workunit.client.1.smithi192.stdout:8/516: rename d6/d10/d46/d5e/d5f/d38/f41 to d6/d10/d46/d7c/fa0 0 2024-04-12T05:09:06.235 INFO:tasks.workunit.client.0.smithi113.stdout:4/621: mknod da/d3c/d83/d47/ccb 0 2024-04-12T05:09:06.235 INFO:tasks.workunit.client.0.smithi113.stdout:5/688: read d0/d10/d27/d1d/d38/fba [2619134,5268] 0 2024-04-12T05:09:06.236 INFO:tasks.workunit.client.1.smithi192.stdout:2/590: mknod d1/d8/d3a/de/d7c/d98/cb0 0 2024-04-12T05:09:06.237 INFO:tasks.workunit.client.1.smithi192.stdout:2/591: readlink d1/da7/lab 0 2024-04-12T05:09:06.237 INFO:tasks.workunit.client.1.smithi192.stdout:0/671: symlink d0/d7b/d9b/d4b/dbc/dce/db0/dbf/le2 0 2024-04-12T05:09:06.241 INFO:tasks.workunit.client.1.smithi192.stdout:7/531: fdatasync d3/d7/d60/d65/fa2 0 2024-04-12T05:09:06.241 INFO:tasks.workunit.client.0.smithi113.stdout:5/689: symlink d0/d10/d27/d1d/d3f/d73/dcd/le3 0 2024-04-12T05:09:06.242 INFO:tasks.workunit.client.1.smithi192.stdout:4/684: mkdir d1/d54/d82/da4/de8 0 2024-04-12T05:09:06.242 INFO:tasks.workunit.client.1.smithi192.stdout:7/532: dread d3/d7/d14/f49 [0,4194304] 0 2024-04-12T05:09:06.244 INFO:tasks.workunit.client.1.smithi192.stdout:5/599: mknod d0/d15/d24/d29/cce 0 2024-04-12T05:09:06.246 INFO:tasks.workunit.client.0.smithi113.stdout:5/690: link d0/d10/d27/l90 d0/d10/d27/d1d/d41/db1/le4 0 2024-04-12T05:09:06.247 INFO:tasks.workunit.client.1.smithi192.stdout:2/592: read d1/d8/d17/f5f [118532,18939] 0 2024-04-12T05:09:06.247 INFO:tasks.workunit.client.0.smithi113.stdout:5/691: rename d0/d10/d27 to d0/d10/d27/d1d/d3f/db0/dda/de5 22 2024-04-12T05:09:06.248 INFO:tasks.workunit.client.1.smithi192.stdout:6/667: rename d3/d8/d10/d4c/f86 to d3/d23/db2/fde 0 2024-04-12T05:09:06.249 INFO:tasks.workunit.client.1.smithi192.stdout:5/600: creat d0/fcf x:0 0 0 2024-04-12T05:09:06.249 INFO:tasks.workunit.client.1.smithi192.stdout:1/600: rename d0/d2/dd/d3c to d0/d2/dd/d3c/d5b/dc5 22 2024-04-12T05:09:06.250 INFO:tasks.workunit.client.1.smithi192.stdout:2/593: stat d1/d8/fc 0 2024-04-12T05:09:06.250 INFO:tasks.workunit.client.0.smithi113.stdout:5/692: symlink d0/d10/d27/d1d/d3f/db0/le6 0 2024-04-12T05:09:06.251 INFO:tasks.workunit.client.1.smithi192.stdout:4/685: dwrite d1/f4b [0,4194304] 0 2024-04-12T05:09:06.252 INFO:tasks.workunit.client.1.smithi192.stdout:5/601: write d0/d15/d24/d29/f2a [4929609,64881] 0 2024-04-12T05:09:06.253 INFO:tasks.workunit.client.1.smithi192.stdout:3/582: rename d5/c35 to d5/d13/d3a/d70/d8c/d8d/ccc 0 2024-04-12T05:09:06.255 INFO:tasks.workunit.client.1.smithi192.stdout:3/583: write d5/d13/d3a/d4e/f9a [993453,97363] 0 2024-04-12T05:09:06.255 INFO:tasks.workunit.client.1.smithi192.stdout:2/594: creat d1/d2d/d73/fb1 x:0 0 0 2024-04-12T05:09:06.256 INFO:tasks.workunit.client.1.smithi192.stdout:7/533: rename d3/d1d/d4a/c63 to d3/d7/d15/cb1 0 2024-04-12T05:09:06.258 INFO:tasks.workunit.client.1.smithi192.stdout:3/584: fdatasync d5/d13/d3c/f68 0 2024-04-12T05:09:06.258 INFO:tasks.workunit.client.1.smithi192.stdout:2/595: fdatasync d1/d8/d3a/f26 0 2024-04-12T05:09:06.262 INFO:tasks.workunit.client.1.smithi192.stdout:1/601: rename d0/d29/c54 to d0/d2/dd/d3c/d5b/d74/d90/cc6 0 2024-04-12T05:09:06.266 INFO:tasks.workunit.client.0.smithi113.stdout:4/622: truncate da/d24/d60/f99 4898764 0 2024-04-12T05:09:06.268 INFO:tasks.workunit.client.1.smithi192.stdout:9/503: rmdir d11/d25 39 2024-04-12T05:09:06.270 INFO:tasks.workunit.client.0.smithi113.stdout:3/600: sync 2024-04-12T05:09:06.272 INFO:tasks.workunit.client.1.smithi192.stdout:5/602: dwrite d0/f1b [0,4194304] 0 2024-04-12T05:09:06.272 INFO:tasks.workunit.client.1.smithi192.stdout:1/602: dread - d0/d2/dd/d3c/f7f zero size 2024-04-12T05:09:06.272 INFO:tasks.workunit.client.1.smithi192.stdout:3/585: mknod d5/d13/d1a/d5f/ccd 0 2024-04-12T05:09:06.273 INFO:tasks.workunit.client.1.smithi192.stdout:2/596: mkdir d1/d2d/d73/da8/db2 0 2024-04-12T05:09:06.274 INFO:tasks.workunit.client.1.smithi192.stdout:6/668: chown d3/d21/d24/f2a 87 1 2024-04-12T05:09:06.275 INFO:tasks.workunit.client.0.smithi113.stdout:3/601: mkdir d0/d13/d47/d4f/dca 0 2024-04-12T05:09:06.276 INFO:tasks.workunit.client.1.smithi192.stdout:4/686: dread d1/d1d/f6c [0,4194304] 0 2024-04-12T05:09:06.277 INFO:tasks.workunit.client.1.smithi192.stdout:6/669: write d3/d8/f1c [6093074,16238] 0 2024-04-12T05:09:06.277 INFO:tasks.workunit.client.1.smithi192.stdout:2/597: chown d1/d8/f10 1857 1 2024-04-12T05:09:06.279 INFO:tasks.workunit.client.0.smithi113.stdout:4/623: rename da/dd/d1b/cc4 to da/d3c/d43/dbc/ccc 0 2024-04-12T05:09:06.284 INFO:tasks.workunit.client.0.smithi113.stdout:4/624: rmdir da/d24/d7e/d79 39 2024-04-12T05:09:06.284 INFO:tasks.workunit.client.0.smithi113.stdout:4/625: stat da 0 2024-04-12T05:09:06.285 INFO:tasks.workunit.client.1.smithi192.stdout:2/598: write d1/f8b [3357142,84170] 0 2024-04-12T05:09:06.286 INFO:tasks.workunit.client.1.smithi192.stdout:7/534: dwrite d3/d1d/f69 [0,4194304] 0 2024-04-12T05:09:06.287 INFO:tasks.workunit.client.0.smithi113.stdout:3/602: fdatasync d0/d13/d15/d28/d2a/d34/f43 0 2024-04-12T05:09:06.288 INFO:tasks.workunit.client.0.smithi113.stdout:4/626: write da/d5a/f91 [4184377,103141] 0 2024-04-12T05:09:06.288 INFO:tasks.workunit.client.0.smithi113.stdout:4/627: readlink da/d24/d7b/lb5 0 2024-04-12T05:09:06.291 INFO:tasks.workunit.client.0.smithi113.stdout:3/603: symlink d0/d80/lcb 0 2024-04-12T05:09:06.293 INFO:tasks.workunit.client.0.smithi113.stdout:4/628: mkdir da/d24/d7e/d79/dcd 0 2024-04-12T05:09:06.296 INFO:tasks.workunit.client.0.smithi113.stdout:3/604: mknod d0/d13/d47/d94/dac/ccc 0 2024-04-12T05:09:06.299 INFO:tasks.workunit.client.1.smithi192.stdout:1/603: creat d0/d2/dd/d21/d62/dbc/fc7 x:0 0 0 2024-04-12T05:09:06.299 INFO:tasks.workunit.client.1.smithi192.stdout:6/670: write d3/dc/d39/fc9 [182783,95000] 0 2024-04-12T05:09:06.299 INFO:tasks.workunit.client.1.smithi192.stdout:9/504: symlink d11/d15/d21/d2c/d1f/d6a/d92/lad 0 2024-04-12T05:09:06.302 INFO:tasks.workunit.client.1.smithi192.stdout:7/535: truncate d3/d11/f37 783380 0 2024-04-12T05:09:06.305 INFO:tasks.workunit.client.1.smithi192.stdout:2/599: dread d1/f44 [0,4194304] 0 2024-04-12T05:09:06.307 INFO:tasks.workunit.client.1.smithi192.stdout:5/603: dwrite d0/d15/f9b [0,4194304] 0 2024-04-12T05:09:06.308 INFO:tasks.workunit.client.1.smithi192.stdout:3/586: dwrite d5/d13/d3a/d4e/d57/faa [0,4194304] 0 2024-04-12T05:09:06.312 INFO:tasks.workunit.client.1.smithi192.stdout:5/604: write d0/d15/d32/d45/fb1 [802056,4477] 0 2024-04-12T05:09:06.317 INFO:tasks.workunit.client.1.smithi192.stdout:5/605: dread - d0/d15/d32/d45/fb0 zero size 2024-04-12T05:09:06.317 INFO:tasks.workunit.client.1.smithi192.stdout:8/517: dread d6/d10/d46/d5e/d5f/d37/f48 [0,4194304] 0 2024-04-12T05:09:06.318 INFO:tasks.workunit.client.1.smithi192.stdout:2/600: dwrite d1/d2d/f5d [0,4194304] 0 2024-04-12T05:09:06.322 INFO:tasks.workunit.client.1.smithi192.stdout:8/518: readlink d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/l76 0 2024-04-12T05:09:06.323 INFO:tasks.workunit.client.0.smithi113.stdout:5/693: dread d0/d10/d27/d1d/d38/d3b/f61 [0,4194304] 0 2024-04-12T05:09:06.323 INFO:tasks.workunit.client.1.smithi192.stdout:8/519: write d6/d1f/d6a/f52 [1460036,64101] 0 2024-04-12T05:09:06.326 INFO:tasks.workunit.client.0.smithi113.stdout:5/694: read d0/d10/d27/d5f/f88 [67658,109392] 0 2024-04-12T05:09:06.330 INFO:tasks.workunit.client.1.smithi192.stdout:0/672: dread d0/d7b/d9b/d4b/dbc/dce/d53/d6d/f9a [0,4194304] 0 2024-04-12T05:09:06.330 INFO:tasks.workunit.client.1.smithi192.stdout:4/687: dread d1/d6b/fd7 [0,4194304] 0 2024-04-12T05:09:06.330 INFO:tasks.workunit.client.0.smithi113.stdout:4/629: dwrite da/d24/f53 [0,4194304] 0 2024-04-12T05:09:06.331 INFO:tasks.workunit.client.1.smithi192.stdout:9/505: dread d11/d15/d21/d2c/d1f/d50/d73/f80 [0,4194304] 0 2024-04-12T05:09:06.333 INFO:tasks.workunit.client.0.smithi113.stdout:4/630: stat da/d70/d80/l9b 0 2024-04-12T05:09:06.336 INFO:tasks.workunit.client.0.smithi113.stdout:4/631: dread da/d4f/f4e [0,4194304] 0 2024-04-12T05:09:06.338 INFO:tasks.workunit.client.0.smithi113.stdout:4/632: unlink da/d3c/d83/l54 0 2024-04-12T05:09:06.341 INFO:tasks.workunit.client.0.smithi113.stdout:4/633: symlink da/d24/d7e/d79/dcd/lce 0 2024-04-12T05:09:06.343 INFO:tasks.workunit.client.0.smithi113.stdout:4/634: creat da/d4f/fcf x:0 0 0 2024-04-12T05:09:06.345 INFO:tasks.workunit.client.0.smithi113.stdout:4/635: unlink da/dd/d3a/c9e 0 2024-04-12T05:09:06.347 INFO:tasks.workunit.client.0.smithi113.stdout:4/636: creat da/d24/d60/db4/fd0 x:0 0 0 2024-04-12T05:09:06.372 INFO:tasks.workunit.client.1.smithi192.stdout:6/671: dread d3/d5/f80 [0,4194304] 0 2024-04-12T05:09:06.386 INFO:tasks.workunit.client.1.smithi192.stdout:1/604: mknod d0/d2/dd/d71/cc8 0 2024-04-12T05:09:06.387 INFO:tasks.workunit.client.1.smithi192.stdout:1/605: chown d0/d2/dd/d3c 6363 1 2024-04-12T05:09:06.392 INFO:tasks.workunit.client.0.smithi113.stdout:4/637: fsync da/d24/d60/fb2 0 2024-04-12T05:09:06.395 INFO:tasks.workunit.client.0.smithi113.stdout:8/656: sync 2024-04-12T05:09:06.397 INFO:tasks.workunit.client.0.smithi113.stdout:2/563: sync 2024-04-12T05:09:06.402 INFO:tasks.workunit.client.0.smithi113.stdout:3/605: dread d0/d13/d15/f2e [0,4194304] 0 2024-04-12T05:09:06.403 INFO:tasks.workunit.client.1.smithi192.stdout:5/606: rename d0/d15/d24/c84 to d0/d15/d3e/dca/db9/cd0 0 2024-04-12T05:09:06.403 INFO:tasks.workunit.client.0.smithi113.stdout:4/638: rmdir da/dd/d1b 39 2024-04-12T05:09:06.403 INFO:tasks.workunit.client.0.smithi113.stdout:5/695: dwrite d0/fc [0,4194304] 0 2024-04-12T05:09:06.405 INFO:tasks.workunit.client.0.smithi113.stdout:2/564: symlink d8/d9/de/d15/d24/d58/lad 0 2024-04-12T05:09:06.408 INFO:tasks.workunit.client.0.smithi113.stdout:3/606: creat d0/d13/d15/d28/d2a/d34/d5f/d7b/fcd x:0 0 0 2024-04-12T05:09:06.410 INFO:tasks.workunit.client.0.smithi113.stdout:2/565: chown d8/d9/de/d12/l8c 30107147 1 2024-04-12T05:09:06.411 INFO:tasks.workunit.client.0.smithi113.stdout:2/566: fdatasync d8/d9/de/d15/d24/d58/f96 0 2024-04-12T05:09:06.413 INFO:tasks.workunit.client.0.smithi113.stdout:5/696: rename d0/d10/d27/d1d/d41/db1 to d0/d10/d27/d1d/d8c/dde/de7 0 2024-04-12T05:09:06.414 INFO:tasks.workunit.client.1.smithi192.stdout:8/520: creat d6/d10/d46/d5e/d5f/d38/d3b/fa1 x:0 0 0 2024-04-12T05:09:06.414 INFO:tasks.workunit.client.1.smithi192.stdout:4/688: mkdir d1/d6b/de9 0 2024-04-12T05:09:06.414 INFO:tasks.workunit.client.1.smithi192.stdout:6/672: truncate d3/d8/d10/d4c/f82 4241213 0 2024-04-12T05:09:06.415 INFO:tasks.workunit.client.1.smithi192.stdout:3/587: creat d5/d13/d3a/d70/d8c/d8d/dbc/fce x:0 0 0 2024-04-12T05:09:06.415 INFO:tasks.workunit.client.1.smithi192.stdout:0/673: mkdir d0/d6/de3 0 2024-04-12T05:09:06.418 INFO:tasks.workunit.client.1.smithi192.stdout:2/601: dwrite d1/f39 [0,4194304] 0 2024-04-12T05:09:06.418 INFO:tasks.workunit.client.1.smithi192.stdout:4/689: chown d1/d54/d55/d64/f91 69 1 2024-04-12T05:09:06.418 INFO:tasks.workunit.client.1.smithi192.stdout:7/536: rmdir d3/d7/d15/d1a 39 2024-04-12T05:09:06.421 INFO:tasks.workunit.client.1.smithi192.stdout:1/606: truncate d0/d2/dd/f32 1324299 0 2024-04-12T05:09:06.421 INFO:tasks.workunit.client.1.smithi192.stdout:8/521: creat d6/d1f/d5d/fa2 x:0 0 0 2024-04-12T05:09:06.422 INFO:tasks.workunit.client.0.smithi113.stdout:2/567: creat d8/d9/de/d15/d24/d58/fae x:0 0 0 2024-04-12T05:09:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:04.672222+0000 mgr.smithi192.ocppyo (mgr.24357) 28 : cluster [DBG] pgmap v22: 65 pgs: 65 active+clean; 2.1 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 60 MiB/s rd, 254 MiB/s wr, 431 op/s 2024-04-12T05:09:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:05.355351+0000 mon.smithi113 (mon.0) 673 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:05.365309+0000 mon.smithi113 (mon.0) 674 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:06.424 INFO:tasks.workunit.client.0.smithi113.stdout:2/568: read d8/d9/d1a/f26 [9885,94164] 0 2024-04-12T05:09:06.428 INFO:tasks.workunit.client.1.smithi192.stdout:4/690: write d1/d54/d55/fd0 [296152,35299] 0 2024-04-12T05:09:06.428 INFO:tasks.workunit.client.0.smithi113.stdout:4/639: link da/d70/d80/c81 da/d24/d7e/d79/cd1 0 2024-04-12T05:09:06.429 INFO:tasks.workunit.client.0.smithi113.stdout:3/607: creat d0/d13/d47/d4f/dca/fce x:0 0 0 2024-04-12T05:09:06.431 INFO:tasks.workunit.client.0.smithi113.stdout:2/569: rename d8/d9/d5b/c6b to d8/d9/d1a/caf 0 2024-04-12T05:09:06.434 INFO:tasks.workunit.client.0.smithi113.stdout:3/608: readlink d0/d13/d15/d28/d2a/d34/l3a 0 2024-04-12T05:09:06.438 INFO:tasks.workunit.client.0.smithi113.stdout:3/609: write d0/d13/f60 [695480,35663] 0 2024-04-12T05:09:06.440 INFO:tasks.workunit.client.0.smithi113.stdout:4/640: dwrite da/d24/d7e/fbe [0,4194304] 0 2024-04-12T05:09:06.441 INFO:tasks.workunit.client.0.smithi113.stdout:2/570: truncate d8/d9/de/d15/f18 1146857 0 2024-04-12T05:09:06.445 INFO:tasks.workunit.client.0.smithi113.stdout:4/641: stat da/d24/d7e/f19 0 2024-04-12T05:09:06.445 INFO:tasks.workunit.client.0.smithi113.stdout:8/657: truncate d9/d1b/fcd 832613 0 2024-04-12T05:09:06.446 INFO:tasks.workunit.client.0.smithi113.stdout:8/658: chown d9/d16 723464966 1 2024-04-12T05:09:06.446 INFO:tasks.workunit.client.1.smithi192.stdout:0/674: dread d0/d6/d15/d1b/d46/f49 [0,4194304] 0 2024-04-12T05:09:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:04.672222+0000 mgr.smithi192.ocppyo (mgr.24357) 28 : cluster [DBG] pgmap v22: 65 pgs: 65 active+clean; 2.1 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 60 MiB/s rd, 254 MiB/s wr, 431 op/s 2024-04-12T05:09:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:05.355351+0000 mon.smithi113 (mon.0) 673 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:05.365309+0000 mon.smithi113 (mon.0) 674 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:06.451 INFO:tasks.workunit.client.0.smithi113.stdout:4/642: creat da/de/fd2 x:0 0 0 2024-04-12T05:09:06.452 INFO:tasks.workunit.client.1.smithi192.stdout:5/607: dread d0/d15/d24/f25 [4194304,4194304] 0 2024-04-12T05:09:06.452 INFO:tasks.workunit.client.0.smithi113.stdout:8/659: dread - d9/d1b/d8e/d93/fb7 zero size 2024-04-12T05:09:06.457 INFO:tasks.workunit.client.0.smithi113.stdout:2/571: creat d8/d9/fb0 x:0 0 0 2024-04-12T05:09:06.458 INFO:tasks.workunit.client.0.smithi113.stdout:4/643: creat da/d70/fd3 x:0 0 0 2024-04-12T05:09:06.462 INFO:tasks.workunit.client.0.smithi113.stdout:4/644: write da/d4a/f5c [2132968,54242] 0 2024-04-12T05:09:06.462 INFO:tasks.workunit.client.1.smithi192.stdout:2/602: mkdir d1/d8/d17/db3 0 2024-04-12T05:09:06.462 INFO:tasks.workunit.client.1.smithi192.stdout:2/603: readlink d1/da7/lab 0 2024-04-12T05:09:06.467 INFO:tasks.workunit.client.0.smithi113.stdout:3/610: write d0/d13/d15/d28/d2a/d34/f43 [1715303,5580] 0 2024-04-12T05:09:06.467 INFO:tasks.workunit.client.0.smithi113.stdout:5/697: dwrite d0/d10/d27/d1d/f49 [4194304,4194304] 0 2024-04-12T05:09:06.469 INFO:tasks.workunit.client.0.smithi113.stdout:8/660: creat d9/d16/fd3 x:0 0 0 2024-04-12T05:09:06.470 INFO:tasks.workunit.client.0.smithi113.stdout:3/611: write d0/fa [499361,10724] 0 2024-04-12T05:09:06.471 INFO:tasks.workunit.client.1.smithi192.stdout:7/537: write d3/d1d/f9f [5300052,130687] 0 2024-04-12T05:09:06.472 INFO:tasks.workunit.client.0.smithi113.stdout:5/698: write d0/d10/d27/d1d/d38/f52 [26580,129655] 0 2024-04-12T05:09:06.473 INFO:tasks.workunit.client.0.smithi113.stdout:8/661: mkdir d9/dd4 0 2024-04-12T05:09:06.474 INFO:tasks.workunit.client.1.smithi192.stdout:1/607: dread d0/d2/dd/d21/d33/f6b [0,4194304] 0 2024-04-12T05:09:06.475 INFO:tasks.workunit.client.0.smithi113.stdout:4/645: read da/dd/f48 [608907,105197] 0 2024-04-12T05:09:06.481 INFO:tasks.workunit.client.1.smithi192.stdout:0/675: creat d0/d6/d15/d55/d29/db5/fe4 x:0 0 0 2024-04-12T05:09:06.483 INFO:tasks.workunit.client.0.smithi113.stdout:5/699: symlink d0/d10/d27/d5f/le8 0 2024-04-12T05:09:06.485 INFO:tasks.workunit.client.0.smithi113.stdout:5/700: write d0/d10/d6e/f3c [4221019,45993] 0 2024-04-12T05:09:06.491 INFO:tasks.workunit.client.0.smithi113.stdout:5/701: dwrite d0/fa8 [0,4194304] 0 2024-04-12T05:09:06.493 INFO:tasks.workunit.client.0.smithi113.stdout:5/702: read - d0/d10/d27/d1d/d3f/d73/d89/f5a zero size 2024-04-12T05:09:06.497 INFO:tasks.workunit.client.0.smithi113.stdout:5/703: link d0/d10/d27/d1d/d3f/c54 d0/d10/ce9 0 2024-04-12T05:09:06.501 INFO:tasks.workunit.client.0.smithi113.stdout:2/572: dread f7 [0,4194304] 0 2024-04-12T05:09:06.501 INFO:tasks.workunit.client.0.smithi113.stdout:2/573: readlink d8/d9/d1a/l71 0 2024-04-12T05:09:06.507 INFO:tasks.workunit.client.0.smithi113.stdout:4/646: write da/dd/d1b/fb1 [5045019,113556] 0 2024-04-12T05:09:06.507 INFO:tasks.workunit.client.0.smithi113.stdout:2/574: getdents d8/d9/d5b 0 2024-04-12T05:09:06.510 INFO:tasks.workunit.client.0.smithi113.stdout:2/575: unlink d8/d16/f19 0 2024-04-12T05:09:06.511 INFO:tasks.workunit.client.0.smithi113.stdout:4/647: unlink da/d3c/d9f/ca4 0 2024-04-12T05:09:06.512 INFO:tasks.workunit.client.0.smithi113.stdout:2/576: chown d8/d9/de/d12/d2b/fa1 5843000 1 2024-04-12T05:09:06.512 INFO:tasks.workunit.client.0.smithi113.stdout:4/648: fsync da/d24/d7e/f19 0 2024-04-12T05:09:06.521 INFO:tasks.workunit.client.0.smithi113.stdout:5/704: truncate d0/d10/d27/d1d/d38/d3b/d6b/fd4 2547282 0 2024-04-12T05:09:06.523 INFO:tasks.workunit.client.0.smithi113.stdout:5/705: mknod d0/d10/d27/d1d/d8c/cea 0 2024-04-12T05:09:06.528 INFO:tasks.workunit.client.0.smithi113.stdout:5/706: dwrite d0/d10/d27/d1d/d8c/fa7 [0,4194304] 0 2024-04-12T05:09:06.529 INFO:tasks.workunit.client.0.smithi113.stdout:4/649: write da/dd/f32 [4403802,83343] 0 2024-04-12T05:09:06.534 INFO:tasks.workunit.client.0.smithi113.stdout:4/650: truncate da/d3c/d43/f75 445232 0 2024-04-12T05:09:06.537 INFO:tasks.workunit.client.0.smithi113.stdout:2/577: dwrite d8/d9/de/d12/d2b/fa1 [0,4194304] 0 2024-04-12T05:09:06.538 INFO:tasks.workunit.client.0.smithi113.stdout:7/684: sync 2024-04-12T05:09:06.538 INFO:tasks.workunit.client.0.smithi113.stdout:9/667: sync 2024-04-12T05:09:06.541 INFO:tasks.workunit.client.0.smithi113.stdout:2/578: write d8/d16/f17 [2301339,98579] 0 2024-04-12T05:09:06.542 INFO:tasks.workunit.client.0.smithi113.stdout:2/579: stat d8/d9/d5b 0 2024-04-12T05:09:06.546 INFO:tasks.workunit.client.1.smithi192.stdout:6/673: write d3/d8/f17 [3329059,122515] 0 2024-04-12T05:09:06.546 INFO:tasks.workunit.client.1.smithi192.stdout:7/538: mknod d3/d7/d14/d1f/cb2 0 2024-04-12T05:09:06.546 INFO:tasks.workunit.client.0.smithi113.stdout:7/685: unlink d0/dc/da9/l59 0 2024-04-12T05:09:06.547 INFO:tasks.workunit.client.1.smithi192.stdout:9/506: sync 2024-04-12T05:09:06.549 INFO:tasks.workunit.client.1.smithi192.stdout:8/522: mknod d6/d10/d46/d5e/d5f/d31/ca3 0 2024-04-12T05:09:06.552 INFO:tasks.workunit.client.0.smithi113.stdout:7/686: creat d0/dc/d6/d9b/dcd/feb x:0 0 0 2024-04-12T05:09:06.552 INFO:tasks.workunit.client.0.smithi113.stdout:4/651: mknod da/d24/d7e/cd4 0 2024-04-12T05:09:06.553 INFO:tasks.workunit.client.0.smithi113.stdout:7/687: chown d0/dd/d11/f20 1607 1 2024-04-12T05:09:06.554 INFO:tasks.workunit.client.1.smithi192.stdout:3/588: symlink d5/d13/d3a/d4e/lcf 0 2024-04-12T05:09:06.554 INFO:tasks.workunit.client.0.smithi113.stdout:2/580: creat d8/d9/d27/d3f/fb1 x:0 0 0 2024-04-12T05:09:06.557 INFO:tasks.workunit.client.0.smithi113.stdout:7/688: fdatasync d0/d6c/fb0 0 2024-04-12T05:09:06.560 INFO:tasks.workunit.client.0.smithi113.stdout:4/652: mkdir da/d3c/dd5 0 2024-04-12T05:09:06.563 INFO:tasks.workunit.client.0.smithi113.stdout:2/581: mkdir d8/d9/de/d43/db2 0 2024-04-12T05:09:06.563 INFO:tasks.workunit.client.1.smithi192.stdout:9/507: mknod d11/d15/d21/d2c/d1f/cae 0 2024-04-12T05:09:06.565 INFO:tasks.workunit.client.0.smithi113.stdout:7/689: symlink d0/dc/d6/d1a/da0/dad/lec 0 2024-04-12T05:09:06.568 INFO:tasks.workunit.client.1.smithi192.stdout:6/674: rmdir d3/d8/d56/d59/d97/dbf 39 2024-04-12T05:09:06.571 INFO:tasks.workunit.client.1.smithi192.stdout:0/676: symlink d0/dc5/le5 0 2024-04-12T05:09:06.572 INFO:tasks.workunit.client.0.smithi113.stdout:4/653: dwrite da/d4f/fcf [0,4194304] 0 2024-04-12T05:09:06.575 INFO:tasks.workunit.client.0.smithi113.stdout:9/668: dread d1/d27/f5d [0,4194304] 0 2024-04-12T05:09:06.575 INFO:tasks.workunit.client.0.smithi113.stdout:7/690: rename d0/d52/f74 to d0/dc/da9/d5c/d85/fed 0 2024-04-12T05:09:06.577 INFO:tasks.workunit.client.0.smithi113.stdout:7/691: readlink d0/dc/d6/d1a/l46 0 2024-04-12T05:09:06.579 INFO:tasks.workunit.client.1.smithi192.stdout:1/608: link d0/d2/dd/d3c/d5b/d7c/la8 d0/d2/d1e/d3f/d56/d9e/lc9 0 2024-04-12T05:09:06.579 INFO:tasks.workunit.client.0.smithi113.stdout:5/707: dread d0/d10/d27/d1d/d3f/d73/d89/f40 [0,4194304] 0 2024-04-12T05:09:06.581 INFO:tasks.workunit.client.0.smithi113.stdout:2/582: getdents d8/d9/de/d12/d44 0 2024-04-12T05:09:06.581 INFO:tasks.workunit.client.1.smithi192.stdout:3/589: link d5/d13/d3c/d36/f8a d5/d13/d94/fd0 0 2024-04-12T05:09:06.587 INFO:tasks.workunit.client.0.smithi113.stdout:4/654: link da/d24/d7b/lc0 da/d24/d7e/d79/dcd/ld6 0 2024-04-12T05:09:06.587 INFO:tasks.workunit.client.0.smithi113.stdout:7/692: creat d0/dd/d11/de6/dbb/fee x:0 0 0 2024-04-12T05:09:06.587 INFO:tasks.workunit.client.1.smithi192.stdout:7/539: rename d3/d7/d60/c97 to d3/d1d/cb3 0 2024-04-12T05:09:06.589 INFO:tasks.workunit.client.1.smithi192.stdout:7/540: chown d3/d7/d60/fa0 36318 1 2024-04-12T05:09:06.590 INFO:tasks.workunit.client.0.smithi113.stdout:2/583: rename d8/d9/d27/d3f to d8/d9/de/d12/d2b/daa/db3 0 2024-04-12T05:09:06.591 INFO:tasks.workunit.client.0.smithi113.stdout:9/669: dwrite d1/d27/d7a/fde [0,4194304] 0 2024-04-12T05:09:06.593 INFO:tasks.workunit.client.0.smithi113.stdout:4/655: creat da/d3c/d83/fd7 x:0 0 0 2024-04-12T05:09:06.593 INFO:tasks.workunit.client.0.smithi113.stdout:7/693: fsync d0/f48 0 2024-04-12T05:09:06.593 INFO:tasks.workunit.client.1.smithi192.stdout:0/677: creat d0/d6/d67/db1/dd6/fe6 x:0 0 0 2024-04-12T05:09:06.596 INFO:tasks.workunit.client.0.smithi113.stdout:2/584: chown d8/d9/de/d15/d38/f51 6 1 2024-04-12T05:09:06.596 INFO:tasks.workunit.client.1.smithi192.stdout:3/590: creat d5/d13/d3c/d36/da0/fd1 x:0 0 0 2024-04-12T05:09:06.600 INFO:tasks.workunit.client.0.smithi113.stdout:5/708: dwrite d0/d10/d27/d1d/d38/d44/fbc [0,4194304] 0 2024-04-12T05:09:06.601 INFO:tasks.workunit.client.0.smithi113.stdout:7/694: creat d0/dc/d6/d9b/fef x:0 0 0 2024-04-12T05:09:06.603 INFO:tasks.workunit.client.1.smithi192.stdout:8/523: getdents d6/d10/d46/d5e/d5f/d37 0 2024-04-12T05:09:06.604 INFO:tasks.workunit.client.0.smithi113.stdout:5/709: write d0/d10/d27/d1d/d38/f52 [1126072,110588] 0 2024-04-12T05:09:06.606 INFO:tasks.workunit.client.1.smithi192.stdout:0/678: stat d0/c52 0 2024-04-12T05:09:06.607 INFO:tasks.workunit.client.0.smithi113.stdout:2/585: unlink d8/d9/de/d12/d44/f89 0 2024-04-12T05:09:06.608 INFO:tasks.workunit.client.0.smithi113.stdout:2/586: write d8/d9/de/f22 [1033152,118601] 0 2024-04-12T05:09:06.609 INFO:tasks.workunit.client.0.smithi113.stdout:2/587: stat d8/d9/d1a/l71 0 2024-04-12T05:09:06.610 INFO:tasks.workunit.client.0.smithi113.stdout:9/670: link d1/d4/d5/d22/ca3 d1/d4/d6d/ce1 0 2024-04-12T05:09:06.611 INFO:tasks.workunit.client.1.smithi192.stdout:0/679: chown d0/fad 3 1 2024-04-12T05:09:06.612 INFO:tasks.workunit.client.1.smithi192.stdout:6/675: getdents d3/d8/d48 0 2024-04-12T05:09:06.614 INFO:tasks.workunit.client.1.smithi192.stdout:7/541: unlink d3/d7/c2f 0 2024-04-12T05:09:06.615 INFO:tasks.workunit.client.1.smithi192.stdout:2/604: symlink d1/d2d/lb4 0 2024-04-12T05:09:06.616 INFO:tasks.workunit.client.0.smithi113.stdout:7/695: mknod d0/cf0 0 2024-04-12T05:09:06.617 INFO:tasks.workunit.client.1.smithi192.stdout:5/608: symlink d0/d15/d3e/ld1 0 2024-04-12T05:09:06.618 INFO:tasks.workunit.client.0.smithi113.stdout:7/696: fsync d0/dc/fde 0 2024-04-12T05:09:06.619 INFO:tasks.workunit.client.1.smithi192.stdout:9/508: getdents d11/d25/d36 0 2024-04-12T05:09:06.620 INFO:tasks.workunit.client.1.smithi192.stdout:1/609: getdents d0/d2/dd/d21 0 2024-04-12T05:09:06.623 INFO:tasks.workunit.client.0.smithi113.stdout:5/710: link d0/d10/d27/d1d/d38/d44/d66/f96 d0/d10/d27/d1d/d3f/db0/feb 0 2024-04-12T05:09:06.623 INFO:tasks.workunit.client.1.smithi192.stdout:8/524: write d6/d10/d46/d5e/d5f/d31/d61/f74 [902432,623] 0 2024-04-12T05:09:06.625 INFO:tasks.workunit.client.0.smithi113.stdout:7/697: symlink d0/dc/de7/lf1 0 2024-04-12T05:09:06.627 INFO:tasks.workunit.client.1.smithi192.stdout:5/609: fsync d0/d15/d24/d47/f4d 0 2024-04-12T05:09:06.627 INFO:tasks.workunit.client.1.smithi192.stdout:6/676: mkdir d3/d8/d56/d59/ddf 0 2024-04-12T05:09:06.627 INFO:tasks.workunit.client.1.smithi192.stdout:1/610: chown d0/d1b/l27 637669058 1 2024-04-12T05:09:06.630 INFO:tasks.workunit.client.1.smithi192.stdout:8/525: write f5 [472252,105692] 0 2024-04-12T05:09:06.631 INFO:tasks.workunit.client.0.smithi113.stdout:4/656: dwrite da/de/f62 [0,4194304] 0 2024-04-12T05:09:06.633 INFO:tasks.workunit.client.1.smithi192.stdout:2/605: symlink d1/d8/d17/d19/d71/da0/lb5 0 2024-04-12T05:09:06.634 INFO:tasks.workunit.client.0.smithi113.stdout:2/588: write d8/d9/de/d12/f4c [3247584,73972] 0 2024-04-12T05:09:06.634 INFO:tasks.workunit.client.0.smithi113.stdout:4/657: read da/d3c/d83/f8a [1954022,101194] 0 2024-04-12T05:09:06.635 INFO:tasks.workunit.client.0.smithi113.stdout:7/698: rename d0/dc/d6/d1a/da0/dad to d0/dc/d6/d78/df2 0 2024-04-12T05:09:06.636 INFO:tasks.workunit.client.0.smithi113.stdout:9/671: dread d1/d4/f77 [0,4194304] 0 2024-04-12T05:09:06.637 INFO:tasks.workunit.client.1.smithi192.stdout:4/691: dwrite d1/d54/d55/d64/f74 [0,4194304] 0 2024-04-12T05:09:06.637 INFO:tasks.workunit.client.1.smithi192.stdout:0/680: dwrite d0/d7b/d9b/d4b/dbc/dce/d53/d6d/f62 [0,4194304] 0 2024-04-12T05:09:06.642 INFO:tasks.workunit.client.0.smithi113.stdout:2/589: creat d8/d9/de/d12/d3c/fb4 x:0 0 0 2024-04-12T05:09:06.643 INFO:tasks.workunit.client.0.smithi113.stdout:5/711: write d0/d10/d27/d1d/d41/f9e [957199,20573] 0 2024-04-12T05:09:06.644 INFO:tasks.workunit.client.0.smithi113.stdout:5/712: read d0/d10/d6e/f87 [479260,28333] 0 2024-04-12T05:09:06.649 INFO:tasks.workunit.client.1.smithi192.stdout:7/542: link d3/d7/d14/f92 d3/d7/fb4 0 2024-04-12T05:09:06.649 INFO:tasks.workunit.client.1.smithi192.stdout:5/610: creat d0/d15/d32/db2/fd2 x:0 0 0 2024-04-12T05:09:06.650 INFO:tasks.workunit.client.1.smithi192.stdout:0/681: write d0/d6/d15/d55/f6c [951166,74567] 0 2024-04-12T05:09:06.650 INFO:tasks.workunit.client.1.smithi192.stdout:6/677: mknod d3/d21/d24/ce0 0 2024-04-12T05:09:06.650 INFO:tasks.workunit.client.1.smithi192.stdout:8/526: dwrite d6/d1f/f2a [0,4194304] 0 2024-04-12T05:09:06.653 INFO:tasks.workunit.client.1.smithi192.stdout:8/527: chown d6/l17 6 1 2024-04-12T05:09:06.653 INFO:tasks.workunit.client.1.smithi192.stdout:9/509: mknod d11/d15/d21/d2c/d1f/d6a/d92/da6/caf 0 2024-04-12T05:09:06.654 INFO:tasks.workunit.client.1.smithi192.stdout:2/606: mknod d1/d2d/d73/da8/cb6 0 2024-04-12T05:09:06.656 INFO:tasks.workunit.client.0.smithi113.stdout:2/590: unlink f7 0 2024-04-12T05:09:06.659 INFO:tasks.workunit.client.1.smithi192.stdout:6/678: truncate d3/dc/d39/fc9 464215 0 2024-04-12T05:09:06.659 INFO:tasks.workunit.client.1.smithi192.stdout:3/591: truncate d5/f19 4238296 0 2024-04-12T05:09:06.659 INFO:tasks.workunit.client.1.smithi192.stdout:5/611: mkdir d0/d15/d3e/dca/dd3 0 2024-04-12T05:09:06.663 INFO:tasks.workunit.client.1.smithi192.stdout:3/592: stat d5/d13/d94/fd0 0 2024-04-12T05:09:06.663 INFO:tasks.workunit.client.1.smithi192.stdout:4/692: mkdir d1/d54/d55/d64/d68/ddd/dea 0 2024-04-12T05:09:06.663 INFO:tasks.workunit.client.1.smithi192.stdout:6/679: chown d3/d26/c2b 1922729 1 2024-04-12T05:09:06.665 INFO:tasks.workunit.client.0.smithi113.stdout:7/699: link d0/dd/d11/f77 d0/dc/d6/d9b/ff3 0 2024-04-12T05:09:06.666 INFO:tasks.workunit.client.0.smithi113.stdout:4/658: rename da/d70/d80/c89 to da/d3c/da5/dad/cd8 0 2024-04-12T05:09:06.667 INFO:tasks.workunit.client.0.smithi113.stdout:5/713: getdents d0/d10/d6e/d93/dc9 0 2024-04-12T05:09:06.671 INFO:tasks.workunit.client.0.smithi113.stdout:5/714: write d0/d10/d6e/fb5 [1726309,33383] 0 2024-04-12T05:09:06.672 INFO:tasks.workunit.client.0.smithi113.stdout:4/659: mkdir da/d3c/d9a/dc1/dd9 0 2024-04-12T05:09:06.674 INFO:tasks.workunit.client.1.smithi192.stdout:4/693: creat d1/d54/d55/d64/feb x:0 0 0 2024-04-12T05:09:06.675 INFO:tasks.workunit.client.0.smithi113.stdout:9/672: dread d1/d27/f95 [0,4194304] 0 2024-04-12T05:09:06.675 INFO:tasks.workunit.client.0.smithi113.stdout:7/700: read d0/dc/d6/d78/f38 [416353,124903] 0 2024-04-12T05:09:06.676 INFO:tasks.workunit.client.1.smithi192.stdout:9/510: read d11/d25/d36/d3d/f5e [1843846,84853] 0 2024-04-12T05:09:06.676 INFO:tasks.workunit.client.0.smithi113.stdout:5/715: rename d0/d10/d27/d5f/f65 to d0/d10/d27/d1d/d8c/db8/dc6/fec 0 2024-04-12T05:09:06.677 INFO:tasks.workunit.client.0.smithi113.stdout:2/591: link d8/d9/de/d12/d2b/daa/db3/c7c d8/d9/de/d12/d44/cb5 0 2024-04-12T05:09:06.677 INFO:tasks.workunit.client.0.smithi113.stdout:9/673: truncate d1/d4/f77 4870294 0 2024-04-12T05:09:06.683 INFO:tasks.workunit.client.1.smithi192.stdout:0/682: dwrite d0/d7b/d9b/d4b/dbc/dce/d95/db4/db8/fcb [0,4194304] 0 2024-04-12T05:09:06.683 INFO:tasks.workunit.client.1.smithi192.stdout:7/543: getdents d3/d1d/d4a/d5e 0 2024-04-12T05:09:06.684 INFO:tasks.workunit.client.0.smithi113.stdout:9/674: read d1/d4/d30/dcb/f81 [12384,1294] 0 2024-04-12T05:09:06.685 INFO:tasks.workunit.client.1.smithi192.stdout:5/612: rename d0/d15/d24/d29/c93 to d0/d15/d32/db2/cd4 0 2024-04-12T05:09:06.687 INFO:tasks.workunit.client.1.smithi192.stdout:2/607: dwrite d1/d8/d17/d19/d60/f76 [4194304,4194304] 0 2024-04-12T05:09:06.689 INFO:tasks.workunit.client.0.smithi113.stdout:2/592: dwrite d8/d9/de/d15/d24/d58/f75 [0,4194304] 0 2024-04-12T05:09:06.689 INFO:tasks.workunit.client.1.smithi192.stdout:1/611: dread d0/d1b/d44/d69/fb7 [0,4194304] 0 2024-04-12T05:09:06.691 INFO:tasks.workunit.client.1.smithi192.stdout:4/694: unlink d1/d54/d82/fb3 0 2024-04-12T05:09:06.694 INFO:tasks.workunit.client.1.smithi192.stdout:5/613: write d0/d15/d32/d69/d7d/fcc [1028114,3432] 0 2024-04-12T05:09:06.694 INFO:tasks.workunit.client.1.smithi192.stdout:7/544: truncate d3/d11/f37 1311024 0 2024-04-12T05:09:06.704 INFO:tasks.workunit.client.1.smithi192.stdout:8/528: link d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/l76 d6/d10/d6c/d9e/la4 0 2024-04-12T05:09:06.706 INFO:tasks.workunit.client.1.smithi192.stdout:0/683: chown d0/c41 295051640 1 2024-04-12T05:09:06.707 INFO:tasks.workunit.client.1.smithi192.stdout:2/608: dwrite d1/d8/d17/d19/d71/d7f/f96 [0,4194304] 0 2024-04-12T05:09:06.708 INFO:tasks.workunit.client.1.smithi192.stdout:7/545: creat d3/d7/d60/d65/fb5 x:0 0 0 2024-04-12T05:09:06.708 INFO:tasks.workunit.client.1.smithi192.stdout:9/511: link d11/d15/d21/d2c/f1a d11/d15/d21/fb0 0 2024-04-12T05:09:06.711 INFO:tasks.workunit.client.1.smithi192.stdout:1/612: read d0/d2/dd/fa9 [2488590,43672] 0 2024-04-12T05:09:06.715 INFO:tasks.workunit.client.1.smithi192.stdout:6/680: write d3/d5/f66 [4335547,99414] 0 2024-04-12T05:09:06.716 INFO:tasks.workunit.client.1.smithi192.stdout:2/609: creat d1/d8/d3a/de/d7c/d98/fb7 x:0 0 0 2024-04-12T05:09:06.717 INFO:tasks.workunit.client.1.smithi192.stdout:1/613: unlink d0/d2/d1e/d3f/d56/d9e/l9a 0 2024-04-12T05:09:06.718 INFO:tasks.workunit.client.1.smithi192.stdout:9/512: mknod d11/d15/d21/d2c/d1f/d50/cb1 0 2024-04-12T05:09:06.719 INFO:tasks.workunit.client.1.smithi192.stdout:0/684: rename d0/d6/d15/d55/d2e/f42 to d0/d7b/d9b/d4b/dbc/dce/d53/fe7 0 2024-04-12T05:09:06.722 INFO:tasks.workunit.client.1.smithi192.stdout:7/546: creat d3/d7/d15/d22/fb6 x:0 0 0 2024-04-12T05:09:06.723 INFO:tasks.workunit.client.1.smithi192.stdout:3/593: write d5/d13/d94/fd0 [5131310,29373] 0 2024-04-12T05:09:06.724 INFO:tasks.workunit.client.1.smithi192.stdout:8/529: truncate d6/f95 1413713 0 2024-04-12T05:09:06.724 INFO:tasks.workunit.client.1.smithi192.stdout:2/610: creat d1/d8/d3a/de/d7c/fb8 x:0 0 0 2024-04-12T05:09:06.726 INFO:tasks.workunit.client.1.smithi192.stdout:0/685: mknod d0/d6/d15/d55/d2e/d66/ce8 0 2024-04-12T05:09:06.727 INFO:tasks.workunit.client.1.smithi192.stdout:7/547: chown d3/d7/d14/d1f/cb2 281025 1 2024-04-12T05:09:06.727 INFO:tasks.workunit.client.1.smithi192.stdout:9/513: creat d11/d15/d21/d2c/d1f/d50/d73/fb2 x:0 0 0 2024-04-12T05:09:06.730 INFO:tasks.workunit.client.0.smithi113.stdout:4/660: truncate da/d4f/fcf 715081 0 2024-04-12T05:09:06.730 INFO:tasks.workunit.client.0.smithi113.stdout:5/716: write d0/d10/d6e/f4a [899068,116654] 0 2024-04-12T05:09:06.730 INFO:tasks.workunit.client.0.smithi113.stdout:2/593: write d8/d9/de/d12/d3c/d8a/fa2 [1040979,116419] 0 2024-04-12T05:09:06.732 INFO:tasks.workunit.client.0.smithi113.stdout:2/594: write d8/d9/de/d12/d2b/f8e [790108,90749] 0 2024-04-12T05:09:06.732 INFO:tasks.workunit.client.1.smithi192.stdout:1/614: creat d0/d2/dd/d3c/d5b/da7/fca x:0 0 0 2024-04-12T05:09:06.733 INFO:tasks.workunit.client.1.smithi192.stdout:3/594: creat d5/d13/d3a/d70/fd2 x:0 0 0 2024-04-12T05:09:06.734 INFO:tasks.workunit.client.0.smithi113.stdout:9/675: getdents d1/df/d4b/d63/dad 0 2024-04-12T05:09:06.735 INFO:tasks.workunit.client.0.smithi113.stdout:4/661: rmdir da/d24/d60/db8 39 2024-04-12T05:09:06.735 INFO:tasks.workunit.client.0.smithi113.stdout:9/676: write d1/d4/d38/f8c [2406173,58760] 0 2024-04-12T05:09:06.736 INFO:tasks.workunit.client.0.smithi113.stdout:4/662: chown da/d3c/d83/d47 579465765 1 2024-04-12T05:09:06.737 INFO:tasks.workunit.client.0.smithi113.stdout:2/595: chown d8/d9/d27/d88/f95 85073185 1 2024-04-12T05:09:06.738 INFO:tasks.workunit.client.1.smithi192.stdout:3/595: read - d5/d13/d94/fc6 zero size 2024-04-12T05:09:06.738 INFO:tasks.workunit.client.0.smithi113.stdout:2/596: write d8/d9/fb0 [154254,52726] 0 2024-04-12T05:09:06.741 INFO:tasks.workunit.client.1.smithi192.stdout:2/611: symlink d1/d8/d17/d19/d60/d9c/lb9 0 2024-04-12T05:09:06.741 INFO:tasks.workunit.client.1.smithi192.stdout:8/530: dread d6/d10/f78 [0,4194304] 0 2024-04-12T05:09:06.741 INFO:tasks.workunit.client.0.smithi113.stdout:9/677: unlink d1/df/db6/ca7 0 2024-04-12T05:09:06.742 INFO:tasks.workunit.client.0.smithi113.stdout:4/663: rmdir da/dd 39 2024-04-12T05:09:06.743 INFO:tasks.workunit.client.1.smithi192.stdout:1/615: chown d0/d2/dd/d3c/d5b/d74/d90/f94 30 1 2024-04-12T05:09:06.743 INFO:tasks.workunit.client.1.smithi192.stdout:4/695: dwrite d1/d54/f9a [0,4194304] 0 2024-04-12T05:09:06.743 INFO:tasks.workunit.client.0.smithi113.stdout:4/664: write da/d24/d60/db4/fd0 [986875,97633] 0 2024-04-12T05:09:06.744 INFO:tasks.workunit.client.1.smithi192.stdout:9/514: creat d11/d25/d36/d3d/fb3 x:0 0 0 2024-04-12T05:09:06.745 INFO:tasks.workunit.client.1.smithi192.stdout:3/596: creat d5/d13/d3a/d70/fd3 x:0 0 0 2024-04-12T05:09:06.748 INFO:tasks.workunit.client.1.smithi192.stdout:2/612: fsync d1/d8/d17/d19/d60/f93 0 2024-04-12T05:09:06.752 INFO:tasks.workunit.client.0.smithi113.stdout:2/597: rmdir d8/d9/de 39 2024-04-12T05:09:06.755 INFO:tasks.workunit.client.0.smithi113.stdout:9/678: mknod d1/df/d4b/ce2 0 2024-04-12T05:09:06.756 INFO:tasks.workunit.client.1.smithi192.stdout:8/531: creat d6/d10/fa5 x:0 0 0 2024-04-12T05:09:06.756 INFO:tasks.workunit.client.0.smithi113.stdout:2/598: creat d8/d9/de/d12/d2b/fb6 x:0 0 0 2024-04-12T05:09:06.756 INFO:tasks.workunit.client.1.smithi192.stdout:1/616: symlink d0/d1b/lcb 0 2024-04-12T05:09:06.758 INFO:tasks.workunit.client.0.smithi113.stdout:9/679: creat d1/d4/d38/d65/dcd/dd7/fe3 x:0 0 0 2024-04-12T05:09:06.759 INFO:tasks.workunit.client.1.smithi192.stdout:8/532: chown d6/d10/d46/d5e 392675789 1 2024-04-12T05:09:06.759 INFO:tasks.workunit.client.1.smithi192.stdout:9/515: stat d11/d25/d36/d6e/d33/da2/f39 0 2024-04-12T05:09:06.760 INFO:tasks.workunit.client.1.smithi192.stdout:2/613: unlink d1/d2d/l58 0 2024-04-12T05:09:06.761 INFO:tasks.workunit.client.0.smithi113.stdout:2/599: creat d8/d9/d27/fb7 x:0 0 0 2024-04-12T05:09:06.762 INFO:tasks.workunit.client.0.smithi113.stdout:9/680: rename d1/df/d4b/d63/d3d/dab to d1/d27/d7a/de4 0 2024-04-12T05:09:06.763 INFO:tasks.workunit.client.1.smithi192.stdout:3/597: mknod d5/d13/d3a/d70/d8c/cd4 0 2024-04-12T05:09:06.764 INFO:tasks.workunit.client.0.smithi113.stdout:2/600: creat d8/d9/de/d12/d2b/d79/fb8 x:0 0 0 2024-04-12T05:09:06.764 INFO:tasks.workunit.client.1.smithi192.stdout:6/681: dread d3/d8/d10/d4c/d68/f77 [0,4194304] 0 2024-04-12T05:09:06.766 INFO:tasks.workunit.client.0.smithi113.stdout:9/681: rename d1/d4/d38/d65/lc5 to d1/d27/le5 0 2024-04-12T05:09:06.767 INFO:tasks.workunit.client.1.smithi192.stdout:4/696: creat d1/d1d/d53/d1c/de1/fec x:0 0 0 2024-04-12T05:09:06.768 INFO:tasks.workunit.client.1.smithi192.stdout:3/598: fsync d5/d13/d3a/d70/d8c/d8d/dbc/fce 0 2024-04-12T05:09:06.768 INFO:tasks.workunit.client.1.smithi192.stdout:1/617: fsync d0/d2/dd/d3c/d5b/da7/fca 0 2024-04-12T05:09:06.770 INFO:tasks.workunit.client.0.smithi113.stdout:2/601: dwrite d8/d9/f69 [0,4194304] 0 2024-04-12T05:09:06.771 INFO:tasks.workunit.client.0.smithi113.stdout:2/602: chown d8/d9/de/d12/d44/c7d 57073 1 2024-04-12T05:09:06.772 INFO:tasks.workunit.client.1.smithi192.stdout:9/516: symlink d11/d25/d36/d6e/lb4 0 2024-04-12T05:09:06.773 INFO:tasks.workunit.client.1.smithi192.stdout:6/682: chown d3/dc/d39/c55 63173924 1 2024-04-12T05:09:06.774 INFO:tasks.workunit.client.0.smithi113.stdout:0/724: sync 2024-04-12T05:09:06.774 INFO:tasks.workunit.client.0.smithi113.stdout:8/662: sync 2024-04-12T05:09:06.774 INFO:tasks.workunit.client.0.smithi113.stdout:6/644: sync 2024-04-12T05:09:06.774 INFO:tasks.workunit.client.0.smithi113.stdout:3/612: sync 2024-04-12T05:09:06.774 INFO:tasks.workunit.client.0.smithi113.stdout:1/778: sync 2024-04-12T05:09:06.775 INFO:tasks.workunit.client.1.smithi192.stdout:7/548: dread d3/d7/d15/f77 [0,4194304] 0 2024-04-12T05:09:06.775 INFO:tasks.workunit.client.0.smithi113.stdout:3/613: chown d0/d7e/f92 0 1 2024-04-12T05:09:06.777 INFO:tasks.workunit.client.0.smithi113.stdout:6/645: chown d8/f22 124 1 2024-04-12T05:09:06.777 INFO:tasks.workunit.client.0.smithi113.stdout:1/779: creat d10/d41/d89/d96/dd3/ffd x:0 0 0 2024-04-12T05:09:06.780 INFO:tasks.workunit.client.1.smithi192.stdout:7/549: readlink d3/d1d/d5a/d5f/l3a 0 2024-04-12T05:09:06.784 INFO:tasks.workunit.client.1.smithi192.stdout:9/517: dwrite d11/d25/d36/d6e/d33/fab [0,4194304] 0 2024-04-12T05:09:06.785 INFO:tasks.workunit.client.1.smithi192.stdout:9/518: chown d11/d15/d21/d2c/d1f/la3 884912 1 2024-04-12T05:09:06.787 INFO:tasks.workunit.client.1.smithi192.stdout:9/519: dread - d11/d15/d21/d2c/d1f/d6a/f7c zero size 2024-04-12T05:09:06.792 INFO:tasks.workunit.client.0.smithi113.stdout:8/663: chown d9/d1b/d8e/d93/d32/d48/d8d/dae/cb6 13588141 1 2024-04-12T05:09:06.793 INFO:tasks.workunit.client.1.smithi192.stdout:3/599: creat d5/d13/d94/fd5 x:0 0 0 2024-04-12T05:09:06.794 INFO:tasks.workunit.client.0.smithi113.stdout:1/780: creat d10/ddc/dde/ffe x:0 0 0 2024-04-12T05:09:06.794 INFO:tasks.workunit.client.0.smithi113.stdout:3/614: mkdir d0/d13/d47/d94/dcf 0 2024-04-12T05:09:06.794 INFO:tasks.workunit.client.0.smithi113.stdout:9/682: write d1/d27/f3c [173110,53750] 0 2024-04-12T05:09:06.795 INFO:tasks.workunit.client.0.smithi113.stdout:2/603: getdents d8/d9/de/d15/d24 0 2024-04-12T05:09:06.798 INFO:tasks.workunit.client.1.smithi192.stdout:6/683: mknod d3/dc/d39/ce1 0 2024-04-12T05:09:06.801 INFO:tasks.workunit.client.0.smithi113.stdout:8/664: read d9/f47 [2333261,106990] 0 2024-04-12T05:09:06.802 INFO:tasks.workunit.client.1.smithi192.stdout:0/686: dread d0/d7b/d7c/dcc/f85 [0,4194304] 0 2024-04-12T05:09:06.803 INFO:tasks.workunit.client.0.smithi113.stdout:4/665: dread da/dd/d1b/f2c [0,4194304] 0 2024-04-12T05:09:06.803 INFO:tasks.workunit.client.0.smithi113.stdout:3/615: rename d0/c17 to d0/d13/d15/d28/d2a/d8f/cd0 0 2024-04-12T05:09:06.803 INFO:tasks.workunit.client.0.smithi113.stdout:0/725: write d1/f2f [1558053,86360] 0 2024-04-12T05:09:06.804 INFO:tasks.workunit.client.0.smithi113.stdout:8/665: write d9/d1b/d8e/d93/d32/d48/f49 [5139616,42075] 0 2024-04-12T05:09:06.805 INFO:tasks.workunit.client.1.smithi192.stdout:3/600: dwrite d5/d13/d3a/d70/d8c/f96 [0,4194304] 0 2024-04-12T05:09:06.805 INFO:tasks.workunit.client.0.smithi113.stdout:9/683: mknod d1/df/d4b/d63/ce6 0 2024-04-12T05:09:06.806 INFO:tasks.workunit.client.1.smithi192.stdout:2/614: link d1/d8/d17/d19/d60/d9c/lb9 d1/d2d/d73/da8/db2/lba 0 2024-04-12T05:09:06.806 INFO:tasks.workunit.client.1.smithi192.stdout:9/520: chown d11/d15/d21/d2c/d1f/d6a/d7e/d89/l9e 86612 1 2024-04-12T05:09:06.806 INFO:tasks.workunit.client.1.smithi192.stdout:4/697: mkdir d1/d1d/d53/d1c/d95/dc6/dd5/de5/ded 0 2024-04-12T05:09:06.807 INFO:tasks.workunit.client.0.smithi113.stdout:6/646: truncate d8/de/d13/fb 3465937 0 2024-04-12T05:09:06.807 INFO:tasks.workunit.client.0.smithi113.stdout:4/666: truncate da/d3c/d83/f40 9375221 0 2024-04-12T05:09:06.807 INFO:tasks.workunit.client.0.smithi113.stdout:2/604: symlink d8/d9/de/d43/db2/lb9 0 2024-04-12T05:09:06.809 INFO:tasks.workunit.client.1.smithi192.stdout:9/521: fsync d11/d15/d21/d2c/d1f/f67 0 2024-04-12T05:09:06.810 INFO:tasks.workunit.client.0.smithi113.stdout:1/781: link d10/d41/d89/d96/d7d/d9f/lc9 d10/d41/d89/df2/lff 0 2024-04-12T05:09:06.811 INFO:tasks.workunit.client.0.smithi113.stdout:8/666: creat d9/d1b/d8e/d93/d32/d48/dc1/fd5 x:0 0 0 2024-04-12T05:09:06.811 INFO:tasks.workunit.client.0.smithi113.stdout:0/726: getdents d1/d80/dcf 0 2024-04-12T05:09:06.812 INFO:tasks.workunit.client.0.smithi113.stdout:3/616: mkdir d0/d13/d47/d94/dcf/dd1 0 2024-04-12T05:09:06.814 INFO:tasks.workunit.client.0.smithi113.stdout:2/605: symlink d8/d9/de/d12/d2b/daa/lba 0 2024-04-12T05:09:06.814 INFO:tasks.workunit.client.0.smithi113.stdout:8/667: stat d9/d1b/d8e/d93 0 2024-04-12T05:09:06.815 INFO:tasks.workunit.client.0.smithi113.stdout:4/667: chown da/d3c/da5/dad/cd8 61788974 1 2024-04-12T05:09:06.818 INFO:tasks.workunit.client.0.smithi113.stdout:6/647: link d8/d45/f49 d8/de/dd5/de7/db3/dc5/fed 0 2024-04-12T05:09:06.819 INFO:tasks.workunit.client.0.smithi113.stdout:1/782: creat d10/d41/d89/d96/dd3/f100 x:0 0 0 2024-04-12T05:09:06.820 INFO:tasks.workunit.client.1.smithi192.stdout:2/615: readlink d1/d8/d3a/de/d7c/d98/la4 0 2024-04-12T05:09:06.820 INFO:tasks.workunit.client.1.smithi192.stdout:7/550: dwrite d3/d7/d15/f50 [0,4194304] 0 2024-04-12T05:09:06.820 INFO:tasks.workunit.client.1.smithi192.stdout:7/551: chown d3/f6f 2943327 1 2024-04-12T05:09:06.821 INFO:tasks.workunit.client.0.smithi113.stdout:8/668: mknod d9/d1b/d8e/d93/d32/d48/d8d/d4a/cd6 0 2024-04-12T05:09:06.822 INFO:tasks.workunit.client.0.smithi113.stdout:1/783: chown d10/d41/d89 0 1 2024-04-12T05:09:06.822 INFO:tasks.workunit.client.0.smithi113.stdout:9/684: dread d1/d27/f5d [0,4194304] 0 2024-04-12T05:09:06.822 INFO:tasks.workunit.client.0.smithi113.stdout:0/727: symlink d1/d6/d16/d2a/d50/db6/d36/le2 0 2024-04-12T05:09:06.824 INFO:tasks.workunit.client.1.smithi192.stdout:3/601: creat d5/d13/d3a/d4e/fd6 x:0 0 0 2024-04-12T05:09:06.825 INFO:tasks.workunit.client.1.smithi192.stdout:3/602: chown d5/d13/d3a/d85/fa4 28301821 1 2024-04-12T05:09:06.825 INFO:tasks.workunit.client.0.smithi113.stdout:1/784: chown d10/d2e/d7c/l87 155228 1 2024-04-12T05:09:06.825 INFO:tasks.workunit.client.0.smithi113.stdout:9/685: chown d1/df/fc3 6 1 2024-04-12T05:09:06.825 INFO:tasks.workunit.client.1.smithi192.stdout:3/603: fdatasync d5/d13/d3c/d84/f9d 0 2024-04-12T05:09:06.826 INFO:tasks.workunit.client.1.smithi192.stdout:3/604: stat d5/f55 0 2024-04-12T05:09:06.826 INFO:tasks.workunit.client.1.smithi192.stdout:8/533: truncate d6/d10/d46/d5e/d5f/d37/f48 3026734 0 2024-04-12T05:09:06.828 INFO:tasks.workunit.client.0.smithi113.stdout:2/606: getdents d8/d9/de/d43 0 2024-04-12T05:09:06.828 INFO:tasks.workunit.client.1.smithi192.stdout:1/618: link d0/d2/d1e/d3f/f87 d0/d1b/d44/d69/fcc 0 2024-04-12T05:09:06.829 INFO:tasks.workunit.client.0.smithi113.stdout:4/668: read da/d3c/d83/f40 [4923942,79300] 0 2024-04-12T05:09:06.829 INFO:tasks.workunit.client.0.smithi113.stdout:8/669: write d9/d16/d3c/f52 [1506713,15113] 0 2024-04-12T05:09:06.831 INFO:tasks.workunit.client.0.smithi113.stdout:3/617: dwrite d0/d13/f6b [0,4194304] 0 2024-04-12T05:09:06.831 INFO:tasks.workunit.client.0.smithi113.stdout:0/728: creat d1/d6/d16/dd5/fe3 x:0 0 0 2024-04-12T05:09:06.831 INFO:tasks.workunit.client.1.smithi192.stdout:3/605: truncate d5/d13/d3c/d84/f9d 95421 0 2024-04-12T05:09:06.831 INFO:tasks.workunit.client.1.smithi192.stdout:1/619: chown d0/d2/dd/c12 228 1 2024-04-12T05:09:06.832 INFO:tasks.workunit.client.0.smithi113.stdout:1/785: mkdir d10/d2e/dd8/df1/d101 0 2024-04-12T05:09:06.836 INFO:tasks.workunit.client.1.smithi192.stdout:8/534: dwrite d6/d10/d46/d5e/d5f/f43 [0,4194304] 0 2024-04-12T05:09:06.839 INFO:tasks.workunit.client.0.smithi113.stdout:2/607: creat d8/d9/de/d12/d3c/fbb x:0 0 0 2024-04-12T05:09:06.839 INFO:tasks.workunit.client.0.smithi113.stdout:4/669: creat da/dd/d3a/fda x:0 0 0 2024-04-12T05:09:06.847 INFO:tasks.workunit.client.0.smithi113.stdout:8/670: dwrite d9/d16/d65/d8c/fa4 [0,4194304] 0 2024-04-12T05:09:06.848 INFO:tasks.workunit.client.1.smithi192.stdout:1/620: dwrite d0/d2/d1e/d3f/d56/f5a [0,4194304] 0 2024-04-12T05:09:06.850 INFO:tasks.workunit.client.0.smithi113.stdout:4/670: creat da/d24/d7b/fdb x:0 0 0 2024-04-12T05:09:06.854 INFO:tasks.workunit.client.0.smithi113.stdout:8/671: unlink d9/d1b/d96/f9b 0 2024-04-12T05:09:06.856 INFO:tasks.workunit.client.0.smithi113.stdout:8/672: mknod d9/cd7 0 2024-04-12T05:09:06.862 INFO:tasks.workunit.client.0.smithi113.stdout:4/671: dwrite da/d4f/f4e [8388608,4194304] 0 2024-04-12T05:09:06.864 INFO:tasks.workunit.client.1.smithi192.stdout:7/552: rename d3/f6f to d3/d1d/d5a/d5f/d6a/fb7 0 2024-04-12T05:09:06.866 INFO:tasks.workunit.client.1.smithi192.stdout:4/698: creat d1/d54/d82/d71/d9e/dc4/dd9/fee x:0 0 0 2024-04-12T05:09:06.867 INFO:tasks.workunit.client.1.smithi192.stdout:4/699: dread - d1/d1d/d53/d1c/fde zero size 2024-04-12T05:09:06.873 INFO:tasks.workunit.client.0.smithi113.stdout:0/729: symlink d1/le4 0 2024-04-12T05:09:06.873 INFO:tasks.workunit.client.0.smithi113.stdout:8/673: link d9/d16/d3c/l6e d9/d1b/d8e/d93/d92/ld8 0 2024-04-12T05:09:06.873 INFO:tasks.workunit.client.0.smithi113.stdout:0/730: chown d1/fc9 1 1 2024-04-12T05:09:06.874 INFO:tasks.workunit.client.1.smithi192.stdout:6/684: getdents d3/d8/d56/d59 0 2024-04-12T05:09:06.877 INFO:tasks.workunit.client.0.smithi113.stdout:0/731: rmdir d1/d43/d59/db2 39 2024-04-12T05:09:06.878 INFO:tasks.workunit.client.0.smithi113.stdout:8/674: dread d9/d1b/d8e/d93/d32/d5b/fb8 [0,4194304] 0 2024-04-12T05:09:06.879 INFO:tasks.workunit.client.1.smithi192.stdout:0/687: getdents d0 0 2024-04-12T05:09:06.880 INFO:tasks.workunit.client.1.smithi192.stdout:8/535: mknod d6/d21/ca6 0 2024-04-12T05:09:06.881 INFO:tasks.workunit.client.1.smithi192.stdout:1/621: creat d0/d29/d8d/d96/fcd x:0 0 0 2024-04-12T05:09:06.883 INFO:tasks.workunit.client.0.smithi113.stdout:4/672: fsync da/d4f/f4e 0 2024-04-12T05:09:06.883 INFO:tasks.workunit.client.0.smithi113.stdout:6/648: truncate d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8f 606188 0 2024-04-12T05:09:06.884 INFO:tasks.workunit.client.1.smithi192.stdout:8/536: truncate d6/d10/d46/d5e/d5f/f4d 500870 0 2024-04-12T05:09:06.884 INFO:tasks.workunit.client.1.smithi192.stdout:1/622: write d0/d2/dd/d3c/f41 [3269709,15144] 0 2024-04-12T05:09:06.886 INFO:tasks.workunit.client.1.smithi192.stdout:9/522: link d11/d15/d21/d2c/d1f/d6a/d92/l9f d11/d15/d21/d2c/d1f/lb5 0 2024-04-12T05:09:06.887 INFO:tasks.workunit.client.0.smithi113.stdout:6/649: chown d8/fda 1 1 2024-04-12T05:09:06.887 INFO:tasks.workunit.client.0.smithi113.stdout:8/675: dread d9/d16/d65/d8c/fb5 [0,4194304] 0 2024-04-12T05:09:06.889 INFO:tasks.workunit.client.1.smithi192.stdout:4/700: rename d1/d54/d55/d64/d68 to d1/d1d/d53/d1c/dd1/def 0 2024-04-12T05:09:06.892 INFO:tasks.workunit.client.1.smithi192.stdout:8/537: dwrite d6/d1f/d5d/f8d [0,4194304] 0 2024-04-12T05:09:06.893 INFO:tasks.workunit.client.0.smithi113.stdout:8/676: chown d9/d16/d3c/f74 145139984 1 2024-04-12T05:09:06.894 INFO:tasks.workunit.client.1.smithi192.stdout:1/623: dwrite d0/d2/d1e/f36 [0,4194304] 0 2024-04-12T05:09:06.895 INFO:tasks.workunit.client.1.smithi192.stdout:8/538: dread - d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/f72 zero size 2024-04-12T05:09:06.896 INFO:tasks.workunit.client.1.smithi192.stdout:2/616: creat d1/d8/d17/d19/fbb x:0 0 0 2024-04-12T05:09:06.897 INFO:tasks.workunit.client.0.smithi113.stdout:3/618: dwrite d0/d13/d15/f3d [4194304,4194304] 0 2024-04-12T05:09:06.901 INFO:tasks.workunit.client.0.smithi113.stdout:1/786: write d10/d2e/dd8/d99/fb8 [228946,98150] 0 2024-04-12T05:09:06.901 INFO:tasks.workunit.client.0.smithi113.stdout:2/608: dwrite d8/d9/d27/f28 [0,4194304] 0 2024-04-12T05:09:06.901 INFO:tasks.workunit.client.1.smithi192.stdout:7/553: mknod d3/d1d/d74/d86/d93/cb8 0 2024-04-12T05:09:06.904 INFO:tasks.workunit.client.0.smithi113.stdout:4/673: link da/d24/d60/daa/c51 da/d24/d60/db4/cdc 0 2024-04-12T05:09:06.904 INFO:tasks.workunit.client.0.smithi113.stdout:0/732: rename d1/d6/d16/d2a/d50/db6/d36/cda to d1/d6/d16/d2a/d50/d52/d7e/ce5 0 2024-04-12T05:09:06.905 INFO:tasks.workunit.client.0.smithi113.stdout:8/677: mknod d9/d16/d7f/d3a/d6c/cd9 0 2024-04-12T05:09:06.907 INFO:tasks.workunit.client.0.smithi113.stdout:9/686: dread d1/d4/f8f [0,4194304] 0 2024-04-12T05:09:06.909 INFO:tasks.workunit.client.1.smithi192.stdout:9/523: symlink d11/d25/d36/d3d/d7a/lb6 0 2024-04-12T05:09:06.910 INFO:tasks.workunit.client.0.smithi113.stdout:3/619: write d0/d13/d47/f4a [815064,1684] 0 2024-04-12T05:09:06.911 INFO:tasks.workunit.client.0.smithi113.stdout:2/609: rmdir d8/d9 39 2024-04-12T05:09:06.912 INFO:tasks.workunit.client.1.smithi192.stdout:4/701: rename d1/d54/d82/da1 to d1/d54/d82/df0 0 2024-04-12T05:09:06.912 INFO:tasks.workunit.client.0.smithi113.stdout:4/674: fsync da/d24/d60/db4/fb7 0 2024-04-12T05:09:06.915 INFO:tasks.workunit.client.0.smithi113.stdout:6/650: dwrite d8/de/d13/d11/f43 [0,4194304] 0 2024-04-12T05:09:06.916 INFO:tasks.workunit.client.0.smithi113.stdout:1/787: stat d10/d41/d89/d96/d7d/f7e 0 2024-04-12T05:09:06.916 INFO:tasks.workunit.client.0.smithi113.stdout:9/687: chown d1/d4/c55 23 1 2024-04-12T05:09:06.918 INFO:tasks.workunit.client.1.smithi192.stdout:8/539: rename d6/d10/d46/d5e/d5f/d37/d4f/l56 to d6/d10/d46/d5e/d5f/d31/d61/la7 0 2024-04-12T05:09:06.918 INFO:tasks.workunit.client.1.smithi192.stdout:8/540: stat d6/d10/d46/d5e/d5f/c32 0 2024-04-12T05:09:06.919 INFO:tasks.workunit.client.1.smithi192.stdout:7/554: mkdir d3/d1d/d74/d86/d93/db9 0 2024-04-12T05:09:06.920 INFO:tasks.workunit.client.1.smithi192.stdout:6/685: creat d3/d8/d10/d4c/fe2 x:0 0 0 2024-04-12T05:09:06.921 INFO:tasks.workunit.client.1.smithi192.stdout:7/555: stat d3/d7/d60/fa0 0 2024-04-12T05:09:06.922 INFO:tasks.workunit.client.1.smithi192.stdout:3/606: getdents d5/d13/d3c 0 2024-04-12T05:09:06.922 INFO:tasks.workunit.client.1.smithi192.stdout:7/556: fsync d3/d1d/f9f 0 2024-04-12T05:09:06.923 INFO:tasks.workunit.client.1.smithi192.stdout:9/524: stat d11/d15/d21/d2c/d1f/d50/c5b 0 2024-04-12T05:09:06.925 INFO:tasks.workunit.client.0.smithi113.stdout:0/733: read d1/d6/d16/f24 [108342,61822] 0 2024-04-12T05:09:06.925 INFO:tasks.workunit.client.0.smithi113.stdout:2/610: chown d8/d9/f30 13 1 2024-04-12T05:09:06.925 INFO:tasks.workunit.client.1.smithi192.stdout:7/557: truncate d3/d7/d60/d65/f8a 383231 0 2024-04-12T05:09:06.926 INFO:tasks.workunit.client.0.smithi113.stdout:9/688: dread - d1/d4/d5/d6/fdf zero size 2024-04-12T05:09:06.929 INFO:tasks.workunit.client.1.smithi192.stdout:9/525: creat d11/d25/d36/d3d/fb7 x:0 0 0 2024-04-12T05:09:06.930 INFO:tasks.workunit.client.0.smithi113.stdout:2/611: dread - d8/d9/de/d43/fa8 zero size 2024-04-12T05:09:06.930 INFO:tasks.workunit.client.0.smithi113.stdout:6/651: symlink d8/de/d13/d11/d36/d53/db8/d63/d66/dc4/lee 0 2024-04-12T05:09:06.930 INFO:tasks.workunit.client.0.smithi113.stdout:8/678: dwrite d9/d16/f1a [0,4194304] 0 2024-04-12T05:09:06.932 INFO:tasks.workunit.client.1.smithi192.stdout:4/702: creat d1/d49/db5/ff1 x:0 0 0 2024-04-12T05:09:06.933 INFO:tasks.workunit.client.0.smithi113.stdout:9/689: mknod d1/df/d29/ce7 0 2024-04-12T05:09:06.933 INFO:tasks.workunit.client.0.smithi113.stdout:3/620: dwrite d0/d13/d15/d28/f45 [0,4194304] 0 2024-04-12T05:09:06.933 INFO:tasks.workunit.client.1.smithi192.stdout:5/614: sync 2024-04-12T05:09:06.934 INFO:tasks.workunit.client.0.smithi113.stdout:8/679: chown d9/d1b/d8e/d93/d32 9 1 2024-04-12T05:09:06.935 INFO:tasks.workunit.client.1.smithi192.stdout:3/607: creat d5/d13/d3a/d8b/da2/da7/fd7 x:0 0 0 2024-04-12T05:09:06.938 INFO:tasks.workunit.client.0.smithi113.stdout:8/680: readlink d9/d16/d65/db1/lbf 0 2024-04-12T05:09:06.938 INFO:tasks.workunit.client.0.smithi113.stdout:1/788: creat d10/d2e/f102 x:0 0 0 2024-04-12T05:09:06.939 INFO:tasks.workunit.client.0.smithi113.stdout:3/621: write d0/d13/d15/d28/f31 [904253,66062] 0 2024-04-12T05:09:06.941 INFO:tasks.workunit.client.1.smithi192.stdout:2/617: rename d1/d2d/c94 to d1/d2d/cbc 0 2024-04-12T05:09:06.943 INFO:tasks.workunit.client.0.smithi113.stdout:9/690: chown d1/df/d4b/d63/d3d/d6f/fa8 40 1 2024-04-12T05:09:06.945 INFO:tasks.workunit.client.0.smithi113.stdout:3/622: mkdir d0/d13/d47/d4f/dd2 0 2024-04-12T05:09:06.947 INFO:tasks.workunit.client.1.smithi192.stdout:1/624: getdents d0/d2/dd/d3c/dab 0 2024-04-12T05:09:06.947 INFO:tasks.workunit.client.1.smithi192.stdout:7/558: creat d3/d1d/d4a/d5e/d6d/fba x:0 0 0 2024-04-12T05:09:06.949 INFO:tasks.workunit.client.0.smithi113.stdout:1/789: mknod d10/d14/d95/c103 0 2024-04-12T05:09:06.949 INFO:tasks.workunit.client.1.smithi192.stdout:4/703: mknod d1/d54/d82/df0/cf2 0 2024-04-12T05:09:06.950 INFO:tasks.workunit.client.0.smithi113.stdout:8/681: fsync d9/d1b/fcd 0 2024-04-12T05:09:06.952 INFO:tasks.workunit.client.0.smithi113.stdout:6/652: write d8/d45/f49 [832645,80607] 0 2024-04-12T05:09:06.952 INFO:tasks.workunit.client.0.smithi113.stdout:1/790: creat d10/d2e/d7c/dc5/dc8/f104 x:0 0 0 2024-04-12T05:09:06.953 INFO:tasks.workunit.client.1.smithi192.stdout:1/625: write d0/d2/dd/d21/d39/f65 [4675212,96903] 0 2024-04-12T05:09:06.956 INFO:tasks.workunit.client.1.smithi192.stdout:7/559: creat d3/d7/d14/d1f/d76/fbb x:0 0 0 2024-04-12T05:09:06.958 INFO:tasks.workunit.client.0.smithi113.stdout:2/612: truncate d8/d9/d27/f6d 796843 0 2024-04-12T05:09:06.960 INFO:tasks.workunit.client.1.smithi192.stdout:0/688: dwrite d0/d7b/d9b/d4b/dbc/dce/d53/fe7 [0,4194304] 0 2024-04-12T05:09:06.962 INFO:tasks.workunit.client.0.smithi113.stdout:6/653: creat d8/de/d13/d11/d36/d53/db7/fef x:0 0 0 2024-04-12T05:09:06.963 INFO:tasks.workunit.client.1.smithi192.stdout:3/608: creat d5/fd8 x:0 0 0 2024-04-12T05:09:06.963 INFO:tasks.workunit.client.0.smithi113.stdout:9/691: getdents d1/d27/d7a/dd2 0 2024-04-12T05:09:06.964 INFO:tasks.workunit.client.0.smithi113.stdout:3/623: getdents d0/d13/d47/d94/dcf 0 2024-04-12T05:09:06.966 INFO:tasks.workunit.client.1.smithi192.stdout:7/560: symlink d3/d1d/d5a/d5f/lbc 0 2024-04-12T05:09:06.967 INFO:tasks.workunit.client.0.smithi113.stdout:3/624: chown d0/d7e/cae 0 1 2024-04-12T05:09:06.968 INFO:tasks.workunit.client.1.smithi192.stdout:3/609: fsync d5/d13/f2f 0 2024-04-12T05:09:06.969 INFO:tasks.workunit.client.1.smithi192.stdout:6/686: dread d3/d8/d10/f71 [0,4194304] 0 2024-04-12T05:09:06.970 INFO:tasks.workunit.client.0.smithi113.stdout:4/675: dread da/dd/d3a/f42 [0,4194304] 0 2024-04-12T05:09:06.970 INFO:tasks.workunit.client.0.smithi113.stdout:3/625: fsync d0/d13/f6a 0 2024-04-12T05:09:06.972 INFO:tasks.workunit.client.1.smithi192.stdout:7/561: chown d3/d1d/d48/c99 18405527 1 2024-04-12T05:09:06.973 INFO:tasks.workunit.client.1.smithi192.stdout:4/704: dwrite d1/d1d/fd2 [4194304,4194304] 0 2024-04-12T05:09:06.975 INFO:tasks.workunit.client.0.smithi113.stdout:6/654: rename d8/de/d13/d11/d36/d53/db8/d63 to d8/de/d13/df0 0 2024-04-12T05:09:06.976 INFO:tasks.workunit.client.0.smithi113.stdout:4/676: write da/d4a/f5c [2382357,119073] 0 2024-04-12T05:09:06.976 INFO:tasks.workunit.client.0.smithi113.stdout:2/613: dwrite d8/d9/de/d12/f9c [0,4194304] 0 2024-04-12T05:09:06.976 INFO:tasks.workunit.client.1.smithi192.stdout:5/615: truncate d0/d15/d32/d69/d7d/fcc 990990 0 2024-04-12T05:09:06.977 INFO:tasks.workunit.client.0.smithi113.stdout:3/626: mkdir d0/d13/d47/d4f/dd3 0 2024-04-12T05:09:06.977 INFO:tasks.workunit.client.0.smithi113.stdout:6/655: read - d8/d45/fe9 zero size 2024-04-12T05:09:06.979 INFO:tasks.workunit.client.1.smithi192.stdout:7/562: write d3/d1d/f41 [1034137,120594] 0 2024-04-12T05:09:06.979 INFO:tasks.workunit.client.1.smithi192.stdout:1/626: dwrite d0/d2/dd/d3c/d5b/d74/d90/fc4 [0,4194304] 0 2024-04-12T05:09:06.980 INFO:tasks.workunit.client.0.smithi113.stdout:8/682: dwrite d9/f47 [0,4194304] 0 2024-04-12T05:09:06.982 INFO:tasks.workunit.client.0.smithi113.stdout:2/614: chown d8/d9/d5b 7547106 1 2024-04-12T05:09:06.989 INFO:tasks.workunit.client.1.smithi192.stdout:2/618: write d1/d2d/f32 [1257520,62122] 0 2024-04-12T05:09:06.989 INFO:tasks.workunit.client.1.smithi192.stdout:9/526: dread d11/d25/d36/d6e/d33/da2/f39 [0,4194304] 0 2024-04-12T05:09:06.991 INFO:tasks.workunit.client.0.smithi113.stdout:9/692: rename d1/df/d4b/d63/d3d/d6f/dba to d1/df/d29/de8 0 2024-04-12T05:09:06.992 INFO:tasks.workunit.client.0.smithi113.stdout:4/677: creat da/d3c/d9f/fdd x:0 0 0 2024-04-12T05:09:06.993 INFO:tasks.workunit.client.0.smithi113.stdout:1/791: dwrite d10/d41/d89/d96/dd3/f50 [0,4194304] 0 2024-04-12T05:09:06.994 INFO:tasks.workunit.client.1.smithi192.stdout:3/610: write d5/f7a [5136036,121062] 0 2024-04-12T05:09:07.336 INFO:tasks.workunit.client.1.smithi192.stdout:1/627: chown d0/d2/dd/f5d 0 1 2024-04-12T05:09:07.336 INFO:tasks.workunit.client.1.smithi192.stdout:4/705: symlink d1/d1d/d53/d1c/dd1/def/ddd/lf3 0 2024-04-12T05:09:07.336 INFO:tasks.workunit.client.1.smithi192.stdout:8/541: dwrite d6/f95 [0,4194304] 0 2024-04-12T05:09:07.356 INFO:tasks.workunit.client.1.smithi192.stdout:2/619: chown d1/d2d/d73/fb1 58 1 2024-04-12T05:09:07.356 INFO:tasks.workunit.client.1.smithi192.stdout:4/706: truncate d1/d54/d55/d64/f91 1025166 0 2024-04-12T05:09:07.356 INFO:tasks.workunit.client.1.smithi192.stdout:2/620: write d1/d8/d3a/de/d7c/fb8 [929489,95205] 0 2024-04-12T05:09:07.356 INFO:tasks.workunit.client.1.smithi192.stdout:7/563: creat d3/d1d/d5a/d5f/fbd x:0 0 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:4/707: truncate d1/d1d/fad 4802718 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:2/621: write d1/d8/d3a/de/d7c/fb8 [623522,18810] 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:9/527: symlink d11/d15/d21/d2c/d1f/d50/d73/lb8 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:3/611: creat d5/d13/d3a/d8b/da2/da7/fd9 x:0 0 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:7/564: rename d3/d1d/d4a/f61 to d3/d11/fbe 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:4/708: mkdir d1/d1d/d53/d1c/dd1/def/ddd/dea/df4 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:7/565: creat d3/d7/d14/d1f/fbf x:0 0 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:9/528: symlink d11/d15/d21/d2c/lb9 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:8/542: dwrite d6/d21/f60 [0,4194304] 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:7/566: mknod d3/d7/d60/cc0 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:9/529: rename d11/d25/d36/d6e/d33/da2/l6d to d11/d25/d36/d3d/d7a/lba 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:8/543: mkdir d6/d10/d6c/d9e/da8 0 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:8/544: chown d6/d10/d46/d5e/d5f/d31/d61 121585 1 2024-04-12T05:09:07.357 INFO:tasks.workunit.client.1.smithi192.stdout:7/567: unlink d3/d7/d15/d1a/l5d 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:7/568: dread - d3/d1d/d5a/d5f/fbd zero size 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:2/622: write d1/d2d/f34 [465966,54299] 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:3/612: truncate d5/d13/f42 18131 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:8/545: dwrite d6/d10/d46/d5e/d5f/f7e [0,4194304] 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:5/616: dread d0/d15/f23 [0,4194304] 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:0/689: dread d0/d6/d15/d1b/d3a/d4f/f24 [0,4194304] 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:2/623: write d1/f70 [898435,83515] 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:3/613: creat d5/d13/d94/fda x:0 0 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:8/546: readlink d6/d10/d46/d5e/d5f/d38/l91 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:7/569: read d3/d7/d15/f24 [1358845,47722] 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:6/687: dwrite d3/d23/db2/fde [0,4194304] 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:5/617: mknod d0/d15/d32/d69/cd5 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:1/628: dwrite d0/d2/dd/d21/d62/d84/f91 [4194304,4194304] 0 2024-04-12T05:09:07.358 INFO:tasks.workunit.client.1.smithi192.stdout:5/618: chown d0/d15/d24/f25 2 1 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:3/614: write d5/d13/d94/fd5 [257839,89733] 0 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:6/688: rmdir d3/d23/d67 39 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:3/615: read d5/d13/d3a/d70/d8c/fb4 [197701,26094] 0 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:8/547: read d6/d10/d46/d5e/d5f/f16 [1246060,95777] 0 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:1/629: creat d0/d2/dd/d3c/d5b/d74/d7d/fce x:0 0 0 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:7/570: dread d3/d7/d14/f58 [0,4194304] 0 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:7/571: read - d3/d1d/d4a/faa zero size 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:3/616: read d5/f55 [2890395,126541] 0 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:1/630: truncate d0/d1b/d44/fb9 361754 0 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:3/617: dread - d5/d13/d3c/d36/da0/fb7 zero size 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:0/690: dread d0/d6/d15/d55/f6c [0,4194304] 0 2024-04-12T05:09:07.359 INFO:tasks.workunit.client.1.smithi192.stdout:8/548: creat d6/d10/d46/d5e/d6e/fa9 x:0 0 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:1/631: mknod d0/d2/dd/d21/d62/dbc/ccf 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:2/624: write d1/d8/f8e [586754,61312] 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:6/689: rename d3/d21/d24/l42 to d3/le3 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:1/632: chown d0/d2/dd/d21/d62/la1 23022 1 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:0/691: creat d0/d6/d67/db1/fe9 x:0 0 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:8/549: mknod d6/d10/d46/d5e/d5f/d37/d7b/caa 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:5/619: link d0/d15/l67 d0/d15/d3e/ld6 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:7/572: rename d3/d1d/f82 to d3/d1d/d4a/d5e/d6d/da3/fc1 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:0/692: creat d0/d6/d15/d55/d2e/d59/fea x:0 0 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:8/550: write d6/d10/d46/f96 [5263788,63729] 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:5/620: unlink d0/d15/d24/f51 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:5/621: chown d0/d15/d3e 79173986 1 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:5/622: dread d0/d15/f23 [4194304,4194304] 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:6/690: dread d3/d21/f8d [0,4194304] 0 2024-04-12T05:09:07.360 INFO:tasks.workunit.client.1.smithi192.stdout:5/623: dwrite d0/d15/d64/fcd [0,4194304] 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:7/573: fsync d3/d1d/d74/f8c 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:6/691: mknod d3/d8/d56/d59/ce4 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:5/624: mkdir d0/d15/d64/d7c/db5/dd7 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:1/633: rename d0/d2/dd/d3c/d5b/d74/d7d/d9b to d0/dd0 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:6/692: mknod d3/d5/d9c/ce5 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:2/625: write d1/d8/fc [1254512,122855] 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:1/634: write d0/d2/dd/d21/d62/d84/f8e [131272,21851] 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:2/626: chown d1/d8/d17/d19/d60/fa6 9 1 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:0/693: getdents d0/d6/d67 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:5/625: creat d0/d15/d64/d7c/db5/dbe/fd8 x:0 0 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:2/627: creat d1/d8/d17/d19/d40/fbd x:0 0 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:5/626: fsync d0/d15/d3e/dca/fb7 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:1/635: write d0/d2/d1e/d3f/d56/d9e/fbb [377347,49467] 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:0/694: dwrite d0/d6/d15/f3d [0,4194304] 0 2024-04-12T05:09:07.361 INFO:tasks.workunit.client.1.smithi192.stdout:6/693: mkdir d3/d8/de6 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:8/551: rename d6/da/f70 to d6/d10/d46/d5e/d5f/fab 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:5/627: creat d0/d15/d32/d69/fd9 x:0 0 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:2/628: creat d1/d8/d17/d19/d60/d9c/fbe x:0 0 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:6/694: mkdir d3/d5/de7 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:8/552: write d6/d1f/f2a [524637,116329] 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:2/629: fsync d1/d8/f8e 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:5/628: chown d0/d15/d32/d45/fb1 6213 1 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:0/695: dwrite d0/d6/d67/f8a [0,4194304] 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:2/630: fdatasync d1/f8b 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:7/574: rename d3/d1d/d5a/d5f/fa4 to d3/d1d/fc2 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:7/575: chown d3/d1d/d5a 1019740 1 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:0/696: dread d0/d6/d15/d55/d2e/f63 [0,4194304] 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:8/553: symlink d6/d10/d6c/lac 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:6/695: truncate d3/f43 1139542 0 2024-04-12T05:09:07.362 INFO:tasks.workunit.client.1.smithi192.stdout:8/554: chown d6/f18 23291 1 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:2/631: creat d1/d8/d3a/de/d7c/fbf x:0 0 0 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:2/632: chown d1/d8/d17/d19/d60/d59/la1 237630 1 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:1/636: rename d0/d2/dd/d21/l3d to d0/d2/dd/d3c/d5b/db6/ld1 0 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:0/697: mkdir d0/d6/d67/db1/deb 0 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:0/698: write d0/d6/dc4/fd9 [919703,15282] 0 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:8/555: mknod d6/d10/d6c/d9e/cad 0 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:2/633: symlink d1/d2d/d73/da8/lc0 0 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:5/629: rename d0/d15/d64/d7c to d0/d15/d3e/dca/dda 0 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:7/576: creat d3/d1d/d74/d86/d93/db9/fc3 x:0 0 0 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:8/556: rmdir d6/d1f/d6a 39 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:7/577: readlink d3/d1d/d5a/d5f/lbc 0 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:2/634: symlink d1/da7/lc1 0 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:0/699: dwrite d0/d7b/d9b/d4b/dbc/dce/f87 [0,4194304] 0 2024-04-12T05:09:07.363 INFO:tasks.workunit.client.1.smithi192.stdout:7/578: chown d3/d7/d15/d1a 3 1 2024-04-12T05:09:07.364 INFO:tasks.workunit.client.1.smithi192.stdout:5/630: dread d0/d21/d5e/f6c [0,4194304] 0 2024-04-12T05:09:07.364 INFO:tasks.workunit.client.1.smithi192.stdout:6/696: getdents d3/d8/d56 0 2024-04-12T05:09:07.364 INFO:tasks.workunit.client.1.smithi192.stdout:2/635: mknod d1/d8/d17/d19/d60/d59/cc2 0 2024-04-12T05:09:07.364 INFO:tasks.workunit.client.1.smithi192.stdout:8/557: dwrite d6/d10/fa5 [0,4194304] 0 2024-04-12T05:09:07.364 INFO:tasks.workunit.client.1.smithi192.stdout:7/579: write d3/d7/d15/d1a/f3e [118105,89681] 0 2024-04-12T05:09:07.364 INFO:tasks.workunit.client.1.smithi192.stdout:0/700: dwrite d0/d6/d15/d55/d2e/d59/f78 [4194304,4194304] 0 2024-04-12T05:09:07.364 INFO:tasks.workunit.client.1.smithi192.stdout:0/701: dwrite d0/d6/d15/d55/fd8 [0,4194304] 0 2024-04-12T05:09:07.366 INFO:tasks.workunit.client.1.smithi192.stdout:6/697: rename d3/d21/d24/d79 to d3/d8/d56/d59/d97/dbf/de8 0 2024-04-12T05:09:07.366 INFO:tasks.workunit.client.1.smithi192.stdout:5/631: link d0/d15/d3e/dca/ccb d0/d15/d32/d69/dc8/cdb 0 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:8/683: rename d9/la to d9/d1b/d8e/d93/d32/lda 0 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:4/678: read - da/d24/d60/db4/fca zero size 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:1/792: symlink d10/d2e/d7c/dc5/dc8/dad/l105 0 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:3/627: dread d0/d7e/f85 [0,4194304] 0 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:2/615: creat d8/d9/de/d12/d3c/fbc x:0 0 0 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:6/656: dwrite d8/de/d13/d11/d36/d53/db7/fd4 [0,4194304] 0 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:2/616: dread - d8/d9/d27/fb7 zero size 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:8/684: link d9/d1b/d8e/d93/d32/d48/dc1/fd5 d9/d16/d65/db1/d59/d78/fdb 0 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:6/657: read d8/d45/f5d [2539572,91446] 0 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:2/617: mkdir d8/d16/dbd 0 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:2/618: chown d8/d9/de/d15 182 1 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:4/679: rename da/d24/d7e/d79/c92 to da/cde 0 2024-04-12T05:09:07.369 INFO:tasks.workunit.client.0.smithi113.stdout:9/693: write d1/d27/f95 [1135446,81180] 0 2024-04-12T05:09:07.370 INFO:tasks.workunit.client.0.smithi113.stdout:2/619: unlink d8/d9/d27/f6d 0 2024-04-12T05:09:07.370 INFO:tasks.workunit.client.0.smithi113.stdout:2/620: write d8/d9/de/d12/d2b/fb6 [831253,54078] 0 2024-04-12T05:09:07.370 INFO:tasks.workunit.client.0.smithi113.stdout:4/680: getdents da/d70 0 2024-04-12T05:09:07.370 INFO:tasks.workunit.client.0.smithi113.stdout:2/621: stat d8/d9/de/d12/d2b 0 2024-04-12T05:09:07.370 INFO:tasks.workunit.client.0.smithi113.stdout:9/694: dwrite d1/d27/f5d [0,4194304] 0 2024-04-12T05:09:07.370 INFO:tasks.workunit.client.0.smithi113.stdout:2/622: dread - d8/d9/de/d12/d2b/d79/fb8 zero size 2024-04-12T05:09:07.370 INFO:tasks.workunit.client.0.smithi113.stdout:2/623: dread - d8/d9/d27/fb7 zero size 2024-04-12T05:09:07.370 INFO:tasks.workunit.client.0.smithi113.stdout:9/695: unlink d1/df/d4b/d63/d3d/d6f/fa8 0 2024-04-12T05:09:07.370 INFO:tasks.workunit.client.0.smithi113.stdout:8/685: dwrite d9/d1b/d8e/d93/f84 [0,4194304] 0 2024-04-12T05:09:07.370 INFO:tasks.workunit.client.0.smithi113.stdout:3/628: dread d0/d13/d15/d28/f3f [0,4194304] 0 2024-04-12T05:09:07.371 INFO:tasks.workunit.client.0.smithi113.stdout:6/658: dwrite d8/d1d/fa0 [0,4194304] 0 2024-04-12T05:09:07.371 INFO:tasks.workunit.client.0.smithi113.stdout:3/629: fsync d0/d13/d47/f4a 0 2024-04-12T05:09:07.371 INFO:tasks.workunit.client.0.smithi113.stdout:3/630: truncate d0/f6 719595 0 2024-04-12T05:09:07.371 INFO:tasks.workunit.client.0.smithi113.stdout:8/686: creat d9/d1b/d8e/d93/fdc x:0 0 0 2024-04-12T05:09:07.371 INFO:tasks.workunit.client.0.smithi113.stdout:3/631: fdatasync d0/d13/f81 0 2024-04-12T05:09:07.371 INFO:tasks.workunit.client.0.smithi113.stdout:2/624: write d8/d16/f6e [4416498,124136] 0 2024-04-12T05:09:07.371 INFO:tasks.workunit.client.0.smithi113.stdout:6/659: link d8/de/d13/d11/d36/d3d/d4c/d31/l80 d8/de/d13/df0/d66/lf1 0 2024-04-12T05:09:07.371 INFO:tasks.workunit.client.0.smithi113.stdout:2/625: symlink d8/d9/de/d43/db2/lbe 0 2024-04-12T05:09:07.371 INFO:tasks.workunit.client.0.smithi113.stdout:9/696: dwrite d1/df/d29/fa2 [0,4194304] 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:8/687: link d9/d16/d7f/d3a/d6c/d90/fbc d9/d1b/d8e/d93/d32/d5b/fdd 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:4/681: dread da/d73/f45 [0,4194304] 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:2/626: write d8/d9/de/d15/d24/d58/f96 [832655,102284] 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:8/688: truncate d9/d1b/d8e/d93/d32/f64 688699 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:4/682: dread da/d24/d7e/fbe [0,4194304] 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:9/697: creat d1/fe9 x:0 0 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:2/627: getdents d8/d9/de/d12/d3c/d8a 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:2/628: stat d8/d9/d1a/f26 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:9/698: fsync d1/df/d4b/d63/dad/fc6 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:4/683: dread da/d24/d60/db4/fb7 [0,4194304] 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:2/629: write d8/d9/f30 [1107413,70509] 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:9/699: symlink d1/d4/d5/lea 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:4/684: dwrite da/dd/d3a/fa3 [0,4194304] 0 2024-04-12T05:09:07.372 INFO:tasks.workunit.client.0.smithi113.stdout:4/685: chown da/d5a/f91 38 1 2024-04-12T05:09:07.373 INFO:tasks.workunit.client.0.smithi113.stdout:4/686: truncate da/dd/d1b/f2c 4437324 0 2024-04-12T05:09:07.373 INFO:tasks.workunit.client.0.smithi113.stdout:4/687: chown da/dd/f6e 13 1 2024-04-12T05:09:07.373 INFO:tasks.workunit.client.0.smithi113.stdout:4/688: symlink da/d24/d7b/ldf 0 2024-04-12T05:09:07.373 INFO:tasks.workunit.client.0.smithi113.stdout:4/689: creat da/d4f/d90/fe0 x:0 0 0 2024-04-12T05:09:07.373 INFO:tasks.workunit.client.0.smithi113.stdout:4/690: mkdir da/d3c/da5/dad/de1 0 2024-04-12T05:09:07.373 INFO:tasks.workunit.client.0.smithi113.stdout:9/700: dwrite d1/d4/d5/d6/fbd [0,4194304] 0 2024-04-12T05:09:07.373 INFO:tasks.workunit.client.0.smithi113.stdout:9/701: mkdir d1/d4/d38/d65/dcd/deb 0 2024-04-12T05:09:07.373 INFO:tasks.workunit.client.0.smithi113.stdout:9/702: stat d1/d4/d38/d65 0 2024-04-12T05:09:07.373 INFO:tasks.workunit.client.0.smithi113.stdout:9/703: creat d1/d27/d7a/dd2/fec x:0 0 0 2024-04-12T05:09:07.373 INFO:tasks.workunit.client.0.smithi113.stdout:9/704: link d1/d4/d38/c47 d1/d27/d7a/de4/ced 0 2024-04-12T05:09:07.373 INFO:tasks.workunit.client.0.smithi113.stdout:6/660: dread d8/de/d13/fa [0,4194304] 0 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:9/705: read d1/df/d4b/d63/d3d/f5f [2873540,73412] 0 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:6/661: stat d8/de/d13/df0/f69 0 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:9/706: mknod d1/d4/d5/cee 0 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:7/701: sync 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:6/662: symlink d8/de/d13/d11/d36/d3d/d4c/lf2 0 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:2/630: dread d8/d9/de/d15/d24/d58/f84 [0,4194304] 0 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:2/631: chown d8/d9/d27/d88/fab 223 1 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:6/663: rename d8/d1d/fa0 to d8/d45/ff3 0 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:9/707: link d1/d4/d30/dcb/fb3 d1/d4/d87/fef 0 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:2/632: mkdir d8/d9/de/d12/dbf 0 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:2/633: stat d8/d9/de/d43/db2/lb9 0 2024-04-12T05:09:07.374 INFO:tasks.workunit.client.0.smithi113.stdout:6/664: symlink d8/de/d13/d11/d36/d3d/d4c/d26/d7d/lf4 0 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:4/691: write da/d3c/d43/f5e [1904553,67787] 0 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:9/708: rename d1/d4/d5/cee to d1/cf0 0 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:2/634: dwrite d8/d9/de/f10 [4194304,4194304] 0 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:2/635: readlink d8/d9/de/d15/d24/la7 0 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:4/692: link da/d73/c6c da/d4f/ce2 0 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:2/636: chown d8/d9/de/f31 2772 1 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:7/702: write d0/dc/da9/d5c/d9a/fab [690638,63017] 0 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:2/637: write d8/d16/f17 [2409162,11929] 0 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:6/665: rmdir d8/de/d13/d11/d36/d3d/d4c/d26 39 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:7/703: unlink d0/dd/d11/f20 0 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:2/638: creat d8/fc0 x:0 0 0 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:4/693: rename da/d24/d7e/fbe to da/d24/fe3 0 2024-04-12T05:09:07.375 INFO:tasks.workunit.client.0.smithi113.stdout:6/666: dwrite d8/de/d13/d11/d36/d53/db7/fef [0,4194304] 0 2024-04-12T05:09:07.376 INFO:tasks.workunit.client.0.smithi113.stdout:4/694: write da/d4f/f5d [2112360,41094] 0 2024-04-12T05:09:07.376 INFO:tasks.workunit.client.0.smithi113.stdout:2/639: dwrite d8/d9/d27/d88/f95 [0,4194304] 0 2024-04-12T05:09:07.376 INFO:tasks.workunit.client.0.smithi113.stdout:4/695: creat da/d24/d7e/d79/dcd/fe4 x:0 0 0 2024-04-12T05:09:07.376 INFO:tasks.workunit.client.0.smithi113.stdout:4/696: truncate da/d4f/d90/fe0 253135 0 2024-04-12T05:09:07.376 INFO:tasks.workunit.client.0.smithi113.stdout:4/697: readlink da/d2b/l98 0 2024-04-12T05:09:07.376 INFO:tasks.workunit.client.0.smithi113.stdout:2/640: rename d8/d9/de/d12/d3c/d8a/fa2 to d8/d9/de/d12/d2b/daa/db3/fc1 0 2024-04-12T05:09:07.376 INFO:tasks.workunit.client.0.smithi113.stdout:6/667: link d8/d45/f5d d8/de/d13/d11/d36/d3d/d4c/d26/d5b/ff5 0 2024-04-12T05:09:07.376 INFO:tasks.workunit.client.0.smithi113.stdout:2/641: write d8/d9/de/d12/d2b/d79/fb8 [552821,69300] 0 2024-04-12T05:09:07.376 INFO:tasks.workunit.client.0.smithi113.stdout:2/642: chown d8/d9/de/d43/fa8 1160647 1 2024-04-12T05:09:07.376 INFO:tasks.workunit.client.0.smithi113.stdout:4/698: getdents da/d4a/da2 0 2024-04-12T05:09:07.376 INFO:tasks.workunit.client.0.smithi113.stdout:6/668: rename d8/de/d13/d11/d36/d3d/d4c/d26/d64/lc7 to d8/de/d17/lf6 0 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:6/669: chown d8/d45/ff3 6465366 1 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:4/699: rename da/d24/d60/c68 to da/d24/d7e/d79/dcd/ce5 0 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:4/700: creat da/d24/d7e/d79/dcd/fe6 x:0 0 0 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:6/670: creat d8/de/d13/d11/d36/d3d/d4c/dd8/ff7 x:0 0 0 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:2/643: write d8/d16/f4e [616765,76693] 0 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:4/701: truncate da/d4f/d90/fe0 1175289 0 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:2/644: mknod d8/d16/cc2 0 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:6/671: fdatasync d8/d45/f5d 0 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:2/645: creat d8/d9/de/d15/d24/d58/fc3 x:0 0 0 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:2/646: fdatasync d8/d9/d27/f28 0 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:2/647: readlink d8/d9/de/d43/l47 0 2024-04-12T05:09:07.377 INFO:tasks.workunit.client.0.smithi113.stdout:4/702: dread da/d4f/f5d [4194304,4194304] 0 2024-04-12T05:09:07.378 INFO:tasks.workunit.client.0.smithi113.stdout:2/648: mknod d8/d9/de/d12/d2b/cc4 0 2024-04-12T05:09:07.378 INFO:tasks.workunit.client.0.smithi113.stdout:4/703: fsync da/d70/fa9 0 2024-04-12T05:09:07.378 INFO:tasks.workunit.client.0.smithi113.stdout:6/672: write d8/de/d13/d11/d36/d3d/f90 [139029,126924] 0 2024-04-12T05:09:07.378 INFO:tasks.workunit.client.0.smithi113.stdout:2/649: dwrite d8/d9/de/d15/d38/f51 [0,4194304] 0 2024-04-12T05:09:07.378 INFO:tasks.workunit.client.0.smithi113.stdout:4/704: dwrite da/d2b/f3e [0,4194304] 0 2024-04-12T05:09:07.378 INFO:tasks.workunit.client.0.smithi113.stdout:4/705: chown da/dd 5729352 1 2024-04-12T05:09:07.378 INFO:tasks.workunit.client.0.smithi113.stdout:2/650: dread - d8/d9/de/d15/d38/f99 zero size 2024-04-12T05:09:07.378 INFO:tasks.workunit.client.0.smithi113.stdout:2/651: symlink d8/d9/d27/d88/d91/lc5 0 2024-04-12T05:09:07.378 INFO:tasks.workunit.client.0.smithi113.stdout:4/706: rename da/c27 to da/d24/d7e/d79/ce7 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:4/707: fdatasync da/d70/fd3 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:2/652: creat d8/d9/de/d12/d2b/daa/fc6 x:0 0 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:2/653: fdatasync d8/d9/de/d15/d24/d58/fae 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:4/708: link da/d24/d60/daa/c8f da/d24/d7e/d79/ce8 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:6/673: dwrite d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8f [0,4194304] 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:6/674: rename d8/d45/fe8 to d8/de/d13/d11/d36/d3d/d4c/d26/d5b/ff8 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:6/675: rename d8/d1d/ccb to d8/de/d13/d11/d36/d53/cf9 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:6/676: read d8/de/d17/f7b [471059,7342] 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:6/677: symlink d8/de/dd5/lfa 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:4/709: dwrite da/d3c/d83/f8a [4194304,4194304] 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:6/678: creat d8/de/d13/d11/d36/d3d/ffb x:0 0 0 2024-04-12T05:09:07.379 INFO:tasks.workunit.client.0.smithi113.stdout:4/710: chown da/de/f62 499710243 1 2024-04-12T05:09:07.380 INFO:tasks.workunit.client.0.smithi113.stdout:6/679: symlink d8/de/d13/d11/d36/d3d/d4c/d31/lfc 0 2024-04-12T05:09:07.380 INFO:tasks.workunit.client.0.smithi113.stdout:4/711: mknod da/d24/ce9 0 2024-04-12T05:09:07.380 INFO:tasks.workunit.client.0.smithi113.stdout:6/680: creat d8/de/dd5/de7/ffd x:0 0 0 2024-04-12T05:09:07.380 INFO:tasks.workunit.client.0.smithi113.stdout:4/712: rename da/d3c/d83/f8a to da/d3c/d83/d47/dbd/fea 0 2024-04-12T05:09:07.380 INFO:tasks.workunit.client.0.smithi113.stdout:6/681: truncate d8/de/f16 4286058 0 2024-04-12T05:09:07.380 INFO:tasks.workunit.client.0.smithi113.stdout:6/682: dread - d8/de/d13/d11/d36/d3d/d4c/dd8/ff7 zero size 2024-04-12T05:09:07.380 INFO:tasks.workunit.client.0.smithi113.stdout:0/734: sync 2024-04-12T05:09:07.380 INFO:tasks.workunit.client.0.smithi113.stdout:5/717: sync 2024-04-12T05:09:07.380 INFO:tasks.workunit.client.0.smithi113.stdout:6/683: creat d8/de/d13/d11/d36/d2c/dca/ffe x:0 0 0 2024-04-12T05:09:07.380 INFO:tasks.workunit.client.0.smithi113.stdout:4/713: write da/d73/f71 [478841,76666] 0 2024-04-12T05:09:07.380 INFO:tasks.workunit.client.0.smithi113.stdout:6/684: rename d8/de/d13/df0/f69 to d8/de/d13/df0/d66/dc4/de3/fff 0 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:6/685: chown d8/de/d13/d11/d36/f4e 14509150 1 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:6/686: chown d8/de/l21 1 1 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:4/714: fdatasync da/d4f/fcf 0 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:6/687: stat d8/de/d17/f27 0 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:5/718: dread d0/d10/d27/d1d/d3f/d73/f76 [0,4194304] 0 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:4/715: mknod da/de/dc8/ceb 0 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:5/719: symlink d0/d10/d27/d1d/d38/d3b/led 0 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:6/688: dread d8/de/d13/d11/d36/d3d/d4c/d31/f62 [0,4194304] 0 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:5/720: mknod d0/d10/d27/d1d/d3f/d73/dcd/cee 0 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:6/689: mkdir d8/de/d13/d11/d36/d3d/d4c/d26/d64/d100 0 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:5/721: stat d0/d10/d27/d1d/d8c/db8/fd5 0 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:5/722: mknod d0/d10/d27/d1d/d38/d44/d4e/cef 0 2024-04-12T05:09:07.381 INFO:tasks.workunit.client.0.smithi113.stdout:5/723: mknod d0/d10/d27/d1d/d38/d44/d66/cf0 0 2024-04-12T05:09:07.383 INFO:tasks.workunit.client.1.smithi192.stdout:8/558: creat d6/d10/d46/d5e/d5f/d38/d3b/fae x:0 0 0 2024-04-12T05:09:07.383 INFO:tasks.workunit.client.1.smithi192.stdout:8/559: write d6/f95 [4369425,43648] 0 2024-04-12T05:09:07.383 INFO:tasks.workunit.client.1.smithi192.stdout:7/580: symlink d3/d7/d14/d1f/d76/lc4 0 2024-04-12T05:09:07.383 INFO:tasks.workunit.client.1.smithi192.stdout:6/698: fdatasync d3/d5/f45 0 2024-04-12T05:09:07.383 INFO:tasks.workunit.client.1.smithi192.stdout:7/581: symlink d3/d1d/d5a/d5f/d7a/lc5 0 2024-04-12T05:09:07.383 INFO:tasks.workunit.client.0.smithi113.stdout:5/724: chown d0/d10/f16 318723499 1 2024-04-12T05:09:07.384 INFO:tasks.workunit.client.0.smithi113.stdout:5/725: write d0/d10/d27/d1d/d38/d44/d4e/f64 [405775,114427] 0 2024-04-12T05:09:07.384 INFO:tasks.workunit.client.0.smithi113.stdout:6/690: dread d8/de/d13/d11/d36/d3d/d4c/d31/f54 [0,4194304] 0 2024-04-12T05:09:07.384 INFO:tasks.workunit.client.0.smithi113.stdout:5/726: chown d0/d10/d27/d1d/d3f/d73/db9 247 1 2024-04-12T05:09:07.384 INFO:tasks.workunit.client.0.smithi113.stdout:6/691: rmdir d8/de/dd5 39 2024-04-12T05:09:07.384 INFO:tasks.workunit.client.0.smithi113.stdout:0/735: getdents d1/d6/d8e 0 2024-04-12T05:09:07.385 INFO:tasks.workunit.client.1.smithi192.stdout:5/632: write d0/d15/d3e/d77/f78 [3640921,103333] 0 2024-04-12T05:09:07.385 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:06.382267+0000 mon.smithi113 (mon.0) 675 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:07.385 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:06.396439+0000 mon.smithi113 (mon.0) 676 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:07.385 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:06.673140+0000 mgr.smithi192.ocppyo (mgr.24357) 29 : cluster [DBG] pgmap v23: 65 pgs: 65 active+clean; 2.1 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 60 MiB/s rd, 254 MiB/s wr, 431 op/s 2024-04-12T05:09:07.385 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:07.119747+0000 mon.smithi113 (mon.0) 677 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:07.385 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:07.142494+0000 mon.smithi113 (mon.0) 678 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:07.385 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:07.149335+0000 mon.smithi192 (mon.1) 50 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.386 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:07.149768+0000 mon.smithi113 (mon.0) 679 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.386 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:07.151393+0000 mon.smithi192 (mon.1) 51 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.386 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:07.151723+0000 mon.smithi113 (mon.0) 680 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.386 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:07.153636+0000 mon.smithi192 (mon.1) 52 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.386 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:07.153977+0000 mon.smithi113 (mon.0) 681 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.386 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:07.154944+0000 mgr.smithi192.ocppyo (mgr.24357) 30 : cephadm [INF] Adjusting osd_memory_target on smithi113 to 1910M 2024-04-12T05:09:07.386 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:07.161611+0000 mon.smithi113 (mon.0) 682 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:07.386 INFO:tasks.workunit.client.1.smithi192.stdout:2/636: link d1/d8/d17/d19/d40/c4f d1/d8/d17/d19/d71/d7f/cc3 0 2024-04-12T05:09:07.387 INFO:tasks.workunit.client.1.smithi192.stdout:2/637: truncate d1/f4b 877249 0 2024-04-12T05:09:07.387 INFO:tasks.workunit.client.1.smithi192.stdout:5/633: write d0/d15/d24/d29/f2a [4299538,126258] 0 2024-04-12T05:09:07.388 INFO:tasks.workunit.client.0.smithi113.stdout:5/727: creat d0/d10/d27/d1d/d38/d3b/d6b/ff1 x:0 0 0 2024-04-12T05:09:07.388 INFO:tasks.workunit.client.1.smithi192.stdout:7/582: creat d3/d1d/d5a/d5f/d7a/fc6 x:0 0 0 2024-04-12T05:09:07.389 INFO:tasks.workunit.client.0.smithi113.stdout:6/692: readlink d8/de/d13/d11/d36/d3d/d4c/d31/l80 0 2024-04-12T05:09:07.390 INFO:tasks.workunit.client.0.smithi113.stdout:5/728: dread - d0/d10/d27/d1d/d3f/d7e/fae zero size 2024-04-12T05:09:07.391 INFO:tasks.workunit.client.1.smithi192.stdout:5/634: write d0/d15/d3e/dca/dda/db5/dbe/fd8 [871183,108881] 0 2024-04-12T05:09:07.391 INFO:tasks.workunit.client.0.smithi113.stdout:6/693: fdatasync d8/de/d13/d11/d36/f35 0 2024-04-12T05:09:07.391 INFO:tasks.workunit.client.1.smithi192.stdout:6/699: dread d3/dc/d39/f3a [0,4194304] 0 2024-04-12T05:09:07.392 INFO:tasks.workunit.client.1.smithi192.stdout:8/560: creat d6/d10/d46/d5e/faf x:0 0 0 2024-04-12T05:09:07.393 INFO:tasks.workunit.client.1.smithi192.stdout:5/635: write d0/d15/d32/d69/d7d/fc9 [3843362,111495] 0 2024-04-12T05:09:07.394 INFO:tasks.workunit.client.1.smithi192.stdout:8/561: fsync d6/d10/d46/d5e/d5f/f4d 0 2024-04-12T05:09:07.396 INFO:tasks.workunit.client.1.smithi192.stdout:7/583: creat d3/d1d/d5a/d5f/d6a/fc7 x:0 0 0 2024-04-12T05:09:07.398 INFO:tasks.workunit.client.1.smithi192.stdout:6/700: dread d3/d8/f17 [4194304,4194304] 0 2024-04-12T05:09:07.398 INFO:tasks.workunit.client.0.smithi113.stdout:0/736: link d1/d6/d16/d8f/fce d1/fe6 0 2024-04-12T05:09:07.399 INFO:tasks.workunit.client.1.smithi192.stdout:6/701: write d3/d8/d10/d4c/fb0 [368838,50718] 0 2024-04-12T05:09:07.401 INFO:tasks.workunit.client.0.smithi113.stdout:6/694: read - d8/de/d13/d11/d36/d3d/d4c/d26/d5b/ff8 zero size 2024-04-12T05:09:07.403 INFO:tasks.workunit.client.1.smithi192.stdout:5/636: symlink d0/d15/d32/d69/d7d/ldc 0 2024-04-12T05:09:07.404 INFO:tasks.workunit.client.1.smithi192.stdout:8/562: readlink d6/d10/d46/d5e/d5f/d31/d61/la7 0 2024-04-12T05:09:07.404 INFO:tasks.workunit.client.0.smithi113.stdout:5/729: dwrite d0/d10/d27/d1d/fbf [0,4194304] 0 2024-04-12T05:09:07.405 INFO:tasks.workunit.client.1.smithi192.stdout:8/563: chown d6/d10/d46/d5e/d5f/f43 30476 1 2024-04-12T05:09:07.409 INFO:tasks.workunit.client.1.smithi192.stdout:7/584: symlink d3/d1d/d74/d86/lc8 0 2024-04-12T05:09:07.411 INFO:tasks.workunit.client.1.smithi192.stdout:2/638: link d1/d8/d17/d19/d60/d59/l9a d1/d8/d17/db3/lc4 0 2024-04-12T05:09:07.412 INFO:tasks.workunit.client.1.smithi192.stdout:8/564: dwrite d6/da/f1e [0,4194304] 0 2024-04-12T05:09:07.412 INFO:tasks.workunit.client.0.smithi113.stdout:6/695: creat d8/de/dd5/f101 x:0 0 0 2024-04-12T05:09:07.413 INFO:tasks.workunit.client.0.smithi113.stdout:0/737: rename d1/d6/d34/c7f to d1/d43/ce7 0 2024-04-12T05:09:07.417 INFO:tasks.workunit.client.0.smithi113.stdout:5/730: rename d0/d10/d27/d1d/d38/d44/fb3 to d0/d10/d27/d1d/d8c/dde/de7/ff2 0 2024-04-12T05:09:07.420 INFO:tasks.workunit.client.0.smithi113.stdout:6/696: mkdir d8/de/d13/d11/d102 0 2024-04-12T05:09:07.421 INFO:tasks.workunit.client.1.smithi192.stdout:6/702: mkdir d3/d8/d22/dd4/de9 0 2024-04-12T05:09:07.421 INFO:tasks.workunit.client.1.smithi192.stdout:5/637: chown d0/d15/f8c 193523827 1 2024-04-12T05:09:07.421 INFO:tasks.workunit.client.1.smithi192.stdout:7/585: rename d3/d1d/d48/c99 to d3/d1d/d5a/d5f/d7a/cc9 0 2024-04-12T05:09:07.422 INFO:tasks.workunit.client.0.smithi113.stdout:5/731: creat d0/d10/d27/d1d/d41/ff3 x:0 0 0 2024-04-12T05:09:07.422 INFO:tasks.workunit.client.1.smithi192.stdout:2/639: symlink d1/d8/lc5 0 2024-04-12T05:09:07.423 INFO:tasks.workunit.client.0.smithi113.stdout:5/732: truncate d0/d10/d6e/fd8 284264 0 2024-04-12T05:09:07.424 INFO:tasks.workunit.client.1.smithi192.stdout:5/638: write d0/d15/f23 [1922255,44103] 0 2024-04-12T05:09:07.424 INFO:tasks.workunit.client.1.smithi192.stdout:7/586: creat d3/d1d/d4a/d5e/fca x:0 0 0 2024-04-12T05:09:07.424 INFO:tasks.workunit.client.0.smithi113.stdout:6/697: unlink d8/de/d13/d11/la5 0 2024-04-12T05:09:07.426 INFO:tasks.workunit.client.1.smithi192.stdout:6/703: creat d3/d8/de6/fea x:0 0 0 2024-04-12T05:09:07.426 INFO:tasks.workunit.client.0.smithi113.stdout:5/733: read - d0/d10/d27/d1d/d8c/f9d zero size 2024-04-12T05:09:07.427 INFO:tasks.workunit.client.0.smithi113.stdout:6/698: rmdir d8/de/dd5/de7/db3/dc5 39 2024-04-12T05:09:07.428 INFO:tasks.workunit.client.1.smithi192.stdout:5/639: unlink d0/d21/ca5 0 2024-04-12T05:09:07.431 INFO:tasks.workunit.client.1.smithi192.stdout:8/565: dwrite d6/d10/d46/d5e/d5f/d38/d3b/f73 [0,4194304] 0 2024-04-12T05:09:07.431 INFO:tasks.workunit.client.1.smithi192.stdout:7/587: dread d3/d1d/d5a/d5f/f31 [0,4194304] 0 2024-04-12T05:09:07.434 INFO:tasks.workunit.client.1.smithi192.stdout:7/588: readlink d3/d1d/d5a/d5f/d6a/l83 0 2024-04-12T05:09:07.435 INFO:tasks.workunit.client.0.smithi113.stdout:6/699: rename d8/d45/f6e to d8/de/dd5/de7/db3/dc5/f103 0 2024-04-12T05:09:07.440 INFO:tasks.workunit.client.1.smithi192.stdout:7/589: unlink d3/d7/d14/l51 0 2024-04-12T05:09:07.441 INFO:tasks.workunit.client.1.smithi192.stdout:7/590: stat d3/d7/d15/d1a/f3e 0 2024-04-12T05:09:07.441 INFO:tasks.workunit.client.0.smithi113.stdout:6/700: unlink d8/de/d13/d11/d36/d3d/d4c/d26/fc1 0 2024-04-12T05:09:07.442 INFO:tasks.workunit.client.0.smithi113.stdout:5/734: write d0/d10/d27/d1d/d3f/d7e/f8b [775871,104874] 0 2024-04-12T05:09:07.445 INFO:tasks.workunit.client.0.smithi113.stdout:6/701: creat d8/de/d13/df0/f104 x:0 0 0 2024-04-12T05:09:07.445 INFO:tasks.workunit.client.1.smithi192.stdout:2/640: write d1/d2d/d73/f7a [569317,88327] 0 2024-04-12T05:09:07.447 INFO:tasks.workunit.client.0.smithi113.stdout:6/702: dread - d8/de/d13/fe0 zero size 2024-04-12T05:09:07.447 INFO:tasks.workunit.client.0.smithi113.stdout:5/735: symlink d0/d10/d27/d1d/d38/d44/d4e/daa/lf4 0 2024-04-12T05:09:07.447 INFO:tasks.workunit.client.1.smithi192.stdout:6/704: truncate d3/d8/d56/f75 3171416 0 2024-04-12T05:09:07.449 INFO:tasks.workunit.client.1.smithi192.stdout:8/566: write d6/d10/d46/d5e/d5f/d37/d4f/f9a [355204,18113] 0 2024-04-12T05:09:07.450 INFO:tasks.workunit.client.0.smithi113.stdout:0/738: dwrite d1/d6/d16/f47 [0,4194304] 0 2024-04-12T05:09:07.451 INFO:tasks.workunit.client.1.smithi192.stdout:6/705: mkdir d3/d8/d56/d59/d97/dbf/deb 0 2024-04-12T05:09:07.452 INFO:tasks.workunit.client.0.smithi113.stdout:5/736: creat d0/ff5 x:0 0 0 2024-04-12T05:09:07.452 INFO:tasks.workunit.client.0.smithi113.stdout:6/703: dread d8/de/d13/d11/d36/d3d/d4c/d31/f62 [0,4194304] 0 2024-04-12T05:09:07.452 INFO:tasks.workunit.client.1.smithi192.stdout:5/640: dwrite d0/f52 [0,4194304] 0 2024-04-12T05:09:07.452 INFO:tasks.workunit.client.0.smithi113.stdout:0/739: write d1/d6/d34/f78 [4880782,81797] 0 2024-04-12T05:09:07.453 INFO:tasks.workunit.client.1.smithi192.stdout:5/641: write d0/fcf [834728,64014] 0 2024-04-12T05:09:07.455 INFO:tasks.workunit.client.1.smithi192.stdout:8/567: write d6/d10/d46/d5e/d5f/d37/f68 [831934,64363] 0 2024-04-12T05:09:07.456 INFO:tasks.workunit.client.1.smithi192.stdout:6/706: creat d3/d8/de6/fec x:0 0 0 2024-04-12T05:09:07.459 INFO:tasks.workunit.client.0.smithi113.stdout:5/737: creat d0/d10/d6e/ff6 x:0 0 0 2024-04-12T05:09:07.459 INFO:tasks.workunit.client.1.smithi192.stdout:8/568: readlink d6/d10/d46/d5e/d5f/d38/l91 0 2024-04-12T05:09:07.463 INFO:tasks.workunit.client.1.smithi192.stdout:5/642: creat d0/d15/d24/d47/fdd x:0 0 0 2024-04-12T05:09:07.469 INFO:tasks.workunit.client.1.smithi192.stdout:8/569: creat d6/d1f/d9c/fb0 x:0 0 0 2024-04-12T05:09:07.470 INFO:tasks.workunit.client.1.smithi192.stdout:5/643: dwrite d0/f6e [0,4194304] 0 2024-04-12T05:09:07.470 INFO:tasks.workunit.client.1.smithi192.stdout:8/570: dread - d6/d1f/d5d/fa2 zero size 2024-04-12T05:09:07.471 INFO:tasks.workunit.client.1.smithi192.stdout:6/707: getdents d3/d23/db5 0 2024-04-12T05:09:07.474 INFO:tasks.workunit.client.0.smithi113.stdout:0/740: truncate d1/d6/d16/f47 302563 0 2024-04-12T05:09:07.476 INFO:tasks.workunit.client.1.smithi192.stdout:8/571: dwrite d6/d21/f60 [0,4194304] 0 2024-04-12T05:09:07.477 INFO:tasks.workunit.client.1.smithi192.stdout:8/572: write d6/d1f/d9c/fb0 [693146,113740] 0 2024-04-12T05:09:07.480 INFO:tasks.workunit.client.1.smithi192.stdout:8/573: chown d6/d10/d46/d5e/d5f/d37/d4f/f9a 18615 1 2024-04-12T05:09:07.481 INFO:tasks.workunit.client.1.smithi192.stdout:8/574: chown d6/d10/l99 1033 1 2024-04-12T05:09:07.484 INFO:tasks.workunit.client.1.smithi192.stdout:5/644: link d0/c3 d0/d15/d32/d69/d7d/cde 0 2024-04-12T05:09:07.486 INFO:tasks.workunit.client.0.smithi113.stdout:5/738: dread d0/d10/d6e/f4f [0,4194304] 0 2024-04-12T05:09:07.486 INFO:tasks.workunit.client.0.smithi113.stdout:0/741: getdents d1/d6/d8e 0 2024-04-12T05:09:07.487 INFO:tasks.workunit.client.1.smithi192.stdout:8/575: dwrite d6/d21/f44 [0,4194304] 0 2024-04-12T05:09:07.487 INFO:tasks.workunit.client.0.smithi113.stdout:5/739: stat d0/d10/d27/d1d/f49 0 2024-04-12T05:09:07.489 INFO:tasks.workunit.client.0.smithi113.stdout:0/742: symlink d1/d6/d8e/le8 0 2024-04-12T05:09:07.491 INFO:tasks.workunit.client.0.smithi113.stdout:5/740: symlink d0/d10/d6e/lf7 0 2024-04-12T05:09:07.493 INFO:tasks.workunit.client.0.smithi113.stdout:0/743: truncate d1/d43/d59/f82 1028097 0 2024-04-12T05:09:07.494 INFO:tasks.workunit.client.1.smithi192.stdout:5/645: mkdir d0/d15/d24/ddf 0 2024-04-12T05:09:07.497 INFO:tasks.workunit.client.0.smithi113.stdout:0/744: mkdir d1/d43/d59/de9 0 2024-04-12T05:09:07.497 INFO:tasks.workunit.client.1.smithi192.stdout:5/646: rmdir d0/d15/d3e/dca/dda/db5 39 2024-04-12T05:09:07.498 INFO:tasks.workunit.client.0.smithi113.stdout:5/741: dwrite d0/d10/f56 [0,4194304] 0 2024-04-12T05:09:07.499 INFO:tasks.workunit.client.1.smithi192.stdout:6/708: dwrite d3/d8/d22/f65 [0,4194304] 0 2024-04-12T05:09:07.500 INFO:tasks.workunit.client.0.smithi113.stdout:0/745: symlink d1/d6/d8e/lea 0 2024-04-12T05:09:07.501 INFO:tasks.workunit.client.0.smithi113.stdout:5/742: mkdir d0/d10/d27/d1d/d3f/d7e/df8 0 2024-04-12T05:09:07.503 INFO:tasks.workunit.client.0.smithi113.stdout:0/746: fdatasync d1/d6/d34/dc3/f2b 0 2024-04-12T05:09:07.508 INFO:tasks.workunit.client.1.smithi192.stdout:8/576: dwrite d6/d10/d46/d5e/d5f/d37/f6f [0,4194304] 0 2024-04-12T05:09:07.508 INFO:tasks.workunit.client.0.smithi113.stdout:0/747: creat d1/d6/d16/d2a/feb x:0 0 0 2024-04-12T05:09:07.509 INFO:tasks.workunit.client.0.smithi113.stdout:5/743: creat d0/d10/d27/d1d/d3f/d7e/df8/ff9 x:0 0 0 2024-04-12T05:09:07.517 INFO:tasks.workunit.client.1.smithi192.stdout:6/709: getdents d3/d8/d56/d59/ddf 0 2024-04-12T05:09:07.517 INFO:tasks.workunit.client.1.smithi192.stdout:5/647: dwrite d0/f61 [0,4194304] 0 2024-04-12T05:09:07.520 INFO:tasks.workunit.client.0.smithi113.stdout:5/744: dwrite d0/d10/d27/d1d/d8c/db8/dc6/fec [0,4194304] 0 2024-04-12T05:09:07.520 INFO:tasks.workunit.client.1.smithi192.stdout:6/710: chown d3/d5 23 1 2024-04-12T05:09:07.523 INFO:tasks.workunit.client.1.smithi192.stdout:5/648: symlink d0/d15/d64/le0 0 2024-04-12T05:09:07.525 INFO:tasks.workunit.client.1.smithi192.stdout:8/577: dwrite d6/d10/d46/d5e/d5f/d38/d3b/fae [0,4194304] 0 2024-04-12T05:09:07.528 INFO:tasks.workunit.client.0.smithi113.stdout:5/745: dwrite d0/d10/d27/d1d/d3f/d7e/fae [0,4194304] 0 2024-04-12T05:09:07.531 INFO:tasks.workunit.client.1.smithi192.stdout:5/649: mkdir d0/d15/d3e/dca/dda/db5/dd7/de1 0 2024-04-12T05:09:07.534 INFO:tasks.workunit.client.1.smithi192.stdout:5/650: fdatasync d0/d15/d32/d69/d7d/fac 0 2024-04-12T05:09:07.537 INFO:tasks.workunit.client.0.smithi113.stdout:5/746: mknod d0/d10/d6e/d93/dc9/cfa 0 2024-04-12T05:09:07.537 INFO:tasks.workunit.client.1.smithi192.stdout:5/651: symlink d0/d21/le2 0 2024-04-12T05:09:07.539 INFO:tasks.workunit.client.1.smithi192.stdout:8/578: dwrite d6/d10/d46/d5e/d5f/d38/d3b/fa1 [0,4194304] 0 2024-04-12T05:09:07.541 INFO:tasks.workunit.client.1.smithi192.stdout:5/652: stat d0/d15/d32/db2 0 2024-04-12T05:09:07.544 INFO:tasks.workunit.client.0.smithi113.stdout:5/747: link d0/d10/d27/d1d/d38/d75/l81 d0/d10/d27/d1d/d3f/db0/lfb 0 2024-04-12T05:09:07.549 INFO:tasks.workunit.client.1.smithi192.stdout:8/579: mkdir d6/d10/d46/d5e/d5f/d31/d80/db1 0 2024-04-12T05:09:07.550 INFO:tasks.workunit.client.0.smithi113.stdout:5/748: rename d0/d10/d27/d1d/d38/c79 to d0/d10/d27/d1d/d8c/db8/cfc 0 2024-04-12T05:09:07.552 INFO:tasks.workunit.client.1.smithi192.stdout:8/580: dread d6/d21/f60 [0,4194304] 0 2024-04-12T05:09:07.553 INFO:tasks.workunit.client.0.smithi113.stdout:5/749: mknod d0/d10/d27/d1d/d38/d44/d66/cfd 0 2024-04-12T05:09:07.557 INFO:tasks.workunit.client.1.smithi192.stdout:8/581: truncate d6/d10/d46/d7c/f8c 826395 0 2024-04-12T05:09:07.558 INFO:tasks.workunit.client.1.smithi192.stdout:8/582: creat d6/d10/d46/d5e/d5f/d37/d4f/fb2 x:0 0 0 2024-04-12T05:09:07.558 INFO:tasks.workunit.client.0.smithi113.stdout:5/750: link d0/d10/d27/d1d/c37 d0/cfe 0 2024-04-12T05:09:07.561 INFO:tasks.workunit.client.1.smithi192.stdout:8/583: unlink d6/d10/d46/d5e/d5f/d37/f68 0 2024-04-12T05:09:07.567 INFO:tasks.workunit.client.1.smithi192.stdout:8/584: dwrite d6/f95 [0,4194304] 0 2024-04-12T05:09:07.567 INFO:tasks.workunit.client.0.smithi113.stdout:0/748: dread d1/d6/d16/d2a/d50/d52/d7e/f8d [0,4194304] 0 2024-04-12T05:09:07.569 INFO:tasks.workunit.client.1.smithi192.stdout:8/585: symlink d6/lb3 0 2024-04-12T05:09:07.574 INFO:tasks.workunit.client.0.smithi113.stdout:0/749: dwrite d1/d6/d16/dd5/fc7 [0,4194304] 0 2024-04-12T05:09:07.577 INFO:tasks.workunit.client.0.smithi113.stdout:5/751: rmdir d0 39 2024-04-12T05:09:07.583 INFO:tasks.workunit.client.0.smithi113.stdout:5/752: link d0/d10/d27/d1d/d3f/d7e/f8b d0/d10/d27/d1d/d3f/d73/fff 0 2024-04-12T05:09:07.586 INFO:tasks.workunit.client.0.smithi113.stdout:1/793: sync 2024-04-12T05:09:07.587 INFO:tasks.workunit.client.0.smithi113.stdout:5/753: unlink d0/d10/d27/d1d/d38/f52 0 2024-04-12T05:09:07.589 INFO:tasks.workunit.client.0.smithi113.stdout:1/794: mknod d10/da5/ddf/c106 0 2024-04-12T05:09:07.591 INFO:tasks.workunit.client.0.smithi113.stdout:1/795: read - d10/d14/d95/dcf/ffa zero size 2024-04-12T05:09:07.591 INFO:tasks.workunit.client.0.smithi113.stdout:0/750: write d1/fe6 [539871,37664] 0 2024-04-12T05:09:07.595 INFO:tasks.workunit.client.0.smithi113.stdout:0/751: rename d1/d6/d16/d8f/fb9 to d1/d6/d8e/fec 0 2024-04-12T05:09:07.596 INFO:tasks.workunit.client.0.smithi113.stdout:0/752: dread - d1/d6/d16/d2a/d50/f85 zero size 2024-04-12T05:09:07.596 INFO:tasks.workunit.client.1.smithi192.stdout:8/586: dread d6/d21/f23 [0,4194304] 0 2024-04-12T05:09:07.600 INFO:tasks.workunit.client.0.smithi113.stdout:5/754: dread d0/d10/d27/d1d/d38/d44/fa3 [0,4194304] 0 2024-04-12T05:09:07.602 INFO:tasks.workunit.client.0.smithi113.stdout:5/755: fsync d0/d10/d27/d1d/d41/f9e 0 2024-04-12T05:09:07.602 INFO:tasks.workunit.client.0.smithi113.stdout:0/753: truncate d1/d6/d34/dc3/f2b 1655027 0 2024-04-12T05:09:07.603 INFO:tasks.workunit.client.0.smithi113.stdout:5/756: chown d0/d10/d27/d1d/d8c/f9d 52538 1 2024-04-12T05:09:07.604 INFO:tasks.workunit.client.1.smithi192.stdout:8/587: write d6/da/fd [2683230,10536] 0 2024-04-12T05:09:07.605 INFO:tasks.workunit.client.0.smithi113.stdout:5/757: fsync d0/d10/d27/d1d/d38/d44/d4e/f64 0 2024-04-12T05:09:07.606 INFO:tasks.workunit.client.1.smithi192.stdout:8/588: mkdir d6/d10/d6c/db4 0 2024-04-12T05:09:07.607 INFO:tasks.workunit.client.0.smithi113.stdout:5/758: symlink d0/d10/d27/d1d/d38/d44/dcb/l100 0 2024-04-12T05:09:07.609 INFO:tasks.workunit.client.0.smithi113.stdout:5/759: fsync d0/d10/d27/d1d/fbf 0 2024-04-12T05:09:07.614 INFO:tasks.workunit.client.0.smithi113.stdout:5/760: getdents d0/d10/d27/d1d/d3f/d73/d89 0 2024-04-12T05:09:07.615 INFO:tasks.workunit.client.0.smithi113.stdout:5/761: write d0/ff5 [402350,23041] 0 2024-04-12T05:09:07.615 INFO:tasks.workunit.client.0.smithi113.stdout:0/754: dread d1/fc9 [0,4194304] 0 2024-04-12T05:09:07.618 INFO:tasks.workunit.client.0.smithi113.stdout:0/755: readlink d1/d6/d16/d2a/d50/d52/lb7 0 2024-04-12T05:09:07.621 INFO:tasks.workunit.client.0.smithi113.stdout:0/756: fdatasync d1/d6/d34/dc3/f1a 0 2024-04-12T05:09:07.623 INFO:tasks.workunit.client.0.smithi113.stdout:5/762: dwrite d0/f31 [0,4194304] 0 2024-04-12T05:09:07.624 INFO:tasks.workunit.client.1.smithi192.stdout:8/589: dread d6/d10/d46/d5e/d5f/d37/d4f/d51/f66 [0,4194304] 0 2024-04-12T05:09:07.625 INFO:tasks.workunit.client.0.smithi113.stdout:5/763: write d0/d10/d6e/ff6 [496544,72932] 0 2024-04-12T05:09:07.625 INFO:tasks.workunit.client.1.smithi192.stdout:8/590: write d6/d1f/f2a [4584072,75672] 0 2024-04-12T05:09:07.625 INFO:tasks.workunit.client.0.smithi113.stdout:0/757: unlink d1/d80/c86 0 2024-04-12T05:09:07.631 INFO:tasks.workunit.client.1.smithi192.stdout:8/591: dwrite d6/d10/d46/d5e/d5f/d38/d3b/fae [0,4194304] 0 2024-04-12T05:09:07.639 INFO:tasks.workunit.client.0.smithi113.stdout:5/764: dread d0/d10/d27/d1d/d3f/f47 [0,4194304] 0 2024-04-12T05:09:07.639 INFO:tasks.workunit.client.1.smithi192.stdout:8/592: truncate d6/d10/fa5 2625714 0 2024-04-12T05:09:07.643 INFO:tasks.workunit.client.1.smithi192.stdout:8/593: mkdir d6/d10/d46/d5e/d5f/d31/d61/d9d/db5 0 2024-04-12T05:09:07.644 INFO:tasks.workunit.client.1.smithi192.stdout:8/594: stat d6/d10/d46/d5e/d5f/d38/d3b/fa1 0 2024-04-12T05:09:07.645 INFO:tasks.workunit.client.0.smithi113.stdout:5/765: write d0/d10/d27/d1d/d38/d44/d66/f96 [5849693,70119] 0 2024-04-12T05:09:07.648 INFO:tasks.workunit.client.1.smithi192.stdout:8/595: dread d6/d1f/d5d/f8d [0,4194304] 0 2024-04-12T05:09:07.654 INFO:tasks.workunit.client.0.smithi113.stdout:0/758: dread d1/d6/d16/d2a/d50/f77 [0,4194304] 0 2024-04-12T05:09:07.657 INFO:tasks.workunit.client.1.smithi192.stdout:8/596: dwrite d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/f72 [0,4194304] 0 2024-04-12T05:09:07.660 INFO:tasks.workunit.client.0.smithi113.stdout:0/759: truncate d1/d6/f5d 367365 0 2024-04-12T05:09:07.660 INFO:tasks.workunit.client.0.smithi113.stdout:5/766: dwrite d0/d10/d27/d5f/f88 [0,4194304] 0 2024-04-12T05:09:07.660 INFO:tasks.workunit.client.1.smithi192.stdout:8/597: truncate d6/d10/d46/d5e/d5f/d37/d4f/d4c/f94 151733 0 2024-04-12T05:09:07.667 INFO:tasks.workunit.client.1.smithi192.stdout:8/598: rename d6/d21/f33 to d6/d10/d46/d5e/d5f/d31/d80/db1/fb6 0 2024-04-12T05:09:07.670 INFO:tasks.workunit.client.1.smithi192.stdout:8/599: read d6/d10/f11 [647134,20967] 0 2024-04-12T05:09:07.673 INFO:tasks.workunit.client.0.smithi113.stdout:0/760: dwrite d1/d6/d16/d2a/d50/db6/fdd [0,4194304] 0 2024-04-12T05:09:07.680 INFO:tasks.workunit.client.1.smithi192.stdout:3/618: sync 2024-04-12T05:09:07.680 INFO:tasks.workunit.client.1.smithi192.stdout:9/530: sync 2024-04-12T05:09:07.680 INFO:tasks.workunit.client.1.smithi192.stdout:4/709: sync 2024-04-12T05:09:07.680 INFO:tasks.workunit.client.0.smithi113.stdout:0/761: read d1/fd7 [1343638,111748] 0 2024-04-12T05:09:07.683 INFO:tasks.workunit.client.1.smithi192.stdout:8/600: dread d6/f18 [0,4194304] 0 2024-04-12T05:09:07.684 INFO:tasks.workunit.client.0.smithi113.stdout:5/767: dread d0/d10/f1b [0,4194304] 0 2024-04-12T05:09:07.687 INFO:tasks.workunit.client.0.smithi113.stdout:0/762: creat d1/d43/d59/de9/fed x:0 0 0 2024-04-12T05:09:07.689 INFO:tasks.workunit.client.1.smithi192.stdout:8/601: dwrite d6/d10/d46/d5e/d5f/f57 [0,4194304] 0 2024-04-12T05:09:07.691 INFO:tasks.workunit.client.0.smithi113.stdout:0/763: symlink d1/d43/d59/lee 0 2024-04-12T05:09:07.695 INFO:tasks.workunit.client.0.smithi113.stdout:5/768: truncate d0/d10/f45 752158 0 2024-04-12T05:09:07.695 INFO:tasks.workunit.client.0.smithi113.stdout:0/764: write d1/d6/f30 [1358112,14068] 0 2024-04-12T05:09:07.697 INFO:tasks.workunit.client.0.smithi113.stdout:0/765: fdatasync d1/d6/d34/f40 0 2024-04-12T05:09:07.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:06.382267+0000 mon.smithi113 (mon.0) 675 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:07.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:06.396439+0000 mon.smithi113 (mon.0) 676 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:07.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:06.673140+0000 mgr.smithi192.ocppyo (mgr.24357) 29 : cluster [DBG] pgmap v23: 65 pgs: 65 active+clean; 2.1 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 60 MiB/s rd, 254 MiB/s wr, 431 op/s 2024-04-12T05:09:07.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:07.119747+0000 mon.smithi113 (mon.0) 677 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:07.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:07.142494+0000 mon.smithi113 (mon.0) 678 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:07.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:07.149335+0000 mon.smithi192 (mon.1) 50 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:07.149768+0000 mon.smithi113 (mon.0) 679 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:07.151393+0000 mon.smithi192 (mon.1) 51 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:07.151723+0000 mon.smithi113 (mon.0) 680 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:07.153636+0000 mon.smithi192 (mon.1) 52 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:07.153977+0000 mon.smithi113 (mon.0) 681 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:07.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:07.154944+0000 mgr.smithi192.ocppyo (mgr.24357) 30 : cephadm [INF] Adjusting osd_memory_target on smithi113 to 1910M 2024-04-12T05:09:07.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:07.161611+0000 mon.smithi113 (mon.0) 682 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:07.701 INFO:tasks.workunit.client.0.smithi113.stdout:0/766: read d1/d6/d16/f24 [96544,16556] 0 2024-04-12T05:09:07.702 INFO:tasks.workunit.client.0.smithi113.stdout:5/769: dread d0/d10/d6e/f4a [0,4194304] 0 2024-04-12T05:09:07.704 INFO:tasks.workunit.client.0.smithi113.stdout:5/770: write d0/d10/d27/d1d/f49 [2570138,120728] 0 2024-04-12T05:09:07.705 INFO:tasks.workunit.client.1.smithi192.stdout:9/531: mkdir d11/d25/d36/d6e/d33/dbb 0 2024-04-12T05:09:07.706 INFO:tasks.workunit.client.1.smithi192.stdout:3/619: mknod d5/d13/d3a/d70/d8c/d8d/cdb 0 2024-04-12T05:09:07.707 INFO:tasks.workunit.client.0.smithi113.stdout:5/771: truncate d0/d10/da1/fa5 655032 0 2024-04-12T05:09:07.708 INFO:tasks.workunit.client.1.smithi192.stdout:8/602: creat d6/da/fb7 x:0 0 0 2024-04-12T05:09:07.709 INFO:tasks.workunit.client.1.smithi192.stdout:4/710: symlink d1/d1d/lf5 0 2024-04-12T05:09:07.709 INFO:tasks.workunit.client.0.smithi113.stdout:5/772: symlink d0/d10/d27/d1d/d8c/db8/l101 0 2024-04-12T05:09:07.710 INFO:tasks.workunit.client.1.smithi192.stdout:8/603: chown d6/l17 78 1 2024-04-12T05:09:07.712 INFO:tasks.workunit.client.0.smithi113.stdout:5/773: creat d0/d10/d27/d1d/d38/d75/f102 x:0 0 0 2024-04-12T05:09:07.712 INFO:tasks.workunit.client.0.smithi113.stdout:0/767: link d1/l6a d1/d6/d34/dc3/lef 0 2024-04-12T05:09:07.713 INFO:tasks.workunit.client.1.smithi192.stdout:3/620: dwrite d5/fd [0,4194304] 0 2024-04-12T05:09:07.714 INFO:tasks.workunit.client.0.smithi113.stdout:5/774: dread - d0/d10/d27/d1d/d3f/d73/d89/f5a zero size 2024-04-12T05:09:07.717 INFO:tasks.workunit.client.1.smithi192.stdout:8/604: mkdir d6/d10/d46/d5e/d5f/d31/d61/d9d/db8 0 2024-04-12T05:09:07.717 INFO:tasks.workunit.client.1.smithi192.stdout:9/532: link d11/d15/d21/c68 d11/d15/d21/d2c/d1f/d6a/d7e/cbc 0 2024-04-12T05:09:07.718 INFO:tasks.workunit.client.0.smithi113.stdout:0/768: dread d1/d6/d8e/fcb [0,4194304] 0 2024-04-12T05:09:07.718 INFO:tasks.workunit.client.0.smithi113.stdout:5/775: rename d0/d10/d27/c2c to d0/d10/d6e/d93/c103 0 2024-04-12T05:09:07.722 INFO:tasks.workunit.client.1.smithi192.stdout:4/711: link d1/d54/d55/d64/fd4 d1/d1d/d53/d1c/d84/dbb/ff6 0 2024-04-12T05:09:07.723 INFO:tasks.workunit.client.1.smithi192.stdout:8/605: creat d6/d10/d46/d5e/d5f/d31/d61/d9d/fb9 x:0 0 0 2024-04-12T05:09:07.725 INFO:tasks.workunit.client.0.smithi113.stdout:5/776: getdents d0/d10/d27/d1d/d8c/db8/dc6 0 2024-04-12T05:09:07.727 INFO:tasks.workunit.client.1.smithi192.stdout:9/533: rename d11/d25/d36/d6e/d33/f4c to d11/d25/d36/d6e/d8f/fbd 0 2024-04-12T05:09:07.729 INFO:tasks.workunit.client.1.smithi192.stdout:4/712: mkdir d1/d1d/d53/d1c/d84/dbb/df7 0 2024-04-12T05:09:07.730 INFO:tasks.workunit.client.0.smithi113.stdout:5/777: link d0/d10/d27/d1d/d38/d44/d4e/daa/lf4 d0/d10/d27/d1d/d3f/d7e/df8/l104 0 2024-04-12T05:09:07.730 INFO:tasks.workunit.client.1.smithi192.stdout:3/621: creat d5/d13/d3a/fdc x:0 0 0 2024-04-12T05:09:07.731 INFO:tasks.workunit.client.0.smithi113.stdout:5/778: chown d0/d10/d27/d1d/d38/d44/dcb/fe1 0 1 2024-04-12T05:09:07.732 INFO:tasks.workunit.client.0.smithi113.stdout:0/769: rename d1/d6/d34/l8b to d1/d6/d16/d2a/d50/d52/lf0 0 2024-04-12T05:09:07.732 INFO:tasks.workunit.client.0.smithi113.stdout:5/779: read - d0/d10/d27/d1d/d3f/db0/dda/fdd zero size 2024-04-12T05:09:07.733 INFO:tasks.workunit.client.0.smithi113.stdout:0/770: dread - d1/d6/d16/dd5/fe3 zero size 2024-04-12T05:09:07.734 INFO:tasks.workunit.client.0.smithi113.stdout:0/771: write d1/f2f [1839080,96699] 0 2024-04-12T05:09:07.736 INFO:tasks.workunit.client.0.smithi113.stdout:5/780: creat d0/d10/d27/d1d/d38/d3b/d6b/f105 x:0 0 0 2024-04-12T05:09:07.736 INFO:tasks.workunit.client.1.smithi192.stdout:3/622: rmdir d5/d13/d3a/d4a 39 2024-04-12T05:09:07.738 INFO:tasks.workunit.client.0.smithi113.stdout:5/781: creat d0/d10/d27/d1d/d8c/dde/f106 x:0 0 0 2024-04-12T05:09:07.739 INFO:tasks.workunit.client.1.smithi192.stdout:8/606: dwrite d6/d10/d46/d5e/d5f/d31/d80/db1/fb6 [0,4194304] 0 2024-04-12T05:09:07.739 INFO:tasks.workunit.client.0.smithi113.stdout:0/772: write d1/d6/d16/f1c [6153190,34148] 0 2024-04-12T05:09:07.741 INFO:tasks.workunit.client.1.smithi192.stdout:4/713: truncate d1/d54/d55/d64/fae 1128442 0 2024-04-12T05:09:07.742 INFO:tasks.workunit.client.1.smithi192.stdout:8/607: write d6/d10/d46/d5e/d5f/d37/d4f/d4c/f64 [26800,64747] 0 2024-04-12T05:09:07.743 INFO:tasks.workunit.client.1.smithi192.stdout:3/623: write d5/d13/d3a/d4a/d61/d9f/fc3 [141200,62950] 0 2024-04-12T05:09:07.747 INFO:tasks.workunit.client.1.smithi192.stdout:9/534: getdents d11/d25/d36/d3d 0 2024-04-12T05:09:07.752 INFO:tasks.workunit.client.1.smithi192.stdout:8/608: creat d6/da/fba x:0 0 0 2024-04-12T05:09:07.754 INFO:tasks.workunit.client.1.smithi192.stdout:3/624: rmdir d5/d13/d3a/d70/d8c/d8d 39 2024-04-12T05:09:07.755 INFO:tasks.workunit.client.1.smithi192.stdout:9/535: symlink d11/d25/d36/d6e/d33/lbe 0 2024-04-12T05:09:07.756 INFO:tasks.workunit.client.0.smithi113.stdout:5/782: dwrite d0/d10/f16 [0,4194304] 0 2024-04-12T05:09:07.757 INFO:tasks.workunit.client.1.smithi192.stdout:9/536: read d11/d25/d36/d6e/d33/da2/f84 [157256,52240] 0 2024-04-12T05:09:07.757 INFO:tasks.workunit.client.1.smithi192.stdout:8/609: truncate d6/d10/d46/d5e/d5f/f1c 334533 0 2024-04-12T05:09:07.758 INFO:tasks.workunit.client.1.smithi192.stdout:9/537: write d11/d25/d36/f38 [1296572,45243] 0 2024-04-12T05:09:07.760 INFO:tasks.workunit.client.1.smithi192.stdout:3/625: symlink d5/d13/d3a/d4e/d57/ldd 0 2024-04-12T05:09:07.761 INFO:tasks.workunit.client.1.smithi192.stdout:4/714: creat d1/d54/d82/d71/d9e/ff8 x:0 0 0 2024-04-12T05:09:07.763 INFO:tasks.workunit.client.0.smithi113.stdout:5/783: write d0/d10/d27/d1d/d38/fb2 [521674,2447] 0 2024-04-12T05:09:07.764 INFO:tasks.workunit.client.1.smithi192.stdout:9/538: dread - d11/d25/d36/d6e/d33/f6c zero size 2024-04-12T05:09:07.765 INFO:tasks.workunit.client.0.smithi113.stdout:5/784: fdatasync d0/d10/d27/d1d/d41/fa0 0 2024-04-12T05:09:07.767 INFO:tasks.workunit.client.0.smithi113.stdout:5/785: rmdir d0/d10/d27/d1d/d38/d44/d4e/daa 39 2024-04-12T05:09:07.768 INFO:tasks.workunit.client.1.smithi192.stdout:3/626: mknod d5/d13/d1a/d53/cde 0 2024-04-12T05:09:07.770 INFO:tasks.workunit.client.0.smithi113.stdout:5/786: creat d0/d10/d27/d1d/d3f/d7e/df8/f107 x:0 0 0 2024-04-12T05:09:07.771 INFO:tasks.workunit.client.0.smithi113.stdout:3/632: sync 2024-04-12T05:09:07.771 INFO:tasks.workunit.client.0.smithi113.stdout:8/689: sync 2024-04-12T05:09:07.773 INFO:tasks.workunit.client.0.smithi113.stdout:5/787: mkdir d0/d10/d27/d1d/d3f/d7e/df8/d108 0 2024-04-12T05:09:07.774 INFO:tasks.workunit.client.0.smithi113.stdout:8/690: truncate d9/d1b/d8e/d93/fdc 760320 0 2024-04-12T05:09:07.775 INFO:tasks.workunit.client.1.smithi192.stdout:3/627: creat d5/d13/d3a/d4a/fdf x:0 0 0 2024-04-12T05:09:07.775 INFO:tasks.workunit.client.0.smithi113.stdout:8/691: readlink d9/d16/l58 0 2024-04-12T05:09:07.777 INFO:tasks.workunit.client.1.smithi192.stdout:3/628: write d5/d13/d3a/d8b/da2/da7/fd9 [770291,56261] 0 2024-04-12T05:09:07.777 INFO:tasks.workunit.client.1.smithi192.stdout:4/715: unlink d1/d54/d82/l9f 0 2024-04-12T05:09:07.777 INFO:tasks.workunit.client.0.smithi113.stdout:5/788: creat d0/d10/d27/d1d/d38/d3b/d6b/f109 x:0 0 0 2024-04-12T05:09:07.779 INFO:tasks.workunit.client.0.smithi113.stdout:8/692: fdatasync d9/d1b/f56 0 2024-04-12T05:09:07.779 INFO:tasks.workunit.client.1.smithi192.stdout:8/610: rename d6/d10/d46/d5e/d5f/d31/l29 to d6/d10/d46/d5e/lbb 0 2024-04-12T05:09:07.781 INFO:tasks.workunit.client.1.smithi192.stdout:3/629: rmdir d5/d13/d3a/d85 39 2024-04-12T05:09:07.784 INFO:tasks.workunit.client.0.smithi113.stdout:3/633: dwrite d0/d13/f19 [0,4194304] 0 2024-04-12T05:09:07.784 INFO:tasks.workunit.client.1.smithi192.stdout:8/611: mkdir d6/d10/d46/d5e/d5f/d38/dbc 0 2024-04-12T05:09:07.785 INFO:tasks.workunit.client.1.smithi192.stdout:8/612: dread - d6/d10/d46/d5e/d5f/fab zero size 2024-04-12T05:09:07.788 INFO:tasks.workunit.client.0.smithi113.stdout:3/634: creat d0/d13/d47/d94/fd4 x:0 0 0 2024-04-12T05:09:07.791 INFO:tasks.workunit.client.1.smithi192.stdout:1/637: sync 2024-04-12T05:09:07.791 INFO:tasks.workunit.client.0.smithi113.stdout:3/635: truncate d0/fa6 2915918 0 2024-04-12T05:09:07.791 INFO:tasks.workunit.client.1.smithi192.stdout:3/630: creat d5/d13/d3a/d85/fe0 x:0 0 0 2024-04-12T05:09:07.798 INFO:tasks.workunit.client.0.smithi113.stdout:0/773: dwrite d1/d6/d34/dc3/f45 [0,4194304] 0 2024-04-12T05:09:07.798 INFO:tasks.workunit.client.0.smithi113.stdout:5/789: dwrite d0/d10/d27/d1d/d38/d3b/d6b/fd4 [0,4194304] 0 2024-04-12T05:09:07.801 INFO:tasks.workunit.client.1.smithi192.stdout:9/539: write d11/d15/f45 [4681070,95661] 0 2024-04-12T05:09:07.802 INFO:tasks.workunit.client.1.smithi192.stdout:1/638: chown d0/d2/dd/d21/d33/fa6 15 1 2024-04-12T05:09:07.802 INFO:tasks.workunit.client.0.smithi113.stdout:8/693: dread d9/f55 [0,4194304] 0 2024-04-12T05:09:07.803 INFO:tasks.workunit.client.0.smithi113.stdout:3/636: fsync d0/d7e/fb3 0 2024-04-12T05:09:07.805 INFO:tasks.workunit.client.0.smithi113.stdout:3/637: chown d0/d13/d15/d28/d2a/f8e 163 1 2024-04-12T05:09:07.806 INFO:tasks.workunit.client.0.smithi113.stdout:8/694: mkdir d9/dd4/dde 0 2024-04-12T05:09:07.809 INFO:tasks.workunit.client.0.smithi113.stdout:5/790: rename d0/l3e to d0/d10/d27/d1d/d3f/l10a 0 2024-04-12T05:09:07.811 INFO:tasks.workunit.client.0.smithi113.stdout:0/774: fsync d1/d6/d16/d2a/f3f 0 2024-04-12T05:09:07.812 INFO:tasks.workunit.client.0.smithi113.stdout:8/695: read d9/f53 [215485,84601] 0 2024-04-12T05:09:07.813 INFO:tasks.workunit.client.0.smithi113.stdout:5/791: rename d0/d10/d27/d1d to d0/d10/d27/d1d/d3f/d7e/df8/d10b 22 2024-04-12T05:09:07.813 INFO:tasks.workunit.client.0.smithi113.stdout:3/638: creat d0/d13/dbf/fd5 x:0 0 0 2024-04-12T05:09:07.815 INFO:tasks.workunit.client.1.smithi192.stdout:3/631: rename d5/d13/d3a/d85 to d5/d13/d3a/d70/d8c/de1 0 2024-04-12T05:09:07.816 INFO:tasks.workunit.client.0.smithi113.stdout:8/696: symlink d9/d16/ldf 0 2024-04-12T05:09:07.817 INFO:tasks.workunit.client.1.smithi192.stdout:1/639: dread d0/d2/d1e/d3f/f87 [0,4194304] 0 2024-04-12T05:09:07.817 INFO:tasks.workunit.client.1.smithi192.stdout:8/613: getdents d6/d10/d46/d5e/d5f/d37/d7b 0 2024-04-12T05:09:07.817 INFO:tasks.workunit.client.0.smithi113.stdout:3/639: readlink d0/d13/d15/d28/d2a/d34/l5a 0 2024-04-12T05:09:07.817 INFO:tasks.workunit.client.0.smithi113.stdout:8/697: dread - d9/d1b/d96/fd1 zero size 2024-04-12T05:09:07.818 INFO:tasks.workunit.client.0.smithi113.stdout:0/775: mkdir d1/d43/df1 0 2024-04-12T05:09:07.819 INFO:tasks.workunit.client.0.smithi113.stdout:8/698: dread - d9/d1b/d8e/d93/fb7 zero size 2024-04-12T05:09:07.819 INFO:tasks.workunit.client.1.smithi192.stdout:4/716: dwrite d1/d1d/d53/d1c/f6d [0,4194304] 0 2024-04-12T05:09:07.820 INFO:tasks.workunit.client.0.smithi113.stdout:0/776: dread - d1/d6/d34/fc6 zero size 2024-04-12T05:09:07.821 INFO:tasks.workunit.client.1.smithi192.stdout:4/717: fdatasync d1/d1d/d53/d1c/f4e 0 2024-04-12T05:09:07.822 INFO:tasks.workunit.client.0.smithi113.stdout:5/792: rename d0/d10/d27/d1d/d8c/dde to d0/d10/d27/d1d/d10c 0 2024-04-12T05:09:07.822 INFO:tasks.workunit.client.1.smithi192.stdout:1/640: unlink d0/d29/d8d/f8f 0 2024-04-12T05:09:07.823 INFO:tasks.workunit.client.0.smithi113.stdout:5/793: truncate d0/f55 1168042 0 2024-04-12T05:09:07.824 INFO:tasks.workunit.client.1.smithi192.stdout:3/632: creat d5/d13/d3a/d4a/d61/d79/fe2 x:0 0 0 2024-04-12T05:09:07.824 INFO:tasks.workunit.client.0.smithi113.stdout:8/699: creat d9/d16/d65/db1/d59/da3/fe0 x:0 0 0 2024-04-12T05:09:07.825 INFO:tasks.workunit.client.0.smithi113.stdout:5/794: chown d0/d10/f45 4206167 1 2024-04-12T05:09:07.826 INFO:tasks.workunit.client.1.smithi192.stdout:7/591: sync 2024-04-12T05:09:07.826 INFO:tasks.workunit.client.1.smithi192.stdout:0/702: sync 2024-04-12T05:09:07.826 INFO:tasks.workunit.client.1.smithi192.stdout:6/711: sync 2024-04-12T05:09:07.826 INFO:tasks.workunit.client.1.smithi192.stdout:2/641: sync 2024-04-12T05:09:07.826 INFO:tasks.workunit.client.1.smithi192.stdout:5/653: sync 2024-04-12T05:09:07.829 INFO:tasks.workunit.client.0.smithi113.stdout:0/777: creat d1/d6/d16/d2a/d50/d52/d7e/ff2 x:0 0 0 2024-04-12T05:09:07.829 INFO:tasks.workunit.client.0.smithi113.stdout:8/700: mknod d9/d16/d65/db1/d59/ce1 0 2024-04-12T05:09:07.829 INFO:tasks.workunit.client.1.smithi192.stdout:7/592: write d3/d7/d60/fa0 [1002292,96515] 0 2024-04-12T05:09:07.831 INFO:tasks.workunit.client.0.smithi113.stdout:8/701: write d9/f47 [1921698,111338] 0 2024-04-12T05:09:07.831 INFO:tasks.workunit.client.1.smithi192.stdout:4/718: rename d1/d1d/fd2 to d1/d1d/d53/d1c/d95/dc6/ff9 0 2024-04-12T05:09:07.832 INFO:tasks.workunit.client.0.smithi113.stdout:5/795: mknod d0/d10/d27/d1d/d3f/d7e/df8/d108/c10d 0 2024-04-12T05:09:07.835 INFO:tasks.workunit.client.0.smithi113.stdout:8/702: fsync d9/d16/d65/fa9 0 2024-04-12T05:09:07.836 INFO:tasks.workunit.client.0.smithi113.stdout:0/778: stat d1/d6/fb0 0 2024-04-12T05:09:07.836 INFO:tasks.workunit.client.1.smithi192.stdout:9/540: dwrite fe [0,4194304] 0 2024-04-12T05:09:07.837 INFO:tasks.workunit.client.0.smithi113.stdout:0/779: stat d1/d6/d16/dd5/dbb/ld9 0 2024-04-12T05:09:07.837 INFO:tasks.workunit.client.1.smithi192.stdout:1/641: mknod d0/d2/dd/cd2 0 2024-04-12T05:09:07.837 INFO:tasks.workunit.client.1.smithi192.stdout:9/541: chown d11/d25/d36/d3d/d7a/c99 92608 1 2024-04-12T05:09:07.838 INFO:tasks.workunit.client.1.smithi192.stdout:1/642: write d0/d1b/fb1 [785446,19699] 0 2024-04-12T05:09:07.840 INFO:tasks.workunit.client.1.smithi192.stdout:6/712: creat d3/dc/fed x:0 0 0 2024-04-12T05:09:07.840 INFO:tasks.workunit.client.1.smithi192.stdout:0/703: unlink d0/d7b/d9b/d4b/cbd 0 2024-04-12T05:09:07.841 INFO:tasks.workunit.client.0.smithi113.stdout:8/703: stat d9/d1b/d8e/d93/d32/d48/d8d/fa0 0 2024-04-12T05:09:07.841 INFO:tasks.workunit.client.0.smithi113.stdout:3/640: dread d0/d13/f6b [0,4194304] 0 2024-04-12T05:09:07.843 INFO:tasks.workunit.client.1.smithi192.stdout:9/542: read - d11/d25/d36/d6e/d71/f95 zero size 2024-04-12T05:09:07.845 INFO:tasks.workunit.client.0.smithi113.stdout:0/780: dread d1/d6/d16/dd5/fc7 [0,4194304] 0 2024-04-12T05:09:07.847 INFO:tasks.workunit.client.1.smithi192.stdout:7/593: readlink d3/d7/l2b 0 2024-04-12T05:09:07.847 INFO:tasks.workunit.client.1.smithi192.stdout:4/719: rmdir d1/d54/d82/d9c 39 2024-04-12T05:09:07.848 INFO:tasks.workunit.client.0.smithi113.stdout:3/641: creat d0/d13/d15/d28/d2a/d8f/fd6 x:0 0 0 2024-04-12T05:09:07.848 INFO:tasks.workunit.client.1.smithi192.stdout:5/654: mknod d0/d15/d32/d69/dc8/ce3 0 2024-04-12T05:09:07.850 INFO:tasks.workunit.client.1.smithi192.stdout:1/643: stat d0/d2/dd/d3c/d5b/d74/d7d 0 2024-04-12T05:09:07.851 INFO:tasks.workunit.client.0.smithi113.stdout:3/642: symlink d0/d13/d47/d4f/dd2/ld7 0 2024-04-12T05:09:07.852 INFO:tasks.workunit.client.0.smithi113.stdout:5/796: dread d0/d10/f45 [0,4194304] 0 2024-04-12T05:09:07.853 INFO:tasks.workunit.client.1.smithi192.stdout:0/704: unlink d0/d6/d15/d1b/d3a/d4f/fda 0 2024-04-12T05:09:07.854 INFO:tasks.workunit.client.0.smithi113.stdout:0/781: dwrite d1/d6/d16/d2a/d50/db6/d36/fad [0,4194304] 0 2024-04-12T05:09:07.854 INFO:tasks.workunit.client.1.smithi192.stdout:9/543: write d11/d15/d21/d2c/d1f/d94/fa0 [985819,29061] 0 2024-04-12T05:09:07.854 INFO:tasks.workunit.client.1.smithi192.stdout:8/614: dwrite d6/d1f/f5c [0,4194304] 0 2024-04-12T05:09:07.854 INFO:tasks.workunit.client.0.smithi113.stdout:3/643: symlink d0/d80/ld8 0 2024-04-12T05:09:07.858 INFO:tasks.workunit.client.1.smithi192.stdout:4/720: write d1/d1d/d53/d1c/fcb [588090,56365] 0 2024-04-12T05:09:07.858 INFO:tasks.workunit.client.0.smithi113.stdout:5/797: write d0/d10/d27/d1d/d3f/f6a [320400,50580] 0 2024-04-12T05:09:07.859 INFO:tasks.workunit.client.0.smithi113.stdout:0/782: creat d1/d6/d16/d2a/d50/db6/ff3 x:0 0 0 2024-04-12T05:09:07.860 INFO:tasks.workunit.client.0.smithi113.stdout:3/644: mknod d0/d13/d15/cd9 0 2024-04-12T05:09:07.865 INFO:tasks.workunit.client.1.smithi192.stdout:8/615: creat d6/d10/d6c/fbd x:0 0 0 2024-04-12T05:09:07.867 INFO:tasks.workunit.client.0.smithi113.stdout:3/645: creat d0/d13/d47/d4f/dd3/fda x:0 0 0 2024-04-12T05:09:07.868 INFO:tasks.workunit.client.1.smithi192.stdout:2/642: dread d1/d8/d17/d19/f5e [0,4194304] 0 2024-04-12T05:09:07.869 INFO:tasks.workunit.client.1.smithi192.stdout:6/713: dwrite d3/dc/fcc [0,4194304] 0 2024-04-12T05:09:07.871 INFO:tasks.workunit.client.0.smithi113.stdout:3/646: read - d0/d7e/f92 zero size 2024-04-12T05:09:07.872 INFO:tasks.workunit.client.1.smithi192.stdout:0/705: write d0/d6/d15/d1b/d46/f4d [4917822,117285] 0 2024-04-12T05:09:07.872 INFO:tasks.workunit.client.0.smithi113.stdout:0/783: rename d1/d6/d16/dd4 to d1/d43/d59/df4 0 2024-04-12T05:09:07.872 INFO:tasks.workunit.client.1.smithi192.stdout:3/633: dread d5/d13/d3c/d36/f49 [0,4194304] 0 2024-04-12T05:09:07.873 INFO:tasks.workunit.client.0.smithi113.stdout:8/704: dread d9/d1b/f61 [0,4194304] 0 2024-04-12T05:09:07.874 INFO:tasks.workunit.client.0.smithi113.stdout:8/705: stat d9/f55 0 2024-04-12T05:09:07.875 INFO:tasks.workunit.client.0.smithi113.stdout:3/647: creat d0/d13/d15/d28/d2a/d34/d5f/fdb x:0 0 0 2024-04-12T05:09:07.876 INFO:tasks.workunit.client.1.smithi192.stdout:8/616: chown d6/d10/d46/d5e/d5f/f88 108 1 2024-04-12T05:09:07.878 INFO:tasks.workunit.client.0.smithi113.stdout:8/706: rmdir d9 39 2024-04-12T05:09:07.881 INFO:tasks.workunit.client.0.smithi113.stdout:0/784: dread d1/d6/d16/d2a/d50/d52/fbf [0,4194304] 0 2024-04-12T05:09:07.884 INFO:tasks.workunit.client.0.smithi113.stdout:5/798: rename d0/d10/d27/d1d/dce/cd3 to d0/d10/d27/c10e 0 2024-04-12T05:09:07.884 INFO:tasks.workunit.client.1.smithi192.stdout:5/655: dwrite d0/d21/fa9 [0,4194304] 0 2024-04-12T05:09:07.884 INFO:tasks.workunit.client.1.smithi192.stdout:3/634: rmdir d5/d13/d3a/d8b/da2/da7 39 2024-04-12T05:09:07.886 INFO:tasks.workunit.client.0.smithi113.stdout:8/707: truncate d9/d1b/d8e/d93/d32/d5b/fa1 1470696 0 2024-04-12T05:09:07.888 INFO:tasks.workunit.client.1.smithi192.stdout:0/706: chown d0/d6/d15/d55/d2e/d66/f70 0 1 2024-04-12T05:09:07.888 INFO:tasks.workunit.client.1.smithi192.stdout:7/594: dwrite d3/d7/d15/d22/fb6 [0,4194304] 0 2024-04-12T05:09:07.888 INFO:tasks.workunit.client.1.smithi192.stdout:1/644: write d0/d2/d1e/d3f/f87 [190361,32370] 0 2024-04-12T05:09:07.889 INFO:tasks.workunit.client.0.smithi113.stdout:8/708: chown d9/d1b/f4f 127 1 2024-04-12T05:09:07.891 INFO:tasks.workunit.client.1.smithi192.stdout:8/617: chown d6/d10/d46/d5e/d5f/c32 1123 1 2024-04-12T05:09:07.891 INFO:tasks.workunit.client.0.smithi113.stdout:8/709: truncate d9/d16/d65/f97 4196995 0 2024-04-12T05:09:07.892 INFO:tasks.workunit.client.1.smithi192.stdout:4/721: truncate d1/d54/f61 3372253 0 2024-04-12T05:09:07.892 INFO:tasks.workunit.client.0.smithi113.stdout:5/799: mkdir d0/d10/d27/d1d/d38/d44/d66/d10f 0 2024-04-12T05:09:07.894 INFO:tasks.workunit.client.0.smithi113.stdout:3/648: getdents d0/d7e 0 2024-04-12T05:09:07.895 INFO:tasks.workunit.client.0.smithi113.stdout:8/710: chown d9/d1b/c1d 844112 1 2024-04-12T05:09:07.896 INFO:tasks.workunit.client.1.smithi192.stdout:0/707: rmdir d0/d7b/d9b/d35 39 2024-04-12T05:09:07.896 INFO:tasks.workunit.client.1.smithi192.stdout:3/635: fsync d5/d13/d1a/d53/f60 0 2024-04-12T05:09:07.896 INFO:tasks.workunit.client.1.smithi192.stdout:9/544: dwrite d11/d25/d36/d3d/fb3 [0,4194304] 0 2024-04-12T05:09:07.896 INFO:tasks.workunit.client.0.smithi113.stdout:0/785: dwrite d1/d6/d8e/fcb [0,4194304] 0 2024-04-12T05:09:07.900 INFO:tasks.workunit.client.0.smithi113.stdout:3/649: read d0/d13/f42 [728268,40757] 0 2024-04-12T05:09:07.901 INFO:tasks.workunit.client.0.smithi113.stdout:3/650: chown d0/d13/d47/d94/dcf 21036220 1 2024-04-12T05:09:07.902 INFO:tasks.workunit.client.0.smithi113.stdout:5/800: stat d0/d10/d27/d1d/d38/d75/l81 0 2024-04-12T05:09:07.905 INFO:tasks.workunit.client.0.smithi113.stdout:8/711: creat d9/d16/d3c/fe2 x:0 0 0 2024-04-12T05:09:07.907 INFO:tasks.workunit.client.1.smithi192.stdout:7/595: mkdir d3/d1d/d5a/d5f/d7a/da6/dcb 0 2024-04-12T05:09:07.907 INFO:tasks.workunit.client.0.smithi113.stdout:0/786: link d1/d6/d16/d2a/d50/d52/le0 d1/d6/d8e/lf5 0 2024-04-12T05:09:07.908 INFO:tasks.workunit.client.1.smithi192.stdout:2/643: link d1/d8/d3a/de/c38 d1/d8/d17/d19/d60/d9c/cc6 0 2024-04-12T05:09:07.910 INFO:tasks.workunit.client.0.smithi113.stdout:3/651: creat d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/fdc x:0 0 0 2024-04-12T05:09:07.914 INFO:tasks.workunit.client.0.smithi113.stdout:5/801: rename d0/d10/d27/d1d/d41/fa0 to d0/d10/d27/d1d/d38/d44/d4e/f110 0 2024-04-12T05:09:07.916 INFO:tasks.workunit.client.1.smithi192.stdout:8/618: creat d6/d10/d46/d5e/d5f/d37/d4f/fbe x:0 0 0 2024-04-12T05:09:07.916 INFO:tasks.workunit.client.1.smithi192.stdout:5/656: mknod d0/d15/d3e/dca/dda/ce4 0 2024-04-12T05:09:07.917 INFO:tasks.workunit.client.1.smithi192.stdout:3/636: truncate d5/d13/d3a/d4a/f90 1036416 0 2024-04-12T05:09:07.917 INFO:tasks.workunit.client.1.smithi192.stdout:1/645: rename d0/d1b/d28/f4d to d0/d2/dd/d3c/d5b/db6/fd3 0 2024-04-12T05:09:07.920 INFO:tasks.workunit.client.1.smithi192.stdout:7/596: fdatasync d3/d1d/d5a/d5f/d6a/fc7 0 2024-04-12T05:09:07.920 INFO:tasks.workunit.client.0.smithi113.stdout:5/802: mknod d0/d10/d27/d1d/d3f/d73/db9/c111 0 2024-04-12T05:09:07.920 INFO:tasks.workunit.client.0.smithi113.stdout:5/803: mknod d0/d10/d6e/d93/dc9/c112 0 2024-04-12T05:09:07.922 INFO:tasks.workunit.client.1.smithi192.stdout:0/708: mknod d0/d6/d15/d19/cec 0 2024-04-12T05:09:07.922 INFO:tasks.workunit.client.1.smithi192.stdout:6/714: dwrite d3/f94 [0,4194304] 0 2024-04-12T05:09:07.922 INFO:tasks.workunit.client.1.smithi192.stdout:9/545: dwrite d11/d25/d36/d6e/d71/f78 [0,4194304] 0 2024-04-12T05:09:07.923 INFO:tasks.workunit.client.0.smithi113.stdout:3/652: write d0/d13/f70 [1139634,40056] 0 2024-04-12T05:09:07.924 INFO:tasks.workunit.client.1.smithi192.stdout:1/646: dread - d0/d2/dd/d3c/f7f zero size 2024-04-12T05:09:07.925 INFO:tasks.workunit.client.0.smithi113.stdout:3/653: fsync d0/d13/d15/d28/d2a/d34/d5f/f96 0 2024-04-12T05:09:07.925 INFO:tasks.workunit.client.0.smithi113.stdout:0/787: truncate d1/d6/d16/d8f/fce 21476 0 2024-04-12T05:09:07.926 INFO:tasks.workunit.client.0.smithi113.stdout:3/654: chown d0/d13/d15/c46 1858288649 1 2024-04-12T05:09:07.928 INFO:tasks.workunit.client.1.smithi192.stdout:8/619: write d6/d10/d46/d5e/d5f/d31/d80/db1/fb6 [776701,73462] 0 2024-04-12T05:09:07.929 INFO:tasks.workunit.client.1.smithi192.stdout:1/647: fsync d0/d2/f50 0 2024-04-12T05:09:07.931 INFO:tasks.workunit.client.1.smithi192.stdout:2/644: chown d1/d8/d17/d19/d60/d59/c81 63465 1 2024-04-12T05:09:07.932 INFO:tasks.workunit.client.0.smithi113.stdout:8/712: dread d9/d16/d65/db1/f2d [0,4194304] 0 2024-04-12T05:09:07.933 INFO:tasks.workunit.client.1.smithi192.stdout:8/620: readlink d6/d10/d46/d7c/l8b 0 2024-04-12T05:09:07.935 INFO:tasks.workunit.client.0.smithi113.stdout:8/713: fdatasync d9/d1b/d96/fba 0 2024-04-12T05:09:07.936 INFO:tasks.workunit.client.0.smithi113.stdout:8/714: mknod d9/d1b/ce3 0 2024-04-12T05:09:07.938 INFO:tasks.workunit.client.1.smithi192.stdout:4/722: dwrite d1/d1d/d53/f89 [0,4194304] 0 2024-04-12T05:09:07.940 INFO:tasks.workunit.client.1.smithi192.stdout:2/645: dread d1/d8/d17/d19/d60/fa6 [0,4194304] 0 2024-04-12T05:09:07.943 INFO:tasks.workunit.client.0.smithi113.stdout:8/715: dread d9/f9d [0,4194304] 0 2024-04-12T05:09:07.944 INFO:tasks.workunit.client.0.smithi113.stdout:8/716: stat d9/d16/d65/c9c 0 2024-04-12T05:09:07.948 INFO:tasks.workunit.client.0.smithi113.stdout:8/717: symlink d9/d1b/d8e/d93/le4 0 2024-04-12T05:09:07.951 INFO:tasks.workunit.client.0.smithi113.stdout:8/718: fsync d9/d16/d65/db1/d59/d68/f87 0 2024-04-12T05:09:07.953 INFO:tasks.workunit.client.0.smithi113.stdout:8/719: write d9/d1b/d8e/d93/d32/f3f [3088893,114505] 0 2024-04-12T05:09:07.957 INFO:tasks.workunit.client.1.smithi192.stdout:6/715: symlink d3/d5/d9c/lee 0 2024-04-12T05:09:07.957 INFO:tasks.workunit.client.0.smithi113.stdout:0/788: fdatasync d1/fa6 0 2024-04-12T05:09:07.959 INFO:tasks.workunit.client.1.smithi192.stdout:3/637: symlink d5/d13/d3a/d8b/da2/da7/le3 0 2024-04-12T05:09:07.961 INFO:tasks.workunit.client.1.smithi192.stdout:3/638: chown d5/d13/f2a 536 1 2024-04-12T05:09:07.962 INFO:tasks.workunit.client.1.smithi192.stdout:5/657: dwrite d0/f3d [4194304,4194304] 0 2024-04-12T05:09:07.964 INFO:tasks.workunit.client.0.smithi113.stdout:8/720: dwrite d9/d1b/d96/fba [4194304,4194304] 0 2024-04-12T05:09:07.964 INFO:tasks.workunit.client.0.smithi113.stdout:0/789: dwrite d1/d6/d8e/fcb [0,4194304] 0 2024-04-12T05:09:07.968 INFO:tasks.workunit.client.0.smithi113.stdout:8/721: fdatasync d9/d1b/d8e/d93/d32/d5b/f83 0 2024-04-12T05:09:07.976 INFO:tasks.workunit.client.0.smithi113.stdout:8/722: rmdir d9/d1b/d8e/d93/d32/d48/dc1/dd0 0 2024-04-12T05:09:07.978 INFO:tasks.workunit.client.0.smithi113.stdout:8/723: truncate d9/d1b/fcd 1532037 0 2024-04-12T05:09:07.980 INFO:tasks.workunit.client.0.smithi113.stdout:8/724: symlink d9/d1b/d96/le5 0 2024-04-12T05:09:07.984 INFO:tasks.workunit.client.0.smithi113.stdout:8/725: truncate d9/d1b/d8e/d93/d32/f60 2024340 0 2024-04-12T05:09:07.984 INFO:tasks.workunit.client.0.smithi113.stdout:8/726: chown d9/d16/d7f/l2b 3282 1 2024-04-12T05:09:07.991 INFO:tasks.workunit.client.0.smithi113.stdout:8/727: dwrite d9/f55 [4194304,4194304] 0 2024-04-12T05:09:07.997 INFO:tasks.workunit.client.0.smithi113.stdout:8/728: dread d9/d16/d65/db1/d59/d78/f9f [0,4194304] 0 2024-04-12T05:09:08.004 INFO:tasks.workunit.client.0.smithi113.stdout:8/729: rename d9/d1b/d8e/d93/d32/d48/f85 to d9/d1b/d8e/d93/fe6 0 2024-04-12T05:09:08.006 INFO:tasks.workunit.client.0.smithi113.stdout:8/730: creat d9/d1b/d96/fe7 x:0 0 0 2024-04-12T05:09:08.016 INFO:tasks.workunit.client.0.smithi113.stdout:8/731: dread d9/d16/d65/f97 [0,4194304] 0 2024-04-12T05:09:08.022 INFO:tasks.workunit.client.0.smithi113.stdout:9/709: sync 2024-04-12T05:09:08.027 INFO:tasks.workunit.client.0.smithi113.stdout:9/710: write d1/df/d29/fbc [711589,53027] 0 2024-04-12T05:09:08.027 INFO:tasks.workunit.client.0.smithi113.stdout:8/732: dwrite d9/d1b/d8e/d93/faf [0,4194304] 0 2024-04-12T05:09:08.029 INFO:tasks.workunit.client.0.smithi113.stdout:9/711: chown d1/d27/d7a/dd2/fdc 955 1 2024-04-12T05:09:08.029 INFO:tasks.workunit.client.0.smithi113.stdout:8/733: chown d9/d16/d65/db1/d59/d68/caa 874323 1 2024-04-12T05:09:08.036 INFO:tasks.workunit.client.0.smithi113.stdout:8/734: write d9/d1b/d8e/d93/f28 [1086885,82866] 0 2024-04-12T05:09:08.037 INFO:tasks.workunit.client.0.smithi113.stdout:9/712: dwrite d1/d4/d87/fbb [0,4194304] 0 2024-04-12T05:09:08.042 INFO:tasks.workunit.client.0.smithi113.stdout:9/713: write d1/d4/d38/f62 [1074827,42649] 0 2024-04-12T05:09:08.042 INFO:tasks.workunit.client.1.smithi192.stdout:7/597: mknod d3/d7/d14/d1f/d76/ccc 0 2024-04-12T05:09:08.043 INFO:tasks.workunit.client.0.smithi113.stdout:9/714: chown d1/d27 2945106 1 2024-04-12T05:09:08.045 INFO:tasks.workunit.client.0.smithi113.stdout:8/735: write d9/d16/d7f/d3a/f5e [3856635,65850] 0 2024-04-12T05:09:08.047 INFO:tasks.workunit.client.1.smithi192.stdout:0/709: creat d0/d7b/d9b/d4b/dbc/dce/d95/fed x:0 0 0 2024-04-12T05:09:08.048 INFO:tasks.workunit.client.0.smithi113.stdout:9/715: mkdir d1/d4/d5/d22/d9e/df1 0 2024-04-12T05:09:08.050 INFO:tasks.workunit.client.0.smithi113.stdout:8/736: rename d9/d1b/f56 to d9/d1b/fe8 0 2024-04-12T05:09:08.051 INFO:tasks.workunit.client.0.smithi113.stdout:9/716: chown d1/df/d4b/d63/c66 47 1 2024-04-12T05:09:08.052 INFO:tasks.workunit.client.0.smithi113.stdout:8/737: creat d9/d1b/d96/fe9 x:0 0 0 2024-04-12T05:09:08.052 INFO:tasks.workunit.client.1.smithi192.stdout:8/621: rmdir d6/d10/d46/d5e/d5f/d37/d4f/d4c 39 2024-04-12T05:09:08.055 INFO:tasks.workunit.client.0.smithi113.stdout:9/717: rename d1/d4/f8f to d1/df/d4b/d63/dad/ff2 0 2024-04-12T05:09:08.056 INFO:tasks.workunit.client.0.smithi113.stdout:0/790: write d1/d6/d16/d2a/d50/f85 [703858,2151] 0 2024-04-12T05:09:08.057 INFO:tasks.workunit.client.1.smithi192.stdout:9/546: mknod d11/d15/d21/d2c/cbf 0 2024-04-12T05:09:08.059 INFO:tasks.workunit.client.1.smithi192.stdout:9/547: readlink d11/d15/d21/d2c/d1f/d50/d73/lb8 0 2024-04-12T05:09:08.061 INFO:tasks.workunit.client.1.smithi192.stdout:3/639: unlink d5/d13/d3c/d84/cb8 0 2024-04-12T05:09:08.064 INFO:tasks.workunit.client.0.smithi113.stdout:0/791: creat d1/d6/d16/ff6 x:0 0 0 2024-04-12T05:09:08.065 INFO:tasks.workunit.client.1.smithi192.stdout:1/648: mknod d0/d2/dd/d21/d62/dbc/cd4 0 2024-04-12T05:09:08.066 INFO:tasks.workunit.client.0.smithi113.stdout:0/792: write d1/f2 [4748266,69284] 0 2024-04-12T05:09:08.068 INFO:tasks.workunit.client.0.smithi113.stdout:8/738: dwrite d9/d1b/d8e/d93/d32/f60 [0,4194304] 0 2024-04-12T05:09:08.069 INFO:tasks.workunit.client.1.smithi192.stdout:9/548: mknod d11/d25/d36/d3d/cc0 0 2024-04-12T05:09:08.071 INFO:tasks.workunit.client.1.smithi192.stdout:7/598: getdents d3/d1d/d5a/d5f/d7a/da6/dcb 0 2024-04-12T05:09:08.075 INFO:tasks.workunit.client.1.smithi192.stdout:0/710: link d0/d7b/d7c/fdb d0/d7b/d9b/d4b/fee 0 2024-04-12T05:09:08.076 INFO:tasks.workunit.client.1.smithi192.stdout:0/711: fdatasync d0/d7b/d7c/dcc/f93 0 2024-04-12T05:09:08.076 INFO:tasks.workunit.client.0.smithi113.stdout:0/793: rmdir d1/d6/d16/d2a/d50 39 2024-04-12T05:09:08.077 INFO:tasks.workunit.client.0.smithi113.stdout:8/739: dwrite d9/d1b/d8e/d93/d32/f60 [4194304,4194304] 0 2024-04-12T05:09:08.077 INFO:tasks.workunit.client.1.smithi192.stdout:1/649: unlink d0/d2/dd/d21/d33/l85 0 2024-04-12T05:09:08.077 INFO:tasks.workunit.client.1.smithi192.stdout:2/646: dread d1/d8/d3a/de/f30 [0,4194304] 0 2024-04-12T05:09:08.078 INFO:tasks.workunit.client.1.smithi192.stdout:0/712: chown d0/d6/d15/d19/f71 2916 1 2024-04-12T05:09:08.079 INFO:tasks.workunit.client.0.smithi113.stdout:8/740: mknod d9/d16/d65/cea 0 2024-04-12T05:09:08.080 INFO:tasks.workunit.client.1.smithi192.stdout:6/716: rename d3/d8/d10/d4c/fb0 to d3/d23/fef 0 2024-04-12T05:09:08.083 INFO:tasks.workunit.client.1.smithi192.stdout:5/658: link d0/f11 d0/d15/d3e/dca/dda/fe5 0 2024-04-12T05:09:08.084 INFO:tasks.workunit.client.1.smithi192.stdout:7/599: mknod d3/d1d/d5a/d5f/d7a/da6/ccd 0 2024-04-12T05:09:08.084 INFO:tasks.workunit.client.1.smithi192.stdout:5/659: write d0/d15/d24/d29/f2a [1846301,112218] 0 2024-04-12T05:09:08.084 INFO:tasks.workunit.client.0.smithi113.stdout:0/794: mknod d1/d43/d59/db2/cf7 0 2024-04-12T05:09:08.089 INFO:tasks.workunit.client.0.smithi113.stdout:8/741: dwrite d9/d16/d65/db1/d59/d68/fa8 [0,4194304] 0 2024-04-12T05:09:08.089 INFO:tasks.workunit.client.1.smithi192.stdout:2/647: unlink d1/d8/d3a/de/l42 0 2024-04-12T05:09:08.092 INFO:tasks.workunit.client.0.smithi113.stdout:0/795: creat d1/d6/d34/dc3/ff8 x:0 0 0 2024-04-12T05:09:08.093 INFO:tasks.workunit.client.0.smithi113.stdout:8/742: mknod d9/d1b/d8e/ceb 0 2024-04-12T05:09:08.094 INFO:tasks.workunit.client.0.smithi113.stdout:8/743: chown d9/d1b/d8e/d93/f84 44 1 2024-04-12T05:09:08.095 INFO:tasks.workunit.client.1.smithi192.stdout:4/723: rename d1/d1d/d53/d1c/d95/cac to d1/d54/d82/da4/da8/cfa 0 2024-04-12T05:09:08.099 INFO:tasks.workunit.client.1.smithi192.stdout:0/713: dread d0/d6/d15/d55/d2e/d66/f86 [0,4194304] 0 2024-04-12T05:09:08.099 INFO:tasks.workunit.client.1.smithi192.stdout:1/650: creat d0/d29/dbf/fd5 x:0 0 0 2024-04-12T05:09:08.100 INFO:tasks.workunit.client.0.smithi113.stdout:0/796: dread d1/d6/d16/d2a/d50/f61 [0,4194304] 0 2024-04-12T05:09:08.100 INFO:tasks.workunit.client.1.smithi192.stdout:8/622: getdents d6/d10/d46/d5e/d5f/d31 0 2024-04-12T05:09:08.101 INFO:tasks.workunit.client.1.smithi192.stdout:0/714: chown d0/d7b/ca8 1395 1 2024-04-12T05:09:08.101 INFO:tasks.workunit.client.1.smithi192.stdout:2/648: fdatasync d1/f4 0 2024-04-12T05:09:08.102 INFO:tasks.workunit.client.1.smithi192.stdout:9/549: creat d11/d15/fc1 x:0 0 0 2024-04-12T05:09:08.102 INFO:tasks.workunit.client.0.smithi113.stdout:0/797: stat d1/d6/d16/dd5/f5f 0 2024-04-12T05:09:08.104 INFO:tasks.workunit.client.0.smithi113.stdout:8/744: write d9/d1b/d8e/d93/f71 [1024478,111435] 0 2024-04-12T05:09:08.108 INFO:tasks.workunit.client.0.smithi113.stdout:8/745: mkdir d9/d16/d65/db1/dec 0 2024-04-12T05:09:08.109 INFO:tasks.workunit.client.1.smithi192.stdout:3/640: rename d5/d13/d3a/d4e/d57/f51 to d5/d13/d3a/d70/d8c/d8d/fe4 0 2024-04-12T05:09:08.109 INFO:tasks.workunit.client.1.smithi192.stdout:9/550: fsync d11/d25/f32 0 2024-04-12T05:09:08.109 INFO:tasks.workunit.client.1.smithi192.stdout:2/649: dread - d1/d8/d17/d19/d60/f93 zero size 2024-04-12T05:09:08.109 INFO:tasks.workunit.client.1.smithi192.stdout:4/724: mkdir d1/d1d/d53/d1c/dd1/def/ddd/dfb 0 2024-04-12T05:09:08.109 INFO:tasks.workunit.client.1.smithi192.stdout:4/725: write d1/d49/f5b [3889133,118936] 0 2024-04-12T05:09:08.110 INFO:tasks.workunit.client.0.smithi113.stdout:0/798: unlink d1/d6/d8e/fba 0 2024-04-12T05:09:08.113 INFO:tasks.workunit.client.0.smithi113.stdout:8/746: write d9/d16/d65/db1/f2d [2020262,59901] 0 2024-04-12T05:09:08.114 INFO:tasks.workunit.client.0.smithi113.stdout:0/799: stat d1/d6/d16/d2a/d50/d52/cb4 0 2024-04-12T05:09:08.115 INFO:tasks.workunit.client.0.smithi113.stdout:0/800: write d1/f2f [2704010,37711] 0 2024-04-12T05:09:08.116 INFO:tasks.workunit.client.1.smithi192.stdout:4/726: dwrite d1/d54/f9a [0,4194304] 0 2024-04-12T05:09:08.118 INFO:tasks.workunit.client.1.smithi192.stdout:1/651: creat d0/d2/dd/d3c/d5b/d74/d7d/fd6 x:0 0 0 2024-04-12T05:09:08.118 INFO:tasks.workunit.client.0.smithi113.stdout:8/747: dread d9/d16/d65/f97 [0,4194304] 0 2024-04-12T05:09:08.119 INFO:tasks.workunit.client.0.smithi113.stdout:8/748: chown d9/d1b/d8e/d93/d32/d48/d8d 1 1 2024-04-12T05:09:08.120 INFO:tasks.workunit.client.0.smithi113.stdout:8/749: readlink d9/d16/d7f/l2b 0 2024-04-12T05:09:08.120 INFO:tasks.workunit.client.1.smithi192.stdout:1/652: chown d0/d2/dd/f3b 42172882 1 2024-04-12T05:09:08.120 INFO:tasks.workunit.client.0.smithi113.stdout:8/750: fsync d9/d16/d3c/f74 0 2024-04-12T05:09:08.123 INFO:tasks.workunit.client.1.smithi192.stdout:8/623: dwrite d6/d1f/d6a/f52 [0,4194304] 0 2024-04-12T05:09:08.123 INFO:tasks.workunit.client.1.smithi192.stdout:1/653: fdatasync d0/d2/dd/d3c/d5b/d74/d7d/fce 0 2024-04-12T05:09:08.124 INFO:tasks.workunit.client.0.smithi113.stdout:8/751: unlink d9/d1b/d8e/d93/d32/d48/d8d/l54 0 2024-04-12T05:09:08.125 INFO:tasks.workunit.client.1.smithi192.stdout:5/660: rename d0/d15/d24/l2e to d0/d15/d32/d69/d7d/le6 0 2024-04-12T05:09:08.128 INFO:tasks.workunit.client.0.smithi113.stdout:8/752: mknod d9/d1b/d8e/d93/d32/d48/d8d/dae/ced 0 2024-04-12T05:09:08.128 INFO:tasks.workunit.client.1.smithi192.stdout:3/641: fdatasync d5/d13/d3c/d36/f49 0 2024-04-12T05:09:08.128 INFO:tasks.workunit.client.1.smithi192.stdout:0/715: dread d0/d6/d15/d55/d2e/f63 [0,4194304] 0 2024-04-12T05:09:08.128 INFO:tasks.workunit.client.1.smithi192.stdout:8/624: fsync d6/d10/d46/d5e/d5f/d31/d80/db1/fb6 0 2024-04-12T05:09:08.131 INFO:tasks.workunit.client.0.smithi113.stdout:8/753: mknod d9/d16/d3c/cee 0 2024-04-12T05:09:08.131 INFO:tasks.workunit.client.1.smithi192.stdout:0/716: readlink d0/d7b/d9b/d4b/dbc/dce/d95/l9c 0 2024-04-12T05:09:08.133 INFO:tasks.workunit.client.1.smithi192.stdout:8/625: chown d6/d10/d46/d5e/d5f/d31/d61/f74 4934 1 2024-04-12T05:09:08.134 INFO:tasks.workunit.client.0.smithi113.stdout:8/754: symlink d9/d1b/d8e/d93/lef 0 2024-04-12T05:09:08.135 INFO:tasks.workunit.client.0.smithi113.stdout:8/755: chown d9/d1b/d8e/d93/f71 9535883 1 2024-04-12T05:09:08.135 INFO:tasks.workunit.client.1.smithi192.stdout:1/654: dread d0/d1b/f35 [0,4194304] 0 2024-04-12T05:09:08.136 INFO:tasks.workunit.client.1.smithi192.stdout:3/642: read d5/d13/d3a/d70/f71 [989806,36609] 0 2024-04-12T05:09:08.138 INFO:tasks.workunit.client.1.smithi192.stdout:3/643: read - d5/d13/d3a/d70/d8c/d8d/dbc/fce zero size 2024-04-12T05:09:08.139 INFO:tasks.workunit.client.1.smithi192.stdout:8/626: dwrite d6/d10/d46/d5e/d5f/d38/d3b/f81 [0,4194304] 0 2024-04-12T05:09:08.143 INFO:tasks.workunit.client.0.smithi113.stdout:8/756: write d9/d1b/d8e/d93/d32/d48/d8d/fa0 [667457,77800] 0 2024-04-12T05:09:08.143 INFO:tasks.workunit.client.1.smithi192.stdout:8/627: dread d6/d10/d46/d5e/d5f/d31/f36 [0,4194304] 0 2024-04-12T05:09:08.147 INFO:tasks.workunit.client.0.smithi113.stdout:8/757: creat d9/d1b/d8e/d93/d32/d48/ff0 x:0 0 0 2024-04-12T05:09:08.149 INFO:tasks.workunit.client.0.smithi113.stdout:8/758: creat d9/dd4/ff1 x:0 0 0 2024-04-12T05:09:08.150 INFO:tasks.workunit.client.0.smithi113.stdout:8/759: mkdir d9/d16/d65/db1/df2 0 2024-04-12T05:09:08.160 INFO:tasks.workunit.client.0.smithi113.stdout:8/760: truncate d9/d1b/d8e/d93/f84 1935880 0 2024-04-12T05:09:08.184 INFO:tasks.workunit.client.0.smithi113.stdout:7/704: sync 2024-04-12T05:09:08.184 INFO:tasks.workunit.client.0.smithi113.stdout:2/654: sync 2024-04-12T05:09:08.186 INFO:tasks.workunit.client.0.smithi113.stdout:7/705: mkdir d0/dc/d6/d9b/d9f/df4 0 2024-04-12T05:09:08.187 INFO:tasks.workunit.client.0.smithi113.stdout:2/655: symlink d8/d9/de/d12/d44/lc7 0 2024-04-12T05:09:08.188 INFO:tasks.workunit.client.0.smithi113.stdout:7/706: chown d0/dc/d6/d24/c2a 41383 1 2024-04-12T05:09:08.190 INFO:tasks.workunit.client.0.smithi113.stdout:2/656: creat d8/d9/de/d12/d3c/d8a/fc8 x:0 0 0 2024-04-12T05:09:08.191 INFO:tasks.workunit.client.0.smithi113.stdout:7/707: creat d0/d6c/ff5 x:0 0 0 2024-04-12T05:09:08.191 INFO:tasks.workunit.client.0.smithi113.stdout:2/657: readlink d8/d9/d27/d88/d91/lc5 0 2024-04-12T05:09:08.192 INFO:tasks.workunit.client.0.smithi113.stdout:7/708: creat d0/dc/ff6 x:0 0 0 2024-04-12T05:09:08.193 INFO:tasks.workunit.client.0.smithi113.stdout:7/709: fsync d0/dd/f66 0 2024-04-12T05:09:08.195 INFO:tasks.workunit.client.0.smithi113.stdout:7/710: chown d0/dd/d11/de6/ca8 18 1 2024-04-12T05:09:08.196 INFO:tasks.workunit.client.0.smithi113.stdout:7/711: dread - d0/dc/d6/fd0 zero size 2024-04-12T05:09:08.197 INFO:tasks.workunit.client.0.smithi113.stdout:2/658: dwrite d8/d9/de/d15/f40 [0,4194304] 0 2024-04-12T05:09:08.198 INFO:tasks.workunit.client.0.smithi113.stdout:7/712: chown d0/dd/d11/l75 2204 1 2024-04-12T05:09:08.205 INFO:tasks.workunit.client.0.smithi113.stdout:7/713: link d0/dc/d6/f28 d0/d52/dc5/ff7 0 2024-04-12T05:09:08.211 INFO:tasks.workunit.client.0.smithi113.stdout:2/659: write d8/d9/d27/f65 [1859071,42423] 0 2024-04-12T05:09:08.213 INFO:tasks.workunit.client.0.smithi113.stdout:2/660: creat d8/d16/dbd/fc9 x:0 0 0 2024-04-12T05:09:08.214 INFO:tasks.workunit.client.0.smithi113.stdout:2/661: fsync d8/d9/de/d12/f9c 0 2024-04-12T05:09:08.218 INFO:tasks.workunit.client.0.smithi113.stdout:2/662: rename d8/d9/de/c64 to d8/d9/de/d12/dbf/cca 0 2024-04-12T05:09:08.220 INFO:tasks.workunit.client.1.smithi192.stdout:2/650: creat d1/d8/d17/d19/d60/d9c/fc7 x:0 0 0 2024-04-12T05:09:08.224 INFO:tasks.workunit.client.1.smithi192.stdout:6/717: write d3/d8/d48/f84 [358291,62472] 0 2024-04-12T05:09:08.224 INFO:tasks.workunit.client.0.smithi113.stdout:2/663: dwrite d8/d9/de/d15/d24/d58/f75 [0,4194304] 0 2024-04-12T05:09:08.224 INFO:tasks.workunit.client.1.smithi192.stdout:6/718: dread - d3/d5/f81 zero size 2024-04-12T05:09:08.225 INFO:tasks.workunit.client.1.smithi192.stdout:9/551: fdatasync d11/d15/f5c 0 2024-04-12T05:09:08.225 INFO:tasks.workunit.client.0.smithi113.stdout:2/664: dread - d8/d9/de/d43/fa8 zero size 2024-04-12T05:09:08.226 INFO:tasks.workunit.client.1.smithi192.stdout:9/552: fsync d11/d25/d36/d3d/fb3 0 2024-04-12T05:09:08.243 INFO:tasks.workunit.client.0.smithi113.stdout:2/665: dread d8/d16/f6e [0,4194304] 0 2024-04-12T05:09:08.246 INFO:tasks.workunit.client.0.smithi113.stdout:2/666: creat d8/d9/de/d12/d3c/fcb x:0 0 0 2024-04-12T05:09:08.253 INFO:tasks.workunit.client.0.smithi113.stdout:2/667: dwrite d8/d9/d27/f72 [0,4194304] 0 2024-04-12T05:09:08.260 INFO:tasks.workunit.client.0.smithi113.stdout:2/668: dwrite d8/d9/d27/d88/f8b [0,4194304] 0 2024-04-12T05:09:08.267 INFO:tasks.workunit.client.0.smithi113.stdout:2/669: dread d8/d9/de/d15/f18 [0,4194304] 0 2024-04-12T05:09:08.268 INFO:tasks.workunit.client.0.smithi113.stdout:2/670: fdatasync d8/d9/de/d12/d3c/fbc 0 2024-04-12T05:09:08.269 INFO:tasks.workunit.client.0.smithi113.stdout:2/671: stat d8/d9/d1a 0 2024-04-12T05:09:08.275 INFO:tasks.workunit.client.0.smithi113.stdout:2/672: dwrite d8/d9/d27/fb7 [0,4194304] 0 2024-04-12T05:09:08.280 INFO:tasks.workunit.client.0.smithi113.stdout:2/673: dread d8/d9/d1a/f26 [0,4194304] 0 2024-04-12T05:09:08.281 INFO:tasks.workunit.client.0.smithi113.stdout:2/674: write d8/d9/f69 [4539291,54439] 0 2024-04-12T05:09:08.282 INFO:tasks.workunit.client.0.smithi113.stdout:2/675: chown d8/d9/d27/d88/d91 1841 1 2024-04-12T05:09:08.285 INFO:tasks.workunit.client.0.smithi113.stdout:2/676: creat d8/d9/de/d12/d2b/fcc x:0 0 0 2024-04-12T05:09:08.286 INFO:tasks.workunit.client.0.smithi113.stdout:2/677: write d8/d9/de/d12/d3c/fcb [234191,51688] 0 2024-04-12T05:09:08.287 INFO:tasks.workunit.client.0.smithi113.stdout:2/678: chown d8/d9/de/d12/d44/c7d 97912 1 2024-04-12T05:09:08.288 INFO:tasks.workunit.client.0.smithi113.stdout:2/679: stat d8/d9/de/d12/d3c/d8a/fc8 0 2024-04-12T05:09:08.293 INFO:tasks.workunit.client.0.smithi113.stdout:2/680: dwrite d8/d9/de/f67 [0,4194304] 0 2024-04-12T05:09:08.294 INFO:tasks.workunit.client.0.smithi113.stdout:2/681: write d8/d9/de/d43/fa8 [490244,38120] 0 2024-04-12T05:09:08.296 INFO:tasks.workunit.client.1.smithi192.stdout:5/661: mkdir d0/d15/d32/d69/d7d/de7 0 2024-04-12T05:09:08.297 INFO:tasks.workunit.client.1.smithi192.stdout:1/655: fsync d0/d2/dd/f5d 0 2024-04-12T05:09:08.297 INFO:tasks.workunit.client.1.smithi192.stdout:6/719: dwrite d3/d8/d22/f2e [0,4194304] 0 2024-04-12T05:09:08.299 INFO:tasks.workunit.client.1.smithi192.stdout:2/651: rename d1/d2d/d73/f74 to d1/d8/d17/db3/fc8 0 2024-04-12T05:09:08.299 INFO:tasks.workunit.client.1.smithi192.stdout:4/727: mkdir d1/d1d/d53/d1c/d84/dbb/df7/dfc 0 2024-04-12T05:09:08.304 INFO:tasks.workunit.client.1.smithi192.stdout:9/553: unlink d11/d15/f42 0 2024-04-12T05:09:08.304 INFO:tasks.workunit.client.0.smithi113.stdout:2/682: truncate d8/f1e 1408461 0 2024-04-12T05:09:08.310 INFO:tasks.workunit.client.0.smithi113.stdout:2/683: rename l4 to d8/d9/de/d43/db2/lcd 0 2024-04-12T05:09:08.312 INFO:tasks.workunit.client.0.smithi113.stdout:2/684: unlink d8/d16/cc2 0 2024-04-12T05:09:08.312 INFO:tasks.workunit.client.0.smithi113.stdout:2/685: chown d8/d9/de/d15/d38/f51 11 1 2024-04-12T05:09:08.314 INFO:tasks.workunit.client.0.smithi113.stdout:2/686: symlink d8/d9/d27/lce 0 2024-04-12T05:09:08.315 INFO:tasks.workunit.client.0.smithi113.stdout:2/687: chown d8/d9/d27/d88/d91/lc5 58 1 2024-04-12T05:09:08.318 INFO:tasks.workunit.client.0.smithi113.stdout:2/688: dread d8/d9/d27/d88/f8b [0,4194304] 0 2024-04-12T05:09:08.327 INFO:tasks.workunit.client.0.smithi113.stdout:2/689: dwrite d8/d9/de/d12/d2b/daa/fc6 [0,4194304] 0 2024-04-12T05:09:08.329 INFO:tasks.workunit.client.0.smithi113.stdout:2/690: symlink d8/d9/de/d12/d2b/daa/db3/lcf 0 2024-04-12T05:09:08.330 INFO:tasks.workunit.client.0.smithi113.stdout:2/691: readlink d8/d9/de/d12/d2b/daa/db3/l4a 0 2024-04-12T05:09:08.331 INFO:tasks.workunit.client.0.smithi113.stdout:2/692: fsync d8/d9/de/d15/d24/d58/fae 0 2024-04-12T05:09:08.341 INFO:tasks.workunit.client.0.smithi113.stdout:2/693: dwrite d8/d9/de/d12/f5d [0,4194304] 0 2024-04-12T05:09:08.363 INFO:tasks.workunit.client.1.smithi192.stdout:7/600: truncate d3/d1d/d48/d7b/f95 1209217 0 2024-04-12T05:09:08.364 INFO:tasks.workunit.client.1.smithi192.stdout:4/728: dread - d1/d1d/d53/d1c/de1/fec zero size 2024-04-12T05:09:08.366 INFO:tasks.workunit.client.0.smithi113.stdout:2/694: truncate d8/fd 1347891 0 2024-04-12T05:09:08.366 INFO:tasks.workunit.client.1.smithi192.stdout:3/644: mkdir d5/d13/d3a/d8b/da2/de5 0 2024-04-12T05:09:08.367 INFO:tasks.workunit.client.1.smithi192.stdout:1/656: mkdir d0/d2/dd/d3c/d5b/da7/dd7 0 2024-04-12T05:09:08.369 INFO:tasks.workunit.client.1.smithi192.stdout:8/628: dwrite d6/d21/f60 [0,4194304] 0 2024-04-12T05:09:08.370 INFO:tasks.workunit.client.1.smithi192.stdout:6/720: write d3/d8/d10/d4c/fdb [519305,108729] 0 2024-04-12T05:09:08.374 INFO:tasks.workunit.client.1.smithi192.stdout:7/601: mkdir d3/d1d/d4a/d5e/d6d/da3/dce 0 2024-04-12T05:09:08.375 INFO:tasks.workunit.client.1.smithi192.stdout:5/662: dwrite d0/d15/d3e/dca/db9/fba [0,4194304] 0 2024-04-12T05:09:08.375 INFO:tasks.workunit.client.1.smithi192.stdout:4/729: mkdir d1/d1d/d53/d1c/dd1/def/ddd/dfd 0 2024-04-12T05:09:08.377 INFO:tasks.workunit.client.1.smithi192.stdout:8/629: creat d6/d10/d46/d5e/d6e/fbf x:0 0 0 2024-04-12T05:09:08.378 INFO:tasks.workunit.client.1.smithi192.stdout:5/663: chown d0/d15/f23 0 1 2024-04-12T05:09:08.379 INFO:tasks.workunit.client.0.smithi113.stdout:4/716: sync 2024-04-12T05:09:08.380 INFO:tasks.workunit.client.1.smithi192.stdout:5/664: dread - d0/d15/d64/fb6 zero size 2024-04-12T05:09:08.381 INFO:tasks.workunit.client.1.smithi192.stdout:7/602: unlink d3/d1d/l9b 0 2024-04-12T05:09:08.382 INFO:tasks.workunit.client.1.smithi192.stdout:1/657: mkdir d0/d2/dd/d21/d33/dd8 0 2024-04-12T05:09:08.383 INFO:tasks.workunit.client.0.smithi113.stdout:4/717: creat da/d3c/da5/fec x:0 0 0 2024-04-12T05:09:08.383 INFO:tasks.workunit.client.1.smithi192.stdout:4/730: mkdir d1/d54/d82/d71/d9e/dc4/dfe 0 2024-04-12T05:09:08.385 INFO:tasks.workunit.client.1.smithi192.stdout:7/603: dread - d3/d1d/d5a/d5f/d6a/fb7 zero size 2024-04-12T05:09:08.385 INFO:tasks.workunit.client.1.smithi192.stdout:1/658: write d0/d2/d1e/f36 [1393597,88658] 0 2024-04-12T05:09:08.386 INFO:tasks.workunit.client.0.smithi113.stdout:4/718: symlink da/led 0 2024-04-12T05:09:08.386 INFO:tasks.workunit.client.1.smithi192.stdout:2/652: getdents d1/d8/d17/d19/d71/da0 0 2024-04-12T05:09:08.389 INFO:tasks.workunit.client.0.smithi113.stdout:4/719: creat da/d24/d7b/fee x:0 0 0 2024-04-12T05:09:08.390 INFO:tasks.workunit.client.1.smithi192.stdout:5/665: creat d0/d15/d24/ddf/fe8 x:0 0 0 2024-04-12T05:09:08.391 INFO:tasks.workunit.client.1.smithi192.stdout:6/721: write d3/d26/f28 [1401489,118903] 0 2024-04-12T05:09:08.391 INFO:tasks.workunit.client.1.smithi192.stdout:7/604: symlink d3/lcf 0 2024-04-12T05:09:08.394 INFO:tasks.workunit.client.1.smithi192.stdout:4/731: mkdir d1/d54/d82/d71/dff 0 2024-04-12T05:09:08.395 INFO:tasks.workunit.client.1.smithi192.stdout:1/659: creat d0/d1b/d28/d9d/fd9 x:0 0 0 2024-04-12T05:09:08.396 INFO:tasks.workunit.client.1.smithi192.stdout:8/630: dwrite d6/d10/d46/d5e/d5f/d31/d61/d9d/fb9 [0,4194304] 0 2024-04-12T05:09:08.405 INFO:tasks.workunit.client.1.smithi192.stdout:7/605: chown d3/d1d/d5a/d5f/c6b 1758 1 2024-04-12T05:09:08.407 INFO:tasks.workunit.client.1.smithi192.stdout:9/554: dwrite d11/d15/d21/d2c/d1f/d50/f98 [0,4194304] 0 2024-04-12T05:09:08.408 INFO:tasks.workunit.client.0.smithi113.stdout:4/720: truncate da/d73/f71 888130 0 2024-04-12T05:09:08.408 INFO:tasks.workunit.client.0.smithi113.stdout:4/721: fsync da/d24/d60/db4/fc7 0 2024-04-12T05:09:08.409 INFO:tasks.workunit.client.0.smithi113.stdout:4/722: readlink da/d24/d7b/ldf 0 2024-04-12T05:09:08.412 INFO:tasks.workunit.client.1.smithi192.stdout:5/666: rename d0/d15/d24/f46 to d0/d15/d32/d69/fe9 0 2024-04-12T05:09:08.414 INFO:tasks.workunit.client.1.smithi192.stdout:3/645: dread d5/d13/d3c/f25 [4194304,4194304] 0 2024-04-12T05:09:08.416 INFO:tasks.workunit.client.0.smithi113.stdout:4/723: creat da/fef x:0 0 0 2024-04-12T05:09:08.416 INFO:tasks.workunit.client.1.smithi192.stdout:9/555: chown d11/d15/d21/d2c/f8a 4372418 1 2024-04-12T05:09:08.418 INFO:tasks.workunit.client.1.smithi192.stdout:7/606: symlink d3/d7/d60/d65/ld0 0 2024-04-12T05:09:08.419 INFO:tasks.workunit.client.0.smithi113.stdout:4/724: symlink da/d3c/da5/lf0 0 2024-04-12T05:09:08.420 INFO:tasks.workunit.client.1.smithi192.stdout:2/653: rename d1/d8/d17/d19/d60/d59/f90 to d1/d8/d17/d19/d60/d59/da3/fc9 0 2024-04-12T05:09:08.420 INFO:tasks.workunit.client.1.smithi192.stdout:8/631: dwrite f5 [0,4194304] 0 2024-04-12T05:09:08.423 INFO:tasks.workunit.client.1.smithi192.stdout:5/667: symlink d0/d15/d3e/dca/dd3/lea 0 2024-04-12T05:09:08.423 INFO:tasks.workunit.client.1.smithi192.stdout:6/722: dwrite d3/d26/f46 [0,4194304] 0 2024-04-12T05:09:08.423 INFO:tasks.workunit.client.1.smithi192.stdout:9/556: mknod d11/d15/d21/d2c/d1f/d6a/cc2 0 2024-04-12T05:09:08.424 INFO:tasks.workunit.client.0.smithi113.stdout:4/725: link da/d3c/d83/fd7 da/d24/d7e/ff1 0 2024-04-12T05:09:08.428 INFO:tasks.workunit.client.1.smithi192.stdout:2/654: write d1/d8/d17/d19/d60/d9c/fbe [755882,89553] 0 2024-04-12T05:09:08.430 INFO:tasks.workunit.client.0.smithi113.stdout:4/726: rename da/d5a/f91 to da/d4f/ff2 0 2024-04-12T05:09:08.431 INFO:tasks.workunit.client.1.smithi192.stdout:8/632: mkdir d6/d10/d46/d5e/d5f/d31/d80/dc0 0 2024-04-12T05:09:08.433 INFO:tasks.workunit.client.1.smithi192.stdout:7/607: rename d3/d7/d15/d22/l32 to d3/d1d/d4a/d5e/d6d/da3/dce/ld1 0 2024-04-12T05:09:08.434 INFO:tasks.workunit.client.0.smithi113.stdout:4/727: symlink da/d24/d7e/d79/lf3 0 2024-04-12T05:09:08.434 INFO:tasks.workunit.client.1.smithi192.stdout:2/655: dread - d1/d2d/d73/f9b zero size 2024-04-12T05:09:08.437 INFO:tasks.workunit.client.0.smithi113.stdout:4/728: mknod da/d4f/cf4 0 2024-04-12T05:09:08.438 INFO:tasks.workunit.client.1.smithi192.stdout:7/608: chown d3/d7/f89 712470 1 2024-04-12T05:09:08.438 INFO:tasks.workunit.client.1.smithi192.stdout:8/633: creat d6/d10/d46/d5e/d5f/d31/d80/fc1 x:0 0 0 2024-04-12T05:09:08.439 INFO:tasks.workunit.client.1.smithi192.stdout:4/732: dread d1/d54/d82/da4/f92 [0,4194304] 0 2024-04-12T05:09:08.439 INFO:tasks.workunit.client.0.smithi113.stdout:4/729: creat da/de/ff5 x:0 0 0 2024-04-12T05:09:08.440 INFO:tasks.workunit.client.1.smithi192.stdout:2/656: write d1/d8/d3a/de/d7c/fbf [665900,125728] 0 2024-04-12T05:09:08.440 INFO:tasks.workunit.client.1.smithi192.stdout:1/660: dread d0/d2/dd/d3c/f41 [0,4194304] 0 2024-04-12T05:09:08.442 INFO:tasks.workunit.client.1.smithi192.stdout:9/557: dwrite d11/d25/d36/d6e/f77 [0,4194304] 0 2024-04-12T05:09:08.443 INFO:tasks.workunit.client.1.smithi192.stdout:7/609: creat d3/d1d/d5a/d5f/fd2 x:0 0 0 2024-04-12T05:09:08.444 INFO:tasks.workunit.client.1.smithi192.stdout:8/634: rmdir d6/d10/d46/d5e/d5f/d37/d7b 39 2024-04-12T05:09:08.446 INFO:tasks.workunit.client.1.smithi192.stdout:2/657: dread - d1/d8/d3a/de/d7c/d98/fb7 zero size 2024-04-12T05:09:08.447 INFO:tasks.workunit.client.0.smithi113.stdout:4/730: creat da/d24/d60/ff6 x:0 0 0 2024-04-12T05:09:08.450 INFO:tasks.workunit.client.1.smithi192.stdout:4/733: creat d1/d1d/d53/d1c/dd1/def/ddd/dea/f100 x:0 0 0 2024-04-12T05:09:08.453 INFO:tasks.workunit.client.1.smithi192.stdout:1/661: mknod d0/d2/dd/d3c/d5b/da7/dd7/cda 0 2024-04-12T05:09:08.454 INFO:tasks.workunit.client.1.smithi192.stdout:5/668: truncate d0/f13 1126892 0 2024-04-12T05:09:08.457 INFO:tasks.workunit.client.1.smithi192.stdout:7/610: creat d3/d7/fd3 x:0 0 0 2024-04-12T05:09:08.460 INFO:tasks.workunit.client.1.smithi192.stdout:8/635: dwrite d6/d10/d46/d5e/d5f/f57 [0,4194304] 0 2024-04-12T05:09:08.465 INFO:tasks.workunit.client.1.smithi192.stdout:1/662: creat d0/d29/d8d/d96/fdb x:0 0 0 2024-04-12T05:09:08.465 INFO:tasks.workunit.client.1.smithi192.stdout:8/636: chown d6/d10/l99 3 1 2024-04-12T05:09:08.466 INFO:tasks.workunit.client.1.smithi192.stdout:7/611: symlink d3/d1d/d74/d86/ld4 0 2024-04-12T05:09:08.467 INFO:tasks.workunit.client.1.smithi192.stdout:9/558: dwrite d11/d25/d36/d3d/fa1 [0,4194304] 0 2024-04-12T05:09:08.467 INFO:tasks.workunit.client.0.smithi113.stdout:4/731: getdents da/d5a 0 2024-04-12T05:09:08.468 INFO:tasks.workunit.client.1.smithi192.stdout:5/669: truncate d0/d15/d32/d69/d7d/fc9 4710742 0 2024-04-12T05:09:08.471 INFO:tasks.workunit.client.1.smithi192.stdout:1/663: chown d0/d2/dd/d3c/d5b/d74/d7d/fd6 42 1 2024-04-12T05:09:08.472 INFO:tasks.workunit.client.1.smithi192.stdout:1/664: truncate d0/d1b/d44/f6e 1007754 0 2024-04-12T05:09:08.476 INFO:tasks.workunit.client.1.smithi192.stdout:9/559: mknod d11/d25/d36/d6e/d71/cc3 0 2024-04-12T05:09:08.478 INFO:tasks.workunit.client.1.smithi192.stdout:1/665: symlink d0/d1b/d28/d9d/ldc 0 2024-04-12T05:09:08.478 INFO:tasks.workunit.client.1.smithi192.stdout:2/658: dread d1/f2 [0,4194304] 0 2024-04-12T05:09:08.481 INFO:tasks.workunit.client.1.smithi192.stdout:8/637: symlink d6/d10/d46/d5e/lc2 0 2024-04-12T05:09:08.482 INFO:tasks.workunit.client.1.smithi192.stdout:4/734: getdents d1/d54/d82/d9c/ddb 0 2024-04-12T05:09:08.483 INFO:tasks.workunit.client.1.smithi192.stdout:5/670: mkdir d0/d15/d3e/dca/dda/deb 0 2024-04-12T05:09:08.483 INFO:tasks.workunit.client.1.smithi192.stdout:9/560: truncate d11/d25/d36/d6e/f56 566837 0 2024-04-12T05:09:08.485 INFO:tasks.workunit.client.1.smithi192.stdout:7/612: link d3/d7/d15/d22/f4d d3/d1d/d5a/d5f/fd5 0 2024-04-12T05:09:08.485 INFO:tasks.workunit.client.1.smithi192.stdout:5/671: stat d0/d21/f2f 0 2024-04-12T05:09:08.485 INFO:tasks.workunit.client.1.smithi192.stdout:2/659: truncate d1/d8/d3a/f77 185098 0 2024-04-12T05:09:08.487 INFO:tasks.workunit.client.1.smithi192.stdout:4/735: mkdir d1/d49/d101 0 2024-04-12T05:09:08.488 INFO:tasks.workunit.client.1.smithi192.stdout:1/666: dwrite d0/d2/d1e/d3f/d56/f93 [0,4194304] 0 2024-04-12T05:09:08.491 INFO:tasks.workunit.client.1.smithi192.stdout:5/672: write d0/d15/d32/d45/fb1 [623356,44531] 0 2024-04-12T05:09:08.493 INFO:tasks.workunit.client.1.smithi192.stdout:4/736: stat d1/d54/d82/da4/da8/cfa 0 2024-04-12T05:09:08.495 INFO:tasks.workunit.client.1.smithi192.stdout:4/737: chown d1/d54/d55/d64/f86 6 1 2024-04-12T05:09:08.497 INFO:tasks.workunit.client.1.smithi192.stdout:7/613: chown d3/d7/d15/d22/f4d 47 1 2024-04-12T05:09:08.498 INFO:tasks.workunit.client.1.smithi192.stdout:5/673: creat d0/d15/d32/db2/fec x:0 0 0 2024-04-12T05:09:08.498 INFO:tasks.workunit.client.1.smithi192.stdout:9/561: dwrite d11/d25/d36/d6e/d33/da2/f7b [0,4194304] 0 2024-04-12T05:09:08.501 INFO:tasks.workunit.client.1.smithi192.stdout:2/660: creat d1/d8/d3a/de/fca x:0 0 0 2024-04-12T05:09:08.504 INFO:tasks.workunit.client.1.smithi192.stdout:5/674: unlink d0/d15/d32/l34 0 2024-04-12T05:09:08.507 INFO:tasks.workunit.client.1.smithi192.stdout:9/562: creat d11/d25/d36/d6e/d33/da2/fc4 x:0 0 0 2024-04-12T05:09:08.510 INFO:tasks.workunit.client.1.smithi192.stdout:2/661: dread d1/d8/d3a/de/d7c/fbf [0,4194304] 0 2024-04-12T05:09:08.511 INFO:tasks.workunit.client.1.smithi192.stdout:4/738: dwrite d1/d1d/f6c [0,4194304] 0 2024-04-12T05:09:08.512 INFO:tasks.workunit.client.1.smithi192.stdout:9/563: creat d11/d15/d21/d2c/d1f/d50/d73/fc5 x:0 0 0 2024-04-12T05:09:08.515 INFO:tasks.workunit.client.1.smithi192.stdout:2/662: creat d1/d2d/fcb x:0 0 0 2024-04-12T05:09:08.518 INFO:tasks.workunit.client.1.smithi192.stdout:1/667: dread d0/d2/f7b [0,4194304] 0 2024-04-12T05:09:08.520 INFO:tasks.workunit.client.1.smithi192.stdout:2/663: creat d1/d8/d17/fcc x:0 0 0 2024-04-12T05:09:08.523 INFO:tasks.workunit.client.1.smithi192.stdout:1/668: rmdir d0/d2/dd/d3c/d5b/db6 39 2024-04-12T05:09:08.525 INFO:tasks.workunit.client.1.smithi192.stdout:2/664: creat d1/d8/d17/db3/fcd x:0 0 0 2024-04-12T05:09:08.526 INFO:tasks.workunit.client.1.smithi192.stdout:4/739: link d1/d1d/d53/c34 d1/d1d/d53/d1c/dd1/def/ddd/dea/df4/c102 0 2024-04-12T05:09:08.526 INFO:tasks.workunit.client.1.smithi192.stdout:4/740: stat d1/c24 0 2024-04-12T05:09:08.528 INFO:tasks.workunit.client.1.smithi192.stdout:1/669: mkdir d0/d1b/d44/d69/ddd 0 2024-04-12T05:09:08.528 INFO:tasks.workunit.client.1.smithi192.stdout:8/638: dread d6/d10/fa5 [0,4194304] 0 2024-04-12T05:09:08.529 INFO:tasks.workunit.client.1.smithi192.stdout:2/665: creat d1/d8/d17/d19/d60/d59/da3/fce x:0 0 0 2024-04-12T05:09:08.530 INFO:tasks.workunit.client.1.smithi192.stdout:5/675: dread d0/f3d [0,4194304] 0 2024-04-12T05:09:08.530 INFO:tasks.workunit.client.1.smithi192.stdout:4/741: mkdir d1/d1d/d53/d1c/dd1/def/ddd/dfb/d103 0 2024-04-12T05:09:08.532 INFO:tasks.workunit.client.1.smithi192.stdout:2/666: write d1/d2d/f32 [2705821,115770] 0 2024-04-12T05:09:08.532 INFO:tasks.workunit.client.1.smithi192.stdout:1/670: unlink d0/d1b/d44/fb9 0 2024-04-12T05:09:08.533 INFO:tasks.workunit.client.1.smithi192.stdout:5/676: stat d0/d15/d32/d45/cc0 0 2024-04-12T05:09:08.534 INFO:tasks.workunit.client.1.smithi192.stdout:8/639: getdents d6/d10/d46/d5e/d5f/d31/d80/dc0 0 2024-04-12T05:09:08.539 INFO:tasks.workunit.client.1.smithi192.stdout:9/564: dwrite d11/d15/d21/d2c/d1f/f4d [0,4194304] 0 2024-04-12T05:09:08.540 INFO:tasks.workunit.client.1.smithi192.stdout:8/640: write d6/f18 [182367,130269] 0 2024-04-12T05:09:08.542 INFO:tasks.workunit.client.1.smithi192.stdout:4/742: write d1/d54/d55/d64/f7d [112808,4639] 0 2024-04-12T05:09:08.545 INFO:tasks.workunit.client.1.smithi192.stdout:1/671: getdents d0/d29/dbf 0 2024-04-12T05:09:08.546 INFO:tasks.workunit.client.1.smithi192.stdout:9/565: mknod d11/d15/d21/d2c/cc6 0 2024-04-12T05:09:08.547 INFO:tasks.workunit.client.1.smithi192.stdout:2/667: link d1/d8/d3a/de/d7c/c89 d1/d8/d3a/de/d7c/d98/ccf 0 2024-04-12T05:09:08.549 INFO:tasks.workunit.client.1.smithi192.stdout:5/677: dwrite d0/d15/d24/d47/fdd [0,4194304] 0 2024-04-12T05:09:08.549 INFO:tasks.workunit.client.1.smithi192.stdout:8/641: mknod d6/d10/d46/d5e/d5f/d31/d80/db1/cc3 0 2024-04-12T05:09:08.549 INFO:tasks.workunit.client.0.smithi113.stdout:6/704: sync 2024-04-12T05:09:08.549 INFO:tasks.workunit.client.0.smithi113.stdout:5/804: sync 2024-04-12T05:09:08.550 INFO:tasks.workunit.client.0.smithi113.stdout:1/796: sync 2024-04-12T05:09:08.552 INFO:tasks.workunit.client.1.smithi192.stdout:2/668: chown d1/d8/f10 11 1 2024-04-12T05:09:08.552 INFO:tasks.workunit.client.0.smithi113.stdout:1/797: unlink d10/d2e/l3d 0 2024-04-12T05:09:08.555 INFO:tasks.workunit.client.0.smithi113.stdout:6/705: write d8/de/d13/df0/d66/dc4/de3/fff [2109492,97177] 0 2024-04-12T05:09:08.555 INFO:tasks.workunit.client.1.smithi192.stdout:8/642: fsync d6/d10/fa5 0 2024-04-12T05:09:08.557 INFO:tasks.workunit.client.1.smithi192.stdout:2/669: write d1/d8/fc [775909,95344] 0 2024-04-12T05:09:08.557 INFO:tasks.workunit.client.0.smithi113.stdout:1/798: symlink d10/d2e/d7c/l107 0 2024-04-12T05:09:08.557 INFO:tasks.workunit.client.0.smithi113.stdout:5/805: rename d0/d10/d27/d1d/d3f/f6c to d0/d10/d6e/d93/f113 0 2024-04-12T05:09:08.558 INFO:tasks.workunit.client.0.smithi113.stdout:1/799: chown d10/d41/d89/d96/l81 339146980 1 2024-04-12T05:09:08.558 INFO:tasks.workunit.client.0.smithi113.stdout:6/706: fsync d8/de/dd5/de7/f6a 0 2024-04-12T05:09:08.561 INFO:tasks.workunit.client.0.smithi113.stdout:1/800: mknod d10/da5/ddf/c108 0 2024-04-12T05:09:08.562 INFO:tasks.workunit.client.1.smithi192.stdout:4/743: dwrite d1/d54/d82/d71/d9e/dc4/dd9/fee [0,4194304] 0 2024-04-12T05:09:08.562 INFO:tasks.workunit.client.0.smithi113.stdout:1/801: dread - d10/d41/d89/ff6 zero size 2024-04-12T05:09:08.562 INFO:tasks.workunit.client.0.smithi113.stdout:6/707: mknod d8/de/dd5/de7/db3/dc5/c105 0 2024-04-12T05:09:08.562 INFO:tasks.workunit.client.1.smithi192.stdout:1/672: dread d0/d2/dd/d3c/f41 [0,4194304] 0 2024-04-12T05:09:08.563 INFO:tasks.workunit.client.1.smithi192.stdout:1/673: chown d0 25958 1 2024-04-12T05:09:08.565 INFO:tasks.workunit.client.1.smithi192.stdout:5/678: rmdir d0/d15/d32/d69/d7d/de7 0 2024-04-12T05:09:08.565 INFO:tasks.workunit.client.1.smithi192.stdout:8/643: dread d6/d10/d46/d5e/d5f/d38/d3b/fae [0,4194304] 0 2024-04-12T05:09:08.565 INFO:tasks.workunit.client.1.smithi192.stdout:1/674: write d0/d2/dd/d3c/d5b/d74/d90/f94 [289626,15893] 0 2024-04-12T05:09:08.570 INFO:tasks.workunit.client.0.smithi113.stdout:1/802: rename d10/ddc to d10/d2e/d31/dc3/d109 0 2024-04-12T05:09:08.574 INFO:tasks.workunit.client.1.smithi192.stdout:5/679: mkdir d0/d15/d3e/dca/db9/ded 0 2024-04-12T05:09:08.579 INFO:tasks.workunit.client.1.smithi192.stdout:2/670: getdents d1/d8/d3a 0 2024-04-12T05:09:08.580 INFO:tasks.workunit.client.0.smithi113.stdout:1/803: dwrite d10/d2e/d31/dc3/d109/dde/ffe [0,4194304] 0 2024-04-12T05:09:08.582 INFO:tasks.workunit.client.1.smithi192.stdout:9/566: truncate d11/d25/d36/d3d/fa1 1455199 0 2024-04-12T05:09:08.584 INFO:tasks.workunit.client.1.smithi192.stdout:4/744: rmdir d1/d49/d101 0 2024-04-12T05:09:08.584 INFO:tasks.workunit.client.0.smithi113.stdout:5/806: dread d0/d10/d27/d1d/d38/f8d [0,4194304] 0 2024-04-12T05:09:08.586 INFO:tasks.workunit.client.0.smithi113.stdout:6/708: dread d8/d45/ff3 [0,4194304] 0 2024-04-12T05:09:08.588 INFO:tasks.workunit.client.0.smithi113.stdout:6/709: chown d8/de/d17/c84 265 1 2024-04-12T05:09:08.589 INFO:tasks.workunit.client.1.smithi192.stdout:1/675: symlink d0/d2/dd/lde 0 2024-04-12T05:09:08.589 INFO:tasks.workunit.client.1.smithi192.stdout:4/745: dwrite d1/d54/d82/da4/f79 [0,4194304] 0 2024-04-12T05:09:08.590 INFO:tasks.workunit.client.0.smithi113.stdout:5/807: creat d0/d10/d27/d1d/d38/d44/f114 x:0 0 0 2024-04-12T05:09:08.591 INFO:tasks.workunit.client.0.smithi113.stdout:6/710: fdatasync d8/de/d13/d11/d36/d3d/d4c/d26/d5b/ff5 0 2024-04-12T05:09:08.593 INFO:tasks.workunit.client.1.smithi192.stdout:2/671: creat d1/d8/d17/fd0 x:0 0 0 2024-04-12T05:09:08.596 INFO:tasks.workunit.client.1.smithi192.stdout:4/746: chown d1/d6b/la2 24488981 1 2024-04-12T05:09:08.596 INFO:tasks.workunit.client.0.smithi113.stdout:3/655: sync 2024-04-12T05:09:08.596 INFO:tasks.workunit.client.0.smithi113.stdout:9/718: sync 2024-04-12T05:09:08.596 INFO:tasks.workunit.client.0.smithi113.stdout:7/714: sync 2024-04-12T05:09:08.597 INFO:tasks.workunit.client.0.smithi113.stdout:8/761: sync 2024-04-12T05:09:08.597 INFO:tasks.workunit.client.0.smithi113.stdout:2/695: sync 2024-04-12T05:09:08.597 INFO:tasks.workunit.client.0.smithi113.stdout:0/801: sync 2024-04-12T05:09:08.598 INFO:tasks.workunit.client.0.smithi113.stdout:6/711: rename d8/de/dd5/f101 to d8/de/d13/df0/d66/f106 0 2024-04-12T05:09:08.602 INFO:tasks.workunit.client.0.smithi113.stdout:6/712: stat d8/de/d17/c3e 0 2024-04-12T05:09:08.603 INFO:tasks.workunit.client.0.smithi113.stdout:8/762: chown d9/d16/d3c/f52 332055 1 2024-04-12T05:09:08.604 INFO:tasks.workunit.client.1.smithi192.stdout:4/747: dwrite d1/d54/d55/d64/f8f [0,4194304] 0 2024-04-12T05:09:08.604 INFO:tasks.workunit.client.0.smithi113.stdout:9/719: mknod d1/d4/d5/d22/cf3 0 2024-04-12T05:09:08.605 INFO:tasks.workunit.client.1.smithi192.stdout:5/680: link d0/d15/d3e/dca/la4 d0/d21/d5e/lee 0 2024-04-12T05:09:08.605 INFO:tasks.workunit.client.0.smithi113.stdout:3/656: read d0/d13/d15/d28/fa5 [4681,32685] 0 2024-04-12T05:09:08.605 INFO:tasks.workunit.client.0.smithi113.stdout:2/696: mknod d8/d9/de/d15/d38/cd0 0 2024-04-12T05:09:08.606 INFO:tasks.workunit.client.0.smithi113.stdout:0/802: stat d1/d6/d16/d2a/d50/db6/d36/fad 0 2024-04-12T05:09:08.607 INFO:tasks.workunit.client.0.smithi113.stdout:1/804: write f2 [500261,74695] 0 2024-04-12T05:09:08.607 INFO:tasks.workunit.client.0.smithi113.stdout:7/715: write d0/dc/d6/d9b/d80/f96 [1705097,96626] 0 2024-04-12T05:09:08.608 INFO:tasks.workunit.client.1.smithi192.stdout:2/672: creat d1/d2d/fd1 x:0 0 0 2024-04-12T05:09:08.610 INFO:tasks.workunit.client.1.smithi192.stdout:5/681: chown d0/d15/d32/d45/fb0 39 1 2024-04-12T05:09:08.611 INFO:tasks.workunit.client.0.smithi113.stdout:5/808: dwrite d0/d10/d27/d1d/d8c/db8/dc6/fc4 [0,4194304] 0 2024-04-12T05:09:08.614 INFO:tasks.workunit.client.0.smithi113.stdout:8/763: mkdir d9/d1b/d8e/d93/d32/d48/dc1/df3 0 2024-04-12T05:09:08.614 INFO:tasks.workunit.client.0.smithi113.stdout:0/803: fdatasync d1/d6/d8e/fec 0 2024-04-12T05:09:08.615 INFO:tasks.workunit.client.0.smithi113.stdout:3/657: mkdir d0/d13/d15/d28/d2a/d8f/ddd 0 2024-04-12T05:09:08.618 INFO:tasks.workunit.client.1.smithi192.stdout:8/644: truncate d6/d10/d46/d5e/d5f/d37/f6f 1353179 0 2024-04-12T05:09:08.618 INFO:tasks.workunit.client.1.smithi192.stdout:4/748: dwrite d1/d1d/d53/d1c/d95/fc0 [0,4194304] 0 2024-04-12T05:09:08.619 INFO:tasks.workunit.client.0.smithi113.stdout:6/713: rename d8/de/d13/d11/d36/d3d/d4c/lf2 to d8/de/d13/d11/d36/d3d/l107 0 2024-04-12T05:09:08.619 INFO:tasks.workunit.client.0.smithi113.stdout:3/658: read d0/d13/d47/f78 [1196957,32168] 0 2024-04-12T05:09:08.619 INFO:tasks.workunit.client.0.smithi113.stdout:0/804: read d1/d43/d59/f82 [516693,124896] 0 2024-04-12T05:09:08.621 INFO:tasks.workunit.client.0.smithi113.stdout:1/805: getdents d10/d2e/d31/d5c/daf 0 2024-04-12T05:09:08.622 INFO:tasks.workunit.client.0.smithi113.stdout:6/714: chown d8/de/d13/d11 116 1 2024-04-12T05:09:08.624 INFO:tasks.workunit.client.0.smithi113.stdout:8/764: mknod d9/d1b/d8e/d93/cf4 0 2024-04-12T05:09:08.625 INFO:tasks.workunit.client.0.smithi113.stdout:8/765: readlink d9/d1b/d96/le5 0 2024-04-12T05:09:08.625 INFO:tasks.workunit.client.0.smithi113.stdout:1/806: creat d10/da5/f10a x:0 0 0 2024-04-12T05:09:08.626 INFO:tasks.workunit.client.1.smithi192.stdout:8/645: symlink d6/d10/lc4 0 2024-04-12T05:09:08.627 INFO:tasks.workunit.client.1.smithi192.stdout:1/676: dwrite d0/d2/dd/f3b [0,4194304] 0 2024-04-12T05:09:08.628 INFO:tasks.workunit.client.0.smithi113.stdout:0/805: creat d1/d80/ff9 x:0 0 0 2024-04-12T05:09:08.630 INFO:tasks.workunit.client.0.smithi113.stdout:2/697: rename d8/d9/de/c93 to d8/d9/de/d12/d2b/d79/cd1 0 2024-04-12T05:09:08.631 INFO:tasks.workunit.client.1.smithi192.stdout:5/682: rename d0/d15/d64/fad to d0/d15/d32/fef 0 2024-04-12T05:09:08.631 INFO:tasks.workunit.client.1.smithi192.stdout:2/673: dread d1/d8/d17/d19/d60/d9c/fbe [0,4194304] 0 2024-04-12T05:09:08.632 INFO:tasks.workunit.client.0.smithi113.stdout:9/720: write d1/df/fc3 [633235,34534] 0 2024-04-12T05:09:08.636 INFO:tasks.workunit.client.1.smithi192.stdout:1/677: rmdir d0/d29/d8d/d96 39 2024-04-12T05:09:08.637 INFO:tasks.workunit.client.0.smithi113.stdout:5/809: getdents d0/d10/d27/d1d/d38/d44/d66 0 2024-04-12T05:09:08.637 INFO:tasks.workunit.client.0.smithi113.stdout:7/716: write d0/dd/d11/f8c [263494,80836] 0 2024-04-12T05:09:08.638 INFO:tasks.workunit.client.0.smithi113.stdout:8/766: chown d9/d16/d7f/d3a/d6c/d90/fbc 9 1 2024-04-12T05:09:08.640 INFO:tasks.workunit.client.1.smithi192.stdout:8/646: mkdir d6/d10/d46/d5e/d5f/d31/d61/dc5 0 2024-04-12T05:09:08.641 INFO:tasks.workunit.client.0.smithi113.stdout:7/717: write d0/dc/f41 [608750,117946] 0 2024-04-12T05:09:08.641 INFO:tasks.workunit.client.1.smithi192.stdout:5/683: creat d0/d15/d32/d45/ff0 x:0 0 0 2024-04-12T05:09:08.643 INFO:tasks.workunit.client.0.smithi113.stdout:4/732: sync 2024-04-12T05:09:08.646 INFO:tasks.workunit.client.1.smithi192.stdout:2/674: read d1/d8/d3a/de/f97 [1497,82098] 0 2024-04-12T05:09:08.648 INFO:tasks.workunit.client.1.smithi192.stdout:1/678: creat d0/d2/dd/d21/d33/dd8/fdf x:0 0 0 2024-04-12T05:09:08.650 INFO:tasks.workunit.client.0.smithi113.stdout:5/810: mkdir d0/d10/d27/d1d/d41/d115 0 2024-04-12T05:09:08.651 INFO:tasks.workunit.client.0.smithi113.stdout:0/806: unlink d1/d6/d16/f4c 0 2024-04-12T05:09:08.652 INFO:tasks.workunit.client.0.smithi113.stdout:9/721: mknod d1/df/d29/de8/cf4 0 2024-04-12T05:09:08.652 INFO:tasks.workunit.client.1.smithi192.stdout:2/675: truncate d1/d8/d17/f5f 626753 0 2024-04-12T05:09:08.655 INFO:tasks.workunit.client.0.smithi113.stdout:5/811: read d0/d10/d27/d1d/d38/f8d [1486529,113004] 0 2024-04-12T05:09:08.655 INFO:tasks.workunit.client.1.smithi192.stdout:1/679: rename d0/d2/dd/d3c/d5b/d74/d7d/fce to d0/d2/dd/d3c/d5b/d74/d7d/fe0 0 2024-04-12T05:09:08.657 INFO:tasks.workunit.client.0.smithi113.stdout:0/807: write d1/d6/d16/d2a/d50/db6/d36/fd1 [1045861,71846] 0 2024-04-12T05:09:08.658 INFO:tasks.workunit.client.1.smithi192.stdout:2/676: mkdir d1/d2d/d73/da8/dd2 0 2024-04-12T05:09:08.659 INFO:tasks.workunit.client.0.smithi113.stdout:2/698: dwrite d8/d9/de/d15/f9d [0,4194304] 0 2024-04-12T05:09:08.662 INFO:tasks.workunit.client.0.smithi113.stdout:5/812: truncate d0/d10/f56 4195375 0 2024-04-12T05:09:08.662 INFO:tasks.workunit.client.0.smithi113.stdout:3/659: getdents d0/d13/d15/d28/d2a/d8b 0 2024-04-12T05:09:08.662 INFO:tasks.workunit.client.1.smithi192.stdout:1/680: creat d0/d2/d1e/d3f/d78/fe1 x:0 0 0 2024-04-12T05:09:08.664 INFO:tasks.workunit.client.0.smithi113.stdout:7/718: dread d0/dd/f66 [0,4194304] 0 2024-04-12T05:09:08.664 INFO:tasks.workunit.client.0.smithi113.stdout:6/715: truncate d8/de/d13/d11/d36/d3d/d4c/d26/d5b/fb2 3237998 0 2024-04-12T05:09:08.664 INFO:tasks.workunit.client.1.smithi192.stdout:2/677: chown d1/d8/d3a/de/c38 43805867 1 2024-04-12T05:09:08.665 INFO:tasks.workunit.client.1.smithi192.stdout:5/684: dread d0/d15/d64/fcd [0,4194304] 0 2024-04-12T05:09:08.670 INFO:tasks.workunit.client.1.smithi192.stdout:5/685: chown d0/d15/f23 5580867 1 2024-04-12T05:09:08.673 INFO:tasks.workunit.client.1.smithi192.stdout:4/749: truncate d1/d54/d82/da4/f92 3863157 0 2024-04-12T05:09:08.674 INFO:tasks.workunit.client.0.smithi113.stdout:2/699: mkdir d8/d9/de/d12/d2b/dd2 0 2024-04-12T05:09:08.676 INFO:tasks.workunit.client.0.smithi113.stdout:6/716: chown d8/c2f 3339249 1 2024-04-12T05:09:08.676 INFO:tasks.workunit.client.0.smithi113.stdout:0/808: link d1/d43/d59/db2/lc5 d1/d6/d16/d2a/d50/lfa 0 2024-04-12T05:09:08.677 INFO:tasks.workunit.client.0.smithi113.stdout:2/700: readlink d8/d9/lf 0 2024-04-12T05:09:08.677 INFO:tasks.workunit.client.0.smithi113.stdout:3/660: symlink d0/d13/d15/d28/d2a/d34/d5f/d7b/lde 0 2024-04-12T05:09:08.679 INFO:tasks.workunit.client.0.smithi113.stdout:5/813: mknod d0/d10/d27/d1d/d3f/db0/dda/c116 0 2024-04-12T05:09:08.680 INFO:tasks.workunit.client.0.smithi113.stdout:7/719: write d0/d52/dc5/ff7 [5226242,74138] 0 2024-04-12T05:09:08.682 INFO:tasks.workunit.client.0.smithi113.stdout:9/722: truncate d1/d27/f5d 3849560 0 2024-04-12T05:09:08.684 INFO:tasks.workunit.client.0.smithi113.stdout:3/661: readlink d0/d7e/l9b 0 2024-04-12T05:09:08.685 INFO:tasks.workunit.client.0.smithi113.stdout:1/807: dwrite d10/d2e/d7c/dc5/ddb/ff8 [4194304,4194304] 0 2024-04-12T05:09:08.685 INFO:tasks.workunit.client.0.smithi113.stdout:8/767: dread d9/d1b/f45 [0,4194304] 0 2024-04-12T05:09:08.685 INFO:tasks.workunit.client.1.smithi192.stdout:2/678: mknod d1/d2d/d73/da8/db2/cd3 0 2024-04-12T05:09:08.686 INFO:tasks.workunit.client.1.smithi192.stdout:5/686: rmdir d0/d15/d3e/dca/dda 39 2024-04-12T05:09:08.686 INFO:tasks.workunit.client.1.smithi192.stdout:4/750: creat d1/d54/d55/dc3/f104 x:0 0 0 2024-04-12T05:09:08.687 INFO:tasks.workunit.client.1.smithi192.stdout:8/647: dwrite d6/d10/d46/d5e/d5f/d37/f6f [0,4194304] 0 2024-04-12T05:09:08.687 INFO:tasks.workunit.client.0.smithi113.stdout:0/809: mknod d1/d6/d16/dd5/dbb/cfb 0 2024-04-12T05:09:08.690 INFO:tasks.workunit.client.0.smithi113.stdout:5/814: rmdir d0/d10/d27/d1d/d3f/d7e 39 2024-04-12T05:09:08.690 INFO:tasks.workunit.client.1.smithi192.stdout:8/648: write d6/da/f30 [1632914,106921] 0 2024-04-12T05:09:08.691 INFO:tasks.workunit.client.0.smithi113.stdout:3/662: chown d0/d13/f81 6 1 2024-04-12T05:09:08.691 INFO:tasks.workunit.client.1.smithi192.stdout:4/751: truncate d1/d54/d82/da4/fe3 669723 0 2024-04-12T05:09:08.692 INFO:tasks.workunit.client.0.smithi113.stdout:4/733: dwrite da/d3c/d43/f7d [0,4194304] 0 2024-04-12T05:09:08.694 INFO:tasks.workunit.client.0.smithi113.stdout:7/720: read d0/dc/d6/d9b/ff3 [2266561,81233] 0 2024-04-12T05:09:08.694 INFO:tasks.workunit.client.0.smithi113.stdout:1/808: chown d10/d41/d5b/fa3 8889 1 2024-04-12T05:09:08.696 INFO:tasks.workunit.client.0.smithi113.stdout:6/717: stat d8/de/d13/d11/d36/d3d/d4c/d26/d5b/cd0 0 2024-04-12T05:09:08.697 INFO:tasks.workunit.client.0.smithi113.stdout:2/701: creat d8/d9/de/d12/fd3 x:0 0 0 2024-04-12T05:09:08.699 INFO:tasks.workunit.client.0.smithi113.stdout:6/718: chown d8/de/d17/f78 0 1 2024-04-12T05:09:08.699 INFO:tasks.workunit.client.0.smithi113.stdout:9/723: fdatasync d1/d4/d5/d6/f48 0 2024-04-12T05:09:08.699 INFO:tasks.workunit.client.0.smithi113.stdout:0/810: rmdir d1/d6/d16/dd5/dd6 39 2024-04-12T05:09:08.701 INFO:tasks.workunit.client.0.smithi113.stdout:7/721: write d0/d6c/ff5 [684977,65786] 0 2024-04-12T05:09:08.701 INFO:tasks.workunit.client.0.smithi113.stdout:8/768: unlink d9/d1b/d8e/d93/d32/d5b/fdd 0 2024-04-12T05:09:08.704 INFO:tasks.workunit.client.0.smithi113.stdout:2/702: readlink d8/d9/de/d12/d3c/d8a/la9 0 2024-04-12T05:09:08.709 INFO:tasks.workunit.client.0.smithi113.stdout:5/815: write d0/d10/d27/d1d/d38/fba [3267626,23643] 0 2024-04-12T05:09:08.711 INFO:tasks.workunit.client.0.smithi113.stdout:0/811: dread d1/d80/f84 [0,4194304] 0 2024-04-12T05:09:08.711 INFO:tasks.workunit.client.1.smithi192.stdout:1/681: dread d0/d2/dd/d21/d39/f49 [0,4194304] 0 2024-04-12T05:09:08.719 INFO:tasks.workunit.client.0.smithi113.stdout:2/703: mkdir d8/d9/d27/d88/d91/dd4 0 2024-04-12T05:09:08.720 INFO:tasks.workunit.client.0.smithi113.stdout:0/812: mknod d1/d43/d59/df4/cfc 0 2024-04-12T05:09:08.721 INFO:tasks.workunit.client.0.smithi113.stdout:6/719: dwrite d8/de/d13/d11/d36/d3d/f90 [0,4194304] 0 2024-04-12T05:09:08.721 INFO:tasks.workunit.client.0.smithi113.stdout:9/724: dread d1/d4/d5/d22/fca [0,4194304] 0 2024-04-12T05:09:08.723 INFO:tasks.workunit.client.0.smithi113.stdout:3/663: link d0/d13/d15/d28/d2a/d8f/cd0 d0/d7e/cdf 0 2024-04-12T05:09:08.724 INFO:tasks.workunit.client.0.smithi113.stdout:1/809: getdents d10/d41 0 2024-04-12T05:09:08.725 INFO:tasks.workunit.client.0.smithi113.stdout:8/769: dwrite d9/d1b/d8e/d93/f71 [0,4194304] 0 2024-04-12T05:09:08.731 INFO:tasks.workunit.client.0.smithi113.stdout:1/810: write d10/d41/d5b/fd2 [846079,31514] 0 2024-04-12T05:09:08.731 INFO:tasks.workunit.client.0.smithi113.stdout:5/816: mknod d0/d10/d27/d1d/d3f/d7e/df8/c117 0 2024-04-12T05:09:08.733 INFO:tasks.workunit.client.0.smithi113.stdout:7/722: getdents d0/dc/d6/d9b/dcd 0 2024-04-12T05:09:08.733 INFO:tasks.workunit.client.0.smithi113.stdout:1/811: fsync d10/d55/d9c/fe5 0 2024-04-12T05:09:08.735 INFO:tasks.workunit.client.0.smithi113.stdout:7/723: stat d0/d52/dc5/ff7 0 2024-04-12T05:09:08.735 INFO:tasks.workunit.client.0.smithi113.stdout:3/664: mkdir d0/d13/d15/d28/d2a/d34/d5f/d7b/de0 0 2024-04-12T05:09:08.736 INFO:tasks.workunit.client.0.smithi113.stdout:2/704: creat d8/d9/de/fd5 x:0 0 0 2024-04-12T05:09:08.737 INFO:tasks.workunit.client.0.smithi113.stdout:3/665: chown d0/d13/d15/d28/d2a/d8b/d91/dad 0 1 2024-04-12T05:09:08.738 INFO:tasks.workunit.client.1.smithi192.stdout:2/679: symlink d1/d8/d17/d19/d60/d59/ld4 0 2024-04-12T05:09:08.738 INFO:tasks.workunit.client.0.smithi113.stdout:6/720: rename d8/de/dd2 to d8/de/d13/d11/d36/d3d/d4c/d26/d108 0 2024-04-12T05:09:08.739 INFO:tasks.workunit.client.0.smithi113.stdout:8/770: creat d9/d16/d7f/d3a/d6c/d90/ff5 x:0 0 0 2024-04-12T05:09:08.740 INFO:tasks.workunit.client.0.smithi113.stdout:1/812: creat d10/d41/d89/d96/f10b x:0 0 0 2024-04-12T05:09:08.740 INFO:tasks.workunit.client.0.smithi113.stdout:5/817: mknod d0/d10/d27/d1d/d3f/d73/db9/c118 0 2024-04-12T05:09:08.741 INFO:tasks.workunit.client.1.smithi192.stdout:4/752: symlink d1/d54/l105 0 2024-04-12T05:09:08.742 INFO:tasks.workunit.client.1.smithi192.stdout:4/753: write d1/d54/d82/f62 [1267874,25572] 0 2024-04-12T05:09:08.744 INFO:tasks.workunit.client.0.smithi113.stdout:7/724: chown d0/dc/l63 44601477 1 2024-04-12T05:09:08.746 INFO:tasks.workunit.client.0.smithi113.stdout:6/721: fdatasync d8/d1d/fac 0 2024-04-12T05:09:08.747 INFO:tasks.workunit.client.0.smithi113.stdout:1/813: mknod d10/d55/c10c 0 2024-04-12T05:09:08.747 INFO:tasks.workunit.client.0.smithi113.stdout:4/734: dread da/d24/fe3 [0,4194304] 0 2024-04-12T05:09:08.748 INFO:tasks.workunit.client.0.smithi113.stdout:9/725: link d1/df/d29/da6/la4 d1/df/d29/de8/dc7/lf5 0 2024-04-12T05:09:08.749 INFO:tasks.workunit.client.0.smithi113.stdout:2/705: fdatasync d8/d9/de/d12/d2b/f54 0 2024-04-12T05:09:08.754 INFO:tasks.workunit.client.1.smithi192.stdout:5/687: getdents d0/d15/d3e/dca 0 2024-04-12T05:09:08.755 INFO:tasks.workunit.client.1.smithi192.stdout:5/688: chown d0/d15/d64/c8a 5298 1 2024-04-12T05:09:08.755 INFO:tasks.workunit.client.0.smithi113.stdout:5/818: dwrite d0/d10/d27/d1d/fc7 [0,4194304] 0 2024-04-12T05:09:08.756 INFO:tasks.workunit.client.0.smithi113.stdout:8/771: fdatasync d9/d1b/d8e/d93/f84 0 2024-04-12T05:09:08.757 INFO:tasks.workunit.client.0.smithi113.stdout:6/722: read - d8/de/d13/d11/d36/d3d/d4c/d26/d5b/ff8 zero size 2024-04-12T05:09:08.758 INFO:tasks.workunit.client.1.smithi192.stdout:8/649: rmdir d6/d10/d46/d5e/d5f/d31/d80/dc0 0 2024-04-12T05:09:08.759 INFO:tasks.workunit.client.0.smithi113.stdout:1/814: dread d10/d2e/dd8/d99/fb8 [0,4194304] 0 2024-04-12T05:09:08.759 INFO:tasks.workunit.client.1.smithi192.stdout:1/682: link d0/d2/d1e/d3f/d56/d9e/lc9 d0/d2/dd/d21/d33/dd8/le2 0 2024-04-12T05:09:08.760 INFO:tasks.workunit.client.0.smithi113.stdout:3/666: getdents d0/d13/d15/d28/d2a/d34 0 2024-04-12T05:09:08.762 INFO:tasks.workunit.client.0.smithi113.stdout:5/819: dread d0/ff5 [0,4194304] 0 2024-04-12T05:09:08.763 INFO:tasks.workunit.client.1.smithi192.stdout:4/754: creat d1/d54/d82/d9c/f106 x:0 0 0 2024-04-12T05:09:08.766 INFO:tasks.workunit.client.1.smithi192.stdout:5/689: dwrite d0/d15/d3e/f9d [0,4194304] 0 2024-04-12T05:09:08.767 INFO:tasks.workunit.client.1.smithi192.stdout:5/690: stat d0/d15/d3e/dca/f88 0 2024-04-12T05:09:08.767 INFO:tasks.workunit.client.0.smithi113.stdout:2/706: mkdir d8/d9/de/d43/dd6 0 2024-04-12T05:09:08.768 INFO:tasks.workunit.client.0.smithi113.stdout:6/723: mknod d8/de/dd5/de7/db3/c109 0 2024-04-12T05:09:08.770 INFO:tasks.workunit.client.0.smithi113.stdout:0/813: write d1/d6/d16/dd5/f37 [318790,107995] 0 2024-04-12T05:09:08.778 INFO:tasks.workunit.client.1.smithi192.stdout:4/755: symlink d1/d54/d82/d71/dff/l107 0 2024-04-12T05:09:08.778 INFO:tasks.workunit.client.1.smithi192.stdout:5/691: creat d0/d15/d3e/dca/dda/db5/dbe/ff1 x:0 0 0 2024-04-12T05:09:08.778 INFO:tasks.workunit.client.1.smithi192.stdout:2/680: write d1/d8/f41 [1122296,12335] 0 2024-04-12T05:09:08.779 INFO:tasks.workunit.client.1.smithi192.stdout:1/683: link d0/d2/d1e/d3f/d56/d9e/l75 d0/d2/dd/d3c/d5b/d74/d90/le3 0 2024-04-12T05:09:08.781 INFO:tasks.workunit.client.0.smithi113.stdout:7/725: dwrite d0/faf [0,4194304] 0 2024-04-12T05:09:08.782 INFO:tasks.workunit.client.1.smithi192.stdout:8/650: link d6/da/f3a d6/d10/d46/d5e/d5f/d37/d4f/fc6 0 2024-04-12T05:09:08.782 INFO:tasks.workunit.client.1.smithi192.stdout:2/681: dread d1/d8/d3a/f77 [0,4194304] 0 2024-04-12T05:09:08.783 INFO:tasks.workunit.client.0.smithi113.stdout:2/707: rmdir d8/d9/d27 39 2024-04-12T05:09:08.783 INFO:tasks.workunit.client.0.smithi113.stdout:3/667: creat d0/d13/d15/d28/d2a/d34/d5f/fe1 x:0 0 0 2024-04-12T05:09:08.783 INFO:tasks.workunit.client.1.smithi192.stdout:1/684: stat d0/d29/d8d/d96/caf 0 2024-04-12T05:09:08.785 INFO:tasks.workunit.client.1.smithi192.stdout:2/682: stat d1/d8/d17/d19/d60/d9c/f9d 0 2024-04-12T05:09:08.786 INFO:tasks.workunit.client.0.smithi113.stdout:5/820: write d0/d10/d27/d1d/d3f/d73/d89/fc1 [844103,36786] 0 2024-04-12T05:09:08.787 INFO:tasks.workunit.client.1.smithi192.stdout:5/692: rename d0/d15/f9b to d0/d15/d3e/dca/dda/db5/dd7/de1/ff2 0 2024-04-12T05:09:08.791 INFO:tasks.workunit.client.1.smithi192.stdout:8/651: unlink d6/d21/f44 0 2024-04-12T05:09:08.791 INFO:tasks.workunit.client.0.smithi113.stdout:8/772: dwrite d9/f9e [0,4194304] 0 2024-04-12T05:09:08.791 INFO:tasks.workunit.client.0.smithi113.stdout:4/735: dwrite da/d70/d80/f8b [0,4194304] 0 2024-04-12T05:09:08.792 INFO:tasks.workunit.client.1.smithi192.stdout:2/683: symlink d1/d8/d17/db3/ld5 0 2024-04-12T05:09:08.798 INFO:tasks.workunit.client.0.smithi113.stdout:5/821: creat d0/d10/d6e/d93/f119 x:0 0 0 2024-04-12T05:09:08.798 INFO:tasks.workunit.client.1.smithi192.stdout:2/684: mknod d1/d8/d3a/de/d7c/d98/cd6 0 2024-04-12T05:09:08.801 INFO:tasks.workunit.client.0.smithi113.stdout:9/726: dwrite d1/d4/d5/f12 [4194304,4194304] 0 2024-04-12T05:09:08.802 INFO:tasks.workunit.client.0.smithi113.stdout:8/773: creat d9/d16/d7f/d3a/d6c/d90/dac/ff6 x:0 0 0 2024-04-12T05:09:08.803 INFO:tasks.workunit.client.1.smithi192.stdout:8/652: creat d6/d10/d46/d5e/d5f/d37/fc7 x:0 0 0 2024-04-12T05:09:08.804 INFO:tasks.workunit.client.0.smithi113.stdout:3/668: mkdir d0/d13/d47/d94/dcf/dd1/de2 0 2024-04-12T05:09:08.805 INFO:tasks.workunit.client.0.smithi113.stdout:2/708: creat d8/d9/d27/d88/fd7 x:0 0 0 2024-04-12T05:09:08.807 INFO:tasks.workunit.client.1.smithi192.stdout:2/685: truncate d1/d8/d17/d19/d60/d9c/f9d 828040 0 2024-04-12T05:09:08.808 INFO:tasks.workunit.client.0.smithi113.stdout:4/736: creat da/d3c/d9a/dc1/dd9/ff7 x:0 0 0 2024-04-12T05:09:08.808 INFO:tasks.workunit.client.0.smithi113.stdout:8/774: write d9/d1b/d96/fe9 [674917,79291] 0 2024-04-12T05:09:08.808 INFO:tasks.workunit.client.1.smithi192.stdout:5/693: getdents d0/d15/da1 0 2024-04-12T05:09:08.810 INFO:tasks.workunit.client.0.smithi113.stdout:7/726: truncate d0/dc/da9/d5c/d85/fed 327744 0 2024-04-12T05:09:08.810 INFO:tasks.workunit.client.0.smithi113.stdout:3/669: chown d0/d13/d47/d4f/lc4 5325 1 2024-04-12T05:09:08.810 INFO:tasks.workunit.client.0.smithi113.stdout:4/737: readlink da/d24/d7e/l20 0 2024-04-12T05:09:08.810 INFO:tasks.workunit.client.0.smithi113.stdout:5/822: symlink d0/d10/d27/d1d/d38/d3b/d6b/l11a 0 2024-04-12T05:09:08.810 INFO:tasks.workunit.client.1.smithi192.stdout:5/694: write d0/d15/d32/d45/ff0 [476548,94583] 0 2024-04-12T05:09:08.811 INFO:tasks.workunit.client.0.smithi113.stdout:1/815: dread d10/d2e/d31/dc3/d109/fe7 [0,4194304] 0 2024-04-12T05:09:08.811 INFO:tasks.workunit.client.0.smithi113.stdout:9/727: symlink d1/lf6 0 2024-04-12T05:09:08.812 INFO:tasks.workunit.client.0.smithi113.stdout:9/728: chown d1 314 1 2024-04-12T05:09:08.812 INFO:tasks.workunit.client.1.smithi192.stdout:8/653: mknod d6/d10/d46/d5e/d5f/d31/d61/d9d/cc8 0 2024-04-12T05:09:08.813 INFO:tasks.workunit.client.1.smithi192.stdout:8/654: stat d6/d1f/d6a/f52 0 2024-04-12T05:09:08.815 INFO:tasks.workunit.client.1.smithi192.stdout:8/655: write d6/d10/d46/d5e/d5f/d38/d3b/f73 [4801144,105490] 0 2024-04-12T05:09:08.816 INFO:tasks.workunit.client.0.smithi113.stdout:8/775: read - d9/d16/d7f/d3a/d6c/d90/ff5 zero size 2024-04-12T05:09:08.816 INFO:tasks.workunit.client.1.smithi192.stdout:5/695: readlink d0/d15/d32/d69/d7d/le6 0 2024-04-12T05:09:08.816 INFO:tasks.workunit.client.0.smithi113.stdout:0/814: getdents d1 0 2024-04-12T05:09:08.817 INFO:tasks.workunit.client.1.smithi192.stdout:2/686: mknod d1/d8/d3a/de/cd7 0 2024-04-12T05:09:08.818 INFO:tasks.workunit.client.1.smithi192.stdout:5/696: truncate d0/d15/d24/ddf/fe8 276924 0 2024-04-12T05:09:08.818 INFO:tasks.workunit.client.0.smithi113.stdout:2/709: creat d8/d9/de/d12/dbf/fd8 x:0 0 0 2024-04-12T05:09:08.819 INFO:tasks.workunit.client.1.smithi192.stdout:8/656: readlink d6/d10/d46/d5e/d5f/d37/d4f/d51/l8a 0 2024-04-12T05:09:08.819 INFO:tasks.workunit.client.0.smithi113.stdout:3/670: write d0/d13/d15/fb4 [1044560,10286] 0 2024-04-12T05:09:08.819 INFO:tasks.workunit.client.0.smithi113.stdout:9/729: mkdir d1/df/d4b/df7 0 2024-04-12T05:09:08.820 INFO:tasks.workunit.client.1.smithi192.stdout:8/657: write d6/d21/f60 [5099130,60186] 0 2024-04-12T05:09:08.822 INFO:tasks.workunit.client.0.smithi113.stdout:8/776: write d9/d1b/d96/fd1 [515477,27181] 0 2024-04-12T05:09:08.823 INFO:tasks.workunit.client.0.smithi113.stdout:7/727: symlink d0/dc/d6/d1a/lf8 0 2024-04-12T05:09:08.824 INFO:tasks.workunit.client.1.smithi192.stdout:2/687: rename d1/d8/d17/db3/fcd to d1/d2d/d73/da8/db2/fd8 0 2024-04-12T05:09:08.825 INFO:tasks.workunit.client.0.smithi113.stdout:0/815: fsync d1/d6/d16/d2a/d50/d52/d7e/ff2 0 2024-04-12T05:09:08.825 INFO:tasks.workunit.client.0.smithi113.stdout:3/671: chown d0/d13/d15 8 1 2024-04-12T05:09:08.826 INFO:tasks.workunit.client.1.smithi192.stdout:5/697: dread d0/d15/d24/d47/f54 [0,4194304] 0 2024-04-12T05:09:08.826 INFO:tasks.workunit.client.0.smithi113.stdout:4/738: getdents da/d3c/da5/dad/de1 0 2024-04-12T05:09:08.827 INFO:tasks.workunit.client.0.smithi113.stdout:8/777: rmdir d9/d1b/d96 39 2024-04-12T05:09:08.828 INFO:tasks.workunit.client.0.smithi113.stdout:1/816: mknod d10/d2e/d7c/dc5/dc8/dad/c10d 0 2024-04-12T05:09:08.829 INFO:tasks.workunit.client.0.smithi113.stdout:5/823: link d0/d10/d27/d1d/d38/d44/dcb/fe1 d0/f11b 0 2024-04-12T05:09:08.829 INFO:tasks.workunit.client.0.smithi113.stdout:4/739: dread - da/d70/fd3 zero size 2024-04-12T05:09:08.829 INFO:tasks.workunit.client.0.smithi113.stdout:6/724: dread d8/de/d17/f78 [0,4194304] 0 2024-04-12T05:09:08.829 INFO:tasks.workunit.client.1.smithi192.stdout:8/658: mkdir d6/d1f/d5d/dc9 0 2024-04-12T05:09:08.829 INFO:tasks.workunit.client.1.smithi192.stdout:8/659: stat d6/d10/lc4 0 2024-04-12T05:09:08.830 INFO:tasks.workunit.client.1.smithi192.stdout:8/660: dread - d6/da/fb7 zero size 2024-04-12T05:09:08.831 INFO:tasks.workunit.client.1.smithi192.stdout:5/698: dread d0/d15/d32/fef [0,4194304] 0 2024-04-12T05:09:08.833 INFO:tasks.workunit.client.0.smithi113.stdout:1/817: write d10/d14/d95/dcf/ffa [1008958,89469] 0 2024-04-12T05:09:08.834 INFO:tasks.workunit.client.1.smithi192.stdout:8/661: mknod d6/d10/d46/d5e/d5f/d31/d80/db1/cca 0 2024-04-12T05:09:08.835 INFO:tasks.workunit.client.1.smithi192.stdout:5/699: mkdir d0/d15/d3e/dca/dd3/df3 0 2024-04-12T05:09:08.835 INFO:tasks.workunit.client.0.smithi113.stdout:7/728: dread d0/dc/d6/d24/f71 [0,4194304] 0 2024-04-12T05:09:08.839 INFO:tasks.workunit.client.0.smithi113.stdout:1/818: mkdir d10/d55/d10e 0 2024-04-12T05:09:08.840 INFO:tasks.workunit.client.0.smithi113.stdout:2/710: dwrite d8/d9/de/d15/d24/d58/f84 [0,4194304] 0 2024-04-12T05:09:08.842 INFO:tasks.workunit.client.0.smithi113.stdout:1/819: read d10/d41/d5b/fa3 [33593,54703] 0 2024-04-12T05:09:08.844 INFO:tasks.workunit.client.0.smithi113.stdout:4/740: dwrite da/d3c/d9f/fba [0,4194304] 0 2024-04-12T05:09:08.844 INFO:tasks.workunit.client.0.smithi113.stdout:2/711: truncate d8/d9/de/fd5 279841 0 2024-04-12T05:09:08.845 INFO:tasks.workunit.client.0.smithi113.stdout:2/712: chown d8/d9/d27 103003693 1 2024-04-12T05:09:08.852 INFO:tasks.workunit.client.1.smithi192.stdout:0/717: sync 2024-04-12T05:09:08.852 INFO:tasks.workunit.client.1.smithi192.stdout:3/646: sync 2024-04-12T05:09:08.855 INFO:tasks.workunit.client.0.smithi113.stdout:1/820: write d10/d41/d89/d96/d7d/fb2 [635840,70936] 0 2024-04-12T05:09:08.856 INFO:tasks.workunit.client.0.smithi113.stdout:1/821: chown d10/d41/d89/d96/f10b 1920521 1 2024-04-12T05:09:08.860 INFO:tasks.workunit.client.1.smithi192.stdout:5/700: read - d0/d15/d3e/dca/dda/f98 zero size 2024-04-12T05:09:08.860 INFO:tasks.workunit.client.1.smithi192.stdout:8/662: rename d6/d10/d46/d5e/d6e/fa9 to d6/d10/d46/d5e/d5f/d31/d80/db1/fcb 0 2024-04-12T05:09:08.860 INFO:tasks.workunit.client.0.smithi113.stdout:1/822: truncate d10/d55/f8c 1589688 0 2024-04-12T05:09:08.861 INFO:tasks.workunit.client.0.smithi113.stdout:1/823: chown d10/d55/ffc 324 1 2024-04-12T05:09:08.861 INFO:tasks.workunit.client.1.smithi192.stdout:0/718: dread - d0/d6/d15/d55/d2e/d66/f70 zero size 2024-04-12T05:09:08.861 INFO:tasks.workunit.client.1.smithi192.stdout:5/701: chown d0/d15/d24/d29/f85 17 1 2024-04-12T05:09:08.862 INFO:tasks.workunit.client.0.smithi113.stdout:0/816: rmdir d1/d80/dcf 0 2024-04-12T05:09:08.863 INFO:tasks.workunit.client.1.smithi192.stdout:2/688: dread d1/d8/d17/d19/d60/f66 [0,4194304] 0 2024-04-12T05:09:08.863 INFO:tasks.workunit.client.0.smithi113.stdout:5/824: read d0/f3a [1733630,53568] 0 2024-04-12T05:09:08.864 INFO:tasks.workunit.client.0.smithi113.stdout:1/824: creat d10/d2e/dd8/d70/f10f x:0 0 0 2024-04-12T05:09:08.864 INFO:tasks.workunit.client.0.smithi113.stdout:3/672: creat d0/dc6/fe3 x:0 0 0 2024-04-12T05:09:08.865 INFO:tasks.workunit.client.0.smithi113.stdout:6/725: mknod d8/de/d13/df0/d66/dc4/c10a 0 2024-04-12T05:09:08.867 INFO:tasks.workunit.client.1.smithi192.stdout:5/702: creat d0/d15/d32/d69/d7d/ff4 x:0 0 0 2024-04-12T05:09:08.870 INFO:tasks.workunit.client.0.smithi113.stdout:1/825: read d10/d2e/d7c/f92 [18912,88031] 0 2024-04-12T05:09:08.870 INFO:tasks.workunit.client.0.smithi113.stdout:0/817: fsync d1/d6/d16/d2a/d50/f7d 0 2024-04-12T05:09:08.872 INFO:tasks.workunit.client.0.smithi113.stdout:9/730: getdents d1/d4/d5/d6 0 2024-04-12T05:09:08.876 INFO:tasks.workunit.client.0.smithi113.stdout:3/673: dread d0/d13/d47/f78 [0,4194304] 0 2024-04-12T05:09:08.876 INFO:tasks.workunit.client.0.smithi113.stdout:8/778: dwrite d9/d16/d65/db1/d59/da3/fe0 [0,4194304] 0 2024-04-12T05:09:08.877 INFO:tasks.workunit.client.1.smithi192.stdout:2/689: rename d1/d8/d17/fcc to d1/d8/d17/d19/d60/d59/fd9 0 2024-04-12T05:09:08.879 INFO:tasks.workunit.client.0.smithi113.stdout:2/713: mkdir d8/d9/d27/dd9 0 2024-04-12T05:09:08.879 INFO:tasks.workunit.client.0.smithi113.stdout:4/741: getdents da/d4a/da2 0 2024-04-12T05:09:08.881 INFO:tasks.workunit.client.0.smithi113.stdout:1/826: symlink d10/d2e/d31/dc3/l110 0 2024-04-12T05:09:08.883 INFO:tasks.workunit.client.1.smithi192.stdout:8/663: dwrite d6/d10/d46/d5e/d5f/d37/d4f/fc6 [4194304,4194304] 0 2024-04-12T05:09:08.883 INFO:tasks.workunit.client.1.smithi192.stdout:0/719: dwrite d0/d7b/d9b/d4b/dbc/dce/f87 [4194304,4194304] 0 2024-04-12T05:09:08.883 INFO:tasks.workunit.client.1.smithi192.stdout:2/690: read d1/d8/d17/d19/d60/fa6 [507770,32165] 0 2024-04-12T05:09:08.884 INFO:tasks.workunit.client.1.smithi192.stdout:3/647: dread d5/d13/d1a/d53/f60 [0,4194304] 0 2024-04-12T05:09:08.884 INFO:tasks.workunit.client.0.smithi113.stdout:9/731: rename d1/df/d4b/f89 to d1/d4/d87/ff8 0 2024-04-12T05:09:08.886 INFO:tasks.workunit.client.1.smithi192.stdout:0/720: dread - d0/fa7 zero size 2024-04-12T05:09:08.886 INFO:tasks.workunit.client.1.smithi192.stdout:5/703: getdents d0/d15/d3e/dca/dda/db5/dd7 0 2024-04-12T05:09:08.889 INFO:tasks.workunit.client.0.smithi113.stdout:0/818: unlink d1/d6/d16/d2a/d50/cd0 0 2024-04-12T05:09:08.889 INFO:tasks.workunit.client.0.smithi113.stdout:1/827: chown d10/d2e/d31/dc3/d109/fe7 4235 1 2024-04-12T05:09:08.892 INFO:tasks.workunit.client.0.smithi113.stdout:1/828: chown d10/d2e/d31/d5c/l8a 18 1 2024-04-12T05:09:08.893 INFO:tasks.workunit.client.0.smithi113.stdout:6/726: dwrite d8/de/d17/f78 [0,4194304] 0 2024-04-12T05:09:08.893 INFO:tasks.workunit.client.1.smithi192.stdout:3/648: truncate d5/d13/d1a/d5f/fa1 248771 0 2024-04-12T05:09:08.895 INFO:tasks.workunit.client.1.smithi192.stdout:2/691: fdatasync d1/d2d/fd1 0 2024-04-12T05:09:08.896 INFO:tasks.workunit.client.0.smithi113.stdout:2/714: rename d8/d9/de/d15/d38 to d8/d9/d27/dda 0 2024-04-12T05:09:08.896 INFO:tasks.workunit.client.1.smithi192.stdout:0/721: fsync d0/d6/d15/d19/f36 0 2024-04-12T05:09:08.899 INFO:tasks.workunit.client.1.smithi192.stdout:0/722: chown d0/d6/d67/dd2 13895523 1 2024-04-12T05:09:08.899 INFO:tasks.workunit.client.0.smithi113.stdout:1/829: chown d10/da5/lb4 392 1 2024-04-12T05:09:08.899 INFO:tasks.workunit.client.0.smithi113.stdout:6/727: fsync d8/de/d13/fa 0 2024-04-12T05:09:08.900 INFO:tasks.workunit.client.1.smithi192.stdout:8/664: dwrite d6/da/f30 [0,4194304] 0 2024-04-12T05:09:08.900 INFO:tasks.workunit.client.0.smithi113.stdout:1/830: dread - d10/d2e/d31/dc3/fd5 zero size 2024-04-12T05:09:08.901 INFO:tasks.workunit.client.0.smithi113.stdout:4/742: link da/d73/f71 da/d3c/d9a/ff8 0 2024-04-12T05:09:08.903 INFO:tasks.workunit.client.0.smithi113.stdout:1/831: write d10/d2e/d31/dc3/fd5 [492101,120430] 0 2024-04-12T05:09:08.905 INFO:tasks.workunit.client.0.smithi113.stdout:2/715: unlink d8/d9/de/d43/l47 0 2024-04-12T05:09:08.905 INFO:tasks.workunit.client.0.smithi113.stdout:4/743: symlink da/d3c/da5/dad/lf9 0 2024-04-12T05:09:08.906 INFO:tasks.workunit.client.1.smithi192.stdout:3/649: mknod d5/d13/d3c/d26/ce6 0 2024-04-12T05:09:08.907 INFO:tasks.workunit.client.1.smithi192.stdout:3/650: fdatasync d5/f2d 0 2024-04-12T05:09:08.909 INFO:tasks.workunit.client.0.smithi113.stdout:8/779: truncate d9/d1b/d96/fe9 513836 0 2024-04-12T05:09:08.909 INFO:tasks.workunit.client.1.smithi192.stdout:2/692: creat d1/d2d/d73/da8/db2/fda x:0 0 0 2024-04-12T05:09:08.910 INFO:tasks.workunit.client.0.smithi113.stdout:5/825: dread d0/d10/d27/d1d/f1e [0,4194304] 0 2024-04-12T05:09:08.913 INFO:tasks.workunit.client.1.smithi192.stdout:8/665: creat d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/fcc x:0 0 0 2024-04-12T05:09:08.914 INFO:tasks.workunit.client.0.smithi113.stdout:6/728: rename d8/de/d17/f7c to d8/de/d13/d11/d36/d3d/d4c/d31/f10b 0 2024-04-12T05:09:08.916 INFO:tasks.workunit.client.0.smithi113.stdout:2/716: fsync d8/d9/de/d15/f29 0 2024-04-12T05:09:08.916 INFO:tasks.workunit.client.1.smithi192.stdout:3/651: dwrite d5/d13/d1a/d5f/fa1 [0,4194304] 0 2024-04-12T05:09:08.919 INFO:tasks.workunit.client.1.smithi192.stdout:2/693: mknod d1/d8/d17/d19/d71/cdb 0 2024-04-12T05:09:08.920 INFO:tasks.workunit.client.0.smithi113.stdout:6/729: chown d8/de/dd5/de7/db3/dc5/cc6 61331 1 2024-04-12T05:09:08.922 INFO:tasks.workunit.client.0.smithi113.stdout:9/732: dwrite d1/d4/d5/f58 [4194304,4194304] 0 2024-04-12T05:09:08.925 INFO:tasks.workunit.client.1.smithi192.stdout:8/666: rmdir d6/d10/d6c 39 2024-04-12T05:09:08.925 INFO:tasks.workunit.client.0.smithi113.stdout:5/826: mkdir d0/d10/d27/d1d/d38/d44/d4e/daa/dbb/d11c 0 2024-04-12T05:09:08.927 INFO:tasks.workunit.client.1.smithi192.stdout:2/694: chown d1/d8/d17/d19/l5c 375811 1 2024-04-12T05:09:08.927 INFO:tasks.workunit.client.1.smithi192.stdout:3/652: dread d5/d13/d1a/d53/f60 [0,4194304] 0 2024-04-12T05:09:08.928 INFO:tasks.workunit.client.1.smithi192.stdout:5/704: truncate d0/f1b 3673348 0 2024-04-12T05:09:08.928 INFO:tasks.workunit.client.0.smithi113.stdout:2/717: rename d8/d9/de/d12/d3c/fcb to d8/d9/d27/dd9/fdb 0 2024-04-12T05:09:08.931 INFO:tasks.workunit.client.0.smithi113.stdout:2/718: chown d8/d9/de/d15/d24 5 1 2024-04-12T05:09:08.931 INFO:tasks.workunit.client.0.smithi113.stdout:6/730: creat d8/de/dd5/f10c x:0 0 0 2024-04-12T05:09:08.933 INFO:tasks.workunit.client.1.smithi192.stdout:2/695: symlink d1/d8/d3a/de/d7c/d98/ldc 0 2024-04-12T05:09:08.933 INFO:tasks.workunit.client.0.smithi113.stdout:3/674: dread d0/d13/d15/f3d [4194304,4194304] 0 2024-04-12T05:09:08.933 INFO:tasks.workunit.client.0.smithi113.stdout:3/675: chown d0/d13 893655 1 2024-04-12T05:09:08.935 INFO:tasks.workunit.client.0.smithi113.stdout:8/780: rmdir d9/d16/d65/db1/df2 0 2024-04-12T05:09:08.937 INFO:tasks.workunit.client.0.smithi113.stdout:3/676: fsync d0/d13/d15/d28/d2a/d34/d5f/f96 0 2024-04-12T05:09:08.937 INFO:tasks.workunit.client.0.smithi113.stdout:5/827: unlink d0/d10/d27/d1d/d8c/fa7 0 2024-04-12T05:09:08.937 INFO:tasks.workunit.client.0.smithi113.stdout:4/744: write da/dd/f6e [627850,115740] 0 2024-04-12T05:09:08.938 INFO:tasks.workunit.client.1.smithi192.stdout:5/705: creat d0/d15/d3e/dca/ff5 x:0 0 0 2024-04-12T05:09:08.939 INFO:tasks.workunit.client.0.smithi113.stdout:2/719: symlink d8/d9/de/d12/d2b/daa/ldc 0 2024-04-12T05:09:08.940 INFO:tasks.workunit.client.0.smithi113.stdout:1/832: dread d10/d41/d89/d96/dd3/f50 [0,4194304] 0 2024-04-12T05:09:08.943 INFO:tasks.workunit.client.0.smithi113.stdout:6/731: mkdir d8/de/d13/d11/d36/d89/d10d 0 2024-04-12T05:09:08.943 INFO:tasks.workunit.client.1.smithi192.stdout:2/696: chown d1/d8/d17/d19/d60/d9c/lb9 1460657624 1 2024-04-12T05:09:08.944 INFO:tasks.workunit.client.1.smithi192.stdout:3/653: unlink d5/d13/d1a/f3b 0 2024-04-12T05:09:08.945 INFO:tasks.workunit.client.1.smithi192.stdout:0/723: dwrite d0/d6/d15/d55/d2e/d66/f70 [0,4194304] 0 2024-04-12T05:09:08.946 INFO:tasks.workunit.client.0.smithi113.stdout:0/819: dread d1/d6/f30 [0,4194304] 0 2024-04-12T05:09:08.947 INFO:tasks.workunit.client.1.smithi192.stdout:2/697: chown d1/d8/d3a/de/d7c/d98/ldc 112840 1 2024-04-12T05:09:08.949 INFO:tasks.workunit.client.0.smithi113.stdout:4/745: symlink da/d3c/da5/lfa 0 2024-04-12T05:09:08.950 INFO:tasks.workunit.client.0.smithi113.stdout:5/828: dread d0/d10/d6e/f4a [0,4194304] 0 2024-04-12T05:09:08.952 INFO:tasks.workunit.client.0.smithi113.stdout:9/733: getdents d1/d27/d7a/de4 0 2024-04-12T05:09:08.954 INFO:tasks.workunit.client.1.smithi192.stdout:8/667: write d6/d10/d46/d5e/d5f/f1c [214507,72404] 0 2024-04-12T05:09:08.954 INFO:tasks.workunit.client.0.smithi113.stdout:4/746: chown da/d24/d7e/ff1 2897252 1 2024-04-12T05:09:08.955 INFO:tasks.workunit.client.1.smithi192.stdout:3/654: unlink d5/d13/d3a/d8b/da2/da7/le3 0 2024-04-12T05:09:08.956 INFO:tasks.workunit.client.1.smithi192.stdout:0/724: symlink d0/d6/d15/d1b/lef 0 2024-04-12T05:09:08.957 INFO:tasks.workunit.client.1.smithi192.stdout:3/655: dread - d5/d13/d1a/d5f/f75 zero size 2024-04-12T05:09:08.957 INFO:tasks.workunit.client.1.smithi192.stdout:2/698: dread d1/d2d/f34 [0,4194304] 0 2024-04-12T05:09:08.958 INFO:tasks.workunit.client.0.smithi113.stdout:8/781: dwrite d9/d16/fd3 [0,4194304] 0 2024-04-12T05:09:08.958 INFO:tasks.workunit.client.0.smithi113.stdout:4/747: dread - da/fef zero size 2024-04-12T05:09:08.958 INFO:tasks.workunit.client.1.smithi192.stdout:3/656: chown d5/f55 1143 1 2024-04-12T05:09:08.961 INFO:tasks.workunit.client.1.smithi192.stdout:8/668: unlink d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/f75 0 2024-04-12T05:09:08.962 INFO:tasks.workunit.client.0.smithi113.stdout:2/720: dread d8/d9/de/d15/d24/d58/f84 [0,4194304] 0 2024-04-12T05:09:08.964 INFO:tasks.workunit.client.1.smithi192.stdout:0/725: mknod d0/d7b/d7c/cf0 0 2024-04-12T05:09:08.965 INFO:tasks.workunit.client.1.smithi192.stdout:2/699: mkdir d1/d8/d3a/de/d7c/ddd 0 2024-04-12T05:09:08.967 INFO:tasks.workunit.client.0.smithi113.stdout:8/782: read d9/d1b/d8e/d93/d32/d5b/f63 [1878378,48158] 0 2024-04-12T05:09:08.968 INFO:tasks.workunit.client.0.smithi113.stdout:3/677: dread d0/d13/d15/d28/f74 [0,4194304] 0 2024-04-12T05:09:08.968 INFO:tasks.workunit.client.1.smithi192.stdout:3/657: symlink d5/d13/d3a/d70/d8c/de1/le7 0 2024-04-12T05:09:08.969 INFO:tasks.workunit.client.0.smithi113.stdout:6/732: write d8/de/d13/df0/d66/dc4/fe6 [2341383,81793] 0 2024-04-12T05:09:08.971 INFO:tasks.workunit.client.1.smithi192.stdout:0/726: mknod d0/d6/d67/cf1 0 2024-04-12T05:09:08.972 INFO:tasks.workunit.client.0.smithi113.stdout:2/721: mkdir d8/d9/de/d12/d2b/ddd 0 2024-04-12T05:09:08.972 INFO:tasks.workunit.client.0.smithi113.stdout:9/734: dwrite d1/df/d29/da6/fdb [0,4194304] 0 2024-04-12T05:09:08.974 INFO:tasks.workunit.client.0.smithi113.stdout:6/733: chown d8/de/d13/d11/ca1 5104 1 2024-04-12T05:09:08.974 INFO:tasks.workunit.client.0.smithi113.stdout:9/735: chown d1/d4/d5/d6/cae 5014 1 2024-04-12T05:09:08.974 INFO:tasks.workunit.client.0.smithi113.stdout:3/678: read - d0/d13/d15/d28/d2a/d34/d5f/fe1 zero size 2024-04-12T05:09:08.977 INFO:tasks.workunit.client.1.smithi192.stdout:3/658: readlink d5/d13/d3c/d36/lc1 0 2024-04-12T05:09:08.979 INFO:tasks.workunit.client.1.smithi192.stdout:5/706: dwrite d0/d15/d24/f4b [0,4194304] 0 2024-04-12T05:09:08.980 INFO:tasks.workunit.client.0.smithi113.stdout:8/783: rmdir d9/d1b/d8e/d93/d32/d48/dc1 39 2024-04-12T05:09:08.981 INFO:tasks.workunit.client.1.smithi192.stdout:0/727: fdatasync d0/d6/d15/d19/f71 0 2024-04-12T05:09:08.982 INFO:tasks.workunit.client.0.smithi113.stdout:1/833: write d10/d41/d89/d96/f49 [897659,2840] 0 2024-04-12T05:09:08.982 INFO:tasks.workunit.client.0.smithi113.stdout:2/722: dread d8/d9/f56 [0,4194304] 0 2024-04-12T05:09:08.982 INFO:tasks.workunit.client.0.smithi113.stdout:3/679: read - d0/d13/d47/f89 zero size 2024-04-12T05:09:08.983 INFO:tasks.workunit.client.0.smithi113.stdout:6/734: unlink d8/de/d13/d11/d36/d3d/d4c/d26/d64/f8c 0 2024-04-12T05:09:08.983 INFO:tasks.workunit.client.0.smithi113.stdout:0/820: write d1/fc9 [1785125,58336] 0 2024-04-12T05:09:08.984 INFO:tasks.workunit.client.0.smithi113.stdout:8/784: chown d9/l14 6703 1 2024-04-12T05:09:08.984 INFO:tasks.workunit.client.0.smithi113.stdout:9/736: mkdir d1/d4/df9 0 2024-04-12T05:09:08.991 INFO:tasks.workunit.client.0.smithi113.stdout:8/785: creat d9/d16/d65/db1/d59/d78/ff7 x:0 0 0 2024-04-12T05:09:08.992 INFO:tasks.workunit.client.0.smithi113.stdout:3/680: creat d0/d13/d15/d28/d2a/d8f/ddd/fe4 x:0 0 0 2024-04-12T05:09:08.992 INFO:tasks.workunit.client.1.smithi192.stdout:8/669: link d6/d1f/d5d/f8d d6/d10/d46/d5e/d5f/d31/d80/fcd 0 2024-04-12T05:09:08.992 INFO:tasks.workunit.client.1.smithi192.stdout:3/659: mknod d5/d13/d3c/d84/ce8 0 2024-04-12T05:09:08.993 INFO:tasks.workunit.client.1.smithi192.stdout:3/660: fdatasync d5/d13/d3c/d36/da0/fb7 0 2024-04-12T05:09:08.993 INFO:tasks.workunit.client.0.smithi113.stdout:1/834: symlink d10/d41/d89/d96/d7d/d9f/l111 0 2024-04-12T05:09:08.994 INFO:tasks.workunit.client.1.smithi192.stdout:5/707: chown d0/d15/d32/d45/f60 770 1 2024-04-12T05:09:08.995 INFO:tasks.workunit.client.0.smithi113.stdout:0/821: truncate d1/fe6 503976 0 2024-04-12T05:09:08.996 INFO:tasks.workunit.client.0.smithi113.stdout:6/735: write d8/d45/ff3 [1289845,22074] 0 2024-04-12T05:09:08.997 INFO:tasks.workunit.client.1.smithi192.stdout:2/700: truncate d1/d2d/f5d 3536569 0 2024-04-12T05:09:08.997 INFO:tasks.workunit.client.0.smithi113.stdout:8/786: unlink d9/d16/d7f/d3a/d6c/d90/lce 0 2024-04-12T05:09:08.997 INFO:tasks.workunit.client.1.smithi192.stdout:3/661: dread d5/d13/d3c/f91 [0,4194304] 0 2024-04-12T05:09:08.998 INFO:tasks.workunit.client.1.smithi192.stdout:2/701: chown d1/d8/d3a/de/l18 272159 1 2024-04-12T05:09:08.999 INFO:tasks.workunit.client.0.smithi113.stdout:8/787: dread - d9/d1b/d96/fe7 zero size 2024-04-12T05:09:08.999 INFO:tasks.workunit.client.0.smithi113.stdout:5/829: dread d0/d10/d27/d1d/d38/d3b/d6b/fd4 [0,4194304] 0 2024-04-12T05:09:09.000 INFO:tasks.workunit.client.0.smithi113.stdout:2/723: rename d8/d9/d27/c4f to d8/d9/d27/d88/d91/cde 0 2024-04-12T05:09:09.000 INFO:tasks.workunit.client.0.smithi113.stdout:6/736: chown d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8d 10951327 1 2024-04-12T05:09:09.001 INFO:tasks.workunit.client.0.smithi113.stdout:5/830: chown d0/d10/d27/d1d/d3f 11525145 1 2024-04-12T05:09:09.003 INFO:tasks.workunit.client.1.smithi192.stdout:8/670: rmdir d6/d10/d46/d5e/d5f/d31 39 2024-04-12T05:09:09.004 INFO:tasks.workunit.client.1.smithi192.stdout:8/671: fdatasync d6/d1f/f5c 0 2024-04-12T05:09:09.006 INFO:tasks.workunit.client.0.smithi113.stdout:1/835: unlink d10/d41/lbf 0 2024-04-12T05:09:09.008 INFO:tasks.workunit.client.0.smithi113.stdout:2/724: write d8/d9/de/f31 [2113411,46815] 0 2024-04-12T05:09:09.013 INFO:tasks.workunit.client.1.smithi192.stdout:5/708: unlink d0/d15/d24/c3a 0 2024-04-12T05:09:09.013 INFO:tasks.workunit.client.1.smithi192.stdout:5/709: dread - d0/d15/d3e/fae zero size 2024-04-12T05:09:09.014 INFO:tasks.workunit.client.1.smithi192.stdout:3/662: mknod d5/d13/d3a/d4a/ce9 0 2024-04-12T05:09:09.016 INFO:tasks.workunit.client.1.smithi192.stdout:0/728: unlink d0/d6/d15/d1b/l77 0 2024-04-12T05:09:09.016 INFO:tasks.workunit.client.0.smithi113.stdout:9/737: dwrite d1/f46 [0,4194304] 0 2024-04-12T05:09:09.017 INFO:tasks.workunit.client.0.smithi113.stdout:3/681: dwrite d0/d13/f42 [4194304,4194304] 0 2024-04-12T05:09:09.018 INFO:tasks.workunit.client.1.smithi192.stdout:0/729: fsync d0/d6/d15/d55/d2e/d59/f78 0 2024-04-12T05:09:09.019 INFO:tasks.workunit.client.1.smithi192.stdout:0/730: fsync d0/d6/d15/d55/d2e/d59/f78 0 2024-04-12T05:09:09.020 INFO:tasks.workunit.client.0.smithi113.stdout:1/836: mkdir d10/d2e/d7c/dc5/dc8/d112 0 2024-04-12T05:09:09.022 INFO:tasks.workunit.client.0.smithi113.stdout:0/822: write d1/d6/d16/d8f/f9e [863289,95038] 0 2024-04-12T05:09:09.026 INFO:tasks.workunit.client.0.smithi113.stdout:1/837: readlink d10/d55/l6d 0 2024-04-12T05:09:09.030 INFO:tasks.workunit.client.0.smithi113.stdout:0/823: dread d1/d80/f84 [0,4194304] 0 2024-04-12T05:09:09.035 INFO:tasks.workunit.client.0.smithi113.stdout:0/824: symlink d1/d6/d16/d2a/d50/lfd 0 2024-04-12T05:09:09.039 INFO:tasks.workunit.client.1.smithi192.stdout:0/731: fdatasync d0/fd0 0 2024-04-12T05:09:09.039 INFO:tasks.workunit.client.1.smithi192.stdout:2/702: dwrite d1/d8/d17/d19/d60/d59/fd9 [0,4194304] 0 2024-04-12T05:09:09.039 INFO:tasks.workunit.client.1.smithi192.stdout:9/567: sync 2024-04-12T05:09:09.039 INFO:tasks.workunit.client.1.smithi192.stdout:7/614: sync 2024-04-12T05:09:09.039 INFO:tasks.workunit.client.1.smithi192.stdout:6/723: sync 2024-04-12T05:09:09.041 INFO:tasks.workunit.client.1.smithi192.stdout:3/663: link d5/d13/d3a/d70/d8c/d8d/f93 d5/d13/d3a/d70/d8c/d8d/dbc/fea 0 2024-04-12T05:09:09.041 INFO:tasks.workunit.client.0.smithi113.stdout:0/825: dread d1/d6/fb0 [0,4194304] 0 2024-04-12T05:09:09.043 INFO:tasks.workunit.client.1.smithi192.stdout:0/732: mkdir d0/d6/d67/db1/dd6/df2 0 2024-04-12T05:09:09.046 INFO:tasks.workunit.client.1.smithi192.stdout:7/615: mknod d3/d7/d15/d1a/cd6 0 2024-04-12T05:09:09.046 INFO:tasks.workunit.client.1.smithi192.stdout:6/724: creat d3/d8/d56/d59/d97/dbf/ff0 x:0 0 0 2024-04-12T05:09:09.050 INFO:tasks.workunit.client.0.smithi113.stdout:0/826: mknod d1/d6/d16/cfe 0 2024-04-12T05:09:09.051 INFO:tasks.workunit.client.1.smithi192.stdout:8/672: write d6/d10/f78 [1659520,26342] 0 2024-04-12T05:09:09.051 INFO:tasks.workunit.client.0.smithi113.stdout:0/827: write d1/d80/ff9 [727865,63292] 0 2024-04-12T05:09:09.052 INFO:tasks.workunit.client.1.smithi192.stdout:5/710: dread d0/d21/f36 [0,4194304] 0 2024-04-12T05:09:09.052 INFO:tasks.workunit.client.1.smithi192.stdout:8/673: stat d6/d10/d46/d5e/d5f/d37/fc7 0 2024-04-12T05:09:09.055 INFO:tasks.workunit.client.1.smithi192.stdout:3/664: mkdir d5/d13/d3a/d4a/d61/d9f/deb 0 2024-04-12T05:09:09.056 INFO:tasks.workunit.client.0.smithi113.stdout:6/737: creat d8/de/d13/d11/d36/d3d/d4c/d26/d108/f10e x:0 0 0 2024-04-12T05:09:09.058 INFO:tasks.workunit.client.0.smithi113.stdout:6/738: dread - d8/de/d13/d11/d36/d3d/d4c/d26/d5b/ff8 zero size 2024-04-12T05:09:09.061 INFO:tasks.workunit.client.0.smithi113.stdout:2/725: dwrite d8/d9/d27/d88/f8b [0,4194304] 0 2024-04-12T05:09:09.061 INFO:tasks.workunit.client.1.smithi192.stdout:0/733: rename d0/d6/d15/d55 to d0/d7b/d9b/d4b/dbc/dce/d53/df3 0 2024-04-12T05:09:09.063 INFO:tasks.workunit.client.0.smithi113.stdout:3/682: fsync d0/d13/d15/f75 0 2024-04-12T05:09:09.065 INFO:tasks.workunit.client.1.smithi192.stdout:0/734: fdatasync d0/d7b/d9b/d4b/dbc/dce/d95/fed 0 2024-04-12T05:09:09.065 INFO:tasks.workunit.client.0.smithi113.stdout:3/683: fsync d0/d13/d15/d28/d2a/d34/f72 0 2024-04-12T05:09:09.066 INFO:tasks.workunit.client.0.smithi113.stdout:8/788: creat d9/d1b/ff8 x:0 0 0 2024-04-12T05:09:09.067 INFO:tasks.workunit.client.0.smithi113.stdout:1/838: write d10/d41/d5b/fb0 [1214192,77003] 0 2024-04-12T05:09:09.070 INFO:tasks.workunit.client.0.smithi113.stdout:6/739: creat d8/de/d13/d11/d36/d53/db8/dce/f10f x:0 0 0 2024-04-12T05:09:09.072 INFO:tasks.workunit.client.0.smithi113.stdout:8/789: dread d9/d16/fd3 [0,4194304] 0 2024-04-12T05:09:09.072 INFO:tasks.workunit.client.1.smithi192.stdout:2/703: write d1/d8/d17/d19/d60/d9c/f9f [1020713,26617] 0 2024-04-12T05:09:09.073 INFO:tasks.workunit.client.0.smithi113.stdout:8/790: chown d9/d1b/d8e/d93/d92 3267 1 2024-04-12T05:09:09.075 INFO:tasks.workunit.client.0.smithi113.stdout:0/828: unlink d1/le4 0 2024-04-12T05:09:09.077 INFO:tasks.workunit.client.0.smithi113.stdout:2/726: mkdir d8/d9/de/d12/d2b/d79/ddf 0 2024-04-12T05:09:09.077 INFO:tasks.workunit.client.0.smithi113.stdout:9/738: creat d1/d4/d5/d6/ffa x:0 0 0 2024-04-12T05:09:09.079 INFO:tasks.workunit.client.0.smithi113.stdout:3/684: unlink d0/d13/d15/d28/d2a/d34/d5f/c93 0 2024-04-12T05:09:09.080 INFO:tasks.workunit.client.1.smithi192.stdout:9/568: getdents d11/d15 0 2024-04-12T05:09:09.081 INFO:tasks.workunit.client.0.smithi113.stdout:0/829: readlink d1/d6/d16/d2a/l49 0 2024-04-12T05:09:09.083 INFO:tasks.workunit.client.0.smithi113.stdout:2/727: mkdir d8/d9/d27/d88/de0 0 2024-04-12T05:09:09.086 INFO:tasks.workunit.client.0.smithi113.stdout:9/739: creat d1/d27/d7a/de4/db7/ffb x:0 0 0 2024-04-12T05:09:09.087 INFO:tasks.workunit.client.0.smithi113.stdout:1/839: write d10/d2e/f4e [2903878,36448] 0 2024-04-12T05:09:09.088 INFO:tasks.workunit.client.0.smithi113.stdout:0/830: dread d1/d6/d34/dc3/f1a [0,4194304] 0 2024-04-12T05:09:09.090 INFO:tasks.workunit.client.0.smithi113.stdout:2/728: creat d8/d9/de/d15/d24/d58/fe1 x:0 0 0 2024-04-12T05:09:09.091 INFO:tasks.workunit.client.1.smithi192.stdout:7/616: link d3/d7/d14/f92 d3/d7/d15/d1a/fd7 0 2024-04-12T05:09:09.092 INFO:tasks.workunit.client.0.smithi113.stdout:0/831: read d1/d6/d16/f1c [593695,3745] 0 2024-04-12T05:09:09.092 INFO:tasks.workunit.client.0.smithi113.stdout:8/791: link d9/d16/d7f/d3a/c44 d9/d1b/d96/cf9 0 2024-04-12T05:09:09.093 INFO:tasks.workunit.client.1.smithi192.stdout:8/674: creat d6/d10/d46/d5e/d5f/d31/d61/d9d/db8/fce x:0 0 0 2024-04-12T05:09:09.094 INFO:tasks.workunit.client.1.smithi192.stdout:7/617: write d3/d7/fd3 [933517,51504] 0 2024-04-12T05:09:09.095 INFO:tasks.workunit.client.0.smithi113.stdout:2/729: dread d8/d9/de/fd5 [0,4194304] 0 2024-04-12T05:09:09.095 INFO:tasks.workunit.client.0.smithi113.stdout:9/740: readlink d1/d4/d38/d65/l92 0 2024-04-12T05:09:09.095 INFO:tasks.workunit.client.1.smithi192.stdout:5/711: creat d0/d15/d3e/dca/dda/da0/ff6 x:0 0 0 2024-04-12T05:09:09.096 INFO:tasks.workunit.client.0.smithi113.stdout:1/840: rmdir d10/d2e/d7c/dc5/dc8/de1 39 2024-04-12T05:09:09.096 INFO:tasks.workunit.client.0.smithi113.stdout:2/730: dread - d8/d9/d27/d88/fa4 zero size 2024-04-12T05:09:09.097 INFO:tasks.workunit.client.0.smithi113.stdout:8/792: symlink d9/d16/d65/db1/lfa 0 2024-04-12T05:09:09.097 INFO:tasks.workunit.client.1.smithi192.stdout:6/725: dwrite d3/d8/d56/d59/d8a/d99/f9f [0,4194304] 0 2024-04-12T05:09:09.099 INFO:tasks.workunit.client.0.smithi113.stdout:8/793: fsync d9/f9d 0 2024-04-12T05:09:09.099 INFO:tasks.workunit.client.0.smithi113.stdout:9/741: readlink d1/df/d4b/d63/d3d/d6f/l94 0 2024-04-12T05:09:09.100 INFO:tasks.workunit.client.0.smithi113.stdout:3/685: dread d0/f57 [0,4194304] 0 2024-04-12T05:09:09.101 INFO:tasks.workunit.client.1.smithi192.stdout:7/618: write d3/d7/d60/d65/f8a [1355575,45502] 0 2024-04-12T05:09:09.101 INFO:tasks.workunit.client.0.smithi113.stdout:2/731: fdatasync d8/d9/de/d12/f35 0 2024-04-12T05:09:09.106 INFO:tasks.workunit.client.0.smithi113.stdout:8/794: creat d9/d16/d65/db1/d59/da3/ffb x:0 0 0 2024-04-12T05:09:09.106 INFO:tasks.workunit.client.1.smithi192.stdout:5/712: fdatasync d0/d15/d3e/dca/fc4 0 2024-04-12T05:09:09.107 INFO:tasks.workunit.client.1.smithi192.stdout:2/704: truncate d1/f4b 159850 0 2024-04-12T05:09:09.108 INFO:tasks.workunit.client.1.smithi192.stdout:5/713: dread - d0/d15/d32/db2/fd2 zero size 2024-04-12T05:09:09.109 INFO:tasks.workunit.client.0.smithi113.stdout:3/686: mkdir d0/d13/d15/d28/d2a/d34/d5f/d7b/de5 0 2024-04-12T05:09:09.111 INFO:tasks.workunit.client.0.smithi113.stdout:5/831: write d0/d10/d6e/f4f [2783529,17405] 0 2024-04-12T05:09:09.112 INFO:tasks.workunit.client.1.smithi192.stdout:3/665: rename d5/d13/d3a/d4e/c5c to d5/d13/cec 0 2024-04-12T05:09:09.112 INFO:tasks.workunit.client.0.smithi113.stdout:2/732: dwrite d8/d9/de/d15/d24/f8d [0,4194304] 0 2024-04-12T05:09:09.115 INFO:tasks.workunit.client.1.smithi192.stdout:7/619: mkdir d3/d11/dd8 0 2024-04-12T05:09:09.116 INFO:tasks.workunit.client.1.smithi192.stdout:3/666: write d5/d13/d3a/d4a/d61/d79/fe2 [913213,78096] 0 2024-04-12T05:09:09.118 INFO:tasks.workunit.client.1.smithi192.stdout:7/620: chown d3/d7/d15/f77 827276430 1 2024-04-12T05:09:09.118 INFO:tasks.workunit.client.0.smithi113.stdout:5/832: write d0/d10/d27/d1d/d38/d44/f114 [383743,113475] 0 2024-04-12T05:09:09.122 INFO:tasks.workunit.client.1.smithi192.stdout:0/735: rename d0/c52 to d0/d7b/d9b/d4b/dbc/dce/d53/df3/d2e/d66/cf4 0 2024-04-12T05:09:09.122 INFO:tasks.workunit.client.1.smithi192.stdout:6/726: mkdir d3/d8/d56/d59/ddf/df1 0 2024-04-12T05:09:09.122 INFO:tasks.workunit.client.0.smithi113.stdout:1/841: dwrite d10/da5/fb3 [0,4194304] 0 2024-04-12T05:09:09.123 INFO:tasks.workunit.client.0.smithi113.stdout:8/795: link d9/d16/fd3 d9/d1b/d8e/d93/d32/d48/d8d/ffc 0 2024-04-12T05:09:09.124 INFO:tasks.workunit.client.1.smithi192.stdout:8/675: dwrite d6/d10/d46/d5e/d5f/d37/d4f/d51/f66 [0,4194304] 0 2024-04-12T05:09:09.125 INFO:tasks.workunit.client.1.smithi192.stdout:7/621: write d3/d7/d15/d1a/f3e [831055,10753] 0 2024-04-12T05:09:09.126 INFO:tasks.workunit.client.0.smithi113.stdout:3/687: creat d0/d13/d47/d94/dcf/dd1/de2/fe6 x:0 0 0 2024-04-12T05:09:09.129 INFO:tasks.workunit.client.0.smithi113.stdout:2/733: rename d8/d9/d27/d88/d91/dd4 to d8/d9/de/d12/d2b/daa/db3/de2 0 2024-04-12T05:09:09.130 INFO:tasks.workunit.client.1.smithi192.stdout:8/676: truncate d6/d10/d46/d5e/d6e/fbf 221381 0 2024-04-12T05:09:09.135 INFO:tasks.workunit.client.0.smithi113.stdout:5/833: write d0/d10/d27/d1d/d38/d44/fbc [3485486,122066] 0 2024-04-12T05:09:09.138 INFO:tasks.workunit.client.0.smithi113.stdout:0/832: dwrite d1/d6/d16/d8f/fce [0,4194304] 0 2024-04-12T05:09:09.142 INFO:tasks.workunit.client.0.smithi113.stdout:3/688: mknod d0/d13/d47/d4f/dd3/ce7 0 2024-04-12T05:09:09.144 INFO:tasks.workunit.client.1.smithi192.stdout:7/622: creat d3/d1d/d74/d86/d93/fd9 x:0 0 0 2024-04-12T05:09:09.144 INFO:tasks.workunit.client.1.smithi192.stdout:5/714: dread d0/d15/d32/f6a [0,4194304] 0 2024-04-12T05:09:09.145 INFO:tasks.workunit.client.1.smithi192.stdout:5/715: write d0/d21/fa9 [2298308,62137] 0 2024-04-12T05:09:09.149 INFO:tasks.workunit.client.0.smithi113.stdout:5/834: write d0/d10/d27/d1d/d38/d75/fd7 [488222,71071] 0 2024-04-12T05:09:09.151 INFO:tasks.workunit.client.1.smithi192.stdout:8/677: symlink d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d98/lcf 0 2024-04-12T05:09:09.151 INFO:tasks.workunit.client.0.smithi113.stdout:2/734: dwrite d8/d9/de/d12/dbf/fd8 [0,4194304] 0 2024-04-12T05:09:09.154 INFO:tasks.workunit.client.1.smithi192.stdout:3/667: link d5/d13/d1a/lc8 d5/d13/d3a/d4a/d61/d79/led 0 2024-04-12T05:09:09.155 INFO:tasks.workunit.client.0.smithi113.stdout:5/835: fdatasync d0/d10/d27/d1d/d38/d44/fa3 0 2024-04-12T05:09:09.157 INFO:tasks.workunit.client.0.smithi113.stdout:8/796: rename d9/d1b/d8e/d93/d32/d48/dc1 to d9/d1b/dfd 0 2024-04-12T05:09:09.159 INFO:tasks.workunit.client.0.smithi113.stdout:1/842: truncate d10/d2e/dd8/d99/fb8 283793 0 2024-04-12T05:09:09.161 INFO:tasks.workunit.client.0.smithi113.stdout:0/833: write d1/d6/d16/d2a/d50/f65 [5295057,100258] 0 2024-04-12T05:09:09.161 INFO:tasks.workunit.client.0.smithi113.stdout:2/735: mknod d8/d9/de/d12/d2b/daa/db3/de2/ce3 0 2024-04-12T05:09:09.161 INFO:tasks.workunit.client.1.smithi192.stdout:0/736: chown d0/d6/faa 6760072 1 2024-04-12T05:09:09.162 INFO:tasks.workunit.client.0.smithi113.stdout:3/689: dwrite d0/d13/d47/d94/fd4 [0,4194304] 0 2024-04-12T05:09:09.162 INFO:tasks.workunit.client.0.smithi113.stdout:1/843: fdatasync d10/d2e/f102 0 2024-04-12T05:09:09.163 INFO:tasks.workunit.client.1.smithi192.stdout:6/727: dread d3/d5/f51 [0,4194304] 0 2024-04-12T05:09:09.163 INFO:tasks.workunit.client.1.smithi192.stdout:2/705: dread d1/d8/d17/d19/d60/f76 [4194304,4194304] 0 2024-04-12T05:09:09.168 INFO:tasks.workunit.client.1.smithi192.stdout:3/668: symlink d5/d13/d3a/d70/d8c/d8d/dbc/lee 0 2024-04-12T05:09:09.168 INFO:tasks.workunit.client.1.smithi192.stdout:5/716: dwrite d0/d15/da1/fa8 [0,4194304] 0 2024-04-12T05:09:09.170 INFO:tasks.workunit.client.0.smithi113.stdout:9/742: unlink d1/df/d4b/d63/l3a 0 2024-04-12T05:09:09.171 INFO:tasks.workunit.client.0.smithi113.stdout:8/797: mknod d9/d1b/d8e/d93/d32/d48/d8d/cfe 0 2024-04-12T05:09:09.173 INFO:tasks.workunit.client.1.smithi192.stdout:2/706: creat d1/d8/d17/d19/d60/d59/da3/fde x:0 0 0 2024-04-12T05:09:09.174 INFO:tasks.workunit.client.0.smithi113.stdout:1/844: rmdir d10/d2e/dd8/df1 39 2024-04-12T05:09:09.175 INFO:tasks.workunit.client.0.smithi113.stdout:5/836: dwrite d0/d10/d6e/fb5 [0,4194304] 0 2024-04-12T05:09:09.175 INFO:tasks.workunit.client.0.smithi113.stdout:1/845: truncate d10/d14/f4d 1062432 0 2024-04-12T05:09:09.175 INFO:tasks.workunit.client.1.smithi192.stdout:0/737: rmdir d0/d7b/d9b 39 2024-04-12T05:09:09.177 INFO:tasks.workunit.client.0.smithi113.stdout:1/846: chown d10/d2e/dd8/d70/f48 309682730 1 2024-04-12T05:09:09.177 INFO:tasks.workunit.client.1.smithi192.stdout:8/678: read d6/d1f/d5d/f8d [2948761,41242] 0 2024-04-12T05:09:09.177 INFO:tasks.workunit.client.1.smithi192.stdout:2/707: chown d1/d8/d17/c8c 63151 1 2024-04-12T05:09:09.177 INFO:tasks.workunit.client.1.smithi192.stdout:3/669: read d5/d13/d3a/d70/d8c/d8d/fe4 [344142,52186] 0 2024-04-12T05:09:09.178 INFO:tasks.workunit.client.1.smithi192.stdout:5/717: mknod d0/d15/da1/cf7 0 2024-04-12T05:09:09.179 INFO:tasks.workunit.client.0.smithi113.stdout:3/690: mknod d0/d13/d15/d28/d2a/d8b/ce8 0 2024-04-12T05:09:09.179 INFO:tasks.workunit.client.1.smithi192.stdout:6/728: symlink d3/d23/d67/lf2 0 2024-04-12T05:09:09.181 INFO:tasks.workunit.client.0.smithi113.stdout:1/847: chown d10/d2e/dd8/ca8 3838984 1 2024-04-12T05:09:09.181 INFO:tasks.workunit.client.0.smithi113.stdout:8/798: creat d9/dd4/dde/fff x:0 0 0 2024-04-12T05:09:09.183 INFO:tasks.workunit.client.0.smithi113.stdout:5/837: symlink d0/d10/d27/d1d/d3f/d73/l11d 0 2024-04-12T05:09:09.183 INFO:tasks.workunit.client.0.smithi113.stdout:3/691: write d0/d13/d15/d28/d2a/fbb [918000,129132] 0 2024-04-12T05:09:09.184 INFO:tasks.workunit.client.1.smithi192.stdout:7/623: dwrite d3/f21 [4194304,4194304] 0 2024-04-12T05:09:09.185 INFO:tasks.workunit.client.0.smithi113.stdout:8/799: write d9/d1b/f61 [7225408,94376] 0 2024-04-12T05:09:09.190 INFO:tasks.workunit.client.0.smithi113.stdout:1/848: write f5 [2407613,29686] 0 2024-04-12T05:09:09.191 INFO:tasks.workunit.client.1.smithi192.stdout:2/708: unlink d1/d8/f8e 0 2024-04-12T05:09:09.191 INFO:tasks.workunit.client.1.smithi192.stdout:5/718: rmdir d0/d15/d32/d69 39 2024-04-12T05:09:09.191 INFO:tasks.workunit.client.1.smithi192.stdout:0/738: mknod d0/d6/dc4/cf5 0 2024-04-12T05:09:09.195 INFO:tasks.workunit.client.1.smithi192.stdout:8/679: dwrite d6/d10/d46/d5e/d5f/d38/d3b/f73 [0,4194304] 0 2024-04-12T05:09:09.197 INFO:tasks.workunit.client.1.smithi192.stdout:8/680: write d6/da/f49 [537801,69695] 0 2024-04-12T05:09:09.198 INFO:tasks.workunit.client.1.smithi192.stdout:6/729: rename d3/d8/d56/d59/d97/dbf/ff0 to d3/d21/d24/ff3 0 2024-04-12T05:09:09.199 INFO:tasks.workunit.client.0.smithi113.stdout:8/800: creat d9/d16/d65/db1/d59/d68/f100 x:0 0 0 2024-04-12T05:09:09.202 INFO:tasks.workunit.client.0.smithi113.stdout:8/801: chown d9/dd4/ff1 0 1 2024-04-12T05:09:09.203 INFO:tasks.workunit.client.0.smithi113.stdout:3/692: dwrite d0/d13/d15/fb4 [0,4194304] 0 2024-04-12T05:09:09.203 INFO:tasks.workunit.client.0.smithi113.stdout:1/849: creat d10/da5/f113 x:0 0 0 2024-04-12T05:09:09.203 INFO:tasks.workunit.client.0.smithi113.stdout:5/838: link d0/cd d0/d10/d27/d1d/d3f/d7e/df8/c11e 0 2024-04-12T05:09:09.205 INFO:tasks.workunit.client.1.smithi192.stdout:7/624: dread - d3/d7/d60/d65/f81 zero size 2024-04-12T05:09:09.208 INFO:tasks.workunit.client.1.smithi192.stdout:7/625: fdatasync d3/d1d/d5a/d5f/d7a/fc6 0 2024-04-12T05:09:09.209 INFO:tasks.workunit.client.1.smithi192.stdout:6/730: symlink d3/dc/lf4 0 2024-04-12T05:09:09.214 INFO:tasks.workunit.client.1.smithi192.stdout:8/681: dwrite d6/da/f49 [0,4194304] 0 2024-04-12T05:09:09.216 INFO:tasks.workunit.client.0.smithi113.stdout:0/834: dwrite d1/d6/d16/dd5/dbb/fc0 [0,4194304] 0 2024-04-12T05:09:09.217 INFO:tasks.workunit.client.0.smithi113.stdout:1/850: read d10/d2e/dd8/f69 [1879230,101020] 0 2024-04-12T05:09:09.219 INFO:tasks.workunit.client.1.smithi192.stdout:3/670: dread d5/f2d [0,4194304] 0 2024-04-12T05:09:09.221 INFO:tasks.workunit.client.0.smithi113.stdout:0/835: unlink d1/d6/d16/dd5/f7b 0 2024-04-12T05:09:09.222 INFO:tasks.workunit.client.1.smithi192.stdout:5/719: dread d0/d21/f33 [0,4194304] 0 2024-04-12T05:09:09.224 INFO:tasks.workunit.client.1.smithi192.stdout:3/671: fsync d5/f14 0 2024-04-12T05:09:09.228 INFO:tasks.workunit.client.1.smithi192.stdout:6/731: dwrite d3/dc/d39/f70 [0,4194304] 0 2024-04-12T05:09:09.228 INFO:tasks.workunit.client.1.smithi192.stdout:5/720: write d0/d15/d24/d29/f85 [297988,123591] 0 2024-04-12T05:09:09.228 INFO:tasks.workunit.client.0.smithi113.stdout:8/802: write d9/d1b/d96/fe9 [934895,40707] 0 2024-04-12T05:09:09.230 INFO:tasks.workunit.client.1.smithi192.stdout:7/626: fsync d3/d1d/d48/d7b/f95 0 2024-04-12T05:09:09.230 INFO:tasks.workunit.client.0.smithi113.stdout:9/743: dread d1/df/db6/f59 [0,4194304] 0 2024-04-12T05:09:09.232 INFO:tasks.workunit.client.0.smithi113.stdout:0/836: creat d1/d6/d16/d2a/d50/fff x:0 0 0 2024-04-12T05:09:09.234 INFO:tasks.workunit.client.0.smithi113.stdout:9/744: symlink d1/d4/d5/lfc 0 2024-04-12T05:09:09.236 INFO:tasks.workunit.client.0.smithi113.stdout:3/693: dread d0/d13/f7d [0,4194304] 0 2024-04-12T05:09:09.236 INFO:tasks.workunit.client.0.smithi113.stdout:8/803: rename d9/dd4/dde/fff to d9/d16/d65/db1/d59/d78/f101 0 2024-04-12T05:09:09.238 INFO:tasks.workunit.client.1.smithi192.stdout:5/721: write d0/d15/d32/d69/d7d/ff4 [194486,39031] 0 2024-04-12T05:09:09.239 INFO:tasks.workunit.client.0.smithi113.stdout:0/837: write d1/d6/d16/d2a/d50/d52/d7e/f8d [4441071,98106] 0 2024-04-12T05:09:09.240 INFO:tasks.workunit.client.1.smithi192.stdout:7/627: rename d3/d1d/d5a/d5f/d7a/lc5 to d3/d1d/d5a/d5f/d7a/da6/lda 0 2024-04-12T05:09:09.241 INFO:tasks.workunit.client.0.smithi113.stdout:8/804: symlink d9/d16/d65/l102 0 2024-04-12T05:09:09.241 INFO:tasks.workunit.client.0.smithi113.stdout:5/839: dread d0/d10/d27/d1d/d3f/d73/fff [0,4194304] 0 2024-04-12T05:09:09.242 INFO:tasks.workunit.client.1.smithi192.stdout:8/682: dwrite d6/d10/d46/d5e/d5f/fab [0,4194304] 0 2024-04-12T05:09:09.244 INFO:tasks.workunit.client.0.smithi113.stdout:0/838: creat d1/d6/d16/f100 x:0 0 0 2024-04-12T05:09:09.246 INFO:tasks.workunit.client.0.smithi113.stdout:3/694: dwrite d0/d13/f42 [4194304,4194304] 0 2024-04-12T05:09:09.247 INFO:tasks.workunit.client.0.smithi113.stdout:0/839: write d1/d6/d16/d2a/d50/f85 [139990,18137] 0 2024-04-12T05:09:09.254 INFO:tasks.workunit.client.1.smithi192.stdout:8/683: unlink d6/d10/d46/d5e/d5f/d31/d61/la7 0 2024-04-12T05:09:09.254 INFO:tasks.workunit.client.0.smithi113.stdout:3/695: creat d0/d13/d47/d94/dcf/dd1/fe9 x:0 0 0 2024-04-12T05:09:09.255 INFO:tasks.workunit.client.1.smithi192.stdout:0/739: write d0/d6/fa3 [461077,56637] 0 2024-04-12T05:09:09.255 INFO:tasks.workunit.client.0.smithi113.stdout:3/696: chown d0/l66 858 1 2024-04-12T05:09:09.256 INFO:tasks.workunit.client.0.smithi113.stdout:0/840: read d1/d6/d16/fb8 [5038159,29918] 0 2024-04-12T05:09:09.260 INFO:tasks.workunit.client.0.smithi113.stdout:3/697: creat d0/d13/d15/d28/dc0/fea x:0 0 0 2024-04-12T05:09:09.260 INFO:tasks.workunit.client.0.smithi113.stdout:0/841: unlink d1/d6/d16/d2a/d50/lfd 0 2024-04-12T05:09:09.262 INFO:tasks.workunit.client.0.smithi113.stdout:3/698: symlink d0/d13/d15/d28/d2a/d34/d5f/leb 0 2024-04-12T05:09:09.263 INFO:tasks.workunit.client.0.smithi113.stdout:0/842: mknod d1/d43/d59/df4/c101 0 2024-04-12T05:09:09.266 INFO:tasks.workunit.client.1.smithi192.stdout:8/684: getdents d6/d10/d46/d5e/d5f/d31/d61/d9d/db8 0 2024-04-12T05:09:09.269 INFO:tasks.workunit.client.1.smithi192.stdout:8/685: fsync d6/d10/d46/d5e/d5f/f88 0 2024-04-12T05:09:09.270 INFO:tasks.workunit.client.0.smithi113.stdout:3/699: link d0/d13/d47/f4a d0/d13/d15/d28/d2a/d34/d5f/fec 0 2024-04-12T05:09:09.271 INFO:tasks.workunit.client.1.smithi192.stdout:5/722: dread d0/f13 [0,4194304] 0 2024-04-12T05:09:09.275 INFO:tasks.workunit.client.0.smithi113.stdout:8/805: write d9/d1b/d8e/d93/d32/f64 [76873,78872] 0 2024-04-12T05:09:09.275 INFO:tasks.workunit.client.0.smithi113.stdout:3/700: truncate d0/d13/d15/d28/d2a/f8e 291332 0 2024-04-12T05:09:09.278 INFO:tasks.workunit.client.1.smithi192.stdout:0/740: dread d0/d6/d67/f8a [0,4194304] 0 2024-04-12T05:09:09.280 INFO:tasks.workunit.client.1.smithi192.stdout:7/628: write d3/d1d/f28 [4230873,70006] 0 2024-04-12T05:09:09.281 INFO:tasks.workunit.client.0.smithi113.stdout:9/745: dread d1/d4/d87/fc9 [0,4194304] 0 2024-04-12T05:09:09.283 INFO:tasks.workunit.client.0.smithi113.stdout:1/851: dwrite d10/d41/d89/d96/d7d/d9f/fbd [0,4194304] 0 2024-04-12T05:09:09.284 INFO:tasks.workunit.client.1.smithi192.stdout:5/723: dwrite d0/d15/d3e/dca/fb7 [0,4194304] 0 2024-04-12T05:09:09.285 INFO:tasks.workunit.client.1.smithi192.stdout:8/686: write d6/d10/d46/d7c/f8c [1196655,14343] 0 2024-04-12T05:09:09.286 INFO:tasks.workunit.client.1.smithi192.stdout:6/732: dwrite d3/d8/f25 [4194304,4194304] 0 2024-04-12T05:09:09.286 INFO:tasks.workunit.client.0.smithi113.stdout:1/852: chown d10/d2e/dd8/d99 1159153 1 2024-04-12T05:09:09.287 INFO:tasks.workunit.client.0.smithi113.stdout:5/840: dwrite d0/d10/d27/d1d/d3f/d73/d89/f62 [0,4194304] 0 2024-04-12T05:09:09.288 INFO:tasks.workunit.client.1.smithi192.stdout:8/687: write d6/d10/d46/f96 [2984748,100178] 0 2024-04-12T05:09:09.289 INFO:tasks.workunit.client.1.smithi192.stdout:6/733: stat d3/d21/d24/ce0 0 2024-04-12T05:09:09.290 INFO:tasks.workunit.client.0.smithi113.stdout:5/841: chown d0/d10/d27/d1d/d8c 150539 1 2024-04-12T05:09:09.290 INFO:tasks.workunit.client.1.smithi192.stdout:5/724: fdatasync d0/d15/d24/f25 0 2024-04-12T05:09:09.290 INFO:tasks.workunit.client.0.smithi113.stdout:9/746: creat d1/d4/d5/d6/ffd x:0 0 0 2024-04-12T05:09:09.294 INFO:tasks.workunit.client.0.smithi113.stdout:0/843: dwrite d1/d6/fb0 [0,4194304] 0 2024-04-12T05:09:09.298 INFO:tasks.workunit.client.1.smithi192.stdout:5/725: creat d0/d15/d24/ff8 x:0 0 0 2024-04-12T05:09:09.299 INFO:tasks.workunit.client.0.smithi113.stdout:9/747: stat d1/df/d29/de8/cf4 0 2024-04-12T05:09:09.300 INFO:tasks.workunit.client.0.smithi113.stdout:5/842: symlink d0/d10/d27/d5f/l11f 0 2024-04-12T05:09:09.304 INFO:tasks.workunit.client.0.smithi113.stdout:8/806: rename d9/d1b/d8e/d93/d32/f3f to d9/d1b/d8e/d93/d32/d48/f103 0 2024-04-12T05:09:09.304 INFO:tasks.workunit.client.1.smithi192.stdout:7/629: truncate d3/d1d/d5a/d5f/f64 5737083 0 2024-04-12T05:09:09.305 INFO:tasks.workunit.client.1.smithi192.stdout:8/688: unlink d6/d10/l99 0 2024-04-12T05:09:09.305 INFO:tasks.workunit.client.0.smithi113.stdout:9/748: chown d1/d4/d5/d22/fdd 110451 1 2024-04-12T05:09:09.306 INFO:tasks.workunit.client.0.smithi113.stdout:0/844: dread d1/d6/d16/d2a/d50/f61 [0,4194304] 0 2024-04-12T05:09:09.307 INFO:tasks.workunit.client.0.smithi113.stdout:5/843: mknod d0/d10/d27/d1d/d3f/d73/c120 0 2024-04-12T05:09:09.307 INFO:tasks.workunit.client.1.smithi192.stdout:7/630: dread - d3/d1d/d4a/d5e/fca zero size 2024-04-12T05:09:09.308 INFO:tasks.workunit.client.1.smithi192.stdout:0/741: dread d0/d7b/d9b/d4b/dbc/dce/f65 [0,4194304] 0 2024-04-12T05:09:09.308 INFO:tasks.workunit.client.0.smithi113.stdout:8/807: truncate d9/d16/d65/fa9 601132 0 2024-04-12T05:09:09.309 INFO:tasks.workunit.client.0.smithi113.stdout:1/853: write d10/d21/f2b [1398115,121432] 0 2024-04-12T05:09:09.310 INFO:tasks.workunit.client.0.smithi113.stdout:9/749: truncate d1/d4/d5/fb9 239582 0 2024-04-12T05:09:09.313 INFO:tasks.workunit.client.1.smithi192.stdout:5/726: mkdir d0/d15/da1/df9 0 2024-04-12T05:09:09.314 INFO:tasks.workunit.client.1.smithi192.stdout:5/727: stat d0/d15/d32/d69/d7d/fc9 0 2024-04-12T05:09:09.316 INFO:tasks.workunit.client.1.smithi192.stdout:8/689: creat d6/d1f/d9c/fd0 x:0 0 0 2024-04-12T05:09:09.335 INFO:tasks.workunit.client.0.smithi113.stdout:7/729: sync 2024-04-12T05:09:09.337 INFO:tasks.workunit.client.1.smithi192.stdout:7/631: chown d3/d7/d14/f3d 108264529 1 2024-04-12T05:09:09.338 INFO:tasks.workunit.client.1.smithi192.stdout:0/742: rename d0/d6/dc4/cf5 to d0/d7b/d9b/d4b/dbc/dce/d53/df3/d2e/d59/cf6 0 2024-04-12T05:09:09.339 INFO:tasks.workunit.client.0.smithi113.stdout:8/808: symlink d9/d1b/d8e/d93/d32/d48/d8d/dae/l104 0 2024-04-12T05:09:09.340 INFO:tasks.workunit.client.0.smithi113.stdout:5/844: creat d0/d10/d27/d1d/d38/d44/d66/d10f/f121 x:0 0 0 2024-04-12T05:09:09.340 INFO:tasks.workunit.client.0.smithi113.stdout:9/750: symlink d1/d4/d5/d22/d9e/lfe 0 2024-04-12T05:09:09.341 INFO:tasks.workunit.client.1.smithi192.stdout:0/743: chown d0/d6/d67/f92 9237 1 2024-04-12T05:09:09.342 INFO:tasks.workunit.client.1.smithi192.stdout:8/690: dread d6/d10/d46/d5e/d5f/d31/d61/d9d/fb9 [0,4194304] 0 2024-04-12T05:09:09.342 INFO:tasks.workunit.client.1.smithi192.stdout:7/632: write d3/d7/d15/f24 [2475857,35953] 0 2024-04-12T05:09:09.342 INFO:tasks.workunit.client.0.smithi113.stdout:3/701: truncate d0/d13/d15/d28/d2a/fbb 586068 0 2024-04-12T05:09:09.343 INFO:tasks.workunit.client.0.smithi113.stdout:9/751: stat d1/df/db6 0 2024-04-12T05:09:09.343 INFO:tasks.workunit.client.0.smithi113.stdout:8/809: truncate d9/dd4/ff1 568793 0 2024-04-12T05:09:09.343 INFO:tasks.workunit.client.1.smithi192.stdout:7/633: dread - d3/d1d/d5a/d5f/fd2 zero size 2024-04-12T05:09:09.344 INFO:tasks.workunit.client.0.smithi113.stdout:7/730: getdents d0/dc/d6/d9b/dcd 0 2024-04-12T05:09:09.345 INFO:tasks.workunit.client.1.smithi192.stdout:5/728: rename d0/d15/d32/d45/f7f to d0/d15/da1/ffa 0 2024-04-12T05:09:09.345 INFO:tasks.workunit.client.1.smithi192.stdout:7/634: dread - d3/d1d/d5a/d5f/d6a/fb7 zero size 2024-04-12T05:09:09.346 INFO:tasks.workunit.client.1.smithi192.stdout:6/734: truncate d3/d5/d9c/fa7 725535 0 2024-04-12T05:09:09.347 INFO:tasks.workunit.client.0.smithi113.stdout:1/854: dread d10/d41/d5b/fb0 [0,4194304] 0 2024-04-12T05:09:09.348 INFO:tasks.workunit.client.0.smithi113.stdout:3/702: creat d0/d13/d15/d28/d2a/d8b/d91/fed x:0 0 0 2024-04-12T05:09:09.348 INFO:tasks.workunit.client.0.smithi113.stdout:0/845: rename d1/d6/d16/d2a/c33 to d1/d43/d59/db2/c102 0 2024-04-12T05:09:09.349 INFO:tasks.workunit.client.0.smithi113.stdout:5/845: getdents d0/d10/d27/d1d/d3f/db0/dda 0 2024-04-12T05:09:09.350 INFO:tasks.workunit.client.0.smithi113.stdout:7/731: truncate d0/d52/fbc 195402 0 2024-04-12T05:09:09.350 INFO:tasks.workunit.client.0.smithi113.stdout:9/752: mkdir d1/d4/d5/d22/d9e/df1/dff 0 2024-04-12T05:09:09.352 INFO:tasks.workunit.client.0.smithi113.stdout:8/810: fdatasync d9/d16/d7f/d3a/d6c/d90/fbc 0 2024-04-12T05:09:09.352 INFO:tasks.workunit.client.0.smithi113.stdout:5/846: readlink d0/d10/d27/d1d/d8c/db8/l101 0 2024-04-12T05:09:09.353 INFO:tasks.workunit.client.1.smithi192.stdout:0/744: creat d0/d6/d67/db1/ff7 x:0 0 0 2024-04-12T05:09:09.353 INFO:tasks.workunit.client.0.smithi113.stdout:7/732: chown d0/dc/da9/d5c/d85/fb1 6113 1 2024-04-12T05:09:09.354 INFO:tasks.workunit.client.0.smithi113.stdout:9/753: rmdir d1/df/d4b/d63/d3d 39 2024-04-12T05:09:09.354 INFO:tasks.workunit.client.0.smithi113.stdout:8/811: truncate d9/d1b/d96/fe7 50673 0 2024-04-12T05:09:09.356 INFO:tasks.workunit.client.1.smithi192.stdout:5/729: rename d0/d15/d24/d47/c76 to d0/d15/da1/cfb 0 2024-04-12T05:09:09.356 INFO:tasks.workunit.client.0.smithi113.stdout:3/703: dread d0/d13/d15/d28/d2a/d34/d5f/f96 [0,4194304] 0 2024-04-12T05:09:09.356 INFO:tasks.workunit.client.0.smithi113.stdout:5/847: mknod d0/d10/d6e/d93/dc9/c122 0 2024-04-12T05:09:09.357 INFO:tasks.workunit.client.1.smithi192.stdout:8/691: dwrite d6/d10/d46/d5e/d5f/d37/d4f/d4c/f55 [0,4194304] 0 2024-04-12T05:09:09.357 INFO:tasks.workunit.client.0.smithi113.stdout:1/855: rename fb to d10/d2e/d31/d5c/f114 0 2024-04-12T05:09:09.358 INFO:tasks.workunit.client.0.smithi113.stdout:7/733: mknod d0/dc/da9/d5c/d85/cf9 0 2024-04-12T05:09:09.358 INFO:tasks.workunit.client.1.smithi192.stdout:7/635: creat d3/d1d/d5a/d5f/d7a/da6/fdb x:0 0 0 2024-04-12T05:09:09.359 INFO:tasks.workunit.client.0.smithi113.stdout:0/846: dread - d1/d6/d16/d2a/d50/f99 zero size 2024-04-12T05:09:09.359 INFO:tasks.workunit.client.1.smithi192.stdout:7/636: stat d3/d1d/d74/d86/faf 0 2024-04-12T05:09:09.360 INFO:tasks.workunit.client.1.smithi192.stdout:6/735: truncate d3/d8/d22/f52 285257 0 2024-04-12T05:09:09.361 INFO:tasks.workunit.client.0.smithi113.stdout:8/812: creat d9/d16/d65/db1/d59/da3/f105 x:0 0 0 2024-04-12T05:09:09.362 INFO:tasks.workunit.client.1.smithi192.stdout:0/745: write d0/d7b/d9b/d4b/dbc/dce/d53/d6d/f9a [3147520,112619] 0 2024-04-12T05:09:09.363 INFO:tasks.workunit.client.1.smithi192.stdout:5/730: symlink d0/d15/d24/d29/lfc 0 2024-04-12T05:09:09.364 INFO:tasks.workunit.client.0.smithi113.stdout:3/704: rmdir d0/d13/d15/d28/d2a 39 2024-04-12T05:09:09.366 INFO:tasks.workunit.client.1.smithi192.stdout:8/692: rename d6/da/f3a to d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d98/fd1 0 2024-04-12T05:09:09.366 INFO:tasks.workunit.client.0.smithi113.stdout:1/856: dwrite d10/d14/f2a [0,4194304] 0 2024-04-12T05:09:09.371 INFO:tasks.workunit.client.0.smithi113.stdout:3/705: chown d0/d13/d15/d28/f45 636 1 2024-04-12T05:09:09.374 INFO:tasks.workunit.client.1.smithi192.stdout:5/731: rename d0/f96 to d0/d15/d3e/d77/ffd 0 2024-04-12T05:09:09.376 INFO:tasks.workunit.client.0.smithi113.stdout:8/813: dwrite d9/d1b/d8e/d93/d32/d5b/f83 [0,4194304] 0 2024-04-12T05:09:09.380 INFO:tasks.workunit.client.0.smithi113.stdout:9/754: getdents d1/d4/d5/d6 0 2024-04-12T05:09:09.380 INFO:tasks.workunit.client.0.smithi113.stdout:5/848: link d0/d10/d27/l7c d0/d10/d27/d1d/d8c/l123 0 2024-04-12T05:09:09.386 INFO:tasks.workunit.client.0.smithi113.stdout:1/857: readlink d10/d2e/d31/dc3/lef 0 2024-04-12T05:09:09.386 INFO:tasks.workunit.client.0.smithi113.stdout:0/847: link d1/d6/d16/d2a/d50/db6/f5c d1/d6/d16/d8f/f103 0 2024-04-12T05:09:09.387 INFO:tasks.workunit.client.0.smithi113.stdout:8/814: rename d9/d16/d7f/d3a/d6c/d90/dac to d9/d16/d65/db1/d106 0 2024-04-12T05:09:09.391 INFO:tasks.workunit.client.0.smithi113.stdout:3/706: truncate d0/d13/d15/d28/d2a/d34/f72 571036 0 2024-04-12T05:09:09.393 INFO:tasks.workunit.client.0.smithi113.stdout:7/734: dwrite d0/dc/d6/d9b/f55 [0,4194304] 0 2024-04-12T05:09:09.393 INFO:tasks.workunit.client.0.smithi113.stdout:9/755: getdents d1/d4/d5/d6 0 2024-04-12T05:09:09.396 INFO:tasks.workunit.client.0.smithi113.stdout:7/735: readlink d0/dd/d11/de6/ld3 0 2024-04-12T05:09:09.397 INFO:tasks.workunit.client.1.smithi192.stdout:0/746: creat d0/d6/db2/ff8 x:0 0 0 2024-04-12T05:09:09.397 INFO:tasks.workunit.client.0.smithi113.stdout:0/848: unlink d1/d6/d16/d2a/d50/db6/l56 0 2024-04-12T05:09:09.397 INFO:tasks.workunit.client.1.smithi192.stdout:6/736: write d3/dc/d39/f4e [3870954,106581] 0 2024-04-12T05:09:09.403 INFO:tasks.workunit.client.1.smithi192.stdout:7/637: rmdir d3/d1d/d5a/d5f/d7a/da6/dcb 0 2024-04-12T05:09:09.403 INFO:tasks.workunit.client.1.smithi192.stdout:0/747: write d0/d6/fa3 [239833,93737] 0 2024-04-12T05:09:09.403 INFO:tasks.workunit.client.1.smithi192.stdout:6/737: truncate d3/d8/d56/d59/fd0 52348 0 2024-04-12T05:09:09.404 INFO:tasks.workunit.client.0.smithi113.stdout:5/849: dwrite d0/d10/d27/d1d/d3f/d73/fff [0,4194304] 0 2024-04-12T05:09:09.406 INFO:tasks.workunit.client.1.smithi192.stdout:5/732: link d0/d15/d24/f4b d0/d15/d3e/dca/dda/ffe 0 2024-04-12T05:09:09.409 INFO:tasks.workunit.client.1.smithi192.stdout:8/693: truncate d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d98/fd1 7831674 0 2024-04-12T05:09:09.413 INFO:tasks.workunit.client.0.smithi113.stdout:0/849: symlink d1/d6/d34/dc3/l104 0 2024-04-12T05:09:09.415 INFO:tasks.workunit.client.0.smithi113.stdout:5/850: stat d0/f11b 0 2024-04-12T05:09:09.415 INFO:tasks.workunit.client.0.smithi113.stdout:8/815: dwrite d9/d1b/d8e/d93/f71 [0,4194304] 0 2024-04-12T05:09:09.418 INFO:tasks.workunit.client.1.smithi192.stdout:8/694: rename d6/c40 to d6/d10/d46/d5e/d5f/cd2 0 2024-04-12T05:09:09.419 INFO:tasks.workunit.client.0.smithi113.stdout:9/756: dwrite d1/d4/d5/f12 [4194304,4194304] 0 2024-04-12T05:09:09.420 INFO:tasks.workunit.client.0.smithi113.stdout:7/736: truncate d0/dc/da9/d5c/d85/f8d 346202 0 2024-04-12T05:09:09.422 INFO:tasks.workunit.client.0.smithi113.stdout:5/851: creat d0/d10/d6e/d93/f124 x:0 0 0 2024-04-12T05:09:09.424 INFO:tasks.workunit.client.1.smithi192.stdout:8/695: fdatasync d6/d10/d46/d5e/d5f/d37/f48 0 2024-04-12T05:09:09.426 INFO:tasks.workunit.client.0.smithi113.stdout:0/850: dread d1/d6/d16/d2a/d50/db6/d36/f74 [0,4194304] 0 2024-04-12T05:09:09.426 INFO:tasks.workunit.client.0.smithi113.stdout:3/707: dread d0/f6 [0,4194304] 0 2024-04-12T05:09:09.428 INFO:tasks.workunit.client.0.smithi113.stdout:0/851: chown d1/d6/d16/dd5/fc7 7252959 1 2024-04-12T05:09:09.430 INFO:tasks.workunit.client.1.smithi192.stdout:8/696: fsync d6/d10/d46/d5e/d5f/d38/d3b/fa1 0 2024-04-12T05:09:09.431 INFO:tasks.workunit.client.1.smithi192.stdout:0/748: write d0/d7b/d9b/d4b/dbc/dce/f65 [1794326,93473] 0 2024-04-12T05:09:09.433 INFO:tasks.workunit.client.0.smithi113.stdout:3/708: stat d0/d13/d15/d28/d2a/fbb 0 2024-04-12T05:09:09.433 INFO:tasks.workunit.client.0.smithi113.stdout:9/757: link d1/d4/d5/l2e d1/d4/d5/d22/d9e/df1/dff/l100 0 2024-04-12T05:09:09.434 INFO:tasks.workunit.client.1.smithi192.stdout:5/733: dread d0/d15/d24/d29/f2a [0,4194304] 0 2024-04-12T05:09:09.434 INFO:tasks.workunit.client.0.smithi113.stdout:7/737: link d0/d6c/l6f d0/dc/da9/d5c/d9a/d9e/lfa 0 2024-04-12T05:09:09.437 INFO:tasks.workunit.client.0.smithi113.stdout:3/709: mkdir d0/d13/d15/d28/d2a/d34/d5f/dee 0 2024-04-12T05:09:09.438 INFO:tasks.workunit.client.1.smithi192.stdout:8/697: creat d6/d1f/d5d/fd3 x:0 0 0 2024-04-12T05:09:09.439 INFO:tasks.workunit.client.0.smithi113.stdout:9/758: write d1/d4/d30/dcb/f3f [4072523,8024] 0 2024-04-12T05:09:09.439 INFO:tasks.workunit.client.1.smithi192.stdout:6/738: write d3/f43 [1570660,37703] 0 2024-04-12T05:09:09.440 INFO:tasks.workunit.client.1.smithi192.stdout:5/734: dread d0/d15/d24/d47/f4d [0,4194304] 0 2024-04-12T05:09:09.440 INFO:tasks.workunit.client.0.smithi113.stdout:7/738: creat d0/dc/d6/d9b/dcd/ffb x:0 0 0 2024-04-12T05:09:09.440 INFO:tasks.workunit.client.1.smithi192.stdout:0/749: dread - d0/fa7 zero size 2024-04-12T05:09:09.441 INFO:tasks.workunit.client.1.smithi192.stdout:6/739: chown d3/d8/dd6 4746 1 2024-04-12T05:09:09.442 INFO:tasks.workunit.client.1.smithi192.stdout:5/735: chown d0/f6e 0 1 2024-04-12T05:09:09.442 INFO:tasks.workunit.client.1.smithi192.stdout:6/740: stat d3/d23/f74 0 2024-04-12T05:09:09.444 INFO:tasks.workunit.client.0.smithi113.stdout:9/759: mkdir d1/df/db6/d101 0 2024-04-12T05:09:09.444 INFO:tasks.workunit.client.1.smithi192.stdout:8/698: creat d6/d10/d6c/fd4 x:0 0 0 2024-04-12T05:09:09.445 INFO:tasks.workunit.client.1.smithi192.stdout:0/750: mkdir d0/d6/d67/dd2/df9 0 2024-04-12T05:09:09.448 INFO:tasks.workunit.client.1.smithi192.stdout:0/751: readlink d0/d6/d15/d1b/d3a/d4f/l94 0 2024-04-12T05:09:09.449 INFO:tasks.workunit.client.1.smithi192.stdout:8/699: symlink d6/d1f/d5d/dc9/ld5 0 2024-04-12T05:09:09.450 INFO:tasks.workunit.client.1.smithi192.stdout:5/736: unlink d0/d15/d3e/dca/dda/da0/cb8 0 2024-04-12T05:09:09.450 INFO:tasks.workunit.client.1.smithi192.stdout:0/752: fdatasync d0/d7b/d9b/d4b/dbc/dce/d53/df3/d29/db5/fe4 0 2024-04-12T05:09:09.451 INFO:tasks.workunit.client.0.smithi113.stdout:9/760: dread d1/d4/d5/d22/f9c [0,4194304] 0 2024-04-12T05:09:09.451 INFO:tasks.workunit.client.1.smithi192.stdout:8/700: stat d6/d10/d46/d5e/d5f/d31/d61/f74 0 2024-04-12T05:09:09.451 INFO:tasks.workunit.client.1.smithi192.stdout:5/737: write d0/d21/fa9 [3710660,39858] 0 2024-04-12T05:09:09.452 INFO:tasks.workunit.client.0.smithi113.stdout:0/852: rename d1/d6/d16/d2a to d1/d43/d59/de9/d105 0 2024-04-12T05:09:09.453 INFO:tasks.workunit.client.1.smithi192.stdout:8/701: read d6/f95 [863978,63933] 0 2024-04-12T05:09:09.454 INFO:tasks.workunit.client.1.smithi192.stdout:0/753: write d0/d7b/d9b/d4b/dbc/dce/d53/df3/fd8 [1452038,20035] 0 2024-04-12T05:09:09.457 INFO:tasks.workunit.client.1.smithi192.stdout:5/738: unlink d0/l28 0 2024-04-12T05:09:09.458 INFO:tasks.workunit.client.0.smithi113.stdout:7/739: getdents d0/dc 0 2024-04-12T05:09:09.460 INFO:tasks.workunit.client.1.smithi192.stdout:0/754: mkdir d0/d7b/d9b/d4b/dbc/dfa 0 2024-04-12T05:09:09.461 INFO:tasks.workunit.client.1.smithi192.stdout:6/741: dread d3/d23/f74 [0,4194304] 0 2024-04-12T05:09:09.462 INFO:tasks.workunit.client.0.smithi113.stdout:5/852: dwrite d0/d10/d27/d1d/d3f/f47 [0,4194304] 0 2024-04-12T05:09:09.462 INFO:tasks.workunit.client.0.smithi113.stdout:0/853: truncate d1/d6/d34/fc6 51370 0 2024-04-12T05:09:09.463 INFO:tasks.workunit.client.1.smithi192.stdout:5/739: creat d0/d15/d3e/dca/dda/deb/fff x:0 0 0 2024-04-12T05:09:09.464 INFO:tasks.workunit.client.1.smithi192.stdout:8/702: creat d6/fd6 x:0 0 0 2024-04-12T05:09:09.466 INFO:tasks.workunit.client.1.smithi192.stdout:8/703: chown d6/d10/d46/d5e/d5f/d31/d61/f74 47 1 2024-04-12T05:09:09.466 INFO:tasks.workunit.client.0.smithi113.stdout:7/740: symlink d0/dd/d11/de6/lfc 0 2024-04-12T05:09:09.466 INFO:tasks.workunit.client.1.smithi192.stdout:0/755: unlink d0/d7b/d9b/d4b/dbc/dce/d53/df3/f6c 0 2024-04-12T05:09:09.467 INFO:tasks.workunit.client.0.smithi113.stdout:0/854: stat d1/d43/d59/de9/d105/d50/db6 0 2024-04-12T05:09:09.468 INFO:tasks.workunit.client.1.smithi192.stdout:6/742: symlink d3/d8/dd6/lf5 0 2024-04-12T05:09:09.469 INFO:tasks.workunit.client.0.smithi113.stdout:9/761: dwrite d1/d4/d5/d6/fdf [0,4194304] 0 2024-04-12T05:09:09.471 INFO:tasks.workunit.client.1.smithi192.stdout:5/740: link d0/d15/d3e/d77/ffd d0/d15/d64/f100 0 2024-04-12T05:09:09.472 INFO:tasks.workunit.client.0.smithi113.stdout:3/710: dwrite d0/d13/d15/d28/d2a/f8e [0,4194304] 0 2024-04-12T05:09:09.473 INFO:tasks.workunit.client.0.smithi113.stdout:5/853: truncate d0/d10/d27/d1d/d10c/f106 425928 0 2024-04-12T05:09:09.475 INFO:tasks.workunit.client.1.smithi192.stdout:6/743: fsync d3/d8/d22/f7c 0 2024-04-12T05:09:09.475 INFO:tasks.workunit.client.0.smithi113.stdout:0/855: write d1/d6/d16/dd5/fe3 [251838,75511] 0 2024-04-12T05:09:09.479 INFO:tasks.workunit.client.1.smithi192.stdout:0/756: creat d0/d7b/d9b/d4b/dbc/dce/d53/df3/d2e/d66/ffb x:0 0 0 2024-04-12T05:09:09.480 INFO:tasks.workunit.client.1.smithi192.stdout:8/704: dwrite d6/d10/d6c/fd4 [0,4194304] 0 2024-04-12T05:09:09.482 INFO:tasks.workunit.client.0.smithi113.stdout:0/856: chown d1/d6/d34/dc3/lef 2110 1 2024-04-12T05:09:09.485 INFO:tasks.workunit.client.0.smithi113.stdout:5/854: link d0/fc d0/d10/d27/d1d/d8c/db8/f125 0 2024-04-12T05:09:09.486 INFO:tasks.workunit.client.0.smithi113.stdout:0/857: symlink d1/d6/d8e/l106 0 2024-04-12T05:09:09.486 INFO:tasks.workunit.client.0.smithi113.stdout:3/711: dwrite d0/d13/d15/f3d [4194304,4194304] 0 2024-04-12T05:09:09.489 INFO:tasks.workunit.client.0.smithi113.stdout:9/762: write d1/d4/d38/d65/f7d [221076,125769] 0 2024-04-12T05:09:09.490 INFO:tasks.workunit.client.0.smithi113.stdout:0/858: chown d1/l6a 0 1 2024-04-12T05:09:09.490 INFO:tasks.workunit.client.1.smithi192.stdout:5/741: rename d0/d15/d3e/dca/f88 to d0/d21/f101 0 2024-04-12T05:09:09.492 INFO:tasks.workunit.client.0.smithi113.stdout:3/712: symlink d0/dc6/lef 0 2024-04-12T05:09:09.493 INFO:tasks.workunit.client.0.smithi113.stdout:5/855: creat d0/d10/d27/d1d/d38/d44/d4e/daa/f126 x:0 0 0 2024-04-12T05:09:09.498 INFO:tasks.workunit.client.0.smithi113.stdout:9/763: symlink d1/d4/d30/dcb/l102 0 2024-04-12T05:09:09.500 INFO:tasks.workunit.client.1.smithi192.stdout:0/757: truncate d0/d7b/d9b/d4b/dbc/dce/d53/df3/d2e/d59/f84 115510 0 2024-04-12T05:09:09.500 INFO:tasks.workunit.client.1.smithi192.stdout:6/744: symlink d3/d8/lf6 0 2024-04-12T05:09:09.501 INFO:tasks.workunit.client.0.smithi113.stdout:3/713: truncate d0/d13/d47/f61 219035 0 2024-04-12T05:09:09.501 INFO:tasks.workunit.client.0.smithi113.stdout:9/764: stat d1/d4/d5/d17 0 2024-04-12T05:09:09.502 INFO:tasks.workunit.client.1.smithi192.stdout:8/705: mkdir d6/d10/d6c/d9e/da8/dd7 0 2024-04-12T05:09:09.503 INFO:tasks.workunit.client.0.smithi113.stdout:5/856: getdents d0/d10/d27/d1d/d3f/d73/d89 0 2024-04-12T05:09:09.504 INFO:tasks.workunit.client.0.smithi113.stdout:0/859: fdatasync d1/f22 0 2024-04-12T05:09:09.504 INFO:tasks.workunit.client.0.smithi113.stdout:3/714: creat d0/d80/ff0 x:0 0 0 2024-04-12T05:09:09.506 INFO:tasks.workunit.client.0.smithi113.stdout:9/765: readlink d1/d4/d5/d22/d9e/df1/dff/l100 0 2024-04-12T05:09:09.508 INFO:tasks.workunit.client.0.smithi113.stdout:5/857: mkdir d0/d127 0 2024-04-12T05:09:09.509 INFO:tasks.workunit.client.0.smithi113.stdout:7/741: dwrite d0/d52/f54 [4194304,4194304] 0 2024-04-12T05:09:09.510 INFO:tasks.workunit.client.1.smithi192.stdout:8/706: symlink d6/d10/d46/d5e/d5f/d38/ld8 0 2024-04-12T05:09:09.512 INFO:tasks.workunit.client.1.smithi192.stdout:1/685: sync 2024-04-12T05:09:09.512 INFO:tasks.workunit.client.1.smithi192.stdout:4/756: sync 2024-04-12T05:09:09.514 INFO:tasks.workunit.client.1.smithi192.stdout:4/757: chown d1/d54/d82/d9c/f106 185 1 2024-04-12T05:09:09.515 INFO:tasks.workunit.client.1.smithi192.stdout:1/686: truncate d0/d2/d1e/d3f/d56/f79 1612540 0 2024-04-12T05:09:09.515 INFO:tasks.workunit.client.0.smithi113.stdout:9/766: creat d1/d4/d5/f103 x:0 0 0 2024-04-12T05:09:09.516 INFO:tasks.workunit.client.1.smithi192.stdout:0/758: link d0/d6/dc4/c5d d0/d7b/d9b/d4b/dbc/dce/d95/db4/cfc 0 2024-04-12T05:09:09.520 INFO:tasks.workunit.client.0.smithi113.stdout:7/742: creat d0/dd/ffd x:0 0 0 2024-04-12T05:09:09.521 INFO:tasks.workunit.client.1.smithi192.stdout:1/687: mknod d0/d2/dd/d21/d62/dbc/ce4 0 2024-04-12T05:09:09.521 INFO:tasks.workunit.client.0.smithi113.stdout:9/767: truncate d1/d4/d5/d22/f9c 905471 0 2024-04-12T05:09:09.522 INFO:tasks.workunit.client.1.smithi192.stdout:4/758: chown d1/d1d/d53/d1c/de1/fec 6 1 2024-04-12T05:09:09.522 INFO:tasks.workunit.client.0.smithi113.stdout:0/860: dwrite d1/d43/d59/de9/d105/d50/db6/fa8 [0,4194304] 0 2024-04-12T05:09:09.523 INFO:tasks.workunit.client.1.smithi192.stdout:8/707: dwrite d6/d10/d46/d5e/d5f/d38/d3b/fae [4194304,4194304] 0 2024-04-12T05:09:09.525 INFO:tasks.workunit.client.0.smithi113.stdout:0/861: readlink d1/d6/d16/la1 0 2024-04-12T05:09:09.525 INFO:tasks.workunit.client.0.smithi113.stdout:3/715: dread d0/fa6 [0,4194304] 0 2024-04-12T05:09:09.525 INFO:tasks.workunit.client.1.smithi192.stdout:1/688: write d0/d2/d1e/d3f/d56/f79 [106896,14587] 0 2024-04-12T05:09:09.527 INFO:tasks.workunit.client.0.smithi113.stdout:0/862: read d1/d43/d59/de9/d105/d50/f7d [1249081,71340] 0 2024-04-12T05:09:09.529 INFO:tasks.workunit.client.1.smithi192.stdout:6/745: write d3/d8/d48/f63 [867680,112312] 0 2024-04-12T05:09:09.531 INFO:tasks.workunit.client.0.smithi113.stdout:3/716: link d0/d7e/cbe d0/d13/d47/d4f/dca/cf1 0 2024-04-12T05:09:09.532 INFO:tasks.workunit.client.1.smithi192.stdout:4/759: read d1/fb6 [2991059,85390] 0 2024-04-12T05:09:09.532 INFO:tasks.workunit.client.0.smithi113.stdout:0/863: creat d1/d43/d59/df4/f107 x:0 0 0 2024-04-12T05:09:09.534 INFO:tasks.workunit.client.1.smithi192.stdout:5/742: mknod d0/d15/d3e/dca/dda/db5/c102 0 2024-04-12T05:09:09.534 INFO:tasks.workunit.client.0.smithi113.stdout:3/717: dread d0/f6 [0,4194304] 0 2024-04-12T05:09:09.535 INFO:tasks.workunit.client.0.smithi113.stdout:3/718: chown d0/d13/d15/cc5 871 1 2024-04-12T05:09:09.536 INFO:tasks.workunit.client.1.smithi192.stdout:5/743: truncate d0/fa 5033298 0 2024-04-12T05:09:09.536 INFO:tasks.workunit.client.1.smithi192.stdout:5/744: chown d0/d15/d32/d69 277 1 2024-04-12T05:09:09.539 INFO:tasks.workunit.client.1.smithi192.stdout:6/746: rename d3/d21/c7f to d3/d8/dd6/cf7 0 2024-04-12T05:09:09.539 INFO:tasks.workunit.client.0.smithi113.stdout:9/768: dread d1/df/fc3 [0,4194304] 0 2024-04-12T05:09:09.541 INFO:tasks.workunit.client.1.smithi192.stdout:0/759: getdents d0/d7b/d9b/d4b/dbc/dce/d53/df3/d2e 0 2024-04-12T05:09:09.541 INFO:tasks.workunit.client.0.smithi113.stdout:3/719: rename d0/d13/d15/d28/f45 to d0/d13/d47/d94/dcf/ff2 0 2024-04-12T05:09:09.542 INFO:tasks.workunit.client.1.smithi192.stdout:1/689: unlink d0/d2/dd/d21/d33/dd8/le2 0 2024-04-12T05:09:09.543 INFO:tasks.workunit.client.0.smithi113.stdout:0/864: dwrite d1/f22 [4194304,4194304] 0 2024-04-12T05:09:09.566 INFO:tasks.workunit.client.1.smithi192.stdout:6/747: dread d3/dc/f3b [0,4194304] 0 2024-04-12T05:09:09.588 INFO:tasks.workunit.client.0.smithi113.stdout:4/748: sync 2024-04-12T05:09:09.590 INFO:tasks.workunit.client.0.smithi113.stdout:9/769: truncate d1/d4/d6d/fc4 163109 0 2024-04-12T05:09:09.592 INFO:tasks.workunit.client.0.smithi113.stdout:0/865: mknod d1/d43/d59/df4/c108 0 2024-04-12T05:09:09.593 INFO:tasks.workunit.client.0.smithi113.stdout:3/720: dread d0/d13/d15/d28/d2a/d34/f63 [0,4194304] 0 2024-04-12T05:09:09.593 INFO:tasks.workunit.client.0.smithi113.stdout:5/858: dread d0/d10/d27/d5f/f88 [0,4194304] 0 2024-04-12T05:09:09.593 INFO:tasks.workunit.client.0.smithi113.stdout:4/749: write da/d4f/f4e [9080841,71261] 0 2024-04-12T05:09:09.596 INFO:tasks.workunit.client.0.smithi113.stdout:3/721: readlink d0/d13/d47/la1 0 2024-04-12T05:09:09.597 INFO:tasks.workunit.client.0.smithi113.stdout:4/750: rmdir da/d3c/d43/dbc 39 2024-04-12T05:09:09.598 INFO:tasks.workunit.client.1.smithi192.stdout:6/748: dread d3/d23/f78 [0,4194304] 0 2024-04-12T05:09:09.598 INFO:tasks.workunit.client.0.smithi113.stdout:5/859: mkdir d0/d10/d27/d5f/d128 0 2024-04-12T05:09:09.602 INFO:tasks.workunit.client.0.smithi113.stdout:7/743: dread d0/dc/d6/fc0 [0,4194304] 0 2024-04-12T05:09:09.602 INFO:tasks.workunit.client.0.smithi113.stdout:9/770: getdents d1/d4/d5/d22/d9e/df1 0 2024-04-12T05:09:09.602 INFO:tasks.workunit.client.0.smithi113.stdout:0/866: mknod d1/d43/d59/c109 0 2024-04-12T05:09:09.603 INFO:tasks.workunit.client.1.smithi192.stdout:0/760: mknod d0/d6/d67/db1/deb/cfd 0 2024-04-12T05:09:09.605 INFO:tasks.workunit.client.1.smithi192.stdout:8/708: getdents d6/d10/d6c/d9e/da8 0 2024-04-12T05:09:09.605 INFO:tasks.workunit.client.0.smithi113.stdout:5/860: creat d0/d10/d27/f129 x:0 0 0 2024-04-12T05:09:09.606 INFO:tasks.workunit.client.0.smithi113.stdout:9/771: mkdir d1/df/d29/de8/d104 0 2024-04-12T05:09:09.606 INFO:tasks.workunit.client.1.smithi192.stdout:8/709: chown d6/d10/d6c/fbd 3635 1 2024-04-12T05:09:09.607 INFO:tasks.workunit.client.0.smithi113.stdout:7/744: rename d0/dc/f83 to d0/dc/ffe 0 2024-04-12T05:09:09.608 INFO:tasks.workunit.client.1.smithi192.stdout:4/760: write d1/d54/d55/d64/f86 [407272,120661] 0 2024-04-12T05:09:09.608 INFO:tasks.workunit.client.1.smithi192.stdout:1/690: unlink d0/d2/dd/d21/d33/c37 0 2024-04-12T05:09:09.610 INFO:tasks.workunit.client.1.smithi192.stdout:0/761: dread d0/d6/dc4/fd9 [0,4194304] 0 2024-04-12T05:09:09.611 INFO:tasks.workunit.client.1.smithi192.stdout:6/749: truncate d3/d21/d24/f2a 699202 0 2024-04-12T05:09:09.611 INFO:tasks.workunit.client.0.smithi113.stdout:9/772: dread - d1/d4/d5/d6/f76 zero size 2024-04-12T05:09:09.613 INFO:tasks.workunit.client.0.smithi113.stdout:5/861: readlink d0/d10/d27/d1d/d38/d3b/d6b/l11a 0 2024-04-12T05:09:09.613 INFO:tasks.workunit.client.0.smithi113.stdout:3/722: dwrite d0/d13/d15/d28/fa5 [0,4194304] 0 2024-04-12T05:09:09.613 INFO:tasks.workunit.client.1.smithi192.stdout:8/710: rename d6/d10/d6c/d9e/da8 to d6/d10/d46/d5e/d5f/d31/d61/dc5/dd9 0 2024-04-12T05:09:09.614 INFO:tasks.workunit.client.1.smithi192.stdout:5/745: link d0/d15/d24/d47/f4d d0/d15/d3e/dca/dda/f103 0 2024-04-12T05:09:09.617 INFO:tasks.workunit.client.1.smithi192.stdout:4/761: creat d1/d54/d82/da4/da8/f108 x:0 0 0 2024-04-12T05:09:09.618 INFO:tasks.workunit.client.0.smithi113.stdout:0/867: dwrite d1/d6/d34/dc3/ff8 [0,4194304] 0 2024-04-12T05:09:09.619 INFO:tasks.workunit.client.1.smithi192.stdout:0/762: read - d0/d6/d15/d1b/f80 zero size 2024-04-12T05:09:09.620 INFO:tasks.workunit.client.0.smithi113.stdout:9/773: mknod d1/d4/d5/d22/d9e/df1/c105 0 2024-04-12T05:09:09.620 INFO:tasks.workunit.client.0.smithi113.stdout:5/862: read d0/d10/d27/d1d/d38/d44/d4e/f64 [155665,65368] 0 2024-04-12T05:09:09.621 INFO:tasks.workunit.client.0.smithi113.stdout:3/723: write d0/d13/d15/d28/d2a/d34/d5f/d7b/fcd [425747,53360] 0 2024-04-12T05:09:09.622 INFO:tasks.workunit.client.1.smithi192.stdout:1/691: dwrite d0/d1b/d44/f6e [0,4194304] 0 2024-04-12T05:09:09.622 INFO:tasks.workunit.client.1.smithi192.stdout:8/711: write d6/d10/d46/d5e/d5f/f7e [4047093,97172] 0 2024-04-12T05:09:09.622 INFO:tasks.workunit.client.1.smithi192.stdout:5/746: symlink d0/d21/l104 0 2024-04-12T05:09:09.624 INFO:tasks.workunit.client.0.smithi113.stdout:4/751: dread da/d3c/d43/f7d [0,4194304] 0 2024-04-12T05:09:09.627 INFO:tasks.workunit.client.1.smithi192.stdout:0/763: write d0/d6/d67/f92 [4647394,19830] 0 2024-04-12T05:09:09.627 INFO:tasks.workunit.client.0.smithi113.stdout:5/863: unlink d0/d10/d6e/d93/f124 0 2024-04-12T05:09:09.628 INFO:tasks.workunit.client.1.smithi192.stdout:8/712: readlink d6/d10/d46/d5e/d5f/d38/ld8 0 2024-04-12T05:09:09.629 INFO:tasks.workunit.client.1.smithi192.stdout:6/750: dwrite d3/d8/d56/d59/d97/dbf/de8/fa0 [4194304,4194304] 0 2024-04-12T05:09:09.634 INFO:tasks.workunit.client.1.smithi192.stdout:0/764: write d0/d6/d67/f8a [2513269,56318] 0 2024-04-12T05:09:09.639 INFO:tasks.workunit.client.0.smithi113.stdout:3/724: getdents d0/d13/d15/d28/d2a/d34/d5f/d7b/de5 0 2024-04-12T05:09:09.640 INFO:tasks.workunit.client.1.smithi192.stdout:5/747: link d0/d15/d32/f4c d0/d15/d3e/dca/dda/da7/f105 0 2024-04-12T05:09:09.640 INFO:tasks.workunit.client.0.smithi113.stdout:5/864: stat d0/d10/d27/d1d/d41 0 2024-04-12T05:09:09.641 INFO:tasks.workunit.client.1.smithi192.stdout:0/765: write d0/d7b/d9b/d4b/dbc/dce/d53/df3/d2e/d66/f70 [1034411,74737] 0 2024-04-12T05:09:09.641 INFO:tasks.workunit.client.0.smithi113.stdout:7/745: dwrite d0/dc/d6/d9b/ff3 [4194304,4194304] 0 2024-04-12T05:09:09.642 INFO:tasks.workunit.client.1.smithi192.stdout:8/713: dread d6/d10/d46/d5e/d5f/d38/d3b/fae [4194304,4194304] 0 2024-04-12T05:09:09.642 INFO:tasks.workunit.client.0.smithi113.stdout:4/752: mknod da/d24/d60/db8/cfb 0 2024-04-12T05:09:09.643 INFO:tasks.workunit.client.0.smithi113.stdout:0/868: dwrite d1/d6/d34/dc3/f2b [0,4194304] 0 2024-04-12T05:09:09.643 INFO:tasks.workunit.client.1.smithi192.stdout:1/692: dread d0/d2/d1e/d3f/d56/f79 [0,4194304] 0 2024-04-12T05:09:09.645 INFO:tasks.workunit.client.0.smithi113.stdout:3/725: stat d0/d13/d15/d28/f31 0 2024-04-12T05:09:09.645 INFO:tasks.workunit.client.1.smithi192.stdout:0/766: dread - d0/d7b/d9b/d4b/dbc/dce/d53/df3/d29/db5/fe4 zero size 2024-04-12T05:09:09.646 INFO:tasks.workunit.client.0.smithi113.stdout:4/753: dread - da/d24/d7b/fee zero size 2024-04-12T05:09:09.647 INFO:tasks.workunit.client.1.smithi192.stdout:1/693: truncate d0/d2/dd/d21/d62/d84/f8e 341681 0 2024-04-12T05:09:09.648 INFO:tasks.workunit.client.0.smithi113.stdout:5/865: creat d0/d10/d27/d1d/d8c/db8/f12a x:0 0 0 2024-04-12T05:09:09.649 INFO:tasks.workunit.client.1.smithi192.stdout:5/748: mkdir d0/d15/d32/d45/d106 0 2024-04-12T05:09:09.651 INFO:tasks.workunit.client.0.smithi113.stdout:3/726: rename d0/d13/d15/l33 to d0/d7e/da3/lf3 0 2024-04-12T05:09:09.652 INFO:tasks.workunit.client.0.smithi113.stdout:0/869: creat d1/d43/d59/de9/f10a x:0 0 0 2024-04-12T05:09:09.653 INFO:tasks.workunit.client.1.smithi192.stdout:1/694: write d0/d2/dd/d21/d62/dbc/fc0 [960623,75811] 0 2024-04-12T05:09:09.655 INFO:tasks.workunit.client.1.smithi192.stdout:8/714: symlink d6/d10/d46/d5e/d5f/d31/lda 0 2024-04-12T05:09:09.655 INFO:tasks.workunit.client.1.smithi192.stdout:5/749: stat d0/d15/d3e/dca/dda/db5/dd7 0 2024-04-12T05:09:09.657 INFO:tasks.workunit.client.1.smithi192.stdout:8/715: write d6/d10/d6c/fbd [120926,102002] 0 2024-04-12T05:09:09.657 INFO:tasks.workunit.client.0.smithi113.stdout:9/774: truncate d1/df/f36 3225891 0 2024-04-12T05:09:09.658 INFO:tasks.workunit.client.1.smithi192.stdout:1/695: fdatasync d0/d2/dd/fa9 0 2024-04-12T05:09:09.661 INFO:tasks.workunit.client.0.smithi113.stdout:9/775: symlink d1/d4/d5/d17/l106 0 2024-04-12T05:09:09.662 INFO:tasks.workunit.client.1.smithi192.stdout:8/716: symlink d6/d10/d46/d5e/d6e/ldb 0 2024-04-12T05:09:09.665 INFO:tasks.workunit.client.0.smithi113.stdout:0/870: chown d1/d43/d59/de9/d105/d50/d52/d7e 191257482 1 2024-04-12T05:09:09.668 INFO:tasks.workunit.client.0.smithi113.stdout:5/866: write d0/d10/d27/d1d/d38/d44/d4e/f64 [635160,14526] 0 2024-04-12T05:09:09.668 INFO:tasks.workunit.client.1.smithi192.stdout:4/762: dwrite d1/d1d/d53/f52 [0,4194304] 0 2024-04-12T05:09:09.669 INFO:tasks.workunit.client.0.smithi113.stdout:7/746: dwrite d0/dc/d6/fd0 [0,4194304] 0 2024-04-12T05:09:09.669 INFO:tasks.workunit.client.0.smithi113.stdout:4/754: dwrite da/d24/d7b/fdb [0,4194304] 0 2024-04-12T05:09:09.675 INFO:tasks.workunit.client.1.smithi192.stdout:5/750: dwrite d0/d15/d64/fcd [0,4194304] 0 2024-04-12T05:09:09.676 INFO:tasks.workunit.client.0.smithi113.stdout:9/776: rename d1/d27/d7a/de4/db7/fd8 to d1/df/d29/f107 0 2024-04-12T05:09:09.676 INFO:tasks.workunit.client.1.smithi192.stdout:0/767: dwrite d0/d7b/d9b/f60 [0,4194304] 0 2024-04-12T05:09:09.685 INFO:tasks.workunit.client.0.smithi113.stdout:0/871: rmdir d1/d43/d59/de9/d105/d50 39 2024-04-12T05:09:09.690 INFO:tasks.workunit.client.0.smithi113.stdout:4/755: creat da/d3c/d43/dbc/ffc x:0 0 0 2024-04-12T05:09:09.691 INFO:tasks.workunit.client.0.smithi113.stdout:3/727: link d0/d13/l14 d0/d13/d15/d28/d2a/d34/d5f/d7b/lf4 0 2024-04-12T05:09:09.691 INFO:tasks.workunit.client.0.smithi113.stdout:0/872: symlink d1/d43/d59/de9/l10b 0 2024-04-12T05:09:09.695 INFO:tasks.workunit.client.0.smithi113.stdout:5/867: dread d0/d10/d27/d1d/d3f/d73/d89/fc1 [0,4194304] 0 2024-04-12T05:09:09.696 INFO:tasks.workunit.client.0.smithi113.stdout:5/868: fsync d0/d10/d27/d1d/d38/d3b/d6b/f105 0 2024-04-12T05:09:09.698 INFO:tasks.workunit.client.0.smithi113.stdout:5/869: write d0/d10/d27/d1d/d38/d44/d66/d10f/f121 [8399,57033] 0 2024-04-12T05:09:09.698 INFO:tasks.workunit.client.1.smithi192.stdout:0/768: chown d0/d7b/d9b/d4b/dbc/dce/d95/db4/ld3 438129 1 2024-04-12T05:09:09.701 INFO:tasks.workunit.client.1.smithi192.stdout:5/751: dread d0/d15/d24/ddf/fe8 [0,4194304] 0 2024-04-12T05:09:09.702 INFO:tasks.workunit.client.0.smithi113.stdout:5/870: dread - d0/d10/d27/d1d/d3f/d73/fad zero size 2024-04-12T05:09:09.703 INFO:tasks.workunit.client.1.smithi192.stdout:1/696: dwrite d0/d2/dd/d3c/f9c [0,4194304] 0 2024-04-12T05:09:09.704 INFO:tasks.workunit.client.0.smithi113.stdout:3/728: dwrite d0/d13/d47/d94/dcf/ff2 [0,4194304] 0 2024-04-12T05:09:09.704 INFO:tasks.workunit.client.1.smithi192.stdout:8/717: dwrite d6/d10/d46/d5e/d5f/d38/d3b/f81 [0,4194304] 0 2024-04-12T05:09:09.705 INFO:tasks.workunit.client.0.smithi113.stdout:3/729: chown d0/d13/d47/f89 46 1 2024-04-12T05:09:09.711 INFO:tasks.workunit.client.0.smithi113.stdout:5/871: link d0/d10/d27/d1d/d3f/d73/d89/c7b d0/d10/d27/d1d/d8c/db8/c12b 0 2024-04-12T05:09:09.713 INFO:tasks.workunit.client.0.smithi113.stdout:3/730: link d0/d13/d47/f50 d0/dc6/ff5 0 2024-04-12T05:09:09.713 INFO:tasks.workunit.client.1.smithi192.stdout:4/763: mkdir d1/d54/d82/d71/d9e/dc4/dfe/d109 0 2024-04-12T05:09:09.715 INFO:tasks.workunit.client.1.smithi192.stdout:0/769: creat d0/d7b/d9b/d4b/d72/ffe x:0 0 0 2024-04-12T05:09:09.716 INFO:tasks.workunit.client.1.smithi192.stdout:0/770: write d0/d6/d67/db1/dd6/fe6 [108924,95428] 0 2024-04-12T05:09:09.719 INFO:tasks.workunit.client.0.smithi113.stdout:5/872: write d0/d10/d27/d1d/d8c/db8/f12a [21734,37228] 0 2024-04-12T05:09:09.720 INFO:tasks.workunit.client.0.smithi113.stdout:3/731: mknod d0/d13/d15/d28/d2a/d8b/cf6 0 2024-04-12T05:09:09.721 INFO:tasks.workunit.client.1.smithi192.stdout:1/697: creat d0/d1b/d44/d69/ddd/fe5 x:0 0 0 2024-04-12T05:09:09.721 INFO:tasks.workunit.client.1.smithi192.stdout:5/752: readlink d0/d15/d24/d29/lb4 0 2024-04-12T05:09:09.722 INFO:tasks.workunit.client.0.smithi113.stdout:5/873: dread - d0/d10/d27/f129 zero size 2024-04-12T05:09:09.722 INFO:tasks.workunit.client.0.smithi113.stdout:3/732: readlink d0/d13/d15/d28/d2a/d34/l5a 0 2024-04-12T05:09:09.723 INFO:tasks.workunit.client.0.smithi113.stdout:7/747: dwrite d0/fa2 [0,4194304] 0 2024-04-12T05:09:09.724 INFO:tasks.workunit.client.1.smithi192.stdout:1/698: symlink d0/d2/dd/d21/d62/le6 0 2024-04-12T05:09:09.724 INFO:tasks.workunit.client.1.smithi192.stdout:9/569: sync 2024-04-12T05:09:09.726 INFO:tasks.workunit.client.0.smithi113.stdout:7/748: write d0/dd/d11/f4a [2457183,118421] 0 2024-04-12T05:09:09.729 INFO:tasks.workunit.client.1.smithi192.stdout:5/753: link d0/d15/d3e/dca/dda/db5/dbe/fd8 d0/d15/da1/df9/f107 0 2024-04-12T05:09:09.730 INFO:tasks.workunit.client.1.smithi192.stdout:0/771: dread d0/d6/d15/d1b/d46/f8c [0,4194304] 0 2024-04-12T05:09:09.731 INFO:tasks.workunit.client.1.smithi192.stdout:1/699: chown d0/d2/dd/d3c/c5f 387 1 2024-04-12T05:09:09.731 INFO:tasks.workunit.client.0.smithi113.stdout:4/756: truncate da/d3c/d43/fc5 7379654 0 2024-04-12T05:09:09.731 INFO:tasks.workunit.client.0.smithi113.stdout:4/757: chown da/dd 0 1 2024-04-12T05:09:09.732 INFO:tasks.workunit.client.1.smithi192.stdout:0/772: fdatasync d0/d6/d15/f23 0 2024-04-12T05:09:09.733 INFO:tasks.workunit.client.1.smithi192.stdout:5/754: write d0/d15/d32/d69/d7d/fac [569324,42554] 0 2024-04-12T05:09:09.734 INFO:tasks.workunit.client.1.smithi192.stdout:5/755: readlink d0/d15/d32/l4a 0 2024-04-12T05:09:09.735 INFO:tasks.workunit.client.1.smithi192.stdout:0/773: readlink d0/d7b/d7c/lae 0 2024-04-12T05:09:09.735 INFO:tasks.workunit.client.1.smithi192.stdout:9/570: truncate d11/d25/d36/d6e/d33/f54 839271 0 2024-04-12T05:09:09.735 INFO:tasks.workunit.client.0.smithi113.stdout:5/874: creat d0/d10/f12c x:0 0 0 2024-04-12T05:09:09.735 INFO:tasks.workunit.client.0.smithi113.stdout:4/758: unlink da/d3c/d83/f40 0 2024-04-12T05:09:09.736 INFO:tasks.workunit.client.0.smithi113.stdout:3/733: getdents d0/d13/d15/d28/d2a/d34/d5f/dee 0 2024-04-12T05:09:09.736 INFO:tasks.workunit.client.1.smithi192.stdout:1/700: write d0/f24 [1740263,77397] 0 2024-04-12T05:09:09.736 INFO:tasks.workunit.client.0.smithi113.stdout:2/736: sync 2024-04-12T05:09:09.737 INFO:tasks.workunit.client.0.smithi113.stdout:6/740: sync 2024-04-12T05:09:09.737 INFO:tasks.workunit.client.1.smithi192.stdout:0/774: readlink d0/d6/l12 0 2024-04-12T05:09:09.739 INFO:tasks.workunit.client.1.smithi192.stdout:9/571: mkdir d11/d25/d36/d6e/dc7 0 2024-04-12T05:09:09.739 INFO:tasks.workunit.client.1.smithi192.stdout:1/701: fdatasync d0/d2/dd/d21/f38 0 2024-04-12T05:09:09.740 INFO:tasks.workunit.client.0.smithi113.stdout:5/875: chown d0/d10/d27/d1d/d3f/d73/db9/c111 240 1 2024-04-12T05:09:09.740 INFO:tasks.workunit.client.1.smithi192.stdout:0/775: mknod d0/d7b/d9b/d4b/dbc/dce/db0/cff 0 2024-04-12T05:09:09.740 INFO:tasks.workunit.client.1.smithi192.stdout:1/702: stat d0/d2/f7b 0 2024-04-12T05:09:09.742 INFO:tasks.workunit.client.0.smithi113.stdout:4/759: mkdir da/d3c/dfd 0 2024-04-12T05:09:09.744 INFO:tasks.workunit.client.0.smithi113.stdout:6/741: unlink d8/de/dd5/de7/ffd 0 2024-04-12T05:09:09.745 INFO:tasks.workunit.client.1.smithi192.stdout:9/572: symlink d11/d25/d60/lc8 0 2024-04-12T05:09:09.746 INFO:tasks.workunit.client.0.smithi113.stdout:3/734: dwrite d0/d13/d15/d28/d2a/d34/f43 [0,4194304] 0 2024-04-12T05:09:09.747 INFO:tasks.workunit.client.1.smithi192.stdout:5/756: getdents d0/d15 0 2024-04-12T05:09:09.748 INFO:tasks.workunit.client.0.smithi113.stdout:5/876: mkdir d0/d10/d27/d5f/d128/d12d 0 2024-04-12T05:09:09.749 INFO:tasks.workunit.client.1.smithi192.stdout:9/573: write d11/d25/d36/f38 [4091712,128217] 0 2024-04-12T05:09:09.749 INFO:tasks.workunit.client.0.smithi113.stdout:6/742: write d8/d45/ff3 [961435,51847] 0 2024-04-12T05:09:09.750 INFO:tasks.workunit.client.1.smithi192.stdout:9/574: dread - d11/d15/fc1 zero size 2024-04-12T05:09:09.755 INFO:tasks.workunit.client.1.smithi192.stdout:1/703: creat d0/d29/d8d/fe7 x:0 0 0 2024-04-12T05:09:09.757 INFO:tasks.workunit.client.0.smithi113.stdout:9/777: dread d1/df/d4b/d63/d3d/d6f/faf [0,4194304] 0 2024-04-12T05:09:09.758 INFO:tasks.workunit.client.1.smithi192.stdout:1/704: read d0/d2/dd/d3c/d5b/d74/d90/f94 [293668,40297] 0 2024-04-12T05:09:09.759 INFO:tasks.workunit.client.1.smithi192.stdout:0/776: link d0/d7b/d9b/d4b/d72/fd1 d0/d7b/d9b/f100 0 2024-04-12T05:09:09.759 INFO:tasks.workunit.client.0.smithi113.stdout:4/760: dwrite da/dd/d3a/fa3 [0,4194304] 0 2024-04-12T05:09:09.760 INFO:tasks.workunit.client.1.smithi192.stdout:1/705: dread - d0/d29/d8d/d96/fcd zero size 2024-04-12T05:09:09.761 INFO:tasks.workunit.client.1.smithi192.stdout:0/777: stat d0/d7b/d9b/d4b/dbc/dce/db0/dbf/fc7 0 2024-04-12T05:09:09.762 INFO:tasks.workunit.client.0.smithi113.stdout:3/735: write d0/d13/d47/d94/dcf/dd1/fe9 [51399,110684] 0 2024-04-12T05:09:09.764 INFO:tasks.workunit.client.1.smithi192.stdout:0/778: symlink d0/d7b/d7c/dbb/l101 0 2024-04-12T05:09:09.766 INFO:tasks.workunit.client.0.smithi113.stdout:6/743: write d8/de/d13/d11/d36/d3d/d4c/d31/f62 [773661,122856] 0 2024-04-12T05:09:09.766 INFO:tasks.workunit.client.1.smithi192.stdout:0/779: readlink d0/d6/d15/d19/lc6 0 2024-04-12T05:09:09.767 INFO:tasks.workunit.client.1.smithi192.stdout:5/757: dwrite d0/f3d [0,4194304] 0 2024-04-12T05:09:09.771 INFO:tasks.workunit.client.0.smithi113.stdout:6/744: fdatasync d8/de/d13/d11/d36/d3d/f90 0 2024-04-12T05:09:09.772 INFO:tasks.workunit.client.1.smithi192.stdout:8/718: write d6/d10/d46/d5e/d5f/d37/f48 [831914,59606] 0 2024-04-12T05:09:09.773 INFO:tasks.workunit.client.0.smithi113.stdout:0/873: dwrite d1/d6/d8e/fec [0,4194304] 0 2024-04-12T05:09:09.773 INFO:tasks.workunit.client.0.smithi113.stdout:4/761: creat da/d3c/da5/dad/de1/ffe x:0 0 0 2024-04-12T05:09:09.774 INFO:tasks.workunit.client.1.smithi192.stdout:5/758: creat d0/d15/d3e/dca/dda/da7/f108 x:0 0 0 2024-04-12T05:09:09.775 INFO:tasks.workunit.client.0.smithi113.stdout:0/874: stat d1/d43/d59/df4/f107 0 2024-04-12T05:09:09.776 INFO:tasks.workunit.client.1.smithi192.stdout:1/706: dwrite d0/d1b/d44/d69/fb7 [0,4194304] 0 2024-04-12T05:09:09.776 INFO:tasks.workunit.client.1.smithi192.stdout:5/759: stat d0/d15/d32/d69/d7d/ldc 0 2024-04-12T05:09:09.782 INFO:tasks.workunit.client.0.smithi113.stdout:0/875: read d1/d43/d59/de9/d105/d50/f77 [93783,4500] 0 2024-04-12T05:09:09.782 INFO:tasks.workunit.client.0.smithi113.stdout:3/736: truncate d0/d13/d47/f78 724833 0 2024-04-12T05:09:09.782 INFO:tasks.workunit.client.1.smithi192.stdout:0/780: creat d0/d7b/d9b/d4b/dbc/dce/d95/db4/f102 x:0 0 0 2024-04-12T05:09:09.782 INFO:tasks.workunit.client.1.smithi192.stdout:4/764: write d1/d1d/d53/d1c/f65 [1386067,42740] 0 2024-04-12T05:09:09.783 INFO:tasks.workunit.client.0.smithi113.stdout:5/877: truncate d0/d10/d27/d1d/d3f/d73/fff 3186349 0 2024-04-12T05:09:09.783 INFO:tasks.workunit.client.0.smithi113.stdout:6/745: creat d8/de/d13/d11/f110 x:0 0 0 2024-04-12T05:09:09.785 INFO:tasks.workunit.client.1.smithi192.stdout:0/781: truncate d0/d6/db2/ff8 537597 0 2024-04-12T05:09:09.785 INFO:tasks.workunit.client.0.smithi113.stdout:4/762: symlink da/d3c/d83/d47/lff 0 2024-04-12T05:09:09.786 INFO:tasks.workunit.client.0.smithi113.stdout:9/778: truncate d1/d4/d5/f12 2417047 0 2024-04-12T05:09:09.788 INFO:tasks.workunit.client.1.smithi192.stdout:0/782: dread - d0/d6/faa zero size 2024-04-12T05:09:09.790 INFO:tasks.workunit.client.1.smithi192.stdout:1/707: read d0/d2/dd/d21/f43 [28919,23900] 0 2024-04-12T05:09:09.790 INFO:tasks.workunit.client.1.smithi192.stdout:8/719: write d6/d10/d46/d5e/d5f/d31/d80/db1/fb6 [2604066,44762] 0 2024-04-12T05:09:09.792 INFO:tasks.workunit.client.1.smithi192.stdout:8/720: chown d6/d1f/d9c/fb0 5 1 2024-04-12T05:09:09.793 INFO:tasks.workunit.client.0.smithi113.stdout:4/763: rmdir da/d3c/d9f 39 2024-04-12T05:09:09.794 INFO:tasks.workunit.client.0.smithi113.stdout:7/749: dread d0/dd/f30 [0,4194304] 0 2024-04-12T05:09:09.795 INFO:tasks.workunit.client.1.smithi192.stdout:4/765: dread - d1/d1d/d53/d1c/d84/fa3 zero size 2024-04-12T05:09:09.796 INFO:tasks.workunit.client.0.smithi113.stdout:6/746: mknod d8/de/d17/c111 0 2024-04-12T05:09:09.797 INFO:tasks.workunit.client.0.smithi113.stdout:9/779: dread d1/d4/d5/fb9 [0,4194304] 0 2024-04-12T05:09:09.798 INFO:tasks.workunit.client.0.smithi113.stdout:4/764: dread - da/dd/d3a/fda zero size 2024-04-12T05:09:09.799 INFO:tasks.workunit.client.1.smithi192.stdout:5/760: creat d0/d15/d3e/dca/f109 x:0 0 0 2024-04-12T05:09:09.801 INFO:tasks.workunit.client.0.smithi113.stdout:2/737: dread d8/d9/de/d15/d24/f3b [0,4194304] 0 2024-04-12T05:09:09.802 INFO:tasks.workunit.client.0.smithi113.stdout:5/878: dwrite d0/d10/d6e/fb5 [0,4194304] 0 2024-04-12T05:09:09.805 INFO:tasks.workunit.client.0.smithi113.stdout:4/765: chown da/d3c/d83/d47/ccb 208624 1 2024-04-12T05:09:09.806 INFO:tasks.workunit.client.0.smithi113.stdout:3/737: symlink d0/d13/d15/d28/d2a/d34/d5f/d7b/de0/lf7 0 2024-04-12T05:09:09.806 INFO:tasks.workunit.client.1.smithi192.stdout:5/761: truncate d0/d21/f33 2024784 0 2024-04-12T05:09:09.808 INFO:tasks.workunit.client.0.smithi113.stdout:4/766: chown da/d3c/d83/d47/la1 146 1 2024-04-12T05:09:09.808 INFO:tasks.workunit.client.0.smithi113.stdout:2/738: write d8/d9/d27/f65 [18531,58389] 0 2024-04-12T05:09:09.809 INFO:tasks.workunit.client.0.smithi113.stdout:7/750: creat d0/dc/da9/fff x:0 0 0 2024-04-12T05:09:09.809 INFO:tasks.workunit.client.1.smithi192.stdout:5/762: mknod d0/d15/d32/d45/d106/c10a 0 2024-04-12T05:09:09.809 INFO:tasks.workunit.client.1.smithi192.stdout:0/783: link d0/c9 d0/d6/d67/c103 0 2024-04-12T05:09:09.811 INFO:tasks.workunit.client.0.smithi113.stdout:3/738: write d0/d13/f60 [234343,99931] 0 2024-04-12T05:09:09.814 INFO:tasks.workunit.client.0.smithi113.stdout:6/747: write d8/de/d13/df0/d66/f106 [40529,87540] 0 2024-04-12T05:09:09.814 INFO:tasks.workunit.client.0.smithi113.stdout:5/879: mkdir d0/d10/d27/d1d/d38/d44/d4e/daa/dbb/d12e 0 2024-04-12T05:09:09.819 INFO:tasks.workunit.client.0.smithi113.stdout:9/780: dwrite d1/df/d29/fa2 [0,4194304] 0 2024-04-12T05:09:09.821 INFO:tasks.workunit.client.1.smithi192.stdout:0/784: dwrite d0/d6/d67/db1/fe9 [0,4194304] 0 2024-04-12T05:09:09.822 INFO:tasks.workunit.client.0.smithi113.stdout:7/751: unlink d0/dd/d11/dd5/fdd 0 2024-04-12T05:09:09.824 INFO:tasks.workunit.client.1.smithi192.stdout:1/708: dread d0/f24 [0,4194304] 0 2024-04-12T05:09:09.826 INFO:tasks.workunit.client.0.smithi113.stdout:6/748: creat d8/de/d13/d11/d36/d53/db7/f112 x:0 0 0 2024-04-12T05:09:09.829 INFO:tasks.workunit.client.1.smithi192.stdout:4/766: dread d1/d54/f94 [0,4194304] 0 2024-04-12T05:09:09.830 INFO:tasks.workunit.client.0.smithi113.stdout:2/739: getdents d8/d9/de/d12/d2b/d79 0 2024-04-12T05:09:09.831 INFO:tasks.workunit.client.1.smithi192.stdout:1/709: symlink d0/d1b/d44/le8 0 2024-04-12T05:09:09.832 INFO:tasks.workunit.client.0.smithi113.stdout:4/767: link da/d3c/d83/d47/lbf da/d3c/l100 0 2024-04-12T05:09:09.832 INFO:tasks.workunit.client.0.smithi113.stdout:5/880: dwrite d0/d10/d27/d1d/d38/d44/d4e/f64 [0,4194304] 0 2024-04-12T05:09:09.834 INFO:tasks.workunit.client.0.smithi113.stdout:3/739: rename d0/d7e/cbe to d0/d13/d47/cf8 0 2024-04-12T05:09:09.835 INFO:tasks.workunit.client.0.smithi113.stdout:9/781: dwrite d1/f46 [4194304,4194304] 0 2024-04-12T05:09:09.836 INFO:tasks.workunit.client.1.smithi192.stdout:0/785: dwrite d0/d6/d67/db1/ff7 [0,4194304] 0 2024-04-12T05:09:09.837 INFO:tasks.workunit.client.0.smithi113.stdout:0/876: dread d1/d6/d16/d8f/f103 [0,4194304] 0 2024-04-12T05:09:09.838 INFO:tasks.workunit.client.1.smithi192.stdout:0/786: chown d0/d6/d15/d1b/d3a 3 1 2024-04-12T05:09:09.840 INFO:tasks.workunit.client.0.smithi113.stdout:3/740: write d0/d13/d15/d28/d2a/d34/d5f/fe1 [91418,257] 0 2024-04-12T05:09:09.842 INFO:tasks.workunit.client.0.smithi113.stdout:5/881: rename d0/d10/d27/d1d/d38/d44 to d0/d10/d27/d1d/d38/d44/d66/d10f/d12f 22 2024-04-12T05:09:09.843 INFO:tasks.workunit.client.1.smithi192.stdout:5/763: dread d0/d15/d3e/d77/f78 [0,4194304] 0 2024-04-12T05:09:09.844 INFO:tasks.workunit.client.0.smithi113.stdout:0/877: write d1/d6/d34/f78 [4362219,97918] 0 2024-04-12T05:09:09.850 INFO:tasks.workunit.client.0.smithi113.stdout:2/740: dwrite d8/d9/de/d43/f5c [0,4194304] 0 2024-04-12T05:09:09.853 INFO:tasks.workunit.client.0.smithi113.stdout:2/741: chown d8/d16/f4e 111883 1 2024-04-12T05:09:09.855 INFO:tasks.workunit.client.1.smithi192.stdout:5/764: unlink d0/d15/d3e/dca/dd3/lea 0 2024-04-12T05:09:09.859 INFO:tasks.workunit.client.0.smithi113.stdout:2/742: mknod d8/d16/ce4 0 2024-04-12T05:09:09.861 INFO:tasks.workunit.client.0.smithi113.stdout:2/743: chown d8/d16/f17 1026092112 1 2024-04-12T05:09:09.862 INFO:tasks.workunit.client.0.smithi113.stdout:6/749: dread d8/de/f16 [0,4194304] 0 2024-04-12T05:09:09.866 INFO:tasks.workunit.client.0.smithi113.stdout:6/750: rename d8/de/d13/d11/d36/d3d/d4c/d31/l39 to d8/de/dd5/l113 0 2024-04-12T05:09:09.871 INFO:tasks.workunit.client.1.smithi192.stdout:1/710: dread d0/d2/dd/d3c/d5b/db6/fd3 [0,4194304] 0 2024-04-12T05:09:09.873 INFO:tasks.workunit.client.1.smithi192.stdout:1/711: creat d0/d1b/d28/d9d/fe9 x:0 0 0 2024-04-12T05:09:09.876 INFO:tasks.workunit.client.1.smithi192.stdout:1/712: rename d0/d2/dd/d21/d39/l92 to d0/d29/dbf/lea 0 2024-04-12T05:09:09.877 INFO:tasks.workunit.client.1.smithi192.stdout:1/713: fsync d0/d2/d1e/d3f/d78/fe1 0 2024-04-12T05:09:09.879 INFO:tasks.workunit.client.1.smithi192.stdout:1/714: mkdir d0/d2/dd/d3c/d5b/db6/deb 0 2024-04-12T05:09:09.919 INFO:tasks.workunit.client.0.smithi113.stdout:1/858: sync 2024-04-12T05:09:09.920 INFO:tasks.workunit.client.0.smithi113.stdout:6/751: rename d8/de/f16 to d8/de/d13/d11/d36/d3d/d4c/d26/f114 0 2024-04-12T05:09:09.922 INFO:tasks.workunit.client.1.smithi192.stdout:4/767: truncate d1/d49/f5b 1566976 0 2024-04-12T05:09:09.924 INFO:tasks.workunit.client.0.smithi113.stdout:9/782: write d1/df/db6/f84 [3396011,120271] 0 2024-04-12T05:09:09.925 INFO:tasks.workunit.client.0.smithi113.stdout:3/741: write d0/d13/d15/d28/d2a/fbb [200332,24452] 0 2024-04-12T05:09:09.926 INFO:tasks.workunit.client.0.smithi113.stdout:6/752: chown d8/de/d13/d11/d36/d3d/d4c/d26/d108/ld6 227744 1 2024-04-12T05:09:09.927 INFO:tasks.workunit.client.0.smithi113.stdout:1/859: write d10/d2e/d31/dc3/d109/fe7 [3513430,60540] 0 2024-04-12T05:09:09.928 INFO:tasks.workunit.client.1.smithi192.stdout:2/709: sync 2024-04-12T05:09:09.928 INFO:tasks.workunit.client.0.smithi113.stdout:6/753: write d8/d1d/fcd [270567,2764] 0 2024-04-12T05:09:09.929 INFO:tasks.workunit.client.1.smithi192.stdout:0/787: dwrite d0/f16 [0,4194304] 0 2024-04-12T05:09:09.930 INFO:tasks.workunit.client.0.smithi113.stdout:3/742: mknod d0/d13/d47/d94/cf9 0 2024-04-12T05:09:09.932 INFO:tasks.workunit.client.1.smithi192.stdout:1/715: write d0/d1b/d28/d9d/f8b [2257769,45171] 0 2024-04-12T05:09:09.933 INFO:tasks.workunit.client.1.smithi192.stdout:5/765: dwrite d0/d15/d3e/d77/ffd [4194304,4194304] 0 2024-04-12T05:09:09.934 INFO:tasks.workunit.client.0.smithi113.stdout:5/882: dread d0/d10/d27/d1d/d38/d44/d66/f96 [0,4194304] 0 2024-04-12T05:09:09.936 INFO:tasks.workunit.client.0.smithi113.stdout:4/768: dread da/d3c/f95 [0,4194304] 0 2024-04-12T05:09:09.936 INFO:tasks.workunit.client.0.smithi113.stdout:1/860: unlink d10/d2e/d31/d5c/l7b 0 2024-04-12T05:09:09.939 INFO:tasks.workunit.client.0.smithi113.stdout:3/743: symlink d0/d13/d15/d28/d2a/d34/d5f/d7b/de0/lfa 0 2024-04-12T05:09:09.941 INFO:tasks.workunit.client.0.smithi113.stdout:1/861: chown d10/d2e/d31/d5c/c83 3936581 1 2024-04-12T05:09:09.944 INFO:tasks.workunit.client.0.smithi113.stdout:4/769: mknod da/d24/d7e/d79/dcd/c101 0 2024-04-12T05:09:09.944 INFO:tasks.workunit.client.1.smithi192.stdout:0/788: chown d0/l74 159 1 2024-04-12T05:09:09.944 INFO:tasks.workunit.client.1.smithi192.stdout:1/716: fsync d0/d1b/d28/f81 0 2024-04-12T05:09:09.944 INFO:tasks.workunit.client.0.smithi113.stdout:7/752: dread d0/dd/d11/f19 [0,4194304] 0 2024-04-12T05:09:09.946 INFO:tasks.workunit.client.0.smithi113.stdout:6/754: dwrite d8/de/d13/d11/d36/d89/fe1 [0,4194304] 0 2024-04-12T05:09:09.949 INFO:tasks.workunit.client.1.smithi192.stdout:0/789: chown d0/d7b/d9b/d4b/dbc/dce/d53/df3 6 1 2024-04-12T05:09:09.950 INFO:tasks.workunit.client.0.smithi113.stdout:9/783: dread d1/df/d29/fbc [0,4194304] 0 2024-04-12T05:09:09.950 INFO:tasks.workunit.client.0.smithi113.stdout:1/862: unlink d10/da5/f113 0 2024-04-12T05:09:09.951 INFO:tasks.workunit.client.0.smithi113.stdout:3/744: symlink d0/d13/d15/d28/d2a/lfb 0 2024-04-12T05:09:09.953 INFO:tasks.workunit.client.0.smithi113.stdout:2/744: dread d8/d9/d27/d88/f8b [0,4194304] 0 2024-04-12T05:09:09.953 INFO:tasks.workunit.client.0.smithi113.stdout:4/770: symlink da/d3c/dd5/l102 0 2024-04-12T05:09:09.954 INFO:tasks.workunit.client.1.smithi192.stdout:4/768: dwrite d1/d54/f6e [0,4194304] 0 2024-04-12T05:09:09.957 INFO:tasks.workunit.client.0.smithi113.stdout:1/863: symlink d10/d14/d95/dcf/l115 0 2024-04-12T05:09:09.958 INFO:tasks.workunit.client.1.smithi192.stdout:2/710: dwrite d1/d8/d3a/de/d7c/fb8 [0,4194304] 0 2024-04-12T05:09:09.958 INFO:tasks.workunit.client.0.smithi113.stdout:4/771: fdatasync da/d24/d60/fb2 0 2024-04-12T05:09:09.960 INFO:tasks.workunit.client.1.smithi192.stdout:5/766: dwrite d0/d15/da1/fa8 [0,4194304] 0 2024-04-12T05:09:09.961 INFO:tasks.workunit.client.0.smithi113.stdout:7/753: rename d0/dc/de7/lf1 to d0/dc/da9/d5c/ddf/l100 0 2024-04-12T05:09:09.963 INFO:tasks.workunit.client.1.smithi192.stdout:1/717: creat d0/fec x:0 0 0 2024-04-12T05:09:09.964 INFO:tasks.workunit.client.0.smithi113.stdout:6/755: creat d8/de/d13/d11/d36/f115 x:0 0 0 2024-04-12T05:09:09.964 INFO:tasks.workunit.client.0.smithi113.stdout:1/864: mkdir d10/d41/d89/d96/d7d/d116 0 2024-04-12T05:09:09.968 INFO:tasks.workunit.client.0.smithi113.stdout:9/784: mkdir d1/d4/d38/d65/dcd/deb/d108 0 2024-04-12T05:09:09.969 INFO:tasks.workunit.client.1.smithi192.stdout:4/769: truncate d1/d1d/d53/d1c/f66 249185 0 2024-04-12T05:09:09.969 INFO:tasks.workunit.client.1.smithi192.stdout:0/790: dwrite d0/d7b/d9b/d4b/dbc/dce/f65 [0,4194304] 0 2024-04-12T05:09:09.970 INFO:tasks.workunit.client.0.smithi113.stdout:4/772: fsync da/dd/d3a/f5b 0 2024-04-12T05:09:09.975 INFO:tasks.workunit.client.0.smithi113.stdout:3/745: dwrite d0/d13/d47/d94/dcf/dd1/de2/fe6 [0,4194304] 0 2024-04-12T05:09:09.978 INFO:tasks.workunit.client.1.smithi192.stdout:5/767: mknod d0/d15/d3e/dca/c10b 0 2024-04-12T05:09:09.982 INFO:tasks.workunit.client.0.smithi113.stdout:5/883: link d0/d10/d27/d1d/d3f/d7e/df8/l104 d0/d10/d27/d1d/d38/d3b/d6b/l130 0 2024-04-12T05:09:09.982 INFO:tasks.workunit.client.0.smithi113.stdout:9/785: chown d1/df/d4b/d63/d3d/d6f/lc1 3701 1 2024-04-12T05:09:09.983 INFO:tasks.workunit.client.1.smithi192.stdout:2/711: truncate d1/d8/d17/d19/d40/f4c 175270 0 2024-04-12T05:09:09.984 INFO:tasks.workunit.client.0.smithi113.stdout:7/754: dwrite d0/dc/d6/d9b/f55 [0,4194304] 0 2024-04-12T05:09:09.986 INFO:tasks.workunit.client.0.smithi113.stdout:8/816: sync 2024-04-12T05:09:09.987 INFO:tasks.workunit.client.0.smithi113.stdout:1/865: dwrite d10/d41/d89/d96/dd3/f50 [0,4194304] 0 2024-04-12T05:09:09.989 INFO:tasks.workunit.client.1.smithi192.stdout:1/718: dwrite d0/d1b/d44/d69/ddd/fe5 [0,4194304] 0 2024-04-12T05:09:09.989 INFO:tasks.workunit.client.1.smithi192.stdout:0/791: chown d0/d6/d15/d1b/f80 3157 1 2024-04-12T05:09:09.991 INFO:tasks.workunit.client.0.smithi113.stdout:6/756: rename d8/de/d13/d11/fd3 to d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f116 0 2024-04-12T05:09:09.994 INFO:tasks.workunit.client.0.smithi113.stdout:5/884: stat d0/d10/d27/d1d/d8c/db8/cfc 0 2024-04-12T05:09:09.996 INFO:tasks.workunit.client.0.smithi113.stdout:5/885: write d0/d10/d27/d1d/d38/d44/fbc [786144,78061] 0 2024-04-12T05:09:09.998 INFO:tasks.workunit.client.1.smithi192.stdout:2/712: mknod d1/da7/cdf 0 2024-04-12T05:09:10.001 INFO:tasks.workunit.client.1.smithi192.stdout:1/719: read d0/d2/dd/f32 [148889,47486] 0 2024-04-12T05:09:10.007 INFO:tasks.workunit.client.1.smithi192.stdout:5/768: dwrite d0/d15/d64/fb6 [0,4194304] 0 2024-04-12T05:09:10.007 INFO:tasks.workunit.client.1.smithi192.stdout:5/769: readlink d0/d15/l22 0 2024-04-12T05:09:10.008 INFO:tasks.workunit.client.1.smithi192.stdout:0/792: creat d0/d7b/d9b/d4b/dbc/dce/d53/df3/d2e/d59/f104 x:0 0 0 2024-04-12T05:09:10.008 INFO:tasks.workunit.client.1.smithi192.stdout:4/770: dwrite d1/d1d/d53/d1c/d95/dc6/ff9 [4194304,4194304] 0 2024-04-12T05:09:10.011 INFO:tasks.workunit.client.1.smithi192.stdout:1/720: creat d0/d2/dd/d3c/d5b/da7/dd7/fed x:0 0 0 2024-04-12T05:09:10.014 INFO:tasks.workunit.client.0.smithi113.stdout:8/817: creat d9/dd4/f107 x:0 0 0 2024-04-12T05:09:10.016 INFO:tasks.workunit.client.1.smithi192.stdout:2/713: creat d1/d8/d17/d19/d60/fe0 x:0 0 0 2024-04-12T05:09:10.021 INFO:tasks.workunit.client.0.smithi113.stdout:5/886: fdatasync d0/fc 0 2024-04-12T05:09:10.023 INFO:tasks.workunit.client.1.smithi192.stdout:1/721: rename d0/d1b/d44/d69/fa2 to d0/d2/dd/d21/d33/dd8/fee 0 2024-04-12T05:09:10.024 INFO:tasks.workunit.client.1.smithi192.stdout:4/771: getdents d1/d1d/d53/d1c 0 2024-04-12T05:09:10.024 INFO:tasks.workunit.client.0.smithi113.stdout:4/773: getdents da/d4f 0 2024-04-12T05:09:10.025 INFO:tasks.workunit.client.1.smithi192.stdout:2/714: write d1/d8/d17/d19/f5e [2146044,95907] 0 2024-04-12T05:09:10.026 INFO:tasks.workunit.client.1.smithi192.stdout:5/770: fsync d0/d15/d3e/dca/dda/da7/f105 0 2024-04-12T05:09:10.027 INFO:tasks.workunit.client.0.smithi113.stdout:2/745: dread d8/d9/de/d15/f9d [0,4194304] 0 2024-04-12T05:09:10.030 INFO:tasks.workunit.client.0.smithi113.stdout:8/818: creat d9/d1b/f108 x:0 0 0 2024-04-12T05:09:10.031 INFO:tasks.workunit.client.0.smithi113.stdout:2/746: dread - d8/d9/d27/d88/fa3 zero size 2024-04-12T05:09:10.031 INFO:tasks.workunit.client.0.smithi113.stdout:5/887: dwrite d0/d10/d27/d1d/d38/d44/d4e/f64 [0,4194304] 0 2024-04-12T05:09:10.035 INFO:tasks.workunit.client.0.smithi113.stdout:5/888: creat d0/d10/d27/d1d/d38/d75/f131 x:0 0 0 2024-04-12T05:09:10.037 INFO:tasks.workunit.client.1.smithi192.stdout:2/715: dread d1/d8/d17/d19/d60/d9c/f9f [0,4194304] 0 2024-04-12T05:09:10.038 INFO:tasks.workunit.client.1.smithi192.stdout:4/772: link d1/d54/ld3 d1/l10a 0 2024-04-12T05:09:10.038 INFO:tasks.workunit.client.1.smithi192.stdout:1/722: dwrite d0/d2/d1e/d3f/d56/f79 [0,4194304] 0 2024-04-12T05:09:10.038 INFO:tasks.workunit.client.0.smithi113.stdout:2/747: rename d8/d9/de/f10 to d8/d9/de/d12/d2b/d79/ddf/fe5 0 2024-04-12T05:09:10.039 INFO:tasks.workunit.client.1.smithi192.stdout:2/716: chown d1/d8/lc5 287060 1 2024-04-12T05:09:10.040 INFO:tasks.workunit.client.0.smithi113.stdout:4/774: read da/d3c/d43/fc5 [5842249,72797] 0 2024-04-12T05:09:10.040 INFO:tasks.workunit.client.0.smithi113.stdout:8/819: truncate d9/d16/d3c/f74 2798067 0 2024-04-12T05:09:10.042 INFO:tasks.workunit.client.0.smithi113.stdout:8/820: chown d9/d16/d7f/d3a/d6c/d90/ff5 161358393 1 2024-04-12T05:09:10.042 INFO:tasks.workunit.client.1.smithi192.stdout:1/723: symlink d0/d29/lef 0 2024-04-12T05:09:10.044 INFO:tasks.workunit.client.1.smithi192.stdout:4/773: rmdir d1/d1d/d53/d1c/dd1/def/ddd/dea 39 2024-04-12T05:09:10.044 INFO:tasks.workunit.client.0.smithi113.stdout:2/748: mknod d8/d9/d27/d88/d91/ce6 0 2024-04-12T05:09:10.045 INFO:tasks.workunit.client.1.smithi192.stdout:2/717: symlink d1/d8/d3a/de/le1 0 2024-04-12T05:09:10.047 INFO:tasks.workunit.client.1.smithi192.stdout:1/724: creat d0/d2/d9f/ff0 x:0 0 0 2024-04-12T05:09:10.050 INFO:tasks.workunit.client.0.smithi113.stdout:8/821: read d9/f9e [416965,12071] 0 2024-04-12T05:09:10.051 INFO:tasks.workunit.client.1.smithi192.stdout:1/725: mknod d0/d2/d1e/cf1 0 2024-04-12T05:09:10.052 INFO:tasks.workunit.client.0.smithi113.stdout:5/889: rename d0/d10/d27/d1d/d3f/d73/d89/f5a to d0/d10/d27/d1d/d38/f132 0 2024-04-12T05:09:10.052 INFO:tasks.workunit.client.1.smithi192.stdout:4/774: dread d1/d54/d82/d71/fb0 [0,4194304] 0 2024-04-12T05:09:10.053 INFO:tasks.workunit.client.1.smithi192.stdout:2/718: mkdir d1/d8/de2 0 2024-04-12T05:09:10.053 INFO:tasks.workunit.client.0.smithi113.stdout:4/775: creat da/f103 x:0 0 0 2024-04-12T05:09:10.054 INFO:tasks.workunit.client.0.smithi113.stdout:7/755: dread d0/dc/d6/d9b/ff3 [0,4194304] 0 2024-04-12T05:09:10.055 INFO:tasks.workunit.client.0.smithi113.stdout:1/866: dread d10/d14/f1f [0,4194304] 0 2024-04-12T05:09:10.055 INFO:tasks.workunit.client.0.smithi113.stdout:2/749: mknod d8/d9/de/d12/ce7 0 2024-04-12T05:09:10.056 INFO:tasks.workunit.client.0.smithi113.stdout:5/890: write d0/d10/d27/d1d/d38/d44/fbc [2409720,71905] 0 2024-04-12T05:09:10.060 INFO:tasks.workunit.client.0.smithi113.stdout:2/750: write d8/d9/de/d12/f78 [4746426,126605] 0 2024-04-12T05:09:10.061 INFO:tasks.workunit.client.1.smithi192.stdout:2/719: fsync d1/d8/d17/d19/d60/d59/da3/fde 0 2024-04-12T05:09:10.063 INFO:tasks.workunit.client.0.smithi113.stdout:5/891: symlink d0/d10/d27/d1d/d10c/l133 0 2024-04-12T05:09:10.064 INFO:tasks.workunit.client.1.smithi192.stdout:3/672: sync 2024-04-12T05:09:10.064 INFO:tasks.workunit.client.1.smithi192.stdout:6/751: sync 2024-04-12T05:09:10.064 INFO:tasks.workunit.client.1.smithi192.stdout:7/638: sync 2024-04-12T05:09:10.066 INFO:tasks.workunit.client.1.smithi192.stdout:3/673: readlink d5/d13/d3c/d36/l3e 0 2024-04-12T05:09:10.067 INFO:tasks.workunit.client.0.smithi113.stdout:3/746: dwrite d0/d13/d47/f50 [0,4194304] 0 2024-04-12T05:09:10.068 INFO:tasks.workunit.client.0.smithi113.stdout:2/751: creat d8/d9/d1a/fe8 x:0 0 0 2024-04-12T05:09:10.068 INFO:tasks.workunit.client.1.smithi192.stdout:4/775: rename d1/d6b to d1/d54/d82/d71/d9e/d10b 0 2024-04-12T05:09:10.070 INFO:tasks.workunit.client.1.smithi192.stdout:0/793: write d0/d7b/d9b/d4b/fee [1359304,89505] 0 2024-04-12T05:09:10.070 INFO:tasks.workunit.client.0.smithi113.stdout:3/747: fdatasync d0/d13/d15/d28/d2a/fbb 0 2024-04-12T05:09:10.072 INFO:tasks.workunit.client.1.smithi192.stdout:0/794: fsync d0/d6/d67/db1/dd6/fe6 0 2024-04-12T05:09:10.075 INFO:tasks.workunit.client.0.smithi113.stdout:4/776: link da/d3c/d9f/fba da/d3c/da5/dad/de1/f104 0 2024-04-12T05:09:10.076 INFO:tasks.workunit.client.0.smithi113.stdout:5/892: creat d0/d10/d27/d1d/d3f/d7e/f134 x:0 0 0 2024-04-12T05:09:10.077 INFO:tasks.workunit.client.1.smithi192.stdout:4/776: readlink d1/d54/ld3 0 2024-04-12T05:09:10.078 INFO:tasks.workunit.client.0.smithi113.stdout:6/757: dwrite d8/de/dd5/de7/db3/dc5/f103 [0,4194304] 0 2024-04-12T05:09:10.079 INFO:tasks.workunit.client.0.smithi113.stdout:5/893: fsync d0/d10/d6e/fb5 0 2024-04-12T05:09:10.080 INFO:tasks.workunit.client.1.smithi192.stdout:7/639: dwrite d3/d11/fbe [0,4194304] 0 2024-04-12T05:09:10.081 INFO:tasks.workunit.client.1.smithi192.stdout:3/674: rename d5/d13/d3a/d4a/d61/d9f/fc3 to d5/d13/d1a/fef 0 2024-04-12T05:09:10.081 INFO:tasks.workunit.client.0.smithi113.stdout:3/748: rmdir d0/d13/dbf 39 2024-04-12T05:09:10.081 INFO:tasks.workunit.client.0.smithi113.stdout:2/752: symlink d8/d9/de/d12/d44/le9 0 2024-04-12T05:09:10.083 INFO:tasks.workunit.client.1.smithi192.stdout:2/720: creat d1/d2d/fe3 x:0 0 0 2024-04-12T05:09:10.084 INFO:tasks.workunit.client.1.smithi192.stdout:7/640: fdatasync d3/d7/d15/d22/fb6 0 2024-04-12T05:09:10.084 INFO:tasks.workunit.client.1.smithi192.stdout:5/771: truncate d0/d15/d24/d29/f2a 4308639 0 2024-04-12T05:09:10.084 INFO:tasks.workunit.client.0.smithi113.stdout:6/758: read d8/d45/ff3 [2636702,72946] 0 2024-04-12T05:09:10.086 INFO:tasks.workunit.client.1.smithi192.stdout:1/726: dread d0/d1b/d28/d9d/f8b [0,4194304] 0 2024-04-12T05:09:10.087 INFO:tasks.workunit.client.0.smithi113.stdout:4/777: truncate da/dd/d3a/f42 3325635 0 2024-04-12T05:09:10.091 INFO:tasks.workunit.client.0.smithi113.stdout:7/756: dread d0/dc/d6/d9b/f25 [0,4194304] 0 2024-04-12T05:09:10.093 INFO:tasks.workunit.client.1.smithi192.stdout:7/641: mkdir d3/d7/d15/d1a/ddc 0 2024-04-12T05:09:10.093 INFO:tasks.workunit.client.0.smithi113.stdout:3/749: creat d0/d13/d15/d28/d2a/d8b/d91/ffc x:0 0 0 2024-04-12T05:09:10.094 INFO:tasks.workunit.client.0.smithi113.stdout:7/757: read d0/dc/f41 [632350,46563] 0 2024-04-12T05:09:10.094 INFO:tasks.workunit.client.0.smithi113.stdout:4/778: fdatasync da/d4f/f5d 0 2024-04-12T05:09:10.094 INFO:tasks.workunit.client.1.smithi192.stdout:1/727: dread d0/d1b/d44/d69/fb7 [0,4194304] 0 2024-04-12T05:09:10.095 INFO:tasks.workunit.client.0.smithi113.stdout:5/894: dwrite d0/d10/d6e/f4f [0,4194304] 0 2024-04-12T05:09:10.095 INFO:tasks.workunit.client.1.smithi192.stdout:0/795: mknod d0/d6/d15/c105 0 2024-04-12T05:09:10.097 INFO:tasks.workunit.client.1.smithi192.stdout:5/772: mknod d0/d15/d3e/d77/c10c 0 2024-04-12T05:09:10.097 INFO:tasks.workunit.client.0.smithi113.stdout:6/759: mknod d8/de/d13/d11/d36/d3d/d4c/d26/d108/c117 0 2024-04-12T05:09:10.098 INFO:tasks.workunit.client.1.smithi192.stdout:2/721: creat d1/d8/d3a/de/d7c/d98/fe4 x:0 0 0 2024-04-12T05:09:10.103 INFO:tasks.workunit.client.0.smithi113.stdout:1/867: truncate d10/d14/f4d 201371 0 2024-04-12T05:09:10.103 INFO:tasks.workunit.client.0.smithi113.stdout:3/750: mknod d0/d13/d47/d4f/dd3/cfd 0 2024-04-12T05:09:10.104 INFO:tasks.workunit.client.0.smithi113.stdout:4/779: mknod da/d3c/d9f/c105 0 2024-04-12T05:09:10.105 INFO:tasks.workunit.client.1.smithi192.stdout:7/642: symlink d3/d1d/d74/d86/ldd 0 2024-04-12T05:09:10.105 INFO:tasks.workunit.client.1.smithi192.stdout:1/728: dwrite d0/d2/dd/d3c/d5b/d74/d90/fc4 [4194304,4194304] 0 2024-04-12T05:09:10.106 INFO:tasks.workunit.client.0.smithi113.stdout:2/753: dread d8/d9/de/d12/d2b/f8e [0,4194304] 0 2024-04-12T05:09:10.109 INFO:tasks.workunit.client.0.smithi113.stdout:6/760: dread - d8/de/dd5/de7/db3/fb6 zero size 2024-04-12T05:09:10.109 INFO:tasks.workunit.client.1.smithi192.stdout:0/796: fdatasync d0/d6/d15/d1b/d3a/f76 0 2024-04-12T05:09:10.114 INFO:tasks.workunit.client.1.smithi192.stdout:5/773: mknod d0/d15/d3e/dca/dda/db5/dd7/c10d 0 2024-04-12T05:09:10.114 INFO:tasks.workunit.client.1.smithi192.stdout:6/752: dwrite d3/d8/d56/d59/d8a/faa [0,4194304] 0 2024-04-12T05:09:10.115 INFO:tasks.workunit.client.0.smithi113.stdout:2/754: fdatasync d8/d9/de/d12/f35 0 2024-04-12T05:09:10.115 INFO:tasks.workunit.client.1.smithi192.stdout:5/774: chown d0/f52 29 1 2024-04-12T05:09:10.116 INFO:tasks.workunit.client.0.smithi113.stdout:6/761: mknod d8/de/d13/c118 0 2024-04-12T05:09:10.116 INFO:tasks.workunit.client.1.smithi192.stdout:2/722: mknod d1/d8/d3a/ce5 0 2024-04-12T05:09:10.117 INFO:tasks.workunit.client.0.smithi113.stdout:7/758: dwrite d0/dc/d6/d9b/dcd/feb [0,4194304] 0 2024-04-12T05:09:10.118 INFO:tasks.workunit.client.0.smithi113.stdout:5/895: dwrite d0/d10/d27/d1d/fbf [0,4194304] 0 2024-04-12T05:09:10.119 INFO:tasks.workunit.client.1.smithi192.stdout:1/729: rmdir d0/d2/dd/d3c 39 2024-04-12T05:09:10.121 INFO:tasks.workunit.client.1.smithi192.stdout:9/575: sync 2024-04-12T05:09:10.122 INFO:tasks.workunit.client.0.smithi113.stdout:1/868: dwrite d10/d2e/f4e [0,4194304] 0 2024-04-12T05:09:10.127 INFO:tasks.workunit.client.1.smithi192.stdout:6/753: mkdir d3/d8/dd6/df8 0 2024-04-12T05:09:10.127 INFO:tasks.workunit.client.1.smithi192.stdout:0/797: creat d0/d6/d67/f106 x:0 0 0 2024-04-12T05:09:10.129 INFO:tasks.workunit.client.0.smithi113.stdout:5/896: dread - d0/d10/d27/d1d/d3f/db0/dda/fdd zero size 2024-04-12T05:09:10.129 INFO:tasks.workunit.client.0.smithi113.stdout:6/762: write d8/de/d13/d11/d36/d3d/d4c/d31/f62 [1662946,124731] 0 2024-04-12T05:09:10.130 INFO:tasks.workunit.client.1.smithi192.stdout:3/675: getdents d5/d13/d3c/d36 0 2024-04-12T05:09:10.130 INFO:tasks.workunit.client.0.smithi113.stdout:3/751: rename d0/d13/d15/d28/d2a/d8b/d91/ffc to d0/d13/d15/d28/ffe 0 2024-04-12T05:09:10.132 INFO:tasks.workunit.client.1.smithi192.stdout:4/777: rename d1/d1d/d53/l38 to d1/d54/d82/d9c/l10c 0 2024-04-12T05:09:10.134 INFO:tasks.workunit.client.1.smithi192.stdout:4/778: chown d1/d1d/d53/d1c/dd1/def/c97 958 1 2024-04-12T05:09:10.136 INFO:tasks.workunit.client.0.smithi113.stdout:6/763: creat d8/de/dd5/f119 x:0 0 0 2024-04-12T05:09:10.138 INFO:tasks.workunit.client.0.smithi113.stdout:7/759: fsync d0/dc/da9/d5c/fdb 0 2024-04-12T05:09:10.139 INFO:tasks.workunit.client.1.smithi192.stdout:0/798: symlink d0/d7b/d9b/d4b/dbc/dce/d9f/l107 0 2024-04-12T05:09:10.139 INFO:tasks.workunit.client.0.smithi113.stdout:3/752: mknod d0/d13/d47/d94/dac/cff 0 2024-04-12T05:09:10.143 INFO:tasks.workunit.client.0.smithi113.stdout:7/760: truncate d0/dd/d11/de6/dbb/fee 399964 0 2024-04-12T05:09:10.144 INFO:tasks.workunit.client.1.smithi192.stdout:5/775: dread d0/d15/d3e/dca/dda/fe5 [0,4194304] 0 2024-04-12T05:09:10.144 INFO:tasks.workunit.client.1.smithi192.stdout:7/643: rename d3/d1d/d5a/d5f/f31 to d3/d1d/d4a/d5e/d6d/fde 0 2024-04-12T05:09:10.146 INFO:tasks.workunit.client.0.smithi113.stdout:5/897: fdatasync d0/d10/d27/d1d/fbf 0 2024-04-12T05:09:10.149 INFO:tasks.workunit.client.0.smithi113.stdout:5/898: write d0/d10/d27/d1d/d38/d3b/d6b/fd4 [1480937,47130] 0 2024-04-12T05:09:10.150 INFO:tasks.workunit.client.0.smithi113.stdout:0/878: sync 2024-04-12T05:09:10.151 INFO:tasks.workunit.client.0.smithi113.stdout:4/780: write da/d73/f1c [140856,23183] 0 2024-04-12T05:09:10.152 INFO:tasks.workunit.client.1.smithi192.stdout:0/799: mkdir d0/d7b/d9b/d4b/dbc/dce/d53/d6d/d108 0 2024-04-12T05:09:10.153 INFO:tasks.workunit.client.0.smithi113.stdout:1/869: getdents d10/d41/d89 0 2024-04-12T05:09:10.155 INFO:tasks.workunit.client.1.smithi192.stdout:3/676: getdents d5/d13/d3c/d26/dca 0 2024-04-12T05:09:10.156 INFO:tasks.workunit.client.0.smithi113.stdout:5/899: symlink d0/d10/d27/d1d/d38/d75/l135 0 2024-04-12T05:09:10.158 INFO:tasks.workunit.client.0.smithi113.stdout:7/761: truncate d0/dc/d6/d1a/f7e 201793 0 2024-04-12T05:09:10.161 INFO:tasks.workunit.client.1.smithi192.stdout:5/776: dwrite d0/d15/d3e/dca/fc4 [0,4194304] 0 2024-04-12T05:09:10.163 INFO:tasks.workunit.client.1.smithi192.stdout:1/730: rename d0/d2/dd to d0/d2/d1e/d3f/df2 0 2024-04-12T05:09:10.164 INFO:tasks.workunit.client.0.smithi113.stdout:6/764: dread d8/de/d13/d11/d36/d3d/d4c/f28 [0,4194304] 0 2024-04-12T05:09:10.167 INFO:tasks.workunit.client.0.smithi113.stdout:0/879: truncate d1/d43/d59/de9/d105/d50/db6/f4d 6198130 0 2024-04-12T05:09:10.167 INFO:tasks.workunit.client.1.smithi192.stdout:7/644: stat d3/d1d/d4a/d5e/d6d/da3/dce/ld1 0 2024-04-12T05:09:10.168 INFO:tasks.workunit.client.1.smithi192.stdout:2/723: write d1/d8/d17/d19/d60/fa6 [203049,38937] 0 2024-04-12T05:09:10.169 INFO:tasks.workunit.client.1.smithi192.stdout:9/576: dread d11/d25/d36/d6e/d33/da2/f39 [4194304,4194304] 0 2024-04-12T05:09:10.170 INFO:tasks.workunit.client.0.smithi113.stdout:7/762: dwrite d0/dd/d11/f77 [4194304,4194304] 0 2024-04-12T05:09:10.172 INFO:tasks.workunit.client.0.smithi113.stdout:5/900: dread d0/d10/d27/d1d/d3f/db0/feb [0,4194304] 0 2024-04-12T05:09:10.173 INFO:tasks.workunit.client.0.smithi113.stdout:0/880: write d1/d6/fb0 [4232291,127482] 0 2024-04-12T05:09:10.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:08.673655+0000 mgr.smithi192.ocppyo (mgr.24357) 31 : cluster [DBG] pgmap v24: 65 pgs: 65 active+clean; 2.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 109 MiB/s rd, 361 MiB/s wr, 658 op/s 2024-04-12T05:09:10.175 INFO:tasks.workunit.client.0.smithi113.stdout:6/765: dread - d8/de/d13/d11/d36/d53/db7/fc9 zero size 2024-04-12T05:09:10.175 INFO:tasks.workunit.client.0.smithi113.stdout:7/763: mkdir d0/dc/da9/d5c/d9a/db6/d101 0 2024-04-12T05:09:10.176 INFO:tasks.workunit.client.0.smithi113.stdout:3/753: dread d0/d13/d15/d28/d2a/f98 [0,4194304] 0 2024-04-12T05:09:10.177 INFO:tasks.workunit.client.0.smithi113.stdout:0/881: chown d1/d43/d59/lb3 872 1 2024-04-12T05:09:10.178 INFO:tasks.workunit.client.0.smithi113.stdout:6/766: dread - d8/de/d13/fe0 zero size 2024-04-12T05:09:10.179 INFO:tasks.workunit.client.0.smithi113.stdout:2/755: dread d8/d9/d27/f28 [0,4194304] 0 2024-04-12T05:09:10.182 INFO:tasks.workunit.client.0.smithi113.stdout:7/764: fdatasync d0/dc/f41 0 2024-04-12T05:09:10.184 INFO:tasks.workunit.client.1.smithi192.stdout:0/800: rename d0/d7b/d9b/d4b/dbc/dce to d0/d7b/d9b/d35/d109 0 2024-04-12T05:09:10.185 INFO:tasks.workunit.client.1.smithi192.stdout:4/779: dwrite d1/d1d/d53/d1c/d84/dbb/ff6 [4194304,4194304] 0 2024-04-12T05:09:10.185 INFO:tasks.workunit.client.0.smithi113.stdout:2/756: mknod d8/d9/d1a/cea 0 2024-04-12T05:09:10.187 INFO:tasks.workunit.client.0.smithi113.stdout:5/901: getdents d0/d10/d27/d1d/d3f/d73 0 2024-04-12T05:09:10.192 INFO:tasks.workunit.client.1.smithi192.stdout:9/577: rename d11/d25/d36/d6e/d33/da2/f59 to d11/d25/d36/d6e/d33/da2/fc9 0 2024-04-12T05:09:10.192 INFO:tasks.workunit.client.0.smithi113.stdout:2/757: write d8/d9/de/d12/f5d [2455265,95556] 0 2024-04-12T05:09:10.192 INFO:tasks.workunit.client.0.smithi113.stdout:3/754: dwrite d0/d13/f70 [0,4194304] 0 2024-04-12T05:09:10.194 INFO:tasks.workunit.client.1.smithi192.stdout:3/677: truncate d5/d13/d3a/d70/d8c/d8d/f93 256460 0 2024-04-12T05:09:10.195 INFO:tasks.workunit.client.1.smithi192.stdout:5/777: write d0/d15/d3e/dca/dda/ffe [2142872,69877] 0 2024-04-12T05:09:10.196 INFO:tasks.workunit.client.1.smithi192.stdout:3/678: truncate d5/d13/d3a/d4a/d61/d79/fe2 1287395 0 2024-04-12T05:09:10.197 INFO:tasks.workunit.client.0.smithi113.stdout:5/902: symlink d0/d10/d27/d1d/d3f/d73/db9/l136 0 2024-04-12T05:09:10.197 INFO:tasks.workunit.client.0.smithi113.stdout:5/903: stat d0/d10/d27/d1d/d41/cdb 0 2024-04-12T05:09:10.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:08.673655+0000 mgr.smithi192.ocppyo (mgr.24357) 31 : cluster [DBG] pgmap v24: 65 pgs: 65 active+clean; 2.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 109 MiB/s rd, 361 MiB/s wr, 658 op/s 2024-04-12T05:09:10.200 INFO:tasks.workunit.client.0.smithi113.stdout:7/765: creat d0/dc/d50/f102 x:0 0 0 2024-04-12T05:09:10.200 INFO:tasks.workunit.client.1.smithi192.stdout:1/731: mkdir d0/d2/d1e/d3f/df2/d3c/d5b/d74/d7d/df3 0 2024-04-12T05:09:10.202 INFO:tasks.workunit.client.0.smithi113.stdout:0/882: write d1/d6/d34/dc3/f95 [1694474,69912] 0 2024-04-12T05:09:10.202 INFO:tasks.workunit.client.0.smithi113.stdout:7/766: fsync d0/dc/da9/d5c/d9a/fab 0 2024-04-12T05:09:10.202 INFO:tasks.workunit.client.0.smithi113.stdout:5/904: rmdir d0/d10/d6e/d93 39 2024-04-12T05:09:10.203 INFO:tasks.workunit.client.1.smithi192.stdout:0/801: rmdir d0/d6/d15/d1b 39 2024-04-12T05:09:10.206 INFO:tasks.workunit.client.0.smithi113.stdout:6/767: dread d8/de/d17/f78 [0,4194304] 0 2024-04-12T05:09:10.207 INFO:tasks.workunit.client.1.smithi192.stdout:4/780: mkdir d1/d1d/d53/d1c/d95/dc6/dd5/de5/d10d 0 2024-04-12T05:09:10.208 INFO:tasks.workunit.client.0.smithi113.stdout:7/767: creat d0/dc/da9/d5c/f103 x:0 0 0 2024-04-12T05:09:10.208 INFO:tasks.workunit.client.0.smithi113.stdout:2/758: creat d8/d9/de/d15/feb x:0 0 0 2024-04-12T05:09:10.209 INFO:tasks.workunit.client.1.smithi192.stdout:9/578: chown d11/d25/d36/d6e/f56 1 1 2024-04-12T05:09:10.210 INFO:tasks.workunit.client.0.smithi113.stdout:6/768: mkdir d8/de/d13/d11/d36/d89/d11a 0 2024-04-12T05:09:10.213 INFO:tasks.workunit.client.0.smithi113.stdout:2/759: creat d8/d9/d1a/fec x:0 0 0 2024-04-12T05:09:10.214 INFO:tasks.workunit.client.1.smithi192.stdout:3/679: rmdir d5/d13/d3a 39 2024-04-12T05:09:10.215 INFO:tasks.workunit.client.0.smithi113.stdout:6/769: write d8/de/d13/df0/d66/f7e [3611344,130920] 0 2024-04-12T05:09:10.215 INFO:tasks.workunit.client.1.smithi192.stdout:1/732: rmdir d0/d29 39 2024-04-12T05:09:10.216 INFO:tasks.workunit.client.0.smithi113.stdout:3/755: truncate d0/d13/f70 1013549 0 2024-04-12T05:09:10.216 INFO:tasks.workunit.client.0.smithi113.stdout:2/760: stat d8/d9/de/d15/d24/d58/f96 0 2024-04-12T05:09:10.216 INFO:tasks.workunit.client.1.smithi192.stdout:1/733: chown d0/d1b/lcb 2 1 2024-04-12T05:09:10.217 INFO:tasks.workunit.client.0.smithi113.stdout:7/768: getdents d0/dc/da9/d5c/d85 0 2024-04-12T05:09:10.219 INFO:tasks.workunit.client.0.smithi113.stdout:5/905: dwrite d0/d10/d27/d1d/d38/d44/d66/f96 [0,4194304] 0 2024-04-12T05:09:10.220 INFO:tasks.workunit.client.0.smithi113.stdout:6/770: write d8/de/d13/d11/d36/d3d/d4c/d26/d108/f10e [375572,10823] 0 2024-04-12T05:09:10.221 INFO:tasks.workunit.client.1.smithi192.stdout:0/802: rename d0/d6 to d0/d7b/d9b/d35/d109/d9f/d10a 0 2024-04-12T05:09:10.222 INFO:tasks.workunit.client.0.smithi113.stdout:2/761: mkdir d8/d9/d27/d88/d91/ded 0 2024-04-12T05:09:10.223 INFO:tasks.workunit.client.0.smithi113.stdout:5/906: unlink d0/d10/d27/d1d/d3f/d73/db9/c118 0 2024-04-12T05:09:10.224 INFO:tasks.workunit.client.0.smithi113.stdout:0/883: rename d1/d6/d16/d8f/f103 to d1/d43/d59/de9/d105/d50/f10c 0 2024-04-12T05:09:10.225 INFO:tasks.workunit.client.0.smithi113.stdout:6/771: rmdir d8/de/d13/d11/d36/d3d/d4c/d26/d7d 39 2024-04-12T05:09:10.225 INFO:tasks.workunit.client.0.smithi113.stdout:5/907: dread - d0/d10/d27/d1d/d38/d3b/d6b/f105 zero size 2024-04-12T05:09:10.226 INFO:tasks.workunit.client.1.smithi192.stdout:5/778: unlink d0/d21/d5e/c75 0 2024-04-12T05:09:10.227 INFO:tasks.workunit.client.0.smithi113.stdout:6/772: write d8/de/d13/d11/f110 [42513,69139] 0 2024-04-12T05:09:10.228 INFO:tasks.workunit.client.0.smithi113.stdout:2/762: mkdir d8/d9/de/d43/dee 0 2024-04-12T05:09:10.229 INFO:tasks.workunit.client.1.smithi192.stdout:1/734: creat d0/d1b/d44/d69/ddd/ff4 x:0 0 0 2024-04-12T05:09:10.229 INFO:tasks.workunit.client.0.smithi113.stdout:0/884: getdents d1/d43/df1 0 2024-04-12T05:09:10.230 INFO:tasks.workunit.client.0.smithi113.stdout:2/763: stat d8/d9/de/f31 0 2024-04-12T05:09:10.233 INFO:tasks.workunit.client.1.smithi192.stdout:2/724: rename d1/d8/d17/db3/ld5 to d1/d8/d3a/de/d7c/ddd/le6 0 2024-04-12T05:09:10.233 INFO:tasks.workunit.client.0.smithi113.stdout:6/773: unlink d8/de/dd5/l113 0 2024-04-12T05:09:10.236 INFO:tasks.workunit.client.1.smithi192.stdout:2/725: chown d1/d8/d17/d19/d40/f5b 14 1 2024-04-12T05:09:10.236 INFO:tasks.workunit.client.1.smithi192.stdout:3/680: mkdir d5/d13/d3a/d8b/df0 0 2024-04-12T05:09:10.237 INFO:tasks.workunit.client.0.smithi113.stdout:6/774: mknod d8/de/dd5/de7/d76/c11b 0 2024-04-12T05:09:10.238 INFO:tasks.workunit.client.0.smithi113.stdout:2/764: mkdir d8/d9/d27/d88/d91/ded/def 0 2024-04-12T05:09:10.239 INFO:tasks.workunit.client.1.smithi192.stdout:1/735: dwrite d0/d1b/d44/fb3 [0,4194304] 0 2024-04-12T05:09:10.239 INFO:tasks.workunit.client.0.smithi113.stdout:7/769: truncate d0/dc/d6/d78/f6a 2557680 0 2024-04-12T05:09:10.240 INFO:tasks.workunit.client.1.smithi192.stdout:5/779: dread d0/d15/d24/ddf/fe8 [0,4194304] 0 2024-04-12T05:09:10.241 INFO:tasks.workunit.client.0.smithi113.stdout:6/775: creat d8/de/dd5/f11c x:0 0 0 2024-04-12T05:09:10.241 INFO:tasks.workunit.client.1.smithi192.stdout:4/781: creat d1/d54/d82/d71/d9e/f10e x:0 0 0 2024-04-12T05:09:10.242 INFO:tasks.workunit.client.1.smithi192.stdout:9/579: dread f3 [0,4194304] 0 2024-04-12T05:09:10.242 INFO:tasks.workunit.client.0.smithi113.stdout:0/885: creat d1/d80/f10d x:0 0 0 2024-04-12T05:09:10.243 INFO:tasks.workunit.client.1.smithi192.stdout:1/736: read d0/d2/d1e/d3f/d56/f5a [275260,93943] 0 2024-04-12T05:09:10.244 INFO:tasks.workunit.client.1.smithi192.stdout:5/780: truncate d0/d15/d32/d69/fd9 250791 0 2024-04-12T05:09:10.244 INFO:tasks.workunit.client.1.smithi192.stdout:0/803: fsync d0/d7b/d7c/fdb 0 2024-04-12T05:09:10.247 INFO:tasks.workunit.client.1.smithi192.stdout:1/737: write d0/d2/d1e/d3f/df2/fa9 [2680679,57415] 0 2024-04-12T05:09:10.248 INFO:tasks.workunit.client.1.smithi192.stdout:9/580: creat d11/d15/fca x:0 0 0 2024-04-12T05:09:10.248 INFO:tasks.workunit.client.1.smithi192.stdout:0/804: dread - d0/d7b/d9b/d35/d109/d53/df3/d2e/d59/fea zero size 2024-04-12T05:09:10.249 INFO:tasks.workunit.client.0.smithi113.stdout:2/765: rename d8/d9/de/d15/d24/d58/f84 to d8/d9/de/d12/d2b/daa/db3/ff0 0 2024-04-12T05:09:10.250 INFO:tasks.workunit.client.0.smithi113.stdout:2/766: chown d8/l25 9 1 2024-04-12T05:09:10.250 INFO:tasks.workunit.client.0.smithi113.stdout:5/908: dread d0/d10/d27/d1d/d38/fba [0,4194304] 0 2024-04-12T05:09:10.251 INFO:tasks.workunit.client.0.smithi113.stdout:0/886: symlink d1/d43/d59/de9/d105/d50/l10e 0 2024-04-12T05:09:10.253 INFO:tasks.workunit.client.0.smithi113.stdout:7/770: dread d0/dc/d6/d9b/d80/f96 [0,4194304] 0 2024-04-12T05:09:10.253 INFO:tasks.workunit.client.0.smithi113.stdout:2/767: creat d8/d9/d1a/ff1 x:0 0 0 2024-04-12T05:09:10.254 INFO:tasks.workunit.client.1.smithi192.stdout:1/738: chown d0/d29/d8d/d96/fdb 35 1 2024-04-12T05:09:10.255 INFO:tasks.workunit.client.1.smithi192.stdout:9/581: dwrite d11/d25/d36/d6e/d33/fab [0,4194304] 0 2024-04-12T05:09:10.258 INFO:tasks.workunit.client.1.smithi192.stdout:4/782: creat d1/d1d/f10f x:0 0 0 2024-04-12T05:09:10.262 INFO:tasks.workunit.client.0.smithi113.stdout:2/768: symlink d8/d9/de/d43/db2/lf2 0 2024-04-12T05:09:10.263 INFO:tasks.workunit.client.0.smithi113.stdout:5/909: link d0/d10/d27/d1d/d3f/f47 d0/d10/d27/d5f/d128/f137 0 2024-04-12T05:09:10.264 INFO:tasks.workunit.client.1.smithi192.stdout:0/805: rename d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d3a/d4f/c8e to d0/d7b/d9b/d35/d109/d9f/d10a/db2/c10b 0 2024-04-12T05:09:10.266 INFO:tasks.workunit.client.0.smithi113.stdout:0/887: dwrite d1/d43/d59/de9/d105/d50/d52/d7e/ff2 [0,4194304] 0 2024-04-12T05:09:10.267 INFO:tasks.workunit.client.0.smithi113.stdout:7/771: creat d0/dc/d6/d24/f104 x:0 0 0 2024-04-12T05:09:10.267 INFO:tasks.workunit.client.1.smithi192.stdout:2/726: dwrite d1/d8/d3a/de/f30 [0,4194304] 0 2024-04-12T05:09:10.269 INFO:tasks.workunit.client.0.smithi113.stdout:5/910: fdatasync d0/d10/d27/d1d/d10c/f106 0 2024-04-12T05:09:10.271 INFO:tasks.workunit.client.1.smithi192.stdout:5/781: getdents d0/d15/d3e/dca/dda 0 2024-04-12T05:09:10.271 INFO:tasks.workunit.client.1.smithi192.stdout:4/783: fsync d1/d1d/f10f 0 2024-04-12T05:09:10.271 INFO:tasks.workunit.client.0.smithi113.stdout:2/769: link d8/d9/de/f31 d8/d9/d27/d88/de0/ff3 0 2024-04-12T05:09:10.272 INFO:tasks.workunit.client.1.smithi192.stdout:8/721: sync 2024-04-12T05:09:10.272 INFO:tasks.workunit.client.1.smithi192.stdout:1/739: link d0/d2/d1e/d3f/df2/d3c/d5b/d74/d7d/fe0 d0/d2/d1e/d3f/df2/d3c/d5b/d74/d7d/ff5 0 2024-04-12T05:09:10.275 INFO:tasks.workunit.client.1.smithi192.stdout:8/722: write d6/d1f/f2a [57162,70743] 0 2024-04-12T05:09:10.275 INFO:tasks.workunit.client.0.smithi113.stdout:6/776: dwrite d8/d1d/fac [0,4194304] 0 2024-04-12T05:09:10.275 INFO:tasks.workunit.client.1.smithi192.stdout:2/727: mkdir d1/d2d/d73/de7 0 2024-04-12T05:09:10.276 INFO:tasks.workunit.client.0.smithi113.stdout:5/911: mknod d0/d10/d27/d1d/d3f/db0/dda/c138 0 2024-04-12T05:09:10.277 INFO:tasks.workunit.client.1.smithi192.stdout:1/740: chown d0/d2/d1e/d3f/df2/d3c/d5b/db6 11134175 1 2024-04-12T05:09:10.278 INFO:tasks.workunit.client.0.smithi113.stdout:2/770: symlink d8/d16/dbd/lf4 0 2024-04-12T05:09:10.278 INFO:tasks.workunit.client.0.smithi113.stdout:7/772: rmdir d0/dc/d6/d1a/da0 39 2024-04-12T05:09:10.279 INFO:tasks.workunit.client.0.smithi113.stdout:5/912: chown d0/d10/d27/d1d/d38/d44/d66/cd0 51232659 1 2024-04-12T05:09:10.279 INFO:tasks.workunit.client.1.smithi192.stdout:5/782: fsync d0/d15/d32/d69/f9e 0 2024-04-12T05:09:10.280 INFO:tasks.workunit.client.1.smithi192.stdout:9/582: dwrite f3 [0,4194304] 0 2024-04-12T05:09:10.287 INFO:tasks.workunit.client.0.smithi113.stdout:7/773: fdatasync d0/dc/d6/d78/f31 0 2024-04-12T05:09:10.289 INFO:tasks.workunit.client.0.smithi113.stdout:6/777: symlink d8/de/d13/d11/d36/d53/db8/dce/l11d 0 2024-04-12T05:09:10.290 INFO:tasks.workunit.client.1.smithi192.stdout:2/728: readlink d1/d8/d17/d19/d60/d9c/lad 0 2024-04-12T05:09:10.293 INFO:tasks.workunit.client.1.smithi192.stdout:0/806: rename d0/d7b/d9b/d35/d109/d95/l9c to d0/l10c 0 2024-04-12T05:09:10.296 INFO:tasks.workunit.client.0.smithi113.stdout:5/913: getdents d0/d10/d27/d1d/d38/d44 0 2024-04-12T05:09:10.300 INFO:tasks.workunit.client.1.smithi192.stdout:1/741: creat d0/d29/d8d/ff6 x:0 0 0 2024-04-12T05:09:10.301 INFO:tasks.workunit.client.1.smithi192.stdout:0/807: chown d0/d7b/d9b/d35/d109/d95/db4/db8/fcb 7732663 1 2024-04-12T05:09:10.303 INFO:tasks.workunit.client.0.smithi113.stdout:7/774: link d0/dc/da9/d5c/d9a/fab d0/dc/da9/d5c/f105 0 2024-04-12T05:09:10.304 INFO:tasks.workunit.client.1.smithi192.stdout:9/583: creat d11/d15/d21/d2c/fcb x:0 0 0 2024-04-12T05:09:10.304 INFO:tasks.workunit.client.0.smithi113.stdout:0/888: dwrite d1/d6/d16/f67 [0,4194304] 0 2024-04-12T05:09:10.304 INFO:tasks.workunit.client.0.smithi113.stdout:6/778: getdents d8/de/d13/d11/d36/d53/db8 0 2024-04-12T05:09:10.307 INFO:tasks.workunit.client.1.smithi192.stdout:0/808: creat d0/d7b/d9b/d35/d109/d95/db4/db8/f10d x:0 0 0 2024-04-12T05:09:10.308 INFO:tasks.workunit.client.0.smithi113.stdout:7/775: readlink d0/d6c/l6f 0 2024-04-12T05:09:10.310 INFO:tasks.workunit.client.1.smithi192.stdout:2/729: write d1/d8/f46 [4669167,30393] 0 2024-04-12T05:09:10.313 INFO:tasks.workunit.client.0.smithi113.stdout:2/771: truncate d8/d9/d27/d88/f95 3128620 0 2024-04-12T05:09:10.314 INFO:tasks.workunit.client.0.smithi113.stdout:0/889: dwrite d1/d43/d59/de9/d105/d50/fff [0,4194304] 0 2024-04-12T05:09:10.317 INFO:tasks.workunit.client.0.smithi113.stdout:5/914: dread d0/d10/d27/d1d/fc7 [0,4194304] 0 2024-04-12T05:09:10.318 INFO:tasks.workunit.client.1.smithi192.stdout:2/730: readlink d1/d8/d3a/l14 0 2024-04-12T05:09:10.319 INFO:tasks.workunit.client.0.smithi113.stdout:6/779: rename d8/de/d13/d11/d36/d3d/d4c/dd8/ff7 to d8/f11e 0 2024-04-12T05:09:10.320 INFO:tasks.workunit.client.1.smithi192.stdout:0/809: mkdir d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d3a/d4f/d10e 0 2024-04-12T05:09:10.320 INFO:tasks.workunit.client.1.smithi192.stdout:5/783: dread d0/d15/d32/d45/f60 [0,4194304] 0 2024-04-12T05:09:10.321 INFO:tasks.workunit.client.1.smithi192.stdout:1/742: dwrite d0/d29/d8d/d96/fdb [0,4194304] 0 2024-04-12T05:09:10.321 INFO:tasks.workunit.client.1.smithi192.stdout:4/784: dwrite d1/d54/d82/d71/d9e/d10b/f7e [0,4194304] 0 2024-04-12T05:09:10.322 INFO:tasks.workunit.client.0.smithi113.stdout:5/915: truncate d0/d10/d27/d1d/d41/ff3 694144 0 2024-04-12T05:09:10.324 INFO:tasks.workunit.client.0.smithi113.stdout:5/916: dread - d0/d10/d27/d1d/d38/d3b/d6b/f105 zero size 2024-04-12T05:09:10.330 INFO:tasks.workunit.client.0.smithi113.stdout:5/917: unlink d0/d10/d27/d1d/d3f/d73/l11d 0 2024-04-12T05:09:10.332 INFO:tasks.workunit.client.0.smithi113.stdout:5/918: fdatasync d0/d10/d6e/f3c 0 2024-04-12T05:09:10.333 INFO:tasks.workunit.client.1.smithi192.stdout:9/584: rename d11/d15/d21/d2c/d1f/d6a/d7e/cbc to d11/d25/d36/d6e/ccc 0 2024-04-12T05:09:10.333 INFO:tasks.workunit.client.1.smithi192.stdout:1/743: readlink d0/d2/d1e/d3f/df2/d21/d62/d84/l86 0 2024-04-12T05:09:10.334 INFO:tasks.workunit.client.0.smithi113.stdout:7/776: write d0/dc/d6/d1a/fb8 [1158252,111687] 0 2024-04-12T05:09:10.336 INFO:tasks.workunit.client.1.smithi192.stdout:4/785: rmdir d1/d49/db5 39 2024-04-12T05:09:10.336 INFO:tasks.workunit.client.1.smithi192.stdout:1/744: chown d0/d1b 8 1 2024-04-12T05:09:10.338 INFO:tasks.workunit.client.1.smithi192.stdout:5/784: mknod d0/d15/d32/d69/dc8/c10e 0 2024-04-12T05:09:10.338 INFO:tasks.workunit.client.1.smithi192.stdout:4/786: stat d1/d1d/d53/l2a 0 2024-04-12T05:09:10.338 INFO:tasks.workunit.client.0.smithi113.stdout:0/890: dwrite d1/fc9 [0,4194304] 0 2024-04-12T05:09:10.339 INFO:tasks.workunit.client.1.smithi192.stdout:9/585: symlink d11/d15/d21/d2c/d1f/d50/d73/lcd 0 2024-04-12T05:09:10.341 INFO:tasks.workunit.client.1.smithi192.stdout:9/586: truncate d11/d15/d21/d2c/d1f/f67 310022 0 2024-04-12T05:09:10.343 INFO:tasks.workunit.client.1.smithi192.stdout:0/810: rename d0/d7b/d9b/d35/d109/d9f/fa5 to d0/d7b/d9b/d35/d109/d9f/d10a/d67/db1/f10f 0 2024-04-12T05:09:10.345 INFO:tasks.workunit.client.0.smithi113.stdout:5/919: dwrite d0/d10/d27/d1d/d8c/db8/f12a [0,4194304] 0 2024-04-12T05:09:10.349 INFO:tasks.workunit.client.0.smithi113.stdout:0/891: creat d1/d43/d59/de9/d105/d50/d52/d7e/f10f x:0 0 0 2024-04-12T05:09:10.350 INFO:tasks.workunit.client.1.smithi192.stdout:4/787: fsync d1/d54/d55/fa6 0 2024-04-12T05:09:10.352 INFO:tasks.workunit.client.1.smithi192.stdout:1/745: fsync d0/d29/d8d/d96/fdb 0 2024-04-12T05:09:10.352 INFO:tasks.workunit.client.1.smithi192.stdout:0/811: unlink d0/d7b/d9b/d4b/l54 0 2024-04-12T05:09:10.353 INFO:tasks.workunit.client.0.smithi113.stdout:5/920: read d0/d10/d27/d1d/d3f/d7e/f8b [1411281,107336] 0 2024-04-12T05:09:10.358 INFO:tasks.workunit.client.1.smithi192.stdout:2/731: write d1/d2d/f5d [1319880,34821] 0 2024-04-12T05:09:10.358 INFO:tasks.workunit.client.1.smithi192.stdout:0/812: fsync d0/d7b/d9b/d35/d109/d53/df3/d2e/f9e 0 2024-04-12T05:09:10.363 INFO:tasks.workunit.client.1.smithi192.stdout:4/788: symlink d1/d54/d82/l110 0 2024-04-12T05:09:10.363 INFO:tasks.workunit.client.0.smithi113.stdout:6/780: dwrite d8/de/d13/d11/d36/d3d/d4c/d26/d64/fb0 [0,4194304] 0 2024-04-12T05:09:10.367 INFO:tasks.workunit.client.1.smithi192.stdout:5/785: dwrite d0/f14 [0,4194304] 0 2024-04-12T05:09:10.368 INFO:tasks.workunit.client.0.smithi113.stdout:2/772: dread d8/d9/de/d12/f9c [0,4194304] 0 2024-04-12T05:09:10.369 INFO:tasks.workunit.client.1.smithi192.stdout:0/813: unlink d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d3a/d4f/l94 0 2024-04-12T05:09:10.374 INFO:tasks.workunit.client.0.smithi113.stdout:5/921: dwrite d0/ff5 [0,4194304] 0 2024-04-12T05:09:10.375 INFO:tasks.workunit.client.0.smithi113.stdout:6/781: mknod d8/de/d13/d11/d36/c11f 0 2024-04-12T05:09:10.375 INFO:tasks.workunit.client.0.smithi113.stdout:7/777: dwrite d0/dc/da9/d5c/d85/f8d [0,4194304] 0 2024-04-12T05:09:10.376 INFO:tasks.workunit.client.1.smithi192.stdout:4/789: mkdir d1/d1d/d53/d1c/d111 0 2024-04-12T05:09:10.376 INFO:tasks.workunit.client.1.smithi192.stdout:5/786: mknod d0/d21/d5e/c10f 0 2024-04-12T05:09:10.380 INFO:tasks.workunit.client.0.smithi113.stdout:0/892: dread d1/d6/d16/fb8 [4194304,4194304] 0 2024-04-12T05:09:10.381 INFO:tasks.workunit.client.0.smithi113.stdout:9/786: sync 2024-04-12T05:09:10.382 INFO:tasks.workunit.client.0.smithi113.stdout:0/893: dread - d1/d80/f10d zero size 2024-04-12T05:09:10.382 INFO:tasks.workunit.client.1.smithi192.stdout:5/787: truncate d0/d15/d3e/fae 445180 0 2024-04-12T05:09:10.383 INFO:tasks.workunit.client.0.smithi113.stdout:5/922: symlink d0/d10/d27/d1d/d3f/d7e/l139 0 2024-04-12T05:09:10.383 INFO:tasks.workunit.client.0.smithi113.stdout:2/773: read d8/d9/d1a/f50 [221099,87703] 0 2024-04-12T05:09:10.384 INFO:tasks.workunit.client.1.smithi192.stdout:5/788: readlink d0/lf 0 2024-04-12T05:09:10.385 INFO:tasks.workunit.client.0.smithi113.stdout:7/778: rename d0/dc/da9/d5c/d9a/d9e/fea to d0/dc/de7/f106 0 2024-04-12T05:09:10.386 INFO:tasks.workunit.client.0.smithi113.stdout:2/774: dread - d8/d9/d27/dda/f99 zero size 2024-04-12T05:09:10.386 INFO:tasks.workunit.client.0.smithi113.stdout:0/894: mkdir d1/d6/d16/d110 0 2024-04-12T05:09:10.387 INFO:tasks.workunit.client.1.smithi192.stdout:0/814: getdents d0/d7b/d9b/d35/d109/d9f/d10a/db2 0 2024-04-12T05:09:10.389 INFO:tasks.workunit.client.1.smithi192.stdout:9/587: dwrite d11/d15/f6b [0,4194304] 0 2024-04-12T05:09:10.391 INFO:tasks.workunit.client.0.smithi113.stdout:6/782: dwrite d8/de/d13/df0/d66/dc4/fe6 [0,4194304] 0 2024-04-12T05:09:10.392 INFO:tasks.workunit.client.1.smithi192.stdout:1/746: dwrite d0/d2/d1e/d3f/df2/d3c/d5b/d74/d90/f94 [0,4194304] 0 2024-04-12T05:09:10.395 INFO:tasks.workunit.client.0.smithi113.stdout:0/895: creat d1/d43/d59/f111 x:0 0 0 2024-04-12T05:09:10.398 INFO:tasks.workunit.client.0.smithi113.stdout:5/923: rename d0/d10/d27/d1d/d38/d44/d4e/c5d to d0/d10/d6e/d93/dc9/c13a 0 2024-04-12T05:09:10.399 INFO:tasks.workunit.client.0.smithi113.stdout:7/779: creat d0/dd/d95/dd4/f107 x:0 0 0 2024-04-12T05:09:10.402 INFO:tasks.workunit.client.1.smithi192.stdout:1/747: mkdir d0/d2/d1e/d3f/df2/d3c/d5b/d7c/df7 0 2024-04-12T05:09:10.403 INFO:tasks.workunit.client.1.smithi192.stdout:0/815: write d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d46/f4d [1262094,113850] 0 2024-04-12T05:09:10.403 INFO:tasks.workunit.client.0.smithi113.stdout:7/780: mkdir d0/dc/d6/d78/daa/d108 0 2024-04-12T05:09:10.404 INFO:tasks.workunit.client.0.smithi113.stdout:2/775: link d8/d9/de/d12/d44/lc7 d8/d9/d27/dd9/lf5 0 2024-04-12T05:09:10.405 INFO:tasks.workunit.client.0.smithi113.stdout:2/776: readlink d8/d9/de/d43/db2/lbe 0 2024-04-12T05:09:10.407 INFO:tasks.workunit.client.1.smithi192.stdout:1/748: creat d0/d1b/d44/ff8 x:0 0 0 2024-04-12T05:09:10.408 INFO:tasks.workunit.client.1.smithi192.stdout:1/749: chown d0/d1b/f53 202137943 1 2024-04-12T05:09:10.409 INFO:tasks.workunit.client.1.smithi192.stdout:1/750: write d0/d2/d1e/d3f/df2/fa9 [3272868,42248] 0 2024-04-12T05:09:10.411 INFO:tasks.workunit.client.1.smithi192.stdout:2/732: dread d1/d2d/f5d [0,4194304] 0 2024-04-12T05:09:10.411 INFO:tasks.workunit.client.0.smithi113.stdout:2/777: dread d8/d9/de/d43/f5c [0,4194304] 0 2024-04-12T05:09:10.411 INFO:tasks.workunit.client.1.smithi192.stdout:5/789: getdents d0/d15/d3e/dca/db9/ded 0 2024-04-12T05:09:10.416 INFO:tasks.workunit.client.1.smithi192.stdout:5/790: creat d0/d15/d3e/dca/db9/f110 x:0 0 0 2024-04-12T05:09:10.421 INFO:tasks.workunit.client.0.smithi113.stdout:2/778: dread - d8/d9/de/d12/d3c/fbc zero size 2024-04-12T05:09:10.421 INFO:tasks.workunit.client.0.smithi113.stdout:5/924: dread d0/f55 [0,4194304] 0 2024-04-12T05:09:10.423 INFO:tasks.workunit.client.0.smithi113.stdout:9/787: dread d1/d4/d38/d65/fcf [0,4194304] 0 2024-04-12T05:09:10.425 INFO:tasks.workunit.client.0.smithi113.stdout:7/781: dwrite d0/dc/d6/f16 [0,4194304] 0 2024-04-12T05:09:10.425 INFO:tasks.workunit.client.1.smithi192.stdout:5/791: rename d0/d15/d3e/dca/dda/db5/dd7/de1/ff2 to d0/d15/d24/d47/f111 0 2024-04-12T05:09:10.427 INFO:tasks.workunit.client.0.smithi113.stdout:9/788: fdatasync d1/df/d29/fd3 0 2024-04-12T05:09:10.427 INFO:tasks.workunit.client.1.smithi192.stdout:9/588: dread fe [0,4194304] 0 2024-04-12T05:09:10.428 INFO:tasks.workunit.client.1.smithi192.stdout:2/733: getdents d1/d2d/d73/da8/db2 0 2024-04-12T05:09:10.429 INFO:tasks.workunit.client.0.smithi113.stdout:7/782: readlink d0/dc/d6/d9b/dcd/le3 0 2024-04-12T05:09:10.431 INFO:tasks.workunit.client.1.smithi192.stdout:4/790: dread d1/d49/f69 [0,4194304] 0 2024-04-12T05:09:10.436 INFO:tasks.workunit.client.0.smithi113.stdout:2/779: dwrite d8/d9/de/d15/d24/d58/fe1 [0,4194304] 0 2024-04-12T05:09:10.437 INFO:tasks.workunit.client.1.smithi192.stdout:9/589: fsync d11/d25/d36/d6e/d8f/fbd 0 2024-04-12T05:09:10.438 INFO:tasks.workunit.client.0.smithi113.stdout:2/780: dread - d8/d9/d27/d88/fa3 zero size 2024-04-12T05:09:10.440 INFO:tasks.workunit.client.0.smithi113.stdout:0/896: truncate d1/f2 1998425 0 2024-04-12T05:09:10.442 INFO:tasks.workunit.client.0.smithi113.stdout:7/783: link d0/dd/d11/de6/c89 d0/dc/d6/d9b/dcd/c109 0 2024-04-12T05:09:10.443 INFO:tasks.workunit.client.0.smithi113.stdout:2/781: chown d8/d9/d1a/c9a 27 1 2024-04-12T05:09:10.443 INFO:tasks.workunit.client.1.smithi192.stdout:0/816: write d0/d7b/d9b/d35/d109/d53/df3/d2e/d66/f86 [28700,122396] 0 2024-04-12T05:09:10.443 INFO:tasks.workunit.client.0.smithi113.stdout:9/789: dwrite d1/d27/d7a/dd2/fec [0,4194304] 0 2024-04-12T05:09:10.444 INFO:tasks.workunit.client.0.smithi113.stdout:0/897: fdatasync d1/d6/d34/fc6 0 2024-04-12T05:09:10.446 INFO:tasks.workunit.client.1.smithi192.stdout:0/817: chown d0/d7b/d9b/d35/d109/d9f/d10a/d67/db1/dd6 0 1 2024-04-12T05:09:10.446 INFO:tasks.workunit.client.1.smithi192.stdout:5/792: truncate d0/d15/d32/d69/d7d/fcc 1406670 0 2024-04-12T05:09:10.447 INFO:tasks.workunit.client.1.smithi192.stdout:1/751: write d0/d2/d1e/d3f/df2/d21/f43 [353886,21149] 0 2024-04-12T05:09:10.447 INFO:tasks.workunit.client.0.smithi113.stdout:9/790: fdatasync d1/d4/d38/f8c 0 2024-04-12T05:09:10.447 INFO:tasks.workunit.client.0.smithi113.stdout:7/784: chown d0/dc/de7/f106 107451049 1 2024-04-12T05:09:10.451 INFO:tasks.workunit.client.0.smithi113.stdout:9/791: fdatasync d1/d4/d5/d6/f76 0 2024-04-12T05:09:10.452 INFO:tasks.workunit.client.1.smithi192.stdout:9/590: mkdir d11/d15/d21/d2c/d1f/d6a/d7e/d89/dce 0 2024-04-12T05:09:10.453 INFO:tasks.workunit.client.0.smithi113.stdout:9/792: fdatasync d1/d27/d7a/de4/db7/ffb 0 2024-04-12T05:09:10.453 INFO:tasks.workunit.client.0.smithi113.stdout:7/785: mknod d0/dd/d95/dd4/c10a 0 2024-04-12T05:09:10.454 INFO:tasks.workunit.client.0.smithi113.stdout:7/786: write d0/dc/da9/d5c/f103 [468829,23870] 0 2024-04-12T05:09:10.455 INFO:tasks.workunit.client.0.smithi113.stdout:2/782: mkdir d8/d9/de/d12/d2b/dd2/df6 0 2024-04-12T05:09:10.457 INFO:tasks.workunit.client.0.smithi113.stdout:5/925: write d0/d10/d27/d1d/d10c/de7/ff2 [650836,95155] 0 2024-04-12T05:09:10.457 INFO:tasks.workunit.client.0.smithi113.stdout:2/783: dread d8/d9/de/d12/d2b/f8e [0,4194304] 0 2024-04-12T05:09:10.460 INFO:tasks.workunit.client.0.smithi113.stdout:2/784: chown d8/d9/d27/dda/f52 30386125 1 2024-04-12T05:09:10.460 INFO:tasks.workunit.client.0.smithi113.stdout:9/793: mkdir d1/d4/df9/d109 0 2024-04-12T05:09:10.461 INFO:tasks.workunit.client.0.smithi113.stdout:5/926: mkdir d0/d10/d27/d1d/d8c/db8/d13b 0 2024-04-12T05:09:10.462 INFO:tasks.workunit.client.0.smithi113.stdout:9/794: chown d1/d4/d5/d6/ffa 2637486 1 2024-04-12T05:09:10.463 INFO:tasks.workunit.client.1.smithi192.stdout:1/752: creat d0/d2/d1e/d3f/ff9 x:0 0 0 2024-04-12T05:09:10.463 INFO:tasks.workunit.client.1.smithi192.stdout:0/818: dread d0/d7b/d9b/d4b/fee [0,4194304] 0 2024-04-12T05:09:10.463 INFO:tasks.workunit.client.1.smithi192.stdout:5/793: dwrite d0/d15/d32/db2/fd2 [0,4194304] 0 2024-04-12T05:09:10.463 INFO:tasks.workunit.client.0.smithi113.stdout:0/898: rmdir d1/d6/d16/dd5/dbb 39 2024-04-12T05:09:10.465 INFO:tasks.workunit.client.1.smithi192.stdout:1/753: write d0/fec [528628,17688] 0 2024-04-12T05:09:10.468 INFO:tasks.workunit.client.1.smithi192.stdout:4/791: creat d1/d1d/d53/d1c/de1/f112 x:0 0 0 2024-04-12T05:09:10.469 INFO:tasks.workunit.client.1.smithi192.stdout:2/734: getdents d1/d2d 0 2024-04-12T05:09:10.471 INFO:tasks.workunit.client.1.smithi192.stdout:0/819: mknod d0/d7b/d9b/d35/d109/db0/c110 0 2024-04-12T05:09:10.472 INFO:tasks.workunit.client.1.smithi192.stdout:1/754: creat d0/d2/d1e/d3f/df2/d3c/d5b/da7/dd7/ffa x:0 0 0 2024-04-12T05:09:10.473 INFO:tasks.workunit.client.1.smithi192.stdout:4/792: write d1/d54/d55/d64/fd4 [6902020,4068] 0 2024-04-12T05:09:10.473 INFO:tasks.workunit.client.0.smithi113.stdout:2/785: getdents d8/d9/de/d43/db2 0 2024-04-12T05:09:10.474 INFO:tasks.workunit.client.0.smithi113.stdout:0/899: link d1/fa7 d1/d6/d16/d110/f112 0 2024-04-12T05:09:10.474 INFO:tasks.workunit.client.1.smithi192.stdout:9/591: creat d11/d15/d21/d2c/fcf x:0 0 0 2024-04-12T05:09:10.475 INFO:tasks.workunit.client.0.smithi113.stdout:0/900: chown d1/d6/d16/c3c 53 1 2024-04-12T05:09:10.475 INFO:tasks.workunit.client.1.smithi192.stdout:0/820: chown d0/d7b/d9b/d35/d109/d9f/d10a/d15/l8b 2 1 2024-04-12T05:09:10.476 INFO:tasks.workunit.client.0.smithi113.stdout:2/786: unlink d8/d16/dbd/fc9 0 2024-04-12T05:09:10.479 INFO:tasks.workunit.client.1.smithi192.stdout:1/755: rename d0/d1b/d28/d9d/fa4 to d0/d1b/d44/d69/ffb 0 2024-04-12T05:09:10.481 INFO:tasks.workunit.client.1.smithi192.stdout:1/756: fdatasync d0/d29/d8d/d96/fdb 0 2024-04-12T05:09:10.482 INFO:tasks.workunit.client.0.smithi113.stdout:5/927: dread d0/d10/f56 [0,4194304] 0 2024-04-12T05:09:10.482 INFO:tasks.workunit.client.1.smithi192.stdout:2/735: dwrite d1/d2d/d73/da8/db2/fd8 [0,4194304] 0 2024-04-12T05:09:10.485 INFO:tasks.workunit.client.1.smithi192.stdout:9/592: mknod d11/d15/d21/cd0 0 2024-04-12T05:09:10.486 INFO:tasks.workunit.client.0.smithi113.stdout:5/928: creat d0/d10/d27/d1d/d41/f13c x:0 0 0 2024-04-12T05:09:10.486 INFO:tasks.workunit.client.0.smithi113.stdout:0/901: creat d1/d43/d59/de9/d105/d50/f113 x:0 0 0 2024-04-12T05:09:10.488 INFO:tasks.workunit.client.0.smithi113.stdout:5/929: dread - d0/d10/d27/d1d/d3f/db0/dda/fdd zero size 2024-04-12T05:09:10.489 INFO:tasks.workunit.client.0.smithi113.stdout:5/930: write d0/d10/d6e/f35 [89201,126439] 0 2024-04-12T05:09:10.490 INFO:tasks.workunit.client.0.smithi113.stdout:5/931: readlink d0/d10/d27/d1d/d38/d75/l81 0 2024-04-12T05:09:10.491 INFO:tasks.workunit.client.0.smithi113.stdout:2/787: write d8/d9/de/f31 [4858774,24862] 0 2024-04-12T05:09:10.492 INFO:tasks.workunit.client.1.smithi192.stdout:0/821: symlink d0/d7b/d9b/d35/d109/d9f/d10a/de3/l111 0 2024-04-12T05:09:10.493 INFO:tasks.workunit.client.0.smithi113.stdout:5/932: write d0/d10/d27/d1d/d38/fba [4811836,29639] 0 2024-04-12T05:09:10.497 INFO:tasks.workunit.client.1.smithi192.stdout:2/736: symlink d1/d2d/d73/de7/le8 0 2024-04-12T05:09:10.498 INFO:tasks.workunit.client.0.smithi113.stdout:2/788: creat d8/d9/de/d12/d3c/d8a/ff7 x:0 0 0 2024-04-12T05:09:10.499 INFO:tasks.workunit.client.0.smithi113.stdout:0/902: rename d1/d43/d59/de9/d105/d50/fff to d1/d43/d59/df4/f114 0 2024-04-12T05:09:10.499 INFO:tasks.workunit.client.1.smithi192.stdout:4/793: dread d1/d54/d82/f62 [0,4194304] 0 2024-04-12T05:09:10.500 INFO:tasks.workunit.client.1.smithi192.stdout:1/757: mknod d0/d2/d1e/d3f/df2/d21/cfc 0 2024-04-12T05:09:10.501 INFO:tasks.workunit.client.1.smithi192.stdout:0/822: dwrite d0/d7b/d9b/d35/d109/d95/db4/db8/fcb [0,4194304] 0 2024-04-12T05:09:10.501 INFO:tasks.workunit.client.0.smithi113.stdout:2/789: chown d8/d9/d27/d88/fd7 2812 1 2024-04-12T05:09:10.503 INFO:tasks.workunit.client.0.smithi113.stdout:7/787: dwrite d0/d52/fbc [0,4194304] 0 2024-04-12T05:09:10.509 INFO:tasks.workunit.client.0.smithi113.stdout:9/795: dwrite d1/d27/f5d [0,4194304] 0 2024-04-12T05:09:10.512 INFO:tasks.workunit.client.0.smithi113.stdout:5/933: write d0/d10/da1/fc3 [912343,127510] 0 2024-04-12T05:09:10.513 INFO:tasks.workunit.client.1.smithi192.stdout:2/737: fdatasync d1/d8/d3a/de/f97 0 2024-04-12T05:09:10.517 INFO:tasks.workunit.client.1.smithi192.stdout:5/794: write d0/d21/f2f [4030250,89355] 0 2024-04-12T05:09:10.518 INFO:tasks.workunit.client.0.smithi113.stdout:7/788: rename d0/d52/dc5 to d0/dc/d10b 0 2024-04-12T05:09:10.518 INFO:tasks.workunit.client.0.smithi113.stdout:2/790: rmdir d8/d9/de/d12/dbf 39 2024-04-12T05:09:10.520 INFO:tasks.workunit.client.0.smithi113.stdout:0/903: write d1/fd7 [3808690,87095] 0 2024-04-12T05:09:10.523 INFO:tasks.workunit.client.0.smithi113.stdout:5/934: symlink d0/d10/d27/d1d/d3f/d7e/df8/d108/l13d 0 2024-04-12T05:09:10.525 INFO:tasks.workunit.client.0.smithi113.stdout:5/935: write d0/d10/d27/d1d/d3f/db0/feb [4930178,39075] 0 2024-04-12T05:09:10.527 INFO:tasks.workunit.client.0.smithi113.stdout:7/789: creat d0/dc/d6/f10c x:0 0 0 2024-04-12T05:09:10.527 INFO:tasks.workunit.client.1.smithi192.stdout:0/823: mknod d0/d7b/d9b/d35/d109/d9f/d10a/c112 0 2024-04-12T05:09:10.530 INFO:tasks.workunit.client.0.smithi113.stdout:7/790: fsync d0/dc/f41 0 2024-04-12T05:09:10.530 INFO:tasks.workunit.client.1.smithi192.stdout:4/794: creat d1/d1d/f113 x:0 0 0 2024-04-12T05:09:10.533 INFO:tasks.workunit.client.0.smithi113.stdout:5/936: getdents d0/d10/d27/d5f/d128 0 2024-04-12T05:09:10.534 INFO:tasks.workunit.client.0.smithi113.stdout:5/937: chown d0/d10/f58 615194 1 2024-04-12T05:09:10.535 INFO:tasks.workunit.client.1.smithi192.stdout:5/795: dwrite d0/d15/d64/f100 [4194304,4194304] 0 2024-04-12T05:09:10.537 INFO:tasks.workunit.client.1.smithi192.stdout:9/593: write d11/d25/d36/d6e/d33/da2/fc9 [1341527,39327] 0 2024-04-12T05:09:10.537 INFO:tasks.workunit.client.0.smithi113.stdout:5/938: creat d0/d10/d6e/d93/f13e x:0 0 0 2024-04-12T05:09:10.539 INFO:tasks.workunit.client.0.smithi113.stdout:5/939: chown d0/d10/d27/d1d/d10c/f106 25 1 2024-04-12T05:09:10.541 INFO:tasks.workunit.client.1.smithi192.stdout:5/796: truncate d0/d15/d24/d29/f85 1186015 0 2024-04-12T05:09:10.541 INFO:tasks.workunit.client.1.smithi192.stdout:0/824: link d0/d7b/d9b/d35/d109/d53/df3/l88 d0/d7b/d9b/d35/d109/db0/l113 0 2024-04-12T05:09:10.542 INFO:tasks.workunit.client.1.smithi192.stdout:5/797: chown d0/d15/d3e/dca/dda/db5 448130554 1 2024-04-12T05:09:10.544 INFO:tasks.workunit.client.1.smithi192.stdout:0/825: write d0/d7b/d9b/d35/d109/d9f/d10a/db2/ff8 [305256,104677] 0 2024-04-12T05:09:10.546 INFO:tasks.workunit.client.1.smithi192.stdout:4/795: rename d1/d49/db5 to d1/d1d/d53/d1c/dd1/def/ddd/d114 0 2024-04-12T05:09:10.549 INFO:tasks.workunit.client.0.smithi113.stdout:9/796: dread d1/d4/d87/ff8 [0,4194304] 0 2024-04-12T05:09:10.550 INFO:tasks.workunit.client.1.smithi192.stdout:5/798: mkdir d0/d15/d3e/dca/db9/d112 0 2024-04-12T05:09:10.550 INFO:tasks.workunit.client.0.smithi113.stdout:0/904: write d1/fa6 [1007128,20113] 0 2024-04-12T05:09:10.551 INFO:tasks.workunit.client.1.smithi192.stdout:0/826: rmdir d0/d7b/d9b/d35/d109/db0/dbf/dde 39 2024-04-12T05:09:10.553 INFO:tasks.workunit.client.0.smithi113.stdout:7/791: dread d0/dd/d11/f5e [0,4194304] 0 2024-04-12T05:09:10.554 INFO:tasks.workunit.client.1.smithi192.stdout:9/594: getdents d11/d15/d21/d2c/d1f/d50 0 2024-04-12T05:09:10.555 INFO:tasks.workunit.client.0.smithi113.stdout:0/905: fdatasync d1/d6/d34/dc3/f1a 0 2024-04-12T05:09:10.555 INFO:tasks.workunit.client.1.smithi192.stdout:4/796: dwrite d1/d1d/d53/f89 [0,4194304] 0 2024-04-12T05:09:10.555 INFO:tasks.workunit.client.1.smithi192.stdout:2/738: write d1/d8/d3a/f77 [665540,45824] 0 2024-04-12T05:09:10.557 INFO:tasks.workunit.client.1.smithi192.stdout:5/799: mkdir d0/d15/d3e/dca/dda/deb/d113 0 2024-04-12T05:09:10.559 INFO:tasks.workunit.client.1.smithi192.stdout:9/595: mknod d11/d25/d36/cd1 0 2024-04-12T05:09:10.559 INFO:tasks.workunit.client.0.smithi113.stdout:0/906: symlink d1/d6/l115 0 2024-04-12T05:09:10.560 INFO:tasks.workunit.client.1.smithi192.stdout:5/800: write d0/d21/f101 [1891439,11836] 0 2024-04-12T05:09:10.560 INFO:tasks.workunit.client.0.smithi113.stdout:7/792: write d0/dc/da9/d5c/d85/fed [1119484,105911] 0 2024-04-12T05:09:10.561 INFO:tasks.workunit.client.1.smithi192.stdout:0/827: creat d0/d7b/d9b/d35/d109/d9f/d10a/dc4/f114 x:0 0 0 2024-04-12T05:09:10.566 INFO:tasks.workunit.client.0.smithi113.stdout:9/797: dwrite d1/d4/d5/d17/f74 [0,4194304] 0 2024-04-12T05:09:10.568 INFO:tasks.workunit.client.1.smithi192.stdout:2/739: creat d1/d8/d17/d19/d60/d59/d68/fe9 x:0 0 0 2024-04-12T05:09:10.570 INFO:tasks.workunit.client.0.smithi113.stdout:9/798: read d1/d4/d5/f58 [2970428,16163] 0 2024-04-12T05:09:10.571 INFO:tasks.workunit.client.1.smithi192.stdout:4/797: mkdir d1/d54/d82/d71/d9e/d10b/de9/d115 0 2024-04-12T05:09:10.572 INFO:tasks.workunit.client.0.smithi113.stdout:0/907: creat d1/d43/d59/de9/d105/d50/db6/d36/f116 x:0 0 0 2024-04-12T05:09:10.573 INFO:tasks.workunit.client.0.smithi113.stdout:9/799: write d1/df/db6/f59 [1962255,120289] 0 2024-04-12T05:09:10.574 INFO:tasks.workunit.client.1.smithi192.stdout:5/801: mkdir d0/d15/d3e/dca/dda/db5/d114 0 2024-04-12T05:09:10.574 INFO:tasks.workunit.client.0.smithi113.stdout:9/800: fdatasync d1/df/db6/f84 0 2024-04-12T05:09:10.575 INFO:tasks.workunit.client.1.smithi192.stdout:0/828: stat d0/d7b/d9b/d35/d109/d9f/d10a/cb6 0 2024-04-12T05:09:10.577 INFO:tasks.workunit.client.0.smithi113.stdout:9/801: readlink d1/d4/d30/l5e 0 2024-04-12T05:09:10.578 INFO:tasks.workunit.client.1.smithi192.stdout:2/740: mkdir d1/d8/d17/d19/d60/d59/d68/dea 0 2024-04-12T05:09:10.579 INFO:tasks.workunit.client.0.smithi113.stdout:9/802: fdatasync d1/df/fc3 0 2024-04-12T05:09:10.581 INFO:tasks.workunit.client.1.smithi192.stdout:9/596: write d11/d15/d21/f65 [197305,11611] 0 2024-04-12T05:09:10.582 INFO:tasks.workunit.client.0.smithi113.stdout:0/908: rename d1/d6/d16/d8f/fce to d1/d43/d59/de9/f117 0 2024-04-12T05:09:10.584 INFO:tasks.workunit.client.1.smithi192.stdout:2/741: dread d1/d2d/d73/da8/db2/fd8 [0,4194304] 0 2024-04-12T05:09:10.585 INFO:tasks.workunit.client.1.smithi192.stdout:4/798: rmdir d1/d1d/d53/d1c/d95/dc6/dd5 39 2024-04-12T05:09:10.585 INFO:tasks.workunit.client.0.smithi113.stdout:9/803: rename d1/d4/d5/d22/cf3 to d1/d4/df9/d109/c10a 0 2024-04-12T05:09:10.587 INFO:tasks.workunit.client.1.smithi192.stdout:9/597: creat d11/d25/d36/d6e/fd2 x:0 0 0 2024-04-12T05:09:10.587 INFO:tasks.workunit.client.1.smithi192.stdout:4/799: write d1/d1d/f10f [311047,91828] 0 2024-04-12T05:09:10.588 INFO:tasks.workunit.client.0.smithi113.stdout:9/804: mknod d1/d4/d6d/c10b 0 2024-04-12T05:09:10.589 INFO:tasks.workunit.client.1.smithi192.stdout:2/742: unlink d1/d8/d17/d19/d60/d59/cc2 0 2024-04-12T05:09:10.589 INFO:tasks.workunit.client.1.smithi192.stdout:4/800: fdatasync d1/d54/d82/da4/f79 0 2024-04-12T05:09:10.590 INFO:tasks.workunit.client.1.smithi192.stdout:9/598: mkdir d11/d25/dd3 0 2024-04-12T05:09:10.591 INFO:tasks.workunit.client.1.smithi192.stdout:4/801: stat d1/d54/d82/da4 0 2024-04-12T05:09:10.592 INFO:tasks.workunit.client.1.smithi192.stdout:4/802: chown d1/d54/d82/d71/dff 5 1 2024-04-12T05:09:10.592 INFO:tasks.workunit.client.1.smithi192.stdout:2/743: mknod d1/d2d/d73/da8/db2/ceb 0 2024-04-12T05:09:10.593 INFO:tasks.workunit.client.1.smithi192.stdout:4/803: dread - d1/d54/d82/d71/d9e/f10e zero size 2024-04-12T05:09:10.594 INFO:tasks.workunit.client.1.smithi192.stdout:2/744: creat d1/d2d/d73/de7/fec x:0 0 0 2024-04-12T05:09:10.596 INFO:tasks.workunit.client.1.smithi192.stdout:9/599: rename d11/d25/d36/d3d/d7a/ca5 to d11/d15/d21/d2c/d1f/d6a/d7e/cd4 0 2024-04-12T05:09:10.599 INFO:tasks.workunit.client.1.smithi192.stdout:4/804: mkdir d1/d54/d55/d64/d116 0 2024-04-12T05:09:10.599 INFO:tasks.workunit.client.1.smithi192.stdout:2/745: write d1/d2d/f34 [1225067,53981] 0 2024-04-12T05:09:10.600 INFO:tasks.workunit.client.1.smithi192.stdout:9/600: mkdir d11/d25/dd3/dd5 0 2024-04-12T05:09:10.603 INFO:tasks.workunit.client.1.smithi192.stdout:5/802: dwrite d0/d15/da1/df9/f107 [0,4194304] 0 2024-04-12T05:09:10.603 INFO:tasks.workunit.client.1.smithi192.stdout:9/601: fdatasync d11/d25/f91 0 2024-04-12T05:09:10.605 INFO:tasks.workunit.client.1.smithi192.stdout:2/746: write d1/d8/d17/d19/f5e [190196,21267] 0 2024-04-12T05:09:10.605 INFO:tasks.workunit.client.0.smithi113.stdout:9/805: dread d1/d27/f3c [0,4194304] 0 2024-04-12T05:09:10.605 INFO:tasks.workunit.client.0.smithi113.stdout:9/806: readlink d1/d4/d5/l72 0 2024-04-12T05:09:10.607 INFO:tasks.workunit.client.0.smithi113.stdout:9/807: dread - d1/d27/d7a/dd2/fdc zero size 2024-04-12T05:09:10.608 INFO:tasks.workunit.client.1.smithi192.stdout:0/829: dwrite d0/d7b/d9b/d35/d109/d53/df3/f48 [0,4194304] 0 2024-04-12T05:09:10.609 INFO:tasks.workunit.client.1.smithi192.stdout:4/805: creat d1/d49/f117 x:0 0 0 2024-04-12T05:09:10.613 INFO:tasks.workunit.client.1.smithi192.stdout:4/806: write d1/d1d/d53/d1c/dd1/def/ddd/d114/ff1 [479775,7728] 0 2024-04-12T05:09:10.616 INFO:tasks.workunit.client.0.smithi113.stdout:9/808: dwrite d1/df/d29/da6/fdb [4194304,4194304] 0 2024-04-12T05:09:10.618 INFO:tasks.workunit.client.1.smithi192.stdout:2/747: rmdir d1/d8/d17/d19 39 2024-04-12T05:09:10.619 INFO:tasks.workunit.client.0.smithi113.stdout:4/781: sync 2024-04-12T05:09:10.619 INFO:tasks.workunit.client.0.smithi113.stdout:1/870: sync 2024-04-12T05:09:10.619 INFO:tasks.workunit.client.0.smithi113.stdout:8/822: sync 2024-04-12T05:09:10.620 INFO:tasks.workunit.client.0.smithi113.stdout:9/809: mknod d1/d4/df9/c10c 0 2024-04-12T05:09:10.621 INFO:tasks.workunit.client.1.smithi192.stdout:9/602: creat d11/d25/d36/d6e/dc7/fd6 x:0 0 0 2024-04-12T05:09:10.622 INFO:tasks.workunit.client.1.smithi192.stdout:2/748: truncate d1/d8/d17/f2a 583355 0 2024-04-12T05:09:10.623 INFO:tasks.workunit.client.1.smithi192.stdout:0/830: rename d0/d7b/d9b/d35/d109/d9f/d10a/d67/ccd to d0/dc5/c115 0 2024-04-12T05:09:10.624 INFO:tasks.workunit.client.0.smithi113.stdout:1/871: creat d10/d41/d89/d96/d7d/f117 x:0 0 0 2024-04-12T05:09:10.626 INFO:tasks.workunit.client.0.smithi113.stdout:4/782: rename da/de to da/d3c/d9a/dc1/d106 0 2024-04-12T05:09:10.627 INFO:tasks.workunit.client.0.smithi113.stdout:1/872: mknod d10/d21/c118 0 2024-04-12T05:09:10.628 INFO:tasks.workunit.client.1.smithi192.stdout:0/831: chown d0/d7b/d9b/d35/d109/d53/df3/d2e/d66/f86 504 1 2024-04-12T05:09:10.629 INFO:tasks.workunit.client.1.smithi192.stdout:0/832: chown d0/d7b/d9b/d35/d109/d53/df3/d2e/f9e 15 1 2024-04-12T05:09:10.629 INFO:tasks.workunit.client.0.smithi113.stdout:4/783: mknod da/d24/d60/db8/c107 0 2024-04-12T05:09:10.631 INFO:tasks.workunit.client.0.smithi113.stdout:9/810: rename d1/d4/d5/d22/cce to d1/d4/d38/d65/c10d 0 2024-04-12T05:09:10.632 INFO:tasks.workunit.client.0.smithi113.stdout:4/784: creat da/d3c/d9a/dc1/d106/dc8/f108 x:0 0 0 2024-04-12T05:09:10.633 INFO:tasks.workunit.client.0.smithi113.stdout:9/811: mkdir d1/d27/d7a/dd2/d10e 0 2024-04-12T05:09:10.633 INFO:tasks.workunit.client.1.smithi192.stdout:0/833: mkdir d0/d7b/d9b/d35/d109/d9f/d10a/d67/db1/dd6/df2/d116 0 2024-04-12T05:09:10.635 INFO:tasks.workunit.client.1.smithi192.stdout:5/803: dread d0/f3d [4194304,4194304] 0 2024-04-12T05:09:10.637 INFO:tasks.workunit.client.1.smithi192.stdout:0/834: mknod d0/d7b/d9b/d35/d109/d53/df3/d29/db5/c117 0 2024-04-12T05:09:10.638 INFO:tasks.workunit.client.1.smithi192.stdout:5/804: creat d0/d15/d32/db2/f115 x:0 0 0 2024-04-12T05:09:10.639 INFO:tasks.workunit.client.1.smithi192.stdout:0/835: chown d0/d7b/d9b/d35/d109/d9f/d10a/c1a 1640 1 2024-04-12T05:09:10.641 INFO:tasks.workunit.client.1.smithi192.stdout:5/805: unlink d0/f52 0 2024-04-12T05:09:10.641 INFO:tasks.workunit.client.0.smithi113.stdout:8/823: dread d9/d16/d65/d8c/fa4 [0,4194304] 0 2024-04-12T05:09:10.644 INFO:tasks.workunit.client.0.smithi113.stdout:8/824: rename d9/c11 to d9/d16/dc5/c109 0 2024-04-12T05:09:10.646 INFO:tasks.workunit.client.1.smithi192.stdout:0/836: getdents d0/d7b/d9b/d35 0 2024-04-12T05:09:10.646 INFO:tasks.workunit.client.1.smithi192.stdout:4/807: write d1/f29 [2090662,54891] 0 2024-04-12T05:09:10.646 INFO:tasks.workunit.client.0.smithi113.stdout:4/785: dread da/dd/f48 [0,4194304] 0 2024-04-12T05:09:10.648 INFO:tasks.workunit.client.1.smithi192.stdout:4/808: chown d1/d1d/d53/db2/lbd 16094 1 2024-04-12T05:09:10.651 INFO:tasks.workunit.client.1.smithi192.stdout:0/837: creat d0/d7b/d9b/d4b/d72/f118 x:0 0 0 2024-04-12T05:09:10.652 INFO:tasks.workunit.client.1.smithi192.stdout:0/838: chown d0/d7b/d9b/d4b 23 1 2024-04-12T05:09:10.654 INFO:tasks.workunit.client.0.smithi113.stdout:8/825: getdents d9/d1b/d8e/d93/d32 0 2024-04-12T05:09:10.654 INFO:tasks.workunit.client.0.smithi113.stdout:8/826: chown d9/d16/d65/db1/dec 54156 1 2024-04-12T05:09:10.655 INFO:tasks.workunit.client.0.smithi113.stdout:8/827: chown d9/le 41444 1 2024-04-12T05:09:10.656 INFO:tasks.workunit.client.0.smithi113.stdout:9/812: getdents d1/d4/df9 0 2024-04-12T05:09:10.658 INFO:tasks.workunit.client.0.smithi113.stdout:4/786: link da/d24/d7b/ldf da/d70/l109 0 2024-04-12T05:09:10.658 INFO:tasks.workunit.client.0.smithi113.stdout:1/873: dread d10/d41/d89/d96/d7d/f79 [0,4194304] 0 2024-04-12T05:09:10.659 INFO:tasks.workunit.client.1.smithi192.stdout:9/603: dwrite d11/d25/d36/d6e/f56 [0,4194304] 0 2024-04-12T05:09:10.661 INFO:tasks.workunit.client.1.smithi192.stdout:0/839: creat d0/d7b/d9b/d35/d109/d53/df3/f119 x:0 0 0 2024-04-12T05:09:10.662 INFO:tasks.workunit.client.0.smithi113.stdout:8/828: mkdir d9/d16/d65/db1/d59/da3/d10a 0 2024-04-12T05:09:10.664 INFO:tasks.workunit.client.0.smithi113.stdout:8/829: dread - d9/d16/d65/db1/d59/d78/ff7 zero size 2024-04-12T05:09:10.666 INFO:tasks.workunit.client.1.smithi192.stdout:4/809: dread d1/f4b [0,4194304] 0 2024-04-12T05:09:10.667 INFO:tasks.workunit.client.0.smithi113.stdout:4/787: creat da/d24/d7b/f10a x:0 0 0 2024-04-12T05:09:10.667 INFO:tasks.workunit.client.0.smithi113.stdout:1/874: write d10/d2e/d31/d5c/f74 [990918,17287] 0 2024-04-12T05:09:10.669 INFO:tasks.workunit.client.0.smithi113.stdout:9/813: mkdir d1/d4/d5/d22/d9e/df1/dff/d10f 0 2024-04-12T05:09:10.670 INFO:tasks.workunit.client.0.smithi113.stdout:9/814: chown d1/df/d29/ce7 9813 1 2024-04-12T05:09:10.672 INFO:tasks.workunit.client.0.smithi113.stdout:4/788: creat da/dd/d3a/f10b x:0 0 0 2024-04-12T05:09:10.673 INFO:tasks.workunit.client.0.smithi113.stdout:8/830: symlink d9/d16/d65/db1/dec/l10b 0 2024-04-12T05:09:10.674 INFO:tasks.workunit.client.1.smithi192.stdout:4/810: unlink d1/d1d/d53/d1c/dd1/def/faa 0 2024-04-12T05:09:10.674 INFO:tasks.workunit.client.1.smithi192.stdout:0/840: link d0/d7b/d9b/d35/d109/d53/df3/l88 d0/d7b/d9b/d35/d109/d95/db4/db8/l11a 0 2024-04-12T05:09:10.675 INFO:tasks.workunit.client.0.smithi113.stdout:8/831: dread - d9/d1b/d8e/d93/fb7 zero size 2024-04-12T05:09:10.675 INFO:tasks.workunit.client.0.smithi113.stdout:1/875: fsync d10/d2e/d7c/f92 0 2024-04-12T05:09:10.675 INFO:tasks.workunit.client.1.smithi192.stdout:9/604: dwrite d11/d25/d36/d6e/f56 [4194304,4194304] 0 2024-04-12T05:09:10.676 INFO:tasks.workunit.client.0.smithi113.stdout:9/815: link d1/d4/f88 d1/d4/d38/d65/dcd/f110 0 2024-04-12T05:09:10.677 INFO:tasks.workunit.client.1.smithi192.stdout:9/605: rename d11/d25 to d11/d25/d36/d6e/d8f/dd7 22 2024-04-12T05:09:10.677 INFO:tasks.workunit.client.0.smithi113.stdout:4/789: fdatasync da/d3c/d43/f75 0 2024-04-12T05:09:10.678 INFO:tasks.workunit.client.1.smithi192.stdout:9/606: read - d11/d25/d36/d3d/fb7 zero size 2024-04-12T05:09:10.679 INFO:tasks.workunit.client.0.smithi113.stdout:4/790: fsync da/d24/d7e/d79/dcd/fe4 0 2024-04-12T05:09:10.680 INFO:tasks.workunit.client.1.smithi192.stdout:4/811: symlink d1/d1d/d53/db2/l118 0 2024-04-12T05:09:10.680 INFO:tasks.workunit.client.0.smithi113.stdout:1/876: creat d10/d2e/d31/dc3/f119 x:0 0 0 2024-04-12T05:09:10.681 INFO:tasks.workunit.client.1.smithi192.stdout:4/812: chown d1/d54/d82/d71/d9e/ff8 1261 1 2024-04-12T05:09:10.683 INFO:tasks.workunit.client.0.smithi113.stdout:1/877: chown d10/d2e/d31/d5c/f114 16543711 1 2024-04-12T05:09:10.684 INFO:tasks.workunit.client.0.smithi113.stdout:9/816: mkdir d1/df/d4b/df7/d111 0 2024-04-12T05:09:10.686 INFO:tasks.workunit.client.1.smithi192.stdout:9/607: link d11/laa d11/d15/d21/ld8 0 2024-04-12T05:09:10.686 INFO:tasks.workunit.client.0.smithi113.stdout:9/817: write d1/d4/d5/d6/ffa [81623,55005] 0 2024-04-12T05:09:10.686 INFO:tasks.workunit.client.0.smithi113.stdout:4/791: creat da/dd/f10c x:0 0 0 2024-04-12T05:09:10.687 INFO:tasks.workunit.client.1.smithi192.stdout:9/608: readlink d11/d15/d21/d2c/d1f/d94/la7 0 2024-04-12T05:09:10.687 INFO:tasks.workunit.client.0.smithi113.stdout:1/878: mknod d10/c11a 0 2024-04-12T05:09:10.688 INFO:tasks.workunit.client.0.smithi113.stdout:9/818: stat d1/df/d4b/d63/d3d/d6f/faf 0 2024-04-12T05:09:10.688 INFO:tasks.workunit.client.0.smithi113.stdout:4/792: symlink da/d3c/d43/dbc/l10d 0 2024-04-12T05:09:10.689 INFO:tasks.workunit.client.0.smithi113.stdout:9/819: chown d1/df/db6/la5 60588 1 2024-04-12T05:09:10.689 INFO:tasks.workunit.client.0.smithi113.stdout:1/879: chown d10/d41/d89/d96/dd3/f7a 4861 1 2024-04-12T05:09:10.690 INFO:tasks.workunit.client.0.smithi113.stdout:8/832: truncate d9/d1b/d8e/d93/f71 192529 0 2024-04-12T05:09:10.690 INFO:tasks.workunit.client.0.smithi113.stdout:9/820: dread - d1/d4/d5/d6/f76 zero size 2024-04-12T05:09:10.691 INFO:tasks.workunit.client.1.smithi192.stdout:9/609: rename d11/d15/d21/d2c/d1f/f4d to d11/d25/d36/d3d/fd9 0 2024-04-12T05:09:10.691 INFO:tasks.workunit.client.0.smithi113.stdout:4/793: chown da/dd/d3a/f42 17 1 2024-04-12T05:09:10.692 INFO:tasks.workunit.client.0.smithi113.stdout:9/821: truncate d1/df/fc3 1123949 0 2024-04-12T05:09:10.693 INFO:tasks.workunit.client.0.smithi113.stdout:8/833: write d9/d16/d65/db1/d59/d78/f101 [981755,39540] 0 2024-04-12T05:09:10.694 INFO:tasks.workunit.client.0.smithi113.stdout:1/880: stat d10/d2e/dd8/d70/l62 0 2024-04-12T05:09:10.694 INFO:tasks.workunit.client.0.smithi113.stdout:9/822: dread - d1/df/d29/fd3 zero size 2024-04-12T05:09:10.694 INFO:tasks.workunit.client.1.smithi192.stdout:9/610: chown d11/l51 335274747 1 2024-04-12T05:09:10.696 INFO:tasks.workunit.client.0.smithi113.stdout:4/794: unlink da/d3c/d43/dbc/ccc 0 2024-04-12T05:09:10.697 INFO:tasks.workunit.client.0.smithi113.stdout:4/795: write da/d2b/f3e [4794608,99835] 0 2024-04-12T05:09:10.699 INFO:tasks.workunit.client.0.smithi113.stdout:8/834: unlink d9/d16/d65/db1/d106/ff6 0 2024-04-12T05:09:10.703 INFO:tasks.workunit.client.0.smithi113.stdout:8/835: creat d9/d16/d7f/d3a/d6c/f10c x:0 0 0 2024-04-12T05:09:10.704 INFO:tasks.workunit.client.0.smithi113.stdout:1/881: getdents d10/d2e/d31/dc3/d109 0 2024-04-12T05:09:10.704 INFO:tasks.workunit.client.0.smithi113.stdout:1/882: write f5 [521129,23639] 0 2024-04-12T05:09:10.705 INFO:tasks.workunit.client.0.smithi113.stdout:8/836: fdatasync d9/d16/d7f/d3a/f5e 0 2024-04-12T05:09:10.710 INFO:tasks.workunit.client.0.smithi113.stdout:8/837: chown d9/d1b/d8e/d93/d32/d48/d8d/d4a/fbd 5 1 2024-04-12T05:09:10.713 INFO:tasks.workunit.client.0.smithi113.stdout:8/838: read - d9/d16/d65/f6a zero size 2024-04-12T05:09:10.713 INFO:tasks.workunit.client.0.smithi113.stdout:4/796: dread da/dd/f32 [0,4194304] 0 2024-04-12T05:09:10.715 INFO:tasks.workunit.client.0.smithi113.stdout:4/797: creat da/d3c/da5/f10e x:0 0 0 2024-04-12T05:09:10.715 INFO:tasks.workunit.client.0.smithi113.stdout:1/883: dwrite d10/d41/d89/d96/dd3/f100 [0,4194304] 0 2024-04-12T05:09:10.718 INFO:tasks.workunit.client.1.smithi192.stdout:7/645: sync 2024-04-12T05:09:10.718 INFO:tasks.workunit.client.1.smithi192.stdout:6/754: sync 2024-04-12T05:09:10.720 INFO:tasks.workunit.client.0.smithi113.stdout:4/798: rename da/d4f to da/d3c/d83/d47/d10f 0 2024-04-12T05:09:10.721 INFO:tasks.workunit.client.0.smithi113.stdout:1/884: creat d10/d2e/d31/dc3/d109/f11b x:0 0 0 2024-04-12T05:09:10.721 INFO:tasks.workunit.client.0.smithi113.stdout:4/799: write da/dd/f10c [337667,79349] 0 2024-04-12T05:09:10.723 INFO:tasks.workunit.client.0.smithi113.stdout:1/885: write d10/d2e/d31/dc3/d109/f11b [870031,3662] 0 2024-04-12T05:09:10.728 INFO:tasks.workunit.client.0.smithi113.stdout:1/886: rename d10/d21/f2b to d10/d2e/d7c/dc5/dc8/d112/f11c 0 2024-04-12T05:09:10.731 INFO:tasks.workunit.client.0.smithi113.stdout:4/800: mkdir da/d24/d7b/d110 0 2024-04-12T05:09:10.732 INFO:tasks.workunit.client.0.smithi113.stdout:1/887: unlink d10/d2e/f54 0 2024-04-12T05:09:10.736 INFO:tasks.workunit.client.0.smithi113.stdout:1/888: creat d10/da5/ddf/f11d x:0 0 0 2024-04-12T05:09:10.738 INFO:tasks.workunit.client.0.smithi113.stdout:4/801: link da/d24/d7e/cd4 da/d3c/d43/dbc/c111 0 2024-04-12T05:09:10.739 INFO:tasks.workunit.client.1.smithi192.stdout:9/611: symlink d11/d15/d21/d2c/lda 0 2024-04-12T05:09:10.739 INFO:tasks.workunit.client.0.smithi113.stdout:1/889: fsync d10/d2e/d31/d5c/f114 0 2024-04-12T05:09:10.745 INFO:tasks.workunit.client.0.smithi113.stdout:1/890: chown d10/c20 1741124 1 2024-04-12T05:09:10.746 INFO:tasks.workunit.client.0.smithi113.stdout:8/839: dwrite d9/d1b/d8e/d93/d32/d5b/fa1 [0,4194304] 0 2024-04-12T05:09:10.747 INFO:tasks.workunit.client.1.smithi192.stdout:6/755: link d3/d8/f6d d3/d8/d22/dd4/ff9 0 2024-04-12T05:09:10.748 INFO:tasks.workunit.client.1.smithi192.stdout:7/646: truncate d3/d7/d15/f29 5040055 0 2024-04-12T05:09:10.750 INFO:tasks.workunit.client.0.smithi113.stdout:1/891: dread d10/d41/d5b/fd2 [0,4194304] 0 2024-04-12T05:09:10.752 INFO:tasks.workunit.client.1.smithi192.stdout:9/612: creat d11/d25/d36/d6e/d33/dbb/fdb x:0 0 0 2024-04-12T05:09:10.752 INFO:tasks.workunit.client.0.smithi113.stdout:1/892: chown d10/d14/d95/c103 28429 1 2024-04-12T05:09:10.752 INFO:tasks.workunit.client.0.smithi113.stdout:8/840: creat d9/d16/f10d x:0 0 0 2024-04-12T05:09:10.753 INFO:tasks.workunit.client.1.smithi192.stdout:0/841: dwrite d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d46/f49 [4194304,4194304] 0 2024-04-12T05:09:10.753 INFO:tasks.workunit.client.0.smithi113.stdout:1/893: chown d10/d14/f1f 99644 1 2024-04-12T05:09:10.756 INFO:tasks.workunit.client.1.smithi192.stdout:9/613: chown d11/d25 118513 1 2024-04-12T05:09:10.758 INFO:tasks.workunit.client.1.smithi192.stdout:9/614: fdatasync d11/d25/f91 0 2024-04-12T05:09:10.758 INFO:tasks.workunit.client.0.smithi113.stdout:4/802: dread da/d73/f71 [0,4194304] 0 2024-04-12T05:09:10.760 INFO:tasks.workunit.client.0.smithi113.stdout:8/841: symlink d9/d16/d65/db1/d59/l10e 0 2024-04-12T05:09:10.762 INFO:tasks.workunit.client.0.smithi113.stdout:8/842: readlink d9/d1b/d8e/d93/le4 0 2024-04-12T05:09:10.762 INFO:tasks.workunit.client.0.smithi113.stdout:1/894: dwrite d10/d2e/d31/dc3/fd5 [0,4194304] 0 2024-04-12T05:09:10.763 INFO:tasks.workunit.client.1.smithi192.stdout:7/647: rmdir d3/d11/dd8 0 2024-04-12T05:09:10.764 INFO:tasks.workunit.client.0.smithi113.stdout:4/803: rmdir da/d24/d60/db4 39 2024-04-12T05:09:10.765 INFO:tasks.workunit.client.1.smithi192.stdout:6/756: dwrite d3/d8/d10/d4c/fdb [0,4194304] 0 2024-04-12T05:09:10.766 INFO:tasks.workunit.client.1.smithi192.stdout:6/757: stat d3/d5/l34 0 2024-04-12T05:09:10.766 INFO:tasks.workunit.client.0.smithi113.stdout:8/843: creat d9/d1b/d96/f10f x:0 0 0 2024-04-12T05:09:10.768 INFO:tasks.workunit.client.1.smithi192.stdout:7/648: readlink d3/d1d/l7f 0 2024-04-12T05:09:10.768 INFO:tasks.workunit.client.1.smithi192.stdout:9/615: unlink d11/d25/d36/d6e/d33/da2/f39 0 2024-04-12T05:09:10.769 INFO:tasks.workunit.client.0.smithi113.stdout:8/844: rmdir d9/d16/d65/db1/dec 39 2024-04-12T05:09:10.770 INFO:tasks.workunit.client.1.smithi192.stdout:0/842: dwrite d0/d7b/d9b/d35/d109/d53/df3/f119 [0,4194304] 0 2024-04-12T05:09:10.771 INFO:tasks.workunit.client.0.smithi113.stdout:9/823: dread d1/d4/d30/dcb/f3f [0,4194304] 0 2024-04-12T05:09:10.773 INFO:tasks.workunit.client.0.smithi113.stdout:8/845: creat d9/dd4/f110 x:0 0 0 2024-04-12T05:09:10.775 INFO:tasks.workunit.client.0.smithi113.stdout:9/824: mknod d1/d4/d5/d22/d9e/df1/dff/c112 0 2024-04-12T05:09:10.775 INFO:tasks.workunit.client.0.smithi113.stdout:1/895: dread d10/d55/f8c [0,4194304] 0 2024-04-12T05:09:10.777 INFO:tasks.workunit.client.0.smithi113.stdout:1/896: symlink d10/d2e/d7c/dc5/dc8/l11e 0 2024-04-12T05:09:10.778 INFO:tasks.workunit.client.0.smithi113.stdout:8/846: mkdir d9/d16/d65/db1/dec/d111 0 2024-04-12T05:09:10.780 INFO:tasks.workunit.client.0.smithi113.stdout:8/847: write d9/d16/d65/db1/d59/da3/ffb [312358,57449] 0 2024-04-12T05:09:10.781 INFO:tasks.workunit.client.1.smithi192.stdout:9/616: read d11/f69 [3405191,30611] 0 2024-04-12T05:09:10.782 INFO:tasks.workunit.client.1.smithi192.stdout:7/649: creat d3/d7/fdf x:0 0 0 2024-04-12T05:09:10.783 INFO:tasks.workunit.client.1.smithi192.stdout:9/617: write d11/d15/d21/d2c/fcb [711732,99950] 0 2024-04-12T05:09:10.787 INFO:tasks.workunit.client.1.smithi192.stdout:7/650: truncate d3/d7/d60/d65/fb5 76666 0 2024-04-12T05:09:10.788 INFO:tasks.workunit.client.1.smithi192.stdout:0/843: dwrite d0/d7b/d9b/f38 [0,4194304] 0 2024-04-12T05:09:10.788 INFO:tasks.workunit.client.1.smithi192.stdout:9/618: mknod d11/d15/d21/d2c/d1f/cdc 0 2024-04-12T05:09:10.790 INFO:tasks.workunit.client.1.smithi192.stdout:7/651: mknod d3/ce0 0 2024-04-12T05:09:10.792 INFO:tasks.workunit.client.1.smithi192.stdout:7/652: chown d3/d1d/d5a 1740774998 1 2024-04-12T05:09:10.793 INFO:tasks.workunit.client.1.smithi192.stdout:9/619: write d11/d25/d36/d6e/d71/f95 [502435,123387] 0 2024-04-12T05:09:10.794 INFO:tasks.workunit.client.1.smithi192.stdout:7/653: dread - d3/d1d/d5a/d5f/fbd zero size 2024-04-12T05:09:10.794 INFO:tasks.workunit.client.1.smithi192.stdout:0/844: creat d0/d7b/d9b/d35/d109/d53/d6d/d108/f11b x:0 0 0 2024-04-12T05:09:10.795 INFO:tasks.workunit.client.1.smithi192.stdout:9/620: mkdir d11/d25/d36/d6e/d33/ddd 0 2024-04-12T05:09:10.801 INFO:tasks.workunit.client.1.smithi192.stdout:0/845: dread d0/d7b/d9b/d35/d109/d53/df3/f48 [0,4194304] 0 2024-04-12T05:09:10.805 INFO:tasks.workunit.client.1.smithi192.stdout:7/654: link d3/d7/d14/f47 d3/d1d/fe1 0 2024-04-12T05:09:10.807 INFO:tasks.workunit.client.0.smithi113.stdout:4/804: dwrite da/dd/d3a/f5b [0,4194304] 0 2024-04-12T05:09:10.808 INFO:tasks.workunit.client.1.smithi192.stdout:6/758: dread d3/d23/db2/fde [0,4194304] 0 2024-04-12T05:09:10.810 INFO:tasks.workunit.client.0.smithi113.stdout:4/805: rmdir da/d70 39 2024-04-12T05:09:10.812 INFO:tasks.workunit.client.1.smithi192.stdout:6/759: truncate d3/d23/f78 271142 0 2024-04-12T05:09:10.813 INFO:tasks.workunit.client.0.smithi113.stdout:4/806: rmdir da/d3c/d9a/dc1/dd9 39 2024-04-12T05:09:10.814 INFO:tasks.workunit.client.0.smithi113.stdout:1/897: truncate d10/d2e/d31/dc3/d109/fe7 3193745 0 2024-04-12T05:09:10.816 INFO:tasks.workunit.client.0.smithi113.stdout:3/756: sync 2024-04-12T05:09:10.816 INFO:tasks.workunit.client.0.smithi113.stdout:4/807: mkdir da/d24/d60/daa/d112 0 2024-04-12T05:09:10.818 INFO:tasks.workunit.client.0.smithi113.stdout:4/808: truncate da/f103 988957 0 2024-04-12T05:09:10.819 INFO:tasks.workunit.client.1.smithi192.stdout:7/655: dwrite d3/d1d/d4a/d5e/d6d/da3/fc1 [0,4194304] 0 2024-04-12T05:09:10.819 INFO:tasks.workunit.client.0.smithi113.stdout:1/898: rename d10/d2e/d31/dc3/f119 to d10/d41/d5b/f11f 0 2024-04-12T05:09:10.821 INFO:tasks.workunit.client.0.smithi113.stdout:4/809: mkdir da/d3c/d83/d113 0 2024-04-12T05:09:10.824 INFO:tasks.workunit.client.0.smithi113.stdout:3/757: rename d0/d13/d15/d28/d2a/d8f/ddd/fe4 to d0/d13/d15/d28/d2a/d34/d5f/f100 0 2024-04-12T05:09:10.825 INFO:tasks.workunit.client.0.smithi113.stdout:3/758: chown d0/d7e/da3/fb9 31989697 1 2024-04-12T05:09:10.825 INFO:tasks.workunit.client.0.smithi113.stdout:4/810: mknod da/d24/d60/daa/c114 0 2024-04-12T05:09:10.827 INFO:tasks.workunit.client.1.smithi192.stdout:9/621: write d11/d25/d36/d6e/fa8 [441671,24400] 0 2024-04-12T05:09:10.829 INFO:tasks.workunit.client.0.smithi113.stdout:3/759: write d0/d13/d15/d28/d2a/d34/d5f/f100 [823770,17795] 0 2024-04-12T05:09:10.832 INFO:tasks.workunit.client.1.smithi192.stdout:6/760: link d3/d8/d10/d4c/d68/f77 d3/d8/d56/d59/d97/dbf/de8/ffa 0 2024-04-12T05:09:10.832 INFO:tasks.workunit.client.0.smithi113.stdout:3/760: read d0/d7e/f85 [626872,76743] 0 2024-04-12T05:09:10.833 INFO:tasks.workunit.client.0.smithi113.stdout:3/761: chown d0/d13/d15/d28/f74 652002 1 2024-04-12T05:09:10.833 INFO:tasks.workunit.client.1.smithi192.stdout:9/622: creat d11/d15/d21/d2c/d1f/d6a/d7e/d89/dce/fde x:0 0 0 2024-04-12T05:09:10.834 INFO:tasks.workunit.client.0.smithi113.stdout:3/762: chown d0/d13/d47/d4f/dd2/ld7 164264240 1 2024-04-12T05:09:10.835 INFO:tasks.workunit.client.1.smithi192.stdout:7/656: getdents d3/d1d/d4a/d5e/d6d/da3 0 2024-04-12T05:09:10.835 INFO:tasks.workunit.client.0.smithi113.stdout:3/763: chown d0/d80/lcb 0 1 2024-04-12T05:09:10.838 INFO:tasks.workunit.client.0.smithi113.stdout:3/764: creat d0/d13/d15/d28/d2a/d34/f101 x:0 0 0 2024-04-12T05:09:10.839 INFO:tasks.workunit.client.1.smithi192.stdout:6/761: link d3/f1e d3/d8/de6/ffb 0 2024-04-12T05:09:10.841 INFO:tasks.workunit.client.1.smithi192.stdout:0/846: truncate d0/d7b/d9b/d35/d109/d53/df3/d2e/d66/f70 3035658 0 2024-04-12T05:09:10.842 INFO:tasks.workunit.client.1.smithi192.stdout:6/762: read d3/d8/f92 [2071204,33788] 0 2024-04-12T05:09:10.845 INFO:tasks.workunit.client.0.smithi113.stdout:4/811: write da/d3c/d9a/ff8 [779871,92407] 0 2024-04-12T05:09:10.846 INFO:tasks.workunit.client.0.smithi113.stdout:4/812: read - da/d3c/d9f/fdd zero size 2024-04-12T05:09:10.848 INFO:tasks.workunit.client.0.smithi113.stdout:4/813: fsync da/d24/d7b/f10a 0 2024-04-12T05:09:10.849 INFO:tasks.workunit.client.1.smithi192.stdout:0/847: symlink d0/d7b/d9b/d35/d109/d53/df3/d2e/d59/l11c 0 2024-04-12T05:09:10.851 INFO:tasks.workunit.client.0.smithi113.stdout:4/814: mkdir da/d24/d115 0 2024-04-12T05:09:10.853 INFO:tasks.workunit.client.0.smithi113.stdout:1/899: dwrite d10/d2e/dd8/f69 [0,4194304] 0 2024-04-12T05:09:10.854 INFO:tasks.workunit.client.1.smithi192.stdout:7/657: write d3/d7/d15/d22/f4d [141660,88010] 0 2024-04-12T05:09:10.854 INFO:tasks.workunit.client.0.smithi113.stdout:4/815: chown da/d73/f1c 143974197 1 2024-04-12T05:09:10.855 INFO:tasks.workunit.client.1.smithi192.stdout:9/623: dread d11/d25/d36/f5f [0,4194304] 0 2024-04-12T05:09:10.857 INFO:tasks.workunit.client.0.smithi113.stdout:3/765: dwrite d0/d13/d15/d28/d2a/d34/f43 [0,4194304] 0 2024-04-12T05:09:10.858 INFO:tasks.workunit.client.0.smithi113.stdout:4/816: creat da/d24/d7b/d110/f116 x:0 0 0 2024-04-12T05:09:10.860 INFO:tasks.workunit.client.0.smithi113.stdout:3/766: mknod d0/d80/c102 0 2024-04-12T05:09:10.864 INFO:tasks.workunit.client.1.smithi192.stdout:9/624: mkdir d11/d25/d36/d6e/d33/da2/ddf 0 2024-04-12T05:09:10.866 INFO:tasks.workunit.client.0.smithi113.stdout:4/817: unlink da/d24/d7b/lc0 0 2024-04-12T05:09:10.866 INFO:tasks.workunit.client.0.smithi113.stdout:3/767: unlink d0/d13/d15/d28/d2a/f8e 0 2024-04-12T05:09:10.870 INFO:tasks.workunit.client.1.smithi192.stdout:7/658: link d3/d1d/d48/d7b/c87 d3/d7/d14/d1f/d76/ce2 0 2024-04-12T05:09:10.871 INFO:tasks.workunit.client.1.smithi192.stdout:6/763: dwrite d3/d8/d10/d4c/f62 [0,4194304] 0 2024-04-12T05:09:10.871 INFO:tasks.workunit.client.1.smithi192.stdout:9/625: mknod d11/d25/dd3/dd5/ce0 0 2024-04-12T05:09:10.874 INFO:tasks.workunit.client.1.smithi192.stdout:7/659: creat d3/d1d/d74/d86/fe3 x:0 0 0 2024-04-12T05:09:10.878 INFO:tasks.workunit.client.1.smithi192.stdout:6/764: rename d3/d8/d22/cb6 to d3/d8/d56/d59/ddf/cfc 0 2024-04-12T05:09:10.880 INFO:tasks.workunit.client.1.smithi192.stdout:0/848: dwrite d0/d7b/d9b/d35/d109/d53/fc3 [0,4194304] 0 2024-04-12T05:09:10.882 INFO:tasks.workunit.client.1.smithi192.stdout:6/765: mknod d3/dc/d39/cfd 0 2024-04-12T05:09:10.887 INFO:tasks.workunit.client.1.smithi192.stdout:6/766: symlink d3/d8/d56/d59/d97/dbf/dd2/lfe 0 2024-04-12T05:09:10.888 INFO:tasks.workunit.client.0.smithi113.stdout:3/768: write d0/d13/fa9 [3558589,10589] 0 2024-04-12T05:09:10.890 INFO:tasks.workunit.client.1.smithi192.stdout:6/767: mknod d3/d5/d9c/dbd/cff 0 2024-04-12T05:09:10.891 INFO:tasks.workunit.client.1.smithi192.stdout:0/849: dwrite d0/fd0 [0,4194304] 0 2024-04-12T05:09:10.904 INFO:tasks.workunit.client.0.smithi113.stdout:1/900: truncate d10/d41/d89/d96/dd3/f100 3964632 0 2024-04-12T05:09:10.907 INFO:tasks.workunit.client.1.smithi192.stdout:9/626: dread f10 [0,4194304] 0 2024-04-12T05:09:10.908 INFO:tasks.workunit.client.1.smithi192.stdout:9/627: fdatasync d11/d15/d21/d2c/d1f/f67 0 2024-04-12T05:09:10.910 INFO:tasks.workunit.client.1.smithi192.stdout:9/628: creat d11/d15/d21/fe1 x:0 0 0 2024-04-12T05:09:10.912 INFO:tasks.workunit.client.1.smithi192.stdout:9/629: mknod d11/d25/d36/d6e/dc7/ce2 0 2024-04-12T05:09:10.915 INFO:tasks.workunit.client.1.smithi192.stdout:9/630: getdents d11/d15/d21/d2c/d1f/d6a/d7e/d89/dce 0 2024-04-12T05:09:10.920 INFO:tasks.workunit.client.1.smithi192.stdout:9/631: dwrite d11/d15/d21/d2c/d1f/d94/fa0 [0,4194304] 0 2024-04-12T05:09:10.926 INFO:tasks.workunit.client.1.smithi192.stdout:1/758: sync 2024-04-12T05:09:10.927 INFO:tasks.workunit.client.1.smithi192.stdout:3/681: sync 2024-04-12T05:09:10.927 INFO:tasks.workunit.client.1.smithi192.stdout:8/723: sync 2024-04-12T05:09:10.929 INFO:tasks.workunit.client.1.smithi192.stdout:3/682: mkdir d5/df1 0 2024-04-12T05:09:10.930 INFO:tasks.workunit.client.1.smithi192.stdout:8/724: symlink d6/d10/d46/d7c/ldc 0 2024-04-12T05:09:10.931 INFO:tasks.workunit.client.0.smithi113.stdout:3/769: write d0/d13/f7d [2302661,19973] 0 2024-04-12T05:09:10.933 INFO:tasks.workunit.client.1.smithi192.stdout:1/759: rename d0/d29/d8d/d96/fcd to d0/d2/d9f/ffd 0 2024-04-12T05:09:10.936 INFO:tasks.workunit.client.1.smithi192.stdout:8/725: symlink d6/d10/d46/d5e/d5f/d37/d4f/d51/ldd 0 2024-04-12T05:09:10.936 INFO:tasks.workunit.client.0.smithi113.stdout:3/770: read d0/d13/f42 [5309986,15821] 0 2024-04-12T05:09:10.939 INFO:tasks.workunit.client.1.smithi192.stdout:0/850: truncate d0/d7b/d9b/d35/d109/d53/d6d/f62 60163 0 2024-04-12T05:09:10.939 INFO:tasks.workunit.client.1.smithi192.stdout:8/726: stat d6/d10/d46/d5e 0 2024-04-12T05:09:10.941 INFO:tasks.workunit.client.1.smithi192.stdout:3/683: dwrite d5/d13/d1a/d5f/fa1 [0,4194304] 0 2024-04-12T05:09:10.943 INFO:tasks.workunit.client.1.smithi192.stdout:7/660: dwrite d3/f9e [0,4194304] 0 2024-04-12T05:09:10.945 INFO:tasks.workunit.client.1.smithi192.stdout:9/632: write d11/d25/d36/d6e/d33/f6c [786247,27358] 0 2024-04-12T05:09:10.946 INFO:tasks.workunit.client.0.smithi113.stdout:3/771: dread d0/f18 [0,4194304] 0 2024-04-12T05:09:10.947 INFO:tasks.workunit.client.0.smithi113.stdout:3/772: write d0/d13/d15/d28/fa5 [473547,45147] 0 2024-04-12T05:09:10.949 INFO:tasks.workunit.client.0.smithi113.stdout:1/901: write d10/d41/d89/d96/dd3/f100 [3710578,85618] 0 2024-04-12T05:09:10.951 INFO:tasks.workunit.client.0.smithi113.stdout:3/773: unlink d0/dc6/ff5 0 2024-04-12T05:09:10.953 INFO:tasks.workunit.client.1.smithi192.stdout:1/760: dwrite d0/d2/d1e/d3f/d56/f93 [0,4194304] 0 2024-04-12T05:09:10.954 INFO:tasks.workunit.client.1.smithi192.stdout:3/684: read - d5/d13/d3a/d70/d8c/de1/fe0 zero size 2024-04-12T05:09:10.955 INFO:tasks.workunit.client.0.smithi113.stdout:3/774: symlink d0/d13/d15/d28/d2a/d8f/l103 0 2024-04-12T05:09:10.956 INFO:tasks.workunit.client.1.smithi192.stdout:0/851: truncate d0/d7b/d9b/d35/d109/d53/d6d/d108/f11b 43555 0 2024-04-12T05:09:10.956 INFO:tasks.workunit.client.1.smithi192.stdout:9/633: rename d11/d25/d36/d6e/dc7/fd6 to d11/d25/d36/d6e/d33/dbb/fe3 0 2024-04-12T05:09:10.957 INFO:tasks.workunit.client.1.smithi192.stdout:8/727: write d6/d10/f11 [3151631,129903] 0 2024-04-12T05:09:10.957 INFO:tasks.workunit.client.1.smithi192.stdout:1/761: write d0/d2/d1e/d3f/df2/d21/d33/f6b [3255741,128238] 0 2024-04-12T05:09:10.958 INFO:tasks.workunit.client.1.smithi192.stdout:3/685: creat d5/d13/d3a/d8b/da2/da7/ff2 x:0 0 0 2024-04-12T05:09:10.959 INFO:tasks.workunit.client.1.smithi192.stdout:0/852: readlink d0/d7b/d9b/d35/d109/d9f/d10a/d67/l8f 0 2024-04-12T05:09:10.960 INFO:tasks.workunit.client.0.smithi113.stdout:1/902: dwrite d10/d2e/d7c/dc5/ddb/ff8 [4194304,4194304] 0 2024-04-12T05:09:10.963 INFO:tasks.workunit.client.1.smithi192.stdout:3/686: mknod d5/d13/d3c/d84/cf3 0 2024-04-12T05:09:10.968 INFO:tasks.workunit.client.1.smithi192.stdout:7/661: truncate f1 387345 0 2024-04-12T05:09:10.969 INFO:tasks.workunit.client.0.smithi113.stdout:1/903: fsync d10/d41/d89/d96/d7d/f79 0 2024-04-12T05:09:10.970 INFO:tasks.workunit.client.1.smithi192.stdout:3/687: creat d5/d13/d3c/d26/ff4 x:0 0 0 2024-04-12T05:09:10.972 INFO:tasks.workunit.client.0.smithi113.stdout:1/904: chown d10/d41/d5b/l6b 784715209 1 2024-04-12T05:09:10.973 INFO:tasks.workunit.client.1.smithi192.stdout:0/853: unlink d0/d7b/d9b/d35/d109/d53/df3/d2e/d66/f70 0 2024-04-12T05:09:10.975 INFO:tasks.workunit.client.1.smithi192.stdout:7/662: truncate d3/d1d/d4a/f70 1210754 0 2024-04-12T05:09:10.975 INFO:tasks.workunit.client.1.smithi192.stdout:0/854: chown d0/d7b/d9b/d35/d109/d95/db4/ldc 1 1 2024-04-12T05:09:10.975 INFO:tasks.workunit.client.0.smithi113.stdout:1/905: read - d10/d55/fee zero size 2024-04-12T05:09:10.976 INFO:tasks.workunit.client.0.smithi113.stdout:3/775: rename d0/d80/ca4 to d0/d13/d15/d28/d2a/d8f/c104 0 2024-04-12T05:09:10.978 INFO:tasks.workunit.client.0.smithi113.stdout:3/776: symlink d0/d13/d47/d94/l105 0 2024-04-12T05:09:10.979 INFO:tasks.workunit.client.0.smithi113.stdout:3/777: fdatasync d0/d13/d47/d94/dcf/dd1/de2/fe6 0 2024-04-12T05:09:10.982 INFO:tasks.workunit.client.1.smithi192.stdout:7/663: rename d3/d1d/d5a/d5f/l3a to d3/d7/d14/d1f/le4 0 2024-04-12T05:09:10.983 INFO:tasks.workunit.client.1.smithi192.stdout:7/664: readlink d3/l44 0 2024-04-12T05:09:10.983 INFO:tasks.workunit.client.1.smithi192.stdout:0/855: link d0/d7b/d9b/d35/d109/db0/c110 d0/d7b/d9b/d35/d109/d53/df3/d2e/c11d 0 2024-04-12T05:09:10.985 INFO:tasks.workunit.client.1.smithi192.stdout:7/665: creat d3/d7/d15/d1a/ddc/fe5 x:0 0 0 2024-04-12T05:09:10.987 INFO:tasks.workunit.client.0.smithi113.stdout:3/778: read d0/f1c [3172799,33726] 0 2024-04-12T05:09:10.989 INFO:tasks.workunit.client.1.smithi192.stdout:0/856: rename d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d3a/d4f to d0/d7b/d9b/d35/d109/d53/df3/d29/db5/d11e 0 2024-04-12T05:09:10.989 INFO:tasks.workunit.client.1.smithi192.stdout:9/634: dread d11/d25/d36/d6e/d33/f55 [0,4194304] 0 2024-04-12T05:09:10.990 INFO:tasks.workunit.client.0.smithi113.stdout:3/779: fsync d0/d13/d15/d28/d2a/d34/d5f/fe1 0 2024-04-12T05:09:10.992 INFO:tasks.workunit.client.1.smithi192.stdout:9/635: fdatasync d11/d25/d36/f5f 0 2024-04-12T05:09:10.993 INFO:tasks.workunit.client.1.smithi192.stdout:9/636: rmdir d11/d25/d60 39 2024-04-12T05:09:10.995 INFO:tasks.workunit.client.1.smithi192.stdout:9/637: fdatasync d11/d25/d36/d6e/d8f/fbd 0 2024-04-12T05:09:10.997 INFO:tasks.workunit.client.1.smithi192.stdout:9/638: creat d11/d25/dd3/fe4 x:0 0 0 2024-04-12T05:09:11.000 INFO:tasks.workunit.client.1.smithi192.stdout:9/639: creat d11/d15/d21/d2c/d1f/d6a/d7e/fe5 x:0 0 0 2024-04-12T05:09:11.002 INFO:tasks.workunit.client.1.smithi192.stdout:0/857: chown d0/d7b/d9b/d35/d109/d53/d6d/f62 64193 1 2024-04-12T05:09:11.003 INFO:tasks.workunit.client.1.smithi192.stdout:9/640: creat d11/d25/d36/d6e/d33/da2/fe6 x:0 0 0 2024-04-12T05:09:11.006 INFO:tasks.workunit.client.1.smithi192.stdout:0/858: mkdir d0/d7b/d9b/d35/d11f 0 2024-04-12T05:09:11.007 INFO:tasks.workunit.client.1.smithi192.stdout:3/688: dread d5/d13/d1a/d53/f58 [0,4194304] 0 2024-04-12T05:09:11.009 INFO:tasks.workunit.client.1.smithi192.stdout:7/666: dread d3/d11/f3b [0,4194304] 0 2024-04-12T05:09:11.010 INFO:tasks.workunit.client.1.smithi192.stdout:3/689: symlink d5/d13/d1a/d5f/lf5 0 2024-04-12T05:09:11.011 INFO:tasks.workunit.client.0.smithi113.stdout:3/780: dwrite d0/f18 [0,4194304] 0 2024-04-12T05:09:11.013 INFO:tasks.workunit.client.1.smithi192.stdout:0/859: symlink d0/d7b/d9b/l120 0 2024-04-12T05:09:11.015 INFO:tasks.workunit.client.0.smithi113.stdout:2/791: sync 2024-04-12T05:09:11.015 INFO:tasks.workunit.client.0.smithi113.stdout:6/783: sync 2024-04-12T05:09:11.015 INFO:tasks.workunit.client.1.smithi192.stdout:7/667: dread d3/f4f [0,4194304] 0 2024-04-12T05:09:11.019 INFO:tasks.workunit.client.1.smithi192.stdout:0/860: readlink d0/d7b/d9b/d35/d109/d9f/d10a/d67/le1 0 2024-04-12T05:09:11.021 INFO:tasks.workunit.client.0.smithi113.stdout:3/781: dwrite d0/d13/f70 [0,4194304] 0 2024-04-12T05:09:11.022 INFO:tasks.workunit.client.1.smithi192.stdout:3/690: getdents d5/d13/d3c/d84 0 2024-04-12T05:09:11.023 INFO:tasks.workunit.client.0.smithi113.stdout:6/784: mkdir d8/de/d13/d11/d36/d3d/d4c/d120 0 2024-04-12T05:09:11.029 INFO:tasks.workunit.client.1.smithi192.stdout:3/691: creat d5/d13/d3a/d8b/ff6 x:0 0 0 2024-04-12T05:09:11.030 INFO:tasks.workunit.client.0.smithi113.stdout:6/785: creat d8/de/d13/d11/d102/f121 x:0 0 0 2024-04-12T05:09:11.030 INFO:tasks.workunit.client.1.smithi192.stdout:3/692: symlink d5/d13/d94/dbd/lf7 0 2024-04-12T05:09:11.034 INFO:tasks.workunit.client.0.smithi113.stdout:6/786: read d8/de/d13/df0/d66/f7e [190307,17413] 0 2024-04-12T05:09:11.037 INFO:tasks.workunit.client.1.smithi192.stdout:9/641: dread d11/d15/d21/d2c/d1f/d50/f98 [0,4194304] 0 2024-04-12T05:09:11.040 INFO:tasks.workunit.client.1.smithi192.stdout:0/861: truncate d0/d7b/d9b/d35/d109/d9f/d10a/d67/f92 3280195 0 2024-04-12T05:09:11.041 INFO:tasks.workunit.client.1.smithi192.stdout:9/642: dread d11/d15/d21/d2c/d1f/d94/fa0 [0,4194304] 0 2024-04-12T05:09:11.044 INFO:tasks.workunit.client.1.smithi192.stdout:9/643: mkdir d11/d25/d36/d6e/d33/dbb/de7 0 2024-04-12T05:09:11.046 INFO:tasks.workunit.client.0.smithi113.stdout:2/792: truncate d8/d9/d27/d88/f8b 1204449 0 2024-04-12T05:09:11.050 INFO:tasks.workunit.client.1.smithi192.stdout:0/862: link d0/d7b/d9b/d35/d109/d53/df3/d2e/d66/fd7 d0/d7b/d9b/d35/d109/d95/db4/db8/f121 0 2024-04-12T05:09:11.051 INFO:tasks.workunit.client.1.smithi192.stdout:7/668: dread d3/d11/f16 [0,4194304] 0 2024-04-12T05:09:11.052 INFO:tasks.workunit.client.0.smithi113.stdout:3/782: dwrite d0/d13/d15/faf [0,4194304] 0 2024-04-12T05:09:11.052 INFO:tasks.workunit.client.1.smithi192.stdout:9/644: link d11/d15/c85 d11/d25/d36/d6e/d33/dbb/de7/ce8 0 2024-04-12T05:09:11.053 INFO:tasks.workunit.client.0.smithi113.stdout:2/793: rename d8/d9/de/d12/f78 to d8/d9/de/d43/dee/ff8 0 2024-04-12T05:09:11.053 INFO:tasks.workunit.client.0.smithi113.stdout:3/783: chown d0/d13/d15/d28/d2a/d34/d5f/f96 1188090191 1 2024-04-12T05:09:11.055 INFO:tasks.workunit.client.0.smithi113.stdout:2/794: write d8/fc0 [65931,62021] 0 2024-04-12T05:09:11.055 INFO:tasks.workunit.client.1.smithi192.stdout:7/669: truncate d3/d11/f46 2830173 0 2024-04-12T05:09:11.055 INFO:tasks.workunit.client.0.smithi113.stdout:2/795: stat d8/d16/dbd 0 2024-04-12T05:09:11.060 INFO:tasks.workunit.client.0.smithi113.stdout:3/784: dread d0/d13/d15/d28/d2a/d34/d5f/f100 [0,4194304] 0 2024-04-12T05:09:11.063 INFO:tasks.workunit.client.1.smithi192.stdout:0/863: creat d0/d7b/f122 x:0 0 0 2024-04-12T05:09:11.063 INFO:tasks.workunit.client.1.smithi192.stdout:3/693: dread d5/f1c [0,4194304] 0 2024-04-12T05:09:11.063 INFO:tasks.workunit.client.1.smithi192.stdout:7/670: rename d3/d1d/d4a/l78 to d3/d1d/d4a/d5e/d6d/da3/le6 0 2024-04-12T05:09:11.063 INFO:tasks.workunit.client.1.smithi192.stdout:9/645: chown d11/d25/d36/d3d/cc0 242030161 1 2024-04-12T05:09:11.063 INFO:tasks.workunit.client.1.smithi192.stdout:0/864: chown d0/d7b/d9b/c3b 1 1 2024-04-12T05:09:11.063 INFO:tasks.workunit.client.1.smithi192.stdout:3/694: mkdir d5/d13/d3a/d4e/df8 0 2024-04-12T05:09:11.063 INFO:tasks.workunit.client.1.smithi192.stdout:7/671: write d3/d7/d14/d1f/f34 [5630761,94138] 0 2024-04-12T05:09:11.064 INFO:tasks.workunit.client.0.smithi113.stdout:2/796: dread d8/d9/de/d12/f57 [0,4194304] 0 2024-04-12T05:09:11.065 INFO:tasks.workunit.client.0.smithi113.stdout:3/785: mkdir d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/d106 0 2024-04-12T05:09:11.066 INFO:tasks.workunit.client.1.smithi192.stdout:9/646: fdatasync d11/d25/d60/f97 0 2024-04-12T05:09:11.067 INFO:tasks.workunit.client.1.smithi192.stdout:0/865: mknod d0/d7b/d9b/d35/d109/d53/df3/d29/c123 0 2024-04-12T05:09:11.068 INFO:tasks.workunit.client.1.smithi192.stdout:3/695: chown d5/d13/d3a/d70/d8c/d8d/f93 248 1 2024-04-12T05:09:11.069 INFO:tasks.workunit.client.0.smithi113.stdout:6/787: dread d8/de/d13/d11/d36/d89/fe1 [0,4194304] 0 2024-04-12T05:09:11.069 INFO:tasks.workunit.client.0.smithi113.stdout:3/786: chown d0/d13/d15/d28/d2a/d34/f72 129879 1 2024-04-12T05:09:11.070 INFO:tasks.workunit.client.0.smithi113.stdout:6/788: chown d8/de 399367804 1 2024-04-12T05:09:11.070 INFO:tasks.workunit.client.1.smithi192.stdout:2/749: sync 2024-04-12T05:09:11.070 INFO:tasks.workunit.client.1.smithi192.stdout:5/806: sync 2024-04-12T05:09:11.070 INFO:tasks.workunit.client.1.smithi192.stdout:9/647: symlink d11/d15/d21/d2c/d1f/d6a/d7e/d89/le9 0 2024-04-12T05:09:11.071 INFO:tasks.workunit.client.0.smithi113.stdout:2/797: dwrite d8/d9/d27/dda/f51 [0,4194304] 0 2024-04-12T05:09:11.071 INFO:tasks.workunit.client.1.smithi192.stdout:0/866: truncate d0/d7b/d9b/d35/d109/db0/dbf/fc7 1024169 0 2024-04-12T05:09:11.071 INFO:tasks.workunit.client.1.smithi192.stdout:3/696: truncate d5/d13/d3c/d26/ff4 803469 0 2024-04-12T05:09:11.072 INFO:tasks.workunit.client.0.smithi113.stdout:2/798: stat d8/d9/de/d15/d24/d58/fc3 0 2024-04-12T05:09:11.073 INFO:tasks.workunit.client.0.smithi113.stdout:3/787: creat d0/d13/d47/d94/dcf/dd1/de2/f107 x:0 0 0 2024-04-12T05:09:11.075 INFO:tasks.workunit.client.0.smithi113.stdout:6/789: unlink d8/de/d13/df0/d66/f7e 0 2024-04-12T05:09:11.076 INFO:tasks.workunit.client.0.smithi113.stdout:6/790: dread - d8/de/d13/d11/d36/d53/db7/fc9 zero size 2024-04-12T05:09:11.077 INFO:tasks.workunit.client.1.smithi192.stdout:0/867: read d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d46/f49 [7683200,112524] 0 2024-04-12T05:09:11.078 INFO:tasks.workunit.client.0.smithi113.stdout:2/799: creat d8/d9/de/d12/d2b/dd2/df6/ff9 x:0 0 0 2024-04-12T05:09:11.079 INFO:tasks.workunit.client.1.smithi192.stdout:5/807: dread d0/d15/d3e/dca/dda/f103 [0,4194304] 0 2024-04-12T05:09:11.080 INFO:tasks.workunit.client.1.smithi192.stdout:9/648: mknod d11/d15/d21/d2c/d1f/d6a/d7e/d89/dce/cea 0 2024-04-12T05:09:11.083 INFO:tasks.workunit.client.1.smithi192.stdout:6/768: sync 2024-04-12T05:09:11.083 INFO:tasks.workunit.client.1.smithi192.stdout:4/813: sync 2024-04-12T05:09:11.084 INFO:tasks.workunit.client.1.smithi192.stdout:0/868: chown d0/d7b/d9b/d35/d109/d9f/d10a/d67/db1/dd6/df2 25973 1 2024-04-12T05:09:11.086 INFO:tasks.workunit.client.1.smithi192.stdout:5/808: mkdir d0/d15/d3e/dca/dda/db5/dd7/d116 0 2024-04-12T05:09:11.086 INFO:tasks.workunit.client.1.smithi192.stdout:6/769: write d3/d8/d56/d59/d8a/d99/f9f [3884276,101259] 0 2024-04-12T05:09:11.087 INFO:tasks.workunit.client.1.smithi192.stdout:1/762: sync 2024-04-12T05:09:11.087 INFO:tasks.workunit.client.1.smithi192.stdout:8/728: sync 2024-04-12T05:09:11.089 INFO:tasks.workunit.client.1.smithi192.stdout:5/809: readlink d0/d15/d32/l5d 0 2024-04-12T05:09:11.089 INFO:tasks.workunit.client.1.smithi192.stdout:7/672: dread d3/d1d/d48/d7b/f95 [0,4194304] 0 2024-04-12T05:09:11.091 INFO:tasks.workunit.client.0.smithi113.stdout:2/800: dread d8/d9/de/d15/d24/f3b [0,4194304] 0 2024-04-12T05:09:11.093 INFO:tasks.workunit.client.0.smithi113.stdout:2/801: write d8/d9/d27/d88/fab [830266,36422] 0 2024-04-12T05:09:11.094 INFO:tasks.workunit.client.1.smithi192.stdout:7/673: write d3/d1d/f9f [1884978,99053] 0 2024-04-12T05:09:11.096 INFO:tasks.workunit.client.0.smithi113.stdout:2/802: dread - d8/d9/de/d15/d24/f7e zero size 2024-04-12T05:09:11.097 INFO:tasks.workunit.client.1.smithi192.stdout:5/810: rmdir d0/d15/d32/d69/dc8 39 2024-04-12T05:09:11.100 INFO:tasks.workunit.client.0.smithi113.stdout:2/803: fdatasync d8/d9/de/d15/d24/f4b 0 2024-04-12T05:09:11.104 INFO:tasks.workunit.client.1.smithi192.stdout:4/814: getdents d1/d54/d82/d71/d9e/dc4/dd9 0 2024-04-12T05:09:11.104 INFO:tasks.workunit.client.1.smithi192.stdout:0/869: dwrite d0/d7b/d9b/d35/d109/d9f/d10a/d67/f106 [0,4194304] 0 2024-04-12T05:09:11.108 INFO:tasks.workunit.client.0.smithi113.stdout:2/804: creat d8/d9/d27/dda/ffa x:0 0 0 2024-04-12T05:09:11.109 INFO:tasks.workunit.client.1.smithi192.stdout:2/750: dwrite d1/d8/d3a/f3b [0,4194304] 0 2024-04-12T05:09:11.111 INFO:tasks.workunit.client.0.smithi113.stdout:2/805: symlink d8/d9/de/d12/d2b/daa/lfb 0 2024-04-12T05:09:11.117 INFO:tasks.workunit.client.1.smithi192.stdout:1/763: write d0/d2/d1e/d3f/df2/d21/f99 [502000,100966] 0 2024-04-12T05:09:11.119 INFO:tasks.workunit.client.0.smithi113.stdout:2/806: creat d8/d9/de/d43/dee/ffc x:0 0 0 2024-04-12T05:09:11.120 INFO:tasks.workunit.client.0.smithi113.stdout:6/791: dread d8/d45/f49 [0,4194304] 0 2024-04-12T05:09:11.120 INFO:tasks.workunit.client.0.smithi113.stdout:3/788: truncate d0/d13/d15/d28/d2a/f98 1344419 0 2024-04-12T05:09:11.122 INFO:tasks.workunit.client.0.smithi113.stdout:6/792: rename d8/de/d13/d11/d36/d3d/d4c/d120 to d8/de/d13/d11/d36/d3d/d4c/d120/d122 22 2024-04-12T05:09:11.128 INFO:tasks.workunit.client.0.smithi113.stdout:6/793: rename d8/la6 to d8/de/dd5/l123 0 2024-04-12T05:09:11.129 INFO:tasks.workunit.client.1.smithi192.stdout:3/697: dwrite d5/d13/d3a/d70/d8c/fb4 [0,4194304] 0 2024-04-12T05:09:11.130 INFO:tasks.workunit.client.0.smithi113.stdout:3/789: symlink d0/d13/d15/d28/d2a/l108 0 2024-04-12T05:09:11.130 INFO:tasks.workunit.client.1.smithi192.stdout:6/770: dread d3/dc/d39/f87 [0,4194304] 0 2024-04-12T05:09:11.131 INFO:tasks.workunit.client.1.smithi192.stdout:0/870: dread d0/d7b/d9b/d35/d109/d53/df3/f119 [0,4194304] 0 2024-04-12T05:09:11.132 INFO:tasks.workunit.client.0.smithi113.stdout:2/807: creat d8/d9/d27/ffd x:0 0 0 2024-04-12T05:09:11.132 INFO:tasks.workunit.client.1.smithi192.stdout:8/729: dread d6/d10/d46/d5e/d5f/d31/d80/fcd [0,4194304] 0 2024-04-12T05:09:11.132 INFO:tasks.workunit.client.1.smithi192.stdout:5/811: dwrite d0/d15/d24/d47/f4d [0,4194304] 0 2024-04-12T05:09:11.133 INFO:tasks.workunit.client.1.smithi192.stdout:9/649: dread d11/d25/d36/d6e/f77 [0,4194304] 0 2024-04-12T05:09:11.133 INFO:tasks.workunit.client.0.smithi113.stdout:7/793: sync 2024-04-12T05:09:11.133 INFO:tasks.workunit.client.0.smithi113.stdout:0/909: sync 2024-04-12T05:09:11.134 INFO:tasks.workunit.client.0.smithi113.stdout:5/940: sync 2024-04-12T05:09:11.134 INFO:tasks.workunit.client.0.smithi113.stdout:6/794: mkdir d8/d45/d124 0 2024-04-12T05:09:11.135 INFO:tasks.workunit.client.0.smithi113.stdout:6/795: fsync d8/de/dd5/f119 0 2024-04-12T05:09:11.137 INFO:tasks.workunit.client.1.smithi192.stdout:5/812: chown d0/d15/d32/d69/cd5 1665978 1 2024-04-12T05:09:11.138 INFO:tasks.workunit.client.1.smithi192.stdout:3/698: mknod d5/d13/d3c/d84/cf9 0 2024-04-12T05:09:11.138 INFO:tasks.workunit.client.0.smithi113.stdout:7/794: mknod d0/dc/d6/d78/c10d 0 2024-04-12T05:09:11.140 INFO:tasks.workunit.client.0.smithi113.stdout:6/796: write d8/de/d13/d11/d36/d3d/d4c/d31/f10b [1824723,38485] 0 2024-04-12T05:09:11.141 INFO:tasks.workunit.client.0.smithi113.stdout:6/797: readlink d8/de/dd5/lfa 0 2024-04-12T05:09:11.145 INFO:tasks.workunit.client.0.smithi113.stdout:7/795: rename d0/dc/d6/d9b/fef to d0/dd/d95/f10e 0 2024-04-12T05:09:11.146 INFO:tasks.workunit.client.0.smithi113.stdout:6/798: symlink d8/de/d13/d11/d36/d3d/d4c/d26/d5b/l125 0 2024-04-12T05:09:11.147 INFO:tasks.workunit.client.1.smithi192.stdout:9/650: chown d11/d15/f45 2678 1 2024-04-12T05:09:11.148 INFO:tasks.workunit.client.1.smithi192.stdout:3/699: creat d5/d13/d94/dbd/ffa x:0 0 0 2024-04-12T05:09:11.150 INFO:tasks.workunit.client.1.smithi192.stdout:8/730: mkdir d6/d10/d6c/d9e/dde 0 2024-04-12T05:09:11.150 INFO:tasks.workunit.client.1.smithi192.stdout:1/764: rename d0/d2/d1e/d3f/df2/d21/d33/dd8/fdf to d0/d2/d1e/d3f/df2/d3c/d5b/ffe 0 2024-04-12T05:09:11.154 INFO:tasks.workunit.client.1.smithi192.stdout:6/771: dwrite d3/d8/f1f [0,4194304] 0 2024-04-12T05:09:11.155 INFO:tasks.workunit.client.1.smithi192.stdout:1/765: chown d0/d1b/c2c 0 1 2024-04-12T05:09:11.155 INFO:tasks.workunit.client.1.smithi192.stdout:9/651: symlink d11/d25/d60/d8c/leb 0 2024-04-12T05:09:11.157 INFO:tasks.workunit.client.1.smithi192.stdout:5/813: dwrite d0/f6e [0,4194304] 0 2024-04-12T05:09:11.158 INFO:tasks.workunit.client.1.smithi192.stdout:1/766: dread - d0/d1b/d44/ff8 zero size 2024-04-12T05:09:11.160 INFO:tasks.workunit.client.1.smithi192.stdout:6/772: write d3/d8/d56/d59/d8a/d99/f9f [4709669,103282] 0 2024-04-12T05:09:11.164 INFO:tasks.workunit.client.0.smithi113.stdout:6/799: dread d8/de/d17/f78 [0,4194304] 0 2024-04-12T05:09:11.164 INFO:tasks.workunit.client.1.smithi192.stdout:8/731: dwrite f5 [0,4194304] 0 2024-04-12T05:09:11.166 INFO:tasks.workunit.client.1.smithi192.stdout:6/773: read d3/d23/db2/fde [2265173,123038] 0 2024-04-12T05:09:11.169 INFO:tasks.workunit.client.1.smithi192.stdout:1/767: fsync d0/d2/d1e/d3f/df2/d3c/d5b/fa5 0 2024-04-12T05:09:11.170 INFO:tasks.workunit.client.1.smithi192.stdout:5/814: creat d0/d15/d3e/dca/dda/db5/dd7/d116/f117 x:0 0 0 2024-04-12T05:09:11.170 INFO:tasks.workunit.client.0.smithi113.stdout:6/800: write d8/de/d13/d11/d36/d3d/d4c/d26/d5b/ff8 [316901,99968] 0 2024-04-12T05:09:11.171 INFO:tasks.workunit.client.0.smithi113.stdout:6/801: dread - d8/de/dd5/f119 zero size 2024-04-12T05:09:11.176 INFO:tasks.workunit.client.0.smithi113.stdout:0/910: truncate d1/d43/d59/de9/f117 2167797 0 2024-04-12T05:09:11.179 INFO:tasks.workunit.client.1.smithi192.stdout:8/732: write d6/d10/d46/d5e/d5f/f57 [2979926,4055] 0 2024-04-12T05:09:11.179 INFO:tasks.workunit.client.1.smithi192.stdout:9/652: dread d11/d15/d21/d2c/d1f/d50/f98 [0,4194304] 0 2024-04-12T05:09:11.179 INFO:tasks.workunit.client.1.smithi192.stdout:2/751: write d1/d8/d17/d19/d60/f65 [1628133,82180] 0 2024-04-12T05:09:11.179 INFO:tasks.workunit.client.1.smithi192.stdout:1/768: chown d0/d2/c67 61728 1 2024-04-12T05:09:11.179 INFO:tasks.workunit.client.1.smithi192.stdout:9/653: fsync d11/d15/d21/d2c/f8a 0 2024-04-12T05:09:11.181 INFO:tasks.workunit.client.0.smithi113.stdout:3/790: write d0/d13/d15/d28/d2a/d34/fbd [73511,43017] 0 2024-04-12T05:09:11.181 INFO:tasks.workunit.client.0.smithi113.stdout:6/802: dwrite d8/de/d13/df0/d66/f106 [0,4194304] 0 2024-04-12T05:09:11.181 INFO:tasks.workunit.client.0.smithi113.stdout:2/808: write d8/d9/de/fd5 [411058,100655] 0 2024-04-12T05:09:11.182 INFO:tasks.workunit.client.1.smithi192.stdout:2/752: dread d1/f8b [4194304,4194304] 0 2024-04-12T05:09:11.182 INFO:tasks.workunit.client.0.smithi113.stdout:8/848: sync 2024-04-12T05:09:11.182 INFO:tasks.workunit.client.0.smithi113.stdout:9/825: sync 2024-04-12T05:09:11.182 INFO:tasks.workunit.client.0.smithi113.stdout:4/818: sync 2024-04-12T05:09:11.183 INFO:tasks.workunit.client.1.smithi192.stdout:0/871: write d0/d7b/d9b/d35/d109/d9f/d10a/d15/d19/f36 [1063064,47839] 0 2024-04-12T05:09:11.185 INFO:tasks.workunit.client.0.smithi113.stdout:9/826: chown d1/d4/d5/f103 176 1 2024-04-12T05:09:11.185 INFO:tasks.workunit.client.0.smithi113.stdout:2/809: readlink d8/d9/de/d12/d2b/daa/lba 0 2024-04-12T05:09:11.185 INFO:tasks.workunit.client.0.smithi113.stdout:0/911: mkdir d1/d43/df1/d118 0 2024-04-12T05:09:11.186 INFO:tasks.workunit.client.1.smithi192.stdout:6/774: getdents d3/d8/d56/d59/d8a 0 2024-04-12T05:09:11.189 INFO:tasks.workunit.client.0.smithi113.stdout:2/810: stat d8/d9/de/d12/d2b/daa/db3/fb1 0 2024-04-12T05:09:11.189 INFO:tasks.workunit.client.0.smithi113.stdout:4/819: chown da/d3c/d9f/fdd 16807 1 2024-04-12T05:09:11.190 INFO:tasks.workunit.client.1.smithi192.stdout:2/753: dwrite d1/d2d/fd1 [0,4194304] 0 2024-04-12T05:09:11.191 INFO:tasks.workunit.client.0.smithi113.stdout:0/912: symlink d1/d43/df1/l119 0 2024-04-12T05:09:11.191 INFO:tasks.workunit.client.0.smithi113.stdout:5/941: dread d0/fc [0,4194304] 0 2024-04-12T05:09:11.192 INFO:tasks.workunit.client.0.smithi113.stdout:6/803: creat d8/d45/d124/f126 x:0 0 0 2024-04-12T05:09:11.194 INFO:tasks.workunit.client.1.smithi192.stdout:0/872: unlink d0/d7b/d9b/d35/d109/d9f/d10a/faa 0 2024-04-12T05:09:11.196 INFO:tasks.workunit.client.0.smithi113.stdout:2/811: truncate d8/d9/de/d12/d2b/fb6 1129865 0 2024-04-12T05:09:11.196 INFO:tasks.workunit.client.0.smithi113.stdout:0/913: rmdir d1/d43/d59/db2 39 2024-04-12T05:09:11.197 INFO:tasks.workunit.client.0.smithi113.stdout:9/827: read d1/d4/f33 [1595635,72604] 0 2024-04-12T05:09:11.197 INFO:tasks.workunit.client.1.smithi192.stdout:1/769: creat d0/d29/d8d/d96/fff x:0 0 0 2024-04-12T05:09:11.198 INFO:tasks.workunit.client.1.smithi192.stdout:3/700: dread d5/d13/d3a/d8b/fba [0,4194304] 0 2024-04-12T05:09:11.199 INFO:tasks.workunit.client.0.smithi113.stdout:2/812: stat d8/d9/de/d43/db2/lbe 0 2024-04-12T05:09:11.200 INFO:tasks.workunit.client.1.smithi192.stdout:0/873: creat d0/d7b/d9b/d35/d109/d9f/f124 x:0 0 0 2024-04-12T05:09:11.200 INFO:tasks.workunit.client.0.smithi113.stdout:5/942: unlink d0/d10/d27/d1d/d41/f13c 0 2024-04-12T05:09:11.200 INFO:tasks.workunit.client.0.smithi113.stdout:3/791: link d0/d13/d15/cd9 d0/d13/d47/d94/dcf/dd1/de2/c109 0 2024-04-12T05:09:11.201 INFO:tasks.workunit.client.1.smithi192.stdout:3/701: write d5/d13/d3a/d70/d8c/fb4 [2604195,28780] 0 2024-04-12T05:09:11.201 INFO:tasks.workunit.client.1.smithi192.stdout:5/815: rmdir d0/d15/d3e/dca/dda/db5/d114 0 2024-04-12T05:09:11.202 INFO:tasks.workunit.client.0.smithi113.stdout:2/813: stat d8/d9/d27/f28 0 2024-04-12T05:09:11.203 INFO:tasks.workunit.client.0.smithi113.stdout:6/804: symlink d8/l127 0 2024-04-12T05:09:11.203 INFO:tasks.workunit.client.0.smithi113.stdout:9/828: mknod d1/d4/d5/d22/d9e/c113 0 2024-04-12T05:09:11.203 INFO:tasks.workunit.client.1.smithi192.stdout:1/770: mknod d0/d2/d1e/d3f/df2/d21/d62/dbc/c100 0 2024-04-12T05:09:11.204 INFO:tasks.workunit.client.0.smithi113.stdout:2/814: fsync d8/d9/de/d12/d3c/d8a/ff7 0 2024-04-12T05:09:11.205 INFO:tasks.workunit.client.0.smithi113.stdout:9/829: chown d1/d4/d5/d22/d9e 36835 1 2024-04-12T05:09:11.205 INFO:tasks.workunit.client.1.smithi192.stdout:1/771: dread d0/d1b/d28/f81 [0,4194304] 0 2024-04-12T05:09:11.206 INFO:tasks.workunit.client.1.smithi192.stdout:0/874: mknod d0/d7b/d9b/d35/d109/d53/df3/d29/db5/d11e/c125 0 2024-04-12T05:09:11.207 INFO:tasks.workunit.client.1.smithi192.stdout:2/754: creat d1/d8/d17/fed x:0 0 0 2024-04-12T05:09:11.208 INFO:tasks.workunit.client.0.smithi113.stdout:5/943: mknod d0/d10/d27/d1d/d3f/d73/c13f 0 2024-04-12T05:09:11.209 INFO:tasks.workunit.client.0.smithi113.stdout:0/914: link d1/d6/d34/f40 d1/d6/d34/dc3/f11a 0 2024-04-12T05:09:11.215 INFO:tasks.workunit.client.1.smithi192.stdout:8/733: dread d6/d10/fa5 [0,4194304] 0 2024-04-12T05:09:11.216 INFO:tasks.workunit.client.0.smithi113.stdout:5/944: chown d0/d10/d27/d1d/d38/d44/d66/cf0 47080431 1 2024-04-12T05:09:11.216 INFO:tasks.workunit.client.0.smithi113.stdout:4/820: dread da/d24/d60/f99 [0,4194304] 0 2024-04-12T05:09:11.216 INFO:tasks.workunit.client.0.smithi113.stdout:0/915: fsync d1/d6/d16/f47 0 2024-04-12T05:09:11.216 INFO:tasks.workunit.client.0.smithi113.stdout:5/945: stat d0/d10/d27/d1d/d38/d44/d66/cd0 0 2024-04-12T05:09:11.216 INFO:tasks.workunit.client.0.smithi113.stdout:7/796: write d0/dc/d6/d78/daa/fa6 [1177109,28596] 0 2024-04-12T05:09:11.216 INFO:tasks.workunit.client.1.smithi192.stdout:0/875: chown d0/d7b/d9b/d35/d109/db0/l113 1 1 2024-04-12T05:09:11.217 INFO:tasks.workunit.client.0.smithi113.stdout:8/849: truncate d9/d1b/d8e/d93/fdc 698039 0 2024-04-12T05:09:11.218 INFO:tasks.workunit.client.0.smithi113.stdout:5/946: readlink d0/d10/d27/l7c 0 2024-04-12T05:09:11.220 INFO:tasks.workunit.client.1.smithi192.stdout:2/755: rename d1/d2d/d73/da8/dd2 to d1/d8/d17/db3/dee 0 2024-04-12T05:09:11.220 INFO:tasks.workunit.client.1.smithi192.stdout:8/734: write d6/d10/d46/d5e/d5f/d37/f6f [1942383,47792] 0 2024-04-12T05:09:11.222 INFO:tasks.workunit.client.0.smithi113.stdout:3/792: write d0/d13/d47/fb2 [432889,122836] 0 2024-04-12T05:09:11.222 INFO:tasks.workunit.client.0.smithi113.stdout:8/850: readlink d9/d1b/d8e/d93/d32/d48/d8d/l8a 0 2024-04-12T05:09:11.222 INFO:tasks.workunit.client.0.smithi113.stdout:8/851: stat d9 0 2024-04-12T05:09:11.222 INFO:tasks.workunit.client.0.smithi113.stdout:5/947: dread - d0/d10/d27/d1d/d38/d75/f102 zero size 2024-04-12T05:09:11.224 INFO:tasks.workunit.client.1.smithi192.stdout:3/702: mkdir d5/d13/d3a/d4a/d61/d9f/deb/dfb 0 2024-04-12T05:09:11.225 INFO:tasks.workunit.client.0.smithi113.stdout:5/948: chown d0/d10/d6e/f4f 59 1 2024-04-12T05:09:11.225 INFO:tasks.workunit.client.1.smithi192.stdout:9/654: write d11/d15/d21/d2c/d1f/d50/f75 [185422,85589] 0 2024-04-12T05:09:11.225 INFO:tasks.workunit.client.0.smithi113.stdout:3/793: symlink d0/d13/d15/l10a 0 2024-04-12T05:09:11.226 INFO:tasks.workunit.client.0.smithi113.stdout:8/852: dread - d9/d1b/d8e/d93/fb7 zero size 2024-04-12T05:09:11.226 INFO:tasks.workunit.client.0.smithi113.stdout:5/949: dread - d0/d10/f12c zero size 2024-04-12T05:09:11.227 INFO:tasks.workunit.client.0.smithi113.stdout:7/797: truncate d0/dc/d6/d9b/f23 1173920 0 2024-04-12T05:09:11.230 INFO:tasks.workunit.client.0.smithi113.stdout:3/794: write d0/f1c [1130184,44809] 0 2024-04-12T05:09:11.232 INFO:tasks.workunit.client.0.smithi113.stdout:8/853: mkdir d9/d1b/d8e/d93/d32/d48/d8d/dae/d112 0 2024-04-12T05:09:11.234 INFO:tasks.workunit.client.0.smithi113.stdout:9/830: dread d1/d4/d5/f58 [4194304,4194304] 0 2024-04-12T05:09:11.234 INFO:tasks.workunit.client.0.smithi113.stdout:7/798: unlink d0/dc/d6/d9b/f5d 0 2024-04-12T05:09:11.235 INFO:tasks.workunit.client.1.smithi192.stdout:6/775: dwrite d3/d23/fb9 [0,4194304] 0 2024-04-12T05:09:11.237 INFO:tasks.workunit.client.0.smithi113.stdout:7/799: fsync d0/dc/d6/d9b/dcd/feb 0 2024-04-12T05:09:11.237 INFO:tasks.workunit.client.0.smithi113.stdout:5/950: symlink d0/d10/d27/d1d/d8c/l140 0 2024-04-12T05:09:11.238 INFO:tasks.workunit.client.1.smithi192.stdout:1/772: link d0/d1b/d44/fb3 d0/d29/d8d/f101 0 2024-04-12T05:09:11.239 INFO:tasks.workunit.client.0.smithi113.stdout:3/795: dread d0/d13/d15/faf [0,4194304] 0 2024-04-12T05:09:11.239 INFO:tasks.workunit.client.0.smithi113.stdout:4/821: write da/d3c/d83/d47/d10f/fcf [1448190,96714] 0 2024-04-12T05:09:11.241 INFO:tasks.workunit.client.1.smithi192.stdout:3/703: creat d5/d13/d3c/d36/da0/ffc x:0 0 0 2024-04-12T05:09:11.244 INFO:tasks.workunit.client.1.smithi192.stdout:8/735: dread d6/d10/d46/d5e/d5f/d31/f36 [0,4194304] 0 2024-04-12T05:09:11.244 INFO:tasks.workunit.client.1.smithi192.stdout:3/704: readlink d5/d13/d3c/l45 0 2024-04-12T05:09:11.244 INFO:tasks.workunit.client.1.smithi192.stdout:0/876: mkdir d0/d7b/d9b/d35/d109/d53/df3/d126 0 2024-04-12T05:09:11.245 INFO:tasks.workunit.client.1.smithi192.stdout:5/816: write d0/d15/d32/d45/f65 [1629837,121885] 0 2024-04-12T05:09:11.248 INFO:tasks.workunit.client.1.smithi192.stdout:2/756: dread d1/d2d/f32 [0,4194304] 0 2024-04-12T05:09:11.249 INFO:tasks.workunit.client.0.smithi113.stdout:4/822: mknod da/d3c/d9f/c117 0 2024-04-12T05:09:11.250 INFO:tasks.workunit.client.1.smithi192.stdout:3/705: stat d5/c8 0 2024-04-12T05:09:11.251 INFO:tasks.workunit.client.0.smithi113.stdout:3/796: truncate d0/d13/d15/f2e 4001816 0 2024-04-12T05:09:11.252 INFO:tasks.workunit.client.1.smithi192.stdout:0/877: mkdir d0/d7b/d9b/d4b/d72/d127 0 2024-04-12T05:09:11.252 INFO:tasks.workunit.client.1.smithi192.stdout:5/817: rmdir d0/d15/d32/d69 39 2024-04-12T05:09:11.253 INFO:tasks.workunit.client.1.smithi192.stdout:5/818: stat d0/d15/d3e/dca/fc4 0 2024-04-12T05:09:11.254 INFO:tasks.workunit.client.0.smithi113.stdout:0/916: dwrite d1/fa7 [0,4194304] 0 2024-04-12T05:09:11.254 INFO:tasks.workunit.client.0.smithi113.stdout:8/854: rename d9/d16/d3c/l6e to d9/dd4/dde/l113 0 2024-04-12T05:09:11.256 INFO:tasks.workunit.client.0.smithi113.stdout:9/831: mknod d1/d4/c114 0 2024-04-12T05:09:11.257 INFO:tasks.workunit.client.1.smithi192.stdout:9/655: rename d11/d15/d21/d2c to d11/d25/dec 0 2024-04-12T05:09:11.257 INFO:tasks.workunit.client.1.smithi192.stdout:9/656: read f10 [2814657,9678] 0 2024-04-12T05:09:11.259 INFO:tasks.workunit.client.1.smithi192.stdout:6/776: creat d3/d8/d56/d59/d97/f100 x:0 0 0 2024-04-12T05:09:11.259 INFO:tasks.workunit.client.1.smithi192.stdout:6/777: chown d3/d21 1177 1 2024-04-12T05:09:11.260 INFO:tasks.workunit.client.1.smithi192.stdout:3/706: symlink d5/d13/d3a/d70/d8c/de1/lfd 0 2024-04-12T05:09:11.261 INFO:tasks.workunit.client.0.smithi113.stdout:3/797: creat d0/d13/d15/d28/f10b x:0 0 0 2024-04-12T05:09:11.262 INFO:tasks.workunit.client.1.smithi192.stdout:3/707: stat d5/d13/d3a/fdc 0 2024-04-12T05:09:11.262 INFO:tasks.workunit.client.1.smithi192.stdout:6/778: write d3/d5/f66 [2342450,83804] 0 2024-04-12T05:09:11.262 INFO:tasks.workunit.client.1.smithi192.stdout:5/819: unlink d0/f14 0 2024-04-12T05:09:11.263 INFO:tasks.workunit.client.0.smithi113.stdout:8/855: getdents d9/dd4 0 2024-04-12T05:09:11.263 INFO:tasks.workunit.client.1.smithi192.stdout:6/779: chown d3/d5/de7 15487921 1 2024-04-12T05:09:11.264 INFO:tasks.workunit.client.0.smithi113.stdout:3/798: rmdir d0/d13/d15/d28/d2a/d34 39 2024-04-12T05:09:11.264 INFO:tasks.workunit.client.1.smithi192.stdout:3/708: write d5/d13/d3a/d70/d8c/fb4 [2783841,48060] 0 2024-04-12T05:09:11.265 INFO:tasks.workunit.client.0.smithi113.stdout:0/917: creat d1/d43/d59/db2/f11b x:0 0 0 2024-04-12T05:09:11.267 INFO:tasks.workunit.client.1.smithi192.stdout:0/878: rename d0/d7b/d9b/d35/d109/d95/db4/db8/f10d to d0/d7b/d9b/d35/d109/d95/db4/db8/f128 0 2024-04-12T05:09:11.270 INFO:tasks.workunit.client.0.smithi113.stdout:3/799: rmdir d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1 39 2024-04-12T05:09:11.270 INFO:tasks.workunit.client.1.smithi192.stdout:9/657: mknod d11/d25/dec/d1f/d6a/d7e/ced 0 2024-04-12T05:09:11.272 INFO:tasks.workunit.client.0.smithi113.stdout:4/823: dread da/dd/d1b/f2c [0,4194304] 0 2024-04-12T05:09:11.276 INFO:tasks.workunit.client.1.smithi192.stdout:1/773: dwrite d0/d2/d1e/d3f/d56/f5a [4194304,4194304] 0 2024-04-12T05:09:11.278 INFO:tasks.workunit.client.1.smithi192.stdout:5/820: mkdir d0/d15/d3e/dca/dda/db5/dbe/d118 0 2024-04-12T05:09:11.279 INFO:tasks.workunit.client.0.smithi113.stdout:4/824: read da/d3c/d83/d47/d10f/fcf [147348,67540] 0 2024-04-12T05:09:11.280 INFO:tasks.workunit.client.0.smithi113.stdout:4/825: stat da/dd 0 2024-04-12T05:09:11.280 INFO:tasks.workunit.client.0.smithi113.stdout:3/800: dwrite d0/d7e/fb3 [0,4194304] 0 2024-04-12T05:09:11.281 INFO:tasks.workunit.client.1.smithi192.stdout:1/774: creat d0/d2/d1e/d3f/df2/d21/d62/dbc/f102 x:0 0 0 2024-04-12T05:09:11.282 INFO:tasks.workunit.client.1.smithi192.stdout:0/879: mkdir d0/d7b/d129 0 2024-04-12T05:09:11.283 INFO:tasks.workunit.client.0.smithi113.stdout:9/832: dread d1/d4/d5/d22/fca [0,4194304] 0 2024-04-12T05:09:11.284 INFO:tasks.workunit.client.1.smithi192.stdout:2/757: rename d1/d8/d17/d19/d71/cdb to d1/d8/d3a/de/cef 0 2024-04-12T05:09:11.285 INFO:tasks.workunit.client.1.smithi192.stdout:2/758: dread - d1/d2d/d73/f9b zero size 2024-04-12T05:09:11.286 INFO:tasks.workunit.client.0.smithi113.stdout:9/833: write d1/d27/d7a/fde [6701,75476] 0 2024-04-12T05:09:11.289 INFO:tasks.workunit.client.1.smithi192.stdout:3/709: rename d5/d13/d3c/d36/da0/fd1 to d5/d13/d3a/d4a/d61/ffe 0 2024-04-12T05:09:11.289 INFO:tasks.workunit.client.0.smithi113.stdout:0/918: dread d1/d43/d59/de9/d105/d50/d52/d7e/ff2 [0,4194304] 0 2024-04-12T05:09:11.290 INFO:tasks.workunit.client.1.smithi192.stdout:1/775: getdents d0/dd0 0 2024-04-12T05:09:11.290 INFO:tasks.workunit.client.0.smithi113.stdout:0/919: fsync d1/d80/f10d 0 2024-04-12T05:09:11.293 INFO:tasks.workunit.client.1.smithi192.stdout:9/658: rename d11/d25/d36/d6e/d71/c9b to d11/d25/dec/d1f/d50/d73/cee 0 2024-04-12T05:09:11.294 INFO:tasks.workunit.client.0.smithi113.stdout:9/834: truncate d1/f79 3052331 0 2024-04-12T05:09:11.296 INFO:tasks.workunit.client.0.smithi113.stdout:3/801: getdents d0/d13/d15/d28/d2a/d8f 0 2024-04-12T05:09:11.296 INFO:tasks.workunit.client.0.smithi113.stdout:5/951: dread d0/d10/d27/d1d/d38/d44/d4e/f110 [0,4194304] 0 2024-04-12T05:09:11.297 INFO:tasks.workunit.client.1.smithi192.stdout:1/776: rmdir d0/d29/d8d 39 2024-04-12T05:09:11.298 INFO:tasks.workunit.client.1.smithi192.stdout:1/777: readlink d0/d2/d1e/d3f/df2/d21/d62/la1 0 2024-04-12T05:09:11.299 INFO:tasks.workunit.client.0.smithi113.stdout:5/952: rmdir d0/d10/da1 39 2024-04-12T05:09:11.299 INFO:tasks.workunit.client.1.smithi192.stdout:0/880: rename d0/d7b/d9b/d35/d109/d9f/d10a/d67/db1/fe9 to d0/d7b/d9b/d35/d109/d9f/d10a/d67/dd2/f12a 0 2024-04-12T05:09:11.300 INFO:tasks.workunit.client.1.smithi192.stdout:5/821: dread d0/d15/d24/d29/f2a [0,4194304] 0 2024-04-12T05:09:11.301 INFO:tasks.workunit.client.1.smithi192.stdout:9/659: creat d11/d25/dd3/fef x:0 0 0 2024-04-12T05:09:11.301 INFO:tasks.workunit.client.0.smithi113.stdout:5/953: dread - d0/d10/d27/d1d/d8c/db8/fd5 zero size 2024-04-12T05:09:11.302 INFO:tasks.workunit.client.0.smithi113.stdout:0/920: rmdir d1/d43/df1/d118 0 2024-04-12T05:09:11.303 INFO:tasks.workunit.client.1.smithi192.stdout:9/660: write d11/d15/f6b [4703022,47647] 0 2024-04-12T05:09:11.305 INFO:tasks.workunit.client.0.smithi113.stdout:5/954: symlink d0/d10/d27/d1d/d38/d44/d66/l141 0 2024-04-12T05:09:11.306 INFO:tasks.workunit.client.1.smithi192.stdout:3/710: rename d5/f2d to d5/d13/d3a/d70/d8c/d8d/fff 0 2024-04-12T05:09:11.307 INFO:tasks.workunit.client.1.smithi192.stdout:5/822: mknod d0/d15/d3e/dca/dda/db5/c119 0 2024-04-12T05:09:11.308 INFO:tasks.workunit.client.1.smithi192.stdout:5/823: chown d0/d15/d64/c73 24 1 2024-04-12T05:09:11.309 INFO:tasks.workunit.client.0.smithi113.stdout:5/955: symlink d0/d10/l142 0 2024-04-12T05:09:11.313 INFO:tasks.workunit.client.0.smithi113.stdout:3/802: dread d0/d13/d47/d94/fd4 [0,4194304] 0 2024-04-12T05:09:11.314 INFO:tasks.workunit.client.1.smithi192.stdout:3/711: rename d5/d13/d3a/d4a/f67 to d5/d13/d3a/d4a/d61/f100 0 2024-04-12T05:09:11.315 INFO:tasks.workunit.client.0.smithi113.stdout:9/835: dread d1/df/db6/f84 [0,4194304] 0 2024-04-12T05:09:11.315 INFO:tasks.workunit.client.0.smithi113.stdout:5/956: link d0/d10/d27/d1d/d8c/db8/dc6/fec d0/d10/d27/d5f/f143 0 2024-04-12T05:09:11.315 INFO:tasks.workunit.client.1.smithi192.stdout:5/824: chown d0/c1f 773082426 1 2024-04-12T05:09:11.316 INFO:tasks.workunit.client.1.smithi192.stdout:9/661: creat d11/d15/ff0 x:0 0 0 2024-04-12T05:09:11.317 INFO:tasks.workunit.client.1.smithi192.stdout:1/778: dread d0/d2/f34 [0,4194304] 0 2024-04-12T05:09:11.317 INFO:tasks.workunit.client.1.smithi192.stdout:3/712: symlink d5/d13/d3a/d4a/d61/l101 0 2024-04-12T05:09:11.318 INFO:tasks.workunit.client.0.smithi113.stdout:5/957: creat d0/d10/d27/d1d/d38/d44/dcb/f144 x:0 0 0 2024-04-12T05:09:11.318 INFO:tasks.workunit.client.1.smithi192.stdout:5/825: mknod d0/d15/d32/db2/c11a 0 2024-04-12T05:09:11.321 INFO:tasks.workunit.client.0.smithi113.stdout:3/803: link d0/d13/d15/d28/d2a/l5b d0/d13/d47/d94/dcf/dd1/l10c 0 2024-04-12T05:09:11.321 INFO:tasks.workunit.client.1.smithi192.stdout:5/826: dread - d0/d15/d3e/dca/dda/da0/ff6 zero size 2024-04-12T05:09:11.321 INFO:tasks.workunit.client.0.smithi113.stdout:3/804: read - d0/d13/d15/d28/d2a/d34/d5f/fdb zero size 2024-04-12T05:09:11.322 INFO:tasks.workunit.client.1.smithi192.stdout:3/713: creat d5/d13/d3a/d4e/f102 x:0 0 0 2024-04-12T05:09:11.323 INFO:tasks.workunit.client.1.smithi192.stdout:1/779: mknod d0/d2/d1e/d3f/df2/d3c/d5b/db6/c103 0 2024-04-12T05:09:11.324 INFO:tasks.workunit.client.0.smithi113.stdout:5/958: symlink d0/d10/d27/d5f/d128/d12d/l145 0 2024-04-12T05:09:11.324 INFO:tasks.workunit.client.1.smithi192.stdout:9/662: dread d11/d25/d36/d6e/d33/f6c [0,4194304] 0 2024-04-12T05:09:11.325 INFO:tasks.workunit.client.1.smithi192.stdout:5/827: link d0/c9 d0/d15/d32/db2/c11b 0 2024-04-12T05:09:11.327 INFO:tasks.workunit.client.0.smithi113.stdout:3/805: rename d0/l1e to d0/d13/d15/d28/d2a/d34/l10d 0 2024-04-12T05:09:11.327 INFO:tasks.workunit.client.1.smithi192.stdout:9/663: chown d11/d15/c46 4 1 2024-04-12T05:09:11.328 INFO:tasks.workunit.client.1.smithi192.stdout:1/780: dread d0/d2/d1e/d3f/df2/d21/d33/f42 [0,4194304] 0 2024-04-12T05:09:11.329 INFO:tasks.workunit.client.1.smithi192.stdout:9/664: rmdir d11/d25/dec/d1f/d6a/d7e 39 2024-04-12T05:09:11.330 INFO:tasks.workunit.client.0.smithi113.stdout:5/959: creat d0/d10/d27/d1d/d3f/d73/f146 x:0 0 0 2024-04-12T05:09:11.331 INFO:tasks.workunit.client.1.smithi192.stdout:1/781: chown d0/d29/d8d/d96/fff 289379392 1 2024-04-12T05:09:11.331 INFO:tasks.workunit.client.1.smithi192.stdout:9/665: truncate d11/d25/d36/d6e/d33/dbb/fdb 94440 0 2024-04-12T05:09:11.332 INFO:tasks.workunit.client.0.smithi113.stdout:5/960: write d0/fa8 [2149184,92927] 0 2024-04-12T05:09:11.334 INFO:tasks.workunit.client.0.smithi113.stdout:3/806: dwrite d0/d13/d15/d28/fa5 [4194304,4194304] 0 2024-04-12T05:09:11.336 INFO:tasks.workunit.client.0.smithi113.stdout:5/961: truncate d0/d10/d27/d1d/d3f/d7e/df8/ff9 263236 0 2024-04-12T05:09:11.337 INFO:tasks.workunit.client.1.smithi192.stdout:1/782: creat d0/d1b/f104 x:0 0 0 2024-04-12T05:09:11.339 INFO:tasks.workunit.client.1.smithi192.stdout:1/783: unlink d0/d2/d1e/d3f/d56/l88 0 2024-04-12T05:09:11.340 INFO:tasks.workunit.client.0.smithi113.stdout:5/962: symlink d0/d10/d27/d1d/d3f/d73/dcd/l147 0 2024-04-12T05:09:11.341 INFO:tasks.workunit.client.0.smithi113.stdout:5/963: write d0/d10/d27/d1d/d3f/f6a [869645,71506] 0 2024-04-12T05:09:11.343 INFO:tasks.workunit.client.0.smithi113.stdout:5/964: stat d0/d10/f16 0 2024-04-12T05:09:11.343 INFO:tasks.workunit.client.1.smithi192.stdout:1/784: rmdir d0/d2/d1e/d3f/df2/d3c/d5b/da7/dd7 39 2024-04-12T05:09:11.345 INFO:tasks.workunit.client.0.smithi113.stdout:1/906: sync 2024-04-12T05:09:11.346 INFO:tasks.workunit.client.1.smithi192.stdout:1/785: truncate d0/d1b/f6a 961791 0 2024-04-12T05:09:11.347 INFO:tasks.workunit.client.0.smithi113.stdout:1/907: creat d10/d2e/d7c/dc5/dc8/de1/f120 x:0 0 0 2024-04-12T05:09:11.347 INFO:tasks.workunit.client.0.smithi113.stdout:1/908: chown d10/d41/d89/lca 348918759 1 2024-04-12T05:09:11.349 INFO:tasks.workunit.client.0.smithi113.stdout:1/909: mkdir d10/d2e/d31/d5c/daf/d121 0 2024-04-12T05:09:11.350 INFO:tasks.workunit.client.0.smithi113.stdout:1/910: fsync d10/d2e/d7c/dc5/dc8/f104 0 2024-04-12T05:09:11.353 INFO:tasks.workunit.client.0.smithi113.stdout:1/911: dread d10/d41/d89/d96/d7d/f79 [0,4194304] 0 2024-04-12T05:09:11.353 INFO:tasks.workunit.client.0.smithi113.stdout:1/912: read - d10/d2e/dd8/d70/fe3 zero size 2024-04-12T05:09:11.354 INFO:tasks.workunit.client.0.smithi113.stdout:1/913: readlink d10/d14/d95/dcf/l115 0 2024-04-12T05:09:11.354 INFO:tasks.workunit.client.1.smithi192.stdout:6/780: read d3/d8/f13 [364509,59649] 0 2024-04-12T05:09:11.358 INFO:tasks.workunit.client.0.smithi113.stdout:1/914: rename d10/d55/f8c to d10/d2e/d31/d5c/daf/f122 0 2024-04-12T05:09:11.360 INFO:tasks.workunit.client.0.smithi113.stdout:1/915: symlink d10/d41/d89/d96/d7d/l123 0 2024-04-12T05:09:11.361 INFO:tasks.workunit.client.1.smithi192.stdout:6/781: truncate d3/dc/fc4 4503447 0 2024-04-12T05:09:11.368 INFO:tasks.workunit.client.0.smithi113.stdout:8/856: dwrite d9/d1b/f45 [4194304,4194304] 0 2024-04-12T05:09:11.374 INFO:tasks.workunit.client.1.smithi192.stdout:2/759: getdents d1/d8/d17/d19/d71 0 2024-04-12T05:09:11.376 INFO:tasks.workunit.client.1.smithi192.stdout:2/760: readlink d1/d8/d17/d19/l54 0 2024-04-12T05:09:11.378 INFO:tasks.workunit.client.1.smithi192.stdout:3/714: rename d5/d13/d3c/d36/da0 to d5/d13/d1a/d5f/d103 0 2024-04-12T05:09:11.378 INFO:tasks.workunit.client.1.smithi192.stdout:3/715: chown d5/d13/lb9 112469 1 2024-04-12T05:09:11.379 INFO:tasks.workunit.client.1.smithi192.stdout:3/716: read - d5/d13/d3c/f31 zero size 2024-04-12T05:09:11.379 INFO:tasks.workunit.client.1.smithi192.stdout:2/761: mkdir d1/d2d/d73/de7/df0 0 2024-04-12T05:09:11.380 INFO:tasks.workunit.client.1.smithi192.stdout:2/762: read - d1/d8/d17/fed zero size 2024-04-12T05:09:11.381 INFO:tasks.workunit.client.0.smithi113.stdout:9/836: truncate d1/df/d29/da6/fdb 8168727 0 2024-04-12T05:09:11.381 INFO:tasks.workunit.client.1.smithi192.stdout:3/717: read d5/d13/d3a/d4e/f9a [460428,108915] 0 2024-04-12T05:09:11.382 INFO:tasks.workunit.client.1.smithi192.stdout:3/718: dread - d5/fd8 zero size 2024-04-12T05:09:11.383 INFO:tasks.workunit.client.0.smithi113.stdout:9/837: creat d1/d27/d7a/f115 x:0 0 0 2024-04-12T05:09:11.385 INFO:tasks.workunit.client.0.smithi113.stdout:9/838: mkdir d1/d116 0 2024-04-12T05:09:11.387 INFO:tasks.workunit.client.1.smithi192.stdout:3/719: chown d5/d13/d3a/c77 1 1 2024-04-12T05:09:11.387 INFO:tasks.workunit.client.0.smithi113.stdout:8/857: truncate d9/d1b/d8e/d93/faf 2993741 0 2024-04-12T05:09:11.388 INFO:tasks.workunit.client.1.smithi192.stdout:3/720: chown d5/d13/d3a/d8b/da2 608160 1 2024-04-12T05:09:11.390 INFO:tasks.workunit.client.1.smithi192.stdout:9/666: rename d11/d15/d21/fe1 to d11/d25/ff1 0 2024-04-12T05:09:11.390 INFO:tasks.workunit.client.1.smithi192.stdout:0/881: write d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d3a/f76 [992194,129274] 0 2024-04-12T05:09:11.391 INFO:tasks.workunit.client.0.smithi113.stdout:3/807: dwrite d0/d13/d47/f89 [0,4194304] 0 2024-04-12T05:09:11.393 INFO:tasks.workunit.client.1.smithi192.stdout:2/763: creat d1/d8/d17/d19/d60/d59/d68/dea/ff1 x:0 0 0 2024-04-12T05:09:11.393 INFO:tasks.workunit.client.0.smithi113.stdout:8/858: mknod d9/d1b/d8e/d93/d92/c114 0 2024-04-12T05:09:11.394 INFO:tasks.workunit.client.0.smithi113.stdout:0/921: truncate d1/fc9 4006969 0 2024-04-12T05:09:11.394 INFO:tasks.workunit.client.0.smithi113.stdout:9/839: getdents d1/d4/d38/d65/dcd/deb 0 2024-04-12T05:09:11.395 INFO:tasks.workunit.client.1.smithi192.stdout:0/882: dread d0/d7b/d9b/d35/d109/d53/d6d/d108/f11b [0,4194304] 0 2024-04-12T05:09:11.396 INFO:tasks.workunit.client.1.smithi192.stdout:0/883: stat d0/d7b/d9b/d35/d109/d9f/d10a/d15/l45 0 2024-04-12T05:09:11.396 INFO:tasks.workunit.client.0.smithi113.stdout:3/808: creat d0/d13/d47/d94/dac/f10e x:0 0 0 2024-04-12T05:09:11.399 INFO:tasks.workunit.client.1.smithi192.stdout:6/782: rename d3/d8/d10/f71 to d3/d5/d9c/f101 0 2024-04-12T05:09:11.401 INFO:tasks.workunit.client.0.smithi113.stdout:0/922: creat d1/d6/d34/f11c x:0 0 0 2024-04-12T05:09:11.403 INFO:tasks.workunit.client.1.smithi192.stdout:2/764: mknod d1/d8/d3a/de/d7c/cf2 0 2024-04-12T05:09:11.406 INFO:tasks.workunit.client.0.smithi113.stdout:9/840: dwrite d1/d4/d5/d22/fca [0,4194304] 0 2024-04-12T05:09:11.406 INFO:tasks.workunit.client.1.smithi192.stdout:0/884: mkdir d0/d7b/d9b/d35/d109/d95/d12b 0 2024-04-12T05:09:11.408 INFO:tasks.workunit.client.0.smithi113.stdout:0/923: symlink d1/d43/d59/de9/l11d 0 2024-04-12T05:09:11.409 INFO:tasks.workunit.client.1.smithi192.stdout:6/783: truncate d3/d8/d10/d4c/d68/f89 554211 0 2024-04-12T05:09:11.413 INFO:tasks.workunit.client.0.smithi113.stdout:0/924: chown d1/d6/d16/dd5/l6e 94845090 1 2024-04-12T05:09:11.414 INFO:tasks.workunit.client.0.smithi113.stdout:9/841: rename d1/d4/d5/d22/d9e to d1/d4/d5/d17/d117 0 2024-04-12T05:09:11.416 INFO:tasks.workunit.client.1.smithi192.stdout:3/721: creat d5/f104 x:0 0 0 2024-04-12T05:09:11.416 INFO:tasks.workunit.client.1.smithi192.stdout:1/786: write d0/d1b/d28/f81 [810904,123817] 0 2024-04-12T05:09:11.417 INFO:tasks.workunit.client.1.smithi192.stdout:2/765: mkdir d1/d8/d3a/df3 0 2024-04-12T05:09:11.418 INFO:tasks.workunit.client.0.smithi113.stdout:9/842: mknod d1/d27/d7a/de4/c118 0 2024-04-12T05:09:11.418 INFO:tasks.workunit.client.1.smithi192.stdout:5/828: dwrite d0/d21/d5e/f6c [0,4194304] 0 2024-04-12T05:09:11.419 INFO:tasks.workunit.client.0.smithi113.stdout:0/925: mkdir d1/d43/d59/de9/d11e 0 2024-04-12T05:09:11.427 INFO:tasks.workunit.client.1.smithi192.stdout:5/829: dwrite d0/d15/d24/f4b [0,4194304] 0 2024-04-12T05:09:11.427 INFO:tasks.workunit.client.1.smithi192.stdout:2/766: dwrite d1/d8/d3a/de/f30 [0,4194304] 0 2024-04-12T05:09:11.429 INFO:tasks.workunit.client.1.smithi192.stdout:2/767: chown d1/d8/d3a/de/d7c/d98/fb7 3188974 1 2024-04-12T05:09:11.430 INFO:tasks.workunit.client.0.smithi113.stdout:1/916: write d10/d14/f4d [407448,25802] 0 2024-04-12T05:09:11.460 INFO:tasks.workunit.client.0.smithi113.stdout:0/926: dread d1/d43/d59/de9/d105/d50/db6/d36/f74 [0,4194304] 0 2024-04-12T05:09:11.465 INFO:tasks.workunit.client.0.smithi113.stdout:3/809: truncate d0/d13/d15/d28/d2a/d34/d5f/d7b/fcd 214235 0 2024-04-12T05:09:11.470 INFO:tasks.workunit.client.0.smithi113.stdout:9/843: write d1/f78 [1858431,93337] 0 2024-04-12T05:09:11.495 INFO:tasks.workunit.client.1.smithi192.stdout:1/787: dread - d0/d2/d1e/d3f/df2/d3c/d5b/da7/dd7/ffa zero size 2024-04-12T05:09:11.498 INFO:tasks.workunit.client.1.smithi192.stdout:6/784: symlink d3/d8/d48/d85/l102 0 2024-04-12T05:09:11.498 INFO:tasks.workunit.client.1.smithi192.stdout:3/722: dread d5/d13/d3a/d4e/f95 [0,4194304] 0 2024-04-12T05:09:11.501 INFO:tasks.workunit.client.0.smithi113.stdout:3/810: unlink d0/d13/d15/c8d 0 2024-04-12T05:09:11.503 INFO:tasks.workunit.client.0.smithi113.stdout:9/844: rename d1/df/d4b/d63/d3d/d6f/l94 to d1/d27/l119 0 2024-04-12T05:09:11.504 INFO:tasks.workunit.client.0.smithi113.stdout:3/811: fdatasync d0/d13/d15/d28/ffe 0 2024-04-12T05:09:11.509 INFO:tasks.workunit.client.0.smithi113.stdout:9/845: rename d1/df/d4b/d63/d3d/f49 to d1/d4/d38/d65/dcd/deb/f11a 0 2024-04-12T05:09:11.511 INFO:tasks.workunit.client.0.smithi113.stdout:3/812: dwrite d0/d13/d15/d28/fa5 [4194304,4194304] 0 2024-04-12T05:09:11.512 INFO:tasks.workunit.client.0.smithi113.stdout:9/846: mkdir d1/d27/d7a/d11b 0 2024-04-12T05:09:11.516 INFO:tasks.workunit.client.0.smithi113.stdout:3/813: mknod d0/d7e/c10f 0 2024-04-12T05:09:11.518 INFO:tasks.workunit.client.0.smithi113.stdout:0/927: unlink d1/d6/d16/dd5/dbb/cfb 0 2024-04-12T05:09:11.522 INFO:tasks.workunit.client.1.smithi192.stdout:0/885: getdents d0 0 2024-04-12T05:09:11.523 INFO:tasks.workunit.client.1.smithi192.stdout:5/830: getdents d0/d15/d3e/dca/db9/ded 0 2024-04-12T05:09:11.525 INFO:tasks.workunit.client.0.smithi113.stdout:9/847: dwrite d1/d27/d7a/fde [4194304,4194304] 0 2024-04-12T05:09:11.525 INFO:tasks.workunit.client.0.smithi113.stdout:1/917: getdents d10/d2e/d7c/dc5/dc8/de1 0 2024-04-12T05:09:11.525 INFO:tasks.workunit.client.0.smithi113.stdout:1/918: chown d10/d2e/l4b 2024 1 2024-04-12T05:09:11.526 INFO:tasks.workunit.client.0.smithi113.stdout:3/814: mkdir d0/d13/d15/d110 0 2024-04-12T05:09:11.531 INFO:tasks.workunit.client.0.smithi113.stdout:9/848: mknod d1/df/d4b/d63/d3d/c11c 0 2024-04-12T05:09:11.536 INFO:tasks.workunit.client.0.smithi113.stdout:3/815: rename d0/d13/d15/faf to d0/d13/d15/d28/d2a/f111 0 2024-04-12T05:09:11.537 INFO:tasks.workunit.client.0.smithi113.stdout:3/816: chown d0/d13/d15/d28/l32 989 1 2024-04-12T05:09:11.539 INFO:tasks.workunit.client.1.smithi192.stdout:5/831: truncate d0/d15/fc3 258422 0 2024-04-12T05:09:11.540 INFO:tasks.workunit.client.1.smithi192.stdout:5/832: readlink d0/d15/d24/l31 0 2024-04-12T05:09:11.541 INFO:tasks.workunit.client.0.smithi113.stdout:3/817: creat d0/d13/d47/d94/dac/f112 x:0 0 0 2024-04-12T05:09:11.541 INFO:tasks.workunit.client.1.smithi192.stdout:1/788: creat d0/d2/d1e/d3f/df2/d21/d39/f105 x:0 0 0 2024-04-12T05:09:11.544 INFO:tasks.workunit.client.1.smithi192.stdout:1/789: chown d0/d2/d1e/d3f/df2/cd2 2701 1 2024-04-12T05:09:11.546 INFO:tasks.workunit.client.1.smithi192.stdout:0/886: creat d0/d7b/d9b/d35/d109/d9f/d10a/d67/dd2/df9/f12c x:0 0 0 2024-04-12T05:09:11.547 INFO:tasks.workunit.client.0.smithi113.stdout:3/818: dwrite d0/d13/d15/f3d [4194304,4194304] 0 2024-04-12T05:09:11.550 INFO:tasks.workunit.client.1.smithi192.stdout:5/833: dwrite d0/d15/d3e/dca/dda/ffe [4194304,4194304] 0 2024-04-12T05:09:11.557 INFO:tasks.workunit.client.1.smithi192.stdout:3/723: getdents d5/d13/d3a 0 2024-04-12T05:09:11.572 INFO:tasks.workunit.client.0.smithi113.stdout:9/849: dread d1/df/db6/f59 [0,4194304] 0 2024-04-12T05:09:11.572 INFO:tasks.workunit.client.0.smithi113.stdout:3/819: write d0/d13/d15/d28/d2a/d8f/fc2 [633797,32359] 0 2024-04-12T05:09:11.573 INFO:tasks.workunit.client.1.smithi192.stdout:5/834: dread d0/d15/d32/d45/fc6 [0,4194304] 0 2024-04-12T05:09:11.575 INFO:tasks.workunit.client.1.smithi192.stdout:3/724: mkdir d5/d13/d3a/d8b/da2/d105 0 2024-04-12T05:09:11.576 INFO:tasks.workunit.client.0.smithi113.stdout:9/850: creat d1/f11d x:0 0 0 2024-04-12T05:09:11.577 INFO:tasks.workunit.client.0.smithi113.stdout:6/805: sync 2024-04-12T05:09:11.577 INFO:tasks.workunit.client.0.smithi113.stdout:3/820: mkdir d0/d13/d15/d28/d2a/d8b/d91/d113 0 2024-04-12T05:09:11.578 INFO:tasks.workunit.client.1.smithi192.stdout:3/725: chown d5/d13/d3c/l45 0 1 2024-04-12T05:09:11.579 INFO:tasks.workunit.client.0.smithi113.stdout:9/851: rename d1/ca9 to d1/d27/d7a/d11b/c11e 0 2024-04-12T05:09:11.580 INFO:tasks.workunit.client.0.smithi113.stdout:6/806: chown d8/de/d13/d11/d36/d3d/d4c/d26/d5b/ff8 9339 1 2024-04-12T05:09:11.582 INFO:tasks.workunit.client.0.smithi113.stdout:3/821: unlink d0/d13/d47/d4f/f90 0 2024-04-12T05:09:11.582 INFO:tasks.workunit.client.1.smithi192.stdout:3/726: readlink d5/d13/d94/dbd/lf7 0 2024-04-12T05:09:11.583 INFO:tasks.workunit.client.1.smithi192.stdout:6/785: mkdir d3/dc/d103 0 2024-04-12T05:09:11.583 INFO:tasks.workunit.client.1.smithi192.stdout:1/790: dread d0/f24 [0,4194304] 0 2024-04-12T05:09:11.585 INFO:tasks.workunit.client.1.smithi192.stdout:5/835: dwrite d0/d15/d32/d45/fb1 [0,4194304] 0 2024-04-12T05:09:11.586 INFO:tasks.workunit.client.0.smithi113.stdout:6/807: mkdir d8/de/d13/d11/d36/d53/db8/dce/d128 0 2024-04-12T05:09:11.586 INFO:tasks.workunit.client.0.smithi113.stdout:1/919: dread d10/d41/d89/d96/f91 [0,4194304] 0 2024-04-12T05:09:11.590 INFO:tasks.workunit.client.1.smithi192.stdout:4/815: sync 2024-04-12T05:09:11.590 INFO:tasks.workunit.client.1.smithi192.stdout:7/674: sync 2024-04-12T05:09:11.591 INFO:tasks.workunit.client.1.smithi192.stdout:1/791: mkdir d0/d2/d1e/d3f/df2/d3c/d106 0 2024-04-12T05:09:11.592 INFO:tasks.workunit.client.1.smithi192.stdout:5/836: write d0/d15/d24/d47/fdd [4213716,75597] 0 2024-04-12T05:09:11.593 INFO:tasks.workunit.client.1.smithi192.stdout:3/727: dwrite d5/d13/d3a/d4e/f102 [0,4194304] 0 2024-04-12T05:09:11.594 INFO:tasks.workunit.client.1.smithi192.stdout:5/837: stat d0/d15/d32/l4a 0 2024-04-12T05:09:11.594 INFO:tasks.workunit.client.1.smithi192.stdout:1/792: chown d0/d2/d1e/d3f/df2/lde 42 1 2024-04-12T05:09:11.595 INFO:tasks.workunit.client.0.smithi113.stdout:1/920: mkdir d10/d41/d89/d96/dd3/d124 0 2024-04-12T05:09:11.595 INFO:tasks.workunit.client.1.smithi192.stdout:5/838: fdatasync d0/d15/d3e/dca/dda/da7/f105 0 2024-04-12T05:09:11.595 INFO:tasks.workunit.client.0.smithi113.stdout:9/852: dwrite d1/d4/d38/d65/dcd/dd7/fe3 [0,4194304] 0 2024-04-12T05:09:11.599 INFO:tasks.workunit.client.0.smithi113.stdout:9/853: creat d1/df/d29/f11f x:0 0 0 2024-04-12T05:09:11.605 INFO:tasks.workunit.client.0.smithi113.stdout:3/822: dwrite d0/d13/d47/f61 [0,4194304] 0 2024-04-12T05:09:11.609 INFO:tasks.workunit.client.0.smithi113.stdout:3/823: write d0/d13/d15/d28/d2a/d34/d5f/fe1 [736397,124375] 0 2024-04-12T05:09:11.611 INFO:tasks.workunit.client.0.smithi113.stdout:9/854: dwrite d1/df/db6/f6b [0,4194304] 0 2024-04-12T05:09:11.614 INFO:tasks.workunit.client.1.smithi192.stdout:1/793: mkdir d0/dd0/d107 0 2024-04-12T05:09:11.615 INFO:tasks.workunit.client.1.smithi192.stdout:5/839: readlink d0/d15/d32/d69/d7d/le6 0 2024-04-12T05:09:11.625 INFO:tasks.workunit.client.0.smithi113.stdout:9/855: creat d1/d4/d38/d65/dcd/f120 x:0 0 0 2024-04-12T05:09:11.626 INFO:tasks.workunit.client.1.smithi192.stdout:4/816: mkdir d1/d1d/d53/d1c/d95/dc6/dd5/de5/ded/d119 0 2024-04-12T05:09:11.627 INFO:tasks.workunit.client.0.smithi113.stdout:9/856: read - d1/d4/d6d/fd4 zero size 2024-04-12T05:09:11.627 INFO:tasks.workunit.client.1.smithi192.stdout:5/840: mkdir d0/d15/d3e/dca/dda/d11c 0 2024-04-12T05:09:11.628 INFO:tasks.workunit.client.0.smithi113.stdout:3/824: getdents d0/d13/dbf 0 2024-04-12T05:09:11.630 INFO:tasks.workunit.client.0.smithi113.stdout:9/857: rmdir d1/df/d4b/df7 39 2024-04-12T05:09:11.631 INFO:tasks.workunit.client.1.smithi192.stdout:4/817: chown d1/d54/d82/da4/fe3 23 1 2024-04-12T05:09:11.631 INFO:tasks.workunit.client.1.smithi192.stdout:6/786: link d3/d8/d56/lb1 d3/d8/d10/l104 0 2024-04-12T05:09:11.633 INFO:tasks.workunit.client.1.smithi192.stdout:6/787: symlink d3/d5/d9c/l105 0 2024-04-12T05:09:11.635 INFO:tasks.workunit.client.0.smithi113.stdout:9/858: unlink d1/d27/d7a/de4/c118 0 2024-04-12T05:09:11.637 INFO:tasks.workunit.client.0.smithi113.stdout:9/859: write d1/d4/d87/fbb [2013874,46635] 0 2024-04-12T05:09:11.637 INFO:tasks.workunit.client.1.smithi192.stdout:5/841: dread d0/d15/d32/d69/d7d/fcc [0,4194304] 0 2024-04-12T05:09:11.639 INFO:tasks.workunit.client.1.smithi192.stdout:6/788: mkdir d3/d8/d56/d59/d97/dbf/deb/d106 0 2024-04-12T05:09:11.639 INFO:tasks.workunit.client.1.smithi192.stdout:5/842: fdatasync d0/d15/d24/d29/f85 0 2024-04-12T05:09:11.642 INFO:tasks.workunit.client.0.smithi113.stdout:9/860: truncate d1/d4/d6d/fc4 767437 0 2024-04-12T05:09:11.644 INFO:tasks.workunit.client.1.smithi192.stdout:5/843: creat d0/d15/d3e/dca/dda/db5/dbe/f11d x:0 0 0 2024-04-12T05:09:11.646 INFO:tasks.workunit.client.1.smithi192.stdout:6/789: read d3/d8/d10/d4c/d68/f77 [3105481,102356] 0 2024-04-12T05:09:11.647 INFO:tasks.workunit.client.1.smithi192.stdout:5/844: mknod d0/d15/d3e/dca/db9/c11e 0 2024-04-12T05:09:11.647 INFO:tasks.workunit.client.0.smithi113.stdout:0/928: write d1/d43/d59/de9/d105/d50/fae [716621,124898] 0 2024-04-12T05:09:11.650 INFO:tasks.workunit.client.1.smithi192.stdout:5/845: chown d0/d15/l62 2071193 1 2024-04-12T05:09:11.651 INFO:tasks.workunit.client.0.smithi113.stdout:9/861: dwrite d1/d4/f77 [0,4194304] 0 2024-04-12T05:09:11.651 INFO:tasks.workunit.client.1.smithi192.stdout:6/790: mknod d3/d8/d22/dd4/c107 0 2024-04-12T05:09:11.652 INFO:tasks.workunit.client.1.smithi192.stdout:2/768: dwrite d1/d8/d3a/f2e [0,4194304] 0 2024-04-12T05:09:11.655 INFO:tasks.workunit.client.0.smithi113.stdout:0/929: read d1/d6/f5d [8641,18732] 0 2024-04-12T05:09:11.657 INFO:tasks.workunit.client.0.smithi113.stdout:0/930: chown d1/d43/d59/db2/ldf 4 1 2024-04-12T05:09:11.657 INFO:tasks.workunit.client.1.smithi192.stdout:5/846: creat d0/d21/d5e/f11f x:0 0 0 2024-04-12T05:09:11.658 INFO:tasks.workunit.client.1.smithi192.stdout:2/769: readlink d1/d8/d17/d19/d60/d59/ld4 0 2024-04-12T05:09:11.659 INFO:tasks.workunit.client.0.smithi113.stdout:3/825: dread d0/d13/d47/d4f/f4c [4194304,4194304] 0 2024-04-12T05:09:11.665 INFO:tasks.workunit.client.1.smithi192.stdout:6/791: mknod d3/d8/d56/d59/ddf/df1/c108 0 2024-04-12T05:09:11.665 INFO:tasks.workunit.client.0.smithi113.stdout:3/826: creat d0/d13/d47/d94/dac/f114 x:0 0 0 2024-04-12T05:09:11.665 INFO:tasks.workunit.client.0.smithi113.stdout:0/931: mknod d1/d6/d8e/c11f 0 2024-04-12T05:09:11.666 INFO:tasks.workunit.client.1.smithi192.stdout:6/792: chown d3/d21/d24/dcd 0 1 2024-04-12T05:09:11.669 INFO:tasks.workunit.client.1.smithi192.stdout:5/847: mkdir d0/d15/d3e/d120 0 2024-04-12T05:09:11.669 INFO:tasks.workunit.client.0.smithi113.stdout:0/932: creat d1/d6/d16/d8f/f120 x:0 0 0 2024-04-12T05:09:11.670 INFO:tasks.workunit.client.0.smithi113.stdout:3/827: rmdir d0/d13/d47/d94/dcf 39 2024-04-12T05:09:11.670 INFO:tasks.workunit.client.1.smithi192.stdout:5/848: truncate d0/fcf 1245136 0 2024-04-12T05:09:11.672 INFO:tasks.workunit.client.1.smithi192.stdout:6/793: read d3/d23/db5/fcb [1597070,98220] 0 2024-04-12T05:09:11.672 INFO:tasks.workunit.client.1.smithi192.stdout:2/770: symlink d1/d8/d3a/de/lf4 0 2024-04-12T05:09:11.673 INFO:tasks.workunit.client.0.smithi113.stdout:9/862: dwrite d1/d4/d38/d65/fcf [0,4194304] 0 2024-04-12T05:09:11.675 INFO:tasks.workunit.client.1.smithi192.stdout:1/794: rename d0/d2/d1e/d3f/df2/d21/d33 to d0/d2/d1e/d3f/df2/d3c/d5b/d74/d90/d108 0 2024-04-12T05:09:11.675 INFO:tasks.workunit.client.0.smithi113.stdout:3/828: unlink d0/d13/f6b 0 2024-04-12T05:09:11.677 INFO:tasks.workunit.client.1.smithi192.stdout:5/849: dread d0/d15/d32/d45/fb1 [0,4194304] 0 2024-04-12T05:09:11.678 INFO:tasks.workunit.client.0.smithi113.stdout:9/863: write d1/d27/f95 [1400732,19460] 0 2024-04-12T05:09:11.680 INFO:tasks.workunit.client.0.smithi113.stdout:9/864: chown d1/d4/d87/fc9 25362129 1 2024-04-12T05:09:11.680 INFO:tasks.workunit.client.1.smithi192.stdout:1/795: dread - d0/d1b/d44/ff8 zero size 2024-04-12T05:09:11.685 INFO:tasks.workunit.client.0.smithi113.stdout:9/865: unlink d1/df/db6/f84 0 2024-04-12T05:09:11.687 INFO:tasks.workunit.client.1.smithi192.stdout:5/850: creat d0/d15/d3e/dca/dd3/f121 x:0 0 0 2024-04-12T05:09:11.688 INFO:tasks.workunit.client.1.smithi192.stdout:5/851: fsync d0/d15/d24/ff8 0 2024-04-12T05:09:11.688 INFO:tasks.workunit.client.0.smithi113.stdout:9/866: creat d1/d4/df9/f121 x:0 0 0 2024-04-12T05:09:11.689 INFO:tasks.workunit.client.1.smithi192.stdout:2/771: fdatasync d1/d8/d17/f5f 0 2024-04-12T05:09:11.689 INFO:tasks.workunit.client.0.smithi113.stdout:3/829: write d0/d13/d15/d28/d2a/f2b [674319,94163] 0 2024-04-12T05:09:11.690 INFO:tasks.workunit.client.0.smithi113.stdout:0/933: getdents d1/d6/d34/dc3/d87 0 2024-04-12T05:09:11.693 INFO:tasks.workunit.client.0.smithi113.stdout:3/830: read d0/d13/d47/d4f/f35 [4442779,108672] 0 2024-04-12T05:09:11.696 INFO:tasks.workunit.client.0.smithi113.stdout:3/831: creat d0/d13/d15/d28/d2a/d8f/ddd/f115 x:0 0 0 2024-04-12T05:09:11.696 INFO:tasks.workunit.client.0.smithi113.stdout:9/867: dread d1/df/f4d [0,4194304] 0 2024-04-12T05:09:11.697 INFO:tasks.workunit.client.0.smithi113.stdout:0/934: rename d1/d6/d34/c83 to d1/d43/d59/de9/d105/d50/db6/c121 0 2024-04-12T05:09:11.699 INFO:tasks.workunit.client.0.smithi113.stdout:9/868: symlink d1/d27/d7a/d11b/l122 0 2024-04-12T05:09:11.700 INFO:tasks.workunit.client.0.smithi113.stdout:0/935: chown d1/d43/d59/de9/d105/d50/d52/l55 1568636 1 2024-04-12T05:09:11.701 INFO:tasks.workunit.client.0.smithi113.stdout:9/869: creat d1/d4/d38/d65/dcd/dd7/f123 x:0 0 0 2024-04-12T05:09:11.704 INFO:tasks.workunit.client.0.smithi113.stdout:9/870: creat d1/df/d29/de8/d104/f124 x:0 0 0 2024-04-12T05:09:11.717 INFO:tasks.workunit.client.0.smithi113.stdout:9/871: write d1/df/d29/f7b [278364,6151] 0 2024-04-12T05:09:11.719 INFO:tasks.workunit.client.0.smithi113.stdout:5/965: sync 2024-04-12T05:09:11.720 INFO:tasks.workunit.client.0.smithi113.stdout:2/815: sync 2024-04-12T05:09:11.720 INFO:tasks.workunit.client.0.smithi113.stdout:3/832: dread d0/f12 [4194304,4194304] 0 2024-04-12T05:09:11.720 INFO:tasks.workunit.client.0.smithi113.stdout:7/800: sync 2024-04-12T05:09:11.720 INFO:tasks.workunit.client.0.smithi113.stdout:4/826: sync 2024-04-12T05:09:11.723 INFO:tasks.workunit.client.0.smithi113.stdout:9/872: rename d1/d4/d5/d17/la1 to d1/d4/d87/l125 0 2024-04-12T05:09:11.724 INFO:tasks.workunit.client.0.smithi113.stdout:1/921: link d10/d41/d89/d96/dd3/f7a d10/d2e/f125 0 2024-04-12T05:09:11.725 INFO:tasks.workunit.client.0.smithi113.stdout:4/827: rmdir da/d70 39 2024-04-12T05:09:11.726 INFO:tasks.workunit.client.0.smithi113.stdout:3/833: truncate d0/f12 1747988 0 2024-04-12T05:09:11.730 INFO:tasks.workunit.client.1.smithi192.stdout:3/728: write d5/d13/d3c/f31 [977416,61] 0 2024-04-12T05:09:11.730 INFO:tasks.workunit.client.1.smithi192.stdout:3/729: stat d5/c8 0 2024-04-12T05:09:11.734 INFO:tasks.workunit.client.1.smithi192.stdout:7/675: dwrite d3/d1d/d4a/d5e/d6d/da3/fab [0,4194304] 0 2024-04-12T05:09:11.760 INFO:tasks.workunit.client.1.smithi192.stdout:5/852: link d0/l7a d0/d15/d3e/dca/dda/db5/dd7/de1/l122 0 2024-04-12T05:09:11.765 INFO:tasks.workunit.client.1.smithi192.stdout:7/676: rename d3/d11/f5c to d3/d1d/d5a/d5f/d7a/da6/fe7 0 2024-04-12T05:09:11.766 INFO:tasks.workunit.client.1.smithi192.stdout:3/730: getdents d5/d13/d3c/d26/dca 0 2024-04-12T05:09:11.766 INFO:tasks.workunit.client.1.smithi192.stdout:7/677: dread - d3/d7/d15/d1a/ddc/fe5 zero size 2024-04-12T05:09:11.771 INFO:tasks.workunit.client.1.smithi192.stdout:5/853: getdents d0/d15/d3e/dca/dda/db5 0 2024-04-12T05:09:11.773 INFO:tasks.workunit.client.1.smithi192.stdout:5/854: truncate d0/d15/d3e/dca/f109 254542 0 2024-04-12T05:09:11.773 INFO:tasks.workunit.client.1.smithi192.stdout:6/794: write d3/d8/d10/d4c/f82 [445776,52887] 0 2024-04-12T05:09:11.774 INFO:tasks.workunit.client.1.smithi192.stdout:7/678: getdents d3/d1d/d5a 0 2024-04-12T05:09:11.774 INFO:tasks.workunit.client.1.smithi192.stdout:6/795: stat d3/dc/cdc 0 2024-04-12T05:09:11.777 INFO:tasks.workunit.client.1.smithi192.stdout:2/772: write d1/d8/d17/d19/d60/d9c/f9d [141355,77158] 0 2024-04-12T05:09:11.777 INFO:tasks.workunit.client.1.smithi192.stdout:7/679: stat d3/d7/d15/cb1 0 2024-04-12T05:09:11.780 INFO:tasks.workunit.client.1.smithi192.stdout:5/855: mknod d0/d15/d3e/dca/dda/deb/d113/c123 0 2024-04-12T05:09:11.781 INFO:tasks.workunit.client.1.smithi192.stdout:6/796: mkdir d3/d8/d56/d59/d97/dbf/de8/d109 0 2024-04-12T05:09:11.782 INFO:tasks.workunit.client.1.smithi192.stdout:1/796: dwrite d0/d1b/d44/d69/ffb [0,4194304] 0 2024-04-12T05:09:11.784 INFO:tasks.workunit.client.1.smithi192.stdout:7/680: mkdir d3/d11/de8 0 2024-04-12T05:09:11.784 INFO:tasks.workunit.client.1.smithi192.stdout:6/797: write d3/d8/de6/fec [895521,39322] 0 2024-04-12T05:09:11.785 INFO:tasks.workunit.client.1.smithi192.stdout:3/731: dread d5/d13/d3c/d26/fc5 [0,4194304] 0 2024-04-12T05:09:11.785 INFO:tasks.workunit.client.1.smithi192.stdout:2/773: rename d1/d8/d3a/de/le1 to d1/d8/de2/lf5 0 2024-04-12T05:09:11.786 INFO:tasks.workunit.client.1.smithi192.stdout:5/856: rmdir d0/d15/d3e/dca/dda/db5/dd7/d116 39 2024-04-12T05:09:11.788 INFO:tasks.workunit.client.1.smithi192.stdout:3/732: read d5/d13/f2f [2092381,38602] 0 2024-04-12T05:09:11.788 INFO:tasks.workunit.client.1.smithi192.stdout:8/736: sync 2024-04-12T05:09:11.789 INFO:tasks.workunit.client.1.smithi192.stdout:6/798: read - d3/d23/d67/fb8 zero size 2024-04-12T05:09:11.789 INFO:tasks.workunit.client.1.smithi192.stdout:3/733: chown d5/d13/d3c/f31 51 1 2024-04-12T05:09:11.790 INFO:tasks.workunit.client.1.smithi192.stdout:8/737: read d6/d10/d46/d5e/d5f/d31/f36 [478408,1593] 0 2024-04-12T05:09:11.791 INFO:tasks.workunit.client.1.smithi192.stdout:1/797: creat d0/d2/d1e/d3f/df2/d3c/d106/f109 x:0 0 0 2024-04-12T05:09:11.792 INFO:tasks.workunit.client.1.smithi192.stdout:7/681: mkdir d3/d1d/d5a/d5f/d6a/de9 0 2024-04-12T05:09:11.793 INFO:tasks.workunit.client.1.smithi192.stdout:5/857: symlink d0/d15/d32/d45/d106/l124 0 2024-04-12T05:09:11.794 INFO:tasks.workunit.client.1.smithi192.stdout:7/682: fsync d3/d1d/d5a/d5f/fd2 0 2024-04-12T05:09:11.794 INFO:tasks.workunit.client.1.smithi192.stdout:5/858: chown d0/d15 21455074 1 2024-04-12T05:09:11.796 INFO:tasks.workunit.client.1.smithi192.stdout:7/683: stat d3/d1d/d4a/d5e/d6d/da3/dce 0 2024-04-12T05:09:11.799 INFO:tasks.workunit.client.1.smithi192.stdout:1/798: rename d0/d2/d1e/d3f/df2/d21/d62/dbc/cc1 to d0/d2/d1e/d3f/df2/d3c/d5b/d74/d90/c10a 0 2024-04-12T05:09:11.800 INFO:tasks.workunit.client.1.smithi192.stdout:3/734: symlink d5/df1/l106 0 2024-04-12T05:09:11.804 INFO:tasks.workunit.client.1.smithi192.stdout:1/799: creat d0/d29/d8d/d96/f10b x:0 0 0 2024-04-12T05:09:11.805 INFO:tasks.workunit.client.1.smithi192.stdout:3/735: creat d5/d13/d1a/d5f/d103/f107 x:0 0 0 2024-04-12T05:09:11.807 INFO:tasks.workunit.client.1.smithi192.stdout:7/684: getdents d3/d1d/d5a 0 2024-04-12T05:09:11.808 INFO:tasks.workunit.client.1.smithi192.stdout:8/738: link d6/d10/d46/d5e/d5f/d37/d4f/fc6 d6/d10/d46/d5e/d5f/d37/d4f/d4c/fdf 0 2024-04-12T05:09:11.809 INFO:tasks.workunit.client.1.smithi192.stdout:3/736: mknod d5/d13/d3a/d70/d8c/d8d/dbc/c108 0 2024-04-12T05:09:11.810 INFO:tasks.workunit.client.1.smithi192.stdout:2/774: dread d1/d8/d17/d19/f5e [0,4194304] 0 2024-04-12T05:09:11.810 INFO:tasks.workunit.client.1.smithi192.stdout:7/685: mknod d3/d1d/d5a/d5f/d7a/da6/cea 0 2024-04-12T05:09:11.812 INFO:tasks.workunit.client.1.smithi192.stdout:5/859: dread d0/d15/d24/d47/fdd [0,4194304] 0 2024-04-12T05:09:11.812 INFO:tasks.workunit.client.1.smithi192.stdout:3/737: chown d5/d13/d3a/d8b/ff6 1 1 2024-04-12T05:09:11.813 INFO:tasks.workunit.client.1.smithi192.stdout:1/800: truncate d0/d2/d1e/d3f/df2/d21/d62/d84/f91 6730640 0 2024-04-12T05:09:11.814 INFO:tasks.workunit.client.1.smithi192.stdout:8/739: mknod d6/d10/d46/d5e/d5f/d31/ce0 0 2024-04-12T05:09:11.815 INFO:tasks.workunit.client.1.smithi192.stdout:3/738: write d5/d13/d3c/d84/f9d [555004,89116] 0 2024-04-12T05:09:11.816 INFO:tasks.workunit.client.1.smithi192.stdout:5/860: truncate d0/d21/d5e/f11f 61464 0 2024-04-12T05:09:11.816 INFO:tasks.workunit.client.1.smithi192.stdout:2/775: rmdir d1/d8/d17/d19/d60/d59/d68 39 2024-04-12T05:09:11.819 INFO:tasks.workunit.client.1.smithi192.stdout:8/740: rename d6/d10/d46/d5e/d5f/d38 to d6/d10/d46/d5e/d5f/d38/dbc/de1 22 2024-04-12T05:09:11.820 INFO:tasks.workunit.client.1.smithi192.stdout:6/799: write d3/f1e [804908,24530] 0 2024-04-12T05:09:11.823 INFO:tasks.workunit.client.1.smithi192.stdout:3/739: dread d5/d13/d3a/d70/d8c/d8d/fe4 [0,4194304] 0 2024-04-12T05:09:11.826 INFO:tasks.workunit.client.1.smithi192.stdout:7/686: write d3/d1d/d4a/d5e/fa5 [954747,66504] 0 2024-04-12T05:09:11.826 INFO:tasks.workunit.client.1.smithi192.stdout:6/800: read d3/d8/f92 [3151934,11374] 0 2024-04-12T05:09:11.828 INFO:tasks.workunit.client.1.smithi192.stdout:1/801: dwrite d0/d2/d1e/d3f/d56/f79 [0,4194304] 0 2024-04-12T05:09:11.830 INFO:tasks.workunit.client.1.smithi192.stdout:1/802: stat d0/d1b/f53 0 2024-04-12T05:09:11.830 INFO:tasks.workunit.client.1.smithi192.stdout:3/740: mknod d5/d13/d3a/d70/c109 0 2024-04-12T05:09:11.831 INFO:tasks.workunit.client.1.smithi192.stdout:7/687: symlink d3/d1d/d4a/d5e/d6d/da3/leb 0 2024-04-12T05:09:11.832 INFO:tasks.workunit.client.1.smithi192.stdout:8/741: truncate d6/d10/d46/d5e/d5f/f16 1032905 0 2024-04-12T05:09:11.835 INFO:tasks.workunit.client.1.smithi192.stdout:1/803: creat d0/dd0/f10c x:0 0 0 2024-04-12T05:09:11.836 INFO:tasks.workunit.client.1.smithi192.stdout:8/742: symlink d6/da/le2 0 2024-04-12T05:09:11.836 INFO:tasks.workunit.client.1.smithi192.stdout:5/861: dread d0/d15/d32/d69/f9e [0,4194304] 0 2024-04-12T05:09:11.837 INFO:tasks.workunit.client.1.smithi192.stdout:3/741: creat d5/d13/d3a/d4a/d61/f10a x:0 0 0 2024-04-12T05:09:11.838 INFO:tasks.workunit.client.1.smithi192.stdout:6/801: dwrite d3/f43 [0,4194304] 0 2024-04-12T05:09:11.839 INFO:tasks.workunit.client.1.smithi192.stdout:8/743: truncate f3 1790425 0 2024-04-12T05:09:11.842 INFO:tasks.workunit.client.1.smithi192.stdout:6/802: creat d3/d8/d56/d59/d8a/d99/f10a x:0 0 0 2024-04-12T05:09:11.843 INFO:tasks.workunit.client.1.smithi192.stdout:1/804: creat d0/d2/d1e/d3f/df2/d3c/d5b/d7c/df7/f10d x:0 0 0 2024-04-12T05:09:11.844 INFO:tasks.workunit.client.1.smithi192.stdout:5/862: creat d0/d15/d3e/dca/dda/db5/dd7/d116/f125 x:0 0 0 2024-04-12T05:09:11.845 INFO:tasks.workunit.client.1.smithi192.stdout:6/803: creat d3/d8/de6/f10b x:0 0 0 2024-04-12T05:09:11.845 INFO:tasks.workunit.client.1.smithi192.stdout:1/805: stat d0/d2/d1e/d3f/df2/d3c/dab 0 2024-04-12T05:09:11.848 INFO:tasks.workunit.client.1.smithi192.stdout:6/804: write d3/d8/d22/f65 [3414554,14065] 0 2024-04-12T05:09:11.849 INFO:tasks.workunit.client.1.smithi192.stdout:1/806: write d0/d2/d1e/d3f/df2/d3c/d5b/d7c/df7/f10d [335426,34825] 0 2024-04-12T05:09:11.852 INFO:tasks.workunit.client.1.smithi192.stdout:6/805: getdents d3/d8/d48/d85 0 2024-04-12T05:09:11.854 INFO:tasks.workunit.client.1.smithi192.stdout:1/807: rename d0/d2/d1e/d3f/df2/f4b to d0/d2/d1e/d3f/df2/d3c/d5b/db6/f10e 0 2024-04-12T05:09:11.862 INFO:tasks.workunit.client.1.smithi192.stdout:1/808: dwrite d0/d2/d1e/d3f/d56/d9e/fbb [4194304,4194304] 0 2024-04-12T05:09:11.864 INFO:tasks.workunit.client.1.smithi192.stdout:8/744: truncate d6/d10/d46/d5e/d5f/f7e 1783819 0 2024-04-12T05:09:11.865 INFO:tasks.workunit.client.1.smithi192.stdout:5/863: truncate d0/d15/d32/d45/fb1 3877800 0 2024-04-12T05:09:11.866 INFO:tasks.workunit.client.1.smithi192.stdout:6/806: write d3/d8/d10/d4c/d68/f77 [3705532,124627] 0 2024-04-12T05:09:11.868 INFO:tasks.workunit.client.0.smithi113.stdout:7/801: rename d0/dc/d6/d9b/d80/f96 to d0/dc/da9/d5c/d9a/db6/d101/f10f 0 2024-04-12T05:09:11.871 INFO:tasks.workunit.client.0.smithi113.stdout:9/873: symlink d1/df/d29/de8/dc7/l126 0 2024-04-12T05:09:11.871 INFO:tasks.workunit.client.0.smithi113.stdout:1/922: symlink d10/d14/l126 0 2024-04-12T05:09:11.873 INFO:tasks.workunit.client.1.smithi192.stdout:3/742: dwrite d5/d13/d1a/d53/f58 [0,4194304] 0 2024-04-12T05:09:11.873 INFO:tasks.workunit.client.1.smithi192.stdout:8/745: dread - d6/d10/d46/d5e/d5f/d31/d80/fc1 zero size 2024-04-12T05:09:11.873 INFO:tasks.workunit.client.1.smithi192.stdout:6/807: write d3/dc/d39/f4e [3909043,58258] 0 2024-04-12T05:09:11.873 INFO:tasks.workunit.client.1.smithi192.stdout:5/864: chown d0/d15/d3e/dca/dda/db5/dbe/f11d 37656402 1 2024-04-12T05:09:11.874 INFO:tasks.workunit.client.1.smithi192.stdout:8/746: chown d6/d10/d46/d5e 23 1 2024-04-12T05:09:11.874 INFO:tasks.workunit.client.1.smithi192.stdout:1/809: rename d0/d2/d1e/d3f/d56/d9e/l80 to d0/d1b/l10f 0 2024-04-12T05:09:11.874 INFO:tasks.workunit.client.0.smithi113.stdout:2/816: getdents d8/d9/d27 0 2024-04-12T05:09:11.877 INFO:tasks.workunit.client.0.smithi113.stdout:3/834: creat d0/d13/d47/d94/dcf/f116 x:0 0 0 2024-04-12T05:09:11.878 INFO:tasks.workunit.client.1.smithi192.stdout:8/747: write d6/d10/d46/d5e/d5f/f1c [1119195,114568] 0 2024-04-12T05:09:11.878 INFO:tasks.workunit.client.1.smithi192.stdout:3/743: mknod d5/d13/d1a/d53/dbb/c10b 0 2024-04-12T05:09:11.881 INFO:tasks.workunit.client.0.smithi113.stdout:9/874: mkdir d1/d4/d38/d127 0 2024-04-12T05:09:11.882 INFO:tasks.workunit.client.1.smithi192.stdout:5/865: creat d0/d15/d32/d45/f126 x:0 0 0 2024-04-12T05:09:11.882 INFO:tasks.workunit.client.1.smithi192.stdout:6/808: truncate d3/d8/d48/f6f 2833156 0 2024-04-12T05:09:11.883 INFO:tasks.workunit.client.1.smithi192.stdout:1/810: truncate d0/d2/d1e/f4a 1048079 0 2024-04-12T05:09:11.885 INFO:tasks.workunit.client.1.smithi192.stdout:3/744: rename d5/f104 to d5/d13/d3a/d8b/da2/de5/f10c 0 2024-04-12T05:09:11.887 INFO:tasks.workunit.client.1.smithi192.stdout:6/809: mknod d3/d21/d24/dd7/c10c 0 2024-04-12T05:09:11.888 INFO:tasks.workunit.client.0.smithi113.stdout:4/828: rename da/fb to da/d24/d7e/d79/dcd/f118 0 2024-04-12T05:09:11.888 INFO:tasks.workunit.client.0.smithi113.stdout:3/835: dread - d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/fdc zero size 2024-04-12T05:09:11.889 INFO:tasks.workunit.client.0.smithi113.stdout:5/966: dwrite d0/d10/d27/d5f/f143 [4194304,4194304] 0 2024-04-12T05:09:11.889 INFO:tasks.workunit.client.1.smithi192.stdout:1/811: dread d0/d2/d1e/d3f/df2/d3c/d5b/d7c/df7/f10d [0,4194304] 0 2024-04-12T05:09:11.891 INFO:tasks.workunit.client.0.smithi113.stdout:9/875: readlink d1/df/d29/de8/dc7/lf5 0 2024-04-12T05:09:11.895 INFO:tasks.workunit.client.1.smithi192.stdout:3/745: dwrite d5/d13/d94/fd5 [0,4194304] 0 2024-04-12T05:09:11.896 INFO:tasks.workunit.client.0.smithi113.stdout:4/829: read da/d3c/d43/f75 [155341,103364] 0 2024-04-12T05:09:11.898 INFO:tasks.workunit.client.1.smithi192.stdout:3/746: dread - d5/d13/d3a/d4a/d61/f10a zero size 2024-04-12T05:09:11.898 INFO:tasks.workunit.client.0.smithi113.stdout:2/817: dwrite d8/d9/d27/d88/fab [0,4194304] 0 2024-04-12T05:09:11.901 INFO:tasks.workunit.client.0.smithi113.stdout:4/830: write da/fef [633386,40556] 0 2024-04-12T05:09:11.902 INFO:tasks.workunit.client.0.smithi113.stdout:7/802: dwrite d0/dc/da9/d5c/d9a/db6/fcf [0,4194304] 0 2024-04-12T05:09:11.902 INFO:tasks.workunit.client.1.smithi192.stdout:6/810: rename d3/d8/d10/c1d to d3/d8/d56/d59/c10d 0 2024-04-12T05:09:11.904 INFO:tasks.workunit.client.1.smithi192.stdout:3/747: dread d5/d13/d3a/d70/d8c/fb4 [0,4194304] 0 2024-04-12T05:09:11.908 INFO:tasks.workunit.client.0.smithi113.stdout:5/967: symlink d0/d10/d27/d1d/d41/d115/l148 0 2024-04-12T05:09:11.908 INFO:tasks.workunit.client.0.smithi113.stdout:9/876: creat d1/df/d4b/df7/d111/f128 x:0 0 0 2024-04-12T05:09:11.909 INFO:tasks.workunit.client.0.smithi113.stdout:1/923: write d10/d2e/d7c/f92 [207170,33673] 0 2024-04-12T05:09:11.911 INFO:tasks.workunit.client.1.smithi192.stdout:3/748: unlink d5/d13/d3a/d8b/ff6 0 2024-04-12T05:09:11.911 INFO:tasks.workunit.client.0.smithi113.stdout:2/818: mkdir d8/d9/d27/d88/de0/dfe 0 2024-04-12T05:09:11.914 INFO:tasks.workunit.client.0.smithi113.stdout:5/968: read d0/d10/f16 [172877,3662] 0 2024-04-12T05:09:11.917 INFO:tasks.workunit.client.0.smithi113.stdout:5/969: fdatasync d0/d10/d27/d1d/d8c/db8/f12a 0 2024-04-12T05:09:11.919 INFO:tasks.workunit.client.1.smithi192.stdout:6/811: mkdir d3/d5/de7/d10e 0 2024-04-12T05:09:11.920 INFO:tasks.workunit.client.1.smithi192.stdout:5/866: truncate d0/d15/d32/db2/fd2 4070160 0 2024-04-12T05:09:11.920 INFO:tasks.workunit.client.0.smithi113.stdout:4/831: symlink da/d3c/d43/dbc/l119 0 2024-04-12T05:09:11.920 INFO:tasks.workunit.client.0.smithi113.stdout:9/877: creat d1/d116/f129 x:0 0 0 2024-04-12T05:09:11.921 INFO:tasks.workunit.client.0.smithi113.stdout:1/924: mknod d10/d41/d89/df2/c127 0 2024-04-12T05:09:11.922 INFO:tasks.workunit.client.1.smithi192.stdout:6/812: read - d3/d21/d24/ff3 zero size 2024-04-12T05:09:11.923 INFO:tasks.workunit.client.0.smithi113.stdout:2/819: link d8/fc0 d8/d9/de/d12/d2b/ddd/fff 0 2024-04-12T05:09:11.924 INFO:tasks.workunit.client.0.smithi113.stdout:7/803: dwrite d0/dd/d95/dd4/f107 [0,4194304] 0 2024-04-12T05:09:11.924 INFO:tasks.workunit.client.0.smithi113.stdout:9/878: dread - d1/d27/d7a/de4/db7/ffb zero size 2024-04-12T05:09:11.924 INFO:tasks.workunit.client.1.smithi192.stdout:8/748: dread d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d98/fd1 [4194304,4194304] 0 2024-04-12T05:09:11.925 INFO:tasks.workunit.client.0.smithi113.stdout:4/832: fsync da/dd/d1b/f2c 0 2024-04-12T05:09:11.927 INFO:tasks.workunit.client.0.smithi113.stdout:2/820: write d8/d9/de/f31 [5547550,117230] 0 2024-04-12T05:09:11.928 INFO:tasks.workunit.client.0.smithi113.stdout:8/859: sync 2024-04-12T05:09:11.929 INFO:tasks.workunit.client.1.smithi192.stdout:3/749: rename d5/d13/d1a/fef to d5/d13/d3a/d70/d8c/f10d 0 2024-04-12T05:09:11.930 INFO:tasks.workunit.client.0.smithi113.stdout:1/925: mkdir d10/d21/d128 0 2024-04-12T05:09:11.930 INFO:tasks.workunit.client.0.smithi113.stdout:4/833: creat da/d3c/d83/d47/d10f/d90/f11a x:0 0 0 2024-04-12T05:09:11.931 INFO:tasks.workunit.client.1.smithi192.stdout:6/813: symlink d3/d8/d56/d59/d97/dbf/de8/d109/l10f 0 2024-04-12T05:09:11.934 INFO:tasks.workunit.client.0.smithi113.stdout:4/834: chown da/d3c/d9a/dc1/d106/dc8 1885659 1 2024-04-12T05:09:11.934 INFO:tasks.workunit.client.0.smithi113.stdout:9/879: dwrite d1/df/d29/de8/d104/f124 [0,4194304] 0 2024-04-12T05:09:11.936 INFO:tasks.workunit.client.0.smithi113.stdout:8/860: mknod d9/d16/d7f/d3a/d6c/d90/c115 0 2024-04-12T05:09:11.936 INFO:tasks.workunit.client.1.smithi192.stdout:5/867: rename d0/d15/d3e/dca/db9 to d0/d15/d3e/dca/dda/deb/d113/d127 0 2024-04-12T05:09:11.937 INFO:tasks.workunit.client.0.smithi113.stdout:7/804: mknod d0/dd/d11/c110 0 2024-04-12T05:09:11.941 INFO:tasks.workunit.client.0.smithi113.stdout:1/926: creat d10/d41/d89/f129 x:0 0 0 2024-04-12T05:09:11.941 INFO:tasks.workunit.client.0.smithi113.stdout:8/861: rmdir d9/d1b/d8e 39 2024-04-12T05:09:11.941 INFO:tasks.workunit.client.1.smithi192.stdout:8/749: mkdir d6/d10/d46/d5e/d5f/d31/d61/de3 0 2024-04-12T05:09:11.942 INFO:tasks.workunit.client.1.smithi192.stdout:6/814: mknod d3/d8/d48/d85/c110 0 2024-04-12T05:09:11.943 INFO:tasks.workunit.client.1.smithi192.stdout:8/750: chown d6/d10/d46/d5e/d5f/d37/d4f/d51/f66 0 1 2024-04-12T05:09:11.945 INFO:tasks.workunit.client.1.smithi192.stdout:8/751: write d6/d1f/f5c [949645,69028] 0 2024-04-12T05:09:11.946 INFO:tasks.workunit.client.0.smithi113.stdout:7/805: creat d0/dc/da9/d5c/d9a/f111 x:0 0 0 2024-04-12T05:09:11.946 INFO:tasks.workunit.client.1.smithi192.stdout:4/818: sync 2024-04-12T05:09:11.946 INFO:tasks.workunit.client.1.smithi192.stdout:0/887: sync 2024-04-12T05:09:11.946 INFO:tasks.workunit.client.1.smithi192.stdout:9/667: sync 2024-04-12T05:09:11.947 INFO:tasks.workunit.client.0.smithi113.stdout:1/927: rmdir d10/d14/d95/dcf 39 2024-04-12T05:09:11.947 INFO:tasks.workunit.client.0.smithi113.stdout:4/835: dwrite da/d3c/d9a/dc1/d106/f62 [0,4194304] 0 2024-04-12T05:09:11.949 INFO:tasks.workunit.client.0.smithi113.stdout:1/928: chown d10/d41/d89/d96/l5a 28949161 1 2024-04-12T05:09:11.949 INFO:tasks.workunit.client.0.smithi113.stdout:4/836: write da/d24/d7b/fdb [2967028,71023] 0 2024-04-12T05:09:11.950 INFO:tasks.workunit.client.0.smithi113.stdout:2/821: getdents d8/d9/de/d12/dbf 0 2024-04-12T05:09:11.951 INFO:tasks.workunit.client.0.smithi113.stdout:1/929: stat d10/d14/l126 0 2024-04-12T05:09:11.951 INFO:tasks.workunit.client.1.smithi192.stdout:6/815: stat d3/d8/d10/d4c/c8c 0 2024-04-12T05:09:11.952 INFO:tasks.workunit.client.1.smithi192.stdout:8/752: symlink d6/d10/d46/d5e/d6e/le4 0 2024-04-12T05:09:11.954 INFO:tasks.workunit.client.1.smithi192.stdout:6/816: chown d3/d8/d56/d59/d97/cc6 195 1 2024-04-12T05:09:11.957 INFO:tasks.workunit.client.0.smithi113.stdout:4/837: write da/d3c/d43/f7d [3383915,120472] 0 2024-04-12T05:09:11.959 INFO:tasks.workunit.client.1.smithi192.stdout:4/819: readlink d1/d54/d82/d9c/lbe 0 2024-04-12T05:09:11.960 INFO:tasks.workunit.client.0.smithi113.stdout:2/822: write d8/d9/de/d12/d2b/daa/db3/fc1 [599558,83688] 0 2024-04-12T05:09:11.960 INFO:tasks.workunit.client.1.smithi192.stdout:0/888: creat d0/d7b/d129/f12d x:0 0 0 2024-04-12T05:09:11.960 INFO:tasks.workunit.client.1.smithi192.stdout:4/820: chown d1/d54/d55/d64/f86 1 1 2024-04-12T05:09:11.962 INFO:tasks.workunit.client.0.smithi113.stdout:8/862: getdents d9/d16/d7f 0 2024-04-12T05:09:11.962 INFO:tasks.workunit.client.1.smithi192.stdout:8/753: creat d6/d10/d46/d5e/d5f/d31/d80/db1/fe5 x:0 0 0 2024-04-12T05:09:11.964 INFO:tasks.workunit.client.0.smithi113.stdout:7/806: getdents d0/d52 0 2024-04-12T05:09:11.965 INFO:tasks.workunit.client.1.smithi192.stdout:6/817: rename d3/d8/lf6 to d3/d8/d56/d59/d97/dbf/de8/l111 0 2024-04-12T05:09:11.965 INFO:tasks.workunit.client.0.smithi113.stdout:9/880: dread d1/d4/d5/d6/fbd [0,4194304] 0 2024-04-12T05:09:11.965 INFO:tasks.workunit.client.0.smithi113.stdout:2/823: mknod d8/d9/de/d12/d3c/d8a/c100 0 2024-04-12T05:09:11.967 INFO:tasks.workunit.client.1.smithi192.stdout:0/889: rmdir d0/d7b/d9b/d35/d109/d9f 39 2024-04-12T05:09:11.968 INFO:tasks.workunit.client.0.smithi113.stdout:4/838: getdents da/d24/d7e 0 2024-04-12T05:09:11.971 INFO:tasks.workunit.client.1.smithi192.stdout:6/818: dread d3/d21/d24/f4f [0,4194304] 0 2024-04-12T05:09:11.972 INFO:tasks.workunit.client.1.smithi192.stdout:3/750: write d5/d13/d3a/d70/d8c/fb3 [764335,43811] 0 2024-04-12T05:09:11.972 INFO:tasks.workunit.client.0.smithi113.stdout:7/807: creat d0/dd/f112 x:0 0 0 2024-04-12T05:09:11.973 INFO:tasks.workunit.client.1.smithi192.stdout:6/819: readlink d3/d5/l3f 0 2024-04-12T05:09:11.973 INFO:tasks.workunit.client.1.smithi192.stdout:9/668: dread d11/d25/d36/d6e/f3e [0,4194304] 0 2024-04-12T05:09:11.974 INFO:tasks.workunit.client.1.smithi192.stdout:8/754: symlink d6/d10/d46/d5e/d5f/d38/dbc/le6 0 2024-04-12T05:09:11.976 INFO:tasks.workunit.client.1.smithi192.stdout:8/755: stat d6/d1f/d5d/fd3 0 2024-04-12T05:09:11.977 INFO:tasks.workunit.client.0.smithi113.stdout:4/839: rename da/d3c/da5/lfa to da/d4a/l11b 0 2024-04-12T05:09:11.977 INFO:tasks.workunit.client.0.smithi113.stdout:8/863: dwrite d9/d16/d65/db1/d59/da3/f105 [0,4194304] 0 2024-04-12T05:09:11.978 INFO:tasks.workunit.client.0.smithi113.stdout:7/808: dread - d0/dc/d6/d24/fe1 zero size 2024-04-12T05:09:11.979 INFO:tasks.workunit.client.1.smithi192.stdout:3/751: mkdir d5/d13/d3a/d70/d8c/de1/d10e 0 2024-04-12T05:09:11.981 INFO:tasks.workunit.client.0.smithi113.stdout:8/864: truncate d9/d16/d65/d8c/fb5 2337550 0 2024-04-12T05:09:11.981 INFO:tasks.workunit.client.0.smithi113.stdout:4/840: fdatasync da/d3c/d43/f75 0 2024-04-12T05:09:11.984 INFO:tasks.workunit.client.0.smithi113.stdout:9/881: link d1/df/d4b/d63/dad/ff2 d1/d4/d38/d65/dcd/dd7/f12a 0 2024-04-12T05:09:11.985 INFO:tasks.workunit.client.0.smithi113.stdout:7/809: creat d0/dc/d6/d9b/d9f/f113 x:0 0 0 2024-04-12T05:09:11.985 INFO:tasks.workunit.client.1.smithi192.stdout:0/890: creat d0/d7b/d9b/f12e x:0 0 0 2024-04-12T05:09:11.985 INFO:tasks.workunit.client.1.smithi192.stdout:6/820: truncate d3/d8/d56/d59/d97/f100 714834 0 2024-04-12T05:09:11.986 INFO:tasks.workunit.client.0.smithi113.stdout:8/865: fsync d9/fb 0 2024-04-12T05:09:11.990 INFO:tasks.workunit.client.0.smithi113.stdout:4/841: creat da/d2b/f11c x:0 0 0 2024-04-12T05:09:11.990 INFO:tasks.workunit.client.1.smithi192.stdout:4/821: dread d1/f3e [4194304,4194304] 0 2024-04-12T05:09:11.991 INFO:tasks.workunit.client.0.smithi113.stdout:1/930: dwrite ff [4194304,4194304] 0 2024-04-12T05:09:11.991 INFO:tasks.workunit.client.1.smithi192.stdout:3/752: write d5/d13/d94/fda [751909,1838] 0 2024-04-12T05:09:11.992 INFO:tasks.workunit.client.1.smithi192.stdout:9/669: dwrite d11/d25/d36/d6e/f56 [4194304,4194304] 0 2024-04-12T05:09:11.992 INFO:tasks.workunit.client.0.smithi113.stdout:4/842: dread - da/d2b/f11c zero size 2024-04-12T05:09:11.994 INFO:tasks.workunit.client.0.smithi113.stdout:8/866: fsync d9/d16/d65/db1/d59/d68/f87 0 2024-04-12T05:09:11.994 INFO:tasks.workunit.client.0.smithi113.stdout:9/882: dread d1/df/d29/fbc [0,4194304] 0 2024-04-12T05:09:11.997 INFO:tasks.workunit.client.0.smithi113.stdout:8/867: write d9/dd4/f107 [974283,106899] 0 2024-04-12T05:09:12.000 INFO:tasks.workunit.client.1.smithi192.stdout:5/868: dwrite d0/d15/d24/d29/f2a [0,4194304] 0 2024-04-12T05:09:12.002 INFO:tasks.workunit.client.0.smithi113.stdout:4/843: fsync da/d24/d60/db4/fc7 0 2024-04-12T05:09:12.005 INFO:tasks.workunit.client.0.smithi113.stdout:2/824: dread d8/d9/de/d12/d2b/daa/db3/ff0 [0,4194304] 0 2024-04-12T05:09:12.006 INFO:tasks.workunit.client.0.smithi113.stdout:7/810: truncate d0/dd/d11/f5e 854776 0 2024-04-12T05:09:12.007 INFO:tasks.workunit.client.1.smithi192.stdout:9/670: readlink d11/d25/d36/d6e/lb4 0 2024-04-12T05:09:12.007 INFO:tasks.workunit.client.1.smithi192.stdout:0/891: chown d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d3a/c51 52 1 2024-04-12T05:09:12.007 INFO:tasks.workunit.client.1.smithi192.stdout:8/756: write d6/d1f/d5d/f8d [1138128,74547] 0 2024-04-12T05:09:12.008 INFO:tasks.workunit.client.1.smithi192.stdout:6/821: dwrite d3/d8/de6/fea [0,4194304] 0 2024-04-12T05:09:12.008 INFO:tasks.workunit.client.1.smithi192.stdout:5/869: chown d0/d15/l22 2837 1 2024-04-12T05:09:12.010 INFO:tasks.workunit.client.1.smithi192.stdout:0/892: creat d0/d7b/d9b/d35/f12f x:0 0 0 2024-04-12T05:09:12.014 INFO:tasks.workunit.client.0.smithi113.stdout:9/883: write d1/df/d29/da6/fdb [2901056,104029] 0 2024-04-12T05:09:12.016 INFO:tasks.workunit.client.1.smithi192.stdout:8/757: mkdir d6/d10/d6c/d9e/dde/de7 0 2024-04-12T05:09:12.017 INFO:tasks.workunit.client.0.smithi113.stdout:8/868: dwrite d9/dd4/ff1 [0,4194304] 0 2024-04-12T05:09:12.020 INFO:tasks.workunit.client.0.smithi113.stdout:7/811: chown d0/dd/d95/f10e 97352 1 2024-04-12T05:09:12.021 INFO:tasks.workunit.client.1.smithi192.stdout:4/822: dwrite d1/f2 [0,4194304] 0 2024-04-12T05:09:12.021 INFO:tasks.workunit.client.1.smithi192.stdout:0/893: rmdir d0/d7b/d9b/d35/d109/d9f/d10a/d67/dd2 39 2024-04-12T05:09:12.024 INFO:tasks.workunit.client.0.smithi113.stdout:7/812: chown d0/dc/d50/f102 118 1 2024-04-12T05:09:12.025 INFO:tasks.workunit.client.1.smithi192.stdout:6/822: dwrite d3/d8/d10/d4c/fe2 [0,4194304] 0 2024-04-12T05:09:12.025 INFO:tasks.workunit.client.1.smithi192.stdout:0/894: chown d0/fad 4 1 2024-04-12T05:09:12.026 INFO:tasks.workunit.client.0.smithi113.stdout:2/825: dwrite d8/d9/d1a/fe8 [0,4194304] 0 2024-04-12T05:09:12.027 INFO:tasks.workunit.client.1.smithi192.stdout:5/870: creat d0/d15/d24/d29/f128 x:0 0 0 2024-04-12T05:09:12.030 INFO:tasks.workunit.client.0.smithi113.stdout:1/931: dwrite d10/d41/d89/df2/ffb [0,4194304] 0 2024-04-12T05:09:12.033 INFO:tasks.workunit.client.1.smithi192.stdout:6/823: dread - d3/d8/d10/d4c/fa6 zero size 2024-04-12T05:09:12.034 INFO:tasks.workunit.client.0.smithi113.stdout:2/826: rmdir d8/d9/de/d12/d2b/d79/ddf 39 2024-04-12T05:09:12.034 INFO:tasks.workunit.client.1.smithi192.stdout:0/895: creat d0/d7b/d9b/d35/d109/d9f/d10a/d67/db1/deb/f130 x:0 0 0 2024-04-12T05:09:12.035 INFO:tasks.workunit.client.0.smithi113.stdout:1/932: chown d10/d2e/d31/dc3/fd5 471 1 2024-04-12T05:09:12.036 INFO:tasks.workunit.client.0.smithi113.stdout:9/884: dwrite d1/df/f36 [0,4194304] 0 2024-04-12T05:09:12.037 INFO:tasks.workunit.client.0.smithi113.stdout:7/813: creat d0/dc/d6/d24/dd2/f114 x:0 0 0 2024-04-12T05:09:12.040 INFO:tasks.workunit.client.1.smithi192.stdout:4/823: rename d1/d54/d82/d9c to d1/d1d/d53/d11a 0 2024-04-12T05:09:12.042 INFO:tasks.workunit.client.1.smithi192.stdout:5/871: symlink d0/d15/d3e/dca/dda/d11c/l129 0 2024-04-12T05:09:12.044 INFO:tasks.workunit.client.1.smithi192.stdout:3/753: truncate d5/d13/d94/fd5 185051 0 2024-04-12T05:09:12.046 INFO:tasks.workunit.client.1.smithi192.stdout:5/872: read d0/d15/d3e/dca/dda/da7/f105 [3297578,30980] 0 2024-04-12T05:09:12.054 INFO:tasks.workunit.client.0.smithi113.stdout:2/827: rmdir d8/d9/de/d15/d24 39 2024-04-12T05:09:12.054 INFO:tasks.workunit.client.0.smithi113.stdout:4/844: write da/d3c/d43/fc5 [2277962,44505] 0 2024-04-12T05:09:12.054 INFO:tasks.workunit.client.0.smithi113.stdout:7/814: fsync d0/dc/d6/d78/f7c 0 2024-04-12T05:09:12.054 INFO:tasks.workunit.client.1.smithi192.stdout:3/754: write d5/d13/d3a/d8b/da2/da7/fd9 [213629,36244] 0 2024-04-12T05:09:12.054 INFO:tasks.workunit.client.1.smithi192.stdout:6/824: link d3/d8/d56/d59/ce4 d3/d8/c112 0 2024-04-12T05:09:12.055 INFO:tasks.workunit.client.0.smithi113.stdout:9/885: rmdir d1/d4/d87 39 2024-04-12T05:09:12.055 INFO:tasks.workunit.client.0.smithi113.stdout:1/933: mknod d10/d41/d89/d96/dd3/d124/c12a 0 2024-04-12T05:09:12.056 INFO:tasks.workunit.client.1.smithi192.stdout:8/758: dwrite d6/da/f1e [4194304,4194304] 0 2024-04-12T05:09:12.056 INFO:tasks.workunit.client.0.smithi113.stdout:7/815: rmdir d0 39 2024-04-12T05:09:12.058 INFO:tasks.workunit.client.1.smithi192.stdout:6/825: write d3/dc/fed [257166,83168] 0 2024-04-12T05:09:12.058 INFO:tasks.workunit.client.1.smithi192.stdout:5/873: creat d0/d15/d3e/dca/dda/db5/dd7/d116/f12a x:0 0 0 2024-04-12T05:09:12.058 INFO:tasks.workunit.client.1.smithi192.stdout:0/896: dwrite d0/d7b/d9b/d35/d109/d53/df3/f119 [0,4194304] 0 2024-04-12T05:09:12.061 INFO:tasks.workunit.client.0.smithi113.stdout:8/869: dwrite d9/d1b/d8e/d93/f71 [0,4194304] 0 2024-04-12T05:09:12.061 INFO:tasks.workunit.client.0.smithi113.stdout:4/845: rmdir da/d70 39 2024-04-12T05:09:12.063 INFO:tasks.workunit.client.1.smithi192.stdout:8/759: dread - d6/d1f/d9c/fd0 zero size 2024-04-12T05:09:12.064 INFO:tasks.workunit.client.1.smithi192.stdout:6/826: chown d3/d8/d22/dd4/ff9 23217 1 2024-04-12T05:09:12.065 INFO:tasks.workunit.client.1.smithi192.stdout:3/755: mknod d5/d13/d3c/d26/dca/c10f 0 2024-04-12T05:09:12.065 INFO:tasks.workunit.client.0.smithi113.stdout:2/828: symlink d8/d9/d27/d88/d91/ded/def/l101 0 2024-04-12T05:09:12.066 INFO:tasks.workunit.client.1.smithi192.stdout:0/897: truncate d0/d7b/d7c/dcc/f97 1918816 0 2024-04-12T05:09:12.069 INFO:tasks.workunit.client.1.smithi192.stdout:6/827: symlink d3/l113 0 2024-04-12T05:09:12.075 INFO:tasks.workunit.client.0.smithi113.stdout:2/829: fdatasync d8/fd 0 2024-04-12T05:09:12.077 INFO:tasks.workunit.client.1.smithi192.stdout:5/874: dwrite d0/d15/d24/d47/fdd [0,4194304] 0 2024-04-12T05:09:12.083 INFO:tasks.workunit.client.0.smithi113.stdout:2/830: creat d8/d9/de/d12/d2b/daa/db3/de2/f102 x:0 0 0 2024-04-12T05:09:12.084 INFO:tasks.workunit.client.0.smithi113.stdout:4/846: dwrite da/d24/d7b/fdb [0,4194304] 0 2024-04-12T05:09:12.085 INFO:tasks.workunit.client.1.smithi192.stdout:6/828: mknod d3/d8/dd6/c114 0 2024-04-12T05:09:12.085 INFO:tasks.workunit.client.1.smithi192.stdout:0/898: rename d0/d7b/d9b/d35/d109/d53/d6d/f9a to d0/d7b/d9b/d35/d109/d53/df3/d2e/d59/f131 0 2024-04-12T05:09:12.087 INFO:tasks.workunit.client.1.smithi192.stdout:5/875: creat d0/d15/d64/f12b x:0 0 0 2024-04-12T05:09:12.091 INFO:tasks.workunit.client.0.smithi113.stdout:9/886: dwrite d1/df/d4b/f7f [0,4194304] 0 2024-04-12T05:09:12.093 INFO:tasks.workunit.client.1.smithi192.stdout:3/756: link d5/d13/d1a/d5f/d103/ffc d5/d13/d3a/d70/d8c/d8d/dbc/f110 0 2024-04-12T05:09:12.095 INFO:tasks.workunit.client.0.smithi113.stdout:4/847: fsync da/d24/d7e/ff1 0 2024-04-12T05:09:12.097 INFO:tasks.workunit.client.0.smithi113.stdout:7/816: dwrite d0/fe0 [0,4194304] 0 2024-04-12T05:09:12.104 INFO:tasks.workunit.client.0.smithi113.stdout:9/887: chown d1/df/d29 1312 1 2024-04-12T05:09:12.104 INFO:tasks.workunit.client.0.smithi113.stdout:8/870: dwrite d9/d1b/d8e/d93/d92/f99 [0,4194304] 0 2024-04-12T05:09:12.105 INFO:tasks.workunit.client.1.smithi192.stdout:8/760: dwrite d6/d10/d46/d5e/d5f/f16 [0,4194304] 0 2024-04-12T05:09:12.105 INFO:tasks.workunit.client.1.smithi192.stdout:5/876: link d0/d15/d24/f4b d0/d15/d24/d29/f12c 0 2024-04-12T05:09:12.106 INFO:tasks.workunit.client.0.smithi113.stdout:2/831: truncate d8/d9/de/d12/dbf/fd8 5063143 0 2024-04-12T05:09:12.107 INFO:tasks.workunit.client.0.smithi113.stdout:1/934: dwrite d10/d41/f4a [8388608,4194304] 0 2024-04-12T05:09:12.112 INFO:tasks.workunit.client.0.smithi113.stdout:2/832: truncate d8/d9/d1a/fec 975566 0 2024-04-12T05:09:12.114 INFO:tasks.workunit.client.1.smithi192.stdout:6/829: dwrite d3/dc/d39/f3a [0,4194304] 0 2024-04-12T05:09:12.114 INFO:tasks.workunit.client.1.smithi192.stdout:5/877: mknod d0/d15/d32/d69/dc8/c12d 0 2024-04-12T05:09:12.114 INFO:tasks.workunit.client.1.smithi192.stdout:6/830: read - d3/d5/f81 zero size 2024-04-12T05:09:12.115 INFO:tasks.workunit.client.0.smithi113.stdout:7/817: dread d0/dc/da9/d5c/d85/fb1 [0,4194304] 0 2024-04-12T05:09:12.117 INFO:tasks.workunit.client.0.smithi113.stdout:7/818: write d0/dc/fde [261151,94500] 0 2024-04-12T05:09:12.118 INFO:tasks.workunit.client.0.smithi113.stdout:2/833: rename d8/d9/d27/d88/d91 to d8/d9/d1a/d103 0 2024-04-12T05:09:12.120 INFO:tasks.workunit.client.1.smithi192.stdout:6/831: read - d3/dc/d39/f7a zero size 2024-04-12T05:09:12.121 INFO:tasks.workunit.client.0.smithi113.stdout:2/834: chown d8/d9/f92 1 1 2024-04-12T05:09:12.121 INFO:tasks.workunit.client.0.smithi113.stdout:8/871: getdents d9/d1b/d8e/d93/d32 0 2024-04-12T05:09:12.124 INFO:tasks.workunit.client.1.smithi192.stdout:0/899: write d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/f80 [571170,100620] 0 2024-04-12T05:09:12.126 INFO:tasks.workunit.client.1.smithi192.stdout:0/900: write d0/d7b/d9b/d35/d109/d53/df3/d29/f2b [1131968,123623] 0 2024-04-12T05:09:12.130 INFO:tasks.workunit.client.0.smithi113.stdout:1/935: getdents d10/d2e/d31/d5c/daf 0 2024-04-12T05:09:12.130 INFO:tasks.workunit.client.0.smithi113.stdout:2/835: mknod d8/d9/d27/d88/de0/c104 0 2024-04-12T05:09:12.132 INFO:tasks.workunit.client.1.smithi192.stdout:5/878: rename d0/d21/f36 to d0/d15/d24/d29/f12e 0 2024-04-12T05:09:12.133 INFO:tasks.workunit.client.0.smithi113.stdout:4/848: dread da/d4a/f9d [0,4194304] 0 2024-04-12T05:09:12.133 INFO:tasks.workunit.client.0.smithi113.stdout:2/836: mkdir d8/d105 0 2024-04-12T05:09:12.133 INFO:tasks.workunit.client.1.smithi192.stdout:3/757: dwrite d5/d13/d3a/d4e/f6c [0,4194304] 0 2024-04-12T05:09:12.135 INFO:tasks.workunit.client.0.smithi113.stdout:4/849: write da/d24/d7b/fdb [3642957,65773] 0 2024-04-12T05:09:12.137 INFO:tasks.workunit.client.1.smithi192.stdout:0/901: dwrite d0/d7b/d9b/d35/d109/d53/df3/d29/f2b [0,4194304] 0 2024-04-12T05:09:12.138 INFO:tasks.workunit.client.0.smithi113.stdout:2/837: rmdir d8/d9/de/d43 39 2024-04-12T05:09:12.138 INFO:tasks.workunit.client.0.smithi113.stdout:8/872: dread d9/d16/d65/fa9 [0,4194304] 0 2024-04-12T05:09:12.140 INFO:tasks.workunit.client.0.smithi113.stdout:8/873: rmdir d9/d16/d65/db4 39 2024-04-12T05:09:12.142 INFO:tasks.workunit.client.1.smithi192.stdout:8/761: link d6/d10/d46/d5e/d5f/d37/d7b/l89 d6/d10/d46/d5e/d5f/d31/le8 0 2024-04-12T05:09:12.142 INFO:tasks.workunit.client.0.smithi113.stdout:2/838: creat d8/d9/de/d12/d2b/dd2/df6/f106 x:0 0 0 2024-04-12T05:09:12.143 INFO:tasks.workunit.client.0.smithi113.stdout:1/936: link d10/d41/ce8 d10/d2e/dd8/d70/c12b 0 2024-04-12T05:09:12.145 INFO:tasks.workunit.client.1.smithi192.stdout:0/902: mknod d0/d7b/d9b/d35/d109/d53/df3/d29/db5/d11e/c132 0 2024-04-12T05:09:12.145 INFO:tasks.workunit.client.1.smithi192.stdout:5/879: dwrite d0/d15/d64/fb6 [0,4194304] 0 2024-04-12T05:09:12.146 INFO:tasks.workunit.client.1.smithi192.stdout:3/758: mknod d5/d13/d3a/d70/d8c/d8d/daf/c111 0 2024-04-12T05:09:12.146 INFO:tasks.workunit.client.0.smithi113.stdout:2/839: write d8/d9/d1a/ff1 [723612,25776] 0 2024-04-12T05:09:12.146 INFO:tasks.workunit.client.0.smithi113.stdout:8/874: creat d9/d16/d65/db1/d59/d78/f116 x:0 0 0 2024-04-12T05:09:12.147 INFO:tasks.workunit.client.1.smithi192.stdout:8/762: write d6/d10/d46/d5e/d5f/d37/fc7 [708157,129585] 0 2024-04-12T05:09:12.148 INFO:tasks.workunit.client.1.smithi192.stdout:8/763: chown d6/d10/d46/d5e/d5f/d31/d61/d9d/db5 133886859 1 2024-04-12T05:09:12.150 INFO:tasks.workunit.client.1.smithi192.stdout:8/764: fdatasync d6/d10/d46/d5e/d5f/d37/d4f/d4c/f94 0 2024-04-12T05:09:12.150 INFO:tasks.workunit.client.1.smithi192.stdout:3/759: creat d5/df1/f112 x:0 0 0 2024-04-12T05:09:12.151 INFO:tasks.workunit.client.1.smithi192.stdout:3/760: fsync d5/d13/d3a/d4e/f102 0 2024-04-12T05:09:12.152 INFO:tasks.workunit.client.0.smithi113.stdout:8/875: truncate d9/d16/d65/db1/d59/da3/f105 4798080 0 2024-04-12T05:09:12.153 INFO:tasks.workunit.client.0.smithi113.stdout:2/840: fdatasync d8/d9/d5b/f85 0 2024-04-12T05:09:12.156 INFO:tasks.workunit.client.0.smithi113.stdout:2/841: creat d8/d9/de/d12/d3c/d8a/f107 x:0 0 0 2024-04-12T05:09:12.160 INFO:tasks.workunit.client.0.smithi113.stdout:2/842: read d8/d9/de/d12/d2b/d79/fb8 [389524,67010] 0 2024-04-12T05:09:12.163 INFO:tasks.workunit.client.1.smithi192.stdout:6/832: truncate d3/d8/d56/d59/d97/dbf/de8/ffa 1391587 0 2024-04-12T05:09:12.165 INFO:tasks.workunit.client.0.smithi113.stdout:7/819: dread d0/dc/d6/d78/f45 [0,4194304] 0 2024-04-12T05:09:12.168 INFO:tasks.workunit.client.1.smithi192.stdout:5/880: write d0/f7 [2657514,33092] 0 2024-04-12T05:09:12.172 INFO:tasks.workunit.client.1.smithi192.stdout:5/881: truncate d0/d15/d3e/dca/ff5 1032290 0 2024-04-12T05:09:12.172 INFO:tasks.workunit.client.1.smithi192.stdout:6/833: dwrite d3/d23/fb9 [0,4194304] 0 2024-04-12T05:09:12.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:10.674039+0000 mgr.smithi192.ocppyo (mgr.24357) 32 : cluster [DBG] pgmap v25: 65 pgs: 65 active+clean; 2.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 82 MiB/s rd, 217 MiB/s wr, 436 op/s 2024-04-12T05:09:12.174 INFO:tasks.workunit.client.1.smithi192.stdout:5/882: write d0/d15/d32/d45/f126 [455756,58046] 0 2024-04-12T05:09:12.175 INFO:tasks.workunit.client.0.smithi113.stdout:6/808: sync 2024-04-12T05:09:12.176 INFO:tasks.workunit.client.1.smithi192.stdout:6/834: write d3/d8/d56/d59/d8a/d99/f9f [2182304,89261] 0 2024-04-12T05:09:12.178 INFO:tasks.workunit.client.0.smithi113.stdout:1/937: dread d10/d41/d89/d96/dd3/f50 [0,4194304] 0 2024-04-12T05:09:12.178 INFO:tasks.workunit.client.1.smithi192.stdout:6/835: truncate d3/d26/f28 5168442 0 2024-04-12T05:09:12.178 INFO:tasks.workunit.client.1.smithi192.stdout:6/836: read d3/d26/f46 [4073688,35439] 0 2024-04-12T05:09:12.179 INFO:tasks.workunit.client.1.smithi192.stdout:5/883: fdatasync d0/d15/d32/d45/fc6 0 2024-04-12T05:09:12.182 INFO:tasks.workunit.client.1.smithi192.stdout:5/884: mkdir d0/d15/d32/d69/dc8/d12f 0 2024-04-12T05:09:12.183 INFO:tasks.workunit.client.1.smithi192.stdout:6/837: rename d3/d5/d9c/dbd/cff to d3/d8/d56/d59/d8a/c115 0 2024-04-12T05:09:12.187 INFO:tasks.workunit.client.0.smithi113.stdout:1/938: dwrite d10/da5/f10a [0,4194304] 0 2024-04-12T05:09:12.192 INFO:tasks.workunit.client.1.smithi192.stdout:6/838: unlink d3/d8/d10/l104 0 2024-04-12T05:09:12.194 INFO:tasks.workunit.client.0.smithi113.stdout:1/939: unlink d10/d14/d95/leb 0 2024-04-12T05:09:12.194 INFO:tasks.workunit.client.0.smithi113.stdout:7/820: write d0/dc/d6/d78/f6a [2543670,47279] 0 2024-04-12T05:09:12.196 INFO:tasks.workunit.client.1.smithi192.stdout:6/839: read d3/d8/f92 [350273,42893] 0 2024-04-12T05:09:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:10.674039+0000 mgr.smithi192.ocppyo (mgr.24357) 32 : cluster [DBG] pgmap v25: 65 pgs: 65 active+clean; 2.7 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 82 MiB/s rd, 217 MiB/s wr, 436 op/s 2024-04-12T05:09:12.199 INFO:tasks.workunit.client.0.smithi113.stdout:8/876: dread d9/d16/d65/d8c/fa4 [0,4194304] 0 2024-04-12T05:09:12.200 INFO:tasks.workunit.client.1.smithi192.stdout:5/885: write d0/d15/d24/f4b [5342443,13670] 0 2024-04-12T05:09:12.203 INFO:tasks.workunit.client.1.smithi192.stdout:8/765: dwrite d6/d10/d46/d7c/fa0 [0,4194304] 0 2024-04-12T05:09:12.204 INFO:tasks.workunit.client.0.smithi113.stdout:7/821: mknod d0/dd/c115 0 2024-04-12T05:09:12.205 INFO:tasks.workunit.client.0.smithi113.stdout:4/850: dwrite da/d24/d60/db4/fc9 [0,4194304] 0 2024-04-12T05:09:12.205 INFO:tasks.workunit.client.0.smithi113.stdout:2/843: dread d8/d9/f69 [0,4194304] 0 2024-04-12T05:09:12.206 INFO:tasks.workunit.client.1.smithi192.stdout:3/761: dwrite d5/d13/f42 [0,4194304] 0 2024-04-12T05:09:12.208 INFO:tasks.workunit.client.0.smithi113.stdout:4/851: fdatasync da/dd/d3a/fda 0 2024-04-12T05:09:12.209 INFO:tasks.workunit.client.1.smithi192.stdout:3/762: write d5/d13/d94/fda [1203544,117831] 0 2024-04-12T05:09:12.210 INFO:tasks.workunit.client.0.smithi113.stdout:1/940: read d10/d21/f40 [4239598,13549] 0 2024-04-12T05:09:12.212 INFO:tasks.workunit.client.0.smithi113.stdout:1/941: stat d10/d2e/d31/d5c/daf/cc1 0 2024-04-12T05:09:12.213 INFO:tasks.workunit.client.0.smithi113.stdout:6/809: dwrite d8/de/d13/d11/d36/d89/f97 [0,4194304] 0 2024-04-12T05:09:12.213 INFO:tasks.workunit.client.0.smithi113.stdout:4/852: rmdir da/d3c/d83/d47/d10f/d90 39 2024-04-12T05:09:12.214 INFO:tasks.workunit.client.1.smithi192.stdout:5/886: truncate d0/f1b 2088001 0 2024-04-12T05:09:12.218 INFO:tasks.workunit.client.1.smithi192.stdout:8/766: truncate d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d98/fd1 5745756 0 2024-04-12T05:09:12.218 INFO:tasks.workunit.client.1.smithi192.stdout:2/776: sync 2024-04-12T05:09:12.218 INFO:tasks.workunit.client.1.smithi192.stdout:7/688: sync 2024-04-12T05:09:12.219 INFO:tasks.workunit.client.0.smithi113.stdout:6/810: chown d8/de/d13/d11/d36/d3d/d4c/d26/f114 0 1 2024-04-12T05:09:12.219 INFO:tasks.workunit.client.1.smithi192.stdout:2/777: chown d1/f39 171423 1 2024-04-12T05:09:12.221 INFO:tasks.workunit.client.0.smithi113.stdout:1/942: truncate d10/d2e/d7c/dc5/dc8/f9e 1903514 0 2024-04-12T05:09:12.222 INFO:tasks.workunit.client.1.smithi192.stdout:5/887: unlink d0/d21/d5e/cbd 0 2024-04-12T05:09:12.224 INFO:tasks.workunit.client.0.smithi113.stdout:4/853: getdents da/dd/d1b 0 2024-04-12T05:09:12.228 INFO:tasks.workunit.client.0.smithi113.stdout:8/877: dwrite d9/d1b/d8e/d93/f66 [0,4194304] 0 2024-04-12T05:09:12.229 INFO:tasks.workunit.client.1.smithi192.stdout:2/778: mkdir d1/d8/d17/d19/d60/df6 0 2024-04-12T05:09:12.229 INFO:tasks.workunit.client.0.smithi113.stdout:4/854: write da/d70/d80/f8b [4809335,32029] 0 2024-04-12T05:09:12.229 INFO:tasks.workunit.client.1.smithi192.stdout:8/767: symlink d6/d10/d46/le9 0 2024-04-12T05:09:12.230 INFO:tasks.workunit.client.1.smithi192.stdout:7/689: dwrite d3/f9e [4194304,4194304] 0 2024-04-12T05:09:12.232 INFO:tasks.workunit.client.1.smithi192.stdout:5/888: creat d0/d15/d32/d69/dc8/d12f/f130 x:0 0 0 2024-04-12T05:09:12.237 INFO:tasks.workunit.client.0.smithi113.stdout:4/855: creat da/d3c/d83/d47/d10f/d90/f11d x:0 0 0 2024-04-12T05:09:12.238 INFO:tasks.workunit.client.1.smithi192.stdout:8/768: truncate d6/d10/d46/d5e/d5f/d31/d80/db1/fe5 708641 0 2024-04-12T05:09:12.239 INFO:tasks.workunit.client.1.smithi192.stdout:5/889: mknod d0/d15/c131 0 2024-04-12T05:09:12.240 INFO:tasks.workunit.client.0.smithi113.stdout:4/856: rmdir da/d3c/da5/dad/de1 39 2024-04-12T05:09:12.248 INFO:tasks.workunit.client.1.smithi192.stdout:5/890: mknod d0/d15/d3e/d120/c132 0 2024-04-12T05:09:12.249 INFO:tasks.workunit.client.1.smithi192.stdout:6/840: dread d3/dc/f69 [0,4194304] 0 2024-04-12T05:09:12.250 INFO:tasks.workunit.client.0.smithi113.stdout:8/878: truncate d9/d16/d65/db1/d59/da3/fe0 3649136 0 2024-04-12T05:09:12.251 INFO:tasks.workunit.client.1.smithi192.stdout:8/769: creat d6/d10/d46/d5e/d5f/fea x:0 0 0 2024-04-12T05:09:12.251 INFO:tasks.workunit.client.1.smithi192.stdout:3/763: dread d5/d13/d1a/d53/f60 [0,4194304] 0 2024-04-12T05:09:12.252 INFO:tasks.workunit.client.1.smithi192.stdout:7/690: write d3/d1d/d5a/d5f/f64 [3130919,114029] 0 2024-04-12T05:09:12.253 INFO:tasks.workunit.client.1.smithi192.stdout:5/891: mknod d0/d15/d3e/dca/dd3/c133 0 2024-04-12T05:09:12.254 INFO:tasks.workunit.client.0.smithi113.stdout:8/879: rmdir d9/d16/d65/db1/d106 39 2024-04-12T05:09:12.257 INFO:tasks.workunit.client.1.smithi192.stdout:8/770: write d6/d1f/d5d/fd3 [868829,83109] 0 2024-04-12T05:09:12.257 INFO:tasks.workunit.client.0.smithi113.stdout:4/857: dwrite da/d3c/d83/fd7 [0,4194304] 0 2024-04-12T05:09:12.258 INFO:tasks.workunit.client.1.smithi192.stdout:6/841: rename d3/d8/d22/dd4/c107 to d3/d8/d56/d59/ddf/df1/c116 0 2024-04-12T05:09:12.259 INFO:tasks.workunit.client.0.smithi113.stdout:1/943: dread d10/d41/d89/d96/d7d/d9f/fbd [0,4194304] 0 2024-04-12T05:09:12.262 INFO:tasks.workunit.client.0.smithi113.stdout:8/880: creat d9/d1b/dfd/df3/f117 x:0 0 0 2024-04-12T05:09:12.262 INFO:tasks.workunit.client.1.smithi192.stdout:5/892: creat d0/d15/d24/ddf/f134 x:0 0 0 2024-04-12T05:09:12.262 INFO:tasks.workunit.client.1.smithi192.stdout:6/842: readlink d3/d5/d9c/l105 0 2024-04-12T05:09:12.264 INFO:tasks.workunit.client.0.smithi113.stdout:4/858: dread - da/d24/d7e/d79/dcd/fe4 zero size 2024-04-12T05:09:12.266 INFO:tasks.workunit.client.0.smithi113.stdout:8/881: creat d9/d16/d65/db1/dec/f118 x:0 0 0 2024-04-12T05:09:12.267 INFO:tasks.workunit.client.1.smithi192.stdout:3/764: dwrite d5/d13/d3c/d84/f9d [0,4194304] 0 2024-04-12T05:09:12.268 INFO:tasks.workunit.client.0.smithi113.stdout:8/882: write d9/d1b/d8e/d93/d32/d5b/f83 [5418403,44554] 0 2024-04-12T05:09:12.268 INFO:tasks.workunit.client.0.smithi113.stdout:4/859: rename da/d2b/fa0 to da/d70/d80/f11e 0 2024-04-12T05:09:12.272 INFO:tasks.workunit.client.1.smithi192.stdout:3/765: creat d5/d13/d3a/d70/d8c/d8d/dbc/f113 x:0 0 0 2024-04-12T05:09:12.276 INFO:tasks.workunit.client.0.smithi113.stdout:4/860: creat da/d3c/d9a/dc1/f11f x:0 0 0 2024-04-12T05:09:12.277 INFO:tasks.workunit.client.0.smithi113.stdout:8/883: stat d9/d16/d65/db1/d59/da3/fe0 0 2024-04-12T05:09:12.279 INFO:tasks.workunit.client.1.smithi192.stdout:7/691: write d3/d11/f37 [2057874,42191] 0 2024-04-12T05:09:12.280 INFO:tasks.workunit.client.0.smithi113.stdout:1/944: write d10/d2e/d7c/dc5/fc7 [262026,1014] 0 2024-04-12T05:09:12.280 INFO:tasks.workunit.client.1.smithi192.stdout:8/771: truncate d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/f72 3333812 0 2024-04-12T05:09:12.281 INFO:tasks.workunit.client.0.smithi113.stdout:8/884: mkdir d9/d16/d65/db4/d119 0 2024-04-12T05:09:12.282 INFO:tasks.workunit.client.1.smithi192.stdout:6/843: write d3/d5/d9c/dbd/fd3 [861335,124099] 0 2024-04-12T05:09:12.282 INFO:tasks.workunit.client.1.smithi192.stdout:7/692: mknod d3/d11/de8/cec 0 2024-04-12T05:09:12.284 INFO:tasks.workunit.client.1.smithi192.stdout:8/772: unlink d6/d10/l67 0 2024-04-12T05:09:12.285 INFO:tasks.workunit.client.0.smithi113.stdout:1/945: read d10/d55/f8b [1936088,103489] 0 2024-04-12T05:09:12.285 INFO:tasks.workunit.client.1.smithi192.stdout:3/766: write d5/d13/d3c/d84/f9d [5212879,54792] 0 2024-04-12T05:09:12.286 INFO:tasks.workunit.client.1.smithi192.stdout:6/844: creat d3/d5/d9e/f117 x:0 0 0 2024-04-12T05:09:12.287 INFO:tasks.workunit.client.1.smithi192.stdout:7/693: creat d3/d1d/d74/d86/fed x:0 0 0 2024-04-12T05:09:12.291 INFO:tasks.workunit.client.1.smithi192.stdout:8/773: creat d6/d10/d46/d5e/d5f/d37/feb x:0 0 0 2024-04-12T05:09:12.291 INFO:tasks.workunit.client.1.smithi192.stdout:5/893: dwrite d0/d15/d32/d45/fb1 [0,4194304] 0 2024-04-12T05:09:12.296 INFO:tasks.workunit.client.1.smithi192.stdout:3/767: mkdir d5/d13/d3a/d4a/d61/d9f/deb/dfb/d114 0 2024-04-12T05:09:12.297 INFO:tasks.workunit.client.1.smithi192.stdout:5/894: creat d0/d21/d5e/f135 x:0 0 0 2024-04-12T05:09:12.300 INFO:tasks.workunit.client.1.smithi192.stdout:3/768: chown d5/d13/d3a/d4e/f95 305388 1 2024-04-12T05:09:12.301 INFO:tasks.workunit.client.1.smithi192.stdout:7/694: dwrite d3/d1d/d74/d86/d93/fd9 [0,4194304] 0 2024-04-12T05:09:12.304 INFO:tasks.workunit.client.1.smithi192.stdout:7/695: rmdir d3/d11/d68 39 2024-04-12T05:09:12.304 INFO:tasks.workunit.client.0.smithi113.stdout:1/946: truncate d10/d2e/d7c/f92 1817309 0 2024-04-12T05:09:12.305 INFO:tasks.workunit.client.1.smithi192.stdout:8/774: dwrite d6/d10/d46/d5e/d5f/d37/d4f/f9a [0,4194304] 0 2024-04-12T05:09:12.306 INFO:tasks.workunit.client.0.smithi113.stdout:1/947: write d10/d41/d89/d96/f10b [54864,122424] 0 2024-04-12T05:09:12.307 INFO:tasks.workunit.client.1.smithi192.stdout:3/769: symlink d5/d13/d3a/d70/d8c/d8d/dbc/l115 0 2024-04-12T05:09:12.307 INFO:tasks.workunit.client.0.smithi113.stdout:8/885: dread d9/d16/d65/db1/f8b [0,4194304] 0 2024-04-12T05:09:12.307 INFO:tasks.workunit.client.1.smithi192.stdout:5/895: mkdir d0/d15/d3e/dca/dda/deb/d113/d127/ded/d136 0 2024-04-12T05:09:12.309 INFO:tasks.workunit.client.1.smithi192.stdout:5/896: chown d0/d15/d32/d69/fd9 2175378 1 2024-04-12T05:09:12.310 INFO:tasks.workunit.client.1.smithi192.stdout:3/770: fsync d5/d13/d3c/d84/f9d 0 2024-04-12T05:09:12.311 INFO:tasks.workunit.client.0.smithi113.stdout:1/948: creat d10/d2e/d7c/dc5/dc8/d112/f12c x:0 0 0 2024-04-12T05:09:12.312 INFO:tasks.workunit.client.0.smithi113.stdout:1/949: fsync d10/d2e/d7c/dc5/dc8/f104 0 2024-04-12T05:09:12.314 INFO:tasks.workunit.client.1.smithi192.stdout:7/696: truncate d3/d7/d14/f47 437634 0 2024-04-12T05:09:12.317 INFO:tasks.workunit.client.0.smithi113.stdout:1/950: mknod d10/d41/d89/df2/c12d 0 2024-04-12T05:09:12.317 INFO:tasks.workunit.client.1.smithi192.stdout:6/845: dwrite d3/d8/d56/d59/d97/dbf/de8/fb4 [0,4194304] 0 2024-04-12T05:09:12.318 INFO:tasks.workunit.client.0.smithi113.stdout:8/886: dwrite d9/dd4/ff1 [0,4194304] 0 2024-04-12T05:09:12.322 INFO:tasks.workunit.client.1.smithi192.stdout:8/775: getdents d6/d10/d46 0 2024-04-12T05:09:12.323 INFO:tasks.workunit.client.1.smithi192.stdout:7/697: mkdir d3/d1d/d4a/d5e/d6d/da3/dce/dee 0 2024-04-12T05:09:12.329 INFO:tasks.workunit.client.1.smithi192.stdout:5/897: getdents d0/d15/d3e/dca 0 2024-04-12T05:09:12.331 INFO:tasks.workunit.client.1.smithi192.stdout:5/898: chown d0/d15/d32/f4c 241527 1 2024-04-12T05:09:12.332 INFO:tasks.workunit.client.1.smithi192.stdout:6/846: dwrite d3/d8/de6/ffb [0,4194304] 0 2024-04-12T05:09:12.333 INFO:tasks.workunit.client.1.smithi192.stdout:8/776: creat d6/d10/d46/d5e/d5f/d38/dbc/fec x:0 0 0 2024-04-12T05:09:12.336 INFO:tasks.workunit.client.0.smithi113.stdout:8/887: dwrite d9/d1b/d8e/d93/d32/d5b/fb8 [0,4194304] 0 2024-04-12T05:09:12.338 INFO:tasks.workunit.client.1.smithi192.stdout:7/698: mknod d3/d7/d15/cef 0 2024-04-12T05:09:12.338 INFO:tasks.workunit.client.1.smithi192.stdout:6/847: fsync d3/d8/d22/dd4/ff9 0 2024-04-12T05:09:12.340 INFO:tasks.workunit.client.1.smithi192.stdout:3/771: write d5/d13/d3a/d4e/d57/faa [773230,105873] 0 2024-04-12T05:09:12.341 INFO:tasks.workunit.client.0.smithi113.stdout:8/888: read d9/d1b/f46 [924364,26526] 0 2024-04-12T05:09:12.344 INFO:tasks.workunit.client.1.smithi192.stdout:6/848: unlink d3/dc/fcc 0 2024-04-12T05:09:12.345 INFO:tasks.workunit.client.1.smithi192.stdout:7/699: readlink d3/d1d/d4a/d5e/d6d/da3/le6 0 2024-04-12T05:09:12.346 INFO:tasks.workunit.client.1.smithi192.stdout:7/700: fdatasync d3/d7/d14/f58 0 2024-04-12T05:09:12.347 INFO:tasks.workunit.client.1.smithi192.stdout:7/701: readlink d3/d1d/d5a/d5f/d6a/l83 0 2024-04-12T05:09:12.347 INFO:tasks.workunit.client.1.smithi192.stdout:3/772: fsync d5/d13/d3c/d26/f4d 0 2024-04-12T05:09:12.350 INFO:tasks.workunit.client.1.smithi192.stdout:7/702: rmdir d3/d1d/d4a/d5e 39 2024-04-12T05:09:12.350 INFO:tasks.workunit.client.0.smithi113.stdout:8/889: dwrite d9/d1b/d96/fd1 [0,4194304] 0 2024-04-12T05:09:12.351 INFO:tasks.workunit.client.0.smithi113.stdout:8/890: fdatasync d9/d1b/d96/f10f 0 2024-04-12T05:09:12.353 INFO:tasks.workunit.client.1.smithi192.stdout:3/773: creat d5/d13/d3a/d8b/da2/de5/f116 x:0 0 0 2024-04-12T05:09:12.354 INFO:tasks.workunit.client.0.smithi113.stdout:8/891: mknod d9/d1b/d8e/c11a 0 2024-04-12T05:09:12.361 INFO:tasks.workunit.client.0.smithi113.stdout:1/951: dwrite d10/d41/d89/d96/dd3/f64 [0,4194304] 0 2024-04-12T05:09:12.363 INFO:tasks.workunit.client.1.smithi192.stdout:6/849: truncate d3/d8/d10/d4c/f62 2295574 0 2024-04-12T05:09:12.364 INFO:tasks.workunit.client.1.smithi192.stdout:6/850: fsync d3/d5/d9e/f117 0 2024-04-12T05:09:12.365 INFO:tasks.workunit.client.1.smithi192.stdout:6/851: readlink d3/d8/d22/l49 0 2024-04-12T05:09:12.366 INFO:tasks.workunit.client.0.smithi113.stdout:1/952: dread d10/d41/d89/d96/dd3/f61 [0,4194304] 0 2024-04-12T05:09:12.369 INFO:tasks.workunit.client.0.smithi113.stdout:1/953: creat d10/f12e x:0 0 0 2024-04-12T05:09:12.371 INFO:tasks.workunit.client.1.smithi192.stdout:7/703: dwrite d3/d1d/d48/d7b/f95 [0,4194304] 0 2024-04-12T05:09:12.372 INFO:tasks.workunit.client.0.smithi113.stdout:8/892: dread d9/d16/d65/db1/d59/d68/f87 [0,4194304] 0 2024-04-12T05:09:12.372 INFO:tasks.workunit.client.0.smithi113.stdout:1/954: mkdir d10/d21/d128/d12f 0 2024-04-12T05:09:12.375 INFO:tasks.workunit.client.1.smithi192.stdout:1/812: sync 2024-04-12T05:09:12.375 INFO:tasks.workunit.client.1.smithi192.stdout:7/704: creat d3/ff0 x:0 0 0 2024-04-12T05:09:12.377 INFO:tasks.workunit.client.0.smithi113.stdout:1/955: link d10/d41/faa d10/d2e/d31/d5c/daf/f130 0 2024-04-12T05:09:12.378 INFO:tasks.workunit.client.0.smithi113.stdout:1/956: fsync d10/d41/d89/ff6 0 2024-04-12T05:09:12.381 INFO:tasks.workunit.client.0.smithi113.stdout:8/893: fsync d9/d16/d3c/f74 0 2024-04-12T05:09:12.382 INFO:tasks.workunit.client.1.smithi192.stdout:6/852: write d3/d5/f72 [1435079,92302] 0 2024-04-12T05:09:12.382 INFO:tasks.workunit.client.1.smithi192.stdout:1/813: truncate d0/d2/d1e/d3f/df2/d3c/dab/fb5 717633 0 2024-04-12T05:09:12.383 INFO:tasks.workunit.client.0.smithi113.stdout:1/957: creat d10/da5/f131 x:0 0 0 2024-04-12T05:09:12.384 INFO:tasks.workunit.client.0.smithi113.stdout:8/894: unlink d9/d16/d7f/ca7 0 2024-04-12T05:09:12.385 INFO:tasks.workunit.client.0.smithi113.stdout:8/895: chown d9/d16/d65/d8c/fa4 2 1 2024-04-12T05:09:12.385 INFO:tasks.workunit.client.1.smithi192.stdout:6/853: stat d3/d21/d24/fd8 0 2024-04-12T05:09:12.386 INFO:tasks.workunit.client.1.smithi192.stdout:3/774: dread d5/d13/f2f [0,4194304] 0 2024-04-12T05:09:12.386 INFO:tasks.workunit.client.0.smithi113.stdout:8/896: write d9/d16/f1a [3555892,83151] 0 2024-04-12T05:09:12.388 INFO:tasks.workunit.client.0.smithi113.stdout:1/958: mkdir d10/d14/d95/d132 0 2024-04-12T05:09:12.389 INFO:tasks.workunit.client.0.smithi113.stdout:8/897: symlink d9/d16/d65/db1/d59/da3/l11b 0 2024-04-12T05:09:12.389 INFO:tasks.workunit.client.1.smithi192.stdout:6/854: creat d3/d8/d56/d59/d97/dbf/deb/f118 x:0 0 0 2024-04-12T05:09:12.391 INFO:tasks.workunit.client.1.smithi192.stdout:3/775: rename d5/d13/d3a/d4e/df8 to d5/d13/d94/dbd/d117 0 2024-04-12T05:09:12.393 INFO:tasks.workunit.client.1.smithi192.stdout:6/855: fdatasync d3/d8/f13 0 2024-04-12T05:09:12.395 INFO:tasks.workunit.client.1.smithi192.stdout:6/856: stat d3/d8/f6d 0 2024-04-12T05:09:12.395 INFO:tasks.workunit.client.0.smithi113.stdout:1/959: dwrite d10/d2e/f102 [0,4194304] 0 2024-04-12T05:09:12.397 INFO:tasks.workunit.client.0.smithi113.stdout:8/898: dread d9/d1b/fcd [0,4194304] 0 2024-04-12T05:09:12.402 INFO:tasks.workunit.client.0.smithi113.stdout:8/899: creat d9/d16/d65/f11c x:0 0 0 2024-04-12T05:09:12.402 INFO:tasks.workunit.client.1.smithi192.stdout:3/776: dwrite d5/fd8 [0,4194304] 0 2024-04-12T05:09:12.404 INFO:tasks.workunit.client.0.smithi113.stdout:8/900: creat d9/d16/d65/db1/d59/d68/f11d x:0 0 0 2024-04-12T05:09:12.406 INFO:tasks.workunit.client.1.smithi192.stdout:7/705: dwrite d3/d7/f89 [0,4194304] 0 2024-04-12T05:09:12.436 INFO:tasks.workunit.client.0.smithi113.stdout:3/836: sync 2024-04-12T05:09:12.436 INFO:tasks.workunit.client.0.smithi113.stdout:0/936: sync 2024-04-12T05:09:12.438 INFO:tasks.workunit.client.1.smithi192.stdout:1/814: dread d0/d2/d1e/d3f/df2/d21/d62/d84/f91 [4194304,4194304] 0 2024-04-12T05:09:12.440 INFO:tasks.workunit.client.1.smithi192.stdout:6/857: unlink d3/d8/d22/dd4/ff9 0 2024-04-12T05:09:12.441 INFO:tasks.workunit.client.1.smithi192.stdout:1/815: truncate d0/d2/d1e/d3f/df2/d3c/d5b/ffe 893341 0 2024-04-12T05:09:12.443 INFO:tasks.workunit.client.0.smithi113.stdout:1/960: symlink d10/d2e/dd8/l133 0 2024-04-12T05:09:12.444 INFO:tasks.workunit.client.0.smithi113.stdout:8/901: write d9/d16/d3c/f52 [2182111,58525] 0 2024-04-12T05:09:12.444 INFO:tasks.workunit.client.1.smithi192.stdout:3/777: creat d5/d13/d3a/f118 x:0 0 0 2024-04-12T05:09:12.445 INFO:tasks.workunit.client.0.smithi113.stdout:3/837: fsync d0/d13/d47/d94/dcf/dd1/fe9 0 2024-04-12T05:09:12.446 INFO:tasks.workunit.client.0.smithi113.stdout:8/902: truncate d9/dd4/f110 623978 0 2024-04-12T05:09:12.447 INFO:tasks.workunit.client.0.smithi113.stdout:8/903: chown d9/d1b/d8e/d93/cf4 1322227 1 2024-04-12T05:09:12.450 INFO:tasks.workunit.client.1.smithi192.stdout:6/858: mkdir d3/d8/dd6/df8/d119 0 2024-04-12T05:09:12.451 INFO:tasks.workunit.client.1.smithi192.stdout:7/706: dread d3/d1d/d4a/d5e/d6d/fde [0,4194304] 0 2024-04-12T05:09:12.452 INFO:tasks.workunit.client.1.smithi192.stdout:3/778: unlink d5/d13/d3c/d26/ff4 0 2024-04-12T05:09:12.452 INFO:tasks.workunit.client.1.smithi192.stdout:1/816: mknod d0/dd0/d107/c110 0 2024-04-12T05:09:12.454 INFO:tasks.workunit.client.1.smithi192.stdout:6/859: mknod d3/d23/db2/dc7/c11a 0 2024-04-12T05:09:12.457 INFO:tasks.workunit.client.1.smithi192.stdout:7/707: rename d3/d1d/fc2 to d3/d7/d60/ff1 0 2024-04-12T05:09:12.458 INFO:tasks.workunit.client.1.smithi192.stdout:7/708: stat d3/d1d/d74/d86/d93/db9 0 2024-04-12T05:09:12.458 INFO:tasks.workunit.client.0.smithi113.stdout:0/937: dwrite d1/d43/d59/de9/d105/f70 [0,4194304] 0 2024-04-12T05:09:12.459 INFO:tasks.workunit.client.0.smithi113.stdout:0/938: stat d1/d43/d59/l89 0 2024-04-12T05:09:12.461 INFO:tasks.workunit.client.0.smithi113.stdout:3/838: rename d0/d13/d15/d28/d2a/d34/f101 to d0/d13/f117 0 2024-04-12T05:09:12.461 INFO:tasks.workunit.client.1.smithi192.stdout:6/860: dread d3/d8/d56/d59/d97/dbf/de8/fb4 [0,4194304] 0 2024-04-12T05:09:12.463 INFO:tasks.workunit.client.1.smithi192.stdout:1/817: mknod d0/d29/c111 0 2024-04-12T05:09:12.463 INFO:tasks.workunit.client.0.smithi113.stdout:8/904: getdents d9/d1b/d8e/d93/d92 0 2024-04-12T05:09:12.465 INFO:tasks.workunit.client.0.smithi113.stdout:8/905: write d9/d1b/d8e/d93/d32/f64 [1432071,78832] 0 2024-04-12T05:09:12.466 INFO:tasks.workunit.client.1.smithi192.stdout:7/709: rename d3/d11/de8 to d3/d7/d15/d1a/df2 0 2024-04-12T05:09:12.468 INFO:tasks.workunit.client.1.smithi192.stdout:7/710: chown d3/d1d/d74/d86/ld4 21967 1 2024-04-12T05:09:12.469 INFO:tasks.workunit.client.1.smithi192.stdout:7/711: chown d3/d7/d15/d22/c5b 40 1 2024-04-12T05:09:12.470 INFO:tasks.workunit.client.0.smithi113.stdout:8/906: rename d9/d1b/d8e/fb3 to d9/d1b/f11e 0 2024-04-12T05:09:12.471 INFO:tasks.workunit.client.0.smithi113.stdout:0/939: mkdir d1/d6/d16/dd5/dd6/dde/d122 0 2024-04-12T05:09:12.473 INFO:tasks.workunit.client.1.smithi192.stdout:6/861: rename d3/dc/d39/f70 to d3/d8/d48/f11b 0 2024-04-12T05:09:12.474 INFO:tasks.workunit.client.1.smithi192.stdout:7/712: mkdir d3/d7/d15/d22/df3 0 2024-04-12T05:09:12.475 INFO:tasks.workunit.client.1.smithi192.stdout:7/713: read - d3/d1d/d5a/d5f/fbd zero size 2024-04-12T05:09:12.476 INFO:tasks.workunit.client.0.smithi113.stdout:0/940: unlink d1/d6/d16/f67 0 2024-04-12T05:09:12.476 INFO:tasks.workunit.client.1.smithi192.stdout:7/714: fdatasync d3/d11/f37 0 2024-04-12T05:09:12.476 INFO:tasks.workunit.client.0.smithi113.stdout:0/941: chown d1/d43/d59/de9/d105/d50 31175 1 2024-04-12T05:09:12.479 INFO:tasks.workunit.client.1.smithi192.stdout:3/779: truncate d5/d13/d3c/d26/f9e 918252 0 2024-04-12T05:09:12.480 INFO:tasks.workunit.client.1.smithi192.stdout:1/818: rename d0/d1b/d28/d9d/fd9 to d0/d2/d1e/d3f/df2/d3c/d5b/db6/deb/f112 0 2024-04-12T05:09:12.481 INFO:tasks.workunit.client.0.smithi113.stdout:0/942: creat d1/d43/d59/de9/d105/d50/db6/f123 x:0 0 0 2024-04-12T05:09:12.483 INFO:tasks.workunit.client.1.smithi192.stdout:6/862: creat d3/d8/d56/d59/d8a/f11c x:0 0 0 2024-04-12T05:09:12.485 INFO:tasks.workunit.client.0.smithi113.stdout:3/839: write d0/d13/d15/d28/d2a/d34/f72 [166635,63370] 0 2024-04-12T05:09:12.488 INFO:tasks.workunit.client.0.smithi113.stdout:3/840: stat d0/d13/d15/d28/d2a/d34/l3a 0 2024-04-12T05:09:12.490 INFO:tasks.workunit.client.0.smithi113.stdout:3/841: readlink d0/d13/d15/d28/d2a/d34/d5f/d7b/lde 0 2024-04-12T05:09:12.491 INFO:tasks.workunit.client.0.smithi113.stdout:0/943: creat d1/d6/f124 x:0 0 0 2024-04-12T05:09:12.492 INFO:tasks.workunit.client.1.smithi192.stdout:1/819: mkdir d0/d2/d1e/d113 0 2024-04-12T05:09:12.494 INFO:tasks.workunit.client.0.smithi113.stdout:8/907: dwrite d9/d1b/d8e/d93/d32/d48/f69 [0,4194304] 0 2024-04-12T05:09:12.495 INFO:tasks.workunit.client.0.smithi113.stdout:8/908: fdatasync d9/d1b/d96/fe7 0 2024-04-12T05:09:12.496 INFO:tasks.workunit.client.1.smithi192.stdout:7/715: truncate d3/d1d/fe1 1188036 0 2024-04-12T05:09:12.497 INFO:tasks.workunit.client.1.smithi192.stdout:7/716: dread - d3/d1d/d4a/d5e/fca zero size 2024-04-12T05:09:12.499 INFO:tasks.workunit.client.1.smithi192.stdout:7/717: chown d3/d1d/d5a/d5f/d7a/da6/cea 8052407 1 2024-04-12T05:09:12.502 INFO:tasks.workunit.client.0.smithi113.stdout:3/842: dread d0/d13/d47/fb2 [0,4194304] 0 2024-04-12T05:09:12.505 INFO:tasks.workunit.client.1.smithi192.stdout:1/820: creat d0/d2/d1e/d3f/df2/d3c/d5b/d7c/f114 x:0 0 0 2024-04-12T05:09:12.506 INFO:tasks.workunit.client.0.smithi113.stdout:0/944: fsync d1/d43/d59/de9/d105/d50/f99 0 2024-04-12T05:09:12.507 INFO:tasks.workunit.client.0.smithi113.stdout:8/909: dwrite d9/d16/d65/db1/d59/d78/f116 [0,4194304] 0 2024-04-12T05:09:12.509 INFO:tasks.workunit.client.1.smithi192.stdout:6/863: link d3/d23/fef d3/d8/d56/d59/d97/dbf/f11d 0 2024-04-12T05:09:12.511 INFO:tasks.workunit.client.0.smithi113.stdout:3/843: rename d0/d13/d47/d94/fc3 to d0/d13/f118 0 2024-04-12T05:09:12.512 INFO:tasks.workunit.client.1.smithi192.stdout:1/821: mkdir d0/d2/d1e/d3f/df2/d21/d62/d115 0 2024-04-12T05:09:12.512 INFO:tasks.workunit.client.1.smithi192.stdout:9/671: sync 2024-04-12T05:09:12.512 INFO:tasks.workunit.client.1.smithi192.stdout:2/779: sync 2024-04-12T05:09:12.512 INFO:tasks.workunit.client.1.smithi192.stdout:0/903: sync 2024-04-12T05:09:12.512 INFO:tasks.workunit.client.1.smithi192.stdout:4/824: sync 2024-04-12T05:09:12.513 INFO:tasks.workunit.client.0.smithi113.stdout:0/945: creat d1/d43/d59/de9/d105/d50/db6/d36/f125 x:0 0 0 2024-04-12T05:09:12.514 INFO:tasks.workunit.client.1.smithi192.stdout:6/864: symlink d3/d8/d56/d59/ddf/l11e 0 2024-04-12T05:09:12.515 INFO:tasks.workunit.client.1.smithi192.stdout:7/718: truncate d3/d7/f19 7139056 0 2024-04-12T05:09:12.515 INFO:tasks.workunit.client.1.smithi192.stdout:4/825: chown d1/d1d/d53/d11a/f106 735487063 1 2024-04-12T05:09:12.519 INFO:tasks.workunit.client.1.smithi192.stdout:3/780: truncate d5/d13/d1a/d53/f58 308413 0 2024-04-12T05:09:12.522 INFO:tasks.workunit.client.1.smithi192.stdout:6/865: read d3/d8/d10/d4c/f62 [2254313,21122] 0 2024-04-12T05:09:12.523 INFO:tasks.workunit.client.0.smithi113.stdout:0/946: rename d1/d43/d59/de9/d105/d50/f10c to d1/d43/d59/db2/f126 0 2024-04-12T05:09:12.525 INFO:tasks.workunit.client.1.smithi192.stdout:0/904: dwrite d0/d7b/d9b/d35/d109/d95/db4/f102 [0,4194304] 0 2024-04-12T05:09:12.528 INFO:tasks.workunit.client.0.smithi113.stdout:3/844: dread d0/d13/d47/d4f/f35 [4194304,4194304] 0 2024-04-12T05:09:12.529 INFO:tasks.workunit.client.0.smithi113.stdout:0/947: creat d1/d6/d16/dd5/dbb/f127 x:0 0 0 2024-04-12T05:09:12.530 INFO:tasks.workunit.client.0.smithi113.stdout:8/910: dwrite d9/d16/d65/db1/d59/d78/fcf [0,4194304] 0 2024-04-12T05:09:12.532 INFO:tasks.workunit.client.1.smithi192.stdout:3/781: dwrite d5/d13/d1a/d5f/d103/f107 [0,4194304] 0 2024-04-12T05:09:12.532 INFO:tasks.workunit.client.1.smithi192.stdout:9/672: creat d11/d25/dec/d1f/d6a/ff2 x:0 0 0 2024-04-12T05:09:12.533 INFO:tasks.workunit.client.1.smithi192.stdout:6/866: dwrite d3/d5/f72 [0,4194304] 0 2024-04-12T05:09:12.534 INFO:tasks.workunit.client.0.smithi113.stdout:3/845: creat d0/d13/d47/f119 x:0 0 0 2024-04-12T05:09:12.538 INFO:tasks.workunit.client.0.smithi113.stdout:8/911: write d9/f9d [687863,59720] 0 2024-04-12T05:09:12.542 INFO:tasks.workunit.client.1.smithi192.stdout:9/673: mkdir d11/d25/d36/d6e/d33/df3 0 2024-04-12T05:09:12.543 INFO:tasks.workunit.client.1.smithi192.stdout:6/867: truncate d3/d8/d48/fd1 332479 0 2024-04-12T05:09:12.544 INFO:tasks.workunit.client.1.smithi192.stdout:1/822: link d0/d2/d1e/d3f/df2/d71/cc8 d0/d2/d1e/d3f/df2/d3c/d5b/db6/c116 0 2024-04-12T05:09:12.545 INFO:tasks.workunit.client.1.smithi192.stdout:0/905: creat d0/d7b/d9b/d4b/f133 x:0 0 0 2024-04-12T05:09:12.547 INFO:tasks.workunit.client.0.smithi113.stdout:3/846: dwrite d0/d7e/da3/fb9 [0,4194304] 0 2024-04-12T05:09:12.549 INFO:tasks.workunit.client.0.smithi113.stdout:3/847: chown d0/d13/d15/d28/l73 13 1 2024-04-12T05:09:12.552 INFO:tasks.workunit.client.1.smithi192.stdout:3/782: dwrite d5/d13/f2f [4194304,4194304] 0 2024-04-12T05:09:12.556 INFO:tasks.workunit.client.1.smithi192.stdout:6/868: getdents d3/d8/d22/dd4/de9 0 2024-04-12T05:09:12.557 INFO:tasks.workunit.client.1.smithi192.stdout:2/780: dread d1/d2d/f3d [0,4194304] 0 2024-04-12T05:09:12.557 INFO:tasks.workunit.client.0.smithi113.stdout:3/848: rename d0/d13/d47/d4f/f4c to d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/f11a 0 2024-04-12T05:09:12.560 INFO:tasks.workunit.client.1.smithi192.stdout:9/674: dwrite f3 [0,4194304] 0 2024-04-12T05:09:12.560 INFO:tasks.workunit.client.0.smithi113.stdout:3/849: creat d0/d13/d47/d94/dcf/dd1/de2/f11b x:0 0 0 2024-04-12T05:09:12.563 INFO:tasks.workunit.client.1.smithi192.stdout:8/777: sync 2024-04-12T05:09:12.564 INFO:tasks.workunit.client.0.smithi113.stdout:8/912: truncate d9/d1b/f45 6089591 0 2024-04-12T05:09:12.566 INFO:tasks.workunit.client.0.smithi113.stdout:3/850: link d0/d13/d15/d28/d2a/d34/fbd d0/d13/d47/d4f/dd3/f11c 0 2024-04-12T05:09:12.567 INFO:tasks.workunit.client.1.smithi192.stdout:5/899: sync 2024-04-12T05:09:12.568 INFO:tasks.workunit.client.1.smithi192.stdout:5/900: truncate d0/d15/d3e/dca/dda/da7/f108 683635 0 2024-04-12T05:09:12.580 INFO:tasks.workunit.client.0.smithi113.stdout:5/970: sync 2024-04-12T05:09:12.580 INFO:tasks.workunit.client.0.smithi113.stdout:9/888: sync 2024-04-12T05:09:12.584 INFO:tasks.workunit.client.0.smithi113.stdout:8/913: rename c4 to d9/d16/d65/db1/d59/da3/c11f 0 2024-04-12T05:09:12.587 INFO:tasks.workunit.client.0.smithi113.stdout:0/948: dread d1/d6/d16/dd5/dbb/fc0 [0,4194304] 0 2024-04-12T05:09:12.587 INFO:tasks.workunit.client.0.smithi113.stdout:3/851: chown d0/d13/d15/f2e 0 1 2024-04-12T05:09:12.588 INFO:tasks.workunit.client.0.smithi113.stdout:9/889: chown d1/d4/d5/d22/cc8 57 1 2024-04-12T05:09:12.592 INFO:tasks.workunit.client.0.smithi113.stdout:5/971: symlink d0/d10/d27/d1d/d3f/d73/dcd/l149 0 2024-04-12T05:09:12.593 INFO:tasks.workunit.client.1.smithi192.stdout:3/783: chown d5/d13/d3a/d4e/f9a 0 1 2024-04-12T05:09:12.594 INFO:tasks.workunit.client.0.smithi113.stdout:8/914: mknod d9/d16/d65/db1/d59/da3/d10a/c120 0 2024-04-12T05:09:12.595 INFO:tasks.workunit.client.0.smithi113.stdout:0/949: write d1/d6/d16/dd5/fc7 [4010194,105990] 0 2024-04-12T05:09:12.595 INFO:tasks.workunit.client.0.smithi113.stdout:5/972: symlink d0/d10/d27/d1d/d38/d44/dcb/l14a 0 2024-04-12T05:09:12.598 INFO:tasks.workunit.client.0.smithi113.stdout:8/915: unlink d9/d1b/f4f 0 2024-04-12T05:09:12.599 INFO:tasks.workunit.client.1.smithi192.stdout:7/719: truncate d3/d1d/d5a/d5f/f30 1804486 0 2024-04-12T05:09:12.599 INFO:tasks.workunit.client.1.smithi192.stdout:8/778: readlink d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/l76 0 2024-04-12T05:09:12.599 INFO:tasks.workunit.client.0.smithi113.stdout:3/852: creat d0/d13/d15/d28/d2a/d34/f11d x:0 0 0 2024-04-12T05:09:12.601 INFO:tasks.workunit.client.1.smithi192.stdout:7/720: dread d3/d1d/d74/d86/d93/fd9 [0,4194304] 0 2024-04-12T05:09:12.602 INFO:tasks.workunit.client.0.smithi113.stdout:3/853: truncate d0/d13/d47/d94/dcf/dd1/de2/f107 273622 0 2024-04-12T05:09:12.603 INFO:tasks.workunit.client.1.smithi192.stdout:2/781: mkdir d1/d8/d17/d19/d60/df7 0 2024-04-12T05:09:12.603 INFO:tasks.workunit.client.1.smithi192.stdout:4/826: truncate d1/d54/d55/d64/f8f 2269197 0 2024-04-12T05:09:12.604 INFO:tasks.workunit.client.0.smithi113.stdout:0/950: mknod d1/d43/d59/de9/d105/d50/d52/c128 0 2024-04-12T05:09:12.608 INFO:tasks.workunit.client.0.smithi113.stdout:0/951: dread d1/d6/d16/dd5/dbb/fc0 [0,4194304] 0 2024-04-12T05:09:12.609 INFO:tasks.workunit.client.1.smithi192.stdout:9/675: rename d11/d25/d36/d6e/d33/l64 to d11/d25/dec/d1f/d6a/d7e/d89/dce/lf4 0 2024-04-12T05:09:12.609 INFO:tasks.workunit.client.1.smithi192.stdout:9/676: chown d11/d25/ff1 41101752 1 2024-04-12T05:09:12.610 INFO:tasks.workunit.client.1.smithi192.stdout:9/677: read - d11/d25/d36/d6e/fd2 zero size 2024-04-12T05:09:12.611 INFO:tasks.workunit.client.0.smithi113.stdout:3/854: creat d0/d13/d47/f11e x:0 0 0 2024-04-12T05:09:12.611 INFO:tasks.workunit.client.0.smithi113.stdout:3/855: stat d0/d7e/da3 0 2024-04-12T05:09:12.612 INFO:tasks.workunit.client.1.smithi192.stdout:1/823: write d0/d2/d1e/d3f/df2/d3c/dab/fb5 [525988,40399] 0 2024-04-12T05:09:12.612 INFO:tasks.workunit.client.0.smithi113.stdout:3/856: readlink d0/d13/d47/d94/l105 0 2024-04-12T05:09:12.614 INFO:tasks.workunit.client.1.smithi192.stdout:8/779: symlink d6/d10/d46/d5e/d5f/d31/d61/dc5/dd9/led 0 2024-04-12T05:09:12.614 INFO:tasks.workunit.client.1.smithi192.stdout:7/721: mknod d3/d1d/d4a/d5e/d6d/cf4 0 2024-04-12T05:09:12.616 INFO:tasks.workunit.client.0.smithi113.stdout:5/973: rename d0/d10/d27/d1d/d3f to d0/d10/d6e/d93/dc9/d14b 0 2024-04-12T05:09:12.617 INFO:tasks.workunit.client.0.smithi113.stdout:3/857: readlink d0/d13/d47/d94/dcf/dd1/l10c 0 2024-04-12T05:09:12.618 INFO:tasks.workunit.client.1.smithi192.stdout:0/906: link d0/f20 d0/d7b/d9b/d35/d109/d95/db4/db8/f134 0 2024-04-12T05:09:12.618 INFO:tasks.workunit.client.0.smithi113.stdout:5/974: chown d0/d10/d27/d1d/d38/d44/dcb/fe1 13394299 1 2024-04-12T05:09:12.618 INFO:tasks.workunit.client.0.smithi113.stdout:8/916: rename d9/d16/d65/db1 to d9/d16/d65/d121 0 2024-04-12T05:09:12.619 INFO:tasks.workunit.client.0.smithi113.stdout:8/917: chown d9/d1b/f61 19 1 2024-04-12T05:09:12.620 INFO:tasks.workunit.client.1.smithi192.stdout:2/782: creat d1/d8/d17/db3/dee/ff8 x:0 0 0 2024-04-12T05:09:12.620 INFO:tasks.workunit.client.1.smithi192.stdout:0/907: stat d0/d7b/d9b/d35/d109/d53/df3/d2e/d59/f131 0 2024-04-12T05:09:12.622 INFO:tasks.workunit.client.0.smithi113.stdout:8/918: write d9/d1b/ff8 [155777,23395] 0 2024-04-12T05:09:12.623 INFO:tasks.workunit.client.0.smithi113.stdout:5/975: chown d0/d10/d6e/d93/dc9/d14b/l10a 1115 1 2024-04-12T05:09:12.624 INFO:tasks.workunit.client.1.smithi192.stdout:8/780: read d6/d10/d46/d5e/d5f/d37/d4f/f9a [1214895,34466] 0 2024-04-12T05:09:12.624 INFO:tasks.workunit.client.1.smithi192.stdout:7/722: dread d3/f4f [0,4194304] 0 2024-04-12T05:09:12.625 INFO:tasks.workunit.client.0.smithi113.stdout:5/976: write d0/f31 [898776,104349] 0 2024-04-12T05:09:12.626 INFO:tasks.workunit.client.1.smithi192.stdout:6/869: dwrite d3/dc/f9a [0,4194304] 0 2024-04-12T05:09:12.626 INFO:tasks.workunit.client.0.smithi113.stdout:3/858: dwrite d0/d13/d15/d28/d2a/f2b [0,4194304] 0 2024-04-12T05:09:12.629 INFO:tasks.workunit.client.1.smithi192.stdout:9/678: mkdir d11/d25/d36/d6e/d33/dbb/df5 0 2024-04-12T05:09:12.630 INFO:tasks.workunit.client.1.smithi192.stdout:8/781: write d6/d10/d46/d5e/d5f/d37/d4f/d51/f66 [1116907,77059] 0 2024-04-12T05:09:12.630 INFO:tasks.workunit.client.1.smithi192.stdout:1/824: mknod d0/d2/d1e/d3f/df2/d21/d62/d84/c117 0 2024-04-12T05:09:12.637 INFO:tasks.workunit.client.1.smithi192.stdout:0/908: dwrite d0/d7b/d9b/d35/d109/d53/df3/d2e/f9e [0,4194304] 0 2024-04-12T05:09:12.639 INFO:tasks.workunit.client.0.smithi113.stdout:3/859: chown d0/d13/d47/d94/dcf 29090824 1 2024-04-12T05:09:12.639 INFO:tasks.workunit.client.0.smithi113.stdout:8/919: fdatasync d9/d1b/d8e/d93/faf 0 2024-04-12T05:09:12.639 INFO:tasks.workunit.client.0.smithi113.stdout:3/860: fsync d0/d13/d15/f3d 0 2024-04-12T05:09:12.640 INFO:tasks.workunit.client.0.smithi113.stdout:3/861: chown d0/d13/d15/d28/l58 43711 1 2024-04-12T05:09:12.641 INFO:tasks.workunit.client.1.smithi192.stdout:3/784: mknod d5/d13/d3a/d4a/d61/d9f/deb/dfb/d114/c119 0 2024-04-12T05:09:12.641 INFO:tasks.workunit.client.1.smithi192.stdout:5/901: getdents d0/d15/d3e/dca/dda/db5/dd7/d116 0 2024-04-12T05:09:12.642 INFO:tasks.workunit.client.1.smithi192.stdout:3/785: read d5/d13/f2f [815291,117626] 0 2024-04-12T05:09:12.642 INFO:tasks.workunit.client.0.smithi113.stdout:0/952: dwrite d1/d43/d59/de9/d105/d50/f99 [0,4194304] 0 2024-04-12T05:09:12.645 INFO:tasks.workunit.client.0.smithi113.stdout:0/953: write d1/d6/d16/f100 [547666,24250] 0 2024-04-12T05:09:12.646 INFO:tasks.workunit.client.0.smithi113.stdout:3/862: symlink d0/l11f 0 2024-04-12T05:09:12.647 INFO:tasks.workunit.client.1.smithi192.stdout:8/782: unlink d6/d10/d46/d7c/l8b 0 2024-04-12T05:09:12.648 INFO:tasks.workunit.client.1.smithi192.stdout:4/827: link d1/d54/fc5 d1/d1d/d53/d1c/d95/dc6/dd5/de5/d10d/f11b 0 2024-04-12T05:09:12.648 INFO:tasks.workunit.client.0.smithi113.stdout:5/977: creat d0/d10/d27/d1d/d38/d44/d4e/daa/dbb/f14c x:0 0 0 2024-04-12T05:09:12.650 INFO:tasks.workunit.client.1.smithi192.stdout:8/783: chown d6/d10/d46/d5e/d6e/l97 60146003 1 2024-04-12T05:09:12.651 INFO:tasks.workunit.client.0.smithi113.stdout:8/920: truncate d9/d1b/d8e/d93/d32/d5b/f63 410565 0 2024-04-12T05:09:12.651 INFO:tasks.workunit.client.0.smithi113.stdout:8/921: readlink d9/d16/d65/d121/l77 0 2024-04-12T05:09:12.654 INFO:tasks.workunit.client.0.smithi113.stdout:8/922: creat d9/dd4/dde/f122 x:0 0 0 2024-04-12T05:09:12.655 INFO:tasks.workunit.client.0.smithi113.stdout:0/954: mkdir d1/d43/d59/de9/d105/d50/d129 0 2024-04-12T05:09:12.656 INFO:tasks.workunit.client.0.smithi113.stdout:3/863: read d0/d13/d15/d28/d2a/d34/d5f/f65 [840026,45739] 0 2024-04-12T05:09:12.656 INFO:tasks.workunit.client.0.smithi113.stdout:8/923: mknod d9/d16/c123 0 2024-04-12T05:09:12.658 INFO:tasks.workunit.client.0.smithi113.stdout:3/864: rename d0/d13/d15 to d0/d13/d15/d28/dc0/d120 22 2024-04-12T05:09:12.660 INFO:tasks.workunit.client.1.smithi192.stdout:5/902: unlink d0/d15/d3e/dca/dda/db5/c119 0 2024-04-12T05:09:12.662 INFO:tasks.workunit.client.0.smithi113.stdout:8/924: dread d9/d1b/d8e/d93/f66 [0,4194304] 0 2024-04-12T05:09:12.663 INFO:tasks.workunit.client.1.smithi192.stdout:2/783: mkdir d1/d8/d3a/df9 0 2024-04-12T05:09:12.664 INFO:tasks.workunit.client.0.smithi113.stdout:0/955: fdatasync d1/d43/d59/de9/d105/d50/d52/fa0 0 2024-04-12T05:09:12.665 INFO:tasks.workunit.client.0.smithi113.stdout:3/865: dwrite d0/d13/d47/d94/dac/f114 [0,4194304] 0 2024-04-12T05:09:12.666 INFO:tasks.workunit.client.1.smithi192.stdout:7/723: symlink d3/d7/lf5 0 2024-04-12T05:09:12.667 INFO:tasks.workunit.client.1.smithi192.stdout:5/903: dwrite d0/d15/d32/d45/ff0 [0,4194304] 0 2024-04-12T05:09:12.669 INFO:tasks.workunit.client.0.smithi113.stdout:3/866: mknod d0/d13/d15/d28/d2a/d8f/ddd/c121 0 2024-04-12T05:09:12.670 INFO:tasks.workunit.client.0.smithi113.stdout:0/956: mkdir d1/d6/d16/dd5/dd6/dde/d122/d12a 0 2024-04-12T05:09:12.671 INFO:tasks.workunit.client.0.smithi113.stdout:3/867: write d0/d13/d15/d28/ffe [748388,35447] 0 2024-04-12T05:09:12.671 INFO:tasks.workunit.client.1.smithi192.stdout:8/784: write d6/d1f/d5d/fa2 [175865,121204] 0 2024-04-12T05:09:12.673 INFO:tasks.workunit.client.1.smithi192.stdout:4/828: symlink d1/d1d/d53/d1c/dd1/def/ddd/d114/l11c 0 2024-04-12T05:09:12.676 INFO:tasks.workunit.client.1.smithi192.stdout:8/785: dread - d6/d10/d46/d5e/d5f/f88 zero size 2024-04-12T05:09:12.677 INFO:tasks.workunit.client.0.smithi113.stdout:0/957: truncate d1/d43/d59/de9/d105/f3f 3363355 0 2024-04-12T05:09:12.677 INFO:tasks.workunit.client.1.smithi192.stdout:5/904: dread d0/d15/d32/d69/f9e [0,4194304] 0 2024-04-12T05:09:12.678 INFO:tasks.workunit.client.0.smithi113.stdout:3/868: rename d0/d13/d15/cc5 to d0/d13/d15/d28/d2a/d34/d5f/c122 0 2024-04-12T05:09:12.678 INFO:tasks.workunit.client.0.smithi113.stdout:0/958: chown d1/d6/l115 46 1 2024-04-12T05:09:12.678 INFO:tasks.workunit.client.1.smithi192.stdout:2/784: fdatasync d1/d8/d17/d19/d60/d9c/fbe 0 2024-04-12T05:09:12.679 INFO:tasks.workunit.client.1.smithi192.stdout:1/825: creat d0/d1b/f118 x:0 0 0 2024-04-12T05:09:12.680 INFO:tasks.workunit.client.0.smithi113.stdout:3/869: dread - d0/d13/d15/d28/d2a/d8f/ddd/f115 zero size 2024-04-12T05:09:12.685 INFO:tasks.workunit.client.0.smithi113.stdout:0/959: mkdir d1/d6/d16/dd5/dd6/dde/d122/d12b 0 2024-04-12T05:09:12.685 INFO:tasks.workunit.client.1.smithi192.stdout:7/724: rename d3/d1d/d5a/d5f/d7a/da6/fdb to d3/d1d/d74/ff6 0 2024-04-12T05:09:12.686 INFO:tasks.workunit.client.0.smithi113.stdout:3/870: creat d0/d13/d15/d110/f123 x:0 0 0 2024-04-12T05:09:12.687 INFO:tasks.workunit.client.0.smithi113.stdout:3/871: chown d0/d13/d15/d28/f6f 743992 1 2024-04-12T05:09:12.687 INFO:tasks.workunit.client.1.smithi192.stdout:6/870: getdents d3/d8/d56/d59/ddf/df1 0 2024-04-12T05:09:12.687 INFO:tasks.workunit.client.1.smithi192.stdout:7/725: dread - d3/d1d/d4a/d5e/d6d/fba zero size 2024-04-12T05:09:12.688 INFO:tasks.workunit.client.0.smithi113.stdout:8/925: write d9/d16/d3c/f4e [1163074,128456] 0 2024-04-12T05:09:12.688 INFO:tasks.workunit.client.1.smithi192.stdout:8/786: creat d6/d1f/d9c/fee x:0 0 0 2024-04-12T05:09:12.690 INFO:tasks.workunit.client.0.smithi113.stdout:0/960: mknod d1/d6/d16/d8f/c12c 0 2024-04-12T05:09:12.690 INFO:tasks.workunit.client.0.smithi113.stdout:3/872: creat d0/d13/d15/d28/d2a/d8b/d91/f124 x:0 0 0 2024-04-12T05:09:12.691 INFO:tasks.workunit.client.1.smithi192.stdout:5/905: mknod d0/d15/d32/d45/c137 0 2024-04-12T05:09:12.693 INFO:tasks.workunit.client.1.smithi192.stdout:8/787: chown d6/d10/d46/d5e/d5f/d38/l84 496 1 2024-04-12T05:09:12.695 INFO:tasks.workunit.client.0.smithi113.stdout:0/961: creat d1/d6/d34/dc3/d87/f12d x:0 0 0 2024-04-12T05:09:12.698 INFO:tasks.workunit.client.1.smithi192.stdout:3/786: getdents d5/d13/d1a/d53 0 2024-04-12T05:09:12.698 INFO:tasks.workunit.client.0.smithi113.stdout:8/926: link d9/d1b/d8e/d93/d32/d48/c75 d9/d1b/d8e/d93/d32/d5b/c124 0 2024-04-12T05:09:12.699 INFO:tasks.workunit.client.1.smithi192.stdout:0/909: dwrite d0/d7b/d9b/d35/d109/d9f/d10a/d15/d19/f71 [4194304,4194304] 0 2024-04-12T05:09:12.701 INFO:tasks.workunit.client.0.smithi113.stdout:8/927: mknod d9/d16/d65/d121/dec/d111/c125 0 2024-04-12T05:09:12.702 INFO:tasks.workunit.client.1.smithi192.stdout:9/679: link d11/d15/l57 d11/d15/d21/lf6 0 2024-04-12T05:09:12.703 INFO:tasks.workunit.client.1.smithi192.stdout:5/906: creat d0/d15/d3e/dca/dda/d11c/f138 x:0 0 0 2024-04-12T05:09:12.703 INFO:tasks.workunit.client.0.smithi113.stdout:3/873: write d0/d13/d47/d4f/dd3/f11c [208407,19190] 0 2024-04-12T05:09:12.706 INFO:tasks.workunit.client.0.smithi113.stdout:8/928: dread d9/d16/d65/d121/f2c [0,4194304] 0 2024-04-12T05:09:12.707 INFO:tasks.workunit.client.1.smithi192.stdout:2/785: creat d1/d8/d17/ffa x:0 0 0 2024-04-12T05:09:12.708 INFO:tasks.workunit.client.1.smithi192.stdout:7/726: dwrite d3/d7/f89 [0,4194304] 0 2024-04-12T05:09:12.711 INFO:tasks.workunit.client.0.smithi113.stdout:3/874: dwrite d0/d13/d47/d94/dcf/ff2 [0,4194304] 0 2024-04-12T05:09:12.713 INFO:tasks.workunit.client.0.smithi113.stdout:3/875: stat d0/d13/d15/d28/d2a/d34/d5f/f96 0 2024-04-12T05:09:12.718 INFO:tasks.workunit.client.1.smithi192.stdout:4/829: rename d1/d54/f99 to d1/d1d/d53/d1c/d95/dc6/dd5/de5/f11d 0 2024-04-12T05:09:12.719 INFO:tasks.workunit.client.0.smithi113.stdout:3/876: fdatasync d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/f11a 0 2024-04-12T05:09:12.720 INFO:tasks.workunit.client.1.smithi192.stdout:2/786: stat d1/d8/d17/fed 0 2024-04-12T05:09:12.721 INFO:tasks.workunit.client.1.smithi192.stdout:8/788: dwrite d6/d10/d46/d5e/d5f/fab [0,4194304] 0 2024-04-12T05:09:12.724 INFO:tasks.workunit.client.1.smithi192.stdout:5/907: dread d0/d15/d24/ddf/fe8 [0,4194304] 0 2024-04-12T05:09:12.725 INFO:tasks.workunit.client.0.smithi113.stdout:0/962: write d1/d43/d59/de9/d105/d50/f77 [4103816,7942] 0 2024-04-12T05:09:12.728 INFO:tasks.workunit.client.0.smithi113.stdout:3/877: dwrite d0/d13/d15/d28/d2a/d34/f72 [0,4194304] 0 2024-04-12T05:09:12.728 INFO:tasks.workunit.client.1.smithi192.stdout:2/787: read d1/d8/d3a/de/f30 [3263986,44089] 0 2024-04-12T05:09:12.731 INFO:tasks.workunit.client.1.smithi192.stdout:9/680: dread d11/d25/d36/d6e/d33/da2/fc9 [0,4194304] 0 2024-04-12T05:09:12.732 INFO:tasks.workunit.client.1.smithi192.stdout:1/826: dwrite d0/d2/d1e/d3f/df2/d21/d39/f82 [0,4194304] 0 2024-04-12T05:09:12.732 INFO:tasks.workunit.client.0.smithi113.stdout:0/963: mkdir d1/d6/d16/dd5/dd6/d12e 0 2024-04-12T05:09:12.733 INFO:tasks.workunit.client.1.smithi192.stdout:1/827: stat d0/d2/d1e/d3f/df2/f32 0 2024-04-12T05:09:12.734 INFO:tasks.workunit.client.0.smithi113.stdout:0/964: chown d1/d43/d59/de9/d105/d50/d52/d7e/f8d 8 1 2024-04-12T05:09:12.735 INFO:tasks.workunit.client.0.smithi113.stdout:0/965: stat d1/d6/d16/d8f 0 2024-04-12T05:09:12.741 INFO:tasks.workunit.client.0.smithi113.stdout:0/966: rename d1/d43/d59/de9/d105/d50/d52/c128 to d1/d43/c12f 0 2024-04-12T05:09:12.751 INFO:tasks.workunit.client.0.smithi113.stdout:0/967: link d1/d6/f30 d1/f130 0 2024-04-12T05:09:12.754 INFO:tasks.workunit.client.0.smithi113.stdout:0/968: fdatasync d1/d43/d59/de9/d105/d50/f85 0 2024-04-12T05:09:12.756 INFO:tasks.workunit.client.1.smithi192.stdout:3/787: symlink d5/d13/d3a/d70/d8c/de1/l11a 0 2024-04-12T05:09:12.757 INFO:tasks.workunit.client.0.smithi113.stdout:7/822: sync 2024-04-12T05:09:12.757 INFO:tasks.workunit.client.1.smithi192.stdout:3/788: chown d5/d13/d3a/d70/d8c/cd4 90295150 1 2024-04-12T05:09:12.758 INFO:tasks.workunit.client.1.smithi192.stdout:3/789: stat d5/d13/d3c/d26 0 2024-04-12T05:09:12.761 INFO:tasks.workunit.client.1.smithi192.stdout:8/789: unlink d6/d10/d46/d5e/d5f/c62 0 2024-04-12T05:09:12.761 INFO:tasks.workunit.client.1.smithi192.stdout:8/790: fsync d6/da/fba 0 2024-04-12T05:09:12.763 INFO:tasks.workunit.client.1.smithi192.stdout:4/830: mkdir d1/d54/d82/d71/d9e/d10b/de9/d11e 0 2024-04-12T05:09:12.764 INFO:tasks.workunit.client.0.smithi113.stdout:0/969: rmdir d1/d6/d34 39 2024-04-12T05:09:12.764 INFO:tasks.workunit.client.1.smithi192.stdout:4/831: fdatasync d1/d54/d82/d71/d9e/f10e 0 2024-04-12T05:09:12.764 INFO:tasks.workunit.client.0.smithi113.stdout:3/878: dread d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/f11a [0,4194304] 0 2024-04-12T05:09:12.766 INFO:tasks.workunit.client.1.smithi192.stdout:5/908: symlink d0/d15/d24/d47/l139 0 2024-04-12T05:09:12.769 INFO:tasks.workunit.client.1.smithi192.stdout:1/828: unlink d0/d2/d1e/d3f/df2/d3c/f7f 0 2024-04-12T05:09:12.770 INFO:tasks.workunit.client.0.smithi113.stdout:3/879: rename d0/d7e/da3/cb5 to d0/d80/c125 0 2024-04-12T05:09:12.770 INFO:tasks.workunit.client.1.smithi192.stdout:0/910: symlink d0/d7b/d9b/d35/d109/d9f/d10a/d67/l135 0 2024-04-12T05:09:12.772 INFO:tasks.workunit.client.1.smithi192.stdout:3/790: symlink d5/d13/d1a/d5f/l11b 0 2024-04-12T05:09:12.774 INFO:tasks.workunit.client.0.smithi113.stdout:3/880: rename d0/d13/d15/d28/d2a/d8b/lc7 to d0/d13/d47/d4f/dd3/l126 0 2024-04-12T05:09:12.775 INFO:tasks.workunit.client.0.smithi113.stdout:3/881: chown d0/le 4 1 2024-04-12T05:09:12.775 INFO:tasks.workunit.client.1.smithi192.stdout:2/788: creat d1/d8/d3a/df9/ffb x:0 0 0 2024-04-12T05:09:12.775 INFO:tasks.workunit.client.1.smithi192.stdout:2/789: chown d1/d8/d17/fd0 1981 1 2024-04-12T05:09:12.777 INFO:tasks.workunit.client.1.smithi192.stdout:2/790: fsync d1/d8/d3a/de/d7c/fb8 0 2024-04-12T05:09:12.777 INFO:tasks.workunit.client.0.smithi113.stdout:3/882: chown d0/d13/d15/cd9 312142 1 2024-04-12T05:09:12.777 INFO:tasks.workunit.client.0.smithi113.stdout:3/883: chown d0/fa8 442078693 1 2024-04-12T05:09:12.778 INFO:tasks.workunit.client.1.smithi192.stdout:6/871: link d3/d8/d10/d4c/lbb d3/d8/d56/d59/d97/l11f 0 2024-04-12T05:09:12.780 INFO:tasks.workunit.client.1.smithi192.stdout:3/791: mkdir d5/d13/d3a/d4a/d61/d79/d11c 0 2024-04-12T05:09:12.781 INFO:tasks.workunit.client.0.smithi113.stdout:3/884: rename d0/d13/d47/d94/dac/f114 to d0/d13/d47/d94/dcf/dd1/de2/f127 0 2024-04-12T05:09:12.782 INFO:tasks.workunit.client.1.smithi192.stdout:5/909: mkdir d0/d15/d3e/dca/dda/db5/dbe/d118/d13a 0 2024-04-12T05:09:12.783 INFO:tasks.workunit.client.0.smithi113.stdout:3/885: write d0/d13/d47/d94/dcf/dd1/de2/fe6 [4128778,30602] 0 2024-04-12T05:09:12.784 INFO:tasks.workunit.client.1.smithi192.stdout:9/681: rename d11/d25/d36/d6e/d33/dbb/fe3 to d11/d15/d21/ff7 0 2024-04-12T05:09:12.787 INFO:tasks.workunit.client.0.smithi113.stdout:3/886: getdents d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/d106 0 2024-04-12T05:09:12.790 INFO:tasks.workunit.client.1.smithi192.stdout:5/910: unlink d0/d15/d32/d45/d106/l124 0 2024-04-12T05:09:12.794 INFO:tasks.workunit.client.1.smithi192.stdout:4/832: getdents d1/d1d/d53/d1c/d95/dc6/dd5/de5 0 2024-04-12T05:09:12.799 INFO:tasks.workunit.client.1.smithi192.stdout:9/682: mknod d11/d25/d36/d6e/d33/dbb/df5/cf8 0 2024-04-12T05:09:12.802 INFO:tasks.workunit.client.1.smithi192.stdout:5/911: mknod d0/d15/d3e/dca/dda/deb/d113/d127/ded/d136/c13b 0 2024-04-12T05:09:12.803 INFO:tasks.workunit.client.1.smithi192.stdout:2/791: getdents d1/d8/d17 0 2024-04-12T05:09:12.805 INFO:tasks.workunit.client.1.smithi192.stdout:2/792: read d1/d8/f46 [2096375,108544] 0 2024-04-12T05:09:12.806 INFO:tasks.workunit.client.1.smithi192.stdout:8/791: rename d6/d10/d46/d5e/d5f/d37/d4f/d4c/fdf to d6/d10/fef 0 2024-04-12T05:09:12.808 INFO:tasks.workunit.client.1.smithi192.stdout:2/793: write d1/d8/d17/d19/d60/f65 [2300168,55303] 0 2024-04-12T05:09:12.809 INFO:tasks.workunit.client.1.smithi192.stdout:5/912: truncate d0/d15/d32/d45/fb0 131527 0 2024-04-12T05:09:12.811 INFO:tasks.workunit.client.1.smithi192.stdout:6/872: rename d3/l113 to d3/d8/d56/d59/ddf/l120 0 2024-04-12T05:09:12.812 INFO:tasks.workunit.client.1.smithi192.stdout:5/913: read d0/d15/d3e/d77/ffd [4570541,97019] 0 2024-04-12T05:09:12.813 INFO:tasks.workunit.client.1.smithi192.stdout:7/727: dwrite d3/d11/f16 [0,4194304] 0 2024-04-12T05:09:12.813 INFO:tasks.workunit.client.1.smithi192.stdout:8/792: mknod d6/d10/d46/d5e/d5f/d31/d61/d9d/db8/cf0 0 2024-04-12T05:09:12.814 INFO:tasks.workunit.client.1.smithi192.stdout:2/794: rmdir d1/d8/d17/d19/d60/d59 39 2024-04-12T05:09:12.819 INFO:tasks.workunit.client.1.smithi192.stdout:6/873: dread d3/d23/f78 [0,4194304] 0 2024-04-12T05:09:12.820 INFO:tasks.workunit.client.1.smithi192.stdout:2/795: creat d1/da7/ffc x:0 0 0 2024-04-12T05:09:12.823 INFO:tasks.workunit.client.1.smithi192.stdout:5/914: rename d0/d15/d3e/dca/c10b to d0/d15/d3e/dca/dd3/df3/c13c 0 2024-04-12T05:09:12.830 INFO:tasks.workunit.client.1.smithi192.stdout:3/792: truncate d5/fd8 2978815 0 2024-04-12T05:09:12.830 INFO:tasks.workunit.client.1.smithi192.stdout:8/793: mknod d6/d10/d46/d5e/d5f/d31/d61/dc5/dd9/dd7/cf1 0 2024-04-12T05:09:12.831 INFO:tasks.workunit.client.1.smithi192.stdout:4/833: write d1/fb6 [4330708,96835] 0 2024-04-12T05:09:12.834 INFO:tasks.workunit.client.1.smithi192.stdout:9/683: truncate d11/d25/d36/d6e/d33/fab 3478153 0 2024-04-12T05:09:12.837 INFO:tasks.workunit.client.1.smithi192.stdout:4/834: read - d1/d1d/d53/d1c/d84/fa9 zero size 2024-04-12T05:09:12.838 INFO:tasks.workunit.client.1.smithi192.stdout:8/794: dwrite d6/d10/d46/d7c/f8c [0,4194304] 0 2024-04-12T05:09:12.839 INFO:tasks.workunit.client.1.smithi192.stdout:4/835: fdatasync d1/d54/d82/d71/d9e/dc4/dd9/fee 0 2024-04-12T05:09:12.840 INFO:tasks.workunit.client.1.smithi192.stdout:2/796: write d1/d8/f10 [2956064,105218] 0 2024-04-12T05:09:12.842 INFO:tasks.workunit.client.1.smithi192.stdout:4/836: chown d1/f2 66128272 1 2024-04-12T05:09:12.843 INFO:tasks.workunit.client.1.smithi192.stdout:5/915: dread - d0/d15/d3e/dca/dda/db5/dbe/f11d zero size 2024-04-12T05:09:12.844 INFO:tasks.workunit.client.1.smithi192.stdout:4/837: stat d1/f29 0 2024-04-12T05:09:12.844 INFO:tasks.workunit.client.1.smithi192.stdout:8/795: creat d6/d10/d46/d5e/d5f/d31/d61/ff2 x:0 0 0 2024-04-12T05:09:12.849 INFO:tasks.workunit.client.1.smithi192.stdout:8/796: chown d6/d10/d46/d7c/f8c 14 1 2024-04-12T05:09:12.850 INFO:tasks.workunit.client.1.smithi192.stdout:7/728: dwrite d3/d1d/d4a/f70 [0,4194304] 0 2024-04-12T05:09:12.851 INFO:tasks.workunit.client.1.smithi192.stdout:3/793: write d5/d13/d1a/d5f/d103/ffc [910893,96593] 0 2024-04-12T05:09:12.855 INFO:tasks.workunit.client.1.smithi192.stdout:2/797: link d1/d8/d3a/de/d7c/d98/fe4 d1/d8/d3a/de/d7c/ffd 0 2024-04-12T05:09:12.857 INFO:tasks.workunit.client.1.smithi192.stdout:3/794: dread - d5/d13/d1a/d5f/d103/fb7 zero size 2024-04-12T05:09:12.858 INFO:tasks.workunit.client.1.smithi192.stdout:8/797: truncate d6/d10/d46/d5e/faf 877782 0 2024-04-12T05:09:12.858 INFO:tasks.workunit.client.1.smithi192.stdout:4/838: dwrite d1/d1d/fad [0,4194304] 0 2024-04-12T05:09:12.863 INFO:tasks.workunit.client.1.smithi192.stdout:7/729: rename d3/d7/d15/f24 to d3/d1d/d74/d86/d93/ff7 0 2024-04-12T05:09:12.864 INFO:tasks.workunit.client.1.smithi192.stdout:5/916: dwrite d0/d15/d24/d47/f54 [0,4194304] 0 2024-04-12T05:09:12.869 INFO:tasks.workunit.client.1.smithi192.stdout:8/798: creat d6/d10/d6c/db4/ff3 x:0 0 0 2024-04-12T05:09:12.869 INFO:tasks.workunit.client.1.smithi192.stdout:7/730: mknod d3/d1d/d5a/cf8 0 2024-04-12T05:09:12.912 INFO:tasks.workunit.client.1.smithi192.stdout:3/795: creat d5/d13/d3a/d70/d8c/de1/d10e/f11d x:0 0 0 2024-04-12T05:09:12.912 INFO:tasks.workunit.client.1.smithi192.stdout:7/731: write d3/d7/d15/f29 [5344884,58638] 0 2024-04-12T05:09:12.914 INFO:tasks.workunit.client.1.smithi192.stdout:8/799: symlink d6/lf4 0 2024-04-12T05:09:12.916 INFO:tasks.workunit.client.1.smithi192.stdout:3/796: chown d5/d13/d1a/fa9 120684 1 2024-04-12T05:09:12.917 INFO:tasks.workunit.client.1.smithi192.stdout:8/800: readlink d6/d10/d46/d5e/d5f/l1b 0 2024-04-12T05:09:12.917 INFO:tasks.workunit.client.1.smithi192.stdout:7/732: mkdir d3/d1d/d4a/d5e/df9 0 2024-04-12T05:09:12.918 INFO:tasks.workunit.client.1.smithi192.stdout:5/917: dwrite d0/d15/d32/d69/d7d/fac [0,4194304] 0 2024-04-12T05:09:12.922 INFO:tasks.workunit.client.1.smithi192.stdout:8/801: mkdir d6/d10/d46/d5e/d5f/d37/d4f/d4c/df5 0 2024-04-12T05:09:12.924 INFO:tasks.workunit.client.1.smithi192.stdout:5/918: getdents d0/d15/d3e/dca 0 2024-04-12T05:09:12.925 INFO:tasks.workunit.client.1.smithi192.stdout:3/797: truncate d5/d13/d3a/d4e/d57/f6e 476986 0 2024-04-12T05:09:12.929 INFO:tasks.workunit.client.1.smithi192.stdout:3/798: rmdir d5/d13/d3a/d4a 39 2024-04-12T05:09:12.929 INFO:tasks.workunit.client.1.smithi192.stdout:2/798: dwrite d1/d8/d17/d19/d40/fbd [0,4194304] 0 2024-04-12T05:09:12.930 INFO:tasks.workunit.client.1.smithi192.stdout:7/733: fsync d3/d1d/d4a/f70 0 2024-04-12T05:09:12.930 INFO:tasks.workunit.client.1.smithi192.stdout:5/919: write d0/d15/d3e/dca/dda/f103 [517086,80171] 0 2024-04-12T05:09:12.930 INFO:tasks.workunit.client.1.smithi192.stdout:8/802: symlink d6/d10/d46/d5e/d5f/d31/d80/db1/lf6 0 2024-04-12T05:09:12.933 INFO:tasks.workunit.client.1.smithi192.stdout:8/803: chown d6/c25 209682924 1 2024-04-12T05:09:12.933 INFO:tasks.workunit.client.1.smithi192.stdout:7/734: dread - d3/d1d/d5a/d5f/fd2 zero size 2024-04-12T05:09:12.939 INFO:tasks.workunit.client.0.smithi113.stdout:1/961: sync 2024-04-12T05:09:12.939 INFO:tasks.workunit.client.0.smithi113.stdout:2/844: sync 2024-04-12T05:09:12.939 INFO:tasks.workunit.client.0.smithi113.stdout:6/811: sync 2024-04-12T05:09:12.939 INFO:tasks.workunit.client.0.smithi113.stdout:9/890: sync 2024-04-12T05:09:12.939 INFO:tasks.workunit.client.0.smithi113.stdout:5/978: sync 2024-04-12T05:09:12.939 INFO:tasks.workunit.client.0.smithi113.stdout:4/861: sync 2024-04-12T05:09:12.940 INFO:tasks.workunit.client.1.smithi192.stdout:4/839: dwrite d1/d54/d55/d64/f74 [0,4194304] 0 2024-04-12T05:09:12.940 INFO:tasks.workunit.client.0.smithi113.stdout:4/862: chown da/d5a 836 1 2024-04-12T05:09:12.941 INFO:tasks.workunit.client.1.smithi192.stdout:2/799: write d1/d8/f46 [3655689,114142] 0 2024-04-12T05:09:12.942 INFO:tasks.workunit.client.0.smithi113.stdout:1/962: chown d10/d2e/d31/d5c/daf/cc1 60288602 1 2024-04-12T05:09:12.942 INFO:tasks.workunit.client.0.smithi113.stdout:6/812: fdatasync d8/de/d13/d11/d36/d53/db7/f112 0 2024-04-12T05:09:12.943 INFO:tasks.workunit.client.1.smithi192.stdout:5/920: symlink d0/d15/d3e/dca/dda/db5/dd7/de1/l13d 0 2024-04-12T05:09:12.944 INFO:tasks.workunit.client.0.smithi113.stdout:9/891: creat d1/d27/d7a/d11b/f12b x:0 0 0 2024-04-12T05:09:12.946 INFO:tasks.workunit.client.0.smithi113.stdout:5/979: mknod d0/d10/d27/d1d/d38/d44/d4e/daa/dbb/c14d 0 2024-04-12T05:09:12.946 INFO:tasks.workunit.client.1.smithi192.stdout:8/804: rename d6/d10/d46/d5e/d5f/c87 to d6/d10/d46/d5e/d5f/d31/d61/d9d/db8/cf7 0 2024-04-12T05:09:12.950 INFO:tasks.workunit.client.0.smithi113.stdout:9/892: rmdir d1/d116 39 2024-04-12T05:09:12.951 INFO:tasks.workunit.client.1.smithi192.stdout:5/921: write d0/d15/f23 [8099255,36333] 0 2024-04-12T05:09:12.951 INFO:tasks.workunit.client.0.smithi113.stdout:2/845: link d8/d9/de/d12/d2b/dd2/df6/ff9 d8/d9/f108 0 2024-04-12T05:09:12.953 INFO:tasks.workunit.client.0.smithi113.stdout:1/963: rmdir d10/d2e/d31/d5c/daf/d121 0 2024-04-12T05:09:12.954 INFO:tasks.workunit.client.1.smithi192.stdout:4/840: mkdir d1/d1d/d11f 0 2024-04-12T05:09:12.954 INFO:tasks.workunit.client.0.smithi113.stdout:6/813: dwrite d8/d1d/fcd [0,4194304] 0 2024-04-12T05:09:12.955 INFO:tasks.workunit.client.1.smithi192.stdout:8/805: symlink d6/d10/d6c/db4/lf8 0 2024-04-12T05:09:12.957 INFO:tasks.workunit.client.1.smithi192.stdout:3/799: link d5/d13/d1a/d5f/ccd d5/d13/d3a/d70/d8c/d8d/c11e 0 2024-04-12T05:09:12.957 INFO:tasks.workunit.client.1.smithi192.stdout:5/922: rmdir d0/d15/d3e 39 2024-04-12T05:09:12.958 INFO:tasks.workunit.client.1.smithi192.stdout:7/735: link d3/d7/d15/d1a/f6e d3/d1d/d5a/d5f/d6a/de9/ffa 0 2024-04-12T05:09:12.960 INFO:tasks.workunit.client.0.smithi113.stdout:9/893: write d1/d4/d30/f75 [4919028,32012] 0 2024-04-12T05:09:12.961 INFO:tasks.workunit.client.0.smithi113.stdout:1/964: symlink d10/d2e/d31/d5c/l134 0 2024-04-12T05:09:12.962 INFO:tasks.workunit.client.1.smithi192.stdout:4/841: symlink d1/d1d/d53/d1c/d95/dc6/l120 0 2024-04-12T05:09:12.962 INFO:tasks.workunit.client.0.smithi113.stdout:5/980: dwrite d0/d10/d27/d1d/d38/d44/dcb/f144 [0,4194304] 0 2024-04-12T05:09:12.964 INFO:tasks.workunit.client.1.smithi192.stdout:2/800: creat d1/d8/d17/d19/d60/d59/ffe x:0 0 0 2024-04-12T05:09:12.964 INFO:tasks.workunit.client.0.smithi113.stdout:2/846: rmdir d8/d9/d27/d88/de0/dfe 0 2024-04-12T05:09:12.965 INFO:tasks.workunit.client.0.smithi113.stdout:2/847: stat d8/d9/d27/dda 0 2024-04-12T05:09:12.966 INFO:tasks.workunit.client.1.smithi192.stdout:5/923: creat d0/d15/d32/d69/d7d/f13e x:0 0 0 2024-04-12T05:09:12.969 INFO:tasks.workunit.client.1.smithi192.stdout:2/801: stat d1/d8/d17/d19/l67 0 2024-04-12T05:09:12.970 INFO:tasks.workunit.client.1.smithi192.stdout:5/924: chown d0/d15/d24/ff8 165 1 2024-04-12T05:09:12.971 INFO:tasks.workunit.client.0.smithi113.stdout:2/848: dread d8/d9/d1a/f26 [0,4194304] 0 2024-04-12T05:09:12.971 INFO:tasks.workunit.client.0.smithi113.stdout:6/814: truncate d8/de/d13/d11/d36/d3d/ffb 934194 0 2024-04-12T05:09:12.973 INFO:tasks.workunit.client.1.smithi192.stdout:8/806: dwrite f5 [4194304,4194304] 0 2024-04-12T05:09:12.974 INFO:tasks.workunit.client.0.smithi113.stdout:9/894: dwrite d1/d27/f3c [0,4194304] 0 2024-04-12T05:09:12.975 INFO:tasks.workunit.client.1.smithi192.stdout:7/736: rename d3/d7/d15/c62 to d3/d1d/d4a/d5e/d6d/da3/dce/dee/cfb 0 2024-04-12T05:09:12.975 INFO:tasks.workunit.client.1.smithi192.stdout:4/842: dwrite d1/d54/f9a [0,4194304] 0 2024-04-12T05:09:12.976 INFO:tasks.workunit.client.1.smithi192.stdout:3/800: dread d5/d13/d3c/d36/f8a [0,4194304] 0 2024-04-12T05:09:12.976 INFO:tasks.workunit.client.0.smithi113.stdout:5/981: rename d0/d10/d6e/d93/dc9/d14b/d73/db9/c111 to d0/d10/d27/d5f/d128/d12d/c14e 0 2024-04-12T05:09:12.978 INFO:tasks.workunit.client.1.smithi192.stdout:8/807: chown d6/d10/d46/d5e/d5f/d31/d61/d9d 3 1 2024-04-12T05:09:12.979 INFO:tasks.workunit.client.1.smithi192.stdout:2/802: mknod d1/cff 0 2024-04-12T05:09:12.980 INFO:tasks.workunit.client.0.smithi113.stdout:6/815: mknod d8/de/dd5/c129 0 2024-04-12T05:09:12.985 INFO:tasks.workunit.client.0.smithi113.stdout:1/965: write d10/d2e/dd8/d70/f48 [3208091,111828] 0 2024-04-12T05:09:12.985 INFO:tasks.workunit.client.1.smithi192.stdout:5/925: dread d0/d15/d32/d69/f9e [0,4194304] 0 2024-04-12T05:09:12.987 INFO:tasks.workunit.client.0.smithi113.stdout:6/816: fsync d8/de/dd5/de7/db3/dc5/fed 0 2024-04-12T05:09:12.987 INFO:tasks.workunit.client.1.smithi192.stdout:4/843: symlink d1/d49/l121 0 2024-04-12T05:09:12.988 INFO:tasks.workunit.client.0.smithi113.stdout:1/966: chown fd 9204350 1 2024-04-12T05:09:12.989 INFO:tasks.workunit.client.0.smithi113.stdout:2/849: dread d8/d9/d27/d88/fab [0,4194304] 0 2024-04-12T05:09:12.989 INFO:tasks.workunit.client.1.smithi192.stdout:3/801: rename d5/d13/d1a/l82 to d5/d13/d3a/d70/d8c/de1/l11f 0 2024-04-12T05:09:12.990 INFO:tasks.workunit.client.1.smithi192.stdout:7/737: truncate d3/d7/f56 3376382 0 2024-04-12T05:09:12.992 INFO:tasks.workunit.client.1.smithi192.stdout:7/738: chown d3/d1d/d4a/d5e 1297193 1 2024-04-12T05:09:12.993 INFO:tasks.workunit.client.1.smithi192.stdout:7/739: readlink d3/d7/d60/d65/ld0 0 2024-04-12T05:09:12.994 INFO:tasks.workunit.client.0.smithi113.stdout:4/863: dread da/dd/d3a/f42 [0,4194304] 0 2024-04-12T05:09:12.995 INFO:tasks.workunit.client.0.smithi113.stdout:2/850: stat d8/d9/de/d43/dee/ffc 0 2024-04-12T05:09:12.996 INFO:tasks.workunit.client.1.smithi192.stdout:2/803: symlink d1/d8/d17/d19/d60/d59/da3/l100 0 2024-04-12T05:09:12.997 INFO:tasks.workunit.client.0.smithi113.stdout:2/851: dread - d8/d9/f108 zero size 2024-04-12T05:09:12.999 INFO:tasks.workunit.client.1.smithi192.stdout:5/926: truncate d0/d15/fc3 657653 0 2024-04-12T05:09:13.001 INFO:tasks.workunit.client.1.smithi192.stdout:4/844: truncate d1/d1d/d53/d1c/f4e 6826095 0 2024-04-12T05:09:13.002 INFO:tasks.workunit.client.1.smithi192.stdout:3/802: write d5/d13/d3a/d4a/d61/f100 [1639670,58855] 0 2024-04-12T05:09:13.004 INFO:tasks.workunit.client.0.smithi113.stdout:6/817: truncate d8/de/d13/d11/d36/d53/db7/fd4 3913085 0 2024-04-12T05:09:13.004 INFO:tasks.workunit.client.0.smithi113.stdout:9/895: write d1/df/d4b/d63/dad/fc6 [3890752,51767] 0 2024-04-12T05:09:13.005 INFO:tasks.workunit.client.1.smithi192.stdout:3/803: dread d5/d13/d3c/d26/f9e [0,4194304] 0 2024-04-12T05:09:13.005 INFO:tasks.workunit.client.1.smithi192.stdout:7/740: link d3/d7/d60/d65/fb5 d3/d1d/d4a/d5e/d6d/ffc 0 2024-04-12T05:09:13.008 INFO:tasks.workunit.client.1.smithi192.stdout:8/808: rename d6/d10/d46/d5e/d5f/d38/dbc/le6 to d6/d10/d46/d5e/d5f/lf9 0 2024-04-12T05:09:13.008 INFO:tasks.workunit.client.1.smithi192.stdout:8/809: readlink d6/d1f/d6a/l69 0 2024-04-12T05:09:13.009 INFO:tasks.workunit.client.0.smithi113.stdout:9/896: stat d1/d4/f88 0 2024-04-12T05:09:13.009 INFO:tasks.workunit.client.1.smithi192.stdout:4/845: symlink d1/d54/d82/df0/l122 0 2024-04-12T05:09:13.011 INFO:tasks.workunit.client.0.smithi113.stdout:1/967: dwrite d10/d41/d89/d96/d7d/f7f [0,4194304] 0 2024-04-12T05:09:13.012 INFO:tasks.workunit.client.0.smithi113.stdout:6/818: creat d8/f12a x:0 0 0 2024-04-12T05:09:13.014 INFO:tasks.workunit.client.0.smithi113.stdout:9/897: creat d1/df/db6/f12c x:0 0 0 2024-04-12T05:09:13.014 INFO:tasks.workunit.client.0.smithi113.stdout:6/819: chown d8/de/d13/d11/d102/f121 538 1 2024-04-12T05:09:13.015 INFO:tasks.workunit.client.1.smithi192.stdout:3/804: symlink d5/d13/d1a/d5f/d103/l120 0 2024-04-12T05:09:13.016 INFO:tasks.workunit.client.1.smithi192.stdout:5/927: symlink d0/d15/d3e/dca/dda/db5/dbe/l13f 0 2024-04-12T05:09:13.017 INFO:tasks.workunit.client.0.smithi113.stdout:1/968: creat d10/d41/d89/d96/d7d/f135 x:0 0 0 2024-04-12T05:09:13.018 INFO:tasks.workunit.client.0.smithi113.stdout:9/898: mkdir d1/df/db6/d12d 0 2024-04-12T05:09:13.020 INFO:tasks.workunit.client.0.smithi113.stdout:9/899: chown d1/d4/d38/d65/c4f 507 1 2024-04-12T05:09:13.020 INFO:tasks.workunit.client.1.smithi192.stdout:8/810: mknod d6/d10/d6c/d9e/cfa 0 2024-04-12T05:09:13.022 INFO:tasks.workunit.client.1.smithi192.stdout:2/804: link d1/d8/d17/d19/d71/d7f/f96 d1/da7/f101 0 2024-04-12T05:09:13.023 INFO:tasks.workunit.client.0.smithi113.stdout:1/969: chown d10/d41/l6a 6 1 2024-04-12T05:09:13.024 INFO:tasks.workunit.client.0.smithi113.stdout:4/864: dread da/d3c/d83/d47/d10f/f72 [0,4194304] 0 2024-04-12T05:09:13.024 INFO:tasks.workunit.client.0.smithi113.stdout:9/900: unlink d1/f11d 0 2024-04-12T05:09:13.027 INFO:tasks.workunit.client.0.smithi113.stdout:4/865: rmdir da/d24/d7e/d79 39 2024-04-12T05:09:13.030 INFO:tasks.workunit.client.0.smithi113.stdout:9/901: unlink d1/df/d4b/df7/d111/f128 0 2024-04-12T05:09:13.030 INFO:tasks.workunit.client.1.smithi192.stdout:5/928: rmdir d0/d15/d3e 39 2024-04-12T05:09:13.034 INFO:tasks.workunit.client.0.smithi113.stdout:6/820: dwrite d8/de/dd5/de7/feb [0,4194304] 0 2024-04-12T05:09:13.035 INFO:tasks.workunit.client.0.smithi113.stdout:9/902: creat d1/df/d4b/df7/d111/f12e x:0 0 0 2024-04-12T05:09:13.040 INFO:tasks.workunit.client.1.smithi192.stdout:3/805: link d5/d13/d1a/d5f/f75 d5/d13/d1a/d53/f121 0 2024-04-12T05:09:13.040 INFO:tasks.workunit.client.0.smithi113.stdout:1/970: dwrite d10/d14/f1f [0,4194304] 0 2024-04-12T05:09:13.042 INFO:tasks.workunit.client.0.smithi113.stdout:6/821: unlink d8/de/d13/d11/d36/d3d/d4c/d26/d5b/f79 0 2024-04-12T05:09:13.045 INFO:tasks.workunit.client.1.smithi192.stdout:7/741: dwrite d3/d7/d15/f77 [4194304,4194304] 0 2024-04-12T05:09:13.047 INFO:tasks.workunit.client.1.smithi192.stdout:4/846: link d1/d54/d82/c85 d1/d1d/d53/d1c/d95/dc6/dd5/de5/ded/d119/c123 0 2024-04-12T05:09:13.048 INFO:tasks.workunit.client.1.smithi192.stdout:5/929: fsync d0/d15/d3e/fb3 0 2024-04-12T05:09:13.050 INFO:tasks.workunit.client.0.smithi113.stdout:2/852: dread d8/d9/de/d12/d2b/daa/db3/f94 [0,4194304] 0 2024-04-12T05:09:13.050 INFO:tasks.workunit.client.0.smithi113.stdout:1/971: dwrite d10/d2e/f102 [4194304,4194304] 0 2024-04-12T05:09:13.053 INFO:tasks.workunit.client.1.smithi192.stdout:2/805: getdents d1/d8 0 2024-04-12T05:09:13.053 INFO:tasks.workunit.client.1.smithi192.stdout:7/742: truncate d3/d7/f7e 4993373 0 2024-04-12T05:09:13.053 INFO:tasks.workunit.client.0.smithi113.stdout:1/972: truncate d10/d41/d89/d96/d7d/f117 519769 0 2024-04-12T05:09:13.054 INFO:tasks.workunit.client.0.smithi113.stdout:2/853: mknod d8/d9/de/d12/d2b/c109 0 2024-04-12T05:09:13.055 INFO:tasks.workunit.client.1.smithi192.stdout:7/743: chown d3/d1d/d4a/d5e/d6d/da3/dce/dee 16611025 1 2024-04-12T05:09:13.055 INFO:tasks.workunit.client.1.smithi192.stdout:3/806: creat d5/d13/d3a/d70/d8c/f122 x:0 0 0 2024-04-12T05:09:13.056 INFO:tasks.workunit.client.1.smithi192.stdout:5/930: symlink d0/d15/d24/d47/l140 0 2024-04-12T05:09:13.056 INFO:tasks.workunit.client.1.smithi192.stdout:8/811: write d6/d10/d46/d5e/d5f/d37/d4f/fb2 [916822,93297] 0 2024-04-12T05:09:13.057 INFO:tasks.workunit.client.0.smithi113.stdout:1/973: truncate d10/d41/d89/f129 254890 0 2024-04-12T05:09:13.059 INFO:tasks.workunit.client.0.smithi113.stdout:2/854: mkdir d8/d9/d5b/d10a 0 2024-04-12T05:09:13.060 INFO:tasks.workunit.client.0.smithi113.stdout:6/822: write d8/de/d17/f59 [4139613,54181] 0 2024-04-12T05:09:13.061 INFO:tasks.workunit.client.1.smithi192.stdout:8/812: write d6/d10/d46/d7c/fa0 [2435156,61731] 0 2024-04-12T05:09:13.065 INFO:tasks.workunit.client.1.smithi192.stdout:2/806: creat d1/d2d/d73/de7/df0/f102 x:0 0 0 2024-04-12T05:09:13.066 INFO:tasks.workunit.client.0.smithi113.stdout:1/974: mkdir d10/d14/d95/d132/d136 0 2024-04-12T05:09:13.066 INFO:tasks.workunit.client.1.smithi192.stdout:7/744: mkdir d3/d1d/d4a/d5e/df9/dfd 0 2024-04-12T05:09:13.069 INFO:tasks.workunit.client.0.smithi113.stdout:6/823: mknod d8/de/dd5/de7/d76/c12b 0 2024-04-12T05:09:13.070 INFO:tasks.workunit.client.1.smithi192.stdout:8/813: truncate d6/d10/fa5 3147459 0 2024-04-12T05:09:13.070 INFO:tasks.workunit.client.0.smithi113.stdout:1/975: dwrite d10/d41/d89/d96/d7d/f7e [0,4194304] 0 2024-04-12T05:09:13.072 INFO:tasks.workunit.client.1.smithi192.stdout:3/807: dwrite d5/d13/d94/dbd/ffa [0,4194304] 0 2024-04-12T05:09:13.072 INFO:tasks.workunit.client.1.smithi192.stdout:2/807: creat d1/d2d/d73/de7/f103 x:0 0 0 2024-04-12T05:09:13.072 INFO:tasks.workunit.client.0.smithi113.stdout:1/976: fdatasync d10/d2e/dd8/d70/f10f 0 2024-04-12T05:09:13.074 INFO:tasks.workunit.client.0.smithi113.stdout:6/824: mknod d8/de/d13/df0/d66/dc4/de3/c12c 0 2024-04-12T05:09:13.074 INFO:tasks.workunit.client.1.smithi192.stdout:5/931: link d0/d15/d32/f6a d0/d15/d3e/dca/dda/deb/d113/d127/f141 0 2024-04-12T05:09:13.075 INFO:tasks.workunit.client.0.smithi113.stdout:2/855: link d8/d9/de/d12/d44/f76 d8/d9/d1a/d103/ded/def/f10b 0 2024-04-12T05:09:13.076 INFO:tasks.workunit.client.1.smithi192.stdout:4/847: link d1/d1d/d53/d1c/dd1/def/l80 d1/d49/l124 0 2024-04-12T05:09:13.076 INFO:tasks.workunit.client.1.smithi192.stdout:3/808: write d5/d13/d94/fda [312732,13163] 0 2024-04-12T05:09:13.079 INFO:tasks.workunit.client.1.smithi192.stdout:7/745: mknod d3/d1d/d4a/d5e/cfe 0 2024-04-12T05:09:13.082 INFO:tasks.workunit.client.1.smithi192.stdout:8/814: dread - d6/d10/d46/d5e/d5f/d38/f7d zero size 2024-04-12T05:09:13.082 INFO:tasks.workunit.client.1.smithi192.stdout:4/848: truncate d1/d54/d82/d71/fb0 834681 0 2024-04-12T05:09:13.083 INFO:tasks.workunit.client.0.smithi113.stdout:4/866: dread da/d3c/d83/f6a [4194304,4194304] 0 2024-04-12T05:09:13.084 INFO:tasks.workunit.client.0.smithi113.stdout:4/867: chown da/d3c/d9a/dc1/d106 174448955 1 2024-04-12T05:09:13.085 INFO:tasks.workunit.client.1.smithi192.stdout:7/746: fsync d3/d7/d60/ff1 0 2024-04-12T05:09:13.086 INFO:tasks.workunit.client.0.smithi113.stdout:1/977: getdents d10/d2e/dd8/d99 0 2024-04-12T05:09:13.086 INFO:tasks.workunit.client.0.smithi113.stdout:2/856: dwrite d8/d9/d1a/fec [0,4194304] 0 2024-04-12T05:09:13.087 INFO:tasks.workunit.client.0.smithi113.stdout:8/929: sync 2024-04-12T05:09:13.091 INFO:tasks.workunit.client.1.smithi192.stdout:8/815: mkdir d6/d1f/d9c/dfb 0 2024-04-12T05:09:13.092 INFO:tasks.workunit.client.1.smithi192.stdout:2/808: dread d1/d8/d17/d19/d60/d9c/f9f [0,4194304] 0 2024-04-12T05:09:13.092 INFO:tasks.workunit.client.1.smithi192.stdout:5/932: dwrite d0/d15/d3e/dca/f109 [0,4194304] 0 2024-04-12T05:09:13.094 INFO:tasks.workunit.client.1.smithi192.stdout:3/809: mkdir d5/d13/d3a/d8b/da2/d123 0 2024-04-12T05:09:13.094 INFO:tasks.workunit.client.0.smithi113.stdout:8/930: unlink d9/cd7 0 2024-04-12T05:09:13.096 INFO:tasks.workunit.client.0.smithi113.stdout:8/931: fsync d9/d16/d65/d121/d59/d78/ff7 0 2024-04-12T05:09:13.096 INFO:tasks.workunit.client.0.smithi113.stdout:6/825: dread d8/fda [0,4194304] 0 2024-04-12T05:09:13.097 INFO:tasks.workunit.client.0.smithi113.stdout:4/868: unlink da/dd/c7f 0 2024-04-12T05:09:13.098 INFO:tasks.workunit.client.0.smithi113.stdout:2/857: truncate d8/d9/de/d15/d24/d58/fae 640023 0 2024-04-12T05:09:13.099 INFO:tasks.workunit.client.1.smithi192.stdout:2/809: dread - d1/d8/d17/fd0 zero size 2024-04-12T05:09:13.099 INFO:tasks.workunit.client.1.smithi192.stdout:8/816: write d6/d10/d46/d5e/d6e/fbf [849676,14411] 0 2024-04-12T05:09:13.099 INFO:tasks.workunit.client.1.smithi192.stdout:7/747: mkdir d3/d7/dff 0 2024-04-12T05:09:13.100 INFO:tasks.workunit.client.1.smithi192.stdout:2/810: chown d1/d2d/d73/f9b 15519858 1 2024-04-12T05:09:13.100 INFO:tasks.workunit.client.1.smithi192.stdout:5/933: chown d0 322 1 2024-04-12T05:09:13.101 INFO:tasks.workunit.client.1.smithi192.stdout:8/817: fdatasync d6/d10/d46/d5e/d5f/f88 0 2024-04-12T05:09:13.102 INFO:tasks.workunit.client.0.smithi113.stdout:4/869: truncate da/d3c/d9a/dc1/d106/dc8/f108 603173 0 2024-04-12T05:09:13.102 INFO:tasks.workunit.client.0.smithi113.stdout:2/858: mknod d8/d9/d27/dda/c10c 0 2024-04-12T05:09:13.102 INFO:tasks.workunit.client.0.smithi113.stdout:4/870: fdatasync da/dd/f6e 0 2024-04-12T05:09:13.103 INFO:tasks.workunit.client.1.smithi192.stdout:4/849: rename d1/d1d/d53/d1c/f36 to d1/d1d/d53/d11a/ddb/ddc/f125 0 2024-04-12T05:09:13.105 INFO:tasks.workunit.client.0.smithi113.stdout:8/932: rename d9/d16/d65/d121/d59/ce1 to d9/d16/c126 0 2024-04-12T05:09:13.105 INFO:tasks.workunit.client.1.smithi192.stdout:3/810: creat d5/d13/d3a/d4e/f124 x:0 0 0 2024-04-12T05:09:13.106 INFO:tasks.workunit.client.0.smithi113.stdout:4/871: creat da/d3c/da5/f120 x:0 0 0 2024-04-12T05:09:13.108 INFO:tasks.workunit.client.1.smithi192.stdout:4/850: fsync d1/d1d/d53/d1c/fb1 0 2024-04-12T05:09:13.109 INFO:tasks.workunit.client.1.smithi192.stdout:7/748: truncate d3/d1d/d48/d7b/f95 2280183 0 2024-04-12T05:09:13.109 INFO:tasks.workunit.client.0.smithi113.stdout:4/872: read - da/d24/f52 zero size 2024-04-12T05:09:13.111 INFO:tasks.workunit.client.1.smithi192.stdout:3/811: read d5/d13/d3a/d4a/d61/f64 [745216,78600] 0 2024-04-12T05:09:13.111 INFO:tasks.workunit.client.0.smithi113.stdout:6/826: rename d8/de/d13/d11/d36/f61 to d8/de/dd5/de7/d76/f12d 0 2024-04-12T05:09:13.112 INFO:tasks.workunit.client.1.smithi192.stdout:3/812: chown d5/d13/d3a/d8b/da2 111372124 1 2024-04-12T05:09:13.113 INFO:tasks.workunit.client.1.smithi192.stdout:7/749: creat d3/d1d/d74/d86/d93/db9/f100 x:0 0 0 2024-04-12T05:09:13.114 INFO:tasks.workunit.client.0.smithi113.stdout:4/873: fsync da/d24/f52 0 2024-04-12T05:09:13.114 INFO:tasks.workunit.client.0.smithi113.stdout:2/859: link d8/d9/d27/dda/cd0 d8/d9/de/d12/d2b/daa/db3/de2/c10d 0 2024-04-12T05:09:13.115 INFO:tasks.workunit.client.1.smithi192.stdout:3/813: fsync d5/d13/d3a/d70/d8c/fb0 0 2024-04-12T05:09:13.115 INFO:tasks.workunit.client.0.smithi113.stdout:4/874: fsync da/d2b/f11c 0 2024-04-12T05:09:13.118 INFO:tasks.workunit.client.0.smithi113.stdout:2/860: fsync d8/d9/d27/dd9/fdb 0 2024-04-12T05:09:13.119 INFO:tasks.workunit.client.1.smithi192.stdout:4/851: rename d1/d1d/d53/d1c/d84/fa9 to d1/d54/d55/f126 0 2024-04-12T05:09:13.120 INFO:tasks.workunit.client.1.smithi192.stdout:4/852: fsync d1/d1d/d53/d1c/de1/f112 0 2024-04-12T05:09:13.120 INFO:tasks.workunit.client.0.smithi113.stdout:6/827: dwrite d8/d45/d124/f126 [0,4194304] 0 2024-04-12T05:09:13.121 INFO:tasks.workunit.client.1.smithi192.stdout:3/814: write d5/d13/d3a/d8b/fba [3243699,22212] 0 2024-04-12T05:09:13.123 INFO:tasks.workunit.client.0.smithi113.stdout:2/861: mkdir d8/d9/de/d12/d44/d10e 0 2024-04-12T05:09:13.124 INFO:tasks.workunit.client.1.smithi192.stdout:3/815: readlink d5/d13/d1a/l47 0 2024-04-12T05:09:13.124 INFO:tasks.workunit.client.0.smithi113.stdout:1/978: truncate f2 1925735 0 2024-04-12T05:09:13.125 INFO:tasks.workunit.client.0.smithi113.stdout:2/862: read d8/fc0 [120550,125290] 0 2024-04-12T05:09:13.127 INFO:tasks.workunit.client.0.smithi113.stdout:1/979: mkdir d10/d55/d9c/d137 0 2024-04-12T05:09:13.128 INFO:tasks.workunit.client.0.smithi113.stdout:6/828: mknod d8/de/dd5/de7/db3/dc5/c12e 0 2024-04-12T05:09:13.131 INFO:tasks.workunit.client.1.smithi192.stdout:4/853: dwrite d1/d54/d82/d71/d9e/f10e [0,4194304] 0 2024-04-12T05:09:13.132 INFO:tasks.workunit.client.1.smithi192.stdout:3/816: symlink d5/d13/d3a/d70/d8c/d8d/l125 0 2024-04-12T05:09:13.132 INFO:tasks.workunit.client.0.smithi113.stdout:4/875: mknod da/d24/d7e/d79/dcd/c121 0 2024-04-12T05:09:13.132 INFO:tasks.workunit.client.0.smithi113.stdout:4/876: chown da/d24/d60/daa/d112 129069 1 2024-04-12T05:09:13.133 INFO:tasks.workunit.client.1.smithi192.stdout:5/934: truncate d0/d15/d3e/dca/fb7 2131191 0 2024-04-12T05:09:13.133 INFO:tasks.workunit.client.0.smithi113.stdout:6/829: dread d8/f22 [0,4194304] 0 2024-04-12T05:09:13.133 INFO:tasks.workunit.client.0.smithi113.stdout:4/877: chown da/d24/d7e/d79 42930 1 2024-04-12T05:09:13.134 INFO:tasks.workunit.client.1.smithi192.stdout:7/750: dread d3/d1d/f28 [0,4194304] 0 2024-04-12T05:09:13.135 INFO:tasks.workunit.client.0.smithi113.stdout:2/863: write d8/d9/de/d12/d2b/daa/db3/ff0 [458662,33094] 0 2024-04-12T05:09:13.135 INFO:tasks.workunit.client.1.smithi192.stdout:3/817: mkdir d5/d13/d1a/d53/dbb/d126 0 2024-04-12T05:09:13.136 INFO:tasks.workunit.client.1.smithi192.stdout:5/935: creat d0/d15/d3e/dca/dda/d11c/f142 x:0 0 0 2024-04-12T05:09:13.138 INFO:tasks.workunit.client.0.smithi113.stdout:2/864: chown d8/d9/d27/d88/fab 393 1 2024-04-12T05:09:13.141 INFO:tasks.workunit.client.0.smithi113.stdout:8/933: write d9/d1b/f46 [227083,31003] 0 2024-04-12T05:09:13.141 INFO:tasks.workunit.client.0.smithi113.stdout:2/865: write d8/d9/d1a/ff1 [1301364,27289] 0 2024-04-12T05:09:13.142 INFO:tasks.workunit.client.1.smithi192.stdout:4/854: creat d1/d54/d82/da4/f127 x:0 0 0 2024-04-12T05:09:13.144 INFO:tasks.workunit.client.0.smithi113.stdout:6/830: dread d8/de/d13/d11/d36/d3d/d4c/d31/f10b [0,4194304] 0 2024-04-12T05:09:13.144 INFO:tasks.workunit.client.1.smithi192.stdout:5/936: mknod d0/d15/d32/db2/c143 0 2024-04-12T05:09:13.146 INFO:tasks.workunit.client.1.smithi192.stdout:5/937: stat d0/f11 0 2024-04-12T05:09:13.146 INFO:tasks.workunit.client.1.smithi192.stdout:3/818: dread d5/d13/d3a/d4e/f95 [0,4194304] 0 2024-04-12T05:09:13.148 INFO:tasks.workunit.client.1.smithi192.stdout:5/938: mknod d0/d21/c144 0 2024-04-12T05:09:13.151 INFO:tasks.workunit.client.1.smithi192.stdout:5/939: mkdir d0/d15/d145 0 2024-04-12T05:09:13.151 INFO:tasks.workunit.client.1.smithi192.stdout:5/940: dread - d0/d21/d5e/f135 zero size 2024-04-12T05:09:13.153 INFO:tasks.workunit.client.1.smithi192.stdout:3/819: rename d5/d13/d94/dbd/lf7 to d5/d13/d3a/l127 0 2024-04-12T05:09:13.155 INFO:tasks.workunit.client.1.smithi192.stdout:4/855: creat d1/d1d/d53/d11a/ddb/f128 x:0 0 0 2024-04-12T05:09:13.157 INFO:tasks.workunit.client.1.smithi192.stdout:5/941: fsync d0/d15/d32/d69/f9e 0 2024-04-12T05:09:13.159 INFO:tasks.workunit.client.1.smithi192.stdout:7/751: write d3/d7/d15/f50 [1647183,83935] 0 2024-04-12T05:09:13.159 INFO:tasks.workunit.client.0.smithi113.stdout:2/866: dread d8/d9/de/d12/d2b/d79/ddf/fe5 [4194304,4194304] 0 2024-04-12T05:09:13.165 INFO:tasks.workunit.client.0.smithi113.stdout:2/867: dwrite d8/d9/de/f67 [0,4194304] 0 2024-04-12T05:09:13.166 INFO:tasks.workunit.client.0.smithi113.stdout:2/868: stat d8/d9/d1a 0 2024-04-12T05:09:13.168 INFO:tasks.workunit.client.0.smithi113.stdout:2/869: creat d8/d9/de/d15/d24/f10f x:0 0 0 2024-04-12T05:09:13.169 INFO:tasks.workunit.client.0.smithi113.stdout:2/870: chown d8/d9/de/f31 309194906 1 2024-04-12T05:09:13.170 INFO:tasks.workunit.client.0.smithi113.stdout:2/871: write d8/d9/d27/f72 [1368484,80294] 0 2024-04-12T05:09:13.170 INFO:tasks.workunit.client.0.smithi113.stdout:2/872: chown d8/c11 175360878 1 2024-04-12T05:09:13.171 INFO:tasks.workunit.client.0.smithi113.stdout:2/873: readlink d8/d9/de/d12/d2b/daa/db3/lcf 0 2024-04-12T05:09:13.189 INFO:tasks.workunit.client.0.smithi113.stdout:6/831: write d8/de/d13/d11/d36/d53/db8/f9d [1565915,28644] 0 2024-04-12T05:09:13.191 INFO:tasks.workunit.client.0.smithi113.stdout:2/874: truncate d8/d9/f69 2417590 0 2024-04-12T05:09:13.192 INFO:tasks.workunit.client.0.smithi113.stdout:6/832: stat d8/de/dd5/lfa 0 2024-04-12T05:09:13.194 INFO:tasks.workunit.client.0.smithi113.stdout:2/875: readlink d8/d9/de/d15/d24/l36 0 2024-04-12T05:09:13.196 INFO:tasks.workunit.client.0.smithi113.stdout:4/878: dwrite da/d24/f53 [0,4194304] 0 2024-04-12T05:09:13.197 INFO:tasks.workunit.client.0.smithi113.stdout:8/934: dwrite d9/d1b/d8e/d93/f4d [0,4194304] 0 2024-04-12T05:09:13.198 INFO:tasks.workunit.client.0.smithi113.stdout:2/876: chown d8/d9/de/d15/d24/d58/f75 1894302774 1 2024-04-12T05:09:13.202 INFO:tasks.workunit.client.0.smithi113.stdout:2/877: creat d8/d9/de/d12/d3c/f110 x:0 0 0 2024-04-12T05:09:13.204 INFO:tasks.workunit.client.0.smithi113.stdout:4/879: unlink da/d24/d7e/d79/ce8 0 2024-04-12T05:09:13.207 INFO:tasks.workunit.client.0.smithi113.stdout:2/878: rmdir d8/d9 39 2024-04-12T05:09:13.207 INFO:tasks.workunit.client.0.smithi113.stdout:6/833: dwrite d8/de/d13/d11/d36/d3d/d4c/d26/d108/f10e [0,4194304] 0 2024-04-12T05:09:13.211 INFO:tasks.workunit.client.0.smithi113.stdout:8/935: link d9/d16/d65/d121/f8b d9/d1b/d8e/d93/d32/d48/d8d/dae/d112/f127 0 2024-04-12T05:09:13.214 INFO:tasks.workunit.client.0.smithi113.stdout:6/834: rename d8/de/d13/df0/d66/f106 to d8/de/d13/d11/d36/d2c/dca/f12f 0 2024-04-12T05:09:13.215 INFO:tasks.workunit.client.0.smithi113.stdout:2/879: dread - d8/d9/de/d12/d2b/daa/db3/fb1 zero size 2024-04-12T05:09:13.217 INFO:tasks.workunit.client.0.smithi113.stdout:8/936: creat d9/d1b/d8e/d93/d92/f128 x:0 0 0 2024-04-12T05:09:13.218 INFO:tasks.workunit.client.0.smithi113.stdout:4/880: rename da/d3c/d9a/dc1/d106/dc8/f108 to da/d3c/d9a/dc1/dd9/f122 0 2024-04-12T05:09:13.219 INFO:tasks.workunit.client.0.smithi113.stdout:6/835: dread - d8/de/d13/d11/d36/d3d/d4c/fdb zero size 2024-04-12T05:09:13.220 INFO:tasks.workunit.client.0.smithi113.stdout:2/880: chown d8/d9/d27/d88/f8b 128091084 1 2024-04-12T05:09:13.221 INFO:tasks.workunit.client.0.smithi113.stdout:2/881: fsync d8/d9/de/d12/f35 0 2024-04-12T05:09:13.221 INFO:tasks.workunit.client.0.smithi113.stdout:4/881: mknod da/dd/d1b/c123 0 2024-04-12T05:09:13.227 INFO:tasks.workunit.client.0.smithi113.stdout:2/882: unlink d8/d9/de/d15/d24/d58/f75 0 2024-04-12T05:09:13.228 INFO:tasks.workunit.client.0.smithi113.stdout:6/836: dwrite d8/de/dd5/de7/feb [0,4194304] 0 2024-04-12T05:09:13.231 INFO:tasks.workunit.client.0.smithi113.stdout:2/883: chown d8/d9/d27/dd9/lf5 1 1 2024-04-12T05:09:13.232 INFO:tasks.workunit.client.0.smithi113.stdout:6/837: creat d8/de/d13/d11/d36/d3d/f130 x:0 0 0 2024-04-12T05:09:13.234 INFO:tasks.workunit.client.0.smithi113.stdout:8/937: dread d9/d1b/d8e/d93/d32/d5b/f63 [0,4194304] 0 2024-04-12T05:09:13.234 INFO:tasks.workunit.client.0.smithi113.stdout:8/938: fdatasync d9/d16/d3c/fe2 0 2024-04-12T05:09:13.241 INFO:tasks.workunit.client.0.smithi113.stdout:8/939: mkdir d9/d1b/d8e/d93/d32/d48/d129 0 2024-04-12T05:09:13.242 INFO:tasks.workunit.client.0.smithi113.stdout:6/838: getdents d8/de/d13/d11/d36/d3d/d4c/d31 0 2024-04-12T05:09:13.243 INFO:tasks.workunit.client.0.smithi113.stdout:4/882: rename da/dd to da/d24/d7e/d124 0 2024-04-12T05:09:13.244 INFO:tasks.workunit.client.0.smithi113.stdout:4/883: dread - da/d3c/d9f/fdd zero size 2024-04-12T05:09:13.245 INFO:tasks.workunit.client.0.smithi113.stdout:8/940: creat d9/d16/d65/d121/d59/d68/dca/f12a x:0 0 0 2024-04-12T05:09:13.247 INFO:tasks.workunit.client.0.smithi113.stdout:6/839: creat d8/de/d13/d11/d36/d53/db8/dce/d128/f131 x:0 0 0 2024-04-12T05:09:13.249 INFO:tasks.workunit.client.0.smithi113.stdout:4/884: fsync da/d24/d7e/d79/dcd/fe6 0 2024-04-12T05:09:13.250 INFO:tasks.workunit.client.0.smithi113.stdout:8/941: fdatasync d9/d1b/d8e/d93/d32/d48/d8d/dae/d112/f127 0 2024-04-12T05:09:13.254 INFO:tasks.workunit.client.0.smithi113.stdout:8/942: write d9/d16/d65/d121/f2d [1941997,65542] 0 2024-04-12T05:09:13.254 INFO:tasks.workunit.client.0.smithi113.stdout:2/884: dwrite d8/d9/d5b/f85 [0,4194304] 0 2024-04-12T05:09:13.256 INFO:tasks.workunit.client.0.smithi113.stdout:8/943: chown d9/d16/d65/d8c/fb5 1407264 1 2024-04-12T05:09:13.257 INFO:tasks.workunit.client.0.smithi113.stdout:6/840: write d8/de/d13/d11/d36/f35 [4481386,17952] 0 2024-04-12T05:09:13.258 INFO:tasks.workunit.client.0.smithi113.stdout:8/944: stat d9/d16/d65/f97 0 2024-04-12T05:09:13.258 INFO:tasks.workunit.client.0.smithi113.stdout:4/885: dread da/d24/d7e/d124/f6e [0,4194304] 0 2024-04-12T05:09:13.260 INFO:tasks.workunit.client.0.smithi113.stdout:8/945: chown d9/d1b/d8e/d93/d32/d48/f103 8566 1 2024-04-12T05:09:13.263 INFO:tasks.workunit.client.0.smithi113.stdout:4/886: mknod da/d3c/d43/c125 0 2024-04-12T05:09:13.265 INFO:tasks.workunit.client.0.smithi113.stdout:8/946: mknod d9/d16/d65/d121/d59/d68/c12b 0 2024-04-12T05:09:13.265 INFO:tasks.workunit.client.0.smithi113.stdout:2/885: dwrite d8/d9/de/d43/dee/ffc [0,4194304] 0 2024-04-12T05:09:13.268 INFO:tasks.workunit.client.0.smithi113.stdout:4/887: symlink da/d24/d60/daa/d112/l126 0 2024-04-12T05:09:13.269 INFO:tasks.workunit.client.0.smithi113.stdout:2/886: chown d8/d9/de/d15/c77 1579 1 2024-04-12T05:09:13.270 INFO:tasks.workunit.client.0.smithi113.stdout:6/841: dwrite d8/de/d13/d11/d36/d53/db8/dce/f10f [0,4194304] 0 2024-04-12T05:09:13.270 INFO:tasks.workunit.client.0.smithi113.stdout:8/947: write d9/d16/d65/d121/d59/d78/fcf [1388659,72453] 0 2024-04-12T05:09:13.274 INFO:tasks.workunit.client.0.smithi113.stdout:4/888: creat da/d24/d60/db8/f127 x:0 0 0 2024-04-12T05:09:13.277 INFO:tasks.workunit.client.0.smithi113.stdout:8/948: truncate d9/d1b/d8e/d93/faf 2206924 0 2024-04-12T05:09:13.278 INFO:tasks.workunit.client.0.smithi113.stdout:8/949: fsync d9/d1b/d8e/d93/f28 0 2024-04-12T05:09:13.281 INFO:tasks.workunit.client.0.smithi113.stdout:2/887: dwrite d8/d9/d1a/f26 [0,4194304] 0 2024-04-12T05:09:13.283 INFO:tasks.workunit.client.0.smithi113.stdout:2/888: write d8/d9/d1a/ff1 [492500,107922] 0 2024-04-12T05:09:13.283 INFO:tasks.workunit.client.1.smithi192.stdout:3/820: write d5/d13/d3a/d4a/d61/d79/f80 [794089,35072] 0 2024-04-12T05:09:13.283 INFO:tasks.workunit.client.1.smithi192.stdout:7/752: creat d3/d7/d15/d1a/ddc/f101 x:0 0 0 2024-04-12T05:09:13.286 INFO:tasks.workunit.client.1.smithi192.stdout:4/856: creat d1/d1d/d53/d1c/dd1/def/ddd/dea/df4/f129 x:0 0 0 2024-04-12T05:09:13.288 INFO:tasks.workunit.client.1.smithi192.stdout:5/942: getdents d0/d15/d3e/dca/dda/deb/d113/d127/ded/d136 0 2024-04-12T05:09:13.288 INFO:tasks.workunit.client.1.smithi192.stdout:4/857: chown d1/d54/d55/d64/l56 69 1 2024-04-12T05:09:13.289 INFO:tasks.workunit.client.1.smithi192.stdout:7/753: creat d3/d7/d14/d1f/f102 x:0 0 0 2024-04-12T05:09:13.292 INFO:tasks.workunit.client.1.smithi192.stdout:7/754: chown d3/l44 1782332218 1 2024-04-12T05:09:13.294 INFO:tasks.workunit.client.0.smithi113.stdout:8/950: rmdir d9/d16/d7f/d3a/d6c 39 2024-04-12T05:09:13.295 INFO:tasks.workunit.client.1.smithi192.stdout:3/821: dwrite d5/d13/d3a/d70/d8c/d8d/dbc/f113 [0,4194304] 0 2024-04-12T05:09:13.298 INFO:tasks.workunit.client.1.smithi192.stdout:4/858: mknod d1/d1d/d53/d1c/dd1/def/ddd/d114/c12a 0 2024-04-12T05:09:13.299 INFO:tasks.workunit.client.0.smithi113.stdout:3/887: sync 2024-04-12T05:09:13.299 INFO:tasks.workunit.client.0.smithi113.stdout:0/970: sync 2024-04-12T05:09:13.299 INFO:tasks.workunit.client.0.smithi113.stdout:7/823: sync 2024-04-12T05:09:13.300 INFO:tasks.workunit.client.1.smithi192.stdout:4/859: chown d1/d54/d82/d71/d9e/d10b/f7e 29019 1 2024-04-12T05:09:13.300 INFO:tasks.workunit.client.1.smithi192.stdout:3/822: mknod d5/d13/d3a/d4a/d61/d79/d11c/c128 0 2024-04-12T05:09:13.304 INFO:tasks.workunit.client.0.smithi113.stdout:0/971: truncate d1/d6/d34/dc3/f11a 8842513 0 2024-04-12T05:09:13.306 INFO:tasks.workunit.client.1.smithi192.stdout:5/943: getdents d0/d15/d32/d69/dc8 0 2024-04-12T05:09:13.307 INFO:tasks.workunit.client.1.smithi192.stdout:4/860: getdents d1/d54/d55/dc3 0 2024-04-12T05:09:13.309 INFO:tasks.workunit.client.0.smithi113.stdout:0/972: mkdir d1/d6/d16/d8f/d131 0 2024-04-12T05:09:13.309 INFO:tasks.workunit.client.0.smithi113.stdout:7/824: dwrite d0/dc/fde [0,4194304] 0 2024-04-12T05:09:13.310 INFO:tasks.workunit.client.1.smithi192.stdout:5/944: read d0/d15/d24/d29/f2a [3330388,77445] 0 2024-04-12T05:09:13.310 INFO:tasks.workunit.client.1.smithi192.stdout:4/861: rmdir d1/d1d/d53/d1c/d95/dc6/dd5/de5/d10d 39 2024-04-12T05:09:13.311 INFO:tasks.workunit.client.0.smithi113.stdout:7/825: readlink d0/dc/d10b/le8 0 2024-04-12T05:09:13.313 INFO:tasks.workunit.client.1.smithi192.stdout:3/823: dread d5/d13/d3a/d8b/fba [0,4194304] 0 2024-04-12T05:09:13.314 INFO:tasks.workunit.client.0.smithi113.stdout:8/951: write d9/d16/d65/f6a [751292,126922] 0 2024-04-12T05:09:13.315 INFO:tasks.workunit.client.0.smithi113.stdout:0/973: truncate d1/d6/d16/f47 1103645 0 2024-04-12T05:09:13.315 INFO:tasks.workunit.client.0.smithi113.stdout:6/842: dwrite d8/faf [0,4194304] 0 2024-04-12T05:09:13.317 INFO:tasks.workunit.client.1.smithi192.stdout:3/824: write d5/d13/d1a/d53/f60 [1153962,44978] 0 2024-04-12T05:09:13.319 INFO:tasks.workunit.client.1.smithi192.stdout:3/825: write d5/d13/f42 [2183008,58944] 0 2024-04-12T05:09:13.320 INFO:tasks.workunit.client.1.smithi192.stdout:5/945: getdents d0/d15/d32/d45/d106 0 2024-04-12T05:09:13.321 INFO:tasks.workunit.client.1.smithi192.stdout:7/755: write d3/d7/d14/d1f/d76/fbb [203315,20203] 0 2024-04-12T05:09:13.322 INFO:tasks.workunit.client.0.smithi113.stdout:7/826: creat d0/dc/d6/d1a/d81/f116 x:0 0 0 2024-04-12T05:09:13.323 INFO:tasks.workunit.client.1.smithi192.stdout:4/862: dwrite d1/d54/d82/d71/d9e/f10e [0,4194304] 0 2024-04-12T05:09:13.324 INFO:tasks.workunit.client.0.smithi113.stdout:8/952: write d9/d1b/d8e/d93/f28 [321026,125045] 0 2024-04-12T05:09:13.326 INFO:tasks.workunit.client.1.smithi192.stdout:4/863: chown d1/d54/d82/d71/d9e/fa0 26074524 1 2024-04-12T05:09:13.328 INFO:tasks.workunit.client.0.smithi113.stdout:0/974: write d1/d43/d59/de9/d105/f70 [138927,6819] 0 2024-04-12T05:09:13.329 INFO:tasks.workunit.client.1.smithi192.stdout:4/864: stat d1/d54/d55/d64/f74 0 2024-04-12T05:09:13.329 INFO:tasks.workunit.client.0.smithi113.stdout:2/889: dwrite d8/d9/f30 [0,4194304] 0 2024-04-12T05:09:13.329 INFO:tasks.workunit.client.1.smithi192.stdout:5/946: dread d0/d21/d5e/f11f [0,4194304] 0 2024-04-12T05:09:13.334 INFO:tasks.workunit.client.1.smithi192.stdout:4/865: mkdir d1/d1d/d53/d1c/d84/d12b 0 2024-04-12T05:09:13.336 INFO:tasks.workunit.client.1.smithi192.stdout:4/866: truncate d1/d1d/d53/d1c/de1/f112 669979 0 2024-04-12T05:09:13.339 INFO:tasks.workunit.client.0.smithi113.stdout:7/827: rename d0/dc/da9/d5c/f105 to d0/d52/f117 0 2024-04-12T05:09:13.340 INFO:tasks.workunit.client.0.smithi113.stdout:6/843: dwrite d8/de/dd5/de7/feb [0,4194304] 0 2024-04-12T05:09:13.340 INFO:tasks.workunit.client.0.smithi113.stdout:3/888: dwrite d0/fa8 [0,4194304] 0 2024-04-12T05:09:13.340 INFO:tasks.workunit.client.0.smithi113.stdout:0/975: creat d1/d6/d34/f132 x:0 0 0 2024-04-12T05:09:13.341 INFO:tasks.workunit.client.1.smithi192.stdout:3/826: link d5/d13/d94/fd5 d5/d13/d3a/d4a/d61/d79/d11c/f129 0 2024-04-12T05:09:13.343 INFO:tasks.workunit.client.1.smithi192.stdout:7/756: dread f1 [0,4194304] 0 2024-04-12T05:09:13.343 INFO:tasks.workunit.client.0.smithi113.stdout:7/828: readlink d0/dc/da9/l4b 0 2024-04-12T05:09:13.344 INFO:tasks.workunit.client.1.smithi192.stdout:7/757: chown d3/d7/d60/d65 53536 1 2024-04-12T05:09:13.344 INFO:tasks.workunit.client.0.smithi113.stdout:3/889: dread - d0/d13/d47/f119 zero size 2024-04-12T05:09:13.344 INFO:tasks.workunit.client.1.smithi192.stdout:4/867: creat d1/d1d/d53/d1c/d95/dc6/dd5/de5/f12c x:0 0 0 2024-04-12T05:09:13.347 INFO:tasks.workunit.client.0.smithi113.stdout:0/976: dread - d1/d43/d59/de9/d105/d50/d52/d7e/f10f zero size 2024-04-12T05:09:13.350 INFO:tasks.workunit.client.0.smithi113.stdout:0/977: write d1/d6/d34/dc3/ff8 [3888693,7787] 0 2024-04-12T05:09:13.351 INFO:tasks.workunit.client.1.smithi192.stdout:4/868: mknod d1/d1d/d53/d1c/dd1/def/ddd/dfb/d103/c12d 0 2024-04-12T05:09:13.351 INFO:tasks.workunit.client.1.smithi192.stdout:7/758: mknod d3/d1d/d5a/d5f/d7a/da6/c103 0 2024-04-12T05:09:13.351 INFO:tasks.workunit.client.1.smithi192.stdout:4/869: chown d1/d54/d55/f30 69945556 1 2024-04-12T05:09:13.351 INFO:tasks.workunit.client.0.smithi113.stdout:8/953: rename d9/d1b/d8e/d93/d32/d48/d8d/c7d to d9/d1b/d8e/d93/d32/d48/d8d/dae/c12c 0 2024-04-12T05:09:13.352 INFO:tasks.workunit.client.0.smithi113.stdout:0/978: chown d1/d6/d34/dc3/lef 139 1 2024-04-12T05:09:13.353 INFO:tasks.workunit.client.1.smithi192.stdout:7/759: read d3/d7/d14/d1f/d76/fbb [9343,90183] 0 2024-04-12T05:09:13.355 INFO:tasks.workunit.client.1.smithi192.stdout:7/760: chown d3/d7/d15/d22/df3 200446 1 2024-04-12T05:09:13.355 INFO:tasks.workunit.client.0.smithi113.stdout:3/890: mknod d0/d13/d15/d28/d2a/d34/d5f/d7b/de5/c128 0 2024-04-12T05:09:13.356 INFO:tasks.workunit.client.0.smithi113.stdout:0/979: read - d1/d43/d59/de9/d105/d50/db6/d36/f125 zero size 2024-04-12T05:09:13.357 INFO:tasks.workunit.client.0.smithi113.stdout:7/829: link d0/dc/cc9 d0/dd/d11/dd5/c118 0 2024-04-12T05:09:13.360 INFO:tasks.workunit.client.0.smithi113.stdout:8/954: dwrite d9/f9d [0,4194304] 0 2024-04-12T05:09:13.360 INFO:tasks.workunit.client.0.smithi113.stdout:3/891: chown d0/d7e/da3/lf3 14 1 2024-04-12T05:09:13.362 INFO:tasks.workunit.client.0.smithi113.stdout:8/955: readlink d9/d1b/d8e/d93/d32/l7b 0 2024-04-12T05:09:13.362 INFO:tasks.workunit.client.0.smithi113.stdout:3/892: fdatasync d0/fa6 0 2024-04-12T05:09:13.364 INFO:tasks.workunit.client.0.smithi113.stdout:0/980: mkdir d1/d43/d59/de9/d105/d50/d52/d7e/d133 0 2024-04-12T05:09:13.366 INFO:tasks.workunit.client.0.smithi113.stdout:0/981: dread - d1/d43/d59/de9/f10a zero size 2024-04-12T05:09:13.367 INFO:tasks.workunit.client.0.smithi113.stdout:7/830: creat d0/dc/d6/d1a/f119 x:0 0 0 2024-04-12T05:09:13.367 INFO:tasks.workunit.client.0.smithi113.stdout:2/890: write d8/fc0 [389718,35173] 0 2024-04-12T05:09:13.368 INFO:tasks.workunit.client.0.smithi113.stdout:0/982: creat d1/d43/d59/de9/d105/d50/db6/f134 x:0 0 0 2024-04-12T05:09:13.368 INFO:tasks.workunit.client.0.smithi113.stdout:6/844: write d8/de/d13/f67 [799961,13965] 0 2024-04-12T05:09:13.369 INFO:tasks.workunit.client.0.smithi113.stdout:2/891: readlink d8/l25 0 2024-04-12T05:09:13.369 INFO:tasks.workunit.client.0.smithi113.stdout:7/831: chown d0/dc/d10b/ce5 39115 1 2024-04-12T05:09:13.373 INFO:tasks.workunit.client.0.smithi113.stdout:6/845: creat d8/de/d13/f132 x:0 0 0 2024-04-12T05:09:13.373 INFO:tasks.workunit.client.0.smithi113.stdout:7/832: creat d0/dd/d95/dd4/f11a x:0 0 0 2024-04-12T05:09:13.374 INFO:tasks.workunit.client.1.smithi192.stdout:3/827: dread d5/d13/d3a/d70/f71 [0,4194304] 0 2024-04-12T05:09:13.375 INFO:tasks.workunit.client.1.smithi192.stdout:3/828: write d5/d13/d3a/d70/d8c/f122 [833809,59791] 0 2024-04-12T05:09:13.375 INFO:tasks.workunit.client.0.smithi113.stdout:6/846: chown d8/de/d13/d11/l6f 83231618 1 2024-04-12T05:09:13.376 INFO:tasks.workunit.client.0.smithi113.stdout:6/847: write d8/faf [2078743,93423] 0 2024-04-12T05:09:13.378 INFO:tasks.workunit.client.1.smithi192.stdout:7/761: write d3/d1d/f41 [1384933,89212] 0 2024-04-12T05:09:13.380 INFO:tasks.workunit.client.0.smithi113.stdout:7/833: symlink d0/dd/d11/de6/l11b 0 2024-04-12T05:09:13.380 INFO:tasks.workunit.client.1.smithi192.stdout:7/762: readlink d3/l91 0 2024-04-12T05:09:13.382 INFO:tasks.workunit.client.1.smithi192.stdout:3/829: getdents d5/d13/d3a/d4a/d61/d79 0 2024-04-12T05:09:13.382 INFO:tasks.workunit.client.1.smithi192.stdout:4/870: truncate d1/d54/d82/d71/d9e/f10e 3564470 0 2024-04-12T05:09:13.382 INFO:tasks.workunit.client.0.smithi113.stdout:6/848: symlink d8/de/d13/df0/d66/dc4/dd9/de5/l133 0 2024-04-12T05:09:13.383 INFO:tasks.workunit.client.1.smithi192.stdout:4/871: fdatasync d1/d54/f61 0 2024-04-12T05:09:13.383 INFO:tasks.workunit.client.1.smithi192.stdout:7/763: dread - d3/d1d/d4a/d5e/d6d/fa8 zero size 2024-04-12T05:09:13.384 INFO:tasks.workunit.client.1.smithi192.stdout:3/830: truncate d5/d13/d1a/d5f/f75 556691 0 2024-04-12T05:09:13.387 INFO:tasks.workunit.client.1.smithi192.stdout:7/764: unlink d3/d1d/d74/d86/ldd 0 2024-04-12T05:09:13.388 INFO:tasks.workunit.client.0.smithi113.stdout:6/849: mkdir d8/de/d13/d11/d36/d3d/d4c/d31/d134 0 2024-04-12T05:09:13.388 INFO:tasks.workunit.client.0.smithi113.stdout:8/956: write d9/d16/d3c/f74 [2944348,102495] 0 2024-04-12T05:09:13.390 INFO:tasks.workunit.client.0.smithi113.stdout:6/850: truncate d8/f11e 998027 0 2024-04-12T05:09:13.391 INFO:tasks.workunit.client.0.smithi113.stdout:8/957: stat d9/d16/d65/d121/d59/da3/ffb 0 2024-04-12T05:09:13.391 INFO:tasks.workunit.client.0.smithi113.stdout:7/834: link d0/dc/d6/d1a/d81/cd8 d0/dc/d6/d1a/c11c 0 2024-04-12T05:09:13.392 INFO:tasks.workunit.client.0.smithi113.stdout:3/893: write d0/d13/d15/f75 [313869,48290] 0 2024-04-12T05:09:13.393 INFO:tasks.workunit.client.0.smithi113.stdout:2/892: dread d8/d9/de/d15/d24/d58/fe1 [0,4194304] 0 2024-04-12T05:09:13.394 INFO:tasks.workunit.client.1.smithi192.stdout:7/765: rename d3/d7/d14/d1f/f102 to d3/d7/d60/d65/f104 0 2024-04-12T05:09:13.394 INFO:tasks.workunit.client.0.smithi113.stdout:6/851: write d8/d45/fae [4828059,19718] 0 2024-04-12T05:09:13.395 INFO:tasks.workunit.client.0.smithi113.stdout:8/958: chown d9/d1b/d8e/d93/d32/d48/d8d 12 1 2024-04-12T05:09:13.396 INFO:tasks.workunit.client.1.smithi192.stdout:4/872: getdents d1/d54 0 2024-04-12T05:09:13.397 INFO:tasks.workunit.client.0.smithi113.stdout:3/894: write d0/d13/d15/d28/d2a/d34/f11d [983402,119263] 0 2024-04-12T05:09:13.397 INFO:tasks.workunit.client.0.smithi113.stdout:7/835: mkdir d0/dc/da9/d5c/d9a/d11d 0 2024-04-12T05:09:13.399 INFO:tasks.workunit.client.1.smithi192.stdout:7/766: unlink d3/d7/d15/d1a/fac 0 2024-04-12T05:09:13.400 INFO:tasks.workunit.client.0.smithi113.stdout:8/959: dread - d9/d16/d65/d121/d59/d68/f100 zero size 2024-04-12T05:09:13.402 INFO:tasks.workunit.client.0.smithi113.stdout:6/852: creat d8/de/d13/f135 x:0 0 0 2024-04-12T05:09:13.403 INFO:tasks.workunit.client.1.smithi192.stdout:4/873: fsync d1/d1d/d53/d1c/d95/dc6/dd5/de5/f11d 0 2024-04-12T05:09:13.403 INFO:tasks.workunit.client.1.smithi192.stdout:3/831: dwrite d5/d13/d3c/d26/fc5 [0,4194304] 0 2024-04-12T05:09:13.405 INFO:tasks.workunit.client.0.smithi113.stdout:0/983: dread d1/d6/f30 [0,4194304] 0 2024-04-12T05:09:13.405 INFO:tasks.workunit.client.1.smithi192.stdout:7/767: truncate d3/d7/d60/d65/f81 463660 0 2024-04-12T05:09:13.405 INFO:tasks.workunit.client.1.smithi192.stdout:3/832: write d5/d13/d3a/d8b/da2/da7/fd9 [1431224,110203] 0 2024-04-12T05:09:13.406 INFO:tasks.workunit.client.0.smithi113.stdout:6/853: fsync d8/de/d17/f72 0 2024-04-12T05:09:13.406 INFO:tasks.workunit.client.0.smithi113.stdout:2/893: dwrite d8/d9/d27/d88/fa4 [0,4194304] 0 2024-04-12T05:09:13.407 INFO:tasks.workunit.client.0.smithi113.stdout:3/895: mkdir d0/d13/d15/d129 0 2024-04-12T05:09:13.410 INFO:tasks.workunit.client.1.smithi192.stdout:7/768: mkdir d3/d1d/d5a/d5f/d7a/da6/d105 0 2024-04-12T05:09:13.411 INFO:tasks.workunit.client.0.smithi113.stdout:6/854: chown d8/de/d13/d11 59218 1 2024-04-12T05:09:13.412 INFO:tasks.workunit.client.1.smithi192.stdout:3/833: truncate d5/d13/d3a/d70/d8c/f96 3635891 0 2024-04-12T05:09:13.413 INFO:tasks.workunit.client.0.smithi113.stdout:2/894: dread - d8/d9/d27/d88/fa3 zero size 2024-04-12T05:09:13.414 INFO:tasks.workunit.client.0.smithi113.stdout:8/960: truncate d9/d1b/d96/fe7 246427 0 2024-04-12T05:09:13.414 INFO:tasks.workunit.client.1.smithi192.stdout:7/769: creat d3/d1d/d74/d86/d93/f106 x:0 0 0 2024-04-12T05:09:13.416 INFO:tasks.workunit.client.0.smithi113.stdout:3/896: fsync d0/d13/d15/f2e 0 2024-04-12T05:09:13.416 INFO:tasks.workunit.client.0.smithi113.stdout:7/836: link d0/dd/d11/ld7 d0/dc/d6/d78/daa/d108/l11e 0 2024-04-12T05:09:13.416 INFO:tasks.workunit.client.1.smithi192.stdout:3/834: mknod d5/d13/d3a/d8b/da2/da7/c12a 0 2024-04-12T05:09:13.417 INFO:tasks.workunit.client.1.smithi192.stdout:7/770: read - d3/d1d/d4a/d5e/fca zero size 2024-04-12T05:09:13.419 INFO:tasks.workunit.client.0.smithi113.stdout:8/961: write d9/d16/d65/d8c/fb5 [1609196,32562] 0 2024-04-12T05:09:13.420 INFO:tasks.workunit.client.0.smithi113.stdout:3/897: fsync d0/d13/d15/d28/d2a/d8b/d91/f124 0 2024-04-12T05:09:13.420 INFO:tasks.workunit.client.1.smithi192.stdout:3/835: creat d5/d13/d3a/d4e/f12b x:0 0 0 2024-04-12T05:09:13.421 INFO:tasks.workunit.client.0.smithi113.stdout:7/837: creat d0/dd/d95/f11f x:0 0 0 2024-04-12T05:09:13.421 INFO:tasks.workunit.client.0.smithi113.stdout:8/962: stat d9/d1b/d8e/d93/cf4 0 2024-04-12T05:09:13.422 INFO:tasks.workunit.client.1.smithi192.stdout:4/874: dread d1/d54/d55/f30 [0,4194304] 0 2024-04-12T05:09:13.423 INFO:tasks.workunit.client.0.smithi113.stdout:3/898: stat d0/d13/d47/d94/dcf/dd1/de2/c109 0 2024-04-12T05:09:13.424 INFO:tasks.workunit.client.0.smithi113.stdout:2/895: dwrite d8/d9/de/f31 [0,4194304] 0 2024-04-12T05:09:13.428 INFO:tasks.workunit.client.0.smithi113.stdout:7/838: dread d0/dc/da9/d5c/d9a/fab [0,4194304] 0 2024-04-12T05:09:13.434 INFO:tasks.workunit.client.0.smithi113.stdout:7/839: rmdir d0/dc/d6/d9b/d9f/dd1 0 2024-04-12T05:09:13.435 INFO:tasks.workunit.client.0.smithi113.stdout:7/840: fsync d0/dc/de7/f106 0 2024-04-12T05:09:13.437 INFO:tasks.workunit.client.0.smithi113.stdout:7/841: chown d0/dd/f112 1598482 1 2024-04-12T05:09:13.437 INFO:tasks.workunit.client.0.smithi113.stdout:3/899: dwrite d0/d13/d47/d4f/f35 [0,4194304] 0 2024-04-12T05:09:13.440 INFO:tasks.workunit.client.0.smithi113.stdout:3/900: rmdir d0/d13/d15/d28/d2a/d8b/d91/dad 39 2024-04-12T05:09:13.450 INFO:tasks.workunit.client.0.smithi113.stdout:3/901: mkdir d0/d13/d47/d4f/dd3/d12a 0 2024-04-12T05:09:13.452 INFO:tasks.workunit.client.0.smithi113.stdout:7/842: dwrite d0/dd/d95/dd4/f107 [0,4194304] 0 2024-04-12T05:09:13.452 INFO:tasks.workunit.client.0.smithi113.stdout:2/896: dread d8/d9/f92 [0,4194304] 0 2024-04-12T05:09:13.453 INFO:tasks.workunit.client.0.smithi113.stdout:3/902: truncate d0/d13/d15/d28/f10b 631836 0 2024-04-12T05:09:13.454 INFO:tasks.workunit.client.0.smithi113.stdout:2/897: truncate d8/d9/d1a/ff1 2343677 0 2024-04-12T05:09:13.454 INFO:tasks.workunit.client.0.smithi113.stdout:3/903: fdatasync d0/d13/d47/d94/dac/f10e 0 2024-04-12T05:09:13.461 INFO:tasks.workunit.client.0.smithi113.stdout:5/982: sync 2024-04-12T05:09:13.461 INFO:tasks.workunit.client.0.smithi113.stdout:9/903: sync 2024-04-12T05:09:13.462 INFO:tasks.workunit.client.0.smithi113.stdout:2/898: unlink d8/d9/d27/dda/f99 0 2024-04-12T05:09:13.465 INFO:tasks.workunit.client.0.smithi113.stdout:2/899: creat d8/d9/de/d12/d3c/d8a/f111 x:0 0 0 2024-04-12T05:09:13.466 INFO:tasks.workunit.client.0.smithi113.stdout:9/904: mkdir d1/d4/d87/d12f 0 2024-04-12T05:09:13.467 INFO:tasks.workunit.client.0.smithi113.stdout:7/843: getdents d0 0 2024-04-12T05:09:13.468 INFO:tasks.workunit.client.0.smithi113.stdout:5/983: creat d0/d10/d6e/d93/dc9/d14b/d73/d89/f14f x:0 0 0 2024-04-12T05:09:13.468 INFO:tasks.workunit.client.0.smithi113.stdout:3/904: getdents d0/d13/d15/d28/d2a/d34/d5f/d7b/de5 0 2024-04-12T05:09:13.469 INFO:tasks.workunit.client.0.smithi113.stdout:5/984: chown d0/d10 77177742 1 2024-04-12T05:09:13.469 INFO:tasks.workunit.client.0.smithi113.stdout:2/900: creat d8/d9/de/d43/db2/f112 x:0 0 0 2024-04-12T05:09:13.470 INFO:tasks.workunit.client.0.smithi113.stdout:7/844: rmdir d0/dc/d6/d78/df2 39 2024-04-12T05:09:13.470 INFO:tasks.workunit.client.0.smithi113.stdout:3/905: fdatasync d0/d13/d47/fb2 0 2024-04-12T05:09:13.472 INFO:tasks.workunit.client.0.smithi113.stdout:2/901: mkdir d8/d16/d113 0 2024-04-12T05:09:13.473 INFO:tasks.workunit.client.0.smithi113.stdout:2/902: readlink d8/d9/lf 0 2024-04-12T05:09:13.473 INFO:tasks.workunit.client.0.smithi113.stdout:7/845: rmdir d0/dc/da9/d5c/d9a/d9e 39 2024-04-12T05:09:13.475 INFO:tasks.workunit.client.0.smithi113.stdout:7/846: fdatasync d0/dc/d6/d9b/dcd/ffb 0 2024-04-12T05:09:13.475 INFO:tasks.workunit.client.0.smithi113.stdout:7/847: chown d0/dd/d95/dd4/c10a 110133 1 2024-04-12T05:09:13.476 INFO:tasks.workunit.client.0.smithi113.stdout:3/906: mkdir d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/d12b 0 2024-04-12T05:09:13.477 INFO:tasks.workunit.client.0.smithi113.stdout:2/903: mkdir d8/d9/d1a/d103/ded/def/d114 0 2024-04-12T05:09:13.477 INFO:tasks.workunit.client.0.smithi113.stdout:5/985: getdents d0/d10/d27/d5f/d128 0 2024-04-12T05:09:13.479 INFO:tasks.workunit.client.0.smithi113.stdout:2/904: creat d8/d9/d1a/d103/f115 x:0 0 0 2024-04-12T05:09:13.480 INFO:tasks.workunit.client.0.smithi113.stdout:2/905: chown d8/d9/d27/d88/fab 202 1 2024-04-12T05:09:13.482 INFO:tasks.workunit.client.0.smithi113.stdout:2/906: fsync d8/d16/f1d 0 2024-04-12T05:09:13.482 INFO:tasks.workunit.client.0.smithi113.stdout:7/848: link d0/dc/d6/fa d0/dc/d6/f120 0 2024-04-12T05:09:13.483 INFO:tasks.workunit.client.0.smithi113.stdout:2/907: truncate d8/d9/d27/ffd 715277 0 2024-04-12T05:09:13.485 INFO:tasks.workunit.client.0.smithi113.stdout:7/849: mknod d0/dc/d6/d9b/c121 0 2024-04-12T05:09:13.486 INFO:tasks.workunit.client.0.smithi113.stdout:2/908: chown d8/d9/d1a/caf 1200128852 1 2024-04-12T05:09:13.487 INFO:tasks.workunit.client.1.smithi192.stdout:7/771: mknod d3/d1d/d5a/d5f/d6a/de9/c107 0 2024-04-12T05:09:13.489 INFO:tasks.workunit.client.0.smithi113.stdout:2/909: fdatasync d8/d9/de/d12/f5d 0 2024-04-12T05:09:13.490 INFO:tasks.workunit.client.0.smithi113.stdout:0/984: truncate d1/d6/d8e/fec 2417638 0 2024-04-12T05:09:13.493 INFO:tasks.workunit.client.0.smithi113.stdout:2/910: symlink d8/d9/de/d43/dd6/l116 0 2024-04-12T05:09:13.494 INFO:tasks.workunit.client.0.smithi113.stdout:9/905: write d1/df/d4b/d63/dad/ff2 [381316,112507] 0 2024-04-12T05:09:13.495 INFO:tasks.workunit.client.1.smithi192.stdout:3/836: symlink d5/d13/d3a/d8b/df0/l12c 0 2024-04-12T05:09:13.496 INFO:tasks.workunit.client.0.smithi113.stdout:2/911: unlink d8/d9/d27/ffd 0 2024-04-12T05:09:13.498 INFO:tasks.workunit.client.1.smithi192.stdout:7/772: truncate d3/d1d/d4a/d5e/d6d/ffc 757813 0 2024-04-12T05:09:13.499 INFO:tasks.workunit.client.0.smithi113.stdout:2/912: fdatasync d8/d9/de/d12/d2b/f8e 0 2024-04-12T05:09:13.505 INFO:tasks.workunit.client.0.smithi113.stdout:9/906: dwrite d1/df/f36 [4194304,4194304] 0 2024-04-12T05:09:13.511 INFO:tasks.workunit.client.0.smithi113.stdout:7/850: dwrite d0/d6c/fb0 [0,4194304] 0 2024-04-12T05:09:13.513 INFO:tasks.workunit.client.1.smithi192.stdout:0/911: sync 2024-04-12T05:09:13.513 INFO:tasks.workunit.client.1.smithi192.stdout:2/811: sync 2024-04-12T05:09:13.513 INFO:tasks.workunit.client.1.smithi192.stdout:1/829: sync 2024-04-12T05:09:13.514 INFO:tasks.workunit.client.1.smithi192.stdout:6/874: sync 2024-04-12T05:09:13.514 INFO:tasks.workunit.client.1.smithi192.stdout:9/684: sync 2024-04-12T05:09:13.514 INFO:tasks.workunit.client.1.smithi192.stdout:3/837: link d5/d13/d3a/d70/d8c/cd4 d5/d13/d1a/d53/c12d 0 2024-04-12T05:09:13.516 INFO:tasks.workunit.client.0.smithi113.stdout:9/907: dwrite d1/d27/d7a/fde [0,4194304] 0 2024-04-12T05:09:13.516 INFO:tasks.workunit.client.0.smithi113.stdout:0/985: symlink d1/d43/d59/de9/d105/d50/l135 0 2024-04-12T05:09:13.517 INFO:tasks.workunit.client.1.smithi192.stdout:2/812: write d1/d8/d17/db3/dee/ff8 [329632,25271] 0 2024-04-12T05:09:13.520 INFO:tasks.workunit.client.1.smithi192.stdout:6/875: creat d3/d8/d56/f121 x:0 0 0 2024-04-12T05:09:13.523 INFO:tasks.workunit.client.0.smithi113.stdout:7/851: symlink d0/dc/da9/d5c/d9a/db6/d101/l122 0 2024-04-12T05:09:13.524 INFO:tasks.workunit.client.0.smithi113.stdout:2/913: link d8/d9/d27/c2d d8/d9/c117 0 2024-04-12T05:09:13.526 INFO:tasks.workunit.client.1.smithi192.stdout:3/838: symlink d5/d13/d1a/d5f/l12e 0 2024-04-12T05:09:13.526 INFO:tasks.workunit.client.0.smithi113.stdout:9/908: dwrite d1/d27/f3c [4194304,4194304] 0 2024-04-12T05:09:13.527 INFO:tasks.workunit.client.0.smithi113.stdout:7/852: mkdir d0/dc/d50/d123 0 2024-04-12T05:09:13.529 INFO:tasks.workunit.client.1.smithi192.stdout:2/813: rmdir d1/d8/d17/db3 39 2024-04-12T05:09:13.531 INFO:tasks.workunit.client.1.smithi192.stdout:6/876: fsync d3/d5/d9c/f101 0 2024-04-12T05:09:13.532 INFO:tasks.workunit.client.1.smithi192.stdout:6/877: stat d3/dc/cdc 0 2024-04-12T05:09:13.533 INFO:tasks.workunit.client.1.smithi192.stdout:9/685: mkdir d11/d25/d36/d3d/d7a/df9 0 2024-04-12T05:09:13.535 INFO:tasks.workunit.client.0.smithi113.stdout:9/909: mkdir d1/df/d4b/d63/dad/d130 0 2024-04-12T05:09:13.535 INFO:tasks.workunit.client.0.smithi113.stdout:0/986: rename d1/d6/d16/c6f to d1/d43/d59/c136 0 2024-04-12T05:09:13.536 INFO:tasks.workunit.client.1.smithi192.stdout:9/686: chown d11/d25/d36/d6e 328 1 2024-04-12T05:09:13.536 INFO:tasks.workunit.client.0.smithi113.stdout:7/853: mknod d0/dc/d6/d1a/c124 0 2024-04-12T05:09:13.536 INFO:tasks.workunit.client.0.smithi113.stdout:9/910: fsync d1/f46 0 2024-04-12T05:09:13.537 INFO:tasks.workunit.client.1.smithi192.stdout:3/839: mknod d5/d13/d1a/d53/c12f 0 2024-04-12T05:09:13.538 INFO:tasks.workunit.client.0.smithi113.stdout:2/914: getdents d8/d9/de/d43/dee 0 2024-04-12T05:09:13.538 INFO:tasks.workunit.client.0.smithi113.stdout:7/854: fsync d0/dc/d6/d9b/ff3 0 2024-04-12T05:09:13.539 INFO:tasks.workunit.client.1.smithi192.stdout:7/773: dread d3/d1d/f69 [0,4194304] 0 2024-04-12T05:09:13.540 INFO:tasks.workunit.client.1.smithi192.stdout:4/875: dwrite d1/d54/d55/d64/f8f [0,4194304] 0 2024-04-12T05:09:13.541 INFO:tasks.workunit.client.1.smithi192.stdout:2/814: fsync d1/d8/d17/db3/dee/ff8 0 2024-04-12T05:09:13.542 INFO:tasks.workunit.client.0.smithi113.stdout:9/911: creat d1/df/d4b/d63/dad/f131 x:0 0 0 2024-04-12T05:09:13.542 INFO:tasks.workunit.client.0.smithi113.stdout:0/987: creat d1/d6/d34/dc3/d87/f137 x:0 0 0 2024-04-12T05:09:13.543 INFO:tasks.workunit.client.0.smithi113.stdout:2/915: mknod d8/d9/de/d15/d24/c118 0 2024-04-12T05:09:13.544 INFO:tasks.workunit.client.0.smithi113.stdout:7/855: read d0/f33 [6144802,128159] 0 2024-04-12T05:09:13.546 INFO:tasks.workunit.client.1.smithi192.stdout:9/687: write d11/d25/dec/d1f/d50/d73/fc5 [6138,57205] 0 2024-04-12T05:09:13.547 INFO:tasks.workunit.client.1.smithi192.stdout:0/912: dread d0/f20 [0,4194304] 0 2024-04-12T05:09:13.548 INFO:tasks.workunit.client.0.smithi113.stdout:0/988: chown d1/d80/f84 15967856 1 2024-04-12T05:09:13.548 INFO:tasks.workunit.client.1.smithi192.stdout:1/830: link d0/d2/d1e/d3f/d56/cad d0/d2/c119 0 2024-04-12T05:09:13.550 INFO:tasks.workunit.client.1.smithi192.stdout:3/840: symlink d5/d13/d1a/d53/dbb/l130 0 2024-04-12T05:09:13.551 INFO:tasks.workunit.client.0.smithi113.stdout:2/916: symlink d8/d9/l119 0 2024-04-12T05:09:13.551 INFO:tasks.workunit.client.0.smithi113.stdout:0/989: write d1/d6/d16/f1c [8047171,29818] 0 2024-04-12T05:09:13.551 INFO:tasks.workunit.client.1.smithi192.stdout:1/831: chown d0/d2/d1e/d3f/df2/d21/d62/dbc/f102 3413502 1 2024-04-12T05:09:13.551 INFO:tasks.workunit.client.1.smithi192.stdout:4/876: dread d1/d1d/d53/d1c/dd1/def/ddd/d114/ff1 [0,4194304] 0 2024-04-12T05:09:13.552 INFO:tasks.workunit.client.1.smithi192.stdout:2/815: rmdir d1/d8/d17/d19/d60 39 2024-04-12T05:09:13.553 INFO:tasks.workunit.client.0.smithi113.stdout:2/917: rmdir d8/d9 39 2024-04-12T05:09:13.556 INFO:tasks.workunit.client.0.smithi113.stdout:0/990: mkdir d1/d80/d138 0 2024-04-12T05:09:13.556 INFO:tasks.workunit.client.0.smithi113.stdout:2/918: truncate d8/d9/d27/dd9/fdb 422322 0 2024-04-12T05:09:13.558 INFO:tasks.workunit.client.1.smithi192.stdout:7/774: truncate d3/d1d/d4a/d5e/d6d/fde 693334 0 2024-04-12T05:09:13.559 INFO:tasks.workunit.client.0.smithi113.stdout:0/991: mknod d1/d80/c139 0 2024-04-12T05:09:13.560 INFO:tasks.workunit.client.0.smithi113.stdout:2/919: creat d8/d9/d27/f11a x:0 0 0 2024-04-12T05:09:13.561 INFO:tasks.workunit.client.1.smithi192.stdout:8/818: sync 2024-04-12T05:09:13.561 INFO:tasks.workunit.client.1.smithi192.stdout:5/947: sync 2024-04-12T05:09:13.563 INFO:tasks.workunit.client.0.smithi113.stdout:0/992: dread d1/d6/d16/f100 [0,4194304] 0 2024-04-12T05:09:13.563 INFO:tasks.workunit.client.0.smithi113.stdout:9/912: write d1/d4/d38/f62 [1549445,53849] 0 2024-04-12T05:09:13.564 INFO:tasks.workunit.client.1.smithi192.stdout:5/948: write d0/f6e [2627233,46669] 0 2024-04-12T05:09:13.565 INFO:tasks.workunit.client.0.smithi113.stdout:2/920: mkdir d8/d9/de/d12/d2b/d79/ddf/d11b 0 2024-04-12T05:09:13.566 INFO:tasks.workunit.client.0.smithi113.stdout:9/913: fsync d1/d4/f77 0 2024-04-12T05:09:13.568 INFO:tasks.workunit.client.0.smithi113.stdout:9/914: chown d1/d27/d7a/d11b/l122 5729 1 2024-04-12T05:09:13.569 INFO:tasks.workunit.client.0.smithi113.stdout:7/856: dwrite d0/dd/d11/de6/f64 [0,4194304] 0 2024-04-12T05:09:13.570 INFO:tasks.workunit.client.0.smithi113.stdout:7/857: stat d0/dd/d11 0 2024-04-12T05:09:13.573 INFO:tasks.workunit.client.0.smithi113.stdout:9/915: rename d1/d4/d5/d6 to d1/d4/d5/d17/d117/d132 0 2024-04-12T05:09:13.576 INFO:tasks.workunit.client.0.smithi113.stdout:7/858: unlink d0/dc/d6/d1a/c3b 0 2024-04-12T05:09:13.577 INFO:tasks.workunit.client.0.smithi113.stdout:9/916: getdents d1/d4/d38/d127 0 2024-04-12T05:09:13.578 INFO:tasks.workunit.client.1.smithi192.stdout:5/949: dread d0/d15/d32/d69/d7d/fc9 [0,4194304] 0 2024-04-12T05:09:13.579 INFO:tasks.workunit.client.1.smithi192.stdout:5/950: chown d0/d15/d3e/dca/dda/da0 7810 1 2024-04-12T05:09:13.580 INFO:tasks.workunit.client.0.smithi113.stdout:2/921: dwrite d8/d9/de/d12/d2b/fa1 [0,4194304] 0 2024-04-12T05:09:13.582 INFO:tasks.workunit.client.0.smithi113.stdout:9/917: creat d1/d27/d7a/dd2/f133 x:0 0 0 2024-04-12T05:09:13.584 INFO:tasks.workunit.client.0.smithi113.stdout:2/922: creat d8/d9/de/d43/dee/f11c x:0 0 0 2024-04-12T05:09:13.585 INFO:tasks.workunit.client.1.smithi192.stdout:5/951: dwrite d0/d15/d32/d45/f126 [0,4194304] 0 2024-04-12T05:09:13.586 INFO:tasks.workunit.client.0.smithi113.stdout:9/918: symlink d1/df/d29/de8/d104/l134 0 2024-04-12T05:09:13.588 INFO:tasks.workunit.client.0.smithi113.stdout:7/859: write d0/dc/f68 [695846,32292] 0 2024-04-12T05:09:13.591 INFO:tasks.workunit.client.0.smithi113.stdout:9/919: creat d1/d4/d5/d22/f135 x:0 0 0 2024-04-12T05:09:13.593 INFO:tasks.workunit.client.0.smithi113.stdout:7/860: creat d0/dc/de7/f125 x:0 0 0 2024-04-12T05:09:13.595 INFO:tasks.workunit.client.0.smithi113.stdout:7/861: chown d0/dc/da9/d5c/cba 7959 1 2024-04-12T05:09:13.595 INFO:tasks.workunit.client.0.smithi113.stdout:2/923: dread d8/f1e [0,4194304] 0 2024-04-12T05:09:13.596 INFO:tasks.workunit.client.0.smithi113.stdout:9/920: rename d1/d4/d5/d17/c69 to d1/d4/d38/d65/dcd/c136 0 2024-04-12T05:09:13.599 INFO:tasks.workunit.client.0.smithi113.stdout:2/924: creat d8/d9/de/d12/d44/f11d x:0 0 0 2024-04-12T05:09:13.607 INFO:tasks.workunit.client.0.smithi113.stdout:9/921: write d1/d4/d6d/fd4 [1014072,54000] 0 2024-04-12T05:09:13.608 INFO:tasks.workunit.client.0.smithi113.stdout:7/862: truncate d0/dc/d6/f28 3631040 0 2024-04-12T05:09:13.608 INFO:tasks.workunit.client.0.smithi113.stdout:2/925: write d8/d9/de/d12/f4c [863795,45203] 0 2024-04-12T05:09:13.613 INFO:tasks.workunit.client.0.smithi113.stdout:7/863: getdents d0/dd/d11/dd5 0 2024-04-12T05:09:13.614 INFO:tasks.workunit.client.0.smithi113.stdout:2/926: dread d8/d9/de/d12/f35 [0,4194304] 0 2024-04-12T05:09:13.616 INFO:tasks.workunit.client.0.smithi113.stdout:9/922: truncate d1/df/d29/de8/d104/f124 276168 0 2024-04-12T05:09:13.617 INFO:tasks.workunit.client.0.smithi113.stdout:4/889: sync 2024-04-12T05:09:13.617 INFO:tasks.workunit.client.0.smithi113.stdout:1/980: sync 2024-04-12T05:09:13.619 INFO:tasks.workunit.client.0.smithi113.stdout:7/864: read d0/dc/da9/d5c/fb9 [677779,47911] 0 2024-04-12T05:09:13.622 INFO:tasks.workunit.client.0.smithi113.stdout:2/927: truncate d8/d9/de/f67 2239909 0 2024-04-12T05:09:13.623 INFO:tasks.workunit.client.0.smithi113.stdout:9/923: mknod d1/d4/d5/d17/d117/df1/dff/d10f/c137 0 2024-04-12T05:09:13.625 INFO:tasks.workunit.client.0.smithi113.stdout:7/865: symlink d0/dc/da9/d5c/d9a/d9e/l126 0 2024-04-12T05:09:13.626 INFO:tasks.workunit.client.0.smithi113.stdout:2/928: dread d8/d9/de/d12/d2b/fa1 [0,4194304] 0 2024-04-12T05:09:13.628 INFO:tasks.workunit.client.0.smithi113.stdout:2/929: chown d8/d9/de/d12/d44/lc7 100485373 1 2024-04-12T05:09:13.629 INFO:tasks.workunit.client.0.smithi113.stdout:2/930: chown d8/d9/de/d12/d3c/d8a/ff7 112041486 1 2024-04-12T05:09:13.629 INFO:tasks.workunit.client.0.smithi113.stdout:7/866: symlink d0/dc/d6/d1a/da0/l127 0 2024-04-12T05:09:13.632 INFO:tasks.workunit.client.0.smithi113.stdout:2/931: fsync d8/d9/d27/d88/de0/ff3 0 2024-04-12T05:09:13.633 INFO:tasks.workunit.client.0.smithi113.stdout:4/890: dread da/d24/d60/f99 [0,4194304] 0 2024-04-12T05:09:13.634 INFO:tasks.workunit.client.0.smithi113.stdout:2/932: dread - d8/d9/de/d43/db2/f112 zero size 2024-04-12T05:09:13.634 INFO:tasks.workunit.client.0.smithi113.stdout:4/891: chown da/d24/d60/db8 290584239 1 2024-04-12T05:09:13.634 INFO:tasks.workunit.client.0.smithi113.stdout:9/924: truncate d1/df/f36 699408 0 2024-04-12T05:09:13.638 INFO:tasks.workunit.client.0.smithi113.stdout:9/925: fdatasync d1/d4/d30/dcb/f81 0 2024-04-12T05:09:13.638 INFO:tasks.workunit.client.0.smithi113.stdout:7/867: write d0/dc/d6/d78/f31 [2882267,68505] 0 2024-04-12T05:09:13.639 INFO:tasks.workunit.client.0.smithi113.stdout:2/933: truncate d8/d9/d27/dda/f52 803191 0 2024-04-12T05:09:13.644 INFO:tasks.workunit.client.0.smithi113.stdout:7/868: rename d0/dd/f112 to d0/dc/da9/f128 0 2024-04-12T05:09:13.645 INFO:tasks.workunit.client.0.smithi113.stdout:2/934: fsync d8/d16/f1d 0 2024-04-12T05:09:13.648 INFO:tasks.workunit.client.0.smithi113.stdout:4/892: dwrite da/d24/d7e/d124/d3a/f88 [0,4194304] 0 2024-04-12T05:09:13.652 INFO:tasks.workunit.client.1.smithi192.stdout:0/913: rename d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d3a/c51 to d0/d7b/d9b/d35/d11f/c136 0 2024-04-12T05:09:13.654 INFO:tasks.workunit.client.0.smithi113.stdout:9/926: creat d1/df/d4b/d63/f138 x:0 0 0 2024-04-12T05:09:13.654 INFO:tasks.workunit.client.1.smithi192.stdout:4/877: fsync d1/d54/d82/f5d 0 2024-04-12T05:09:13.656 INFO:tasks.workunit.client.1.smithi192.stdout:1/832: creat d0/d2/d1e/d3f/d56/f11a x:0 0 0 2024-04-12T05:09:13.658 INFO:tasks.workunit.client.0.smithi113.stdout:7/869: mkdir d0/dc/d6/d78/df2/d129 0 2024-04-12T05:09:13.660 INFO:tasks.workunit.client.1.smithi192.stdout:6/878: dwrite d3/f37 [4194304,4194304] 0 2024-04-12T05:09:13.660 INFO:tasks.workunit.client.0.smithi113.stdout:2/935: mkdir d8/d9/de/d12/d2b/d79/d11e 0 2024-04-12T05:09:13.661 INFO:tasks.workunit.client.1.smithi192.stdout:7/775: fdatasync d3/d7/d60/f66 0 2024-04-12T05:09:13.669 INFO:tasks.workunit.client.1.smithi192.stdout:9/688: dwrite f10 [0,4194304] 0 2024-04-12T05:09:13.671 INFO:tasks.workunit.client.1.smithi192.stdout:2/816: rename d1/d2d/faf to d1/d8/d17/d19/d71/f104 0 2024-04-12T05:09:13.679 INFO:tasks.workunit.client.1.smithi192.stdout:4/878: unlink d1/d54/d55/dc3/f104 0 2024-04-12T05:09:13.679 INFO:tasks.workunit.client.1.smithi192.stdout:0/914: mkdir d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d137 0 2024-04-12T05:09:13.682 INFO:tasks.workunit.client.1.smithi192.stdout:1/833: mkdir d0/d29/d8d/d96/d11b 0 2024-04-12T05:09:13.683 INFO:tasks.workunit.client.1.smithi192.stdout:7/776: creat d3/d7/d15/d1a/f108 x:0 0 0 2024-04-12T05:09:13.688 INFO:tasks.workunit.client.1.smithi192.stdout:9/689: unlink d11/d25/d36/c58 0 2024-04-12T05:09:13.690 INFO:tasks.workunit.client.1.smithi192.stdout:7/777: dwrite d3/d7/d15/f29 [0,4194304] 0 2024-04-12T05:09:13.696 INFO:tasks.workunit.client.1.smithi192.stdout:3/841: link d5/d13/d3c/l29 d5/d13/d3a/d70/d8c/de1/l131 0 2024-04-12T05:09:13.697 INFO:tasks.workunit.client.1.smithi192.stdout:8/819: dread d6/d10/d46/d5e/d5f/d37/d4f/d4c/f55 [0,4194304] 0 2024-04-12T05:09:13.699 INFO:tasks.workunit.client.1.smithi192.stdout:9/690: write d11/d25/d36/d3d/fb3 [2868683,83272] 0 2024-04-12T05:09:13.700 INFO:tasks.workunit.client.1.smithi192.stdout:6/879: creat d3/f122 x:0 0 0 2024-04-12T05:09:13.702 INFO:tasks.workunit.client.1.smithi192.stdout:7/778: mknod d3/d1d/d74/d86/c109 0 2024-04-12T05:09:13.706 INFO:tasks.workunit.client.1.smithi192.stdout:2/817: mkdir d1/d8/d17/d19/d60/d59/d68/d105 0 2024-04-12T05:09:13.707 INFO:tasks.workunit.client.1.smithi192.stdout:3/842: truncate d5/d13/f18 7596318 0 2024-04-12T05:09:13.708 INFO:tasks.workunit.client.1.smithi192.stdout:8/820: mkdir d6/d10/d6c/d9e/dde/dfc 0 2024-04-12T05:09:13.709 INFO:tasks.workunit.client.1.smithi192.stdout:3/843: dread - d5/d13/d3a/d4e/f12b zero size 2024-04-12T05:09:13.710 INFO:tasks.workunit.client.1.smithi192.stdout:4/879: write d1/d54/d82/f48 [599377,47373] 0 2024-04-12T05:09:13.711 INFO:tasks.workunit.client.1.smithi192.stdout:0/915: rename d0/d7b/d9b/d35/d109/d9f/d10a/d67/cf1 to d0/d7b/d9b/d35/d109/d9f/d10a/c138 0 2024-04-12T05:09:13.713 INFO:tasks.workunit.client.1.smithi192.stdout:2/818: unlink d1/d8/d3a/de/d7c/d98/fe4 0 2024-04-12T05:09:13.714 INFO:tasks.workunit.client.1.smithi192.stdout:3/844: fsync d5/d13/d3a/d8b/da2/de5/f116 0 2024-04-12T05:09:13.715 INFO:tasks.workunit.client.1.smithi192.stdout:9/691: creat d11/d25/d36/d6e/d33/df3/ffa x:0 0 0 2024-04-12T05:09:13.716 INFO:tasks.workunit.client.1.smithi192.stdout:5/952: dwrite d0/d21/f33 [0,4194304] 0 2024-04-12T05:09:13.728 INFO:tasks.workunit.client.1.smithi192.stdout:4/880: truncate d1/d54/d82/da4/f127 835064 0 2024-04-12T05:09:13.729 INFO:tasks.workunit.client.1.smithi192.stdout:9/692: dread d11/d25/dec/f8a [0,4194304] 0 2024-04-12T05:09:13.730 INFO:tasks.workunit.client.1.smithi192.stdout:1/834: dwrite d0/d2/d1e/d3f/df2/f32 [0,4194304] 0 2024-04-12T05:09:13.737 INFO:tasks.workunit.client.1.smithi192.stdout:8/821: dread d6/d1f/f5c [0,4194304] 0 2024-04-12T05:09:13.750 INFO:tasks.workunit.client.1.smithi192.stdout:4/881: dread d1/d54/f94 [0,4194304] 0 2024-04-12T05:09:13.751 INFO:tasks.workunit.client.0.smithi113.stdout:1/981: symlink d10/d41/l138 0 2024-04-12T05:09:13.751 INFO:tasks.workunit.client.0.smithi113.stdout:9/927: dwrite d1/d27/f95 [0,4194304] 0 2024-04-12T05:09:13.753 INFO:tasks.workunit.client.0.smithi113.stdout:0/993: truncate d1/d6/fb0 482800 0 2024-04-12T05:09:13.756 INFO:tasks.workunit.client.0.smithi113.stdout:2/936: rmdir d8/d9/de/d15/d24/d58 39 2024-04-12T05:09:13.758 INFO:tasks.workunit.client.0.smithi113.stdout:9/928: creat d1/d4/d5/d17/d117/df1/f139 x:0 0 0 2024-04-12T05:09:13.759 INFO:tasks.workunit.client.0.smithi113.stdout:0/994: write d1/d43/d59/de9/d105/d50/f7d [2339827,40485] 0 2024-04-12T05:09:13.759 INFO:tasks.workunit.client.0.smithi113.stdout:7/870: dwrite d0/d6c/fb0 [4194304,4194304] 0 2024-04-12T05:09:13.760 INFO:tasks.workunit.client.0.smithi113.stdout:2/937: dread - d8/d9/d27/d88/fa3 zero size 2024-04-12T05:09:13.763 INFO:tasks.workunit.client.0.smithi113.stdout:7/871: chown d0/dc/da9/fff 7120 1 2024-04-12T05:09:13.764 INFO:tasks.workunit.client.1.smithi192.stdout:7/779: getdents d3/d7/d14/d1f 0 2024-04-12T05:09:13.764 INFO:tasks.workunit.client.0.smithi113.stdout:9/929: creat d1/d27/d7a/f13a x:0 0 0 2024-04-12T05:09:13.764 INFO:tasks.workunit.client.0.smithi113.stdout:2/938: dread - d8/d9/de/d12/d3c/fb4 zero size 2024-04-12T05:09:13.765 INFO:tasks.workunit.client.0.smithi113.stdout:7/872: chown d0/fa2 96216348 1 2024-04-12T05:09:13.766 INFO:tasks.workunit.client.0.smithi113.stdout:9/930: write d1/d27/d7a/de4/db7/ffb [881157,95504] 0 2024-04-12T05:09:13.766 INFO:tasks.workunit.client.0.smithi113.stdout:2/939: write d8/d9/de/d12/f9c [2505853,97076] 0 2024-04-12T05:09:13.767 INFO:tasks.workunit.client.0.smithi113.stdout:0/995: symlink d1/d43/d59/df4/l13a 0 2024-04-12T05:09:13.767 INFO:tasks.workunit.client.1.smithi192.stdout:3/845: symlink d5/d13/d3a/d4a/d61/l132 0 2024-04-12T05:09:13.769 INFO:tasks.workunit.client.0.smithi113.stdout:9/931: chown d1/df/d4b/d63/d3d/c7c 18430 1 2024-04-12T05:09:13.771 INFO:tasks.workunit.client.0.smithi113.stdout:7/873: truncate d0/dd/d11/f5e 1363593 0 2024-04-12T05:09:13.772 INFO:tasks.workunit.client.1.smithi192.stdout:5/953: unlink d0/d15/d24/c4f 0 2024-04-12T05:09:13.773 INFO:tasks.workunit.client.0.smithi113.stdout:2/940: mknod d8/d9/de/d12/d2b/d79/ddf/d11b/c11f 0 2024-04-12T05:09:13.776 INFO:tasks.workunit.client.0.smithi113.stdout:0/996: creat d1/d80/d138/f13b x:0 0 0 2024-04-12T05:09:13.776 INFO:tasks.workunit.client.0.smithi113.stdout:2/941: symlink d8/d9/de/d12/d2b/d79/d11e/l120 0 2024-04-12T05:09:13.779 INFO:tasks.workunit.client.0.smithi113.stdout:0/997: truncate d1/d6/d16/dd5/f37 1279993 0 2024-04-12T05:09:13.783 INFO:tasks.workunit.client.1.smithi192.stdout:8/822: creat d6/d10/d6c/db4/ffd x:0 0 0 2024-04-12T05:09:13.786 INFO:tasks.workunit.client.1.smithi192.stdout:6/880: rename d3/d8/d56/cc1 to d3/c123 0 2024-04-12T05:09:13.788 INFO:tasks.workunit.client.1.smithi192.stdout:3/846: creat d5/d13/d3c/d36/f133 x:0 0 0 2024-04-12T05:09:13.790 INFO:tasks.workunit.client.0.smithi113.stdout:0/998: link d1/fa6 d1/d43/d59/de9/d105/d50/d129/f13c 0 2024-04-12T05:09:13.791 INFO:tasks.workunit.client.1.smithi192.stdout:3/847: dread d5/d13/d3a/d70/d8c/f122 [0,4194304] 0 2024-04-12T05:09:13.792 INFO:tasks.workunit.client.0.smithi113.stdout:2/942: dwrite d8/d9/de/f67 [0,4194304] 0 2024-04-12T05:09:13.792 INFO:tasks.workunit.client.1.smithi192.stdout:5/954: fdatasync d0/d15/d3e/f71 0 2024-04-12T05:09:13.793 INFO:tasks.workunit.client.1.smithi192.stdout:5/955: write d0/fcf [1188957,37902] 0 2024-04-12T05:09:13.794 INFO:tasks.workunit.client.0.smithi113.stdout:0/999: creat d1/d80/f13d x:0 0 0 2024-04-12T05:09:13.797 INFO:tasks.workunit.client.1.smithi192.stdout:9/693: creat d11/d25/d36/d6e/d33/da2/ddf/ffb x:0 0 0 2024-04-12T05:09:13.798 INFO:tasks.workunit.client.0.smithi113.stdout:1/982: dread d10/d41/f4a [8388608,4194304] 0 2024-04-12T05:09:13.801 INFO:tasks.workunit.client.0.smithi113.stdout:2/943: dwrite d8/d9/d27/dd9/fdb [0,4194304] 0 2024-04-12T05:09:13.811 INFO:tasks.workunit.client.1.smithi192.stdout:9/694: dread - d11/d25/d36/d6e/d33/da2/fc4 zero size 2024-04-12T05:09:13.813 INFO:tasks.workunit.client.1.smithi192.stdout:4/882: getdents d1/d1d/d53/d1c/d95/dc6 0 2024-04-12T05:09:13.814 INFO:tasks.workunit.client.1.smithi192.stdout:6/881: creat d3/f124 x:0 0 0 2024-04-12T05:09:13.815 INFO:tasks.workunit.client.1.smithi192.stdout:6/882: stat d3/d8/d22/dd4 0 2024-04-12T05:09:13.816 INFO:tasks.workunit.client.1.smithi192.stdout:9/695: creat d11/d25/dec/d1f/d94/ffc x:0 0 0 2024-04-12T05:09:13.817 INFO:tasks.workunit.client.1.smithi192.stdout:9/696: chown d11/d25/d36/d6e/d33/dbb 15602909 1 2024-04-12T05:09:13.818 INFO:tasks.workunit.client.1.smithi192.stdout:6/883: creat d3/d8/de6/f125 x:0 0 0 2024-04-12T05:09:13.818 INFO:tasks.workunit.client.1.smithi192.stdout:3/848: dread d5/d13/d1a/d5f/d103/f107 [0,4194304] 0 2024-04-12T05:09:13.819 INFO:tasks.workunit.client.1.smithi192.stdout:6/884: fsync d3/f122 0 2024-04-12T05:09:13.824 INFO:tasks.workunit.client.1.smithi192.stdout:3/849: creat d5/d13/d94/dbd/d117/f134 x:0 0 0 2024-04-12T05:09:13.824 INFO:tasks.workunit.client.1.smithi192.stdout:0/916: write d0/d7b/d9b/d35/d109/d53/df3/d2e/d66/f82 [414150,30309] 0 2024-04-12T05:09:13.829 INFO:tasks.workunit.client.1.smithi192.stdout:1/835: write d0/d1b/d44/d69/faa [683937,102746] 0 2024-04-12T05:09:13.829 INFO:tasks.workunit.client.0.smithi113.stdout:2/944: dread d8/d9/de/d12/f5d [0,4194304] 0 2024-04-12T05:09:13.829 INFO:tasks.workunit.client.1.smithi192.stdout:9/697: dwrite d11/d25/ff1 [0,4194304] 0 2024-04-12T05:09:13.831 INFO:tasks.workunit.client.1.smithi192.stdout:3/850: dread - d5/d13/d3a/d70/d8c/de1/d10e/f11d zero size 2024-04-12T05:09:13.831 INFO:tasks.workunit.client.0.smithi113.stdout:1/983: dread d10/d41/f60 [0,4194304] 0 2024-04-12T05:09:13.831 INFO:tasks.workunit.client.1.smithi192.stdout:6/885: dwrite d3/dc/d39/f4e [4194304,4194304] 0 2024-04-12T05:09:13.832 INFO:tasks.workunit.client.0.smithi113.stdout:2/945: chown d8/d9/de/d12/d2b/daa/db3/fb1 14717766 1 2024-04-12T05:09:13.832 INFO:tasks.workunit.client.1.smithi192.stdout:1/836: fsync d0/d2/d1e/d3f/df2/d3c/dab/fb5 0 2024-04-12T05:09:13.833 INFO:tasks.workunit.client.0.smithi113.stdout:1/984: dread - d10/d2e/d7c/ff9 zero size 2024-04-12T05:09:13.834 INFO:tasks.workunit.client.1.smithi192.stdout:8/823: write d6/d1f/d9c/fb0 [1267503,2658] 0 2024-04-12T05:09:13.837 INFO:tasks.workunit.client.0.smithi113.stdout:1/985: chown d10/f9a 191 1 2024-04-12T05:09:13.838 INFO:tasks.workunit.client.1.smithi192.stdout:5/956: write d0/d15/d3e/dca/dda/deb/d113/d127/fba [2901485,95932] 0 2024-04-12T05:09:13.840 INFO:tasks.workunit.client.0.smithi113.stdout:2/946: mkdir d8/d9/de/d15/d24/d58/d121 0 2024-04-12T05:09:13.841 INFO:tasks.workunit.client.1.smithi192.stdout:1/837: truncate d0/d2/d1e/d3f/df2/d3c/d5b/d74/d90/d108/fa6 1385847 0 2024-04-12T05:09:13.842 INFO:tasks.workunit.client.0.smithi113.stdout:2/947: stat d8/d9/de/d12/dbf/fd8 0 2024-04-12T05:09:13.842 INFO:tasks.workunit.client.1.smithi192.stdout:5/957: read - d0/d15/d3e/dca/dda/db5/dd7/d116/f12a zero size 2024-04-12T05:09:13.845 INFO:tasks.workunit.client.1.smithi192.stdout:9/698: symlink d11/d25/dec/d1f/d6a/d7e/d89/lfd 0 2024-04-12T05:09:13.845 INFO:tasks.workunit.client.1.smithi192.stdout:1/838: truncate d0/d1b/d44/ff8 166462 0 2024-04-12T05:09:13.845 INFO:tasks.workunit.client.1.smithi192.stdout:8/824: dwrite d6/d10/d46/f96 [0,4194304] 0 2024-04-12T05:09:13.846 INFO:tasks.workunit.client.0.smithi113.stdout:6/855: sync 2024-04-12T05:09:13.846 INFO:tasks.workunit.client.0.smithi113.stdout:2/948: creat d8/d9/de/d12/d44/f122 x:0 0 0 2024-04-12T05:09:13.848 INFO:tasks.workunit.client.1.smithi192.stdout:8/825: stat d6/d10/d46/d5e/d5f/d37 0 2024-04-12T05:09:13.848 INFO:tasks.workunit.client.0.smithi113.stdout:1/986: rename d10/d2e/d7c/dc5/dc8/dad/lb9 to d10/d2e/d31/dc3/d109/dde/l139 0 2024-04-12T05:09:13.848 INFO:tasks.workunit.client.0.smithi113.stdout:6/856: dread - d8/de/d13/d11/d36/d3d/d4c/d31/fc0 zero size 2024-04-12T05:09:13.850 INFO:tasks.workunit.client.1.smithi192.stdout:8/826: truncate d6/d10/f11 3942191 0 2024-04-12T05:09:13.850 INFO:tasks.workunit.client.1.smithi192.stdout:1/839: truncate d0/d2/d1e/d3f/df2/d3c/d106/f109 58020 0 2024-04-12T05:09:13.850 INFO:tasks.workunit.client.0.smithi113.stdout:2/949: creat d8/d9/de/d12/d2b/dd2/f123 x:0 0 0 2024-04-12T05:09:13.851 INFO:tasks.workunit.client.1.smithi192.stdout:4/883: truncate d1/d1d/fad 2206504 0 2024-04-12T05:09:13.857 INFO:tasks.workunit.client.1.smithi192.stdout:6/886: write d3/d5/f51 [1123754,125544] 0 2024-04-12T05:09:13.859 INFO:tasks.workunit.client.1.smithi192.stdout:0/917: getdents d0/d7b/d7c/dbb 0 2024-04-12T05:09:13.859 INFO:tasks.workunit.client.1.smithi192.stdout:1/840: chown d0/d2/d1e/d3f/d56/d9e/l75 3247034 1 2024-04-12T05:09:13.862 INFO:tasks.workunit.client.1.smithi192.stdout:8/827: fsync d6/d10/d46/d5e/d5f/f4d 0 2024-04-12T05:09:13.864 INFO:tasks.workunit.client.0.smithi113.stdout:1/987: dwrite d10/d41/d89/d96/f49 [0,4194304] 0 2024-04-12T05:09:13.865 INFO:tasks.workunit.client.1.smithi192.stdout:4/884: symlink d1/d1d/d53/d1c/dd1/def/l12e 0 2024-04-12T05:09:13.866 INFO:tasks.workunit.client.0.smithi113.stdout:1/988: readlink d10/d14/l68 0 2024-04-12T05:09:13.869 INFO:tasks.workunit.client.0.smithi113.stdout:6/857: dread d8/de/d13/d11/d36/d89/f97 [0,4194304] 0 2024-04-12T05:09:13.870 INFO:tasks.workunit.client.1.smithi192.stdout:9/699: write d11/d25/d36/f81 [683000,11608] 0 2024-04-12T05:09:13.871 INFO:tasks.workunit.client.1.smithi192.stdout:8/828: dwrite d6/da/fba [0,4194304] 0 2024-04-12T05:09:13.874 INFO:tasks.workunit.client.0.smithi113.stdout:1/989: dwrite d10/d14/f2a [4194304,4194304] 0 2024-04-12T05:09:13.875 INFO:tasks.workunit.client.1.smithi192.stdout:6/887: rename d3/d8/d48/d85/l102 to d3/d8/dd6/df8/d119/l126 0 2024-04-12T05:09:13.878 INFO:tasks.workunit.client.0.smithi113.stdout:6/858: mkdir d8/de/d13/d11/d36/d3d/d4c/d26/d5b/d136 0 2024-04-12T05:09:13.880 INFO:tasks.workunit.client.0.smithi113.stdout:1/990: rename d10/d2e/d7c/dc5/fc7 to d10/d41/d89/d96/d7d/d9f/f13a 0 2024-04-12T05:09:13.884 INFO:tasks.workunit.client.1.smithi192.stdout:1/841: fdatasync d0/d2/d1e/f36 0 2024-04-12T05:09:13.884 INFO:tasks.workunit.client.1.smithi192.stdout:3/851: dwrite d5/d13/d94/fd5 [0,4194304] 0 2024-04-12T05:09:13.884 INFO:tasks.workunit.client.1.smithi192.stdout:4/885: creat d1/d1d/d53/d1c/d95/dc6/dd5/f12f x:0 0 0 2024-04-12T05:09:13.884 INFO:tasks.workunit.client.1.smithi192.stdout:5/958: link d0/f61 d0/d15/d3e/dca/dda/deb/d113/d127/ded/d136/f146 0 2024-04-12T05:09:13.885 INFO:tasks.workunit.client.0.smithi113.stdout:2/950: dread d8/d9/d1a/fe8 [0,4194304] 0 2024-04-12T05:09:13.888 INFO:tasks.workunit.client.0.smithi113.stdout:1/991: write d10/d2e/d7c/dc5/dc8/d112/f11c [1285427,77633] 0 2024-04-12T05:09:13.888 INFO:tasks.workunit.client.0.smithi113.stdout:2/951: chown d8/d9/d27/dda/c10c 546 1 2024-04-12T05:09:13.889 INFO:tasks.workunit.client.1.smithi192.stdout:1/842: write d0/d2/d1e/d3f/d56/f93 [4341685,751] 0 2024-04-12T05:09:13.889 INFO:tasks.workunit.client.0.smithi113.stdout:6/859: dwrite d8/de/dd5/de7/db3/dc5/f103 [4194304,4194304] 0 2024-04-12T05:09:13.890 INFO:tasks.workunit.client.1.smithi192.stdout:1/843: stat d0/d2/d1e/d3f/df2/d3c/d5b/d7c/lac 0 2024-04-12T05:09:13.892 INFO:tasks.workunit.client.1.smithi192.stdout:0/918: write d0/d7b/d9b/d35/d109/d9f/d10a/d67/dd2/f12a [1127811,113884] 0 2024-04-12T05:09:13.892 INFO:tasks.workunit.client.0.smithi113.stdout:6/860: read - d8/de/d13/d11/d36/d53/db7/f112 zero size 2024-04-12T05:09:13.895 INFO:tasks.workunit.client.0.smithi113.stdout:2/952: truncate d8/d16/f4e 58024 0 2024-04-12T05:09:13.896 INFO:tasks.workunit.client.1.smithi192.stdout:5/959: unlink d0/d15/d64/ca2 0 2024-04-12T05:09:13.896 INFO:tasks.workunit.client.0.smithi113.stdout:2/953: readlink d8/d9/de/d12/d2b/daa/db3/lcf 0 2024-04-12T05:09:13.897 INFO:tasks.workunit.client.1.smithi192.stdout:4/886: dwrite d1/d54/d82/d71/d9e/d10b/f7e [0,4194304] 0 2024-04-12T05:09:13.898 INFO:tasks.workunit.client.0.smithi113.stdout:1/992: truncate d10/d41/faa 57774 0 2024-04-12T05:09:13.906 INFO:tasks.workunit.client.0.smithi113.stdout:1/993: getdents d10/d55/d9c/d137 0 2024-04-12T05:09:13.910 INFO:tasks.workunit.client.1.smithi192.stdout:5/960: dwrite d0/d15/d3e/dca/f109 [0,4194304] 0 2024-04-12T05:09:13.910 INFO:tasks.workunit.client.1.smithi192.stdout:1/844: fsync d0/d2/d1e/d3f/df2/d21/d62/d84/f91 0 2024-04-12T05:09:13.912 INFO:tasks.workunit.client.0.smithi113.stdout:2/954: dread d8/d9/de/d12/d2b/ddd/fff [0,4194304] 0 2024-04-12T05:09:13.912 INFO:tasks.workunit.client.1.smithi192.stdout:4/887: creat d1/d1d/d53/db2/f130 x:0 0 0 2024-04-12T05:09:13.914 INFO:tasks.workunit.client.0.smithi113.stdout:1/994: rename d10/d2e/dd8/d70/f48 to d10/d2e/d7c/dc5/f13b 0 2024-04-12T05:09:13.915 INFO:tasks.workunit.client.0.smithi113.stdout:2/955: chown d8/d9/de/d12/d44/f76 70960544 1 2024-04-12T05:09:13.918 INFO:tasks.workunit.client.0.smithi113.stdout:6/861: dread d8/de/d13/d11/f43 [0,4194304] 0 2024-04-12T05:09:13.919 INFO:tasks.workunit.client.0.smithi113.stdout:2/956: mknod d8/d9/de/d12/d2b/c124 0 2024-04-12T05:09:13.919 INFO:tasks.workunit.client.1.smithi192.stdout:9/700: write d11/d25/d36/d6e/d33/f54 [1865690,59380] 0 2024-04-12T05:09:13.922 INFO:tasks.workunit.client.0.smithi113.stdout:1/995: creat d10/d21/d128/d12f/f13c x:0 0 0 2024-04-12T05:09:13.924 INFO:tasks.workunit.client.1.smithi192.stdout:4/888: rmdir d1/d1d/d53/d1c/d95/dc6/dd5/de5 39 2024-04-12T05:09:13.925 INFO:tasks.workunit.client.1.smithi192.stdout:9/701: creat d11/d25/d36/d6e/d33/da2/ffe x:0 0 0 2024-04-12T05:09:13.927 INFO:tasks.workunit.client.0.smithi113.stdout:1/996: mknod d10/d41/d89/c13d 0 2024-04-12T05:09:13.928 INFO:tasks.workunit.client.1.smithi192.stdout:0/919: getdents d0/d7b/d9b/d35/d109/db0/dbf/dde 0 2024-04-12T05:09:13.932 INFO:tasks.workunit.client.1.smithi192.stdout:4/889: rename d1/d54/f9a to d1/d1d/f131 0 2024-04-12T05:09:13.934 INFO:tasks.workunit.client.1.smithi192.stdout:8/829: write d6/d10/fa5 [1615843,111941] 0 2024-04-12T05:09:13.938 INFO:tasks.workunit.client.1.smithi192.stdout:9/702: rename d11/d15/d21/f65 to d11/d25/dec/d1f/d6a/d7e/d89/fff 0 2024-04-12T05:09:13.940 INFO:tasks.workunit.client.1.smithi192.stdout:6/888: write d3/d5/d9c/fa7 [506581,114660] 0 2024-04-12T05:09:13.941 INFO:tasks.workunit.client.1.smithi192.stdout:4/890: creat d1/d1d/d53/d1c/dd1/f132 x:0 0 0 2024-04-12T05:09:13.943 INFO:tasks.workunit.client.0.smithi113.stdout:2/957: dwrite d8/d9/d27/fb7 [0,4194304] 0 2024-04-12T05:09:13.944 INFO:tasks.workunit.client.1.smithi192.stdout:8/830: dwrite d6/d10/d46/d5e/d5f/f43 [0,4194304] 0 2024-04-12T05:09:13.945 INFO:tasks.workunit.client.0.smithi113.stdout:6/862: dwrite d8/de/d13/d11/d36/d3d/d4c/fdb [0,4194304] 0 2024-04-12T05:09:13.946 INFO:tasks.workunit.client.1.smithi192.stdout:9/703: unlink d11/d25/f32 0 2024-04-12T05:09:13.948 INFO:tasks.workunit.client.1.smithi192.stdout:3/852: dread d5/d13/d3a/d70/d8c/fb3 [0,4194304] 0 2024-04-12T05:09:13.949 INFO:tasks.workunit.client.1.smithi192.stdout:8/831: write d6/d10/f11 [139788,31526] 0 2024-04-12T05:09:13.951 INFO:tasks.workunit.client.0.smithi113.stdout:2/958: creat d8/d9/de/d12/d3c/d8a/f125 x:0 0 0 2024-04-12T05:09:13.953 INFO:tasks.workunit.client.0.smithi113.stdout:6/863: symlink d8/de/d13/df0/d66/dc4/dd9/l137 0 2024-04-12T05:09:13.953 INFO:tasks.workunit.client.1.smithi192.stdout:3/853: chown d5/d13/d3a/d4a/d61/d79/l7f 32268394 1 2024-04-12T05:09:13.954 INFO:tasks.workunit.client.1.smithi192.stdout:6/889: dwrite d3/d5/d9c/dbd/fd3 [0,4194304] 0 2024-04-12T05:09:13.959 INFO:tasks.workunit.client.1.smithi192.stdout:6/890: dread - d3/d8/d56/f121 zero size 2024-04-12T05:09:13.962 INFO:tasks.workunit.client.0.smithi113.stdout:1/997: dwrite d10/d41/d89/d96/d7d/f79 [4194304,4194304] 0 2024-04-12T05:09:13.965 INFO:tasks.workunit.client.0.smithi113.stdout:6/864: mknod d8/de/d13/d11/d36/d3d/d4c/d26/c138 0 2024-04-12T05:09:13.971 INFO:tasks.workunit.client.0.smithi113.stdout:6/865: chown d8/de/c18 3 1 2024-04-12T05:09:13.972 INFO:tasks.workunit.client.1.smithi192.stdout:9/704: dwrite d11/d25/dec/d1f/d94/ffc [0,4194304] 0 2024-04-12T05:09:13.972 INFO:tasks.workunit.client.1.smithi192.stdout:1/845: dread d0/d2/f50 [0,4194304] 0 2024-04-12T05:09:13.972 INFO:tasks.workunit.client.1.smithi192.stdout:6/891: write d3/f37 [2176057,130665] 0 2024-04-12T05:09:13.972 INFO:tasks.workunit.client.1.smithi192.stdout:0/920: dread d0/d7b/d9b/d35/d109/d53/df3/fd8 [0,4194304] 0 2024-04-12T05:09:13.972 INFO:tasks.workunit.client.0.smithi113.stdout:2/959: dwrite d8/d9/de/d12/d2b/fa1 [0,4194304] 0 2024-04-12T05:09:13.973 INFO:tasks.workunit.client.1.smithi192.stdout:0/921: fdatasync d0/fd0 0 2024-04-12T05:09:13.975 INFO:tasks.workunit.client.1.smithi192.stdout:8/832: truncate d6/d10/d46/d5e/d5f/d38/dbc/fec 766739 0 2024-04-12T05:09:13.975 INFO:tasks.workunit.client.0.smithi113.stdout:6/866: readlink d8/de/d13/d11/d36/l30 0 2024-04-12T05:09:13.975 INFO:tasks.workunit.client.0.smithi113.stdout:6/867: stat d8/de 0 2024-04-12T05:09:13.977 INFO:tasks.workunit.client.1.smithi192.stdout:1/846: mkdir d0/d29/d8d/d11c 0 2024-04-12T05:09:13.980 INFO:tasks.workunit.client.0.smithi113.stdout:2/960: dread d8/d9/de/d15/d24/d58/f96 [0,4194304] 0 2024-04-12T05:09:13.984 INFO:tasks.workunit.client.1.smithi192.stdout:3/854: creat d5/d13/d3a/d8b/da2/f135 x:0 0 0 2024-04-12T05:09:13.988 INFO:tasks.workunit.client.1.smithi192.stdout:1/847: unlink d0/d2/d1e/d3f/df2/d21/d62/d84/l86 0 2024-04-12T05:09:13.989 INFO:tasks.workunit.client.1.smithi192.stdout:4/891: creat d1/d54/d82/d71/d9e/f133 x:0 0 0 2024-04-12T05:09:13.990 INFO:tasks.workunit.client.0.smithi113.stdout:2/961: unlink d8/d9/de/d12/d2b/cc4 0 2024-04-12T05:09:13.991 INFO:tasks.workunit.client.1.smithi192.stdout:8/833: dwrite d6/d10/d46/d5e/d5f/f88 [0,4194304] 0 2024-04-12T05:09:13.992 INFO:tasks.workunit.client.0.smithi113.stdout:6/868: rmdir d8/de/d13/d11/d36/d3d/d4c/d120 0 2024-04-12T05:09:13.992 INFO:tasks.workunit.client.1.smithi192.stdout:0/922: getdents d0/d7b/d9b/d35/d109/d53/d6d 0 2024-04-12T05:09:13.992 INFO:tasks.workunit.client.1.smithi192.stdout:6/892: dread d3/d8/fa3 [0,4194304] 0 2024-04-12T05:09:13.993 INFO:tasks.workunit.client.0.smithi113.stdout:2/962: chown d8/d9/d27/f65 460 1 2024-04-12T05:09:13.994 INFO:tasks.workunit.client.0.smithi113.stdout:1/998: write d10/d21/fd1 [394967,11815] 0 2024-04-12T05:09:13.996 INFO:tasks.workunit.client.1.smithi192.stdout:8/834: mknod d6/d1f/d6a/cfe 0 2024-04-12T05:09:13.996 INFO:tasks.workunit.client.0.smithi113.stdout:6/869: mknod d8/de/d13/d11/d36/c139 0 2024-04-12T05:09:13.997 INFO:tasks.workunit.client.0.smithi113.stdout:1/999: mkdir d10/d2e/d7c/dc5/dc8/dad/d13e 0 2024-04-12T05:09:13.998 INFO:tasks.workunit.client.0.smithi113.stdout:6/870: truncate d8/f12a 965252 0 2024-04-12T05:09:13.998 INFO:tasks.workunit.client.1.smithi192.stdout:4/892: mknod d1/d1d/d53/db2/c134 0 2024-04-12T05:09:13.999 INFO:tasks.workunit.client.0.smithi113.stdout:2/963: creat d8/f126 x:0 0 0 2024-04-12T05:09:14.001 INFO:tasks.workunit.client.1.smithi192.stdout:8/835: creat d6/d21/fff x:0 0 0 2024-04-12T05:09:14.001 INFO:tasks.workunit.client.0.smithi113.stdout:2/964: mkdir d8/d9/d1a/d103/d127 0 2024-04-12T05:09:14.003 INFO:tasks.workunit.client.0.smithi113.stdout:6/871: symlink d8/de/d13/d11/d36/l13a 0 2024-04-12T05:09:14.003 INFO:tasks.workunit.client.1.smithi192.stdout:9/705: dwrite d11/f12 [0,4194304] 0 2024-04-12T05:09:14.003 INFO:tasks.workunit.client.0.smithi113.stdout:8/963: sync 2024-04-12T05:09:14.004 INFO:tasks.workunit.client.0.smithi113.stdout:3/907: sync 2024-04-12T05:09:14.005 INFO:tasks.workunit.client.0.smithi113.stdout:2/965: write d8/d9/de/d12/d2b/fb6 [1833792,65181] 0 2024-04-12T05:09:14.010 INFO:tasks.workunit.client.1.smithi192.stdout:0/923: rename d0/d7b/d9b/d35/d109/d53/df3/d2e/d59/f78 to d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/f139 0 2024-04-12T05:09:14.010 INFO:tasks.workunit.client.1.smithi192.stdout:1/848: link d0/d2/d1e/d3f/d56/d9e/c83 d0/d2/d1e/d3f/df2/d3c/dab/c11d 0 2024-04-12T05:09:14.011 INFO:tasks.workunit.client.1.smithi192.stdout:6/893: dread d3/d23/f76 [0,4194304] 0 2024-04-12T05:09:14.012 INFO:tasks.workunit.client.1.smithi192.stdout:8/836: truncate d6/d21/f23 2674717 0 2024-04-12T05:09:14.015 INFO:tasks.workunit.client.1.smithi192.stdout:0/924: write d0/d7b/d9b/d35/d109/d9f/d10a/d67/dd2/f12a [2053999,53814] 0 2024-04-12T05:09:14.024 INFO:tasks.workunit.client.1.smithi192.stdout:8/837: mknod d6/d10/d6c/d9e/dde/dfc/c100 0 2024-04-12T05:09:14.027 INFO:tasks.workunit.client.1.smithi192.stdout:3/855: write d5/d13/d3a/d8b/da2/da7/fd7 [598955,6724] 0 2024-04-12T05:09:14.029 INFO:tasks.workunit.client.1.smithi192.stdout:1/849: dwrite d0/d2/d1e/d3f/df2/d3c/d5b/db6/fd3 [0,4194304] 0 2024-04-12T05:09:14.034 INFO:tasks.workunit.client.1.smithi192.stdout:1/850: chown d0/d2/d1e/f36 30545317 1 2024-04-12T05:09:14.038 INFO:tasks.workunit.client.1.smithi192.stdout:9/706: dwrite fe [4194304,4194304] 0 2024-04-12T05:09:14.048 INFO:tasks.workunit.client.1.smithi192.stdout:0/925: dread d0/d7b/d7c/dcc/f93 [0,4194304] 0 2024-04-12T05:09:14.049 INFO:tasks.workunit.client.1.smithi192.stdout:4/893: dread d1/d54/d82/d71/d9e/fc7 [0,4194304] 0 2024-04-12T05:09:14.052 INFO:tasks.workunit.client.1.smithi192.stdout:8/838: rmdir d6/d1f/d9c/dfb 0 2024-04-12T05:09:14.053 INFO:tasks.workunit.client.1.smithi192.stdout:1/851: write d0/d2/d1e/d3f/df2/d21/d62/d84/f8e [1250396,90441] 0 2024-04-12T05:09:14.054 INFO:tasks.workunit.client.1.smithi192.stdout:6/894: dread d3/d8/de6/ffb [0,4194304] 0 2024-04-12T05:09:14.055 INFO:tasks.workunit.client.1.smithi192.stdout:9/707: symlink d11/d25/d36/d3d/d7a/df9/l100 0 2024-04-12T05:09:14.058 INFO:tasks.workunit.client.1.smithi192.stdout:3/856: link d5/d13/d3a/d70/c109 d5/d13/d3a/d4a/d61/d9f/deb/dfb/c136 0 2024-04-12T05:09:14.059 INFO:tasks.workunit.client.1.smithi192.stdout:0/926: chown d0/d7b/d9b/d35/d109/d9f/d10a/d67/c103 6 1 2024-04-12T05:09:14.061 INFO:tasks.workunit.client.1.smithi192.stdout:8/839: creat d6/d10/d46/d5e/d5f/d37/d7b/f101 x:0 0 0 2024-04-12T05:09:14.063 INFO:tasks.workunit.client.1.smithi192.stdout:3/857: rmdir d5/d13/d3c 39 2024-04-12T05:09:14.064 INFO:tasks.workunit.client.1.smithi192.stdout:9/708: symlink d11/d25/dec/d1f/d6a/d7e/d89/dce/l101 0 2024-04-12T05:09:14.065 INFO:tasks.workunit.client.1.smithi192.stdout:1/852: creat d0/d1b/f11e x:0 0 0 2024-04-12T05:09:14.066 INFO:tasks.workunit.client.0.smithi113.stdout:2/966: fsync d8/d9/de/d15/f9d 0 2024-04-12T05:09:14.068 INFO:tasks.workunit.client.1.smithi192.stdout:8/840: unlink d6/d10/d46/d5e/d5f/cd2 0 2024-04-12T05:09:14.069 INFO:tasks.workunit.client.1.smithi192.stdout:3/858: rmdir d5/d13/d3a/d4a/d61/d79/d11c 39 2024-04-12T05:09:14.070 INFO:tasks.workunit.client.1.smithi192.stdout:4/894: dwrite d1/d54/d55/d64/f74 [0,4194304] 0 2024-04-12T05:09:14.070 INFO:tasks.workunit.client.1.smithi192.stdout:0/927: mkdir d0/d7b/d9b/d35/d109/d53/df3/d2e/d59/d13a 0 2024-04-12T05:09:14.071 INFO:tasks.workunit.client.0.smithi113.stdout:6/872: dread d8/de/d13/d11/d36/d53/db8/fc2 [0,4194304] 0 2024-04-12T05:09:14.072 INFO:tasks.workunit.client.0.smithi113.stdout:8/964: mkdir d9/d16/d65/db4/dcb/d12d 0 2024-04-12T05:09:14.072 INFO:tasks.workunit.client.1.smithi192.stdout:3/859: chown d5/d13/d3a/d4e/d57 15 1 2024-04-12T05:09:14.072 INFO:tasks.workunit.client.0.smithi113.stdout:8/965: chown d9/d1b/d8e/d93 12655005 1 2024-04-12T05:09:14.074 INFO:tasks.workunit.client.1.smithi192.stdout:8/841: dread - d6/d10/d46/d5e/d5f/d37/d7b/f101 zero size 2024-04-12T05:09:14.074 INFO:tasks.workunit.client.1.smithi192.stdout:9/709: rename d11/d25/d60/f97 to d11/d25/d36/d3d/d7a/df9/f102 0 2024-04-12T05:09:14.075 INFO:tasks.workunit.client.0.smithi113.stdout:2/967: creat d8/d16/dbd/f128 x:0 0 0 2024-04-12T05:09:14.076 INFO:tasks.workunit.client.1.smithi192.stdout:3/860: fdatasync d5/d13/d1a/d5f/d103/f107 0 2024-04-12T05:09:14.078 INFO:tasks.workunit.client.1.smithi192.stdout:9/710: symlink d11/d25/dec/d1f/d6a/d92/l103 0 2024-04-12T05:09:14.079 INFO:tasks.workunit.client.1.smithi192.stdout:8/842: creat d6/d1f/f102 x:0 0 0 2024-04-12T05:09:14.080 INFO:tasks.workunit.client.1.smithi192.stdout:3/861: mkdir d5/d13/d3a/d8b/da2/d123/d137 0 2024-04-12T05:09:14.080 INFO:tasks.workunit.client.1.smithi192.stdout:8/843: readlink d6/d10/d46/d5e/d5f/l4e 0 2024-04-12T05:09:14.081 INFO:tasks.workunit.client.1.smithi192.stdout:8/844: dread - d6/d21/fff zero size 2024-04-12T05:09:14.083 INFO:tasks.workunit.client.1.smithi192.stdout:3/862: dread d5/d13/d3a/d70/d8c/d8d/fff [0,4194304] 0 2024-04-12T05:09:14.084 INFO:tasks.workunit.client.1.smithi192.stdout:9/711: symlink d11/d25/d36/d6e/d8f/l104 0 2024-04-12T05:09:14.084 INFO:tasks.workunit.client.0.smithi113.stdout:8/966: mkdir d9/d1b/d8e/d93/d32/d48/d8d/dae/d112/d12e 0 2024-04-12T05:09:14.084 INFO:tasks.workunit.client.1.smithi192.stdout:3/863: chown d5/d13/d1a/d5f/d103/l120 1422 1 2024-04-12T05:09:14.085 INFO:tasks.workunit.client.0.smithi113.stdout:6/873: link d8/de/f94 d8/de/d13/d11/d36/d53/f13b 0 2024-04-12T05:09:14.086 INFO:tasks.workunit.client.1.smithi192.stdout:8/845: mkdir d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d103 0 2024-04-12T05:09:14.086 INFO:tasks.workunit.client.0.smithi113.stdout:2/968: dwrite d8/fc0 [0,4194304] 0 2024-04-12T05:09:14.090 INFO:tasks.workunit.client.0.smithi113.stdout:6/874: write d8/de/d13/d11/d36/d53/db8/dce/d128/f131 [353333,68696] 0 2024-04-12T05:09:14.093 INFO:tasks.workunit.client.0.smithi113.stdout:6/875: write d8/de/dd5/f11c [144278,102656] 0 2024-04-12T05:09:14.096 INFO:tasks.workunit.client.0.smithi113.stdout:2/969: mknod d8/d9/de/d12/d2b/dd2/c129 0 2024-04-12T05:09:14.096 INFO:tasks.workunit.client.1.smithi192.stdout:8/846: rename d6/d10/d46/d5e/d5f/d31/d61/de3 to d6/d10/d46/d5e/d5f/d37/d7b/d104 0 2024-04-12T05:09:14.097 INFO:tasks.workunit.client.0.smithi113.stdout:8/967: fsync d9/d16/d7f/d3a/d6c/d90/fbc 0 2024-04-12T05:09:14.098 INFO:tasks.workunit.client.0.smithi113.stdout:2/970: read d8/d16/f61 [814810,8027] 0 2024-04-12T05:09:14.099 INFO:tasks.workunit.client.0.smithi113.stdout:8/968: chown d9/d16/d65/d121/dec/f118 14953642 1 2024-04-12T05:09:14.101 INFO:tasks.workunit.client.1.smithi192.stdout:8/847: symlink d6/d10/d46/d5e/d5f/d31/d80/l105 0 2024-04-12T05:09:14.103 INFO:tasks.workunit.client.1.smithi192.stdout:1/853: dwrite d0/d2/d1e/d3f/df2/d3c/d5b/db6/f10e [4194304,4194304] 0 2024-04-12T05:09:14.104 INFO:tasks.workunit.client.1.smithi192.stdout:9/712: truncate d11/d15/f5c 327036 0 2024-04-12T05:09:14.104 INFO:tasks.workunit.client.0.smithi113.stdout:8/969: creat d9/d16/d7f/d3a/d6c/f12f x:0 0 0 2024-04-12T05:09:14.105 INFO:tasks.workunit.client.0.smithi113.stdout:8/970: write d9/d16/d65/f6a [1039145,71523] 0 2024-04-12T05:09:14.107 INFO:tasks.workunit.client.0.smithi113.stdout:6/876: getdents d8/de/d13/d11/d36/d53/db8/dce/d128 0 2024-04-12T05:09:14.107 INFO:tasks.workunit.client.1.smithi192.stdout:1/854: dread - d0/dd0/f10c zero size 2024-04-12T05:09:14.108 INFO:tasks.workunit.client.1.smithi192.stdout:0/928: dwrite d0/d7b/d9b/d35/d109/d53/df3/d2e/f63 [0,4194304] 0 2024-04-12T05:09:14.108 INFO:tasks.workunit.client.1.smithi192.stdout:9/713: fdatasync d11/d25/d36/d6e/f56 0 2024-04-12T05:09:14.109 INFO:tasks.workunit.client.0.smithi113.stdout:6/877: write d8/de/d13/d11/d36/f35 [1745230,71090] 0 2024-04-12T05:09:14.114 INFO:tasks.workunit.client.0.smithi113.stdout:8/971: fdatasync d9/d1b/d8e/d93/d32/d48/d8d/d4a/fbd 0 2024-04-12T05:09:14.117 INFO:tasks.workunit.client.1.smithi192.stdout:3/864: dwrite d5/d13/d3a/d70/d8c/d8d/f93 [0,4194304] 0 2024-04-12T05:09:14.118 INFO:tasks.workunit.client.1.smithi192.stdout:0/929: write d0/d7b/d9b/d35/d109/d9f/d10a/d67/db1/dd6/fe6 [785736,54168] 0 2024-04-12T05:09:14.119 INFO:tasks.workunit.client.1.smithi192.stdout:9/714: creat d11/d25/d36/d3d/d7a/df9/f105 x:0 0 0 2024-04-12T05:09:14.120 INFO:tasks.workunit.client.0.smithi113.stdout:8/972: rmdir d9/d1b/d8e/d93/d92 39 2024-04-12T05:09:14.120 INFO:tasks.workunit.client.1.smithi192.stdout:8/848: dread d6/d10/d46/d5e/d5f/d37/d4f/fb2 [0,4194304] 0 2024-04-12T05:09:14.122 INFO:tasks.workunit.client.1.smithi192.stdout:1/855: unlink d0/d2/d1e/d3f/df2/d3c/d5b/d74/d7d/fe0 0 2024-04-12T05:09:14.123 INFO:tasks.workunit.client.1.smithi192.stdout:9/715: chown d11/d25/d36/d6e/d33/dbb/de7 812 1 2024-04-12T05:09:14.124 INFO:tasks.workunit.client.1.smithi192.stdout:3/865: chown d5/d13/d1a/lc8 3264 1 2024-04-12T05:09:14.124 INFO:tasks.workunit.client.1.smithi192.stdout:0/930: mknod d0/d7b/d9b/d35/d109/db0/c13b 0 2024-04-12T05:09:14.128 INFO:tasks.workunit.client.1.smithi192.stdout:8/849: rename d6/d10/d6c/db4 to d6/d1f/d6a/d106 0 2024-04-12T05:09:14.128 INFO:tasks.workunit.client.0.smithi113.stdout:8/973: mkdir d9/d16/d65/db4/dcb/d130 0 2024-04-12T05:09:14.135 INFO:tasks.workunit.client.1.smithi192.stdout:9/716: mknod d11/d25/d36/d6e/d33/da2/c106 0 2024-04-12T05:09:14.135 INFO:tasks.workunit.client.1.smithi192.stdout:1/856: read d0/d2/d1e/d3f/df2/d21/d62/d84/f8e [293715,66317] 0 2024-04-12T05:09:14.141 INFO:tasks.workunit.client.1.smithi192.stdout:9/717: creat d11/d25/d36/d6e/d33/ddd/f107 x:0 0 0 2024-04-12T05:09:14.142 INFO:tasks.workunit.client.1.smithi192.stdout:1/857: write d0/d29/d8d/d96/fdb [2558529,56519] 0 2024-04-12T05:09:14.148 INFO:tasks.workunit.client.1.smithi192.stdout:9/718: rmdir d11/d25/dec/d1f/d6a/d7e 39 2024-04-12T05:09:14.151 INFO:tasks.workunit.client.1.smithi192.stdout:9/719: dread - d11/d25/d36/d3d/fb7 zero size 2024-04-12T05:09:14.151 INFO:tasks.workunit.client.1.smithi192.stdout:1/858: getdents d0/d2/d1e/d3f/df2/d71 0 2024-04-12T05:09:14.153 INFO:tasks.workunit.client.1.smithi192.stdout:1/859: write d0/d2/d1e/d3f/d56/f11a [554338,34567] 0 2024-04-12T05:09:14.153 INFO:tasks.workunit.client.1.smithi192.stdout:3/866: write d5/d13/d3a/d4e/f9a [1149598,124965] 0 2024-04-12T05:09:14.159 INFO:tasks.workunit.client.1.smithi192.stdout:8/850: dread d6/d1f/f2a [0,4194304] 0 2024-04-12T05:09:14.163 INFO:tasks.workunit.client.1.smithi192.stdout:0/931: dwrite d0/d7b/d9b/d35/d109/d9f/d10a/dc4/fba [0,4194304] 0 2024-04-12T05:09:14.165 INFO:tasks.workunit.client.0.smithi113.stdout:8/974: dread d9/d1b/d8e/d93/d32/d5b/fb8 [0,4194304] 0 2024-04-12T05:09:14.165 INFO:tasks.workunit.client.1.smithi192.stdout:8/851: mknod d6/d10/d6c/d9e/dde/c107 0 2024-04-12T05:09:14.166 INFO:tasks.workunit.client.0.smithi113.stdout:8/975: fsync d9/d16/d65/d121/d59/d78/ff7 0 2024-04-12T05:09:14.169 INFO:tasks.workunit.client.1.smithi192.stdout:8/852: fsync d6/d10/d46/d5e/d5f/d38/dbc/fec 0 2024-04-12T05:09:14.170 INFO:tasks.workunit.client.1.smithi192.stdout:0/932: mkdir d0/d7b/d9b/d35/d109/d9f/d10a/db2/d13c 0 2024-04-12T05:09:14.171 INFO:tasks.workunit.client.1.smithi192.stdout:3/867: dwrite d5/d13/d3a/d4a/fdf [0,4194304] 0 2024-04-12T05:09:14.173 INFO:tasks.workunit.client.1.smithi192.stdout:8/853: write d6/d1f/d6a/d106/ffd [1042653,91880] 0 2024-04-12T05:09:14.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:12.674809+0000 mgr.smithi192.ocppyo (mgr.24357) 33 : cluster [DBG] pgmap v26: 65 pgs: 65 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 141 MiB/s rd, 336 MiB/s wr, 714 op/s 2024-04-12T05:09:14.174 INFO:tasks.workunit.client.1.smithi192.stdout:3/868: read d5/d13/d3a/d4a/fdf [1859863,45042] 0 2024-04-12T05:09:14.180 INFO:tasks.workunit.client.1.smithi192.stdout:8/854: creat d6/d10/d46/d5e/d5f/d38/dbc/f108 x:0 0 0 2024-04-12T05:09:14.180 INFO:tasks.workunit.client.1.smithi192.stdout:9/720: dread d11/d25/f52 [0,4194304] 0 2024-04-12T05:09:14.181 INFO:tasks.workunit.client.1.smithi192.stdout:3/869: creat d5/d13/d3a/d4a/d61/d9f/f138 x:0 0 0 2024-04-12T05:09:14.181 INFO:tasks.workunit.client.1.smithi192.stdout:1/860: dread d0/d29/d8d/f101 [0,4194304] 0 2024-04-12T05:09:14.182 INFO:tasks.workunit.client.1.smithi192.stdout:9/721: chown d11/d25/d36/d6e/d71 185430 1 2024-04-12T05:09:14.184 INFO:tasks.workunit.client.1.smithi192.stdout:8/855: unlink d6/d10/d46/d5e/d5f/d31/lda 0 2024-04-12T05:09:14.186 INFO:tasks.workunit.client.1.smithi192.stdout:9/722: creat d11/d25/d36/d6e/d71/f108 x:0 0 0 2024-04-12T05:09:14.188 INFO:tasks.workunit.client.1.smithi192.stdout:1/861: stat d0/d2/d1e/d3f/df2/d3c/d5b/d74/d90/cc6 0 2024-04-12T05:09:14.189 INFO:tasks.workunit.client.1.smithi192.stdout:0/933: dwrite d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d46/f8c [0,4194304] 0 2024-04-12T05:09:14.190 INFO:tasks.workunit.client.1.smithi192.stdout:8/856: stat d6/d10/d46/d5e/d5f/d31/d61/dc5/dd9 0 2024-04-12T05:09:14.192 INFO:tasks.workunit.client.1.smithi192.stdout:0/934: chown d0/l10c 4829507 1 2024-04-12T05:09:14.194 INFO:tasks.workunit.client.1.smithi192.stdout:3/870: dwrite d5/d13/d1a/d5f/fa1 [0,4194304] 0 2024-04-12T05:09:14.194 INFO:tasks.workunit.client.1.smithi192.stdout:1/862: unlink d0/d2/d1e/d3f/df2/d21/f99 0 2024-04-12T05:09:14.197 INFO:tasks.workunit.client.1.smithi192.stdout:0/935: chown d0/d7b/d9b/d35/d109/d53/df3/fd8 0 1 2024-04-12T05:09:14.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:12.674809+0000 mgr.smithi192.ocppyo (mgr.24357) 33 : cluster [DBG] pgmap v26: 65 pgs: 65 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 141 MiB/s rd, 336 MiB/s wr, 714 op/s 2024-04-12T05:09:14.198 INFO:tasks.workunit.client.1.smithi192.stdout:9/723: rename d11/d25/dec/d1f/d6a/d7e/ced to d11/d25/d36/d6e/d33/ddd/c109 0 2024-04-12T05:09:14.204 INFO:tasks.workunit.client.1.smithi192.stdout:8/857: dwrite d6/da/f30 [0,4194304] 0 2024-04-12T05:09:14.207 INFO:tasks.workunit.client.1.smithi192.stdout:0/936: dread d0/d7b/d9b/d35/d109/d53/d6d/f62 [0,4194304] 0 2024-04-12T05:09:14.207 INFO:tasks.workunit.client.1.smithi192.stdout:3/871: dread d5/d13/d1a/d5f/d103/f107 [0,4194304] 0 2024-04-12T05:09:14.209 INFO:tasks.workunit.client.1.smithi192.stdout:1/863: dwrite d0/d1b/f118 [0,4194304] 0 2024-04-12T05:09:14.211 INFO:tasks.workunit.client.1.smithi192.stdout:7/780: sync 2024-04-12T05:09:14.212 INFO:tasks.workunit.client.1.smithi192.stdout:2/819: sync 2024-04-12T05:09:14.213 INFO:tasks.workunit.client.1.smithi192.stdout:8/858: rename d6/da to d6/d10/d109 0 2024-04-12T05:09:14.217 INFO:tasks.workunit.client.1.smithi192.stdout:9/724: dwrite d11/f69 [0,4194304] 0 2024-04-12T05:09:14.222 INFO:tasks.workunit.client.1.smithi192.stdout:2/820: readlink d1/d8/d17/l1a 0 2024-04-12T05:09:14.222 INFO:tasks.workunit.client.1.smithi192.stdout:8/859: chown d6/d10 462 1 2024-04-12T05:09:14.225 INFO:tasks.workunit.client.1.smithi192.stdout:9/725: creat d11/d25/d36/d6e/d33/da2/f10a x:0 0 0 2024-04-12T05:09:14.228 INFO:tasks.workunit.client.1.smithi192.stdout:3/872: creat d5/d13/d3c/d84/f139 x:0 0 0 2024-04-12T05:09:14.229 INFO:tasks.workunit.client.1.smithi192.stdout:2/821: fdatasync d1/d8/d17/f5f 0 2024-04-12T05:09:14.233 INFO:tasks.workunit.client.1.smithi192.stdout:0/937: dwrite d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d46/f4d [4194304,4194304] 0 2024-04-12T05:09:14.234 INFO:tasks.workunit.client.1.smithi192.stdout:8/860: mknod d6/d10/d46/d5e/d5f/d31/d80/db1/c10a 0 2024-04-12T05:09:14.237 INFO:tasks.workunit.client.1.smithi192.stdout:8/861: dread - d6/d10/d46/d5e/d5f/d31/d80/db1/fcb zero size 2024-04-12T05:09:14.239 INFO:tasks.workunit.client.1.smithi192.stdout:1/864: getdents d0/d1b/d28/d9d 0 2024-04-12T05:09:14.240 INFO:tasks.workunit.client.1.smithi192.stdout:7/781: link d3/d7/d14/d1f/c9c d3/d1d/d48/c10a 0 2024-04-12T05:09:14.240 INFO:tasks.workunit.client.1.smithi192.stdout:3/873: mknod d5/d13/d1a/d5f/c13a 0 2024-04-12T05:09:14.244 INFO:tasks.workunit.client.1.smithi192.stdout:7/782: chown d3/d7/d14/d1f/d76 53465131 1 2024-04-12T05:09:14.245 INFO:tasks.workunit.client.1.smithi192.stdout:2/822: rename d1/d2d/cbc to d1/d8/d3a/df9/c106 0 2024-04-12T05:09:14.248 INFO:tasks.workunit.client.0.smithi113.stdout:6/878: write d8/de/d13/d11/f43 [3530582,81732] 0 2024-04-12T05:09:14.249 INFO:tasks.workunit.client.0.smithi113.stdout:4/893: sync 2024-04-12T05:09:14.249 INFO:tasks.workunit.client.0.smithi113.stdout:5/986: sync 2024-04-12T05:09:14.249 INFO:tasks.workunit.client.1.smithi192.stdout:3/874: symlink d5/d13/d3a/d8b/l13b 0 2024-04-12T05:09:14.249 INFO:tasks.workunit.client.0.smithi113.stdout:6/879: readlink d8/de/l21 0 2024-04-12T05:09:14.251 INFO:tasks.workunit.client.0.smithi113.stdout:4/894: chown da/d24/d7e/d124/d1b 1440562 1 2024-04-12T05:09:14.252 INFO:tasks.workunit.client.0.smithi113.stdout:8/976: write d9/d1b/d8e/d93/d32/d48/ff0 [1007933,53863] 0 2024-04-12T05:09:14.253 INFO:tasks.workunit.client.1.smithi192.stdout:8/862: dwrite d6/d10/d46/d5e/d5f/f57 [0,4194304] 0 2024-04-12T05:09:14.253 INFO:tasks.workunit.client.1.smithi192.stdout:7/783: read d3/f21 [1778430,117205] 0 2024-04-12T05:09:14.254 INFO:tasks.workunit.client.1.smithi192.stdout:2/823: creat d1/d2d/f107 x:0 0 0 2024-04-12T05:09:14.256 INFO:tasks.workunit.client.1.smithi192.stdout:3/875: fsync d5/d13/d3c/d36/f49 0 2024-04-12T05:09:14.257 INFO:tasks.workunit.client.0.smithi113.stdout:2/971: rename d8/d9/de/d12/f9c to d8/d9/de/d15/d24/d58/d121/f12a 0 2024-04-12T05:09:14.257 INFO:tasks.workunit.client.0.smithi113.stdout:6/880: truncate d8/de/d13/d11/d36/d3d/d4c/d31/f62 2042751 0 2024-04-12T05:09:14.258 INFO:tasks.workunit.client.0.smithi113.stdout:4/895: creat da/d3c/d83/d47/d10f/d90/f128 x:0 0 0 2024-04-12T05:09:14.259 INFO:tasks.workunit.client.1.smithi192.stdout:2/824: dread - d1/d2d/fe3 zero size 2024-04-12T05:09:14.260 INFO:tasks.workunit.client.0.smithi113.stdout:5/987: truncate d0/d10/d27/d1d/d38/d44/fa3 4103031 0 2024-04-12T05:09:14.263 INFO:tasks.workunit.client.0.smithi113.stdout:2/972: mkdir d8/d9/de/d43/d12b 0 2024-04-12T05:09:14.265 INFO:tasks.workunit.client.1.smithi192.stdout:1/865: write d0/d2/d1e/d3f/df2/d3c/d5b/d74/d90/d108/fa6 [1496667,82163] 0 2024-04-12T05:09:14.265 INFO:tasks.workunit.client.0.smithi113.stdout:6/881: fsync d8/de/f19 0 2024-04-12T05:09:14.266 INFO:tasks.workunit.client.1.smithi192.stdout:8/863: rename d6/d10/d46/d5e/d5f/d38/ld8 to d6/d10/d46/d5e/d5f/d37/d4f/d4c/l10b 0 2024-04-12T05:09:14.267 INFO:tasks.workunit.client.1.smithi192.stdout:1/866: fsync d0/d2/d1e/d3f/df2/d3c/d5b/d74/d90/d108/f6b 0 2024-04-12T05:09:14.268 INFO:tasks.workunit.client.1.smithi192.stdout:8/864: chown d6/d10/d6c/fd4 8 1 2024-04-12T05:09:14.268 INFO:tasks.workunit.client.1.smithi192.stdout:3/876: truncate d5/d13/d3a/d4e/f102 3006352 0 2024-04-12T05:09:14.269 INFO:tasks.workunit.client.0.smithi113.stdout:4/896: creat da/d70/f129 x:0 0 0 2024-04-12T05:09:14.270 INFO:tasks.workunit.client.1.smithi192.stdout:0/938: truncate d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d46/f4d 1367004 0 2024-04-12T05:09:14.270 INFO:tasks.workunit.client.0.smithi113.stdout:5/988: chown d0/d10/d27/d1d/d41/c67 227585 1 2024-04-12T05:09:14.270 INFO:tasks.workunit.client.0.smithi113.stdout:2/973: creat d8/d9/de/d43/dd6/f12c x:0 0 0 2024-04-12T05:09:14.273 INFO:tasks.workunit.client.0.smithi113.stdout:4/897: mknod da/d24/d7b/c12a 0 2024-04-12T05:09:14.274 INFO:tasks.workunit.client.0.smithi113.stdout:4/898: chown da/d3c/da5/f10e 121301 1 2024-04-12T05:09:14.275 INFO:tasks.workunit.client.1.smithi192.stdout:1/867: creat d0/d29/dbf/f11f x:0 0 0 2024-04-12T05:09:14.276 INFO:tasks.workunit.client.1.smithi192.stdout:8/865: dwrite d6/d10/d109/f30 [0,4194304] 0 2024-04-12T05:09:14.277 INFO:tasks.workunit.client.1.smithi192.stdout:0/939: fsync d0/fa7 0 2024-04-12T05:09:14.278 INFO:tasks.workunit.client.1.smithi192.stdout:1/868: dread - d0/d1b/f104 zero size 2024-04-12T05:09:14.278 INFO:tasks.workunit.client.1.smithi192.stdout:3/877: creat d5/d13/d3a/d8b/df0/f13c x:0 0 0 2024-04-12T05:09:14.278 INFO:tasks.workunit.client.1.smithi192.stdout:7/784: link d3/d7/c79 d3/d1d/d5a/d5f/d7a/da6/c10b 0 2024-04-12T05:09:14.279 INFO:tasks.workunit.client.0.smithi113.stdout:8/977: dread d9/d1b/d8e/d93/d32/f60 [4194304,4194304] 0 2024-04-12T05:09:14.279 INFO:tasks.workunit.client.0.smithi113.stdout:4/899: unlink l4 0 2024-04-12T05:09:14.281 INFO:tasks.workunit.client.0.smithi113.stdout:8/978: chown d9/d1b/d8e/d93/f84 0 1 2024-04-12T05:09:14.283 INFO:tasks.workunit.client.1.smithi192.stdout:1/869: truncate d0/d1b/d28/d9d/fe9 805272 0 2024-04-12T05:09:14.284 INFO:tasks.workunit.client.0.smithi113.stdout:2/974: link d8/d9/f108 d8/d9/de/d12/d2b/f12d 0 2024-04-12T05:09:14.285 INFO:tasks.workunit.client.1.smithi192.stdout:7/785: write d3/d7/d15/f77 [483069,16527] 0 2024-04-12T05:09:14.286 INFO:tasks.workunit.client.0.smithi113.stdout:5/989: write d0/d10/d27/d1d/d38/fb2 [1092799,95904] 0 2024-04-12T05:09:14.287 INFO:tasks.workunit.client.0.smithi113.stdout:4/900: symlink da/l12b 0 2024-04-12T05:09:14.288 INFO:tasks.workunit.client.1.smithi192.stdout:8/866: truncate d6/f18 1778780 0 2024-04-12T05:09:14.288 INFO:tasks.workunit.client.1.smithi192.stdout:1/870: stat d0/d29/d8d 0 2024-04-12T05:09:14.290 INFO:tasks.workunit.client.1.smithi192.stdout:8/867: write d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/fcc [443540,59132] 0 2024-04-12T05:09:14.290 INFO:tasks.workunit.client.0.smithi113.stdout:5/990: read - d0/d10/d27/d1d/d38/d3b/d6b/ff1 zero size 2024-04-12T05:09:14.291 INFO:tasks.workunit.client.1.smithi192.stdout:7/786: rmdir d3/d1d/d5a/d5f/d6a 39 2024-04-12T05:09:14.292 INFO:tasks.workunit.client.1.smithi192.stdout:7/787: chown d3/d1d/d5a/d5f/fd5 263225 1 2024-04-12T05:09:14.293 INFO:tasks.workunit.client.1.smithi192.stdout:8/868: mkdir d6/d10/d46/d5e/d5f/d31/d80/d10c 0 2024-04-12T05:09:14.295 INFO:tasks.workunit.client.0.smithi113.stdout:4/901: rename da/d70 to da/d12c 0 2024-04-12T05:09:14.296 INFO:tasks.workunit.client.1.smithi192.stdout:2/825: dread d1/d8/d17/d19/f45 [0,4194304] 0 2024-04-12T05:09:14.296 INFO:tasks.workunit.client.1.smithi192.stdout:7/788: symlink d3/d7/d14/d1f/d76/l10c 0 2024-04-12T05:09:14.297 INFO:tasks.workunit.client.0.smithi113.stdout:2/975: getdents d8/d9/de/d12 0 2024-04-12T05:09:14.298 INFO:tasks.workunit.client.0.smithi113.stdout:4/902: readlink da/d3c/d83/d47/l7a 0 2024-04-12T05:09:14.298 INFO:tasks.workunit.client.1.smithi192.stdout:3/878: link d5/d13/d3a/d70/fd3 d5/d13/d3a/d8b/da2/d123/d137/f13d 0 2024-04-12T05:09:14.299 INFO:tasks.workunit.client.0.smithi113.stdout:5/991: dwrite d0/d10/d6e/fb5 [0,4194304] 0 2024-04-12T05:09:14.303 INFO:tasks.workunit.client.0.smithi113.stdout:4/903: symlink da/d3c/d83/d113/l12d 0 2024-04-12T05:09:14.303 INFO:tasks.workunit.client.0.smithi113.stdout:8/979: write d9/d16/d65/d121/d59/d68/fa8 [2464732,130955] 0 2024-04-12T05:09:14.306 INFO:tasks.workunit.client.0.smithi113.stdout:8/980: read - d9/d1b/dfd/df3/f117 zero size 2024-04-12T05:09:14.307 INFO:tasks.workunit.client.1.smithi192.stdout:1/871: rename d0/d2/d1e/d3f/d56/lae to d0/l120 0 2024-04-12T05:09:14.307 INFO:tasks.workunit.client.1.smithi192.stdout:7/789: mknod d3/d1d/d5a/d5f/d7a/da6/d105/c10d 0 2024-04-12T05:09:14.309 INFO:tasks.workunit.client.0.smithi113.stdout:2/976: truncate d8/d9/de/d15/d24/f7e 14689 0 2024-04-12T05:09:14.309 INFO:tasks.workunit.client.1.smithi192.stdout:8/869: dwrite d6/fd6 [0,4194304] 0 2024-04-12T05:09:14.310 INFO:tasks.workunit.client.0.smithi113.stdout:5/992: dwrite d0/d10/d6e/d93/dc9/d14b/d73/d89/f62 [4194304,4194304] 0 2024-04-12T05:09:14.312 INFO:tasks.workunit.client.1.smithi192.stdout:3/879: symlink d5/d13/d3a/d4a/d61/d9f/deb/l13e 0 2024-04-12T05:09:14.312 INFO:tasks.workunit.client.1.smithi192.stdout:2/826: dread d1/d2d/f34 [0,4194304] 0 2024-04-12T05:09:14.314 INFO:tasks.workunit.client.1.smithi192.stdout:7/790: dread - d3/d1d/d74/d86/d93/f106 zero size 2024-04-12T05:09:14.318 INFO:tasks.workunit.client.0.smithi113.stdout:2/977: mkdir d8/d9/de/d12/d2b/d79/ddf/d11b/d12e 0 2024-04-12T05:09:14.318 INFO:tasks.workunit.client.1.smithi192.stdout:0/940: dread d0/d7b/d9b/d35/d109/d9f/d10a/d67/f92 [0,4194304] 0 2024-04-12T05:09:14.324 INFO:tasks.workunit.client.1.smithi192.stdout:3/880: chown d5/d13/d3a/d8b/fae 279836 1 2024-04-12T05:09:14.328 INFO:tasks.workunit.client.1.smithi192.stdout:7/791: dwrite d3/d1d/d5a/d5f/fbd [0,4194304] 0 2024-04-12T05:09:14.332 INFO:tasks.workunit.client.1.smithi192.stdout:7/792: chown d3/d1d/d48/d7b/f95 54795717 1 2024-04-12T05:09:14.332 INFO:tasks.workunit.client.0.smithi113.stdout:5/993: dread d0/d10/d27/d1d/d8c/db8/f125 [0,4194304] 0 2024-04-12T05:09:14.334 INFO:tasks.workunit.client.1.smithi192.stdout:2/827: dread d1/d2d/d73/f7a [0,4194304] 0 2024-04-12T05:09:14.334 INFO:tasks.workunit.client.1.smithi192.stdout:0/941: link d0/d7b/d9b/d35/d109/db0/dbf/le2 d0/d7b/d9b/d4b/dbc/l13d 0 2024-04-12T05:09:14.336 INFO:tasks.workunit.client.1.smithi192.stdout:7/793: symlink d3/d1d/d4a/d5e/d6d/l10e 0 2024-04-12T05:09:14.337 INFO:tasks.workunit.client.0.smithi113.stdout:5/994: creat d0/d10/d27/d1d/d38/d75/f150 x:0 0 0 2024-04-12T05:09:14.339 INFO:tasks.workunit.client.1.smithi192.stdout:0/942: dread d0/d7b/d9b/d35/d109/d53/df3/d2e/d59/f84 [0,4194304] 0 2024-04-12T05:09:14.340 INFO:tasks.workunit.client.0.smithi113.stdout:2/978: dread d8/d9/d27/d88/f8b [0,4194304] 0 2024-04-12T05:09:14.342 INFO:tasks.workunit.client.0.smithi113.stdout:2/979: chown d8/d9/de/fd5 1912384 1 2024-04-12T05:09:14.345 INFO:tasks.workunit.client.1.smithi192.stdout:8/870: truncate d6/d10/d46/d5e/d5f/d31/d80/db1/fb6 3080740 0 2024-04-12T05:09:14.346 INFO:tasks.workunit.client.0.smithi113.stdout:2/980: rmdir d8/d9/de/d12/d2b/dd2 39 2024-04-12T05:09:14.346 INFO:tasks.workunit.client.0.smithi113.stdout:2/981: chown d8/f1e 178 1 2024-04-12T05:09:14.347 INFO:tasks.workunit.client.0.smithi113.stdout:8/981: dread d9/d16/d65/d121/d59/d78/f116 [0,4194304] 0 2024-04-12T05:09:14.347 INFO:tasks.workunit.client.0.smithi113.stdout:5/995: write d0/d10/d27/d1d/d38/d3b/d6b/f109 [357409,33315] 0 2024-04-12T05:09:14.347 INFO:tasks.workunit.client.0.smithi113.stdout:2/982: stat d8/d9/de/d15/c80 0 2024-04-12T05:09:14.348 INFO:tasks.workunit.client.1.smithi192.stdout:0/943: link d0/d7b/d9b/d35/d109/d9f/d10a/db2/ff8 d0/d7b/d7c/f13e 0 2024-04-12T05:09:14.349 INFO:tasks.workunit.client.0.smithi113.stdout:2/983: write d8/d9/f30 [1335069,52257] 0 2024-04-12T05:09:14.352 INFO:tasks.workunit.client.1.smithi192.stdout:1/872: dwrite d0/d2/d1e/d3f/df2/d3c/d5b/da7/fca [0,4194304] 0 2024-04-12T05:09:14.355 INFO:tasks.workunit.client.0.smithi113.stdout:5/996: creat d0/d10/d6e/d93/dc9/d14b/f151 x:0 0 0 2024-04-12T05:09:14.357 INFO:tasks.workunit.client.0.smithi113.stdout:8/982: rename d9/d1b/d8e/d93/d32/d48/d8d/f73 to d9/d16/d65/d121/d59/da3/f131 0 2024-04-12T05:09:14.357 INFO:tasks.workunit.client.1.smithi192.stdout:3/881: dwrite d5/f19 [0,4194304] 0 2024-04-12T05:09:14.360 INFO:tasks.workunit.client.0.smithi113.stdout:2/984: truncate d8/d9/de/d15/d24/f7e 823893 0 2024-04-12T05:09:14.360 INFO:tasks.workunit.client.0.smithi113.stdout:8/983: chown d9/d16/d65/d121/d59/d78/f116 104471085 1 2024-04-12T05:09:14.361 INFO:tasks.workunit.client.0.smithi113.stdout:5/997: creat d0/d127/f152 x:0 0 0 2024-04-12T05:09:14.361 INFO:tasks.workunit.client.1.smithi192.stdout:0/944: creat d0/d7b/d9b/f13f x:0 0 0 2024-04-12T05:09:14.361 INFO:tasks.workunit.client.1.smithi192.stdout:8/871: link d6/d21/l34 d6/d10/d46/d5e/d5f/d37/d4f/d4c/l10d 0 2024-04-12T05:09:14.362 INFO:tasks.workunit.client.1.smithi192.stdout:2/828: dwrite d1/d8/d3a/de/d7c/fbf [0,4194304] 0 2024-04-12T05:09:14.365 INFO:tasks.workunit.client.1.smithi192.stdout:1/873: creat d0/d2/d1e/d3f/df2/d3c/d5b/db6/deb/f121 x:0 0 0 2024-04-12T05:09:14.366 INFO:tasks.workunit.client.1.smithi192.stdout:8/872: write d6/d1f/d5d/fd3 [1817928,93361] 0 2024-04-12T05:09:14.372 INFO:tasks.workunit.client.1.smithi192.stdout:0/945: symlink d0/d7b/d9b/d4b/l140 0 2024-04-12T05:09:14.372 INFO:tasks.workunit.client.1.smithi192.stdout:1/874: dread - d0/d2/d1e/d3f/df2/d21/d62/dbc/fc7 zero size 2024-04-12T05:09:14.376 INFO:tasks.workunit.client.1.smithi192.stdout:8/873: rename d6/d10/d46/d5e/d5f/d38/d3b to d6/d1f/d10e 0 2024-04-12T05:09:14.377 INFO:tasks.workunit.client.1.smithi192.stdout:2/829: creat d1/d8/d3a/de/d7c/f108 x:0 0 0 2024-04-12T05:09:14.378 INFO:tasks.workunit.client.1.smithi192.stdout:0/946: mknod d0/d7b/d9b/d35/c141 0 2024-04-12T05:09:14.379 INFO:tasks.workunit.client.1.smithi192.stdout:8/874: mkdir d6/d10/d46/d5e/d5f/d31/d61/d9d/d10f 0 2024-04-12T05:09:14.380 INFO:tasks.workunit.client.1.smithi192.stdout:0/947: truncate d0/d7b/d9b/d4b/f133 264192 0 2024-04-12T05:09:14.380 INFO:tasks.workunit.client.1.smithi192.stdout:2/830: stat d1/d8/d3a/de/d7c/d98/fb7 0 2024-04-12T05:09:14.383 INFO:tasks.workunit.client.1.smithi192.stdout:1/875: creat d0/d2/d1e/d3f/df2/d3c/d5b/d74/d7d/f122 x:0 0 0 2024-04-12T05:09:14.384 INFO:tasks.workunit.client.1.smithi192.stdout:8/875: creat d6/d10/d46/d5e/d5f/d31/d61/d9d/db8/f110 x:0 0 0 2024-04-12T05:09:14.386 INFO:tasks.workunit.client.1.smithi192.stdout:3/882: truncate d5/d13/d3a/d8b/fba 1498744 0 2024-04-12T05:09:14.388 INFO:tasks.workunit.client.0.smithi113.stdout:5/998: truncate d0/d10/d27/d1d/d10c/de7/ff2 485840 0 2024-04-12T05:09:14.389 INFO:tasks.workunit.client.0.smithi113.stdout:7/874: sync 2024-04-12T05:09:14.389 INFO:tasks.workunit.client.0.smithi113.stdout:9/932: sync 2024-04-12T05:09:14.391 INFO:tasks.workunit.client.1.smithi192.stdout:2/831: dwrite d1/d8/d3a/f2e [0,4194304] 0 2024-04-12T05:09:14.391 INFO:tasks.workunit.client.1.smithi192.stdout:8/876: chown d6/d10/d46/d5e/d5f/d37/d4f/d4c/l10b 620790 1 2024-04-12T05:09:14.392 INFO:tasks.workunit.client.0.smithi113.stdout:5/999: unlink d0/fc 0 2024-04-12T05:09:14.392 INFO:tasks.workunit.client.0.smithi113.stdout:8/984: dwrite d9/d16/d65/f97 [0,4194304] 0 2024-04-12T05:09:14.393 INFO:tasks.workunit.client.1.smithi192.stdout:2/832: chown d1/d8/d3a/de/lf4 2093583464 1 2024-04-12T05:09:14.394 INFO:tasks.workunit.client.0.smithi113.stdout:9/933: rmdir d1/d4/d5/d17 39 2024-04-12T05:09:14.395 INFO:tasks.workunit.client.1.smithi192.stdout:3/883: mkdir d5/d13/d3a/d4a/d61/d9f/deb/dfb/d114/d13f 0 2024-04-12T05:09:14.395 INFO:tasks.workunit.client.1.smithi192.stdout:8/877: read d6/d10/fa5 [1123818,77645] 0 2024-04-12T05:09:14.395 INFO:tasks.workunit.client.0.smithi113.stdout:9/934: write d1/d27/f3c [5130531,64121] 0 2024-04-12T05:09:14.397 INFO:tasks.workunit.client.0.smithi113.stdout:7/875: mkdir d0/dc/d6/d78/d12a 0 2024-04-12T05:09:14.400 INFO:tasks.workunit.client.0.smithi113.stdout:2/985: dread d8/d9/de/d15/f9d [0,4194304] 0 2024-04-12T05:09:14.400 INFO:tasks.workunit.client.0.smithi113.stdout:7/876: creat d0/dd/d95/f12b x:0 0 0 2024-04-12T05:09:14.404 INFO:tasks.workunit.client.0.smithi113.stdout:9/935: mknod d1/df/d29/de8/d104/c13b 0 2024-04-12T05:09:14.404 INFO:tasks.workunit.client.0.smithi113.stdout:8/985: getdents d9 0 2024-04-12T05:09:14.404 INFO:tasks.workunit.client.0.smithi113.stdout:2/986: mknod d8/d9/d27/dda/c12f 0 2024-04-12T05:09:14.404 INFO:tasks.workunit.client.1.smithi192.stdout:1/876: write d0/d2/f50 [363905,58394] 0 2024-04-12T05:09:14.407 INFO:tasks.workunit.client.0.smithi113.stdout:8/986: chown d9/d16/d65/d121/l77 6 1 2024-04-12T05:09:14.408 INFO:tasks.workunit.client.0.smithi113.stdout:9/936: write d1/d4/d30/dcb/f3f [1516587,51554] 0 2024-04-12T05:09:14.410 INFO:tasks.workunit.client.0.smithi113.stdout:8/987: fsync d9/d1b/d96/fd1 0 2024-04-12T05:09:14.410 INFO:tasks.workunit.client.1.smithi192.stdout:0/948: dread d0/d7b/d7c/dcc/f85 [0,4194304] 0 2024-04-12T05:09:14.412 INFO:tasks.workunit.client.1.smithi192.stdout:2/833: rename d1/d2d/f3d to d1/d8/d3a/f109 0 2024-04-12T05:09:14.412 INFO:tasks.workunit.client.1.smithi192.stdout:3/884: write d5/d13/d1a/fa9 [854073,89754] 0 2024-04-12T05:09:14.412 INFO:tasks.workunit.client.1.smithi192.stdout:1/877: fdatasync d0/d2/d1e/d3f/df2/d3c/d5b/db6/deb/f112 0 2024-04-12T05:09:14.413 INFO:tasks.workunit.client.1.smithi192.stdout:2/834: readlink d1/d8/d3a/de/lf4 0 2024-04-12T05:09:14.414 INFO:tasks.workunit.client.0.smithi113.stdout:7/877: dwrite d0/d52/fbc [0,4194304] 0 2024-04-12T05:09:14.415 INFO:tasks.workunit.client.0.smithi113.stdout:9/937: write d1/df/d4b/d63/dad/ff2 [3954923,4772] 0 2024-04-12T05:09:14.415 INFO:tasks.workunit.client.1.smithi192.stdout:1/878: chown d0/d2/l14 396089 1 2024-04-12T05:09:14.416 INFO:tasks.workunit.client.1.smithi192.stdout:2/835: chown d1/d8/d3a/de/d7c/f7d 10096929 1 2024-04-12T05:09:14.417 INFO:tasks.workunit.client.0.smithi113.stdout:9/938: stat d1/f46 0 2024-04-12T05:09:14.420 INFO:tasks.workunit.client.0.smithi113.stdout:9/939: stat d1/df/d4b/d63/d3d 0 2024-04-12T05:09:14.421 INFO:tasks.workunit.client.0.smithi113.stdout:8/988: read d9/d1b/d8e/d93/fe6 [444081,14123] 0 2024-04-12T05:09:14.421 INFO:tasks.workunit.client.1.smithi192.stdout:3/885: getdents d5/d13/d3a/d8b/da2 0 2024-04-12T05:09:14.422 INFO:tasks.workunit.client.0.smithi113.stdout:2/987: dwrite d8/d9/d1a/f50 [0,4194304] 0 2024-04-12T05:09:14.423 INFO:tasks.workunit.client.1.smithi192.stdout:2/836: mkdir d1/d8/d17/d19/d60/d59/da3/d10a 0 2024-04-12T05:09:14.426 INFO:tasks.workunit.client.0.smithi113.stdout:2/988: truncate d8/d9/de/d43/db2/f112 257302 0 2024-04-12T05:09:14.427 INFO:tasks.workunit.client.1.smithi192.stdout:2/837: symlink d1/d8/d17/d19/d71/d7f/l10b 0 2024-04-12T05:09:14.427 INFO:tasks.workunit.client.0.smithi113.stdout:9/940: symlink d1/d4/d5/d17/d117/df1/dff/l13c 0 2024-04-12T05:09:14.429 INFO:tasks.workunit.client.1.smithi192.stdout:0/949: write d0/d7b/d9b/d35/d109/d9f/d10a/d67/f92 [3088447,107071] 0 2024-04-12T05:09:14.431 INFO:tasks.workunit.client.1.smithi192.stdout:2/838: creat d1/d2d/d73/da8/f10c x:0 0 0 2024-04-12T05:09:14.432 INFO:tasks.workunit.client.1.smithi192.stdout:0/950: symlink d0/d7b/d7c/dbb/l142 0 2024-04-12T05:09:14.434 INFO:tasks.workunit.client.0.smithi113.stdout:2/989: rename d8/d9/de/d12/d3c/fbc to d8/d9/d1a/d103/ded/f130 0 2024-04-12T05:09:14.435 INFO:tasks.workunit.client.0.smithi113.stdout:9/941: fsync d1/df/d29/de8/d104/f124 0 2024-04-12T05:09:14.437 INFO:tasks.workunit.client.0.smithi113.stdout:9/942: write d1/d4/d38/f62 [782076,46816] 0 2024-04-12T05:09:14.438 INFO:tasks.workunit.client.1.smithi192.stdout:0/951: creat d0/d7b/d9b/d35/d109/d9f/d10a/f143 x:0 0 0 2024-04-12T05:09:14.441 INFO:tasks.workunit.client.0.smithi113.stdout:9/943: write d1/d4/d5/d17/d117/d132/ffa [517777,79837] 0 2024-04-12T05:09:14.442 INFO:tasks.workunit.client.1.smithi192.stdout:2/839: write d1/d8/d3a/f95 [4849029,15667] 0 2024-04-12T05:09:14.444 INFO:tasks.workunit.client.0.smithi113.stdout:9/944: symlink d1/d4/d5/d17/l13d 0 2024-04-12T05:09:14.450 INFO:tasks.workunit.client.0.smithi113.stdout:9/945: getdents d1/df/d4b/d63/d3d 0 2024-04-12T05:09:14.450 INFO:tasks.workunit.client.1.smithi192.stdout:2/840: dwrite d1/d2d/fe3 [0,4194304] 0 2024-04-12T05:09:14.452 INFO:tasks.workunit.client.1.smithi192.stdout:3/886: dread d5/d13/f18 [0,4194304] 0 2024-04-12T05:09:14.452 INFO:tasks.workunit.client.1.smithi192.stdout:2/841: dread - d1/d8/d17/ffa zero size 2024-04-12T05:09:14.452 INFO:tasks.workunit.client.1.smithi192.stdout:0/952: dwrite d0/d7b/d9b/d35/d109/d53/f9d [0,4194304] 0 2024-04-12T05:09:14.458 INFO:tasks.workunit.client.0.smithi113.stdout:2/990: write d8/d9/de/d12/d3c/fbb [201570,78831] 0 2024-04-12T05:09:14.460 INFO:tasks.workunit.client.0.smithi113.stdout:8/989: dread d9/d16/d65/d121/f8b [0,4194304] 0 2024-04-12T05:09:14.460 INFO:tasks.workunit.client.1.smithi192.stdout:5/961: sync 2024-04-12T05:09:14.465 INFO:tasks.workunit.client.1.smithi192.stdout:3/887: unlink d5/d13/d3c/d84/cf9 0 2024-04-12T05:09:14.466 INFO:tasks.workunit.client.1.smithi192.stdout:5/962: mkdir d0/d15/d3e/dca/d147 0 2024-04-12T05:09:14.467 INFO:tasks.workunit.client.1.smithi192.stdout:0/953: mknod d0/d7b/d9b/d35/d109/d9f/d10a/de3/c144 0 2024-04-12T05:09:14.468 INFO:tasks.workunit.client.1.smithi192.stdout:3/888: rmdir d5/d13/d3a/d4e 39 2024-04-12T05:09:14.470 INFO:tasks.workunit.client.1.smithi192.stdout:3/889: mknod d5/d13/d94/c140 0 2024-04-12T05:09:14.471 INFO:tasks.workunit.client.1.smithi192.stdout:2/842: dwrite d1/d8/d3a/df9/ffb [0,4194304] 0 2024-04-12T05:09:14.474 INFO:tasks.workunit.client.1.smithi192.stdout:0/954: creat d0/d7b/d9b/d35/d109/d9f/d10a/db2/f145 x:0 0 0 2024-04-12T05:09:14.479 INFO:tasks.workunit.client.1.smithi192.stdout:2/843: unlink d1/d8/d17/d19/d60/d59/d68/c7b 0 2024-04-12T05:09:14.481 INFO:tasks.workunit.client.0.smithi113.stdout:8/990: readlink d9/d16/d7f/l2b 0 2024-04-12T05:09:14.482 INFO:tasks.workunit.client.1.smithi192.stdout:0/955: creat d0/d7b/d9b/d35/d109/d53/df3/d29/db5/f146 x:0 0 0 2024-04-12T05:09:14.483 INFO:tasks.workunit.client.1.smithi192.stdout:3/890: link d5/l37 d5/d13/d3a/d8b/da2/da7/l141 0 2024-04-12T05:09:14.485 INFO:tasks.workunit.client.1.smithi192.stdout:0/956: symlink d0/d7b/d9b/d35/d109/d9f/d10a/dc4/l147 0 2024-04-12T05:09:14.487 INFO:tasks.workunit.client.0.smithi113.stdout:2/991: dwrite d8/d9/de/d43/dee/ff8 [0,4194304] 0 2024-04-12T05:09:14.491 INFO:tasks.workunit.client.1.smithi192.stdout:2/844: dwrite d1/d2d/fd1 [4194304,4194304] 0 2024-04-12T05:09:14.491 INFO:tasks.workunit.client.0.smithi113.stdout:2/992: rmdir d8/d9/de/d12/d3c/d8a 39 2024-04-12T05:09:14.493 INFO:tasks.workunit.client.0.smithi113.stdout:2/993: write d8/d9/d27/f11a [843203,112091] 0 2024-04-12T05:09:14.496 INFO:tasks.workunit.client.0.smithi113.stdout:8/991: link d9/d16/d7f/d3a/la6 d9/d1b/d96/l132 0 2024-04-12T05:09:14.502 INFO:tasks.workunit.client.1.smithi192.stdout:0/957: readlink d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/l98 0 2024-04-12T05:09:14.502 INFO:tasks.workunit.client.1.smithi192.stdout:3/891: creat d5/d13/d3a/d8b/da2/d105/f142 x:0 0 0 2024-04-12T05:09:14.502 INFO:tasks.workunit.client.1.smithi192.stdout:2/845: dread - d1/d2d/d73/f8d zero size 2024-04-12T05:09:14.505 INFO:tasks.workunit.client.1.smithi192.stdout:3/892: creat d5/d13/d1a/d53/f143 x:0 0 0 2024-04-12T05:09:14.505 INFO:tasks.workunit.client.0.smithi113.stdout:2/994: symlink d8/d9/de/d15/l131 0 2024-04-12T05:09:14.505 INFO:tasks.workunit.client.0.smithi113.stdout:8/992: write d9/d1b/d8e/d93/f66 [4949780,72824] 0 2024-04-12T05:09:14.505 INFO:tasks.workunit.client.0.smithi113.stdout:2/995: dread - d8/f126 zero size 2024-04-12T05:09:14.505 INFO:tasks.workunit.client.0.smithi113.stdout:8/993: fsync d9/d16/d7f/d3a/d6c/f12f 0 2024-04-12T05:09:14.505 INFO:tasks.workunit.client.0.smithi113.stdout:2/996: write d8/d9/de/f31 [2996659,59284] 0 2024-04-12T05:09:14.506 INFO:tasks.workunit.client.0.smithi113.stdout:2/997: readlink d8/d9/de/d12/d2b/d79/d11e/l120 0 2024-04-12T05:09:14.506 INFO:tasks.workunit.client.1.smithi192.stdout:2/846: creat d1/d8/de2/f10d x:0 0 0 2024-04-12T05:09:14.507 INFO:tasks.workunit.client.0.smithi113.stdout:8/994: symlink d9/dd4/l133 0 2024-04-12T05:09:14.507 INFO:tasks.workunit.client.0.smithi113.stdout:2/998: readlink d8/d9/de/d43/dd6/l116 0 2024-04-12T05:09:14.508 INFO:tasks.workunit.client.1.smithi192.stdout:0/958: write d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d46/f49 [7528831,90609] 0 2024-04-12T05:09:14.509 INFO:tasks.workunit.client.0.smithi113.stdout:8/995: symlink d9/d16/d3c/l134 0 2024-04-12T05:09:14.510 INFO:tasks.workunit.client.1.smithi192.stdout:3/893: mkdir d5/d13/d3a/d8b/da2/d123/d144 0 2024-04-12T05:09:14.514 INFO:tasks.workunit.client.0.smithi113.stdout:2/999: creat d8/d9/de/d12/d2b/dd2/f132 x:0 0 0 2024-04-12T05:09:14.517 INFO:tasks.workunit.client.1.smithi192.stdout:0/959: creat d0/d7b/d9b/d35/d109/d53/df3/d29/f148 x:0 0 0 2024-04-12T05:09:14.518 INFO:tasks.workunit.client.1.smithi192.stdout:2/847: dwrite d1/d2d/d73/de7/df0/f102 [0,4194304] 0 2024-04-12T05:09:14.518 INFO:tasks.workunit.client.1.smithi192.stdout:3/894: truncate d5/d13/d3a/d8b/da2/f135 712265 0 2024-04-12T05:09:14.518 INFO:tasks.workunit.client.0.smithi113.stdout:8/996: write d9/d16/d7f/d3a/d6c/d90/ff5 [1016758,59746] 0 2024-04-12T05:09:14.519 INFO:tasks.workunit.client.1.smithi192.stdout:3/895: read d5/f19 [238288,32458] 0 2024-04-12T05:09:14.521 INFO:tasks.workunit.client.0.smithi113.stdout:9/946: dwrite d1/d4/d30/f75 [0,4194304] 0 2024-04-12T05:09:14.523 INFO:tasks.workunit.client.0.smithi113.stdout:8/997: creat d9/d16/d65/d121/d59/d68/dca/f135 x:0 0 0 2024-04-12T05:09:14.523 INFO:tasks.workunit.client.1.smithi192.stdout:2/848: mknod d1/d2d/d73/da8/c10e 0 2024-04-12T05:09:14.526 INFO:tasks.workunit.client.1.smithi192.stdout:2/849: write d1/d8/d17/d19/d60/fa6 [757040,128145] 0 2024-04-12T05:09:14.527 INFO:tasks.workunit.client.1.smithi192.stdout:3/896: unlink d5/d13/d3a/d70/d8c/d8d/c11e 0 2024-04-12T05:09:14.528 INFO:tasks.workunit.client.0.smithi113.stdout:9/947: link d1/d4/d5/d17/d117/df1/f139 d1/d4/d38/f13e 0 2024-04-12T05:09:14.532 INFO:tasks.workunit.client.1.smithi192.stdout:2/850: rmdir d1/d8/d17/d19/d60/df7 0 2024-04-12T05:09:14.536 INFO:tasks.workunit.client.0.smithi113.stdout:9/948: dwrite d1/d116/f129 [0,4194304] 0 2024-04-12T05:09:14.537 INFO:tasks.workunit.client.0.smithi113.stdout:9/949: fdatasync d1/d27/d7a/dd2/f133 0 2024-04-12T05:09:14.537 INFO:tasks.workunit.client.0.smithi113.stdout:9/950: fdatasync d1/d27/d7a/de4/db7/ffb 0 2024-04-12T05:09:14.541 INFO:tasks.workunit.client.1.smithi192.stdout:3/897: write d5/d13/f2a [1474277,55819] 0 2024-04-12T05:09:14.543 INFO:tasks.workunit.client.0.smithi113.stdout:9/951: dread d1/d4/f33 [0,4194304] 0 2024-04-12T05:09:14.546 INFO:tasks.workunit.client.1.smithi192.stdout:3/898: read - d5/d13/d94/dbd/fc4 zero size 2024-04-12T05:09:14.546 INFO:tasks.workunit.client.0.smithi113.stdout:8/998: dwrite d9/d16/d65/fa9 [0,4194304] 0 2024-04-12T05:09:14.547 INFO:tasks.workunit.client.1.smithi192.stdout:0/960: dread d0/d7b/d9b/d35/d109/d95/db4/f102 [0,4194304] 0 2024-04-12T05:09:14.548 INFO:tasks.workunit.client.0.smithi113.stdout:9/952: unlink d1/df/db6/f6b 0 2024-04-12T05:09:14.549 INFO:tasks.workunit.client.0.smithi113.stdout:9/953: readlink d1/df/db6/lc2 0 2024-04-12T05:09:14.550 INFO:tasks.workunit.client.0.smithi113.stdout:9/954: dread - d1/d4/d38/f13e zero size 2024-04-12T05:09:14.550 INFO:tasks.workunit.client.1.smithi192.stdout:3/899: mkdir d5/d13/d3a/d4a/d61/d145 0 2024-04-12T05:09:14.551 INFO:tasks.workunit.client.1.smithi192.stdout:2/851: dwrite d1/d8/f22 [8388608,4194304] 0 2024-04-12T05:09:14.552 INFO:tasks.workunit.client.1.smithi192.stdout:0/961: creat d0/d7b/d7c/dbb/f149 x:0 0 0 2024-04-12T05:09:14.555 INFO:tasks.workunit.client.0.smithi113.stdout:9/955: dread - d1/fe9 zero size 2024-04-12T05:09:14.556 INFO:tasks.workunit.client.1.smithi192.stdout:2/852: mkdir d1/d2d/d73/de7/df0/d10f 0 2024-04-12T05:09:14.557 INFO:tasks.workunit.client.0.smithi113.stdout:8/999: unlink d9/d1b/d8e/d93/d32/d5b/f83 0 2024-04-12T05:09:14.557 INFO:tasks.workunit.client.0.smithi113.stdout:9/956: stat d1/d4/df9/d109/c10a 0 2024-04-12T05:09:14.558 INFO:tasks.workunit.client.0.smithi113.stdout:9/957: chown d1/d4/d38/ld5 0 1 2024-04-12T05:09:14.559 INFO:tasks.workunit.client.1.smithi192.stdout:0/962: symlink d0/d7b/d9b/d35/d109/d9f/d10a/l14a 0 2024-04-12T05:09:14.560 INFO:tasks.workunit.client.1.smithi192.stdout:3/900: mkdir d5/d13/d3a/d4a/d61/d145/d146 0 2024-04-12T05:09:14.562 INFO:tasks.workunit.client.1.smithi192.stdout:0/963: creat d0/d7b/d9b/d4b/dbc/dfa/f14b x:0 0 0 2024-04-12T05:09:14.563 INFO:tasks.workunit.client.0.smithi113.stdout:9/958: creat d1/d4/d5/f13f x:0 0 0 2024-04-12T05:09:14.564 INFO:tasks.workunit.client.1.smithi192.stdout:2/853: getdents d1/d8/d17/d19/d60/d59/d68/dea 0 2024-04-12T05:09:14.564 INFO:tasks.workunit.client.1.smithi192.stdout:0/964: read d0/d7b/d9b/d35/d109/d53/fe7 [2984075,21382] 0 2024-04-12T05:09:14.566 INFO:tasks.workunit.client.0.smithi113.stdout:9/959: write d1/df/d29/fbc [1375825,4670] 0 2024-04-12T05:09:14.569 INFO:tasks.workunit.client.0.smithi113.stdout:3/908: sync 2024-04-12T05:09:14.574 INFO:tasks.workunit.client.0.smithi113.stdout:9/960: getdents d1/d27/d7a/d11b 0 2024-04-12T05:09:14.574 INFO:tasks.workunit.client.0.smithi113.stdout:3/909: dread d0/d13/d15/d28/d2a/d34/d5f/f96 [0,4194304] 0 2024-04-12T05:09:14.575 INFO:tasks.workunit.client.1.smithi192.stdout:3/901: dwrite d5/d13/d3a/d4a/d61/d79/f80 [0,4194304] 0 2024-04-12T05:09:14.579 INFO:tasks.workunit.client.0.smithi113.stdout:9/961: mkdir d1/d4/d5/d17/d117/d140 0 2024-04-12T05:09:14.581 INFO:tasks.workunit.client.1.smithi192.stdout:0/965: link d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d46/f49 d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/d46/f14c 0 2024-04-12T05:09:14.581 INFO:tasks.workunit.client.1.smithi192.stdout:3/902: fsync d5/d13/d3a/d70/d8c/de1/fe0 0 2024-04-12T05:09:14.581 INFO:tasks.workunit.client.1.smithi192.stdout:2/854: dwrite d1/d8/d17/d19/f31 [0,4194304] 0 2024-04-12T05:09:14.582 INFO:tasks.workunit.client.0.smithi113.stdout:9/962: mkdir d1/d116/d141 0 2024-04-12T05:09:14.583 INFO:tasks.workunit.client.0.smithi113.stdout:3/910: getdents d0/d13/d47 0 2024-04-12T05:09:14.588 INFO:tasks.workunit.client.1.smithi192.stdout:2/855: mknod d1/d8/d17/d19/d60/d59/c110 0 2024-04-12T05:09:14.589 INFO:tasks.workunit.client.0.smithi113.stdout:3/911: creat d0/d13/d15/d28/d2a/d34/d5f/d7b/de5/f12c x:0 0 0 2024-04-12T05:09:14.590 INFO:tasks.workunit.client.1.smithi192.stdout:0/966: truncate d0/d7b/d9b/d35/d109/d53/df3/d2e/d66/fd7 477280 0 2024-04-12T05:09:14.591 INFO:tasks.workunit.client.1.smithi192.stdout:3/903: dread d5/d13/d3c/d26/f9e [0,4194304] 0 2024-04-12T05:09:14.591 INFO:tasks.workunit.client.0.smithi113.stdout:3/912: mknod d0/d13/c12d 0 2024-04-12T05:09:14.592 INFO:tasks.workunit.client.1.smithi192.stdout:2/856: rename d1/d2d/d73/da8/db2/ceb to d1/da7/c111 0 2024-04-12T05:09:14.593 INFO:tasks.workunit.client.0.smithi113.stdout:3/913: write d0/d13/d15/d28/d2a/d34/d5f/fdb [583261,90371] 0 2024-04-12T05:09:14.593 INFO:tasks.workunit.client.0.smithi113.stdout:9/963: rename d1/d116 to d1/d4/d5/d17/d117/df1/dff/d142 0 2024-04-12T05:09:14.595 INFO:tasks.workunit.client.0.smithi113.stdout:9/964: stat d1/f79 0 2024-04-12T05:09:14.595 INFO:tasks.workunit.client.1.smithi192.stdout:3/904: symlink d5/d13/d1a/d53/dbb/d126/l147 0 2024-04-12T05:09:14.596 INFO:tasks.workunit.client.0.smithi113.stdout:7/878: sync 2024-04-12T05:09:14.596 INFO:tasks.workunit.client.0.smithi113.stdout:6/882: sync 2024-04-12T05:09:14.596 INFO:tasks.workunit.client.0.smithi113.stdout:4/904: sync 2024-04-12T05:09:14.597 INFO:tasks.workunit.client.0.smithi113.stdout:3/914: mkdir d0/d13/d47/d4f/dca/d12e 0 2024-04-12T05:09:14.597 INFO:tasks.workunit.client.0.smithi113.stdout:4/905: chown da/d24/f53 0 1 2024-04-12T05:09:14.598 INFO:tasks.workunit.client.0.smithi113.stdout:7/879: dread - d0/dc/d50/f102 zero size 2024-04-12T05:09:14.598 INFO:tasks.workunit.client.1.smithi192.stdout:2/857: fdatasync d1/d8/d3a/fae 0 2024-04-12T05:09:14.599 INFO:tasks.workunit.client.0.smithi113.stdout:3/915: fsync d0/f18 0 2024-04-12T05:09:14.600 INFO:tasks.workunit.client.1.smithi192.stdout:0/967: dwrite d0/d7b/d9b/d4b/dbc/dfa/f14b [0,4194304] 0 2024-04-12T05:09:14.600 INFO:tasks.workunit.client.0.smithi113.stdout:4/906: dread - da/d5a/fab zero size 2024-04-12T05:09:14.600 INFO:tasks.workunit.client.0.smithi113.stdout:6/883: write d8/de/d13/d11/d36/f35 [2615760,83844] 0 2024-04-12T05:09:14.601 INFO:tasks.workunit.client.0.smithi113.stdout:3/916: chown d0/d13/d15 4187240 1 2024-04-12T05:09:14.601 INFO:tasks.workunit.client.1.smithi192.stdout:2/858: dread - d1/d2d/d73/de7/fec zero size 2024-04-12T05:09:14.602 INFO:tasks.workunit.client.0.smithi113.stdout:7/880: symlink d0/l12c 0 2024-04-12T05:09:14.605 INFO:tasks.workunit.client.1.smithi192.stdout:3/905: write d5/d13/d3a/d4a/f5a [2851253,40231] 0 2024-04-12T05:09:14.606 INFO:tasks.workunit.client.0.smithi113.stdout:9/965: dwrite d1/d4/d38/f8c [0,4194304] 0 2024-04-12T05:09:14.606 INFO:tasks.workunit.client.0.smithi113.stdout:6/884: write d8/de/d17/f59 [4061336,94873] 0 2024-04-12T05:09:14.611 INFO:tasks.workunit.client.0.smithi113.stdout:4/907: write da/d24/d7e/d124/d3a/f88 [474088,215] 0 2024-04-12T05:09:14.613 INFO:tasks.workunit.client.1.smithi192.stdout:2/859: creat d1/f112 x:0 0 0 2024-04-12T05:09:14.614 INFO:tasks.workunit.client.1.smithi192.stdout:0/968: dread d0/d7b/d9b/d35/d109/d9f/d10a/d15/f3d [0,4194304] 0 2024-04-12T05:09:14.617 INFO:tasks.workunit.client.0.smithi113.stdout:3/917: rename d0/d13/d15/fb4 to d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/d12b/f12f 0 2024-04-12T05:09:14.617 INFO:tasks.workunit.client.1.smithi192.stdout:3/906: getdents d5/d13/d3a 0 2024-04-12T05:09:14.618 INFO:tasks.workunit.client.0.smithi113.stdout:9/966: rename d1/f26 to d1/d27/d7a/d11b/f143 0 2024-04-12T05:09:14.623 INFO:tasks.workunit.client.1.smithi192.stdout:3/907: fdatasync d5/d13/d94/fd0 0 2024-04-12T05:09:14.624 INFO:tasks.workunit.client.0.smithi113.stdout:4/908: fdatasync da/d24/d7e/faf 0 2024-04-12T05:09:14.624 INFO:tasks.workunit.client.1.smithi192.stdout:3/908: chown d5/d13/d1a/d5f/d103/ffc 318952 1 2024-04-12T05:09:14.624 INFO:tasks.workunit.client.1.smithi192.stdout:0/969: dwrite d0/d7b/d9b/d35/d109/d9f/d10a/d15/d19/f36 [4194304,4194304] 0 2024-04-12T05:09:14.628 INFO:tasks.workunit.client.0.smithi113.stdout:3/918: link d0/d7e/da3/lf3 d0/d7e/l130 0 2024-04-12T05:09:14.629 INFO:tasks.workunit.client.1.smithi192.stdout:3/909: unlink d5/d13/d3c/d84/ce8 0 2024-04-12T05:09:14.629 INFO:tasks.workunit.client.1.smithi192.stdout:2/860: dwrite d1/f70 [0,4194304] 0 2024-04-12T05:09:14.631 INFO:tasks.workunit.client.0.smithi113.stdout:4/909: write da/d24/d7e/d124/d3a/fda [750845,69066] 0 2024-04-12T05:09:14.634 INFO:tasks.workunit.client.1.smithi192.stdout:3/910: fsync d5/d13/d3a/d8b/da2/da7/ff2 0 2024-04-12T05:09:14.634 INFO:tasks.workunit.client.0.smithi113.stdout:3/919: dread - d0/d13/d47/d4f/dca/fce zero size 2024-04-12T05:09:14.634 INFO:tasks.workunit.client.0.smithi113.stdout:6/885: dwrite d8/f40 [0,4194304] 0 2024-04-12T05:09:14.635 INFO:tasks.workunit.client.0.smithi113.stdout:7/881: getdents d0/dc/da9/d5c/d9a 0 2024-04-12T05:09:14.635 INFO:tasks.workunit.client.1.smithi192.stdout:2/861: chown d1/d8/d17/d19/c61 5086983 1 2024-04-12T05:09:14.636 INFO:tasks.workunit.client.0.smithi113.stdout:4/910: mkdir da/d2b/d12e 0 2024-04-12T05:09:14.639 INFO:tasks.workunit.client.0.smithi113.stdout:3/920: creat d0/d80/f131 x:0 0 0 2024-04-12T05:09:14.640 INFO:tasks.workunit.client.0.smithi113.stdout:9/967: dread d1/d4/d5/f58 [0,4194304] 0 2024-04-12T05:09:14.642 INFO:tasks.workunit.client.0.smithi113.stdout:9/968: chown d1/df/d4b/df7 3126987 1 2024-04-12T05:09:14.643 INFO:tasks.workunit.client.0.smithi113.stdout:4/911: creat da/d3c/d9a/dc1/d106/dc8/f12f x:0 0 0 2024-04-12T05:09:14.644 INFO:tasks.workunit.client.1.smithi192.stdout:2/862: mknod d1/d8/d3a/de/d7c/d98/c113 0 2024-04-12T05:09:14.644 INFO:tasks.workunit.client.1.smithi192.stdout:0/970: dwrite d0/d7b/d9b/d35/d109/d53/df3/d29/fdd [0,4194304] 0 2024-04-12T05:09:14.646 INFO:tasks.workunit.client.1.smithi192.stdout:3/911: write d5/d13/d3a/d4a/d61/ffe [624880,87132] 0 2024-04-12T05:09:14.646 INFO:tasks.workunit.client.0.smithi113.stdout:4/912: chown da/d3c/d9a/ff8 73 1 2024-04-12T05:09:14.647 INFO:tasks.workunit.client.0.smithi113.stdout:9/969: creat d1/d4/d38/d65/dcd/deb/f144 x:0 0 0 2024-04-12T05:09:14.648 INFO:tasks.workunit.client.1.smithi192.stdout:3/912: readlink d5/d13/d1a/d53/dbb/l130 0 2024-04-12T05:09:14.649 INFO:tasks.workunit.client.0.smithi113.stdout:4/913: fdatasync da/d24/d60/db4/fc7 0 2024-04-12T05:09:14.650 INFO:tasks.workunit.client.1.smithi192.stdout:3/913: write d5/d13/d94/fda [569772,48847] 0 2024-04-12T05:09:14.651 INFO:tasks.workunit.client.0.smithi113.stdout:3/921: dwrite d0/d13/d47/f89 [0,4194304] 0 2024-04-12T05:09:14.651 INFO:tasks.workunit.client.1.smithi192.stdout:2/863: dread - d1/d8/d17/db3/fc8 zero size 2024-04-12T05:09:14.653 INFO:tasks.workunit.client.0.smithi113.stdout:9/970: mkdir d1/d4/d5/d17/d117/df1/dff/d145 0 2024-04-12T05:09:14.653 INFO:tasks.workunit.client.1.smithi192.stdout:3/914: fsync d5/d13/d94/fc6 0 2024-04-12T05:09:14.655 INFO:tasks.workunit.client.0.smithi113.stdout:4/914: stat da/d24/d7e/d79/dcd/ce5 0 2024-04-12T05:09:14.657 INFO:tasks.workunit.client.0.smithi113.stdout:9/971: rmdir d1/df/d4b/d63 39 2024-04-12T05:09:14.659 INFO:tasks.workunit.client.0.smithi113.stdout:9/972: truncate d1/d27/d7a/dd2/fdc 363795 0 2024-04-12T05:09:14.659 INFO:tasks.workunit.client.0.smithi113.stdout:3/922: mkdir d0/d13/d15/d28/d2a/d8f/d132 0 2024-04-12T05:09:14.661 INFO:tasks.workunit.client.0.smithi113.stdout:4/915: dread - da/d3c/da5/dad/de1/ffe zero size 2024-04-12T05:09:14.664 INFO:tasks.workunit.client.1.smithi192.stdout:0/971: write d0/d7b/d9b/f100 [993174,75891] 0 2024-04-12T05:09:14.664 INFO:tasks.workunit.client.1.smithi192.stdout:2/864: link d1/d2d/d73/c75 d1/d8/d3a/de/d7c/c114 0 2024-04-12T05:09:14.665 INFO:tasks.workunit.client.0.smithi113.stdout:9/973: chown d1/df/d4b/d63/dad/ff2 10751203 1 2024-04-12T05:09:14.665 INFO:tasks.workunit.client.0.smithi113.stdout:9/974: chown d1/d27/d7a/d11b 37273012 1 2024-04-12T05:09:14.665 INFO:tasks.workunit.client.1.smithi192.stdout:2/865: readlink d1/d8/d3a/de/d7c/d98/la5 0 2024-04-12T05:09:14.666 INFO:tasks.workunit.client.0.smithi113.stdout:4/916: truncate da/d3c/da5/fec 622428 0 2024-04-12T05:09:14.668 INFO:tasks.workunit.client.0.smithi113.stdout:6/886: write d8/de/d13/fa [3397510,18938] 0 2024-04-12T05:09:14.671 INFO:tasks.workunit.client.1.smithi192.stdout:0/972: creat d0/d7b/d9b/d35/d109/d53/df3/d29/db5/d11e/f14d x:0 0 0 2024-04-12T05:09:14.671 INFO:tasks.workunit.client.0.smithi113.stdout:4/917: mknod da/d24/c130 0 2024-04-12T05:09:14.674 INFO:tasks.workunit.client.0.smithi113.stdout:6/887: unlink d8/de/d13/d11/d36/d53/db7/fba 0 2024-04-12T05:09:14.675 INFO:tasks.workunit.client.0.smithi113.stdout:7/882: dwrite d0/dc/d10b/ff7 [0,4194304] 0 2024-04-12T05:09:14.676 INFO:tasks.workunit.client.1.smithi192.stdout:3/915: dwrite d5/fd8 [0,4194304] 0 2024-04-12T05:09:14.676 INFO:tasks.workunit.client.0.smithi113.stdout:7/883: chown d0/dc/d6/d9b/fcb 2417902 1 2024-04-12T05:09:14.677 INFO:tasks.workunit.client.0.smithi113.stdout:9/975: link d1/df/f4d d1/d4/d5/d17/d117/df1/dff/d10f/f146 0 2024-04-12T05:09:14.677 INFO:tasks.workunit.client.0.smithi113.stdout:4/918: creat da/d3c/d9a/dc1/dd9/f131 x:0 0 0 2024-04-12T05:09:14.678 INFO:tasks.workunit.client.1.smithi192.stdout:0/973: rmdir d0/d7b/d9b/d4b/d72/d127 0 2024-04-12T05:09:14.678 INFO:tasks.workunit.client.1.smithi192.stdout:6/895: sync 2024-04-12T05:09:14.679 INFO:tasks.workunit.client.1.smithi192.stdout:9/726: sync 2024-04-12T05:09:14.679 INFO:tasks.workunit.client.1.smithi192.stdout:4/895: sync 2024-04-12T05:09:14.679 INFO:tasks.workunit.client.1.smithi192.stdout:0/974: chown d0/d7b/d9b/d35/d109/d9f/d10a/d67/db1/dd6/df2/d116 156226 1 2024-04-12T05:09:14.680 INFO:tasks.workunit.client.0.smithi113.stdout:6/888: symlink d8/de/d13/d11/d36/l13c 0 2024-04-12T05:09:14.681 INFO:tasks.workunit.client.1.smithi192.stdout:3/916: creat d5/d13/d3a/d4e/f148 x:0 0 0 2024-04-12T05:09:14.683 INFO:tasks.workunit.client.1.smithi192.stdout:2/866: write d1/d8/d17/f5f [1027513,99179] 0 2024-04-12T05:09:14.688 INFO:tasks.workunit.client.0.smithi113.stdout:6/889: write d8/de/d13/d11/f43 [712212,100491] 0 2024-04-12T05:09:14.688 INFO:tasks.workunit.client.1.smithi192.stdout:3/917: mknod d5/d13/d3a/d4a/d61/d9f/deb/dfb/d114/c149 0 2024-04-12T05:09:14.690 INFO:tasks.workunit.client.0.smithi113.stdout:7/884: rename d0/dc/d6 to d0/dc/d50/d123/d12d 0 2024-04-12T05:09:14.690 INFO:tasks.workunit.client.1.smithi192.stdout:2/867: chown d1/d8/d3a/de/d7c/d98/ccf 70160 1 2024-04-12T05:09:14.691 INFO:tasks.workunit.client.1.smithi192.stdout:9/727: link d11/f12 d11/d25/d36/d6e/d33/dbb/de7/f10b 0 2024-04-12T05:09:14.692 INFO:tasks.workunit.client.1.smithi192.stdout:4/896: rmdir d1/d1d/d53/d1c/d95/dc6/dd5/de5/ded/d119 39 2024-04-12T05:09:14.693 INFO:tasks.workunit.client.0.smithi113.stdout:6/890: readlink d8/de/d13/d11/d36/l13c 0 2024-04-12T05:09:14.694 INFO:tasks.workunit.client.1.smithi192.stdout:6/896: creat d3/d8/d56/d59/d97/f127 x:0 0 0 2024-04-12T05:09:14.694 INFO:tasks.workunit.client.0.smithi113.stdout:9/976: dwrite d1/d4/d5/f13f [0,4194304] 0 2024-04-12T05:09:14.694 INFO:tasks.workunit.client.1.smithi192.stdout:3/918: read - d5/d13/d3a/d70/fd2 zero size 2024-04-12T05:09:14.697 INFO:tasks.workunit.client.1.smithi192.stdout:9/728: mkdir d11/d25/dd3/dd5/d10c 0 2024-04-12T05:09:14.697 INFO:tasks.workunit.client.1.smithi192.stdout:2/868: write d1/d8/d17/d19/d60/f76 [5942551,128064] 0 2024-04-12T05:09:14.700 INFO:tasks.workunit.client.1.smithi192.stdout:6/897: unlink d3/d8/d10/d4c/f62 0 2024-04-12T05:09:14.701 INFO:tasks.workunit.client.0.smithi113.stdout:3/923: dread d0/d13/f42 [0,4194304] 0 2024-04-12T05:09:14.701 INFO:tasks.workunit.client.0.smithi113.stdout:9/977: creat d1/d4/d5/d17/d117/f147 x:0 0 0 2024-04-12T05:09:14.702 INFO:tasks.workunit.client.0.smithi113.stdout:3/924: chown d0/f1c 105458 1 2024-04-12T05:09:14.702 INFO:tasks.workunit.client.1.smithi192.stdout:3/919: mkdir d5/d13/d3a/d4e/d57/d14a 0 2024-04-12T05:09:14.702 INFO:tasks.workunit.client.1.smithi192.stdout:2/869: mkdir d1/d2d/d73/da8/db2/d115 0 2024-04-12T05:09:14.703 INFO:tasks.workunit.client.0.smithi113.stdout:7/885: truncate d0/f48 598137 0 2024-04-12T05:09:14.705 INFO:tasks.workunit.client.0.smithi113.stdout:3/925: write d0/d13/d15/d28/ffe [843890,9590] 0 2024-04-12T05:09:14.706 INFO:tasks.workunit.client.1.smithi192.stdout:6/898: unlink d3/d8/d56/d59/d97/f127 0 2024-04-12T05:09:14.706 INFO:tasks.workunit.client.0.smithi113.stdout:9/978: unlink d1/f46 0 2024-04-12T05:09:14.707 INFO:tasks.workunit.client.0.smithi113.stdout:6/891: truncate d8/de/d13/d11/d36/d3d/d4c/d26/f114 2627624 0 2024-04-12T05:09:14.708 INFO:tasks.workunit.client.0.smithi113.stdout:6/892: chown d8/d1d/fcc 11141237 1 2024-04-12T05:09:14.709 INFO:tasks.workunit.client.1.smithi192.stdout:3/920: creat d5/d13/d1a/d53/dbb/d126/f14b x:0 0 0 2024-04-12T05:09:14.713 INFO:tasks.workunit.client.1.smithi192.stdout:2/870: read d1/d8/d3a/de/f30 [576481,1111] 0 2024-04-12T05:09:14.714 INFO:tasks.workunit.client.1.smithi192.stdout:4/897: write d1/d54/d82/d71/d9e/d10b/fd7 [4076977,65157] 0 2024-04-12T05:09:14.717 INFO:tasks.workunit.client.0.smithi113.stdout:7/886: truncate d0/d52/f117 1680061 0 2024-04-12T05:09:14.717 INFO:tasks.workunit.client.1.smithi192.stdout:6/899: dwrite d3/d8/d56/d59/d97/f100 [0,4194304] 0 2024-04-12T05:09:14.719 INFO:tasks.workunit.client.0.smithi113.stdout:9/979: creat d1/df/db6/d101/f148 x:0 0 0 2024-04-12T05:09:14.719 INFO:tasks.workunit.client.0.smithi113.stdout:4/919: dwrite da/d3c/d9a/dc1/dd9/ff7 [0,4194304] 0 2024-04-12T05:09:14.720 INFO:tasks.workunit.client.1.smithi192.stdout:2/871: fsync d1/d8/d17/d19/d60/d9c/fc7 0 2024-04-12T05:09:14.722 INFO:tasks.workunit.client.0.smithi113.stdout:3/926: creat d0/d13/d15/d28/f133 x:0 0 0 2024-04-12T05:09:14.723 INFO:tasks.workunit.client.1.smithi192.stdout:4/898: symlink d1/d1d/d53/d11a/ddb/ddc/l135 0 2024-04-12T05:09:14.723 INFO:tasks.workunit.client.0.smithi113.stdout:4/920: readlink da/d3c/da5/dad/lf9 0 2024-04-12T05:09:14.723 INFO:tasks.workunit.client.1.smithi192.stdout:3/921: dread d5/d13/d3c/d26/f9e [0,4194304] 0 2024-04-12T05:09:14.724 INFO:tasks.workunit.client.0.smithi113.stdout:4/921: write da/d12c/f129 [245656,64092] 0 2024-04-12T05:09:14.725 INFO:tasks.workunit.client.0.smithi113.stdout:9/980: symlink d1/d4/d6d/l149 0 2024-04-12T05:09:14.728 INFO:tasks.workunit.client.0.smithi113.stdout:7/887: truncate d0/dc/d50/f72 1249936 0 2024-04-12T05:09:14.728 INFO:tasks.workunit.client.0.smithi113.stdout:3/927: symlink d0/d13/d15/d28/dc0/l134 0 2024-04-12T05:09:14.730 INFO:tasks.workunit.client.0.smithi113.stdout:7/888: chown d0/dc/d50/d123/d12d/d9b/f84 6 1 2024-04-12T05:09:14.732 INFO:tasks.workunit.client.0.smithi113.stdout:3/928: chown d0/d13/f1f 0 1 2024-04-12T05:09:14.735 INFO:tasks.workunit.client.0.smithi113.stdout:7/889: chown d0/dc/da9/d5c/d9a/d9e/fbd 1933288 1 2024-04-12T05:09:14.738 INFO:tasks.workunit.client.1.smithi192.stdout:9/729: dread d11/d15/d21/fb0 [4194304,4194304] 0 2024-04-12T05:09:14.739 INFO:tasks.workunit.client.0.smithi113.stdout:4/922: dwrite da/d73/f1c [0,4194304] 0 2024-04-12T05:09:14.739 INFO:tasks.workunit.client.0.smithi113.stdout:9/981: link d1/df/db6/c50 d1/df/d29/de8/dc7/c14a 0 2024-04-12T05:09:14.744 INFO:tasks.workunit.client.0.smithi113.stdout:6/893: dread d8/de/f94 [0,4194304] 0 2024-04-12T05:09:14.745 INFO:tasks.workunit.client.1.smithi192.stdout:2/872: dwrite d1/d8/d3a/df9/ffb [0,4194304] 0 2024-04-12T05:09:14.746 INFO:tasks.workunit.client.0.smithi113.stdout:7/890: symlink d0/dc/d50/d123/d12d/d78/df2/d129/l12e 0 2024-04-12T05:09:14.746 INFO:tasks.workunit.client.0.smithi113.stdout:3/929: dwrite d0/d13/d47/d94/dac/f10e [0,4194304] 0 2024-04-12T05:09:14.751 INFO:tasks.workunit.client.1.smithi192.stdout:6/900: dwrite d3/dc/d39/fc9 [0,4194304] 0 2024-04-12T05:09:14.752 INFO:tasks.workunit.client.0.smithi113.stdout:9/982: dwrite d1/df/d4b/f7f [0,4194304] 0 2024-04-12T05:09:14.755 INFO:tasks.workunit.client.0.smithi113.stdout:9/983: fdatasync d1/d4/d38/d65/dcd/dd7/f12a 0 2024-04-12T05:09:14.757 INFO:tasks.workunit.client.0.smithi113.stdout:6/894: creat d8/de/d13/df0/d66/dc4/f13d x:0 0 0 2024-04-12T05:09:14.762 INFO:tasks.workunit.client.0.smithi113.stdout:7/891: rename d0/dc/l63 to d0/dc/da9/d5c/l12f 0 2024-04-12T05:09:14.770 INFO:tasks.workunit.client.0.smithi113.stdout:3/930: creat d0/d13/d15/d28/d2a/d34/d5f/d7b/f135 x:0 0 0 2024-04-12T05:09:14.770 INFO:tasks.workunit.client.1.smithi192.stdout:4/899: fsync d1/f33 0 2024-04-12T05:09:14.772 INFO:tasks.workunit.client.1.smithi192.stdout:3/922: mknod d5/d13/d3a/d4e/c14c 0 2024-04-12T05:09:14.772 INFO:tasks.workunit.client.1.smithi192.stdout:4/900: dread - d1/d1d/d53/d1c/d95/dc6/dd5/de5/f12c zero size 2024-04-12T05:09:14.773 INFO:tasks.workunit.client.1.smithi192.stdout:9/730: unlink d11/d25/dec/d1f/d50/cb1 0 2024-04-12T05:09:14.774 INFO:tasks.workunit.client.0.smithi113.stdout:7/892: dwrite d0/dc/d50/d123/d12d/d78/f6a [0,4194304] 0 2024-04-12T05:09:14.775 INFO:tasks.workunit.client.1.smithi192.stdout:9/731: chown d11/d25/d36/d6e/d33/df3/ffa 382961 1 2024-04-12T05:09:14.776 INFO:tasks.workunit.client.1.smithi192.stdout:6/901: rmdir d3/d8/de6 39 2024-04-12T05:09:14.776 INFO:tasks.workunit.client.0.smithi113.stdout:3/931: creat d0/d13/d15/d28/d2a/d8f/d132/f136 x:0 0 0 2024-04-12T05:09:14.779 INFO:tasks.workunit.client.1.smithi192.stdout:3/923: creat d5/df1/f14d x:0 0 0 2024-04-12T05:09:14.781 INFO:tasks.workunit.client.1.smithi192.stdout:2/873: unlink d1/d2d/fcb 0 2024-04-12T05:09:14.784 INFO:tasks.workunit.client.1.smithi192.stdout:4/901: creat d1/d1d/d53/d1c/d111/f136 x:0 0 0 2024-04-12T05:09:14.786 INFO:tasks.workunit.client.1.smithi192.stdout:6/902: mknod d3/d8/d56/d59/c128 0 2024-04-12T05:09:14.787 INFO:tasks.workunit.client.1.smithi192.stdout:2/874: unlink d1/d8/d17/d19/d40/f5b 0 2024-04-12T05:09:14.789 INFO:tasks.workunit.client.1.smithi192.stdout:3/924: creat d5/d13/d3a/d4a/d61/d79/d11c/f14e x:0 0 0 2024-04-12T05:09:14.790 INFO:tasks.workunit.client.1.smithi192.stdout:6/903: fdatasync d3/d8/d56/d59/d8a/d99/fb3 0 2024-04-12T05:09:14.792 INFO:tasks.workunit.client.0.smithi113.stdout:4/923: write da/d24/d7e/d124/f84 [2862688,69787] 0 2024-04-12T05:09:14.792 INFO:tasks.workunit.client.0.smithi113.stdout:9/984: dread d1/d4/d5/d17/f74 [0,4194304] 0 2024-04-12T05:09:14.793 INFO:tasks.workunit.client.1.smithi192.stdout:9/732: getdents d11/d25/d36/d3d/d7a/df9 0 2024-04-12T05:09:14.793 INFO:tasks.workunit.client.1.smithi192.stdout:2/875: mkdir d1/d8/d17/db3/d116 0 2024-04-12T05:09:14.793 INFO:tasks.workunit.client.0.smithi113.stdout:9/985: fsync d1/d4/d5/d22/f135 0 2024-04-12T05:09:14.795 INFO:tasks.workunit.client.0.smithi113.stdout:4/924: mkdir da/d24/d7e/d124/d1b/d132 0 2024-04-12T05:09:14.797 INFO:tasks.workunit.client.1.smithi192.stdout:4/902: dwrite d1/d1d/d53/f89 [0,4194304] 0 2024-04-12T05:09:14.797 INFO:tasks.workunit.client.1.smithi192.stdout:3/925: creat d5/d13/d3a/d8b/da2/d123/d144/f14f x:0 0 0 2024-04-12T05:09:14.797 INFO:tasks.workunit.client.0.smithi113.stdout:4/925: readlink da/d24/d7e/d79/dcd/ld6 0 2024-04-12T05:09:14.799 INFO:tasks.workunit.client.1.smithi192.stdout:6/904: symlink d3/d8/d56/d59/d97/dbf/deb/l129 0 2024-04-12T05:09:14.801 INFO:tasks.workunit.client.0.smithi113.stdout:4/926: mkdir da/d3c/d133 0 2024-04-12T05:09:14.802 INFO:tasks.workunit.client.1.smithi192.stdout:3/926: write d5/d13/d3a/d4a/d61/ffe [1714994,103178] 0 2024-04-12T05:09:14.805 INFO:tasks.workunit.client.0.smithi113.stdout:4/927: write da/d24/d7e/d124/f6e [125247,60337] 0 2024-04-12T05:09:14.808 INFO:tasks.workunit.client.0.smithi113.stdout:6/895: fsync d8/de/d13/d11/d36/d3d/d4c/d26/f114 0 2024-04-12T05:09:14.810 INFO:tasks.workunit.client.1.smithi192.stdout:3/927: creat d5/d13/d3a/d70/d8c/d8d/f150 x:0 0 0 2024-04-12T05:09:14.811 INFO:tasks.workunit.client.0.smithi113.stdout:3/932: write d0/d13/d47/d4f/dca/fce [473156,82646] 0 2024-04-12T05:09:14.814 INFO:tasks.workunit.client.0.smithi113.stdout:9/986: write d1/d4/d38/f8b [2164412,96624] 0 2024-04-12T05:09:14.815 INFO:tasks.workunit.client.1.smithi192.stdout:3/928: rename d5/d13/d3c/d26/f9e to d5/d13/d94/f151 0 2024-04-12T05:09:14.816 INFO:tasks.workunit.client.0.smithi113.stdout:3/933: stat d0/d13/d15/d28/d2a/d8f/l103 0 2024-04-12T05:09:14.821 INFO:tasks.workunit.client.0.smithi113.stdout:7/893: dwrite d0/dc/ff6 [0,4194304] 0 2024-04-12T05:09:14.823 INFO:tasks.workunit.client.0.smithi113.stdout:3/934: rmdir d0/d13 39 2024-04-12T05:09:14.825 INFO:tasks.workunit.client.0.smithi113.stdout:9/987: dwrite d1/df/db6/f12c [0,4194304] 0 2024-04-12T05:09:14.825 INFO:tasks.workunit.client.0.smithi113.stdout:6/896: write d8/de/d13/d11/d36/f4e [1624604,24119] 0 2024-04-12T05:09:14.828 INFO:tasks.workunit.client.1.smithi192.stdout:6/905: dread d3/d8/f1f [4194304,4194304] 0 2024-04-12T05:09:14.828 INFO:tasks.workunit.client.0.smithi113.stdout:7/894: chown d0/dc/d50/d123/d12d/d1a/f34 3022654 1 2024-04-12T05:09:14.830 INFO:tasks.workunit.client.0.smithi113.stdout:7/895: stat d0/dc/d50/d123/d12d/fd0 0 2024-04-12T05:09:14.830 INFO:tasks.workunit.client.1.smithi192.stdout:9/733: write d11/d15/f5c [821143,77670] 0 2024-04-12T05:09:14.833 INFO:tasks.workunit.client.1.smithi192.stdout:4/903: write d1/d1d/d53/d1c/fb1 [93654,72427] 0 2024-04-12T05:09:14.833 INFO:tasks.workunit.client.0.smithi113.stdout:6/897: mkdir d8/d1d/d13e 0 2024-04-12T05:09:14.835 INFO:tasks.workunit.client.1.smithi192.stdout:3/929: write d5/f7a [6223092,17434] 0 2024-04-12T05:09:14.836 INFO:tasks.workunit.client.0.smithi113.stdout:3/935: mkdir d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/d106/d137 0 2024-04-12T05:09:14.838 INFO:tasks.workunit.client.0.smithi113.stdout:3/936: fsync d0/d7e/f85 0 2024-04-12T05:09:14.839 INFO:tasks.workunit.client.0.smithi113.stdout:4/928: dread da/d24/d7e/d124/d3a/fa3 [0,4194304] 0 2024-04-12T05:09:14.841 INFO:tasks.workunit.client.0.smithi113.stdout:9/988: dwrite d1/f78 [0,4194304] 0 2024-04-12T05:09:14.843 INFO:tasks.workunit.client.0.smithi113.stdout:4/929: fsync da/d24/d7e/d124/f84 0 2024-04-12T05:09:14.843 INFO:tasks.workunit.client.1.smithi192.stdout:3/930: mknod d5/d13/d1a/d5f/c152 0 2024-04-12T05:09:14.845 INFO:tasks.workunit.client.1.smithi192.stdout:6/906: dwrite d3/dc/d39/f3a [4194304,4194304] 0 2024-04-12T05:09:14.847 INFO:tasks.workunit.client.0.smithi113.stdout:7/896: dwrite d0/dc/d50/d123/d12d/d1a/fb8 [4194304,4194304] 0 2024-04-12T05:09:14.847 INFO:tasks.workunit.client.1.smithi192.stdout:2/876: dwrite d1/d8/d17/d19/d60/d9c/fc7 [0,4194304] 0 2024-04-12T05:09:14.848 INFO:tasks.workunit.client.1.smithi192.stdout:6/907: chown d3/d21/d24/c96 110170 1 2024-04-12T05:09:14.850 INFO:tasks.workunit.client.1.smithi192.stdout:3/931: fsync d5/d13/d3a/d4e/fd6 0 2024-04-12T05:09:14.851 INFO:tasks.workunit.client.0.smithi113.stdout:4/930: fsync da/d24/d7e/d79/dcd/fe6 0 2024-04-12T05:09:14.853 INFO:tasks.workunit.client.1.smithi192.stdout:6/908: rename d3/d23/f74 to d3/dc/d39/f12a 0 2024-04-12T05:09:14.854 INFO:tasks.workunit.client.0.smithi113.stdout:9/989: mkdir d1/d27/d7a/d14b 0 2024-04-12T05:09:14.855 INFO:tasks.workunit.client.0.smithi113.stdout:4/931: write da/d24/d7b/fee [774657,41712] 0 2024-04-12T05:09:14.855 INFO:tasks.workunit.client.1.smithi192.stdout:9/734: truncate f10 3014178 0 2024-04-12T05:09:14.856 INFO:tasks.workunit.client.0.smithi113.stdout:7/897: truncate d0/dc/da9/d5c/d9a/db6/d101/f10f 1999833 0 2024-04-12T05:09:14.857 INFO:tasks.workunit.client.0.smithi113.stdout:7/898: readlink d0/dc/d10b/le8 0 2024-04-12T05:09:14.858 INFO:tasks.workunit.client.0.smithi113.stdout:7/899: chown d0/dc/d50/d123/d12d 6 1 2024-04-12T05:09:14.858 INFO:tasks.workunit.client.0.smithi113.stdout:9/990: read d1/d27/f3c [3361357,123171] 0 2024-04-12T05:09:14.860 INFO:tasks.workunit.client.0.smithi113.stdout:9/991: chown d1/df/d4b/df7/d111 2 1 2024-04-12T05:09:14.861 INFO:tasks.workunit.client.1.smithi192.stdout:4/904: write d1/f33 [1099700,127370] 0 2024-04-12T05:09:14.862 INFO:tasks.workunit.client.1.smithi192.stdout:6/909: symlink d3/d8/d48/l12b 0 2024-04-12T05:09:14.865 INFO:tasks.workunit.client.0.smithi113.stdout:4/932: mknod da/d3c/d83/d47/d10f/c134 0 2024-04-12T05:09:14.867 INFO:tasks.workunit.client.0.smithi113.stdout:7/900: chown d0/dc/d50/d123/d12d/d78/daa/la3 1 1 2024-04-12T05:09:14.867 INFO:tasks.workunit.client.0.smithi113.stdout:9/992: chown d1/d4/d38/d65/l1d 110898 1 2024-04-12T05:09:14.868 INFO:tasks.workunit.client.0.smithi113.stdout:4/933: read - da/d3c/d43/dbc/ffc zero size 2024-04-12T05:09:14.869 INFO:tasks.workunit.client.1.smithi192.stdout:2/877: dwrite d1/d2d/d73/de7/f103 [0,4194304] 0 2024-04-12T05:09:14.869 INFO:tasks.workunit.client.1.smithi192.stdout:3/932: link d5/d13/d94/ca6 d5/d13/d3a/d70/d8c/d8d/dbc/c153 0 2024-04-12T05:09:14.869 INFO:tasks.workunit.client.1.smithi192.stdout:4/905: write d1/d1d/d53/d1c/f65 [1891474,98127] 0 2024-04-12T05:09:14.870 INFO:tasks.workunit.client.1.smithi192.stdout:6/910: chown d3/d8/c3d 8287992 1 2024-04-12T05:09:14.871 INFO:tasks.workunit.client.0.smithi113.stdout:4/934: chown da/d3c/d9a/dc1 7263146 1 2024-04-12T05:09:14.871 INFO:tasks.workunit.client.0.smithi113.stdout:9/993: dread - d1/d4/d38/d65/dcd/dd7/f123 zero size 2024-04-12T05:09:14.872 INFO:tasks.workunit.client.1.smithi192.stdout:9/735: dread d11/d25/d36/d6e/d71/f95 [0,4194304] 0 2024-04-12T05:09:14.875 INFO:tasks.workunit.client.0.smithi113.stdout:3/937: dwrite d0/d13/d15/f2e [0,4194304] 0 2024-04-12T05:09:14.875 INFO:tasks.workunit.client.1.smithi192.stdout:3/933: rename d5/df1 to d5/d13/d3c/d26/d154 0 2024-04-12T05:09:14.877 INFO:tasks.workunit.client.1.smithi192.stdout:4/906: symlink d1/d1d/d53/d1c/d95/dc6/l137 0 2024-04-12T05:09:14.877 INFO:tasks.workunit.client.0.smithi113.stdout:7/901: symlink d0/dc/d50/d123/d12d/l130 0 2024-04-12T05:09:14.878 INFO:tasks.workunit.client.0.smithi113.stdout:4/935: creat da/d3c/d9f/f135 x:0 0 0 2024-04-12T05:09:14.879 INFO:tasks.workunit.client.1.smithi192.stdout:9/736: truncate d11/d25/dec/d1f/d50/d73/f80 622661 0 2024-04-12T05:09:14.880 INFO:tasks.workunit.client.0.smithi113.stdout:9/994: creat d1/df/d29/de8/dc7/f14c x:0 0 0 2024-04-12T05:09:14.880 INFO:tasks.workunit.client.0.smithi113.stdout:4/936: chown da/d24/fe3 135 1 2024-04-12T05:09:14.880 INFO:tasks.workunit.client.1.smithi192.stdout:3/934: symlink d5/d13/d3a/d4a/l155 0 2024-04-12T05:09:14.881 INFO:tasks.workunit.client.0.smithi113.stdout:9/995: write d1/d4/d5/f103 [473820,57310] 0 2024-04-12T05:09:14.882 INFO:tasks.workunit.client.1.smithi192.stdout:3/935: dread - d5/d13/d3a/d8b/da2/d105/f142 zero size 2024-04-12T05:09:14.883 INFO:tasks.workunit.client.0.smithi113.stdout:3/938: creat d0/d13/d15/d28/d2a/d34/f138 x:0 0 0 2024-04-12T05:09:14.884 INFO:tasks.workunit.client.0.smithi113.stdout:4/937: creat da/d3c/d9f/f136 x:0 0 0 2024-04-12T05:09:14.886 INFO:tasks.workunit.client.0.smithi113.stdout:3/939: write d0/d7e/fb3 [1285058,84511] 0 2024-04-12T05:09:14.886 INFO:tasks.workunit.client.1.smithi192.stdout:9/737: rename d11/d25/dec/d1f/d6a/d7e/d89 to d11/d25/dd3/d10d 0 2024-04-12T05:09:14.887 INFO:tasks.workunit.client.0.smithi113.stdout:3/940: fsync d0/d80/ff0 0 2024-04-12T05:09:14.888 INFO:tasks.workunit.client.0.smithi113.stdout:7/902: read d0/dc/f18 [7149714,41535] 0 2024-04-12T05:09:14.889 INFO:tasks.workunit.client.0.smithi113.stdout:4/938: symlink da/d24/d115/l137 0 2024-04-12T05:09:14.889 INFO:tasks.workunit.client.0.smithi113.stdout:6/898: dread d8/de/d17/f72 [0,4194304] 0 2024-04-12T05:09:14.891 INFO:tasks.workunit.client.1.smithi192.stdout:3/936: mknod d5/d13/d94/c156 0 2024-04-12T05:09:14.892 INFO:tasks.workunit.client.1.smithi192.stdout:2/878: getdents d1/d8 0 2024-04-12T05:09:14.893 INFO:tasks.workunit.client.0.smithi113.stdout:3/941: mknod d0/d13/d47/d94/dcf/c139 0 2024-04-12T05:09:14.893 INFO:tasks.workunit.client.1.smithi192.stdout:9/738: symlink d11/d25/dd3/d10d/dce/l10e 0 2024-04-12T05:09:14.894 INFO:tasks.workunit.client.1.smithi192.stdout:2/879: fdatasync d1/d8/d17/d19/f31 0 2024-04-12T05:09:14.895 INFO:tasks.workunit.client.1.smithi192.stdout:3/937: mkdir d5/d13/d3a/d70/d157 0 2024-04-12T05:09:14.898 INFO:tasks.workunit.client.1.smithi192.stdout:9/739: mkdir d11/d25/d36/d6e/d33/da2/ddf/d10f 0 2024-04-12T05:09:14.898 INFO:tasks.workunit.client.1.smithi192.stdout:3/938: dread - d5/d13/d3a/d8b/da2/d123/d144/f14f zero size 2024-04-12T05:09:14.899 INFO:tasks.workunit.client.0.smithi113.stdout:3/942: creat d0/d13/d15/d28/d2a/d34/d5f/f13a x:0 0 0 2024-04-12T05:09:14.899 INFO:tasks.workunit.client.0.smithi113.stdout:4/939: dwrite da/d24/d7e/faf [0,4194304] 0 2024-04-12T05:09:14.899 INFO:tasks.workunit.client.0.smithi113.stdout:6/899: dread d8/de/d13/d11/d36/d3d/ffb [0,4194304] 0 2024-04-12T05:09:14.901 INFO:tasks.workunit.client.0.smithi113.stdout:4/940: write da/d3c/d9a/dc1/d106/f62 [4586981,20506] 0 2024-04-12T05:09:14.903 INFO:tasks.workunit.client.1.smithi192.stdout:6/911: dwrite d3/d21/f7d [0,4194304] 0 2024-04-12T05:09:14.903 INFO:tasks.workunit.client.0.smithi113.stdout:9/996: write d1/d4/d5/d17/d117/df1/dff/d10f/f146 [133767,80068] 0 2024-04-12T05:09:14.904 INFO:tasks.workunit.client.0.smithi113.stdout:7/903: write d0/dc/d50/d123/d12d/d78/f45 [3650870,90702] 0 2024-04-12T05:09:14.906 INFO:tasks.workunit.client.1.smithi192.stdout:9/740: mknod d11/d25/dd3/d10d/dce/c110 0 2024-04-12T05:09:14.906 INFO:tasks.workunit.client.0.smithi113.stdout:3/943: stat d0/d13/d15/d28/d2a/d8f/ddd/f115 0 2024-04-12T05:09:14.909 INFO:tasks.workunit.client.0.smithi113.stdout:4/941: creat da/d24/d60/daa/d112/f138 x:0 0 0 2024-04-12T05:09:14.909 INFO:tasks.workunit.client.0.smithi113.stdout:3/944: creat d0/d13/d47/d4f/dd3/f13b x:0 0 0 2024-04-12T05:09:14.910 INFO:tasks.workunit.client.1.smithi192.stdout:6/912: symlink d3/d23/db2/l12c 0 2024-04-12T05:09:14.910 INFO:tasks.workunit.client.0.smithi113.stdout:9/997: symlink d1/d4/d5/d17/d117/df1/dff/d10f/l14d 0 2024-04-12T05:09:14.911 INFO:tasks.workunit.client.0.smithi113.stdout:3/945: rmdir d0/d7e 39 2024-04-12T05:09:14.912 INFO:tasks.workunit.client.0.smithi113.stdout:7/904: truncate d0/dd/d95/dd4/f11a 379299 0 2024-04-12T05:09:14.913 INFO:tasks.workunit.client.1.smithi192.stdout:3/939: rename d5/d13/d1a/l5e to d5/d13/d3a/d8b/da2/d123/d137/l158 0 2024-04-12T05:09:14.913 INFO:tasks.workunit.client.0.smithi113.stdout:4/942: symlink da/d3c/d83/d47/dbd/l139 0 2024-04-12T05:09:14.914 INFO:tasks.workunit.client.0.smithi113.stdout:3/946: read - d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/fdc zero size 2024-04-12T05:09:14.915 INFO:tasks.workunit.client.1.smithi192.stdout:4/907: dwrite d1/d1d/d53/d1c/dc1/fce [0,4194304] 0 2024-04-12T05:09:14.916 INFO:tasks.workunit.client.0.smithi113.stdout:3/947: write d0/d13/d47/d94/dac/f112 [249486,116621] 0 2024-04-12T05:09:14.918 INFO:tasks.workunit.client.0.smithi113.stdout:3/948: readlink d0/le 0 2024-04-12T05:09:14.918 INFO:tasks.workunit.client.0.smithi113.stdout:4/943: fsync da/d24/d7e/d124/d1b/fb1 0 2024-04-12T05:09:14.921 INFO:tasks.workunit.client.1.smithi192.stdout:9/741: unlink d11/d25/dec/d1f/lb5 0 2024-04-12T05:09:14.921 INFO:tasks.workunit.client.1.smithi192.stdout:2/880: write d1/d8/d17/d19/d60/f99 [360127,32270] 0 2024-04-12T05:09:14.922 INFO:tasks.workunit.client.0.smithi113.stdout:4/944: read da/d3c/d83/d47/dbd/fea [4228878,127428] 0 2024-04-12T05:09:14.922 INFO:tasks.workunit.client.1.smithi192.stdout:3/940: rename d5/d13/d3a/d4e/l65 to d5/d13/d3a/d70/l159 0 2024-04-12T05:09:14.924 INFO:tasks.workunit.client.1.smithi192.stdout:4/908: write d1/d54/d82/f48 [324568,60530] 0 2024-04-12T05:09:14.927 INFO:tasks.workunit.client.1.smithi192.stdout:2/881: write d1/d8/d17/d19/d60/f65 [401345,42641] 0 2024-04-12T05:09:14.928 INFO:tasks.workunit.client.1.smithi192.stdout:6/913: truncate d3/d8/d56/d59/d97/dbf/de8/ffa 2131698 0 2024-04-12T05:09:14.930 INFO:tasks.workunit.client.0.smithi113.stdout:7/905: truncate d0/dd/d11/f77 2620364 0 2024-04-12T05:09:14.932 INFO:tasks.workunit.client.0.smithi113.stdout:4/945: unlink da/d3c/d83/d47/d10f/ce2 0 2024-04-12T05:09:14.934 INFO:tasks.workunit.client.1.smithi192.stdout:9/742: link d11/d25/dec/d1f/d50/d73/f8b d11/d25/d36/f111 0 2024-04-12T05:09:14.934 INFO:tasks.workunit.client.0.smithi113.stdout:7/906: truncate d0/dc/f41 2451052 0 2024-04-12T05:09:14.935 INFO:tasks.workunit.client.0.smithi113.stdout:4/946: symlink da/d24/d60/db8/l13a 0 2024-04-12T05:09:14.936 INFO:tasks.workunit.client.1.smithi192.stdout:6/914: getdents d3 0 2024-04-12T05:09:14.938 INFO:tasks.workunit.client.1.smithi192.stdout:2/882: dread d1/d8/d17/d19/d60/fa6 [0,4194304] 0 2024-04-12T05:09:14.941 INFO:tasks.workunit.client.1.smithi192.stdout:6/915: creat d3/d5/d9c/f12d x:0 0 0 2024-04-12T05:09:14.941 INFO:tasks.workunit.client.0.smithi113.stdout:7/907: getdents d0/dc/d50/d123 0 2024-04-12T05:09:14.944 INFO:tasks.workunit.client.1.smithi192.stdout:9/743: link d11/d25/dd3/d10d/dce/fde d11/d25/d36/d6e/d33/da2/f112 0 2024-04-12T05:09:14.945 INFO:tasks.workunit.client.0.smithi113.stdout:4/947: dwrite da/d3c/d83/d47/d10f/f5d [0,4194304] 0 2024-04-12T05:09:14.948 INFO:tasks.workunit.client.0.smithi113.stdout:9/998: dwrite d1/d4/d5/d22/f9c [0,4194304] 0 2024-04-12T05:09:14.951 INFO:tasks.workunit.client.0.smithi113.stdout:7/908: read d0/d6c/f70 [4060991,105762] 0 2024-04-12T05:09:14.952 INFO:tasks.workunit.client.1.smithi192.stdout:7/794: sync 2024-04-12T05:09:14.952 INFO:tasks.workunit.client.1.smithi192.stdout:6/916: unlink d3/dc/fc4 0 2024-04-12T05:09:14.952 INFO:tasks.workunit.client.1.smithi192.stdout:1/879: sync 2024-04-12T05:09:14.952 INFO:tasks.workunit.client.1.smithi192.stdout:8/878: sync 2024-04-12T05:09:14.952 INFO:tasks.workunit.client.0.smithi113.stdout:3/949: dwrite d0/f6 [0,4194304] 0 2024-04-12T05:09:14.955 INFO:tasks.workunit.client.1.smithi192.stdout:8/879: truncate d6/d10/d46/d5e/d6e/fbf 931483 0 2024-04-12T05:09:14.956 INFO:tasks.workunit.client.0.smithi113.stdout:9/999: stat d1/d4/d5/d17/d117/df1/dff/d10f/f146 0 2024-04-12T05:09:14.960 INFO:tasks.workunit.client.1.smithi192.stdout:1/880: rmdir d0/d2/d1e/d3f/df2/d3c/d5b/d7c 39 2024-04-12T05:09:14.961 INFO:tasks.workunit.client.0.smithi113.stdout:7/909: chown d0/dd/d11/f5e 66 1 2024-04-12T05:09:14.962 INFO:tasks.workunit.client.1.smithi192.stdout:3/941: dwrite d5/d13/f2f [8388608,4194304] 0 2024-04-12T05:09:14.963 INFO:tasks.workunit.client.1.smithi192.stdout:4/909: dread d1/d54/d82/f48 [0,4194304] 0 2024-04-12T05:09:14.963 INFO:tasks.workunit.client.0.smithi113.stdout:3/950: unlink d0/l66 0 2024-04-12T05:09:14.966 INFO:tasks.workunit.client.1.smithi192.stdout:7/795: dread d3/d1d/d4a/d5e/d6d/ffc [0,4194304] 0 2024-04-12T05:09:14.968 INFO:tasks.workunit.client.0.smithi113.stdout:7/910: readlink d0/dd/d11/de6/l69 0 2024-04-12T05:09:14.968 INFO:tasks.workunit.client.1.smithi192.stdout:4/910: chown d1/d49/c4a 75 1 2024-04-12T05:09:14.973 INFO:tasks.workunit.client.1.smithi192.stdout:3/942: rmdir d5/d13/d3a/d8b/da2/d123 39 2024-04-12T05:09:14.974 INFO:tasks.workunit.client.1.smithi192.stdout:7/796: truncate d3/d1d/d74/d86/fed 925265 0 2024-04-12T05:09:14.976 INFO:tasks.workunit.client.1.smithi192.stdout:7/797: readlink d3/d7/l2b 0 2024-04-12T05:09:14.976 INFO:tasks.workunit.client.1.smithi192.stdout:4/911: unlink d1/d54/d55/d64/f7d 0 2024-04-12T05:09:14.976 INFO:tasks.workunit.client.0.smithi113.stdout:4/948: dwrite da/d3c/d83/d47/d10f/f72 [4194304,4194304] 0 2024-04-12T05:09:14.978 INFO:tasks.workunit.client.1.smithi192.stdout:1/881: dwrite d0/d1b/d44/d69/ffb [0,4194304] 0 2024-04-12T05:09:14.981 INFO:tasks.workunit.client.1.smithi192.stdout:7/798: readlink d3/d7/d15/d1a/l2a 0 2024-04-12T05:09:14.982 INFO:tasks.workunit.client.0.smithi113.stdout:4/949: rmdir da/d3c/d43/dbc 39 2024-04-12T05:09:14.985 INFO:tasks.workunit.client.1.smithi192.stdout:9/744: dwrite d11/d25/d36/d6e/d33/da2/f84 [0,4194304] 0 2024-04-12T05:09:14.985 INFO:tasks.workunit.client.1.smithi192.stdout:3/943: unlink d5/d13/d3a/d4e/f102 0 2024-04-12T05:09:14.990 INFO:tasks.workunit.client.1.smithi192.stdout:4/912: chown d1/d1d/d53/d1c/d95/dc6/dd5/de5/ded 20851 1 2024-04-12T05:09:14.990 INFO:tasks.workunit.client.1.smithi192.stdout:1/882: dread - d0/d2/d1e/d3f/df2/d3c/d5b/d74/d90/d108/f68 zero size 2024-04-12T05:09:14.991 INFO:tasks.workunit.client.1.smithi192.stdout:8/880: truncate d6/d10/d46/d5e/d5f/f16 3510217 0 2024-04-12T05:09:14.994 INFO:tasks.workunit.client.1.smithi192.stdout:3/944: symlink d5/d13/d94/dbd/l15a 0 2024-04-12T05:09:14.995 INFO:tasks.workunit.client.1.smithi192.stdout:7/799: dwrite d3/d1d/d4a/d5e/d6d/fba [0,4194304] 0 2024-04-12T05:09:14.995 INFO:tasks.workunit.client.1.smithi192.stdout:1/883: write d0/d29/dbf/f11f [539195,111984] 0 2024-04-12T05:09:14.997 INFO:tasks.workunit.client.1.smithi192.stdout:9/745: rename d11/d15/d21/l87 to d11/d25/dd3/l113 0 2024-04-12T05:09:14.998 INFO:tasks.workunit.client.1.smithi192.stdout:8/881: symlink d6/d10/d46/d5e/d5f/d37/d7b/l111 0 2024-04-12T05:09:15.003 INFO:tasks.workunit.client.1.smithi192.stdout:7/800: unlink d3/d7/d14/d1f/fbf 0 2024-04-12T05:09:15.004 INFO:tasks.workunit.client.0.smithi113.stdout:4/950: truncate da/d3c/d9a/dc1/d106/f62 2601475 0 2024-04-12T05:09:15.005 INFO:tasks.workunit.client.0.smithi113.stdout:3/951: dwrite d0/d13/d15/d28/d2a/d34/d5f/d7b/fcd [0,4194304] 0 2024-04-12T05:09:15.008 INFO:tasks.workunit.client.1.smithi192.stdout:9/746: creat d11/d25/d36/d6e/d8f/d9c/f114 x:0 0 0 2024-04-12T05:09:15.011 INFO:tasks.workunit.client.0.smithi113.stdout:7/911: dwrite d0/dc/d50/d123/d12d/d1a/f2b [0,4194304] 0 2024-04-12T05:09:15.013 INFO:tasks.workunit.client.0.smithi113.stdout:3/952: truncate d0/d13/d15/d28/d2a/d34/fbd 956809 0 2024-04-12T05:09:15.014 INFO:tasks.workunit.client.0.smithi113.stdout:3/953: chown d0/d13/d15/d28/d2a/d34/d5f/d7b/lde 5136498 1 2024-04-12T05:09:15.015 INFO:tasks.workunit.client.1.smithi192.stdout:7/801: truncate d3/d1d/d4a/f70 1257274 0 2024-04-12T05:09:15.015 INFO:tasks.workunit.client.0.smithi113.stdout:4/951: rename da/d24/d60/daa to da/d24/d7e/d124/d1b/d13b 0 2024-04-12T05:09:15.015 INFO:tasks.workunit.client.0.smithi113.stdout:3/954: chown d0/d13/d15/d28/d2a/d8f 78153222 1 2024-04-12T05:09:15.021 INFO:tasks.workunit.client.1.smithi192.stdout:7/802: mknod d3/d7/d15/d22/c10f 0 2024-04-12T05:09:15.025 INFO:tasks.workunit.client.0.smithi113.stdout:7/912: getdents d0/dc/da9/d5c/ddf 0 2024-04-12T05:09:15.029 INFO:tasks.workunit.client.0.smithi113.stdout:3/955: link d0/d13/d47/f78 d0/d13/d15/d28/d2a/d8b/d95/f13c 0 2024-04-12T05:09:15.030 INFO:tasks.workunit.client.1.smithi192.stdout:8/882: truncate d6/d10/d46/d5e/d5f/f57 1380866 0 2024-04-12T05:09:15.030 INFO:tasks.workunit.client.1.smithi192.stdout:7/803: creat d3/d7/dff/f110 x:0 0 0 2024-04-12T05:09:15.031 INFO:tasks.workunit.client.0.smithi113.stdout:7/913: rename d0/dc/d50/d123/d12d/d78/daa/cbf to d0/dc/d50/d123/d12d/d9b/d9f/df4/c131 0 2024-04-12T05:09:15.032 INFO:tasks.workunit.client.1.smithi192.stdout:3/945: dwrite d5/d13/d3a/d8b/fae [0,4194304] 0 2024-04-12T05:09:15.034 INFO:tasks.workunit.client.0.smithi113.stdout:3/956: mkdir d0/d13/d47/d13d 0 2024-04-12T05:09:15.034 INFO:tasks.workunit.client.1.smithi192.stdout:7/804: fdatasync d3/d7/d14/d1f/d76/fa7 0 2024-04-12T05:09:15.037 INFO:tasks.workunit.client.1.smithi192.stdout:4/913: dread d1/d1d/d53/f52 [0,4194304] 0 2024-04-12T05:09:15.041 INFO:tasks.workunit.client.0.smithi113.stdout:3/957: dwrite d0/d13/f7d [4194304,4194304] 0 2024-04-12T05:09:15.089 INFO:tasks.workunit.client.1.smithi192.stdout:0/975: sync 2024-04-12T05:09:15.089 INFO:tasks.workunit.client.1.smithi192.stdout:5/963: sync 2024-04-12T05:09:15.090 INFO:tasks.workunit.client.0.smithi113.stdout:7/914: mkdir d0/d6c/d132 0 2024-04-12T05:09:15.091 INFO:tasks.workunit.client.1.smithi192.stdout:7/805: symlink d3/d1d/d74/d86/l111 0 2024-04-12T05:09:15.094 INFO:tasks.workunit.client.0.smithi113.stdout:7/915: dread d0/dc/d50/d123/d12d/d1a/f2b [0,4194304] 0 2024-04-12T05:09:15.094 INFO:tasks.workunit.client.1.smithi192.stdout:9/747: dread d11/d25/d36/d6e/d33/dbb/de7/f10b [0,4194304] 0 2024-04-12T05:09:15.095 INFO:tasks.workunit.client.1.smithi192.stdout:1/884: dread d0/d2/d1e/d3f/df2/d3c/f9c [0,4194304] 0 2024-04-12T05:09:15.095 INFO:tasks.workunit.client.0.smithi113.stdout:7/916: truncate d0/dd/d95/f11f 522907 0 2024-04-12T05:09:15.097 INFO:tasks.workunit.client.0.smithi113.stdout:7/917: fsync d0/d6c/ff5 0 2024-04-12T05:09:15.098 INFO:tasks.workunit.client.0.smithi113.stdout:3/958: dread d0/d13/d15/d28/d2a/d34/f63 [0,4194304] 0 2024-04-12T05:09:15.099 INFO:tasks.workunit.client.0.smithi113.stdout:7/918: read d0/dd/d95/f11f [451606,28304] 0 2024-04-12T05:09:15.099 INFO:tasks.workunit.client.1.smithi192.stdout:0/976: unlink d0/fd0 0 2024-04-12T05:09:15.100 INFO:tasks.workunit.client.1.smithi192.stdout:4/914: dwrite d1/d1d/d53/d1c/f4e [0,4194304] 0 2024-04-12T05:09:15.101 INFO:tasks.workunit.client.1.smithi192.stdout:0/977: chown d0/d7b/d9b/d4b/c61 5534 1 2024-04-12T05:09:15.103 INFO:tasks.workunit.client.0.smithi113.stdout:3/959: dread d0/d13/d47/f89 [0,4194304] 0 2024-04-12T05:09:15.104 INFO:tasks.workunit.client.1.smithi192.stdout:7/806: truncate d3/d1d/d74/f8c 1730175 0 2024-04-12T05:09:15.106 INFO:tasks.workunit.client.1.smithi192.stdout:9/748: mkdir d11/d25/dec/d1f/d6a/d92/d115 0 2024-04-12T05:09:15.107 INFO:tasks.workunit.client.1.smithi192.stdout:9/749: fdatasync d11/d25/dec/fcf 0 2024-04-12T05:09:15.107 INFO:tasks.workunit.client.0.smithi113.stdout:3/960: symlink d0/d13/d47/d4f/dca/l13e 0 2024-04-12T05:09:15.109 INFO:tasks.workunit.client.1.smithi192.stdout:9/750: write d11/d25/dec/d1f/d94/ffc [721624,83557] 0 2024-04-12T05:09:15.110 INFO:tasks.workunit.client.1.smithi192.stdout:1/885: rename d0/d29/lef to d0/d1b/d28/d9d/l123 0 2024-04-12T05:09:15.112 INFO:tasks.workunit.client.1.smithi192.stdout:8/883: write d6/d10/d109/fb7 [982346,127693] 0 2024-04-12T05:09:15.113 INFO:tasks.workunit.client.0.smithi113.stdout:3/961: mkdir d0/d13/d15/d13f 0 2024-04-12T05:09:15.113 INFO:tasks.workunit.client.1.smithi192.stdout:8/884: fsync d6/d10/d46/d7c/fa0 0 2024-04-12T05:09:15.114 INFO:tasks.workunit.client.0.smithi113.stdout:3/962: stat d0/d13/d47/d94/dcf/dd1/de2/f107 0 2024-04-12T05:09:15.115 INFO:tasks.workunit.client.1.smithi192.stdout:5/964: symlink d0/d15/d3e/l148 0 2024-04-12T05:09:15.116 INFO:tasks.workunit.client.1.smithi192.stdout:0/978: mknod d0/d7b/d9b/d35/d109/d53/d6d/d108/c14e 0 2024-04-12T05:09:15.120 INFO:tasks.workunit.client.1.smithi192.stdout:0/979: readlink d0/d7b/d9b/d35/d109/d95/db4/db8/l11a 0 2024-04-12T05:09:15.120 INFO:tasks.workunit.client.1.smithi192.stdout:1/886: mknod d0/d2/d1e/d3f/df2/d3c/c124 0 2024-04-12T05:09:15.121 INFO:tasks.workunit.client.0.smithi113.stdout:3/963: dwrite d0/d80/f131 [0,4194304] 0 2024-04-12T05:09:15.121 INFO:tasks.workunit.client.1.smithi192.stdout:0/980: chown d0/d7b/d7c/lae 430 1 2024-04-12T05:09:15.122 INFO:tasks.workunit.client.0.smithi113.stdout:3/964: stat d0/d13/d15/f3d 0 2024-04-12T05:09:15.123 INFO:tasks.workunit.client.1.smithi192.stdout:3/946: dwrite d5/f19 [0,4194304] 0 2024-04-12T05:09:15.124 INFO:tasks.workunit.client.0.smithi113.stdout:3/965: dread - d0/d13/d15/d28/d2a/d34/d5f/d7b/de5/f12c zero size 2024-04-12T05:09:15.125 INFO:tasks.workunit.client.1.smithi192.stdout:8/885: mkdir d6/d10/d46/d5e/d5f/d31/d80/db1/d112 0 2024-04-12T05:09:15.126 INFO:tasks.workunit.client.0.smithi113.stdout:3/966: dread - d0/d13/d15/d28/d2a/d34/d5f/d7b/f135 zero size 2024-04-12T05:09:15.127 INFO:tasks.workunit.client.1.smithi192.stdout:0/981: write d0/d7b/d9b/d4b/dbc/dfa/f14b [1438381,30530] 0 2024-04-12T05:09:15.128 INFO:tasks.workunit.client.0.smithi113.stdout:3/967: write d0/d13/f117 [779194,17391] 0 2024-04-12T05:09:15.129 INFO:tasks.workunit.client.1.smithi192.stdout:1/887: readlink d0/d29/dbf/lea 0 2024-04-12T05:09:15.129 INFO:tasks.workunit.client.1.smithi192.stdout:8/886: stat d6/d10/d46/d5e/d5f/d31/d61/d9d/db8/f110 0 2024-04-12T05:09:15.133 INFO:tasks.workunit.client.0.smithi113.stdout:7/919: dread d0/dd/f1c [0,4194304] 0 2024-04-12T05:09:15.133 INFO:tasks.workunit.client.1.smithi192.stdout:4/915: dread d1/d54/d82/d71/f7b [0,4194304] 0 2024-04-12T05:09:15.133 INFO:tasks.workunit.client.1.smithi192.stdout:8/887: dread d6/d10/d46/d5e/d5f/d38/dbc/fec [0,4194304] 0 2024-04-12T05:09:15.135 INFO:tasks.workunit.client.0.smithi113.stdout:3/968: symlink d0/d13/d15/l140 0 2024-04-12T05:09:15.136 INFO:tasks.workunit.client.1.smithi192.stdout:9/751: creat d11/d25/d36/d3d/f116 x:0 0 0 2024-04-12T05:09:15.136 INFO:tasks.workunit.client.0.smithi113.stdout:7/920: creat d0/dc/d50/d123/f133 x:0 0 0 2024-04-12T05:09:15.137 INFO:tasks.workunit.client.1.smithi192.stdout:1/888: mkdir d0/d1b/d44/d69/d125 0 2024-04-12T05:09:15.138 INFO:tasks.workunit.client.0.smithi113.stdout:3/969: rmdir d0/d13/d47/d4f/dd3 39 2024-04-12T05:09:15.139 INFO:tasks.workunit.client.1.smithi192.stdout:0/982: truncate d0/d7b/d9b/d35/d109/d53/fe7 2740025 0 2024-04-12T05:09:15.140 INFO:tasks.workunit.client.0.smithi113.stdout:7/921: symlink d0/dd/d11/de6/l134 0 2024-04-12T05:09:15.141 INFO:tasks.workunit.client.1.smithi192.stdout:4/916: creat d1/d54/d82/da4/da8/f138 x:0 0 0 2024-04-12T05:09:15.142 INFO:tasks.workunit.client.1.smithi192.stdout:8/888: symlink d6/d10/d46/d5e/d5f/d31/d61/d9d/l113 0 2024-04-12T05:09:15.143 INFO:tasks.workunit.client.0.smithi113.stdout:7/922: creat d0/dc/d50/d123/d12d/d9b/d9f/f135 x:0 0 0 2024-04-12T05:09:15.143 INFO:tasks.workunit.client.0.smithi113.stdout:7/923: chown d0/dd/fb5 111 1 2024-04-12T05:09:15.144 INFO:tasks.workunit.client.1.smithi192.stdout:1/889: unlink d0/d29/c111 0 2024-04-12T05:09:15.144 INFO:tasks.workunit.client.0.smithi113.stdout:7/924: chown d0/dc/f42 599206297 1 2024-04-12T05:09:15.146 INFO:tasks.workunit.client.1.smithi192.stdout:9/752: symlink d11/d25/d36/d6e/d33/da2/l117 0 2024-04-12T05:09:15.148 INFO:tasks.workunit.client.0.smithi113.stdout:7/925: fsync d0/dc/d50/d123/d12d/fa 0 2024-04-12T05:09:15.149 INFO:tasks.workunit.client.1.smithi192.stdout:4/917: mknod d1/d54/d82/d71/c139 0 2024-04-12T05:09:15.150 INFO:tasks.workunit.client.0.smithi113.stdout:7/926: write d0/dc/de7/f125 [158126,84346] 0 2024-04-12T05:09:15.153 INFO:tasks.workunit.client.1.smithi192.stdout:1/890: rename d0/d1b/d44/d69/ddd to d0/d2/d1e/d3f/df2/d3c/d5b/da7/dd7/d126 0 2024-04-12T05:09:15.155 INFO:tasks.workunit.client.0.smithi113.stdout:3/970: dread d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/d12b/f12f [0,4194304] 0 2024-04-12T05:09:15.156 INFO:tasks.workunit.client.0.smithi113.stdout:7/927: dread d0/dc/d50/d123/d12d/d78/f38 [0,4194304] 0 2024-04-12T05:09:15.156 INFO:tasks.workunit.client.1.smithi192.stdout:9/753: write d11/d25/d36/d6e/d71/f95 [497975,103467] 0 2024-04-12T05:09:15.157 INFO:tasks.workunit.client.1.smithi192.stdout:4/918: dwrite d1/d1d/d53/d1c/d84/dbb/ff6 [4194304,4194304] 0 2024-04-12T05:09:15.161 INFO:tasks.workunit.client.1.smithi192.stdout:0/983: unlink d0/d7b/d9b/d4b/dbc/dfa/f14b 0 2024-04-12T05:09:15.161 INFO:tasks.workunit.client.0.smithi113.stdout:7/928: mkdir d0/dc/d50/d123/d12d/d24/dd2/d136 0 2024-04-12T05:09:15.163 INFO:tasks.workunit.client.1.smithi192.stdout:8/889: getdents d6/d10/d6c/d9e/dde/de7 0 2024-04-12T05:09:15.166 INFO:tasks.workunit.client.0.smithi113.stdout:3/971: dwrite d0/d7e/fb3 [4194304,4194304] 0 2024-04-12T05:09:15.169 INFO:tasks.workunit.client.1.smithi192.stdout:5/965: getdents d0/d15/d3e/dca/dda/deb 0 2024-04-12T05:09:15.171 INFO:tasks.workunit.client.0.smithi113.stdout:3/972: mkdir d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/d106/d137/d141 0 2024-04-12T05:09:15.173 INFO:tasks.workunit.client.0.smithi113.stdout:3/973: dread - d0/d13/d15/d28/dc0/fea zero size 2024-04-12T05:09:15.175 INFO:tasks.workunit.client.0.smithi113.stdout:3/974: chown d0/d13/f1f 5047390 1 2024-04-12T05:09:15.177 INFO:tasks.workunit.client.1.smithi192.stdout:0/984: creat d0/d7b/d9b/d35/d109/d53/d6d/d108/f14f x:0 0 0 2024-04-12T05:09:15.177 INFO:tasks.workunit.client.1.smithi192.stdout:8/890: mkdir d6/d10/d6c/d9e/dde/d114 0 2024-04-12T05:09:15.178 INFO:tasks.workunit.client.0.smithi113.stdout:3/975: mknod d0/d13/d47/c142 0 2024-04-12T05:09:15.178 INFO:tasks.workunit.client.1.smithi192.stdout:3/947: getdents d5/d13/d3a/d8b/da2/d123 0 2024-04-12T05:09:15.179 INFO:tasks.workunit.client.0.smithi113.stdout:7/929: dwrite d0/f33 [4194304,4194304] 0 2024-04-12T05:09:15.179 INFO:tasks.workunit.client.1.smithi192.stdout:3/948: chown d5/d13/d3a/d4e/lcf 2476 1 2024-04-12T05:09:15.180 INFO:tasks.workunit.client.0.smithi113.stdout:7/930: dread - d0/dc/d50/d123/d12d/d24/fe1 zero size 2024-04-12T05:09:15.185 INFO:tasks.workunit.client.0.smithi113.stdout:3/976: getdents d0/d13 0 2024-04-12T05:09:15.187 INFO:tasks.workunit.client.0.smithi113.stdout:3/977: write d0/d13/d15/f3d [7635826,114743] 0 2024-04-12T05:09:15.189 INFO:tasks.workunit.client.1.smithi192.stdout:8/891: mkdir d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d98/d115 0 2024-04-12T05:09:15.190 INFO:tasks.workunit.client.1.smithi192.stdout:9/754: dwrite d11/d25/d36/d3d/f79 [0,4194304] 0 2024-04-12T05:09:15.191 INFO:tasks.workunit.client.1.smithi192.stdout:5/966: link d0/d15/d3e/dca/dda/deb/d113/d127/c11e d0/d15/d32/d69/dc8/c149 0 2024-04-12T05:09:15.196 INFO:tasks.workunit.client.0.smithi113.stdout:3/978: link d0/d13/d47/f78 d0/d13/d15/d28/d2a/d8f/ddd/f143 0 2024-04-12T05:09:15.197 INFO:tasks.workunit.client.1.smithi192.stdout:8/892: read - d6/d10/d46/d5e/d5f/d31/d80/db1/fcb zero size 2024-04-12T05:09:15.197 INFO:tasks.workunit.client.1.smithi192.stdout:4/919: dread d1/d54/d82/da4/f79 [0,4194304] 0 2024-04-12T05:09:15.197 INFO:tasks.workunit.client.1.smithi192.stdout:9/755: dread - d11/d15/ff0 zero size 2024-04-12T05:09:15.198 INFO:tasks.workunit.client.1.smithi192.stdout:5/967: fdatasync d0/d15/d64/fcd 0 2024-04-12T05:09:15.199 INFO:tasks.workunit.client.1.smithi192.stdout:8/893: creat d6/d10/d46/d5e/d5f/d38/dbc/f116 x:0 0 0 2024-04-12T05:09:15.200 INFO:tasks.workunit.client.0.smithi113.stdout:3/979: creat d0/d13/d47/d13d/f144 x:0 0 0 2024-04-12T05:09:15.201 INFO:tasks.workunit.client.1.smithi192.stdout:4/920: write d1/d1d/d53/d1c/d111/f136 [282616,78513] 0 2024-04-12T05:09:15.202 INFO:tasks.workunit.client.0.smithi113.stdout:3/980: fsync d0/f57 0 2024-04-12T05:09:15.203 INFO:tasks.workunit.client.0.smithi113.stdout:3/981: fsync d0/d13/d47/f11e 0 2024-04-12T05:09:15.205 INFO:tasks.workunit.client.1.smithi192.stdout:9/756: getdents d11/d25/d60 0 2024-04-12T05:09:15.207 INFO:tasks.workunit.client.0.smithi113.stdout:7/931: dread d0/dc/fde [0,4194304] 0 2024-04-12T05:09:15.209 INFO:tasks.workunit.client.0.smithi113.stdout:7/932: creat d0/d6c/f137 x:0 0 0 2024-04-12T05:09:15.210 INFO:tasks.workunit.client.1.smithi192.stdout:4/921: dread - d1/d1d/d53/d1c/dd1/def/ddd/dea/f100 zero size 2024-04-12T05:09:15.211 INFO:tasks.workunit.client.0.smithi113.stdout:7/933: symlink d0/dc/da9/d5c/ddf/l138 0 2024-04-12T05:09:15.212 INFO:tasks.workunit.client.1.smithi192.stdout:1/891: write d0/d2/d1e/d3f/df2/d3c/d5b/d74/d7d/fd6 [886469,61762] 0 2024-04-12T05:09:15.213 INFO:tasks.workunit.client.1.smithi192.stdout:3/949: dread d5/d13/d3a/d8b/da2/da7/fd9 [0,4194304] 0 2024-04-12T05:09:15.213 INFO:tasks.workunit.client.0.smithi113.stdout:3/982: write d0/d13/d15/d28/f6f [1611978,22962] 0 2024-04-12T05:09:15.213 INFO:tasks.workunit.client.0.smithi113.stdout:7/934: dread - d0/dc/d50/d123/d12d/d9b/d80/fca zero size 2024-04-12T05:09:15.214 INFO:tasks.workunit.client.1.smithi192.stdout:5/968: dwrite d0/d15/d24/d29/f128 [0,4194304] 0 2024-04-12T05:09:15.214 INFO:tasks.workunit.client.1.smithi192.stdout:8/894: rmdir d6/d10/d46/d5e/d5f/d37/d4f/d51/d65 39 2024-04-12T05:09:15.219 INFO:tasks.workunit.client.1.smithi192.stdout:5/969: fsync d0/d21/d5e/f135 0 2024-04-12T05:09:15.219 INFO:tasks.workunit.client.1.smithi192.stdout:0/985: dwrite d0/d7b/d9b/d35/d109/d9f/d10a/fa3 [0,4194304] 0 2024-04-12T05:09:15.220 INFO:tasks.workunit.client.1.smithi192.stdout:8/895: chown d6/d10/d6c/d9e/la4 133670 1 2024-04-12T05:09:15.221 INFO:tasks.workunit.client.1.smithi192.stdout:4/922: mkdir d1/d1d/d11f/d13a 0 2024-04-12T05:09:15.222 INFO:tasks.workunit.client.0.smithi113.stdout:3/983: dwrite d0/d13/d47/d13d/f144 [0,4194304] 0 2024-04-12T05:09:15.224 INFO:tasks.workunit.client.0.smithi113.stdout:3/984: chown d0/d13/d47/f4a 1 1 2024-04-12T05:09:15.227 INFO:tasks.workunit.client.1.smithi192.stdout:3/950: mknod d5/d13/d3a/d8b/da2/d123/d137/c15b 0 2024-04-12T05:09:15.227 INFO:tasks.workunit.client.1.smithi192.stdout:0/986: creat d0/d7b/d9b/d35/d109/d53/df3/d2e/d66/f150 x:0 0 0 2024-04-12T05:09:15.228 INFO:tasks.workunit.client.0.smithi113.stdout:3/985: mkdir d0/d13/d15/d28/d2a/d34/d5f/d7b/de5/d145 0 2024-04-12T05:09:15.231 INFO:tasks.workunit.client.1.smithi192.stdout:4/923: creat d1/d1d/d53/d11a/ddb/f13b x:0 0 0 2024-04-12T05:09:15.233 INFO:tasks.workunit.client.0.smithi113.stdout:7/935: dread d0/dc/da9/d5c/d85/fa4 [0,4194304] 0 2024-04-12T05:09:15.233 INFO:tasks.workunit.client.1.smithi192.stdout:0/987: chown d0/d7b/d9b/d35/d109/d9f/d10a/dc4/fd9 28433 1 2024-04-12T05:09:15.234 INFO:tasks.workunit.client.0.smithi113.stdout:7/936: chown d0/dc/da9/l65 4702682 1 2024-04-12T05:09:15.234 INFO:tasks.workunit.client.0.smithi113.stdout:3/986: truncate d0/d13/f118 466970 0 2024-04-12T05:09:15.234 INFO:tasks.workunit.client.1.smithi192.stdout:9/757: truncate d11/d15/f5c 130247 0 2024-04-12T05:09:15.237 INFO:tasks.workunit.client.0.smithi113.stdout:7/937: dread d0/dd/d95/dd4/f11a [0,4194304] 0 2024-04-12T05:09:15.237 INFO:tasks.workunit.client.1.smithi192.stdout:1/892: dwrite d0/d2/d1e/d3f/df2/d21/d39/f82 [0,4194304] 0 2024-04-12T05:09:15.237 INFO:tasks.workunit.client.1.smithi192.stdout:0/988: write d0/d7b/d9b/f12e [167912,77162] 0 2024-04-12T05:09:15.238 INFO:tasks.workunit.client.0.smithi113.stdout:3/987: chown d0/d13/d15/d28/d2a/d8f/cd0 6 1 2024-04-12T05:09:15.239 INFO:tasks.workunit.client.0.smithi113.stdout:7/938: write d0/dd/f66 [4948203,100622] 0 2024-04-12T05:09:15.239 INFO:tasks.workunit.client.1.smithi192.stdout:5/970: rename d0/d15/d3e/dca/ca3 to d0/d15/d32/d69/c14a 0 2024-04-12T05:09:15.240 INFO:tasks.workunit.client.1.smithi192.stdout:4/924: symlink d1/d1d/d53/d1c/d84/l13c 0 2024-04-12T05:09:15.241 INFO:tasks.workunit.client.1.smithi192.stdout:5/971: stat d0/d15/d3e/dca/dda/ffe 0 2024-04-12T05:09:15.243 INFO:tasks.workunit.client.0.smithi113.stdout:7/939: dread d0/dc/da9/d5c/d85/fb1 [0,4194304] 0 2024-04-12T05:09:15.245 INFO:tasks.workunit.client.1.smithi192.stdout:3/951: truncate d5/d13/d3a/d4e/f95 1445902 0 2024-04-12T05:09:15.246 INFO:tasks.workunit.client.1.smithi192.stdout:0/989: write d0/d7b/d9b/d35/d109/f65 [2820240,128086] 0 2024-04-12T05:09:15.247 INFO:tasks.workunit.client.1.smithi192.stdout:9/758: link d11/d25/d36/d6e/d33/f55 d11/d25/d36/d3d/d7a/df9/f118 0 2024-04-12T05:09:15.247 INFO:tasks.workunit.client.1.smithi192.stdout:1/893: read d0/d1b/f60 [190807,90251] 0 2024-04-12T05:09:15.248 INFO:tasks.workunit.client.0.smithi113.stdout:3/988: creat d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/f146 x:0 0 0 2024-04-12T05:09:15.250 INFO:tasks.workunit.client.0.smithi113.stdout:7/940: truncate d0/fa2 4545426 0 2024-04-12T05:09:15.251 INFO:tasks.workunit.client.1.smithi192.stdout:4/925: rename d1/d54/f94 to d1/d1d/d53/f13d 0 2024-04-12T05:09:15.251 INFO:tasks.workunit.client.1.smithi192.stdout:5/972: symlink d0/d15/d3e/d120/l14b 0 2024-04-12T05:09:15.254 INFO:tasks.workunit.client.0.smithi113.stdout:3/989: dwrite d0/d7e/f85 [0,4194304] 0 2024-04-12T05:09:15.254 INFO:tasks.workunit.client.0.smithi113.stdout:7/941: creat d0/dc/d50/d123/d12d/d78/df2/f139 x:0 0 0 2024-04-12T05:09:15.255 INFO:tasks.workunit.client.1.smithi192.stdout:1/894: creat d0/d2/d1e/d3f/df2/d3c/f127 x:0 0 0 2024-04-12T05:09:15.258 INFO:tasks.workunit.client.0.smithi113.stdout:3/990: symlink d0/d13/d15/d28/d2a/d34/d5f/l147 0 2024-04-12T05:09:15.263 INFO:tasks.workunit.client.0.smithi113.stdout:3/991: symlink d0/d13/d47/d4f/l148 0 2024-04-12T05:09:15.265 INFO:tasks.workunit.client.0.smithi113.stdout:7/942: truncate d0/dc/da9/d5c/d85/f8d 617854 0 2024-04-12T05:09:15.267 INFO:tasks.workunit.client.1.smithi192.stdout:9/759: dwrite d11/d25/dec/d1f/f67 [0,4194304] 0 2024-04-12T05:09:15.268 INFO:tasks.workunit.client.1.smithi192.stdout:1/895: creat d0/d2/d1e/d3f/df2/d21/d62/f128 x:0 0 0 2024-04-12T05:09:15.269 INFO:tasks.workunit.client.1.smithi192.stdout:3/952: truncate d5/f14 10722423 0 2024-04-12T05:09:15.271 INFO:tasks.workunit.client.1.smithi192.stdout:0/990: dwrite d0/d7b/d9b/d35/d109/d53/f9d [4194304,4194304] 0 2024-04-12T05:09:15.280 INFO:tasks.workunit.client.0.smithi113.stdout:7/943: write d0/dc/d50/d123/d12d/d1a/f29 [4028251,79540] 0 2024-04-12T05:09:15.284 INFO:tasks.workunit.client.0.smithi113.stdout:7/944: symlink d0/dc/d50/d123/d12d/d24/dd2/l13a 0 2024-04-12T05:09:15.284 INFO:tasks.workunit.client.1.smithi192.stdout:5/973: dwrite d0/d15/f23 [4194304,4194304] 0 2024-04-12T05:09:15.285 INFO:tasks.workunit.client.0.smithi113.stdout:7/945: chown d0/dd/d11/de6/l7f 41501 1 2024-04-12T05:09:15.287 INFO:tasks.workunit.client.1.smithi192.stdout:8/896: dread d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d98/fd1 [0,4194304] 0 2024-04-12T05:09:15.287 INFO:tasks.workunit.client.1.smithi192.stdout:9/760: dwrite d11/d25/f91 [0,4194304] 0 2024-04-12T05:09:15.292 INFO:tasks.workunit.client.1.smithi192.stdout:0/991: symlink d0/d7b/d9b/d35/d109/d53/df3/d29/db5/l151 0 2024-04-12T05:09:15.292 INFO:tasks.workunit.client.0.smithi113.stdout:7/946: link d0/dc/d50/d123/d12d/d9b/f23 d0/dc/d50/d123/d12d/d9b/d9f/df4/f13b 0 2024-04-12T05:09:15.294 INFO:tasks.workunit.client.1.smithi192.stdout:5/974: rmdir d0/d15/d24 39 2024-04-12T05:09:15.295 INFO:tasks.workunit.client.1.smithi192.stdout:1/896: unlink d0/d2/d1e/d3f/df2/d3c/d5b/d74/d90/le3 0 2024-04-12T05:09:15.296 INFO:tasks.workunit.client.1.smithi192.stdout:4/926: mknod d1/d54/d82/d71/d9e/dc4/dfe/d109/c13e 0 2024-04-12T05:09:15.297 INFO:tasks.workunit.client.0.smithi113.stdout:7/947: creat d0/dc/d50/d123/d12d/d9b/d9f/df4/f13c x:0 0 0 2024-04-12T05:09:15.297 INFO:tasks.workunit.client.1.smithi192.stdout:8/897: rmdir d6/d10/d46/d5e/d5f/d31/d61/dc5/dd9 39 2024-04-12T05:09:15.298 INFO:tasks.workunit.client.1.smithi192.stdout:4/927: chown d1/d1d/d53/d1c/d84/fa3 0 1 2024-04-12T05:09:15.299 INFO:tasks.workunit.client.1.smithi192.stdout:5/975: chown d0/d15/d3e/dca/dda/deb/d113/d127/f141 410519 1 2024-04-12T05:09:15.301 INFO:tasks.workunit.client.1.smithi192.stdout:0/992: mknod d0/d7b/d9b/d35/d109/d9f/d10a/d67/db1/dd6/c152 0 2024-04-12T05:09:15.301 INFO:tasks.workunit.client.1.smithi192.stdout:5/976: chown d0/d21/f101 761 1 2024-04-12T05:09:15.302 INFO:tasks.workunit.client.1.smithi192.stdout:3/953: link d5/d13/d3c/d26/l33 d5/d13/d1a/d5f/l15c 0 2024-04-12T05:09:15.305 INFO:tasks.workunit.client.1.smithi192.stdout:1/897: mknod d0/d2/d1e/d3f/df2/d3c/dab/c129 0 2024-04-12T05:09:15.305 INFO:tasks.workunit.client.1.smithi192.stdout:4/928: write d1/f2 [1116915,70250] 0 2024-04-12T05:09:15.307 INFO:tasks.workunit.client.1.smithi192.stdout:3/954: write d5/d13/d1a/d53/f143 [205895,3303] 0 2024-04-12T05:09:15.309 INFO:tasks.workunit.client.1.smithi192.stdout:0/993: rmdir d0/d7b/d9b/d4b/dbc 39 2024-04-12T05:09:15.311 INFO:tasks.workunit.client.1.smithi192.stdout:5/977: creat d0/d15/d32/d69/dc8/d12f/f14c x:0 0 0 2024-04-12T05:09:15.311 INFO:tasks.workunit.client.1.smithi192.stdout:8/898: dwrite d6/d10/d46/d5e/d5f/fea [0,4194304] 0 2024-04-12T05:09:15.312 INFO:tasks.workunit.client.0.smithi113.stdout:7/948: dwrite d0/dc/d50/d123/d12d/d9b/f23 [0,4194304] 0 2024-04-12T05:09:15.313 INFO:tasks.workunit.client.0.smithi113.stdout:7/949: chown d0/dc/d10b/ce5 1829005818 1 2024-04-12T05:09:15.313 INFO:tasks.workunit.client.1.smithi192.stdout:4/929: mkdir d1/d1d/d53/d1c/d84/dbb/df7/d13f 0 2024-04-12T05:09:15.316 INFO:tasks.workunit.client.1.smithi192.stdout:5/978: creat d0/f14d x:0 0 0 2024-04-12T05:09:15.320 INFO:tasks.workunit.client.0.smithi113.stdout:7/950: unlink d0/dc/d50/d123/d12d/d24/f104 0 2024-04-12T05:09:15.320 INFO:tasks.workunit.client.1.smithi192.stdout:1/898: rename d0/d2/d1e/d3f/df2/d3c/d5b to d0/d2/d1e/d3f/df2/d21/d62/d115/d12a 0 2024-04-12T05:09:15.321 INFO:tasks.workunit.client.1.smithi192.stdout:9/761: dread d11/d25/d36/f63 [0,4194304] 0 2024-04-12T05:09:15.323 INFO:tasks.workunit.client.1.smithi192.stdout:0/994: creat d0/d7b/d9b/d35/d109/d9f/d10a/db2/d13c/f153 x:0 0 0 2024-04-12T05:09:15.323 INFO:tasks.workunit.client.0.smithi113.stdout:7/951: creat d0/dc/d50/d123/d12d/d9b/d9f/f13d x:0 0 0 2024-04-12T05:09:15.324 INFO:tasks.workunit.client.1.smithi192.stdout:8/899: creat d6/d10/d46/d5e/d5f/d31/d61/f117 x:0 0 0 2024-04-12T05:09:15.324 INFO:tasks.workunit.client.0.smithi113.stdout:7/952: dread - d0/dc/da9/d5c/d9a/f111 zero size 2024-04-12T05:09:15.325 INFO:tasks.workunit.client.1.smithi192.stdout:1/899: mknod d0/d29/dbf/c12b 0 2024-04-12T05:09:15.326 INFO:tasks.workunit.client.0.smithi113.stdout:7/953: write d0/d6c/fb0 [5162395,111836] 0 2024-04-12T05:09:15.328 INFO:tasks.workunit.client.1.smithi192.stdout:8/900: chown d6/d10/d46/d5e/d5f/d37/d4f/d4c/f64 99766 1 2024-04-12T05:09:15.328 INFO:tasks.workunit.client.1.smithi192.stdout:9/762: truncate d11/d25/d36/d3d/fb7 979607 0 2024-04-12T05:09:15.330 INFO:tasks.workunit.client.1.smithi192.stdout:4/930: link d1/d1d/d53/d1c/dc1/cca d1/d54/d82/da4/de8/c140 0 2024-04-12T05:09:15.332 INFO:tasks.workunit.client.1.smithi192.stdout:1/900: dread - d0/d29/dbf/fd5 zero size 2024-04-12T05:09:15.333 INFO:tasks.workunit.client.1.smithi192.stdout:8/901: truncate f5 9423066 0 2024-04-12T05:09:15.333 INFO:tasks.workunit.client.0.smithi113.stdout:7/954: write d0/d52/f117 [339245,68296] 0 2024-04-12T05:09:15.333 INFO:tasks.workunit.client.1.smithi192.stdout:5/979: dwrite d0/d15/d24/d47/f4d [0,4194304] 0 2024-04-12T05:09:15.335 INFO:tasks.workunit.client.0.smithi113.stdout:7/955: mknod d0/dd/d95/c13e 0 2024-04-12T05:09:15.336 INFO:tasks.workunit.client.0.smithi113.stdout:6/900: sync 2024-04-12T05:09:15.336 INFO:tasks.workunit.client.1.smithi192.stdout:9/763: mknod d11/d25/d36/d6e/dc7/c119 0 2024-04-12T05:09:15.341 INFO:tasks.workunit.client.1.smithi192.stdout:1/901: chown d0/d2/d1e/d3f/df2/d3c/f9c 3095716 1 2024-04-12T05:09:15.341 INFO:tasks.workunit.client.1.smithi192.stdout:3/955: dwrite d5/d13/d3a/d8b/fba [0,4194304] 0 2024-04-12T05:09:15.342 INFO:tasks.workunit.client.1.smithi192.stdout:5/980: mknod d0/c14e 0 2024-04-12T05:09:15.342 INFO:tasks.workunit.client.0.smithi113.stdout:6/901: symlink d8/de/d13/d11/d36/d3d/d4c/d26/d7d/l13f 0 2024-04-12T05:09:15.344 INFO:tasks.workunit.client.0.smithi113.stdout:6/902: stat d8/de/d13/df0/d66/dc4/c10a 0 2024-04-12T05:09:15.344 INFO:tasks.workunit.client.0.smithi113.stdout:7/956: dread d0/dc/d50/d123/d12d/d78/daa/fa6 [0,4194304] 0 2024-04-12T05:09:15.346 INFO:tasks.workunit.client.1.smithi192.stdout:8/902: rmdir d6/d10/d46/d5e/d5f/d31 39 2024-04-12T05:09:15.346 INFO:tasks.workunit.client.0.smithi113.stdout:6/903: fsync d8/de/d13/d11/d36/d89/fe1 0 2024-04-12T05:09:15.351 INFO:tasks.workunit.client.0.smithi113.stdout:7/957: getdents d0/dc/da9/d5c/d9a 0 2024-04-12T05:09:15.352 INFO:tasks.workunit.client.1.smithi192.stdout:4/931: write d1/d54/d55/fa6 [3858742,92671] 0 2024-04-12T05:09:15.352 INFO:tasks.workunit.client.1.smithi192.stdout:3/956: dread - d5/d13/d3a/d8b/da2/de5/f10c zero size 2024-04-12T05:09:15.355 INFO:tasks.workunit.client.1.smithi192.stdout:4/932: stat d1/d54/d82/d71/d9e/d10b/de9/d115 0 2024-04-12T05:09:15.356 INFO:tasks.workunit.client.1.smithi192.stdout:5/981: getdents d0/d15/d32/d69/dc8 0 2024-04-12T05:09:15.357 INFO:tasks.workunit.client.1.smithi192.stdout:3/957: mknod d5/d13/d3a/d4a/c15d 0 2024-04-12T05:09:15.358 INFO:tasks.workunit.client.1.smithi192.stdout:8/903: dread d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d98/fd1 [0,4194304] 0 2024-04-12T05:09:15.359 INFO:tasks.workunit.client.1.smithi192.stdout:4/933: creat d1/d54/d55/dc3/f141 x:0 0 0 2024-04-12T05:09:15.359 INFO:tasks.workunit.client.0.smithi113.stdout:7/958: write d0/dc/d50/d123/d12d/d9b/fcb [767200,116103] 0 2024-04-12T05:09:15.360 INFO:tasks.workunit.client.1.smithi192.stdout:0/995: truncate d0/d7b/d9b/d35/d109/d9f/d10a/d15/d1b/f139 582366 0 2024-04-12T05:09:15.364 INFO:tasks.workunit.client.0.smithi113.stdout:6/904: dwrite d8/de/d13/d11/d36/d2c/fa3 [0,4194304] 0 2024-04-12T05:09:15.365 INFO:tasks.workunit.client.1.smithi192.stdout:0/996: fsync d0/d7b/d9b/d4b/f133 0 2024-04-12T05:09:15.366 INFO:tasks.workunit.client.0.smithi113.stdout:7/959: unlink d0/dc/d50/d123/d12d/d9b/f25 0 2024-04-12T05:09:15.367 INFO:tasks.workunit.client.1.smithi192.stdout:8/904: read d6/d21/f23 [615620,20546] 0 2024-04-12T05:09:15.367 INFO:tasks.workunit.client.1.smithi192.stdout:9/764: dread d11/d25/d36/d6e/f3e [0,4194304] 0 2024-04-12T05:09:15.368 INFO:tasks.workunit.client.1.smithi192.stdout:5/982: getdents d0/d15/d3e/dca/dda/db5/dbe/d118 0 2024-04-12T05:09:15.369 INFO:tasks.workunit.client.1.smithi192.stdout:4/934: rename d1/c17 to d1/d1d/d53/d1c/dd1/def/ddd/d114/c142 0 2024-04-12T05:09:15.369 INFO:tasks.workunit.client.0.smithi113.stdout:6/905: mkdir d8/d140 0 2024-04-12T05:09:15.375 INFO:tasks.workunit.client.0.smithi113.stdout:6/906: dwrite d8/f11e [0,4194304] 0 2024-04-12T05:09:15.376 INFO:tasks.workunit.client.1.smithi192.stdout:3/958: dwrite d5/d13/d3a/d70/d8c/de1/d10e/f11d [0,4194304] 0 2024-04-12T05:09:15.376 INFO:tasks.workunit.client.1.smithi192.stdout:9/765: read - d11/d25/d36/d6e/fd2 zero size 2024-04-12T05:09:15.377 INFO:tasks.workunit.client.1.smithi192.stdout:1/902: dwrite d0/d2/d1e/d3f/df2/f3b [0,4194304] 0 2024-04-12T05:09:15.380 INFO:tasks.workunit.client.1.smithi192.stdout:9/766: dread - d11/d25/dd3/fef zero size 2024-04-12T05:09:15.381 INFO:tasks.workunit.client.1.smithi192.stdout:1/903: dread - d0/d2/d1e/d3f/ff9 zero size 2024-04-12T05:09:15.382 INFO:tasks.workunit.client.1.smithi192.stdout:5/983: creat d0/d15/da1/df9/f14f x:0 0 0 2024-04-12T05:09:15.382 INFO:tasks.workunit.client.0.smithi113.stdout:6/907: creat d8/de/d13/d11/d36/d3d/d4c/d31/d83/f141 x:0 0 0 2024-04-12T05:09:15.388 INFO:tasks.workunit.client.1.smithi192.stdout:5/984: dread d0/d15/f79 [0,4194304] 0 2024-04-12T05:09:15.388 INFO:tasks.workunit.client.1.smithi192.stdout:0/997: dread d0/d7b/d9b/d35/d109/d95/db4/f102 [0,4194304] 0 2024-04-12T05:09:15.389 INFO:tasks.workunit.client.1.smithi192.stdout:4/935: creat d1/d54/d82/da4/f143 x:0 0 0 2024-04-12T05:09:15.394 INFO:tasks.workunit.client.0.smithi113.stdout:6/908: dread d8/f22 [0,4194304] 0 2024-04-12T05:09:15.394 INFO:tasks.workunit.client.1.smithi192.stdout:3/959: dwrite d5/d13/d3c/d36/f133 [0,4194304] 0 2024-04-12T05:09:15.401 INFO:tasks.workunit.client.1.smithi192.stdout:0/998: creat d0/d7b/d9b/d35/d109/f154 x:0 0 0 2024-04-12T05:09:15.403 INFO:tasks.workunit.client.1.smithi192.stdout:5/985: getdents d0/d15/d3e/dca/dda/db5/dbe/d118 0 2024-04-12T05:09:15.404 INFO:tasks.workunit.client.0.smithi113.stdout:6/909: dwrite d8/d1d/fac [0,4194304] 0 2024-04-12T05:09:15.405 INFO:tasks.workunit.client.1.smithi192.stdout:0/999: dread - d0/d7b/d9b/d35/d109/d9f/d10a/d67/db1/f10f zero size 2024-04-12T05:09:15.406 INFO:tasks.workunit.client.1.smithi192.stdout:5/986: fdatasync d0/d15/d3e/dca/dda/deb/d113/d127/f141 0 2024-04-12T05:09:15.411 INFO:tasks.workunit.client.1.smithi192.stdout:5/987: rmdir d0/d15/d3e/dca/dda/db5 39 2024-04-12T05:09:15.411 INFO:tasks.workunit.client.1.smithi192.stdout:8/905: write d6/d10/d46/d5e/d5f/d31/d80/db1/fb6 [204339,54022] 0 2024-04-12T05:09:15.414 INFO:tasks.workunit.client.1.smithi192.stdout:8/906: write d6/d1f/d9c/fd0 [74962,117998] 0 2024-04-12T05:09:15.414 INFO:tasks.workunit.client.0.smithi113.stdout:7/960: dread d0/dc/f42 [0,4194304] 0 2024-04-12T05:09:15.417 INFO:tasks.workunit.client.1.smithi192.stdout:5/988: chown d0/d15/d3e/dca/dda/deb/d113/d127/c11e 24436276 1 2024-04-12T05:09:15.420 INFO:tasks.workunit.client.0.smithi113.stdout:7/961: write d0/f12 [646984,43687] 0 2024-04-12T05:09:15.420 INFO:tasks.workunit.client.1.smithi192.stdout:8/907: unlink d6/d10/d46/d5e/d5f/l24 0 2024-04-12T05:09:15.421 INFO:tasks.workunit.client.1.smithi192.stdout:1/904: dread d0/d2/d1e/d3f/d56/d9e/fbb [0,4194304] 0 2024-04-12T05:09:15.421 INFO:tasks.workunit.client.0.smithi113.stdout:7/962: fdatasync d0/dc/d50/d123/d12d/d9b/f23 0 2024-04-12T05:09:15.422 INFO:tasks.workunit.client.1.smithi192.stdout:5/989: creat d0/d15/d3e/d120/f150 x:0 0 0 2024-04-12T05:09:15.422 INFO:tasks.workunit.client.0.smithi113.stdout:7/963: truncate d0/dc/da9/f128 843275 0 2024-04-12T05:09:15.424 INFO:tasks.workunit.client.0.smithi113.stdout:7/964: read d0/dc/da9/d5c/d85/fed [567656,119551] 0 2024-04-12T05:09:15.424 INFO:tasks.workunit.client.1.smithi192.stdout:4/936: dwrite d1/d1d/d53/d1c/f6d [4194304,4194304] 0 2024-04-12T05:09:15.428 INFO:tasks.workunit.client.1.smithi192.stdout:8/908: mkdir d6/d10/d46/d5e/d5f/d31/d61/d9d/db5/d118 0 2024-04-12T05:09:15.430 INFO:tasks.workunit.client.1.smithi192.stdout:1/905: fdatasync d0/d2/d1e/d3f/df2/d21/f47 0 2024-04-12T05:09:15.431 INFO:tasks.workunit.client.1.smithi192.stdout:3/960: dread d5/fd [0,4194304] 0 2024-04-12T05:09:15.431 INFO:tasks.workunit.client.1.smithi192.stdout:4/937: creat d1/d54/d82/da4/f144 x:0 0 0 2024-04-12T05:09:15.433 INFO:tasks.workunit.client.0.smithi113.stdout:7/965: read d0/dd/d11/f5e [748707,129416] 0 2024-04-12T05:09:15.435 INFO:tasks.workunit.client.0.smithi113.stdout:7/966: chown d0/dc/d50/d123/d12d/d78/daa 1890696973 1 2024-04-12T05:09:15.436 INFO:tasks.workunit.client.1.smithi192.stdout:4/938: unlink d1/d54/d82/da4/f144 0 2024-04-12T05:09:15.438 INFO:tasks.workunit.client.0.smithi113.stdout:7/967: truncate d0/dc/d50/d123/d12d/fa 1722404 0 2024-04-12T05:09:15.439 INFO:tasks.workunit.client.1.smithi192.stdout:5/990: dread d0/f13 [0,4194304] 0 2024-04-12T05:09:15.440 INFO:tasks.workunit.client.0.smithi113.stdout:7/968: mkdir d0/dd/d95/dd4/d13f 0 2024-04-12T05:09:15.442 INFO:tasks.workunit.client.1.smithi192.stdout:8/909: dwrite d6/d10/d109/f30 [4194304,4194304] 0 2024-04-12T05:09:15.442 INFO:tasks.workunit.client.1.smithi192.stdout:4/939: write d1/d1d/d53/f89 [2803415,123600] 0 2024-04-12T05:09:15.443 INFO:tasks.workunit.client.1.smithi192.stdout:1/906: write d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d74/d90/d108/f42 [527369,84962] 0 2024-04-12T05:09:15.445 INFO:tasks.workunit.client.1.smithi192.stdout:5/991: mkdir d0/d15/d32/d69/d7d/d151 0 2024-04-12T05:09:15.445 INFO:tasks.workunit.client.0.smithi113.stdout:7/969: link d0/dc/da9/d5c/ddf/l100 d0/dc/d50/d123/d12d/d78/daa/l140 0 2024-04-12T05:09:15.447 INFO:tasks.workunit.client.1.smithi192.stdout:1/907: dread - d0/d1b/f104 zero size 2024-04-12T05:09:15.447 INFO:tasks.workunit.client.1.smithi192.stdout:4/940: write d1/d1d/d53/db2/f130 [322642,108133] 0 2024-04-12T05:09:15.449 INFO:tasks.workunit.client.1.smithi192.stdout:8/910: rmdir d6/d10/d6c/d9e 39 2024-04-12T05:09:15.450 INFO:tasks.workunit.client.1.smithi192.stdout:1/908: chown d0/d29/dbf/f11f 41 1 2024-04-12T05:09:15.451 INFO:tasks.workunit.client.1.smithi192.stdout:5/992: dread - d0/d15/d32/db2/f115 zero size 2024-04-12T05:09:15.451 INFO:tasks.workunit.client.0.smithi113.stdout:7/970: link d0/dc/d50/d123/d12d/d9b/f55 d0/d6c/f141 0 2024-04-12T05:09:15.453 INFO:tasks.workunit.client.1.smithi192.stdout:5/993: chown d0/d15/d32/d69 84 1 2024-04-12T05:09:15.454 INFO:tasks.workunit.client.1.smithi192.stdout:8/911: truncate d6/d10/d109/f1e 9176145 0 2024-04-12T05:09:15.454 INFO:tasks.workunit.client.0.smithi113.stdout:7/971: fdatasync d0/dc/d50/d123/d12d/d9b/fb4 0 2024-04-12T05:09:15.457 INFO:tasks.workunit.client.1.smithi192.stdout:4/941: dwrite d1/d54/d82/da4/f143 [0,4194304] 0 2024-04-12T05:09:15.457 INFO:tasks.workunit.client.1.smithi192.stdout:5/994: creat d0/d15/da1/f152 x:0 0 0 2024-04-12T05:09:15.466 INFO:tasks.workunit.client.1.smithi192.stdout:3/961: dread d5/d13/d3a/d4a/d61/d79/f80 [0,4194304] 0 2024-04-12T05:09:15.466 INFO:tasks.workunit.client.1.smithi192.stdout:8/912: creat d6/d1f/f119 x:0 0 0 2024-04-12T05:09:15.466 INFO:tasks.workunit.client.0.smithi113.stdout:7/972: rename d0/dc/d50/d123/d12d/d78/f31 to d0/dc/d50/d123/d12d/d9b/d9f/db3/f142 0 2024-04-12T05:09:15.468 INFO:tasks.workunit.client.0.smithi113.stdout:7/973: write d0/dd/d95/dd4/f107 [4166448,121822] 0 2024-04-12T05:09:15.479 INFO:tasks.workunit.client.1.smithi192.stdout:4/942: dwrite d1/d54/f61 [0,4194304] 0 2024-04-12T05:09:15.481 INFO:tasks.workunit.client.1.smithi192.stdout:5/995: dread d0/d15/d24/d29/f12c [0,4194304] 0 2024-04-12T05:09:15.491 INFO:tasks.workunit.client.1.smithi192.stdout:8/913: dwrite d6/d10/d46/d5e/d5f/d37/d4f/d4c/f55 [0,4194304] 0 2024-04-12T05:09:15.491 INFO:tasks.workunit.client.1.smithi192.stdout:3/962: dwrite d5/d13/d3c/f25 [0,4194304] 0 2024-04-12T05:09:15.493 INFO:tasks.workunit.client.1.smithi192.stdout:8/914: chown d6/d10/d46/f96 297 1 2024-04-12T05:09:15.493 INFO:tasks.workunit.client.1.smithi192.stdout:4/943: dread d1/d54/d55/d64/fd4 [4194304,4194304] 0 2024-04-12T05:09:15.494 INFO:tasks.workunit.client.1.smithi192.stdout:4/944: stat d1/f29 0 2024-04-12T05:09:15.498 INFO:tasks.workunit.client.1.smithi192.stdout:4/945: readlink d1/d54/d82/da4/da8/lb8 0 2024-04-12T05:09:15.498 INFO:tasks.workunit.client.1.smithi192.stdout:8/915: write d6/d10/d46/d5e/d5f/f88 [2835465,18146] 0 2024-04-12T05:09:15.503 INFO:tasks.workunit.client.1.smithi192.stdout:3/963: creat d5/d13/d3a/d8b/da2/d105/f15e x:0 0 0 2024-04-12T05:09:15.504 INFO:tasks.workunit.client.1.smithi192.stdout:3/964: fdatasync d5/d13/d3a/d70/d8c/de1/fe0 0 2024-04-12T05:09:15.506 INFO:tasks.workunit.client.1.smithi192.stdout:3/965: write d5/d13/d3a/d8b/da2/d105/f15e [135409,94085] 0 2024-04-12T05:09:15.508 INFO:tasks.workunit.client.0.smithi113.stdout:4/952: sync 2024-04-12T05:09:15.509 INFO:tasks.workunit.client.1.smithi192.stdout:4/946: creat d1/d1d/d53/d1c/d95/dc6/dd5/de5/f145 x:0 0 0 2024-04-12T05:09:15.510 INFO:tasks.workunit.client.1.smithi192.stdout:3/966: read d5/d13/d3a/d70/d8c/fb0 [2366341,78315] 0 2024-04-12T05:09:15.512 INFO:tasks.workunit.client.1.smithi192.stdout:3/967: read d5/d13/d3a/d70/d8c/d8d/dbc/f110 [846298,3640] 0 2024-04-12T05:09:15.513 INFO:tasks.workunit.client.1.smithi192.stdout:4/947: mkdir d1/d1d/d53/d1c/dd1/def/ddd/d146 0 2024-04-12T05:09:15.514 INFO:tasks.workunit.client.1.smithi192.stdout:8/916: truncate d6/d10/d46/d5e/d5f/fea 1835404 0 2024-04-12T05:09:15.514 INFO:tasks.workunit.client.0.smithi113.stdout:7/974: mknod d0/dc/da9/c143 0 2024-04-12T05:09:15.515 INFO:tasks.workunit.client.0.smithi113.stdout:7/975: write d0/d52/f117 [1983430,9911] 0 2024-04-12T05:09:15.515 INFO:tasks.workunit.client.1.smithi192.stdout:8/917: fsync d6/d10/d46/d5e/d5f/d37/feb 0 2024-04-12T05:09:15.516 INFO:tasks.workunit.client.1.smithi192.stdout:3/968: creat d5/d13/d3a/d4a/d61/d9f/deb/f15f x:0 0 0 2024-04-12T05:09:15.516 INFO:tasks.workunit.client.0.smithi113.stdout:4/953: stat da/d3c/d43/dbc/c111 0 2024-04-12T05:09:15.517 INFO:tasks.workunit.client.1.smithi192.stdout:4/948: creat d1/d54/d82/d71/d9e/dc4/dfe/d109/f147 x:0 0 0 2024-04-12T05:09:15.518 INFO:tasks.workunit.client.0.smithi113.stdout:7/976: creat d0/dc/d50/d123/f144 x:0 0 0 2024-04-12T05:09:15.519 INFO:tasks.workunit.client.0.smithi113.stdout:7/977: readlink d0/l8 0 2024-04-12T05:09:15.519 INFO:tasks.workunit.client.1.smithi192.stdout:8/918: symlink d6/d10/d6c/d9e/dde/dfc/l11a 0 2024-04-12T05:09:15.520 INFO:tasks.workunit.client.1.smithi192.stdout:4/949: mknod d1/d1d/d53/d1c/de1/c148 0 2024-04-12T05:09:15.521 INFO:tasks.workunit.client.1.smithi192.stdout:8/919: dread d6/d10/d46/d5e/d5f/d37/d4f/d4c/f94 [0,4194304] 0 2024-04-12T05:09:15.525 INFO:tasks.workunit.client.0.smithi113.stdout:4/954: dwrite da/d3c/da5/f10e [0,4194304] 0 2024-04-12T05:09:15.525 INFO:tasks.workunit.client.1.smithi192.stdout:4/950: mkdir d1/d54/d82/d149 0 2024-04-12T05:09:15.527 INFO:tasks.workunit.client.1.smithi192.stdout:8/920: fsync d6/d10/d46/d5e/d5f/d37/d4f/fbe 0 2024-04-12T05:09:15.528 INFO:tasks.workunit.client.1.smithi192.stdout:3/969: write d5/d13/d3a/d8b/fc2 [2315265,108647] 0 2024-04-12T05:09:15.528 INFO:tasks.workunit.client.0.smithi113.stdout:7/978: dread d0/dc/d10b/ff7 [4194304,4194304] 0 2024-04-12T05:09:15.530 INFO:tasks.workunit.client.0.smithi113.stdout:4/955: creat da/d24/d7e/d124/d1b/d13b/d112/f13c x:0 0 0 2024-04-12T05:09:15.530 INFO:tasks.workunit.client.0.smithi113.stdout:4/956: stat da/led 0 2024-04-12T05:09:15.532 INFO:tasks.workunit.client.1.smithi192.stdout:4/951: symlink d1/d1d/d53/d1c/d95/dc6/dd5/de5/ded/d119/l14a 0 2024-04-12T05:09:15.534 INFO:tasks.workunit.client.0.smithi113.stdout:4/957: read da/d24/d7e/ff1 [1833409,66239] 0 2024-04-12T05:09:15.534 INFO:tasks.workunit.client.1.smithi192.stdout:8/921: mknod d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d98/c11b 0 2024-04-12T05:09:15.536 INFO:tasks.workunit.client.0.smithi113.stdout:7/979: rename d0/dc/da9/d5c/l12f to d0/dc/da9/d5c/d85/l145 0 2024-04-12T05:09:15.536 INFO:tasks.workunit.client.1.smithi192.stdout:4/952: mknod d1/d54/d82/da4/de8/c14b 0 2024-04-12T05:09:15.539 INFO:tasks.workunit.client.0.smithi113.stdout:3/992: sync 2024-04-12T05:09:15.539 INFO:tasks.workunit.client.0.smithi113.stdout:7/980: fsync d0/dd/d11/f5e 0 2024-04-12T05:09:15.541 INFO:tasks.workunit.client.1.smithi192.stdout:8/922: dwrite d6/d1f/f119 [0,4194304] 0 2024-04-12T05:09:15.542 INFO:tasks.workunit.client.0.smithi113.stdout:4/958: dwrite da/d3c/d83/d47/d10f/d90/f128 [0,4194304] 0 2024-04-12T05:09:15.543 INFO:tasks.workunit.client.0.smithi113.stdout:7/981: dread - d0/dc/d50/d123/d12d/d78/df2/f139 zero size 2024-04-12T05:09:15.545 INFO:tasks.workunit.client.0.smithi113.stdout:3/993: creat d0/d13/d15/d28/d2a/d34/d5f/d7b/dc1/f149 x:0 0 0 2024-04-12T05:09:15.545 INFO:tasks.workunit.client.0.smithi113.stdout:7/982: dread - d0/dc/d50/d123/d12d/d24/dd2/f114 zero size 2024-04-12T05:09:15.545 INFO:tasks.workunit.client.1.smithi192.stdout:4/953: rename d1/d54/d82/d71/d9e/d10b/de9/d11e to d1/d1d/d53/d1c/d84/dbb/df7/dfc/d14c 0 2024-04-12T05:09:15.547 INFO:tasks.workunit.client.0.smithi113.stdout:7/983: truncate d0/dc/f68 1871660 0 2024-04-12T05:09:15.547 INFO:tasks.workunit.client.0.smithi113.stdout:3/994: read d0/d13/d47/d94/dac/f112 [124927,59830] 0 2024-04-12T05:09:15.547 INFO:tasks.workunit.client.1.smithi192.stdout:4/954: chown d1/d1d/d53/d1c/d95/dc6/dd5/de5/f145 640648963 1 2024-04-12T05:09:15.550 INFO:tasks.workunit.client.1.smithi192.stdout:8/923: symlink d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d98/d115/l11c 0 2024-04-12T05:09:15.551 INFO:tasks.workunit.client.0.smithi113.stdout:7/984: fdatasync d0/d6c/f141 0 2024-04-12T05:09:15.553 INFO:tasks.workunit.client.0.smithi113.stdout:3/995: mknod d0/d13/d47/d94/dac/c14a 0 2024-04-12T05:09:15.553 INFO:tasks.workunit.client.1.smithi192.stdout:8/924: fdatasync d6/d10/d46/f77 0 2024-04-12T05:09:15.556 INFO:tasks.workunit.client.1.smithi192.stdout:8/925: creat d6/d21/f11d x:0 0 0 2024-04-12T05:09:15.558 INFO:tasks.workunit.client.1.smithi192.stdout:4/955: dwrite d1/d54/d82/d71/fb0 [0,4194304] 0 2024-04-12T05:09:15.561 INFO:tasks.workunit.client.0.smithi113.stdout:7/985: read d0/dc/d50/d123/d12d/d1a/fb8 [1929901,125414] 0 2024-04-12T05:09:15.562 INFO:tasks.workunit.client.0.smithi113.stdout:3/996: getdents d0/d13/d15/d28/d2a/d8f/ddd 0 2024-04-12T05:09:15.565 INFO:tasks.workunit.client.1.smithi192.stdout:4/956: read d1/d1d/d53/d1c/fcb [97364,101686] 0 2024-04-12T05:09:15.566 INFO:tasks.workunit.client.0.smithi113.stdout:3/997: mknod d0/d13/d47/d94/dac/c14b 0 2024-04-12T05:09:15.569 INFO:tasks.workunit.client.0.smithi113.stdout:3/998: symlink d0/d13/d47/d4f/dca/d12e/l14c 0 2024-04-12T05:09:15.569 INFO:tasks.workunit.client.1.smithi192.stdout:4/957: rmdir d1/d54/d82/d71/d9e/d10b/de9/d115 0 2024-04-12T05:09:15.572 INFO:tasks.workunit.client.1.smithi192.stdout:4/958: creat d1/d54/f14d x:0 0 0 2024-04-12T05:09:15.572 INFO:tasks.workunit.client.0.smithi113.stdout:3/999: unlink d0/d13/d15/d28/f6f 0 2024-04-12T05:09:15.582 INFO:tasks.workunit.client.1.smithi192.stdout:8/926: write d6/d10/d46/d5e/faf [656087,53614] 0 2024-04-12T05:09:15.585 INFO:tasks.workunit.client.0.smithi113.stdout:7/986: dwrite d0/d6c/f141 [0,4194304] 0 2024-04-12T05:09:15.586 INFO:tasks.workunit.client.1.smithi192.stdout:8/927: symlink d6/d10/d46/d5e/d5f/l11e 0 2024-04-12T05:09:15.591 INFO:tasks.workunit.client.1.smithi192.stdout:8/928: dread d6/d10/d46/d5e/d5f/d31/d80/db1/fe5 [0,4194304] 0 2024-04-12T05:09:15.591 INFO:tasks.workunit.client.1.smithi192.stdout:4/959: dwrite d1/d54/d82/d71/d9e/fa0 [0,4194304] 0 2024-04-12T05:09:15.592 INFO:tasks.workunit.client.0.smithi113.stdout:7/987: dwrite d0/dc/d50/d123/f144 [0,4194304] 0 2024-04-12T05:09:15.598 INFO:tasks.workunit.client.1.smithi192.stdout:4/960: fdatasync d1/d54/f6e 0 2024-04-12T05:09:15.601 INFO:tasks.workunit.client.1.smithi192.stdout:8/929: symlink d6/d10/d46/d5e/d5f/d31/d61/d9d/d10f/l11f 0 2024-04-12T05:09:15.605 INFO:tasks.workunit.client.1.smithi192.stdout:4/961: creat d1/d1d/d53/d1c/d95/dc6/dd5/f14e x:0 0 0 2024-04-12T05:09:15.605 INFO:tasks.workunit.client.1.smithi192.stdout:4/962: fsync d1/f33 0 2024-04-12T05:09:15.607 INFO:tasks.workunit.client.1.smithi192.stdout:4/963: fsync d1/d54/d82/d71/d9e/dc4/dd9/fee 0 2024-04-12T05:09:15.609 INFO:tasks.workunit.client.1.smithi192.stdout:4/964: dread - d1/d54/d82/da4/da8/f138 zero size 2024-04-12T05:09:15.611 INFO:tasks.workunit.client.1.smithi192.stdout:8/930: dwrite d6/d10/d46/d5e/d5f/d31/d80/db1/fcb [0,4194304] 0 2024-04-12T05:09:15.614 INFO:tasks.workunit.client.1.smithi192.stdout:4/965: symlink d1/d1d/d53/d1c/dd1/def/ddd/d114/l14f 0 2024-04-12T05:09:15.615 INFO:tasks.workunit.client.1.smithi192.stdout:8/931: symlink d6/d10/d46/d5e/d5f/d31/d80/l120 0 2024-04-12T05:09:15.615 INFO:tasks.workunit.client.1.smithi192.stdout:4/966: chown d1/d1d/d53/d1c/f65 13334 1 2024-04-12T05:09:15.619 INFO:tasks.workunit.client.1.smithi192.stdout:4/967: mkdir d1/d1d/d53/d11a/ddb/d150 0 2024-04-12T05:09:15.625 INFO:tasks.workunit.client.0.smithi113.stdout:7/988: dwrite d0/dd/f73 [0,4194304] 0 2024-04-12T05:09:15.628 INFO:tasks.workunit.client.0.smithi113.stdout:7/989: mkdir d0/dc/d50/d123/d12d/d78/daa/d146 0 2024-04-12T05:09:15.630 INFO:tasks.workunit.client.0.smithi113.stdout:7/990: mknod d0/dc/de7/c147 0 2024-04-12T05:09:15.630 INFO:tasks.workunit.client.0.smithi113.stdout:7/991: stat d0/dc/d50/d123/d12d/c3d 0 2024-04-12T05:09:15.637 INFO:tasks.workunit.client.1.smithi192.stdout:8/932: dread f3 [0,4194304] 0 2024-04-12T05:09:15.638 INFO:tasks.workunit.client.0.smithi113.stdout:7/992: write d0/dd/ffd [948305,72283] 0 2024-04-12T05:09:15.639 INFO:tasks.workunit.client.0.smithi113.stdout:7/993: stat d0/dc/d50/f102 0 2024-04-12T05:09:15.641 INFO:tasks.workunit.client.1.smithi192.stdout:4/968: dread d1/d1d/d53/d1c/d95/dc6/ff9 [4194304,4194304] 0 2024-04-12T05:09:15.642 INFO:tasks.workunit.client.0.smithi113.stdout:7/994: mknod d0/dc/d50/d123/d12d/d24/c148 0 2024-04-12T05:09:15.645 INFO:tasks.workunit.client.1.smithi192.stdout:4/969: read d1/d54/d82/d71/d9e/d10b/fd7 [2846009,114659] 0 2024-04-12T05:09:15.645 INFO:tasks.workunit.client.0.smithi113.stdout:7/995: unlink d0/fa2 0 2024-04-12T05:09:15.647 INFO:tasks.workunit.client.1.smithi192.stdout:4/970: mknod d1/d54/c151 0 2024-04-12T05:09:15.648 INFO:tasks.workunit.client.0.smithi113.stdout:7/996: chown d0/dc/da9/d5c/d85/l145 9773973 1 2024-04-12T05:09:15.653 INFO:tasks.workunit.client.1.smithi192.stdout:8/933: dwrite d6/d10/d46/d5e/d5f/d37/d4f/fc6 [0,4194304] 0 2024-04-12T05:09:15.654 INFO:tasks.workunit.client.0.smithi113.stdout:7/997: write d0/dc/d50/d123/d12d/fa [2753945,13457] 0 2024-04-12T05:09:15.655 INFO:tasks.workunit.client.1.smithi192.stdout:4/971: dread d1/d1d/d53/d1c/f66 [0,4194304] 0 2024-04-12T05:09:15.657 INFO:tasks.workunit.client.1.smithi192.stdout:4/972: fdatasync d1/d54/d82/da4/f143 0 2024-04-12T05:09:15.662 INFO:tasks.workunit.client.1.smithi192.stdout:4/973: mknod d1/d1d/d53/d1c/dd1/def/ddd/c152 0 2024-04-12T05:09:15.664 INFO:tasks.workunit.client.1.smithi192.stdout:4/974: rename d1/d1d/d53/d1c to d1/d1d/d53/d1c/d95/d153 22 2024-04-12T05:09:15.665 INFO:tasks.workunit.client.0.smithi113.stdout:6/910: sync 2024-04-12T05:09:15.666 INFO:tasks.workunit.client.0.smithi113.stdout:6/911: write d8/de/d13/d11/d36/d3d/d4c/d26/d108/f10e [1844511,103346] 0 2024-04-12T05:09:15.668 INFO:tasks.workunit.client.0.smithi113.stdout:6/912: chown d8/de/d13/d11/d36/d3d/d4c/d31 68153721 1 2024-04-12T05:09:15.669 INFO:tasks.workunit.client.0.smithi113.stdout:6/913: write d8/f40 [3914545,31671] 0 2024-04-12T05:09:15.670 INFO:tasks.workunit.client.1.smithi192.stdout:8/934: dwrite d6/d10/d46/d5e/d5f/f16 [0,4194304] 0 2024-04-12T05:09:15.671 INFO:tasks.workunit.client.0.smithi113.stdout:7/998: write d0/dd/d11/f4a [3097613,26677] 0 2024-04-12T05:09:15.674 INFO:tasks.workunit.client.0.smithi113.stdout:6/914: rename d8/d1d/l9e to d8/de/d13/df0/d66/dc4/de3/l142 0 2024-04-12T05:09:15.678 INFO:tasks.workunit.client.1.smithi192.stdout:4/975: dwrite d1/d54/d82/d71/d9e/ff8 [0,4194304] 0 2024-04-12T05:09:15.678 INFO:tasks.workunit.client.0.smithi113.stdout:6/915: mknod d8/de/d13/d11/d36/d3d/d93/c143 0 2024-04-12T05:09:15.680 INFO:tasks.workunit.client.0.smithi113.stdout:7/999: read d0/dd/d11/f4a [869898,128334] 0 2024-04-12T05:09:15.682 INFO:tasks.workunit.client.0.smithi113.stdout:6/916: creat d8/de/d13/d11/d36/d53/f144 x:0 0 0 2024-04-12T05:09:15.683 INFO:tasks.workunit.client.1.smithi192.stdout:8/935: dwrite d6/d1f/d6a/d106/ffd [0,4194304] 0 2024-04-12T05:09:15.684 INFO:tasks.workunit.client.1.smithi192.stdout:4/976: creat d1/d1d/d53/d1c/dd1/def/ddd/d146/f154 x:0 0 0 2024-04-12T05:09:15.688 INFO:tasks.workunit.client.0.smithi113.stdout:6/917: truncate d8/de/d13/d11/d36/d89/fe1 1844729 0 2024-04-12T05:09:15.688 INFO:tasks.workunit.client.1.smithi192.stdout:8/936: unlink d6/d10/d46/d5e/d5f/d31/ce0 0 2024-04-12T05:09:15.689 INFO:tasks.workunit.client.1.smithi192.stdout:4/977: mkdir d1/d54/d82/d71/dff/d155 0 2024-04-12T05:09:15.693 INFO:tasks.workunit.client.1.smithi192.stdout:4/978: symlink d1/d54/d55/d64/l156 0 2024-04-12T05:09:15.695 INFO:tasks.workunit.client.1.smithi192.stdout:8/937: getdents d6/d10/d6c/d9e 0 2024-04-12T05:09:15.695 INFO:tasks.workunit.client.0.smithi113.stdout:6/918: unlink d8/de/d13/d11/d36/cad 0 2024-04-12T05:09:15.697 INFO:tasks.workunit.client.1.smithi192.stdout:4/979: fdatasync d1/d1d/d53/d1c/fde 0 2024-04-12T05:09:15.701 INFO:tasks.workunit.client.1.smithi192.stdout:6/917: sync 2024-04-12T05:09:15.701 INFO:tasks.workunit.client.1.smithi192.stdout:7/807: sync 2024-04-12T05:09:15.701 INFO:tasks.workunit.client.1.smithi192.stdout:2/883: sync 2024-04-12T05:09:15.701 INFO:tasks.workunit.client.1.smithi192.stdout:9/767: sync 2024-04-12T05:09:15.703 INFO:tasks.workunit.client.1.smithi192.stdout:8/938: rmdir d6/d10 39 2024-04-12T05:09:15.703 INFO:tasks.workunit.client.0.smithi113.stdout:6/919: write d8/de/d13/d11/d36/d53/db8/f9b [1000663,45642] 0 2024-04-12T05:09:15.703 INFO:tasks.workunit.client.1.smithi192.stdout:2/884: chown d1/d8/d3a/de/d7c/d98/la4 807534 1 2024-04-12T05:09:15.705 INFO:tasks.workunit.client.0.smithi113.stdout:6/920: creat d8/d1d/d13e/f145 x:0 0 0 2024-04-12T05:09:15.706 INFO:tasks.workunit.client.1.smithi192.stdout:6/918: readlink d3/d23/l61 0 2024-04-12T05:09:15.708 INFO:tasks.workunit.client.1.smithi192.stdout:4/980: truncate d1/fb6 1613583 0 2024-04-12T05:09:15.709 INFO:tasks.workunit.client.1.smithi192.stdout:2/885: mknod d1/d2d/d73/de7/c117 0 2024-04-12T05:09:15.710 INFO:tasks.workunit.client.1.smithi192.stdout:7/808: mknod d3/d1d/d5a/d5f/d6a/c112 0 2024-04-12T05:09:15.711 INFO:tasks.workunit.client.0.smithi113.stdout:6/921: write d8/de/d13/d11/d36/d53/db7/fef [1439373,56440] 0 2024-04-12T05:09:15.712 INFO:tasks.workunit.client.1.smithi192.stdout:4/981: dread - d1/d1d/d53/d1c/dd1/def/ddd/dea/f100 zero size 2024-04-12T05:09:15.713 INFO:tasks.workunit.client.1.smithi192.stdout:2/886: chown d1/d8/d17/d19/d60/d59/d68/d105 408 1 2024-04-12T05:09:15.715 INFO:tasks.workunit.client.1.smithi192.stdout:7/809: mknod d3/d1d/c113 0 2024-04-12T05:09:15.715 INFO:tasks.workunit.client.1.smithi192.stdout:8/939: dwrite d6/d10/fef [0,4194304] 0 2024-04-12T05:09:15.715 INFO:tasks.workunit.client.1.smithi192.stdout:2/887: chown d1/d8/f29 240171 1 2024-04-12T05:09:15.718 INFO:tasks.workunit.client.0.smithi113.stdout:6/922: write d8/de/d13/d11/d36/d3d/d4c/d26/d64/fb0 [255582,46785] 0 2024-04-12T05:09:15.718 INFO:tasks.workunit.client.1.smithi192.stdout:8/940: readlink d6/d10/d46/d5e/d5f/d37/d4f/d51/l86 0 2024-04-12T05:09:15.718 INFO:tasks.workunit.client.1.smithi192.stdout:7/810: mkdir d3/d1d/d48/d114 0 2024-04-12T05:09:15.719 INFO:tasks.workunit.client.1.smithi192.stdout:4/982: dread - d1/d1d/d53/d1c/d95/dc6/dd5/f12f zero size 2024-04-12T05:09:15.719 INFO:tasks.workunit.client.1.smithi192.stdout:9/768: truncate d11/d25/d36/d3d/fb3 3607123 0 2024-04-12T05:09:15.723 INFO:tasks.workunit.client.1.smithi192.stdout:4/983: creat d1/d1d/d53/d1c/dd1/def/ddd/d146/f157 x:0 0 0 2024-04-12T05:09:15.724 INFO:tasks.workunit.client.1.smithi192.stdout:9/769: creat d11/d25/d36/d6e/d33/da2/ddf/f11a x:0 0 0 2024-04-12T05:09:15.725 INFO:tasks.workunit.client.1.smithi192.stdout:8/941: rename d6/d10/d46/d5e/lc2 to d6/d10/d46/d5e/d5f/d31/d80/d10c/l121 0 2024-04-12T05:09:15.726 INFO:tasks.workunit.client.1.smithi192.stdout:8/942: chown d6/d1f/d9c/fee 2411 1 2024-04-12T05:09:15.727 INFO:tasks.workunit.client.1.smithi192.stdout:7/811: rmdir d3/d7/d15/d22/df3 0 2024-04-12T05:09:15.728 INFO:tasks.workunit.client.1.smithi192.stdout:4/984: mknod d1/d49/c158 0 2024-04-12T05:09:15.731 INFO:tasks.workunit.client.1.smithi192.stdout:5/996: sync 2024-04-12T05:09:15.732 INFO:tasks.workunit.client.1.smithi192.stdout:3/970: sync 2024-04-12T05:09:15.732 INFO:tasks.workunit.client.1.smithi192.stdout:1/909: sync 2024-04-12T05:09:15.736 INFO:tasks.workunit.client.1.smithi192.stdout:8/943: creat d6/d10/d46/d5e/d5f/d37/d4f/d51/f122 x:0 0 0 2024-04-12T05:09:15.736 INFO:tasks.workunit.client.1.smithi192.stdout:3/971: mknod d5/d13/d3a/d8b/c160 0 2024-04-12T05:09:15.745 INFO:tasks.workunit.client.0.smithi113.stdout:6/923: dread d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8f [0,4194304] 0 2024-04-12T05:09:15.749 INFO:tasks.workunit.client.1.smithi192.stdout:5/997: dread d0/d15/d24/d29/f85 [0,4194304] 0 2024-04-12T05:09:15.750 INFO:tasks.workunit.client.0.smithi113.stdout:6/924: dread d8/de/d13/d11/d36/d53/db8/f9d [0,4194304] 0 2024-04-12T05:09:15.753 INFO:tasks.workunit.client.0.smithi113.stdout:6/925: chown d8/de/d13/d11/d36/d3d/d4c/d26/d5b/fb2 467 1 2024-04-12T05:09:15.755 INFO:tasks.workunit.client.1.smithi192.stdout:9/770: dwrite d11/d25/d36/d3d/f5e [0,4194304] 0 2024-04-12T05:09:15.756 INFO:tasks.workunit.client.0.smithi113.stdout:6/926: read d8/de/d13/d11/d36/d3d/d4c/d26/f114 [1662630,121009] 0 2024-04-12T05:09:15.756 INFO:tasks.workunit.client.1.smithi192.stdout:7/812: dwrite d3/d7/fb4 [0,4194304] 0 2024-04-12T05:09:15.759 INFO:tasks.workunit.client.1.smithi192.stdout:9/771: readlink d11/d25/dd3/d10d/dce/l10e 0 2024-04-12T05:09:15.759 INFO:tasks.workunit.client.1.smithi192.stdout:7/813: truncate d3/d1d/f9f 5798937 0 2024-04-12T05:09:15.769 INFO:tasks.workunit.client.0.smithi113.stdout:6/927: dwrite d8/de/d13/d11/d36/d53/fe2 [0,4194304] 0 2024-04-12T05:09:15.772 INFO:tasks.workunit.client.0.smithi113.stdout:4/959: sync 2024-04-12T05:09:15.774 INFO:tasks.workunit.client.0.smithi113.stdout:6/928: symlink d8/d45/l146 0 2024-04-12T05:09:15.775 INFO:tasks.workunit.client.0.smithi113.stdout:6/929: stat d8/de/dd5/de7/d76/c11b 0 2024-04-12T05:09:15.780 INFO:tasks.workunit.client.0.smithi113.stdout:6/930: unlink d8/c14 0 2024-04-12T05:09:15.784 INFO:tasks.workunit.client.0.smithi113.stdout:6/931: symlink d8/de/d13/d11/d36/d3d/d4c/d26/d64/l147 0 2024-04-12T05:09:15.786 INFO:tasks.workunit.client.1.smithi192.stdout:5/998: creat d0/d15/d32/db2/f153 x:0 0 0 2024-04-12T05:09:15.786 INFO:tasks.workunit.client.1.smithi192.stdout:7/814: fdatasync d3/f21 0 2024-04-12T05:09:15.786 INFO:tasks.workunit.client.1.smithi192.stdout:8/944: dread d6/d10/f78 [0,4194304] 0 2024-04-12T05:09:15.786 INFO:tasks.workunit.client.1.smithi192.stdout:5/999: creat d0/d15/d64/f154 x:0 0 0 2024-04-12T05:09:15.786 INFO:tasks.workunit.client.1.smithi192.stdout:3/972: dread d5/d13/f18 [4194304,4194304] 0 2024-04-12T05:09:15.786 INFO:tasks.workunit.client.1.smithi192.stdout:3/973: readlink d5/d13/d3a/l69 0 2024-04-12T05:09:15.787 INFO:tasks.workunit.client.1.smithi192.stdout:7/815: dwrite d3/d7/d15/d1a/fd7 [0,4194304] 0 2024-04-12T05:09:15.787 INFO:tasks.workunit.client.1.smithi192.stdout:9/772: dread f3 [0,4194304] 0 2024-04-12T05:09:15.787 INFO:tasks.workunit.client.1.smithi192.stdout:1/910: truncate d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/da7/dd7/d126/ff4 556127 0 2024-04-12T05:09:15.789 INFO:tasks.workunit.client.0.smithi113.stdout:6/932: rename d8/de/d13/d11/d36/d3d/d4c/d26/d64/cdf to d8/d45/d124/c148 0 2024-04-12T05:09:15.790 INFO:tasks.workunit.client.1.smithi192.stdout:9/773: readlink d11/d25/d60/d8c/leb 0 2024-04-12T05:09:15.790 INFO:tasks.workunit.client.1.smithi192.stdout:1/911: chown d0/d2/d1e/d3f/df2/d21/d62/dbc/fc2 638356 1 2024-04-12T05:09:15.796 INFO:tasks.workunit.client.1.smithi192.stdout:1/912: creat d0/d2/f12c x:0 0 0 2024-04-12T05:09:15.798 INFO:tasks.workunit.client.0.smithi113.stdout:6/933: rmdir d8/de/d13/d11/d36/d89/d10d 0 2024-04-12T05:09:15.798 INFO:tasks.workunit.client.1.smithi192.stdout:7/816: creat d3/d11/d68/f115 x:0 0 0 2024-04-12T05:09:15.799 INFO:tasks.workunit.client.0.smithi113.stdout:6/934: chown d8/de/d13/d11/d36/d3d/d4c/d26/fdd 236516814 1 2024-04-12T05:09:15.801 INFO:tasks.workunit.client.1.smithi192.stdout:1/913: creat d0/d2/d1e/d3f/d78/f12d x:0 0 0 2024-04-12T05:09:15.805 INFO:tasks.workunit.client.1.smithi192.stdout:7/817: dwrite d3/d1d/f41 [0,4194304] 0 2024-04-12T05:09:15.807 INFO:tasks.workunit.client.1.smithi192.stdout:8/945: truncate d6/d10/d46/d5e/d6e/fbf 587262 0 2024-04-12T05:09:15.814 INFO:tasks.workunit.client.1.smithi192.stdout:1/914: truncate d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d74/d90/fc4 234517 0 2024-04-12T05:09:15.815 INFO:tasks.workunit.client.1.smithi192.stdout:8/946: unlink d6/d10/d46/d5e/d5f/d31/d80/fcd 0 2024-04-12T05:09:15.821 INFO:tasks.workunit.client.1.smithi192.stdout:8/947: readlink d6/d10/d46/d5e/d5f/l11e 0 2024-04-12T05:09:15.825 INFO:tasks.workunit.client.1.smithi192.stdout:3/974: dwrite d5/d13/d3a/d4e/d57/f6e [0,4194304] 0 2024-04-12T05:09:15.826 INFO:tasks.workunit.client.1.smithi192.stdout:7/818: dwrite d3/d1d/f9f [0,4194304] 0 2024-04-12T05:09:15.827 INFO:tasks.workunit.client.1.smithi192.stdout:9/774: dwrite d11/d15/fca [0,4194304] 0 2024-04-12T05:09:15.828 INFO:tasks.workunit.client.1.smithi192.stdout:1/915: chown d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d7c/df7 39 1 2024-04-12T05:09:15.830 INFO:tasks.workunit.client.1.smithi192.stdout:9/775: fdatasync d11/d25/d36/d6e/d33/df3/ffa 0 2024-04-12T05:09:15.831 INFO:tasks.workunit.client.1.smithi192.stdout:7/819: fdatasync d3/d7/d15/d1a/f108 0 2024-04-12T05:09:15.833 INFO:tasks.workunit.client.1.smithi192.stdout:9/776: fdatasync d11/d25/d36/d6e/d71/f95 0 2024-04-12T05:09:15.836 INFO:tasks.workunit.client.1.smithi192.stdout:1/916: write d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d74/d90/d108/fa6 [1047114,69958] 0 2024-04-12T05:09:15.839 INFO:tasks.workunit.client.1.smithi192.stdout:7/820: chown d3/d7/d14/d1f/d76/l10c 396073 1 2024-04-12T05:09:15.842 INFO:tasks.workunit.client.1.smithi192.stdout:8/948: mkdir d6/d10/d46/d5e/d5f/d37/d123 0 2024-04-12T05:09:15.847 INFO:tasks.workunit.client.1.smithi192.stdout:7/821: rename d3/d1d/f41 to d3/d1d/d5a/d5f/d6a/de9/f116 0 2024-04-12T05:09:15.847 INFO:tasks.workunit.client.1.smithi192.stdout:8/949: creat d6/d10/d46/d5e/d5f/d37/d7b/f124 x:0 0 0 2024-04-12T05:09:15.850 INFO:tasks.workunit.client.1.smithi192.stdout:7/822: symlink d3/d7/d15/d1a/l117 0 2024-04-12T05:09:15.851 INFO:tasks.workunit.client.1.smithi192.stdout:3/975: getdents d5/d13/d94/dbd/d117 0 2024-04-12T05:09:15.852 INFO:tasks.workunit.client.1.smithi192.stdout:1/917: getdents d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d74 0 2024-04-12T05:09:15.854 INFO:tasks.workunit.client.1.smithi192.stdout:3/976: write d5/d13/d3a/d70/d8c/d8d/dbc/fea [1021574,65084] 0 2024-04-12T05:09:15.856 INFO:tasks.workunit.client.1.smithi192.stdout:7/823: creat d3/d1d/d4a/d5e/d6d/da3/dce/dee/f118 x:0 0 0 2024-04-12T05:09:15.857 INFO:tasks.workunit.client.1.smithi192.stdout:8/950: getdents d6/d1f/d5d 0 2024-04-12T05:09:15.861 INFO:tasks.workunit.client.1.smithi192.stdout:7/824: mknod d3/d1d/d5a/d5f/c119 0 2024-04-12T05:09:15.862 INFO:tasks.workunit.client.1.smithi192.stdout:9/777: truncate d11/d25/d36/d3d/fd9 92977 0 2024-04-12T05:09:15.864 INFO:tasks.workunit.client.1.smithi192.stdout:3/977: truncate d5/d13/d3c/d26/f44 2489519 0 2024-04-12T05:09:15.864 INFO:tasks.workunit.client.1.smithi192.stdout:7/825: read d3/d7/d14/f58 [98666,76001] 0 2024-04-12T05:09:15.866 INFO:tasks.workunit.client.1.smithi192.stdout:1/918: dwrite d0/d29/d8d/d96/f10b [0,4194304] 0 2024-04-12T05:09:15.868 INFO:tasks.workunit.client.1.smithi192.stdout:4/985: sync 2024-04-12T05:09:15.868 INFO:tasks.workunit.client.1.smithi192.stdout:2/888: sync 2024-04-12T05:09:15.868 INFO:tasks.workunit.client.1.smithi192.stdout:6/919: sync 2024-04-12T05:09:15.868 INFO:tasks.workunit.client.1.smithi192.stdout:3/978: readlink d5/d13/d3a/d4a/d61/l101 0 2024-04-12T05:09:15.876 INFO:tasks.workunit.client.1.smithi192.stdout:3/979: readlink d5/d13/d1a/d5f/d103/l120 0 2024-04-12T05:09:15.877 INFO:tasks.workunit.client.1.smithi192.stdout:8/951: dwrite d6/d10/d46/d5e/d5f/d31/d80/fc1 [0,4194304] 0 2024-04-12T05:09:15.879 INFO:tasks.workunit.client.1.smithi192.stdout:2/889: chown d1/d8/d3a/de/l37 6435 1 2024-04-12T05:09:15.880 INFO:tasks.workunit.client.1.smithi192.stdout:4/986: truncate d1/d54/d82/da4/da8/f108 610740 0 2024-04-12T05:09:15.884 INFO:tasks.workunit.client.1.smithi192.stdout:2/890: creat d1/d8/d17/d19/d60/d59/d68/dea/f118 x:0 0 0 2024-04-12T05:09:15.888 INFO:tasks.workunit.client.1.smithi192.stdout:7/826: link d3/d1d/d74/d86/lc8 d3/d1d/d4a/d5e/d6d/l11a 0 2024-04-12T05:09:15.890 INFO:tasks.workunit.client.1.smithi192.stdout:4/987: rename d1/d54/d55/fd0 to d1/d1d/d53/f159 0 2024-04-12T05:09:15.890 INFO:tasks.workunit.client.1.smithi192.stdout:1/919: write d0/d2/d1e/f4a [961731,12769] 0 2024-04-12T05:09:15.892 INFO:tasks.workunit.client.1.smithi192.stdout:3/980: fsync d5/d13/d3c/d26/f4d 0 2024-04-12T05:09:15.892 INFO:tasks.workunit.client.1.smithi192.stdout:6/920: write d3/d5/d9c/f101 [227411,66031] 0 2024-04-12T05:09:15.895 INFO:tasks.workunit.client.1.smithi192.stdout:9/778: dwrite d11/d25/dec/d1f/d50/d73/f8b [0,4194304] 0 2024-04-12T05:09:15.897 INFO:tasks.workunit.client.1.smithi192.stdout:2/891: creat d1/d8/d17/f119 x:0 0 0 2024-04-12T05:09:15.899 INFO:tasks.workunit.client.1.smithi192.stdout:8/952: truncate d6/d10/d46/d5e/d5f/d37/d4f/fc6 2556200 0 2024-04-12T05:09:15.901 INFO:tasks.workunit.client.1.smithi192.stdout:9/779: chown d11/d25/d36/d6e 7 1 2024-04-12T05:09:15.903 INFO:tasks.workunit.client.1.smithi192.stdout:3/981: creat d5/d13/d3a/d4e/d57/f161 x:0 0 0 2024-04-12T05:09:15.903 INFO:tasks.workunit.client.1.smithi192.stdout:2/892: truncate d1/d2d/f107 995704 0 2024-04-12T05:09:15.903 INFO:tasks.workunit.client.1.smithi192.stdout:7/827: dwrite d3/d7/d15/f50 [4194304,4194304] 0 2024-04-12T05:09:15.904 INFO:tasks.workunit.client.1.smithi192.stdout:6/921: rename d3/d8/d22/dd4 to d3/d8/de6/d12e 0 2024-04-12T05:09:15.909 INFO:tasks.workunit.client.1.smithi192.stdout:8/953: mkdir d6/d1f/d5d/d125 0 2024-04-12T05:09:15.909 INFO:tasks.workunit.client.1.smithi192.stdout:4/988: write d1/d1d/d53/d1c/d95/dc6/dd5/de5/f11d [485765,117000] 0 2024-04-12T05:09:15.909 INFO:tasks.workunit.client.1.smithi192.stdout:1/920: rename d0/d2/d1e/d3f/df2/d21/d62/d115 to d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/db6/deb/d12e 22 2024-04-12T05:09:15.911 INFO:tasks.workunit.client.1.smithi192.stdout:3/982: symlink d5/d13/d3a/d8b/l162 0 2024-04-12T05:09:15.914 INFO:tasks.workunit.client.1.smithi192.stdout:2/893: write d1/d8/d17/d19/d60/d59/d68/dea/f118 [823625,110826] 0 2024-04-12T05:09:15.914 INFO:tasks.workunit.client.1.smithi192.stdout:7/828: creat d3/d7/d60/f11b x:0 0 0 2024-04-12T05:09:15.919 INFO:tasks.workunit.client.1.smithi192.stdout:8/954: creat d6/d10/d46/d5e/d5f/d31/d80/d10c/f126 x:0 0 0 2024-04-12T05:09:15.919 INFO:tasks.workunit.client.1.smithi192.stdout:6/922: creat d3/d21/f12f x:0 0 0 2024-04-12T05:09:15.922 INFO:tasks.workunit.client.1.smithi192.stdout:8/955: fdatasync d6/d1f/f119 0 2024-04-12T05:09:15.922 INFO:tasks.workunit.client.1.smithi192.stdout:2/894: readlink d1/d8/d3a/de/d7c/d98/la4 0 2024-04-12T05:09:15.923 INFO:tasks.workunit.client.1.smithi192.stdout:9/780: dwrite d11/d25/dec/fcb [0,4194304] 0 2024-04-12T05:09:15.923 INFO:tasks.workunit.client.1.smithi192.stdout:4/989: rename d1/d1d/d53/d1c/dd1/def/ddd/d146/f154 to d1/d54/d55/dc3/f15a 0 2024-04-12T05:09:15.928 INFO:tasks.workunit.client.1.smithi192.stdout:3/983: creat d5/d13/d94/dbd/d117/f163 x:0 0 0 2024-04-12T05:09:15.929 INFO:tasks.workunit.client.1.smithi192.stdout:2/895: fsync d1/d8/d17/d19/d60/fa6 0 2024-04-12T05:09:15.930 INFO:tasks.workunit.client.1.smithi192.stdout:8/956: read d6/d1f/d10e/fae [7857526,54968] 0 2024-04-12T05:09:15.930 INFO:tasks.workunit.client.1.smithi192.stdout:9/781: mknod d11/d25/d36/d6e/dc7/c11b 0 2024-04-12T05:09:15.931 INFO:tasks.workunit.client.1.smithi192.stdout:3/984: dread - d5/d13/d3a/d4a/d61/f10a zero size 2024-04-12T05:09:15.933 INFO:tasks.workunit.client.1.smithi192.stdout:4/990: symlink d1/d54/d55/d64/d116/l15b 0 2024-04-12T05:09:15.934 INFO:tasks.workunit.client.1.smithi192.stdout:9/782: write d11/d25/d36/d3d/fb7 [66604,100527] 0 2024-04-12T05:09:15.939 INFO:tasks.workunit.client.1.smithi192.stdout:8/957: symlink d6/d10/d46/d5e/d5f/d31/d61/dc5/dd9/dd7/l127 0 2024-04-12T05:09:15.941 INFO:tasks.workunit.client.1.smithi192.stdout:6/923: write d3/dc/d39/f87 [3083584,30706] 0 2024-04-12T05:09:15.942 INFO:tasks.workunit.client.1.smithi192.stdout:9/783: unlink d11/d25/d36/d6e/d33/da2/l117 0 2024-04-12T05:09:15.945 INFO:tasks.workunit.client.1.smithi192.stdout:3/985: dwrite d5/d13/d3c/d36/f133 [4194304,4194304] 0 2024-04-12T05:09:15.946 INFO:tasks.workunit.client.1.smithi192.stdout:2/896: link d1/d8/d17/d19/d60/d9c/f9d d1/d8/d17/d19/d60/d59/da3/d10a/f11a 0 2024-04-12T05:09:15.948 INFO:tasks.workunit.client.1.smithi192.stdout:4/991: truncate d1/d1d/d53/d1c/dd1/def/ddd/d114/ff1 960334 0 2024-04-12T05:09:15.949 INFO:tasks.workunit.client.1.smithi192.stdout:9/784: mknod d11/d25/dd3/d10d/c11c 0 2024-04-12T05:09:15.950 INFO:tasks.workunit.client.1.smithi192.stdout:1/921: dwrite d0/d2/d1e/d3f/df2/d21/d62/d84/f91 [4194304,4194304] 0 2024-04-12T05:09:15.951 INFO:tasks.workunit.client.1.smithi192.stdout:6/924: creat d3/d8/d56/d59/d97/dbf/deb/f130 x:0 0 0 2024-04-12T05:09:15.954 INFO:tasks.workunit.client.1.smithi192.stdout:2/897: rename d1/d8/d3a/f3b to d1/d8/d17/db3/dee/f11b 0 2024-04-12T05:09:15.955 INFO:tasks.workunit.client.1.smithi192.stdout:8/958: creat d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d103/f128 x:0 0 0 2024-04-12T05:09:15.955 INFO:tasks.workunit.client.1.smithi192.stdout:4/992: mknod d1/d1d/d53/d1c/d111/c15c 0 2024-04-12T05:09:15.959 INFO:tasks.workunit.client.1.smithi192.stdout:6/925: write d3/d8/de6/ffb [745064,29512] 0 2024-04-12T05:09:15.961 INFO:tasks.workunit.client.1.smithi192.stdout:3/986: dread d5/d13/d3a/d4e/f95 [0,4194304] 0 2024-04-12T05:09:15.963 INFO:tasks.workunit.client.1.smithi192.stdout:4/993: write d1/d1d/d53/d1c/d95/dc6/ff9 [8297138,56523] 0 2024-04-12T05:09:15.965 INFO:tasks.workunit.client.1.smithi192.stdout:4/994: dread - d1/d54/d82/da4/da8/f138 zero size 2024-04-12T05:09:15.967 INFO:tasks.workunit.client.1.smithi192.stdout:4/995: stat d1/d49/f69 0 2024-04-12T05:09:15.968 INFO:tasks.workunit.client.1.smithi192.stdout:3/987: fsync d5/d13/d1a/d5f/d103/ffc 0 2024-04-12T05:09:15.969 INFO:tasks.workunit.client.1.smithi192.stdout:8/959: symlink d6/d10/d46/d5e/d5f/d31/d80/db1/d112/l129 0 2024-04-12T05:09:15.971 INFO:tasks.workunit.client.1.smithi192.stdout:9/785: truncate d11/d25/d36/d6e/d33/da2/f84 691141 0 2024-04-12T05:09:15.971 INFO:tasks.workunit.client.1.smithi192.stdout:1/922: link d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d74/d90/d108/l4e d0/d2/d1e/d3f/df2/d21/d62/d115/l12f 0 2024-04-12T05:09:15.972 INFO:tasks.workunit.client.0.smithi113.stdout:6/935: sync 2024-04-12T05:09:15.972 INFO:tasks.workunit.client.0.smithi113.stdout:4/960: sync 2024-04-12T05:09:15.972 INFO:tasks.workunit.client.1.smithi192.stdout:3/988: mknod d5/c164 0 2024-04-12T05:09:15.974 INFO:tasks.workunit.client.0.smithi113.stdout:4/961: truncate da/d24/d60/db4/fc7 874477 0 2024-04-12T05:09:15.975 INFO:tasks.workunit.client.0.smithi113.stdout:4/962: stat da/d3c/d43/f75 0 2024-04-12T05:09:15.976 INFO:tasks.workunit.client.1.smithi192.stdout:6/926: dwrite d3/d21/d24/f4f [0,4194304] 0 2024-04-12T05:09:15.976 INFO:tasks.workunit.client.1.smithi192.stdout:4/996: rename d1/d54/d82/d71 to d1/d1d/d53/d11a/ddb/ddc/d15d 0 2024-04-12T05:09:15.976 INFO:tasks.workunit.client.0.smithi113.stdout:6/936: dread d8/de/d17/f7b [0,4194304] 0 2024-04-12T05:09:15.979 INFO:tasks.workunit.client.0.smithi113.stdout:4/963: fdatasync da/d24/d7e/d124/d3a/f42 0 2024-04-12T05:09:15.981 INFO:tasks.workunit.client.0.smithi113.stdout:6/937: fdatasync d8/de/dd5/de7/db3/fb6 0 2024-04-12T05:09:15.982 INFO:tasks.workunit.client.1.smithi192.stdout:4/997: chown d1/f3e 221726551 1 2024-04-12T05:09:15.985 INFO:tasks.workunit.client.1.smithi192.stdout:8/960: rename d6/d10/d46/d5e/d5f/d37/d4f/d51/l8a to d6/d1f/d9c/l12a 0 2024-04-12T05:09:15.987 INFO:tasks.workunit.client.1.smithi192.stdout:4/998: read - d1/d1d/d53/d1c/dd1/f132 zero size 2024-04-12T05:09:15.988 INFO:tasks.workunit.client.0.smithi113.stdout:4/964: unlink da/d24/c96 0 2024-04-12T05:09:15.990 INFO:tasks.workunit.client.1.smithi192.stdout:1/923: dwrite d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/db6/deb/f112 [0,4194304] 0 2024-04-12T05:09:15.992 INFO:tasks.workunit.client.0.smithi113.stdout:4/965: creat da/d24/d7e/d124/d1b/d13b/d112/f13d x:0 0 0 2024-04-12T05:09:15.993 INFO:tasks.workunit.client.1.smithi192.stdout:6/927: truncate d3/d8/d56/f8e 48937 0 2024-04-12T05:09:15.993 INFO:tasks.workunit.client.1.smithi192.stdout:2/898: dread d1/d8/d3a/de/d7c/fb8 [0,4194304] 0 2024-04-12T05:09:15.996 INFO:tasks.workunit.client.1.smithi192.stdout:1/924: write d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/db6/fd3 [251532,35553] 0 2024-04-12T05:09:15.998 INFO:tasks.workunit.client.1.smithi192.stdout:3/989: link d5/d13/d3a/d70/d8c/d8d/fe4 d5/d13/d1a/d53/dbb/f165 0 2024-04-12T05:09:15.999 INFO:tasks.workunit.client.1.smithi192.stdout:4/999: rename d1/d1d/d53/d1c/d95/dc6/ff9 to d1/d1d/d11f/f15e 0 2024-04-12T05:09:15.999 INFO:tasks.workunit.client.1.smithi192.stdout:9/786: dread d11/d25/dec/d1f/d50/f98 [0,4194304] 0 2024-04-12T05:09:15.999 INFO:tasks.workunit.client.0.smithi113.stdout:4/966: dwrite da/d3c/d83/d47/d10f/f72 [4194304,4194304] 0 2024-04-12T05:09:15.999 INFO:tasks.workunit.client.1.smithi192.stdout:8/961: chown d6/d10/d46/d5e/d5f/d31/d80/d10c/l121 2224 1 2024-04-12T05:09:16.007 INFO:tasks.workunit.client.0.smithi113.stdout:6/938: dwrite d8/de/d13/d11/d36/d3d/d4c/d31/f54 [4194304,4194304] 0 2024-04-12T05:09:16.009 INFO:tasks.workunit.client.1.smithi192.stdout:6/928: rename d3/d8/c3d to d3/d5/d9c/dbd/c131 0 2024-04-12T05:09:16.011 INFO:tasks.workunit.client.1.smithi192.stdout:2/899: write d1/d2d/f32 [10149,104651] 0 2024-04-12T05:09:16.011 INFO:tasks.workunit.client.1.smithi192.stdout:9/787: mknod d11/d25/dec/d1f/d94/c11d 0 2024-04-12T05:09:16.015 INFO:tasks.workunit.client.0.smithi113.stdout:4/967: dread da/d24/d7e/d79/dcd/f118 [0,4194304] 0 2024-04-12T05:09:16.018 INFO:tasks.workunit.client.0.smithi113.stdout:4/968: chown da/d3c/d83/d47/d10f/f5d 88 1 2024-04-12T05:09:16.018 INFO:tasks.workunit.client.1.smithi192.stdout:9/788: creat d11/d25/dec/d1f/d50/f11e x:0 0 0 2024-04-12T05:09:16.018 INFO:tasks.workunit.client.1.smithi192.stdout:6/929: dwrite d3/f1e [0,4194304] 0 2024-04-12T05:09:16.019 INFO:tasks.workunit.client.0.smithi113.stdout:6/939: dwrite d8/de/d13/f135 [0,4194304] 0 2024-04-12T05:09:16.020 INFO:tasks.workunit.client.1.smithi192.stdout:1/925: link d0/d2/d1e/d3f/d56/d9e/c83 d0/d1b/d28/d9d/c130 0 2024-04-12T05:09:16.022 INFO:tasks.workunit.client.0.smithi113.stdout:4/969: rmdir da/d24/d7b/d110 39 2024-04-12T05:09:16.022 INFO:tasks.workunit.client.1.smithi192.stdout:6/930: mknod d3/d8/d22/c132 0 2024-04-12T05:09:16.024 INFO:tasks.workunit.client.0.smithi113.stdout:4/970: write da/d24/d7e/d124/d3a/fda [231442,57716] 0 2024-04-12T05:09:16.024 INFO:tasks.workunit.client.1.smithi192.stdout:6/931: chown d3/d23/db2 256341 1 2024-04-12T05:09:16.029 INFO:tasks.workunit.client.1.smithi192.stdout:6/932: creat d3/f133 x:0 0 0 2024-04-12T05:09:16.031 INFO:tasks.workunit.client.1.smithi192.stdout:1/926: rename d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d7c/c89 to d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/da7/dd7/c131 0 2024-04-12T05:09:16.031 INFO:tasks.workunit.client.1.smithi192.stdout:3/990: dwrite d5/f1c [0,4194304] 0 2024-04-12T05:09:16.035 INFO:tasks.workunit.client.1.smithi192.stdout:3/991: fsync d5/d13/d3a/d8b/fb5 0 2024-04-12T05:09:16.037 INFO:tasks.workunit.client.0.smithi113.stdout:4/971: dread da/d24/d60/f99 [0,4194304] 0 2024-04-12T05:09:16.038 INFO:tasks.workunit.client.1.smithi192.stdout:3/992: chown d5/d13/d3a/d4e/lcf 69157 1 2024-04-12T05:09:16.039 INFO:tasks.workunit.client.1.smithi192.stdout:1/927: symlink d0/d29/d8d/d11c/l132 0 2024-04-12T05:09:16.040 INFO:tasks.workunit.client.0.smithi113.stdout:4/972: rmdir da/d3c/da5/dad/de1 39 2024-04-12T05:09:16.044 INFO:tasks.workunit.client.0.smithi113.stdout:4/973: creat da/d3c/d83/d47/d10f/d90/f13e x:0 0 0 2024-04-12T05:09:16.045 INFO:tasks.workunit.client.1.smithi192.stdout:9/789: dwrite d11/d25/d36/d6e/d33/da2/fc4 [0,4194304] 0 2024-04-12T05:09:16.045 INFO:tasks.workunit.client.1.smithi192.stdout:3/993: symlink d5/d13/d3a/d4a/d61/d9f/deb/dfb/d114/l166 0 2024-04-12T05:09:16.048 INFO:tasks.workunit.client.0.smithi113.stdout:4/974: truncate da/f66 2392888 0 2024-04-12T05:09:16.048 INFO:tasks.workunit.client.1.smithi192.stdout:9/790: write d11/d25/dec/f8a [187111,105570] 0 2024-04-12T05:09:16.052 INFO:tasks.workunit.client.0.smithi113.stdout:4/975: dread da/d24/d7e/d124/d3a/fa3 [0,4194304] 0 2024-04-12T05:09:16.054 INFO:tasks.workunit.client.1.smithi192.stdout:8/962: dread d6/d10/d109/f49 [0,4194304] 0 2024-04-12T05:09:16.055 INFO:tasks.workunit.client.1.smithi192.stdout:9/791: mkdir d11/d15/d11f 0 2024-04-12T05:09:16.056 INFO:tasks.workunit.client.1.smithi192.stdout:9/792: chown d11/d25/dec/lda 124873588 1 2024-04-12T05:09:16.058 INFO:tasks.workunit.client.1.smithi192.stdout:8/963: rmdir d6/d10/d46/d5e/d5f/d37/d4f/d4c/df5 0 2024-04-12T05:09:16.059 INFO:tasks.workunit.client.1.smithi192.stdout:8/964: chown d6/d10/d46/d5e/d5f 14286 1 2024-04-12T05:09:16.061 INFO:tasks.workunit.client.1.smithi192.stdout:8/965: fsync d6/d1f/d10e/f81 0 2024-04-12T05:09:16.062 INFO:tasks.workunit.client.1.smithi192.stdout:8/966: chown d6/d10/f78 3681 1 2024-04-12T05:09:16.063 INFO:tasks.workunit.client.1.smithi192.stdout:9/793: dread d11/d25/d36/d6e/d71/f95 [0,4194304] 0 2024-04-12T05:09:16.065 INFO:tasks.workunit.client.0.smithi113.stdout:6/940: dread d8/f40 [0,4194304] 0 2024-04-12T05:09:16.066 INFO:tasks.workunit.client.1.smithi192.stdout:9/794: write d11/d25/d36/d6e/d33/da2/fc4 [1963747,130707] 0 2024-04-12T05:09:16.068 INFO:tasks.workunit.client.1.smithi192.stdout:8/967: dread d6/d10/d46/d5e/d6e/fbf [0,4194304] 0 2024-04-12T05:09:16.069 INFO:tasks.workunit.client.1.smithi192.stdout:9/795: readlink d11/laa 0 2024-04-12T05:09:16.069 INFO:tasks.workunit.client.1.smithi192.stdout:3/994: truncate d5/d13/d3a/d4a/f5a 3534462 0 2024-04-12T05:09:16.072 INFO:tasks.workunit.client.0.smithi113.stdout:6/941: mknod d8/de/d13/d11/d36/d2c/c149 0 2024-04-12T05:09:16.074 INFO:tasks.workunit.client.1.smithi192.stdout:3/995: read - d5/d13/d3a/d4a/d61/d9f/f138 zero size 2024-04-12T05:09:16.075 INFO:tasks.workunit.client.0.smithi113.stdout:6/942: link d8/de/dd5/lfa d8/de/dd5/de7/d76/l14a 0 2024-04-12T05:09:16.075 INFO:tasks.workunit.client.1.smithi192.stdout:6/933: dread d3/f37 [0,4194304] 0 2024-04-12T05:09:16.077 INFO:tasks.workunit.client.1.smithi192.stdout:1/928: dwrite d0/d2/d1e/d3f/df2/d21/f47 [0,4194304] 0 2024-04-12T05:09:16.081 INFO:tasks.workunit.client.1.smithi192.stdout:3/996: mkdir d5/d13/d3a/d70/d8c/de1/d167 0 2024-04-12T05:09:16.082 INFO:tasks.workunit.client.0.smithi113.stdout:6/943: creat d8/de/d13/df0/d66/dc4/dd9/de5/f14b x:0 0 0 2024-04-12T05:09:16.082 INFO:tasks.workunit.client.1.smithi192.stdout:8/968: rename d6/d1f/d6a/f52 to d6/d10/d46/d5e/d5f/d31/f12b 0 2024-04-12T05:09:16.083 INFO:tasks.workunit.client.0.smithi113.stdout:6/944: chown d8/f40 312016 1 2024-04-12T05:09:16.083 INFO:tasks.workunit.client.1.smithi192.stdout:8/969: readlink d6/l17 0 2024-04-12T05:09:16.084 INFO:tasks.workunit.client.1.smithi192.stdout:6/934: mkdir d3/d5/d9c/d134 0 2024-04-12T05:09:16.086 INFO:tasks.workunit.client.0.smithi113.stdout:6/945: rmdir d8/de/d13/d11/d36/d53/db8 39 2024-04-12T05:09:16.088 INFO:tasks.workunit.client.0.smithi113.stdout:6/946: symlink d8/d1d/l14c 0 2024-04-12T05:09:16.089 INFO:tasks.workunit.client.1.smithi192.stdout:9/796: dwrite d11/d25/d36/d6e/d33/da2/f112 [0,4194304] 0 2024-04-12T05:09:16.090 INFO:tasks.workunit.client.1.smithi192.stdout:8/970: dread d6/d10/f78 [0,4194304] 0 2024-04-12T05:09:16.092 INFO:tasks.workunit.client.0.smithi113.stdout:6/947: creat d8/de/d13/d11/d36/d53/db8/f14d x:0 0 0 2024-04-12T05:09:16.094 INFO:tasks.workunit.client.1.smithi192.stdout:8/971: write d6/d10/d46/d5e/d5f/d31/d61/ff2 [857157,72050] 0 2024-04-12T05:09:16.094 INFO:tasks.workunit.client.1.smithi192.stdout:9/797: rmdir d11/d25/d36 39 2024-04-12T05:09:16.098 INFO:tasks.workunit.client.1.smithi192.stdout:6/935: dwrite d3/dc/d39/fc9 [0,4194304] 0 2024-04-12T05:09:16.099 INFO:tasks.workunit.client.0.smithi113.stdout:6/948: creat d8/de/d13/d11/d36/d3d/d4c/d26/d5b/f14e x:0 0 0 2024-04-12T05:09:16.101 INFO:tasks.workunit.client.1.smithi192.stdout:3/997: truncate d5/d13/d3c/d26/fc5 1376487 0 2024-04-12T05:09:16.102 INFO:tasks.workunit.client.0.smithi113.stdout:6/949: creat d8/d1d/f14f x:0 0 0 2024-04-12T05:09:16.104 INFO:tasks.workunit.client.1.smithi192.stdout:6/936: mknod d3/d5/d9c/d134/c135 0 2024-04-12T05:09:16.105 INFO:tasks.workunit.client.1.smithi192.stdout:9/798: fdatasync d11/d25/d36/d6e/d33/da2/f84 0 2024-04-12T05:09:16.105 INFO:tasks.workunit.client.0.smithi113.stdout:6/950: mkdir d8/de/d13/d11/d36/d3d/d4c/d26/d108/d150 0 2024-04-12T05:09:16.107 INFO:tasks.workunit.client.1.smithi192.stdout:3/998: chown d5/d13/d3a/d70/d8c/de1/cb1 3500 1 2024-04-12T05:09:16.109 INFO:tasks.workunit.client.0.smithi113.stdout:6/951: fsync d8/de/d13/d11/d36/f115 0 2024-04-12T05:09:16.111 INFO:tasks.workunit.client.1.smithi192.stdout:3/999: fsync d5/d13/d3a/d8b/fae 0 2024-04-12T05:09:16.111 INFO:tasks.workunit.client.1.smithi192.stdout:1/929: dwrite d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d7c/df7/f10d [0,4194304] 0 2024-04-12T05:09:16.115 INFO:tasks.workunit.client.1.smithi192.stdout:9/799: unlink d11/d25/dec/d1f/d50/d73/lb8 0 2024-04-12T05:09:16.115 INFO:tasks.workunit.client.1.smithi192.stdout:8/972: dwrite d6/d10/d46/d5e/d5f/d31/d80/db1/fb6 [0,4194304] 0 2024-04-12T05:09:16.115 INFO:tasks.workunit.client.1.smithi192.stdout:6/937: symlink d3/d8/d56/d59/ddf/df1/l136 0 2024-04-12T05:09:16.116 INFO:tasks.workunit.client.0.smithi113.stdout:6/952: rename d8/de/d13/d11/d36/f35 to d8/de/d13/d11/d36/d3d/d4c/d26/d5b/d136/f151 0 2024-04-12T05:09:16.122 INFO:tasks.workunit.client.1.smithi192.stdout:6/938: symlink d3/dc/l137 0 2024-04-12T05:09:16.123 INFO:tasks.workunit.client.1.smithi192.stdout:6/939: dread - d3/d5/d9e/f117 zero size 2024-04-12T05:09:16.124 INFO:tasks.workunit.client.1.smithi192.stdout:9/800: rename d11/d25/d36/d3d/d7a to d11/d25/d36/d6e/d8f/d120 0 2024-04-12T05:09:16.126 INFO:tasks.workunit.client.1.smithi192.stdout:8/973: mkdir d6/d1f/d5d/d12c 0 2024-04-12T05:09:16.126 INFO:tasks.workunit.client.1.smithi192.stdout:1/930: dwrite d0/d2/d1e/d3f/df2/d21/d62/d84/f91 [4194304,4194304] 0 2024-04-12T05:09:16.129 INFO:tasks.workunit.client.1.smithi192.stdout:6/940: unlink d3/d23/c5e 0 2024-04-12T05:09:16.130 INFO:tasks.workunit.client.1.smithi192.stdout:6/941: write d3/d8/d56/d59/d97/dbf/de8/fa0 [3644181,115079] 0 2024-04-12T05:09:16.133 INFO:tasks.workunit.client.1.smithi192.stdout:1/931: read d0/d2/d1e/d3f/df2/d21/f43 [360011,111489] 0 2024-04-12T05:09:16.135 INFO:tasks.workunit.client.1.smithi192.stdout:9/801: getdents d11 0 2024-04-12T05:09:16.137 INFO:tasks.workunit.client.1.smithi192.stdout:1/932: creat d0/d2/d9f/f133 x:0 0 0 2024-04-12T05:09:16.139 INFO:tasks.workunit.client.1.smithi192.stdout:1/933: chown d0/d2/d1e/d3f/df2/d21/d62/dbc/cd4 2 1 2024-04-12T05:09:16.142 INFO:tasks.workunit.client.1.smithi192.stdout:1/934: creat d0/d29/d8d/d11c/f134 x:0 0 0 2024-04-12T05:09:16.143 INFO:tasks.workunit.client.1.smithi192.stdout:1/935: truncate d0/dd0/f10c 748031 0 2024-04-12T05:09:16.144 INFO:tasks.workunit.client.1.smithi192.stdout:6/942: write d3/d8/d48/f63 [1557200,1681] 0 2024-04-12T05:09:16.146 INFO:tasks.workunit.client.0.smithi113.stdout:6/953: dwrite d8/de/f19 [0,4194304] 0 2024-04-12T05:09:16.147 INFO:tasks.workunit.client.0.smithi113.stdout:6/954: dread - d8/de/d13/d11/d36/d53/db7/f112 zero size 2024-04-12T05:09:16.149 INFO:tasks.workunit.client.0.smithi113.stdout:6/955: read d8/de/d13/d11/d36/d53/db8/dce/d128/f131 [359608,51206] 0 2024-04-12T05:09:16.151 INFO:tasks.workunit.client.1.smithi192.stdout:8/974: dwrite d6/d10/d46/d5e/d6e/fbf [0,4194304] 0 2024-04-12T05:09:16.151 INFO:tasks.workunit.client.0.smithi113.stdout:6/956: fsync d8/d45/fe9 0 2024-04-12T05:09:16.155 INFO:tasks.workunit.client.1.smithi192.stdout:8/975: mkdir d6/d10/d46/d5e/d5f/d31/d80/db1/d12d 0 2024-04-12T05:09:16.157 INFO:tasks.workunit.client.0.smithi113.stdout:6/957: unlink d8/faf 0 2024-04-12T05:09:16.159 INFO:tasks.workunit.client.1.smithi192.stdout:1/936: truncate d0/d2/f50 2035690 0 2024-04-12T05:09:16.161 INFO:tasks.workunit.client.1.smithi192.stdout:8/976: getdents d6 0 2024-04-12T05:09:16.162 INFO:tasks.workunit.client.0.smithi113.stdout:6/958: symlink d8/de/d13/d11/d36/l152 0 2024-04-12T05:09:16.166 INFO:tasks.workunit.client.0.smithi113.stdout:6/959: truncate d8/de/d13/d11/d36/d3d/d4c/d26/fdd 118532 0 2024-04-12T05:09:16.168 INFO:tasks.workunit.client.1.smithi192.stdout:8/977: symlink d6/d10/d46/d5e/d5f/d31/d80/db1/l12e 0 2024-04-12T05:09:16.169 INFO:tasks.workunit.client.1.smithi192.stdout:1/937: dread d0/f20 [4194304,4194304] 0 2024-04-12T05:09:16.169 INFO:tasks.workunit.client.0.smithi113.stdout:6/960: dread d8/f40 [0,4194304] 0 2024-04-12T05:09:16.171 INFO:tasks.workunit.client.1.smithi192.stdout:1/938: write d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/db6/fd3 [2215263,39296] 0 2024-04-12T05:09:16.172 INFO:tasks.workunit.client.0.smithi113.stdout:6/961: symlink d8/de/d13/d11/l153 0 2024-04-12T05:09:16.173 INFO:tasks.workunit.client.1.smithi192.stdout:1/939: chown d0/d1b/d28/d9d/f8b 0 1 2024-04-12T05:09:16.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:14.675207+0000 mgr.smithi192.ocppyo (mgr.24357) 34 : cluster [DBG] pgmap v27: 65 pgs: 65 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 108 MiB/s rd, 226 MiB/s wr, 505 op/s 2024-04-12T05:09:16.178 INFO:tasks.workunit.client.1.smithi192.stdout:8/978: dwrite d6/d1f/d5d/fd3 [0,4194304] 0 2024-04-12T05:09:16.179 INFO:tasks.workunit.client.1.smithi192.stdout:6/943: dread d3/d21/f8d [0,4194304] 0 2024-04-12T05:09:16.181 INFO:tasks.workunit.client.1.smithi192.stdout:6/944: chown d3/d8/d48 320 1 2024-04-12T05:09:16.182 INFO:tasks.workunit.client.1.smithi192.stdout:8/979: mknod d6/d10/d46/d5e/d5f/d31/d80/db1/d12d/c12f 0 2024-04-12T05:09:16.184 INFO:tasks.workunit.client.1.smithi192.stdout:1/940: truncate d0/d29/d8d/d96/f10b 1118927 0 2024-04-12T05:09:16.187 INFO:tasks.workunit.client.1.smithi192.stdout:8/980: getdents d6/d10/d46/d5e 0 2024-04-12T05:09:16.189 INFO:tasks.workunit.client.1.smithi192.stdout:8/981: rmdir d6/d10/d46/d5e/d5f/d37/d4f/d51/d65 39 2024-04-12T05:09:16.193 INFO:tasks.workunit.client.1.smithi192.stdout:1/941: dread d0/d2/d1e/d3f/df2/d21/d62/dbc/fc0 [0,4194304] 0 2024-04-12T05:09:16.193 INFO:tasks.workunit.client.1.smithi192.stdout:8/982: rename d6/d10/d46/d5e/d5f/d31/d61/d9d/cc8 to d6/d10/d46/d5e/d5f/d37/c130 0 2024-04-12T05:09:16.194 INFO:tasks.workunit.client.1.smithi192.stdout:8/983: chown d6/d10/d46/d5e/d5f/d31/d61/d9d/l113 1607626 1 2024-04-12T05:09:16.196 INFO:tasks.workunit.client.1.smithi192.stdout:8/984: write d6/d10/d46/d5e/d5f/d31/d80/d10c/f126 [129445,97138] 0 2024-04-12T05:09:16.198 INFO:tasks.workunit.client.1.smithi192.stdout:1/942: link d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/fa5 d0/d2/d1e/d3f/df2/d3c/dab/f135 0 2024-04-12T05:09:16.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:14.675207+0000 mgr.smithi192.ocppyo (mgr.24357) 34 : cluster [DBG] pgmap v27: 65 pgs: 65 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 108 MiB/s rd, 226 MiB/s wr, 505 op/s 2024-04-12T05:09:16.199 INFO:tasks.workunit.client.1.smithi192.stdout:8/985: rename d6/d10/d46/d5e/d5f to d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d103/d131 22 2024-04-12T05:09:16.201 INFO:tasks.workunit.client.1.smithi192.stdout:8/986: creat d6/d10/d46/f132 x:0 0 0 2024-04-12T05:09:16.201 INFO:tasks.workunit.client.1.smithi192.stdout:8/987: chown d6/d1f/d6a/d106 24384 1 2024-04-12T05:09:16.204 INFO:tasks.workunit.client.1.smithi192.stdout:8/988: symlink d6/d10/d46/d5e/d5f/d37/d7b/d104/l133 0 2024-04-12T05:09:16.204 INFO:tasks.workunit.client.1.smithi192.stdout:1/943: dwrite d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/db6/fd3 [0,4194304] 0 2024-04-12T05:09:16.205 INFO:tasks.workunit.client.1.smithi192.stdout:8/989: dread - d6/d10/d46/f132 zero size 2024-04-12T05:09:16.206 INFO:tasks.workunit.client.1.smithi192.stdout:1/944: readlink d0/d29/dbf/lea 0 2024-04-12T05:09:16.209 INFO:tasks.workunit.client.1.smithi192.stdout:8/990: truncate d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/d103/f128 940244 0 2024-04-12T05:09:16.211 INFO:tasks.workunit.client.1.smithi192.stdout:8/991: fdatasync d6/d10/d46/f85 0 2024-04-12T05:09:16.212 INFO:tasks.workunit.client.1.smithi192.stdout:8/992: chown d6/d10/d46/d5e/faf 1806355 1 2024-04-12T05:09:16.215 INFO:tasks.workunit.client.1.smithi192.stdout:8/993: stat d6/d10/d46/d5e/d5f/d37/d4f/d51/d65/f72 0 2024-04-12T05:09:16.215 INFO:tasks.workunit.client.1.smithi192.stdout:8/994: readlink d6/d10/d46/d5e/d5f/l4e 0 2024-04-12T05:09:16.219 INFO:tasks.workunit.client.1.smithi192.stdout:8/995: truncate d6/d10/d46/d5e/d5f/d31/d80/db1/fe5 741683 0 2024-04-12T05:09:16.219 INFO:tasks.workunit.client.1.smithi192.stdout:1/945: write d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/fb8 [52834,112220] 0 2024-04-12T05:09:16.224 INFO:tasks.workunit.client.1.smithi192.stdout:1/946: rename d0/d2/d1e/d3f/ff9 to d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d74/d90/f136 0 2024-04-12T05:09:16.225 INFO:tasks.workunit.client.1.smithi192.stdout:8/996: write d6/d10/d46/d5e/d5f/d31/f36 [1790070,21524] 0 2024-04-12T05:09:16.226 INFO:tasks.workunit.client.1.smithi192.stdout:1/947: fsync d0/d2/d1e/d3f/df2/d21/d62/dbc/fc0 0 2024-04-12T05:09:16.229 INFO:tasks.workunit.client.1.smithi192.stdout:8/997: unlink d6/d1f/d9c/l12a 0 2024-04-12T05:09:16.230 INFO:tasks.workunit.client.1.smithi192.stdout:8/998: chown d6/d10/d46/d5e/d5f/d31/d61/ff2 4262696 1 2024-04-12T05:09:16.230 INFO:tasks.workunit.client.1.smithi192.stdout:8/999: readlink d6/lf4 0 2024-04-12T05:09:16.232 INFO:tasks.workunit.client.1.smithi192.stdout:1/948: write d0/d2/d1e/d3f/df2/d21/d39/f49 [3243928,22598] 0 2024-04-12T05:09:16.233 INFO:tasks.workunit.client.1.smithi192.stdout:1/949: write d0/d1b/d44/d69/faa [234371,84745] 0 2024-04-12T05:09:16.235 INFO:tasks.workunit.client.1.smithi192.stdout:1/950: creat d0/d1b/d44/d69/f137 x:0 0 0 2024-04-12T05:09:16.237 INFO:tasks.workunit.client.1.smithi192.stdout:1/951: mkdir d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d74/d138 0 2024-04-12T05:09:16.239 INFO:tasks.workunit.client.1.smithi192.stdout:1/952: getdents d0/d1b/d28/d9d 0 2024-04-12T05:09:16.244 INFO:tasks.workunit.client.1.smithi192.stdout:1/953: dwrite d0/d2/d1e/d3f/df2/d21/d62/d84/f91 [0,4194304] 0 2024-04-12T05:09:16.255 INFO:tasks.workunit.client.1.smithi192.stdout:1/954: dwrite d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/da7/dd7/d126/fe5 [0,4194304] 0 2024-04-12T05:09:16.264 INFO:tasks.workunit.client.1.smithi192.stdout:1/955: dwrite d0/d2/d1e/d3f/df2/d21/d62/f128 [0,4194304] 0 2024-04-12T05:09:16.267 INFO:tasks.workunit.client.1.smithi192.stdout:1/956: rename d0/d2/d1e/d3f/d78/f12d to d0/dd0/f139 0 2024-04-12T05:09:16.273 INFO:tasks.workunit.client.1.smithi192.stdout:1/957: dwrite d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/db6/f10e [4194304,4194304] 0 2024-04-12T05:09:16.286 INFO:tasks.workunit.client.1.smithi192.stdout:1/958: dwrite d0/d2/d1e/d3f/df2/d21/d62/dbc/fc0 [0,4194304] 0 2024-04-12T05:09:16.287 INFO:tasks.workunit.client.1.smithi192.stdout:1/959: chown d0/d2/d1e/c2e 0 1 2024-04-12T05:09:16.289 INFO:tasks.workunit.client.0.smithi113.stdout:4/976: sync 2024-04-12T05:09:16.290 INFO:tasks.workunit.client.1.smithi192.stdout:1/960: symlink d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d7c/l13a 0 2024-04-12T05:09:16.291 INFO:tasks.workunit.client.0.smithi113.stdout:4/977: write da/d3c/da5/f120 [558545,118785] 0 2024-04-12T05:09:16.292 INFO:tasks.workunit.client.0.smithi113.stdout:4/978: stat da/d3c/d83/f6a 0 2024-04-12T05:09:16.292 INFO:tasks.workunit.client.1.smithi192.stdout:1/961: mkdir d0/d2/d1e/d3f/d13b 0 2024-04-12T05:09:16.311 INFO:tasks.workunit.client.0.smithi113.stdout:4/979: dread da/d24/fe3 [0,4194304] 0 2024-04-12T05:09:16.313 INFO:tasks.workunit.client.0.smithi113.stdout:4/980: mknod da/d3c/d9a/c13f 0 2024-04-12T05:09:16.319 INFO:tasks.workunit.client.0.smithi113.stdout:4/981: write da/d24/f52 [864098,130964] 0 2024-04-12T05:09:16.320 INFO:tasks.workunit.client.0.smithi113.stdout:4/982: dread - da/d3c/d43/dbc/ffc zero size 2024-04-12T05:09:16.326 INFO:tasks.workunit.client.0.smithi113.stdout:4/983: dwrite da/d3c/d9f/fdd [0,4194304] 0 2024-04-12T05:09:16.327 INFO:tasks.workunit.client.0.smithi113.stdout:4/984: truncate da/d24/f52 1165923 0 2024-04-12T05:09:16.328 INFO:tasks.workunit.client.0.smithi113.stdout:4/985: write da/d3c/d9a/dc1/d106/dc8/f12f [265252,13482] 0 2024-04-12T05:09:16.337 INFO:tasks.workunit.client.0.smithi113.stdout:4/986: write da/d3c/da5/fec [993355,43388] 0 2024-04-12T05:09:16.339 INFO:tasks.workunit.client.0.smithi113.stdout:4/987: chown da/d24/d7e/d124/d1b/d13b/d112/l126 407696 1 2024-04-12T05:09:16.345 INFO:tasks.workunit.client.0.smithi113.stdout:4/988: dwrite da/d24/d7e/d124/f10c [0,4194304] 0 2024-04-12T05:09:16.348 INFO:tasks.workunit.client.0.smithi113.stdout:6/962: sync 2024-04-12T05:09:16.352 INFO:tasks.workunit.client.0.smithi113.stdout:4/989: rename da/d3c/d83/d47/lbf to da/d24/d7b/l140 0 2024-04-12T05:09:16.353 INFO:tasks.workunit.client.0.smithi113.stdout:4/990: chown da/cde 16316 1 2024-04-12T05:09:16.356 INFO:tasks.workunit.client.0.smithi113.stdout:4/991: mkdir da/d3c/d133/d141 0 2024-04-12T05:09:16.357 INFO:tasks.workunit.client.0.smithi113.stdout:6/963: dwrite d8/de/d13/fe0 [0,4194304] 0 2024-04-12T05:09:16.359 INFO:tasks.workunit.client.0.smithi113.stdout:4/992: rmdir da/d3c/dd5 39 2024-04-12T05:09:16.361 INFO:tasks.workunit.client.0.smithi113.stdout:6/964: unlink d8/de/d13/d11/d36/d3d/d4c/d26/l4b 0 2024-04-12T05:09:16.363 INFO:tasks.workunit.client.0.smithi113.stdout:6/965: readlink d8/de/l4f 0 2024-04-12T05:09:16.364 INFO:tasks.workunit.client.0.smithi113.stdout:4/993: mknod da/d24/d7b/d110/c142 0 2024-04-12T05:09:16.366 INFO:tasks.workunit.client.0.smithi113.stdout:6/966: mknod d8/de/d13/d11/d36/d3d/d4c/d31/d83/c154 0 2024-04-12T05:09:16.367 INFO:tasks.workunit.client.0.smithi113.stdout:6/967: dread - d8/de/dd5/f119 zero size 2024-04-12T05:09:16.369 INFO:tasks.workunit.client.0.smithi113.stdout:4/994: getdents da/d3c/da5 0 2024-04-12T05:09:16.371 INFO:tasks.workunit.client.0.smithi113.stdout:6/968: rename d8/de/dd5/de7/d76/f12d to d8/de/d13/d11/f155 0 2024-04-12T05:09:16.374 INFO:tasks.workunit.client.0.smithi113.stdout:6/969: rename d8/de/d17/c1e to d8/de/d13/d11/d36/d3d/d4c/d26/d108/d150/c156 0 2024-04-12T05:09:16.380 INFO:tasks.workunit.client.0.smithi113.stdout:6/970: dwrite d8/de/d13/d11/d36/d53/db7/f112 [0,4194304] 0 2024-04-12T05:09:16.381 INFO:tasks.workunit.client.0.smithi113.stdout:4/995: dwrite da/d3c/d9f/fba [0,4194304] 0 2024-04-12T05:09:16.386 INFO:tasks.workunit.client.0.smithi113.stdout:6/971: readlink d8/de/d13/d11/d36/d53/db8/dce/lea 0 2024-04-12T05:09:16.393 INFO:tasks.workunit.client.0.smithi113.stdout:6/972: creat d8/de/d13/d11/d36/d89/d11a/f157 x:0 0 0 2024-04-12T05:09:16.396 INFO:tasks.workunit.client.0.smithi113.stdout:4/996: dread da/d24/d7e/d124/f48 [0,4194304] 0 2024-04-12T05:09:16.396 INFO:tasks.workunit.client.0.smithi113.stdout:6/973: symlink d8/de/d13/d11/d36/d3d/d4c/l158 0 2024-04-12T05:09:16.398 INFO:tasks.workunit.client.0.smithi113.stdout:4/997: creat da/d3c/dfd/f143 x:0 0 0 2024-04-12T05:09:16.399 INFO:tasks.workunit.client.0.smithi113.stdout:6/974: symlink d8/de/d13/d11/d36/d53/db8/l159 0 2024-04-12T05:09:16.400 INFO:tasks.workunit.client.0.smithi113.stdout:4/998: write da/d3c/da5/f10e [3655198,42585] 0 2024-04-12T05:09:16.402 INFO:tasks.workunit.client.0.smithi113.stdout:6/975: creat d8/de/d13/d11/d36/d3d/d4c/d31/f15a x:0 0 0 2024-04-12T05:09:16.404 INFO:tasks.workunit.client.0.smithi113.stdout:6/976: rmdir d8/de 39 2024-04-12T05:09:16.409 INFO:tasks.workunit.client.0.smithi113.stdout:4/999: dwrite da/d24/d7e/d124/d3a/fa3 [4194304,4194304] 0 2024-04-12T05:09:16.414 INFO:tasks.workunit.client.0.smithi113.stdout:6/977: dwrite d8/de/d13/d11/f155 [0,4194304] 0 2024-04-12T05:09:16.428 INFO:tasks.workunit.client.0.smithi113.stdout:6/978: write d8/de/dd5/de7/f57 [3119463,111849] 0 2024-04-12T05:09:16.428 INFO:tasks.workunit.client.0.smithi113.stdout:6/979: write d8/d45/d124/f126 [3515498,22510] 0 2024-04-12T05:09:16.434 INFO:tasks.workunit.client.0.smithi113.stdout:6/980: rename d8/de/d13/d11/d36/d3d/d4c/d26/la7 to d8/de/d13/d11/d36/d3d/d4c/d31/d83/l15b 0 2024-04-12T05:09:16.436 INFO:tasks.workunit.client.0.smithi113.stdout:6/981: chown d8/de/d13/d11/d36/d3d/d4c/d26/d7d/f8d 220 1 2024-04-12T05:09:16.443 INFO:tasks.workunit.client.0.smithi113.stdout:6/982: rename d8/de/d13/d11/d36/d3d/d93/caa to d8/de/d13/d11/d36/d89/c15c 0 2024-04-12T05:09:16.444 INFO:tasks.workunit.client.0.smithi113.stdout:6/983: dread - d8/de/d13/d11/d36/d89/d11a/f157 zero size 2024-04-12T05:09:16.452 INFO:tasks.workunit.client.0.smithi113.stdout:6/984: dwrite d8/de/d13/d11/d36/d53/db7/fc9 [0,4194304] 0 2024-04-12T05:09:16.458 INFO:tasks.workunit.client.1.smithi192.stdout:7/829: sync 2024-04-12T05:09:16.460 INFO:tasks.workunit.client.0.smithi113.stdout:6/985: link d8/de/d13/d11/d36/d53/db7/f112 d8/de/d13/d11/d36/d3d/d4c/d26/d64/d100/f15d 0 2024-04-12T05:09:16.462 INFO:tasks.workunit.client.0.smithi113.stdout:6/986: read d8/de/d13/fe0 [4159917,96236] 0 2024-04-12T05:09:16.464 INFO:tasks.workunit.client.0.smithi113.stdout:6/987: dread d8/de/d13/d11/d36/d3d/ffb [0,4194304] 0 2024-04-12T05:09:16.505 INFO:tasks.workunit.client.1.smithi192.stdout:7/830: dread d3/d1d/d74/d86/d93/fd9 [0,4194304] 0 2024-04-12T05:09:16.506 INFO:tasks.workunit.client.1.smithi192.stdout:7/831: chown d3/d1d/d74/d86/c109 32679 1 2024-04-12T05:09:16.507 INFO:tasks.workunit.client.1.smithi192.stdout:7/832: readlink d3/d7/d15/d1a/l117 0 2024-04-12T05:09:16.518 INFO:tasks.workunit.client.0.smithi113.stdout:6/988: rename d8/de/d13/d11/d36/d3d/d4c/d26/d5b/ff8 to d8/de/d13/d11/d36/d53/db7/f15e 0 2024-04-12T05:09:16.519 INFO:tasks.workunit.client.0.smithi113.stdout:6/989: stat d8/d45/c75 0 2024-04-12T05:09:16.530 INFO:tasks.workunit.client.0.smithi113.stdout:6/990: rename d8/de/d13/d11/d36/d3d/d4c/d26 to d8/de/d13/d11/d36/d3d/d4c/dd8/d15f 0 2024-04-12T05:09:16.537 INFO:tasks.workunit.client.0.smithi113.stdout:6/991: dwrite d8/de/d13/d11/d36/d2c/fa3 [0,4194304] 0 2024-04-12T05:09:16.544 INFO:tasks.workunit.client.0.smithi113.stdout:6/992: dwrite d8/de/d13/d11/d36/d3d/d4c/d31/f54 [0,4194304] 0 2024-04-12T05:09:16.548 INFO:tasks.workunit.client.0.smithi113.stdout:6/993: mkdir d8/de/d13/d11/d36/d3d/d4c/dd8/d15f/d5b/d136/d160 0 2024-04-12T05:09:16.550 INFO:tasks.workunit.client.0.smithi113.stdout:6/994: mknod d8/de/c161 0 2024-04-12T05:09:16.561 INFO:tasks.workunit.client.0.smithi113.stdout:6/995: dwrite d8/de/d13/d11/d36/d3d/d4c/dd8/d15f/d5b/fb2 [0,4194304] 0 2024-04-12T05:09:16.567 INFO:tasks.workunit.client.0.smithi113.stdout:6/996: write d8/de/d13/d11/d36/d3d/d4c/dd8/d15f/f114 [985641,122193] 0 2024-04-12T05:09:16.568 INFO:tasks.workunit.client.0.smithi113.stdout:6/997: fdatasync d8/d1d/d13e/f145 0 2024-04-12T05:09:16.572 INFO:tasks.workunit.client.0.smithi113.stdout:6/998: unlink d8/de/d13/d11/d36/d3d/d4c/dd8/d15f/c77 0 2024-04-12T05:09:16.574 INFO:tasks.workunit.client.0.smithi113.stdout:6/999: write d8/de/d13/d11/d36/d53/db8/f9b [816140,81955] 0 2024-04-12T05:09:16.578 INFO:tasks.workunit.client.0.smithi113.stderr:+ rm -rf -- ./tmp.blyVfylW1H 2024-04-12T05:09:16.607 INFO:tasks.workunit.client.1.smithi192.stdout:7/833: write d3/d1d/d5a/d5f/f30 [1084478,54365] 0 2024-04-12T05:09:16.617 INFO:tasks.workunit.client.1.smithi192.stdout:7/834: dwrite d3/d7/fdf [0,4194304] 0 2024-04-12T05:09:16.620 INFO:tasks.workunit.client.1.smithi192.stdout:7/835: creat d3/d7/d15/d1a/df2/f11c x:0 0 0 2024-04-12T05:09:16.623 INFO:tasks.workunit.client.1.smithi192.stdout:7/836: unlink d3/d1d/d48/c10a 0 2024-04-12T05:09:16.623 INFO:tasks.workunit.client.1.smithi192.stdout:7/837: stat d3/d1d/d74/cae 0 2024-04-12T05:09:16.626 INFO:tasks.workunit.client.1.smithi192.stdout:7/838: mkdir d3/d7/d60/d65/d11d 0 2024-04-12T05:09:16.629 INFO:tasks.workunit.client.1.smithi192.stdout:7/839: creat d3/d7/d15/d22/f11e x:0 0 0 2024-04-12T05:09:16.631 INFO:tasks.workunit.client.1.smithi192.stdout:7/840: creat d3/d1d/d5a/f11f x:0 0 0 2024-04-12T05:09:16.641 INFO:tasks.workunit.client.1.smithi192.stdout:7/841: creat d3/d1d/d4a/d5e/d6d/da3/dce/dee/f120 x:0 0 0 2024-04-12T05:09:16.653 INFO:tasks.workunit.client.1.smithi192.stdout:7/842: write d3/d1d/d74/d86/d93/db9/fc3 [727464,15507] 0 2024-04-12T05:09:16.657 INFO:tasks.workunit.client.1.smithi192.stdout:7/843: truncate d3/d7/f10 4094777 0 2024-04-12T05:09:16.657 INFO:tasks.workunit.client.1.smithi192.stdout:7/844: fdatasync d3/d7/d15/d22/f4d 0 2024-04-12T05:09:16.659 INFO:tasks.workunit.client.1.smithi192.stdout:7/845: rmdir d3/d1d/d5a/d5f/d7a/da6 39 2024-04-12T05:09:16.664 INFO:tasks.workunit.client.1.smithi192.stdout:7/846: rename d3/d7/d14/f47 to d3/d1d/d48/d114/f121 0 2024-04-12T05:09:16.666 INFO:tasks.workunit.client.1.smithi192.stdout:7/847: mknod d3/d1d/d48/c122 0 2024-04-12T05:09:16.673 INFO:tasks.workunit.client.1.smithi192.stdout:7/848: dwrite d3/d7/d15/d1a/f108 [0,4194304] 0 2024-04-12T05:09:16.679 INFO:tasks.workunit.client.1.smithi192.stdout:7/849: dread d3/d1d/d5a/d5f/fd5 [0,4194304] 0 2024-04-12T05:09:16.702 INFO:tasks.workunit.client.1.smithi192.stdout:7/850: truncate d3/d7/d15/d22/fb6 1584726 0 2024-04-12T05:09:16.704 INFO:tasks.workunit.client.1.smithi192.stdout:2/900: sync 2024-04-12T05:09:16.704 INFO:tasks.workunit.client.1.smithi192.stdout:6/945: sync 2024-04-12T05:09:16.705 INFO:tasks.workunit.client.1.smithi192.stdout:9/802: sync 2024-04-12T05:09:16.745 INFO:tasks.workunit.client.1.smithi192.stdout:1/962: sync 2024-04-12T05:09:16.753 INFO:tasks.workunit.client.1.smithi192.stdout:2/901: creat d1/d8/f11c x:0 0 0 2024-04-12T05:09:16.755 INFO:tasks.workunit.client.1.smithi192.stdout:1/963: read d0/d2/d1e/d3f/d56/f79 [4045556,103576] 0 2024-04-12T05:09:16.783 INFO:tasks.workunit.client.1.smithi192.stdout:6/946: truncate d3/d8/d10/d4c/d68/f77 2940722 0 2024-04-12T05:09:16.789 INFO:tasks.workunit.client.1.smithi192.stdout:9/803: getdents d11/d25/d36 0 2024-04-12T05:09:16.795 INFO:tasks.workunit.client.1.smithi192.stdout:9/804: rename d11/d25/dec/d1f/d6a/d7e/c9d to d11/d25/d36/d6e/d33/dbb/df5/c121 0 2024-04-12T05:09:16.801 INFO:tasks.workunit.client.1.smithi192.stdout:9/805: getdents d11/d25/dec/d1f/d6a/d92/da6 0 2024-04-12T05:09:16.803 INFO:tasks.workunit.client.1.smithi192.stdout:1/964: write d0/d2/d1e/d3f/df2/f1d [3081390,89251] 0 2024-04-12T05:09:16.803 INFO:tasks.workunit.client.1.smithi192.stdout:6/947: truncate d3/d21/f7d 2867573 0 2024-04-12T05:09:16.810 INFO:tasks.workunit.client.1.smithi192.stdout:2/902: dwrite d1/d8/d17/d19/fbb [0,4194304] 0 2024-04-12T05:09:16.812 INFO:tasks.workunit.client.1.smithi192.stdout:1/965: dread - d0/d2/d1e/d3f/df2/d21/d39/f105 zero size 2024-04-12T05:09:16.812 INFO:tasks.workunit.client.1.smithi192.stdout:7/851: creat d3/d1d/d74/f123 x:0 0 0 2024-04-12T05:09:16.813 INFO:tasks.workunit.client.1.smithi192.stdout:6/948: fdatasync d3/d8/d56/f8e 0 2024-04-12T05:09:16.818 INFO:tasks.workunit.client.1.smithi192.stdout:2/903: mkdir d1/d8/d3a/de/d7c/d11d 0 2024-04-12T05:09:16.820 INFO:tasks.workunit.client.1.smithi192.stdout:2/904: write d1/d2d/fd1 [4904161,44506] 0 2024-04-12T05:09:16.822 INFO:tasks.workunit.client.1.smithi192.stdout:9/806: link d11/d25/d36/f38 d11/d25/d36/d6e/d8f/d120/df9/f122 0 2024-04-12T05:09:16.825 INFO:tasks.workunit.client.1.smithi192.stdout:2/905: symlink d1/d8/d17/d19/d60/d59/d68/dea/l11e 0 2024-04-12T05:09:16.826 INFO:tasks.workunit.client.1.smithi192.stdout:9/807: mkdir d11/d25/d36/d6e/d33/dbb/df5/d123 0 2024-04-12T05:09:16.837 INFO:tasks.workunit.client.1.smithi192.stdout:6/949: dread d3/d21/d24/f2a [0,4194304] 0 2024-04-12T05:09:16.838 INFO:tasks.workunit.client.1.smithi192.stdout:9/808: dread - d11/d25/d36/d6e/d33/ddd/f107 zero size 2024-04-12T05:09:16.839 INFO:tasks.workunit.client.1.smithi192.stdout:2/906: link d1/d2d/fe3 d1/f11f 0 2024-04-12T05:09:16.841 INFO:tasks.workunit.client.1.smithi192.stdout:6/950: chown d3/d5/d9e/cd5 0 1 2024-04-12T05:09:16.842 INFO:tasks.workunit.client.1.smithi192.stdout:9/809: mkdir d11/d25/d36/d6e/d33/dbb/d124 0 2024-04-12T05:09:16.845 INFO:tasks.workunit.client.1.smithi192.stdout:6/951: unlink d3/d23/cd9 0 2024-04-12T05:09:16.845 INFO:tasks.workunit.client.1.smithi192.stdout:1/966: dwrite d0/d29/d8d/d96/fff [0,4194304] 0 2024-04-12T05:09:16.847 INFO:tasks.workunit.client.1.smithi192.stdout:1/967: stat d0/d2/d1e/d3f/d56/f93 0 2024-04-12T05:09:16.851 INFO:tasks.workunit.client.1.smithi192.stdout:2/907: rename d1/d8/d17/d19/d60/f99 to d1/d8/d3a/de/f120 0 2024-04-12T05:09:16.852 INFO:tasks.workunit.client.1.smithi192.stdout:6/952: dread d3/dc/d39/fc9 [0,4194304] 0 2024-04-12T05:09:16.852 INFO:tasks.workunit.client.1.smithi192.stdout:2/908: readlink d1/d8/d17/d19/l5c 0 2024-04-12T05:09:16.857 INFO:tasks.workunit.client.1.smithi192.stdout:2/909: chown d1/d2d/f5d 888 1 2024-04-12T05:09:16.859 INFO:tasks.workunit.client.1.smithi192.stdout:9/810: dwrite d11/d25/d36/d6e/d33/dbb/fdb [0,4194304] 0 2024-04-12T05:09:16.861 INFO:tasks.workunit.client.1.smithi192.stdout:2/910: unlink d1/d2d/d73/da8/lc0 0 2024-04-12T05:09:16.865 INFO:tasks.workunit.client.1.smithi192.stdout:2/911: mknod d1/d8/d17/db3/dee/c121 0 2024-04-12T05:09:16.866 INFO:tasks.workunit.client.1.smithi192.stdout:1/968: write d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d74/d7d/ff5 [589772,93681] 0 2024-04-12T05:09:16.867 INFO:tasks.workunit.client.1.smithi192.stdout:6/953: write d3/d8/d56/d59/d97/dbf/f11d [47893,84689] 0 2024-04-12T05:09:16.869 INFO:tasks.workunit.client.1.smithi192.stdout:9/811: dwrite d11/d25/d36/f111 [4194304,4194304] 0 2024-04-12T05:09:16.871 INFO:tasks.workunit.client.1.smithi192.stdout:6/954: dread - d3/d8/d10/d4c/fa6 zero size 2024-04-12T05:09:16.871 INFO:tasks.workunit.client.1.smithi192.stdout:2/912: truncate d1/d8/d17/d19/f31 4330627 0 2024-04-12T05:09:16.874 INFO:tasks.workunit.client.1.smithi192.stdout:6/955: readlink d3/d8/d56/d59/ddf/df1/l136 0 2024-04-12T05:09:16.877 INFO:tasks.workunit.client.1.smithi192.stdout:6/956: creat d3/d8/d56/d59/d97/dbf/de8/d109/f138 x:0 0 0 2024-04-12T05:09:16.879 INFO:tasks.workunit.client.1.smithi192.stdout:2/913: rename d1/d8/d3a/df9/ffb to d1/d2d/f122 0 2024-04-12T05:09:16.886 INFO:tasks.workunit.client.1.smithi192.stdout:2/914: dwrite d1/d8/f11c [0,4194304] 0 2024-04-12T05:09:16.887 INFO:tasks.workunit.client.1.smithi192.stdout:1/969: write d0/d2/d1e/d3f/df2/d21/d39/f105 [880432,2671] 0 2024-04-12T05:09:16.890 INFO:tasks.workunit.client.1.smithi192.stdout:1/970: mknod d0/d2/c13c 0 2024-04-12T05:09:16.891 INFO:tasks.workunit.client.1.smithi192.stdout:2/915: truncate d1/d2d/f48 1088589 0 2024-04-12T05:09:16.896 INFO:tasks.workunit.client.1.smithi192.stdout:2/916: creat d1/d8/d17/d19/d71/d7f/f123 x:0 0 0 2024-04-12T05:09:16.899 INFO:tasks.workunit.client.1.smithi192.stdout:2/917: rename d1/d8/lc5 to d1/d8/d3a/de/d7c/d98/l124 0 2024-04-12T05:09:16.903 INFO:tasks.workunit.client.1.smithi192.stdout:1/971: dwrite d0/d1b/f6a [0,4194304] 0 2024-04-12T05:09:16.907 INFO:tasks.workunit.client.1.smithi192.stdout:1/972: fsync d0/d2/f7b 0 2024-04-12T05:09:16.909 INFO:tasks.workunit.client.1.smithi192.stdout:2/918: dwrite d1/d8/d17/f119 [0,4194304] 0 2024-04-12T05:09:16.910 INFO:tasks.workunit.client.1.smithi192.stdout:2/919: write d1/d8/d3a/f95 [5395412,56294] 0 2024-04-12T05:09:16.911 INFO:tasks.workunit.client.1.smithi192.stdout:2/920: readlink d1/l7 0 2024-04-12T05:09:16.911 INFO:tasks.workunit.client.1.smithi192.stdout:1/973: mkdir d0/d2/d1e/d113/d13d 0 2024-04-12T05:09:16.919 INFO:tasks.workunit.client.1.smithi192.stdout:1/974: symlink d0/d2/d1e/d3f/d56/d9e/l13e 0 2024-04-12T05:09:16.922 INFO:tasks.workunit.client.1.smithi192.stdout:1/975: truncate d0/d2/d1e/f36 2118418 0 2024-04-12T05:09:16.924 INFO:tasks.workunit.client.1.smithi192.stdout:1/976: mkdir d0/d2/d1e/d3f/df2/d13f 0 2024-04-12T05:09:16.927 INFO:tasks.workunit.client.1.smithi192.stdout:1/977: unlink d0/d2/d1e/d3f/df2/d21/d62/le6 0 2024-04-12T05:09:16.934 INFO:tasks.workunit.client.1.smithi192.stdout:1/978: dwrite d0/d2/d1e/d3f/df2/d21/d39/f82 [0,4194304] 0 2024-04-12T05:09:16.935 INFO:tasks.workunit.client.1.smithi192.stdout:2/921: dread d1/d8/f29 [0,4194304] 0 2024-04-12T05:09:16.944 INFO:tasks.workunit.client.1.smithi192.stdout:1/979: symlink d0/d1b/d44/l140 0 2024-04-12T05:09:16.944 INFO:tasks.workunit.client.1.smithi192.stdout:2/922: mkdir d1/d8/d17/d19/d60/d59/da3/d10a/d125 0 2024-04-12T05:09:16.944 INFO:tasks.workunit.client.1.smithi192.stdout:2/923: read d1/f44 [568938,126574] 0 2024-04-12T05:09:16.944 INFO:tasks.workunit.client.1.smithi192.stdout:1/980: link d0/d2/l57 d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/l141 0 2024-04-12T05:09:16.946 INFO:tasks.workunit.client.1.smithi192.stdout:1/981: symlink d0/d2/d1e/d3f/df2/d21/d62/d84/l142 0 2024-04-12T05:09:16.949 INFO:tasks.workunit.client.1.smithi192.stdout:1/982: creat d0/d2/d1e/d3f/df2/d21/d62/dbc/f143 x:0 0 0 2024-04-12T05:09:16.951 INFO:tasks.workunit.client.1.smithi192.stdout:1/983: mknod d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d74/d7d/df3/c144 0 2024-04-12T05:09:16.953 INFO:tasks.workunit.client.1.smithi192.stdout:1/984: fsync d0/d2/d1e/d3f/df2/fa9 0 2024-04-12T05:09:16.955 INFO:tasks.workunit.client.1.smithi192.stdout:1/985: mkdir d0/d29/d8d/d96/d11b/d145 0 2024-04-12T05:09:16.962 INFO:tasks.workunit.client.1.smithi192.stdout:2/924: write d1/f11f [4834916,39603] 0 2024-04-12T05:09:16.964 INFO:tasks.workunit.client.1.smithi192.stdout:2/925: read - d1/d2d/d73/f9b zero size 2024-04-12T05:09:16.967 INFO:tasks.workunit.client.1.smithi192.stdout:1/986: truncate d0/d1b/d44/d69/ffb 3634301 0 2024-04-12T05:09:16.967 INFO:tasks.workunit.client.1.smithi192.stdout:2/926: creat d1/d8/d17/db3/f126 x:0 0 0 2024-04-12T05:09:16.970 INFO:tasks.workunit.client.1.smithi192.stdout:2/927: mknod d1/d8/d3a/c127 0 2024-04-12T05:09:16.970 INFO:tasks.workunit.client.1.smithi192.stdout:2/928: chown d1/d8/d17/d19/d60/f65 45126 1 2024-04-12T05:09:16.971 INFO:tasks.workunit.client.1.smithi192.stdout:1/987: rename d0/d2/d1e/d3f/df2/d21/d62/d84/c117 to d0/d29/c146 0 2024-04-12T05:09:16.973 INFO:tasks.workunit.client.1.smithi192.stdout:2/929: symlink d1/d2d/d73/da8/db2/d115/l128 0 2024-04-12T05:09:16.974 INFO:tasks.workunit.client.1.smithi192.stdout:1/988: mkdir d0/d29/d8d/d96/d11b/d145/d147 0 2024-04-12T05:09:16.976 INFO:tasks.workunit.client.1.smithi192.stdout:1/989: write d0/d2/d1e/d3f/df2/d21/d39/f49 [3555016,111154] 0 2024-04-12T05:09:16.980 INFO:tasks.workunit.client.1.smithi192.stdout:1/990: symlink d0/d2/d1e/d3f/d13b/l148 0 2024-04-12T05:09:16.985 INFO:tasks.workunit.client.1.smithi192.stdout:1/991: dread d0/d2/d1e/d3f/df2/d3c/dab/fb5 [0,4194304] 0 2024-04-12T05:09:16.987 INFO:tasks.workunit.client.1.smithi192.stdout:2/930: dwrite d1/d8/f29 [0,4194304] 0 2024-04-12T05:09:16.988 INFO:tasks.workunit.client.1.smithi192.stdout:1/992: read d0/d2/d1e/d3f/d56/f11a [59844,8239] 0 2024-04-12T05:09:16.992 INFO:tasks.workunit.client.1.smithi192.stdout:2/931: write d1/d2d/d73/da8/f10c [778383,13919] 0 2024-04-12T05:09:16.993 INFO:tasks.workunit.client.1.smithi192.stdout:1/993: symlink d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/d74/d7d/l149 0 2024-04-12T05:09:17.000 INFO:tasks.workunit.client.1.smithi192.stdout:1/994: dwrite d0/d2/d1e/d3f/df2/d21/d62/d84/f8e [0,4194304] 0 2024-04-12T05:09:17.010 INFO:tasks.workunit.client.1.smithi192.stdout:1/995: truncate d0/d2/d1e/d3f/df2/d21/d62/d84/f91 786783 0 2024-04-12T05:09:17.012 INFO:tasks.workunit.client.1.smithi192.stdout:1/996: write d0/d2/d1e/d3f/d56/fbe [2049582,40722] 0 2024-04-12T05:09:17.017 INFO:tasks.workunit.client.1.smithi192.stdout:1/997: link d0/d2/d1e/d3f/df2/d21/d62/d115/d12a/db6/ld1 d0/d29/d8d/d11c/l14a 0 2024-04-12T05:09:17.019 INFO:tasks.workunit.client.1.smithi192.stdout:1/998: rmdir d0/d2/d1e/d3f/d13b 39 2024-04-12T05:09:17.030 INFO:tasks.workunit.client.1.smithi192.stdout:1/999: dwrite d0/d2/d1e/f36 [0,4194304] 0 2024-04-12T05:09:17.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:15.909521+0000 mon.smithi192 (mon.1) 53 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:09:17.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:15.909521+0000 mon.smithi192 (mon.1) 53 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:09:17.323 INFO:tasks.workunit.client.1.smithi192.stdout:7/852: sync 2024-04-12T05:09:17.333 INFO:tasks.workunit.client.1.smithi192.stdout:7/853: dwrite d3/d1d/f69 [0,4194304] 0 2024-04-12T05:09:17.336 INFO:tasks.workunit.client.1.smithi192.stdout:7/854: mknod d3/d1d/d5a/d5f/d6a/c124 0 2024-04-12T05:09:17.340 INFO:tasks.workunit.client.1.smithi192.stdout:7/855: creat d3/d1d/d4a/d5e/df9/dfd/f125 x:0 0 0 2024-04-12T05:09:17.341 INFO:tasks.workunit.client.1.smithi192.stdout:7/856: rename d3/d7 to d3/d7/d14/d1f/d126 22 2024-04-12T05:09:17.343 INFO:tasks.workunit.client.1.smithi192.stdout:7/857: write d3/d1d/d74/d86/fed [930102,16761] 0 2024-04-12T05:09:17.345 INFO:tasks.workunit.client.1.smithi192.stdout:7/858: chown d3/d1d/d5a/d5f/d7a/da6/d105/c10d 33 1 2024-04-12T05:09:17.347 INFO:tasks.workunit.client.1.smithi192.stdout:7/859: read d3/d7/d60/f66 [2482284,21776] 0 2024-04-12T05:09:17.349 INFO:tasks.workunit.client.1.smithi192.stdout:7/860: unlink d3/d7/d15/d22/f4d 0 2024-04-12T05:09:17.350 INFO:tasks.workunit.client.1.smithi192.stdout:7/861: chown d3/d1d/d74/d86/c109 385337748 1 2024-04-12T05:09:17.353 INFO:tasks.workunit.client.1.smithi192.stdout:7/862: getdents d3/d7/d60/d65 0 2024-04-12T05:09:17.355 INFO:tasks.workunit.client.1.smithi192.stdout:7/863: unlink d3/d1d/f2e 0 2024-04-12T05:09:17.359 INFO:tasks.workunit.client.1.smithi192.stdout:7/864: rename d3/f21 to d3/d7/f127 0 2024-04-12T05:09:17.371 INFO:tasks.workunit.client.1.smithi192.stdout:7/865: mknod d3/d1d/d5a/d5f/d7a/da6/c128 0 2024-04-12T05:09:17.372 INFO:tasks.workunit.client.1.smithi192.stdout:7/866: stat d3/d1d/d5a/d5f/d6a/de9/c107 0 2024-04-12T05:09:17.380 INFO:tasks.workunit.client.1.smithi192.stdout:7/867: dwrite d3/d11/d68/f115 [0,4194304] 0 2024-04-12T05:09:17.386 INFO:tasks.workunit.client.1.smithi192.stdout:7/868: creat d3/d7/d15/f129 x:0 0 0 2024-04-12T05:09:17.388 INFO:tasks.workunit.client.1.smithi192.stdout:7/869: creat d3/d7/d60/d65/f12a x:0 0 0 2024-04-12T05:09:17.391 INFO:tasks.workunit.client.1.smithi192.stdout:7/870: fsync d3/d1d/d74/f8c 0 2024-04-12T05:09:17.393 INFO:tasks.workunit.client.1.smithi192.stdout:7/871: mkdir d3/d1d/d4a/d5e/d12b 0 2024-04-12T05:09:17.404 INFO:tasks.workunit.client.1.smithi192.stdout:7/872: dread d3/d7/d60/ff1 [0,4194304] 0 2024-04-12T05:09:17.405 INFO:tasks.workunit.client.1.smithi192.stdout:7/873: chown d3/d7/d14/d1f/f34 377 1 2024-04-12T05:09:17.408 INFO:tasks.workunit.client.1.smithi192.stdout:7/874: write d3/d1d/d5a/d5f/d6a/de9/f116 [4123643,40972] 0 2024-04-12T05:09:17.417 INFO:tasks.workunit.client.1.smithi192.stdout:7/875: dwrite d3/d7/d15/d1a/f3e [0,4194304] 0 2024-04-12T05:09:17.424 INFO:tasks.workunit.client.1.smithi192.stdout:7/876: dread d3/d1d/d74/d86/d93/ff7 [0,4194304] 0 2024-04-12T05:09:17.429 INFO:tasks.workunit.client.1.smithi192.stdout:7/877: getdents d3/d1d/d5a/d5f 0 2024-04-12T05:09:17.551 INFO:tasks.workunit.client.1.smithi192.stdout:9/812: sync 2024-04-12T05:09:17.554 INFO:tasks.workunit.client.1.smithi192.stdout:9/813: getdents d11/d25/dec/d1f/d6a 0 2024-04-12T05:09:17.564 INFO:tasks.workunit.client.1.smithi192.stdout:9/814: dread d11/d25/d36/d6e/f3e [0,4194304] 0 2024-04-12T05:09:17.566 INFO:tasks.workunit.client.1.smithi192.stdout:9/815: truncate d11/d25/dd3/fe4 753647 0 2024-04-12T05:09:17.672 INFO:tasks.workunit.client.1.smithi192.stdout:6/957: sync 2024-04-12T05:09:17.672 INFO:tasks.workunit.client.1.smithi192.stdout:2/932: sync 2024-04-12T05:09:17.673 INFO:tasks.workunit.client.1.smithi192.stdout:2/933: read d1/d8/d3a/de/d7c/fb8 [3961125,36727] 0 2024-04-12T05:09:17.674 INFO:tasks.workunit.client.1.smithi192.stdout:6/958: mknod d3/d23/db5/c139 0 2024-04-12T05:09:17.677 INFO:tasks.workunit.client.1.smithi192.stdout:2/934: rename d1/d8/d17/db3/lc4 to d1/d8/d17/db3/d116/l129 0 2024-04-12T05:09:17.678 INFO:tasks.workunit.client.1.smithi192.stdout:6/959: creat d3/d8/d56/d59/d97/f13a x:0 0 0 2024-04-12T05:09:17.680 INFO:tasks.workunit.client.1.smithi192.stdout:2/935: symlink d1/d8/d3a/l12a 0 2024-04-12T05:09:17.680 INFO:tasks.workunit.client.1.smithi192.stdout:6/960: write d3/d5/d9c/dbd/fd3 [377802,53183] 0 2024-04-12T05:09:17.681 INFO:tasks.workunit.client.1.smithi192.stdout:2/936: readlink d1/l64 0 2024-04-12T05:09:17.681 INFO:tasks.workunit.client.1.smithi192.stdout:6/961: readlink d3/d8/lba 0 2024-04-12T05:09:17.684 INFO:tasks.workunit.client.1.smithi192.stdout:6/962: mkdir d3/d5/d9c/d134/d13b 0 2024-04-12T05:09:17.685 INFO:tasks.workunit.client.1.smithi192.stdout:2/937: mkdir d1/d8/d17/d19/d71/da0/d12b 0 2024-04-12T05:09:17.687 INFO:tasks.workunit.client.1.smithi192.stdout:6/963: truncate d3/d8/f13 940816 0 2024-04-12T05:09:17.688 INFO:tasks.workunit.client.1.smithi192.stdout:6/964: write d3/d8/de6/f10b [1001304,94575] 0 2024-04-12T05:09:17.691 INFO:tasks.workunit.client.1.smithi192.stdout:6/965: truncate d3/d8/d56/d59/d97/dbf/de8/d109/f138 582297 0 2024-04-12T05:09:17.696 INFO:tasks.workunit.client.1.smithi192.stdout:6/966: write d3/dc/f69 [2266906,33998] 0 2024-04-12T05:09:17.696 INFO:tasks.workunit.client.1.smithi192.stdout:6/967: read d3/d8/de6/ffb [1109299,55108] 0 2024-04-12T05:09:17.700 INFO:tasks.workunit.client.1.smithi192.stdout:6/968: creat d3/d8/d48/f13c x:0 0 0 2024-04-12T05:09:17.701 INFO:tasks.workunit.client.1.smithi192.stdout:6/969: chown d3/d23/fb9 534885 1 2024-04-12T05:09:17.704 INFO:tasks.workunit.client.1.smithi192.stdout:6/970: rmdir d3/d8/de6/d12e 39 2024-04-12T05:09:17.738 INFO:tasks.workunit.client.1.smithi192.stdout:6/971: dread d3/d8/d56/d59/d8a/faa [0,4194304] 0 2024-04-12T05:09:17.743 INFO:tasks.workunit.client.1.smithi192.stdout:6/972: rename d3/d8/d56/d59/d8a to d3/d23/db2/dc7/d13d 0 2024-04-12T05:09:17.747 INFO:tasks.workunit.client.1.smithi192.stdout:6/973: unlink d3/dc/f9a 0 2024-04-12T05:09:17.763 INFO:tasks.workunit.client.1.smithi192.stdout:6/974: write d3/d8/f13 [677959,75498] 0 2024-04-12T05:09:17.776 INFO:tasks.workunit.client.1.smithi192.stdout:7/878: sync 2024-04-12T05:09:17.776 INFO:tasks.workunit.client.1.smithi192.stdout:9/816: sync 2024-04-12T05:09:17.784 INFO:tasks.workunit.client.1.smithi192.stdout:7/879: dwrite d3/d7/dff/f110 [0,4194304] 0 2024-04-12T05:09:17.802 INFO:tasks.workunit.client.1.smithi192.stdout:6/975: truncate d3/d8/d56/d59/d97/dbf/de8/ffa 462955 0 2024-04-12T05:09:17.802 INFO:tasks.workunit.client.1.smithi192.stdout:9/817: symlink d11/d25/dd3/d10d/dce/l125 0 2024-04-12T05:09:17.805 INFO:tasks.workunit.client.1.smithi192.stdout:6/976: creat d3/d5/f13e x:0 0 0 2024-04-12T05:09:17.806 INFO:tasks.workunit.client.1.smithi192.stdout:7/880: link d3/d1d/l7f d3/d1d/d5a/d5f/d6a/l12c 0 2024-04-12T05:09:17.806 INFO:tasks.workunit.client.1.smithi192.stdout:6/977: chown d3/d8/d10/c57 1727061 1 2024-04-12T05:09:17.809 INFO:tasks.workunit.client.1.smithi192.stdout:7/881: mkdir d3/d1d/d4a/d5e/df9/d12d 0 2024-04-12T05:09:17.814 INFO:tasks.workunit.client.1.smithi192.stdout:9/818: dwrite d11/d25/d36/d6e/d8f/d9c/f114 [0,4194304] 0 2024-04-12T05:09:17.820 INFO:tasks.workunit.client.1.smithi192.stdout:7/882: dwrite d3/d7/d15/f50 [4194304,4194304] 0 2024-04-12T05:09:17.825 INFO:tasks.workunit.client.1.smithi192.stdout:9/819: truncate d11/d25/d36/d6e/fa8 1149053 0 2024-04-12T05:09:17.828 INFO:tasks.workunit.client.1.smithi192.stdout:9/820: dread - d11/d15/fc1 zero size 2024-04-12T05:09:17.829 INFO:tasks.workunit.client.1.smithi192.stdout:7/883: creat d3/d1d/d4a/d5e/d12b/f12e x:0 0 0 2024-04-12T05:09:17.833 INFO:tasks.workunit.client.1.smithi192.stdout:2/938: sync 2024-04-12T05:09:17.838 INFO:tasks.workunit.client.1.smithi192.stdout:7/884: creat d3/d11/f12f x:0 0 0 2024-04-12T05:09:17.840 INFO:tasks.workunit.client.1.smithi192.stdout:6/978: dread d3/dc/d39/f3a [0,4194304] 0 2024-04-12T05:09:17.840 INFO:tasks.workunit.client.1.smithi192.stdout:7/885: stat d3/d7/d15/d1a/df2/f11c 0 2024-04-12T05:09:17.850 INFO:tasks.workunit.client.1.smithi192.stdout:6/979: link d3/d21/d24/ff3 d3/d8/d22/f13f 0 2024-04-12T05:09:17.851 INFO:tasks.workunit.client.1.smithi192.stdout:9/821: dwrite d11/d25/dec/d1f/f7f [0,4194304] 0 2024-04-12T05:09:17.859 INFO:tasks.workunit.client.1.smithi192.stdout:9/822: rename d11/d25/d36/d6e/d71/f78 to d11/d25/d36/d6e/d8f/d120/df9/f126 0 2024-04-12T05:09:17.862 INFO:tasks.workunit.client.1.smithi192.stdout:2/939: dwrite d1/d8/d3a/f3f [0,4194304] 0 2024-04-12T05:09:17.869 INFO:tasks.workunit.client.1.smithi192.stdout:2/940: symlink d1/d8/d17/d19/d60/d59/da3/l12c 0 2024-04-12T05:09:17.869 INFO:tasks.workunit.client.1.smithi192.stdout:9/823: link d11/d25/dec/d1f/cdc d11/d25/dd3/dd5/d10c/c127 0 2024-04-12T05:09:17.872 INFO:tasks.workunit.client.1.smithi192.stdout:7/886: dread d3/d11/fbe [0,4194304] 0 2024-04-12T05:09:17.872 INFO:tasks.workunit.client.1.smithi192.stdout:6/980: dwrite d3/d8/de6/fec [0,4194304] 0 2024-04-12T05:09:17.872 INFO:tasks.workunit.client.1.smithi192.stdout:2/941: fdatasync d1/d8/d3a/de/d7c/fb8 0 2024-04-12T05:09:17.874 INFO:tasks.workunit.client.1.smithi192.stdout:6/981: chown d3/d5/f80 759800 1 2024-04-12T05:09:17.878 INFO:tasks.workunit.client.1.smithi192.stdout:6/982: fsync d3/d21/d24/f2a 0 2024-04-12T05:09:17.879 INFO:tasks.workunit.client.1.smithi192.stdout:6/983: readlink d3/d5/l3f 0 2024-04-12T05:09:17.882 INFO:tasks.workunit.client.1.smithi192.stdout:6/984: symlink d3/d23/db2/dc7/l140 0 2024-04-12T05:09:17.883 INFO:tasks.workunit.client.1.smithi192.stdout:2/942: getdents d1/d8/d17/db3/dee 0 2024-04-12T05:09:17.884 INFO:tasks.workunit.client.1.smithi192.stdout:7/887: write d3/d7/d60/d65/f81 [1034835,4316] 0 2024-04-12T05:09:17.885 INFO:tasks.workunit.client.1.smithi192.stdout:6/985: write d3/d8/d56/d59/d97/dbf/de8/d109/f138 [1567369,8456] 0 2024-04-12T05:09:17.887 INFO:tasks.workunit.client.1.smithi192.stdout:6/986: chown d3/d23/f76 2 1 2024-04-12T05:09:17.887 INFO:tasks.workunit.client.1.smithi192.stdout:2/943: creat d1/f12d x:0 0 0 2024-04-12T05:09:17.889 INFO:tasks.workunit.client.1.smithi192.stdout:7/888: mkdir d3/d1d/d48/d130 0 2024-04-12T05:09:17.892 INFO:tasks.workunit.client.1.smithi192.stdout:6/987: fsync d3/f1e 0 2024-04-12T05:09:17.892 INFO:tasks.workunit.client.1.smithi192.stdout:7/889: stat d3/d1d/d74/d86/faf 0 2024-04-12T05:09:17.892 INFO:tasks.workunit.client.1.smithi192.stdout:9/824: dwrite d11/d25/dd3/fe4 [0,4194304] 0 2024-04-12T05:09:17.895 INFO:tasks.workunit.client.1.smithi192.stdout:7/890: unlink d3/d1d/d74/d86/d93/f106 0 2024-04-12T05:09:17.898 INFO:tasks.workunit.client.1.smithi192.stdout:7/891: rmdir d3/d1d/d5a/d5f/d6a 39 2024-04-12T05:09:17.903 INFO:tasks.workunit.client.1.smithi192.stdout:6/988: rename d3/d26 to d3/d8/d56/d59/d141 0 2024-04-12T05:09:17.904 INFO:tasks.workunit.client.1.smithi192.stdout:7/892: fdatasync d3/d1d/fe1 0 2024-04-12T05:09:17.906 INFO:tasks.workunit.client.1.smithi192.stdout:9/825: rename d11/d25/d36/d6e/dc7/ce2 to d11/d25/dd3/dd5/c128 0 2024-04-12T05:09:17.907 INFO:tasks.workunit.client.1.smithi192.stdout:7/893: mknod d3/d1d/d48/d130/c131 0 2024-04-12T05:09:17.914 INFO:tasks.workunit.client.1.smithi192.stdout:7/894: truncate d3/d1d/d5a/d5f/d7a/fc6 221297 0 2024-04-12T05:09:17.915 INFO:tasks.workunit.client.1.smithi192.stdout:9/826: write d11/d25/d36/d6e/d33/fab [2124093,40090] 0 2024-04-12T05:09:17.916 INFO:tasks.workunit.client.1.smithi192.stdout:7/895: creat d3/d1d/d74/d86/f132 x:0 0 0 2024-04-12T05:09:17.919 INFO:tasks.workunit.client.1.smithi192.stdout:9/827: creat d11/d25/dec/f129 x:0 0 0 2024-04-12T05:09:17.920 INFO:tasks.workunit.client.1.smithi192.stdout:7/896: creat d3/d1d/d4a/d5e/df9/d12d/f133 x:0 0 0 2024-04-12T05:09:17.921 INFO:tasks.workunit.client.1.smithi192.stdout:9/828: mknod d11/d25/d36/d6e/d71/c12a 0 2024-04-12T05:09:17.923 INFO:tasks.workunit.client.1.smithi192.stdout:7/897: fdatasync d3/d1d/d74/d86/fe3 0 2024-04-12T05:09:17.927 INFO:tasks.workunit.client.1.smithi192.stdout:7/898: truncate d3/d7/d15/d1a/ddc/fe5 801265 0 2024-04-12T05:09:17.928 INFO:tasks.workunit.client.1.smithi192.stdout:7/899: dread - d3/d7/d60/f11b zero size 2024-04-12T05:09:17.928 INFO:tasks.workunit.client.1.smithi192.stdout:9/829: write d11/d25/dec/d1f/d6a/f7c [546286,62180] 0 2024-04-12T05:09:17.931 INFO:tasks.workunit.client.1.smithi192.stdout:7/900: mknod d3/d1d/d48/d7b/c134 0 2024-04-12T05:09:17.932 INFO:tasks.workunit.client.1.smithi192.stdout:6/989: dread d3/d23/db2/dc7/d13d/d99/f9f [0,4194304] 0 2024-04-12T05:09:17.933 INFO:tasks.workunit.client.1.smithi192.stdout:9/830: rename d11/l27 to d11/d25/dec/d1f/d6a/d92/da6/l12b 0 2024-04-12T05:09:17.936 INFO:tasks.workunit.client.1.smithi192.stdout:7/901: rename d3/d7/f7e to d3/d1d/d48/f135 0 2024-04-12T05:09:17.937 INFO:tasks.workunit.client.1.smithi192.stdout:6/990: dread d3/d23/db2/dc7/d13d/d99/f9f [0,4194304] 0 2024-04-12T05:09:17.940 INFO:tasks.workunit.client.1.smithi192.stdout:9/831: chown d11/d25/dec/d1f/d6a/d92/d115 235530559 1 2024-04-12T05:09:17.944 INFO:tasks.workunit.client.1.smithi192.stdout:9/832: mkdir d11/d25/d36/d6e/d33/dbb/df5/d12c 0 2024-04-12T05:09:17.947 INFO:tasks.workunit.client.1.smithi192.stdout:9/833: mkdir d11/d15/d12d 0 2024-04-12T05:09:17.949 INFO:tasks.workunit.client.1.smithi192.stdout:9/834: write fe [135709,115215] 0 2024-04-12T05:09:17.953 INFO:tasks.workunit.client.1.smithi192.stdout:9/835: chown d11/d25/dec/d1f/d6a/d92/d115 15766 1 2024-04-12T05:09:17.954 INFO:tasks.workunit.client.1.smithi192.stdout:7/902: dwrite d3/d7/d15/d1a/ddc/fe5 [0,4194304] 0 2024-04-12T05:09:17.954 INFO:tasks.workunit.client.1.smithi192.stdout:6/991: dwrite d3/d8/d56/d59/d97/dbf/de8/ffa [0,4194304] 0 2024-04-12T05:09:17.957 INFO:tasks.workunit.client.1.smithi192.stdout:6/992: readlink d3/d8/d56/d59/d97/dbf/de8/d109/l10f 0 2024-04-12T05:09:17.957 INFO:tasks.workunit.client.1.smithi192.stdout:7/903: mkdir d3/d7/dff/d136 0 2024-04-12T05:09:17.961 INFO:tasks.workunit.client.1.smithi192.stdout:7/904: symlink d3/d1d/d48/d130/l137 0 2024-04-12T05:09:17.962 INFO:tasks.workunit.client.1.smithi192.stdout:6/993: creat d3/d5/d9c/d134/d13b/f142 x:0 0 0 2024-04-12T05:09:17.968 INFO:tasks.workunit.client.1.smithi192.stdout:6/994: getdents d3/d8/d22 0 2024-04-12T05:09:17.973 INFO:tasks.workunit.client.1.smithi192.stdout:7/905: dwrite d3/d1d/d4a/d5e/d6d/fa1 [0,4194304] 0 2024-04-12T05:09:17.979 INFO:tasks.workunit.client.1.smithi192.stdout:7/906: mknod d3/c138 0 2024-04-12T05:09:17.979 INFO:tasks.workunit.client.1.smithi192.stdout:6/995: dwrite d3/d8/de6/fec [0,4194304] 0 2024-04-12T05:09:17.989 INFO:tasks.workunit.client.1.smithi192.stdout:7/907: symlink d3/d7/d60/d65/d11d/l139 0 2024-04-12T05:09:17.989 INFO:tasks.workunit.client.1.smithi192.stdout:6/996: link d3/d8/dd6/lf5 d3/d23/l143 0 2024-04-12T05:09:17.990 INFO:tasks.workunit.client.1.smithi192.stdout:7/908: readlink d3/d7/d15/d1a/l2a 0 2024-04-12T05:09:17.991 INFO:tasks.workunit.client.1.smithi192.stdout:6/997: chown d3/d8/d56/d59/d141 331 1 2024-04-12T05:09:17.992 INFO:tasks.workunit.client.1.smithi192.stdout:6/998: read - d3/d8/d48/fbe zero size 2024-04-12T05:09:17.992 INFO:tasks.workunit.client.1.smithi192.stdout:7/909: rmdir d3/d7/d15/d1a/ddc 39 2024-04-12T05:09:17.995 INFO:tasks.workunit.client.1.smithi192.stdout:6/999: rename d3/d5/f45 to d3/d8/dd6/f144 0 2024-04-12T05:09:17.996 INFO:tasks.workunit.client.1.smithi192.stdout:7/910: mknod d3/d7/d14/d1f/c13a 0 2024-04-12T05:09:18.002 INFO:tasks.workunit.client.1.smithi192.stdout:7/911: dwrite d3/d1d/d74/d86/fed [0,4194304] 0 2024-04-12T05:09:18.016 INFO:tasks.workunit.client.1.smithi192.stdout:7/912: dread d3/d1d/d4a/d5e/d6d/fba [0,4194304] 0 2024-04-12T05:09:18.021 INFO:tasks.workunit.client.1.smithi192.stdout:7/913: write d3/d1d/d74/f8c [2067833,48137] 0 2024-04-12T05:09:18.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:16.675547+0000 mgr.smithi192.ocppyo (mgr.24357) 35 : cluster [DBG] pgmap v28: 65 pgs: 65 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 108 MiB/s rd, 226 MiB/s wr, 505 op/s 2024-04-12T05:09:18.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:16.675547+0000 mgr.smithi192.ocppyo (mgr.24357) 35 : cluster [DBG] pgmap v28: 65 pgs: 65 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 108 MiB/s rd, 226 MiB/s wr, 505 op/s 2024-04-12T05:09:18.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.352+0000 7f49bfa5f700 1 -- 172.21.15.113:0/4053255478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49b80f7810 msgr2=0x7f49b80f7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:18.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.352+0000 7f49bfa5f700 1 --2- 172.21.15.113:0/4053255478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49b80f7810 0x7f49b80f7c30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f49b40047f0 tx=0x7f49b4009cb0).stop 2024-04-12T05:09:18.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.352+0000 7f49bfa5f700 1 -- 172.21.15.113:0/4053255478 shutdown_connections 2024-04-12T05:09:18.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.352+0000 7f49bfa5f700 1 --2- 172.21.15.113:0/4053255478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49b80f8ab0 0x7f49b80f8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:18.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.352+0000 7f49bfa5f700 1 --2- 172.21.15.113:0/4053255478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49b80f7810 0x7f49b80f7c30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:18.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.352+0000 7f49bfa5f700 1 -- 172.21.15.113:0/4053255478 >> 172.21.15.113:0/4053255478 conn(0x7f49b80f2f50 msgr2=0x7f49b80f5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:18.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.352+0000 7f49bfa5f700 1 -- 172.21.15.113:0/4053255478 shutdown_connections 2024-04-12T05:09:18.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.352+0000 7f49bfa5f700 1 -- 172.21.15.113:0/4053255478 wait complete. 2024-04-12T05:09:18.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.353+0000 7f49bfa5f700 1 Processor -- start 2024-04-12T05:09:18.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.353+0000 7f49bfa5f700 1 -- start start 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.353+0000 7f49bfa5f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49b80f7810 0x7f49b806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.353+0000 7f49bfa5f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49b80f8ab0 0x7f49b806b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.353+0000 7f49bfa5f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b806cf80 con 0x7f49b80f8ab0 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.353+0000 7f49bfa5f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b806b520 con 0x7f49b80f7810 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.353+0000 7f49bcffa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49b80f8ab0 0x7f49b806b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.353+0000 7f49bcffa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49b80f8ab0 0x7f49b806b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38586/0 (socket says 172.21.15.113:38586) 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.353+0000 7f49bcffa700 1 -- 172.21.15.113:0/1893571127 learned_addr learned my addr 172.21.15.113:0/1893571127 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.353+0000 7f49bd7fb700 1 --2- 172.21.15.113:0/1893571127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49b80f7810 0x7f49b806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.354+0000 7f49bcffa700 1 -- 172.21.15.113:0/1893571127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49b80f7810 msgr2=0x7f49b806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.354+0000 7f49bcffa700 1 --2- 172.21.15.113:0/1893571127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49b80f7810 0x7f49b806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.354+0000 7f49bcffa700 1 -- 172.21.15.113:0/1893571127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b4005040 con 0x7f49b80f8ab0 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.354+0000 7f49bcffa700 1 --2- 172.21.15.113:0/1893571127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49b80f8ab0 0x7f49b806b030 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f49a800e8f0 tx=0x7f49a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.354+0000 7f49ae7fc700 1 -- 172.21.15.113:0/1893571127 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49a8007b40 con 0x7f49b80f8ab0 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.354+0000 7f49ae7fc700 1 -- 172.21.15.113:0/1893571127 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49a800f040 con 0x7f49b80f8ab0 2024-04-12T05:09:18.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.354+0000 7f49bfa5f700 1 -- 172.21.15.113:0/1893571127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b806b7a0 con 0x7f49b80f8ab0 2024-04-12T05:09:18.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.354+0000 7f49ae7fc700 1 -- 172.21.15.113:0/1893571127 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49a8014880 con 0x7f49b80f8ab0 2024-04-12T05:09:18.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.354+0000 7f49bfa5f700 1 -- 172.21.15.113:0/1893571127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b806bd60 con 0x7f49b80f8ab0 2024-04-12T05:09:18.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.355+0000 7f49bfa5f700 1 -- 172.21.15.113:0/1893571127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b80fffd0 con 0x7f49b80f8ab0 2024-04-12T05:09:18.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.355+0000 7f49ae7fc700 1 -- 172.21.15.113:0/1893571127 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7f49a8014ab0 con 0x7f49b80f8ab0 2024-04-12T05:09:18.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.356+0000 7f49ae7fc700 1 --2- 172.21.15.113:0/1893571127 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f49a4066dc0 0x7f49a4069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:18.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.356+0000 7f49ae7fc700 1 -- 172.21.15.113:0/1893571127 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f49a8083f00 con 0x7f49b80f8ab0 2024-04-12T05:09:18.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.356+0000 7f49bd7fb700 1 --2- 172.21.15.113:0/1893571127 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f49a4066dc0 0x7f49a4069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:09:18.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.356+0000 7f49bd7fb700 1 --2- 172.21.15.113:0/1893571127 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f49a4066dc0 0x7f49a4069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f49b40047c0 tx=0x7f49b4009cb0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:18.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.361+0000 7f49ae7fc700 1 -- 172.21.15.113:0/1893571127 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f49a8051fc0 con 0x7f49b80f8ab0 2024-04-12T05:09:18.509 INFO:tasks.workunit.client.1.smithi192.stdout:2/944: sync 2024-04-12T05:09:18.510 INFO:tasks.workunit.client.1.smithi192.stdout:9/836: sync 2024-04-12T05:09:18.512 INFO:tasks.workunit.client.1.smithi192.stdout:2/945: mkdir d1/d8/d17/d19/d71/d7f/d12e 0 2024-04-12T05:09:18.515 INFO:tasks.workunit.client.1.smithi192.stdout:9/837: rename d11/d25/dec/d1f/d50/c5b to d11/d25/dec/d1f/d6a/c12e 0 2024-04-12T05:09:18.521 INFO:tasks.workunit.client.1.smithi192.stdout:2/946: truncate d1/f70 2319399 0 2024-04-12T05:09:18.521 INFO:tasks.workunit.client.1.smithi192.stdout:9/838: write d11/d25/d36/d6e/d33/f55 [1034702,34023] 0 2024-04-12T05:09:18.522 INFO:tasks.workunit.client.1.smithi192.stdout:2/947: readlink d1/d8/d17/d19/d60/d59/ld4 0 2024-04-12T05:09:18.523 INFO:tasks.workunit.client.1.smithi192.stdout:2/948: chown d1/d8/d17/d19/d60/d59/c81 129874 1 2024-04-12T05:09:18.524 INFO:tasks.workunit.client.1.smithi192.stdout:9/839: mknod d11/d25/d60/c12f 0 2024-04-12T05:09:18.525 INFO:tasks.workunit.client.1.smithi192.stdout:9/840: read d11/d25/dd3/fe4 [3613743,33346] 0 2024-04-12T05:09:18.528 INFO:tasks.workunit.client.1.smithi192.stdout:2/949: rename d1/d8/d3a/de/d7c/d98/la4 to d1/d8/d3a/de/d7c/d98/l12f 0 2024-04-12T05:09:18.532 INFO:tasks.workunit.client.1.smithi192.stdout:2/950: symlink d1/d8/d17/d19/d71/da0/d12b/l130 0 2024-04-12T05:09:18.534 INFO:tasks.workunit.client.1.smithi192.stdout:2/951: stat d1/d8/d3a/l82 0 2024-04-12T05:09:18.535 INFO:tasks.workunit.client.1.smithi192.stdout:9/841: write d11/d25/d36/d6e/d33/da2/fe6 [580078,54384] 0 2024-04-12T05:09:18.538 INFO:tasks.workunit.client.1.smithi192.stdout:2/952: truncate d1/d8/d3a/de/f30 3517295 0 2024-04-12T05:09:18.541 INFO:tasks.workunit.client.1.smithi192.stdout:2/953: mknod d1/c131 0 2024-04-12T05:09:18.544 INFO:tasks.workunit.client.1.smithi192.stdout:2/954: unlink d1/d2d/d73/f9b 0 2024-04-12T05:09:18.544 INFO:tasks.workunit.client.1.smithi192.stdout:9/842: write d11/d25/dec/d1f/d50/d73/f80 [444516,124037] 0 2024-04-12T05:09:18.548 INFO:tasks.workunit.client.1.smithi192.stdout:9/843: read d11/d25/f70 [114830,89042] 0 2024-04-12T05:09:18.548 INFO:tasks.workunit.client.1.smithi192.stdout:2/955: creat d1/d8/d17/d19/f132 x:0 0 0 2024-04-12T05:09:18.553 INFO:tasks.workunit.client.1.smithi192.stdout:9/844: link d11/d25/d36/d6e/d71/c12a d11/d25/d36/d6e/d33/ddd/c130 0 2024-04-12T05:09:18.556 INFO:tasks.workunit.client.1.smithi192.stdout:2/956: dwrite d1/d8/d17/d19/fbb [0,4194304] 0 2024-04-12T05:09:18.556 INFO:tasks.workunit.client.1.smithi192.stdout:9/845: mknod d11/d25/d36/d6e/d33/df3/c131 0 2024-04-12T05:09:18.558 INFO:tasks.workunit.client.1.smithi192.stdout:9/846: chown d11/d25/dec/cbf 14247 1 2024-04-12T05:09:18.559 INFO:tasks.workunit.client.1.smithi192.stdout:9/847: symlink d11/d25/dec/d1f/l132 0 2024-04-12T05:09:18.560 INFO:tasks.workunit.client.1.smithi192.stdout:9/848: chown d11/d25/d36/d6e/d33/c8d 3358 1 2024-04-12T05:09:18.563 INFO:tasks.workunit.client.1.smithi192.stdout:2/957: truncate d1/d8/d3a/de/d7c/fb8 2407067 0 2024-04-12T05:09:18.565 INFO:tasks.workunit.client.1.smithi192.stdout:9/849: symlink d11/d15/d21/l133 0 2024-04-12T05:09:18.565 INFO:tasks.workunit.client.1.smithi192.stdout:2/958: stat d1/d8/d17/d19/d71/f104 0 2024-04-12T05:09:18.570 INFO:tasks.workunit.client.1.smithi192.stdout:9/850: dread - d11/d25/dec/d1f/d6a/ff2 zero size 2024-04-12T05:09:18.573 INFO:tasks.workunit.client.1.smithi192.stdout:2/959: rename d1/d2d/lb4 to d1/d2d/d73/da8/db2/l133 0 2024-04-12T05:09:18.575 INFO:tasks.workunit.client.1.smithi192.stdout:2/960: fdatasync d1/d8/d17/d19/d60/d9c/f9d 0 2024-04-12T05:09:18.579 INFO:tasks.workunit.client.1.smithi192.stdout:2/961: unlink d1/d8/d17/d19/d71/d7f/cc3 0 2024-04-12T05:09:18.580 INFO:tasks.workunit.client.1.smithi192.stdout:2/962: read d1/d2d/d73/f7a [457406,15306] 0 2024-04-12T05:09:18.589 INFO:tasks.workunit.client.1.smithi192.stdout:2/963: dwrite d1/d8/d3a/de/fca [0,4194304] 0 2024-04-12T05:09:18.602 INFO:tasks.workunit.client.1.smithi192.stdout:2/964: dwrite d1/da7/f101 [0,4194304] 0 2024-04-12T05:09:18.605 INFO:tasks.workunit.client.1.smithi192.stdout:2/965: mknod d1/d8/d17/d19/d71/d7f/d12e/c134 0 2024-04-12T05:09:18.608 INFO:tasks.workunit.client.1.smithi192.stdout:2/966: chown d1/d8/d17/d19/d60/d59/da3/fc9 3657428 1 2024-04-12T05:09:18.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.613+0000 7f49bfa5f700 1 -- 172.21.15.113:0/1893571127 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f49b8059120 con 0x7f49a4066dc0 2024-04-12T05:09:18.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.615+0000 7f49ae7fc700 1 -- 172.21.15.113:0/1893571127 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f49b8059120 con 0x7f49a4066dc0 2024-04-12T05:09:18.618 INFO:tasks.workunit.client.1.smithi192.stdout:2/967: dwrite d1/d8/d17/d19/d60/d59/da3/fde [0,4194304] 0 2024-04-12T05:09:18.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.618+0000 7f49bfa5f700 1 -- 172.21.15.113:0/1893571127 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f49a4066dc0 msgr2=0x7f49a4069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:18.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.618+0000 7f49bfa5f700 1 --2- 172.21.15.113:0/1893571127 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f49a4066dc0 0x7f49a4069280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f49b40047c0 tx=0x7f49b4009cb0).stop 2024-04-12T05:09:18.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.618+0000 7f49bfa5f700 1 -- 172.21.15.113:0/1893571127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49b80f8ab0 msgr2=0x7f49b806b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:18.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.618+0000 7f49bfa5f700 1 --2- 172.21.15.113:0/1893571127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49b80f8ab0 0x7f49b806b030 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f49a800e8f0 tx=0x7f49a800b040).stop 2024-04-12T05:09:18.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.618+0000 7f49bfa5f700 1 -- 172.21.15.113:0/1893571127 shutdown_connections 2024-04-12T05:09:18.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.618+0000 7f49bfa5f700 1 --2- 172.21.15.113:0/1893571127 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f49a4066dc0 0x7f49a4069280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:18.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.618+0000 7f49bfa5f700 1 --2- 172.21.15.113:0/1893571127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49b80f7810 0x7f49b806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:18.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.618+0000 7f49bfa5f700 1 --2- 172.21.15.113:0/1893571127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49b80f8ab0 0x7f49b806b030 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:18.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.618+0000 7f49bfa5f700 1 -- 172.21.15.113:0/1893571127 >> 172.21.15.113:0/1893571127 conn(0x7f49b80f2f50 msgr2=0x7f49b80fc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:18.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.618+0000 7f49bfa5f700 1 -- 172.21.15.113:0/1893571127 shutdown_connections 2024-04-12T05:09:18.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.618+0000 7f49bfa5f700 1 -- 172.21.15.113:0/1893571127 wait complete. 2024-04-12T05:09:18.621 INFO:tasks.workunit.client.1.smithi192.stdout:2/968: write d1/d8/d17/d19/f132 [524851,23586] 0 2024-04-12T05:09:18.622 INFO:tasks.workunit.client.1.smithi192.stdout:2/969: chown d1/d8/d17/d19/d60/d59/d68/dea 1564923 1 2024-04-12T05:09:18.623 INFO:tasks.workunit.client.1.smithi192.stdout:2/970: chown d1/d2d/d73/fac 715 1 2024-04-12T05:09:18.633 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:09:18.634 INFO:tasks.workunit.client.1.smithi192.stdout:2/971: dwrite d1/f4 [4194304,4194304] 0 2024-04-12T05:09:18.638 INFO:tasks.workunit.client.1.smithi192.stdout:2/972: mkdir d1/d8/d17/db3/dee/d135 0 2024-04-12T05:09:18.639 INFO:tasks.workunit.client.1.smithi192.stdout:2/973: creat d1/d2d/f136 x:0 0 0 2024-04-12T05:09:18.643 INFO:tasks.workunit.client.1.smithi192.stdout:7/914: sync 2024-04-12T05:09:18.654 INFO:tasks.workunit.client.1.smithi192.stdout:2/974: dwrite d1/d8/d3a/de/f97 [0,4194304] 0 2024-04-12T05:09:18.660 INFO:tasks.workunit.client.1.smithi192.stdout:7/915: dwrite d3/d1d/d5a/d5f/d7a/fc6 [0,4194304] 0 2024-04-12T05:09:18.663 INFO:tasks.workunit.client.1.smithi192.stdout:7/916: fsync d3/d1d/d74/d86/fed 0 2024-04-12T05:09:18.664 INFO:tasks.workunit.client.1.smithi192.stdout:7/917: stat d3/d7/d14/d1f/d76/lc4 0 2024-04-12T05:09:18.665 INFO:tasks.workunit.client.1.smithi192.stdout:7/918: read d3/d1d/f69 [3793485,60082] 0 2024-04-12T05:09:18.666 INFO:tasks.workunit.client.1.smithi192.stdout:2/975: rename d1/da7/c111 to d1/d8/d17/d19/d60/d59/d68/d105/c137 0 2024-04-12T05:09:18.670 INFO:tasks.workunit.client.1.smithi192.stdout:2/976: read d1/d8/d3a/f2e [2277137,64833] 0 2024-04-12T05:09:18.670 INFO:tasks.workunit.client.1.smithi192.stdout:7/919: mkdir d3/d1d/d5a/d5f/d7a/da6/d13b 0 2024-04-12T05:09:18.674 INFO:tasks.workunit.client.1.smithi192.stdout:7/920: rmdir d3/d1d/d4a/d5e/df9 39 2024-04-12T05:09:18.675 INFO:tasks.workunit.client.1.smithi192.stdout:7/921: dread - d3/d7/d60/d65/f12a zero size 2024-04-12T05:09:18.680 INFO:tasks.workunit.client.1.smithi192.stdout:7/922: rename d3/d1d/d4a/d5e/d6d/da3/fab to d3/d7/d60/d65/d11d/f13c 0 2024-04-12T05:09:18.683 INFO:tasks.workunit.client.1.smithi192.stdout:2/977: dwrite d1/f8b [0,4194304] 0 2024-04-12T05:09:18.685 INFO:tasks.workunit.client.1.smithi192.stdout:7/923: rename d3/d7/c79 to d3/d1d/d4a/d5e/d6d/c13d 0 2024-04-12T05:09:18.685 INFO:tasks.workunit.client.1.smithi192.stdout:2/978: write d1/d8/d3a/de/fca [2504505,18287] 0 2024-04-12T05:09:18.690 INFO:tasks.workunit.client.1.smithi192.stdout:2/979: rmdir d1/d8/d17/d19/d71/d7f/d12e 39 2024-04-12T05:09:18.692 INFO:tasks.workunit.client.1.smithi192.stdout:7/924: creat d3/d1d/d5a/d5f/d6a/de9/f13e x:0 0 0 2024-04-12T05:09:18.694 INFO:tasks.workunit.client.1.smithi192.stdout:2/980: rmdir d1/d8/d17/d19/d60/d59/d68/dea 39 2024-04-12T05:09:18.695 INFO:tasks.workunit.client.1.smithi192.stdout:7/925: mknod d3/d1d/d4a/d5e/d6d/da3/c13f 0 2024-04-12T05:09:18.696 INFO:tasks.workunit.client.1.smithi192.stdout:2/981: stat d1/d8/d17/d19/d60/d59/d68/fe9 0 2024-04-12T05:09:18.698 INFO:tasks.workunit.client.1.smithi192.stdout:7/926: chown d3/d1d/d5a/d5f/c6b 592184379 1 2024-04-12T05:09:18.700 INFO:tasks.workunit.client.1.smithi192.stdout:2/982: creat d1/d8/d17/d19/d60/f138 x:0 0 0 2024-04-12T05:09:18.701 INFO:tasks.workunit.client.1.smithi192.stdout:7/927: fdatasync d3/d1d/d4a/d5e/d6d/ffc 0 2024-04-12T05:09:18.702 INFO:tasks.workunit.client.1.smithi192.stdout:2/983: write d1/d8/d3a/f95 [3367407,53246] 0 2024-04-12T05:09:18.707 INFO:tasks.workunit.client.1.smithi192.stdout:7/928: creat d3/d1d/d4a/d5e/f140 x:0 0 0 2024-04-12T05:09:18.708 INFO:tasks.workunit.client.1.smithi192.stdout:2/984: truncate d1/f4 1899546 0 2024-04-12T05:09:18.712 INFO:tasks.workunit.client.1.smithi192.stdout:2/985: unlink d1/d8/f10 0 2024-04-12T05:09:18.714 INFO:tasks.workunit.client.1.smithi192.stdout:2/986: write d1/d8/d17/db3/f126 [503424,103076] 0 2024-04-12T05:09:18.716 INFO:tasks.workunit.client.1.smithi192.stdout:7/929: dwrite d3/d1d/d4a/d5e/df9/dfd/f125 [0,4194304] 0 2024-04-12T05:09:18.722 INFO:tasks.workunit.client.1.smithi192.stdout:2/987: chown d1/d8/d17/d19/d60/d59/d68/dea/l11e 8379697 1 2024-04-12T05:09:18.722 INFO:tasks.workunit.client.1.smithi192.stdout:7/930: truncate d3/d7/f127 4209550 0 2024-04-12T05:09:18.724 INFO:tasks.workunit.client.1.smithi192.stdout:7/931: symlink d3/d1d/d5a/d5f/d7a/da6/l141 0 2024-04-12T05:09:18.738 INFO:tasks.workunit.client.1.smithi192.stdout:7/932: dwrite d3/d11/f46 [0,4194304] 0 2024-04-12T05:09:18.742 INFO:tasks.workunit.client.1.smithi192.stdout:7/933: creat d3/d7/d14/d1f/f142 x:0 0 0 2024-04-12T05:09:18.743 INFO:tasks.workunit.client.1.smithi192.stdout:7/934: chown d3/d1d/d5a/d5f/c6b 5 1 2024-04-12T05:09:18.743 INFO:tasks.workunit.client.1.smithi192.stdout:7/935: fsync d3/d11/f12f 0 2024-04-12T05:09:18.745 INFO:tasks.workunit.client.1.smithi192.stdout:7/936: dread d3/d1d/d74/d86/d93/ff7 [0,4194304] 0 2024-04-12T05:09:18.751 INFO:tasks.workunit.client.1.smithi192.stdout:7/937: rmdir d3/d1d/d5a/d5f/d7a/da6/d13b 0 2024-04-12T05:09:18.753 INFO:tasks.workunit.client.1.smithi192.stdout:7/938: symlink d3/d1d/d4a/d5e/d12b/l143 0 2024-04-12T05:09:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.754+0000 7fd33e806700 1 -- 172.21.15.113:0/4064609505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3380f6450 msgr2=0x7fd3380f8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.754+0000 7fd33e806700 1 --2- 172.21.15.113:0/4064609505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3380f6450 0x7fd3380f8bd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fd3200047f0 tx=0x7fd320009cb0).stop 2024-04-12T05:09:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.754+0000 7fd33e806700 1 -- 172.21.15.113:0/4064609505 shutdown_connections 2024-04-12T05:09:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.754+0000 7fd33e806700 1 --2- 172.21.15.113:0/4064609505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3380f6450 0x7fd3380f8bd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.754+0000 7fd33e806700 1 --2- 172.21.15.113:0/4064609505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3380f5ab0 0x7fd3380f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.754+0000 7fd33e806700 1 -- 172.21.15.113:0/4064609505 >> 172.21.15.113:0/4064609505 conn(0x7fd3380f15d0 msgr2=0x7fd3380f39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.754+0000 7fd33e806700 1 -- 172.21.15.113:0/4064609505 shutdown_connections 2024-04-12T05:09:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.754+0000 7fd33e806700 1 -- 172.21.15.113:0/4064609505 wait complete. 2024-04-12T05:09:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.754+0000 7fd33e806700 1 Processor -- start 2024-04-12T05:09:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.754+0000 7fd33e806700 1 -- start start 2024-04-12T05:09:18.757 INFO:tasks.workunit.client.1.smithi192.stdout:7/939: mknod d3/d1d/d74/c144 0 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd33e806700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3380f5ab0 0x7fd338107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd33e806700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3380f6450 0x7fd3381084b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd33e806700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd338108a60 con 0x7fd3380f6450 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd33e806700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd338108ba0 con 0x7fd3380f5ab0 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd3377fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3380f6450 0x7fd3381084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd3377fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3380f6450 0x7fd3381084b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38604/0 (socket says 172.21.15.113:38604) 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd3377fe700 1 -- 172.21.15.113:0/619419515 learned_addr learned my addr 172.21.15.113:0/619419515 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd337fff700 1 --2- 172.21.15.113:0/619419515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3380f5ab0 0x7fd338107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd3377fe700 1 -- 172.21.15.113:0/619419515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3380f5ab0 msgr2=0x7fd338107fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd3377fe700 1 --2- 172.21.15.113:0/619419515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3380f5ab0 0x7fd338107fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd3377fe700 1 -- 172.21.15.113:0/619419515 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd320005040 con 0x7fd3380f6450 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.755+0000 7fd3377fe700 1 --2- 172.21.15.113:0/619419515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3380f6450 0x7fd3381084b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fd320013620 tx=0x7fd320009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.756+0000 7fd3357fa700 1 -- 172.21.15.113:0/619419515 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd32000e070 con 0x7fd3380f6450 2024-04-12T05:09:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.756+0000 7fd33e806700 1 -- 172.21.15.113:0/619419515 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33810d5e0 con 0x7fd3380f6450 2024-04-12T05:09:18.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.756+0000 7fd33e806700 1 -- 172.21.15.113:0/619419515 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33810dba0 con 0x7fd3380f6450 2024-04-12T05:09:18.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.756+0000 7fd3357fa700 1 -- 172.21.15.113:0/619419515 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd320007e90 con 0x7fd3380f6450 2024-04-12T05:09:18.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.756+0000 7fd3357fa700 1 -- 172.21.15.113:0/619419515 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd320018070 con 0x7fd3380f6450 2024-04-12T05:09:18.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.757+0000 7fd33e806700 1 -- 172.21.15.113:0/619419515 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd318005120 con 0x7fd3380f6450 2024-04-12T05:09:18.761 INFO:tasks.workunit.client.1.smithi192.stdout:7/940: mkdir d3/d1d/d4a/d5e/d6d/da3/dce/d145 0 2024-04-12T05:09:18.761 INFO:tasks.workunit.client.1.smithi192.stdout:7/941: dread - d3/d7/d14/d1f/f142 zero size 2024-04-12T05:09:18.761 INFO:tasks.workunit.client.1.smithi192.stdout:7/942: write d3/d7/d15/d1a/f3e [4297725,54710] 0 2024-04-12T05:09:18.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.758+0000 7fd3357fa700 1 -- 172.21.15.113:0/619419515 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7fd32001d420 con 0x7fd3380f6450 2024-04-12T05:09:18.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.758+0000 7fd3357fa700 1 --2- 172.21.15.113:0/619419515 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fd324066dc0 0x7fd324069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:18.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.758+0000 7fd3357fa700 1 -- 172.21.15.113:0/619419515 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7fd320085070 con 0x7fd3380f6450 2024-04-12T05:09:18.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.759+0000 7fd337fff700 1 --2- 172.21.15.113:0/619419515 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fd324066dc0 0x7fd324069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:09:18.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.759+0000 7fd337fff700 1 --2- 172.21.15.113:0/619419515 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fd324066dc0 0x7fd324069280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fd33805fad0 tx=0x7fd328009d80).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:18.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:18.764+0000 7fd3357fa700 1 -- 172.21.15.113:0/619419515 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fd320053110 con 0x7fd3380f6450 2024-04-12T05:09:18.796 INFO:tasks.workunit.client.1.smithi192.stdout:7/943: mkdir d3/d1d/d5a/d5f/d7a/d146 0 2024-04-12T05:09:18.817 INFO:tasks.workunit.client.1.smithi192.stdout:7/944: dwrite d3/d1d/d4a/faa [0,4194304] 0 2024-04-12T05:09:18.822 INFO:tasks.workunit.client.1.smithi192.stdout:7/945: creat d3/d11/f147 x:0 0 0 2024-04-12T05:09:18.833 INFO:tasks.workunit.client.1.smithi192.stdout:7/946: dwrite d3/d11/f3b [4194304,4194304] 0 2024-04-12T05:09:18.839 INFO:tasks.workunit.client.1.smithi192.stdout:7/947: rename d3/d11/f37 to d3/d7/d15/f148 0 2024-04-12T05:09:18.841 INFO:tasks.workunit.client.1.smithi192.stdout:7/948: creat d3/d7/d15/d1a/df2/f149 x:0 0 0 2024-04-12T05:09:18.854 INFO:tasks.workunit.client.1.smithi192.stdout:7/949: dwrite d3/d1d/fe1 [0,4194304] 0 2024-04-12T05:09:18.862 INFO:tasks.workunit.client.1.smithi192.stdout:7/950: dread d3/d1d/d5a/d5f/d7a/fc6 [0,4194304] 0 2024-04-12T05:09:18.862 INFO:tasks.workunit.client.1.smithi192.stdout:7/951: dread - d3/d7/d15/d22/f11e zero size 2024-04-12T05:09:18.867 INFO:tasks.workunit.client.1.smithi192.stdout:7/952: truncate d3/d7/d15/d1a/fd7 2701691 0 2024-04-12T05:09:18.871 INFO:tasks.workunit.client.1.smithi192.stdout:7/953: rename d3/d7/d15/d1a/df2/cec to d3/d11/c14a 0 2024-04-12T05:09:18.873 INFO:tasks.workunit.client.1.smithi192.stdout:7/954: symlink d3/d1d/d5a/d5f/l14b 0 2024-04-12T05:09:18.876 INFO:tasks.workunit.client.1.smithi192.stdout:7/955: truncate d3/d7/f19 3652621 0 2024-04-12T05:09:18.883 INFO:tasks.workunit.client.1.smithi192.stdout:7/956: dwrite d3/d1d/d5a/d5f/d7a/fc6 [0,4194304] 0 2024-04-12T05:09:18.897 INFO:tasks.workunit.client.1.smithi192.stdout:7/957: write d3/d7/d15/f148 [353117,90396] 0 2024-04-12T05:09:18.900 INFO:tasks.workunit.client.1.smithi192.stdout:7/958: truncate d3/d1d/d74/d86/d93/ff7 208953 0 2024-04-12T05:09:18.902 INFO:tasks.workunit.client.1.smithi192.stdout:7/959: symlink d3/d1d/d4a/d5e/d6d/da3/l14c 0 2024-04-12T05:09:18.905 INFO:tasks.workunit.client.1.smithi192.stdout:7/960: read d3/d1d/d4a/d5e/d6d/da3/fc1 [1228254,21952] 0 2024-04-12T05:09:18.907 INFO:tasks.workunit.client.1.smithi192.stdout:7/961: fdatasync d3/d7/d60/d65/f104 0 2024-04-12T05:09:18.909 INFO:tasks.workunit.client.1.smithi192.stdout:7/962: unlink d3/d7/d60/d65/f8a 0 2024-04-12T05:09:18.911 INFO:tasks.workunit.client.1.smithi192.stdout:7/963: mknod d3/d1d/d48/d114/c14d 0 2024-04-12T05:09:18.917 INFO:tasks.workunit.client.1.smithi192.stdout:7/964: dwrite d3/d7/d15/f50 [4194304,4194304] 0 2024-04-12T05:09:18.920 INFO:tasks.workunit.client.1.smithi192.stdout:7/965: creat d3/d1d/d5a/d5f/d6a/de9/f14e x:0 0 0 2024-04-12T05:09:18.924 INFO:tasks.workunit.client.1.smithi192.stdout:7/966: rename d3/d1d/d5a/d5f/d6a/fc7 to d3/d7/dff/d136/f14f 0 2024-04-12T05:09:18.926 INFO:tasks.workunit.client.1.smithi192.stdout:7/967: chown d3/d7/f56 5426350 1 2024-04-12T05:09:18.927 INFO:tasks.workunit.client.1.smithi192.stdout:7/968: dread - d3/d1d/d74/d86/fe3 zero size 2024-04-12T05:09:18.928 INFO:tasks.workunit.client.1.smithi192.stdout:7/969: chown d3/d1d/d4a/d5e/df9/dfd/f125 9 1 2024-04-12T05:09:18.944 INFO:tasks.workunit.client.1.smithi192.stdout:7/970: dwrite d3/d7/d15/d22/fb6 [0,4194304] 0 2024-04-12T05:09:18.948 INFO:tasks.workunit.client.1.smithi192.stdout:7/971: fsync d3/d1d/d5a/d5f/fbd 0 2024-04-12T05:09:18.951 INFO:tasks.workunit.client.1.smithi192.stdout:7/972: truncate d3/d7/d14/d1f/f142 528749 0 2024-04-12T05:09:18.951 INFO:tasks.workunit.client.1.smithi192.stdout:7/973: fsync d3/d1d/d4a/faa 0 2024-04-12T05:09:18.955 INFO:tasks.workunit.client.1.smithi192.stdout:7/974: symlink d3/d11/l150 0 2024-04-12T05:09:18.955 INFO:tasks.workunit.client.1.smithi192.stdout:7/975: readlink d3/d7/l52 0 2024-04-12T05:09:18.956 INFO:tasks.workunit.client.1.smithi192.stdout:7/976: stat d3/d1d/d5a/f11f 0 2024-04-12T05:09:18.960 INFO:tasks.workunit.client.1.smithi192.stdout:7/977: creat d3/d1d/d5a/d5f/f151 x:0 0 0 2024-04-12T05:09:18.961 INFO:tasks.workunit.client.1.smithi192.stdout:7/978: write d3/d7/d15/d22/fb6 [1602922,1496] 0 2024-04-12T05:09:18.968 INFO:tasks.workunit.client.1.smithi192.stdout:7/979: dwrite d3/d7/d15/d22/fb6 [0,4194304] 0 2024-04-12T05:09:18.972 INFO:tasks.workunit.client.1.smithi192.stdout:7/980: symlink d3/d1d/d48/d114/l152 0 2024-04-12T05:09:18.975 INFO:tasks.workunit.client.1.smithi192.stdout:7/981: fsync d3/d1d/d4a/d5e/d6d/f96 0 2024-04-12T05:09:18.981 INFO:tasks.workunit.client.1.smithi192.stdout:7/982: dwrite d3/d7/d60/ff1 [0,4194304] 0 2024-04-12T05:09:19.002 INFO:tasks.workunit.client.1.smithi192.stdout:7/983: dread d3/d7/f10 [0,4194304] 0 2024-04-12T05:09:19.021 INFO:tasks.workunit.client.1.smithi192.stdout:7/984: dwrite d3/d1d/d74/d86/d93/fd9 [0,4194304] 0 2024-04-12T05:09:19.021 INFO:tasks.workunit.client.1.smithi192.stdout:9/851: sync 2024-04-12T05:09:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.026+0000 7fd33e806700 1 -- 172.21.15.113:0/619419515 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd318000bc0 con 0x7fd324066dc0 2024-04-12T05:09:19.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.029+0000 7fd3357fa700 1 -- 172.21.15.113:0/619419515 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fd318000bc0 con 0x7fd324066dc0 2024-04-12T05:09:19.032 INFO:tasks.workunit.client.1.smithi192.stdout:7/985: link d3/d1d/fe1 d3/d1d/d74/f153 0 2024-04-12T05:09:19.032 INFO:tasks.workunit.client.1.smithi192.stdout:9/852: rename d11/laa to d11/d25/dec/d1f/d6a/d92/da6/l134 0 2024-04-12T05:09:19.032 INFO:tasks.workunit.client.1.smithi192.stdout:7/986: mkdir d3/d1d/d4a/d5e/d6d/da3/dce/dee/d154 0 2024-04-12T05:09:19.033 INFO:tasks.workunit.client.1.smithi192.stdout:7/987: rmdir d3/d1d/d74/d86/d93/db9 39 2024-04-12T05:09:19.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.031+0000 7fd33e806700 1 -- 172.21.15.113:0/619419515 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fd324066dc0 msgr2=0x7fd324069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.031+0000 7fd33e806700 1 --2- 172.21.15.113:0/619419515 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fd324066dc0 0x7fd324069280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fd33805fad0 tx=0x7fd328009d80).stop 2024-04-12T05:09:19.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.031+0000 7fd33e806700 1 -- 172.21.15.113:0/619419515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3380f6450 msgr2=0x7fd3381084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.031+0000 7fd33e806700 1 --2- 172.21.15.113:0/619419515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3380f6450 0x7fd3381084b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fd320013620 tx=0x7fd320009cb0).stop 2024-04-12T05:09:19.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.032+0000 7fd33e806700 1 -- 172.21.15.113:0/619419515 shutdown_connections 2024-04-12T05:09:19.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.032+0000 7fd33e806700 1 --2- 172.21.15.113:0/619419515 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fd324066dc0 0x7fd324069280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.032+0000 7fd33e806700 1 --2- 172.21.15.113:0/619419515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3380f5ab0 0x7fd338107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.032+0000 7fd33e806700 1 --2- 172.21.15.113:0/619419515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3380f6450 0x7fd3381084b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.032+0000 7fd33e806700 1 -- 172.21.15.113:0/619419515 >> 172.21.15.113:0/619419515 conn(0x7fd3380f15d0 msgr2=0x7fd3380fc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:19.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.032+0000 7fd33e806700 1 -- 172.21.15.113:0/619419515 shutdown_connections 2024-04-12T05:09:19.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.032+0000 7fd33e806700 1 -- 172.21.15.113:0/619419515 wait complete. 2024-04-12T05:09:19.038 INFO:tasks.workunit.client.1.smithi192.stdout:7/988: rmdir d3/d1d/d74 39 2024-04-12T05:09:19.041 INFO:tasks.workunit.client.1.smithi192.stdout:9/853: dread - d11/d25/d36/d6e/f30 zero size 2024-04-12T05:09:19.045 INFO:tasks.workunit.client.1.smithi192.stdout:9/854: symlink d11/d25/dd3/d10d/l135 0 2024-04-12T05:09:19.046 INFO:tasks.workunit.client.1.smithi192.stdout:9/855: dread - d11/d25/d36/d6e/fd2 zero size 2024-04-12T05:09:19.053 INFO:tasks.workunit.client.1.smithi192.stdout:9/856: symlink d11/d25/dd3/l136 0 2024-04-12T05:09:19.060 INFO:tasks.workunit.client.1.smithi192.stdout:9/857: mkdir d11/d25/d36/d6e/d33/df3/d137 0 2024-04-12T05:09:19.061 INFO:tasks.workunit.client.1.smithi192.stdout:9/858: chown fe 5531 1 2024-04-12T05:09:19.070 INFO:tasks.workunit.client.1.smithi192.stdout:9/859: unlink d11/d25/dec/d1f/d6a/d7e/cd4 0 2024-04-12T05:09:19.073 INFO:tasks.workunit.client.1.smithi192.stdout:2/988: sync 2024-04-12T05:09:19.077 INFO:tasks.workunit.client.1.smithi192.stdout:7/989: dread d3/d7/fdf [0,4194304] 0 2024-04-12T05:09:19.078 INFO:tasks.workunit.client.1.smithi192.stdout:2/989: truncate d1/d8/f84 805455 0 2024-04-12T05:09:19.078 INFO:tasks.workunit.client.1.smithi192.stdout:7/990: symlink d3/d1d/d48/l155 0 2024-04-12T05:09:19.082 INFO:tasks.workunit.client.1.smithi192.stdout:7/991: stat d3/d7/d15/f3c 0 2024-04-12T05:09:19.086 INFO:tasks.workunit.client.1.smithi192.stdout:2/990: mkdir d1/d8/d17/d19/d60/d139 0 2024-04-12T05:09:19.090 INFO:tasks.workunit.client.1.smithi192.stdout:7/992: rmdir d3/d7/d15/d1a/ddc 39 2024-04-12T05:09:19.094 INFO:tasks.workunit.client.1.smithi192.stdout:2/991: write d1/d8/d3a/de/d7c/f7d [759623,28850] 0 2024-04-12T05:09:19.098 INFO:tasks.workunit.client.1.smithi192.stdout:7/993: chown d3/d1d/d74/d86/lc8 97266 1 2024-04-12T05:09:19.105 INFO:tasks.workunit.client.1.smithi192.stdout:9/860: dread d11/d25/d36/d3d/f5e [0,4194304] 0 2024-04-12T05:09:19.105 INFO:tasks.workunit.client.1.smithi192.stdout:7/994: dwrite d3/d7/d60/d65/f12a [0,4194304] 0 2024-04-12T05:09:19.107 INFO:tasks.workunit.client.1.smithi192.stdout:9/861: dread - d11/d25/d36/d6e/d8f/d120/df9/f102 zero size 2024-04-12T05:09:19.108 INFO:tasks.workunit.client.1.smithi192.stdout:2/992: write d1/d8/d17/db3/fc8 [663904,117390] 0 2024-04-12T05:09:19.113 INFO:tasks.workunit.client.1.smithi192.stdout:9/862: unlink f6 0 2024-04-12T05:09:19.115 INFO:tasks.workunit.client.1.smithi192.stdout:7/995: dwrite d3/d7/d15/d1a/df2/f11c [0,4194304] 0 2024-04-12T05:09:19.124 INFO:tasks.workunit.client.1.smithi192.stdout:2/993: write d1/d8/d17/f2a [319137,55463] 0 2024-04-12T05:09:19.124 INFO:tasks.workunit.client.1.smithi192.stdout:9/863: truncate d11/d25/d36/d6e/d8f/d120/df9/f118 2351141 0 2024-04-12T05:09:19.125 INFO:tasks.workunit.client.1.smithi192.stdout:7/996: creat d3/d1d/d5a/d5f/d7a/d146/f156 x:0 0 0 2024-04-12T05:09:19.127 INFO:tasks.workunit.client.1.smithi192.stdout:7/997: chown d3/d1d/d5a/d5f/c6b 212497 1 2024-04-12T05:09:19.129 INFO:tasks.workunit.client.1.smithi192.stdout:2/994: symlink d1/d8/d17/d19/d60/d59/l13a 0 2024-04-12T05:09:19.132 INFO:tasks.workunit.client.1.smithi192.stdout:7/998: creat d3/d1d/d4a/d5e/f157 x:0 0 0 2024-04-12T05:09:19.136 INFO:tasks.workunit.client.1.smithi192.stdout:2/995: dwrite d1/d8/d17/db3/f126 [0,4194304] 0 2024-04-12T05:09:19.144 INFO:tasks.workunit.client.1.smithi192.stdout:2/996: getdents d1/d8/d17/d19 0 2024-04-12T05:09:19.149 INFO:tasks.workunit.client.1.smithi192.stdout:9/864: symlink d11/d25/d36/d6e/d33/l138 0 2024-04-12T05:09:19.149 INFO:tasks.workunit.client.1.smithi192.stdout:2/997: mkdir d1/d8/d3a/de/d7c/d11d/d13b 0 2024-04-12T05:09:19.150 INFO:tasks.workunit.client.1.smithi192.stdout:2/998: chown d1/d8/d17/d19/d60 0 1 2024-04-12T05:09:19.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.148+0000 7fb69b7fb700 1 -- 172.21.15.113:0/931710539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6940f6a60 msgr2=0x7fb6940f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.148+0000 7fb69b7fb700 1 --2- 172.21.15.113:0/931710539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6940f6a60 0x7fb6940f6ee0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fb6900047f0 tx=0x7fb690009cb0).stop 2024-04-12T05:09:19.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.148+0000 7fb69b7fb700 1 -- 172.21.15.113:0/931710539 shutdown_connections 2024-04-12T05:09:19.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.148+0000 7fb69b7fb700 1 --2- 172.21.15.113:0/931710539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6940f6a60 0x7fb6940f6ee0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.148+0000 7fb69b7fb700 1 --2- 172.21.15.113:0/931710539 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6940f7d10 0x7fb6940f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:09:19.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.148+0000 7fb69b7fb700 1 -- 172.21.15.113:0/931710539 >> 172.21.15.113:0/931710539 conn(0x7fb6940f1610 msgr2=0x7fb6940f3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:19.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.148+0000 7fb69b7fb700 1 -- 172.21.15.113:0/931710539 shutdown_connections 2024-04-12T05:09:19.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.148+0000 7fb69b7fb700 1 -- 172.21.15.113:0/931710539 wait complete. 2024-04-12T05:09:19.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.149+0000 7fb69b7fb700 1 Processor -- start 2024-04-12T05:09:19.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.149+0000 7fb69b7fb700 1 -- start start 2024-04-12T05:09:19.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.149+0000 7fb69b7fb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6940f6a60 0x7fb694106f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:19.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.149+0000 7fb69b7fb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6940f7d10 0x7fb694107430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:19.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.149+0000 7fb69b7fb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb694107a70 con 0x7fb6940f6a60 2024-04-12T05:09:19.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.149+0000 7fb69b7fb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb694107bb0 con 0x7fb6940f7d10 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.149+0000 7fb699597700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6940f6a60 0x7fb694106f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.149+0000 7fb699597700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6940f6a60 0x7fb694106f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38628/0 (socket says 172.21.15.113:38628) 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.149+0000 7fb699597700 1 -- 172.21.15.113:0/1030730173 learned_addr learned my addr 172.21.15.113:0/1030730173 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.150+0000 7fb699597700 1 -- 172.21.15.113:0/1030730173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6940f7d10 msgr2=0x7fb694107430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.150+0000 7fb699597700 1 --2- 172.21.15.113:0/1030730173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6940f7d10 0x7fb694107430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.150+0000 7fb699597700 1 -- 172.21.15.113:0/1030730173 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb690005040 con 0x7fb6940f6a60 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.150+0000 7fb699597700 1 --2- 172.21.15.113:0/1030730173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6940f6a60 0x7fb694106f40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fb68800b3f0 tx=0x7fb688006f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.150+0000 7fb6867fc700 1 -- 172.21.15.113:0/1030730173 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb68800bea0 con 0x7fb6940f6a60 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.150+0000 7fb6867fc700 1 -- 172.21.15.113:0/1030730173 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb688012aa0 con 0x7fb6940f6a60 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.150+0000 7fb69b7fb700 1 -- 172.21.15.113:0/1030730173 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69410c650 con 0x7fb6940f6a60 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.150+0000 7fb6867fc700 1 -- 172.21.15.113:0/1030730173 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb68800db60 con 0x7fb6940f6a60 2024-04-12T05:09:19.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.150+0000 7fb69b7fb700 1 -- 172.21.15.113:0/1030730173 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69410cc10 con 0x7fb6940f6a60 2024-04-12T05:09:19.154 INFO:tasks.workunit.client.1.smithi192.stdout:9/865: creat d11/d15/d11f/f139 x:0 0 0 2024-04-12T05:09:19.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.151+0000 7fb69b7fb700 1 -- 172.21.15.113:0/1030730173 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6940fefa0 con 0x7fb6940f6a60 2024-04-12T05:09:19.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.152+0000 7fb6867fc700 1 -- 172.21.15.113:0/1030730173 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7fb68800dda0 con 0x7fb6940f6a60 2024-04-12T05:09:19.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.152+0000 7fb6867fc700 1 --2- 172.21.15.113:0/1030730173 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb680066dc0 0x7fb680069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:19.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.152+0000 7fb6867fc700 1 -- 172.21.15.113:0/1030730173 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7fb6880840c0 con 0x7fb6940f6a60 2024-04-12T05:09:19.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.152+0000 7fb698d96700 1 --2- 172.21.15.113:0/1030730173 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb680066dc0 0x7fb680069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:09:19.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.153+0000 7fb698d96700 1 --2- 172.21.15.113:0/1030730173 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb680066dc0 0x7fb680069280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb690013560 tx=0x7fb690009cb0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:19.155 INFO:tasks.workunit.client.1.smithi192.stdout:2/999: rename d1/d2d/d73/fac to d1/d8/d17/d19/d60/d59/f13c 0 2024-04-12T05:09:19.159 INFO:tasks.workunit.client.1.smithi192.stdout:9/866: rename d11/d15/d21/cd0 to d11/d25/d36/d3d/c13a 0 2024-04-12T05:09:19.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.159+0000 7fb6867fc700 1 -- 172.21.15.113:0/1030730173 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fb688052270 con 0x7fb6940f6a60 2024-04-12T05:09:19.161 INFO:tasks.workunit.client.1.smithi192.stdout:9/867: creat d11/d25/f13b x:0 0 0 2024-04-12T05:09:19.164 INFO:tasks.workunit.client.1.smithi192.stdout:9/868: creat d11/d25/dec/d1f/f13c x:0 0 0 2024-04-12T05:09:19.166 INFO:tasks.workunit.client.1.smithi192.stdout:9/869: creat d11/d25/dec/d1f/d50/f13d x:0 0 0 2024-04-12T05:09:19.167 INFO:tasks.workunit.client.1.smithi192.stdout:9/870: chown d11/d25/d36/d6e/d33/da2/fe6 61 1 2024-04-12T05:09:19.169 INFO:tasks.workunit.client.1.smithi192.stdout:9/871: creat d11/d25/d36/d6e/dc7/f13e x:0 0 0 2024-04-12T05:09:19.181 INFO:tasks.workunit.client.1.smithi192.stdout:9/872: dwrite d11/d15/d21/fb0 [0,4194304] 0 2024-04-12T05:09:19.184 INFO:tasks.workunit.client.1.smithi192.stdout:9/873: creat d11/d25/d36/d6e/d33/dbb/f13f x:0 0 0 2024-04-12T05:09:19.199 INFO:tasks.workunit.client.1.smithi192.stdout:9/874: dwrite d11/d25/d36/d6e/f3e [0,4194304] 0 2024-04-12T05:09:19.208 INFO:tasks.workunit.client.1.smithi192.stdout:9/875: dwrite d11/d25/d36/d6e/dc7/f13e [0,4194304] 0 2024-04-12T05:09:19.214 INFO:tasks.workunit.client.1.smithi192.stdout:9/876: rename d11/d15 to d11/d25/d36/d6e/d33/da2/ddf/d10f/d140 0 2024-04-12T05:09:19.231 INFO:tasks.workunit.client.1.smithi192.stdout:9/877: rmdir d11/d25/d60 39 2024-04-12T05:09:19.234 INFO:tasks.workunit.client.1.smithi192.stdout:9/878: creat d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d11f/f141 x:0 0 0 2024-04-12T05:09:19.235 INFO:tasks.workunit.client.1.smithi192.stdout:9/879: chown d11/f69 1858 1 2024-04-12T05:09:19.240 INFO:tasks.workunit.client.1.smithi192.stdout:9/880: creat d11/d25/d36/d6e/d8f/d9c/f142 x:0 0 0 2024-04-12T05:09:19.243 INFO:tasks.workunit.client.1.smithi192.stdout:9/881: symlink d11/d25/d36/d6e/d33/dbb/df5/d123/l143 0 2024-04-12T05:09:19.244 INFO:tasks.workunit.client.1.smithi192.stdout:9/882: chown d11/l51 527375845 1 2024-04-12T05:09:19.277 INFO:tasks.workunit.client.1.smithi192.stdout:9/883: rename d11/d25/d36/d3d/fb3 to d11/d25/dd3/d10d/dce/f144 0 2024-04-12T05:09:19.279 INFO:tasks.workunit.client.1.smithi192.stdout:9/884: write d11/d25/d36/d6e/d8f/fbd [5133548,95005] 0 2024-04-12T05:09:19.282 INFO:tasks.workunit.client.1.smithi192.stdout:9/885: creat d11/d25/d60/f145 x:0 0 0 2024-04-12T05:09:19.285 INFO:tasks.workunit.client.1.smithi192.stdout:9/886: rmdir d11/d25/dec/d1f/d50/d73 39 2024-04-12T05:09:19.290 INFO:tasks.workunit.client.1.smithi192.stdout:9/887: dwrite d11/d25/d36/d3d/f79 [8388608,4194304] 0 2024-04-12T05:09:19.297 INFO:tasks.workunit.client.1.smithi192.stdout:9/888: fdatasync d11/d25/d36/d6e/d8f/d120/df9/f126 0 2024-04-12T05:09:19.305 INFO:tasks.workunit.client.1.smithi192.stdout:9/889: write d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d21/f6f [208277,63845] 0 2024-04-12T05:09:19.312 INFO:tasks.workunit.client.1.smithi192.stdout:9/890: dwrite d11/d25/dec/d1f/f67 [4194304,4194304] 0 2024-04-12T05:09:19.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.401+0000 7fb69b7fb700 1 -- 172.21.15.113:0/1030730173 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb694059120 con 0x7fb680066dc0 2024-04-12T05:09:19.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.417+0000 7fb6867fc700 1 -- 172.21.15.113:0/1030730173 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7fb694059120 con 0x7fb680066dc0 2024-04-12T05:09:19.419 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:09:19.420 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (7m) 39s ago 8m 26.2M - 0.23.0 ba2b418f427c aae81b8048e6 2024-04-12T05:09:19.420 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (8m) 39s ago 8m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:09:19.420 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (7m) 14s ago 7m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:09:19.420 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (6m) 39s ago 8m 60.7M - 8.3.5 dad864ee21e9 7d83fd029dea 2024-04-12T05:09:19.420 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (3m) 39s ago 3m 97.7M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:09:19.420 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (2m) 39s ago 2m 15.4M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:09:19.421 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (3m) 14s ago 3m 109M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:09:19.421 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (2m) 14s ago 2m 12.9M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:09:19.421 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:9283 running (9m) 39s ago 9m 382M - 16.2.15-1-g8963e9de 727f386679cb 3fab7f291e6b 2024-04-12T05:09:19.421 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283 running (58s) 14s ago 7m 479M - 18.2.2-1187-gd540ebac bb2c5eac26a4 550d8cc53ae9 2024-04-12T05:09:19.421 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (9m) 39s ago 9m 49.3M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:09:19.421 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (7m) 14s ago 7m 41.4M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:09:19.422 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (8m) 39s ago 8m 22.5M - 1.3.1 1dbe0e931976 046bcfb16579 2024-04-12T05:09:19.422 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (7m) 14s ago 7m 28.1M - 1.3.1 1dbe0e931976 bc604318825c 2024-04-12T05:09:19.422 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (6m) 39s ago 6m 109M 1910M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:09:19.422 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (6m) 39s ago 6m 124M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:09:19.422 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (5m) 39s ago 5m 102M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:09:19.423 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (5m) 14s ago 5m 130M 4982M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:09:19.423 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (5m) 14s ago 5m 121M 4982M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:09:19.423 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (4m) 14s ago 4m 110M 4982M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:09:19.423 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (7m) 14s ago 7m 62.7M - 2.33.4 514e6a882f6e b4259537bd55 2024-04-12T05:09:19.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.419+0000 7fb69b7fb700 1 -- 172.21.15.113:0/1030730173 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb680066dc0 msgr2=0x7fb680069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.419+0000 7fb69b7fb700 1 --2- 172.21.15.113:0/1030730173 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb680066dc0 0x7fb680069280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb690013560 tx=0x7fb690009cb0).stop 2024-04-12T05:09:19.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.419+0000 7fb69b7fb700 1 -- 172.21.15.113:0/1030730173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6940f6a60 msgr2=0x7fb694106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.419+0000 7fb69b7fb700 1 --2- 172.21.15.113:0/1030730173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6940f6a60 0x7fb694106f40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fb68800b3f0 tx=0x7fb688006f90).stop 2024-04-12T05:09:19.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.419+0000 7fb69b7fb700 1 -- 172.21.15.113:0/1030730173 shutdown_connections 2024-04-12T05:09:19.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.419+0000 7fb69b7fb700 1 --2- 172.21.15.113:0/1030730173 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb680066dc0 0x7fb680069280 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.419+0000 7fb69b7fb700 1 --2- 172.21.15.113:0/1030730173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6940f6a60 0x7fb694106f40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.419+0000 7fb69b7fb700 1 --2- 172.21.15.113:0/1030730173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6940f7d10 0x7fb694107430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:09:19.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.419+0000 7fb69b7fb700 1 -- 172.21.15.113:0/1030730173 >> 172.21.15.113:0/1030730173 conn(0x7fb6940f1610 msgr2=0x7fb6940fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:19.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.419+0000 7fb69b7fb700 1 -- 172.21.15.113:0/1030730173 shutdown_connections 2024-04-12T05:09:19.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.419+0000 7fb69b7fb700 1 -- 172.21.15.113:0/1030730173 wait complete. 2024-04-12T05:09:19.520 INFO:tasks.workunit.client.1.smithi192.stdout:7/999: sync 2024-04-12T05:09:19.520 INFO:tasks.workunit.client.1.smithi192.stdout:9/891: sync 2024-04-12T05:09:19.523 INFO:tasks.workunit.client.1.smithi192.stdout:9/892: creat d11/d25/dec/d1f/d6a/f146 x:0 0 0 2024-04-12T05:09:19.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.523+0000 7fba0a9e2700 1 -- 172.21.15.113:0/793675278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba040fa950 msgr2=0x7fba040fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.523+0000 7fba0a9e2700 1 --2- 172.21.15.113:0/793675278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba040fa950 0x7fba040fcda0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fb9f80047f0 tx=0x7fb9f8009cb0).stop 2024-04-12T05:09:19.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.523+0000 7fba0a9e2700 1 -- 172.21.15.113:0/793675278 shutdown_connections 2024-04-12T05:09:19.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.523+0000 7fba0a9e2700 1 --2- 172.21.15.113:0/793675278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba040fa950 0x7fba040fcda0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.523+0000 7fba0a9e2700 1 --2- 172.21.15.113:0/793675278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba04061280 0x7fba040fa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.523+0000 7fba0a9e2700 1 -- 172.21.15.113:0/793675278 >> 172.21.15.113:0/793675278 conn(0x7fba040f1e50 msgr2=0x7fba040f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:19.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.523+0000 7fba0a9e2700 1 -- 172.21.15.113:0/793675278 shutdown_connections 2024-04-12T05:09:19.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.523+0000 7fba0a9e2700 1 -- 172.21.15.113:0/793675278 wait complete. 2024-04-12T05:09:19.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.524+0000 7fba0a9e2700 1 Processor -- start 2024-04-12T05:09:19.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.524+0000 7fba0a9e2700 1 -- start start 2024-04-12T05:09:19.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.524+0000 7fba0a9e2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba04061280 0x7fba04107f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:19.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.524+0000 7fba0a9e2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba040fa950 0x7fba04108400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:19.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.524+0000 7fba0a9e2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba041089b0 con 0x7fba04061280 2024-04-12T05:09:19.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.524+0000 7fba0a9e2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba04108af0 con 0x7fba040fa950 2024-04-12T05:09:19.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.524+0000 7fba03fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba04061280 0x7fba04107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.524+0000 7fba03fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba04061280 0x7fba04107f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38650/0 (socket says 172.21.15.113:38650) 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.524+0000 7fba03fff700 1 -- 172.21.15.113:0/3877047586 learned_addr learned my addr 172.21.15.113:0/3877047586 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.524+0000 7fba037fe700 1 --2- 172.21.15.113:0/3877047586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba040fa950 0x7fba04108400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.525+0000 7fba03fff700 1 -- 172.21.15.113:0/3877047586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba040fa950 msgr2=0x7fba04108400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.525+0000 7fba03fff700 1 --2- 172.21.15.113:0/3877047586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba040fa950 0x7fba04108400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.525+0000 7fba03fff700 1 -- 172.21.15.113:0/3877047586 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f8005040 con 0x7fba04061280 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.525+0000 7fba03fff700 1 --2- 172.21.15.113:0/3877047586 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba04061280 0x7fba04107f10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fb9f40041d0 tx=0x7fb9f4009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.525+0000 7fba017fa700 1 -- 172.21.15.113:0/3877047586 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f4007b40 con 0x7fba04061280 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.525+0000 7fba017fa700 1 -- 172.21.15.113:0/3877047586 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9f400f040 con 0x7fba04061280 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.525+0000 7fba0a9e2700 1 -- 172.21.15.113:0/3877047586 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0410d590 con 0x7fba04061280 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.525+0000 7fba017fa700 1 -- 172.21.15.113:0/3877047586 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f40147b0 con 0x7fba04061280 2024-04-12T05:09:19.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.525+0000 7fba0a9e2700 1 -- 172.21.15.113:0/3877047586 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0410db50 con 0x7fba04061280 2024-04-12T05:09:19.529 INFO:tasks.workunit.client.1.smithi192.stdout:9/893: write d11/d25/d36/d3d/f5e [2373889,72059] 0 2024-04-12T05:09:19.529 INFO:tasks.workunit.client.1.smithi192.stdout:9/894: read - d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/fc1 zero size 2024-04-12T05:09:19.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.526+0000 7fba0a9e2700 1 -- 172.21.15.113:0/3877047586 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba040fffb0 con 0x7fba04061280 2024-04-12T05:09:19.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.526+0000 7fba017fa700 1 -- 172.21.15.113:0/3877047586 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7fb9f4009110 con 0x7fba04061280 2024-04-12T05:09:19.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.527+0000 7fba017fa700 1 --2- 172.21.15.113:0/3877047586 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb9ec066dc0 0x7fb9ec069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:19.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.527+0000 7fba017fa700 1 -- 172.21.15.113:0/3877047586 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7fb9f40843f0 con 0x7fba04061280 2024-04-12T05:09:19.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.527+0000 7fba037fe700 1 --2- 172.21.15.113:0/3877047586 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb9ec066dc0 0x7fb9ec069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:09:19.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.527+0000 7fba037fe700 1 --2- 172.21.15.113:0/3877047586 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb9ec066dc0 0x7fb9ec069280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fb9f8004f10 tx=0x7fb9f800b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:19.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.532+0000 7fba017fa700 1 -- 172.21.15.113:0/3877047586 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fb9f4052490 con 0x7fba04061280 2024-04-12T05:09:19.541 INFO:tasks.workunit.client.1.smithi192.stdout:9/895: dread d11/d25/d36/d6e/d33/dbb/fdb [0,4194304] 0 2024-04-12T05:09:19.543 INFO:tasks.workunit.client.1.smithi192.stdout:9/896: symlink d11/d25/dec/d1f/d6a/d7e/l147 0 2024-04-12T05:09:19.546 INFO:tasks.workunit.client.1.smithi192.stdout:9/897: unlink d11/d25/d36/d6e/d8f/l104 0 2024-04-12T05:09:19.549 INFO:tasks.workunit.client.1.smithi192.stdout:9/898: creat d11/d25/d36/d6e/d33/ddd/f148 x:0 0 0 2024-04-12T05:09:19.549 INFO:tasks.workunit.client.1.smithi192.stdout:9/899: dread - d11/d25/f13b zero size 2024-04-12T05:09:19.554 INFO:tasks.workunit.client.1.smithi192.stdout:9/900: dread d11/d25/dec/d1f/f67 [4194304,4194304] 0 2024-04-12T05:09:19.556 INFO:tasks.workunit.client.1.smithi192.stdout:9/901: creat d11/d25/d60/f149 x:0 0 0 2024-04-12T05:09:19.558 INFO:tasks.workunit.client.1.smithi192.stdout:9/902: mknod d11/d25/dec/d1f/d94/c14a 0 2024-04-12T05:09:19.560 INFO:tasks.workunit.client.1.smithi192.stdout:9/903: creat d11/d25/d36/d6e/d33/da2/ddf/d10f/f14b x:0 0 0 2024-04-12T05:09:19.565 INFO:tasks.workunit.client.1.smithi192.stdout:9/904: rename d11/d25/d36/d6e/d33/ddd/f107 to d11/d25/d36/d6e/d8f/f14c 0 2024-04-12T05:09:19.565 INFO:tasks.workunit.client.1.smithi192.stdout:9/905: readlink d11/d25/dec/lda 0 2024-04-12T05:09:19.567 INFO:tasks.workunit.client.1.smithi192.stdout:9/906: dread - d11/d25/dec/fcf zero size 2024-04-12T05:09:19.570 INFO:tasks.workunit.client.1.smithi192.stdout:9/907: creat d11/d25/d36/d6e/d33/da2/ddf/f14d x:0 0 0 2024-04-12T05:09:19.571 INFO:tasks.workunit.client.1.smithi192.stdout:9/908: dread - d11/d25/dec/d1f/d6a/f146 zero size 2024-04-12T05:09:19.573 INFO:tasks.workunit.client.1.smithi192.stdout:9/909: mknod d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d11f/c14e 0 2024-04-12T05:09:19.586 INFO:tasks.workunit.client.1.smithi192.stdout:9/910: dwrite d11/d25/d36/d6e/d33/da2/fc9 [0,4194304] 0 2024-04-12T05:09:19.591 INFO:tasks.workunit.client.1.smithi192.stdout:9/911: mknod d11/d25/dec/d1f/d6a/d92/da6/c14f 0 2024-04-12T05:09:19.595 INFO:tasks.workunit.client.1.smithi192.stdout:9/912: mkdir d11/d25/dec/d1f/d6a/d92/d115/d150 0 2024-04-12T05:09:19.598 INFO:tasks.workunit.client.1.smithi192.stdout:9/913: unlink d11/d25/d36/d6e/d8f/d120/df9/f102 0 2024-04-12T05:09:19.601 INFO:tasks.workunit.client.1.smithi192.stdout:9/914: creat d11/d25/dec/d1f/d6a/d92/d115/d150/f151 x:0 0 0 2024-04-12T05:09:19.607 INFO:tasks.workunit.client.1.smithi192.stdout:9/915: dwrite d11/d25/d36/d6e/d33/f54 [0,4194304] 0 2024-04-12T05:09:19.613 INFO:tasks.workunit.client.1.smithi192.stdout:9/916: dwrite d11/d25/dec/f1a [4194304,4194304] 0 2024-04-12T05:09:19.618 INFO:tasks.workunit.client.1.smithi192.stdout:9/917: mkdir d11/d152 0 2024-04-12T05:09:19.624 INFO:tasks.workunit.client.1.smithi192.stdout:9/918: link d11/d25/d36/d6e/d33/dbb/fdb d11/d25/d36/d6e/f153 0 2024-04-12T05:09:19.639 INFO:tasks.workunit.client.1.smithi192.stdout:9/919: write d11/d25/dd3/fe4 [5129771,130816] 0 2024-04-12T05:09:19.642 INFO:tasks.workunit.client.1.smithi192.stdout:9/920: mkdir d11/d25/d36/d6e/d33/dbb/df5/d154 0 2024-04-12T05:09:19.644 INFO:tasks.workunit.client.1.smithi192.stdout:9/921: chown d11/d25/d36/d6e/d33/da2/f84 939351333 1 2024-04-12T05:09:19.647 INFO:tasks.workunit.client.1.smithi192.stdout:9/922: mknod d11/d25/d36/d6e/d33/da2/ddf/d10f/c155 0 2024-04-12T05:09:19.647 INFO:tasks.workunit.client.1.smithi192.stdout:9/923: dread - d11/d25/dec/fcf zero size 2024-04-12T05:09:19.648 INFO:tasks.workunit.client.1.smithi192.stdout:9/924: dread - d11/d25/f13b zero size 2024-04-12T05:09:19.649 INFO:tasks.workunit.client.1.smithi192.stdout:9/925: write d11/d25/d36/d3d/f116 [672941,31787] 0 2024-04-12T05:09:19.651 INFO:tasks.workunit.client.1.smithi192.stdout:9/926: rmdir d11/d25/d60/d8c 39 2024-04-12T05:09:19.654 INFO:tasks.workunit.client.1.smithi192.stdout:9/927: readlink d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d21/ld8 0 2024-04-12T05:09:19.655 INFO:tasks.workunit.client.1.smithi192.stdout:9/928: fdatasync d11/d25/d36/d6e/d33/f55 0 2024-04-12T05:09:19.656 INFO:tasks.workunit.client.1.smithi192.stdout:9/929: write d11/d25/d36/d6e/d33/da2/fc9 [2183891,85397] 0 2024-04-12T05:09:19.659 INFO:tasks.workunit.client.1.smithi192.stdout:9/930: rmdir d11/d25/d36/d6e/d33/ddd 39 2024-04-12T05:09:19.664 INFO:tasks.workunit.client.1.smithi192.stdout:9/931: link d11/d25/d36/d6e/d33/lbe d11/d25/d36/d6e/d33/df3/l156 0 2024-04-12T05:09:19.667 INFO:tasks.workunit.client.1.smithi192.stdout:9/932: mkdir d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d21/d157 0 2024-04-12T05:09:19.667 INFO:tasks.workunit.client.1.smithi192.stdout:9/933: stat d11/d25/d36/d3d 0 2024-04-12T05:09:19.668 INFO:tasks.workunit.client.1.smithi192.stdout:9/934: write d11/d25/dec/d1f/d6a/f146 [817694,62967] 0 2024-04-12T05:09:19.684 INFO:tasks.workunit.client.1.smithi192.stdout:9/935: dread d11/d25/d36/d6e/d33/da2/fc4 [0,4194304] 0 2024-04-12T05:09:19.691 INFO:tasks.workunit.client.1.smithi192.stdout:9/936: dwrite d11/d25/dec/d1f/d50/f13d [0,4194304] 0 2024-04-12T05:09:19.692 INFO:tasks.workunit.client.1.smithi192.stdout:9/937: rename d11/d25/d36/d6e to d11/d25/d36/d6e/d33/dbb/d158 22 2024-04-12T05:09:19.700 INFO:tasks.workunit.client.1.smithi192.stdout:9/938: creat d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d21/f159 x:0 0 0 2024-04-12T05:09:19.701 INFO:tasks.workunit.client.1.smithi192.stdout:9/939: stat d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d21/d157 0 2024-04-12T05:09:19.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.882+0000 7fba0a9e2700 1 -- 172.21.15.113:0/3877047586 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fba040598f0 con 0x7fba04061280 2024-04-12T05:09:19.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.883+0000 7fba017fa700 1 -- 172.21.15.113:0/3877047586 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+850 (secure 0 0 0) 0x7fb9f4052080 con 0x7fba04061280 2024-04-12T05:09:19.886 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:09:19.886 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:09:19.886 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:09:19.886 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:09:19.886 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:09:19.886 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:09:19.886 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 13, 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:09:19.887 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:09:19.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.885+0000 7fba0a9e2700 1 -- 172.21.15.113:0/3877047586 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb9ec066dc0 msgr2=0x7fb9ec069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.885+0000 7fba0a9e2700 1 --2- 172.21.15.113:0/3877047586 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb9ec066dc0 0x7fb9ec069280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fb9f8004f10 tx=0x7fb9f800b040).stop 2024-04-12T05:09:19.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.885+0000 7fba0a9e2700 1 -- 172.21.15.113:0/3877047586 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba04061280 msgr2=0x7fba04107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.885+0000 7fba0a9e2700 1 --2- 172.21.15.113:0/3877047586 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba04061280 0x7fba04107f10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fb9f40041d0 tx=0x7fb9f4009f60).stop 2024-04-12T05:09:19.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.885+0000 7fba0a9e2700 1 -- 172.21.15.113:0/3877047586 shutdown_connections 2024-04-12T05:09:19.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.885+0000 7fba0a9e2700 1 --2- 172.21.15.113:0/3877047586 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fb9ec066dc0 0x7fb9ec069280 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.885+0000 7fba0a9e2700 1 --2- 172.21.15.113:0/3877047586 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba04061280 0x7fba04107f10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.885+0000 7fba0a9e2700 1 --2- 172.21.15.113:0/3877047586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba040fa950 0x7fba04108400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.885+0000 7fba0a9e2700 1 -- 172.21.15.113:0/3877047586 >> 172.21.15.113:0/3877047586 conn(0x7fba040f1e50 msgr2=0x7fba040f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:19.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.885+0000 7fba0a9e2700 1 -- 172.21.15.113:0/3877047586 shutdown_connections 2024-04-12T05:09:19.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.885+0000 7fba0a9e2700 1 -- 172.21.15.113:0/3877047586 wait complete. 2024-04-12T05:09:19.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:18.615404+0000 mgr.smithi192.ocppyo (mgr.24357) 36 : audit [DBG] from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:19.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:18.676027+0000 mgr.smithi192.ocppyo (mgr.24357) 37 : cluster [DBG] pgmap v29: 65 pgs: 65 active+clean; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 153 MiB/s rd, 343 MiB/s wr, 747 op/s 2024-04-12T05:09:19.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:19.028735+0000 mgr.smithi192.ocppyo (mgr.24357) 38 : audit [DBG] from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:19.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.989+0000 7f9cf1c93700 1 -- 172.21.15.113:0/80238650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cec0f6ff0 msgr2=0x7f9cec0f7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.989+0000 7f9cf1c93700 1 --2- 172.21.15.113:0/80238650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cec0f6ff0 0x7f9cec0f7a20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f9ce0004870 tx=0x7f9ce0009cb0).stop 2024-04-12T05:09:19.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.989+0000 7f9cf1c93700 1 -- 172.21.15.113:0/80238650 shutdown_connections 2024-04-12T05:09:19.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.989+0000 7f9cf1c93700 1 --2- 172.21.15.113:0/80238650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cec0f6ff0 0x7f9cec0f7a20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.989+0000 7f9cf1c93700 1 --2- 172.21.15.113:0/80238650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cec0f66e0 0x7f9cec0f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:19.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.989+0000 7f9cf1c93700 1 -- 172.21.15.113:0/80238650 >> 172.21.15.113:0/80238650 conn(0x7f9cec0f1ec0 msgr2=0x7f9cec0f4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:19.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.989+0000 7f9cf1c93700 1 -- 172.21.15.113:0/80238650 shutdown_connections 2024-04-12T05:09:19.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.989+0000 7f9cf1c93700 1 -- 172.21.15.113:0/80238650 wait complete. 2024-04-12T05:09:19.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.989+0000 7f9cf1c93700 1 Processor -- start 2024-04-12T05:09:19.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.989+0000 7f9cf1c93700 1 -- start start 2024-04-12T05:09:19.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9cf1c93700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cec0f66e0 0x7f9cec10a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:19.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9cf1c93700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cec0f6ff0 0x7f9cec10a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:19.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9cf1c93700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec10af50 con 0x7f9cec0f6ff0 2024-04-12T05:09:19.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9cf1c93700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec10b090 con 0x7f9cec0f66e0 2024-04-12T05:09:19.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9ceaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cec0f6ff0 0x7f9cec10a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:19.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9ceaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cec0f6ff0 0x7f9cec10a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38672/0 (socket says 172.21.15.113:38672) 2024-04-12T05:09:19.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9ceaffd700 1 -- 172.21.15.113:0/1019053710 learned_addr learned my addr 172.21.15.113:0/1019053710 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:09:19.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9ceaffd700 1 -- 172.21.15.113:0/1019053710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cec0f66e0 msgr2=0x7f9cec10a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:19.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9ceaffd700 1 --2- 172.21.15.113:0/1019053710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cec0f66e0 0x7f9cec10a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:09:19.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9ceaffd700 1 -- 172.21.15.113:0/1019053710 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce0005040 con 0x7f9cec0f6ff0 2024-04-12T05:09:19.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9ceaffd700 1 --2- 172.21.15.113:0/1019053710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cec0f6ff0 0x7f9cec10a9a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f9ce0014040 tx=0x7f9ce0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:19.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9ce8ff9700 1 -- 172.21.15.113:0/1019053710 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ce00092e0 con 0x7f9cec0f6ff0 2024-04-12T05:09:19.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9ce8ff9700 1 -- 172.21.15.113:0/1019053710 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ce0009440 con 0x7f9cec0f6ff0 2024-04-12T05:09:19.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9cf1c93700 1 -- 172.21.15.113:0/1019053710 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cec10fad0 con 0x7f9cec0f6ff0 2024-04-12T05:09:19.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9ce8ff9700 1 -- 172.21.15.113:0/1019053710 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ce001a430 con 0x7f9cec0f6ff0 2024-04-12T05:09:19.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.990+0000 7f9cf1c93700 1 -- 172.21.15.113:0/1019053710 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cec110090 con 0x7f9cec0f6ff0 2024-04-12T05:09:19.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.991+0000 7f9cf1c93700 1 -- 172.21.15.113:0/1019053710 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cec0fddc0 con 0x7f9cec0f6ff0 2024-04-12T05:09:19.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.992+0000 7f9ce8ff9700 1 -- 172.21.15.113:0/1019053710 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7f9ce0018040 con 0x7f9cec0f6ff0 2024-04-12T05:09:19.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.992+0000 7f9ce8ff9700 1 --2- 172.21.15.113:0/1019053710 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f9cd4066dc0 0x7f9cd4069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:19.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.992+0000 7f9ce8ff9700 1 -- 172.21.15.113:0/1019053710 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f9ce00855c0 con 0x7f9cec0f6ff0 2024-04-12T05:09:19.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.993+0000 7f9ceb7fe700 1 --2- 172.21.15.113:0/1019053710 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f9cd4066dc0 0x7f9cd4069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:09:19.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.993+0000 7f9ceb7fe700 1 --2- 172.21.15.113:0/1019053710 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f9cd4066dc0 0x7f9cd4069280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f9cdc004c60 tx=0x7f9cdc00b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:20.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:19.997+0000 7f9ce8ff9700 1 -- 172.21.15.113:0/1019053710 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f9ce0053660 con 0x7f9cec0f6ff0 2024-04-12T05:09:20.172 INFO:tasks.workunit.client.1.smithi192.stdout:9/940: sync 2024-04-12T05:09:20.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:18.615404+0000 mgr.smithi192.ocppyo (mgr.24357) 36 : audit [DBG] from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:20.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:18.676027+0000 mgr.smithi192.ocppyo (mgr.24357) 37 : cluster [DBG] pgmap v29: 65 pgs: 65 active+clean; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 153 MiB/s rd, 343 MiB/s wr, 747 op/s 2024-04-12T05:09:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:19.028735+0000 mgr.smithi192.ocppyo (mgr.24357) 38 : audit [DBG] from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:20.174 INFO:tasks.workunit.client.1.smithi192.stdout:9/941: dread - d11/d25/dec/d1f/d6a/d7e/fe5 zero size 2024-04-12T05:09:20.175 INFO:tasks.workunit.client.1.smithi192.stdout:9/942: chown d11/d25/d36/d6e/d33/da2/f10a 11 1 2024-04-12T05:09:20.176 INFO:tasks.workunit.client.1.smithi192.stdout:9/943: dread - d11/d25/d60/f145 zero size 2024-04-12T05:09:20.180 INFO:tasks.workunit.client.1.smithi192.stdout:9/944: rmdir d11/d25/d36/d6e/d33/dbb/df5/d154 0 2024-04-12T05:09:20.186 INFO:tasks.workunit.client.1.smithi192.stdout:9/945: dread d11/d25/d36/d3d/fb7 [0,4194304] 0 2024-04-12T05:09:20.188 INFO:tasks.workunit.client.1.smithi192.stdout:9/946: mknod d11/d152/c15a 0 2024-04-12T05:09:20.190 INFO:tasks.workunit.client.1.smithi192.stdout:9/947: write d11/d25/dec/d1f/d6a/d92/d115/d150/f151 [28865,6981] 0 2024-04-12T05:09:20.192 INFO:tasks.workunit.client.1.smithi192.stdout:9/948: write d11/d25/d36/d6e/d33/da2/ddf/d10f/f14b [530719,33187] 0 2024-04-12T05:09:20.198 INFO:tasks.workunit.client.1.smithi192.stdout:9/949: dwrite d11/d25/d36/d3d/f116 [0,4194304] 0 2024-04-12T05:09:20.202 INFO:tasks.workunit.client.1.smithi192.stdout:9/950: creat d11/d25/d36/d6e/d33/dbb/df5/d12c/f15b x:0 0 0 2024-04-12T05:09:20.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.235+0000 7f9cf1c93700 1 -- 172.21.15.113:0/1019053710 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cec059120 con 0x7f9cd4066dc0 2024-04-12T05:09:20.239 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.236+0000 7f9ce8ff9700 1 -- 172.21.15.113:0/1019053710 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f9cec059120 con 0x7f9cd4066dc0 2024-04-12T05:09:20.239 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:09:20.239 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:09:20.239 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:09:20.239 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:09:20.239 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [], 2024-04-12T05:09:20.239 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "1/2 daemons upgraded", 2024-04-12T05:09:20.240 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "", 2024-04-12T05:09:20.240 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:09:20.240 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:09:20.240 INFO:tasks.workunit.client.1.smithi192.stdout:9/951: write d11/d25/f52 [1051870,68716] 0 2024-04-12T05:09:20.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.238+0000 7f9cf1c93700 1 -- 172.21.15.113:0/1019053710 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f9cd4066dc0 msgr2=0x7f9cd4069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:20.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.238+0000 7f9cf1c93700 1 --2- 172.21.15.113:0/1019053710 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f9cd4066dc0 0x7f9cd4069280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f9cdc004c60 tx=0x7f9cdc00b040).stop 2024-04-12T05:09:20.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.238+0000 7f9cf1c93700 1 -- 172.21.15.113:0/1019053710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cec0f6ff0 msgr2=0x7f9cec10a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:20.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.238+0000 7f9cf1c93700 1 --2- 172.21.15.113:0/1019053710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cec0f6ff0 0x7f9cec10a9a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f9ce0014040 tx=0x7f9ce0009cb0).stop 2024-04-12T05:09:20.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.238+0000 7f9cf1c93700 1 -- 172.21.15.113:0/1019053710 shutdown_connections 2024-04-12T05:09:20.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.238+0000 7f9cf1c93700 1 --2- 172.21.15.113:0/1019053710 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f9cd4066dc0 0x7f9cd4069280 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:20.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.238+0000 7f9cf1c93700 1 --2- 172.21.15.113:0/1019053710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cec0f66e0 0x7f9cec10a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:09:20.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.238+0000 7f9cf1c93700 1 --2- 172.21.15.113:0/1019053710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cec0f6ff0 0x7f9cec10a9a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:20.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.238+0000 7f9cf1c93700 1 -- 172.21.15.113:0/1019053710 >> 172.21.15.113:0/1019053710 conn(0x7f9cec0f1ec0 msgr2=0x7f9cec0fb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:20.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.238+0000 7f9cf1c93700 1 -- 172.21.15.113:0/1019053710 shutdown_connections 2024-04-12T05:09:20.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:20.238+0000 7f9cf1c93700 1 -- 172.21.15.113:0/1019053710 wait complete. 2024-04-12T05:09:20.242 INFO:tasks.workunit.client.1.smithi192.stdout:9/952: read d11/d25/dec/d1f/f7f [3705270,69344] 0 2024-04-12T05:09:20.243 INFO:tasks.workunit.client.1.smithi192.stdout:9/953: fsync d11/d25/d36/d6e/d33/da2/fc4 0 2024-04-12T05:09:20.247 INFO:tasks.workunit.client.1.smithi192.stdout:9/954: creat d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/f15c x:0 0 0 2024-04-12T05:09:20.248 INFO:tasks.workunit.client.1.smithi192.stdout:9/955: chown d11/d25/d36/d6e/d8f/d120/c99 12322676 1 2024-04-12T05:09:20.257 INFO:tasks.workunit.client.1.smithi192.stdout:9/956: dwrite d11/d25/d36/d6e/d33/f6c [0,4194304] 0 2024-04-12T05:09:20.261 INFO:tasks.workunit.client.1.smithi192.stdout:9/957: chown d11/d25/dec/d1f/d50/d73/fb2 28680294 1 2024-04-12T05:09:20.271 INFO:tasks.workunit.client.1.smithi192.stdout:9/958: read d11/d25/dec/f8a [51866,73631] 0 2024-04-12T05:09:20.275 INFO:tasks.workunit.client.1.smithi192.stdout:9/959: getdents d11/d25/dec/d1f/d6a 0 2024-04-12T05:09:20.286 INFO:tasks.workunit.client.1.smithi192.stdout:9/960: dread d11/d25/f52 [0,4194304] 0 2024-04-12T05:09:20.290 INFO:tasks.workunit.client.1.smithi192.stdout:9/961: rename d11/f12 to d11/d25/d36/d6e/dc7/f15d 0 2024-04-12T05:09:20.292 INFO:tasks.workunit.client.1.smithi192.stdout:9/962: symlink d11/d25/dd3/l15e 0 2024-04-12T05:09:20.294 INFO:tasks.workunit.client.1.smithi192.stdout:9/963: mkdir d11/d25/d36/d6e/d71/d15f 0 2024-04-12T05:09:20.299 INFO:tasks.workunit.client.1.smithi192.stdout:9/964: dwrite d11/d25/d36/d6e/d8f/d9c/f114 [0,4194304] 0 2024-04-12T05:09:20.303 INFO:tasks.workunit.client.1.smithi192.stdout:9/965: truncate d11/d25/d36/d6e/f153 3154063 0 2024-04-12T05:09:20.306 INFO:tasks.workunit.client.1.smithi192.stdout:9/966: write d11/d25/dd3/d10d/fff [897464,24868] 0 2024-04-12T05:09:20.314 INFO:tasks.workunit.client.1.smithi192.stdout:9/967: dwrite d11/d25/d36/d6e/d8f/d9c/f142 [0,4194304] 0 2024-04-12T05:09:20.322 INFO:tasks.workunit.client.1.smithi192.stdout:9/968: rmdir d11/d25/dec/d1f/d6a/d92/d115/d150 39 2024-04-12T05:09:20.323 INFO:tasks.workunit.client.1.smithi192.stdout:9/969: readlink d11/d25/dec/lda 0 2024-04-12T05:09:20.614 INFO:tasks.workunit.client.1.smithi192.stdout:9/970: sync 2024-04-12T05:09:20.616 INFO:tasks.workunit.client.1.smithi192.stdout:9/971: mknod d11/d152/c160 0 2024-04-12T05:09:20.618 INFO:tasks.workunit.client.1.smithi192.stdout:9/972: rmdir d11/d25/d36/d6e/dc7 39 2024-04-12T05:09:20.622 INFO:tasks.workunit.client.1.smithi192.stdout:9/973: symlink d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d21/d157/l161 0 2024-04-12T05:09:20.624 INFO:tasks.workunit.client.1.smithi192.stdout:9/974: mknod d11/d25/d36/c162 0 2024-04-12T05:09:20.633 INFO:tasks.workunit.client.1.smithi192.stdout:9/975: dwrite d11/d25/d36/d6e/dc7/f15d [0,4194304] 0 2024-04-12T05:09:20.637 INFO:tasks.workunit.client.1.smithi192.stdout:9/976: mknod d11/d25/dd3/dd5/c163 0 2024-04-12T05:09:20.643 INFO:tasks.workunit.client.1.smithi192.stdout:9/977: write d11/d25/d36/d6e/f30 [437040,24884] 0 2024-04-12T05:09:20.647 INFO:tasks.workunit.client.1.smithi192.stdout:9/978: unlink d11/d25/dec/d1f/d94/la7 0 2024-04-12T05:09:20.651 INFO:tasks.workunit.client.1.smithi192.stdout:9/979: rename d11/d25/dec/d1f/c3f to d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d21/d157/c164 0 2024-04-12T05:09:20.890 INFO:tasks.workunit.client.1.smithi192.stdout:9/980: sync 2024-04-12T05:09:20.893 INFO:tasks.workunit.client.1.smithi192.stdout:9/981: mkdir d11/d25/d36/d6e/d8f/d120/d165 0 2024-04-12T05:09:20.894 INFO:tasks.workunit.client.1.smithi192.stdout:9/982: chown d11/d25/d36/d6e/d33/da2/fc4 716 1 2024-04-12T05:09:20.895 INFO:tasks.workunit.client.1.smithi192.stdout:9/983: creat d11/d25/d36/d6e/d8f/d9c/f166 x:0 0 0 2024-04-12T05:09:20.897 INFO:tasks.workunit.client.1.smithi192.stdout:9/984: chown d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/c46 516666 1 2024-04-12T05:09:20.899 INFO:tasks.workunit.client.1.smithi192.stdout:9/985: truncate d11/d25/d36/d6e/d8f/d120/df9/f126 4308758 0 2024-04-12T05:09:20.906 INFO:tasks.workunit.client.1.smithi192.stdout:9/986: dwrite d11/d25/d36/d6e/d33/da2/fc4 [4194304,4194304] 0 2024-04-12T05:09:20.910 INFO:tasks.workunit.client.1.smithi192.stdout:9/987: unlink c0 0 2024-04-12T05:09:20.913 INFO:tasks.workunit.client.1.smithi192.stdout:9/988: dread d11/d25/dec/d1f/d50/f13d [0,4194304] 0 2024-04-12T05:09:20.917 INFO:tasks.workunit.client.1.smithi192.stdout:9/989: getdents d11/d25/d36/d6e/d71/d15f 0 2024-04-12T05:09:20.918 INFO:tasks.workunit.client.1.smithi192.stdout:9/990: stat d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d21/ld8 0 2024-04-12T05:09:20.920 INFO:tasks.workunit.client.1.smithi192.stdout:9/991: rmdir d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d21/d157 39 2024-04-12T05:09:20.925 INFO:tasks.workunit.client.1.smithi192.stdout:9/992: link d11/d25/d36/d6e/dc7/f13e d11/d25/d36/d6e/d71/d15f/f167 0 2024-04-12T05:09:20.926 INFO:tasks.workunit.client.1.smithi192.stdout:9/993: read - d11/d25/d36/d6e/d8f/d120/df9/f105 zero size 2024-04-12T05:09:20.928 INFO:tasks.workunit.client.1.smithi192.stdout:9/994: write d11/d25/dec/d1f/f13c [961438,120916] 0 2024-04-12T05:09:20.931 INFO:tasks.workunit.client.1.smithi192.stdout:9/995: symlink d11/d25/d36/d6e/l168 0 2024-04-12T05:09:20.933 INFO:tasks.workunit.client.1.smithi192.stdout:9/996: symlink d11/d152/l169 0 2024-04-12T05:09:20.933 INFO:tasks.workunit.client.1.smithi192.stdout:9/997: stat d11/d25/d36/d6e/dc7/f15d 0 2024-04-12T05:09:20.936 INFO:tasks.workunit.client.1.smithi192.stdout:9/998: truncate d11/d25/d36/d6e/d71/f108 833521 0 2024-04-12T05:09:20.943 INFO:tasks.workunit.client.1.smithi192.stdout:9/999: getdents d11/d25/d36/d6e/d33/da2/ddf/d10f/d140/d21/d157 0 2024-04-12T05:09:20.948 INFO:tasks.workunit.client.1.smithi192.stderr:+ rm -rf -- ./tmp.Eu8tZstG0i 2024-04-12T05:09:21.173 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:19.402736+0000 mgr.smithi192.ocppyo (mgr.24357) 39 : audit [DBG] from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:21.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:19.884976+0000 mon.smithi113 (mon.0) 683 : audit [DBG] from='client.? 172.21.15.113:0/3877047586' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:09:21.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:20.237746+0000 mgr.smithi192.ocppyo (mgr.24357) 40 : audit [DBG] from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:21.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:19.402736+0000 mgr.smithi192.ocppyo (mgr.24357) 39 : audit [DBG] from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:21.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:19.884976+0000 mon.smithi113 (mon.0) 683 : audit [DBG] from='client.? 172.21.15.113:0/3877047586' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:09:21.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:20.237746+0000 mgr.smithi192.ocppyo (mgr.24357) 40 : audit [DBG] from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:20.676355+0000 mgr.smithi192.ocppyo (mgr.24357) 41 : cluster [DBG] pgmap v30: 65 pgs: 65 active+clean; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 104 MiB/s rd, 236 MiB/s wr, 520 op/s 2024-04-12T05:09:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:21.422404+0000 mon.smithi113 (mon.0) 684 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:21.429676+0000 mon.smithi113 (mon.0) 685 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:21.431818+0000 mon.smithi192 (mon.1) 54 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:21.432192+0000 mon.smithi113 (mon.0) 686 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:21.433078+0000 mon.smithi192 (mon.1) 55 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:21.433367+0000 mon.smithi113 (mon.0) 687 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:21.434340+0000 mon.smithi192 (mon.1) 56 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:21.434596+0000 mon.smithi113 (mon.0) 688 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:21.439725+0000 mon.smithi113 (mon.0) 689 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:22.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:21.443203+0000 mon.smithi192 (mon.1) 57 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:09:22.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:21.444175+0000 mon.smithi192 (mon.1) 58 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:09:22.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:20.676355+0000 mgr.smithi192.ocppyo (mgr.24357) 41 : cluster [DBG] pgmap v30: 65 pgs: 65 active+clean; 4.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 104 MiB/s rd, 236 MiB/s wr, 520 op/s 2024-04-12T05:09:22.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:21.422404+0000 mon.smithi113 (mon.0) 684 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:22.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:21.429676+0000 mon.smithi113 (mon.0) 685 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:22.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:21.431818+0000 mon.smithi192 (mon.1) 54 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:21.432192+0000 mon.smithi113 (mon.0) 686 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:21.433078+0000 mon.smithi192 (mon.1) 55 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:21.433367+0000 mon.smithi113 (mon.0) 687 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:21.434340+0000 mon.smithi192 (mon.1) 56 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:21.434596+0000 mon.smithi113 (mon.0) 688 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T05:09:22.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:21.439725+0000 mon.smithi113 (mon.0) 689 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:22.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:21.443203+0000 mon.smithi192 (mon.1) 57 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:09:22.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:21.444175+0000 mon.smithi192 (mon.1) 58 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:21.435323+0000 mgr.smithi192.ocppyo (mgr.24357) 42 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 2251M 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:21.445341+0000 mgr.smithi192.ocppyo (mgr.24357) 43 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.conf 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:21.445562+0000 mgr.smithi192.ocppyo (mgr.24357) 44 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:21.760331+0000 mgr.smithi192.ocppyo (mgr.24357) 45 : cephadm [INF] Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:21.760503+0000 mgr.smithi192.ocppyo (mgr.24357) 46 : cephadm [INF] Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:22.126194+0000 mgr.smithi192.ocppyo (mgr.24357) 47 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:22.126416+0000 mgr.smithi192.ocppyo (mgr.24357) 48 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:22.760698+0000 mon.smithi113 (mon.0) 690 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:22.763931+0000 mon.smithi113 (mon.0) 691 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:22.769358+0000 mon.smithi113 (mon.0) 692 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:22.772089+0000 mon.smithi113 (mon.0) 693 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:22.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:22.777264+0000 mon.smithi113 (mon.0) 694 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:23.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:21.435323+0000 mgr.smithi192.ocppyo (mgr.24357) 42 : cephadm [INF] Adjusting osd_memory_target on smithi192 to 2251M 2024-04-12T05:09:23.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:21.445341+0000 mgr.smithi192.ocppyo (mgr.24357) 43 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.conf 2024-04-12T05:09:23.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:21.445562+0000 mgr.smithi192.ocppyo (mgr.24357) 44 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-04-12T05:09:23.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:21.760331+0000 mgr.smithi192.ocppyo (mgr.24357) 45 : cephadm [INF] Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:09:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:21.760503+0000 mgr.smithi192.ocppyo (mgr.24357) 46 : cephadm [INF] Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:09:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:22.126194+0000 mgr.smithi192.ocppyo (mgr.24357) 47 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:09:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:22.126416+0000 mgr.smithi192.ocppyo (mgr.24357) 48 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:09:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:22.760698+0000 mon.smithi113 (mon.0) 690 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:22.763931+0000 mon.smithi113 (mon.0) 691 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:22.769358+0000 mon.smithi113 (mon.0) 692 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:22.772089+0000 mon.smithi113 (mon.0) 693 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:22.777264+0000 mon.smithi113 (mon.0) 694 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:24.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:22.440853+0000 mgr.smithi192.ocppyo (mgr.24357) 49 : cephadm [INF] Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:09:24.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:22.440986+0000 mgr.smithi192.ocppyo (mgr.24357) 50 : cephadm [INF] Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:09:24.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:22.676815+0000 mgr.smithi192.ocppyo (mgr.24357) 51 : cluster [DBG] pgmap v31: 65 pgs: 65 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 109 MiB/s rd, 258 MiB/s wr, 620 op/s 2024-04-12T05:09:24.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:22.914813+0000 mon.smithi113 (mon.0) 695 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:24.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:22.920278+0000 mon.smithi113 (mon.0) 696 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:24.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:22.925695+0000 mon.smithi113 (mon.0) 697 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:24.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:22.932018+0000 mon.smithi113 (mon.0) 698 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:24.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:22.932768+0000 mgr.smithi192.ocppyo (mgr.24357) 52 : cephadm [INF] Reconfiguring alertmanager.smithi113 (dependencies changed)... 2024-04-12T05:09:24.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:22.944407+0000 mgr.smithi192.ocppyo (mgr.24357) 53 : cephadm [INF] Reconfiguring daemon alertmanager.smithi113 on smithi113 2024-04-12T05:09:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:22.440853+0000 mgr.smithi192.ocppyo (mgr.24357) 49 : cephadm [INF] Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:09:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:22.440986+0000 mgr.smithi192.ocppyo (mgr.24357) 50 : cephadm [INF] Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:09:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:22.676815+0000 mgr.smithi192.ocppyo (mgr.24357) 51 : cluster [DBG] pgmap v31: 65 pgs: 65 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 109 MiB/s rd, 258 MiB/s wr, 620 op/s 2024-04-12T05:09:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:22.914813+0000 mon.smithi113 (mon.0) 695 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:22.920278+0000 mon.smithi113 (mon.0) 696 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:22.925695+0000 mon.smithi113 (mon.0) 697 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:22.932018+0000 mon.smithi113 (mon.0) 698 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:22.932768+0000 mgr.smithi192.ocppyo (mgr.24357) 52 : cephadm [INF] Reconfiguring alertmanager.smithi113 (dependencies changed)... 2024-04-12T05:09:24.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:22.944407+0000 mgr.smithi192.ocppyo (mgr.24357) 53 : cephadm [INF] Reconfiguring daemon alertmanager.smithi113 on smithi113 2024-04-12T05:09:26.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:24.677190+0000 mgr.smithi192.ocppyo (mgr.24357) 54 : cluster [DBG] pgmap v32: 65 pgs: 65 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 50 MiB/s rd, 139 MiB/s wr, 342 op/s 2024-04-12T05:09:26.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:24.677190+0000 mgr.smithi192.ocppyo (mgr.24357) 54 : cluster [DBG] pgmap v32: 65 pgs: 65 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 50 MiB/s rd, 139 MiB/s wr, 342 op/s 2024-04-12T05:09:28.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:26.677560+0000 mgr.smithi192.ocppyo (mgr.24357) 55 : cluster [DBG] pgmap v33: 65 pgs: 65 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 50 MiB/s rd, 139 MiB/s wr, 342 op/s 2024-04-12T05:09:28.235 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:26.677560+0000 mgr.smithi192.ocppyo (mgr.24357) 55 : cluster [DBG] pgmap v33: 65 pgs: 65 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 50 MiB/s rd, 139 MiB/s wr, 342 op/s 2024-04-12T05:09:29.673 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-12T05:09:29.673 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-12T05:09:30.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:28.678135+0000 mgr.smithi192.ocppyo (mgr.24357) 56 : cluster [DBG] pgmap v34: 65 pgs: 65 active+clean; 2.4 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 50 MiB/s rd, 140 MiB/s wr, 437 op/s 2024-04-12T05:09:30.235 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:28.678135+0000 mgr.smithi192.ocppyo (mgr.24357) 56 : cluster [DBG] pgmap v34: 65 pgs: 65 active+clean; 2.4 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 50 MiB/s rd, 140 MiB/s wr, 437 op/s 2024-04-12T05:09:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:30.910469+0000 mon.smithi192 (mon.1) 59 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:09:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:30.910469+0000 mon.smithi192 (mon.1) 59 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:09:32.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:30.679105+0000 mgr.smithi192.ocppyo (mgr.24357) 57 : cluster [DBG] pgmap v35: 65 pgs: 65 active+clean; 2.4 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 5.1 MiB/s rd, 23 MiB/s wr, 195 op/s 2024-04-12T05:09:32.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:30.679105+0000 mgr.smithi192.ocppyo (mgr.24357) 57 : cluster [DBG] pgmap v35: 65 pgs: 65 active+clean; 2.4 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 5.1 MiB/s rd, 23 MiB/s wr, 195 op/s 2024-04-12T05:09:33.940 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-12T05:09:33.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-12T05:09:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:32.679638+0000 mgr.smithi192.ocppyo (mgr.24357) 58 : cluster [DBG] pgmap v36: 65 pgs: 65 active+clean; 877 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 5.2 MiB/s rd, 23 MiB/s wr, 298 op/s 2024-04-12T05:09:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:33.642403+0000 mon.smithi113 (mon.0) 699 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:33.654913+0000 mon.smithi113 (mon.0) 700 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:34.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:32.679638+0000 mgr.smithi192.ocppyo (mgr.24357) 58 : cluster [DBG] pgmap v36: 65 pgs: 65 active+clean; 877 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 5.2 MiB/s rd, 23 MiB/s wr, 298 op/s 2024-04-12T05:09:34.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:33.642403+0000 mon.smithi113 (mon.0) 699 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:34.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:33.654913+0000 mon.smithi113 (mon.0) 700 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:09:35.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:33.676589+0000 mgr.smithi192.ocppyo (mgr.24357) 59 : cephadm [INF] Reconfiguring grafana.smithi113 (dependencies changed)... 2024-04-12T05:09:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:09:33.695398+0000 mgr.smithi192.ocppyo (mgr.24357) 60 : cephadm [INF] Reconfiguring daemon grafana.smithi113 on smithi113 2024-04-12T05:09:35.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:33.676589+0000 mgr.smithi192.ocppyo (mgr.24357) 59 : cephadm [INF] Reconfiguring grafana.smithi113 (dependencies changed)... 2024-04-12T05:09:35.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:09:33.695398+0000 mgr.smithi192.ocppyo (mgr.24357) 60 : cephadm [INF] Reconfiguring daemon grafana.smithi113 on smithi113 2024-04-12T05:09:36.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:34.680023+0000 mgr.smithi192.ocppyo (mgr.24357) 61 : cluster [DBG] pgmap v37: 65 pgs: 65 active+clean; 877 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 31 KiB/s rd, 1.5 MiB/s wr, 198 op/s 2024-04-12T05:09:36.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:34.680023+0000 mgr.smithi192.ocppyo (mgr.24357) 61 : cluster [DBG] pgmap v37: 65 pgs: 65 active+clean; 877 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 31 KiB/s rd, 1.5 MiB/s wr, 198 op/s 2024-04-12T05:09:38.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:36.680390+0000 mgr.smithi192.ocppyo (mgr.24357) 62 : cluster [DBG] pgmap v38: 65 pgs: 65 active+clean; 877 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 31 KiB/s rd, 1.5 MiB/s wr, 198 op/s 2024-04-12T05:09:38.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:36.680390+0000 mgr.smithi192.ocppyo (mgr.24357) 62 : cluster [DBG] pgmap v38: 65 pgs: 65 active+clean; 877 MiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 31 KiB/s rd, 1.5 MiB/s wr, 198 op/s 2024-04-12T05:09:40.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:38.680899+0000 mgr.smithi192.ocppyo (mgr.24357) 63 : cluster [DBG] pgmap v39: 65 pgs: 65 active+clean; 325 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 50 KiB/s rd, 2.3 MiB/s wr, 358 op/s 2024-04-12T05:09:40.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:38.680899+0000 mgr.smithi192.ocppyo (mgr.24357) 63 : cluster [DBG] pgmap v39: 65 pgs: 65 active+clean; 325 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 50 KiB/s rd, 2.3 MiB/s wr, 358 op/s 2024-04-12T05:09:42.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:40.681252+0000 mgr.smithi192.ocppyo (mgr.24357) 64 : cluster [DBG] pgmap v40: 65 pgs: 65 active+clean; 325 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 36 KiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-04-12T05:09:42.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:40.681252+0000 mgr.smithi192.ocppyo (mgr.24357) 64 : cluster [DBG] pgmap v40: 65 pgs: 65 active+clean; 325 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 36 KiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-04-12T05:09:44.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:42.681711+0000 mgr.smithi192.ocppyo (mgr.24357) 65 : cluster [DBG] pgmap v41: 65 pgs: 65 active+clean; 290 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 57 KiB/s rd, 2.4 MiB/s wr, 470 op/s 2024-04-12T05:09:44.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:42.681711+0000 mgr.smithi192.ocppyo (mgr.24357) 65 : cluster [DBG] pgmap v41: 65 pgs: 65 active+clean; 290 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 57 KiB/s rd, 2.4 MiB/s wr, 470 op/s 2024-04-12T05:09:46.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:44.682039+0000 mgr.smithi192.ocppyo (mgr.24357) 66 : cluster [DBG] pgmap v42: 65 pgs: 65 active+clean; 290 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 41 KiB/s rd, 1.6 MiB/s wr, 367 op/s 2024-04-12T05:09:46.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:45.910425+0000 mon.smithi192 (mon.1) 60 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:09:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:44.682039+0000 mgr.smithi192.ocppyo (mgr.24357) 66 : cluster [DBG] pgmap v42: 65 pgs: 65 active+clean; 290 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 41 KiB/s rd, 1.6 MiB/s wr, 367 op/s 2024-04-12T05:09:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:45.910425+0000 mon.smithi192 (mon.1) 60 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:09:48.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:46.682337+0000 mgr.smithi192.ocppyo (mgr.24357) 67 : cluster [DBG] pgmap v43: 65 pgs: 65 active+clean; 290 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 41 KiB/s rd, 1.6 MiB/s wr, 367 op/s 2024-04-12T05:09:48.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:46.682337+0000 mgr.smithi192.ocppyo (mgr.24357) 67 : cluster [DBG] pgmap v43: 65 pgs: 65 active+clean; 290 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 41 KiB/s rd, 1.6 MiB/s wr, 367 op/s 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.367+0000 7f8e09d9b700 1 -- 172.21.15.113:0/2249896482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e040f66f0 msgr2=0x7f8e040f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.367+0000 7f8e09d9b700 1 --2- 172.21.15.113:0/2249896482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e040f66f0 0x7f8e040f6b10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f8df40047f0 tx=0x7f8df4009cb0).stop 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.367+0000 7f8e09d9b700 1 -- 172.21.15.113:0/2249896482 shutdown_connections 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.367+0000 7f8e09d9b700 1 --2- 172.21.15.113:0/2249896482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e040f7990 0x7f8e040f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.367+0000 7f8e09d9b700 1 --2- 172.21.15.113:0/2249896482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e040f66f0 0x7f8e040f6b10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.367+0000 7f8e09d9b700 1 -- 172.21.15.113:0/2249896482 >> 172.21.15.113:0/2249896482 conn(0x7f8e040f1e50 msgr2=0x7f8e040f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.367+0000 7f8e09d9b700 1 -- 172.21.15.113:0/2249896482 shutdown_connections 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.367+0000 7f8e09d9b700 1 -- 172.21.15.113:0/2249896482 wait complete. 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.367+0000 7f8e09d9b700 1 Processor -- start 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e09d9b700 1 -- start start 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e09d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e040f66f0 0x7f8e04105e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e09d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e040f7990 0x7f8e04106380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e09d9b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e04106930 con 0x7f8e040f7990 2024-04-12T05:09:50.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e09d9b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e04106a70 con 0x7f8e040f66f0 2024-04-12T05:09:50.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e02ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e040f7990 0x7f8e04106380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:50.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e02ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e040f7990 0x7f8e04106380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51972/0 (socket says 172.21.15.113:51972) 2024-04-12T05:09:50.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e02ffd700 1 -- 172.21.15.113:0/4123739439 learned_addr learned my addr 172.21.15.113:0/4123739439 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:09:50.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e02ffd700 1 -- 172.21.15.113:0/4123739439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e040f66f0 msgr2=0x7f8e04105e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:50.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e02ffd700 1 --2- 172.21.15.113:0/4123739439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e040f66f0 0x7f8e04105e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:09:50.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e02ffd700 1 -- 172.21.15.113:0/4123739439 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df4005040 con 0x7f8e040f7990 2024-04-12T05:09:50.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.368+0000 7f8e02ffd700 1 --2- 172.21.15.113:0/4123739439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e040f7990 0x7f8e04106380 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8df800e910 tx=0x7f8df800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:50.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.369+0000 7f8e00ff9700 1 -- 172.21.15.113:0/4123739439 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8df800eeb0 con 0x7f8e040f7990 2024-04-12T05:09:50.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.369+0000 7f8e09d9b700 1 -- 172.21.15.113:0/4123739439 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0410b510 con 0x7f8e040f7990 2024-04-12T05:09:50.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.369+0000 7f8e09d9b700 1 -- 172.21.15.113:0/4123739439 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0410bad0 con 0x7f8e040f7990 2024-04-12T05:09:50.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.369+0000 7f8e00ff9700 1 -- 172.21.15.113:0/4123739439 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8df800f040 con 0x7f8e040f7990 2024-04-12T05:09:50.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.369+0000 7f8e00ff9700 1 -- 172.21.15.113:0/4123739439 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8df800c8a0 con 0x7f8e040f7990 2024-04-12T05:09:50.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.370+0000 7f8e09d9b700 1 -- 172.21.15.113:0/4123739439 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df0005150 con 0x7f8e040f7990 2024-04-12T05:09:50.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.371+0000 7f8e00ff9700 1 -- 172.21.15.113:0/4123739439 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7f8df8007b40 con 0x7f8e040f7990 2024-04-12T05:09:50.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.372+0000 7f8e00ff9700 1 --2- 172.21.15.113:0/4123739439 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dec066e10 0x7f8dec0692d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:50.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.372+0000 7f8e00ff9700 1 -- 172.21.15.113:0/4123739439 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f8df8083e70 con 0x7f8e040f7990 2024-04-12T05:09:50.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.372+0000 7f8e037fe700 1 --2- 172.21.15.113:0/4123739439 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dec066e10 0x7f8dec0692d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:09:50.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.372+0000 7f8e037fe700 1 --2- 172.21.15.113:0/4123739439 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dec066e10 0x7f8dec0692d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f8df4013560 tx=0x7f8df4009cb0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:50.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.376+0000 7f8e00ff9700 1 -- 172.21.15.113:0/4123739439 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f8df8051f50 con 0x7f8e040f7990 2024-04-12T05:09:50.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:48.682783+0000 mgr.smithi192.ocppyo (mgr.24357) 68 : cluster [DBG] pgmap v44: 65 pgs: 65 active+clean; 294 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 59 KiB/s rd, 2.3 MiB/s wr, 574 op/s 2024-04-12T05:09:50.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:48.682783+0000 mgr.smithi192.ocppyo (mgr.24357) 68 : cluster [DBG] pgmap v44: 65 pgs: 65 active+clean; 294 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 59 KiB/s rd, 2.3 MiB/s wr, 574 op/s 2024-04-12T05:09:50.704 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:09:50.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.631+0000 7f8e09d9b700 1 -- 172.21.15.113:0/4123739439 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8df0000bf0 con 0x7f8dec066e10 2024-04-12T05:09:50.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.633+0000 7f8e00ff9700 1 -- 172.21.15.113:0/4123739439 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f8df0000bf0 con 0x7f8dec066e10 2024-04-12T05:09:50.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.635+0000 7f8e09d9b700 1 -- 172.21.15.113:0/4123739439 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dec066e10 msgr2=0x7f8dec0692d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:50.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.635+0000 7f8e09d9b700 1 --2- 172.21.15.113:0/4123739439 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dec066e10 0x7f8dec0692d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f8df4013560 tx=0x7f8df4009cb0).stop 2024-04-12T05:09:50.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.635+0000 7f8e09d9b700 1 -- 172.21.15.113:0/4123739439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e040f7990 msgr2=0x7f8e04106380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:50.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.635+0000 7f8e09d9b700 1 --2- 172.21.15.113:0/4123739439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e040f7990 0x7f8e04106380 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8df800e910 tx=0x7f8df800b040).stop 2024-04-12T05:09:50.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.635+0000 7f8e09d9b700 1 -- 172.21.15.113:0/4123739439 shutdown_connections 2024-04-12T05:09:50.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.635+0000 7f8e09d9b700 1 --2- 172.21.15.113:0/4123739439 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dec066e10 0x7f8dec0692d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:50.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.635+0000 7f8e09d9b700 1 --2- 172.21.15.113:0/4123739439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e040f66f0 0x7f8e04105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:09:50.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.635+0000 7f8e09d9b700 1 --2- 172.21.15.113:0/4123739439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e040f7990 0x7f8e04106380 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:50.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.635+0000 7f8e09d9b700 1 -- 172.21.15.113:0/4123739439 >> 172.21.15.113:0/4123739439 conn(0x7f8e040f1e50 msgr2=0x7f8e040f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:50.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.635+0000 7f8e09d9b700 1 -- 172.21.15.113:0/4123739439 shutdown_connections 2024-04-12T05:09:50.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.635+0000 7f8e09d9b700 1 -- 172.21.15.113:0/4123739439 wait complete. 2024-04-12T05:09:50.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.758+0000 7fe226750700 1 -- 172.21.15.113:0/381077353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2200f7990 msgr2=0x7fe2200f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:50.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.758+0000 7fe226750700 1 --2- 172.21.15.113:0/381077353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2200f7990 0x7fe2200f7df0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fe20c0047f0 tx=0x7fe20c009cb0).stop 2024-04-12T05:09:50.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.758+0000 7fe226750700 1 -- 172.21.15.113:0/381077353 shutdown_connections 2024-04-12T05:09:50.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.758+0000 7fe226750700 1 --2- 172.21.15.113:0/381077353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2200f7990 0x7fe2200f7df0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:50.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.758+0000 7fe226750700 1 --2- 172.21.15.113:0/381077353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2200f66f0 0x7fe2200f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:50.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.758+0000 7fe226750700 1 -- 172.21.15.113:0/381077353 >> 172.21.15.113:0/381077353 conn(0x7fe2200f1e50 msgr2=0x7fe2200f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:50.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.758+0000 7fe226750700 1 -- 172.21.15.113:0/381077353 shutdown_connections 2024-04-12T05:09:50.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.758+0000 7fe226750700 1 -- 172.21.15.113:0/381077353 wait complete. 2024-04-12T05:09:50.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.759+0000 7fe226750700 1 Processor -- start 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.759+0000 7fe226750700 1 -- start start 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe226750700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2200f66f0 0x7fe220106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe226750700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2200f7990 0x7fe220107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe226750700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe220107930 con 0x7fe2200f66f0 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe226750700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe220107a70 con 0x7fe2200f7990 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe21ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2200f66f0 0x7fe220106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe21ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2200f66f0 0x7fe220106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51992/0 (socket says 172.21.15.113:51992) 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe21ffff700 1 -- 172.21.15.113:0/3352573052 learned_addr learned my addr 172.21.15.113:0/3352573052 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe21f7fe700 1 --2- 172.21.15.113:0/3352573052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2200f7990 0x7fe220107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe21ffff700 1 -- 172.21.15.113:0/3352573052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2200f7990 msgr2=0x7fe220107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe21ffff700 1 --2- 172.21.15.113:0/3352573052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2200f7990 0x7fe220107380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe21ffff700 1 -- 172.21.15.113:0/3352573052 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c005040 con 0x7fe2200f66f0 2024-04-12T05:09:50.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe21ffff700 1 --2- 172.21.15.113:0/3352573052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2200f66f0 0x7fe220106e90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fe21800db00 tx=0x7fe218004e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:50.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe21d7fa700 1 -- 172.21.15.113:0/3352573052 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe218007e70 con 0x7fe2200f66f0 2024-04-12T05:09:50.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe226750700 1 -- 172.21.15.113:0/3352573052 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe220060650 con 0x7fe2200f66f0 2024-04-12T05:09:50.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe21d7fa700 1 -- 172.21.15.113:0/3352573052 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe21800f040 con 0x7fe2200f66f0 2024-04-12T05:09:50.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.760+0000 7fe21d7fa700 1 -- 172.21.15.113:0/3352573052 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe218013c10 con 0x7fe2200f66f0 2024-04-12T05:09:50.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.761+0000 7fe226750700 1 -- 172.21.15.113:0/3352573052 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe220060c10 con 0x7fe2200f66f0 2024-04-12T05:09:50.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.762+0000 7fe226750700 1 -- 172.21.15.113:0/3352573052 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2200feeb0 con 0x7fe2200f66f0 2024-04-12T05:09:50.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.762+0000 7fe21d7fa700 1 -- 172.21.15.113:0/3352573052 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7fe21800b750 con 0x7fe2200f66f0 2024-04-12T05:09:50.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.762+0000 7fe21d7fa700 1 --2- 172.21.15.113:0/3352573052 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe204066dc0 0x7fe204069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:50.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.763+0000 7fe21d7fa700 1 -- 172.21.15.113:0/3352573052 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7fe218083f00 con 0x7fe2200f66f0 2024-04-12T05:09:50.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.763+0000 7fe21f7fe700 1 --2- 172.21.15.113:0/3352573052 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe204066dc0 0x7fe204069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:09:50.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.763+0000 7fe21f7fe700 1 --2- 172.21.15.113:0/3352573052 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe204066dc0 0x7fe204069280 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe20c004f10 tx=0x7fe20c00b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:50.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:50.768+0000 7fe21d7fa700 1 -- 172.21.15.113:0/3352573052 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fe218051fa0 con 0x7fe2200f66f0 2024-04-12T05:09:51.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.072+0000 7fe226750700 1 -- 172.21.15.113:0/3352573052 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe220060ef0 con 0x7fe204066dc0 2024-04-12T05:09:51.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.074+0000 7fe21d7fa700 1 -- 172.21.15.113:0/3352573052 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fe220060ef0 con 0x7fe204066dc0 2024-04-12T05:09:51.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.076+0000 7fe226750700 1 -- 172.21.15.113:0/3352573052 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe204066dc0 msgr2=0x7fe204069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:51.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.076+0000 7fe226750700 1 --2- 172.21.15.113:0/3352573052 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe204066dc0 0x7fe204069280 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe20c004f10 tx=0x7fe20c00b040).stop 2024-04-12T05:09:51.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.076+0000 7fe226750700 1 -- 172.21.15.113:0/3352573052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2200f66f0 msgr2=0x7fe220106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:51.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.076+0000 7fe226750700 1 --2- 172.21.15.113:0/3352573052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2200f66f0 0x7fe220106e90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fe21800db00 tx=0x7fe218004e80).stop 2024-04-12T05:09:51.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.076+0000 7fe226750700 1 -- 172.21.15.113:0/3352573052 shutdown_connections 2024-04-12T05:09:51.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.076+0000 7fe226750700 1 --2- 172.21.15.113:0/3352573052 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe204066dc0 0x7fe204069280 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:51.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.076+0000 7fe226750700 1 --2- 172.21.15.113:0/3352573052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2200f66f0 0x7fe220106e90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:51.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.076+0000 7fe226750700 1 --2- 172.21.15.113:0/3352573052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2200f7990 0x7fe220107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:51.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.076+0000 7fe226750700 1 -- 172.21.15.113:0/3352573052 >> 172.21.15.113:0/3352573052 conn(0x7fe2200f1e50 msgr2=0x7fe2200fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:51.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.076+0000 7fe226750700 1 -- 172.21.15.113:0/3352573052 shutdown_connections 2024-04-12T05:09:51.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.076+0000 7fe226750700 1 -- 172.21.15.113:0/3352573052 wait complete. 2024-04-12T05:09:51.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.227+0000 7f8de74a8700 1 -- 172.21.15.113:0/3401173044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de00fe670 msgr2=0x7f8de00ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:51.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.227+0000 7f8de74a8700 1 --2- 172.21.15.113:0/3401173044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de00fe670 0x7f8de00ff080 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f8dd00047f0 tx=0x7f8dd0009cb0).stop 2024-04-12T05:09:51.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.227+0000 7f8de74a8700 1 -- 172.21.15.113:0/3401173044 shutdown_connections 2024-04-12T05:09:51.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.227+0000 7f8de74a8700 1 --2- 172.21.15.113:0/3401173044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de00fe670 0x7f8de00ff080 secure :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0x7f8dd00047f0 tx=0x7f8dd0009cb0).stop 2024-04-12T05:09:51.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.227+0000 7f8de74a8700 1 --2- 172.21.15.113:0/3401173044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de00fdd60 0x7f8de00fe180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:09:51.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.227+0000 7f8de74a8700 1 -- 172.21.15.113:0/3401173044 >> 172.21.15.113:0/3401173044 conn(0x7f8de006d0d0 msgr2=0x7f8de006f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:51.232 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.230+0000 7f8de74a8700 1 -- 172.21.15.113:0/3401173044 shutdown_connections 2024-04-12T05:09:51.232 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.230+0000 7f8de74a8700 1 -- 172.21.15.113:0/3401173044 wait complete. 2024-04-12T05:09:51.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.231+0000 7f8de74a8700 1 Processor -- start 2024-04-12T05:09:51.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.231+0000 7f8de74a8700 1 -- start start 2024-04-12T05:09:51.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.231+0000 7f8de74a8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de00fdd60 0x7f8de010e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:51.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.231+0000 7f8de74a8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de010eca0 0x7f8de0113d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:51.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.231+0000 7f8de74a8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de010f200 con 0x7f8de010eca0 2024-04-12T05:09:51.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.231+0000 7f8de74a8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de010f340 con 0x7f8de00fdd60 2024-04-12T05:09:51.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.232+0000 7f8de5244700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de00fdd60 0x7f8de010e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:51.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.232+0000 7f8de5244700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de00fdd60 0x7f8de010e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.113:44370/0 (socket says 172.21.15.113:44370) 2024-04-12T05:09:51.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.232+0000 7f8de5244700 1 -- 172.21.15.113:0/470406273 learned_addr learned my addr 172.21.15.113:0/470406273 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:09:51.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.232+0000 7f8de5244700 1 -- 172.21.15.113:0/470406273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de010eca0 msgr2=0x7f8de0113d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:09:51.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.232+0000 7f8de5244700 1 --2- 172.21.15.113:0/470406273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de010eca0 0x7f8de0113d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:09:51.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.232+0000 7f8de5244700 1 -- 172.21.15.113:0/470406273 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd0005040 con 0x7f8de00fdd60 2024-04-12T05:09:51.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.232+0000 7f8de5244700 1 --2- 172.21.15.113:0/470406273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de00fdd60 0x7f8de010e7b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f8ddc00b410 tx=0x7f8ddc009060).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:51.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.233+0000 7f8dd67fc700 1 -- 172.21.15.113:0/470406273 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ddc0125f0 con 0x7f8de00fdd60 2024-04-12T05:09:51.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.234+0000 7f8dd67fc700 1 -- 172.21.15.113:0/470406273 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ddc012c00 con 0x7f8de00fdd60 2024-04-12T05:09:51.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.234+0000 7f8dd67fc700 1 -- 172.21.15.113:0/470406273 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ddc00db40 con 0x7f8de00fdd60 2024-04-12T05:09:51.239 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.235+0000 7f8de74a8700 1 -- 172.21.15.113:0/470406273 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de0071590 con 0x7f8de00fdd60 2024-04-12T05:09:51.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.235+0000 7f8de74a8700 1 -- 172.21.15.113:0/470406273 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de0071b20 con 0x7f8de00fdd60 2024-04-12T05:09:51.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.236+0000 7f8de74a8700 1 -- 172.21.15.113:0/470406273 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de0106690 con 0x7f8de00fdd60 2024-04-12T05:09:51.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.237+0000 7f8dd67fc700 1 -- 172.21.15.113:0/470406273 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7f8ddc00dca0 con 0x7f8de00fdd60 2024-04-12T05:09:51.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.237+0000 7f8dd67fc700 1 --2- 172.21.15.113:0/470406273 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dcc066ee0 0x7f8dcc0693a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:51.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.237+0000 7f8dd67fc700 1 -- 172.21.15.113:0/470406273 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f8ddc083bf0 con 0x7f8de00fdd60 2024-04-12T05:09:51.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.240+0000 7f8de4a43700 1 --2- 172.21.15.113:0/470406273 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dcc066ee0 0x7f8dcc0693a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:09:51.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.241+0000 7f8de4a43700 1 --2- 172.21.15.113:0/470406273 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dcc066ee0 0x7f8dcc0693a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f8dd0000c00 tx=0x7f8dd000b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:51.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.247+0000 7f8dd67fc700 1 -- 172.21.15.113:0/470406273 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f8ddc051de0 con 0x7f8de00fdd60 2024-04-12T05:09:51.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.507+0000 7f8de74a8700 1 -- 172.21.15.113:0/470406273 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8de0059120 con 0x7f8dcc066ee0 2024-04-12T05:09:51.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.517+0000 7f8dd67fc700 1 -- 172.21.15.113:0/470406273 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f8de0059120 con 0x7f8dcc066ee0 2024-04-12T05:09:51.519 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:09:51.519 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 starting - - - - 2024-04-12T05:09:51.519 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (8m) 71s ago 8m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:09:51.519 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (8m) 46s ago 8m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:09:51.519 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (7m) 71s ago 8m 60.7M - 8.3.5 dad864ee21e9 7d83fd029dea 2024-04-12T05:09:51.519 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (3m) 71s ago 3m 97.7M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (3m) 71s ago 3m 15.4M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (3m) 46s ago 3m 109M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (3m) 46s ago 3m 12.9M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:9283 running (10m) 71s ago 10m 382M - 16.2.15-1-g8963e9de 727f386679cb 3fab7f291e6b 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283 running (90s) 46s ago 8m 479M - 18.2.2-1187-gd540ebac bb2c5eac26a4 550d8cc53ae9 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (10m) 71s ago 10m 49.3M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (8m) 46s ago 8m 41.4M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (8m) 71s ago 8m 22.5M - 1.3.1 1dbe0e931976 046bcfb16579 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (8m) 46s ago 8m 28.1M - 1.3.1 1dbe0e931976 bc604318825c 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (7m) 71s ago 7m 109M 1910M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (6m) 71s ago 6m 124M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (6m) 71s ago 6m 102M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:09:51.520 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (6m) 46s ago 6m 130M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:09:51.521 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (5m) 46s ago 5m 121M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:09:51.521 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (5m) 46s ago 5m 110M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:09:51.521 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (7m) 46s ago 7m 62.7M - 2.33.4 514e6a882f6e b4259537bd55 2024-04-12T05:09:51.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.519+0000 7f8de74a8700 1 -- 172.21.15.113:0/470406273 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dcc066ee0 msgr2=0x7f8dcc0693a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:51.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.519+0000 7f8de74a8700 1 --2- 172.21.15.113:0/470406273 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dcc066ee0 0x7f8dcc0693a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f8dd0000c00 tx=0x7f8dd000b040).stop 2024-04-12T05:09:51.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.519+0000 7f8de74a8700 1 -- 172.21.15.113:0/470406273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de00fdd60 msgr2=0x7f8de010e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:51.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.519+0000 7f8de74a8700 1 --2- 172.21.15.113:0/470406273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de00fdd60 0x7f8de010e7b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f8ddc00b410 tx=0x7f8ddc009060).stop 2024-04-12T05:09:51.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.519+0000 7f8de74a8700 1 -- 172.21.15.113:0/470406273 shutdown_connections 2024-04-12T05:09:51.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.519+0000 7f8de74a8700 1 --2- 172.21.15.113:0/470406273 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f8dcc066ee0 0x7f8dcc0693a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:51.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.519+0000 7f8de74a8700 1 --2- 172.21.15.113:0/470406273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8de00fdd60 0x7f8de010e7b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:51.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.519+0000 7f8de74a8700 1 --2- 172.21.15.113:0/470406273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8de010eca0 0x7f8de0113d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:09:51.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.519+0000 7f8de74a8700 1 -- 172.21.15.113:0/470406273 >> 172.21.15.113:0/470406273 conn(0x7f8de006d0d0 msgr2=0x7f8de0102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:51.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.519+0000 7f8de74a8700 1 -- 172.21.15.113:0/470406273 shutdown_connections 2024-04-12T05:09:51.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.519+0000 7f8de74a8700 1 -- 172.21.15.113:0/470406273 wait complete. 2024-04-12T05:09:51.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.633+0000 7f0170920700 1 -- 172.21.15.113:0/1385774581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016c0611c0 msgr2=0x7f016c0fa490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:51.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.633+0000 7f0170920700 1 --2- 172.21.15.113:0/1385774581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016c0611c0 0x7f016c0fa490 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f015c0047f0 tx=0x7f015c009cb0).stop 2024-04-12T05:09:51.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.633+0000 7f0170920700 1 -- 172.21.15.113:0/1385774581 shutdown_connections 2024-04-12T05:09:51.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.633+0000 7f0170920700 1 --2- 172.21.15.113:0/1385774581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c0fa980 0x7f016c0fcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:51.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.633+0000 7f0170920700 1 --2- 172.21.15.113:0/1385774581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016c0611c0 0x7f016c0fa490 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:51.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.633+0000 7f0170920700 1 -- 172.21.15.113:0/1385774581 >> 172.21.15.113:0/1385774581 conn(0x7f016c0f1e50 msgr2=0x7f016c0f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:51.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.634+0000 7f0170920700 1 -- 172.21.15.113:0/1385774581 shutdown_connections 2024-04-12T05:09:51.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.634+0000 7f0170920700 1 -- 172.21.15.113:0/1385774581 wait complete. 2024-04-12T05:09:51.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.634+0000 7f0170920700 1 Processor -- start 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.634+0000 7f0170920700 1 -- start start 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.634+0000 7f0170920700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c0611c0 0x7f016c0f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.634+0000 7f0170920700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016c0fa980 0x7f016c0f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.634+0000 7f0170920700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c0f82e0 con 0x7f016c0fa980 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.634+0000 7f0170920700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c0f6880 con 0x7f016c0611c0 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f0163fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016c0fa980 0x7f016c0f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f016a59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c0611c0 0x7f016c0f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f0163fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016c0fa980 0x7f016c0f6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52030/0 (socket says 172.21.15.113:52030) 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f0163fff700 1 -- 172.21.15.113:0/2025770684 learned_addr learned my addr 172.21.15.113:0/2025770684 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f016a59c700 1 -- 172.21.15.113:0/2025770684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016c0fa980 msgr2=0x7f016c0f6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f016a59c700 1 --2- 172.21.15.113:0/2025770684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016c0fa980 0x7f016c0f6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f016a59c700 1 -- 172.21.15.113:0/2025770684 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c005040 con 0x7f016c0611c0 2024-04-12T05:09:51.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f016a59c700 1 --2- 172.21.15.113:0/2025770684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c0611c0 0x7f016c0f7d30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f015c013560 tx=0x7f015c009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:51.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f01637fe700 1 -- 172.21.15.113:0/2025770684 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f015c018070 con 0x7f016c0611c0 2024-04-12T05:09:51.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f01637fe700 1 -- 172.21.15.113:0/2025770684 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f015c007e90 con 0x7f016c0611c0 2024-04-12T05:09:51.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f0170920700 1 -- 172.21.15.113:0/2025770684 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016c0f6aa0 con 0x7f016c0611c0 2024-04-12T05:09:51.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f01637fe700 1 -- 172.21.15.113:0/2025770684 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f015c012620 con 0x7f016c0611c0 2024-04-12T05:09:51.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.635+0000 7f0170920700 1 -- 172.21.15.113:0/2025770684 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016c0f7000 con 0x7f016c0611c0 2024-04-12T05:09:51.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.637+0000 7f0170920700 1 -- 172.21.15.113:0/2025770684 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016c100000 con 0x7f016c0611c0 2024-04-12T05:09:51.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.637+0000 7f01637fe700 1 -- 172.21.15.113:0/2025770684 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7f015c01d420 con 0x7f016c0611c0 2024-04-12T05:09:51.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.637+0000 7f01637fe700 1 --2- 172.21.15.113:0/2025770684 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0158066dc0 0x7f0158069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:51.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.637+0000 7f01637fe700 1 -- 172.21.15.113:0/2025770684 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f015c085130 con 0x7f016c0611c0 2024-04-12T05:09:51.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.637+0000 7f0163fff700 1 --2- 172.21.15.113:0/2025770684 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0158066dc0 0x7f0158069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:09:51.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.638+0000 7f0163fff700 1 --2- 172.21.15.113:0/2025770684 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0158066dc0 0x7f0158069280 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0154004e20 tx=0x7f015400b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:51.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:51.644+0000 7f01637fe700 1 -- 172.21.15.113:0/2025770684 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f015c0531d0 con 0x7f016c0611c0 2024-04-12T05:09:52.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.030+0000 7f0170920700 1 -- 172.21.15.113:0/2025770684 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f016c0598f0 con 0x7f016c0611c0 2024-04-12T05:09:52.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.031+0000 7f01637fe700 1 -- 172.21.15.113:0/2025770684 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+850 (secure 0 0 0) 0x7f015c052dc0 con 0x7f016c0611c0 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:09:52.033 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:09:52.034 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:09:52.034 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:09:52.034 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:09:52.034 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 13, 2024-04-12T05:09:52.034 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:09:52.034 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:09:52.034 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:09:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.033+0000 7f0170920700 1 -- 172.21.15.113:0/2025770684 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0158066dc0 msgr2=0x7f0158069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.033+0000 7f0170920700 1 --2- 172.21.15.113:0/2025770684 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0158066dc0 0x7f0158069280 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0154004e20 tx=0x7f015400b040).stop 2024-04-12T05:09:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.033+0000 7f0170920700 1 -- 172.21.15.113:0/2025770684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c0611c0 msgr2=0x7f016c0f7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.033+0000 7f0170920700 1 --2- 172.21.15.113:0/2025770684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c0611c0 0x7f016c0f7d30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f015c013560 tx=0x7f015c009cb0).stop 2024-04-12T05:09:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.033+0000 7f0170920700 1 -- 172.21.15.113:0/2025770684 shutdown_connections 2024-04-12T05:09:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.033+0000 7f0170920700 1 --2- 172.21.15.113:0/2025770684 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f0158066dc0 0x7f0158069280 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.033+0000 7f0170920700 1 --2- 172.21.15.113:0/2025770684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c0611c0 0x7f016c0f7d30 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0x7f015c013560 tx=0x7f015c009cb0).stop 2024-04-12T05:09:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.033+0000 7f0170920700 1 --2- 172.21.15.113:0/2025770684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f016c0fa980 0x7f016c0f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.033+0000 7f0170920700 1 -- 172.21.15.113:0/2025770684 >> 172.21.15.113:0/2025770684 conn(0x7f016c0f1e50 msgr2=0x7f016c0f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.034+0000 7f0170920700 1 -- 172.21.15.113:0/2025770684 shutdown_connections 2024-04-12T05:09:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.034+0000 7f0170920700 1 -- 172.21.15.113:0/2025770684 wait complete. 2024-04-12T05:09:52.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.175+0000 7fe51046c700 1 -- 172.21.15.113:0/74324307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5080fa9e0 msgr2=0x7fe5080fcdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:52.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.175+0000 7fe51046c700 1 --2- 172.21.15.113:0/74324307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5080fa9e0 0x7fe5080fcdd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fe4f8004730 tx=0x7fe4f8009cb0).stop 2024-04-12T05:09:52.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.175+0000 7fe51046c700 1 -- 172.21.15.113:0/74324307 shutdown_connections 2024-04-12T05:09:52.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.175+0000 7fe51046c700 1 --2- 172.21.15.113:0/74324307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5080fa9e0 0x7fe5080fcdd0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0x7fe4f8004730 tx=0x7fe4f8009cb0).stop 2024-04-12T05:09:52.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.175+0000 7fe51046c700 1 --2- 172.21.15.113:0/74324307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe508061710 0x7fe5080fa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:52.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.175+0000 7fe51046c700 1 -- 172.21.15.113:0/74324307 >> 172.21.15.113:0/74324307 conn(0x7fe5080f1f10 msgr2=0x7fe5080f4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:52.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.176+0000 7fe51046c700 1 -- 172.21.15.113:0/74324307 shutdown_connections 2024-04-12T05:09:52.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.176+0000 7fe51046c700 1 -- 172.21.15.113:0/74324307 wait complete. 2024-04-12T05:09:52.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.177+0000 7fe51046c700 1 Processor -- start 2024-04-12T05:09:52.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.177+0000 7fe51046c700 1 -- start start 2024-04-12T05:09:52.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.177+0000 7fe51046c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe508061710 0x7fe508069ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:52.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.177+0000 7fe51046c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe50806a4e0 0x7fe50811c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:52.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.177+0000 7fe51046c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50806a9b0 con 0x7fe50806a4e0 2024-04-12T05:09:52.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.177+0000 7fe51046c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50806aaf0 con 0x7fe508061710 2024-04-12T05:09:52.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.177+0000 7fe50da07700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe50806a4e0 0x7fe50811c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:52.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.177+0000 7fe50da07700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe50806a4e0 0x7fe50811c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52048/0 (socket says 172.21.15.113:52048) 2024-04-12T05:09:52.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.177+0000 7fe50da07700 1 -- 172.21.15.113:0/2393989815 learned_addr learned my addr 172.21.15.113:0/2393989815 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:09:52.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.177+0000 7fe50e208700 1 --2- 172.21.15.113:0/2393989815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe508061710 0x7fe508069ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:09:52.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.178+0000 7fe50da07700 1 -- 172.21.15.113:0/2393989815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe508061710 msgr2=0x7fe508069ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:52.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.178+0000 7fe50da07700 1 --2- 172.21.15.113:0/2393989815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe508061710 0x7fe508069ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:52.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.178+0000 7fe50da07700 1 -- 172.21.15.113:0/2393989815 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f8005040 con 0x7fe50806a4e0 2024-04-12T05:09:52.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.178+0000 7fe50da07700 1 --2- 172.21.15.113:0/2393989815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe50806a4e0 0x7fe50811c520 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fe4f8004650 tx=0x7fe4f8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:52.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.178+0000 7fe4ff7fe700 1 -- 172.21.15.113:0/2393989815 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4f8009320 con 0x7fe50806a4e0 2024-04-12T05:09:52.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.178+0000 7fe4ff7fe700 1 -- 172.21.15.113:0/2393989815 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4f8007e90 con 0x7fe50806a4e0 2024-04-12T05:09:52.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.178+0000 7fe4ff7fe700 1 -- 172.21.15.113:0/2393989815 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4f801d3a0 con 0x7fe50806a4e0 2024-04-12T05:09:52.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.180+0000 7fe51046c700 1 -- 172.21.15.113:0/2393989815 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe50811ca10 con 0x7fe50806a4e0 2024-04-12T05:09:52.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.180+0000 7fe51046c700 1 -- 172.21.15.113:0/2393989815 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50811ce30 con 0x7fe50806a4e0 2024-04-12T05:09:52.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.181+0000 7fe51046c700 1 -- 172.21.15.113:0/2393989815 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe508047540 con 0x7fe50806a4e0 2024-04-12T05:09:52.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.183+0000 7fe4ff7fe700 1 -- 172.21.15.113:0/2393989815 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 84062+0+0 (secure 0 0 0) 0x7fe4f8014420 con 0x7fe50806a4e0 2024-04-12T05:09:52.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.183+0000 7fe4ff7fe700 1 --2- 172.21.15.113:0/2393989815 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe4f4066e10 0x7fe4f40692d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:09:52.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.183+0000 7fe4ff7fe700 1 -- 172.21.15.113:0/2393989815 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7fe4f8085320 con 0x7fe50806a4e0 2024-04-12T05:09:52.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.184+0000 7fe50e208700 1 --2- 172.21.15.113:0/2393989815 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe4f4066e10 0x7fe4f40692d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:09:52.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.187+0000 7fe50e208700 1 --2- 172.21.15.113:0/2393989815 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe4f4066e10 0x7fe4f40692d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe5040043d0 tx=0x7fe50400b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:09:52.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.193+0000 7fe4ff7fe700 1 -- 172.21.15.113:0/2393989815 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fe4f8053300 con 0x7fe50806a4e0 2024-04-12T05:09:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:50.633373+0000 mgr.smithi192.ocppyo (mgr.24357) 69 : audit [DBG] from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:50.683086+0000 mgr.smithi192.ocppyo (mgr.24357) 70 : cluster [DBG] pgmap v45: 65 pgs: 65 active+clean; 294 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.5 MiB/s wr, 415 op/s 2024-04-12T05:09:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:51.074460+0000 mgr.smithi192.ocppyo (mgr.24357) 71 : audit [DBG] from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:52.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.426+0000 7fe51046c700 1 -- 172.21.15.113:0/2393989815 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe508059120 con 0x7fe4f4066e10 2024-04-12T05:09:52.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.428+0000 7fe4ff7fe700 1 -- 172.21.15.113:0/2393989815 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fe508059120 con 0x7fe4f4066e10 2024-04-12T05:09:52.430 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:09:52.430 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:09:52.430 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:09:52.430 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:09:52.430 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [], 2024-04-12T05:09:52.430 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "1/2 daemons upgraded", 2024-04-12T05:09:52.430 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "", 2024-04-12T05:09:52.430 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:09:52.431 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:09:52.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.430+0000 7fe51046c700 1 -- 172.21.15.113:0/2393989815 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe4f4066e10 msgr2=0x7fe4f40692d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:52.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.430+0000 7fe51046c700 1 --2- 172.21.15.113:0/2393989815 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe4f4066e10 0x7fe4f40692d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe5040043d0 tx=0x7fe50400b040).stop 2024-04-12T05:09:52.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.430+0000 7fe51046c700 1 -- 172.21.15.113:0/2393989815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe50806a4e0 msgr2=0x7fe50811c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:09:52.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.430+0000 7fe51046c700 1 --2- 172.21.15.113:0/2393989815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe50806a4e0 0x7fe50811c520 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fe4f8004650 tx=0x7fe4f8009cb0).stop 2024-04-12T05:09:52.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.430+0000 7fe51046c700 1 -- 172.21.15.113:0/2393989815 shutdown_connections 2024-04-12T05:09:52.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.430+0000 7fe51046c700 1 --2- 172.21.15.113:0/2393989815 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fe4f4066e10 0x7fe4f40692d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:52.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.430+0000 7fe51046c700 1 --2- 172.21.15.113:0/2393989815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe508061710 0x7fe508069ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:52.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.430+0000 7fe51046c700 1 --2- 172.21.15.113:0/2393989815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe50806a4e0 0x7fe50811c520 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:09:52.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.430+0000 7fe51046c700 1 -- 172.21.15.113:0/2393989815 >> 172.21.15.113:0/2393989815 conn(0x7fe5080f1f10 msgr2=0x7fe5080f4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:09:52.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.430+0000 7fe51046c700 1 -- 172.21.15.113:0/2393989815 shutdown_connections 2024-04-12T05:09:52.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:09:52.430+0000 7fe51046c700 1 -- 172.21.15.113:0/2393989815 wait complete. 2024-04-12T05:09:52.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:50.633373+0000 mgr.smithi192.ocppyo (mgr.24357) 69 : audit [DBG] from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:52.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:50.683086+0000 mgr.smithi192.ocppyo (mgr.24357) 70 : cluster [DBG] pgmap v45: 65 pgs: 65 active+clean; 294 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.5 MiB/s wr, 415 op/s 2024-04-12T05:09:52.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:51.074460+0000 mgr.smithi192.ocppyo (mgr.24357) 71 : audit [DBG] from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:51.509105+0000 mgr.smithi192.ocppyo (mgr.24357) 72 : audit [DBG] from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:52.031736+0000 mon.smithi192 (mon.1) 61 : audit [DBG] from='client.? 172.21.15.113:0/2025770684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:09:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:51.509105+0000 mgr.smithi192.ocppyo (mgr.24357) 72 : audit [DBG] from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:52.031736+0000 mon.smithi192 (mon.1) 61 : audit [DBG] from='client.? 172.21.15.113:0/2025770684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:09:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:09:52.428976+0000 mgr.smithi192.ocppyo (mgr.24357) 73 : audit [DBG] from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:52.683501+0000 mgr.smithi192.ocppyo (mgr.24357) 74 : cluster [DBG] pgmap v46: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 63 KiB/s rd, 2.4 MiB/s wr, 645 op/s 2024-04-12T05:09:54.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:09:52.428976+0000 mgr.smithi192.ocppyo (mgr.24357) 73 : audit [DBG] from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:09:54.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:52.683501+0000 mgr.smithi192.ocppyo (mgr.24357) 74 : cluster [DBG] pgmap v46: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 63 KiB/s rd, 2.4 MiB/s wr, 645 op/s 2024-04-12T05:09:56.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:54.683817+0000 mgr.smithi192.ocppyo (mgr.24357) 75 : cluster [DBG] pgmap v47: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 1.5 MiB/s wr, 437 op/s 2024-04-12T05:09:56.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:54.683817+0000 mgr.smithi192.ocppyo (mgr.24357) 75 : cluster [DBG] pgmap v47: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 1.5 MiB/s wr, 437 op/s 2024-04-12T05:09:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:09:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:56.684105+0000 mgr.smithi192.ocppyo (mgr.24357) 76 : cluster [DBG] pgmap v48: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 1.5 MiB/s wr, 437 op/s 2024-04-12T05:09:58.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:09:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:56.684105+0000 mgr.smithi192.ocppyo (mgr.24357) 76 : cluster [DBG] pgmap v48: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 1.5 MiB/s wr, 437 op/s 2024-04-12T05:10:00.272 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:09:58.684645+0000 mgr.smithi192.ocppyo (mgr.24357) 77 : cluster [DBG] pgmap v49: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 64 KiB/s rd, 2.4 MiB/s wr, 657 op/s 2024-04-12T05:10:00.273 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:00.000130+0000 mon.smithi113 (mon.0) 701 : cluster [INF] overall HEALTH_OK 2024-04-12T05:10:00.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:09:58.684645+0000 mgr.smithi192.ocppyo (mgr.24357) 77 : cluster [DBG] pgmap v49: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 64 KiB/s rd, 2.4 MiB/s wr, 657 op/s 2024-04-12T05:10:00.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:00.000130+0000 mon.smithi113 (mon.0) 701 : cluster [INF] overall HEALTH_OK 2024-04-12T05:10:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:00.371968+0000 mon.smithi113 (mon.0) 702 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:00.381834+0000 mon.smithi113 (mon.0) 703 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:00.414338+0000 mgr.smithi192.ocppyo (mgr.24357) 78 : cephadm [INF] Reconfiguring node-exporter.smithi113 (dependencies changed)... 2024-04-12T05:10:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:00.414680+0000 mgr.smithi192.ocppyo (mgr.24357) 79 : cephadm [INF] Reconfiguring daemon node-exporter.smithi113 on smithi113 2024-04-12T05:10:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:00.684949+0000 mgr.smithi192.ocppyo (mgr.24357) 80 : cluster [DBG] pgmap v50: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 46 KiB/s rd, 1.7 MiB/s wr, 450 op/s 2024-04-12T05:10:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:00.910897+0000 mon.smithi192 (mon.1) 62 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:10:01.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:00.371968+0000 mon.smithi113 (mon.0) 702 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:01.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:00.381834+0000 mon.smithi113 (mon.0) 703 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:01.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:00.414338+0000 mgr.smithi192.ocppyo (mgr.24357) 78 : cephadm [INF] Reconfiguring node-exporter.smithi113 (dependencies changed)... 2024-04-12T05:10:01.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:00.414680+0000 mgr.smithi192.ocppyo (mgr.24357) 79 : cephadm [INF] Reconfiguring daemon node-exporter.smithi113 on smithi113 2024-04-12T05:10:01.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:00.684949+0000 mgr.smithi192.ocppyo (mgr.24357) 80 : cluster [DBG] pgmap v50: 65 pgs: 65 active+clean; 280 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 46 KiB/s rd, 1.7 MiB/s wr, 450 op/s 2024-04-12T05:10:01.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:00.910897+0000 mon.smithi192 (mon.1) 62 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:10:02.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:01.389351+0000 mon.smithi113 (mon.0) 704 : cluster [DBG] mgrmap e25: smithi192.ocppyo(active, since 90s), standbys: smithi113.ctsazg 2024-04-12T05:10:02.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:01.389351+0000 mon.smithi113 (mon.0) 704 : cluster [DBG] mgrmap e25: smithi192.ocppyo(active, since 90s), standbys: smithi113.ctsazg 2024-04-12T05:10:04.017 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:02.685388+0000 mgr.smithi192.ocppyo (mgr.24357) 81 : cluster [DBG] pgmap v51: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 68 KiB/s rd, 2.4 MiB/s wr, 674 op/s 2024-04-12T05:10:04.017 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:02.765501+0000 mon.smithi113 (mon.0) 705 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:04.018 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:02.774808+0000 mon.smithi113 (mon.0) 706 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:04.018 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:02.785468+0000 mgr.smithi192.ocppyo (mgr.24357) 82 : cephadm [INF] Reconfiguring node-exporter.smithi192 (dependencies changed)... 2024-04-12T05:10:04.018 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:02.785778+0000 mgr.smithi192.ocppyo (mgr.24357) 83 : cephadm [INF] Reconfiguring daemon node-exporter.smithi192 on smithi192 2024-04-12T05:10:04.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:02.685388+0000 mgr.smithi192.ocppyo (mgr.24357) 81 : cluster [DBG] pgmap v51: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 68 KiB/s rd, 2.4 MiB/s wr, 674 op/s 2024-04-12T05:10:04.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:02.765501+0000 mon.smithi113 (mon.0) 705 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:04.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:02.774808+0000 mon.smithi113 (mon.0) 706 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:04.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:02.785468+0000 mgr.smithi192.ocppyo (mgr.24357) 82 : cephadm [INF] Reconfiguring node-exporter.smithi192 (dependencies changed)... 2024-04-12T05:10:04.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:02.785778+0000 mgr.smithi192.ocppyo (mgr.24357) 83 : cephadm [INF] Reconfiguring daemon node-exporter.smithi192 on smithi192 2024-04-12T05:10:06.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:04.685680+0000 mgr.smithi192.ocppyo (mgr.24357) 84 : cluster [DBG] pgmap v52: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 45 KiB/s rd, 1.6 MiB/s wr, 444 op/s 2024-04-12T05:10:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:05.123465+0000 mon.smithi113 (mon.0) 707 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:05.129080+0000 mon.smithi113 (mon.0) 708 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:05.161416+0000 mgr.smithi192.ocppyo (mgr.24357) 85 : cephadm [INF] Reconfiguring prometheus.smithi192 (dependencies changed)... 2024-04-12T05:10:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:04.685680+0000 mgr.smithi192.ocppyo (mgr.24357) 84 : cluster [DBG] pgmap v52: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 45 KiB/s rd, 1.6 MiB/s wr, 444 op/s 2024-04-12T05:10:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:05.123465+0000 mon.smithi113 (mon.0) 707 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:05.129080+0000 mon.smithi113 (mon.0) 708 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:05.161416+0000 mgr.smithi192.ocppyo (mgr.24357) 85 : cephadm [INF] Reconfiguring prometheus.smithi192 (dependencies changed)... 2024-04-12T05:10:07.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:05.403457+0000 mgr.smithi192.ocppyo (mgr.24357) 86 : cephadm [INF] Reconfiguring daemon prometheus.smithi192 on smithi192 2024-04-12T05:10:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:05.403457+0000 mgr.smithi192.ocppyo (mgr.24357) 86 : cephadm [INF] Reconfiguring daemon prometheus.smithi192 on smithi192 2024-04-12T05:10:08.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:08 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:06.686017+0000 mgr.smithi192.ocppyo (mgr.24357) 87 : cluster [DBG] pgmap v53: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 45 KiB/s rd, 1.6 MiB/s wr, 444 op/s 2024-04-12T05:10:08.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:08 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:06.686017+0000 mgr.smithi192.ocppyo (mgr.24357) 87 : cluster [DBG] pgmap v53: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 45 KiB/s rd, 1.6 MiB/s wr, 444 op/s 2024-04-12T05:10:10.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:08.686527+0000 mgr.smithi192.ocppyo (mgr.24357) 88 : cluster [DBG] pgmap v54: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 64 KiB/s rd, 2.3 MiB/s wr, 658 op/s 2024-04-12T05:10:10.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:08.686527+0000 mgr.smithi192.ocppyo (mgr.24357) 88 : cluster [DBG] pgmap v54: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 64 KiB/s rd, 2.3 MiB/s wr, 658 op/s 2024-04-12T05:10:12.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:10.686939+0000 mgr.smithi192.ocppyo (mgr.24357) 89 : cluster [DBG] pgmap v55: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 1.5 MiB/s wr, 438 op/s 2024-04-12T05:10:12.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:10.686939+0000 mgr.smithi192.ocppyo (mgr.24357) 89 : cluster [DBG] pgmap v55: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 42 KiB/s rd, 1.5 MiB/s wr, 438 op/s 2024-04-12T05:10:14.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:12.687363+0000 mgr.smithi192.ocppyo (mgr.24357) 90 : cluster [DBG] pgmap v56: 65 pgs: 65 active+clean; 277 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 62 KiB/s rd, 2.2 MiB/s wr, 650 op/s 2024-04-12T05:10:14.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:12.687363+0000 mgr.smithi192.ocppyo (mgr.24357) 90 : cluster [DBG] pgmap v56: 65 pgs: 65 active+clean; 277 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 62 KiB/s rd, 2.2 MiB/s wr, 650 op/s 2024-04-12T05:10:16.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:14.687718+0000 mgr.smithi192.ocppyo (mgr.24357) 91 : cluster [DBG] pgmap v57: 65 pgs: 65 active+clean; 277 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.5 MiB/s wr, 426 op/s 2024-04-12T05:10:16.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:15.911097+0000 mon.smithi192 (mon.1) 63 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:10:16.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:14.687718+0000 mgr.smithi192.ocppyo (mgr.24357) 91 : cluster [DBG] pgmap v57: 65 pgs: 65 active+clean; 277 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.5 MiB/s wr, 426 op/s 2024-04-12T05:10:16.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:15.911097+0000 mon.smithi192 (mon.1) 63 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:10:18.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:16.688115+0000 mgr.smithi192.ocppyo (mgr.24357) 92 : cluster [DBG] pgmap v58: 65 pgs: 65 active+clean; 277 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.5 MiB/s wr, 426 op/s 2024-04-12T05:10:18.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:16.688115+0000 mgr.smithi192.ocppyo (mgr.24357) 92 : cluster [DBG] pgmap v58: 65 pgs: 65 active+clean; 277 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 40 KiB/s rd, 1.5 MiB/s wr, 426 op/s 2024-04-12T05:10:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:18.688691+0000 mgr.smithi192.ocppyo (mgr.24357) 93 : cluster [DBG] pgmap v59: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 58 KiB/s rd, 2.1 MiB/s wr, 634 op/s 2024-04-12T05:10:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:18.688691+0000 mgr.smithi192.ocppyo (mgr.24357) 93 : cluster [DBG] pgmap v59: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 58 KiB/s rd, 2.1 MiB/s wr, 634 op/s 2024-04-12T05:10:22.342 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-12T05:10:22.342 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-12T05:10:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:20.689060+0000 mgr.smithi192.ocppyo (mgr.24357) 94 : cluster [DBG] pgmap v60: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 39 KiB/s rd, 1.4 MiB/s wr, 420 op/s 2024-04-12T05:10:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:20.689060+0000 mgr.smithi192.ocppyo (mgr.24357) 94 : cluster [DBG] pgmap v60: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 39 KiB/s rd, 1.4 MiB/s wr, 420 op/s 2024-04-12T05:10:22.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.540+0000 7f4b41cb0700 1 -- 172.21.15.113:0/3202411537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fa980 msgr2=0x7f4b3c0fcd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:22.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.540+0000 7f4b41cb0700 1 --2- 172.21.15.113:0/3202411537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fa980 0x7f4b3c0fcd70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f4b240047f0 tx=0x7f4b24009cb0).stop 2024-04-12T05:10:22.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.540+0000 7f4b41cb0700 1 -- 172.21.15.113:0/3202411537 shutdown_connections 2024-04-12T05:10:22.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.540+0000 7f4b41cb0700 1 --2- 172.21.15.113:0/3202411537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fa980 0x7f4b3c0fcd70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:22.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.540+0000 7f4b41cb0700 1 --2- 172.21.15.113:0/3202411537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c061790 0x7f4b3c0fa490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:22.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.540+0000 7f4b41cb0700 1 -- 172.21.15.113:0/3202411537 >> 172.21.15.113:0/3202411537 conn(0x7f4b3c0f1f10 msgr2=0x7f4b3c0f4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:22.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.540+0000 7f4b41cb0700 1 -- 172.21.15.113:0/3202411537 shutdown_connections 2024-04-12T05:10:22.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.540+0000 7f4b41cb0700 1 -- 172.21.15.113:0/3202411537 wait complete. 2024-04-12T05:10:22.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.541+0000 7f4b41cb0700 1 Processor -- start 2024-04-12T05:10:22.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.541+0000 7f4b41cb0700 1 -- start start 2024-04-12T05:10:22.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.541+0000 7f4b41cb0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c061790 0x7f4b3c107f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:22.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.541+0000 7f4b41cb0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fa980 0x7f4b3c108480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:22.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.541+0000 7f4b41cb0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c108a30 con 0x7f4b3c0fa980 2024-04-12T05:10:22.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.541+0000 7f4b41cb0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c108b70 con 0x7f4b3c061790 2024-04-12T05:10:22.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.541+0000 7f4b33fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fa980 0x7f4b3c108480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:22.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.541+0000 7f4b33fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fa980 0x7f4b3c108480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44284/0 (socket says 172.21.15.113:44284) 2024-04-12T05:10:22.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.541+0000 7f4b33fff700 1 -- 172.21.15.113:0/1942008175 learned_addr learned my addr 172.21.15.113:0/1942008175 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:10:22.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.542+0000 7f4b3b7fe700 1 --2- 172.21.15.113:0/1942008175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c061790 0x7f4b3c107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:22.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.542+0000 7f4b33fff700 1 -- 172.21.15.113:0/1942008175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c061790 msgr2=0x7f4b3c107f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:22.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.542+0000 7f4b33fff700 1 --2- 172.21.15.113:0/1942008175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c061790 0x7f4b3c107f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:22.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.542+0000 7f4b33fff700 1 -- 172.21.15.113:0/1942008175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b24005040 con 0x7f4b3c0fa980 2024-04-12T05:10:22.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.542+0000 7f4b33fff700 1 --2- 172.21.15.113:0/1942008175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fa980 0x7f4b3c108480 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f4b24005010 tx=0x7f4b24009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:22.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.542+0000 7f4b397fa700 1 -- 172.21.15.113:0/1942008175 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b2400e030 con 0x7f4b3c0fa980 2024-04-12T05:10:22.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.542+0000 7f4b397fa700 1 -- 172.21.15.113:0/1942008175 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b24007e90 con 0x7f4b3c0fa980 2024-04-12T05:10:22.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.542+0000 7f4b41cb0700 1 -- 172.21.15.113:0/1942008175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c10d5b0 con 0x7f4b3c0fa980 2024-04-12T05:10:22.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.542+0000 7f4b397fa700 1 -- 172.21.15.113:0/1942008175 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b24018070 con 0x7f4b3c0fa980 2024-04-12T05:10:22.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.542+0000 7f4b41cb0700 1 -- 172.21.15.113:0/1942008175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c10db70 con 0x7f4b3c0fa980 2024-04-12T05:10:22.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.543+0000 7f4b41cb0700 1 -- 172.21.15.113:0/1942008175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c0fffe0 con 0x7f4b3c0fa980 2024-04-12T05:10:22.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.544+0000 7f4b397fa700 1 -- 172.21.15.113:0/1942008175 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 25) v1 ==== 83960+0+0 (secure 0 0 0) 0x7f4b2401d420 con 0x7f4b3c0fa980 2024-04-12T05:10:22.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.544+0000 7f4b397fa700 1 --2- 172.21.15.113:0/1942008175 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4b28066ca0 0x7f4b28069160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:22.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.544+0000 7f4b397fa700 1 -- 172.21.15.113:0/1942008175 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f4b240854e0 con 0x7f4b3c0fa980 2024-04-12T05:10:22.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.544+0000 7f4b3b7fe700 1 --2- 172.21.15.113:0/1942008175 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4b28066ca0 0x7f4b28069160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:10:22.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.545+0000 7f4b3b7fe700 1 --2- 172.21.15.113:0/1942008175 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4b28066ca0 0x7f4b28069160 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4b3c061560 tx=0x7f4b2c009d80).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:22.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.549+0000 7f4b397fa700 1 -- 172.21.15.113:0/1942008175 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f4b240538a0 con 0x7f4b3c0fa980 2024-04-12T05:10:22.686 DEBUG:teuthology.parallel:result is None 2024-04-12T05:10:22.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.830+0000 7f4b41cb0700 1 -- 172.21.15.113:0/1942008175 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b3c059120 con 0x7f4b28066ca0 2024-04-12T05:10:22.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.832+0000 7f4b397fa700 1 -- 172.21.15.113:0/1942008175 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f4b3c059120 con 0x7f4b28066ca0 2024-04-12T05:10:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.834+0000 7f4b41cb0700 1 -- 172.21.15.113:0/1942008175 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4b28066ca0 msgr2=0x7f4b28069160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.834+0000 7f4b41cb0700 1 --2- 172.21.15.113:0/1942008175 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4b28066ca0 0x7f4b28069160 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4b3c061560 tx=0x7f4b2c009d80).stop 2024-04-12T05:10:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.834+0000 7f4b41cb0700 1 -- 172.21.15.113:0/1942008175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fa980 msgr2=0x7f4b3c108480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.834+0000 7f4b41cb0700 1 --2- 172.21.15.113:0/1942008175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fa980 0x7f4b3c108480 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f4b24005010 tx=0x7f4b24009cb0).stop 2024-04-12T05:10:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.834+0000 7f4b41cb0700 1 -- 172.21.15.113:0/1942008175 shutdown_connections 2024-04-12T05:10:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.834+0000 7f4b41cb0700 1 --2- 172.21.15.113:0/1942008175 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4b28066ca0 0x7f4b28069160 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.834+0000 7f4b41cb0700 1 --2- 172.21.15.113:0/1942008175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c061790 0x7f4b3c107f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.834+0000 7f4b41cb0700 1 --2- 172.21.15.113:0/1942008175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fa980 0x7f4b3c108480 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.834+0000 7f4b41cb0700 1 -- 172.21.15.113:0/1942008175 >> 172.21.15.113:0/1942008175 conn(0x7f4b3c0f1f10 msgr2=0x7f4b3c0f4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.834+0000 7f4b41cb0700 1 -- 172.21.15.113:0/1942008175 shutdown_connections 2024-04-12T05:10:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.834+0000 7f4b41cb0700 1 -- 172.21.15.113:0/1942008175 wait complete. 2024-04-12T05:10:22.847 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:10:22.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.957+0000 7f61e8fc8700 1 -- 172.21.15.113:0/2209618087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61e40f8aa0 msgr2=0x7f61e40f8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:22.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.957+0000 7f61e8fc8700 1 --2- 172.21.15.113:0/2209618087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61e40f8aa0 0x7f61e40f8f00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f61d80047f0 tx=0x7f61d8009cb0).stop 2024-04-12T05:10:22.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.957+0000 7f61e8fc8700 1 -- 172.21.15.113:0/2209618087 shutdown_connections 2024-04-12T05:10:22.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.957+0000 7f61e8fc8700 1 --2- 172.21.15.113:0/2209618087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61e40f8aa0 0x7f61e40f8f00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:22.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.957+0000 7f61e8fc8700 1 --2- 172.21.15.113:0/2209618087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e40f7800 0x7f61e40f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:22.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.957+0000 7f61e8fc8700 1 -- 172.21.15.113:0/2209618087 >> 172.21.15.113:0/2209618087 conn(0x7f61e40f2f80 msgr2=0x7f61e40f53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:22.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.957+0000 7f61e8fc8700 1 -- 172.21.15.113:0/2209618087 shutdown_connections 2024-04-12T05:10:22.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.957+0000 7f61e8fc8700 1 -- 172.21.15.113:0/2209618087 wait complete. 2024-04-12T05:10:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.958+0000 7f61e8fc8700 1 Processor -- start 2024-04-12T05:10:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.958+0000 7f61e8fc8700 1 -- start start 2024-04-12T05:10:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.958+0000 7f61e8fc8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61e40f7800 0x7f61e4107f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.958+0000 7f61e8fc8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e40f8aa0 0x7f61e4108420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.958+0000 7f61e8fc8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41089d0 con 0x7f61e40f7800 2024-04-12T05:10:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.958+0000 7f61e8fc8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e4108b10 con 0x7f61e40f8aa0 2024-04-12T05:10:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.958+0000 7f61e259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61e40f7800 0x7f61e4107f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.958+0000 7f61e259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61e40f7800 0x7f61e4107f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44308/0 (socket says 172.21.15.113:44308) 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.958+0000 7f61e259c700 1 -- 172.21.15.113:0/475728037 learned_addr learned my addr 172.21.15.113:0/475728037 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.958+0000 7f61e1d9b700 1 --2- 172.21.15.113:0/475728037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e40f8aa0 0x7f61e4108420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.959+0000 7f61e259c700 1 -- 172.21.15.113:0/475728037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e40f8aa0 msgr2=0x7f61e4108420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.959+0000 7f61e259c700 1 --2- 172.21.15.113:0/475728037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e40f8aa0 0x7f61e4108420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.959+0000 7f61e259c700 1 -- 172.21.15.113:0/475728037 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d8005040 con 0x7f61e40f7800 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.959+0000 7f61e259c700 1 --2- 172.21.15.113:0/475728037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61e40f7800 0x7f61e4107f30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f61d4002770 tx=0x7f61d4002a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.959+0000 7f61d37fe700 1 -- 172.21.15.113:0/475728037 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61d4007b40 con 0x7f61e40f7800 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.959+0000 7f61e8fc8700 1 -- 172.21.15.113:0/475728037 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e410d5b0 con 0x7f61e40f7800 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.959+0000 7f61d37fe700 1 -- 172.21.15.113:0/475728037 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61d400f040 con 0x7f61e40f7800 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.959+0000 7f61d37fe700 1 -- 172.21.15.113:0/475728037 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61d4014820 con 0x7f61e40f7800 2024-04-12T05:10:22.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.959+0000 7f61e8fc8700 1 -- 172.21.15.113:0/475728037 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e410daf0 con 0x7f61e40f7800 2024-04-12T05:10:22.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.960+0000 7f61e8fc8700 1 -- 172.21.15.113:0/475728037 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e40fffc0 con 0x7f61e40f7800 2024-04-12T05:10:22.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.960+0000 7f61d37fe700 1 -- 172.21.15.113:0/475728037 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 25) v1 ==== 83960+0+0 (secure 0 0 0) 0x7f61d4007ca0 con 0x7f61e40f7800 2024-04-12T05:10:22.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.961+0000 7f61d37fe700 1 --2- 172.21.15.113:0/475728037 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f61cc066ca0 0x7f61cc069160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:22.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.961+0000 7f61d37fe700 1 -- 172.21.15.113:0/475728037 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f61d4083cb0 con 0x7f61e40f7800 2024-04-12T05:10:22.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.961+0000 7f61e1d9b700 1 --2- 172.21.15.113:0/475728037 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f61cc066ca0 0x7f61cc069160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:10:22.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.962+0000 7f61e1d9b700 1 --2- 172.21.15.113:0/475728037 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f61cc066ca0 0x7f61cc069160 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f61d8004f10 tx=0x7f61d800b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:22.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:22.966+0000 7f61d37fe700 1 -- 172.21.15.113:0/475728037 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f61d4052090 con 0x7f61e40f7800 2024-04-12T05:10:23.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.245+0000 7f61e8fc8700 1 -- 172.21.15.113:0/475728037 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f61e4059120 con 0x7f61cc066ca0 2024-04-12T05:10:23.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.246+0000 7f61d37fe700 1 -- 172.21.15.113:0/475728037 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f61e4059120 con 0x7f61cc066ca0 2024-04-12T05:10:23.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.248+0000 7f61e8fc8700 1 -- 172.21.15.113:0/475728037 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f61cc066ca0 msgr2=0x7f61cc069160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:23.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.248+0000 7f61e8fc8700 1 --2- 172.21.15.113:0/475728037 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f61cc066ca0 0x7f61cc069160 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f61d8004f10 tx=0x7f61d800b040).stop 2024-04-12T05:10:23.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.248+0000 7f61e8fc8700 1 -- 172.21.15.113:0/475728037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61e40f7800 msgr2=0x7f61e4107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:23.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.248+0000 7f61e8fc8700 1 --2- 172.21.15.113:0/475728037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61e40f7800 0x7f61e4107f30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f61d4002770 tx=0x7f61d4002a80).stop 2024-04-12T05:10:23.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.248+0000 7f61e8fc8700 1 -- 172.21.15.113:0/475728037 shutdown_connections 2024-04-12T05:10:23.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.248+0000 7f61e8fc8700 1 --2- 172.21.15.113:0/475728037 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f61cc066ca0 0x7f61cc069160 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:23.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.248+0000 7f61e8fc8700 1 --2- 172.21.15.113:0/475728037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61e40f7800 0x7f61e4107f30 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:23.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.248+0000 7f61e8fc8700 1 --2- 172.21.15.113:0/475728037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e40f8aa0 0x7f61e4108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:23.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.248+0000 7f61e8fc8700 1 -- 172.21.15.113:0/475728037 >> 172.21.15.113:0/475728037 conn(0x7f61e40f2f80 msgr2=0x7f61e40fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:23.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.248+0000 7f61e8fc8700 1 -- 172.21.15.113:0/475728037 shutdown_connections 2024-04-12T05:10:23.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.248+0000 7f61e8fc8700 1 -- 172.21.15.113:0/475728037 wait complete. 2024-04-12T05:10:23.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.357+0000 7f4ad3da5700 1 -- 172.21.15.113:0/4122423123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4acc0f7070 msgr2=0x7f4acc0f74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:23.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.357+0000 7f4ad3da5700 1 --2- 172.21.15.113:0/4122423123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4acc0f7070 0x7f4acc0f74f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f4abc0047f0 tx=0x7f4abc009cb0).stop 2024-04-12T05:10:23.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.357+0000 7f4ad3da5700 1 -- 172.21.15.113:0/4122423123 shutdown_connections 2024-04-12T05:10:23.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.357+0000 7f4ad3da5700 1 --2- 172.21.15.113:0/4122423123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4acc0f7070 0x7f4acc0f74f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:23.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.357+0000 7f4ad3da5700 1 --2- 172.21.15.113:0/4122423123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4acc0f5e70 0x7f4acc0f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:23.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.357+0000 7f4ad3da5700 1 -- 172.21.15.113:0/4122423123 >> 172.21.15.113:0/4122423123 conn(0x7f4acc0f15d0 msgr2=0x7f4acc0f39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:23.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.357+0000 7f4ad3da5700 1 -- 172.21.15.113:0/4122423123 shutdown_connections 2024-04-12T05:10:23.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.357+0000 7f4ad3da5700 1 -- 172.21.15.113:0/4122423123 wait complete. 2024-04-12T05:10:23.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.358+0000 7f4ad3da5700 1 Processor -- start 2024-04-12T05:10:23.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.358+0000 7f4ad3da5700 1 -- start start 2024-04-12T05:10:23.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.358+0000 7f4ad3da5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4acc0f5e70 0x7f4acc106f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:23.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.358+0000 7f4ad3da5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4acc0f7070 0x7f4acc107460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:23.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.358+0000 7f4ad3da5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc107a10 con 0x7f4acc0f5e70 2024-04-12T05:10:23.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.358+0000 7f4ad3da5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc107b50 con 0x7f4acc0f7070 2024-04-12T05:10:23.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.358+0000 7f4ad1b41700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4acc0f5e70 0x7f4acc106f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:23.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.358+0000 7f4ad1b41700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4acc0f5e70 0x7f4acc106f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44324/0 (socket says 172.21.15.113:44324) 2024-04-12T05:10:23.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.358+0000 7f4ad1b41700 1 -- 172.21.15.113:0/3221645488 learned_addr learned my addr 172.21.15.113:0/3221645488 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:10:23.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.359+0000 7f4ad1b41700 1 -- 172.21.15.113:0/3221645488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4acc0f7070 msgr2=0x7f4acc107460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:10:23.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.359+0000 7f4ad1b41700 1 --2- 172.21.15.113:0/3221645488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4acc0f7070 0x7f4acc107460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:10:23.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.359+0000 7f4ad1b41700 1 -- 172.21.15.113:0/3221645488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc005040 con 0x7f4acc0f5e70 2024-04-12T05:10:23.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.359+0000 7f4ad1b41700 1 --2- 172.21.15.113:0/3221645488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4acc0f5e70 0x7f4acc106f70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f4ac800e910 tx=0x7f4ac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:23.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.359+0000 7f4ac2ffd700 1 -- 172.21.15.113:0/3221645488 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ac800eeb0 con 0x7f4acc0f5e70 2024-04-12T05:10:23.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.359+0000 7f4ad3da5700 1 -- 172.21.15.113:0/3221645488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4acc10c5f0 con 0x7f4acc0f5e70 2024-04-12T05:10:23.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.359+0000 7f4ad3da5700 1 -- 172.21.15.113:0/3221645488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4acc10cbb0 con 0x7f4acc0f5e70 2024-04-12T05:10:23.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.360+0000 7f4ac2ffd700 1 -- 172.21.15.113:0/3221645488 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ac800f040 con 0x7f4acc0f5e70 2024-04-12T05:10:23.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.360+0000 7f4ac2ffd700 1 -- 172.21.15.113:0/3221645488 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ac800c810 con 0x7f4acc0f5e70 2024-04-12T05:10:23.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.361+0000 7f4ad3da5700 1 -- 172.21.15.113:0/3221645488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4acc0fe670 con 0x7f4acc0f5e70 2024-04-12T05:10:23.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.362+0000 7f4ac2ffd700 1 -- 172.21.15.113:0/3221645488 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 25) v1 ==== 83960+0+0 (secure 0 0 0) 0x7f4ac80090d0 con 0x7f4acc0f5e70 2024-04-12T05:10:23.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.362+0000 7f4ac2ffd700 1 --2- 172.21.15.113:0/3221645488 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4ab8066ca0 0x7f4ab8069160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:23.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.362+0000 7f4ac2ffd700 1 -- 172.21.15.113:0/3221645488 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f4ac8083bd0 con 0x7f4acc0f5e70 2024-04-12T05:10:23.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.362+0000 7f4ad1340700 1 --2- 172.21.15.113:0/3221645488 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4ab8066ca0 0x7f4ab8069160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:10:23.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.363+0000 7f4ad1340700 1 --2- 172.21.15.113:0/3221645488 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4ab8066ca0 0x7f4ab8069160 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f4abc013560 tx=0x7f4abc009cb0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:23.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.367+0000 7f4ac2ffd700 1 -- 172.21.15.113:0/3221645488 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f4ac8051f90 con 0x7f4acc0f5e70 2024-04-12T05:10:23.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.665+0000 7f4ad3da5700 1 -- 172.21.15.113:0/3221645488 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4acc0fbb50 con 0x7f4ab8066ca0 2024-04-12T05:10:23.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.675+0000 7f4ac2ffd700 1 -- 172.21.15.113:0/3221645488 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f4acc0fbb50 con 0x7f4ab8066ca0 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 starting - - - - 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (9m) 103s ago 9m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (8m) 78s ago 8m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 starting - - - - 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (4m) 103s ago 4m 97.7M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (4m) 103s ago 4m 15.4M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (4m) 78s ago 4m 109M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (3m) 78s ago 3m 12.9M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:9283 running (10m) 103s ago 10m 382M - 16.2.15-1-g8963e9de 727f386679cb 3fab7f291e6b 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283 running (2m) 78s ago 8m 479M - 18.2.2-1187-gd540ebac bb2c5eac26a4 550d8cc53ae9 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (10m) 103s ago 10m 49.3M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (8m) 78s ago 8m 41.4M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:10:23.677 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 starting - - - - 2024-04-12T05:10:23.678 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 starting - - - - 2024-04-12T05:10:23.678 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (8m) 103s ago 8m 109M 1910M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:10:23.678 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (7m) 103s ago 7m 124M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:10:23.678 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (6m) 103s ago 6m 102M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:10:23.678 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (6m) 78s ago 6m 130M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:10:23.678 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (6m) 78s ago 6m 121M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:10:23.678 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (5m) 78s ago 5m 110M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:10:23.678 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (8m) 78s ago 8m 62.7M - 2.33.4 514e6a882f6e b4259537bd55 2024-04-12T05:10:23.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.677+0000 7f4ad3da5700 1 -- 172.21.15.113:0/3221645488 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4ab8066ca0 msgr2=0x7f4ab8069160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:23.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.677+0000 7f4ad3da5700 1 --2- 172.21.15.113:0/3221645488 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4ab8066ca0 0x7f4ab8069160 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f4abc013560 tx=0x7f4abc009cb0).stop 2024-04-12T05:10:23.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.677+0000 7f4ad3da5700 1 -- 172.21.15.113:0/3221645488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4acc0f5e70 msgr2=0x7f4acc106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:23.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.677+0000 7f4ad3da5700 1 --2- 172.21.15.113:0/3221645488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4acc0f5e70 0x7f4acc106f70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f4ac800e910 tx=0x7f4ac800b040).stop 2024-04-12T05:10:23.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.677+0000 7f4ad3da5700 1 -- 172.21.15.113:0/3221645488 shutdown_connections 2024-04-12T05:10:23.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.677+0000 7f4ad3da5700 1 --2- 172.21.15.113:0/3221645488 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f4ab8066ca0 0x7f4ab8069160 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:23.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.677+0000 7f4ad3da5700 1 --2- 172.21.15.113:0/3221645488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4acc0f5e70 0x7f4acc106f70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:23.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.677+0000 7f4ad3da5700 1 --2- 172.21.15.113:0/3221645488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4acc0f7070 0x7f4acc107460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:10:23.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.677+0000 7f4ad3da5700 1 -- 172.21.15.113:0/3221645488 >> 172.21.15.113:0/3221645488 conn(0x7f4acc0f15d0 msgr2=0x7f4acc0fb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:23.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.677+0000 7f4ad3da5700 1 -- 172.21.15.113:0/3221645488 shutdown_connections 2024-04-12T05:10:23.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.677+0000 7f4ad3da5700 1 -- 172.21.15.113:0/3221645488 wait complete. 2024-04-12T05:10:23.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.782+0000 7f12f2e29700 1 -- 172.21.15.113:0/3695242967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ec0fab90 msgr2=0x7f12ec0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:23.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.782+0000 7f12f2e29700 1 --2- 172.21.15.113:0/3695242967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ec0fab90 0x7f12ec0fcfe0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f12d80047f0 tx=0x7f12d8009cb0).stop 2024-04-12T05:10:23.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.782+0000 7f12f2e29700 1 -- 172.21.15.113:0/3695242967 shutdown_connections 2024-04-12T05:10:23.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.782+0000 7f12f2e29700 1 --2- 172.21.15.113:0/3695242967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ec0fab90 0x7f12ec0fcfe0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:23.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.782+0000 7f12f2e29700 1 --2- 172.21.15.113:0/3695242967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12ec061300 0x7f12ec0fa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:23.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.782+0000 7f12f2e29700 1 -- 172.21.15.113:0/3695242967 >> 172.21.15.113:0/3695242967 conn(0x7f12ec0f2070 msgr2=0x7f12ec0f44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:23.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.782+0000 7f12f2e29700 1 -- 172.21.15.113:0/3695242967 shutdown_connections 2024-04-12T05:10:23.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.782+0000 7f12f2e29700 1 -- 172.21.15.113:0/3695242967 wait complete. 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.782+0000 7f12f2e29700 1 Processor -- start 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.782+0000 7f12f2e29700 1 -- start start 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12f2e29700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12ec061300 0x7f12ec108170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12f2e29700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ec0fab90 0x7f12ec108660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12f2e29700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec108c10 con 0x7f12ec0fab90 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12f2e29700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec108d50 con 0x7f12ec061300 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12ebfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ec0fab90 0x7f12ec108660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12ebfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ec0fab90 0x7f12ec108660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44348/0 (socket says 172.21.15.113:44348) 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12ebfff700 1 -- 172.21.15.113:0/2914794577 learned_addr learned my addr 172.21.15.113:0/2914794577 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12f0bc5700 1 --2- 172.21.15.113:0/2914794577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12ec061300 0x7f12ec108170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12ebfff700 1 -- 172.21.15.113:0/2914794577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12ec061300 msgr2=0x7f12ec108170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12ebfff700 1 --2- 172.21.15.113:0/2914794577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12ec061300 0x7f12ec108170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:23.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12ebfff700 1 -- 172.21.15.113:0/2914794577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d8005040 con 0x7f12ec0fab90 2024-04-12T05:10:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12ebfff700 1 --2- 172.21.15.113:0/2914794577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ec0fab90 0x7f12ec108660 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f12d8013620 tx=0x7f12d8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12e9ffb700 1 -- 172.21.15.113:0/2914794577 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12d800e030 con 0x7f12ec0fab90 2024-04-12T05:10:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12e9ffb700 1 -- 172.21.15.113:0/2914794577 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12d8007e90 con 0x7f12ec0fab90 2024-04-12T05:10:23.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12f2e29700 1 -- 172.21.15.113:0/2914794577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ec10d790 con 0x7f12ec0fab90 2024-04-12T05:10:23.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.783+0000 7f12e9ffb700 1 -- 172.21.15.113:0/2914794577 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12d8018070 con 0x7f12ec0fab90 2024-04-12T05:10:23.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.784+0000 7f12f2e29700 1 -- 172.21.15.113:0/2914794577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ec10dcd0 con 0x7f12ec0fab90 2024-04-12T05:10:23.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.785+0000 7f12f2e29700 1 -- 172.21.15.113:0/2914794577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ec05fb50 con 0x7f12ec0fab90 2024-04-12T05:10:23.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.785+0000 7f12e9ffb700 1 -- 172.21.15.113:0/2914794577 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 25) v1 ==== 83960+0+0 (secure 0 0 0) 0x7f12d801d420 con 0x7f12ec0fab90 2024-04-12T05:10:23.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.786+0000 7f12e9ffb700 1 --2- 172.21.15.113:0/2914794577 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f12dc066ca0 0x7f12dc069160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:23.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.786+0000 7f12e9ffb700 1 -- 172.21.15.113:0/2914794577 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f12d8085380 con 0x7f12ec0fab90 2024-04-12T05:10:23.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.786+0000 7f12f0bc5700 1 --2- 172.21.15.113:0/2914794577 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f12dc066ca0 0x7f12dc069160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:10:23.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.786+0000 7f12f0bc5700 1 --2- 172.21.15.113:0/2914794577 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f12dc066ca0 0x7f12dc069160 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f12e0004e60 tx=0x7f12e000b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:23.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:23.791+0000 7f12e9ffb700 1 -- 172.21.15.113:0/2914794577 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f12d80536c0 con 0x7f12ec0fab90 2024-04-12T05:10:24.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.164+0000 7f12f2e29700 1 -- 172.21.15.113:0/2914794577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f12ec1001f0 con 0x7f12ec0fab90 2024-04-12T05:10:24.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.166+0000 7f12e9ffb700 1 -- 172.21.15.113:0/2914794577 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+850 (secure 0 0 0) 0x7f12d80532b0 con 0x7f12ec0fab90 2024-04-12T05:10:24.168 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 13, 2024-04-12T05:10:24.169 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:10:24.170 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:10:24.170 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:10:24.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.170+0000 7f12f2e29700 1 -- 172.21.15.113:0/2914794577 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f12dc066ca0 msgr2=0x7f12dc069160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:24.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.170+0000 7f12f2e29700 1 --2- 172.21.15.113:0/2914794577 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f12dc066ca0 0x7f12dc069160 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f12e0004e60 tx=0x7f12e000b040).stop 2024-04-12T05:10:24.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.170+0000 7f12f2e29700 1 -- 172.21.15.113:0/2914794577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ec0fab90 msgr2=0x7f12ec108660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:24.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.170+0000 7f12f2e29700 1 --2- 172.21.15.113:0/2914794577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ec0fab90 0x7f12ec108660 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f12d8013620 tx=0x7f12d8009cb0).stop 2024-04-12T05:10:24.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.170+0000 7f12f2e29700 1 -- 172.21.15.113:0/2914794577 shutdown_connections 2024-04-12T05:10:24.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.170+0000 7f12f2e29700 1 --2- 172.21.15.113:0/2914794577 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f12dc066ca0 0x7f12dc069160 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:24.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.170+0000 7f12f2e29700 1 --2- 172.21.15.113:0/2914794577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12ec061300 0x7f12ec108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:24.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.170+0000 7f12f2e29700 1 --2- 172.21.15.113:0/2914794577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12ec0fab90 0x7f12ec108660 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:24.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.170+0000 7f12f2e29700 1 -- 172.21.15.113:0/2914794577 >> 172.21.15.113:0/2914794577 conn(0x7f12ec0f2070 msgr2=0x7f12ec0f44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:24.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.170+0000 7f12f2e29700 1 -- 172.21.15.113:0/2914794577 shutdown_connections 2024-04-12T05:10:24.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.170+0000 7f12f2e29700 1 -- 172.21.15.113:0/2914794577 wait complete. 2024-04-12T05:10:24.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.307+0000 7fae09827700 1 -- 172.21.15.113:0/3657069173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae040fa970 msgr2=0x7fae040fcdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:24.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.307+0000 7fae09827700 1 --2- 172.21.15.113:0/3657069173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae040fa970 0x7fae040fcdc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fadf80047f0 tx=0x7fadf8009cb0).stop 2024-04-12T05:10:24.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.307+0000 7fae09827700 1 -- 172.21.15.113:0/3657069173 shutdown_connections 2024-04-12T05:10:24.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.307+0000 7fae09827700 1 --2- 172.21.15.113:0/3657069173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae040fa970 0x7fae040fcdc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:24.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.307+0000 7fae09827700 1 --2- 172.21.15.113:0/3657069173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae04061280 0x7fae040fa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:24.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.307+0000 7fae09827700 1 -- 172.21.15.113:0/3657069173 >> 172.21.15.113:0/3657069173 conn(0x7fae040f1e50 msgr2=0x7fae040f4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:24.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.307+0000 7fae09827700 1 -- 172.21.15.113:0/3657069173 shutdown_connections 2024-04-12T05:10:24.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.307+0000 7fae09827700 1 -- 172.21.15.113:0/3657069173 wait complete. 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.307+0000 7fae09827700 1 Processor -- start 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae09827700 1 -- start start 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae09827700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae04061280 0x7fae04108070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae09827700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae040fa970 0x7fae04108560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae09827700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae04108b10 con 0x7fae040fa970 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae09827700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae04108c50 con 0x7fae04061280 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae02ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae04061280 0x7fae04108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae027fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae040fa970 0x7fae04108560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae02ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae04061280 0x7fae04108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.113:54082/0 (socket says 172.21.15.113:54082) 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae02ffd700 1 -- 172.21.15.113:0/2281225612 learned_addr learned my addr 172.21.15.113:0/2281225612 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae027fc700 1 -- 172.21.15.113:0/2281225612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae04061280 msgr2=0x7fae04108070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae027fc700 1 --2- 172.21.15.113:0/2281225612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae04061280 0x7fae04108070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:24.311 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae027fc700 1 -- 172.21.15.113:0/2281225612 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf8005040 con 0x7fae040fa970 2024-04-12T05:10:24.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fae027fc700 1 --2- 172.21.15.113:0/2281225612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae040fa970 0x7fae04108560 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fadf8013560 tx=0x7fadf8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:24.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fadebfff700 1 -- 172.21.15.113:0/2281225612 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf8018070 con 0x7fae040fa970 2024-04-12T05:10:24.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.308+0000 7fadebfff700 1 -- 172.21.15.113:0/2281225612 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadf8007e90 con 0x7fae040fa970 2024-04-12T05:10:24.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.309+0000 7fae09827700 1 -- 172.21.15.113:0/2281225612 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae040f61d0 con 0x7fae040fa970 2024-04-12T05:10:24.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.309+0000 7fadebfff700 1 -- 172.21.15.113:0/2281225612 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf8012620 con 0x7fae040fa970 2024-04-12T05:10:24.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.309+0000 7fae09827700 1 -- 172.21.15.113:0/2281225612 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae040f6790 con 0x7fae040fa970 2024-04-12T05:10:24.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.309+0000 7fae09827700 1 -- 172.21.15.113:0/2281225612 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae040fffd0 con 0x7fae040fa970 2024-04-12T05:10:24.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.311+0000 7fadebfff700 1 -- 172.21.15.113:0/2281225612 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 25) v1 ==== 83960+0+0 (secure 0 0 0) 0x7fadf801d420 con 0x7fae040fa970 2024-04-12T05:10:24.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.311+0000 7fadebfff700 1 --2- 172.21.15.113:0/2281225612 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fadec066cf0 0x7fadec0691b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:24.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.311+0000 7fadebfff700 1 -- 172.21.15.113:0/2281225612 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7fadf8085280 con 0x7fae040fa970 2024-04-12T05:10:24.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.311+0000 7fae02ffd700 1 --2- 172.21.15.113:0/2281225612 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fadec066cf0 0x7fadec0691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:10:24.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.312+0000 7fae02ffd700 1 --2- 172.21.15.113:0/2281225612 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fadec066cf0 0x7fadec0691b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fadf4004e80 tx=0x7fadf400b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:24.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.315+0000 7fadebfff700 1 -- 172.21.15.113:0/2281225612 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fadf8053560 con 0x7fae040fa970 2024-04-12T05:10:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:22.689597+0000 mgr.smithi192.ocppyo (mgr.24357) 95 : cluster [DBG] pgmap v61: 65 pgs: 65 active+clean; 269 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 56 KiB/s rd, 2.1 MiB/s wr, 609 op/s 2024-04-12T05:10:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:22.832425+0000 mgr.smithi192.ocppyo (mgr.24357) 96 : audit [DBG] from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:23.246829+0000 mgr.smithi192.ocppyo (mgr.24357) 97 : audit [DBG] from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:24.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:22.689597+0000 mgr.smithi192.ocppyo (mgr.24357) 95 : cluster [DBG] pgmap v61: 65 pgs: 65 active+clean; 269 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 56 KiB/s rd, 2.1 MiB/s wr, 609 op/s 2024-04-12T05:10:24.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:22.832425+0000 mgr.smithi192.ocppyo (mgr.24357) 96 : audit [DBG] from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:24.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:23.246829+0000 mgr.smithi192.ocppyo (mgr.24357) 97 : audit [DBG] from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:24.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.593+0000 7fae09827700 1 -- 172.21.15.113:0/2281225612 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fae040f7bf0 con 0x7fadec066cf0 2024-04-12T05:10:24.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.595+0000 7fadebfff700 1 -- 172.21.15.113:0/2281225612 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fae040f7bf0 con 0x7fadec066cf0 2024-04-12T05:10:24.597 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:10:24.597 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:10:24.597 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:10:24.597 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:10:24.597 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [], 2024-04-12T05:10:24.597 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "1/2 daemons upgraded", 2024-04-12T05:10:24.597 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "", 2024-04-12T05:10:24.597 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:10:24.597 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:10:24.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.597+0000 7fae09827700 1 -- 172.21.15.113:0/2281225612 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fadec066cf0 msgr2=0x7fadec0691b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:24.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.597+0000 7fae09827700 1 --2- 172.21.15.113:0/2281225612 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fadec066cf0 0x7fadec0691b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fadf4004e80 tx=0x7fadf400b040).stop 2024-04-12T05:10:24.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.597+0000 7fae09827700 1 -- 172.21.15.113:0/2281225612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae040fa970 msgr2=0x7fae04108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:24.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.597+0000 7fae09827700 1 --2- 172.21.15.113:0/2281225612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae040fa970 0x7fae04108560 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fadf8013560 tx=0x7fadf8009cb0).stop 2024-04-12T05:10:24.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.597+0000 7fae09827700 1 -- 172.21.15.113:0/2281225612 shutdown_connections 2024-04-12T05:10:24.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.597+0000 7fae09827700 1 --2- 172.21.15.113:0/2281225612 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7fadec066cf0 0x7fadec0691b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:24.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.597+0000 7fae09827700 1 --2- 172.21.15.113:0/2281225612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae04061280 0x7fae04108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:24.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.597+0000 7fae09827700 1 --2- 172.21.15.113:0/2281225612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae040fa970 0x7fae04108560 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:24.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.597+0000 7fae09827700 1 -- 172.21.15.113:0/2281225612 >> 172.21.15.113:0/2281225612 conn(0x7fae040f1e50 msgr2=0x7fae040f4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:24.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.597+0000 7fae09827700 1 -- 172.21.15.113:0/2281225612 shutdown_connections 2024-04-12T05:10:24.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:24.597+0000 7fae09827700 1 -- 172.21.15.113:0/2281225612 wait complete. 2024-04-12T05:10:25.334 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-12T05:10:25.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-12T05:10:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:23.666889+0000 mgr.smithi192.ocppyo (mgr.24357) 98 : audit [DBG] from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:24.167269+0000 mon.smithi113 (mon.0) 709 : audit [DBG] from='client.? 172.21.15.113:0/2914794577' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:10:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:23.666889+0000 mgr.smithi192.ocppyo (mgr.24357) 98 : audit [DBG] from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:24.167269+0000 mon.smithi113 (mon.0) 709 : audit [DBG] from='client.? 172.21.15.113:0/2914794577' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:10:25.607 DEBUG:teuthology.parallel:result is None 2024-04-12T05:10:25.608 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-12T05:10:25.660 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-12T05:10:25.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-12T05:10:25.696 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-12T05:10:25.696 DEBUG:teuthology.parallel:result is None 2024-04-12T05:10:26.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:24.595360+0000 mgr.smithi192.ocppyo (mgr.24357) 99 : audit [DBG] from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:26.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:24.689923+0000 mgr.smithi192.ocppyo (mgr.24357) 100 : cluster [DBG] pgmap v62: 65 pgs: 65 active+clean; 269 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 36 KiB/s rd, 1.3 MiB/s wr, 397 op/s 2024-04-12T05:10:26.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:24.595360+0000 mgr.smithi192.ocppyo (mgr.24357) 99 : audit [DBG] from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:26.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:24.689923+0000 mgr.smithi192.ocppyo (mgr.24357) 100 : cluster [DBG] pgmap v62: 65 pgs: 65 active+clean; 269 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 36 KiB/s rd, 1.3 MiB/s wr, 397 op/s 2024-04-12T05:10:28.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:26.690478+0000 mgr.smithi192.ocppyo (mgr.24357) 101 : cluster [DBG] pgmap v63: 65 pgs: 65 active+clean; 269 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 36 KiB/s rd, 1.3 MiB/s wr, 397 op/s 2024-04-12T05:10:28.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:26.690478+0000 mgr.smithi192.ocppyo (mgr.24357) 101 : cluster [DBG] pgmap v63: 65 pgs: 65 active+clean; 269 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 36 KiB/s rd, 1.3 MiB/s wr, 397 op/s 2024-04-12T05:10:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:28.691022+0000 mgr.smithi192.ocppyo (mgr.24357) 102 : cluster [DBG] pgmap v64: 65 pgs: 65 active+clean; 261 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 50 KiB/s rd, 1.7 MiB/s wr, 538 op/s 2024-04-12T05:10:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:29.608107+0000 mon.smithi113 (mon.0) 710 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:29.615310+0000 mon.smithi113 (mon.0) 711 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:29.635727+0000 mon.smithi192 (mon.1) 64 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:10:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:29.639259+0000 mon.smithi192 (mon.1) 65 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:10:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:29.641964+0000 mon.smithi192 (mon.1) 66 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:10:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:29.647722+0000 mon.smithi192 (mon.1) 67 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:10:30.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:28.691022+0000 mgr.smithi192.ocppyo (mgr.24357) 102 : cluster [DBG] pgmap v64: 65 pgs: 65 active+clean; 261 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 50 KiB/s rd, 1.7 MiB/s wr, 538 op/s 2024-04-12T05:10:30.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:29.608107+0000 mon.smithi113 (mon.0) 710 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:30.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:29.615310+0000 mon.smithi113 (mon.0) 711 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:30.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:29.635727+0000 mon.smithi192 (mon.1) 64 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:10:30.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:29.639259+0000 mon.smithi192 (mon.1) 65 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:10:30.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:29.641964+0000 mon.smithi192 (mon.1) 66 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:10:30.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:29.647722+0000 mon.smithi192 (mon.1) 67 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:10:31.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:29.636077+0000 mgr.smithi192.ocppyo (mgr.24357) 103 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:10:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:29.639558+0000 mgr.smithi192.ocppyo (mgr.24357) 104 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:10:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:29.642266+0000 mgr.smithi192.ocppyo (mgr.24357) 105 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:10:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:29.649914+0000 mgr.smithi192.ocppyo (mgr.24357) 106 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi192.ocppyo) 2024-04-12T05:10:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:29.650386+0000 mgr.smithi192.ocppyo (mgr.24357) 107 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi192.ocppyo) 2024-04-12T05:10:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:30.911277+0000 mon.smithi192 (mon.1) 68 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:10:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:29.636077+0000 mgr.smithi192.ocppyo (mgr.24357) 103 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:10:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:29.639558+0000 mgr.smithi192.ocppyo (mgr.24357) 104 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:10:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:29.642266+0000 mgr.smithi192.ocppyo (mgr.24357) 105 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:10:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:29.649914+0000 mgr.smithi192.ocppyo (mgr.24357) 106 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi192.ocppyo) 2024-04-12T05:10:31.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:29.650386+0000 mgr.smithi192.ocppyo (mgr.24357) 107 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.smithi192.ocppyo) 2024-04-12T05:10:31.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:30.911277+0000 mon.smithi192 (mon.1) 68 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:10:32.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:30.691570+0000 mgr.smithi192.ocppyo (mgr.24357) 108 : cluster [DBG] pgmap v65: 65 pgs: 65 active+clean; 261 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 32 KiB/s rd, 1.0 MiB/s wr, 329 op/s 2024-04-12T05:10:32.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:30.691570+0000 mgr.smithi192.ocppyo (mgr.24357) 108 : cluster [DBG] pgmap v65: 65 pgs: 65 active+clean; 261 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 32 KiB/s rd, 1.0 MiB/s wr, 329 op/s 2024-04-12T05:10:34.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:32.692343+0000 mgr.smithi192.ocppyo (mgr.24357) 109 : cluster [DBG] pgmap v66: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 1.0 MiB/s wr, 354 op/s 2024-04-12T05:10:34.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:32.692343+0000 mgr.smithi192.ocppyo (mgr.24357) 109 : cluster [DBG] pgmap v66: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 34 KiB/s rd, 1.0 MiB/s wr, 354 op/s 2024-04-12T05:10:36.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:36 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:34.692881+0000 mgr.smithi192.ocppyo (mgr.24357) 110 : cluster [DBG] pgmap v67: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 391 KiB/s wr, 165 op/s 2024-04-12T05:10:36.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:36 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:34.692881+0000 mgr.smithi192.ocppyo (mgr.24357) 110 : cluster [DBG] pgmap v67: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 391 KiB/s wr, 165 op/s 2024-04-12T05:10:38.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:36.693290+0000 mgr.smithi192.ocppyo (mgr.24357) 111 : cluster [DBG] pgmap v68: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 391 KiB/s wr, 165 op/s 2024-04-12T05:10:38.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:36.693290+0000 mgr.smithi192.ocppyo (mgr.24357) 111 : cluster [DBG] pgmap v68: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 391 KiB/s wr, 165 op/s 2024-04-12T05:10:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:38.693968+0000 mgr.smithi192.ocppyo (mgr.24357) 112 : cluster [DBG] pgmap v69: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 391 KiB/s wr, 165 op/s 2024-04-12T05:10:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:39.548999+0000 mon.smithi113 (mon.0) 712 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:39.550796+0000 mon.smithi192 (mon.1) 69 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.ctsazg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:10:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:39.551349+0000 mon.smithi113 (mon.0) 713 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.ctsazg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:10:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:39.552442+0000 mon.smithi192 (mon.1) 70 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:10:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:39.553278+0000 mon.smithi192 (mon.1) 71 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:10:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:38.693968+0000 mgr.smithi192.ocppyo (mgr.24357) 112 : cluster [DBG] pgmap v69: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 391 KiB/s wr, 165 op/s 2024-04-12T05:10:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:39.548999+0000 mon.smithi113 (mon.0) 712 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:39.550796+0000 mon.smithi192 (mon.1) 69 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.ctsazg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:10:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:39.551349+0000 mon.smithi113 (mon.0) 713 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.ctsazg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:10:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:39.552442+0000 mon.smithi192 (mon.1) 70 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:10:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:39.553278+0000 mon.smithi192 (mon.1) 71 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:10:41.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:39.543417+0000 mgr.smithi192.ocppyo (mgr.24357) 113 : cephadm [INF] Upgrade: Updating mgr.smithi113.ctsazg 2024-04-12T05:10:41.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:39.554131+0000 mgr.smithi192.ocppyo (mgr.24357) 114 : cephadm [INF] Deploying daemon mgr.smithi113.ctsazg on smithi113 2024-04-12T05:10:41.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:39.543417+0000 mgr.smithi192.ocppyo (mgr.24357) 113 : cephadm [INF] Upgrade: Updating mgr.smithi113.ctsazg 2024-04-12T05:10:41.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:39.554131+0000 mgr.smithi192.ocppyo (mgr.24357) 114 : cephadm [INF] Deploying daemon mgr.smithi113.ctsazg on smithi113 2024-04-12T05:10:42.477 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:42 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:40.694459+0000 mgr.smithi192.ocppyo (mgr.24357) 115 : cluster [DBG] pgmap v70: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 18 KiB/s wr, 24 op/s 2024-04-12T05:10:42.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:42 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:40.694459+0000 mgr.smithi192.ocppyo (mgr.24357) 115 : cluster [DBG] pgmap v70: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 18 KiB/s wr, 24 op/s 2024-04-12T05:10:44.478 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:42.695130+0000 mgr.smithi192.ocppyo (mgr.24357) 116 : cluster [DBG] pgmap v71: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 18 KiB/s wr, 24 op/s 2024-04-12T05:10:44.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:42.695130+0000 mgr.smithi192.ocppyo (mgr.24357) 116 : cluster [DBG] pgmap v71: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 18 KiB/s wr, 24 op/s 2024-04-12T05:10:46.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:44.695701+0000 mgr.smithi192.ocppyo (mgr.24357) 117 : cluster [DBG] pgmap v72: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-12T05:10:46.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:45.690073+0000 mon.smithi113 (mon.0) 714 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:46.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:45.698037+0000 mon.smithi113 (mon.0) 715 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:46.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:45.735282+0000 mon.smithi192 (mon.1) 72 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:10:46.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:45.911455+0000 mon.smithi192 (mon.1) 73 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:10:46.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:46.055136+0000 mon.smithi113 (mon.0) 716 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:46.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:46.060425+0000 mon.smithi113 (mon.0) 717 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:46.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:44.695701+0000 mgr.smithi192.ocppyo (mgr.24357) 117 : cluster [DBG] pgmap v72: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-12T05:10:46.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:45.690073+0000 mon.smithi113 (mon.0) 714 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:46.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:45.698037+0000 mon.smithi113 (mon.0) 715 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:46.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:45.735282+0000 mon.smithi192 (mon.1) 72 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:10:46.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:45.911455+0000 mon.smithi192 (mon.1) 73 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:10:46.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:46.055136+0000 mon.smithi113 (mon.0) 716 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:46.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:46.060425+0000 mon.smithi113 (mon.0) 717 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:48.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:46.696122+0000 mgr.smithi192.ocppyo (mgr.24357) 118 : cluster [DBG] pgmap v73: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-12T05:10:48.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:46.696122+0000 mgr.smithi192.ocppyo (mgr.24357) 118 : cluster [DBG] pgmap v73: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-12T05:10:50.340 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:48.696649+0000 mgr.smithi192.ocppyo (mgr.24357) 119 : cluster [DBG] pgmap v74: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-12T05:10:50.341 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:49.613993+0000 mon.smithi113 (mon.0) 718 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:50.341 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:49.619836+0000 mon.smithi113 (mon.0) 719 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:50.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:48.696649+0000 mgr.smithi192.ocppyo (mgr.24357) 119 : cluster [DBG] pgmap v74: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-04-12T05:10:50.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:49.613993+0000 mon.smithi113 (mon.0) 718 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:50.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:49.619836+0000 mon.smithi113 (mon.0) 719 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:50.697141+0000 mgr.smithi192.ocppyo (mgr.24357) 120 : cluster [DBG] pgmap v75: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:10:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:51.443117+0000 mon.smithi113 (mon.0) 720 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:51.449923+0000 mon.smithi113 (mon.0) 721 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:52.140791+0000 mon.smithi113 (mon.0) 722 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:52 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:52.146881+0000 mon.smithi113 (mon.0) 723 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:52.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:50.697141+0000 mgr.smithi192.ocppyo (mgr.24357) 120 : cluster [DBG] pgmap v75: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:10:52.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:51.443117+0000 mon.smithi113 (mon.0) 720 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:52.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:51.449923+0000 mon.smithi113 (mon.0) 721 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:52.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:52.140791+0000 mon.smithi113 (mon.0) 722 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:52.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:52.146881+0000 mon.smithi113 (mon.0) 723 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:53.395 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:52.696781+0000 mon.smithi113 (mon.0) 724 : cluster [DBG] Standby manager daemon smithi113.ctsazg restarted 2024-04-12T05:10:53.395 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:52.696919+0000 mon.smithi113 (mon.0) 725 : cluster [DBG] Standby manager daemon smithi113.ctsazg started 2024-04-12T05:10:53.395 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:52.700720+0000 mon.smithi113 (mon.0) 726 : audit [DBG] from='mgr.? 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.ctsazg/crt"}]: dispatch 2024-04-12T05:10:53.395 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:52.701671+0000 mon.smithi113 (mon.0) 727 : audit [DBG] from='mgr.? 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:10:53.395 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:52.704188+0000 mon.smithi113 (mon.0) 728 : audit [DBG] from='mgr.? 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.ctsazg/key"}]: dispatch 2024-04-12T05:10:53.395 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:52.704600+0000 mon.smithi113 (mon.0) 729 : audit [DBG] from='mgr.? 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:10:53.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:52.696781+0000 mon.smithi113 (mon.0) 724 : cluster [DBG] Standby manager daemon smithi113.ctsazg restarted 2024-04-12T05:10:53.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:52.696919+0000 mon.smithi113 (mon.0) 725 : cluster [DBG] Standby manager daemon smithi113.ctsazg started 2024-04-12T05:10:53.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:52.700720+0000 mon.smithi113 (mon.0) 726 : audit [DBG] from='mgr.? 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.ctsazg/crt"}]: dispatch 2024-04-12T05:10:53.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:52.701671+0000 mon.smithi113 (mon.0) 727 : audit [DBG] from='mgr.? 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:10:53.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:52.704188+0000 mon.smithi113 (mon.0) 728 : audit [DBG] from='mgr.? 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi113.ctsazg/key"}]: dispatch 2024-04-12T05:10:53.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:52.704600+0000 mon.smithi113 (mon.0) 729 : audit [DBG] from='mgr.? 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:10:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:52.697704+0000 mgr.smithi192.ocppyo (mgr.24357) 121 : cluster [DBG] pgmap v76: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:10:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:54 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:53.239408+0000 mon.smithi113 (mon.0) 730 : cluster [DBG] mgrmap e26: smithi192.ocppyo(active, since 2m), standbys: smithi113.ctsazg 2024-04-12T05:10:54.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:52.697704+0000 mgr.smithi192.ocppyo (mgr.24357) 121 : cluster [DBG] pgmap v76: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:10:54.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:53.239408+0000 mon.smithi113 (mon.0) 730 : cluster [DBG] mgrmap e26: smithi192.ocppyo(active, since 2m), standbys: smithi113.ctsazg 2024-04-12T05:10:54.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 -- 172.21.15.113:0/605229681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540f8cb0 msgr2=0x7f91540f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 --2- 172.21.15.113:0/605229681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540f8cb0 0x7f91540f9110 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f91400047f0 tx=0x7f9140009cb0).stop 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 -- 172.21.15.113:0/605229681 shutdown_connections 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 --2- 172.21.15.113:0/605229681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540f8cb0 0x7f91540f9110 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 --2- 172.21.15.113:0/605229681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540f7a10 0x7f91540f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 -- 172.21.15.113:0/605229681 >> 172.21.15.113:0/605229681 conn(0x7f91540f3150 msgr2=0x7f91540f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 -- 172.21.15.113:0/605229681 shutdown_connections 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 -- 172.21.15.113:0/605229681 wait complete. 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 Processor -- start 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 -- start start 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540f7a10 0x7f9154108210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540f8cb0 0x7f9154108700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9154108d40 con 0x7f91540f7a10 2024-04-12T05:10:54.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.701+0000 7f915ae5a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9154108e80 con 0x7f91540f8cb0 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9158bf6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540f7a10 0x7f9154108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9158bf6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540f7a10 0x7f9154108210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38658/0 (socket says 172.21.15.113:38658) 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9158bf6700 1 -- 172.21.15.113:0/2868985694 learned_addr learned my addr 172.21.15.113:0/2868985694 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9153fff700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540f8cb0 0x7f9154108700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9158bf6700 1 -- 172.21.15.113:0/2868985694 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540f8cb0 msgr2=0x7f9154108700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9158bf6700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540f8cb0 0x7f9154108700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9158bf6700 1 -- 172.21.15.113:0/2868985694 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9140005040 con 0x7f91540f7a10 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9153fff700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540f8cb0 0x7f9154108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9158bf6700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540f7a10 0x7f9154108210 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f914800b480 tx=0x7f91480090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9151ffb700 1 -- 172.21.15.113:0/2868985694 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9148002500 con 0x7f91540f7a10 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9151ffb700 1 -- 172.21.15.113:0/2868985694 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9148002b10 con 0x7f91540f7a10 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f915ae5a700 1 -- 172.21.15.113:0/2868985694 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915410d920 con 0x7f91540f7a10 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f9151ffb700 1 -- 172.21.15.113:0/2868985694 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9148012d60 con 0x7f91540f7a10 2024-04-12T05:10:54.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.702+0000 7f915ae5a700 1 -- 172.21.15.113:0/2868985694 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915410de60 con 0x7f91540f7a10 2024-04-12T05:10:54.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.703+0000 7f915ae5a700 1 -- 172.21.15.113:0/2868985694 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915405fb50 con 0x7f91540f7a10 2024-04-12T05:10:54.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.704+0000 7f9151ffb700 1 -- 172.21.15.113:0/2868985694 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f9148002660 con 0x7f91540f7a10 2024-04-12T05:10:54.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.704+0000 7f9151ffb700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f914406c6a0 0x7f914406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:54.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.704+0000 7f9151ffb700 1 -- 172.21.15.113:0/2868985694 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f914808b6c0 con 0x7f91540f7a10 2024-04-12T05:10:54.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.705+0000 7f9153fff700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f914406c6a0 0x7f914406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:10:54.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.705+0000 7f9153fff700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f914406c6a0 0x7f914406eb60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9140004f10 tx=0x7f914000b040).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:54.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.709+0000 7f9151ffb700 1 -- 172.21.15.113:0/2868985694 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f91480598c0 con 0x7f91540f7a10 2024-04-12T05:10:54.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.940+0000 7f915ae5a700 1 -- 172.21.15.113:0/2868985694 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f91541022d0 con 0x7f914406c6a0 2024-04-12T05:10:54.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.942+0000 7f9151ffb700 1 -- 172.21.15.113:0/2868985694 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f9140019040 con 0x7f914406c6a0 2024-04-12T05:10:54.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.944+0000 7f915ae5a700 1 -- 172.21.15.113:0/2868985694 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f914406c6a0 msgr2=0x7f914406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:54.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.944+0000 7f915ae5a700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f914406c6a0 0x7f914406eb60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9140004f10 tx=0x7f914000b040).stop 2024-04-12T05:10:54.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.944+0000 7f915ae5a700 1 -- 172.21.15.113:0/2868985694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540f7a10 msgr2=0x7f9154108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:54.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.944+0000 7f915ae5a700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540f7a10 0x7f9154108210 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f914800b480 tx=0x7f91480090f0).stop 2024-04-12T05:10:54.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.944+0000 7f915ae5a700 1 -- 172.21.15.113:0/2868985694 shutdown_connections 2024-04-12T05:10:54.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.944+0000 7f915ae5a700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f914406c6a0 0x7f914406eb60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:54.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.944+0000 7f915ae5a700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540f7a10 0x7f9154108210 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:54.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.944+0000 7f915ae5a700 1 --2- 172.21.15.113:0/2868985694 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540f8cb0 0x7f9154108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:54.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.944+0000 7f915ae5a700 1 -- 172.21.15.113:0/2868985694 >> 172.21.15.113:0/2868985694 conn(0x7f91540f3150 msgr2=0x7f91540fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:54.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.944+0000 7f915ae5a700 1 -- 172.21.15.113:0/2868985694 shutdown_connections 2024-04-12T05:10:54.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:54.944+0000 7f915ae5a700 1 -- 172.21.15.113:0/2868985694 wait complete. 2024-04-12T05:10:54.956 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:10:55.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 -- 172.21.15.113:0/715102305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75100f7990 msgr2=0x7f75100f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 --2- 172.21.15.113:0/715102305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75100f7990 0x7f75100f7df0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f750c0047f0 tx=0x7f750c009cb0).stop 2024-04-12T05:10:55.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 -- 172.21.15.113:0/715102305 shutdown_connections 2024-04-12T05:10:55.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 --2- 172.21.15.113:0/715102305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75100f7990 0x7f75100f7df0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 --2- 172.21.15.113:0/715102305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75100f66f0 0x7f75100f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 -- 172.21.15.113:0/715102305 >> 172.21.15.113:0/715102305 conn(0x7f75100f1e50 msgr2=0x7f75100f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:55.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 -- 172.21.15.113:0/715102305 shutdown_connections 2024-04-12T05:10:55.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 -- 172.21.15.113:0/715102305 wait complete. 2024-04-12T05:10:55.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 Processor -- start 2024-04-12T05:10:55.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 -- start start 2024-04-12T05:10:55.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75100f66f0 0x7f7510107f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:55.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75100f7990 0x7f7510108410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:55.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75101089c0 con 0x7f75100f7990 2024-04-12T05:10:55.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.043+0000 7f7518322700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7510108b00 con 0x7f75100f66f0 2024-04-12T05:10:55.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f75158bd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75100f7990 0x7f7510108410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f75158bd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75100f7990 0x7f7510108410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38672/0 (socket says 172.21.15.113:38672) 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f75158bd700 1 -- 172.21.15.113:0/3756092216 learned_addr learned my addr 172.21.15.113:0/3756092216 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f75158bd700 1 -- 172.21.15.113:0/3756092216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75100f66f0 msgr2=0x7f7510107f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f75158bd700 1 --2- 172.21.15.113:0/3756092216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75100f66f0 0x7f7510107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f75158bd700 1 -- 172.21.15.113:0/3756092216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c005040 con 0x7f75100f7990 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f75158bd700 1 --2- 172.21.15.113:0/3756092216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75100f7990 0x7f7510108410 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f750c00b120 tx=0x7f750c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f75077fe700 1 -- 172.21.15.113:0/3756092216 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f750c00e030 con 0x7f75100f7990 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f75077fe700 1 -- 172.21.15.113:0/3756092216 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f750c007e90 con 0x7f75100f7990 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f7518322700 1 -- 172.21.15.113:0/3756092216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751010d540 con 0x7f75100f7990 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f75077fe700 1 -- 172.21.15.113:0/3756092216 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f750c018070 con 0x7f75100f7990 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.044+0000 7f7518322700 1 -- 172.21.15.113:0/3756092216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751010db00 con 0x7f75100f7990 2024-04-12T05:10:55.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.045+0000 7f7518322700 1 -- 172.21.15.113:0/3756092216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75100feeb0 con 0x7f75100f7990 2024-04-12T05:10:55.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.045+0000 7f75077fe700 1 -- 172.21.15.113:0/3756092216 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f750c01d960 con 0x7f75100f7990 2024-04-12T05:10:55.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.046+0000 7f75077fe700 1 --2- 172.21.15.113:0/3756092216 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f74fc06c6a0 0x7f74fc06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:55.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.046+0000 7f75077fe700 1 -- 172.21.15.113:0/3756092216 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f750c08cbd0 con 0x7f75100f7990 2024-04-12T05:10:55.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.046+0000 7f75160be700 1 --2- 172.21.15.113:0/3756092216 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f74fc06c6a0 0x7f74fc06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:10:55.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.047+0000 7f75160be700 1 --2- 172.21.15.113:0/3756092216 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f74fc06c6a0 0x7f74fc06eb60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f75100f77f0 tx=0x7f7500007be0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:55.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.051+0000 7f75077fe700 1 -- 172.21.15.113:0/3756092216 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f750c05add0 con 0x7f75100f7990 2024-04-12T05:10:55.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.263+0000 7f7518322700 1 -- 172.21.15.113:0/3756092216 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f751010e190 con 0x7f74fc06c6a0 2024-04-12T05:10:55.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.264+0000 7f75077fe700 1 -- 172.21.15.113:0/3756092216 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f751010e190 con 0x7f74fc06c6a0 2024-04-12T05:10:55.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.266+0000 7f7518322700 1 -- 172.21.15.113:0/3756092216 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f74fc06c6a0 msgr2=0x7f74fc06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.266+0000 7f7518322700 1 --2- 172.21.15.113:0/3756092216 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f74fc06c6a0 0x7f74fc06eb60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f75100f77f0 tx=0x7f7500007be0).stop 2024-04-12T05:10:55.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.266+0000 7f7518322700 1 -- 172.21.15.113:0/3756092216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75100f7990 msgr2=0x7f7510108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.266+0000 7f7518322700 1 --2- 172.21.15.113:0/3756092216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75100f7990 0x7f7510108410 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f750c00b120 tx=0x7f750c009cb0).stop 2024-04-12T05:10:55.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.266+0000 7f7518322700 1 -- 172.21.15.113:0/3756092216 shutdown_connections 2024-04-12T05:10:55.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.266+0000 7f7518322700 1 --2- 172.21.15.113:0/3756092216 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f74fc06c6a0 0x7f74fc06eb60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.266+0000 7f7518322700 1 --2- 172.21.15.113:0/3756092216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75100f66f0 0x7f7510107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:10:55.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.266+0000 7f7518322700 1 --2- 172.21.15.113:0/3756092216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75100f7990 0x7f7510108410 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.266+0000 7f7518322700 1 -- 172.21.15.113:0/3756092216 >> 172.21.15.113:0/3756092216 conn(0x7f75100f1e50 msgr2=0x7f7510060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:55.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.267+0000 7f7518322700 1 -- 172.21.15.113:0/3756092216 shutdown_connections 2024-04-12T05:10:55.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.267+0000 7f7518322700 1 -- 172.21.15.113:0/3756092216 wait complete. 2024-04-12T05:10:55.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.366+0000 7f504b3aa700 1 -- 172.21.15.113:0/4023704517 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50440f8aa0 msgr2=0x7f50440f8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.366+0000 7f504b3aa700 1 --2- 172.21.15.113:0/4023704517 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50440f8aa0 0x7f50440f8f00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f50400047f0 tx=0x7f5040009cb0).stop 2024-04-12T05:10:55.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 -- 172.21.15.113:0/4023704517 shutdown_connections 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 --2- 172.21.15.113:0/4023704517 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50440f8aa0 0x7f50440f8f00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 --2- 172.21.15.113:0/4023704517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50440f7800 0x7f50440f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 -- 172.21.15.113:0/4023704517 >> 172.21.15.113:0/4023704517 conn(0x7f50440f2f80 msgr2=0x7f50440f53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 -- 172.21.15.113:0/4023704517 shutdown_connections 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 -- 172.21.15.113:0/4023704517 wait complete. 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 Processor -- start 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 -- start start 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50440f7800 0x7f5044107f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50440f8aa0 0x7f5044108450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5044108a00 con 0x7f50440f8aa0 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.367+0000 7f504b3aa700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5044108b40 con 0x7f50440f7800 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f5048945700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50440f8aa0 0x7f5044108450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:55.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f5048945700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50440f8aa0 0x7f5044108450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38684/0 (socket says 172.21.15.113:38684) 2024-04-12T05:10:55.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f5048945700 1 -- 172.21.15.113:0/3546173516 learned_addr learned my addr 172.21.15.113:0/3546173516 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:10:55.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f5049146700 1 --2- 172.21.15.113:0/3546173516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50440f7800 0x7f5044107f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:55.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f5048945700 1 -- 172.21.15.113:0/3546173516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50440f7800 msgr2=0x7f5044107f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f5048945700 1 --2- 172.21.15.113:0/3546173516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50440f7800 0x7f5044107f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f5048945700 1 -- 172.21.15.113:0/3546173516 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5040005040 con 0x7f50440f8aa0 2024-04-12T05:10:55.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f5048945700 1 --2- 172.21.15.113:0/3546173516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50440f8aa0 0x7f5044108450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f5040013620 tx=0x7f5040009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:55.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f503a7fc700 1 -- 172.21.15.113:0/3546173516 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504000e070 con 0x7f50440f8aa0 2024-04-12T05:10:55.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f503a7fc700 1 -- 172.21.15.113:0/3546173516 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5040007e90 con 0x7f50440f8aa0 2024-04-12T05:10:55.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f503a7fc700 1 -- 172.21.15.113:0/3546173516 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5040018070 con 0x7f50440f8aa0 2024-04-12T05:10:55.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f504b3aa700 1 -- 172.21.15.113:0/3546173516 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504410d580 con 0x7f50440f8aa0 2024-04-12T05:10:55.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.368+0000 7f504b3aa700 1 -- 172.21.15.113:0/3546173516 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504410db40 con 0x7f50440f8aa0 2024-04-12T05:10:55.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.369+0000 7f504b3aa700 1 -- 172.21.15.113:0/3546173516 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50440fffe0 con 0x7f50440f8aa0 2024-04-12T05:10:55.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.369+0000 7f503a7fc700 1 -- 172.21.15.113:0/3546173516 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f504001d420 con 0x7f50440f8aa0 2024-04-12T05:10:55.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.370+0000 7f503a7fc700 1 --2- 172.21.15.113:0/3546173516 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f503006c6a0 0x7f503006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:55.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.370+0000 7f503a7fc700 1 -- 172.21.15.113:0/3546173516 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f504008c4a0 con 0x7f50440f8aa0 2024-04-12T05:10:55.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.370+0000 7f5049146700 1 --2- 172.21.15.113:0/3546173516 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f503006c6a0 0x7f503006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:10:55.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.371+0000 7f5049146700 1 --2- 172.21.15.113:0/3546173516 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f503006c6a0 0x7f503006eb60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f50440f8900 tx=0x7f5034009d60).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:55.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.375+0000 7f503a7fc700 1 -- 172.21.15.113:0/3546173516 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f504005a6a0 con 0x7f50440f8aa0 2024-04-12T05:10:55.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.571+0000 7f504b3aa700 1 -- 172.21.15.113:0/3546173516 --> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5044059120 con 0x7f503006c6a0 2024-04-12T05:10:55.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.582+0000 7f503a7fc700 1 -- 172.21.15.113:0/3546173516 <== mgr.24357 v2:172.21.15.192:6828/2257000201 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5044059120 con 0x7f503006c6a0 2024-04-12T05:10:55.584 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:10:55.584 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (82s) 4s ago 9m 13.9M - 0.23.0 ba2b418f427c b2262b7eedd3 2024-04-12T05:10:55.584 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (9m) 4s ago 9m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:10:55.584 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (9m) 5s ago 9m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:10:55.584 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (55s) 4s ago 9m 43.2M - 8.3.5 dad864ee21e9 d33ad1514a4c 2024-04-12T05:10:55.584 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (4m) 4s ago 4m 121M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:10:55.584 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (4m) 4s ago 4m 16.3M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:10:55.584 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (4m) 5s ago 4m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (4m) 5s ago 4m 13.8M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (10s) 4s ago 11m 74.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283 running (2m) 5s ago 9m 497M - 18.2.2-1187-gd540ebac bb2c5eac26a4 550d8cc53ae9 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (11m) 4s ago 11m 51.6M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (9m) 5s ago 9m 43.6M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (53s) 4s ago 9m 11.6M - 1.3.1 1dbe0e931976 b61fc2c12125 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (51s) 5s ago 9m 20.4M - 1.3.1 1dbe0e931976 2b882fe39886 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (8m) 4s ago 8m 226M 1910M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (7m) 4s ago 7m 240M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (7m) 4s ago 7m 202M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (7m) 5s ago 7m 265M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (6m) 5s ago 6m 232M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (6m) 5s ago 6m 225M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:10:55.585 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (28s) 5s ago 8m 50.0M - 2.33.4 514e6a882f6e e8cb4d261f11 2024-04-12T05:10:55.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.584+0000 7f504b3aa700 1 -- 172.21.15.113:0/3546173516 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f503006c6a0 msgr2=0x7f503006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.584+0000 7f504b3aa700 1 --2- 172.21.15.113:0/3546173516 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f503006c6a0 0x7f503006eb60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f50440f8900 tx=0x7f5034009d60).stop 2024-04-12T05:10:55.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.584+0000 7f504b3aa700 1 -- 172.21.15.113:0/3546173516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50440f8aa0 msgr2=0x7f5044108450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.584+0000 7f504b3aa700 1 --2- 172.21.15.113:0/3546173516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50440f8aa0 0x7f5044108450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f5040013620 tx=0x7f5040009cb0).stop 2024-04-12T05:10:55.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.584+0000 7f504b3aa700 1 -- 172.21.15.113:0/3546173516 shutdown_connections 2024-04-12T05:10:55.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.584+0000 7f504b3aa700 1 --2- 172.21.15.113:0/3546173516 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f503006c6a0 0x7f503006eb60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.584+0000 7f504b3aa700 1 --2- 172.21.15.113:0/3546173516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50440f7800 0x7f5044107f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.584+0000 7f504b3aa700 1 --2- 172.21.15.113:0/3546173516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50440f8aa0 0x7f5044108450 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.584+0000 7f504b3aa700 1 -- 172.21.15.113:0/3546173516 >> 172.21.15.113:0/3546173516 conn(0x7f50440f2f80 msgr2=0x7f50440fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:55.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.584+0000 7f504b3aa700 1 -- 172.21.15.113:0/3546173516 shutdown_connections 2024-04-12T05:10:55.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.584+0000 7f504b3aa700 1 -- 172.21.15.113:0/3546173516 wait complete. 2024-04-12T05:10:55.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.680+0000 7f1d93d96700 1 -- 172.21.15.113:0/3229380833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 msgr2=0x7f1d8c0f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.680+0000 7f1d93d96700 1 --2- 172.21.15.113:0/3229380833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c0f9110 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f1d7c0047f0 tx=0x7f1d7c009cb0).stop 2024-04-12T05:10:55.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.680+0000 7f1d93d96700 1 -- 172.21.15.113:0/3229380833 shutdown_connections 2024-04-12T05:10:55.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.680+0000 7f1d93d96700 1 --2- 172.21.15.113:0/3229380833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c0f9110 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.680+0000 7f1d93d96700 1 --2- 172.21.15.113:0/3229380833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c0f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.680+0000 7f1d93d96700 1 -- 172.21.15.113:0/3229380833 >> 172.21.15.113:0/3229380833 conn(0x7f1d8c0f3150 msgr2=0x7f1d8c0f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:55.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.680+0000 7f1d93d96700 1 -- 172.21.15.113:0/3229380833 shutdown_connections 2024-04-12T05:10:55.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.680+0000 7f1d93d96700 1 -- 172.21.15.113:0/3229380833 wait complete. 2024-04-12T05:10:55.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.681+0000 7f1d93d96700 1 Processor -- start 2024-04-12T05:10:55.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.681+0000 7f1d93d96700 1 -- start start 2024-04-12T05:10:55.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.681+0000 7f1d93d96700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c108190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.681+0000 7f1d93d96700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c108680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.681+0000 7f1d93d96700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c108c30 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.681+0000 7f1d93d96700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c108d70 con 0x7f1d8c0f8cb0 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.681+0000 7f1d91b32700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c108190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.681+0000 7f1d91b32700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c108190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38690/0 (socket says 172.21.15.113:38690) 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.681+0000 7f1d91b32700 1 -- 172.21.15.113:0/1068955629 learned_addr learned my addr 172.21.15.113:0/1068955629 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.682+0000 7f1d91b32700 1 -- 172.21.15.113:0/1068955629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c0f8cb0 msgr2=0x7f1d8c108680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.682+0000 7f1d91b32700 1 --2- 172.21.15.113:0/1068955629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c108680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.682+0000 7f1d91b32700 1 -- 172.21.15.113:0/1068955629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c005040 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.682+0000 7f1d91b32700 1 --2- 172.21.15.113:0/1068955629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c108190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f1d8800c790 tx=0x7f1d88007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.682+0000 7f1d82ffd700 1 -- 172.21.15.113:0/1068955629 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d880106f0 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.682+0000 7f1d93d96700 1 -- 172.21.15.113:0/1068955629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8c10d810 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.682+0000 7f1d82ffd700 1 -- 172.21.15.113:0/1068955629 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d88010d00 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.682+0000 7f1d82ffd700 1 -- 172.21.15.113:0/1068955629 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d88012e30 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.682+0000 7f1d93d96700 1 -- 172.21.15.113:0/1068955629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8c10ddd0 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.683+0000 7f1d93d96700 1 -- 172.21.15.113:0/1068955629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d8c05fb50 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.683+0000 7f1d82ffd700 1 -- 172.21.15.113:0/1068955629 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f1d88012400 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.684+0000 7f1d82ffd700 1 --2- 172.21.15.113:0/1068955629 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f1d7806c630 0x7f1d7806eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:55.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.684+0000 7f1d82ffd700 1 -- 172.21.15.113:0/1068955629 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5531+0+0 (secure 0 0 0) 0x7f1d8808b650 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.684+0000 7f1d91331700 1 --2- 172.21.15.113:0/1068955629 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f1d7806c630 0x7f1d7806eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:10:55.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.685+0000 7f1d91331700 1 --2- 172.21.15.113:0/1068955629 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f1d7806c630 0x7f1d7806eaf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1d7c013620 tx=0x7f1d7c009cb0).ready entity=mgr.24357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:55.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.689+0000 7f1d82ffd700 1 -- 172.21.15.113:0/1068955629 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f1d88059900 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.938+0000 7f1d82ffd700 1 -- 172.21.15.113:0/1068955629 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 27) v1 ==== 45177+0+0 (secure 0 0 0) 0x7f1d88056240 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.938+0000 7f1d82ffd700 1 -- 172.21.15.113:0/1068955629 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f1d7806c630 msgr2=0x7f1d7806eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.938+0000 7f1d82ffd700 1 --2- 172.21.15.113:0/1068955629 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f1d7806c630 0x7f1d7806eaf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1d7c013620 tx=0x7f1d7c009cb0).stop 2024-04-12T05:10:55.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.976+0000 7f1d93d96700 1 -- 172.21.15.113:0/1068955629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1d8c0598f0 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.976+0000 7f1d82ffd700 1 -- 172.21.15.113:0/1068955629 <== mon.0 v2:172.21.15.113:3300/0 8 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f1d88059900 con 0x7f1d8c0f7a10 2024-04-12T05:10:55.978 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:10:55.978 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:10:55.978 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:10:55.978 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:10:55.978 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:10:55.978 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 12, 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:10:55.979 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:10:55.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.978+0000 7f1d93d96700 1 -- 172.21.15.113:0/1068955629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f7a10 msgr2=0x7f1d8c108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:55.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.978+0000 7f1d93d96700 1 --2- 172.21.15.113:0/1068955629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c108190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f1d8800c790 tx=0x7f1d88007f90).stop 2024-04-12T05:10:55.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.978+0000 7f1d93d96700 1 -- 172.21.15.113:0/1068955629 shutdown_connections 2024-04-12T05:10:55.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.978+0000 7f1d93d96700 1 --2- 172.21.15.113:0/1068955629 >> [v2:172.21.15.192:6828/2257000201,v1:172.21.15.192:6829/2257000201] conn(0x7f1d7806c630 0x7f1d7806eaf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.978+0000 7f1d93d96700 1 --2- 172.21.15.113:0/1068955629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d8c0f7a10 0x7f1d8c108190 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:55.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.978+0000 7f1d93d96700 1 --2- 172.21.15.113:0/1068955629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c0f8cb0 0x7f1d8c108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:10:55.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.978+0000 7f1d93d96700 1 -- 172.21.15.113:0/1068955629 >> 172.21.15.113:0/1068955629 conn(0x7f1d8c0f3150 msgr2=0x7f1d8c0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:55.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.978+0000 7f1d93d96700 1 -- 172.21.15.113:0/1068955629 shutdown_connections 2024-04-12T05:10:55.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:55.978+0000 7f1d93d96700 1 -- 172.21.15.113:0/1068955629 wait complete. 2024-04-12T05:10:56.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.098+0000 7f36633ef700 1 -- 172.21.15.113:0/616636595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f365c06cbd0 msgr2=0x7f365c06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:56.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.098+0000 7f36633ef700 1 --2- 172.21.15.113:0/616636595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f365c06cbd0 0x7f365c06b0d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f36580047f0 tx=0x7f3658009cb0).stop 2024-04-12T05:10:56.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.098+0000 7f36633ef700 1 -- 172.21.15.113:0/616636595 shutdown_connections 2024-04-12T05:10:56.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.098+0000 7f36633ef700 1 --2- 172.21.15.113:0/616636595 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365c06b5c0 0x7f365c06ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:56.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.098+0000 7f36633ef700 1 --2- 172.21.15.113:0/616636595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f365c06cbd0 0x7f365c06b0d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:56.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.098+0000 7f36633ef700 1 -- 172.21.15.113:0/616636595 >> 172.21.15.113:0/616636595 conn(0x7f365c0f3150 msgr2=0x7f365c0f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.098+0000 7f36633ef700 1 -- 172.21.15.113:0/616636595 shutdown_connections 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.098+0000 7f36633ef700 1 -- 172.21.15.113:0/616636595 wait complete. 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f36633ef700 1 Processor -- start 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f36633ef700 1 -- start start 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f36633ef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365c06b5c0 0x7f365c10e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f36633ef700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f365c06cbd0 0x7f365c10ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f36633ef700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c10f390 con 0x7f365c06cbd0 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f36633ef700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c10f4d0 con 0x7f365c06b5c0 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f366098a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f365c06cbd0 0x7f365c10ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f366098a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f365c06cbd0 0x7f365c10ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38840/0 (socket says 172.21.15.113:38840) 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f366118b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365c06b5c0 0x7f365c10e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:10:56.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f366098a700 1 -- 172.21.15.113:0/3191525773 learned_addr learned my addr 172.21.15.113:0/3191525773 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:10:56.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f366098a700 1 -- 172.21.15.113:0/3191525773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365c06b5c0 msgr2=0x7f365c10e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.099+0000 7f366098a700 1 --2- 172.21.15.113:0/3191525773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365c06b5c0 0x7f365c10e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.100+0000 7f366098a700 1 -- 172.21.15.113:0/3191525773 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3658005040 con 0x7f365c06cbd0 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.100+0000 7f366118b700 1 --2- 172.21.15.113:0/3191525773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365c06b5c0 0x7f365c10e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.100+0000 7f366098a700 1 --2- 172.21.15.113:0/3191525773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f365c06cbd0 0x7f365c10ede0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f364c00e8f0 tx=0x7f364c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.101+0000 7f36527fc700 1 -- 172.21.15.113:0/3191525773 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f364c007b40 con 0x7f365c06cbd0 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.101+0000 7f36633ef700 1 -- 172.21.15.113:0/3191525773 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365c113f70 con 0x7f365c06cbd0 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.101+0000 7f36527fc700 1 -- 172.21.15.113:0/3191525773 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f364c00f040 con 0x7f365c06cbd0 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.101+0000 7f36527fc700 1 -- 172.21.15.113:0/3191525773 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f364c014920 con 0x7f365c06cbd0 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.101+0000 7f36633ef700 1 -- 172.21.15.113:0/3191525773 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365c114420 con 0x7f365c06cbd0 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.102+0000 7f36527fc700 1 -- 172.21.15.113:0/3191525773 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 27) v1 ==== 45177+0+0 (secure 0 0 0) 0x7f364c014b60 con 0x7f365c06cbd0 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.102+0000 7f36527fc700 1 -- 172.21.15.113:0/3191525773 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f364c04f170 con 0x7f365c06cbd0 2024-04-12T05:10:56.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.102+0000 7f36633ef700 1 -- 172.21.15.113:0/3191525773 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365c05fb50 con 0x7f365c06cbd0 2024-04-12T05:10:56.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.108+0000 7f36527fc700 1 -- 172.21.15.113:0/3191525773 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f364c019020 con 0x7f365c06cbd0 2024-04-12T05:10:56.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:54.698028+0000 mgr.smithi192.ocppyo (mgr.24357) 122 : cluster [DBG] pgmap v77: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:10:56.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:54.942822+0000 mgr.smithi192.ocppyo (mgr.24357) 123 : audit [DBG] from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:56.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.265200+0000 mgr.smithi192.ocppyo (mgr.24357) 124 : audit [DBG] from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:56.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.775370+0000 mon.smithi113 (mon.0) 731 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.781466+0000 mon.smithi113 (mon.0) 732 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.783418+0000 mon.smithi192 (mon.1) 74 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.784337+0000 mon.smithi192 (mon.1) 75 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.789030+0000 mon.smithi113 (mon.0) 733 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.928442+0000 mon.smithi192 (mon.1) 76 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.933939+0000 mon.smithi192 (mon.1) 77 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mgr fail", "who": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.934271+0000 mon.smithi113 (mon.0) 734 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mgr fail", "who": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:55.940713+0000 mon.smithi113 (mon.0) 735 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.940976+0000 mon.smithi113 (mon.0) 736 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd='[{"prefix": "mgr fail", "who": "smithi192.ocppyo"}]': finished 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:55.941056+0000 mon.smithi113 (mon.0) 737 : cluster [DBG] mgrmap e27: smithi113.ctsazg(active, starting, since 0.00633137s) 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.942299+0000 mon.smithi113 (mon.0) 738 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.942419+0000 mon.smithi113 (mon.0) 739 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.942564+0000 mon.smithi113 (mon.0) 740 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:10:56.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.942766+0000 mon.smithi113 (mon.0) 741 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.943149+0000 mon.smithi113 (mon.0) 742 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.943384+0000 mon.smithi113 (mon.0) 743 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.943650+0000 mon.smithi113 (mon.0) 744 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi113.ctsazg", "id": "smithi113.ctsazg"}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.943859+0000 mon.smithi113 (mon.0) 745 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.944102+0000 mon.smithi113 (mon.0) 746 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.944334+0000 mon.smithi113 (mon.0) 747 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.944559+0000 mon.smithi113 (mon.0) 748 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.944737+0000 mon.smithi113 (mon.0) 749 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.945095+0000 mon.smithi113 (mon.0) 750 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.945349+0000 mon.smithi113 (mon.0) 751 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.945695+0000 mon.smithi113 (mon.0) 752 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.946114+0000 mon.smithi113 (mon.0) 753 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:10:56.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:55.975882+0000 mon.smithi113 (mon.0) 754 : cluster [INF] Manager daemon smithi113.ctsazg is now available 2024-04-12T05:10:56.427 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.977163+0000 mon.smithi113 (mon.0) 755 : audit [DBG] from='client.? 172.21.15.113:0/1068955629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:10:56.427 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:55.998784+0000 mon.smithi113 (mon.0) 756 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:10:56.427 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:56.019416+0000 mon.smithi113 (mon.0) 757 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:10:56.427 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:56.028909+0000 mon.smithi113 (mon.0) 758 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:10:56.427 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:56.046730+0000 mon.smithi113 (mon.0) 759 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/trash_purge_schedule"}]: dispatch 2024-04-12T05:10:56.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:54.698028+0000 mgr.smithi192.ocppyo (mgr.24357) 122 : cluster [DBG] pgmap v77: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:10:56.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:54.942822+0000 mgr.smithi192.ocppyo (mgr.24357) 123 : audit [DBG] from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:56.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.265200+0000 mgr.smithi192.ocppyo (mgr.24357) 124 : audit [DBG] from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:56.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.775370+0000 mon.smithi113 (mon.0) 731 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:56.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.781466+0000 mon.smithi113 (mon.0) 732 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:56.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.783418+0000 mon.smithi192 (mon.1) 74 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:10:56.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.784337+0000 mon.smithi192 (mon.1) 75 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.789030+0000 mon.smithi113 (mon.0) 733 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.928442+0000 mon.smithi192 (mon.1) 76 : audit [DBG] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.933939+0000 mon.smithi192 (mon.1) 77 : audit [INF] from='mgr.24357 172.21.15.192:0/3785106306' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mgr fail", "who": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.934271+0000 mon.smithi113 (mon.0) 734 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "mgr fail", "who": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:55.940713+0000 mon.smithi113 (mon.0) 735 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.940976+0000 mon.smithi113 (mon.0) 736 : audit [INF] from='mgr.24357 ' entity='mgr.smithi192.ocppyo' cmd='[{"prefix": "mgr fail", "who": "smithi192.ocppyo"}]': finished 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:55.941056+0000 mon.smithi113 (mon.0) 737 : cluster [DBG] mgrmap e27: smithi113.ctsazg(active, starting, since 0.00633137s) 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.942299+0000 mon.smithi113 (mon.0) 738 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.942419+0000 mon.smithi113 (mon.0) 739 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.942564+0000 mon.smithi113 (mon.0) 740 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.942766+0000 mon.smithi113 (mon.0) 741 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.943149+0000 mon.smithi113 (mon.0) 742 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.943384+0000 mon.smithi113 (mon.0) 743 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:10:56.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.943650+0000 mon.smithi113 (mon.0) 744 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi113.ctsazg", "id": "smithi113.ctsazg"}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.943859+0000 mon.smithi113 (mon.0) 745 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.944102+0000 mon.smithi113 (mon.0) 746 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.944334+0000 mon.smithi113 (mon.0) 747 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.944559+0000 mon.smithi113 (mon.0) 748 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.944737+0000 mon.smithi113 (mon.0) 749 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.945095+0000 mon.smithi113 (mon.0) 750 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.945349+0000 mon.smithi113 (mon.0) 751 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.945695+0000 mon.smithi113 (mon.0) 752 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.946114+0000 mon.smithi113 (mon.0) 753 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:55.975882+0000 mon.smithi113 (mon.0) 754 : cluster [INF] Manager daemon smithi113.ctsazg is now available 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.977163+0000 mon.smithi113 (mon.0) 755 : audit [DBG] from='client.? 172.21.15.113:0/1068955629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:55.998784+0000 mon.smithi113 (mon.0) 756 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:56.019416+0000 mon.smithi113 (mon.0) 757 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:56.028909+0000 mon.smithi113 (mon.0) 758 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:10:56.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:56.046730+0000 mon.smithi113 (mon.0) 759 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/trash_purge_schedule"}]: dispatch 2024-04-12T05:10:56.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.940+0000 7f36527fc700 1 -- 172.21.15.113:0/3191525773 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 28) v1 ==== 45600+0+0 (secure 0 0 0) 0x7f364c007ca0 con 0x7f365c06cbd0 2024-04-12T05:10:56.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.940+0000 7f36527fc700 1 --2- 172.21.15.113:0/3191525773 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f364803d380 0x7f364803f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:10:56.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.940+0000 7f36527fc700 1 -- 172.21.15.113:0/3191525773 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f364803fc60 con 0x7f364803d380 2024-04-12T05:10:56.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.948+0000 7f366118b700 1 --2- 172.21.15.113:0/3191525773 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f364803d380 0x7f364803f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:10:56.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.949+0000 7f366118b700 1 --2- 172.21.15.113:0/3191525773 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f364803d380 0x7f364803f770 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f36580047c0 tx=0x7f3658009cb0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:10:56.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.951+0000 7f36527fc700 1 -- 172.21.15.113:0/3191525773 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f364803fc60 con 0x7f364803d380 2024-04-12T05:10:56.954 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:10:56.954 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:10:56.955 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:10:56.955 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:10:56.955 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:10:56.955 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr" 2024-04-12T05:10:56.955 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:10:56.955 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "2/2 daemons upgraded", 2024-04-12T05:10:56.955 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "", 2024-04-12T05:10:56.955 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:10:56.955 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:10:56.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.954+0000 7f36633ef700 1 -- 172.21.15.113:0/3191525773 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f364803d380 msgr2=0x7f364803f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:56.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.954+0000 7f36633ef700 1 --2- 172.21.15.113:0/3191525773 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f364803d380 0x7f364803f770 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f36580047c0 tx=0x7f3658009cb0).stop 2024-04-12T05:10:56.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.954+0000 7f36633ef700 1 -- 172.21.15.113:0/3191525773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f365c06cbd0 msgr2=0x7f365c10ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:10:56.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.954+0000 7f36633ef700 1 --2- 172.21.15.113:0/3191525773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f365c06cbd0 0x7f365c10ede0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f364c00e8f0 tx=0x7f364c00b040).stop 2024-04-12T05:10:56.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.954+0000 7f36633ef700 1 -- 172.21.15.113:0/3191525773 shutdown_connections 2024-04-12T05:10:56.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.954+0000 7f36633ef700 1 --2- 172.21.15.113:0/3191525773 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f364803d380 0x7f364803f770 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:56.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.954+0000 7f36633ef700 1 --2- 172.21.15.113:0/3191525773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f365c06b5c0 0x7f365c10e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:56.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.954+0000 7f36633ef700 1 --2- 172.21.15.113:0/3191525773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f365c06cbd0 0x7f365c10ede0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:10:56.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.954+0000 7f36633ef700 1 -- 172.21.15.113:0/3191525773 >> 172.21.15.113:0/3191525773 conn(0x7f365c0f3150 msgr2=0x7f365c0f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:10:56.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.954+0000 7f36633ef700 1 -- 172.21.15.113:0/3191525773 shutdown_connections 2024-04-12T05:10:56.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:10:56.954+0000 7f36633ef700 1 -- 172.21.15.113:0/3191525773 wait complete. 2024-04-12T05:10:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:56.342232+0000 mgr.smithi113.ctsazg (mgr.14656) 1 : cephadm [INF] [12/Apr/2024:05:10:56] ENGINE Bus STARTING 2024-04-12T05:10:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:56.457268+0000 mgr.smithi113.ctsazg (mgr.14656) 2 : cephadm [INF] [12/Apr/2024:05:10:56] ENGINE Serving on https://172.21.15.113:7150 2024-04-12T05:10:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:56.559256+0000 mgr.smithi113.ctsazg (mgr.14656) 3 : cephadm [INF] [12/Apr/2024:05:10:56] ENGINE Serving on http://172.21.15.113:8765 2024-04-12T05:10:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:10:56.559421+0000 mgr.smithi113.ctsazg (mgr.14656) 4 : cephadm [INF] [12/Apr/2024:05:10:56] ENGINE Bus STARTED 2024-04-12T05:10:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:56.943379+0000 mon.smithi113 (mon.0) 760 : cluster [DBG] mgrmap e28: smithi113.ctsazg(active, since 1.00865s) 2024-04-12T05:10:57.428 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:56.342232+0000 mgr.smithi113.ctsazg (mgr.14656) 1 : cephadm [INF] [12/Apr/2024:05:10:56] ENGINE Bus STARTING 2024-04-12T05:10:57.428 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:56.457268+0000 mgr.smithi113.ctsazg (mgr.14656) 2 : cephadm [INF] [12/Apr/2024:05:10:56] ENGINE Serving on https://172.21.15.113:7150 2024-04-12T05:10:57.428 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:56.559256+0000 mgr.smithi113.ctsazg (mgr.14656) 3 : cephadm [INF] [12/Apr/2024:05:10:56] ENGINE Serving on http://172.21.15.113:8765 2024-04-12T05:10:57.428 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:10:56.559421+0000 mgr.smithi113.ctsazg (mgr.14656) 4 : cephadm [INF] [12/Apr/2024:05:10:56] ENGINE Bus STARTED 2024-04-12T05:10:57.428 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:56.943379+0000 mon.smithi113 (mon.0) 760 : cluster [DBG] mgrmap e28: smithi113.ctsazg(active, since 1.00865s) 2024-04-12T05:10:58.570 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:10:56.952812+0000 mgr.smithi113.ctsazg (mgr.14656) 5 : audit [DBG] from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:58.570 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:58 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:56.954874+0000 mgr.smithi113.ctsazg (mgr.14656) 6 : cluster [DBG] pgmap v3: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:10:58.604 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:10:56.952812+0000 mgr.smithi113.ctsazg (mgr.14656) 5 : audit [DBG] from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:10:58.604 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:56.954874+0000 mgr.smithi113.ctsazg (mgr.14656) 6 : cluster [DBG] pgmap v3: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:10:59.523 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:10:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:57.944740+0000 mgr.smithi113.ctsazg (mgr.14656) 7 : cluster [DBG] pgmap v4: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:10:59.553 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:10:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:57.944740+0000 mgr.smithi113.ctsazg (mgr.14656) 7 : cluster [DBG] pgmap v4: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:00.586 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:59.246645+0000 mon.smithi113 (mon.0) 761 : cluster [DBG] mgrmap e29: smithi113.ctsazg(active, since 3s) 2024-04-12T05:11:00.586 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:00.041737+0000 mon.smithi113 (mon.0) 762 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:00.586 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:00.047962+0000 mon.smithi113 (mon.0) 763 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:00.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:59.246645+0000 mon.smithi113 (mon.0) 761 : cluster [DBG] mgrmap e29: smithi113.ctsazg(active, since 3s) 2024-04-12T05:11:00.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:00.041737+0000 mon.smithi113 (mon.0) 762 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:00.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:00.047962+0000 mon.smithi113 (mon.0) 763 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:01.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:10:59.945102+0000 mgr.smithi113.ctsazg (mgr.14656) 8 : cluster [DBG] pgmap v5: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:01.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:00.735394+0000 mon.smithi113 (mon.0) 764 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:01.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:00.739888+0000 mon.smithi113 (mon.0) 765 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:01.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:01.123236+0000 mon.smithi113 (mon.0) 766 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:01.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:01.128998+0000 mon.smithi113 (mon.0) 767 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:10:59.945102+0000 mgr.smithi113.ctsazg (mgr.14656) 8 : cluster [DBG] pgmap v5: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:00.735394+0000 mon.smithi113 (mon.0) 764 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:00.739888+0000 mon.smithi113 (mon.0) 765 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:01.123236+0000 mon.smithi113 (mon.0) 766 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:01.128998+0000 mon.smithi113 (mon.0) 767 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:02.944 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:01.763760+0000 mon.smithi113 (mon.0) 768 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:02.945 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:01.767308+0000 mon.smithi113 (mon.0) 769 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:02.945 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:01.945442+0000 mgr.smithi113.ctsazg (mgr.14656) 9 : cluster [DBG] pgmap v6: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:03.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:01.763760+0000 mon.smithi113 (mon.0) 768 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:03.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:01.767308+0000 mon.smithi113 (mon.0) 769 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:03.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:02 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:01.945442+0000 mgr.smithi113.ctsazg (mgr.14656) 9 : cluster [DBG] pgmap v6: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:03.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:03.075992+0000 mon.smithi113 (mon.0) 770 : cluster [DBG] Standby manager daemon smithi192.ocppyo started 2024-04-12T05:11:03.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:03.077521+0000 mon.smithi192 (mon.1) 78 : audit [DBG] from='mgr.? 172.21.15.192:0/3146860867' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/crt"}]: dispatch 2024-04-12T05:11:03.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:03.077969+0000 mon.smithi192 (mon.1) 79 : audit [DBG] from='mgr.? 172.21.15.192:0/3146860867' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:11:03.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:03.079747+0000 mon.smithi192 (mon.1) 80 : audit [DBG] from='mgr.? 172.21.15.192:0/3146860867' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/key"}]: dispatch 2024-04-12T05:11:03.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:03.080685+0000 mon.smithi192 (mon.1) 81 : audit [DBG] from='mgr.? 172.21.15.192:0/3146860867' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:11:04.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:03.075992+0000 mon.smithi113 (mon.0) 770 : cluster [DBG] Standby manager daemon smithi192.ocppyo started 2024-04-12T05:11:04.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:03.077521+0000 mon.smithi192 (mon.1) 78 : audit [DBG] from='mgr.? 172.21.15.192:0/3146860867' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/crt"}]: dispatch 2024-04-12T05:11:04.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:03.077969+0000 mon.smithi192 (mon.1) 79 : audit [DBG] from='mgr.? 172.21.15.192:0/3146860867' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:11:04.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:03.079747+0000 mon.smithi192 (mon.1) 80 : audit [DBG] from='mgr.? 172.21.15.192:0/3146860867' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/key"}]: dispatch 2024-04-12T05:11:04.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:03.080685+0000 mon.smithi192 (mon.1) 81 : audit [DBG] from='mgr.? 172.21.15.192:0/3146860867' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:11:05.062 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:03.778961+0000 mon.smithi113 (mon.0) 771 : cluster [DBG] mgrmap e30: smithi113.ctsazg(active, since 7s), standbys: smithi192.ocppyo 2024-04-12T05:11:05.063 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:03.779137+0000 mon.smithi113 (mon.0) 772 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi192.ocppyo", "id": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:11:05.063 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:04 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:03.945847+0000 mgr.smithi113.ctsazg (mgr.14656) 10 : cluster [DBG] pgmap v7: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-12T05:11:05.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:03.778961+0000 mon.smithi113 (mon.0) 771 : cluster [DBG] mgrmap e30: smithi113.ctsazg(active, since 7s), standbys: smithi192.ocppyo 2024-04-12T05:11:05.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:03.779137+0000 mon.smithi113 (mon.0) 772 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi192.ocppyo", "id": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:11:05.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:03.945847+0000 mgr.smithi113.ctsazg (mgr.14656) 10 : cluster [DBG] pgmap v7: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-12T05:11:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:05.093622+0000 mon.smithi113 (mon.0) 773 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:05.097583+0000 mon.smithi113 (mon.0) 774 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:05.098706+0000 mon.smithi113 (mon.0) 775 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:11:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:05.099362+0000 mon.smithi113 (mon.0) 776 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:11:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:05.100208+0000 mgr.smithi113.ctsazg (mgr.14656) 11 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.conf 2024-04-12T05:11:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:05.100368+0000 mgr.smithi113.ctsazg (mgr.14656) 12 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-04-12T05:11:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:05.364582+0000 mgr.smithi113.ctsazg (mgr.14656) 13 : cephadm [INF] Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:11:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:05.364716+0000 mgr.smithi113.ctsazg (mgr.14656) 14 : cephadm [INF] Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:11:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:05.093622+0000 mon.smithi113 (mon.0) 773 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:05.097583+0000 mon.smithi113 (mon.0) 774 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:06.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:05.098706+0000 mon.smithi113 (mon.0) 775 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:11:06.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:05.099362+0000 mon.smithi113 (mon.0) 776 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:11:06.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:05.100208+0000 mgr.smithi113.ctsazg (mgr.14656) 11 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.conf 2024-04-12T05:11:06.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:05.100368+0000 mgr.smithi113.ctsazg (mgr.14656) 12 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-04-12T05:11:06.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:05.364582+0000 mgr.smithi113.ctsazg (mgr.14656) 13 : cephadm [INF] Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:11:06.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:05.364716+0000 mgr.smithi113.ctsazg (mgr.14656) 14 : cephadm [INF] Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:11:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:05.629029+0000 mgr.smithi113.ctsazg (mgr.14656) 15 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:11:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:05.629246+0000 mgr.smithi113.ctsazg (mgr.14656) 16 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:11:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:05.893461+0000 mgr.smithi113.ctsazg (mgr.14656) 17 : cephadm [INF] Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:11:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:05.893589+0000 mgr.smithi113.ctsazg (mgr.14656) 18 : cephadm [INF] Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:11:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:05.946118+0000 mgr.smithi113.ctsazg (mgr.14656) 19 : cluster [DBG] pgmap v8: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T05:11:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:06.161701+0000 mon.smithi113 (mon.0) 777 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:06.166595+0000 mon.smithi113 (mon.0) 778 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:06.172107+0000 mon.smithi113 (mon.0) 779 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:07.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:06.177206+0000 mon.smithi113 (mon.0) 780 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:07.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:06.183065+0000 mon.smithi113 (mon.0) 781 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:07.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:06.315130+0000 mgr.smithi113.ctsazg (mgr.14656) 20 : cephadm [INF] Reconfiguring prometheus.smithi192 (dependencies changed)... 2024-04-12T05:11:07.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:06.540204+0000 mgr.smithi113.ctsazg (mgr.14656) 21 : cephadm [INF] Reconfiguring daemon prometheus.smithi192 on smithi192 2024-04-12T05:11:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:05.629029+0000 mgr.smithi113.ctsazg (mgr.14656) 15 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:11:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:05.629246+0000 mgr.smithi113.ctsazg (mgr.14656) 16 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:11:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:05.893461+0000 mgr.smithi113.ctsazg (mgr.14656) 17 : cephadm [INF] Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:11:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:05.893589+0000 mgr.smithi113.ctsazg (mgr.14656) 18 : cephadm [INF] Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:11:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:05.946118+0000 mgr.smithi113.ctsazg (mgr.14656) 19 : cluster [DBG] pgmap v8: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T05:11:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:06.161701+0000 mon.smithi113 (mon.0) 777 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:06.166595+0000 mon.smithi113 (mon.0) 778 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:06.172107+0000 mon.smithi113 (mon.0) 779 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:06.177206+0000 mon.smithi113 (mon.0) 780 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:06.183065+0000 mon.smithi113 (mon.0) 781 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:06.315130+0000 mgr.smithi113.ctsazg (mgr.14656) 20 : cephadm [INF] Reconfiguring prometheus.smithi192 (dependencies changed)... 2024-04-12T05:11:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:06.540204+0000 mgr.smithi113.ctsazg (mgr.14656) 21 : cephadm [INF] Reconfiguring daemon prometheus.smithi192 on smithi192 2024-04-12T05:11:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:07.946679+0000 mgr.smithi113.ctsazg (mgr.14656) 22 : cluster [DBG] pgmap v9: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T05:11:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:07.946679+0000 mgr.smithi113.ctsazg (mgr.14656) 22 : cluster [DBG] pgmap v9: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T05:11:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:09.947053+0000 mgr.smithi113.ctsazg (mgr.14656) 23 : cluster [DBG] pgmap v10: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T05:11:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:11.019685+0000 mon.smithi113 (mon.0) 782 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:11:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:09.947053+0000 mgr.smithi113.ctsazg (mgr.14656) 23 : cluster [DBG] pgmap v10: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T05:11:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:11.019685+0000 mon.smithi113 (mon.0) 782 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:11:12.872 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:11.635522+0000 mon.smithi113 (mon.0) 783 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:12.872 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:11.640051+0000 mon.smithi113 (mon.0) 784 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:12.872 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:11.671029+0000 mon.smithi113 (mon.0) 785 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:11:12.872 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:11.671644+0000 mgr.smithi113.ctsazg (mgr.14656) 24 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:11:12.872 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:11.677680+0000 mon.smithi113 (mon.0) 786 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:12.873 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:11.947406+0000 mgr.smithi113.ctsazg (mgr.14656) 25 : cluster [DBG] pgmap v11: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T05:11:12.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:11.635522+0000 mon.smithi113 (mon.0) 783 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:12.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:11.640051+0000 mon.smithi113 (mon.0) 784 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:12.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:11.671029+0000 mon.smithi113 (mon.0) 785 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:11:12.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:11.671644+0000 mgr.smithi113.ctsazg (mgr.14656) 24 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:11:12.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:11.677680+0000 mon.smithi113 (mon.0) 786 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:12.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:11.947406+0000 mgr.smithi113.ctsazg (mgr.14656) 25 : cluster [DBG] pgmap v11: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T05:11:13.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:13.248080+0000 mgr.smithi113.ctsazg (mgr.14656) 26 : cephadm [INF] Upgrade: Updating mgr.smithi192.ocppyo 2024-04-12T05:11:13.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:13.248498+0000 mon.smithi113 (mon.0) 787 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.ocppyo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:11:13.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:13.249452+0000 mon.smithi113 (mon.0) 788 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:11:13.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:13.250319+0000 mon.smithi113 (mon.0) 789 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:11:13.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:13.251228+0000 mgr.smithi113.ctsazg (mgr.14656) 27 : cephadm [INF] Deploying daemon mgr.smithi192.ocppyo on smithi192 2024-04-12T05:11:13.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:13.248080+0000 mgr.smithi113.ctsazg (mgr.14656) 26 : cephadm [INF] Upgrade: Updating mgr.smithi192.ocppyo 2024-04-12T05:11:13.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:13.248498+0000 mon.smithi113 (mon.0) 787 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.ocppyo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:11:13.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:13.249452+0000 mon.smithi113 (mon.0) 788 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:11:13.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:13.250319+0000 mon.smithi113 (mon.0) 789 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:11:13.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:13.251228+0000 mgr.smithi113.ctsazg (mgr.14656) 27 : cephadm [INF] Deploying daemon mgr.smithi192.ocppyo on smithi192 2024-04-12T05:11:14.685 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:13.948244+0000 mgr.smithi113.ctsazg (mgr.14656) 28 : cluster [DBG] pgmap v12: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T05:11:14.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:13.948244+0000 mgr.smithi113.ctsazg (mgr.14656) 28 : cluster [DBG] pgmap v12: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T05:11:17.408 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:15.948614+0000 mgr.smithi113.ctsazg (mgr.14656) 29 : cluster [DBG] pgmap v13: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:17.409 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:16.886018+0000 mon.smithi113 (mon.0) 790 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:17.409 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:16.890402+0000 mon.smithi113 (mon.0) 791 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:17.409 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:16.923464+0000 mon.smithi113 (mon.0) 792 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:15.948614+0000 mgr.smithi113.ctsazg (mgr.14656) 29 : cluster [DBG] pgmap v13: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:16.886018+0000 mon.smithi113 (mon.0) 790 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:16.890402+0000 mon.smithi113 (mon.0) 791 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:16.923464+0000 mon.smithi113 (mon.0) 792 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:19.270 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:17.949365+0000 mgr.smithi113.ctsazg (mgr.14656) 30 : cluster [DBG] pgmap v14: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:17.949365+0000 mgr.smithi113.ctsazg (mgr.14656) 30 : cluster [DBG] pgmap v14: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:21.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:19.949714+0000 mgr.smithi113.ctsazg (mgr.14656) 31 : cluster [DBG] pgmap v15: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:21.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:21.004162+0000 mon.smithi113 (mon.0) 793 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:19.949714+0000 mgr.smithi113.ctsazg (mgr.14656) 31 : cluster [DBG] pgmap v15: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:21.004162+0000 mon.smithi113 (mon.0) 793 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:22.271 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:21.011564+0000 mon.smithi113 (mon.0) 794 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:22.271 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:21.699526+0000 mon.smithi113 (mon.0) 795 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:22.271 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:21.704139+0000 mon.smithi113 (mon.0) 796 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:22.286 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:21.011564+0000 mon.smithi113 (mon.0) 794 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:22.286 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:21.699526+0000 mon.smithi113 (mon.0) 795 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:22.286 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:21.704139+0000 mon.smithi113 (mon.0) 796 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:23.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:21.950068+0000 mgr.smithi113.ctsazg (mgr.14656) 32 : cluster [DBG] pgmap v16: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:21.950068+0000 mgr.smithi113.ctsazg (mgr.14656) 32 : cluster [DBG] pgmap v16: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:23.914415+0000 mon.smithi113 (mon.0) 797 : cluster [DBG] Standby manager daemon smithi192.ocppyo restarted 2024-04-12T05:11:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:23.914562+0000 mon.smithi113 (mon.0) 798 : cluster [DBG] Standby manager daemon smithi192.ocppyo started 2024-04-12T05:11:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:23.915789+0000 mon.smithi192 (mon.1) 82 : audit [DBG] from='mgr.? 172.21.15.192:0/1308084764' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/crt"}]: dispatch 2024-04-12T05:11:24.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:23.916227+0000 mon.smithi192 (mon.1) 83 : audit [DBG] from='mgr.? 172.21.15.192:0/1308084764' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:11:24.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:23.918034+0000 mon.smithi192 (mon.1) 84 : audit [DBG] from='mgr.? 172.21.15.192:0/1308084764' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/key"}]: dispatch 2024-04-12T05:11:24.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:23.919029+0000 mon.smithi192 (mon.1) 85 : audit [DBG] from='mgr.? 172.21.15.192:0/1308084764' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:11:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:23.914415+0000 mon.smithi113 (mon.0) 797 : cluster [DBG] Standby manager daemon smithi192.ocppyo restarted 2024-04-12T05:11:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:23.914562+0000 mon.smithi113 (mon.0) 798 : cluster [DBG] Standby manager daemon smithi192.ocppyo started 2024-04-12T05:11:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:23.915789+0000 mon.smithi192 (mon.1) 82 : audit [DBG] from='mgr.? 172.21.15.192:0/1308084764' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/crt"}]: dispatch 2024-04-12T05:11:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:23.916227+0000 mon.smithi192 (mon.1) 83 : audit [DBG] from='mgr.? 172.21.15.192:0/1308084764' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:11:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:23.918034+0000 mon.smithi192 (mon.1) 84 : audit [DBG] from='mgr.? 172.21.15.192:0/1308084764' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/key"}]: dispatch 2024-04-12T05:11:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:23.919029+0000 mon.smithi192 (mon.1) 85 : audit [DBG] from='mgr.? 172.21.15.192:0/1308084764' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:11:25.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:23.950678+0000 mgr.smithi113.ctsazg (mgr.14656) 33 : cluster [DBG] pgmap v17: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:25.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:24.034620+0000 mon.smithi113 (mon.0) 799 : cluster [DBG] mgrmap e31: smithi113.ctsazg(active, since 28s), standbys: smithi192.ocppyo 2024-04-12T05:11:25.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:24.931791+0000 mon.smithi113 (mon.0) 800 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:25.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:24.936516+0000 mon.smithi113 (mon.0) 801 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:25.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:24.937428+0000 mon.smithi113 (mon.0) 802 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:11:25.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:24.938240+0000 mon.smithi113 (mon.0) 803 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:11:25.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:24.942864+0000 mon.smithi113 (mon.0) 804 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:23.950678+0000 mgr.smithi113.ctsazg (mgr.14656) 33 : cluster [DBG] pgmap v17: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:24.034620+0000 mon.smithi113 (mon.0) 799 : cluster [DBG] mgrmap e31: smithi113.ctsazg(active, since 28s), standbys: smithi192.ocppyo 2024-04-12T05:11:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:24.931791+0000 mon.smithi113 (mon.0) 800 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:24.936516+0000 mon.smithi113 (mon.0) 801 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:24.937428+0000 mon.smithi113 (mon.0) 802 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:11:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:24.938240+0000 mon.smithi113 (mon.0) 803 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:11:25.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:24.942864+0000 mon.smithi113 (mon.0) 804 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.324 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.081378+0000 mon.smithi113 (mon.0) 805 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:26.324 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.083891+0000 mon.smithi113 (mon.0) 806 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.324 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:25.084540+0000 mgr.smithi113.ctsazg (mgr.14656) 34 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-04-12T05:11:26.324 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.088831+0000 mon.smithi113 (mon.0) 807 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.324 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.089553+0000 mon.smithi113 (mon.0) 808 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi113.ctsazg"}]: dispatch 2024-04-12T05:11:26.324 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.094841+0000 mon.smithi113 (mon.0) 809 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi113.ctsazg"}]': finished 2024-04-12T05:11:26.324 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.095481+0000 mon.smithi113 (mon.0) 810 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi192.ocppyo"}]: dispatch 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.101091+0000 mon.smithi113 (mon.0) 811 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi192.ocppyo"}]': finished 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.103381+0000 mon.smithi113 (mon.0) 812 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:25.103949+0000 mgr.smithi113.ctsazg (mgr.14656) 35 : cephadm [INF] Upgrade: Setting container_image for all rgw 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.109625+0000 mon.smithi113 (mon.0) 813 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.110535+0000 mon.smithi113 (mon.0) 814 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:25.111069+0000 mgr.smithi113.ctsazg (mgr.14656) 36 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.116753+0000 mon.smithi113 (mon.0) 815 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.117685+0000 mon.smithi113 (mon.0) 816 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:25.118225+0000 mgr.smithi113.ctsazg (mgr.14656) 37 : cephadm [INF] Upgrade: Setting container_image for all cephfs-mirror 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.123872+0000 mon.smithi113 (mon.0) 817 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.124778+0000 mon.smithi113 (mon.0) 818 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:25.125306+0000 mgr.smithi113.ctsazg (mgr.14656) 38 : cephadm [INF] Upgrade: Setting container_image for all ceph-exporter 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.131263+0000 mon.smithi113 (mon.0) 819 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.132183+0000 mon.smithi113 (mon.0) 820 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.325 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:25.132704+0000 mgr.smithi113.ctsazg (mgr.14656) 39 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-04-12T05:11:26.326 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.138288+0000 mon.smithi113 (mon.0) 821 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.326 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.139199+0000 mon.smithi113 (mon.0) 822 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.326 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:25.139693+0000 mgr.smithi113.ctsazg (mgr.14656) 40 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-04-12T05:11:26.326 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.145400+0000 mon.smithi113 (mon.0) 823 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.326 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.146387+0000 mon.smithi113 (mon.0) 824 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.326 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:25.146899+0000 mgr.smithi113.ctsazg (mgr.14656) 41 : cephadm [INF] Upgrade: Setting container_image for all nvmeof 2024-04-12T05:11:26.326 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:25.152331+0000 mon.smithi113 (mon.0) 825 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.326 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:26.019895+0000 mon.smithi113 (mon.0) 826 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:11:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.081378+0000 mon.smithi113 (mon.0) 805 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.083891+0000 mon.smithi113 (mon.0) 806 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:25.084540+0000 mgr.smithi113.ctsazg (mgr.14656) 34 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-04-12T05:11:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.088831+0000 mon.smithi113 (mon.0) 807 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.089553+0000 mon.smithi113 (mon.0) 808 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi113.ctsazg"}]: dispatch 2024-04-12T05:11:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.094841+0000 mon.smithi113 (mon.0) 809 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi113.ctsazg"}]': finished 2024-04-12T05:11:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.095481+0000 mon.smithi113 (mon.0) 810 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi192.ocppyo"}]: dispatch 2024-04-12T05:11:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.101091+0000 mon.smithi113 (mon.0) 811 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi192.ocppyo"}]': finished 2024-04-12T05:11:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.103381+0000 mon.smithi113 (mon.0) 812 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:25.103949+0000 mgr.smithi113.ctsazg (mgr.14656) 35 : cephadm [INF] Upgrade: Setting container_image for all rgw 2024-04-12T05:11:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.109625+0000 mon.smithi113 (mon.0) 813 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.110535+0000 mon.smithi113 (mon.0) 814 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:25.111069+0000 mgr.smithi113.ctsazg (mgr.14656) 36 : cephadm [INF] Upgrade: Setting container_image for all rbd-mirror 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.116753+0000 mon.smithi113 (mon.0) 815 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.117685+0000 mon.smithi113 (mon.0) 816 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:25.118225+0000 mgr.smithi113.ctsazg (mgr.14656) 37 : cephadm [INF] Upgrade: Setting container_image for all cephfs-mirror 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.123872+0000 mon.smithi113 (mon.0) 817 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.124778+0000 mon.smithi113 (mon.0) 818 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:25.125306+0000 mgr.smithi113.ctsazg (mgr.14656) 38 : cephadm [INF] Upgrade: Setting container_image for all ceph-exporter 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.131263+0000 mon.smithi113 (mon.0) 819 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.132183+0000 mon.smithi113 (mon.0) 820 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:25.132704+0000 mgr.smithi113.ctsazg (mgr.14656) 39 : cephadm [INF] Upgrade: Setting container_image for all iscsi 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.138288+0000 mon.smithi113 (mon.0) 821 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.139199+0000 mon.smithi113 (mon.0) 822 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:25.139693+0000 mgr.smithi113.ctsazg (mgr.14656) 40 : cephadm [INF] Upgrade: Setting container_image for all nfs 2024-04-12T05:11:26.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.145400+0000 mon.smithi113 (mon.0) 823 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.146387+0000 mon.smithi113 (mon.0) 824 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:26.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:25.146899+0000 mgr.smithi113.ctsazg (mgr.14656) 41 : cephadm [INF] Upgrade: Setting container_image for all nvmeof 2024-04-12T05:11:26.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:25.152331+0000 mon.smithi113 (mon.0) 825 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:26.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:26.019895+0000 mon.smithi113 (mon.0) 826 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:11:27.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.061+0000 7fbb4e0ca700 1 -- 172.21.15.113:0/396030259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb480f8cb0 msgr2=0x7fbb480f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.061+0000 7fbb4e0ca700 1 --2- 172.21.15.113:0/396030259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb480f8cb0 0x7fbb480f9110 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fbb300047f0 tx=0x7fbb30009cb0).stop 2024-04-12T05:11:27.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.061+0000 7fbb4e0ca700 1 -- 172.21.15.113:0/396030259 shutdown_connections 2024-04-12T05:11:27.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.061+0000 7fbb4e0ca700 1 --2- 172.21.15.113:0/396030259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb480f8cb0 0x7fbb480f9110 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.061+0000 7fbb4e0ca700 1 --2- 172.21.15.113:0/396030259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480f7a10 0x7fbb480f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:11:27.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.061+0000 7fbb4e0ca700 1 -- 172.21.15.113:0/396030259 >> 172.21.15.113:0/396030259 conn(0x7fbb480f3150 msgr2=0x7fbb480f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:27.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.061+0000 7fbb4e0ca700 1 -- 172.21.15.113:0/396030259 shutdown_connections 2024-04-12T05:11:27.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.061+0000 7fbb4e0ca700 1 -- 172.21.15.113:0/396030259 wait complete. 2024-04-12T05:11:27.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.062+0000 7fbb4e0ca700 1 Processor -- start 2024-04-12T05:11:27.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.062+0000 7fbb4e0ca700 1 -- start start 2024-04-12T05:11:27.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.062+0000 7fbb4e0ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480f7a10 0x7fbb48108210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.062+0000 7fbb4e0ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb480f8cb0 0x7fbb48108700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.062+0000 7fbb4e0ca700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb48108d40 con 0x7fbb480f8cb0 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.062+0000 7fbb4e0ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb48108e80 con 0x7fbb480f7a10 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb46ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb480f8cb0 0x7fbb48108700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb46ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb480f8cb0 0x7fbb48108700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40638/0 (socket says 172.21.15.113:40638) 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb46ffd700 1 -- 172.21.15.113:0/2017489625 learned_addr learned my addr 172.21.15.113:0/2017489625 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb477fe700 1 --2- 172.21.15.113:0/2017489625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480f7a10 0x7fbb48108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb46ffd700 1 -- 172.21.15.113:0/2017489625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480f7a10 msgr2=0x7fbb48108210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb46ffd700 1 --2- 172.21.15.113:0/2017489625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480f7a10 0x7fbb48108210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb46ffd700 1 -- 172.21.15.113:0/2017489625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb30005040 con 0x7fbb480f8cb0 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb46ffd700 1 --2- 172.21.15.113:0/2017489625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb480f8cb0 0x7fbb48108700 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fbb3000b120 tx=0x7fbb30009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb44ff9700 1 -- 172.21.15.113:0/2017489625 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb3000e030 con 0x7fbb480f8cb0 2024-04-12T05:11:27.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb4e0ca700 1 -- 172.21.15.113:0/2017489625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb4810d8c0 con 0x7fbb480f8cb0 2024-04-12T05:11:27.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb44ff9700 1 -- 172.21.15.113:0/2017489625 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb30007e90 con 0x7fbb480f8cb0 2024-04-12T05:11:27.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb44ff9700 1 -- 172.21.15.113:0/2017489625 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb30018070 con 0x7fbb480f8cb0 2024-04-12T05:11:27.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.063+0000 7fbb4e0ca700 1 -- 172.21.15.113:0/2017489625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4810dda0 con 0x7fbb480f8cb0 2024-04-12T05:11:27.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.064+0000 7fbb3e7fc700 1 -- 172.21.15.113:0/2017489625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb2c005120 con 0x7fbb480f8cb0 2024-04-12T05:11:27.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.065+0000 7fbb44ff9700 1 -- 172.21.15.113:0/2017489625 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 31) v1 ==== 90781+0+0 (secure 0 0 0) 0x7fbb3001d420 con 0x7fbb480f8cb0 2024-04-12T05:11:27.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.065+0000 7fbb44ff9700 1 --2- 172.21.15.113:0/2017489625 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fbb3406c7a0 0x7fbb3406ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:27.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.065+0000 7fbb44ff9700 1 -- 172.21.15.113:0/2017489625 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fbb3008cfa0 con 0x7fbb480f8cb0 2024-04-12T05:11:27.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.065+0000 7fbb477fe700 1 --2- 172.21.15.113:0/2017489625 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fbb3406c7a0 0x7fbb3406ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:11:27.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.065+0000 7fbb477fe700 1 --2- 172.21.15.113:0/2017489625 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fbb3406c7a0 0x7fbb3406ec60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fbb480f8b10 tx=0x7fbb38009780).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:27.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.070+0000 7fbb44ff9700 1 -- 172.21.15.113:0/2017489625 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fbb3005ae80 con 0x7fbb480f8cb0 2024-04-12T05:11:27.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.311+0000 7fbb3e7fc700 1 -- 172.21.15.113:0/2017489625 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb2c000bc0 con 0x7fbb3406c7a0 2024-04-12T05:11:27.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.313+0000 7fbb44ff9700 1 -- 172.21.15.113:0/2017489625 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7fbb2c000bc0 con 0x7fbb3406c7a0 2024-04-12T05:11:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.315+0000 7fbb3e7fc700 1 -- 172.21.15.113:0/2017489625 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fbb3406c7a0 msgr2=0x7fbb3406ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.315+0000 7fbb3e7fc700 1 --2- 172.21.15.113:0/2017489625 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fbb3406c7a0 0x7fbb3406ec60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fbb480f8b10 tx=0x7fbb38009780).stop 2024-04-12T05:11:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.315+0000 7fbb3e7fc700 1 -- 172.21.15.113:0/2017489625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb480f8cb0 msgr2=0x7fbb48108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.315+0000 7fbb3e7fc700 1 --2- 172.21.15.113:0/2017489625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb480f8cb0 0x7fbb48108700 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fbb3000b120 tx=0x7fbb30009cb0).stop 2024-04-12T05:11:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.315+0000 7fbb3e7fc700 1 -- 172.21.15.113:0/2017489625 shutdown_connections 2024-04-12T05:11:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.315+0000 7fbb3e7fc700 1 --2- 172.21.15.113:0/2017489625 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fbb3406c7a0 0x7fbb3406ec60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.315+0000 7fbb3e7fc700 1 --2- 172.21.15.113:0/2017489625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb480f7a10 0x7fbb48108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.315+0000 7fbb3e7fc700 1 --2- 172.21.15.113:0/2017489625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb480f8cb0 0x7fbb48108700 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.315+0000 7fbb3e7fc700 1 -- 172.21.15.113:0/2017489625 >> 172.21.15.113:0/2017489625 conn(0x7fbb480f3150 msgr2=0x7fbb480fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:27.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.315+0000 7fbb3e7fc700 1 -- 172.21.15.113:0/2017489625 shutdown_connections 2024-04-12T05:11:27.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.315+0000 7fbb3e7fc700 1 -- 172.21.15.113:0/2017489625 wait complete. 2024-04-12T05:11:27.328 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:11:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:25.951045+0000 mgr.smithi113.ctsazg (mgr.14656) 42 : cluster [DBG] pgmap v18: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:27.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.423+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/3792945337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ff40f8cb0 msgr2=0x7f7ff40f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.423+0000 7f7ffb7fd700 1 --2- 172.21.15.113:0/3792945337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ff40f8cb0 0x7f7ff40f9110 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f7fe80047f0 tx=0x7f7fe8009cb0).stop 2024-04-12T05:11:27.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.423+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/3792945337 shutdown_connections 2024-04-12T05:11:27.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.423+0000 7f7ffb7fd700 1 --2- 172.21.15.113:0/3792945337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ff40f8cb0 0x7f7ff40f9110 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.423+0000 7f7ffb7fd700 1 --2- 172.21.15.113:0/3792945337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ff40f7a10 0x7f7ff40f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.423+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/3792945337 >> 172.21.15.113:0/3792945337 conn(0x7f7ff40f3150 msgr2=0x7f7ff40f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:27.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.423+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/3792945337 shutdown_connections 2024-04-12T05:11:27.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.423+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/3792945337 wait complete. 2024-04-12T05:11:27.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.423+0000 7f7ffb7fd700 1 Processor -- start 2024-04-12T05:11:27.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.423+0000 7f7ffb7fd700 1 -- start start 2024-04-12T05:11:27.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ffb7fd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ff40f7a10 0x7f7ff4108190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:27.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ffb7fd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ff40f8cb0 0x7f7ff4108680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:27.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ffb7fd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff4108c30 con 0x7f7ff40f7a10 2024-04-12T05:11:27.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ffb7fd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff4108d70 con 0x7f7ff40f8cb0 2024-04-12T05:11:27.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ff9599700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ff40f7a10 0x7f7ff4108190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:27.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ff9599700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ff40f7a10 0x7f7ff4108190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40654/0 (socket says 172.21.15.113:40654) 2024-04-12T05:11:27.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ff9599700 1 -- 172.21.15.113:0/2404083314 learned_addr learned my addr 172.21.15.113:0/2404083314 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:11:27.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ff8d98700 1 --2- 172.21.15.113:0/2404083314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ff40f8cb0 0x7f7ff4108680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:27.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ff9599700 1 -- 172.21.15.113:0/2404083314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ff40f8cb0 msgr2=0x7f7ff4108680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ff9599700 1 --2- 172.21.15.113:0/2404083314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ff40f8cb0 0x7f7ff4108680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ff9599700 1 -- 172.21.15.113:0/2404083314 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe8005040 con 0x7f7ff40f7a10 2024-04-12T05:11:27.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ff9599700 1 --2- 172.21.15.113:0/2404083314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ff40f7a10 0x7f7ff4108190 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f7fec00db00 tx=0x7f7fec004e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:27.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ff27fc700 1 -- 172.21.15.113:0/2404083314 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fec00b750 con 0x7f7ff40f7a10 2024-04-12T05:11:27.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ff27fc700 1 -- 172.21.15.113:0/2404083314 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fec00f040 con 0x7f7ff40f7a10 2024-04-12T05:11:27.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/2404083314 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff410d810 con 0x7f7ff40f7a10 2024-04-12T05:11:27.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ff27fc700 1 -- 172.21.15.113:0/2404083314 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fec013be0 con 0x7f7ff40f7a10 2024-04-12T05:11:27.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.424+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/2404083314 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff410ddd0 con 0x7f7ff40f7a10 2024-04-12T05:11:27.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.425+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/2404083314 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff405fb50 con 0x7f7ff40f7a10 2024-04-12T05:11:27.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.426+0000 7f7ff27fc700 1 -- 172.21.15.113:0/2404083314 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 31) v1 ==== 90781+0+0 (secure 0 0 0) 0x7f7fec0133f0 con 0x7f7ff40f7a10 2024-04-12T05:11:27.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.426+0000 7f7ff27fc700 1 --2- 172.21.15.113:0/2404083314 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7fdc06c7c0 0x7f7fdc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:27.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.426+0000 7f7ff27fc700 1 -- 172.21.15.113:0/2404083314 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f7fec08b500 con 0x7f7ff40f7a10 2024-04-12T05:11:27.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.426+0000 7f7ff8d98700 1 --2- 172.21.15.113:0/2404083314 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7fdc06c7c0 0x7f7fdc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:11:27.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.427+0000 7f7ff8d98700 1 --2- 172.21.15.113:0/2404083314 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7fdc06c7c0 0x7f7fdc06ec80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f7fe8004f10 tx=0x7f7fe800b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:27.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.431+0000 7f7ff27fc700 1 -- 172.21.15.113:0/2404083314 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f7fec0593e0 con 0x7f7ff40f7a10 2024-04-12T05:11:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:25.951045+0000 mgr.smithi113.ctsazg (mgr.14656) 42 : cluster [DBG] pgmap v18: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:27.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.644+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/2404083314 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ff4059120 con 0x7f7fdc06c7c0 2024-04-12T05:11:27.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.646+0000 7f7ff27fc700 1 -- 172.21.15.113:0/2404083314 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f7ff4059120 con 0x7f7fdc06c7c0 2024-04-12T05:11:27.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.648+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/2404083314 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7fdc06c7c0 msgr2=0x7f7fdc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.648+0000 7f7ffb7fd700 1 --2- 172.21.15.113:0/2404083314 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7fdc06c7c0 0x7f7fdc06ec80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f7fe8004f10 tx=0x7f7fe800b040).stop 2024-04-12T05:11:27.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.648+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/2404083314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ff40f7a10 msgr2=0x7f7ff4108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.648+0000 7f7ffb7fd700 1 --2- 172.21.15.113:0/2404083314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ff40f7a10 0x7f7ff4108190 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f7fec00db00 tx=0x7f7fec004e80).stop 2024-04-12T05:11:27.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.648+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/2404083314 shutdown_connections 2024-04-12T05:11:27.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.648+0000 7f7ffb7fd700 1 --2- 172.21.15.113:0/2404083314 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7fdc06c7c0 0x7f7fdc06ec80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.648+0000 7f7ffb7fd700 1 --2- 172.21.15.113:0/2404083314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ff40f7a10 0x7f7ff4108190 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.648+0000 7f7ffb7fd700 1 --2- 172.21.15.113:0/2404083314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ff40f8cb0 0x7f7ff4108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.648+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/2404083314 >> 172.21.15.113:0/2404083314 conn(0x7f7ff40f3150 msgr2=0x7f7ff40fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:27.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.648+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/2404083314 shutdown_connections 2024-04-12T05:11:27.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.648+0000 7f7ffb7fd700 1 -- 172.21.15.113:0/2404083314 wait complete. 2024-04-12T05:11:27.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.747+0000 7fc483216700 1 -- 172.21.15.113:0/375098025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc47c0f66f0 msgr2=0x7fc47c0f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.747+0000 7fc483216700 1 --2- 172.21.15.113:0/375098025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc47c0f66f0 0x7fc47c0f6b10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fc474009b60 tx=0x7fc474004d10).stop 2024-04-12T05:11:27.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.747+0000 7fc483216700 1 -- 172.21.15.113:0/375098025 shutdown_connections 2024-04-12T05:11:27.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.747+0000 7fc483216700 1 --2- 172.21.15.113:0/375098025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc47c0f7990 0x7fc47c0f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.747+0000 7fc483216700 1 --2- 172.21.15.113:0/375098025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc47c0f66f0 0x7fc47c0f6b10 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.747+0000 7fc483216700 1 -- 172.21.15.113:0/375098025 >> 172.21.15.113:0/375098025 conn(0x7fc47c0f1e50 msgr2=0x7fc47c0f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:27.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.747+0000 7fc483216700 1 -- 172.21.15.113:0/375098025 shutdown_connections 2024-04-12T05:11:27.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.747+0000 7fc483216700 1 -- 172.21.15.113:0/375098025 wait complete. 2024-04-12T05:11:27.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc483216700 1 Processor -- start 2024-04-12T05:11:27.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc483216700 1 -- start start 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc483216700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc47c0f66f0 0x7fc47c106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc483216700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc47c0f7990 0x7fc47c107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc483216700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c107930 con 0x7fc47c0f7990 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc483216700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c107a70 con 0x7fc47c0f66f0 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc47bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc47c0f7990 0x7fc47c107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc47bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc47c0f7990 0x7fc47c107380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40672/0 (socket says 172.21.15.113:40672) 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc47bfff700 1 -- 172.21.15.113:0/503365290 learned_addr learned my addr 172.21.15.113:0/503365290 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc480fb2700 1 --2- 172.21.15.113:0/503365290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc47c0f66f0 0x7fc47c106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc47bfff700 1 -- 172.21.15.113:0/503365290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc47c0f66f0 msgr2=0x7fc47c106e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc47bfff700 1 --2- 172.21.15.113:0/503365290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc47c0f66f0 0x7fc47c106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc47bfff700 1 -- 172.21.15.113:0/503365290 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4740097c0 con 0x7fc47c0f7990 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc47bfff700 1 --2- 172.21.15.113:0/503365290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc47c0f7990 0x7fc47c107380 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fc4700041d0 tx=0x7fc470009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:27.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc479ffb700 1 -- 172.21.15.113:0/503365290 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc470002d90 con 0x7fc47c0f7990 2024-04-12T05:11:27.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc483216700 1 -- 172.21.15.113:0/503365290 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47c10c510 con 0x7fc47c0f7990 2024-04-12T05:11:27.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.748+0000 7fc479ffb700 1 -- 172.21.15.113:0/503365290 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc47000f040 con 0x7fc47c0f7990 2024-04-12T05:11:27.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.749+0000 7fc479ffb700 1 -- 172.21.15.113:0/503365290 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc47000d800 con 0x7fc47c0f7990 2024-04-12T05:11:27.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.749+0000 7fc483216700 1 -- 172.21.15.113:0/503365290 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47c10cad0 con 0x7fc47c0f7990 2024-04-12T05:11:27.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.750+0000 7fc483216700 1 -- 172.21.15.113:0/503365290 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47c0feeb0 con 0x7fc47c0f7990 2024-04-12T05:11:27.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.750+0000 7fc479ffb700 1 -- 172.21.15.113:0/503365290 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 31) v1 ==== 90781+0+0 (secure 0 0 0) 0x7fc47000dac0 con 0x7fc47c0f7990 2024-04-12T05:11:27.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.750+0000 7fc479ffb700 1 --2- 172.21.15.113:0/503365290 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc46406c7c0 0x7fc46406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:27.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.750+0000 7fc479ffb700 1 -- 172.21.15.113:0/503365290 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fc47008ba80 con 0x7fc47c0f7990 2024-04-12T05:11:27.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.751+0000 7fc480fb2700 1 --2- 172.21.15.113:0/503365290 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc46406c7c0 0x7fc46406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:11:27.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.751+0000 7fc480fb2700 1 --2- 172.21.15.113:0/503365290 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc46406c7c0 0x7fc46406ec80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fc474009750 tx=0x7fc474004d10).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:27.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.755+0000 7fc479ffb700 1 -- 172.21.15.113:0/503365290 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fc470059960 con 0x7fc47c0f7990 2024-04-12T05:11:27.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.958+0000 7fc483216700 1 -- 172.21.15.113:0/503365290 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc47c059120 con 0x7fc46406c7c0 2024-04-12T05:11:27.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.968+0000 7fc479ffb700 1 -- 172.21.15.113:0/503365290 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fc47c059120 con 0x7fc46406c7c0 2024-04-12T05:11:27.970 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:11:27.970 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (115s) 26s ago 10m 14.0M - 0.23.0 ba2b418f427c b2262b7eedd3 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (10m) 26s ago 10m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (9m) 6s ago 9m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (88s) 26s ago 10m 43.3M - 8.3.5 dad864ee21e9 d33ad1514a4c 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (5m) 26s ago 5m 121M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (5m) 26s ago 5m 16.4M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (5m) 6s ago 5m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (5m) 6s ago 5m 14.2M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (42s) 26s ago 11m 471M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (11s) 6s ago 9m 49.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (11m) 26s ago 11m 50.0M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (9m) 6s ago 9m 44.4M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (85s) 26s ago 10m 15.6M - 1.3.1 1dbe0e931976 b61fc2c12125 2024-04-12T05:11:27.971 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (83s) 6s ago 9m 20.5M - 1.3.1 1dbe0e931976 2b882fe39886 2024-04-12T05:11:27.972 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (9m) 26s ago 9m 226M 1910M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:11:27.972 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (8m) 26s ago 8m 240M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:11:27.972 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (8m) 26s ago 8m 202M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:11:27.972 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (7m) 6s ago 7m 264M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:11:27.972 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (7m) 6s ago 7m 232M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:11:27.972 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (6m) 6s ago 6m 226M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:11:27.972 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (16s) 6s ago 9m 47.1M - 2.33.4 514e6a882f6e 50ee0cc5543a 2024-04-12T05:11:27.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.970+0000 7fc483216700 1 -- 172.21.15.113:0/503365290 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc46406c7c0 msgr2=0x7fc46406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.970+0000 7fc483216700 1 --2- 172.21.15.113:0/503365290 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc46406c7c0 0x7fc46406ec80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fc474009750 tx=0x7fc474004d10).stop 2024-04-12T05:11:27.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.970+0000 7fc483216700 1 -- 172.21.15.113:0/503365290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc47c0f7990 msgr2=0x7fc47c107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:27.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.970+0000 7fc483216700 1 --2- 172.21.15.113:0/503365290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc47c0f7990 0x7fc47c107380 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fc4700041d0 tx=0x7fc470009f60).stop 2024-04-12T05:11:27.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.970+0000 7fc483216700 1 -- 172.21.15.113:0/503365290 shutdown_connections 2024-04-12T05:11:27.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.970+0000 7fc483216700 1 --2- 172.21.15.113:0/503365290 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc46406c7c0 0x7fc46406ec80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.970+0000 7fc483216700 1 --2- 172.21.15.113:0/503365290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc47c0f66f0 0x7fc47c106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.970+0000 7fc483216700 1 --2- 172.21.15.113:0/503365290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc47c0f7990 0x7fc47c107380 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:27.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.970+0000 7fc483216700 1 -- 172.21.15.113:0/503365290 >> 172.21.15.113:0/503365290 conn(0x7fc47c0f1e50 msgr2=0x7fc47c0fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:27.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.971+0000 7fc483216700 1 -- 172.21.15.113:0/503365290 shutdown_connections 2024-04-12T05:11:27.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:27.971+0000 7fc483216700 1 -- 172.21.15.113:0/503365290 wait complete. 2024-04-12T05:11:28.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.068+0000 7f7b8ada9700 1 -- 172.21.15.113:0/1223906923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b840f7a10 msgr2=0x7f7b840f7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:28.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.068+0000 7f7b8ada9700 1 --2- 172.21.15.113:0/1223906923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b840f7a10 0x7f7b840f7e30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7b780047f0 tx=0x7f7b78009cb0).stop 2024-04-12T05:11:28.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.068+0000 7f7b8ada9700 1 -- 172.21.15.113:0/1223906923 shutdown_connections 2024-04-12T05:11:28.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.068+0000 7f7b8ada9700 1 --2- 172.21.15.113:0/1223906923 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b840f8cb0 0x7f7b840f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.068+0000 7f7b8ada9700 1 --2- 172.21.15.113:0/1223906923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b840f7a10 0x7f7b840f7e30 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.068+0000 7f7b8ada9700 1 -- 172.21.15.113:0/1223906923 >> 172.21.15.113:0/1223906923 conn(0x7f7b840f3150 msgr2=0x7f7b840f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:28.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.068+0000 7f7b8ada9700 1 -- 172.21.15.113:0/1223906923 shutdown_connections 2024-04-12T05:11:28.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.068+0000 7f7b8ada9700 1 -- 172.21.15.113:0/1223906923 wait complete. 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b8ada9700 1 Processor -- start 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b8ada9700 1 -- start start 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b8ada9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b840f7a10 0x7f7b8406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b8ada9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b840f8cb0 0x7f7b8406b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b8ada9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8406cf80 con 0x7f7b840f7a10 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b8ada9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8406b520 con 0x7f7b840f8cb0 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b88b45700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b840f7a10 0x7f7b8406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b88b45700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b840f7a10 0x7f7b8406c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40690/0 (socket says 172.21.15.113:40690) 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b88b45700 1 -- 172.21.15.113:0/2913093334 learned_addr learned my addr 172.21.15.113:0/2913093334 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b83fff700 1 --2- 172.21.15.113:0/2913093334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b840f8cb0 0x7f7b8406b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b88b45700 1 -- 172.21.15.113:0/2913093334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b840f8cb0 msgr2=0x7f7b8406b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b88b45700 1 --2- 172.21.15.113:0/2913093334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b840f8cb0 0x7f7b8406b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.069+0000 7f7b88b45700 1 -- 172.21.15.113:0/2913093334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b78005040 con 0x7f7b840f7a10 2024-04-12T05:11:28.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.070+0000 7f7b88b45700 1 --2- 172.21.15.113:0/2913093334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b840f7a10 0x7f7b8406c9d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f7b7800b120 tx=0x7f7b78009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:28.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.070+0000 7f7b81ffb700 1 -- 172.21.15.113:0/2913093334 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b7800e030 con 0x7f7b840f7a10 2024-04-12T05:11:28.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.070+0000 7f7b81ffb700 1 -- 172.21.15.113:0/2913093334 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b78007e90 con 0x7f7b840f7a10 2024-04-12T05:11:28.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.070+0000 7f7b8ada9700 1 -- 172.21.15.113:0/2913093334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8406b740 con 0x7f7b840f7a10 2024-04-12T05:11:28.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.070+0000 7f7b81ffb700 1 -- 172.21.15.113:0/2913093334 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b78018070 con 0x7f7b840f7a10 2024-04-12T05:11:28.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.070+0000 7f7b8ada9700 1 -- 172.21.15.113:0/2913093334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8406bca0 con 0x7f7b840f7a10 2024-04-12T05:11:28.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.071+0000 7f7b8ada9700 1 -- 172.21.15.113:0/2913093334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8405fb50 con 0x7f7b840f7a10 2024-04-12T05:11:28.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.071+0000 7f7b81ffb700 1 -- 172.21.15.113:0/2913093334 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 31) v1 ==== 90781+0+0 (secure 0 0 0) 0x7f7b7801d960 con 0x7f7b840f7a10 2024-04-12T05:11:28.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.072+0000 7f7b81ffb700 1 --2- 172.21.15.113:0/2913093334 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b7406c7c0 0x7f7b7406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:28.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.072+0000 7f7b81ffb700 1 -- 172.21.15.113:0/2913093334 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f7b7808cf60 con 0x7f7b840f7a10 2024-04-12T05:11:28.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.072+0000 7f7b83fff700 1 --2- 172.21.15.113:0/2913093334 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b7406c7c0 0x7f7b7406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:11:28.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.072+0000 7f7b83fff700 1 --2- 172.21.15.113:0/2913093334 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b7406c7c0 0x7f7b7406ec80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f7b70004b60 tx=0x7f7b70009040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:28.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.077+0000 7f7b81ffb700 1 -- 172.21.15.113:0/2913093334 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f7b780c0390 con 0x7f7b840f7a10 2024-04-12T05:11:28.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.413+0000 7f7b8ada9700 1 -- 172.21.15.113:0/2913093334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7b841001d0 con 0x7f7b840f7a10 2024-04-12T05:11:28.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.413+0000 7f7b81ffb700 1 -- 172.21.15.113:0/2913093334 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f7b7805ae40 con 0x7f7b840f7a10 2024-04-12T05:11:28.416 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:11:28.416 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:11:28.416 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:11:28.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:11:28.416 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:11:28.416 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:11:28.416 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:11:28.416 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:11:28.416 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:11:28.417 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:11:28.417 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:11:28.417 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:11:28.417 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:11:28.417 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:11:28.417 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 12, 2024-04-12T05:11:28.417 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:11:28.417 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:11:28.417 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:11:28.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.415+0000 7f7b8ada9700 1 -- 172.21.15.113:0/2913093334 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b7406c7c0 msgr2=0x7f7b7406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:28.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.415+0000 7f7b8ada9700 1 --2- 172.21.15.113:0/2913093334 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b7406c7c0 0x7f7b7406ec80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f7b70004b60 tx=0x7f7b70009040).stop 2024-04-12T05:11:28.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.415+0000 7f7b8ada9700 1 -- 172.21.15.113:0/2913093334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b840f7a10 msgr2=0x7f7b8406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:28.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.415+0000 7f7b8ada9700 1 --2- 172.21.15.113:0/2913093334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b840f7a10 0x7f7b8406c9d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f7b7800b120 tx=0x7f7b78009cb0).stop 2024-04-12T05:11:28.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.415+0000 7f7b8ada9700 1 -- 172.21.15.113:0/2913093334 shutdown_connections 2024-04-12T05:11:28.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.415+0000 7f7b8ada9700 1 --2- 172.21.15.113:0/2913093334 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b7406c7c0 0x7f7b7406ec80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.415+0000 7f7b8ada9700 1 --2- 172.21.15.113:0/2913093334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b840f7a10 0x7f7b8406c9d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.415+0000 7f7b8ada9700 1 --2- 172.21.15.113:0/2913093334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b840f8cb0 0x7f7b8406b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.415+0000 7f7b8ada9700 1 -- 172.21.15.113:0/2913093334 >> 172.21.15.113:0/2913093334 conn(0x7f7b840f3150 msgr2=0x7f7b840fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:28.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.415+0000 7f7b8ada9700 1 -- 172.21.15.113:0/2913093334 shutdown_connections 2024-04-12T05:11:28.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.415+0000 7f7b8ada9700 1 -- 172.21.15.113:0/2913093334 wait complete. 2024-04-12T05:11:28.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:26.820301+0000 mgr.smithi113.ctsazg (mgr.14656) 43 : cephadm [INF] Upgrade: Updating node-exporter.smithi113 (1/2) 2024-04-12T05:11:28.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:26.820692+0000 mgr.smithi113.ctsazg (mgr.14656) 44 : cephadm [INF] Deploying daemon node-exporter.smithi113 on smithi113 2024-04-12T05:11:28.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:27.313267+0000 mgr.smithi113.ctsazg (mgr.14656) 45 : audit [DBG] from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:11:28.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:26.820301+0000 mgr.smithi113.ctsazg (mgr.14656) 43 : cephadm [INF] Upgrade: Updating node-exporter.smithi113 (1/2) 2024-04-12T05:11:28.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:26.820692+0000 mgr.smithi113.ctsazg (mgr.14656) 44 : cephadm [INF] Deploying daemon node-exporter.smithi113 on smithi113 2024-04-12T05:11:28.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:27.313267+0000 mgr.smithi113.ctsazg (mgr.14656) 45 : audit [DBG] from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:11:28.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.518+0000 7efd764bf700 1 -- 172.21.15.113:0/1041653591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f6710 msgr2=0x7efd700f6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:28.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.518+0000 7efd764bf700 1 --2- 172.21.15.113:0/1041653591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f6710 0x7efd700f6b30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7efd60004810 tx=0x7efd60009cb0).stop 2024-04-12T05:11:28.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.518+0000 7efd764bf700 1 -- 172.21.15.113:0/1041653591 shutdown_connections 2024-04-12T05:11:28.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.518+0000 7efd764bf700 1 --2- 172.21.15.113:0/1041653591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd700f7020 0x7efd700f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.518+0000 7efd764bf700 1 --2- 172.21.15.113:0/1041653591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f6710 0x7efd700f6b30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.518+0000 7efd764bf700 1 -- 172.21.15.113:0/1041653591 >> 172.21.15.113:0/1041653591 conn(0x7efd700f1f10 msgr2=0x7efd700f4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:28.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.518+0000 7efd764bf700 1 -- 172.21.15.113:0/1041653591 shutdown_connections 2024-04-12T05:11:28.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.518+0000 7efd764bf700 1 -- 172.21.15.113:0/1041653591 wait complete. 2024-04-12T05:11:28.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.519+0000 7efd764bf700 1 Processor -- start 2024-04-12T05:11:28.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.519+0000 7efd764bf700 1 -- start start 2024-04-12T05:11:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.519+0000 7efd764bf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f6710 0x7efd70105f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.519+0000 7efd764bf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd700f7020 0x7efd70106430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.519+0000 7efd764bf700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd701069e0 con 0x7efd700f6710 2024-04-12T05:11:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.519+0000 7efd764bf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd70106b20 con 0x7efd700f7020 2024-04-12T05:11:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd6ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f6710 0x7efd70105f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd6ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f6710 0x7efd70105f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40702/0 (socket says 172.21.15.113:40702) 2024-04-12T05:11:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd6ffff700 1 -- 172.21.15.113:0/3671201531 learned_addr learned my addr 172.21.15.113:0/3671201531 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:11:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd6f7fe700 1 --2- 172.21.15.113:0/3671201531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd700f7020 0x7efd70106430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd6ffff700 1 -- 172.21.15.113:0/3671201531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd700f7020 msgr2=0x7efd70106430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd6ffff700 1 --2- 172.21.15.113:0/3671201531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd700f7020 0x7efd70106430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd6ffff700 1 -- 172.21.15.113:0/3671201531 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd60005040 con 0x7efd700f6710 2024-04-12T05:11:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd6ffff700 1 --2- 172.21.15.113:0/3671201531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f6710 0x7efd70105f40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7efd60000c00 tx=0x7efd60009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd6d7fa700 1 -- 172.21.15.113:0/3671201531 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd60009610 con 0x7efd700f6710 2024-04-12T05:11:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd6d7fa700 1 -- 172.21.15.113:0/3671201531 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd60004290 con 0x7efd700f6710 2024-04-12T05:11:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd6d7fa700 1 -- 172.21.15.113:0/3671201531 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd6000f670 con 0x7efd700f6710 2024-04-12T05:11:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd764bf700 1 -- 172.21.15.113:0/3671201531 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7010b560 con 0x7efd700f6710 2024-04-12T05:11:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.520+0000 7efd764bf700 1 -- 172.21.15.113:0/3671201531 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7010bb20 con 0x7efd700f6710 2024-04-12T05:11:28.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.522+0000 7efd764bf700 1 -- 172.21.15.113:0/3671201531 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd700fde10 con 0x7efd700f6710 2024-04-12T05:11:28.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.522+0000 7efd6d7fa700 1 -- 172.21.15.113:0/3671201531 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 31) v1 ==== 90781+0+0 (secure 0 0 0) 0x7efd60018030 con 0x7efd700f6710 2024-04-12T05:11:28.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.522+0000 7efd6d7fa700 1 --2- 172.21.15.113:0/3671201531 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7efd5806c7c0 0x7efd5806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:28.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.523+0000 7efd6d7fa700 1 -- 172.21.15.113:0/3671201531 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7efd6008ecc0 con 0x7efd700f6710 2024-04-12T05:11:28.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.523+0000 7efd6f7fe700 1 --2- 172.21.15.113:0/3671201531 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7efd5806c7c0 0x7efd5806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:11:28.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.523+0000 7efd6f7fe700 1 --2- 172.21.15.113:0/3671201531 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7efd5806c7c0 0x7efd5806ec80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7efd64004de0 tx=0x7efd6400b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:28.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.528+0000 7efd6d7fa700 1 -- 172.21.15.113:0/3671201531 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7efd6005cba0 con 0x7efd700f6710 2024-04-12T05:11:28.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.754+0000 7efd764bf700 1 -- 172.21.15.113:0/3671201531 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd70059120 con 0x7efd5806c7c0 2024-04-12T05:11:28.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.755+0000 7efd6d7fa700 1 -- 172.21.15.113:0/3671201531 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7efd70059120 con 0x7efd5806c7c0 2024-04-12T05:11:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:11:28.757 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:11:28.758 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:11:28.758 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:11:28.758 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:11:28.758 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr" 2024-04-12T05:11:28.758 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:11:28.758 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "2/2 daemons upgraded", 2024-04-12T05:11:28.758 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-12T05:11:28.758 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:11:28.758 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:11:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.758+0000 7efd764bf700 1 -- 172.21.15.113:0/3671201531 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7efd5806c7c0 msgr2=0x7efd5806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.758+0000 7efd764bf700 1 --2- 172.21.15.113:0/3671201531 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7efd5806c7c0 0x7efd5806ec80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7efd64004de0 tx=0x7efd6400b040).stop 2024-04-12T05:11:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.758+0000 7efd764bf700 1 -- 172.21.15.113:0/3671201531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f6710 msgr2=0x7efd70105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.758+0000 7efd764bf700 1 --2- 172.21.15.113:0/3671201531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f6710 0x7efd70105f40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7efd60000c00 tx=0x7efd60009cb0).stop 2024-04-12T05:11:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.758+0000 7efd764bf700 1 -- 172.21.15.113:0/3671201531 shutdown_connections 2024-04-12T05:11:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.758+0000 7efd764bf700 1 --2- 172.21.15.113:0/3671201531 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7efd5806c7c0 0x7efd5806ec80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.758+0000 7efd764bf700 1 --2- 172.21.15.113:0/3671201531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd700f6710 0x7efd70105f40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.758+0000 7efd764bf700 1 --2- 172.21.15.113:0/3671201531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd700f7020 0x7efd70106430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.758+0000 7efd764bf700 1 -- 172.21.15.113:0/3671201531 >> 172.21.15.113:0/3671201531 conn(0x7efd700f1f10 msgr2=0x7efd700fb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.758+0000 7efd764bf700 1 -- 172.21.15.113:0/3671201531 shutdown_connections 2024-04-12T05:11:28.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:28.758+0000 7efd764bf700 1 -- 172.21.15.113:0/3671201531 wait complete. 2024-04-12T05:11:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:27.646370+0000 mgr.smithi113.ctsazg (mgr.14656) 46 : audit [DBG] from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:11:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:27.951538+0000 mgr.smithi113.ctsazg (mgr.14656) 47 : cluster [DBG] pgmap v19: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:27.960116+0000 mgr.smithi113.ctsazg (mgr.14656) 48 : audit [DBG] from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:11:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:28.414210+0000 mon.smithi113 (mon.0) 827 : audit [DBG] from='client.? 172.21.15.113:0/2913093334' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:27.646370+0000 mgr.smithi113.ctsazg (mgr.14656) 46 : audit [DBG] from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:11:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:27.951538+0000 mgr.smithi113.ctsazg (mgr.14656) 47 : cluster [DBG] pgmap v19: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:27.960116+0000 mgr.smithi113.ctsazg (mgr.14656) 48 : audit [DBG] from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:11:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:28.414210+0000 mon.smithi113 (mon.0) 827 : audit [DBG] from='client.? 172.21.15.113:0/2913093334' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:28.756117+0000 mgr.smithi113.ctsazg (mgr.14656) 49 : audit [DBG] from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:11:30.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:28.756117+0000 mgr.smithi113.ctsazg (mgr.14656) 49 : audit [DBG] from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:11:31.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:29.951902+0000 mgr.smithi113.ctsazg (mgr.14656) 50 : cluster [DBG] pgmap v20: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:31.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:29.951902+0000 mgr.smithi113.ctsazg (mgr.14656) 50 : cluster [DBG] pgmap v20: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:32.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:31.833850+0000 mon.smithi113 (mon.0) 828 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:32.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:31.837981+0000 mon.smithi113 (mon.0) 829 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:32.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:31.952438+0000 mgr.smithi113.ctsazg (mgr.14656) 51 : cluster [DBG] pgmap v21: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:31.833850+0000 mon.smithi113 (mon.0) 828 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:31.837981+0000 mon.smithi113 (mon.0) 829 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:31.952438+0000 mgr.smithi113.ctsazg (mgr.14656) 51 : cluster [DBG] pgmap v21: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:33.414129+0000 mgr.smithi113.ctsazg (mgr.14656) 52 : cephadm [INF] Upgrade: Updating node-exporter.smithi192 (2/2) 2024-04-12T05:11:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:33.414556+0000 mgr.smithi113.ctsazg (mgr.14656) 53 : cephadm [INF] Deploying daemon node-exporter.smithi192 on smithi192 2024-04-12T05:11:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:33.414129+0000 mgr.smithi113.ctsazg (mgr.14656) 52 : cephadm [INF] Upgrade: Updating node-exporter.smithi192 (2/2) 2024-04-12T05:11:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:33.414556+0000 mgr.smithi113.ctsazg (mgr.14656) 53 : cephadm [INF] Deploying daemon node-exporter.smithi192 on smithi192 2024-04-12T05:11:35.119 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:33.953144+0000 mgr.smithi113.ctsazg (mgr.14656) 54 : cluster [DBG] pgmap v22: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:33.953144+0000 mgr.smithi113.ctsazg (mgr.14656) 54 : cluster [DBG] pgmap v22: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:35.953675+0000 mgr.smithi113.ctsazg (mgr.14656) 55 : cluster [DBG] pgmap v23: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:35.953675+0000 mgr.smithi113.ctsazg (mgr.14656) 55 : cluster [DBG] pgmap v23: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:37.954390+0000 mgr.smithi113.ctsazg (mgr.14656) 56 : cluster [DBG] pgmap v24: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:38.085330+0000 mon.smithi113 (mon.0) 830 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:38.090031+0000 mon.smithi113 (mon.0) 831 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:38.139738+0000 mon.smithi113 (mon.0) 832 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:39.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:38.410410+0000 mon.smithi113 (mon.0) 833 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:39.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:38.415158+0000 mon.smithi113 (mon.0) 834 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:37.954390+0000 mgr.smithi113.ctsazg (mgr.14656) 56 : cluster [DBG] pgmap v24: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:38.085330+0000 mon.smithi113 (mon.0) 830 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:38.090031+0000 mon.smithi113 (mon.0) 831 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:38.139738+0000 mon.smithi113 (mon.0) 832 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:38.410410+0000 mon.smithi113 (mon.0) 833 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:39.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:38.415158+0000 mon.smithi113 (mon.0) 834 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:39.954776+0000 mgr.smithi113.ctsazg (mgr.14656) 57 : cluster [DBG] pgmap v25: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:41.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:41.020085+0000 mon.smithi113 (mon.0) 835 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:11:41.439 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:39.954776+0000 mgr.smithi113.ctsazg (mgr.14656) 57 : cluster [DBG] pgmap v25: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:41.439 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:41.020085+0000 mon.smithi113 (mon.0) 835 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:11:43.362 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:41.955035+0000 mgr.smithi113.ctsazg (mgr.14656) 58 : cluster [DBG] pgmap v26: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:43.362 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:42.093495+0000 mon.smithi113 (mon.0) 836 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:43.362 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:42.099296+0000 mon.smithi113 (mon.0) 837 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:43.362 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:42.576449+0000 mon.smithi113 (mon.0) 838 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:43.362 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:42.581695+0000 mon.smithi113 (mon.0) 839 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:43.362 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:42.789580+0000 mon.smithi113 (mon.0) 840 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:43.362 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:42.794144+0000 mon.smithi113 (mon.0) 841 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:41.955035+0000 mgr.smithi113.ctsazg (mgr.14656) 58 : cluster [DBG] pgmap v26: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:42.093495+0000 mon.smithi113 (mon.0) 836 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:42.099296+0000 mon.smithi113 (mon.0) 837 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:42.576449+0000 mon.smithi113 (mon.0) 838 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:42.581695+0000 mon.smithi113 (mon.0) 839 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:42.789580+0000 mon.smithi113 (mon.0) 840 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:42.794144+0000 mon.smithi113 (mon.0) 841 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:44.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:43.214834+0000 mon.smithi113 (mon.0) 842 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:44.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:44 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:43.219995+0000 mon.smithi113 (mon.0) 843 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:44.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:43.214834+0000 mon.smithi113 (mon.0) 842 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:44.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:43.219995+0000 mon.smithi113 (mon.0) 843 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:43.955426+0000 mgr.smithi113.ctsazg (mgr.14656) 59 : cluster [DBG] pgmap v27: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:43.955426+0000 mgr.smithi113.ctsazg (mgr.14656) 59 : cluster [DBG] pgmap v27: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:45.873917+0000 mon.smithi113 (mon.0) 844 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:45.877961+0000 mon.smithi113 (mon.0) 845 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:45.955686+0000 mgr.smithi113.ctsazg (mgr.14656) 60 : cluster [DBG] pgmap v28: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.746281+0000 mon.smithi113 (mon.0) 846 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.750347+0000 mon.smithi113 (mon.0) 847 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.751192+0000 mon.smithi113 (mon.0) 848 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:11:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.751814+0000 mon.smithi113 (mon.0) 849 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:11:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.755689+0000 mon.smithi113 (mon.0) 850 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:45.873917+0000 mon.smithi113 (mon.0) 844 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:45.877961+0000 mon.smithi113 (mon.0) 845 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:45.955686+0000 mgr.smithi113.ctsazg (mgr.14656) 60 : cluster [DBG] pgmap v28: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.746281+0000 mon.smithi113 (mon.0) 846 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.750347+0000 mon.smithi113 (mon.0) 847 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.751192+0000 mon.smithi113 (mon.0) 848 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:11:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.751814+0000 mon.smithi113 (mon.0) 849 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:11:47.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:46 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.755689+0000 mon.smithi113 (mon.0) 850 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:47.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.892942+0000 mon.smithi113 (mon.0) 851 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:47.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.895180+0000 mon.smithi113 (mon.0) 852 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:47.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.897331+0000 mon.smithi113 (mon.0) 853 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:47.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.898263+0000 mon.smithi113 (mon.0) 854 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:47.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.899237+0000 mon.smithi113 (mon.0) 855 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:47.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.900139+0000 mon.smithi113 (mon.0) 856 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:47.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.901005+0000 mon.smithi113 (mon.0) 857 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:47.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.901854+0000 mon.smithi113 (mon.0) 858 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:47.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.902802+0000 mon.smithi113 (mon.0) 859 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:47.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:46.903875+0000 mon.smithi113 (mon.0) 860 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.892942+0000 mon.smithi113 (mon.0) 851 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.895180+0000 mon.smithi113 (mon.0) 852 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.897331+0000 mon.smithi113 (mon.0) 853 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.898263+0000 mon.smithi113 (mon.0) 854 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.899237+0000 mon.smithi113 (mon.0) 855 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.900139+0000 mon.smithi113 (mon.0) 856 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.901005+0000 mon.smithi113 (mon.0) 857 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:48.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.901854+0000 mon.smithi113 (mon.0) 858 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:48.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.902802+0000 mon.smithi113 (mon.0) 859 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:48.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:46.903875+0000 mon.smithi113 (mon.0) 860 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:11:48.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:47.956267+0000 mgr.smithi113.ctsazg (mgr.14656) 61 : cluster [DBG] pgmap v29: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:48.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:48 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:48.422048+0000 mgr.smithi113.ctsazg (mgr.14656) 62 : cephadm [INF] Upgrade: Updating prometheus.smithi192 2024-04-12T05:11:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:47.956267+0000 mgr.smithi113.ctsazg (mgr.14656) 61 : cluster [DBG] pgmap v29: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:48 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:48.422048+0000 mgr.smithi113.ctsazg (mgr.14656) 62 : cephadm [INF] Upgrade: Updating prometheus.smithi192 2024-04-12T05:11:50.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:11:48.638066+0000 mgr.smithi113.ctsazg (mgr.14656) 63 : cephadm [INF] Deploying daemon prometheus.smithi192 on smithi192 2024-04-12T05:11:50.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:11:48.638066+0000 mgr.smithi113.ctsazg (mgr.14656) 63 : cephadm [INF] Deploying daemon prometheus.smithi192 on smithi192 2024-04-12T05:11:51.103 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:49.956739+0000 mgr.smithi113.ctsazg (mgr.14656) 64 : cluster [DBG] pgmap v30: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:50 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:49.956739+0000 mgr.smithi113.ctsazg (mgr.14656) 64 : cluster [DBG] pgmap v30: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:51.957208+0000 mgr.smithi113.ctsazg (mgr.14656) 65 : cluster [DBG] pgmap v31: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:51.957208+0000 mgr.smithi113.ctsazg (mgr.14656) 65 : cluster [DBG] pgmap v31: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:55.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:53.957749+0000 mgr.smithi113.ctsazg (mgr.14656) 66 : cluster [DBG] pgmap v32: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:55.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:54.225538+0000 mon.smithi113 (mon.0) 861 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:55.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:54.232324+0000 mon.smithi113 (mon.0) 862 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:55.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:54.270434+0000 mon.smithi113 (mon.0) 863 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:55.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:53.957749+0000 mgr.smithi113.ctsazg (mgr.14656) 66 : cluster [DBG] pgmap v32: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:55.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:54.225538+0000 mon.smithi113 (mon.0) 861 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:55.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:54.232324+0000 mon.smithi113 (mon.0) 862 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:55.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:54.270434+0000 mon.smithi113 (mon.0) 863 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:11:56.514 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:56.021827+0000 mon.smithi113 (mon.0) 864 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:11:56.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:56.021827+0000 mon.smithi113 (mon.0) 864 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:11:57.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:55.958146+0000 mgr.smithi113.ctsazg (mgr.14656) 67 : cluster [DBG] pgmap v33: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:57.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:55.958146+0000 mgr.smithi113.ctsazg (mgr.14656) 67 : cluster [DBG] pgmap v33: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:58.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.868+0000 7fe4b030b700 1 -- 172.21.15.113:0/1515820419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4a80f66f0 msgr2=0x7fe4a80f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:58.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.868+0000 7fe4b030b700 1 --2- 172.21.15.113:0/1515820419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4a80f66f0 0x7fe4a80f6b10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fe4980047f0 tx=0x7fe498009cb0).stop 2024-04-12T05:11:58.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.868+0000 7fe4b030b700 1 -- 172.21.15.113:0/1515820419 shutdown_connections 2024-04-12T05:11:58.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.868+0000 7fe4b030b700 1 --2- 172.21.15.113:0/1515820419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4a80f7990 0x7fe4a80f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:58.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.868+0000 7fe4b030b700 1 --2- 172.21.15.113:0/1515820419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4a80f66f0 0x7fe4a80f6b10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:58.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.868+0000 7fe4b030b700 1 -- 172.21.15.113:0/1515820419 >> 172.21.15.113:0/1515820419 conn(0x7fe4a80f1e50 msgr2=0x7fe4a80f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:58.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.868+0000 7fe4b030b700 1 -- 172.21.15.113:0/1515820419 shutdown_connections 2024-04-12T05:11:58.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.868+0000 7fe4b030b700 1 -- 172.21.15.113:0/1515820419 wait complete. 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.869+0000 7fe4b030b700 1 Processor -- start 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.869+0000 7fe4b030b700 1 -- start start 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4b030b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4a80f66f0 0x7fe4a8061060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4b030b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4a80f7990 0x7fe4a810ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4b030b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a810d370 con 0x7fe4a80f66f0 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4b030b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a810d4b0 con 0x7fe4a80f7990 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4ae0a7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4a80f66f0 0x7fe4a8061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4ae0a7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4a80f66f0 0x7fe4a8061060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52808/0 (socket says 172.21.15.113:52808) 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4ae0a7700 1 -- 172.21.15.113:0/1239301265 learned_addr learned my addr 172.21.15.113:0/1239301265 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4ad8a6700 1 --2- 172.21.15.113:0/1239301265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4a80f7990 0x7fe4a810ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4ae0a7700 1 -- 172.21.15.113:0/1239301265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4a80f7990 msgr2=0x7fe4a810ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:58.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4ae0a7700 1 --2- 172.21.15.113:0/1239301265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4a80f7990 0x7fe4a810ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:58.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4ae0a7700 1 -- 172.21.15.113:0/1239301265 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe498005040 con 0x7fe4a80f66f0 2024-04-12T05:11:58.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4ae0a7700 1 --2- 172.21.15.113:0/1239301265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4a80f66f0 0x7fe4a8061060 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fe498000c00 tx=0x7fe498009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:58.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe49f7fe700 1 -- 172.21.15.113:0/1239301265 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe498018070 con 0x7fe4a80f66f0 2024-04-12T05:11:58.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4b030b700 1 -- 172.21.15.113:0/1239301265 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a810d6b0 con 0x7fe4a80f66f0 2024-04-12T05:11:58.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe49f7fe700 1 -- 172.21.15.113:0/1239301265 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe498007e90 con 0x7fe4a80f66f0 2024-04-12T05:11:58.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe49f7fe700 1 -- 172.21.15.113:0/1239301265 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe49801c380 con 0x7fe4a80f66f0 2024-04-12T05:11:58.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.870+0000 7fe4b030b700 1 -- 172.21.15.113:0/1239301265 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a810dc10 con 0x7fe4a80f66f0 2024-04-12T05:11:58.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.871+0000 7fe4b030b700 1 -- 172.21.15.113:0/1239301265 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a80feed0 con 0x7fe4a80f66f0 2024-04-12T05:11:58.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.872+0000 7fe49f7fe700 1 -- 172.21.15.113:0/1239301265 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 31) v1 ==== 90781+0+0 (secure 0 0 0) 0x7fe4980123c0 con 0x7fe4a80f66f0 2024-04-12T05:11:58.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.872+0000 7fe49f7fe700 1 --2- 172.21.15.113:0/1239301265 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe49406c7c0 0x7fe49406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:58.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.872+0000 7fe49f7fe700 1 -- 172.21.15.113:0/1239301265 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fe49808c770 con 0x7fe4a80f66f0 2024-04-12T05:11:58.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.872+0000 7fe4ad8a6700 1 --2- 172.21.15.113:0/1239301265 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe49406c7c0 0x7fe49406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:11:58.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.873+0000 7fe4ad8a6700 1 --2- 172.21.15.113:0/1239301265 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe49406c7c0 0x7fe49406ec80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fe4a4004ee0 tx=0x7fe4a400b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:58.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:58.877+0000 7fe49f7fe700 1 -- 172.21.15.113:0/1239301265 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fe49805a650 con 0x7fe4a80f66f0 2024-04-12T05:11:59.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.083+0000 7fe4b030b700 1 -- 172.21.15.113:0/1239301265 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4a8059120 con 0x7fe49406c7c0 2024-04-12T05:11:59.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.084+0000 7fe49f7fe700 1 -- 172.21.15.113:0/1239301265 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fe4a8059120 con 0x7fe49406c7c0 2024-04-12T05:11:59.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.086+0000 7fe4b030b700 1 -- 172.21.15.113:0/1239301265 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe49406c7c0 msgr2=0x7fe49406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.086+0000 7fe4b030b700 1 --2- 172.21.15.113:0/1239301265 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe49406c7c0 0x7fe49406ec80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fe4a4004ee0 tx=0x7fe4a400b040).stop 2024-04-12T05:11:59.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.086+0000 7fe4b030b700 1 -- 172.21.15.113:0/1239301265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4a80f66f0 msgr2=0x7fe4a8061060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.086+0000 7fe4b030b700 1 --2- 172.21.15.113:0/1239301265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4a80f66f0 0x7fe4a8061060 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fe498000c00 tx=0x7fe498009cb0).stop 2024-04-12T05:11:59.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.087+0000 7fe4b030b700 1 -- 172.21.15.113:0/1239301265 shutdown_connections 2024-04-12T05:11:59.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.087+0000 7fe4b030b700 1 --2- 172.21.15.113:0/1239301265 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe49406c7c0 0x7fe49406ec80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.087+0000 7fe4b030b700 1 --2- 172.21.15.113:0/1239301265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4a80f66f0 0x7fe4a8061060 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.087+0000 7fe4b030b700 1 --2- 172.21.15.113:0/1239301265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4a80f7990 0x7fe4a810ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.087+0000 7fe4b030b700 1 -- 172.21.15.113:0/1239301265 >> 172.21.15.113:0/1239301265 conn(0x7fe4a80f1e50 msgr2=0x7fe4a80fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:59.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.087+0000 7fe4b030b700 1 -- 172.21.15.113:0/1239301265 shutdown_connections 2024-04-12T05:11:59.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.087+0000 7fe4b030b700 1 -- 172.21.15.113:0/1239301265 wait complete. 2024-04-12T05:11:59.098 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:11:59.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.186+0000 7fd81c497700 1 -- 172.21.15.113:0/3278975674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8140f7800 msgr2=0x7fd8140f7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.186+0000 7fd81c497700 1 --2- 172.21.15.113:0/3278975674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8140f7800 0x7fd8140f7c20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fd8040047f0 tx=0x7fd804009cb0).stop 2024-04-12T05:11:59.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.186+0000 7fd81c497700 1 -- 172.21.15.113:0/3278975674 shutdown_connections 2024-04-12T05:11:59.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.186+0000 7fd81c497700 1 --2- 172.21.15.113:0/3278975674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8140f8aa0 0x7fd8140f8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:11:59.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.186+0000 7fd81c497700 1 --2- 172.21.15.113:0/3278975674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8140f7800 0x7fd8140f7c20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.186+0000 7fd81c497700 1 -- 172.21.15.113:0/3278975674 >> 172.21.15.113:0/3278975674 conn(0x7fd8140f2f80 msgr2=0x7fd8140f53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:59.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.186+0000 7fd81c497700 1 -- 172.21.15.113:0/3278975674 shutdown_connections 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.186+0000 7fd81c497700 1 -- 172.21.15.113:0/3278975674 wait complete. 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd81c497700 1 Processor -- start 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd81c497700 1 -- start start 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd81c497700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8140f7800 0x7fd814108000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd81c497700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8140f8aa0 0x7fd8141084f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd81c497700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd814108b30 con 0x7fd8140f7800 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd81c497700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd814108c70 con 0x7fd8140f8aa0 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd81a233700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8140f7800 0x7fd814108000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd81a233700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8140f7800 0x7fd814108000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52822/0 (socket says 172.21.15.113:52822) 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd81a233700 1 -- 172.21.15.113:0/619262139 learned_addr learned my addr 172.21.15.113:0/619262139 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd819a32700 1 --2- 172.21.15.113:0/619262139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8140f8aa0 0x7fd8141084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd81a233700 1 -- 172.21.15.113:0/619262139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8140f8aa0 msgr2=0x7fd8141084f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.187+0000 7fd81a233700 1 --2- 172.21.15.113:0/619262139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8140f8aa0 0x7fd8141084f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.188+0000 7fd81a233700 1 -- 172.21.15.113:0/619262139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd804005040 con 0x7fd8140f7800 2024-04-12T05:11:59.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.188+0000 7fd81a233700 1 --2- 172.21.15.113:0/619262139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8140f7800 0x7fd814108000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fd80400b120 tx=0x7fd804009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:59.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.188+0000 7fd80b7fe700 1 -- 172.21.15.113:0/619262139 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80400e030 con 0x7fd8140f7800 2024-04-12T05:11:59.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.188+0000 7fd80b7fe700 1 -- 172.21.15.113:0/619262139 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd804007e90 con 0x7fd8140f7800 2024-04-12T05:11:59.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.188+0000 7fd81c497700 1 -- 172.21.15.113:0/619262139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81410d6b0 con 0x7fd8140f7800 2024-04-12T05:11:59.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.188+0000 7fd80b7fe700 1 -- 172.21.15.113:0/619262139 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd804018070 con 0x7fd8140f7800 2024-04-12T05:11:59.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.188+0000 7fd81c497700 1 -- 172.21.15.113:0/619262139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81410db90 con 0x7fd8140f7800 2024-04-12T05:11:59.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.189+0000 7fd81c497700 1 -- 172.21.15.113:0/619262139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd814100040 con 0x7fd8140f7800 2024-04-12T05:11:59.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.190+0000 7fd80b7fe700 1 -- 172.21.15.113:0/619262139 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 31) v1 ==== 90781+0+0 (secure 0 0 0) 0x7fd80401d9d0 con 0x7fd8140f7800 2024-04-12T05:11:59.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.190+0000 7fd80b7fe700 1 --2- 172.21.15.113:0/619262139 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fd80006c7a0 0x7fd80006ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:59.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.190+0000 7fd80b7fe700 1 -- 172.21.15.113:0/619262139 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fd80408d2b0 con 0x7fd8140f7800 2024-04-12T05:11:59.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.190+0000 7fd819a32700 1 --2- 172.21.15.113:0/619262139 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fd80006c7a0 0x7fd80006ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:11:59.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.191+0000 7fd819a32700 1 --2- 172.21.15.113:0/619262139 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fd80006c7a0 0x7fd80006ec60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fd81000b910 tx=0x7fd810008040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:59.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.194+0000 7fd80b7fe700 1 -- 172.21.15.113:0/619262139 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fd80405b190 con 0x7fd8140f7800 2024-04-12T05:11:59.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.399+0000 7fd81c497700 1 -- 172.21.15.113:0/619262139 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd814059120 con 0x7fd80006c7a0 2024-04-12T05:11:59.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.400+0000 7fd80b7fe700 1 -- 172.21.15.113:0/619262139 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fd814059120 con 0x7fd80006c7a0 2024-04-12T05:11:59.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.402+0000 7fd81c497700 1 -- 172.21.15.113:0/619262139 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fd80006c7a0 msgr2=0x7fd80006ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.402+0000 7fd81c497700 1 --2- 172.21.15.113:0/619262139 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fd80006c7a0 0x7fd80006ec60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fd81000b910 tx=0x7fd810008040).stop 2024-04-12T05:11:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.402+0000 7fd81c497700 1 -- 172.21.15.113:0/619262139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8140f7800 msgr2=0x7fd814108000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.402+0000 7fd81c497700 1 --2- 172.21.15.113:0/619262139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8140f7800 0x7fd814108000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fd80400b120 tx=0x7fd804009cb0).stop 2024-04-12T05:11:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.402+0000 7fd81c497700 1 -- 172.21.15.113:0/619262139 shutdown_connections 2024-04-12T05:11:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.402+0000 7fd81c497700 1 --2- 172.21.15.113:0/619262139 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fd80006c7a0 0x7fd80006ec60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.402+0000 7fd81c497700 1 --2- 172.21.15.113:0/619262139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8140f7800 0x7fd814108000 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.402+0000 7fd81c497700 1 --2- 172.21.15.113:0/619262139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8140f8aa0 0x7fd8141084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.402+0000 7fd81c497700 1 -- 172.21.15.113:0/619262139 >> 172.21.15.113:0/619262139 conn(0x7fd8140f2f80 msgr2=0x7fd8140fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.403+0000 7fd81c497700 1 -- 172.21.15.113:0/619262139 shutdown_connections 2024-04-12T05:11:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.403+0000 7fd81c497700 1 -- 172.21.15.113:0/619262139 wait complete. 2024-04-12T05:11:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:57.958944+0000 mgr.smithi113.ctsazg (mgr.14656) 68 : cluster [DBG] pgmap v34: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:58.050059+0000 mon.smithi113 (mon.0) 865 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:58.056783+0000 mon.smithi113 (mon.0) 866 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:58.694411+0000 mon.smithi113 (mon.0) 867 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:11:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:58.699258+0000 mon.smithi113 (mon.0) 868 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:57.958944+0000 mgr.smithi113.ctsazg (mgr.14656) 68 : cluster [DBG] pgmap v34: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:11:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:58.050059+0000 mon.smithi113 (mon.0) 865 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:58.056783+0000 mon.smithi113 (mon.0) 866 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:58.694411+0000 mon.smithi113 (mon.0) 867 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:11:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:58.699258+0000 mon.smithi113 (mon.0) 868 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:11:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.500+0000 7fb07c928700 1 -- 172.21.15.113:0/2468087865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0780f8cb0 msgr2=0x7fb0780f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.500+0000 7fb07c928700 1 --2- 172.21.15.113:0/2468087865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0780f8cb0 0x7fb0780f9110 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fb0680047f0 tx=0x7fb068009cb0).stop 2024-04-12T05:11:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.501+0000 7fb07c928700 1 -- 172.21.15.113:0/2468087865 shutdown_connections 2024-04-12T05:11:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.501+0000 7fb07c928700 1 --2- 172.21.15.113:0/2468087865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0780f8cb0 0x7fb0780f9110 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.501+0000 7fb07c928700 1 --2- 172.21.15.113:0/2468087865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0780f7a10 0x7fb0780f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.501+0000 7fb07c928700 1 -- 172.21.15.113:0/2468087865 >> 172.21.15.113:0/2468087865 conn(0x7fb0780f3150 msgr2=0x7fb0780f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.501+0000 7fb07c928700 1 -- 172.21.15.113:0/2468087865 shutdown_connections 2024-04-12T05:11:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.501+0000 7fb07c928700 1 -- 172.21.15.113:0/2468087865 wait complete. 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.501+0000 7fb07c928700 1 Processor -- start 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.501+0000 7fb07c928700 1 -- start start 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb07c928700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0780f7a10 0x7fb07806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb07c928700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0780f8cb0 0x7fb07806b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb07c928700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07806cf80 con 0x7fb0780f8cb0 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb07c928700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07806b520 con 0x7fb0780f7a10 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb075d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0780f8cb0 0x7fb07806b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb075d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0780f8cb0 0x7fb07806b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52840/0 (socket says 172.21.15.113:52840) 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb075d9b700 1 -- 172.21.15.113:0/3995397980 learned_addr learned my addr 172.21.15.113:0/3995397980 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb07659c700 1 --2- 172.21.15.113:0/3995397980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0780f7a10 0x7fb07806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb075d9b700 1 -- 172.21.15.113:0/3995397980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0780f7a10 msgr2=0x7fb07806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb075d9b700 1 --2- 172.21.15.113:0/3995397980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0780f7a10 0x7fb07806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb075d9b700 1 -- 172.21.15.113:0/3995397980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb068005040 con 0x7fb0780f8cb0 2024-04-12T05:11:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb075d9b700 1 --2- 172.21.15.113:0/3995397980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0780f8cb0 0x7fb07806b030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fb068013560 tx=0x7fb068009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb0677fe700 1 -- 172.21.15.113:0/3995397980 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb068018070 con 0x7fb0780f8cb0 2024-04-12T05:11:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb0677fe700 1 -- 172.21.15.113:0/3995397980 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb068007e90 con 0x7fb0780f8cb0 2024-04-12T05:11:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb0677fe700 1 -- 172.21.15.113:0/3995397980 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb06801c380 con 0x7fb0780f8cb0 2024-04-12T05:11:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb07c928700 1 -- 172.21.15.113:0/3995397980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07806b740 con 0x7fb0780f8cb0 2024-04-12T05:11:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.502+0000 7fb07c928700 1 -- 172.21.15.113:0/3995397980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07806bca0 con 0x7fb0780f8cb0 2024-04-12T05:11:59.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.503+0000 7fb07c928700 1 -- 172.21.15.113:0/3995397980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb078047540 con 0x7fb0780f8cb0 2024-04-12T05:11:59.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.504+0000 7fb0677fe700 1 -- 172.21.15.113:0/3995397980 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 31) v1 ==== 90781+0+0 (secure 0 0 0) 0x7fb0680123c0 con 0x7fb0780f8cb0 2024-04-12T05:11:59.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.504+0000 7fb0677fe700 1 --2- 172.21.15.113:0/3995397980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb06006c7c0 0x7fb06006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:59.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.504+0000 7fb0677fe700 1 -- 172.21.15.113:0/3995397980 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fb06808c780 con 0x7fb0780f8cb0 2024-04-12T05:11:59.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.504+0000 7fb07659c700 1 --2- 172.21.15.113:0/3995397980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb06006c7c0 0x7fb06006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:11:59.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.505+0000 7fb07659c700 1 --2- 172.21.15.113:0/3995397980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb06006c7c0 0x7fb06006ec80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fb0780f8b10 tx=0x7fb06c00b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:59.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.509+0000 7fb0677fe700 1 -- 172.21.15.113:0/3995397980 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fb06805a660 con 0x7fb0780f8cb0 2024-04-12T05:11:59.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.704+0000 7fb07c928700 1 -- 172.21.15.113:0/3995397980 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0781020e0 con 0x7fb06006c7c0 2024-04-12T05:11:59.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.713+0000 7fb0677fe700 1 -- 172.21.15.113:0/3995397980 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fb0781020e0 con 0x7fb06006c7c0 2024-04-12T05:11:59.715 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (2m) 17s ago 11m 17.5M - 0.23.0 ba2b418f427c b2262b7eedd3 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (10m) 17s ago 10m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (10m) 1s ago 10m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (119s) 17s ago 10m 43.5M - 8.3.5 dad864ee21e9 d33ad1514a4c 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (5m) 17s ago 5m 121M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (5m) 17s ago 5m 16.8M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (5m) 1s ago 5m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (5m) 1s ago 5m 14.4M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (74s) 17s ago 12m 491M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (43s) 1s ago 10m 446M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (12m) 17s ago 12m 56.1M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (10m) 1s ago 10m 49.2M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (28s) 17s ago 10m 8028k - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (22s) 1s ago 10m 8456k - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (9m) 17s ago 9m 226M 1910M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:11:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (9m) 17s ago 9m 240M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:11:59.717 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (8m) 17s ago 8m 202M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:11:59.717 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (8m) 1s ago 8m 263M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:11:59.717 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (7m) 1s ago 7m 233M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:11:59.717 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (7m) 1s ago 7m 226M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:11:59.717 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (6s) 1s ago 9m 38.0M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:11:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.715+0000 7fb07c928700 1 -- 172.21.15.113:0/3995397980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb06006c7c0 msgr2=0x7fb06006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.715+0000 7fb07c928700 1 --2- 172.21.15.113:0/3995397980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb06006c7c0 0x7fb06006ec80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fb0780f8b10 tx=0x7fb06c00b040).stop 2024-04-12T05:11:59.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.715+0000 7fb07c928700 1 -- 172.21.15.113:0/3995397980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0780f8cb0 msgr2=0x7fb07806b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.715+0000 7fb07c928700 1 --2- 172.21.15.113:0/3995397980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0780f8cb0 0x7fb07806b030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fb068013560 tx=0x7fb068009cb0).stop 2024-04-12T05:11:59.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.715+0000 7fb07c928700 1 -- 172.21.15.113:0/3995397980 shutdown_connections 2024-04-12T05:11:59.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.715+0000 7fb07c928700 1 --2- 172.21.15.113:0/3995397980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb06006c7c0 0x7fb06006ec80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.715+0000 7fb07c928700 1 --2- 172.21.15.113:0/3995397980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0780f7a10 0x7fb07806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.715+0000 7fb07c928700 1 --2- 172.21.15.113:0/3995397980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0780f8cb0 0x7fb07806b030 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.716+0000 7fb07c928700 1 -- 172.21.15.113:0/3995397980 >> 172.21.15.113:0/3995397980 conn(0x7fb0780f3150 msgr2=0x7fb0780fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:59.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.716+0000 7fb07c928700 1 -- 172.21.15.113:0/3995397980 shutdown_connections 2024-04-12T05:11:59.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.716+0000 7fb07c928700 1 -- 172.21.15.113:0/3995397980 wait complete. 2024-04-12T05:11:59.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 -- 172.21.15.113:0/779706287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09440f66f0 msgr2=0x7f09440f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 --2- 172.21.15.113:0/779706287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09440f66f0 0x7f09440f6b10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f09340047f0 tx=0x7f0934009cb0).stop 2024-04-12T05:11:59.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 -- 172.21.15.113:0/779706287 shutdown_connections 2024-04-12T05:11:59.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 --2- 172.21.15.113:0/779706287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09440f7990 0x7f09440f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:11:59.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 --2- 172.21.15.113:0/779706287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09440f66f0 0x7f09440f6b10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 -- 172.21.15.113:0/779706287 >> 172.21.15.113:0/779706287 conn(0x7f09440f1e50 msgr2=0x7f09440f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:11:59.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 -- 172.21.15.113:0/779706287 shutdown_connections 2024-04-12T05:11:59.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 -- 172.21.15.113:0/779706287 wait complete. 2024-04-12T05:11:59.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 Processor -- start 2024-04-12T05:11:59.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 -- start start 2024-04-12T05:11:59.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09440f66f0 0x7f0944061040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:59.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09440f7990 0x7f094410ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:59.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094410d2f0 con 0x7f09440f66f0 2024-04-12T05:11:59.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.812+0000 7f094c43b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094410d430 con 0x7f09440f7990 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f094a1d7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09440f66f0 0x7f0944061040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f094a1d7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09440f66f0 0x7f0944061040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52856/0 (socket says 172.21.15.113:52856) 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f094a1d7700 1 -- 172.21.15.113:0/1347875709 learned_addr learned my addr 172.21.15.113:0/1347875709 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f09499d6700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09440f7990 0x7f094410ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f094a1d7700 1 -- 172.21.15.113:0/1347875709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09440f7990 msgr2=0x7f094410ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f094a1d7700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09440f7990 0x7f094410ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f094a1d7700 1 -- 172.21.15.113:0/1347875709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0934005040 con 0x7f09440f66f0 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f09499d6700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09440f7990 0x7f094410ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f094a1d7700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09440f66f0 0x7f0944061040 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f093400b120 tx=0x7f0934009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f093b7fe700 1 -- 172.21.15.113:0/1347875709 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f093400e030 con 0x7f09440f66f0 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f093b7fe700 1 -- 172.21.15.113:0/1347875709 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0934007e90 con 0x7f09440f66f0 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f093b7fe700 1 -- 172.21.15.113:0/1347875709 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0934018070 con 0x7f09440f66f0 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f094c43b700 1 -- 172.21.15.113:0/1347875709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094410d6b0 con 0x7f09440f66f0 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.813+0000 7f094c43b700 1 -- 172.21.15.113:0/1347875709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094410db40 con 0x7f09440f66f0 2024-04-12T05:11:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.814+0000 7f094c43b700 1 -- 172.21.15.113:0/1347875709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09440fef30 con 0x7f09440f66f0 2024-04-12T05:11:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.815+0000 7f093b7fe700 1 -- 172.21.15.113:0/1347875709 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 31) v1 ==== 90781+0+0 (secure 0 0 0) 0x7f093401d960 con 0x7f09440f66f0 2024-04-12T05:11:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.815+0000 7f093b7fe700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f093006c7c0 0x7f093006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:11:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.815+0000 7f093b7fe700 1 -- 172.21.15.113:0/1347875709 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f093408cf80 con 0x7f09440f66f0 2024-04-12T05:11:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.815+0000 7f09499d6700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f093006c7c0 0x7f093006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:11:59.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.816+0000 7f09499d6700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f093006c7c0 0x7f093006ec80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f0940009990 tx=0x7f0940008040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:11:59.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:11:59.820+0000 7f093b7fe700 1 -- 172.21.15.113:0/1347875709 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f093405ae60 con 0x7f09440f66f0 2024-04-12T05:12:00.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.088+0000 7f094c43b700 1 -- 172.21.15.113:0/1347875709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0944100da0 con 0x7f09440f66f0 2024-04-12T05:12:00.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.088+0000 7f093b7fe700 1 -- 172.21.15.113:0/1347875709 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f093405a9f0 con 0x7f09440f66f0 2024-04-12T05:12:00.090 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:12:00.091 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 12, 2024-04-12T05:12:00.092 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:12:00.092 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:12:00.092 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:12:00.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.090+0000 7f094c43b700 1 -- 172.21.15.113:0/1347875709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f093006c7c0 msgr2=0x7f093006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:00.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.090+0000 7f094c43b700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f093006c7c0 0x7f093006ec80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f0940009990 tx=0x7f0940008040).stop 2024-04-12T05:12:00.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.090+0000 7f094c43b700 1 -- 172.21.15.113:0/1347875709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09440f66f0 msgr2=0x7f0944061040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:00.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.090+0000 7f094c43b700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09440f66f0 0x7f0944061040 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f093400b120 tx=0x7f0934009cb0).stop 2024-04-12T05:12:00.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.090+0000 7f094c43b700 1 -- 172.21.15.113:0/1347875709 shutdown_connections 2024-04-12T05:12:00.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.090+0000 7f094c43b700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f093006c7c0 0x7f093006ec80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:00.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.090+0000 7f094c43b700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09440f66f0 0x7f0944061040 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:00.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.090+0000 7f094c43b700 1 --2- 172.21.15.113:0/1347875709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09440f7990 0x7f094410ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:00.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.090+0000 7f094c43b700 1 -- 172.21.15.113:0/1347875709 >> 172.21.15.113:0/1347875709 conn(0x7f09440f1e50 msgr2=0x7f09440fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:00.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.090+0000 7f094c43b700 1 -- 172.21.15.113:0/1347875709 shutdown_connections 2024-04-12T05:12:00.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.090+0000 7f094c43b700 1 -- 172.21.15.113:0/1347875709 wait complete. 2024-04-12T05:12:00.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.189+0000 7f3c7510e700 1 -- 172.21.15.113:0/717528538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f66f0 msgr2=0x7f3c700f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:00.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.189+0000 7f3c7510e700 1 --2- 172.21.15.113:0/717528538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f66f0 0x7f3c700f6b10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3c600047f0 tx=0x7f3c60009cb0).stop 2024-04-12T05:12:00.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.189+0000 7f3c7510e700 1 -- 172.21.15.113:0/717528538 shutdown_connections 2024-04-12T05:12:00.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.189+0000 7f3c7510e700 1 --2- 172.21.15.113:0/717528538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f7990 0x7f3c700f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:00.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.189+0000 7f3c7510e700 1 --2- 172.21.15.113:0/717528538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f66f0 0x7f3c700f6b10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:00.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.189+0000 7f3c7510e700 1 -- 172.21.15.113:0/717528538 >> 172.21.15.113:0/717528538 conn(0x7f3c700f1e50 msgr2=0x7f3c700f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:00.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.189+0000 7f3c7510e700 1 -- 172.21.15.113:0/717528538 shutdown_connections 2024-04-12T05:12:00.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.189+0000 7f3c7510e700 1 -- 172.21.15.113:0/717528538 wait complete. 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.190+0000 7f3c7510e700 1 Processor -- start 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.190+0000 7f3c7510e700 1 -- start start 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.190+0000 7f3c7510e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f66f0 0x7f3c70106f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.190+0000 7f3c7510e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f7990 0x7f3c70107480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.190+0000 7f3c7510e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70107a30 con 0x7f3c700f7990 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.190+0000 7f3c7510e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70107b70 con 0x7f3c700f66f0 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c6e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f7990 0x7f3c70107480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c6e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f7990 0x7f3c70107480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52874/0 (socket says 172.21.15.113:52874) 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c6e59c700 1 -- 172.21.15.113:0/624523674 learned_addr learned my addr 172.21.15.113:0/624523674 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c6ed9d700 1 --2- 172.21.15.113:0/624523674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f66f0 0x7f3c70106f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c6e59c700 1 -- 172.21.15.113:0/624523674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f66f0 msgr2=0x7f3c70106f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c6e59c700 1 --2- 172.21.15.113:0/624523674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f66f0 0x7f3c70106f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c6e59c700 1 -- 172.21.15.113:0/624523674 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c60005040 con 0x7f3c700f7990 2024-04-12T05:12:00.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c6e59c700 1 --2- 172.21.15.113:0/624523674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f7990 0x7f3c70107480 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f3c6400e910 tx=0x7f3c6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:00.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c57fff700 1 -- 172.21.15.113:0/624523674 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c64007b40 con 0x7f3c700f7990 2024-04-12T05:12:00.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c57fff700 1 -- 172.21.15.113:0/624523674 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c6400f040 con 0x7f3c700f7990 2024-04-12T05:12:00.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c7510e700 1 -- 172.21.15.113:0/624523674 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7010c610 con 0x7f3c700f7990 2024-04-12T05:12:00.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c57fff700 1 -- 172.21.15.113:0/624523674 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c64014920 con 0x7f3c700f7990 2024-04-12T05:12:00.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.191+0000 7f3c7510e700 1 -- 172.21.15.113:0/624523674 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7010cbd0 con 0x7f3c700f7990 2024-04-12T05:12:00.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.192+0000 7f3c7510e700 1 -- 172.21.15.113:0/624523674 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c700feed0 con 0x7f3c700f7990 2024-04-12T05:12:00.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.193+0000 7f3c57fff700 1 -- 172.21.15.113:0/624523674 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 31) v1 ==== 90781+0+0 (secure 0 0 0) 0x7f3c64007ca0 con 0x7f3c700f7990 2024-04-12T05:12:00.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.193+0000 7f3c57fff700 1 --2- 172.21.15.113:0/624523674 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f3c5806c7c0 0x7f3c5806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:00.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.193+0000 7f3c57fff700 1 -- 172.21.15.113:0/624523674 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f3c6408b400 con 0x7f3c700f7990 2024-04-12T05:12:00.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.193+0000 7f3c6ed9d700 1 --2- 172.21.15.113:0/624523674 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f3c5806c7c0 0x7f3c5806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:12:00.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.193+0000 7f3c6ed9d700 1 --2- 172.21.15.113:0/624523674 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f3c5806c7c0 0x7f3c5806ec80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3c600047c0 tx=0x7f3c60009cb0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:00.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:59.085218+0000 mgr.smithi113.ctsazg (mgr.14656) 69 : audit [DBG] from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:00.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:59.401277+0000 mgr.smithi113.ctsazg (mgr.14656) 70 : audit [DBG] from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:00.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.198+0000 7f3c57fff700 1 -- 172.21.15.113:0/624523674 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f3c640592e0 con 0x7f3c700f7990 2024-04-12T05:12:00.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.403+0000 7f3c7510e700 1 -- 172.21.15.113:0/624523674 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c70059120 con 0x7f3c5806c7c0 2024-04-12T05:12:00.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.404+0000 7f3c57fff700 1 -- 172.21.15.113:0/624523674 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f3c70059120 con 0x7f3c5806c7c0 2024-04-12T05:12:00.406 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:12:00.406 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:12:00.406 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:12:00.406 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:12:00.406 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:12:00.406 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr" 2024-04-12T05:12:00.406 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:12:00.406 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "2/2 daemons upgraded", 2024-04-12T05:12:00.407 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-12T05:12:00.407 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:12:00.407 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:12:00.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.406+0000 7f3c7510e700 1 -- 172.21.15.113:0/624523674 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f3c5806c7c0 msgr2=0x7f3c5806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:00.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.406+0000 7f3c7510e700 1 --2- 172.21.15.113:0/624523674 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f3c5806c7c0 0x7f3c5806ec80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3c600047c0 tx=0x7f3c60009cb0).stop 2024-04-12T05:12:00.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.406+0000 7f3c7510e700 1 -- 172.21.15.113:0/624523674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f7990 msgr2=0x7f3c70107480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:00.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.406+0000 7f3c7510e700 1 --2- 172.21.15.113:0/624523674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f7990 0x7f3c70107480 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f3c6400e910 tx=0x7f3c6400b040).stop 2024-04-12T05:12:00.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.406+0000 7f3c7510e700 1 -- 172.21.15.113:0/624523674 shutdown_connections 2024-04-12T05:12:00.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.406+0000 7f3c7510e700 1 --2- 172.21.15.113:0/624523674 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f3c5806c7c0 0x7f3c5806ec80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:00.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.406+0000 7f3c7510e700 1 --2- 172.21.15.113:0/624523674 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f66f0 0x7f3c70106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:00.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.406+0000 7f3c7510e700 1 --2- 172.21.15.113:0/624523674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f7990 0x7f3c70107480 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:00.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.406+0000 7f3c7510e700 1 -- 172.21.15.113:0/624523674 >> 172.21.15.113:0/624523674 conn(0x7f3c700f1e50 msgr2=0x7f3c700fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:00.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.406+0000 7f3c7510e700 1 -- 172.21.15.113:0/624523674 shutdown_connections 2024-04-12T05:12:00.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:00.406+0000 7f3c7510e700 1 -- 172.21.15.113:0/624523674 wait complete. 2024-04-12T05:12:00.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:59.085218+0000 mgr.smithi113.ctsazg (mgr.14656) 69 : audit [DBG] from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:00.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:59.401277+0000 mgr.smithi113.ctsazg (mgr.14656) 70 : audit [DBG] from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:11:59.705590+0000 mgr.smithi113.ctsazg (mgr.14656) 71 : audit [DBG] from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:11:59.959284+0000 mgr.smithi113.ctsazg (mgr.14656) 72 : cluster [DBG] pgmap v35: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:00.089205+0000 mon.smithi113 (mon.0) 869 : audit [DBG] from='client.? 172.21.15.113:0/1347875709' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:00.404662+0000 mgr.smithi113.ctsazg (mgr.14656) 73 : audit [DBG] from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:11:59.705590+0000 mgr.smithi113.ctsazg (mgr.14656) 71 : audit [DBG] from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:11:59.959284+0000 mgr.smithi113.ctsazg (mgr.14656) 72 : cluster [DBG] pgmap v35: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:01.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:00.089205+0000 mon.smithi113 (mon.0) 869 : audit [DBG] from='client.? 172.21.15.113:0/1347875709' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:01.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:00.404662+0000 mgr.smithi113.ctsazg (mgr.14656) 73 : audit [DBG] from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:01.959975+0000 mgr.smithi113.ctsazg (mgr.14656) 74 : cluster [DBG] pgmap v36: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.026766+0000 mon.smithi113 (mon.0) 870 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.033497+0000 mon.smithi113 (mon.0) 871 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.034420+0000 mon.smithi113 (mon.0) 872 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:12:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.035241+0000 mon.smithi113 (mon.0) 873 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:12:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.041880+0000 mon.smithi113 (mon.0) 874 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:02.067590+0000 mon.smithi113 (mon.0) 875 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.179165+0000 mon.smithi113 (mon.0) 876 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.179681+0000 mgr.smithi113.ctsazg (mgr.14656) 75 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.185313+0000 mon.smithi113 (mon.0) 877 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.187928+0000 mon.smithi113 (mon.0) 878 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.190325+0000 mon.smithi113 (mon.0) 879 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.191291+0000 mon.smithi113 (mon.0) 880 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.192245+0000 mon.smithi113 (mon.0) 881 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.193210+0000 mon.smithi113 (mon.0) 882 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.194179+0000 mon.smithi113 (mon.0) 883 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.195138+0000 mon.smithi113 (mon.0) 884 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.196167+0000 mon.smithi113 (mon.0) 885 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.197430+0000 mon.smithi113 (mon.0) 886 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:02.198583+0000 mon.smithi113 (mon.0) 887 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:01.959975+0000 mgr.smithi113.ctsazg (mgr.14656) 74 : cluster [DBG] pgmap v36: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.026766+0000 mon.smithi113 (mon.0) 870 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.033497+0000 mon.smithi113 (mon.0) 871 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.034420+0000 mon.smithi113 (mon.0) 872 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.035241+0000 mon.smithi113 (mon.0) 873 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.041880+0000 mon.smithi113 (mon.0) 874 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:02.067590+0000 mon.smithi113 (mon.0) 875 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.179165+0000 mon.smithi113 (mon.0) 876 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.179681+0000 mgr.smithi113.ctsazg (mgr.14656) 75 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.185313+0000 mon.smithi113 (mon.0) 877 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.187928+0000 mon.smithi113 (mon.0) 878 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.190325+0000 mon.smithi113 (mon.0) 879 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.191291+0000 mon.smithi113 (mon.0) 880 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.192245+0000 mon.smithi113 (mon.0) 881 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.193210+0000 mon.smithi113 (mon.0) 882 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.194179+0000 mon.smithi113 (mon.0) 883 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.195138+0000 mon.smithi113 (mon.0) 884 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.196167+0000 mon.smithi113 (mon.0) 885 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.197430+0000 mon.smithi113 (mon.0) 886 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:03.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:02.198583+0000 mon.smithi113 (mon.0) 887 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:12:03.896809+0000 mgr.smithi113.ctsazg (mgr.14656) 76 : cephadm [INF] Upgrade: Updating alertmanager.smithi113 2024-04-12T05:12:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:12:03.902917+0000 mgr.smithi113.ctsazg (mgr.14656) 77 : cephadm [INF] Deploying daemon alertmanager.smithi113 on smithi113 2024-04-12T05:12:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:03.960286+0000 mgr.smithi113.ctsazg (mgr.14656) 78 : cluster [DBG] pgmap v37: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:12:03.896809+0000 mgr.smithi113.ctsazg (mgr.14656) 76 : cephadm [INF] Upgrade: Updating alertmanager.smithi113 2024-04-12T05:12:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:12:03.902917+0000 mgr.smithi113.ctsazg (mgr.14656) 77 : cephadm [INF] Deploying daemon alertmanager.smithi113 on smithi113 2024-04-12T05:12:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:03.960286+0000 mgr.smithi113.ctsazg (mgr.14656) 78 : cluster [DBG] pgmap v37: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:07.403 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:05.960698+0000 mgr.smithi113.ctsazg (mgr.14656) 79 : cluster [DBG] pgmap v38: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:05.960698+0000 mgr.smithi113.ctsazg (mgr.14656) 79 : cluster [DBG] pgmap v38: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:09.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:07.961348+0000 mgr.smithi113.ctsazg (mgr.14656) 80 : cluster [DBG] pgmap v39: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:07.961348+0000 mgr.smithi113.ctsazg (mgr.14656) 80 : cluster [DBG] pgmap v39: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:10.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:09.660033+0000 mon.smithi113 (mon.0) 888 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:10.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:09.666130+0000 mon.smithi113 (mon.0) 889 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:10.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:09.677561+0000 mon.smithi113 (mon.0) 890 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:10.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:10 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:09.961731+0000 mgr.smithi113.ctsazg (mgr.14656) 81 : cluster [DBG] pgmap v40: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:10.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:09.660033+0000 mon.smithi113 (mon.0) 888 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:10.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:09.666130+0000 mon.smithi113 (mon.0) 889 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:10.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:09.677561+0000 mon.smithi113 (mon.0) 890 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:10.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:09.961731+0000 mgr.smithi113.ctsazg (mgr.14656) 81 : cluster [DBG] pgmap v40: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:11.904 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:11.022047+0000 mon.smithi113 (mon.0) 891 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:12:11.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:11.022047+0000 mon.smithi113 (mon.0) 891 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:12:12.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:11.962081+0000 mgr.smithi113.ctsazg (mgr.14656) 82 : cluster [DBG] pgmap v41: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:13.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:11.962081+0000 mgr.smithi113.ctsazg (mgr.14656) 82 : cluster [DBG] pgmap v41: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:15.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:13.705335+0000 mon.smithi113 (mon.0) 892 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:15.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:13.709131+0000 mon.smithi113 (mon.0) 893 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:15.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:13.962413+0000 mgr.smithi113.ctsazg (mgr.14656) 83 : cluster [DBG] pgmap v42: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:15.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:14.107527+0000 mon.smithi113 (mon.0) 894 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:15.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:14 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:14.112669+0000 mon.smithi113 (mon.0) 895 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:15.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:13.705335+0000 mon.smithi113 (mon.0) 892 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:15.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:13.709131+0000 mon.smithi113 (mon.0) 893 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:15.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:13.962413+0000 mgr.smithi113.ctsazg (mgr.14656) 83 : cluster [DBG] pgmap v42: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:15.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:14.107527+0000 mon.smithi113 (mon.0) 894 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:15.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:14.112669+0000 mon.smithi113 (mon.0) 895 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:15.920 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:14.752388+0000 mon.smithi113 (mon.0) 896 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:15.920 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:14.758104+0000 mon.smithi113 (mon.0) 897 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:16.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:14.752388+0000 mon.smithi113 (mon.0) 896 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:16.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:14.758104+0000 mon.smithi113 (mon.0) 897 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:16.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:15.962728+0000 mgr.smithi113.ctsazg (mgr.14656) 84 : cluster [DBG] pgmap v43: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:17.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:15.962728+0000 mgr.smithi113.ctsazg (mgr.14656) 84 : cluster [DBG] pgmap v43: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:18.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:17.882504+0000 mon.smithi113 (mon.0) 898 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:17.886744+0000 mon.smithi113 (mon.0) 899 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:17.887475+0000 mon.smithi113 (mon.0) 900 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:17.888069+0000 mon.smithi113 (mon.0) 901 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:17.893884+0000 mon.smithi113 (mon.0) 902 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:17.963181+0000 mgr.smithi113.ctsazg (mgr.14656) 85 : cluster [DBG] pgmap v44: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.029607+0000 mon.smithi113 (mon.0) 903 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.029951+0000 mgr.smithi113.ctsazg (mgr.14656) 86 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.035157+0000 mon.smithi113 (mon.0) 904 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.037662+0000 mon.smithi113 (mon.0) 905 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.040146+0000 mon.smithi113 (mon.0) 906 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.041057+0000 mon.smithi113 (mon.0) 907 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.041931+0000 mon.smithi113 (mon.0) 908 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.042814+0000 mon.smithi113 (mon.0) 909 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.043715+0000 mon.smithi113 (mon.0) 910 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:18.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.044591+0000 mon.smithi113 (mon.0) 911 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:18.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.045527+0000 mon.smithi113 (mon.0) 912 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:18.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.046811+0000 mon.smithi113 (mon.0) 913 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:18.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.047991+0000 mon.smithi113 (mon.0) 914 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:18.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:18.049102+0000 mon.smithi113 (mon.0) 915 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:19.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:17.882504+0000 mon.smithi113 (mon.0) 898 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:19.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:17.886744+0000 mon.smithi113 (mon.0) 899 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:19.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:17.887475+0000 mon.smithi113 (mon.0) 900 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:12:19.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:17.888069+0000 mon.smithi113 (mon.0) 901 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:17.893884+0000 mon.smithi113 (mon.0) 902 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:17.963181+0000 mgr.smithi113.ctsazg (mgr.14656) 85 : cluster [DBG] pgmap v44: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.029607+0000 mon.smithi113 (mon.0) 903 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.029951+0000 mgr.smithi113.ctsazg (mgr.14656) 86 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.035157+0000 mon.smithi113 (mon.0) 904 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.037662+0000 mon.smithi113 (mon.0) 905 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.040146+0000 mon.smithi113 (mon.0) 906 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.041057+0000 mon.smithi113 (mon.0) 907 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.041931+0000 mon.smithi113 (mon.0) 908 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.042814+0000 mon.smithi113 (mon.0) 909 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.043715+0000 mon.smithi113 (mon.0) 910 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.044591+0000 mon.smithi113 (mon.0) 911 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.045527+0000 mon.smithi113 (mon.0) 912 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.046811+0000 mon.smithi113 (mon.0) 913 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:19.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.047991+0000 mon.smithi113 (mon.0) 914 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:19.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:18.049102+0000 mon.smithi113 (mon.0) 915 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:12:19.567367+0000 mgr.smithi113.ctsazg (mgr.14656) 87 : cephadm [INF] Upgrade: Updating grafana.smithi113 2024-04-12T05:12:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:12:19.579287+0000 mgr.smithi113.ctsazg (mgr.14656) 88 : cephadm [INF] Deploying daemon grafana.smithi113 on smithi113 2024-04-12T05:12:20.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:12:19.567367+0000 mgr.smithi113.ctsazg (mgr.14656) 87 : cephadm [INF] Upgrade: Updating grafana.smithi113 2024-04-12T05:12:20.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:12:19.579287+0000 mgr.smithi113.ctsazg (mgr.14656) 88 : cephadm [INF] Deploying daemon grafana.smithi113 on smithi113 2024-04-12T05:12:21.040 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:19.963524+0000 mgr.smithi113.ctsazg (mgr.14656) 89 : cluster [DBG] pgmap v45: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:21.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:19.963524+0000 mgr.smithi113.ctsazg (mgr.14656) 89 : cluster [DBG] pgmap v45: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:23.355 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:23 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:21.964228+0000 mgr.smithi113.ctsazg (mgr.14656) 90 : cluster [DBG] pgmap v46: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:23 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:21.964228+0000 mgr.smithi113.ctsazg (mgr.14656) 90 : cluster [DBG] pgmap v46: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:23.964596+0000 mgr.smithi113.ctsazg (mgr.14656) 91 : cluster [DBG] pgmap v47: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:24.779640+0000 mon.smithi113 (mon.0) 916 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:24.784487+0000 mon.smithi113 (mon.0) 917 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:25 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:24.803217+0000 mon.smithi113 (mon.0) 918 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:23.964596+0000 mgr.smithi113.ctsazg (mgr.14656) 91 : cluster [DBG] pgmap v47: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:24.779640+0000 mon.smithi113 (mon.0) 916 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:24.784487+0000 mon.smithi113 (mon.0) 917 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:24.803217+0000 mon.smithi113 (mon.0) 918 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:25.964922+0000 mgr.smithi113.ctsazg (mgr.14656) 92 : cluster [DBG] pgmap v48: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:27 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:26.023527+0000 mon.smithi113 (mon.0) 919 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:12:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:25.964922+0000 mgr.smithi113.ctsazg (mgr.14656) 92 : cluster [DBG] pgmap v48: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:27 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:26.023527+0000 mon.smithi113 (mon.0) 919 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:12:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:27.965381+0000 mgr.smithi113.ctsazg (mgr.14656) 93 : cluster [DBG] pgmap v49: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:28.036332+0000 mon.smithi113 (mon.0) 920 : cluster [DBG] mgrmap e32: smithi113.ctsazg(active, since 92s), standbys: smithi192.ocppyo 2024-04-12T05:12:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:27.965381+0000 mgr.smithi113.ctsazg (mgr.14656) 93 : cluster [DBG] pgmap v49: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:28.036332+0000 mon.smithi113 (mon.0) 920 : cluster [DBG] mgrmap e32: smithi113.ctsazg(active, since 92s), standbys: smithi192.ocppyo 2024-04-12T05:12:30.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.515+0000 7f9f2215e700 1 -- 172.21.15.113:0/1918208207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f1c0f66f0 msgr2=0x7f9f1c0f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:30.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.515+0000 7f9f2215e700 1 --2- 172.21.15.113:0/1918208207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f1c0f66f0 0x7f9f1c0f6b10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f9f0c0047f0 tx=0x7f9f0c009cb0).stop 2024-04-12T05:12:30.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.515+0000 7f9f2215e700 1 -- 172.21.15.113:0/1918208207 shutdown_connections 2024-04-12T05:12:30.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.515+0000 7f9f2215e700 1 --2- 172.21.15.113:0/1918208207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f1c0f7990 0x7f9f1c0f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:12:30.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.515+0000 7f9f2215e700 1 --2- 172.21.15.113:0/1918208207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f1c0f66f0 0x7f9f1c0f6b10 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:30.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.515+0000 7f9f2215e700 1 -- 172.21.15.113:0/1918208207 >> 172.21.15.113:0/1918208207 conn(0x7f9f1c0f1e50 msgr2=0x7f9f1c0f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:30.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.515+0000 7f9f2215e700 1 -- 172.21.15.113:0/1918208207 shutdown_connections 2024-04-12T05:12:30.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.515+0000 7f9f2215e700 1 -- 172.21.15.113:0/1918208207 wait complete. 2024-04-12T05:12:30.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.515+0000 7f9f2215e700 1 Processor -- start 2024-04-12T05:12:30.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f2215e700 1 -- start start 2024-04-12T05:12:30.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f2215e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f1c0f66f0 0x7f9f1c107fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f2215e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f1c0f7990 0x7f9f1c1084c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f2215e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c108b00 con 0x7f9f1c0f66f0 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f2215e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c108c40 con 0x7f9f1c0f7990 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f1b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f1c0f66f0 0x7f9f1c107fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f1b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f1c0f66f0 0x7f9f1c107fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46476/0 (socket says 172.21.15.113:46476) 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f1b7fe700 1 -- 172.21.15.113:0/4194705920 learned_addr learned my addr 172.21.15.113:0/4194705920 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f1b7fe700 1 -- 172.21.15.113:0/4194705920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f1c0f7990 msgr2=0x7f9f1c1084c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f1affd700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f1c0f7990 0x7f9f1c1084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f1b7fe700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f1c0f7990 0x7f9f1c1084c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f1b7fe700 1 -- 172.21.15.113:0/4194705920 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c005040 con 0x7f9f1c0f66f0 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f1affd700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f1c0f7990 0x7f9f1c1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f1b7fe700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f1c0f66f0 0x7f9f1c107fd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f9f0c00b120 tx=0x7f9f0c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f18ff9700 1 -- 172.21.15.113:0/4194705920 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f0c00e030 con 0x7f9f1c0f66f0 2024-04-12T05:12:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f2215e700 1 -- 172.21.15.113:0/4194705920 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1c10d680 con 0x7f9f1c0f66f0 2024-04-12T05:12:30.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f18ff9700 1 -- 172.21.15.113:0/4194705920 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f0c007e90 con 0x7f9f1c0f66f0 2024-04-12T05:12:30.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f18ff9700 1 -- 172.21.15.113:0/4194705920 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f0c018070 con 0x7f9f1c0f66f0 2024-04-12T05:12:30.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.516+0000 7f9f2215e700 1 -- 172.21.15.113:0/4194705920 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1c10db60 con 0x7f9f1c0f66f0 2024-04-12T05:12:30.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.517+0000 7f9f2215e700 1 -- 172.21.15.113:0/4194705920 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1c0fef30 con 0x7f9f1c0f66f0 2024-04-12T05:12:30.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.518+0000 7f9f18ff9700 1 -- 172.21.15.113:0/4194705920 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f9f0c01d9d0 con 0x7f9f1c0f66f0 2024-04-12T05:12:30.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.518+0000 7f9f18ff9700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9f0406c680 0x7f9f0406eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:30.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.519+0000 7f9f18ff9700 1 -- 172.21.15.113:0/4194705920 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f9f0c08d030 con 0x7f9f1c0f66f0 2024-04-12T05:12:30.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.519+0000 7f9f1affd700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9f0406c680 0x7f9f0406eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:12:30.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.519+0000 7f9f1affd700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9f0406c680 0x7f9f0406eb40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f9f1000a900 tx=0x7f9f10008040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.523+0000 7f9f18ff9700 1 -- 172.21.15.113:0/4194705920 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f9f0c05b130 con 0x7f9f1c0f66f0 2024-04-12T05:12:30.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:29.285664+0000 mon.smithi113 (mon.0) 921 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:30.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:29.291807+0000 mon.smithi113 (mon.0) 922 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:30.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:29.925525+0000 mon.smithi113 (mon.0) 923 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:30.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:29.931719+0000 mon.smithi113 (mon.0) 924 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:30.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:29.285664+0000 mon.smithi113 (mon.0) 921 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:30.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:29.291807+0000 mon.smithi113 (mon.0) 922 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:30.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:29.925525+0000 mon.smithi113 (mon.0) 923 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:30.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:29.931719+0000 mon.smithi113 (mon.0) 924 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:30.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.733+0000 7f9f2215e700 1 -- 172.21.15.113:0/4194705920 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f1c059120 con 0x7f9f0406c680 2024-04-12T05:12:30.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.735+0000 7f9f18ff9700 1 -- 172.21.15.113:0/4194705920 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f9f1c059120 con 0x7f9f0406c680 2024-04-12T05:12:30.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.737+0000 7f9f2215e700 1 -- 172.21.15.113:0/4194705920 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9f0406c680 msgr2=0x7f9f0406eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:30.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.737+0000 7f9f2215e700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9f0406c680 0x7f9f0406eb40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f9f1000a900 tx=0x7f9f10008040).stop 2024-04-12T05:12:30.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.737+0000 7f9f2215e700 1 -- 172.21.15.113:0/4194705920 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f1c0f66f0 msgr2=0x7f9f1c107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:30.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.737+0000 7f9f2215e700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f1c0f66f0 0x7f9f1c107fd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f9f0c00b120 tx=0x7f9f0c009cb0).stop 2024-04-12T05:12:30.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.737+0000 7f9f2215e700 1 -- 172.21.15.113:0/4194705920 shutdown_connections 2024-04-12T05:12:30.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.737+0000 7f9f2215e700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9f0406c680 0x7f9f0406eb40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:30.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.737+0000 7f9f2215e700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f1c0f66f0 0x7f9f1c107fd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:30.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.737+0000 7f9f2215e700 1 --2- 172.21.15.113:0/4194705920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f1c0f7990 0x7f9f1c1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:30.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.737+0000 7f9f2215e700 1 -- 172.21.15.113:0/4194705920 >> 172.21.15.113:0/4194705920 conn(0x7f9f1c0f1e50 msgr2=0x7f9f1c0fc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:30.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.737+0000 7f9f2215e700 1 -- 172.21.15.113:0/4194705920 shutdown_connections 2024-04-12T05:12:30.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.737+0000 7f9f2215e700 1 -- 172.21.15.113:0/4194705920 wait complete. 2024-04-12T05:12:30.749 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:12:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.839+0000 7febafaa3700 1 -- 172.21.15.113:0/2049127563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba80fa950 msgr2=0x7feba80fcd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.839+0000 7febafaa3700 1 --2- 172.21.15.113:0/2049127563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba80fa950 0x7feba80fcd40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7feb9c0047f0 tx=0x7feb9c009cb0).stop 2024-04-12T05:12:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.839+0000 7febafaa3700 1 -- 172.21.15.113:0/2049127563 shutdown_connections 2024-04-12T05:12:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.839+0000 7febafaa3700 1 --2- 172.21.15.113:0/2049127563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba80fa950 0x7feba80fcd40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.839+0000 7febafaa3700 1 --2- 172.21.15.113:0/2049127563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feba8061790 0x7feba80fa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.839+0000 7febafaa3700 1 -- 172.21.15.113:0/2049127563 >> 172.21.15.113:0/2049127563 conn(0x7feba80f1ee0 msgr2=0x7feba80f4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.839+0000 7febafaa3700 1 -- 172.21.15.113:0/2049127563 shutdown_connections 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.840+0000 7febafaa3700 1 -- 172.21.15.113:0/2049127563 wait complete. 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.840+0000 7febafaa3700 1 Processor -- start 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.840+0000 7febafaa3700 1 -- start start 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.840+0000 7febafaa3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba8061790 0x7feba8107f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.840+0000 7febafaa3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feba80fa950 0x7feba8108460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.840+0000 7febafaa3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba8108a10 con 0x7feba8061790 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.840+0000 7febafaa3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba8108b50 con 0x7feba80fa950 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.840+0000 7febad83f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba8061790 0x7feba8107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.840+0000 7febad83f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba8061790 0x7feba8107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46482/0 (socket says 172.21.15.113:46482) 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.840+0000 7febad83f700 1 -- 172.21.15.113:0/3539133199 learned_addr learned my addr 172.21.15.113:0/3539133199 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.841+0000 7febad03e700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feba80fa950 0x7feba8108460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.841+0000 7febad83f700 1 -- 172.21.15.113:0/3539133199 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feba80fa950 msgr2=0x7feba8108460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.841+0000 7febad83f700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feba80fa950 0x7feba8108460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.841+0000 7febad83f700 1 -- 172.21.15.113:0/3539133199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c005040 con 0x7feba8061790 2024-04-12T05:12:30.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.841+0000 7febad03e700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feba80fa950 0x7feba8108460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:12:30.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.841+0000 7febad83f700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba8061790 0x7feba8107f70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7feba400c790 tx=0x7feba4007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:30.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.841+0000 7feb9af78700 1 -- 172.21.15.113:0/3539133199 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feba400e680 con 0x7feba8061790 2024-04-12T05:12:30.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.841+0000 7feb9af78700 1 -- 172.21.15.113:0/3539133199 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feba400ec90 con 0x7feba8061790 2024-04-12T05:12:30.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.841+0000 7febafaa3700 1 -- 172.21.15.113:0/3539133199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba810d5f0 con 0x7feba8061790 2024-04-12T05:12:30.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.841+0000 7feb9af78700 1 -- 172.21.15.113:0/3539133199 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feba4013df0 con 0x7feba8061790 2024-04-12T05:12:30.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.841+0000 7febafaa3700 1 -- 172.21.15.113:0/3539133199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba810dbb0 con 0x7feba8061790 2024-04-12T05:12:30.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.842+0000 7febafaa3700 1 -- 172.21.15.113:0/3539133199 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba80fffb0 con 0x7feba8061790 2024-04-12T05:12:30.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.842+0000 7feb9af78700 1 -- 172.21.15.113:0/3539133199 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7feba400e7e0 con 0x7feba8061790 2024-04-12T05:12:30.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.843+0000 7feb9af78700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7feb9406c6a0 0x7feb9406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:30.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.843+0000 7febad03e700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7feb9406c6a0 0x7feb9406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:12:30.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.843+0000 7feb9af78700 1 -- 172.21.15.113:0/3539133199 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7feba408b9e0 con 0x7feba8061790 2024-04-12T05:12:30.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.843+0000 7febad03e700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7feb9406c6a0 0x7feb9406eb60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7feb9c007cf0 tx=0x7feb9c00b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:30.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:30.848+0000 7feb9af78700 1 -- 172.21.15.113:0/3539133199 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7feba4059ae0 con 0x7feba8061790 2024-04-12T05:12:31.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.059+0000 7febafaa3700 1 -- 172.21.15.113:0/3539133199 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feba8059120 con 0x7feb9406c6a0 2024-04-12T05:12:31.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.060+0000 7feb9af78700 1 -- 172.21.15.113:0/3539133199 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7feba8059120 con 0x7feb9406c6a0 2024-04-12T05:12:31.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.063+0000 7febafaa3700 1 -- 172.21.15.113:0/3539133199 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7feb9406c6a0 msgr2=0x7feb9406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.063+0000 7febafaa3700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7feb9406c6a0 0x7feb9406eb60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7feb9c007cf0 tx=0x7feb9c00b040).stop 2024-04-12T05:12:31.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.063+0000 7febafaa3700 1 -- 172.21.15.113:0/3539133199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba8061790 msgr2=0x7feba8107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.063+0000 7febafaa3700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba8061790 0x7feba8107f70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7feba400c790 tx=0x7feba4007f10).stop 2024-04-12T05:12:31.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.063+0000 7febafaa3700 1 -- 172.21.15.113:0/3539133199 shutdown_connections 2024-04-12T05:12:31.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.063+0000 7febafaa3700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7feb9406c6a0 0x7feb9406eb60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.063+0000 7febafaa3700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feba8061790 0x7feba8107f70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.063+0000 7febafaa3700 1 --2- 172.21.15.113:0/3539133199 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feba80fa950 0x7feba8108460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.063+0000 7febafaa3700 1 -- 172.21.15.113:0/3539133199 >> 172.21.15.113:0/3539133199 conn(0x7feba80f1ee0 msgr2=0x7feba80f42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:31.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.063+0000 7febafaa3700 1 -- 172.21.15.113:0/3539133199 shutdown_connections 2024-04-12T05:12:31.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.063+0000 7febafaa3700 1 -- 172.21.15.113:0/3539133199 wait complete. 2024-04-12T05:12:31.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.162+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/2496171945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9a00f7990 msgr2=0x7fb9a00f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.162+0000 7fb9a5b4e700 1 --2- 172.21.15.113:0/2496171945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9a00f7990 0x7fb9a00f7df0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fb9880047f0 tx=0x7fb988009cb0).stop 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.162+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/2496171945 shutdown_connections 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.162+0000 7fb9a5b4e700 1 --2- 172.21.15.113:0/2496171945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9a00f7990 0x7fb9a00f7df0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.162+0000 7fb9a5b4e700 1 --2- 172.21.15.113:0/2496171945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9a00f66f0 0x7fb9a00f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.162+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/2496171945 >> 172.21.15.113:0/2496171945 conn(0x7fb9a00f1e50 msgr2=0x7fb9a00f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.162+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/2496171945 shutdown_connections 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.162+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/2496171945 wait complete. 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.162+0000 7fb9a5b4e700 1 Processor -- start 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.162+0000 7fb9a5b4e700 1 -- start start 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb9a5b4e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9a00f66f0 0x7fb9a010a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb9a5b4e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9a00f7990 0x7fb9a010a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb9a5b4e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a010ac00 con 0x7fb9a00f7990 2024-04-12T05:12:31.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb9a5b4e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a010ad40 con 0x7fb9a00f66f0 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb99effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9a00f7990 0x7fb9a010a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb99effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9a00f7990 0x7fb9a010a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46502/0 (socket says 172.21.15.113:46502) 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb99effd700 1 -- 172.21.15.113:0/3053395183 learned_addr learned my addr 172.21.15.113:0/3053395183 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb99f7fe700 1 --2- 172.21.15.113:0/3053395183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9a00f66f0 0x7fb9a010a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb99effd700 1 -- 172.21.15.113:0/3053395183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9a00f66f0 msgr2=0x7fb9a010a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb99effd700 1 --2- 172.21.15.113:0/3053395183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9a00f66f0 0x7fb9a010a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb99effd700 1 -- 172.21.15.113:0/3053395183 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb988005040 con 0x7fb9a00f7990 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb99effd700 1 --2- 172.21.15.113:0/3053395183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9a00f7990 0x7fb9a010a650 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fb988013620 tx=0x7fb988009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb99cff9700 1 -- 172.21.15.113:0/3053395183 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb98800e070 con 0x7fb9a00f7990 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/3053395183 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a010f780 con 0x7fb9a00f7990 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb99cff9700 1 -- 172.21.15.113:0/3053395183 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb988007e90 con 0x7fb9a00f7990 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb99cff9700 1 -- 172.21.15.113:0/3053395183 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb988018070 con 0x7fb9a00f7990 2024-04-12T05:12:31.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.163+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/3053395183 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a010fcc0 con 0x7fb9a00f7990 2024-04-12T05:12:31.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.164+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/3053395183 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a00feeb0 con 0x7fb9a00f7990 2024-04-12T05:12:31.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.165+0000 7fb99cff9700 1 -- 172.21.15.113:0/3053395183 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7fb98801d420 con 0x7fb9a00f7990 2024-04-12T05:12:31.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.165+0000 7fb99cff9700 1 --2- 172.21.15.113:0/3053395183 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb98c06c6a0 0x7fb98c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:31.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.165+0000 7fb99cff9700 1 -- 172.21.15.113:0/3053395183 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fb98808c5a0 con 0x7fb9a00f7990 2024-04-12T05:12:31.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.165+0000 7fb99f7fe700 1 --2- 172.21.15.113:0/3053395183 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb98c06c6a0 0x7fb98c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:12:31.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.166+0000 7fb99f7fe700 1 --2- 172.21.15.113:0/3053395183 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb98c06c6a0 0x7fb98c06eb60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb998009a00 tx=0x7fb99800b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:31.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.170+0000 7fb99cff9700 1 -- 172.21.15.113:0/3053395183 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fb98805a6a0 con 0x7fb9a00f7990 2024-04-12T05:12:31.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.369+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/3053395183 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9a0110300 con 0x7fb98c06c6a0 2024-04-12T05:12:31.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.379+0000 7fb99cff9700 1 -- 172.21.15.113:0/3053395183 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fb9a0110300 con 0x7fb98c06c6a0 2024-04-12T05:12:31.381 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:12:31.381 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (22s) 2s ago 11m 15.2M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:12:31.381 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (11m) 2s ago 11m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:12:31.381 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (10m) 17s ago 10m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:12:31.381 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (7s) 2s ago 11m 38.2M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:12:31.381 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (6m) 2s ago 6m 120M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:12:31.381 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (6m) 2s ago 6m 17.0M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (6m) 17s ago 6m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (6m) 17s ago 6m 14.4M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (106s) 2s ago 12m 498M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (75s) 17s ago 10m 446M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (12m) 2s ago 13m 59.8M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (10m) 17s ago 10m 49.8M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (60s) 2s ago 11m 8744k - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (53s) 17s ago 10m 15.8M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (10m) 2s ago 10m 226M 1910M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (9m) 2s ago 9m 240M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (9m) 2s ago 9m 202M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (8m) 17s ago 8m 263M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (8m) 17s ago 8m 233M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:12:31.382 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (7m) 17s ago 7m 226M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:12:31.383 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (37s) 17s ago 10m 42.0M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:12:31.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.381+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/3053395183 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb98c06c6a0 msgr2=0x7fb98c06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.381+0000 7fb9a5b4e700 1 --2- 172.21.15.113:0/3053395183 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb98c06c6a0 0x7fb98c06eb60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb998009a00 tx=0x7fb99800b040).stop 2024-04-12T05:12:31.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.381+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/3053395183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9a00f7990 msgr2=0x7fb9a010a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.381+0000 7fb9a5b4e700 1 --2- 172.21.15.113:0/3053395183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9a00f7990 0x7fb9a010a650 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fb988013620 tx=0x7fb988009cb0).stop 2024-04-12T05:12:31.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.381+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/3053395183 shutdown_connections 2024-04-12T05:12:31.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.381+0000 7fb9a5b4e700 1 --2- 172.21.15.113:0/3053395183 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb98c06c6a0 0x7fb98c06eb60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.381+0000 7fb9a5b4e700 1 --2- 172.21.15.113:0/3053395183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9a00f66f0 0x7fb9a010a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.381+0000 7fb9a5b4e700 1 --2- 172.21.15.113:0/3053395183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9a00f7990 0x7fb9a010a650 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.381+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/3053395183 >> 172.21.15.113:0/3053395183 conn(0x7fb9a00f1e50 msgr2=0x7fb9a0060d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:31.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.381+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/3053395183 shutdown_connections 2024-04-12T05:12:31.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.381+0000 7fb9a5b4e700 1 -- 172.21.15.113:0/3053395183 wait complete. 2024-04-12T05:12:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:31 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:29.965730+0000 mgr.smithi113.ctsazg (mgr.14656) 94 : cluster [DBG] pgmap v50: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:31.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.481+0000 7f8578ea6700 1 -- 172.21.15.113:0/178249276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85740f7a10 msgr2=0x7f85740f7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.481+0000 7f8578ea6700 1 --2- 172.21.15.113:0/178249276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85740f7a10 0x7f85740f7e30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f85640047f0 tx=0x7f8564009cb0).stop 2024-04-12T05:12:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.481+0000 7f8578ea6700 1 -- 172.21.15.113:0/178249276 shutdown_connections 2024-04-12T05:12:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.481+0000 7f8578ea6700 1 --2- 172.21.15.113:0/178249276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85740f8cb0 0x7f85740f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.481+0000 7f8578ea6700 1 --2- 172.21.15.113:0/178249276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85740f7a10 0x7f85740f7e30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.481+0000 7f8578ea6700 1 -- 172.21.15.113:0/178249276 >> 172.21.15.113:0/178249276 conn(0x7f85740f3150 msgr2=0x7f85740f5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.482+0000 7f8578ea6700 1 -- 172.21.15.113:0/178249276 shutdown_connections 2024-04-12T05:12:31.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.482+0000 7f8578ea6700 1 -- 172.21.15.113:0/178249276 wait complete. 2024-04-12T05:12:31.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.482+0000 7f8578ea6700 1 Processor -- start 2024-04-12T05:12:31.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.482+0000 7f8578ea6700 1 -- start start 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.482+0000 7f8578ea6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85740f7a10 0x7f8574108190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.482+0000 7f8578ea6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85740f8cb0 0x7f8574108680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.482+0000 7f8578ea6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8574108c30 con 0x7f85740f7a10 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.482+0000 7f8578ea6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8574108d70 con 0x7f85740f8cb0 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f857259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85740f7a10 0x7f8574108190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f857259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85740f7a10 0x7f8574108190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46522/0 (socket says 172.21.15.113:46522) 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f857259c700 1 -- 172.21.15.113:0/2243125902 learned_addr learned my addr 172.21.15.113:0/2243125902 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f8571d9b700 1 --2- 172.21.15.113:0/2243125902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85740f8cb0 0x7f8574108680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f857259c700 1 -- 172.21.15.113:0/2243125902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85740f8cb0 msgr2=0x7f8574108680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f857259c700 1 --2- 172.21.15.113:0/2243125902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85740f8cb0 0x7f8574108680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f857259c700 1 -- 172.21.15.113:0/2243125902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8564005040 con 0x7f85740f7a10 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f857259c700 1 --2- 172.21.15.113:0/2243125902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85740f7a10 0x7f8574108190 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f8564013620 tx=0x7f8564009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:31.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f85637fe700 1 -- 172.21.15.113:0/2243125902 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f856400e070 con 0x7f85740f7a10 2024-04-12T05:12:31.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f8578ea6700 1 -- 172.21.15.113:0/2243125902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857410d7b0 con 0x7f85740f7a10 2024-04-12T05:12:31.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f85637fe700 1 -- 172.21.15.113:0/2243125902 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8564007e90 con 0x7f85740f7a10 2024-04-12T05:12:31.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f85637fe700 1 -- 172.21.15.113:0/2243125902 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8564018070 con 0x7f85740f7a10 2024-04-12T05:12:31.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.483+0000 7f8578ea6700 1 -- 172.21.15.113:0/2243125902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857410dd70 con 0x7f85740f7a10 2024-04-12T05:12:31.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.484+0000 7f8578ea6700 1 -- 172.21.15.113:0/2243125902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f857405fb50 con 0x7f85740f7a10 2024-04-12T05:12:31.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.485+0000 7f85637fe700 1 -- 172.21.15.113:0/2243125902 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f856401d420 con 0x7f85740f7a10 2024-04-12T05:12:31.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.485+0000 7f85637fe700 1 --2- 172.21.15.113:0/2243125902 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f855c06c6a0 0x7f855c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:31.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.485+0000 7f85637fe700 1 -- 172.21.15.113:0/2243125902 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f856408c450 con 0x7f85740f7a10 2024-04-12T05:12:31.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.485+0000 7f8571d9b700 1 --2- 172.21.15.113:0/2243125902 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f855c06c6a0 0x7f855c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:12:31.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.486+0000 7f8571d9b700 1 --2- 172.21.15.113:0/2243125902 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f855c06c6a0 0x7f855c06eb60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f8568009170 tx=0x7f8568009d60).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:31.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.490+0000 7f85637fe700 1 -- 172.21.15.113:0/2243125902 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f856405a550 con 0x7f85740f7a10 2024-04-12T05:12:31.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:29.965730+0000 mgr.smithi113.ctsazg (mgr.14656) 94 : cluster [DBG] pgmap v50: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:31.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.770+0000 7f8578ea6700 1 -- 172.21.15.113:0/2243125902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8574102110 con 0x7f85740f7a10 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.771+0000 7f85637fe700 1 -- 172.21.15.113:0/2243125902 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f856401de60 con 0x7f85740f7a10 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:12:31.773 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:12:31.774 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:12:31.774 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:12:31.774 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 12, 2024-04-12T05:12:31.774 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:12:31.774 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:12:31.774 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:12:31.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.773+0000 7f8578ea6700 1 -- 172.21.15.113:0/2243125902 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f855c06c6a0 msgr2=0x7f855c06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.773+0000 7f8578ea6700 1 --2- 172.21.15.113:0/2243125902 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f855c06c6a0 0x7f855c06eb60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f8568009170 tx=0x7f8568009d60).stop 2024-04-12T05:12:31.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.773+0000 7f8578ea6700 1 -- 172.21.15.113:0/2243125902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85740f7a10 msgr2=0x7f8574108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.773+0000 7f8578ea6700 1 --2- 172.21.15.113:0/2243125902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85740f7a10 0x7f8574108190 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f8564013620 tx=0x7f8564009cb0).stop 2024-04-12T05:12:31.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.774+0000 7f8578ea6700 1 -- 172.21.15.113:0/2243125902 shutdown_connections 2024-04-12T05:12:31.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.774+0000 7f8578ea6700 1 --2- 172.21.15.113:0/2243125902 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f855c06c6a0 0x7f855c06eb60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.774+0000 7f8578ea6700 1 --2- 172.21.15.113:0/2243125902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85740f7a10 0x7f8574108190 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.774+0000 7f8578ea6700 1 --2- 172.21.15.113:0/2243125902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85740f8cb0 0x7f8574108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.774+0000 7f8578ea6700 1 -- 172.21.15.113:0/2243125902 >> 172.21.15.113:0/2243125902 conn(0x7f85740f3150 msgr2=0x7f85740fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:31.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.774+0000 7f8578ea6700 1 -- 172.21.15.113:0/2243125902 shutdown_connections 2024-04-12T05:12:31.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.774+0000 7f8578ea6700 1 -- 172.21.15.113:0/2243125902 wait complete. 2024-04-12T05:12:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.879+0000 7f4d00684700 1 -- 172.21.15.113:0/3877732232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cf80f80e0 msgr2=0x7f4cf80f8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.879+0000 7f4d00684700 1 --2- 172.21.15.113:0/3877732232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cf80f80e0 0x7f4cf80f8b10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f4ce80047f0 tx=0x7f4ce8009cb0).stop 2024-04-12T05:12:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.879+0000 7f4d00684700 1 -- 172.21.15.113:0/3877732232 shutdown_connections 2024-04-12T05:12:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.879+0000 7f4d00684700 1 --2- 172.21.15.113:0/3877732232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cf80f80e0 0x7f4cf80f8b10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.879+0000 7f4d00684700 1 --2- 172.21.15.113:0/3877732232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cf80f77d0 0x7f4cf80f7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.879+0000 7f4d00684700 1 -- 172.21.15.113:0/3877732232 >> 172.21.15.113:0/3877732232 conn(0x7f4cf80f2fb0 msgr2=0x7f4cf80f53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.879+0000 7f4d00684700 1 -- 172.21.15.113:0/3877732232 shutdown_connections 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.879+0000 7f4d00684700 1 -- 172.21.15.113:0/3877732232 wait complete. 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.880+0000 7f4d00684700 1 Processor -- start 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.880+0000 7f4d00684700 1 -- start start 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.880+0000 7f4d00684700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cf80f77d0 0x7f4cf8107f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.880+0000 7f4d00684700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cf80f80e0 0x7f4cf8108460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.880+0000 7f4d00684700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf8108a10 con 0x7f4cf80f77d0 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.880+0000 7f4d00684700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf8108b50 con 0x7f4cf80f80e0 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.880+0000 7f4cfe420700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cf80f77d0 0x7f4cf8107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.880+0000 7f4cfe420700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cf80f77d0 0x7f4cf8107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46534/0 (socket says 172.21.15.113:46534) 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.880+0000 7f4cfe420700 1 -- 172.21.15.113:0/3016413434 learned_addr learned my addr 172.21.15.113:0/3016413434 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.880+0000 7f4cfdc1f700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cf80f80e0 0x7f4cf8108460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.881+0000 7f4cfe420700 1 -- 172.21.15.113:0/3016413434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cf80f80e0 msgr2=0x7f4cf8108460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.881+0000 7f4cfe420700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cf80f80e0 0x7f4cf8108460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:31.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.881+0000 7f4cfe420700 1 -- 172.21.15.113:0/3016413434 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce8005040 con 0x7f4cf80f77d0 2024-04-12T05:12:31.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.881+0000 7f4cfdc1f700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cf80f80e0 0x7f4cf8108460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:12:31.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.881+0000 7f4cfe420700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cf80f77d0 0x7f4cf8107f70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4cf40041d0 tx=0x7f4cf4009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:31.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.881+0000 7f4cef7fe700 1 -- 172.21.15.113:0/3016413434 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cf4007b40 con 0x7f4cf80f77d0 2024-04-12T05:12:31.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.881+0000 7f4cef7fe700 1 -- 172.21.15.113:0/3016413434 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4cf400f040 con 0x7f4cf80f77d0 2024-04-12T05:12:31.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.881+0000 7f4cef7fe700 1 -- 172.21.15.113:0/3016413434 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cf400d800 con 0x7f4cf80f77d0 2024-04-12T05:12:31.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.881+0000 7f4d00684700 1 -- 172.21.15.113:0/3016413434 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf810d5f0 con 0x7f4cf80f77d0 2024-04-12T05:12:31.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.881+0000 7f4d00684700 1 -- 172.21.15.113:0/3016413434 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf810dbb0 con 0x7f4cf80f77d0 2024-04-12T05:12:31.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.882+0000 7f4d00684700 1 -- 172.21.15.113:0/3016413434 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf80ffff0 con 0x7f4cf80f77d0 2024-04-12T05:12:31.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.882+0000 7f4cef7fe700 1 -- 172.21.15.113:0/3016413434 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f4cf4007ca0 con 0x7f4cf80f77d0 2024-04-12T05:12:31.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.883+0000 7f4cef7fe700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f4ce406c6a0 0x7f4ce406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:12:31.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.883+0000 7f4cef7fe700 1 -- 172.21.15.113:0/3016413434 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f4cf408b1d0 con 0x7f4cf80f77d0 2024-04-12T05:12:31.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.883+0000 7f4cfdc1f700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f4ce406c6a0 0x7f4ce406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:12:31.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.883+0000 7f4cfdc1f700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f4ce406c6a0 0x7f4ce406eb60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4ce80043d0 tx=0x7f4ce800b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:12:31.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:31.888+0000 7f4cef7fe700 1 -- 172.21.15.113:0/3016413434 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f4cf40592d0 con 0x7f4cf80f77d0 2024-04-12T05:12:32.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.104+0000 7f4d00684700 1 -- 172.21.15.113:0/3016413434 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cf8059120 con 0x7f4ce406c6a0 2024-04-12T05:12:32.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.105+0000 7f4cef7fe700 1 -- 172.21.15.113:0/3016413434 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f4cf8059120 con 0x7f4ce406c6a0 2024-04-12T05:12:32.107 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:12:32.107 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:12:32.107 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:12:32.107 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-12T05:12:32.108 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:12:32.108 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr" 2024-04-12T05:12:32.108 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:12:32.108 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "2/2 daemons upgraded", 2024-04-12T05:12:32.108 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading grafana daemons", 2024-04-12T05:12:32.108 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:12:32.108 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:12:32.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.107+0000 7f4d00684700 1 -- 172.21.15.113:0/3016413434 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f4ce406c6a0 msgr2=0x7f4ce406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:32.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.107+0000 7f4d00684700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f4ce406c6a0 0x7f4ce406eb60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4ce80043d0 tx=0x7f4ce800b040).stop 2024-04-12T05:12:32.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.107+0000 7f4d00684700 1 -- 172.21.15.113:0/3016413434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cf80f77d0 msgr2=0x7f4cf8107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:12:32.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.107+0000 7f4d00684700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cf80f77d0 0x7f4cf8107f70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4cf40041d0 tx=0x7f4cf4009f60).stop 2024-04-12T05:12:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.107+0000 7f4d00684700 1 -- 172.21.15.113:0/3016413434 shutdown_connections 2024-04-12T05:12:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.107+0000 7f4d00684700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f4ce406c6a0 0x7f4ce406eb60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.107+0000 7f4d00684700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cf80f77d0 0x7f4cf8107f70 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.107+0000 7f4d00684700 1 --2- 172.21.15.113:0/3016413434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cf80f80e0 0x7f4cf8108460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:12:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.107+0000 7f4d00684700 1 -- 172.21.15.113:0/3016413434 >> 172.21.15.113:0/3016413434 conn(0x7f4cf80f2fb0 msgr2=0x7f4cf80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:12:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.107+0000 7f4d00684700 1 -- 172.21.15.113:0/3016413434 shutdown_connections 2024-04-12T05:12:32.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:12:32.107+0000 7f4d00684700 1 -- 172.21.15.113:0/3016413434 wait complete. 2024-04-12T05:12:32.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:30.735439+0000 mgr.smithi113.ctsazg (mgr.14656) 95 : audit [DBG] from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:32.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:31.061272+0000 mgr.smithi113.ctsazg (mgr.14656) 96 : audit [DBG] from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:32.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:31.370913+0000 mgr.smithi113.ctsazg (mgr.14656) 97 : audit [DBG] from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:32.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:31.771583+0000 mon.smithi113 (mon.0) 925 : audit [DBG] from='client.? 172.21.15.113:0/2243125902' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:32.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:30.735439+0000 mgr.smithi113.ctsazg (mgr.14656) 95 : audit [DBG] from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:32.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:31.061272+0000 mgr.smithi113.ctsazg (mgr.14656) 96 : audit [DBG] from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:32.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:31.370913+0000 mgr.smithi113.ctsazg (mgr.14656) 97 : audit [DBG] from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:32.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:31.771583+0000 mon.smithi113 (mon.0) 925 : audit [DBG] from='client.? 172.21.15.113:0/2243125902' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:31.966217+0000 mgr.smithi113.ctsazg (mgr.14656) 98 : cluster [DBG] pgmap v51: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:32.105951+0000 mgr.smithi113.ctsazg (mgr.14656) 99 : audit [DBG] from='client.14764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.205593+0000 mon.smithi113 (mon.0) 926 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.211756+0000 mon.smithi113 (mon.0) 927 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.212546+0000 mon.smithi113 (mon.0) 928 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:12:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.213180+0000 mon.smithi113 (mon.0) 929 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:12:33.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:33 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.219300+0000 mon.smithi113 (mon.0) 930 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:33.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:31.966217+0000 mgr.smithi113.ctsazg (mgr.14656) 98 : cluster [DBG] pgmap v51: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:33.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:32.105951+0000 mgr.smithi113.ctsazg (mgr.14656) 99 : audit [DBG] from='client.14764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:12:33.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.205593+0000 mon.smithi113 (mon.0) 926 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:33.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.211756+0000 mon.smithi113 (mon.0) 927 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:33.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.212546+0000 mon.smithi113 (mon.0) 928 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:12:33.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.213180+0000 mon.smithi113 (mon.0) 929 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:12:33.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.219300+0000 mon.smithi113 (mon.0) 930 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:34.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.354552+0000 mon.smithi113 (mon.0) 931 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:12:34.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.354919+0000 mgr.smithi113.ctsazg (mgr.14656) 100 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:12:34.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.360022+0000 mon.smithi113 (mon.0) 932 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:34.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.362216+0000 mon.smithi113 (mon.0) 933 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.364260+0000 mon.smithi113 (mon.0) 934 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.365102+0000 mon.smithi113 (mon.0) 935 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.365965+0000 mon.smithi113 (mon.0) 936 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.366840+0000 mon.smithi113 (mon.0) 937 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.367677+0000 mon.smithi113 (mon.0) 938 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.368529+0000 mon.smithi113 (mon.0) 939 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.369480+0000 mon.smithi113 (mon.0) 940 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.370662+0000 mon.smithi113 (mon.0) 941 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.371676+0000 mon.smithi113 (mon.0) 942 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.372667+0000 mon.smithi113 (mon.0) 943 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.373721+0000 mon.smithi113 (mon.0) 944 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.374640+0000 mon.smithi113 (mon.0) 945 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.375541+0000 mon.smithi113 (mon.0) 946 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:12:33.376096+0000 mgr.smithi113.ctsazg (mgr.14656) 101 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.380120+0000 mon.smithi113 (mon.0) 947 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.381340+0000 mon.smithi113 (mon.0) 948 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.385175+0000 mon.smithi113 (mon.0) 949 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.385787+0000 mon.smithi113 (mon.0) 950 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.388902+0000 mon.smithi113 (mon.0) 951 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-12T05:12:34.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.389493+0000 mon.smithi113 (mon.0) 952 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.389955+0000 mon.smithi113 (mon.0) 953 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.390387+0000 mon.smithi113 (mon.0) 954 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.390839+0000 mon.smithi113 (mon.0) 955 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.393807+0000 mon.smithi113 (mon.0) 956 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.394339+0000 mon.smithi113 (mon.0) 957 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.397114+0000 mon.smithi113 (mon.0) 958 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.397676+0000 mon.smithi113 (mon.0) 959 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.398156+0000 mon.smithi113 (mon.0) 960 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.400985+0000 mon.smithi113 (mon.0) 961 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.401486+0000 mon.smithi113 (mon.0) 962 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.404253+0000 mon.smithi113 (mon.0) 963 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.404875+0000 mon.smithi113 (mon.0) 964 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.407703+0000 mon.smithi113 (mon.0) 965 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-12T05:12:34.677 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.408274+0000 mon.smithi113 (mon.0) 966 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.411318+0000 mon.smithi113 (mon.0) 967 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.411977+0000 mon.smithi113 (mon.0) 968 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.412450+0000 mon.smithi113 (mon.0) 969 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.412916+0000 mon.smithi113 (mon.0) 970 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.413352+0000 mon.smithi113 (mon.0) 971 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.413765+0000 mon.smithi113 (mon.0) 972 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.414188+0000 mon.smithi113 (mon.0) 973 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.414601+0000 mon.smithi113 (mon.0) 974 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.418021+0000 mon.smithi113 (mon.0) 975 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:12:33.418471+0000 mgr.smithi113.ctsazg (mgr.14656) 102 : cephadm [INF] Upgrade: Complete! 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.418810+0000 mon.smithi113 (mon.0) 976 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.421252+0000 mon.smithi113 (mon.0) 977 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.421589+0000 mon.smithi113 (mon.0) 978 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:34.678 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.422797+0000 mon.smithi113 (mon.0) 979 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:12:34.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.423326+0000 mon.smithi113 (mon.0) 980 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:12:34.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.427198+0000 mon.smithi113 (mon.0) 981 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:34.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.563687+0000 mon.smithi113 (mon.0) 982 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:34.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.565064+0000 mon.smithi113 (mon.0) 983 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:12:34.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.565576+0000 mon.smithi113 (mon.0) 984 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:12:34.679 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:33.569992+0000 mon.smithi113 (mon.0) 985 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.354552+0000 mon.smithi113 (mon.0) 931 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:12:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.354919+0000 mgr.smithi113.ctsazg (mgr.14656) 100 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T05:12:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.360022+0000 mon.smithi113 (mon.0) 932 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.362216+0000 mon.smithi113 (mon.0) 933 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.364260+0000 mon.smithi113 (mon.0) 934 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.365102+0000 mon.smithi113 (mon.0) 935 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.365965+0000 mon.smithi113 (mon.0) 936 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.366840+0000 mon.smithi113 (mon.0) 937 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.367677+0000 mon.smithi113 (mon.0) 938 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.368529+0000 mon.smithi113 (mon.0) 939 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.369480+0000 mon.smithi113 (mon.0) 940 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.370662+0000 mon.smithi113 (mon.0) 941 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.371676+0000 mon.smithi113 (mon.0) 942 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.372667+0000 mon.smithi113 (mon.0) 943 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.373721+0000 mon.smithi113 (mon.0) 944 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.374640+0000 mon.smithi113 (mon.0) 945 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.375541+0000 mon.smithi113 (mon.0) 946 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:12:33.376096+0000 mgr.smithi113.ctsazg (mgr.14656) 101 : cephadm [INF] Upgrade: Finalizing container_image settings 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.380120+0000 mon.smithi113 (mon.0) 947 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:34.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.381340+0000 mon.smithi113 (mon.0) 948 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.385175+0000 mon.smithi113 (mon.0) 949 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.385787+0000 mon.smithi113 (mon.0) 950 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.388902+0000 mon.smithi113 (mon.0) 951 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.389493+0000 mon.smithi113 (mon.0) 952 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.389955+0000 mon.smithi113 (mon.0) 953 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.390387+0000 mon.smithi113 (mon.0) 954 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.390839+0000 mon.smithi113 (mon.0) 955 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.393807+0000 mon.smithi113 (mon.0) 956 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.394339+0000 mon.smithi113 (mon.0) 957 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.397114+0000 mon.smithi113 (mon.0) 958 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.397676+0000 mon.smithi113 (mon.0) 959 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.398156+0000 mon.smithi113 (mon.0) 960 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-12T05:12:34.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.400985+0000 mon.smithi113 (mon.0) 961 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.401486+0000 mon.smithi113 (mon.0) 962 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.404253+0000 mon.smithi113 (mon.0) 963 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.404875+0000 mon.smithi113 (mon.0) 964 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.407703+0000 mon.smithi113 (mon.0) 965 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.408274+0000 mon.smithi113 (mon.0) 966 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.411318+0000 mon.smithi113 (mon.0) 967 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.411977+0000 mon.smithi113 (mon.0) 968 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.412450+0000 mon.smithi113 (mon.0) 969 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.412916+0000 mon.smithi113 (mon.0) 970 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.413352+0000 mon.smithi113 (mon.0) 971 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.413765+0000 mon.smithi113 (mon.0) 972 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.414188+0000 mon.smithi113 (mon.0) 973 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:12:34.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.414601+0000 mon.smithi113 (mon.0) 974 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.418021+0000 mon.smithi113 (mon.0) 975 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:12:33.418471+0000 mgr.smithi113.ctsazg (mgr.14656) 102 : cephadm [INF] Upgrade: Complete! 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.418810+0000 mon.smithi113 (mon.0) 976 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.421252+0000 mon.smithi113 (mon.0) 977 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.421589+0000 mon.smithi113 (mon.0) 978 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.422797+0000 mon.smithi113 (mon.0) 979 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.423326+0000 mon.smithi113 (mon.0) 980 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.427198+0000 mon.smithi113 (mon.0) 981 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.563687+0000 mon.smithi113 (mon.0) 982 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.565064+0000 mon.smithi113 (mon.0) 983 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.565576+0000 mon.smithi113 (mon.0) 984 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:12:34.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:33.569992+0000 mon.smithi113 (mon.0) 985 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:35.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:33.966543+0000 mgr.smithi113.ctsazg (mgr.14656) 103 : cluster [DBG] pgmap v52: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:35.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:34.576437+0000 mon.smithi113 (mon.0) 986 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:12:35.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:33.966543+0000 mgr.smithi113.ctsazg (mgr.14656) 103 : cluster [DBG] pgmap v52: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:35.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:34.576437+0000 mon.smithi113 (mon.0) 986 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:12:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:35.966936+0000 mgr.smithi113.ctsazg (mgr.14656) 104 : cluster [DBG] pgmap v53: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:36.032841+0000 mon.smithi113 (mon.0) 987 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:35.966936+0000 mgr.smithi113.ctsazg (mgr.14656) 104 : cluster [DBG] pgmap v53: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:36.032841+0000 mon.smithi113 (mon.0) 987 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:12:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:37.967390+0000 mgr.smithi113.ctsazg (mgr.14656) 105 : cluster [DBG] pgmap v54: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:37.967390+0000 mgr.smithi113.ctsazg (mgr.14656) 105 : cluster [DBG] pgmap v54: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:12:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:39.967931+0000 mgr.smithi113.ctsazg (mgr.14656) 106 : cluster [DBG] pgmap v55: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:41 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:41.022503+0000 mon.smithi113 (mon.0) 988 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:12:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:39.967931+0000 mgr.smithi113.ctsazg (mgr.14656) 106 : cluster [DBG] pgmap v55: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:41.022503+0000 mon.smithi113 (mon.0) 988 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:12:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:41.968604+0000 mgr.smithi113.ctsazg (mgr.14656) 107 : cluster [DBG] pgmap v56: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:43.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:43 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:41.968604+0000 mgr.smithi113.ctsazg (mgr.14656) 107 : cluster [DBG] pgmap v56: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:45 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:43.969212+0000 mgr.smithi113.ctsazg (mgr.14656) 108 : cluster [DBG] pgmap v57: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:45 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:43.969212+0000 mgr.smithi113.ctsazg (mgr.14656) 108 : cluster [DBG] pgmap v57: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:47 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:45.969618+0000 mgr.smithi113.ctsazg (mgr.14656) 109 : cluster [DBG] pgmap v58: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:47 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:45.969618+0000 mgr.smithi113.ctsazg (mgr.14656) 109 : cluster [DBG] pgmap v58: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:49 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:47.970345+0000 mgr.smithi113.ctsazg (mgr.14656) 110 : cluster [DBG] pgmap v59: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:49.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:49 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:47.970345+0000 mgr.smithi113.ctsazg (mgr.14656) 110 : cluster [DBG] pgmap v59: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:49.970729+0000 mgr.smithi113.ctsazg (mgr.14656) 111 : cluster [DBG] pgmap v60: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:49.970729+0000 mgr.smithi113.ctsazg (mgr.14656) 111 : cluster [DBG] pgmap v60: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:51.971343+0000 mgr.smithi113.ctsazg (mgr.14656) 112 : cluster [DBG] pgmap v61: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:51.971343+0000 mgr.smithi113.ctsazg (mgr.14656) 112 : cluster [DBG] pgmap v61: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:55 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:53.971708+0000 mgr.smithi113.ctsazg (mgr.14656) 113 : cluster [DBG] pgmap v62: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:55.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:55 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:53.971708+0000 mgr.smithi113.ctsazg (mgr.14656) 113 : cluster [DBG] pgmap v62: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:56.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:56 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:12:56.022791+0000 mon.smithi113 (mon.0) 989 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:12:56.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:12:56.022791+0000 mon.smithi113 (mon.0) 989 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:12:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:57 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:55.972283+0000 mgr.smithi113.ctsazg (mgr.14656) 114 : cluster [DBG] pgmap v63: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:57.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:57 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:55.972283+0000 mgr.smithi113.ctsazg (mgr.14656) 114 : cluster [DBG] pgmap v63: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:12:59 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:57.972872+0000 mgr.smithi113.ctsazg (mgr.14656) 115 : cluster [DBG] pgmap v64: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:12:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:12:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:57.972872+0000 mgr.smithi113.ctsazg (mgr.14656) 115 : cluster [DBG] pgmap v64: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:01 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:12:59.973308+0000 mgr.smithi113.ctsazg (mgr.14656) 116 : cluster [DBG] pgmap v65: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:01.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:01 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:12:59.973308+0000 mgr.smithi113.ctsazg (mgr.14656) 116 : cluster [DBG] pgmap v65: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:02.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.220+0000 7fe315acd700 1 -- 172.21.15.113:0/280080669 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3100fa950 msgr2=0x7fe3100fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:02.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.220+0000 7fe315acd700 1 --2- 172.21.15.113:0/280080669 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3100fa950 0x7fe3100fcda0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fe3000047f0 tx=0x7fe300009cb0).stop 2024-04-12T05:13:02.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.220+0000 7fe315acd700 1 -- 172.21.15.113:0/280080669 shutdown_connections 2024-04-12T05:13:02.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.220+0000 7fe315acd700 1 --2- 172.21.15.113:0/280080669 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3100fa950 0x7fe3100fcda0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.220+0000 7fe315acd700 1 --2- 172.21.15.113:0/280080669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe310061280 0x7fe3100fa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.220+0000 7fe315acd700 1 -- 172.21.15.113:0/280080669 >> 172.21.15.113:0/280080669 conn(0x7fe3100f1e50 msgr2=0x7fe3100f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.220+0000 7fe315acd700 1 -- 172.21.15.113:0/280080669 shutdown_connections 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.220+0000 7fe315acd700 1 -- 172.21.15.113:0/280080669 wait complete. 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.220+0000 7fe315acd700 1 Processor -- start 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe315acd700 1 -- start start 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe315acd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe310061280 0x7fe31010a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe315acd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3100fa950 0x7fe31010a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe315acd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31010ac10 con 0x7fe3100fa950 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe315acd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31010ad50 con 0x7fe310061280 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe30effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3100fa950 0x7fe31010a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:02.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe30effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3100fa950 0x7fe31010a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:32948/0 (socket says 172.21.15.113:32948) 2024-04-12T05:13:02.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe30effd700 1 -- 172.21.15.113:0/3450391015 learned_addr learned my addr 172.21.15.113:0/3450391015 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:02.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe30f7fe700 1 --2- 172.21.15.113:0/3450391015 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe310061280 0x7fe31010a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:02.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe30effd700 1 -- 172.21.15.113:0/3450391015 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe310061280 msgr2=0x7fe31010a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:02.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe30effd700 1 --2- 172.21.15.113:0/3450391015 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe310061280 0x7fe31010a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:02.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe30effd700 1 -- 172.21.15.113:0/3450391015 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe300005040 con 0x7fe3100fa950 2024-04-12T05:13:02.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.221+0000 7fe30effd700 1 --2- 172.21.15.113:0/3450391015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3100fa950 0x7fe31010a660 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fe300013620 tx=0x7fe300009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:02.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.222+0000 7fe30cf74700 1 -- 172.21.15.113:0/3450391015 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe30000e070 con 0x7fe3100fa950 2024-04-12T05:13:02.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.222+0000 7fe315acd700 1 -- 172.21.15.113:0/3450391015 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31010f790 con 0x7fe3100fa950 2024-04-12T05:13:02.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.222+0000 7fe30cf74700 1 -- 172.21.15.113:0/3450391015 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe300007e90 con 0x7fe3100fa950 2024-04-12T05:13:02.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.222+0000 7fe30cf74700 1 -- 172.21.15.113:0/3450391015 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe300018070 con 0x7fe3100fa950 2024-04-12T05:13:02.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.222+0000 7fe315acd700 1 -- 172.21.15.113:0/3450391015 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31010fd50 con 0x7fe3100fa950 2024-04-12T05:13:02.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.223+0000 7fe315acd700 1 -- 172.21.15.113:0/3450391015 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3100fffd0 con 0x7fe3100fa950 2024-04-12T05:13:02.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.223+0000 7fe30cf74700 1 -- 172.21.15.113:0/3450391015 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7fe30001d420 con 0x7fe3100fa950 2024-04-12T05:13:02.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.224+0000 7fe30cf74700 1 --2- 172.21.15.113:0/3450391015 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe2fc06c6a0 0x7fe2fc06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:02.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.224+0000 7fe30f7fe700 1 --2- 172.21.15.113:0/3450391015 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe2fc06c6a0 0x7fe2fc06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:02.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.224+0000 7fe30cf74700 1 -- 172.21.15.113:0/3450391015 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fe30008c390 con 0x7fe3100fa950 2024-04-12T05:13:02.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.224+0000 7fe30f7fe700 1 --2- 172.21.15.113:0/3450391015 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe2fc06c6a0 0x7fe2fc06eb60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe2f8004de0 tx=0x7fe2f800b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:02.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.229+0000 7fe30cf74700 1 -- 172.21.15.113:0/3450391015 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fe30005a490 con 0x7fe3100fa950 2024-04-12T05:13:02.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.441+0000 7fe315acd700 1 -- 172.21.15.113:0/3450391015 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe310110390 con 0x7fe2fc06c6a0 2024-04-12T05:13:02.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.442+0000 7fe30cf74700 1 -- 172.21.15.113:0/3450391015 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fe310110390 con 0x7fe2fc06c6a0 2024-04-12T05:13:02.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.444+0000 7fe315acd700 1 -- 172.21.15.113:0/3450391015 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe2fc06c6a0 msgr2=0x7fe2fc06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:02.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.444+0000 7fe315acd700 1 --2- 172.21.15.113:0/3450391015 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe2fc06c6a0 0x7fe2fc06eb60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe2f8004de0 tx=0x7fe2f800b040).stop 2024-04-12T05:13:02.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.444+0000 7fe315acd700 1 -- 172.21.15.113:0/3450391015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3100fa950 msgr2=0x7fe31010a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:02.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.444+0000 7fe315acd700 1 --2- 172.21.15.113:0/3450391015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3100fa950 0x7fe31010a660 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fe300013620 tx=0x7fe300009cb0).stop 2024-04-12T05:13:02.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.444+0000 7fe315acd700 1 -- 172.21.15.113:0/3450391015 shutdown_connections 2024-04-12T05:13:02.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.444+0000 7fe315acd700 1 --2- 172.21.15.113:0/3450391015 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe2fc06c6a0 0x7fe2fc06eb60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:02.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.444+0000 7fe315acd700 1 --2- 172.21.15.113:0/3450391015 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe310061280 0x7fe31010a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:02.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.444+0000 7fe315acd700 1 --2- 172.21.15.113:0/3450391015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3100fa950 0x7fe31010a660 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:02.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.444+0000 7fe315acd700 1 -- 172.21.15.113:0/3450391015 >> 172.21.15.113:0/3450391015 conn(0x7fe3100f1e50 msgr2=0x7fe3100f61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:02.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.444+0000 7fe315acd700 1 -- 172.21.15.113:0/3450391015 shutdown_connections 2024-04-12T05:13:02.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:02.444+0000 7fe315acd700 1 -- 172.21.15.113:0/3450391015 wait complete. 2024-04-12T05:13:03.256 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-12T05:13:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:01.973880+0000 mgr.smithi113.ctsazg (mgr.14656) 117 : cluster [DBG] pgmap v66: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:03 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:02.442542+0000 mgr.smithi113.ctsazg (mgr.14656) 118 : audit [DBG] from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:01.973880+0000 mgr.smithi113.ctsazg (mgr.14656) 117 : cluster [DBG] pgmap v66: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:02.442542+0000 mgr.smithi113.ctsazg (mgr.14656) 118 : audit [DBG] from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:05.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.395+0000 7fccb7abf700 1 -- 172.21.15.113:0/671266436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb00fc7f0 msgr2=0x7fccb00fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:05.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.395+0000 7fccb7abf700 1 --2- 172.21.15.113:0/671266436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb00fc7f0 0x7fccb00fcbd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fccac0047f0 tx=0x7fccac009cb0).stop 2024-04-12T05:13:05.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.395+0000 7fccb7abf700 1 -- 172.21.15.113:0/671266436 shutdown_connections 2024-04-12T05:13:05.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.395+0000 7fccb7abf700 1 --2- 172.21.15.113:0/671266436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb00fd150 0x7fccb00f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:05.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.395+0000 7fccb7abf700 1 --2- 172.21.15.113:0/671266436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb00fc7f0 0x7fccb00fcbd0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:05.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.395+0000 7fccb7abf700 1 -- 172.21.15.113:0/671266436 >> 172.21.15.113:0/671266436 conn(0x7fccb0070590 msgr2=0x7fccb00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:05.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.395+0000 7fccb7abf700 1 -- 172.21.15.113:0/671266436 shutdown_connections 2024-04-12T05:13:05.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.395+0000 7fccb7abf700 1 -- 172.21.15.113:0/671266436 wait complete. 2024-04-12T05:13:05.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.395+0000 7fccb7abf700 1 Processor -- start 2024-04-12T05:13:05.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.395+0000 7fccb7abf700 1 -- start start 2024-04-12T05:13:05.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb7abf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb00fc7f0 0x7fccb0109120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb7abf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb00fd150 0x7fccb0109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb7abf700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0109b00 con 0x7fccb00fd150 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb7abf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0109c40 con 0x7fccb00fc7f0 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb505a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb00fd150 0x7fccb0109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb505a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb00fd150 0x7fccb0109610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:32968/0 (socket says 172.21.15.113:32968) 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb505a700 1 -- 172.21.15.113:0/255249311 learned_addr learned my addr 172.21.15.113:0/255249311 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb585b700 1 --2- 172.21.15.113:0/255249311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb00fc7f0 0x7fccb0109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb505a700 1 -- 172.21.15.113:0/255249311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb00fc7f0 msgr2=0x7fccb0109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb505a700 1 --2- 172.21.15.113:0/255249311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb00fc7f0 0x7fccb0109120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb505a700 1 -- 172.21.15.113:0/255249311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac005040 con 0x7fccb00fd150 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb505a700 1 --2- 172.21.15.113:0/255249311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb00fd150 0x7fccb0109610 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fcca0002820 tx=0x7fcca0009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fcca6f78700 1 -- 172.21.15.113:0/255249311 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcca0007b40 con 0x7fccb00fd150 2024-04-12T05:13:05.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fcca6f78700 1 -- 172.21.15.113:0/255249311 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcca000f040 con 0x7fccb00fd150 2024-04-12T05:13:05.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb7abf700 1 -- 172.21.15.113:0/255249311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb0109ef0 con 0x7fccb00fd150 2024-04-12T05:13:05.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fcca6f78700 1 -- 172.21.15.113:0/255249311 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcca00148b0 con 0x7fccb00fd150 2024-04-12T05:13:05.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.396+0000 7fccb7abf700 1 -- 172.21.15.113:0/255249311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb01124e0 con 0x7fccb00fd150 2024-04-12T05:13:05.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.397+0000 7fccb7abf700 1 -- 172.21.15.113:0/255249311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb005f590 con 0x7fccb00fd150 2024-04-12T05:13:05.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.398+0000 7fcca6f78700 1 -- 172.21.15.113:0/255249311 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7fcca0007ca0 con 0x7fccb00fd150 2024-04-12T05:13:05.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.399+0000 7fcca6f78700 1 --2- 172.21.15.113:0/255249311 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fcc9c06c770 0x7fcc9c06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:05.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.399+0000 7fcca6f78700 1 -- 172.21.15.113:0/255249311 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fcca008b390 con 0x7fccb00fd150 2024-04-12T05:13:05.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.399+0000 7fccb585b700 1 --2- 172.21.15.113:0/255249311 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fcc9c06c770 0x7fcc9c06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:05.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.399+0000 7fccb585b700 1 --2- 172.21.15.113:0/255249311 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fcc9c06c770 0x7fcc9c06ec30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fccac004f10 tx=0x7fccac00b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:05.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.403+0000 7fcca6f78700 1 -- 172.21.15.113:0/255249311 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fcca00559c0 con 0x7fccb00fd150 2024-04-12T05:13:05.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:03.974227+0000 mgr.smithi113.ctsazg (mgr.14656) 119 : cluster [DBG] pgmap v67: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:03.974227+0000 mgr.smithi113.ctsazg (mgr.14656) 119 : cluster [DBG] pgmap v67: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:05.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.667+0000 7fccb7abf700 1 -- 172.21.15.113:0/255249311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fccb0047540 con 0x7fccb00fd150 2024-04-12T05:13:05.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.668+0000 7fcca6f78700 1 -- 172.21.15.113:0/255249311 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7fcca0019300 con 0x7fccb00fd150 2024-04-12T05:13:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.670+0000 7fccb7abf700 1 -- 172.21.15.113:0/255249311 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fcc9c06c770 msgr2=0x7fcc9c06ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.670+0000 7fccb7abf700 1 --2- 172.21.15.113:0/255249311 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fcc9c06c770 0x7fcc9c06ec30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fccac004f10 tx=0x7fccac00b040).stop 2024-04-12T05:13:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.670+0000 7fccb7abf700 1 -- 172.21.15.113:0/255249311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb00fd150 msgr2=0x7fccb0109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.670+0000 7fccb7abf700 1 --2- 172.21.15.113:0/255249311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb00fd150 0x7fccb0109610 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fcca0002820 tx=0x7fcca0009f20).stop 2024-04-12T05:13:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.670+0000 7fccb7abf700 1 -- 172.21.15.113:0/255249311 shutdown_connections 2024-04-12T05:13:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.670+0000 7fccb7abf700 1 --2- 172.21.15.113:0/255249311 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fcc9c06c770 0x7fcc9c06ec30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.670+0000 7fccb7abf700 1 --2- 172.21.15.113:0/255249311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb00fc7f0 0x7fccb0109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.670+0000 7fccb7abf700 1 --2- 172.21.15.113:0/255249311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb00fd150 0x7fccb0109610 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.670+0000 7fccb7abf700 1 -- 172.21.15.113:0/255249311 >> 172.21.15.113:0/255249311 conn(0x7fccb0070590 msgr2=0x7fccb00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.670+0000 7fccb7abf700 1 -- 172.21.15.113:0/255249311 shutdown_connections 2024-04-12T05:13:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:05.670+0000 7fccb7abf700 1 -- 172.21.15.113:0/255249311 wait complete. 2024-04-12T05:13:05.682 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:13:06.401 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:06 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:05.668773+0000 mon.smithi113 (mon.0) 990 : audit [DBG] from='client.? 172.21.15.113:0/255249311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:13:06.422 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-12T05:13:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:05.668773+0000 mon.smithi113 (mon.0) 990 : audit [DBG] from='client.? 172.21.15.113:0/255249311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:13:07.111 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:07 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:05.974568+0000 mgr.smithi113.ctsazg (mgr.14656) 120 : cluster [DBG] pgmap v68: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:05.974568+0000 mgr.smithi113.ctsazg (mgr.14656) 120 : cluster [DBG] pgmap v68: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:08.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.571+0000 7f36cc2c2700 1 -- 172.21.15.113:0/1297566819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c40fd150 msgr2=0x7f36c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:08.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.571+0000 7f36cc2c2700 1 --2- 172.21.15.113:0/1297566819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c40fd150 0x7f36c40f6930 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f36c00047f0 tx=0x7f36c0009cb0).stop 2024-04-12T05:13:08.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 -- 172.21.15.113:0/1297566819 shutdown_connections 2024-04-12T05:13:08.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 --2- 172.21.15.113:0/1297566819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c40fd150 0x7f36c40f6930 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:08.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 --2- 172.21.15.113:0/1297566819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36c40fc7f0 0x7f36c40fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:08.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 -- 172.21.15.113:0/1297566819 >> 172.21.15.113:0/1297566819 conn(0x7f36c4070590 msgr2=0x7f36c40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:08.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 -- 172.21.15.113:0/1297566819 shutdown_connections 2024-04-12T05:13:08.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 -- 172.21.15.113:0/1297566819 wait complete. 2024-04-12T05:13:08.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 Processor -- start 2024-04-12T05:13:08.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 -- start start 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36c40fc7f0 0x7f36c410ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c40fd150 0x7f36c410d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c410d9c0 con 0x7f36c40fd150 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.572+0000 7f36cc2c2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c410db00 con 0x7f36c40fc7f0 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36c985d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c40fd150 0x7f36c410d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36c985d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c40fd150 0x7f36c410d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:32996/0 (socket says 172.21.15.113:32996) 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36c985d700 1 -- 172.21.15.113:0/2304815605 learned_addr learned my addr 172.21.15.113:0/2304815605 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36ca05e700 1 --2- 172.21.15.113:0/2304815605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36c40fc7f0 0x7f36c410ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36c985d700 1 -- 172.21.15.113:0/2304815605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36c40fc7f0 msgr2=0x7f36c410ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36c985d700 1 --2- 172.21.15.113:0/2304815605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36c40fc7f0 0x7f36c410ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36c985d700 1 -- 172.21.15.113:0/2304815605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c0005040 con 0x7f36c40fd150 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36c985d700 1 --2- 172.21.15.113:0/2304815605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c40fd150 0x7f36c410d3c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f36c00099f0 tx=0x7f36c0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36bb779700 1 -- 172.21.15.113:0/2304815605 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36c000e030 con 0x7f36c40fd150 2024-04-12T05:13:08.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36bb779700 1 -- 172.21.15.113:0/2304815605 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36c0007e90 con 0x7f36c40fd150 2024-04-12T05:13:08.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36cc2c2700 1 -- 172.21.15.113:0/2304815605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c4107130 con 0x7f36c40fd150 2024-04-12T05:13:08.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36bb779700 1 -- 172.21.15.113:0/2304815605 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36c0018070 con 0x7f36c40fd150 2024-04-12T05:13:08.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.573+0000 7f36cc2c2700 1 -- 172.21.15.113:0/2304815605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c4107690 con 0x7f36c40fd150 2024-04-12T05:13:08.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.574+0000 7f36cc2c2700 1 -- 172.21.15.113:0/2304815605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c405f590 con 0x7f36c40fd150 2024-04-12T05:13:08.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.574+0000 7f36bb779700 1 -- 172.21.15.113:0/2304815605 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f36c00094c0 con 0x7f36c40fd150 2024-04-12T05:13:08.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.575+0000 7f36bb779700 1 --2- 172.21.15.113:0/2304815605 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f36b006c6a0 0x7f36b006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:08.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.575+0000 7f36ca05e700 1 --2- 172.21.15.113:0/2304815605 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f36b006c6a0 0x7f36b006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:08.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.575+0000 7f36bb779700 1 -- 172.21.15.113:0/2304815605 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f36c008ca00 con 0x7f36c40fd150 2024-04-12T05:13:08.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.575+0000 7f36ca05e700 1 --2- 172.21.15.113:0/2304815605 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f36b006c6a0 0x7f36b006eb60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f36b4004dc0 tx=0x7f36b400b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:08.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.580+0000 7f36bb779700 1 -- 172.21.15.113:0/2304815605 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f36c005ab00 con 0x7f36c40fd150 2024-04-12T05:13:08.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.847+0000 7f36cc2c2700 1 -- 172.21.15.113:0/2304815605 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f36c4047540 con 0x7f36c40fd150 2024-04-12T05:13:08.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.847+0000 7f36bb779700 1 -- 172.21.15.113:0/2304815605 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f36c005a690 con 0x7f36c40fd150 2024-04-12T05:13:08.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.849+0000 7f36cc2c2700 1 -- 172.21.15.113:0/2304815605 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f36b006c6a0 msgr2=0x7f36b006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:08.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.849+0000 7f36cc2c2700 1 --2- 172.21.15.113:0/2304815605 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f36b006c6a0 0x7f36b006eb60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f36b4004dc0 tx=0x7f36b400b040).stop 2024-04-12T05:13:08.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.849+0000 7f36cc2c2700 1 -- 172.21.15.113:0/2304815605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c40fd150 msgr2=0x7f36c410d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:08.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.849+0000 7f36cc2c2700 1 --2- 172.21.15.113:0/2304815605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c40fd150 0x7f36c410d3c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f36c00099f0 tx=0x7f36c0009cb0).stop 2024-04-12T05:13:08.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.849+0000 7f36cc2c2700 1 -- 172.21.15.113:0/2304815605 shutdown_connections 2024-04-12T05:13:08.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.849+0000 7f36cc2c2700 1 --2- 172.21.15.113:0/2304815605 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f36b006c6a0 0x7f36b006eb60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:08.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.849+0000 7f36cc2c2700 1 --2- 172.21.15.113:0/2304815605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36c40fc7f0 0x7f36c410ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:08.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.849+0000 7f36cc2c2700 1 --2- 172.21.15.113:0/2304815605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36c40fd150 0x7f36c410d3c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:08.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.849+0000 7f36cc2c2700 1 -- 172.21.15.113:0/2304815605 >> 172.21.15.113:0/2304815605 conn(0x7f36c4070590 msgr2=0x7f36c40fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:08.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.849+0000 7f36cc2c2700 1 -- 172.21.15.113:0/2304815605 shutdown_connections 2024-04-12T05:13:08.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:08.849+0000 7f36cc2c2700 1 -- 172.21.15.113:0/2304815605 wait complete. 2024-04-12T05:13:08.861 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)" 2024-04-12T05:13:09.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:07.975007+0000 mgr.smithi113.ctsazg (mgr.14656) 121 : cluster [DBG] pgmap v69: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:09.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:09 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:08.848388+0000 mon.smithi113 (mon.0) 991 : audit [DBG] from='client.? 172.21.15.113:0/2304815605' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:13:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:07.975007+0000 mgr.smithi113.ctsazg (mgr.14656) 121 : cluster [DBG] pgmap v69: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:08.848388+0000 mon.smithi113 (mon.0) 991 : audit [DBG] from='client.? 172.21.15.113:0/2304815605' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:13:09.587 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-12T05:13:11.313 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:09.975350+0000 mgr.smithi113.ctsazg (mgr.14656) 122 : cluster [DBG] pgmap v70: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:11.314 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:11.023015+0000 mon.smithi113 (mon.0) 992 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:13:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:09.975350+0000 mgr.smithi113.ctsazg (mgr.14656) 122 : cluster [DBG] pgmap v70: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:11.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:11.023015+0000 mon.smithi113 (mon.0) 992 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:13:11.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.568+0000 7f9b724df700 1 -- 172.21.15.113:0/2983525801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b6c0fe210 msgr2=0x7f9b6c0f5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:11.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.568+0000 7f9b724df700 1 --2- 172.21.15.113:0/2983525801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b6c0fe210 0x7f9b6c0f5f80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f9b600047f0 tx=0x7f9b60009cb0).stop 2024-04-12T05:13:11.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.568+0000 7f9b724df700 1 -- 172.21.15.113:0/2983525801 shutdown_connections 2024-04-12T05:13:11.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.568+0000 7f9b724df700 1 --2- 172.21.15.113:0/2983525801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b6c0fe210 0x7f9b6c0f5f80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:11.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.568+0000 7f9b724df700 1 --2- 172.21.15.113:0/2983525801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b6c0fd8b0 0x7f9b6c0fdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:11.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.568+0000 7f9b724df700 1 -- 172.21.15.113:0/2983525801 >> 172.21.15.113:0/2983525801 conn(0x7f9b6c06ce40 msgr2=0x7f9b6c06d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:11.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.568+0000 7f9b724df700 1 -- 172.21.15.113:0/2983525801 shutdown_connections 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.568+0000 7f9b724df700 1 -- 172.21.15.113:0/2983525801 wait complete. 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b724df700 1 Processor -- start 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b724df700 1 -- start start 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b724df700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b6c0fd8b0 0x7f9b6c10dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b724df700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b6c0fe210 0x7f9b6c10e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b724df700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c10eae0 con 0x7f9b6c0fe210 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b724df700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c10ec20 con 0x7f9b6c0fd8b0 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b6b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b6c0fe210 0x7f9b6c10e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b6b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b6c0fe210 0x7f9b6c10e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33704/0 (socket says 172.21.15.113:33704) 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b6b7fe700 1 -- 172.21.15.113:0/3930197978 learned_addr learned my addr 172.21.15.113:0/3930197978 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b6bfff700 1 --2- 172.21.15.113:0/3930197978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b6c0fd8b0 0x7f9b6c10dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b6b7fe700 1 -- 172.21.15.113:0/3930197978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b6c0fd8b0 msgr2=0x7f9b6c10dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:11.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b6b7fe700 1 --2- 172.21.15.113:0/3930197978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b6c0fd8b0 0x7f9b6c10dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.569+0000 7f9b6b7fe700 1 -- 172.21.15.113:0/3930197978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b60005040 con 0x7f9b6c0fe210 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.570+0000 7f9b6b7fe700 1 --2- 172.21.15.113:0/3930197978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b6c0fe210 0x7f9b6c10e4e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f9b60005010 tx=0x7f9b60009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.570+0000 7f9b697fa700 1 -- 172.21.15.113:0/3930197978 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b6000e030 con 0x7f9b6c0fe210 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.570+0000 7f9b724df700 1 -- 172.21.15.113:0/3930197978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6c108250 con 0x7f9b6c0fe210 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.570+0000 7f9b697fa700 1 -- 172.21.15.113:0/3930197978 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b60007e90 con 0x7f9b6c0fe210 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.570+0000 7f9b697fa700 1 -- 172.21.15.113:0/3930197978 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b60018070 con 0x7f9b6c0fe210 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.570+0000 7f9b724df700 1 -- 172.21.15.113:0/3930197978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6c108730 con 0x7f9b6c0fe210 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.571+0000 7f9b724df700 1 -- 172.21.15.113:0/3930197978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6c060670 con 0x7f9b6c0fe210 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.571+0000 7f9b697fa700 1 -- 172.21.15.113:0/3930197978 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f9b6001d670 con 0x7f9b6c0fe210 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.572+0000 7f9b697fa700 1 --2- 172.21.15.113:0/3930197978 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9b5406c6a0 0x7f9b5406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.572+0000 7f9b697fa700 1 -- 172.21.15.113:0/3930197978 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f9b6008c9d0 con 0x7f9b6c0fe210 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.572+0000 7f9b6bfff700 1 --2- 172.21.15.113:0/3930197978 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9b5406c6a0 0x7f9b5406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.572+0000 7f9b6bfff700 1 --2- 172.21.15.113:0/3930197978 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9b5406c6a0 0x7f9b5406eb60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9b5c004dc0 tx=0x7f9b5c00b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:11.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.576+0000 7f9b697fa700 1 -- 172.21.15.113:0/3930197978 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f9b6005aad0 con 0x7f9b6c0fe210 2024-04-12T05:13:11.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.844+0000 7f9b724df700 1 -- 172.21.15.113:0/3930197978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9b6c109710 con 0x7f9b6c0fe210 2024-04-12T05:13:11.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.844+0000 7f9b697fa700 1 -- 172.21.15.113:0/3930197978 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7f9b6005a660 con 0x7f9b6c0fe210 2024-04-12T05:13:11.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.846+0000 7f9b724df700 1 -- 172.21.15.113:0/3930197978 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9b5406c6a0 msgr2=0x7f9b5406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:11.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.846+0000 7f9b724df700 1 --2- 172.21.15.113:0/3930197978 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9b5406c6a0 0x7f9b5406eb60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9b5c004dc0 tx=0x7f9b5c00b040).stop 2024-04-12T05:13:11.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.846+0000 7f9b724df700 1 -- 172.21.15.113:0/3930197978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b6c0fe210 msgr2=0x7f9b6c10e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:11.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.846+0000 7f9b724df700 1 --2- 172.21.15.113:0/3930197978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b6c0fe210 0x7f9b6c10e4e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f9b60005010 tx=0x7f9b60009cb0).stop 2024-04-12T05:13:11.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.847+0000 7f9b724df700 1 -- 172.21.15.113:0/3930197978 shutdown_connections 2024-04-12T05:13:11.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.847+0000 7f9b724df700 1 --2- 172.21.15.113:0/3930197978 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f9b5406c6a0 0x7f9b5406eb60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:11.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.847+0000 7f9b724df700 1 --2- 172.21.15.113:0/3930197978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b6c0fd8b0 0x7f9b6c10dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:11.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.847+0000 7f9b724df700 1 --2- 172.21.15.113:0/3930197978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b6c0fe210 0x7f9b6c10e4e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:11.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.847+0000 7f9b724df700 1 -- 172.21.15.113:0/3930197978 >> 172.21.15.113:0/3930197978 conn(0x7f9b6c06ce40 msgr2=0x7f9b6c100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:11.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.847+0000 7f9b724df700 1 -- 172.21.15.113:0/3930197978 shutdown_connections 2024-04-12T05:13:11.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:11.847+0000 7f9b724df700 1 -- 172.21.15.113:0/3930197978 wait complete. 2024-04-12T05:13:11.859 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:13:12.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:12 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:11.845530+0000 mon.smithi113 (mon.0) 993 : audit [DBG] from='client.? 172.21.15.113:0/3930197978' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:13:12.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:11.845530+0000 mon.smithi113 (mon.0) 993 : audit [DBG] from='client.? 172.21.15.113:0/3930197978' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:13:12.635 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-12T05:13:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:13 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:11.975878+0000 mgr.smithi113.ctsazg (mgr.14656) 123 : cluster [DBG] pgmap v71: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:11.975878+0000 mgr.smithi113.ctsazg (mgr.14656) 123 : cluster [DBG] pgmap v71: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:14.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.630+0000 7fe0a7867700 1 -- 172.21.15.113:0/2614716739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0a006b9c0 msgr2=0x7fe0a0103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:14.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.630+0000 7fe0a7867700 1 --2- 172.21.15.113:0/2614716739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0a006b9c0 0x7fe0a0103cc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fe09c0047f0 tx=0x7fe09c009cb0).stop 2024-04-12T05:13:14.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.630+0000 7fe0a7867700 1 -- 172.21.15.113:0/2614716739 shutdown_connections 2024-04-12T05:13:14.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.630+0000 7fe0a7867700 1 --2- 172.21.15.113:0/2614716739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0a006b9c0 0x7fe0a0103cc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:14.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.630+0000 7fe0a7867700 1 --2- 172.21.15.113:0/2614716739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0a006b0f0 0x7fe0a006b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:14.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.630+0000 7fe0a7867700 1 -- 172.21.15.113:0/2614716739 >> 172.21.15.113:0/2614716739 conn(0x7fe0a00f34b0 msgr2=0x7fe0a00f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:14.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.630+0000 7fe0a7867700 1 -- 172.21.15.113:0/2614716739 shutdown_connections 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.630+0000 7fe0a7867700 1 -- 172.21.15.113:0/2614716739 wait complete. 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.631+0000 7fe0a7867700 1 Processor -- start 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.631+0000 7fe0a7867700 1 -- start start 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.631+0000 7fe0a7867700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0a006b0f0 0x7fe0a0110320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.631+0000 7fe0a7867700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0a006b9c0 0x7fe0a0110810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.631+0000 7fe0a7867700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a0110e10 con 0x7fe0a006b0f0 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.631+0000 7fe0a7867700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a0110f50 con 0x7fe0a006b9c0 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.631+0000 7fe0a5603700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0a006b0f0 0x7fe0a0110320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.631+0000 7fe0a5603700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0a006b0f0 0x7fe0a0110320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33724/0 (socket says 172.21.15.113:33724) 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.631+0000 7fe0a5603700 1 -- 172.21.15.113:0/3670442669 learned_addr learned my addr 172.21.15.113:0/3670442669 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.632+0000 7fe0a4e02700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0a006b9c0 0x7fe0a0110810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.632+0000 7fe0a5603700 1 -- 172.21.15.113:0/3670442669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0a006b9c0 msgr2=0x7fe0a0110810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.632+0000 7fe0a5603700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0a006b9c0 0x7fe0a0110810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:14.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.632+0000 7fe0a5603700 1 -- 172.21.15.113:0/3670442669 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c005040 con 0x7fe0a006b0f0 2024-04-12T05:13:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.632+0000 7fe0a4e02700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0a006b9c0 0x7fe0a0110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:13:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.632+0000 7fe0a5603700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0a006b0f0 0x7fe0a0110320 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fe0900041d0 tx=0x7fe090009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.632+0000 7fe0967fc700 1 -- 172.21.15.113:0/3670442669 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe090007b40 con 0x7fe0a006b0f0 2024-04-12T05:13:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.632+0000 7fe0a7867700 1 -- 172.21.15.113:0/3670442669 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a010a5e0 con 0x7fe0a006b0f0 2024-04-12T05:13:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.632+0000 7fe0967fc700 1 -- 172.21.15.113:0/3670442669 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe09000f040 con 0x7fe0a006b0f0 2024-04-12T05:13:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.632+0000 7fe0967fc700 1 -- 172.21.15.113:0/3670442669 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe09000d800 con 0x7fe0a006b0f0 2024-04-12T05:13:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.632+0000 7fe0a7867700 1 -- 172.21.15.113:0/3670442669 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a010ab20 con 0x7fe0a006b0f0 2024-04-12T05:13:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.633+0000 7fe0a7867700 1 -- 172.21.15.113:0/3670442669 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a005f590 con 0x7fe0a006b0f0 2024-04-12T05:13:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.634+0000 7fe0967fc700 1 -- 172.21.15.113:0/3670442669 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7fe090007ca0 con 0x7fe0a006b0f0 2024-04-12T05:13:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.634+0000 7fe0967fc700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe08c06c6a0 0x7fe08c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.634+0000 7fe0967fc700 1 -- 172.21.15.113:0/3670442669 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fe09008b250 con 0x7fe0a006b0f0 2024-04-12T05:13:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.634+0000 7fe0a4e02700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe08c06c6a0 0x7fe08c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.634+0000 7fe0a4e02700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe08c06c6a0 0x7fe08c06eb60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fe09c004540 tx=0x7fe09c00b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:14.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.639+0000 7fe0967fc700 1 -- 172.21.15.113:0/3670442669 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fe090059350 con 0x7fe0a006b0f0 2024-04-12T05:13:14.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:14.842+0000 7fe0a7867700 1 -- 172.21.15.113:0/3670442669 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0a010bab0 con 0x7fe08c06c6a0 2024-04-12T05:13:15.388 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:13.976319+0000 mgr.smithi113.ctsazg (mgr.14656) 124 : cluster [DBG] pgmap v72: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:13.976319+0000 mgr.smithi113.ctsazg (mgr.14656) 124 : cluster [DBG] pgmap v72: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:16.316 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:16 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:14.844046+0000 mgr.smithi113.ctsazg (mgr.14656) 125 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:16.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.361+0000 7fe0967fc700 1 -- 172.21.15.113:0/3670442669 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5081 (secure 0 0 0) 0x7fe0a010bab0 con 0x7fe08c06c6a0 2024-04-12T05:13:16.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.364+0000 7fe0a7867700 1 -- 172.21.15.113:0/3670442669 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe08c06c6a0 msgr2=0x7fe08c06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:16.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.364+0000 7fe0a7867700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe08c06c6a0 0x7fe08c06eb60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fe09c004540 tx=0x7fe09c00b040).stop 2024-04-12T05:13:16.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.364+0000 7fe0a7867700 1 -- 172.21.15.113:0/3670442669 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0a006b0f0 msgr2=0x7fe0a0110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:16.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.364+0000 7fe0a7867700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0a006b0f0 0x7fe0a0110320 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fe0900041d0 tx=0x7fe090009f60).stop 2024-04-12T05:13:16.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.364+0000 7fe0a7867700 1 -- 172.21.15.113:0/3670442669 shutdown_connections 2024-04-12T05:13:16.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.364+0000 7fe0a7867700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fe08c06c6a0 0x7fe08c06eb60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:16.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.364+0000 7fe0a7867700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0a006b0f0 0x7fe0a0110320 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:16.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.364+0000 7fe0a7867700 1 --2- 172.21.15.113:0/3670442669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0a006b9c0 0x7fe0a0110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:16.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.364+0000 7fe0a7867700 1 -- 172.21.15.113:0/3670442669 >> 172.21.15.113:0/3670442669 conn(0x7fe0a00f34b0 msgr2=0x7fe0a0103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:16.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.364+0000 7fe0a7867700 1 -- 172.21.15.113:0/3670442669 shutdown_connections 2024-04-12T05:13:16.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:16.364+0000 7fe0a7867700 1 -- 172.21.15.113:0/3670442669 wait complete. 2024-04-12T05:13:16.377 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:13:16.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:14.844046+0000 mgr.smithi113.ctsazg (mgr.14656) 125 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:17.041 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch ps' 2024-04-12T05:13:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:15.976701+0000 mgr.smithi113.ctsazg (mgr.14656) 126 : cluster [DBG] pgmap v73: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:17.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:15.976701+0000 mgr.smithi113.ctsazg (mgr.14656) 126 : cluster [DBG] pgmap v73: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:18.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.871+0000 7f91e3437700 1 -- 172.21.15.113:0/2045980259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91dc0fd150 msgr2=0x7f91dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:18.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.871+0000 7f91e3437700 1 --2- 172.21.15.113:0/2045980259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91dc0fd150 0x7f91dc0f6930 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f91d80047f0 tx=0x7f91d8009cb0).stop 2024-04-12T05:13:18.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.871+0000 7f91e3437700 1 -- 172.21.15.113:0/2045980259 shutdown_connections 2024-04-12T05:13:18.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.871+0000 7f91e3437700 1 --2- 172.21.15.113:0/2045980259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91dc0fd150 0x7f91dc0f6930 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.871+0000 7f91e3437700 1 --2- 172.21.15.113:0/2045980259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc0fc7f0 0x7f91dc0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.871+0000 7f91e3437700 1 -- 172.21.15.113:0/2045980259 >> 172.21.15.113:0/2045980259 conn(0x7f91dc070590 msgr2=0x7f91dc0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.871+0000 7f91e3437700 1 -- 172.21.15.113:0/2045980259 shutdown_connections 2024-04-12T05:13:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.871+0000 7f91e3437700 1 -- 172.21.15.113:0/2045980259 wait complete. 2024-04-12T05:13:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.872+0000 7f91e3437700 1 Processor -- start 2024-04-12T05:13:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.872+0000 7f91e3437700 1 -- start start 2024-04-12T05:13:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.872+0000 7f91e3437700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc0fc7f0 0x7f91dc10e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.872+0000 7f91e3437700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91dc0fd150 0x7f91dc10e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.872+0000 7f91e3437700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc10ebe0 con 0x7f91dc0fd150 2024-04-12T05:13:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.872+0000 7f91e3437700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc108200 con 0x7f91dc0fc7f0 2024-04-12T05:13:18.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.872+0000 7f91e09d2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91dc0fd150 0x7f91dc10e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.872+0000 7f91e09d2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91dc0fd150 0x7f91dc10e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33746/0 (socket says 172.21.15.113:33746) 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.872+0000 7f91e09d2700 1 -- 172.21.15.113:0/1110539548 learned_addr learned my addr 172.21.15.113:0/1110539548 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.873+0000 7f91e09d2700 1 -- 172.21.15.113:0/1110539548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc0fc7f0 msgr2=0x7f91dc10e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.873+0000 7f91e11d3700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc0fc7f0 0x7f91dc10e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.873+0000 7f91e09d2700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc0fc7f0 0x7f91dc10e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.873+0000 7f91e09d2700 1 -- 172.21.15.113:0/1110539548 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d8005040 con 0x7f91dc0fd150 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.873+0000 7f91e11d3700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc0fc7f0 0x7f91dc10e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.873+0000 7f91e09d2700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91dc0fd150 0x7f91dc10e550 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f91d80099f0 tx=0x7f91d8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.873+0000 7f91d27fc700 1 -- 172.21.15.113:0/1110539548 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91d800e030 con 0x7f91dc0fd150 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.873+0000 7f91d27fc700 1 -- 172.21.15.113:0/1110539548 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91d8007e90 con 0x7f91dc0fd150 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.873+0000 7f91e3437700 1 -- 172.21.15.113:0/1110539548 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91dc108450 con 0x7f91dc0fd150 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.873+0000 7f91d27fc700 1 -- 172.21.15.113:0/1110539548 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91d8018070 con 0x7f91dc0fd150 2024-04-12T05:13:18.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.873+0000 7f91e3437700 1 -- 172.21.15.113:0/1110539548 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91dc108930 con 0x7f91dc0fd150 2024-04-12T05:13:18.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.874+0000 7f91e3437700 1 -- 172.21.15.113:0/1110539548 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91dc05f590 con 0x7f91dc0fd150 2024-04-12T05:13:18.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.874+0000 7f91d27fc700 1 -- 172.21.15.113:0/1110539548 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f91d801d420 con 0x7f91dc0fd150 2024-04-12T05:13:18.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.875+0000 7f91d27fc700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f91c806c6a0 0x7f91c806eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:18.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.875+0000 7f91d27fc700 1 -- 172.21.15.113:0/1110539548 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f91d808cbf0 con 0x7f91dc0fd150 2024-04-12T05:13:18.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.875+0000 7f91e11d3700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f91c806c6a0 0x7f91c806eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:18.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.875+0000 7f91e11d3700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f91c806c6a0 0x7f91c806eb60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f91cc00a900 tx=0x7f91cc008040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:18.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:18.879+0000 7f91d27fc700 1 -- 172.21.15.113:0/1110539548 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f91d805acf0 con 0x7f91dc0fd150 2024-04-12T05:13:19.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.077+0000 7f91e3437700 1 -- 172.21.15.113:0/1110539548 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f91dc109780 con 0x7f91c806c6a0 2024-04-12T05:13:19.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.086+0000 7f91d27fc700 1 -- 172.21.15.113:0/1110539548 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f91dc109780 con 0x7f91c806c6a0 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (70s) 49s ago 12m 15.2M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (12m) 49s ago 12m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (11m) 65s ago 11m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (54s) 49s ago 12m 38.2M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (7m) 49s ago 7m 120M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (6m) 49s ago 6m 17.0M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (7m) 65s ago 7m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (6m) 65s ago 6m 14.4M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (2m) 49s ago 13m 498M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (2m) 65s ago 11m 446M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (13m) 49s ago 13m 59.8M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:13:19.089 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (11m) 65s ago 11m 49.8M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:13:19.090 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (107s) 49s ago 12m 8744k - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:13:19.090 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (101s) 65s ago 11m 15.8M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:13:19.090 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (10m) 49s ago 10m 226M 1910M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:13:19.090 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (10m) 49s ago 10m 240M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:13:19.090 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (9m) 49s ago 9m 202M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:13:19.090 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (9m) 65s ago 9m 263M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:13:19.090 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (9m) 65s ago 9m 233M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:13:19.090 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (8m) 65s ago 8m 226M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:13:19.090 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (85s) 65s ago 11m 42.0M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:13:19.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.088+0000 7f91e3437700 1 -- 172.21.15.113:0/1110539548 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f91c806c6a0 msgr2=0x7f91c806eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:19.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.088+0000 7f91e3437700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f91c806c6a0 0x7f91c806eb60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f91cc00a900 tx=0x7f91cc008040).stop 2024-04-12T05:13:19.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.088+0000 7f91e3437700 1 -- 172.21.15.113:0/1110539548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91dc0fd150 msgr2=0x7f91dc10e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:19.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.088+0000 7f91e3437700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91dc0fd150 0x7f91dc10e550 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f91d80099f0 tx=0x7f91d8009cb0).stop 2024-04-12T05:13:19.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.088+0000 7f91e3437700 1 -- 172.21.15.113:0/1110539548 shutdown_connections 2024-04-12T05:13:19.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.088+0000 7f91e3437700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f91c806c6a0 0x7f91c806eb60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:19.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.089+0000 7f91e3437700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc0fc7f0 0x7f91dc10e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:19.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.089+0000 7f91e3437700 1 --2- 172.21.15.113:0/1110539548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91dc0fd150 0x7f91dc10e550 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:19.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.089+0000 7f91e3437700 1 -- 172.21.15.113:0/1110539548 >> 172.21.15.113:0/1110539548 conn(0x7f91dc070590 msgr2=0x7f91dc060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:19.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.089+0000 7f91e3437700 1 -- 172.21.15.113:0/1110539548 shutdown_connections 2024-04-12T05:13:19.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:19.089+0000 7f91e3437700 1 -- 172.21.15.113:0/1110539548 wait complete. 2024-04-12T05:13:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:19 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:17.977204+0000 mgr.smithi113.ctsazg (mgr.14656) 127 : cluster [DBG] pgmap v74: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:17.977204+0000 mgr.smithi113.ctsazg (mgr.14656) 127 : cluster [DBG] pgmap v74: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:19.730 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-12T05:13:19.731 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:13:19.731 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-04-12T05:13:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:20 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:19.078651+0000 mgr.smithi113.ctsazg (mgr.14656) 128 : audit [DBG] from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:20 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:19.078651+0000 mgr.smithi113.ctsazg (mgr.14656) 128 : audit [DBG] from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:21.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:19.977615+0000 mgr.smithi113.ctsazg (mgr.14656) 129 : cluster [DBG] pgmap v75: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:21 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:19.977615+0000 mgr.smithi113.ctsazg (mgr.14656) 129 : cluster [DBG] pgmap v75: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:21.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.776+0000 7fb092290700 1 -- 172.21.15.113:0/822128844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08c0fd8f0 msgr2=0x7fb08c0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:21.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.776+0000 7fb092290700 1 --2- 172.21.15.113:0/822128844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08c0fd8f0 0x7fb08c0fdcd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fb07c0047f0 tx=0x7fb07c009cb0).stop 2024-04-12T05:13:21.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.776+0000 7fb092290700 1 -- 172.21.15.113:0/822128844 shutdown_connections 2024-04-12T05:13:21.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.776+0000 7fb092290700 1 --2- 172.21.15.113:0/822128844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb08c0fe250 0x7fb08c0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:21.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.776+0000 7fb092290700 1 --2- 172.21.15.113:0/822128844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08c0fd8f0 0x7fb08c0fdcd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:21.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.776+0000 7fb092290700 1 -- 172.21.15.113:0/822128844 >> 172.21.15.113:0/822128844 conn(0x7fb08c0f34b0 msgr2=0x7fb08c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:21.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.776+0000 7fb092290700 1 -- 172.21.15.113:0/822128844 shutdown_connections 2024-04-12T05:13:21.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.776+0000 7fb092290700 1 -- 172.21.15.113:0/822128844 wait complete. 2024-04-12T05:13:21.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.776+0000 7fb092290700 1 Processor -- start 2024-04-12T05:13:21.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.776+0000 7fb092290700 1 -- start start 2024-04-12T05:13:21.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb092290700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb08c0fd8f0 0x7fb08c10dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:21.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb092290700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08c0fe250 0x7fb08c10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:21.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb092290700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c10ead0 con 0x7fb08c0fe250 2024-04-12T05:13:21.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb092290700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c10ec10 con 0x7fb08c0fd8f0 2024-04-12T05:13:21.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb08b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08c0fe250 0x7fb08c10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:21.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb08b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08c0fe250 0x7fb08c10e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42446/0 (socket says 172.21.15.113:42446) 2024-04-12T05:13:21.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb08b7fe700 1 -- 172.21.15.113:0/597910131 learned_addr learned my addr 172.21.15.113:0/597910131 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:21.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb08bfff700 1 --2- 172.21.15.113:0/597910131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb08c0fd8f0 0x7fb08c10dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:21.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb08b7fe700 1 -- 172.21.15.113:0/597910131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb08c0fd8f0 msgr2=0x7fb08c10dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:21.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb08b7fe700 1 --2- 172.21.15.113:0/597910131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb08c0fd8f0 0x7fb08c10dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb08b7fe700 1 -- 172.21.15.113:0/597910131 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07c005040 con 0x7fb08c0fe250 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb08b7fe700 1 --2- 172.21.15.113:0/597910131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08c0fe250 0x7fb08c10e4d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fb080002780 tx=0x7fb080009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb089775700 1 -- 172.21.15.113:0/597910131 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb080007b40 con 0x7fb08c0fe250 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb089775700 1 -- 172.21.15.113:0/597910131 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb08000f040 con 0x7fb08c0fe250 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb092290700 1 -- 172.21.15.113:0/597910131 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08c1082a0 con 0x7fb08c0fe250 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb089775700 1 -- 172.21.15.113:0/597910131 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb080014780 con 0x7fb08c0fe250 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.777+0000 7fb092290700 1 -- 172.21.15.113:0/597910131 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08c108860 con 0x7fb08c0fe250 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.778+0000 7fb092290700 1 -- 172.21.15.113:0/597910131 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08c05f590 con 0x7fb08c0fe250 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.779+0000 7fb089775700 1 -- 172.21.15.113:0/597910131 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7fb080007ca0 con 0x7fb08c0fe250 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.779+0000 7fb089775700 1 --2- 172.21.15.113:0/597910131 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb07406c6a0 0x7fb07406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.779+0000 7fb089775700 1 -- 172.21.15.113:0/597910131 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fb08008b650 con 0x7fb08c0fe250 2024-04-12T05:13:21.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.779+0000 7fb08bfff700 1 --2- 172.21.15.113:0/597910131 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb07406c6a0 0x7fb07406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:21.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.780+0000 7fb08bfff700 1 --2- 172.21.15.113:0/597910131 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb07406c6a0 0x7fb07406eb60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb07c004f10 tx=0x7fb07c00b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:21.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.784+0000 7fb089775700 1 -- 172.21.15.113:0/597910131 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fb080059750 con 0x7fb08c0fe250 2024-04-12T05:13:21.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.973+0000 7fb092290700 1 -- 172.21.15.113:0/597910131 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7fb08c047540 con 0x7fb08c0fe250 2024-04-12T05:13:21.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.977+0000 7fb089775700 1 -- 172.21.15.113:0/597910131 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v49)=0 v49) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb0800592e0 con 0x7fb08c0fe250 2024-04-12T05:13:21.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.980+0000 7fb092290700 1 -- 172.21.15.113:0/597910131 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb07406c6a0 msgr2=0x7fb07406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:21.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.980+0000 7fb092290700 1 --2- 172.21.15.113:0/597910131 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb07406c6a0 0x7fb07406eb60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb07c004f10 tx=0x7fb07c00b040).stop 2024-04-12T05:13:21.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.980+0000 7fb092290700 1 -- 172.21.15.113:0/597910131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08c0fe250 msgr2=0x7fb08c10e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:21.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.980+0000 7fb092290700 1 --2- 172.21.15.113:0/597910131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08c0fe250 0x7fb08c10e4d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fb080002780 tx=0x7fb080009ee0).stop 2024-04-12T05:13:21.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.980+0000 7fb092290700 1 -- 172.21.15.113:0/597910131 shutdown_connections 2024-04-12T05:13:21.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.980+0000 7fb092290700 1 --2- 172.21.15.113:0/597910131 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb07406c6a0 0x7fb07406eb60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:21.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.980+0000 7fb092290700 1 --2- 172.21.15.113:0/597910131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb08c0fd8f0 0x7fb08c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:21.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.980+0000 7fb092290700 1 --2- 172.21.15.113:0/597910131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb08c0fe250 0x7fb08c10e4d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:21.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.980+0000 7fb092290700 1 -- 172.21.15.113:0/597910131 >> 172.21.15.113:0/597910131 conn(0x7fb08c0f34b0 msgr2=0x7fb08c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:21.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.980+0000 7fb092290700 1 -- 172.21.15.113:0/597910131 shutdown_connections 2024-04-12T05:13:21.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:21.980+0000 7fb092290700 1 -- 172.21.15.113:0/597910131 wait complete. 2024-04-12T05:13:22.626 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-12T05:13:22.627 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:13:22.627 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-12T05:13:23.283 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:21.978047+0000 mgr.smithi113.ctsazg (mgr.14656) 130 : cluster [DBG] pgmap v76: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:23.283 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:21.978596+0000 mon.smithi113 (mon.0) 994 : audit [INF] from='client.? 172.21.15.113:0/597910131' entity='client.admin' 2024-04-12T05:13:23.283 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:22.030460+0000 mon.smithi113 (mon.0) 995 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:13:23.283 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:22.450748+0000 mon.smithi113 (mon.0) 996 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:13:23.283 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:22.451407+0000 mon.smithi113 (mon.0) 997 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:13:23.283 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:22 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:22.455955+0000 mon.smithi113 (mon.0) 998 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:21.978047+0000 mgr.smithi113.ctsazg (mgr.14656) 130 : cluster [DBG] pgmap v76: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:21.978596+0000 mon.smithi113 (mon.0) 994 : audit [INF] from='client.? 172.21.15.113:0/597910131' entity='client.admin' 2024-04-12T05:13:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:22.030460+0000 mon.smithi113 (mon.0) 995 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:13:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:22.450748+0000 mon.smithi113 (mon.0) 996 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:13:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:22.451407+0000 mon.smithi113 (mon.0) 997 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:13:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:22 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:22.455955+0000 mon.smithi113 (mon.0) 998 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:24.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.704+0000 7f12fb966700 1 -- 172.21.15.113:0/3888541240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12f40fd150 msgr2=0x7f12f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:24.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.704+0000 7f12fb966700 1 --2- 172.21.15.113:0/3888541240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12f40fd150 0x7f12f40f6930 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f12f00047f0 tx=0x7f12f0009cb0).stop 2024-04-12T05:13:24.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.704+0000 7f12fb966700 1 -- 172.21.15.113:0/3888541240 shutdown_connections 2024-04-12T05:13:24.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.704+0000 7f12fb966700 1 --2- 172.21.15.113:0/3888541240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12f40fd150 0x7f12f40f6930 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:24.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.704+0000 7f12fb966700 1 --2- 172.21.15.113:0/3888541240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12f40fc7f0 0x7f12f40fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:24.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.704+0000 7f12fb966700 1 -- 172.21.15.113:0/3888541240 >> 172.21.15.113:0/3888541240 conn(0x7f12f4070590 msgr2=0x7f12f40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.704+0000 7f12fb966700 1 -- 172.21.15.113:0/3888541240 shutdown_connections 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.704+0000 7f12fb966700 1 -- 172.21.15.113:0/3888541240 wait complete. 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12fb966700 1 Processor -- start 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12fb966700 1 -- start start 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12fb966700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12f40fc7f0 0x7f12f410e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12fb966700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12f40fd150 0x7f12f410e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12fb966700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f410ebe0 con 0x7f12f40fd150 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12fb966700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f4108200 con 0x7f12f40fc7f0 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12f8f01700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12f40fd150 0x7f12f410e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12f8f01700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12f40fd150 0x7f12f410e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42476/0 (socket says 172.21.15.113:42476) 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12f8f01700 1 -- 172.21.15.113:0/2592475515 learned_addr learned my addr 172.21.15.113:0/2592475515 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12f9702700 1 --2- 172.21.15.113:0/2592475515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12f40fc7f0 0x7f12f410e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12f8f01700 1 -- 172.21.15.113:0/2592475515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12f40fc7f0 msgr2=0x7f12f410e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:24.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12f8f01700 1 --2- 172.21.15.113:0/2592475515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12f40fc7f0 0x7f12f410e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:24.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12f8f01700 1 -- 172.21.15.113:0/2592475515 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f0005040 con 0x7f12f40fd150 2024-04-12T05:13:24.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12f8f01700 1 --2- 172.21.15.113:0/2592475515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12f40fd150 0x7f12f410e550 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f12f00099f0 tx=0x7f12f0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:24.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.705+0000 7f12ea7fc700 1 -- 172.21.15.113:0/2592475515 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12f000e030 con 0x7f12f40fd150 2024-04-12T05:13:24.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.706+0000 7f12ea7fc700 1 -- 172.21.15.113:0/2592475515 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12f0007e90 con 0x7f12f40fd150 2024-04-12T05:13:24.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.706+0000 7f12fb966700 1 -- 172.21.15.113:0/2592475515 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f4108450 con 0x7f12f40fd150 2024-04-12T05:13:24.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.706+0000 7f12ea7fc700 1 -- 172.21.15.113:0/2592475515 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12f0018070 con 0x7f12f40fd150 2024-04-12T05:13:24.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.706+0000 7f12fb966700 1 -- 172.21.15.113:0/2592475515 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f4108930 con 0x7f12f40fd150 2024-04-12T05:13:24.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.706+0000 7f12fb966700 1 -- 172.21.15.113:0/2592475515 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f405f590 con 0x7f12f40fd150 2024-04-12T05:13:24.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.707+0000 7f12ea7fc700 1 -- 172.21.15.113:0/2592475515 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f12f001d420 con 0x7f12f40fd150 2024-04-12T05:13:24.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.707+0000 7f12ea7fc700 1 --2- 172.21.15.113:0/2592475515 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f12e006c6a0 0x7f12e006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:24.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.708+0000 7f12f9702700 1 --2- 172.21.15.113:0/2592475515 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f12e006c6a0 0x7f12e006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:24.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.708+0000 7f12ea7fc700 1 -- 172.21.15.113:0/2592475515 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f12f008cbf0 con 0x7f12f40fd150 2024-04-12T05:13:24.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.708+0000 7f12f9702700 1 --2- 172.21.15.113:0/2592475515 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f12e006c6a0 0x7f12e006eb60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f12e400b910 tx=0x7f12e4008040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:24.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.712+0000 7f12ea7fc700 1 -- 172.21.15.113:0/2592475515 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f12f005acf0 con 0x7f12f40fd150 2024-04-12T05:13:24.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.898+0000 7f12fb966700 1 -- 172.21.15.113:0/2592475515 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f12f41096e0 con 0x7f12f40fd150 2024-04-12T05:13:24.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.899+0000 7f12ea7fc700 1 -- 172.21.15.113:0/2592475515 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v49)=0 v49) v1 ==== 155+0+0 (secure 0 0 0) 0x7f12f005a880 con 0x7f12f40fd150 2024-04-12T05:13:24.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.901+0000 7f12fb966700 1 -- 172.21.15.113:0/2592475515 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f12e006c6a0 msgr2=0x7f12e006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:24.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.901+0000 7f12fb966700 1 --2- 172.21.15.113:0/2592475515 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f12e006c6a0 0x7f12e006eb60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f12e400b910 tx=0x7f12e4008040).stop 2024-04-12T05:13:24.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.901+0000 7f12fb966700 1 -- 172.21.15.113:0/2592475515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12f40fd150 msgr2=0x7f12f410e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:24.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.901+0000 7f12fb966700 1 --2- 172.21.15.113:0/2592475515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12f40fd150 0x7f12f410e550 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f12f00099f0 tx=0x7f12f0009cb0).stop 2024-04-12T05:13:24.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.901+0000 7f12fb966700 1 -- 172.21.15.113:0/2592475515 shutdown_connections 2024-04-12T05:13:24.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.901+0000 7f12fb966700 1 --2- 172.21.15.113:0/2592475515 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f12e006c6a0 0x7f12e006eb60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:24.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.901+0000 7f12fb966700 1 --2- 172.21.15.113:0/2592475515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12f40fc7f0 0x7f12f410e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:24.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.901+0000 7f12fb966700 1 --2- 172.21.15.113:0/2592475515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12f40fd150 0x7f12f410e550 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:24.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.901+0000 7f12fb966700 1 -- 172.21.15.113:0/2592475515 >> 172.21.15.113:0/2592475515 conn(0x7f12f4070590 msgr2=0x7f12f4060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:24.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.901+0000 7f12fb966700 1 -- 172.21.15.113:0/2592475515 shutdown_connections 2024-04-12T05:13:24.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:24.901+0000 7f12fb966700 1 -- 172.21.15.113:0/2592475515 wait complete. 2024-04-12T05:13:25.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:24 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:23.978332+0000 mgr.smithi113.ctsazg (mgr.14656) 131 : cluster [DBG] pgmap v77: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:24 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:23.978332+0000 mgr.smithi113.ctsazg (mgr.14656) 131 : cluster [DBG] pgmap v77: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:25.592 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-12T05:13:27.265 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:25.978739+0000 mgr.smithi113.ctsazg (mgr.14656) 132 : cluster [DBG] pgmap v78: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:27.266 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:26 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:26.023481+0000 mon.smithi113 (mon.0) 999 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:13:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:25.978739+0000 mgr.smithi113.ctsazg (mgr.14656) 132 : cluster [DBG] pgmap v78: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:26 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:26.023481+0000 mon.smithi113 (mon.0) 999 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:13:27.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.518+0000 7fb290216700 1 -- 172.21.15.113:0/3592118599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880fd180 msgr2=0x7fb2880f6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:27.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.518+0000 7fb290216700 1 --2- 172.21.15.113:0/3592118599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880fd180 0x7fb2880f6960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fb2840047f0 tx=0x7fb284009cb0).stop 2024-04-12T05:13:27.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.519+0000 7fb290216700 1 -- 172.21.15.113:0/3592118599 shutdown_connections 2024-04-12T05:13:27.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.519+0000 7fb290216700 1 --2- 172.21.15.113:0/3592118599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880fd180 0x7fb2880f6960 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:27.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.519+0000 7fb290216700 1 --2- 172.21.15.113:0/3592118599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880fc820 0x7fb2880fcc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:27.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.519+0000 7fb290216700 1 -- 172.21.15.113:0/3592118599 >> 172.21.15.113:0/3592118599 conn(0x7fb2880f2400 msgr2=0x7fb2880f4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:27.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.519+0000 7fb290216700 1 -- 172.21.15.113:0/3592118599 shutdown_connections 2024-04-12T05:13:27.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.519+0000 7fb290216700 1 -- 172.21.15.113:0/3592118599 wait complete. 2024-04-12T05:13:27.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.519+0000 7fb290216700 1 Processor -- start 2024-04-12T05:13:27.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb290216700 1 -- start start 2024-04-12T05:13:27.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb290216700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880fc820 0x7fb28810cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:27.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb290216700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880fd180 0x7fb28810d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:27.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb290216700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28810db10 con 0x7fb2880fc820 2024-04-12T05:13:27.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb290216700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb288107180 con 0x7fb2880fd180 2024-04-12T05:13:27.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb28dfb2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880fc820 0x7fb28810cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:27.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb28dfb2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880fc820 0x7fb28810cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42506/0 (socket says 172.21.15.113:42506) 2024-04-12T05:13:27.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb28dfb2700 1 -- 172.21.15.113:0/1774969170 learned_addr learned my addr 172.21.15.113:0/1774969170 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb28d7b1700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880fd180 0x7fb28810d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb28dfb2700 1 -- 172.21.15.113:0/1774969170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880fd180 msgr2=0x7fb28810d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb28dfb2700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880fd180 0x7fb28810d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb28dfb2700 1 -- 172.21.15.113:0/1774969170 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb284005040 con 0x7fb2880fc820 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb28d7b1700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880fd180 0x7fb28810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb28dfb2700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880fc820 0x7fb28810cf90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fb27800b4b0 tx=0x7fb2780090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb27effd700 1 -- 172.21.15.113:0/1774969170 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb278002500 con 0x7fb2880fc820 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb27effd700 1 -- 172.21.15.113:0/1774969170 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb278002b10 con 0x7fb2880fc820 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb290216700 1 -- 172.21.15.113:0/1774969170 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb288107430 con 0x7fb2880fc820 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.520+0000 7fb27effd700 1 -- 172.21.15.113:0/1774969170 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb278012cf0 con 0x7fb2880fc820 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.521+0000 7fb290216700 1 -- 172.21.15.113:0/1774969170 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb288107970 con 0x7fb2880fc820 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.521+0000 7fb290216700 1 -- 172.21.15.113:0/1774969170 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb288060330 con 0x7fb2880fc820 2024-04-12T05:13:27.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.522+0000 7fb27effd700 1 -- 172.21.15.113:0/1774969170 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7fb278002660 con 0x7fb2880fc820 2024-04-12T05:13:27.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.522+0000 7fb27effd700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb27406c6a0 0x7fb27406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:27.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.523+0000 7fb27effd700 1 -- 172.21.15.113:0/1774969170 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fb27808b840 con 0x7fb2880fc820 2024-04-12T05:13:27.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.523+0000 7fb28d7b1700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb27406c6a0 0x7fb27406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:27.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.523+0000 7fb28d7b1700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb27406c6a0 0x7fb27406eb60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb2840043d0 tx=0x7fb28400b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:27.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.527+0000 7fb27effd700 1 -- 172.21.15.113:0/1774969170 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fb278059940 con 0x7fb2880fc820 2024-04-12T05:13:27.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.716+0000 7fb290216700 1 -- 172.21.15.113:0/1774969170 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fb288108610 con 0x7fb2880fc820 2024-04-12T05:13:27.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.716+0000 7fb27effd700 1 -- 172.21.15.113:0/1774969170 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v49)=0 v49) v1 ==== 163+0+0 (secure 0 0 0) 0x7fb2780594d0 con 0x7fb2880fc820 2024-04-12T05:13:27.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.718+0000 7fb290216700 1 -- 172.21.15.113:0/1774969170 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb27406c6a0 msgr2=0x7fb27406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:27.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.718+0000 7fb290216700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb27406c6a0 0x7fb27406eb60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb2840043d0 tx=0x7fb28400b040).stop 2024-04-12T05:13:27.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.718+0000 7fb290216700 1 -- 172.21.15.113:0/1774969170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880fc820 msgr2=0x7fb28810cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:27.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.718+0000 7fb290216700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880fc820 0x7fb28810cf90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fb27800b4b0 tx=0x7fb2780090f0).stop 2024-04-12T05:13:27.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.718+0000 7fb290216700 1 -- 172.21.15.113:0/1774969170 shutdown_connections 2024-04-12T05:13:27.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.718+0000 7fb290216700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fb27406c6a0 0x7fb27406eb60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:27.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.718+0000 7fb290216700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880fc820 0x7fb28810cf90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:27.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.718+0000 7fb290216700 1 --2- 172.21.15.113:0/1774969170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880fd180 0x7fb28810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:27.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.718+0000 7fb290216700 1 -- 172.21.15.113:0/1774969170 >> 172.21.15.113:0/1774969170 conn(0x7fb2880f2400 msgr2=0x7fb2880fabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:27.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.718+0000 7fb290216700 1 -- 172.21.15.113:0/1774969170 shutdown_connections 2024-04-12T05:13:27.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:27.718+0000 7fb290216700 1 -- 172.21.15.113:0/1774969170 wait complete. 2024-04-12T05:13:28.408 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-12T05:13:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:27.979451+0000 mgr.smithi113.ctsazg (mgr.14656) 133 : cluster [DBG] pgmap v79: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:28 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:27.979451+0000 mgr.smithi113.ctsazg (mgr.14656) 133 : cluster [DBG] pgmap v79: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:30.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.323+0000 7f7902ea4700 1 -- 172.21.15.113:0/4228788098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78fc0fd8f0 msgr2=0x7f78fc0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:30.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.323+0000 7f7902ea4700 1 --2- 172.21.15.113:0/4228788098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78fc0fd8f0 0x7f78fc0fdcd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f78f00047f0 tx=0x7f78f0009cb0).stop 2024-04-12T05:13:30.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.323+0000 7f7902ea4700 1 -- 172.21.15.113:0/4228788098 shutdown_connections 2024-04-12T05:13:30.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.323+0000 7f7902ea4700 1 --2- 172.21.15.113:0/4228788098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78fc0fe250 0x7f78fc0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:30.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.323+0000 7f7902ea4700 1 --2- 172.21.15.113:0/4228788098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78fc0fd8f0 0x7f78fc0fdcd0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:30.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.323+0000 7f7902ea4700 1 -- 172.21.15.113:0/4228788098 >> 172.21.15.113:0/4228788098 conn(0x7f78fc0f34b0 msgr2=0x7f78fc0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:30.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.323+0000 7f7902ea4700 1 -- 172.21.15.113:0/4228788098 shutdown_connections 2024-04-12T05:13:30.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.323+0000 7f7902ea4700 1 -- 172.21.15.113:0/4228788098 wait complete. 2024-04-12T05:13:30.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7902ea4700 1 Processor -- start 2024-04-12T05:13:30.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7902ea4700 1 -- start start 2024-04-12T05:13:30.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7902ea4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78fc0fd8f0 0x7f78fc10dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:30.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7902ea4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78fc0fe250 0x7f78fc10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:30.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7902ea4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc10ead0 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7902ea4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc10ec10 con 0x7f78fc0fe250 2024-04-12T05:13:30.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7900c40700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78fc0fd8f0 0x7f78fc10dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:30.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7900c40700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78fc0fd8f0 0x7f78fc10dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41886/0 (socket says 172.21.15.113:41886) 2024-04-12T05:13:30.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7900c40700 1 -- 172.21.15.113:0/2087683118 learned_addr learned my addr 172.21.15.113:0/2087683118 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:30.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7900c40700 1 -- 172.21.15.113:0/2087683118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78fc0fe250 msgr2=0x7f78fc10e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:30.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7900c40700 1 --2- 172.21.15.113:0/2087683118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78fc0fe250 0x7f78fc10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:30.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.324+0000 7f7900c40700 1 -- 172.21.15.113:0/2087683118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f0005040 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.325+0000 7f7900c40700 1 --2- 172.21.15.113:0/2087683118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78fc0fd8f0 0x7f78fc10dfe0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f78f000b120 tx=0x7f78f0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:30.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.325+0000 7f78f9ffb700 1 -- 172.21.15.113:0/2087683118 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78f000e030 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.325+0000 7f7902ea4700 1 -- 172.21.15.113:0/2087683118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc108240 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.325+0000 7f78f9ffb700 1 -- 172.21.15.113:0/2087683118 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78f0007e90 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.325+0000 7f78f9ffb700 1 -- 172.21.15.113:0/2087683118 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78f0018070 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.325+0000 7f7902ea4700 1 -- 172.21.15.113:0/2087683118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc1087a0 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.326+0000 7f7902ea4700 1 -- 172.21.15.113:0/2087683118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78fc05f590 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.326+0000 7f78f9ffb700 1 -- 172.21.15.113:0/2087683118 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f78f001d420 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.327+0000 7f78f9ffb700 1 --2- 172.21.15.113:0/2087683118 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f78ec06c6a0 0x7f78ec06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:30.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.327+0000 7f78f9ffb700 1 -- 172.21.15.113:0/2087683118 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f78f008cc00 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.327+0000 7f78fbfff700 1 --2- 172.21.15.113:0/2087683118 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f78ec06c6a0 0x7f78ec06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:30.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.327+0000 7f78fbfff700 1 --2- 172.21.15.113:0/2087683118 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f78ec06c6a0 0x7f78ec06eb60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f78fc109920 tx=0x7f78e8007bc0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:30.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.332+0000 7f78f9ffb700 1 -- 172.21.15.113:0/2087683118 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f78f0092020 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.519+0000 7f7902ea4700 1 -- 172.21.15.113:0/2087683118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f78fc047540 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.519+0000 7f78f9ffb700 1 -- 172.21.15.113:0/2087683118 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v49)=0 v49) v1 ==== 135+0+0 (secure 0 0 0) 0x7f78f005ad00 con 0x7f78fc0fd8f0 2024-04-12T05:13:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.521+0000 7f7902ea4700 1 -- 172.21.15.113:0/2087683118 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f78ec06c6a0 msgr2=0x7f78ec06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.521+0000 7f7902ea4700 1 --2- 172.21.15.113:0/2087683118 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f78ec06c6a0 0x7f78ec06eb60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f78fc109920 tx=0x7f78e8007bc0).stop 2024-04-12T05:13:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.521+0000 7f7902ea4700 1 -- 172.21.15.113:0/2087683118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78fc0fd8f0 msgr2=0x7f78fc10dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.521+0000 7f7902ea4700 1 --2- 172.21.15.113:0/2087683118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78fc0fd8f0 0x7f78fc10dfe0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f78f000b120 tx=0x7f78f0009cb0).stop 2024-04-12T05:13:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.522+0000 7f7902ea4700 1 -- 172.21.15.113:0/2087683118 shutdown_connections 2024-04-12T05:13:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.522+0000 7f7902ea4700 1 --2- 172.21.15.113:0/2087683118 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f78ec06c6a0 0x7f78ec06eb60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.522+0000 7f7902ea4700 1 --2- 172.21.15.113:0/2087683118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78fc0fd8f0 0x7f78fc10dfe0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:30.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.522+0000 7f7902ea4700 1 --2- 172.21.15.113:0/2087683118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78fc0fe250 0x7f78fc10e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:30.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.522+0000 7f7902ea4700 1 -- 172.21.15.113:0/2087683118 >> 172.21.15.113:0/2087683118 conn(0x7f78fc0f34b0 msgr2=0x7f78fc0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:30.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.522+0000 7f7902ea4700 1 -- 172.21.15.113:0/2087683118 shutdown_connections 2024-04-12T05:13:30.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:30.522+0000 7f7902ea4700 1 -- 172.21.15.113:0/2087683118 wait complete. 2024-04-12T05:13:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:29.979798+0000 mgr.smithi113.ctsazg (mgr.14656) 134 : cluster [DBG] pgmap v80: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:31.214 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-12T05:13:31.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:30 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:29.979798+0000 mgr.smithi113.ctsazg (mgr.14656) 134 : cluster [DBG] pgmap v80: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:33.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.022+0000 7f8f5f479700 1 -- 172.21.15.113:0/452076522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fc7f0 msgr2=0x7f8f580fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:33.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.022+0000 7f8f5f479700 1 --2- 172.21.15.113:0/452076522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fc7f0 0x7f8f580fcbd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f8f480047f0 tx=0x7f8f48009cb0).stop 2024-04-12T05:13:33.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.022+0000 7f8f5f479700 1 -- 172.21.15.113:0/452076522 shutdown_connections 2024-04-12T05:13:33.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.022+0000 7f8f5f479700 1 --2- 172.21.15.113:0/452076522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f580fd150 0x7f8f580f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:33.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.023+0000 7f8f5f479700 1 --2- 172.21.15.113:0/452076522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fc7f0 0x7f8f580fcbd0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:33.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.023+0000 7f8f5f479700 1 -- 172.21.15.113:0/452076522 >> 172.21.15.113:0/452076522 conn(0x7f8f58070590 msgr2=0x7f8f580f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:33.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.023+0000 7f8f5f479700 1 -- 172.21.15.113:0/452076522 shutdown_connections 2024-04-12T05:13:33.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.023+0000 7f8f5f479700 1 -- 172.21.15.113:0/452076522 wait complete. 2024-04-12T05:13:33.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.023+0000 7f8f5f479700 1 Processor -- start 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.023+0000 7f8f5f479700 1 -- start start 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.023+0000 7f8f5f479700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fc7f0 0x7f8f58109160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.023+0000 7f8f5f479700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f580fd150 0x7f8f58109650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.023+0000 7f8f5f479700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f58109bd0 con 0x7f8f580fc7f0 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.023+0000 7f8f5f479700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f58109d10 con 0x7f8f580fd150 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f5d215700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fc7f0 0x7f8f58109160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f5d215700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fc7f0 0x7f8f58109160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41900/0 (socket says 172.21.15.113:41900) 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f5d215700 1 -- 172.21.15.113:0/3331626285 learned_addr learned my addr 172.21.15.113:0/3331626285 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f5d215700 1 -- 172.21.15.113:0/3331626285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f580fd150 msgr2=0x7f8f58109650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f5d215700 1 --2- 172.21.15.113:0/3331626285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f580fd150 0x7f8f58109650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f5d215700 1 -- 172.21.15.113:0/3331626285 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f48005040 con 0x7f8f580fc7f0 2024-04-12T05:13:33.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f5d215700 1 --2- 172.21.15.113:0/3331626285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fc7f0 0x7f8f58109160 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f8f4800b120 tx=0x7f8f48009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:33.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f4e7fc700 1 -- 172.21.15.113:0/3331626285 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f4800e030 con 0x7f8f580fc7f0 2024-04-12T05:13:33.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f5f479700 1 -- 172.21.15.113:0/3331626285 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f58109f60 con 0x7f8f580fc7f0 2024-04-12T05:13:33.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f4e7fc700 1 -- 172.21.15.113:0/3331626285 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f48007e90 con 0x7f8f580fc7f0 2024-04-12T05:13:33.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f4e7fc700 1 -- 172.21.15.113:0/3331626285 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f48018070 con 0x7f8f580fc7f0 2024-04-12T05:13:33.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.024+0000 7f8f5f479700 1 -- 172.21.15.113:0/3331626285 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f581124a0 con 0x7f8f580fc7f0 2024-04-12T05:13:33.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.025+0000 7f8f5f479700 1 -- 172.21.15.113:0/3331626285 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5805f590 con 0x7f8f580fc7f0 2024-04-12T05:13:33.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.026+0000 7f8f4e7fc700 1 -- 172.21.15.113:0/3331626285 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f8f4801d420 con 0x7f8f580fc7f0 2024-04-12T05:13:33.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.026+0000 7f8f4e7fc700 1 --2- 172.21.15.113:0/3331626285 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f8f4406c6a0 0x7f8f4406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:33.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.026+0000 7f8f4e7fc700 1 -- 172.21.15.113:0/3331626285 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f8f4808cbe0 con 0x7f8f580fc7f0 2024-04-12T05:13:33.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.026+0000 7f8f5ca14700 1 --2- 172.21.15.113:0/3331626285 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f8f4406c6a0 0x7f8f4406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:33.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.026+0000 7f8f5ca14700 1 --2- 172.21.15.113:0/3331626285 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f8f4406c6a0 0x7f8f4406eb60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8f58060ee0 tx=0x7f8f54006c30).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:33.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.031+0000 7f8f4e7fc700 1 -- 172.21.15.113:0/3331626285 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f8f48057250 con 0x7f8f580fc7f0 2024-04-12T05:13:33.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.238+0000 7f8f5f479700 1 -- 172.21.15.113:0/3331626285 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f58059120 con 0x7f8f4406c6a0 2024-04-12T05:13:33.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.244+0000 7f8f4e7fc700 1 -- 172.21.15.113:0/3331626285 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f8f58059120 con 0x7f8f4406c6a0 2024-04-12T05:13:33.247 INFO:teuthology.orchestra.run.smithi113.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:13:33.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.246+0000 7f8f5f479700 1 -- 172.21.15.113:0/3331626285 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f8f4406c6a0 msgr2=0x7f8f4406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:33.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.246+0000 7f8f5f479700 1 --2- 172.21.15.113:0/3331626285 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f8f4406c6a0 0x7f8f4406eb60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8f58060ee0 tx=0x7f8f54006c30).stop 2024-04-12T05:13:33.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.246+0000 7f8f5f479700 1 -- 172.21.15.113:0/3331626285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fc7f0 msgr2=0x7f8f58109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:33.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.246+0000 7f8f5f479700 1 --2- 172.21.15.113:0/3331626285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fc7f0 0x7f8f58109160 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f8f4800b120 tx=0x7f8f48009cb0).stop 2024-04-12T05:13:33.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.246+0000 7f8f5f479700 1 -- 172.21.15.113:0/3331626285 shutdown_connections 2024-04-12T05:13:33.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.246+0000 7f8f5f479700 1 --2- 172.21.15.113:0/3331626285 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f8f4406c6a0 0x7f8f4406eb60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:33.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.246+0000 7f8f5f479700 1 --2- 172.21.15.113:0/3331626285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f580fc7f0 0x7f8f58109160 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:33.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.246+0000 7f8f5f479700 1 --2- 172.21.15.113:0/3331626285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f580fd150 0x7f8f58109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:33.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.246+0000 7f8f5f479700 1 -- 172.21.15.113:0/3331626285 >> 172.21.15.113:0/3331626285 conn(0x7f8f58070590 msgr2=0x7f8f580fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:33.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.246+0000 7f8f5f479700 1 -- 172.21.15.113:0/3331626285 shutdown_connections 2024-04-12T05:13:33.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:33.246+0000 7f8f5f479700 1 -- 172.21.15.113:0/3331626285 wait complete. 2024-04-12T05:13:33.268 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:31.980442+0000 mgr.smithi113.ctsazg (mgr.14656) 135 : cluster [DBG] pgmap v81: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:33.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:32 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:31.980442+0000 mgr.smithi113.ctsazg (mgr.14656) 135 : cluster [DBG] pgmap v81: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:33.977 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-12T05:13:33.978 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:13:33.978 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-12T05:13:34.613 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:33.241481+0000 mgr.smithi113.ctsazg (mgr.14656) 136 : audit [DBG] from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:34.613 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:13:33.242032+0000 mgr.smithi113.ctsazg (mgr.14656) 137 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:13:34.613 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:33.245881+0000 mon.smithi113 (mon.0) 1000 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:34.613 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:33.275692+0000 mon.smithi113 (mon.0) 1001 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:13:34.613 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:33.697181+0000 mon.smithi113 (mon.0) 1002 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:13:34.613 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:33.697863+0000 mon.smithi113 (mon.0) 1003 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:13:34.614 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:34 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:33.702566+0000 mon.smithi113 (mon.0) 1004 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:34.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:33.241481+0000 mgr.smithi113.ctsazg (mgr.14656) 136 : audit [DBG] from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:34.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:13:33.242032+0000 mgr.smithi113.ctsazg (mgr.14656) 137 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:13:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:33.245881+0000 mon.smithi113 (mon.0) 1000 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:33.275692+0000 mon.smithi113 (mon.0) 1001 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:13:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:33.697181+0000 mon.smithi113 (mon.0) 1002 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:13:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:33.697863+0000 mon.smithi113 (mon.0) 1003 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:13:34.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:34 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:33.702566+0000 mon.smithi113 (mon.0) 1004 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:35.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:13:33.838522+0000 mgr.smithi113.ctsazg (mgr.14656) 138 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:13:35.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:33.980740+0000 mgr.smithi113.ctsazg (mgr.14656) 139 : cluster [DBG] pgmap v82: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:35.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:13:33.838522+0000 mgr.smithi113.ctsazg (mgr.14656) 138 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:13:35.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:33.980740+0000 mgr.smithi113.ctsazg (mgr.14656) 139 : cluster [DBG] pgmap v82: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:37.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.988+0000 7f25d5c16700 1 -- 172.21.15.113:0/1978999338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d00fd150 msgr2=0x7f25d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.988+0000 7f25d5c16700 1 --2- 172.21.15.113:0/1978999338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d00fd150 0x7f25d00f6930 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f25c00047f0 tx=0x7f25c0009cb0).stop 2024-04-12T05:13:37.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.988+0000 7f25d5c16700 1 -- 172.21.15.113:0/1978999338 shutdown_connections 2024-04-12T05:13:37.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.988+0000 7f25d5c16700 1 --2- 172.21.15.113:0/1978999338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d00fd150 0x7f25d00f6930 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.988+0000 7f25d5c16700 1 --2- 172.21.15.113:0/1978999338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d00fc7f0 0x7f25d00fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:37.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.988+0000 7f25d5c16700 1 -- 172.21.15.113:0/1978999338 >> 172.21.15.113:0/1978999338 conn(0x7f25d0070590 msgr2=0x7f25d00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:37.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.988+0000 7f25d5c16700 1 -- 172.21.15.113:0/1978999338 shutdown_connections 2024-04-12T05:13:37.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.988+0000 7f25d5c16700 1 -- 172.21.15.113:0/1978999338 wait complete. 2024-04-12T05:13:37.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25d5c16700 1 Processor -- start 2024-04-12T05:13:37.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25d5c16700 1 -- start start 2024-04-12T05:13:37.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25d5c16700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d00fc7f0 0x7f25d0109150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25d5c16700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d00fd150 0x7f25d0109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25d5c16700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0109bc0 con 0x7f25d00fd150 2024-04-12T05:13:37.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25d5c16700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0109d00 con 0x7f25d00fc7f0 2024-04-12T05:13:37.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25ceffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d00fd150 0x7f25d0109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:37.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25ceffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d00fd150 0x7f25d0109640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41920/0 (socket says 172.21.15.113:41920) 2024-04-12T05:13:37.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25ceffd700 1 -- 172.21.15.113:0/3625017680 learned_addr learned my addr 172.21.15.113:0/3625017680 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:37.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25cf7fe700 1 --2- 172.21.15.113:0/3625017680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d00fc7f0 0x7f25d0109150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:37.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25ceffd700 1 -- 172.21.15.113:0/3625017680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d00fc7f0 msgr2=0x7f25d0109150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25ceffd700 1 --2- 172.21.15.113:0/3625017680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d00fc7f0 0x7f25d0109150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.989+0000 7f25ceffd700 1 -- 172.21.15.113:0/3625017680 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c0005040 con 0x7f25d00fd150 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.990+0000 7f25ceffd700 1 --2- 172.21.15.113:0/3625017680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d00fd150 0x7f25d0109640 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f25c00099f0 tx=0x7f25c0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.990+0000 7f25ccff9700 1 -- 172.21.15.113:0/3625017680 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25c000e030 con 0x7f25d00fd150 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.990+0000 7f25ccff9700 1 -- 172.21.15.113:0/3625017680 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25c0007e90 con 0x7f25d00fd150 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.990+0000 7f25d5c16700 1 -- 172.21.15.113:0/3625017680 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d0109f50 con 0x7f25d00fd150 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.990+0000 7f25ccff9700 1 -- 172.21.15.113:0/3625017680 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25c0018070 con 0x7f25d00fd150 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.990+0000 7f25d5c16700 1 -- 172.21.15.113:0/3625017680 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d0112490 con 0x7f25d00fd150 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.991+0000 7f25d5c16700 1 -- 172.21.15.113:0/3625017680 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d005f590 con 0x7f25d00fd150 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.991+0000 7f25ccff9700 1 -- 172.21.15.113:0/3625017680 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f25c001d420 con 0x7f25d00fd150 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.992+0000 7f25ccff9700 1 --2- 172.21.15.113:0/3625017680 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f25bc06c6a0 0x7f25bc06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.992+0000 7f25cf7fe700 1 --2- 172.21.15.113:0/3625017680 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f25bc06c6a0 0x7f25bc06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.992+0000 7f25ccff9700 1 -- 172.21.15.113:0/3625017680 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f25c008cb40 con 0x7f25d00fd150 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.992+0000 7f25cf7fe700 1 --2- 172.21.15.113:0/3625017680 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f25bc06c6a0 0x7f25bc06eb60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f25b800b910 tx=0x7f25b8008040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:37.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:36.996+0000 7f25ccff9700 1 -- 172.21.15.113:0/3625017680 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f25c00571b0 con 0x7f25d00fd150 2024-04-12T05:13:37.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.202+0000 7f25d5c16700 1 -- 172.21.15.113:0/3625017680 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f25d0059120 con 0x7f25bc06c6a0 2024-04-12T05:13:37.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.203+0000 7f25ccff9700 1 -- 172.21.15.113:0/3625017680 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f25d0059120 con 0x7f25bc06c6a0 2024-04-12T05:13:37.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.205+0000 7f25d5c16700 1 -- 172.21.15.113:0/3625017680 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f25bc06c6a0 msgr2=0x7f25bc06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.205+0000 7f25d5c16700 1 --2- 172.21.15.113:0/3625017680 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f25bc06c6a0 0x7f25bc06eb60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f25b800b910 tx=0x7f25b8008040).stop 2024-04-12T05:13:37.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.205+0000 7f25d5c16700 1 -- 172.21.15.113:0/3625017680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d00fd150 msgr2=0x7f25d0109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.205+0000 7f25d5c16700 1 --2- 172.21.15.113:0/3625017680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d00fd150 0x7f25d0109640 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f25c00099f0 tx=0x7f25c0009cb0).stop 2024-04-12T05:13:37.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.206+0000 7f25d5c16700 1 -- 172.21.15.113:0/3625017680 shutdown_connections 2024-04-12T05:13:37.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.206+0000 7f25d5c16700 1 --2- 172.21.15.113:0/3625017680 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f25bc06c6a0 0x7f25bc06eb60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.206+0000 7f25d5c16700 1 --2- 172.21.15.113:0/3625017680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25d00fc7f0 0x7f25d0109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.206+0000 7f25d5c16700 1 --2- 172.21.15.113:0/3625017680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25d00fd150 0x7f25d0109640 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.206+0000 7f25d5c16700 1 -- 172.21.15.113:0/3625017680 >> 172.21.15.113:0/3625017680 conn(0x7f25d0070590 msgr2=0x7f25d00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:37.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.206+0000 7f25d5c16700 1 -- 172.21.15.113:0/3625017680 shutdown_connections 2024-04-12T05:13:37.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.206+0000 7f25d5c16700 1 -- 172.21.15.113:0/3625017680 wait complete. 2024-04-12T05:13:37.217 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:13:37.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.311+0000 7f7b6c459700 1 -- 172.21.15.113:0/3427145414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b640fd970 msgr2=0x7f7b640fdd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.311+0000 7f7b6c459700 1 --2- 172.21.15.113:0/3427145414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b640fd970 0x7f7b640fdd50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f7b5c009b60 tx=0x7f7b5c004d10).stop 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.311+0000 7f7b6c459700 1 -- 172.21.15.113:0/3427145414 shutdown_connections 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.311+0000 7f7b6c459700 1 --2- 172.21.15.113:0/3427145414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b640fe2d0 0x7f7b640f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.311+0000 7f7b6c459700 1 --2- 172.21.15.113:0/3427145414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b640fd970 0x7f7b640fdd50 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.311+0000 7f7b6c459700 1 -- 172.21.15.113:0/3427145414 >> 172.21.15.113:0/3427145414 conn(0x7f7b640f3550 msgr2=0x7f7b640f5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.311+0000 7f7b6c459700 1 -- 172.21.15.113:0/3427145414 shutdown_connections 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.311+0000 7f7b6c459700 1 -- 172.21.15.113:0/3427145414 wait complete. 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.311+0000 7f7b6c459700 1 Processor -- start 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.311+0000 7f7b6c459700 1 -- start start 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6c459700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b640fd970 0x7f7b6406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6c459700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b640fe2d0 0x7f7b6406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6c459700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6406bab0 con 0x7f7b640fd970 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6c459700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6406bbf0 con 0x7f7b640fe2d0 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6a1f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b640fd970 0x7f7b6406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:37.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6a1f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b640fd970 0x7f7b6406b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41936/0 (socket says 172.21.15.113:41936) 2024-04-12T05:13:37.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6a1f5700 1 -- 172.21.15.113:0/378442526 learned_addr learned my addr 172.21.15.113:0/378442526 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:37.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6a1f5700 1 -- 172.21.15.113:0/378442526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b640fe2d0 msgr2=0x7f7b6406b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6a1f5700 1 --2- 172.21.15.113:0/378442526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b640fe2d0 0x7f7b6406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:37.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6a1f5700 1 -- 172.21.15.113:0/378442526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c0097c0 con 0x7f7b640fd970 2024-04-12T05:13:37.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6a1f5700 1 --2- 172.21.15.113:0/378442526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b640fd970 0x7f7b6406b0d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f7b5c009040 tx=0x7f7b5c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:37.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b637fe700 1 -- 172.21.15.113:0/378442526 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b5c00e070 con 0x7f7b640fd970 2024-04-12T05:13:37.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b637fe700 1 -- 172.21.15.113:0/378442526 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b5c00bec0 con 0x7f7b640fd970 2024-04-12T05:13:37.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b637fe700 1 -- 172.21.15.113:0/378442526 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b5c019070 con 0x7f7b640fd970 2024-04-12T05:13:37.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6c459700 1 -- 172.21.15.113:0/378442526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6406be40 con 0x7f7b640fd970 2024-04-12T05:13:37.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.312+0000 7f7b6c459700 1 -- 172.21.15.113:0/378442526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b641146c0 con 0x7f7b640fd970 2024-04-12T05:13:37.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.313+0000 7f7b6c459700 1 -- 172.21.15.113:0/378442526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b64060370 con 0x7f7b640fd970 2024-04-12T05:13:37.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.314+0000 7f7b637fe700 1 -- 172.21.15.113:0/378442526 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f7b5c01d6b0 con 0x7f7b640fd970 2024-04-12T05:13:37.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.314+0000 7f7b637fe700 1 --2- 172.21.15.113:0/378442526 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b4c06c770 0x7f7b4c06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.314+0000 7f7b637fe700 1 -- 172.21.15.113:0/378442526 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f7b5c08cc00 con 0x7f7b640fd970 2024-04-12T05:13:37.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.314+0000 7f7b699f4700 1 --2- 172.21.15.113:0/378442526 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b4c06c770 0x7f7b4c06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:37.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.315+0000 7f7b699f4700 1 --2- 172.21.15.113:0/378442526 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b4c06c770 0x7f7b4c06ec30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7b6406c870 tx=0x7f7b58006c30).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:37.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.319+0000 7f7b637fe700 1 -- 172.21.15.113:0/378442526 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f7b5c0571b0 con 0x7f7b640fd970 2024-04-12T05:13:37.537 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.535+0000 7f7b6c459700 1 -- 172.21.15.113:0/378442526 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b6406c6c0 con 0x7f7b4c06c770 2024-04-12T05:13:37.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.536+0000 7f7b637fe700 1 -- 172.21.15.113:0/378442526 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f7b6406c6c0 con 0x7f7b4c06c770 2024-04-12T05:13:37.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.538+0000 7f7b6c459700 1 -- 172.21.15.113:0/378442526 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b4c06c770 msgr2=0x7f7b4c06ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.538+0000 7f7b6c459700 1 --2- 172.21.15.113:0/378442526 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b4c06c770 0x7f7b4c06ec30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7b6406c870 tx=0x7f7b58006c30).stop 2024-04-12T05:13:37.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.538+0000 7f7b6c459700 1 -- 172.21.15.113:0/378442526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b640fd970 msgr2=0x7f7b6406b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.538+0000 7f7b6c459700 1 --2- 172.21.15.113:0/378442526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b640fd970 0x7f7b6406b0d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f7b5c009040 tx=0x7f7b5c004d10).stop 2024-04-12T05:13:37.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.538+0000 7f7b6c459700 1 -- 172.21.15.113:0/378442526 shutdown_connections 2024-04-12T05:13:37.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.538+0000 7f7b6c459700 1 --2- 172.21.15.113:0/378442526 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f7b4c06c770 0x7f7b4c06ec30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.538+0000 7f7b6c459700 1 --2- 172.21.15.113:0/378442526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b640fd970 0x7f7b6406b0d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.538+0000 7f7b6c459700 1 --2- 172.21.15.113:0/378442526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b640fe2d0 0x7f7b6406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:37.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.538+0000 7f7b6c459700 1 -- 172.21.15.113:0/378442526 >> 172.21.15.113:0/378442526 conn(0x7f7b640f3550 msgr2=0x7f7b640fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:37.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.538+0000 7f7b6c459700 1 -- 172.21.15.113:0/378442526 shutdown_connections 2024-04-12T05:13:37.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.538+0000 7f7b6c459700 1 -- 172.21.15.113:0/378442526 wait complete. 2024-04-12T05:13:37.628 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:35.981132+0000 mgr.smithi113.ctsazg (mgr.14656) 140 : cluster [DBG] pgmap v83: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:37.628 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:36.460899+0000 mon.smithi113 (mon.0) 1005 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:37.628 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:13:36.461221+0000 mgr.smithi113.ctsazg (mgr.14656) 141 : cephadm [INF] Upgrade: Target is version 18.2.2-1187-gd540ebac (reef) 2024-04-12T05:13:37.628 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:13:36.461282+0000 mgr.smithi113.ctsazg (mgr.14656) 142 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3'] 2024-04-12T05:13:37.628 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:36.462382+0000 mon.smithi113 (mon.0) 1006 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:13:37.628 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:36.467371+0000 mon.smithi113 (mon.0) 1007 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:37.628 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:36.468748+0000 mon.smithi113 (mon.0) 1008 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:13:37.629 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:13:36.469300+0000 mgr.smithi113.ctsazg (mgr.14656) 143 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-04-12T05:13:37.629 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:36.473240+0000 mon.smithi113 (mon.0) 1009 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:37.629 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:37 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:36.474252+0000 mon.smithi113 (mon.0) 1010 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T05:13:37.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.641+0000 7f112d48f700 1 -- 172.21.15.113:0/255103654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11280fc7f0 msgr2=0x7f11280fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.641+0000 7f112d48f700 1 --2- 172.21.15.113:0/255103654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11280fc7f0 0x7f11280fcbd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f11180047f0 tx=0x7f1118009cb0).stop 2024-04-12T05:13:37.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.641+0000 7f112d48f700 1 -- 172.21.15.113:0/255103654 shutdown_connections 2024-04-12T05:13:37.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.641+0000 7f112d48f700 1 --2- 172.21.15.113:0/255103654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11280fd150 0x7f11280f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.641+0000 7f112d48f700 1 --2- 172.21.15.113:0/255103654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11280fc7f0 0x7f11280fcbd0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.641+0000 7f112d48f700 1 -- 172.21.15.113:0/255103654 >> 172.21.15.113:0/255103654 conn(0x7f1128070590 msgr2=0x7f11280f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:37.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.641+0000 7f112d48f700 1 -- 172.21.15.113:0/255103654 shutdown_connections 2024-04-12T05:13:37.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.641+0000 7f112d48f700 1 -- 172.21.15.113:0/255103654 wait complete. 2024-04-12T05:13:37.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.641+0000 7f112d48f700 1 Processor -- start 2024-04-12T05:13:37.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f112d48f700 1 -- start start 2024-04-12T05:13:37.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f112d48f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11280fc7f0 0x7f11281090f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f112d48f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11280fd150 0x7f11281095e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f112d48f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1128109ad0 con 0x7f11280fc7f0 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f112d48f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1128109c10 con 0x7f11280fd150 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f1126ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11280fc7f0 0x7f11281090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f1126ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11280fc7f0 0x7f11281090f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41948/0 (socket says 172.21.15.113:41948) 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f1126ffd700 1 -- 172.21.15.113:0/993345700 learned_addr learned my addr 172.21.15.113:0/993345700 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f1126ffd700 1 -- 172.21.15.113:0/993345700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11280fd150 msgr2=0x7f11281095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f11267fc700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11280fd150 0x7f11281095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f1126ffd700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11280fd150 0x7f11281095e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f1126ffd700 1 -- 172.21.15.113:0/993345700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1118005040 con 0x7f11280fc7f0 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.642+0000 7f11267fc700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11280fd150 0x7f11281095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.643+0000 7f1126ffd700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11280fc7f0 0x7f11281090f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f111800b120 tx=0x7f1118009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.643+0000 7f110ffff700 1 -- 172.21.15.113:0/993345700 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f111800e030 con 0x7f11280fc7f0 2024-04-12T05:13:37.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.643+0000 7f112d48f700 1 -- 172.21.15.113:0/993345700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1128109e60 con 0x7f11280fc7f0 2024-04-12T05:13:37.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.643+0000 7f110ffff700 1 -- 172.21.15.113:0/993345700 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1118007e90 con 0x7f11280fc7f0 2024-04-12T05:13:37.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.643+0000 7f110ffff700 1 -- 172.21.15.113:0/993345700 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1118018070 con 0x7f11280fc7f0 2024-04-12T05:13:37.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.643+0000 7f112d48f700 1 -- 172.21.15.113:0/993345700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1128112450 con 0x7f11280fc7f0 2024-04-12T05:13:37.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.644+0000 7f112d48f700 1 -- 172.21.15.113:0/993345700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112805f590 con 0x7f11280fc7f0 2024-04-12T05:13:37.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.644+0000 7f110ffff700 1 -- 172.21.15.113:0/993345700 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f111801d420 con 0x7f11280fc7f0 2024-04-12T05:13:37.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.645+0000 7f110ffff700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f111006c6a0 0x7f111006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.645+0000 7f110ffff700 1 -- 172.21.15.113:0/993345700 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f111808cc10 con 0x7f11280fc7f0 2024-04-12T05:13:37.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.645+0000 7f11267fc700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f111006c6a0 0x7f111006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:37.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.645+0000 7f11267fc700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f111006c6a0 0x7f111006eb60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f1128060ee0 tx=0x7f111c009040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:37.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.650+0000 7f110ffff700 1 -- 172.21.15.113:0/993345700 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f1118057280 con 0x7f11280fc7f0 2024-04-12T05:13:37.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.855+0000 7f112d48f700 1 -- 172.21.15.113:0/993345700 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1128059120 con 0x7f111006c6a0 2024-04-12T05:13:37.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.865+0000 7f110ffff700 1 -- 172.21.15.113:0/993345700 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f1128059120 con 0x7f111006c6a0 2024-04-12T05:13:37.868 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:13:37.868 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (89s) 68s ago 12m 15.2M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:13:37.868 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (12m) 68s ago 12m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:13:37.868 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (12m) 84s ago 12m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:13:37.868 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (73s) 68s ago 12m 38.2M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:13:37.868 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (7m) 68s ago 7m 120M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:13:37.868 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (7m) 68s ago 7m 17.0M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:13:37.868 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (7m) 84s ago 7m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:13:37.868 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (7m) 84s ago 7m 14.4M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:13:37.868 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (2m) 68s ago 13m 498M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:13:37.868 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (2m) 84s ago 12m 446M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:13:37.869 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (14m) 68s ago 14m 59.8M 2048M 16.2.15-1-g8963e9de 727f386679cb e27d7c02f242 2024-04-12T05:13:37.869 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (11m) 84s ago 11m 49.8M 2048M 16.2.15-1-g8963e9de 727f386679cb adce07508243 2024-04-12T05:13:37.869 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (2m) 68s ago 12m 8744k - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:13:37.869 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (2m) 84s ago 11m 15.8M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:13:37.869 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (11m) 68s ago 11m 226M 1910M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:13:37.869 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (10m) 68s ago 10m 240M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:13:37.869 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (10m) 68s ago 10m 202M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:13:37.869 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (9m) 84s ago 9m 263M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:13:37.869 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (9m) 84s ago 9m 233M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:13:37.869 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (8m) 84s ago 8m 226M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:13:37.869 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (104s) 84s ago 11m 42.0M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:13:37.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.868+0000 7f112d48f700 1 -- 172.21.15.113:0/993345700 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f111006c6a0 msgr2=0x7f111006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.868+0000 7f112d48f700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f111006c6a0 0x7f111006eb60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f1128060ee0 tx=0x7f111c009040).stop 2024-04-12T05:13:37.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.868+0000 7f112d48f700 1 -- 172.21.15.113:0/993345700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11280fc7f0 msgr2=0x7f11281090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.868+0000 7f112d48f700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11280fc7f0 0x7f11281090f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f111800b120 tx=0x7f1118009cb0).stop 2024-04-12T05:13:37.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.868+0000 7f112d48f700 1 -- 172.21.15.113:0/993345700 shutdown_connections 2024-04-12T05:13:37.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.868+0000 7f112d48f700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f111006c6a0 0x7f111006eb60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.868+0000 7f112d48f700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11280fc7f0 0x7f11281090f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.868+0000 7f112d48f700 1 --2- 172.21.15.113:0/993345700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11280fd150 0x7f11281095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.868+0000 7f112d48f700 1 -- 172.21.15.113:0/993345700 >> 172.21.15.113:0/993345700 conn(0x7f1128070590 msgr2=0x7f11280fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:37.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.868+0000 7f112d48f700 1 -- 172.21.15.113:0/993345700 shutdown_connections 2024-04-12T05:13:37.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.868+0000 7f112d48f700 1 -- 172.21.15.113:0/993345700 wait complete. 2024-04-12T05:13:37.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:35.981132+0000 mgr.smithi113.ctsazg (mgr.14656) 140 : cluster [DBG] pgmap v83: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:37.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:36.460899+0000 mon.smithi113 (mon.0) 1005 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:37.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:13:36.461221+0000 mgr.smithi113.ctsazg (mgr.14656) 141 : cephadm [INF] Upgrade: Target is version 18.2.2-1187-gd540ebac (reef) 2024-04-12T05:13:37.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:13:36.461282+0000 mgr.smithi113.ctsazg (mgr.14656) 142 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3'] 2024-04-12T05:13:37.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:36.462382+0000 mon.smithi113 (mon.0) 1006 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:13:37.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:36.467371+0000 mon.smithi113 (mon.0) 1007 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:37.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:36.468748+0000 mon.smithi113 (mon.0) 1008 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:13:37.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:13:36.469300+0000 mgr.smithi113.ctsazg (mgr.14656) 143 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-04-12T05:13:37.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:36.473240+0000 mon.smithi113 (mon.0) 1009 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:37.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:37 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:36.474252+0000 mon.smithi113 (mon.0) 1010 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T05:13:37.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.965+0000 7fc39853a700 1 -- 172.21.15.113:0/2564609505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3900fe270 msgr2=0x7fc3900f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.965+0000 7fc39853a700 1 --2- 172.21.15.113:0/2564609505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3900fe270 0x7fc3900f7a50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fc38c0047f0 tx=0x7fc38c009cb0).stop 2024-04-12T05:13:37.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.965+0000 7fc39853a700 1 -- 172.21.15.113:0/2564609505 shutdown_connections 2024-04-12T05:13:37.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.965+0000 7fc39853a700 1 --2- 172.21.15.113:0/2564609505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3900fe270 0x7fc3900f7a50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.965+0000 7fc39853a700 1 --2- 172.21.15.113:0/2564609505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3900fd910 0x7fc3900fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.965+0000 7fc39853a700 1 -- 172.21.15.113:0/2564609505 >> 172.21.15.113:0/2564609505 conn(0x7fc3900f3530 msgr2=0x7fc3900f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:37.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.965+0000 7fc39853a700 1 -- 172.21.15.113:0/2564609505 shutdown_connections 2024-04-12T05:13:37.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.965+0000 7fc39853a700 1 -- 172.21.15.113:0/2564609505 wait complete. 2024-04-12T05:13:37.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.966+0000 7fc39853a700 1 Processor -- start 2024-04-12T05:13:37.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.966+0000 7fc39853a700 1 -- start start 2024-04-12T05:13:37.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc39853a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3900fd910 0x7fc39010dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc39853a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3900fe270 0x7fc39010e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc39853a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39010eab0 con 0x7fc3900fd910 2024-04-12T05:13:37.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc39853a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39010ebf0 con 0x7fc3900fe270 2024-04-12T05:13:37.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc3962d6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3900fd910 0x7fc39010dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:37.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc3962d6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3900fd910 0x7fc39010dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41964/0 (socket says 172.21.15.113:41964) 2024-04-12T05:13:37.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc3962d6700 1 -- 172.21.15.113:0/1758271651 learned_addr learned my addr 172.21.15.113:0/1758271651 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:37.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc395ad5700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3900fe270 0x7fc39010e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:37.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc3962d6700 1 -- 172.21.15.113:0/1758271651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3900fe270 msgr2=0x7fc39010e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:37.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc3962d6700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3900fe270 0x7fc39010e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:37.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc3962d6700 1 -- 172.21.15.113:0/1758271651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c005040 con 0x7fc3900fd910 2024-04-12T05:13:37.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc395ad5700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3900fe270 0x7fc39010e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:13:37.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc3962d6700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3900fd910 0x7fc39010dfc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fc3840041d0 tx=0x7fc384009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:37.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc3837fe700 1 -- 172.21.15.113:0/1758271651 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc384007b40 con 0x7fc3900fd910 2024-04-12T05:13:37.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc39853a700 1 -- 172.21.15.113:0/1758271651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc390108280 con 0x7fc3900fd910 2024-04-12T05:13:37.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc3837fe700 1 -- 172.21.15.113:0/1758271651 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc38400f040 con 0x7fc3900fd910 2024-04-12T05:13:37.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc3837fe700 1 -- 172.21.15.113:0/1758271651 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc38400d800 con 0x7fc3900fd910 2024-04-12T05:13:37.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.967+0000 7fc39853a700 1 -- 172.21.15.113:0/1758271651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc390108840 con 0x7fc3900fd910 2024-04-12T05:13:37.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.968+0000 7fc39853a700 1 -- 172.21.15.113:0/1758271651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39005f570 con 0x7fc3900fd910 2024-04-12T05:13:37.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.969+0000 7fc3837fe700 1 -- 172.21.15.113:0/1758271651 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7fc384007ca0 con 0x7fc3900fd910 2024-04-12T05:13:37.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.969+0000 7fc3837fe700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc37c06c6a0 0x7fc37c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:37.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.970+0000 7fc3837fe700 1 -- 172.21.15.113:0/1758271651 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fc38408b270 con 0x7fc3900fd910 2024-04-12T05:13:37.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.970+0000 7fc395ad5700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc37c06c6a0 0x7fc37c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:37.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.970+0000 7fc395ad5700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc37c06c6a0 0x7fc37c06eb60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc38c0043d0 tx=0x7fc38c00b040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:37.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:37.974+0000 7fc3837fe700 1 -- 172.21.15.113:0/1758271651 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7fc384059370 con 0x7fc3900fd910 2024-04-12T05:13:38.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.245+0000 7fc39853a700 1 -- 172.21.15.113:0/1758271651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc3900ffca0 con 0x7fc3900fd910 2024-04-12T05:13:38.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.245+0000 7fc3837fe700 1 -- 172.21.15.113:0/1758271651 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+743 (secure 0 0 0) 0x7fc384058f00 con 0x7fc3900fd910 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:13:38.248 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:13:38.249 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:13:38.249 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 12, 2024-04-12T05:13:38.249 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:13:38.249 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:13:38.249 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:13:38.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.247+0000 7fc39853a700 1 -- 172.21.15.113:0/1758271651 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc37c06c6a0 msgr2=0x7fc37c06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:38.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.247+0000 7fc39853a700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc37c06c6a0 0x7fc37c06eb60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc38c0043d0 tx=0x7fc38c00b040).stop 2024-04-12T05:13:38.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.247+0000 7fc39853a700 1 -- 172.21.15.113:0/1758271651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3900fd910 msgr2=0x7fc39010dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:38.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.247+0000 7fc39853a700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3900fd910 0x7fc39010dfc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fc3840041d0 tx=0x7fc384009f60).stop 2024-04-12T05:13:38.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.247+0000 7fc39853a700 1 -- 172.21.15.113:0/1758271651 shutdown_connections 2024-04-12T05:13:38.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.247+0000 7fc39853a700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7fc37c06c6a0 0x7fc37c06eb60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.247+0000 7fc39853a700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3900fd910 0x7fc39010dfc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.247+0000 7fc39853a700 1 --2- 172.21.15.113:0/1758271651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3900fe270 0x7fc39010e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.247+0000 7fc39853a700 1 -- 172.21.15.113:0/1758271651 >> 172.21.15.113:0/1758271651 conn(0x7fc3900f3530 msgr2=0x7fc3900fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:38.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.248+0000 7fc39853a700 1 -- 172.21.15.113:0/1758271651 shutdown_connections 2024-04-12T05:13:38.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.248+0000 7fc39853a700 1 -- 172.21.15.113:0/1758271651 wait complete. 2024-04-12T05:13:38.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.350+0000 7f16f107f700 1 -- 172.21.15.113:0/3379498367 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16ec0fd8f0 msgr2=0x7f16ec0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:38.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.350+0000 7f16f107f700 1 --2- 172.21.15.113:0/3379498367 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16ec0fd8f0 0x7f16ec0fdcd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f16dc0047f0 tx=0x7f16dc009cb0).stop 2024-04-12T05:13:38.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.350+0000 7f16f107f700 1 -- 172.21.15.113:0/3379498367 shutdown_connections 2024-04-12T05:13:38.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.350+0000 7f16f107f700 1 --2- 172.21.15.113:0/3379498367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ec0fe250 0x7f16ec0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.350+0000 7f16f107f700 1 --2- 172.21.15.113:0/3379498367 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16ec0fd8f0 0x7f16ec0fdcd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.350+0000 7f16f107f700 1 -- 172.21.15.113:0/3379498367 >> 172.21.15.113:0/3379498367 conn(0x7f16ec0f34b0 msgr2=0x7f16ec0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:38.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.350+0000 7f16f107f700 1 -- 172.21.15.113:0/3379498367 shutdown_connections 2024-04-12T05:13:38.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.350+0000 7f16f107f700 1 -- 172.21.15.113:0/3379498367 wait complete. 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.351+0000 7f16f107f700 1 Processor -- start 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16f107f700 1 -- start start 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16f107f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ec0fd8f0 0x7f16ec10dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16f107f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16ec0fe250 0x7f16ec10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16f107f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec10ead0 con 0x7f16ec0fe250 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16f107f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec10ec10 con 0x7f16ec0fd8f0 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16ea59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16ec0fe250 0x7f16ec10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16ea59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16ec0fe250 0x7f16ec10e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41984/0 (socket says 172.21.15.113:41984) 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16ea59c700 1 -- 172.21.15.113:0/258865874 learned_addr learned my addr 172.21.15.113:0/258865874 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16ead9d700 1 --2- 172.21.15.113:0/258865874 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ec0fd8f0 0x7f16ec10dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16ea59c700 1 -- 172.21.15.113:0/258865874 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ec0fd8f0 msgr2=0x7f16ec10dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:38.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16ea59c700 1 --2- 172.21.15.113:0/258865874 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ec0fd8f0 0x7f16ec10dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.352+0000 7f16ea59c700 1 -- 172.21.15.113:0/258865874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc005040 con 0x7f16ec0fe250 2024-04-12T05:13:38.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.353+0000 7f16ea59c700 1 --2- 172.21.15.113:0/258865874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16ec0fe250 0x7f16ec10e4d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f16e000c790 tx=0x7f16e0007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:38.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.353+0000 7f16d3fff700 1 -- 172.21.15.113:0/258865874 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16e000e680 con 0x7f16ec0fe250 2024-04-12T05:13:38.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.353+0000 7f16f107f700 1 -- 172.21.15.113:0/258865874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec1082a0 con 0x7f16ec0fe250 2024-04-12T05:13:38.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.353+0000 7f16d3fff700 1 -- 172.21.15.113:0/258865874 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16e000ec90 con 0x7f16ec0fe250 2024-04-12T05:13:38.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.353+0000 7f16d3fff700 1 -- 172.21.15.113:0/258865874 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16e0012c70 con 0x7f16ec0fe250 2024-04-12T05:13:38.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.353+0000 7f16f107f700 1 -- 172.21.15.113:0/258865874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec108860 con 0x7f16ec0fe250 2024-04-12T05:13:38.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.354+0000 7f16f107f700 1 -- 172.21.15.113:0/258865874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec05f590 con 0x7f16ec0fe250 2024-04-12T05:13:38.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.355+0000 7f16d3fff700 1 -- 172.21.15.113:0/258865874 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f16e0012490 con 0x7f16ec0fe250 2024-04-12T05:13:38.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.355+0000 7f16d3fff700 1 --2- 172.21.15.113:0/258865874 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f16d406c6b0 0x7f16d406eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:38.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.355+0000 7f16d3fff700 1 -- 172.21.15.113:0/258865874 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f16e008af90 con 0x7f16ec0fe250 2024-04-12T05:13:38.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.355+0000 7f16ead9d700 1 --2- 172.21.15.113:0/258865874 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f16d406c6b0 0x7f16d406eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:38.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.355+0000 7f16ead9d700 1 --2- 172.21.15.113:0/258865874 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f16d406c6b0 0x7f16d406eb70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f16dc004f10 tx=0x7f16dc009cb0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:38.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.360+0000 7f16d3fff700 1 -- 172.21.15.113:0/258865874 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f16e0059090 con 0x7f16ec0fe250 2024-04-12T05:13:38.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.612+0000 7f16f107f700 1 -- 172.21.15.113:0/258865874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f16ec047540 con 0x7f16ec0fe250 2024-04-12T05:13:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.612+0000 7f16d3fff700 1 -- 172.21.15.113:0/258865874 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1779 (secure 0 0 0) 0x7f16e0058c20 con 0x7f16ec0fe250 2024-04-12T05:13:38.615 INFO:teuthology.orchestra.run.smithi113.stdout:e18 2024-04-12T05:13:38.615 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:13:38.615 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:13:38.615 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:13:38.616 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:13:38.616 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:13:38.616 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:13:38.616 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 18 2024-04-12T05:13:38.616 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 2024-04-12T05:13:38.616 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:13:38.616 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:13:38.616 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:13:38.616 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:13:38.616 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:13:38.616 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:13:38.617 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:13:38.617 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:13:38.617 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:13:38.617 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:13:38.617 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:13:38.617 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:13:38.617 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:13:38.617 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=14426,1=24261} 2024-04-12T05:13:38.617 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:13:38.618 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:13:38.618 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:13:38.618 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:13:38.618 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:13:38.618 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:13:38.618 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:13:38.618 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:13:38.618 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:13:38.618 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:13:38.619 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:13:38.619 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:13:38.619 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:13:38.619 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:13:38.619 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:13:38.619 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:13:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.614+0000 7f16f107f700 1 -- 172.21.15.113:0/258865874 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f16d406c6b0 msgr2=0x7f16d406eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.614+0000 7f16f107f700 1 --2- 172.21.15.113:0/258865874 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f16d406c6b0 0x7f16d406eb70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f16dc004f10 tx=0x7f16dc009cb0).stop 2024-04-12T05:13:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.614+0000 7f16f107f700 1 -- 172.21.15.113:0/258865874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16ec0fe250 msgr2=0x7f16ec10e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.614+0000 7f16f107f700 1 --2- 172.21.15.113:0/258865874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16ec0fe250 0x7f16ec10e4d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f16e000c790 tx=0x7f16e0007f70).stop 2024-04-12T05:13:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.614+0000 7f16f107f700 1 -- 172.21.15.113:0/258865874 shutdown_connections 2024-04-12T05:13:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.614+0000 7f16f107f700 1 --2- 172.21.15.113:0/258865874 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f16d406c6b0 0x7f16d406eb70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.614+0000 7f16f107f700 1 --2- 172.21.15.113:0/258865874 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ec0fd8f0 0x7f16ec10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.614+0000 7f16f107f700 1 --2- 172.21.15.113:0/258865874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16ec0fe250 0x7f16ec10e4d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.614+0000 7f16f107f700 1 -- 172.21.15.113:0/258865874 >> 172.21.15.113:0/258865874 conn(0x7f16ec0f34b0 msgr2=0x7f16ec0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:38.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.615+0000 7f16f107f700 1 -- 172.21.15.113:0/258865874 shutdown_connections 2024-04-12T05:13:38.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.615+0000 7f16f107f700 1 -- 172.21.15.113:0/258865874 wait complete. 2024-04-12T05:13:38.621 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 18 2024-04-12T05:13:38.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:37.204130+0000 mgr.smithi113.ctsazg (mgr.14656) 144 : audit [DBG] from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:38.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:37.537008+0000 mgr.smithi113.ctsazg (mgr.14656) 145 : audit [DBG] from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:38.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:38.246426+0000 mon.smithi113 (mon.0) 1011 : audit [DBG] from='client.? 172.21.15.113:0/1758271651' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:13:38.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:38.345430+0000 mon.smithi113 (mon.0) 1012 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:38.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:38.346326+0000 mon.smithi113 (mon.0) 1013 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:13:38.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:38.346901+0000 mon.smithi113 (mon.0) 1014 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:13:38.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:38 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:38.347400+0000 mon.smithi113 (mon.0) 1015 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:13:38.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.716+0000 7f5ef555f700 1 -- 172.21.15.113:0/905615399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ef00fe2d0 msgr2=0x7f5ef00f7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:38.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.716+0000 7f5ef555f700 1 --2- 172.21.15.113:0/905615399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ef00fe2d0 0x7f5ef00f7ab0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f5ee00047f0 tx=0x7f5ee0009cb0).stop 2024-04-12T05:13:38.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.716+0000 7f5ef555f700 1 -- 172.21.15.113:0/905615399 shutdown_connections 2024-04-12T05:13:38.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.716+0000 7f5ef555f700 1 --2- 172.21.15.113:0/905615399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ef00fe2d0 0x7f5ef00f7ab0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.716+0000 7f5ef555f700 1 --2- 172.21.15.113:0/905615399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ef00fd970 0x7f5ef00fdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:13:38.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.716+0000 7f5ef555f700 1 -- 172.21.15.113:0/905615399 >> 172.21.15.113:0/905615399 conn(0x7f5ef00f3550 msgr2=0x7f5ef00f5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:38.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.716+0000 7f5ef555f700 1 -- 172.21.15.113:0/905615399 shutdown_connections 2024-04-12T05:13:38.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.716+0000 7f5ef555f700 1 -- 172.21.15.113:0/905615399 wait complete. 2024-04-12T05:13:38.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5ef555f700 1 Processor -- start 2024-04-12T05:13:38.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5ef555f700 1 -- start start 2024-04-12T05:13:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5ef555f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ef00fd970 0x7f5ef010e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5ef555f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ef00fe2d0 0x7f5ef010e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5ef555f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef010ebf0 con 0x7f5ef00fe2d0 2024-04-12T05:13:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5ef555f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef0108210 con 0x7f5ef00fd970 2024-04-12T05:13:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5eee7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ef00fe2d0 0x7f5ef010e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5eee7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ef00fe2d0 0x7f5ef010e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41994/0 (socket says 172.21.15.113:41994) 2024-04-12T05:13:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5eee7fc700 1 -- 172.21.15.113:0/1793079980 learned_addr learned my addr 172.21.15.113:0/1793079980 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5eeeffd700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ef00fd970 0x7f5ef010e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5eee7fc700 1 -- 172.21.15.113:0/1793079980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ef00fd970 msgr2=0x7f5ef010e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5eee7fc700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ef00fd970 0x7f5ef010e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5eee7fc700 1 -- 172.21.15.113:0/1793079980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee0005040 con 0x7f5ef00fe2d0 2024-04-12T05:13:38.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5eeeffd700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ef00fd970 0x7f5ef010e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:13:38.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.717+0000 7f5eee7fc700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ef00fe2d0 0x7f5ef010e560 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f5ee00099f0 tx=0x7f5ee0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:38.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.718+0000 7f5ed7fff700 1 -- 172.21.15.113:0/1793079980 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ee000e030 con 0x7f5ef00fe2d0 2024-04-12T05:13:38.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.718+0000 7f5ed7fff700 1 -- 172.21.15.113:0/1793079980 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ee0007e90 con 0x7f5ef00fe2d0 2024-04-12T05:13:38.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.718+0000 7f5ed7fff700 1 -- 172.21.15.113:0/1793079980 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ee0018070 con 0x7f5ef00fe2d0 2024-04-12T05:13:38.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.718+0000 7f5ef555f700 1 -- 172.21.15.113:0/1793079980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef0108460 con 0x7f5ef00fe2d0 2024-04-12T05:13:38.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.718+0000 7f5ef555f700 1 -- 172.21.15.113:0/1793079980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef0108940 con 0x7f5ef00fe2d0 2024-04-12T05:13:38.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.719+0000 7f5ef555f700 1 -- 172.21.15.113:0/1793079980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef0060370 con 0x7f5ef00fe2d0 2024-04-12T05:13:38.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.719+0000 7f5ed7fff700 1 -- 172.21.15.113:0/1793079980 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f5ee001d420 con 0x7f5ef00fe2d0 2024-04-12T05:13:38.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.720+0000 7f5ed7fff700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f5ed806c6a0 0x7f5ed806eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:38.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.720+0000 7f5ed7fff700 1 -- 172.21.15.113:0/1793079980 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f5ee008cb70 con 0x7f5ef00fe2d0 2024-04-12T05:13:38.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.720+0000 7f5eeeffd700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f5ed806c6a0 0x7f5ed806eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:38.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.720+0000 7f5eeeffd700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f5ed806c6a0 0x7f5ed806eb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f5ee4009990 tx=0x7f5ee4008040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:38.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.724+0000 7f5ed7fff700 1 -- 172.21.15.113:0/1793079980 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f5ee005ac70 con 0x7f5ef00fe2d0 2024-04-12T05:13:38.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.935+0000 7f5ef555f700 1 -- 172.21.15.113:0/1793079980 --> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ef01097e0 con 0x7f5ed806c6a0 2024-04-12T05:13:38.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.937+0000 7f5ed7fff700 1 -- 172.21.15.113:0/1793079980 <== mgr.14656 v2:172.21.15.113:6800/3082109414 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f5ef01097e0 con 0x7f5ed806c6a0 2024-04-12T05:13:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:13:38.939 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:13:38.939 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:13:38.939 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:13:38.939 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:13:38.939 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr" 2024-04-12T05:13:38.939 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:13:38.939 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "2/21 daemons upgraded", 2024-04-12T05:13:38.939 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mon daemons", 2024-04-12T05:13:38.939 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:13:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:13:38.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.939+0000 7f5ef555f700 1 -- 172.21.15.113:0/1793079980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f5ed806c6a0 msgr2=0x7f5ed806eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:38.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.939+0000 7f5ef555f700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f5ed806c6a0 0x7f5ed806eb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f5ee4009990 tx=0x7f5ee4008040).stop 2024-04-12T05:13:38.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.939+0000 7f5ef555f700 1 -- 172.21.15.113:0/1793079980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ef00fe2d0 msgr2=0x7f5ef010e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:38.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.939+0000 7f5ef555f700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ef00fe2d0 0x7f5ef010e560 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f5ee00099f0 tx=0x7f5ee0009cb0).stop 2024-04-12T05:13:38.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.939+0000 7f5ef555f700 1 -- 172.21.15.113:0/1793079980 shutdown_connections 2024-04-12T05:13:38.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.939+0000 7f5ef555f700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f5ed806c6a0 0x7f5ed806eb60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.939+0000 7f5ef555f700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ef00fd970 0x7f5ef010e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.939+0000 7f5ef555f700 1 --2- 172.21.15.113:0/1793079980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ef00fe2d0 0x7f5ef010e560 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.939+0000 7f5ef555f700 1 -- 172.21.15.113:0/1793079980 >> 172.21.15.113:0/1793079980 conn(0x7f5ef00f3550 msgr2=0x7f5ef00fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.939+0000 7f5ef555f700 1 -- 172.21.15.113:0/1793079980 shutdown_connections 2024-04-12T05:13:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:38.939+0000 7f5ef555f700 1 -- 172.21.15.113:0/1793079980 wait complete. 2024-04-12T05:13:38.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:37.204130+0000 mgr.smithi113.ctsazg (mgr.14656) 144 : audit [DBG] from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:38.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:37.537008+0000 mgr.smithi113.ctsazg (mgr.14656) 145 : audit [DBG] from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:38.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:38.246426+0000 mon.smithi113 (mon.0) 1011 : audit [DBG] from='client.? 172.21.15.113:0/1758271651' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:13:38.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:38.345430+0000 mon.smithi113 (mon.0) 1012 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:38.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:38.346326+0000 mon.smithi113 (mon.0) 1013 : audit [INF] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:13:38.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:38.346901+0000 mon.smithi113 (mon.0) 1014 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:13:38.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:38.347400+0000 mon.smithi113 (mon.0) 1015 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:13:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.040+0000 7f6eba4a4700 1 -- 172.21.15.113:0/754903111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6eb40fda00 msgr2=0x7f6eb40fdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.040+0000 7f6eba4a4700 1 --2- 172.21.15.113:0/754903111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6eb40fda00 0x7f6eb40fdde0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f6ea40047f0 tx=0x7f6ea4009cb0).stop 2024-04-12T05:13:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.040+0000 7f6eba4a4700 1 -- 172.21.15.113:0/754903111 shutdown_connections 2024-04-12T05:13:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.040+0000 7f6eba4a4700 1 --2- 172.21.15.113:0/754903111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb40fe360 0x7f6eb40f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.040+0000 7f6eba4a4700 1 --2- 172.21.15.113:0/754903111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6eb40fda00 0x7f6eb40fdde0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.040+0000 7f6eba4a4700 1 -- 172.21.15.113:0/754903111 >> 172.21.15.113:0/754903111 conn(0x7f6eb406cea0 msgr2=0x7f6eb406d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.040+0000 7f6eba4a4700 1 -- 172.21.15.113:0/754903111 shutdown_connections 2024-04-12T05:13:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.040+0000 7f6eba4a4700 1 -- 172.21.15.113:0/754903111 wait complete. 2024-04-12T05:13:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.040+0000 7f6eba4a4700 1 Processor -- start 2024-04-12T05:13:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.040+0000 7f6eba4a4700 1 -- start start 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eba4a4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6eb40fda00 0x7f6eb410dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eba4a4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb40fe360 0x7f6eb410e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eba4a4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb410eac0 con 0x7f6eb40fda00 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eba4a4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb410ec00 con 0x7f6eb40fe360 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6eb40fda00 0x7f6eb410dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6eb40fda00 0x7f6eb410dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42002/0 (socket says 172.21.15.113:42002) 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb3fff700 1 -- 172.21.15.113:0/2413665709 learned_addr learned my addr 172.21.15.113:0/2413665709 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb37fe700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb40fe360 0x7f6eb410e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb3fff700 1 -- 172.21.15.113:0/2413665709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb40fe360 msgr2=0x7f6eb410e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb3fff700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb40fe360 0x7f6eb410e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb3fff700 1 -- 172.21.15.113:0/2413665709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea4005040 con 0x7f6eb40fda00 2024-04-12T05:13:39.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb37fe700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb40fe360 0x7f6eb410e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:13:39.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb3fff700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6eb40fda00 0x7f6eb410dfd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f6ea400b120 tx=0x7f6ea4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:39.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb17fa700 1 -- 172.21.15.113:0/2413665709 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ea400e030 con 0x7f6eb40fda00 2024-04-12T05:13:39.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb17fa700 1 -- 172.21.15.113:0/2413665709 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ea4007e90 con 0x7f6eb40fda00 2024-04-12T05:13:39.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eba4a4700 1 -- 172.21.15.113:0/2413665709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb4108230 con 0x7f6eb40fda00 2024-04-12T05:13:39.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eb17fa700 1 -- 172.21.15.113:0/2413665709 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ea4018070 con 0x7f6eb40fda00 2024-04-12T05:13:39.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.041+0000 7f6eba4a4700 1 -- 172.21.15.113:0/2413665709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb4108790 con 0x7f6eb40fda00 2024-04-12T05:13:39.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.042+0000 7f6eba4a4700 1 -- 172.21.15.113:0/2413665709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb4047540 con 0x7f6eb40fda00 2024-04-12T05:13:39.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.043+0000 7f6eb17fa700 1 -- 172.21.15.113:0/2413665709 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 32) v1 ==== 90679+0+0 (secure 0 0 0) 0x7f6ea401d420 con 0x7f6eb40fda00 2024-04-12T05:13:39.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.043+0000 7f6eb17fa700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f6ea006c6a0 0x7f6ea006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:13:39.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.043+0000 7f6eb37fe700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f6ea006c6a0 0x7f6ea006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:13:39.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.043+0000 7f6eb17fa700 1 -- 172.21.15.113:0/2413665709 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f6ea408cb30 con 0x7f6eb40fda00 2024-04-12T05:13:39.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.044+0000 7f6eb37fe700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f6ea006c6a0 0x7f6ea006eb60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f6eb4109910 tx=0x7f6e9c009040).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:13:39.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.048+0000 7f6eb17fa700 1 -- 172.21.15.113:0/2413665709 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181111 (secure 0 0 0) 0x7f6ea405ac30 con 0x7f6eb40fda00 2024-04-12T05:13:39.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.326+0000 7f6eba4a4700 1 -- 172.21.15.113:0/2413665709 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6eb4109650 con 0x7f6eb40fda00 2024-04-12T05:13:39.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.326+0000 7f6eb17fa700 1 -- 172.21.15.113:0/2413665709 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f6ea405a7c0 con 0x7f6eb40fda00 2024-04-12T05:13:39.329 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_OK 2024-04-12T05:13:39.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.328+0000 7f6eba4a4700 1 -- 172.21.15.113:0/2413665709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f6ea006c6a0 msgr2=0x7f6ea006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:39.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.328+0000 7f6eba4a4700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f6ea006c6a0 0x7f6ea006eb60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f6eb4109910 tx=0x7f6e9c009040).stop 2024-04-12T05:13:39.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.328+0000 7f6eba4a4700 1 -- 172.21.15.113:0/2413665709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6eb40fda00 msgr2=0x7f6eb410dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:13:39.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.328+0000 7f6eba4a4700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6eb40fda00 0x7f6eb410dfd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f6ea400b120 tx=0x7f6ea4009cb0).stop 2024-04-12T05:13:39.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.328+0000 7f6eba4a4700 1 -- 172.21.15.113:0/2413665709 shutdown_connections 2024-04-12T05:13:39.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.328+0000 7f6eba4a4700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.113:6800/3082109414,v1:172.21.15.113:6801/3082109414] conn(0x7f6ea006c6a0 0x7f6ea006eb60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:39.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.328+0000 7f6eba4a4700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6eb40fda00 0x7f6eb410dfd0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:39.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.328+0000 7f6eba4a4700 1 --2- 172.21.15.113:0/2413665709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6eb40fe360 0x7f6eb410e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:13:39.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.328+0000 7f6eba4a4700 1 -- 172.21.15.113:0/2413665709 >> 172.21.15.113:0/2413665709 conn(0x7f6eb406cea0 msgr2=0x7f6eb40fbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:13:39.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.328+0000 7f6eba4a4700 1 -- 172.21.15.113:0/2413665709 shutdown_connections 2024-04-12T05:13:39.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:13:39.328+0000 7f6eba4a4700 1 -- 172.21.15.113:0/2413665709 wait complete. 2024-04-12T05:13:39.856 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:37.857478+0000 mgr.smithi113.ctsazg (mgr.14656) 146 : audit [DBG] from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:39.856 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cluster 2024-04-12T05:13:37.981546+0000 mgr.smithi113.ctsazg (mgr.14656) 147 : cluster [DBG] pgmap v84: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:39.856 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:13:38.341309+0000 mgr.smithi113.ctsazg (mgr.14656) 148 : cephadm [INF] Upgrade: Updating mon.smithi113 2024-04-12T05:13:39.856 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: cephadm 2024-04-12T05:13:38.348260+0000 mgr.smithi113.ctsazg (mgr.14656) 149 : cephadm [INF] Deploying daemon mon.smithi113 on smithi113 2024-04-12T05:13:39.856 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:38.613502+0000 mon.smithi113 (mon.0) 1016 : audit [DBG] from='client.? 172.21.15.113:0/258865874' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:13:39.856 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:39 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: audit 2024-04-12T05:13:39.327359+0000 mon.smithi113 (mon.0) 1017 : audit [DBG] from='client.? 172.21.15.113:0/2413665709' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:13:39.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:37.857478+0000 mgr.smithi113.ctsazg (mgr.14656) 146 : audit [DBG] from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:13:39.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:37.981546+0000 mgr.smithi113.ctsazg (mgr.14656) 147 : cluster [DBG] pgmap v84: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:39.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:13:38.341309+0000 mgr.smithi113.ctsazg (mgr.14656) 148 : cephadm [INF] Upgrade: Updating mon.smithi113 2024-04-12T05:13:39.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:13:38.348260+0000 mgr.smithi113.ctsazg (mgr.14656) 149 : cephadm [INF] Deploying daemon mon.smithi113 on smithi113 2024-04-12T05:13:39.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:38.613502+0000 mon.smithi113 (mon.0) 1016 : audit [DBG] from='client.? 172.21.15.113:0/258865874' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:13:39.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:39.327359+0000 mon.smithi113 (mon.0) 1017 : audit [DBG] from='client.? 172.21.15.113:0/2413665709' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:13:40.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:40 smithi113 systemd[1]: Stopping Ceph mon.smithi113 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:13:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T05:13:40.322+0000 7fc48b02e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi113 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-12T05:13:40.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[39678]: debug 2024-04-12T05:13:40.322+0000 7fc48b02e700 -1 mon.smithi113@0(leader) e2 *** Got Signal Terminated *** 2024-04-12T05:13:41.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:40 smithi113 bash[103166]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113 2024-04-12T05:13:41.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 systemd[1]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi113.service: Succeeded. 2024-04-12T05:13:41.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 systemd[1]: Stopped Ceph mon.smithi113 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:13:41.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 systemd[1]: Starting Ceph mon.smithi113 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:13:42.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T05:13:42.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable), process ceph-mon, pid 7 2024-04-12T05:13:42.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: pidfile_write: ignore empty --pid-file 2024-04-12T05:13:42.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: load: jerasure load: lrc 2024-04-12T05:13:42.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: RocksDB version: 7.9.2 2024-04-12T05:13:42.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Git sha 0 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Compile date 2024-04-08 23:21:21 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: DB SUMMARY 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: DB Session ID: B98XM4WH7POW8FIS7TLL 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: CURRENT file: CURRENT 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: MANIFEST file: MANIFEST-000008 size: 1174 Bytes 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi113/store.db dir, Total Num: 1, files: 000026.sst 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi113/store.db: 000024.log size: 2750023 ; 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.error_if_exists: 0 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.create_if_missing: 0 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.paranoid_checks: 1 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-12T05:13:42.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.env: 0x5616abb167a0 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.fs: PosixFileSystem 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.info_log: 0x5616ada173a0 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.statistics: (nil) 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.use_fsync: 0 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_log_file_size: 0 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T05:13:42.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.allow_fallocate: 1 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.use_direct_reads: 0 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.db_log_dir: 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.wal_dir: 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T05:13:42.179 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.write_buffer_manager: 0x5616acc8c140 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.unordered_write: 0 2024-04-12T05:13:42.180 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.row_cache: None 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.wal_filter: None 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.two_write_queues: 0 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.wal_compression: 0 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.atomic_flush: 0 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T05:13:42.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.log_readahead_size: 0 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_background_jobs: 2 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_background_compactions: -1 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_subcompactions: 1 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T05:13:42.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_open_files: -1 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T05:13:42.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_background_flushes: -1 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Compression algorithms supported: 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: kZSTD supported: 0 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: kXpressCompression supported: 0 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: kZlibCompression supported: 1 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: kSnappyCompression supported: 1 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: kLZ4Compression supported: 1 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: kBZip2Compression supported: 0 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi113/store.db/MANIFEST-000008 2024-04-12T05:13:42.184 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.merge_operator: 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_filter: None 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5616ada174c0) 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: cache_index_and_filter_blocks: 1 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T05:13:42.185 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: pin_top_level_index_and_filter: 1 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: index_type: 0 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: data_block_index_type: 0 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: index_shortening: 1 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: checksum: 4 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: no_block_cache: 0 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_cache: 0x5616acd2d1f0 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_cache_name: BinnedLRUCache 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_cache_options: 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: capacity : 536870912 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: num_shard_bits : 4 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: strict_capacity_limit : 0 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: high_pri_pool_ratio: 0.000 2024-04-12T05:13:42.186 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_cache_compressed: (nil) 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: persistent_cache: (nil) 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_size: 4096 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_size_deviation: 10 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_restart_interval: 16 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: index_block_restart_interval: 1 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: metadata_block_size: 4096 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: partition_filters: 0 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: use_delta_encoding: 1 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: filter_policy: bloomfilter 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: whole_key_filtering: 1 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: verify_compression: 0 2024-04-12T05:13:42.187 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: read_amp_bytes_per_bit: 0 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: format_version: 5 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: enable_index_compression: 1 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: block_align: 0 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: max_auto_readahead_size: 262144 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: prepopulate_block_cache: 0 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: initial_auto_readahead_size: 8192 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: num_file_reads_for_auto_readahead: 2 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compression: NoCompression 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T05:13:42.188 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.num_levels: 7 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T05:13:42.189 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-12T05:13:42.190 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T05:13:42.191 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.arena_block_size: 1048576 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T05:13:42.192 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.table_properties_collectors: 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.inplace_update_support: 0 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.bloom_locality: 0 2024-04-12T05:13:42.193 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.max_successive_merges: 0 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.ttl: 2592000 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.enable_blob_files: false 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.min_blob_size: 0 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T05:13:42.194 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.blob_file_starting_level: 0 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 26.sst 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi113/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 28, last_sequence is 11367, log_number is 24,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 24 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f3d56d38-0d65-40db-aa37-b157612f96e8 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712898821895004, "job": 1, "event": "recovery_started", "wal_files": [24]} 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #24 mode 2 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712898821901959, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 29, "file_size": 2742564, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11368, "largest_seqno": 11925, "table_properties": {"data_size": 2738024, "index_size": 3009, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4288, "raw_average_key_size": 21, "raw_value_size": 2731846, "raw_average_value_size": 13391, "num_data_blocks": 133, "num_entries": 204, "num_filter_entries": 204, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712898821, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f3d56d38-0d65-40db-aa37-b157612f96e8", "db_session_id": "B98XM4WH7POW8FIS7TLL", "orig_file_number": 29, "seqno_to_time_mapping": "N/A"}} 2024-04-12T05:13:42.195 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712898821902030, "job": 1, "event": "recovery_finished"} 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [db/version_set.cc:5047] Creating manifest 31 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi113/store.db/000024.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5616acdcc000 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: DB pointer 0x5616acdba000 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ** DB Stats ** 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T05:13:42.196 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ** Compaction Stats [default] ** 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: L0 1/0 2.62 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 708.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: L6 1/0 26.87 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Sum 2/0 29.49 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 708.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 708.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ** Compaction Stats [default] ** 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 708.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T05:13:42.197 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-12T05:13:42.198 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-12T05:13:42.198 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-12T05:13:42.198 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T05:13:42.198 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-12T05:13:42.198 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T05:13:42.198 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T05:13:42.198 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T05:13:42.198 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T05:13:42.198 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Cumulative compaction: 0.00 GB write, 242.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:13:42.198 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Interval compaction: 0.00 GB write, 242.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:13:42.198 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T05:13:42.199 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Block cache BinnedLRUCache@0x5616acd2d1f0#7 capacity: 512.00 MB usage: 3.67 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-12T05:13:42.199 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,3.08 KB,0.000587106%) Misc(1,0.00 KB,0%) 2024-04-12T05:13:42.199 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-12T05:13:42.199 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T05:13:42.199 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: starting mon.smithi113 rank 0 at public addrs [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] at bind addrs [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi113 fsid 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T05:13:42.199 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: mon.smithi113@-1(???) e2 preinit fsid 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T05:13:42.199 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: mon.smithi113@-1(???).mds e18 new map 2024-04-12T05:13:42.199 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: mon.smithi113@-1(???).mds e18 print_map 2024-04-12T05:13:42.199 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: e18 2024-04-12T05:13:42.200 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:13:42.200 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:13:42.200 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: legacy client fscid: 1 2024-04-12T05:13:42.200 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-12T05:13:42.200 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Filesystem 'cephfs' (1) 2024-04-12T05:13:42.200 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: fs_name cephfs 2024-04-12T05:13:42.200 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: epoch 18 2024-04-12T05:13:42.200 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:13:42.201 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:13:42.201 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:13:42.201 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: tableserver 0 2024-04-12T05:13:42.201 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: root 0 2024-04-12T05:13:42.201 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: session_timeout 60 2024-04-12T05:13:42.201 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: session_autoclose 300 2024-04-12T05:13:42.201 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: max_file_size 1099511627776 2024-04-12T05:13:42.201 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: max_xattr_size 65536 2024-04-12T05:13:42.201 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: required_client_features {} 2024-04-12T05:13:42.201 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: last_failure 0 2024-04-12T05:13:42.202 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: last_failure_osd_epoch 0 2024-04-12T05:13:42.202 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:13:42.202 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: max_mds 2 2024-04-12T05:13:42.202 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: in 0,1 2024-04-12T05:13:42.202 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: up {0=14426,1=24261} 2024-04-12T05:13:42.202 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: failed 2024-04-12T05:13:42.202 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: damaged 2024-04-12T05:13:42.202 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: stopped 2024-04-12T05:13:42.202 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: data_pools [3] 2024-04-12T05:13:42.202 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: metadata_pool 2 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: inline_data disabled 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: balancer 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: bal_rank_mask -1 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: standby_count_wanted 1 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: [mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: [mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: Standby daemons: 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: [mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout: [mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: mon.smithi113@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-12T05:13:42.203 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: mon.smithi113@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:13:42.204 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: mon.smithi113@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:13:42.204 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: mon.smithi113@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:13:42.204 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 ceph-mon[103371]: mon.smithi113@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-04-12T05:13:42.204 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 bash[103333]: 98cfc95ea472e54d8d995bddb343d33f478f3954c94dcfafc78471eb0e494b8d 2024-04-12T05:13:42.204 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:13:41 smithi113 systemd[1]: Started Ceph mon.smithi113 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:13:45.299 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:40.330862+0000 mon.smithi192 (mon.1) 86 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:40.331610+0000 mon.smithi192 (mon.1) 87 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:41.023689+0000 mon.smithi192 (mon.1) 88 : audit [DBG] from='mgr.14656 172.21.15.113:0/37031107' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:41.982425+0000 mgr.smithi113.ctsazg (mgr.14656) 152 : cluster [DBG] pgmap v86: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:43.982916+0000 mgr.smithi113.ctsazg (mgr.14656) 153 : cluster [DBG] pgmap v87: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:44.921517+0000 mon.smithi113 (mon.0) 1 : cluster [INF] mon.smithi113 calling monitor election 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:44.935946+0000 mon.smithi113 (mon.0) 2 : cluster [INF] mon.smithi113 is new leader, mons smithi113,smithi192 in quorum (ranks 0,1) 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:44.941218+0000 mon.smithi113 (mon.0) 3 : cluster [DBG] monmap e2: 2 mons at {smithi113=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:44.941844+0000 mon.smithi113 (mon.0) 4 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:44.941879+0000 mon.smithi113 (mon.0) 5 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:44.942934+0000 mon.smithi113 (mon.0) 6 : cluster [DBG] mgrmap e32: smithi113.ctsazg(active, since 2m), standbys: smithi192.ocppyo 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:44.943411+0000 mon.smithi113 (mon.0) 7 : cluster [INF] overall HEALTH_OK 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:44.959924+0000 mon.smithi113 (mon.0) 8 : cluster [DBG] mgrmap e33: smithi113.ctsazg(active, since 2m), standbys: smithi192.ocppyo 2024-04-12T05:13:45.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:44 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:44.960001+0000 mon.smithi113 (mon.0) 9 : audit [INF] from='mgr.14656 ' entity='' 2024-04-12T05:13:51.863 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:51.719656+0000 mon.smithi113 (mon.0) 10 : cluster [DBG] Standby manager daemon smithi192.ocppyo restarted 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:51.719723+0000 mon.smithi113 (mon.0) 11 : cluster [DBG] Standby manager daemon smithi192.ocppyo started 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.722229+0000 mon.smithi192 (mon.1) 89 : audit [DBG] from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/crt"}]: dispatch 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.722727+0000 mon.smithi192 (mon.1) 90 : audit [DBG] from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.725207+0000 mon.smithi192 (mon.1) 91 : audit [DBG] from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/key"}]: dispatch 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.725558+0000 mon.smithi192 (mon.1) 92 : audit [DBG] from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:51.744552+0000 mon.smithi113 (mon.0) 12 : cluster [INF] Active manager daemon smithi113.ctsazg restarted 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:51.744867+0000 mon.smithi113 (mon.0) 13 : cluster [INF] Activating manager daemon smithi113.ctsazg 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:51.750637+0000 mon.smithi113 (mon.0) 14 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:51.750883+0000 mon.smithi113 (mon.0) 15 : cluster [DBG] mgrmap e34: smithi113.ctsazg(active, starting, since 0.00611264s), standbys: smithi192.ocppyo 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.752044+0000 mon.smithi113 (mon.0) 16 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.752168+0000 mon.smithi113 (mon.0) 17 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.753516+0000 mon.smithi113 (mon.0) 18 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.753950+0000 mon.smithi113 (mon.0) 19 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:13:51.864 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.754204+0000 mon.smithi113 (mon.0) 20 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.754561+0000 mon.smithi113 (mon.0) 21 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.755884+0000 mon.smithi113 (mon.0) 22 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi113.ctsazg", "id": "smithi113.ctsazg"}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.756063+0000 mon.smithi113 (mon.0) 23 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi192.ocppyo", "id": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.756226+0000 mon.smithi113 (mon.0) 24 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.756444+0000 mon.smithi113 (mon.0) 25 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.756635+0000 mon.smithi113 (mon.0) 26 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.756809+0000 mon.smithi113 (mon.0) 27 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.756975+0000 mon.smithi113 (mon.0) 28 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.757159+0000 mon.smithi113 (mon.0) 29 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.757360+0000 mon.smithi113 (mon.0) 30 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.757642+0000 mon.smithi113 (mon.0) 31 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:13:51.865 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:51 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.757996+0000 mon.smithi113 (mon.0) 32 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:13:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:51.775415+0000 mon.smithi113 (mon.0) 33 : cluster [INF] Manager daemon smithi113.ctsazg is now available 2024-04-12T05:13:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.796618+0000 mon.smithi113 (mon.0) 34 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:13:53.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.815982+0000 mon.smithi113 (mon.0) 35 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:13:53.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.824245+0000 mon.smithi113 (mon.0) 36 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:13:53.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:52 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:51.839133+0000 mon.smithi113 (mon.0) 37 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/trash_purge_schedule"}]: dispatch 2024-04-12T05:13:53.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:52.781122+0000 mon.smithi113 (mon.0) 38 : cluster [DBG] mgrmap e35: smithi113.ctsazg(active, since 1.03635s), standbys: smithi192.ocppyo 2024-04-12T05:13:53.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:53 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:13:52.980640+0000 mgr.smithi113.ctsazg (mgr.34104) 2 : cephadm [INF] [12/Apr/2024:05:13:52] ENGINE Bus STARTING 2024-04-12T05:13:54.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:13:53.082230+0000 mgr.smithi113.ctsazg (mgr.34104) 3 : cephadm [INF] [12/Apr/2024:05:13:53] ENGINE Serving on http://172.21.15.113:8765 2024-04-12T05:13:54.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:13:53.195864+0000 mgr.smithi113.ctsazg (mgr.34104) 4 : cephadm [INF] [12/Apr/2024:05:13:53] ENGINE Serving on https://172.21.15.113:7150 2024-04-12T05:13:54.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:13:53.196032+0000 mgr.smithi113.ctsazg (mgr.34104) 5 : cephadm [INF] [12/Apr/2024:05:13:53] ENGINE Bus STARTED 2024-04-12T05:13:54.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:53.755982+0000 mgr.smithi113.ctsazg (mgr.34104) 6 : cluster [DBG] pgmap v4: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:54.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:54 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:53.784773+0000 mon.smithi113 (mon.0) 39 : cluster [DBG] mgrmap e36: smithi113.ctsazg(active, since 2s), standbys: smithi192.ocppyo 2024-04-12T05:13:57.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:55.756269+0000 mgr.smithi113.ctsazg (mgr.34104) 7 : cluster [DBG] pgmap v5: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:13:57.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:55.791515+0000 mon.smithi113 (mon.0) 40 : cluster [DBG] mgrmap e37: smithi113.ctsazg(active, since 4s), standbys: smithi192.ocppyo 2024-04-12T05:13:57.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:56.062242+0000 mon.smithi113 (mon.0) 41 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:57.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:56.069142+0000 mon.smithi113 (mon.0) 42 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:57.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:56.704307+0000 mon.smithi113 (mon.0) 43 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:57.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:56.710155+0000 mon.smithi113 (mon.0) 44 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:57.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:56.748106+0000 mon.smithi113 (mon.0) 45 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:57.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:56.752362+0000 mon.smithi113 (mon.0) 46 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:58.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:57.385774+0000 mon.smithi113 (mon.0) 47 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:58.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:13:57.392305+0000 mon.smithi113 (mon.0) 48 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:13:58.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:13:58 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:57.756573+0000 mgr.smithi113.ctsazg (mgr.34104) 8 : cluster [DBG] pgmap v6: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:01.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:00 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:13:59.757228+0000 mgr.smithi113.ctsazg (mgr.34104) 9 : cluster [DBG] pgmap v7: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-12T05:14:02.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:14:01.063931+0000 mgr.smithi113.ctsazg (mgr.34104) 10 : cephadm [INF] Detected new or changed devices on smithi113 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:01.068352+0000 mon.smithi113 (mon.0) 49 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:01.074643+0000 mon.smithi113 (mon.0) 50 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:01.075832+0000 mon.smithi113 (mon.0) 51 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:01.076468+0000 mon.smithi113 (mon.0) 52 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:14:01.077365+0000 mgr.smithi113.ctsazg (mgr.34104) 11 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.conf 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:14:01.077509+0000 mgr.smithi113.ctsazg (mgr.34104) 12 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.conf 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:14:01.342084+0000 mgr.smithi113.ctsazg (mgr.34104) 13 : cephadm [INF] Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:14:01.342220+0000 mgr.smithi113.ctsazg (mgr.34104) 14 : cephadm [INF] Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:14:01.606443+0000 mgr.smithi113.ctsazg (mgr.34104) 15 : cephadm [INF] Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:14:01.606572+0000 mgr.smithi113.ctsazg (mgr.34104) 16 : cephadm [INF] Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:14:01.757536+0000 mgr.smithi113.ctsazg (mgr.34104) 17 : cluster [DBG] pgmap v8: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:14:01.870526+0000 mgr.smithi113.ctsazg (mgr.34104) 18 : cephadm [INF] Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:14:02.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:14:01.870670+0000 mgr.smithi113.ctsazg (mgr.34104) 19 : cephadm [INF] Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:14:03.433 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:02.139214+0000 mon.smithi113 (mon.0) 53 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:03.433 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:02.141528+0000 mon.smithi113 (mon.0) 54 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:03.434 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:02.145218+0000 mon.smithi113 (mon.0) 55 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:03.434 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:02.148224+0000 mon.smithi113 (mon.0) 56 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:03.434 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:02.151421+0000 mon.smithi113 (mon.0) 57 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:03.434 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:02.289267+0000 mon.smithi113 (mon.0) 58 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:14:03.434 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:02.291435+0000 mon.smithi113 (mon.0) 59 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:14:03.434 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:02.292290+0000 mon.smithi113 (mon.0) 60 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T05:14:04.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cluster 2024-04-12T05:14:03.758172+0000 mgr.smithi113.ctsazg (mgr.34104) 20 : cluster [DBG] pgmap v9: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T05:14:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:04.065553+0000 mon.smithi113 (mon.0) 61 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:04.066467+0000 mon.smithi113 (mon.0) 62 : audit [INF] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:14:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:04.067145+0000 mon.smithi113 (mon.0) 63 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:14:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: audit 2024-04-12T05:14:04.067884+0000 mon.smithi113 (mon.0) 64 : audit [DBG] from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:14:04.060630+0000 mgr.smithi113.ctsazg (mgr.34104) 21 : cephadm [INF] Upgrade: Updating mon.smithi192 2024-04-12T05:14:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: cephadm 2024-04-12T05:14:04.068815+0000 mgr.smithi113.ctsazg (mgr.34104) 22 : cephadm [INF] Deploying daemon mon.smithi192 on smithi192 2024-04-12T05:14:05.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:05 smithi192 systemd[1]: Stopping Ceph mon.smithi192 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:14:06.287 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: debug 2024-04-12T05:14:06.016+0000 7f6a2f63c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi192 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-12T05:14:06.287 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[42618]: debug 2024-04-12T05:14:06.016+0000 7f6a2f63c700 -1 mon.smithi192@1(peon) e2 *** Got Signal Terminated *** 2024-04-12T05:14:06.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:06 smithi192 bash[85111]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192 2024-04-12T05:14:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:06 smithi192 systemd[1]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi192.service: Succeeded. 2024-04-12T05:14:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:06 smithi192 systemd[1]: Stopped Ceph mon.smithi192 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:14:07.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:06 smithi192 systemd[1]: Starting Ceph mon.smithi192 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:14:07.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T05:14:07.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable), process ceph-mon, pid 7 2024-04-12T05:14:07.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: pidfile_write: ignore empty --pid-file 2024-04-12T05:14:07.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: load: jerasure load: lrc 2024-04-12T05:14:07.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: RocksDB version: 7.9.2 2024-04-12T05:14:07.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Git sha 0 2024-04-12T05:14:07.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Compile date 2024-04-08 23:21:21 2024-04-12T05:14:07.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: DB SUMMARY 2024-04-12T05:14:07.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: DB Session ID: NUTFXUABANDCIGRC1TFK 2024-04-12T05:14:07.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: CURRENT file: CURRENT 2024-04-12T05:14:07.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: MANIFEST file: MANIFEST-000005 size: 1193 Bytes 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi192/store.db dir, Total Num: 1, files: 000025.sst 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi192/store.db: 000023.log size: 7739762 ; 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.error_if_exists: 0 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.create_if_missing: 0 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.paranoid_checks: 1 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.env: 0x559bf94fb7a0 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.fs: PosixFileSystem 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.info_log: 0x559bfc24f3a0 2024-04-12T05:14:07.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.statistics: (nil) 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.use_fsync: 0 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_log_file_size: 0 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.allow_fallocate: 1 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.use_direct_reads: 0 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T05:14:07.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.db_log_dir: 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.wal_dir: 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.write_buffer_manager: 0x559bfb4c4140 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T05:14:07.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.unordered_write: 0 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.row_cache: None 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.wal_filter: None 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T05:14:07.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.two_write_queues: 0 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.wal_compression: 0 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.atomic_flush: 0 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.log_readahead_size: 0 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T05:14:07.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T05:14:07.955 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T05:14:07.956 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-12T05:14:07.956 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_background_jobs: 2 2024-04-12T05:14:07.956 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_background_compactions: -1 2024-04-12T05:14:07.956 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_subcompactions: 1 2024-04-12T05:14:07.956 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T05:14:07.956 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T05:14:07.956 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T05:14:07.956 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T05:14:07.956 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_open_files: -1 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_background_flushes: -1 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Compression algorithms supported: 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T05:14:07.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: kZSTD supported: 0 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: kXpressCompression supported: 0 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: kZlibCompression supported: 1 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: kSnappyCompression supported: 1 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: kLZ4Compression supported: 1 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: kBZip2Compression supported: 0 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi192/store.db/MANIFEST-000005 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.merge_operator: 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_filter: None 2024-04-12T05:14:07.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559bfc24f4c0) 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: cache_index_and_filter_blocks: 1 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: pin_top_level_index_and_filter: 1 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: index_type: 0 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: data_block_index_type: 0 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: index_shortening: 1 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T05:14:07.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: checksum: 4 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: no_block_cache: 0 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache: 0x559bfb5651f0 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache_name: BinnedLRUCache 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache_options: 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: capacity : 536870912 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: num_shard_bits : 4 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: strict_capacity_limit : 0 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: high_pri_pool_ratio: 0.000 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache_compressed: (nil) 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: persistent_cache: (nil) 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_size: 4096 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_size_deviation: 10 2024-04-12T05:14:07.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_restart_interval: 16 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: index_block_restart_interval: 1 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: metadata_block_size: 4096 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: partition_filters: 0 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: use_delta_encoding: 1 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: filter_policy: bloomfilter 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: whole_key_filtering: 1 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: verify_compression: 0 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: read_amp_bytes_per_bit: 0 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: format_version: 5 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: enable_index_compression: 1 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_align: 0 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: max_auto_readahead_size: 262144 2024-04-12T05:14:07.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: prepopulate_block_cache: 0 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: initial_auto_readahead_size: 8192 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: num_file_reads_for_auto_readahead: 2 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compression: NoCompression 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.num_levels: 7 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T05:14:07.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T05:14:07.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T05:14:07.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.arena_block_size: 1048576 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T05:14:07.965 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.table_properties_collectors: 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.inplace_update_support: 0 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T05:14:07.966 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.bloom_locality: 0 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.max_successive_merges: 0 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.ttl: 2592000 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.enable_blob_files: false 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.min_blob_size: 0 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T05:14:07.967 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.blob_file_starting_level: 0 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 25.sst 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi192/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 27, last_sequence is 11878, log_number is 23,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 23 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9bd4ee1b-dab7-4d23-b02f-a3d15b9a94a6 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712898847476494, "job": 1, "event": "recovery_started", "wal_files": [23]} 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #23 mode 2 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712898847492969, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 28, "file_size": 5081856, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11883, "largest_seqno": 12874, "table_properties": {"data_size": 5075151, "index_size": 4854, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 837, "raw_key_size": 6824, "raw_average_key_size": 20, "raw_value_size": 5065281, "raw_average_value_size": 15585, "num_data_blocks": 212, "num_entries": 325, "num_filter_entries": 325, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712898847, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9bd4ee1b-dab7-4d23-b02f-a3d15b9a94a6", "db_session_id": "NUTFXUABANDCIGRC1TFK", "orig_file_number": 28, "seqno_to_time_mapping": "N/A"}} 2024-04-12T05:14:07.968 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712898847493120, "job": 1, "event": "recovery_finished"} 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [db/version_set.cc:5047] Creating manifest 30 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi192/store.db/000023.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559bfb604000 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: DB pointer 0x559bfb5f2000 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** DB Stats ** 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T05:14:07.969 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** Compaction Stats [default] ** 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: L0 1/0 4.85 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 645.8 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: L6 1/0 26.87 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Sum 2/0 31.72 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 645.8 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 645.8 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** Compaction Stats [default] ** 2024-04-12T05:14:07.970 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 645.8 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Flush(GB): cumulative 0.005, interval 0.005 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative compaction: 0.00 GB write, 251.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:14:07.971 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval compaction: 0.00 GB write, 251.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T05:14:07.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T05:14:07.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Block cache BinnedLRUCache@0x559bfb5651f0#7 capacity: 512.00 MB usage: 5.98 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-12T05:14:07.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.91 KB,0.000172853%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-04-12T05:14:07.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-04-12T05:14:07.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T05:14:07.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: starting mon.smithi192 rank 1 at public addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] at bind addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi192 fsid 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T05:14:07.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: mon.smithi192@-1(???) e2 preinit fsid 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T05:14:07.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: mon.smithi192@-1(???).mds e18 new map 2024-04-12T05:14:07.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: mon.smithi192@-1(???).mds e18 print_map 2024-04-12T05:14:07.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: e18 2024-04-12T05:14:07.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: legacy client fscid: 1 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Filesystem 'cephfs' (1) 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: fs_name cephfs 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: epoch 18 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: tableserver 0 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: root 0 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: session_timeout 60 2024-04-12T05:14:07.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: session_autoclose 300 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: max_file_size 1099511627776 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: max_xattr_size 65536 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: required_client_features {} 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: last_failure 0 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: last_failure_osd_epoch 0 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: max_mds 2 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: in 0,1 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: up {0=14426,1=24261} 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: failed 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: damaged 2024-04-12T05:14:07.974 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: stopped 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: data_pools [3] 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: metadata_pool 2 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: inline_data disabled 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: balancer 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: bal_rank_mask -1 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: standby_count_wanted 1 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: [mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: [mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Standby daemons: 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: [mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:07.975 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: [mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:07.976 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: mon.smithi192@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-12T05:14:07.976 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: mon.smithi192@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:14:07.976 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: mon.smithi192@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:14:07.976 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: mon.smithi192@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-12T05:14:07.976 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 ceph-mon[85304]: mon.smithi192@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-04-12T05:14:07.976 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 bash[85266]: e99449ab860c2b654f7c1c33a48d7995dd6145243745264ae13762df9be563ad 2024-04-12T05:14:07.976 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:07 smithi192 systemd[1]: Started Ceph mon.smithi192 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:14:08.589 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mkfs 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T05:14:08.589 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mkfs 52321e2c-f889-11ee-bc8c-c7b262605968 2024-04-12T05:14:08.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:14:08.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:14:08.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:14:08.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:14:08.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Reconfiguring prometheus.smithi192 (dependencies changed)... 2024-04-12T05:14:08.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Reconfiguring daemon prometheus.smithi192 on smithi192 2024-04-12T05:14:08.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Updating mgr.smithi192.ocppyo 2024-04-12T05:14:08.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Deploying daemon mgr.smithi192.ocppyo on smithi192 2024-04-12T05:14:08.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all mgr 2024-04-12T05:14:08.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all rgw 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all rbd-mirror 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all ceph-exporter 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all iscsi 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all nfs 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all nvmeof 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Updating node-exporter.smithi113 (1/2) 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Deploying daemon node-exporter.smithi113 on smithi113 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Updating node-exporter.smithi192 (2/2) 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Deploying daemon node-exporter.smithi192 on smithi192 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Updating prometheus.smithi192 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Deploying daemon prometheus.smithi192 on smithi192 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Updating alertmanager.smithi113 2024-04-12T05:14:08.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Deploying daemon alertmanager.smithi113 on smithi113 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Updating grafana.smithi113 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Deploying daemon grafana.smithi113 on smithi113 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Finalizing container_image settings 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Complete! 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v61: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v62: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v63: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v64: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v65: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v66: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v67: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v68: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v69: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v70: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v71: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v72: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v73: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v74: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v75: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v76: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v77: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v78: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v79: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v80: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v81: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v82: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v83: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.928 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Target is version 18.2.2-1187-gd540ebac (reef) 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3'] 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all mgr 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v84: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Updating mon.smithi113 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Deploying daemon mon.smithi113 on smithi113 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v86: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v87: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mon.smithi113 calling monitor election 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mon.smithi113 is new leader, mons smithi113,smithi192 in quorum (ranks 0,1) 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: monmap e2: 2 mons at {smithi113=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: osdmap e48: 6 total, 6 up, 6 in 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mgrmap e32: smithi113.ctsazg(active, since 2m), standbys: smithi192.ocppyo 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: overall HEALTH_OK 2024-04-12T05:14:08.929 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mgrmap e33: smithi113.ctsazg(active, since 2m), standbys: smithi192.ocppyo 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.14656 ' entity='' 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Standby manager daemon smithi192.ocppyo restarted 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Standby manager daemon smithi192.ocppyo started 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/crt"}]: dispatch 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/key"}]: dispatch 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Active manager daemon smithi113.ctsazg restarted 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Activating manager daemon smithi113.ctsazg 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: osdmap e49: 6 total, 6 up, 6 in 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mgrmap e34: smithi113.ctsazg(active, starting, since 0.00611264s), standbys: smithi192.ocppyo 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:14:08.930 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi113.ctsazg", "id": "smithi113.ctsazg"}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi192.ocppyo", "id": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:14:08.931 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:14:08.932 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Manager daemon smithi113.ctsazg is now available 2024-04-12T05:14:08.932 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:14:08.932 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:14:08.932 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:14:08.932 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/trash_purge_schedule"}]: dispatch 2024-04-12T05:14:08.932 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mgrmap e35: smithi113.ctsazg(active, since 1.03635s), standbys: smithi192.ocppyo 2024-04-12T05:14:08.932 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: [12/Apr/2024:05:13:52] ENGINE Bus STARTING 2024-04-12T05:14:08.932 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: [12/Apr/2024:05:13:53] ENGINE Serving on http://172.21.15.113:8765 2024-04-12T05:14:08.932 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: [12/Apr/2024:05:13:53] ENGINE Serving on https://172.21.15.113:7150 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: [12/Apr/2024:05:13:53] ENGINE Bus STARTED 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v4: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mgrmap e36: smithi113.ctsazg(active, since 2s), standbys: smithi192.ocppyo 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v5: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mgrmap e37: smithi113.ctsazg(active, since 4s), standbys: smithi192.ocppyo 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.933 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v6: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v7: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Detected new or changed devices on smithi113 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi113:/etc/ceph/ceph.conf 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi192:/etc/ceph/ceph.conf 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:14:08.934 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v8: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v9: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Upgrade: Updating mon.smithi192 2024-04-12T05:14:08.935 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: Deploying daemon mon.smithi192 on smithi192 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mon.smithi113 calling monitor election 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mon.smithi192 calling monitor election 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mon.smithi113 is new leader, mons smithi113,smithi192 in quorum (ranks 0,1) 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: monmap e3: 2 mons at {smithi113=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: osdmap e49: 6 total, 6 up, 6 in 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: mgrmap e37: smithi113.ctsazg(active, since 15s), standbys: smithi192.ocppyo 2024-04-12T05:14:08.936 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: overall HEALTH_OK 2024-04-12T05:14:08.937 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.937 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.937 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:14:08.937 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:08 smithi113 ceph-mon[103371]: pgmap v11: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Reconfiguring prometheus.smithi192 (dependencies changed)... 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Reconfiguring daemon prometheus.smithi192 on smithi192 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Updating mgr.smithi192.ocppyo 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Deploying daemon mgr.smithi192.ocppyo on smithi192 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all mgr 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all rgw 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all rbd-mirror 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-12T05:14:08.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all ceph-exporter 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all iscsi 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all nfs 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all nvmeof 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Updating node-exporter.smithi113 (1/2) 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Deploying daemon node-exporter.smithi113 on smithi113 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Updating node-exporter.smithi192 (2/2) 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Deploying daemon node-exporter.smithi192 on smithi192 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Updating prometheus.smithi192 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Deploying daemon prometheus.smithi192 on smithi192 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Updating alertmanager.smithi113 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Deploying daemon alertmanager.smithi113 on smithi113 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Updating grafana.smithi113 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Deploying daemon grafana.smithi113 on smithi113 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Finalizing container_image settings 2024-04-12T05:14:08.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Complete! 2024-04-12T05:14:08.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v61: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v62: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v63: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v64: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v65: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v66: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v67: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v68: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v69: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v70: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.951 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v71: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v72: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v73: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v74: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v75: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v76: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v77: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v78: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.952 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v79: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v80: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v81: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v82: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v83: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Target is version 18.2.2-1187-gd540ebac (reef) 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3'] 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all mgr 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v84: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Updating mon.smithi113 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Deploying daemon mon.smithi113 on smithi113 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v86: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v87: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mon.smithi113 calling monitor election 2024-04-12T05:14:08.953 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mon.smithi113 is new leader, mons smithi113,smithi192 in quorum (ranks 0,1) 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: monmap e2: 2 mons at {smithi113=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: osdmap e48: 6 total, 6 up, 6 in 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mgrmap e32: smithi113.ctsazg(active, since 2m), standbys: smithi192.ocppyo 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: overall HEALTH_OK 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mgrmap e33: smithi113.ctsazg(active, since 2m), standbys: smithi192.ocppyo 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.14656 ' entity='' 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Standby manager daemon smithi192.ocppyo restarted 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Standby manager daemon smithi192.ocppyo started 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/crt"}]: dispatch 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.ocppyo/key"}]: dispatch 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.? 172.21.15.192:0/1126654284' entity='mgr.smithi192.ocppyo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Active manager daemon smithi113.ctsazg restarted 2024-04-12T05:14:08.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Activating manager daemon smithi113.ctsazg 2024-04-12T05:14:08.955 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: osdmap e49: 6 total, 6 up, 6 in 2024-04-12T05:14:08.955 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mgrmap e34: smithi113.ctsazg(active, starting, since 0.00611264s), standbys: smithi192.ocppyo 2024-04-12T05:14:08.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:14:08.957 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi113.ctsazg", "id": "smithi113.ctsazg"}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr metadata", "who": "smithi192.ocppyo", "id": "smithi192.ocppyo"}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:14:08.958 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Manager daemon smithi113.ctsazg is now available 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/mirror_snapshot_schedule"}]: dispatch 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi113.ctsazg/trash_purge_schedule"}]: dispatch 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mgrmap e35: smithi113.ctsazg(active, since 1.03635s), standbys: smithi192.ocppyo 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: [12/Apr/2024:05:13:52] ENGINE Bus STARTING 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: [12/Apr/2024:05:13:53] ENGINE Serving on http://172.21.15.113:8765 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: [12/Apr/2024:05:13:53] ENGINE Serving on https://172.21.15.113:7150 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: [12/Apr/2024:05:13:53] ENGINE Bus STARTED 2024-04-12T05:14:08.959 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v4: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mgrmap e36: smithi113.ctsazg(active, since 2s), standbys: smithi192.ocppyo 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v5: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mgrmap e37: smithi113.ctsazg(active, since 4s), standbys: smithi192.ocppyo 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v6: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v7: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Detected new or changed devices on smithi113 2024-04-12T05:14:08.960 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi113:/etc/ceph/ceph.conf 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi192:/etc/ceph/ceph.conf 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.conf 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v8: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Updating smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.961 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v9: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Upgrade: Updating mon.smithi192 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: Deploying daemon mon.smithi192 on smithi192 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi113"}]: dispatch 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mon.smithi113 calling monitor election 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mon.smithi192 calling monitor election 2024-04-12T05:14:08.962 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-04-12T05:14:08.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mon.smithi113 is new leader, mons smithi113,smithi192 in quorum (ranks 0,1) 2024-04-12T05:14:08.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: monmap e3: 2 mons at {smithi113=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-12T05:14:08.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi192.ammjap=up:active} 2 up:standby 2024-04-12T05:14:08.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: osdmap e49: 6 total, 6 up, 6 in 2024-04-12T05:14:08.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: mgrmap e37: smithi113.ctsazg(active, since 15s), standbys: smithi192.ocppyo 2024-04-12T05:14:08.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: overall HEALTH_OK 2024-04-12T05:14:08.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:08.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:14:08.963 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:08 smithi192 ceph-mon[85304]: pgmap v11: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T05:14:09.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.441+0000 7fbe254e8700 1 -- 172.21.15.113:0/2403118419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe200fe250 msgr2=0x7fbe200f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:09.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.441+0000 7fbe254e8700 1 --2- 172.21.15.113:0/2403118419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe200fe250 0x7fbe200f7a30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fbe100047f0 tx=0x7fbe10009cb0).stop 2024-04-12T05:14:09.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.441+0000 7fbe254e8700 1 -- 172.21.15.113:0/2403118419 shutdown_connections 2024-04-12T05:14:09.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.441+0000 7fbe254e8700 1 --2- 172.21.15.113:0/2403118419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe200fe250 0x7fbe200f7a30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.441+0000 7fbe254e8700 1 --2- 172.21.15.113:0/2403118419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe200fd8f0 0x7fbe200fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.441+0000 7fbe254e8700 1 -- 172.21.15.113:0/2403118419 >> 172.21.15.113:0/2403118419 conn(0x7fbe200f34b0 msgr2=0x7fbe200f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:09.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.441+0000 7fbe254e8700 1 -- 172.21.15.113:0/2403118419 shutdown_connections 2024-04-12T05:14:09.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.441+0000 7fbe254e8700 1 -- 172.21.15.113:0/2403118419 wait complete. 2024-04-12T05:14:09.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.442+0000 7fbe254e8700 1 Processor -- start 2024-04-12T05:14:09.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.442+0000 7fbe254e8700 1 -- start start 2024-04-12T05:14:09.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.442+0000 7fbe254e8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe200fd8f0 0x7fbe2010dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.442+0000 7fbe254e8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe200fe250 0x7fbe2010e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.442+0000 7fbe254e8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2010eae0 con 0x7fbe200fd8f0 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.442+0000 7fbe254e8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2010ec20 con 0x7fbe200fe250 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.442+0000 7fbe1effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe200fd8f0 0x7fbe2010dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.442+0000 7fbe1effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe200fd8f0 0x7fbe2010dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41642/0 (socket says 172.21.15.113:41642) 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.442+0000 7fbe1effd700 1 -- 172.21.15.113:0/3613869987 learned_addr learned my addr 172.21.15.113:0/3613869987 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe1e7fc700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe200fe250 0x7fbe2010e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe1e7fc700 1 -- 172.21.15.113:0/3613869987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe200fe250 msgr2=0x7fbe2010e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe1e7fc700 1 -- 172.21.15.113:0/3613869987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe200fe250 msgr2=0x7fbe2010e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe1e7fc700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe200fe250 0x7fbe2010e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe1e7fc700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe200fe250 0x7fbe2010e4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe1effd700 1 -- 172.21.15.113:0/3613869987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe200fe250 msgr2=0x7fbe2010e4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe1effd700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe200fe250 0x7fbe2010e4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe1effd700 1 -- 172.21.15.113:0/3613869987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe10005040 con 0x7fbe200fd8f0 2024-04-12T05:14:09.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe1effd700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe200fd8f0 0x7fbe2010dff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fbe140041d0 tx=0x7fbe14009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:09.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe07fff700 1 -- 172.21.15.113:0/3613869987 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe14007b40 con 0x7fbe200fd8f0 2024-04-12T05:14:09.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe07fff700 1 -- 172.21.15.113:0/3613869987 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe1400f040 con 0x7fbe200fd8f0 2024-04-12T05:14:09.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe07fff700 1 -- 172.21.15.113:0/3613869987 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe1400d800 con 0x7fbe200fd8f0 2024-04-12T05:14:09.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe254e8700 1 -- 172.21.15.113:0/3613869987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe201082b0 con 0x7fbe200fd8f0 2024-04-12T05:14:09.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.443+0000 7fbe254e8700 1 -- 172.21.15.113:0/3613869987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe20108870 con 0x7fbe200fd8f0 2024-04-12T05:14:09.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.444+0000 7fbe254e8700 1 -- 172.21.15.113:0/3613869987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2005f590 con 0x7fbe200fd8f0 2024-04-12T05:14:09.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.445+0000 7fbe07fff700 1 -- 172.21.15.113:0/3613869987 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7fbe14007ca0 con 0x7fbe200fd8f0 2024-04-12T05:14:09.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.445+0000 7fbe07fff700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe0806cbc0 0x7fbe0806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:09.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.445+0000 7fbe1e7fc700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe0806cbc0 0x7fbe0806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:09.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.445+0000 7fbe07fff700 1 -- 172.21.15.113:0/3613869987 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fbe1408b990 con 0x7fbe200fd8f0 2024-04-12T05:14:09.449 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.446+0000 7fbe1e7fc700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe0806cbc0 0x7fbe0806f080 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fbe10004f10 tx=0x7fbe1000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:09.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.450+0000 7fbe07fff700 1 -- 172.21.15.113:0/3613869987 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbe14059380 con 0x7fbe200fd8f0 2024-04-12T05:14:09.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.656+0000 7fbe254e8700 1 -- 172.21.15.113:0/3613869987 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe20109780 con 0x7fbe0806cbc0 2024-04-12T05:14:09.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.657+0000 7fbe07fff700 1 -- 172.21.15.113:0/3613869987 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fbe20109780 con 0x7fbe0806cbc0 2024-04-12T05:14:09.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.659+0000 7fbe254e8700 1 -- 172.21.15.113:0/3613869987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe0806cbc0 msgr2=0x7fbe0806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:09.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.659+0000 7fbe254e8700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe0806cbc0 0x7fbe0806f080 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fbe10004f10 tx=0x7fbe1000b040).stop 2024-04-12T05:14:09.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.660+0000 7fbe254e8700 1 -- 172.21.15.113:0/3613869987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe200fd8f0 msgr2=0x7fbe2010dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:09.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.660+0000 7fbe254e8700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe200fd8f0 0x7fbe2010dff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fbe140041d0 tx=0x7fbe14009f60).stop 2024-04-12T05:14:09.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.660+0000 7fbe254e8700 1 -- 172.21.15.113:0/3613869987 shutdown_connections 2024-04-12T05:14:09.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.660+0000 7fbe254e8700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe0806cbc0 0x7fbe0806f080 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.660+0000 7fbe254e8700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe200fd8f0 0x7fbe2010dff0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.660+0000 7fbe254e8700 1 --2- 172.21.15.113:0/3613869987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe200fe250 0x7fbe2010e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.660+0000 7fbe254e8700 1 -- 172.21.15.113:0/3613869987 >> 172.21.15.113:0/3613869987 conn(0x7fbe200f34b0 msgr2=0x7fbe200fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:09.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.660+0000 7fbe254e8700 1 -- 172.21.15.113:0/3613869987 shutdown_connections 2024-04-12T05:14:09.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.660+0000 7fbe254e8700 1 -- 172.21.15.113:0/3613869987 wait complete. 2024-04-12T05:14:09.672 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:14:09.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.758+0000 7efcc4b5d700 1 -- 172.21.15.113:0/3899087178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc00fd150 msgr2=0x7efcc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:09.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.758+0000 7efcc4b5d700 1 --2- 172.21.15.113:0/3899087178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc00fd150 0x7efcc00f6930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7efcb00047f0 tx=0x7efcb0009cb0).stop 2024-04-12T05:14:09.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.758+0000 7efcc4b5d700 1 -- 172.21.15.113:0/3899087178 shutdown_connections 2024-04-12T05:14:09.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.758+0000 7efcc4b5d700 1 --2- 172.21.15.113:0/3899087178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc00fd150 0x7efcc00f6930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.758+0000 7efcc4b5d700 1 --2- 172.21.15.113:0/3899087178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc00fc7f0 0x7efcc00fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.758+0000 7efcc4b5d700 1 -- 172.21.15.113:0/3899087178 >> 172.21.15.113:0/3899087178 conn(0x7efcc00f23b0 msgr2=0x7efcc00f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:09.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.758+0000 7efcc4b5d700 1 -- 172.21.15.113:0/3899087178 shutdown_connections 2024-04-12T05:14:09.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.758+0000 7efcc4b5d700 1 -- 172.21.15.113:0/3899087178 wait complete. 2024-04-12T05:14:09.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.759+0000 7efcc4b5d700 1 Processor -- start 2024-04-12T05:14:09.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.759+0000 7efcc4b5d700 1 -- start start 2024-04-12T05:14:09.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.759+0000 7efcc4b5d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc00fc7f0 0x7efcc01090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:09.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.759+0000 7efcc4b5d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc00fd150 0x7efcc01095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:09.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.759+0000 7efcc4b5d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc0109aa0 con 0x7efcc00fc7f0 2024-04-12T05:14:09.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.759+0000 7efcc4b5d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc0109be0 con 0x7efcc00fd150 2024-04-12T05:14:09.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcbe59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc00fc7f0 0x7efcc01090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:09.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcbe59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc00fc7f0 0x7efcc01090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41664/0 (socket says 172.21.15.113:41664) 2024-04-12T05:14:09.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcbe59c700 1 -- 172.21.15.113:0/809153777 learned_addr learned my addr 172.21.15.113:0/809153777 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:09.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcbdd9b700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc00fd150 0x7efcc01095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:09.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcbe59c700 1 -- 172.21.15.113:0/809153777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc00fd150 msgr2=0x7efcc01095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:09.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcbe59c700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc00fd150 0x7efcc01095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcbe59c700 1 -- 172.21.15.113:0/809153777 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb0005040 con 0x7efcc00fc7f0 2024-04-12T05:14:09.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcbdd9b700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc00fd150 0x7efcc01095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:14:09.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcbe59c700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc00fc7f0 0x7efcc01090c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efcb40041d0 tx=0x7efcb4009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:09.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcaf7fe700 1 -- 172.21.15.113:0/809153777 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcb4007b40 con 0x7efcc00fc7f0 2024-04-12T05:14:09.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcaf7fe700 1 -- 172.21.15.113:0/809153777 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efcb400f040 con 0x7efcc00fc7f0 2024-04-12T05:14:09.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcc4b5d700 1 -- 172.21.15.113:0/809153777 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc0109e90 con 0x7efcc00fc7f0 2024-04-12T05:14:09.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcaf7fe700 1 -- 172.21.15.113:0/809153777 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcb400d800 con 0x7efcc00fc7f0 2024-04-12T05:14:09.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.760+0000 7efcc4b5d700 1 -- 172.21.15.113:0/809153777 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc01124b0 con 0x7efcc00fc7f0 2024-04-12T05:14:09.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.761+0000 7efcc4b5d700 1 -- 172.21.15.113:0/809153777 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc0060370 con 0x7efcc00fc7f0 2024-04-12T05:14:09.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.762+0000 7efcaf7fe700 1 -- 172.21.15.113:0/809153777 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7efcb4009110 con 0x7efcc00fc7f0 2024-04-12T05:14:09.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.762+0000 7efcaf7fe700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efca806cbc0 0x7efca806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:09.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.762+0000 7efcaf7fe700 1 -- 172.21.15.113:0/809153777 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7efcb408baa0 con 0x7efcc00fc7f0 2024-04-12T05:14:09.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.762+0000 7efcbdd9b700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efca806cbc0 0x7efca806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:09.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.762+0000 7efcbdd9b700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efca806cbc0 0x7efca806f080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7efcb00043d0 tx=0x7efcb000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:09.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.766+0000 7efcaf7fe700 1 -- 172.21.15.113:0/809153777 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7efcb4055a00 con 0x7efcc00fc7f0 2024-04-12T05:14:09.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.973+0000 7efcc4b5d700 1 -- 172.21.15.113:0/809153777 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efcc0059120 con 0x7efca806cbc0 2024-04-12T05:14:09.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.974+0000 7efcaf7fe700 1 -- 172.21.15.113:0/809153777 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7efcc0059120 con 0x7efca806cbc0 2024-04-12T05:14:09.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.976+0000 7efcc4b5d700 1 -- 172.21.15.113:0/809153777 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efca806cbc0 msgr2=0x7efca806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:09.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.976+0000 7efcc4b5d700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efca806cbc0 0x7efca806f080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7efcb00043d0 tx=0x7efcb000b040).stop 2024-04-12T05:14:09.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.976+0000 7efcc4b5d700 1 -- 172.21.15.113:0/809153777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc00fc7f0 msgr2=0x7efcc01090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:09.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.976+0000 7efcc4b5d700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc00fc7f0 0x7efcc01090c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efcb40041d0 tx=0x7efcb4009f60).stop 2024-04-12T05:14:09.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.976+0000 7efcc4b5d700 1 -- 172.21.15.113:0/809153777 shutdown_connections 2024-04-12T05:14:09.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.976+0000 7efcc4b5d700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efca806cbc0 0x7efca806f080 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.976+0000 7efcc4b5d700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc00fc7f0 0x7efcc01090c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.976+0000 7efcc4b5d700 1 --2- 172.21.15.113:0/809153777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc00fd150 0x7efcc01095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:09.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.976+0000 7efcc4b5d700 1 -- 172.21.15.113:0/809153777 >> 172.21.15.113:0/809153777 conn(0x7efcc00f23b0 msgr2=0x7efcc00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:09.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.976+0000 7efcc4b5d700 1 -- 172.21.15.113:0/809153777 shutdown_connections 2024-04-12T05:14:09.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:09.976+0000 7efcc4b5d700 1 -- 172.21.15.113:0/809153777 wait complete. 2024-04-12T05:14:10.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.075+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/92451709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fcc0f9200 msgr2=0x7f8fcc101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:10.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.075+0000 7f8fd1e9f700 1 --2- 172.21.15.113:0/92451709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fcc0f9200 0x7f8fcc101910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f8fbc0047f0 tx=0x7f8fbc009cb0).stop 2024-04-12T05:14:10.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.075+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/92451709 shutdown_connections 2024-04-12T05:14:10.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.075+0000 7f8fd1e9f700 1 --2- 172.21.15.113:0/92451709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fcc0f9200 0x7f8fcc101910 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.075+0000 7f8fd1e9f700 1 --2- 172.21.15.113:0/92451709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fcc0f88a0 0x7f8fcc0f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.075+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/92451709 >> 172.21.15.113:0/92451709 conn(0x7f8fcc070590 msgr2=0x7f8fcc0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:10.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.075+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/92451709 shutdown_connections 2024-04-12T05:14:10.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.075+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/92451709 wait complete. 2024-04-12T05:14:10.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.075+0000 7f8fd1e9f700 1 Processor -- start 2024-04-12T05:14:10.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fd1e9f700 1 -- start start 2024-04-12T05:14:10.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fd1e9f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fcc0f88a0 0x7f8fcc10e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:10.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fd1e9f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fcc0f9200 0x7f8fcc10e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:10.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fd1e9f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc10eb70 con 0x7f8fcc0f88a0 2024-04-12T05:14:10.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fd1e9f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc10ecb0 con 0x7f8fcc0f9200 2024-04-12T05:14:10.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fcc0f88a0 0x7f8fcc10e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fcc0f88a0 0x7f8fcc10e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41682/0 (socket says 172.21.15.113:41682) 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcb7fe700 1 -- 172.21.15.113:0/1740263276 learned_addr learned my addr 172.21.15.113:0/1740263276 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcaffd700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fcc0f9200 0x7f8fcc10e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcaffd700 1 -- 172.21.15.113:0/1740263276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fcc0f9200 msgr2=0x7f8fcc10e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcaffd700 1 -- 172.21.15.113:0/1740263276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fcc0f9200 msgr2=0x7f8fcc10e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcaffd700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fcc0f9200 0x7f8fcc10e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcaffd700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fcc0f9200 0x7f8fcc10e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcb7fe700 1 -- 172.21.15.113:0/1740263276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fcc0f9200 msgr2=0x7f8fcc10e570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcb7fe700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fcc0f9200 0x7f8fcc10e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcb7fe700 1 -- 172.21.15.113:0/1740263276 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc005040 con 0x7f8fcc0f88a0 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.076+0000 7f8fcb7fe700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fcc0f88a0 0x7f8fcc10e080 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f8fc000c740 tx=0x7f8fc0007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.077+0000 7f8fc8ff9700 1 -- 172.21.15.113:0/1740263276 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fc000e680 con 0x7f8fcc0f88a0 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.077+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/1740263276 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc108340 con 0x7f8fcc0f88a0 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.077+0000 7f8fc8ff9700 1 -- 172.21.15.113:0/1740263276 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8fc000ec90 con 0x7f8fcc0f88a0 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.077+0000 7f8fc8ff9700 1 -- 172.21.15.113:0/1740263276 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fc0013d60 con 0x7f8fcc0f88a0 2024-04-12T05:14:10.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.077+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/1740263276 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc108880 con 0x7f8fcc0f88a0 2024-04-12T05:14:10.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.078+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/1740263276 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc05f590 con 0x7f8fcc0f88a0 2024-04-12T05:14:10.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.078+0000 7f8fc8ff9700 1 -- 172.21.15.113:0/1740263276 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f8fc000e7e0 con 0x7f8fcc0f88a0 2024-04-12T05:14:10.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.078+0000 7f8fc8ff9700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fb406cbc0 0x7f8fb406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:10.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.079+0000 7f8fc8ff9700 1 -- 172.21.15.113:0/1740263276 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f8fc008c160 con 0x7f8fcc0f88a0 2024-04-12T05:14:10.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.079+0000 7f8fcaffd700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fb406cbc0 0x7f8fb406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:10.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.079+0000 7f8fcaffd700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fb406cbc0 0x7f8fb406f080 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f8fbc004f10 tx=0x7f8fbc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:10.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.083+0000 7f8fc8ff9700 1 -- 172.21.15.113:0/1740263276 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8fc0059c60 con 0x7f8fcc0f88a0 2024-04-12T05:14:10.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.280+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/1740263276 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fcc108b00 con 0x7f8fb406cbc0 2024-04-12T05:14:10.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.290+0000 7f8fc8ff9700 1 -- 172.21.15.113:0/1740263276 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f8fcc108b00 con 0x7f8fb406cbc0 2024-04-12T05:14:10.292 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:14:10.292 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (2m) 13s ago 13m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:14:10.292 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (13m) 13s ago 13m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:14:10.292 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (12m) 14s ago 12m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (106s) 13s ago 12m 84.0M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (7m) 13s ago 7m 121M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (7m) 13s ago 7m 17.6M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (7m) 14s ago 7m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (7m) 14s ago 7m 15.0M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (3m) 13s ago 14m 468M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (2m) 14s ago 12m 446M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (28s) 13s ago 14m 27.2M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 starting - - - 2048M 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (2m) 13s ago 12m 16.3M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (2m) 14s ago 12m 17.0M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (11m) 13s ago 11m 227M 1910M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (11m) 13s ago 11m 239M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (10m) 13s ago 10m 202M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (10m) 14s ago 10m 264M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:14:10.293 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (9m) 14s ago 9m 233M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:14:10.294 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (9m) 14s ago 9m 226M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:14:10.294 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (2m) 14s ago 12m 54.8M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:14:10.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.292+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/1740263276 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fb406cbc0 msgr2=0x7f8fb406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:10.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.292+0000 7f8fd1e9f700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fb406cbc0 0x7f8fb406f080 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f8fbc004f10 tx=0x7f8fbc00b040).stop 2024-04-12T05:14:10.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.292+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/1740263276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fcc0f88a0 msgr2=0x7f8fcc10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.292+0000 7f8fd1e9f700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fcc0f88a0 0x7f8fcc10e080 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f8fc000c740 tx=0x7f8fc0007ef0).stop 2024-04-12T05:14:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.292+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/1740263276 shutdown_connections 2024-04-12T05:14:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.292+0000 7f8fd1e9f700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fb406cbc0 0x7f8fb406f080 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.292+0000 7f8fd1e9f700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fcc0f88a0 0x7f8fcc10e080 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.292+0000 7f8fd1e9f700 1 --2- 172.21.15.113:0/1740263276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fcc0f9200 0x7f8fcc10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.292+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/1740263276 >> 172.21.15.113:0/1740263276 conn(0x7f8fcc070590 msgr2=0x7f8fcc0fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.292+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/1740263276 shutdown_connections 2024-04-12T05:14:10.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.292+0000 7f8fd1e9f700 1 -- 172.21.15.113:0/1740263276 wait complete. 2024-04-12T05:14:10.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.389+0000 7f7abd661700 1 -- 172.21.15.113:0/1132773944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab80fe360 msgr2=0x7f7ab80f5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:10.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.389+0000 7f7abd661700 1 --2- 172.21.15.113:0/1132773944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab80fe360 0x7f7ab80f5fc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7aa80047f0 tx=0x7f7aa8009cb0).stop 2024-04-12T05:14:10.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.389+0000 7f7abd661700 1 -- 172.21.15.113:0/1132773944 shutdown_connections 2024-04-12T05:14:10.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.389+0000 7f7abd661700 1 --2- 172.21.15.113:0/1132773944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab80fe360 0x7f7ab80f5fc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.389+0000 7f7abd661700 1 --2- 172.21.15.113:0/1132773944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab80fda00 0x7f7ab80fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.389+0000 7f7abd661700 1 -- 172.21.15.113:0/1132773944 >> 172.21.15.113:0/1132773944 conn(0x7f7ab806cea0 msgr2=0x7f7ab806d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:10.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.390+0000 7f7abd661700 1 -- 172.21.15.113:0/1132773944 shutdown_connections 2024-04-12T05:14:10.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.390+0000 7f7abd661700 1 -- 172.21.15.113:0/1132773944 wait complete. 2024-04-12T05:14:10.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.390+0000 7f7abd661700 1 Processor -- start 2024-04-12T05:14:10.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.390+0000 7f7abd661700 1 -- start start 2024-04-12T05:14:10.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7abd661700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab80fda00 0x7f7ab810dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:10.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7abd661700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab80fe360 0x7f7ab810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:10.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7abd661700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab810eae0 con 0x7f7ab80fe360 2024-04-12T05:14:10.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7abd661700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab810ec20 con 0x7f7ab80fda00 2024-04-12T05:14:10.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab67fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab80fe360 0x7f7ab810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:10.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab67fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab80fe360 0x7f7ab810e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43088/0 (socket says 172.21.15.113:43088) 2024-04-12T05:14:10.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab67fc700 1 -- 172.21.15.113:0/899898484 learned_addr learned my addr 172.21.15.113:0/899898484 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:10.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab6ffd700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab80fda00 0x7f7ab810dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:10.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab6ffd700 1 -- 172.21.15.113:0/899898484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab80fda00 msgr2=0x7f7ab810dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-12T05:14:10.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab6ffd700 1 -- 172.21.15.113:0/899898484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab80fda00 msgr2=0x7f7ab810dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:14:10.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab6ffd700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab80fda00 0x7f7ab810dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:14:10.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab6ffd700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab80fda00 0x7f7ab810dff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:14:10.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab67fc700 1 -- 172.21.15.113:0/899898484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab80fda00 msgr2=0x7f7ab810dff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:14:10.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab67fc700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab80fda00 0x7f7ab810dff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab67fc700 1 -- 172.21.15.113:0/899898484 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa8005040 con 0x7f7ab80fe360 2024-04-12T05:14:10.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7ab67fc700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab80fe360 0x7f7ab810e4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f7aa8005010 tx=0x7f7aa8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:10.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7a9ffff700 1 -- 172.21.15.113:0/899898484 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7aa800e030 con 0x7f7ab80fe360 2024-04-12T05:14:10.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7a9ffff700 1 -- 172.21.15.113:0/899898484 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7aa8007e90 con 0x7f7ab80fe360 2024-04-12T05:14:10.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7abd661700 1 -- 172.21.15.113:0/899898484 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab8108250 con 0x7f7ab80fe360 2024-04-12T05:14:10.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7a9ffff700 1 -- 172.21.15.113:0/899898484 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7aa8018070 con 0x7f7ab80fe360 2024-04-12T05:14:10.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.391+0000 7f7abd661700 1 -- 172.21.15.113:0/899898484 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab81087b0 con 0x7f7ab80fe360 2024-04-12T05:14:10.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.392+0000 7f7abd661700 1 -- 172.21.15.113:0/899898484 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab8047540 con 0x7f7ab80fe360 2024-04-12T05:14:10.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.393+0000 7f7a9ffff700 1 -- 172.21.15.113:0/899898484 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f7aa801d6e0 con 0x7f7ab80fe360 2024-04-12T05:14:10.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.393+0000 7f7a9ffff700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7aa006cb60 0x7f7aa006f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:10.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.394+0000 7f7a9ffff700 1 -- 172.21.15.113:0/899898484 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f7aa808d5c0 con 0x7f7ab80fe360 2024-04-12T05:14:10.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.394+0000 7f7ab6ffd700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7aa006cb60 0x7f7aa006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:10.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.394+0000 7f7ab6ffd700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7aa006cb60 0x7f7aa006f020 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f7aac009fd0 tx=0x7f7aac009dd0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:10.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.397+0000 7f7a9ffff700 1 -- 172.21.15.113:0/899898484 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7aa805aff0 con 0x7f7ab80fe360 2024-04-12T05:14:10.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.663+0000 7f7abd661700 1 -- 172.21.15.113:0/899898484 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7ab80ffd80 con 0x7f7ab80fe360 2024-04-12T05:14:10.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.664+0000 7f7a9ffff700 1 -- 172.21.15.113:0/899898484 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+742 (secure 0 0 0) 0x7f7aa805ab80 con 0x7f7ab80fe360 2024-04-12T05:14:10.666 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:14:10.666 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:14:10.666 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:14:10.666 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:14:10.666 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:14:10.666 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:14:10.666 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:14:10.666 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:14:10.666 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:14:10.667 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:14:10.667 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:14:10.667 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:14:10.667 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:14:10.667 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:14:10.667 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 10, 2024-04-12T05:14:10.667 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:14:10.667 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:14:10.667 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:14:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.666+0000 7f7abd661700 1 -- 172.21.15.113:0/899898484 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7aa006cb60 msgr2=0x7f7aa006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.666+0000 7f7abd661700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7aa006cb60 0x7f7aa006f020 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f7aac009fd0 tx=0x7f7aac009dd0).stop 2024-04-12T05:14:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.666+0000 7f7abd661700 1 -- 172.21.15.113:0/899898484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab80fe360 msgr2=0x7f7ab810e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.666+0000 7f7abd661700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab80fe360 0x7f7ab810e4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f7aa8005010 tx=0x7f7aa8009cb0).stop 2024-04-12T05:14:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.666+0000 7f7abd661700 1 -- 172.21.15.113:0/899898484 shutdown_connections 2024-04-12T05:14:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.666+0000 7f7abd661700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7aa006cb60 0x7f7aa006f020 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.666+0000 7f7abd661700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab80fda00 0x7f7ab810dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.666+0000 7f7abd661700 1 --2- 172.21.15.113:0/899898484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ab80fe360 0x7f7ab810e4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.666+0000 7f7abd661700 1 -- 172.21.15.113:0/899898484 >> 172.21.15.113:0/899898484 conn(0x7f7ab806cea0 msgr2=0x7f7ab80fbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.666+0000 7f7abd661700 1 -- 172.21.15.113:0/899898484 shutdown_connections 2024-04-12T05:14:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.666+0000 7f7abd661700 1 -- 172.21.15.113:0/899898484 wait complete. 2024-04-12T05:14:10.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.764+0000 7f98af35f700 1 -- 172.21.15.113:0/2539356537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a80fe270 msgr2=0x7f98a80f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:10.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.764+0000 7f98af35f700 1 --2- 172.21.15.113:0/2539356537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a80fe270 0x7f98a80f7a50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f98a40047f0 tx=0x7f98a4009cb0).stop 2024-04-12T05:14:10.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.764+0000 7f98af35f700 1 -- 172.21.15.113:0/2539356537 shutdown_connections 2024-04-12T05:14:10.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.764+0000 7f98af35f700 1 --2- 172.21.15.113:0/2539356537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a80fe270 0x7f98a80f7a50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.764+0000 7f98af35f700 1 --2- 172.21.15.113:0/2539356537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98a80fd910 0x7f98a80fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.764+0000 7f98af35f700 1 -- 172.21.15.113:0/2539356537 >> 172.21.15.113:0/2539356537 conn(0x7f98a80f3530 msgr2=0x7f98a80f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:10.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.764+0000 7f98af35f700 1 -- 172.21.15.113:0/2539356537 shutdown_connections 2024-04-12T05:14:10.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.764+0000 7f98af35f700 1 -- 172.21.15.113:0/2539356537 wait complete. 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.764+0000 7f98af35f700 1 Processor -- start 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.764+0000 7f98af35f700 1 -- start start 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98af35f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a80fd910 0x7f98a806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98af35f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98a80fe270 0x7f98a806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98af35f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a806bab0 con 0x7f98a80fd910 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98af35f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a806bbf0 con 0x7f98a80fe270 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ad0fb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a80fd910 0x7f98a806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ad0fb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a80fd910 0x7f98a806b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43104/0 (socket says 172.21.15.113:43104) 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ad0fb700 1 -- 172.21.15.113:0/2235815767 learned_addr learned my addr 172.21.15.113:0/2235815767 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ac8fa700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98a80fe270 0x7f98a806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ac8fa700 1 -- 172.21.15.113:0/2235815767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98a80fe270 msgr2=0x7f98a806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ac8fa700 1 -- 172.21.15.113:0/2235815767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98a80fe270 msgr2=0x7f98a806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ac8fa700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98a80fe270 0x7f98a806b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ac8fa700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98a80fe270 0x7f98a806b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:14:10.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ad0fb700 1 -- 172.21.15.113:0/2235815767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98a80fe270 msgr2=0x7f98a806b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T05:14:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ad0fb700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98a80fe270 0x7f98a806b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ad0fb700 1 -- 172.21.15.113:0/2235815767 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a4005040 con 0x7f98a80fd910 2024-04-12T05:14:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98ad0fb700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a80fd910 0x7f98a806b0d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f98980041d0 tx=0x7f9898009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f989e7fc700 1 -- 172.21.15.113:0/2235815767 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9898002d90 con 0x7f98a80fd910 2024-04-12T05:14:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f989e7fc700 1 -- 172.21.15.113:0/2235815767 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f989800f040 con 0x7f98a80fd910 2024-04-12T05:14:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98af35f700 1 -- 172.21.15.113:0/2235815767 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a806bea0 con 0x7f98a80fd910 2024-04-12T05:14:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f989e7fc700 1 -- 172.21.15.113:0/2235815767 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f989800d800 con 0x7f98a80fd910 2024-04-12T05:14:10.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.765+0000 7f98af35f700 1 -- 172.21.15.113:0/2235815767 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a81146d0 con 0x7f98a80fd910 2024-04-12T05:14:10.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.766+0000 7f98af35f700 1 -- 172.21.15.113:0/2235815767 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a805f570 con 0x7f98a80fd910 2024-04-12T05:14:10.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.767+0000 7f989e7fc700 1 -- 172.21.15.113:0/2235815767 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f989800dac0 con 0x7f98a80fd910 2024-04-12T05:14:10.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.767+0000 7f989e7fc700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f989406cbc0 0x7f989406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:10.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.767+0000 7f989e7fc700 1 -- 172.21.15.113:0/2235815767 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f989808bfc0 con 0x7f98a80fd910 2024-04-12T05:14:10.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.767+0000 7f98ac8fa700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f989406cbc0 0x7f989406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:10.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.767+0000 7f98ac8fa700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f989406cbc0 0x7f989406f080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f98a806c870 tx=0x7f98a4009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:10.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:10.771+0000 7f989e7fc700 1 -- 172.21.15.113:0/2235815767 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9898055f20 con 0x7f98a80fd910 2024-04-12T05:14:11.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.002+0000 7f98af35f700 1 -- 172.21.15.113:0/2235815767 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f98a806c650 con 0x7f98a80fd910 2024-04-12T05:14:11.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.003+0000 7f989e7fc700 1 -- 172.21.15.113:0/2235815767 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1859 (secure 0 0 0) 0x7f9898059540 con 0x7f98a80fd910 2024-04-12T05:14:11.005 INFO:teuthology.orchestra.run.smithi113.stdout:e18 2024-04-12T05:14:11.005 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:14:11.005 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:14:11.005 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:14:11.005 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:14:11.005 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 18 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:14:11.006 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=14426,1=24261} 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:14:11.007 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:14:11.008 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:14:11.008 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:11.008 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:11.008 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:14:11.008 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:14:11.008 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:14:11.008 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:14:11.008 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:11.008 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:11.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.005+0000 7f98af35f700 1 -- 172.21.15.113:0/2235815767 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f989406cbc0 msgr2=0x7f989406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:11.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.005+0000 7f98af35f700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f989406cbc0 0x7f989406f080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f98a806c870 tx=0x7f98a4009cb0).stop 2024-04-12T05:14:11.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.005+0000 7f98af35f700 1 -- 172.21.15.113:0/2235815767 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a80fd910 msgr2=0x7f98a806b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:11.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.005+0000 7f98af35f700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a80fd910 0x7f98a806b0d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f98980041d0 tx=0x7f9898009f60).stop 2024-04-12T05:14:11.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.005+0000 7f98af35f700 1 -- 172.21.15.113:0/2235815767 shutdown_connections 2024-04-12T05:14:11.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.005+0000 7f98af35f700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f989406cbc0 0x7f989406f080 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.005+0000 7f98af35f700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98a80fd910 0x7f98a806b0d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.005+0000 7f98af35f700 1 --2- 172.21.15.113:0/2235815767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98a80fe270 0x7f98a806b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.005+0000 7f98af35f700 1 -- 172.21.15.113:0/2235815767 >> 172.21.15.113:0/2235815767 conn(0x7f98a80f3530 msgr2=0x7f98a80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:11.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.005+0000 7f98af35f700 1 -- 172.21.15.113:0/2235815767 shutdown_connections 2024-04-12T05:14:11.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.005+0000 7f98af35f700 1 -- 172.21.15.113:0/2235815767 wait complete. 2024-04-12T05:14:11.010 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 18 2024-04-12T05:14:11.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.104+0000 7f9021f9d700 1 -- 172.21.15.113:0/974196902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f901c0f6710 msgr2=0x7f901c0f6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:11.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.104+0000 7f9021f9d700 1 --2- 172.21.15.113:0/974196902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f901c0f6710 0x7f901c0f6af0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f90100047f0 tx=0x7f9010009cb0).stop 2024-04-12T05:14:11.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.104+0000 7f9021f9d700 1 -- 172.21.15.113:0/974196902 shutdown_connections 2024-04-12T05:14:11.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.104+0000 7f9021f9d700 1 --2- 172.21.15.113:0/974196902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f901c0f6fe0 0x7f901c103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.104+0000 7f9021f9d700 1 --2- 172.21.15.113:0/974196902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f901c0f6710 0x7f901c0f6af0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.104+0000 7f9021f9d700 1 -- 172.21.15.113:0/974196902 >> 172.21.15.113:0/974196902 conn(0x7f901c070590 msgr2=0x7f901c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:11.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.104+0000 7f9021f9d700 1 -- 172.21.15.113:0/974196902 shutdown_connections 2024-04-12T05:14:11.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.104+0000 7f9021f9d700 1 -- 172.21.15.113:0/974196902 wait complete. 2024-04-12T05:14:11.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.104+0000 7f9021f9d700 1 Processor -- start 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f9021f9d700 1 -- start start 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f9021f9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f901c0f6710 0x7f901c110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f9021f9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f901c0f6fe0 0x7f901c110750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f9021f9d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c110d50 con 0x7f901c0f6710 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f9021f9d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c110e90 con 0x7f901c0f6fe0 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f901c0f6710 0x7f901c110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f901c0f6710 0x7f901c110260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43116/0 (socket says 172.21.15.113:43116) 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f901b7fe700 1 -- 172.21.15.113:0/233000424 learned_addr learned my addr 172.21.15.113:0/233000424 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f901affd700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f901c0f6fe0 0x7f901c110750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f901b7fe700 1 -- 172.21.15.113:0/233000424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f901c0f6fe0 msgr2=0x7f901c110750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f901b7fe700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f901c0f6fe0 0x7f901c110750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f901b7fe700 1 -- 172.21.15.113:0/233000424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9010005040 con 0x7f901c0f6710 2024-04-12T05:14:11.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f901affd700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f901c0f6fe0 0x7f901c110750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:14:11.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f901b7fe700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f901c0f6710 0x7f901c110260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f90100047c0 tx=0x7f9010009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:11.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f9018ff9700 1 -- 172.21.15.113:0/233000424 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f901000e030 con 0x7f901c0f6710 2024-04-12T05:14:11.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f9018ff9700 1 -- 172.21.15.113:0/233000424 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9010007e90 con 0x7f901c0f6710 2024-04-12T05:14:11.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f9021f9d700 1 -- 172.21.15.113:0/233000424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901c10a4c0 con 0x7f901c0f6710 2024-04-12T05:14:11.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.105+0000 7f9018ff9700 1 -- 172.21.15.113:0/233000424 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9010018070 con 0x7f901c0f6710 2024-04-12T05:14:11.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.106+0000 7f9021f9d700 1 -- 172.21.15.113:0/233000424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901c10a9a0 con 0x7f901c0f6710 2024-04-12T05:14:11.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.106+0000 7f9021f9d700 1 -- 172.21.15.113:0/233000424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901c05f590 con 0x7f901c0f6710 2024-04-12T05:14:11.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.107+0000 7f9018ff9700 1 -- 172.21.15.113:0/233000424 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f901001d420 con 0x7f901c0f6710 2024-04-12T05:14:11.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.107+0000 7f9018ff9700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f900406cb60 0x7f900406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:11.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.107+0000 7f9018ff9700 1 -- 172.21.15.113:0/233000424 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f901008d320 con 0x7f901c0f6710 2024-04-12T05:14:11.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.107+0000 7f901affd700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f900406cb60 0x7f900406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:11.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.108+0000 7f901affd700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f900406cb60 0x7f900406f020 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f901c10bba0 tx=0x7f900c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:11.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.112+0000 7f9018ff9700 1 -- 172.21.15.113:0/233000424 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f901005ad10 con 0x7f901c0f6710 2024-04-12T05:14:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:10 smithi113 ceph-mon[103371]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:10 smithi113 ceph-mon[103371]: pgmap v12: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T05:14:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:10 smithi113 ceph-mon[103371]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:10 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/899898484' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:14:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:10 smithi192 ceph-mon[85304]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:10 smithi192 ceph-mon[85304]: pgmap v12: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T05:14:11.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:10 smithi192 ceph-mon[85304]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:11.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:10 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/899898484' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:14:11.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.317+0000 7f9021f9d700 1 -- 172.21.15.113:0/233000424 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f901c10b9f0 con 0x7f900406cb60 2024-04-12T05:14:11.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.318+0000 7f9018ff9700 1 -- 172.21.15.113:0/233000424 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f901c10b9f0 con 0x7f900406cb60 2024-04-12T05:14:11.320 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:14:11.320 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:14:11.320 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:14:11.320 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:14:11.321 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:14:11.321 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr" 2024-04-12T05:14:11.321 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:14:11.321 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "3/21 daemons upgraded", 2024-04-12T05:14:11.321 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mon daemons", 2024-04-12T05:14:11.321 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:14:11.321 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:14:11.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.320+0000 7f9021f9d700 1 -- 172.21.15.113:0/233000424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f900406cb60 msgr2=0x7f900406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:11.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.320+0000 7f9021f9d700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f900406cb60 0x7f900406f020 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f901c10bba0 tx=0x7f900c00b040).stop 2024-04-12T05:14:11.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.320+0000 7f9021f9d700 1 -- 172.21.15.113:0/233000424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f901c0f6710 msgr2=0x7f901c110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:11.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.320+0000 7f9021f9d700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f901c0f6710 0x7f901c110260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f90100047c0 tx=0x7f9010009cb0).stop 2024-04-12T05:14:11.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.320+0000 7f9021f9d700 1 -- 172.21.15.113:0/233000424 shutdown_connections 2024-04-12T05:14:11.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.320+0000 7f9021f9d700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f900406cb60 0x7f900406f020 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.320+0000 7f9021f9d700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f901c0f6710 0x7f901c110260 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.320+0000 7f9021f9d700 1 --2- 172.21.15.113:0/233000424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f901c0f6fe0 0x7f901c110750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.320+0000 7f9021f9d700 1 -- 172.21.15.113:0/233000424 >> 172.21.15.113:0/233000424 conn(0x7f901c070590 msgr2=0x7f901c06d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:11.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.321+0000 7f9021f9d700 1 -- 172.21.15.113:0/233000424 shutdown_connections 2024-04-12T05:14:11.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.321+0000 7f9021f9d700 1 -- 172.21.15.113:0/233000424 wait complete. 2024-04-12T05:14:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.418+0000 7f3d5f650700 1 -- 172.21.15.113:0/3100480019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d580fd150 msgr2=0x7f3d580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.418+0000 7f3d5f650700 1 --2- 172.21.15.113:0/3100480019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d580fd150 0x7f3d580f6930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f3d480047f0 tx=0x7f3d48009cb0).stop 2024-04-12T05:14:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.418+0000 7f3d5f650700 1 -- 172.21.15.113:0/3100480019 shutdown_connections 2024-04-12T05:14:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.418+0000 7f3d5f650700 1 --2- 172.21.15.113:0/3100480019 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d580fd150 0x7f3d580f6930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.418+0000 7f3d5f650700 1 --2- 172.21.15.113:0/3100480019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d580fc7f0 0x7f3d580fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.418+0000 7f3d5f650700 1 -- 172.21.15.113:0/3100480019 >> 172.21.15.113:0/3100480019 conn(0x7f3d58070590 msgr2=0x7f3d580f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.418+0000 7f3d5f650700 1 -- 172.21.15.113:0/3100480019 shutdown_connections 2024-04-12T05:14:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.418+0000 7f3d5f650700 1 -- 172.21.15.113:0/3100480019 wait complete. 2024-04-12T05:14:11.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.418+0000 7f3d5f650700 1 Processor -- start 2024-04-12T05:14:11.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.418+0000 7f3d5f650700 1 -- start start 2024-04-12T05:14:11.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5f650700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d580fc7f0 0x7f3d5810dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:11.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5f650700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d580fd150 0x7f3d5810e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:11.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5f650700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5810eab0 con 0x7f3d580fc7f0 2024-04-12T05:14:11.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5f650700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5810ebf0 con 0x7f3d580fd150 2024-04-12T05:14:11.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5d3ec700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d580fc7f0 0x7f3d5810dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:11.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5d3ec700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d580fc7f0 0x7f3d5810dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43132/0 (socket says 172.21.15.113:43132) 2024-04-12T05:14:11.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5d3ec700 1 -- 172.21.15.113:0/3015280077 learned_addr learned my addr 172.21.15.113:0/3015280077 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5cbeb700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d580fd150 0x7f3d5810e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5cbeb700 1 -- 172.21.15.113:0/3015280077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d580fd150 msgr2=0x7f3d5810e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5cbeb700 1 -- 172.21.15.113:0/3015280077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d580fd150 msgr2=0x7f3d5810e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5cbeb700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d580fd150 0x7f3d5810e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5d3ec700 1 -- 172.21.15.113:0/3015280077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d580fd150 msgr2=0x7f3d5810e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5cbeb700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d580fd150 0x7f3d5810e4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5d3ec700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d580fd150 0x7f3d5810e4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5d3ec700 1 -- 172.21.15.113:0/3015280077 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d48005040 con 0x7f3d580fc7f0 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5d3ec700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d580fc7f0 0x7f3d5810dfc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f3d540027a0 tx=0x7f3d54009f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d4e7fc700 1 -- 172.21.15.113:0/3015280077 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d54007b40 con 0x7f3d580fc7f0 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d4e7fc700 1 -- 172.21.15.113:0/3015280077 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d5400f040 con 0x7f3d580fc7f0 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5f650700 1 -- 172.21.15.113:0/3015280077 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d58108280 con 0x7f3d580fc7f0 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d4e7fc700 1 -- 172.21.15.113:0/3015280077 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d54014910 con 0x7f3d580fc7f0 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.419+0000 7f3d5f650700 1 -- 172.21.15.113:0/3015280077 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d58108840 con 0x7f3d580fc7f0 2024-04-12T05:14:11.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.420+0000 7f3d5f650700 1 -- 172.21.15.113:0/3015280077 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5805f590 con 0x7f3d580fc7f0 2024-04-12T05:14:11.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.421+0000 7f3d4e7fc700 1 -- 172.21.15.113:0/3015280077 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f3d54007ca0 con 0x7f3d580fc7f0 2024-04-12T05:14:11.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.421+0000 7f3d4e7fc700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d4406cbc0 0x7f3d4406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:11.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.421+0000 7f3d4e7fc700 1 -- 172.21.15.113:0/3015280077 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f3d5408b9d0 con 0x7f3d580fc7f0 2024-04-12T05:14:11.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.421+0000 7f3d5cbeb700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d4406cbc0 0x7f3d4406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:11.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.421+0000 7f3d5cbeb700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d4406cbc0 0x7f3d4406f080 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f3d48004f10 tx=0x7f3d4800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:11.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.425+0000 7f3d4e7fc700 1 -- 172.21.15.113:0/3015280077 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3d540593c0 con 0x7f3d580fc7f0 2024-04-12T05:14:11.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.696+0000 7f3d5f650700 1 -- 172.21.15.113:0/3015280077 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3d58047540 con 0x7f3d580fc7f0 2024-04-12T05:14:11.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.696+0000 7f3d4e7fc700 1 -- 172.21.15.113:0/3015280077 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f3d54058f50 con 0x7f3d580fc7f0 2024-04-12T05:14:11.698 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_OK 2024-04-12T05:14:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.698+0000 7f3d5f650700 1 -- 172.21.15.113:0/3015280077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d4406cbc0 msgr2=0x7f3d4406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.698+0000 7f3d5f650700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d4406cbc0 0x7f3d4406f080 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f3d48004f10 tx=0x7f3d4800b040).stop 2024-04-12T05:14:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.698+0000 7f3d5f650700 1 -- 172.21.15.113:0/3015280077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d580fc7f0 msgr2=0x7f3d5810dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.698+0000 7f3d5f650700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d580fc7f0 0x7f3d5810dfc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f3d540027a0 tx=0x7f3d54009f70).stop 2024-04-12T05:14:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.698+0000 7f3d5f650700 1 -- 172.21.15.113:0/3015280077 shutdown_connections 2024-04-12T05:14:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.698+0000 7f3d5f650700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d4406cbc0 0x7f3d4406f080 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.698+0000 7f3d5f650700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d580fc7f0 0x7f3d5810dfc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.698+0000 7f3d5f650700 1 --2- 172.21.15.113:0/3015280077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d580fd150 0x7f3d5810e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.698+0000 7f3d5f650700 1 -- 172.21.15.113:0/3015280077 >> 172.21.15.113:0/3015280077 conn(0x7f3d58070590 msgr2=0x7f3d58060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.698+0000 7f3d5f650700 1 -- 172.21.15.113:0/3015280077 shutdown_connections 2024-04-12T05:14:11.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:11.698+0000 7f3d5f650700 1 -- 172.21.15.113:0/3015280077 wait complete. 2024-04-12T05:14:12.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:11 smithi113 ceph-mon[103371]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:12.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:11 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2235815767' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:14:12.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:11 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:12.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:11 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:12.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:11 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3015280077' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:14:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:11 smithi192 ceph-mon[85304]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:11 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2235815767' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:14:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:11 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:11 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:11 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3015280077' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:14:12.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:12 smithi192 ceph-mon[85304]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:12.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:12 smithi192 ceph-mon[85304]: pgmap v13: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:12.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:12 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:12.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:12 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:13.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:12 smithi113 ceph-mon[103371]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:13.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:12 smithi113 ceph-mon[103371]: pgmap v13: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:13.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:12 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:13.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:12 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:15.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:14 smithi113 ceph-mon[103371]: pgmap v14: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:15.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:14 smithi192 ceph-mon[85304]: pgmap v14: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:17.356 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:16 smithi113 ceph-mon[103371]: pgmap v15: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:17.356 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:16 smithi113 ceph-mon[103371]: Detected new or changed devices on smithi192 2024-04-12T05:14:17.357 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:17.357 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:17.357 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:17.357 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:14:17.357 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:17.357 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:14:17.357 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:14:17.357 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:16 smithi192 ceph-mon[85304]: pgmap v15: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:16 smithi192 ceph-mon[85304]: Detected new or changed devices on smithi192 2024-04-12T05:14:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:14:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:14:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:14:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:18.393 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:17 smithi113 ceph-mon[103371]: Reconfiguring mon.smithi113 (monmap changed)... 2024-04-12T05:14:18.394 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:17 smithi113 ceph-mon[103371]: Reconfiguring daemon mon.smithi113 on smithi113 2024-04-12T05:14:18.394 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:17 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:18.394 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:17 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:18.394 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:17 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.ctsazg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:14:18.394 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:17 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:14:18.394 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:17 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:18.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:17 smithi192 ceph-mon[85304]: Reconfiguring mon.smithi113 (monmap changed)... 2024-04-12T05:14:18.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:17 smithi192 ceph-mon[85304]: Reconfiguring daemon mon.smithi113 on smithi113 2024-04-12T05:14:18.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:17 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:17 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:17 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi113.ctsazg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:14:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:17 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:14:18.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:17 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:19.068 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:19 smithi113 ceph-mon[103371]: Reconfiguring mgr.smithi113.ctsazg (monmap changed)... 2024-04-12T05:14:19.069 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:19 smithi113 ceph-mon[103371]: Reconfiguring daemon mgr.smithi113.ctsazg on smithi113 2024-04-12T05:14:19.069 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:19 smithi113 ceph-mon[103371]: pgmap v16: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:19 smithi192 ceph-mon[85304]: Reconfiguring mgr.smithi113.ctsazg (monmap changed)... 2024-04-12T05:14:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:19 smithi192 ceph-mon[85304]: Reconfiguring daemon mgr.smithi113.ctsazg on smithi113 2024-04-12T05:14:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:19 smithi192 ceph-mon[85304]: pgmap v16: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:20 smithi113 ceph-mon[103371]: Reconfiguring crash.smithi113 (monmap changed)... 2024-04-12T05:14:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:14:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:20 smithi113 ceph-mon[103371]: Reconfiguring daemon crash.smithi113 on smithi113 2024-04-12T05:14:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:20 smithi113 ceph-mon[103371]: pgmap v17: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:20 smithi192 ceph-mon[85304]: Reconfiguring crash.smithi113 (monmap changed)... 2024-04-12T05:14:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:14:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:20 smithi192 ceph-mon[85304]: Reconfiguring daemon crash.smithi113 on smithi113 2024-04-12T05:14:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:20 smithi192 ceph-mon[85304]: pgmap v17: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:21 smithi113 ceph-mon[103371]: Reconfiguring osd.0 (monmap changed)... 2024-04-12T05:14:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:14:22.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:22.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:21 smithi113 ceph-mon[103371]: Reconfiguring daemon osd.0 on smithi113 2024-04-12T05:14:22.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:14:22.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:22.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:22.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:21 smithi192 ceph-mon[85304]: Reconfiguring osd.0 (monmap changed)... 2024-04-12T05:14:22.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:14:22.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:22.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:21 smithi192 ceph-mon[85304]: Reconfiguring daemon osd.0 on smithi113 2024-04-12T05:14:22.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:14:23.140 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:22 smithi113 ceph-mon[103371]: pgmap v18: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:23.140 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:23.140 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:23.140 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:14:23.140 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:22 smithi192 ceph-mon[85304]: pgmap v18: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:14:23.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:23.880 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:23 smithi113 ceph-mon[103371]: Reconfiguring osd.1 (monmap changed)... 2024-04-12T05:14:23.880 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:23 smithi113 ceph-mon[103371]: Reconfiguring daemon osd.1 on smithi113 2024-04-12T05:14:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:23 smithi192 ceph-mon[85304]: Reconfiguring osd.1 (monmap changed)... 2024-04-12T05:14:24.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:23 smithi192 ceph-mon[85304]: Reconfiguring daemon osd.1 on smithi113 2024-04-12T05:14:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:24 smithi113 ceph-mon[103371]: pgmap v19: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:24 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:24 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:24 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:14:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:24 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:24 smithi192 ceph-mon[85304]: pgmap v19: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:24 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:24 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:24 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:14:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:24 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:26.364 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:25 smithi113 ceph-mon[103371]: Reconfiguring osd.2 (monmap changed)... 2024-04-12T05:14:26.365 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:25 smithi113 ceph-mon[103371]: Reconfiguring daemon osd.2 on smithi113 2024-04-12T05:14:26.365 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:25 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:26.365 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:25 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:26.365 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:25 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ovmadh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:14:26.365 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:25 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:26.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:25 smithi192 ceph-mon[85304]: Reconfiguring osd.2 (monmap changed)... 2024-04-12T05:14:26.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:25 smithi192 ceph-mon[85304]: Reconfiguring daemon osd.2 on smithi113 2024-04-12T05:14:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:25 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:25 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:25 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ovmadh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:14:26.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:25 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:27.268 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:26 smithi113 ceph-mon[103371]: pgmap v20: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:27.268 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:26 smithi113 ceph-mon[103371]: Reconfiguring mds.cephfs.smithi113.ovmadh (monmap changed)... 2024-04-12T05:14:27.268 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:26 smithi113 ceph-mon[103371]: Reconfiguring daemon mds.cephfs.smithi113.ovmadh on smithi113 2024-04-12T05:14:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:26 smithi192 ceph-mon[85304]: pgmap v20: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:27.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:26 smithi192 ceph-mon[85304]: Reconfiguring mds.cephfs.smithi113.ovmadh (monmap changed)... 2024-04-12T05:14:27.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:26 smithi192 ceph-mon[85304]: Reconfiguring daemon mds.cephfs.smithi113.ovmadh on smithi113 2024-04-12T05:14:28.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:28 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:28.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:28 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:28.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:28 smithi192 ceph-mon[85304]: Reconfiguring mds.cephfs.smithi113.ydsbei (monmap changed)... 2024-04-12T05:14:28.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:28 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ydsbei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:14:28.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:28 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:28.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:28 smithi192 ceph-mon[85304]: Reconfiguring daemon mds.cephfs.smithi113.ydsbei on smithi113 2024-04-12T05:14:28.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:28 smithi192 ceph-mon[85304]: pgmap v21: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:28.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:28 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:28.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:28 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:28.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:28 smithi113 ceph-mon[103371]: Reconfiguring mds.cephfs.smithi113.ydsbei (monmap changed)... 2024-04-12T05:14:28.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:28 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ydsbei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:14:28.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:28 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:28.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:28 smithi113 ceph-mon[103371]: Reconfiguring daemon mds.cephfs.smithi113.ydsbei on smithi113 2024-04-12T05:14:28.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:28 smithi113 ceph-mon[103371]: pgmap v21: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:30.136 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:29 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:30.136 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:29 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:30.136 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:29 smithi192 ceph-mon[85304]: Reconfiguring crash.smithi192 (monmap changed)... 2024-04-12T05:14:30.136 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:29 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:14:30.137 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:29 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:30.137 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:29 smithi192 ceph-mon[85304]: Reconfiguring daemon crash.smithi192 on smithi192 2024-04-12T05:14:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:29 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:29 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:29 smithi113 ceph-mon[103371]: Reconfiguring crash.smithi192 (monmap changed)... 2024-04-12T05:14:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:29 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:14:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:29 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:29 smithi113 ceph-mon[103371]: Reconfiguring daemon crash.smithi192 on smithi192 2024-04-12T05:14:30.858 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:30 smithi192 ceph-mon[85304]: pgmap v22: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:30.858 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:30.858 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:30.858 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.ocppyo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:14:30.858 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:14:30.858 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:30 smithi113 ceph-mon[103371]: pgmap v22: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.ocppyo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T05:14:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T05:14:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:31.999 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:31 smithi192 ceph-mon[85304]: Reconfiguring mgr.smithi192.ocppyo (monmap changed)... 2024-04-12T05:14:31.999 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:31 smithi192 ceph-mon[85304]: Reconfiguring daemon mgr.smithi192.ocppyo on smithi192 2024-04-12T05:14:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:31 smithi113 ceph-mon[103371]: Reconfiguring mgr.smithi192.ocppyo (monmap changed)... 2024-04-12T05:14:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:31 smithi113 ceph-mon[103371]: Reconfiguring daemon mgr.smithi192.ocppyo on smithi192 2024-04-12T05:14:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:32 smithi113 ceph-mon[103371]: pgmap v23: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:32 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:32 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:32 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:14:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:32 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:14:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:32 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:33.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:32 smithi192 ceph-mon[85304]: pgmap v23: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:33.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:32 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:33.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:32 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:33.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:32 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T05:14:33.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:32 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T05:14:33.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:32 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:34.137 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:33 smithi192 ceph-mon[85304]: Reconfiguring mon.smithi192 (monmap changed)... 2024-04-12T05:14:34.137 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:33 smithi192 ceph-mon[85304]: Reconfiguring daemon mon.smithi192 on smithi192 2024-04-12T05:14:34.137 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:33 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:34.137 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:33 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:34.137 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:33 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:14:34.137 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:33 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:33 smithi113 ceph-mon[103371]: Reconfiguring mon.smithi192 (monmap changed)... 2024-04-12T05:14:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:33 smithi113 ceph-mon[103371]: Reconfiguring daemon mon.smithi192 on smithi192 2024-04-12T05:14:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:33 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:33 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:33 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:14:34.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:33 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:34 smithi113 ceph-mon[103371]: Reconfiguring osd.3 (monmap changed)... 2024-04-12T05:14:35.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:34 smithi113 ceph-mon[103371]: Reconfiguring daemon osd.3 on smithi192 2024-04-12T05:14:35.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:34 smithi113 ceph-mon[103371]: pgmap v24: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:34 smithi192 ceph-mon[85304]: Reconfiguring osd.3 (monmap changed)... 2024-04-12T05:14:35.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:34 smithi192 ceph-mon[85304]: Reconfiguring daemon osd.3 on smithi192 2024-04-12T05:14:35.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:34 smithi192 ceph-mon[85304]: pgmap v24: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:36.778 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:36.779 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:36.779 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:36 smithi192 ceph-mon[85304]: Reconfiguring osd.4 (monmap changed)... 2024-04-12T05:14:36.780 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:14:36.781 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:36.782 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:36 smithi192 ceph-mon[85304]: Reconfiguring daemon osd.4 on smithi192 2024-04-12T05:14:36.783 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:36 smithi192 ceph-mon[85304]: pgmap v25: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:36.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:36.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:36.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:36 smithi113 ceph-mon[103371]: Reconfiguring osd.4 (monmap changed)... 2024-04-12T05:14:36.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:14:36.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:36.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:36 smithi113 ceph-mon[103371]: Reconfiguring daemon osd.4 on smithi192 2024-04-12T05:14:36.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:36 smithi113 ceph-mon[103371]: pgmap v25: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:37.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:14:37.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:37.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:37.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:37 smithi192 ceph-mon[85304]: Reconfiguring osd.5 (monmap changed)... 2024-04-12T05:14:37.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:14:37.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:37.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:37 smithi192 ceph-mon[85304]: Reconfiguring daemon osd.5 on smithi192 2024-04-12T05:14:37.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:14:37.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:37.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:37.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:37 smithi113 ceph-mon[103371]: Reconfiguring osd.5 (monmap changed)... 2024-04-12T05:14:37.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:14:37.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:37.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:37 smithi113 ceph-mon[103371]: Reconfiguring daemon osd.5 on smithi192 2024-04-12T05:14:38.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:38 smithi192 ceph-mon[85304]: pgmap v26: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:38.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:38.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:38.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.ammjap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:14:38.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:38.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:38 smithi113 ceph-mon[103371]: pgmap v26: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:38.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:38.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:38.926 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.ammjap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:14:38.927 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:39.809 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:39 smithi192 ceph-mon[85304]: Reconfiguring mds.cephfs.smithi192.ammjap (monmap changed)... 2024-04-12T05:14:39.809 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:39 smithi192 ceph-mon[85304]: Reconfiguring daemon mds.cephfs.smithi192.ammjap on smithi192 2024-04-12T05:14:39.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:39 smithi113 ceph-mon[103371]: Reconfiguring mds.cephfs.smithi192.ammjap (monmap changed)... 2024-04-12T05:14:39.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:39 smithi113 ceph-mon[103371]: Reconfiguring daemon mds.cephfs.smithi192.ammjap on smithi192 2024-04-12T05:14:41.178 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:40 smithi113 ceph-mon[103371]: pgmap v27: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:41.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:41.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:41.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:40 smithi113 ceph-mon[103371]: Reconfiguring mds.cephfs.smithi192.dtnaoi (monmap changed)... 2024-04-12T05:14:41.181 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.dtnaoi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:14:41.182 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:41.183 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:40 smithi113 ceph-mon[103371]: Reconfiguring daemon mds.cephfs.smithi192.dtnaoi on smithi192 2024-04-12T05:14:41.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:40 smithi192 ceph-mon[85304]: pgmap v27: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:41.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:41.201 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:41.201 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:40 smithi192 ceph-mon[85304]: Reconfiguring mds.cephfs.smithi192.dtnaoi (monmap changed)... 2024-04-12T05:14:41.201 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.dtnaoi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:14:41.201 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:41.202 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:40 smithi192 ceph-mon[85304]: Reconfiguring daemon mds.cephfs.smithi192.dtnaoi on smithi192 2024-04-12T05:14:41.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.801+0000 7f96fa86e700 1 -- 172.21.15.113:0/343545957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96f40fd910 msgr2=0x7f96f40fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:41.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.801+0000 7f96fa86e700 1 --2- 172.21.15.113:0/343545957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96f40fd910 0x7f96f40fdcf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f96e80047f0 tx=0x7f96e8009cb0).stop 2024-04-12T05:14:41.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.801+0000 7f96fa86e700 1 -- 172.21.15.113:0/343545957 shutdown_connections 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.801+0000 7f96fa86e700 1 --2- 172.21.15.113:0/343545957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96f40fe270 0x7f96f40f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.801+0000 7f96fa86e700 1 --2- 172.21.15.113:0/343545957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96f40fd910 0x7f96f40fdcf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.801+0000 7f96fa86e700 1 -- 172.21.15.113:0/343545957 >> 172.21.15.113:0/343545957 conn(0x7f96f40f3530 msgr2=0x7f96f40f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.801+0000 7f96fa86e700 1 -- 172.21.15.113:0/343545957 shutdown_connections 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.801+0000 7f96fa86e700 1 -- 172.21.15.113:0/343545957 wait complete. 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.801+0000 7f96fa86e700 1 Processor -- start 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96fa86e700 1 -- start start 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96fa86e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96f40fd910 0x7f96f410dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96fa86e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96f40fe270 0x7f96f410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96fa86e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f410ead0 con 0x7f96f40fd910 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96fa86e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f410ec10 con 0x7f96f40fe270 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96f3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96f40fd910 0x7f96f410dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:41.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96f3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96f40fd910 0x7f96f410dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47108/0 (socket says 172.21.15.113:47108) 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96f3fff700 1 -- 172.21.15.113:0/3757579744 learned_addr learned my addr 172.21.15.113:0/3757579744 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96f37fe700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96f40fe270 0x7f96f410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96f3fff700 1 -- 172.21.15.113:0/3757579744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96f40fe270 msgr2=0x7f96f410e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96f3fff700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96f40fe270 0x7f96f410e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96f3fff700 1 -- 172.21.15.113:0/3757579744 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e8005040 con 0x7f96f40fd910 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96f37fe700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96f40fe270 0x7f96f410e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.802+0000 7f96f3fff700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96f40fd910 0x7f96f410dfe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f96e8005010 tx=0x7f96e8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.803+0000 7f96f17fa700 1 -- 172.21.15.113:0/3757579744 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96e800e030 con 0x7f96f40fd910 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.803+0000 7f96f17fa700 1 -- 172.21.15.113:0/3757579744 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96e8007e90 con 0x7f96f40fd910 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.803+0000 7f96fa86e700 1 -- 172.21.15.113:0/3757579744 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f4108240 con 0x7f96f40fd910 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.803+0000 7f96f17fa700 1 -- 172.21.15.113:0/3757579744 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96e8018070 con 0x7f96f40fd910 2024-04-12T05:14:41.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.803+0000 7f96fa86e700 1 -- 172.21.15.113:0/3757579744 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f4108720 con 0x7f96f40fd910 2024-04-12T05:14:41.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.804+0000 7f96fa86e700 1 -- 172.21.15.113:0/3757579744 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f405f570 con 0x7f96f40fd910 2024-04-12T05:14:41.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.804+0000 7f96f17fa700 1 -- 172.21.15.113:0/3757579744 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f96e801d6e0 con 0x7f96f40fd910 2024-04-12T05:14:41.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.805+0000 7f96f17fa700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f96dc06cb60 0x7f96dc06f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:41.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.805+0000 7f96f37fe700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f96dc06cb60 0x7f96dc06f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:41.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.805+0000 7f96f17fa700 1 -- 172.21.15.113:0/3757579744 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f96e808d290 con 0x7f96f40fd910 2024-04-12T05:14:41.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.805+0000 7f96f37fe700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f96dc06cb60 0x7f96dc06f020 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f96f4109920 tx=0x7f96e4009de0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:41.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:41.809+0000 7f96f17fa700 1 -- 172.21.15.113:0/3757579744 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f96e805ac80 con 0x7f96f40fd910 2024-04-12T05:14:42.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.031+0000 7f96fa86e700 1 -- 172.21.15.113:0/3757579744 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f96f4109700 con 0x7f96dc06cb60 2024-04-12T05:14:42.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.032+0000 7f96f17fa700 1 -- 172.21.15.113:0/3757579744 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f96f4109700 con 0x7f96dc06cb60 2024-04-12T05:14:42.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.034+0000 7f96fa86e700 1 -- 172.21.15.113:0/3757579744 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f96dc06cb60 msgr2=0x7f96dc06f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.034+0000 7f96fa86e700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f96dc06cb60 0x7f96dc06f020 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f96f4109920 tx=0x7f96e4009de0).stop 2024-04-12T05:14:42.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.034+0000 7f96fa86e700 1 -- 172.21.15.113:0/3757579744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96f40fd910 msgr2=0x7f96f410dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.034+0000 7f96fa86e700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96f40fd910 0x7f96f410dfe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f96e8005010 tx=0x7f96e8009cb0).stop 2024-04-12T05:14:42.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.034+0000 7f96fa86e700 1 -- 172.21.15.113:0/3757579744 shutdown_connections 2024-04-12T05:14:42.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.034+0000 7f96fa86e700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f96dc06cb60 0x7f96dc06f020 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.034+0000 7f96fa86e700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f96f40fd910 0x7f96f410dfe0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.034+0000 7f96fa86e700 1 --2- 172.21.15.113:0/3757579744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96f40fe270 0x7f96f410e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.034+0000 7f96fa86e700 1 -- 172.21.15.113:0/3757579744 >> 172.21.15.113:0/3757579744 conn(0x7f96f40f3530 msgr2=0x7f96f40fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:42.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.034+0000 7f96fa86e700 1 -- 172.21.15.113:0/3757579744 shutdown_connections 2024-04-12T05:14:42.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.034+0000 7f96fa86e700 1 -- 172.21.15.113:0/3757579744 wait complete. 2024-04-12T05:14:42.046 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:14:42.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.137+0000 7fb88940d700 1 -- 172.21.15.113:0/2419951060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8840fd150 msgr2=0x7fb8840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.137+0000 7fb88940d700 1 --2- 172.21.15.113:0/2419951060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8840fd150 0x7fb8840f6930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fb8780047f0 tx=0x7fb878009cb0).stop 2024-04-12T05:14:42.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.137+0000 7fb88940d700 1 -- 172.21.15.113:0/2419951060 shutdown_connections 2024-04-12T05:14:42.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.137+0000 7fb88940d700 1 --2- 172.21.15.113:0/2419951060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8840fd150 0x7fb8840f6930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.137+0000 7fb88940d700 1 --2- 172.21.15.113:0/2419951060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8840fc7f0 0x7fb8840fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.137+0000 7fb88940d700 1 -- 172.21.15.113:0/2419951060 >> 172.21.15.113:0/2419951060 conn(0x7fb884070590 msgr2=0x7fb8840f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.137+0000 7fb88940d700 1 -- 172.21.15.113:0/2419951060 shutdown_connections 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.137+0000 7fb88940d700 1 -- 172.21.15.113:0/2419951060 wait complete. 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.138+0000 7fb88940d700 1 Processor -- start 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.138+0000 7fb88940d700 1 -- start start 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.138+0000 7fb88940d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8840fc7f0 0x7fb88410df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.138+0000 7fb88940d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8840fd150 0x7fb88410e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.138+0000 7fb88940d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88410ea60 con 0x7fb8840fd150 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.138+0000 7fb88940d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88410eba0 con 0x7fb8840fc7f0 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.138+0000 7fb8827fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8840fd150 0x7fb88410e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.139+0000 7fb8827fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8840fd150 0x7fb88410e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47114/0 (socket says 172.21.15.113:47114) 2024-04-12T05:14:42.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.139+0000 7fb8827fc700 1 -- 172.21.15.113:0/2358527453 learned_addr learned my addr 172.21.15.113:0/2358527453 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:42.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.139+0000 7fb882ffd700 1 --2- 172.21.15.113:0/2358527453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8840fc7f0 0x7fb88410df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:42.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.139+0000 7fb8827fc700 1 -- 172.21.15.113:0/2358527453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8840fc7f0 msgr2=0x7fb88410df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.139+0000 7fb8827fc700 1 --2- 172.21.15.113:0/2358527453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8840fc7f0 0x7fb88410df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.139+0000 7fb8827fc700 1 -- 172.21.15.113:0/2358527453 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb878005040 con 0x7fb8840fd150 2024-04-12T05:14:42.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.139+0000 7fb8827fc700 1 --2- 172.21.15.113:0/2358527453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8840fd150 0x7fb88410e460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fb8780099f0 tx=0x7fb878009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:42.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.139+0000 7fb86bfff700 1 -- 172.21.15.113:0/2358527453 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb87800e030 con 0x7fb8840fd150 2024-04-12T05:14:42.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.139+0000 7fb86bfff700 1 -- 172.21.15.113:0/2358527453 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb878007e90 con 0x7fb8840fd150 2024-04-12T05:14:42.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.139+0000 7fb86bfff700 1 -- 172.21.15.113:0/2358527453 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb878018070 con 0x7fb8840fd150 2024-04-12T05:14:42.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.140+0000 7fb88940d700 1 -- 172.21.15.113:0/2358527453 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb884108240 con 0x7fb8840fd150 2024-04-12T05:14:42.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.140+0000 7fb88940d700 1 -- 172.21.15.113:0/2358527453 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb884108720 con 0x7fb8840fd150 2024-04-12T05:14:42.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.141+0000 7fb88940d700 1 -- 172.21.15.113:0/2358527453 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88405f590 con 0x7fb8840fd150 2024-04-12T05:14:42.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.141+0000 7fb86bfff700 1 -- 172.21.15.113:0/2358527453 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7fb87801d470 con 0x7fb8840fd150 2024-04-12T05:14:42.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.141+0000 7fb86bfff700 1 --2- 172.21.15.113:0/2358527453 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb86c06cbc0 0x7fb86c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:42.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.142+0000 7fb86bfff700 1 -- 172.21.15.113:0/2358527453 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fb87808cfe0 con 0x7fb8840fd150 2024-04-12T05:14:42.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.142+0000 7fb882ffd700 1 --2- 172.21.15.113:0/2358527453 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb86c06cbc0 0x7fb86c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:42.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.142+0000 7fb882ffd700 1 --2- 172.21.15.113:0/2358527453 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb86c06cbc0 0x7fb86c06f080 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fb874004d20 tx=0x7fb874009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:42.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.146+0000 7fb86bfff700 1 -- 172.21.15.113:0/2358527453 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb87805a9d0 con 0x7fb8840fd150 2024-04-12T05:14:42.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.360+0000 7fb88940d700 1 -- 172.21.15.113:0/2358527453 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb884108db0 con 0x7fb86c06cbc0 2024-04-12T05:14:42.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.361+0000 7fb86bfff700 1 -- 172.21.15.113:0/2358527453 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7fb884108db0 con 0x7fb86c06cbc0 2024-04-12T05:14:42.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.363+0000 7fb88940d700 1 -- 172.21.15.113:0/2358527453 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb86c06cbc0 msgr2=0x7fb86c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.363+0000 7fb88940d700 1 --2- 172.21.15.113:0/2358527453 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb86c06cbc0 0x7fb86c06f080 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fb874004d20 tx=0x7fb874009040).stop 2024-04-12T05:14:42.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.364+0000 7fb88940d700 1 -- 172.21.15.113:0/2358527453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8840fd150 msgr2=0x7fb88410e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.364+0000 7fb88940d700 1 --2- 172.21.15.113:0/2358527453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8840fd150 0x7fb88410e460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fb8780099f0 tx=0x7fb878009cb0).stop 2024-04-12T05:14:42.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.364+0000 7fb88940d700 1 -- 172.21.15.113:0/2358527453 shutdown_connections 2024-04-12T05:14:42.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.364+0000 7fb88940d700 1 --2- 172.21.15.113:0/2358527453 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb86c06cbc0 0x7fb86c06f080 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.364+0000 7fb88940d700 1 --2- 172.21.15.113:0/2358527453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8840fc7f0 0x7fb88410df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.364+0000 7fb88940d700 1 --2- 172.21.15.113:0/2358527453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8840fd150 0x7fb88410e460 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.364+0000 7fb88940d700 1 -- 172.21.15.113:0/2358527453 >> 172.21.15.113:0/2358527453 conn(0x7fb884070590 msgr2=0x7fb884060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:42.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.364+0000 7fb88940d700 1 -- 172.21.15.113:0/2358527453 shutdown_connections 2024-04-12T05:14:42.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.364+0000 7fb88940d700 1 -- 172.21.15.113:0/2358527453 wait complete. 2024-04-12T05:14:42.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.464+0000 7fef7af08700 1 -- 172.21.15.113:0/1224252715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef740fc7f0 msgr2=0x7fef740fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.464+0000 7fef7af08700 1 --2- 172.21.15.113:0/1224252715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef740fc7f0 0x7fef740fcbd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fef680047f0 tx=0x7fef68009cb0).stop 2024-04-12T05:14:42.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.464+0000 7fef7af08700 1 -- 172.21.15.113:0/1224252715 shutdown_connections 2024-04-12T05:14:42.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.464+0000 7fef7af08700 1 --2- 172.21.15.113:0/1224252715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef740fd150 0x7fef740f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:14:42.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.464+0000 7fef7af08700 1 --2- 172.21.15.113:0/1224252715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef740fc7f0 0x7fef740fcbd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.464+0000 7fef7af08700 1 -- 172.21.15.113:0/1224252715 >> 172.21.15.113:0/1224252715 conn(0x7fef74070590 msgr2=0x7fef740f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:42.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.464+0000 7fef7af08700 1 -- 172.21.15.113:0/1224252715 shutdown_connections 2024-04-12T05:14:42.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.464+0000 7fef7af08700 1 -- 172.21.15.113:0/1224252715 wait complete. 2024-04-12T05:14:42.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef7af08700 1 Processor -- start 2024-04-12T05:14:42.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef7af08700 1 -- start start 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef7af08700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef740fc7f0 0x7fef7410e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef7af08700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef740fd150 0x7fef7410e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef7af08700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7410ebe0 con 0x7fef740fd150 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef7af08700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef74108200 con 0x7fef740fc7f0 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef73fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef740fd150 0x7fef7410e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef73fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef740fd150 0x7fef7410e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47128/0 (socket says 172.21.15.113:47128) 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef73fff700 1 -- 172.21.15.113:0/1261098351 learned_addr learned my addr 172.21.15.113:0/1261098351 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef78ca4700 1 --2- 172.21.15.113:0/1261098351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef740fc7f0 0x7fef7410e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef73fff700 1 -- 172.21.15.113:0/1261098351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef740fc7f0 msgr2=0x7fef7410e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef73fff700 1 --2- 172.21.15.113:0/1261098351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef740fc7f0 0x7fef7410e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef73fff700 1 -- 172.21.15.113:0/1261098351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef68005040 con 0x7fef740fd150 2024-04-12T05:14:42.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef73fff700 1 --2- 172.21.15.113:0/1261098351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef740fd150 0x7fef7410e550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fef6000b480 tx=0x7fef600090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:42.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef71ffb700 1 -- 172.21.15.113:0/1261098351 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef60002500 con 0x7fef740fd150 2024-04-12T05:14:42.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef71ffb700 1 -- 172.21.15.113:0/1261098351 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef60002b10 con 0x7fef740fd150 2024-04-12T05:14:42.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.465+0000 7fef7af08700 1 -- 172.21.15.113:0/1261098351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef741084b0 con 0x7fef740fd150 2024-04-12T05:14:42.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.466+0000 7fef71ffb700 1 -- 172.21.15.113:0/1261098351 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef60012c90 con 0x7fef740fd150 2024-04-12T05:14:42.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.466+0000 7fef7af08700 1 -- 172.21.15.113:0/1261098351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef741089f0 con 0x7fef740fd150 2024-04-12T05:14:42.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.467+0000 7fef7af08700 1 -- 172.21.15.113:0/1261098351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7405f590 con 0x7fef740fd150 2024-04-12T05:14:42.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.467+0000 7fef71ffb700 1 -- 172.21.15.113:0/1261098351 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7fef60002660 con 0x7fef740fd150 2024-04-12T05:14:42.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.467+0000 7fef71ffb700 1 --2- 172.21.15.113:0/1261098351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef6406cbc0 0x7fef6406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:42.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.468+0000 7fef78ca4700 1 --2- 172.21.15.113:0/1261098351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef6406cbc0 0x7fef6406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:42.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.468+0000 7fef71ffb700 1 -- 172.21.15.113:0/1261098351 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fef6008be60 con 0x7fef740fd150 2024-04-12T05:14:42.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.468+0000 7fef78ca4700 1 --2- 172.21.15.113:0/1261098351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef6406cbc0 0x7fef6406f080 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fef68004f10 tx=0x7fef6800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:42.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.472+0000 7fef71ffb700 1 -- 172.21.15.113:0/1261098351 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fef60059960 con 0x7fef740fd150 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all mon 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi113"}]: dispatch 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi113"}]': finished 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi192"}]: dispatch 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi192"}]': finished 2024-04-12T05:14:42.508 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:42 smithi113 ceph-mon[103371]: pgmap v28: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:42.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.673+0000 7fef7af08700 1 -- 172.21.15.113:0/1261098351 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fef741096e0 con 0x7fef6406cbc0 2024-04-12T05:14:42.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.683+0000 7fef71ffb700 1 -- 172.21.15.113:0/1261098351 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fef741096e0 con 0x7fef6406cbc0 2024-04-12T05:14:42.685 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:14:42.686 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (2m) 45s ago 13m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:14:42.686 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (13m) 45s ago 13m 6639k - 16.2.15-1-g8963e9de 727f386679cb d13346ffdadf 2024-04-12T05:14:42.686 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (13m) 31s ago 13m 6639k - 16.2.15-1-g8963e9de 727f386679cb 730c0b2ae147 2024-04-12T05:14:42.686 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (2m) 45s ago 13m 84.0M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (8m) 45s ago 8m 121M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (8m) 45s ago 8m 17.6M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (8m) 31s ago 8m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (8m) 31s ago 8m 15.2M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (3m) 45s ago 15m 468M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (3m) 31s ago 13m 447M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (61s) 45s ago 15m 27.2M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (35s) 31s ago 13m 23.4M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (3m) 45s ago 13m 16.3M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (3m) 31s ago 12m 17.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (12m) 45s ago 12m 227M 1910M 16.2.15-1-g8963e9de 727f386679cb 3eaa70d0bdaa 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (11m) 45s ago 11m 239M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:14:42.692 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (11m) 45s ago 11m 202M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:14:42.693 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (10m) 31s ago 10m 264M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:14:42.693 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (10m) 31s ago 10m 232M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:14:42.693 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (9m) 31s ago 9m 226M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:14:42.693 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (2m) 31s ago 12m 57.4M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:14:42.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.685+0000 7fef7af08700 1 -- 172.21.15.113:0/1261098351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef6406cbc0 msgr2=0x7fef6406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.685+0000 7fef7af08700 1 --2- 172.21.15.113:0/1261098351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef6406cbc0 0x7fef6406f080 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fef68004f10 tx=0x7fef6800b040).stop 2024-04-12T05:14:42.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.685+0000 7fef7af08700 1 -- 172.21.15.113:0/1261098351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef740fd150 msgr2=0x7fef7410e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.685+0000 7fef7af08700 1 --2- 172.21.15.113:0/1261098351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef740fd150 0x7fef7410e550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fef6000b480 tx=0x7fef600090f0).stop 2024-04-12T05:14:42.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.685+0000 7fef7af08700 1 -- 172.21.15.113:0/1261098351 shutdown_connections 2024-04-12T05:14:42.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.685+0000 7fef7af08700 1 --2- 172.21.15.113:0/1261098351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef6406cbc0 0x7fef6406f080 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.685+0000 7fef7af08700 1 --2- 172.21.15.113:0/1261098351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef740fc7f0 0x7fef7410e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.685+0000 7fef7af08700 1 --2- 172.21.15.113:0/1261098351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef740fd150 0x7fef7410e550 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.685+0000 7fef7af08700 1 -- 172.21.15.113:0/1261098351 >> 172.21.15.113:0/1261098351 conn(0x7fef74070590 msgr2=0x7fef74060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:42.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.685+0000 7fef7af08700 1 -- 172.21.15.113:0/1261098351 shutdown_connections 2024-04-12T05:14:42.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.685+0000 7fef7af08700 1 -- 172.21.15.113:0/1261098351 wait complete. 2024-04-12T05:14:42.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:42.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:42.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:14:42.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:14:42.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:14:42.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all mon 2024-04-12T05:14:42.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:42.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi113"}]: dispatch 2024-04-12T05:14:42.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi113"}]': finished 2024-04-12T05:14:42.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi192"}]: dispatch 2024-04-12T05:14:42.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi192"}]': finished 2024-04-12T05:14:42.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:42 smithi192 ceph-mon[85304]: pgmap v28: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:42.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.782+0000 7ff342eef700 1 -- 172.21.15.113:0/431455638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c0fc7f0 msgr2=0x7ff33c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.782+0000 7ff342eef700 1 --2- 172.21.15.113:0/431455638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c0fc7f0 0x7ff33c0fcbd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7ff3300047f0 tx=0x7ff330009cb0).stop 2024-04-12T05:14:42.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.782+0000 7ff342eef700 1 -- 172.21.15.113:0/431455638 shutdown_connections 2024-04-12T05:14:42.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.782+0000 7ff342eef700 1 --2- 172.21.15.113:0/431455638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff33c0fd150 0x7ff33c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:14:42.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.782+0000 7ff342eef700 1 --2- 172.21.15.113:0/431455638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c0fc7f0 0x7ff33c0fcbd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.782+0000 7ff342eef700 1 -- 172.21.15.113:0/431455638 >> 172.21.15.113:0/431455638 conn(0x7ff33c070590 msgr2=0x7ff33c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.782+0000 7ff342eef700 1 -- 172.21.15.113:0/431455638 shutdown_connections 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.782+0000 7ff342eef700 1 -- 172.21.15.113:0/431455638 wait complete. 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff342eef700 1 Processor -- start 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff342eef700 1 -- start start 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff342eef700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c0fc7f0 0x7ff33c10e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff342eef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff33c0fd150 0x7ff33c10e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff342eef700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c10ebe0 con 0x7ff33c0fc7f0 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff342eef700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c108200 con 0x7ff33c0fd150 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff340c8b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c0fc7f0 0x7ff33c10e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff340c8b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c0fc7f0 0x7ff33c10e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47148/0 (socket says 172.21.15.113:47148) 2024-04-12T05:14:42.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff340c8b700 1 -- 172.21.15.113:0/3978520512 learned_addr learned my addr 172.21.15.113:0/3978520512 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:42.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff33bfff700 1 --2- 172.21.15.113:0/3978520512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff33c0fd150 0x7ff33c10e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:42.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff340c8b700 1 -- 172.21.15.113:0/3978520512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff33c0fd150 msgr2=0x7ff33c10e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:42.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff340c8b700 1 --2- 172.21.15.113:0/3978520512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff33c0fd150 0x7ff33c10e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:42.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff340c8b700 1 -- 172.21.15.113:0/3978520512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff330005040 con 0x7ff33c0fc7f0 2024-04-12T05:14:42.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.783+0000 7ff340c8b700 1 --2- 172.21.15.113:0/3978520512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c0fc7f0 0x7ff33c10e060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7ff33000b120 tx=0x7ff330009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:42.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.784+0000 7ff339ffb700 1 -- 172.21.15.113:0/3978520512 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff33000e030 con 0x7ff33c0fc7f0 2024-04-12T05:14:42.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.784+0000 7ff342eef700 1 -- 172.21.15.113:0/3978520512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33c108450 con 0x7ff33c0fc7f0 2024-04-12T05:14:42.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.784+0000 7ff339ffb700 1 -- 172.21.15.113:0/3978520512 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff330007e90 con 0x7ff33c0fc7f0 2024-04-12T05:14:42.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.784+0000 7ff339ffb700 1 -- 172.21.15.113:0/3978520512 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff330018070 con 0x7ff33c0fc7f0 2024-04-12T05:14:42.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.784+0000 7ff342eef700 1 -- 172.21.15.113:0/3978520512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33c108930 con 0x7ff33c0fc7f0 2024-04-12T05:14:42.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.785+0000 7ff342eef700 1 -- 172.21.15.113:0/3978520512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33c05f590 con 0x7ff33c0fc7f0 2024-04-12T05:14:42.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.785+0000 7ff339ffb700 1 -- 172.21.15.113:0/3978520512 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7ff33001d420 con 0x7ff33c0fc7f0 2024-04-12T05:14:42.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.786+0000 7ff339ffb700 1 --2- 172.21.15.113:0/3978520512 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff32c06cbc0 0x7ff32c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:42.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.786+0000 7ff339ffb700 1 -- 172.21.15.113:0/3978520512 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7ff33008d440 con 0x7ff33c0fc7f0 2024-04-12T05:14:42.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.786+0000 7ff33bfff700 1 --2- 172.21.15.113:0/3978520512 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff32c06cbc0 0x7ff32c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:42.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.786+0000 7ff33bfff700 1 --2- 172.21.15.113:0/3978520512 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff32c06cbc0 0x7ff32c06f080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ff33c1099a0 tx=0x7ff328008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:42.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:42.790+0000 7ff339ffb700 1 -- 172.21.15.113:0/3978520512 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff33005ae30 con 0x7ff33c0fc7f0 2024-04-12T05:14:43.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.072+0000 7ff342eef700 1 -- 172.21.15.113:0/3978520512 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff33c109030 con 0x7ff33c0fc7f0 2024-04-12T05:14:43.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.072+0000 7ff339ffb700 1 -- 172.21.15.113:0/3978520512 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+742 (secure 0 0 0) 0x7ff33005a9c0 con 0x7ff33c0fc7f0 2024-04-12T05:14:43.075 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:14:43.075 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:14:43.075 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:14:43.075 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:14:43.075 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:14:43.075 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:14:43.075 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:14:43.075 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:14:43.075 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6 2024-04-12T05:14:43.075 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:14:43.076 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:14:43.076 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:14:43.076 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:14:43.076 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:14:43.076 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 10, 2024-04-12T05:14:43.076 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:14:43.076 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:14:43.076 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:14:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.074+0000 7ff342eef700 1 -- 172.21.15.113:0/3978520512 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff32c06cbc0 msgr2=0x7ff32c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.074+0000 7ff342eef700 1 --2- 172.21.15.113:0/3978520512 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff32c06cbc0 0x7ff32c06f080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ff33c1099a0 tx=0x7ff328008040).stop 2024-04-12T05:14:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.074+0000 7ff342eef700 1 -- 172.21.15.113:0/3978520512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c0fc7f0 msgr2=0x7ff33c10e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.074+0000 7ff342eef700 1 --2- 172.21.15.113:0/3978520512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c0fc7f0 0x7ff33c10e060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7ff33000b120 tx=0x7ff330009cb0).stop 2024-04-12T05:14:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.074+0000 7ff342eef700 1 -- 172.21.15.113:0/3978520512 shutdown_connections 2024-04-12T05:14:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.074+0000 7ff342eef700 1 --2- 172.21.15.113:0/3978520512 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff32c06cbc0 0x7ff32c06f080 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.074+0000 7ff342eef700 1 --2- 172.21.15.113:0/3978520512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff33c0fc7f0 0x7ff33c10e060 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.074+0000 7ff342eef700 1 --2- 172.21.15.113:0/3978520512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff33c0fd150 0x7ff33c10e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.074+0000 7ff342eef700 1 -- 172.21.15.113:0/3978520512 >> 172.21.15.113:0/3978520512 conn(0x7ff33c070590 msgr2=0x7ff33c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.074+0000 7ff342eef700 1 -- 172.21.15.113:0/3978520512 shutdown_connections 2024-04-12T05:14:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.074+0000 7ff342eef700 1 -- 172.21.15.113:0/3978520512 wait complete. 2024-04-12T05:14:43.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.176+0000 7f7b745cd700 1 -- 172.21.15.113:0/2350539404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b6c0f8920 msgr2=0x7f7b6c0f8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.176+0000 7f7b745cd700 1 --2- 172.21.15.113:0/2350539404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b6c0f8920 0x7f7b6c0f8d00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f7b680047f0 tx=0x7f7b68009cb0).stop 2024-04-12T05:14:43.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.176+0000 7f7b745cd700 1 -- 172.21.15.113:0/2350539404 shutdown_connections 2024-04-12T05:14:43.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.176+0000 7f7b745cd700 1 --2- 172.21.15.113:0/2350539404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6c0f91f0 0x7f7b6c101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.176+0000 7f7b745cd700 1 --2- 172.21.15.113:0/2350539404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b6c0f8920 0x7f7b6c0f8d00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.176+0000 7f7b745cd700 1 -- 172.21.15.113:0/2350539404 >> 172.21.15.113:0/2350539404 conn(0x7f7b6c070590 msgr2=0x7f7b6c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:43.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.176+0000 7f7b745cd700 1 -- 172.21.15.113:0/2350539404 shutdown_connections 2024-04-12T05:14:43.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.177+0000 7f7b745cd700 1 -- 172.21.15.113:0/2350539404 wait complete. 2024-04-12T05:14:43.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.177+0000 7f7b745cd700 1 Processor -- start 2024-04-12T05:14:43.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.177+0000 7f7b745cd700 1 -- start start 2024-04-12T05:14:43.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b745cd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b6c0f8920 0x7f7b6c10e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:43.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b745cd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6c0f91f0 0x7f7b6c10e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:43.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b745cd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c10eb40 con 0x7f7b6c0f8920 2024-04-12T05:14:43.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b745cd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c10ec80 con 0x7f7b6c0f91f0 2024-04-12T05:14:43.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b72369700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b6c0f8920 0x7f7b6c10e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:43.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b72369700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b6c0f8920 0x7f7b6c10e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47156/0 (socket says 172.21.15.113:47156) 2024-04-12T05:14:43.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b72369700 1 -- 172.21.15.113:0/3059574520 learned_addr learned my addr 172.21.15.113:0/3059574520 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:43.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b71b68700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6c0f91f0 0x7f7b6c10e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:43.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b72369700 1 -- 172.21.15.113:0/3059574520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6c0f91f0 msgr2=0x7f7b6c10e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b72369700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6c0f91f0 0x7f7b6c10e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b72369700 1 -- 172.21.15.113:0/3059574520 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b68005040 con 0x7f7b6c0f8920 2024-04-12T05:14:43.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b71b68700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6c0f91f0 0x7f7b6c10e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:14:43.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b72369700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b6c0f8920 0x7f7b6c10e050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f7b68005010 tx=0x7f7b68009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:43.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b637fe700 1 -- 172.21.15.113:0/3059574520 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b6800e030 con 0x7f7b6c0f8920 2024-04-12T05:14:43.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b637fe700 1 -- 172.21.15.113:0/3059574520 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b68007e90 con 0x7f7b6c0f8920 2024-04-12T05:14:43.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b637fe700 1 -- 172.21.15.113:0/3059574520 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b68018070 con 0x7f7b6c0f8920 2024-04-12T05:14:43.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.178+0000 7f7b745cd700 1 -- 172.21.15.113:0/3059574520 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c1082b0 con 0x7f7b6c0f8920 2024-04-12T05:14:43.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.179+0000 7f7b745cd700 1 -- 172.21.15.113:0/3059574520 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c108790 con 0x7f7b6c0f8920 2024-04-12T05:14:43.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.179+0000 7f7b745cd700 1 -- 172.21.15.113:0/3059574520 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c05f590 con 0x7f7b6c0f8920 2024-04-12T05:14:43.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.180+0000 7f7b637fe700 1 -- 172.21.15.113:0/3059574520 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f7b6801d6e0 con 0x7f7b6c0f8920 2024-04-12T05:14:43.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.181+0000 7f7b637fe700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b5806cb60 0x7f7b5806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:43.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.181+0000 7f7b637fe700 1 -- 172.21.15.113:0/3059574520 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f7b6808d290 con 0x7f7b6c0f8920 2024-04-12T05:14:43.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.181+0000 7f7b71b68700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b5806cb60 0x7f7b5806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:43.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.181+0000 7f7b71b68700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b5806cb60 0x7f7b5806f020 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f7b6c109990 tx=0x7f7b5c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:43.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.188+0000 7f7b637fe700 1 -- 172.21.15.113:0/3059574520 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7b6805ac80 con 0x7f7b6c0f8920 2024-04-12T05:14:43.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.432+0000 7f7b745cd700 1 -- 172.21.15.113:0/3059574520 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7b6c109770 con 0x7f7b6c0f8920 2024-04-12T05:14:43.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.432+0000 7f7b637fe700 1 -- 172.21.15.113:0/3059574520 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1859 (secure 0 0 0) 0x7f7b6805a810 con 0x7f7b6c0f8920 2024-04-12T05:14:43.436 INFO:teuthology.orchestra.run.smithi113.stdout:e18 2024-04-12T05:14:43.436 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:14:43.436 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:14:43.437 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:14:43.437 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:14:43.437 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:14:43.437 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:14:43.437 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 18 2024-04-12T05:14:43.437 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:14:43.437 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:14:43.438 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:14:43.438 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:14:43.438 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:14:43.438 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:14:43.439 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:14:43.439 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:14:43.439 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:14:43.439 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:14:43.440 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:14:43.440 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:14:43.440 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:14:43.440 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:14:43.440 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:14:43.440 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=14426,1=24261} 2024-04-12T05:14:43.440 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:14:43.440 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:14:43.440 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:43.441 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.434+0000 7f7b745cd700 1 -- 172.21.15.113:0/3059574520 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b5806cb60 msgr2=0x7f7b5806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.434+0000 7f7b745cd700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b5806cb60 0x7f7b5806f020 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f7b6c109990 tx=0x7f7b5c00b040).stop 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.434+0000 7f7b745cd700 1 -- 172.21.15.113:0/3059574520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b6c0f8920 msgr2=0x7f7b6c10e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.434+0000 7f7b745cd700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b6c0f8920 0x7f7b6c10e050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f7b68005010 tx=0x7f7b68009cb0).stop 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.435+0000 7f7b745cd700 1 -- 172.21.15.113:0/3059574520 shutdown_connections 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.435+0000 7f7b745cd700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b5806cb60 0x7f7b5806f020 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.435+0000 7f7b745cd700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b6c0f8920 0x7f7b6c10e050 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.435+0000 7f7b745cd700 1 --2- 172.21.15.113:0/3059574520 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b6c0f91f0 0x7f7b6c10e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.435+0000 7f7b745cd700 1 -- 172.21.15.113:0/3059574520 >> 172.21.15.113:0/3059574520 conn(0x7f7b6c070590 msgr2=0x7f7b6c0fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.435+0000 7f7b745cd700 1 -- 172.21.15.113:0/3059574520 shutdown_connections 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.435+0000 7f7b745cd700 1 -- 172.21.15.113:0/3059574520 wait complete. 2024-04-12T05:14:43.443 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 18 2024-04-12T05:14:43.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.539+0000 7fb385823700 1 -- 172.21.15.113:0/151396931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3800fd8b0 msgr2=0x7fb3800fdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.539+0000 7fb385823700 1 --2- 172.21.15.113:0/151396931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3800fd8b0 0x7fb3800fdc90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fb3700047f0 tx=0x7fb370009cb0).stop 2024-04-12T05:14:43.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.539+0000 7fb385823700 1 -- 172.21.15.113:0/151396931 shutdown_connections 2024-04-12T05:14:43.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.539+0000 7fb385823700 1 --2- 172.21.15.113:0/151396931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3800fe210 0x7fb3800f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:14:43.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.539+0000 7fb385823700 1 --2- 172.21.15.113:0/151396931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3800fd8b0 0x7fb3800fdc90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.539+0000 7fb385823700 1 -- 172.21.15.113:0/151396931 >> 172.21.15.113:0/151396931 conn(0x7fb38006ce40 msgr2=0x7fb38006d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:43.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.539+0000 7fb385823700 1 -- 172.21.15.113:0/151396931 shutdown_connections 2024-04-12T05:14:43.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.539+0000 7fb385823700 1 -- 172.21.15.113:0/151396931 wait complete. 2024-04-12T05:14:43.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.539+0000 7fb385823700 1 Processor -- start 2024-04-12T05:14:43.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb385823700 1 -- start start 2024-04-12T05:14:43.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb385823700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3800fd8b0 0x7fb38010e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:43.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb385823700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3800fe210 0x7fb38010e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:43.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb385823700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38010ec40 con 0x7fb3800fd8b0 2024-04-12T05:14:43.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb385823700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb380108260 con 0x7fb3800fe210 2024-04-12T05:14:43.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb37effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3800fd8b0 0x7fb38010e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:43.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb37effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3800fd8b0 0x7fb38010e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47168/0 (socket says 172.21.15.113:47168) 2024-04-12T05:14:43.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb37effd700 1 -- 172.21.15.113:0/154594969 learned_addr learned my addr 172.21.15.113:0/154594969 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:43.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb37e7fc700 1 --2- 172.21.15.113:0/154594969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3800fe210 0x7fb38010e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:43.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb37effd700 1 -- 172.21.15.113:0/154594969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3800fe210 msgr2=0x7fb38010e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb37effd700 1 --2- 172.21.15.113:0/154594969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3800fe210 0x7fb38010e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb37effd700 1 -- 172.21.15.113:0/154594969 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb370005040 con 0x7fb3800fd8b0 2024-04-12T05:14:43.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb37effd700 1 --2- 172.21.15.113:0/154594969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3800fd8b0 0x7fb38010e0c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb37000b120 tx=0x7fb370009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:43.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.540+0000 7fb367fff700 1 -- 172.21.15.113:0/154594969 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb37000e030 con 0x7fb3800fd8b0 2024-04-12T05:14:43.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.541+0000 7fb385823700 1 -- 172.21.15.113:0/154594969 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3801084b0 con 0x7fb3800fd8b0 2024-04-12T05:14:43.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.541+0000 7fb367fff700 1 -- 172.21.15.113:0/154594969 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb370007e90 con 0x7fb3800fd8b0 2024-04-12T05:14:43.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.541+0000 7fb367fff700 1 -- 172.21.15.113:0/154594969 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb370018070 con 0x7fb3800fd8b0 2024-04-12T05:14:43.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.541+0000 7fb385823700 1 -- 172.21.15.113:0/154594969 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb380108990 con 0x7fb3800fd8b0 2024-04-12T05:14:43.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.542+0000 7fb385823700 1 -- 172.21.15.113:0/154594969 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb380060670 con 0x7fb3800fd8b0 2024-04-12T05:14:43.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.542+0000 7fb367fff700 1 -- 172.21.15.113:0/154594969 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7fb37001d420 con 0x7fb3800fd8b0 2024-04-12T05:14:43.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.543+0000 7fb367fff700 1 --2- 172.21.15.113:0/154594969 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb36806cbc0 0x7fb36806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:43.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.543+0000 7fb37e7fc700 1 --2- 172.21.15.113:0/154594969 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb36806cbc0 0x7fb36806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:43.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.543+0000 7fb367fff700 1 -- 172.21.15.113:0/154594969 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fb37008d2e0 con 0x7fb3800fd8b0 2024-04-12T05:14:43.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.543+0000 7fb37e7fc700 1 --2- 172.21.15.113:0/154594969 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb36806cbc0 0x7fb36806f080 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb380109a00 tx=0x7fb374009060).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:43.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.547+0000 7fb367fff700 1 -- 172.21.15.113:0/154594969 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb37005acd0 con 0x7fb3800fd8b0 2024-04-12T05:14:43.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.758+0000 7fb385823700 1 -- 172.21.15.113:0/154594969 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3801097b0 con 0x7fb36806cbc0 2024-04-12T05:14:43.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.759+0000 7fb367fff700 1 -- 172.21.15.113:0/154594969 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7fb3801097b0 con 0x7fb36806cbc0 2024-04-12T05:14:43.761 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:14:43.762 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:14:43.762 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:14:43.762 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:14:43.762 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:14:43.762 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:14:43.762 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr" 2024-04-12T05:14:43.762 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:14:43.762 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "4/21 daemons upgraded", 2024-04-12T05:14:43.762 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading crash daemons", 2024-04-12T05:14:43.762 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:14:43.762 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:14:43.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.762+0000 7fb385823700 1 -- 172.21.15.113:0/154594969 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb36806cbc0 msgr2=0x7fb36806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.762+0000 7fb385823700 1 --2- 172.21.15.113:0/154594969 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb36806cbc0 0x7fb36806f080 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb380109a00 tx=0x7fb374009060).stop 2024-04-12T05:14:43.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.762+0000 7fb385823700 1 -- 172.21.15.113:0/154594969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3800fd8b0 msgr2=0x7fb38010e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.762+0000 7fb385823700 1 --2- 172.21.15.113:0/154594969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3800fd8b0 0x7fb38010e0c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb37000b120 tx=0x7fb370009cb0).stop 2024-04-12T05:14:43.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.762+0000 7fb385823700 1 -- 172.21.15.113:0/154594969 shutdown_connections 2024-04-12T05:14:43.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.762+0000 7fb385823700 1 --2- 172.21.15.113:0/154594969 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb36806cbc0 0x7fb36806f080 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.762+0000 7fb385823700 1 --2- 172.21.15.113:0/154594969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3800fd8b0 0x7fb38010e0c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.762+0000 7fb385823700 1 --2- 172.21.15.113:0/154594969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3800fe210 0x7fb38010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.762+0000 7fb385823700 1 -- 172.21.15.113:0/154594969 >> 172.21.15.113:0/154594969 conn(0x7fb38006ce40 msgr2=0x7fb380100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:43.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.762+0000 7fb385823700 1 -- 172.21.15.113:0/154594969 shutdown_connections 2024-04-12T05:14:43.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.762+0000 7fb385823700 1 -- 172.21.15.113:0/154594969 wait complete. 2024-04-12T05:14:43.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.861+0000 7f4bddc85700 1 -- 172.21.15.113:0/4004157797 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd80fd8f0 msgr2=0x7f4bd80fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.861+0000 7f4bddc85700 1 --2- 172.21.15.113:0/4004157797 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd80fd8f0 0x7f4bd80fdcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4bc00047f0 tx=0x7f4bc0009cb0).stop 2024-04-12T05:14:43.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.861+0000 7f4bddc85700 1 -- 172.21.15.113:0/4004157797 shutdown_connections 2024-04-12T05:14:43.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.861+0000 7f4bddc85700 1 --2- 172.21.15.113:0/4004157797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4bd80fe250 0x7f4bd80f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.861+0000 7f4bddc85700 1 --2- 172.21.15.113:0/4004157797 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd80fd8f0 0x7f4bd80fdcd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.861+0000 7f4bddc85700 1 -- 172.21.15.113:0/4004157797 >> 172.21.15.113:0/4004157797 conn(0x7f4bd80f34b0 msgr2=0x7f4bd80f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:43.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.861+0000 7f4bddc85700 1 -- 172.21.15.113:0/4004157797 shutdown_connections 2024-04-12T05:14:43.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.861+0000 7f4bddc85700 1 -- 172.21.15.113:0/4004157797 wait complete. 2024-04-12T05:14:43.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.862+0000 7f4bddc85700 1 Processor -- start 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.862+0000 7f4bddc85700 1 -- start start 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.862+0000 7f4bddc85700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4bd80fd8f0 0x7f4bd810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.862+0000 7f4bddc85700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd80fe250 0x7f4bd810e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.862+0000 7f4bddc85700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd810ead0 con 0x7f4bd80fe250 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.862+0000 7f4bddc85700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd810ec10 con 0x7f4bd80fd8f0 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.862+0000 7f4bd6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd80fe250 0x7f4bd810e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.862+0000 7f4bd6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd80fe250 0x7f4bd810e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47184/0 (socket says 172.21.15.113:47184) 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.862+0000 7f4bd6ffd700 1 -- 172.21.15.113:0/2882814506 learned_addr learned my addr 172.21.15.113:0/2882814506 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.862+0000 7f4bd77fe700 1 --2- 172.21.15.113:0/2882814506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4bd80fd8f0 0x7f4bd810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.863+0000 7f4bd6ffd700 1 -- 172.21.15.113:0/2882814506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4bd80fd8f0 msgr2=0x7f4bd810dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.863+0000 7f4bd6ffd700 1 --2- 172.21.15.113:0/2882814506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4bd80fd8f0 0x7f4bd810dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:43.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.863+0000 7f4bd6ffd700 1 -- 172.21.15.113:0/2882814506 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc0005040 con 0x7f4bd80fe250 2024-04-12T05:14:43.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.863+0000 7f4bd6ffd700 1 --2- 172.21.15.113:0/2882814506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd80fe250 0x7f4bd810e4d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4bc8002780 tx=0x7f4bc8009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:43.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.863+0000 7f4bd4ff9700 1 -- 172.21.15.113:0/2882814506 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bc8007b40 con 0x7f4bd80fe250 2024-04-12T05:14:43.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.863+0000 7f4bd4ff9700 1 -- 172.21.15.113:0/2882814506 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bc800f040 con 0x7f4bd80fe250 2024-04-12T05:14:43.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.863+0000 7f4bddc85700 1 -- 172.21.15.113:0/2882814506 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd81082a0 con 0x7f4bd80fe250 2024-04-12T05:14:43.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.863+0000 7f4bd4ff9700 1 -- 172.21.15.113:0/2882814506 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bc8014780 con 0x7f4bd80fe250 2024-04-12T05:14:43.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.863+0000 7f4bddc85700 1 -- 172.21.15.113:0/2882814506 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd8108860 con 0x7f4bd80fe250 2024-04-12T05:14:43.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.864+0000 7f4bddc85700 1 -- 172.21.15.113:0/2882814506 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd805f590 con 0x7f4bd80fe250 2024-04-12T05:14:43.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.864+0000 7f4bd4ff9700 1 -- 172.21.15.113:0/2882814506 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f4bc8007ca0 con 0x7f4bd80fe250 2024-04-12T05:14:43.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.865+0000 7f4bd4ff9700 1 --2- 172.21.15.113:0/2882814506 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4bc406cbc0 0x7f4bc406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:14:43.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.865+0000 7f4bd4ff9700 1 -- 172.21.15.113:0/2882814506 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f4bc808bef0 con 0x7f4bd80fe250 2024-04-12T05:14:43.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.865+0000 7f4bd77fe700 1 --2- 172.21.15.113:0/2882814506 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4bc406cbc0 0x7f4bc406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:14:43.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.865+0000 7f4bd77fe700 1 --2- 172.21.15.113:0/2882814506 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4bc406cbc0 0x7f4bc406f080 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f4bc0004f10 tx=0x7f4bc000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:14:43.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:43.869+0000 7f4bd4ff9700 1 -- 172.21.15.113:0/2882814506 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4bc8059860 con 0x7f4bd80fe250 2024-04-12T05:14:44.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.141+0000 7f4bddc85700 1 -- 172.21.15.113:0/2882814506 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4bd8047540 con 0x7f4bd80fe250 2024-04-12T05:14:44.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.141+0000 7f4bd4ff9700 1 -- 172.21.15.113:0/2882814506 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f4bc80593f0 con 0x7f4bd80fe250 2024-04-12T05:14:44.144 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_OK 2024-04-12T05:14:44.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.143+0000 7f4bddc85700 1 -- 172.21.15.113:0/2882814506 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4bc406cbc0 msgr2=0x7f4bc406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:44.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.143+0000 7f4bddc85700 1 --2- 172.21.15.113:0/2882814506 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4bc406cbc0 0x7f4bc406f080 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f4bc0004f10 tx=0x7f4bc000b040).stop 2024-04-12T05:14:44.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.143+0000 7f4bddc85700 1 -- 172.21.15.113:0/2882814506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd80fe250 msgr2=0x7f4bd810e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:14:44.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.143+0000 7f4bddc85700 1 --2- 172.21.15.113:0/2882814506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd80fe250 0x7f4bd810e4d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4bc8002780 tx=0x7f4bc8009ee0).stop 2024-04-12T05:14:44.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.143+0000 7f4bddc85700 1 -- 172.21.15.113:0/2882814506 shutdown_connections 2024-04-12T05:14:44.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.143+0000 7f4bddc85700 1 --2- 172.21.15.113:0/2882814506 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4bc406cbc0 0x7f4bc406f080 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:44.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.143+0000 7f4bddc85700 1 --2- 172.21.15.113:0/2882814506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4bd80fd8f0 0x7f4bd810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:44.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.143+0000 7f4bddc85700 1 --2- 172.21.15.113:0/2882814506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4bd80fe250 0x7f4bd810e4d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:14:44.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.143+0000 7f4bddc85700 1 -- 172.21.15.113:0/2882814506 >> 172.21.15.113:0/2882814506 conn(0x7f4bd80f34b0 msgr2=0x7f4bd80fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:14:44.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.143+0000 7f4bddc85700 1 -- 172.21.15.113:0/2882814506 shutdown_connections 2024-04-12T05:14:44.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:14:44.143+0000 7f4bddc85700 1 -- 172.21.15.113:0/2882814506 wait complete. 2024-04-12T05:14:44.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:43 smithi113 ceph-mon[103371]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:44.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:43 smithi113 ceph-mon[103371]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:44.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:43 smithi113 ceph-mon[103371]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:44.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:43 smithi113 ceph-mon[103371]: Upgrade: Updating crash.smithi113 (1/2) 2024-04-12T05:14:44.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:44.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:14:44.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:44.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:43 smithi113 ceph-mon[103371]: Deploying daemon crash.smithi113 on smithi113 2024-04-12T05:14:44.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:43 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3978520512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:14:44.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:43 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3059574520' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:14:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:43 smithi192 ceph-mon[85304]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:43 smithi192 ceph-mon[85304]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:43 smithi192 ceph-mon[85304]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:43 smithi192 ceph-mon[85304]: Upgrade: Updating crash.smithi113 (1/2) 2024-04-12T05:14:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi113", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:14:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:43 smithi192 ceph-mon[85304]: Deploying daemon crash.smithi113 on smithi113 2024-04-12T05:14:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:43 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3978520512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:14:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:43 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3059574520' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:14:45.296 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:44 smithi113 ceph-mon[103371]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:45.296 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:44 smithi113 ceph-mon[103371]: pgmap v29: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:45.296 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:44 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2882814506' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:14:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:44 smithi192 ceph-mon[85304]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:14:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:44 smithi192 ceph-mon[85304]: pgmap v29: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:44 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2882814506' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:14:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:47 smithi113 ceph-mon[103371]: pgmap v30: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:47 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:47 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:47 smithi192 ceph-mon[85304]: pgmap v30: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:47 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:47 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:49.379 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:49 smithi192 ceph-mon[85304]: pgmap v31: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:49.379 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:49 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:49.379 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:49 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:14:49.379 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:49 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:49 smithi113 ceph-mon[103371]: pgmap v31: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:49 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:49 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-12T05:14:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:49 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:14:50.113 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:50 smithi192 ceph-mon[85304]: Upgrade: Updating crash.smithi192 (2/2) 2024-04-12T05:14:50.113 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:50 smithi192 ceph-mon[85304]: Deploying daemon crash.smithi192 on smithi192 2024-04-12T05:14:50.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:50 smithi113 ceph-mon[103371]: Upgrade: Updating crash.smithi192 (2/2) 2024-04-12T05:14:50.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:50 smithi113 ceph-mon[103371]: Deploying daemon crash.smithi192 on smithi192 2024-04-12T05:14:51.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:51 smithi192 ceph-mon[85304]: pgmap v32: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:51 smithi113 ceph-mon[103371]: pgmap v32: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:52.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:52.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:52.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:14:52.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:52 smithi113 ceph-mon[103371]: pgmap v33: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:52.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:14:52.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:52.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:52.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:14:52.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:52 smithi192 ceph-mon[85304]: pgmap v33: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:52.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:14:54.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:54 smithi192 ceph-mon[85304]: pgmap v34: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:55.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:54 smithi113 ceph-mon[103371]: pgmap v34: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:56.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:56.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:56.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:56 smithi192 ceph-mon[85304]: pgmap v35: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:56.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:56.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:56.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:56.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:56.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:56.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:56.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:56 smithi113 ceph-mon[103371]: pgmap v35: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:56.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:56.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:56.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:56.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:57.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:57 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:57.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:57 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:57.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:57 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:57.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:57 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:14:58.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:14:58 smithi113 ceph-mon[103371]: pgmap v36: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:14:58.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:14:58 smithi192 ceph-mon[85304]: pgmap v36: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:00.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:00 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:00.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:00 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:00.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:00 smithi113 ceph-mon[103371]: pgmap v37: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:00.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:00 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:00.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:00 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:00.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:00 smithi192 ceph-mon[85304]: pgmap v37: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:15:01.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all crash 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi113"}]: dispatch 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi113"}]': finished 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi192"}]: dispatch 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi192"}]': finished 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T05:15:01.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T05:15:01.676 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:01 smithi113 ceph-mon[103371]: Upgrade: osd.0 is safe to restart 2024-04-12T05:15:01.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:01.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:01.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all crash 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi113"}]: dispatch 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi113"}]': finished 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi192"}]: dispatch 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi192"}]': finished 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T05:15:01.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T05:15:01.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:01 smithi192 ceph-mon[85304]: Upgrade: osd.0 is safe to restart 2024-04-12T05:15:02.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:02 smithi113 ceph-mon[103371]: pgmap v38: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:02.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:02.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:15:02.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:15:02.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:02 smithi192 ceph-mon[85304]: pgmap v38: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:02.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:02.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T05:15:02.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:15:03.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:03 smithi113 ceph-mon[103371]: Upgrade: Updating osd.0 2024-04-12T05:15:03.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:03 smithi113 ceph-mon[103371]: Deploying daemon osd.0 on smithi113 2024-04-12T05:15:03.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:03 smithi192 ceph-mon[85304]: Upgrade: Updating osd.0 2024-04-12T05:15:03.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:03 smithi192 ceph-mon[85304]: Deploying daemon osd.0 on smithi113 2024-04-12T05:15:04.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:04 smithi113 ceph-mon[103371]: pgmap v39: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:04.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:04 smithi192 ceph-mon[85304]: pgmap v39: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:05.789 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:05 smithi113 systemd[1]: Stopping Ceph osd.0 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:15:05.789 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0[52570]: debug 2024-04-12T05:15:05.725+0000 7fb9f3d5c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-12T05:15:05.789 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0[52570]: debug 2024-04-12T05:15:05.725+0000 7fb9f3d5c700 -1 osd.0 49 *** Got signal Terminated *** 2024-04-12T05:15:05.790 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:05 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0[52570]: debug 2024-04-12T05:15:05.725+0000 7fb9f3d5c700 -1 osd.0 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:15:06.123 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:05 smithi113 ceph-mon[103371]: osd.0 marked itself down and dead 2024-04-12T05:15:06.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:05 smithi192 ceph-mon[85304]: osd.0 marked itself down and dead 2024-04-12T05:15:06.424 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:06 smithi113 bash[116431]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0 2024-04-12T05:15:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:06 smithi113 ceph-mon[103371]: pgmap v40: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:06 smithi113 ceph-mon[103371]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:15:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:06 smithi113 ceph-mon[103371]: osdmap e50: 6 total, 5 up, 6 in 2024-04-12T05:15:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:06 smithi113 ceph-mon[103371]: osdmap e51: 6 total, 5 up, 6 in 2024-04-12T05:15:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:06 smithi192 ceph-mon[85304]: pgmap v40: 65 pgs: 65 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:06 smithi192 ceph-mon[85304]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:15:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:06 smithi192 ceph-mon[85304]: osdmap e50: 6 total, 5 up, 6 in 2024-04-12T05:15:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:06 smithi192 ceph-mon[85304]: osdmap e51: 6 total, 5 up, 6 in 2024-04-12T05:15:08.158 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:08.158 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:15:08.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:08.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:15:08.424 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:08 smithi113 systemd[1]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.0.service: Succeeded. 2024-04-12T05:15:08.424 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:08 smithi113 systemd[1]: Stopped Ceph osd.0 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:15:08.424 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:08 smithi113 systemd[1]: Starting Ceph osd.0 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:15:08.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:08 smithi113 ceph-mon[103371]: pgmap v43: 65 pgs: 9 stale+active+clean, 56 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:08.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:08 smithi192 ceph-mon[85304]: pgmap v43: 65 pgs: 9 stale+active+clean, 56 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:09.612 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:09 smithi113 bash[116708]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-12T05:15:09.612 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:09 smithi113 bash[116708]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-12T05:15:09.612 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:09 smithi113 bash[116708]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-12T05:15:09.613 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:09 smithi113 bash[116708]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-12T05:15:09.613 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:09 smithi113 bash[116708]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-12T05:15:09.613 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:09 smithi113 bash[116708]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-12T05:15:09.613 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:09 smithi113 bash[116708]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-12T05:15:10.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:09 smithi113 ceph-mon[103371]: Health check failed: Degraded data redundancy: 63/324 objects degraded (19.444%), 19 pgs degraded (PG_DEGRADED) 2024-04-12T05:15:10.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:09 smithi192 ceph-mon[85304]: Health check failed: Degraded data redundancy: 63/324 objects degraded (19.444%), 19 pgs degraded (PG_DEGRADED) 2024-04-12T05:15:10.883 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:10 smithi113 ceph-osd[117003]: -- 172.21.15.113:0/1252998786 <== mon.0 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555d8b8431e0 con 0x555d8b813000 2024-04-12T05:15:10.883 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:10 smithi113 bash[116964]: 03af9c08ee5989c5e4079bad4d0b1e846496c4be0af31a2cd026b7c8f2d13cd7 2024-04-12T05:15:10.883 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:10 smithi113 systemd[1]: Started Ceph osd.0 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:15:11.174 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:11 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0[116980]: 2024-04-12T05:15:11.003+0000 7f66ec77a7c0 -1 Falling back to public interface 2024-04-12T05:15:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:10 smithi113 ceph-mon[103371]: pgmap v44: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:10 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:11.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:10 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:11.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:10 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:15:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:10 smithi192 ceph-mon[85304]: pgmap v44: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:10 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:10 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:11.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:10 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:15:13.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:12 smithi113 ceph-mon[103371]: pgmap v45: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:13.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:12 smithi192 ceph-mon[85304]: pgmap v45: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:14.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.260+0000 7f4acf16d700 1 -- 172.21.15.113:0/3727280924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac80fd8f0 msgr2=0x7f4ac80fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:14.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.260+0000 7f4acf16d700 1 --2- 172.21.15.113:0/3727280924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac80fd8f0 0x7f4ac80fdcd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4abc0047f0 tx=0x7f4abc009cb0).stop 2024-04-12T05:15:14.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.260+0000 7f4acf16d700 1 -- 172.21.15.113:0/3727280924 shutdown_connections 2024-04-12T05:15:14.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.260+0000 7f4acf16d700 1 --2- 172.21.15.113:0/3727280924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac80fe250 0x7f4ac80f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.260+0000 7f4acf16d700 1 --2- 172.21.15.113:0/3727280924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac80fd8f0 0x7f4ac80fdcd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.260+0000 7f4acf16d700 1 -- 172.21.15.113:0/3727280924 >> 172.21.15.113:0/3727280924 conn(0x7f4ac80f34b0 msgr2=0x7f4ac80f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:14.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.260+0000 7f4acf16d700 1 -- 172.21.15.113:0/3727280924 shutdown_connections 2024-04-12T05:15:14.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.260+0000 7f4acf16d700 1 -- 172.21.15.113:0/3727280924 wait complete. 2024-04-12T05:15:14.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4acf16d700 1 Processor -- start 2024-04-12T05:15:14.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4acf16d700 1 -- start start 2024-04-12T05:15:14.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4acf16d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac80fd8f0 0x7f4ac806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4acf16d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac80fe250 0x7f4ac806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4acf16d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac806bab0 con 0x7f4ac80fd8f0 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4acf16d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac806bbf0 con 0x7f4ac80fe250 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4accf09700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac80fd8f0 0x7f4ac806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4accf09700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac80fd8f0 0x7f4ac806b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55910/0 (socket says 172.21.15.113:55910) 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4accf09700 1 -- 172.21.15.113:0/3952317753 learned_addr learned my addr 172.21.15.113:0/3952317753 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4ac7fff700 1 --2- 172.21.15.113:0/3952317753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac80fe250 0x7f4ac806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4accf09700 1 -- 172.21.15.113:0/3952317753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac80fe250 msgr2=0x7f4ac806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4accf09700 1 --2- 172.21.15.113:0/3952317753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac80fe250 0x7f4ac806b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.261+0000 7f4accf09700 1 -- 172.21.15.113:0/3952317753 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc005040 con 0x7f4ac80fd8f0 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.262+0000 7f4accf09700 1 --2- 172.21.15.113:0/3952317753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac80fd8f0 0x7f4ac806b0d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f4abc0043a0 tx=0x7f4abc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:14.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.262+0000 7f4ac5ffb700 1 -- 172.21.15.113:0/3952317753 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4abc00e030 con 0x7f4ac80fd8f0 2024-04-12T05:15:14.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.262+0000 7f4acf16d700 1 -- 172.21.15.113:0/3952317753 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac806be40 con 0x7f4ac80fd8f0 2024-04-12T05:15:14.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.262+0000 7f4ac5ffb700 1 -- 172.21.15.113:0/3952317753 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4abc007e90 con 0x7f4ac80fd8f0 2024-04-12T05:15:14.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.262+0000 7f4ac5ffb700 1 -- 172.21.15.113:0/3952317753 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4abc018070 con 0x7f4ac80fd8f0 2024-04-12T05:15:14.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.262+0000 7f4acf16d700 1 -- 172.21.15.113:0/3952317753 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac8114610 con 0x7f4ac80fd8f0 2024-04-12T05:15:14.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.263+0000 7f4acf16d700 1 -- 172.21.15.113:0/3952317753 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac805f590 con 0x7f4ac80fd8f0 2024-04-12T05:15:14.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.264+0000 7f4ac5ffb700 1 -- 172.21.15.113:0/3952317753 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f4abc0094c0 con 0x7f4ac80fd8f0 2024-04-12T05:15:14.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.264+0000 7f4ac5ffb700 1 --2- 172.21.15.113:0/3952317753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ab806cbc0 0x7f4ab806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:14.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.264+0000 7f4ac5ffb700 1 -- 172.21.15.113:0/3952317753 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f4abc08d1f0 con 0x7f4ac80fd8f0 2024-04-12T05:15:14.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.264+0000 7f4ac7fff700 1 --2- 172.21.15.113:0/3952317753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ab806cbc0 0x7f4ab806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:14.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.265+0000 7f4ac7fff700 1 --2- 172.21.15.113:0/3952317753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ab806cbc0 0x7f4ab806f080 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f4ac806c870 tx=0x7f4ab400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:14.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.270+0000 7f4ac5ffb700 1 -- 172.21.15.113:0/3952317753 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4abc057150 con 0x7f4ac80fd8f0 2024-04-12T05:15:14.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.505+0000 7f4acf16d700 1 -- 172.21.15.113:0/3952317753 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ac8059120 con 0x7f4ab806cbc0 2024-04-12T05:15:14.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.506+0000 7f4ac5ffb700 1 -- 172.21.15.113:0/3952317753 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f4ac8059120 con 0x7f4ab806cbc0 2024-04-12T05:15:14.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.508+0000 7f4acf16d700 1 -- 172.21.15.113:0/3952317753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ab806cbc0 msgr2=0x7f4ab806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:14.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.508+0000 7f4acf16d700 1 --2- 172.21.15.113:0/3952317753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ab806cbc0 0x7f4ab806f080 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f4ac806c870 tx=0x7f4ab400b040).stop 2024-04-12T05:15:14.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.508+0000 7f4acf16d700 1 -- 172.21.15.113:0/3952317753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac80fd8f0 msgr2=0x7f4ac806b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:14.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.508+0000 7f4acf16d700 1 --2- 172.21.15.113:0/3952317753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac80fd8f0 0x7f4ac806b0d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f4abc0043a0 tx=0x7f4abc009cb0).stop 2024-04-12T05:15:14.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.508+0000 7f4acf16d700 1 -- 172.21.15.113:0/3952317753 shutdown_connections 2024-04-12T05:15:14.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.508+0000 7f4acf16d700 1 --2- 172.21.15.113:0/3952317753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ab806cbc0 0x7f4ab806f080 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.508+0000 7f4acf16d700 1 --2- 172.21.15.113:0/3952317753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ac80fd8f0 0x7f4ac806b0d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.508+0000 7f4acf16d700 1 --2- 172.21.15.113:0/3952317753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ac80fe250 0x7f4ac806b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.508+0000 7f4acf16d700 1 -- 172.21.15.113:0/3952317753 >> 172.21.15.113:0/3952317753 conn(0x7f4ac80f34b0 msgr2=0x7f4ac80fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:14.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.508+0000 7f4acf16d700 1 -- 172.21.15.113:0/3952317753 shutdown_connections 2024-04-12T05:15:14.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.508+0000 7f4acf16d700 1 -- 172.21.15.113:0/3952317753 wait complete. 2024-04-12T05:15:14.522 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:15:14.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.612+0000 7f639c4ca700 1 -- 172.21.15.113:0/3711362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940f6fc0 msgr2=0x7f6394103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:14.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.612+0000 7f639c4ca700 1 --2- 172.21.15.113:0/3711362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940f6fc0 0x7f6394103d60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f6384004820 tx=0x7f6384009cb0).stop 2024-04-12T05:15:14.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.612+0000 7f639c4ca700 1 -- 172.21.15.113:0/3711362 shutdown_connections 2024-04-12T05:15:14.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.612+0000 7f639c4ca700 1 --2- 172.21.15.113:0/3711362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940f6fc0 0x7f6394103d60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.612+0000 7f639c4ca700 1 --2- 172.21.15.113:0/3711362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63940f66f0 0x7f63940f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.612+0000 7f639c4ca700 1 -- 172.21.15.113:0/3711362 >> 172.21.15.113:0/3711362 conn(0x7f6394070590 msgr2=0x7f63940f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:14.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.612+0000 7f639c4ca700 1 -- 172.21.15.113:0/3711362 shutdown_connections 2024-04-12T05:15:14.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.612+0000 7f639c4ca700 1 -- 172.21.15.113:0/3711362 wait complete. 2024-04-12T05:15:14.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.613+0000 7f639c4ca700 1 Processor -- start 2024-04-12T05:15:14.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.613+0000 7f639c4ca700 1 -- start start 2024-04-12T05:15:14.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.613+0000 7f639c4ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63940f66f0 0x7f63941102f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:14.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.613+0000 7f639c4ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940f6fc0 0x7f63941107e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:14.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.613+0000 7f639c4ca700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6394110de0 con 0x7f63940f6fc0 2024-04-12T05:15:14.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.613+0000 7f639c4ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6394110f20 con 0x7f63940f66f0 2024-04-12T05:15:14.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.613+0000 7f6399a65700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940f6fc0 0x7f63941107e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:14.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.613+0000 7f6399a65700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940f6fc0 0x7f63941107e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55932/0 (socket says 172.21.15.113:55932) 2024-04-12T05:15:14.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.613+0000 7f6399a65700 1 -- 172.21.15.113:0/2842708254 learned_addr learned my addr 172.21.15.113:0/2842708254 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:14.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.614+0000 7f639a266700 1 --2- 172.21.15.113:0/2842708254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63940f66f0 0x7f63941102f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:14.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.614+0000 7f6399a65700 1 -- 172.21.15.113:0/2842708254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63940f66f0 msgr2=0x7f63941102f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:14.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.614+0000 7f6399a65700 1 --2- 172.21.15.113:0/2842708254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63940f66f0 0x7f63941102f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.614+0000 7f6399a65700 1 -- 172.21.15.113:0/2842708254 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6384005040 con 0x7f63940f6fc0 2024-04-12T05:15:14.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.614+0000 7f6399a65700 1 --2- 172.21.15.113:0/2842708254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940f6fc0 0x7f63941107e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f6384005010 tx=0x7f6384009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:14.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.614+0000 7f638b7fe700 1 -- 172.21.15.113:0/2842708254 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f638400e030 con 0x7f63940f6fc0 2024-04-12T05:15:14.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.614+0000 7f638b7fe700 1 -- 172.21.15.113:0/2842708254 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6384007e90 con 0x7f63940f6fc0 2024-04-12T05:15:14.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.614+0000 7f638b7fe700 1 -- 172.21.15.113:0/2842708254 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6384018070 con 0x7f63940f6fc0 2024-04-12T05:15:14.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.614+0000 7f639c4ca700 1 -- 172.21.15.113:0/2842708254 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639410a550 con 0x7f63940f6fc0 2024-04-12T05:15:14.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.614+0000 7f639c4ca700 1 -- 172.21.15.113:0/2842708254 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639410aa30 con 0x7f63940f6fc0 2024-04-12T05:15:14.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.615+0000 7f639c4ca700 1 -- 172.21.15.113:0/2842708254 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f639405f590 con 0x7f63940f6fc0 2024-04-12T05:15:14.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.616+0000 7f638b7fe700 1 -- 172.21.15.113:0/2842708254 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f638401d420 con 0x7f63940f6fc0 2024-04-12T05:15:14.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.616+0000 7f638b7fe700 1 --2- 172.21.15.113:0/2842708254 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638006cb60 0x7f638006f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:14.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.616+0000 7f638b7fe700 1 -- 172.21.15.113:0/2842708254 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f638408d2d0 con 0x7f63940f6fc0 2024-04-12T05:15:14.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.616+0000 7f639a266700 1 --2- 172.21.15.113:0/2842708254 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638006cb60 0x7f638006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:14.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.617+0000 7f639a266700 1 --2- 172.21.15.113:0/2842708254 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638006cb60 0x7f638006f020 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6390009170 tx=0x7f6390009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:14.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.623+0000 7f638b7fe700 1 -- 172.21.15.113:0/2842708254 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f638405acc0 con 0x7f63940f6fc0 2024-04-12T05:15:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.850+0000 7f639c4ca700 1 -- 172.21.15.113:0/2842708254 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f639410b9e0 con 0x7f638006cb60 2024-04-12T05:15:14.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.851+0000 7f638b7fe700 1 -- 172.21.15.113:0/2842708254 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f639410b9e0 con 0x7f638006cb60 2024-04-12T05:15:14.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.853+0000 7f639c4ca700 1 -- 172.21.15.113:0/2842708254 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638006cb60 msgr2=0x7f638006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:14.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.853+0000 7f639c4ca700 1 --2- 172.21.15.113:0/2842708254 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638006cb60 0x7f638006f020 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6390009170 tx=0x7f6390009d60).stop 2024-04-12T05:15:14.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.853+0000 7f639c4ca700 1 -- 172.21.15.113:0/2842708254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940f6fc0 msgr2=0x7f63941107e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.853+0000 7f639c4ca700 1 --2- 172.21.15.113:0/2842708254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940f6fc0 0x7f63941107e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f6384005010 tx=0x7f6384009cb0).stop 2024-04-12T05:15:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.853+0000 7f639c4ca700 1 -- 172.21.15.113:0/2842708254 shutdown_connections 2024-04-12T05:15:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.853+0000 7f639c4ca700 1 --2- 172.21.15.113:0/2842708254 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638006cb60 0x7f638006f020 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.853+0000 7f639c4ca700 1 --2- 172.21.15.113:0/2842708254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63940f66f0 0x7f63941102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.853+0000 7f639c4ca700 1 --2- 172.21.15.113:0/2842708254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63940f6fc0 0x7f63941107e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.853+0000 7f639c4ca700 1 -- 172.21.15.113:0/2842708254 >> 172.21.15.113:0/2842708254 conn(0x7f6394070590 msgr2=0x7f639406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.853+0000 7f639c4ca700 1 -- 172.21.15.113:0/2842708254 shutdown_connections 2024-04-12T05:15:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.853+0000 7f639c4ca700 1 -- 172.21.15.113:0/2842708254 wait complete. 2024-04-12T05:15:14.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.959+0000 7fd0f186e700 1 -- 172.21.15.113:0/708179007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0ec0fd8f0 msgr2=0x7fd0ec0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:14.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.959+0000 7fd0f186e700 1 --2- 172.21.15.113:0/708179007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0ec0fd8f0 0x7fd0ec0fdcd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fd0dc0047f0 tx=0x7fd0dc009cb0).stop 2024-04-12T05:15:14.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.959+0000 7fd0f186e700 1 -- 172.21.15.113:0/708179007 shutdown_connections 2024-04-12T05:15:14.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.959+0000 7fd0f186e700 1 --2- 172.21.15.113:0/708179007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0ec0fe250 0x7fd0ec0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:15:14.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.959+0000 7fd0f186e700 1 --2- 172.21.15.113:0/708179007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0ec0fd8f0 0x7fd0ec0fdcd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.959+0000 7fd0f186e700 1 -- 172.21.15.113:0/708179007 >> 172.21.15.113:0/708179007 conn(0x7fd0ec0f34b0 msgr2=0x7fd0ec0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:14.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.959+0000 7fd0f186e700 1 -- 172.21.15.113:0/708179007 shutdown_connections 2024-04-12T05:15:14.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.959+0000 7fd0f186e700 1 -- 172.21.15.113:0/708179007 wait complete. 2024-04-12T05:15:14.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.959+0000 7fd0f186e700 1 Processor -- start 2024-04-12T05:15:14.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0f186e700 1 -- start start 2024-04-12T05:15:14.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0f186e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0ec0fd8f0 0x7fd0ec06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:14.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0f186e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0ec0fe250 0x7fd0ec06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:14.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0f186e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec06bab0 con 0x7fd0ec0fd8f0 2024-04-12T05:15:14.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0f186e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec06bbf0 con 0x7fd0ec0fe250 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0eaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0ec0fd8f0 0x7fd0ec06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0eaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0ec0fd8f0 0x7fd0ec06b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55956/0 (socket says 172.21.15.113:55956) 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0eaffd700 1 -- 172.21.15.113:0/834026 learned_addr learned my addr 172.21.15.113:0/834026 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0ea7fc700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0ec0fe250 0x7fd0ec06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0eaffd700 1 -- 172.21.15.113:0/834026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0ec0fe250 msgr2=0x7fd0ec06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0eaffd700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0ec0fe250 0x7fd0ec06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0eaffd700 1 -- 172.21.15.113:0/834026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc005040 con 0x7fd0ec0fd8f0 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0ea7fc700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0ec0fe250 0x7fd0ec06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.960+0000 7fd0eaffd700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0ec0fd8f0 0x7fd0ec06b0d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd0dc00b120 tx=0x7fd0dc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.961+0000 7fd0d3fff700 1 -- 172.21.15.113:0/834026 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0dc00e030 con 0x7fd0ec0fd8f0 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.961+0000 7fd0f186e700 1 -- 172.21.15.113:0/834026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ec06be40 con 0x7fd0ec0fd8f0 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.961+0000 7fd0d3fff700 1 -- 172.21.15.113:0/834026 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0dc007e90 con 0x7fd0ec0fd8f0 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.961+0000 7fd0d3fff700 1 -- 172.21.15.113:0/834026 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0dc018070 con 0x7fd0ec0fd8f0 2024-04-12T05:15:14.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.961+0000 7fd0f186e700 1 -- 172.21.15.113:0/834026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ec114760 con 0x7fd0ec0fd8f0 2024-04-12T05:15:14.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.962+0000 7fd0f186e700 1 -- 172.21.15.113:0/834026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ec05f590 con 0x7fd0ec0fd8f0 2024-04-12T05:15:14.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.962+0000 7fd0d3fff700 1 -- 172.21.15.113:0/834026 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7fd0dc01d420 con 0x7fd0ec0fd8f0 2024-04-12T05:15:14.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.963+0000 7fd0d3fff700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0d406cbc0 0x7fd0d406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:14.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.963+0000 7fd0d3fff700 1 -- 172.21.15.113:0/834026 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fd0dc08d380 con 0x7fd0ec0fd8f0 2024-04-12T05:15:14.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.963+0000 7fd0ea7fc700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0d406cbc0 0x7fd0d406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:14.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.963+0000 7fd0ea7fc700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0d406cbc0 0x7fd0d406f080 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fd0ec06c870 tx=0x7fd0e0008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:14.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:14.967+0000 7fd0d3fff700 1 -- 172.21.15.113:0/834026 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd0dc0572e0 con 0x7fd0ec0fd8f0 2024-04-12T05:15:15.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:14 smithi113 ceph-mon[103371]: pgmap v46: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:15.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:14 smithi192 ceph-mon[85304]: pgmap v46: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:15.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.202+0000 7fd0f186e700 1 -- 172.21.15.113:0/834026 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0ec059120 con 0x7fd0d406cbc0 2024-04-12T05:15:15.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.214+0000 7fd0d3fff700 1 -- 172.21.15.113:0/834026 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd0ec059120 con 0x7fd0d406cbc0 2024-04-12T05:15:15.216 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:15:15.216 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (3m) 19s ago 14m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:15:15.216 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (29s) 19s ago 14m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:15:15.216 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (24s) 19s ago 13m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:15:15.216 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (2m) 19s ago 14m 82.4M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (8m) 19s ago 8m 120M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (8m) 19s ago 8m 17.9M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (8m) 19s ago 8m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (8m) 19s ago 8m 15.3M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (4m) 19s ago 15m 492M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (3m) 19s ago 13m 448M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (93s) 19s ago 15m 41.6M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (68s) 19s ago 13m 33.0M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (3m) 19s ago 13m 16.3M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (3m) 19s ago 13m 17.8M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 starting - - - 1910M 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (12m) 19s ago 12m 240M 1910M 16.2.15-1-g8963e9de 727f386679cb 866869552c7a 2024-04-12T05:15:15.217 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (11m) 19s ago 11m 202M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:15:15.218 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (11m) 19s ago 11m 264M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:15:15.218 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (10m) 19s ago 10m 231M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:15:15.218 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (10m) 19s ago 10m 226M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:15:15.218 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (3m) 19s ago 13m 59.8M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:15:15.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.216+0000 7fd0f186e700 1 -- 172.21.15.113:0/834026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0d406cbc0 msgr2=0x7fd0d406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:15.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.216+0000 7fd0f186e700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0d406cbc0 0x7fd0d406f080 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fd0ec06c870 tx=0x7fd0e0008040).stop 2024-04-12T05:15:15.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.216+0000 7fd0f186e700 1 -- 172.21.15.113:0/834026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0ec0fd8f0 msgr2=0x7fd0ec06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:15.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.217+0000 7fd0f186e700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0ec0fd8f0 0x7fd0ec06b0d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd0dc00b120 tx=0x7fd0dc009cb0).stop 2024-04-12T05:15:15.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.217+0000 7fd0f186e700 1 -- 172.21.15.113:0/834026 shutdown_connections 2024-04-12T05:15:15.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.217+0000 7fd0f186e700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0d406cbc0 0x7fd0d406f080 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.217+0000 7fd0f186e700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0ec0fd8f0 0x7fd0ec06b0d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.217+0000 7fd0f186e700 1 --2- 172.21.15.113:0/834026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0ec0fe250 0x7fd0ec06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.217+0000 7fd0f186e700 1 -- 172.21.15.113:0/834026 >> 172.21.15.113:0/834026 conn(0x7fd0ec0f34b0 msgr2=0x7fd0ec0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:15.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.217+0000 7fd0f186e700 1 -- 172.21.15.113:0/834026 shutdown_connections 2024-04-12T05:15:15.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.217+0000 7fd0f186e700 1 -- 172.21.15.113:0/834026 wait complete. 2024-04-12T05:15:15.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.333+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/4184134768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb140fe270 msgr2=0x7fbb140f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:15.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.333+0000 7fbb1a7b0700 1 --2- 172.21.15.113:0/4184134768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb140fe270 0x7fbb140f7a50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbb080047f0 tx=0x7fbb08009cb0).stop 2024-04-12T05:15:15.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.333+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/4184134768 shutdown_connections 2024-04-12T05:15:15.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.333+0000 7fbb1a7b0700 1 --2- 172.21.15.113:0/4184134768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb140fe270 0x7fbb140f7a50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.333+0000 7fbb1a7b0700 1 --2- 172.21.15.113:0/4184134768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb140fd910 0x7fbb140fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.333+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/4184134768 >> 172.21.15.113:0/4184134768 conn(0x7fbb140f3510 msgr2=0x7fbb140f5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:15.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.333+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/4184134768 shutdown_connections 2024-04-12T05:15:15.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.333+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/4184134768 wait complete. 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.334+0000 7fbb1a7b0700 1 Processor -- start 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.334+0000 7fbb1a7b0700 1 -- start start 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.334+0000 7fbb1a7b0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb140fd910 0x7fbb1410dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.334+0000 7fbb1a7b0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb140fe270 0x7fbb1410e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.334+0000 7fbb1a7b0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1410eac0 con 0x7fbb140fd910 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.334+0000 7fbb1a7b0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1410ec00 con 0x7fbb140fe270 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.334+0000 7fbb13fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb140fd910 0x7fbb1410dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.334+0000 7fbb13fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb140fd910 0x7fbb1410dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55962/0 (socket says 172.21.15.113:55962) 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.334+0000 7fbb13fff700 1 -- 172.21.15.113:0/3134594518 learned_addr learned my addr 172.21.15.113:0/3134594518 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.334+0000 7fbb137fe700 1 --2- 172.21.15.113:0/3134594518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb140fe270 0x7fbb1410e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.335+0000 7fbb13fff700 1 -- 172.21.15.113:0/3134594518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb140fe270 msgr2=0x7fbb1410e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.335+0000 7fbb13fff700 1 --2- 172.21.15.113:0/3134594518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb140fe270 0x7fbb1410e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.335+0000 7fbb13fff700 1 -- 172.21.15.113:0/3134594518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb08005040 con 0x7fbb140fd910 2024-04-12T05:15:15.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.335+0000 7fbb13fff700 1 --2- 172.21.15.113:0/3134594518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb140fd910 0x7fbb1410dfd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fbb040041d0 tx=0x7fbb04009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:15.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.335+0000 7fbb117fa700 1 -- 172.21.15.113:0/3134594518 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb04007b40 con 0x7fbb140fd910 2024-04-12T05:15:15.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.335+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/3134594518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb14108290 con 0x7fbb140fd910 2024-04-12T05:15:15.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.335+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/3134594518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb14108850 con 0x7fbb140fd910 2024-04-12T05:15:15.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.335+0000 7fbb117fa700 1 -- 172.21.15.113:0/3134594518 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb0400f040 con 0x7fbb140fd910 2024-04-12T05:15:15.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.335+0000 7fbb117fa700 1 -- 172.21.15.113:0/3134594518 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb04014510 con 0x7fbb140fd910 2024-04-12T05:15:15.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.336+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/3134594518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb00005150 con 0x7fbb140fd910 2024-04-12T05:15:15.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.337+0000 7fbb117fa700 1 -- 172.21.15.113:0/3134594518 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7fbb04009110 con 0x7fbb140fd910 2024-04-12T05:15:15.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.337+0000 7fbb117fa700 1 --2- 172.21.15.113:0/3134594518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbafc070fd0 0x7fbafc073490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:15.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.337+0000 7fbb117fa700 1 -- 172.21.15.113:0/3134594518 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fbb0408bc40 con 0x7fbb140fd910 2024-04-12T05:15:15.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.337+0000 7fbb137fe700 1 --2- 172.21.15.113:0/3134594518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbafc070fd0 0x7fbafc073490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:15.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.338+0000 7fbb137fe700 1 --2- 172.21.15.113:0/3134594518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbafc070fd0 0x7fbafc073490 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fbb14109910 tx=0x7fbb0800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:15.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.342+0000 7fbb117fa700 1 -- 172.21.15.113:0/3134594518 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbb040595b0 con 0x7fbb140fd910 2024-04-12T05:15:15.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.650+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/3134594518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbb00005af0 con 0x7fbb140fd910 2024-04-12T05:15:15.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.651+0000 7fbb117fa700 1 -- 172.21.15.113:0/3134594518 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7fbb04059140 con 0x7fbb140fd910 2024-04-12T05:15:15.653 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:15:15.653 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:15:15.653 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:15:15.653 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:15:15.653 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:15:15.653 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:15:15.653 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:15:15.653 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:15:15.653 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 5 2024-04-12T05:15:15.653 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:15:15.654 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:15:15.654 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:15:15.654 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:15:15.654 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:15:15.654 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 9, 2024-04-12T05:15:15.654 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:15:15.654 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:15:15.654 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:15:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.653+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/3134594518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbafc070fd0 msgr2=0x7fbafc073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.653+0000 7fbb1a7b0700 1 --2- 172.21.15.113:0/3134594518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbafc070fd0 0x7fbafc073490 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fbb14109910 tx=0x7fbb0800b040).stop 2024-04-12T05:15:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.653+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/3134594518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb140fd910 msgr2=0x7fbb1410dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.653+0000 7fbb1a7b0700 1 --2- 172.21.15.113:0/3134594518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb140fd910 0x7fbb1410dfd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fbb040041d0 tx=0x7fbb04009f60).stop 2024-04-12T05:15:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.653+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/3134594518 shutdown_connections 2024-04-12T05:15:15.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.653+0000 7fbb1a7b0700 1 --2- 172.21.15.113:0/3134594518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbafc070fd0 0x7fbafc073490 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.653+0000 7fbb1a7b0700 1 --2- 172.21.15.113:0/3134594518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb140fd910 0x7fbb1410dfd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.653+0000 7fbb1a7b0700 1 --2- 172.21.15.113:0/3134594518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb140fe270 0x7fbb1410e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.653+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/3134594518 >> 172.21.15.113:0/3134594518 conn(0x7fbb140f3510 msgr2=0x7fbb140fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:15.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.653+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/3134594518 shutdown_connections 2024-04-12T05:15:15.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.653+0000 7fbb1a7b0700 1 -- 172.21.15.113:0/3134594518 wait complete. 2024-04-12T05:15:15.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.760+0000 7f1c9def4700 1 -- 172.21.15.113:0/1215852992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c980fd8f0 msgr2=0x7f1c980fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:15.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.760+0000 7f1c9def4700 1 --2- 172.21.15.113:0/1215852992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c980fd8f0 0x7f1c980fdcd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f1c880047f0 tx=0x7f1c88009cb0).stop 2024-04-12T05:15:15.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.760+0000 7f1c9def4700 1 -- 172.21.15.113:0/1215852992 shutdown_connections 2024-04-12T05:15:15.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.760+0000 7f1c9def4700 1 --2- 172.21.15.113:0/1215852992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c980fe250 0x7f1c980f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.760+0000 7f1c9def4700 1 --2- 172.21.15.113:0/1215852992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c980fd8f0 0x7f1c980fdcd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.760+0000 7f1c9def4700 1 -- 172.21.15.113:0/1215852992 >> 172.21.15.113:0/1215852992 conn(0x7f1c980f34b0 msgr2=0x7f1c980f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:15.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.760+0000 7f1c9def4700 1 -- 172.21.15.113:0/1215852992 shutdown_connections 2024-04-12T05:15:15.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.760+0000 7f1c9def4700 1 -- 172.21.15.113:0/1215852992 wait complete. 2024-04-12T05:15:15.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.761+0000 7f1c9def4700 1 Processor -- start 2024-04-12T05:15:15.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.761+0000 7f1c9def4700 1 -- start start 2024-04-12T05:15:15.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.761+0000 7f1c9def4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c980fd8f0 0x7f1c9810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:15.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.761+0000 7f1c9def4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c980fe250 0x7f1c9810e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:15.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.761+0000 7f1c9def4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9810ead0 con 0x7f1c980fd8f0 2024-04-12T05:15:15.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.761+0000 7f1c9def4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9810ec10 con 0x7f1c980fe250 2024-04-12T05:15:15.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.761+0000 7f1c977fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c980fd8f0 0x7f1c9810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.761+0000 7f1c977fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c980fd8f0 0x7f1c9810dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55980/0 (socket says 172.21.15.113:55980) 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.762+0000 7f1c977fe700 1 -- 172.21.15.113:0/3066338923 learned_addr learned my addr 172.21.15.113:0/3066338923 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.762+0000 7f1c96ffd700 1 --2- 172.21.15.113:0/3066338923 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c980fe250 0x7f1c9810e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.762+0000 7f1c977fe700 1 -- 172.21.15.113:0/3066338923 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c980fe250 msgr2=0x7f1c9810e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.762+0000 7f1c977fe700 1 --2- 172.21.15.113:0/3066338923 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c980fe250 0x7f1c9810e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.762+0000 7f1c977fe700 1 -- 172.21.15.113:0/3066338923 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c88005040 con 0x7f1c980fd8f0 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.762+0000 7f1c977fe700 1 --2- 172.21.15.113:0/3066338923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c980fd8f0 0x7f1c9810dfe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f1c88005010 tx=0x7f1c88009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.762+0000 7f1c94ff9700 1 -- 172.21.15.113:0/3066338923 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c8800e030 con 0x7f1c980fd8f0 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.762+0000 7f1c94ff9700 1 -- 172.21.15.113:0/3066338923 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c88007e90 con 0x7f1c980fd8f0 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.762+0000 7f1c9def4700 1 -- 172.21.15.113:0/3066338923 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c98108240 con 0x7f1c980fd8f0 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.762+0000 7f1c94ff9700 1 -- 172.21.15.113:0/3066338923 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c88018070 con 0x7f1c980fd8f0 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.762+0000 7f1c9def4700 1 -- 172.21.15.113:0/3066338923 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c981087a0 con 0x7f1c980fd8f0 2024-04-12T05:15:15.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.763+0000 7f1c9def4700 1 -- 172.21.15.113:0/3066338923 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9805f590 con 0x7f1c980fd8f0 2024-04-12T05:15:15.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.764+0000 7f1c94ff9700 1 -- 172.21.15.113:0/3066338923 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f1c8801d670 con 0x7f1c980fd8f0 2024-04-12T05:15:15.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.765+0000 7f1c94ff9700 1 --2- 172.21.15.113:0/3066338923 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c8006cbc0 0x7f1c8006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:15.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.765+0000 7f1c94ff9700 1 -- 172.21.15.113:0/3066338923 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f1c8808d7c0 con 0x7f1c980fd8f0 2024-04-12T05:15:15.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.765+0000 7f1c96ffd700 1 --2- 172.21.15.113:0/3066338923 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c8006cbc0 0x7f1c8006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:15.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.765+0000 7f1c96ffd700 1 --2- 172.21.15.113:0/3066338923 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c8006cbc0 0x7f1c8006f080 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1c98109920 tx=0x7f1c8c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:15.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:15.769+0000 7f1c94ff9700 1 -- 172.21.15.113:0/3066338923 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1c8805b1b0 con 0x7f1c980fd8f0 2024-04-12T05:15:16.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.018+0000 7f1c9def4700 1 -- 172.21.15.113:0/3066338923 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1c980ffc40 con 0x7f1c980fd8f0 2024-04-12T05:15:16.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.019+0000 7f1c94ff9700 1 -- 172.21.15.113:0/3066338923 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1859 (secure 0 0 0) 0x7f1c8800c0c0 con 0x7f1c980fd8f0 2024-04-12T05:15:16.021 INFO:teuthology.orchestra.run.smithi113.stdout:e18 2024-04-12T05:15:16.021 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:15:16.021 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:15:16.021 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:15:16.021 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:15:16.021 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:15:16.021 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:15:16.021 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 18 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:15:16.022 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=14426,1=24261} 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:15:16.023 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:15:16.024 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:15:16.024 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:15:16.024 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:15:16.024 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:15:16.024 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.021+0000 7f1c9def4700 1 -- 172.21.15.113:0/3066338923 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c8006cbc0 msgr2=0x7f1c8006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.021+0000 7f1c9def4700 1 --2- 172.21.15.113:0/3066338923 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c8006cbc0 0x7f1c8006f080 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1c98109920 tx=0x7f1c8c00b040).stop 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.021+0000 7f1c9def4700 1 -- 172.21.15.113:0/3066338923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c980fd8f0 msgr2=0x7f1c9810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.021+0000 7f1c9def4700 1 --2- 172.21.15.113:0/3066338923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c980fd8f0 0x7f1c9810dfe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f1c88005010 tx=0x7f1c88009cb0).stop 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.021+0000 7f1c9def4700 1 -- 172.21.15.113:0/3066338923 shutdown_connections 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.021+0000 7f1c9def4700 1 --2- 172.21.15.113:0/3066338923 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c8006cbc0 0x7f1c8006f080 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.021+0000 7f1c9def4700 1 --2- 172.21.15.113:0/3066338923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c980fd8f0 0x7f1c9810dfe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.021+0000 7f1c9def4700 1 --2- 172.21.15.113:0/3066338923 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c980fe250 0x7f1c9810e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.021+0000 7f1c9def4700 1 -- 172.21.15.113:0/3066338923 >> 172.21.15.113:0/3066338923 conn(0x7f1c980f34b0 msgr2=0x7f1c980fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.021+0000 7f1c9def4700 1 -- 172.21.15.113:0/3066338923 shutdown_connections 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.021+0000 7f1c9def4700 1 -- 172.21.15.113:0/3066338923 wait complete. 2024-04-12T05:15:16.025 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 18 2024-04-12T05:15:16.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.124+0000 7f1b84005700 1 -- 172.21.15.113:0/2289569146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b7c0f9200 msgr2=0x7f1b7c101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:16.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.124+0000 7f1b84005700 1 --2- 172.21.15.113:0/2289569146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b7c0f9200 0x7f1b7c101910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f1b6c0047f0 tx=0x7f1b6c009cb0).stop 2024-04-12T05:15:16.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.124+0000 7f1b84005700 1 -- 172.21.15.113:0/2289569146 shutdown_connections 2024-04-12T05:15:16.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.124+0000 7f1b84005700 1 --2- 172.21.15.113:0/2289569146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b7c0f9200 0x7f1b7c101910 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.124+0000 7f1b84005700 1 --2- 172.21.15.113:0/2289569146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b7c0f88a0 0x7f1b7c0f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.124+0000 7f1b84005700 1 -- 172.21.15.113:0/2289569146 >> 172.21.15.113:0/2289569146 conn(0x7f1b7c070590 msgr2=0x7f1b7c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:16.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.124+0000 7f1b84005700 1 -- 172.21.15.113:0/2289569146 shutdown_connections 2024-04-12T05:15:16.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.124+0000 7f1b84005700 1 -- 172.21.15.113:0/2289569146 wait complete. 2024-04-12T05:15:16.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.125+0000 7f1b84005700 1 Processor -- start 2024-04-12T05:15:16.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.125+0000 7f1b84005700 1 -- start start 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.125+0000 7f1b84005700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b7c0f88a0 0x7f1b7c10e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.125+0000 7f1b84005700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b7c0f9200 0x7f1b7c10e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.125+0000 7f1b84005700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c10eb70 con 0x7f1b7c0f88a0 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.125+0000 7f1b84005700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c10ecb0 con 0x7f1b7c0f9200 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b81da1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b7c0f88a0 0x7f1b7c10e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b81da1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b7c0f88a0 0x7f1b7c10e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56008/0 (socket says 172.21.15.113:56008) 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b81da1700 1 -- 172.21.15.113:0/2065664132 learned_addr learned my addr 172.21.15.113:0/2065664132 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b815a0700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b7c0f9200 0x7f1b7c10e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b81da1700 1 -- 172.21.15.113:0/2065664132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b7c0f9200 msgr2=0x7f1b7c10e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b81da1700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b7c0f9200 0x7f1b7c10e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b81da1700 1 -- 172.21.15.113:0/2065664132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c005040 con 0x7f1b7c0f88a0 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b815a0700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b7c0f9200 0x7f1b7c10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:15:16.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b81da1700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b7c0f88a0 0x7f1b7c10e080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f1b78002770 tx=0x7f1b78002b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:16.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b72ffd700 1 -- 172.21.15.113:0/2065664132 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b78002d80 con 0x7f1b7c0f88a0 2024-04-12T05:15:16.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b72ffd700 1 -- 172.21.15.113:0/2065664132 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b7800f040 con 0x7f1b7c0f88a0 2024-04-12T05:15:16.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b72ffd700 1 -- 172.21.15.113:0/2065664132 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b7800d800 con 0x7f1b7c0f88a0 2024-04-12T05:15:16.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b84005700 1 -- 172.21.15.113:0/2065664132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7c108340 con 0x7f1b7c0f88a0 2024-04-12T05:15:16.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.126+0000 7f1b84005700 1 -- 172.21.15.113:0/2065664132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7c108880 con 0x7f1b7c0f88a0 2024-04-12T05:15:16.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.127+0000 7f1b84005700 1 -- 172.21.15.113:0/2065664132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7c05f590 con 0x7f1b7c0f88a0 2024-04-12T05:15:16.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.128+0000 7f1b72ffd700 1 -- 172.21.15.113:0/2065664132 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f1b78007b40 con 0x7f1b7c0f88a0 2024-04-12T05:15:16.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.129+0000 7f1b72ffd700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b6806cbc0 0x7f1b6806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:16.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.129+0000 7f1b72ffd700 1 -- 172.21.15.113:0/2065664132 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f1b7808be70 con 0x7f1b7c0f88a0 2024-04-12T05:15:16.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.129+0000 7f1b815a0700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b6806cbc0 0x7f1b6806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:16.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.130+0000 7f1b815a0700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b6806cbc0 0x7f1b6806f080 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1b7c1099c0 tx=0x7f1b6c009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:16.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.133+0000 7f1b72ffd700 1 -- 172.21.15.113:0/2065664132 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1b78059820 con 0x7f1b7c0f88a0 2024-04-12T05:15:16.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:15 smithi113 ceph-mon[103371]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:16.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:15 smithi113 ceph-mon[103371]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:16.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:15 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:16.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:15 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:16.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:15 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3134594518' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:16.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:15 smithi192 ceph-mon[85304]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:16.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:15 smithi192 ceph-mon[85304]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:16.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:15 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:16.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:15 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:16.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:15 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3134594518' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:16.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.440+0000 7f1b84005700 1 -- 172.21.15.113:0/2065664132 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b7c108b50 con 0x7f1b6806cbc0 2024-04-12T05:15:16.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.441+0000 7f1b72ffd700 1 -- 172.21.15.113:0/2065664132 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1b7c108b50 con 0x7f1b6806cbc0 2024-04-12T05:15:16.443 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "7/21 daemons upgraded", 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:15:16.444 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:15:16.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.445+0000 7f1b84005700 1 -- 172.21.15.113:0/2065664132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b6806cbc0 msgr2=0x7f1b6806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:16.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.445+0000 7f1b84005700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b6806cbc0 0x7f1b6806f080 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1b7c1099c0 tx=0x7f1b6c009cb0).stop 2024-04-12T05:15:16.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.445+0000 7f1b84005700 1 -- 172.21.15.113:0/2065664132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b7c0f88a0 msgr2=0x7f1b7c10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:16.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.445+0000 7f1b84005700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b7c0f88a0 0x7f1b7c10e080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f1b78002770 tx=0x7f1b78002b30).stop 2024-04-12T05:15:16.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.445+0000 7f1b84005700 1 -- 172.21.15.113:0/2065664132 shutdown_connections 2024-04-12T05:15:16.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.445+0000 7f1b84005700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b6806cbc0 0x7f1b6806f080 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.445+0000 7f1b84005700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b7c0f88a0 0x7f1b7c10e080 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.445+0000 7f1b84005700 1 --2- 172.21.15.113:0/2065664132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b7c0f9200 0x7f1b7c10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.445+0000 7f1b84005700 1 -- 172.21.15.113:0/2065664132 >> 172.21.15.113:0/2065664132 conn(0x7f1b7c070590 msgr2=0x7f1b7c0fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:16.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.445+0000 7f1b84005700 1 -- 172.21.15.113:0/2065664132 shutdown_connections 2024-04-12T05:15:16.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.445+0000 7f1b84005700 1 -- 172.21.15.113:0/2065664132 wait complete. 2024-04-12T05:15:16.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.554+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/1730096027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e740f88a0 msgr2=0x7f7e740f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:16.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.554+0000 7f7e7bb9f700 1 --2- 172.21.15.113:0/1730096027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e740f88a0 0x7f7e740f8c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f7e700047f0 tx=0x7f7e70009cb0).stop 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.554+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/1730096027 shutdown_connections 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.554+0000 7f7e7bb9f700 1 --2- 172.21.15.113:0/1730096027 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e740f9200 0x7f7e74101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.554+0000 7f7e7bb9f700 1 --2- 172.21.15.113:0/1730096027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e740f88a0 0x7f7e740f8c80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.554+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/1730096027 >> 172.21.15.113:0/1730096027 conn(0x7f7e740f23b0 msgr2=0x7f7e740f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.554+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/1730096027 shutdown_connections 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.554+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/1730096027 wait complete. 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.554+0000 7f7e7bb9f700 1 Processor -- start 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7bb9f700 1 -- start start 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7bb9f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e740f88a0 0x7f7e7410e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7bb9f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e740f9200 0x7f7e7410e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7bb9f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7410eb50 con 0x7f7e740f88a0 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7bb9f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7410ec90 con 0x7f7e740f9200 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7993b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e740f88a0 0x7f7e7410e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:16.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7993b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e740f88a0 0x7f7e7410e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56038/0 (socket says 172.21.15.113:56038) 2024-04-12T05:15:16.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7993b700 1 -- 172.21.15.113:0/330876948 learned_addr learned my addr 172.21.15.113:0/330876948 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:16.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7913a700 1 --2- 172.21.15.113:0/330876948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e740f9200 0x7f7e7410e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:16.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7993b700 1 -- 172.21.15.113:0/330876948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e740f9200 msgr2=0x7f7e7410e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:16.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7993b700 1 --2- 172.21.15.113:0/330876948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e740f9200 0x7f7e7410e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7993b700 1 -- 172.21.15.113:0/330876948 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e70005040 con 0x7f7e740f88a0 2024-04-12T05:15:16.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e7993b700 1 --2- 172.21.15.113:0/330876948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e740f88a0 0x7f7e7410e060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f7e70005010 tx=0x7f7e70009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:16.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.555+0000 7f7e6affd700 1 -- 172.21.15.113:0/330876948 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e7000e030 con 0x7f7e740f88a0 2024-04-12T05:15:16.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.556+0000 7f7e6affd700 1 -- 172.21.15.113:0/330876948 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e70007e90 con 0x7f7e740f88a0 2024-04-12T05:15:16.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.556+0000 7f7e6affd700 1 -- 172.21.15.113:0/330876948 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e70018070 con 0x7f7e740f88a0 2024-04-12T05:15:16.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.556+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/330876948 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e741082c0 con 0x7f7e740f88a0 2024-04-12T05:15:16.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.556+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/330876948 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e741087a0 con 0x7f7e740f88a0 2024-04-12T05:15:16.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.557+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/330876948 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e74060390 con 0x7f7e740f88a0 2024-04-12T05:15:16.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.557+0000 7f7e6affd700 1 -- 172.21.15.113:0/330876948 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 37) v1 ==== 90954+0+0 (secure 0 0 0) 0x7f7e7001d670 con 0x7f7e740f88a0 2024-04-12T05:15:16.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.558+0000 7f7e6affd700 1 --2- 172.21.15.113:0/330876948 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7e6006cbc0 0x7f7e6006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:16.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.558+0000 7f7e6affd700 1 -- 172.21.15.113:0/330876948 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f7e7008d270 con 0x7f7e740f88a0 2024-04-12T05:15:16.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.558+0000 7f7e7913a700 1 --2- 172.21.15.113:0/330876948 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7e6006cbc0 0x7f7e6006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:16.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.559+0000 7f7e7913a700 1 --2- 172.21.15.113:0/330876948 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7e6006cbc0 0x7f7e6006f080 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7e741099a0 tx=0x7f7e6400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:16.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.563+0000 7f7e6affd700 1 -- 172.21.15.113:0/330876948 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7e7005ac60 con 0x7f7e740f88a0 2024-04-12T05:15:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.850+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/330876948 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7e741096e0 con 0x7f7e740f88a0 2024-04-12T05:15:16.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.851+0000 7f7e6affd700 1 -- 172.21.15.113:0/330876948 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1328 (secure 0 0 0) 0x7f7e7005a7f0 con 0x7f7e740f88a0 2024-04-12T05:15:16.854 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 63/324 objects degraded (19.444%), 19 pgs degraded 2024-04-12T05:15:16.854 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T05:15:16.854 INFO:teuthology.orchestra.run.smithi113.stdout: osd.0 (root=default,host=smithi113) is down 2024-04-12T05:15:16.854 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 63/324 objects degraded (19.444%), 19 pgs degraded 2024-04-12T05:15:16.854 INFO:teuthology.orchestra.run.smithi113.stdout: pg 1.0 is active+undersized+degraded, acting [3,1] 2024-04-12T05:15:16.854 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.0 is active+undersized+degraded, acting [3,1] 2024-04-12T05:15:16.854 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1 is active+undersized+degraded, acting [2,1] 2024-04-12T05:15:16.854 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-04-12T05:15:16.854 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.4 is active+undersized+degraded, acting [1,4] 2024-04-12T05:15:16.854 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.5 is active+undersized+degraded, acting [3,4] 2024-04-12T05:15:16.854 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.8 is active+undersized+degraded, acting [3,5] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.9 is active+undersized+degraded, acting [1,4] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.f is active+undersized+degraded, acting [4,5] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.12 is active+undersized+degraded, acting [3,1] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.13 is active+undersized+degraded, acting [4,2] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.15 is active+undersized+degraded, acting [1,3] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.19 is active+undersized+degraded, acting [4,2] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1b is active+undersized+degraded, acting [1,5] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1d is active+undersized+degraded, acting [3,5] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1e is active+undersized+degraded, acting [2,5] 2024-04-12T05:15:16.855 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1f is active+undersized+degraded, acting [3,4] 2024-04-12T05:15:16.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.854+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/330876948 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7e6006cbc0 msgr2=0x7f7e6006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:16.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.854+0000 7f7e7bb9f700 1 --2- 172.21.15.113:0/330876948 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7e6006cbc0 0x7f7e6006f080 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7e741099a0 tx=0x7f7e6400b040).stop 2024-04-12T05:15:16.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.854+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/330876948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e740f88a0 msgr2=0x7f7e7410e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:16.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.854+0000 7f7e7bb9f700 1 --2- 172.21.15.113:0/330876948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e740f88a0 0x7f7e7410e060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f7e70005010 tx=0x7f7e70009cb0).stop 2024-04-12T05:15:16.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.854+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/330876948 shutdown_connections 2024-04-12T05:15:16.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.854+0000 7f7e7bb9f700 1 --2- 172.21.15.113:0/330876948 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7e6006cbc0 0x7f7e6006f080 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.854+0000 7f7e7bb9f700 1 --2- 172.21.15.113:0/330876948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e740f88a0 0x7f7e7410e060 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.854+0000 7f7e7bb9f700 1 --2- 172.21.15.113:0/330876948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e740f9200 0x7f7e7410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:16.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.854+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/330876948 >> 172.21.15.113:0/330876948 conn(0x7f7e740f23b0 msgr2=0x7f7e740fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:16.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.854+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/330876948 shutdown_connections 2024-04-12T05:15:16.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:16.854+0000 7f7e7bb9f700 1 -- 172.21.15.113:0/330876948 wait complete. 2024-04-12T05:15:17.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:16 smithi113 ceph-mon[103371]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:17.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:16 smithi113 ceph-mon[103371]: pgmap v47: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:17.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:17.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:17.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:16 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3066338923' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:15:17.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:16 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/330876948' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:15:17.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:16 smithi192 ceph-mon[85304]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:16 smithi192 ceph-mon[85304]: pgmap v47: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:16 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3066338923' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:15:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:16 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/330876948' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:15:17.630 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0[116980]: 2024-04-12T05:15:17.476+0000 7f66ec77a7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-12T05:15:18.174 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:17 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0[116980]: 2024-04-12T05:15:17.902+0000 7f66ec77a7c0 -1 osd.0 49 log_to_monitors true 2024-04-12T05:15:18.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:17 smithi113 ceph-mon[103371]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:18.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:17 smithi113 ceph-mon[103371]: from='osd.0 [v2:172.21.15.113:6802/2361630947,v1:172.21.15.113:6803/2361630947]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T05:15:18.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:17 smithi192 ceph-mon[85304]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:18.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:17 smithi192 ceph-mon[85304]: from='osd.0 [v2:172.21.15.113:6802/2361630947,v1:172.21.15.113:6803/2361630947]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T05:15:18.969 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:15:18 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0[116980]: 2024-04-12T05:15:18.678+0000 7f66d9bcc700 -1 osd.0 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:15:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:18 smithi113 ceph-mon[103371]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:18 smithi113 ceph-mon[103371]: from='osd.0 [v2:172.21.15.113:6802/2361630947,v1:172.21.15.113:6803/2361630947]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T05:15:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:18 smithi113 ceph-mon[103371]: osdmap e52: 6 total, 5 up, 6 in 2024-04-12T05:15:19.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:18 smithi113 ceph-mon[103371]: from='osd.0 [v2:172.21.15.113:6802/2361630947,v1:172.21.15.113:6803/2361630947]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:15:19.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:18 smithi113 ceph-mon[103371]: from='osd.0 [v2:172.21.15.113:6802/2361630947,v1:172.21.15.113:6803/2361630947]' entity='osd.0' 2024-04-12T05:15:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:18 smithi192 ceph-mon[85304]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:18 smithi192 ceph-mon[85304]: from='osd.0 [v2:172.21.15.113:6802/2361630947,v1:172.21.15.113:6803/2361630947]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T05:15:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:18 smithi192 ceph-mon[85304]: osdmap e52: 6 total, 5 up, 6 in 2024-04-12T05:15:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:18 smithi192 ceph-mon[85304]: from='osd.0 [v2:172.21.15.113:6802/2361630947,v1:172.21.15.113:6803/2361630947]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:15:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:18 smithi192 ceph-mon[85304]: from='osd.0 [v2:172.21.15.113:6802/2361630947,v1:172.21.15.113:6803/2361630947]' entity='osd.0' 2024-04-12T05:15:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:19 smithi113 ceph-mon[103371]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:15:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:19 smithi113 ceph-mon[103371]: osd.0 [v2:172.21.15.113:6802/2361630947,v1:172.21.15.113:6803/2361630947] boot 2024-04-12T05:15:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:19 smithi113 ceph-mon[103371]: osdmap e53: 6 total, 6 up, 6 in 2024-04-12T05:15:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:19 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:15:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:19 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:19 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:19 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:15:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:19 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:15:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:19 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:19 smithi192 ceph-mon[85304]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:15:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:19 smithi192 ceph-mon[85304]: osd.0 [v2:172.21.15.113:6802/2361630947,v1:172.21.15.113:6803/2361630947] boot 2024-04-12T05:15:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:19 smithi192 ceph-mon[85304]: osdmap e53: 6 total, 6 up, 6 in 2024-04-12T05:15:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:19 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T05:15:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:19 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:19 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:19 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:15:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:19 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:15:20.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:19 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:20 smithi113 ceph-mon[103371]: pgmap v51: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:15:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:15:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:20 smithi113 ceph-mon[103371]: osdmap e54: 6 total, 6 up, 6 in 2024-04-12T05:15:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:20 smithi192 ceph-mon[85304]: pgmap v51: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:15:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:15:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:20 smithi192 ceph-mon[85304]: osdmap e54: 6 total, 6 up, 6 in 2024-04-12T05:15:22.270 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:21 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:15:22.270 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:21 smithi113 ceph-mon[103371]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-12T05:15:22.270 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:22.270 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:15:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:21 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:15:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:21 smithi192 ceph-mon[85304]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-12T05:15:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:15:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:22 smithi113 ceph-mon[103371]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:22 smithi192 ceph-mon[85304]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 63/324 objects degraded (19.444%) 2024-04-12T05:15:24.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:23 smithi113 ceph-mon[103371]: mgrmap e38: smithi113.ctsazg(active, since 91s), standbys: smithi192.ocppyo 2024-04-12T05:15:24.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:23 smithi192 ceph-mon[85304]: mgrmap e38: smithi113.ctsazg(active, since 91s), standbys: smithi192.ocppyo 2024-04-12T05:15:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:24 smithi113 ceph-mon[103371]: pgmap v54: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-04-12T05:15:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:24 smithi113 ceph-mon[103371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 63/324 objects degraded (19.444%), 19 pgs degraded) 2024-04-12T05:15:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:24 smithi113 ceph-mon[103371]: Cluster is now healthy 2024-04-12T05:15:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:24 smithi192 ceph-mon[85304]: pgmap v54: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 853 B/s wr, 0 op/s 2024-04-12T05:15:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:24 smithi192 ceph-mon[85304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 63/324 objects degraded (19.444%), 19 pgs degraded) 2024-04-12T05:15:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:24 smithi192 ceph-mon[85304]: Cluster is now healthy 2024-04-12T05:15:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:27 smithi113 ceph-mon[103371]: pgmap v55: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 655 B/s wr, 0 op/s 2024-04-12T05:15:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:27 smithi192 ceph-mon[85304]: pgmap v55: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 655 B/s wr, 0 op/s 2024-04-12T05:15:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:29 smithi113 ceph-mon[103371]: pgmap v56: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 632 B/s wr, 0 op/s 2024-04-12T05:15:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:29 smithi192 ceph-mon[85304]: pgmap v56: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 632 B/s wr, 0 op/s 2024-04-12T05:15:31.427 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:31 smithi113 ceph-mon[103371]: pgmap v57: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:15:31.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:31 smithi192 ceph-mon[85304]: pgmap v57: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-04-12T05:15:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:32 smithi113 ceph-mon[103371]: pgmap v58: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 461 B/s wr, 0 op/s 2024-04-12T05:15:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:32 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:33.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:32 smithi192 ceph-mon[85304]: pgmap v58: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 461 B/s wr, 0 op/s 2024-04-12T05:15:33.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:32 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:34 smithi113 ceph-mon[103371]: pgmap v59: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-04-12T05:15:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:34 smithi192 ceph-mon[85304]: pgmap v59: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-04-12T05:15:35.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:35 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:15:36.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:35 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:15:37.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:36 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:15:37.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:36 smithi113 ceph-mon[103371]: Upgrade: osd.1 is safe to restart 2024-04-12T05:15:37.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:36 smithi113 ceph-mon[103371]: pgmap v60: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:37.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:37.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:15:37.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:37.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:36 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T05:15:37.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:36 smithi192 ceph-mon[85304]: Upgrade: osd.1 is safe to restart 2024-04-12T05:15:37.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:36 smithi192 ceph-mon[85304]: pgmap v60: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:37.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:37.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:15:37.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:38.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:37 smithi113 ceph-mon[103371]: Upgrade: Updating osd.1 2024-04-12T05:15:38.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:15:38.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:15:38.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:37 smithi113 ceph-mon[103371]: Deploying daemon osd.1 on smithi113 2024-04-12T05:15:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:37 smithi192 ceph-mon[85304]: Upgrade: Updating osd.1 2024-04-12T05:15:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T05:15:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:15:38.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:37 smithi192 ceph-mon[85304]: Deploying daemon osd.1 on smithi113 2024-04-12T05:15:38.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:38 smithi113 ceph-mon[103371]: pgmap v61: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:39.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:38 smithi192 ceph-mon[85304]: pgmap v61: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:40.030 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:40 smithi113 systemd[1]: Stopping Ceph osd.1 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:15:40.674 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[58952]: debug 2024-04-12T05:15:40.301+0000 7feb4710f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-12T05:15:40.674 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[58952]: debug 2024-04-12T05:15:40.301+0000 7feb4710f700 -1 osd.1 54 *** Got signal Terminated *** 2024-04-12T05:15:40.674 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:40 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[58952]: debug 2024-04-12T05:15:40.301+0000 7feb4710f700 -1 osd.1 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:15:41.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:40 smithi113 ceph-mon[103371]: pgmap v62: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:41.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:40 smithi113 ceph-mon[103371]: osd.1 marked itself down and dead 2024-04-12T05:15:41.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:40 smithi192 ceph-mon[85304]: pgmap v62: 65 pgs: 65 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:41.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:40 smithi192 ceph-mon[85304]: osd.1 marked itself down and dead 2024-04-12T05:15:41.674 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:41 smithi113 bash[123577]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1 2024-04-12T05:15:42.100 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:41 smithi113 ceph-mon[103371]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:15:42.100 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:41 smithi113 ceph-mon[103371]: osdmap e55: 6 total, 5 up, 6 in 2024-04-12T05:15:42.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:41 smithi192 ceph-mon[85304]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:15:42.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:41 smithi192 ceph-mon[85304]: osdmap e55: 6 total, 5 up, 6 in 2024-04-12T05:15:43.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:42 smithi113 ceph-mon[103371]: pgmap v64: 65 pgs: 12 stale+active+clean, 53 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:43.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:42 smithi113 ceph-mon[103371]: osdmap e56: 6 total, 5 up, 6 in 2024-04-12T05:15:43.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:42 smithi192 ceph-mon[85304]: pgmap v64: 65 pgs: 12 stale+active+clean, 53 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:15:43.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:42 smithi192 ceph-mon[85304]: osdmap e56: 6 total, 5 up, 6 in 2024-04-12T05:15:43.674 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:43 smithi113 systemd[1]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.1.service: Succeeded. 2024-04-12T05:15:43.674 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:43 smithi113 systemd[1]: Stopped Ceph osd.1 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:15:43.674 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:43 smithi113 systemd[1]: Starting Ceph osd.1 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:15:44.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:43 smithi113 ceph-mon[103371]: Health check failed: Degraded data redundancy: 51/324 objects degraded (15.741%), 16 pgs degraded (PG_DEGRADED) 2024-04-12T05:15:44.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:43 smithi192 ceph-mon[85304]: Health check failed: Degraded data redundancy: 51/324 objects degraded (15.741%), 16 pgs degraded (PG_DEGRADED) 2024-04-12T05:15:44.674 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:44 smithi113 bash[123845]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-12T05:15:44.674 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:44 smithi113 bash[123845]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-12T05:15:44.674 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:44 smithi113 bash[123845]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-12T05:15:44.674 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:44 smithi113 bash[123845]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-12T05:15:44.675 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:44 smithi113 bash[123845]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-12T05:15:44.675 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:44 smithi113 bash[123845]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-12T05:15:44.675 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:44 smithi113 bash[123845]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-12T05:15:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:44 smithi113 ceph-mon[103371]: pgmap v66: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:45.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:44 smithi192 ceph-mon[85304]: pgmap v66: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:45.924 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:45 smithi113 ceph-osd[124141]: -- 172.21.15.113:0/3870853124 <== mon.0 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5592f05891e0 con 0x5592f0559000 2024-04-12T05:15:45.924 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:45 smithi113 bash[124104]: cad088ac8df16589e9db3f8169bbd77820bf6807565bf019ddda998c2dc59b93 2024-04-12T05:15:45.924 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:45 smithi113 systemd[1]: Started Ceph osd.1 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:15:46.424 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:46 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[124118]: 2024-04-12T05:15:46.160+0000 7f9aa2aa87c0 -1 Falling back to public interface 2024-04-12T05:15:46.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.963+0000 7f92f153b700 1 -- 172.21.15.113:0/1764846286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0fd150 msgr2=0x7f92ec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:46.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.963+0000 7f92f153b700 1 --2- 172.21.15.113:0/1764846286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0fd150 0x7f92ec0f6930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f92e00047f0 tx=0x7f92e0009cb0).stop 2024-04-12T05:15:46.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.963+0000 7f92f153b700 1 -- 172.21.15.113:0/1764846286 shutdown_connections 2024-04-12T05:15:46.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.963+0000 7f92f153b700 1 --2- 172.21.15.113:0/1764846286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0fd150 0x7f92ec0f6930 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:46.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.963+0000 7f92f153b700 1 --2- 172.21.15.113:0/1764846286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92ec0fc7f0 0x7f92ec0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:46.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.963+0000 7f92f153b700 1 -- 172.21.15.113:0/1764846286 >> 172.21.15.113:0/1764846286 conn(0x7f92ec0f23b0 msgr2=0x7f92ec0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:46.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.963+0000 7f92f153b700 1 -- 172.21.15.113:0/1764846286 shutdown_connections 2024-04-12T05:15:46.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.964+0000 7f92f153b700 1 -- 172.21.15.113:0/1764846286 wait complete. 2024-04-12T05:15:46.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.964+0000 7f92f153b700 1 Processor -- start 2024-04-12T05:15:46.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92f153b700 1 -- start start 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92f153b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0fc7f0 0x7f92ec10cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92f153b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92ec0fd150 0x7f92ec10d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92f153b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec10da60 con 0x7f92ec0fc7f0 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92f153b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec10dba0 con 0x7f92ec0fd150 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92eaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0fc7f0 0x7f92ec10cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92eaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0fc7f0 0x7f92ec10cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54858/0 (socket says 172.21.15.113:54858) 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92eaffd700 1 -- 172.21.15.113:0/2982656126 learned_addr learned my addr 172.21.15.113:0/2982656126 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92ea7fc700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92ec0fd150 0x7f92ec10d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92eaffd700 1 -- 172.21.15.113:0/2982656126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92ec0fd150 msgr2=0x7f92ec10d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92eaffd700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92ec0fd150 0x7f92ec10d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92eaffd700 1 -- 172.21.15.113:0/2982656126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e0005040 con 0x7f92ec0fc7f0 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92ea7fc700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92ec0fd150 0x7f92ec10d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:15:46.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92eaffd700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0fc7f0 0x7f92ec10cf70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f92dc0041d0 tx=0x7f92dc009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:46.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92d3fff700 1 -- 172.21.15.113:0/2982656126 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92dc002d90 con 0x7f92ec0fc7f0 2024-04-12T05:15:46.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.965+0000 7f92f153b700 1 -- 172.21.15.113:0/2982656126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ec107230 con 0x7f92ec0fc7f0 2024-04-12T05:15:46.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.966+0000 7f92d3fff700 1 -- 172.21.15.113:0/2982656126 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92dc00f040 con 0x7f92ec0fc7f0 2024-04-12T05:15:46.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.966+0000 7f92d3fff700 1 -- 172.21.15.113:0/2982656126 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92dc00d800 con 0x7f92ec0fc7f0 2024-04-12T05:15:46.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.966+0000 7f92f153b700 1 -- 172.21.15.113:0/2982656126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ec107770 con 0x7f92ec0fc7f0 2024-04-12T05:15:46.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.966+0000 7f92f153b700 1 -- 172.21.15.113:0/2982656126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ec060370 con 0x7f92ec0fc7f0 2024-04-12T05:15:46.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.968+0000 7f92d3fff700 1 -- 172.21.15.113:0/2982656126 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f92dc007b40 con 0x7f92ec0fc7f0 2024-04-12T05:15:46.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.968+0000 7f92d3fff700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f92d406cab0 0x7f92d406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:46.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.968+0000 7f92d3fff700 1 -- 172.21.15.113:0/2982656126 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f92dc08c050 con 0x7f92ec0fc7f0 2024-04-12T05:15:46.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.968+0000 7f92ea7fc700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f92d406cab0 0x7f92d406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:46.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.968+0000 7f92ea7fc700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f92d406cab0 0x7f92d406ef70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f92ec1088b0 tx=0x7f92e0009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:46.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:46.972+0000 7f92d3fff700 1 -- 172.21.15.113:0/2982656126 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f92dc059c60 con 0x7f92ec0fc7f0 2024-04-12T05:15:47.165 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:46 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:47.165 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:46 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:47.166 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:46 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:15:47.166 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:46 smithi113 ceph-mon[103371]: pgmap v67: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:47.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.186+0000 7f92f153b700 1 -- 172.21.15.113:0/2982656126 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f92ec107aa0 con 0x7f92d406cab0 2024-04-12T05:15:47.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.187+0000 7f92d3fff700 1 -- 172.21.15.113:0/2982656126 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f92ec107aa0 con 0x7f92d406cab0 2024-04-12T05:15:47.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.189+0000 7f92f153b700 1 -- 172.21.15.113:0/2982656126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f92d406cab0 msgr2=0x7f92d406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.189+0000 7f92f153b700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f92d406cab0 0x7f92d406ef70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f92ec1088b0 tx=0x7f92e0009cb0).stop 2024-04-12T05:15:47.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.189+0000 7f92f153b700 1 -- 172.21.15.113:0/2982656126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0fc7f0 msgr2=0x7f92ec10cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.189+0000 7f92f153b700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0fc7f0 0x7f92ec10cf70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f92dc0041d0 tx=0x7f92dc009f60).stop 2024-04-12T05:15:47.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.190+0000 7f92f153b700 1 -- 172.21.15.113:0/2982656126 shutdown_connections 2024-04-12T05:15:47.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.190+0000 7f92f153b700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f92d406cab0 0x7f92d406ef70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.190+0000 7f92f153b700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92ec0fc7f0 0x7f92ec10cf70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.190+0000 7f92f153b700 1 --2- 172.21.15.113:0/2982656126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92ec0fd150 0x7f92ec10d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.190+0000 7f92f153b700 1 -- 172.21.15.113:0/2982656126 >> 172.21.15.113:0/2982656126 conn(0x7f92ec0f23b0 msgr2=0x7f92ec0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:47.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.190+0000 7f92f153b700 1 -- 172.21.15.113:0/2982656126 shutdown_connections 2024-04-12T05:15:47.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.190+0000 7f92f153b700 1 -- 172.21.15.113:0/2982656126 wait complete. 2024-04-12T05:15:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:46 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:46 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:46 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:15:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:46 smithi192 ceph-mon[85304]: pgmap v67: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:47.202 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:15:47.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.291+0000 7fd4e53dc700 1 -- 172.21.15.113:0/2765207041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4e00fe360 msgr2=0x7fd4e00f5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.291+0000 7fd4e53dc700 1 --2- 172.21.15.113:0/2765207041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4e00fe360 0x7fd4e00f5fc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd4d40047f0 tx=0x7fd4d4009cb0).stop 2024-04-12T05:15:47.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.291+0000 7fd4e53dc700 1 -- 172.21.15.113:0/2765207041 shutdown_connections 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.291+0000 7fd4e53dc700 1 --2- 172.21.15.113:0/2765207041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4e00fe360 0x7fd4e00f5fc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.291+0000 7fd4e53dc700 1 --2- 172.21.15.113:0/2765207041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00fda00 0x7fd4e00fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.291+0000 7fd4e53dc700 1 -- 172.21.15.113:0/2765207041 >> 172.21.15.113:0/2765207041 conn(0x7fd4e006cea0 msgr2=0x7fd4e006d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.292+0000 7fd4e53dc700 1 -- 172.21.15.113:0/2765207041 shutdown_connections 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.292+0000 7fd4e53dc700 1 -- 172.21.15.113:0/2765207041 wait complete. 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.292+0000 7fd4e53dc700 1 Processor -- start 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.292+0000 7fd4e53dc700 1 -- start start 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.292+0000 7fd4e53dc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4e00fda00 0x7fd4e01101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.292+0000 7fd4e53dc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00fe360 0x7fd4e0110690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.292+0000 7fd4e53dc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e0110c90 con 0x7fd4e00fda00 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.292+0000 7fd4e53dc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e0110dd0 con 0x7fd4e00fe360 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4e00fda00 0x7fd4e01101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:47.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4e00fda00 0x7fd4e01101a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54880/0 (socket says 172.21.15.113:54880) 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4deffd700 1 -- 172.21.15.113:0/1373463577 learned_addr learned my addr 172.21.15.113:0/1373463577 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4de7fc700 1 --2- 172.21.15.113:0/1373463577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00fe360 0x7fd4e0110690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4deffd700 1 -- 172.21.15.113:0/1373463577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00fe360 msgr2=0x7fd4e0110690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4deffd700 1 --2- 172.21.15.113:0/1373463577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00fe360 0x7fd4e0110690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4deffd700 1 -- 172.21.15.113:0/1373463577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d4005040 con 0x7fd4e00fda00 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4deffd700 1 --2- 172.21.15.113:0/1373463577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4e00fda00 0x7fd4e01101a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd4d0002770 tx=0x7fd4d0002a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4c7fff700 1 -- 172.21.15.113:0/1373463577 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4d0007b40 con 0x7fd4e00fda00 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4c7fff700 1 -- 172.21.15.113:0/1373463577 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4d000f040 con 0x7fd4e00fda00 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4e53dc700 1 -- 172.21.15.113:0/1373463577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e010a460 con 0x7fd4e00fda00 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4c7fff700 1 -- 172.21.15.113:0/1373463577 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4d0014820 con 0x7fd4e00fda00 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.293+0000 7fd4e53dc700 1 -- 172.21.15.113:0/1373463577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e010a9a0 con 0x7fd4e00fda00 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.294+0000 7fd4e53dc700 1 -- 172.21.15.113:0/1373463577 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e0047540 con 0x7fd4e00fda00 2024-04-12T05:15:47.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.295+0000 7fd4c7fff700 1 -- 172.21.15.113:0/1373463577 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd4d0007ca0 con 0x7fd4e00fda00 2024-04-12T05:15:47.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.295+0000 7fd4c7fff700 1 --2- 172.21.15.113:0/1373463577 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd4c806caa0 0x7fd4c806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:47.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.295+0000 7fd4c7fff700 1 -- 172.21.15.113:0/1373463577 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fd4d008b7e0 con 0x7fd4e00fda00 2024-04-12T05:15:47.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.295+0000 7fd4de7fc700 1 --2- 172.21.15.113:0/1373463577 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd4c806caa0 0x7fd4c806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:47.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.295+0000 7fd4de7fc700 1 --2- 172.21.15.113:0/1373463577 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd4c806caa0 0x7fd4c806ef60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd4e010bae0 tx=0x7fd4d400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:47.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.299+0000 7fd4c7fff700 1 -- 172.21.15.113:0/1373463577 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd4d00593f0 con 0x7fd4e00fda00 2024-04-12T05:15:47.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.510+0000 7fd4e53dc700 1 -- 172.21.15.113:0/1373463577 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4e010ac70 con 0x7fd4c806caa0 2024-04-12T05:15:47.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.511+0000 7fd4c7fff700 1 -- 172.21.15.113:0/1373463577 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fd4e010ac70 con 0x7fd4c806caa0 2024-04-12T05:15:47.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.513+0000 7fd4e53dc700 1 -- 172.21.15.113:0/1373463577 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd4c806caa0 msgr2=0x7fd4c806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.513+0000 7fd4e53dc700 1 --2- 172.21.15.113:0/1373463577 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd4c806caa0 0x7fd4c806ef60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd4e010bae0 tx=0x7fd4d400b040).stop 2024-04-12T05:15:47.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.513+0000 7fd4e53dc700 1 -- 172.21.15.113:0/1373463577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4e00fda00 msgr2=0x7fd4e01101a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.513+0000 7fd4e53dc700 1 --2- 172.21.15.113:0/1373463577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4e00fda00 0x7fd4e01101a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd4d0002770 tx=0x7fd4d0002a80).stop 2024-04-12T05:15:47.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.513+0000 7fd4e53dc700 1 -- 172.21.15.113:0/1373463577 shutdown_connections 2024-04-12T05:15:47.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.513+0000 7fd4e53dc700 1 --2- 172.21.15.113:0/1373463577 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd4c806caa0 0x7fd4c806ef60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.513+0000 7fd4e53dc700 1 --2- 172.21.15.113:0/1373463577 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4e00fda00 0x7fd4e01101a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.513+0000 7fd4e53dc700 1 --2- 172.21.15.113:0/1373463577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00fe360 0x7fd4e0110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.513+0000 7fd4e53dc700 1 -- 172.21.15.113:0/1373463577 >> 172.21.15.113:0/1373463577 conn(0x7fd4e006cea0 msgr2=0x7fd4e00f78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:47.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.513+0000 7fd4e53dc700 1 -- 172.21.15.113:0/1373463577 shutdown_connections 2024-04-12T05:15:47.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.513+0000 7fd4e53dc700 1 -- 172.21.15.113:0/1373463577 wait complete. 2024-04-12T05:15:47.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.613+0000 7fe14f77a700 1 -- 172.21.15.113:0/468905586 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1480fe250 msgr2=0x7fe1480f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.613+0000 7fe14f77a700 1 --2- 172.21.15.113:0/468905586 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1480fe250 0x7fe1480f7a30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fe1380047f0 tx=0x7fe138009cb0).stop 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.613+0000 7fe14f77a700 1 -- 172.21.15.113:0/468905586 shutdown_connections 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.613+0000 7fe14f77a700 1 --2- 172.21.15.113:0/468905586 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1480fe250 0x7fe1480f7a30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.613+0000 7fe14f77a700 1 --2- 172.21.15.113:0/468905586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1480fd8f0 0x7fe1480fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.613+0000 7fe14f77a700 1 -- 172.21.15.113:0/468905586 >> 172.21.15.113:0/468905586 conn(0x7fe1480f34b0 msgr2=0x7fe1480f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.613+0000 7fe14f77a700 1 -- 172.21.15.113:0/468905586 shutdown_connections 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.614+0000 7fe14f77a700 1 -- 172.21.15.113:0/468905586 wait complete. 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.614+0000 7fe14f77a700 1 Processor -- start 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.614+0000 7fe14f77a700 1 -- start start 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.614+0000 7fe14f77a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1480fd8f0 0x7fe14810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.614+0000 7fe14f77a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1480fe250 0x7fe14810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.614+0000 7fe14f77a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14810ebb0 con 0x7fe1480fe250 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.614+0000 7fe14f77a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1481081d0 con 0x7fe1480fd8f0 2024-04-12T05:15:47.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.614+0000 7fe14cd15700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1480fe250 0x7fe14810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.614+0000 7fe14cd15700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1480fe250 0x7fe14810e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54894/0 (socket says 172.21.15.113:54894) 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.614+0000 7fe14cd15700 1 -- 172.21.15.113:0/1214856578 learned_addr learned my addr 172.21.15.113:0/1214856578 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.615+0000 7fe14d516700 1 --2- 172.21.15.113:0/1214856578 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1480fd8f0 0x7fe14810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.615+0000 7fe14cd15700 1 -- 172.21.15.113:0/1214856578 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1480fd8f0 msgr2=0x7fe14810e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.615+0000 7fe14cd15700 1 --2- 172.21.15.113:0/1214856578 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1480fd8f0 0x7fe14810e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.615+0000 7fe14cd15700 1 -- 172.21.15.113:0/1214856578 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe138005040 con 0x7fe1480fe250 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.615+0000 7fe14cd15700 1 --2- 172.21.15.113:0/1214856578 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1480fe250 0x7fe14810e520 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fe138000c00 tx=0x7fe138009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.615+0000 7fe13e7fc700 1 -- 172.21.15.113:0/1214856578 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe13800e030 con 0x7fe1480fe250 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.615+0000 7fe13e7fc700 1 -- 172.21.15.113:0/1214856578 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe138007e90 con 0x7fe1480fe250 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.615+0000 7fe14f77a700 1 -- 172.21.15.113:0/1214856578 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe148108420 con 0x7fe1480fe250 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.615+0000 7fe13e7fc700 1 -- 172.21.15.113:0/1214856578 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe138018070 con 0x7fe1480fe250 2024-04-12T05:15:47.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.615+0000 7fe14f77a700 1 -- 172.21.15.113:0/1214856578 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe148108900 con 0x7fe1480fe250 2024-04-12T05:15:47.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.616+0000 7fe14f77a700 1 -- 172.21.15.113:0/1214856578 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14805f590 con 0x7fe1480fe250 2024-04-12T05:15:47.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.617+0000 7fe13e7fc700 1 -- 172.21.15.113:0/1214856578 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe13801d420 con 0x7fe1480fe250 2024-04-12T05:15:47.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.617+0000 7fe13e7fc700 1 --2- 172.21.15.113:0/1214856578 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406cab0 0x7fe13406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:47.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.617+0000 7fe13e7fc700 1 -- 172.21.15.113:0/1214856578 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fe13808d110 con 0x7fe1480fe250 2024-04-12T05:15:47.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.617+0000 7fe14d516700 1 --2- 172.21.15.113:0/1214856578 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406cab0 0x7fe13406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:47.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.618+0000 7fe14d516700 1 --2- 172.21.15.113:0/1214856578 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406cab0 0x7fe13406ef70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fe14400b910 tx=0x7fe144008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:47.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.622+0000 7fe13e7fc700 1 -- 172.21.15.113:0/1214856578 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe13805ac60 con 0x7fe1480fe250 2024-04-12T05:15:47.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.843+0000 7fe14f77a700 1 -- 172.21.15.113:0/1214856578 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1481096b0 con 0x7fe13406cab0 2024-04-12T05:15:47.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.852+0000 7fe13e7fc700 1 -- 172.21.15.113:0/1214856578 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fe1481096b0 con 0x7fe13406cab0 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (3m) 32s ago 14m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (61s) 32s ago 14m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (56s) 52s ago 14m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (3m) 32s ago 14m 88.5M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (9m) 32s ago 9m 120M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (9m) 32s ago 9m 18.0M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (9m) 52s ago 9m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (9m) 52s ago 9m 15.3M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (5m) 32s ago 16m 494M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (4m) 52s ago 14m 448M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (2m) 32s ago 16m 42.2M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:15:47.855 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (100s) 52s ago 14m 33.0M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:15:47.856 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (4m) 32s ago 14m 16.5M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:15:47.856 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (4m) 52s ago 13m 17.8M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:15:47.856 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (37s) 32s ago 13m 11.3M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:15:47.856 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 starting - - - 1910M 2024-04-12T05:15:47.856 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (12m) 32s ago 12m 202M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:15:47.856 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (11m) 52s ago 11m 264M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:15:47.856 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (11m) 52s ago 11m 231M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:15:47.856 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (11m) 52s ago 11m 226M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:15:47.856 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (3m) 52s ago 13m 59.8M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:15:47.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.854+0000 7fe14f77a700 1 -- 172.21.15.113:0/1214856578 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406cab0 msgr2=0x7fe13406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.854+0000 7fe14f77a700 1 --2- 172.21.15.113:0/1214856578 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406cab0 0x7fe13406ef70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fe14400b910 tx=0x7fe144008040).stop 2024-04-12T05:15:47.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.854+0000 7fe14f77a700 1 -- 172.21.15.113:0/1214856578 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1480fe250 msgr2=0x7fe14810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.855+0000 7fe14f77a700 1 --2- 172.21.15.113:0/1214856578 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1480fe250 0x7fe14810e520 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fe138000c00 tx=0x7fe138009cb0).stop 2024-04-12T05:15:47.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.855+0000 7fe14f77a700 1 -- 172.21.15.113:0/1214856578 shutdown_connections 2024-04-12T05:15:47.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.855+0000 7fe14f77a700 1 --2- 172.21.15.113:0/1214856578 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406cab0 0x7fe13406ef70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.855+0000 7fe14f77a700 1 --2- 172.21.15.113:0/1214856578 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1480fd8f0 0x7fe14810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.855+0000 7fe14f77a700 1 --2- 172.21.15.113:0/1214856578 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1480fe250 0x7fe14810e520 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.855+0000 7fe14f77a700 1 -- 172.21.15.113:0/1214856578 >> 172.21.15.113:0/1214856578 conn(0x7fe1480f34b0 msgr2=0x7fe1480fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:47.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.855+0000 7fe14f77a700 1 -- 172.21.15.113:0/1214856578 shutdown_connections 2024-04-12T05:15:47.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.855+0000 7fe14f77a700 1 -- 172.21.15.113:0/1214856578 wait complete. 2024-04-12T05:15:47.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.952+0000 7f3754499700 1 -- 172.21.15.113:0/72835088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f374c0fe250 msgr2=0x7f374c0f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.952+0000 7f3754499700 1 --2- 172.21.15.113:0/72835088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f374c0fe250 0x7f374c0f7a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f373c0047f0 tx=0x7f373c009cb0).stop 2024-04-12T05:15:47.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 -- 172.21.15.113:0/72835088 shutdown_connections 2024-04-12T05:15:47.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 --2- 172.21.15.113:0/72835088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f374c0fe250 0x7f374c0f7a30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 --2- 172.21.15.113:0/72835088 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f374c0fd8f0 0x7f374c0fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 -- 172.21.15.113:0/72835088 >> 172.21.15.113:0/72835088 conn(0x7f374c0f34b0 msgr2=0x7f374c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 -- 172.21.15.113:0/72835088 shutdown_connections 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 -- 172.21.15.113:0/72835088 wait complete. 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 Processor -- start 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 -- start start 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f374c0fd8f0 0x7f374c06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f374c0fe250 0x7f374c06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c06bab0 con 0x7f374c0fe250 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.953+0000 7f3754499700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c06bbf0 con 0x7f374c0fd8f0 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f3751a34700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f374c0fe250 0x7f374c06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f3751a34700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f374c0fe250 0x7f374c06b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54912/0 (socket says 172.21.15.113:54912) 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f3751a34700 1 -- 172.21.15.113:0/3391500267 learned_addr learned my addr 172.21.15.113:0/3391500267 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f3752235700 1 --2- 172.21.15.113:0/3391500267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f374c0fd8f0 0x7f374c06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f3751a34700 1 -- 172.21.15.113:0/3391500267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f374c0fd8f0 msgr2=0x7f374c06b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:47.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f3751a34700 1 --2- 172.21.15.113:0/3391500267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f374c0fd8f0 0x7f374c06b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:47.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f3751a34700 1 -- 172.21.15.113:0/3391500267 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c005040 con 0x7f374c0fe250 2024-04-12T05:15:47.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f3751a34700 1 --2- 172.21.15.113:0/3391500267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f374c0fe250 0x7f374c06b5c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f373c0099f0 tx=0x7f373c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:47.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f37437fe700 1 -- 172.21.15.113:0/3391500267 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f373c00e030 con 0x7f374c0fe250 2024-04-12T05:15:47.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f3754499700 1 -- 172.21.15.113:0/3391500267 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374c06be40 con 0x7f374c0fe250 2024-04-12T05:15:47.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f37437fe700 1 -- 172.21.15.113:0/3391500267 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f373c007e90 con 0x7f374c0fe250 2024-04-12T05:15:47.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f37437fe700 1 -- 172.21.15.113:0/3391500267 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f373c018070 con 0x7f374c0fe250 2024-04-12T05:15:47.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.954+0000 7f3754499700 1 -- 172.21.15.113:0/3391500267 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374c114690 con 0x7f374c0fe250 2024-04-12T05:15:47.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.955+0000 7f3754499700 1 -- 172.21.15.113:0/3391500267 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374c05f590 con 0x7f374c0fe250 2024-04-12T05:15:47.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.956+0000 7f37437fe700 1 -- 172.21.15.113:0/3391500267 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f373c0094c0 con 0x7f374c0fe250 2024-04-12T05:15:47.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.956+0000 7f37437fe700 1 --2- 172.21.15.113:0/3391500267 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f373806caa0 0x7f373806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:47.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.956+0000 7f37437fe700 1 -- 172.21.15.113:0/3391500267 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f373c08cf60 con 0x7f374c0fe250 2024-04-12T05:15:47.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.956+0000 7f3752235700 1 --2- 172.21.15.113:0/3391500267 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f373806caa0 0x7f373806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:47.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.956+0000 7f3752235700 1 --2- 172.21.15.113:0/3391500267 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f373806caa0 0x7f373806ef60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f3748004dc0 tx=0x7f374800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:47.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:47.961+0000 7f37437fe700 1 -- 172.21.15.113:0/3391500267 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f373c0c02c0 con 0x7f374c0fe250 2024-04-12T05:15:48.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.267+0000 7f3754499700 1 -- 172.21.15.113:0/3391500267 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f374c047540 con 0x7f374c0fe250 2024-04-12T05:15:48.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.267+0000 7f37437fe700 1 -- 172.21.15.113:0/3391500267 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f373c0570e0 con 0x7f374c0fe250 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4, 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:15:48.270 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:15:48.271 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:15:48.271 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:15:48.271 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:15:48.271 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 8, 2024-04-12T05:15:48.271 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 5 2024-04-12T05:15:48.271 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:15:48.271 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:15:48.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.269+0000 7f3754499700 1 -- 172.21.15.113:0/3391500267 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f373806caa0 msgr2=0x7f373806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:48.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.269+0000 7f3754499700 1 --2- 172.21.15.113:0/3391500267 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f373806caa0 0x7f373806ef60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f3748004dc0 tx=0x7f374800b040).stop 2024-04-12T05:15:48.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.269+0000 7f3754499700 1 -- 172.21.15.113:0/3391500267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f374c0fe250 msgr2=0x7f374c06b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:48.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.269+0000 7f3754499700 1 --2- 172.21.15.113:0/3391500267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f374c0fe250 0x7f374c06b5c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f373c0099f0 tx=0x7f373c009cb0).stop 2024-04-12T05:15:48.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.269+0000 7f3754499700 1 -- 172.21.15.113:0/3391500267 shutdown_connections 2024-04-12T05:15:48.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.269+0000 7f3754499700 1 --2- 172.21.15.113:0/3391500267 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f373806caa0 0x7f373806ef60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.269+0000 7f3754499700 1 --2- 172.21.15.113:0/3391500267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f374c0fd8f0 0x7f374c06b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.269+0000 7f3754499700 1 --2- 172.21.15.113:0/3391500267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f374c0fe250 0x7f374c06b5c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.269+0000 7f3754499700 1 -- 172.21.15.113:0/3391500267 >> 172.21.15.113:0/3391500267 conn(0x7f374c0f34b0 msgr2=0x7f374c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:48.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.269+0000 7f3754499700 1 -- 172.21.15.113:0/3391500267 shutdown_connections 2024-04-12T05:15:48.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.270+0000 7f3754499700 1 -- 172.21.15.113:0/3391500267 wait complete. 2024-04-12T05:15:48.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.371+0000 7f0ba6574700 1 -- 172.21.15.113:0/1741395619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00fd8b0 msgr2=0x7f0ba00fdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:48.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.371+0000 7f0ba6574700 1 --2- 172.21.15.113:0/1741395619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00fd8b0 0x7f0ba00fdc90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f0b900047f0 tx=0x7f0b90009cb0).stop 2024-04-12T05:15:48.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.371+0000 7f0ba6574700 1 -- 172.21.15.113:0/1741395619 shutdown_connections 2024-04-12T05:15:48.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.371+0000 7f0ba6574700 1 --2- 172.21.15.113:0/1741395619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ba00fe210 0x7f0ba00f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.371+0000 7f0ba6574700 1 --2- 172.21.15.113:0/1741395619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00fd8b0 0x7f0ba00fdc90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.371+0000 7f0ba6574700 1 -- 172.21.15.113:0/1741395619 >> 172.21.15.113:0/1741395619 conn(0x7f0ba006ce40 msgr2=0x7f0ba006d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:48.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.371+0000 7f0ba6574700 1 -- 172.21.15.113:0/1741395619 shutdown_connections 2024-04-12T05:15:48.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.371+0000 7f0ba6574700 1 -- 172.21.15.113:0/1741395619 wait complete. 2024-04-12T05:15:48.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.372+0000 7f0ba6574700 1 Processor -- start 2024-04-12T05:15:48.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.372+0000 7f0ba6574700 1 -- start start 2024-04-12T05:15:48.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.372+0000 7f0ba6574700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00fd8b0 0x7f0ba010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:48.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.372+0000 7f0ba6574700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ba00fe210 0x7f0ba010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:48.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.372+0000 7f0ba6574700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba010ead0 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.372+0000 7f0ba6574700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba010ec10 con 0x7f0ba00fe210 2024-04-12T05:15:48.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.372+0000 7f0b9ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00fd8b0 0x7f0ba010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:48.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.372+0000 7f0b9ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00fd8b0 0x7f0ba010dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54934/0 (socket says 172.21.15.113:54934) 2024-04-12T05:15:48.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.372+0000 7f0b9ffff700 1 -- 172.21.15.113:0/2641152340 learned_addr learned my addr 172.21.15.113:0/2641152340 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:48.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.372+0000 7f0b9f7fe700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ba00fe210 0x7f0ba010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:48.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.373+0000 7f0b9ffff700 1 -- 172.21.15.113:0/2641152340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ba00fe210 msgr2=0x7f0ba010e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:48.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.373+0000 7f0b9ffff700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ba00fe210 0x7f0ba010e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.373+0000 7f0b9ffff700 1 -- 172.21.15.113:0/2641152340 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b90005040 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.373+0000 7f0b9f7fe700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ba00fe210 0x7f0ba010e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:15:48.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.373+0000 7f0b9ffff700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00fd8b0 0x7f0ba010dfe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f0b90005010 tx=0x7f0b90009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:48.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.373+0000 7f0b9d7fa700 1 -- 172.21.15.113:0/2641152340 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b9000e030 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.373+0000 7f0b9d7fa700 1 -- 172.21.15.113:0/2641152340 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b90007e90 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.373+0000 7f0ba6574700 1 -- 172.21.15.113:0/2641152340 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba0108240 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.373+0000 7f0b9d7fa700 1 -- 172.21.15.113:0/2641152340 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b90018070 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.373+0000 7f0ba6574700 1 -- 172.21.15.113:0/2641152340 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba0108720 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.374+0000 7f0ba6574700 1 -- 172.21.15.113:0/2641152340 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba0060670 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.374+0000 7f0b9d7fa700 1 -- 172.21.15.113:0/2641152340 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0b9001d6e0 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.375+0000 7f0b9d7fa700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8806caa0 0x7f0b8806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:48.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.375+0000 7f0b9d7fa700 1 -- 172.21.15.113:0/2641152340 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f0b9008ce80 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.375+0000 7f0b9f7fe700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8806caa0 0x7f0b8806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:48.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.375+0000 7f0b9f7fe700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8806caa0 0x7f0b8806ef60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f0b94009a00 tx=0x7f0b9400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:48.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.379+0000 7f0b9d7fa700 1 -- 172.21.15.113:0/2641152340 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0b9005aa90 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.626+0000 7f0ba6574700 1 -- 172.21.15.113:0/2641152340 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0ba0109760 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.627+0000 7f0b9d7fa700 1 -- 172.21.15.113:0/2641152340 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1859 (secure 0 0 0) 0x7f0b9005a620 con 0x7f0ba00fd8b0 2024-04-12T05:15:48.629 INFO:teuthology.orchestra.run.smithi113.stdout:e18 2024-04-12T05:15:48.629 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:15:48.629 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:15:48.629 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:15:48.629 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:15:48.629 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:15:48.629 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:15:48.629 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 18 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:15:48.630 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=14426,1=24261} 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:15:48.631 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:15:48.632 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:15:48.632 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:15:48.632 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:15:48.632 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:15:48.632 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:15:48.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.629+0000 7f0ba6574700 1 -- 172.21.15.113:0/2641152340 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8806caa0 msgr2=0x7f0b8806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:48.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.629+0000 7f0ba6574700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8806caa0 0x7f0b8806ef60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f0b94009a00 tx=0x7f0b9400b040).stop 2024-04-12T05:15:48.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.629+0000 7f0ba6574700 1 -- 172.21.15.113:0/2641152340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00fd8b0 msgr2=0x7f0ba010dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:48.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.629+0000 7f0ba6574700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00fd8b0 0x7f0ba010dfe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f0b90005010 tx=0x7f0b90009cb0).stop 2024-04-12T05:15:48.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.629+0000 7f0ba6574700 1 -- 172.21.15.113:0/2641152340 shutdown_connections 2024-04-12T05:15:48.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.629+0000 7f0ba6574700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8806caa0 0x7f0b8806ef60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.629+0000 7f0ba6574700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ba00fd8b0 0x7f0ba010dfe0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.629+0000 7f0ba6574700 1 --2- 172.21.15.113:0/2641152340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ba00fe210 0x7f0ba010e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.629+0000 7f0ba6574700 1 -- 172.21.15.113:0/2641152340 >> 172.21.15.113:0/2641152340 conn(0x7f0ba006ce40 msgr2=0x7f0ba0100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:48.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.629+0000 7f0ba6574700 1 -- 172.21.15.113:0/2641152340 shutdown_connections 2024-04-12T05:15:48.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.629+0000 7f0ba6574700 1 -- 172.21.15.113:0/2641152340 wait complete. 2024-04-12T05:15:48.633 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 18 2024-04-12T05:15:48.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.731+0000 7f124e94f700 1 -- 172.21.15.113:0/3250177967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480fd150 msgr2=0x7f12480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:48.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.731+0000 7f124e94f700 1 --2- 172.21.15.113:0/3250177967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480fd150 0x7f12480f6930 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f12380047f0 tx=0x7f1238009cb0).stop 2024-04-12T05:15:48.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.731+0000 7f124e94f700 1 -- 172.21.15.113:0/3250177967 shutdown_connections 2024-04-12T05:15:48.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.731+0000 7f124e94f700 1 --2- 172.21.15.113:0/3250177967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480fd150 0x7f12480f6930 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.731+0000 7f124e94f700 1 --2- 172.21.15.113:0/3250177967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12480fc7f0 0x7f12480fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.731+0000 7f124e94f700 1 -- 172.21.15.113:0/3250177967 >> 172.21.15.113:0/3250177967 conn(0x7f1248070590 msgr2=0x7f12480f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:48.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.731+0000 7f124e94f700 1 -- 172.21.15.113:0/3250177967 shutdown_connections 2024-04-12T05:15:48.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.731+0000 7f124e94f700 1 -- 172.21.15.113:0/3250177967 wait complete. 2024-04-12T05:15:48.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.731+0000 7f124e94f700 1 Processor -- start 2024-04-12T05:15:48.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f124e94f700 1 -- start start 2024-04-12T05:15:48.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f124e94f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12480fc7f0 0x7f1248109210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:48.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f124e94f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480fd150 0x7f1248109700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:48.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f124e94f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1248109bf0 con 0x7f12480fd150 2024-04-12T05:15:48.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f124e94f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1248109d30 con 0x7f12480fc7f0 2024-04-12T05:15:48.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f12477fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480fd150 0x7f1248109700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:48.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f12477fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480fd150 0x7f1248109700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54954/0 (socket says 172.21.15.113:54954) 2024-04-12T05:15:48.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f12477fe700 1 -- 172.21.15.113:0/934203559 learned_addr learned my addr 172.21.15.113:0/934203559 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:48.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f1247fff700 1 --2- 172.21.15.113:0/934203559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12480fc7f0 0x7f1248109210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:48.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f12477fe700 1 -- 172.21.15.113:0/934203559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12480fc7f0 msgr2=0x7f1248109210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:48.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f12477fe700 1 --2- 172.21.15.113:0/934203559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12480fc7f0 0x7f1248109210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f12477fe700 1 -- 172.21.15.113:0/934203559 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1238005040 con 0x7f12480fd150 2024-04-12T05:15:48.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.732+0000 7f12477fe700 1 --2- 172.21.15.113:0/934203559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480fd150 0x7f1248109700 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1238000c00 tx=0x7f1238009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:48.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.733+0000 7f12457fa700 1 -- 172.21.15.113:0/934203559 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1238018070 con 0x7f12480fd150 2024-04-12T05:15:48.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.733+0000 7f124e94f700 1 -- 172.21.15.113:0/934203559 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1248109f80 con 0x7f12480fd150 2024-04-12T05:15:48.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.733+0000 7f12457fa700 1 -- 172.21.15.113:0/934203559 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1238007e90 con 0x7f12480fd150 2024-04-12T05:15:48.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.733+0000 7f12457fa700 1 -- 172.21.15.113:0/934203559 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12380123c0 con 0x7f12480fd150 2024-04-12T05:15:48.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.733+0000 7f124e94f700 1 -- 172.21.15.113:0/934203559 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1248112570 con 0x7f12480fd150 2024-04-12T05:15:48.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.734+0000 7f124e94f700 1 -- 172.21.15.113:0/934203559 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124805f590 con 0x7f12480fd150 2024-04-12T05:15:48.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.734+0000 7f12457fa700 1 -- 172.21.15.113:0/934203559 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f123801d420 con 0x7f12480fd150 2024-04-12T05:15:48.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.735+0000 7f12457fa700 1 --2- 172.21.15.113:0/934203559 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f123406caa0 0x7f123406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:48.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.735+0000 7f12457fa700 1 -- 172.21.15.113:0/934203559 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f123808cbe0 con 0x7f12480fd150 2024-04-12T05:15:48.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.735+0000 7f1247fff700 1 --2- 172.21.15.113:0/934203559 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f123406caa0 0x7f123406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:48.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.735+0000 7f1247fff700 1 --2- 172.21.15.113:0/934203559 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f123406caa0 0x7f123406ef60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f1230009f20 tx=0x7f123000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:48.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.739+0000 7f12457fa700 1 -- 172.21.15.113:0/934203559 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1238056d60 con 0x7f12480fd150 2024-04-12T05:15:48.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.964+0000 7f124e94f700 1 -- 172.21.15.113:0/934203559 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1248059120 con 0x7f123406caa0 2024-04-12T05:15:48.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.965+0000 7f12457fa700 1 -- 172.21.15.113:0/934203559 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1248059120 con 0x7f123406caa0 2024-04-12T05:15:48.967 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:15:48.967 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:15:48.967 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:15:48.967 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:15:48.968 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:15:48.968 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:15:48.968 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:15:48.968 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:15:48.968 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:15:48.968 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "7/21 daemons upgraded", 2024-04-12T05:15:48.968 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:15:48.968 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:15:48.968 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:15:48.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.968+0000 7f124e94f700 1 -- 172.21.15.113:0/934203559 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f123406caa0 msgr2=0x7f123406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:48.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.968+0000 7f124e94f700 1 --2- 172.21.15.113:0/934203559 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f123406caa0 0x7f123406ef60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f1230009f20 tx=0x7f123000b040).stop 2024-04-12T05:15:48.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.968+0000 7f124e94f700 1 -- 172.21.15.113:0/934203559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480fd150 msgr2=0x7f1248109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:48.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.968+0000 7f124e94f700 1 --2- 172.21.15.113:0/934203559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480fd150 0x7f1248109700 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1238000c00 tx=0x7f1238009cb0).stop 2024-04-12T05:15:48.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.969+0000 7f124e94f700 1 -- 172.21.15.113:0/934203559 shutdown_connections 2024-04-12T05:15:48.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.969+0000 7f124e94f700 1 --2- 172.21.15.113:0/934203559 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f123406caa0 0x7f123406ef60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.969+0000 7f124e94f700 1 --2- 172.21.15.113:0/934203559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12480fc7f0 0x7f1248109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.969+0000 7f124e94f700 1 --2- 172.21.15.113:0/934203559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12480fd150 0x7f1248109700 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:48.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.969+0000 7f124e94f700 1 -- 172.21.15.113:0/934203559 >> 172.21.15.113:0/934203559 conn(0x7f1248070590 msgr2=0x7f12480fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:48.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.969+0000 7f124e94f700 1 -- 172.21.15.113:0/934203559 shutdown_connections 2024-04-12T05:15:48.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:48.969+0000 7f124e94f700 1 -- 172.21.15.113:0/934203559 wait complete. 2024-04-12T05:15:49.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.070+0000 7fc33a335700 1 -- 172.21.15.113:0/1802516272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3340fc7f0 msgr2=0x7fc3340fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:49.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.070+0000 7fc33a335700 1 --2- 172.21.15.113:0/1802516272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3340fc7f0 0x7fc3340fcbd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc3240047f0 tx=0x7fc324009cb0).stop 2024-04-12T05:15:49.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.070+0000 7fc33a335700 1 -- 172.21.15.113:0/1802516272 shutdown_connections 2024-04-12T05:15:49.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.070+0000 7fc33a335700 1 --2- 172.21.15.113:0/1802516272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3340fd150 0x7fc3340f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:49.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.070+0000 7fc33a335700 1 --2- 172.21.15.113:0/1802516272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3340fc7f0 0x7fc3340fcbd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:49.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.070+0000 7fc33a335700 1 -- 172.21.15.113:0/1802516272 >> 172.21.15.113:0/1802516272 conn(0x7fc33406bdc0 msgr2=0x7fc33406c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:49.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.071+0000 7fc33a335700 1 -- 172.21.15.113:0/1802516272 shutdown_connections 2024-04-12T05:15:49.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.071+0000 7fc33a335700 1 -- 172.21.15.113:0/1802516272 wait complete. 2024-04-12T05:15:49.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.071+0000 7fc33a335700 1 Processor -- start 2024-04-12T05:15:49.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.071+0000 7fc33a335700 1 -- start start 2024-04-12T05:15:49.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc33a335700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3340fc7f0 0x7fc334106f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:49.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc33a335700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3340fd150 0x7fc334107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:49.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc33a335700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3340f6eb0 con 0x7fc3340fc7f0 2024-04-12T05:15:49.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc33a335700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc334107940 con 0x7fc3340fd150 2024-04-12T05:15:49.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc333fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3340fc7f0 0x7fc334106f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:49.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc333fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3340fc7f0 0x7fc334106f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54978/0 (socket says 172.21.15.113:54978) 2024-04-12T05:15:49.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc333fff700 1 -- 172.21.15.113:0/917660077 learned_addr learned my addr 172.21.15.113:0/917660077 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:15:49.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc3337fe700 1 --2- 172.21.15.113:0/917660077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3340fd150 0x7fc334107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:49.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc333fff700 1 -- 172.21.15.113:0/917660077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3340fd150 msgr2=0x7fc334107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:49.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc333fff700 1 --2- 172.21.15.113:0/917660077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3340fd150 0x7fc334107450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:49.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc333fff700 1 -- 172.21.15.113:0/917660077 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc324005040 con 0x7fc3340fc7f0 2024-04-12T05:15:49.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc333fff700 1 --2- 172.21.15.113:0/917660077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3340fc7f0 0x7fc334106f60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc3240043a0 tx=0x7fc324009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:49.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc3317fa700 1 -- 172.21.15.113:0/917660077 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc32400e030 con 0x7fc3340fc7f0 2024-04-12T05:15:49.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc3317fa700 1 -- 172.21.15.113:0/917660077 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc324007e90 con 0x7fc3340fc7f0 2024-04-12T05:15:49.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc33a335700 1 -- 172.21.15.113:0/917660077 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc334107bc0 con 0x7fc3340fc7f0 2024-04-12T05:15:49.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc3317fa700 1 -- 172.21.15.113:0/917660077 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc324018070 con 0x7fc3340fc7f0 2024-04-12T05:15:49.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.072+0000 7fc33a335700 1 -- 172.21.15.113:0/917660077 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc334114790 con 0x7fc3340fc7f0 2024-04-12T05:15:49.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.073+0000 7fc33a335700 1 -- 172.21.15.113:0/917660077 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33405f090 con 0x7fc3340fc7f0 2024-04-12T05:15:49.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.074+0000 7fc3317fa700 1 -- 172.21.15.113:0/917660077 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc324009590 con 0x7fc3340fc7f0 2024-04-12T05:15:49.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.074+0000 7fc3317fa700 1 --2- 172.21.15.113:0/917660077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc31c06ca40 0x7fc31c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:15:49.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.074+0000 7fc3317fa700 1 -- 172.21.15.113:0/917660077 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fc32408d010 con 0x7fc3340fc7f0 2024-04-12T05:15:49.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.074+0000 7fc3337fe700 1 --2- 172.21.15.113:0/917660077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc31c06ca40 0x7fc31c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:15:49.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.075+0000 7fc3337fe700 1 --2- 172.21.15.113:0/917660077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc31c06ca40 0x7fc31c06ef00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fc3340f68b0 tx=0x7fc32800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:15:49.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.083+0000 7fc3317fa700 1 -- 172.21.15.113:0/917660077 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc32405ac20 con 0x7fc3340fc7f0 2024-04-12T05:15:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:48 smithi113 ceph-mon[103371]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:48 smithi113 ceph-mon[103371]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:48 smithi113 ceph-mon[103371]: pgmap v68: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:48 smithi113 ceph-mon[103371]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:48 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3391500267' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:48 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2641152340' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:15:49.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:48 smithi192 ceph-mon[85304]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:49.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:48 smithi192 ceph-mon[85304]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:49.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:48 smithi192 ceph-mon[85304]: pgmap v68: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:48 smithi192 ceph-mon[85304]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:48 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3391500267' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:49.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:48 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2641152340' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:15:49.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.375+0000 7fc33a335700 1 -- 172.21.15.113:0/917660077 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc3340f9a20 con 0x7fc3340fc7f0 2024-04-12T05:15:49.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.375+0000 7fc3317fa700 1 -- 172.21.15.113:0/917660077 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1160 (secure 0 0 0) 0x7fc32405a7b0 con 0x7fc3340fc7f0 2024-04-12T05:15:49.378 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 51/324 objects degraded (15.741%), 16 pgs degraded 2024-04-12T05:15:49.378 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T05:15:49.378 INFO:teuthology.orchestra.run.smithi113.stdout: osd.1 (root=default,host=smithi113) is down 2024-04-12T05:15:49.378 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 51/324 objects degraded (15.741%), 16 pgs degraded 2024-04-12T05:15:49.378 INFO:teuthology.orchestra.run.smithi113.stdout: pg 1.0 is active+undersized+degraded, acting [3,0] 2024-04-12T05:15:49.378 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.0 is active+undersized+degraded, acting [3,0] 2024-04-12T05:15:49.378 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-04-12T05:15:49.378 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.2 is active+undersized+degraded, acting [5,0] 2024-04-12T05:15:49.378 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-04-12T05:15:49.378 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.4 is active+undersized+degraded, acting [0,4] 2024-04-12T05:15:49.378 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-04-12T05:15:49.379 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.9 is active+undersized+degraded, acting [4,0] 2024-04-12T05:15:49.379 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.a is active+undersized+degraded, acting [4,3] 2024-04-12T05:15:49.379 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.d is active+undersized+degraded, acting [3,2] 2024-04-12T05:15:49.379 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-04-12T05:15:49.379 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.11 is active+undersized+degraded, acting [4,3] 2024-04-12T05:15:49.379 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.12 is active+undersized+degraded, acting [3,0] 2024-04-12T05:15:49.379 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.15 is active+undersized+degraded, acting [3,0] 2024-04-12T05:15:49.379 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-04-12T05:15:49.379 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1b is active+undersized+degraded, acting [0,5] 2024-04-12T05:15:49.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.379+0000 7fc33a335700 1 -- 172.21.15.113:0/917660077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc31c06ca40 msgr2=0x7fc31c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:49.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.379+0000 7fc33a335700 1 --2- 172.21.15.113:0/917660077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc31c06ca40 0x7fc31c06ef00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fc3340f68b0 tx=0x7fc32800b040).stop 2024-04-12T05:15:49.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.379+0000 7fc33a335700 1 -- 172.21.15.113:0/917660077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3340fc7f0 msgr2=0x7fc334106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:15:49.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.379+0000 7fc33a335700 1 --2- 172.21.15.113:0/917660077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3340fc7f0 0x7fc334106f60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc3240043a0 tx=0x7fc324009cb0).stop 2024-04-12T05:15:49.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.379+0000 7fc33a335700 1 -- 172.21.15.113:0/917660077 shutdown_connections 2024-04-12T05:15:49.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.379+0000 7fc33a335700 1 --2- 172.21.15.113:0/917660077 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc31c06ca40 0x7fc31c06ef00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:49.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.379+0000 7fc33a335700 1 --2- 172.21.15.113:0/917660077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3340fc7f0 0x7fc334106f60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:49.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.379+0000 7fc33a335700 1 --2- 172.21.15.113:0/917660077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3340fd150 0x7fc334107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:15:49.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.379+0000 7fc33a335700 1 -- 172.21.15.113:0/917660077 >> 172.21.15.113:0/917660077 conn(0x7fc33406bdc0 msgr2=0x7fc3340ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:15:49.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.379+0000 7fc33a335700 1 -- 172.21.15.113:0/917660077 shutdown_connections 2024-04-12T05:15:49.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:15:49.379+0000 7fc33a335700 1 -- 172.21.15.113:0/917660077 wait complete. 2024-04-12T05:15:50.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:49 smithi113 ceph-mon[103371]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:50.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:49 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/917660077' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:15:50.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:49 smithi192 ceph-mon[85304]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:15:50.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:49 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/917660077' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:15:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:50 smithi113 ceph-mon[103371]: pgmap v69: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:50 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:50 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:51.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:50 smithi192 ceph-mon[85304]: pgmap v69: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:51.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:50 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:51.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:50 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:52 smithi113 ceph-mon[103371]: pgmap v70: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:15:52.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:52.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:52.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:52 smithi192 ceph-mon[85304]: pgmap v70: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:52.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:52.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:15:53.424 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[124118]: 2024-04-12T05:15:53.015+0000 7f9aa2aa87c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-12T05:15:53.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:53 smithi113 ceph-mon[103371]: from='osd.1 [v2:172.21.15.113:6810/1423470151,v1:172.21.15.113:6811/1423470151]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T05:15:53.924 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[124118]: 2024-04-12T05:15:53.524+0000 7f9aa2aa87c0 -1 osd.1 54 log_to_monitors true 2024-04-12T05:15:53.924 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:15:53 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[124118]: 2024-04-12T05:15:53.602+0000 7f9a93f02700 -1 osd.1 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:15:53.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:53 smithi192 ceph-mon[85304]: from='osd.1 [v2:172.21.15.113:6810/1423470151,v1:172.21.15.113:6811/1423470151]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T05:15:54.907 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:54 smithi113 ceph-mon[103371]: from='osd.1 [v2:172.21.15.113:6810/1423470151,v1:172.21.15.113:6811/1423470151]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T05:15:54.907 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:54 smithi113 ceph-mon[103371]: osdmap e57: 6 total, 5 up, 6 in 2024-04-12T05:15:54.907 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:54 smithi113 ceph-mon[103371]: from='osd.1 [v2:172.21.15.113:6810/1423470151,v1:172.21.15.113:6811/1423470151]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:15:54.907 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:54 smithi113 ceph-mon[103371]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:54.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:54 smithi192 ceph-mon[85304]: from='osd.1 [v2:172.21.15.113:6810/1423470151,v1:172.21.15.113:6811/1423470151]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T05:15:54.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:54 smithi192 ceph-mon[85304]: osdmap e57: 6 total, 5 up, 6 in 2024-04-12T05:15:54.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:54 smithi192 ceph-mon[85304]: from='osd.1 [v2:172.21.15.113:6810/1423470151,v1:172.21.15.113:6811/1423470151]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:15:54.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:54 smithi192 ceph-mon[85304]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:15:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: osd.1 [v2:172.21.15.113:6810/1423470151,v1:172.21.15.113:6811/1423470151] boot 2024-04-12T05:15:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: osdmap e58: 6 total, 6 up, 6 in 2024-04-12T05:15:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:15:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:15:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:15:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:15:55.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:55.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:55.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:55.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:15:55.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:15:55.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: osd.1 [v2:172.21.15.113:6810/1423470151,v1:172.21.15.113:6811/1423470151] boot 2024-04-12T05:15:55.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: osdmap e58: 6 total, 6 up, 6 in 2024-04-12T05:15:55.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T05:15:55.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:55.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:55.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:15:55.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:15:55.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:15:55.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:15:55.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:55.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:55.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:15:55.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:15:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:56 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:15:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:56 smithi113 ceph-mon[103371]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-12T05:15:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:56 smithi113 ceph-mon[103371]: osdmap e59: 6 total, 6 up, 6 in 2024-04-12T05:15:56.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:56 smithi113 ceph-mon[103371]: pgmap v75: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:56.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:56 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:15:56.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:56 smithi192 ceph-mon[85304]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-12T05:15:56.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:56 smithi192 ceph-mon[85304]: osdmap e59: 6 total, 6 up, 6 in 2024-04-12T05:15:56.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:56 smithi192 ceph-mon[85304]: pgmap v75: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:59.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:58 smithi113 ceph-mon[103371]: pgmap v76: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:15:59.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:58 smithi192 ceph-mon[85304]: pgmap v76: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 51/324 objects degraded (15.741%) 2024-04-12T05:16:00.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:59 smithi113 ceph-mon[103371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 51/324 objects degraded (15.741%), 16 pgs degraded) 2024-04-12T05:16:00.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:15:59 smithi113 ceph-mon[103371]: Cluster is now healthy 2024-04-12T05:16:00.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:59 smithi192 ceph-mon[85304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 51/324 objects degraded (15.741%), 16 pgs degraded) 2024-04-12T05:16:00.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:15:59 smithi192 ceph-mon[85304]: Cluster is now healthy 2024-04-12T05:16:01.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:00 smithi113 ceph-mon[103371]: pgmap v77: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:01.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:00 smithi192 ceph-mon[85304]: pgmap v77: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:03.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:02 smithi113 ceph-mon[103371]: pgmap v78: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:03.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:02 smithi192 ceph-mon[85304]: pgmap v78: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:05.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:04 smithi113 ceph-mon[103371]: pgmap v79: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:05.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:04 smithi192 ceph-mon[85304]: pgmap v79: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:06 smithi113 ceph-mon[103371]: pgmap v80: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:16:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:06 smithi192 ceph-mon[85304]: pgmap v80: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:16:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:09.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:08 smithi113 ceph-mon[103371]: pgmap v81: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:09.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:08 smithi192 ceph-mon[85304]: pgmap v81: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:10 smithi113 ceph-mon[103371]: pgmap v82: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:11.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:10 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:16:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:10 smithi192 ceph-mon[85304]: pgmap v82: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:10 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:16:12.157 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:11 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:16:12.157 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:11 smithi113 ceph-mon[103371]: Upgrade: osd.2 is safe to restart 2024-04-12T05:16:12.157 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:11 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:12.157 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:11 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:16:12.157 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:11 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:16:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:11 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T05:16:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:11 smithi192 ceph-mon[85304]: Upgrade: osd.2 is safe to restart 2024-04-12T05:16:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:11 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:12.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:11 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T05:16:12.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:11 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:16:13.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:12 smithi113 ceph-mon[103371]: Upgrade: Updating osd.2 2024-04-12T05:16:13.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:12 smithi113 ceph-mon[103371]: Deploying daemon osd.2 on smithi113 2024-04-12T05:16:13.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:12 smithi113 ceph-mon[103371]: pgmap v83: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:13.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:12 smithi192 ceph-mon[85304]: Upgrade: Updating osd.2 2024-04-12T05:16:13.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:12 smithi192 ceph-mon[85304]: Deploying daemon osd.2 on smithi113 2024-04-12T05:16:13.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:12 smithi192 ceph-mon[85304]: pgmap v83: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:15.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:14 smithi113 ceph-mon[103371]: pgmap v84: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:15.175 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:14 smithi113 systemd[1]: Stopping Ceph osd.2 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:16:15.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:14 smithi192 ceph-mon[85304]: pgmap v84: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:15.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[64924]: debug 2024-04-12T05:16:15.195+0000 7f67a8fd5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-12T05:16:15.675 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[64924]: debug 2024-04-12T05:16:15.195+0000 7f67a8fd5700 -1 osd.2 59 *** Got signal Terminated *** 2024-04-12T05:16:15.675 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:15 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[64924]: debug 2024-04-12T05:16:15.195+0000 7f67a8fd5700 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:16:16.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:15 smithi113 ceph-mon[103371]: osd.2 marked itself down and dead 2024-04-12T05:16:16.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:15 smithi192 ceph-mon[85304]: osd.2 marked itself down and dead 2024-04-12T05:16:16.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:16 smithi113 bash[130845]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2 2024-04-12T05:16:17.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:16 smithi113 ceph-mon[103371]: pgmap v85: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:17.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:16 smithi113 ceph-mon[103371]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:16:17.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:16 smithi113 ceph-mon[103371]: osdmap e60: 6 total, 5 up, 6 in 2024-04-12T05:16:17.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:16 smithi192 ceph-mon[85304]: pgmap v85: 65 pgs: 65 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:17.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:16 smithi192 ceph-mon[85304]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:16:17.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:16 smithi192 ceph-mon[85304]: osdmap e60: 6 total, 5 up, 6 in 2024-04-12T05:16:18.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:17 smithi113 ceph-mon[103371]: osdmap e61: 6 total, 5 up, 6 in 2024-04-12T05:16:18.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:17 smithi192 ceph-mon[85304]: osdmap e61: 6 total, 5 up, 6 in 2024-04-12T05:16:18.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:18 smithi113 systemd[1]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.2.service: Succeeded. 2024-04-12T05:16:18.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:18 smithi113 systemd[1]: Stopped Ceph osd.2 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:16:18.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:18 smithi113 systemd[1]: Starting Ceph osd.2 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:16:19.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:18 smithi113 ceph-mon[103371]: pgmap v88: 65 pgs: 9 stale+active+clean, 56 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:19.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:18 smithi192 ceph-mon[85304]: pgmap v88: 65 pgs: 9 stale+active+clean, 56 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-12T05:16:19.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.489+0000 7f6052cf5700 1 -- 172.21.15.113:0/2201878065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f604c0f9260 msgr2=0x7f604c101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:19.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.489+0000 7f6052cf5700 1 --2- 172.21.15.113:0/2201878065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f604c0f9260 0x7f604c101970 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f60380047f0 tx=0x7f6038009cb0).stop 2024-04-12T05:16:19.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.489+0000 7f6052cf5700 1 -- 172.21.15.113:0/2201878065 shutdown_connections 2024-04-12T05:16:19.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.489+0000 7f6052cf5700 1 --2- 172.21.15.113:0/2201878065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f604c0f9260 0x7f604c101970 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:19.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.489+0000 7f6052cf5700 1 --2- 172.21.15.113:0/2201878065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f604c0f8900 0x7f604c0f8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:16:19.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.489+0000 7f6052cf5700 1 -- 172.21.15.113:0/2201878065 >> 172.21.15.113:0/2201878065 conn(0x7f604c070590 msgr2=0x7f604c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:19.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.489+0000 7f6052cf5700 1 -- 172.21.15.113:0/2201878065 shutdown_connections 2024-04-12T05:16:19.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.489+0000 7f6052cf5700 1 -- 172.21.15.113:0/2201878065 wait complete. 2024-04-12T05:16:19.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.490+0000 7f6052cf5700 1 Processor -- start 2024-04-12T05:16:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.490+0000 7f6052cf5700 1 -- start start 2024-04-12T05:16:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.490+0000 7f6052cf5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f604c0f8900 0x7f604c10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.490+0000 7f6052cf5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f604c0f9260 0x7f604c10e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.490+0000 7f6052cf5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c10ebb0 con 0x7f604c0f9260 2024-04-12T05:16:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.490+0000 7f6052cf5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c1081d0 con 0x7f604c0f8900 2024-04-12T05:16:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.490+0000 7f604bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f604c0f9260 0x7f604c10e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.490+0000 7f604bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f604c0f9260 0x7f604c10e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54178/0 (socket says 172.21.15.113:54178) 2024-04-12T05:16:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.490+0000 7f604bfff700 1 -- 172.21.15.113:0/868488521 learned_addr learned my addr 172.21.15.113:0/868488521 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.490+0000 7f6050a91700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f604c0f8900 0x7f604c10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.491+0000 7f604bfff700 1 -- 172.21.15.113:0/868488521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f604c0f8900 msgr2=0x7f604c10e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:19.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.491+0000 7f604bfff700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f604c0f8900 0x7f604c10e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:19.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.491+0000 7f604bfff700 1 -- 172.21.15.113:0/868488521 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6038005040 con 0x7f604c0f9260 2024-04-12T05:16:19.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.491+0000 7f6050a91700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f604c0f8900 0x7f604c10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:16:19.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.491+0000 7f604bfff700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f604c0f9260 0x7f604c10e520 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f6038009310 tx=0x7f6038009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:19.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.491+0000 7f6049ffb700 1 -- 172.21.15.113:0/868488521 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f603800e030 con 0x7f604c0f9260 2024-04-12T05:16:19.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.491+0000 7f6049ffb700 1 -- 172.21.15.113:0/868488521 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6038007e90 con 0x7f604c0f9260 2024-04-12T05:16:19.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.491+0000 7f6052cf5700 1 -- 172.21.15.113:0/868488521 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c108420 con 0x7f604c0f9260 2024-04-12T05:16:19.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.491+0000 7f6049ffb700 1 -- 172.21.15.113:0/868488521 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6038018070 con 0x7f604c0f9260 2024-04-12T05:16:19.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.491+0000 7f6052cf5700 1 -- 172.21.15.113:0/868488521 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c108900 con 0x7f604c0f9260 2024-04-12T05:16:19.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.492+0000 7f6052cf5700 1 -- 172.21.15.113:0/868488521 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c05f590 con 0x7f604c0f9260 2024-04-12T05:16:19.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.492+0000 7f6049ffb700 1 -- 172.21.15.113:0/868488521 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f603801d420 con 0x7f604c0f9260 2024-04-12T05:16:19.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.493+0000 7f6049ffb700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f603c06caa0 0x7f603c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:19.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.493+0000 7f6050a91700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f603c06caa0 0x7f603c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:19.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.493+0000 7f6049ffb700 1 -- 172.21.15.113:0/868488521 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f603808d230 con 0x7f604c0f9260 2024-04-12T05:16:19.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.493+0000 7f6050a91700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f603c06caa0 0x7f603c06ef60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6040009990 tx=0x7f6040008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:19.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.498+0000 7f6049ffb700 1 -- 172.21.15.113:0/868488521 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f603805ae40 con 0x7f604c0f9260 2024-04-12T05:16:19.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:19 smithi113 bash[131123]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-12T05:16:19.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:19 smithi113 bash[131123]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-12T05:16:19.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:19 smithi113 bash[131123]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-12T05:16:19.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:19 smithi113 bash[131123]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-12T05:16:19.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:19 smithi113 bash[131123]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-12T05:16:19.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:19 smithi113 bash[131123]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-12T05:16:19.674 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:19 smithi113 bash[131123]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-12T05:16:19.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.710+0000 7f6052cf5700 1 -- 172.21.15.113:0/868488521 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f604c109720 con 0x7f603c06caa0 2024-04-12T05:16:19.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.711+0000 7f6049ffb700 1 -- 172.21.15.113:0/868488521 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f604c109720 con 0x7f603c06caa0 2024-04-12T05:16:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.713+0000 7f6052cf5700 1 -- 172.21.15.113:0/868488521 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f603c06caa0 msgr2=0x7f603c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.713+0000 7f6052cf5700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f603c06caa0 0x7f603c06ef60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6040009990 tx=0x7f6040008040).stop 2024-04-12T05:16:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.713+0000 7f6052cf5700 1 -- 172.21.15.113:0/868488521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f604c0f9260 msgr2=0x7f604c10e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.713+0000 7f6052cf5700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f604c0f9260 0x7f604c10e520 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f6038009310 tx=0x7f6038009cb0).stop 2024-04-12T05:16:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.713+0000 7f6052cf5700 1 -- 172.21.15.113:0/868488521 shutdown_connections 2024-04-12T05:16:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.714+0000 7f6052cf5700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f603c06caa0 0x7f603c06ef60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.714+0000 7f6052cf5700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f604c0f8900 0x7f604c10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.714+0000 7f6052cf5700 1 --2- 172.21.15.113:0/868488521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f604c0f9260 0x7f604c10e520 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.714+0000 7f6052cf5700 1 -- 172.21.15.113:0/868488521 >> 172.21.15.113:0/868488521 conn(0x7f604c070590 msgr2=0x7f604c0fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.714+0000 7f6052cf5700 1 -- 172.21.15.113:0/868488521 shutdown_connections 2024-04-12T05:16:19.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.714+0000 7f6052cf5700 1 -- 172.21.15.113:0/868488521 wait complete. 2024-04-12T05:16:19.726 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:16:19.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.814+0000 7f143fe05700 1 -- 172.21.15.113:0/3812536844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14380fd150 msgr2=0x7f14380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:19.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.814+0000 7f143fe05700 1 --2- 172.21.15.113:0/3812536844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14380fd150 0x7f14380f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f14340047f0 tx=0x7f1434009cb0).stop 2024-04-12T05:16:19.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.814+0000 7f143fe05700 1 -- 172.21.15.113:0/3812536844 shutdown_connections 2024-04-12T05:16:19.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.814+0000 7f143fe05700 1 --2- 172.21.15.113:0/3812536844 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14380fd150 0x7f14380f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:19.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.814+0000 7f143fe05700 1 --2- 172.21.15.113:0/3812536844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14380fc7f0 0x7f14380fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:19.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.814+0000 7f143fe05700 1 -- 172.21.15.113:0/3812536844 >> 172.21.15.113:0/3812536844 conn(0x7f1438070590 msgr2=0x7f14380f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:19.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.814+0000 7f143fe05700 1 -- 172.21.15.113:0/3812536844 shutdown_connections 2024-04-12T05:16:19.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.814+0000 7f143fe05700 1 -- 172.21.15.113:0/3812536844 wait complete. 2024-04-12T05:16:19.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.814+0000 7f143fe05700 1 Processor -- start 2024-04-12T05:16:19.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143fe05700 1 -- start start 2024-04-12T05:16:19.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143fe05700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14380fc7f0 0x7f14381090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:19.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143fe05700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14380fd150 0x7f14381095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:19.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143fe05700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1438109aa0 con 0x7f14380fd150 2024-04-12T05:16:19.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143fe05700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1438109be0 con 0x7f14380fc7f0 2024-04-12T05:16:19.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143d3a0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14380fd150 0x7f14381095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:19.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143d3a0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14380fd150 0x7f14381095b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54192/0 (socket says 172.21.15.113:54192) 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143d3a0700 1 -- 172.21.15.113:0/647187108 learned_addr learned my addr 172.21.15.113:0/647187108 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143dba1700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14380fc7f0 0x7f14381090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143d3a0700 1 -- 172.21.15.113:0/647187108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14380fc7f0 msgr2=0x7f14381090c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143d3a0700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14380fc7f0 0x7f14381090c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143d3a0700 1 -- 172.21.15.113:0/647187108 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1434005040 con 0x7f14380fd150 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143dba1700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14380fc7f0 0x7f14381090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.815+0000 7f143d3a0700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14380fd150 0x7f14381095b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f14340099f0 tx=0x7f1434009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.816+0000 7f142effd700 1 -- 172.21.15.113:0/647187108 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f143400e030 con 0x7f14380fd150 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.816+0000 7f143fe05700 1 -- 172.21.15.113:0/647187108 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1438109e30 con 0x7f14380fd150 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.816+0000 7f142effd700 1 -- 172.21.15.113:0/647187108 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1434007e90 con 0x7f14380fd150 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.816+0000 7f142effd700 1 -- 172.21.15.113:0/647187108 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1434018070 con 0x7f14380fd150 2024-04-12T05:16:19.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.816+0000 7f143fe05700 1 -- 172.21.15.113:0/647187108 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1438112420 con 0x7f14380fd150 2024-04-12T05:16:19.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.816+0000 7f143fe05700 1 -- 172.21.15.113:0/647187108 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143805f590 con 0x7f14380fd150 2024-04-12T05:16:19.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.817+0000 7f142effd700 1 -- 172.21.15.113:0/647187108 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f143401d420 con 0x7f14380fd150 2024-04-12T05:16:19.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.817+0000 7f142effd700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f142406caf0 0x7f142406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:19.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.817+0000 7f142effd700 1 -- 172.21.15.113:0/647187108 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f143408d250 con 0x7f14380fd150 2024-04-12T05:16:19.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.817+0000 7f143dba1700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f142406caf0 0x7f142406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:19.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.818+0000 7f143dba1700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f142406caf0 0x7f142406efb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f14280041d0 tx=0x7f1428009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:19.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:19.822+0000 7f142effd700 1 -- 172.21.15.113:0/647187108 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1434057310 con 0x7f14380fd150 2024-04-12T05:16:20.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.030+0000 7f143fe05700 1 -- 172.21.15.113:0/647187108 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1438059120 con 0x7f142406caf0 2024-04-12T05:16:20.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.031+0000 7f142effd700 1 -- 172.21.15.113:0/647187108 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1438059120 con 0x7f142406caf0 2024-04-12T05:16:20.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.034+0000 7f143fe05700 1 -- 172.21.15.113:0/647187108 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f142406caf0 msgr2=0x7f142406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.034+0000 7f143fe05700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f142406caf0 0x7f142406efb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f14280041d0 tx=0x7f1428009040).stop 2024-04-12T05:16:20.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.034+0000 7f143fe05700 1 -- 172.21.15.113:0/647187108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14380fd150 msgr2=0x7f14381095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.034+0000 7f143fe05700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14380fd150 0x7f14381095b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f14340099f0 tx=0x7f1434009cb0).stop 2024-04-12T05:16:20.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.034+0000 7f143fe05700 1 -- 172.21.15.113:0/647187108 shutdown_connections 2024-04-12T05:16:20.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.034+0000 7f143fe05700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f142406caf0 0x7f142406efb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.034+0000 7f143fe05700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14380fc7f0 0x7f14381090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.034+0000 7f143fe05700 1 --2- 172.21.15.113:0/647187108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14380fd150 0x7f14381095b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.034+0000 7f143fe05700 1 -- 172.21.15.113:0/647187108 >> 172.21.15.113:0/647187108 conn(0x7f1438070590 msgr2=0x7f14380fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:20.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.034+0000 7f143fe05700 1 -- 172.21.15.113:0/647187108 shutdown_connections 2024-04-12T05:16:20.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.034+0000 7f143fe05700 1 -- 172.21.15.113:0/647187108 wait complete. 2024-04-12T05:16:20.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.149+0000 7f6e8bf98700 1 -- 172.21.15.113:0/3546829661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e840f91f0 msgr2=0x7f6e84101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.149+0000 7f6e8bf98700 1 --2- 172.21.15.113:0/3546829661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e840f91f0 0x7f6e84101990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f6e740047f0 tx=0x7f6e74009cb0).stop 2024-04-12T05:16:20.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 -- 172.21.15.113:0/3546829661 shutdown_connections 2024-04-12T05:16:20.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 --2- 172.21.15.113:0/3546829661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e840f91f0 0x7f6e84101990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 --2- 172.21.15.113:0/3546829661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e840f8920 0x7f6e840f8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:16:20.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 -- 172.21.15.113:0/3546829661 >> 172.21.15.113:0/3546829661 conn(0x7f6e84070590 msgr2=0x7f6e840f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:20.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 -- 172.21.15.113:0/3546829661 shutdown_connections 2024-04-12T05:16:20.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 -- 172.21.15.113:0/3546829661 wait complete. 2024-04-12T05:16:20.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 Processor -- start 2024-04-12T05:16:20.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 -- start start 2024-04-12T05:16:20.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e840f8920 0x7f6e84110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:20.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e840f91f0 0x7f6e84110760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:20.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e84110df0 con 0x7f6e840f8920 2024-04-12T05:16:20.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.150+0000 7f6e8bf98700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8410a460 con 0x7f6e840f91f0 2024-04-12T05:16:20.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e89d34700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e840f8920 0x7f6e84110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:20.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e89d34700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e840f8920 0x7f6e84110270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54198/0 (socket says 172.21.15.113:54198) 2024-04-12T05:16:20.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e89d34700 1 -- 172.21.15.113:0/1462694756 learned_addr learned my addr 172.21.15.113:0/1462694756 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:20.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e89d34700 1 -- 172.21.15.113:0/1462694756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e840f91f0 msgr2=0x7f6e84110760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e89533700 1 --2- 172.21.15.113:0/1462694756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e840f91f0 0x7f6e84110760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:20.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e89d34700 1 --2- 172.21.15.113:0/1462694756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e840f91f0 0x7f6e84110760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e89d34700 1 -- 172.21.15.113:0/1462694756 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74005040 con 0x7f6e840f8920 2024-04-12T05:16:20.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e89d34700 1 --2- 172.21.15.113:0/1462694756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e840f8920 0x7f6e84110270 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f6e8000b4b0 tx=0x7f6e800090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:20.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e7affd700 1 -- 172.21.15.113:0/1462694756 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e80002500 con 0x7f6e840f8920 2024-04-12T05:16:20.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e7affd700 1 -- 172.21.15.113:0/1462694756 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e80002b10 con 0x7f6e840f8920 2024-04-12T05:16:20.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e8bf98700 1 -- 172.21.15.113:0/1462694756 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8410a710 con 0x7f6e840f8920 2024-04-12T05:16:20.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e7affd700 1 -- 172.21.15.113:0/1462694756 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e80012cf0 con 0x7f6e840f8920 2024-04-12T05:16:20.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.151+0000 7f6e8bf98700 1 -- 172.21.15.113:0/1462694756 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8410ac50 con 0x7f6e840f8920 2024-04-12T05:16:20.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.152+0000 7f6e8bf98700 1 -- 172.21.15.113:0/1462694756 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8405f590 con 0x7f6e840f8920 2024-04-12T05:16:20.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.153+0000 7f6e7affd700 1 -- 172.21.15.113:0/1462694756 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6e80002660 con 0x7f6e840f8920 2024-04-12T05:16:20.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.153+0000 7f6e7affd700 1 --2- 172.21.15.113:0/1462694756 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6e7006cab0 0x7f6e7006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:20.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.153+0000 7f6e7affd700 1 -- 172.21.15.113:0/1462694756 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f6e8008bd80 con 0x7f6e840f8920 2024-04-12T05:16:20.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.157+0000 7f6e89533700 1 --2- 172.21.15.113:0/1462694756 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6e7006cab0 0x7f6e7006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:20.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.157+0000 7f6e89533700 1 --2- 172.21.15.113:0/1462694756 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6e7006cab0 0x7f6e7006ef70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f6e740043d0 tx=0x7f6e74009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:20.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.158+0000 7f6e7affd700 1 -- 172.21.15.113:0/1462694756 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6e8008f030 con 0x7f6e840f8920 2024-04-12T05:16:20.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.360+0000 7f6e8bf98700 1 -- 172.21.15.113:0/1462694756 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e8410b960 con 0x7f6e7006cab0 2024-04-12T05:16:20.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.370+0000 7f6e7affd700 1 -- 172.21.15.113:0/1462694756 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6e8410b960 con 0x7f6e7006cab0 2024-04-12T05:16:20.372 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:16:20.372 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (4m) 30s ago 15m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:16:20.372 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (94s) 30s ago 15m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:16:20.372 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (89s) 85s ago 14m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:16:20.372 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (3m) 30s ago 15m 89.2M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:16:20.372 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (10m) 30s ago 10m 120M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:16:20.372 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (9m) 30s ago 9m 18.1M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:16:20.372 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (10m) 85s ago 10m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:16:20.372 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (9m) 85s ago 9m 15.3M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (5m) 30s ago 16m 496M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (5m) 85s ago 14m 448M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (2m) 30s ago 16m 44.5M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (2m) 85s ago 14m 33.0M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (4m) 30s ago 15m 16.3M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (4m) 85s ago 14m 17.8M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (70s) 30s ago 13m 93.0M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (35s) 30s ago 13m 11.4M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (12m) 30s ago 12m 201M 1910M 16.2.15-1-g8963e9de 727f386679cb e20a37d7e826 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (12m) 85s ago 12m 264M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (12m) 85s ago 12m 231M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:16:20.373 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (11m) 85s ago 11m 226M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:16:20.374 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (4m) 85s ago 14m 59.8M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:16:20.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.372+0000 7f6e8bf98700 1 -- 172.21.15.113:0/1462694756 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6e7006cab0 msgr2=0x7f6e7006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.372+0000 7f6e8bf98700 1 --2- 172.21.15.113:0/1462694756 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6e7006cab0 0x7f6e7006ef70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f6e740043d0 tx=0x7f6e74009cb0).stop 2024-04-12T05:16:20.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.372+0000 7f6e8bf98700 1 -- 172.21.15.113:0/1462694756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e840f8920 msgr2=0x7f6e84110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.372+0000 7f6e8bf98700 1 --2- 172.21.15.113:0/1462694756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e840f8920 0x7f6e84110270 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f6e8000b4b0 tx=0x7f6e800090f0).stop 2024-04-12T05:16:20.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.372+0000 7f6e8bf98700 1 -- 172.21.15.113:0/1462694756 shutdown_connections 2024-04-12T05:16:20.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.372+0000 7f6e8bf98700 1 --2- 172.21.15.113:0/1462694756 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6e7006cab0 0x7f6e7006ef70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.372+0000 7f6e8bf98700 1 --2- 172.21.15.113:0/1462694756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e840f8920 0x7f6e84110270 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.372+0000 7f6e8bf98700 1 --2- 172.21.15.113:0/1462694756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e840f91f0 0x7f6e84110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.372+0000 7f6e8bf98700 1 -- 172.21.15.113:0/1462694756 >> 172.21.15.113:0/1462694756 conn(0x7f6e84070590 msgr2=0x7f6e840f6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:20.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.372+0000 7f6e8bf98700 1 -- 172.21.15.113:0/1462694756 shutdown_connections 2024-04-12T05:16:20.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.372+0000 7f6e8bf98700 1 -- 172.21.15.113:0/1462694756 wait complete. 2024-04-12T05:16:20.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:19 smithi113 ceph-mon[103371]: Health check failed: Degraded data redundancy: 39/324 objects degraded (12.037%), 12 pgs degraded (PG_DEGRADED) 2024-04-12T05:16:20.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:19 smithi192 ceph-mon[85304]: Health check failed: Degraded data redundancy: 39/324 objects degraded (12.037%), 12 pgs degraded (PG_DEGRADED) 2024-04-12T05:16:20.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.475+0000 7fa7cd45d700 1 -- 172.21.15.113:0/3255280092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7c80fe250 msgr2=0x7fa7c80f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.475+0000 7fa7cd45d700 1 --2- 172.21.15.113:0/3255280092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7c80fe250 0x7fa7c80f7a30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fa7b0004730 tx=0x7fa7b0009cb0).stop 2024-04-12T05:16:20.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.475+0000 7fa7cd45d700 1 -- 172.21.15.113:0/3255280092 shutdown_connections 2024-04-12T05:16:20.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.475+0000 7fa7cd45d700 1 --2- 172.21.15.113:0/3255280092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7c80fe250 0x7fa7c80f7a30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.475+0000 7fa7cd45d700 1 --2- 172.21.15.113:0/3255280092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7c80fd8f0 0x7fa7c80fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.475+0000 7fa7cd45d700 1 -- 172.21.15.113:0/3255280092 >> 172.21.15.113:0/3255280092 conn(0x7fa7c80f34b0 msgr2=0x7fa7c80f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:20.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.475+0000 7fa7cd45d700 1 -- 172.21.15.113:0/3255280092 shutdown_connections 2024-04-12T05:16:20.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.475+0000 7fa7cd45d700 1 -- 172.21.15.113:0/3255280092 wait complete. 2024-04-12T05:16:20.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.476+0000 7fa7cd45d700 1 Processor -- start 2024-04-12T05:16:20.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.476+0000 7fa7cd45d700 1 -- start start 2024-04-12T05:16:20.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.476+0000 7fa7cd45d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7c80fd8f0 0x7fa7c810dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:20.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.476+0000 7fa7cd45d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7c80fe250 0x7fa7c810e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:20.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.476+0000 7fa7cd45d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c810eab0 con 0x7fa7c80fe250 2024-04-12T05:16:20.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.476+0000 7fa7cd45d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c810ebf0 con 0x7fa7c80fd8f0 2024-04-12T05:16:20.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7c67fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7c80fe250 0x7fa7c810e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:20.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7c67fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7c80fe250 0x7fa7c810e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33726/0 (socket says 172.21.15.113:33726) 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7c67fc700 1 -- 172.21.15.113:0/1626144320 learned_addr learned my addr 172.21.15.113:0/1626144320 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7c6ffd700 1 --2- 172.21.15.113:0/1626144320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7c80fd8f0 0x7fa7c810dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7c67fc700 1 -- 172.21.15.113:0/1626144320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7c80fd8f0 msgr2=0x7fa7c810dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7c67fc700 1 --2- 172.21.15.113:0/1626144320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7c80fd8f0 0x7fa7c810dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7c67fc700 1 -- 172.21.15.113:0/1626144320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b0005040 con 0x7fa7c80fe250 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7c67fc700 1 --2- 172.21.15.113:0/1626144320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7c80fe250 0x7fa7c810e4b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fa7b00099f0 tx=0x7fa7b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7bffff700 1 -- 172.21.15.113:0/1626144320 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7b0019070 con 0x7fa7c80fe250 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7bffff700 1 -- 172.21.15.113:0/1626144320 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7b0013380 con 0x7fa7c80fe250 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7cd45d700 1 -- 172.21.15.113:0/1626144320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c8108220 con 0x7fa7c80fe250 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7bffff700 1 -- 172.21.15.113:0/1626144320 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7b0014bb0 con 0x7fa7c80fe250 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.477+0000 7fa7cd45d700 1 -- 172.21.15.113:0/1626144320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c8108780 con 0x7fa7c80fe250 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.478+0000 7fa7cd45d700 1 -- 172.21.15.113:0/1626144320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c805fdb0 con 0x7fa7c80fe250 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.479+0000 7fa7bffff700 1 -- 172.21.15.113:0/1626144320 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa7b0014d10 con 0x7fa7c80fe250 2024-04-12T05:16:20.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.479+0000 7fa7bffff700 1 --2- 172.21.15.113:0/1626144320 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa7b406cab0 0x7fa7b406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:20.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.479+0000 7fa7c6ffd700 1 --2- 172.21.15.113:0/1626144320 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa7b406cab0 0x7fa7b406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:20.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.479+0000 7fa7bffff700 1 -- 172.21.15.113:0/1626144320 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fa7b008ca00 con 0x7fa7c80fe250 2024-04-12T05:16:20.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.480+0000 7fa7c6ffd700 1 --2- 172.21.15.113:0/1626144320 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa7b406cab0 0x7fa7b406ef70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa7b8001370 tx=0x7fa7b800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:20.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.484+0000 7fa7bffff700 1 -- 172.21.15.113:0/1626144320 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa7b005a610 con 0x7fa7c80fe250 2024-04-12T05:16:20.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.764+0000 7fa7cd45d700 1 -- 172.21.15.113:0/1626144320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa7c80ffc40 con 0x7fa7c80fe250 2024-04-12T05:16:20.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.764+0000 7fa7bffff700 1 -- 172.21.15.113:0/1626144320 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fa7b005a1a0 con 0x7fa7c80fe250 2024-04-12T05:16:20.767 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:16:20.767 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:16:20.767 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:16:20.767 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:16:20.767 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:16:20.767 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:16:20.767 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:16:20.767 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:16:20.767 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 3, 2024-04-12T05:16:20.767 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:16:20.768 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:16:20.768 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:16:20.768 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:16:20.768 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:16:20.768 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:16:20.768 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 7, 2024-04-12T05:16:20.768 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:16:20.768 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:16:20.768 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:16:20.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.767+0000 7fa7cd45d700 1 -- 172.21.15.113:0/1626144320 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa7b406cab0 msgr2=0x7fa7b406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.767+0000 7fa7cd45d700 1 --2- 172.21.15.113:0/1626144320 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa7b406cab0 0x7fa7b406ef70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa7b8001370 tx=0x7fa7b800b040).stop 2024-04-12T05:16:20.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.767+0000 7fa7cd45d700 1 -- 172.21.15.113:0/1626144320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7c80fe250 msgr2=0x7fa7c810e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.767+0000 7fa7cd45d700 1 --2- 172.21.15.113:0/1626144320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7c80fe250 0x7fa7c810e4b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fa7b00099f0 tx=0x7fa7b000b040).stop 2024-04-12T05:16:20.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.767+0000 7fa7cd45d700 1 -- 172.21.15.113:0/1626144320 shutdown_connections 2024-04-12T05:16:20.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.767+0000 7fa7cd45d700 1 --2- 172.21.15.113:0/1626144320 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa7b406cab0 0x7fa7b406ef70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.767+0000 7fa7cd45d700 1 --2- 172.21.15.113:0/1626144320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa7c80fd8f0 0x7fa7c810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.767+0000 7fa7cd45d700 1 --2- 172.21.15.113:0/1626144320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7c80fe250 0x7fa7c810e4b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.767+0000 7fa7cd45d700 1 -- 172.21.15.113:0/1626144320 >> 172.21.15.113:0/1626144320 conn(0x7fa7c80f34b0 msgr2=0x7fa7c80fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:20.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.767+0000 7fa7cd45d700 1 -- 172.21.15.113:0/1626144320 shutdown_connections 2024-04-12T05:16:20.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.767+0000 7fa7cd45d700 1 -- 172.21.15.113:0/1626144320 wait complete. 2024-04-12T05:16:20.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.873+0000 7f0b88110700 1 -- 172.21.15.113:0/3322271864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b800f9200 msgr2=0x7f0b80101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.873+0000 7f0b88110700 1 --2- 172.21.15.113:0/3322271864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b800f9200 0x7f0b80101910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f0b700047f0 tx=0x7f0b70009cb0).stop 2024-04-12T05:16:20.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.873+0000 7f0b88110700 1 -- 172.21.15.113:0/3322271864 shutdown_connections 2024-04-12T05:16:20.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.873+0000 7f0b88110700 1 --2- 172.21.15.113:0/3322271864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b800f9200 0x7f0b80101910 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.873+0000 7f0b88110700 1 --2- 172.21.15.113:0/3322271864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b800f88a0 0x7f0b800f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.873+0000 7f0b88110700 1 -- 172.21.15.113:0/3322271864 >> 172.21.15.113:0/3322271864 conn(0x7f0b80070590 msgr2=0x7f0b800f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:20.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.873+0000 7f0b88110700 1 -- 172.21.15.113:0/3322271864 shutdown_connections 2024-04-12T05:16:20.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.873+0000 7f0b88110700 1 -- 172.21.15.113:0/3322271864 wait complete. 2024-04-12T05:16:20.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b88110700 1 Processor -- start 2024-04-12T05:16:20.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b88110700 1 -- start start 2024-04-12T05:16:20.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b88110700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b800f88a0 0x7f0b800f6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:20.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b88110700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b800f9200 0x7f0b800f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:20.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b88110700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b800f7230 con 0x7f0b800f9200 2024-04-12T05:16:20.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b88110700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b800f7370 con 0x7f0b800f88a0 2024-04-12T05:16:20.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b856ab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b800f9200 0x7f0b800f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:20.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b856ab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b800f9200 0x7f0b800f6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33756/0 (socket says 172.21.15.113:33756) 2024-04-12T05:16:20.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b856ab700 1 -- 172.21.15.113:0/3128275193 learned_addr learned my addr 172.21.15.113:0/3128275193 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:20.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b85eac700 1 --2- 172.21.15.113:0/3128275193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b800f88a0 0x7f0b800f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:20.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b856ab700 1 -- 172.21.15.113:0/3128275193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b800f88a0 msgr2=0x7f0b800f6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:20.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b856ab700 1 --2- 172.21.15.113:0/3128275193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b800f88a0 0x7f0b800f6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:20.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b856ab700 1 -- 172.21.15.113:0/3128275193 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b70005040 con 0x7f0b800f9200 2024-04-12T05:16:20.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.874+0000 7f0b856ab700 1 --2- 172.21.15.113:0/3128275193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b800f9200 0x7f0b800f6d40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0b70009380 tx=0x7f0b70009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:20.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.875+0000 7f0b76ffd700 1 -- 172.21.15.113:0/3128275193 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b70018070 con 0x7f0b800f9200 2024-04-12T05:16:20.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.875+0000 7f0b88110700 1 -- 172.21.15.113:0/3128275193 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b800f75c0 con 0x7f0b800f9200 2024-04-12T05:16:20.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.875+0000 7f0b76ffd700 1 -- 172.21.15.113:0/3128275193 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b70007e90 con 0x7f0b800f9200 2024-04-12T05:16:20.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.875+0000 7f0b76ffd700 1 -- 172.21.15.113:0/3128275193 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b700123c0 con 0x7f0b800f9200 2024-04-12T05:16:20.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.875+0000 7f0b88110700 1 -- 172.21.15.113:0/3128275193 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b801147d0 con 0x7f0b800f9200 2024-04-12T05:16:20.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.876+0000 7f0b88110700 1 -- 172.21.15.113:0/3128275193 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8005f590 con 0x7f0b800f9200 2024-04-12T05:16:20.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.876+0000 7f0b76ffd700 1 -- 172.21.15.113:0/3128275193 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0b7001d420 con 0x7f0b800f9200 2024-04-12T05:16:20.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.877+0000 7f0b76ffd700 1 --2- 172.21.15.113:0/3128275193 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b6c06caa0 0x7f0b6c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:20.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.877+0000 7f0b76ffd700 1 -- 172.21.15.113:0/3128275193 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f0b7008c860 con 0x7f0b800f9200 2024-04-12T05:16:20.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.877+0000 7f0b85eac700 1 --2- 172.21.15.113:0/3128275193 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b6c06caa0 0x7f0b6c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:20.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.877+0000 7f0b85eac700 1 --2- 172.21.15.113:0/3128275193 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b6c06caa0 0x7f0b6c06ef60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0b7c004ee0 tx=0x7f0b7c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:20.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:20.881+0000 7f0b76ffd700 1 -- 172.21.15.113:0/3128275193 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0b700569e0 con 0x7f0b800f9200 2024-04-12T05:16:20.924 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:20 smithi113 ceph-osd[131538]: -- 172.21.15.113:0/3146777539 <== mon.0 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c62b9e31e0 con 0x55c62b9b3000 2024-04-12T05:16:20.924 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:20 smithi113 bash[131458]: 70ada4b6cb092e5081ff6e4be1ffaaac285a6d5ea113f31643aacd78b0f5cd12 2024-04-12T05:16:20.924 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:20 smithi113 systemd[1]: Started Ceph osd.2 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:16:21.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.158+0000 7f0b88110700 1 -- 172.21.15.113:0/3128275193 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0b80047540 con 0x7f0b800f9200 2024-04-12T05:16:21.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.158+0000 7f0b76ffd700 1 -- 172.21.15.113:0/3128275193 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1859 (secure 0 0 0) 0x7f0b7005a000 con 0x7f0b800f9200 2024-04-12T05:16:21.161 INFO:teuthology.orchestra.run.smithi113.stdout:e18 2024-04-12T05:16:21.161 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:16:21.161 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:16:21.161 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:16:21.161 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:16:21.161 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:16:21.161 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:16:21.161 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 18 2024-04-12T05:16:21.161 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:16:21.161 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:16:21.162 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=14426,1=24261} 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:16:21.163 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:16:21.164 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:16:21.164 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:16:21.164 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:16:21.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.161+0000 7f0b88110700 1 -- 172.21.15.113:0/3128275193 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b6c06caa0 msgr2=0x7f0b6c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:21.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.161+0000 7f0b88110700 1 --2- 172.21.15.113:0/3128275193 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b6c06caa0 0x7f0b6c06ef60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0b7c004ee0 tx=0x7f0b7c00b040).stop 2024-04-12T05:16:21.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.161+0000 7f0b88110700 1 -- 172.21.15.113:0/3128275193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b800f9200 msgr2=0x7f0b800f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:21.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.161+0000 7f0b88110700 1 --2- 172.21.15.113:0/3128275193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b800f9200 0x7f0b800f6d40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0b70009380 tx=0x7f0b70009cb0).stop 2024-04-12T05:16:21.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.161+0000 7f0b88110700 1 -- 172.21.15.113:0/3128275193 shutdown_connections 2024-04-12T05:16:21.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.161+0000 7f0b88110700 1 --2- 172.21.15.113:0/3128275193 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b6c06caa0 0x7f0b6c06ef60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.161+0000 7f0b88110700 1 --2- 172.21.15.113:0/3128275193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b800f88a0 0x7f0b800f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.161+0000 7f0b88110700 1 --2- 172.21.15.113:0/3128275193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b800f9200 0x7f0b800f6d40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.161+0000 7f0b88110700 1 -- 172.21.15.113:0/3128275193 >> 172.21.15.113:0/3128275193 conn(0x7f0b80070590 msgr2=0x7f0b80100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:21.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.161+0000 7f0b88110700 1 -- 172.21.15.113:0/3128275193 shutdown_connections 2024-04-12T05:16:21.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.161+0000 7f0b88110700 1 -- 172.21.15.113:0/3128275193 wait complete. 2024-04-12T05:16:21.165 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 18 2024-04-12T05:16:21.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.260+0000 7fc8f56e1700 1 -- 172.21.15.113:0/1663667426 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f00f8890 msgr2=0x7fc8f00f8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:21.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.260+0000 7fc8f56e1700 1 --2- 172.21.15.113:0/1663667426 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f00f8890 0x7fc8f00f8c70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fc8e00047f0 tx=0x7fc8e0009cb0).stop 2024-04-12T05:16:21.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.260+0000 7fc8f56e1700 1 -- 172.21.15.113:0/1663667426 shutdown_connections 2024-04-12T05:16:21.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.260+0000 7fc8f56e1700 1 --2- 172.21.15.113:0/1663667426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f00f91f0 0x7fc8f0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.260+0000 7fc8f56e1700 1 --2- 172.21.15.113:0/1663667426 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f00f8890 0x7fc8f00f8c70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.260+0000 7fc8f56e1700 1 -- 172.21.15.113:0/1663667426 >> 172.21.15.113:0/1663667426 conn(0x7fc8f00f23c0 msgr2=0x7fc8f00f47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:21.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.261+0000 7fc8f56e1700 1 -- 172.21.15.113:0/1663667426 shutdown_connections 2024-04-12T05:16:21.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.261+0000 7fc8f56e1700 1 -- 172.21.15.113:0/1663667426 wait complete. 2024-04-12T05:16:21.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.261+0000 7fc8f56e1700 1 Processor -- start 2024-04-12T05:16:21.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.261+0000 7fc8f56e1700 1 -- start start 2024-04-12T05:16:21.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.261+0000 7fc8f56e1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f00f8890 0x7fc8f010e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:21.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.261+0000 7fc8f56e1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f00f91f0 0x7fc8f010e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:21.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.261+0000 7fc8f56e1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f010eb50 con 0x7fc8f00f8890 2024-04-12T05:16:21.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.261+0000 7fc8f56e1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f010ec90 con 0x7fc8f00f91f0 2024-04-12T05:16:21.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8eeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f00f8890 0x7fc8f010e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:21.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8eeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f00f8890 0x7fc8f010e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33764/0 (socket says 172.21.15.113:33764) 2024-04-12T05:16:21.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8eeffd700 1 -- 172.21.15.113:0/2440514358 learned_addr learned my addr 172.21.15.113:0/2440514358 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:21.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8ee7fc700 1 --2- 172.21.15.113:0/2440514358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f00f91f0 0x7fc8f010e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:21.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8eeffd700 1 -- 172.21.15.113:0/2440514358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f00f91f0 msgr2=0x7fc8f010e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:21.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8eeffd700 1 --2- 172.21.15.113:0/2440514358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f00f91f0 0x7fc8f010e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8eeffd700 1 -- 172.21.15.113:0/2440514358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e0005040 con 0x7fc8f00f8890 2024-04-12T05:16:21.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8eeffd700 1 --2- 172.21.15.113:0/2440514358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f00f8890 0x7fc8f010e060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc8e0005010 tx=0x7fc8e0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:21.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8d7fff700 1 -- 172.21.15.113:0/2440514358 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e000e030 con 0x7fc8f00f8890 2024-04-12T05:16:21.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8d7fff700 1 -- 172.21.15.113:0/2440514358 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8e0007e90 con 0x7fc8f00f8890 2024-04-12T05:16:21.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8f56e1700 1 -- 172.21.15.113:0/2440514358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f01082c0 con 0x7fc8f00f8890 2024-04-12T05:16:21.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8d7fff700 1 -- 172.21.15.113:0/2440514358 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e0018070 con 0x7fc8f00f8890 2024-04-12T05:16:21.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.262+0000 7fc8f56e1700 1 -- 172.21.15.113:0/2440514358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f01087a0 con 0x7fc8f00f8890 2024-04-12T05:16:21.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.263+0000 7fc8f56e1700 1 -- 172.21.15.113:0/2440514358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f0060370 con 0x7fc8f00f8890 2024-04-12T05:16:21.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.264+0000 7fc8d7fff700 1 -- 172.21.15.113:0/2440514358 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc8e001d670 con 0x7fc8f00f8890 2024-04-12T05:16:21.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.264+0000 7fc8d7fff700 1 --2- 172.21.15.113:0/2440514358 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc8d806caa0 0x7fc8d806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:21.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.264+0000 7fc8d7fff700 1 -- 172.21.15.113:0/2440514358 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fc8e008cf70 con 0x7fc8f00f8890 2024-04-12T05:16:21.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.264+0000 7fc8ee7fc700 1 --2- 172.21.15.113:0/2440514358 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc8d806caa0 0x7fc8d806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:21.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.264+0000 7fc8ee7fc700 1 --2- 172.21.15.113:0/2440514358 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc8d806caa0 0x7fc8d806ef60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fc8f01099a0 tx=0x7fc8e400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:21.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.268+0000 7fc8d7fff700 1 -- 172.21.15.113:0/2440514358 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc8e00c02c0 con 0x7fc8f00f8890 2024-04-12T05:16:21.424 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:21 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[131495]: 2024-04-12T05:16:21.124+0000 7f5241f587c0 -1 Falling back to public interface 2024-04-12T05:16:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:20 smithi113 ceph-mon[103371]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:20 smithi113 ceph-mon[103371]: pgmap v89: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:21.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:20 smithi113 ceph-mon[103371]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:21.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:21.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:21.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:16:21.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:20 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1626144320' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:16:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:20 smithi192 ceph-mon[85304]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:20 smithi192 ceph-mon[85304]: pgmap v89: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:20 smithi192 ceph-mon[85304]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:16:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:20 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1626144320' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:16:21.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.483+0000 7fc8f56e1700 1 -- 172.21.15.113:0/2440514358 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8f00fee20 con 0x7fc8d806caa0 2024-04-12T05:16:21.486 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.484+0000 7fc8d7fff700 1 -- 172.21.15.113:0/2440514358 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fc8f00fee20 con 0x7fc8d806caa0 2024-04-12T05:16:21.486 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:16:21.486 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:16:21.486 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:16:21.486 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:16:21.486 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:16:21.486 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:16:21.486 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:16:21.486 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:16:21.487 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:16:21.487 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "8/21 daemons upgraded", 2024-04-12T05:16:21.487 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:16:21.487 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:16:21.487 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:16:21.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.486+0000 7fc8f56e1700 1 -- 172.21.15.113:0/2440514358 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc8d806caa0 msgr2=0x7fc8d806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:21.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.486+0000 7fc8f56e1700 1 --2- 172.21.15.113:0/2440514358 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc8d806caa0 0x7fc8d806ef60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fc8f01099a0 tx=0x7fc8e400b040).stop 2024-04-12T05:16:21.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.486+0000 7fc8f56e1700 1 -- 172.21.15.113:0/2440514358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f00f8890 msgr2=0x7fc8f010e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:21.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.486+0000 7fc8f56e1700 1 --2- 172.21.15.113:0/2440514358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f00f8890 0x7fc8f010e060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc8e0005010 tx=0x7fc8e0009cb0).stop 2024-04-12T05:16:21.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.486+0000 7fc8f56e1700 1 -- 172.21.15.113:0/2440514358 shutdown_connections 2024-04-12T05:16:21.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.486+0000 7fc8f56e1700 1 --2- 172.21.15.113:0/2440514358 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc8d806caa0 0x7fc8d806ef60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.486+0000 7fc8f56e1700 1 --2- 172.21.15.113:0/2440514358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f00f8890 0x7fc8f010e060 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.486+0000 7fc8f56e1700 1 --2- 172.21.15.113:0/2440514358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f00f91f0 0x7fc8f010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.486+0000 7fc8f56e1700 1 -- 172.21.15.113:0/2440514358 >> 172.21.15.113:0/2440514358 conn(0x7fc8f00f23c0 msgr2=0x7fc8f00fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:21.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.486+0000 7fc8f56e1700 1 -- 172.21.15.113:0/2440514358 shutdown_connections 2024-04-12T05:16:21.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.486+0000 7fc8f56e1700 1 -- 172.21.15.113:0/2440514358 wait complete. 2024-04-12T05:16:21.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.597+0000 7f79a33eb700 1 -- 172.21.15.113:0/285706335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f799c0fc7f0 msgr2=0x7f799c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:21.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.597+0000 7f79a33eb700 1 --2- 172.21.15.113:0/285706335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f799c0fc7f0 0x7f799c0fcbd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f79940047f0 tx=0x7f7994009cb0).stop 2024-04-12T05:16:21.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.597+0000 7f79a33eb700 1 -- 172.21.15.113:0/285706335 shutdown_connections 2024-04-12T05:16:21.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.597+0000 7f79a33eb700 1 --2- 172.21.15.113:0/285706335 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f799c0fd150 0x7f799c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.597+0000 7f79a33eb700 1 --2- 172.21.15.113:0/285706335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f799c0fc7f0 0x7f799c0fcbd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.597+0000 7f79a33eb700 1 -- 172.21.15.113:0/285706335 >> 172.21.15.113:0/285706335 conn(0x7f799c070590 msgr2=0x7f799c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:21.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.597+0000 7f79a33eb700 1 -- 172.21.15.113:0/285706335 shutdown_connections 2024-04-12T05:16:21.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.597+0000 7f79a33eb700 1 -- 172.21.15.113:0/285706335 wait complete. 2024-04-12T05:16:21.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.598+0000 7f79a33eb700 1 Processor -- start 2024-04-12T05:16:21.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.598+0000 7f79a33eb700 1 -- start start 2024-04-12T05:16:21.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.598+0000 7f79a33eb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f799c0fc7f0 0x7f799c1101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:21.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.598+0000 7f79a33eb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f799c0fd150 0x7f799c1106c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:21.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.598+0000 7f79a33eb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c110cc0 con 0x7f799c0fd150 2024-04-12T05:16:21.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.598+0000 7f79a33eb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c110e00 con 0x7f799c0fc7f0 2024-04-12T05:16:21.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.598+0000 7f79a0986700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f799c0fd150 0x7f799c1106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:21.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.598+0000 7f79a0986700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f799c0fd150 0x7f799c1106c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33784/0 (socket says 172.21.15.113:33784) 2024-04-12T05:16:21.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.598+0000 7f79a0986700 1 -- 172.21.15.113:0/1620886042 learned_addr learned my addr 172.21.15.113:0/1620886042 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:21.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.598+0000 7f79a1187700 1 --2- 172.21.15.113:0/1620886042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f799c0fc7f0 0x7f799c1101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:21.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.599+0000 7f79a0986700 1 -- 172.21.15.113:0/1620886042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f799c0fc7f0 msgr2=0x7f799c1101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:21.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.599+0000 7f79a0986700 1 --2- 172.21.15.113:0/1620886042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f799c0fc7f0 0x7f799c1101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.599+0000 7f79a0986700 1 -- 172.21.15.113:0/1620886042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7994005040 con 0x7f799c0fd150 2024-04-12T05:16:21.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.599+0000 7f79a0986700 1 --2- 172.21.15.113:0/1620886042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f799c0fd150 0x7f799c1106c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f7990002780 tx=0x7f7990009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:21.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.599+0000 7f799a7fc700 1 -- 172.21.15.113:0/1620886042 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7990007b40 con 0x7f799c0fd150 2024-04-12T05:16:21.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.599+0000 7f799a7fc700 1 -- 172.21.15.113:0/1620886042 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f799000f040 con 0x7f799c0fd150 2024-04-12T05:16:21.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.599+0000 7f79a33eb700 1 -- 172.21.15.113:0/1620886042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799c10a490 con 0x7f799c0fd150 2024-04-12T05:16:21.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.599+0000 7f799a7fc700 1 -- 172.21.15.113:0/1620886042 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7990014780 con 0x7f799c0fd150 2024-04-12T05:16:21.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.599+0000 7f79a33eb700 1 -- 172.21.15.113:0/1620886042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799c10aa50 con 0x7f799c0fd150 2024-04-12T05:16:21.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.600+0000 7f79a33eb700 1 -- 172.21.15.113:0/1620886042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799c05f590 con 0x7f799c0fd150 2024-04-12T05:16:21.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.600+0000 7f799a7fc700 1 -- 172.21.15.113:0/1620886042 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7990007ca0 con 0x7f799c0fd150 2024-04-12T05:16:21.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.601+0000 7f799a7fc700 1 --2- 172.21.15.113:0/1620886042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f798406caa0 0x7f798406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:21.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.601+0000 7f799a7fc700 1 -- 172.21.15.113:0/1620886042 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f799008bb00 con 0x7f799c0fd150 2024-04-12T05:16:21.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.601+0000 7f79a1187700 1 --2- 172.21.15.113:0/1620886042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f798406caa0 0x7f798406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:21.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.601+0000 7f79a1187700 1 --2- 172.21.15.113:0/1620886042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f798406caa0 0x7f798406ef60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7994004f10 tx=0x7f799400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:21.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.605+0000 7f799a7fc700 1 -- 172.21.15.113:0/1620886042 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7990059710 con 0x7f799c0fd150 2024-04-12T05:16:21.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.879+0000 7f79a33eb700 1 -- 172.21.15.113:0/1620886042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f799c047540 con 0x7f799c0fd150 2024-04-12T05:16:21.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.879+0000 7f799a7fc700 1 -- 172.21.15.113:0/1620886042 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+941 (secure 0 0 0) 0x7f79900592a0 con 0x7f799c0fd150 2024-04-12T05:16:21.882 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 39/324 objects degraded (12.037%), 12 pgs degraded 2024-04-12T05:16:21.882 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T05:16:21.882 INFO:teuthology.orchestra.run.smithi113.stdout: osd.2 (root=default,host=smithi113) is down 2024-04-12T05:16:21.882 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 39/324 objects degraded (12.037%), 12 pgs degraded 2024-04-12T05:16:21.882 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1 is active+undersized+degraded, acting [1,0] 2024-04-12T05:16:21.882 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.3 is active+undersized+degraded, acting [5,1] 2024-04-12T05:16:21.882 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.7 is active+undersized+degraded, acting [3,4] 2024-04-12T05:16:21.882 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-04-12T05:16:21.883 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.e is active+undersized+degraded, acting [0,3] 2024-04-12T05:16:21.883 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-04-12T05:16:21.883 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-04-12T05:16:21.883 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.16 is active+undersized+degraded, acting [5,3] 2024-04-12T05:16:21.883 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.17 is active+undersized+degraded, acting [5,1] 2024-04-12T05:16:21.883 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.19 is active+undersized+degraded, acting [0,4] 2024-04-12T05:16:21.883 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1c is active+undersized+degraded, acting [4,5] 2024-04-12T05:16:21.883 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-04-12T05:16:21.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.881+0000 7f79a33eb700 1 -- 172.21.15.113:0/1620886042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f798406caa0 msgr2=0x7f798406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:21.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.881+0000 7f79a33eb700 1 --2- 172.21.15.113:0/1620886042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f798406caa0 0x7f798406ef60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7994004f10 tx=0x7f799400b040).stop 2024-04-12T05:16:21.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.881+0000 7f79a33eb700 1 -- 172.21.15.113:0/1620886042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f799c0fd150 msgr2=0x7f799c1106c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:21.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.881+0000 7f79a33eb700 1 --2- 172.21.15.113:0/1620886042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f799c0fd150 0x7f799c1106c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f7990002780 tx=0x7f7990009ee0).stop 2024-04-12T05:16:21.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.881+0000 7f79a33eb700 1 -- 172.21.15.113:0/1620886042 shutdown_connections 2024-04-12T05:16:21.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.881+0000 7f79a33eb700 1 --2- 172.21.15.113:0/1620886042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f798406caa0 0x7f798406ef60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.881+0000 7f79a33eb700 1 --2- 172.21.15.113:0/1620886042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f799c0fc7f0 0x7f799c1101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.881+0000 7f79a33eb700 1 --2- 172.21.15.113:0/1620886042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f799c0fd150 0x7f799c1106c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:21.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.881+0000 7f79a33eb700 1 -- 172.21.15.113:0/1620886042 >> 172.21.15.113:0/1620886042 conn(0x7f799c070590 msgr2=0x7f799c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:21.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.881+0000 7f79a33eb700 1 -- 172.21.15.113:0/1620886042 shutdown_connections 2024-04-12T05:16:21.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:21.881+0000 7f79a33eb700 1 -- 172.21.15.113:0/1620886042 wait complete. 2024-04-12T05:16:22.270 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:21 smithi113 ceph-mon[103371]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:22.270 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:21 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3128275193' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:16:22.270 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:22.271 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:16:22.271 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:21 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1620886042' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:16:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:21 smithi192 ceph-mon[85304]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:21 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3128275193' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:16:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:16:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:21 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1620886042' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:16:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:22 smithi113 ceph-mon[103371]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:22 smithi113 ceph-mon[103371]: pgmap v90: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:22 smithi192 ceph-mon[85304]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:22 smithi192 ceph-mon[85304]: pgmap v90: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:24 smithi113 ceph-mon[103371]: pgmap v91: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:24 smithi192 ceph-mon[85304]: pgmap v91: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:26.903 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:26 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:26.903 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:26 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:26.904 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:26 smithi113 ceph-mon[103371]: pgmap v92: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:26.904 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:26 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:26.904 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:26 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:26.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:26 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:26.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:26 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:26.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:26 smithi192 ceph-mon[85304]: pgmap v92: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:26.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:26 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:26.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:26 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:28 smithi113 ceph-mon[103371]: pgmap v93: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:29.175 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:28 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[131495]: 2024-04-12T05:16:28.978+0000 7f5241f587c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-12T05:16:29.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:28 smithi192 ceph-mon[85304]: pgmap v93: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:29.810 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:29 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[131495]: 2024-04-12T05:16:29.473+0000 7f5241f587c0 -1 osd.2 60 log_to_monitors true 2024-04-12T05:16:30.127 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:29 smithi113 ceph-mon[103371]: from='osd.2 [v2:172.21.15.113:6818/2501657120,v1:172.21.15.113:6819/2501657120]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T05:16:30.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:29 smithi192 ceph-mon[85304]: from='osd.2 [v2:172.21.15.113:6818/2501657120,v1:172.21.15.113:6819/2501657120]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T05:16:30.852 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[131495]: 2024-04-12T05:16:30.550+0000 7f522f3aa700 -1 osd.2 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:16:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: pgmap v94: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='osd.2 [v2:172.21.15.113:6818/2501657120,v1:172.21.15.113:6819/2501657120]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T05:16:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: osdmap e62: 6 total, 5 up, 6 in 2024-04-12T05:16:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='osd.2 [v2:172.21.15.113:6818/2501657120,v1:172.21.15.113:6819/2501657120]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:16:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:31.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:31.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:16:31.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:16:31.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:31.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:16:31.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:16:31.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:16:31.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:16:31.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:16:31.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:30 smithi113 ceph-mon[103371]: from='osd.2 [v2:172.21.15.113:6818/2501657120,v1:172.21.15.113:6819/2501657120]' entity='osd.2' 2024-04-12T05:16:31.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: pgmap v94: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='osd.2 [v2:172.21.15.113:6818/2501657120,v1:172.21.15.113:6819/2501657120]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: osdmap e62: 6 total, 5 up, 6 in 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='osd.2 [v2:172.21.15.113:6818/2501657120,v1:172.21.15.113:6819/2501657120]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:16:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:16:31.200 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:30 smithi192 ceph-mon[85304]: from='osd.2 [v2:172.21.15.113:6818/2501657120,v1:172.21.15.113:6819/2501657120]' entity='osd.2' 2024-04-12T05:16:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:31 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:16:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:31 smithi113 ceph-mon[103371]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-12T05:16:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:31 smithi113 ceph-mon[103371]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:16:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:31 smithi113 ceph-mon[103371]: osd.2 [v2:172.21.15.113:6818/2501657120,v1:172.21.15.113:6819/2501657120] boot 2024-04-12T05:16:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:31 smithi113 ceph-mon[103371]: osdmap e63: 6 total, 6 up, 6 in 2024-04-12T05:16:32.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:31 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:16:32.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:31 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:16:32.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:31 smithi192 ceph-mon[85304]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-12T05:16:32.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:31 smithi192 ceph-mon[85304]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:16:32.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:31 smithi192 ceph-mon[85304]: osd.2 [v2:172.21.15.113:6818/2501657120,v1:172.21.15.113:6819/2501657120] boot 2024-04-12T05:16:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:31 smithi192 ceph-mon[85304]: osdmap e63: 6 total, 6 up, 6 in 2024-04-12T05:16:32.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:31 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T05:16:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:32 smithi113 ceph-mon[103371]: pgmap v97: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:32 smithi113 ceph-mon[103371]: osdmap e64: 6 total, 6 up, 6 in 2024-04-12T05:16:33.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:32 smithi192 ceph-mon[85304]: pgmap v97: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 259 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 39/324 objects degraded (12.037%) 2024-04-12T05:16:33.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:32 smithi192 ceph-mon[85304]: osdmap e64: 6 total, 6 up, 6 in 2024-04-12T05:16:34.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:33 smithi113 ceph-mon[103371]: Health check update: Degraded data redundancy: 13/324 objects degraded (4.012%), 4 pgs degraded (PG_DEGRADED) 2024-04-12T05:16:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:33 smithi192 ceph-mon[85304]: Health check update: Degraded data redundancy: 13/324 objects degraded (4.012%), 4 pgs degraded (PG_DEGRADED) 2024-04-12T05:16:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:34 smithi113 ceph-mon[103371]: pgmap v99: 65 pgs: 4 active+undersized+degraded, 5 active+undersized, 56 active+clean; 259 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 13/324 objects degraded (4.012%) 2024-04-12T05:16:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:34 smithi192 ceph-mon[85304]: pgmap v99: 65 pgs: 4 active+undersized+degraded, 5 active+undersized, 56 active+clean; 259 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 13/324 objects degraded (4.012%) 2024-04-12T05:16:36.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:35 smithi113 ceph-mon[103371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/324 objects degraded (4.012%), 4 pgs degraded) 2024-04-12T05:16:36.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:35 smithi113 ceph-mon[103371]: Cluster is now healthy 2024-04-12T05:16:36.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:35 smithi192 ceph-mon[85304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/324 objects degraded (4.012%), 4 pgs degraded) 2024-04-12T05:16:36.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:35 smithi192 ceph-mon[85304]: Cluster is now healthy 2024-04-12T05:16:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:36 smithi113 ceph-mon[103371]: pgmap v100: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:16:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:36 smithi192 ceph-mon[85304]: pgmap v100: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:16:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:38 smithi113 ceph-mon[103371]: pgmap v101: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:38 smithi192 ceph-mon[85304]: pgmap v101: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:40 smithi113 ceph-mon[103371]: pgmap v102: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:40 smithi192 ceph-mon[85304]: pgmap v102: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:43.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:42 smithi113 ceph-mon[103371]: pgmap v103: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:43.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:43.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:42 smithi192 ceph-mon[85304]: pgmap v103: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:43.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:44 smithi113 ceph-mon[103371]: pgmap v104: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:45.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:44 smithi192 ceph-mon[85304]: pgmap v104: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:46.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:45 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:16:46.175 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:45 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:16:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:46 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:16:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:46 smithi113 ceph-mon[103371]: Upgrade: osd.3 is safe to restart 2024-04-12T05:16:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:46 smithi113 ceph-mon[103371]: pgmap v105: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:46 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T05:16:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:46 smithi192 ceph-mon[85304]: Upgrade: osd.3 is safe to restart 2024-04-12T05:16:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:46 smithi192 ceph-mon[85304]: pgmap v105: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:48.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:48 smithi113 ceph-mon[103371]: Upgrade: Updating osd.3 2024-04-12T05:16:48.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:48 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:48.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:48 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:16:48.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:48 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:16:48.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:48 smithi113 ceph-mon[103371]: Deploying daemon osd.3 on smithi192 2024-04-12T05:16:48.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:48 smithi192 ceph-mon[85304]: Upgrade: Updating osd.3 2024-04-12T05:16:48.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:48 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:48.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:48 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T05:16:48.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:48 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:16:48.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:48 smithi192 ceph-mon[85304]: Deploying daemon osd.3 on smithi192 2024-04-12T05:16:49.421 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:49 smithi192 ceph-mon[85304]: pgmap v106: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:49 smithi113 ceph-mon[103371]: pgmap v106: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:50.759 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:50 smithi192 systemd[1]: Stopping Ceph osd.3 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:16:51.010 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[48787]: debug 2024-04-12T05:16:50.756+0000 7f07799dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-12T05:16:51.010 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[48787]: debug 2024-04-12T05:16:50.756+0000 7f07799dc700 -1 osd.3 64 *** Got signal Terminated *** 2024-04-12T05:16:51.010 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:50 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[48787]: debug 2024-04-12T05:16:50.756+0000 7f07799dc700 -1 osd.3 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:16:51.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:51 smithi192 ceph-mon[85304]: pgmap v107: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:51.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:51 smithi192 ceph-mon[85304]: osd.3 marked itself down and dead 2024-04-12T05:16:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:51 smithi113 ceph-mon[103371]: pgmap v107: 65 pgs: 65 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:51 smithi113 ceph-mon[103371]: osd.3 marked itself down and dead 2024-04-12T05:16:51.698 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:51 smithi192 bash[97386]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3 2024-04-12T05:16:51.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.995+0000 7fe4e6c49700 1 -- 172.21.15.113:0/298700548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4e00fe250 msgr2=0x7fe4e00f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.995+0000 7fe4e6c49700 1 --2- 172.21.15.113:0/298700548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4e00fe250 0x7fe4e00f7a30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe4cc004730 tx=0x7fe4cc009cb0).stop 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.995+0000 7fe4e6c49700 1 -- 172.21.15.113:0/298700548 shutdown_connections 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.995+0000 7fe4e6c49700 1 --2- 172.21.15.113:0/298700548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4e00fe250 0x7fe4e00f7a30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.995+0000 7fe4e6c49700 1 --2- 172.21.15.113:0/298700548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4e00fd8f0 0x7fe4e00fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.995+0000 7fe4e6c49700 1 -- 172.21.15.113:0/298700548 >> 172.21.15.113:0/298700548 conn(0x7fe4e00f34b0 msgr2=0x7fe4e00f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.995+0000 7fe4e6c49700 1 -- 172.21.15.113:0/298700548 shutdown_connections 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.995+0000 7fe4e6c49700 1 -- 172.21.15.113:0/298700548 wait complete. 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.995+0000 7fe4e6c49700 1 Processor -- start 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.995+0000 7fe4e6c49700 1 -- start start 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e6c49700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4e00fd8f0 0x7fe4e01103b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e6c49700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4e00fe250 0x7fe4e01108a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e6c49700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0110ea0 con 0x7fe4e00fd8f0 2024-04-12T05:16:51.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e6c49700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0110fe0 con 0x7fe4e00fe250 2024-04-12T05:16:52.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e49e5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4e00fd8f0 0x7fe4e01103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e49e5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4e00fd8f0 0x7fe4e01103b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36582/0 (socket says 172.21.15.113:36582) 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e49e5700 1 -- 172.21.15.113:0/2219094452 learned_addr learned my addr 172.21.15.113:0/2219094452 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4dffff700 1 --2- 172.21.15.113:0/2219094452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4e00fe250 0x7fe4e01108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e49e5700 1 -- 172.21.15.113:0/2219094452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4e00fe250 msgr2=0x7fe4e01108a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e49e5700 1 --2- 172.21.15.113:0/2219094452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4e00fe250 0x7fe4e01108a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e49e5700 1 -- 172.21.15.113:0/2219094452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc005040 con 0x7fe4e00fd8f0 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e49e5700 1 --2- 172.21.15.113:0/2219094452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4e00fd8f0 0x7fe4e01103b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe4d400e910 tx=0x7fe4d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4ddffb700 1 -- 172.21.15.113:0/2219094452 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4d4007b40 con 0x7fe4e00fd8f0 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4ddffb700 1 -- 172.21.15.113:0/2219094452 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4d400f040 con 0x7fe4e00fd8f0 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4e6c49700 1 -- 172.21.15.113:0/2219094452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e010a610 con 0x7fe4e00fd8f0 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.996+0000 7fe4ddffb700 1 -- 172.21.15.113:0/2219094452 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4d4014870 con 0x7fe4e00fd8f0 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.997+0000 7fe4e6c49700 1 -- 172.21.15.113:0/2219094452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e010ab20 con 0x7fe4e00fd8f0 2024-04-12T05:16:52.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.998+0000 7fe4e6c49700 1 -- 172.21.15.113:0/2219094452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e005f590 con 0x7fe4e00fd8f0 2024-04-12T05:16:52.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.998+0000 7fe4ddffb700 1 -- 172.21.15.113:0/2219094452 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe4d4014ae0 con 0x7fe4e00fd8f0 2024-04-12T05:16:52.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.998+0000 7fe4ddffb700 1 --2- 172.21.15.113:0/2219094452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4d006caa0 0x7fe4d006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:52.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.998+0000 7fe4ddffb700 1 -- 172.21.15.113:0/2219094452 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6047+0+0 (secure 0 0 0) 0x7fe4d408b6d0 con 0x7fe4e00fd8f0 2024-04-12T05:16:52.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.999+0000 7fe4dffff700 1 --2- 172.21.15.113:0/2219094452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4d006caa0 0x7fe4d006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:51.999+0000 7fe4dffff700 1 --2- 172.21.15.113:0/2219094452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4d006caa0 0x7fe4d006ef60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe4cc004da0 tx=0x7fe4cc016040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:52.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.003+0000 7fe4ddffb700 1 -- 172.21.15.113:0/2219094452 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe4d40592e0 con 0x7fe4e00fd8f0 2024-04-12T05:16:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:52 smithi113 ceph-mon[103371]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:16:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:52 smithi113 ceph-mon[103371]: osdmap e65: 6 total, 5 up, 6 in 2024-04-12T05:16:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [0, 3]}]: dispatch 2024-04-12T05:16:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.19", "id": [0, 3]}]: dispatch 2024-04-12T05:16:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [0, 3]}]: dispatch 2024-04-12T05:16:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.4", "id": [1, 5]}]: dispatch 2024-04-12T05:16:52.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [4, 0]}]: dispatch 2024-04-12T05:16:52.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [1, 3]}]: dispatch 2024-04-12T05:16:52.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 3]}]: dispatch 2024-04-12T05:16:52.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:52.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:16:52.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.228+0000 7fe4e6c49700 1 -- 172.21.15.113:0/2219094452 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4e010bb10 con 0x7fe4d006caa0 2024-04-12T05:16:52.232 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.229+0000 7fe4ddffb700 1 -- 172.21.15.113:0/2219094452 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe4e010bb10 con 0x7fe4d006caa0 2024-04-12T05:16:52.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.231+0000 7fe4e6c49700 1 -- 172.21.15.113:0/2219094452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4d006caa0 msgr2=0x7fe4d006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.231+0000 7fe4e6c49700 1 --2- 172.21.15.113:0/2219094452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4d006caa0 0x7fe4d006ef60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe4cc004da0 tx=0x7fe4cc016040).stop 2024-04-12T05:16:52.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.231+0000 7fe4e6c49700 1 -- 172.21.15.113:0/2219094452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4e00fd8f0 msgr2=0x7fe4e01103b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.231+0000 7fe4e6c49700 1 --2- 172.21.15.113:0/2219094452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4e00fd8f0 0x7fe4e01103b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe4d400e910 tx=0x7fe4d400b040).stop 2024-04-12T05:16:52.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.231+0000 7fe4e6c49700 1 -- 172.21.15.113:0/2219094452 shutdown_connections 2024-04-12T05:16:52.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.231+0000 7fe4e6c49700 1 --2- 172.21.15.113:0/2219094452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4d006caa0 0x7fe4d006ef60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.231+0000 7fe4e6c49700 1 --2- 172.21.15.113:0/2219094452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4e00fd8f0 0x7fe4e01103b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.231+0000 7fe4e6c49700 1 --2- 172.21.15.113:0/2219094452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4e00fe250 0x7fe4e01108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.231+0000 7fe4e6c49700 1 -- 172.21.15.113:0/2219094452 >> 172.21.15.113:0/2219094452 conn(0x7fe4e00f34b0 msgr2=0x7fe4e00fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:52.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.231+0000 7fe4e6c49700 1 -- 172.21.15.113:0/2219094452 shutdown_connections 2024-04-12T05:16:52.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.231+0000 7fe4e6c49700 1 -- 172.21.15.113:0/2219094452 wait complete. 2024-04-12T05:16:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:16:52.298 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:52 smithi192 ceph-mon[85304]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:16:52.298 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:52 smithi192 ceph-mon[85304]: osdmap e65: 6 total, 5 up, 6 in 2024-04-12T05:16:52.298 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [0, 3]}]: dispatch 2024-04-12T05:16:52.299 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.19", "id": [0, 3]}]: dispatch 2024-04-12T05:16:52.299 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [0, 3]}]: dispatch 2024-04-12T05:16:52.299 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.4", "id": [1, 5]}]: dispatch 2024-04-12T05:16:52.299 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [4, 0]}]: dispatch 2024-04-12T05:16:52.299 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [1, 3]}]: dispatch 2024-04-12T05:16:52.299 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 3]}]: dispatch 2024-04-12T05:16:52.299 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:52.299 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:16:52.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.331+0000 7fe4a447a700 1 -- 172.21.15.113:0/1763922305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c0fd830 msgr2=0x7fe49c0fdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.331+0000 7fe4a447a700 1 --2- 172.21.15.113:0/1763922305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c0fd830 0x7fe49c0fdc10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe48c0047f0 tx=0x7fe48c009cb0).stop 2024-04-12T05:16:52.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.331+0000 7fe4a447a700 1 -- 172.21.15.113:0/1763922305 shutdown_connections 2024-04-12T05:16:52.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.331+0000 7fe4a447a700 1 --2- 172.21.15.113:0/1763922305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe49c0fe190 0x7fe49c0f7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:16:52.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.331+0000 7fe4a447a700 1 --2- 172.21.15.113:0/1763922305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c0fd830 0x7fe49c0fdc10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.331+0000 7fe4a447a700 1 -- 172.21.15.113:0/1763922305 >> 172.21.15.113:0/1763922305 conn(0x7fe49c0f3450 msgr2=0x7fe49c0f5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:52.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.331+0000 7fe4a447a700 1 -- 172.21.15.113:0/1763922305 shutdown_connections 2024-04-12T05:16:52.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.331+0000 7fe4a447a700 1 -- 172.21.15.113:0/1763922305 wait complete. 2024-04-12T05:16:52.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.331+0000 7fe4a447a700 1 Processor -- start 2024-04-12T05:16:52.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a447a700 1 -- start start 2024-04-12T05:16:52.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a447a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c0fd830 0x7fe49c1102f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a447a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe49c0fe190 0x7fe49c1107e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a447a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c110e70 con 0x7fe49c0fd830 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a447a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c10a280 con 0x7fe49c0fe190 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a2216700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c0fd830 0x7fe49c1102f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a2216700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c0fd830 0x7fe49c1102f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36602/0 (socket says 172.21.15.113:36602) 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a2216700 1 -- 172.21.15.113:0/2807391235 learned_addr learned my addr 172.21.15.113:0/2807391235 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a1a15700 1 --2- 172.21.15.113:0/2807391235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe49c0fe190 0x7fe49c1107e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a2216700 1 -- 172.21.15.113:0/2807391235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe49c0fe190 msgr2=0x7fe49c1107e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a2216700 1 --2- 172.21.15.113:0/2807391235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe49c0fe190 0x7fe49c1107e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a2216700 1 -- 172.21.15.113:0/2807391235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c005040 con 0x7fe49c0fd830 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a2216700 1 --2- 172.21.15.113:0/2807391235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c0fd830 0x7fe49c1102f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe48c00b120 tx=0x7fe48c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4937fe700 1 -- 172.21.15.113:0/2807391235 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe48c00e030 con 0x7fe49c0fd830 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4937fe700 1 -- 172.21.15.113:0/2807391235 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe48c007e90 con 0x7fe49c0fd830 2024-04-12T05:16:52.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4937fe700 1 -- 172.21.15.113:0/2807391235 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe48c018070 con 0x7fe49c0fd830 2024-04-12T05:16:52.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.332+0000 7fe4a447a700 1 -- 172.21.15.113:0/2807391235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49c10a4d0 con 0x7fe49c0fd830 2024-04-12T05:16:52.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.333+0000 7fe4a447a700 1 -- 172.21.15.113:0/2807391235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49c10a9b0 con 0x7fe49c0fd830 2024-04-12T05:16:52.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.334+0000 7fe4a447a700 1 -- 172.21.15.113:0/2807391235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49c05f470 con 0x7fe49c0fd830 2024-04-12T05:16:52.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.334+0000 7fe4937fe700 1 -- 172.21.15.113:0/2807391235 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe48c01d420 con 0x7fe49c0fd830 2024-04-12T05:16:52.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.334+0000 7fe4937fe700 1 --2- 172.21.15.113:0/2807391235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe48806caf0 0x7fe48806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:52.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.334+0000 7fe4937fe700 1 -- 172.21.15.113:0/2807391235 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6250+0+0 (secure 0 0 0) 0x7fe48c08d0e0 con 0x7fe49c0fd830 2024-04-12T05:16:52.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.334+0000 7fe4a1a15700 1 --2- 172.21.15.113:0/2807391235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe48806caf0 0x7fe48806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.335+0000 7fe4a1a15700 1 --2- 172.21.15.113:0/2807391235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe48806caf0 0x7fe48806efb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe49c10bc30 tx=0x7fe498009060).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:52.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.339+0000 7fe4937fe700 1 -- 172.21.15.113:0/2807391235 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe48c05ac20 con 0x7fe49c0fd830 2024-04-12T05:16:52.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.544+0000 7fe4a447a700 1 -- 172.21.15.113:0/2807391235 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe49c046d80 con 0x7fe48806caf0 2024-04-12T05:16:52.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.545+0000 7fe4937fe700 1 -- 172.21.15.113:0/2807391235 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe49800e620 con 0x7fe48806caf0 2024-04-12T05:16:52.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.547+0000 7fe4a447a700 1 -- 172.21.15.113:0/2807391235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe48806caf0 msgr2=0x7fe48806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.547+0000 7fe4a447a700 1 --2- 172.21.15.113:0/2807391235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe48806caf0 0x7fe48806efb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe49c10bc30 tx=0x7fe498009060).stop 2024-04-12T05:16:52.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.547+0000 7fe4a447a700 1 -- 172.21.15.113:0/2807391235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c0fd830 msgr2=0x7fe49c1102f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.547+0000 7fe4a447a700 1 --2- 172.21.15.113:0/2807391235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c0fd830 0x7fe49c1102f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe48c00b120 tx=0x7fe48c009cb0).stop 2024-04-12T05:16:52.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.547+0000 7fe4a447a700 1 -- 172.21.15.113:0/2807391235 shutdown_connections 2024-04-12T05:16:52.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.547+0000 7fe4a447a700 1 --2- 172.21.15.113:0/2807391235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe48806caf0 0x7fe48806efb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.547+0000 7fe4a447a700 1 --2- 172.21.15.113:0/2807391235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe49c0fd830 0x7fe49c1102f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.547+0000 7fe4a447a700 1 --2- 172.21.15.113:0/2807391235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe49c0fe190 0x7fe49c1107e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.547+0000 7fe4a447a700 1 -- 172.21.15.113:0/2807391235 >> 172.21.15.113:0/2807391235 conn(0x7fe49c0f3450 msgr2=0x7fe49c0f93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:52.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.548+0000 7fe4a447a700 1 -- 172.21.15.113:0/2807391235 shutdown_connections 2024-04-12T05:16:52.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.548+0000 7fe4a447a700 1 -- 172.21.15.113:0/2807391235 wait complete. 2024-04-12T05:16:52.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.646+0000 7fb3300f9700 1 -- 172.21.15.113:0/365753717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3280fd950 msgr2=0x7fb3280fdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.646+0000 7fb3300f9700 1 --2- 172.21.15.113:0/365753717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3280fd950 0x7fb3280fdd30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb3180047f0 tx=0x7fb318009cb0).stop 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.646+0000 7fb3300f9700 1 -- 172.21.15.113:0/365753717 shutdown_connections 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.646+0000 7fb3300f9700 1 --2- 172.21.15.113:0/365753717 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3280fe2b0 0x7fb328060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.646+0000 7fb3300f9700 1 --2- 172.21.15.113:0/365753717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3280fd950 0x7fb3280fdd30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.646+0000 7fb3300f9700 1 -- 172.21.15.113:0/365753717 >> 172.21.15.113:0/365753717 conn(0x7fb32806c760 msgr2=0x7fb32806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.646+0000 7fb3300f9700 1 -- 172.21.15.113:0/365753717 shutdown_connections 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.646+0000 7fb3300f9700 1 -- 172.21.15.113:0/365753717 wait complete. 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.646+0000 7fb3300f9700 1 Processor -- start 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.646+0000 7fb3300f9700 1 -- start start 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb3300f9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3280fd950 0x7fb32810e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb3300f9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3280fe2b0 0x7fb32810e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb3300f9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32810ec10 con 0x7fb3280fd950 2024-04-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb3300f9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32810ed50 con 0x7fb3280fe2b0 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb32de95700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3280fd950 0x7fb32810e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb32de95700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3280fd950 0x7fb32810e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36622/0 (socket says 172.21.15.113:36622) 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb32de95700 1 -- 172.21.15.113:0/3054174184 learned_addr learned my addr 172.21.15.113:0/3054174184 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb32d694700 1 --2- 172.21.15.113:0/3054174184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3280fe2b0 0x7fb32810e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb32de95700 1 -- 172.21.15.113:0/3054174184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3280fe2b0 msgr2=0x7fb32810e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb32de95700 1 --2- 172.21.15.113:0/3054174184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3280fe2b0 0x7fb32810e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb32de95700 1 -- 172.21.15.113:0/3054174184 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb318005040 con 0x7fb3280fd950 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb32de95700 1 --2- 172.21.15.113:0/3054174184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3280fd950 0x7fb32810e120 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb318009380 tx=0x7fb318009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb31effd700 1 -- 172.21.15.113:0/3054174184 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb318018070 con 0x7fb3280fd950 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb3300f9700 1 -- 172.21.15.113:0/3054174184 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb328108380 con 0x7fb3280fd950 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb31effd700 1 -- 172.21.15.113:0/3054174184 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb318007e90 con 0x7fb3280fd950 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb31effd700 1 -- 172.21.15.113:0/3054174184 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3180123c0 con 0x7fb3280fd950 2024-04-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.647+0000 7fb3300f9700 1 -- 172.21.15.113:0/3054174184 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3281088e0 con 0x7fb3280fd950 2024-04-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.648+0000 7fb3300f9700 1 -- 172.21.15.113:0/3054174184 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32805f570 con 0x7fb3280fd950 2024-04-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.649+0000 7fb31effd700 1 -- 172.21.15.113:0/3054174184 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb31801d420 con 0x7fb3280fd950 2024-04-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.649+0000 7fb31effd700 1 --2- 172.21.15.113:0/3054174184 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb31406caa0 0x7fb31406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.649+0000 7fb31effd700 1 -- 172.21.15.113:0/3054174184 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6250+0+0 (secure 0 0 0) 0x7fb31808c950 con 0x7fb3280fd950 2024-04-12T05:16:52.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.649+0000 7fb32d694700 1 --2- 172.21.15.113:0/3054174184 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb31406caa0 0x7fb31406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.649+0000 7fb32d694700 1 --2- 172.21.15.113:0/3054174184 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb31406caa0 0x7fb31406ef60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fb328109a60 tx=0x7fb32400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.653+0000 7fb31effd700 1 -- 172.21.15.113:0/3054174184 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb31805a490 con 0x7fb3280fd950 2024-04-12T05:16:52.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.862+0000 7fb3300f9700 1 -- 172.21.15.113:0/3054174184 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3280ffca0 con 0x7fb31406caa0 2024-04-12T05:16:52.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.872+0000 7fb31effd700 1 -- 172.21.15.113:0/3054174184 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fb3280ffca0 con 0x7fb31406caa0 2024-04-12T05:16:52.875 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:16:52.875 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (4m) 27s ago 15m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:16:52.875 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (2m) 27s ago 15m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:16:52.875 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (2m) 117s ago 15m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:16:52.875 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (4m) 27s ago 15m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:16:52.875 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (10m) 27s ago 10m 120M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:16:52.875 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (10m) 27s ago 10m 18.2M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:16:52.875 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (10m) 117s ago 10m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (10m) 117s ago 10m 15.3M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (6m) 27s ago 17m 501M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (5m) 117s ago 15m 448M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (3m) 27s ago 17m 46.6M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (2m) 117s ago 15m 33.0M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (5m) 27s ago 15m 16.6M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (5m) 117s ago 15m 17.8M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (102s) 27s ago 14m 93.5M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (67s) 27s ago 13m 93.9M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (32s) 27s ago 13m 13.7M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (13m) 117s ago 13m 264M 2251M 16.2.15-1-g8963e9de 727f386679cb d073fb539478 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (12m) 117s ago 12m 231M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (12m) 117s ago 12m 226M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:16:52.876 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (4m) 117s ago 14m 59.8M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:16:52.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.874+0000 7fb3300f9700 1 -- 172.21.15.113:0/3054174184 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb31406caa0 msgr2=0x7fb31406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.874+0000 7fb3300f9700 1 --2- 172.21.15.113:0/3054174184 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb31406caa0 0x7fb31406ef60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fb328109a60 tx=0x7fb32400b040).stop 2024-04-12T05:16:52.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.874+0000 7fb3300f9700 1 -- 172.21.15.113:0/3054174184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3280fd950 msgr2=0x7fb32810e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.874+0000 7fb3300f9700 1 --2- 172.21.15.113:0/3054174184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3280fd950 0x7fb32810e120 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb318009380 tx=0x7fb318009cb0).stop 2024-04-12T05:16:52.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.874+0000 7fb3300f9700 1 -- 172.21.15.113:0/3054174184 shutdown_connections 2024-04-12T05:16:52.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.874+0000 7fb3300f9700 1 --2- 172.21.15.113:0/3054174184 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb31406caa0 0x7fb31406ef60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.874+0000 7fb3300f9700 1 --2- 172.21.15.113:0/3054174184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3280fd950 0x7fb32810e120 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.874+0000 7fb3300f9700 1 --2- 172.21.15.113:0/3054174184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3280fe2b0 0x7fb32810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.874+0000 7fb3300f9700 1 -- 172.21.15.113:0/3054174184 >> 172.21.15.113:0/3054174184 conn(0x7fb32806c760 msgr2=0x7fb3280fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:52.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.874+0000 7fb3300f9700 1 -- 172.21.15.113:0/3054174184 shutdown_connections 2024-04-12T05:16:52.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.874+0000 7fb3300f9700 1 -- 172.21.15.113:0/3054174184 wait complete. 2024-04-12T05:16:52.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.971+0000 7fdd2d79d700 1 -- 172.21.15.113:0/64822031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280f92d0 msgr2=0x7fdd28101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.971+0000 7fdd2d79d700 1 --2- 172.21.15.113:0/64822031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280f92d0 0x7fdd28101a70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fdd1c004820 tx=0x7fdd1c009cb0).stop 2024-04-12T05:16:52.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.971+0000 7fdd2d79d700 1 -- 172.21.15.113:0/64822031 shutdown_connections 2024-04-12T05:16:52.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.971+0000 7fdd2d79d700 1 --2- 172.21.15.113:0/64822031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280f92d0 0x7fdd28101a70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.971+0000 7fdd2d79d700 1 --2- 172.21.15.113:0/64822031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280f8a00 0x7fdd280f8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.971+0000 7fdd2d79d700 1 -- 172.21.15.113:0/64822031 >> 172.21.15.113:0/64822031 conn(0x7fdd280f2490 msgr2=0x7fdd280f48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:52.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.971+0000 7fdd2d79d700 1 -- 172.21.15.113:0/64822031 shutdown_connections 2024-04-12T05:16:52.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.971+0000 7fdd2d79d700 1 -- 172.21.15.113:0/64822031 wait complete. 2024-04-12T05:16:52.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.972+0000 7fdd2d79d700 1 Processor -- start 2024-04-12T05:16:52.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.972+0000 7fdd2d79d700 1 -- start start 2024-04-12T05:16:52.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.972+0000 7fdd2d79d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280f8a00 0x7fdd2810e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.972+0000 7fdd2d79d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280f92d0 0x7fdd2810e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.972+0000 7fdd2d79d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2810ec20 con 0x7fdd280f92d0 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.972+0000 7fdd2d79d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2810ed60 con 0x7fdd280f8a00 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd267fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280f92d0 0x7fdd2810e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd267fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280f92d0 0x7fdd2810e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36650/0 (socket says 172.21.15.113:36650) 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd267fc700 1 -- 172.21.15.113:0/1854322302 learned_addr learned my addr 172.21.15.113:0/1854322302 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd26ffd700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280f8a00 0x7fdd2810e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd267fc700 1 -- 172.21.15.113:0/1854322302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280f8a00 msgr2=0x7fdd2810e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd267fc700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280f8a00 0x7fdd2810e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd267fc700 1 -- 172.21.15.113:0/1854322302 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c005040 con 0x7fdd280f92d0 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd26ffd700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280f8a00 0x7fdd2810e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd267fc700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280f92d0 0x7fdd2810e620 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fdd1c009360 tx=0x7fdd1c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:52.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd0ffff700 1 -- 172.21.15.113:0/1854322302 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd1c018070 con 0x7fdd280f92d0 2024-04-12T05:16:52.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd0ffff700 1 -- 172.21.15.113:0/1854322302 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd1c007e90 con 0x7fdd280f92d0 2024-04-12T05:16:52.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd2d79d700 1 -- 172.21.15.113:0/1854322302 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd281083e0 con 0x7fdd280f92d0 2024-04-12T05:16:52.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd0ffff700 1 -- 172.21.15.113:0/1854322302 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd1c01c3c0 con 0x7fdd280f92d0 2024-04-12T05:16:52.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.973+0000 7fdd2d79d700 1 -- 172.21.15.113:0/1854322302 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd281088c0 con 0x7fdd280f92d0 2024-04-12T05:16:52.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.974+0000 7fdd2d79d700 1 -- 172.21.15.113:0/1854322302 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd28060330 con 0x7fdd280f92d0 2024-04-12T05:16:52.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.975+0000 7fdd0ffff700 1 -- 172.21.15.113:0/1854322302 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdd1c01d420 con 0x7fdd280f92d0 2024-04-12T05:16:52.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.975+0000 7fdd0ffff700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006caa0 0x7fdd1006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:52.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.975+0000 7fdd0ffff700 1 -- 172.21.15.113:0/1854322302 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6250+0+0 (secure 0 0 0) 0x7fdd1c08d0f0 con 0x7fdd280f92d0 2024-04-12T05:16:52.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.975+0000 7fdd26ffd700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006caa0 0x7fdd1006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:52.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.975+0000 7fdd26ffd700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006caa0 0x7fdd1006ef60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fdd18011510 tx=0x7fdd1800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:52.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:52.979+0000 7fdd0ffff700 1 -- 172.21.15.113:0/1854322302 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fdd1c05acf0 con 0x7fdd280f92d0 2024-04-12T05:16:53.031 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:53 smithi192 ceph-mon[85304]: pgmap v109: 65 pgs: 16 stale+active+clean, 49 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:53.031 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [0, 3]}]': finished 2024-04-12T05:16:53.031 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.19", "id": [0, 3]}]': finished 2024-04-12T05:16:53.031 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [0, 3]}]': finished 2024-04-12T05:16:53.031 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.4", "id": [1, 5]}]': finished 2024-04-12T05:16:53.031 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [4, 0]}]': finished 2024-04-12T05:16:53.032 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [1, 3]}]': finished 2024-04-12T05:16:53.032 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 3]}]': finished 2024-04-12T05:16:53.032 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:53 smithi192 ceph-mon[85304]: osdmap e66: 6 total, 5 up, 6 in 2024-04-12T05:16:53.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.252+0000 7fdd2d79d700 1 -- 172.21.15.113:0/1854322302 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdd28109850 con 0x7fdd280f92d0 2024-04-12T05:16:53.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.253+0000 7fdd0ffff700 1 -- 172.21.15.113:0/1854322302 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fdd1c004490 con 0x7fdd280f92d0 2024-04-12T05:16:53.256 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:16:53.256 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:16:53.256 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:16:53.256 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:16:53.256 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:16:53.256 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:16:53.256 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:16:53.256 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:16:53.256 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2, 2024-04-12T05:16:53.256 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:16:53.257 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:16:53.257 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:16:53.257 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:16:53.257 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:16:53.257 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:16:53.257 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6, 2024-04-12T05:16:53.257 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 7 2024-04-12T05:16:53.257 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:16:53.257 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:16:53.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.255+0000 7fdd2d79d700 1 -- 172.21.15.113:0/1854322302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006caa0 msgr2=0x7fdd1006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:53.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.255+0000 7fdd2d79d700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006caa0 0x7fdd1006ef60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fdd18011510 tx=0x7fdd1800b040).stop 2024-04-12T05:16:53.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.255+0000 7fdd2d79d700 1 -- 172.21.15.113:0/1854322302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280f92d0 msgr2=0x7fdd2810e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:53.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.255+0000 7fdd2d79d700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280f92d0 0x7fdd2810e620 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fdd1c009360 tx=0x7fdd1c009cb0).stop 2024-04-12T05:16:53.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.255+0000 7fdd2d79d700 1 -- 172.21.15.113:0/1854322302 shutdown_connections 2024-04-12T05:16:53.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.255+0000 7fdd2d79d700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006caa0 0x7fdd1006ef60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.255+0000 7fdd2d79d700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280f8a00 0x7fdd2810e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.255+0000 7fdd2d79d700 1 --2- 172.21.15.113:0/1854322302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280f92d0 0x7fdd2810e620 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.255+0000 7fdd2d79d700 1 -- 172.21.15.113:0/1854322302 >> 172.21.15.113:0/1854322302 conn(0x7fdd280f2490 msgr2=0x7fdd280fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:53.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.255+0000 7fdd2d79d700 1 -- 172.21.15.113:0/1854322302 shutdown_connections 2024-04-12T05:16:53.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.255+0000 7fdd2d79d700 1 -- 172.21.15.113:0/1854322302 wait complete. 2024-04-12T05:16:53.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.355+0000 7f9bc4de9700 1 -- 172.21.15.113:0/2292311636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc00fd150 msgr2=0x7f9bc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:53.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.355+0000 7f9bc4de9700 1 --2- 172.21.15.113:0/2292311636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc00fd150 0x7f9bc00f6930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f9bb40047f0 tx=0x7f9bb4009cb0).stop 2024-04-12T05:16:53.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.355+0000 7f9bc4de9700 1 -- 172.21.15.113:0/2292311636 shutdown_connections 2024-04-12T05:16:53.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.355+0000 7f9bc4de9700 1 --2- 172.21.15.113:0/2292311636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc00fd150 0x7f9bc00f6930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.355+0000 7f9bc4de9700 1 --2- 172.21.15.113:0/2292311636 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bc00fc7f0 0x7f9bc00fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.355+0000 7f9bc4de9700 1 -- 172.21.15.113:0/2292311636 >> 172.21.15.113:0/2292311636 conn(0x7f9bc0070590 msgr2=0x7f9bc00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:53.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.355+0000 7f9bc4de9700 1 -- 172.21.15.113:0/2292311636 shutdown_connections 2024-04-12T05:16:53.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.355+0000 7f9bc4de9700 1 -- 172.21.15.113:0/2292311636 wait complete. 2024-04-12T05:16:53.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.356+0000 7f9bc4de9700 1 Processor -- start 2024-04-12T05:16:53.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.356+0000 7f9bc4de9700 1 -- start start 2024-04-12T05:16:53.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.356+0000 7f9bc4de9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc00fc7f0 0x7f9bc01090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:53.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.356+0000 7f9bc4de9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bc00fd150 0x7f9bc01095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:53.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.356+0000 7f9bc4de9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc0109aa0 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.356+0000 7f9bc4de9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc0109be0 con 0x7f9bc00fd150 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9bbe59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc00fc7f0 0x7f9bc01090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9bbe59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc00fc7f0 0x7f9bc01090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36662/0 (socket says 172.21.15.113:36662) 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9bbe59c700 1 -- 172.21.15.113:0/1578723700 learned_addr learned my addr 172.21.15.113:0/1578723700 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9bbdd9b700 1 --2- 172.21.15.113:0/1578723700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bc00fd150 0x7f9bc01095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9bbe59c700 1 -- 172.21.15.113:0/1578723700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bc00fd150 msgr2=0x7f9bc01095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9bbe59c700 1 --2- 172.21.15.113:0/1578723700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bc00fd150 0x7f9bc01095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9bbe59c700 1 -- 172.21.15.113:0/1578723700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb4005040 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9bbe59c700 1 --2- 172.21.15.113:0/1578723700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc00fc7f0 0x7f9bc01090c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9bb000c890 tx=0x7f9bb0007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9baf7fe700 1 -- 172.21.15.113:0/1578723700 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bb0004200 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9bc4de9700 1 -- 172.21.15.113:0/1578723700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc0109e90 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9baf7fe700 1 -- 172.21.15.113:0/1578723700 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bb000a9d0 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9baf7fe700 1 -- 172.21.15.113:0/1578723700 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bb00039e0 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.357+0000 7f9bc4de9700 1 -- 172.21.15.113:0/1578723700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc01124b0 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.358+0000 7f9bc4de9700 1 -- 172.21.15.113:0/1578723700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc005f590 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.359+0000 7f9baf7fe700 1 -- 172.21.15.113:0/1578723700 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9bb0003c50 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.360+0000 7f9baf7fe700 1 --2- 172.21.15.113:0/1578723700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ba806caa0 0x7f9ba806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:53.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.360+0000 7f9bbdd9b700 1 --2- 172.21.15.113:0/1578723700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ba806caa0 0x7f9ba806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:53.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.360+0000 7f9baf7fe700 1 -- 172.21.15.113:0/1578723700 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f9bb008bd70 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.360+0000 7f9bbdd9b700 1 --2- 172.21.15.113:0/1578723700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ba806caa0 0x7f9ba806ef60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9bc0060f00 tx=0x7f9bb400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:53.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.363+0000 7f9baf7fe700 1 -- 172.21.15.113:0/1578723700 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9bb0055dc0 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:53 smithi113 ceph-mon[103371]: pgmap v109: 65 pgs: 16 stale+active+clean, 49 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:16:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [0, 3]}]': finished 2024-04-12T05:16:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.19", "id": [0, 3]}]': finished 2024-04-12T05:16:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [0, 3]}]': finished 2024-04-12T05:16:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.4", "id": [1, 5]}]': finished 2024-04-12T05:16:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [4, 0]}]': finished 2024-04-12T05:16:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [1, 3]}]': finished 2024-04-12T05:16:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 3]}]': finished 2024-04-12T05:16:53.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:53 smithi113 ceph-mon[103371]: osdmap e66: 6 total, 5 up, 6 in 2024-04-12T05:16:53.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.591+0000 7f9bc4de9700 1 -- 172.21.15.113:0/1578723700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9bc00feb80 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.592+0000 7f9baf7fe700 1 -- 172.21.15.113:0/1578723700 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1859 (secure 0 0 0) 0x7f9bb0016070 con 0x7f9bc00fc7f0 2024-04-12T05:16:53.595 INFO:teuthology.orchestra.run.smithi113.stdout:e18 2024-04-12T05:16:53.595 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:16:53.595 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:16:53.595 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:16:53.595 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:16:53.595 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 18 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:16:53.596 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=14426,1=24261} 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:16:53.597 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:16:53.598 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:16:53.598 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:16:53.598 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:16:53.598 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:16:53.598 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:16:53.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.594+0000 7f9bc4de9700 1 -- 172.21.15.113:0/1578723700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ba806caa0 msgr2=0x7f9ba806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:53.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.594+0000 7f9bc4de9700 1 --2- 172.21.15.113:0/1578723700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ba806caa0 0x7f9ba806ef60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9bc0060f00 tx=0x7f9bb400b040).stop 2024-04-12T05:16:53.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.594+0000 7f9bc4de9700 1 -- 172.21.15.113:0/1578723700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc00fc7f0 msgr2=0x7f9bc01090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:53.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.594+0000 7f9bc4de9700 1 --2- 172.21.15.113:0/1578723700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc00fc7f0 0x7f9bc01090c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9bb000c890 tx=0x7f9bb0007df0).stop 2024-04-12T05:16:53.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.594+0000 7f9bc4de9700 1 -- 172.21.15.113:0/1578723700 shutdown_connections 2024-04-12T05:16:53.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.594+0000 7f9bc4de9700 1 --2- 172.21.15.113:0/1578723700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ba806caa0 0x7f9ba806ef60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.594+0000 7f9bc4de9700 1 --2- 172.21.15.113:0/1578723700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc00fc7f0 0x7f9bc01090c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.594+0000 7f9bc4de9700 1 --2- 172.21.15.113:0/1578723700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bc00fd150 0x7f9bc01095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.594+0000 7f9bc4de9700 1 -- 172.21.15.113:0/1578723700 >> 172.21.15.113:0/1578723700 conn(0x7f9bc0070590 msgr2=0x7f9bc00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:53.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.594+0000 7f9bc4de9700 1 -- 172.21.15.113:0/1578723700 shutdown_connections 2024-04-12T05:16:53.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.594+0000 7f9bc4de9700 1 -- 172.21.15.113:0/1578723700 wait complete. 2024-04-12T05:16:53.599 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 18 2024-04-12T05:16:53.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.693+0000 7fc08b6f6700 1 -- 172.21.15.113:0/1348326242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0840fc7f0 msgr2=0x7fc0840fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:53.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.693+0000 7fc08b6f6700 1 --2- 172.21.15.113:0/1348326242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0840fc7f0 0x7fc0840fcbd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc0740047f0 tx=0x7fc074009cb0).stop 2024-04-12T05:16:53.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.693+0000 7fc08b6f6700 1 -- 172.21.15.113:0/1348326242 shutdown_connections 2024-04-12T05:16:53.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.693+0000 7fc08b6f6700 1 --2- 172.21.15.113:0/1348326242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0840fd150 0x7fc0840f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.693+0000 7fc08b6f6700 1 --2- 172.21.15.113:0/1348326242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0840fc7f0 0x7fc0840fcbd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.693+0000 7fc08b6f6700 1 -- 172.21.15.113:0/1348326242 >> 172.21.15.113:0/1348326242 conn(0x7fc084070590 msgr2=0x7fc0840f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:53.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.693+0000 7fc08b6f6700 1 -- 172.21.15.113:0/1348326242 shutdown_connections 2024-04-12T05:16:53.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.693+0000 7fc08b6f6700 1 -- 172.21.15.113:0/1348326242 wait complete. 2024-04-12T05:16:53.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.693+0000 7fc08b6f6700 1 Processor -- start 2024-04-12T05:16:53.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.693+0000 7fc08b6f6700 1 -- start start 2024-04-12T05:16:53.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc08b6f6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0840fc7f0 0x7fc08410cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:53.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc08b6f6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0840fd150 0x7fc08410d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc08b6f6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08410da20 con 0x7fc0840fc7f0 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc08b6f6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08410db60 con 0x7fc0840fd150 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc089492700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0840fc7f0 0x7fc08410cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc089492700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0840fc7f0 0x7fc08410cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36676/0 (socket says 172.21.15.113:36676) 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc089492700 1 -- 172.21.15.113:0/3624005693 learned_addr learned my addr 172.21.15.113:0/3624005693 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc088c91700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0840fd150 0x7fc08410d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc089492700 1 -- 172.21.15.113:0/3624005693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0840fd150 msgr2=0x7fc08410d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc089492700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0840fd150 0x7fc08410d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc089492700 1 -- 172.21.15.113:0/3624005693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc074005040 con 0x7fc0840fc7f0 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc088c91700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0840fd150 0x7fc08410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc089492700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0840fc7f0 0x7fc08410cf30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc074005010 tx=0x7fc074009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc07a7fc700 1 -- 172.21.15.113:0/3624005693 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc07400e030 con 0x7fc0840fc7f0 2024-04-12T05:16:53.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc07a7fc700 1 -- 172.21.15.113:0/3624005693 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc074007e90 con 0x7fc0840fc7f0 2024-04-12T05:16:53.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc08b6f6700 1 -- 172.21.15.113:0/3624005693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc084107190 con 0x7fc0840fc7f0 2024-04-12T05:16:53.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc07a7fc700 1 -- 172.21.15.113:0/3624005693 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc074018070 con 0x7fc0840fc7f0 2024-04-12T05:16:53.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.694+0000 7fc08b6f6700 1 -- 172.21.15.113:0/3624005693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0841076f0 con 0x7fc0840fc7f0 2024-04-12T05:16:53.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.695+0000 7fc08b6f6700 1 -- 172.21.15.113:0/3624005693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08405f590 con 0x7fc0840fc7f0 2024-04-12T05:16:53.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.696+0000 7fc07a7fc700 1 -- 172.21.15.113:0/3624005693 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc07401d6e0 con 0x7fc0840fc7f0 2024-04-12T05:16:53.700 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:53 smithi192 systemd[1]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.3.service: Succeeded. 2024-04-12T05:16:53.700 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:53 smithi192 systemd[1]: Stopped Ceph osd.3 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:16:53.701 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:53 smithi192 systemd[1]: Starting Ceph osd.3 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:16:53.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.696+0000 7fc07a7fc700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc07006ca30 0x7fc07006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:53.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.696+0000 7fc07a7fc700 1 -- 172.21.15.113:0/3624005693 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fc07408d150 con 0x7fc0840fc7f0 2024-04-12T05:16:53.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.696+0000 7fc088c91700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc07006ca30 0x7fc07006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:53.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.696+0000 7fc088c91700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc07006ca30 0x7fc07006eef0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fc084108870 tx=0x7fc08000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:53.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.700+0000 7fc07a7fc700 1 -- 172.21.15.113:0/3624005693 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc07405a920 con 0x7fc0840fc7f0 2024-04-12T05:16:53.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.905+0000 7fc08b6f6700 1 -- 172.21.15.113:0/3624005693 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0840feb40 con 0x7fc07006ca30 2024-04-12T05:16:53.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.906+0000 7fc07a7fc700 1 -- 172.21.15.113:0/3624005693 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fc0840feb40 con 0x7fc07006ca30 2024-04-12T05:16:53.909 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:16:53.909 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:16:53.909 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:16:53.910 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:16:53.910 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:16:53.910 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:16:53.910 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:16:53.910 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:16:53.910 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:16:53.910 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "9/21 daemons upgraded", 2024-04-12T05:16:53.910 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:16:53.910 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:16:53.910 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:16:53.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.909+0000 7fc08b6f6700 1 -- 172.21.15.113:0/3624005693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc07006ca30 msgr2=0x7fc07006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:53.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.909+0000 7fc08b6f6700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc07006ca30 0x7fc07006eef0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fc084108870 tx=0x7fc08000b040).stop 2024-04-12T05:16:53.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.909+0000 7fc08b6f6700 1 -- 172.21.15.113:0/3624005693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0840fc7f0 msgr2=0x7fc08410cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:53.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.909+0000 7fc08b6f6700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0840fc7f0 0x7fc08410cf30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc074005010 tx=0x7fc074009cb0).stop 2024-04-12T05:16:53.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.909+0000 7fc08b6f6700 1 -- 172.21.15.113:0/3624005693 shutdown_connections 2024-04-12T05:16:53.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.909+0000 7fc08b6f6700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc07006ca30 0x7fc07006eef0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.909+0000 7fc08b6f6700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0840fc7f0 0x7fc08410cf30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.909+0000 7fc08b6f6700 1 --2- 172.21.15.113:0/3624005693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0840fd150 0x7fc08410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:53.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.909+0000 7fc08b6f6700 1 -- 172.21.15.113:0/3624005693 >> 172.21.15.113:0/3624005693 conn(0x7fc084070590 msgr2=0x7fc0840fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:53.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.909+0000 7fc08b6f6700 1 -- 172.21.15.113:0/3624005693 shutdown_connections 2024-04-12T05:16:53.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:53.909+0000 7fc08b6f6700 1 -- 172.21.15.113:0/3624005693 wait complete. 2024-04-12T05:16:54.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.007+0000 7f85599be700 1 -- 172.21.15.113:0/751520720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85540fc7f0 msgr2=0x7f85540fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:54.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.007+0000 7f85599be700 1 --2- 172.21.15.113:0/751520720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85540fc7f0 0x7f85540fcbd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f85440047f0 tx=0x7f8544009cb0).stop 2024-04-12T05:16:54.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.007+0000 7f85599be700 1 -- 172.21.15.113:0/751520720 shutdown_connections 2024-04-12T05:16:54.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.007+0000 7f85599be700 1 --2- 172.21.15.113:0/751520720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85540fd150 0x7f85540f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:54.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.007+0000 7f85599be700 1 --2- 172.21.15.113:0/751520720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85540fc7f0 0x7f85540fcbd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:54.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.007+0000 7f85599be700 1 -- 172.21.15.113:0/751520720 >> 172.21.15.113:0/751520720 conn(0x7f855406bdc0 msgr2=0x7f855406c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:54.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.007+0000 7f85599be700 1 -- 172.21.15.113:0/751520720 shutdown_connections 2024-04-12T05:16:54.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.007+0000 7f85599be700 1 -- 172.21.15.113:0/751520720 wait complete. 2024-04-12T05:16:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.008+0000 7f85599be700 1 Processor -- start 2024-04-12T05:16:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.008+0000 7f85599be700 1 -- start start 2024-04-12T05:16:54.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.008+0000 7f85599be700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85540fc7f0 0x7f855410d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.008+0000 7f85599be700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85540fd150 0x7f855410d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.008+0000 7f85599be700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855410db70 con 0x7f85540fc7f0 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.008+0000 7f85599be700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855410dcb0 con 0x7f85540fd150 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.008+0000 7f8552ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85540fc7f0 0x7f855410d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.008+0000 7f8552ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85540fc7f0 0x7f855410d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36704/0 (socket says 172.21.15.113:36704) 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.008+0000 7f8552ffd700 1 -- 172.21.15.113:0/3996243281 learned_addr learned my addr 172.21.15.113:0/3996243281 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.009+0000 7f85527fc700 1 --2- 172.21.15.113:0/3996243281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85540fd150 0x7f855410d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.009+0000 7f8552ffd700 1 -- 172.21.15.113:0/3996243281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85540fd150 msgr2=0x7f855410d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.009+0000 7f8552ffd700 1 --2- 172.21.15.113:0/3996243281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85540fd150 0x7f855410d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.009+0000 7f8552ffd700 1 -- 172.21.15.113:0/3996243281 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8544005040 con 0x7f85540fc7f0 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.009+0000 7f8552ffd700 1 --2- 172.21.15.113:0/3996243281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85540fc7f0 0x7f855410d080 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f8544009380 tx=0x7f8544009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.009+0000 7f853bfff700 1 -- 172.21.15.113:0/3996243281 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8544018070 con 0x7f85540fc7f0 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.009+0000 7f853bfff700 1 -- 172.21.15.113:0/3996243281 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8544007e90 con 0x7f85540fc7f0 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.009+0000 7f85599be700 1 -- 172.21.15.113:0/3996243281 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85541072e0 con 0x7f85540fc7f0 2024-04-12T05:16:54.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.009+0000 7f853bfff700 1 -- 172.21.15.113:0/3996243281 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f854401c380 con 0x7f85540fc7f0 2024-04-12T05:16:54.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.009+0000 7f85599be700 1 -- 172.21.15.113:0/3996243281 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85541077c0 con 0x7f85540fc7f0 2024-04-12T05:16:54.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.010+0000 7f85599be700 1 -- 172.21.15.113:0/3996243281 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855405f090 con 0x7f85540fc7f0 2024-04-12T05:16:54.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.010+0000 7f853bfff700 1 -- 172.21.15.113:0/3996243281 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f85440123c0 con 0x7f85540fc7f0 2024-04-12T05:16:54.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.011+0000 7f853bfff700 1 --2- 172.21.15.113:0/3996243281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f853c06cab0 0x7f853c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:16:54.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.011+0000 7f853bfff700 1 -- 172.21.15.113:0/3996243281 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f854408cc90 con 0x7f85540fc7f0 2024-04-12T05:16:54.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.011+0000 7f85527fc700 1 --2- 172.21.15.113:0/3996243281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f853c06cab0 0x7f853c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:16:54.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.011+0000 7f85527fc700 1 --2- 172.21.15.113:0/3996243281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f853c06cab0 0x7f853c06ef70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f85541089c0 tx=0x7f854800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:16:54.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.015+0000 7f853bfff700 1 -- 172.21.15.113:0/3996243281 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f854405a6f0 con 0x7f85540fc7f0 2024-04-12T05:16:54.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:54 smithi192 ceph-mon[85304]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:54.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:54 smithi192 ceph-mon[85304]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:54.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:54 smithi192 ceph-mon[85304]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:54.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:54 smithi192 ceph-mon[85304]: osdmap e67: 6 total, 5 up, 6 in 2024-04-12T05:16:54.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:54 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1854322302' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:16:54.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:54 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1578723700' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:16:54.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:54 smithi192 ceph-mon[85304]: osdmap e68: 6 total, 5 up, 6 in 2024-04-12T05:16:54.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.375+0000 7f85599be700 1 -- 172.21.15.113:0/3996243281 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8554047540 con 0x7f85540fc7f0 2024-04-12T05:16:54.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.376+0000 7f853bfff700 1 -- 172.21.15.113:0/3996243281 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1456 (secure 0 0 0) 0x7f854405a280 con 0x7f85540fc7f0 2024-04-12T05:16:54.379 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 1 pg peering; Degraded data redundancy: 65/324 objects degraded (20.062%), 18 pgs degraded 2024-04-12T05:16:54.379 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T05:16:54.379 INFO:teuthology.orchestra.run.smithi113.stdout: osd.3 (root=default,host=smithi192) is down 2024-04-12T05:16:54.379 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering 2024-04-12T05:16:54.379 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.19 is stuck peering for 95s, current state peering, last acting [4,2] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 65/324 objects degraded (20.062%), 18 pgs degraded 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 1.0 is active+undersized+degraded, acting [0,1] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.7 is active+undersized+degraded, acting [4,2] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.8 is active+undersized+degraded, acting [5,0] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.a is active+undersized+degraded, acting [1,4] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.d is active+undersized+degraded, acting [1,2] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.12 is active+undersized+degraded, acting [1,0] 2024-04-12T05:16:54.380 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.14 is active+undersized+degraded, acting [4,5] 2024-04-12T05:16:54.381 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-04-12T05:16:54.381 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-04-12T05:16:54.381 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-04-12T05:16:54.381 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1a is active+undersized+degraded, acting [4,5] 2024-04-12T05:16:54.381 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1d is active+undersized+degraded, acting [5,0] 2024-04-12T05:16:54.381 INFO:teuthology.orchestra.run.smithi113.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-04-12T05:16:54.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.378+0000 7f85599be700 1 -- 172.21.15.113:0/3996243281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f853c06cab0 msgr2=0x7f853c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:54.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.378+0000 7f85599be700 1 --2- 172.21.15.113:0/3996243281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f853c06cab0 0x7f853c06ef70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f85541089c0 tx=0x7f854800b040).stop 2024-04-12T05:16:54.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.378+0000 7f85599be700 1 -- 172.21.15.113:0/3996243281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85540fc7f0 msgr2=0x7f855410d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:16:54.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.378+0000 7f85599be700 1 --2- 172.21.15.113:0/3996243281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85540fc7f0 0x7f855410d080 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f8544009380 tx=0x7f8544009cb0).stop 2024-04-12T05:16:54.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.378+0000 7f85599be700 1 -- 172.21.15.113:0/3996243281 shutdown_connections 2024-04-12T05:16:54.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.378+0000 7f85599be700 1 --2- 172.21.15.113:0/3996243281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f853c06cab0 0x7f853c06ef70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:54.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.378+0000 7f85599be700 1 --2- 172.21.15.113:0/3996243281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f85540fc7f0 0x7f855410d080 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:54.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.378+0000 7f85599be700 1 --2- 172.21.15.113:0/3996243281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85540fd150 0x7f855410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:16:54.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.378+0000 7f85599be700 1 -- 172.21.15.113:0/3996243281 >> 172.21.15.113:0/3996243281 conn(0x7f855406bdc0 msgr2=0x7f85540ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:16:54.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.378+0000 7f85599be700 1 -- 172.21.15.113:0/3996243281 shutdown_connections 2024-04-12T05:16:54.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:16:54.378+0000 7f85599be700 1 -- 172.21.15.113:0/3996243281 wait complete. 2024-04-12T05:16:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:54 smithi113 ceph-mon[103371]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:54 smithi113 ceph-mon[103371]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:54 smithi113 ceph-mon[103371]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:54 smithi113 ceph-mon[103371]: osdmap e67: 6 total, 5 up, 6 in 2024-04-12T05:16:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:54 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1854322302' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:16:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:54 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1578723700' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:16:54.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:54 smithi113 ceph-mon[103371]: osdmap e68: 6 total, 5 up, 6 in 2024-04-12T05:16:54.910 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:54 smithi192 bash[97661]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-12T05:16:54.910 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:54 smithi192 bash[97661]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-12T05:16:54.910 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:54 smithi192 bash[97661]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-12T05:16:54.910 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:54 smithi192 bash[97661]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-12T05:16:54.910 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:54 smithi192 bash[97661]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-12T05:16:54.911 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:54 smithi192 bash[97661]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-12T05:16:54.911 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:54 smithi192 bash[97661]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-12T05:16:55.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:55 smithi192 ceph-mon[85304]: pgmap v112: 65 pgs: 3 remapped+peering, 3 peering, 18 active+undersized+degraded, 1 stale+active+clean, 15 active+undersized, 25 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/324 objects degraded (20.062%) 2024-04-12T05:16:55.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:55 smithi192 ceph-mon[85304]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:55.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:55 smithi192 ceph-mon[85304]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T05:16:55.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:55 smithi192 ceph-mon[85304]: Health check failed: Degraded data redundancy: 65/324 objects degraded (20.062%), 18 pgs degraded (PG_DEGRADED) 2024-04-12T05:16:55.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:55 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3996243281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:16:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:55 smithi113 ceph-mon[103371]: pgmap v112: 65 pgs: 3 remapped+peering, 3 peering, 18 active+undersized+degraded, 1 stale+active+clean, 15 active+undersized, 25 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 65/324 objects degraded (20.062%) 2024-04-12T05:16:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:55 smithi113 ceph-mon[103371]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:16:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:55 smithi113 ceph-mon[103371]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T05:16:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:55 smithi113 ceph-mon[103371]: Health check failed: Degraded data redundancy: 65/324 objects degraded (20.062%), 18 pgs degraded (PG_DEGRADED) 2024-04-12T05:16:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:55 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3996243281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:16:55.816 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:55 smithi192 ceph-osd[97982]: -- 172.21.15.192:0/3356613601 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e73a6431e0 con 0x55e73a613000 2024-04-12T05:16:55.816 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:55 smithi192 bash[97944]: 537f5471b31d7aedc696e29156ff543df7213470a146b516d3205c4126eb6c85 2024-04-12T05:16:56.198 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:55 smithi192 systemd[1]: Started Ceph osd.3 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:16:56.535 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:16:56 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[97959]: 2024-04-12T05:16:56.253+0000 7f6e20d6e7c0 -1 Falling back to public interface 2024-04-12T05:16:57.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:56 smithi113 ceph-mon[103371]: pgmap v114: 65 pgs: 1 active+recovering+undersized+remapped, 3 remapped+peering, 3 peering, 19 active+undersized+degraded, 17 active+undersized, 22 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 0 B/s, 98 objects/s recovering 2024-04-12T05:16:57.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:57.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:57.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:16:57.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:56 smithi113 ceph-mon[103371]: osdmap e69: 6 total, 5 up, 6 in 2024-04-12T05:16:57.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:56 smithi192 ceph-mon[85304]: pgmap v114: 65 pgs: 1 active+recovering+undersized+remapped, 3 remapped+peering, 3 peering, 19 active+undersized+degraded, 17 active+undersized, 22 active+clean; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 0 B/s, 98 objects/s recovering 2024-04-12T05:16:57.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:57.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:16:57.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:16:57.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:56 smithi192 ceph-mon[85304]: osdmap e69: 6 total, 5 up, 6 in 2024-04-12T05:16:58.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:58 smithi113 ceph-mon[103371]: osdmap e70: 6 total, 5 up, 6 in 2024-04-12T05:16:58.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:58 smithi192 ceph-mon[85304]: osdmap e70: 6 total, 5 up, 6 in 2024-04-12T05:16:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:59 smithi113 ceph-mon[103371]: pgmap v117: 65 pgs: 1 active+clean+remapped, 22 active+clean, 19 active+undersized+degraded, 2 peering, 3 remapped+peering, 17 active+undersized, 1 active+recovering+undersized+remapped; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 3/324 objects misplaced (0.926%); 0 B/s, 98 objects/s recovering 2024-04-12T05:16:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:16:59 smithi113 ceph-mon[103371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-12T05:16:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:59 smithi192 ceph-mon[85304]: pgmap v117: 65 pgs: 1 active+clean+remapped, 22 active+clean, 19 active+undersized+degraded, 2 peering, 3 remapped+peering, 17 active+undersized, 1 active+recovering+undersized+remapped; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 3/324 objects misplaced (0.926%); 0 B/s, 98 objects/s recovering 2024-04-12T05:16:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:16:59 smithi192 ceph-mon[85304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-12T05:17:00.851 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:00 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:00.851 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:00 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:00.851 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:00 smithi192 ceph-mon[85304]: pgmap v118: 65 pgs: 6 active+clean+remapped, 22 active+clean, 19 active+undersized+degraded, 17 active+undersized, 1 active+recovering+undersized+remapped; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 11/324 objects misplaced (3.395%); 0 B/s, 189 objects/s recovering 2024-04-12T05:17:00.851 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:00 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:00.851 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:00 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:00.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:00 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:00.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:00 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:00.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:00 smithi113 ceph-mon[103371]: pgmap v118: 65 pgs: 6 active+clean+remapped, 22 active+clean, 19 active+undersized+degraded, 17 active+undersized, 1 active+recovering+undersized+remapped; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 11/324 objects misplaced (3.395%); 0 B/s, 189 objects/s recovering 2024-04-12T05:17:00.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:00 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:00.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:00 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:03.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:02 smithi113 ceph-mon[103371]: pgmap v119: 65 pgs: 7 active+clean+remapped, 22 active+clean, 19 active+undersized+degraded, 17 active+undersized; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 11/324 objects misplaced (3.395%); 0 B/s, 175 objects/s recovering 2024-04-12T05:17:03.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:02 smithi113 ceph-mon[103371]: Health check update: Degraded data redundancy: 67/324 objects degraded (20.679%), 19 pgs degraded (PG_DEGRADED) 2024-04-12T05:17:03.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:02 smithi192 ceph-mon[85304]: pgmap v119: 65 pgs: 7 active+clean+remapped, 22 active+clean, 19 active+undersized+degraded, 17 active+undersized; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 11/324 objects misplaced (3.395%); 0 B/s, 175 objects/s recovering 2024-04-12T05:17:03.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:02 smithi192 ceph-mon[85304]: Health check update: Degraded data redundancy: 67/324 objects degraded (20.679%), 19 pgs degraded (PG_DEGRADED) 2024-04-12T05:17:03.949 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:17:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[97959]: 2024-04-12T05:17:03.625+0000 7f6e20d6e7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-12T05:17:04.448 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[97959]: 2024-04-12T05:17:04.125+0000 7f6e20d6e7c0 -1 osd.3 64 log_to_monitors true 2024-04-12T05:17:05.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: pgmap v120: 65 pgs: 7 active+clean+remapped, 22 active+clean, 19 active+undersized+degraded, 17 active+undersized; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 11/324 objects misplaced (3.395%); 0 B/s, 112 objects/s recovering 2024-04-12T05:17:05.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='osd.3 [v2:172.21.15.192:6800/2734551320,v1:172.21.15.192:6801/2734551320]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:17:05.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:17:05.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:05.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:05.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:17:05.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:17:05.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:05.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:17:05.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:05.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:05.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:05.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:04 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:17:05.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: pgmap v120: 65 pgs: 7 active+clean+remapped, 22 active+clean, 19 active+undersized+degraded, 17 active+undersized; 259 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 11/324 objects misplaced (3.395%); 0 B/s, 112 objects/s recovering 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='osd.3 [v2:172.21.15.192:6800/2734551320,v1:172.21.15.192:6801/2734551320]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:05.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:17:05.948 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:17:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[97959]: 2024-04-12T05:17:05.691+0000 7f6e0e1c0700 -1 osd.3 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:17:05.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:05 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:17:05.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:05 smithi192 ceph-mon[85304]: Upgrade: unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-04-12T05:17:05.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:05 smithi192 ceph-mon[85304]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T05:17:05.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:05 smithi192 ceph-mon[85304]: osdmap e71: 6 total, 5 up, 6 in 2024-04-12T05:17:05.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:05 smithi192 ceph-mon[85304]: from='osd.3 [v2:172.21.15.192:6800/2734551320,v1:172.21.15.192:6801/2734551320]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:17:05.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:05 smithi192 ceph-mon[85304]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:17:05.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:05 smithi192 ceph-mon[85304]: from='osd.3 ' entity='osd.3' 2024-04-12T05:17:06.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:05 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:17:06.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:05 smithi113 ceph-mon[103371]: Upgrade: unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-04-12T05:17:06.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:05 smithi113 ceph-mon[103371]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T05:17:06.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:05 smithi113 ceph-mon[103371]: osdmap e71: 6 total, 5 up, 6 in 2024-04-12T05:17:06.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:05 smithi113 ceph-mon[103371]: from='osd.3 [v2:172.21.15.192:6800/2734551320,v1:172.21.15.192:6801/2734551320]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:17:06.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:05 smithi113 ceph-mon[103371]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:17:06.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:05 smithi113 ceph-mon[103371]: from='osd.3 ' entity='osd.3' 2024-04-12T05:17:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:06 smithi113 ceph-mon[103371]: pgmap v122: 65 pgs: 7 active+clean+remapped, 22 active+clean, 19 active+undersized+degraded, 17 active+undersized; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 11/324 objects misplaced (3.395%); 0 B/s, 102 objects/s recovering 2024-04-12T05:17:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:06 smithi113 ceph-mon[103371]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:17:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:06 smithi113 ceph-mon[103371]: osd.3 [v2:172.21.15.192:6800/2734551320,v1:172.21.15.192:6801/2734551320] boot 2024-04-12T05:17:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:06 smithi113 ceph-mon[103371]: osdmap e72: 6 total, 6 up, 6 in 2024-04-12T05:17:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:17:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:17:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:06 smithi192 ceph-mon[85304]: pgmap v122: 65 pgs: 7 active+clean+remapped, 22 active+clean, 19 active+undersized+degraded, 17 active+undersized; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 67/324 objects degraded (20.679%); 11/324 objects misplaced (3.395%); 0 B/s, 102 objects/s recovering 2024-04-12T05:17:07.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:06 smithi192 ceph-mon[85304]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:17:07.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:06 smithi192 ceph-mon[85304]: osd.3 [v2:172.21.15.192:6800/2734551320,v1:172.21.15.192:6801/2734551320] boot 2024-04-12T05:17:07.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:06 smithi192 ceph-mon[85304]: osdmap e72: 6 total, 6 up, 6 in 2024-04-12T05:17:07.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T05:17:07.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:07.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:17:08.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:07 smithi113 ceph-mon[103371]: osdmap e73: 6 total, 6 up, 6 in 2024-04-12T05:17:08.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:07 smithi192 ceph-mon[85304]: osdmap e73: 6 total, 6 up, 6 in 2024-04-12T05:17:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:08 smithi113 ceph-mon[103371]: pgmap v125: 65 pgs: 1 peering, 3 remapped+peering, 3 active+clean+remapped, 30 active+clean, 16 active+undersized+degraded, 12 active+undersized; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 58/324 objects degraded (17.901%); 8/324 objects misplaced (2.469%) 2024-04-12T05:17:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:08 smithi113 ceph-mon[103371]: Health check update: Degraded data redundancy: 58/324 objects degraded (17.901%), 16 pgs degraded (PG_DEGRADED) 2024-04-12T05:17:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:08 smithi113 ceph-mon[103371]: osdmap e74: 6 total, 6 up, 6 in 2024-04-12T05:17:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:08 smithi192 ceph-mon[85304]: pgmap v125: 65 pgs: 1 peering, 3 remapped+peering, 3 active+clean+remapped, 30 active+clean, 16 active+undersized+degraded, 12 active+undersized; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 58/324 objects degraded (17.901%); 8/324 objects misplaced (2.469%) 2024-04-12T05:17:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:08 smithi192 ceph-mon[85304]: Health check update: Degraded data redundancy: 58/324 objects degraded (17.901%), 16 pgs degraded (PG_DEGRADED) 2024-04-12T05:17:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:08 smithi192 ceph-mon[85304]: osdmap e74: 6 total, 6 up, 6 in 2024-04-12T05:17:10.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:09 smithi113 ceph-mon[103371]: osdmap e75: 6 total, 6 up, 6 in 2024-04-12T05:17:10.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:09 smithi192 ceph-mon[85304]: osdmap e75: 6 total, 6 up, 6 in 2024-04-12T05:17:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:10 smithi113 ceph-mon[103371]: pgmap v128: 65 pgs: 1 peering, 3 remapped+peering, 1 active+clean+remapped, 40 active+clean, 2 active+undersized+remapped, 11 active+undersized+degraded, 7 active+undersized; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 43/324 objects degraded (13.272%); 5.0 MiB/s, 1 objects/s recovering 2024-04-12T05:17:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:10 smithi113 ceph-mon[103371]: osdmap e76: 6 total, 6 up, 6 in 2024-04-12T05:17:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:10 smithi192 ceph-mon[85304]: pgmap v128: 65 pgs: 1 peering, 3 remapped+peering, 1 active+clean+remapped, 40 active+clean, 2 active+undersized+remapped, 11 active+undersized+degraded, 7 active+undersized; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 43/324 objects degraded (13.272%); 5.0 MiB/s, 1 objects/s recovering 2024-04-12T05:17:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:10 smithi192 ceph-mon[85304]: osdmap e76: 6 total, 6 up, 6 in 2024-04-12T05:17:12.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:12 smithi113 ceph-mon[103371]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T05:17:12.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:12 smithi113 ceph-mon[103371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/324 objects degraded (13.272%), 11 pgs degraded) 2024-04-12T05:17:12.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:12 smithi192 ceph-mon[85304]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T05:17:12.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:12 smithi192 ceph-mon[85304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/324 objects degraded (13.272%), 11 pgs degraded) 2024-04-12T05:17:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:13 smithi113 ceph-mon[103371]: pgmap v130: 65 pgs: 2 peering, 2 remapped+peering, 59 active+clean, 2 active+undersized+remapped; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 4.1 MiB/s, 0 objects/s recovering 2024-04-12T05:17:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:13 smithi192 ceph-mon[85304]: pgmap v130: 65 pgs: 2 peering, 2 remapped+peering, 59 active+clean, 2 active+undersized+remapped; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 4.1 MiB/s, 0 objects/s recovering 2024-04-12T05:17:14.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:14 smithi113 ceph-mon[103371]: pgmap v131: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 62 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%); 3.3 MiB/s, 0 objects/s recovering 2024-04-12T05:17:14.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:14 smithi113 ceph-mon[103371]: Health check failed: Degraded data redundancy: 725/324 objects degraded (223.765%), 1 pg degraded (PG_DEGRADED) 2024-04-12T05:17:14.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:14 smithi192 ceph-mon[85304]: pgmap v131: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 62 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%); 3.3 MiB/s, 0 objects/s recovering 2024-04-12T05:17:14.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:14 smithi192 ceph-mon[85304]: Health check failed: Degraded data redundancy: 725/324 objects degraded (223.765%), 1 pg degraded (PG_DEGRADED) 2024-04-12T05:17:16.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:15 smithi113 ceph-mon[103371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-12T05:17:16.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:15 smithi192 ceph-mon[85304]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-12T05:17:17.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:16 smithi113 ceph-mon[103371]: pgmap v132: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%); 2.5 MiB/s, 0 objects/s recovering 2024-04-12T05:17:17.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:16 smithi192 ceph-mon[85304]: pgmap v132: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%); 2.5 MiB/s, 0 objects/s recovering 2024-04-12T05:17:19.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:18 smithi113 ceph-mon[103371]: pgmap v133: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%) 2024-04-12T05:17:19.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:18 smithi192 ceph-mon[85304]: pgmap v133: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%) 2024-04-12T05:17:20.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:19 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:17:20.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:19 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:17:21.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:20 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:17:21.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:20 smithi113 ceph-mon[103371]: Upgrade: osd.4 is safe to restart 2024-04-12T05:17:21.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:20 smithi113 ceph-mon[103371]: pgmap v134: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%) 2024-04-12T05:17:21.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:20 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T05:17:21.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:20 smithi192 ceph-mon[85304]: Upgrade: osd.4 is safe to restart 2024-04-12T05:17:21.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:20 smithi192 ceph-mon[85304]: pgmap v134: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%) 2024-04-12T05:17:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:22 smithi192 ceph-mon[85304]: Upgrade: Updating osd.4 2024-04-12T05:17:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:17:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:17:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:22 smithi192 ceph-mon[85304]: Deploying daemon osd.4 on smithi192 2024-04-12T05:17:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:22 smithi192 ceph-mon[85304]: pgmap v135: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%); 0 B/s, 18 objects/s recovering 2024-04-12T05:17:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:17:22.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:22 smithi113 ceph-mon[103371]: Upgrade: Updating osd.4 2024-04-12T05:17:22.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:22.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T05:17:22.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:17:22.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:22 smithi113 ceph-mon[103371]: Deploying daemon osd.4 on smithi192 2024-04-12T05:17:22.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:22 smithi113 ceph-mon[103371]: pgmap v135: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%); 0 B/s, 18 objects/s recovering 2024-04-12T05:17:22.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:17:24.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.494+0000 7f5198eee700 1 -- 172.21.15.113:0/3899382851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51940fd910 msgr2=0x7f51940fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:24.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.494+0000 7f5198eee700 1 --2- 172.21.15.113:0/3899382851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51940fd910 0x7f51940fdcf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f51840047f0 tx=0x7f5184009cb0).stop 2024-04-12T05:17:24.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.494+0000 7f5198eee700 1 -- 172.21.15.113:0/3899382851 shutdown_connections 2024-04-12T05:17:24.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.494+0000 7f5198eee700 1 --2- 172.21.15.113:0/3899382851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51940fe270 0x7f51940f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:24.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.494+0000 7f5198eee700 1 --2- 172.21.15.113:0/3899382851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51940fd910 0x7f51940fdcf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:24.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.494+0000 7f5198eee700 1 -- 172.21.15.113:0/3899382851 >> 172.21.15.113:0/3899382851 conn(0x7f51940f3530 msgr2=0x7f51940f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:24.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.494+0000 7f5198eee700 1 -- 172.21.15.113:0/3899382851 shutdown_connections 2024-04-12T05:17:24.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.494+0000 7f5198eee700 1 -- 172.21.15.113:0/3899382851 wait complete. 2024-04-12T05:17:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.495+0000 7f5198eee700 1 Processor -- start 2024-04-12T05:17:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.495+0000 7f5198eee700 1 -- start start 2024-04-12T05:17:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.495+0000 7f5198eee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51940fd910 0x7f519410e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.495+0000 7f5198eee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51940fe270 0x7f519410e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.495+0000 7f5198eee700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519410ec20 con 0x7f51940fd910 2024-04-12T05:17:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.495+0000 7f5198eee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519410ed60 con 0x7f51940fe270 2024-04-12T05:17:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.495+0000 7f519259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51940fd910 0x7f519410e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.495+0000 7f519259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51940fd910 0x7f519410e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60782/0 (socket says 172.21.15.113:60782) 2024-04-12T05:17:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.495+0000 7f519259c700 1 -- 172.21.15.113:0/3310038030 learned_addr learned my addr 172.21.15.113:0/3310038030 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.496+0000 7f5191d9b700 1 --2- 172.21.15.113:0/3310038030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51940fe270 0x7f519410e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:24.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.496+0000 7f519259c700 1 -- 172.21.15.113:0/3310038030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51940fe270 msgr2=0x7f519410e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:24.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.496+0000 7f519259c700 1 --2- 172.21.15.113:0/3310038030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51940fe270 0x7f519410e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:24.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.496+0000 7f519259c700 1 -- 172.21.15.113:0/3310038030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5184005040 con 0x7f51940fd910 2024-04-12T05:17:24.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.496+0000 7f519259c700 1 --2- 172.21.15.113:0/3310038030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51940fd910 0x7f519410e130 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5184009380 tx=0x7f5184009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:24.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.496+0000 7f518b7fe700 1 -- 172.21.15.113:0/3310038030 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5184018070 con 0x7f51940fd910 2024-04-12T05:17:24.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.496+0000 7f518b7fe700 1 -- 172.21.15.113:0/3310038030 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5184007e90 con 0x7f51940fd910 2024-04-12T05:17:24.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.496+0000 7f5198eee700 1 -- 172.21.15.113:0/3310038030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5194108390 con 0x7f51940fd910 2024-04-12T05:17:24.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.496+0000 7f518b7fe700 1 -- 172.21.15.113:0/3310038030 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51840123c0 con 0x7f51940fd910 2024-04-12T05:17:24.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.496+0000 7f5198eee700 1 -- 172.21.15.113:0/3310038030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51941088f0 con 0x7f51940fd910 2024-04-12T05:17:24.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.497+0000 7f5198eee700 1 -- 172.21.15.113:0/3310038030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519405f570 con 0x7f51940fd910 2024-04-12T05:17:24.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.498+0000 7f518b7fe700 1 -- 172.21.15.113:0/3310038030 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f518401d420 con 0x7f51940fd910 2024-04-12T05:17:24.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.498+0000 7f518b7fe700 1 --2- 172.21.15.113:0/3310038030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f518006caa0 0x7f518006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:24.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.498+0000 7f5191d9b700 1 --2- 172.21.15.113:0/3310038030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f518006caa0 0x7f518006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:24.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.498+0000 7f518b7fe700 1 -- 172.21.15.113:0/3310038030 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6308+0+0 (secure 0 0 0) 0x7f518408c990 con 0x7f51940fd910 2024-04-12T05:17:24.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.499+0000 7f5191d9b700 1 --2- 172.21.15.113:0/3310038030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f518006caa0 0x7f518006ef60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5194109a70 tx=0x7f517c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:24.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.503+0000 7f518b7fe700 1 -- 172.21.15.113:0/3310038030 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f518405a490 con 0x7f51940fd910 2024-04-12T05:17:24.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.710+0000 7f5198eee700 1 -- 172.21.15.113:0/3310038030 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f51940ffc60 con 0x7f518006caa0 2024-04-12T05:17:24.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.711+0000 7f518b7fe700 1 -- 172.21.15.113:0/3310038030 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f51940ffc60 con 0x7f518006caa0 2024-04-12T05:17:24.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.713+0000 7f5198eee700 1 -- 172.21.15.113:0/3310038030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f518006caa0 msgr2=0x7f518006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:24.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.713+0000 7f5198eee700 1 --2- 172.21.15.113:0/3310038030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f518006caa0 0x7f518006ef60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5194109a70 tx=0x7f517c00b040).stop 2024-04-12T05:17:24.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.713+0000 7f5198eee700 1 -- 172.21.15.113:0/3310038030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51940fd910 msgr2=0x7f519410e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:24.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.713+0000 7f5198eee700 1 --2- 172.21.15.113:0/3310038030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51940fd910 0x7f519410e130 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5184009380 tx=0x7f5184009cb0).stop 2024-04-12T05:17:24.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.713+0000 7f5198eee700 1 -- 172.21.15.113:0/3310038030 shutdown_connections 2024-04-12T05:17:24.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.713+0000 7f5198eee700 1 --2- 172.21.15.113:0/3310038030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f518006caa0 0x7f518006ef60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:24.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.713+0000 7f5198eee700 1 --2- 172.21.15.113:0/3310038030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51940fd910 0x7f519410e130 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:24.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.713+0000 7f5198eee700 1 --2- 172.21.15.113:0/3310038030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51940fe270 0x7f519410e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:24.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.713+0000 7f5198eee700 1 -- 172.21.15.113:0/3310038030 >> 172.21.15.113:0/3310038030 conn(0x7f51940f3530 msgr2=0x7f51940fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:24.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.714+0000 7f5198eee700 1 -- 172.21.15.113:0/3310038030 shutdown_connections 2024-04-12T05:17:24.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.714+0000 7f5198eee700 1 -- 172.21.15.113:0/3310038030 wait complete. 2024-04-12T05:17:24.726 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:17:24.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.813+0000 7fe2cdb05700 1 -- 172.21.15.113:0/1321445061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80fc7f0 msgr2=0x7fe2c80fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:24.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.813+0000 7fe2cdb05700 1 --2- 172.21.15.113:0/1321445061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80fc7f0 0x7fe2c80fcbd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fe2bc0047f0 tx=0x7fe2bc009cb0).stop 2024-04-12T05:17:24.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.813+0000 7fe2cdb05700 1 -- 172.21.15.113:0/1321445061 shutdown_connections 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.813+0000 7fe2cdb05700 1 --2- 172.21.15.113:0/1321445061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2c80fd150 0x7fe2c80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.813+0000 7fe2cdb05700 1 --2- 172.21.15.113:0/1321445061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80fc7f0 0x7fe2c80fcbd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.813+0000 7fe2cdb05700 1 -- 172.21.15.113:0/1321445061 >> 172.21.15.113:0/1321445061 conn(0x7fe2c8070590 msgr2=0x7fe2c80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.813+0000 7fe2cdb05700 1 -- 172.21.15.113:0/1321445061 shutdown_connections 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.813+0000 7fe2cdb05700 1 -- 172.21.15.113:0/1321445061 wait complete. 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.813+0000 7fe2cdb05700 1 Processor -- start 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2cdb05700 1 -- start start 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2cdb05700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2c80fc7f0 0x7fe2c810e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2cdb05700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80fd150 0x7fe2c810e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2cdb05700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c810eb80 con 0x7fe2c80fd150 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2cdb05700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c810ecc0 con 0x7fe2c80fc7f0 2024-04-12T05:17:24.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2c6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80fd150 0x7fe2c810e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:24.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2c6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80fd150 0x7fe2c810e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60800/0 (socket says 172.21.15.113:60800) 2024-04-12T05:17:24.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2c6ffd700 1 -- 172.21.15.113:0/3320672017 learned_addr learned my addr 172.21.15.113:0/3320672017 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:24.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2c77fe700 1 --2- 172.21.15.113:0/3320672017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2c80fc7f0 0x7fe2c810e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:24.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2c6ffd700 1 -- 172.21.15.113:0/3320672017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2c80fc7f0 msgr2=0x7fe2c810e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:24.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2c6ffd700 1 --2- 172.21.15.113:0/3320672017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2c80fc7f0 0x7fe2c810e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:24.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2c6ffd700 1 -- 172.21.15.113:0/3320672017 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc005040 con 0x7fe2c80fd150 2024-04-12T05:17:24.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.814+0000 7fe2c6ffd700 1 --2- 172.21.15.113:0/3320672017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80fd150 0x7fe2c810e580 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe2b800e8c0 tx=0x7fe2b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:24.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.815+0000 7fe2c4ff9700 1 -- 172.21.15.113:0/3320672017 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2b8007b40 con 0x7fe2c80fd150 2024-04-12T05:17:24.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.815+0000 7fe2cdb05700 1 -- 172.21.15.113:0/3320672017 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c8108390 con 0x7fe2c80fd150 2024-04-12T05:17:24.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.815+0000 7fe2c4ff9700 1 -- 172.21.15.113:0/3320672017 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2b800f040 con 0x7fe2c80fd150 2024-04-12T05:17:24.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.815+0000 7fe2c4ff9700 1 -- 172.21.15.113:0/3320672017 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2b8014830 con 0x7fe2c80fd150 2024-04-12T05:17:24.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.815+0000 7fe2cdb05700 1 -- 172.21.15.113:0/3320672017 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c8108950 con 0x7fe2c80fd150 2024-04-12T05:17:24.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.816+0000 7fe2c4ff9700 1 -- 172.21.15.113:0/3320672017 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe2b8014af0 con 0x7fe2c80fd150 2024-04-12T05:17:24.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.816+0000 7fe2cdb05700 1 -- 172.21.15.113:0/3320672017 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c805f590 con 0x7fe2c80fd150 2024-04-12T05:17:24.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.816+0000 7fe2c4ff9700 1 --2- 172.21.15.113:0/3320672017 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe2b006caa0 0x7fe2b006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:24.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.817+0000 7fe2c4ff9700 1 -- 172.21.15.113:0/3320672017 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6308+0+0 (secure 0 0 0) 0x7fe2b808b8d0 con 0x7fe2c80fd150 2024-04-12T05:17:24.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.817+0000 7fe2c77fe700 1 --2- 172.21.15.113:0/3320672017 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe2b006caa0 0x7fe2b006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:24.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.817+0000 7fe2c77fe700 1 --2- 172.21.15.113:0/3320672017 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe2b006caa0 0x7fe2b006ef60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe2bc0047c0 tx=0x7fe2bc009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:24.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:24.821+0000 7fe2c4ff9700 1 -- 172.21.15.113:0/3320672017 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe2b8055a00 con 0x7fe2c80fd150 2024-04-12T05:17:25.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.028+0000 7fe2cdb05700 1 -- 172.21.15.113:0/3320672017 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2c8108f90 con 0x7fe2b006caa0 2024-04-12T05:17:25.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.029+0000 7fe2c4ff9700 1 -- 172.21.15.113:0/3320672017 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fe2c8108f90 con 0x7fe2b006caa0 2024-04-12T05:17:25.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.031+0000 7fe2cdb05700 1 -- 172.21.15.113:0/3320672017 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe2b006caa0 msgr2=0x7fe2b006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.031+0000 7fe2cdb05700 1 --2- 172.21.15.113:0/3320672017 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe2b006caa0 0x7fe2b006ef60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe2bc0047c0 tx=0x7fe2bc009cb0).stop 2024-04-12T05:17:25.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.031+0000 7fe2cdb05700 1 -- 172.21.15.113:0/3320672017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80fd150 msgr2=0x7fe2c810e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.031+0000 7fe2cdb05700 1 --2- 172.21.15.113:0/3320672017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80fd150 0x7fe2c810e580 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe2b800e8c0 tx=0x7fe2b800b040).stop 2024-04-12T05:17:25.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.031+0000 7fe2cdb05700 1 -- 172.21.15.113:0/3320672017 shutdown_connections 2024-04-12T05:17:25.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.031+0000 7fe2cdb05700 1 --2- 172.21.15.113:0/3320672017 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe2b006caa0 0x7fe2b006ef60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.031+0000 7fe2cdb05700 1 --2- 172.21.15.113:0/3320672017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2c80fc7f0 0x7fe2c810e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.031+0000 7fe2cdb05700 1 --2- 172.21.15.113:0/3320672017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2c80fd150 0x7fe2c810e580 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.031+0000 7fe2cdb05700 1 -- 172.21.15.113:0/3320672017 >> 172.21.15.113:0/3320672017 conn(0x7fe2c8070590 msgr2=0x7fe2c8060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:25.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.031+0000 7fe2cdb05700 1 -- 172.21.15.113:0/3320672017 shutdown_connections 2024-04-12T05:17:25.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.031+0000 7fe2cdb05700 1 -- 172.21.15.113:0/3320672017 wait complete. 2024-04-12T05:17:25.034 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:24 smithi192 ceph-mon[85304]: pgmap v136: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%); 0 B/s, 28 objects/s recovering 2024-04-12T05:17:25.035 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:24 smithi192 systemd[1]: Stopping Ceph osd.4 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:17:25.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.129+0000 7fbce9fd2700 1 -- 172.21.15.113:0/89478427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbce40fd150 msgr2=0x7fbce40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.129+0000 7fbce9fd2700 1 --2- 172.21.15.113:0/89478427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbce40fd150 0x7fbce40f6930 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fbcd40047f0 tx=0x7fbcd4009cb0).stop 2024-04-12T05:17:25.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.129+0000 7fbce9fd2700 1 -- 172.21.15.113:0/89478427 shutdown_connections 2024-04-12T05:17:25.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.129+0000 7fbce9fd2700 1 --2- 172.21.15.113:0/89478427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbce40fd150 0x7fbce40f6930 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.129+0000 7fbce9fd2700 1 --2- 172.21.15.113:0/89478427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbce40fc7f0 0x7fbce40fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.129+0000 7fbce9fd2700 1 -- 172.21.15.113:0/89478427 >> 172.21.15.113:0/89478427 conn(0x7fbce40f23b0 msgr2=0x7fbce40f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:25.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.129+0000 7fbce9fd2700 1 -- 172.21.15.113:0/89478427 shutdown_connections 2024-04-12T05:17:25.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.129+0000 7fbce9fd2700 1 -- 172.21.15.113:0/89478427 wait complete. 2024-04-12T05:17:25.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.130+0000 7fbce9fd2700 1 Processor -- start 2024-04-12T05:17:25.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.130+0000 7fbce9fd2700 1 -- start start 2024-04-12T05:17:25.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.130+0000 7fbce9fd2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbce40fc7f0 0x7fbce410d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:25.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.130+0000 7fbce9fd2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbce40fd150 0x7fbce410d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:25.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.130+0000 7fbce9fd2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce410db10 con 0x7fbce40fd150 2024-04-12T05:17:25.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.130+0000 7fbce9fd2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce410dc50 con 0x7fbce40fc7f0 2024-04-12T05:17:25.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.130+0000 7fbce2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbce40fd150 0x7fbce410d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:25.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.130+0000 7fbce2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbce40fd150 0x7fbce410d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60812/0 (socket says 172.21.15.113:60812) 2024-04-12T05:17:25.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.130+0000 7fbce2ffd700 1 -- 172.21.15.113:0/1900121772 learned_addr learned my addr 172.21.15.113:0/1900121772 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:25.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.130+0000 7fbce37fe700 1 --2- 172.21.15.113:0/1900121772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbce40fc7f0 0x7fbce410d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:25.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.131+0000 7fbce2ffd700 1 -- 172.21.15.113:0/1900121772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbce40fc7f0 msgr2=0x7fbce410d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.131+0000 7fbce2ffd700 1 --2- 172.21.15.113:0/1900121772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbce40fc7f0 0x7fbce410d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.131+0000 7fbce2ffd700 1 -- 172.21.15.113:0/1900121772 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd4005040 con 0x7fbce40fd150 2024-04-12T05:17:25.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.131+0000 7fbce2ffd700 1 --2- 172.21.15.113:0/1900121772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbce40fd150 0x7fbce410d510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fbcd4000c00 tx=0x7fbcd4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:25.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.131+0000 7fbce0ff9700 1 -- 172.21.15.113:0/1900121772 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcd4018070 con 0x7fbce40fd150 2024-04-12T05:17:25.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.131+0000 7fbce0ff9700 1 -- 172.21.15.113:0/1900121772 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcd4007e90 con 0x7fbce40fd150 2024-04-12T05:17:25.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.131+0000 7fbce9fd2700 1 -- 172.21.15.113:0/1900121772 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce4107280 con 0x7fbce40fd150 2024-04-12T05:17:25.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.131+0000 7fbce0ff9700 1 -- 172.21.15.113:0/1900121772 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcd401de20 con 0x7fbce40fd150 2024-04-12T05:17:25.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.131+0000 7fbce9fd2700 1 -- 172.21.15.113:0/1900121772 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce41077e0 con 0x7fbce40fd150 2024-04-12T05:17:25.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.132+0000 7fbce9fd2700 1 -- 172.21.15.113:0/1900121772 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce4060390 con 0x7fbce40fd150 2024-04-12T05:17:25.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.133+0000 7fbce0ff9700 1 -- 172.21.15.113:0/1900121772 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbcd401d420 con 0x7fbce40fd150 2024-04-12T05:17:25.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.133+0000 7fbce0ff9700 1 --2- 172.21.15.113:0/1900121772 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbcd006ca30 0x7fbcd006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:25.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.134+0000 7fbce37fe700 1 --2- 172.21.15.113:0/1900121772 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbcd006ca30 0x7fbcd006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:25.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.134+0000 7fbce0ff9700 1 -- 172.21.15.113:0/1900121772 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6308+0+0 (secure 0 0 0) 0x7fbcd408cc50 con 0x7fbce40fd150 2024-04-12T05:17:25.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.134+0000 7fbce37fe700 1 --2- 172.21.15.113:0/1900121772 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbcd006ca30 0x7fbcd006eef0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fbccc004ee0 tx=0x7fbccc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:25.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.137+0000 7fbce0ff9700 1 -- 172.21.15.113:0/1900121772 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbcd405a710 con 0x7fbce40fd150 2024-04-12T05:17:25.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:24 smithi113 ceph-mon[103371]: pgmap v136: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%); 0 B/s, 28 objects/s recovering 2024-04-12T05:17:25.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.336+0000 7fbce9fd2700 1 -- 172.21.15.113:0/1900121772 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbce40feb40 con 0x7fbcd006ca30 2024-04-12T05:17:25.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.346+0000 7fbce0ff9700 1 -- 172.21.15.113:0/1900121772 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fbce40feb40 con 0x7fbcd006ca30 2024-04-12T05:17:25.348 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:17:25.348 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (5m) 59s ago 16m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:17:25.348 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (2m) 59s ago 16m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:17:25.348 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (2m) 25s ago 15m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:17:25.348 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (5m) 59s ago 16m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:17:25.348 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (11m) 59s ago 11m 120M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:17:25.348 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (11m) 59s ago 11m 18.2M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (11m) 25s ago 11m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (11m) 25s ago 11m 15.9M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (6m) 59s ago 17m 501M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (6m) 25s ago 15m 449M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (3m) 59s ago 17m 46.6M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (3m) 25s ago 15m 36.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (5m) 59s ago 16m 16.6M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (5m) 25s ago 15m 17.5M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (2m) 59s ago 15m 93.5M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (100s) 59s ago 14m 93.9M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (65s) 59s ago 14m 13.7M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (29s) 25s ago 13m 11.3M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (13m) 25s ago 13m 232M 2251M 16.2.15-1-g8963e9de 727f386679cb 27a8e1a554b8 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (12m) 25s ago 12m 232M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:17:25.349 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (5m) 25s ago 15m 60.2M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:17:25.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.348+0000 7fbce9fd2700 1 -- 172.21.15.113:0/1900121772 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbcd006ca30 msgr2=0x7fbcd006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.348+0000 7fbce9fd2700 1 --2- 172.21.15.113:0/1900121772 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbcd006ca30 0x7fbcd006eef0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fbccc004ee0 tx=0x7fbccc00b040).stop 2024-04-12T05:17:25.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.348+0000 7fbce9fd2700 1 -- 172.21.15.113:0/1900121772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbce40fd150 msgr2=0x7fbce410d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.348+0000 7fbce9fd2700 1 --2- 172.21.15.113:0/1900121772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbce40fd150 0x7fbce410d510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fbcd4000c00 tx=0x7fbcd4009cb0).stop 2024-04-12T05:17:25.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.348+0000 7fbce9fd2700 1 -- 172.21.15.113:0/1900121772 shutdown_connections 2024-04-12T05:17:25.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.348+0000 7fbce9fd2700 1 --2- 172.21.15.113:0/1900121772 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbcd006ca30 0x7fbcd006eef0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.348+0000 7fbce9fd2700 1 --2- 172.21.15.113:0/1900121772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbce40fc7f0 0x7fbce410d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.348+0000 7fbce9fd2700 1 --2- 172.21.15.113:0/1900121772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbce40fd150 0x7fbce410d510 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.348+0000 7fbce9fd2700 1 -- 172.21.15.113:0/1900121772 >> 172.21.15.113:0/1900121772 conn(0x7fbce40f23b0 msgr2=0x7fbce40fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:25.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.348+0000 7fbce9fd2700 1 -- 172.21.15.113:0/1900121772 shutdown_connections 2024-04-12T05:17:25.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.348+0000 7fbce9fd2700 1 -- 172.21.15.113:0/1900121772 wait complete. 2024-04-12T05:17:25.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.445+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/2094673630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680f9200 msgr2=0x7f0f68101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.445+0000 7f0f6ffd9700 1 --2- 172.21.15.113:0/2094673630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680f9200 0x7f0f68101910 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0f640047f0 tx=0x7f0f64009cb0).stop 2024-04-12T05:17:25.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.445+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/2094673630 shutdown_connections 2024-04-12T05:17:25.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.445+0000 7f0f6ffd9700 1 --2- 172.21.15.113:0/2094673630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680f9200 0x7f0f68101910 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.445+0000 7f0f6ffd9700 1 --2- 172.21.15.113:0/2094673630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f680f88a0 0x7f0f680f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.445+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/2094673630 >> 172.21.15.113:0/2094673630 conn(0x7f0f68070590 msgr2=0x7f0f680f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:25.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.445+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/2094673630 shutdown_connections 2024-04-12T05:17:25.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.445+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/2094673630 wait complete. 2024-04-12T05:17:25.449 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[54140]: debug 2024-04-12T05:17:25.030+0000 7fe7776a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-12T05:17:25.449 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[54140]: debug 2024-04-12T05:17:25.030+0000 7fe7776a3700 -1 osd.4 76 *** Got signal Terminated *** 2024-04-12T05:17:25.449 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:25 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[54140]: debug 2024-04-12T05:17:25.030+0000 7fe7776a3700 -1 osd.4 76 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:17:25.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.446+0000 7f0f6ffd9700 1 Processor -- start 2024-04-12T05:17:25.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.446+0000 7f0f6ffd9700 1 -- start start 2024-04-12T05:17:25.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.446+0000 7f0f6ffd9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f680f88a0 0x7f0f680f6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:25.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.446+0000 7f0f6ffd9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680f9200 0x7f0f680f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:25.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.446+0000 7f0f6ffd9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f680f7230 con 0x7f0f680f9200 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.446+0000 7f0f6ffd9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f680f7370 con 0x7f0f680f88a0 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.446+0000 7f0f6d574700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680f9200 0x7f0f680f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.446+0000 7f0f6d574700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680f9200 0x7f0f680f6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60834/0 (socket says 172.21.15.113:60834) 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.446+0000 7f0f6d574700 1 -- 172.21.15.113:0/4163148056 learned_addr learned my addr 172.21.15.113:0/4163148056 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.446+0000 7f0f6dd75700 1 --2- 172.21.15.113:0/4163148056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f680f88a0 0x7f0f680f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.447+0000 7f0f6d574700 1 -- 172.21.15.113:0/4163148056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f680f88a0 msgr2=0x7f0f680f6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.447+0000 7f0f6d574700 1 --2- 172.21.15.113:0/4163148056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f680f88a0 0x7f0f680f6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.447+0000 7f0f6d574700 1 -- 172.21.15.113:0/4163148056 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f64005040 con 0x7f0f680f9200 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.447+0000 7f0f6d574700 1 --2- 172.21.15.113:0/4163148056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680f9200 0x7f0f680f6d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f0f64009380 tx=0x7f0f64009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.447+0000 7f0f5effd700 1 -- 172.21.15.113:0/4163148056 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f64018070 con 0x7f0f680f9200 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.447+0000 7f0f5effd700 1 -- 172.21.15.113:0/4163148056 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f64007e90 con 0x7f0f680f9200 2024-04-12T05:17:25.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.447+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/4163148056 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f680f75c0 con 0x7f0f680f9200 2024-04-12T05:17:25.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.447+0000 7f0f5effd700 1 -- 172.21.15.113:0/4163148056 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f640123c0 con 0x7f0f680f9200 2024-04-12T05:17:25.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.447+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/4163148056 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f681147a0 con 0x7f0f680f9200 2024-04-12T05:17:25.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.448+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/4163148056 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6805f590 con 0x7f0f680f9200 2024-04-12T05:17:25.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.448+0000 7f0f5effd700 1 -- 172.21.15.113:0/4163148056 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0f6401d420 con 0x7f0f680f9200 2024-04-12T05:17:25.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.449+0000 7f0f5effd700 1 --2- 172.21.15.113:0/4163148056 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f5406caa0 0x7f0f5406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:25.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.449+0000 7f0f5effd700 1 -- 172.21.15.113:0/4163148056 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6308+0+0 (secure 0 0 0) 0x7f0f6408c8f0 con 0x7f0f680f9200 2024-04-12T05:17:25.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.449+0000 7f0f6dd75700 1 --2- 172.21.15.113:0/4163148056 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f5406caa0 0x7f0f5406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:25.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.449+0000 7f0f6dd75700 1 --2- 172.21.15.113:0/4163148056 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f5406caa0 0x7f0f5406ef60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f0f580048b0 tx=0x7f0f5800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:25.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.453+0000 7f0f5effd700 1 -- 172.21.15.113:0/4163148056 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0f64056960 con 0x7f0f680f9200 2024-04-12T05:17:25.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.724+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/4163148056 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0f68047540 con 0x7f0f680f9200 2024-04-12T05:17:25.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.724+0000 7f0f5effd700 1 -- 172.21.15.113:0/4163148056 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f0f64059f80 con 0x7f0f680f9200 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 2, 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:17:25.727 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:17:25.728 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:17:25.728 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 6, 2024-04-12T05:17:25.728 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 8 2024-04-12T05:17:25.728 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:17:25.728 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:17:25.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.726+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/4163148056 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f5406caa0 msgr2=0x7f0f5406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.726+0000 7f0f6ffd9700 1 --2- 172.21.15.113:0/4163148056 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f5406caa0 0x7f0f5406ef60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f0f580048b0 tx=0x7f0f5800b040).stop 2024-04-12T05:17:25.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.726+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/4163148056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680f9200 msgr2=0x7f0f680f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.726+0000 7f0f6ffd9700 1 --2- 172.21.15.113:0/4163148056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680f9200 0x7f0f680f6d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f0f64009380 tx=0x7f0f64009cb0).stop 2024-04-12T05:17:25.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.726+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/4163148056 shutdown_connections 2024-04-12T05:17:25.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.726+0000 7f0f6ffd9700 1 --2- 172.21.15.113:0/4163148056 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f5406caa0 0x7f0f5406ef60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.726+0000 7f0f6ffd9700 1 --2- 172.21.15.113:0/4163148056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f680f88a0 0x7f0f680f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.726+0000 7f0f6ffd9700 1 --2- 172.21.15.113:0/4163148056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f680f9200 0x7f0f680f6d40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.726+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/4163148056 >> 172.21.15.113:0/4163148056 conn(0x7f0f68070590 msgr2=0x7f0f681012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:25.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.726+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/4163148056 shutdown_connections 2024-04-12T05:17:25.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.726+0000 7f0f6ffd9700 1 -- 172.21.15.113:0/4163148056 wait complete. 2024-04-12T05:17:25.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 -- 172.21.15.113:0/2499111545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29340f9200 msgr2=0x7f2934101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 --2- 172.21.15.113:0/2499111545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29340f9200 0x7f2934101910 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2924004730 tx=0x7f2924009cb0).stop 2024-04-12T05:17:25.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 -- 172.21.15.113:0/2499111545 shutdown_connections 2024-04-12T05:17:25.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 --2- 172.21.15.113:0/2499111545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29340f9200 0x7f2934101910 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 --2- 172.21.15.113:0/2499111545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29340f88a0 0x7f29340f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 -- 172.21.15.113:0/2499111545 >> 172.21.15.113:0/2499111545 conn(0x7f29340f23b0 msgr2=0x7f29340f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:25.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 -- 172.21.15.113:0/2499111545 shutdown_connections 2024-04-12T05:17:25.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 -- 172.21.15.113:0/2499111545 wait complete. 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 Processor -- start 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 -- start start 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29340f88a0 0x7f293410e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29340f9200 0x7f293410e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.826+0000 7f293b7e2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293410ebe0 con 0x7f29340f9200 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f293b7e2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293410ed20 con 0x7f29340f88a0 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f2938d7d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29340f9200 0x7f293410e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f2938d7d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29340f9200 0x7f293410e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60866/0 (socket says 172.21.15.113:60866) 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f2938d7d700 1 -- 172.21.15.113:0/3192955247 learned_addr learned my addr 172.21.15.113:0/3192955247 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f293957e700 1 --2- 172.21.15.113:0/3192955247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29340f88a0 0x7f293410e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f2938d7d700 1 -- 172.21.15.113:0/3192955247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29340f88a0 msgr2=0x7f293410e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f2938d7d700 1 --2- 172.21.15.113:0/3192955247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29340f88a0 0x7f293410e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f2938d7d700 1 -- 172.21.15.113:0/3192955247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2930005040 con 0x7f29340f9200 2024-04-12T05:17:25.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f2938d7d700 1 --2- 172.21.15.113:0/3192955247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29340f9200 0x7f293410e5e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f2924009e30 tx=0x7f292400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:25.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f292a7fc700 1 -- 172.21.15.113:0/3192955247 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2924019070 con 0x7f29340f9200 2024-04-12T05:17:25.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f292a7fc700 1 -- 172.21.15.113:0/3192955247 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2924013380 con 0x7f29340f9200 2024-04-12T05:17:25.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f293b7e2700 1 -- 172.21.15.113:0/3192955247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2924005040 con 0x7f29340f9200 2024-04-12T05:17:25.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f292a7fc700 1 -- 172.21.15.113:0/3192955247 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2924014ca0 con 0x7f29340f9200 2024-04-12T05:17:25.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.827+0000 7f293b7e2700 1 -- 172.21.15.113:0/3192955247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29341086a0 con 0x7f29340f9200 2024-04-12T05:17:25.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.828+0000 7f293b7e2700 1 -- 172.21.15.113:0/3192955247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2934060370 con 0x7f29340f9200 2024-04-12T05:17:25.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.829+0000 7f292a7fc700 1 -- 172.21.15.113:0/3192955247 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f29240144b0 con 0x7f29340f9200 2024-04-12T05:17:25.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.829+0000 7f292a7fc700 1 --2- 172.21.15.113:0/3192955247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f292006caa0 0x7f292006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:25.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.829+0000 7f292a7fc700 1 -- 172.21.15.113:0/3192955247 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6308+0+0 (secure 0 0 0) 0x7f292408cb30 con 0x7f29340f9200 2024-04-12T05:17:25.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.829+0000 7f293957e700 1 --2- 172.21.15.113:0/3192955247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f292006caa0 0x7f292006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:25.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.830+0000 7f293957e700 1 --2- 172.21.15.113:0/3192955247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f292006caa0 0x7f292006ef60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f2930004870 tx=0x7f293000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:25.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:25.834+0000 7f292a7fc700 1 -- 172.21.15.113:0/3192955247 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f292405a630 con 0x7f29340f9200 2024-04-12T05:17:26.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.072+0000 7f293b7e2700 1 -- 172.21.15.113:0/3192955247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2934109770 con 0x7f29340f9200 2024-04-12T05:17:26.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.073+0000 7f292a7fc700 1 -- 172.21.15.113:0/3192955247 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1859 (secure 0 0 0) 0x7f292405a1c0 con 0x7f29340f9200 2024-04-12T05:17:26.075 INFO:teuthology.orchestra.run.smithi113.stdout:e18 2024-04-12T05:17:26.075 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:17:26.075 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:17:26.075 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:17:26.075 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:17:26.075 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:17:26.075 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:17:26.075 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 18 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:17:26.076 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=14426,1=24261} 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:17:26.077 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:17:26.078 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:17:26.078 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:17:26.078 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:17:26.078 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:17:26.078 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:17:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.075+0000 7f293b7e2700 1 -- 172.21.15.113:0/3192955247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f292006caa0 msgr2=0x7f292006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.075+0000 7f293b7e2700 1 --2- 172.21.15.113:0/3192955247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f292006caa0 0x7f292006ef60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f2930004870 tx=0x7f293000b040).stop 2024-04-12T05:17:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.075+0000 7f293b7e2700 1 -- 172.21.15.113:0/3192955247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29340f9200 msgr2=0x7f293410e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.075+0000 7f293b7e2700 1 --2- 172.21.15.113:0/3192955247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29340f9200 0x7f293410e5e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f2924009e30 tx=0x7f292400b040).stop 2024-04-12T05:17:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.075+0000 7f293b7e2700 1 -- 172.21.15.113:0/3192955247 shutdown_connections 2024-04-12T05:17:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.075+0000 7f293b7e2700 1 --2- 172.21.15.113:0/3192955247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f292006caa0 0x7f292006ef60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.075+0000 7f293b7e2700 1 --2- 172.21.15.113:0/3192955247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29340f88a0 0x7f293410e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.075+0000 7f293b7e2700 1 --2- 172.21.15.113:0/3192955247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29340f9200 0x7f293410e5e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.075+0000 7f293b7e2700 1 -- 172.21.15.113:0/3192955247 >> 172.21.15.113:0/3192955247 conn(0x7f29340f23b0 msgr2=0x7f29340fc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.075+0000 7f293b7e2700 1 -- 172.21.15.113:0/3192955247 shutdown_connections 2024-04-12T05:17:26.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.075+0000 7f293b7e2700 1 -- 172.21.15.113:0/3192955247 wait complete. 2024-04-12T05:17:26.080 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 18 2024-04-12T05:17:26.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:25 smithi113 ceph-mon[103371]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:26.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:25 smithi113 ceph-mon[103371]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:26.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:25 smithi113 ceph-mon[103371]: osd.4 marked itself down and dead 2024-04-12T05:17:26.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:25 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/4163148056' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.173+0000 7f47947f1700 1 -- 172.21.15.113:0/2360267965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c0fe360 msgr2=0x7f478c0f5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.173+0000 7f47947f1700 1 --2- 172.21.15.113:0/2360267965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c0fe360 0x7f478c0f5fc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f477c0047f0 tx=0x7f477c009cb0).stop 2024-04-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 -- 172.21.15.113:0/2360267965 shutdown_connections 2024-04-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 --2- 172.21.15.113:0/2360267965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c0fe360 0x7f478c0f5fc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 --2- 172.21.15.113:0/2360267965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f478c0fda00 0x7f478c0fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 -- 172.21.15.113:0/2360267965 >> 172.21.15.113:0/2360267965 conn(0x7f478c06cea0 msgr2=0x7f478c06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:26.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 -- 172.21.15.113:0/2360267965 shutdown_connections 2024-04-12T05:17:26.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 -- 172.21.15.113:0/2360267965 wait complete. 2024-04-12T05:17:26.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 Processor -- start 2024-04-12T05:17:26.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 -- start start 2024-04-12T05:17:26.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c0fda00 0x7f478c0f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:26.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f478c0fe360 0x7f478c0f7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:26.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c0f7c70 con 0x7f478c0fda00 2024-04-12T05:17:26.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.174+0000 7f47947f1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c0f7db0 con 0x7f478c0fe360 2024-04-12T05:17:26.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f479258d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c0fda00 0x7f478c0f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:26.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f479258d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c0fda00 0x7f478c0f7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60872/0 (socket says 172.21.15.113:60872) 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f479258d700 1 -- 172.21.15.113:0/1297911613 learned_addr learned my addr 172.21.15.113:0/1297911613 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f4791d8c700 1 --2- 172.21.15.113:0/1297911613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f478c0fe360 0x7f478c0f7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f479258d700 1 -- 172.21.15.113:0/1297911613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f478c0fe360 msgr2=0x7f478c0f7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f479258d700 1 --2- 172.21.15.113:0/1297911613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f478c0fe360 0x7f478c0f7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f479258d700 1 -- 172.21.15.113:0/1297911613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c005040 con 0x7f478c0fda00 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f479258d700 1 --2- 172.21.15.113:0/1297911613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c0fda00 0x7f478c0f7290 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f478800e910 tx=0x7f478800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f47837fe700 1 -- 172.21.15.113:0/1297911613 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4788007b40 con 0x7f478c0fda00 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f47837fe700 1 -- 172.21.15.113:0/1297911613 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f478800f040 con 0x7f478c0fda00 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f47947f1700 1 -- 172.21.15.113:0/1297911613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478c0f8000 con 0x7f478c0fda00 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f47837fe700 1 -- 172.21.15.113:0/1297911613 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4788014920 con 0x7f478c0fda00 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.175+0000 7f47947f1700 1 -- 172.21.15.113:0/1297911613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478c1147c0 con 0x7f478c0fda00 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.176+0000 7f47947f1700 1 -- 172.21.15.113:0/1297911613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478c047540 con 0x7f478c0fda00 2024-04-12T05:17:26.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.177+0000 7f47837fe700 1 -- 172.21.15.113:0/1297911613 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4788007ca0 con 0x7f478c0fda00 2024-04-12T05:17:26.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.177+0000 7f47837fe700 1 --2- 172.21.15.113:0/1297911613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f477806caa0 0x7f477806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:26.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.177+0000 7f47837fe700 1 -- 172.21.15.113:0/1297911613 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6308+0+0 (secure 0 0 0) 0x7f478808b830 con 0x7f478c0fda00 2024-04-12T05:17:26.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.177+0000 7f4791d8c700 1 --2- 172.21.15.113:0/1297911613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f477806caa0 0x7f477806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:26.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.177+0000 7f4791d8c700 1 --2- 172.21.15.113:0/1297911613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f477806caa0 0x7f477806ef60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f478c0f8a30 tx=0x7f477c009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:26.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.181+0000 7f47837fe700 1 -- 172.21.15.113:0/1297911613 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f47880558a0 con 0x7f478c0fda00 2024-04-12T05:17:26.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:25 smithi192 ceph-mon[85304]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:26.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:25 smithi192 ceph-mon[85304]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:26.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:25 smithi192 ceph-mon[85304]: osd.4 marked itself down and dead 2024-04-12T05:17:26.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:25 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/4163148056' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:26.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.385+0000 7f47947f1700 1 -- 172.21.15.113:0/1297911613 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f478c059120 con 0x7f477806caa0 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.387+0000 7f47837fe700 1 -- 172.21.15.113:0/1297911613 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f478c059120 con 0x7f477806caa0 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "10/21 daemons upgraded", 2024-04-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:17:26.390 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:17:26.390 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:17:26.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.389+0000 7f47947f1700 1 -- 172.21.15.113:0/1297911613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f477806caa0 msgr2=0x7f477806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:26.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.389+0000 7f47947f1700 1 --2- 172.21.15.113:0/1297911613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f477806caa0 0x7f477806ef60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f478c0f8a30 tx=0x7f477c009cb0).stop 2024-04-12T05:17:26.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.389+0000 7f47947f1700 1 -- 172.21.15.113:0/1297911613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c0fda00 msgr2=0x7f478c0f7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:26.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.389+0000 7f47947f1700 1 --2- 172.21.15.113:0/1297911613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c0fda00 0x7f478c0f7290 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f478800e910 tx=0x7f478800b040).stop 2024-04-12T05:17:26.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.389+0000 7f47947f1700 1 -- 172.21.15.113:0/1297911613 shutdown_connections 2024-04-12T05:17:26.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.389+0000 7f47947f1700 1 --2- 172.21.15.113:0/1297911613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f477806caa0 0x7f477806ef60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.389+0000 7f47947f1700 1 --2- 172.21.15.113:0/1297911613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f478c0fda00 0x7f478c0f7290 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.389+0000 7f47947f1700 1 --2- 172.21.15.113:0/1297911613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f478c0fe360 0x7f478c0f7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.389+0000 7f47947f1700 1 -- 172.21.15.113:0/1297911613 >> 172.21.15.113:0/1297911613 conn(0x7f478c06cea0 msgr2=0x7f478c0fbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.389+0000 7f47947f1700 1 -- 172.21.15.113:0/1297911613 shutdown_connections 2024-04-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.389+0000 7f47947f1700 1 -- 172.21.15.113:0/1297911613 wait complete. 2024-04-12T05:17:26.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.487+0000 7f716f801700 1 -- 172.21.15.113:0/1443284617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71680f88a0 msgr2=0x7f71680f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:26.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.487+0000 7f716f801700 1 --2- 172.21.15.113:0/1443284617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71680f88a0 0x7f71680f8c80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f71640047f0 tx=0x7f7164009cb0).stop 2024-04-12T05:17:26.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.487+0000 7f716f801700 1 -- 172.21.15.113:0/1443284617 shutdown_connections 2024-04-12T05:17:26.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.487+0000 7f716f801700 1 --2- 172.21.15.113:0/1443284617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71680f9200 0x7f7168101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.487+0000 7f716f801700 1 --2- 172.21.15.113:0/1443284617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71680f88a0 0x7f71680f8c80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.487+0000 7f716f801700 1 -- 172.21.15.113:0/1443284617 >> 172.21.15.113:0/1443284617 conn(0x7f7168070590 msgr2=0x7f71680f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:26.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.487+0000 7f716f801700 1 -- 172.21.15.113:0/1443284617 shutdown_connections 2024-04-12T05:17:26.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.487+0000 7f716f801700 1 -- 172.21.15.113:0/1443284617 wait complete. 2024-04-12T05:17:26.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.488+0000 7f716f801700 1 Processor -- start 2024-04-12T05:17:26.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.488+0000 7f716f801700 1 -- start start 2024-04-12T05:17:26.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.488+0000 7f716f801700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71680f88a0 0x7f716810e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:26.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.488+0000 7f716f801700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71680f9200 0x7f716810e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:26.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.488+0000 7f716f801700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716810ec20 con 0x7f71680f9200 2024-04-12T05:17:26.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.488+0000 7f716f801700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716810ed60 con 0x7f71680f88a0 2024-04-12T05:17:26.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.488+0000 7f716cd9c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71680f9200 0x7f716810e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.488+0000 7f716cd9c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71680f9200 0x7f716810e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60884/0 (socket says 172.21.15.113:60884) 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.488+0000 7f716cd9c700 1 -- 172.21.15.113:0/1055202407 learned_addr learned my addr 172.21.15.113:0/1055202407 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.488+0000 7f716d59d700 1 --2- 172.21.15.113:0/1055202407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71680f88a0 0x7f716810e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.489+0000 7f716cd9c700 1 -- 172.21.15.113:0/1055202407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71680f88a0 msgr2=0x7f716810e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.489+0000 7f716cd9c700 1 --2- 172.21.15.113:0/1055202407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71680f88a0 0x7f716810e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.489+0000 7f716cd9c700 1 -- 172.21.15.113:0/1055202407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7164005040 con 0x7f71680f9200 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.489+0000 7f716cd9c700 1 --2- 172.21.15.113:0/1055202407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71680f9200 0x7f716810e620 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f715800e8f0 tx=0x7f715800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.489+0000 7f715e7fc700 1 -- 172.21.15.113:0/1055202407 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7158007b40 con 0x7f71680f9200 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.489+0000 7f715e7fc700 1 -- 172.21.15.113:0/1055202407 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f715800f040 con 0x7f71680f9200 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.489+0000 7f715e7fc700 1 -- 172.21.15.113:0/1055202407 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7158014880 con 0x7f71680f9200 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.489+0000 7f716f801700 1 -- 172.21.15.113:0/1055202407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71681083f0 con 0x7f71680f9200 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.489+0000 7f716f801700 1 -- 172.21.15.113:0/1055202407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7168108930 con 0x7f71680f9200 2024-04-12T05:17:26.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.490+0000 7f716f801700 1 -- 172.21.15.113:0/1055202407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716805f590 con 0x7f71680f9200 2024-04-12T05:17:26.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.491+0000 7f715e7fc700 1 -- 172.21.15.113:0/1055202407 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7158009090 con 0x7f71680f9200 2024-04-12T05:17:26.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.491+0000 7f715e7fc700 1 --2- 172.21.15.113:0/1055202407 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f715406caa0 0x7f715406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:26.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.491+0000 7f715e7fc700 1 -- 172.21.15.113:0/1055202407 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6308+0+0 (secure 0 0 0) 0x7f715808c080 con 0x7f71680f9200 2024-04-12T05:17:26.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.491+0000 7f716d59d700 1 --2- 172.21.15.113:0/1055202407 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f715406caa0 0x7f715406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:26.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.492+0000 7f716d59d700 1 --2- 172.21.15.113:0/1055202407 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f715406caa0 0x7f715406ef60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f71640047c0 tx=0x7f7164009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:26.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.495+0000 7f715e7fc700 1 -- 172.21.15.113:0/1055202407 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7158059c00 con 0x7f71680f9200 2024-04-12T05:17:26.698 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:26 smithi192 bash[104281]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4 2024-04-12T05:17:26.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.762+0000 7f716f801700 1 -- 172.21.15.113:0/1055202407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7168047540 con 0x7f71680f9200 2024-04-12T05:17:26.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.762+0000 7f715e7fc700 1 -- 172.21.15.113:0/1055202407 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+353 (secure 0 0 0) 0x7f7158019180 con 0x7f71680f9200 2024-04-12T05:17:26.764 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 725/324 objects degraded (223.765%), 1 pg degraded 2024-04-12T05:17:26.765 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T05:17:26.765 INFO:teuthology.orchestra.run.smithi113.stdout: osd.4 (root=default,host=smithi192) is down 2024-04-12T05:17:26.765 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 725/324 objects degraded (223.765%), 1 pg degraded 2024-04-12T05:17:26.765 INFO:teuthology.orchestra.run.smithi113.stdout: pg 3.1c is active+recovery_wait+undersized+degraded+remapped, acting [5,1] 2024-04-12T05:17:26.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.764+0000 7f716f801700 1 -- 172.21.15.113:0/1055202407 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f715406caa0 msgr2=0x7f715406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:26.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.764+0000 7f716f801700 1 --2- 172.21.15.113:0/1055202407 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f715406caa0 0x7f715406ef60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f71640047c0 tx=0x7f7164009cb0).stop 2024-04-12T05:17:26.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.764+0000 7f716f801700 1 -- 172.21.15.113:0/1055202407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71680f9200 msgr2=0x7f716810e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:26.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.764+0000 7f716f801700 1 --2- 172.21.15.113:0/1055202407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71680f9200 0x7f716810e620 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f715800e8f0 tx=0x7f715800b040).stop 2024-04-12T05:17:26.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.764+0000 7f716f801700 1 -- 172.21.15.113:0/1055202407 shutdown_connections 2024-04-12T05:17:26.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.764+0000 7f716f801700 1 --2- 172.21.15.113:0/1055202407 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f715406caa0 0x7f715406ef60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.764+0000 7f716f801700 1 --2- 172.21.15.113:0/1055202407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71680f88a0 0x7f716810e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.764+0000 7f716f801700 1 --2- 172.21.15.113:0/1055202407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71680f9200 0x7f716810e620 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:26.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.764+0000 7f716f801700 1 -- 172.21.15.113:0/1055202407 >> 172.21.15.113:0/1055202407 conn(0x7f7168070590 msgr2=0x7f71680fc250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:26.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.764+0000 7f716f801700 1 -- 172.21.15.113:0/1055202407 shutdown_connections 2024-04-12T05:17:26.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:26.764+0000 7f716f801700 1 -- 172.21.15.113:0/1055202407 wait complete. 2024-04-12T05:17:27.118 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:26 smithi192 ceph-mon[85304]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:27.119 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:26 smithi192 ceph-mon[85304]: pgmap v137: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%); 0 B/s, 19 objects/s recovering 2024-04-12T05:17:27.119 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:26 smithi192 ceph-mon[85304]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:17:27.119 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:26 smithi192 ceph-mon[85304]: osdmap e77: 6 total, 5 up, 6 in 2024-04-12T05:17:27.119 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:26 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3192955247' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:17:27.119 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:26 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1055202407' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:17:27.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:26 smithi113 ceph-mon[103371]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:27.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:26 smithi113 ceph-mon[103371]: pgmap v137: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 725/324 objects degraded (223.765%); 0 B/s, 19 objects/s recovering 2024-04-12T05:17:27.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:26 smithi113 ceph-mon[103371]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:17:27.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:26 smithi113 ceph-mon[103371]: osdmap e77: 6 total, 5 up, 6 in 2024-04-12T05:17:27.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:26 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3192955247' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:17:27.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:26 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1055202407' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:17:28.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:27 smithi113 ceph-mon[103371]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:28.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:27 smithi113 ceph-mon[103371]: osdmap e78: 6 total, 5 up, 6 in 2024-04-12T05:17:28.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:27 smithi192 ceph-mon[85304]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:28.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:27 smithi192 ceph-mon[85304]: osdmap e78: 6 total, 5 up, 6 in 2024-04-12T05:17:28.695 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:28 smithi192 systemd[1]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.4.service: Succeeded. 2024-04-12T05:17:28.695 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:28 smithi192 systemd[1]: Stopped Ceph osd.4 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:17:28.695 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:28 smithi192 systemd[1]: Starting Ceph osd.4 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:17:28.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:28 smithi192 ceph-mon[85304]: pgmap v140: 65 pgs: 5 stale+active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+undersized+remapped, 2 active+undersized, 54 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 730/324 objects degraded (225.309%); 0 B/s, 28 objects/s recovering 2024-04-12T05:17:28.948 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:28 smithi192 ceph-mon[85304]: Health check update: Degraded data redundancy: 730/324 objects degraded (225.309%), 3 pgs degraded (PG_DEGRADED) 2024-04-12T05:17:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:28 smithi113 ceph-mon[103371]: pgmap v140: 65 pgs: 5 stale+active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+undersized+remapped, 2 active+undersized, 54 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 730/324 objects degraded (225.309%); 0 B/s, 28 objects/s recovering 2024-04-12T05:17:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:28 smithi113 ceph-mon[103371]: Health check update: Degraded data redundancy: 730/324 objects degraded (225.309%), 3 pgs degraded (PG_DEGRADED) 2024-04-12T05:17:29.698 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:29 smithi192 bash[104553]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-12T05:17:29.699 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:29 smithi192 bash[104553]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-12T05:17:29.699 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:29 smithi192 bash[104553]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-12T05:17:29.699 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:29 smithi192 bash[104553]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-12T05:17:29.699 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:29 smithi192 bash[104553]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-12T05:17:29.699 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:29 smithi192 bash[104553]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-12T05:17:29.699 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:29 smithi192 bash[104553]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-12T05:17:30.884 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:30 smithi192 ceph-osd[104871]: -- 172.21.15.192:0/1152895975 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564801ff11e0 con 0x564801fc0c00 2024-04-12T05:17:30.884 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:30 smithi192 bash[104833]: 0bcafbab30e6158bf51fe334b7e9d94b53afa113a37dfea48a5e2333e610621b 2024-04-12T05:17:30.884 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:30 smithi192 systemd[1]: Started Ceph osd.4 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:17:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:30 smithi113 ceph-mon[103371]: pgmap v141: 65 pgs: 2 stale+active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 1 active+recovering+undersized+remapped, 10 active+undersized, 42 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 751/324 objects degraded (231.790%); 0 B/s, 28 objects/s recovering 2024-04-12T05:17:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:17:31.198 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:31 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:17:31.103+0000 7f8497f0f7c0 -1 Falling back to public interface 2024-04-12T05:17:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:30 smithi192 ceph-mon[85304]: pgmap v141: 65 pgs: 2 stale+active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 1 active+recovering+undersized+remapped, 10 active+undersized, 42 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 751/324 objects degraded (231.790%); 0 B/s, 28 objects/s recovering 2024-04-12T05:17:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:31.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:17:33.139 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:32 smithi192 ceph-mon[85304]: pgmap v142: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovering+undersized+remapped, 16 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 779/324 objects degraded (240.432%); 0 B/s, 14 objects/s recovering 2024-04-12T05:17:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:32 smithi113 ceph-mon[103371]: pgmap v142: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovering+undersized+remapped, 16 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 779/324 objects degraded (240.432%); 0 B/s, 14 objects/s recovering 2024-04-12T05:17:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:34 smithi113 ceph-mon[103371]: pgmap v143: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovering+undersized+remapped, 16 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 779/324 objects degraded (240.432%); 0 B/s, 29 objects/s recovering 2024-04-12T05:17:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:34 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:34 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:34 smithi192 ceph-mon[85304]: pgmap v143: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovering+undersized+remapped, 16 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 779/324 objects degraded (240.432%); 0 B/s, 29 objects/s recovering 2024-04-12T05:17:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:34 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:34 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:36.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:36.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:36.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:36 smithi192 ceph-mon[85304]: pgmap v144: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovering+undersized+remapped, 16 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 779/324 objects degraded (240.432%); 0 B/s, 23 objects/s recovering 2024-04-12T05:17:36.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:36.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:36.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:36 smithi113 ceph-mon[103371]: pgmap v144: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovering+undersized+remapped, 16 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 779/324 objects degraded (240.432%); 0 B/s, 23 objects/s recovering 2024-04-12T05:17:38.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:38.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:17:38.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:37 smithi113 ceph-mon[103371]: Health check update: Degraded data redundancy: 779/324 objects degraded (240.432%), 17 pgs degraded (PG_DEGRADED) 2024-04-12T05:17:38.199 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:17:38.010+0000 7f8497f0f7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-12T05:17:38.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:38.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:17:38.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:37 smithi192 ceph-mon[85304]: Health check update: Degraded data redundancy: 779/324 objects degraded (240.432%), 17 pgs degraded (PG_DEGRADED) 2024-04-12T05:17:38.698 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:38 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:17:38.575+0000 7f8497f0f7c0 -1 osd.4 76 log_to_monitors true 2024-04-12T05:17:39.048 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:38 smithi192 ceph-mon[85304]: pgmap v145: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovering+undersized+remapped, 16 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 779/324 objects degraded (240.432%); 0 B/s, 31 objects/s recovering 2024-04-12T05:17:39.048 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:38 smithi192 ceph-mon[85304]: from='osd.4 [v2:172.21.15.192:6808/2295139175,v1:172.21.15.192:6809/2295139175]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:17:39.048 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:38 smithi192 ceph-mon[85304]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:17:39.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:38 smithi113 ceph-mon[103371]: pgmap v145: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovering+undersized+remapped, 16 active+undersized, 31 active+clean; 259 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 779/324 objects degraded (240.432%); 0 B/s, 31 objects/s recovering 2024-04-12T05:17:39.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:38 smithi113 ceph-mon[103371]: from='osd.4 [v2:172.21.15.192:6808/2295139175,v1:172.21.15.192:6809/2295139175]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:17:39.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:38 smithi113 ceph-mon[103371]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T05:17:39.948 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:17:39.662+0000 7f8485361700 -1 osd.4 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:17:39.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T05:17:39.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: osdmap e79: 6 total, 5 up, 6 in 2024-04-12T05:17:39.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='osd.4 [v2:172.21.15.192:6808/2295139175,v1:172.21.15.192:6809/2295139175]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:17:39.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:17:39.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:39.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:39.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:17:39.949 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:17:39.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:39.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:17:39.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:39.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:39.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:39.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:17:39.950 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:39 smithi192 ceph-mon[85304]: from='osd.4 ' entity='osd.4' 2024-04-12T05:17:40.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T05:17:40.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: osdmap e79: 6 total, 5 up, 6 in 2024-04-12T05:17:40.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='osd.4 [v2:172.21.15.192:6808/2295139175,v1:172.21.15.192:6809/2295139175]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:17:40.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:17:40.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:40.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:40.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:17:40.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:17:40.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:40.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:17:40.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:40.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:40.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:40.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:17:40.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:39 smithi113 ceph-mon[103371]: from='osd.4 ' entity='osd.4' 2024-04-12T05:17:41.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:40 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:17:41.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:40 smithi113 ceph-mon[103371]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-12T05:17:41.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:40 smithi113 ceph-mon[103371]: pgmap v147: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovering+undersized+remapped, 16 active+undersized, 31 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 779/324 objects degraded (240.432%); 0 B/s, 23 objects/s recovering 2024-04-12T05:17:41.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:40 smithi113 ceph-mon[103371]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:17:41.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:40 smithi113 ceph-mon[103371]: osd.4 [v2:172.21.15.192:6808/2295139175,v1:172.21.15.192:6809/2295139175] boot 2024-04-12T05:17:41.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:40 smithi113 ceph-mon[103371]: osdmap e80: 6 total, 6 up, 6 in 2024-04-12T05:17:41.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:17:41.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:40 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:17:41.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:40 smithi192 ceph-mon[85304]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-12T05:17:41.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:40 smithi192 ceph-mon[85304]: pgmap v147: 65 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+undersized+degraded, 1 active+recovering+undersized+remapped, 16 active+undersized, 31 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 779/324 objects degraded (240.432%); 0 B/s, 23 objects/s recovering 2024-04-12T05:17:41.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:40 smithi192 ceph-mon[85304]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:17:41.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:40 smithi192 ceph-mon[85304]: osd.4 [v2:172.21.15.192:6808/2295139175,v1:172.21.15.192:6809/2295139175] boot 2024-04-12T05:17:41.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:40 smithi192 ceph-mon[85304]: osdmap e80: 6 total, 6 up, 6 in 2024-04-12T05:17:41.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T05:17:42.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:41 smithi113 ceph-mon[103371]: osdmap e81: 6 total, 6 up, 6 in 2024-04-12T05:17:42.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:41 smithi192 ceph-mon[85304]: osdmap e81: 6 total, 6 up, 6 in 2024-04-12T05:17:43.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:42 smithi113 ceph-mon[103371]: pgmap v150: 65 pgs: 11 peering, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+undersized+degraded, 1 active+recovering+undersized+remapped, 11 active+undersized, 31 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 753/324 objects degraded (232.407%); 0 B/s, 19 objects/s recovering 2024-04-12T05:17:43.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:42 smithi113 ceph-mon[103371]: Health check update: Degraded data redundancy: 753/324 objects degraded (232.407%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T05:17:43.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:42 smithi192 ceph-mon[85304]: pgmap v150: 65 pgs: 11 peering, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+undersized+degraded, 1 active+recovering+undersized+remapped, 11 active+undersized, 31 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 753/324 objects degraded (232.407%); 0 B/s, 19 objects/s recovering 2024-04-12T05:17:43.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:42 smithi192 ceph-mon[85304]: Health check update: Degraded data redundancy: 753/324 objects degraded (232.407%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T05:17:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:44 smithi113 ceph-mon[103371]: pgmap v151: 65 pgs: 11 peering, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 3 active+undersized, 47 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 6/324 objects degraded (1.852%); 0 B/s, 19 objects/s recovering 2024-04-12T05:17:45.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:44 smithi192 ceph-mon[85304]: pgmap v151: 65 pgs: 11 peering, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 3 active+undersized, 47 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 6/324 objects degraded (1.852%); 0 B/s, 19 objects/s recovering 2024-04-12T05:17:46.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:45 smithi113 ceph-mon[103371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/324 objects degraded (1.852%), 3 pgs degraded) 2024-04-12T05:17:46.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:45 smithi113 ceph-mon[103371]: Cluster is now healthy 2024-04-12T05:17:46.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:45 smithi192 ceph-mon[85304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/324 objects degraded (1.852%), 3 pgs degraded) 2024-04-12T05:17:46.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:45 smithi192 ceph-mon[85304]: Cluster is now healthy 2024-04-12T05:17:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:46 smithi113 ceph-mon[103371]: pgmap v152: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 16 objects/s recovering 2024-04-12T05:17:47.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:46 smithi192 ceph-mon[85304]: pgmap v152: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 16 objects/s recovering 2024-04-12T05:17:49.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:48 smithi113 ceph-mon[103371]: pgmap v153: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 28 objects/s recovering 2024-04-12T05:17:49.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:48 smithi192 ceph-mon[85304]: pgmap v153: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 28 objects/s recovering 2024-04-12T05:17:51.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:50 smithi113 ceph-mon[103371]: pgmap v154: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 23 objects/s recovering 2024-04-12T05:17:51.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:50 smithi192 ceph-mon[85304]: pgmap v154: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 23 objects/s recovering 2024-04-12T05:17:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:52 smithi113 ceph-mon[103371]: pgmap v155: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 21 objects/s recovering 2024-04-12T05:17:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:17:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.9"}]: dispatch 2024-04-12T05:17:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:52 smithi192 ceph-mon[85304]: pgmap v155: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 21 objects/s recovering 2024-04-12T05:17:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:17:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:17:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.9"}]: dispatch 2024-04-12T05:17:54.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.9"}]': finished 2024-04-12T05:17:54.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:53 smithi192 ceph-mon[85304]: osdmap e82: 6 total, 6 up, 6 in 2024-04-12T05:17:54.350 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "2.9"}]': finished 2024-04-12T05:17:54.351 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:53 smithi113 ceph-mon[103371]: osdmap e82: 6 total, 6 up, 6 in 2024-04-12T05:17:55.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:54 smithi192 ceph-mon[85304]: pgmap v157: 65 pgs: 1 peering, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 23 objects/s recovering 2024-04-12T05:17:55.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:54 smithi192 ceph-mon[85304]: osdmap e83: 6 total, 6 up, 6 in 2024-04-12T05:17:55.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:54 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:17:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:54 smithi113 ceph-mon[103371]: pgmap v157: 65 pgs: 1 peering, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 23 objects/s recovering 2024-04-12T05:17:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:54 smithi113 ceph-mon[103371]: osdmap e83: 6 total, 6 up, 6 in 2024-04-12T05:17:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:54 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:17:56.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:55 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:17:56.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:55 smithi192 ceph-mon[85304]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-12T05:17:56.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:55 smithi192 ceph-mon[85304]: osdmap e84: 6 total, 6 up, 6 in 2024-04-12T05:17:56.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:55 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:17:56.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:55 smithi113 ceph-mon[103371]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-12T05:17:56.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:55 smithi113 ceph-mon[103371]: osdmap e84: 6 total, 6 up, 6 in 2024-04-12T05:17:56.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.874+0000 7f29fb65a700 1 -- 172.21.15.113:0/2047737246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29f40fe250 msgr2=0x7f29f40f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:56.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.874+0000 7f29fb65a700 1 --2- 172.21.15.113:0/2047737246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29f40fe250 0x7f29f40f7a30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f29f00047f0 tx=0x7f29f0009cb0).stop 2024-04-12T05:17:56.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.874+0000 7f29fb65a700 1 -- 172.21.15.113:0/2047737246 shutdown_connections 2024-04-12T05:17:56.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.874+0000 7f29fb65a700 1 --2- 172.21.15.113:0/2047737246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29f40fe250 0x7f29f40f7a30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:56.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.874+0000 7f29fb65a700 1 --2- 172.21.15.113:0/2047737246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40fd8f0 0x7f29f40fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:56.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.874+0000 7f29fb65a700 1 -- 172.21.15.113:0/2047737246 >> 172.21.15.113:0/2047737246 conn(0x7f29f40f34b0 msgr2=0x7f29f40f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:56.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.874+0000 7f29fb65a700 1 -- 172.21.15.113:0/2047737246 shutdown_connections 2024-04-12T05:17:56.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.874+0000 7f29fb65a700 1 -- 172.21.15.113:0/2047737246 wait complete. 2024-04-12T05:17:56.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29fb65a700 1 Processor -- start 2024-04-12T05:17:56.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29fb65a700 1 -- start start 2024-04-12T05:17:56.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29fb65a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40fd8f0 0x7f29f410e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29fb65a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29f40fe250 0x7f29f410e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29fb65a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f410ec30 con 0x7f29f40fe250 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29fb65a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f410ed70 con 0x7f29f40fd8f0 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29f8bf5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29f40fe250 0x7f29f410e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29f8bf5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29f40fe250 0x7f29f410e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55638/0 (socket says 172.21.15.113:55638) 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29f93f6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40fd8f0 0x7f29f410e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29f8bf5700 1 -- 172.21.15.113:0/430613827 learned_addr learned my addr 172.21.15.113:0/430613827 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29f8bf5700 1 -- 172.21.15.113:0/430613827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40fd8f0 msgr2=0x7f29f410e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29f8bf5700 1 --2- 172.21.15.113:0/430613827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40fd8f0 0x7f29f410e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29f8bf5700 1 -- 172.21.15.113:0/430613827 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f0005040 con 0x7f29f40fe250 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29f93f6700 1 --2- 172.21.15.113:0/430613827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40fd8f0 0x7f29f410e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29f8bf5700 1 --2- 172.21.15.113:0/430613827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29f40fe250 0x7f29f410e630 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f29f0009360 tx=0x7f29f0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.875+0000 7f29e67fc700 1 -- 172.21.15.113:0/430613827 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29f0018070 con 0x7f29f40fe250 2024-04-12T05:17:56.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.876+0000 7f29e67fc700 1 -- 172.21.15.113:0/430613827 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29f0007e90 con 0x7f29f40fe250 2024-04-12T05:17:56.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.876+0000 7f29e67fc700 1 -- 172.21.15.113:0/430613827 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29f00123c0 con 0x7f29f40fe250 2024-04-12T05:17:56.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.876+0000 7f29fb65a700 1 -- 172.21.15.113:0/430613827 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f41083a0 con 0x7f29f40fe250 2024-04-12T05:17:56.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.876+0000 7f29fb65a700 1 -- 172.21.15.113:0/430613827 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f4108900 con 0x7f29f40fe250 2024-04-12T05:17:56.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.877+0000 7f29fb65a700 1 -- 172.21.15.113:0/430613827 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f405f590 con 0x7f29f40fe250 2024-04-12T05:17:56.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.877+0000 7f29e67fc700 1 -- 172.21.15.113:0/430613827 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f29f001d420 con 0x7f29f40fe250 2024-04-12T05:17:56.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.877+0000 7f29e67fc700 1 --2- 172.21.15.113:0/430613827 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f29e006cab0 0x7f29e006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:56.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.877+0000 7f29e67fc700 1 -- 172.21.15.113:0/430613827 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6250+0+0 (secure 0 0 0) 0x7f29f008caa0 con 0x7f29f40fe250 2024-04-12T05:17:56.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.877+0000 7f29f93f6700 1 --2- 172.21.15.113:0/430613827 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f29e006cab0 0x7f29e006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:56.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.878+0000 7f29f93f6700 1 --2- 172.21.15.113:0/430613827 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f29e006cab0 0x7f29e006ef70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f29e8011510 tx=0x7f29e800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:56.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:56.882+0000 7f29e67fc700 1 -- 172.21.15.113:0/430613827 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f29f005a5e0 con 0x7f29f40fe250 2024-04-12T05:17:57.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.092+0000 7f29fb65a700 1 -- 172.21.15.113:0/430613827 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f29f41098d0 con 0x7f29e006cab0 2024-04-12T05:17:57.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.093+0000 7f29e67fc700 1 -- 172.21.15.113:0/430613827 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f29f41098d0 con 0x7f29e006cab0 2024-04-12T05:17:57.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.095+0000 7f29fb65a700 1 -- 172.21.15.113:0/430613827 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f29e006cab0 msgr2=0x7f29e006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.095+0000 7f29fb65a700 1 --2- 172.21.15.113:0/430613827 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f29e006cab0 0x7f29e006ef70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f29e8011510 tx=0x7f29e800b040).stop 2024-04-12T05:17:57.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.095+0000 7f29fb65a700 1 -- 172.21.15.113:0/430613827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29f40fe250 msgr2=0x7f29f410e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.095+0000 7f29fb65a700 1 --2- 172.21.15.113:0/430613827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29f40fe250 0x7f29f410e630 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f29f0009360 tx=0x7f29f0009cb0).stop 2024-04-12T05:17:57.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.095+0000 7f29fb65a700 1 -- 172.21.15.113:0/430613827 shutdown_connections 2024-04-12T05:17:57.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.095+0000 7f29fb65a700 1 --2- 172.21.15.113:0/430613827 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f29e006cab0 0x7f29e006ef70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.095+0000 7f29fb65a700 1 --2- 172.21.15.113:0/430613827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40fd8f0 0x7f29f410e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.095+0000 7f29fb65a700 1 --2- 172.21.15.113:0/430613827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29f40fe250 0x7f29f410e630 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.095+0000 7f29fb65a700 1 -- 172.21.15.113:0/430613827 >> 172.21.15.113:0/430613827 conn(0x7f29f40f34b0 msgr2=0x7f29f40fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:57.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.095+0000 7f29fb65a700 1 -- 172.21.15.113:0/430613827 shutdown_connections 2024-04-12T05:17:57.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.095+0000 7f29fb65a700 1 -- 172.21.15.113:0/430613827 wait complete. 2024-04-12T05:17:57.107 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:17:57.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.194+0000 7f1e92e75700 1 -- 172.21.15.113:0/2986530328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e8c0fd910 msgr2=0x7f1e8c0fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.194+0000 7f1e92e75700 1 --2- 172.21.15.113:0/2986530328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e8c0fd910 0x7f1e8c0fdcf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f1e800047f0 tx=0x7f1e80009cb0).stop 2024-04-12T05:17:57.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 -- 172.21.15.113:0/2986530328 shutdown_connections 2024-04-12T05:17:57.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 --2- 172.21.15.113:0/2986530328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e8c0fe270 0x7f1e8c0f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 --2- 172.21.15.113:0/2986530328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e8c0fd910 0x7f1e8c0fdcf0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 -- 172.21.15.113:0/2986530328 >> 172.21.15.113:0/2986530328 conn(0x7f1e8c0f3530 msgr2=0x7f1e8c0f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:57.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 -- 172.21.15.113:0/2986530328 shutdown_connections 2024-04-12T05:17:57.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 -- 172.21.15.113:0/2986530328 wait complete. 2024-04-12T05:17:57.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 Processor -- start 2024-04-12T05:17:57.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 -- start start 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e8c0fd910 0x7f1e8c10e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e8c0fe270 0x7f1e8c10e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c10ec30 con 0x7f1e8c0fe270 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.195+0000 7f1e92e75700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c10ed70 con 0x7f1e8c0fd910 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e8bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e8c0fe270 0x7f1e8c10e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e8bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e8c0fe270 0x7f1e8c10e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55642/0 (socket says 172.21.15.113:55642) 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e8bfff700 1 -- 172.21.15.113:0/2625229232 learned_addr learned my addr 172.21.15.113:0/2625229232 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e90c11700 1 --2- 172.21.15.113:0/2625229232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e8c0fd910 0x7f1e8c10e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e8bfff700 1 -- 172.21.15.113:0/2625229232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e8c0fd910 msgr2=0x7f1e8c10e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e8bfff700 1 --2- 172.21.15.113:0/2625229232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e8c0fd910 0x7f1e8c10e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e8bfff700 1 -- 172.21.15.113:0/2625229232 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e80005040 con 0x7f1e8c0fe270 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e8bfff700 1 --2- 172.21.15.113:0/2625229232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e8c0fe270 0x7f1e8c10e630 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f1e7800e940 tx=0x7f1e7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e89ffb700 1 -- 172.21.15.113:0/2625229232 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e78007b40 con 0x7f1e8c0fe270 2024-04-12T05:17:57.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e89ffb700 1 -- 172.21.15.113:0/2625229232 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e7800f040 con 0x7f1e8c0fe270 2024-04-12T05:17:57.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e92e75700 1 -- 172.21.15.113:0/2625229232 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c1083a0 con 0x7f1e8c0fe270 2024-04-12T05:17:57.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e89ffb700 1 -- 172.21.15.113:0/2625229232 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e780148e0 con 0x7f1e8c0fe270 2024-04-12T05:17:57.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.196+0000 7f1e92e75700 1 -- 172.21.15.113:0/2625229232 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c108960 con 0x7f1e8c0fe270 2024-04-12T05:17:57.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.197+0000 7f1e92e75700 1 -- 172.21.15.113:0/2625229232 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c05f570 con 0x7f1e8c0fe270 2024-04-12T05:17:57.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.198+0000 7f1e89ffb700 1 -- 172.21.15.113:0/2625229232 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1e78007ca0 con 0x7f1e8c0fe270 2024-04-12T05:17:57.201 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:56 smithi192 ceph-mon[85304]: pgmap v160: 65 pgs: 1 peering, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 19 objects/s recovering 2024-04-12T05:17:57.201 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:56 smithi192 ceph-mon[85304]: osdmap e85: 6 total, 6 up, 6 in 2024-04-12T05:17:57.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.198+0000 7f1e89ffb700 1 --2- 172.21.15.113:0/2625229232 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1e7c06caa0 0x7f1e7c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:57.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.198+0000 7f1e89ffb700 1 -- 172.21.15.113:0/2625229232 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6250+0+0 (secure 0 0 0) 0x7f1e7808b7d0 con 0x7f1e8c0fe270 2024-04-12T05:17:57.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.198+0000 7f1e90c11700 1 --2- 172.21.15.113:0/2625229232 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1e7c06caa0 0x7f1e7c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:57.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.198+0000 7f1e90c11700 1 --2- 172.21.15.113:0/2625229232 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1e7c06caa0 0x7f1e7c06ef60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f1e800047c0 tx=0x7f1e80009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:57.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.202+0000 7f1e89ffb700 1 -- 172.21.15.113:0/2625229232 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1e78059310 con 0x7f1e8c0fe270 2024-04-12T05:17:57.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.407+0000 7f1e92e75700 1 -- 172.21.15.113:0/2625229232 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e8c0ffc60 con 0x7f1e7c06caa0 2024-04-12T05:17:57.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.408+0000 7f1e89ffb700 1 -- 172.21.15.113:0/2625229232 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f1e8c0ffc60 con 0x7f1e7c06caa0 2024-04-12T05:17:57.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.411+0000 7f1e92e75700 1 -- 172.21.15.113:0/2625229232 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1e7c06caa0 msgr2=0x7f1e7c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.411+0000 7f1e92e75700 1 --2- 172.21.15.113:0/2625229232 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1e7c06caa0 0x7f1e7c06ef60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f1e800047c0 tx=0x7f1e80009cb0).stop 2024-04-12T05:17:57.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.411+0000 7f1e92e75700 1 -- 172.21.15.113:0/2625229232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e8c0fe270 msgr2=0x7f1e8c10e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.411+0000 7f1e92e75700 1 --2- 172.21.15.113:0/2625229232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e8c0fe270 0x7f1e8c10e630 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f1e7800e940 tx=0x7f1e7800b040).stop 2024-04-12T05:17:57.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.411+0000 7f1e92e75700 1 -- 172.21.15.113:0/2625229232 shutdown_connections 2024-04-12T05:17:57.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.411+0000 7f1e92e75700 1 --2- 172.21.15.113:0/2625229232 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1e7c06caa0 0x7f1e7c06ef60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.411+0000 7f1e92e75700 1 --2- 172.21.15.113:0/2625229232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e8c0fd910 0x7f1e8c10e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.411+0000 7f1e92e75700 1 --2- 172.21.15.113:0/2625229232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e8c0fe270 0x7f1e8c10e630 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.411+0000 7f1e92e75700 1 -- 172.21.15.113:0/2625229232 >> 172.21.15.113:0/2625229232 conn(0x7f1e8c0f3530 msgr2=0x7f1e8c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:57.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.411+0000 7f1e92e75700 1 -- 172.21.15.113:0/2625229232 shutdown_connections 2024-04-12T05:17:57.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.411+0000 7f1e92e75700 1 -- 172.21.15.113:0/2625229232 wait complete. 2024-04-12T05:17:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:56 smithi113 ceph-mon[103371]: pgmap v160: 65 pgs: 1 peering, 1 active+recovering+undersized+remapped, 63 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 19 objects/s recovering 2024-04-12T05:17:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:56 smithi113 ceph-mon[103371]: osdmap e85: 6 total, 6 up, 6 in 2024-04-12T05:17:57.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.509+0000 7fc131300700 1 -- 172.21.15.113:0/1454870533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc12c0fd150 msgr2=0x7fc12c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.509+0000 7fc131300700 1 --2- 172.21.15.113:0/1454870533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc12c0fd150 0x7fc12c0f6930 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fc11c0047f0 tx=0x7fc11c009cb0).stop 2024-04-12T05:17:57.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.509+0000 7fc131300700 1 -- 172.21.15.113:0/1454870533 shutdown_connections 2024-04-12T05:17:57.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.509+0000 7fc131300700 1 --2- 172.21.15.113:0/1454870533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc12c0fd150 0x7fc12c0f6930 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.509+0000 7fc131300700 1 --2- 172.21.15.113:0/1454870533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc12c0fc7f0 0x7fc12c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.509+0000 7fc131300700 1 -- 172.21.15.113:0/1454870533 >> 172.21.15.113:0/1454870533 conn(0x7fc12c0f23b0 msgr2=0x7fc12c0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.509+0000 7fc131300700 1 -- 172.21.15.113:0/1454870533 shutdown_connections 2024-04-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.509+0000 7fc131300700 1 -- 172.21.15.113:0/1454870533 wait complete. 2024-04-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc131300700 1 Processor -- start 2024-04-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc131300700 1 -- start start 2024-04-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc131300700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc12c0fc7f0 0x7fc12c110360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc131300700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc12c0fd150 0x7fc12c110850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc131300700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c110e50 con 0x7fc12c0fd150 2024-04-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc131300700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c110f90 con 0x7fc12c0fc7f0 2024-04-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc12a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc12c0fd150 0x7fc12c110850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc12affd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc12c0fc7f0 0x7fc12c110360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc12affd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc12c0fc7f0 0x7fc12c110360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.113:34010/0 (socket says 172.21.15.113:34010) 2024-04-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc12affd700 1 -- 172.21.15.113:0/2756234059 learned_addr learned my addr 172.21.15.113:0/2756234059 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc12a7fc700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc12c0fd150 0x7fc12c110850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55662/0 (socket says 172.21.15.113:55662) 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc12a7fc700 1 -- 172.21.15.113:0/2756234059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc12c0fc7f0 msgr2=0x7fc12c110360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc12a7fc700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc12c0fc7f0 0x7fc12c110360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc12a7fc700 1 -- 172.21.15.113:0/2756234059 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c005040 con 0x7fc12c0fd150 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc12affd700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc12c0fc7f0 0x7fc12c110360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.510+0000 7fc12a7fc700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc12c0fd150 0x7fc12c110850 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc11c000c00 tx=0x7fc11c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.511+0000 7fc123fff700 1 -- 172.21.15.113:0/2756234059 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc11c018070 con 0x7fc12c0fd150 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.511+0000 7fc123fff700 1 -- 172.21.15.113:0/2756234059 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc11c007e90 con 0x7fc12c0fd150 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.511+0000 7fc131300700 1 -- 172.21.15.113:0/2756234059 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c10a5c0 con 0x7fc12c0fd150 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.511+0000 7fc123fff700 1 -- 172.21.15.113:0/2756234059 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc11c0123c0 con 0x7fc12c0fd150 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.511+0000 7fc131300700 1 -- 172.21.15.113:0/2756234059 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c10ab20 con 0x7fc12c0fd150 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.512+0000 7fc131300700 1 -- 172.21.15.113:0/2756234059 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c060390 con 0x7fc12c0fd150 2024-04-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.512+0000 7fc123fff700 1 -- 172.21.15.113:0/2756234059 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc11c01d420 con 0x7fc12c0fd150 2024-04-12T05:17:57.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.512+0000 7fc123fff700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc11806caa0 0x7fc11806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:57.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.513+0000 7fc123fff700 1 -- 172.21.15.113:0/2756234059 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6250+0+0 (secure 0 0 0) 0x7fc11c08ca20 con 0x7fc12c0fd150 2024-04-12T05:17:57.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.513+0000 7fc12affd700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc11806caa0 0x7fc11806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:57.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.513+0000 7fc12affd700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc11806caa0 0x7fc11806ef60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc114007d30 tx=0x7fc11400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:57.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.517+0000 7fc123fff700 1 -- 172.21.15.113:0/2756234059 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc11c05a560 con 0x7fc12c0fd150 2024-04-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.715+0000 7fc131300700 1 -- 172.21.15.113:0/2756234059 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc12c10b160 con 0x7fc11806caa0 2024-04-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.724+0000 7fc123fff700 1 -- 172.21.15.113:0/2756234059 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc12c10b160 con 0x7fc11806caa0 2024-04-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (5m) 92s ago 17m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (3m) 92s ago 16m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (3m) 23s ago 16m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (5m) 92s ago 16m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (11m) 92s ago 11m 120M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (11m) 92s ago 11m 18.2M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (11m) 23s ago 11m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (11m) 23s ago 11m 16.0M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (7m) 92s ago 18m 501M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (6m) 23s ago 16m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (4m) 92s ago 18m 46.6M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (3m) 23s ago 16m 36.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (6m) 92s ago 16m 16.6M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (6m) 23s ago 16m 17.5M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (2m) 92s ago 15m 93.5M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:17:57.728 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (2m) 92s ago 15m 93.9M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:17:57.728 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (97s) 92s ago 14m 13.7M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:17:57.728 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (62s) 23s ago 14m 152M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:17:57.728 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (27s) 23s ago 13m 13.8M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:17:57.728 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (13m) 23s ago 13m 258M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:17:57.728 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (6m) 23s ago 15m 60.2M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.726+0000 7fc131300700 1 -- 172.21.15.113:0/2756234059 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc11806caa0 msgr2=0x7fc11806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.726+0000 7fc131300700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc11806caa0 0x7fc11806ef60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc114007d30 tx=0x7fc11400b040).stop 2024-04-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.726+0000 7fc131300700 1 -- 172.21.15.113:0/2756234059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc12c0fd150 msgr2=0x7fc12c110850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.726+0000 7fc131300700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc12c0fd150 0x7fc12c110850 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc11c000c00 tx=0x7fc11c009cb0).stop 2024-04-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.726+0000 7fc131300700 1 -- 172.21.15.113:0/2756234059 shutdown_connections 2024-04-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.726+0000 7fc131300700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc11806caa0 0x7fc11806ef60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.726+0000 7fc131300700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc12c0fc7f0 0x7fc12c110360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.726+0000 7fc131300700 1 --2- 172.21.15.113:0/2756234059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc12c0fd150 0x7fc12c110850 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.726+0000 7fc131300700 1 -- 172.21.15.113:0/2756234059 >> 172.21.15.113:0/2756234059 conn(0x7fc12c0f23b0 msgr2=0x7fc12c0613a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.726+0000 7fc131300700 1 -- 172.21.15.113:0/2756234059 shutdown_connections 2024-04-12T05:17:57.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.726+0000 7fc131300700 1 -- 172.21.15.113:0/2756234059 wait complete. 2024-04-12T05:17:57.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.824+0000 7ff45bdd0700 1 -- 172.21.15.113:0/4256573143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4540fe250 msgr2=0x7ff4540f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.824+0000 7ff45bdd0700 1 --2- 172.21.15.113:0/4256573143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4540fe250 0x7ff4540f7a30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ff4440047f0 tx=0x7ff444009cb0).stop 2024-04-12T05:17:57.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.824+0000 7ff45bdd0700 1 -- 172.21.15.113:0/4256573143 shutdown_connections 2024-04-12T05:17:57.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.824+0000 7ff45bdd0700 1 --2- 172.21.15.113:0/4256573143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4540fe250 0x7ff4540f7a30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.824+0000 7ff45bdd0700 1 --2- 172.21.15.113:0/4256573143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4540fd8f0 0x7ff4540fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.824+0000 7ff45bdd0700 1 -- 172.21.15.113:0/4256573143 >> 172.21.15.113:0/4256573143 conn(0x7ff4540f34b0 msgr2=0x7ff4540f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:57.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.824+0000 7ff45bdd0700 1 -- 172.21.15.113:0/4256573143 shutdown_connections 2024-04-12T05:17:57.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.824+0000 7ff45bdd0700 1 -- 172.21.15.113:0/4256573143 wait complete. 2024-04-12T05:17:57.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.825+0000 7ff45bdd0700 1 Processor -- start 2024-04-12T05:17:57.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.825+0000 7ff45bdd0700 1 -- start start 2024-04-12T05:17:57.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.825+0000 7ff45bdd0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4540fd8f0 0x7ff45406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:57.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.825+0000 7ff45bdd0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4540fe250 0x7ff45406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:57.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.825+0000 7ff45bdd0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45406bab0 con 0x7ff4540fe250 2024-04-12T05:17:57.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.825+0000 7ff45bdd0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45406bbf0 con 0x7ff4540fd8f0 2024-04-12T05:17:57.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.825+0000 7ff45936b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4540fe250 0x7ff45406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:57.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.825+0000 7ff45936b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4540fe250 0x7ff45406b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55688/0 (socket says 172.21.15.113:55688) 2024-04-12T05:17:57.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.825+0000 7ff45936b700 1 -- 172.21.15.113:0/157958362 learned_addr learned my addr 172.21.15.113:0/157958362 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:57.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.825+0000 7ff459b6c700 1 --2- 172.21.15.113:0/157958362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4540fd8f0 0x7ff45406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:57.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.826+0000 7ff45936b700 1 -- 172.21.15.113:0/157958362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4540fd8f0 msgr2=0x7ff45406b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:57.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.826+0000 7ff45936b700 1 --2- 172.21.15.113:0/157958362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4540fd8f0 0x7ff45406b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:57.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.826+0000 7ff45936b700 1 -- 172.21.15.113:0/157958362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff444005040 con 0x7ff4540fe250 2024-04-12T05:17:57.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.826+0000 7ff45936b700 1 --2- 172.21.15.113:0/157958362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4540fe250 0x7ff45406b5c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7ff444000c00 tx=0x7ff444009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:57.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.826+0000 7ff44affd700 1 -- 172.21.15.113:0/157958362 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff444018070 con 0x7ff4540fe250 2024-04-12T05:17:57.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.826+0000 7ff44affd700 1 -- 172.21.15.113:0/157958362 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff444007e90 con 0x7ff4540fe250 2024-04-12T05:17:57.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.826+0000 7ff45bdd0700 1 -- 172.21.15.113:0/157958362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45406be40 con 0x7ff4540fe250 2024-04-12T05:17:57.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.826+0000 7ff44affd700 1 -- 172.21.15.113:0/157958362 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff44401de20 con 0x7ff4540fe250 2024-04-12T05:17:57.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.826+0000 7ff45bdd0700 1 -- 172.21.15.113:0/157958362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff454114740 con 0x7ff4540fe250 2024-04-12T05:17:57.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.827+0000 7ff45bdd0700 1 -- 172.21.15.113:0/157958362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45405f590 con 0x7ff4540fe250 2024-04-12T05:17:57.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.828+0000 7ff44affd700 1 -- 172.21.15.113:0/157958362 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff44401d420 con 0x7ff4540fe250 2024-04-12T05:17:57.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.828+0000 7ff44affd700 1 --2- 172.21.15.113:0/157958362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff44006ca30 0x7ff44006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:57.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.828+0000 7ff44affd700 1 -- 172.21.15.113:0/157958362 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6250+0+0 (secure 0 0 0) 0x7ff44408c900 con 0x7ff4540fe250 2024-04-12T05:17:57.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.828+0000 7ff459b6c700 1 --2- 172.21.15.113:0/157958362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff44006ca30 0x7ff44006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:57.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.828+0000 7ff459b6c700 1 --2- 172.21.15.113:0/157958362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff44006ca30 0x7ff44006eef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7ff4500048b0 tx=0x7ff45000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:57.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:57.832+0000 7ff44affd700 1 -- 172.21.15.113:0/157958362 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff4440569b0 con 0x7ff4540fe250 2024-04-12T05:17:58.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.103+0000 7ff45bdd0700 1 -- 172.21.15.113:0/157958362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff454047540 con 0x7ff4540fe250 2024-04-12T05:17:58.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.103+0000 7ff44affd700 1 -- 172.21.15.113:0/157958362 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7ff444059fd0 con 0x7ff4540fe250 2024-04-12T05:17:58.105 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:17:58.105 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 5 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:17:58.106 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:17:58.107 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 5, 2024-04-12T05:17:58.107 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 9 2024-04-12T05:17:58.107 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:17:58.107 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:17:58.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.105+0000 7ff45bdd0700 1 -- 172.21.15.113:0/157958362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff44006ca30 msgr2=0x7ff44006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.105+0000 7ff45bdd0700 1 --2- 172.21.15.113:0/157958362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff44006ca30 0x7ff44006eef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7ff4500048b0 tx=0x7ff45000b040).stop 2024-04-12T05:17:58.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.105+0000 7ff45bdd0700 1 -- 172.21.15.113:0/157958362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4540fe250 msgr2=0x7ff45406b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.105+0000 7ff45bdd0700 1 --2- 172.21.15.113:0/157958362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4540fe250 0x7ff45406b5c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7ff444000c00 tx=0x7ff444009cb0).stop 2024-04-12T05:17:58.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.105+0000 7ff45bdd0700 1 -- 172.21.15.113:0/157958362 shutdown_connections 2024-04-12T05:17:58.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.105+0000 7ff45bdd0700 1 --2- 172.21.15.113:0/157958362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff44006ca30 0x7ff44006eef0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.105+0000 7ff45bdd0700 1 --2- 172.21.15.113:0/157958362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4540fd8f0 0x7ff45406b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.105+0000 7ff45bdd0700 1 --2- 172.21.15.113:0/157958362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4540fe250 0x7ff45406b5c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.105+0000 7ff45bdd0700 1 -- 172.21.15.113:0/157958362 >> 172.21.15.113:0/157958362 conn(0x7ff4540f34b0 msgr2=0x7ff4540fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:58.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.105+0000 7ff45bdd0700 1 -- 172.21.15.113:0/157958362 shutdown_connections 2024-04-12T05:17:58.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.105+0000 7ff45bdd0700 1 -- 172.21.15.113:0/157958362 wait complete. 2024-04-12T05:17:58.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.203+0000 7feb02a26700 1 -- 172.21.15.113:0/1775446318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feafc0fe250 msgr2=0x7feafc0f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.203+0000 7feb02a26700 1 --2- 172.21.15.113:0/1775446318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feafc0fe250 0x7feafc0f7a30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7feaf00047f0 tx=0x7feaf0009cb0).stop 2024-04-12T05:17:58.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.203+0000 7feb02a26700 1 -- 172.21.15.113:0/1775446318 shutdown_connections 2024-04-12T05:17:58.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.203+0000 7feb02a26700 1 --2- 172.21.15.113:0/1775446318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feafc0fe250 0x7feafc0f7a30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.203+0000 7feb02a26700 1 --2- 172.21.15.113:0/1775446318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feafc0fd8f0 0x7feafc0fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.203+0000 7feb02a26700 1 -- 172.21.15.113:0/1775446318 >> 172.21.15.113:0/1775446318 conn(0x7feafc0f34b0 msgr2=0x7feafc0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:58.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.203+0000 7feb02a26700 1 -- 172.21.15.113:0/1775446318 shutdown_connections 2024-04-12T05:17:58.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.203+0000 7feb02a26700 1 -- 172.21.15.113:0/1775446318 wait complete. 2024-04-12T05:17:58.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.204+0000 7feb02a26700 1 Processor -- start 2024-04-12T05:17:58.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.204+0000 7feb02a26700 1 -- start start 2024-04-12T05:17:58.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.204+0000 7feb02a26700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feafc0fd8f0 0x7feafc06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:58.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.204+0000 7feb02a26700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feafc0fe250 0x7feafc06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:58.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.204+0000 7feb02a26700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc06bab0 con 0x7feafc0fe250 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.204+0000 7feb02a26700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc06bbf0 con 0x7feafc0fd8f0 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feafb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feafc0fe250 0x7feafc06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feafb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feafc0fe250 0x7feafc06b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55698/0 (socket says 172.21.15.113:55698) 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feafb7fe700 1 -- 172.21.15.113:0/1232442830 learned_addr learned my addr 172.21.15.113:0/1232442830 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feafbfff700 1 --2- 172.21.15.113:0/1232442830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feafc0fd8f0 0x7feafc06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feafb7fe700 1 -- 172.21.15.113:0/1232442830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feafc0fd8f0 msgr2=0x7feafc06b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feafb7fe700 1 --2- 172.21.15.113:0/1232442830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feafc0fd8f0 0x7feafc06b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feafb7fe700 1 -- 172.21.15.113:0/1232442830 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf0005040 con 0x7feafc0fe250 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feafb7fe700 1 --2- 172.21.15.113:0/1232442830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feafc0fe250 0x7feafc06b5c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7feaf0000c00 tx=0x7feaf0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feaf97fa700 1 -- 172.21.15.113:0/1232442830 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaf0018070 con 0x7feafc0fe250 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feaf97fa700 1 -- 172.21.15.113:0/1232442830 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feaf0007e90 con 0x7feafc0fe250 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feb02a26700 1 -- 172.21.15.113:0/1232442830 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feafc06be40 con 0x7feafc0fe250 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feaf97fa700 1 -- 172.21.15.113:0/1232442830 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaf001de20 con 0x7feafc0fe250 2024-04-12T05:17:58.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.205+0000 7feb02a26700 1 -- 172.21.15.113:0/1232442830 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feafc114740 con 0x7feafc0fe250 2024-04-12T05:17:58.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.206+0000 7feb02a26700 1 -- 172.21.15.113:0/1232442830 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feafc05f590 con 0x7feafc0fe250 2024-04-12T05:17:58.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.207+0000 7feaf97fa700 1 -- 172.21.15.113:0/1232442830 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7feaf001d420 con 0x7feafc0fe250 2024-04-12T05:17:58.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.207+0000 7feaf97fa700 1 --2- 172.21.15.113:0/1232442830 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feae406ca30 0x7feae406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:58.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.207+0000 7feaf97fa700 1 -- 172.21.15.113:0/1232442830 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6250+0+0 (secure 0 0 0) 0x7feaf008c970 con 0x7feafc0fe250 2024-04-12T05:17:58.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.207+0000 7feafbfff700 1 --2- 172.21.15.113:0/1232442830 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feae406ca30 0x7feae406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:58.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.207+0000 7feafbfff700 1 --2- 172.21.15.113:0/1232442830 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feae406ca30 0x7feae406eef0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7feaec0048b0 tx=0x7feaec00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:58.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.211+0000 7feaf97fa700 1 -- 172.21.15.113:0/1232442830 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7feaf0056a20 con 0x7feafc0fe250 2024-04-12T05:17:58.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:57 smithi113 ceph-mon[103371]: osdmap e86: 6 total, 6 up, 6 in 2024-04-12T05:17:58.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.439+0000 7feb02a26700 1 -- 172.21.15.113:0/1232442830 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feafc047540 con 0x7feafc0fe250 2024-04-12T05:17:58.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.439+0000 7feaf97fa700 1 -- 172.21.15.113:0/1232442830 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1859 (secure 0 0 0) 0x7feaf005a040 con 0x7feafc0fe250 2024-04-12T05:17:58.442 INFO:teuthology.orchestra.run.smithi113.stdout:e18 2024-04-12T05:17:58.442 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:17:58.442 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:17:58.442 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:17:58.442 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 18 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:17:58.443 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:17:58.444 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:17:58.444 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:17:58.444 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:17:58.444 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:17:58.444 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=14426,1=24261} 2024-04-12T05:17:58.444 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:17:58.444 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:17:58.444 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:17:58.444 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:17:58.444 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:17:58.445 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:17:58.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.441+0000 7feb02a26700 1 -- 172.21.15.113:0/1232442830 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feae406ca30 msgr2=0x7feae406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.441+0000 7feb02a26700 1 --2- 172.21.15.113:0/1232442830 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feae406ca30 0x7feae406eef0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7feaec0048b0 tx=0x7feaec00b040).stop 2024-04-12T05:17:58.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.442+0000 7feb02a26700 1 -- 172.21.15.113:0/1232442830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feafc0fe250 msgr2=0x7feafc06b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.442+0000 7feb02a26700 1 --2- 172.21.15.113:0/1232442830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feafc0fe250 0x7feafc06b5c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7feaf0000c00 tx=0x7feaf0009cb0).stop 2024-04-12T05:17:58.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.442+0000 7feb02a26700 1 -- 172.21.15.113:0/1232442830 shutdown_connections 2024-04-12T05:17:58.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.442+0000 7feb02a26700 1 --2- 172.21.15.113:0/1232442830 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feae406ca30 0x7feae406eef0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.442+0000 7feb02a26700 1 --2- 172.21.15.113:0/1232442830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feafc0fd8f0 0x7feafc06b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.442+0000 7feb02a26700 1 --2- 172.21.15.113:0/1232442830 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feafc0fe250 0x7feafc06b5c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.442+0000 7feb02a26700 1 -- 172.21.15.113:0/1232442830 >> 172.21.15.113:0/1232442830 conn(0x7feafc0f34b0 msgr2=0x7feafc0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:58.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.442+0000 7feb02a26700 1 -- 172.21.15.113:0/1232442830 shutdown_connections 2024-04-12T05:17:58.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.442+0000 7feb02a26700 1 -- 172.21.15.113:0/1232442830 wait complete. 2024-04-12T05:17:58.447 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 18 2024-04-12T05:17:58.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:57 smithi192 ceph-mon[85304]: osdmap e86: 6 total, 6 up, 6 in 2024-04-12T05:17:58.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.541+0000 7f5397ba1700 1 -- 172.21.15.113:0/1963859103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53900fd150 msgr2=0x7f53900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.541+0000 7f5397ba1700 1 --2- 172.21.15.113:0/1963859103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53900fd150 0x7f53900f6930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f538c004730 tx=0x7f538c009cb0).stop 2024-04-12T05:17:58.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.541+0000 7f5397ba1700 1 -- 172.21.15.113:0/1963859103 shutdown_connections 2024-04-12T05:17:58.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.541+0000 7f5397ba1700 1 --2- 172.21.15.113:0/1963859103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53900fd150 0x7f53900f6930 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.541+0000 7f5397ba1700 1 --2- 172.21.15.113:0/1963859103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53900fc7f0 0x7f53900fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.541+0000 7f5397ba1700 1 -- 172.21.15.113:0/1963859103 >> 172.21.15.113:0/1963859103 conn(0x7f53900f23b0 msgr2=0x7f53900f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:58.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.541+0000 7f5397ba1700 1 -- 172.21.15.113:0/1963859103 shutdown_connections 2024-04-12T05:17:58.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.541+0000 7f5397ba1700 1 -- 172.21.15.113:0/1963859103 wait complete. 2024-04-12T05:17:58.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.542+0000 7f5397ba1700 1 Processor -- start 2024-04-12T05:17:58.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.542+0000 7f5397ba1700 1 -- start start 2024-04-12T05:17:58.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.542+0000 7f5397ba1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53900fc7f0 0x7f53900f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:58.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.542+0000 7f5397ba1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53900fd150 0x7f53900f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:58.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.542+0000 7f5397ba1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900f8f20 con 0x7f53900fc7f0 2024-04-12T05:17:58.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.542+0000 7f5397ba1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900f9060 con 0x7f53900fd150 2024-04-12T05:17:58.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.542+0000 7f539593d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53900fc7f0 0x7f53900f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:58.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.542+0000 7f539593d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53900fc7f0 0x7f53900f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55722/0 (socket says 172.21.15.113:55722) 2024-04-12T05:17:58.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.542+0000 7f539593d700 1 -- 172.21.15.113:0/2101799563 learned_addr learned my addr 172.21.15.113:0/2101799563 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:58.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.542+0000 7f539513c700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53900fd150 0x7f53900f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:58.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.543+0000 7f539593d700 1 -- 172.21.15.113:0/2101799563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53900fd150 msgr2=0x7f53900f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.543+0000 7f539593d700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53900fd150 0x7f53900f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.543+0000 7f539593d700 1 -- 172.21.15.113:0/2101799563 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5384005040 con 0x7f53900fc7f0 2024-04-12T05:17:58.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.543+0000 7f539513c700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53900fd150 0x7f53900f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:17:58.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.543+0000 7f539593d700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53900fc7f0 0x7f53900f8540 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5384009380 tx=0x7f5384009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:58.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.543+0000 7f5382ffd700 1 -- 172.21.15.113:0/2101799563 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f538400e560 con 0x7f53900fc7f0 2024-04-12T05:17:58.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.543+0000 7f5397ba1700 1 -- 172.21.15.113:0/2101799563 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538c005040 con 0x7f53900fc7f0 2024-04-12T05:17:58.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.543+0000 7f5382ffd700 1 -- 172.21.15.113:0/2101799563 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f538400f040 con 0x7f53900fc7f0 2024-04-12T05:17:58.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.543+0000 7f5382ffd700 1 -- 172.21.15.113:0/2101799563 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5384014b00 con 0x7f53900fc7f0 2024-04-12T05:17:58.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.543+0000 7f5397ba1700 1 -- 172.21.15.113:0/2101799563 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53901146a0 con 0x7f53900fc7f0 2024-04-12T05:17:58.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.544+0000 7f5397ba1700 1 -- 172.21.15.113:0/2101799563 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5390060370 con 0x7f53900fc7f0 2024-04-12T05:17:58.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.544+0000 7f5382ffd700 1 -- 172.21.15.113:0/2101799563 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f538400e6c0 con 0x7f53900fc7f0 2024-04-12T05:17:58.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.544+0000 7f5382ffd700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f537c06caa0 0x7f537c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:58.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.545+0000 7f5382ffd700 1 -- 172.21.15.113:0/2101799563 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6250+0+0 (secure 0 0 0) 0x7f538408b890 con 0x7f53900fc7f0 2024-04-12T05:17:58.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.545+0000 7f539513c700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f537c06caa0 0x7f537c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:58.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.545+0000 7f539513c700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f537c06caa0 0x7f537c06ef60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f538c004da0 tx=0x7f538c016040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:58.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.549+0000 7f5382ffd700 1 -- 172.21.15.113:0/2101799563 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5384055940 con 0x7f53900fc7f0 2024-04-12T05:17:58.755 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.753+0000 7f5397ba1700 1 -- 172.21.15.113:0/2101799563 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f53900f9a90 con 0x7f537c06caa0 2024-04-12T05:17:58.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.754+0000 7f5382ffd700 1 -- 172.21.15.113:0/2101799563 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f53900f9a90 con 0x7f537c06caa0 2024-04-12T05:17:58.756 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:17:58.756 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:17:58.756 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:17:58.756 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:17:58.756 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:17:58.756 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:17:58.757 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:17:58.757 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:17:58.757 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:17:58.757 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "11/21 daemons upgraded", 2024-04-12T05:17:58.757 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:17:58.757 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:17:58.757 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:17:58.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.756+0000 7f5397ba1700 1 -- 172.21.15.113:0/2101799563 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f537c06caa0 msgr2=0x7f537c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.756+0000 7f5397ba1700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f537c06caa0 0x7f537c06ef60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f538c004da0 tx=0x7f538c016040).stop 2024-04-12T05:17:58.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.756+0000 7f5397ba1700 1 -- 172.21.15.113:0/2101799563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53900fc7f0 msgr2=0x7f53900f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.756+0000 7f5397ba1700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53900fc7f0 0x7f53900f8540 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5384009380 tx=0x7f5384009cb0).stop 2024-04-12T05:17:58.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.756+0000 7f5397ba1700 1 -- 172.21.15.113:0/2101799563 shutdown_connections 2024-04-12T05:17:58.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.756+0000 7f5397ba1700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f537c06caa0 0x7f537c06ef60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.756+0000 7f5397ba1700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53900fc7f0 0x7f53900f8540 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.756+0000 7f5397ba1700 1 --2- 172.21.15.113:0/2101799563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53900fd150 0x7f53900f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.756+0000 7f5397ba1700 1 -- 172.21.15.113:0/2101799563 >> 172.21.15.113:0/2101799563 conn(0x7f53900f23b0 msgr2=0x7f5390061340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:58.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.756+0000 7f5397ba1700 1 -- 172.21.15.113:0/2101799563 shutdown_connections 2024-04-12T05:17:58.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.756+0000 7f5397ba1700 1 -- 172.21.15.113:0/2101799563 wait complete. 2024-04-12T05:17:58.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.854+0000 7fc4abcad700 1 -- 172.21.15.113:0/2745590339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4a40fd150 msgr2=0x7fc4a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.854+0000 7fc4abcad700 1 --2- 172.21.15.113:0/2745590339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4a40fd150 0x7fc4a40f6930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc4a00047f0 tx=0x7fc4a0009cb0).stop 2024-04-12T05:17:58.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.854+0000 7fc4abcad700 1 -- 172.21.15.113:0/2745590339 shutdown_connections 2024-04-12T05:17:58.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.854+0000 7fc4abcad700 1 --2- 172.21.15.113:0/2745590339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4a40fd150 0x7fc4a40f6930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.854+0000 7fc4abcad700 1 --2- 172.21.15.113:0/2745590339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4a40fc7f0 0x7fc4a40fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.854+0000 7fc4abcad700 1 -- 172.21.15.113:0/2745590339 >> 172.21.15.113:0/2745590339 conn(0x7fc4a4070590 msgr2=0x7fc4a40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:58.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.854+0000 7fc4abcad700 1 -- 172.21.15.113:0/2745590339 shutdown_connections 2024-04-12T05:17:58.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.854+0000 7fc4abcad700 1 -- 172.21.15.113:0/2745590339 wait complete. 2024-04-12T05:17:58.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.855+0000 7fc4abcad700 1 Processor -- start 2024-04-12T05:17:58.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.855+0000 7fc4abcad700 1 -- start start 2024-04-12T05:17:58.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.855+0000 7fc4abcad700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4a40fc7f0 0x7fc4a410e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:58.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.855+0000 7fc4abcad700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4a40fd150 0x7fc4a410e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:58.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.855+0000 7fc4abcad700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a410eb90 con 0x7fc4a40fc7f0 2024-04-12T05:17:58.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.855+0000 7fc4abcad700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a410ecd0 con 0x7fc4a40fd150 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.855+0000 7fc4a9a49700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4a40fc7f0 0x7fc4a410e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.855+0000 7fc4a9a49700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4a40fc7f0 0x7fc4a410e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55748/0 (socket says 172.21.15.113:55748) 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.855+0000 7fc4a9a49700 1 -- 172.21.15.113:0/1392431916 learned_addr learned my addr 172.21.15.113:0/1392431916 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.856+0000 7fc4a9248700 1 --2- 172.21.15.113:0/1392431916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4a40fd150 0x7fc4a410e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.856+0000 7fc4a9a49700 1 -- 172.21.15.113:0/1392431916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4a40fd150 msgr2=0x7fc4a410e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.856+0000 7fc4a9a49700 1 --2- 172.21.15.113:0/1392431916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4a40fd150 0x7fc4a410e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.856+0000 7fc4a9a49700 1 -- 172.21.15.113:0/1392431916 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a0005040 con 0x7fc4a40fc7f0 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.856+0000 7fc4a9a49700 1 --2- 172.21.15.113:0/1392431916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4a40fc7f0 0x7fc4a410e0a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc49800e8f0 tx=0x7fc49800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.856+0000 7fc496ffd700 1 -- 172.21.15.113:0/1392431916 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc498007b40 con 0x7fc4a40fc7f0 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.856+0000 7fc496ffd700 1 -- 172.21.15.113:0/1392431916 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc49800f040 con 0x7fc4a40fc7f0 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.856+0000 7fc4abcad700 1 -- 172.21.15.113:0/1392431916 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a41083a0 con 0x7fc4a40fc7f0 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.856+0000 7fc496ffd700 1 -- 172.21.15.113:0/1392431916 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc498014880 con 0x7fc4a40fc7f0 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.856+0000 7fc4abcad700 1 -- 172.21.15.113:0/1392431916 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a4108960 con 0x7fc4a40fc7f0 2024-04-12T05:17:58.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.857+0000 7fc4abcad700 1 -- 172.21.15.113:0/1392431916 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a405f590 con 0x7fc4a40fc7f0 2024-04-12T05:17:58.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.857+0000 7fc496ffd700 1 -- 172.21.15.113:0/1392431916 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc498014ab0 con 0x7fc4a40fc7f0 2024-04-12T05:17:58.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.858+0000 7fc496ffd700 1 --2- 172.21.15.113:0/1392431916 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc49006caa0 0x7fc49006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:17:58.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.858+0000 7fc4a9248700 1 --2- 172.21.15.113:0/1392431916 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc49006caa0 0x7fc49006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:17:58.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.858+0000 7fc496ffd700 1 -- 172.21.15.113:0/1392431916 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6250+0+0 (secure 0 0 0) 0x7fc49808b970 con 0x7fc4a40fc7f0 2024-04-12T05:17:58.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.858+0000 7fc4a9248700 1 --2- 172.21.15.113:0/1392431916 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc49006caa0 0x7fc49006ef60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc4a4109a20 tx=0x7fc4a0009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:17:58.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:58.862+0000 7fc496ffd700 1 -- 172.21.15.113:0/1392431916 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc498055a20 con 0x7fc4a40fc7f0 2024-04-12T05:17:59.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.130+0000 7fc4abcad700 1 -- 172.21.15.113:0/1392431916 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc4a4047540 con 0x7fc4a40fc7f0 2024-04-12T05:17:59.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.130+0000 7fc496ffd700 1 -- 172.21.15.113:0/1392431916 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fc498019020 con 0x7fc4a40fc7f0 2024-04-12T05:17:59.132 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_OK 2024-04-12T05:17:59.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.132+0000 7fc4abcad700 1 -- 172.21.15.113:0/1392431916 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc49006caa0 msgr2=0x7fc49006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:59.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.132+0000 7fc4abcad700 1 --2- 172.21.15.113:0/1392431916 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc49006caa0 0x7fc49006ef60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc4a4109a20 tx=0x7fc4a0009cb0).stop 2024-04-12T05:17:59.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.132+0000 7fc4abcad700 1 -- 172.21.15.113:0/1392431916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4a40fc7f0 msgr2=0x7fc4a410e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:17:59.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.132+0000 7fc4abcad700 1 --2- 172.21.15.113:0/1392431916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4a40fc7f0 0x7fc4a410e0a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc49800e8f0 tx=0x7fc49800b040).stop 2024-04-12T05:17:59.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.132+0000 7fc4abcad700 1 -- 172.21.15.113:0/1392431916 shutdown_connections 2024-04-12T05:17:59.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.132+0000 7fc4abcad700 1 --2- 172.21.15.113:0/1392431916 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc49006caa0 0x7fc49006ef60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:59.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.132+0000 7fc4abcad700 1 --2- 172.21.15.113:0/1392431916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4a40fc7f0 0x7fc4a410e0a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:59.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.132+0000 7fc4abcad700 1 --2- 172.21.15.113:0/1392431916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4a40fd150 0x7fc4a410e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:17:59.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.132+0000 7fc4abcad700 1 -- 172.21.15.113:0/1392431916 >> 172.21.15.113:0/1392431916 conn(0x7fc4a4070590 msgr2=0x7fc4a4060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:17:59.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.132+0000 7fc4abcad700 1 -- 172.21.15.113:0/1392431916 shutdown_connections 2024-04-12T05:17:59.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:17:59.132+0000 7fc4abcad700 1 -- 172.21.15.113:0/1392431916 wait complete. 2024-04-12T05:17:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:58 smithi113 ceph-mon[103371]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:58 smithi113 ceph-mon[103371]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:58 smithi113 ceph-mon[103371]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:58 smithi113 ceph-mon[103371]: pgmap v163: 65 pgs: 1 active+recovering+undersized+remapped, 1 peering, 63 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 29 objects/s recovering 2024-04-12T05:17:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:58 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/157958362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:58 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1232442830' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:17:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:58 smithi192 ceph-mon[85304]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:58 smithi192 ceph-mon[85304]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:58 smithi192 ceph-mon[85304]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:17:59.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:58 smithi192 ceph-mon[85304]: pgmap v163: 65 pgs: 1 active+recovering+undersized+remapped, 1 peering, 63 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 29 objects/s recovering 2024-04-12T05:17:59.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:58 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/157958362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:17:59.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:58 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1232442830' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:18:00.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:59 smithi113 ceph-mon[103371]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:18:00.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:17:59 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1392431916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:18:00.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:59 smithi192 ceph-mon[85304]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:18:00.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:17:59 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1392431916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:18:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:00 smithi113 ceph-mon[103371]: pgmap v164: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s, 20 objects/s recovering 2024-04-12T05:18:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:00 smithi192 ceph-mon[85304]: pgmap v164: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s, 20 objects/s recovering 2024-04-12T05:18:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:02 smithi113 ceph-mon[103371]: pgmap v165: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s, 17 objects/s recovering 2024-04-12T05:18:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:02 smithi192 ceph-mon[85304]: pgmap v165: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s, 17 objects/s recovering 2024-04-12T05:18:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:04 smithi113 ceph-mon[103371]: pgmap v166: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1024 KiB/s, 29 objects/s recovering 2024-04-12T05:18:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:04 smithi192 ceph-mon[85304]: pgmap v166: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1024 KiB/s, 29 objects/s recovering 2024-04-12T05:18:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:06 smithi113 ceph-mon[103371]: pgmap v167: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 829 KiB/s, 23 objects/s recovering 2024-04-12T05:18:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:18:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:06 smithi192 ceph-mon[85304]: pgmap v167: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 829 KiB/s, 23 objects/s recovering 2024-04-12T05:18:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:18:09.364 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:08 smithi113 ceph-mon[103371]: pgmap v168: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 753 KiB/s, 32 objects/s recovering 2024-04-12T05:18:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:08 smithi192 ceph-mon[85304]: pgmap v168: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 753 KiB/s, 32 objects/s recovering 2024-04-12T05:18:10.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:09 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:18:10.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:09 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:18:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:10 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:18:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:10 smithi113 ceph-mon[103371]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-12T05:18:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:10 smithi113 ceph-mon[103371]: pgmap v169: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 682 KiB/s, 19 objects/s recovering 2024-04-12T05:18:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:10 smithi113 ceph-mon[103371]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-04-12T05:18:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:10 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:18:11.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:10 smithi192 ceph-mon[85304]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-12T05:18:11.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:10 smithi192 ceph-mon[85304]: pgmap v169: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 682 KiB/s, 19 objects/s recovering 2024-04-12T05:18:11.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:10 smithi192 ceph-mon[85304]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-04-12T05:18:12.269 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:11 smithi113 ceph-mon[103371]: osdmap e87: 6 total, 6 up, 6 in 2024-04-12T05:18:12.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:11 smithi192 ceph-mon[85304]: osdmap e87: 6 total, 6 up, 6 in 2024-04-12T05:18:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:12 smithi113 ceph-mon[103371]: pgmap v171: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 23 objects/s recovering 2024-04-12T05:18:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:12 smithi113 ceph-mon[103371]: osdmap e88: 6 total, 6 up, 6 in 2024-04-12T05:18:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:12 smithi192 ceph-mon[85304]: pgmap v171: 65 pgs: 1 active+recovering+undersized+remapped, 64 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 23 objects/s recovering 2024-04-12T05:18:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:12 smithi192 ceph-mon[85304]: osdmap e88: 6 total, 6 up, 6 in 2024-04-12T05:18:14.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:14 smithi113 ceph-mon[103371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-04-12T05:18:14.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:14 smithi113 ceph-mon[103371]: Cluster is now healthy 2024-04-12T05:18:14.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:14 smithi192 ceph-mon[85304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-04-12T05:18:14.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:14 smithi192 ceph-mon[85304]: Cluster is now healthy 2024-04-12T05:18:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:15 smithi113 ceph-mon[103371]: pgmap v173: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 20 objects/s recovering 2024-04-12T05:18:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:15 smithi192 ceph-mon[85304]: pgmap v173: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 20 objects/s recovering 2024-04-12T05:18:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:17 smithi113 ceph-mon[103371]: pgmap v174: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 6 objects/s recovering 2024-04-12T05:18:17.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:17 smithi192 ceph-mon[85304]: pgmap v174: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 6 objects/s recovering 2024-04-12T05:18:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:19 smithi113 ceph-mon[103371]: pgmap v175: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 6 objects/s recovering 2024-04-12T05:18:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:19 smithi192 ceph-mon[85304]: pgmap v175: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 6 objects/s recovering 2024-04-12T05:18:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:21 smithi113 ceph-mon[103371]: pgmap v176: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 5 objects/s recovering 2024-04-12T05:18:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:21 smithi192 ceph-mon[85304]: pgmap v176: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 5 objects/s recovering 2024-04-12T05:18:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:18:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:18:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:23 smithi113 ceph-mon[103371]: pgmap v177: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 5 objects/s recovering 2024-04-12T05:18:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:23 smithi192 ceph-mon[85304]: pgmap v177: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 5 objects/s recovering 2024-04-12T05:18:25.196 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:25 smithi192 ceph-mon[85304]: pgmap v178: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 4 objects/s recovering 2024-04-12T05:18:25.196 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:25 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:18:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:25 smithi113 ceph-mon[103371]: pgmap v178: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s, 4 objects/s recovering 2024-04-12T05:18:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:25 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:18:26.312 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:26 smithi192 ceph-mon[85304]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:18:26.312 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:26 smithi192 ceph-mon[85304]: Upgrade: osd.5 is safe to restart 2024-04-12T05:18:26.313 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:26 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:26.313 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:26 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:18:26.313 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:26 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:18:26.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:26 smithi113 ceph-mon[103371]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T05:18:26.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:26 smithi113 ceph-mon[103371]: Upgrade: osd.5 is safe to restart 2024-04-12T05:18:26.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:26 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:26.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:26 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T05:18:26.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:26 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:18:27.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:27 smithi192 ceph-mon[85304]: pgmap v179: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:27.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:27 smithi192 ceph-mon[85304]: Upgrade: Updating osd.5 2024-04-12T05:18:27.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:27 smithi192 ceph-mon[85304]: Deploying daemon osd.5 on smithi192 2024-04-12T05:18:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:27 smithi113 ceph-mon[103371]: pgmap v179: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:27 smithi113 ceph-mon[103371]: Upgrade: Updating osd.5 2024-04-12T05:18:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:27 smithi113 ceph-mon[103371]: Deploying daemon osd.5 on smithi192 2024-04-12T05:18:29.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:29 smithi192 ceph-mon[85304]: pgmap v180: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:29.199 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:28 smithi192 systemd[1]: Stopping Ceph osd.5 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:18:29.199 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[59730]: debug 2024-04-12T05:18:29.148+0000 7f4409ffb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-12T05:18:29.199 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[59730]: debug 2024-04-12T05:18:29.148+0000 7f4409ffb700 -1 osd.5 88 *** Got signal Terminated *** 2024-04-12T05:18:29.199 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:29 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[59730]: debug 2024-04-12T05:18:29.148+0000 7f4409ffb700 -1 osd.5 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:18:29.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.244+0000 7f9840a9e700 1 -- 172.21.15.113:0/4075523043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f983c0fc7f0 msgr2=0x7f983c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:29.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.244+0000 7f9840a9e700 1 --2- 172.21.15.113:0/4075523043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f983c0fc7f0 0x7f983c0fcbd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f982c0047f0 tx=0x7f982c009cb0).stop 2024-04-12T05:18:29.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.244+0000 7f9840a9e700 1 -- 172.21.15.113:0/4075523043 shutdown_connections 2024-04-12T05:18:29.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.244+0000 7f9840a9e700 1 --2- 172.21.15.113:0/4075523043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f983c0fd150 0x7f983c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.244+0000 7f9840a9e700 1 --2- 172.21.15.113:0/4075523043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f983c0fc7f0 0x7f983c0fcbd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.244+0000 7f9840a9e700 1 -- 172.21.15.113:0/4075523043 >> 172.21.15.113:0/4075523043 conn(0x7f983c070590 msgr2=0x7f983c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:29.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.244+0000 7f9840a9e700 1 -- 172.21.15.113:0/4075523043 shutdown_connections 2024-04-12T05:18:29.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.244+0000 7f9840a9e700 1 -- 172.21.15.113:0/4075523043 wait complete. 2024-04-12T05:18:29.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f9840a9e700 1 Processor -- start 2024-04-12T05:18:29.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f9840a9e700 1 -- start start 2024-04-12T05:18:29.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f9840a9e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f983c0fc7f0 0x7f983c109230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:29.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f9840a9e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f983c0fd150 0x7f983c109720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f9840a9e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c109c60 con 0x7f983c0fc7f0 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f9840a9e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c109da0 con 0x7f983c0fd150 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f983a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f983c0fc7f0 0x7f983c109230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f983a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f983c0fc7f0 0x7f983c109230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55178/0 (socket says 172.21.15.113:55178) 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f983a59c700 1 -- 172.21.15.113:0/3667126222 learned_addr learned my addr 172.21.15.113:0/3667126222 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f9839d9b700 1 --2- 172.21.15.113:0/3667126222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f983c0fd150 0x7f983c109720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f983a59c700 1 -- 172.21.15.113:0/3667126222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f983c0fd150 msgr2=0x7f983c109720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f983a59c700 1 --2- 172.21.15.113:0/3667126222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f983c0fd150 0x7f983c109720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.245+0000 7f983a59c700 1 -- 172.21.15.113:0/3667126222 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c005040 con 0x7f983c0fc7f0 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.246+0000 7f983a59c700 1 --2- 172.21.15.113:0/3667126222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f983c0fc7f0 0x7f983c109230 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f982c009380 tx=0x7f982c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.246+0000 7f982b7fe700 1 -- 172.21.15.113:0/3667126222 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f982c018070 con 0x7f983c0fc7f0 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.246+0000 7f9840a9e700 1 -- 172.21.15.113:0/3667126222 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983c10a020 con 0x7f983c0fc7f0 2024-04-12T05:18:29.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.246+0000 7f982b7fe700 1 -- 172.21.15.113:0/3667126222 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f982c007e90 con 0x7f983c0fc7f0 2024-04-12T05:18:29.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.246+0000 7f982b7fe700 1 -- 172.21.15.113:0/3667126222 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f982c0123c0 con 0x7f983c0fc7f0 2024-04-12T05:18:29.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.246+0000 7f9840a9e700 1 -- 172.21.15.113:0/3667126222 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983c112570 con 0x7f983c0fc7f0 2024-04-12T05:18:29.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.247+0000 7f9840a9e700 1 -- 172.21.15.113:0/3667126222 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983c05f590 con 0x7f983c0fc7f0 2024-04-12T05:18:29.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.247+0000 7f982b7fe700 1 -- 172.21.15.113:0/3667126222 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f982c01d420 con 0x7f983c0fc7f0 2024-04-12T05:18:29.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.248+0000 7f982b7fe700 1 --2- 172.21.15.113:0/3667126222 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f982406caa0 0x7f982406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:29.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.248+0000 7f982b7fe700 1 -- 172.21.15.113:0/3667126222 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f982c08c9f0 con 0x7f983c0fc7f0 2024-04-12T05:18:29.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.248+0000 7f9839d9b700 1 --2- 172.21.15.113:0/3667126222 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f982406caa0 0x7f982406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:29.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.248+0000 7f9839d9b700 1 --2- 172.21.15.113:0/3667126222 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f982406caa0 0x7f982406ef60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f983c060f00 tx=0x7f983000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:29.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.252+0000 7f982b7fe700 1 -- 172.21.15.113:0/3667126222 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f982c056a00 con 0x7f983c0fc7f0 2024-04-12T05:18:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:29 smithi113 ceph-mon[103371]: pgmap v180: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:29.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.463+0000 7f9840a9e700 1 -- 172.21.15.113:0/3667126222 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f983c059120 con 0x7f982406caa0 2024-04-12T05:18:29.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.464+0000 7f982b7fe700 1 -- 172.21.15.113:0/3667126222 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f983c059120 con 0x7f982406caa0 2024-04-12T05:18:29.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.466+0000 7f9840a9e700 1 -- 172.21.15.113:0/3667126222 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f982406caa0 msgr2=0x7f982406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:29.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.466+0000 7f9840a9e700 1 --2- 172.21.15.113:0/3667126222 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f982406caa0 0x7f982406ef60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f983c060f00 tx=0x7f983000b040).stop 2024-04-12T05:18:29.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.466+0000 7f9840a9e700 1 -- 172.21.15.113:0/3667126222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f983c0fc7f0 msgr2=0x7f983c109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:29.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.466+0000 7f9840a9e700 1 --2- 172.21.15.113:0/3667126222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f983c0fc7f0 0x7f983c109230 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f982c009380 tx=0x7f982c009cb0).stop 2024-04-12T05:18:29.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.466+0000 7f9840a9e700 1 -- 172.21.15.113:0/3667126222 shutdown_connections 2024-04-12T05:18:29.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.466+0000 7f9840a9e700 1 --2- 172.21.15.113:0/3667126222 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f982406caa0 0x7f982406ef60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.466+0000 7f9840a9e700 1 --2- 172.21.15.113:0/3667126222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f983c0fc7f0 0x7f983c109230 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.466+0000 7f9840a9e700 1 --2- 172.21.15.113:0/3667126222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f983c0fd150 0x7f983c109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.466+0000 7f9840a9e700 1 -- 172.21.15.113:0/3667126222 >> 172.21.15.113:0/3667126222 conn(0x7f983c070590 msgr2=0x7f983c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:29.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.466+0000 7f9840a9e700 1 -- 172.21.15.113:0/3667126222 shutdown_connections 2024-04-12T05:18:29.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.466+0000 7f9840a9e700 1 -- 172.21.15.113:0/3667126222 wait complete. 2024-04-12T05:18:29.479 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:18:29.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.566+0000 7fa447611700 1 -- 172.21.15.113:0/2924991021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4400fd090 msgr2=0x7fa4400f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:29.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.566+0000 7fa447611700 1 --2- 172.21.15.113:0/2924991021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4400fd090 0x7fa4400f6870 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fa43c0047f0 tx=0x7fa43c009cb0).stop 2024-04-12T05:18:29.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.566+0000 7fa447611700 1 -- 172.21.15.113:0/2924991021 shutdown_connections 2024-04-12T05:18:29.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.566+0000 7fa447611700 1 --2- 172.21.15.113:0/2924991021 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4400fd090 0x7fa4400f6870 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.566+0000 7fa447611700 1 --2- 172.21.15.113:0/2924991021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4400fc730 0x7fa4400fcb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.566+0000 7fa447611700 1 -- 172.21.15.113:0/2924991021 >> 172.21.15.113:0/2924991021 conn(0x7fa4400f2310 msgr2=0x7fa4400f4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:29.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.566+0000 7fa447611700 1 -- 172.21.15.113:0/2924991021 shutdown_connections 2024-04-12T05:18:29.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.566+0000 7fa447611700 1 -- 172.21.15.113:0/2924991021 wait complete. 2024-04-12T05:18:29.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.566+0000 7fa447611700 1 Processor -- start 2024-04-12T05:18:29.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.566+0000 7fa447611700 1 -- start start 2024-04-12T05:18:29.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa447611700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4400fc730 0x7fa4400f8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa447611700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4400fd090 0x7fa4400f8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa447611700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4400f8f20 con 0x7fa4400fc730 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa447611700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4400f9060 con 0x7fa4400fd090 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa4453ad700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4400fc730 0x7fa4400f8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa4453ad700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4400fc730 0x7fa4400f8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55196/0 (socket says 172.21.15.113:55196) 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa4453ad700 1 -- 172.21.15.113:0/1110177807 learned_addr learned my addr 172.21.15.113:0/1110177807 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa4453ad700 1 -- 172.21.15.113:0/1110177807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4400fd090 msgr2=0x7fa4400f8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa4453ad700 1 --2- 172.21.15.113:0/1110177807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4400fd090 0x7fa4400f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa4453ad700 1 -- 172.21.15.113:0/1110177807 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43c005040 con 0x7fa4400fc730 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa4453ad700 1 --2- 172.21.15.113:0/1110177807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4400fc730 0x7fa4400f8480 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa4300041d0 tx=0x7fa430009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa4367fc700 1 -- 172.21.15.113:0/1110177807 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa430002d90 con 0x7fa4400fc730 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa4367fc700 1 -- 172.21.15.113:0/1110177807 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa43000f040 con 0x7fa4400fc730 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa447611700 1 -- 172.21.15.113:0/1110177807 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4401142b0 con 0x7fa4400fc730 2024-04-12T05:18:29.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa4367fc700 1 -- 172.21.15.113:0/1110177807 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa43000d800 con 0x7fa4400fc730 2024-04-12T05:18:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.567+0000 7fa447611700 1 -- 172.21.15.113:0/1110177807 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa440114730 con 0x7fa4400fc730 2024-04-12T05:18:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.568+0000 7fa447611700 1 -- 172.21.15.113:0/1110177807 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa440060120 con 0x7fa4400fc730 2024-04-12T05:18:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.569+0000 7fa4367fc700 1 -- 172.21.15.113:0/1110177807 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa43000da90 con 0x7fa4400fc730 2024-04-12T05:18:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.569+0000 7fa4367fc700 1 --2- 172.21.15.113:0/1110177807 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa42c06caf0 0x7fa42c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.569+0000 7fa4367fc700 1 -- 172.21.15.113:0/1110177807 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6221+0+0 (secure 0 0 0) 0x7fa43008bd10 con 0x7fa4400fc730 2024-04-12T05:18:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.569+0000 7fa444bac700 1 --2- 172.21.15.113:0/1110177807 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa42c06caf0 0x7fa42c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.569+0000 7fa444bac700 1 --2- 172.21.15.113:0/1110177807 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa42c06caf0 0x7fa42c06efb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa4400f9e30 tx=0x7fa43c009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:29.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.574+0000 7fa4367fc700 1 -- 172.21.15.113:0/1110177807 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa430055de0 con 0x7fa4400fc730 2024-04-12T05:18:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.781+0000 7fa447611700 1 -- 172.21.15.113:0/1110177807 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4400f9c80 con 0x7fa42c06caf0 2024-04-12T05:18:29.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.782+0000 7fa4367fc700 1 -- 172.21.15.113:0/1110177807 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fa4400f9c80 con 0x7fa42c06caf0 2024-04-12T05:18:29.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.784+0000 7fa447611700 1 -- 172.21.15.113:0/1110177807 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa42c06caf0 msgr2=0x7fa42c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:29.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.784+0000 7fa447611700 1 --2- 172.21.15.113:0/1110177807 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa42c06caf0 0x7fa42c06efb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa4400f9e30 tx=0x7fa43c009cb0).stop 2024-04-12T05:18:29.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.784+0000 7fa447611700 1 -- 172.21.15.113:0/1110177807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4400fc730 msgr2=0x7fa4400f8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:29.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.784+0000 7fa447611700 1 --2- 172.21.15.113:0/1110177807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4400fc730 0x7fa4400f8480 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa4300041d0 tx=0x7fa430009f60).stop 2024-04-12T05:18:29.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.784+0000 7fa447611700 1 -- 172.21.15.113:0/1110177807 shutdown_connections 2024-04-12T05:18:29.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.784+0000 7fa447611700 1 --2- 172.21.15.113:0/1110177807 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa42c06caf0 0x7fa42c06efb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.784+0000 7fa447611700 1 --2- 172.21.15.113:0/1110177807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4400fc730 0x7fa4400f8480 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.784+0000 7fa447611700 1 --2- 172.21.15.113:0/1110177807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4400fd090 0x7fa4400f8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:18:29.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.784+0000 7fa447611700 1 -- 172.21.15.113:0/1110177807 >> 172.21.15.113:0/1110177807 conn(0x7fa4400f2310 msgr2=0x7fa440061150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:29.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.784+0000 7fa447611700 1 -- 172.21.15.113:0/1110177807 shutdown_connections 2024-04-12T05:18:29.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.784+0000 7fa447611700 1 -- 172.21.15.113:0/1110177807 wait complete. 2024-04-12T05:18:29.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.882+0000 7f716669d700 1 -- 172.21.15.113:0/2133824485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71600fc7f0 msgr2=0x7f71600fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:29.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.882+0000 7f716669d700 1 --2- 172.21.15.113:0/2133824485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71600fc7f0 0x7f71600fcbd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f71500047f0 tx=0x7f7150009cb0).stop 2024-04-12T05:18:29.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.882+0000 7f716669d700 1 -- 172.21.15.113:0/2133824485 shutdown_connections 2024-04-12T05:18:29.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.882+0000 7f716669d700 1 --2- 172.21.15.113:0/2133824485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71600fd150 0x7f71600f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.882+0000 7f716669d700 1 --2- 172.21.15.113:0/2133824485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71600fc7f0 0x7f71600fcbd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.882+0000 7f716669d700 1 -- 172.21.15.113:0/2133824485 >> 172.21.15.113:0/2133824485 conn(0x7f7160070590 msgr2=0x7f71600f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:29.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.882+0000 7f716669d700 1 -- 172.21.15.113:0/2133824485 shutdown_connections 2024-04-12T05:18:29.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.882+0000 7f716669d700 1 -- 172.21.15.113:0/2133824485 wait complete. 2024-04-12T05:18:29.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.883+0000 7f716669d700 1 Processor -- start 2024-04-12T05:18:29.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.883+0000 7f716669d700 1 -- start start 2024-04-12T05:18:29.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.883+0000 7f716669d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71600fc7f0 0x7f71601090d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.883+0000 7f716669d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71600fd150 0x7f71601095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.883+0000 7f716669d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7160109ab0 con 0x7f71600fd150 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.883+0000 7f716669d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7160109bf0 con 0x7f71600fc7f0 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71600fd150 0x7f71601095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715ffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71600fc7f0 0x7f71601090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71600fd150 0x7f71601095c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55220/0 (socket says 172.21.15.113:55220) 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715f7fe700 1 -- 172.21.15.113:0/1970533863 learned_addr learned my addr 172.21.15.113:0/1970533863 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715ffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71600fc7f0 0x7f71601090d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.113:46640/0 (socket says 172.21.15.113:46640) 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715f7fe700 1 -- 172.21.15.113:0/1970533863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71600fc7f0 msgr2=0x7f71601090d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715f7fe700 1 --2- 172.21.15.113:0/1970533863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71600fc7f0 0x7f71601090d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715f7fe700 1 -- 172.21.15.113:0/1970533863 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7150005040 con 0x7f71600fd150 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715f7fe700 1 --2- 172.21.15.113:0/1970533863 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71600fd150 0x7f71601095c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f715400c710 tx=0x7f7154007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715d7fa700 1 -- 172.21.15.113:0/1970533863 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f715400e680 con 0x7f71600fd150 2024-04-12T05:18:29.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715d7fa700 1 -- 172.21.15.113:0/1970533863 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f715400ec90 con 0x7f71600fd150 2024-04-12T05:18:29.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f716669d700 1 -- 172.21.15.113:0/1970533863 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7160109ea0 con 0x7f71600fd150 2024-04-12T05:18:29.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f715d7fa700 1 -- 172.21.15.113:0/1970533863 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7154013d40 con 0x7f71600fd150 2024-04-12T05:18:29.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.884+0000 7f716669d700 1 -- 172.21.15.113:0/1970533863 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71601124c0 con 0x7f71600fd150 2024-04-12T05:18:29.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.885+0000 7f716669d700 1 -- 172.21.15.113:0/1970533863 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716005f590 con 0x7f71600fd150 2024-04-12T05:18:29.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.886+0000 7f715d7fa700 1 -- 172.21.15.113:0/1970533863 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f715400e7e0 con 0x7f71600fd150 2024-04-12T05:18:29.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.886+0000 7f715d7fa700 1 --2- 172.21.15.113:0/1970533863 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714806caa0 0x7f714806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:29.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.886+0000 7f715d7fa700 1 -- 172.21.15.113:0/1970533863 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f715408bbb0 con 0x7f71600fd150 2024-04-12T05:18:29.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.886+0000 7f715ffff700 1 --2- 172.21.15.113:0/1970533863 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714806caa0 0x7f714806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:29.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.886+0000 7f715ffff700 1 --2- 172.21.15.113:0/1970533863 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714806caa0 0x7f714806ef60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f7150007cf0 tx=0x7f715000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:29.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:29.890+0000 7f715d7fa700 1 -- 172.21.15.113:0/1970533863 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7154092080 con 0x7f71600fd150 2024-04-12T05:18:30.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.093+0000 7f716669d700 1 -- 172.21.15.113:0/1970533863 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7160060d30 con 0x7f714806caa0 2024-04-12T05:18:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.102+0000 7f715d7fa700 1 -- 172.21.15.113:0/1970533863 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7160060d30 con 0x7f714806caa0 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (6m) 2m ago 17m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (3m) 2m ago 17m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (3m) 55s ago 16m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (6m) 2m ago 17m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (12m) 2m ago 12m 120M - 16.2.15-1-g8963e9de 727f386679cb 31d426702d78 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (12m) 2m ago 12m 18.2M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (12m) 55s ago 12m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (12m) 55s ago 12m 16.0M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (7m) 2m ago 18m 501M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (7m) 55s ago 16m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (4m) 2m ago 19m 46.6M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:18:30.106 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (4m) 55s ago 16m 36.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:18:30.107 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (6m) 2m ago 17m 16.6M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:18:30.107 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (6m) 55s ago 16m 17.5M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:18:30.107 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (3m) 2m ago 16m 93.5M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:18:30.107 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (2m) 2m ago 15m 93.9M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:18:30.107 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (2m) 2m ago 15m 13.7M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:18:30.107 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (94s) 55s ago 14m 152M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:18:30.107 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (59s) 55s ago 14m 13.8M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:18:30.107 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (13m) 55s ago 13m 258M 2251M 16.2.15-1-g8963e9de 727f386679cb a0c3882d8b75 2024-04-12T05:18:30.107 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (6m) 55s ago 16m 60.2M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:18:30.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.104+0000 7f716669d700 1 -- 172.21.15.113:0/1970533863 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714806caa0 msgr2=0x7f714806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.104+0000 7f716669d700 1 --2- 172.21.15.113:0/1970533863 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714806caa0 0x7f714806ef60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f7150007cf0 tx=0x7f715000b040).stop 2024-04-12T05:18:30.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.104+0000 7f716669d700 1 -- 172.21.15.113:0/1970533863 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71600fd150 msgr2=0x7f71601095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.104+0000 7f716669d700 1 --2- 172.21.15.113:0/1970533863 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71600fd150 0x7f71601095c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f715400c710 tx=0x7f7154007ec0).stop 2024-04-12T05:18:30.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.105+0000 7f716669d700 1 -- 172.21.15.113:0/1970533863 shutdown_connections 2024-04-12T05:18:30.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.105+0000 7f716669d700 1 --2- 172.21.15.113:0/1970533863 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714806caa0 0x7f714806ef60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.105+0000 7f716669d700 1 --2- 172.21.15.113:0/1970533863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71600fc7f0 0x7f71601090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.105+0000 7f716669d700 1 --2- 172.21.15.113:0/1970533863 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71600fd150 0x7f71601095c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.105+0000 7f716669d700 1 -- 172.21.15.113:0/1970533863 >> 172.21.15.113:0/1970533863 conn(0x7f7160070590 msgr2=0x7f71600fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:30.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.105+0000 7f716669d700 1 -- 172.21.15.113:0/1970533863 shutdown_connections 2024-04-12T05:18:30.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.105+0000 7f716669d700 1 -- 172.21.15.113:0/1970533863 wait complete. 2024-04-12T05:18:30.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.201+0000 7faec97e4700 1 -- 172.21.15.113:0/314779809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec40fd8f0 msgr2=0x7faec40fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.201+0000 7faec97e4700 1 --2- 172.21.15.113:0/314779809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec40fd8f0 0x7faec40fdcd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7faeb80047f0 tx=0x7faeb8009cb0).stop 2024-04-12T05:18:30.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.202+0000 7faec97e4700 1 -- 172.21.15.113:0/314779809 shutdown_connections 2024-04-12T05:18:30.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.202+0000 7faec97e4700 1 --2- 172.21.15.113:0/314779809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faec40fe250 0x7faec40f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.202+0000 7faec97e4700 1 --2- 172.21.15.113:0/314779809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec40fd8f0 0x7faec40fdcd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.202+0000 7faec97e4700 1 -- 172.21.15.113:0/314779809 >> 172.21.15.113:0/314779809 conn(0x7faec40f34b0 msgr2=0x7faec40f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:30.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.202+0000 7faec97e4700 1 -- 172.21.15.113:0/314779809 shutdown_connections 2024-04-12T05:18:30.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.202+0000 7faec97e4700 1 -- 172.21.15.113:0/314779809 wait complete. 2024-04-12T05:18:30.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.202+0000 7faec97e4700 1 Processor -- start 2024-04-12T05:18:30.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.202+0000 7faec97e4700 1 -- start start 2024-04-12T05:18:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec97e4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec40fd8f0 0x7faec410dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec97e4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faec40fe250 0x7faec410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec97e4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec410ead0 con 0x7faec40fd8f0 2024-04-12T05:18:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec97e4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec410ec10 con 0x7faec40fe250 2024-04-12T05:18:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec40fd8f0 0x7faec410dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec40fd8f0 0x7faec410dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55230/0 (socket says 172.21.15.113:55230) 2024-04-12T05:18:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec2ffd700 1 -- 172.21.15.113:0/905081700 learned_addr learned my addr 172.21.15.113:0/905081700 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:18:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec27fc700 1 --2- 172.21.15.113:0/905081700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faec40fe250 0x7faec410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec2ffd700 1 -- 172.21.15.113:0/905081700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faec40fe250 msgr2=0x7faec410e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec2ffd700 1 --2- 172.21.15.113:0/905081700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faec40fe250 0x7faec410e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec2ffd700 1 -- 172.21.15.113:0/905081700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb8005040 con 0x7faec40fd8f0 2024-04-12T05:18:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec2ffd700 1 --2- 172.21.15.113:0/905081700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec40fd8f0 0x7faec410dfe0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7faeb8005010 tx=0x7faeb8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faeabfff700 1 -- 172.21.15.113:0/905081700 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeb800e030 con 0x7faec40fd8f0 2024-04-12T05:18:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faeabfff700 1 -- 172.21.15.113:0/905081700 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faeb8007e90 con 0x7faec40fd8f0 2024-04-12T05:18:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec97e4700 1 -- 172.21.15.113:0/905081700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec4108240 con 0x7faec40fd8f0 2024-04-12T05:18:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faeabfff700 1 -- 172.21.15.113:0/905081700 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeb8018070 con 0x7faec40fd8f0 2024-04-12T05:18:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.203+0000 7faec97e4700 1 -- 172.21.15.113:0/905081700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec41087a0 con 0x7faec40fd8f0 2024-04-12T05:18:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.204+0000 7faec97e4700 1 -- 172.21.15.113:0/905081700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec405f590 con 0x7faec40fd8f0 2024-04-12T05:18:30.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.205+0000 7faeabfff700 1 -- 172.21.15.113:0/905081700 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faeb801d670 con 0x7faec40fd8f0 2024-04-12T05:18:30.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.205+0000 7faeabfff700 1 --2- 172.21.15.113:0/905081700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeac06caa0 0x7faeac06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:30.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.205+0000 7faeabfff700 1 -- 172.21.15.113:0/905081700 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6221+0+0 (secure 0 0 0) 0x7faeb808d0e0 con 0x7faec40fd8f0 2024-04-12T05:18:30.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.205+0000 7faec27fc700 1 --2- 172.21.15.113:0/905081700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeac06caa0 0x7faeac06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:30.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.206+0000 7faec27fc700 1 --2- 172.21.15.113:0/905081700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeac06caa0 0x7faeac06ef60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7faec4109920 tx=0x7faeb400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:30.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.210+0000 7faeabfff700 1 -- 172.21.15.113:0/905081700 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7faeb805ab80 con 0x7faec40fd8f0 2024-04-12T05:18:30.423 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:30 smithi192 ceph-mon[85304]: osd.5 marked itself down and dead 2024-04-12T05:18:30.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:30 smithi113 ceph-mon[103371]: osd.5 marked itself down and dead 2024-04-12T05:18:30.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.476+0000 7faec97e4700 1 -- 172.21.15.113:0/905081700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faec40ffc40 con 0x7faec40fd8f0 2024-04-12T05:18:30.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.477+0000 7faeabfff700 1 -- 172.21.15.113:0/905081700 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+740 (secure 0 0 0) 0x7faeb805a710 con 0x7faec40fd8f0 2024-04-12T05:18:30.480 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:18:30.480 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:18:30.480 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:18:30.480 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 5 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 4, 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 9 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:18:30.481 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:18:30.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.479+0000 7faec97e4700 1 -- 172.21.15.113:0/905081700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeac06caa0 msgr2=0x7faeac06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.479+0000 7faec97e4700 1 --2- 172.21.15.113:0/905081700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeac06caa0 0x7faeac06ef60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7faec4109920 tx=0x7faeb400b040).stop 2024-04-12T05:18:30.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.479+0000 7faec97e4700 1 -- 172.21.15.113:0/905081700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec40fd8f0 msgr2=0x7faec410dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.479+0000 7faec97e4700 1 --2- 172.21.15.113:0/905081700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec40fd8f0 0x7faec410dfe0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7faeb8005010 tx=0x7faeb8009cb0).stop 2024-04-12T05:18:30.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.479+0000 7faec97e4700 1 -- 172.21.15.113:0/905081700 shutdown_connections 2024-04-12T05:18:30.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.479+0000 7faec97e4700 1 --2- 172.21.15.113:0/905081700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeac06caa0 0x7faeac06ef60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.479+0000 7faec97e4700 1 --2- 172.21.15.113:0/905081700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faec40fd8f0 0x7faec410dfe0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.479+0000 7faec97e4700 1 --2- 172.21.15.113:0/905081700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faec40fe250 0x7faec410e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.479+0000 7faec97e4700 1 -- 172.21.15.113:0/905081700 >> 172.21.15.113:0/905081700 conn(0x7faec40f34b0 msgr2=0x7faec40fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:30.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.479+0000 7faec97e4700 1 -- 172.21.15.113:0/905081700 shutdown_connections 2024-04-12T05:18:30.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.479+0000 7faec97e4700 1 -- 172.21.15.113:0/905081700 wait complete. 2024-04-12T05:18:30.581 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.578+0000 7f93d9d85700 1 -- 172.21.15.113:0/2553630954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d40f8900 msgr2=0x7f93d40f8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.581 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.578+0000 7f93d9d85700 1 --2- 172.21.15.113:0/2553630954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d40f8900 0x7f93d40f8ce0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f93c40047f0 tx=0x7f93c4009cb0).stop 2024-04-12T05:18:30.581 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.578+0000 7f93d9d85700 1 -- 172.21.15.113:0/2553630954 shutdown_connections 2024-04-12T05:18:30.581 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.578+0000 7f93d9d85700 1 --2- 172.21.15.113:0/2553630954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d40f9260 0x7f93d4101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.581 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.578+0000 7f93d9d85700 1 --2- 172.21.15.113:0/2553630954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d40f8900 0x7f93d40f8ce0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.578+0000 7f93d9d85700 1 -- 172.21.15.113:0/2553630954 >> 172.21.15.113:0/2553630954 conn(0x7f93d4070590 msgr2=0x7f93d40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:30.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.578+0000 7f93d9d85700 1 -- 172.21.15.113:0/2553630954 shutdown_connections 2024-04-12T05:18:30.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.578+0000 7f93d9d85700 1 -- 172.21.15.113:0/2553630954 wait complete. 2024-04-12T05:18:30.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.578+0000 7f93d9d85700 1 Processor -- start 2024-04-12T05:18:30.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d9d85700 1 -- start start 2024-04-12T05:18:30.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d9d85700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d40f8900 0x7f93d410e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:30.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d9d85700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d40f9260 0x7f93d410e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:30.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d9d85700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d410eb30 con 0x7f93d40f8900 2024-04-12T05:18:30.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d9d85700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d410ec70 con 0x7f93d40f9260 2024-04-12T05:18:30.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d40f8900 0x7f93d410e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:30.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d40f8900 0x7f93d410e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46588/0 (socket says 172.21.15.113:46588) 2024-04-12T05:18:30.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d37fe700 1 -- 172.21.15.113:0/3882661494 learned_addr learned my addr 172.21.15.113:0/3882661494 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:18:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d2ffd700 1 --2- 172.21.15.113:0/3882661494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d40f9260 0x7f93d410e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d37fe700 1 -- 172.21.15.113:0/3882661494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d40f9260 msgr2=0x7f93d410e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d37fe700 1 --2- 172.21.15.113:0/3882661494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d40f9260 0x7f93d410e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.579+0000 7f93d37fe700 1 -- 172.21.15.113:0/3882661494 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c4005040 con 0x7f93d40f8900 2024-04-12T05:18:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.580+0000 7f93d37fe700 1 --2- 172.21.15.113:0/3882661494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d40f8900 0x7f93d410e040 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f93c4009380 tx=0x7f93c4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.580+0000 7f93d0ff9700 1 -- 172.21.15.113:0/3882661494 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c400e070 con 0x7f93d40f8900 2024-04-12T05:18:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.580+0000 7f93d0ff9700 1 -- 172.21.15.113:0/3882661494 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93c4007e90 con 0x7f93d40f8900 2024-04-12T05:18:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.580+0000 7f93d9d85700 1 -- 172.21.15.113:0/3882661494 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d41082a0 con 0x7f93d40f8900 2024-04-12T05:18:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.580+0000 7f93d0ff9700 1 -- 172.21.15.113:0/3882661494 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c4018070 con 0x7f93d40f8900 2024-04-12T05:18:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.580+0000 7f93d9d85700 1 -- 172.21.15.113:0/3882661494 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d4108780 con 0x7f93d40f8900 2024-04-12T05:18:30.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.581+0000 7f93d9d85700 1 -- 172.21.15.113:0/3882661494 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d405f590 con 0x7f93d40f8900 2024-04-12T05:18:30.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.581+0000 7f93d0ff9700 1 -- 172.21.15.113:0/3882661494 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f93c401d420 con 0x7f93d40f8900 2024-04-12T05:18:30.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.582+0000 7f93d0ff9700 1 --2- 172.21.15.113:0/3882661494 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f93c006caa0 0x7f93c006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:30.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.582+0000 7f93d0ff9700 1 -- 172.21.15.113:0/3882661494 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f93c408c8e0 con 0x7f93d40f8900 2024-04-12T05:18:30.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.582+0000 7f93d2ffd700 1 --2- 172.21.15.113:0/3882661494 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f93c006caa0 0x7f93c006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:30.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.582+0000 7f93d2ffd700 1 --2- 172.21.15.113:0/3882661494 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f93c006caa0 0x7f93c006ef60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f93d4109980 tx=0x7f93bc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:30.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.586+0000 7f93d0ff9700 1 -- 172.21.15.113:0/3882661494 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f93c405a440 con 0x7f93d40f8900 2024-04-12T05:18:30.698 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:30 smithi192 bash[111264]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5 2024-04-12T05:18:30.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.812+0000 7f93d9d85700 1 -- 172.21.15.113:0/3882661494 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f93d41097c0 con 0x7f93d40f8900 2024-04-12T05:18:30.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.812+0000 7f93d0ff9700 1 -- 172.21.15.113:0/3882661494 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1859 (secure 0 0 0) 0x7f93c4059fd0 con 0x7f93d40f8900 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:e18 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 18 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:12:34.568261+0000 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:18:30.816 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 0 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=14426,1=24261} 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:18:30.817 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:14426} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.113:6826/1262609849,v1:172.21.15.113:6827/1262609849] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{1:24261} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.192:6824/616350866,v1:172.21.15.192:6825/616350866] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:18:30.818 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:18:30.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.814+0000 7f93d9d85700 1 -- 172.21.15.113:0/3882661494 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f93c006caa0 msgr2=0x7f93c006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.814+0000 7f93d9d85700 1 --2- 172.21.15.113:0/3882661494 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f93c006caa0 0x7f93c006ef60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f93d4109980 tx=0x7f93bc00b040).stop 2024-04-12T05:18:30.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.814+0000 7f93d9d85700 1 -- 172.21.15.113:0/3882661494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d40f8900 msgr2=0x7f93d410e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.814+0000 7f93d9d85700 1 --2- 172.21.15.113:0/3882661494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d40f8900 0x7f93d410e040 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f93c4009380 tx=0x7f93c4009cb0).stop 2024-04-12T05:18:30.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.814+0000 7f93d9d85700 1 -- 172.21.15.113:0/3882661494 shutdown_connections 2024-04-12T05:18:30.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.814+0000 7f93d9d85700 1 --2- 172.21.15.113:0/3882661494 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f93c006caa0 0x7f93c006ef60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.814+0000 7f93d9d85700 1 --2- 172.21.15.113:0/3882661494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d40f8900 0x7f93d410e040 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.814+0000 7f93d9d85700 1 --2- 172.21.15.113:0/3882661494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d40f9260 0x7f93d410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.814+0000 7f93d9d85700 1 -- 172.21.15.113:0/3882661494 >> 172.21.15.113:0/3882661494 conn(0x7f93d4070590 msgr2=0x7f93d40fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:30.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.814+0000 7f93d9d85700 1 -- 172.21.15.113:0/3882661494 shutdown_connections 2024-04-12T05:18:30.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.814+0000 7f93d9d85700 1 -- 172.21.15.113:0/3882661494 wait complete. 2024-04-12T05:18:30.820 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 18 2024-04-12T05:18:30.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.912+0000 7ff81059f700 1 -- 172.21.15.113:0/1442057835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8080fe270 msgr2=0x7ff8080f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.912+0000 7ff81059f700 1 --2- 172.21.15.113:0/1442057835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8080fe270 0x7ff8080f7a50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ff7f80047f0 tx=0x7ff7f8009cb0).stop 2024-04-12T05:18:30.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 -- 172.21.15.113:0/1442057835 shutdown_connections 2024-04-12T05:18:30.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 --2- 172.21.15.113:0/1442057835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8080fe270 0x7ff8080f7a50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 --2- 172.21.15.113:0/1442057835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8080fd910 0x7ff8080fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 -- 172.21.15.113:0/1442057835 >> 172.21.15.113:0/1442057835 conn(0x7ff8080f3530 msgr2=0x7ff8080f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 -- 172.21.15.113:0/1442057835 shutdown_connections 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 -- 172.21.15.113:0/1442057835 wait complete. 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 Processor -- start 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 -- start start 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8080fd910 0x7ff80810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8080fe270 0x7ff80810e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80810ead0 con 0x7ff8080fd910 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.913+0000 7ff81059f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80810ec10 con 0x7ff8080fe270 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff80e33b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8080fd910 0x7ff80810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff80e33b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8080fd910 0x7ff80810dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46612/0 (socket says 172.21.15.113:46612) 2024-04-12T05:18:30.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff80e33b700 1 -- 172.21.15.113:0/3805837305 learned_addr learned my addr 172.21.15.113:0/3805837305 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:18:30.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff80db3a700 1 --2- 172.21.15.113:0/3805837305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8080fe270 0x7ff80810e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:30.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff80e33b700 1 -- 172.21.15.113:0/3805837305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8080fe270 msgr2=0x7ff80810e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:30.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff80e33b700 1 --2- 172.21.15.113:0/3805837305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8080fe270 0x7ff80810e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:30.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff80e33b700 1 -- 172.21.15.113:0/3805837305 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f8005040 con 0x7ff8080fd910 2024-04-12T05:18:30.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff80e33b700 1 --2- 172.21.15.113:0/3805837305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8080fd910 0x7ff80810dfe0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff8040041d0 tx=0x7ff804009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:30.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff7ff7fe700 1 -- 172.21.15.113:0/3805837305 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff804007b40 con 0x7ff8080fd910 2024-04-12T05:18:30.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff81059f700 1 -- 172.21.15.113:0/3805837305 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8081082a0 con 0x7ff8080fd910 2024-04-12T05:18:30.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff7ff7fe700 1 -- 172.21.15.113:0/3805837305 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff80400f040 con 0x7ff8080fd910 2024-04-12T05:18:30.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff7ff7fe700 1 -- 172.21.15.113:0/3805837305 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8040147b0 con 0x7ff8080fd910 2024-04-12T05:18:30.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.914+0000 7ff81059f700 1 -- 172.21.15.113:0/3805837305 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff808108860 con 0x7ff8080fd910 2024-04-12T05:18:30.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.915+0000 7ff81059f700 1 -- 172.21.15.113:0/3805837305 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80805f570 con 0x7ff8080fd910 2024-04-12T05:18:30.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.915+0000 7ff7ff7fe700 1 -- 172.21.15.113:0/3805837305 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff804009110 con 0x7ff8080fd910 2024-04-12T05:18:30.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.916+0000 7ff7ff7fe700 1 --2- 172.21.15.113:0/3805837305 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff7f406caa0 0x7ff7f406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:30.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.916+0000 7ff7ff7fe700 1 -- 172.21.15.113:0/3805837305 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6221+0+0 (secure 0 0 0) 0x7ff80408bda0 con 0x7ff8080fd910 2024-04-12T05:18:30.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.916+0000 7ff80db3a700 1 --2- 172.21.15.113:0/3805837305 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff7f406caa0 0x7ff7f406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:30.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.916+0000 7ff80db3a700 1 --2- 172.21.15.113:0/3805837305 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff7f406caa0 0x7ff7f406ef60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7ff808109920 tx=0x7ff7f800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:30.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:30.920+0000 7ff7ff7fe700 1 -- 172.21.15.113:0/3805837305 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff804059900 con 0x7ff8080fd910 2024-04-12T05:18:31.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.144+0000 7ff81059f700 1 -- 172.21.15.113:0/3805837305 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff808108ae0 con 0x7ff7f406caa0 2024-04-12T05:18:31.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.145+0000 7ff7ff7fe700 1 -- 172.21.15.113:0/3805837305 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7ff808108ae0 con 0x7ff7f406caa0 2024-04-12T05:18:31.148 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:18:31.148 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:18:31.148 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:18:31.148 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:18:31.149 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:18:31.149 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:18:31.149 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:18:31.149 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:18:31.149 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:18:31.149 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "11/21 daemons upgraded", 2024-04-12T05:18:31.149 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T05:18:31.149 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:18:31.149 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:18:31.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.147+0000 7ff81059f700 1 -- 172.21.15.113:0/3805837305 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff7f406caa0 msgr2=0x7ff7f406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:31.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.147+0000 7ff81059f700 1 --2- 172.21.15.113:0/3805837305 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff7f406caa0 0x7ff7f406ef60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7ff808109920 tx=0x7ff7f800b040).stop 2024-04-12T05:18:31.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.147+0000 7ff81059f700 1 -- 172.21.15.113:0/3805837305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8080fd910 msgr2=0x7ff80810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:31.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.147+0000 7ff81059f700 1 --2- 172.21.15.113:0/3805837305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8080fd910 0x7ff80810dfe0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff8040041d0 tx=0x7ff804009f60).stop 2024-04-12T05:18:31.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.147+0000 7ff81059f700 1 -- 172.21.15.113:0/3805837305 shutdown_connections 2024-04-12T05:18:31.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.147+0000 7ff81059f700 1 --2- 172.21.15.113:0/3805837305 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff7f406caa0 0x7ff7f406ef60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:31.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.147+0000 7ff81059f700 1 --2- 172.21.15.113:0/3805837305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8080fd910 0x7ff80810dfe0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:31.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.147+0000 7ff81059f700 1 --2- 172.21.15.113:0/3805837305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8080fe270 0x7ff80810e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:31.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.147+0000 7ff81059f700 1 -- 172.21.15.113:0/3805837305 >> 172.21.15.113:0/3805837305 conn(0x7ff8080f3530 msgr2=0x7ff8080fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:31.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.147+0000 7ff81059f700 1 -- 172.21.15.113:0/3805837305 shutdown_connections 2024-04-12T05:18:31.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.147+0000 7ff81059f700 1 -- 172.21.15.113:0/3805837305 wait complete. 2024-04-12T05:18:31.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.245+0000 7f911cd8f700 1 -- 172.21.15.113:0/1565750665 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180fd150 msgr2=0x7f91180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.245+0000 7f911cd8f700 1 --2- 172.21.15.113:0/1565750665 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180fd150 0x7f91180f6930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f91080047f0 tx=0x7f9108009cb0).stop 2024-04-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.245+0000 7f911cd8f700 1 -- 172.21.15.113:0/1565750665 shutdown_connections 2024-04-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.245+0000 7f911cd8f700 1 --2- 172.21.15.113:0/1565750665 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180fd150 0x7f91180f6930 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.245+0000 7f911cd8f700 1 --2- 172.21.15.113:0/1565750665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91180fc7f0 0x7f91180fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.245+0000 7f911cd8f700 1 -- 172.21.15.113:0/1565750665 >> 172.21.15.113:0/1565750665 conn(0x7f9118070590 msgr2=0x7f91180f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.245+0000 7f911cd8f700 1 -- 172.21.15.113:0/1565750665 shutdown_connections 2024-04-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.245+0000 7f911cd8f700 1 -- 172.21.15.113:0/1565750665 wait complete. 2024-04-12T05:18:31.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.246+0000 7f911cd8f700 1 Processor -- start 2024-04-12T05:18:31.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.246+0000 7f911cd8f700 1 -- start start 2024-04-12T05:18:31.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.246+0000 7f911cd8f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91180fc7f0 0x7f91180f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:31.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.246+0000 7f911cd8f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180fd150 0x7f91180f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:31.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.246+0000 7f911cd8f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91180f8f20 con 0x7f91180fd150 2024-04-12T05:18:31.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.246+0000 7f911cd8f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91180f9060 con 0x7f91180fc7f0 2024-04-12T05:18:31.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.246+0000 7f9115d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180fd150 0x7f91180f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.246+0000 7f9115d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180fd150 0x7f91180f8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:46616/0 (socket says 172.21.15.113:46616) 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.246+0000 7f9115d9b700 1 -- 172.21.15.113:0/795283436 learned_addr learned my addr 172.21.15.113:0/795283436 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.247+0000 7f911659c700 1 --2- 172.21.15.113:0/795283436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91180fc7f0 0x7f91180f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.247+0000 7f9115d9b700 1 -- 172.21.15.113:0/795283436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91180fc7f0 msgr2=0x7f91180f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.247+0000 7f9115d9b700 1 --2- 172.21.15.113:0/795283436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91180fc7f0 0x7f91180f8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.247+0000 7f9115d9b700 1 -- 172.21.15.113:0/795283436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9108005040 con 0x7f91180fd150 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.247+0000 7f9115d9b700 1 --2- 172.21.15.113:0/795283436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180fd150 0x7f91180f8a30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f9108000c00 tx=0x7f9108009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.247+0000 7f910f7fe700 1 -- 172.21.15.113:0/795283436 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9108018070 con 0x7f91180fd150 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.247+0000 7f910f7fe700 1 -- 172.21.15.113:0/795283436 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9108007e90 con 0x7f91180fd150 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.247+0000 7f911cd8f700 1 -- 172.21.15.113:0/795283436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91180f92e0 con 0x7f91180fd150 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.247+0000 7f910f7fe700 1 -- 172.21.15.113:0/795283436 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91080123c0 con 0x7f91180fd150 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.247+0000 7f911cd8f700 1 -- 172.21.15.113:0/795283436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91181146f0 con 0x7f91180fd150 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.248+0000 7f911cd8f700 1 -- 172.21.15.113:0/795283436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911805f590 con 0x7f91180fd150 2024-04-12T05:18:31.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.248+0000 7f910f7fe700 1 -- 172.21.15.113:0/795283436 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9108012630 con 0x7f91180fd150 2024-04-12T05:18:31.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.249+0000 7f910f7fe700 1 --2- 172.21.15.113:0/795283436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f910406caa0 0x7f910406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:18:31.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.249+0000 7f911659c700 1 --2- 172.21.15.113:0/795283436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f910406caa0 0x7f910406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:18:31.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.249+0000 7f910f7fe700 1 -- 172.21.15.113:0/795283436 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6221+0+0 (secure 0 0 0) 0x7f910808c900 con 0x7f91180fd150 2024-04-12T05:18:31.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.249+0000 7f911659c700 1 --2- 172.21.15.113:0/795283436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f910406caa0 0x7f910406ef60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f91000092f0 tx=0x7f910000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:18:31.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.253+0000 7f910f7fe700 1 -- 172.21.15.113:0/795283436 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9108056a40 con 0x7f91180fd150 2024-04-12T05:18:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:31 smithi113 ceph-mon[103371]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:18:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:31 smithi113 ceph-mon[103371]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:18:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:31 smithi113 ceph-mon[103371]: pgmap v181: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:31 smithi113 ceph-mon[103371]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:18:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:31 smithi113 ceph-mon[103371]: Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2024-04-12T05:18:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:31 smithi113 ceph-mon[103371]: osdmap e89: 6 total, 5 up, 6 in 2024-04-12T05:18:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:31 smithi113 ceph-mon[103371]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:18:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:31 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/905081700' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:18:31.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:31 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3882661494' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:18:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:31 smithi192 ceph-mon[85304]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:18:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:31 smithi192 ceph-mon[85304]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:18:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:31 smithi192 ceph-mon[85304]: pgmap v181: 65 pgs: 65 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:31 smithi192 ceph-mon[85304]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T05:18:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:31 smithi192 ceph-mon[85304]: Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2024-04-12T05:18:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:31 smithi192 ceph-mon[85304]: osdmap e89: 6 total, 5 up, 6 in 2024-04-12T05:18:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:31 smithi192 ceph-mon[85304]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:18:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:31 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/905081700' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:18:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:31 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3882661494' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:18:31.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.522+0000 7f911cd8f700 1 -- 172.21.15.113:0/795283436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9118047540 con 0x7f91180fd150 2024-04-12T05:18:31.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.522+0000 7f910f7fe700 1 -- 172.21.15.113:0/795283436 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+343 (secure 0 0 0) 0x7f910805a060 con 0x7f91180fd150 2024-04-12T05:18:31.526 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_WARN 1 osds down; all OSDs are running quincy or later but require_osd_release < quincy 2024-04-12T05:18:31.526 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T05:18:31.526 INFO:teuthology.orchestra.run.smithi113.stdout: osd.5 (root=default,host=smithi192) is down 2024-04-12T05:18:31.526 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running quincy or later but require_osd_release < quincy 2024-04-12T05:18:31.526 INFO:teuthology.orchestra.run.smithi113.stdout: all OSDs are running quincy or later but require_osd_release < quincy 2024-04-12T05:18:31.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.524+0000 7f911cd8f700 1 -- 172.21.15.113:0/795283436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f910406caa0 msgr2=0x7f910406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:31.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.524+0000 7f911cd8f700 1 --2- 172.21.15.113:0/795283436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f910406caa0 0x7f910406ef60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f91000092f0 tx=0x7f910000b040).stop 2024-04-12T05:18:31.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.524+0000 7f911cd8f700 1 -- 172.21.15.113:0/795283436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180fd150 msgr2=0x7f91180f8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:18:31.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.524+0000 7f911cd8f700 1 --2- 172.21.15.113:0/795283436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180fd150 0x7f91180f8a30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f9108000c00 tx=0x7f9108009cb0).stop 2024-04-12T05:18:31.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.525+0000 7f911cd8f700 1 -- 172.21.15.113:0/795283436 shutdown_connections 2024-04-12T05:18:31.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.525+0000 7f911cd8f700 1 --2- 172.21.15.113:0/795283436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f910406caa0 0x7f910406ef60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:31.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.525+0000 7f911cd8f700 1 --2- 172.21.15.113:0/795283436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91180fc7f0 0x7f91180f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:31.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.525+0000 7f911cd8f700 1 --2- 172.21.15.113:0/795283436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91180fd150 0x7f91180f8a30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:18:31.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.525+0000 7f911cd8f700 1 -- 172.21.15.113:0/795283436 >> 172.21.15.113:0/795283436 conn(0x7f9118070590 msgr2=0x7f9118060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:18:31.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.525+0000 7f911cd8f700 1 -- 172.21.15.113:0/795283436 shutdown_connections 2024-04-12T05:18:31.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:18:31.525+0000 7f911cd8f700 1 -- 172.21.15.113:0/795283436 wait complete. 2024-04-12T05:18:32.241 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:32 smithi192 ceph-mon[85304]: osdmap e90: 6 total, 5 up, 6 in 2024-04-12T05:18:32.241 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:32 smithi192 ceph-mon[85304]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:18:32.241 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:32 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/795283436' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:18:32.241 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:32 smithi192 ceph-mon[85304]: pgmap v184: 65 pgs: 14 stale+active+clean, 51 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:32.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:32 smithi113 ceph-mon[103371]: osdmap e90: 6 total, 5 up, 6 in 2024-04-12T05:18:32.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:32 smithi113 ceph-mon[103371]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:18:32.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:32 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/795283436' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:18:32.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:32 smithi113 ceph-mon[103371]: pgmap v184: 65 pgs: 14 stale+active+clean, 51 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:32.698 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:32 smithi192 systemd[1]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.5.service: Succeeded. 2024-04-12T05:18:32.699 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:32 smithi192 systemd[1]: Stopped Ceph osd.5 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:18:32.699 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:32 smithi192 systemd[1]: Starting Ceph osd.5 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:18:33.663 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:33 smithi192 bash[111540]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-12T05:18:33.663 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:33 smithi192 bash[111540]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-12T05:18:33.664 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:33 smithi192 bash[111540]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-12T05:18:33.664 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:33 smithi192 bash[111540]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-12T05:18:33.664 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:33 smithi192 bash[111540]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-12T05:18:33.664 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:33 smithi192 bash[111540]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-12T05:18:33.664 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:33 smithi192 bash[111540]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-12T05:18:34.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:33 smithi192 ceph-mon[85304]: Health check failed: Degraded data redundancy: 7/324 objects degraded (2.160%), 2 pgs degraded (PG_DEGRADED) 2024-04-12T05:18:34.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:33 smithi113 ceph-mon[103371]: Health check failed: Degraded data redundancy: 7/324 objects degraded (2.160%), 2 pgs degraded (PG_DEGRADED) 2024-04-12T05:18:34.698 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:34 smithi192 ceph-osd[111860]: -- 172.21.15.192:0/941187790 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5595221d11e0 con 0x5595221a1000 2024-04-12T05:18:34.698 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:34 smithi192 bash[111822]: fbe8b39c4b06f58b83a956a29d68c30575bdc091cc2355bbd9e3c8bef8a06b0a 2024-04-12T05:18:34.699 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:34 smithi192 systemd[1]: Started Ceph osd.5 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:18:35.199 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:35 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:18:35.024+0000 7ff4122597c0 -1 Falling back to public interface 2024-04-12T05:18:35.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:34 smithi192 ceph-mon[85304]: pgmap v185: 65 pgs: 4 active+undersized, 2 active+undersized+degraded, 11 stale+active+clean, 48 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7/324 objects degraded (2.160%) 2024-04-12T05:18:35.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:34 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:35.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:34 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:35.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:34 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:18:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:34 smithi113 ceph-mon[103371]: pgmap v185: 65 pgs: 4 active+undersized, 2 active+undersized+degraded, 11 stale+active+clean, 48 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7/324 objects degraded (2.160%) 2024-04-12T05:18:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:34 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:34 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:34 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:18:37.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:36 smithi192 ceph-mon[85304]: pgmap v186: 65 pgs: 15 active+undersized, 14 active+undersized+degraded, 36 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 50/324 objects degraded (15.432%) 2024-04-12T05:18:37.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:37.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:18:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:36 smithi113 ceph-mon[103371]: pgmap v186: 65 pgs: 15 active+undersized, 14 active+undersized+degraded, 36 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 50/324 objects degraded (15.432%) 2024-04-12T05:18:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:18:39.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:38 smithi192 ceph-mon[85304]: pgmap v187: 65 pgs: 15 active+undersized, 14 active+undersized+degraded, 36 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 50/324 objects degraded (15.432%) 2024-04-12T05:18:39.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:39.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:38 smithi113 ceph-mon[103371]: pgmap v187: 65 pgs: 15 active+undersized, 14 active+undersized+degraded, 36 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 50/324 objects degraded (15.432%) 2024-04-12T05:18:39.449 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:39.449 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:40.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:40.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:40.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:40 smithi113 ceph-mon[103371]: pgmap v188: 65 pgs: 15 active+undersized, 14 active+undersized+degraded, 36 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 50/324 objects degraded (15.432%) 2024-04-12T05:18:40.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:40.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:40.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:40 smithi192 ceph-mon[85304]: pgmap v188: 65 pgs: 15 active+undersized, 14 active+undersized+degraded, 36 active+clean; 259 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 50/324 objects degraded (15.432%) 2024-04-12T05:18:41.198 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:40 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:18:40.856+0000 7ff4122597c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-12T05:18:41.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:41 smithi113 ceph-mon[103371]: from='osd.5 [v2:172.21.15.192:6816/2093584908,v1:172.21.15.192:6817/2093584908]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:18:41.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:41 smithi113 ceph-mon[103371]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:18:41.699 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:41 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:18:41.293+0000 7ff4122597c0 -1 osd.5 88 log_to_monitors true 2024-04-12T05:18:41.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:41 smithi192 ceph-mon[85304]: from='osd.5 [v2:172.21.15.192:6816/2093584908,v1:172.21.15.192:6817/2093584908]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:18:41.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:41 smithi192 ceph-mon[85304]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T05:18:42.448 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:18:42 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:18:42.277+0000 7ff3ff6ab700 -1 osd.5 88 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T05:18:42.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:42 smithi192 ceph-mon[85304]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T05:18:42.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:42 smithi192 ceph-mon[85304]: osdmap e91: 6 total, 5 up, 6 in 2024-04-12T05:18:42.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:42 smithi192 ceph-mon[85304]: from='osd.5 [v2:172.21.15.192:6816/2093584908,v1:172.21.15.192:6817/2093584908]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:18:42.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:42 smithi192 ceph-mon[85304]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:18:42.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:42 smithi192 ceph-mon[85304]: pgmap v190: 65 pgs: 15 active+undersized, 14 active+undersized+degraded, 36 active+clean; 259 MiB data, 999 MiB used, 535 GiB / 536 GiB avail; 50/324 objects degraded (15.432%) 2024-04-12T05:18:42.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:42 smithi192 ceph-mon[85304]: Health check update: Degraded data redundancy: 50/324 objects degraded (15.432%), 14 pgs degraded (PG_DEGRADED) 2024-04-12T05:18:42.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:42 smithi192 ceph-mon[85304]: from='osd.5 ' entity='osd.5' 2024-04-12T05:18:42.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:42 smithi113 ceph-mon[103371]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T05:18:42.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:42 smithi113 ceph-mon[103371]: osdmap e91: 6 total, 5 up, 6 in 2024-04-12T05:18:42.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:42 smithi113 ceph-mon[103371]: from='osd.5 [v2:172.21.15.192:6816/2093584908,v1:172.21.15.192:6817/2093584908]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:18:42.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:42 smithi113 ceph-mon[103371]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-12T05:18:42.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:42 smithi113 ceph-mon[103371]: pgmap v190: 65 pgs: 15 active+undersized, 14 active+undersized+degraded, 36 active+clean; 259 MiB data, 999 MiB used, 535 GiB / 536 GiB avail; 50/324 objects degraded (15.432%) 2024-04-12T05:18:42.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:42 smithi113 ceph-mon[103371]: Health check update: Degraded data redundancy: 50/324 objects degraded (15.432%), 14 pgs degraded (PG_DEGRADED) 2024-04-12T05:18:42.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:42 smithi113 ceph-mon[103371]: from='osd.5 ' entity='osd.5' 2024-04-12T05:18:43.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:18:43.275+0000 7f142642e700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:18:44.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:44.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:44.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:18:44.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:18:44.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:44.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:18:44.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:18:44.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:18:44.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:18:44.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:18:44.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:44.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-12T05:18:44.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2024-04-12T05:18:44.176 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: osd.5 [v2:172.21.15.192:6816/2093584908,v1:172.21.15.192:6817/2093584908] boot 2024-04-12T05:18:44.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-04-12T05:18:44.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: osdmap e92: 6 total, 6 up, 6 in 2024-04-12T05:18:44.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:18:44.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-12T05:18:44.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T05:18:44.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:18:44.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: osdmap e93: 6 total, 6 up, 6 in 2024-04-12T05:18:44.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-12T05:18:44.177 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:43 smithi113 ceph-mon[103371]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-12T05:18:44.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:44.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:44.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-12T05:18:44.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: osd.5 [v2:172.21.15.192:6816/2093584908,v1:172.21.15.192:6817/2093584908] boot 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: osdmap e92: 6 total, 6 up, 6 in 2024-04-12T05:18:44.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T05:18:44.452 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-12T05:18:44.452 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T05:18:44.452 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T05:18:44.452 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: osdmap e93: 6 total, 6 up, 6 in 2024-04-12T05:18:44.452 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-12T05:18:44.452 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:43 smithi192 ceph-mon[85304]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-12T05:18:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all osd 2024-04-12T05:18:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: Upgrade: Setting require_osd_release to 18 reef 2024-04-12T05:18:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-12T05:18:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: pgmap v193: 65 pgs: 2 peering, 14 active+undersized, 13 active+undersized+degraded, 36 active+clean; 259 MiB data, 999 MiB used, 535 GiB / 536 GiB avail; 46/324 objects degraded (14.198%) 2024-04-12T05:18:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: osdmap e94: 6 total, 6 up, 6 in 2024-04-12T05:18:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.113:6826/2944373762,v1:172.21.15.113:6827/2944373762] up:boot 2024-04-12T05:18:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.192:6824/2802712879,v1:172.21.15.192:6825/2802712879] up:boot 2024-04-12T05:18:45.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:18:45.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:18:45.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:18:45.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:45.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ovmadh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:18:45.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:44 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:18:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all osd 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: Upgrade: Setting require_osd_release to 18 reef 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: pgmap v193: 65 pgs: 2 peering, 14 active+undersized, 13 active+undersized+degraded, 36 active+clean; 259 MiB data, 999 MiB used, 535 GiB / 536 GiB avail; 46/324 objects degraded (14.198%) 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: osdmap e94: 6 total, 6 up, 6 in 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.113:6826/2944373762,v1:172.21.15.113:6827/2944373762] up:boot 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.192:6824/2802712879,v1:172.21.15.192:6825/2802712879] up:boot 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ovmadh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:18:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:44 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:18:46.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:46 smithi113 ceph-mon[103371]: Upgrade: Updating mds.cephfs.smithi113.ovmadh 2024-04-12T05:18:46.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:46 smithi113 ceph-mon[103371]: Deploying daemon mds.cephfs.smithi113.ovmadh on smithi113 2024-04-12T05:18:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:46 smithi192 ceph-mon[85304]: Upgrade: Updating mds.cephfs.smithi113.ovmadh 2024-04-12T05:18:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:46 smithi192 ceph-mon[85304]: Deploying daemon mds.cephfs.smithi113.ovmadh on smithi113 2024-04-12T05:18:47.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:47 smithi113 ceph-mon[103371]: pgmap v195: 65 pgs: 5 peering, 10 active+undersized, 5 active+undersized+degraded, 45 active+clean; 259 MiB data, 999 MiB used, 535 GiB / 536 GiB avail; 15/324 objects degraded (4.630%) 2024-04-12T05:18:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:47 smithi113 ceph-mon[103371]: osdmap e95: 6 total, 6 up, 6 in 2024-04-12T05:18:47.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:47 smithi113 ceph-mon[103371]: Health check update: Degraded data redundancy: 15/324 objects degraded (4.630%), 5 pgs degraded (PG_DEGRADED) 2024-04-12T05:18:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:47 smithi192 ceph-mon[85304]: pgmap v195: 65 pgs: 5 peering, 10 active+undersized, 5 active+undersized+degraded, 45 active+clean; 259 MiB data, 999 MiB used, 535 GiB / 536 GiB avail; 15/324 objects degraded (4.630%) 2024-04-12T05:18:47.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:47 smithi192 ceph-mon[85304]: osdmap e95: 6 total, 6 up, 6 in 2024-04-12T05:18:47.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:47 smithi192 ceph-mon[85304]: Health check update: Degraded data redundancy: 15/324 objects degraded (4.630%), 5 pgs degraded (PG_DEGRADED) 2024-04-12T05:18:48.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:48 smithi113 ceph-mon[103371]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:18:48.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:48 smithi192 ceph-mon[85304]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:18:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:49 smithi113 ceph-mon[103371]: pgmap v197: 65 pgs: 5 peering, 1 active+undersized, 59 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:49 smithi113 ceph-mon[103371]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/324 objects degraded (4.630%), 5 pgs degraded) 2024-04-12T05:18:49.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:49 smithi192 ceph-mon[85304]: pgmap v197: 65 pgs: 5 peering, 1 active+undersized, 59 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:49.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:49 smithi192 ceph-mon[85304]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/324 objects degraded (4.630%), 5 pgs degraded) 2024-04-12T05:18:50.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:50 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:18:50.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:50 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:18:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:51 smithi113 ceph-mon[103371]: pgmap v198: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:51 smithi192 ceph-mon[85304]: pgmap v198: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:18:53.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:52 smithi113 ceph-mon[103371]: pgmap v199: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:53.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:53.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:53.175 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:18:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:53.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:18:53.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:52 smithi192 ceph-mon[85304]: pgmap v199: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:53.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:53.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:53.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:18:54.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:53 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] up:boot 2024-04-12T05:18:54.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:53 smithi113 ceph-mon[103371]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:18:54.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:18:54.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:53 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] up:boot 2024-04-12T05:18:54.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:53 smithi192 ceph-mon[85304]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:18:54.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:18:55.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:54 smithi113 ceph-mon[103371]: pgmap v200: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:55.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:54 smithi192 ceph-mon[85304]: pgmap v200: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:57.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:56 smithi113 ceph-mon[103371]: pgmap v201: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:57.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:56 smithi192 ceph-mon[85304]: pgmap v201: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:57 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:57 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:58.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:57 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:58.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:57 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:58.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:57 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:58.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:57 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:59.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:58 smithi113 ceph-mon[103371]: pgmap v202: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:59.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:58 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:59.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:18:58 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:58 smithi192 ceph-mon[85304]: pgmap v202: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:18:59.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:58 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:18:59.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:18:58 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:00 smithi113 ceph-mon[103371]: pgmap v203: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:00 smithi192 ceph-mon[85304]: pgmap v203: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:01.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.628+0000 7f07984b7700 1 -- 172.21.15.113:0/1225546819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07900f8a10 msgr2=0x7f07900f8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:01.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.628+0000 7f07984b7700 1 --2- 172.21.15.113:0/1225546819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07900f8a10 0x7f07900f8df0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f078c0047f0 tx=0x7f078c009cb0).stop 2024-04-12T05:19:01.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.628+0000 7f07984b7700 1 -- 172.21.15.113:0/1225546819 shutdown_connections 2024-04-12T05:19:01.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.628+0000 7f07984b7700 1 --2- 172.21.15.113:0/1225546819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07900f92e0 0x7f0790101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:19:01.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.628+0000 7f07984b7700 1 --2- 172.21.15.113:0/1225546819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07900f8a10 0x7f07900f8df0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.628+0000 7f07984b7700 1 -- 172.21.15.113:0/1225546819 >> 172.21.15.113:0/1225546819 conn(0x7f079006c760 msgr2=0x7f079006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.628+0000 7f07984b7700 1 -- 172.21.15.113:0/1225546819 shutdown_connections 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.628+0000 7f07984b7700 1 -- 172.21.15.113:0/1225546819 wait complete. 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f07984b7700 1 Processor -- start 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f07984b7700 1 -- start start 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f07984b7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07900f8a10 0x7f07900f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f07984b7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07900f92e0 0x7f07900f6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f07984b7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07900f73a0 con 0x7f07900f8a10 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f07984b7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07900f74e0 con 0x7f07900f92e0 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f0796253700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07900f8a10 0x7f07900f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f0796253700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07900f8a10 0x7f07900f69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36014/0 (socket says 172.21.15.113:36014) 2024-04-12T05:19:01.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f0796253700 1 -- 172.21.15.113:0/1029803990 learned_addr learned my addr 172.21.15.113:0/1029803990 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:01.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f0795a52700 1 --2- 172.21.15.113:0/1029803990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07900f92e0 0x7f07900f6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:01.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f0796253700 1 -- 172.21.15.113:0/1029803990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07900f92e0 msgr2=0x7f07900f6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:01.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f0796253700 1 --2- 172.21.15.113:0/1029803990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07900f92e0 0x7f07900f6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:01.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f0796253700 1 -- 172.21.15.113:0/1029803990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c005040 con 0x7f07900f8a10 2024-04-12T05:19:01.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.629+0000 7f0796253700 1 --2- 172.21.15.113:0/1029803990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07900f8a10 0x7f07900f69c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f078c013640 tx=0x7f078c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:01.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.630+0000 7f07877fe700 1 -- 172.21.15.113:0/1029803990 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f078c00e030 con 0x7f07900f8a10 2024-04-12T05:19:01.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.630+0000 7f07877fe700 1 -- 172.21.15.113:0/1029803990 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f078c007e90 con 0x7f07900f8a10 2024-04-12T05:19:01.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.630+0000 7f07877fe700 1 -- 172.21.15.113:0/1029803990 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f078c018070 con 0x7f07900f8a10 2024-04-12T05:19:01.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.630+0000 7f07984b7700 1 -- 172.21.15.113:0/1029803990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07900f7730 con 0x7f07900f8a10 2024-04-12T05:19:01.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.630+0000 7f07984b7700 1 -- 172.21.15.113:0/1029803990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0790114740 con 0x7f07900f8a10 2024-04-12T05:19:01.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.631+0000 7f07984b7700 1 -- 172.21.15.113:0/1029803990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079005f550 con 0x7f07900f8a10 2024-04-12T05:19:01.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.631+0000 7f07877fe700 1 -- 172.21.15.113:0/1029803990 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f078c01d420 con 0x7f07900f8a10 2024-04-12T05:19:01.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.632+0000 7f07877fe700 1 --2- 172.21.15.113:0/1029803990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f077c06cab0 0x7f077c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:01.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.632+0000 7f07877fe700 1 -- 172.21.15.113:0/1029803990 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f078c08d1e0 con 0x7f07900f8a10 2024-04-12T05:19:01.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.632+0000 7f0795a52700 1 --2- 172.21.15.113:0/1029803990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f077c06cab0 0x7f077c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:01.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.632+0000 7f0795a52700 1 --2- 172.21.15.113:0/1029803990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f077c06cab0 0x7f077c06ef70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f07900f8160 tx=0x7f0780008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:01.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.636+0000 7f07877fe700 1 -- 172.21.15.113:0/1029803990 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f078c057200 con 0x7f07900f8a10 2024-04-12T05:19:01.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.842+0000 7f07984b7700 1 -- 172.21.15.113:0/1029803990 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0790059120 con 0x7f077c06cab0 2024-04-12T05:19:01.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.843+0000 7f07877fe700 1 -- 172.21.15.113:0/1029803990 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0790059120 con 0x7f077c06cab0 2024-04-12T05:19:01.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.846+0000 7f07984b7700 1 -- 172.21.15.113:0/1029803990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f077c06cab0 msgr2=0x7f077c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:01.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.846+0000 7f07984b7700 1 --2- 172.21.15.113:0/1029803990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f077c06cab0 0x7f077c06ef70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f07900f8160 tx=0x7f0780008040).stop 2024-04-12T05:19:01.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.846+0000 7f07984b7700 1 -- 172.21.15.113:0/1029803990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07900f8a10 msgr2=0x7f07900f69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:01.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.846+0000 7f07984b7700 1 --2- 172.21.15.113:0/1029803990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07900f8a10 0x7f07900f69c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f078c013640 tx=0x7f078c009cb0).stop 2024-04-12T05:19:01.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.846+0000 7f07984b7700 1 -- 172.21.15.113:0/1029803990 shutdown_connections 2024-04-12T05:19:01.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.846+0000 7f07984b7700 1 --2- 172.21.15.113:0/1029803990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f077c06cab0 0x7f077c06ef70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:01.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.846+0000 7f07984b7700 1 --2- 172.21.15.113:0/1029803990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07900f8a10 0x7f07900f69c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:01.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.846+0000 7f07984b7700 1 --2- 172.21.15.113:0/1029803990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07900f92e0 0x7f07900f6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:01.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.846+0000 7f07984b7700 1 -- 172.21.15.113:0/1029803990 >> 172.21.15.113:0/1029803990 conn(0x7f079006c760 msgr2=0x7f07900fc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:01.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.846+0000 7f07984b7700 1 -- 172.21.15.113:0/1029803990 shutdown_connections 2024-04-12T05:19:01.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.846+0000 7f07984b7700 1 -- 172.21.15.113:0/1029803990 wait complete. 2024-04-12T05:19:01.858 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:19:01.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.948+0000 7ff676382700 1 -- 172.21.15.113:0/1569065144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fe250 msgr2=0x7ff6700f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:01.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.948+0000 7ff676382700 1 --2- 172.21.15.113:0/1569065144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fe250 0x7ff6700f7a30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7ff6640047f0 tx=0x7ff664009cb0).stop 2024-04-12T05:19:01.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.948+0000 7ff676382700 1 -- 172.21.15.113:0/1569065144 shutdown_connections 2024-04-12T05:19:01.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.948+0000 7ff676382700 1 --2- 172.21.15.113:0/1569065144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fe250 0x7ff6700f7a30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:01.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.948+0000 7ff676382700 1 --2- 172.21.15.113:0/1569065144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fd8f0 0x7ff6700fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:01.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.948+0000 7ff676382700 1 -- 172.21.15.113:0/1569065144 >> 172.21.15.113:0/1569065144 conn(0x7ff6700f34b0 msgr2=0x7ff6700f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:01.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.948+0000 7ff676382700 1 -- 172.21.15.113:0/1569065144 shutdown_connections 2024-04-12T05:19:01.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.948+0000 7ff676382700 1 -- 172.21.15.113:0/1569065144 wait complete. 2024-04-12T05:19:01.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.949+0000 7ff676382700 1 Processor -- start 2024-04-12T05:19:01.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.949+0000 7ff676382700 1 -- start start 2024-04-12T05:19:01.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.949+0000 7ff676382700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fd8f0 0x7ff67010e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.949+0000 7ff676382700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fe250 0x7ff67010e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.949+0000 7ff676382700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67010eb70 con 0x7ff6700fd8f0 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.949+0000 7ff676382700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67010ecb0 con 0x7ff6700fe250 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff66ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fd8f0 0x7ff67010e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff66ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fd8f0 0x7ff67010e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36032/0 (socket says 172.21.15.113:36032) 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff66ffff700 1 -- 172.21.15.113:0/308130137 learned_addr learned my addr 172.21.15.113:0/308130137 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff66f7fe700 1 --2- 172.21.15.113:0/308130137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fe250 0x7ff67010e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff66ffff700 1 -- 172.21.15.113:0/308130137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fe250 msgr2=0x7ff67010e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff66ffff700 1 --2- 172.21.15.113:0/308130137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fe250 0x7ff67010e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff66ffff700 1 -- 172.21.15.113:0/308130137 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664005040 con 0x7ff6700fd8f0 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff66ffff700 1 --2- 172.21.15.113:0/308130137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fd8f0 0x7ff67010e080 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7ff6600041d0 tx=0x7ff660009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff66d7fa700 1 -- 172.21.15.113:0/308130137 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff660007b40 con 0x7ff6700fd8f0 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff66d7fa700 1 -- 172.21.15.113:0/308130137 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff66000f040 con 0x7ff6700fd8f0 2024-04-12T05:19:01.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff676382700 1 -- 172.21.15.113:0/308130137 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff670108340 con 0x7ff6700fd8f0 2024-04-12T05:19:01.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff66d7fa700 1 -- 172.21.15.113:0/308130137 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6600147b0 con 0x7ff6700fd8f0 2024-04-12T05:19:01.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.950+0000 7ff676382700 1 -- 172.21.15.113:0/308130137 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff670108880 con 0x7ff6700fd8f0 2024-04-12T05:19:01.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.951+0000 7ff676382700 1 -- 172.21.15.113:0/308130137 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67005f590 con 0x7ff6700fd8f0 2024-04-12T05:19:01.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.952+0000 7ff66d7fa700 1 -- 172.21.15.113:0/308130137 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff660009110 con 0x7ff6700fd8f0 2024-04-12T05:19:01.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.952+0000 7ff66d7fa700 1 --2- 172.21.15.113:0/308130137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 0x7ff65806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:01.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.952+0000 7ff66d7fa700 1 -- 172.21.15.113:0/308130137 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff66008be50 con 0x7ff6700fd8f0 2024-04-12T05:19:01.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.952+0000 7ff66f7fe700 1 --2- 172.21.15.113:0/308130137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 0x7ff65806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:01.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.952+0000 7ff66f7fe700 1 --2- 172.21.15.113:0/308130137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 0x7ff65806ef60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7ff6701099c0 tx=0x7ff66400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:01.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:01.956+0000 7ff66d7fa700 1 -- 172.21.15.113:0/308130137 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff660059900 con 0x7ff6700fd8f0 2024-04-12T05:19:02.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.174+0000 7ff676382700 1 -- 172.21.15.113:0/308130137 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff670108b00 con 0x7ff65806caa0 2024-04-12T05:19:02.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.175+0000 7ff66d7fa700 1 -- 172.21.15.113:0/308130137 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff670108b00 con 0x7ff65806caa0 2024-04-12T05:19:02.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.177+0000 7ff676382700 1 -- 172.21.15.113:0/308130137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 msgr2=0x7ff65806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:02.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.177+0000 7ff676382700 1 --2- 172.21.15.113:0/308130137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 0x7ff65806ef60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7ff6701099c0 tx=0x7ff66400b040).stop 2024-04-12T05:19:02.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.177+0000 7ff676382700 1 -- 172.21.15.113:0/308130137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fd8f0 msgr2=0x7ff67010e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:02.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.177+0000 7ff676382700 1 --2- 172.21.15.113:0/308130137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fd8f0 0x7ff67010e080 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7ff6600041d0 tx=0x7ff660009f60).stop 2024-04-12T05:19:02.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.177+0000 7ff676382700 1 -- 172.21.15.113:0/308130137 shutdown_connections 2024-04-12T05:19:02.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.177+0000 7ff676382700 1 --2- 172.21.15.113:0/308130137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 0x7ff65806ef60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.177+0000 7ff676382700 1 --2- 172.21.15.113:0/308130137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fd8f0 0x7ff67010e080 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.177+0000 7ff676382700 1 --2- 172.21.15.113:0/308130137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fe250 0x7ff67010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.177+0000 7ff676382700 1 -- 172.21.15.113:0/308130137 >> 172.21.15.113:0/308130137 conn(0x7ff6700f34b0 msgr2=0x7ff6700fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:02.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.177+0000 7ff676382700 1 -- 172.21.15.113:0/308130137 shutdown_connections 2024-04-12T05:19:02.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.177+0000 7ff676382700 1 -- 172.21.15.113:0/308130137 wait complete. 2024-04-12T05:19:02.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.282+0000 7f80fe118700 1 -- 172.21.15.113:0/4063545851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80f80f9200 msgr2=0x7f80f8101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:02.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.282+0000 7f80fe118700 1 --2- 172.21.15.113:0/4063545851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80f80f9200 0x7f80f8101910 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f80ec0047f0 tx=0x7f80ec009cb0).stop 2024-04-12T05:19:02.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.282+0000 7f80fe118700 1 -- 172.21.15.113:0/4063545851 shutdown_connections 2024-04-12T05:19:02.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.282+0000 7f80fe118700 1 --2- 172.21.15.113:0/4063545851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80f80f9200 0x7f80f8101910 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.282+0000 7f80fe118700 1 --2- 172.21.15.113:0/4063545851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80f80f88a0 0x7f80f80f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:19:02.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.282+0000 7f80fe118700 1 -- 172.21.15.113:0/4063545851 >> 172.21.15.113:0/4063545851 conn(0x7f80f8070590 msgr2=0x7f80f80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:02.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.282+0000 7f80fe118700 1 -- 172.21.15.113:0/4063545851 shutdown_connections 2024-04-12T05:19:02.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.282+0000 7f80fe118700 1 -- 172.21.15.113:0/4063545851 wait complete. 2024-04-12T05:19:02.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.283+0000 7f80fe118700 1 Processor -- start 2024-04-12T05:19:02.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.288+0000 7f80fe118700 1 -- start start 2024-04-12T05:19:02.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.288+0000 7f80fe118700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80f80f88a0 0x7f80f8114790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:02.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.288+0000 7f80fe118700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80f80f9200 0x7f80f8114c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:02.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.288+0000 7f80fe118700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f8115310 con 0x7f80f80f9200 2024-04-12T05:19:02.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.288+0000 7f80fe118700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f810e980 con 0x7f80f80f88a0 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.288+0000 7f80f6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80f80f9200 0x7f80f8114c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.288+0000 7f80f6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80f80f9200 0x7f80f8114c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36042/0 (socket says 172.21.15.113:36042) 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.288+0000 7f80f6ffd700 1 -- 172.21.15.113:0/1428217622 learned_addr learned my addr 172.21.15.113:0/1428217622 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.289+0000 7f80f77fe700 1 --2- 172.21.15.113:0/1428217622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80f80f88a0 0x7f80f8114790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.289+0000 7f80f6ffd700 1 -- 172.21.15.113:0/1428217622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80f80f88a0 msgr2=0x7f80f8114790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.289+0000 7f80f6ffd700 1 --2- 172.21.15.113:0/1428217622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80f80f88a0 0x7f80f8114790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.289+0000 7f80f6ffd700 1 -- 172.21.15.113:0/1428217622 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec005040 con 0x7f80f80f9200 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.289+0000 7f80f6ffd700 1 --2- 172.21.15.113:0/1428217622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80f80f9200 0x7f80f8114c80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f80ec013640 tx=0x7f80ec009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.289+0000 7f80f4ff9700 1 -- 172.21.15.113:0/1428217622 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80ec00e030 con 0x7f80f80f9200 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.289+0000 7f80fe118700 1 -- 172.21.15.113:0/1428217622 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f810ec00 con 0x7f80f80f9200 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.289+0000 7f80f4ff9700 1 -- 172.21.15.113:0/1428217622 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80ec007e90 con 0x7f80f80f9200 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.289+0000 7f80f4ff9700 1 -- 172.21.15.113:0/1428217622 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80ec018070 con 0x7f80f80f9200 2024-04-12T05:19:02.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.289+0000 7f80fe118700 1 -- 172.21.15.113:0/1428217622 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f810f0e0 con 0x7f80f80f9200 2024-04-12T05:19:02.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.290+0000 7f80fe118700 1 -- 172.21.15.113:0/1428217622 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f805f590 con 0x7f80f80f9200 2024-04-12T05:19:02.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.291+0000 7f80f4ff9700 1 -- 172.21.15.113:0/1428217622 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f80ec01d420 con 0x7f80f80f9200 2024-04-12T05:19:02.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.291+0000 7f80f4ff9700 1 --2- 172.21.15.113:0/1428217622 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80e006cab0 0x7f80e006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:02.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.291+0000 7f80f4ff9700 1 -- 172.21.15.113:0/1428217622 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f80ec01de40 con 0x7f80f80f9200 2024-04-12T05:19:02.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.292+0000 7f80f77fe700 1 --2- 172.21.15.113:0/1428217622 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80e006cab0 0x7f80e006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:02.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.292+0000 7f80f77fe700 1 --2- 172.21.15.113:0/1428217622 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80e006cab0 0x7f80e006ef70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f80e800b910 tx=0x7f80e8008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:02.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.295+0000 7f80f4ff9700 1 -- 172.21.15.113:0/1428217622 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f80ec05b200 con 0x7f80f80f9200 2024-04-12T05:19:02.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.503+0000 7f80fe118700 1 -- 172.21.15.113:0/1428217622 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f80f8104710 con 0x7f80e006cab0 2024-04-12T05:19:02.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.513+0000 7f80f4ff9700 1 -- 172.21.15.113:0/1428217622 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f80f8104710 con 0x7f80e006cab0 2024-04-12T05:19:02.515 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:19:02.515 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (6m) 5s ago 18m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:19:02.515 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (4m) 5s ago 18m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:19:02.515 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (4m) 24s ago 17m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:19:02.515 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (6m) 5s ago 17m 91.0M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:19:02.515 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (10s) 5s ago 12m 13.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:19:02.516 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (12m) 5s ago 12m 18.9M - 16.2.15-1-g8963e9de 727f386679cb 6cf529afbd55 2024-04-12T05:19:02.516 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (12m) 24s ago 12m 108M - 16.2.15-1-g8963e9de 727f386679cb 2dcb9f34b6c3 2024-04-12T05:19:02.516 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (12m) 24s ago 12m 16.3M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:19:02.516 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (8m) 5s ago 19m 505M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:19:02.516 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (7m) 24s ago 17m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:19:02.516 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (5m) 5s ago 19m 57.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:19:02.516 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (4m) 24s ago 17m 38.9M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:19:02.516 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (7m) 5s ago 17m 16.4M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:19:02.516 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (7m) 24s ago 17m 17.8M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:19:02.516 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (3m) 5s ago 16m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:19:02.517 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (3m) 5s ago 16m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:19:02.517 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (2m) 5s ago 15m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:19:02.517 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (2m) 24s ago 15m 155M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:19:02.517 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (92s) 24s ago 14m 100M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:19:02.517 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (28s) 24s ago 14m 13.7M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:19:02.517 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (7m) 24s ago 17m 60.2M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:19:02.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.515+0000 7f80fe118700 1 -- 172.21.15.113:0/1428217622 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80e006cab0 msgr2=0x7f80e006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:02.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.515+0000 7f80fe118700 1 --2- 172.21.15.113:0/1428217622 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80e006cab0 0x7f80e006ef70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f80e800b910 tx=0x7f80e8008040).stop 2024-04-12T05:19:02.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.515+0000 7f80fe118700 1 -- 172.21.15.113:0/1428217622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80f80f9200 msgr2=0x7f80f8114c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:02.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.515+0000 7f80fe118700 1 --2- 172.21.15.113:0/1428217622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80f80f9200 0x7f80f8114c80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f80ec013640 tx=0x7f80ec009cb0).stop 2024-04-12T05:19:02.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.515+0000 7f80fe118700 1 -- 172.21.15.113:0/1428217622 shutdown_connections 2024-04-12T05:19:02.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.515+0000 7f80fe118700 1 --2- 172.21.15.113:0/1428217622 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80e006cab0 0x7f80e006ef70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.515+0000 7f80fe118700 1 --2- 172.21.15.113:0/1428217622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80f80f88a0 0x7f80f8114790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.515+0000 7f80fe118700 1 --2- 172.21.15.113:0/1428217622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80f80f9200 0x7f80f8114c80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.515+0000 7f80fe118700 1 -- 172.21.15.113:0/1428217622 >> 172.21.15.113:0/1428217622 conn(0x7f80f8070590 msgr2=0x7f80f80f6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:02.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.515+0000 7f80fe118700 1 -- 172.21.15.113:0/1428217622 shutdown_connections 2024-04-12T05:19:02.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.515+0000 7f80fe118700 1 -- 172.21.15.113:0/1428217622 wait complete. 2024-04-12T05:19:02.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.613+0000 7fa2b2d45700 1 -- 172.21.15.113:0/2102414287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2ac0fd8d0 msgr2=0x7fa2ac0fdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:02.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.613+0000 7fa2b2d45700 1 --2- 172.21.15.113:0/2102414287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2ac0fd8d0 0x7fa2ac0fdcb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa2a00047f0 tx=0x7fa2a0009cb0).stop 2024-04-12T05:19:02.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.613+0000 7fa2b2d45700 1 -- 172.21.15.113:0/2102414287 shutdown_connections 2024-04-12T05:19:02.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.613+0000 7fa2b2d45700 1 --2- 172.21.15.113:0/2102414287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ac0fe230 0x7fa2ac0f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.613+0000 7fa2b2d45700 1 --2- 172.21.15.113:0/2102414287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2ac0fd8d0 0x7fa2ac0fdcb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.613+0000 7fa2b2d45700 1 -- 172.21.15.113:0/2102414287 >> 172.21.15.113:0/2102414287 conn(0x7fa2ac06ce40 msgr2=0x7fa2ac06d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:02.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.613+0000 7fa2b2d45700 1 -- 172.21.15.113:0/2102414287 shutdown_connections 2024-04-12T05:19:02.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.613+0000 7fa2b2d45700 1 -- 172.21.15.113:0/2102414287 wait complete. 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.614+0000 7fa2b2d45700 1 Processor -- start 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.614+0000 7fa2b2d45700 1 -- start start 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.614+0000 7fa2b2d45700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ac0fd8d0 0x7fa2ac10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.614+0000 7fa2b2d45700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2ac0fe230 0x7fa2ac10e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.614+0000 7fa2b2d45700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac10eb20 con 0x7fa2ac0fe230 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.614+0000 7fa2b2d45700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac10ec60 con 0x7fa2ac0fd8d0 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.614+0000 7fa2abfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2ac0fe230 0x7fa2ac10e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.614+0000 7fa2abfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2ac0fe230 0x7fa2ac10e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36058/0 (socket says 172.21.15.113:36058) 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.614+0000 7fa2abfff700 1 -- 172.21.15.113:0/3976054845 learned_addr learned my addr 172.21.15.113:0/3976054845 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.614+0000 7fa2b0ae1700 1 --2- 172.21.15.113:0/3976054845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ac0fd8d0 0x7fa2ac10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.615+0000 7fa2abfff700 1 -- 172.21.15.113:0/3976054845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ac0fd8d0 msgr2=0x7fa2ac10e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:02.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.615+0000 7fa2abfff700 1 --2- 172.21.15.113:0/3976054845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ac0fd8d0 0x7fa2ac10e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.615+0000 7fa2abfff700 1 -- 172.21.15.113:0/3976054845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a0005040 con 0x7fa2ac0fe230 2024-04-12T05:19:02.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.615+0000 7fa2abfff700 1 --2- 172.21.15.113:0/3976054845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2ac0fe230 0x7fa2ac10e520 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa298002770 tx=0x7fa298002a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:02.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.615+0000 7fa2a9ffb700 1 -- 172.21.15.113:0/3976054845 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa298007b40 con 0x7fa2ac0fe230 2024-04-12T05:19:02.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.615+0000 7fa2a9ffb700 1 -- 172.21.15.113:0/3976054845 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa29800f040 con 0x7fa2ac0fe230 2024-04-12T05:19:02.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.615+0000 7fa2b2d45700 1 -- 172.21.15.113:0/3976054845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac1082f0 con 0x7fa2ac0fe230 2024-04-12T05:19:02.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.615+0000 7fa2a9ffb700 1 -- 172.21.15.113:0/3976054845 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa298014820 con 0x7fa2ac0fe230 2024-04-12T05:19:02.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.615+0000 7fa2b2d45700 1 -- 172.21.15.113:0/3976054845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ac108830 con 0x7fa2ac0fe230 2024-04-12T05:19:02.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.616+0000 7fa2b2d45700 1 -- 172.21.15.113:0/3976054845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ac05f090 con 0x7fa2ac0fe230 2024-04-12T05:19:02.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.616+0000 7fa2a9ffb700 1 -- 172.21.15.113:0/3976054845 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa298007ca0 con 0x7fa2ac0fe230 2024-04-12T05:19:02.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.617+0000 7fa2a9ffb700 1 --2- 172.21.15.113:0/3976054845 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa29c06caa0 0x7fa29c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:02.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.617+0000 7fa2a9ffb700 1 -- 172.21.15.113:0/3976054845 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa29808b940 con 0x7fa2ac0fe230 2024-04-12T05:19:02.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.617+0000 7fa2b0ae1700 1 --2- 172.21.15.113:0/3976054845 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa29c06caa0 0x7fa29c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:02.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.617+0000 7fa2b0ae1700 1 --2- 172.21.15.113:0/3976054845 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa29c06caa0 0x7fa29c06ef60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa2a0004f10 tx=0x7fa2a000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:02.623 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.621+0000 7fa2a9ffb700 1 -- 172.21.15.113:0/3976054845 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa2980593f0 con 0x7fa2ac0fe230 2024-04-12T05:19:02.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.908+0000 7fa2b2d45700 1 -- 172.21.15.113:0/3976054845 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa2ac0fab20 con 0x7fa2ac0fe230 2024-04-12T05:19:02.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.909+0000 7fa2a9ffb700 1 -- 172.21.15.113:0/3976054845 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fa298058f80 con 0x7fa2ac0fe230 2024-04-12T05:19:02.911 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:19:02.911 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:19:02.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:19:02.911 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:19:02.911 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:19:02.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:19:02.911 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:19:02.911 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:19:02.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:19:02.912 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:19:02.912 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:19:02.912 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 3, 2024-04-12T05:19:02.912 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 1 2024-04-12T05:19:02.912 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:19:02.912 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:19:02.912 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 3, 2024-04-12T05:19:02.912 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 11 2024-04-12T05:19:02.912 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:19:02.912 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:19:02.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.911+0000 7fa2b2d45700 1 -- 172.21.15.113:0/3976054845 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa29c06caa0 msgr2=0x7fa29c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:02.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.911+0000 7fa2b2d45700 1 --2- 172.21.15.113:0/3976054845 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa29c06caa0 0x7fa29c06ef60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa2a0004f10 tx=0x7fa2a000b040).stop 2024-04-12T05:19:02.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.911+0000 7fa2b2d45700 1 -- 172.21.15.113:0/3976054845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2ac0fe230 msgr2=0x7fa2ac10e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:02.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.911+0000 7fa2b2d45700 1 --2- 172.21.15.113:0/3976054845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2ac0fe230 0x7fa2ac10e520 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa298002770 tx=0x7fa298002a80).stop 2024-04-12T05:19:02.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.911+0000 7fa2b2d45700 1 -- 172.21.15.113:0/3976054845 shutdown_connections 2024-04-12T05:19:02.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.911+0000 7fa2b2d45700 1 --2- 172.21.15.113:0/3976054845 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa29c06caa0 0x7fa29c06ef60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.911+0000 7fa2b2d45700 1 --2- 172.21.15.113:0/3976054845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2ac0fd8d0 0x7fa2ac10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.911+0000 7fa2b2d45700 1 --2- 172.21.15.113:0/3976054845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa2ac0fe230 0x7fa2ac10e520 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:02.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.911+0000 7fa2b2d45700 1 -- 172.21.15.113:0/3976054845 >> 172.21.15.113:0/3976054845 conn(0x7fa2ac06ce40 msgr2=0x7fa2ac100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:02.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.911+0000 7fa2b2d45700 1 -- 172.21.15.113:0/3976054845 shutdown_connections 2024-04-12T05:19:02.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:02.911+0000 7fa2b2d45700 1 -- 172.21.15.113:0/3976054845 wait complete. 2024-04-12T05:19:03.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.015+0000 7f5bdeea6700 1 -- 172.21.15.113:0/2828886404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd80fc890 msgr2=0x7f5bd80f60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.015+0000 7f5bdeea6700 1 --2- 172.21.15.113:0/2828886404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd80fc890 0x7f5bd80f60d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f5bc40047f0 tx=0x7f5bc4009cb0).stop 2024-04-12T05:19:03.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.015+0000 7f5bdeea6700 1 -- 172.21.15.113:0/2828886404 shutdown_connections 2024-04-12T05:19:03.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.015+0000 7f5bdeea6700 1 --2- 172.21.15.113:0/2828886404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd80fc890 0x7f5bd80f60d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.015+0000 7f5bdeea6700 1 --2- 172.21.15.113:0/2828886404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bd80fbf30 0x7f5bd80fc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.015+0000 7f5bdeea6700 1 -- 172.21.15.113:0/2828886404 >> 172.21.15.113:0/2828886404 conn(0x7f5bd806ce40 msgr2=0x7f5bd806d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:03.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.016+0000 7f5bdeea6700 1 -- 172.21.15.113:0/2828886404 shutdown_connections 2024-04-12T05:19:03.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.016+0000 7f5bdeea6700 1 -- 172.21.15.113:0/2828886404 wait complete. 2024-04-12T05:19:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bdeea6700 1 Processor -- start 2024-04-12T05:19:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bdeea6700 1 -- start start 2024-04-12T05:19:03.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bdeea6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bd80fbf30 0x7f5bd810e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bdeea6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd80fc890 0x7f5bd810e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bdeea6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd810ec30 con 0x7f5bd80fc890 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bdeea6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd810ed70 con 0x7f5bd80fbf30 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bd7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd80fc890 0x7f5bd810e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bd7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd80fc890 0x7f5bd810e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36078/0 (socket says 172.21.15.113:36078) 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bd7fff700 1 -- 172.21.15.113:0/459426218 learned_addr learned my addr 172.21.15.113:0/459426218 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bdcc42700 1 --2- 172.21.15.113:0/459426218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bd80fbf30 0x7f5bd810e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bd7fff700 1 -- 172.21.15.113:0/459426218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bd80fbf30 msgr2=0x7f5bd810e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bd7fff700 1 --2- 172.21.15.113:0/459426218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bd80fbf30 0x7f5bd810e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.017+0000 7f5bd7fff700 1 -- 172.21.15.113:0/459426218 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc4005040 con 0x7f5bd80fc890 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.018+0000 7f5bd7fff700 1 --2- 172.21.15.113:0/459426218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd80fc890 0x7f5bd810e630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f5bc4013640 tx=0x7f5bc4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.018+0000 7f5bd5ffb700 1 -- 172.21.15.113:0/459426218 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bc400e030 con 0x7f5bd80fc890 2024-04-12T05:19:03.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.018+0000 7f5bd5ffb700 1 -- 172.21.15.113:0/459426218 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bc4007e90 con 0x7f5bd80fc890 2024-04-12T05:19:03.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.018+0000 7f5bd5ffb700 1 -- 172.21.15.113:0/459426218 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bc4018070 con 0x7f5bd80fc890 2024-04-12T05:19:03.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.018+0000 7f5bdeea6700 1 -- 172.21.15.113:0/459426218 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd81083a0 con 0x7f5bd80fc890 2024-04-12T05:19:03.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.018+0000 7f5bdeea6700 1 -- 172.21.15.113:0/459426218 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd8108830 con 0x7f5bd80fc890 2024-04-12T05:19:03.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.019+0000 7f5bdeea6700 1 -- 172.21.15.113:0/459426218 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd8060670 con 0x7f5bd80fc890 2024-04-12T05:19:03.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.020+0000 7f5bd5ffb700 1 -- 172.21.15.113:0/459426218 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5bc401d420 con 0x7f5bd80fc890 2024-04-12T05:19:03.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.020+0000 7f5bd5ffb700 1 --2- 172.21.15.113:0/459426218 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5bc806cab0 0x7f5bc806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:03.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.020+0000 7f5bd5ffb700 1 -- 172.21.15.113:0/459426218 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5bc408d310 con 0x7f5bd80fc890 2024-04-12T05:19:03.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.020+0000 7f5bdcc42700 1 --2- 172.21.15.113:0/459426218 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5bc806cab0 0x7f5bc806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:03.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.021+0000 7f5bdcc42700 1 --2- 172.21.15.113:0/459426218 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5bc806cab0 0x7f5bc806ef70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f5bcc0041d0 tx=0x7f5bcc009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:03.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.025+0000 7f5bd5ffb700 1 -- 172.21.15.113:0/459426218 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5bc405ad00 con 0x7f5bd80fc890 2024-04-12T05:19:03.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.260+0000 7f5bdeea6700 1 -- 172.21.15.113:0/459426218 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5bd8047540 con 0x7f5bd80fc890 2024-04-12T05:19:03.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.261+0000 7f5bd5ffb700 1 -- 172.21.15.113:0/459426218 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f5bc405a890 con 0x7f5bd80fc890 2024-04-12T05:19:03.263 INFO:teuthology.orchestra.run.smithi113.stdout:e22 2024-04-12T05:19:03.263 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:19:03.263 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:19:03.263 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:19:03.263 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:19:03.263 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:19:03.263 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:19:03.263 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:19:03.264 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:19:03.265 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:19:03.266 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:14438} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/2511637182,v1:172.21.15.113:6829/2511637182] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:19:03.266 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:19:03.266 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:19:03.266 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44303} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/2802712879,v1:172.21.15.192:6825/2802712879] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:19:03.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.263+0000 7f5bdeea6700 1 -- 172.21.15.113:0/459426218 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5bc806cab0 msgr2=0x7f5bc806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.263+0000 7f5bdeea6700 1 --2- 172.21.15.113:0/459426218 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5bc806cab0 0x7f5bc806ef70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f5bcc0041d0 tx=0x7f5bcc009040).stop 2024-04-12T05:19:03.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.263+0000 7f5bdeea6700 1 -- 172.21.15.113:0/459426218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd80fc890 msgr2=0x7f5bd810e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.263+0000 7f5bdeea6700 1 --2- 172.21.15.113:0/459426218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd80fc890 0x7f5bd810e630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f5bc4013640 tx=0x7f5bc4009cb0).stop 2024-04-12T05:19:03.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.263+0000 7f5bdeea6700 1 -- 172.21.15.113:0/459426218 shutdown_connections 2024-04-12T05:19:03.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.263+0000 7f5bdeea6700 1 --2- 172.21.15.113:0/459426218 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5bc806cab0 0x7f5bc806ef70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.263+0000 7f5bdeea6700 1 --2- 172.21.15.113:0/459426218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bd80fbf30 0x7f5bd810e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.263+0000 7f5bdeea6700 1 --2- 172.21.15.113:0/459426218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bd80fc890 0x7f5bd810e630 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.263+0000 7f5bdeea6700 1 -- 172.21.15.113:0/459426218 >> 172.21.15.113:0/459426218 conn(0x7f5bd806ce40 msgr2=0x7f5bd80ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:03.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.263+0000 7f5bdeea6700 1 -- 172.21.15.113:0/459426218 shutdown_connections 2024-04-12T05:19:03.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.263+0000 7f5bdeea6700 1 -- 172.21.15.113:0/459426218 wait complete. 2024-04-12T05:19:03.267 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 22 2024-04-12T05:19:03.308 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: pgmap v204: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:03.308 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:03.308 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:03.308 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:03.308 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:19:03.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:19:03.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:03.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:19:03.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:03.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:03.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:03.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:03.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:02 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3976054845' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:03.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.364+0000 7f157c6f3700 1 -- 172.21.15.113:0/1352478368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15740fda00 msgr2=0x7f15740fdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.364+0000 7f157c6f3700 1 --2- 172.21.15.113:0/1352478368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15740fda00 0x7f15740fdde0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f15700047f0 tx=0x7f1570009cb0).stop 2024-04-12T05:19:03.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.364+0000 7f157c6f3700 1 -- 172.21.15.113:0/1352478368 shutdown_connections 2024-04-12T05:19:03.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.364+0000 7f157c6f3700 1 --2- 172.21.15.113:0/1352478368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15740fe360 0x7f15740f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:19:03.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.364+0000 7f157c6f3700 1 --2- 172.21.15.113:0/1352478368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15740fda00 0x7f15740fdde0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.364+0000 7f157c6f3700 1 -- 172.21.15.113:0/1352478368 >> 172.21.15.113:0/1352478368 conn(0x7f157406cea0 msgr2=0x7f157406d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:03.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.364+0000 7f157c6f3700 1 -- 172.21.15.113:0/1352478368 shutdown_connections 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.364+0000 7f157c6f3700 1 -- 172.21.15.113:0/1352478368 wait complete. 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.365+0000 7f157c6f3700 1 Processor -- start 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.365+0000 7f157c6f3700 1 -- start start 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.365+0000 7f157c6f3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15740fda00 0x7f157410e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.365+0000 7f157c6f3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15740fe360 0x7f157410e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.365+0000 7f157c6f3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157410ebc0 con 0x7f15740fe360 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.365+0000 7f157c6f3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15741081e0 con 0x7f15740fda00 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.365+0000 7f1579c8e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15740fe360 0x7f157410e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.365+0000 7f1579c8e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15740fe360 0x7f157410e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36096/0 (socket says 172.21.15.113:36096) 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.365+0000 7f1579c8e700 1 -- 172.21.15.113:0/3180038425 learned_addr learned my addr 172.21.15.113:0/3180038425 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.365+0000 7f157a48f700 1 --2- 172.21.15.113:0/3180038425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15740fda00 0x7f157410e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.366+0000 7f1579c8e700 1 -- 172.21.15.113:0/3180038425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15740fda00 msgr2=0x7f157410e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.366+0000 7f1579c8e700 1 --2- 172.21.15.113:0/3180038425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15740fda00 0x7f157410e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.366+0000 7f1579c8e700 1 -- 172.21.15.113:0/3180038425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1570005040 con 0x7f15740fe360 2024-04-12T05:19:03.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.366+0000 7f1579c8e700 1 --2- 172.21.15.113:0/3180038425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15740fe360 0x7f157410e530 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f156400b4b0 tx=0x7f15640090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:03.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.366+0000 7f156b7fe700 1 -- 172.21.15.113:0/3180038425 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1564002500 con 0x7f15740fe360 2024-04-12T05:19:03.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.366+0000 7f156b7fe700 1 -- 172.21.15.113:0/3180038425 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1564002b10 con 0x7f15740fe360 2024-04-12T05:19:03.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.366+0000 7f157c6f3700 1 -- 172.21.15.113:0/3180038425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1574108490 con 0x7f15740fe360 2024-04-12T05:19:03.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.366+0000 7f156b7fe700 1 -- 172.21.15.113:0/3180038425 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1564012cb0 con 0x7f15740fe360 2024-04-12T05:19:03.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.366+0000 7f157c6f3700 1 -- 172.21.15.113:0/3180038425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15741089d0 con 0x7f15740fe360 2024-04-12T05:19:03.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.367+0000 7f157c6f3700 1 -- 172.21.15.113:0/3180038425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1574047540 con 0x7f15740fe360 2024-04-12T05:19:03.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.367+0000 7f156b7fe700 1 -- 172.21.15.113:0/3180038425 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1564002660 con 0x7f15740fe360 2024-04-12T05:19:03.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.368+0000 7f156b7fe700 1 --2- 172.21.15.113:0/3180038425 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f156006caa0 0x7f156006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:03.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.368+0000 7f156b7fe700 1 -- 172.21.15.113:0/3180038425 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f156408be90 con 0x7f15740fe360 2024-04-12T05:19:03.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.368+0000 7f157a48f700 1 --2- 172.21.15.113:0/3180038425 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f156006caa0 0x7f156006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:03.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.368+0000 7f157a48f700 1 --2- 172.21.15.113:0/3180038425 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f156006caa0 0x7f156006ef60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f1570007cf0 tx=0x7f157000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:03.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.372+0000 7f156b7fe700 1 -- 172.21.15.113:0/3180038425 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1564059940 con 0x7f15740fe360 2024-04-12T05:19:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: pgmap v204: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:02 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3976054845' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:03.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.582+0000 7f157c6f3700 1 -- 172.21.15.113:0/3180038425 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1574108c50 con 0x7f156006caa0 2024-04-12T05:19:03.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.583+0000 7f156b7fe700 1 -- 172.21.15.113:0/3180038425 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1574108c50 con 0x7f156006caa0 2024-04-12T05:19:03.585 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:19:03.585 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:19:03.585 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:19:03.585 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:19:03.585 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:19:03.585 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:19:03.585 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:19:03.585 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:19:03.586 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:19:03.586 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:19:03.586 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "13/21 daemons upgraded", 2024-04-12T05:19:03.586 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:19:03.586 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:19:03.586 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:19:03.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.585+0000 7f157c6f3700 1 -- 172.21.15.113:0/3180038425 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f156006caa0 msgr2=0x7f156006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.585+0000 7f157c6f3700 1 --2- 172.21.15.113:0/3180038425 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f156006caa0 0x7f156006ef60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f1570007cf0 tx=0x7f157000b040).stop 2024-04-12T05:19:03.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.585+0000 7f157c6f3700 1 -- 172.21.15.113:0/3180038425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15740fe360 msgr2=0x7f157410e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.585+0000 7f157c6f3700 1 --2- 172.21.15.113:0/3180038425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15740fe360 0x7f157410e530 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f156400b4b0 tx=0x7f15640090f0).stop 2024-04-12T05:19:03.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.585+0000 7f157c6f3700 1 -- 172.21.15.113:0/3180038425 shutdown_connections 2024-04-12T05:19:03.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.585+0000 7f157c6f3700 1 --2- 172.21.15.113:0/3180038425 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f156006caa0 0x7f156006ef60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.585+0000 7f157c6f3700 1 --2- 172.21.15.113:0/3180038425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15740fda00 0x7f157410e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.585+0000 7f157c6f3700 1 --2- 172.21.15.113:0/3180038425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15740fe360 0x7f157410e530 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.585+0000 7f157c6f3700 1 -- 172.21.15.113:0/3180038425 >> 172.21.15.113:0/3180038425 conn(0x7f157406cea0 msgr2=0x7f15740fbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:03.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.585+0000 7f157c6f3700 1 -- 172.21.15.113:0/3180038425 shutdown_connections 2024-04-12T05:19:03.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.585+0000 7f157c6f3700 1 -- 172.21.15.113:0/3180038425 wait complete. 2024-04-12T05:19:03.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.686+0000 7fc347c90700 1 -- 172.21.15.113:0/3456586853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3400fe210 msgr2=0x7fc3400f5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.686+0000 7fc347c90700 1 --2- 172.21.15.113:0/3456586853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3400fe210 0x7fc3400f5f80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fc33c0047f0 tx=0x7fc33c009cb0).stop 2024-04-12T05:19:03.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.686+0000 7fc347c90700 1 -- 172.21.15.113:0/3456586853 shutdown_connections 2024-04-12T05:19:03.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.686+0000 7fc347c90700 1 --2- 172.21.15.113:0/3456586853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3400fe210 0x7fc3400f5f80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.686+0000 7fc347c90700 1 --2- 172.21.15.113:0/3456586853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3400fd8b0 0x7fc3400fdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.686+0000 7fc347c90700 1 -- 172.21.15.113:0/3456586853 >> 172.21.15.113:0/3456586853 conn(0x7fc34006ce40 msgr2=0x7fc34006d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:03.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.687+0000 7fc347c90700 1 -- 172.21.15.113:0/3456586853 shutdown_connections 2024-04-12T05:19:03.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.687+0000 7fc347c90700 1 -- 172.21.15.113:0/3456586853 wait complete. 2024-04-12T05:19:03.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.687+0000 7fc347c90700 1 Processor -- start 2024-04-12T05:19:03.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.687+0000 7fc347c90700 1 -- start start 2024-04-12T05:19:03.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.687+0000 7fc347c90700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3400fd8b0 0x7fc3400f7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:03.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.687+0000 7fc347c90700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3400fe210 0x7fc3400f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.687+0000 7fc347c90700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3400f7c30 con 0x7fc3400fe210 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.687+0000 7fc347c90700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3400f7d70 con 0x7fc3400fd8b0 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc34522b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3400fe210 0x7fc3400f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc34522b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3400fe210 0x7fc3400f7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36116/0 (socket says 172.21.15.113:36116) 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc34522b700 1 -- 172.21.15.113:0/1002753965 learned_addr learned my addr 172.21.15.113:0/1002753965 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc345a2c700 1 --2- 172.21.15.113:0/1002753965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3400fd8b0 0x7fc3400f7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc34522b700 1 -- 172.21.15.113:0/1002753965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3400fd8b0 msgr2=0x7fc3400f7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc34522b700 1 --2- 172.21.15.113:0/1002753965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3400fd8b0 0x7fc3400f7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc34522b700 1 -- 172.21.15.113:0/1002753965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c005040 con 0x7fc3400fe210 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc34522b700 1 --2- 172.21.15.113:0/1002753965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3400fe210 0x7fc3400f7740 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fc33c005010 tx=0x7fc33c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc336ffd700 1 -- 172.21.15.113:0/1002753965 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc33c00e030 con 0x7fc3400fe210 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc336ffd700 1 -- 172.21.15.113:0/1002753965 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc33c007e90 con 0x7fc3400fe210 2024-04-12T05:19:03.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc347c90700 1 -- 172.21.15.113:0/1002753965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3400f7fc0 con 0x7fc3400fe210 2024-04-12T05:19:03.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc336ffd700 1 -- 172.21.15.113:0/1002753965 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc33c018070 con 0x7fc3400fe210 2024-04-12T05:19:03.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.688+0000 7fc347c90700 1 -- 172.21.15.113:0/1002753965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc340114600 con 0x7fc3400fe210 2024-04-12T05:19:03.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.689+0000 7fc347c90700 1 -- 172.21.15.113:0/1002753965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc340060670 con 0x7fc3400fe210 2024-04-12T05:19:03.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.690+0000 7fc336ffd700 1 -- 172.21.15.113:0/1002753965 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc33c009590 con 0x7fc3400fe210 2024-04-12T05:19:03.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.690+0000 7fc336ffd700 1 --2- 172.21.15.113:0/1002753965 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc32c06caf0 0x7fc32c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:03.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.690+0000 7fc336ffd700 1 -- 172.21.15.113:0/1002753965 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc33c08d220 con 0x7fc3400fe210 2024-04-12T05:19:03.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.690+0000 7fc345a2c700 1 --2- 172.21.15.113:0/1002753965 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc32c06caf0 0x7fc32c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:03.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.691+0000 7fc345a2c700 1 --2- 172.21.15.113:0/1002753965 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc32c06caf0 0x7fc32c06efb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc330004dc0 tx=0x7fc33000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:03.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.698+0000 7fc336ffd700 1 -- 172.21.15.113:0/1002753965 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc33c05ac10 con 0x7fc3400fe210 2024-04-12T05:19:03.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.987+0000 7fc347c90700 1 -- 172.21.15.113:0/1002753965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc340047540 con 0x7fc3400fe210 2024-04-12T05:19:03.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.987+0000 7fc336ffd700 1 -- 172.21.15.113:0/1002753965 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc33c05a7a0 con 0x7fc3400fe210 2024-04-12T05:19:03.990 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:19:03.990 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:19:03.990 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:19:03.990 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:19:03.990 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:19:03.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.989+0000 7fc347c90700 1 -- 172.21.15.113:0/1002753965 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc32c06caf0 msgr2=0x7fc32c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.989+0000 7fc347c90700 1 --2- 172.21.15.113:0/1002753965 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc32c06caf0 0x7fc32c06efb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc330004dc0 tx=0x7fc33000b040).stop 2024-04-12T05:19:03.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.989+0000 7fc347c90700 1 -- 172.21.15.113:0/1002753965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3400fe210 msgr2=0x7fc3400f7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:03.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.989+0000 7fc347c90700 1 --2- 172.21.15.113:0/1002753965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3400fe210 0x7fc3400f7740 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fc33c005010 tx=0x7fc33c009cb0).stop 2024-04-12T05:19:03.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.989+0000 7fc347c90700 1 -- 172.21.15.113:0/1002753965 shutdown_connections 2024-04-12T05:19:03.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.989+0000 7fc347c90700 1 --2- 172.21.15.113:0/1002753965 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc32c06caf0 0x7fc32c06efb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.989+0000 7fc347c90700 1 --2- 172.21.15.113:0/1002753965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3400fd8b0 0x7fc3400f7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.989+0000 7fc347c90700 1 --2- 172.21.15.113:0/1002753965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3400fe210 0x7fc3400f7740 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:03.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.989+0000 7fc347c90700 1 -- 172.21.15.113:0/1002753965 >> 172.21.15.113:0/1002753965 conn(0x7fc34006ce40 msgr2=0x7fc340100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:03.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.989+0000 7fc347c90700 1 -- 172.21.15.113:0/1002753965 shutdown_connections 2024-04-12T05:19:03.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:03.989+0000 7fc347c90700 1 -- 172.21.15.113:0/1002753965 wait complete. 2024-04-12T05:19:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:03 smithi113 ceph-mon[103371]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:03 smithi113 ceph-mon[103371]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:03 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/459426218' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:19:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:03 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:03 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ydsbei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:19:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:03 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:19:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:03 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1002753965' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:19:04.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:04 smithi192 ceph-mon[85304]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:04.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:04 smithi192 ceph-mon[85304]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:04 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/459426218' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:19:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi113.ydsbei", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:19:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:04 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:19:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:04 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1002753965' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:19:05.313 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:05 smithi113 ceph-mon[103371]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:05.313 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:05 smithi113 ceph-mon[103371]: Upgrade: Updating mds.cephfs.smithi113.ydsbei 2024-04-12T05:19:05.313 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:05 smithi113 ceph-mon[103371]: Deploying daemon mds.cephfs.smithi113.ydsbei on smithi113 2024-04-12T05:19:05.313 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:05 smithi113 ceph-mon[103371]: pgmap v205: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:05 smithi192 ceph-mon[85304]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:05.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:05 smithi192 ceph-mon[85304]: Upgrade: Updating mds.cephfs.smithi113.ydsbei 2024-04-12T05:19:05.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:05 smithi192 ceph-mon[85304]: Deploying daemon mds.cephfs.smithi113.ydsbei on smithi113 2024-04-12T05:19:05.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:05 smithi192 ceph-mon[85304]: pgmap v205: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:06 smithi113 ceph-mon[103371]: osdmap e96: 6 total, 6 up, 6 in 2024-04-12T05:19:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:06 smithi113 ceph-mon[103371]: pgmap v207: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:07.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:06 smithi113 ceph-mon[103371]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:19:07.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:06 smithi192 ceph-mon[85304]: osdmap e96: 6 total, 6 up, 6 in 2024-04-12T05:19:07.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:06 smithi192 ceph-mon[85304]: pgmap v207: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:07.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:06 smithi192 ceph-mon[85304]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:19:08.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:19:08.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:19:08.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:19:08.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:19:09.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:08 smithi113 ceph-mon[103371]: pgmap v208: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:09.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:08 smithi192 ceph-mon[85304]: pgmap v208: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:11.141 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:10 smithi113 ceph-mon[103371]: pgmap v209: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:11.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:10 smithi192 ceph-mon[85304]: pgmap v209: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:12.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:12 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:12.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:12 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:12.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:12 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:19:12.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:12 smithi113 ceph-mon[103371]: pgmap v210: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:12.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:12 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] up:boot 2024-04-12T05:19:12.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:12 smithi113 ceph-mon[103371]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:19:12.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:12 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:19:12.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:12 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:12.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:12 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:12 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:19:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:12 smithi192 ceph-mon[85304]: pgmap v210: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:12 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] up:boot 2024-04-12T05:19:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:12 smithi192 ceph-mon[85304]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:19:12.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:12 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:19:15.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:14 smithi113 ceph-mon[103371]: pgmap v211: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:15.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:14 smithi192 ceph-mon[85304]: pgmap v211: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:16.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:16.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:16.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:16 smithi113 ceph-mon[103371]: pgmap v212: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:16.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:16.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:16 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:16.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:16.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:16.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:16 smithi192 ceph-mon[85304]: pgmap v212: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:16.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:16.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:16 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:19.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:18 smithi113 ceph-mon[103371]: pgmap v213: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:19.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:18 smithi192 ceph-mon[85304]: pgmap v213: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:20.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:20.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:20.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:19:20.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:19:20.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:20.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:19:20.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:20.912 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:20.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:20.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:20 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:20.913 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:20 smithi192 ceph-mon[85304]: pgmap v214: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:20.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:20.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:20.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:19:20.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:19:20.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:20.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:19:20.924 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:20.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:20.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:20.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:20 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:20.925 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:20 smithi113 ceph-mon[103371]: pgmap v214: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:22.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:22 smithi192 ceph-mon[85304]: Upgrade: Updating mds.cephfs.smithi192.ammjap 2024-04-12T05:19:22.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:22.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.ammjap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:19:22.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:19:22.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:22 smithi192 ceph-mon[85304]: Deploying daemon mds.cephfs.smithi192.ammjap on smithi192 2024-04-12T05:19:22.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:19:22.674 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:22 smithi192 ceph-mon[85304]: pgmap v215: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:22.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:22 smithi113 ceph-mon[103371]: Upgrade: Updating mds.cephfs.smithi192.ammjap 2024-04-12T05:19:22.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:22.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.ammjap", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:19:22.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:19:22.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:22 smithi113 ceph-mon[103371]: Deploying daemon mds.cephfs.smithi192.ammjap on smithi192 2024-04-12T05:19:22.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:19:22.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:22 smithi113 ceph-mon[103371]: pgmap v215: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:24 smithi113 ceph-mon[103371]: osdmap e97: 6 total, 6 up, 6 in 2024-04-12T05:19:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:24 smithi113 ceph-mon[103371]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:19:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:24 smithi113 ceph-mon[103371]: pgmap v217: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:24.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:24 smithi192 ceph-mon[85304]: osdmap e97: 6 total, 6 up, 6 in 2024-04-12T05:19:24.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:24 smithi192 ceph-mon[85304]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:19:24.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:24 smithi192 ceph-mon[85304]: pgmap v217: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:25.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:25 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:19:25.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:25 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:19:26.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:26 smithi113 ceph-mon[103371]: pgmap v218: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:26.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:26 smithi192 ceph-mon[85304]: pgmap v218: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:28 smithi113 ceph-mon[103371]: pgmap v219: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:28 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:28 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:28 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:19:29.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:28 smithi192 ceph-mon[85304]: pgmap v219: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:29.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:28 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:29.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:28 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:29.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:28 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:19:29.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:29 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] up:boot 2024-04-12T05:19:29.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:29 smithi192 ceph-mon[85304]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:19:29.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:29 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:19:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:29 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] up:boot 2024-04-12T05:19:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:29 smithi113 ceph-mon[103371]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:19:30.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:29 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:19:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:30 smithi113 ceph-mon[103371]: pgmap v220: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:31.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:30 smithi192 ceph-mon[85304]: pgmap v220: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:32 smithi113 ceph-mon[103371]: pgmap v221: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:32 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:33.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:32 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:33.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:32 smithi192 ceph-mon[85304]: pgmap v221: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:33.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:32 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:33.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:32 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:34.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.103+0000 7fdc6babc700 1 -- 172.21.15.113:0/1179678717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc640f8a10 msgr2=0x7fdc640f8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.103+0000 7fdc6babc700 1 --2- 172.21.15.113:0/1179678717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc640f8a10 0x7fdc640f8df0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fdc540047f0 tx=0x7fdc54009cb0).stop 2024-04-12T05:19:34.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.103+0000 7fdc6babc700 1 -- 172.21.15.113:0/1179678717 shutdown_connections 2024-04-12T05:19:34.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.103+0000 7fdc6babc700 1 --2- 172.21.15.113:0/1179678717 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc640f92e0 0x7fdc64101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.103+0000 7fdc6babc700 1 --2- 172.21.15.113:0/1179678717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc640f8a10 0x7fdc640f8df0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.103+0000 7fdc6babc700 1 -- 172.21.15.113:0/1179678717 >> 172.21.15.113:0/1179678717 conn(0x7fdc6406c760 msgr2=0x7fdc6406cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:34.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.103+0000 7fdc6babc700 1 -- 172.21.15.113:0/1179678717 shutdown_connections 2024-04-12T05:19:34.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.103+0000 7fdc6babc700 1 -- 172.21.15.113:0/1179678717 wait complete. 2024-04-12T05:19:34.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.104+0000 7fdc6babc700 1 Processor -- start 2024-04-12T05:19:34.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.104+0000 7fdc6babc700 1 -- start start 2024-04-12T05:19:34.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc6babc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc640f8a10 0x7fdc6410e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc6babc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc640f92e0 0x7fdc6410e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc6babc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6410eb60 con 0x7fdc640f8a10 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc6babc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6410eca0 con 0x7fdc640f92e0 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc69858700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc640f8a10 0x7fdc6410e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc69858700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc640f8a10 0x7fdc6410e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38032/0 (socket says 172.21.15.113:38032) 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc69858700 1 -- 172.21.15.113:0/3503148990 learned_addr learned my addr 172.21.15.113:0/3503148990 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc69057700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc640f92e0 0x7fdc6410e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc69858700 1 -- 172.21.15.113:0/3503148990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc640f92e0 msgr2=0x7fdc6410e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc69858700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc640f92e0 0x7fdc6410e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc69858700 1 -- 172.21.15.113:0/3503148990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc54005040 con 0x7fdc640f8a10 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc69057700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc640f92e0 0x7fdc6410e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc69858700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc640f8a10 0x7fdc6410e070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fdc5400b120 tx=0x7fdc54009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:34.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc5af78700 1 -- 172.21.15.113:0/3503148990 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc5400e030 con 0x7fdc640f8a10 2024-04-12T05:19:34.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc6babc700 1 -- 172.21.15.113:0/3503148990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc641082d0 con 0x7fdc640f8a10 2024-04-12T05:19:34.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc5af78700 1 -- 172.21.15.113:0/3503148990 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc54007e90 con 0x7fdc640f8a10 2024-04-12T05:19:34.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc5af78700 1 -- 172.21.15.113:0/3503148990 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc54018070 con 0x7fdc640f8a10 2024-04-12T05:19:34.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.105+0000 7fdc6babc700 1 -- 172.21.15.113:0/3503148990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc641087b0 con 0x7fdc640f8a10 2024-04-12T05:19:34.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.106+0000 7fdc6babc700 1 -- 172.21.15.113:0/3503148990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6405f550 con 0x7fdc640f8a10 2024-04-12T05:19:34.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.107+0000 7fdc5af78700 1 -- 172.21.15.113:0/3503148990 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdc5401d420 con 0x7fdc640f8a10 2024-04-12T05:19:34.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.107+0000 7fdc5af78700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdc5006caa0 0x7fdc5006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:34.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.107+0000 7fdc5af78700 1 -- 172.21.15.113:0/3503148990 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fdc5408d300 con 0x7fdc640f8a10 2024-04-12T05:19:34.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.107+0000 7fdc69057700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdc5006caa0 0x7fdc5006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:34.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.108+0000 7fdc69057700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdc5006caa0 0x7fdc5006ef60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdc60007fd0 tx=0x7fdc60009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:34.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.112+0000 7fdc5af78700 1 -- 172.21.15.113:0/3503148990 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fdc5405adb0 con 0x7fdc640f8a10 2024-04-12T05:19:34.157 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:33 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:34.157 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:33 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:34.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.318+0000 7fdc6babc700 1 -- 172.21.15.113:0/3503148990 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc640fefb0 con 0x7fdc5006caa0 2024-04-12T05:19:34.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.319+0000 7fdc5af78700 1 -- 172.21.15.113:0/3503148990 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdc640fefb0 con 0x7fdc5006caa0 2024-04-12T05:19:34.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.321+0000 7fdc6babc700 1 -- 172.21.15.113:0/3503148990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdc5006caa0 msgr2=0x7fdc5006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.321+0000 7fdc6babc700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdc5006caa0 0x7fdc5006ef60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdc60007fd0 tx=0x7fdc60009040).stop 2024-04-12T05:19:34.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.321+0000 7fdc6babc700 1 -- 172.21.15.113:0/3503148990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc640f8a10 msgr2=0x7fdc6410e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.321+0000 7fdc6babc700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc640f8a10 0x7fdc6410e070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fdc5400b120 tx=0x7fdc54009cb0).stop 2024-04-12T05:19:34.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.322+0000 7fdc6babc700 1 -- 172.21.15.113:0/3503148990 shutdown_connections 2024-04-12T05:19:34.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.322+0000 7fdc6babc700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdc5006caa0 0x7fdc5006ef60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.322+0000 7fdc6babc700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc640f8a10 0x7fdc6410e070 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.322+0000 7fdc6babc700 1 --2- 172.21.15.113:0/3503148990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc640f92e0 0x7fdc6410e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.322+0000 7fdc6babc700 1 -- 172.21.15.113:0/3503148990 >> 172.21.15.113:0/3503148990 conn(0x7fdc6406c760 msgr2=0x7fdc640fc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:34.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.322+0000 7fdc6babc700 1 -- 172.21.15.113:0/3503148990 shutdown_connections 2024-04-12T05:19:34.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.322+0000 7fdc6babc700 1 -- 172.21.15.113:0/3503148990 wait complete. 2024-04-12T05:19:34.334 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:19:34.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.421+0000 7f502f00e700 1 -- 172.21.15.113:0/4286554012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50280fd8f0 msgr2=0x7f50280fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.421+0000 7f502f00e700 1 --2- 172.21.15.113:0/4286554012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50280fd8f0 0x7f50280fdcd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f501c0047f0 tx=0x7f501c009cb0).stop 2024-04-12T05:19:34.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.421+0000 7f502f00e700 1 -- 172.21.15.113:0/4286554012 shutdown_connections 2024-04-12T05:19:34.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.421+0000 7f502f00e700 1 --2- 172.21.15.113:0/4286554012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50280fe250 0x7f50280f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:19:34.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.421+0000 7f502f00e700 1 --2- 172.21.15.113:0/4286554012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50280fd8f0 0x7f50280fdcd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.421+0000 7f502f00e700 1 -- 172.21.15.113:0/4286554012 >> 172.21.15.113:0/4286554012 conn(0x7f50280f34b0 msgr2=0x7f50280f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:34.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.421+0000 7f502f00e700 1 -- 172.21.15.113:0/4286554012 shutdown_connections 2024-04-12T05:19:34.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.421+0000 7f502f00e700 1 -- 172.21.15.113:0/4286554012 wait complete. 2024-04-12T05:19:34.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:33 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:34.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:33 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:19:34.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f502f00e700 1 Processor -- start 2024-04-12T05:19:34.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f502f00e700 1 -- start start 2024-04-12T05:19:34.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f502f00e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50280fd8f0 0x7f502810e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:34.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f502f00e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50280fe250 0x7f502810e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:34.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f502f00e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502810ec80 con 0x7f50280fe250 2024-04-12T05:19:34.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f502f00e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50281082a0 con 0x7f50280fd8f0 2024-04-12T05:19:34.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f5027fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50280fe250 0x7f502810e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:34.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f5027fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50280fe250 0x7f502810e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38050/0 (socket says 172.21.15.113:38050) 2024-04-12T05:19:34.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f5027fff700 1 -- 172.21.15.113:0/20599729 learned_addr learned my addr 172.21.15.113:0/20599729 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:34.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f502cdaa700 1 --2- 172.21.15.113:0/20599729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50280fd8f0 0x7f502810e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f5027fff700 1 -- 172.21.15.113:0/20599729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50280fd8f0 msgr2=0x7f502810e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f5027fff700 1 --2- 172.21.15.113:0/20599729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50280fd8f0 0x7f502810e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.422+0000 7f5027fff700 1 -- 172.21.15.113:0/20599729 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c005040 con 0x7f50280fe250 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.423+0000 7f5027fff700 1 --2- 172.21.15.113:0/20599729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50280fe250 0x7f502810e5f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f501400b480 tx=0x7f50140090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.423+0000 7f5025ffb700 1 -- 172.21.15.113:0/20599729 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5014002500 con 0x7f50280fe250 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.423+0000 7f502f00e700 1 -- 172.21.15.113:0/20599729 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5028108550 con 0x7f50280fe250 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.423+0000 7f5025ffb700 1 -- 172.21.15.113:0/20599729 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5014002b10 con 0x7f50280fe250 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.423+0000 7f5025ffb700 1 -- 172.21.15.113:0/20599729 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5014012c60 con 0x7f50280fe250 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.423+0000 7f502f00e700 1 -- 172.21.15.113:0/20599729 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5028108a90 con 0x7f50280fe250 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.424+0000 7f502f00e700 1 -- 172.21.15.113:0/20599729 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502805f590 con 0x7f50280fe250 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.424+0000 7f5025ffb700 1 -- 172.21.15.113:0/20599729 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5014002660 con 0x7f50280fe250 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.425+0000 7f5025ffb700 1 --2- 172.21.15.113:0/20599729 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f501806cab0 0x7f501806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.425+0000 7f5025ffb700 1 -- 172.21.15.113:0/20599729 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f501408be50 con 0x7f50280fe250 2024-04-12T05:19:34.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.425+0000 7f502cdaa700 1 --2- 172.21.15.113:0/20599729 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f501806cab0 0x7f501806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:34.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.425+0000 7f502cdaa700 1 --2- 172.21.15.113:0/20599729 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f501806cab0 0x7f501806ef70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f501c004f10 tx=0x7f501c009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:34.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.429+0000 7f5025ffb700 1 -- 172.21.15.113:0/20599729 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5014059900 con 0x7f50280fe250 2024-04-12T05:19:34.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.633+0000 7f502f00e700 1 -- 172.21.15.113:0/20599729 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f50281097f0 con 0x7f501806cab0 2024-04-12T05:19:34.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.635+0000 7f5025ffb700 1 -- 172.21.15.113:0/20599729 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f50281097f0 con 0x7f501806cab0 2024-04-12T05:19:34.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.637+0000 7f502f00e700 1 -- 172.21.15.113:0/20599729 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f501806cab0 msgr2=0x7f501806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.637+0000 7f502f00e700 1 --2- 172.21.15.113:0/20599729 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f501806cab0 0x7f501806ef70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f501c004f10 tx=0x7f501c009cb0).stop 2024-04-12T05:19:34.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.637+0000 7f502f00e700 1 -- 172.21.15.113:0/20599729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50280fe250 msgr2=0x7f502810e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.637+0000 7f502f00e700 1 --2- 172.21.15.113:0/20599729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50280fe250 0x7f502810e5f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f501400b480 tx=0x7f50140090f0).stop 2024-04-12T05:19:34.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.637+0000 7f502f00e700 1 -- 172.21.15.113:0/20599729 shutdown_connections 2024-04-12T05:19:34.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.637+0000 7f502f00e700 1 --2- 172.21.15.113:0/20599729 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f501806cab0 0x7f501806ef70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.637+0000 7f502f00e700 1 --2- 172.21.15.113:0/20599729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50280fd8f0 0x7f502810e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.637+0000 7f502f00e700 1 --2- 172.21.15.113:0/20599729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50280fe250 0x7f502810e5f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.637+0000 7f502f00e700 1 -- 172.21.15.113:0/20599729 >> 172.21.15.113:0/20599729 conn(0x7f50280f34b0 msgr2=0x7f50280fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:34.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.637+0000 7f502f00e700 1 -- 172.21.15.113:0/20599729 shutdown_connections 2024-04-12T05:19:34.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.637+0000 7f502f00e700 1 -- 172.21.15.113:0/20599729 wait complete. 2024-04-12T05:19:34.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.735+0000 7fcec45c9700 1 -- 172.21.15.113:0/3976050570 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcebc0fc820 msgr2=0x7fcebc0fcc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.735+0000 7fcec45c9700 1 --2- 172.21.15.113:0/3976050570 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcebc0fc820 0x7fcebc0fcc00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fceb80047f0 tx=0x7fceb8009cb0).stop 2024-04-12T05:19:34.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.735+0000 7fcec45c9700 1 -- 172.21.15.113:0/3976050570 shutdown_connections 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.735+0000 7fcec45c9700 1 --2- 172.21.15.113:0/3976050570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc0fd180 0x7fcebc0f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.735+0000 7fcec45c9700 1 --2- 172.21.15.113:0/3976050570 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcebc0fc820 0x7fcebc0fcc00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.735+0000 7fcec45c9700 1 -- 172.21.15.113:0/3976050570 >> 172.21.15.113:0/3976050570 conn(0x7fcebc0f2400 msgr2=0x7fcebc0f4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.736+0000 7fcec45c9700 1 -- 172.21.15.113:0/3976050570 shutdown_connections 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.736+0000 7fcec45c9700 1 -- 172.21.15.113:0/3976050570 wait complete. 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.736+0000 7fcec45c9700 1 Processor -- start 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.736+0000 7fcec45c9700 1 -- start start 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.736+0000 7fcec45c9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc0fc820 0x7fcebc0f8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.736+0000 7fcec45c9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcebc0fd180 0x7fcebc0f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.736+0000 7fcec45c9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0f8f50 con 0x7fcebc0fd180 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.736+0000 7fcec45c9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0f9090 con 0x7fcebc0fc820 2024-04-12T05:19:34.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.736+0000 7fcec1b64700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcebc0fd180 0x7fcebc0f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.736+0000 7fcec1b64700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcebc0fd180 0x7fcebc0f8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38054/0 (socket says 172.21.15.113:38054) 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.736+0000 7fcec1b64700 1 -- 172.21.15.113:0/2707139569 learned_addr learned my addr 172.21.15.113:0/2707139569 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.737+0000 7fcec2365700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc0fc820 0x7fcebc0f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.737+0000 7fcec1b64700 1 -- 172.21.15.113:0/2707139569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc0fc820 msgr2=0x7fcebc0f8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.737+0000 7fcec1b64700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc0fc820 0x7fcebc0f8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.737+0000 7fcec1b64700 1 -- 172.21.15.113:0/2707139569 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb8005040 con 0x7fcebc0fd180 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.737+0000 7fcec2365700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc0fc820 0x7fcebc0f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.737+0000 7fcec1b64700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcebc0fd180 0x7fcebc0f8a60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fceac00b480 tx=0x7fceac0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.737+0000 7fceb37fe700 1 -- 172.21.15.113:0/2707139569 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceac002500 con 0x7fcebc0fd180 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.737+0000 7fcec45c9700 1 -- 172.21.15.113:0/2707139569 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc0f9340 con 0x7fcebc0fd180 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.737+0000 7fceb37fe700 1 -- 172.21.15.113:0/2707139569 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fceac002b10 con 0x7fcebc0fd180 2024-04-12T05:19:34.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.737+0000 7fceb37fe700 1 -- 172.21.15.113:0/2707139569 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fceac012c90 con 0x7fcebc0fd180 2024-04-12T05:19:34.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.737+0000 7fcec45c9700 1 -- 172.21.15.113:0/2707139569 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc114740 con 0x7fcebc0fd180 2024-04-12T05:19:34.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.738+0000 7fcec45c9700 1 -- 172.21.15.113:0/2707139569 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc05f510 con 0x7fcebc0fd180 2024-04-12T05:19:34.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.738+0000 7fceb37fe700 1 -- 172.21.15.113:0/2707139569 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fceac002660 con 0x7fcebc0fd180 2024-04-12T05:19:34.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.739+0000 7fceb37fe700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcea806cab0 0x7fcea806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:34.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.739+0000 7fceb37fe700 1 -- 172.21.15.113:0/2707139569 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fceac08bef0 con 0x7fcebc0fd180 2024-04-12T05:19:34.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.739+0000 7fcec2365700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcea806cab0 0x7fcea806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:34.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.739+0000 7fcec2365700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcea806cab0 0x7fcea806ef70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fceb8007cf0 tx=0x7fceb8009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:34.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.743+0000 7fceb37fe700 1 -- 172.21.15.113:0/2707139569 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fceac055f10 con 0x7fcebc0fd180 2024-04-12T05:19:34.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.942+0000 7fcec45c9700 1 -- 172.21.15.113:0/2707139569 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcebc0fb5d0 con 0x7fcea806cab0 2024-04-12T05:19:34.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.951+0000 7fceb37fe700 1 -- 172.21.15.113:0/2707139569 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fcebc0fb5d0 con 0x7fcea806cab0 2024-04-12T05:19:34.953 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:19:34.953 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (7m) 19s ago 18m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (4m) 19s ago 18m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (4m) 2s ago 18m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (7m) 19s ago 18m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (42s) 19s ago 13m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (24s) 19s ago 13m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (6s) 2s ago 13m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (13m) 2s ago 13m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (8m) 19s ago 19m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (8m) 2s ago 17m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (5m) 19s ago 20m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (5m) 2s ago 17m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:19:34.954 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (8m) 19s ago 18m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:19:34.955 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (7m) 2s ago 17m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:19:34.955 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (4m) 19s ago 17m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:19:34.955 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (3m) 19s ago 16m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:19:34.955 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (3m) 19s ago 16m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:19:34.955 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (2m) 2s ago 15m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:19:34.955 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (2m) 2s ago 15m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:19:34.955 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (60s) 2s ago 14m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:19:34.955 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (7m) 2s ago 17m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:19:34.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.953+0000 7fcec45c9700 1 -- 172.21.15.113:0/2707139569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcea806cab0 msgr2=0x7fcea806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.953+0000 7fcec45c9700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcea806cab0 0x7fcea806ef70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fceb8007cf0 tx=0x7fceb8009cb0).stop 2024-04-12T05:19:34.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.953+0000 7fcec45c9700 1 -- 172.21.15.113:0/2707139569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcebc0fd180 msgr2=0x7fcebc0f8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:34.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.953+0000 7fcec45c9700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcebc0fd180 0x7fcebc0f8a60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fceac00b480 tx=0x7fceac0090f0).stop 2024-04-12T05:19:34.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.953+0000 7fcec45c9700 1 -- 172.21.15.113:0/2707139569 shutdown_connections 2024-04-12T05:19:34.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.953+0000 7fcec45c9700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcea806cab0 0x7fcea806ef70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.953+0000 7fcec45c9700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcebc0fc820 0x7fcebc0f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.953+0000 7fcec45c9700 1 --2- 172.21.15.113:0/2707139569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcebc0fd180 0x7fcebc0f8a60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:34.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.953+0000 7fcec45c9700 1 -- 172.21.15.113:0/2707139569 >> 172.21.15.113:0/2707139569 conn(0x7fcebc0f2400 msgr2=0x7fcebc060e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:34.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.953+0000 7fcec45c9700 1 -- 172.21.15.113:0/2707139569 shutdown_connections 2024-04-12T05:19:34.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:34.953+0000 7fcec45c9700 1 -- 172.21.15.113:0/2707139569 wait complete. 2024-04-12T05:19:35.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.051+0000 7fc52f70a700 1 -- 172.21.15.113:0/2461508725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5280fc830 msgr2=0x7fc5280fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.051+0000 7fc52f70a700 1 --2- 172.21.15.113:0/2461508725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5280fc830 0x7fc5280fcc10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fc5240047f0 tx=0x7fc524009cb0).stop 2024-04-12T05:19:35.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.051+0000 7fc52f70a700 1 -- 172.21.15.113:0/2461508725 shutdown_connections 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.051+0000 7fc52f70a700 1 --2- 172.21.15.113:0/2461508725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5280fd190 0x7fc5280f6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.051+0000 7fc52f70a700 1 --2- 172.21.15.113:0/2461508725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5280fc830 0x7fc5280fcc10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.051+0000 7fc52f70a700 1 -- 172.21.15.113:0/2461508725 >> 172.21.15.113:0/2461508725 conn(0x7fc52806c760 msgr2=0x7fc52806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.052+0000 7fc52f70a700 1 -- 172.21.15.113:0/2461508725 shutdown_connections 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.052+0000 7fc52f70a700 1 -- 172.21.15.113:0/2461508725 wait complete. 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.052+0000 7fc52f70a700 1 Processor -- start 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.052+0000 7fc52f70a700 1 -- start start 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.052+0000 7fc52f70a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5280fc830 0x7fc52810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.052+0000 7fc52f70a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5280fd190 0x7fc52810d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.052+0000 7fc52f70a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52810d9c0 con 0x7fc5280fc830 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.052+0000 7fc52f70a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52810db00 con 0x7fc5280fd190 2024-04-12T05:19:35.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc52d4a6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5280fc830 0x7fc52810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc52d4a6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5280fc830 0x7fc52810ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38068/0 (socket says 172.21.15.113:38068) 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc52d4a6700 1 -- 172.21.15.113:0/4236329126 learned_addr learned my addr 172.21.15.113:0/4236329126 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc52cca5700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5280fd190 0x7fc52810d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc52d4a6700 1 -- 172.21.15.113:0/4236329126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5280fd190 msgr2=0x7fc52810d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc52d4a6700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5280fd190 0x7fc52810d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc52d4a6700 1 -- 172.21.15.113:0/4236329126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc524005040 con 0x7fc5280fc830 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc52cca5700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5280fd190 0x7fc52810d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc52d4a6700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5280fc830 0x7fc52810ced0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fc524005010 tx=0x7fc524009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc51e7fc700 1 -- 172.21.15.113:0/4236329126 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc52400e030 con 0x7fc5280fc830 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc51e7fc700 1 -- 172.21.15.113:0/4236329126 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc524007e90 con 0x7fc5280fc830 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc51e7fc700 1 -- 172.21.15.113:0/4236329126 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc524018070 con 0x7fc5280fc830 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc52f70a700 1 -- 172.21.15.113:0/4236329126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc528107130 con 0x7fc5280fc830 2024-04-12T05:19:35.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.053+0000 7fc52f70a700 1 -- 172.21.15.113:0/4236329126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc528107690 con 0x7fc5280fc830 2024-04-12T05:19:35.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.054+0000 7fc52f70a700 1 -- 172.21.15.113:0/4236329126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52805f570 con 0x7fc5280fc830 2024-04-12T05:19:35.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.054+0000 7fc51e7fc700 1 -- 172.21.15.113:0/4236329126 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc52401d6e0 con 0x7fc5280fc830 2024-04-12T05:19:35.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.055+0000 7fc51e7fc700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc51406caa0 0x7fc51406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:35.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.055+0000 7fc51e7fc700 1 -- 172.21.15.113:0/4236329126 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc52408d060 con 0x7fc5280fc830 2024-04-12T05:19:35.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.055+0000 7fc52cca5700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc51406caa0 0x7fc51406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:35.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.055+0000 7fc52cca5700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc51406caa0 0x7fc51406ef60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fc528108810 tx=0x7fc51800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:35.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.059+0000 7fc51e7fc700 1 -- 172.21.15.113:0/4236329126 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc52405abd0 con 0x7fc5280fc830 2024-04-12T05:19:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:34 smithi192 ceph-mon[85304]: pgmap v222: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:35.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.333+0000 7fc52f70a700 1 -- 172.21.15.113:0/4236329126 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc5280feb80 con 0x7fc5280fc830 2024-04-12T05:19:35.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.334+0000 7fc51e7fc700 1 -- 172.21.15.113:0/4236329126 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fc52405a760 con 0x7fc5280fc830 2024-04-12T05:19:35.336 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:19:35.336 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:19:35.337 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:19:35.338 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:19:35.338 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:19:35.338 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:19:35.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.336+0000 7fc52f70a700 1 -- 172.21.15.113:0/4236329126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc51406caa0 msgr2=0x7fc51406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.336+0000 7fc52f70a700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc51406caa0 0x7fc51406ef60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fc528108810 tx=0x7fc51800b040).stop 2024-04-12T05:19:35.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.336+0000 7fc52f70a700 1 -- 172.21.15.113:0/4236329126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5280fc830 msgr2=0x7fc52810ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.336+0000 7fc52f70a700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5280fc830 0x7fc52810ced0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fc524005010 tx=0x7fc524009cb0).stop 2024-04-12T05:19:35.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.336+0000 7fc52f70a700 1 -- 172.21.15.113:0/4236329126 shutdown_connections 2024-04-12T05:19:35.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.336+0000 7fc52f70a700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc51406caa0 0x7fc51406ef60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.336+0000 7fc52f70a700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5280fc830 0x7fc52810ced0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.336+0000 7fc52f70a700 1 --2- 172.21.15.113:0/4236329126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5280fd190 0x7fc52810d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.336+0000 7fc52f70a700 1 -- 172.21.15.113:0/4236329126 >> 172.21.15.113:0/4236329126 conn(0x7fc52806c760 msgr2=0x7fc5280fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:35.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.336+0000 7fc52f70a700 1 -- 172.21.15.113:0/4236329126 shutdown_connections 2024-04-12T05:19:35.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.336+0000 7fc52f70a700 1 -- 172.21.15.113:0/4236329126 wait complete. 2024-04-12T05:19:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:34 smithi113 ceph-mon[103371]: pgmap v222: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:35.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.434+0000 7f9f00766700 1 -- 172.21.15.113:0/4004005623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef80fd150 msgr2=0x7f9ef80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.436 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.434+0000 7f9f00766700 1 --2- 172.21.15.113:0/4004005623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef80fd150 0x7f9ef80f6930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f9ef40047f0 tx=0x7f9ef4009cb0).stop 2024-04-12T05:19:35.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.434+0000 7f9f00766700 1 -- 172.21.15.113:0/4004005623 shutdown_connections 2024-04-12T05:19:35.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.434+0000 7f9f00766700 1 --2- 172.21.15.113:0/4004005623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef80fd150 0x7f9ef80f6930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.434+0000 7f9f00766700 1 --2- 172.21.15.113:0/4004005623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ef80fc7f0 0x7f9ef80fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.434+0000 7f9f00766700 1 -- 172.21.15.113:0/4004005623 >> 172.21.15.113:0/4004005623 conn(0x7f9ef8070590 msgr2=0x7f9ef80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:35.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.434+0000 7f9f00766700 1 -- 172.21.15.113:0/4004005623 shutdown_connections 2024-04-12T05:19:35.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.434+0000 7f9f00766700 1 -- 172.21.15.113:0/4004005623 wait complete. 2024-04-12T05:19:35.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.435+0000 7f9f00766700 1 Processor -- start 2024-04-12T05:19:35.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.435+0000 7f9f00766700 1 -- start start 2024-04-12T05:19:35.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.435+0000 7f9f00766700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef80fc7f0 0x7f9ef81090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:35.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.435+0000 7f9f00766700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ef80fd150 0x7f9ef81095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:35.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.435+0000 7f9f00766700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef8109aa0 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.435+0000 7f9f00766700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef8109be0 con 0x7f9ef80fd150 2024-04-12T05:19:35.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.435+0000 7f9efe502700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef80fc7f0 0x7f9ef81090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:35.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.435+0000 7f9efe502700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef80fc7f0 0x7f9ef81090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38086/0 (socket says 172.21.15.113:38086) 2024-04-12T05:19:35.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.435+0000 7f9efe502700 1 -- 172.21.15.113:0/3228870325 learned_addr learned my addr 172.21.15.113:0/3228870325 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:35.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.435+0000 7f9efdd01700 1 --2- 172.21.15.113:0/3228870325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ef80fd150 0x7f9ef81095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:35.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.436+0000 7f9efe502700 1 -- 172.21.15.113:0/3228870325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ef80fd150 msgr2=0x7f9ef81095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.436+0000 7f9efe502700 1 --2- 172.21.15.113:0/3228870325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ef80fd150 0x7f9ef81095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.436+0000 7f9efe502700 1 -- 172.21.15.113:0/3228870325 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef4005040 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.436+0000 7f9efe502700 1 --2- 172.21.15.113:0/3228870325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef80fc7f0 0x7f9ef81090c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f9ee80041d0 tx=0x7f9ee8009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:35.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.436+0000 7f9eef7fe700 1 -- 172.21.15.113:0/3228870325 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ee8007b40 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.436+0000 7f9eef7fe700 1 -- 172.21.15.113:0/3228870325 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ee800f040 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.436+0000 7f9f00766700 1 -- 172.21.15.113:0/3228870325 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef8109e90 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.436+0000 7f9eef7fe700 1 -- 172.21.15.113:0/3228870325 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ee80147b0 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.436+0000 7f9f00766700 1 -- 172.21.15.113:0/3228870325 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef8112480 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.437+0000 7f9f00766700 1 -- 172.21.15.113:0/3228870325 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef805f590 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.437+0000 7f9eef7fe700 1 -- 172.21.15.113:0/3228870325 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9ee8009110 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.438+0000 7f9eef7fe700 1 --2- 172.21.15.113:0/3228870325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ee406caa0 0x7f9ee406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:35.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.438+0000 7f9eef7fe700 1 -- 172.21.15.113:0/3228870325 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9ee808be50 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.438+0000 7f9efdd01700 1 --2- 172.21.15.113:0/3228870325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ee406caa0 0x7f9ee406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:35.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.438+0000 7f9efdd01700 1 --2- 172.21.15.113:0/3228870325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ee406caa0 0x7f9ee406ef60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9ef8060ee0 tx=0x7f9ef400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:35.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.442+0000 7f9eef7fe700 1 -- 172.21.15.113:0/3228870325 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9ee8055e70 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.673+0000 7f9f00766700 1 -- 172.21.15.113:0/3228870325 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9ef8047540 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.673+0000 7f9eef7fe700 1 -- 172.21.15.113:0/3228870325 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f9ee8059490 con 0x7f9ef80fc7f0 2024-04-12T05:19:35.676 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:19:35.676 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:19:35.676 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:19:35.676 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:19:35.676 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:19:35.676 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:19:35.676 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:19:35.676 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:19:35.676 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:19:35.676 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:19:35.677 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:19:35.677 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:19:35.677 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:19:35.677 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:19:35.677 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:19:35.677 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:19:35.677 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:19:35.677 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:19:35.677 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:19:35.677 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:19:35.678 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:19:35.679 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:19:35.679 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:19:35.679 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:19:35.679 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:19:35.679 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:19:35.679 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:19:35.679 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:19:35.679 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:19:35.679 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:19:35.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.675+0000 7f9f00766700 1 -- 172.21.15.113:0/3228870325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ee406caa0 msgr2=0x7f9ee406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.675+0000 7f9f00766700 1 --2- 172.21.15.113:0/3228870325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ee406caa0 0x7f9ee406ef60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9ef8060ee0 tx=0x7f9ef400b040).stop 2024-04-12T05:19:35.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.675+0000 7f9f00766700 1 -- 172.21.15.113:0/3228870325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef80fc7f0 msgr2=0x7f9ef81090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.675+0000 7f9f00766700 1 --2- 172.21.15.113:0/3228870325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef80fc7f0 0x7f9ef81090c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f9ee80041d0 tx=0x7f9ee8009f60).stop 2024-04-12T05:19:35.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.675+0000 7f9f00766700 1 -- 172.21.15.113:0/3228870325 shutdown_connections 2024-04-12T05:19:35.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.675+0000 7f9f00766700 1 --2- 172.21.15.113:0/3228870325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ee406caa0 0x7f9ee406ef60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.675+0000 7f9f00766700 1 --2- 172.21.15.113:0/3228870325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ef80fc7f0 0x7f9ef81090c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.675+0000 7f9f00766700 1 --2- 172.21.15.113:0/3228870325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ef80fd150 0x7f9ef81095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.676+0000 7f9f00766700 1 -- 172.21.15.113:0/3228870325 >> 172.21.15.113:0/3228870325 conn(0x7f9ef8070590 msgr2=0x7f9ef80fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:35.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.676+0000 7f9f00766700 1 -- 172.21.15.113:0/3228870325 shutdown_connections 2024-04-12T05:19:35.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.676+0000 7f9f00766700 1 -- 172.21.15.113:0/3228870325 wait complete. 2024-04-12T05:19:35.682 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:19:35.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.774+0000 7f8345a58700 1 -- 172.21.15.113:0/502913825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83400fc7f0 msgr2=0x7f83400fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.774+0000 7f8345a58700 1 --2- 172.21.15.113:0/502913825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83400fc7f0 0x7f83400fcbd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f83300047f0 tx=0x7f8330009cb0).stop 2024-04-12T05:19:35.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.774+0000 7f8345a58700 1 -- 172.21.15.113:0/502913825 shutdown_connections 2024-04-12T05:19:35.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.774+0000 7f8345a58700 1 --2- 172.21.15.113:0/502913825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83400fd150 0x7f83400f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.774+0000 7f8345a58700 1 --2- 172.21.15.113:0/502913825 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83400fc7f0 0x7f83400fcbd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.774+0000 7f8345a58700 1 -- 172.21.15.113:0/502913825 >> 172.21.15.113:0/502913825 conn(0x7f83400f23b0 msgr2=0x7f83400f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:35.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.774+0000 7f8345a58700 1 -- 172.21.15.113:0/502913825 shutdown_connections 2024-04-12T05:19:35.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.774+0000 7f8345a58700 1 -- 172.21.15.113:0/502913825 wait complete. 2024-04-12T05:19:35.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.775+0000 7f8345a58700 1 Processor -- start 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.775+0000 7f8345a58700 1 -- start start 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.775+0000 7f8345a58700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83400fc7f0 0x7f834010e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.775+0000 7f8345a58700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83400fd150 0x7f834010e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.775+0000 7f8345a58700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834010ec20 con 0x7f83400fc7f0 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.775+0000 7f8345a58700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834010ed60 con 0x7f83400fd150 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.775+0000 7f833effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83400fc7f0 0x7f834010e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.775+0000 7f833effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83400fc7f0 0x7f834010e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38110/0 (socket says 172.21.15.113:38110) 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.775+0000 7f833effd700 1 -- 172.21.15.113:0/947860035 learned_addr learned my addr 172.21.15.113:0/947860035 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.775+0000 7f833e7fc700 1 --2- 172.21.15.113:0/947860035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83400fd150 0x7f834010e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.776+0000 7f833effd700 1 -- 172.21.15.113:0/947860035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83400fd150 msgr2=0x7f834010e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.776+0000 7f833effd700 1 --2- 172.21.15.113:0/947860035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83400fd150 0x7f834010e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.776+0000 7f833effd700 1 -- 172.21.15.113:0/947860035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8330005040 con 0x7f83400fc7f0 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.776+0000 7f833effd700 1 --2- 172.21.15.113:0/947860035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83400fc7f0 0x7f834010e130 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f8330009380 tx=0x7f8330009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:35.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.776+0000 7f8327fff700 1 -- 172.21.15.113:0/947860035 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8330018070 con 0x7f83400fc7f0 2024-04-12T05:19:35.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.776+0000 7f8327fff700 1 -- 172.21.15.113:0/947860035 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8330007e90 con 0x7f83400fc7f0 2024-04-12T05:19:35.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.776+0000 7f8345a58700 1 -- 172.21.15.113:0/947860035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8340108390 con 0x7f83400fc7f0 2024-04-12T05:19:35.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.776+0000 7f8327fff700 1 -- 172.21.15.113:0/947860035 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83300123c0 con 0x7f83400fc7f0 2024-04-12T05:19:35.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.776+0000 7f8345a58700 1 -- 172.21.15.113:0/947860035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83401088f0 con 0x7f83400fc7f0 2024-04-12T05:19:35.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.777+0000 7f8345a58700 1 -- 172.21.15.113:0/947860035 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8340060370 con 0x7f83400fc7f0 2024-04-12T05:19:35.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.777+0000 7f8327fff700 1 -- 172.21.15.113:0/947860035 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f833001d420 con 0x7f83400fc7f0 2024-04-12T05:19:35.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.778+0000 7f8327fff700 1 --2- 172.21.15.113:0/947860035 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832806caa0 0x7f832806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:35.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.778+0000 7f8327fff700 1 -- 172.21.15.113:0/947860035 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f833008c9e0 con 0x7f83400fc7f0 2024-04-12T05:19:35.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.778+0000 7f833e7fc700 1 --2- 172.21.15.113:0/947860035 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832806caa0 0x7f832806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:35.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.778+0000 7f833e7fc700 1 --2- 172.21.15.113:0/947860035 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832806caa0 0x7f832806ef60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8340109a70 tx=0x7f833400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:35.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.782+0000 7f8327fff700 1 -- 172.21.15.113:0/947860035 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f833005a490 con 0x7f83400fc7f0 2024-04-12T05:19:35.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.986+0000 7f8345a58700 1 -- 172.21.15.113:0/947860035 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8340108f30 con 0x7f832806caa0 2024-04-12T05:19:35.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.987+0000 7f8327fff700 1 -- 172.21.15.113:0/947860035 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8340108f30 con 0x7f832806caa0 2024-04-12T05:19:35.990 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:19:35.990 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:19:35.990 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:19:35.990 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:19:35.990 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:19:35.990 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:19:35.990 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:19:35.990 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:19:35.990 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:19:35.990 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:19:35.990 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:19:35.991 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:19:35.991 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:19:35.991 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:19:35.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.989+0000 7f8345a58700 1 -- 172.21.15.113:0/947860035 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832806caa0 msgr2=0x7f832806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.989+0000 7f8345a58700 1 --2- 172.21.15.113:0/947860035 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832806caa0 0x7f832806ef60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8340109a70 tx=0x7f833400b040).stop 2024-04-12T05:19:35.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.989+0000 7f8345a58700 1 -- 172.21.15.113:0/947860035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83400fc7f0 msgr2=0x7f834010e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:35.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.989+0000 7f8345a58700 1 --2- 172.21.15.113:0/947860035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83400fc7f0 0x7f834010e130 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f8330009380 tx=0x7f8330009cb0).stop 2024-04-12T05:19:35.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.989+0000 7f8345a58700 1 -- 172.21.15.113:0/947860035 shutdown_connections 2024-04-12T05:19:35.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.989+0000 7f8345a58700 1 --2- 172.21.15.113:0/947860035 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832806caa0 0x7f832806ef60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.989+0000 7f8345a58700 1 --2- 172.21.15.113:0/947860035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83400fc7f0 0x7f834010e130 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.989+0000 7f8345a58700 1 --2- 172.21.15.113:0/947860035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83400fd150 0x7f834010e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:35.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.990+0000 7f8345a58700 1 -- 172.21.15.113:0/947860035 >> 172.21.15.113:0/947860035 conn(0x7f83400f23b0 msgr2=0x7f8340061340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:35.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.990+0000 7f8345a58700 1 -- 172.21.15.113:0/947860035 shutdown_connections 2024-04-12T05:19:35.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:35.990+0000 7f8345a58700 1 -- 172.21.15.113:0/947860035 wait complete. 2024-04-12T05:19:36.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.088+0000 7febba121700 1 -- 172.21.15.113:0/2855358093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fd150 msgr2=0x7febb40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.088+0000 7febba121700 1 --2- 172.21.15.113:0/2855358093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fd150 0x7febb40f6930 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7feba80047f0 tx=0x7feba8009cb0).stop 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.089+0000 7febba121700 1 -- 172.21.15.113:0/2855358093 shutdown_connections 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.089+0000 7febba121700 1 --2- 172.21.15.113:0/2855358093 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fd150 0x7febb40f6930 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.089+0000 7febba121700 1 --2- 172.21.15.113:0/2855358093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fc7f0 0x7febb40fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.089+0000 7febba121700 1 -- 172.21.15.113:0/2855358093 >> 172.21.15.113:0/2855358093 conn(0x7febb4070590 msgr2=0x7febb40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.089+0000 7febba121700 1 -- 172.21.15.113:0/2855358093 shutdown_connections 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.089+0000 7febba121700 1 -- 172.21.15.113:0/2855358093 wait complete. 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.089+0000 7febba121700 1 Processor -- start 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.089+0000 7febba121700 1 -- start start 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.089+0000 7febba121700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb40f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.089+0000 7febba121700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 0x7febb40f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febba121700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb40f8f20 con 0x7febb40fc7f0 2024-04-12T05:19:36.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febba121700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb40f9060 con 0x7febb40fd150 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febb37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb40f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febb37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb40f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38128/0 (socket says 172.21.15.113:38128) 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febb37fe700 1 -- 172.21.15.113:0/2107728054 learned_addr learned my addr 172.21.15.113:0/2107728054 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febb2ffd700 1 --2- 172.21.15.113:0/2107728054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 0x7febb40f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febb37fe700 1 -- 172.21.15.113:0/2107728054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 msgr2=0x7febb40f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febb37fe700 1 --2- 172.21.15.113:0/2107728054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 0x7febb40f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febb37fe700 1 -- 172.21.15.113:0/2107728054 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba8005040 con 0x7febb40fc7f0 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febb37fe700 1 --2- 172.21.15.113:0/2107728054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb40f8540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7feba4002770 tx=0x7feba4002a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febb0ff9700 1 -- 172.21.15.113:0/2107728054 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feba4007b40 con 0x7febb40fc7f0 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febb0ff9700 1 -- 172.21.15.113:0/2107728054 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feba400f040 con 0x7febb40fc7f0 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febba121700 1 -- 172.21.15.113:0/2107728054 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb40f9310 con 0x7febb40fc7f0 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febb0ff9700 1 -- 172.21.15.113:0/2107728054 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feba4014820 con 0x7febb40fc7f0 2024-04-12T05:19:36.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.090+0000 7febba121700 1 -- 172.21.15.113:0/2107728054 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb4114620 con 0x7febb40fc7f0 2024-04-12T05:19:36.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.091+0000 7febba121700 1 -- 172.21.15.113:0/2107728054 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb405f590 con 0x7febb40fc7f0 2024-04-12T05:19:36.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.092+0000 7febb0ff9700 1 -- 172.21.15.113:0/2107728054 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7feba4007ca0 con 0x7febb40fc7f0 2024-04-12T05:19:36.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.092+0000 7febb0ff9700 1 --2- 172.21.15.113:0/2107728054 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06caa0 0x7feb9c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:19:36.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.092+0000 7febb2ffd700 1 --2- 172.21.15.113:0/2107728054 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06caa0 0x7feb9c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:19:36.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.092+0000 7febb0ff9700 1 -- 172.21.15.113:0/2107728054 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7feba408b940 con 0x7febb40fc7f0 2024-04-12T05:19:36.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.093+0000 7febb2ffd700 1 --2- 172.21.15.113:0/2107728054 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06caa0 0x7feb9c06ef60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7febb40f9ce0 tx=0x7feba800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:19:36.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.097+0000 7febb0ff9700 1 -- 172.21.15.113:0/2107728054 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7feba4055960 con 0x7febb40fc7f0 2024-04-12T05:19:36.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.364+0000 7febba121700 1 -- 172.21.15.113:0/2107728054 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7febb4047540 con 0x7febb40fc7f0 2024-04-12T05:19:36.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.364+0000 7febb0ff9700 1 -- 172.21.15.113:0/2107728054 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7feba4058f80 con 0x7febb40fc7f0 2024-04-12T05:19:36.367 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:19:36.367 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:19:36.367 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:19:36.367 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:19:36.367 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:19:36.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.366+0000 7febba121700 1 -- 172.21.15.113:0/2107728054 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06caa0 msgr2=0x7feb9c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.366+0000 7febba121700 1 --2- 172.21.15.113:0/2107728054 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06caa0 0x7feb9c06ef60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7febb40f9ce0 tx=0x7feba800b040).stop 2024-04-12T05:19:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.366+0000 7febba121700 1 -- 172.21.15.113:0/2107728054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 msgr2=0x7febb40f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:19:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.366+0000 7febba121700 1 --2- 172.21.15.113:0/2107728054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb40f8540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7feba4002770 tx=0x7feba4002a80).stop 2024-04-12T05:19:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.366+0000 7febba121700 1 -- 172.21.15.113:0/2107728054 shutdown_connections 2024-04-12T05:19:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.366+0000 7febba121700 1 --2- 172.21.15.113:0/2107728054 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06caa0 0x7feb9c06ef60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.366+0000 7febba121700 1 --2- 172.21.15.113:0/2107728054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb40f8540 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.366+0000 7febba121700 1 --2- 172.21.15.113:0/2107728054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 0x7febb40f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:19:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.366+0000 7febba121700 1 -- 172.21.15.113:0/2107728054 >> 172.21.15.113:0/2107728054 conn(0x7febb4070590 msgr2=0x7febb4060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:19:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.366+0000 7febba121700 1 -- 172.21.15.113:0/2107728054 shutdown_connections 2024-04-12T05:19:36.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:19:36.366+0000 7febba121700 1 -- 172.21.15.113:0/2107728054 wait complete. 2024-04-12T05:19:36.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:35 smithi113 ceph-mon[103371]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:36.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:35 smithi113 ceph-mon[103371]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:36.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:35 smithi113 ceph-mon[103371]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:36.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:35 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/4236329126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:36.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:35 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3228870325' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:19:36.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:35 smithi192 ceph-mon[85304]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:36.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:35 smithi192 ceph-mon[85304]: from='client.34452 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:36.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:35 smithi192 ceph-mon[85304]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:36.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:35 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/4236329126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:19:36.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:35 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3228870325' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:19:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:36 smithi113 ceph-mon[103371]: pgmap v223: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:36 smithi113 ceph-mon[103371]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:36 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2107728054' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:19:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:19:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:37 smithi192 ceph-mon[85304]: pgmap v223: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:37 smithi192 ceph-mon[85304]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:19:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:37 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2107728054' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:19:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:19:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:39 smithi113 ceph-mon[103371]: pgmap v224: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:39 smithi192 ceph-mon[85304]: pgmap v224: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:41 smithi113 ceph-mon[103371]: pgmap v225: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:41 smithi192 ceph-mon[85304]: pgmap v225: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:43 smithi113 ceph-mon[103371]: pgmap v226: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:43.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:43 smithi192 ceph-mon[85304]: pgmap v226: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:45 smithi113 ceph-mon[103371]: pgmap v227: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:45 smithi192 ceph-mon[85304]: pgmap v227: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:47 smithi113 ceph-mon[103371]: pgmap v228: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:47 smithi192 ceph-mon[85304]: pgmap v228: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:49 smithi113 ceph-mon[103371]: pgmap v229: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:49.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:49 smithi192 ceph-mon[85304]: pgmap v229: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:51 smithi113 ceph-mon[103371]: pgmap v230: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:51 smithi192 ceph-mon[85304]: pgmap v230: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:19:52.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:19:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:53 smithi113 ceph-mon[103371]: pgmap v231: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:53 smithi192 ceph-mon[85304]: pgmap v231: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:55 smithi113 ceph-mon[103371]: pgmap v232: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:55.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:55 smithi192 ceph-mon[85304]: pgmap v232: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:57 smithi113 ceph-mon[103371]: pgmap v233: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:57.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:57 smithi192 ceph-mon[85304]: pgmap v233: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:19:59 smithi113 ceph-mon[103371]: pgmap v234: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:19:59.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:19:59 smithi192 ceph-mon[85304]: pgmap v234: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:00.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:19:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:20:00.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:19:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:20:00.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:19:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-12T05:20:00.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:19:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:20:00.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:19:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-12T05:20:00.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:00 smithi113 ceph-mon[103371]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:20:00.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:00 smithi113 ceph-mon[103371]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:20:00.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:00 smithi113 ceph-mon[103371]: fs cephfs is degraded 2024-04-12T05:20:00.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:00 smithi113 ceph-mon[103371]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:20:00.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:00 smithi113 ceph-mon[103371]: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:20:00.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:00 smithi192 ceph-mon[85304]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:20:00.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:00 smithi192 ceph-mon[85304]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:20:00.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:00 smithi192 ceph-mon[85304]: fs cephfs is degraded 2024-04-12T05:20:00.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:00 smithi192 ceph-mon[85304]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:20:00.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:00 smithi192 ceph-mon[85304]: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:20:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:01 smithi113 ceph-mon[103371]: pgmap v235: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:01 smithi192 ceph-mon[85304]: pgmap v235: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:03 smithi113 ceph-mon[103371]: pgmap v236: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:03 smithi192 ceph-mon[85304]: pgmap v236: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:05 smithi113 ceph-mon[103371]: pgmap v237: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:05 smithi192 ceph-mon[85304]: pgmap v237: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:06.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.476+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3963590983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5780fd8f0 msgr2=0x7fb5780fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:06.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.476+0000 7fb57e1ff700 1 --2- 172.21.15.113:0/3963590983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5780fd8f0 0x7fb5780fdcd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fb5680047f0 tx=0x7fb568009cb0).stop 2024-04-12T05:20:06.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.477+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3963590983 shutdown_connections 2024-04-12T05:20:06.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.477+0000 7fb57e1ff700 1 --2- 172.21.15.113:0/3963590983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5780fe250 0x7fb5780f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:06.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.477+0000 7fb57e1ff700 1 --2- 172.21.15.113:0/3963590983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5780fd8f0 0x7fb5780fdcd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:06.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.477+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3963590983 >> 172.21.15.113:0/3963590983 conn(0x7fb5780f34b0 msgr2=0x7fb5780f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:06.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.477+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3963590983 shutdown_connections 2024-04-12T05:20:06.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.477+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3963590983 wait complete. 2024-04-12T05:20:06.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.477+0000 7fb57e1ff700 1 Processor -- start 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.477+0000 7fb57e1ff700 1 -- start start 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb57e1ff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5780fd8f0 0x7fb57810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb57e1ff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5780fe250 0x7fb57810e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb57e1ff700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57810ead0 con 0x7fb5780fd8f0 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb57e1ff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57810ec10 con 0x7fb5780fe250 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb5777fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5780fd8f0 0x7fb57810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb5777fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5780fd8f0 0x7fb57810dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41066/0 (socket says 172.21.15.113:41066) 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb5777fe700 1 -- 172.21.15.113:0/3156384811 learned_addr learned my addr 172.21.15.113:0/3156384811 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb576ffd700 1 --2- 172.21.15.113:0/3156384811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5780fe250 0x7fb57810e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb5777fe700 1 -- 172.21.15.113:0/3156384811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5780fe250 msgr2=0x7fb57810e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb5777fe700 1 --2- 172.21.15.113:0/3156384811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5780fe250 0x7fb57810e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:06.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb5777fe700 1 -- 172.21.15.113:0/3156384811 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb568005040 con 0x7fb5780fd8f0 2024-04-12T05:20:06.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb5777fe700 1 --2- 172.21.15.113:0/3156384811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5780fd8f0 0x7fb57810dfe0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fb568005010 tx=0x7fb568009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:06.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb574ff9700 1 -- 172.21.15.113:0/3156384811 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb56800e030 con 0x7fb5780fd8f0 2024-04-12T05:20:06.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.478+0000 7fb574ff9700 1 -- 172.21.15.113:0/3156384811 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb568007e90 con 0x7fb5780fd8f0 2024-04-12T05:20:06.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.479+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3156384811 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb578108240 con 0x7fb5780fd8f0 2024-04-12T05:20:06.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.479+0000 7fb574ff9700 1 -- 172.21.15.113:0/3156384811 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb568018070 con 0x7fb5780fd8f0 2024-04-12T05:20:06.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.479+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3156384811 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb578108720 con 0x7fb5780fd8f0 2024-04-12T05:20:06.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.480+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3156384811 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57805f590 con 0x7fb5780fd8f0 2024-04-12T05:20:06.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.480+0000 7fb574ff9700 1 -- 172.21.15.113:0/3156384811 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb568009610 con 0x7fb5780fd8f0 2024-04-12T05:20:06.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.480+0000 7fb574ff9700 1 --2- 172.21.15.113:0/3156384811 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb56006ca30 0x7fb56006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:06.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.481+0000 7fb574ff9700 1 -- 172.21.15.113:0/3156384811 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb56808d0f0 con 0x7fb5780fd8f0 2024-04-12T05:20:06.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.481+0000 7fb576ffd700 1 --2- 172.21.15.113:0/3156384811 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb56006ca30 0x7fb56006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:06.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.481+0000 7fb576ffd700 1 --2- 172.21.15.113:0/3156384811 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb56006ca30 0x7fb56006eef0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb56c009a00 tx=0x7fb56c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:06.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.485+0000 7fb574ff9700 1 -- 172.21.15.113:0/3156384811 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb56805aba0 con 0x7fb5780fd8f0 2024-04-12T05:20:06.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.690+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3156384811 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5780ffc40 con 0x7fb56006ca30 2024-04-12T05:20:06.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.691+0000 7fb574ff9700 1 -- 172.21.15.113:0/3156384811 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb5780ffc40 con 0x7fb56006ca30 2024-04-12T05:20:06.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.694+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3156384811 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb56006ca30 msgr2=0x7fb56006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:06.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.694+0000 7fb57e1ff700 1 --2- 172.21.15.113:0/3156384811 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb56006ca30 0x7fb56006eef0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb56c009a00 tx=0x7fb56c00b040).stop 2024-04-12T05:20:06.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.694+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3156384811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5780fd8f0 msgr2=0x7fb57810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:06.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.694+0000 7fb57e1ff700 1 --2- 172.21.15.113:0/3156384811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5780fd8f0 0x7fb57810dfe0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fb568005010 tx=0x7fb568009cb0).stop 2024-04-12T05:20:06.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.694+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3156384811 shutdown_connections 2024-04-12T05:20:06.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.694+0000 7fb57e1ff700 1 --2- 172.21.15.113:0/3156384811 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb56006ca30 0x7fb56006eef0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:06.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.694+0000 7fb57e1ff700 1 --2- 172.21.15.113:0/3156384811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5780fd8f0 0x7fb57810dfe0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:06.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.694+0000 7fb57e1ff700 1 --2- 172.21.15.113:0/3156384811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5780fe250 0x7fb57810e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:06.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.694+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3156384811 >> 172.21.15.113:0/3156384811 conn(0x7fb5780f34b0 msgr2=0x7fb5780fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:06.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.694+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3156384811 shutdown_connections 2024-04-12T05:20:06.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.694+0000 7fb57e1ff700 1 -- 172.21.15.113:0/3156384811 wait complete. 2024-04-12T05:20:06.707 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:20:06.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.793+0000 7f9186e16700 1 -- 172.21.15.113:0/2686022498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91800fc7f0 msgr2=0x7f91800fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.793+0000 7f9186e16700 1 --2- 172.21.15.113:0/2686022498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91800fc7f0 0x7f91800fcbd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f91740047f0 tx=0x7f9174009cb0).stop 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.793+0000 7f9186e16700 1 -- 172.21.15.113:0/2686022498 shutdown_connections 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.793+0000 7f9186e16700 1 --2- 172.21.15.113:0/2686022498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91800fd150 0x7f91800f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.793+0000 7f9186e16700 1 --2- 172.21.15.113:0/2686022498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91800fc7f0 0x7f91800fcbd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.793+0000 7f9186e16700 1 -- 172.21.15.113:0/2686022498 >> 172.21.15.113:0/2686022498 conn(0x7f9180070590 msgr2=0x7f91800f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.793+0000 7f9186e16700 1 -- 172.21.15.113:0/2686022498 shutdown_connections 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.793+0000 7f9186e16700 1 -- 172.21.15.113:0/2686022498 wait complete. 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.793+0000 7f9186e16700 1 Processor -- start 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.793+0000 7f9186e16700 1 -- start start 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9186e16700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91800fc7f0 0x7f918010dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9186e16700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91800fd150 0x7f918010e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:06.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9186e16700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918010eae0 con 0x7f91800fc7f0 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9186e16700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918010ec20 con 0x7f91800fd150 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9184bb2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91800fc7f0 0x7f918010dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9184bb2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91800fc7f0 0x7f918010dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41084/0 (socket says 172.21.15.113:41084) 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9184bb2700 1 -- 172.21.15.113:0/1626895012 learned_addr learned my addr 172.21.15.113:0/1626895012 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f917ffff700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91800fd150 0x7f918010e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9184bb2700 1 -- 172.21.15.113:0/1626895012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91800fd150 msgr2=0x7f918010e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9184bb2700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91800fd150 0x7f918010e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9184bb2700 1 -- 172.21.15.113:0/1626895012 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9174005040 con 0x7f91800fc7f0 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f917ffff700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91800fd150 0x7f918010e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9184bb2700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91800fc7f0 0x7f918010dff0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f9174005010 tx=0x7f9174009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f917dffb700 1 -- 172.21.15.113:0/1626895012 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f917400e030 con 0x7f91800fc7f0 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f9186e16700 1 -- 172.21.15.113:0/1626895012 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9180108250 con 0x7f91800fc7f0 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f917dffb700 1 -- 172.21.15.113:0/1626895012 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9174007e90 con 0x7f91800fc7f0 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.794+0000 7f917dffb700 1 -- 172.21.15.113:0/1626895012 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9174018070 con 0x7f91800fc7f0 2024-04-12T05:20:06.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.795+0000 7f9186e16700 1 -- 172.21.15.113:0/1626895012 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91801087b0 con 0x7f91800fc7f0 2024-04-12T05:20:06.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.795+0000 7f9186e16700 1 -- 172.21.15.113:0/1626895012 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918005f590 con 0x7f91800fc7f0 2024-04-12T05:20:06.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.796+0000 7f917dffb700 1 -- 172.21.15.113:0/1626895012 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f917401d6e0 con 0x7f91800fc7f0 2024-04-12T05:20:06.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.796+0000 7f917dffb700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f917006ca30 0x7f917006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:06.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.796+0000 7f917dffb700 1 -- 172.21.15.113:0/1626895012 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f917408d170 con 0x7f91800fc7f0 2024-04-12T05:20:06.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.796+0000 7f917ffff700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f917006ca30 0x7f917006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:06.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.797+0000 7f917ffff700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f917006ca30 0x7f917006eef0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9180109930 tx=0x7f916c009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:06.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:06.801+0000 7f917dffb700 1 -- 172.21.15.113:0/1626895012 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f917405ac20 con 0x7f91800fc7f0 2024-04-12T05:20:07.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.007+0000 7f9186e16700 1 -- 172.21.15.113:0/1626895012 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9180109780 con 0x7f917006ca30 2024-04-12T05:20:07.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.009+0000 7f917dffb700 1 -- 172.21.15.113:0/1626895012 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9180109780 con 0x7f917006ca30 2024-04-12T05:20:07.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.011+0000 7f9186e16700 1 -- 172.21.15.113:0/1626895012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f917006ca30 msgr2=0x7f917006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.011+0000 7f9186e16700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f917006ca30 0x7f917006eef0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9180109930 tx=0x7f916c009d60).stop 2024-04-12T05:20:07.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.011+0000 7f9186e16700 1 -- 172.21.15.113:0/1626895012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91800fc7f0 msgr2=0x7f918010dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.011+0000 7f9186e16700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91800fc7f0 0x7f918010dff0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f9174005010 tx=0x7f9174009cb0).stop 2024-04-12T05:20:07.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.011+0000 7f9186e16700 1 -- 172.21.15.113:0/1626895012 shutdown_connections 2024-04-12T05:20:07.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.011+0000 7f9186e16700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f917006ca30 0x7f917006eef0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.011+0000 7f9186e16700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91800fc7f0 0x7f918010dff0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.011+0000 7f9186e16700 1 --2- 172.21.15.113:0/1626895012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91800fd150 0x7f918010e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.011+0000 7f9186e16700 1 -- 172.21.15.113:0/1626895012 >> 172.21.15.113:0/1626895012 conn(0x7f9180070590 msgr2=0x7f9180060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:07.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.011+0000 7f9186e16700 1 -- 172.21.15.113:0/1626895012 shutdown_connections 2024-04-12T05:20:07.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.011+0000 7f9186e16700 1 -- 172.21.15.113:0/1626895012 wait complete. 2024-04-12T05:20:07.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.110+0000 7fb3122ba700 1 -- 172.21.15.113:0/1866841078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb30c0fe360 msgr2=0x7fb30c0f5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.110+0000 7fb3122ba700 1 --2- 172.21.15.113:0/1866841078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb30c0fe360 0x7fb30c0f5fc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fb2fc0047f0 tx=0x7fb2fc009cb0).stop 2024-04-12T05:20:07.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.110+0000 7fb3122ba700 1 -- 172.21.15.113:0/1866841078 shutdown_connections 2024-04-12T05:20:07.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.110+0000 7fb3122ba700 1 --2- 172.21.15.113:0/1866841078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb30c0fe360 0x7fb30c0f5fc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.110+0000 7fb3122ba700 1 --2- 172.21.15.113:0/1866841078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb30c0fda00 0x7fb30c0fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.110+0000 7fb3122ba700 1 -- 172.21.15.113:0/1866841078 >> 172.21.15.113:0/1866841078 conn(0x7fb30c06cea0 msgr2=0x7fb30c06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:07.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.110+0000 7fb3122ba700 1 -- 172.21.15.113:0/1866841078 shutdown_connections 2024-04-12T05:20:07.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.110+0000 7fb3122ba700 1 -- 172.21.15.113:0/1866841078 wait complete. 2024-04-12T05:20:07.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.111+0000 7fb3122ba700 1 Processor -- start 2024-04-12T05:20:07.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.111+0000 7fb3122ba700 1 -- start start 2024-04-12T05:20:07.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.111+0000 7fb3122ba700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb30c0fda00 0x7fb30c10e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:07.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.111+0000 7fb3122ba700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb30c0fe360 0x7fb30c10e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.111+0000 7fb3122ba700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c10ec20 con 0x7fb30c0fe360 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.111+0000 7fb3122ba700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c10ed60 con 0x7fb30c0fda00 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.111+0000 7fb30b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb30c0fe360 0x7fb30c10e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb30b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb30c0fe360 0x7fb30c10e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41106/0 (socket says 172.21.15.113:41106) 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb30b7fe700 1 -- 172.21.15.113:0/762087390 learned_addr learned my addr 172.21.15.113:0/762087390 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb30bfff700 1 --2- 172.21.15.113:0/762087390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb30c0fda00 0x7fb30c10e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb30b7fe700 1 -- 172.21.15.113:0/762087390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb30c0fda00 msgr2=0x7fb30c10e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb30b7fe700 1 --2- 172.21.15.113:0/762087390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb30c0fda00 0x7fb30c10e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb30b7fe700 1 -- 172.21.15.113:0/762087390 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc005040 con 0x7fb30c0fe360 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb30b7fe700 1 --2- 172.21.15.113:0/762087390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb30c0fe360 0x7fb30c10e620 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb2fc009380 tx=0x7fb2fc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb309775700 1 -- 172.21.15.113:0/762087390 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2fc018070 con 0x7fb30c0fe360 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb309775700 1 -- 172.21.15.113:0/762087390 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2fc007e90 con 0x7fb30c0fe360 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb3122ba700 1 -- 172.21.15.113:0/762087390 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb30c108390 con 0x7fb30c0fe360 2024-04-12T05:20:07.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb309775700 1 -- 172.21.15.113:0/762087390 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2fc0123c0 con 0x7fb30c0fe360 2024-04-12T05:20:07.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.112+0000 7fb3122ba700 1 -- 172.21.15.113:0/762087390 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb30c1088f0 con 0x7fb30c0fe360 2024-04-12T05:20:07.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.113+0000 7fb3122ba700 1 -- 172.21.15.113:0/762087390 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb30c047540 con 0x7fb30c0fe360 2024-04-12T05:20:07.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.114+0000 7fb309775700 1 -- 172.21.15.113:0/762087390 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb2fc01d420 con 0x7fb30c0fe360 2024-04-12T05:20:07.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.114+0000 7fb309775700 1 --2- 172.21.15.113:0/762087390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2f406caa0 0x7fb2f406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:07.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.114+0000 7fb309775700 1 -- 172.21.15.113:0/762087390 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb2fc08c9e0 con 0x7fb30c0fe360 2024-04-12T05:20:07.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.114+0000 7fb30bfff700 1 --2- 172.21.15.113:0/762087390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2f406caa0 0x7fb2f406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:07.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.114+0000 7fb30bfff700 1 --2- 172.21.15.113:0/762087390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2f406caa0 0x7fb2f406ef60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb300004ec0 tx=0x7fb30000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:07.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.118+0000 7fb309775700 1 -- 172.21.15.113:0/762087390 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb2fc05a490 con 0x7fb30c0fe360 2024-04-12T05:20:07.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.315+0000 7fb3122ba700 1 -- 172.21.15.113:0/762087390 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb30c0ffd80 con 0x7fb2f406caa0 2024-04-12T05:20:07.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.325+0000 7fb309775700 1 -- 172.21.15.113:0/762087390 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb30c0ffd80 con 0x7fb2f406caa0 2024-04-12T05:20:07.328 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:20:07.328 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (7m) 52s ago 19m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:20:07.328 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (5m) 52s ago 19m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:20:07.328 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (5m) 34s ago 18m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:20:07.328 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (7m) 52s ago 18m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:20:07.328 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (75s) 52s ago 13m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:20:07.328 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (56s) 52s ago 13m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:20:07.328 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (39s) 34s ago 13m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (13m) 34s ago 13m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (9m) 52s ago 20m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (8m) 34s ago 18m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (6m) 52s ago 20m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (6m) 34s ago 18m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (8m) 52s ago 18m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (8m) 34s ago 18m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (4m) 52s ago 17m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (4m) 52s ago 17m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (3m) 52s ago 16m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (3m) 34s ago 16m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (2m) 34s ago 15m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (93s) 34s ago 15m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:20:07.329 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (8m) 34s ago 18m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:20:07.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.327+0000 7fb3122ba700 1 -- 172.21.15.113:0/762087390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2f406caa0 msgr2=0x7fb2f406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.327+0000 7fb3122ba700 1 --2- 172.21.15.113:0/762087390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2f406caa0 0x7fb2f406ef60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb300004ec0 tx=0x7fb30000b040).stop 2024-04-12T05:20:07.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.327+0000 7fb3122ba700 1 -- 172.21.15.113:0/762087390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb30c0fe360 msgr2=0x7fb30c10e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.327+0000 7fb3122ba700 1 --2- 172.21.15.113:0/762087390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb30c0fe360 0x7fb30c10e620 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb2fc009380 tx=0x7fb2fc009cb0).stop 2024-04-12T05:20:07.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.327+0000 7fb3122ba700 1 -- 172.21.15.113:0/762087390 shutdown_connections 2024-04-12T05:20:07.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.327+0000 7fb3122ba700 1 --2- 172.21.15.113:0/762087390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2f406caa0 0x7fb2f406ef60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.327+0000 7fb3122ba700 1 --2- 172.21.15.113:0/762087390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb30c0fda00 0x7fb30c10e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.327+0000 7fb3122ba700 1 --2- 172.21.15.113:0/762087390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb30c0fe360 0x7fb30c10e620 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.327+0000 7fb3122ba700 1 -- 172.21.15.113:0/762087390 >> 172.21.15.113:0/762087390 conn(0x7fb30c06cea0 msgr2=0x7fb30c0fbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:07.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.327+0000 7fb3122ba700 1 -- 172.21.15.113:0/762087390 shutdown_connections 2024-04-12T05:20:07.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.327+0000 7fb3122ba700 1 -- 172.21.15.113:0/762087390 wait complete. 2024-04-12T05:20:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:07 smithi113 ceph-mon[103371]: pgmap v238: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:20:07.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.424+0000 7f8151d1d700 1 -- 172.21.15.113:0/1328672712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f814c0fc7f0 msgr2=0x7f814c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.424+0000 7f8151d1d700 1 --2- 172.21.15.113:0/1328672712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f814c0fc7f0 0x7f814c0fcbd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f813c0047f0 tx=0x7f813c009cb0).stop 2024-04-12T05:20:07.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.424+0000 7f8151d1d700 1 -- 172.21.15.113:0/1328672712 shutdown_connections 2024-04-12T05:20:07.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.424+0000 7f8151d1d700 1 --2- 172.21.15.113:0/1328672712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f814c0fd150 0x7f814c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.424+0000 7f8151d1d700 1 --2- 172.21.15.113:0/1328672712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f814c0fc7f0 0x7f814c0fcbd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.424+0000 7f8151d1d700 1 -- 172.21.15.113:0/1328672712 >> 172.21.15.113:0/1328672712 conn(0x7f814c070590 msgr2=0x7f814c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:07.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.424+0000 7f8151d1d700 1 -- 172.21.15.113:0/1328672712 shutdown_connections 2024-04-12T05:20:07.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.424+0000 7f8151d1d700 1 -- 172.21.15.113:0/1328672712 wait complete. 2024-04-12T05:20:07.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.425+0000 7f8151d1d700 1 Processor -- start 2024-04-12T05:20:07.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.425+0000 7f8151d1d700 1 -- start start 2024-04-12T05:20:07.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f8151d1d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f814c0fc7f0 0x7f814c10d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:07.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f8151d1d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f814c0fd150 0x7f814c10d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:07.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f8151d1d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814c10db10 con 0x7f814c0fc7f0 2024-04-12T05:20:07.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f8151d1d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814c10dc50 con 0x7f814c0fd150 2024-04-12T05:20:07.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f814b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f814c0fc7f0 0x7f814c10d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:07.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f814b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f814c0fc7f0 0x7f814c10d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41128/0 (socket says 172.21.15.113:41128) 2024-04-12T05:20:07.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f814b7fe700 1 -- 172.21.15.113:0/758644429 learned_addr learned my addr 172.21.15.113:0/758644429 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:07.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f814affd700 1 --2- 172.21.15.113:0/758644429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f814c0fd150 0x7f814c10d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:07.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f814b7fe700 1 -- 172.21.15.113:0/758644429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f814c0fd150 msgr2=0x7f814c10d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f814b7fe700 1 --2- 172.21.15.113:0/758644429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f814c0fd150 0x7f814c10d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f814b7fe700 1 -- 172.21.15.113:0/758644429 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c005040 con 0x7f814c0fc7f0 2024-04-12T05:20:07.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f814b7fe700 1 --2- 172.21.15.113:0/758644429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f814c0fc7f0 0x7f814c10d020 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f813c009380 tx=0x7f813c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:07.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f8148ff9700 1 -- 172.21.15.113:0/758644429 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f813c018070 con 0x7f814c0fc7f0 2024-04-12T05:20:07.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f8148ff9700 1 -- 172.21.15.113:0/758644429 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f813c007e90 con 0x7f814c0fc7f0 2024-04-12T05:20:07.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f8151d1d700 1 -- 172.21.15.113:0/758644429 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f814c1072c0 con 0x7f814c0fc7f0 2024-04-12T05:20:07.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f8148ff9700 1 -- 172.21.15.113:0/758644429 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f813c0123c0 con 0x7f814c0fc7f0 2024-04-12T05:20:07.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.426+0000 7f8151d1d700 1 -- 172.21.15.113:0/758644429 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f814c107820 con 0x7f814c0fc7f0 2024-04-12T05:20:07.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.427+0000 7f8151d1d700 1 -- 172.21.15.113:0/758644429 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f814c05f590 con 0x7f814c0fc7f0 2024-04-12T05:20:07.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.428+0000 7f8148ff9700 1 -- 172.21.15.113:0/758644429 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f813c01d420 con 0x7f814c0fc7f0 2024-04-12T05:20:07.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.428+0000 7f8148ff9700 1 --2- 172.21.15.113:0/758644429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f813406caa0 0x7f813406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:07.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.428+0000 7f8148ff9700 1 -- 172.21.15.113:0/758644429 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f813c08cad0 con 0x7f814c0fc7f0 2024-04-12T05:20:07.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.428+0000 7f814affd700 1 --2- 172.21.15.113:0/758644429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f813406caa0 0x7f813406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:07.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.428+0000 7f814affd700 1 --2- 172.21.15.113:0/758644429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f813406caa0 0x7f813406ef60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f814c1089a0 tx=0x7f814000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:07.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.433+0000 7f8148ff9700 1 -- 172.21.15.113:0/758644429 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f813c056af0 con 0x7f814c0fc7f0 2024-04-12T05:20:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:07 smithi192 ceph-mon[85304]: pgmap v238: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:20:07.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.700+0000 7f8151d1d700 1 -- 172.21.15.113:0/758644429 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f814c047540 con 0x7f814c0fc7f0 2024-04-12T05:20:07.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.700+0000 7f8148ff9700 1 -- 172.21.15.113:0/758644429 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f813c05a110 con 0x7f814c0fc7f0 2024-04-12T05:20:07.704 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:20:07.704 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:20:07.704 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:20:07.705 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:20:07.705 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:20:07.705 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:20:07.705 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:20:07.705 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:20:07.705 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:20:07.705 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:20:07.705 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:20:07.705 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:20:07.706 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:20:07.706 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:20:07.706 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:20:07.706 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:20:07.706 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:20:07.706 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:20:07.707 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:20:07.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.703+0000 7f8151d1d700 1 -- 172.21.15.113:0/758644429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f813406caa0 msgr2=0x7f813406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.703+0000 7f8151d1d700 1 --2- 172.21.15.113:0/758644429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f813406caa0 0x7f813406ef60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f814c1089a0 tx=0x7f814000b040).stop 2024-04-12T05:20:07.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.703+0000 7f8151d1d700 1 -- 172.21.15.113:0/758644429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f814c0fc7f0 msgr2=0x7f814c10d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.703+0000 7f8151d1d700 1 --2- 172.21.15.113:0/758644429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f814c0fc7f0 0x7f814c10d020 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f813c009380 tx=0x7f813c009cb0).stop 2024-04-12T05:20:07.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.703+0000 7f8151d1d700 1 -- 172.21.15.113:0/758644429 shutdown_connections 2024-04-12T05:20:07.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.703+0000 7f8151d1d700 1 --2- 172.21.15.113:0/758644429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f813406caa0 0x7f813406ef60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.703+0000 7f8151d1d700 1 --2- 172.21.15.113:0/758644429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f814c0fc7f0 0x7f814c10d020 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.703+0000 7f8151d1d700 1 --2- 172.21.15.113:0/758644429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f814c0fd150 0x7f814c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.703+0000 7f8151d1d700 1 -- 172.21.15.113:0/758644429 >> 172.21.15.113:0/758644429 conn(0x7f814c070590 msgr2=0x7f814c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:07.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.703+0000 7f8151d1d700 1 -- 172.21.15.113:0/758644429 shutdown_connections 2024-04-12T05:20:07.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.703+0000 7f8151d1d700 1 -- 172.21.15.113:0/758644429 wait complete. 2024-04-12T05:20:07.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.801+0000 7fbe15792700 1 -- 172.21.15.113:0/4167549223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe100fd120 msgr2=0x7fbe100f7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.801+0000 7fbe15792700 1 --2- 172.21.15.113:0/4167549223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe100fd120 0x7fbe100f7090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fbe040047f0 tx=0x7fbe04009cb0).stop 2024-04-12T05:20:07.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.801+0000 7fbe15792700 1 -- 172.21.15.113:0/4167549223 shutdown_connections 2024-04-12T05:20:07.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.801+0000 7fbe15792700 1 --2- 172.21.15.113:0/4167549223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe100fd120 0x7fbe100f7090 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.801+0000 7fbe15792700 1 --2- 172.21.15.113:0/4167549223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe100fc7c0 0x7fbe100fcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.801+0000 7fbe15792700 1 -- 172.21.15.113:0/4167549223 >> 172.21.15.113:0/4167549223 conn(0x7fbe1006cea0 msgr2=0x7fbe1006d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:07.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.801+0000 7fbe15792700 1 -- 172.21.15.113:0/4167549223 shutdown_connections 2024-04-12T05:20:07.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.801+0000 7fbe15792700 1 -- 172.21.15.113:0/4167549223 wait complete. 2024-04-12T05:20:07.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.801+0000 7fbe15792700 1 Processor -- start 2024-04-12T05:20:07.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe15792700 1 -- start start 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe15792700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe100fc7c0 0x7fbe101090a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe15792700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe100fd120 0x7fbe10109590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe15792700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe10109a80 con 0x7fbe100fd120 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe15792700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe10109bc0 con 0x7fbe100fc7c0 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe0e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe100fd120 0x7fbe10109590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe0e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe100fd120 0x7fbe10109590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41138/0 (socket says 172.21.15.113:41138) 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe0e7fc700 1 -- 172.21.15.113:0/1878926759 learned_addr learned my addr 172.21.15.113:0/1878926759 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe0effd700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe100fc7c0 0x7fbe101090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe0e7fc700 1 -- 172.21.15.113:0/1878926759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe100fc7c0 msgr2=0x7fbe101090a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe0e7fc700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe100fc7c0 0x7fbe101090a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe0e7fc700 1 -- 172.21.15.113:0/1878926759 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe04005040 con 0x7fbe100fd120 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe0effd700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe100fc7c0 0x7fbe101090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbe0e7fc700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe100fd120 0x7fbe10109590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbe0400b120 tx=0x7fbe04009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:07.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.802+0000 7fbdf7fff700 1 -- 172.21.15.113:0/1878926759 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe0400e030 con 0x7fbe100fd120 2024-04-12T05:20:07.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.803+0000 7fbdf7fff700 1 -- 172.21.15.113:0/1878926759 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe04007e90 con 0x7fbe100fd120 2024-04-12T05:20:07.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.803+0000 7fbdf7fff700 1 -- 172.21.15.113:0/1878926759 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe04018070 con 0x7fbe100fd120 2024-04-12T05:20:07.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.803+0000 7fbe15792700 1 -- 172.21.15.113:0/1878926759 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe10109e10 con 0x7fbe100fd120 2024-04-12T05:20:07.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.803+0000 7fbe15792700 1 -- 172.21.15.113:0/1878926759 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe101123a0 con 0x7fbe100fd120 2024-04-12T05:20:07.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.804+0000 7fbe15792700 1 -- 172.21.15.113:0/1878926759 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe10047540 con 0x7fbe100fd120 2024-04-12T05:20:07.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.804+0000 7fbdf7fff700 1 -- 172.21.15.113:0/1878926759 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbe0401d420 con 0x7fbe100fd120 2024-04-12T05:20:07.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.804+0000 7fbdf7fff700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbdf806caa0 0x7fbdf806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:07.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.804+0000 7fbdf7fff700 1 -- 172.21.15.113:0/1878926759 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbe0408d300 con 0x7fbe100fd120 2024-04-12T05:20:07.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.805+0000 7fbe0effd700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbdf806caa0 0x7fbdf806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:07.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.805+0000 7fbe0effd700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbdf806caa0 0x7fbdf806ef60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fbe00004b80 tx=0x7fbe00009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:07.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:07.809+0000 7fbdf7fff700 1 -- 172.21.15.113:0/1878926759 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbe04057320 con 0x7fbe100fd120 2024-04-12T05:20:08.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.039+0000 7fbe15792700 1 -- 172.21.15.113:0/1878926759 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbe100feba0 con 0x7fbe100fd120 2024-04-12T05:20:08.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.039+0000 7fbdf7fff700 1 -- 172.21.15.113:0/1878926759 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7fbe0405a940 con 0x7fbe100fd120 2024-04-12T05:20:08.043 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:20:08.043 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:20:08.043 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:20:08.043 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:20:08.043 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:20:08.043 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:20:08.043 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:20:08.043 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:20:08.044 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:20:08.045 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:20:08.046 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:20:08.046 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:20:08.046 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:20:08.046 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:20:08.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.042+0000 7fbe15792700 1 -- 172.21.15.113:0/1878926759 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbdf806caa0 msgr2=0x7fbdf806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:08.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.042+0000 7fbe15792700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbdf806caa0 0x7fbdf806ef60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fbe00004b80 tx=0x7fbe00009040).stop 2024-04-12T05:20:08.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.042+0000 7fbe15792700 1 -- 172.21.15.113:0/1878926759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe100fd120 msgr2=0x7fbe10109590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:08.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.042+0000 7fbe15792700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe100fd120 0x7fbe10109590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbe0400b120 tx=0x7fbe04009cb0).stop 2024-04-12T05:20:08.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.042+0000 7fbe15792700 1 -- 172.21.15.113:0/1878926759 shutdown_connections 2024-04-12T05:20:08.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.042+0000 7fbe15792700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbdf806caa0 0x7fbdf806ef60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.042+0000 7fbe15792700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe100fc7c0 0x7fbe101090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.042+0000 7fbe15792700 1 --2- 172.21.15.113:0/1878926759 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe100fd120 0x7fbe10109590 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.042+0000 7fbe15792700 1 -- 172.21.15.113:0/1878926759 >> 172.21.15.113:0/1878926759 conn(0x7fbe1006cea0 msgr2=0x7fbe100fac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:08.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.042+0000 7fbe15792700 1 -- 172.21.15.113:0/1878926759 shutdown_connections 2024-04-12T05:20:08.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.042+0000 7fbe15792700 1 -- 172.21.15.113:0/1878926759 wait complete. 2024-04-12T05:20:08.048 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:20:08.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.142+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3739147596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e300fd150 msgr2=0x7f5e300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:08.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.142+0000 7f5e36cb4700 1 --2- 172.21.15.113:0/3739147596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e300fd150 0x7f5e300f6930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f5e1c0047f0 tx=0x7f5e1c009cb0).stop 2024-04-12T05:20:08.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.142+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3739147596 shutdown_connections 2024-04-12T05:20:08.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.142+0000 7f5e36cb4700 1 --2- 172.21.15.113:0/3739147596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e300fd150 0x7f5e300f6930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.142+0000 7f5e36cb4700 1 --2- 172.21.15.113:0/3739147596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e300fc7f0 0x7f5e300fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.142+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3739147596 >> 172.21.15.113:0/3739147596 conn(0x7f5e30070590 msgr2=0x7f5e300f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:08.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.142+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3739147596 shutdown_connections 2024-04-12T05:20:08.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.142+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3739147596 wait complete. 2024-04-12T05:20:08.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.142+0000 7f5e36cb4700 1 Processor -- start 2024-04-12T05:20:08.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e36cb4700 1 -- start start 2024-04-12T05:20:08.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e36cb4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e300fc7f0 0x7f5e3010dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:08.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e36cb4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e300fd150 0x7f5e3010e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:08.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e36cb4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3010eab0 con 0x7f5e300fc7f0 2024-04-12T05:20:08.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e36cb4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3010ebf0 con 0x7f5e300fd150 2024-04-12T05:20:08.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e34a50700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e300fc7f0 0x7f5e3010dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:08.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e34a50700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e300fc7f0 0x7f5e3010dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41152/0 (socket says 172.21.15.113:41152) 2024-04-12T05:20:08.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e34a50700 1 -- 172.21.15.113:0/3599792699 learned_addr learned my addr 172.21.15.113:0/3599792699 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:08.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e2ffff700 1 --2- 172.21.15.113:0/3599792699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e300fd150 0x7f5e3010e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:08.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e34a50700 1 -- 172.21.15.113:0/3599792699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e300fd150 msgr2=0x7f5e3010e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:08.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e34a50700 1 --2- 172.21.15.113:0/3599792699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e300fd150 0x7f5e3010e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e34a50700 1 -- 172.21.15.113:0/3599792699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c005040 con 0x7f5e300fc7f0 2024-04-12T05:20:08.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e34a50700 1 --2- 172.21.15.113:0/3599792699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e300fc7f0 0x7f5e3010dfc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f5e2400c740 tx=0x7f5e24007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:08.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e2dffb700 1 -- 172.21.15.113:0/3599792699 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e2400e680 con 0x7f5e300fc7f0 2024-04-12T05:20:08.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e2dffb700 1 -- 172.21.15.113:0/3599792699 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e2400ec90 con 0x7f5e300fc7f0 2024-04-12T05:20:08.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3599792699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e30108280 con 0x7f5e300fc7f0 2024-04-12T05:20:08.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.143+0000 7f5e2dffb700 1 -- 172.21.15.113:0/3599792699 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e24013d30 con 0x7f5e300fc7f0 2024-04-12T05:20:08.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.144+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3599792699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e30108840 con 0x7f5e300fc7f0 2024-04-12T05:20:08.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.145+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3599792699 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3005f590 con 0x7f5e300fc7f0 2024-04-12T05:20:08.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.145+0000 7f5e2dffb700 1 -- 172.21.15.113:0/3599792699 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5e24013410 con 0x7f5e300fc7f0 2024-04-12T05:20:08.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.145+0000 7f5e2dffb700 1 --2- 172.21.15.113:0/3599792699 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e2006caa0 0x7f5e2006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:08.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.145+0000 7f5e2dffb700 1 -- 172.21.15.113:0/3599792699 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5e2408c090 con 0x7f5e300fc7f0 2024-04-12T05:20:08.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.145+0000 7f5e2ffff700 1 --2- 172.21.15.113:0/3599792699 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e2006caa0 0x7f5e2006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:08.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.146+0000 7f5e2ffff700 1 --2- 172.21.15.113:0/3599792699 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e2006caa0 0x7f5e2006ef60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5e30109900 tx=0x7f5e1c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:08.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.150+0000 7f5e2dffb700 1 -- 172.21.15.113:0/3599792699 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5e24059b40 con 0x7f5e300fc7f0 2024-04-12T05:20:08.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.356+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3599792699 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e30108ee0 con 0x7f5e2006caa0 2024-04-12T05:20:08.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.357+0000 7f5e2dffb700 1 -- 172.21.15.113:0/3599792699 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5e30108ee0 con 0x7f5e2006caa0 2024-04-12T05:20:08.360 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:20:08.361 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:20:08.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.359+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3599792699 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e2006caa0 msgr2=0x7f5e2006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:08.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.359+0000 7f5e36cb4700 1 --2- 172.21.15.113:0/3599792699 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e2006caa0 0x7f5e2006ef60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5e30109900 tx=0x7f5e1c00b040).stop 2024-04-12T05:20:08.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.359+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3599792699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e300fc7f0 msgr2=0x7f5e3010dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:08.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.359+0000 7f5e36cb4700 1 --2- 172.21.15.113:0/3599792699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e300fc7f0 0x7f5e3010dfc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f5e2400c740 tx=0x7f5e24007ef0).stop 2024-04-12T05:20:08.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.359+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3599792699 shutdown_connections 2024-04-12T05:20:08.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.359+0000 7f5e36cb4700 1 --2- 172.21.15.113:0/3599792699 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e2006caa0 0x7f5e2006ef60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.359+0000 7f5e36cb4700 1 --2- 172.21.15.113:0/3599792699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e300fc7f0 0x7f5e3010dfc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.359+0000 7f5e36cb4700 1 --2- 172.21.15.113:0/3599792699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e300fd150 0x7f5e3010e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.359+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3599792699 >> 172.21.15.113:0/3599792699 conn(0x7f5e30070590 msgr2=0x7f5e30060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:08.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.360+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3599792699 shutdown_connections 2024-04-12T05:20:08.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.360+0000 7f5e36cb4700 1 -- 172.21.15.113:0/3599792699 wait complete. 2024-04-12T05:20:08.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:08 smithi113 ceph-mon[103371]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:08.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:08 smithi113 ceph-mon[103371]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:08.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:08 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/758644429' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:20:08.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:08 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1878926759' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:20:08.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:08 smithi192 ceph-mon[85304]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:08.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:08 smithi192 ceph-mon[85304]: from='client.34480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:08.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:08 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/758644429' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:20:08.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:08 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1878926759' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:20:08.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.458+0000 7f0580edb700 1 -- 172.21.15.113:0/119777309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f057c0fd150 msgr2=0x7f057c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:08.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.458+0000 7f0580edb700 1 --2- 172.21.15.113:0/119777309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f057c0fd150 0x7f057c0f6930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f056c0047f0 tx=0x7f056c009cb0).stop 2024-04-12T05:20:08.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.458+0000 7f0580edb700 1 -- 172.21.15.113:0/119777309 shutdown_connections 2024-04-12T05:20:08.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.458+0000 7f0580edb700 1 --2- 172.21.15.113:0/119777309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f057c0fd150 0x7f057c0f6930 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.458+0000 7f0580edb700 1 --2- 172.21.15.113:0/119777309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f057c0fc7f0 0x7f057c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.458+0000 7f0580edb700 1 -- 172.21.15.113:0/119777309 >> 172.21.15.113:0/119777309 conn(0x7f057c070590 msgr2=0x7f057c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:08.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.458+0000 7f0580edb700 1 -- 172.21.15.113:0/119777309 shutdown_connections 2024-04-12T05:20:08.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.458+0000 7f0580edb700 1 -- 172.21.15.113:0/119777309 wait complete. 2024-04-12T05:20:08.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.459+0000 7f0580edb700 1 Processor -- start 2024-04-12T05:20:08.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.459+0000 7f0580edb700 1 -- start start 2024-04-12T05:20:08.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.459+0000 7f0580edb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f057c0fc7f0 0x7f057c1101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:08.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.459+0000 7f0580edb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f057c0fd150 0x7f057c1106c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:08.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.459+0000 7f0580edb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c110cc0 con 0x7f057c0fd150 2024-04-12T05:20:08.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.459+0000 7f0580edb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c110e00 con 0x7f057c0fc7f0 2024-04-12T05:20:08.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.459+0000 7f0579d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f057c0fd150 0x7f057c1106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:08.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.459+0000 7f0579d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f057c0fd150 0x7f057c1106c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41170/0 (socket says 172.21.15.113:41170) 2024-04-12T05:20:08.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.459+0000 7f0579d9b700 1 -- 172.21.15.113:0/3916989415 learned_addr learned my addr 172.21.15.113:0/3916989415 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:08.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.459+0000 7f057a59c700 1 --2- 172.21.15.113:0/3916989415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f057c0fc7f0 0x7f057c1101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:08.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.459+0000 7f0579d9b700 1 -- 172.21.15.113:0/3916989415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f057c0fc7f0 msgr2=0x7f057c1101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:08.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.460+0000 7f0579d9b700 1 --2- 172.21.15.113:0/3916989415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f057c0fc7f0 0x7f057c1101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.460+0000 7f0579d9b700 1 -- 172.21.15.113:0/3916989415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c005040 con 0x7f057c0fd150 2024-04-12T05:20:08.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.460+0000 7f0579d9b700 1 --2- 172.21.15.113:0/3916989415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f057c0fd150 0x7f057c1106c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f056c0099f0 tx=0x7f056c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:08.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.460+0000 7f05737fe700 1 -- 172.21.15.113:0/3916989415 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056c00e030 con 0x7f057c0fd150 2024-04-12T05:20:08.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.460+0000 7f05737fe700 1 -- 172.21.15.113:0/3916989415 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f056c007e90 con 0x7f057c0fd150 2024-04-12T05:20:08.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.460+0000 7f0580edb700 1 -- 172.21.15.113:0/3916989415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057c10a430 con 0x7f057c0fd150 2024-04-12T05:20:08.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.460+0000 7f05737fe700 1 -- 172.21.15.113:0/3916989415 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056c018070 con 0x7f057c0fd150 2024-04-12T05:20:08.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.460+0000 7f0580edb700 1 -- 172.21.15.113:0/3916989415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057c10a990 con 0x7f057c0fd150 2024-04-12T05:20:08.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.461+0000 7f0580edb700 1 -- 172.21.15.113:0/3916989415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057c05f590 con 0x7f057c0fd150 2024-04-12T05:20:08.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.461+0000 7f05737fe700 1 -- 172.21.15.113:0/3916989415 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f056c01d420 con 0x7f057c0fd150 2024-04-12T05:20:08.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.462+0000 7f05737fe700 1 --2- 172.21.15.113:0/3916989415 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f056806caa0 0x7f056806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:08.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.462+0000 7f05737fe700 1 -- 172.21.15.113:0/3916989415 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f056c08d240 con 0x7f057c0fd150 2024-04-12T05:20:08.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.462+0000 7f057a59c700 1 --2- 172.21.15.113:0/3916989415 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f056806caa0 0x7f056806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:08.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.462+0000 7f057a59c700 1 --2- 172.21.15.113:0/3916989415 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f056806caa0 0x7f056806ef60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f05640041d0 tx=0x7f0564009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:08.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.466+0000 7f05737fe700 1 -- 172.21.15.113:0/3916989415 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f056c05acf0 con 0x7f057c0fd150 2024-04-12T05:20:08.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.734+0000 7f0580edb700 1 -- 172.21.15.113:0/3916989415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f057c10afd0 con 0x7f057c0fd150 2024-04-12T05:20:08.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.734+0000 7f05737fe700 1 -- 172.21.15.113:0/3916989415 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f056c05a880 con 0x7f057c0fd150 2024-04-12T05:20:08.738 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:20:08.738 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:20:08.738 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:20:08.738 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:20:08.738 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:20:08.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.736+0000 7f0580edb700 1 -- 172.21.15.113:0/3916989415 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f056806caa0 msgr2=0x7f056806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:08.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.736+0000 7f0580edb700 1 --2- 172.21.15.113:0/3916989415 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f056806caa0 0x7f056806ef60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f05640041d0 tx=0x7f0564009040).stop 2024-04-12T05:20:08.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.736+0000 7f0580edb700 1 -- 172.21.15.113:0/3916989415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f057c0fd150 msgr2=0x7f057c1106c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:08.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.736+0000 7f0580edb700 1 --2- 172.21.15.113:0/3916989415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f057c0fd150 0x7f057c1106c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f056c0099f0 tx=0x7f056c009cb0).stop 2024-04-12T05:20:08.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.736+0000 7f0580edb700 1 -- 172.21.15.113:0/3916989415 shutdown_connections 2024-04-12T05:20:08.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.736+0000 7f0580edb700 1 --2- 172.21.15.113:0/3916989415 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f056806caa0 0x7f056806ef60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.736+0000 7f0580edb700 1 --2- 172.21.15.113:0/3916989415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f057c0fc7f0 0x7f057c1101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.736+0000 7f0580edb700 1 --2- 172.21.15.113:0/3916989415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f057c0fd150 0x7f057c1106c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:08.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.736+0000 7f0580edb700 1 -- 172.21.15.113:0/3916989415 >> 172.21.15.113:0/3916989415 conn(0x7f057c070590 msgr2=0x7f057c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:08.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.736+0000 7f0580edb700 1 -- 172.21.15.113:0/3916989415 shutdown_connections 2024-04-12T05:20:08.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:08.737+0000 7f0580edb700 1 -- 172.21.15.113:0/3916989415 wait complete. 2024-04-12T05:20:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:09 smithi113 ceph-mon[103371]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:09 smithi113 ceph-mon[103371]: pgmap v239: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:09 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3916989415' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:20:09.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:09 smithi192 ceph-mon[85304]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:09.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:09 smithi192 ceph-mon[85304]: pgmap v239: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:09.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:09 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3916989415' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:20:10.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:10 smithi113 ceph-mon[103371]: from='client.34496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:10.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:10 smithi192 ceph-mon[85304]: from='client.34496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:11 smithi113 ceph-mon[103371]: pgmap v240: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:11 smithi192 ceph-mon[85304]: pgmap v240: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:13 smithi113 ceph-mon[103371]: pgmap v241: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:13.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:13 smithi192 ceph-mon[85304]: pgmap v241: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:15 smithi113 ceph-mon[103371]: pgmap v242: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:15 smithi192 ceph-mon[85304]: pgmap v242: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:17 smithi113 ceph-mon[103371]: pgmap v243: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:17 smithi192 ceph-mon[85304]: pgmap v243: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:19 smithi113 ceph-mon[103371]: pgmap v244: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:19 smithi192 ceph-mon[85304]: pgmap v244: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:21 smithi113 ceph-mon[103371]: pgmap v245: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:21 smithi192 ceph-mon[85304]: pgmap v245: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:20:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:20:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:23 smithi113 ceph-mon[103371]: pgmap v246: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:23 smithi192 ceph-mon[85304]: pgmap v246: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:24 smithi113 ceph-mon[103371]: pgmap v247: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:24.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:24 smithi192 ceph-mon[85304]: pgmap v247: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:27.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:26 smithi113 ceph-mon[103371]: pgmap v248: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:27.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:26 smithi192 ceph-mon[85304]: pgmap v248: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:29.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:28 smithi113 ceph-mon[103371]: pgmap v249: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:29.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:28 smithi192 ceph-mon[85304]: pgmap v249: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:31.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:30 smithi113 ceph-mon[103371]: pgmap v250: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:31.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:30 smithi192 ceph-mon[85304]: pgmap v250: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:33.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:32 smithi113 ceph-mon[103371]: pgmap v251: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:33.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:32 smithi192 ceph-mon[85304]: pgmap v251: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:35.174 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:34 smithi113 ceph-mon[103371]: pgmap v252: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:35.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:34 smithi192 ceph-mon[85304]: pgmap v252: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:37.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:36 smithi192 ceph-mon[85304]: pgmap v253: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:37.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:20:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:36 smithi113 ceph-mon[103371]: pgmap v253: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:20:38.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.849+0000 7f02f7c0f700 1 -- 172.21.15.113:0/2367597373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02f00f9200 msgr2=0x7f02f0101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:38.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.849+0000 7f02f7c0f700 1 --2- 172.21.15.113:0/2367597373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02f00f9200 0x7f02f0101910 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f02e00047f0 tx=0x7f02e0009cb0).stop 2024-04-12T05:20:38.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.849+0000 7f02f7c0f700 1 -- 172.21.15.113:0/2367597373 shutdown_connections 2024-04-12T05:20:38.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.849+0000 7f02f7c0f700 1 --2- 172.21.15.113:0/2367597373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02f00f9200 0x7f02f0101910 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:38.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.849+0000 7f02f7c0f700 1 --2- 172.21.15.113:0/2367597373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f00f88a0 0x7f02f00f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:38.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.849+0000 7f02f7c0f700 1 -- 172.21.15.113:0/2367597373 >> 172.21.15.113:0/2367597373 conn(0x7f02f0070590 msgr2=0x7f02f00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:38.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.849+0000 7f02f7c0f700 1 -- 172.21.15.113:0/2367597373 shutdown_connections 2024-04-12T05:20:38.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.849+0000 7f02f7c0f700 1 -- 172.21.15.113:0/2367597373 wait complete. 2024-04-12T05:20:38.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f7c0f700 1 Processor -- start 2024-04-12T05:20:38.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f7c0f700 1 -- start start 2024-04-12T05:20:38.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f7c0f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02f00f88a0 0x7f02f010e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:38.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f7c0f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f00f9200 0x7f02f010e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f7c0f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f010eb70 con 0x7f02f00f88a0 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f7c0f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f010ecb0 con 0x7f02f00f9200 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f59ab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02f00f88a0 0x7f02f010e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f59ab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02f00f88a0 0x7f02f010e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47166/0 (socket says 172.21.15.113:47166) 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f59ab700 1 -- 172.21.15.113:0/1640879615 learned_addr learned my addr 172.21.15.113:0/1640879615 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f51aa700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f00f9200 0x7f02f010e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f59ab700 1 -- 172.21.15.113:0/1640879615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f00f9200 msgr2=0x7f02f010e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f59ab700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f00f9200 0x7f02f010e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f59ab700 1 -- 172.21.15.113:0/1640879615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e0005040 con 0x7f02f00f88a0 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f51aa700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f00f9200 0x7f02f010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.850+0000 7f02f59ab700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02f00f88a0 0x7f02f010e080 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f02ec00c790 tx=0x7f02ec004330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.851+0000 7f02e6ffd700 1 -- 172.21.15.113:0/1640879615 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02ec0106f0 con 0x7f02f00f88a0 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.851+0000 7f02f7c0f700 1 -- 172.21.15.113:0/1640879615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f0108340 con 0x7f02f00f88a0 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.851+0000 7f02e6ffd700 1 -- 172.21.15.113:0/1640879615 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02ec010d00 con 0x7f02f00f88a0 2024-04-12T05:20:38.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.851+0000 7f02e6ffd700 1 -- 172.21.15.113:0/1640879615 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02ec012da0 con 0x7f02f00f88a0 2024-04-12T05:20:38.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.851+0000 7f02f7c0f700 1 -- 172.21.15.113:0/1640879615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f0108880 con 0x7f02f00f88a0 2024-04-12T05:20:38.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.852+0000 7f02f7c0f700 1 -- 172.21.15.113:0/1640879615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f005f590 con 0x7f02f00f88a0 2024-04-12T05:20:38.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.852+0000 7f02e6ffd700 1 -- 172.21.15.113:0/1640879615 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f02ec012400 con 0x7f02f00f88a0 2024-04-12T05:20:38.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.852+0000 7f02e6ffd700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f02dc06caa0 0x7f02dc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:38.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.853+0000 7f02e6ffd700 1 -- 172.21.15.113:0/1640879615 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f02ec08b750 con 0x7f02f00f88a0 2024-04-12T05:20:38.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.853+0000 7f02f51aa700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f02dc06caa0 0x7f02dc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:38.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.853+0000 7f02f51aa700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f02dc06caa0 0x7f02dc06ef60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f02e00043d0 tx=0x7f02e000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:38.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:38.857+0000 7f02e6ffd700 1 -- 172.21.15.113:0/1640879615 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f02ec059200 con 0x7f02f00f88a0 2024-04-12T05:20:39.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.065+0000 7f02f7c0f700 1 -- 172.21.15.113:0/1640879615 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f02f0108b00 con 0x7f02dc06caa0 2024-04-12T05:20:39.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.067+0000 7f02e6ffd700 1 -- 172.21.15.113:0/1640879615 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f02f0108b00 con 0x7f02dc06caa0 2024-04-12T05:20:39.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.069+0000 7f02f7c0f700 1 -- 172.21.15.113:0/1640879615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f02dc06caa0 msgr2=0x7f02dc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.069+0000 7f02f7c0f700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f02dc06caa0 0x7f02dc06ef60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f02e00043d0 tx=0x7f02e000b040).stop 2024-04-12T05:20:39.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.069+0000 7f02f7c0f700 1 -- 172.21.15.113:0/1640879615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02f00f88a0 msgr2=0x7f02f010e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.069+0000 7f02f7c0f700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02f00f88a0 0x7f02f010e080 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f02ec00c790 tx=0x7f02ec004330).stop 2024-04-12T05:20:39.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.069+0000 7f02f7c0f700 1 -- 172.21.15.113:0/1640879615 shutdown_connections 2024-04-12T05:20:39.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.069+0000 7f02f7c0f700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f02dc06caa0 0x7f02dc06ef60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.069+0000 7f02f7c0f700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02f00f88a0 0x7f02f010e080 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.069+0000 7f02f7c0f700 1 --2- 172.21.15.113:0/1640879615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f00f9200 0x7f02f010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.069+0000 7f02f7c0f700 1 -- 172.21.15.113:0/1640879615 >> 172.21.15.113:0/1640879615 conn(0x7f02f0070590 msgr2=0x7f02f00fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:39.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.069+0000 7f02f7c0f700 1 -- 172.21.15.113:0/1640879615 shutdown_connections 2024-04-12T05:20:39.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.069+0000 7f02f7c0f700 1 -- 172.21.15.113:0/1640879615 wait complete. 2024-04-12T05:20:39.081 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:20:39.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.168+0000 7fce6b23a700 1 -- 172.21.15.113:0/1286871562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce640fc7f0 msgr2=0x7fce640fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.168+0000 7fce6b23a700 1 --2- 172.21.15.113:0/1286871562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce640fc7f0 0x7fce640fcbd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fce580047f0 tx=0x7fce58009cb0).stop 2024-04-12T05:20:39.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.168+0000 7fce6b23a700 1 -- 172.21.15.113:0/1286871562 shutdown_connections 2024-04-12T05:20:39.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.168+0000 7fce6b23a700 1 --2- 172.21.15.113:0/1286871562 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640fd150 0x7fce640f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:20:39.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.168+0000 7fce6b23a700 1 --2- 172.21.15.113:0/1286871562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce640fc7f0 0x7fce640fcbd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.168+0000 7fce6b23a700 1 -- 172.21.15.113:0/1286871562 >> 172.21.15.113:0/1286871562 conn(0x7fce64070590 msgr2=0x7fce640f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:39.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.168+0000 7fce6b23a700 1 -- 172.21.15.113:0/1286871562 shutdown_connections 2024-04-12T05:20:39.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.168+0000 7fce6b23a700 1 -- 172.21.15.113:0/1286871562 wait complete. 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce6b23a700 1 Processor -- start 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce6b23a700 1 -- start start 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce6b23a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640fc7f0 0x7fce640f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce6b23a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce640fd150 0x7fce640f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce6b23a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce640f8f20 con 0x7fce640fd150 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce6b23a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce640f9060 con 0x7fce640fc7f0 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce63fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce640fd150 0x7fce640f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce63fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce640fd150 0x7fce640f8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47184/0 (socket says 172.21.15.113:47184) 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce63fff700 1 -- 172.21.15.113:0/3688142188 learned_addr learned my addr 172.21.15.113:0/3688142188 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce63fff700 1 -- 172.21.15.113:0/3688142188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640fc7f0 msgr2=0x7fce640f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce63fff700 1 --2- 172.21.15.113:0/3688142188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640fc7f0 0x7fce640f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce63fff700 1 -- 172.21.15.113:0/3688142188 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce58005040 con 0x7fce640fd150 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce63fff700 1 --2- 172.21.15.113:0/3688142188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce640fd150 0x7fce640f8a30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fce5000b480 tx=0x7fce500090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.169+0000 7fce61ffb700 1 -- 172.21.15.113:0/3688142188 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce50002500 con 0x7fce640fd150 2024-04-12T05:20:39.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.170+0000 7fce61ffb700 1 -- 172.21.15.113:0/3688142188 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce50002b10 con 0x7fce640fd150 2024-04-12T05:20:39.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.170+0000 7fce6b23a700 1 -- 172.21.15.113:0/3688142188 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce640f9310 con 0x7fce640fd150 2024-04-12T05:20:39.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.170+0000 7fce61ffb700 1 -- 172.21.15.113:0/3688142188 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce50012c90 con 0x7fce640fd150 2024-04-12T05:20:39.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.170+0000 7fce6b23a700 1 -- 172.21.15.113:0/3688142188 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce64114720 con 0x7fce640fd150 2024-04-12T05:20:39.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.171+0000 7fce6b23a700 1 -- 172.21.15.113:0/3688142188 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6405f590 con 0x7fce640fd150 2024-04-12T05:20:39.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.171+0000 7fce61ffb700 1 -- 172.21.15.113:0/3688142188 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fce50002660 con 0x7fce640fd150 2024-04-12T05:20:39.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.171+0000 7fce61ffb700 1 --2- 172.21.15.113:0/3688142188 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fce5406caa0 0x7fce5406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:39.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.171+0000 7fce61ffb700 1 -- 172.21.15.113:0/3688142188 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fce5008bf20 con 0x7fce640fd150 2024-04-12T05:20:39.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.171+0000 7fce68fd6700 1 --2- 172.21.15.113:0/3688142188 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fce5406caa0 0x7fce5406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:39.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.172+0000 7fce68fd6700 1 --2- 172.21.15.113:0/3688142188 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fce5406caa0 0x7fce5406ef60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fce580047f0 tx=0x7fce58009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:39.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.176+0000 7fce61ffb700 1 -- 172.21.15.113:0/3688142188 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fce500bf2c0 con 0x7fce640fd150 2024-04-12T05:20:39.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:38 smithi192 ceph-mon[85304]: pgmap v254: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:39.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.381+0000 7fce6b23a700 1 -- 172.21.15.113:0/3688142188 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce640fb5a0 con 0x7fce5406caa0 2024-04-12T05:20:39.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.382+0000 7fce61ffb700 1 -- 172.21.15.113:0/3688142188 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fce640fb5a0 con 0x7fce5406caa0 2024-04-12T05:20:39.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.384+0000 7fce6b23a700 1 -- 172.21.15.113:0/3688142188 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fce5406caa0 msgr2=0x7fce5406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.384+0000 7fce6b23a700 1 --2- 172.21.15.113:0/3688142188 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fce5406caa0 0x7fce5406ef60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fce580047f0 tx=0x7fce58009cb0).stop 2024-04-12T05:20:39.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.384+0000 7fce6b23a700 1 -- 172.21.15.113:0/3688142188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce640fd150 msgr2=0x7fce640f8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.384+0000 7fce6b23a700 1 --2- 172.21.15.113:0/3688142188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce640fd150 0x7fce640f8a30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fce5000b480 tx=0x7fce500090f0).stop 2024-04-12T05:20:39.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.384+0000 7fce6b23a700 1 -- 172.21.15.113:0/3688142188 shutdown_connections 2024-04-12T05:20:39.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.384+0000 7fce6b23a700 1 --2- 172.21.15.113:0/3688142188 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fce5406caa0 0x7fce5406ef60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.384+0000 7fce6b23a700 1 --2- 172.21.15.113:0/3688142188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640fc7f0 0x7fce640f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:20:39.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.384+0000 7fce6b23a700 1 --2- 172.21.15.113:0/3688142188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce640fd150 0x7fce640f8a30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.384+0000 7fce6b23a700 1 -- 172.21.15.113:0/3688142188 >> 172.21.15.113:0/3688142188 conn(0x7fce64070590 msgr2=0x7fce64060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:39.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.384+0000 7fce6b23a700 1 -- 172.21.15.113:0/3688142188 shutdown_connections 2024-04-12T05:20:39.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.384+0000 7fce6b23a700 1 -- 172.21.15.113:0/3688142188 wait complete. 2024-04-12T05:20:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:38 smithi113 ceph-mon[103371]: pgmap v254: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:39.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.483+0000 7fc1b196c700 1 -- 172.21.15.113:0/2387494175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1ac0fc7f0 msgr2=0x7fc1ac0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.483+0000 7fc1b196c700 1 --2- 172.21.15.113:0/2387494175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1ac0fc7f0 0x7fc1ac0fcbd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fc1940047f0 tx=0x7fc194009cb0).stop 2024-04-12T05:20:39.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.483+0000 7fc1b196c700 1 -- 172.21.15.113:0/2387494175 shutdown_connections 2024-04-12T05:20:39.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.483+0000 7fc1b196c700 1 --2- 172.21.15.113:0/2387494175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ac0fd150 0x7fc1ac0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.483+0000 7fc1b196c700 1 --2- 172.21.15.113:0/2387494175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1ac0fc7f0 0x7fc1ac0fcbd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.483+0000 7fc1b196c700 1 -- 172.21.15.113:0/2387494175 >> 172.21.15.113:0/2387494175 conn(0x7fc1ac0f23b0 msgr2=0x7fc1ac0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:39.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.483+0000 7fc1b196c700 1 -- 172.21.15.113:0/2387494175 shutdown_connections 2024-04-12T05:20:39.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.483+0000 7fc1b196c700 1 -- 172.21.15.113:0/2387494175 wait complete. 2024-04-12T05:20:39.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.483+0000 7fc1b196c700 1 Processor -- start 2024-04-12T05:20:39.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1b196c700 1 -- start start 2024-04-12T05:20:39.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1b196c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1ac0fc7f0 0x7fc1ac10cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:39.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1b196c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ac0fd150 0x7fc1ac10d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:39.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1b196c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac10da10 con 0x7fc1ac0fc7f0 2024-04-12T05:20:39.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1b196c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac10db50 con 0x7fc1ac0fd150 2024-04-12T05:20:39.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1aaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1ac0fc7f0 0x7fc1ac10cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:39.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1aaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1ac0fc7f0 0x7fc1ac10cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47188/0 (socket says 172.21.15.113:47188) 2024-04-12T05:20:39.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1aaffd700 1 -- 172.21.15.113:0/1730889244 learned_addr learned my addr 172.21.15.113:0/1730889244 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:39.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1aa7fc700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ac0fd150 0x7fc1ac10d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:39.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1aaffd700 1 -- 172.21.15.113:0/1730889244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ac0fd150 msgr2=0x7fc1ac10d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1aaffd700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ac0fd150 0x7fc1ac10d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1aaffd700 1 -- 172.21.15.113:0/1730889244 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194005040 con 0x7fc1ac0fc7f0 2024-04-12T05:20:39.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1aa7fc700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ac0fd150 0x7fc1ac10d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:20:39.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1aaffd700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1ac0fc7f0 0x7fc1ac10cf20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fc194005010 tx=0x7fc194009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:39.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1a3fff700 1 -- 172.21.15.113:0/1730889244 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc19400e030 con 0x7fc1ac0fc7f0 2024-04-12T05:20:39.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1b196c700 1 -- 172.21.15.113:0/1730889244 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac107180 con 0x7fc1ac0fc7f0 2024-04-12T05:20:39.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1a3fff700 1 -- 172.21.15.113:0/1730889244 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc194007e90 con 0x7fc1ac0fc7f0 2024-04-12T05:20:39.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.484+0000 7fc1a3fff700 1 -- 172.21.15.113:0/1730889244 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc194018070 con 0x7fc1ac0fc7f0 2024-04-12T05:20:39.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.485+0000 7fc1b196c700 1 -- 172.21.15.113:0/1730889244 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac107660 con 0x7fc1ac0fc7f0 2024-04-12T05:20:39.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.486+0000 7fc1b196c700 1 -- 172.21.15.113:0/1730889244 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ac060370 con 0x7fc1ac0fc7f0 2024-04-12T05:20:39.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.486+0000 7fc1a3fff700 1 -- 172.21.15.113:0/1730889244 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc19401d6e0 con 0x7fc1ac0fc7f0 2024-04-12T05:20:39.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.487+0000 7fc1a3fff700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc19806caa0 0x7fc19806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:39.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.487+0000 7fc1aa7fc700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc19806caa0 0x7fc19806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:39.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.487+0000 7fc1a3fff700 1 -- 172.21.15.113:0/1730889244 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc19408d060 con 0x7fc1ac0fc7f0 2024-04-12T05:20:39.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.487+0000 7fc1aa7fc700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc19806caa0 0x7fc19806ef60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fc1ac108860 tx=0x7fc19c009de0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:39.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.491+0000 7fc1a3fff700 1 -- 172.21.15.113:0/1730889244 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc19405abd0 con 0x7fc1ac0fc7f0 2024-04-12T05:20:39.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.690+0000 7fc1b196c700 1 -- 172.21.15.113:0/1730889244 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1ac0feb40 con 0x7fc19806caa0 2024-04-12T05:20:39.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.699+0000 7fc1a3fff700 1 -- 172.21.15.113:0/1730889244 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fc1ac0feb40 con 0x7fc19806caa0 2024-04-12T05:20:39.701 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (8m) 84s ago 19m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (5m) 84s ago 19m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (5m) 67s ago 19m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (8m) 84s ago 19m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (107s) 84s ago 14m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (88s) 84s ago 14m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (71s) 67s ago 14m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (14m) 67s ago 14m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (9m) 84s ago 20m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (9m) 67s ago 19m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (6m) 84s ago 21m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (6m) 67s ago 18m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (9m) 84s ago 19m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:20:39.702 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (9m) 67s ago 18m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:20:39.703 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (5m) 84s ago 18m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:20:39.703 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (4m) 84s ago 17m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:20:39.703 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (4m) 84s ago 17m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:20:39.703 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (3m) 67s ago 16m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:20:39.703 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (3m) 67s ago 16m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:20:39.703 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (2m) 67s ago 15m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:20:39.703 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (8m) 67s ago 18m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:20:39.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.701+0000 7fc1b196c700 1 -- 172.21.15.113:0/1730889244 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc19806caa0 msgr2=0x7fc19806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.701+0000 7fc1b196c700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc19806caa0 0x7fc19806ef60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fc1ac108860 tx=0x7fc19c009de0).stop 2024-04-12T05:20:39.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.701+0000 7fc1b196c700 1 -- 172.21.15.113:0/1730889244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1ac0fc7f0 msgr2=0x7fc1ac10cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.701+0000 7fc1b196c700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1ac0fc7f0 0x7fc1ac10cf20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fc194005010 tx=0x7fc194009cb0).stop 2024-04-12T05:20:39.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.701+0000 7fc1b196c700 1 -- 172.21.15.113:0/1730889244 shutdown_connections 2024-04-12T05:20:39.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.701+0000 7fc1b196c700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc19806caa0 0x7fc19806ef60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.701+0000 7fc1b196c700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1ac0fc7f0 0x7fc1ac10cf20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.701+0000 7fc1b196c700 1 --2- 172.21.15.113:0/1730889244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1ac0fd150 0x7fc1ac10d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.701+0000 7fc1b196c700 1 -- 172.21.15.113:0/1730889244 >> 172.21.15.113:0/1730889244 conn(0x7fc1ac0f23b0 msgr2=0x7fc1ac0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:39.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.701+0000 7fc1b196c700 1 -- 172.21.15.113:0/1730889244 shutdown_connections 2024-04-12T05:20:39.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.701+0000 7fc1b196c700 1 -- 172.21.15.113:0/1730889244 wait complete. 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.798+0000 7efc70dd7700 1 -- 172.21.15.113:0/837934518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc6c0f91f0 msgr2=0x7efc6c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.798+0000 7efc70dd7700 1 --2- 172.21.15.113:0/837934518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc6c0f91f0 0x7efc6c101900 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7efc5c0047f0 tx=0x7efc5c009cb0).stop 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.798+0000 7efc70dd7700 1 -- 172.21.15.113:0/837934518 shutdown_connections 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.798+0000 7efc70dd7700 1 --2- 172.21.15.113:0/837934518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc6c0f91f0 0x7efc6c101900 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.798+0000 7efc70dd7700 1 --2- 172.21.15.113:0/837934518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc6c0f8890 0x7efc6c0f8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.798+0000 7efc70dd7700 1 -- 172.21.15.113:0/837934518 >> 172.21.15.113:0/837934518 conn(0x7efc6c0f23c0 msgr2=0x7efc6c0f47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.798+0000 7efc70dd7700 1 -- 172.21.15.113:0/837934518 shutdown_connections 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.798+0000 7efc70dd7700 1 -- 172.21.15.113:0/837934518 wait complete. 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc70dd7700 1 Processor -- start 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc70dd7700 1 -- start start 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc70dd7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc6c0f8890 0x7efc6c10e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:39.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc70dd7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc6c0f91f0 0x7efc6c10e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc70dd7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c10eb50 con 0x7efc6c0f91f0 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc70dd7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c10ec90 con 0x7efc6c0f8890 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc69d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc6c0f91f0 0x7efc6c10e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc69d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc6c0f91f0 0x7efc6c10e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47204/0 (socket says 172.21.15.113:47204) 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc69d9b700 1 -- 172.21.15.113:0/1446766736 learned_addr learned my addr 172.21.15.113:0/1446766736 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc6a59c700 1 --2- 172.21.15.113:0/1446766736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc6c0f8890 0x7efc6c10e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc69d9b700 1 -- 172.21.15.113:0/1446766736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc6c0f8890 msgr2=0x7efc6c10e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc69d9b700 1 --2- 172.21.15.113:0/1446766736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc6c0f8890 0x7efc6c10e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc69d9b700 1 -- 172.21.15.113:0/1446766736 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c005040 con 0x7efc6c0f91f0 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc69d9b700 1 --2- 172.21.15.113:0/1446766736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc6c0f91f0 0x7efc6c10e550 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7efc5c005010 tx=0x7efc5c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc5b7fe700 1 -- 172.21.15.113:0/1446766736 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc5c00e030 con 0x7efc6c0f91f0 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.799+0000 7efc70dd7700 1 -- 172.21.15.113:0/1446766736 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6c1082c0 con 0x7efc6c0f91f0 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.800+0000 7efc5b7fe700 1 -- 172.21.15.113:0/1446766736 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc5c007e90 con 0x7efc6c0f91f0 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.800+0000 7efc5b7fe700 1 -- 172.21.15.113:0/1446766736 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc5c018070 con 0x7efc6c0f91f0 2024-04-12T05:20:39.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.800+0000 7efc70dd7700 1 -- 172.21.15.113:0/1446766736 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc6c1087a0 con 0x7efc6c0f91f0 2024-04-12T05:20:39.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.800+0000 7efc70dd7700 1 -- 172.21.15.113:0/1446766736 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4c005150 con 0x7efc6c0f91f0 2024-04-12T05:20:39.804 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.801+0000 7efc5b7fe700 1 -- 172.21.15.113:0/1446766736 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7efc5c009590 con 0x7efc6c0f91f0 2024-04-12T05:20:39.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.801+0000 7efc5b7fe700 1 --2- 172.21.15.113:0/1446766736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc5406caa0 0x7efc5406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:39.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.801+0000 7efc5b7fe700 1 -- 172.21.15.113:0/1446766736 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7efc5c08d010 con 0x7efc6c0f91f0 2024-04-12T05:20:39.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.802+0000 7efc6a59c700 1 --2- 172.21.15.113:0/1446766736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc5406caa0 0x7efc5406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:39.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.802+0000 7efc6a59c700 1 --2- 172.21.15.113:0/1446766736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc5406caa0 0x7efc5406ef60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7efc60004de0 tx=0x7efc6000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:39.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:39.806+0000 7efc5b7fe700 1 -- 172.21.15.113:0/1446766736 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7efc5c05aac0 con 0x7efc6c0f91f0 2024-04-12T05:20:40.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.076+0000 7efc70dd7700 1 -- 172.21.15.113:0/1446766736 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efc4c006030 con 0x7efc6c0f91f0 2024-04-12T05:20:40.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.076+0000 7efc5b7fe700 1 -- 172.21.15.113:0/1446766736 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7efc5c05a650 con 0x7efc6c0f91f0 2024-04-12T05:20:40.079 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:20:40.079 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:20:40.079 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:20:40.079 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:20:40.080 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:20:40.080 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:20:40.080 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:20:40.080 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:20:40.080 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:20:40.080 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:20:40.080 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:20:40.080 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:20:40.080 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:20:40.081 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:20:40.081 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:20:40.081 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:20:40.081 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:20:40.081 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:20:40.081 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:20:40.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.079+0000 7efc70dd7700 1 -- 172.21.15.113:0/1446766736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc5406caa0 msgr2=0x7efc5406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.079+0000 7efc70dd7700 1 --2- 172.21.15.113:0/1446766736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc5406caa0 0x7efc5406ef60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7efc60004de0 tx=0x7efc6000b040).stop 2024-04-12T05:20:40.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.079+0000 7efc70dd7700 1 -- 172.21.15.113:0/1446766736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc6c0f91f0 msgr2=0x7efc6c10e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.079+0000 7efc70dd7700 1 --2- 172.21.15.113:0/1446766736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc6c0f91f0 0x7efc6c10e550 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7efc5c005010 tx=0x7efc5c009cb0).stop 2024-04-12T05:20:40.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.079+0000 7efc70dd7700 1 -- 172.21.15.113:0/1446766736 shutdown_connections 2024-04-12T05:20:40.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.079+0000 7efc70dd7700 1 --2- 172.21.15.113:0/1446766736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc5406caa0 0x7efc5406ef60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.079+0000 7efc70dd7700 1 --2- 172.21.15.113:0/1446766736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc6c0f8890 0x7efc6c10e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.079+0000 7efc70dd7700 1 --2- 172.21.15.113:0/1446766736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc6c0f91f0 0x7efc6c10e550 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.079+0000 7efc70dd7700 1 -- 172.21.15.113:0/1446766736 >> 172.21.15.113:0/1446766736 conn(0x7efc6c0f23c0 msgr2=0x7efc6c0fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:40.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.079+0000 7efc70dd7700 1 -- 172.21.15.113:0/1446766736 shutdown_connections 2024-04-12T05:20:40.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.079+0000 7efc70dd7700 1 -- 172.21.15.113:0/1446766736 wait complete. 2024-04-12T05:20:40.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.176+0000 7f2a5eda1700 1 -- 172.21.15.113:0/3698373478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a580fd150 msgr2=0x7f2a580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.176+0000 7f2a5eda1700 1 --2- 172.21.15.113:0/3698373478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a580fd150 0x7f2a580f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f2a440047f0 tx=0x7f2a44009cb0).stop 2024-04-12T05:20:40.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.176+0000 7f2a5eda1700 1 -- 172.21.15.113:0/3698373478 shutdown_connections 2024-04-12T05:20:40.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.176+0000 7f2a5eda1700 1 --2- 172.21.15.113:0/3698373478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a580fd150 0x7f2a580f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.176+0000 7f2a5eda1700 1 --2- 172.21.15.113:0/3698373478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a580fc7f0 0x7f2a580fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.176+0000 7f2a5eda1700 1 -- 172.21.15.113:0/3698373478 >> 172.21.15.113:0/3698373478 conn(0x7f2a580f23b0 msgr2=0x7f2a580f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:40.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.176+0000 7f2a5eda1700 1 -- 172.21.15.113:0/3698373478 shutdown_connections 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.176+0000 7f2a5eda1700 1 -- 172.21.15.113:0/3698373478 wait complete. 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.177+0000 7f2a5eda1700 1 Processor -- start 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.177+0000 7f2a5eda1700 1 -- start start 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.177+0000 7f2a5eda1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a580fc7f0 0x7f2a58109100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.177+0000 7f2a5eda1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a580fd150 0x7f2a581095f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.177+0000 7f2a5eda1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a58109ae0 con 0x7f2a580fd150 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.177+0000 7f2a5eda1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a58109c20 con 0x7f2a580fc7f0 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.177+0000 7f2a57fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a580fd150 0x7f2a581095f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.177+0000 7f2a57fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a580fd150 0x7f2a581095f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:47208/0 (socket says 172.21.15.113:47208) 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.177+0000 7f2a57fff700 1 -- 172.21.15.113:0/2283044293 learned_addr learned my addr 172.21.15.113:0/2283044293 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.177+0000 7f2a5cb3d700 1 --2- 172.21.15.113:0/2283044293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a580fc7f0 0x7f2a58109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.178+0000 7f2a57fff700 1 -- 172.21.15.113:0/2283044293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a580fc7f0 msgr2=0x7f2a58109100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.178+0000 7f2a57fff700 1 --2- 172.21.15.113:0/2283044293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a580fc7f0 0x7f2a58109100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.178+0000 7f2a57fff700 1 -- 172.21.15.113:0/2283044293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a44005040 con 0x7f2a580fd150 2024-04-12T05:20:40.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.178+0000 7f2a57fff700 1 --2- 172.21.15.113:0/2283044293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a580fd150 0x7f2a581095f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f2a440099f0 tx=0x7f2a44009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:40.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.178+0000 7f2a55ffb700 1 -- 172.21.15.113:0/2283044293 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a4400e030 con 0x7f2a580fd150 2024-04-12T05:20:40.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.178+0000 7f2a55ffb700 1 -- 172.21.15.113:0/2283044293 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a44007e90 con 0x7f2a580fd150 2024-04-12T05:20:40.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.178+0000 7f2a5eda1700 1 -- 172.21.15.113:0/2283044293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a58109e70 con 0x7f2a580fd150 2024-04-12T05:20:40.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.178+0000 7f2a55ffb700 1 -- 172.21.15.113:0/2283044293 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a44018070 con 0x7f2a580fd150 2024-04-12T05:20:40.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.178+0000 7f2a5eda1700 1 -- 172.21.15.113:0/2283044293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a58112460 con 0x7f2a580fd150 2024-04-12T05:20:40.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.179+0000 7f2a5eda1700 1 -- 172.21.15.113:0/2283044293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a58060390 con 0x7f2a580fd150 2024-04-12T05:20:40.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.180+0000 7f2a55ffb700 1 -- 172.21.15.113:0/2283044293 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2a440094c0 con 0x7f2a580fd150 2024-04-12T05:20:40.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.180+0000 7f2a55ffb700 1 --2- 172.21.15.113:0/2283044293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a4806caa0 0x7f2a4806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:40.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.180+0000 7f2a55ffb700 1 -- 172.21.15.113:0/2283044293 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2a4408d0c0 con 0x7f2a580fd150 2024-04-12T05:20:40.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.180+0000 7f2a5cb3d700 1 --2- 172.21.15.113:0/2283044293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a4806caa0 0x7f2a4806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:40.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.180+0000 7f2a5cb3d700 1 --2- 172.21.15.113:0/2283044293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a4806caa0 0x7f2a4806ef60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2a4c004dc0 tx=0x7f2a4c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:40.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.184+0000 7f2a55ffb700 1 -- 172.21.15.113:0/2283044293 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2a440570e0 con 0x7f2a580fd150 2024-04-12T05:20:40.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:39 smithi192 ceph-mon[85304]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.412+0000 7f2a5eda1700 1 -- 172.21.15.113:0/2283044293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2a58047540 con 0x7f2a580fd150 2024-04-12T05:20:40.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.412+0000 7f2a55ffb700 1 -- 172.21.15.113:0/2283044293 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f2a4405a700 con 0x7f2a580fd150 2024-04-12T05:20:40.415 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:20:40.415 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:20:40.415 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:20:40.415 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:20:40.415 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:20:40.415 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:20:40.415 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:20:40.415 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:20:40.415 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:20:40.415 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:20:40.416 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:20:40.417 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:20:40.418 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:20:40.418 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:20:40.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.414+0000 7f2a5eda1700 1 -- 172.21.15.113:0/2283044293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a4806caa0 msgr2=0x7f2a4806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.414+0000 7f2a5eda1700 1 --2- 172.21.15.113:0/2283044293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a4806caa0 0x7f2a4806ef60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2a4c004dc0 tx=0x7f2a4c00b040).stop 2024-04-12T05:20:40.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.414+0000 7f2a5eda1700 1 -- 172.21.15.113:0/2283044293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a580fd150 msgr2=0x7f2a581095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.415+0000 7f2a5eda1700 1 --2- 172.21.15.113:0/2283044293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a580fd150 0x7f2a581095f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f2a440099f0 tx=0x7f2a44009cb0).stop 2024-04-12T05:20:40.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.415+0000 7f2a5eda1700 1 -- 172.21.15.113:0/2283044293 shutdown_connections 2024-04-12T05:20:40.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.415+0000 7f2a5eda1700 1 --2- 172.21.15.113:0/2283044293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a4806caa0 0x7f2a4806ef60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.415+0000 7f2a5eda1700 1 --2- 172.21.15.113:0/2283044293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a580fc7f0 0x7f2a58109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.415+0000 7f2a5eda1700 1 --2- 172.21.15.113:0/2283044293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a580fd150 0x7f2a581095f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.415+0000 7f2a5eda1700 1 -- 172.21.15.113:0/2283044293 >> 172.21.15.113:0/2283044293 conn(0x7f2a580f23b0 msgr2=0x7f2a580fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:40.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.415+0000 7f2a5eda1700 1 -- 172.21.15.113:0/2283044293 shutdown_connections 2024-04-12T05:20:40.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.415+0000 7f2a5eda1700 1 -- 172.21.15.113:0/2283044293 wait complete. 2024-04-12T05:20:40.419 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:20:40.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:39 smithi113 ceph-mon[103371]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.514+0000 7ffbc1a42700 1 -- 172.21.15.113:0/3787305407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbbc0fda00 msgr2=0x7ffbbc0fdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.514+0000 7ffbc1a42700 1 --2- 172.21.15.113:0/3787305407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbbc0fda00 0x7ffbbc0fdde0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7ffbb00047f0 tx=0x7ffbb0009cb0).stop 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.514+0000 7ffbc1a42700 1 -- 172.21.15.113:0/3787305407 shutdown_connections 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.514+0000 7ffbc1a42700 1 --2- 172.21.15.113:0/3787305407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbbc0fe360 0x7ffbbc0f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.514+0000 7ffbc1a42700 1 --2- 172.21.15.113:0/3787305407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbbc0fda00 0x7ffbbc0fdde0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.514+0000 7ffbc1a42700 1 -- 172.21.15.113:0/3787305407 >> 172.21.15.113:0/3787305407 conn(0x7ffbbc06cea0 msgr2=0x7ffbbc06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.514+0000 7ffbc1a42700 1 -- 172.21.15.113:0/3787305407 shutdown_connections 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.514+0000 7ffbc1a42700 1 -- 172.21.15.113:0/3787305407 wait complete. 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.514+0000 7ffbc1a42700 1 Processor -- start 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.514+0000 7ffbc1a42700 1 -- start start 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbc1a42700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbbc0fda00 0x7ffbbc10e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbc1a42700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbbc0fe360 0x7ffbbc10e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:40.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbc1a42700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc10ec20 con 0x7ffbbc0fe360 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbc1a42700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc10ed60 con 0x7ffbbc0fda00 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbba7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbbc0fe360 0x7ffbbc10e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbba7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbbc0fe360 0x7ffbbc10e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51532/0 (socket says 172.21.15.113:51532) 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbba7fc700 1 -- 172.21.15.113:0/2089816448 learned_addr learned my addr 172.21.15.113:0/2089816448 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbbaffd700 1 --2- 172.21.15.113:0/2089816448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbbc0fda00 0x7ffbbc10e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbba7fc700 1 -- 172.21.15.113:0/2089816448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbbc0fda00 msgr2=0x7ffbbc10e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbba7fc700 1 --2- 172.21.15.113:0/2089816448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbbc0fda00 0x7ffbbc10e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbba7fc700 1 -- 172.21.15.113:0/2089816448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb0005040 con 0x7ffbbc0fe360 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbba7fc700 1 --2- 172.21.15.113:0/2089816448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbbc0fe360 0x7ffbbc10e620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7ffbac00e910 tx=0x7ffbac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffba3fff700 1 -- 172.21.15.113:0/2089816448 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbac00eeb0 con 0x7ffbbc0fe360 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffba3fff700 1 -- 172.21.15.113:0/2089816448 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffbac00f040 con 0x7ffbbc0fe360 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffba3fff700 1 -- 172.21.15.113:0/2089816448 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbac00c8a0 con 0x7ffbbc0fe360 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbc1a42700 1 -- 172.21.15.113:0/2089816448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc108390 con 0x7ffbbc0fe360 2024-04-12T05:20:40.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.515+0000 7ffbc1a42700 1 -- 172.21.15.113:0/2089816448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc108950 con 0x7ffbbc0fe360 2024-04-12T05:20:40.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.516+0000 7ffbc1a42700 1 -- 172.21.15.113:0/2089816448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc047540 con 0x7ffbbc0fe360 2024-04-12T05:20:40.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.517+0000 7ffba3fff700 1 -- 172.21.15.113:0/2089816448 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ffbac0090d0 con 0x7ffbbc0fe360 2024-04-12T05:20:40.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.517+0000 7ffba3fff700 1 --2- 172.21.15.113:0/2089816448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffba406caa0 0x7ffba406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:40.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.517+0000 7ffbbaffd700 1 --2- 172.21.15.113:0/2089816448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffba406caa0 0x7ffba406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:40.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.517+0000 7ffba3fff700 1 -- 172.21.15.113:0/2089816448 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ffbac08b920 con 0x7ffbbc0fe360 2024-04-12T05:20:40.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.518+0000 7ffbbaffd700 1 --2- 172.21.15.113:0/2089816448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffba406caa0 0x7ffba406ef60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ffbb0004e60 tx=0x7ffbb000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:40.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.522+0000 7ffba3fff700 1 -- 172.21.15.113:0/2089816448 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ffbac0593d0 con 0x7ffbbc0fe360 2024-04-12T05:20:40.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.730+0000 7ffbc1a42700 1 -- 172.21.15.113:0/2089816448 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbbc0ffd80 con 0x7ffba406caa0 2024-04-12T05:20:40.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.731+0000 7ffba3fff700 1 -- 172.21.15.113:0/2089816448 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ffbbc0ffd80 con 0x7ffba406caa0 2024-04-12T05:20:40.733 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:20:40.733 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:20:40.733 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:20:40.733 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:20:40.733 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:20:40.733 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:20:40.733 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:20:40.734 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:20:40.734 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:20:40.734 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:20:40.734 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:20:40.734 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:20:40.734 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:20:40.734 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:20:40.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.733+0000 7ffbc1a42700 1 -- 172.21.15.113:0/2089816448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffba406caa0 msgr2=0x7ffba406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.733+0000 7ffbc1a42700 1 --2- 172.21.15.113:0/2089816448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffba406caa0 0x7ffba406ef60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ffbb0004e60 tx=0x7ffbb000b040).stop 2024-04-12T05:20:40.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.733+0000 7ffbc1a42700 1 -- 172.21.15.113:0/2089816448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbbc0fe360 msgr2=0x7ffbbc10e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.733+0000 7ffbc1a42700 1 --2- 172.21.15.113:0/2089816448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbbc0fe360 0x7ffbbc10e620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7ffbac00e910 tx=0x7ffbac00b040).stop 2024-04-12T05:20:40.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.733+0000 7ffbc1a42700 1 -- 172.21.15.113:0/2089816448 shutdown_connections 2024-04-12T05:20:40.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.733+0000 7ffbc1a42700 1 --2- 172.21.15.113:0/2089816448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffba406caa0 0x7ffba406ef60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.733+0000 7ffbc1a42700 1 --2- 172.21.15.113:0/2089816448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbbc0fda00 0x7ffbbc10e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.733+0000 7ffbc1a42700 1 --2- 172.21.15.113:0/2089816448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffbbc0fe360 0x7ffbbc10e620 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.733+0000 7ffbc1a42700 1 -- 172.21.15.113:0/2089816448 >> 172.21.15.113:0/2089816448 conn(0x7ffbbc06cea0 msgr2=0x7ffbbc0fbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:40.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.733+0000 7ffbc1a42700 1 -- 172.21.15.113:0/2089816448 shutdown_connections 2024-04-12T05:20:40.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.733+0000 7ffbc1a42700 1 -- 172.21.15.113:0/2089816448 wait complete. 2024-04-12T05:20:40.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.831+0000 7feead110700 1 -- 172.21.15.113:0/200619662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feea80fc7f0 msgr2=0x7feea80fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.831+0000 7feead110700 1 --2- 172.21.15.113:0/200619662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feea80fc7f0 0x7feea80fcbd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fee900047f0 tx=0x7fee90009cb0).stop 2024-04-12T05:20:40.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.831+0000 7feead110700 1 -- 172.21.15.113:0/200619662 shutdown_connections 2024-04-12T05:20:40.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.831+0000 7feead110700 1 --2- 172.21.15.113:0/200619662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feea80fd150 0x7feea80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.831+0000 7feead110700 1 --2- 172.21.15.113:0/200619662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feea80fc7f0 0x7feea80fcbd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.831+0000 7feead110700 1 -- 172.21.15.113:0/200619662 >> 172.21.15.113:0/200619662 conn(0x7feea8070590 msgr2=0x7feea80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:40.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.831+0000 7feead110700 1 -- 172.21.15.113:0/200619662 shutdown_connections 2024-04-12T05:20:40.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.831+0000 7feead110700 1 -- 172.21.15.113:0/200619662 wait complete. 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.831+0000 7feead110700 1 Processor -- start 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feead110700 1 -- start start 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feead110700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feea80fc7f0 0x7feea810e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feead110700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feea80fd150 0x7feea810e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feead110700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea810ec20 con 0x7feea80fc7f0 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feead110700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea810ed60 con 0x7feea80fd150 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feea6d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feea80fc7f0 0x7feea810e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feea6d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feea80fc7f0 0x7feea810e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51548/0 (socket says 172.21.15.113:51548) 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feea6d9d700 1 -- 172.21.15.113:0/1740614377 learned_addr learned my addr 172.21.15.113:0/1740614377 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feea659c700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feea80fd150 0x7feea810e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feea6d9d700 1 -- 172.21.15.113:0/1740614377 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feea80fd150 msgr2=0x7feea810e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:40.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feea6d9d700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feea80fd150 0x7feea810e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:40.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feea6d9d700 1 -- 172.21.15.113:0/1740614377 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee90005040 con 0x7feea80fc7f0 2024-04-12T05:20:40.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feea659c700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feea80fd150 0x7feea810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:20:40.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.832+0000 7feea6d9d700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feea80fc7f0 0x7feea810e130 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fee90009380 tx=0x7fee90009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:40.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.833+0000 7fee9ffff700 1 -- 172.21.15.113:0/1740614377 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee90018070 con 0x7feea80fc7f0 2024-04-12T05:20:40.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.833+0000 7fee9ffff700 1 -- 172.21.15.113:0/1740614377 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee90007e90 con 0x7feea80fc7f0 2024-04-12T05:20:40.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.833+0000 7feead110700 1 -- 172.21.15.113:0/1740614377 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea8108390 con 0x7feea80fc7f0 2024-04-12T05:20:40.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.833+0000 7fee9ffff700 1 -- 172.21.15.113:0/1740614377 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee900123c0 con 0x7feea80fc7f0 2024-04-12T05:20:40.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.833+0000 7feead110700 1 -- 172.21.15.113:0/1740614377 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea81088f0 con 0x7feea80fc7f0 2024-04-12T05:20:40.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.834+0000 7feead110700 1 -- 172.21.15.113:0/1740614377 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea805f590 con 0x7feea80fc7f0 2024-04-12T05:20:40.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.834+0000 7fee9ffff700 1 -- 172.21.15.113:0/1740614377 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fee9001d420 con 0x7feea80fc7f0 2024-04-12T05:20:40.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.835+0000 7fee9ffff700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fee9406caa0 0x7fee9406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:20:40.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.835+0000 7fee9ffff700 1 -- 172.21.15.113:0/1740614377 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fee9008cad0 con 0x7feea80fc7f0 2024-04-12T05:20:40.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.835+0000 7feea659c700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fee9406caa0 0x7fee9406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:20:40.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.835+0000 7feea659c700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fee9406caa0 0x7fee9406ef60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fee98009f20 tx=0x7fee9800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:20:40.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:40.839+0000 7fee9ffff700 1 -- 172.21.15.113:0/1740614377 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fee9005a580 con 0x7feea80fc7f0 2024-04-12T05:20:41.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.105+0000 7feead110700 1 -- 172.21.15.113:0/1740614377 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7feea8047540 con 0x7feea80fc7f0 2024-04-12T05:20:41.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.105+0000 7fee9ffff700 1 -- 172.21.15.113:0/1740614377 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fee9005a110 con 0x7feea80fc7f0 2024-04-12T05:20:41.107 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:20:41.108 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:20:41.108 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:20:41.108 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:20:41.108 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:20:41.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.107+0000 7feead110700 1 -- 172.21.15.113:0/1740614377 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fee9406caa0 msgr2=0x7fee9406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:41.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.107+0000 7feead110700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fee9406caa0 0x7fee9406ef60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fee98009f20 tx=0x7fee9800b040).stop 2024-04-12T05:20:41.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.107+0000 7feead110700 1 -- 172.21.15.113:0/1740614377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feea80fc7f0 msgr2=0x7feea810e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:20:41.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.107+0000 7feead110700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feea80fc7f0 0x7feea810e130 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fee90009380 tx=0x7fee90009cb0).stop 2024-04-12T05:20:41.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.107+0000 7feead110700 1 -- 172.21.15.113:0/1740614377 shutdown_connections 2024-04-12T05:20:41.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.107+0000 7feead110700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fee9406caa0 0x7fee9406ef60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:41.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.107+0000 7feead110700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feea80fc7f0 0x7feea810e130 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:41.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.108+0000 7feead110700 1 --2- 172.21.15.113:0/1740614377 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feea80fd150 0x7feea810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:20:41.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.108+0000 7feead110700 1 -- 172.21.15.113:0/1740614377 >> 172.21.15.113:0/1740614377 conn(0x7feea8070590 msgr2=0x7feea8060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:20:41.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.108+0000 7feead110700 1 -- 172.21.15.113:0/1740614377 shutdown_connections 2024-04-12T05:20:41.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:20:41.108+0000 7feead110700 1 -- 172.21.15.113:0/1740614377 wait complete. 2024-04-12T05:20:41.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:40 smithi192 ceph-mon[85304]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:41.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:40 smithi192 ceph-mon[85304]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:41.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:40 smithi192 ceph-mon[85304]: pgmap v255: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:41.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:40 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1446766736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:20:41.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:40 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2283044293' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:20:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:40 smithi113 ceph-mon[103371]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:40 smithi113 ceph-mon[103371]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:40 smithi113 ceph-mon[103371]: pgmap v255: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:40 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1446766736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:20:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:40 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2283044293' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:20:42.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:41 smithi192 ceph-mon[85304]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:42.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:41 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1740614377' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:20:42.270 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:41 smithi113 ceph-mon[103371]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:20:42.270 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:41 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1740614377' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:20:43.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:42 smithi192 ceph-mon[85304]: pgmap v256: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:42 smithi113 ceph-mon[103371]: pgmap v256: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:45.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:44 smithi192 ceph-mon[85304]: pgmap v257: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:44 smithi113 ceph-mon[103371]: pgmap v257: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:46 smithi113 ceph-mon[103371]: pgmap v258: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:47.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:46 smithi192 ceph-mon[85304]: pgmap v258: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:48 smithi113 ceph-mon[103371]: pgmap v259: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:49.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:48 smithi192 ceph-mon[85304]: pgmap v259: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:50 smithi113 ceph-mon[103371]: pgmap v260: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:50 smithi192 ceph-mon[85304]: pgmap v260: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:52.269 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:51 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:20:52.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:51 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:20:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:52 smithi113 ceph-mon[103371]: pgmap v261: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:52 smithi192 ceph-mon[85304]: pgmap v261: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:54 smithi113 ceph-mon[103371]: pgmap v262: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:55.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:54 smithi192 ceph-mon[85304]: pgmap v262: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:56 smithi113 ceph-mon[103371]: pgmap v263: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:57.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:56 smithi192 ceph-mon[85304]: pgmap v263: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:20:58 smithi113 ceph-mon[103371]: pgmap v264: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:20:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:20:58 smithi192 ceph-mon[85304]: pgmap v264: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:00 smithi113 ceph-mon[103371]: pgmap v265: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:01.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:00 smithi192 ceph-mon[85304]: pgmap v265: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:02 smithi113 ceph-mon[103371]: pgmap v266: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:02 smithi192 ceph-mon[85304]: pgmap v266: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:04 smithi113 ceph-mon[103371]: pgmap v267: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:04 smithi192 ceph-mon[85304]: pgmap v267: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:06 smithi113 ceph-mon[103371]: pgmap v268: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:21:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:06 smithi192 ceph-mon[85304]: pgmap v268: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:21:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:08 smithi113 ceph-mon[103371]: pgmap v269: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:08 smithi192 ceph-mon[85304]: pgmap v269: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:11.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.219+0000 7f84d8b06700 1 -- 172.21.15.113:0/491915129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84d40fd8b0 msgr2=0x7f84d40fdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:11.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.219+0000 7f84d8b06700 1 --2- 172.21.15.113:0/491915129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84d40fd8b0 0x7f84d40fdc90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f84c80047f0 tx=0x7f84c8009cb0).stop 2024-04-12T05:21:11.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.219+0000 7f84d8b06700 1 -- 172.21.15.113:0/491915129 shutdown_connections 2024-04-12T05:21:11.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.219+0000 7f84d8b06700 1 --2- 172.21.15.113:0/491915129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d40fe210 0x7f84d40f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:21:11.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.219+0000 7f84d8b06700 1 --2- 172.21.15.113:0/491915129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84d40fd8b0 0x7f84d40fdc90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.219+0000 7f84d8b06700 1 -- 172.21.15.113:0/491915129 >> 172.21.15.113:0/491915129 conn(0x7f84d406ce40 msgr2=0x7f84d406d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:11.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.219+0000 7f84d8b06700 1 -- 172.21.15.113:0/491915129 shutdown_connections 2024-04-12T05:21:11.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.219+0000 7f84d8b06700 1 -- 172.21.15.113:0/491915129 wait complete. 2024-04-12T05:21:11.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.220+0000 7f84d8b06700 1 Processor -- start 2024-04-12T05:21:11.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.220+0000 7f84d8b06700 1 -- start start 2024-04-12T05:21:11.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.220+0000 7f84d8b06700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d40fd8b0 0x7f84d410e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:11.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.220+0000 7f84d8b06700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84d40fe210 0x7f84d410e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:11.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.220+0000 7f84d8b06700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d410ec40 con 0x7f84d40fe210 2024-04-12T05:21:11.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.220+0000 7f84d8b06700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d4108260 con 0x7f84d40fd8b0 2024-04-12T05:21:11.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.220+0000 7f84d1d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84d40fe210 0x7f84d410e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:11.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.220+0000 7f84d1d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84d40fe210 0x7f84d410e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41330/0 (socket says 172.21.15.113:41330) 2024-04-12T05:21:11.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.220+0000 7f84d1d9b700 1 -- 172.21.15.113:0/15084790 learned_addr learned my addr 172.21.15.113:0/15084790 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:11.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.220+0000 7f84d259c700 1 --2- 172.21.15.113:0/15084790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d40fd8b0 0x7f84d410e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:11.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.221+0000 7f84d1d9b700 1 -- 172.21.15.113:0/15084790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d40fd8b0 msgr2=0x7f84d410e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:11.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.221+0000 7f84d1d9b700 1 --2- 172.21.15.113:0/15084790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d40fd8b0 0x7f84d410e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.221+0000 7f84d1d9b700 1 -- 172.21.15.113:0/15084790 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c8005040 con 0x7f84d40fe210 2024-04-12T05:21:11.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.221+0000 7f84d1d9b700 1 --2- 172.21.15.113:0/15084790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84d40fe210 0x7f84d410e5b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f84c400b4b0 tx=0x7f84c40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:11.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.221+0000 7f84c37fe700 1 -- 172.21.15.113:0/15084790 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84c4002500 con 0x7f84d40fe210 2024-04-12T05:21:11.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.221+0000 7f84c37fe700 1 -- 172.21.15.113:0/15084790 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84c4002b10 con 0x7f84d40fe210 2024-04-12T05:21:11.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.221+0000 7f84d8b06700 1 -- 172.21.15.113:0/15084790 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d4108510 con 0x7f84d40fe210 2024-04-12T05:21:11.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.221+0000 7f84c37fe700 1 -- 172.21.15.113:0/15084790 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84c4012cf0 con 0x7f84d40fe210 2024-04-12T05:21:11.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.221+0000 7f84d8b06700 1 -- 172.21.15.113:0/15084790 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d4108a50 con 0x7f84d40fe210 2024-04-12T05:21:11.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.222+0000 7f84d8b06700 1 -- 172.21.15.113:0/15084790 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d4060670 con 0x7f84d40fe210 2024-04-12T05:21:11.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.222+0000 7f84c37fe700 1 -- 172.21.15.113:0/15084790 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f84c4002660 con 0x7f84d40fe210 2024-04-12T05:21:11.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.223+0000 7f84c37fe700 1 --2- 172.21.15.113:0/15084790 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84bc06caa0 0x7f84bc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:11.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.223+0000 7f84c37fe700 1 -- 172.21.15.113:0/15084790 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f84c408bf30 con 0x7f84d40fe210 2024-04-12T05:21:11.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.223+0000 7f84d259c700 1 --2- 172.21.15.113:0/15084790 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84bc06caa0 0x7f84bc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:11.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.223+0000 7f84d259c700 1 --2- 172.21.15.113:0/15084790 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84bc06caa0 0x7f84bc06ef60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f84c8007cf0 tx=0x7f84c800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:11.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.227+0000 7f84c37fe700 1 -- 172.21.15.113:0/15084790 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f84c40bf2c0 con 0x7f84d40fe210 2024-04-12T05:21:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:10 smithi113 ceph-mon[103371]: pgmap v270: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:11.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.441+0000 7f84d8b06700 1 -- 172.21.15.113:0/15084790 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f84d41097b0 con 0x7f84bc06caa0 2024-04-12T05:21:11.444 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.442+0000 7f84c37fe700 1 -- 172.21.15.113:0/15084790 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f84d41097b0 con 0x7f84bc06caa0 2024-04-12T05:21:11.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.444+0000 7f84d8b06700 1 -- 172.21.15.113:0/15084790 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84bc06caa0 msgr2=0x7f84bc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:11.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.444+0000 7f84d8b06700 1 --2- 172.21.15.113:0/15084790 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84bc06caa0 0x7f84bc06ef60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f84c8007cf0 tx=0x7f84c800b040).stop 2024-04-12T05:21:11.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.444+0000 7f84d8b06700 1 -- 172.21.15.113:0/15084790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84d40fe210 msgr2=0x7f84d410e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:11.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.444+0000 7f84d8b06700 1 --2- 172.21.15.113:0/15084790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84d40fe210 0x7f84d410e5b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f84c400b4b0 tx=0x7f84c40090f0).stop 2024-04-12T05:21:11.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.444+0000 7f84d8b06700 1 -- 172.21.15.113:0/15084790 shutdown_connections 2024-04-12T05:21:11.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.444+0000 7f84d8b06700 1 --2- 172.21.15.113:0/15084790 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84bc06caa0 0x7f84bc06ef60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.444+0000 7f84d8b06700 1 --2- 172.21.15.113:0/15084790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d40fd8b0 0x7f84d410e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.444+0000 7f84d8b06700 1 --2- 172.21.15.113:0/15084790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84d40fe210 0x7f84d410e5b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.444+0000 7f84d8b06700 1 -- 172.21.15.113:0/15084790 >> 172.21.15.113:0/15084790 conn(0x7f84d406ce40 msgr2=0x7f84d4100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:11.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.445+0000 7f84d8b06700 1 -- 172.21.15.113:0/15084790 shutdown_connections 2024-04-12T05:21:11.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.445+0000 7f84d8b06700 1 -- 172.21.15.113:0/15084790 wait complete. 2024-04-12T05:21:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:10 smithi192 ceph-mon[85304]: pgmap v270: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:11.457 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:21:11.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.543+0000 7f7706cbd700 1 -- 172.21.15.113:0/3215704596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77000fd150 msgr2=0x7f77000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:11.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.543+0000 7f7706cbd700 1 --2- 172.21.15.113:0/3215704596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77000fd150 0x7f77000f6930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f76ec0047f0 tx=0x7f76ec009cb0).stop 2024-04-12T05:21:11.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.543+0000 7f7706cbd700 1 -- 172.21.15.113:0/3215704596 shutdown_connections 2024-04-12T05:21:11.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.543+0000 7f7706cbd700 1 --2- 172.21.15.113:0/3215704596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77000fd150 0x7f77000f6930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.543+0000 7f7706cbd700 1 --2- 172.21.15.113:0/3215704596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77000fc7f0 0x7f77000fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.543+0000 7f7706cbd700 1 -- 172.21.15.113:0/3215704596 >> 172.21.15.113:0/3215704596 conn(0x7f7700070590 msgr2=0x7f77000f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.544+0000 7f7706cbd700 1 -- 172.21.15.113:0/3215704596 shutdown_connections 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.544+0000 7f7706cbd700 1 -- 172.21.15.113:0/3215704596 wait complete. 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.544+0000 7f7706cbd700 1 Processor -- start 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.544+0000 7f7706cbd700 1 -- start start 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.544+0000 7f7706cbd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77000fc7f0 0x7f77000f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.544+0000 7f7706cbd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77000fd150 0x7f77000f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.544+0000 7f7706cbd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77000f8f20 con 0x7f77000fd150 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.544+0000 7f7706cbd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77000f9060 con 0x7f77000fc7f0 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.544+0000 7f76fffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77000fd150 0x7f77000f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.544+0000 7f76fffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77000fd150 0x7f77000f8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41354/0 (socket says 172.21.15.113:41354) 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.544+0000 7f76fffff700 1 -- 172.21.15.113:0/1789752637 learned_addr learned my addr 172.21.15.113:0/1789752637 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.545+0000 7f7704a59700 1 --2- 172.21.15.113:0/1789752637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77000fc7f0 0x7f77000f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:11.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.545+0000 7f76fffff700 1 -- 172.21.15.113:0/1789752637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77000fc7f0 msgr2=0x7f77000f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:11.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.545+0000 7f76fffff700 1 --2- 172.21.15.113:0/1789752637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77000fc7f0 0x7f77000f8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.545+0000 7f76fffff700 1 -- 172.21.15.113:0/1789752637 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec005040 con 0x7f77000fd150 2024-04-12T05:21:11.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.545+0000 7f76fffff700 1 --2- 172.21.15.113:0/1789752637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77000fd150 0x7f77000f8a30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f76ec000c00 tx=0x7f76ec009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:11.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.545+0000 7f76fdffb700 1 -- 172.21.15.113:0/1789752637 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76ec018070 con 0x7f77000fd150 2024-04-12T05:21:11.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.545+0000 7f76fdffb700 1 -- 172.21.15.113:0/1789752637 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76ec007e90 con 0x7f77000fd150 2024-04-12T05:21:11.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.545+0000 7f7706cbd700 1 -- 172.21.15.113:0/1789752637 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77000f92e0 con 0x7f77000fd150 2024-04-12T05:21:11.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.545+0000 7f76fdffb700 1 -- 172.21.15.113:0/1789752637 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76ec01de20 con 0x7f77000fd150 2024-04-12T05:21:11.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.545+0000 7f7706cbd700 1 -- 172.21.15.113:0/1789752637 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7700114710 con 0x7f77000fd150 2024-04-12T05:21:11.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.546+0000 7f7706cbd700 1 -- 172.21.15.113:0/1789752637 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770005f590 con 0x7f77000fd150 2024-04-12T05:21:11.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.547+0000 7f76fdffb700 1 -- 172.21.15.113:0/1789752637 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f76ec01d420 con 0x7f77000fd150 2024-04-12T05:21:11.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.547+0000 7f76fdffb700 1 --2- 172.21.15.113:0/1789752637 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f76f006ca30 0x7f76f006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:11.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.547+0000 7f76fdffb700 1 -- 172.21.15.113:0/1789752637 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f76ec08c990 con 0x7f77000fd150 2024-04-12T05:21:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.547+0000 7f7704a59700 1 --2- 172.21.15.113:0/1789752637 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f76f006ca30 0x7f76f006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:11.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.547+0000 7f7704a59700 1 --2- 172.21.15.113:0/1789752637 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f76f006ca30 0x7f76f006eef0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f76f4004ee0 tx=0x7f76f400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:11.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.551+0000 7f76fdffb700 1 -- 172.21.15.113:0/1789752637 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f76ec0569b0 con 0x7f77000fd150 2024-04-12T05:21:11.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.758+0000 7f7706cbd700 1 -- 172.21.15.113:0/1789752637 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f77000fb5a0 con 0x7f76f006ca30 2024-04-12T05:21:11.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.759+0000 7f76fdffb700 1 -- 172.21.15.113:0/1789752637 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f77000fb5a0 con 0x7f76f006ca30 2024-04-12T05:21:11.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.761+0000 7f7706cbd700 1 -- 172.21.15.113:0/1789752637 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f76f006ca30 msgr2=0x7f76f006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:11.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.761+0000 7f7706cbd700 1 --2- 172.21.15.113:0/1789752637 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f76f006ca30 0x7f76f006eef0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f76f4004ee0 tx=0x7f76f400b040).stop 2024-04-12T05:21:11.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.761+0000 7f7706cbd700 1 -- 172.21.15.113:0/1789752637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77000fd150 msgr2=0x7f77000f8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:11.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.761+0000 7f7706cbd700 1 --2- 172.21.15.113:0/1789752637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77000fd150 0x7f77000f8a30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f76ec000c00 tx=0x7f76ec009cb0).stop 2024-04-12T05:21:11.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.761+0000 7f7706cbd700 1 -- 172.21.15.113:0/1789752637 shutdown_connections 2024-04-12T05:21:11.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.761+0000 7f7706cbd700 1 --2- 172.21.15.113:0/1789752637 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f76f006ca30 0x7f76f006eef0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.761+0000 7f7706cbd700 1 --2- 172.21.15.113:0/1789752637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77000fc7f0 0x7f77000f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.761+0000 7f7706cbd700 1 --2- 172.21.15.113:0/1789752637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77000fd150 0x7f77000f8a30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.761+0000 7f7706cbd700 1 -- 172.21.15.113:0/1789752637 >> 172.21.15.113:0/1789752637 conn(0x7f7700070590 msgr2=0x7f7700060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:11.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.761+0000 7f7706cbd700 1 -- 172.21.15.113:0/1789752637 shutdown_connections 2024-04-12T05:21:11.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.761+0000 7f7706cbd700 1 -- 172.21.15.113:0/1789752637 wait complete. 2024-04-12T05:21:11.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.860+0000 7f0505703700 1 -- 172.21.15.113:0/1493136799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000fd150 msgr2=0x7f05000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:11.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.860+0000 7f0505703700 1 --2- 172.21.15.113:0/1493136799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000fd150 0x7f05000f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f04f00047f0 tx=0x7f04f0009cb0).stop 2024-04-12T05:21:11.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.860+0000 7f0505703700 1 -- 172.21.15.113:0/1493136799 shutdown_connections 2024-04-12T05:21:11.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.860+0000 7f0505703700 1 --2- 172.21.15.113:0/1493136799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000fd150 0x7f05000f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.860+0000 7f0505703700 1 --2- 172.21.15.113:0/1493136799 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05000fc7f0 0x7f05000fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.860+0000 7f0505703700 1 -- 172.21.15.113:0/1493136799 >> 172.21.15.113:0/1493136799 conn(0x7f05000f23b0 msgr2=0x7f05000f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:11.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.860+0000 7f0505703700 1 -- 172.21.15.113:0/1493136799 shutdown_connections 2024-04-12T05:21:11.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.860+0000 7f0505703700 1 -- 172.21.15.113:0/1493136799 wait complete. 2024-04-12T05:21:11.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f0505703700 1 Processor -- start 2024-04-12T05:21:11.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f0505703700 1 -- start start 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f0505703700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05000fc7f0 0x7f0500106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f0505703700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000fd150 0x7f05001073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f0505703700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0500061a90 con 0x7f05000fd150 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f0505703700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05001078b0 con 0x7f05000fc7f0 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f04fe7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000fd150 0x7f05001073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f04fe7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000fd150 0x7f05001073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41382/0 (socket says 172.21.15.113:41382) 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f04fe7fc700 1 -- 172.21.15.113:0/2759679639 learned_addr learned my addr 172.21.15.113:0/2759679639 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f04feffd700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05000fc7f0 0x7f0500106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f04fe7fc700 1 -- 172.21.15.113:0/2759679639 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05000fc7f0 msgr2=0x7f0500106ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.861+0000 7f04fe7fc700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05000fc7f0 0x7f0500106ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.862+0000 7f04fe7fc700 1 -- 172.21.15.113:0/2759679639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f0005040 con 0x7f05000fd150 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.862+0000 7f04feffd700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05000fc7f0 0x7f0500106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:21:11.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.862+0000 7f04fe7fc700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000fd150 0x7f05001073c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f04f00099f0 tx=0x7f04f0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:11.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.862+0000 7f04e7fff700 1 -- 172.21.15.113:0/2759679639 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04f000e030 con 0x7f05000fd150 2024-04-12T05:21:11.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.862+0000 7f0505703700 1 -- 172.21.15.113:0/2759679639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0500107b30 con 0x7f05000fd150 2024-04-12T05:21:11.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.862+0000 7f04e7fff700 1 -- 172.21.15.113:0/2759679639 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04f0007e90 con 0x7f05000fd150 2024-04-12T05:21:11.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.862+0000 7f04e7fff700 1 -- 172.21.15.113:0/2759679639 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04f0018070 con 0x7f05000fd150 2024-04-12T05:21:11.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.862+0000 7f0505703700 1 -- 172.21.15.113:0/2759679639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0500114700 con 0x7f05000fd150 2024-04-12T05:21:11.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.863+0000 7f0505703700 1 -- 172.21.15.113:0/2759679639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0500060370 con 0x7f05000fd150 2024-04-12T05:21:11.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.863+0000 7f04e7fff700 1 -- 172.21.15.113:0/2759679639 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f04f0009590 con 0x7f05000fd150 2024-04-12T05:21:11.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.864+0000 7f04e7fff700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04e806ca30 0x7f04e806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:11.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.864+0000 7f04e7fff700 1 -- 172.21.15.113:0/2759679639 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f04f008d0f0 con 0x7f05000fd150 2024-04-12T05:21:11.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.864+0000 7f04feffd700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04e806ca30 0x7f04e806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:11.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.864+0000 7f04feffd700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04e806ca30 0x7f04e806eef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f04f4009250 tx=0x7f04f4009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:11.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:11.868+0000 7f04e7fff700 1 -- 172.21.15.113:0/2759679639 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f04f005aba0 con 0x7f05000fd150 2024-04-12T05:21:12.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.068+0000 7f0505703700 1 -- 172.21.15.113:0/2759679639 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0500059120 con 0x7f04e806ca30 2024-04-12T05:21:12.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.077+0000 7f04e7fff700 1 -- 172.21.15.113:0/2759679639 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f0500059120 con 0x7f04e806ca30 2024-04-12T05:21:12.079 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (9m) 116s ago 20m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (6m) 116s ago 20m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (6m) 99s ago 19m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (8m) 116s ago 19m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (2m) 116s ago 14m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (2m) 116s ago 14m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (103s) 99s ago 14m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (14m) 99s ago 14m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (10m) 116s ago 21m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (9m) 99s ago 19m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (7m) 116s ago 21m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (7m) 99s ago 19m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (9m) 116s ago 19m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (9m) 99s ago 19m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (6m) 116s ago 18m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:21:12.080 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (5m) 116s ago 18m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:21:12.081 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (4m) 116s ago 17m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:21:12.081 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (4m) 99s ago 17m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:21:12.081 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (3m) 99s ago 16m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:21:12.081 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (2m) 99s ago 16m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:21:12.081 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (9m) 99s ago 19m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:21:12.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.079+0000 7f0505703700 1 -- 172.21.15.113:0/2759679639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04e806ca30 msgr2=0x7f04e806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.079+0000 7f0505703700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04e806ca30 0x7f04e806eef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f04f4009250 tx=0x7f04f4009da0).stop 2024-04-12T05:21:12.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.079+0000 7f0505703700 1 -- 172.21.15.113:0/2759679639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000fd150 msgr2=0x7f05001073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.079+0000 7f0505703700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000fd150 0x7f05001073c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f04f00099f0 tx=0x7f04f0009cb0).stop 2024-04-12T05:21:12.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.079+0000 7f0505703700 1 -- 172.21.15.113:0/2759679639 shutdown_connections 2024-04-12T05:21:12.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.079+0000 7f0505703700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04e806ca30 0x7f04e806eef0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.079+0000 7f0505703700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05000fc7f0 0x7f0500106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.079+0000 7f0505703700 1 --2- 172.21.15.113:0/2759679639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05000fd150 0x7f05001073c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.079+0000 7f0505703700 1 -- 172.21.15.113:0/2759679639 >> 172.21.15.113:0/2759679639 conn(0x7f05000f23b0 msgr2=0x7f05000fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:12.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.080+0000 7f0505703700 1 -- 172.21.15.113:0/2759679639 shutdown_connections 2024-04-12T05:21:12.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.080+0000 7f0505703700 1 -- 172.21.15.113:0/2759679639 wait complete. 2024-04-12T05:21:12.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.176+0000 7f518e928700 1 -- 172.21.15.113:0/3540556246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51880fc7f0 msgr2=0x7f51880fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.176+0000 7f518e928700 1 --2- 172.21.15.113:0/3540556246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51880fc7f0 0x7f51880fcbd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f517c0047f0 tx=0x7f517c009cb0).stop 2024-04-12T05:21:12.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.177+0000 7f518e928700 1 -- 172.21.15.113:0/3540556246 shutdown_connections 2024-04-12T05:21:12.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.177+0000 7f518e928700 1 --2- 172.21.15.113:0/3540556246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51880fd150 0x7f51880f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.177+0000 7f518e928700 1 --2- 172.21.15.113:0/3540556246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51880fc7f0 0x7f51880fcbd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.177+0000 7f518e928700 1 -- 172.21.15.113:0/3540556246 >> 172.21.15.113:0/3540556246 conn(0x7f5188070590 msgr2=0x7f51880f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:12.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.177+0000 7f518e928700 1 -- 172.21.15.113:0/3540556246 shutdown_connections 2024-04-12T05:21:12.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.177+0000 7f518e928700 1 -- 172.21.15.113:0/3540556246 wait complete. 2024-04-12T05:21:12.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.177+0000 7f518e928700 1 Processor -- start 2024-04-12T05:21:12.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.177+0000 7f518e928700 1 -- start start 2024-04-12T05:21:12.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f518e928700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51880fc7f0 0x7f518810e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:12.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f518e928700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51880fd150 0x7f518810e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:12.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f518e928700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518810ec10 con 0x7f51880fc7f0 2024-04-12T05:21:12.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f518e928700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518810ed50 con 0x7f51880fd150 2024-04-12T05:21:12.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f5187fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51880fc7f0 0x7f518810e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f5187fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51880fc7f0 0x7f518810e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41410/0 (socket says 172.21.15.113:41410) 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f5187fff700 1 -- 172.21.15.113:0/3119035545 learned_addr learned my addr 172.21.15.113:0/3119035545 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f51877fe700 1 --2- 172.21.15.113:0/3119035545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51880fd150 0x7f518810e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f5187fff700 1 -- 172.21.15.113:0/3119035545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51880fd150 msgr2=0x7f518810e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f5187fff700 1 --2- 172.21.15.113:0/3119035545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51880fd150 0x7f518810e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f5187fff700 1 -- 172.21.15.113:0/3119035545 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c005040 con 0x7f51880fc7f0 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f5187fff700 1 --2- 172.21.15.113:0/3119035545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51880fc7f0 0x7f518810e120 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f517c009380 tx=0x7f517c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f51857fa700 1 -- 172.21.15.113:0/3119035545 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f517c018070 con 0x7f51880fc7f0 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f51857fa700 1 -- 172.21.15.113:0/3119035545 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f517c007e90 con 0x7f51880fc7f0 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f51857fa700 1 -- 172.21.15.113:0/3119035545 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f517c0123c0 con 0x7f51880fc7f0 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f518e928700 1 -- 172.21.15.113:0/3119035545 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51881083c0 con 0x7f51880fc7f0 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.178+0000 7f518e928700 1 -- 172.21.15.113:0/3119035545 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5188108920 con 0x7f51880fc7f0 2024-04-12T05:21:12.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.179+0000 7f518e928700 1 -- 172.21.15.113:0/3119035545 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518805f590 con 0x7f51880fc7f0 2024-04-12T05:21:12.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.180+0000 7f51857fa700 1 -- 172.21.15.113:0/3119035545 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f517c01d420 con 0x7f51880fc7f0 2024-04-12T05:21:12.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.180+0000 7f51857fa700 1 --2- 172.21.15.113:0/3119035545 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f517006caa0 0x7f517006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:12.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.180+0000 7f51857fa700 1 -- 172.21.15.113:0/3119035545 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f517c08cb20 con 0x7f51880fc7f0 2024-04-12T05:21:12.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.180+0000 7f51877fe700 1 --2- 172.21.15.113:0/3119035545 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f517006caa0 0x7f517006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:12.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.180+0000 7f51877fe700 1 --2- 172.21.15.113:0/3119035545 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f517006caa0 0x7f517006ef60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5178009f20 tx=0x7f517800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:12.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.185+0000 7f51857fa700 1 -- 172.21.15.113:0/3119035545 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f517c056b40 con 0x7f51880fc7f0 2024-04-12T05:21:12.453 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.451+0000 7f518e928700 1 -- 172.21.15.113:0/3119035545 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5188047540 con 0x7f51880fc7f0 2024-04-12T05:21:12.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.452+0000 7f51857fa700 1 -- 172.21.15.113:0/3119035545 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f517c05a160 con 0x7f51880fc7f0 2024-04-12T05:21:12.454 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:21:12.454 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:21:12.454 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:21:12.454 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:21:12.454 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:21:12.454 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:21:12.455 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:21:12.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.454+0000 7f518e928700 1 -- 172.21.15.113:0/3119035545 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f517006caa0 msgr2=0x7f517006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.454+0000 7f518e928700 1 --2- 172.21.15.113:0/3119035545 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f517006caa0 0x7f517006ef60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5178009f20 tx=0x7f517800b040).stop 2024-04-12T05:21:12.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.454+0000 7f518e928700 1 -- 172.21.15.113:0/3119035545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51880fc7f0 msgr2=0x7f518810e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.454+0000 7f518e928700 1 --2- 172.21.15.113:0/3119035545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51880fc7f0 0x7f518810e120 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f517c009380 tx=0x7f517c009cb0).stop 2024-04-12T05:21:12.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.454+0000 7f518e928700 1 -- 172.21.15.113:0/3119035545 shutdown_connections 2024-04-12T05:21:12.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.454+0000 7f518e928700 1 --2- 172.21.15.113:0/3119035545 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f517006caa0 0x7f517006ef60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.454+0000 7f518e928700 1 --2- 172.21.15.113:0/3119035545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51880fc7f0 0x7f518810e120 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.454+0000 7f518e928700 1 --2- 172.21.15.113:0/3119035545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51880fd150 0x7f518810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.454+0000 7f518e928700 1 -- 172.21.15.113:0/3119035545 >> 172.21.15.113:0/3119035545 conn(0x7f5188070590 msgr2=0x7f5188060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:12.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.454+0000 7f518e928700 1 -- 172.21.15.113:0/3119035545 shutdown_connections 2024-04-12T05:21:12.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.454+0000 7f518e928700 1 -- 172.21.15.113:0/3119035545 wait complete. 2024-04-12T05:21:12.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.552+0000 7f98119bb700 1 -- 172.21.15.113:0/2580595362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f980c0fc7f0 msgr2=0x7f980c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.552+0000 7f98119bb700 1 --2- 172.21.15.113:0/2580595362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f980c0fc7f0 0x7f980c0fcbd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f98000047f0 tx=0x7f9800009cb0).stop 2024-04-12T05:21:12.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.552+0000 7f98119bb700 1 -- 172.21.15.113:0/2580595362 shutdown_connections 2024-04-12T05:21:12.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.552+0000 7f98119bb700 1 --2- 172.21.15.113:0/2580595362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f980c0fd150 0x7f980c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.552+0000 7f98119bb700 1 --2- 172.21.15.113:0/2580595362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f980c0fc7f0 0x7f980c0fcbd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.552+0000 7f98119bb700 1 -- 172.21.15.113:0/2580595362 >> 172.21.15.113:0/2580595362 conn(0x7f980c070590 msgr2=0x7f980c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:12.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.552+0000 7f98119bb700 1 -- 172.21.15.113:0/2580595362 shutdown_connections 2024-04-12T05:21:12.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.552+0000 7f98119bb700 1 -- 172.21.15.113:0/2580595362 wait complete. 2024-04-12T05:21:12.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.553+0000 7f98119bb700 1 Processor -- start 2024-04-12T05:21:12.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.553+0000 7f98119bb700 1 -- start start 2024-04-12T05:21:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.553+0000 7f98119bb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f980c0fc7f0 0x7f980c1090f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.553+0000 7f98119bb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f980c0fd150 0x7f980c1095e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.553+0000 7f98119bb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c109ad0 con 0x7f980c0fd150 2024-04-12T05:21:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.553+0000 7f98119bb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c109c10 con 0x7f980c0fc7f0 2024-04-12T05:21:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.553+0000 7f980a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f980c0fd150 0x7f980c1095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.553+0000 7f980a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f980c0fd150 0x7f980c1095e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41432/0 (socket says 172.21.15.113:41432) 2024-04-12T05:21:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.553+0000 7f980a7fc700 1 -- 172.21.15.113:0/1492057369 learned_addr learned my addr 172.21.15.113:0/1492057369 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.553+0000 7f980affd700 1 --2- 172.21.15.113:0/1492057369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f980c0fc7f0 0x7f980c1090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.554+0000 7f980a7fc700 1 -- 172.21.15.113:0/1492057369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f980c0fc7f0 msgr2=0x7f980c1090f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.554+0000 7f980a7fc700 1 --2- 172.21.15.113:0/1492057369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f980c0fc7f0 0x7f980c1090f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.554+0000 7f980a7fc700 1 -- 172.21.15.113:0/1492057369 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9800005040 con 0x7f980c0fd150 2024-04-12T05:21:12.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.554+0000 7f980a7fc700 1 --2- 172.21.15.113:0/1492057369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f980c0fd150 0x7f980c1095e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f97fc00c790 tx=0x7f97fc007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:12.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.554+0000 7f97f3fff700 1 -- 172.21.15.113:0/1492057369 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97fc00e680 con 0x7f980c0fd150 2024-04-12T05:21:12.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.554+0000 7f97f3fff700 1 -- 172.21.15.113:0/1492057369 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97fc00ec90 con 0x7f980c0fd150 2024-04-12T05:21:12.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.554+0000 7f97f3fff700 1 -- 172.21.15.113:0/1492057369 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97fc012c70 con 0x7f980c0fd150 2024-04-12T05:21:12.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.554+0000 7f98119bb700 1 -- 172.21.15.113:0/1492057369 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c109ec0 con 0x7f980c0fd150 2024-04-12T05:21:12.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.554+0000 7f98119bb700 1 -- 172.21.15.113:0/1492057369 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c1124b0 con 0x7f980c0fd150 2024-04-12T05:21:12.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.555+0000 7f98119bb700 1 -- 172.21.15.113:0/1492057369 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980c05f590 con 0x7f980c0fd150 2024-04-12T05:21:12.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.555+0000 7f97f3fff700 1 -- 172.21.15.113:0/1492057369 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f97fc012490 con 0x7f980c0fd150 2024-04-12T05:21:12.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.556+0000 7f97f3fff700 1 --2- 172.21.15.113:0/1492057369 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97f406caa0 0x7f97f406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:12.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.556+0000 7f97f3fff700 1 -- 172.21.15.113:0/1492057369 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f97fc08b650 con 0x7f980c0fd150 2024-04-12T05:21:12.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.556+0000 7f980affd700 1 --2- 172.21.15.113:0/1492057369 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97f406caa0 0x7f97f406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:12.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.556+0000 7f980affd700 1 --2- 172.21.15.113:0/1492057369 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97f406caa0 0x7f97f406ef60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f9800007cf0 tx=0x7f980000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:12.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.560+0000 7f97f3fff700 1 -- 172.21.15.113:0/1492057369 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f97fc055670 con 0x7f980c0fd150 2024-04-12T05:21:12.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.788+0000 7f98119bb700 1 -- 172.21.15.113:0/1492057369 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f980c047540 con 0x7f980c0fd150 2024-04-12T05:21:12.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.788+0000 7f97f3fff700 1 -- 172.21.15.113:0/1492057369 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f97fc058c90 con 0x7f980c0fd150 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:21:12.791 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:21:12.792 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:21:12.793 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:21:12.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.790+0000 7f98119bb700 1 -- 172.21.15.113:0/1492057369 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97f406caa0 msgr2=0x7f97f406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.790+0000 7f98119bb700 1 --2- 172.21.15.113:0/1492057369 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97f406caa0 0x7f97f406ef60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f9800007cf0 tx=0x7f980000b040).stop 2024-04-12T05:21:12.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.790+0000 7f98119bb700 1 -- 172.21.15.113:0/1492057369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f980c0fd150 msgr2=0x7f980c1095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.790+0000 7f98119bb700 1 --2- 172.21.15.113:0/1492057369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f980c0fd150 0x7f980c1095e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f97fc00c790 tx=0x7f97fc007f70).stop 2024-04-12T05:21:12.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.790+0000 7f98119bb700 1 -- 172.21.15.113:0/1492057369 shutdown_connections 2024-04-12T05:21:12.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.790+0000 7f98119bb700 1 --2- 172.21.15.113:0/1492057369 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97f406caa0 0x7f97f406ef60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.790+0000 7f98119bb700 1 --2- 172.21.15.113:0/1492057369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f980c0fc7f0 0x7f980c1090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.790+0000 7f98119bb700 1 --2- 172.21.15.113:0/1492057369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f980c0fd150 0x7f980c1095e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.790+0000 7f98119bb700 1 -- 172.21.15.113:0/1492057369 >> 172.21.15.113:0/1492057369 conn(0x7f980c070590 msgr2=0x7f980c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:12.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.790+0000 7f98119bb700 1 -- 172.21.15.113:0/1492057369 shutdown_connections 2024-04-12T05:21:12.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.790+0000 7f98119bb700 1 -- 172.21.15.113:0/1492057369 wait complete. 2024-04-12T05:21:12.795 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:21:12.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.892+0000 7f176eadd700 1 -- 172.21.15.113:0/1861388174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17680fd830 msgr2=0x7f17680fdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.892+0000 7f176eadd700 1 --2- 172.21.15.113:0/1861388174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17680fd830 0x7f17680fdc10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f175c0047f0 tx=0x7f175c009cb0).stop 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.892+0000 7f176eadd700 1 -- 172.21.15.113:0/1861388174 shutdown_connections 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.892+0000 7f176eadd700 1 --2- 172.21.15.113:0/1861388174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17680fe190 0x7f17680f7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.892+0000 7f176eadd700 1 --2- 172.21.15.113:0/1861388174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17680fd830 0x7f17680fdc10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.892+0000 7f176eadd700 1 -- 172.21.15.113:0/1861388174 >> 172.21.15.113:0/1861388174 conn(0x7f17680f3450 msgr2=0x7f17680f5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.892+0000 7f176eadd700 1 -- 172.21.15.113:0/1861388174 shutdown_connections 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.892+0000 7f176eadd700 1 -- 172.21.15.113:0/1861388174 wait complete. 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.892+0000 7f176eadd700 1 Processor -- start 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.892+0000 7f176eadd700 1 -- start start 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f176eadd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17680fd830 0x7f176810e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f176eadd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17680fe190 0x7f176810e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f176eadd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176810eb90 con 0x7f17680fe190 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f176eadd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176810ecd0 con 0x7f17680fd830 2024-04-12T05:21:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f1767fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17680fe190 0x7f176810e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f1767fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17680fe190 0x7f176810e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41454/0 (socket says 172.21.15.113:41454) 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f1767fff700 1 -- 172.21.15.113:0/1457021720 learned_addr learned my addr 172.21.15.113:0/1457021720 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f176c879700 1 --2- 172.21.15.113:0/1457021720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17680fd830 0x7f176810e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f1767fff700 1 -- 172.21.15.113:0/1457021720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17680fd830 msgr2=0x7f176810e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f1767fff700 1 --2- 172.21.15.113:0/1457021720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17680fd830 0x7f176810e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f1767fff700 1 -- 172.21.15.113:0/1457021720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175c005040 con 0x7f17680fe190 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f1767fff700 1 --2- 172.21.15.113:0/1457021720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17680fe190 0x7f176810e590 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f175400c790 tx=0x7f1754007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f1765f76700 1 -- 172.21.15.113:0/1457021720 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f175400e680 con 0x7f17680fe190 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f1765f76700 1 -- 172.21.15.113:0/1457021720 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f175400ec90 con 0x7f17680fe190 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f176eadd700 1 -- 172.21.15.113:0/1457021720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1768108150 con 0x7f17680fe190 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.893+0000 7f1765f76700 1 -- 172.21.15.113:0/1457021720 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1754012c70 con 0x7f17680fe190 2024-04-12T05:21:12.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.894+0000 7f176eadd700 1 -- 172.21.15.113:0/1457021720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1768108710 con 0x7f17680fe190 2024-04-12T05:21:12.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.894+0000 7f176eadd700 1 -- 172.21.15.113:0/1457021720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176805f470 con 0x7f17680fe190 2024-04-12T05:21:12.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.895+0000 7f1765f76700 1 -- 172.21.15.113:0/1457021720 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1754012490 con 0x7f17680fe190 2024-04-12T05:21:12.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.895+0000 7f1765f76700 1 --2- 172.21.15.113:0/1457021720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f175806caf0 0x7f175806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:12.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.895+0000 7f1765f76700 1 -- 172.21.15.113:0/1457021720 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f175408b730 con 0x7f17680fe190 2024-04-12T05:21:12.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.895+0000 7f176c879700 1 --2- 172.21.15.113:0/1457021720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f175806caf0 0x7f175806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:12.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.896+0000 7f176c879700 1 --2- 172.21.15.113:0/1457021720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f175806caf0 0x7f175806efb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f175c007cf0 tx=0x7f175c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:12.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:12.900+0000 7f1765f76700 1 -- 172.21.15.113:0/1457021720 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f17540591e0 con 0x7f17680fe190 2024-04-12T05:21:13.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.104+0000 7f176eadd700 1 -- 172.21.15.113:0/1457021720 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1768109790 con 0x7f175806caf0 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.105+0000 7f1765f76700 1 -- 172.21.15.113:0/1457021720 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1768109790 con 0x7f175806caf0 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:21:13.108 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:21:13.109 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:21:13.109 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:21:13.109 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:21:13.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.108+0000 7f176eadd700 1 -- 172.21.15.113:0/1457021720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f175806caf0 msgr2=0x7f175806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:13.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.108+0000 7f176eadd700 1 --2- 172.21.15.113:0/1457021720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f175806caf0 0x7f175806efb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f175c007cf0 tx=0x7f175c00b040).stop 2024-04-12T05:21:13.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.108+0000 7f176eadd700 1 -- 172.21.15.113:0/1457021720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17680fe190 msgr2=0x7f176810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:13.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.108+0000 7f176eadd700 1 --2- 172.21.15.113:0/1457021720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17680fe190 0x7f176810e590 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f175400c790 tx=0x7f1754007f70).stop 2024-04-12T05:21:13.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.108+0000 7f176eadd700 1 -- 172.21.15.113:0/1457021720 shutdown_connections 2024-04-12T05:21:13.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.108+0000 7f176eadd700 1 --2- 172.21.15.113:0/1457021720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f175806caf0 0x7f175806efb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:13.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.108+0000 7f176eadd700 1 --2- 172.21.15.113:0/1457021720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17680fd830 0x7f176810e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:13.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.108+0000 7f176eadd700 1 --2- 172.21.15.113:0/1457021720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17680fe190 0x7f176810e590 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:13.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.108+0000 7f176eadd700 1 -- 172.21.15.113:0/1457021720 >> 172.21.15.113:0/1457021720 conn(0x7f17680f3450 msgr2=0x7f17680fbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:13.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.108+0000 7f176eadd700 1 -- 172.21.15.113:0/1457021720 shutdown_connections 2024-04-12T05:21:13.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.108+0000 7f176eadd700 1 -- 172.21.15.113:0/1457021720 wait complete. 2024-04-12T05:21:13.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.206+0000 7ffa85970700 1 -- 172.21.15.113:0/314420203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa800fc7f0 msgr2=0x7ffa800fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:13.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.206+0000 7ffa85970700 1 --2- 172.21.15.113:0/314420203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa800fc7f0 0x7ffa800fcbd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7ffa740047f0 tx=0x7ffa74009cb0).stop 2024-04-12T05:21:13.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.206+0000 7ffa85970700 1 -- 172.21.15.113:0/314420203 shutdown_connections 2024-04-12T05:21:13.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.206+0000 7ffa85970700 1 --2- 172.21.15.113:0/314420203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa800fd150 0x7ffa800f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:13.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.206+0000 7ffa85970700 1 --2- 172.21.15.113:0/314420203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa800fc7f0 0x7ffa800fcbd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:13.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.206+0000 7ffa85970700 1 -- 172.21.15.113:0/314420203 >> 172.21.15.113:0/314420203 conn(0x7ffa80070590 msgr2=0x7ffa800f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:13.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.206+0000 7ffa85970700 1 -- 172.21.15.113:0/314420203 shutdown_connections 2024-04-12T05:21:13.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.206+0000 7ffa85970700 1 -- 172.21.15.113:0/314420203 wait complete. 2024-04-12T05:21:13.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.206+0000 7ffa85970700 1 Processor -- start 2024-04-12T05:21:13.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa85970700 1 -- start start 2024-04-12T05:21:13.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa85970700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa800fc7f0 0x7ffa801090f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa85970700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa800fd150 0x7ffa801095e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa85970700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80109ad0 con 0x7ffa800fc7f0 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa85970700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80109c10 con 0x7ffa800fd150 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa7effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa800fc7f0 0x7ffa801090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa7effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa800fc7f0 0x7ffa801090f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41466/0 (socket says 172.21.15.113:41466) 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa7effd700 1 -- 172.21.15.113:0/2655126301 learned_addr learned my addr 172.21.15.113:0/2655126301 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa7e7fc700 1 --2- 172.21.15.113:0/2655126301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa800fd150 0x7ffa801095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa7effd700 1 -- 172.21.15.113:0/2655126301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa800fd150 msgr2=0x7ffa801095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa7effd700 1 --2- 172.21.15.113:0/2655126301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa800fd150 0x7ffa801095e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa7effd700 1 -- 172.21.15.113:0/2655126301 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa74005040 con 0x7ffa800fc7f0 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa7effd700 1 --2- 172.21.15.113:0/2655126301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa800fc7f0 0x7ffa801090f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7ffa740043a0 tx=0x7ffa74009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa67fff700 1 -- 172.21.15.113:0/2655126301 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa7400e030 con 0x7ffa800fc7f0 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa67fff700 1 -- 172.21.15.113:0/2655126301 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa74007e90 con 0x7ffa800fc7f0 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa85970700 1 -- 172.21.15.113:0/2655126301 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa80109e60 con 0x7ffa800fc7f0 2024-04-12T05:21:13.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.207+0000 7ffa67fff700 1 -- 172.21.15.113:0/2655126301 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa74018070 con 0x7ffa800fc7f0 2024-04-12T05:21:13.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.208+0000 7ffa85970700 1 -- 172.21.15.113:0/2655126301 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa80112450 con 0x7ffa800fc7f0 2024-04-12T05:21:13.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.208+0000 7ffa85970700 1 -- 172.21.15.113:0/2655126301 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa8005f590 con 0x7ffa800fc7f0 2024-04-12T05:21:13.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.209+0000 7ffa67fff700 1 -- 172.21.15.113:0/2655126301 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ffa740094c0 con 0x7ffa800fc7f0 2024-04-12T05:21:13.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.209+0000 7ffa67fff700 1 --2- 172.21.15.113:0/2655126301 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffa6806cab0 0x7ffa6806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:13.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.209+0000 7ffa67fff700 1 -- 172.21.15.113:0/2655126301 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ffa7408d000 con 0x7ffa800fc7f0 2024-04-12T05:21:13.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.209+0000 7ffa7e7fc700 1 --2- 172.21.15.113:0/2655126301 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffa6806cab0 0x7ffa6806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:13.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.210+0000 7ffa7e7fc700 1 --2- 172.21.15.113:0/2655126301 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffa6806cab0 0x7ffa6806ef70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ffa80060ee0 tx=0x7ffa7000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:13.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.214+0000 7ffa67fff700 1 -- 172.21.15.113:0/2655126301 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ffa74057020 con 0x7ffa800fc7f0 2024-04-12T05:21:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:12 smithi113 ceph-mon[103371]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:12 smithi113 ceph-mon[103371]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:12 smithi113 ceph-mon[103371]: pgmap v271: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:12 smithi113 ceph-mon[103371]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:12 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3119035545' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:21:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:12 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1492057369' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:21:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:12 smithi192 ceph-mon[85304]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:13.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:12 smithi192 ceph-mon[85304]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:13.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:12 smithi192 ceph-mon[85304]: pgmap v271: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:13.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:12 smithi192 ceph-mon[85304]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:13.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:12 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3119035545' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:21:13.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:12 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1492057369' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:21:13.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.479+0000 7ffa85970700 1 -- 172.21.15.113:0/2655126301 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ffa80047540 con 0x7ffa800fc7f0 2024-04-12T05:21:13.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.479+0000 7ffa67fff700 1 -- 172.21.15.113:0/2655126301 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ffa7405a640 con 0x7ffa800fc7f0 2024-04-12T05:21:13.481 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:21:13.481 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:21:13.481 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:21:13.481 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:21:13.482 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:21:13.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.481+0000 7ffa85970700 1 -- 172.21.15.113:0/2655126301 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffa6806cab0 msgr2=0x7ffa6806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:13.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.481+0000 7ffa85970700 1 --2- 172.21.15.113:0/2655126301 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffa6806cab0 0x7ffa6806ef70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ffa80060ee0 tx=0x7ffa7000b040).stop 2024-04-12T05:21:13.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.481+0000 7ffa85970700 1 -- 172.21.15.113:0/2655126301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa800fc7f0 msgr2=0x7ffa801090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:13.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.481+0000 7ffa85970700 1 --2- 172.21.15.113:0/2655126301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa800fc7f0 0x7ffa801090f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7ffa740043a0 tx=0x7ffa74009cb0).stop 2024-04-12T05:21:13.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.481+0000 7ffa85970700 1 -- 172.21.15.113:0/2655126301 shutdown_connections 2024-04-12T05:21:13.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.481+0000 7ffa85970700 1 --2- 172.21.15.113:0/2655126301 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffa6806cab0 0x7ffa6806ef70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:13.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.481+0000 7ffa85970700 1 --2- 172.21.15.113:0/2655126301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa800fc7f0 0x7ffa801090f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:13.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.481+0000 7ffa85970700 1 --2- 172.21.15.113:0/2655126301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa800fd150 0x7ffa801095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:13.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.481+0000 7ffa85970700 1 -- 172.21.15.113:0/2655126301 >> 172.21.15.113:0/2655126301 conn(0x7ffa80070590 msgr2=0x7ffa800fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:13.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.481+0000 7ffa85970700 1 -- 172.21.15.113:0/2655126301 shutdown_connections 2024-04-12T05:21:13.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:13.481+0000 7ffa85970700 1 -- 172.21.15.113:0/2655126301 wait complete. 2024-04-12T05:21:14.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:13 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2655126301' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:21:14.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:13 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2655126301' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:21:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:15 smithi113 ceph-mon[103371]: from='client.34552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:15 smithi113 ceph-mon[103371]: pgmap v272: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:15 smithi192 ceph-mon[85304]: from='client.34552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:15 smithi192 ceph-mon[85304]: pgmap v272: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:17 smithi113 ceph-mon[103371]: pgmap v273: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:17.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:17 smithi192 ceph-mon[85304]: pgmap v273: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:19 smithi113 ceph-mon[103371]: pgmap v274: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:19 smithi192 ceph-mon[85304]: pgmap v274: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:21 smithi113 ceph-mon[103371]: pgmap v275: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:21 smithi192 ceph-mon[85304]: pgmap v275: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:22.269 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:21:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:21:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:23 smithi113 ceph-mon[103371]: pgmap v276: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:23 smithi192 ceph-mon[85304]: pgmap v276: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:25 smithi113 ceph-mon[103371]: pgmap v277: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:25 smithi192 ceph-mon[85304]: pgmap v277: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:27 smithi113 ceph-mon[103371]: pgmap v278: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:27 smithi192 ceph-mon[85304]: pgmap v278: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:29 smithi113 ceph-mon[103371]: pgmap v279: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:29 smithi192 ceph-mon[85304]: pgmap v279: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:31 smithi113 ceph-mon[103371]: pgmap v280: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:31 smithi192 ceph-mon[85304]: pgmap v280: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:33 smithi113 ceph-mon[103371]: pgmap v281: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:33.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:33 smithi192 ceph-mon[85304]: pgmap v281: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:35 smithi113 ceph-mon[103371]: pgmap v282: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:35.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:35 smithi192 ceph-mon[85304]: pgmap v282: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:37 smithi113 ceph-mon[103371]: pgmap v283: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:21:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:37 smithi192 ceph-mon[85304]: pgmap v283: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:21:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:39 smithi113 ceph-mon[103371]: pgmap v284: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:39 smithi192 ceph-mon[85304]: pgmap v284: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:41 smithi113 ceph-mon[103371]: pgmap v285: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:41 smithi192 ceph-mon[85304]: pgmap v285: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:43 smithi113 ceph-mon[103371]: pgmap v286: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:43.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:43 smithi192 ceph-mon[85304]: pgmap v286: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:43.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.588+0000 7f994abb7700 1 -- 172.21.15.113:0/3540181130 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440fe250 msgr2=0x7f99440f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:43.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.588+0000 7f994abb7700 1 --2- 172.21.15.113:0/3540181130 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440fe250 0x7f99440f7a30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f99300047f0 tx=0x7f9930009cb0).stop 2024-04-12T05:21:43.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.588+0000 7f994abb7700 1 -- 172.21.15.113:0/3540181130 shutdown_connections 2024-04-12T05:21:43.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.588+0000 7f994abb7700 1 --2- 172.21.15.113:0/3540181130 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440fe250 0x7f99440f7a30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:43.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.588+0000 7f994abb7700 1 --2- 172.21.15.113:0/3540181130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99440fd8f0 0x7f99440fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.588+0000 7f994abb7700 1 -- 172.21.15.113:0/3540181130 >> 172.21.15.113:0/3540181130 conn(0x7f99440f34b0 msgr2=0x7f99440f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.588+0000 7f994abb7700 1 -- 172.21.15.113:0/3540181130 shutdown_connections 2024-04-12T05:21:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.588+0000 7f994abb7700 1 -- 172.21.15.113:0/3540181130 wait complete. 2024-04-12T05:21:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f994abb7700 1 Processor -- start 2024-04-12T05:21:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f994abb7700 1 -- start start 2024-04-12T05:21:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f994abb7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99440fd8f0 0x7f994406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f994abb7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440fe250 0x7f994406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f994abb7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994406bab0 con 0x7f99440fe250 2024-04-12T05:21:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f994abb7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994406bbf0 con 0x7f99440fd8f0 2024-04-12T05:21:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f9943fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440fe250 0x7f994406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:43.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f9943fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440fe250 0x7f994406b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33816/0 (socket says 172.21.15.113:33816) 2024-04-12T05:21:43.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f9943fff700 1 -- 172.21.15.113:0/1827972804 learned_addr learned my addr 172.21.15.113:0/1827972804 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:43.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f9948953700 1 --2- 172.21.15.113:0/1827972804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99440fd8f0 0x7f994406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:43.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f9943fff700 1 -- 172.21.15.113:0/1827972804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99440fd8f0 msgr2=0x7f994406b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:43.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f9943fff700 1 --2- 172.21.15.113:0/1827972804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99440fd8f0 0x7f994406b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:43.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.589+0000 7f9943fff700 1 -- 172.21.15.113:0/1827972804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9930005040 con 0x7f99440fe250 2024-04-12T05:21:43.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.590+0000 7f9943fff700 1 --2- 172.21.15.113:0/1827972804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440fe250 0x7f994406b5c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f99300099f0 tx=0x7f9930009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:43.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.590+0000 7f9941ffb700 1 -- 172.21.15.113:0/1827972804 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f993000e030 con 0x7f99440fe250 2024-04-12T05:21:43.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.590+0000 7f9941ffb700 1 -- 172.21.15.113:0/1827972804 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9930007e90 con 0x7f99440fe250 2024-04-12T05:21:43.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.590+0000 7f994abb7700 1 -- 172.21.15.113:0/1827972804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994406be40 con 0x7f99440fe250 2024-04-12T05:21:43.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.590+0000 7f9941ffb700 1 -- 172.21.15.113:0/1827972804 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9930018070 con 0x7f99440fe250 2024-04-12T05:21:43.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.590+0000 7f994abb7700 1 -- 172.21.15.113:0/1827972804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9944114690 con 0x7f99440fe250 2024-04-12T05:21:43.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.591+0000 7f994abb7700 1 -- 172.21.15.113:0/1827972804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994405f590 con 0x7f99440fe250 2024-04-12T05:21:43.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.591+0000 7f9941ffb700 1 -- 172.21.15.113:0/1827972804 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f99300094c0 con 0x7f99440fe250 2024-04-12T05:21:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.592+0000 7f9941ffb700 1 --2- 172.21.15.113:0/1827972804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f993406caa0 0x7f993406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.592+0000 7f9941ffb700 1 -- 172.21.15.113:0/1827972804 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f993008d080 con 0x7f99440fe250 2024-04-12T05:21:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.592+0000 7f9948953700 1 --2- 172.21.15.113:0/1827972804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f993406caa0 0x7f993406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:43.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.592+0000 7f9948953700 1 --2- 172.21.15.113:0/1827972804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f993406caa0 0x7f993406ef60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f9938004e60 tx=0x7f993800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:43.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.596+0000 7f9941ffb700 1 -- 172.21.15.113:0/1827972804 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f993005ab30 con 0x7f99440fe250 2024-04-12T05:21:43.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.803+0000 7f994abb7700 1 -- 172.21.15.113:0/1827972804 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9944059120 con 0x7f993406caa0 2024-04-12T05:21:43.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.804+0000 7f9941ffb700 1 -- 172.21.15.113:0/1827972804 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9944059120 con 0x7f993406caa0 2024-04-12T05:21:43.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.806+0000 7f994abb7700 1 -- 172.21.15.113:0/1827972804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f993406caa0 msgr2=0x7f993406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:43.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.806+0000 7f994abb7700 1 --2- 172.21.15.113:0/1827972804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f993406caa0 0x7f993406ef60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f9938004e60 tx=0x7f993800b040).stop 2024-04-12T05:21:43.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.806+0000 7f994abb7700 1 -- 172.21.15.113:0/1827972804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440fe250 msgr2=0x7f994406b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:43.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.806+0000 7f994abb7700 1 --2- 172.21.15.113:0/1827972804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440fe250 0x7f994406b5c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f99300099f0 tx=0x7f9930009cb0).stop 2024-04-12T05:21:43.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.806+0000 7f994abb7700 1 -- 172.21.15.113:0/1827972804 shutdown_connections 2024-04-12T05:21:43.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.806+0000 7f994abb7700 1 --2- 172.21.15.113:0/1827972804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f993406caa0 0x7f993406ef60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:43.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.806+0000 7f994abb7700 1 --2- 172.21.15.113:0/1827972804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99440fd8f0 0x7f994406b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:43.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.806+0000 7f994abb7700 1 --2- 172.21.15.113:0/1827972804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99440fe250 0x7f994406b5c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:43.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.806+0000 7f994abb7700 1 -- 172.21.15.113:0/1827972804 >> 172.21.15.113:0/1827972804 conn(0x7f99440f34b0 msgr2=0x7f99440fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:43.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.806+0000 7f994abb7700 1 -- 172.21.15.113:0/1827972804 shutdown_connections 2024-04-12T05:21:43.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.806+0000 7f994abb7700 1 -- 172.21.15.113:0/1827972804 wait complete. 2024-04-12T05:21:43.819 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:21:43.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.905+0000 7f793ec15700 1 -- 172.21.15.113:0/2879719749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79380fc7f0 msgr2=0x7f79380fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:43.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.905+0000 7f793ec15700 1 --2- 172.21.15.113:0/2879719749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79380fc7f0 0x7f79380fcbd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f792c0047f0 tx=0x7f792c009cb0).stop 2024-04-12T05:21:43.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.905+0000 7f793ec15700 1 -- 172.21.15.113:0/2879719749 shutdown_connections 2024-04-12T05:21:43.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.905+0000 7f793ec15700 1 --2- 172.21.15.113:0/2879719749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79380fd150 0x7f79380f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:43.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.905+0000 7f793ec15700 1 --2- 172.21.15.113:0/2879719749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79380fc7f0 0x7f79380fcbd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:43.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.905+0000 7f793ec15700 1 -- 172.21.15.113:0/2879719749 >> 172.21.15.113:0/2879719749 conn(0x7f7938070590 msgr2=0x7f79380f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.905+0000 7f793ec15700 1 -- 172.21.15.113:0/2879719749 shutdown_connections 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.905+0000 7f793ec15700 1 -- 172.21.15.113:0/2879719749 wait complete. 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.906+0000 7f793ec15700 1 Processor -- start 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.906+0000 7f793ec15700 1 -- start start 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.906+0000 7f793ec15700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79380fc7f0 0x7f793810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.906+0000 7f793ec15700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79380fd150 0x7f793810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.906+0000 7f793ec15700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793810eb20 con 0x7f79380fc7f0 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.906+0000 7f793ec15700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793810ec60 con 0x7f79380fd150 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.906+0000 7f793c9b1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79380fc7f0 0x7f793810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.906+0000 7f793c9b1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79380fc7f0 0x7f793810e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33836/0 (socket says 172.21.15.113:33836) 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.906+0000 7f793c9b1700 1 -- 172.21.15.113:0/3589990431 learned_addr learned my addr 172.21.15.113:0/3589990431 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.906+0000 7f7937fff700 1 --2- 172.21.15.113:0/3589990431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79380fd150 0x7f793810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:43.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.907+0000 7f793c9b1700 1 -- 172.21.15.113:0/3589990431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79380fd150 msgr2=0x7f793810e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:43.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.907+0000 7f793c9b1700 1 --2- 172.21.15.113:0/3589990431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79380fd150 0x7f793810e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:43.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.907+0000 7f793c9b1700 1 -- 172.21.15.113:0/3589990431 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792c005040 con 0x7f79380fc7f0 2024-04-12T05:21:43.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.907+0000 7f793c9b1700 1 --2- 172.21.15.113:0/3589990431 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79380fc7f0 0x7f793810e030 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f792c0043a0 tx=0x7f792c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:43.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.907+0000 7f7935ffb700 1 -- 172.21.15.113:0/3589990431 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f792c00e030 con 0x7f79380fc7f0 2024-04-12T05:21:43.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.907+0000 7f793ec15700 1 -- 172.21.15.113:0/3589990431 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7938108290 con 0x7f79380fc7f0 2024-04-12T05:21:43.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.907+0000 7f7935ffb700 1 -- 172.21.15.113:0/3589990431 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f792c007e90 con 0x7f79380fc7f0 2024-04-12T05:21:43.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.907+0000 7f7935ffb700 1 -- 172.21.15.113:0/3589990431 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f792c018070 con 0x7f79380fc7f0 2024-04-12T05:21:43.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.907+0000 7f793ec15700 1 -- 172.21.15.113:0/3589990431 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79381087f0 con 0x7f79380fc7f0 2024-04-12T05:21:43.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.908+0000 7f793ec15700 1 -- 172.21.15.113:0/3589990431 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f793805f590 con 0x7f79380fc7f0 2024-04-12T05:21:43.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.908+0000 7f7935ffb700 1 -- 172.21.15.113:0/3589990431 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f792c0094c0 con 0x7f79380fc7f0 2024-04-12T05:21:43.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.909+0000 7f7935ffb700 1 --2- 172.21.15.113:0/3589990431 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f792806cab0 0x7f792806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:43.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.909+0000 7f7935ffb700 1 -- 172.21.15.113:0/3589990431 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f792c08d000 con 0x7f79380fc7f0 2024-04-12T05:21:43.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.909+0000 7f7937fff700 1 --2- 172.21.15.113:0/3589990431 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f792806cab0 0x7f792806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:43.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.909+0000 7f7937fff700 1 --2- 172.21.15.113:0/3589990431 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f792806cab0 0x7f792806ef70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f7938109970 tx=0x7f792400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:43.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:43.913+0000 7f7935ffb700 1 -- 172.21.15.113:0/3589990431 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f792c05aab0 con 0x7f79380fc7f0 2024-04-12T05:21:44.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.120+0000 7f793ec15700 1 -- 172.21.15.113:0/3589990431 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7938108e80 con 0x7f792806cab0 2024-04-12T05:21:44.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.121+0000 7f7935ffb700 1 -- 172.21.15.113:0/3589990431 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7938108e80 con 0x7f792806cab0 2024-04-12T05:21:44.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.124+0000 7f793ec15700 1 -- 172.21.15.113:0/3589990431 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f792806cab0 msgr2=0x7f792806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.124+0000 7f793ec15700 1 --2- 172.21.15.113:0/3589990431 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f792806cab0 0x7f792806ef70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f7938109970 tx=0x7f792400b040).stop 2024-04-12T05:21:44.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.124+0000 7f793ec15700 1 -- 172.21.15.113:0/3589990431 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79380fc7f0 msgr2=0x7f793810e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.124+0000 7f793ec15700 1 --2- 172.21.15.113:0/3589990431 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79380fc7f0 0x7f793810e030 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f792c0043a0 tx=0x7f792c009cb0).stop 2024-04-12T05:21:44.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.124+0000 7f793ec15700 1 -- 172.21.15.113:0/3589990431 shutdown_connections 2024-04-12T05:21:44.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.124+0000 7f793ec15700 1 --2- 172.21.15.113:0/3589990431 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f792806cab0 0x7f792806ef70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.124+0000 7f793ec15700 1 --2- 172.21.15.113:0/3589990431 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79380fc7f0 0x7f793810e030 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.124+0000 7f793ec15700 1 --2- 172.21.15.113:0/3589990431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79380fd150 0x7f793810e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.124+0000 7f793ec15700 1 -- 172.21.15.113:0/3589990431 >> 172.21.15.113:0/3589990431 conn(0x7f7938070590 msgr2=0x7f7938060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:44.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.124+0000 7f793ec15700 1 -- 172.21.15.113:0/3589990431 shutdown_connections 2024-04-12T05:21:44.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.124+0000 7f793ec15700 1 -- 172.21.15.113:0/3589990431 wait complete. 2024-04-12T05:21:44.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.222+0000 7f31f4ba0700 1 -- 172.21.15.113:0/4122600049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f00fd150 msgr2=0x7f31f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.222+0000 7f31f4ba0700 1 --2- 172.21.15.113:0/4122600049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f00fd150 0x7f31f00f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f31e00047f0 tx=0x7f31e0009cb0).stop 2024-04-12T05:21:44.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.222+0000 7f31f4ba0700 1 -- 172.21.15.113:0/4122600049 shutdown_connections 2024-04-12T05:21:44.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.222+0000 7f31f4ba0700 1 --2- 172.21.15.113:0/4122600049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f00fd150 0x7f31f00f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.222+0000 7f31f4ba0700 1 --2- 172.21.15.113:0/4122600049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f00fc7f0 0x7f31f00fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:21:44.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.222+0000 7f31f4ba0700 1 -- 172.21.15.113:0/4122600049 >> 172.21.15.113:0/4122600049 conn(0x7f31f0070590 msgr2=0x7f31f00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:44.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.222+0000 7f31f4ba0700 1 -- 172.21.15.113:0/4122600049 shutdown_connections 2024-04-12T05:21:44.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.222+0000 7f31f4ba0700 1 -- 172.21.15.113:0/4122600049 wait complete. 2024-04-12T05:21:44.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.223+0000 7f31f4ba0700 1 Processor -- start 2024-04-12T05:21:44.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.223+0000 7f31f4ba0700 1 -- start start 2024-04-12T05:21:44.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.223+0000 7f31f4ba0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f00fc7f0 0x7f31f010e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:44.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.223+0000 7f31f4ba0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f00fd150 0x7f31f010e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:44.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.223+0000 7f31f4ba0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f010ec00 con 0x7f31f00fd150 2024-04-12T05:21:44.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.223+0000 7f31f4ba0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f0108220 con 0x7f31f00fc7f0 2024-04-12T05:21:44.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.223+0000 7f31edd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f00fd150 0x7f31f010e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:44.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.223+0000 7f31edd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f00fd150 0x7f31f010e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33840/0 (socket says 172.21.15.113:33840) 2024-04-12T05:21:44.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.223+0000 7f31edd9b700 1 -- 172.21.15.113:0/2231426216 learned_addr learned my addr 172.21.15.113:0/2231426216 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:44.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.224+0000 7f31ee59c700 1 --2- 172.21.15.113:0/2231426216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f00fc7f0 0x7f31f010e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:44.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.224+0000 7f31edd9b700 1 -- 172.21.15.113:0/2231426216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f00fc7f0 msgr2=0x7f31f010e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.224+0000 7f31edd9b700 1 --2- 172.21.15.113:0/2231426216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f00fc7f0 0x7f31f010e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.224+0000 7f31edd9b700 1 -- 172.21.15.113:0/2231426216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e0005040 con 0x7f31f00fd150 2024-04-12T05:21:44.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.224+0000 7f31edd9b700 1 --2- 172.21.15.113:0/2231426216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f00fd150 0x7f31f010e570 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f31e00099f0 tx=0x7f31e0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:44.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.224+0000 7f31df7fe700 1 -- 172.21.15.113:0/2231426216 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31e000e030 con 0x7f31f00fd150 2024-04-12T05:21:44.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.224+0000 7f31df7fe700 1 -- 172.21.15.113:0/2231426216 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31e0007e90 con 0x7f31f00fd150 2024-04-12T05:21:44.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.224+0000 7f31f4ba0700 1 -- 172.21.15.113:0/2231426216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f0108470 con 0x7f31f00fd150 2024-04-12T05:21:44.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.224+0000 7f31df7fe700 1 -- 172.21.15.113:0/2231426216 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31e0018070 con 0x7f31f00fd150 2024-04-12T05:21:44.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.224+0000 7f31f4ba0700 1 -- 172.21.15.113:0/2231426216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f0108950 con 0x7f31f00fd150 2024-04-12T05:21:44.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.225+0000 7f31f4ba0700 1 -- 172.21.15.113:0/2231426216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f005f590 con 0x7f31f00fd150 2024-04-12T05:21:44.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.225+0000 7f31df7fe700 1 -- 172.21.15.113:0/2231426216 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f31e001d420 con 0x7f31f00fd150 2024-04-12T05:21:44.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.226+0000 7f31df7fe700 1 --2- 172.21.15.113:0/2231426216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f31d806caa0 0x7f31d806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:44.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.226+0000 7f31ee59c700 1 --2- 172.21.15.113:0/2231426216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f31d806caa0 0x7f31d806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:44.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.226+0000 7f31df7fe700 1 -- 172.21.15.113:0/2231426216 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f31e008d2f0 con 0x7f31f00fd150 2024-04-12T05:21:44.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.226+0000 7f31ee59c700 1 --2- 172.21.15.113:0/2231426216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f31d806caa0 0x7f31d806ef60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f31e400b910 tx=0x7f31e4008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:44.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.230+0000 7f31df7fe700 1 -- 172.21.15.113:0/2231426216 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f31e005ada0 con 0x7f31f00fd150 2024-04-12T05:21:44.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.430+0000 7f31f4ba0700 1 -- 172.21.15.113:0/2231426216 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f31f0109770 con 0x7f31d806caa0 2024-04-12T05:21:44.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.440+0000 7f31df7fe700 1 -- 172.21.15.113:0/2231426216 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f31f0109770 con 0x7f31d806caa0 2024-04-12T05:21:44.443 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:21:44.443 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (9m) 2m ago 20m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:21:44.443 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (6m) 2m ago 20m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:21:44.443 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (6m) 2m ago 20m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (9m) 2m ago 20m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (2m) 2m ago 15m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (2m) 2m ago 15m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (2m) 2m ago 15m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (15m) 2m ago 15m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (10m) 2m ago 22m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (10m) 2m ago 20m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (8m) 2m ago 22m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (7m) 2m ago 20m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (10m) 2m ago 20m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (10m) 2m ago 19m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (6m) 2m ago 19m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (5m) 2m ago 18m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:21:44.444 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (5m) 2m ago 18m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:21:44.445 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (4m) 2m ago 17m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:21:44.445 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (4m) 2m ago 17m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:21:44.445 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (3m) 2m ago 17m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:21:44.445 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (9m) 2m ago 19m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:21:44.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.442+0000 7f31f4ba0700 1 -- 172.21.15.113:0/2231426216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f31d806caa0 msgr2=0x7f31d806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.442+0000 7f31f4ba0700 1 --2- 172.21.15.113:0/2231426216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f31d806caa0 0x7f31d806ef60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f31e400b910 tx=0x7f31e4008040).stop 2024-04-12T05:21:44.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.442+0000 7f31f4ba0700 1 -- 172.21.15.113:0/2231426216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f00fd150 msgr2=0x7f31f010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.442+0000 7f31f4ba0700 1 --2- 172.21.15.113:0/2231426216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f00fd150 0x7f31f010e570 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f31e00099f0 tx=0x7f31e0009cb0).stop 2024-04-12T05:21:44.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.442+0000 7f31f4ba0700 1 -- 172.21.15.113:0/2231426216 shutdown_connections 2024-04-12T05:21:44.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.442+0000 7f31f4ba0700 1 --2- 172.21.15.113:0/2231426216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f31d806caa0 0x7f31d806ef60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.442+0000 7f31f4ba0700 1 --2- 172.21.15.113:0/2231426216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31f00fc7f0 0x7f31f010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.442+0000 7f31f4ba0700 1 --2- 172.21.15.113:0/2231426216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31f00fd150 0x7f31f010e570 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.442+0000 7f31f4ba0700 1 -- 172.21.15.113:0/2231426216 >> 172.21.15.113:0/2231426216 conn(0x7f31f0070590 msgr2=0x7f31f0060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:44.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.442+0000 7f31f4ba0700 1 -- 172.21.15.113:0/2231426216 shutdown_connections 2024-04-12T05:21:44.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.442+0000 7f31f4ba0700 1 -- 172.21.15.113:0/2231426216 wait complete. 2024-04-12T05:21:44.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.539+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1615734496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb480fe270 msgr2=0x7ffb480f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.539+0000 7ffb4e9f0700 1 --2- 172.21.15.113:0/1615734496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb480fe270 0x7ffb480f7a50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7ffb3c0047f0 tx=0x7ffb3c009cb0).stop 2024-04-12T05:21:44.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.539+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1615734496 shutdown_connections 2024-04-12T05:21:44.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.539+0000 7ffb4e9f0700 1 --2- 172.21.15.113:0/1615734496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb480fe270 0x7ffb480f7a50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.539+0000 7ffb4e9f0700 1 --2- 172.21.15.113:0/1615734496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb480fd910 0x7ffb480fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.539+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1615734496 >> 172.21.15.113:0/1615734496 conn(0x7ffb480f3510 msgr2=0x7ffb480f5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:44.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.539+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1615734496 shutdown_connections 2024-04-12T05:21:44.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.539+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1615734496 wait complete. 2024-04-12T05:21:44.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.540+0000 7ffb4e9f0700 1 Processor -- start 2024-04-12T05:21:44.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.540+0000 7ffb4e9f0700 1 -- start start 2024-04-12T05:21:44.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.540+0000 7ffb4e9f0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb480fd910 0x7ffb4806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:44.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.540+0000 7ffb4e9f0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb480fe270 0x7ffb4806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:44.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.540+0000 7ffb4e9f0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4806bab0 con 0x7ffb480fe270 2024-04-12T05:21:44.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.540+0000 7ffb4e9f0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4806bbf0 con 0x7ffb480fd910 2024-04-12T05:21:44.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.540+0000 7ffb477fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb480fe270 0x7ffb4806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:44.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb477fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb480fe270 0x7ffb4806b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33848/0 (socket says 172.21.15.113:33848) 2024-04-12T05:21:44.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb477fe700 1 -- 172.21.15.113:0/1403773585 learned_addr learned my addr 172.21.15.113:0/1403773585 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:44.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb47fff700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb480fd910 0x7ffb4806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:44.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb477fe700 1 -- 172.21.15.113:0/1403773585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb480fd910 msgr2=0x7ffb4806b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb477fe700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb480fd910 0x7ffb4806b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb477fe700 1 -- 172.21.15.113:0/1403773585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3c005040 con 0x7ffb480fe270 2024-04-12T05:21:44.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb47fff700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb480fd910 0x7ffb4806b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:21:44.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb477fe700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb480fe270 0x7ffb4806b5c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7ffb3c0099f0 tx=0x7ffb3c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:44.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb457fa700 1 -- 172.21.15.113:0/1403773585 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb3c00e030 con 0x7ffb480fe270 2024-04-12T05:21:44.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb457fa700 1 -- 172.21.15.113:0/1403773585 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb3c007e90 con 0x7ffb480fe270 2024-04-12T05:21:44.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1403773585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4806be40 con 0x7ffb480fe270 2024-04-12T05:21:44.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb457fa700 1 -- 172.21.15.113:0/1403773585 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb3c018070 con 0x7ffb480fe270 2024-04-12T05:21:44.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.541+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1403773585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb481145e0 con 0x7ffb480fe270 2024-04-12T05:21:44.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.542+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1403773585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb48060650 con 0x7ffb480fe270 2024-04-12T05:21:44.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.543+0000 7ffb457fa700 1 -- 172.21.15.113:0/1403773585 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ffb3c009590 con 0x7ffb480fe270 2024-04-12T05:21:44.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.543+0000 7ffb457fa700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb3006ca30 0x7ffb3006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:44.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.543+0000 7ffb457fa700 1 -- 172.21.15.113:0/1403773585 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ffb3c08d0f0 con 0x7ffb480fe270 2024-04-12T05:21:44.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.543+0000 7ffb47fff700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb3006ca30 0x7ffb3006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:44.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.543+0000 7ffb47fff700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb3006ca30 0x7ffb3006eef0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ffb38009250 tx=0x7ffb38009de0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:44.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.547+0000 7ffb457fa700 1 -- 172.21.15.113:0/1403773585 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ffb3c057110 con 0x7ffb480fe270 2024-04-12T05:21:44.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.814+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1403773585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffb48047540 con 0x7ffb480fe270 2024-04-12T05:21:44.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.815+0000 7ffb457fa700 1 -- 172.21.15.113:0/1403773585 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7ffb3c05a730 con 0x7ffb480fe270 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:21:44.819 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:21:44.820 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:21:44.820 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:21:44.820 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:21:44.820 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:21:44.820 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:21:44.820 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:21:44.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.817+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1403773585 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb3006ca30 msgr2=0x7ffb3006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.817+0000 7ffb4e9f0700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb3006ca30 0x7ffb3006eef0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ffb38009250 tx=0x7ffb38009de0).stop 2024-04-12T05:21:44.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.817+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1403773585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb480fe270 msgr2=0x7ffb4806b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.817+0000 7ffb4e9f0700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb480fe270 0x7ffb4806b5c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7ffb3c0099f0 tx=0x7ffb3c009cb0).stop 2024-04-12T05:21:44.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.817+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1403773585 shutdown_connections 2024-04-12T05:21:44.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.817+0000 7ffb4e9f0700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb3006ca30 0x7ffb3006eef0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.817+0000 7ffb4e9f0700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb480fd910 0x7ffb4806b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.817+0000 7ffb4e9f0700 1 --2- 172.21.15.113:0/1403773585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb480fe270 0x7ffb4806b5c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.817+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1403773585 >> 172.21.15.113:0/1403773585 conn(0x7ffb480f3510 msgr2=0x7ffb480fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:44.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.817+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1403773585 shutdown_connections 2024-04-12T05:21:44.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.817+0000 7ffb4e9f0700 1 -- 172.21.15.113:0/1403773585 wait complete. 2024-04-12T05:21:44.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.915+0000 7f242e2c8700 1 -- 172.21.15.113:0/3177499365 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24280fc7f0 msgr2=0x7f24280fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.915+0000 7f242e2c8700 1 --2- 172.21.15.113:0/3177499365 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24280fc7f0 0x7f24280fcbd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f24180047f0 tx=0x7f2418009cb0).stop 2024-04-12T05:21:44.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.915+0000 7f242e2c8700 1 -- 172.21.15.113:0/3177499365 shutdown_connections 2024-04-12T05:21:44.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.915+0000 7f242e2c8700 1 --2- 172.21.15.113:0/3177499365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24280fd150 0x7f24280f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.915+0000 7f242e2c8700 1 --2- 172.21.15.113:0/3177499365 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24280fc7f0 0x7f24280fcbd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.915+0000 7f242e2c8700 1 -- 172.21.15.113:0/3177499365 >> 172.21.15.113:0/3177499365 conn(0x7f2428070590 msgr2=0x7f24280f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:44.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.915+0000 7f242e2c8700 1 -- 172.21.15.113:0/3177499365 shutdown_connections 2024-04-12T05:21:44.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.915+0000 7f242e2c8700 1 -- 172.21.15.113:0/3177499365 wait complete. 2024-04-12T05:21:44.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.916+0000 7f242e2c8700 1 Processor -- start 2024-04-12T05:21:44.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.916+0000 7f242e2c8700 1 -- start start 2024-04-12T05:21:44.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.916+0000 7f242e2c8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24280fc7f0 0x7f242810e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.916+0000 7f242e2c8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24280fd150 0x7f242810e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.916+0000 7f242e2c8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242810eb10 con 0x7f24280fd150 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.916+0000 7f242e2c8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242810ec50 con 0x7f24280fc7f0 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f24277fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24280fd150 0x7f242810e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f24277fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24280fd150 0x7f242810e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33866/0 (socket says 172.21.15.113:33866) 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f24277fe700 1 -- 172.21.15.113:0/2386349588 learned_addr learned my addr 172.21.15.113:0/2386349588 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f2427fff700 1 --2- 172.21.15.113:0/2386349588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24280fc7f0 0x7f242810e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f24277fe700 1 -- 172.21.15.113:0/2386349588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24280fc7f0 msgr2=0x7f242810e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f24277fe700 1 --2- 172.21.15.113:0/2386349588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24280fc7f0 0x7f242810e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f24277fe700 1 -- 172.21.15.113:0/2386349588 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2418005040 con 0x7f24280fd150 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f24277fe700 1 --2- 172.21.15.113:0/2386349588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24280fd150 0x7f242810e510 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f241c0041d0 tx=0x7f241c009f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:44.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f2425775700 1 -- 172.21.15.113:0/2386349588 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f241c007b40 con 0x7f24280fd150 2024-04-12T05:21:44.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f2425775700 1 -- 172.21.15.113:0/2386349588 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f241c00f040 con 0x7f24280fd150 2024-04-12T05:21:44.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f2425775700 1 -- 172.21.15.113:0/2386349588 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f241c013770 con 0x7f24280fd150 2024-04-12T05:21:44.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f242e2c8700 1 -- 172.21.15.113:0/2386349588 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24281082e0 con 0x7f24280fd150 2024-04-12T05:21:44.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.917+0000 7f242e2c8700 1 -- 172.21.15.113:0/2386349588 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24281088a0 con 0x7f24280fd150 2024-04-12T05:21:44.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.918+0000 7f242e2c8700 1 -- 172.21.15.113:0/2386349588 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242805f590 con 0x7f24280fd150 2024-04-12T05:21:44.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.919+0000 7f2425775700 1 -- 172.21.15.113:0/2386349588 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f241c013a30 con 0x7f24280fd150 2024-04-12T05:21:44.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.919+0000 7f2425775700 1 --2- 172.21.15.113:0/2386349588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f241006caa0 0x7f241006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:44.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.919+0000 7f2425775700 1 -- 172.21.15.113:0/2386349588 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f241c08b790 con 0x7f24280fd150 2024-04-12T05:21:44.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.919+0000 7f2427fff700 1 --2- 172.21.15.113:0/2386349588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f241006caa0 0x7f241006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:44.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.919+0000 7f2427fff700 1 --2- 172.21.15.113:0/2386349588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f241006caa0 0x7f241006ef60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f2418004f10 tx=0x7f241800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:44.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:44.923+0000 7f2425775700 1 -- 172.21.15.113:0/2386349588 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f241c059240 con 0x7f24280fd150 2024-04-12T05:21:45.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.153+0000 7f242e2c8700 1 -- 172.21.15.113:0/2386349588 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2428047540 con 0x7f24280fd150 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.153+0000 7f2425775700 1 -- 172.21.15.113:0/2386349588 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f241c058dd0 con 0x7f24280fd150 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:21:45.157 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:21:45.158 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:21:45.159 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:21:45.160 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:21:45.160 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:21:45.160 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:21:45.160 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:21:45.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.155+0000 7f242e2c8700 1 -- 172.21.15.113:0/2386349588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f241006caa0 msgr2=0x7f241006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:45.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.155+0000 7f242e2c8700 1 --2- 172.21.15.113:0/2386349588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f241006caa0 0x7f241006ef60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f2418004f10 tx=0x7f241800b040).stop 2024-04-12T05:21:45.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.155+0000 7f242e2c8700 1 -- 172.21.15.113:0/2386349588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24280fd150 msgr2=0x7f242810e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:45.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.155+0000 7f242e2c8700 1 --2- 172.21.15.113:0/2386349588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24280fd150 0x7f242810e510 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f241c0041d0 tx=0x7f241c009f40).stop 2024-04-12T05:21:45.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.155+0000 7f242e2c8700 1 -- 172.21.15.113:0/2386349588 shutdown_connections 2024-04-12T05:21:45.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.155+0000 7f242e2c8700 1 --2- 172.21.15.113:0/2386349588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f241006caa0 0x7f241006ef60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.155+0000 7f242e2c8700 1 --2- 172.21.15.113:0/2386349588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24280fc7f0 0x7f242810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.155+0000 7f242e2c8700 1 --2- 172.21.15.113:0/2386349588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24280fd150 0x7f242810e510 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.155+0000 7f242e2c8700 1 -- 172.21.15.113:0/2386349588 >> 172.21.15.113:0/2386349588 conn(0x7f2428070590 msgr2=0x7f2428060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:45.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.156+0000 7f242e2c8700 1 -- 172.21.15.113:0/2386349588 shutdown_connections 2024-04-12T05:21:45.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.156+0000 7f242e2c8700 1 -- 172.21.15.113:0/2386349588 wait complete. 2024-04-12T05:21:45.162 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:21:45.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.255+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/553833296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9c40fd150 msgr2=0x7fe9c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:45.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.255+0000 7fe9cb5b7700 1 --2- 172.21.15.113:0/553833296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9c40fd150 0x7fe9c40f6930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fe9b40047f0 tx=0x7fe9b4009cb0).stop 2024-04-12T05:21:45.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.255+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/553833296 shutdown_connections 2024-04-12T05:21:45.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.255+0000 7fe9cb5b7700 1 --2- 172.21.15.113:0/553833296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9c40fd150 0x7fe9c40f6930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.255+0000 7fe9cb5b7700 1 --2- 172.21.15.113:0/553833296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9c40fc7f0 0x7fe9c40fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.255+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/553833296 >> 172.21.15.113:0/553833296 conn(0x7fe9c4070590 msgr2=0x7fe9c40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:45.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.255+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/553833296 shutdown_connections 2024-04-12T05:21:45.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.255+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/553833296 wait complete. 2024-04-12T05:21:45.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.255+0000 7fe9cb5b7700 1 Processor -- start 2024-04-12T05:21:45.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9cb5b7700 1 -- start start 2024-04-12T05:21:45.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9cb5b7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9c40fc7f0 0x7fe9c410ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:45.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9cb5b7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9c40fd150 0x7fe9c410d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:45.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9cb5b7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c410d9c0 con 0x7fe9c40fc7f0 2024-04-12T05:21:45.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9cb5b7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c410db00 con 0x7fe9c40fd150 2024-04-12T05:21:45.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9c9353700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9c40fc7f0 0x7fe9c410ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:45.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9c9353700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9c40fc7f0 0x7fe9c410ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33884/0 (socket says 172.21.15.113:33884) 2024-04-12T05:21:45.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9c9353700 1 -- 172.21.15.113:0/3195332611 learned_addr learned my addr 172.21.15.113:0/3195332611 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:45.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9c9353700 1 -- 172.21.15.113:0/3195332611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9c40fd150 msgr2=0x7fe9c410d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:45.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9c9353700 1 --2- 172.21.15.113:0/3195332611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9c40fd150 0x7fe9c410d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:21:45.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9c9353700 1 -- 172.21.15.113:0/3195332611 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b4005040 con 0x7fe9c40fc7f0 2024-04-12T05:21:45.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9c9353700 1 --2- 172.21.15.113:0/3195332611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9c40fc7f0 0x7fe9c410ced0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fe9c0002770 tx=0x7fe9c0002b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:45.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9ba7fc700 1 -- 172.21.15.113:0/3195332611 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c0002d80 con 0x7fe9c40fc7f0 2024-04-12T05:21:45.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9ba7fc700 1 -- 172.21.15.113:0/3195332611 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9c000f040 con 0x7fe9c40fc7f0 2024-04-12T05:21:45.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/3195332611 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c4107190 con 0x7fe9c40fc7f0 2024-04-12T05:21:45.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.256+0000 7fe9ba7fc700 1 -- 172.21.15.113:0/3195332611 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c000d800 con 0x7fe9c40fc7f0 2024-04-12T05:21:45.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.257+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/3195332611 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c41076d0 con 0x7fe9c40fc7f0 2024-04-12T05:21:45.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.257+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/3195332611 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c405f590 con 0x7fe9c40fc7f0 2024-04-12T05:21:45.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.258+0000 7fe9ba7fc700 1 -- 172.21.15.113:0/3195332611 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe9c0009110 con 0x7fe9c40fc7f0 2024-04-12T05:21:45.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.258+0000 7fe9ba7fc700 1 --2- 172.21.15.113:0/3195332611 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe9b006caa0 0x7fe9b006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:45.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.258+0000 7fe9ba7fc700 1 -- 172.21.15.113:0/3195332611 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe9c008bac0 con 0x7fe9c40fc7f0 2024-04-12T05:21:45.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.258+0000 7fe9c8b52700 1 --2- 172.21.15.113:0/3195332611 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe9b006caa0 0x7fe9b006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:45.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.259+0000 7fe9c8b52700 1 --2- 172.21.15.113:0/3195332611 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe9b006caa0 0x7fe9b006ef60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe9c4108810 tx=0x7fe9b4009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:45.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.263+0000 7fe9ba7fc700 1 -- 172.21.15.113:0/3195332611 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe9c0059570 con 0x7fe9c40fc7f0 2024-04-12T05:21:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:45 smithi113 ceph-mon[103371]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:45 smithi113 ceph-mon[103371]: pgmap v287: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:45 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1403773585' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:21:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:45 smithi192 ceph-mon[85304]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:45 smithi192 ceph-mon[85304]: pgmap v287: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:45 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1403773585' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:21:45.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.468+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/3195332611 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9c4107950 con 0x7fe9b006caa0 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.470+0000 7fe9ba7fc700 1 -- 172.21.15.113:0/3195332611 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe9c4107950 con 0x7fe9b006caa0 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:21:45.473 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:21:45.474 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:21:45.474 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:21:45.474 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:21:45.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.472+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/3195332611 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe9b006caa0 msgr2=0x7fe9b006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:45.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.472+0000 7fe9cb5b7700 1 --2- 172.21.15.113:0/3195332611 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe9b006caa0 0x7fe9b006ef60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe9c4108810 tx=0x7fe9b4009cb0).stop 2024-04-12T05:21:45.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.472+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/3195332611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9c40fc7f0 msgr2=0x7fe9c410ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:45.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.472+0000 7fe9cb5b7700 1 --2- 172.21.15.113:0/3195332611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9c40fc7f0 0x7fe9c410ced0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fe9c0002770 tx=0x7fe9c0002b30).stop 2024-04-12T05:21:45.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.472+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/3195332611 shutdown_connections 2024-04-12T05:21:45.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.472+0000 7fe9cb5b7700 1 --2- 172.21.15.113:0/3195332611 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe9b006caa0 0x7fe9b006ef60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.472+0000 7fe9cb5b7700 1 --2- 172.21.15.113:0/3195332611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9c40fc7f0 0x7fe9c410ced0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.472+0000 7fe9cb5b7700 1 --2- 172.21.15.113:0/3195332611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9c40fd150 0x7fe9c410d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:21:45.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.472+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/3195332611 >> 172.21.15.113:0/3195332611 conn(0x7fe9c4070590 msgr2=0x7fe9c40fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:45.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.472+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/3195332611 shutdown_connections 2024-04-12T05:21:45.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.472+0000 7fe9cb5b7700 1 -- 172.21.15.113:0/3195332611 wait complete. 2024-04-12T05:21:45.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.570+0000 7ff8626f4700 1 -- 172.21.15.113:0/189923583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff85c0fc7f0 msgr2=0x7ff85c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:45.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.570+0000 7ff8626f4700 1 --2- 172.21.15.113:0/189923583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff85c0fc7f0 0x7ff85c0fcbd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7ff84c0047f0 tx=0x7ff84c009cb0).stop 2024-04-12T05:21:45.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.570+0000 7ff8626f4700 1 -- 172.21.15.113:0/189923583 shutdown_connections 2024-04-12T05:21:45.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.570+0000 7ff8626f4700 1 --2- 172.21.15.113:0/189923583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff85c0fd150 0x7ff85c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.570+0000 7ff8626f4700 1 --2- 172.21.15.113:0/189923583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff85c0fc7f0 0x7ff85c0fcbd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.570+0000 7ff8626f4700 1 -- 172.21.15.113:0/189923583 >> 172.21.15.113:0/189923583 conn(0x7ff85c0f23b0 msgr2=0x7ff85c0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:45.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.570+0000 7ff8626f4700 1 -- 172.21.15.113:0/189923583 shutdown_connections 2024-04-12T05:21:45.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.570+0000 7ff8626f4700 1 -- 172.21.15.113:0/189923583 wait complete. 2024-04-12T05:21:45.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.571+0000 7ff8626f4700 1 Processor -- start 2024-04-12T05:21:45.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.571+0000 7ff8626f4700 1 -- start start 2024-04-12T05:21:45.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.571+0000 7ff8626f4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff85c0fc7f0 0x7ff85c10e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:45.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.571+0000 7ff8626f4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff85c0fd150 0x7ff85c10e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:45.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.571+0000 7ff8626f4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c10eb00 con 0x7ff85c0fd150 2024-04-12T05:21:45.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.571+0000 7ff8626f4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c10ec40 con 0x7ff85c0fc7f0 2024-04-12T05:21:45.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.571+0000 7ff85b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff85c0fd150 0x7ff85c10e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:45.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.571+0000 7ff85b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff85c0fd150 0x7ff85c10e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33892/0 (socket says 172.21.15.113:33892) 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.571+0000 7ff85b7fe700 1 -- 172.21.15.113:0/649143405 learned_addr learned my addr 172.21.15.113:0/649143405 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.572+0000 7ff85bfff700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff85c0fc7f0 0x7ff85c10e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.572+0000 7ff85b7fe700 1 -- 172.21.15.113:0/649143405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff85c0fc7f0 msgr2=0x7ff85c10e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.572+0000 7ff85b7fe700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff85c0fc7f0 0x7ff85c10e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.572+0000 7ff85b7fe700 1 -- 172.21.15.113:0/649143405 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c005040 con 0x7ff85c0fd150 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.572+0000 7ff85bfff700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff85c0fc7f0 0x7ff85c10e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.572+0000 7ff85b7fe700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff85c0fd150 0x7ff85c10e500 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7ff844002780 tx=0x7ff844009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.572+0000 7ff8597fa700 1 -- 172.21.15.113:0/649143405 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff844007b40 con 0x7ff85c0fd150 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.572+0000 7ff8626f4700 1 -- 172.21.15.113:0/649143405 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85c1082d0 con 0x7ff85c0fd150 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.572+0000 7ff8597fa700 1 -- 172.21.15.113:0/649143405 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff84400f040 con 0x7ff85c0fd150 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.572+0000 7ff8597fa700 1 -- 172.21.15.113:0/649143405 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff844014780 con 0x7ff85c0fd150 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.572+0000 7ff8626f4700 1 -- 172.21.15.113:0/649143405 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85c108890 con 0x7ff85c0fd150 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.573+0000 7ff8626f4700 1 -- 172.21.15.113:0/649143405 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85c060390 con 0x7ff85c0fd150 2024-04-12T05:21:45.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.573+0000 7ff8597fa700 1 -- 172.21.15.113:0/649143405 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff844009110 con 0x7ff85c0fd150 2024-04-12T05:21:45.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.574+0000 7ff8597fa700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff84806caa0 0x7ff84806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:21:45.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.574+0000 7ff8597fa700 1 -- 172.21.15.113:0/649143405 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff84408bdf0 con 0x7ff85c0fd150 2024-04-12T05:21:45.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.574+0000 7ff85bfff700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff84806caa0 0x7ff84806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:21:45.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.574+0000 7ff85bfff700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff84806caa0 0x7ff84806ef60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7ff84c004540 tx=0x7ff84c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:21:45.581 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.578+0000 7ff8597fa700 1 -- 172.21.15.113:0/649143405 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff8440598a0 con 0x7ff85c0fd150 2024-04-12T05:21:45.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.844+0000 7ff8626f4700 1 -- 172.21.15.113:0/649143405 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff85c047540 con 0x7ff85c0fd150 2024-04-12T05:21:45.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.844+0000 7ff8597fa700 1 -- 172.21.15.113:0/649143405 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ff844059430 con 0x7ff85c0fd150 2024-04-12T05:21:45.848 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:21:45.848 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:21:45.848 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:21:45.848 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:21:45.848 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:21:45.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.846+0000 7ff8626f4700 1 -- 172.21.15.113:0/649143405 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff84806caa0 msgr2=0x7ff84806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:45.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.846+0000 7ff8626f4700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff84806caa0 0x7ff84806ef60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7ff84c004540 tx=0x7ff84c00b040).stop 2024-04-12T05:21:45.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.846+0000 7ff8626f4700 1 -- 172.21.15.113:0/649143405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff85c0fd150 msgr2=0x7ff85c10e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:21:45.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.846+0000 7ff8626f4700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff85c0fd150 0x7ff85c10e500 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7ff844002780 tx=0x7ff844009ee0).stop 2024-04-12T05:21:45.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.846+0000 7ff8626f4700 1 -- 172.21.15.113:0/649143405 shutdown_connections 2024-04-12T05:21:45.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.846+0000 7ff8626f4700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff84806caa0 0x7ff84806ef60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.846+0000 7ff8626f4700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff85c0fc7f0 0x7ff85c10e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.846+0000 7ff8626f4700 1 --2- 172.21.15.113:0/649143405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff85c0fd150 0x7ff85c10e500 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:21:45.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.846+0000 7ff8626f4700 1 -- 172.21.15.113:0/649143405 >> 172.21.15.113:0/649143405 conn(0x7ff85c0f23b0 msgr2=0x7ff85c061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:21:45.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.846+0000 7ff8626f4700 1 -- 172.21.15.113:0/649143405 shutdown_connections 2024-04-12T05:21:45.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:21:45.846+0000 7ff8626f4700 1 -- 172.21.15.113:0/649143405 wait complete. 2024-04-12T05:21:46.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:46 smithi113 ceph-mon[103371]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:46.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:46 smithi113 ceph-mon[103371]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:46.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:46 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2386349588' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:21:46.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:46 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/649143405' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:21:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:46 smithi192 ceph-mon[85304]: from='client.34564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:46 smithi192 ceph-mon[85304]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:46 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2386349588' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:21:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:46 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/649143405' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:21:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:47 smithi113 ceph-mon[103371]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:47 smithi113 ceph-mon[103371]: pgmap v288: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:47 smithi192 ceph-mon[85304]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:21:47.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:47 smithi192 ceph-mon[85304]: pgmap v288: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:49 smithi113 ceph-mon[103371]: pgmap v289: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:49.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:49 smithi192 ceph-mon[85304]: pgmap v289: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:51 smithi113 ceph-mon[103371]: pgmap v290: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:51 smithi192 ceph-mon[85304]: pgmap v290: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:21:52.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:21:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:53 smithi113 ceph-mon[103371]: pgmap v291: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:53 smithi192 ceph-mon[85304]: pgmap v291: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:55 smithi113 ceph-mon[103371]: pgmap v292: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:55.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:55 smithi192 ceph-mon[85304]: pgmap v292: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:57 smithi113 ceph-mon[103371]: pgmap v293: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:57.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:57 smithi192 ceph-mon[85304]: pgmap v293: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:21:59 smithi113 ceph-mon[103371]: pgmap v294: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:21:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:21:59 smithi192 ceph-mon[85304]: pgmap v294: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:01 smithi113 ceph-mon[103371]: pgmap v295: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:01 smithi192 ceph-mon[85304]: pgmap v295: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:03 smithi113 ceph-mon[103371]: pgmap v296: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:03 smithi192 ceph-mon[85304]: pgmap v296: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:05 smithi113 ceph-mon[103371]: pgmap v297: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:05 smithi192 ceph-mon[85304]: pgmap v297: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:07 smithi113 ceph-mon[103371]: pgmap v298: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:22:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:07 smithi192 ceph-mon[85304]: pgmap v298: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:22:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:09 smithi113 ceph-mon[103371]: pgmap v299: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:09 smithi192 ceph-mon[85304]: pgmap v299: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:11 smithi113 ceph-mon[103371]: pgmap v300: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:11 smithi192 ceph-mon[85304]: pgmap v300: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:12.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:12 smithi113 ceph-mon[103371]: pgmap v301: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:12.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:12 smithi192 ceph-mon[85304]: pgmap v301: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:15.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:14 smithi192 ceph-mon[85304]: pgmap v302: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:14 smithi113 ceph-mon[103371]: pgmap v302: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:15.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.960+0000 7f914dcc6700 1 -- 172.21.15.113:0/2461682922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91480fc7f0 msgr2=0x7f91480fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:15.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.960+0000 7f914dcc6700 1 --2- 172.21.15.113:0/2461682922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91480fc7f0 0x7f91480fcbd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f913c0047f0 tx=0x7f913c009cb0).stop 2024-04-12T05:22:15.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.960+0000 7f914dcc6700 1 -- 172.21.15.113:0/2461682922 shutdown_connections 2024-04-12T05:22:15.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.960+0000 7f914dcc6700 1 --2- 172.21.15.113:0/2461682922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91480fd150 0x7f91480f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:15.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.960+0000 7f914dcc6700 1 --2- 172.21.15.113:0/2461682922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91480fc7f0 0x7f91480fcbd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:15.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.960+0000 7f914dcc6700 1 -- 172.21.15.113:0/2461682922 >> 172.21.15.113:0/2461682922 conn(0x7f9148070590 msgr2=0x7f91480f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:15.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.960+0000 7f914dcc6700 1 -- 172.21.15.113:0/2461682922 shutdown_connections 2024-04-12T05:22:15.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.960+0000 7f914dcc6700 1 -- 172.21.15.113:0/2461682922 wait complete. 2024-04-12T05:22:15.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f914dcc6700 1 Processor -- start 2024-04-12T05:22:15.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f914dcc6700 1 -- start start 2024-04-12T05:22:15.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f914dcc6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91480fc7f0 0x7f914810e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:15.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f914dcc6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91480fd150 0x7f914810e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:15.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f914dcc6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914810ec10 con 0x7f91480fd150 2024-04-12T05:22:15.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f914dcc6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914810ed50 con 0x7f91480fc7f0 2024-04-12T05:22:15.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f9146ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91480fd150 0x7f914810e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:15.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f9146ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91480fd150 0x7f914810e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52808/0 (socket says 172.21.15.113:52808) 2024-04-12T05:22:15.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f9146ffd700 1 -- 172.21.15.113:0/4050559866 learned_addr learned my addr 172.21.15.113:0/4050559866 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:15.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f91477fe700 1 --2- 172.21.15.113:0/4050559866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91480fc7f0 0x7f914810e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:15.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f9146ffd700 1 -- 172.21.15.113:0/4050559866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91480fc7f0 msgr2=0x7f914810e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:15.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f9146ffd700 1 --2- 172.21.15.113:0/4050559866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91480fc7f0 0x7f914810e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:15.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.961+0000 7f9146ffd700 1 -- 172.21.15.113:0/4050559866 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c005040 con 0x7f91480fd150 2024-04-12T05:22:15.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.962+0000 7f9146ffd700 1 --2- 172.21.15.113:0/4050559866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91480fd150 0x7f914810e610 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f913800e940 tx=0x7f913800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:15.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.962+0000 7f9144ff9700 1 -- 172.21.15.113:0/4050559866 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9138007b40 con 0x7f91480fd150 2024-04-12T05:22:15.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.962+0000 7f914dcc6700 1 -- 172.21.15.113:0/4050559866 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91481083c0 con 0x7f91480fd150 2024-04-12T05:22:15.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.962+0000 7f9144ff9700 1 -- 172.21.15.113:0/4050559866 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f913800f040 con 0x7f91480fd150 2024-04-12T05:22:15.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.962+0000 7f9144ff9700 1 -- 172.21.15.113:0/4050559866 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91380148e0 con 0x7f91480fd150 2024-04-12T05:22:15.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.962+0000 7f914dcc6700 1 -- 172.21.15.113:0/4050559866 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9148108980 con 0x7f91480fd150 2024-04-12T05:22:15.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.963+0000 7f914dcc6700 1 -- 172.21.15.113:0/4050559866 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914805f590 con 0x7f91480fd150 2024-04-12T05:22:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.963+0000 7f9144ff9700 1 -- 172.21.15.113:0/4050559866 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9138007ca0 con 0x7f91480fd150 2024-04-12T05:22:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.964+0000 7f9144ff9700 1 --2- 172.21.15.113:0/4050559866 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913006caa0 0x7f913006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.964+0000 7f9144ff9700 1 -- 172.21.15.113:0/4050559866 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f913808b860 con 0x7f91480fd150 2024-04-12T05:22:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.964+0000 7f91477fe700 1 --2- 172.21.15.113:0/4050559866 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913006caa0 0x7f913006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.964+0000 7f91477fe700 1 --2- 172.21.15.113:0/4050559866 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913006caa0 0x7f913006ef60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f913c0047c0 tx=0x7f913c009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:15.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:15.968+0000 7f9144ff9700 1 -- 172.21.15.113:0/4050559866 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9138055880 con 0x7f91480fd150 2024-04-12T05:22:16.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.172+0000 7f914dcc6700 1 -- 172.21.15.113:0/4050559866 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9148108fc0 con 0x7f913006caa0 2024-04-12T05:22:16.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.174+0000 7f9144ff9700 1 -- 172.21.15.113:0/4050559866 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9148108fc0 con 0x7f913006caa0 2024-04-12T05:22:16.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.176+0000 7f914dcc6700 1 -- 172.21.15.113:0/4050559866 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913006caa0 msgr2=0x7f913006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.176+0000 7f914dcc6700 1 --2- 172.21.15.113:0/4050559866 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913006caa0 0x7f913006ef60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f913c0047c0 tx=0x7f913c009cb0).stop 2024-04-12T05:22:16.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.176+0000 7f914dcc6700 1 -- 172.21.15.113:0/4050559866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91480fd150 msgr2=0x7f914810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.176+0000 7f914dcc6700 1 --2- 172.21.15.113:0/4050559866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91480fd150 0x7f914810e610 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f913800e940 tx=0x7f913800b040).stop 2024-04-12T05:22:16.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.176+0000 7f914dcc6700 1 -- 172.21.15.113:0/4050559866 shutdown_connections 2024-04-12T05:22:16.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.176+0000 7f914dcc6700 1 --2- 172.21.15.113:0/4050559866 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913006caa0 0x7f913006ef60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.176+0000 7f914dcc6700 1 --2- 172.21.15.113:0/4050559866 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91480fc7f0 0x7f914810e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.176+0000 7f914dcc6700 1 --2- 172.21.15.113:0/4050559866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91480fd150 0x7f914810e610 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.176+0000 7f914dcc6700 1 -- 172.21.15.113:0/4050559866 >> 172.21.15.113:0/4050559866 conn(0x7f9148070590 msgr2=0x7f9148060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:16.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.176+0000 7f914dcc6700 1 -- 172.21.15.113:0/4050559866 shutdown_connections 2024-04-12T05:22:16.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.176+0000 7f914dcc6700 1 -- 172.21.15.113:0/4050559866 wait complete. 2024-04-12T05:22:16.188 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:22:16.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.276+0000 7fcf8040e700 1 -- 172.21.15.113:0/3300301238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf780fd150 msgr2=0x7fcf780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.276+0000 7fcf8040e700 1 --2- 172.21.15.113:0/3300301238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf780fd150 0x7fcf780f6930 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fcf740047f0 tx=0x7fcf74009cb0).stop 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.276+0000 7fcf8040e700 1 -- 172.21.15.113:0/3300301238 shutdown_connections 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.276+0000 7fcf8040e700 1 --2- 172.21.15.113:0/3300301238 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf780fd150 0x7fcf780f6930 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.276+0000 7fcf8040e700 1 --2- 172.21.15.113:0/3300301238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf780fc7f0 0x7fcf780fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.276+0000 7fcf8040e700 1 -- 172.21.15.113:0/3300301238 >> 172.21.15.113:0/3300301238 conn(0x7fcf78070590 msgr2=0x7fcf780f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.276+0000 7fcf8040e700 1 -- 172.21.15.113:0/3300301238 shutdown_connections 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.276+0000 7fcf8040e700 1 -- 172.21.15.113:0/3300301238 wait complete. 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.276+0000 7fcf8040e700 1 Processor -- start 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.276+0000 7fcf8040e700 1 -- start start 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf8040e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf780fc7f0 0x7fcf7810dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf8040e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf780fd150 0x7fcf7810e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf8040e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7810eab0 con 0x7fcf780fc7f0 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf8040e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7810ebf0 con 0x7fcf780fd150 2024-04-12T05:22:16.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf7e1aa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf780fc7f0 0x7fcf7810dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf7e1aa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf780fc7f0 0x7fcf7810dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52830/0 (socket says 172.21.15.113:52830) 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf7e1aa700 1 -- 172.21.15.113:0/1867343594 learned_addr learned my addr 172.21.15.113:0/1867343594 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf7d9a9700 1 --2- 172.21.15.113:0/1867343594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf780fd150 0x7fcf7810e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf7e1aa700 1 -- 172.21.15.113:0/1867343594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf780fd150 msgr2=0x7fcf7810e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf7e1aa700 1 --2- 172.21.15.113:0/1867343594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf780fd150 0x7fcf7810e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf7e1aa700 1 -- 172.21.15.113:0/1867343594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf74005040 con 0x7fcf780fc7f0 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf7e1aa700 1 --2- 172.21.15.113:0/1867343594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf780fc7f0 0x7fcf7810dfc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fcf680041d0 tx=0x7fcf68009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf6f7fe700 1 -- 172.21.15.113:0/1867343594 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf68007b40 con 0x7fcf780fc7f0 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf6f7fe700 1 -- 172.21.15.113:0/1867343594 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf6800f040 con 0x7fcf780fc7f0 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf8040e700 1 -- 172.21.15.113:0/1867343594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf78108280 con 0x7fcf780fc7f0 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf6f7fe700 1 -- 172.21.15.113:0/1867343594 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf680147b0 con 0x7fcf780fc7f0 2024-04-12T05:22:16.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.277+0000 7fcf8040e700 1 -- 172.21.15.113:0/1867343594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf78108840 con 0x7fcf780fc7f0 2024-04-12T05:22:16.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.278+0000 7fcf8040e700 1 -- 172.21.15.113:0/1867343594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7805f590 con 0x7fcf780fc7f0 2024-04-12T05:22:16.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.279+0000 7fcf6f7fe700 1 -- 172.21.15.113:0/1867343594 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcf68009110 con 0x7fcf780fc7f0 2024-04-12T05:22:16.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.279+0000 7fcf6f7fe700 1 --2- 172.21.15.113:0/1867343594 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcf6406caa0 0x7fcf6406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:16.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.279+0000 7fcf6f7fe700 1 -- 172.21.15.113:0/1867343594 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fcf6808be50 con 0x7fcf780fc7f0 2024-04-12T05:22:16.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.279+0000 7fcf7d9a9700 1 --2- 172.21.15.113:0/1867343594 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcf6406caa0 0x7fcf6406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:16.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.279+0000 7fcf7d9a9700 1 --2- 172.21.15.113:0/1867343594 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcf6406caa0 0x7fcf6406ef60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fcf78109900 tx=0x7fcf7400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:16.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.283+0000 7fcf6f7fe700 1 -- 172.21.15.113:0/1867343594 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fcf68059900 con 0x7fcf780fc7f0 2024-04-12T05:22:16.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.490+0000 7fcf8040e700 1 -- 172.21.15.113:0/1867343594 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf78108ee0 con 0x7fcf6406caa0 2024-04-12T05:22:16.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.491+0000 7fcf6f7fe700 1 -- 172.21.15.113:0/1867343594 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcf78108ee0 con 0x7fcf6406caa0 2024-04-12T05:22:16.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.493+0000 7fcf8040e700 1 -- 172.21.15.113:0/1867343594 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcf6406caa0 msgr2=0x7fcf6406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.493+0000 7fcf8040e700 1 --2- 172.21.15.113:0/1867343594 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcf6406caa0 0x7fcf6406ef60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fcf78109900 tx=0x7fcf7400b040).stop 2024-04-12T05:22:16.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.493+0000 7fcf8040e700 1 -- 172.21.15.113:0/1867343594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf780fc7f0 msgr2=0x7fcf7810dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.493+0000 7fcf8040e700 1 --2- 172.21.15.113:0/1867343594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf780fc7f0 0x7fcf7810dfc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fcf680041d0 tx=0x7fcf68009f60).stop 2024-04-12T05:22:16.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.493+0000 7fcf8040e700 1 -- 172.21.15.113:0/1867343594 shutdown_connections 2024-04-12T05:22:16.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.493+0000 7fcf8040e700 1 --2- 172.21.15.113:0/1867343594 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcf6406caa0 0x7fcf6406ef60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.493+0000 7fcf8040e700 1 --2- 172.21.15.113:0/1867343594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf780fc7f0 0x7fcf7810dfc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.493+0000 7fcf8040e700 1 --2- 172.21.15.113:0/1867343594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf780fd150 0x7fcf7810e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.493+0000 7fcf8040e700 1 -- 172.21.15.113:0/1867343594 >> 172.21.15.113:0/1867343594 conn(0x7fcf78070590 msgr2=0x7fcf78060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:16.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.493+0000 7fcf8040e700 1 -- 172.21.15.113:0/1867343594 shutdown_connections 2024-04-12T05:22:16.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.493+0000 7fcf8040e700 1 -- 172.21.15.113:0/1867343594 wait complete. 2024-04-12T05:22:16.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.591+0000 7f7be6dc8700 1 -- 172.21.15.113:0/1689052300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be00fc7f0 msgr2=0x7f7be00fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.591+0000 7f7be6dc8700 1 --2- 172.21.15.113:0/1689052300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be00fc7f0 0x7f7be00fcbd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f7bd40047f0 tx=0x7f7bd4009cb0).stop 2024-04-12T05:22:16.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.591+0000 7f7be6dc8700 1 -- 172.21.15.113:0/1689052300 shutdown_connections 2024-04-12T05:22:16.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.591+0000 7f7be6dc8700 1 --2- 172.21.15.113:0/1689052300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7be00fd150 0x7f7be00f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.591+0000 7f7be6dc8700 1 --2- 172.21.15.113:0/1689052300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be00fc7f0 0x7f7be00fcbd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.591+0000 7f7be6dc8700 1 -- 172.21.15.113:0/1689052300 >> 172.21.15.113:0/1689052300 conn(0x7f7be0070590 msgr2=0x7f7be00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:16.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.591+0000 7f7be6dc8700 1 -- 172.21.15.113:0/1689052300 shutdown_connections 2024-04-12T05:22:16.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.591+0000 7f7be6dc8700 1 -- 172.21.15.113:0/1689052300 wait complete. 2024-04-12T05:22:16.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.592+0000 7f7be6dc8700 1 Processor -- start 2024-04-12T05:22:16.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.592+0000 7f7be6dc8700 1 -- start start 2024-04-12T05:22:16.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.592+0000 7f7be6dc8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be00fc7f0 0x7f7be01091f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:16.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.592+0000 7f7be6dc8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7be00fd150 0x7f7be01096e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:16.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.592+0000 7f7be6dc8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be0109bd0 con 0x7f7be00fc7f0 2024-04-12T05:22:16.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.592+0000 7f7be6dc8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be0109d10 con 0x7f7be00fd150 2024-04-12T05:22:16.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.592+0000 7f7be4b64700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be00fc7f0 0x7f7be01091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:16.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.592+0000 7f7be4b64700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be00fc7f0 0x7f7be01091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52842/0 (socket says 172.21.15.113:52842) 2024-04-12T05:22:16.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.592+0000 7f7be4b64700 1 -- 172.21.15.113:0/2172570754 learned_addr learned my addr 172.21.15.113:0/2172570754 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:16.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.592+0000 7f7bdffff700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7be00fd150 0x7f7be01096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:16.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.593+0000 7f7be4b64700 1 -- 172.21.15.113:0/2172570754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7be00fd150 msgr2=0x7f7be01096e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.593+0000 7f7be4b64700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7be00fd150 0x7f7be01096e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.593+0000 7f7be4b64700 1 -- 172.21.15.113:0/2172570754 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd4005040 con 0x7f7be00fc7f0 2024-04-12T05:22:16.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.593+0000 7f7bdffff700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7be00fd150 0x7f7be01096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:22:16.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.593+0000 7f7be4b64700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be00fc7f0 0x7f7be01091f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f7bd4009380 tx=0x7f7bd4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:16.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.593+0000 7f7bddffb700 1 -- 172.21.15.113:0/2172570754 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bd4018070 con 0x7f7be00fc7f0 2024-04-12T05:22:16.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.593+0000 7f7be6dc8700 1 -- 172.21.15.113:0/2172570754 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be0109f60 con 0x7f7be00fc7f0 2024-04-12T05:22:16.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.593+0000 7f7bddffb700 1 -- 172.21.15.113:0/2172570754 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bd4007e90 con 0x7f7be00fc7f0 2024-04-12T05:22:16.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.593+0000 7f7bddffb700 1 -- 172.21.15.113:0/2172570754 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bd401c380 con 0x7f7be00fc7f0 2024-04-12T05:22:16.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.593+0000 7f7be6dc8700 1 -- 172.21.15.113:0/2172570754 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be0112550 con 0x7f7be00fc7f0 2024-04-12T05:22:16.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.594+0000 7f7be6dc8700 1 -- 172.21.15.113:0/2172570754 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be005f590 con 0x7f7be00fc7f0 2024-04-12T05:22:16.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.594+0000 7f7bddffb700 1 -- 172.21.15.113:0/2172570754 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7bd40123c0 con 0x7f7be00fc7f0 2024-04-12T05:22:16.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.595+0000 7f7bddffb700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7bd006cab0 0x7f7bd006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:16.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.595+0000 7f7bddffb700 1 -- 172.21.15.113:0/2172570754 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7bd408cc60 con 0x7f7be00fc7f0 2024-04-12T05:22:16.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.595+0000 7f7bdffff700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7bd006cab0 0x7f7bd006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:16.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.595+0000 7f7bdffff700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7bd006cab0 0x7f7bd006ef70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7bcc007fd0 tx=0x7f7bcc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:16.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.599+0000 7f7bddffb700 1 -- 172.21.15.113:0/2172570754 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7bd4056c80 con 0x7f7be00fc7f0 2024-04-12T05:22:16.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.800+0000 7f7be6dc8700 1 -- 172.21.15.113:0/2172570754 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7be0059120 con 0x7f7bd006cab0 2024-04-12T05:22:16.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.810+0000 7f7bddffb700 1 -- 172.21.15.113:0/2172570754 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7be0059120 con 0x7f7bd006cab0 2024-04-12T05:22:16.812 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:22:16.812 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (10m) 3m ago 21m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:22:16.812 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (7m) 3m ago 21m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:22:16.812 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (7m) 2m ago 20m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:22:16.812 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (9m) 3m ago 21m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:22:16.812 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (3m) 3m ago 16m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:22:16.812 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (3m) 3m ago 15m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:22:16.812 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (2m) 2m ago 15m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (15m) 2m ago 15m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (11m) 3m ago 22m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (11m) 2m ago 20m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (8m) 3m ago 22m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (8m) 2m ago 20m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (10m) 3m ago 20m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (10m) 2m ago 20m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (7m) 3m ago 19m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (6m) 3m ago 19m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (5m) 3m ago 18m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (5m) 2m ago 18m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (4m) 2m ago 17m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (3m) 2m ago 17m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:22:16.813 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (10m) 2m ago 20m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:22:16.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.812+0000 7f7be6dc8700 1 -- 172.21.15.113:0/2172570754 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7bd006cab0 msgr2=0x7f7bd006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.812+0000 7f7be6dc8700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7bd006cab0 0x7f7bd006ef70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7bcc007fd0 tx=0x7f7bcc00b040).stop 2024-04-12T05:22:16.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.812+0000 7f7be6dc8700 1 -- 172.21.15.113:0/2172570754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be00fc7f0 msgr2=0x7f7be01091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.812+0000 7f7be6dc8700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be00fc7f0 0x7f7be01091f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f7bd4009380 tx=0x7f7bd4009cb0).stop 2024-04-12T05:22:16.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.812+0000 7f7be6dc8700 1 -- 172.21.15.113:0/2172570754 shutdown_connections 2024-04-12T05:22:16.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.812+0000 7f7be6dc8700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7bd006cab0 0x7f7bd006ef70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.812+0000 7f7be6dc8700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be00fc7f0 0x7f7be01091f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.812+0000 7f7be6dc8700 1 --2- 172.21.15.113:0/2172570754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7be00fd150 0x7f7be01096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.812+0000 7f7be6dc8700 1 -- 172.21.15.113:0/2172570754 >> 172.21.15.113:0/2172570754 conn(0x7f7be0070590 msgr2=0x7f7be00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:16.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.812+0000 7f7be6dc8700 1 -- 172.21.15.113:0/2172570754 shutdown_connections 2024-04-12T05:22:16.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.812+0000 7f7be6dc8700 1 -- 172.21.15.113:0/2172570754 wait complete. 2024-04-12T05:22:16.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.909+0000 7fdd2d132700 1 -- 172.21.15.113:0/1190325125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280fc800 msgr2=0x7fdd280fcbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.909+0000 7fdd2d132700 1 --2- 172.21.15.113:0/1190325125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280fc800 0x7fdd280fcbe0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fdd180047f0 tx=0x7fdd18009cb0).stop 2024-04-12T05:22:16.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.909+0000 7fdd2d132700 1 -- 172.21.15.113:0/1190325125 shutdown_connections 2024-04-12T05:22:16.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.909+0000 7fdd2d132700 1 --2- 172.21.15.113:0/1190325125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280fd160 0x7fdd280f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:22:16.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.909+0000 7fdd2d132700 1 --2- 172.21.15.113:0/1190325125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280fc800 0x7fdd280fcbe0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.909+0000 7fdd2d132700 1 -- 172.21.15.113:0/1190325125 >> 172.21.15.113:0/1190325125 conn(0x7fdd280f23c0 msgr2=0x7fdd280f47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:16.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.909+0000 7fdd2d132700 1 -- 172.21.15.113:0/1190325125 shutdown_connections 2024-04-12T05:22:16.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.909+0000 7fdd2d132700 1 -- 172.21.15.113:0/1190325125 wait complete. 2024-04-12T05:22:16.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.910+0000 7fdd2d132700 1 Processor -- start 2024-04-12T05:22:16.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.910+0000 7fdd2d132700 1 -- start start 2024-04-12T05:22:16.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd2d132700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280fc800 0x7fdd28109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:16.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd2d132700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280fd160 0x7fdd28109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd2d132700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28109be0 con 0x7fdd280fc800 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd2d132700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28109d20 con 0x7fdd280fd160 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd26d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280fc800 0x7fdd28109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd26d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280fc800 0x7fdd28109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52858/0 (socket says 172.21.15.113:52858) 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd26d9d700 1 -- 172.21.15.113:0/78200584 learned_addr learned my addr 172.21.15.113:0/78200584 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd2659c700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280fd160 0x7fdd28109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd26d9d700 1 -- 172.21.15.113:0/78200584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280fd160 msgr2=0x7fdd28109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd26d9d700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280fd160 0x7fdd28109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd26d9d700 1 -- 172.21.15.113:0/78200584 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd18005040 con 0x7fdd280fc800 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd2659c700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280fd160 0x7fdd28109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd26d9d700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280fc800 0x7fdd28109170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fdd18013640 tx=0x7fdd18009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd0ffff700 1 -- 172.21.15.113:0/78200584 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd1800e030 con 0x7fdd280fc800 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd0ffff700 1 -- 172.21.15.113:0/78200584 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd18007e90 con 0x7fdd280fc800 2024-04-12T05:22:16.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd2d132700 1 -- 172.21.15.113:0/78200584 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd28109f70 con 0x7fdd280fc800 2024-04-12T05:22:16.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd0ffff700 1 -- 172.21.15.113:0/78200584 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd18018070 con 0x7fdd280fc800 2024-04-12T05:22:16.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.911+0000 7fdd2d132700 1 -- 172.21.15.113:0/78200584 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd281124b0 con 0x7fdd280fc800 2024-04-12T05:22:16.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.912+0000 7fdd2d132700 1 -- 172.21.15.113:0/78200584 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd28060370 con 0x7fdd280fc800 2024-04-12T05:22:16.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.913+0000 7fdd0ffff700 1 -- 172.21.15.113:0/78200584 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdd1801d420 con 0x7fdd280fc800 2024-04-12T05:22:16.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.913+0000 7fdd0ffff700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006cab0 0x7fdd1006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:16.916 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.913+0000 7fdd0ffff700 1 -- 172.21.15.113:0/78200584 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fdd1808d360 con 0x7fdd280fc800 2024-04-12T05:22:16.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.913+0000 7fdd2659c700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006cab0 0x7fdd1006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:16.917 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.914+0000 7fdd2659c700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006cab0 0x7fdd1006ef70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fdd280614f0 tx=0x7fdd1c008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:16.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:16.918+0000 7fdd0ffff700 1 -- 172.21.15.113:0/78200584 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fdd18057380 con 0x7fdd280fc800 2024-04-12T05:22:17.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.185+0000 7fdd2d132700 1 -- 172.21.15.113:0/78200584 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdd28061330 con 0x7fdd280fc800 2024-04-12T05:22:17.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.186+0000 7fdd0ffff700 1 -- 172.21.15.113:0/78200584 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fdd1805a9a0 con 0x7fdd280fc800 2024-04-12T05:22:17.188 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:22:17.188 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:22:17.189 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:22:17.189 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:22:17.189 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:22:17.189 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:22:17.189 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:22:17.189 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:22:17.189 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:22:17.189 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:22:17.189 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:22:17.189 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:22:17.190 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:22:17.190 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:22:17.190 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:22:17.190 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:22:17.190 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:22:17.190 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:22:17.190 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:22:17.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.188+0000 7fdd2d132700 1 -- 172.21.15.113:0/78200584 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006cab0 msgr2=0x7fdd1006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.188+0000 7fdd2d132700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006cab0 0x7fdd1006ef70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fdd280614f0 tx=0x7fdd1c008040).stop 2024-04-12T05:22:17.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.188+0000 7fdd2d132700 1 -- 172.21.15.113:0/78200584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280fc800 msgr2=0x7fdd28109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.188+0000 7fdd2d132700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280fc800 0x7fdd28109170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fdd18013640 tx=0x7fdd18009cb0).stop 2024-04-12T05:22:17.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.188+0000 7fdd2d132700 1 -- 172.21.15.113:0/78200584 shutdown_connections 2024-04-12T05:22:17.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.188+0000 7fdd2d132700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd1006cab0 0x7fdd1006ef70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.188+0000 7fdd2d132700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd280fc800 0x7fdd28109170 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.188+0000 7fdd2d132700 1 --2- 172.21.15.113:0/78200584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd280fd160 0x7fdd28109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.188+0000 7fdd2d132700 1 -- 172.21.15.113:0/78200584 >> 172.21.15.113:0/78200584 conn(0x7fdd280f23c0 msgr2=0x7fdd280fabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.188+0000 7fdd2d132700 1 -- 172.21.15.113:0/78200584 shutdown_connections 2024-04-12T05:22:17.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.188+0000 7fdd2d132700 1 -- 172.21.15.113:0/78200584 wait complete. 2024-04-12T05:22:17.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:16 smithi192 ceph-mon[85304]: pgmap v303: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:17.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.287+0000 7f216891c700 1 -- 172.21.15.113:0/3803053518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21640fd150 msgr2=0x7f21640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.287+0000 7f216891c700 1 --2- 172.21.15.113:0/3803053518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21640fd150 0x7f21640f6930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f21540047f0 tx=0x7f2154009cb0).stop 2024-04-12T05:22:17.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.287+0000 7f216891c700 1 -- 172.21.15.113:0/3803053518 shutdown_connections 2024-04-12T05:22:17.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.287+0000 7f216891c700 1 --2- 172.21.15.113:0/3803053518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21640fd150 0x7f21640f6930 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.287+0000 7f216891c700 1 --2- 172.21.15.113:0/3803053518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21640fc7f0 0x7f21640fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.287+0000 7f216891c700 1 -- 172.21.15.113:0/3803053518 >> 172.21.15.113:0/3803053518 conn(0x7f21640f23b0 msgr2=0x7f21640f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:17.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.287+0000 7f216891c700 1 -- 172.21.15.113:0/3803053518 shutdown_connections 2024-04-12T05:22:17.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.287+0000 7f216891c700 1 -- 172.21.15.113:0/3803053518 wait complete. 2024-04-12T05:22:17.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.288+0000 7f216891c700 1 Processor -- start 2024-04-12T05:22:17.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.288+0000 7f216891c700 1 -- start start 2024-04-12T05:22:17.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.288+0000 7f216891c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21640fc7f0 0x7f216410cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:17.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.288+0000 7f216891c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21640fd150 0x7f216410d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:17.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.288+0000 7f216891c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216410da60 con 0x7f21640fc7f0 2024-04-12T05:22:17.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.288+0000 7f216891c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216410dba0 con 0x7f21640fd150 2024-04-12T05:22:17.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.288+0000 7f216259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21640fc7f0 0x7f216410cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:17.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.288+0000 7f216259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21640fc7f0 0x7f216410cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52880/0 (socket says 172.21.15.113:52880) 2024-04-12T05:22:17.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.288+0000 7f216259c700 1 -- 172.21.15.113:0/1082562281 learned_addr learned my addr 172.21.15.113:0/1082562281 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:17.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.288+0000 7f2161d9b700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21640fd150 0x7f216410d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:17.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.289+0000 7f216259c700 1 -- 172.21.15.113:0/1082562281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21640fd150 msgr2=0x7f216410d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.289+0000 7f216259c700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21640fd150 0x7f216410d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.289+0000 7f216259c700 1 -- 172.21.15.113:0/1082562281 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2154005040 con 0x7f21640fc7f0 2024-04-12T05:22:17.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.289+0000 7f2161d9b700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21640fd150 0x7f216410d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:22:17.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.289+0000 7f216259c700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21640fc7f0 0x7f216410cf70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f21580041d0 tx=0x7f2158009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:17.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.289+0000 7f21537fe700 1 -- 172.21.15.113:0/1082562281 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2158002d90 con 0x7f21640fc7f0 2024-04-12T05:22:17.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.289+0000 7f21537fe700 1 -- 172.21.15.113:0/1082562281 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f215800f040 con 0x7f21640fc7f0 2024-04-12T05:22:17.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.289+0000 7f216891c700 1 -- 172.21.15.113:0/1082562281 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2164107230 con 0x7f21640fc7f0 2024-04-12T05:22:17.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.289+0000 7f21537fe700 1 -- 172.21.15.113:0/1082562281 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f215800d800 con 0x7f21640fc7f0 2024-04-12T05:22:17.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.289+0000 7f216891c700 1 -- 172.21.15.113:0/1082562281 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2164107770 con 0x7f21640fc7f0 2024-04-12T05:22:17.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.290+0000 7f216891c700 1 -- 172.21.15.113:0/1082562281 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2164060370 con 0x7f21640fc7f0 2024-04-12T05:22:17.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.290+0000 7f21537fe700 1 -- 172.21.15.113:0/1082562281 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f215800dac0 con 0x7f21640fc7f0 2024-04-12T05:22:17.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.291+0000 7f21537fe700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f214c06cab0 0x7f214c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:17.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.291+0000 7f21537fe700 1 -- 172.21.15.113:0/1082562281 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f215808bea0 con 0x7f21640fc7f0 2024-04-12T05:22:17.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.291+0000 7f2161d9b700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f214c06cab0 0x7f214c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:17.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.291+0000 7f2161d9b700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f214c06cab0 0x7f214c06ef70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f21641088b0 tx=0x7f2154009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:17.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.295+0000 7f21537fe700 1 -- 172.21.15.113:0/1082562281 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2158059950 con 0x7f21640fc7f0 2024-04-12T05:22:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:16 smithi113 ceph-mon[103371]: pgmap v303: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:17.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.524+0000 7f216891c700 1 -- 172.21.15.113:0/1082562281 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f21641086f0 con 0x7f21640fc7f0 2024-04-12T05:22:17.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.524+0000 7f21537fe700 1 -- 172.21.15.113:0/1082562281 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f21580594e0 con 0x7f21640fc7f0 2024-04-12T05:22:17.527 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:22:17.527 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:22:17.527 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:22:17.527 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:22:17.527 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:22:17.527 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:22:17.527 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:22:17.527 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:22:17.528 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:22:17.529 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:22:17.530 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:22:17.530 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:22:17.530 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:22:17.530 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:22:17.530 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:22:17.530 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:22:17.530 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:22:17.530 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:22:17.530 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:22:17.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.526+0000 7f216891c700 1 -- 172.21.15.113:0/1082562281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f214c06cab0 msgr2=0x7f214c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.526+0000 7f216891c700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f214c06cab0 0x7f214c06ef70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f21641088b0 tx=0x7f2154009cb0).stop 2024-04-12T05:22:17.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.526+0000 7f216891c700 1 -- 172.21.15.113:0/1082562281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21640fc7f0 msgr2=0x7f216410cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.526+0000 7f216891c700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21640fc7f0 0x7f216410cf70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f21580041d0 tx=0x7f2158009f60).stop 2024-04-12T05:22:17.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.526+0000 7f216891c700 1 -- 172.21.15.113:0/1082562281 shutdown_connections 2024-04-12T05:22:17.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.526+0000 7f216891c700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f214c06cab0 0x7f214c06ef70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.526+0000 7f216891c700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21640fc7f0 0x7f216410cf70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.526+0000 7f216891c700 1 --2- 172.21.15.113:0/1082562281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21640fd150 0x7f216410d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.526+0000 7f216891c700 1 -- 172.21.15.113:0/1082562281 >> 172.21.15.113:0/1082562281 conn(0x7f21640f23b0 msgr2=0x7f21640fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:17.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.526+0000 7f216891c700 1 -- 172.21.15.113:0/1082562281 shutdown_connections 2024-04-12T05:22:17.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.526+0000 7f216891c700 1 -- 172.21.15.113:0/1082562281 wait complete. 2024-04-12T05:22:17.532 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:22:17.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.626+0000 7fc710116700 1 -- 172.21.15.113:0/2344978787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7080fc7f0 msgr2=0x7fc7080fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.626+0000 7fc710116700 1 --2- 172.21.15.113:0/2344978787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7080fc7f0 0x7fc7080fcbd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fc6f80047f0 tx=0x7fc6f8009cb0).stop 2024-04-12T05:22:17.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.626+0000 7fc710116700 1 -- 172.21.15.113:0/2344978787 shutdown_connections 2024-04-12T05:22:17.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.626+0000 7fc710116700 1 --2- 172.21.15.113:0/2344978787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7080fd150 0x7fc7080f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.626+0000 7fc710116700 1 --2- 172.21.15.113:0/2344978787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7080fc7f0 0x7fc7080fcbd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.626+0000 7fc710116700 1 -- 172.21.15.113:0/2344978787 >> 172.21.15.113:0/2344978787 conn(0x7fc708070590 msgr2=0x7fc7080f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:17.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.626+0000 7fc710116700 1 -- 172.21.15.113:0/2344978787 shutdown_connections 2024-04-12T05:22:17.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.626+0000 7fc710116700 1 -- 172.21.15.113:0/2344978787 wait complete. 2024-04-12T05:22:17.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.626+0000 7fc710116700 1 Processor -- start 2024-04-12T05:22:17.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.626+0000 7fc710116700 1 -- start start 2024-04-12T05:22:17.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc710116700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7080fc7f0 0x7fc70810dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:17.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc710116700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7080fd150 0x7fc70810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc710116700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70810eae0 con 0x7fc7080fc7f0 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc710116700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70810ec20 con 0x7fc7080fd150 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc70deb2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7080fc7f0 0x7fc70810dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc70deb2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7080fc7f0 0x7fc70810dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52894/0 (socket says 172.21.15.113:52894) 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc70deb2700 1 -- 172.21.15.113:0/2916328458 learned_addr learned my addr 172.21.15.113:0/2916328458 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc70deb2700 1 -- 172.21.15.113:0/2916328458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7080fd150 msgr2=0x7fc70810e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc70deb2700 1 --2- 172.21.15.113:0/2916328458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7080fd150 0x7fc70810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc70deb2700 1 -- 172.21.15.113:0/2916328458 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f8005040 con 0x7fc7080fc7f0 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc70deb2700 1 --2- 172.21.15.113:0/2916328458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7080fc7f0 0x7fc70810dff0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fc6f800b120 tx=0x7fc6f8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc6feffd700 1 -- 172.21.15.113:0/2916328458 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6f800e030 con 0x7fc7080fc7f0 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc6feffd700 1 -- 172.21.15.113:0/2916328458 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6f8007e90 con 0x7fc7080fc7f0 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc710116700 1 -- 172.21.15.113:0/2916328458 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc708108250 con 0x7fc7080fc7f0 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc6feffd700 1 -- 172.21.15.113:0/2916328458 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6f8018070 con 0x7fc7080fc7f0 2024-04-12T05:22:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.627+0000 7fc710116700 1 -- 172.21.15.113:0/2916328458 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7081087b0 con 0x7fc7080fc7f0 2024-04-12T05:22:17.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.628+0000 7fc710116700 1 -- 172.21.15.113:0/2916328458 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70805f590 con 0x7fc7080fc7f0 2024-04-12T05:22:17.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.629+0000 7fc6feffd700 1 -- 172.21.15.113:0/2916328458 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc6f801d420 con 0x7fc7080fc7f0 2024-04-12T05:22:17.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.629+0000 7fc6feffd700 1 --2- 172.21.15.113:0/2916328458 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6f406caa0 0x7fc6f406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:17.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.629+0000 7fc6feffd700 1 -- 172.21.15.113:0/2916328458 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc6f808d300 con 0x7fc7080fc7f0 2024-04-12T05:22:17.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.629+0000 7fc70d6b1700 1 --2- 172.21.15.113:0/2916328458 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6f406caa0 0x7fc6f406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:17.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.630+0000 7fc70d6b1700 1 --2- 172.21.15.113:0/2916328458 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6f406caa0 0x7fc6f406ef60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc708109930 tx=0x7fc704007b40).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:17.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.634+0000 7fc6feffd700 1 -- 172.21.15.113:0/2916328458 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc6f805adb0 con 0x7fc7080fc7f0 2024-04-12T05:22:17.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.839+0000 7fc710116700 1 -- 172.21.15.113:0/2916328458 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc708109780 con 0x7fc6f406caa0 2024-04-12T05:22:17.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.841+0000 7fc6feffd700 1 -- 172.21.15.113:0/2916328458 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc708109780 con 0x7fc6f406caa0 2024-04-12T05:22:17.843 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:22:17.843 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:22:17.843 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:22:17.843 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:22:17.843 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:22:17.843 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:22:17.843 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:22:17.844 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:22:17.844 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:22:17.844 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:22:17.844 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:22:17.844 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:22:17.844 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:22:17.844 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:22:17.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.843+0000 7fc710116700 1 -- 172.21.15.113:0/2916328458 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6f406caa0 msgr2=0x7fc6f406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.843+0000 7fc710116700 1 --2- 172.21.15.113:0/2916328458 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6f406caa0 0x7fc6f406ef60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc708109930 tx=0x7fc704007b40).stop 2024-04-12T05:22:17.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.843+0000 7fc710116700 1 -- 172.21.15.113:0/2916328458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7080fc7f0 msgr2=0x7fc70810dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.843+0000 7fc710116700 1 --2- 172.21.15.113:0/2916328458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7080fc7f0 0x7fc70810dff0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fc6f800b120 tx=0x7fc6f8009cb0).stop 2024-04-12T05:22:17.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.843+0000 7fc710116700 1 -- 172.21.15.113:0/2916328458 shutdown_connections 2024-04-12T05:22:17.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.843+0000 7fc710116700 1 --2- 172.21.15.113:0/2916328458 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6f406caa0 0x7fc6f406ef60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.843+0000 7fc710116700 1 --2- 172.21.15.113:0/2916328458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7080fc7f0 0x7fc70810dff0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.843+0000 7fc710116700 1 --2- 172.21.15.113:0/2916328458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7080fd150 0x7fc70810e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:22:17.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.843+0000 7fc710116700 1 -- 172.21.15.113:0/2916328458 >> 172.21.15.113:0/2916328458 conn(0x7fc708070590 msgr2=0x7fc708060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:17.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.843+0000 7fc710116700 1 -- 172.21.15.113:0/2916328458 shutdown_connections 2024-04-12T05:22:17.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.843+0000 7fc710116700 1 -- 172.21.15.113:0/2916328458 wait complete. 2024-04-12T05:22:17.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.944+0000 7f5caf99c700 1 -- 172.21.15.113:0/3629664765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca80fc7f0 msgr2=0x7f5ca80fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.944+0000 7f5caf99c700 1 --2- 172.21.15.113:0/3629664765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca80fc7f0 0x7f5ca80fcbd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f5ca40047f0 tx=0x7f5ca4009cb0).stop 2024-04-12T05:22:17.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.944+0000 7f5caf99c700 1 -- 172.21.15.113:0/3629664765 shutdown_connections 2024-04-12T05:22:17.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.944+0000 7f5caf99c700 1 --2- 172.21.15.113:0/3629664765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ca80fd150 0x7f5ca80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.944+0000 7f5caf99c700 1 --2- 172.21.15.113:0/3629664765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca80fc7f0 0x7f5ca80fcbd0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.944+0000 7f5caf99c700 1 -- 172.21.15.113:0/3629664765 >> 172.21.15.113:0/3629664765 conn(0x7f5ca8070590 msgr2=0x7f5ca80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:17.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.944+0000 7f5caf99c700 1 -- 172.21.15.113:0/3629664765 shutdown_connections 2024-04-12T05:22:17.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.944+0000 7f5caf99c700 1 -- 172.21.15.113:0/3629664765 wait complete. 2024-04-12T05:22:17.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.945+0000 7f5caf99c700 1 Processor -- start 2024-04-12T05:22:17.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.945+0000 7f5caf99c700 1 -- start start 2024-04-12T05:22:17.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.945+0000 7f5caf99c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ca80fc7f0 0x7f5ca810d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:17.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.945+0000 7f5caf99c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca80fd150 0x7f5ca810d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:17.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.945+0000 7f5caf99c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca810db10 con 0x7f5ca80fd150 2024-04-12T05:22:17.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.945+0000 7f5caf99c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca810dc50 con 0x7f5ca80fc7f0 2024-04-12T05:22:17.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.945+0000 7f5cacf37700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca80fd150 0x7f5ca810d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:17.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.945+0000 7f5cacf37700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca80fd150 0x7f5ca810d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52922/0 (socket says 172.21.15.113:52922) 2024-04-12T05:22:17.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.945+0000 7f5cacf37700 1 -- 172.21.15.113:0/2567023086 learned_addr learned my addr 172.21.15.113:0/2567023086 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:17.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.945+0000 7f5cad738700 1 --2- 172.21.15.113:0/2567023086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ca80fc7f0 0x7f5ca810d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:17.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.946+0000 7f5cacf37700 1 -- 172.21.15.113:0/2567023086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ca80fc7f0 msgr2=0x7f5ca810d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:17.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.946+0000 7f5cacf37700 1 --2- 172.21.15.113:0/2567023086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ca80fc7f0 0x7f5ca810d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:17.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.946+0000 7f5cacf37700 1 -- 172.21.15.113:0/2567023086 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca4005040 con 0x7f5ca80fd150 2024-04-12T05:22:17.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.946+0000 7f5cacf37700 1 --2- 172.21.15.113:0/2567023086 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca80fd150 0x7f5ca810d510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f5c9800e940 tx=0x7f5c9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:17.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.950+0000 7f5c9e7fc700 1 -- 172.21.15.113:0/2567023086 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c9800ee80 con 0x7f5ca80fd150 2024-04-12T05:22:17.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.950+0000 7f5caf99c700 1 -- 172.21.15.113:0/2567023086 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca81072c0 con 0x7f5ca80fd150 2024-04-12T05:22:17.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.950+0000 7f5c9e7fc700 1 -- 172.21.15.113:0/2567023086 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c9800f040 con 0x7f5ca80fd150 2024-04-12T05:22:17.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.950+0000 7f5c9e7fc700 1 -- 172.21.15.113:0/2567023086 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c9800c810 con 0x7f5ca80fd150 2024-04-12T05:22:17.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.950+0000 7f5caf99c700 1 -- 172.21.15.113:0/2567023086 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca8107880 con 0x7f5ca80fd150 2024-04-12T05:22:17.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.951+0000 7f5caf99c700 1 -- 172.21.15.113:0/2567023086 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca805f590 con 0x7f5ca80fd150 2024-04-12T05:22:17.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.952+0000 7f5c9e7fc700 1 -- 172.21.15.113:0/2567023086 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5c980090d0 con 0x7f5ca80fd150 2024-04-12T05:22:17.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.952+0000 7f5c9e7fc700 1 --2- 172.21.15.113:0/2567023086 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5c9406ca40 0x7f5c9406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:17.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.952+0000 7f5c9e7fc700 1 -- 172.21.15.113:0/2567023086 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5c9808c0c0 con 0x7f5ca80fd150 2024-04-12T05:22:17.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.952+0000 7f5cad738700 1 --2- 172.21.15.113:0/2567023086 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5c9406ca40 0x7f5c9406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:17.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.953+0000 7f5cad738700 1 --2- 172.21.15.113:0/2567023086 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5c9406ca40 0x7f5c9406ef00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f5ca4004e60 tx=0x7f5ca400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:17.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:17.956+0000 7f5c9e7fc700 1 -- 172.21.15.113:0/2567023086 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5c980561a0 con 0x7f5ca80fd150 2024-04-12T05:22:18.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:17 smithi192 ceph-mon[85304]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:18.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:17 smithi192 ceph-mon[85304]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:18.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:17 smithi192 ceph-mon[85304]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:18.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:17 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/78200584' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:22:18.199 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:17 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1082562281' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:22:18.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.221+0000 7f5caf99c700 1 -- 172.21.15.113:0/2567023086 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5ca8047540 con 0x7f5ca80fd150 2024-04-12T05:22:18.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.221+0000 7f5c9e7fc700 1 -- 172.21.15.113:0/2567023086 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f5c980597c0 con 0x7f5ca80fd150 2024-04-12T05:22:18.223 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:22:18.223 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:22:18.224 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:22:18.224 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:22:18.224 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:22:18.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.223+0000 7f5caf99c700 1 -- 172.21.15.113:0/2567023086 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5c9406ca40 msgr2=0x7f5c9406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:18.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.223+0000 7f5caf99c700 1 --2- 172.21.15.113:0/2567023086 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5c9406ca40 0x7f5c9406ef00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f5ca4004e60 tx=0x7f5ca400b040).stop 2024-04-12T05:22:18.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.223+0000 7f5caf99c700 1 -- 172.21.15.113:0/2567023086 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca80fd150 msgr2=0x7f5ca810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:18.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.223+0000 7f5caf99c700 1 --2- 172.21.15.113:0/2567023086 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca80fd150 0x7f5ca810d510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f5c9800e940 tx=0x7f5c9800b040).stop 2024-04-12T05:22:18.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.223+0000 7f5caf99c700 1 -- 172.21.15.113:0/2567023086 shutdown_connections 2024-04-12T05:22:18.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.223+0000 7f5caf99c700 1 --2- 172.21.15.113:0/2567023086 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5c9406ca40 0x7f5c9406ef00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:18.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.223+0000 7f5caf99c700 1 --2- 172.21.15.113:0/2567023086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ca80fc7f0 0x7f5ca810d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:18.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.223+0000 7f5caf99c700 1 --2- 172.21.15.113:0/2567023086 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ca80fd150 0x7f5ca810d510 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:18.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.223+0000 7f5caf99c700 1 -- 172.21.15.113:0/2567023086 >> 172.21.15.113:0/2567023086 conn(0x7f5ca8070590 msgr2=0x7f5ca80fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:18.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.223+0000 7f5caf99c700 1 -- 172.21.15.113:0/2567023086 shutdown_connections 2024-04-12T05:22:18.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:18.224+0000 7f5caf99c700 1 -- 172.21.15.113:0/2567023086 wait complete. 2024-04-12T05:22:18.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:17 smithi113 ceph-mon[103371]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:18.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:17 smithi113 ceph-mon[103371]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:18.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:17 smithi113 ceph-mon[103371]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:18.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:17 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/78200584' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:22:18.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:17 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1082562281' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:22:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:18 smithi113 ceph-mon[103371]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:18 smithi113 ceph-mon[103371]: pgmap v304: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:18 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2567023086' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:22:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:18 smithi192 ceph-mon[85304]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:18 smithi192 ceph-mon[85304]: pgmap v304: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:18 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2567023086' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:22:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:20 smithi113 ceph-mon[103371]: pgmap v305: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:20 smithi192 ceph-mon[85304]: pgmap v305: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:22.270 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:22:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:22:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:22 smithi113 ceph-mon[103371]: pgmap v306: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:22 smithi192 ceph-mon[85304]: pgmap v306: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:24 smithi113 ceph-mon[103371]: pgmap v307: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:24 smithi192 ceph-mon[85304]: pgmap v307: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:26 smithi113 ceph-mon[103371]: pgmap v308: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:26 smithi192 ceph-mon[85304]: pgmap v308: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:28 smithi113 ceph-mon[103371]: pgmap v309: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:28 smithi192 ceph-mon[85304]: pgmap v309: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:30 smithi113 ceph-mon[103371]: pgmap v310: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:31.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:30 smithi192 ceph-mon[85304]: pgmap v310: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:32 smithi113 ceph-mon[103371]: pgmap v311: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:33.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:32 smithi192 ceph-mon[85304]: pgmap v311: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:34 smithi113 ceph-mon[103371]: pgmap v312: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:35.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:34 smithi192 ceph-mon[85304]: pgmap v312: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:36 smithi113 ceph-mon[103371]: pgmap v313: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:22:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:36 smithi192 ceph-mon[85304]: pgmap v313: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:22:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:38 smithi113 ceph-mon[103371]: pgmap v314: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:38 smithi192 ceph-mon[85304]: pgmap v314: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:40 smithi113 ceph-mon[103371]: pgmap v315: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:40 smithi192 ceph-mon[85304]: pgmap v315: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:42 smithi113 ceph-mon[103371]: pgmap v316: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:43.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:42 smithi192 ceph-mon[85304]: pgmap v316: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:44 smithi113 ceph-mon[103371]: pgmap v317: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:44 smithi192 ceph-mon[85304]: pgmap v317: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:46 smithi113 ceph-mon[103371]: pgmap v318: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:47.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:46 smithi192 ceph-mon[85304]: pgmap v318: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:48.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.335+0000 7fc553e44700 1 -- 172.21.15.113:0/1720313286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc54c0fc7f0 msgr2=0x7fc54c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:48.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.335+0000 7fc553e44700 1 --2- 172.21.15.113:0/1720313286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc54c0fc7f0 0x7fc54c0fcbd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fc5480047f0 tx=0x7fc548009cb0).stop 2024-04-12T05:22:48.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.335+0000 7fc553e44700 1 -- 172.21.15.113:0/1720313286 shutdown_connections 2024-04-12T05:22:48.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.335+0000 7fc553e44700 1 --2- 172.21.15.113:0/1720313286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c0fd150 0x7fc54c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.335+0000 7fc553e44700 1 --2- 172.21.15.113:0/1720313286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc54c0fc7f0 0x7fc54c0fcbd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.335+0000 7fc553e44700 1 -- 172.21.15.113:0/1720313286 >> 172.21.15.113:0/1720313286 conn(0x7fc54c070590 msgr2=0x7fc54c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:48.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.335+0000 7fc553e44700 1 -- 172.21.15.113:0/1720313286 shutdown_connections 2024-04-12T05:22:48.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.335+0000 7fc553e44700 1 -- 172.21.15.113:0/1720313286 wait complete. 2024-04-12T05:22:48.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.336+0000 7fc553e44700 1 Processor -- start 2024-04-12T05:22:48.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.336+0000 7fc553e44700 1 -- start start 2024-04-12T05:22:48.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.336+0000 7fc553e44700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc54c0fc7f0 0x7fc54c10cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:48.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.336+0000 7fc553e44700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c0fd150 0x7fc54c10d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:48.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.336+0000 7fc553e44700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c10d9f0 con 0x7fc54c0fc7f0 2024-04-12T05:22:48.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.336+0000 7fc553e44700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c10db30 con 0x7fc54c0fd150 2024-04-12T05:22:48.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.336+0000 7fc551be0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc54c0fc7f0 0x7fc54c10cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:48.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc551be0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc54c0fc7f0 0x7fc54c10cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59802/0 (socket says 172.21.15.113:59802) 2024-04-12T05:22:48.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc551be0700 1 -- 172.21.15.113:0/929361779 learned_addr learned my addr 172.21.15.113:0/929361779 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc5513df700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c0fd150 0x7fc54c10d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc551be0700 1 -- 172.21.15.113:0/929361779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c0fd150 msgr2=0x7fc54c10d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc551be0700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c0fd150 0x7fc54c10d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc551be0700 1 -- 172.21.15.113:0/929361779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc548005040 con 0x7fc54c0fc7f0 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc5513df700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c0fd150 0x7fc54c10d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc551be0700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc54c0fc7f0 0x7fc54c10cf00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fc54800b120 tx=0x7fc548009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc542ffd700 1 -- 172.21.15.113:0/929361779 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc54800e030 con 0x7fc54c0fc7f0 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc553e44700 1 -- 172.21.15.113:0/929361779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54c107160 con 0x7fc54c0fc7f0 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc542ffd700 1 -- 172.21.15.113:0/929361779 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc548007e90 con 0x7fc54c0fc7f0 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc542ffd700 1 -- 172.21.15.113:0/929361779 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc548018070 con 0x7fc54c0fc7f0 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.337+0000 7fc553e44700 1 -- 172.21.15.113:0/929361779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54c1076c0 con 0x7fc54c0fc7f0 2024-04-12T05:22:48.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.338+0000 7fc553e44700 1 -- 172.21.15.113:0/929361779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54c05f590 con 0x7fc54c0fc7f0 2024-04-12T05:22:48.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.339+0000 7fc542ffd700 1 -- 172.21.15.113:0/929361779 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc54801d420 con 0x7fc54c0fc7f0 2024-04-12T05:22:48.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.339+0000 7fc542ffd700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc53806caa0 0x7fc53806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:48.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.339+0000 7fc542ffd700 1 -- 172.21.15.113:0/929361779 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc54808d320 con 0x7fc54c0fc7f0 2024-04-12T05:22:48.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.339+0000 7fc5513df700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc53806caa0 0x7fc53806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:48.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.340+0000 7fc5513df700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc53806caa0 0x7fc53806ef60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fc54c108840 tx=0x7fc53c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:48.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.344+0000 7fc542ffd700 1 -- 172.21.15.113:0/929361779 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc54805add0 con 0x7fc54c0fc7f0 2024-04-12T05:22:48.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.548+0000 7fc553e44700 1 -- 172.21.15.113:0/929361779 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc54c108620 con 0x7fc53806caa0 2024-04-12T05:22:48.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.550+0000 7fc542ffd700 1 -- 172.21.15.113:0/929361779 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc54c108620 con 0x7fc53806caa0 2024-04-12T05:22:48.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.552+0000 7fc553e44700 1 -- 172.21.15.113:0/929361779 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc53806caa0 msgr2=0x7fc53806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:48.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.552+0000 7fc553e44700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc53806caa0 0x7fc53806ef60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fc54c108840 tx=0x7fc53c009040).stop 2024-04-12T05:22:48.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.552+0000 7fc553e44700 1 -- 172.21.15.113:0/929361779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc54c0fc7f0 msgr2=0x7fc54c10cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:48.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.552+0000 7fc553e44700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc54c0fc7f0 0x7fc54c10cf00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fc54800b120 tx=0x7fc548009cb0).stop 2024-04-12T05:22:48.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.552+0000 7fc553e44700 1 -- 172.21.15.113:0/929361779 shutdown_connections 2024-04-12T05:22:48.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.552+0000 7fc553e44700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc53806caa0 0x7fc53806ef60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.552+0000 7fc553e44700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc54c0fc7f0 0x7fc54c10cf00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.552+0000 7fc553e44700 1 --2- 172.21.15.113:0/929361779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc54c0fd150 0x7fc54c10d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.552+0000 7fc553e44700 1 -- 172.21.15.113:0/929361779 >> 172.21.15.113:0/929361779 conn(0x7fc54c070590 msgr2=0x7fc54c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:48.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.552+0000 7fc553e44700 1 -- 172.21.15.113:0/929361779 shutdown_connections 2024-04-12T05:22:48.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.552+0000 7fc553e44700 1 -- 172.21.15.113:0/929361779 wait complete. 2024-04-12T05:22:48.564 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:22:48.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.652+0000 7f07a2405700 1 -- 172.21.15.113:0/2313477896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f079c0fe270 msgr2=0x7f079c0f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:48.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.652+0000 7f07a2405700 1 --2- 172.21.15.113:0/2313477896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f079c0fe270 0x7f079c0f7a50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f07840047f0 tx=0x7f0784009cb0).stop 2024-04-12T05:22:48.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.652+0000 7f07a2405700 1 -- 172.21.15.113:0/2313477896 shutdown_connections 2024-04-12T05:22:48.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.652+0000 7f07a2405700 1 --2- 172.21.15.113:0/2313477896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f079c0fe270 0x7f079c0f7a50 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.652+0000 7f07a2405700 1 --2- 172.21.15.113:0/2313477896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c0fd910 0x7f079c0fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.652+0000 7f07a2405700 1 -- 172.21.15.113:0/2313477896 >> 172.21.15.113:0/2313477896 conn(0x7f079c0f3530 msgr2=0x7f079c0f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:48.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.652+0000 7f07a2405700 1 -- 172.21.15.113:0/2313477896 shutdown_connections 2024-04-12T05:22:48.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.652+0000 7f07a2405700 1 -- 172.21.15.113:0/2313477896 wait complete. 2024-04-12T05:22:48.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.653+0000 7f07a2405700 1 Processor -- start 2024-04-12T05:22:48.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.653+0000 7f07a2405700 1 -- start start 2024-04-12T05:22:48.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.653+0000 7f07a2405700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f079c0fd910 0x7f079c10dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:48.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.653+0000 7f07a2405700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c0fe270 0x7f079c10e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:48.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.653+0000 7f07a2405700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c10eae0 con 0x7f079c0fd910 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.653+0000 7f07a2405700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c10ec20 con 0x7f079c0fe270 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.653+0000 7f079bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f079c0fd910 0x7f079c10dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.653+0000 7f079bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f079c0fd910 0x7f079c10dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59816/0 (socket says 172.21.15.113:59816) 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.653+0000 7f079bfff700 1 -- 172.21.15.113:0/1474266918 learned_addr learned my addr 172.21.15.113:0/1474266918 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.654+0000 7f079b7fe700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c0fe270 0x7f079c10e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.654+0000 7f079bfff700 1 -- 172.21.15.113:0/1474266918 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c0fe270 msgr2=0x7f079c10e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.654+0000 7f079bfff700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c0fe270 0x7f079c10e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.654+0000 7f079bfff700 1 -- 172.21.15.113:0/1474266918 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0784005040 con 0x7f079c0fd910 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.654+0000 7f079b7fe700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c0fe270 0x7f079c10e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.654+0000 7f079bfff700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f079c0fd910 0x7f079c10dff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f078c0041d0 tx=0x7f078c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.654+0000 7f07997fa700 1 -- 172.21.15.113:0/1474266918 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f078c002d90 con 0x7f079c0fd910 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.654+0000 7f07997fa700 1 -- 172.21.15.113:0/1474266918 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f078c00f040 con 0x7f079c0fd910 2024-04-12T05:22:48.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.654+0000 7f07a2405700 1 -- 172.21.15.113:0/1474266918 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079c1082b0 con 0x7f079c0fd910 2024-04-12T05:22:48.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.654+0000 7f07997fa700 1 -- 172.21.15.113:0/1474266918 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f078c00d800 con 0x7f079c0fd910 2024-04-12T05:22:48.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.654+0000 7f07a2405700 1 -- 172.21.15.113:0/1474266918 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079c108870 con 0x7f079c0fd910 2024-04-12T05:22:48.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.655+0000 7f07a2405700 1 -- 172.21.15.113:0/1474266918 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079c05f570 con 0x7f079c0fd910 2024-04-12T05:22:48.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.655+0000 7f07997fa700 1 -- 172.21.15.113:0/1474266918 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f078c00da70 con 0x7f079c0fd910 2024-04-12T05:22:48.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.656+0000 7f07997fa700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f078806caa0 0x7f078806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:48.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.656+0000 7f07997fa700 1 -- 172.21.15.113:0/1474266918 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f078c08bdd0 con 0x7f079c0fd910 2024-04-12T05:22:48.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.656+0000 7f079b7fe700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f078806caa0 0x7f078806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:48.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.656+0000 7f079b7fe700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f078806caa0 0x7f078806ef60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f07840043d0 tx=0x7f078400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:48.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.660+0000 7f07997fa700 1 -- 172.21.15.113:0/1474266918 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f078c059880 con 0x7f079c0fd910 2024-04-12T05:22:48.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.867+0000 7f07a2405700 1 -- 172.21.15.113:0/1474266918 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f079c108b40 con 0x7f078806caa0 2024-04-12T05:22:48.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.869+0000 7f07997fa700 1 -- 172.21.15.113:0/1474266918 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f079c108b40 con 0x7f078806caa0 2024-04-12T05:22:48.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.871+0000 7f07a2405700 1 -- 172.21.15.113:0/1474266918 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f078806caa0 msgr2=0x7f078806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:48.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.871+0000 7f07a2405700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f078806caa0 0x7f078806ef60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f07840043d0 tx=0x7f078400b040).stop 2024-04-12T05:22:48.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.871+0000 7f07a2405700 1 -- 172.21.15.113:0/1474266918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f079c0fd910 msgr2=0x7f079c10dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:48.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.871+0000 7f07a2405700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f079c0fd910 0x7f079c10dff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f078c0041d0 tx=0x7f078c009f60).stop 2024-04-12T05:22:48.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.871+0000 7f07a2405700 1 -- 172.21.15.113:0/1474266918 shutdown_connections 2024-04-12T05:22:48.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.871+0000 7f07a2405700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f078806caa0 0x7f078806ef60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.871+0000 7f07a2405700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f079c0fd910 0x7f079c10dff0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.871+0000 7f07a2405700 1 --2- 172.21.15.113:0/1474266918 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c0fe270 0x7f079c10e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.871+0000 7f07a2405700 1 -- 172.21.15.113:0/1474266918 >> 172.21.15.113:0/1474266918 conn(0x7f079c0f3530 msgr2=0x7f079c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:48.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.871+0000 7f07a2405700 1 -- 172.21.15.113:0/1474266918 shutdown_connections 2024-04-12T05:22:48.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.871+0000 7f07a2405700 1 -- 172.21.15.113:0/1474266918 wait complete. 2024-04-12T05:22:48.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.969+0000 7f135a439700 1 -- 172.21.15.113:0/1942884937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13540fd150 msgr2=0x7f13540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:48.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.969+0000 7f135a439700 1 --2- 172.21.15.113:0/1942884937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13540fd150 0x7f13540f6930 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f13480047f0 tx=0x7f1348009cb0).stop 2024-04-12T05:22:48.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.969+0000 7f135a439700 1 -- 172.21.15.113:0/1942884937 shutdown_connections 2024-04-12T05:22:48.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.969+0000 7f135a439700 1 --2- 172.21.15.113:0/1942884937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13540fd150 0x7f13540f6930 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.969+0000 7f135a439700 1 --2- 172.21.15.113:0/1942884937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13540fc7f0 0x7f13540fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.969+0000 7f135a439700 1 -- 172.21.15.113:0/1942884937 >> 172.21.15.113:0/1942884937 conn(0x7f1354070590 msgr2=0x7f13540f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:48.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.969+0000 7f135a439700 1 -- 172.21.15.113:0/1942884937 shutdown_connections 2024-04-12T05:22:48.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.969+0000 7f135a439700 1 -- 172.21.15.113:0/1942884937 wait complete. 2024-04-12T05:22:48.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.970+0000 7f135a439700 1 Processor -- start 2024-04-12T05:22:48.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.970+0000 7f135a439700 1 -- start start 2024-04-12T05:22:48.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.970+0000 7f135a439700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13540fc7f0 0x7f135410dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:48.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.970+0000 7f135a439700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13540fd150 0x7f135410e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:48.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.970+0000 7f135a439700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135410eab0 con 0x7f13540fc7f0 2024-04-12T05:22:48.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.970+0000 7f135a439700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135410ebf0 con 0x7f13540fd150 2024-04-12T05:22:48.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.970+0000 7f1353fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13540fc7f0 0x7f135410dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:48.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f1353fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13540fc7f0 0x7f135410dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59838/0 (socket says 172.21.15.113:59838) 2024-04-12T05:22:48.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f1353fff700 1 -- 172.21.15.113:0/1165967384 learned_addr learned my addr 172.21.15.113:0/1165967384 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:48.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f13537fe700 1 --2- 172.21.15.113:0/1165967384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13540fd150 0x7f135410e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:48.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f1353fff700 1 -- 172.21.15.113:0/1165967384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13540fd150 msgr2=0x7f135410e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f1353fff700 1 --2- 172.21.15.113:0/1165967384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13540fd150 0x7f135410e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f1353fff700 1 -- 172.21.15.113:0/1165967384 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1348005040 con 0x7f13540fc7f0 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f1353fff700 1 --2- 172.21.15.113:0/1165967384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13540fc7f0 0x7f135410dfc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f13440041d0 tx=0x7f1344009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f13517fa700 1 -- 172.21.15.113:0/1165967384 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1344002d90 con 0x7f13540fc7f0 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f135a439700 1 -- 172.21.15.113:0/1165967384 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1354108280 con 0x7f13540fc7f0 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f13517fa700 1 -- 172.21.15.113:0/1165967384 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f134400f040 con 0x7f13540fc7f0 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f13517fa700 1 -- 172.21.15.113:0/1165967384 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f134400d800 con 0x7f13540fc7f0 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.971+0000 7f135a439700 1 -- 172.21.15.113:0/1165967384 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1354108840 con 0x7f13540fc7f0 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.972+0000 7f135a439700 1 -- 172.21.15.113:0/1165967384 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135405f590 con 0x7f13540fc7f0 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.972+0000 7f13517fa700 1 -- 172.21.15.113:0/1165967384 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f134400dac0 con 0x7f13540fc7f0 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.973+0000 7f13517fa700 1 --2- 172.21.15.113:0/1165967384 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f133c06caa0 0x7f133c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.973+0000 7f13517fa700 1 -- 172.21.15.113:0/1165967384 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f134408bea0 con 0x7f13540fc7f0 2024-04-12T05:22:48.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.973+0000 7f13537fe700 1 --2- 172.21.15.113:0/1165967384 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f133c06caa0 0x7f133c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:48.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.973+0000 7f13537fe700 1 --2- 172.21.15.113:0/1165967384 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f133c06caa0 0x7f133c06ef60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f1348004f10 tx=0x7f134800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:48.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:48.977+0000 7f13517fa700 1 -- 172.21.15.113:0/1165967384 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1344059950 con 0x7f13540fc7f0 2024-04-12T05:22:49.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.174+0000 7f135a439700 1 -- 172.21.15.113:0/1165967384 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1354108ee0 con 0x7f133c06caa0 2024-04-12T05:22:49.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.184+0000 7f13517fa700 1 -- 172.21.15.113:0/1165967384 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1354108ee0 con 0x7f133c06caa0 2024-04-12T05:22:49.186 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:22:49.186 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (10m) 3m ago 21m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:22:49.186 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (8m) 3m ago 21m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (7m) 3m ago 21m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (10m) 3m ago 21m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (3m) 3m ago 16m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (3m) 3m ago 16m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (3m) 3m ago 16m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (16m) 3m ago 16m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (12m) 3m ago 23m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (11m) 3m ago 21m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (9m) 3m ago 23m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (8m) 3m ago 21m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (11m) 3m ago 21m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (11m) 3m ago 20m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:22:49.187 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (7m) 3m ago 20m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:22:49.188 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (7m) 3m ago 19m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:22:49.188 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (6m) 3m ago 19m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:22:49.188 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (5m) 3m ago 18m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:22:49.188 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (5m) 3m ago 18m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:22:49.188 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (4m) 3m ago 18m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:22:49.188 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (10m) 3m ago 20m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:22:49.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.186+0000 7f135a439700 1 -- 172.21.15.113:0/1165967384 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f133c06caa0 msgr2=0x7f133c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:49.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.186+0000 7f135a439700 1 --2- 172.21.15.113:0/1165967384 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f133c06caa0 0x7f133c06ef60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f1348004f10 tx=0x7f134800b040).stop 2024-04-12T05:22:49.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.186+0000 7f135a439700 1 -- 172.21.15.113:0/1165967384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13540fc7f0 msgr2=0x7f135410dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:49.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.186+0000 7f135a439700 1 --2- 172.21.15.113:0/1165967384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13540fc7f0 0x7f135410dfc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f13440041d0 tx=0x7f1344009f60).stop 2024-04-12T05:22:49.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.186+0000 7f135a439700 1 -- 172.21.15.113:0/1165967384 shutdown_connections 2024-04-12T05:22:49.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.186+0000 7f135a439700 1 --2- 172.21.15.113:0/1165967384 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f133c06caa0 0x7f133c06ef60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.186+0000 7f135a439700 1 --2- 172.21.15.113:0/1165967384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13540fc7f0 0x7f135410dfc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.186+0000 7f135a439700 1 --2- 172.21.15.113:0/1165967384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13540fd150 0x7f135410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.186+0000 7f135a439700 1 -- 172.21.15.113:0/1165967384 >> 172.21.15.113:0/1165967384 conn(0x7f1354070590 msgr2=0x7f1354060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:49.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.186+0000 7f135a439700 1 -- 172.21.15.113:0/1165967384 shutdown_connections 2024-04-12T05:22:49.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.186+0000 7f135a439700 1 -- 172.21.15.113:0/1165967384 wait complete. 2024-04-12T05:22:49.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.283+0000 7ff929174700 1 -- 172.21.15.113:0/3945543262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9240fc210 msgr2=0x7ff9240f70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:49.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.283+0000 7ff929174700 1 --2- 172.21.15.113:0/3945543262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9240fc210 0x7ff9240f70d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7ff9180047f0 tx=0x7ff918009cb0).stop 2024-04-12T05:22:49.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.283+0000 7ff929174700 1 -- 172.21.15.113:0/3945543262 shutdown_connections 2024-04-12T05:22:49.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.283+0000 7ff929174700 1 --2- 172.21.15.113:0/3945543262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9240fc210 0x7ff9240f70d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.283+0000 7ff929174700 1 --2- 172.21.15.113:0/3945543262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9240fb8b0 0x7ff9240fbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:22:49.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.283+0000 7ff929174700 1 -- 172.21.15.113:0/3945543262 >> 172.21.15.113:0/3945543262 conn(0x7ff92406cea0 msgr2=0x7ff92406d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:49.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.283+0000 7ff929174700 1 -- 172.21.15.113:0/3945543262 shutdown_connections 2024-04-12T05:22:49.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.283+0000 7ff929174700 1 -- 172.21.15.113:0/3945543262 wait complete. 2024-04-12T05:22:49.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.284+0000 7ff929174700 1 Processor -- start 2024-04-12T05:22:49.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.284+0000 7ff929174700 1 -- start start 2024-04-12T05:22:49.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.284+0000 7ff929174700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9240fb8b0 0x7ff924109250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:49.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.284+0000 7ff929174700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9240fc210 0x7ff924109740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:49.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.284+0000 7ff929174700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff924109cc0 con 0x7ff9240fc210 2024-04-12T05:22:49.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.284+0000 7ff929174700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff924109e00 con 0x7ff9240fb8b0 2024-04-12T05:22:49.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff92259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9240fc210 0x7ff924109740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:49.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff92259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9240fc210 0x7ff924109740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59858/0 (socket says 172.21.15.113:59858) 2024-04-12T05:22:49.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff92259c700 1 -- 172.21.15.113:0/824360064 learned_addr learned my addr 172.21.15.113:0/824360064 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:49.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff922d9d700 1 --2- 172.21.15.113:0/824360064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9240fb8b0 0x7ff924109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:49.289 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff92259c700 1 -- 172.21.15.113:0/824360064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9240fb8b0 msgr2=0x7ff924109250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:49.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff92259c700 1 --2- 172.21.15.113:0/824360064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9240fb8b0 0x7ff924109250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff92259c700 1 -- 172.21.15.113:0/824360064 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff918005040 con 0x7ff9240fc210 2024-04-12T05:22:49.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff92259c700 1 --2- 172.21.15.113:0/824360064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9240fc210 0x7ff924109740 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7ff9180099f0 tx=0x7ff918009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:49.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff90bfff700 1 -- 172.21.15.113:0/824360064 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91800e030 con 0x7ff9240fc210 2024-04-12T05:22:49.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff929174700 1 -- 172.21.15.113:0/824360064 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92410a050 con 0x7ff9240fc210 2024-04-12T05:22:49.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff90bfff700 1 -- 172.21.15.113:0/824360064 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff918007e90 con 0x7ff9240fc210 2024-04-12T05:22:49.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff90bfff700 1 -- 172.21.15.113:0/824360064 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff918018070 con 0x7ff9240fc210 2024-04-12T05:22:49.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.285+0000 7ff929174700 1 -- 172.21.15.113:0/824360064 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff924112590 con 0x7ff9240fc210 2024-04-12T05:22:49.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.286+0000 7ff929174700 1 -- 172.21.15.113:0/824360064 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff924047540 con 0x7ff9240fc210 2024-04-12T05:22:49.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.287+0000 7ff90bfff700 1 -- 172.21.15.113:0/824360064 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff91801d420 con 0x7ff9240fc210 2024-04-12T05:22:49.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.287+0000 7ff90bfff700 1 --2- 172.21.15.113:0/824360064 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff90c06caa0 0x7ff90c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:49.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.287+0000 7ff90bfff700 1 -- 172.21.15.113:0/824360064 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff91808d170 con 0x7ff9240fc210 2024-04-12T05:22:49.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.287+0000 7ff922d9d700 1 --2- 172.21.15.113:0/824360064 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff90c06caa0 0x7ff90c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:49.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.287+0000 7ff922d9d700 1 --2- 172.21.15.113:0/824360064 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff90c06caa0 0x7ff90c06ef60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff914006fd0 tx=0x7ff914009060).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:49.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.291+0000 7ff90bfff700 1 -- 172.21.15.113:0/824360064 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff918057190 con 0x7ff9240fc210 2024-04-12T05:22:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:49 smithi113 ceph-mon[103371]: pgmap v319: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:49.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:49 smithi192 ceph-mon[85304]: pgmap v319: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:49.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.566+0000 7ff929174700 1 -- 172.21.15.113:0/824360064 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff9240fec70 con 0x7ff9240fc210 2024-04-12T05:22:49.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.566+0000 7ff90bfff700 1 -- 172.21.15.113:0/824360064 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7ff91805a7b0 con 0x7ff9240fc210 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:22:49.569 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:22:49.570 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:22:49.570 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:22:49.570 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:22:49.570 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:22:49.570 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:22:49.570 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:22:49.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.568+0000 7ff929174700 1 -- 172.21.15.113:0/824360064 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff90c06caa0 msgr2=0x7ff90c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:49.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.568+0000 7ff929174700 1 --2- 172.21.15.113:0/824360064 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff90c06caa0 0x7ff90c06ef60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff914006fd0 tx=0x7ff914009060).stop 2024-04-12T05:22:49.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.568+0000 7ff929174700 1 -- 172.21.15.113:0/824360064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9240fc210 msgr2=0x7ff924109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:49.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.568+0000 7ff929174700 1 --2- 172.21.15.113:0/824360064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9240fc210 0x7ff924109740 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7ff9180099f0 tx=0x7ff918009cb0).stop 2024-04-12T05:22:49.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.568+0000 7ff929174700 1 -- 172.21.15.113:0/824360064 shutdown_connections 2024-04-12T05:22:49.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.568+0000 7ff929174700 1 --2- 172.21.15.113:0/824360064 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff90c06caa0 0x7ff90c06ef60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.568+0000 7ff929174700 1 --2- 172.21.15.113:0/824360064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9240fb8b0 0x7ff924109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.568+0000 7ff929174700 1 --2- 172.21.15.113:0/824360064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9240fc210 0x7ff924109740 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.568+0000 7ff929174700 1 -- 172.21.15.113:0/824360064 >> 172.21.15.113:0/824360064 conn(0x7ff92406cea0 msgr2=0x7ff9240fad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:49.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.568+0000 7ff929174700 1 -- 172.21.15.113:0/824360064 shutdown_connections 2024-04-12T05:22:49.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.569+0000 7ff929174700 1 -- 172.21.15.113:0/824360064 wait complete. 2024-04-12T05:22:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.666+0000 7f7c365ce700 1 -- 172.21.15.113:0/4168832810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c300fe250 msgr2=0x7f7c300f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.666+0000 7f7c365ce700 1 --2- 172.21.15.113:0/4168832810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c300fe250 0x7f7c300f7a30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f7c200047f0 tx=0x7f7c20009cb0).stop 2024-04-12T05:22:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.666+0000 7f7c365ce700 1 -- 172.21.15.113:0/4168832810 shutdown_connections 2024-04-12T05:22:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.666+0000 7f7c365ce700 1 --2- 172.21.15.113:0/4168832810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c300fe250 0x7f7c300f7a30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.666+0000 7f7c365ce700 1 --2- 172.21.15.113:0/4168832810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c300fd8f0 0x7f7c300fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.666+0000 7f7c365ce700 1 -- 172.21.15.113:0/4168832810 >> 172.21.15.113:0/4168832810 conn(0x7f7c300f34b0 msgr2=0x7f7c300f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.666+0000 7f7c365ce700 1 -- 172.21.15.113:0/4168832810 shutdown_connections 2024-04-12T05:22:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.666+0000 7f7c365ce700 1 -- 172.21.15.113:0/4168832810 wait complete. 2024-04-12T05:22:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c365ce700 1 Processor -- start 2024-04-12T05:22:49.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c365ce700 1 -- start start 2024-04-12T05:22:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c365ce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c300fd8f0 0x7f7c3010e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c365ce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c300fe250 0x7f7c3010e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c365ce700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3010eb70 con 0x7f7c300fd8f0 2024-04-12T05:22:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c365ce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3010ecb0 con 0x7f7c300fe250 2024-04-12T05:22:49.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c2ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c300fd8f0 0x7f7c3010e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c2ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c300fd8f0 0x7f7c3010e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59862/0 (socket says 172.21.15.113:59862) 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c2ffff700 1 -- 172.21.15.113:0/380745101 learned_addr learned my addr 172.21.15.113:0/380745101 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c2f7fe700 1 --2- 172.21.15.113:0/380745101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c300fe250 0x7f7c3010e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c2ffff700 1 -- 172.21.15.113:0/380745101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c300fe250 msgr2=0x7f7c3010e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c2ffff700 1 --2- 172.21.15.113:0/380745101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c300fe250 0x7f7c3010e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c2ffff700 1 -- 172.21.15.113:0/380745101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c20005040 con 0x7f7c300fd8f0 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.667+0000 7f7c2ffff700 1 --2- 172.21.15.113:0/380745101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c300fd8f0 0x7f7c3010e080 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f7c180041d0 tx=0x7f7c18009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.668+0000 7f7c2d7fa700 1 -- 172.21.15.113:0/380745101 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c18007b40 con 0x7f7c300fd8f0 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.668+0000 7f7c365ce700 1 -- 172.21.15.113:0/380745101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c30108340 con 0x7f7c300fd8f0 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.668+0000 7f7c2d7fa700 1 -- 172.21.15.113:0/380745101 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c1800f040 con 0x7f7c300fd8f0 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.668+0000 7f7c2d7fa700 1 -- 172.21.15.113:0/380745101 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c180147b0 con 0x7f7c300fd8f0 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.668+0000 7f7c365ce700 1 -- 172.21.15.113:0/380745101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c30108880 con 0x7f7c300fd8f0 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.669+0000 7f7c365ce700 1 -- 172.21.15.113:0/380745101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3005f590 con 0x7f7c300fd8f0 2024-04-12T05:22:49.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.669+0000 7f7c2d7fa700 1 -- 172.21.15.113:0/380745101 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7c18009110 con 0x7f7c300fd8f0 2024-04-12T05:22:49.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.670+0000 7f7c2d7fa700 1 --2- 172.21.15.113:0/380745101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1c06caa0 0x7f7c1c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:49.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.670+0000 7f7c2d7fa700 1 -- 172.21.15.113:0/380745101 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7c1808bf10 con 0x7f7c300fd8f0 2024-04-12T05:22:49.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.670+0000 7f7c2f7fe700 1 --2- 172.21.15.113:0/380745101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1c06caa0 0x7f7c1c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:49.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.670+0000 7f7c2f7fe700 1 --2- 172.21.15.113:0/380745101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1c06caa0 0x7f7c1c06ef60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f7c301099c0 tx=0x7f7c2000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:49.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.674+0000 7f7c2d7fa700 1 -- 172.21.15.113:0/380745101 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7c18059900 con 0x7f7c300fd8f0 2024-04-12T05:22:49.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.902+0000 7f7c365ce700 1 -- 172.21.15.113:0/380745101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7c30109800 con 0x7f7c300fd8f0 2024-04-12T05:22:49.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.902+0000 7f7c2d7fa700 1 -- 172.21.15.113:0/380745101 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f7c18059490 con 0x7f7c300fd8f0 2024-04-12T05:22:49.905 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:22:49.905 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:22:49.905 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:22:49.905 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:22:49.905 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:22:49.905 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:22:49.905 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:22:49.905 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:22:49.905 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:22:49.905 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:22:49.905 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:22:49.906 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:22:49.907 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:22:49.908 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:22:49.908 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:22:49.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.904+0000 7f7c365ce700 1 -- 172.21.15.113:0/380745101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1c06caa0 msgr2=0x7f7c1c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:49.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.904+0000 7f7c365ce700 1 --2- 172.21.15.113:0/380745101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1c06caa0 0x7f7c1c06ef60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f7c301099c0 tx=0x7f7c2000b040).stop 2024-04-12T05:22:49.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.904+0000 7f7c365ce700 1 -- 172.21.15.113:0/380745101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c300fd8f0 msgr2=0x7f7c3010e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:49.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.904+0000 7f7c365ce700 1 --2- 172.21.15.113:0/380745101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c300fd8f0 0x7f7c3010e080 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f7c180041d0 tx=0x7f7c18009f60).stop 2024-04-12T05:22:49.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.905+0000 7f7c365ce700 1 -- 172.21.15.113:0/380745101 shutdown_connections 2024-04-12T05:22:49.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.905+0000 7f7c365ce700 1 --2- 172.21.15.113:0/380745101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1c06caa0 0x7f7c1c06ef60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.905+0000 7f7c365ce700 1 --2- 172.21.15.113:0/380745101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c300fd8f0 0x7f7c3010e080 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.905+0000 7f7c365ce700 1 --2- 172.21.15.113:0/380745101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c300fe250 0x7f7c3010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:49.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.905+0000 7f7c365ce700 1 -- 172.21.15.113:0/380745101 >> 172.21.15.113:0/380745101 conn(0x7f7c300f34b0 msgr2=0x7f7c300fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:49.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.905+0000 7f7c365ce700 1 -- 172.21.15.113:0/380745101 shutdown_connections 2024-04-12T05:22:49.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:49.905+0000 7f7c365ce700 1 -- 172.21.15.113:0/380745101 wait complete. 2024-04-12T05:22:49.909 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:22:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.008+0000 7fdd161bc700 1 -- 172.21.15.113:0/2224288774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd100f8940 msgr2=0x7fdd100f8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.008+0000 7fdd161bc700 1 --2- 172.21.15.113:0/2224288774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd100f8940 0x7fdd100f8d20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fdd000047f0 tx=0x7fdd00009cb0).stop 2024-04-12T05:22:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.008+0000 7fdd161bc700 1 -- 172.21.15.113:0/2224288774 shutdown_connections 2024-04-12T05:22:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.008+0000 7fdd161bc700 1 --2- 172.21.15.113:0/2224288774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd100f9210 0x7fdd101019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.008+0000 7fdd161bc700 1 --2- 172.21.15.113:0/2224288774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd100f8940 0x7fdd100f8d20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.008+0000 7fdd161bc700 1 -- 172.21.15.113:0/2224288774 >> 172.21.15.113:0/2224288774 conn(0x7fdd10070590 msgr2=0x7fdd100f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.008+0000 7fdd161bc700 1 -- 172.21.15.113:0/2224288774 shutdown_connections 2024-04-12T05:22:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.008+0000 7fdd161bc700 1 -- 172.21.15.113:0/2224288774 wait complete. 2024-04-12T05:22:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd161bc700 1 Processor -- start 2024-04-12T05:22:50.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd161bc700 1 -- start start 2024-04-12T05:22:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd161bc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd100f8940 0x7fdd1010e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd161bc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd100f9210 0x7fdd1010e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd161bc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1010eb40 con 0x7fdd100f8940 2024-04-12T05:22:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd161bc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1010ec80 con 0x7fdd100f9210 2024-04-12T05:22:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd0f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd100f8940 0x7fdd1010e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd0f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd100f8940 0x7fdd1010e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59878/0 (socket says 172.21.15.113:59878) 2024-04-12T05:22:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd0f7fe700 1 -- 172.21.15.113:0/2623913211 learned_addr learned my addr 172.21.15.113:0/2623913211 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd0effd700 1 --2- 172.21.15.113:0/2623913211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd100f9210 0x7fdd1010e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:50.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd0f7fe700 1 -- 172.21.15.113:0/2623913211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd100f9210 msgr2=0x7fdd1010e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd0f7fe700 1 --2- 172.21.15.113:0/2623913211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd100f9210 0x7fdd1010e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd0f7fe700 1 -- 172.21.15.113:0/2623913211 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd00005040 con 0x7fdd100f8940 2024-04-12T05:22:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.009+0000 7fdd0f7fe700 1 --2- 172.21.15.113:0/2623913211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd100f8940 0x7fdd1010e050 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fdd000043a0 tx=0x7fdd00009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.010+0000 7fdd0cff9700 1 -- 172.21.15.113:0/2623913211 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd0000e030 con 0x7fdd100f8940 2024-04-12T05:22:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.010+0000 7fdd0cff9700 1 -- 172.21.15.113:0/2623913211 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd00007e90 con 0x7fdd100f8940 2024-04-12T05:22:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.010+0000 7fdd161bc700 1 -- 172.21.15.113:0/2623913211 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd101082b0 con 0x7fdd100f8940 2024-04-12T05:22:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.010+0000 7fdd0cff9700 1 -- 172.21.15.113:0/2623913211 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd00018070 con 0x7fdd100f8940 2024-04-12T05:22:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.010+0000 7fdd161bc700 1 -- 172.21.15.113:0/2623913211 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd10108790 con 0x7fdd100f8940 2024-04-12T05:22:50.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.011+0000 7fdd161bc700 1 -- 172.21.15.113:0/2623913211 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1005f590 con 0x7fdd100f8940 2024-04-12T05:22:50.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.011+0000 7fdd0cff9700 1 -- 172.21.15.113:0/2623913211 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdd000094c0 con 0x7fdd100f8940 2024-04-12T05:22:50.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.012+0000 7fdd0cff9700 1 --2- 172.21.15.113:0/2623913211 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcf806cab0 0x7fdcf806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:50.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.012+0000 7fdd0cff9700 1 -- 172.21.15.113:0/2623913211 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fdd0008cfe0 con 0x7fdd100f8940 2024-04-12T05:22:50.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.012+0000 7fdd0effd700 1 --2- 172.21.15.113:0/2623913211 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcf806cab0 0x7fdcf806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:50.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.012+0000 7fdd0effd700 1 --2- 172.21.15.113:0/2623913211 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcf806cab0 0x7fdcf806ef70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fdd10109990 tx=0x7fdd0400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:50.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.016+0000 7fdd0cff9700 1 -- 172.21.15.113:0/2623913211 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fdd0005aa10 con 0x7fdd100f8940 2024-04-12T05:22:50.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.219+0000 7fdd161bc700 1 -- 172.21.15.113:0/2623913211 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd10109770 con 0x7fdcf806cab0 2024-04-12T05:22:50.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.221+0000 7fdd0cff9700 1 -- 172.21.15.113:0/2623913211 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdd10109770 con 0x7fdcf806cab0 2024-04-12T05:22:50.223 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:22:50.223 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:22:50.223 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:22:50.223 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:22:50.223 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:22:50.223 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:22:50.223 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:22:50.223 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:22:50.223 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:22:50.223 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:22:50.224 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:22:50.224 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:22:50.224 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:22:50.224 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:22:50.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.223+0000 7fdd161bc700 1 -- 172.21.15.113:0/2623913211 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcf806cab0 msgr2=0x7fdcf806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:50.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.223+0000 7fdd161bc700 1 --2- 172.21.15.113:0/2623913211 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcf806cab0 0x7fdcf806ef70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fdd10109990 tx=0x7fdd0400b040).stop 2024-04-12T05:22:50.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.223+0000 7fdd161bc700 1 -- 172.21.15.113:0/2623913211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd100f8940 msgr2=0x7fdd1010e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:50.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.223+0000 7fdd161bc700 1 --2- 172.21.15.113:0/2623913211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd100f8940 0x7fdd1010e050 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fdd000043a0 tx=0x7fdd00009cb0).stop 2024-04-12T05:22:50.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.223+0000 7fdd161bc700 1 -- 172.21.15.113:0/2623913211 shutdown_connections 2024-04-12T05:22:50.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.223+0000 7fdd161bc700 1 --2- 172.21.15.113:0/2623913211 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcf806cab0 0x7fdcf806ef70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.223+0000 7fdd161bc700 1 --2- 172.21.15.113:0/2623913211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd100f8940 0x7fdd1010e050 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.223+0000 7fdd161bc700 1 --2- 172.21.15.113:0/2623913211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd100f9210 0x7fdd1010e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.223+0000 7fdd161bc700 1 -- 172.21.15.113:0/2623913211 >> 172.21.15.113:0/2623913211 conn(0x7fdd10070590 msgr2=0x7fdd100fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:50.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.223+0000 7fdd161bc700 1 -- 172.21.15.113:0/2623913211 shutdown_connections 2024-04-12T05:22:50.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.223+0000 7fdd161bc700 1 -- 172.21.15.113:0/2623913211 wait complete. 2024-04-12T05:22:50.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.327+0000 7f13c5e7b700 1 -- 172.21.15.113:0/1427026550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c00f9200 msgr2=0x7f13c0101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:50.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.327+0000 7f13c5e7b700 1 --2- 172.21.15.113:0/1427026550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c00f9200 0x7f13c0101910 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f13b00047f0 tx=0x7f13b0009cb0).stop 2024-04-12T05:22:50.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.327+0000 7f13c5e7b700 1 -- 172.21.15.113:0/1427026550 shutdown_connections 2024-04-12T05:22:50.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.327+0000 7f13c5e7b700 1 --2- 172.21.15.113:0/1427026550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c00f9200 0x7f13c0101910 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.327+0000 7f13c5e7b700 1 --2- 172.21.15.113:0/1427026550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13c00f88a0 0x7f13c00f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.327+0000 7f13c5e7b700 1 -- 172.21.15.113:0/1427026550 >> 172.21.15.113:0/1427026550 conn(0x7f13c0070590 msgr2=0x7f13c00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:50.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.327+0000 7f13c5e7b700 1 -- 172.21.15.113:0/1427026550 shutdown_connections 2024-04-12T05:22:50.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.327+0000 7f13c5e7b700 1 -- 172.21.15.113:0/1427026550 wait complete. 2024-04-12T05:22:50.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13c5e7b700 1 Processor -- start 2024-04-12T05:22:50.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13c5e7b700 1 -- start start 2024-04-12T05:22:50.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13c5e7b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c00f88a0 0x7f13c010e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:50.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13c5e7b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13c00f9200 0x7f13c010e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:50.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13c5e7b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c010eb70 con 0x7f13c00f88a0 2024-04-12T05:22:50.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13c5e7b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c010ecb0 con 0x7f13c00f9200 2024-04-12T05:22:50.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13bf7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c00f88a0 0x7f13c010e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:50.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13bf7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c00f88a0 0x7f13c010e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34782/0 (socket says 172.21.15.113:34782) 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13bf7fe700 1 -- 172.21.15.113:0/3874875351 learned_addr learned my addr 172.21.15.113:0/3874875351 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13beffd700 1 --2- 172.21.15.113:0/3874875351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13c00f9200 0x7f13c010e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13bf7fe700 1 -- 172.21.15.113:0/3874875351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13c00f9200 msgr2=0x7f13c010e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13bf7fe700 1 --2- 172.21.15.113:0/3874875351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13c00f9200 0x7f13c010e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.328+0000 7f13bf7fe700 1 -- 172.21.15.113:0/3874875351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b0005040 con 0x7f13c00f88a0 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.329+0000 7f13bf7fe700 1 --2- 172.21.15.113:0/3874875351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c00f88a0 0x7f13c010e080 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f13a800c790 tx=0x7f13a8004330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.329+0000 7f13bcff9700 1 -- 172.21.15.113:0/3874875351 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13a80106f0 con 0x7f13c00f88a0 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.329+0000 7f13bcff9700 1 -- 172.21.15.113:0/3874875351 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13a8010d00 con 0x7f13c00f88a0 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.329+0000 7f13bcff9700 1 -- 172.21.15.113:0/3874875351 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13a8012da0 con 0x7f13c00f88a0 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.329+0000 7f13c5e7b700 1 -- 172.21.15.113:0/3874875351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c0108340 con 0x7f13c00f88a0 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.329+0000 7f13c5e7b700 1 -- 172.21.15.113:0/3874875351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c0108880 con 0x7f13c00f88a0 2024-04-12T05:22:50.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.330+0000 7f13c5e7b700 1 -- 172.21.15.113:0/3874875351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c005f590 con 0x7f13c00f88a0 2024-04-12T05:22:50.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.330+0000 7f13bcff9700 1 -- 172.21.15.113:0/3874875351 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f13a8012400 con 0x7f13c00f88a0 2024-04-12T05:22:50.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.331+0000 7f13bcff9700 1 --2- 172.21.15.113:0/3874875351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f13ac06caa0 0x7f13ac06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:22:50.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.331+0000 7f13bcff9700 1 -- 172.21.15.113:0/3874875351 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f13a808b6f0 con 0x7f13c00f88a0 2024-04-12T05:22:50.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.331+0000 7f13beffd700 1 --2- 172.21.15.113:0/3874875351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f13ac06caa0 0x7f13ac06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:22:50.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.331+0000 7f13beffd700 1 --2- 172.21.15.113:0/3874875351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f13ac06caa0 0x7f13ac06ef60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f13b0000c00 tx=0x7f13b000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:22:50.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.335+0000 7f13bcff9700 1 -- 172.21.15.113:0/3874875351 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f13a80591a0 con 0x7f13c00f88a0 2024-04-12T05:22:50.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:50 smithi113 ceph-mon[103371]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:50.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:50 smithi113 ceph-mon[103371]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:50.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:50 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/824360064' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:22:50.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:50 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/380745101' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:22:50.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:50 smithi192 ceph-mon[85304]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:50.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:50 smithi192 ceph-mon[85304]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:50.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:50 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/824360064' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:22:50.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:50 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/380745101' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:22:50.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.599+0000 7f13c5e7b700 1 -- 172.21.15.113:0/3874875351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f13c0047540 con 0x7f13c00f88a0 2024-04-12T05:22:50.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.599+0000 7f13bcff9700 1 -- 172.21.15.113:0/3874875351 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f13a8058d30 con 0x7f13c00f88a0 2024-04-12T05:22:50.602 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:22:50.602 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:22:50.602 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:22:50.602 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:22:50.602 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:22:50.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.601+0000 7f13c5e7b700 1 -- 172.21.15.113:0/3874875351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f13ac06caa0 msgr2=0x7f13ac06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:50.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.601+0000 7f13c5e7b700 1 --2- 172.21.15.113:0/3874875351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f13ac06caa0 0x7f13ac06ef60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f13b0000c00 tx=0x7f13b000b040).stop 2024-04-12T05:22:50.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.601+0000 7f13c5e7b700 1 -- 172.21.15.113:0/3874875351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c00f88a0 msgr2=0x7f13c010e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:22:50.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.601+0000 7f13c5e7b700 1 --2- 172.21.15.113:0/3874875351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c00f88a0 0x7f13c010e080 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f13a800c790 tx=0x7f13a8004330).stop 2024-04-12T05:22:50.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.601+0000 7f13c5e7b700 1 -- 172.21.15.113:0/3874875351 shutdown_connections 2024-04-12T05:22:50.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.601+0000 7f13c5e7b700 1 --2- 172.21.15.113:0/3874875351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f13ac06caa0 0x7f13ac06ef60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.602+0000 7f13c5e7b700 1 --2- 172.21.15.113:0/3874875351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13c00f88a0 0x7f13c010e080 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.602+0000 7f13c5e7b700 1 --2- 172.21.15.113:0/3874875351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13c00f9200 0x7f13c010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:22:50.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.602+0000 7f13c5e7b700 1 -- 172.21.15.113:0/3874875351 >> 172.21.15.113:0/3874875351 conn(0x7f13c0070590 msgr2=0x7f13c00fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:22:50.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.602+0000 7f13c5e7b700 1 -- 172.21.15.113:0/3874875351 shutdown_connections 2024-04-12T05:22:50.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:22:50.602+0000 7f13c5e7b700 1 -- 172.21.15.113:0/3874875351 wait complete. 2024-04-12T05:22:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:51 smithi113 ceph-mon[103371]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:51 smithi113 ceph-mon[103371]: pgmap v320: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:51 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3874875351' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:22:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:51 smithi192 ceph-mon[85304]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:51 smithi192 ceph-mon[85304]: pgmap v320: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:51.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:51 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3874875351' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:22:52.269 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:52 smithi113 ceph-mon[103371]: from='client.34636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:52.270 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:22:52.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:52 smithi192 ceph-mon[85304]: from='client.34636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:22:52.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:22:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:53 smithi113 ceph-mon[103371]: pgmap v321: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:53 smithi192 ceph-mon[85304]: pgmap v321: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:55 smithi113 ceph-mon[103371]: pgmap v322: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:55.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:55 smithi192 ceph-mon[85304]: pgmap v322: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:57 smithi113 ceph-mon[103371]: pgmap v323: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:57.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:57 smithi192 ceph-mon[85304]: pgmap v323: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:22:59 smithi113 ceph-mon[103371]: pgmap v324: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:22:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:22:59 smithi192 ceph-mon[85304]: pgmap v324: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:01 smithi113 ceph-mon[103371]: pgmap v325: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:01 smithi192 ceph-mon[85304]: pgmap v325: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:03 smithi113 ceph-mon[103371]: pgmap v326: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:03 smithi192 ceph-mon[85304]: pgmap v326: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:05 smithi113 ceph-mon[103371]: pgmap v327: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:05 smithi192 ceph-mon[85304]: pgmap v327: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:07 smithi113 ceph-mon[103371]: pgmap v328: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:23:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:07 smithi192 ceph-mon[85304]: pgmap v328: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:23:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:09 smithi113 ceph-mon[103371]: pgmap v329: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:09.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:09 smithi192 ceph-mon[85304]: pgmap v329: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:11 smithi113 ceph-mon[103371]: pgmap v330: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:11 smithi192 ceph-mon[85304]: pgmap v330: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:13 smithi113 ceph-mon[103371]: pgmap v331: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:13 smithi192 ceph-mon[85304]: pgmap v331: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:15 smithi113 ceph-mon[103371]: pgmap v332: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:15 smithi192 ceph-mon[85304]: pgmap v332: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:17 smithi113 ceph-mon[103371]: pgmap v333: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:17.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:17 smithi192 ceph-mon[85304]: pgmap v333: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:19 smithi113 ceph-mon[103371]: pgmap v334: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:19 smithi192 ceph-mon[85304]: pgmap v334: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:20.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.711+0000 7fef3414c700 1 -- 172.21.15.113:0/3228692785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0fd150 msgr2=0x7fef2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:20.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.711+0000 7fef3414c700 1 --2- 172.21.15.113:0/3228692785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0fd150 0x7fef2c0f6930 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fef1c0047f0 tx=0x7fef1c009cb0).stop 2024-04-12T05:23:20.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.711+0000 7fef3414c700 1 -- 172.21.15.113:0/3228692785 shutdown_connections 2024-04-12T05:23:20.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.711+0000 7fef3414c700 1 --2- 172.21.15.113:0/3228692785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0fd150 0x7fef2c0f6930 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:20.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.711+0000 7fef3414c700 1 --2- 172.21.15.113:0/3228692785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef2c0fc7f0 0x7fef2c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:20.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.711+0000 7fef3414c700 1 -- 172.21.15.113:0/3228692785 >> 172.21.15.113:0/3228692785 conn(0x7fef2c070590 msgr2=0x7fef2c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:20.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.712+0000 7fef3414c700 1 -- 172.21.15.113:0/3228692785 shutdown_connections 2024-04-12T05:23:20.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.712+0000 7fef3414c700 1 -- 172.21.15.113:0/3228692785 wait complete. 2024-04-12T05:23:20.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.712+0000 7fef3414c700 1 Processor -- start 2024-04-12T05:23:20.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.712+0000 7fef3414c700 1 -- start start 2024-04-12T05:23:20.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.712+0000 7fef3414c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0fc7f0 0x7fef2c109060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:20.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.712+0000 7fef3414c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef2c0fd150 0x7fef2c109550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.712+0000 7fef3414c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c109a90 con 0x7fef2c0fc7f0 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.712+0000 7fef3414c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c109bd0 con 0x7fef2c0fd150 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef31ee8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0fc7f0 0x7fef2c109060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef31ee8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0fc7f0 0x7fef2c109060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40952/0 (socket says 172.21.15.113:40952) 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef31ee8700 1 -- 172.21.15.113:0/2172051466 learned_addr learned my addr 172.21.15.113:0/2172051466 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef316e7700 1 --2- 172.21.15.113:0/2172051466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef2c0fd150 0x7fef2c109550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef31ee8700 1 -- 172.21.15.113:0/2172051466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef2c0fd150 msgr2=0x7fef2c109550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef31ee8700 1 --2- 172.21.15.113:0/2172051466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef2c0fd150 0x7fef2c109550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef31ee8700 1 -- 172.21.15.113:0/2172051466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c005040 con 0x7fef2c0fc7f0 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef31ee8700 1 --2- 172.21.15.113:0/2172051466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0fc7f0 0x7fef2c109060 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fef2800c740 tx=0x7fef28007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef22ffd700 1 -- 172.21.15.113:0/2172051466 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef2800e680 con 0x7fef2c0fc7f0 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef22ffd700 1 -- 172.21.15.113:0/2172051466 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef2800ec90 con 0x7fef2c0fc7f0 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef3414c700 1 -- 172.21.15.113:0/2172051466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2c109e80 con 0x7fef2c0fc7f0 2024-04-12T05:23:20.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef22ffd700 1 -- 172.21.15.113:0/2172051466 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef28013d60 con 0x7fef2c0fc7f0 2024-04-12T05:23:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.713+0000 7fef3414c700 1 -- 172.21.15.113:0/2172051466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c112470 con 0x7fef2c0fc7f0 2024-04-12T05:23:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.714+0000 7fef3414c700 1 -- 172.21.15.113:0/2172051466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2c05f590 con 0x7fef2c0fc7f0 2024-04-12T05:23:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.715+0000 7fef22ffd700 1 -- 172.21.15.113:0/2172051466 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fef2800e7e0 con 0x7fef2c0fc7f0 2024-04-12T05:23:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.715+0000 7fef22ffd700 1 --2- 172.21.15.113:0/2172051466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef1806caa0 0x7fef1806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:20.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.715+0000 7fef22ffd700 1 -- 172.21.15.113:0/2172051466 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fef2808c150 con 0x7fef2c0fc7f0 2024-04-12T05:23:20.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.715+0000 7fef316e7700 1 --2- 172.21.15.113:0/2172051466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef1806caa0 0x7fef1806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:20.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.716+0000 7fef316e7700 1 --2- 172.21.15.113:0/2172051466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef1806caa0 0x7fef1806ef60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fef1c004f10 tx=0x7fef1c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:20.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.720+0000 7fef22ffd700 1 -- 172.21.15.113:0/2172051466 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fef28056170 con 0x7fef2c0fc7f0 2024-04-12T05:23:20.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.924+0000 7fef3414c700 1 -- 172.21.15.113:0/2172051466 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fef2c060d90 con 0x7fef1806caa0 2024-04-12T05:23:20.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.925+0000 7fef22ffd700 1 -- 172.21.15.113:0/2172051466 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fef2c060d90 con 0x7fef1806caa0 2024-04-12T05:23:20.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.927+0000 7fef3414c700 1 -- 172.21.15.113:0/2172051466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef1806caa0 msgr2=0x7fef1806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:20.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.927+0000 7fef3414c700 1 --2- 172.21.15.113:0/2172051466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef1806caa0 0x7fef1806ef60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fef1c004f10 tx=0x7fef1c00b040).stop 2024-04-12T05:23:20.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.927+0000 7fef3414c700 1 -- 172.21.15.113:0/2172051466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0fc7f0 msgr2=0x7fef2c109060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:20.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.928+0000 7fef3414c700 1 --2- 172.21.15.113:0/2172051466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0fc7f0 0x7fef2c109060 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fef2800c740 tx=0x7fef28007ef0).stop 2024-04-12T05:23:20.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.928+0000 7fef3414c700 1 -- 172.21.15.113:0/2172051466 shutdown_connections 2024-04-12T05:23:20.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.928+0000 7fef3414c700 1 --2- 172.21.15.113:0/2172051466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fef1806caa0 0x7fef1806ef60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:20.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.928+0000 7fef3414c700 1 --2- 172.21.15.113:0/2172051466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef2c0fc7f0 0x7fef2c109060 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:20.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.928+0000 7fef3414c700 1 --2- 172.21.15.113:0/2172051466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef2c0fd150 0x7fef2c109550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:20.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.928+0000 7fef3414c700 1 -- 172.21.15.113:0/2172051466 >> 172.21.15.113:0/2172051466 conn(0x7fef2c070590 msgr2=0x7fef2c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:20.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.928+0000 7fef3414c700 1 -- 172.21.15.113:0/2172051466 shutdown_connections 2024-04-12T05:23:20.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:20.928+0000 7fef3414c700 1 -- 172.21.15.113:0/2172051466 wait complete. 2024-04-12T05:23:20.941 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:23:21.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.027+0000 7f3582704700 1 -- 172.21.15.113:0/572241346 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f357c0f8900 msgr2=0x7f357c0f8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.027+0000 7f3582704700 1 --2- 172.21.15.113:0/572241346 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f357c0f8900 0x7f357c0f8ce0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f35700047f0 tx=0x7f3570009cb0).stop 2024-04-12T05:23:21.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.027+0000 7f3582704700 1 -- 172.21.15.113:0/572241346 shutdown_connections 2024-04-12T05:23:21.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.027+0000 7f3582704700 1 --2- 172.21.15.113:0/572241346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f357c0f9260 0x7f357c101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.027+0000 7f3582704700 1 --2- 172.21.15.113:0/572241346 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f357c0f8900 0x7f357c0f8ce0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.027+0000 7f3582704700 1 -- 172.21.15.113:0/572241346 >> 172.21.15.113:0/572241346 conn(0x7f357c070590 msgr2=0x7f357c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:21.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.027+0000 7f3582704700 1 -- 172.21.15.113:0/572241346 shutdown_connections 2024-04-12T05:23:21.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.027+0000 7f3582704700 1 -- 172.21.15.113:0/572241346 wait complete. 2024-04-12T05:23:21.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.027+0000 7f3582704700 1 Processor -- start 2024-04-12T05:23:21.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.027+0000 7f3582704700 1 -- start start 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f3582704700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f357c0f8900 0x7f357c110220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f3582704700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f357c0f9260 0x7f357c110710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f3582704700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357c110d10 con 0x7f357c0f8900 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f3582704700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357c110e50 con 0x7f357c0f9260 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f357bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f357c0f8900 0x7f357c110220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f357bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f357c0f8900 0x7f357c110220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40970/0 (socket says 172.21.15.113:40970) 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f357bfff700 1 -- 172.21.15.113:0/180426335 learned_addr learned my addr 172.21.15.113:0/180426335 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f357b7fe700 1 --2- 172.21.15.113:0/180426335 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f357c0f9260 0x7f357c110710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f357bfff700 1 -- 172.21.15.113:0/180426335 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f357c0f9260 msgr2=0x7f357c110710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f357bfff700 1 --2- 172.21.15.113:0/180426335 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f357c0f9260 0x7f357c110710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f357bfff700 1 -- 172.21.15.113:0/180426335 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3570005040 con 0x7f357c0f8900 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f357bfff700 1 --2- 172.21.15.113:0/180426335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f357c0f8900 0x7f357c110220 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f3570009380 tx=0x7f3570009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:21.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f35797fa700 1 -- 172.21.15.113:0/180426335 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f357000e070 con 0x7f357c0f8900 2024-04-12T05:23:21.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f35797fa700 1 -- 172.21.15.113:0/180426335 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3570007e90 con 0x7f357c0f8900 2024-04-12T05:23:21.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f3582704700 1 -- 172.21.15.113:0/180426335 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f357c10a4c0 con 0x7f357c0f8900 2024-04-12T05:23:21.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.028+0000 7f35797fa700 1 -- 172.21.15.113:0/180426335 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3570018070 con 0x7f357c0f8900 2024-04-12T05:23:21.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.029+0000 7f3582704700 1 -- 172.21.15.113:0/180426335 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f357c10a9a0 con 0x7f357c0f8900 2024-04-12T05:23:21.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.029+0000 7f3582704700 1 -- 172.21.15.113:0/180426335 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f357c05f590 con 0x7f357c0f8900 2024-04-12T05:23:21.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.030+0000 7f35797fa700 1 -- 172.21.15.113:0/180426335 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f357001d420 con 0x7f357c0f8900 2024-04-12T05:23:21.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.030+0000 7f35797fa700 1 --2- 172.21.15.113:0/180426335 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f356406caa0 0x7f356406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:21.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.030+0000 7f35797fa700 1 -- 172.21.15.113:0/180426335 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f357008ca30 con 0x7f357c0f8900 2024-04-12T05:23:21.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.030+0000 7f357b7fe700 1 --2- 172.21.15.113:0/180426335 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f356406caa0 0x7f356406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:21.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.031+0000 7f357b7fe700 1 --2- 172.21.15.113:0/180426335 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f356406caa0 0x7f356406ef60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f357c10bba0 tx=0x7f356c009de0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:21.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.035+0000 7f35797fa700 1 -- 172.21.15.113:0/180426335 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3570056a50 con 0x7f357c0f8900 2024-04-12T05:23:21.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.247+0000 7f3582704700 1 -- 172.21.15.113:0/180426335 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f357c10b9e0 con 0x7f356406caa0 2024-04-12T05:23:21.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.248+0000 7f35797fa700 1 -- 172.21.15.113:0/180426335 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f357c10b9e0 con 0x7f356406caa0 2024-04-12T05:23:21.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.250+0000 7f3582704700 1 -- 172.21.15.113:0/180426335 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f356406caa0 msgr2=0x7f356406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.250+0000 7f3582704700 1 --2- 172.21.15.113:0/180426335 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f356406caa0 0x7f356406ef60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f357c10bba0 tx=0x7f356c009de0).stop 2024-04-12T05:23:21.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.250+0000 7f3582704700 1 -- 172.21.15.113:0/180426335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f357c0f8900 msgr2=0x7f357c110220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.250+0000 7f3582704700 1 --2- 172.21.15.113:0/180426335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f357c0f8900 0x7f357c110220 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f3570009380 tx=0x7f3570009cb0).stop 2024-04-12T05:23:21.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.250+0000 7f3582704700 1 -- 172.21.15.113:0/180426335 shutdown_connections 2024-04-12T05:23:21.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.250+0000 7f3582704700 1 --2- 172.21.15.113:0/180426335 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f356406caa0 0x7f356406ef60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.250+0000 7f3582704700 1 --2- 172.21.15.113:0/180426335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f357c0f8900 0x7f357c110220 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.250+0000 7f3582704700 1 --2- 172.21.15.113:0/180426335 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f357c0f9260 0x7f357c110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.250+0000 7f3582704700 1 -- 172.21.15.113:0/180426335 >> 172.21.15.113:0/180426335 conn(0x7f357c070590 msgr2=0x7f357c0f6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:21.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.251+0000 7f3582704700 1 -- 172.21.15.113:0/180426335 shutdown_connections 2024-04-12T05:23:21.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.251+0000 7f3582704700 1 -- 172.21.15.113:0/180426335 wait complete. 2024-04-12T05:23:21.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.349+0000 7f4c52563700 1 -- 172.21.15.113:0/2077418847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c0fc7f0 msgr2=0x7f4c4c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.349+0000 7f4c52563700 1 --2- 172.21.15.113:0/2077418847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c0fc7f0 0x7f4c4c0fcbd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f4c400047f0 tx=0x7f4c40009cb0).stop 2024-04-12T05:23:21.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.349+0000 7f4c52563700 1 -- 172.21.15.113:0/2077418847 shutdown_connections 2024-04-12T05:23:21.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.349+0000 7f4c52563700 1 --2- 172.21.15.113:0/2077418847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c4c0fd150 0x7f4c4c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:21.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.349+0000 7f4c52563700 1 --2- 172.21.15.113:0/2077418847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c0fc7f0 0x7f4c4c0fcbd0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.350+0000 7f4c52563700 1 -- 172.21.15.113:0/2077418847 >> 172.21.15.113:0/2077418847 conn(0x7f4c4c0f23b0 msgr2=0x7f4c4c0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:21.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.350+0000 7f4c52563700 1 -- 172.21.15.113:0/2077418847 shutdown_connections 2024-04-12T05:23:21.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.350+0000 7f4c52563700 1 -- 172.21.15.113:0/2077418847 wait complete. 2024-04-12T05:23:21.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.350+0000 7f4c52563700 1 Processor -- start 2024-04-12T05:23:21.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.350+0000 7f4c52563700 1 -- start start 2024-04-12T05:23:21.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c52563700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c4c0fc7f0 0x7f4c4c109190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:21.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c52563700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c0fd150 0x7f4c4c109680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:21.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c52563700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c109c00 con 0x7f4c4c0fd150 2024-04-12T05:23:21.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c52563700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c109d40 con 0x7f4c4c0fc7f0 2024-04-12T05:23:21.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c4b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c0fd150 0x7f4c4c109680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c4b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c0fd150 0x7f4c4c109680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40992/0 (socket says 172.21.15.113:40992) 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c4b7fe700 1 -- 172.21.15.113:0/2050102720 learned_addr learned my addr 172.21.15.113:0/2050102720 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c4b7fe700 1 -- 172.21.15.113:0/2050102720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c4c0fc7f0 msgr2=0x7f4c4c109190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c4b7fe700 1 --2- 172.21.15.113:0/2050102720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c4c0fc7f0 0x7f4c4c109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c4b7fe700 1 -- 172.21.15.113:0/2050102720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c40005040 con 0x7f4c4c0fd150 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c4b7fe700 1 --2- 172.21.15.113:0/2050102720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c0fd150 0x7f4c4c109680 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f4c3c00b480 tx=0x7f4c3c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c497fa700 1 -- 172.21.15.113:0/2050102720 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c3c002500 con 0x7f4c4c0fd150 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c497fa700 1 -- 172.21.15.113:0/2050102720 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c3c002b10 con 0x7f4c4c0fd150 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c52563700 1 -- 172.21.15.113:0/2050102720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c109ff0 con 0x7f4c4c0fd150 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c497fa700 1 -- 172.21.15.113:0/2050102720 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c3c012c90 con 0x7f4c4c0fd150 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.351+0000 7f4c52563700 1 -- 172.21.15.113:0/2050102720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c112580 con 0x7f4c4c0fd150 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.352+0000 7f4c52563700 1 -- 172.21.15.113:0/2050102720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c060370 con 0x7f4c4c0fd150 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.353+0000 7f4c497fa700 1 -- 172.21.15.113:0/2050102720 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4c3c002660 con 0x7f4c4c0fd150 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.353+0000 7f4c497fa700 1 --2- 172.21.15.113:0/2050102720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c3406caa0 0x7f4c3406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:21.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.353+0000 7f4c497fa700 1 -- 172.21.15.113:0/2050102720 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f4c3c08bed0 con 0x7f4c4c0fd150 2024-04-12T05:23:21.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.353+0000 7f4c4bfff700 1 --2- 172.21.15.113:0/2050102720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c3406caa0 0x7f4c3406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:21.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.353+0000 7f4c4bfff700 1 --2- 172.21.15.113:0/2050102720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c3406caa0 0x7f4c3406ef60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4c400047f0 tx=0x7f4c40009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:21.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.357+0000 7f4c497fa700 1 -- 172.21.15.113:0/2050102720 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4c3c055ef0 con 0x7f4c4c0fd150 2024-04-12T05:23:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:21 smithi113 ceph-mon[103371]: pgmap v335: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:21 smithi192 ceph-mon[85304]: pgmap v335: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:21.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.555+0000 7f4c52563700 1 -- 172.21.15.113:0/2050102720 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c4c059120 con 0x7f4c3406caa0 2024-04-12T05:23:21.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.565+0000 7f4c497fa700 1 -- 172.21.15.113:0/2050102720 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4c4c059120 con 0x7f4c3406caa0 2024-04-12T05:23:21.568 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:23:21.568 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (11m) 4m ago 22m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:23:21.568 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (8m) 4m ago 22m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (8m) 3m ago 21m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (10m) 4m ago 22m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (4m) 4m ago 17m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (4m) 4m ago 16m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (3m) 3m ago 17m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (16m) 3m ago 16m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (12m) 4m ago 23m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (12m) 3m ago 21m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (9m) 4m ago 23m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (9m) 3m ago 21m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (11m) 4m ago 22m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:23:21.569 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (11m) 3m ago 21m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:23:21.570 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (8m) 4m ago 20m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:23:21.570 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (7m) 4m ago 20m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:23:21.570 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (7m) 4m ago 19m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:23:21.570 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (6m) 3m ago 19m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:23:21.570 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (5m) 3m ago 19m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:23:21.570 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (4m) 3m ago 18m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:23:21.570 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (11m) 3m ago 21m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:23:21.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.567+0000 7f4c52563700 1 -- 172.21.15.113:0/2050102720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c3406caa0 msgr2=0x7f4c3406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.567+0000 7f4c52563700 1 --2- 172.21.15.113:0/2050102720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c3406caa0 0x7f4c3406ef60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4c400047f0 tx=0x7f4c40009cb0).stop 2024-04-12T05:23:21.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.567+0000 7f4c52563700 1 -- 172.21.15.113:0/2050102720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c0fd150 msgr2=0x7f4c4c109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.567+0000 7f4c52563700 1 --2- 172.21.15.113:0/2050102720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c0fd150 0x7f4c4c109680 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f4c3c00b480 tx=0x7f4c3c0090f0).stop 2024-04-12T05:23:21.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.567+0000 7f4c52563700 1 -- 172.21.15.113:0/2050102720 shutdown_connections 2024-04-12T05:23:21.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.567+0000 7f4c52563700 1 --2- 172.21.15.113:0/2050102720 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c3406caa0 0x7f4c3406ef60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.567+0000 7f4c52563700 1 --2- 172.21.15.113:0/2050102720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c4c0fc7f0 0x7f4c4c109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:21.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.567+0000 7f4c52563700 1 --2- 172.21.15.113:0/2050102720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c4c0fd150 0x7f4c4c109680 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.567+0000 7f4c52563700 1 -- 172.21.15.113:0/2050102720 >> 172.21.15.113:0/2050102720 conn(0x7f4c4c0f23b0 msgr2=0x7f4c4c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:21.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.567+0000 7f4c52563700 1 -- 172.21.15.113:0/2050102720 shutdown_connections 2024-04-12T05:23:21.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.567+0000 7f4c52563700 1 -- 172.21.15.113:0/2050102720 wait complete. 2024-04-12T05:23:21.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.664+0000 7f21b5be1700 1 -- 172.21.15.113:0/3183870842 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b00f9200 msgr2=0x7f21b0101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.664+0000 7f21b5be1700 1 --2- 172.21.15.113:0/3183870842 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b00f9200 0x7f21b0101910 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f21a40047f0 tx=0x7f21a4009cb0).stop 2024-04-12T05:23:21.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.664+0000 7f21b5be1700 1 -- 172.21.15.113:0/3183870842 shutdown_connections 2024-04-12T05:23:21.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.664+0000 7f21b5be1700 1 --2- 172.21.15.113:0/3183870842 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b00f9200 0x7f21b0101910 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.664+0000 7f21b5be1700 1 --2- 172.21.15.113:0/3183870842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21b00f88a0 0x7f21b00f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.664+0000 7f21b5be1700 1 -- 172.21.15.113:0/3183870842 >> 172.21.15.113:0/3183870842 conn(0x7f21b0070590 msgr2=0x7f21b00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:21.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.665+0000 7f21b5be1700 1 -- 172.21.15.113:0/3183870842 shutdown_connections 2024-04-12T05:23:21.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.665+0000 7f21b5be1700 1 -- 172.21.15.113:0/3183870842 wait complete. 2024-04-12T05:23:21.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.665+0000 7f21b5be1700 1 Processor -- start 2024-04-12T05:23:21.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.665+0000 7f21b5be1700 1 -- start start 2024-04-12T05:23:21.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.665+0000 7f21b5be1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b00f88a0 0x7f21b010e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:21.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.665+0000 7f21b5be1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21b00f9200 0x7f21b010e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.665+0000 7f21b5be1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b010eb70 con 0x7f21b00f88a0 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.665+0000 7f21b5be1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b010ecb0 con 0x7f21b00f9200 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21af7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b00f88a0 0x7f21b010e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21af7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b00f88a0 0x7f21b010e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41014/0 (socket says 172.21.15.113:41014) 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21af7fe700 1 -- 172.21.15.113:0/803932927 learned_addr learned my addr 172.21.15.113:0/803932927 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21aeffd700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21b00f9200 0x7f21b010e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21af7fe700 1 -- 172.21.15.113:0/803932927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21b00f9200 msgr2=0x7f21b010e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21af7fe700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21b00f9200 0x7f21b010e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21af7fe700 1 -- 172.21.15.113:0/803932927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a4005040 con 0x7f21b00f88a0 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21aeffd700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21b00f9200 0x7f21b010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21af7fe700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b00f88a0 0x7f21b010e080 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f21a0002770 tx=0x7f21a0002b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21acff9700 1 -- 172.21.15.113:0/803932927 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21a0002d80 con 0x7f21b00f88a0 2024-04-12T05:23:21.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21acff9700 1 -- 172.21.15.113:0/803932927 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21a000f040 con 0x7f21b00f88a0 2024-04-12T05:23:21.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21b5be1700 1 -- 172.21.15.113:0/803932927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b0108340 con 0x7f21b00f88a0 2024-04-12T05:23:21.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21acff9700 1 -- 172.21.15.113:0/803932927 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21a000d800 con 0x7f21b00f88a0 2024-04-12T05:23:21.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.666+0000 7f21b5be1700 1 -- 172.21.15.113:0/803932927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b0108880 con 0x7f21b00f88a0 2024-04-12T05:23:21.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.667+0000 7f21b5be1700 1 -- 172.21.15.113:0/803932927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b005f590 con 0x7f21b00f88a0 2024-04-12T05:23:21.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.668+0000 7f21acff9700 1 -- 172.21.15.113:0/803932927 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f21a0009110 con 0x7f21b00f88a0 2024-04-12T05:23:21.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.668+0000 7f21acff9700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f219806caa0 0x7f219806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:21.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.668+0000 7f21acff9700 1 -- 172.21.15.113:0/803932927 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f21a008b9b0 con 0x7f21b00f88a0 2024-04-12T05:23:21.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.668+0000 7f21aeffd700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f219806caa0 0x7f219806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:21.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.668+0000 7f21aeffd700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f219806caa0 0x7f219806ef60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f21b01099c0 tx=0x7f21a400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:21.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.673+0000 7f21acff9700 1 -- 172.21.15.113:0/803932927 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f21a0059460 con 0x7f21b00f88a0 2024-04-12T05:23:21.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.941+0000 7f21b5be1700 1 -- 172.21.15.113:0/803932927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f21b0109800 con 0x7f21b00f88a0 2024-04-12T05:23:21.944 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.941+0000 7f21acff9700 1 -- 172.21.15.113:0/803932927 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f21a0058ff0 con 0x7f21b00f88a0 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:23:21.945 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:23:21.946 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:23:21.946 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:23:21.946 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:23:21.946 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:23:21.946 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:23:21.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.943+0000 7f21b5be1700 1 -- 172.21.15.113:0/803932927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f219806caa0 msgr2=0x7f219806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.943+0000 7f21b5be1700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f219806caa0 0x7f219806ef60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f21b01099c0 tx=0x7f21a400b040).stop 2024-04-12T05:23:21.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.943+0000 7f21b5be1700 1 -- 172.21.15.113:0/803932927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b00f88a0 msgr2=0x7f21b010e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:21.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.943+0000 7f21b5be1700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b00f88a0 0x7f21b010e080 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f21a0002770 tx=0x7f21a0002b30).stop 2024-04-12T05:23:21.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.943+0000 7f21b5be1700 1 -- 172.21.15.113:0/803932927 shutdown_connections 2024-04-12T05:23:21.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.943+0000 7f21b5be1700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f219806caa0 0x7f219806ef60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.943+0000 7f21b5be1700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21b00f88a0 0x7f21b010e080 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.943+0000 7f21b5be1700 1 --2- 172.21.15.113:0/803932927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21b00f9200 0x7f21b010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:21.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.943+0000 7f21b5be1700 1 -- 172.21.15.113:0/803932927 >> 172.21.15.113:0/803932927 conn(0x7f21b0070590 msgr2=0x7f21b00fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:21.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.944+0000 7f21b5be1700 1 -- 172.21.15.113:0/803932927 shutdown_connections 2024-04-12T05:23:21.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:21.944+0000 7f21b5be1700 1 -- 172.21.15.113:0/803932927 wait complete. 2024-04-12T05:23:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.043+0000 7f5110494700 1 -- 172.21.15.113:0/1657088730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51080fe360 msgr2=0x7f51080f5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.043+0000 7f5110494700 1 --2- 172.21.15.113:0/1657088730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51080fe360 0x7f51080f5fc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f51040047f0 tx=0x7f5104009cb0).stop 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.043+0000 7f5110494700 1 -- 172.21.15.113:0/1657088730 shutdown_connections 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.043+0000 7f5110494700 1 --2- 172.21.15.113:0/1657088730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51080fe360 0x7f51080f5fc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.043+0000 7f5110494700 1 --2- 172.21.15.113:0/1657088730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51080fda00 0x7f51080fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.043+0000 7f5110494700 1 -- 172.21.15.113:0/1657088730 >> 172.21.15.113:0/1657088730 conn(0x7f510806cea0 msgr2=0x7f510806d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.043+0000 7f5110494700 1 -- 172.21.15.113:0/1657088730 shutdown_connections 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.043+0000 7f5110494700 1 -- 172.21.15.113:0/1657088730 wait complete. 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.043+0000 7f5110494700 1 Processor -- start 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.043+0000 7f5110494700 1 -- start start 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f5110494700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51080fda00 0x7f51081101b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f5110494700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51080fe360 0x7f51081106a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f5110494700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5108110ca0 con 0x7f51080fe360 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f5110494700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5108110de0 con 0x7f51080fda00 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f510da2f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51080fe360 0x7f51081106a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:22.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f510da2f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51080fe360 0x7f51081106a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41028/0 (socket says 172.21.15.113:41028) 2024-04-12T05:23:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f510da2f700 1 -- 172.21.15.113:0/866141558 learned_addr learned my addr 172.21.15.113:0/866141558 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f510da2f700 1 -- 172.21.15.113:0/866141558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51080fda00 msgr2=0x7f51081101b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f510da2f700 1 --2- 172.21.15.113:0/866141558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51080fda00 0x7f51081101b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f510da2f700 1 -- 172.21.15.113:0/866141558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5104005040 con 0x7f51080fe360 2024-04-12T05:23:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f510da2f700 1 --2- 172.21.15.113:0/866141558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51080fe360 0x7f51081106a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f510400b120 tx=0x7f5104009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f50fb7fe700 1 -- 172.21.15.113:0/866141558 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f510400e030 con 0x7f51080fe360 2024-04-12T05:23:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f5110494700 1 -- 172.21.15.113:0/866141558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510810a410 con 0x7f51080fe360 2024-04-12T05:23:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f50fb7fe700 1 -- 172.21.15.113:0/866141558 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5104007e90 con 0x7f51080fe360 2024-04-12T05:23:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f50fb7fe700 1 -- 172.21.15.113:0/866141558 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5104018070 con 0x7f51080fe360 2024-04-12T05:23:22.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.044+0000 7f5110494700 1 -- 172.21.15.113:0/866141558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f510810a970 con 0x7f51080fe360 2024-04-12T05:23:22.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.045+0000 7f5110494700 1 -- 172.21.15.113:0/866141558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5108047540 con 0x7f51080fe360 2024-04-12T05:23:22.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.046+0000 7f50fb7fe700 1 -- 172.21.15.113:0/866141558 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f510401d420 con 0x7f51080fe360 2024-04-12T05:23:22.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.046+0000 7f50fb7fe700 1 --2- 172.21.15.113:0/866141558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50f406caa0 0x7f50f406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:22.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.047+0000 7f50fb7fe700 1 -- 172.21.15.113:0/866141558 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f510408d720 con 0x7f51080fe360 2024-04-12T05:23:22.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.047+0000 7f510e230700 1 --2- 172.21.15.113:0/866141558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50f406caa0 0x7f50f406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:22.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.047+0000 7f510e230700 1 --2- 172.21.15.113:0/866141558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50f406caa0 0x7f50f406ef60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f50fc004a90 tx=0x7f50fc007b40).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:22.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.051+0000 7f50fb7fe700 1 -- 172.21.15.113:0/866141558 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f510405b110 con 0x7f51080fe360 2024-04-12T05:23:22.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.282+0000 7f5110494700 1 -- 172.21.15.113:0/866141558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f51080fcbf0 con 0x7f51080fe360 2024-04-12T05:23:22.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.282+0000 7f50fb7fe700 1 -- 172.21.15.113:0/866141558 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f510405aca0 con 0x7f51080fe360 2024-04-12T05:23:22.286 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:23:22.286 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:23:22.286 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:23:22.286 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:23:22.286 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:23:22.286 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:23:22.287 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:23:22.288 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:23:22.288 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:23:22.288 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:23:22.288 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:23:22.288 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:23:22.288 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:23:22.288 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:23:22.288 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:23:22.288 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:23:22.288 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:23:22.288 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:23:22.289 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:23:22.289 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:23:22.289 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:23:22.289 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:23:22.289 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:23:22.289 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:23:22.289 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:23:22.289 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:23:22.289 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:23:22.290 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:23:22.290 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:23:22.290 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:23:22.290 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.284+0000 7f5110494700 1 -- 172.21.15.113:0/866141558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50f406caa0 msgr2=0x7f50f406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.284+0000 7f5110494700 1 --2- 172.21.15.113:0/866141558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50f406caa0 0x7f50f406ef60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f50fc004a90 tx=0x7f50fc007b40).stop 2024-04-12T05:23:22.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.284+0000 7f5110494700 1 -- 172.21.15.113:0/866141558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51080fe360 msgr2=0x7f51081106a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.284+0000 7f5110494700 1 --2- 172.21.15.113:0/866141558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51080fe360 0x7f51081106a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f510400b120 tx=0x7f5104009cb0).stop 2024-04-12T05:23:22.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.284+0000 7f5110494700 1 -- 172.21.15.113:0/866141558 shutdown_connections 2024-04-12T05:23:22.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.284+0000 7f5110494700 1 --2- 172.21.15.113:0/866141558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50f406caa0 0x7f50f406ef60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.284+0000 7f5110494700 1 --2- 172.21.15.113:0/866141558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51080fda00 0x7f51081101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:22.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.284+0000 7f5110494700 1 --2- 172.21.15.113:0/866141558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51080fe360 0x7f51081106a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.284+0000 7f5110494700 1 -- 172.21.15.113:0/866141558 >> 172.21.15.113:0/866141558 conn(0x7f510806cea0 msgr2=0x7f51080f78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:22.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.284+0000 7f5110494700 1 -- 172.21.15.113:0/866141558 shutdown_connections 2024-04-12T05:23:22.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.285+0000 7f5110494700 1 -- 172.21.15.113:0/866141558 wait complete. 2024-04-12T05:23:22.291 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:23:22.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.384+0000 7f22cd72f700 1 -- 172.21.15.113:0/976762742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22c80fd970 msgr2=0x7f22c80fdd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.384+0000 7f22cd72f700 1 --2- 172.21.15.113:0/976762742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22c80fd970 0x7f22c80fdd50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f22bc0047f0 tx=0x7f22bc009cb0).stop 2024-04-12T05:23:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.384+0000 7f22cd72f700 1 -- 172.21.15.113:0/976762742 shutdown_connections 2024-04-12T05:23:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.384+0000 7f22cd72f700 1 --2- 172.21.15.113:0/976762742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22c80fe2d0 0x7f22c80f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.384+0000 7f22cd72f700 1 --2- 172.21.15.113:0/976762742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22c80fd970 0x7f22c80fdd50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.384+0000 7f22cd72f700 1 -- 172.21.15.113:0/976762742 >> 172.21.15.113:0/976762742 conn(0x7f22c80f3550 msgr2=0x7f22c80f5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.384+0000 7f22cd72f700 1 -- 172.21.15.113:0/976762742 shutdown_connections 2024-04-12T05:23:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.384+0000 7f22cd72f700 1 -- 172.21.15.113:0/976762742 wait complete. 2024-04-12T05:23:22.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.384+0000 7f22cd72f700 1 Processor -- start 2024-04-12T05:23:22.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22cd72f700 1 -- start start 2024-04-12T05:23:22.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22cd72f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22c80fd970 0x7f22c810e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:22.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22cd72f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22c80fe2d0 0x7f22c810e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:22.391 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22cd72f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c810ebe0 con 0x7f22c80fd970 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22cd72f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c8108200 con 0x7f22c80fe2d0 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22c6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22c80fd970 0x7f22c810e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22c6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22c80fd970 0x7f22c810e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41044/0 (socket says 172.21.15.113:41044) 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22c6ffd700 1 -- 172.21.15.113:0/3958633100 learned_addr learned my addr 172.21.15.113:0/3958633100 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22c67fc700 1 --2- 172.21.15.113:0/3958633100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22c80fe2d0 0x7f22c810e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22c6ffd700 1 -- 172.21.15.113:0/3958633100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22c80fe2d0 msgr2=0x7f22c810e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22c6ffd700 1 --2- 172.21.15.113:0/3958633100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22c80fe2d0 0x7f22c810e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22c6ffd700 1 -- 172.21.15.113:0/3958633100 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc005040 con 0x7f22c80fd970 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.385+0000 7f22c6ffd700 1 --2- 172.21.15.113:0/3958633100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22c80fd970 0x7f22c810e060 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f22bc00b120 tx=0x7f22bc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.386+0000 7f22affff700 1 -- 172.21.15.113:0/3958633100 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22bc00e030 con 0x7f22c80fd970 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.386+0000 7f22cd72f700 1 -- 172.21.15.113:0/3958633100 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c8108450 con 0x7f22c80fd970 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.386+0000 7f22affff700 1 -- 172.21.15.113:0/3958633100 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22bc007e90 con 0x7f22c80fd970 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.386+0000 7f22affff700 1 -- 172.21.15.113:0/3958633100 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22bc018070 con 0x7f22c80fd970 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.386+0000 7f22cd72f700 1 -- 172.21.15.113:0/3958633100 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c8108850 con 0x7f22c80fd970 2024-04-12T05:23:22.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.386+0000 7f22cd72f700 1 -- 172.21.15.113:0/3958633100 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c8060370 con 0x7f22c80fd970 2024-04-12T05:23:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.387+0000 7f22affff700 1 -- 172.21.15.113:0/3958633100 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f22bc01d420 con 0x7f22c80fd970 2024-04-12T05:23:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.387+0000 7f22affff700 1 --2- 172.21.15.113:0/3958633100 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f22b006caa0 0x7f22b006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.387+0000 7f22affff700 1 -- 172.21.15.113:0/3958633100 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f22bc08d300 con 0x7f22c80fd970 2024-04-12T05:23:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.387+0000 7f22c67fc700 1 --2- 172.21.15.113:0/3958633100 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f22b006caa0 0x7f22b006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:22.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.388+0000 7f22c67fc700 1 --2- 172.21.15.113:0/3958633100 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f22b006caa0 0x7f22b006ef60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f22c81099a0 tx=0x7f22b8008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:22.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.392+0000 7f22affff700 1 -- 172.21.15.113:0/3958633100 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f22bc05adb0 con 0x7f22c80fd970 2024-04-12T05:23:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:22 smithi113 ceph-mon[103371]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:23:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:22 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/803932927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:23:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:22 smithi192 ceph-mon[85304]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:23:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:22 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/803932927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:23:22.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.596+0000 7f22cd72f700 1 -- 172.21.15.113:0/3958633100 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f22c8109820 con 0x7f22b006caa0 2024-04-12T05:23:22.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.597+0000 7f22affff700 1 -- 172.21.15.113:0/3958633100 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f22b800b4f0 con 0x7f22b006caa0 2024-04-12T05:23:22.600 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:23:22.600 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:23:22.601 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:23:22.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.599+0000 7f22cd72f700 1 -- 172.21.15.113:0/3958633100 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f22b006caa0 msgr2=0x7f22b006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.599+0000 7f22cd72f700 1 --2- 172.21.15.113:0/3958633100 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f22b006caa0 0x7f22b006ef60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f22c81099a0 tx=0x7f22b8008040).stop 2024-04-12T05:23:22.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.599+0000 7f22cd72f700 1 -- 172.21.15.113:0/3958633100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22c80fd970 msgr2=0x7f22c810e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.599+0000 7f22cd72f700 1 --2- 172.21.15.113:0/3958633100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22c80fd970 0x7f22c810e060 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f22bc00b120 tx=0x7f22bc009cb0).stop 2024-04-12T05:23:22.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.599+0000 7f22cd72f700 1 -- 172.21.15.113:0/3958633100 shutdown_connections 2024-04-12T05:23:22.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.599+0000 7f22cd72f700 1 --2- 172.21.15.113:0/3958633100 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f22b006caa0 0x7f22b006ef60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.599+0000 7f22cd72f700 1 --2- 172.21.15.113:0/3958633100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22c80fd970 0x7f22c810e060 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.599+0000 7f22cd72f700 1 --2- 172.21.15.113:0/3958633100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22c80fe2d0 0x7f22c810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.599+0000 7f22cd72f700 1 -- 172.21.15.113:0/3958633100 >> 172.21.15.113:0/3958633100 conn(0x7f22c80f3550 msgr2=0x7f22c80fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:22.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.599+0000 7f22cd72f700 1 -- 172.21.15.113:0/3958633100 shutdown_connections 2024-04-12T05:23:22.603 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.599+0000 7f22cd72f700 1 -- 172.21.15.113:0/3958633100 wait complete. 2024-04-12T05:23:22.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.697+0000 7f1284070700 1 -- 172.21.15.113:0/3926307913 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f127c0fd140 msgr2=0x7f127c0f7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.697+0000 7f1284070700 1 --2- 172.21.15.113:0/3926307913 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f127c0fd140 0x7f127c0f7090 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f12780047f0 tx=0x7f1278009cb0).stop 2024-04-12T05:23:22.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.697+0000 7f1284070700 1 -- 172.21.15.113:0/3926307913 shutdown_connections 2024-04-12T05:23:22.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.697+0000 7f1284070700 1 --2- 172.21.15.113:0/3926307913 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f127c0fd140 0x7f127c0f7090 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.697+0000 7f1284070700 1 --2- 172.21.15.113:0/3926307913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f127c0fc870 0x7f127c0fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.697+0000 7f1284070700 1 -- 172.21.15.113:0/3926307913 >> 172.21.15.113:0/3926307913 conn(0x7f127c06ce40 msgr2=0x7f127c06d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:22.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.697+0000 7f1284070700 1 -- 172.21.15.113:0/3926307913 shutdown_connections 2024-04-12T05:23:22.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.697+0000 7f1284070700 1 -- 172.21.15.113:0/3926307913 wait complete. 2024-04-12T05:23:22.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.698+0000 7f1284070700 1 Processor -- start 2024-04-12T05:23:22.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.698+0000 7f1284070700 1 -- start start 2024-04-12T05:23:22.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.698+0000 7f1284070700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f127c0fc870 0x7f127c10cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:22.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.698+0000 7f1284070700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f127c0fd140 0x7f127c10d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:22.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.698+0000 7f1284070700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c10da70 con 0x7f127c0fd140 2024-04-12T05:23:22.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.698+0000 7f1284070700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c10dbb0 con 0x7f127c0fc870 2024-04-12T05:23:22.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.698+0000 7f128160b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f127c0fd140 0x7f127c10d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:22.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.698+0000 7f128160b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f127c0fd140 0x7f127c10d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41068/0 (socket says 172.21.15.113:41068) 2024-04-12T05:23:22.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.698+0000 7f128160b700 1 -- 172.21.15.113:0/1228381317 learned_addr learned my addr 172.21.15.113:0/1228381317 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:22.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.699+0000 7f128160b700 1 -- 172.21.15.113:0/1228381317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f127c0fc870 msgr2=0x7f127c10cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.699+0000 7f128160b700 1 --2- 172.21.15.113:0/1228381317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f127c0fc870 0x7f127c10cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:22.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.699+0000 7f128160b700 1 -- 172.21.15.113:0/1228381317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1278005040 con 0x7f127c0fd140 2024-04-12T05:23:22.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.699+0000 7f128160b700 1 --2- 172.21.15.113:0/1228381317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f127c0fd140 0x7f127c10d470 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1278000c00 tx=0x7f1278009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:22.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.699+0000 7f126effd700 1 -- 172.21.15.113:0/1228381317 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f127800e030 con 0x7f127c0fd140 2024-04-12T05:23:22.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.699+0000 7f126effd700 1 -- 172.21.15.113:0/1228381317 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1278007e90 con 0x7f127c0fd140 2024-04-12T05:23:22.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.699+0000 7f1284070700 1 -- 172.21.15.113:0/1228381317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127c107230 con 0x7f127c0fd140 2024-04-12T05:23:22.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.699+0000 7f126effd700 1 -- 172.21.15.113:0/1228381317 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1278018070 con 0x7f127c0fd140 2024-04-12T05:23:22.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.699+0000 7f1284070700 1 -- 172.21.15.113:0/1228381317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f127c107790 con 0x7f127c0fd140 2024-04-12T05:23:22.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.700+0000 7f1284070700 1 -- 172.21.15.113:0/1228381317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127c060670 con 0x7f127c0fd140 2024-04-12T05:23:22.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.701+0000 7f126effd700 1 -- 172.21.15.113:0/1228381317 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f127801d420 con 0x7f127c0fd140 2024-04-12T05:23:22.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.701+0000 7f126effd700 1 --2- 172.21.15.113:0/1228381317 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f126806cab0 0x7f126806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:22.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.702+0000 7f1281e0c700 1 --2- 172.21.15.113:0/1228381317 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f126806cab0 0x7f126806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:22.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.702+0000 7f126effd700 1 -- 172.21.15.113:0/1228381317 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f127808d0f0 con 0x7f127c0fd140 2024-04-12T05:23:22.705 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.702+0000 7f1281e0c700 1 --2- 172.21.15.113:0/1228381317 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f126806cab0 0x7f126806ef70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f1270004b90 tx=0x7f1270009b60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:22.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.705+0000 7f126effd700 1 -- 172.21.15.113:0/1228381317 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f127805ab60 con 0x7f127c0fd140 2024-04-12T05:23:22.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.977+0000 7f1284070700 1 -- 172.21.15.113:0/1228381317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f127c047540 con 0x7f127c0fd140 2024-04-12T05:23:22.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.977+0000 7f126effd700 1 -- 172.21.15.113:0/1228381317 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f127805a6f0 con 0x7f127c0fd140 2024-04-12T05:23:22.981 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:23:22.981 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:23:22.981 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:23:22.981 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:23:22.981 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:23:22.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.980+0000 7f1284070700 1 -- 172.21.15.113:0/1228381317 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f126806cab0 msgr2=0x7f126806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.980+0000 7f1284070700 1 --2- 172.21.15.113:0/1228381317 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f126806cab0 0x7f126806ef70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f1270004b90 tx=0x7f1270009b60).stop 2024-04-12T05:23:22.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.980+0000 7f1284070700 1 -- 172.21.15.113:0/1228381317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f127c0fd140 msgr2=0x7f127c10d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:22.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.980+0000 7f1284070700 1 --2- 172.21.15.113:0/1228381317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f127c0fd140 0x7f127c10d470 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1278000c00 tx=0x7f1278009cb0).stop 2024-04-12T05:23:22.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.980+0000 7f1284070700 1 -- 172.21.15.113:0/1228381317 shutdown_connections 2024-04-12T05:23:22.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.980+0000 7f1284070700 1 --2- 172.21.15.113:0/1228381317 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f126806cab0 0x7f126806ef70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.980+0000 7f1284070700 1 --2- 172.21.15.113:0/1228381317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f127c0fc870 0x7f127c10cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:22.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.980+0000 7f1284070700 1 --2- 172.21.15.113:0/1228381317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f127c0fd140 0x7f127c10d470 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:22.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.980+0000 7f1284070700 1 -- 172.21.15.113:0/1228381317 >> 172.21.15.113:0/1228381317 conn(0x7f127c06ce40 msgr2=0x7f127c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:22.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.980+0000 7f1284070700 1 -- 172.21.15.113:0/1228381317 shutdown_connections 2024-04-12T05:23:22.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:22.980+0000 7f1284070700 1 -- 172.21.15.113:0/1228381317 wait complete. 2024-04-12T05:23:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:23 smithi113 ceph-mon[103371]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:23 smithi113 ceph-mon[103371]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:23 smithi113 ceph-mon[103371]: pgmap v336: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:23 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/866141558' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:23:23.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:23 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1228381317' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:23:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:23 smithi192 ceph-mon[85304]: from='client.34648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:23 smithi192 ceph-mon[85304]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:23 smithi192 ceph-mon[85304]: pgmap v336: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:23 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/866141558' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:23:23.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:23 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1228381317' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:23:24.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:24 smithi113 ceph-mon[103371]: from='client.34664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:24.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:24 smithi192 ceph-mon[85304]: from='client.34664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:25 smithi113 ceph-mon[103371]: pgmap v337: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:25 smithi192 ceph-mon[85304]: pgmap v337: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:27 smithi113 ceph-mon[103371]: pgmap v338: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:27.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:27 smithi192 ceph-mon[85304]: pgmap v338: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:29 smithi113 ceph-mon[103371]: pgmap v339: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:29 smithi192 ceph-mon[85304]: pgmap v339: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:31 smithi113 ceph-mon[103371]: pgmap v340: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:31 smithi192 ceph-mon[85304]: pgmap v340: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:33 smithi113 ceph-mon[103371]: pgmap v341: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:33.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:33 smithi192 ceph-mon[85304]: pgmap v341: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:35 smithi113 ceph-mon[103371]: pgmap v342: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:35.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:35 smithi192 ceph-mon[85304]: pgmap v342: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:37 smithi113 ceph-mon[103371]: pgmap v343: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:23:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:37 smithi192 ceph-mon[85304]: pgmap v343: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:23:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:39 smithi113 ceph-mon[103371]: pgmap v344: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:39 smithi192 ceph-mon[85304]: pgmap v344: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:41 smithi113 ceph-mon[103371]: pgmap v345: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:41 smithi192 ceph-mon[85304]: pgmap v345: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:43 smithi113 ceph-mon[103371]: pgmap v346: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:43.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:43 smithi192 ceph-mon[85304]: pgmap v346: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:45 smithi113 ceph-mon[103371]: pgmap v347: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:45 smithi192 ceph-mon[85304]: pgmap v347: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:47 smithi113 ceph-mon[103371]: pgmap v348: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:47.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:47 smithi192 ceph-mon[85304]: pgmap v348: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:49 smithi113 ceph-mon[103371]: pgmap v349: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:49.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:49 smithi192 ceph-mon[85304]: pgmap v349: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:51 smithi113 ceph-mon[103371]: pgmap v350: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:51 smithi192 ceph-mon[85304]: pgmap v350: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:23:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:52 smithi113 ceph-mon[103371]: pgmap v351: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:52.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:23:52.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:52 smithi192 ceph-mon[85304]: pgmap v351: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:53.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.093+0000 7fb00cb4b700 1 -- 172.21.15.113:0/586611702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0080fc7f0 msgr2=0x7fb0080fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.093+0000 7fb00cb4b700 1 --2- 172.21.15.113:0/586611702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0080fc7f0 0x7fb0080fcbd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7faffc0047f0 tx=0x7faffc009cb0).stop 2024-04-12T05:23:53.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.093+0000 7fb00cb4b700 1 -- 172.21.15.113:0/586611702 shutdown_connections 2024-04-12T05:23:53.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.093+0000 7fb00cb4b700 1 --2- 172.21.15.113:0/586611702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0080fd150 0x7fb0080f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.093+0000 7fb00cb4b700 1 --2- 172.21.15.113:0/586611702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0080fc7f0 0x7fb0080fcbd0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.093+0000 7fb00cb4b700 1 -- 172.21.15.113:0/586611702 >> 172.21.15.113:0/586611702 conn(0x7fb008070590 msgr2=0x7fb0080f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:53.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.093+0000 7fb00cb4b700 1 -- 172.21.15.113:0/586611702 shutdown_connections 2024-04-12T05:23:53.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.093+0000 7fb00cb4b700 1 -- 172.21.15.113:0/586611702 wait complete. 2024-04-12T05:23:53.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.094+0000 7fb00cb4b700 1 Processor -- start 2024-04-12T05:23:53.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.094+0000 7fb00cb4b700 1 -- start start 2024-04-12T05:23:53.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.094+0000 7fb00cb4b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0080fc7f0 0x7fb0080f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:53.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.094+0000 7fb00cb4b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0080fd150 0x7fb0080f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:53.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.094+0000 7fb00cb4b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0080f8f20 con 0x7fb0080fc7f0 2024-04-12T05:23:53.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.094+0000 7fb00cb4b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0080f9060 con 0x7fb0080fd150 2024-04-12T05:23:53.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.094+0000 7fb00659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0080fc7f0 0x7fb0080f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.094+0000 7fb00659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0080fc7f0 0x7fb0080f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54180/0 (socket says 172.21.15.113:54180) 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.094+0000 7fb00659c700 1 -- 172.21.15.113:0/1678104445 learned_addr learned my addr 172.21.15.113:0/1678104445 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.094+0000 7fb005d9b700 1 --2- 172.21.15.113:0/1678104445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0080fd150 0x7fb0080f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.095+0000 7fb00659c700 1 -- 172.21.15.113:0/1678104445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0080fd150 msgr2=0x7fb0080f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.095+0000 7fb00659c700 1 --2- 172.21.15.113:0/1678104445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0080fd150 0x7fb0080f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.095+0000 7fb00659c700 1 -- 172.21.15.113:0/1678104445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc005040 con 0x7fb0080fc7f0 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.095+0000 7fb00659c700 1 --2- 172.21.15.113:0/1678104445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0080fc7f0 0x7fb0080f8540 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7faffc00b120 tx=0x7faffc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.095+0000 7faff77fe700 1 -- 172.21.15.113:0/1678104445 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faffc00e030 con 0x7fb0080fc7f0 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.095+0000 7faff77fe700 1 -- 172.21.15.113:0/1678104445 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faffc007e90 con 0x7fb0080fc7f0 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.095+0000 7fb00cb4b700 1 -- 172.21.15.113:0/1678104445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0080f92b0 con 0x7fb0080fc7f0 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.095+0000 7faff77fe700 1 -- 172.21.15.113:0/1678104445 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faffc018070 con 0x7fb0080fc7f0 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.095+0000 7fb00cb4b700 1 -- 172.21.15.113:0/1678104445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb008114650 con 0x7fb0080fc7f0 2024-04-12T05:23:53.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.096+0000 7fb00cb4b700 1 -- 172.21.15.113:0/1678104445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00805f590 con 0x7fb0080fc7f0 2024-04-12T05:23:53.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.097+0000 7faff77fe700 1 -- 172.21.15.113:0/1678104445 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faffc01d420 con 0x7fb0080fc7f0 2024-04-12T05:23:53.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.097+0000 7faff77fe700 1 --2- 172.21.15.113:0/1678104445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faff006caa0 0x7faff006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:53.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.097+0000 7faff77fe700 1 -- 172.21.15.113:0/1678104445 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7faffc08d4c0 con 0x7fb0080fc7f0 2024-04-12T05:23:53.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.097+0000 7fb005d9b700 1 --2- 172.21.15.113:0/1678104445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faff006caa0 0x7faff006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:53.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.097+0000 7fb005d9b700 1 --2- 172.21.15.113:0/1678104445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faff006caa0 0x7faff006ef60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fb0080f9ce0 tx=0x7faff8009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:53.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.101+0000 7faff77fe700 1 -- 172.21.15.113:0/1678104445 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7faffc057560 con 0x7fb0080fc7f0 2024-04-12T05:23:53.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.310+0000 7fb00cb4b700 1 -- 172.21.15.113:0/1678104445 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0080fb5a0 con 0x7faff006caa0 2024-04-12T05:23:53.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.311+0000 7faff77fe700 1 -- 172.21.15.113:0/1678104445 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb0080fb5a0 con 0x7faff006caa0 2024-04-12T05:23:53.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.313+0000 7fb00cb4b700 1 -- 172.21.15.113:0/1678104445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faff006caa0 msgr2=0x7faff006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.313+0000 7fb00cb4b700 1 --2- 172.21.15.113:0/1678104445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faff006caa0 0x7faff006ef60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fb0080f9ce0 tx=0x7faff8009040).stop 2024-04-12T05:23:53.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.313+0000 7fb00cb4b700 1 -- 172.21.15.113:0/1678104445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0080fc7f0 msgr2=0x7fb0080f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.313+0000 7fb00cb4b700 1 --2- 172.21.15.113:0/1678104445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0080fc7f0 0x7fb0080f8540 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7faffc00b120 tx=0x7faffc009cb0).stop 2024-04-12T05:23:53.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.313+0000 7fb00cb4b700 1 -- 172.21.15.113:0/1678104445 shutdown_connections 2024-04-12T05:23:53.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.313+0000 7fb00cb4b700 1 --2- 172.21.15.113:0/1678104445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faff006caa0 0x7faff006ef60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.313+0000 7fb00cb4b700 1 --2- 172.21.15.113:0/1678104445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0080fc7f0 0x7fb0080f8540 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.313+0000 7fb00cb4b700 1 --2- 172.21.15.113:0/1678104445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0080fd150 0x7fb0080f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.313+0000 7fb00cb4b700 1 -- 172.21.15.113:0/1678104445 >> 172.21.15.113:0/1678104445 conn(0x7fb008070590 msgr2=0x7fb008060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:53.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.314+0000 7fb00cb4b700 1 -- 172.21.15.113:0/1678104445 shutdown_connections 2024-04-12T05:23:53.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.314+0000 7fb00cb4b700 1 -- 172.21.15.113:0/1678104445 wait complete. 2024-04-12T05:23:53.326 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:23:53.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.412+0000 7f8748b96700 1 -- 172.21.15.113:0/1641065108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87440fd8f0 msgr2=0x7f87440fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.412+0000 7f8748b96700 1 --2- 172.21.15.113:0/1641065108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87440fd8f0 0x7f87440fdcd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f87340047f0 tx=0x7f8734009cb0).stop 2024-04-12T05:23:53.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.413+0000 7f8748b96700 1 -- 172.21.15.113:0/1641065108 shutdown_connections 2024-04-12T05:23:53.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.413+0000 7f8748b96700 1 --2- 172.21.15.113:0/1641065108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87440fe250 0x7f87440f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.413+0000 7f8748b96700 1 --2- 172.21.15.113:0/1641065108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87440fd8f0 0x7f87440fdcd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.413+0000 7f8748b96700 1 -- 172.21.15.113:0/1641065108 >> 172.21.15.113:0/1641065108 conn(0x7f87440f34b0 msgr2=0x7f87440f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:53.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.413+0000 7f8748b96700 1 -- 172.21.15.113:0/1641065108 shutdown_connections 2024-04-12T05:23:53.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.413+0000 7f8748b96700 1 -- 172.21.15.113:0/1641065108 wait complete. 2024-04-12T05:23:53.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.413+0000 7f8748b96700 1 Processor -- start 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.413+0000 7f8748b96700 1 -- start start 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f8748b96700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87440fd8f0 0x7f874406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f8748b96700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87440fe250 0x7f874406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f8748b96700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874406bab0 con 0x7f87440fd8f0 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f8748b96700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874406bbf0 con 0x7f87440fe250 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f874259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87440fd8f0 0x7f874406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f874259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87440fd8f0 0x7f874406b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54198/0 (socket says 172.21.15.113:54198) 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f874259c700 1 -- 172.21.15.113:0/2495711784 learned_addr learned my addr 172.21.15.113:0/2495711784 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f8741d9b700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87440fe250 0x7f874406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f874259c700 1 -- 172.21.15.113:0/2495711784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87440fe250 msgr2=0x7f874406b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f874259c700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87440fe250 0x7f874406b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f874259c700 1 -- 172.21.15.113:0/2495711784 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8734005040 con 0x7f87440fd8f0 2024-04-12T05:23:53.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f8741d9b700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87440fe250 0x7f874406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:23:53.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f874259c700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87440fd8f0 0x7f874406b0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f8734005010 tx=0x7f8734009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:53.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f87337fe700 1 -- 172.21.15.113:0/2495711784 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f873400e030 con 0x7f87440fd8f0 2024-04-12T05:23:53.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f87337fe700 1 -- 172.21.15.113:0/2495711784 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8734007e90 con 0x7f87440fd8f0 2024-04-12T05:23:53.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f8748b96700 1 -- 172.21.15.113:0/2495711784 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874406be40 con 0x7f87440fd8f0 2024-04-12T05:23:53.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f87337fe700 1 -- 172.21.15.113:0/2495711784 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8734018070 con 0x7f87440fd8f0 2024-04-12T05:23:53.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.414+0000 7f8748b96700 1 -- 172.21.15.113:0/2495711784 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87441145f0 con 0x7f87440fd8f0 2024-04-12T05:23:53.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.415+0000 7f8748b96700 1 -- 172.21.15.113:0/2495711784 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874405f590 con 0x7f87440fd8f0 2024-04-12T05:23:53.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.416+0000 7f87337fe700 1 -- 172.21.15.113:0/2495711784 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f873401d6e0 con 0x7f87440fd8f0 2024-04-12T05:23:53.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.416+0000 7f87337fe700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f872c06ca30 0x7f872c06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:53.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.416+0000 7f87337fe700 1 -- 172.21.15.113:0/2495711784 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f873408d100 con 0x7f87440fd8f0 2024-04-12T05:23:53.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.416+0000 7f8741d9b700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f872c06ca30 0x7f872c06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:53.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.416+0000 7f8741d9b700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f872c06ca30 0x7f872c06eef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f874406c870 tx=0x7f8738009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:53.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.420+0000 7f87337fe700 1 -- 172.21.15.113:0/2495711784 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f873405abb0 con 0x7f87440fd8f0 2024-04-12T05:23:53.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.625+0000 7f8748b96700 1 -- 172.21.15.113:0/2495711784 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8744059120 con 0x7f872c06ca30 2024-04-12T05:23:53.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.626+0000 7f87337fe700 1 -- 172.21.15.113:0/2495711784 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8744059120 con 0x7f872c06ca30 2024-04-12T05:23:53.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.628+0000 7f8748b96700 1 -- 172.21.15.113:0/2495711784 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f872c06ca30 msgr2=0x7f872c06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.628+0000 7f8748b96700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f872c06ca30 0x7f872c06eef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f874406c870 tx=0x7f8738009d60).stop 2024-04-12T05:23:53.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.628+0000 7f8748b96700 1 -- 172.21.15.113:0/2495711784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87440fd8f0 msgr2=0x7f874406b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.628+0000 7f8748b96700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87440fd8f0 0x7f874406b0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f8734005010 tx=0x7f8734009cb0).stop 2024-04-12T05:23:53.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.628+0000 7f8748b96700 1 -- 172.21.15.113:0/2495711784 shutdown_connections 2024-04-12T05:23:53.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.628+0000 7f8748b96700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f872c06ca30 0x7f872c06eef0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.628+0000 7f8748b96700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87440fd8f0 0x7f874406b0d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.628+0000 7f8748b96700 1 --2- 172.21.15.113:0/2495711784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87440fe250 0x7f874406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.628+0000 7f8748b96700 1 -- 172.21.15.113:0/2495711784 >> 172.21.15.113:0/2495711784 conn(0x7f87440f34b0 msgr2=0x7f87440fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:53.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.628+0000 7f8748b96700 1 -- 172.21.15.113:0/2495711784 shutdown_connections 2024-04-12T05:23:53.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.628+0000 7f8748b96700 1 -- 172.21.15.113:0/2495711784 wait complete. 2024-04-12T05:23:53.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.726+0000 7f433267b700 1 -- 172.21.15.113:0/3808528262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f432c0fc7f0 msgr2=0x7f432c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.726+0000 7f433267b700 1 --2- 172.21.15.113:0/3808528262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f432c0fc7f0 0x7f432c0fcbd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f431c0047f0 tx=0x7f431c009cb0).stop 2024-04-12T05:23:53.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.726+0000 7f433267b700 1 -- 172.21.15.113:0/3808528262 shutdown_connections 2024-04-12T05:23:53.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.726+0000 7f433267b700 1 --2- 172.21.15.113:0/3808528262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f432c0fd150 0x7f432c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.726+0000 7f433267b700 1 --2- 172.21.15.113:0/3808528262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f432c0fc7f0 0x7f432c0fcbd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.726+0000 7f433267b700 1 -- 172.21.15.113:0/3808528262 >> 172.21.15.113:0/3808528262 conn(0x7f432c070590 msgr2=0x7f432c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:53.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.726+0000 7f433267b700 1 -- 172.21.15.113:0/3808528262 shutdown_connections 2024-04-12T05:23:53.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.726+0000 7f433267b700 1 -- 172.21.15.113:0/3808528262 wait complete. 2024-04-12T05:23:53.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.727+0000 7f433267b700 1 Processor -- start 2024-04-12T05:23:53.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.727+0000 7f433267b700 1 -- start start 2024-04-12T05:23:53.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.727+0000 7f433267b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f432c0fc7f0 0x7f432c10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:53.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.727+0000 7f433267b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f432c0fd150 0x7f432c10e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:53.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.727+0000 7f433267b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c10eb20 con 0x7f432c0fd150 2024-04-12T05:23:53.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.727+0000 7f433267b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c10ec60 con 0x7f432c0fc7f0 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f432b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f432c0fd150 0x7f432c10e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f432b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f432c0fd150 0x7f432c10e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54210/0 (socket says 172.21.15.113:54210) 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f432b7fe700 1 -- 172.21.15.113:0/971859501 learned_addr learned my addr 172.21.15.113:0/971859501 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f432bfff700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f432c0fc7f0 0x7f432c10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f432b7fe700 1 -- 172.21.15.113:0/971859501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f432c0fc7f0 msgr2=0x7f432c10e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f432b7fe700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f432c0fc7f0 0x7f432c10e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f432b7fe700 1 -- 172.21.15.113:0/971859501 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c005040 con 0x7f432c0fd150 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f432bfff700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f432c0fc7f0 0x7f432c10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f432b7fe700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f432c0fd150 0x7f432c10e520 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f4320002820 tx=0x7f4320009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f43297fa700 1 -- 172.21.15.113:0/971859501 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4320007b40 con 0x7f432c0fd150 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f43297fa700 1 -- 172.21.15.113:0/971859501 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f432000f040 con 0x7f432c0fd150 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f433267b700 1 -- 172.21.15.113:0/971859501 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c1082f0 con 0x7f432c0fd150 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f43297fa700 1 -- 172.21.15.113:0/971859501 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4320014910 con 0x7f432c0fd150 2024-04-12T05:23:53.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.728+0000 7f433267b700 1 -- 172.21.15.113:0/971859501 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c1088b0 con 0x7f432c0fd150 2024-04-12T05:23:53.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.729+0000 7f433267b700 1 -- 172.21.15.113:0/971859501 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c05f590 con 0x7f432c0fd150 2024-04-12T05:23:53.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.730+0000 7f43297fa700 1 -- 172.21.15.113:0/971859501 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4320007ca0 con 0x7f432c0fd150 2024-04-12T05:23:53.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.730+0000 7f43297fa700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f431406caa0 0x7f431406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:53.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.730+0000 7f43297fa700 1 -- 172.21.15.113:0/971859501 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f432008b7f0 con 0x7f432c0fd150 2024-04-12T05:23:53.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.730+0000 7f432bfff700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f431406caa0 0x7f431406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:53.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.730+0000 7f432bfff700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f431406caa0 0x7f431406ef60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f431c0043d0 tx=0x7f431c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:53.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.734+0000 7f43297fa700 1 -- 172.21.15.113:0/971859501 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f43200592a0 con 0x7f432c0fd150 2024-04-12T05:23:53.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.931+0000 7f433267b700 1 -- 172.21.15.113:0/971859501 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f432c108f50 con 0x7f431406caa0 2024-04-12T05:23:53.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.941+0000 7f43297fa700 1 -- 172.21.15.113:0/971859501 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f432c108f50 con 0x7f431406caa0 2024-04-12T05:23:53.943 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (11m) 4m ago 22m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (9m) 4m ago 22m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (9m) 4m ago 22m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (11m) 4m ago 22m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (5m) 4m ago 17m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (4m) 4m ago 17m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (4m) 4m ago 17m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (17m) 4m ago 17m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (13m) 4m ago 24m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (12m) 4m ago 22m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (10m) 4m ago 24m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (9m) 4m ago 22m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (12m) 4m ago 22m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:23:53.944 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (12m) 4m ago 22m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:23:53.945 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (8m) 4m ago 21m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:23:53.945 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (8m) 4m ago 20m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:23:53.945 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (7m) 4m ago 20m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:23:53.945 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (6m) 4m ago 20m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:23:53.945 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (6m) 4m ago 19m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:23:53.945 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (5m) 4m ago 19m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:23:53.945 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (12m) 4m ago 21m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:23:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.943+0000 7f433267b700 1 -- 172.21.15.113:0/971859501 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f431406caa0 msgr2=0x7f431406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.943+0000 7f433267b700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f431406caa0 0x7f431406ef60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f431c0043d0 tx=0x7f431c00b040).stop 2024-04-12T05:23:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.943+0000 7f433267b700 1 -- 172.21.15.113:0/971859501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f432c0fd150 msgr2=0x7f432c10e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.943+0000 7f433267b700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f432c0fd150 0x7f432c10e520 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f4320002820 tx=0x7f4320009f20).stop 2024-04-12T05:23:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.943+0000 7f433267b700 1 -- 172.21.15.113:0/971859501 shutdown_connections 2024-04-12T05:23:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.943+0000 7f433267b700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f431406caa0 0x7f431406ef60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.943+0000 7f433267b700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f432c0fc7f0 0x7f432c10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.943+0000 7f433267b700 1 --2- 172.21.15.113:0/971859501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f432c0fd150 0x7f432c10e520 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.943+0000 7f433267b700 1 -- 172.21.15.113:0/971859501 >> 172.21.15.113:0/971859501 conn(0x7f432c070590 msgr2=0x7f432c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.943+0000 7f433267b700 1 -- 172.21.15.113:0/971859501 shutdown_connections 2024-04-12T05:23:53.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:53.943+0000 7f433267b700 1 -- 172.21.15.113:0/971859501 wait complete. 2024-04-12T05:23:54.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.040+0000 7febb92a3700 1 -- 172.21.15.113:0/3403647912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 msgr2=0x7febb40fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.040+0000 7febb92a3700 1 --2- 172.21.15.113:0/3403647912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb40fcbd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7feba80047f0 tx=0x7feba8009cb0).stop 2024-04-12T05:23:54.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.040+0000 7febb92a3700 1 -- 172.21.15.113:0/3403647912 shutdown_connections 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.040+0000 7febb92a3700 1 --2- 172.21.15.113:0/3403647912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 0x7febb40f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.040+0000 7febb92a3700 1 --2- 172.21.15.113:0/3403647912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb40fcbd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.040+0000 7febb92a3700 1 -- 172.21.15.113:0/3403647912 >> 172.21.15.113:0/3403647912 conn(0x7febb40f23b0 msgr2=0x7febb40f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.040+0000 7febb92a3700 1 -- 172.21.15.113:0/3403647912 shutdown_connections 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.040+0000 7febb92a3700 1 -- 172.21.15.113:0/3403647912 wait complete. 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb92a3700 1 Processor -- start 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb92a3700 1 -- start start 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb92a3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb4109190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb92a3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 0x7febb4109680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb92a3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb4109c00 con 0x7febb40fc7f0 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb92a3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb4109d40 con 0x7febb40fd150 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb4109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb4109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54234/0 (socket says 172.21.15.113:54234) 2024-04-12T05:23:54.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb2ffd700 1 -- 172.21.15.113:0/2436113143 learned_addr learned my addr 172.21.15.113:0/2436113143 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:54.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb27fc700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 0x7febb4109680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:54.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb2ffd700 1 -- 172.21.15.113:0/2436113143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 msgr2=0x7febb4109680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb2ffd700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 0x7febb4109680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb2ffd700 1 -- 172.21.15.113:0/2436113143 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba8005040 con 0x7febb40fc7f0 2024-04-12T05:23:54.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.041+0000 7febb27fc700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 0x7febb4109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:23:54.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.042+0000 7febb2ffd700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb4109190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7feba8013640 tx=0x7feba8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:54.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.042+0000 7feb9bfff700 1 -- 172.21.15.113:0/2436113143 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feba800e030 con 0x7febb40fc7f0 2024-04-12T05:23:54.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.042+0000 7feb9bfff700 1 -- 172.21.15.113:0/2436113143 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feba8007e90 con 0x7febb40fc7f0 2024-04-12T05:23:54.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.042+0000 7febb92a3700 1 -- 172.21.15.113:0/2436113143 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb4109f90 con 0x7febb40fc7f0 2024-04-12T05:23:54.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.042+0000 7feb9bfff700 1 -- 172.21.15.113:0/2436113143 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feba8018070 con 0x7febb40fc7f0 2024-04-12T05:23:54.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.042+0000 7febb92a3700 1 -- 172.21.15.113:0/2436113143 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb41124d0 con 0x7febb40fc7f0 2024-04-12T05:23:54.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.043+0000 7febb92a3700 1 -- 172.21.15.113:0/2436113143 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb4060390 con 0x7febb40fc7f0 2024-04-12T05:23:54.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.043+0000 7feb9bfff700 1 -- 172.21.15.113:0/2436113143 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7feba801d420 con 0x7febb40fc7f0 2024-04-12T05:23:54.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.043+0000 7feb9bfff700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06cab0 0x7feb9c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:54.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.044+0000 7feb9bfff700 1 -- 172.21.15.113:0/2436113143 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7feba808d250 con 0x7febb40fc7f0 2024-04-12T05:23:54.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.044+0000 7febb27fc700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06cab0 0x7feb9c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:54.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.044+0000 7febb27fc700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06cab0 0x7feb9c06ef70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7febb40614d0 tx=0x7feba4008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:54.050 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.048+0000 7feb9bfff700 1 -- 172.21.15.113:0/2436113143 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7feba8057270 con 0x7febb40fc7f0 2024-04-12T05:23:54.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.315+0000 7febb92a3700 1 -- 172.21.15.113:0/2436113143 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7febb4061310 con 0x7febb40fc7f0 2024-04-12T05:23:54.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.316+0000 7feb9bfff700 1 -- 172.21.15.113:0/2436113143 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7feba805a890 con 0x7febb40fc7f0 2024-04-12T05:23:54.318 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:23:54.318 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:23:54.318 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:23:54.319 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:23:54.320 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:23:54.320 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:23:54.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.318+0000 7febb92a3700 1 -- 172.21.15.113:0/2436113143 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06cab0 msgr2=0x7feb9c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.318+0000 7febb92a3700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06cab0 0x7feb9c06ef70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7febb40614d0 tx=0x7feba4008040).stop 2024-04-12T05:23:54.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.318+0000 7febb92a3700 1 -- 172.21.15.113:0/2436113143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 msgr2=0x7febb4109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.318+0000 7febb92a3700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb4109190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7feba8013640 tx=0x7feba8009cb0).stop 2024-04-12T05:23:54.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.318+0000 7febb92a3700 1 -- 172.21.15.113:0/2436113143 shutdown_connections 2024-04-12T05:23:54.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.318+0000 7febb92a3700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb9c06cab0 0x7feb9c06ef70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.318+0000 7febb92a3700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7febb40fc7f0 0x7febb4109190 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.318+0000 7febb92a3700 1 --2- 172.21.15.113:0/2436113143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febb40fd150 0x7febb4109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.318+0000 7febb92a3700 1 -- 172.21.15.113:0/2436113143 >> 172.21.15.113:0/2436113143 conn(0x7febb40f23b0 msgr2=0x7febb40fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:54.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.318+0000 7febb92a3700 1 -- 172.21.15.113:0/2436113143 shutdown_connections 2024-04-12T05:23:54.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.318+0000 7febb92a3700 1 -- 172.21.15.113:0/2436113143 wait complete. 2024-04-12T05:23:54.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.416+0000 7ff002784700 1 -- 172.21.15.113:0/912272664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feffc0fc7f0 msgr2=0x7feffc0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.416+0000 7ff002784700 1 --2- 172.21.15.113:0/912272664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feffc0fc7f0 0x7feffc0fcbd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fefec0047f0 tx=0x7fefec009cb0).stop 2024-04-12T05:23:54.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.416+0000 7ff002784700 1 -- 172.21.15.113:0/912272664 shutdown_connections 2024-04-12T05:23:54.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.416+0000 7ff002784700 1 --2- 172.21.15.113:0/912272664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0fd150 0x7feffc0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.416+0000 7ff002784700 1 --2- 172.21.15.113:0/912272664 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feffc0fc7f0 0x7feffc0fcbd0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.416+0000 7ff002784700 1 -- 172.21.15.113:0/912272664 >> 172.21.15.113:0/912272664 conn(0x7feffc0f23b0 msgr2=0x7feffc0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:54.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.416+0000 7ff002784700 1 -- 172.21.15.113:0/912272664 shutdown_connections 2024-04-12T05:23:54.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.416+0000 7ff002784700 1 -- 172.21.15.113:0/912272664 wait complete. 2024-04-12T05:23:54.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.417+0000 7ff002784700 1 Processor -- start 2024-04-12T05:23:54.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.417+0000 7ff002784700 1 -- start start 2024-04-12T05:23:54.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.417+0000 7ff002784700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feffc0fc7f0 0x7feffc10cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:54.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.417+0000 7ff002784700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0fd150 0x7feffc10d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:54.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.417+0000 7ff002784700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc10da30 con 0x7feffc0fc7f0 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.417+0000 7ff002784700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc10db70 con 0x7feffc0fd150 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.417+0000 7feffbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feffc0fc7f0 0x7feffc10cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.417+0000 7feffbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feffc0fc7f0 0x7feffc10cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54244/0 (socket says 172.21.15.113:54244) 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.417+0000 7feffbfff700 1 -- 172.21.15.113:0/3126383949 learned_addr learned my addr 172.21.15.113:0/3126383949 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.418+0000 7feffb7fe700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0fd150 0x7feffc10d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.418+0000 7feffbfff700 1 -- 172.21.15.113:0/3126383949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0fd150 msgr2=0x7feffc10d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.418+0000 7feffbfff700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0fd150 0x7feffc10d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.418+0000 7feffbfff700 1 -- 172.21.15.113:0/3126383949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec005040 con 0x7feffc0fc7f0 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.418+0000 7feffb7fe700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0fd150 0x7feffc10d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.418+0000 7feffbfff700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feffc0fc7f0 0x7feffc10cf40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fefec00b120 tx=0x7fefec009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.418+0000 7feff97fa700 1 -- 172.21.15.113:0/3126383949 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefec00e030 con 0x7feffc0fc7f0 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.418+0000 7feff97fa700 1 -- 172.21.15.113:0/3126383949 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefec007e90 con 0x7feffc0fc7f0 2024-04-12T05:23:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.418+0000 7ff002784700 1 -- 172.21.15.113:0/3126383949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc1071a0 con 0x7feffc0fc7f0 2024-04-12T05:23:54.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.418+0000 7feff97fa700 1 -- 172.21.15.113:0/3126383949 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefec018070 con 0x7feffc0fc7f0 2024-04-12T05:23:54.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.418+0000 7ff002784700 1 -- 172.21.15.113:0/3126383949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc107700 con 0x7feffc0fc7f0 2024-04-12T05:23:54.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.419+0000 7ff002784700 1 -- 172.21.15.113:0/3126383949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feffc060370 con 0x7feffc0fc7f0 2024-04-12T05:23:54.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.419+0000 7feff97fa700 1 -- 172.21.15.113:0/3126383949 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fefec01d420 con 0x7feffc0fc7f0 2024-04-12T05:23:54.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.420+0000 7feff97fa700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fefe406caa0 0x7fefe406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:54.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.420+0000 7feff97fa700 1 -- 172.21.15.113:0/3126383949 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fefec08d300 con 0x7feffc0fc7f0 2024-04-12T05:23:54.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.420+0000 7feffb7fe700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fefe406caa0 0x7fefe406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:54.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.420+0000 7feffb7fe700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fefe406caa0 0x7fefe406ef60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7feffc108880 tx=0x7feff0009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:54.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.424+0000 7feff97fa700 1 -- 172.21.15.113:0/3126383949 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fefec05adb0 con 0x7feffc0fc7f0 2024-04-12T05:23:54.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.652+0000 7ff002784700 1 -- 172.21.15.113:0/3126383949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feffc1086c0 con 0x7feffc0fc7f0 2024-04-12T05:23:54.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.652+0000 7feff97fa700 1 -- 172.21.15.113:0/3126383949 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7fefec05a940 con 0x7feffc0fc7f0 2024-04-12T05:23:54.655 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:23:54.655 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:23:54.655 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:23:54.656 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:23:54.657 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:23:54.657 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:23:54.657 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:23:54.657 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:23:54.657 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:23:54.657 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:23:54.657 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:23:54.657 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:23:54.658 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:23:54.658 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:23:54.658 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:23:54.658 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:23:54.658 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:23:54.658 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:23:54.658 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:23:54.658 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:23:54.658 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:23:54.658 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:23:54.659 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:23:54.659 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:23:54.659 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:23:54.659 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:23:54.659 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:23:54.659 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:23:54.659 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:23:54.659 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:23:54.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.654+0000 7ff002784700 1 -- 172.21.15.113:0/3126383949 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fefe406caa0 msgr2=0x7fefe406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.654+0000 7ff002784700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fefe406caa0 0x7fefe406ef60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7feffc108880 tx=0x7feff0009040).stop 2024-04-12T05:23:54.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.654+0000 7ff002784700 1 -- 172.21.15.113:0/3126383949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feffc0fc7f0 msgr2=0x7feffc10cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.654+0000 7ff002784700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feffc0fc7f0 0x7feffc10cf40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fefec00b120 tx=0x7fefec009cb0).stop 2024-04-12T05:23:54.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.654+0000 7ff002784700 1 -- 172.21.15.113:0/3126383949 shutdown_connections 2024-04-12T05:23:54.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.654+0000 7ff002784700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fefe406caa0 0x7fefe406ef60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.654+0000 7ff002784700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feffc0fc7f0 0x7feffc10cf40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.654+0000 7ff002784700 1 --2- 172.21.15.113:0/3126383949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0fd150 0x7feffc10d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.654+0000 7ff002784700 1 -- 172.21.15.113:0/3126383949 >> 172.21.15.113:0/3126383949 conn(0x7feffc0f23b0 msgr2=0x7feffc0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:54.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.655+0000 7ff002784700 1 -- 172.21.15.113:0/3126383949 shutdown_connections 2024-04-12T05:23:54.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.655+0000 7ff002784700 1 -- 172.21.15.113:0/3126383949 wait complete. 2024-04-12T05:23:54.661 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:23:54.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.754+0000 7f7f93ad7700 1 -- 172.21.15.113:0/3948376068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f8c0fe250 msgr2=0x7f7f8c0f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.754+0000 7f7f93ad7700 1 --2- 172.21.15.113:0/3948376068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f8c0fe250 0x7f7f8c0f7a30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f7f880047f0 tx=0x7f7f88009cb0).stop 2024-04-12T05:23:54.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.754+0000 7f7f93ad7700 1 -- 172.21.15.113:0/3948376068 shutdown_connections 2024-04-12T05:23:54.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.754+0000 7f7f93ad7700 1 --2- 172.21.15.113:0/3948376068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f8c0fe250 0x7f7f8c0f7a30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.754+0000 7f7f93ad7700 1 --2- 172.21.15.113:0/3948376068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f8c0fd8f0 0x7f7f8c0fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.754+0000 7f7f93ad7700 1 -- 172.21.15.113:0/3948376068 >> 172.21.15.113:0/3948376068 conn(0x7f7f8c0f34b0 msgr2=0x7f7f8c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:54.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.754+0000 7f7f93ad7700 1 -- 172.21.15.113:0/3948376068 shutdown_connections 2024-04-12T05:23:54.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.754+0000 7f7f93ad7700 1 -- 172.21.15.113:0/3948376068 wait complete. 2024-04-12T05:23:54.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.754+0000 7f7f93ad7700 1 Processor -- start 2024-04-12T05:23:54.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f93ad7700 1 -- start start 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f93ad7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f8c0fd8f0 0x7f7f8c10e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f93ad7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f8c0fe250 0x7f7f8c10e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f93ad7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c10ec30 con 0x7f7f8c0fe250 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f93ad7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c10ed70 con 0x7f7f8c0fd8f0 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f91072700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f8c0fe250 0x7f7f8c10e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f91072700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f8c0fe250 0x7f7f8c10e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54262/0 (socket says 172.21.15.113:54262) 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f91072700 1 -- 172.21.15.113:0/1048846927 learned_addr learned my addr 172.21.15.113:0/1048846927 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f91072700 1 -- 172.21.15.113:0/1048846927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f8c0fd8f0 msgr2=0x7f7f8c10e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f91072700 1 --2- 172.21.15.113:0/1048846927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f8c0fd8f0 0x7f7f8c10e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f91072700 1 -- 172.21.15.113:0/1048846927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f88005040 con 0x7f7f8c0fe250 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f91072700 1 --2- 172.21.15.113:0/1048846927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f8c0fe250 0x7f7f8c10e630 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f7f88000c00 tx=0x7f7f88009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f82f78700 1 -- 172.21.15.113:0/1048846927 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f88018070 con 0x7f7f8c0fe250 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f93ad7700 1 -- 172.21.15.113:0/1048846927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8c1083a0 con 0x7f7f8c0fe250 2024-04-12T05:23:54.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f82f78700 1 -- 172.21.15.113:0/1048846927 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f88007e90 con 0x7f7f8c0fe250 2024-04-12T05:23:54.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f82f78700 1 -- 172.21.15.113:0/1048846927 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f880123c0 con 0x7f7f8c0fe250 2024-04-12T05:23:54.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.755+0000 7f7f93ad7700 1 -- 172.21.15.113:0/1048846927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8c108900 con 0x7f7f8c0fe250 2024-04-12T05:23:54.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.756+0000 7f7f93ad7700 1 -- 172.21.15.113:0/1048846927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8c05f590 con 0x7f7f8c0fe250 2024-04-12T05:23:54.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.757+0000 7f7f82f78700 1 -- 172.21.15.113:0/1048846927 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7f8801d420 con 0x7f7f8c0fe250 2024-04-12T05:23:54.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.757+0000 7f7f82f78700 1 --2- 172.21.15.113:0/1048846927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7f7806caa0 0x7f7f7806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:54.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.758+0000 7f7f91873700 1 --2- 172.21.15.113:0/1048846927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7f7806caa0 0x7f7f7806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:54.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.758+0000 7f7f82f78700 1 -- 172.21.15.113:0/1048846927 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7f8808caf0 con 0x7f7f8c0fe250 2024-04-12T05:23:54.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.758+0000 7f7f91873700 1 --2- 172.21.15.113:0/1048846927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7f7806caa0 0x7f7f7806ef60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7f7c004c60 tx=0x7f7f7c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:54.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.762+0000 7f7f82f78700 1 -- 172.21.15.113:0/1048846927 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7f8805a5a0 con 0x7f7f8c0fe250 2024-04-12T05:23:54.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.969+0000 7f7f93ad7700 1 -- 172.21.15.113:0/1048846927 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f8c1098d0 con 0x7f7f7806caa0 2024-04-12T05:23:54.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.970+0000 7f7f82f78700 1 -- 172.21.15.113:0/1048846927 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7f8c1098d0 con 0x7f7f7806caa0 2024-04-12T05:23:54.972 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:23:54.972 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:23:54.972 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:23:54.973 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:23:54.973 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:23:54.973 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:23:54.973 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:23:54.973 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:23:54.973 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:23:54.973 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:23:54.973 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:23:54.973 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:23:54.973 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:23:54.973 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:23:54.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.972+0000 7f7f93ad7700 1 -- 172.21.15.113:0/1048846927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7f7806caa0 msgr2=0x7f7f7806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.972+0000 7f7f93ad7700 1 --2- 172.21.15.113:0/1048846927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7f7806caa0 0x7f7f7806ef60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7f7c004c60 tx=0x7f7f7c00b040).stop 2024-04-12T05:23:54.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.972+0000 7f7f93ad7700 1 -- 172.21.15.113:0/1048846927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f8c0fe250 msgr2=0x7f7f8c10e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:54.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.972+0000 7f7f93ad7700 1 --2- 172.21.15.113:0/1048846927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f8c0fe250 0x7f7f8c10e630 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f7f88000c00 tx=0x7f7f88009cb0).stop 2024-04-12T05:23:54.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.972+0000 7f7f93ad7700 1 -- 172.21.15.113:0/1048846927 shutdown_connections 2024-04-12T05:23:54.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.972+0000 7f7f93ad7700 1 --2- 172.21.15.113:0/1048846927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7f7806caa0 0x7f7f7806ef60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.972+0000 7f7f93ad7700 1 --2- 172.21.15.113:0/1048846927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7f8c0fd8f0 0x7f7f8c10e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:54.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.972+0000 7f7f93ad7700 1 --2- 172.21.15.113:0/1048846927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f8c0fe250 0x7f7f8c10e630 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:54.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.972+0000 7f7f93ad7700 1 -- 172.21.15.113:0/1048846927 >> 172.21.15.113:0/1048846927 conn(0x7f7f8c0f34b0 msgr2=0x7f7f8c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:54.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.972+0000 7f7f93ad7700 1 -- 172.21.15.113:0/1048846927 shutdown_connections 2024-04-12T05:23:54.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:54.972+0000 7f7f93ad7700 1 -- 172.21.15.113:0/1048846927 wait complete. 2024-04-12T05:23:55.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.071+0000 7f204898c700 1 -- 172.21.15.113:0/718973184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20440fc7f0 msgr2=0x7f20440fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:55.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.071+0000 7f204898c700 1 --2- 172.21.15.113:0/718973184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20440fc7f0 0x7f20440fcbd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f20340047f0 tx=0x7f2034009cb0).stop 2024-04-12T05:23:55.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.071+0000 7f204898c700 1 -- 172.21.15.113:0/718973184 shutdown_connections 2024-04-12T05:23:55.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.072+0000 7f204898c700 1 --2- 172.21.15.113:0/718973184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20440fd150 0x7f20440f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:55.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.072+0000 7f204898c700 1 --2- 172.21.15.113:0/718973184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20440fc7f0 0x7f20440fcbd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:55.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.072+0000 7f204898c700 1 -- 172.21.15.113:0/718973184 >> 172.21.15.113:0/718973184 conn(0x7f2044070590 msgr2=0x7f20440f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:55.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.072+0000 7f204898c700 1 -- 172.21.15.113:0/718973184 shutdown_connections 2024-04-12T05:23:55.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.072+0000 7f204898c700 1 -- 172.21.15.113:0/718973184 wait complete. 2024-04-12T05:23:55.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.072+0000 7f204898c700 1 Processor -- start 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204898c700 1 -- start start 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204898c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20440fc7f0 0x7f204410cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204898c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20440fd150 0x7f204410d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204898c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204410d9f0 con 0x7f20440fc7f0 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204898c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204410db30 con 0x7f20440fd150 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20440fc7f0 0x7f204410cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20440fc7f0 0x7f204410cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54278/0 (socket says 172.21.15.113:54278) 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204259c700 1 -- 172.21.15.113:0/3318852258 learned_addr learned my addr 172.21.15.113:0/3318852258 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204259c700 1 -- 172.21.15.113:0/3318852258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20440fd150 msgr2=0x7f204410d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204259c700 1 --2- 172.21.15.113:0/3318852258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20440fd150 0x7f204410d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204259c700 1 -- 172.21.15.113:0/3318852258 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2034005040 con 0x7f20440fc7f0 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204259c700 1 --2- 172.21.15.113:0/3318852258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20440fc7f0 0x7f204410cf00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f2034013640 tx=0x7f2034009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:55.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f20337fe700 1 -- 172.21.15.113:0/3318852258 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f203400e030 con 0x7f20440fc7f0 2024-04-12T05:23:55.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f20337fe700 1 -- 172.21.15.113:0/3318852258 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2034007e90 con 0x7f20440fc7f0 2024-04-12T05:23:55.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.073+0000 7f204898c700 1 -- 172.21.15.113:0/3318852258 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2044107160 con 0x7f20440fc7f0 2024-04-12T05:23:55.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.074+0000 7f20337fe700 1 -- 172.21.15.113:0/3318852258 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2034018070 con 0x7f20440fc7f0 2024-04-12T05:23:55.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.074+0000 7f204898c700 1 -- 172.21.15.113:0/3318852258 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20441076c0 con 0x7f20440fc7f0 2024-04-12T05:23:55.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.075+0000 7f204898c700 1 -- 172.21.15.113:0/3318852258 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204405f590 con 0x7f20440fc7f0 2024-04-12T05:23:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.075+0000 7f20337fe700 1 -- 172.21.15.113:0/3318852258 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f203401d420 con 0x7f20440fc7f0 2024-04-12T05:23:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.075+0000 7f20337fe700 1 --2- 172.21.15.113:0/3318852258 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f202c06cab0 0x7f202c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:23:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.075+0000 7f20337fe700 1 -- 172.21.15.113:0/3318852258 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f203408d270 con 0x7f20440fc7f0 2024-04-12T05:23:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.075+0000 7f2041d9b700 1 --2- 172.21.15.113:0/3318852258 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f202c06cab0 0x7f202c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:23:55.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.076+0000 7f2041d9b700 1 --2- 172.21.15.113:0/3318852258 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f202c06cab0 0x7f202c06ef70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f2044108840 tx=0x7f2038007b40).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:23:55.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.080+0000 7f20337fe700 1 -- 172.21.15.113:0/3318852258 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f203405ad20 con 0x7f20440fc7f0 2024-04-12T05:23:55.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.345+0000 7f204898c700 1 -- 172.21.15.113:0/3318852258 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2044108620 con 0x7f20440fc7f0 2024-04-12T05:23:55.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.346+0000 7f20337fe700 1 -- 172.21.15.113:0/3318852258 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f203405a8b0 con 0x7f20440fc7f0 2024-04-12T05:23:55.348 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:23:55.348 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:23:55.348 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:23:55.348 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:23:55.348 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:23:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.348+0000 7f204898c700 1 -- 172.21.15.113:0/3318852258 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f202c06cab0 msgr2=0x7f202c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.348+0000 7f204898c700 1 --2- 172.21.15.113:0/3318852258 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f202c06cab0 0x7f202c06ef70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f2044108840 tx=0x7f2038007b40).stop 2024-04-12T05:23:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.348+0000 7f204898c700 1 -- 172.21.15.113:0/3318852258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20440fc7f0 msgr2=0x7f204410cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:23:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.348+0000 7f204898c700 1 --2- 172.21.15.113:0/3318852258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20440fc7f0 0x7f204410cf00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f2034013640 tx=0x7f2034009cb0).stop 2024-04-12T05:23:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.348+0000 7f204898c700 1 -- 172.21.15.113:0/3318852258 shutdown_connections 2024-04-12T05:23:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.348+0000 7f204898c700 1 --2- 172.21.15.113:0/3318852258 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f202c06cab0 0x7f202c06ef70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:55.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.348+0000 7f204898c700 1 --2- 172.21.15.113:0/3318852258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20440fc7f0 0x7f204410cf00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:23:55.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.348+0000 7f204898c700 1 --2- 172.21.15.113:0/3318852258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20440fd150 0x7f204410d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:23:55.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.348+0000 7f204898c700 1 -- 172.21.15.113:0/3318852258 >> 172.21.15.113:0/3318852258 conn(0x7f2044070590 msgr2=0x7f20440fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:23:55.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.348+0000 7f204898c700 1 -- 172.21.15.113:0/3318852258 shutdown_connections 2024-04-12T05:23:55.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:23:55.348+0000 7f204898c700 1 -- 172.21.15.113:0/3318852258 wait complete. 2024-04-12T05:23:55.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:54 smithi113 ceph-mon[103371]: from='client.34672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:55.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:54 smithi113 ceph-mon[103371]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:55.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:54 smithi113 ceph-mon[103371]: pgmap v352: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:55.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:54 smithi113 ceph-mon[103371]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:55.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:54 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2436113143' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:23:55.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:54 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3126383949' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:23:55.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:54 smithi192 ceph-mon[85304]: from='client.34672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:55.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:54 smithi192 ceph-mon[85304]: from='client.34676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:55.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:54 smithi192 ceph-mon[85304]: pgmap v352: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:55.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:54 smithi192 ceph-mon[85304]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:55.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:54 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2436113143' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:23:55.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:54 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3126383949' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:23:56.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:55 smithi113 ceph-mon[103371]: from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:56.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:55 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3318852258' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:23:56.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:55 smithi192 ceph-mon[85304]: from='client.34692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:23:56.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:55 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3318852258' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:23:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:56 smithi113 ceph-mon[103371]: pgmap v353: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:57.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:56 smithi192 ceph-mon[85304]: pgmap v353: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:23:58 smithi113 ceph-mon[103371]: pgmap v354: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:23:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:23:58 smithi192 ceph-mon[85304]: pgmap v354: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:00 smithi113 ceph-mon[103371]: pgmap v355: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:00 smithi192 ceph-mon[85304]: pgmap v355: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:02 smithi113 ceph-mon[103371]: pgmap v356: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:03.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:02 smithi192 ceph-mon[85304]: pgmap v356: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:04 smithi113 ceph-mon[103371]: pgmap v357: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:05.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:04 smithi192 ceph-mon[85304]: pgmap v357: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:06 smithi113 ceph-mon[103371]: pgmap v358: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:24:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:06 smithi192 ceph-mon[85304]: pgmap v358: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:24:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:08 smithi113 ceph-mon[103371]: pgmap v359: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:08 smithi192 ceph-mon[85304]: pgmap v359: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:10 smithi113 ceph-mon[103371]: pgmap v360: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:10 smithi192 ceph-mon[85304]: pgmap v360: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:12 smithi113 ceph-mon[103371]: pgmap v361: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:12 smithi192 ceph-mon[85304]: pgmap v361: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:14 smithi113 ceph-mon[103371]: pgmap v362: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:15.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:14 smithi192 ceph-mon[85304]: pgmap v362: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:16 smithi113 ceph-mon[103371]: pgmap v363: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:17.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:16 smithi192 ceph-mon[85304]: pgmap v363: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:19 smithi113 ceph-mon[103371]: pgmap v364: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:19 smithi192 ceph-mon[85304]: pgmap v364: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:21 smithi113 ceph-mon[103371]: pgmap v365: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:21 smithi192 ceph-mon[85304]: pgmap v365: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:22.269 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:24:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:24:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:23 smithi113 ceph-mon[103371]: pgmap v366: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:23 smithi192 ceph-mon[85304]: pgmap v366: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:25 smithi113 ceph-mon[103371]: pgmap v367: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:25 smithi192 ceph-mon[85304]: pgmap v367: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:25.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.460+0000 7f2f76563700 1 -- 172.21.15.113:0/2904585566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700fd910 msgr2=0x7f2f700fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:25.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.460+0000 7f2f76563700 1 --2- 172.21.15.113:0/2904585566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700fd910 0x7f2f700fdcf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f2f600047f0 tx=0x7f2f60009cb0).stop 2024-04-12T05:24:25.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.460+0000 7f2f76563700 1 -- 172.21.15.113:0/2904585566 shutdown_connections 2024-04-12T05:24:25.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.460+0000 7f2f76563700 1 --2- 172.21.15.113:0/2904585566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700fe270 0x7f2f700f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.460+0000 7f2f76563700 1 --2- 172.21.15.113:0/2904585566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700fd910 0x7f2f700fdcf0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.460+0000 7f2f76563700 1 -- 172.21.15.113:0/2904585566 >> 172.21.15.113:0/2904585566 conn(0x7f2f700f3530 msgr2=0x7f2f700f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:25.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.461+0000 7f2f76563700 1 -- 172.21.15.113:0/2904585566 shutdown_connections 2024-04-12T05:24:25.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.461+0000 7f2f76563700 1 -- 172.21.15.113:0/2904585566 wait complete. 2024-04-12T05:24:25.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.461+0000 7f2f76563700 1 Processor -- start 2024-04-12T05:24:25.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.461+0000 7f2f76563700 1 -- start start 2024-04-12T05:24:25.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f76563700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700fd910 0x7f2f7010e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:25.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f76563700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700fe270 0x7f2f7010e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:25.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f76563700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7010eba0 con 0x7f2f700fe270 2024-04-12T05:24:25.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f76563700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7010ece0 con 0x7f2f700fd910 2024-04-12T05:24:25.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f6f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700fe270 0x7f2f7010e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:25.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f6f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700fe270 0x7f2f7010e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43376/0 (socket says 172.21.15.113:43376) 2024-04-12T05:24:25.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f6f7fe700 1 -- 172.21.15.113:0/228477482 learned_addr learned my addr 172.21.15.113:0/228477482 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:25.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f6ffff700 1 --2- 172.21.15.113:0/228477482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700fd910 0x7f2f7010e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:25.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f6f7fe700 1 -- 172.21.15.113:0/228477482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700fd910 msgr2=0x7f2f7010e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:25.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f6f7fe700 1 --2- 172.21.15.113:0/228477482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700fd910 0x7f2f7010e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f6f7fe700 1 -- 172.21.15.113:0/228477482 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f60005040 con 0x7f2f700fe270 2024-04-12T05:24:25.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f6f7fe700 1 --2- 172.21.15.113:0/228477482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700fe270 0x7f2f7010e5a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f2f64002820 tx=0x7f2f64009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:25.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/228477482 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f64007b40 con 0x7f2f700fe270 2024-04-12T05:24:25.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/228477482 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f6400f040 con 0x7f2f700fe270 2024-04-12T05:24:25.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f76563700 1 -- 172.21.15.113:0/228477482 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f70108370 con 0x7f2f700fe270 2024-04-12T05:24:25.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/228477482 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f640148b0 con 0x7f2f700fe270 2024-04-12T05:24:25.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.462+0000 7f2f76563700 1 -- 172.21.15.113:0/228477482 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f701088b0 con 0x7f2f700fe270 2024-04-12T05:24:25.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.463+0000 7f2f76563700 1 -- 172.21.15.113:0/228477482 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7005f570 con 0x7f2f700fe270 2024-04-12T05:24:25.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.464+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/228477482 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2f64007ca0 con 0x7f2f700fe270 2024-04-12T05:24:25.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.464+0000 7f2f6d7fa700 1 --2- 172.21.15.113:0/228477482 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f5806caa0 0x7f2f5806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:25.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.464+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/228477482 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2f6408b8f0 con 0x7f2f700fe270 2024-04-12T05:24:25.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.464+0000 7f2f6ffff700 1 --2- 172.21.15.113:0/228477482 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f5806caa0 0x7f2f5806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:25.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.465+0000 7f2f6ffff700 1 --2- 172.21.15.113:0/228477482 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f5806caa0 0x7f2f5806ef60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2f60004f10 tx=0x7f2f6000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:25.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.469+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/228477482 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2f640593a0 con 0x7f2f700fe270 2024-04-12T05:24:25.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.674+0000 7f2f76563700 1 -- 172.21.15.113:0/228477482 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f70109810 con 0x7f2f5806caa0 2024-04-12T05:24:25.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.675+0000 7f2f6d7fa700 1 -- 172.21.15.113:0/228477482 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2f70109810 con 0x7f2f5806caa0 2024-04-12T05:24:25.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.677+0000 7f2f76563700 1 -- 172.21.15.113:0/228477482 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f5806caa0 msgr2=0x7f2f5806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:25.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.677+0000 7f2f76563700 1 --2- 172.21.15.113:0/228477482 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f5806caa0 0x7f2f5806ef60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2f60004f10 tx=0x7f2f6000b040).stop 2024-04-12T05:24:25.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.677+0000 7f2f76563700 1 -- 172.21.15.113:0/228477482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700fe270 msgr2=0x7f2f7010e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:25.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.677+0000 7f2f76563700 1 --2- 172.21.15.113:0/228477482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700fe270 0x7f2f7010e5a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f2f64002820 tx=0x7f2f64009f20).stop 2024-04-12T05:24:25.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.677+0000 7f2f76563700 1 -- 172.21.15.113:0/228477482 shutdown_connections 2024-04-12T05:24:25.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.677+0000 7f2f76563700 1 --2- 172.21.15.113:0/228477482 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f5806caa0 0x7f2f5806ef60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.677+0000 7f2f76563700 1 --2- 172.21.15.113:0/228477482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f700fd910 0x7f2f7010e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.677+0000 7f2f76563700 1 --2- 172.21.15.113:0/228477482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f700fe270 0x7f2f7010e5a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.677+0000 7f2f76563700 1 -- 172.21.15.113:0/228477482 >> 172.21.15.113:0/228477482 conn(0x7f2f700f3530 msgr2=0x7f2f700fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:25.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.677+0000 7f2f76563700 1 -- 172.21.15.113:0/228477482 shutdown_connections 2024-04-12T05:24:25.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.677+0000 7f2f76563700 1 -- 172.21.15.113:0/228477482 wait complete. 2024-04-12T05:24:25.690 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:24:25.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.776+0000 7f5a794c4700 1 -- 172.21.15.113:0/403439655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a740fd150 msgr2=0x7f5a740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:25.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.776+0000 7f5a794c4700 1 --2- 172.21.15.113:0/403439655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a740fd150 0x7f5a740f6930 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f5a680047f0 tx=0x7f5a68009cb0).stop 2024-04-12T05:24:25.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.776+0000 7f5a794c4700 1 -- 172.21.15.113:0/403439655 shutdown_connections 2024-04-12T05:24:25.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.776+0000 7f5a794c4700 1 --2- 172.21.15.113:0/403439655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a740fd150 0x7f5a740f6930 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.776+0000 7f5a794c4700 1 --2- 172.21.15.113:0/403439655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a740fc7f0 0x7f5a740fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.776+0000 7f5a794c4700 1 -- 172.21.15.113:0/403439655 >> 172.21.15.113:0/403439655 conn(0x7f5a740f23b0 msgr2=0x7f5a740f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:25.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.776+0000 7f5a794c4700 1 -- 172.21.15.113:0/403439655 shutdown_connections 2024-04-12T05:24:25.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.776+0000 7f5a794c4700 1 -- 172.21.15.113:0/403439655 wait complete. 2024-04-12T05:24:25.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.777+0000 7f5a794c4700 1 Processor -- start 2024-04-12T05:24:25.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.777+0000 7f5a794c4700 1 -- start start 2024-04-12T05:24:25.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.777+0000 7f5a794c4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a740fc7f0 0x7f5a74109160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:25.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.777+0000 7f5a794c4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a740fd150 0x7f5a74109650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:25.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.777+0000 7f5a794c4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74109b40 con 0x7f5a740fd150 2024-04-12T05:24:25.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.777+0000 7f5a794c4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74109c80 con 0x7f5a740fc7f0 2024-04-12T05:24:25.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.777+0000 7f5a727fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a740fd150 0x7f5a74109650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:25.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.777+0000 7f5a727fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a740fd150 0x7f5a74109650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43392/0 (socket says 172.21.15.113:43392) 2024-04-12T05:24:25.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.777+0000 7f5a727fc700 1 -- 172.21.15.113:0/3655189291 learned_addr learned my addr 172.21.15.113:0/3655189291 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:25.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.777+0000 7f5a72ffd700 1 --2- 172.21.15.113:0/3655189291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a740fc7f0 0x7f5a74109160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:25.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.778+0000 7f5a727fc700 1 -- 172.21.15.113:0/3655189291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a740fc7f0 msgr2=0x7f5a74109160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.778+0000 7f5a727fc700 1 --2- 172.21.15.113:0/3655189291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a740fc7f0 0x7f5a74109160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.778+0000 7f5a727fc700 1 -- 172.21.15.113:0/3655189291 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a68005040 con 0x7f5a740fd150 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.778+0000 7f5a727fc700 1 --2- 172.21.15.113:0/3655189291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a740fd150 0x7f5a74109650 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f5a680099f0 tx=0x7f5a68009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.778+0000 7f5a5bfff700 1 -- 172.21.15.113:0/3655189291 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a6800e030 con 0x7f5a740fd150 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.778+0000 7f5a5bfff700 1 -- 172.21.15.113:0/3655189291 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a68007e90 con 0x7f5a740fd150 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.778+0000 7f5a794c4700 1 -- 172.21.15.113:0/3655189291 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a74109ed0 con 0x7f5a740fd150 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.778+0000 7f5a5bfff700 1 -- 172.21.15.113:0/3655189291 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a68018070 con 0x7f5a740fd150 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.778+0000 7f5a794c4700 1 -- 172.21.15.113:0/3655189291 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a74112440 con 0x7f5a740fd150 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.779+0000 7f5a794c4700 1 -- 172.21.15.113:0/3655189291 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a74060370 con 0x7f5a740fd150 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.779+0000 7f5a5bfff700 1 -- 172.21.15.113:0/3655189291 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5a6801d420 con 0x7f5a740fd150 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.780+0000 7f5a5bfff700 1 --2- 172.21.15.113:0/3655189291 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a5c06caa0 0x7f5a5c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.780+0000 7f5a5bfff700 1 -- 172.21.15.113:0/3655189291 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5a6808d2f0 con 0x7f5a740fd150 2024-04-12T05:24:25.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.780+0000 7f5a72ffd700 1 --2- 172.21.15.113:0/3655189291 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a5c06caa0 0x7f5a5c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:25.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.780+0000 7f5a72ffd700 1 --2- 172.21.15.113:0/3655189291 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a5c06caa0 0x7f5a5c06ef60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f5a640041d0 tx=0x7f5a64009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:25.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.784+0000 7f5a5bfff700 1 -- 172.21.15.113:0/3655189291 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5a68057310 con 0x7f5a740fd150 2024-04-12T05:24:25.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.991+0000 7f5a794c4700 1 -- 172.21.15.113:0/3655189291 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a74059120 con 0x7f5a5c06caa0 2024-04-12T05:24:25.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.992+0000 7f5a5bfff700 1 -- 172.21.15.113:0/3655189291 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5a74059120 con 0x7f5a5c06caa0 2024-04-12T05:24:25.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.994+0000 7f5a794c4700 1 -- 172.21.15.113:0/3655189291 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a5c06caa0 msgr2=0x7f5a5c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:25.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.994+0000 7f5a794c4700 1 --2- 172.21.15.113:0/3655189291 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a5c06caa0 0x7f5a5c06ef60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f5a640041d0 tx=0x7f5a64009040).stop 2024-04-12T05:24:25.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.994+0000 7f5a794c4700 1 -- 172.21.15.113:0/3655189291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a740fd150 msgr2=0x7f5a74109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:25.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.994+0000 7f5a794c4700 1 --2- 172.21.15.113:0/3655189291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a740fd150 0x7f5a74109650 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f5a680099f0 tx=0x7f5a68009cb0).stop 2024-04-12T05:24:25.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.994+0000 7f5a794c4700 1 -- 172.21.15.113:0/3655189291 shutdown_connections 2024-04-12T05:24:25.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.994+0000 7f5a794c4700 1 --2- 172.21.15.113:0/3655189291 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a5c06caa0 0x7f5a5c06ef60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.994+0000 7f5a794c4700 1 --2- 172.21.15.113:0/3655189291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a740fc7f0 0x7f5a74109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.994+0000 7f5a794c4700 1 --2- 172.21.15.113:0/3655189291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a740fd150 0x7f5a74109650 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:25.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.994+0000 7f5a794c4700 1 -- 172.21.15.113:0/3655189291 >> 172.21.15.113:0/3655189291 conn(0x7f5a740f23b0 msgr2=0x7f5a740fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:25.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.994+0000 7f5a794c4700 1 -- 172.21.15.113:0/3655189291 shutdown_connections 2024-04-12T05:24:25.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:25.994+0000 7f5a794c4700 1 -- 172.21.15.113:0/3655189291 wait complete. 2024-04-12T05:24:26.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.093+0000 7f7c2deab700 1 -- 172.21.15.113:0/2975174544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c280fe250 msgr2=0x7f7c280f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:26.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.093+0000 7f7c2deab700 1 --2- 172.21.15.113:0/2975174544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c280fe250 0x7f7c280f7a30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f7c1c0047f0 tx=0x7f7c1c009cb0).stop 2024-04-12T05:24:26.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.093+0000 7f7c2deab700 1 -- 172.21.15.113:0/2975174544 shutdown_connections 2024-04-12T05:24:26.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.093+0000 7f7c2deab700 1 --2- 172.21.15.113:0/2975174544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c280fe250 0x7f7c280f7a30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.093+0000 7f7c2deab700 1 --2- 172.21.15.113:0/2975174544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c280fd8f0 0x7f7c280fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.093+0000 7f7c2deab700 1 -- 172.21.15.113:0/2975174544 >> 172.21.15.113:0/2975174544 conn(0x7f7c280f34b0 msgr2=0x7f7c280f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:26.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.093+0000 7f7c2deab700 1 -- 172.21.15.113:0/2975174544 shutdown_connections 2024-04-12T05:24:26.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.093+0000 7f7c2deab700 1 -- 172.21.15.113:0/2975174544 wait complete. 2024-04-12T05:24:26.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.094+0000 7f7c2deab700 1 Processor -- start 2024-04-12T05:24:26.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.094+0000 7f7c2deab700 1 -- start start 2024-04-12T05:24:26.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.094+0000 7f7c2deab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c280fd8f0 0x7f7c2806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:26.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.094+0000 7f7c2deab700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c280fe250 0x7f7c2806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.094+0000 7f7c2deab700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2806bab0 con 0x7f7c280fd8f0 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.094+0000 7f7c2deab700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2806bbf0 con 0x7f7c280fe250 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.094+0000 7f7c277fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c280fd8f0 0x7f7c2806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.094+0000 7f7c277fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c280fd8f0 0x7f7c2806b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43408/0 (socket says 172.21.15.113:43408) 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.094+0000 7f7c277fe700 1 -- 172.21.15.113:0/3519221247 learned_addr learned my addr 172.21.15.113:0/3519221247 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.095+0000 7f7c26ffd700 1 --2- 172.21.15.113:0/3519221247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c280fe250 0x7f7c2806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.095+0000 7f7c277fe700 1 -- 172.21.15.113:0/3519221247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c280fe250 msgr2=0x7f7c2806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.095+0000 7f7c277fe700 1 --2- 172.21.15.113:0/3519221247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c280fe250 0x7f7c2806b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.095+0000 7f7c277fe700 1 -- 172.21.15.113:0/3519221247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c005040 con 0x7f7c280fd8f0 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.095+0000 7f7c277fe700 1 --2- 172.21.15.113:0/3519221247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c280fd8f0 0x7f7c2806b0d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f7c180041d0 tx=0x7f7c18009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.095+0000 7f7c24ff9700 1 -- 172.21.15.113:0/3519221247 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c18007b40 con 0x7f7c280fd8f0 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.095+0000 7f7c24ff9700 1 -- 172.21.15.113:0/3519221247 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c1800f040 con 0x7f7c280fd8f0 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.095+0000 7f7c2deab700 1 -- 172.21.15.113:0/3519221247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2806bea0 con 0x7f7c280fd8f0 2024-04-12T05:24:26.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.095+0000 7f7c24ff9700 1 -- 172.21.15.113:0/3519221247 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c1800d800 con 0x7f7c280fd8f0 2024-04-12T05:24:26.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.095+0000 7f7c2deab700 1 -- 172.21.15.113:0/3519221247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c281146d0 con 0x7f7c280fd8f0 2024-04-12T05:24:26.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.096+0000 7f7c2deab700 1 -- 172.21.15.113:0/3519221247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2805f590 con 0x7f7c280fd8f0 2024-04-12T05:24:26.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.096+0000 7f7c24ff9700 1 -- 172.21.15.113:0/3519221247 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7c18009110 con 0x7f7c280fd8f0 2024-04-12T05:24:26.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.097+0000 7f7c24ff9700 1 --2- 172.21.15.113:0/3519221247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1006caa0 0x7f7c1006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:26.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.097+0000 7f7c24ff9700 1 -- 172.21.15.113:0/3519221247 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7c1808b940 con 0x7f7c280fd8f0 2024-04-12T05:24:26.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.097+0000 7f7c26ffd700 1 --2- 172.21.15.113:0/3519221247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1006caa0 0x7f7c1006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:26.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.097+0000 7f7c26ffd700 1 --2- 172.21.15.113:0/3519221247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1006caa0 0x7f7c1006ef60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7c1c004f10 tx=0x7f7c1c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:26.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.101+0000 7f7c24ff9700 1 -- 172.21.15.113:0/3519221247 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7c18055960 con 0x7f7c280fd8f0 2024-04-12T05:24:26.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.303+0000 7f7c2deab700 1 -- 172.21.15.113:0/3519221247 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c2806c6c0 con 0x7f7c1006caa0 2024-04-12T05:24:26.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.313+0000 7f7c24ff9700 1 -- 172.21.15.113:0/3519221247 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7c2806c6c0 con 0x7f7c1006caa0 2024-04-12T05:24:26.315 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (12m) 5m ago 23m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (9m) 5m ago 23m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (9m) 4m ago 22m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (12m) 5m ago 23m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (5m) 5m ago 18m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (5m) 5m ago 18m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (4m) 4m ago 18m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (18m) 4m ago 18m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (13m) 5m ago 24m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (13m) 4m ago 22m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (10m) 5m ago 24m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (10m) 4m ago 22m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (12m) 5m ago 23m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:24:26.316 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (12m) 4m ago 22m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:24:26.317 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (9m) 5m ago 22m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:24:26.317 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (8m) 5m ago 21m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:24:26.317 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (8m) 5m ago 21m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:24:26.317 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (7m) 4m ago 20m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:24:26.317 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (6m) 4m ago 20m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:24:26.317 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (5m) 4m ago 19m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:24:26.317 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (12m) 4m ago 22m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:24:26.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.315+0000 7f7c2deab700 1 -- 172.21.15.113:0/3519221247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1006caa0 msgr2=0x7f7c1006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:26.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.315+0000 7f7c2deab700 1 --2- 172.21.15.113:0/3519221247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1006caa0 0x7f7c1006ef60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7c1c004f10 tx=0x7f7c1c00b040).stop 2024-04-12T05:24:26.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.315+0000 7f7c2deab700 1 -- 172.21.15.113:0/3519221247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c280fd8f0 msgr2=0x7f7c2806b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:26.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.315+0000 7f7c2deab700 1 --2- 172.21.15.113:0/3519221247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c280fd8f0 0x7f7c2806b0d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f7c180041d0 tx=0x7f7c18009f60).stop 2024-04-12T05:24:26.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.315+0000 7f7c2deab700 1 -- 172.21.15.113:0/3519221247 shutdown_connections 2024-04-12T05:24:26.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.315+0000 7f7c2deab700 1 --2- 172.21.15.113:0/3519221247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c1006caa0 0x7f7c1006ef60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.315+0000 7f7c2deab700 1 --2- 172.21.15.113:0/3519221247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c280fd8f0 0x7f7c2806b0d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.315+0000 7f7c2deab700 1 --2- 172.21.15.113:0/3519221247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c280fe250 0x7f7c2806b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.315+0000 7f7c2deab700 1 -- 172.21.15.113:0/3519221247 >> 172.21.15.113:0/3519221247 conn(0x7f7c280f34b0 msgr2=0x7f7c280fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:26.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.315+0000 7f7c2deab700 1 -- 172.21.15.113:0/3519221247 shutdown_connections 2024-04-12T05:24:26.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.315+0000 7f7c2deab700 1 -- 172.21.15.113:0/3519221247 wait complete. 2024-04-12T05:24:26.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.412+0000 7feddf8f4700 1 -- 172.21.15.113:0/2103654628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedd80fc7f0 msgr2=0x7fedd80fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.412+0000 7feddf8f4700 1 --2- 172.21.15.113:0/2103654628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedd80fc7f0 0x7fedd80fcbd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fedd40047f0 tx=0x7fedd4009cb0).stop 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.412+0000 7feddf8f4700 1 -- 172.21.15.113:0/2103654628 shutdown_connections 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.412+0000 7feddf8f4700 1 --2- 172.21.15.113:0/2103654628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd80fd150 0x7fedd80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.412+0000 7feddf8f4700 1 --2- 172.21.15.113:0/2103654628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedd80fc7f0 0x7fedd80fcbd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.412+0000 7feddf8f4700 1 -- 172.21.15.113:0/2103654628 >> 172.21.15.113:0/2103654628 conn(0x7fedd80f23b0 msgr2=0x7fedd80f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.413+0000 7feddf8f4700 1 -- 172.21.15.113:0/2103654628 shutdown_connections 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.413+0000 7feddf8f4700 1 -- 172.21.15.113:0/2103654628 wait complete. 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.413+0000 7feddf8f4700 1 Processor -- start 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.413+0000 7feddf8f4700 1 -- start start 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.413+0000 7feddf8f4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedd80fc7f0 0x7fedd81090f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.413+0000 7feddf8f4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd80fd150 0x7fedd81095e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.413+0000 7feddf8f4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8109ad0 con 0x7fedd80fc7f0 2024-04-12T05:24:26.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.413+0000 7feddf8f4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8109c10 con 0x7fedd80fd150 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.413+0000 7feddd690700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedd80fc7f0 0x7fedd81090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.413+0000 7feddd690700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedd80fc7f0 0x7fedd81090f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43422/0 (socket says 172.21.15.113:43422) 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.413+0000 7feddd690700 1 -- 172.21.15.113:0/855854399 learned_addr learned my addr 172.21.15.113:0/855854399 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.414+0000 7feddce8f700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd80fd150 0x7fedd81095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.414+0000 7feddd690700 1 -- 172.21.15.113:0/855854399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd80fd150 msgr2=0x7fedd81095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.414+0000 7feddd690700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd80fd150 0x7fedd81095e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.414+0000 7feddd690700 1 -- 172.21.15.113:0/855854399 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd4005040 con 0x7fedd80fc7f0 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.414+0000 7feddce8f700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd80fd150 0x7fedd81095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.414+0000 7feddd690700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedd80fc7f0 0x7fedd81090f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fedd4005010 tx=0x7fedd4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.414+0000 7fedce7fc700 1 -- 172.21.15.113:0/855854399 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedd400e030 con 0x7fedd80fc7f0 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.414+0000 7fedce7fc700 1 -- 172.21.15.113:0/855854399 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedd4007e90 con 0x7fedd80fc7f0 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.414+0000 7feddf8f4700 1 -- 172.21.15.113:0/855854399 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd8109e60 con 0x7fedd80fc7f0 2024-04-12T05:24:26.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.414+0000 7fedce7fc700 1 -- 172.21.15.113:0/855854399 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedd4018070 con 0x7fedd80fc7f0 2024-04-12T05:24:26.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.414+0000 7feddf8f4700 1 -- 172.21.15.113:0/855854399 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd8112450 con 0x7fedd80fc7f0 2024-04-12T05:24:26.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.415+0000 7feddf8f4700 1 -- 172.21.15.113:0/855854399 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd8060390 con 0x7fedd80fc7f0 2024-04-12T05:24:26.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.416+0000 7fedce7fc700 1 -- 172.21.15.113:0/855854399 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fedd401d6e0 con 0x7fedd80fc7f0 2024-04-12T05:24:26.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.416+0000 7fedce7fc700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fedc406ca30 0x7fedc406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:26.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.416+0000 7feddce8f700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fedc406ca30 0x7fedc406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:26.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.416+0000 7fedce7fc700 1 -- 172.21.15.113:0/855854399 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fedd408d120 con 0x7fedd80fc7f0 2024-04-12T05:24:26.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.416+0000 7feddce8f700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fedc406ca30 0x7fedc406eef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fedd8061490 tx=0x7fedc800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:26.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.421+0000 7fedce7fc700 1 -- 172.21.15.113:0/855854399 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fedd405abd0 con 0x7fedd80fc7f0 2024-04-12T05:24:26.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.689+0000 7feddf8f4700 1 -- 172.21.15.113:0/855854399 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fedd80feb80 con 0x7fedd80fc7f0 2024-04-12T05:24:26.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.690+0000 7fedce7fc700 1 -- 172.21.15.113:0/855854399 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fedd405a760 con 0x7fedd80fc7f0 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:24:26.693 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:24:26.694 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:24:26.694 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:24:26.694 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:24:26.694 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:24:26.694 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:24:26.694 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:24:26.694 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:24:26.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.692+0000 7feddf8f4700 1 -- 172.21.15.113:0/855854399 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fedc406ca30 msgr2=0x7fedc406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:26.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.692+0000 7feddf8f4700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fedc406ca30 0x7fedc406eef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fedd8061490 tx=0x7fedc800b040).stop 2024-04-12T05:24:26.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.692+0000 7feddf8f4700 1 -- 172.21.15.113:0/855854399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedd80fc7f0 msgr2=0x7fedd81090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:26.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.692+0000 7feddf8f4700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedd80fc7f0 0x7fedd81090f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fedd4005010 tx=0x7fedd4009cb0).stop 2024-04-12T05:24:26.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.692+0000 7feddf8f4700 1 -- 172.21.15.113:0/855854399 shutdown_connections 2024-04-12T05:24:26.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.692+0000 7feddf8f4700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fedc406ca30 0x7fedc406eef0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.692+0000 7feddf8f4700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fedd80fc7f0 0x7fedd81090f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.692+0000 7feddf8f4700 1 --2- 172.21.15.113:0/855854399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd80fd150 0x7fedd81095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.692+0000 7feddf8f4700 1 -- 172.21.15.113:0/855854399 >> 172.21.15.113:0/855854399 conn(0x7fedd80f23b0 msgr2=0x7fedd80fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:26.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.692+0000 7feddf8f4700 1 -- 172.21.15.113:0/855854399 shutdown_connections 2024-04-12T05:24:26.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.692+0000 7feddf8f4700 1 -- 172.21.15.113:0/855854399 wait complete. 2024-04-12T05:24:26.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.790+0000 7ffb42051700 1 -- 172.21.15.113:0/531312261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0fe360 msgr2=0x7ffb3c0f5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:26.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.790+0000 7ffb42051700 1 --2- 172.21.15.113:0/531312261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0fe360 0x7ffb3c0f5fc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7ffb240047f0 tx=0x7ffb24009cb0).stop 2024-04-12T05:24:26.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.790+0000 7ffb42051700 1 -- 172.21.15.113:0/531312261 shutdown_connections 2024-04-12T05:24:26.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.790+0000 7ffb42051700 1 --2- 172.21.15.113:0/531312261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0fe360 0x7ffb3c0f5fc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.790+0000 7ffb42051700 1 --2- 172.21.15.113:0/531312261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c0fda00 0x7ffb3c0fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:24:26.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.790+0000 7ffb42051700 1 -- 172.21.15.113:0/531312261 >> 172.21.15.113:0/531312261 conn(0x7ffb3c06cea0 msgr2=0x7ffb3c06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:26.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.790+0000 7ffb42051700 1 -- 172.21.15.113:0/531312261 shutdown_connections 2024-04-12T05:24:26.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.790+0000 7ffb42051700 1 -- 172.21.15.113:0/531312261 wait complete. 2024-04-12T05:24:26.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb42051700 1 Processor -- start 2024-04-12T05:24:26.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb42051700 1 -- start start 2024-04-12T05:24:26.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb42051700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c0fda00 0x7ffb3c10e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:26.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb42051700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0fe360 0x7ffb3c10e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:26.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb42051700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c10eb80 con 0x7ffb3c0fe360 2024-04-12T05:24:26.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb42051700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c1081a0 con 0x7ffb3c0fda00 2024-04-12T05:24:26.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb3affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0fe360 0x7ffb3c10e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:26.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb3affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0fe360 0x7ffb3c10e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43448/0 (socket says 172.21.15.113:43448) 2024-04-12T05:24:26.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb3affd700 1 -- 172.21.15.113:0/1246833475 learned_addr learned my addr 172.21.15.113:0/1246833475 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:26.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb3b7fe700 1 --2- 172.21.15.113:0/1246833475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c0fda00 0x7ffb3c10e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:26.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb3affd700 1 -- 172.21.15.113:0/1246833475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c0fda00 msgr2=0x7ffb3c10e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:26.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb3affd700 1 --2- 172.21.15.113:0/1246833475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c0fda00 0x7ffb3c10e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:26.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.791+0000 7ffb3affd700 1 -- 172.21.15.113:0/1246833475 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb24005040 con 0x7ffb3c0fe360 2024-04-12T05:24:26.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.792+0000 7ffb3affd700 1 --2- 172.21.15.113:0/1246833475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0fe360 0x7ffb3c10e4f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7ffb2400b120 tx=0x7ffb24009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:26.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.792+0000 7ffb38ff9700 1 -- 172.21.15.113:0/1246833475 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb2400e030 con 0x7ffb3c0fe360 2024-04-12T05:24:26.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.792+0000 7ffb42051700 1 -- 172.21.15.113:0/1246833475 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3c1083f0 con 0x7ffb3c0fe360 2024-04-12T05:24:26.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.792+0000 7ffb38ff9700 1 -- 172.21.15.113:0/1246833475 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb24007e90 con 0x7ffb3c0fe360 2024-04-12T05:24:26.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.792+0000 7ffb38ff9700 1 -- 172.21.15.113:0/1246833475 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb24018070 con 0x7ffb3c0fe360 2024-04-12T05:24:26.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.792+0000 7ffb42051700 1 -- 172.21.15.113:0/1246833475 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3c1088d0 con 0x7ffb3c0fe360 2024-04-12T05:24:26.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.793+0000 7ffb42051700 1 -- 172.21.15.113:0/1246833475 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3c047540 con 0x7ffb3c0fe360 2024-04-12T05:24:26.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.793+0000 7ffb38ff9700 1 -- 172.21.15.113:0/1246833475 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ffb2401d420 con 0x7ffb3c0fe360 2024-04-12T05:24:26.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.794+0000 7ffb38ff9700 1 --2- 172.21.15.113:0/1246833475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb2806caa0 0x7ffb2806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:26.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.794+0000 7ffb38ff9700 1 -- 172.21.15.113:0/1246833475 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ffb2408d300 con 0x7ffb3c0fe360 2024-04-12T05:24:26.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.794+0000 7ffb3b7fe700 1 --2- 172.21.15.113:0/1246833475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb2806caa0 0x7ffb2806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:26.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.794+0000 7ffb3b7fe700 1 --2- 172.21.15.113:0/1246833475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb2806caa0 0x7ffb2806ef60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7ffb2c00b910 tx=0x7ffb2c008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:26.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:26.798+0000 7ffb38ff9700 1 -- 172.21.15.113:0/1246833475 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ffb2405adb0 con 0x7ffb3c0fe360 2024-04-12T05:24:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.031+0000 7ffb42051700 1 -- 172.21.15.113:0/1246833475 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ffb3c109680 con 0x7ffb3c0fe360 2024-04-12T05:24:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.031+0000 7ffb38ff9700 1 -- 172.21.15.113:0/1246833475 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7ffb2405a940 con 0x7ffb3c0fe360 2024-04-12T05:24:27.034 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:24:27.034 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:24:27.034 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:24:27.035 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:24:27.035 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:24:27.035 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:24:27.035 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:24:27.035 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:24:27.035 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:24:27.035 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:24:27.035 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:24:27.035 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:24:27.036 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:24:27.037 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:24:27.038 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:24:27.038 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:24:27.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.034+0000 7ffb42051700 1 -- 172.21.15.113:0/1246833475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb2806caa0 msgr2=0x7ffb2806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:27.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.034+0000 7ffb42051700 1 --2- 172.21.15.113:0/1246833475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb2806caa0 0x7ffb2806ef60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7ffb2c00b910 tx=0x7ffb2c008040).stop 2024-04-12T05:24:27.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.034+0000 7ffb42051700 1 -- 172.21.15.113:0/1246833475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0fe360 msgr2=0x7ffb3c10e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:27.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.034+0000 7ffb42051700 1 --2- 172.21.15.113:0/1246833475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0fe360 0x7ffb3c10e4f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7ffb2400b120 tx=0x7ffb24009cb0).stop 2024-04-12T05:24:27.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.034+0000 7ffb42051700 1 -- 172.21.15.113:0/1246833475 shutdown_connections 2024-04-12T05:24:27.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.034+0000 7ffb42051700 1 --2- 172.21.15.113:0/1246833475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb2806caa0 0x7ffb2806ef60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.034+0000 7ffb42051700 1 --2- 172.21.15.113:0/1246833475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c0fda00 0x7ffb3c10e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.034+0000 7ffb42051700 1 --2- 172.21.15.113:0/1246833475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb3c0fe360 0x7ffb3c10e4f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.034+0000 7ffb42051700 1 -- 172.21.15.113:0/1246833475 >> 172.21.15.113:0/1246833475 conn(0x7ffb3c06cea0 msgr2=0x7ffb3c0fbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:27.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.034+0000 7ffb42051700 1 -- 172.21.15.113:0/1246833475 shutdown_connections 2024-04-12T05:24:27.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.034+0000 7ffb42051700 1 -- 172.21.15.113:0/1246833475 wait complete. 2024-04-12T05:24:27.039 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:24:27.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.134+0000 7f26886dd700 1 -- 172.21.15.113:0/3307664962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26800fc7f0 msgr2=0x7f26800fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.134+0000 7f26886dd700 1 --2- 172.21.15.113:0/3307664962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26800fc7f0 0x7f26800fcbd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f267c0047f0 tx=0x7f267c009cb0).stop 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.134+0000 7f26886dd700 1 -- 172.21.15.113:0/3307664962 shutdown_connections 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.134+0000 7f26886dd700 1 --2- 172.21.15.113:0/3307664962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26800fd150 0x7f26800f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.134+0000 7f26886dd700 1 --2- 172.21.15.113:0/3307664962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26800fc7f0 0x7f26800fcbd0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.134+0000 7f26886dd700 1 -- 172.21.15.113:0/3307664962 >> 172.21.15.113:0/3307664962 conn(0x7f2680070590 msgr2=0x7f26800f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.134+0000 7f26886dd700 1 -- 172.21.15.113:0/3307664962 shutdown_connections 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.134+0000 7f26886dd700 1 -- 172.21.15.113:0/3307664962 wait complete. 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.134+0000 7f26886dd700 1 Processor -- start 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f26886dd700 1 -- start start 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f26886dd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26800fc7f0 0x7f268010e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f26886dd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26800fd150 0x7f268010e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:27.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f26886dd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268010ebe0 con 0x7f26800fd150 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f26886dd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2680108200 con 0x7f26800fc7f0 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f2685c78700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26800fd150 0x7f268010e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f2685c78700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26800fd150 0x7f268010e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43466/0 (socket says 172.21.15.113:43466) 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f2685c78700 1 -- 172.21.15.113:0/4275967454 learned_addr learned my addr 172.21.15.113:0/4275967454 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f2686479700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26800fc7f0 0x7f268010e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f2685c78700 1 -- 172.21.15.113:0/4275967454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26800fc7f0 msgr2=0x7f268010e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f2685c78700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26800fc7f0 0x7f268010e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f2685c78700 1 -- 172.21.15.113:0/4275967454 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267c005040 con 0x7f26800fd150 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f2686479700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26800fc7f0 0x7f268010e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f2685c78700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26800fd150 0x7f268010e550 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f267000b480 tx=0x7f26700090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f26777fe700 1 -- 172.21.15.113:0/4275967454 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2670002500 con 0x7f26800fd150 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.136+0000 7f26886dd700 1 -- 172.21.15.113:0/4275967454 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26801084b0 con 0x7f26800fd150 2024-04-12T05:24:27.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.135+0000 7f26777fe700 1 -- 172.21.15.113:0/4275967454 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2670002b10 con 0x7f26800fd150 2024-04-12T05:24:27.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.136+0000 7f26777fe700 1 -- 172.21.15.113:0/4275967454 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2670012c90 con 0x7f26800fd150 2024-04-12T05:24:27.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.136+0000 7f26886dd700 1 -- 172.21.15.113:0/4275967454 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26801089f0 con 0x7f26800fd150 2024-04-12T05:24:27.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.136+0000 7f26886dd700 1 -- 172.21.15.113:0/4275967454 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268005f590 con 0x7f26800fd150 2024-04-12T05:24:27.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.137+0000 7f26777fe700 1 -- 172.21.15.113:0/4275967454 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2670002660 con 0x7f26800fd150 2024-04-12T05:24:27.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.137+0000 7f26777fe700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f266c06caa0 0x7f266c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:27.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.137+0000 7f26777fe700 1 -- 172.21.15.113:0/4275967454 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f267008bef0 con 0x7f26800fd150 2024-04-12T05:24:27.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.137+0000 7f2686479700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f266c06caa0 0x7f266c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:27.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.138+0000 7f2686479700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f266c06caa0 0x7f266c06ef60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f267c007cf0 tx=0x7f267c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:27.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.142+0000 7f26777fe700 1 -- 172.21.15.113:0/4275967454 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f26700599a0 con 0x7f26800fd150 2024-04-12T05:24:27.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.348+0000 7f26886dd700 1 -- 172.21.15.113:0/4275967454 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f26801036b0 con 0x7f266c06caa0 2024-04-12T05:24:27.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.349+0000 7f26777fe700 1 -- 172.21.15.113:0/4275967454 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f267c019040 con 0x7f266c06caa0 2024-04-12T05:24:27.352 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:24:27.352 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:24:27.352 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:24:27.352 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:24:27.352 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:24:27.352 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:24:27.352 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:24:27.352 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:24:27.352 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:24:27.353 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:24:27.353 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:24:27.353 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:24:27.353 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:24:27.353 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:24:27.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.352+0000 7f26886dd700 1 -- 172.21.15.113:0/4275967454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f266c06caa0 msgr2=0x7f266c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:27.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.352+0000 7f26886dd700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f266c06caa0 0x7f266c06ef60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f267c007cf0 tx=0x7f267c00b040).stop 2024-04-12T05:24:27.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.352+0000 7f26886dd700 1 -- 172.21.15.113:0/4275967454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26800fd150 msgr2=0x7f268010e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:27.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.352+0000 7f26886dd700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26800fd150 0x7f268010e550 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f267000b480 tx=0x7f26700090f0).stop 2024-04-12T05:24:27.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.352+0000 7f26886dd700 1 -- 172.21.15.113:0/4275967454 shutdown_connections 2024-04-12T05:24:27.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.352+0000 7f26886dd700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f266c06caa0 0x7f266c06ef60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.352+0000 7f26886dd700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26800fc7f0 0x7f268010e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.352+0000 7f26886dd700 1 --2- 172.21.15.113:0/4275967454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26800fd150 0x7f268010e550 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.352+0000 7f26886dd700 1 -- 172.21.15.113:0/4275967454 >> 172.21.15.113:0/4275967454 conn(0x7f2680070590 msgr2=0x7f2680060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:27.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.352+0000 7f26886dd700 1 -- 172.21.15.113:0/4275967454 shutdown_connections 2024-04-12T05:24:27.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.352+0000 7f26886dd700 1 -- 172.21.15.113:0/4275967454 wait complete. 2024-04-12T05:24:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:27 smithi113 ceph-mon[103371]: from='client.34700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:24:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:27 smithi113 ceph-mon[103371]: pgmap v368: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:27 smithi113 ceph-mon[103371]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:24:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:27 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/855854399' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:24:27.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:27 smithi192 ceph-mon[85304]: from='client.34700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:24:27.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:27 smithi192 ceph-mon[85304]: pgmap v368: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:27.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:27 smithi192 ceph-mon[85304]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:24:27.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:27 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/855854399' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:24:27.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.450+0000 7fdb96202700 1 -- 172.21.15.113:0/1981101803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb900fd150 msgr2=0x7fdb900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:27.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.450+0000 7fdb96202700 1 --2- 172.21.15.113:0/1981101803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb900fd150 0x7fdb900f6930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fdb780047f0 tx=0x7fdb78009cb0).stop 2024-04-12T05:24:27.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.450+0000 7fdb96202700 1 -- 172.21.15.113:0/1981101803 shutdown_connections 2024-04-12T05:24:27.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.450+0000 7fdb96202700 1 --2- 172.21.15.113:0/1981101803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb900fd150 0x7fdb900f6930 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.450+0000 7fdb96202700 1 --2- 172.21.15.113:0/1981101803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb900fc7f0 0x7fdb900fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.450+0000 7fdb96202700 1 -- 172.21.15.113:0/1981101803 >> 172.21.15.113:0/1981101803 conn(0x7fdb90070590 msgr2=0x7fdb900f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:27.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.450+0000 7fdb96202700 1 -- 172.21.15.113:0/1981101803 shutdown_connections 2024-04-12T05:24:27.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.450+0000 7fdb96202700 1 -- 172.21.15.113:0/1981101803 wait complete. 2024-04-12T05:24:27.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.451+0000 7fdb96202700 1 Processor -- start 2024-04-12T05:24:27.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.451+0000 7fdb96202700 1 -- start start 2024-04-12T05:24:27.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.451+0000 7fdb96202700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb900fc7f0 0x7fdb9010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:27.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.451+0000 7fdb96202700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb900fd150 0x7fdb9010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:27.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.451+0000 7fdb96202700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9010ead0 con 0x7fdb900fd150 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.451+0000 7fdb96202700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9010ec10 con 0x7fdb900fc7f0 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.451+0000 7fdb87fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb900fd150 0x7fdb9010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.451+0000 7fdb87fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb900fd150 0x7fdb9010e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43492/0 (socket says 172.21.15.113:43492) 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.451+0000 7fdb87fff700 1 -- 172.21.15.113:0/3534254714 learned_addr learned my addr 172.21.15.113:0/3534254714 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.451+0000 7fdb8f7fe700 1 --2- 172.21.15.113:0/3534254714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb900fc7f0 0x7fdb9010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.452+0000 7fdb87fff700 1 -- 172.21.15.113:0/3534254714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb900fc7f0 msgr2=0x7fdb9010dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.452+0000 7fdb87fff700 1 --2- 172.21.15.113:0/3534254714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb900fc7f0 0x7fdb9010dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.452+0000 7fdb87fff700 1 -- 172.21.15.113:0/3534254714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb78005040 con 0x7fdb900fd150 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.452+0000 7fdb87fff700 1 --2- 172.21.15.113:0/3534254714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb900fd150 0x7fdb9010e4d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fdb780099f0 tx=0x7fdb78009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.452+0000 7fdb8d7fa700 1 -- 172.21.15.113:0/3534254714 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb7800e030 con 0x7fdb900fd150 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.452+0000 7fdb8d7fa700 1 -- 172.21.15.113:0/3534254714 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb78007e90 con 0x7fdb900fd150 2024-04-12T05:24:27.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.452+0000 7fdb96202700 1 -- 172.21.15.113:0/3534254714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb90108240 con 0x7fdb900fd150 2024-04-12T05:24:27.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.452+0000 7fdb8d7fa700 1 -- 172.21.15.113:0/3534254714 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb78018070 con 0x7fdb900fd150 2024-04-12T05:24:27.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.452+0000 7fdb96202700 1 -- 172.21.15.113:0/3534254714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb901087a0 con 0x7fdb900fd150 2024-04-12T05:24:27.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.453+0000 7fdb96202700 1 -- 172.21.15.113:0/3534254714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9005f590 con 0x7fdb900fd150 2024-04-12T05:24:27.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.453+0000 7fdb8d7fa700 1 -- 172.21.15.113:0/3534254714 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdb780094c0 con 0x7fdb900fd150 2024-04-12T05:24:27.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.454+0000 7fdb8d7fa700 1 --2- 172.21.15.113:0/3534254714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdb7c06caa0 0x7fdb7c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:27.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.454+0000 7fdb8d7fa700 1 -- 172.21.15.113:0/3534254714 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fdb7808d010 con 0x7fdb900fd150 2024-04-12T05:24:27.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.454+0000 7fdb8f7fe700 1 --2- 172.21.15.113:0/3534254714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdb7c06caa0 0x7fdb7c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:27.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.454+0000 7fdb8f7fe700 1 --2- 172.21.15.113:0/3534254714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdb7c06caa0 0x7fdb7c06ef60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fdb80004de0 tx=0x7fdb8000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:27.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.458+0000 7fdb8d7fa700 1 -- 172.21.15.113:0/3534254714 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fdb7805aac0 con 0x7fdb900fd150 2024-04-12T05:24:27.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.722+0000 7fdb96202700 1 -- 172.21.15.113:0/3534254714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdb90047540 con 0x7fdb900fd150 2024-04-12T05:24:27.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.722+0000 7fdb8d7fa700 1 -- 172.21.15.113:0/3534254714 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fdb7805a650 con 0x7fdb900fd150 2024-04-12T05:24:27.725 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:24:27.725 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:24:27.725 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:24:27.725 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:24:27.725 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:24:27.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.724+0000 7fdb96202700 1 -- 172.21.15.113:0/3534254714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdb7c06caa0 msgr2=0x7fdb7c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:27.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.724+0000 7fdb96202700 1 --2- 172.21.15.113:0/3534254714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdb7c06caa0 0x7fdb7c06ef60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fdb80004de0 tx=0x7fdb8000b040).stop 2024-04-12T05:24:27.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.724+0000 7fdb96202700 1 -- 172.21.15.113:0/3534254714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb900fd150 msgr2=0x7fdb9010e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:27.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.724+0000 7fdb96202700 1 --2- 172.21.15.113:0/3534254714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb900fd150 0x7fdb9010e4d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fdb780099f0 tx=0x7fdb78009cb0).stop 2024-04-12T05:24:27.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.724+0000 7fdb96202700 1 -- 172.21.15.113:0/3534254714 shutdown_connections 2024-04-12T05:24:27.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.724+0000 7fdb96202700 1 --2- 172.21.15.113:0/3534254714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdb7c06caa0 0x7fdb7c06ef60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.724+0000 7fdb96202700 1 --2- 172.21.15.113:0/3534254714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb900fc7f0 0x7fdb9010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.724+0000 7fdb96202700 1 --2- 172.21.15.113:0/3534254714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb900fd150 0x7fdb9010e4d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:27.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.724+0000 7fdb96202700 1 -- 172.21.15.113:0/3534254714 >> 172.21.15.113:0/3534254714 conn(0x7fdb90070590 msgr2=0x7fdb90060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:27.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.724+0000 7fdb96202700 1 -- 172.21.15.113:0/3534254714 shutdown_connections 2024-04-12T05:24:27.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:27.725+0000 7fdb96202700 1 -- 172.21.15.113:0/3534254714 wait complete. 2024-04-12T05:24:28.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:28 smithi113 ceph-mon[103371]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:24:28.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:28 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1246833475' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:24:28.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:28 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3534254714' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:24:28.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:28 smithi192 ceph-mon[85304]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:24:28.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:28 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1246833475' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:24:28.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:28 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3534254714' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:24:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:29 smithi113 ceph-mon[103371]: from='client.34720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:24:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:29 smithi113 ceph-mon[103371]: pgmap v369: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:29 smithi192 ceph-mon[85304]: from='client.34720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:24:29.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:29 smithi192 ceph-mon[85304]: pgmap v369: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:31 smithi113 ceph-mon[103371]: pgmap v370: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:31.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:31 smithi192 ceph-mon[85304]: pgmap v370: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:33 smithi113 ceph-mon[103371]: pgmap v371: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:33.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:33 smithi192 ceph-mon[85304]: pgmap v371: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:35 smithi113 ceph-mon[103371]: pgmap v372: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:35.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:35 smithi192 ceph-mon[85304]: pgmap v372: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:37 smithi113 ceph-mon[103371]: pgmap v373: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:24:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:37 smithi192 ceph-mon[85304]: pgmap v373: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:24:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:39 smithi113 ceph-mon[103371]: pgmap v374: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:39 smithi192 ceph-mon[85304]: pgmap v374: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:41 smithi113 ceph-mon[103371]: pgmap v375: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:41 smithi192 ceph-mon[85304]: pgmap v375: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:43 smithi113 ceph-mon[103371]: pgmap v376: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:43.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:43 smithi192 ceph-mon[85304]: pgmap v376: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:45 smithi113 ceph-mon[103371]: pgmap v377: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:45 smithi192 ceph-mon[85304]: pgmap v377: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:47 smithi113 ceph-mon[103371]: pgmap v378: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:47 smithi192 ceph-mon[85304]: pgmap v378: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:49 smithi113 ceph-mon[103371]: pgmap v379: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:49.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:49 smithi192 ceph-mon[85304]: pgmap v379: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:51 smithi113 ceph-mon[103371]: pgmap v380: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:51 smithi192 ceph-mon[85304]: pgmap v380: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:24:52.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:24:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:53 smithi113 ceph-mon[103371]: pgmap v381: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:53 smithi192 ceph-mon[85304]: pgmap v381: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:55 smithi113 ceph-mon[103371]: pgmap v382: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:55.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:55 smithi192 ceph-mon[85304]: pgmap v382: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:57 smithi113 ceph-mon[103371]: pgmap v383: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:57.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:57 smithi192 ceph-mon[85304]: pgmap v383: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:57.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.835+0000 7fc9a7cee700 1 -- 172.21.15.113:0/2918260599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9a00fd150 msgr2=0x7fc9a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:57.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.835+0000 7fc9a7cee700 1 --2- 172.21.15.113:0/2918260599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9a00fd150 0x7fc9a00f6930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fc99c0047f0 tx=0x7fc99c009cb0).stop 2024-04-12T05:24:57.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.835+0000 7fc9a7cee700 1 -- 172.21.15.113:0/2918260599 shutdown_connections 2024-04-12T05:24:57.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.835+0000 7fc9a7cee700 1 --2- 172.21.15.113:0/2918260599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9a00fd150 0x7fc9a00f6930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:57.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.836+0000 7fc9a7cee700 1 --2- 172.21.15.113:0/2918260599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9a00fc7f0 0x7fc9a00fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:57.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.836+0000 7fc9a7cee700 1 -- 172.21.15.113:0/2918260599 >> 172.21.15.113:0/2918260599 conn(0x7fc9a0070590 msgr2=0x7fc9a00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:57.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.836+0000 7fc9a7cee700 1 -- 172.21.15.113:0/2918260599 shutdown_connections 2024-04-12T05:24:57.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.836+0000 7fc9a7cee700 1 -- 172.21.15.113:0/2918260599 wait complete. 2024-04-12T05:24:57.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.836+0000 7fc9a7cee700 1 Processor -- start 2024-04-12T05:24:57.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.836+0000 7fc9a7cee700 1 -- start start 2024-04-12T05:24:57.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.836+0000 7fc9a7cee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9a00fc7f0 0x7fc9a010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:57.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.836+0000 7fc9a7cee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9a00fd150 0x7fc9a010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:57.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.836+0000 7fc9a7cee700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a010ead0 con 0x7fc9a00fd150 2024-04-12T05:24:57.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.836+0000 7fc9a7cee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a010ec10 con 0x7fc9a00fc7f0 2024-04-12T05:24:57.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc9a5289700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9a00fd150 0x7fc9a010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc9a5289700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9a00fd150 0x7fc9a010e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55700/0 (socket says 172.21.15.113:55700) 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc9a5289700 1 -- 172.21.15.113:0/1910508171 learned_addr learned my addr 172.21.15.113:0/1910508171 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc9a5a8a700 1 --2- 172.21.15.113:0/1910508171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9a00fc7f0 0x7fc9a010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc9a5289700 1 -- 172.21.15.113:0/1910508171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9a00fc7f0 msgr2=0x7fc9a010dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc9a5289700 1 --2- 172.21.15.113:0/1910508171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9a00fc7f0 0x7fc9a010dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc9a5289700 1 -- 172.21.15.113:0/1910508171 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c005040 con 0x7fc9a00fd150 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc9a5289700 1 --2- 172.21.15.113:0/1910508171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9a00fd150 0x7fc9a010e4d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fc99c0099f0 tx=0x7fc99c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc992ffd700 1 -- 172.21.15.113:0/1910508171 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc99c00e030 con 0x7fc9a00fd150 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc9a7cee700 1 -- 172.21.15.113:0/1910508171 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a0108240 con 0x7fc9a00fd150 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc992ffd700 1 -- 172.21.15.113:0/1910508171 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc99c007e90 con 0x7fc9a00fd150 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc992ffd700 1 -- 172.21.15.113:0/1910508171 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc99c018070 con 0x7fc9a00fd150 2024-04-12T05:24:57.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.837+0000 7fc9a7cee700 1 -- 172.21.15.113:0/1910508171 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a01087a0 con 0x7fc9a00fd150 2024-04-12T05:24:57.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.838+0000 7fc9a7cee700 1 -- 172.21.15.113:0/1910508171 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a005f590 con 0x7fc9a00fd150 2024-04-12T05:24:57.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.839+0000 7fc992ffd700 1 -- 172.21.15.113:0/1910508171 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc99c0094c0 con 0x7fc9a00fd150 2024-04-12T05:24:57.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.839+0000 7fc992ffd700 1 --2- 172.21.15.113:0/1910508171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc98c06caa0 0x7fc98c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:57.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.839+0000 7fc992ffd700 1 -- 172.21.15.113:0/1910508171 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc99c08d100 con 0x7fc9a00fd150 2024-04-12T05:24:57.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.839+0000 7fc9a5a8a700 1 --2- 172.21.15.113:0/1910508171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc98c06caa0 0x7fc98c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:57.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.840+0000 7fc9a5a8a700 1 --2- 172.21.15.113:0/1910508171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc98c06caa0 0x7fc98c06ef60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fc994004de0 tx=0x7fc99400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:57.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:57.844+0000 7fc992ffd700 1 -- 172.21.15.113:0/1910508171 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc99c05abb0 con 0x7fc9a00fd150 2024-04-12T05:24:58.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.050+0000 7fc9a7cee700 1 -- 172.21.15.113:0/1910508171 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9a0109700 con 0x7fc98c06caa0 2024-04-12T05:24:58.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.051+0000 7fc992ffd700 1 -- 172.21.15.113:0/1910508171 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc9a0109700 con 0x7fc98c06caa0 2024-04-12T05:24:58.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.053+0000 7fc9a7cee700 1 -- 172.21.15.113:0/1910508171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc98c06caa0 msgr2=0x7fc98c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.053+0000 7fc9a7cee700 1 --2- 172.21.15.113:0/1910508171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc98c06caa0 0x7fc98c06ef60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fc994004de0 tx=0x7fc99400b040).stop 2024-04-12T05:24:58.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.053+0000 7fc9a7cee700 1 -- 172.21.15.113:0/1910508171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9a00fd150 msgr2=0x7fc9a010e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.053+0000 7fc9a7cee700 1 --2- 172.21.15.113:0/1910508171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9a00fd150 0x7fc9a010e4d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fc99c0099f0 tx=0x7fc99c009cb0).stop 2024-04-12T05:24:58.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.053+0000 7fc9a7cee700 1 -- 172.21.15.113:0/1910508171 shutdown_connections 2024-04-12T05:24:58.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.053+0000 7fc9a7cee700 1 --2- 172.21.15.113:0/1910508171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc98c06caa0 0x7fc98c06ef60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.053+0000 7fc9a7cee700 1 --2- 172.21.15.113:0/1910508171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9a00fc7f0 0x7fc9a010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.053+0000 7fc9a7cee700 1 --2- 172.21.15.113:0/1910508171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9a00fd150 0x7fc9a010e4d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.053+0000 7fc9a7cee700 1 -- 172.21.15.113:0/1910508171 >> 172.21.15.113:0/1910508171 conn(0x7fc9a0070590 msgr2=0x7fc9a0060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:58.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.053+0000 7fc9a7cee700 1 -- 172.21.15.113:0/1910508171 shutdown_connections 2024-04-12T05:24:58.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.053+0000 7fc9a7cee700 1 -- 172.21.15.113:0/1910508171 wait complete. 2024-04-12T05:24:58.066 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:24:58.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.153+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/4075410750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e540fd150 msgr2=0x7f5e540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.153+0000 7f5e5c3ec700 1 --2- 172.21.15.113:0/4075410750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e540fd150 0x7f5e540f6930 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f5e440047f0 tx=0x7f5e44009cb0).stop 2024-04-12T05:24:58.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.153+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/4075410750 shutdown_connections 2024-04-12T05:24:58.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.153+0000 7f5e5c3ec700 1 --2- 172.21.15.113:0/4075410750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e540fd150 0x7f5e540f6930 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.153+0000 7f5e5c3ec700 1 --2- 172.21.15.113:0/4075410750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e540fc7f0 0x7f5e540fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:24:58.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.153+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/4075410750 >> 172.21.15.113:0/4075410750 conn(0x7f5e54070590 msgr2=0x7f5e540f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.153+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/4075410750 shutdown_connections 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.154+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/4075410750 wait complete. 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.154+0000 7f5e5c3ec700 1 Processor -- start 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.154+0000 7f5e5c3ec700 1 -- start start 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.154+0000 7f5e5c3ec700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e540fc7f0 0x7f5e54109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.154+0000 7f5e5c3ec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e540fd150 0x7f5e54109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.154+0000 7f5e5c3ec700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e54109be0 con 0x7f5e540fc7f0 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.154+0000 7f5e5c3ec700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e54109d20 con 0x7f5e540fd150 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.154+0000 7f5e5a188700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e540fc7f0 0x7f5e54109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.154+0000 7f5e5a188700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e540fc7f0 0x7f5e54109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55718/0 (socket says 172.21.15.113:55718) 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.154+0000 7f5e5a188700 1 -- 172.21.15.113:0/1163108836 learned_addr learned my addr 172.21.15.113:0/1163108836 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.155+0000 7f5e59987700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e540fd150 0x7f5e54109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:58.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.155+0000 7f5e5a188700 1 -- 172.21.15.113:0/1163108836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e540fd150 msgr2=0x7f5e54109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.155+0000 7f5e5a188700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e540fd150 0x7f5e54109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.155+0000 7f5e5a188700 1 -- 172.21.15.113:0/1163108836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e44005040 con 0x7f5e540fc7f0 2024-04-12T05:24:58.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.155+0000 7f5e59987700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e540fd150 0x7f5e54109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:24:58.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.155+0000 7f5e5a188700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e540fc7f0 0x7f5e54109170 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f5e5000b4b0 tx=0x7f5e500090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:58.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.155+0000 7f5e4b7fe700 1 -- 172.21.15.113:0/1163108836 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e50002500 con 0x7f5e540fc7f0 2024-04-12T05:24:58.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.155+0000 7f5e4b7fe700 1 -- 172.21.15.113:0/1163108836 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e50002b10 con 0x7f5e540fc7f0 2024-04-12T05:24:58.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.155+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/1163108836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e54109fd0 con 0x7f5e540fc7f0 2024-04-12T05:24:58.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.155+0000 7f5e4b7fe700 1 -- 172.21.15.113:0/1163108836 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e50012cf0 con 0x7f5e540fc7f0 2024-04-12T05:24:58.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.155+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/1163108836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e54112560 con 0x7f5e540fc7f0 2024-04-12T05:24:58.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.156+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/1163108836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5405f590 con 0x7f5e540fc7f0 2024-04-12T05:24:58.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.156+0000 7f5e4b7fe700 1 -- 172.21.15.113:0/1163108836 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5e50002660 con 0x7f5e540fc7f0 2024-04-12T05:24:58.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.157+0000 7f5e4b7fe700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e4006caa0 0x7f5e4006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:58.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.157+0000 7f5e4b7fe700 1 -- 172.21.15.113:0/1163108836 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5e5008bf30 con 0x7f5e540fc7f0 2024-04-12T05:24:58.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.157+0000 7f5e59987700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e4006caa0 0x7f5e4006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:58.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.157+0000 7f5e59987700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e4006caa0 0x7f5e4006ef60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f5e440043d0 tx=0x7f5e4400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:58.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.161+0000 7f5e4b7fe700 1 -- 172.21.15.113:0/1163108836 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5e500bf2c0 con 0x7f5e540fc7f0 2024-04-12T05:24:58.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.366+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/1163108836 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e54059120 con 0x7f5e4006caa0 2024-04-12T05:24:58.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.367+0000 7f5e4b7fe700 1 -- 172.21.15.113:0/1163108836 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5e54059120 con 0x7f5e4006caa0 2024-04-12T05:24:58.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.369+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/1163108836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e4006caa0 msgr2=0x7f5e4006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.369+0000 7f5e5c3ec700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e4006caa0 0x7f5e4006ef60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f5e440043d0 tx=0x7f5e4400b040).stop 2024-04-12T05:24:58.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.369+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/1163108836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e540fc7f0 msgr2=0x7f5e54109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.369+0000 7f5e5c3ec700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e540fc7f0 0x7f5e54109170 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f5e5000b4b0 tx=0x7f5e500090f0).stop 2024-04-12T05:24:58.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.369+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/1163108836 shutdown_connections 2024-04-12T05:24:58.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.369+0000 7f5e5c3ec700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5e4006caa0 0x7f5e4006ef60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.369+0000 7f5e5c3ec700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e540fc7f0 0x7f5e54109170 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.369+0000 7f5e5c3ec700 1 --2- 172.21.15.113:0/1163108836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e540fd150 0x7f5e54109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.369+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/1163108836 >> 172.21.15.113:0/1163108836 conn(0x7f5e54070590 msgr2=0x7f5e540fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:58.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.369+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/1163108836 shutdown_connections 2024-04-12T05:24:58.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.369+0000 7f5e5c3ec700 1 -- 172.21.15.113:0/1163108836 wait complete. 2024-04-12T05:24:58.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.467+0000 7effcbf01700 1 -- 172.21.15.113:0/2199180850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc40fc7f0 msgr2=0x7effc40fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.467+0000 7effcbf01700 1 --2- 172.21.15.113:0/2199180850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc40fc7f0 0x7effc40fcbd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7effb40047f0 tx=0x7effb4009cb0).stop 2024-04-12T05:24:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.467+0000 7effcbf01700 1 -- 172.21.15.113:0/2199180850 shutdown_connections 2024-04-12T05:24:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.467+0000 7effcbf01700 1 --2- 172.21.15.113:0/2199180850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc40fd150 0x7effc40f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.467+0000 7effcbf01700 1 --2- 172.21.15.113:0/2199180850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc40fc7f0 0x7effc40fcbd0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.467+0000 7effcbf01700 1 -- 172.21.15.113:0/2199180850 >> 172.21.15.113:0/2199180850 conn(0x7effc4070590 msgr2=0x7effc40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.467+0000 7effcbf01700 1 -- 172.21.15.113:0/2199180850 shutdown_connections 2024-04-12T05:24:58.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.467+0000 7effcbf01700 1 -- 172.21.15.113:0/2199180850 wait complete. 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effcbf01700 1 Processor -- start 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effcbf01700 1 -- start start 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effcbf01700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc40fc7f0 0x7effc4109120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effcbf01700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc40fd150 0x7effc4109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effcbf01700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc4109b00 con 0x7effc40fd150 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effcbf01700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc4109c40 con 0x7effc40fc7f0 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effc949c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc40fd150 0x7effc4109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effc949c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc40fd150 0x7effc4109610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55748/0 (socket says 172.21.15.113:55748) 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effc949c700 1 -- 172.21.15.113:0/633240202 learned_addr learned my addr 172.21.15.113:0/633240202 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effc9c9d700 1 --2- 172.21.15.113:0/633240202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc40fc7f0 0x7effc4109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effc949c700 1 -- 172.21.15.113:0/633240202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc40fc7f0 msgr2=0x7effc4109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effc949c700 1 --2- 172.21.15.113:0/633240202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc40fc7f0 0x7effc4109120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effc949c700 1 -- 172.21.15.113:0/633240202 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb4005040 con 0x7effc40fd150 2024-04-12T05:24:58.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.468+0000 7effc949c700 1 --2- 172.21.15.113:0/633240202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc40fd150 0x7effc4109610 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7effc0002820 tx=0x7effc0009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:58.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.469+0000 7effbaffd700 1 -- 172.21.15.113:0/633240202 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effc0007b40 con 0x7effc40fd150 2024-04-12T05:24:58.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.469+0000 7effcbf01700 1 -- 172.21.15.113:0/633240202 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc4109ef0 con 0x7effc40fd150 2024-04-12T05:24:58.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.469+0000 7effbaffd700 1 -- 172.21.15.113:0/633240202 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effc000f040 con 0x7effc40fd150 2024-04-12T05:24:58.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.469+0000 7effbaffd700 1 -- 172.21.15.113:0/633240202 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effc00148b0 con 0x7effc40fd150 2024-04-12T05:24:58.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.469+0000 7effcbf01700 1 -- 172.21.15.113:0/633240202 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc41124e0 con 0x7effc40fd150 2024-04-12T05:24:58.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.470+0000 7effcbf01700 1 -- 172.21.15.113:0/633240202 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc405f590 con 0x7effc40fd150 2024-04-12T05:24:58.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.470+0000 7effbaffd700 1 -- 172.21.15.113:0/633240202 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7effc0007ca0 con 0x7effc40fd150 2024-04-12T05:24:58.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.470+0000 7effbaffd700 1 --2- 172.21.15.113:0/633240202 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7effb006caa0 0x7effb006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:58.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.471+0000 7effbaffd700 1 -- 172.21.15.113:0/633240202 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7effc008b8a0 con 0x7effc40fd150 2024-04-12T05:24:58.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.471+0000 7effc9c9d700 1 --2- 172.21.15.113:0/633240202 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7effb006caa0 0x7effb006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:58.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.471+0000 7effc9c9d700 1 --2- 172.21.15.113:0/633240202 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7effb006caa0 0x7effb006ef60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7effb4004f10 tx=0x7effb400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:58.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.475+0000 7effbaffd700 1 -- 172.21.15.113:0/633240202 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7effc00558c0 con 0x7effc40fd150 2024-04-12T05:24:58.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.671+0000 7effcbf01700 1 -- 172.21.15.113:0/633240202 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7effc4060d30 con 0x7effb006caa0 2024-04-12T05:24:58.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.680+0000 7effbaffd700 1 -- 172.21.15.113:0/633240202 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7effc4060d30 con 0x7effb006caa0 2024-04-12T05:24:58.684 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:24:58.684 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (12m) 5m ago 24m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:24:58.684 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (10m) 5m ago 23m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:24:58.684 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (10m) 5m ago 23m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:24:58.684 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (12m) 5m ago 23m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:24:58.684 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (6m) 5m ago 18m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:24:58.684 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (5m) 5m ago 18m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (5m) 5m ago 18m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (18m) 5m ago 18m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (14m) 5m ago 25m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (13m) 5m ago 23m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (11m) 5m ago 25m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (10m) 5m ago 23m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (13m) 5m ago 23m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (13m) 5m ago 23m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (9m) 5m ago 22m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (9m) 5m ago 22m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (8m) 5m ago 21m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:24:58.685 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (8m) 5m ago 21m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:24:58.686 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (7m) 5m ago 20m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:24:58.686 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (6m) 5m ago 20m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:24:58.686 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (13m) 5m ago 22m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:24:58.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.683+0000 7effcbf01700 1 -- 172.21.15.113:0/633240202 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7effb006caa0 msgr2=0x7effb006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.683+0000 7effcbf01700 1 --2- 172.21.15.113:0/633240202 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7effb006caa0 0x7effb006ef60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7effb4004f10 tx=0x7effb400b040).stop 2024-04-12T05:24:58.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.683+0000 7effcbf01700 1 -- 172.21.15.113:0/633240202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc40fd150 msgr2=0x7effc4109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.683+0000 7effcbf01700 1 --2- 172.21.15.113:0/633240202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc40fd150 0x7effc4109610 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7effc0002820 tx=0x7effc0009f20).stop 2024-04-12T05:24:58.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.683+0000 7effcbf01700 1 -- 172.21.15.113:0/633240202 shutdown_connections 2024-04-12T05:24:58.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.683+0000 7effcbf01700 1 --2- 172.21.15.113:0/633240202 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7effb006caa0 0x7effb006ef60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.683+0000 7effcbf01700 1 --2- 172.21.15.113:0/633240202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc40fc7f0 0x7effc4109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.683+0000 7effcbf01700 1 --2- 172.21.15.113:0/633240202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effc40fd150 0x7effc4109610 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.683+0000 7effcbf01700 1 -- 172.21.15.113:0/633240202 >> 172.21.15.113:0/633240202 conn(0x7effc4070590 msgr2=0x7effc40fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:58.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.683+0000 7effcbf01700 1 -- 172.21.15.113:0/633240202 shutdown_connections 2024-04-12T05:24:58.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.683+0000 7effcbf01700 1 -- 172.21.15.113:0/633240202 wait complete. 2024-04-12T05:24:58.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.779+0000 7f0afd4c6700 1 -- 172.21.15.113:0/560089362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0af80fc7e0 msgr2=0x7f0af80fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.779+0000 7f0afd4c6700 1 --2- 172.21.15.113:0/560089362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0af80fc7e0 0x7f0af80fcbc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f0ae80047f0 tx=0x7f0ae8009cb0).stop 2024-04-12T05:24:58.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.779+0000 7f0afd4c6700 1 -- 172.21.15.113:0/560089362 shutdown_connections 2024-04-12T05:24:58.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.779+0000 7f0afd4c6700 1 --2- 172.21.15.113:0/560089362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0af80fd140 0x7f0af80f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.779+0000 7f0afd4c6700 1 --2- 172.21.15.113:0/560089362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0af80fc7e0 0x7f0af80fcbc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.779+0000 7f0afd4c6700 1 -- 172.21.15.113:0/560089362 >> 172.21.15.113:0/560089362 conn(0x7f0af80f23c0 msgr2=0x7f0af80f47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:58.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.780+0000 7f0afd4c6700 1 -- 172.21.15.113:0/560089362 shutdown_connections 2024-04-12T05:24:58.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.780+0000 7f0afd4c6700 1 -- 172.21.15.113:0/560089362 wait complete. 2024-04-12T05:24:58.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.780+0000 7f0afd4c6700 1 Processor -- start 2024-04-12T05:24:58.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.780+0000 7f0afd4c6700 1 -- start start 2024-04-12T05:24:58.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.780+0000 7f0afd4c6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0af80fc7e0 0x7f0af810cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:58.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.780+0000 7f0afd4c6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0af80fd140 0x7f0af810d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:58.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0afd4c6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af810d9d0 con 0x7f0af80fd140 2024-04-12T05:24:58.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0afd4c6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af810db10 con 0x7f0af80fc7e0 2024-04-12T05:24:58.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0af67fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0af80fd140 0x7f0af810d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:58.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0af67fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0af80fd140 0x7f0af810d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55768/0 (socket says 172.21.15.113:55768) 2024-04-12T05:24:58.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0af67fc700 1 -- 172.21.15.113:0/67801867 learned_addr learned my addr 172.21.15.113:0/67801867 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:58.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0af6ffd700 1 --2- 172.21.15.113:0/67801867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0af80fc7e0 0x7f0af810cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:58.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0af67fc700 1 -- 172.21.15.113:0/67801867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0af80fc7e0 msgr2=0x7f0af810cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:58.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0af67fc700 1 --2- 172.21.15.113:0/67801867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0af80fc7e0 0x7f0af810cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:58.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0af67fc700 1 -- 172.21.15.113:0/67801867 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae8005040 con 0x7f0af80fd140 2024-04-12T05:24:58.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0af67fc700 1 --2- 172.21.15.113:0/67801867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0af80fd140 0x7f0af810d3d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f0aec0044e0 tx=0x7f0aec002a50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:58.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0adffff700 1 -- 172.21.15.113:0/67801867 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aec007b40 con 0x7f0af80fd140 2024-04-12T05:24:58.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0afd4c6700 1 -- 172.21.15.113:0/67801867 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af81071a0 con 0x7f0af80fd140 2024-04-12T05:24:58.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0adffff700 1 -- 172.21.15.113:0/67801867 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aec00f040 con 0x7f0af80fd140 2024-04-12T05:24:58.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0adffff700 1 -- 172.21.15.113:0/67801867 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aec014850 con 0x7f0af80fd140 2024-04-12T05:24:58.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.781+0000 7f0afd4c6700 1 -- 172.21.15.113:0/67801867 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af81076e0 con 0x7f0af80fd140 2024-04-12T05:24:58.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.782+0000 7f0afd4c6700 1 -- 172.21.15.113:0/67801867 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af8060370 con 0x7f0af80fd140 2024-04-12T05:24:58.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.783+0000 7f0adffff700 1 -- 172.21.15.113:0/67801867 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0aec009110 con 0x7f0af80fd140 2024-04-12T05:24:58.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.783+0000 7f0adffff700 1 --2- 172.21.15.113:0/67801867 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0ae006caa0 0x7f0ae006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:58.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.783+0000 7f0af6ffd700 1 --2- 172.21.15.113:0/67801867 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0ae006caa0 0x7f0ae006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:58.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.783+0000 7f0adffff700 1 -- 172.21.15.113:0/67801867 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0aec08b960 con 0x7f0af80fd140 2024-04-12T05:24:58.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.784+0000 7f0af6ffd700 1 --2- 172.21.15.113:0/67801867 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0ae006caa0 0x7f0ae006ef60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f0ae8004f10 tx=0x7f0ae800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:58.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:58.788+0000 7f0adffff700 1 -- 172.21.15.113:0/67801867 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0aec059410 con 0x7f0af80fd140 2024-04-12T05:24:59.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.056+0000 7f0afd4c6700 1 -- 172.21.15.113:0/67801867 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0af8047540 con 0x7f0af80fd140 2024-04-12T05:24:59.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.056+0000 7f0adffff700 1 -- 172.21.15.113:0/67801867 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f0aec058fa0 con 0x7f0af80fd140 2024-04-12T05:24:59.060 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:24:59.060 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:24:59.060 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:24:59.060 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:24:59.060 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:24:59.060 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:24:59.060 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:24:59.060 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:24:59.060 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:24:59.061 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:24:59.061 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:24:59.061 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:24:59.061 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:24:59.061 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:24:59.061 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:24:59.061 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:24:59.061 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:24:59.061 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:24:59.061 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:24:59.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.058+0000 7f0afd4c6700 1 -- 172.21.15.113:0/67801867 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0ae006caa0 msgr2=0x7f0ae006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.058+0000 7f0afd4c6700 1 --2- 172.21.15.113:0/67801867 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0ae006caa0 0x7f0ae006ef60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f0ae8004f10 tx=0x7f0ae800b040).stop 2024-04-12T05:24:59.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.058+0000 7f0afd4c6700 1 -- 172.21.15.113:0/67801867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0af80fd140 msgr2=0x7f0af810d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.058+0000 7f0afd4c6700 1 --2- 172.21.15.113:0/67801867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0af80fd140 0x7f0af810d3d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f0aec0044e0 tx=0x7f0aec002a50).stop 2024-04-12T05:24:59.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.058+0000 7f0afd4c6700 1 -- 172.21.15.113:0/67801867 shutdown_connections 2024-04-12T05:24:59.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.058+0000 7f0afd4c6700 1 --2- 172.21.15.113:0/67801867 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0ae006caa0 0x7f0ae006ef60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.058+0000 7f0afd4c6700 1 --2- 172.21.15.113:0/67801867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0af80fc7e0 0x7f0af810cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.058+0000 7f0afd4c6700 1 --2- 172.21.15.113:0/67801867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0af80fd140 0x7f0af810d3d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.058+0000 7f0afd4c6700 1 -- 172.21.15.113:0/67801867 >> 172.21.15.113:0/67801867 conn(0x7f0af80f23c0 msgr2=0x7f0af80faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:59.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.058+0000 7f0afd4c6700 1 -- 172.21.15.113:0/67801867 shutdown_connections 2024-04-12T05:24:59.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.058+0000 7f0afd4c6700 1 -- 172.21.15.113:0/67801867 wait complete. 2024-04-12T05:24:59.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.157+0000 7f84f0641700 1 -- 172.21.15.113:0/1953547886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84e80fd8f0 msgr2=0x7f84e80fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.157+0000 7f84f0641700 1 --2- 172.21.15.113:0/1953547886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84e80fd8f0 0x7f84e80fdcd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f84e40047f0 tx=0x7f84e4009cb0).stop 2024-04-12T05:24:59.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 -- 172.21.15.113:0/1953547886 shutdown_connections 2024-04-12T05:24:59.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 --2- 172.21.15.113:0/1953547886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84e80fe250 0x7f84e80f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 --2- 172.21.15.113:0/1953547886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84e80fd8f0 0x7f84e80fdcd0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 -- 172.21.15.113:0/1953547886 >> 172.21.15.113:0/1953547886 conn(0x7f84e80f34b0 msgr2=0x7f84e80f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:59.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 -- 172.21.15.113:0/1953547886 shutdown_connections 2024-04-12T05:24:59.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 -- 172.21.15.113:0/1953547886 wait complete. 2024-04-12T05:24:59.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 Processor -- start 2024-04-12T05:24:59.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 -- start start 2024-04-12T05:24:59.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84e80fd8f0 0x7f84e810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84e80fe250 0x7f84e810e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e810ead0 con 0x7f84e80fe250 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.158+0000 7f84f0641700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e810ec10 con 0x7f84e80fd8f0 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84edbdc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84e80fe250 0x7f84e810e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84edbdc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84e80fe250 0x7f84e810e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55778/0 (socket says 172.21.15.113:55778) 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84edbdc700 1 -- 172.21.15.113:0/528241322 learned_addr learned my addr 172.21.15.113:0/528241322 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84ee3dd700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84e80fd8f0 0x7f84e810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84edbdc700 1 -- 172.21.15.113:0/528241322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84e80fd8f0 msgr2=0x7f84e810dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84edbdc700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84e80fd8f0 0x7f84e810dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84edbdc700 1 -- 172.21.15.113:0/528241322 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e4005040 con 0x7f84e80fe250 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84ee3dd700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84e80fd8f0 0x7f84e810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84edbdc700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84e80fe250 0x7f84e810e4d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f84d8002780 tx=0x7f84d8009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84df7fe700 1 -- 172.21.15.113:0/528241322 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84d8007b40 con 0x7f84e80fe250 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84f0641700 1 -- 172.21.15.113:0/528241322 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e81082a0 con 0x7f84e80fe250 2024-04-12T05:24:59.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84df7fe700 1 -- 172.21.15.113:0/528241322 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84d800f040 con 0x7f84e80fe250 2024-04-12T05:24:59.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84df7fe700 1 -- 172.21.15.113:0/528241322 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84d8014780 con 0x7f84e80fe250 2024-04-12T05:24:59.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.159+0000 7f84f0641700 1 -- 172.21.15.113:0/528241322 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e8108860 con 0x7f84e80fe250 2024-04-12T05:24:59.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.160+0000 7f84f0641700 1 -- 172.21.15.113:0/528241322 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e805f590 con 0x7f84e80fe250 2024-04-12T05:24:59.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.161+0000 7f84df7fe700 1 -- 172.21.15.113:0/528241322 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f84d8007ca0 con 0x7f84e80fe250 2024-04-12T05:24:59.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.161+0000 7f84df7fe700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84d406caa0 0x7f84d406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:59.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.161+0000 7f84df7fe700 1 -- 172.21.15.113:0/528241322 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f84d808bde0 con 0x7f84e80fe250 2024-04-12T05:24:59.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.161+0000 7f84ee3dd700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84d406caa0 0x7f84d406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:59.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.162+0000 7f84ee3dd700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84d406caa0 0x7f84d406ef60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f84e4004540 tx=0x7f84e400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:59.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.166+0000 7f84df7fe700 1 -- 172.21.15.113:0/528241322 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f84d8059890 con 0x7f84e80fe250 2024-04-12T05:24:59.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.396+0000 7f84f0641700 1 -- 172.21.15.113:0/528241322 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f84e8047540 con 0x7f84e80fe250 2024-04-12T05:24:59.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.396+0000 7f84df7fe700 1 -- 172.21.15.113:0/528241322 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f84d8059420 con 0x7f84e80fe250 2024-04-12T05:24:59.400 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:24:59.400 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:24:59.400 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:24:59.400 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:24:59.400 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:24:59.401 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:24:59.402 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:24:59.403 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:24:59.403 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:24:59.403 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:24:59.403 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:24:59.403 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:24:59.403 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:24:59.403 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:24:59.403 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:24:59.403 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:24:59.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.399+0000 7f84f0641700 1 -- 172.21.15.113:0/528241322 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84d406caa0 msgr2=0x7f84d406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.399+0000 7f84f0641700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84d406caa0 0x7f84d406ef60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f84e4004540 tx=0x7f84e400b040).stop 2024-04-12T05:24:59.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.399+0000 7f84f0641700 1 -- 172.21.15.113:0/528241322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84e80fe250 msgr2=0x7f84e810e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.399+0000 7f84f0641700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84e80fe250 0x7f84e810e4d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f84d8002780 tx=0x7f84d8009ee0).stop 2024-04-12T05:24:59.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.399+0000 7f84f0641700 1 -- 172.21.15.113:0/528241322 shutdown_connections 2024-04-12T05:24:59.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.399+0000 7f84f0641700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84d406caa0 0x7f84d406ef60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.399+0000 7f84f0641700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84e80fd8f0 0x7f84e810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.399+0000 7f84f0641700 1 --2- 172.21.15.113:0/528241322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84e80fe250 0x7f84e810e4d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.399+0000 7f84f0641700 1 -- 172.21.15.113:0/528241322 >> 172.21.15.113:0/528241322 conn(0x7f84e80f34b0 msgr2=0x7f84e80fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.399+0000 7f84f0641700 1 -- 172.21.15.113:0/528241322 shutdown_connections 2024-04-12T05:24:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.399+0000 7f84f0641700 1 -- 172.21.15.113:0/528241322 wait complete. 2024-04-12T05:24:59.405 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:24:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:59 smithi113 ceph-mon[103371]: pgmap v384: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:59 smithi113 ceph-mon[103371]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:24:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:24:59 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/67801867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:24:59.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:59 smithi192 ceph-mon[85304]: pgmap v384: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:24:59.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:59 smithi192 ceph-mon[85304]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:24:59.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:24:59 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/67801867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:24:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.498+0000 7f2bc64c6700 1 -- 172.21.15.113:0/1983480563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bc00fe250 msgr2=0x7f2bc00f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.498+0000 7f2bc64c6700 1 --2- 172.21.15.113:0/1983480563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bc00fe250 0x7f2bc00f7a30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f2ba80047f0 tx=0x7f2ba8009cb0).stop 2024-04-12T05:24:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.498+0000 7f2bc64c6700 1 -- 172.21.15.113:0/1983480563 shutdown_connections 2024-04-12T05:24:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.498+0000 7f2bc64c6700 1 --2- 172.21.15.113:0/1983480563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bc00fe250 0x7f2bc00f7a30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.498+0000 7f2bc64c6700 1 --2- 172.21.15.113:0/1983480563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bc00fd8f0 0x7f2bc00fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.498+0000 7f2bc64c6700 1 -- 172.21.15.113:0/1983480563 >> 172.21.15.113:0/1983480563 conn(0x7f2bc00f34b0 msgr2=0x7f2bc00f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.498+0000 7f2bc64c6700 1 -- 172.21.15.113:0/1983480563 shutdown_connections 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.498+0000 7f2bc64c6700 1 -- 172.21.15.113:0/1983480563 wait complete. 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bc64c6700 1 Processor -- start 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bc64c6700 1 -- start start 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bc64c6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bc00fd8f0 0x7f2bc0110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bc64c6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bc00fe250 0x7f2bc0110750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bc64c6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc0110d50 con 0x7f2bc00fd8f0 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bc64c6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc0110e90 con 0x7f2bc00fe250 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bbffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bc00fd8f0 0x7f2bc0110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bbffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bc00fd8f0 0x7f2bc0110260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55788/0 (socket says 172.21.15.113:55788) 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bbffff700 1 -- 172.21.15.113:0/3305237333 learned_addr learned my addr 172.21.15.113:0/3305237333 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bbf7fe700 1 --2- 172.21.15.113:0/3305237333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bc00fe250 0x7f2bc0110750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:59.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bbffff700 1 -- 172.21.15.113:0/3305237333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bc00fe250 msgr2=0x7f2bc0110750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bbffff700 1 --2- 172.21.15.113:0/3305237333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bc00fe250 0x7f2bc0110750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.499+0000 7f2bbffff700 1 -- 172.21.15.113:0/3305237333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba8005040 con 0x7f2bc00fd8f0 2024-04-12T05:24:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.500+0000 7f2bbffff700 1 --2- 172.21.15.113:0/3305237333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bc00fd8f0 0x7f2bc0110260 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f2bb00041d0 tx=0x7f2bb0009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.500+0000 7f2bbd7fa700 1 -- 172.21.15.113:0/3305237333 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bb0007b40 con 0x7f2bc00fd8f0 2024-04-12T05:24:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.500+0000 7f2bc64c6700 1 -- 172.21.15.113:0/3305237333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc010a520 con 0x7f2bc00fd8f0 2024-04-12T05:24:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.500+0000 7f2bbd7fa700 1 -- 172.21.15.113:0/3305237333 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bb000f040 con 0x7f2bc00fd8f0 2024-04-12T05:24:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.500+0000 7f2bbd7fa700 1 -- 172.21.15.113:0/3305237333 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bb00147b0 con 0x7f2bc00fd8f0 2024-04-12T05:24:59.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.500+0000 7f2bc64c6700 1 -- 172.21.15.113:0/3305237333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc010aa60 con 0x7f2bc00fd8f0 2024-04-12T05:24:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.501+0000 7f2bc64c6700 1 -- 172.21.15.113:0/3305237333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc005f590 con 0x7f2bc00fd8f0 2024-04-12T05:24:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.501+0000 7f2bbd7fa700 1 -- 172.21.15.113:0/3305237333 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2bb0009110 con 0x7f2bc00fd8f0 2024-04-12T05:24:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.502+0000 7f2bbd7fa700 1 --2- 172.21.15.113:0/3305237333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bac06caa0 0x7f2bac06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.502+0000 7f2bbd7fa700 1 -- 172.21.15.113:0/3305237333 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2bb008bf10 con 0x7f2bc00fd8f0 2024-04-12T05:24:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.502+0000 7f2bbf7fe700 1 --2- 172.21.15.113:0/3305237333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bac06caa0 0x7f2bac06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:59.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.502+0000 7f2bbf7fe700 1 --2- 172.21.15.113:0/3305237333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bac06caa0 0x7f2bac06ef60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f2bc010bba0 tx=0x7f2ba800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:59.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.506+0000 7f2bbd7fa700 1 -- 172.21.15.113:0/3305237333 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2bb0059900 con 0x7f2bc00fd8f0 2024-04-12T05:24:59.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.710+0000 7f2bc64c6700 1 -- 172.21.15.113:0/3305237333 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bc010ace0 con 0x7f2bac06caa0 2024-04-12T05:24:59.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.711+0000 7f2bbd7fa700 1 -- 172.21.15.113:0/3305237333 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2bc010ace0 con 0x7f2bac06caa0 2024-04-12T05:24:59.715 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:24:59.715 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:24:59.715 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:24:59.715 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:24:59.715 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:24:59.715 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:24:59.715 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:24:59.715 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:24:59.715 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:24:59.716 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:24:59.716 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:24:59.716 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:24:59.716 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:24:59.716 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:24:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.713+0000 7f2bc64c6700 1 -- 172.21.15.113:0/3305237333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bac06caa0 msgr2=0x7f2bac06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.713+0000 7f2bc64c6700 1 --2- 172.21.15.113:0/3305237333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bac06caa0 0x7f2bac06ef60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f2bc010bba0 tx=0x7f2ba800b040).stop 2024-04-12T05:24:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.713+0000 7f2bc64c6700 1 -- 172.21.15.113:0/3305237333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bc00fd8f0 msgr2=0x7f2bc0110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.713+0000 7f2bc64c6700 1 --2- 172.21.15.113:0/3305237333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bc00fd8f0 0x7f2bc0110260 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f2bb00041d0 tx=0x7f2bb0009f60).stop 2024-04-12T05:24:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.714+0000 7f2bc64c6700 1 -- 172.21.15.113:0/3305237333 shutdown_connections 2024-04-12T05:24:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.714+0000 7f2bc64c6700 1 --2- 172.21.15.113:0/3305237333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bac06caa0 0x7f2bac06ef60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.714+0000 7f2bc64c6700 1 --2- 172.21.15.113:0/3305237333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2bc00fd8f0 0x7f2bc0110260 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.714+0000 7f2bc64c6700 1 --2- 172.21.15.113:0/3305237333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bc00fe250 0x7f2bc0110750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.714+0000 7f2bc64c6700 1 -- 172.21.15.113:0/3305237333 >> 172.21.15.113:0/3305237333 conn(0x7f2bc00f34b0 msgr2=0x7f2bc006b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.714+0000 7f2bc64c6700 1 -- 172.21.15.113:0/3305237333 shutdown_connections 2024-04-12T05:24:59.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.714+0000 7f2bc64c6700 1 -- 172.21.15.113:0/3305237333 wait complete. 2024-04-12T05:24:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 -- 172.21.15.113:0/242731292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f540fc7f0 msgr2=0x7f6f540fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 --2- 172.21.15.113:0/242731292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f540fc7f0 0x7f6f540fcbd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f6f480047f0 tx=0x7f6f48009cb0).stop 2024-04-12T05:24:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 -- 172.21.15.113:0/242731292 shutdown_connections 2024-04-12T05:24:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 --2- 172.21.15.113:0/242731292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f540fd150 0x7f6f540f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 --2- 172.21.15.113:0/242731292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f540fc7f0 0x7f6f540fcbd0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 -- 172.21.15.113:0/242731292 >> 172.21.15.113:0/242731292 conn(0x7f6f54070590 msgr2=0x7f6f540f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 -- 172.21.15.113:0/242731292 shutdown_connections 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 -- 172.21.15.113:0/242731292 wait complete. 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 Processor -- start 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 -- start start 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f540fc7f0 0x7f6f540f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f540fd150 0x7f6f540f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540f8f20 con 0x7f6f540fd150 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.812+0000 7f6f5acfc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540f9060 con 0x7f6f540fc7f0 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f53fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f540fd150 0x7f6f540f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f53fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f540fd150 0x7f6f540f8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55808/0 (socket says 172.21.15.113:55808) 2024-04-12T05:24:59.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f53fff700 1 -- 172.21.15.113:0/2787495613 learned_addr learned my addr 172.21.15.113:0/2787495613 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:24:59.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f58a98700 1 --2- 172.21.15.113:0/2787495613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f540fc7f0 0x7f6f540f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:59.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f53fff700 1 -- 172.21.15.113:0/2787495613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f540fc7f0 msgr2=0x7f6f540f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:24:59.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f53fff700 1 --2- 172.21.15.113:0/2787495613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f540fc7f0 0x7f6f540f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:24:59.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f53fff700 1 -- 172.21.15.113:0/2787495613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f48005040 con 0x7f6f540fd150 2024-04-12T05:24:59.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f53fff700 1 --2- 172.21.15.113:0/2787495613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f540fd150 0x7f6f540f8a30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f6f40002820 tx=0x7f6f40009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:59.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f51ffb700 1 -- 172.21.15.113:0/2787495613 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f40007b40 con 0x7f6f540fd150 2024-04-12T05:24:59.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f51ffb700 1 -- 172.21.15.113:0/2787495613 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f4000f040 con 0x7f6f540fd150 2024-04-12T05:24:59.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f5acfc700 1 -- 172.21.15.113:0/2787495613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f540f9310 con 0x7f6f540fd150 2024-04-12T05:24:59.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f51ffb700 1 -- 172.21.15.113:0/2787495613 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f40014910 con 0x7f6f540fd150 2024-04-12T05:24:59.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.813+0000 7f6f5acfc700 1 -- 172.21.15.113:0/2787495613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f54114710 con 0x7f6f540fd150 2024-04-12T05:24:59.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.814+0000 7f6f5acfc700 1 -- 172.21.15.113:0/2787495613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5405f590 con 0x7f6f540fd150 2024-04-12T05:24:59.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.815+0000 7f6f51ffb700 1 -- 172.21.15.113:0/2787495613 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6f40007ca0 con 0x7f6f540fd150 2024-04-12T05:24:59.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.815+0000 7f6f51ffb700 1 --2- 172.21.15.113:0/2787495613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4406caa0 0x7f6f4406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:24:59.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.815+0000 7f6f51ffb700 1 -- 172.21.15.113:0/2787495613 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6f4008b7d0 con 0x7f6f540fd150 2024-04-12T05:24:59.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.815+0000 7f6f58a98700 1 --2- 172.21.15.113:0/2787495613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4406caa0 0x7f6f4406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:24:59.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.816+0000 7f6f58a98700 1 --2- 172.21.15.113:0/2787495613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4406caa0 0x7f6f4406ef60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f6f48000c00 tx=0x7f6f4800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:24:59.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:24:59.820+0000 7f6f51ffb700 1 -- 172.21.15.113:0/2787495613 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6f400557f0 con 0x7f6f540fd150 2024-04-12T05:25:00.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.085+0000 7f6f5acfc700 1 -- 172.21.15.113:0/2787495613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6f54047540 con 0x7f6f540fd150 2024-04-12T05:25:00.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.085+0000 7f6f51ffb700 1 -- 172.21.15.113:0/2787495613 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6f40058e10 con 0x7f6f540fd150 2024-04-12T05:25:00.089 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:25:00.089 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:25:00.089 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:25:00.089 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:25:00.089 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:25:00.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.088+0000 7f6f5acfc700 1 -- 172.21.15.113:0/2787495613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4406caa0 msgr2=0x7f6f4406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:00.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.088+0000 7f6f5acfc700 1 --2- 172.21.15.113:0/2787495613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4406caa0 0x7f6f4406ef60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f6f48000c00 tx=0x7f6f4800b040).stop 2024-04-12T05:25:00.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.088+0000 7f6f5acfc700 1 -- 172.21.15.113:0/2787495613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f540fd150 msgr2=0x7f6f540f8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:00.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.088+0000 7f6f5acfc700 1 --2- 172.21.15.113:0/2787495613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f540fd150 0x7f6f540f8a30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f6f40002820 tx=0x7f6f40009f20).stop 2024-04-12T05:25:00.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.088+0000 7f6f5acfc700 1 -- 172.21.15.113:0/2787495613 shutdown_connections 2024-04-12T05:25:00.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.088+0000 7f6f5acfc700 1 --2- 172.21.15.113:0/2787495613 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4406caa0 0x7f6f4406ef60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:00.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.088+0000 7f6f5acfc700 1 --2- 172.21.15.113:0/2787495613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f540fc7f0 0x7f6f540f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:00.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.088+0000 7f6f5acfc700 1 --2- 172.21.15.113:0/2787495613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f540fd150 0x7f6f540f8a30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:00.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.088+0000 7f6f5acfc700 1 -- 172.21.15.113:0/2787495613 >> 172.21.15.113:0/2787495613 conn(0x7f6f54070590 msgr2=0x7f6f54060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:00.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.088+0000 7f6f5acfc700 1 -- 172.21.15.113:0/2787495613 shutdown_connections 2024-04-12T05:25:00.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:00.088+0000 7f6f5acfc700 1 -- 172.21.15.113:0/2787495613 wait complete. 2024-04-12T05:25:00.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:00 smithi113 ceph-mon[103371]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:00.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:00 smithi113 ceph-mon[103371]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:00.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:00 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/528241322' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:25:00.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:00 smithi192 ceph-mon[85304]: from='client.34732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:00.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:00 smithi192 ceph-mon[85304]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:00.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:00 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/528241322' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:25:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:01 smithi113 ceph-mon[103371]: from='client.34748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:01.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:01 smithi113 ceph-mon[103371]: pgmap v385: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:01.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:01 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2787495613' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:25:01.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:01 smithi192 ceph-mon[85304]: from='client.34748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:01.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:01 smithi192 ceph-mon[85304]: pgmap v385: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:01.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:01 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2787495613' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:25:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:03 smithi113 ceph-mon[103371]: pgmap v386: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:03 smithi192 ceph-mon[85304]: pgmap v386: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:05 smithi113 ceph-mon[103371]: pgmap v387: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:05.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:05 smithi192 ceph-mon[85304]: pgmap v387: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:07 smithi113 ceph-mon[103371]: pgmap v388: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:25:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:07 smithi192 ceph-mon[85304]: pgmap v388: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:25:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:09 smithi113 ceph-mon[103371]: pgmap v389: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:09 smithi192 ceph-mon[85304]: pgmap v389: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:11 smithi113 ceph-mon[103371]: pgmap v390: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:11 smithi192 ceph-mon[85304]: pgmap v390: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:13 smithi113 ceph-mon[103371]: pgmap v391: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:13 smithi192 ceph-mon[85304]: pgmap v391: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:15 smithi113 ceph-mon[103371]: pgmap v392: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:15 smithi192 ceph-mon[85304]: pgmap v392: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:17 smithi113 ceph-mon[103371]: pgmap v393: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:17.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:17 smithi192 ceph-mon[85304]: pgmap v393: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:19 smithi113 ceph-mon[103371]: pgmap v394: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:19 smithi192 ceph-mon[85304]: pgmap v394: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:21 smithi113 ceph-mon[103371]: pgmap v395: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:21 smithi192 ceph-mon[85304]: pgmap v395: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:25:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:25:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:23 smithi113 ceph-mon[103371]: pgmap v396: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:23 smithi192 ceph-mon[85304]: pgmap v396: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:25 smithi113 ceph-mon[103371]: pgmap v397: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:25 smithi192 ceph-mon[85304]: pgmap v397: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:27 smithi113 ceph-mon[103371]: pgmap v398: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:27.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:27 smithi192 ceph-mon[85304]: pgmap v398: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:29 smithi113 ceph-mon[103371]: pgmap v399: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:29.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:29 smithi192 ceph-mon[85304]: pgmap v399: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:30.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.201+0000 7fbec094b700 1 -- 172.21.15.113:0/1399150241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0fe270 msgr2=0x7fbebc0f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:30.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.201+0000 7fbec094b700 1 --2- 172.21.15.113:0/1399150241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0fe270 0x7fbebc0f7a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fbeac0047f0 tx=0x7fbeac009cb0).stop 2024-04-12T05:25:30.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.201+0000 7fbec094b700 1 -- 172.21.15.113:0/1399150241 shutdown_connections 2024-04-12T05:25:30.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.201+0000 7fbec094b700 1 --2- 172.21.15.113:0/1399150241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0fe270 0x7fbebc0f7a50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.201+0000 7fbec094b700 1 --2- 172.21.15.113:0/1399150241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbebc0fd910 0x7fbebc0fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.201+0000 7fbec094b700 1 -- 172.21.15.113:0/1399150241 >> 172.21.15.113:0/1399150241 conn(0x7fbebc0f3530 msgr2=0x7fbebc0f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:30.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.201+0000 7fbec094b700 1 -- 172.21.15.113:0/1399150241 shutdown_connections 2024-04-12T05:25:30.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.201+0000 7fbec094b700 1 -- 172.21.15.113:0/1399150241 wait complete. 2024-04-12T05:25:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.202+0000 7fbec094b700 1 Processor -- start 2024-04-12T05:25:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.202+0000 7fbec094b700 1 -- start start 2024-04-12T05:25:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.202+0000 7fbec094b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbebc0fd910 0x7fbebc10dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.202+0000 7fbec094b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0fe270 0x7fbebc10e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.202+0000 7fbec094b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc10eae0 con 0x7fbebc0fe270 2024-04-12T05:25:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.202+0000 7fbec094b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbebc10ec20 con 0x7fbebc0fd910 2024-04-12T05:25:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeb9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0fe270 0x7fbebc10e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeb9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0fe270 0x7fbebc10e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48514/0 (socket says 172.21.15.113:48514) 2024-04-12T05:25:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeb9d9b700 1 -- 172.21.15.113:0/2826053178 learned_addr learned my addr 172.21.15.113:0/2826053178 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:25:30.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeba59c700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbebc0fd910 0x7fbebc10dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeb9d9b700 1 -- 172.21.15.113:0/2826053178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbebc0fd910 msgr2=0x7fbebc10dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeb9d9b700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbebc0fd910 0x7fbebc10dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeb9d9b700 1 -- 172.21.15.113:0/2826053178 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac005040 con 0x7fbebc0fe270 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeba59c700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbebc0fd910 0x7fbebc10dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeb9d9b700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0fe270 0x7fbebc10e4e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fbeac0099f0 tx=0x7fbeac009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeab7fe700 1 -- 172.21.15.113:0/2826053178 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeac00e030 con 0x7fbebc0fe270 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeab7fe700 1 -- 172.21.15.113:0/2826053178 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbeac007e90 con 0x7fbebc0fe270 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbec094b700 1 -- 172.21.15.113:0/2826053178 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc108250 con 0x7fbebc0fe270 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbeab7fe700 1 -- 172.21.15.113:0/2826053178 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeac018070 con 0x7fbebc0fe270 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.203+0000 7fbec094b700 1 -- 172.21.15.113:0/2826053178 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc1087b0 con 0x7fbebc0fe270 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.204+0000 7fbec094b700 1 -- 172.21.15.113:0/2826053178 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc05f570 con 0x7fbebc0fe270 2024-04-12T05:25:30.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.205+0000 7fbeab7fe700 1 -- 172.21.15.113:0/2826053178 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbeac009590 con 0x7fbebc0fe270 2024-04-12T05:25:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.205+0000 7fbeab7fe700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbea406caa0 0x7fbea406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.205+0000 7fbeab7fe700 1 -- 172.21.15.113:0/2826053178 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbeac08d010 con 0x7fbebc0fe270 2024-04-12T05:25:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.205+0000 7fbeba59c700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbea406caa0 0x7fbea406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:30.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.206+0000 7fbeba59c700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbea406caa0 0x7fbea406ef60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fbeb0004c80 tx=0x7fbeb000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:30.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.210+0000 7fbeab7fe700 1 -- 172.21.15.113:0/2826053178 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbeac05aac0 con 0x7fbebc0fe270 2024-04-12T05:25:30.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.415+0000 7fbec094b700 1 -- 172.21.15.113:0/2826053178 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbebc0ffc60 con 0x7fbea406caa0 2024-04-12T05:25:30.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.416+0000 7fbeab7fe700 1 -- 172.21.15.113:0/2826053178 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbebc0ffc60 con 0x7fbea406caa0 2024-04-12T05:25:30.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.418+0000 7fbec094b700 1 -- 172.21.15.113:0/2826053178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbea406caa0 msgr2=0x7fbea406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:30.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.418+0000 7fbec094b700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbea406caa0 0x7fbea406ef60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fbeb0004c80 tx=0x7fbeb000b040).stop 2024-04-12T05:25:30.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.418+0000 7fbec094b700 1 -- 172.21.15.113:0/2826053178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0fe270 msgr2=0x7fbebc10e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:30.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.418+0000 7fbec094b700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0fe270 0x7fbebc10e4e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fbeac0099f0 tx=0x7fbeac009cb0).stop 2024-04-12T05:25:30.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.419+0000 7fbec094b700 1 -- 172.21.15.113:0/2826053178 shutdown_connections 2024-04-12T05:25:30.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.419+0000 7fbec094b700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbea406caa0 0x7fbea406ef60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.419+0000 7fbec094b700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbebc0fd910 0x7fbebc10dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.419+0000 7fbec094b700 1 --2- 172.21.15.113:0/2826053178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbebc0fe270 0x7fbebc10e4e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.419+0000 7fbec094b700 1 -- 172.21.15.113:0/2826053178 >> 172.21.15.113:0/2826053178 conn(0x7fbebc0f3530 msgr2=0x7fbebc0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:30.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.419+0000 7fbec094b700 1 -- 172.21.15.113:0/2826053178 shutdown_connections 2024-04-12T05:25:30.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.419+0000 7fbec094b700 1 -- 172.21.15.113:0/2826053178 wait complete. 2024-04-12T05:25:30.431 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:25:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.518+0000 7f8fc259d700 1 -- 172.21.15.113:0/2928750978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fbc0fcf40 msgr2=0x7f8fbc0f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.518+0000 7f8fc259d700 1 --2- 172.21.15.113:0/2928750978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fbc0fcf40 0x7f8fbc0f6d40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f8fb80047f0 tx=0x7f8fb8009cb0).stop 2024-04-12T05:25:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.518+0000 7f8fc259d700 1 -- 172.21.15.113:0/2928750978 shutdown_connections 2024-04-12T05:25:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.518+0000 7f8fc259d700 1 --2- 172.21.15.113:0/2928750978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fbc0fcf40 0x7f8fbc0f6d40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.518+0000 7f8fc259d700 1 --2- 172.21.15.113:0/2928750978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc0fc5e0 0x7f8fbc0fc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.518+0000 7f8fc259d700 1 -- 172.21.15.113:0/2928750978 >> 172.21.15.113:0/2928750978 conn(0x7f8fbc0f1910 msgr2=0x7f8fbc0f3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:30.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.518+0000 7f8fc259d700 1 -- 172.21.15.113:0/2928750978 shutdown_connections 2024-04-12T05:25:30.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.518+0000 7f8fc259d700 1 -- 172.21.15.113:0/2928750978 wait complete. 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.518+0000 7f8fc259d700 1 Processor -- start 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc259d700 1 -- start start 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc259d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc0fc5e0 0x7f8fbc10ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc259d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fbc10d350 0x7f8fbc106df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc259d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc10d8e0 con 0x7f8fbc10d350 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc259d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc10da20 con 0x7f8fbc0fc5e0 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc0d9a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fbc10d350 0x7f8fbc106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc0d9a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fbc10d350 0x7f8fbc106df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40504/0 (socket says 172.21.15.113:40504) 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc0d9a700 1 -- 172.21.15.113:0/57184996 learned_addr learned my addr 172.21.15.113:0/57184996 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc159b700 1 --2- 172.21.15.113:0/57184996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc0fc5e0 0x7f8fbc10ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc0d9a700 1 -- 172.21.15.113:0/57184996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc0fc5e0 msgr2=0x7f8fbc10ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc0d9a700 1 --2- 172.21.15.113:0/57184996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc0fc5e0 0x7f8fbc10ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc0d9a700 1 -- 172.21.15.113:0/57184996 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb8005040 con 0x7f8fbc10d350 2024-04-12T05:25:30.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fc0d9a700 1 --2- 172.21.15.113:0/57184996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fbc10d350 0x7f8fbc106df0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f8fb80099f0 tx=0x7f8fb8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:30.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.519+0000 7f8fb27fc700 1 -- 172.21.15.113:0/57184996 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fb800e030 con 0x7f8fbc10d350 2024-04-12T05:25:30.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.520+0000 7f8fc259d700 1 -- 172.21.15.113:0/57184996 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fbc107340 con 0x7f8fbc10d350 2024-04-12T05:25:30.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.520+0000 7f8fb27fc700 1 -- 172.21.15.113:0/57184996 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8fb8007e90 con 0x7f8fbc10d350 2024-04-12T05:25:30.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.520+0000 7f8fb27fc700 1 -- 172.21.15.113:0/57184996 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fb8018070 con 0x7f8fbc10d350 2024-04-12T05:25:30.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.520+0000 7f8fc259d700 1 -- 172.21.15.113:0/57184996 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fbc107900 con 0x7f8fbc10d350 2024-04-12T05:25:30.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.521+0000 7f8fc259d700 1 -- 172.21.15.113:0/57184996 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fbc05f130 con 0x7f8fbc10d350 2024-04-12T05:25:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.521+0000 7f8fb27fc700 1 -- 172.21.15.113:0/57184996 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8fb801d420 con 0x7f8fbc10d350 2024-04-12T05:25:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.521+0000 7f8fb27fc700 1 --2- 172.21.15.113:0/57184996 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fa806caa0 0x7f8fa806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.522+0000 7f8fb27fc700 1 -- 172.21.15.113:0/57184996 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f8fb808d2e0 con 0x7f8fbc10d350 2024-04-12T05:25:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.522+0000 7f8fc159b700 1 --2- 172.21.15.113:0/57184996 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fa806caa0 0x7f8fa806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:30.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.522+0000 7f8fc159b700 1 --2- 172.21.15.113:0/57184996 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fa806caa0 0x7f8fa806ef60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f8fac004b60 tx=0x7f8fac009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:30.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.526+0000 7f8fb27fc700 1 -- 172.21.15.113:0/57184996 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8fb805ad90 con 0x7f8fbc10d350 2024-04-12T05:25:30.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.735+0000 7f8fc259d700 1 -- 172.21.15.113:0/57184996 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fbc1084e0 con 0x7f8fa806caa0 2024-04-12T05:25:30.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.736+0000 7f8fb27fc700 1 -- 172.21.15.113:0/57184996 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8fbc1084e0 con 0x7f8fa806caa0 2024-04-12T05:25:30.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.738+0000 7f8fc259d700 1 -- 172.21.15.113:0/57184996 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fa806caa0 msgr2=0x7f8fa806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:30.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.738+0000 7f8fc259d700 1 --2- 172.21.15.113:0/57184996 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fa806caa0 0x7f8fa806ef60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f8fac004b60 tx=0x7f8fac009040).stop 2024-04-12T05:25:30.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.738+0000 7f8fc259d700 1 -- 172.21.15.113:0/57184996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fbc10d350 msgr2=0x7f8fbc106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:30.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.738+0000 7f8fc259d700 1 --2- 172.21.15.113:0/57184996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fbc10d350 0x7f8fbc106df0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f8fb80099f0 tx=0x7f8fb8009cb0).stop 2024-04-12T05:25:30.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.739+0000 7f8fc259d700 1 -- 172.21.15.113:0/57184996 shutdown_connections 2024-04-12T05:25:30.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.739+0000 7f8fc259d700 1 --2- 172.21.15.113:0/57184996 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8fa806caa0 0x7f8fa806ef60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.739+0000 7f8fc259d700 1 --2- 172.21.15.113:0/57184996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc0fc5e0 0x7f8fbc10ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.739+0000 7f8fc259d700 1 --2- 172.21.15.113:0/57184996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fbc10d350 0x7f8fbc106df0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.739+0000 7f8fc259d700 1 -- 172.21.15.113:0/57184996 >> 172.21.15.113:0/57184996 conn(0x7f8fbc0f1910 msgr2=0x7f8fbc0fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:30.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.739+0000 7f8fc259d700 1 -- 172.21.15.113:0/57184996 shutdown_connections 2024-04-12T05:25:30.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.739+0000 7f8fc259d700 1 -- 172.21.15.113:0/57184996 wait complete. 2024-04-12T05:25:30.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.836+0000 7f65d8dbc700 1 -- 172.21.15.113:0/1449642661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65d40fd8f0 msgr2=0x7f65d40fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:30.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.836+0000 7f65d8dbc700 1 --2- 172.21.15.113:0/1449642661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65d40fd8f0 0x7f65d40fdcd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f65cc0047f0 tx=0x7f65cc009cb0).stop 2024-04-12T05:25:30.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.836+0000 7f65d8dbc700 1 -- 172.21.15.113:0/1449642661 shutdown_connections 2024-04-12T05:25:30.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.836+0000 7f65d8dbc700 1 --2- 172.21.15.113:0/1449642661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65d40fe250 0x7f65d40f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.836+0000 7f65d8dbc700 1 --2- 172.21.15.113:0/1449642661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65d40fd8f0 0x7f65d40fdcd0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.836+0000 7f65d8dbc700 1 -- 172.21.15.113:0/1449642661 >> 172.21.15.113:0/1449642661 conn(0x7f65d40f34b0 msgr2=0x7f65d40f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:30.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.836+0000 7f65d8dbc700 1 -- 172.21.15.113:0/1449642661 shutdown_connections 2024-04-12T05:25:30.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.836+0000 7f65d8dbc700 1 -- 172.21.15.113:0/1449642661 wait complete. 2024-04-12T05:25:30.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.837+0000 7f65d8dbc700 1 Processor -- start 2024-04-12T05:25:30.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.837+0000 7f65d8dbc700 1 -- start start 2024-04-12T05:25:30.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.837+0000 7f65d8dbc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65d40fd8f0 0x7f65d410e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:30.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.837+0000 7f65d8dbc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65d40fe250 0x7f65d410e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:30.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.837+0000 7f65d8dbc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d410ec00 con 0x7f65d40fd8f0 2024-04-12T05:25:30.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.837+0000 7f65d8dbc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d410ed40 con 0x7f65d40fe250 2024-04-12T05:25:30.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.837+0000 7f65d259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65d40fd8f0 0x7f65d410e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.837+0000 7f65d259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65d40fd8f0 0x7f65d410e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40532/0 (socket says 172.21.15.113:40532) 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.837+0000 7f65d259c700 1 -- 172.21.15.113:0/252115947 learned_addr learned my addr 172.21.15.113:0/252115947 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.837+0000 7f65d1d9b700 1 --2- 172.21.15.113:0/252115947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65d40fe250 0x7f65d410e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.838+0000 7f65d259c700 1 -- 172.21.15.113:0/252115947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65d40fe250 msgr2=0x7f65d410e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.838+0000 7f65d259c700 1 --2- 172.21.15.113:0/252115947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65d40fe250 0x7f65d410e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.838+0000 7f65d259c700 1 -- 172.21.15.113:0/252115947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc005040 con 0x7f65d40fd8f0 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.838+0000 7f65d259c700 1 --2- 172.21.15.113:0/252115947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65d40fd8f0 0x7f65d410e110 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f65cc009380 tx=0x7f65cc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.838+0000 7f65c37fe700 1 -- 172.21.15.113:0/252115947 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65cc018070 con 0x7f65d40fd8f0 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.838+0000 7f65c37fe700 1 -- 172.21.15.113:0/252115947 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65cc007e90 con 0x7f65d40fd8f0 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.838+0000 7f65d8dbc700 1 -- 172.21.15.113:0/252115947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d4108370 con 0x7f65d40fd8f0 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.838+0000 7f65c37fe700 1 -- 172.21.15.113:0/252115947 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65cc0123c0 con 0x7f65d40fd8f0 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.838+0000 7f65d8dbc700 1 -- 172.21.15.113:0/252115947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d41088d0 con 0x7f65d40fd8f0 2024-04-12T05:25:30.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.839+0000 7f65d8dbc700 1 -- 172.21.15.113:0/252115947 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d405f590 con 0x7f65d40fd8f0 2024-04-12T05:25:30.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.839+0000 7f65c37fe700 1 -- 172.21.15.113:0/252115947 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f65cc01d420 con 0x7f65d40fd8f0 2024-04-12T05:25:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.840+0000 7f65c37fe700 1 --2- 172.21.15.113:0/252115947 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f65bc06caa0 0x7f65bc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.840+0000 7f65c37fe700 1 -- 172.21.15.113:0/252115947 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f65cc08c9e0 con 0x7f65d40fd8f0 2024-04-12T05:25:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.840+0000 7f65d1d9b700 1 --2- 172.21.15.113:0/252115947 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f65bc06caa0 0x7f65bc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:30.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.840+0000 7f65d1d9b700 1 --2- 172.21.15.113:0/252115947 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f65bc06caa0 0x7f65bc06ef60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f65d4109a50 tx=0x7f65c800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:30.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:30.844+0000 7f65c37fe700 1 -- 172.21.15.113:0/252115947 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f65cc05a490 con 0x7f65d40fd8f0 2024-04-12T05:25:31.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.042+0000 7f65d8dbc700 1 -- 172.21.15.113:0/252115947 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f65d40ffc40 con 0x7f65bc06caa0 2024-04-12T05:25:31.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.052+0000 7f65c37fe700 1 -- 172.21.15.113:0/252115947 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f65d40ffc40 con 0x7f65bc06caa0 2024-04-12T05:25:31.055 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:25:31.055 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (13m) 6m ago 24m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:25:31.055 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (10m) 6m ago 24m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:25:31.055 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (10m) 5m ago 23m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:25:31.055 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (13m) 6m ago 24m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:25:31.055 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (6m) 6m ago 19m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:25:31.055 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (6m) 6m ago 19m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:25:31.055 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (6m) 5m ago 19m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:25:31.055 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (19m) 5m ago 19m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:25:31.055 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (14m) 6m ago 25m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:25:31.055 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (14m) 5m ago 23m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:25:31.056 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (11m) 6m ago 26m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:25:31.056 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (11m) 5m ago 23m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:25:31.056 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (13m) 6m ago 24m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:25:31.056 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (13m) 5m ago 23m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:25:31.056 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (10m) 6m ago 23m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:25:31.056 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (9m) 6m ago 22m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:25:31.056 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (9m) 6m ago 22m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:25:31.056 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (8m) 5m ago 21m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:25:31.056 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (8m) 5m ago 21m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:25:31.056 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (6m) 5m ago 20m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:25:31.056 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (13m) 5m ago 23m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:25:31.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.054+0000 7f65d8dbc700 1 -- 172.21.15.113:0/252115947 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f65bc06caa0 msgr2=0x7f65bc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.054+0000 7f65d8dbc700 1 --2- 172.21.15.113:0/252115947 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f65bc06caa0 0x7f65bc06ef60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f65d4109a50 tx=0x7f65c800b040).stop 2024-04-12T05:25:31.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.054+0000 7f65d8dbc700 1 -- 172.21.15.113:0/252115947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65d40fd8f0 msgr2=0x7f65d410e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.054+0000 7f65d8dbc700 1 --2- 172.21.15.113:0/252115947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65d40fd8f0 0x7f65d410e110 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f65cc009380 tx=0x7f65cc009cb0).stop 2024-04-12T05:25:31.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.054+0000 7f65d8dbc700 1 -- 172.21.15.113:0/252115947 shutdown_connections 2024-04-12T05:25:31.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.054+0000 7f65d8dbc700 1 --2- 172.21.15.113:0/252115947 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f65bc06caa0 0x7f65bc06ef60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.054+0000 7f65d8dbc700 1 --2- 172.21.15.113:0/252115947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65d40fd8f0 0x7f65d410e110 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.054+0000 7f65d8dbc700 1 --2- 172.21.15.113:0/252115947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65d40fe250 0x7f65d410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.054+0000 7f65d8dbc700 1 -- 172.21.15.113:0/252115947 >> 172.21.15.113:0/252115947 conn(0x7f65d40f34b0 msgr2=0x7f65d40fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:31.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.054+0000 7f65d8dbc700 1 -- 172.21.15.113:0/252115947 shutdown_connections 2024-04-12T05:25:31.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.055+0000 7f65d8dbc700 1 -- 172.21.15.113:0/252115947 wait complete. 2024-04-12T05:25:31.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.152+0000 7f90e4fe5700 1 -- 172.21.15.113:0/3255224370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90e00f9200 msgr2=0x7f90e0101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.152+0000 7f90e4fe5700 1 --2- 172.21.15.113:0/3255224370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90e00f9200 0x7f90e0101910 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f90d40047f0 tx=0x7f90d4009cb0).stop 2024-04-12T05:25:31.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.152+0000 7f90e4fe5700 1 -- 172.21.15.113:0/3255224370 shutdown_connections 2024-04-12T05:25:31.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.153+0000 7f90e4fe5700 1 --2- 172.21.15.113:0/3255224370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90e00f9200 0x7f90e0101910 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.153+0000 7f90e4fe5700 1 --2- 172.21.15.113:0/3255224370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e00f88a0 0x7f90e00f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.153+0000 7f90e4fe5700 1 -- 172.21.15.113:0/3255224370 >> 172.21.15.113:0/3255224370 conn(0x7f90e0070590 msgr2=0x7f90e00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:31.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.153+0000 7f90e4fe5700 1 -- 172.21.15.113:0/3255224370 shutdown_connections 2024-04-12T05:25:31.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.153+0000 7f90e4fe5700 1 -- 172.21.15.113:0/3255224370 wait complete. 2024-04-12T05:25:31.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.153+0000 7f90e4fe5700 1 Processor -- start 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.153+0000 7f90e4fe5700 1 -- start start 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90e4fe5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90e00f88a0 0x7f90e010e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90e4fe5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e00f9200 0x7f90e010e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90e4fe5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e010eb80 con 0x7f90e00f88a0 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90e4fe5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e010ecc0 con 0x7f90e00f9200 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90de59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90e00f88a0 0x7f90e010e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90de59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90e00f88a0 0x7f90e010e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40552/0 (socket says 172.21.15.113:40552) 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90de59c700 1 -- 172.21.15.113:0/1811853833 learned_addr learned my addr 172.21.15.113:0/1811853833 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90ddd9b700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e00f9200 0x7f90e010e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90de59c700 1 -- 172.21.15.113:0/1811853833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e00f9200 msgr2=0x7f90e010e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90de59c700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e00f9200 0x7f90e010e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90de59c700 1 -- 172.21.15.113:0/1811853833 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4005040 con 0x7f90e00f88a0 2024-04-12T05:25:31.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90ddd9b700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e00f9200 0x7f90e010e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:25:31.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90de59c700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90e00f88a0 0x7f90e010e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f90d000c790 tx=0x7f90d000cb50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:31.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90cf7fe700 1 -- 172.21.15.113:0/1811853833 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90d00106f0 con 0x7f90e00f88a0 2024-04-12T05:25:31.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90e4fe5700 1 -- 172.21.15.113:0/1811853833 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e0108350 con 0x7f90e00f88a0 2024-04-12T05:25:31.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90e4fe5700 1 -- 172.21.15.113:0/1811853833 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e0108890 con 0x7f90e00f88a0 2024-04-12T05:25:31.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90cf7fe700 1 -- 172.21.15.113:0/1811853833 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90d0010d00 con 0x7f90e00f88a0 2024-04-12T05:25:31.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.154+0000 7f90cf7fe700 1 -- 172.21.15.113:0/1811853833 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90d0012e50 con 0x7f90e00f88a0 2024-04-12T05:25:31.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.155+0000 7f90e4fe5700 1 -- 172.21.15.113:0/1811853833 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e005f590 con 0x7f90e00f88a0 2024-04-12T05:25:31.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.156+0000 7f90cf7fe700 1 -- 172.21.15.113:0/1811853833 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f90d0010850 con 0x7f90e00f88a0 2024-04-12T05:25:31.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.156+0000 7f90cf7fe700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f90c806caa0 0x7f90c806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:31.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.156+0000 7f90cf7fe700 1 -- 172.21.15.113:0/1811853833 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f90d008b6b0 con 0x7f90e00f88a0 2024-04-12T05:25:31.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.156+0000 7f90ddd9b700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f90c806caa0 0x7f90c806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:31.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.156+0000 7f90ddd9b700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f90c806caa0 0x7f90c806ef60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f90d4007cf0 tx=0x7f90d400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:31.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.160+0000 7f90cf7fe700 1 -- 172.21.15.113:0/1811853833 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f90d0059160 con 0x7f90e00f88a0 2024-04-12T05:25:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:31 smithi113 ceph-mon[103371]: pgmap v400: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:31.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.428+0000 7f90e4fe5700 1 -- 172.21.15.113:0/1811853833 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f90e0047540 con 0x7f90e00f88a0 2024-04-12T05:25:31.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.429+0000 7f90cf7fe700 1 -- 172.21.15.113:0/1811853833 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f90d0058cf0 con 0x7f90e00f88a0 2024-04-12T05:25:31.431 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:25:31.431 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:25:31.431 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:25:31.431 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:25:31.432 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:25:31.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.431+0000 7f90e4fe5700 1 -- 172.21.15.113:0/1811853833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f90c806caa0 msgr2=0x7f90c806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.431+0000 7f90e4fe5700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f90c806caa0 0x7f90c806ef60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f90d4007cf0 tx=0x7f90d400b040).stop 2024-04-12T05:25:31.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.431+0000 7f90e4fe5700 1 -- 172.21.15.113:0/1811853833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90e00f88a0 msgr2=0x7f90e010e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.431+0000 7f90e4fe5700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90e00f88a0 0x7f90e010e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f90d000c790 tx=0x7f90d000cb50).stop 2024-04-12T05:25:31.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.431+0000 7f90e4fe5700 1 -- 172.21.15.113:0/1811853833 shutdown_connections 2024-04-12T05:25:31.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.431+0000 7f90e4fe5700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f90c806caa0 0x7f90c806ef60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.431+0000 7f90e4fe5700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90e00f88a0 0x7f90e010e090 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.431+0000 7f90e4fe5700 1 --2- 172.21.15.113:0/1811853833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90e00f9200 0x7f90e010e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.431+0000 7f90e4fe5700 1 -- 172.21.15.113:0/1811853833 >> 172.21.15.113:0/1811853833 conn(0x7f90e0070590 msgr2=0x7f90e00fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:31.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.431+0000 7f90e4fe5700 1 -- 172.21.15.113:0/1811853833 shutdown_connections 2024-04-12T05:25:31.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.431+0000 7f90e4fe5700 1 -- 172.21.15.113:0/1811853833 wait complete. 2024-04-12T05:25:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:31 smithi192 ceph-mon[85304]: pgmap v400: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:31.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.528+0000 7f68d48bb700 1 -- 172.21.15.113:0/987041416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68d00fe250 msgr2=0x7f68d00f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.528+0000 7f68d48bb700 1 --2- 172.21.15.113:0/987041416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68d00fe250 0x7f68d00f7a30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f68c40047f0 tx=0x7f68c4009cb0).stop 2024-04-12T05:25:31.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.529+0000 7f68d48bb700 1 -- 172.21.15.113:0/987041416 shutdown_connections 2024-04-12T05:25:31.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.529+0000 7f68d48bb700 1 --2- 172.21.15.113:0/987041416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68d00fe250 0x7f68d00f7a30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.529+0000 7f68d48bb700 1 --2- 172.21.15.113:0/987041416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68d00fd8f0 0x7f68d00fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.529+0000 7f68d48bb700 1 -- 172.21.15.113:0/987041416 >> 172.21.15.113:0/987041416 conn(0x7f68d00f34b0 msgr2=0x7f68d00f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:31.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.529+0000 7f68d48bb700 1 -- 172.21.15.113:0/987041416 shutdown_connections 2024-04-12T05:25:31.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.529+0000 7f68d48bb700 1 -- 172.21.15.113:0/987041416 wait complete. 2024-04-12T05:25:31.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.529+0000 7f68d48bb700 1 Processor -- start 2024-04-12T05:25:31.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.529+0000 7f68d48bb700 1 -- start start 2024-04-12T05:25:31.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68d48bb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68d00fd8f0 0x7f68d010e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:31.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68d48bb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68d00fe250 0x7f68d010e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:31.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68d48bb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d010eb70 con 0x7f68d00fd8f0 2024-04-12T05:25:31.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68d48bb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d010ecb0 con 0x7f68d00fe250 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68ce59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68d00fd8f0 0x7f68d010e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68ce59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68d00fd8f0 0x7f68d010e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40562/0 (socket says 172.21.15.113:40562) 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68ce59c700 1 -- 172.21.15.113:0/3533278862 learned_addr learned my addr 172.21.15.113:0/3533278862 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68cdd9b700 1 --2- 172.21.15.113:0/3533278862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68d00fe250 0x7f68d010e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68ce59c700 1 -- 172.21.15.113:0/3533278862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68d00fe250 msgr2=0x7f68d010e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68ce59c700 1 --2- 172.21.15.113:0/3533278862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68d00fe250 0x7f68d010e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68ce59c700 1 -- 172.21.15.113:0/3533278862 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c4005040 con 0x7f68d00fd8f0 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68ce59c700 1 --2- 172.21.15.113:0/3533278862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68d00fd8f0 0x7f68d010e080 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f68c00041d0 tx=0x7f68c0009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68bf7fe700 1 -- 172.21.15.113:0/3533278862 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68c0007b40 con 0x7f68d00fd8f0 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68d48bb700 1 -- 172.21.15.113:0/3533278862 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d0108340 con 0x7f68d00fd8f0 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68bf7fe700 1 -- 172.21.15.113:0/3533278862 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68c000f040 con 0x7f68d00fd8f0 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68bf7fe700 1 -- 172.21.15.113:0/3533278862 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68c00147b0 con 0x7f68d00fd8f0 2024-04-12T05:25:31.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.530+0000 7f68d48bb700 1 -- 172.21.15.113:0/3533278862 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d0108880 con 0x7f68d00fd8f0 2024-04-12T05:25:31.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.531+0000 7f68d48bb700 1 -- 172.21.15.113:0/3533278862 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d005f590 con 0x7f68d00fd8f0 2024-04-12T05:25:31.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.532+0000 7f68bf7fe700 1 -- 172.21.15.113:0/3533278862 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f68c0009110 con 0x7f68d00fd8f0 2024-04-12T05:25:31.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.532+0000 7f68bf7fe700 1 --2- 172.21.15.113:0/3533278862 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f68b806caa0 0x7f68b806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:31.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.532+0000 7f68bf7fe700 1 -- 172.21.15.113:0/3533278862 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f68c008be40 con 0x7f68d00fd8f0 2024-04-12T05:25:31.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.532+0000 7f68cdd9b700 1 --2- 172.21.15.113:0/3533278862 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f68b806caa0 0x7f68b806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:31.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.533+0000 7f68cdd9b700 1 --2- 172.21.15.113:0/3533278862 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f68b806caa0 0x7f68b806ef60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f68d01099c0 tx=0x7f68c400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:31.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.537+0000 7f68bf7fe700 1 -- 172.21.15.113:0/3533278862 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f68c00598f0 con 0x7f68d00fd8f0 2024-04-12T05:25:31.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.766+0000 7f68d48bb700 1 -- 172.21.15.113:0/3533278862 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f68d0109800 con 0x7f68d00fd8f0 2024-04-12T05:25:31.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.766+0000 7f68bf7fe700 1 -- 172.21.15.113:0/3533278862 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f68c0059480 con 0x7f68d00fd8f0 2024-04-12T05:25:31.768 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:25:31.768 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:25:31.769 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:25:31.770 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:25:31.771 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:25:31.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.768+0000 7f68d48bb700 1 -- 172.21.15.113:0/3533278862 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f68b806caa0 msgr2=0x7f68b806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.768+0000 7f68d48bb700 1 --2- 172.21.15.113:0/3533278862 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f68b806caa0 0x7f68b806ef60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f68d01099c0 tx=0x7f68c400b040).stop 2024-04-12T05:25:31.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.768+0000 7f68d48bb700 1 -- 172.21.15.113:0/3533278862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68d00fd8f0 msgr2=0x7f68d010e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.768+0000 7f68d48bb700 1 --2- 172.21.15.113:0/3533278862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68d00fd8f0 0x7f68d010e080 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f68c00041d0 tx=0x7f68c0009f60).stop 2024-04-12T05:25:31.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.768+0000 7f68d48bb700 1 -- 172.21.15.113:0/3533278862 shutdown_connections 2024-04-12T05:25:31.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.768+0000 7f68d48bb700 1 --2- 172.21.15.113:0/3533278862 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f68b806caa0 0x7f68b806ef60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.768+0000 7f68d48bb700 1 --2- 172.21.15.113:0/3533278862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68d00fd8f0 0x7f68d010e080 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.768+0000 7f68d48bb700 1 --2- 172.21.15.113:0/3533278862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68d00fe250 0x7f68d010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.768+0000 7f68d48bb700 1 -- 172.21.15.113:0/3533278862 >> 172.21.15.113:0/3533278862 conn(0x7f68d00f34b0 msgr2=0x7f68d00fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:31.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.768+0000 7f68d48bb700 1 -- 172.21.15.113:0/3533278862 shutdown_connections 2024-04-12T05:25:31.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.768+0000 7f68d48bb700 1 -- 172.21.15.113:0/3533278862 wait complete. 2024-04-12T05:25:31.773 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:25:31.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.867+0000 7f3959412700 1 -- 172.21.15.113:0/540319558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39540fd150 msgr2=0x7f39540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.867+0000 7f3959412700 1 --2- 172.21.15.113:0/540319558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39540fd150 0x7f39540f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f393c0047f0 tx=0x7f393c009cb0).stop 2024-04-12T05:25:31.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.867+0000 7f3959412700 1 -- 172.21.15.113:0/540319558 shutdown_connections 2024-04-12T05:25:31.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.867+0000 7f3959412700 1 --2- 172.21.15.113:0/540319558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39540fd150 0x7f39540f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.867+0000 7f3959412700 1 --2- 172.21.15.113:0/540319558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39540fc7f0 0x7f39540fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:25:31.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.867+0000 7f3959412700 1 -- 172.21.15.113:0/540319558 >> 172.21.15.113:0/540319558 conn(0x7f39540f23b0 msgr2=0x7f39540f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:31.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.867+0000 7f3959412700 1 -- 172.21.15.113:0/540319558 shutdown_connections 2024-04-12T05:25:31.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.867+0000 7f3959412700 1 -- 172.21.15.113:0/540319558 wait complete. 2024-04-12T05:25:31.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.867+0000 7f3959412700 1 Processor -- start 2024-04-12T05:25:31.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f3959412700 1 -- start start 2024-04-12T05:25:31.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f3959412700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39540fc7f0 0x7f395410e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:31.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f3959412700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39540fd150 0x7f395410e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:31.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f3959412700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395410ebd0 con 0x7f39540fd150 2024-04-12T05:25:31.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f3959412700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39541081f0 con 0x7f39540fc7f0 2024-04-12T05:25:31.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f39527fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39540fd150 0x7f395410e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:31.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f39527fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39540fd150 0x7f395410e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40586/0 (socket says 172.21.15.113:40586) 2024-04-12T05:25:31.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f39527fc700 1 -- 172.21.15.113:0/2051410868 learned_addr learned my addr 172.21.15.113:0/2051410868 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:25:31.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f3952ffd700 1 --2- 172.21.15.113:0/2051410868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39540fc7f0 0x7f395410e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:31.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f39527fc700 1 -- 172.21.15.113:0/2051410868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39540fc7f0 msgr2=0x7f395410e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:31.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f39527fc700 1 --2- 172.21.15.113:0/2051410868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39540fc7f0 0x7f395410e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:31.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f39527fc700 1 -- 172.21.15.113:0/2051410868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c005040 con 0x7f39540fd150 2024-04-12T05:25:31.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.868+0000 7f39527fc700 1 --2- 172.21.15.113:0/2051410868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39540fd150 0x7f395410e540 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f393c0099f0 tx=0x7f393c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:31.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.869+0000 7f394bfff700 1 -- 172.21.15.113:0/2051410868 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f393c00e030 con 0x7f39540fd150 2024-04-12T05:25:31.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.869+0000 7f394bfff700 1 -- 172.21.15.113:0/2051410868 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f393c007e90 con 0x7f39540fd150 2024-04-12T05:25:31.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.869+0000 7f3959412700 1 -- 172.21.15.113:0/2051410868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3954108440 con 0x7f39540fd150 2024-04-12T05:25:31.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.869+0000 7f394bfff700 1 -- 172.21.15.113:0/2051410868 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f393c018070 con 0x7f39540fd150 2024-04-12T05:25:31.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.869+0000 7f3959412700 1 -- 172.21.15.113:0/2051410868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3954108920 con 0x7f39540fd150 2024-04-12T05:25:31.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.869+0000 7f3959412700 1 -- 172.21.15.113:0/2051410868 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3954060390 con 0x7f39540fd150 2024-04-12T05:25:31.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.870+0000 7f394bfff700 1 -- 172.21.15.113:0/2051410868 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f393c01d420 con 0x7f39540fd150 2024-04-12T05:25:31.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.870+0000 7f394bfff700 1 --2- 172.21.15.113:0/2051410868 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394006caa0 0x7f394006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:31.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.871+0000 7f394bfff700 1 -- 172.21.15.113:0/2051410868 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f393c08d240 con 0x7f39540fd150 2024-04-12T05:25:31.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.871+0000 7f3952ffd700 1 --2- 172.21.15.113:0/2051410868 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394006caa0 0x7f394006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:31.874 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.871+0000 7f3952ffd700 1 --2- 172.21.15.113:0/2051410868 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394006caa0 0x7f394006ef60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f394400b910 tx=0x7f3944008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:31.877 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:31.875+0000 7f394bfff700 1 -- 172.21.15.113:0/2051410868 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f393c05acf0 con 0x7f39540fd150 2024-04-12T05:25:32.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.079+0000 7f3959412700 1 -- 172.21.15.113:0/2051410868 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39541035f0 con 0x7f394006caa0 2024-04-12T05:25:32.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.080+0000 7f394bfff700 1 -- 172.21.15.113:0/2051410868 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f39541035f0 con 0x7f394006caa0 2024-04-12T05:25:32.082 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:25:32.083 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:25:32.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.082+0000 7f3959412700 1 -- 172.21.15.113:0/2051410868 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394006caa0 msgr2=0x7f394006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:32.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.082+0000 7f3959412700 1 --2- 172.21.15.113:0/2051410868 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394006caa0 0x7f394006ef60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f394400b910 tx=0x7f3944008040).stop 2024-04-12T05:25:32.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.082+0000 7f3959412700 1 -- 172.21.15.113:0/2051410868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39540fd150 msgr2=0x7f395410e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:32.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.082+0000 7f3959412700 1 --2- 172.21.15.113:0/2051410868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39540fd150 0x7f395410e540 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f393c0099f0 tx=0x7f393c009cb0).stop 2024-04-12T05:25:32.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.082+0000 7f3959412700 1 -- 172.21.15.113:0/2051410868 shutdown_connections 2024-04-12T05:25:32.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.082+0000 7f3959412700 1 --2- 172.21.15.113:0/2051410868 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394006caa0 0x7f394006ef60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:32.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.082+0000 7f3959412700 1 --2- 172.21.15.113:0/2051410868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39540fc7f0 0x7f395410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:32.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.082+0000 7f3959412700 1 --2- 172.21.15.113:0/2051410868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39540fd150 0x7f395410e540 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:32.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.082+0000 7f3959412700 1 -- 172.21.15.113:0/2051410868 >> 172.21.15.113:0/2051410868 conn(0x7f39540f23b0 msgr2=0x7f3954061340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:32.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.083+0000 7f3959412700 1 -- 172.21.15.113:0/2051410868 shutdown_connections 2024-04-12T05:25:32.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.083+0000 7f3959412700 1 -- 172.21.15.113:0/2051410868 wait complete. 2024-04-12T05:25:32.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.181+0000 7fc9c9c24700 1 -- 172.21.15.113:0/584595024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9c40fc890 msgr2=0x7fc9c40fcc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.181+0000 7fc9c9c24700 1 --2- 172.21.15.113:0/584595024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9c40fc890 0x7fc9c40fcc70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fc9b40047f0 tx=0x7fc9b4009cb0).stop 2024-04-12T05:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.181+0000 7fc9c9c24700 1 -- 172.21.15.113:0/584595024 shutdown_connections 2024-04-12T05:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.181+0000 7fc9c9c24700 1 --2- 172.21.15.113:0/584595024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9c40fd1f0 0x7fc9c40f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.181+0000 7fc9c9c24700 1 --2- 172.21.15.113:0/584595024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9c40fc890 0x7fc9c40fcc70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.181+0000 7fc9c9c24700 1 -- 172.21.15.113:0/584595024 >> 172.21.15.113:0/584595024 conn(0x7fc9c40f2490 msgr2=0x7fc9c40f48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.181+0000 7fc9c9c24700 1 -- 172.21.15.113:0/584595024 shutdown_connections 2024-04-12T05:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.181+0000 7fc9c9c24700 1 -- 172.21.15.113:0/584595024 wait complete. 2024-04-12T05:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.182+0000 7fc9c9c24700 1 Processor -- start 2024-04-12T05:25:32.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.182+0000 7fc9c9c24700 1 -- start start 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.182+0000 7fc9c9c24700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9c40fc890 0x7fc9c41090e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.182+0000 7fc9c9c24700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9c40fd1f0 0x7fc9c41095d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.182+0000 7fc9c9c24700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4109ac0 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.182+0000 7fc9c9c24700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4109c00 con 0x7fc9c40fc890 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.182+0000 7fc9bbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9c40fd1f0 0x7fc9c41095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.182+0000 7fc9bbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9c40fd1f0 0x7fc9c41095d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40608/0 (socket says 172.21.15.113:40608) 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.182+0000 7fc9bbfff700 1 -- 172.21.15.113:0/1899790044 learned_addr learned my addr 172.21.15.113:0/1899790044 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.182+0000 7fc9c37fe700 1 --2- 172.21.15.113:0/1899790044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9c40fc890 0x7fc9c41090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.183+0000 7fc9bbfff700 1 -- 172.21.15.113:0/1899790044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9c40fc890 msgr2=0x7fc9c41090e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.183+0000 7fc9bbfff700 1 --2- 172.21.15.113:0/1899790044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9c40fc890 0x7fc9c41090e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.183+0000 7fc9bbfff700 1 -- 172.21.15.113:0/1899790044 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4005040 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.183+0000 7fc9bbfff700 1 --2- 172.21.15.113:0/1899790044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9c40fd1f0 0x7fc9c41095d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fc9ac00c710 tx=0x7fc9ac007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:32.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.183+0000 7fc9c17fa700 1 -- 172.21.15.113:0/1899790044 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9ac00e680 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.183+0000 7fc9c17fa700 1 -- 172.21.15.113:0/1899790044 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9ac00ec90 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.183+0000 7fc9c9c24700 1 -- 172.21.15.113:0/1899790044 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c4109eb0 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.183+0000 7fc9c17fa700 1 -- 172.21.15.113:0/1899790044 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9ac013d40 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.183+0000 7fc9c9c24700 1 -- 172.21.15.113:0/1899790044 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c41124d0 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.184+0000 7fc9c9c24700 1 -- 172.21.15.113:0/1899790044 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c4060330 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.184+0000 7fc9c17fa700 1 -- 172.21.15.113:0/1899790044 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc9ac00e7e0 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.185+0000 7fc9c17fa700 1 --2- 172.21.15.113:0/1899790044 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9b006caa0 0x7fc9b006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:25:32.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.185+0000 7fc9c17fa700 1 -- 172.21.15.113:0/1899790044 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc9ac08bb50 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.185+0000 7fc9c37fe700 1 --2- 172.21.15.113:0/1899790044 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9b006caa0 0x7fc9b006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:25:32.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.185+0000 7fc9c37fe700 1 --2- 172.21.15.113:0/1899790044 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9b006caa0 0x7fc9b006ef60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc9b4007cf0 tx=0x7fc9b400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:25:32.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.189+0000 7fc9c17fa700 1 -- 172.21.15.113:0/1899790044 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc9ac055b70 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:32 smithi113 ceph-mon[103371]: from='client.34756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:32.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:32 smithi113 ceph-mon[103371]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:32.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:32 smithi113 ceph-mon[103371]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:32.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:32 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1811853833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:25:32.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:32 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3533278862' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:25:32.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:32 smithi192 ceph-mon[85304]: from='client.34756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:32.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:32 smithi192 ceph-mon[85304]: from='client.34760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:32.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:32 smithi192 ceph-mon[85304]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:32.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:32 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1811853833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:25:32.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:32 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3533278862' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:25:32.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.458+0000 7fc9c9c24700 1 -- 172.21.15.113:0/1899790044 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc9c4061220 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.458+0000 7fc9c17fa700 1 -- 172.21.15.113:0/1899790044 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc9ac059190 con 0x7fc9c40fd1f0 2024-04-12T05:25:32.461 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:25:32.461 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:25:32.461 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:25:32.461 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:25:32.461 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:25:32.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.460+0000 7fc9c9c24700 1 -- 172.21.15.113:0/1899790044 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9b006caa0 msgr2=0x7fc9b006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:32.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.460+0000 7fc9c9c24700 1 --2- 172.21.15.113:0/1899790044 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9b006caa0 0x7fc9b006ef60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc9b4007cf0 tx=0x7fc9b400b040).stop 2024-04-12T05:25:32.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.460+0000 7fc9c9c24700 1 -- 172.21.15.113:0/1899790044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9c40fd1f0 msgr2=0x7fc9c41095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:25:32.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.460+0000 7fc9c9c24700 1 --2- 172.21.15.113:0/1899790044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9c40fd1f0 0x7fc9c41095d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fc9ac00c710 tx=0x7fc9ac007ec0).stop 2024-04-12T05:25:32.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.460+0000 7fc9c9c24700 1 -- 172.21.15.113:0/1899790044 shutdown_connections 2024-04-12T05:25:32.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.460+0000 7fc9c9c24700 1 --2- 172.21.15.113:0/1899790044 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9b006caa0 0x7fc9b006ef60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:32.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.460+0000 7fc9c9c24700 1 --2- 172.21.15.113:0/1899790044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9c40fc890 0x7fc9c41090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:32.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.460+0000 7fc9c9c24700 1 --2- 172.21.15.113:0/1899790044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9c40fd1f0 0x7fc9c41095d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:25:32.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.460+0000 7fc9c9c24700 1 -- 172.21.15.113:0/1899790044 >> 172.21.15.113:0/1899790044 conn(0x7fc9c40f2490 msgr2=0x7fc9c40fac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:25:32.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.460+0000 7fc9c9c24700 1 -- 172.21.15.113:0/1899790044 shutdown_connections 2024-04-12T05:25:32.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:25:32.460+0000 7fc9c9c24700 1 -- 172.21.15.113:0/1899790044 wait complete. 2024-04-12T05:25:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:33 smithi113 ceph-mon[103371]: pgmap v401: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:33 smithi113 ceph-mon[103371]: from='client.34776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:33 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1899790044' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:25:33.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:33 smithi192 ceph-mon[85304]: pgmap v401: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:33.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:33 smithi192 ceph-mon[85304]: from='client.34776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:25:33.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:33 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1899790044' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:25:34.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:34 smithi192 ceph-mon[85304]: pgmap v402: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:34.674 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:34 smithi113 ceph-mon[103371]: pgmap v402: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:37.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:36 smithi113 ceph-mon[103371]: pgmap v403: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:37.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:25:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:36 smithi192 ceph-mon[85304]: pgmap v403: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:25:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:38 smithi113 ceph-mon[103371]: pgmap v404: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:38 smithi192 ceph-mon[85304]: pgmap v404: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:40 smithi113 ceph-mon[103371]: pgmap v405: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:41.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:40 smithi192 ceph-mon[85304]: pgmap v405: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:42 smithi113 ceph-mon[103371]: pgmap v406: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:43.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:42 smithi192 ceph-mon[85304]: pgmap v406: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:45 smithi113 ceph-mon[103371]: pgmap v407: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:45 smithi192 ceph-mon[85304]: pgmap v407: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:47 smithi113 ceph-mon[103371]: pgmap v408: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:47.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:47 smithi192 ceph-mon[85304]: pgmap v408: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:49 smithi113 ceph-mon[103371]: pgmap v409: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:49.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:49 smithi192 ceph-mon[85304]: pgmap v409: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:51 smithi113 ceph-mon[103371]: pgmap v410: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:51 smithi192 ceph-mon[85304]: pgmap v410: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:52.269 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:25:52.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:25:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:53 smithi113 ceph-mon[103371]: pgmap v411: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:53 smithi192 ceph-mon[85304]: pgmap v411: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:55 smithi113 ceph-mon[103371]: pgmap v412: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:55.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:55 smithi192 ceph-mon[85304]: pgmap v412: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:57 smithi113 ceph-mon[103371]: pgmap v413: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:57.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:57 smithi192 ceph-mon[85304]: pgmap v413: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:25:59 smithi113 ceph-mon[103371]: pgmap v414: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:25:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:25:59 smithi192 ceph-mon[85304]: pgmap v414: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:01 smithi113 ceph-mon[103371]: pgmap v415: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:01 smithi192 ceph-mon[85304]: pgmap v415: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:02.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.572+0000 7f4affd53700 1 -- 172.21.15.113:0/2188610011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af80fd150 msgr2=0x7f4af80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:02.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.572+0000 7f4affd53700 1 --2- 172.21.15.113:0/2188610011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af80fd150 0x7f4af80f6930 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4ae80047f0 tx=0x7f4ae8009cb0).stop 2024-04-12T05:26:02.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.572+0000 7f4affd53700 1 -- 172.21.15.113:0/2188610011 shutdown_connections 2024-04-12T05:26:02.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.572+0000 7f4affd53700 1 --2- 172.21.15.113:0/2188610011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af80fd150 0x7f4af80f6930 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:02.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.572+0000 7f4affd53700 1 --2- 172.21.15.113:0/2188610011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4af80fc7f0 0x7f4af80fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:02.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.572+0000 7f4affd53700 1 -- 172.21.15.113:0/2188610011 >> 172.21.15.113:0/2188610011 conn(0x7f4af80f23b0 msgr2=0x7f4af80f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:02.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.572+0000 7f4affd53700 1 -- 172.21.15.113:0/2188610011 shutdown_connections 2024-04-12T05:26:02.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.572+0000 7f4affd53700 1 -- 172.21.15.113:0/2188610011 wait complete. 2024-04-12T05:26:02.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.572+0000 7f4affd53700 1 Processor -- start 2024-04-12T05:26:02.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.572+0000 7f4affd53700 1 -- start start 2024-04-12T05:26:02.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4affd53700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af80fc7f0 0x7f4af8109160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:02.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4affd53700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4af80fd150 0x7f4af8109650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:02.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4affd53700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8109b40 con 0x7f4af80fc7f0 2024-04-12T05:26:02.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4affd53700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8109c80 con 0x7f4af80fd150 2024-04-12T05:26:02.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4afdaef700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af80fc7f0 0x7f4af8109160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:02.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4afdaef700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af80fc7f0 0x7f4af8109160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57486/0 (socket says 172.21.15.113:57486) 2024-04-12T05:26:02.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4afdaef700 1 -- 172.21.15.113:0/2645040504 learned_addr learned my addr 172.21.15.113:0/2645040504 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:02.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4afdaef700 1 -- 172.21.15.113:0/2645040504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4af80fd150 msgr2=0x7f4af8109650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:02.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4afd2ee700 1 --2- 172.21.15.113:0/2645040504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4af80fd150 0x7f4af8109650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:02.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4afdaef700 1 --2- 172.21.15.113:0/2645040504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4af80fd150 0x7f4af8109650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:02.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4afdaef700 1 -- 172.21.15.113:0/2645040504 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae8005040 con 0x7f4af80fc7f0 2024-04-12T05:26:02.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4afdaef700 1 --2- 172.21.15.113:0/2645040504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af80fc7f0 0x7f4af8109160 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f4af400c790 tx=0x7f4af4004330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:02.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4aeeffd700 1 -- 172.21.15.113:0/2645040504 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4af40106f0 con 0x7f4af80fc7f0 2024-04-12T05:26:02.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4aeeffd700 1 -- 172.21.15.113:0/2645040504 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4af4010d00 con 0x7f4af80fc7f0 2024-04-12T05:26:02.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4aeeffd700 1 -- 172.21.15.113:0/2645040504 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4af4012da0 con 0x7f4af80fc7f0 2024-04-12T05:26:02.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.573+0000 7f4affd53700 1 -- 172.21.15.113:0/2645040504 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af8109f30 con 0x7f4af80fc7f0 2024-04-12T05:26:02.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.574+0000 7f4affd53700 1 -- 172.21.15.113:0/2645040504 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af81124a0 con 0x7f4af80fc7f0 2024-04-12T05:26:02.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.574+0000 7f4affd53700 1 -- 172.21.15.113:0/2645040504 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af8060370 con 0x7f4af80fc7f0 2024-04-12T05:26:02.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.575+0000 7f4aeeffd700 1 -- 172.21.15.113:0/2645040504 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4af4012400 con 0x7f4af80fc7f0 2024-04-12T05:26:02.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.576+0000 7f4aeeffd700 1 --2- 172.21.15.113:0/2645040504 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ae406caa0 0x7f4ae406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:02.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.576+0000 7f4aeeffd700 1 -- 172.21.15.113:0/2645040504 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f4af408bf90 con 0x7f4af80fc7f0 2024-04-12T05:26:02.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.576+0000 7f4afd2ee700 1 --2- 172.21.15.113:0/2645040504 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ae406caa0 0x7f4ae406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:02.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.576+0000 7f4afd2ee700 1 --2- 172.21.15.113:0/2645040504 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ae406caa0 0x7f4ae406ef60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f4ae8000c00 tx=0x7f4ae800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:02.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.580+0000 7f4aeeffd700 1 -- 172.21.15.113:0/2645040504 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4af4055fb0 con 0x7f4af80fc7f0 2024-04-12T05:26:02.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.787+0000 7f4affd53700 1 -- 172.21.15.113:0/2645040504 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4af8059120 con 0x7f4ae406caa0 2024-04-12T05:26:02.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.788+0000 7f4aeeffd700 1 -- 172.21.15.113:0/2645040504 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4af8059120 con 0x7f4ae406caa0 2024-04-12T05:26:02.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.790+0000 7f4affd53700 1 -- 172.21.15.113:0/2645040504 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ae406caa0 msgr2=0x7f4ae406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:02.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.790+0000 7f4affd53700 1 --2- 172.21.15.113:0/2645040504 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ae406caa0 0x7f4ae406ef60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f4ae8000c00 tx=0x7f4ae800b040).stop 2024-04-12T05:26:02.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.790+0000 7f4affd53700 1 -- 172.21.15.113:0/2645040504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af80fc7f0 msgr2=0x7f4af8109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:02.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.790+0000 7f4affd53700 1 --2- 172.21.15.113:0/2645040504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af80fc7f0 0x7f4af8109160 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f4af400c790 tx=0x7f4af4004330).stop 2024-04-12T05:26:02.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.790+0000 7f4affd53700 1 -- 172.21.15.113:0/2645040504 shutdown_connections 2024-04-12T05:26:02.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.790+0000 7f4affd53700 1 --2- 172.21.15.113:0/2645040504 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ae406caa0 0x7f4ae406ef60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:02.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.790+0000 7f4affd53700 1 --2- 172.21.15.113:0/2645040504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af80fc7f0 0x7f4af8109160 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:02.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.790+0000 7f4affd53700 1 --2- 172.21.15.113:0/2645040504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4af80fd150 0x7f4af8109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:02.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.790+0000 7f4affd53700 1 -- 172.21.15.113:0/2645040504 >> 172.21.15.113:0/2645040504 conn(0x7f4af80f23b0 msgr2=0x7f4af80fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:02.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.790+0000 7f4affd53700 1 -- 172.21.15.113:0/2645040504 shutdown_connections 2024-04-12T05:26:02.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.790+0000 7f4affd53700 1 -- 172.21.15.113:0/2645040504 wait complete. 2024-04-12T05:26:02.802 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:26:02.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.890+0000 7f64f373e700 1 -- 172.21.15.113:0/2793280796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64ec0fc7f0 msgr2=0x7f64ec0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:02.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.890+0000 7f64f373e700 1 --2- 172.21.15.113:0/2793280796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64ec0fc7f0 0x7f64ec0fcbd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f64e80047f0 tx=0x7f64e8009cb0).stop 2024-04-12T05:26:02.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.890+0000 7f64f373e700 1 -- 172.21.15.113:0/2793280796 shutdown_connections 2024-04-12T05:26:02.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.890+0000 7f64f373e700 1 --2- 172.21.15.113:0/2793280796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64ec0fd150 0x7f64ec0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:02.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.890+0000 7f64f373e700 1 --2- 172.21.15.113:0/2793280796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64ec0fc7f0 0x7f64ec0fcbd0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:02.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.890+0000 7f64f373e700 1 -- 172.21.15.113:0/2793280796 >> 172.21.15.113:0/2793280796 conn(0x7f64ec070590 msgr2=0x7f64ec0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:02.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.890+0000 7f64f373e700 1 -- 172.21.15.113:0/2793280796 shutdown_connections 2024-04-12T05:26:02.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.890+0000 7f64f373e700 1 -- 172.21.15.113:0/2793280796 wait complete. 2024-04-12T05:26:02.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.891+0000 7f64f373e700 1 Processor -- start 2024-04-12T05:26:02.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.891+0000 7f64f373e700 1 -- start start 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.891+0000 7f64f373e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64ec0fc7f0 0x7f64ec10cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.891+0000 7f64f373e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64ec0fd150 0x7f64ec10d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.891+0000 7f64f373e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec10da20 con 0x7f64ec0fc7f0 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.891+0000 7f64f373e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec10db60 con 0x7f64ec0fd150 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.891+0000 7f64f14da700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64ec0fc7f0 0x7f64ec10cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.891+0000 7f64f14da700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64ec0fc7f0 0x7f64ec10cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57512/0 (socket says 172.21.15.113:57512) 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.891+0000 7f64f14da700 1 -- 172.21.15.113:0/3384354600 learned_addr learned my addr 172.21.15.113:0/3384354600 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.891+0000 7f64f0cd9700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64ec0fd150 0x7f64ec10d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.892+0000 7f64f14da700 1 -- 172.21.15.113:0/3384354600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64ec0fd150 msgr2=0x7f64ec10d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.892+0000 7f64f14da700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64ec0fd150 0x7f64ec10d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.892+0000 7f64f14da700 1 -- 172.21.15.113:0/3384354600 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e8005040 con 0x7f64ec0fc7f0 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.892+0000 7f64f0cd9700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64ec0fd150 0x7f64ec10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:26:02.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.892+0000 7f64f14da700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64ec0fc7f0 0x7f64ec10cf30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f64e8005010 tx=0x7f64e8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:02.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.892+0000 7f64e27fc700 1 -- 172.21.15.113:0/3384354600 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64e800e030 con 0x7f64ec0fc7f0 2024-04-12T05:26:02.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.892+0000 7f64f373e700 1 -- 172.21.15.113:0/3384354600 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ec107190 con 0x7f64ec0fc7f0 2024-04-12T05:26:02.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.892+0000 7f64e27fc700 1 -- 172.21.15.113:0/3384354600 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64e8007e90 con 0x7f64ec0fc7f0 2024-04-12T05:26:02.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.892+0000 7f64e27fc700 1 -- 172.21.15.113:0/3384354600 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64e8018070 con 0x7f64ec0fc7f0 2024-04-12T05:26:02.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.892+0000 7f64f373e700 1 -- 172.21.15.113:0/3384354600 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ec1076f0 con 0x7f64ec0fc7f0 2024-04-12T05:26:02.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.893+0000 7f64f373e700 1 -- 172.21.15.113:0/3384354600 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ec05f590 con 0x7f64ec0fc7f0 2024-04-12T05:26:02.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.893+0000 7f64e27fc700 1 -- 172.21.15.113:0/3384354600 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f64e801d6e0 con 0x7f64ec0fc7f0 2024-04-12T05:26:02.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.894+0000 7f64e27fc700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64d806ca30 0x7f64d806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:02.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.894+0000 7f64e27fc700 1 -- 172.21.15.113:0/3384354600 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f64e808d100 con 0x7f64ec0fc7f0 2024-04-12T05:26:02.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.894+0000 7f64f0cd9700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64d806ca30 0x7f64d806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:02.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.894+0000 7f64f0cd9700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64d806ca30 0x7f64d806eef0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f64ec108870 tx=0x7f64dc009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:02.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:02.898+0000 7f64e27fc700 1 -- 172.21.15.113:0/3384354600 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f64e805a920 con 0x7f64ec0fc7f0 2024-04-12T05:26:03.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.106+0000 7f64f373e700 1 -- 172.21.15.113:0/3384354600 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f64ec0feb40 con 0x7f64d806ca30 2024-04-12T05:26:03.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.107+0000 7f64e27fc700 1 -- 172.21.15.113:0/3384354600 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f64ec0feb40 con 0x7f64d806ca30 2024-04-12T05:26:03.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.109+0000 7f64f373e700 1 -- 172.21.15.113:0/3384354600 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64d806ca30 msgr2=0x7f64d806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.109+0000 7f64f373e700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64d806ca30 0x7f64d806eef0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f64ec108870 tx=0x7f64dc009da0).stop 2024-04-12T05:26:03.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.109+0000 7f64f373e700 1 -- 172.21.15.113:0/3384354600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64ec0fc7f0 msgr2=0x7f64ec10cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.109+0000 7f64f373e700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64ec0fc7f0 0x7f64ec10cf30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f64e8005010 tx=0x7f64e8009cb0).stop 2024-04-12T05:26:03.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.109+0000 7f64f373e700 1 -- 172.21.15.113:0/3384354600 shutdown_connections 2024-04-12T05:26:03.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.109+0000 7f64f373e700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64d806ca30 0x7f64d806eef0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.109+0000 7f64f373e700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64ec0fc7f0 0x7f64ec10cf30 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.109+0000 7f64f373e700 1 --2- 172.21.15.113:0/3384354600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64ec0fd150 0x7f64ec10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.109+0000 7f64f373e700 1 -- 172.21.15.113:0/3384354600 >> 172.21.15.113:0/3384354600 conn(0x7f64ec070590 msgr2=0x7f64ec0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:03.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.110+0000 7f64f373e700 1 -- 172.21.15.113:0/3384354600 shutdown_connections 2024-04-12T05:26:03.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.110+0000 7f64f373e700 1 -- 172.21.15.113:0/3384354600 wait complete. 2024-04-12T05:26:03.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.207+0000 7f1b57de2700 1 -- 172.21.15.113:0/1770555433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b500fe250 msgr2=0x7f1b500f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.207+0000 7f1b57de2700 1 --2- 172.21.15.113:0/1770555433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b500fe250 0x7f1b500f7a30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f1b440047f0 tx=0x7f1b44009cb0).stop 2024-04-12T05:26:03.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.207+0000 7f1b57de2700 1 -- 172.21.15.113:0/1770555433 shutdown_connections 2024-04-12T05:26:03.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.207+0000 7f1b57de2700 1 --2- 172.21.15.113:0/1770555433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b500fe250 0x7f1b500f7a30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.207+0000 7f1b57de2700 1 --2- 172.21.15.113:0/1770555433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b500fd8f0 0x7f1b500fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.207+0000 7f1b57de2700 1 -- 172.21.15.113:0/1770555433 >> 172.21.15.113:0/1770555433 conn(0x7f1b500f34b0 msgr2=0x7f1b500f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.207+0000 7f1b57de2700 1 -- 172.21.15.113:0/1770555433 shutdown_connections 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.207+0000 7f1b57de2700 1 -- 172.21.15.113:0/1770555433 wait complete. 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b57de2700 1 Processor -- start 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b57de2700 1 -- start start 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b57de2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b500fd8f0 0x7f1b5006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b57de2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b500fe250 0x7f1b5006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b57de2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5006bab0 con 0x7f1b500fe250 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b57de2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5006bbf0 con 0x7f1b500fd8f0 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b5537d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b500fe250 0x7f1b5006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b5537d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b500fe250 0x7f1b5006b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57536/0 (socket says 172.21.15.113:57536) 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b5537d700 1 -- 172.21.15.113:0/224354112 learned_addr learned my addr 172.21.15.113:0/224354112 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b55b7e700 1 --2- 172.21.15.113:0/224354112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b500fd8f0 0x7f1b5006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b5537d700 1 -- 172.21.15.113:0/224354112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b500fd8f0 msgr2=0x7f1b5006b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b5537d700 1 --2- 172.21.15.113:0/224354112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b500fd8f0 0x7f1b5006b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b5537d700 1 -- 172.21.15.113:0/224354112 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44005040 con 0x7f1b500fe250 2024-04-12T05:26:03.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.208+0000 7f1b5537d700 1 --2- 172.21.15.113:0/224354112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b500fe250 0x7f1b5006b5c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f1b440099f0 tx=0x7f1b44009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:03.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.209+0000 7f1b4effd700 1 -- 172.21.15.113:0/224354112 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b4400e030 con 0x7f1b500fe250 2024-04-12T05:26:03.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.209+0000 7f1b57de2700 1 -- 172.21.15.113:0/224354112 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5006be40 con 0x7f1b500fe250 2024-04-12T05:26:03.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.209+0000 7f1b4effd700 1 -- 172.21.15.113:0/224354112 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b44007e90 con 0x7f1b500fe250 2024-04-12T05:26:03.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.209+0000 7f1b4effd700 1 -- 172.21.15.113:0/224354112 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b44018070 con 0x7f1b500fe250 2024-04-12T05:26:03.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.209+0000 7f1b57de2700 1 -- 172.21.15.113:0/224354112 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b50114600 con 0x7f1b500fe250 2024-04-12T05:26:03.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.209+0000 7f1b57de2700 1 -- 172.21.15.113:0/224354112 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5005f590 con 0x7f1b500fe250 2024-04-12T05:26:03.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.210+0000 7f1b4effd700 1 -- 172.21.15.113:0/224354112 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1b4401d420 con 0x7f1b500fe250 2024-04-12T05:26:03.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.210+0000 7f1b4effd700 1 --2- 172.21.15.113:0/224354112 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b3806caa0 0x7f1b3806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:03.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.210+0000 7f1b4effd700 1 -- 172.21.15.113:0/224354112 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1b4408d2f0 con 0x7f1b500fe250 2024-04-12T05:26:03.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.210+0000 7f1b55b7e700 1 --2- 172.21.15.113:0/224354112 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b3806caa0 0x7f1b3806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:03.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.211+0000 7f1b55b7e700 1 --2- 172.21.15.113:0/224354112 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b3806caa0 0x7f1b3806ef60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f1b480041d0 tx=0x7f1b48009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:03.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.215+0000 7f1b4effd700 1 -- 172.21.15.113:0/224354112 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1b44057310 con 0x7f1b500fe250 2024-04-12T05:26:03.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.411+0000 7f1b57de2700 1 -- 172.21.15.113:0/224354112 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b50059120 con 0x7f1b3806caa0 2024-04-12T05:26:03.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.421+0000 7f1b4effd700 1 -- 172.21.15.113:0/224354112 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1b50059120 con 0x7f1b3806caa0 2024-04-12T05:26:03.423 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (13m) 6m ago 25m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (11m) 6m ago 25m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (11m) 6m ago 24m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (13m) 6m ago 24m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (7m) 6m ago 19m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (6m) 6m ago 19m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (6m) 6m ago 19m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (19m) 6m ago 19m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (15m) 6m ago 26m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (14m) 6m ago 24m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (12m) 6m ago 26m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (11m) 6m ago 24m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (14m) 6m ago 24m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (14m) 6m ago 24m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:26:03.424 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (10m) 6m ago 23m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:26:03.425 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (10m) 6m ago 23m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:26:03.425 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (9m) 6m ago 22m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:26:03.425 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (9m) 6m ago 22m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:26:03.425 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (8m) 6m ago 21m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:26:03.425 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (7m) 6m ago 21m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:26:03.425 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (14m) 6m ago 24m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:26:03.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:03 smithi113 ceph-mon[103371]: pgmap v416: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:03.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.423+0000 7f1b57de2700 1 -- 172.21.15.113:0/224354112 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b3806caa0 msgr2=0x7f1b3806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.423+0000 7f1b57de2700 1 --2- 172.21.15.113:0/224354112 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b3806caa0 0x7f1b3806ef60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f1b480041d0 tx=0x7f1b48009040).stop 2024-04-12T05:26:03.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.423+0000 7f1b57de2700 1 -- 172.21.15.113:0/224354112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b500fe250 msgr2=0x7f1b5006b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.423+0000 7f1b57de2700 1 --2- 172.21.15.113:0/224354112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b500fe250 0x7f1b5006b5c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f1b440099f0 tx=0x7f1b44009cb0).stop 2024-04-12T05:26:03.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.423+0000 7f1b57de2700 1 -- 172.21.15.113:0/224354112 shutdown_connections 2024-04-12T05:26:03.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.423+0000 7f1b57de2700 1 --2- 172.21.15.113:0/224354112 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b3806caa0 0x7f1b3806ef60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.423+0000 7f1b57de2700 1 --2- 172.21.15.113:0/224354112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b500fd8f0 0x7f1b5006b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.423+0000 7f1b57de2700 1 --2- 172.21.15.113:0/224354112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b500fe250 0x7f1b5006b5c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.423+0000 7f1b57de2700 1 -- 172.21.15.113:0/224354112 >> 172.21.15.113:0/224354112 conn(0x7f1b500f34b0 msgr2=0x7f1b500fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:03.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.423+0000 7f1b57de2700 1 -- 172.21.15.113:0/224354112 shutdown_connections 2024-04-12T05:26:03.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.423+0000 7f1b57de2700 1 -- 172.21.15.113:0/224354112 wait complete. 2024-04-12T05:26:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:03 smithi192 ceph-mon[85304]: pgmap v416: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:03.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.520+0000 7fb2e471a700 1 -- 172.21.15.113:0/111137108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2dc0fd8f0 msgr2=0x7fb2dc0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.520+0000 7fb2e471a700 1 --2- 172.21.15.113:0/111137108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2dc0fd8f0 0x7fb2dc0fdcd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fb2d00047f0 tx=0x7fb2d0009cb0).stop 2024-04-12T05:26:03.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.520+0000 7fb2e471a700 1 -- 172.21.15.113:0/111137108 shutdown_connections 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.520+0000 7fb2e471a700 1 --2- 172.21.15.113:0/111137108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2dc0fe250 0x7fb2dc0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.520+0000 7fb2e471a700 1 --2- 172.21.15.113:0/111137108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2dc0fd8f0 0x7fb2dc0fdcd0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.520+0000 7fb2e471a700 1 -- 172.21.15.113:0/111137108 >> 172.21.15.113:0/111137108 conn(0x7fb2dc0f34b0 msgr2=0x7fb2dc0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.520+0000 7fb2e471a700 1 -- 172.21.15.113:0/111137108 shutdown_connections 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.520+0000 7fb2e471a700 1 -- 172.21.15.113:0/111137108 wait complete. 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.521+0000 7fb2e471a700 1 Processor -- start 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.521+0000 7fb2e471a700 1 -- start start 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.521+0000 7fb2e471a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2dc0fd8f0 0x7fb2dc10e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.521+0000 7fb2e471a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2dc0fe250 0x7fb2dc10e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.521+0000 7fb2e471a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc10ec80 con 0x7fb2dc0fe250 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.521+0000 7fb2e471a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc1082a0 con 0x7fb2dc0fd8f0 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.521+0000 7fb2e1cb5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2dc0fe250 0x7fb2dc10e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.521+0000 7fb2e24b6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2dc0fd8f0 0x7fb2dc10e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:03.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.521+0000 7fb2e1cb5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2dc0fe250 0x7fb2dc10e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57548/0 (socket says 172.21.15.113:57548) 2024-04-12T05:26:03.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.521+0000 7fb2e1cb5700 1 -- 172.21.15.113:0/165708010 learned_addr learned my addr 172.21.15.113:0/165708010 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:03.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.522+0000 7fb2e1cb5700 1 -- 172.21.15.113:0/165708010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2dc0fd8f0 msgr2=0x7fb2dc10e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.522+0000 7fb2e1cb5700 1 --2- 172.21.15.113:0/165708010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2dc0fd8f0 0x7fb2dc10e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.522+0000 7fb2e1cb5700 1 -- 172.21.15.113:0/165708010 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0005040 con 0x7fb2dc0fe250 2024-04-12T05:26:03.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.522+0000 7fb2e1cb5700 1 --2- 172.21.15.113:0/165708010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2dc0fe250 0x7fb2dc10e5f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb2d800b480 tx=0x7fb2d80090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:03.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.522+0000 7fb2cf7fe700 1 -- 172.21.15.113:0/165708010 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2d8002500 con 0x7fb2dc0fe250 2024-04-12T05:26:03.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.522+0000 7fb2cf7fe700 1 -- 172.21.15.113:0/165708010 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2d8002b10 con 0x7fb2dc0fe250 2024-04-12T05:26:03.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.522+0000 7fb2cf7fe700 1 -- 172.21.15.113:0/165708010 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2d8012c60 con 0x7fb2dc0fe250 2024-04-12T05:26:03.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.522+0000 7fb2e471a700 1 -- 172.21.15.113:0/165708010 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc108550 con 0x7fb2dc0fe250 2024-04-12T05:26:03.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.522+0000 7fb2e471a700 1 -- 172.21.15.113:0/165708010 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc108a90 con 0x7fb2dc0fe250 2024-04-12T05:26:03.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.523+0000 7fb2e471a700 1 -- 172.21.15.113:0/165708010 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc05f590 con 0x7fb2dc0fe250 2024-04-12T05:26:03.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.523+0000 7fb2cf7fe700 1 -- 172.21.15.113:0/165708010 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb2d8002660 con 0x7fb2dc0fe250 2024-04-12T05:26:03.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.524+0000 7fb2cf7fe700 1 --2- 172.21.15.113:0/165708010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2c806cab0 0x7fb2c806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:03.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.524+0000 7fb2cf7fe700 1 -- 172.21.15.113:0/165708010 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb2d808be50 con 0x7fb2dc0fe250 2024-04-12T05:26:03.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.524+0000 7fb2e24b6700 1 --2- 172.21.15.113:0/165708010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2c806cab0 0x7fb2c806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:03.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.524+0000 7fb2e24b6700 1 --2- 172.21.15.113:0/165708010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2c806cab0 0x7fb2c806ef70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fb2d0004f10 tx=0x7fb2d0009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:03.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.528+0000 7fb2cf7fe700 1 -- 172.21.15.113:0/165708010 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb2d8059900 con 0x7fb2dc0fe250 2024-04-12T05:26:03.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.795+0000 7fb2e471a700 1 -- 172.21.15.113:0/165708010 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb2dc109780 con 0x7fb2dc0fe250 2024-04-12T05:26:03.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.795+0000 7fb2cf7fe700 1 -- 172.21.15.113:0/165708010 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fb2d8059490 con 0x7fb2dc0fe250 2024-04-12T05:26:03.798 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:26:03.798 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:26:03.798 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:26:03.798 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:26:03.798 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:26:03.798 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:26:03.798 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:26:03.798 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:26:03.798 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:26:03.799 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:26:03.799 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:26:03.799 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:26:03.799 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:26:03.799 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:26:03.799 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:26:03.799 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:26:03.799 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:26:03.799 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:26:03.799 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:26:03.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.797+0000 7fb2e471a700 1 -- 172.21.15.113:0/165708010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2c806cab0 msgr2=0x7fb2c806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.797+0000 7fb2e471a700 1 --2- 172.21.15.113:0/165708010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2c806cab0 0x7fb2c806ef70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fb2d0004f10 tx=0x7fb2d0009cb0).stop 2024-04-12T05:26:03.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.797+0000 7fb2e471a700 1 -- 172.21.15.113:0/165708010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2dc0fe250 msgr2=0x7fb2dc10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.797+0000 7fb2e471a700 1 --2- 172.21.15.113:0/165708010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2dc0fe250 0x7fb2dc10e5f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb2d800b480 tx=0x7fb2d80090f0).stop 2024-04-12T05:26:03.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.797+0000 7fb2e471a700 1 -- 172.21.15.113:0/165708010 shutdown_connections 2024-04-12T05:26:03.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.797+0000 7fb2e471a700 1 --2- 172.21.15.113:0/165708010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb2c806cab0 0x7fb2c806ef70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.797+0000 7fb2e471a700 1 --2- 172.21.15.113:0/165708010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2dc0fd8f0 0x7fb2dc10e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.797+0000 7fb2e471a700 1 --2- 172.21.15.113:0/165708010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2dc0fe250 0x7fb2dc10e5f0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.797+0000 7fb2e471a700 1 -- 172.21.15.113:0/165708010 >> 172.21.15.113:0/165708010 conn(0x7fb2dc0f34b0 msgr2=0x7fb2dc0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:03.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.797+0000 7fb2e471a700 1 -- 172.21.15.113:0/165708010 shutdown_connections 2024-04-12T05:26:03.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.798+0000 7fb2e471a700 1 -- 172.21.15.113:0/165708010 wait complete. 2024-04-12T05:26:03.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.894+0000 7f8705aae700 1 -- 172.21.15.113:0/476911938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87000fc7f0 msgr2=0x7f87000fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.894+0000 7f8705aae700 1 --2- 172.21.15.113:0/476911938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87000fc7f0 0x7f87000fcbd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f86f00047f0 tx=0x7f86f0009cb0).stop 2024-04-12T05:26:03.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.895+0000 7f8705aae700 1 -- 172.21.15.113:0/476911938 shutdown_connections 2024-04-12T05:26:03.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.895+0000 7f8705aae700 1 --2- 172.21.15.113:0/476911938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87000fd150 0x7f87000f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:26:03.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.895+0000 7f8705aae700 1 --2- 172.21.15.113:0/476911938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87000fc7f0 0x7f87000fcbd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.895+0000 7f8705aae700 1 -- 172.21.15.113:0/476911938 >> 172.21.15.113:0/476911938 conn(0x7f8700070590 msgr2=0x7f87000f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:03.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.895+0000 7f8705aae700 1 -- 172.21.15.113:0/476911938 shutdown_connections 2024-04-12T05:26:03.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.895+0000 7f8705aae700 1 -- 172.21.15.113:0/476911938 wait complete. 2024-04-12T05:26:03.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.895+0000 7f8705aae700 1 Processor -- start 2024-04-12T05:26:03.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f8705aae700 1 -- start start 2024-04-12T05:26:03.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f8705aae700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87000fc7f0 0x7f870010cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:03.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f8705aae700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87000fd150 0x7f870010d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:03.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f8705aae700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870010daf0 con 0x7f87000fc7f0 2024-04-12T05:26:03.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f8705aae700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8700107110 con 0x7f87000fd150 2024-04-12T05:26:03.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f86ff7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87000fc7f0 0x7f870010cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:03.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f86ff7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87000fc7f0 0x7f870010cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57564/0 (socket says 172.21.15.113:57564) 2024-04-12T05:26:03.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f86ff7fe700 1 -- 172.21.15.113:0/2219011053 learned_addr learned my addr 172.21.15.113:0/2219011053 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:03.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f86feffd700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87000fd150 0x7f870010d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:03.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f86ff7fe700 1 -- 172.21.15.113:0/2219011053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87000fd150 msgr2=0x7f870010d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:03.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f86ff7fe700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87000fd150 0x7f870010d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:03.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f86ff7fe700 1 -- 172.21.15.113:0/2219011053 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f0005040 con 0x7f87000fc7f0 2024-04-12T05:26:03.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f86feffd700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87000fd150 0x7f870010d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:26:03.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.896+0000 7f86ff7fe700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87000fc7f0 0x7f870010cf70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f86f000b120 tx=0x7f86f0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:03.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.897+0000 7f86fcf74700 1 -- 172.21.15.113:0/2219011053 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86f000e030 con 0x7f87000fc7f0 2024-04-12T05:26:03.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.897+0000 7f8705aae700 1 -- 172.21.15.113:0/2219011053 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8700107360 con 0x7f87000fc7f0 2024-04-12T05:26:03.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.897+0000 7f86fcf74700 1 -- 172.21.15.113:0/2219011053 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86f0007e90 con 0x7f87000fc7f0 2024-04-12T05:26:03.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.897+0000 7f86fcf74700 1 -- 172.21.15.113:0/2219011053 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86f0018070 con 0x7f87000fc7f0 2024-04-12T05:26:03.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.897+0000 7f8705aae700 1 -- 172.21.15.113:0/2219011053 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8700107840 con 0x7f87000fc7f0 2024-04-12T05:26:03.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.898+0000 7f8705aae700 1 -- 172.21.15.113:0/2219011053 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870005f590 con 0x7f87000fc7f0 2024-04-12T05:26:03.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.898+0000 7f86fcf74700 1 -- 172.21.15.113:0/2219011053 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f86f001d420 con 0x7f87000fc7f0 2024-04-12T05:26:03.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.899+0000 7f86fcf74700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f86ec06caa0 0x7f86ec06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:03.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.899+0000 7f86feffd700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f86ec06caa0 0x7f86ec06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:03.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.899+0000 7f86fcf74700 1 -- 172.21.15.113:0/2219011053 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f86f008d300 con 0x7f87000fc7f0 2024-04-12T05:26:03.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.899+0000 7f86feffd700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f86ec06caa0 0x7f86ec06ef60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f87001088b0 tx=0x7f86e8008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:03.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:03.903+0000 7f86fcf74700 1 -- 172.21.15.113:0/2219011053 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f86f005adb0 con 0x7f87000fc7f0 2024-04-12T05:26:04.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.136+0000 7f8705aae700 1 -- 172.21.15.113:0/2219011053 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8700047540 con 0x7f87000fc7f0 2024-04-12T05:26:04.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.136+0000 7f86fcf74700 1 -- 172.21.15.113:0/2219011053 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f86f005a940 con 0x7f87000fc7f0 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:26:04.139 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:26:04.140 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:26:04.141 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:26:04.142 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:26:04.142 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:26:04.142 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:26:04.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.138+0000 7f8705aae700 1 -- 172.21.15.113:0/2219011053 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f86ec06caa0 msgr2=0x7f86ec06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:04.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.138+0000 7f8705aae700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f86ec06caa0 0x7f86ec06ef60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f87001088b0 tx=0x7f86e8008040).stop 2024-04-12T05:26:04.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.138+0000 7f8705aae700 1 -- 172.21.15.113:0/2219011053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87000fc7f0 msgr2=0x7f870010cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:04.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.138+0000 7f8705aae700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87000fc7f0 0x7f870010cf70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f86f000b120 tx=0x7f86f0009cb0).stop 2024-04-12T05:26:04.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.138+0000 7f8705aae700 1 -- 172.21.15.113:0/2219011053 shutdown_connections 2024-04-12T05:26:04.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.138+0000 7f8705aae700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f86ec06caa0 0x7f86ec06ef60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.138+0000 7f8705aae700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87000fc7f0 0x7f870010cf70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.138+0000 7f8705aae700 1 --2- 172.21.15.113:0/2219011053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87000fd150 0x7f870010d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.138+0000 7f8705aae700 1 -- 172.21.15.113:0/2219011053 >> 172.21.15.113:0/2219011053 conn(0x7f8700070590 msgr2=0x7f87000fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:04.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.138+0000 7f8705aae700 1 -- 172.21.15.113:0/2219011053 shutdown_connections 2024-04-12T05:26:04.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.138+0000 7f8705aae700 1 -- 172.21.15.113:0/2219011053 wait complete. 2024-04-12T05:26:04.144 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:26:04.239 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.237+0000 7fa65df31700 1 -- 172.21.15.113:0/1593367115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6580fd150 msgr2=0x7fa6580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:04.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.237+0000 7fa65df31700 1 --2- 172.21.15.113:0/1593367115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6580fd150 0x7fa6580f6930 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fa64c0047f0 tx=0x7fa64c009cb0).stop 2024-04-12T05:26:04.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.237+0000 7fa65df31700 1 -- 172.21.15.113:0/1593367115 shutdown_connections 2024-04-12T05:26:04.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.237+0000 7fa65df31700 1 --2- 172.21.15.113:0/1593367115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6580fd150 0x7fa6580f6930 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.237+0000 7fa65df31700 1 --2- 172.21.15.113:0/1593367115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6580fc7f0 0x7fa6580fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.237+0000 7fa65df31700 1 -- 172.21.15.113:0/1593367115 >> 172.21.15.113:0/1593367115 conn(0x7fa6580f23b0 msgr2=0x7fa6580f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:04.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.237+0000 7fa65df31700 1 -- 172.21.15.113:0/1593367115 shutdown_connections 2024-04-12T05:26:04.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.237+0000 7fa65df31700 1 -- 172.21.15.113:0/1593367115 wait complete. 2024-04-12T05:26:04.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.238+0000 7fa65df31700 1 Processor -- start 2024-04-12T05:26:04.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.238+0000 7fa65df31700 1 -- start start 2024-04-12T05:26:04.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.238+0000 7fa65df31700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6580fc7f0 0x7fa65810cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:04.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.238+0000 7fa65df31700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6580fd150 0x7fa65810d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:04.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.238+0000 7fa65df31700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65810da00 con 0x7fa6580fd150 2024-04-12T05:26:04.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.238+0000 7fa65df31700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65810db40 con 0x7fa6580fc7f0 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.238+0000 7fa656ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6580fd150 0x7fa65810d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.238+0000 7fa656ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6580fd150 0x7fa65810d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57578/0 (socket says 172.21.15.113:57578) 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.238+0000 7fa656ffd700 1 -- 172.21.15.113:0/1034392138 learned_addr learned my addr 172.21.15.113:0/1034392138 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.239+0000 7fa6577fe700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6580fc7f0 0x7fa65810cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.239+0000 7fa656ffd700 1 -- 172.21.15.113:0/1034392138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6580fc7f0 msgr2=0x7fa65810cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.239+0000 7fa656ffd700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6580fc7f0 0x7fa65810cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.239+0000 7fa656ffd700 1 -- 172.21.15.113:0/1034392138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c005040 con 0x7fa6580fd150 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.239+0000 7fa6577fe700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6580fc7f0 0x7fa65810cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.239+0000 7fa656ffd700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6580fd150 0x7fa65810d400 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fa64c0099f0 tx=0x7fa64c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.239+0000 7fa654ff9700 1 -- 172.21.15.113:0/1034392138 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa64c00e030 con 0x7fa6580fd150 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.239+0000 7fa65df31700 1 -- 172.21.15.113:0/1034392138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa658107170 con 0x7fa6580fd150 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.239+0000 7fa654ff9700 1 -- 172.21.15.113:0/1034392138 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa64c007e90 con 0x7fa6580fd150 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.239+0000 7fa654ff9700 1 -- 172.21.15.113:0/1034392138 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa64c018070 con 0x7fa6580fd150 2024-04-12T05:26:04.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.239+0000 7fa65df31700 1 -- 172.21.15.113:0/1034392138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6581076d0 con 0x7fa6580fd150 2024-04-12T05:26:04.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.240+0000 7fa65df31700 1 -- 172.21.15.113:0/1034392138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa658060390 con 0x7fa6580fd150 2024-04-12T05:26:04.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.240+0000 7fa654ff9700 1 -- 172.21.15.113:0/1034392138 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa64c01d420 con 0x7fa6580fd150 2024-04-12T05:26:04.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.241+0000 7fa654ff9700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa64006caa0 0x7fa64006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:04.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.241+0000 7fa654ff9700 1 -- 172.21.15.113:0/1034392138 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa64c08d2f0 con 0x7fa6580fd150 2024-04-12T05:26:04.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.241+0000 7fa6577fe700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa64006caa0 0x7fa64006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:04.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.241+0000 7fa6577fe700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa64006caa0 0x7fa64006ef60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa648004ba0 tx=0x7fa648009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:04.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.245+0000 7fa654ff9700 1 -- 172.21.15.113:0/1034392138 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa64c05ada0 con 0x7fa6580fd150 2024-04-12T05:26:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:04 smithi113 ceph-mon[103371]: from='client.34784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:04 smithi113 ceph-mon[103371]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:04.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:04 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/165708010' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:26:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:04 smithi192 ceph-mon[85304]: from='client.34784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:04 smithi192 ceph-mon[85304]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:04.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:04 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/165708010' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:26:04.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.454+0000 7fa65df31700 1 -- 172.21.15.113:0/1034392138 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6580feb40 con 0x7fa64006caa0 2024-04-12T05:26:04.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.455+0000 7fa654ff9700 1 -- 172.21.15.113:0/1034392138 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa6580feb40 con 0x7fa64006caa0 2024-04-12T05:26:04.458 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:26:04.458 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:26:04.458 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:26:04.458 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:26:04.458 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:26:04.458 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:26:04.458 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:26:04.458 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:26:04.458 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:26:04.458 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:26:04.458 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:26:04.459 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:26:04.459 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:26:04.459 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:26:04.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.457+0000 7fa65df31700 1 -- 172.21.15.113:0/1034392138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa64006caa0 msgr2=0x7fa64006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:04.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.457+0000 7fa65df31700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa64006caa0 0x7fa64006ef60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa648004ba0 tx=0x7fa648009040).stop 2024-04-12T05:26:04.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.457+0000 7fa65df31700 1 -- 172.21.15.113:0/1034392138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6580fd150 msgr2=0x7fa65810d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:04.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.457+0000 7fa65df31700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6580fd150 0x7fa65810d400 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fa64c0099f0 tx=0x7fa64c009cb0).stop 2024-04-12T05:26:04.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.458+0000 7fa65df31700 1 -- 172.21.15.113:0/1034392138 shutdown_connections 2024-04-12T05:26:04.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.458+0000 7fa65df31700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa64006caa0 0x7fa64006ef60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.458+0000 7fa65df31700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6580fc7f0 0x7fa65810cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.460 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.458+0000 7fa65df31700 1 --2- 172.21.15.113:0/1034392138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6580fd150 0x7fa65810d400 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.458+0000 7fa65df31700 1 -- 172.21.15.113:0/1034392138 >> 172.21.15.113:0/1034392138 conn(0x7fa6580f23b0 msgr2=0x7fa6580fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:04.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.458+0000 7fa65df31700 1 -- 172.21.15.113:0/1034392138 shutdown_connections 2024-04-12T05:26:04.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.458+0000 7fa65df31700 1 -- 172.21.15.113:0/1034392138 wait complete. 2024-04-12T05:26:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.555+0000 7efe2d2af700 1 -- 172.21.15.113:0/2461925854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe280fd2d0 msgr2=0x7efe280f70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.555+0000 7efe2d2af700 1 --2- 172.21.15.113:0/2461925854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe280fd2d0 0x7efe280f70d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7efe1c0047f0 tx=0x7efe1c009cb0).stop 2024-04-12T05:26:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.555+0000 7efe2d2af700 1 -- 172.21.15.113:0/2461925854 shutdown_connections 2024-04-12T05:26:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.555+0000 7efe2d2af700 1 --2- 172.21.15.113:0/2461925854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe280fd2d0 0x7efe280f70d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.555+0000 7efe2d2af700 1 --2- 172.21.15.113:0/2461925854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe280fc970 0x7efe280fcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.555+0000 7efe2d2af700 1 -- 172.21.15.113:0/2461925854 >> 172.21.15.113:0/2461925854 conn(0x7efe2806ce50 msgr2=0x7efe2806d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:04.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.555+0000 7efe2d2af700 1 -- 172.21.15.113:0/2461925854 shutdown_connections 2024-04-12T05:26:04.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.555+0000 7efe2d2af700 1 -- 172.21.15.113:0/2461925854 wait complete. 2024-04-12T05:26:04.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.556+0000 7efe2d2af700 1 Processor -- start 2024-04-12T05:26:04.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.556+0000 7efe2d2af700 1 -- start start 2024-04-12T05:26:04.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe2d2af700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe280fc970 0x7efe2810e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:04.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe2d2af700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe280fd2d0 0x7efe2810e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe2d2af700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2810eb60 con 0x7efe280fd2d0 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe2d2af700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2810eca0 con 0x7efe280fc970 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe267fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe280fd2d0 0x7efe2810e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe267fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe280fd2d0 0x7efe2810e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57594/0 (socket says 172.21.15.113:57594) 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe267fc700 1 -- 172.21.15.113:0/190558785 learned_addr learned my addr 172.21.15.113:0/190558785 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe26ffd700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe280fc970 0x7efe2810e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe267fc700 1 -- 172.21.15.113:0/190558785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe280fc970 msgr2=0x7efe2810e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe267fc700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe280fc970 0x7efe2810e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe267fc700 1 -- 172.21.15.113:0/190558785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c005040 con 0x7efe280fd2d0 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe26ffd700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe280fc970 0x7efe2810e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe267fc700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe280fd2d0 0x7efe2810e560 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7efe1c00b120 tx=0x7efe1c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe0ffff700 1 -- 172.21.15.113:0/190558785 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe1c00e030 con 0x7efe280fd2d0 2024-04-12T05:26:04.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe0ffff700 1 -- 172.21.15.113:0/190558785 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe1c007e90 con 0x7efe280fd2d0 2024-04-12T05:26:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe2d2af700 1 -- 172.21.15.113:0/190558785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe281082d0 con 0x7efe280fd2d0 2024-04-12T05:26:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe0ffff700 1 -- 172.21.15.113:0/190558785 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe1c018070 con 0x7efe280fd2d0 2024-04-12T05:26:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.557+0000 7efe2d2af700 1 -- 172.21.15.113:0/190558785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe281087b0 con 0x7efe280fd2d0 2024-04-12T05:26:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.558+0000 7efe2d2af700 1 -- 172.21.15.113:0/190558785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe28047540 con 0x7efe280fd2d0 2024-04-12T05:26:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.559+0000 7efe0ffff700 1 -- 172.21.15.113:0/190558785 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7efe1c009640 con 0x7efe280fd2d0 2024-04-12T05:26:04.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.559+0000 7efe0ffff700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efe1006caa0 0x7efe1006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:04.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.559+0000 7efe26ffd700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efe1006caa0 0x7efe1006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:04.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.559+0000 7efe0ffff700 1 -- 172.21.15.113:0/190558785 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7efe1c08d310 con 0x7efe280fd2d0 2024-04-12T05:26:04.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.560+0000 7efe26ffd700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efe1006caa0 0x7efe1006ef60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7efe28060740 tx=0x7efe18009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:04.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.564+0000 7efe0ffff700 1 -- 172.21.15.113:0/190558785 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7efe1c05adc0 con 0x7efe280fd2d0 2024-04-12T05:26:04.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.830+0000 7efe2d2af700 1 -- 172.21.15.113:0/190558785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efe28108df0 con 0x7efe280fd2d0 2024-04-12T05:26:04.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.830+0000 7efe0ffff700 1 -- 172.21.15.113:0/190558785 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7efe1c05a950 con 0x7efe280fd2d0 2024-04-12T05:26:04.832 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:26:04.833 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:26:04.833 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:26:04.833 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:26:04.833 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:26:04.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.832+0000 7efe2d2af700 1 -- 172.21.15.113:0/190558785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efe1006caa0 msgr2=0x7efe1006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:04.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.832+0000 7efe2d2af700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efe1006caa0 0x7efe1006ef60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7efe28060740 tx=0x7efe18009040).stop 2024-04-12T05:26:04.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.832+0000 7efe2d2af700 1 -- 172.21.15.113:0/190558785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe280fd2d0 msgr2=0x7efe2810e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:04.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.832+0000 7efe2d2af700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe280fd2d0 0x7efe2810e560 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7efe1c00b120 tx=0x7efe1c009cb0).stop 2024-04-12T05:26:04.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.832+0000 7efe2d2af700 1 -- 172.21.15.113:0/190558785 shutdown_connections 2024-04-12T05:26:04.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.832+0000 7efe2d2af700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efe1006caa0 0x7efe1006ef60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.832+0000 7efe2d2af700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe280fc970 0x7efe2810e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.832+0000 7efe2d2af700 1 --2- 172.21.15.113:0/190558785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe280fd2d0 0x7efe2810e560 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:04.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.832+0000 7efe2d2af700 1 -- 172.21.15.113:0/190558785 >> 172.21.15.113:0/190558785 conn(0x7efe2806ce50 msgr2=0x7efe280f5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:04.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.832+0000 7efe2d2af700 1 -- 172.21.15.113:0/190558785 shutdown_connections 2024-04-12T05:26:04.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:04.832+0000 7efe2d2af700 1 -- 172.21.15.113:0/190558785 wait complete. 2024-04-12T05:26:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:05 smithi113 ceph-mon[103371]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:05 smithi113 ceph-mon[103371]: pgmap v417: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:05 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2219011053' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:26:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:05 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/190558785' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:26:05.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:05 smithi192 ceph-mon[85304]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:05.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:05 smithi192 ceph-mon[85304]: pgmap v417: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:05.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:05 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2219011053' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:26:05.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:05 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/190558785' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:26:06.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:06 smithi113 ceph-mon[103371]: from='client.34804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:06.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:06 smithi192 ceph-mon[85304]: from='client.34804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:07 smithi113 ceph-mon[103371]: pgmap v418: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:26:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:07 smithi192 ceph-mon[85304]: pgmap v418: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:26:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:09 smithi113 ceph-mon[103371]: pgmap v419: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:09 smithi192 ceph-mon[85304]: pgmap v419: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:11 smithi113 ceph-mon[103371]: pgmap v420: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:11 smithi192 ceph-mon[85304]: pgmap v420: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:13 smithi113 ceph-mon[103371]: pgmap v421: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:13 smithi192 ceph-mon[85304]: pgmap v421: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:15 smithi113 ceph-mon[103371]: pgmap v422: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:15.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:15 smithi192 ceph-mon[85304]: pgmap v422: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:17 smithi113 ceph-mon[103371]: pgmap v423: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:17 smithi192 ceph-mon[85304]: pgmap v423: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:19 smithi113 ceph-mon[103371]: pgmap v424: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:19 smithi192 ceph-mon[85304]: pgmap v424: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:21 smithi113 ceph-mon[103371]: pgmap v425: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:21 smithi192 ceph-mon[85304]: pgmap v425: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:26:22.450 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:26:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:23 smithi113 ceph-mon[103371]: pgmap v426: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:23 smithi192 ceph-mon[85304]: pgmap v426: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:25 smithi113 ceph-mon[103371]: pgmap v427: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:25.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:25 smithi192 ceph-mon[85304]: pgmap v427: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:27 smithi113 ceph-mon[103371]: pgmap v428: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:27.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:27 smithi192 ceph-mon[85304]: pgmap v428: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:29.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:29 smithi113 ceph-mon[103371]: pgmap v429: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:29.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:29 smithi192 ceph-mon[85304]: pgmap v429: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:31 smithi113 ceph-mon[103371]: pgmap v430: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:31 smithi192 ceph-mon[85304]: pgmap v430: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:33 smithi113 ceph-mon[103371]: pgmap v431: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:33.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:33 smithi192 ceph-mon[85304]: pgmap v431: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:34.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.943+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2984905539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c0fc7f0 msgr2=0x7ff79c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:34.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.943+0000 7ff7a1d99700 1 --2- 172.21.15.113:0/2984905539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c0fc7f0 0x7ff79c0fcbd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7ff78c0047f0 tx=0x7ff78c009cb0).stop 2024-04-12T05:26:34.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.943+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2984905539 shutdown_connections 2024-04-12T05:26:34.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.943+0000 7ff7a1d99700 1 --2- 172.21.15.113:0/2984905539 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79c0fd150 0x7ff79c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:34.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.943+0000 7ff7a1d99700 1 --2- 172.21.15.113:0/2984905539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c0fc7f0 0x7ff79c0fcbd0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:34.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.943+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2984905539 >> 172.21.15.113:0/2984905539 conn(0x7ff79c070590 msgr2=0x7ff79c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:34.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.943+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2984905539 shutdown_connections 2024-04-12T05:26:34.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.943+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2984905539 wait complete. 2024-04-12T05:26:34.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff7a1d99700 1 Processor -- start 2024-04-12T05:26:34.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff7a1d99700 1 -- start start 2024-04-12T05:26:34.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff7a1d99700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c0fc7f0 0x7ff79c10dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:34.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff7a1d99700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79c0fd150 0x7ff79c10e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:34.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff7a1d99700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c10eae0 con 0x7ff79c0fc7f0 2024-04-12T05:26:34.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff7a1d99700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c10ec20 con 0x7ff79c0fd150 2024-04-12T05:26:34.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff79b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c0fc7f0 0x7ff79c10dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:34.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff79b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c0fc7f0 0x7ff79c10dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54736/0 (socket says 172.21.15.113:54736) 2024-04-12T05:26:34.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff79b7fe700 1 -- 172.21.15.113:0/2161707881 learned_addr learned my addr 172.21.15.113:0/2161707881 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:34.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff79affd700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79c0fd150 0x7ff79c10e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:34.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff79b7fe700 1 -- 172.21.15.113:0/2161707881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79c0fd150 msgr2=0x7ff79c10e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:34.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff79b7fe700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79c0fd150 0x7ff79c10e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:34.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff79b7fe700 1 -- 172.21.15.113:0/2161707881 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c005040 con 0x7ff79c0fc7f0 2024-04-12T05:26:34.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.944+0000 7ff79affd700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79c0fd150 0x7ff79c10e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:26:34.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.945+0000 7ff79b7fe700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c0fc7f0 0x7ff79c10dff0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7ff78c005010 tx=0x7ff78c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:34.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.945+0000 7ff798ff9700 1 -- 172.21.15.113:0/2161707881 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff78c00e030 con 0x7ff79c0fc7f0 2024-04-12T05:26:34.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.945+0000 7ff798ff9700 1 -- 172.21.15.113:0/2161707881 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff78c007e90 con 0x7ff79c0fc7f0 2024-04-12T05:26:34.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.945+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2161707881 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79c108250 con 0x7ff79c0fc7f0 2024-04-12T05:26:34.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.945+0000 7ff798ff9700 1 -- 172.21.15.113:0/2161707881 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff78c018070 con 0x7ff79c0fc7f0 2024-04-12T05:26:34.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.945+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2161707881 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c1087b0 con 0x7ff79c0fc7f0 2024-04-12T05:26:34.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.946+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2161707881 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79c05f590 con 0x7ff79c0fc7f0 2024-04-12T05:26:34.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.946+0000 7ff798ff9700 1 -- 172.21.15.113:0/2161707881 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff78c01d6e0 con 0x7ff79c0fc7f0 2024-04-12T05:26:34.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.947+0000 7ff798ff9700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff78406caa0 0x7ff78406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:34.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.947+0000 7ff798ff9700 1 -- 172.21.15.113:0/2161707881 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff78c08d060 con 0x7ff79c0fc7f0 2024-04-12T05:26:34.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.947+0000 7ff79affd700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff78406caa0 0x7ff78406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:34.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.947+0000 7ff79affd700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff78406caa0 0x7ff78406ef60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff79c109930 tx=0x7ff790009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:34.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:34.951+0000 7ff798ff9700 1 -- 172.21.15.113:0/2161707881 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff78c05abd0 con 0x7ff79c0fc7f0 2024-04-12T05:26:35.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.162+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2161707881 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff79c108e40 con 0x7ff78406caa0 2024-04-12T05:26:35.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.163+0000 7ff798ff9700 1 -- 172.21.15.113:0/2161707881 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff79c108e40 con 0x7ff78406caa0 2024-04-12T05:26:35.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.165+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2161707881 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff78406caa0 msgr2=0x7ff78406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.165+0000 7ff7a1d99700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff78406caa0 0x7ff78406ef60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff79c109930 tx=0x7ff790009d60).stop 2024-04-12T05:26:35.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.165+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2161707881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c0fc7f0 msgr2=0x7ff79c10dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.165+0000 7ff7a1d99700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c0fc7f0 0x7ff79c10dff0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7ff78c005010 tx=0x7ff78c009cb0).stop 2024-04-12T05:26:35.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.165+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2161707881 shutdown_connections 2024-04-12T05:26:35.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.165+0000 7ff7a1d99700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff78406caa0 0x7ff78406ef60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.165+0000 7ff7a1d99700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff79c0fc7f0 0x7ff79c10dff0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.165+0000 7ff7a1d99700 1 --2- 172.21.15.113:0/2161707881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff79c0fd150 0x7ff79c10e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.165+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2161707881 >> 172.21.15.113:0/2161707881 conn(0x7ff79c070590 msgr2=0x7ff79c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:35.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.165+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2161707881 shutdown_connections 2024-04-12T05:26:35.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.165+0000 7ff7a1d99700 1 -- 172.21.15.113:0/2161707881 wait complete. 2024-04-12T05:26:35.181 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:26:35.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.264+0000 7f2ef201e700 1 -- 172.21.15.113:0/1834379863 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec0f88a0 msgr2=0x7f2eec0f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.264+0000 7f2ef201e700 1 --2- 172.21.15.113:0/1834379863 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec0f88a0 0x7f2eec0f8c80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f2ed40047f0 tx=0x7f2ed4009cb0).stop 2024-04-12T05:26:35.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.264+0000 7f2ef201e700 1 -- 172.21.15.113:0/1834379863 shutdown_connections 2024-04-12T05:26:35.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.264+0000 7f2ef201e700 1 --2- 172.21.15.113:0/1834379863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2eec0f9200 0x7f2eec101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.264+0000 7f2ef201e700 1 --2- 172.21.15.113:0/1834379863 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec0f88a0 0x7f2eec0f8c80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.264+0000 7f2ef201e700 1 -- 172.21.15.113:0/1834379863 >> 172.21.15.113:0/1834379863 conn(0x7f2eec070590 msgr2=0x7f2eec0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:35.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.264+0000 7f2ef201e700 1 -- 172.21.15.113:0/1834379863 shutdown_connections 2024-04-12T05:26:35.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.264+0000 7f2ef201e700 1 -- 172.21.15.113:0/1834379863 wait complete. 2024-04-12T05:26:35.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.264+0000 7f2ef201e700 1 Processor -- start 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.264+0000 7f2ef201e700 1 -- start start 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2ef201e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec0f88a0 0x7f2eec10e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2ef201e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2eec0f9200 0x7f2eec10e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2ef201e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec10eb10 con 0x7f2eec0f88a0 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2ef201e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec10ec50 con 0x7f2eec0f9200 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2eeb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec0f88a0 0x7f2eec10e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2eeb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec0f88a0 0x7f2eec10e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54760/0 (socket says 172.21.15.113:54760) 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2eeb7fe700 1 -- 172.21.15.113:0/307906099 learned_addr learned my addr 172.21.15.113:0/307906099 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2eeb7fe700 1 -- 172.21.15.113:0/307906099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2eec0f9200 msgr2=0x7f2eec10e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2eeb7fe700 1 --2- 172.21.15.113:0/307906099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2eec0f9200 0x7f2eec10e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2eeb7fe700 1 -- 172.21.15.113:0/307906099 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed4005040 con 0x7f2eec0f88a0 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2eeb7fe700 1 --2- 172.21.15.113:0/307906099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec0f88a0 0x7f2eec10e020 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f2ed400b120 tx=0x7f2ed4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:35.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2ee8ff9700 1 -- 172.21.15.113:0/307906099 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ed400e030 con 0x7f2eec0f88a0 2024-04-12T05:26:35.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2ee8ff9700 1 -- 172.21.15.113:0/307906099 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ed4007e90 con 0x7f2eec0f88a0 2024-04-12T05:26:35.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2ef201e700 1 -- 172.21.15.113:0/307906099 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eec108280 con 0x7f2eec0f88a0 2024-04-12T05:26:35.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2ee8ff9700 1 -- 172.21.15.113:0/307906099 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ed4018070 con 0x7f2eec0f88a0 2024-04-12T05:26:35.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.265+0000 7f2ef201e700 1 -- 172.21.15.113:0/307906099 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eec108760 con 0x7f2eec0f88a0 2024-04-12T05:26:35.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.266+0000 7f2ef201e700 1 -- 172.21.15.113:0/307906099 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec05f590 con 0x7f2eec0f88a0 2024-04-12T05:26:35.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.267+0000 7f2ee8ff9700 1 -- 172.21.15.113:0/307906099 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2ed401d420 con 0x7f2eec0f88a0 2024-04-12T05:26:35.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.267+0000 7f2ee8ff9700 1 --2- 172.21.15.113:0/307906099 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2ed806caa0 0x7f2ed806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:35.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.267+0000 7f2ee8ff9700 1 -- 172.21.15.113:0/307906099 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2ed408d300 con 0x7f2eec0f88a0 2024-04-12T05:26:35.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.267+0000 7f2eeaffd700 1 --2- 172.21.15.113:0/307906099 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2ed806caa0 0x7f2ed806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:35.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.268+0000 7f2eeaffd700 1 --2- 172.21.15.113:0/307906099 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2ed806caa0 0x7f2ed806ef60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f2eec109960 tx=0x7f2ee40049c0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:35.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.272+0000 7f2ee8ff9700 1 -- 172.21.15.113:0/307906099 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2ed405adb0 con 0x7f2eec0f88a0 2024-04-12T05:26:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:35 smithi113 ceph-mon[103371]: pgmap v432: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:35.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:35 smithi192 ceph-mon[85304]: pgmap v432: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:35.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.477+0000 7f2ef201e700 1 -- 172.21.15.113:0/307906099 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2eec1097b0 con 0x7f2ed806caa0 2024-04-12T05:26:35.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.479+0000 7f2ee8ff9700 1 -- 172.21.15.113:0/307906099 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2eec1097b0 con 0x7f2ed806caa0 2024-04-12T05:26:35.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.481+0000 7f2ef201e700 1 -- 172.21.15.113:0/307906099 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2ed806caa0 msgr2=0x7f2ed806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.481+0000 7f2ef201e700 1 --2- 172.21.15.113:0/307906099 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2ed806caa0 0x7f2ed806ef60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f2eec109960 tx=0x7f2ee40049c0).stop 2024-04-12T05:26:35.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.481+0000 7f2ef201e700 1 -- 172.21.15.113:0/307906099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec0f88a0 msgr2=0x7f2eec10e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.484 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.481+0000 7f2ef201e700 1 --2- 172.21.15.113:0/307906099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec0f88a0 0x7f2eec10e020 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f2ed400b120 tx=0x7f2ed4009cb0).stop 2024-04-12T05:26:35.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.481+0000 7f2ef201e700 1 -- 172.21.15.113:0/307906099 shutdown_connections 2024-04-12T05:26:35.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.481+0000 7f2ef201e700 1 --2- 172.21.15.113:0/307906099 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2ed806caa0 0x7f2ed806ef60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.481+0000 7f2ef201e700 1 --2- 172.21.15.113:0/307906099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec0f88a0 0x7f2eec10e020 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.481+0000 7f2ef201e700 1 --2- 172.21.15.113:0/307906099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2eec0f9200 0x7f2eec10e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:26:35.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.481+0000 7f2ef201e700 1 -- 172.21.15.113:0/307906099 >> 172.21.15.113:0/307906099 conn(0x7f2eec070590 msgr2=0x7f2eec0fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:35.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.481+0000 7f2ef201e700 1 -- 172.21.15.113:0/307906099 shutdown_connections 2024-04-12T05:26:35.485 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.481+0000 7f2ef201e700 1 -- 172.21.15.113:0/307906099 wait complete. 2024-04-12T05:26:35.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.579+0000 7f520b885700 1 -- 172.21.15.113:0/772106191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52040fd150 msgr2=0x7f52040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.579+0000 7f520b885700 1 --2- 172.21.15.113:0/772106191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52040fd150 0x7f52040f6930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f51f40047f0 tx=0x7f51f4009cb0).stop 2024-04-12T05:26:35.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.579+0000 7f520b885700 1 -- 172.21.15.113:0/772106191 shutdown_connections 2024-04-12T05:26:35.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.579+0000 7f520b885700 1 --2- 172.21.15.113:0/772106191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52040fd150 0x7f52040f6930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.579+0000 7f520b885700 1 --2- 172.21.15.113:0/772106191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52040fc7f0 0x7f52040fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.579+0000 7f520b885700 1 -- 172.21.15.113:0/772106191 >> 172.21.15.113:0/772106191 conn(0x7f52040f23b0 msgr2=0x7f52040f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.579+0000 7f520b885700 1 -- 172.21.15.113:0/772106191 shutdown_connections 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.579+0000 7f520b885700 1 -- 172.21.15.113:0/772106191 wait complete. 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.580+0000 7f520b885700 1 Processor -- start 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.580+0000 7f520b885700 1 -- start start 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.580+0000 7f520b885700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52040fc7f0 0x7f52040f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.580+0000 7f520b885700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52040fd150 0x7f52040f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.580+0000 7f520b885700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52040f8f20 con 0x7f52040fd150 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.580+0000 7f520b885700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52040f9060 con 0x7f52040fc7f0 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.580+0000 7f5208e20700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52040fd150 0x7f52040f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.580+0000 7f5208e20700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52040fd150 0x7f52040f8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54786/0 (socket says 172.21.15.113:54786) 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.580+0000 7f5208e20700 1 -- 172.21.15.113:0/1901070714 learned_addr learned my addr 172.21.15.113:0/1901070714 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:35.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.580+0000 7f5209621700 1 --2- 172.21.15.113:0/1901070714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52040fc7f0 0x7f52040f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:35.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.581+0000 7f5208e20700 1 -- 172.21.15.113:0/1901070714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52040fc7f0 msgr2=0x7f52040f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.581+0000 7f5208e20700 1 --2- 172.21.15.113:0/1901070714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52040fc7f0 0x7f52040f8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.581+0000 7f5208e20700 1 -- 172.21.15.113:0/1901070714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f4005040 con 0x7f52040fd150 2024-04-12T05:26:35.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.581+0000 7f5208e20700 1 --2- 172.21.15.113:0/1901070714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52040fd150 0x7f52040f8a30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f51f40099f0 tx=0x7f51f4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:35.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.581+0000 7f51fa7fc700 1 -- 172.21.15.113:0/1901070714 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51f400e030 con 0x7f52040fd150 2024-04-12T05:26:35.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.581+0000 7f51fa7fc700 1 -- 172.21.15.113:0/1901070714 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51f4007e90 con 0x7f52040fd150 2024-04-12T05:26:35.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.581+0000 7f520b885700 1 -- 172.21.15.113:0/1901070714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52040f92b0 con 0x7f52040fd150 2024-04-12T05:26:35.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.581+0000 7f51fa7fc700 1 -- 172.21.15.113:0/1901070714 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51f4018070 con 0x7f52040fd150 2024-04-12T05:26:35.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.581+0000 7f520b885700 1 -- 172.21.15.113:0/1901070714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52041146a0 con 0x7f52040fd150 2024-04-12T05:26:35.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.582+0000 7f520b885700 1 -- 172.21.15.113:0/1901070714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5204060390 con 0x7f52040fd150 2024-04-12T05:26:35.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.583+0000 7f51fa7fc700 1 -- 172.21.15.113:0/1901070714 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f51f401d420 con 0x7f52040fd150 2024-04-12T05:26:35.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.583+0000 7f51fa7fc700 1 --2- 172.21.15.113:0/1901070714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f51f006caa0 0x7f51f006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:35.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.583+0000 7f51fa7fc700 1 -- 172.21.15.113:0/1901070714 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f51f408d630 con 0x7f52040fd150 2024-04-12T05:26:35.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.583+0000 7f5209621700 1 --2- 172.21.15.113:0/1901070714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f51f006caa0 0x7f51f006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:35.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.583+0000 7f5209621700 1 --2- 172.21.15.113:0/1901070714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f51f006caa0 0x7f51f006ef60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f5200004d20 tx=0x7f5200009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:35.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.587+0000 7f51fa7fc700 1 -- 172.21.15.113:0/1901070714 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f51f40575d0 con 0x7f52040fd150 2024-04-12T05:26:35.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.784+0000 7f520b885700 1 -- 172.21.15.113:0/1901070714 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f52040f9ec0 con 0x7f51f006caa0 2024-04-12T05:26:35.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.794+0000 7f51fa7fc700 1 -- 172.21.15.113:0/1901070714 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f52040f9ec0 con 0x7f51f006caa0 2024-04-12T05:26:35.797 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:26:35.797 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (14m) 7m ago 25m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:26:35.797 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (11m) 7m ago 25m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:26:35.797 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (11m) 7m ago 25m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (14m) 7m ago 25m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (7m) 7m ago 20m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (7m) 7m ago 20m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (7m) 7m ago 20m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (20m) 7m ago 20m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (15m) 7m ago 26m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (15m) 7m ago 24m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (12m) 7m ago 27m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (12m) 7m ago 24m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (15m) 7m ago 25m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (14m) 7m ago 24m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (11m) 7m ago 24m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:26:35.798 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (10m) 7m ago 23m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:26:35.799 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (10m) 7m ago 23m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:26:35.799 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (9m) 7m ago 22m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:26:35.799 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (9m) 7m ago 22m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:26:35.799 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (8m) 7m ago 21m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:26:35.799 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (14m) 7m ago 24m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:26:35.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.796+0000 7f520b885700 1 -- 172.21.15.113:0/1901070714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f51f006caa0 msgr2=0x7f51f006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.796+0000 7f520b885700 1 --2- 172.21.15.113:0/1901070714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f51f006caa0 0x7f51f006ef60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f5200004d20 tx=0x7f5200009040).stop 2024-04-12T05:26:35.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.796+0000 7f520b885700 1 -- 172.21.15.113:0/1901070714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52040fd150 msgr2=0x7f52040f8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.796+0000 7f520b885700 1 --2- 172.21.15.113:0/1901070714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52040fd150 0x7f52040f8a30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f51f40099f0 tx=0x7f51f4009cb0).stop 2024-04-12T05:26:35.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.796+0000 7f520b885700 1 -- 172.21.15.113:0/1901070714 shutdown_connections 2024-04-12T05:26:35.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.796+0000 7f520b885700 1 --2- 172.21.15.113:0/1901070714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f51f006caa0 0x7f51f006ef60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.796+0000 7f520b885700 1 --2- 172.21.15.113:0/1901070714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52040fc7f0 0x7f52040f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.796+0000 7f520b885700 1 --2- 172.21.15.113:0/1901070714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52040fd150 0x7f52040f8a30 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.796+0000 7f520b885700 1 -- 172.21.15.113:0/1901070714 >> 172.21.15.113:0/1901070714 conn(0x7f52040f23b0 msgr2=0x7f5204061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:35.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.796+0000 7f520b885700 1 -- 172.21.15.113:0/1901070714 shutdown_connections 2024-04-12T05:26:35.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.796+0000 7f520b885700 1 -- 172.21.15.113:0/1901070714 wait complete. 2024-04-12T05:26:35.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.893+0000 7fe19d5a6700 1 -- 172.21.15.113:0/567238633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1980f88d0 msgr2=0x7fe1980f8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.893+0000 7fe19d5a6700 1 --2- 172.21.15.113:0/567238633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1980f88d0 0x7fe1980f8cb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fe18c0047f0 tx=0x7fe18c009cb0).stop 2024-04-12T05:26:35.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.893+0000 7fe19d5a6700 1 -- 172.21.15.113:0/567238633 shutdown_connections 2024-04-12T05:26:35.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.893+0000 7fe19d5a6700 1 --2- 172.21.15.113:0/567238633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1980f9230 0x7fe198101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.893+0000 7fe19d5a6700 1 --2- 172.21.15.113:0/567238633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1980f88d0 0x7fe1980f8cb0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.893+0000 7fe19d5a6700 1 -- 172.21.15.113:0/567238633 >> 172.21.15.113:0/567238633 conn(0x7fe1980f2400 msgr2=0x7fe1980f4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:35.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.893+0000 7fe19d5a6700 1 -- 172.21.15.113:0/567238633 shutdown_connections 2024-04-12T05:26:35.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.894+0000 7fe19d5a6700 1 -- 172.21.15.113:0/567238633 wait complete. 2024-04-12T05:26:35.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.894+0000 7fe19d5a6700 1 Processor -- start 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.894+0000 7fe19d5a6700 1 -- start start 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.894+0000 7fe19d5a6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1980f88d0 0x7fe19810e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.894+0000 7fe19d5a6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1980f9230 0x7fe19810e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe19d5a6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19810eb60 con 0x7fe1980f88d0 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe19d5a6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19810eca0 con 0x7fe1980f9230 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe196ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1980f88d0 0x7fe19810e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe196ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1980f88d0 0x7fe19810e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54812/0 (socket says 172.21.15.113:54812) 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe196ffd700 1 -- 172.21.15.113:0/3779996332 learned_addr learned my addr 172.21.15.113:0/3779996332 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe1967fc700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1980f9230 0x7fe19810e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe196ffd700 1 -- 172.21.15.113:0/3779996332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1980f9230 msgr2=0x7fe19810e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe196ffd700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1980f9230 0x7fe19810e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe196ffd700 1 -- 172.21.15.113:0/3779996332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c005040 con 0x7fe1980f88d0 2024-04-12T05:26:35.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe1967fc700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1980f9230 0x7fe19810e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:26:35.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe196ffd700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1980f88d0 0x7fe19810e070 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fe18c005010 tx=0x7fe18c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:35.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe17ffff700 1 -- 172.21.15.113:0/3779996332 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe18c00e030 con 0x7fe1980f88d0 2024-04-12T05:26:35.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe19d5a6700 1 -- 172.21.15.113:0/3779996332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1981082d0 con 0x7fe1980f88d0 2024-04-12T05:26:35.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe17ffff700 1 -- 172.21.15.113:0/3779996332 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe18c007e90 con 0x7fe1980f88d0 2024-04-12T05:26:35.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe17ffff700 1 -- 172.21.15.113:0/3779996332 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe18c018070 con 0x7fe1980f88d0 2024-04-12T05:26:35.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.895+0000 7fe19d5a6700 1 -- 172.21.15.113:0/3779996332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1981087b0 con 0x7fe1980f88d0 2024-04-12T05:26:35.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.896+0000 7fe19d5a6700 1 -- 172.21.15.113:0/3779996332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe198060330 con 0x7fe1980f88d0 2024-04-12T05:26:35.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.897+0000 7fe17ffff700 1 -- 172.21.15.113:0/3779996332 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe18c01d6e0 con 0x7fe1980f88d0 2024-04-12T05:26:35.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.897+0000 7fe17ffff700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe18006ca30 0x7fe18006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:35.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.897+0000 7fe17ffff700 1 -- 172.21.15.113:0/3779996332 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe18c08d080 con 0x7fe1980f88d0 2024-04-12T05:26:35.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.897+0000 7fe1967fc700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe18006ca30 0x7fe18006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:35.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.897+0000 7fe1967fc700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe18006ca30 0x7fe18006eef0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fe188009a00 tx=0x7fe18800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:35.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:35.901+0000 7fe17ffff700 1 -- 172.21.15.113:0/3779996332 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe18c05ab30 con 0x7fe1980f88d0 2024-04-12T05:26:36.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.168+0000 7fe19d5a6700 1 -- 172.21.15.113:0/3779996332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe1981097f0 con 0x7fe1980f88d0 2024-04-12T05:26:36.172 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.168+0000 7fe17ffff700 1 -- 172.21.15.113:0/3779996332 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fe18c05a6c0 con 0x7fe1980f88d0 2024-04-12T05:26:36.172 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:26:36.172 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:26:36.172 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:26:36.172 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:26:36.172 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:26:36.172 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:26:36.172 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:26:36.172 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:26:36.172 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:26:36.172 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:26:36.173 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:26:36.173 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:26:36.173 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:26:36.173 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:26:36.173 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:26:36.173 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:26:36.173 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:26:36.173 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:26:36.173 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:26:36.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.170+0000 7fe19d5a6700 1 -- 172.21.15.113:0/3779996332 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe18006ca30 msgr2=0x7fe18006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.170+0000 7fe19d5a6700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe18006ca30 0x7fe18006eef0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fe188009a00 tx=0x7fe18800b040).stop 2024-04-12T05:26:36.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.170+0000 7fe19d5a6700 1 -- 172.21.15.113:0/3779996332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1980f88d0 msgr2=0x7fe19810e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.170+0000 7fe19d5a6700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1980f88d0 0x7fe19810e070 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fe18c005010 tx=0x7fe18c009cb0).stop 2024-04-12T05:26:36.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.170+0000 7fe19d5a6700 1 -- 172.21.15.113:0/3779996332 shutdown_connections 2024-04-12T05:26:36.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.170+0000 7fe19d5a6700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe18006ca30 0x7fe18006eef0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.170+0000 7fe19d5a6700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1980f88d0 0x7fe19810e070 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.170+0000 7fe19d5a6700 1 --2- 172.21.15.113:0/3779996332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1980f9230 0x7fe19810e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.170+0000 7fe19d5a6700 1 -- 172.21.15.113:0/3779996332 >> 172.21.15.113:0/3779996332 conn(0x7fe1980f2400 msgr2=0x7fe1980fc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:36.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.170+0000 7fe19d5a6700 1 -- 172.21.15.113:0/3779996332 shutdown_connections 2024-04-12T05:26:36.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.170+0000 7fe19d5a6700 1 -- 172.21.15.113:0/3779996332 wait complete. 2024-04-12T05:26:36.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.268+0000 7fe6b1957700 1 -- 172.21.15.113:0/4280198424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6ac0fd140 msgr2=0x7fe6ac0f6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.268+0000 7fe6b1957700 1 --2- 172.21.15.113:0/4280198424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6ac0fd140 0x7fe6ac0f6920 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fe69c0047f0 tx=0x7fe69c009cb0).stop 2024-04-12T05:26:36.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.268+0000 7fe6b1957700 1 -- 172.21.15.113:0/4280198424 shutdown_connections 2024-04-12T05:26:36.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.268+0000 7fe6b1957700 1 --2- 172.21.15.113:0/4280198424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6ac0fd140 0x7fe6ac0f6920 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.268+0000 7fe6b1957700 1 --2- 172.21.15.113:0/4280198424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6ac0fc7e0 0x7fe6ac0fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.268+0000 7fe6b1957700 1 -- 172.21.15.113:0/4280198424 >> 172.21.15.113:0/4280198424 conn(0x7fe6ac0f23c0 msgr2=0x7fe6ac0f47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:36.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.268+0000 7fe6b1957700 1 -- 172.21.15.113:0/4280198424 shutdown_connections 2024-04-12T05:26:36.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.268+0000 7fe6b1957700 1 -- 172.21.15.113:0/4280198424 wait complete. 2024-04-12T05:26:36.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6b1957700 1 Processor -- start 2024-04-12T05:26:36.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6b1957700 1 -- start start 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6b1957700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6ac0fc7e0 0x7fe6ac109140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6b1957700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6ac0fd140 0x7fe6ac109630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6b1957700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac109b20 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6b1957700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac109c60 con 0x7fe6ac0fd140 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6aaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6ac0fc7e0 0x7fe6ac109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6aaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6ac0fc7e0 0x7fe6ac109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54820/0 (socket says 172.21.15.113:54820) 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6aaffd700 1 -- 172.21.15.113:0/291001826 learned_addr learned my addr 172.21.15.113:0/291001826 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6aa7fc700 1 --2- 172.21.15.113:0/291001826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6ac0fd140 0x7fe6ac109630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6aaffd700 1 -- 172.21.15.113:0/291001826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6ac0fd140 msgr2=0x7fe6ac109630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6aaffd700 1 --2- 172.21.15.113:0/291001826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6ac0fd140 0x7fe6ac109630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.269+0000 7fe6aaffd700 1 -- 172.21.15.113:0/291001826 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c005040 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.270+0000 7fe6aaffd700 1 --2- 172.21.15.113:0/291001826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6ac0fc7e0 0x7fe6ac109140 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fe69400c740 tx=0x7fe694007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.270+0000 7fe6a3fff700 1 -- 172.21.15.113:0/291001826 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe69400e680 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.270+0000 7fe6a3fff700 1 -- 172.21.15.113:0/291001826 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe69400ec90 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.270+0000 7fe6b1957700 1 -- 172.21.15.113:0/291001826 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ac109f10 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.270+0000 7fe6a3fff700 1 -- 172.21.15.113:0/291001826 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe694013d30 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.270+0000 7fe6b1957700 1 -- 172.21.15.113:0/291001826 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6ac1124b0 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.271+0000 7fe6b1957700 1 -- 172.21.15.113:0/291001826 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6ac060370 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.271+0000 7fe6a3fff700 1 -- 172.21.15.113:0/291001826 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe694013410 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.272+0000 7fe6a3fff700 1 --2- 172.21.15.113:0/291001826 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe69806caa0 0x7fe69806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:36.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.272+0000 7fe6a3fff700 1 -- 172.21.15.113:0/291001826 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe69408c070 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.272+0000 7fe6aa7fc700 1 --2- 172.21.15.113:0/291001826 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe69806caa0 0x7fe69806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:36.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.272+0000 7fe6aa7fc700 1 --2- 172.21.15.113:0/291001826 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe69806caa0 0x7fe69806ef60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fe6ac061470 tx=0x7fe69c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:36.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.276+0000 7fe6a3fff700 1 -- 172.21.15.113:0/291001826 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe694056090 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.506+0000 7fe6b1957700 1 -- 172.21.15.113:0/291001826 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe6ac047540 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.507+0000 7fe6a3fff700 1 -- 172.21.15.113:0/291001826 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7fe6940596b0 con 0x7fe6ac0fc7e0 2024-04-12T05:26:36.510 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:26:36.511 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:26:36.512 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:26:36.513 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:26:36.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.509+0000 7fe6b1957700 1 -- 172.21.15.113:0/291001826 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe69806caa0 msgr2=0x7fe69806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.509+0000 7fe6b1957700 1 --2- 172.21.15.113:0/291001826 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe69806caa0 0x7fe69806ef60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fe6ac061470 tx=0x7fe69c00b040).stop 2024-04-12T05:26:36.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.509+0000 7fe6b1957700 1 -- 172.21.15.113:0/291001826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6ac0fc7e0 msgr2=0x7fe6ac109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.509+0000 7fe6b1957700 1 --2- 172.21.15.113:0/291001826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6ac0fc7e0 0x7fe6ac109140 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fe69400c740 tx=0x7fe694007ef0).stop 2024-04-12T05:26:36.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.509+0000 7fe6b1957700 1 -- 172.21.15.113:0/291001826 shutdown_connections 2024-04-12T05:26:36.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.509+0000 7fe6b1957700 1 --2- 172.21.15.113:0/291001826 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe69806caa0 0x7fe69806ef60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.509+0000 7fe6b1957700 1 --2- 172.21.15.113:0/291001826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6ac0fc7e0 0x7fe6ac109140 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.509+0000 7fe6b1957700 1 --2- 172.21.15.113:0/291001826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6ac0fd140 0x7fe6ac109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.509+0000 7fe6b1957700 1 -- 172.21.15.113:0/291001826 >> 172.21.15.113:0/291001826 conn(0x7fe6ac0f23c0 msgr2=0x7fe6ac0faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:36.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.509+0000 7fe6b1957700 1 -- 172.21.15.113:0/291001826 shutdown_connections 2024-04-12T05:26:36.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.509+0000 7fe6b1957700 1 -- 172.21.15.113:0/291001826 wait complete. 2024-04-12T05:26:36.515 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:26:36.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.608+0000 7f91d1670700 1 -- 172.21.15.113:0/4177069699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91cc0fd150 msgr2=0x7f91cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.608+0000 7f91d1670700 1 --2- 172.21.15.113:0/4177069699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91cc0fd150 0x7f91cc0f6930 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f91c00047f0 tx=0x7f91c0009cb0).stop 2024-04-12T05:26:36.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.608+0000 7f91d1670700 1 -- 172.21.15.113:0/4177069699 shutdown_connections 2024-04-12T05:26:36.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.608+0000 7f91d1670700 1 --2- 172.21.15.113:0/4177069699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91cc0fd150 0x7f91cc0f6930 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.608+0000 7f91d1670700 1 --2- 172.21.15.113:0/4177069699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91cc0fc7f0 0x7f91cc0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:26:36.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.609+0000 7f91d1670700 1 -- 172.21.15.113:0/4177069699 >> 172.21.15.113:0/4177069699 conn(0x7f91cc0f23b0 msgr2=0x7f91cc0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:36.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.609+0000 7f91d1670700 1 -- 172.21.15.113:0/4177069699 shutdown_connections 2024-04-12T05:26:36.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.609+0000 7f91d1670700 1 -- 172.21.15.113:0/4177069699 wait complete. 2024-04-12T05:26:36.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.609+0000 7f91d1670700 1 Processor -- start 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.609+0000 7f91d1670700 1 -- start start 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91d1670700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91cc0fc7f0 0x7f91cc10e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91d1670700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91cc0fd150 0x7f91cc10e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91d1670700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91cc10ec10 con 0x7f91cc0fd150 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91d1670700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91cc108230 con 0x7f91cc0fc7f0 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91ca7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91cc0fd150 0x7f91cc10e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91ca7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91cc0fd150 0x7f91cc10e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54838/0 (socket says 172.21.15.113:54838) 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91ca7fc700 1 -- 172.21.15.113:0/3027792234 learned_addr learned my addr 172.21.15.113:0/3027792234 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91caffd700 1 --2- 172.21.15.113:0/3027792234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91cc0fc7f0 0x7f91cc10e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91ca7fc700 1 -- 172.21.15.113:0/3027792234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91cc0fc7f0 msgr2=0x7f91cc10e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91ca7fc700 1 --2- 172.21.15.113:0/3027792234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91cc0fc7f0 0x7f91cc10e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91ca7fc700 1 -- 172.21.15.113:0/3027792234 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c0005040 con 0x7f91cc0fd150 2024-04-12T05:26:36.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91ca7fc700 1 --2- 172.21.15.113:0/3027792234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91cc0fd150 0x7f91cc10e580 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f91c00099f0 tx=0x7f91c0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:36.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91b3fff700 1 -- 172.21.15.113:0/3027792234 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91c000e030 con 0x7f91cc0fd150 2024-04-12T05:26:36.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91b3fff700 1 -- 172.21.15.113:0/3027792234 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91c0007e90 con 0x7f91cc0fd150 2024-04-12T05:26:36.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91d1670700 1 -- 172.21.15.113:0/3027792234 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91cc108480 con 0x7f91cc0fd150 2024-04-12T05:26:36.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91b3fff700 1 -- 172.21.15.113:0/3027792234 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91c0018070 con 0x7f91cc0fd150 2024-04-12T05:26:36.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.610+0000 7f91d1670700 1 -- 172.21.15.113:0/3027792234 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91cc108960 con 0x7f91cc0fd150 2024-04-12T05:26:36.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.611+0000 7f91d1670700 1 -- 172.21.15.113:0/3027792234 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91cc060390 con 0x7f91cc0fd150 2024-04-12T05:26:36.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.612+0000 7f91b3fff700 1 -- 172.21.15.113:0/3027792234 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f91c001d420 con 0x7f91cc0fd150 2024-04-12T05:26:36.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.612+0000 7f91b3fff700 1 --2- 172.21.15.113:0/3027792234 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91b406caa0 0x7f91b406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:36.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.612+0000 7f91b3fff700 1 -- 172.21.15.113:0/3027792234 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f91c008d2f0 con 0x7f91cc0fd150 2024-04-12T05:26:36.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.612+0000 7f91caffd700 1 --2- 172.21.15.113:0/3027792234 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91b406caa0 0x7f91b406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:36.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.613+0000 7f91caffd700 1 --2- 172.21.15.113:0/3027792234 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91b406caa0 0x7f91b406ef60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f91bc00b910 tx=0x7f91bc008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:36.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.617+0000 7f91b3fff700 1 -- 172.21.15.113:0/3027792234 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f91c005ada0 con 0x7f91cc0fd150 2024-04-12T05:26:36.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.822+0000 7f91d1670700 1 -- 172.21.15.113:0/3027792234 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f91cc109780 con 0x7f91b406caa0 2024-04-12T05:26:36.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.823+0000 7f91b3fff700 1 -- 172.21.15.113:0/3027792234 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f91cc109780 con 0x7f91b406caa0 2024-04-12T05:26:36.827 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:26:36.827 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:26:36.827 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:26:36.827 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:26:36.827 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:26:36.827 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:26:36.827 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:26:36.827 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:26:36.827 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:26:36.828 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:26:36.828 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:26:36.828 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:26:36.828 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:26:36.828 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:26:36.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.825+0000 7f91d1670700 1 -- 172.21.15.113:0/3027792234 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91b406caa0 msgr2=0x7f91b406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.825+0000 7f91d1670700 1 --2- 172.21.15.113:0/3027792234 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91b406caa0 0x7f91b406ef60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f91bc00b910 tx=0x7f91bc008040).stop 2024-04-12T05:26:36.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.825+0000 7f91d1670700 1 -- 172.21.15.113:0/3027792234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91cc0fd150 msgr2=0x7f91cc10e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.825+0000 7f91d1670700 1 --2- 172.21.15.113:0/3027792234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91cc0fd150 0x7f91cc10e580 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f91c00099f0 tx=0x7f91c0009cb0).stop 2024-04-12T05:26:36.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.826+0000 7f91d1670700 1 -- 172.21.15.113:0/3027792234 shutdown_connections 2024-04-12T05:26:36.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.826+0000 7f91d1670700 1 --2- 172.21.15.113:0/3027792234 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91b406caa0 0x7f91b406ef60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.826+0000 7f91d1670700 1 --2- 172.21.15.113:0/3027792234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91cc0fc7f0 0x7f91cc10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.826+0000 7f91d1670700 1 --2- 172.21.15.113:0/3027792234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91cc0fd150 0x7f91cc10e580 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.826+0000 7f91d1670700 1 -- 172.21.15.113:0/3027792234 >> 172.21.15.113:0/3027792234 conn(0x7f91cc0f23b0 msgr2=0x7f91cc060b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:36.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.826+0000 7f91d1670700 1 -- 172.21.15.113:0/3027792234 shutdown_connections 2024-04-12T05:26:36.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.826+0000 7f91d1670700 1 -- 172.21.15.113:0/3027792234 wait complete. 2024-04-12T05:26:36.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.925+0000 7fa049207700 1 -- 172.21.15.113:0/3840316259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0440fc7f0 msgr2=0x7fa0440fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.925+0000 7fa049207700 1 --2- 172.21.15.113:0/3840316259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0440fc7f0 0x7fa0440fcbd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fa0340047f0 tx=0x7fa034009cb0).stop 2024-04-12T05:26:36.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.926+0000 7fa049207700 1 -- 172.21.15.113:0/3840316259 shutdown_connections 2024-04-12T05:26:36.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.926+0000 7fa049207700 1 --2- 172.21.15.113:0/3840316259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0440fd150 0x7fa0440f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:26:36.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.926+0000 7fa049207700 1 --2- 172.21.15.113:0/3840316259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0440fc7f0 0x7fa0440fcbd0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.926+0000 7fa049207700 1 -- 172.21.15.113:0/3840316259 >> 172.21.15.113:0/3840316259 conn(0x7fa044070590 msgr2=0x7fa0440f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:36.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.926+0000 7fa049207700 1 -- 172.21.15.113:0/3840316259 shutdown_connections 2024-04-12T05:26:36.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.926+0000 7fa049207700 1 -- 172.21.15.113:0/3840316259 wait complete. 2024-04-12T05:26:36.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.926+0000 7fa049207700 1 Processor -- start 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.926+0000 7fa049207700 1 -- start start 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa049207700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0440fc7f0 0x7fa044109140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa049207700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0440fd150 0x7fa044109630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa049207700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa044109bb0 con 0x7fa0440fc7f0 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa049207700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa044109cf0 con 0x7fa0440fd150 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa042d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0440fc7f0 0x7fa044109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa042d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0440fc7f0 0x7fa044109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54856/0 (socket says 172.21.15.113:54856) 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa042d9d700 1 -- 172.21.15.113:0/467279106 learned_addr learned my addr 172.21.15.113:0/467279106 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa04259c700 1 --2- 172.21.15.113:0/467279106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0440fd150 0x7fa044109630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa042d9d700 1 -- 172.21.15.113:0/467279106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0440fd150 msgr2=0x7fa044109630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa042d9d700 1 --2- 172.21.15.113:0/467279106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0440fd150 0x7fa044109630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:36.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa042d9d700 1 -- 172.21.15.113:0/467279106 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa034005040 con 0x7fa0440fc7f0 2024-04-12T05:26:36.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa042d9d700 1 --2- 172.21.15.113:0/467279106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0440fc7f0 0x7fa044109140 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fa03400b120 tx=0x7fa034009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:36.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa03bfff700 1 -- 172.21.15.113:0/467279106 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa03400e030 con 0x7fa0440fc7f0 2024-04-12T05:26:36.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa049207700 1 -- 172.21.15.113:0/467279106 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa044109f40 con 0x7fa0440fc7f0 2024-04-12T05:26:36.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa03bfff700 1 -- 172.21.15.113:0/467279106 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa034007e90 con 0x7fa0440fc7f0 2024-04-12T05:26:36.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.927+0000 7fa049207700 1 -- 172.21.15.113:0/467279106 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa044112480 con 0x7fa0440fc7f0 2024-04-12T05:26:36.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.928+0000 7fa03bfff700 1 -- 172.21.15.113:0/467279106 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa034018070 con 0x7fa0440fc7f0 2024-04-12T05:26:36.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.928+0000 7fa049207700 1 -- 172.21.15.113:0/467279106 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04405f590 con 0x7fa0440fc7f0 2024-04-12T05:26:36.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.929+0000 7fa03bfff700 1 -- 172.21.15.113:0/467279106 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa03401d420 con 0x7fa0440fc7f0 2024-04-12T05:26:36.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.929+0000 7fa03bfff700 1 --2- 172.21.15.113:0/467279106 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa03006caa0 0x7fa03006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:26:36.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.929+0000 7fa04259c700 1 --2- 172.21.15.113:0/467279106 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa03006caa0 0x7fa03006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:26:36.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.929+0000 7fa03bfff700 1 -- 172.21.15.113:0/467279106 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa03408d300 con 0x7fa0440fc7f0 2024-04-12T05:26:36.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.930+0000 7fa04259c700 1 --2- 172.21.15.113:0/467279106 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa03006caa0 0x7fa03006ef60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa044060f00 tx=0x7fa02c008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:26:36.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:36.934+0000 7fa03bfff700 1 -- 172.21.15.113:0/467279106 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa034057320 con 0x7fa0440fc7f0 2024-04-12T05:26:37.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.199+0000 7fa049207700 1 -- 172.21.15.113:0/467279106 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa044060ce0 con 0x7fa0440fc7f0 2024-04-12T05:26:37.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.200+0000 7fa03bfff700 1 -- 172.21.15.113:0/467279106 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa03405a940 con 0x7fa0440fc7f0 2024-04-12T05:26:37.203 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:26:37.203 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:26:37.203 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:26:37.203 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:26:37.204 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:26:37.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.202+0000 7fa049207700 1 -- 172.21.15.113:0/467279106 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa03006caa0 msgr2=0x7fa03006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:37.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.202+0000 7fa049207700 1 --2- 172.21.15.113:0/467279106 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa03006caa0 0x7fa03006ef60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa044060f00 tx=0x7fa02c008040).stop 2024-04-12T05:26:37.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.202+0000 7fa049207700 1 -- 172.21.15.113:0/467279106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0440fc7f0 msgr2=0x7fa044109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:26:37.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.202+0000 7fa049207700 1 --2- 172.21.15.113:0/467279106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0440fc7f0 0x7fa044109140 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fa03400b120 tx=0x7fa034009cb0).stop 2024-04-12T05:26:37.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.202+0000 7fa049207700 1 -- 172.21.15.113:0/467279106 shutdown_connections 2024-04-12T05:26:37.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.202+0000 7fa049207700 1 --2- 172.21.15.113:0/467279106 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa03006caa0 0x7fa03006ef60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:37.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.202+0000 7fa049207700 1 --2- 172.21.15.113:0/467279106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0440fc7f0 0x7fa044109140 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:37.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.202+0000 7fa049207700 1 --2- 172.21.15.113:0/467279106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0440fd150 0x7fa044109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:26:37.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.202+0000 7fa049207700 1 -- 172.21.15.113:0/467279106 >> 172.21.15.113:0/467279106 conn(0x7fa044070590 msgr2=0x7fa0440fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:26:37.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.202+0000 7fa049207700 1 -- 172.21.15.113:0/467279106 shutdown_connections 2024-04-12T05:26:37.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:26:37.202+0000 7fa049207700 1 -- 172.21.15.113:0/467279106 wait complete. 2024-04-12T05:26:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:37 smithi113 ceph-mon[103371]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:37 smithi113 ceph-mon[103371]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:37 smithi113 ceph-mon[103371]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:37 smithi113 ceph-mon[103371]: pgmap v433: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:37.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:37 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3779996332' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:26:37.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:37 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/291001826' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:26:37.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:26:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:37 smithi192 ceph-mon[85304]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:37 smithi192 ceph-mon[85304]: from='client.34816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:37 smithi192 ceph-mon[85304]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:37 smithi192 ceph-mon[85304]: pgmap v433: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:37 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3779996332' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:26:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:37 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/291001826' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:26:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:26:38.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:38 smithi113 ceph-mon[103371]: from='client.34832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:38.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:38 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/467279106' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:26:38.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:38 smithi192 ceph-mon[85304]: from='client.34832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:26:38.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:38 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/467279106' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:26:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:39 smithi113 ceph-mon[103371]: pgmap v434: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:39 smithi192 ceph-mon[85304]: pgmap v434: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:41 smithi113 ceph-mon[103371]: pgmap v435: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:41.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:41 smithi192 ceph-mon[85304]: pgmap v435: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:43 smithi113 ceph-mon[103371]: pgmap v436: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:43.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:43 smithi192 ceph-mon[85304]: pgmap v436: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:45.402 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:45 smithi113 ceph-mon[103371]: pgmap v437: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:45 smithi192 ceph-mon[85304]: pgmap v437: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:47 smithi113 ceph-mon[103371]: pgmap v438: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:47.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:47 smithi192 ceph-mon[85304]: pgmap v438: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:49 smithi113 ceph-mon[103371]: pgmap v439: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:49.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:49 smithi192 ceph-mon[85304]: pgmap v439: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:51.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:51 smithi113 ceph-mon[103371]: pgmap v440: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:51 smithi192 ceph-mon[85304]: pgmap v440: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:26:52.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:26:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:53 smithi113 ceph-mon[103371]: pgmap v441: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:53.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:53 smithi192 ceph-mon[85304]: pgmap v441: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:55 smithi113 ceph-mon[103371]: pgmap v442: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:55.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:55 smithi192 ceph-mon[85304]: pgmap v442: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:57 smithi113 ceph-mon[103371]: pgmap v443: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:57.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:57 smithi192 ceph-mon[85304]: pgmap v443: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:26:59 smithi113 ceph-mon[103371]: pgmap v444: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:26:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:26:59 smithi192 ceph-mon[85304]: pgmap v444: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:01 smithi113 ceph-mon[103371]: pgmap v445: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:01.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:01 smithi192 ceph-mon[85304]: pgmap v445: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:03.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:03 smithi113 ceph-mon[103371]: pgmap v446: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:03.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:03 smithi192 ceph-mon[85304]: pgmap v446: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:05 smithi113 ceph-mon[103371]: pgmap v447: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:05.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:05 smithi192 ceph-mon[85304]: pgmap v447: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:07.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.313+0000 7f99d8a59700 1 -- 172.21.15.113:0/4050058435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d40fc7f0 msgr2=0x7f99d40fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:07.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.313+0000 7f99d8a59700 1 --2- 172.21.15.113:0/4050058435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d40fc7f0 0x7f99d40fcbd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f99c40047f0 tx=0x7f99c4009cb0).stop 2024-04-12T05:27:07.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.313+0000 7f99d8a59700 1 -- 172.21.15.113:0/4050058435 shutdown_connections 2024-04-12T05:27:07.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.313+0000 7f99d8a59700 1 --2- 172.21.15.113:0/4050058435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99d40fd150 0x7f99d40f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:27:07.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.313+0000 7f99d8a59700 1 --2- 172.21.15.113:0/4050058435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d40fc7f0 0x7f99d40fcbd0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:07.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.313+0000 7f99d8a59700 1 -- 172.21.15.113:0/4050058435 >> 172.21.15.113:0/4050058435 conn(0x7f99d4070590 msgr2=0x7f99d40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:07.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.313+0000 7f99d8a59700 1 -- 172.21.15.113:0/4050058435 shutdown_connections 2024-04-12T05:27:07.316 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.313+0000 7f99d8a59700 1 -- 172.21.15.113:0/4050058435 wait complete. 2024-04-12T05:27:07.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.314+0000 7f99d8a59700 1 Processor -- start 2024-04-12T05:27:07.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.314+0000 7f99d8a59700 1 -- start start 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.314+0000 7f99d8a59700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d40fc7f0 0x7f99d4109160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.314+0000 7f99d8a59700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99d40fd150 0x7f99d4109650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.314+0000 7f99d8a59700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d4109bd0 con 0x7f99d40fc7f0 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.314+0000 7f99d8a59700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d4109d10 con 0x7f99d40fd150 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.314+0000 7f99d259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d40fc7f0 0x7f99d4109160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.314+0000 7f99d259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d40fc7f0 0x7f99d4109160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39976/0 (socket says 172.21.15.113:39976) 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.314+0000 7f99d259c700 1 -- 172.21.15.113:0/3526263635 learned_addr learned my addr 172.21.15.113:0/3526263635 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.314+0000 7f99d1d9b700 1 --2- 172.21.15.113:0/3526263635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99d40fd150 0x7f99d4109650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.315+0000 7f99d259c700 1 -- 172.21.15.113:0/3526263635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99d40fd150 msgr2=0x7f99d4109650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.315+0000 7f99d259c700 1 --2- 172.21.15.113:0/3526263635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99d40fd150 0x7f99d4109650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.315+0000 7f99d259c700 1 -- 172.21.15.113:0/3526263635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c4005040 con 0x7f99d40fc7f0 2024-04-12T05:27:07.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.315+0000 7f99d259c700 1 --2- 172.21.15.113:0/3526263635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d40fc7f0 0x7f99d4109160 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f99c400b120 tx=0x7f99c4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:07.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.315+0000 7f99c37fe700 1 -- 172.21.15.113:0/3526263635 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99c400e030 con 0x7f99d40fc7f0 2024-04-12T05:27:07.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.315+0000 7f99d8a59700 1 -- 172.21.15.113:0/3526263635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d4109f60 con 0x7f99d40fc7f0 2024-04-12T05:27:07.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.315+0000 7f99c37fe700 1 -- 172.21.15.113:0/3526263635 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99c4007e90 con 0x7f99d40fc7f0 2024-04-12T05:27:07.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.315+0000 7f99c37fe700 1 -- 172.21.15.113:0/3526263635 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99c4018070 con 0x7f99d40fc7f0 2024-04-12T05:27:07.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.315+0000 7f99d8a59700 1 -- 172.21.15.113:0/3526263635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d41124a0 con 0x7f99d40fc7f0 2024-04-12T05:27:07.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.316+0000 7f99d8a59700 1 -- 172.21.15.113:0/3526263635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d405f590 con 0x7f99d40fc7f0 2024-04-12T05:27:07.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.316+0000 7f99c37fe700 1 -- 172.21.15.113:0/3526263635 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f99c401d420 con 0x7f99d40fc7f0 2024-04-12T05:27:07.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.317+0000 7f99c37fe700 1 --2- 172.21.15.113:0/3526263635 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f99bc06caa0 0x7f99bc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:07.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.317+0000 7f99c37fe700 1 -- 172.21.15.113:0/3526263635 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f99c408d2e0 con 0x7f99d40fc7f0 2024-04-12T05:27:07.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.317+0000 7f99d1d9b700 1 --2- 172.21.15.113:0/3526263635 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f99bc06caa0 0x7f99bc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:07.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.317+0000 7f99d1d9b700 1 --2- 172.21.15.113:0/3526263635 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f99bc06caa0 0x7f99bc06ef60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f99d4060ee0 tx=0x7f99c8008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:07.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.321+0000 7f99c37fe700 1 -- 172.21.15.113:0/3526263635 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f99c4057300 con 0x7f99d40fc7f0 2024-04-12T05:27:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:07 smithi113 ceph-mon[103371]: pgmap v448: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:07.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:27:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:07 smithi192 ceph-mon[85304]: pgmap v448: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:07.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:27:07.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.527+0000 7f99d8a59700 1 -- 172.21.15.113:0/3526263635 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99d4059120 con 0x7f99bc06caa0 2024-04-12T05:27:07.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.528+0000 7f99c37fe700 1 -- 172.21.15.113:0/3526263635 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f99d4059120 con 0x7f99bc06caa0 2024-04-12T05:27:07.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.530+0000 7f99d8a59700 1 -- 172.21.15.113:0/3526263635 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f99bc06caa0 msgr2=0x7f99bc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:07.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.530+0000 7f99d8a59700 1 --2- 172.21.15.113:0/3526263635 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f99bc06caa0 0x7f99bc06ef60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f99d4060ee0 tx=0x7f99c8008040).stop 2024-04-12T05:27:07.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.530+0000 7f99d8a59700 1 -- 172.21.15.113:0/3526263635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d40fc7f0 msgr2=0x7f99d4109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:07.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.530+0000 7f99d8a59700 1 --2- 172.21.15.113:0/3526263635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d40fc7f0 0x7f99d4109160 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f99c400b120 tx=0x7f99c4009cb0).stop 2024-04-12T05:27:07.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.530+0000 7f99d8a59700 1 -- 172.21.15.113:0/3526263635 shutdown_connections 2024-04-12T05:27:07.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.530+0000 7f99d8a59700 1 --2- 172.21.15.113:0/3526263635 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f99bc06caa0 0x7f99bc06ef60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:07.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.530+0000 7f99d8a59700 1 --2- 172.21.15.113:0/3526263635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d40fc7f0 0x7f99d4109160 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:07.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.530+0000 7f99d8a59700 1 --2- 172.21.15.113:0/3526263635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99d40fd150 0x7f99d4109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:07.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.530+0000 7f99d8a59700 1 -- 172.21.15.113:0/3526263635 >> 172.21.15.113:0/3526263635 conn(0x7f99d4070590 msgr2=0x7f99d40fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:07.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.530+0000 7f99d8a59700 1 -- 172.21.15.113:0/3526263635 shutdown_connections 2024-04-12T05:27:07.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.530+0000 7f99d8a59700 1 -- 172.21.15.113:0/3526263635 wait complete. 2024-04-12T05:27:07.542 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:27:07.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.629+0000 7fb0e65d1700 1 -- 172.21.15.113:0/4108334499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00f7060 msgr2=0x7fb0e0103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:07.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.629+0000 7fb0e65d1700 1 --2- 172.21.15.113:0/4108334499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00f7060 0x7fb0e0103e00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fb0c80047f0 tx=0x7fb0c8009cb0).stop 2024-04-12T05:27:07.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.629+0000 7fb0e65d1700 1 -- 172.21.15.113:0/4108334499 shutdown_connections 2024-04-12T05:27:07.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.629+0000 7fb0e65d1700 1 --2- 172.21.15.113:0/4108334499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00f7060 0x7fb0e0103e00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:07.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.629+0000 7fb0e65d1700 1 --2- 172.21.15.113:0/4108334499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0e00f6790 0x7fb0e00f6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:27:07.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.629+0000 7fb0e65d1700 1 -- 172.21.15.113:0/4108334499 >> 172.21.15.113:0/4108334499 conn(0x7fb0e00f2430 msgr2=0x7fb0e00f4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:07.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.629+0000 7fb0e65d1700 1 -- 172.21.15.113:0/4108334499 shutdown_connections 2024-04-12T05:27:07.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.629+0000 7fb0e65d1700 1 -- 172.21.15.113:0/4108334499 wait complete. 2024-04-12T05:27:07.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.629+0000 7fb0e65d1700 1 Processor -- start 2024-04-12T05:27:07.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.629+0000 7fb0e65d1700 1 -- start start 2024-04-12T05:27:07.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0e65d1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00f6790 0x7fb0e01101c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:07.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0e65d1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0e00f7060 0x7fb0e01106b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:07.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0e65d1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e0110d40 con 0x7fb0e00f6790 2024-04-12T05:27:07.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0e65d1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e010a360 con 0x7fb0e00f7060 2024-04-12T05:27:07.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0dffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00f6790 0x7fb0e01101c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:07.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0dffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00f6790 0x7fb0e01101c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39996/0 (socket says 172.21.15.113:39996) 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0dffff700 1 -- 172.21.15.113:0/1100860596 learned_addr learned my addr 172.21.15.113:0/1100860596 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0dffff700 1 -- 172.21.15.113:0/1100860596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0e00f7060 msgr2=0x7fb0e01106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0dffff700 1 --2- 172.21.15.113:0/1100860596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0e00f7060 0x7fb0e01106b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0dffff700 1 -- 172.21.15.113:0/1100860596 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c8005040 con 0x7fb0e00f6790 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0dffff700 1 --2- 172.21.15.113:0/1100860596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00f6790 0x7fb0e01101c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fb0d000b4b0 tx=0x7fb0d00090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0dd7fa700 1 -- 172.21.15.113:0/1100860596 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0d0002500 con 0x7fb0e00f6790 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0dd7fa700 1 -- 172.21.15.113:0/1100860596 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0d0002b10 con 0x7fb0e00f6790 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0e65d1700 1 -- 172.21.15.113:0/1100860596 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e010a610 con 0x7fb0e00f6790 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0dd7fa700 1 -- 172.21.15.113:0/1100860596 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0d0012cf0 con 0x7fb0e00f6790 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.630+0000 7fb0e65d1700 1 -- 172.21.15.113:0/1100860596 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e010abd0 con 0x7fb0e00f6790 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.631+0000 7fb0e65d1700 1 -- 172.21.15.113:0/1100860596 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e0060390 con 0x7fb0e00f6790 2024-04-12T05:27:07.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.632+0000 7fb0dd7fa700 1 -- 172.21.15.113:0/1100860596 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb0d0002660 con 0x7fb0e00f6790 2024-04-12T05:27:07.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.632+0000 7fb0dd7fa700 1 --2- 172.21.15.113:0/1100860596 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb0cc06caa0 0x7fb0cc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:07.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.632+0000 7fb0dd7fa700 1 -- 172.21.15.113:0/1100860596 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb0d008bf30 con 0x7fb0e00f6790 2024-04-12T05:27:07.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.632+0000 7fb0df7fe700 1 --2- 172.21.15.113:0/1100860596 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb0cc06caa0 0x7fb0cc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:07.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.633+0000 7fb0df7fe700 1 --2- 172.21.15.113:0/1100860596 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb0cc06caa0 0x7fb0cc06ef60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fb0c80047c0 tx=0x7fb0c8009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:07.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.637+0000 7fb0dd7fa700 1 -- 172.21.15.113:0/1100860596 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb0d00bf2c0 con 0x7fb0e00f6790 2024-04-12T05:27:07.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.844+0000 7fb0e65d1700 1 -- 172.21.15.113:0/1100860596 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0e010b8b0 con 0x7fb0cc06caa0 2024-04-12T05:27:07.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.845+0000 7fb0dd7fa700 1 -- 172.21.15.113:0/1100860596 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb0e010b8b0 con 0x7fb0cc06caa0 2024-04-12T05:27:07.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.847+0000 7fb0e65d1700 1 -- 172.21.15.113:0/1100860596 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb0cc06caa0 msgr2=0x7fb0cc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:07.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.847+0000 7fb0e65d1700 1 --2- 172.21.15.113:0/1100860596 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb0cc06caa0 0x7fb0cc06ef60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fb0c80047c0 tx=0x7fb0c8009cb0).stop 2024-04-12T05:27:07.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.847+0000 7fb0e65d1700 1 -- 172.21.15.113:0/1100860596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00f6790 msgr2=0x7fb0e01101c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:07.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.847+0000 7fb0e65d1700 1 --2- 172.21.15.113:0/1100860596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00f6790 0x7fb0e01101c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fb0d000b4b0 tx=0x7fb0d00090f0).stop 2024-04-12T05:27:07.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.847+0000 7fb0e65d1700 1 -- 172.21.15.113:0/1100860596 shutdown_connections 2024-04-12T05:27:07.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.847+0000 7fb0e65d1700 1 --2- 172.21.15.113:0/1100860596 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb0cc06caa0 0x7fb0cc06ef60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:07.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.847+0000 7fb0e65d1700 1 --2- 172.21.15.113:0/1100860596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0e00f6790 0x7fb0e01101c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:07.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.847+0000 7fb0e65d1700 1 --2- 172.21.15.113:0/1100860596 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0e00f7060 0x7fb0e01106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:27:07.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.847+0000 7fb0e65d1700 1 -- 172.21.15.113:0/1100860596 >> 172.21.15.113:0/1100860596 conn(0x7fb0e00f2430 msgr2=0x7fb0e00f3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:07.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.847+0000 7fb0e65d1700 1 -- 172.21.15.113:0/1100860596 shutdown_connections 2024-04-12T05:27:07.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.847+0000 7fb0e65d1700 1 -- 172.21.15.113:0/1100860596 wait complete. 2024-04-12T05:27:07.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.945+0000 7f473060c700 1 -- 172.21.15.113:0/968757427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47280fd150 msgr2=0x7f47280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:07.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.945+0000 7f473060c700 1 --2- 172.21.15.113:0/968757427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47280fd150 0x7f47280f6930 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f47240047f0 tx=0x7f4724009cb0).stop 2024-04-12T05:27:07.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.945+0000 7f473060c700 1 -- 172.21.15.113:0/968757427 shutdown_connections 2024-04-12T05:27:07.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.945+0000 7f473060c700 1 --2- 172.21.15.113:0/968757427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47280fd150 0x7f47280f6930 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:07.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.945+0000 7f473060c700 1 --2- 172.21.15.113:0/968757427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47280fc7f0 0x7f47280fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:07.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.945+0000 7f473060c700 1 -- 172.21.15.113:0/968757427 >> 172.21.15.113:0/968757427 conn(0x7f47280f23b0 msgr2=0x7f47280f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.945+0000 7f473060c700 1 -- 172.21.15.113:0/968757427 shutdown_connections 2024-04-12T05:27:07.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.945+0000 7f473060c700 1 -- 172.21.15.113:0/968757427 wait complete. 2024-04-12T05:27:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.946+0000 7f473060c700 1 Processor -- start 2024-04-12T05:27:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.946+0000 7f473060c700 1 -- start start 2024-04-12T05:27:07.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.946+0000 7f473060c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47280fc7f0 0x7f472810cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f473060c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47280fd150 0x7f472810d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f473060c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472810da60 con 0x7f47280fc7f0 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f473060c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472810dba0 con 0x7f47280fd150 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f472e3a8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47280fc7f0 0x7f472810cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f472e3a8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47280fc7f0 0x7f472810cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40014/0 (socket says 172.21.15.113:40014) 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f472e3a8700 1 -- 172.21.15.113:0/3104204454 learned_addr learned my addr 172.21.15.113:0/3104204454 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f472dba7700 1 --2- 172.21.15.113:0/3104204454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47280fd150 0x7f472810d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f472e3a8700 1 -- 172.21.15.113:0/3104204454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47280fd150 msgr2=0x7f472810d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f472e3a8700 1 --2- 172.21.15.113:0/3104204454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47280fd150 0x7f472810d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f472e3a8700 1 -- 172.21.15.113:0/3104204454 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4724005040 con 0x7f47280fc7f0 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f472e3a8700 1 --2- 172.21.15.113:0/3104204454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47280fc7f0 0x7f472810cf70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f47180041d0 tx=0x7f4718009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f471f7fe700 1 -- 172.21.15.113:0/3104204454 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4718002d90 con 0x7f47280fc7f0 2024-04-12T05:27:07.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f473060c700 1 -- 172.21.15.113:0/3104204454 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4728107230 con 0x7f47280fc7f0 2024-04-12T05:27:07.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f471f7fe700 1 -- 172.21.15.113:0/3104204454 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f471800f040 con 0x7f47280fc7f0 2024-04-12T05:27:07.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f471f7fe700 1 -- 172.21.15.113:0/3104204454 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f471800d800 con 0x7f47280fc7f0 2024-04-12T05:27:07.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.947+0000 7f473060c700 1 -- 172.21.15.113:0/3104204454 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4728107770 con 0x7f47280fc7f0 2024-04-12T05:27:07.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.948+0000 7f473060c700 1 -- 172.21.15.113:0/3104204454 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4728060370 con 0x7f47280fc7f0 2024-04-12T05:27:07.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.949+0000 7f471f7fe700 1 -- 172.21.15.113:0/3104204454 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f471800dae0 con 0x7f47280fc7f0 2024-04-12T05:27:07.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.949+0000 7f471f7fe700 1 --2- 172.21.15.113:0/3104204454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471406caa0 0x7f471406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:07.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.949+0000 7f471f7fe700 1 -- 172.21.15.113:0/3104204454 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f471808be80 con 0x7f47280fc7f0 2024-04-12T05:27:07.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.949+0000 7f472dba7700 1 --2- 172.21.15.113:0/3104204454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471406caa0 0x7f471406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:07.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.949+0000 7f472dba7700 1 --2- 172.21.15.113:0/3104204454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471406caa0 0x7f471406ef60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f4724004f10 tx=0x7f472400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:07.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:07.953+0000 7f471f7fe700 1 -- 172.21.15.113:0/3104204454 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4718059930 con 0x7f47280fc7f0 2024-04-12T05:27:08.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.149+0000 7f473060c700 1 -- 172.21.15.113:0/3104204454 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f47281079f0 con 0x7f471406caa0 2024-04-12T05:27:08.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.159+0000 7f471f7fe700 1 -- 172.21.15.113:0/3104204454 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f47281079f0 con 0x7f471406caa0 2024-04-12T05:27:08.162 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:27:08.162 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (14m) 7m ago 26m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:27:08.162 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (12m) 7m ago 26m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:27:08.162 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (12m) 7m ago 25m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:27:08.162 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (14m) 7m ago 25m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:27:08.162 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (8m) 7m ago 20m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (7m) 7m ago 20m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (7m) 7m ago 20m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (20m) 7m ago 20m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (16m) 7m ago 27m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (15m) 7m ago 25m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (13m) 7m ago 27m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (13m) 7m ago 25m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (15m) 7m ago 25m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (15m) 7m ago 25m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (11m) 7m ago 24m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (11m) 7m ago 24m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:27:08.163 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (10m) 7m ago 23m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:27:08.164 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (10m) 7m ago 23m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:27:08.164 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (9m) 7m ago 22m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:27:08.164 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (8m) 7m ago 22m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:27:08.164 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (15m) 7m ago 25m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:27:08.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.161+0000 7f473060c700 1 -- 172.21.15.113:0/3104204454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471406caa0 msgr2=0x7f471406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.161+0000 7f473060c700 1 --2- 172.21.15.113:0/3104204454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471406caa0 0x7f471406ef60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f4724004f10 tx=0x7f472400b040).stop 2024-04-12T05:27:08.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.162+0000 7f473060c700 1 -- 172.21.15.113:0/3104204454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47280fc7f0 msgr2=0x7f472810cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.162+0000 7f473060c700 1 --2- 172.21.15.113:0/3104204454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47280fc7f0 0x7f472810cf70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f47180041d0 tx=0x7f4718009f60).stop 2024-04-12T05:27:08.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.162+0000 7f473060c700 1 -- 172.21.15.113:0/3104204454 shutdown_connections 2024-04-12T05:27:08.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.162+0000 7f473060c700 1 --2- 172.21.15.113:0/3104204454 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471406caa0 0x7f471406ef60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.162+0000 7f473060c700 1 --2- 172.21.15.113:0/3104204454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47280fc7f0 0x7f472810cf70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.162+0000 7f473060c700 1 --2- 172.21.15.113:0/3104204454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47280fd150 0x7f472810d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.162+0000 7f473060c700 1 -- 172.21.15.113:0/3104204454 >> 172.21.15.113:0/3104204454 conn(0x7f47280f23b0 msgr2=0x7f47280fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:08.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.162+0000 7f473060c700 1 -- 172.21.15.113:0/3104204454 shutdown_connections 2024-04-12T05:27:08.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.162+0000 7f473060c700 1 -- 172.21.15.113:0/3104204454 wait complete. 2024-04-12T05:27:08.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.257+0000 7f454099a700 1 -- 172.21.15.113:0/3360746051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f453c0fd910 msgr2=0x7f453c0fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.257+0000 7f454099a700 1 --2- 172.21.15.113:0/3360746051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f453c0fd910 0x7f453c0fdcf0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f45300047f0 tx=0x7f4530009cb0).stop 2024-04-12T05:27:08.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.258+0000 7f454099a700 1 -- 172.21.15.113:0/3360746051 shutdown_connections 2024-04-12T05:27:08.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.258+0000 7f454099a700 1 --2- 172.21.15.113:0/3360746051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f453c0fe270 0x7f453c0f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.258+0000 7f454099a700 1 --2- 172.21.15.113:0/3360746051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f453c0fd910 0x7f453c0fdcf0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.258+0000 7f454099a700 1 -- 172.21.15.113:0/3360746051 >> 172.21.15.113:0/3360746051 conn(0x7f453c0f3510 msgr2=0x7f453c0f5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:08.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.258+0000 7f454099a700 1 -- 172.21.15.113:0/3360746051 shutdown_connections 2024-04-12T05:27:08.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.258+0000 7f454099a700 1 -- 172.21.15.113:0/3360746051 wait complete. 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.258+0000 7f454099a700 1 Processor -- start 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.258+0000 7f454099a700 1 -- start start 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f454099a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f453c0fd910 0x7f453c10e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f454099a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f453c0fe270 0x7f453c10e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f454099a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453c10eb10 con 0x7f453c0fe270 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f454099a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453c10ec50 con 0x7f453c0fd910 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f4539d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f453c0fe270 0x7f453c10e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f4539d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f453c0fe270 0x7f453c10e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40034/0 (socket says 172.21.15.113:40034) 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f4539d9b700 1 -- 172.21.15.113:0/804635400 learned_addr learned my addr 172.21.15.113:0/804635400 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f453a59c700 1 --2- 172.21.15.113:0/804635400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f453c0fd910 0x7f453c10e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f4539d9b700 1 -- 172.21.15.113:0/804635400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f453c0fd910 msgr2=0x7f453c10e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f4539d9b700 1 --2- 172.21.15.113:0/804635400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f453c0fd910 0x7f453c10e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f4539d9b700 1 -- 172.21.15.113:0/804635400 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4530005040 con 0x7f453c0fe270 2024-04-12T05:27:08.263 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f4539d9b700 1 --2- 172.21.15.113:0/804635400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f453c0fe270 0x7f453c10e510 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f452c00c790 tx=0x7f452c007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:08.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f452b7fe700 1 -- 172.21.15.113:0/804635400 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f452c0106f0 con 0x7f453c0fe270 2024-04-12T05:27:08.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f452b7fe700 1 -- 172.21.15.113:0/804635400 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f452c010d00 con 0x7f453c0fe270 2024-04-12T05:27:08.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f454099a700 1 -- 172.21.15.113:0/804635400 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453c1082e0 con 0x7f453c0fe270 2024-04-12T05:27:08.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f452b7fe700 1 -- 172.21.15.113:0/804635400 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f452c012d50 con 0x7f453c0fe270 2024-04-12T05:27:08.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.259+0000 7f454099a700 1 -- 172.21.15.113:0/804635400 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453c1088a0 con 0x7f453c0fe270 2024-04-12T05:27:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.260+0000 7f454099a700 1 -- 172.21.15.113:0/804635400 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f453c060650 con 0x7f453c0fe270 2024-04-12T05:27:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.261+0000 7f452b7fe700 1 -- 172.21.15.113:0/804635400 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f452c012570 con 0x7f453c0fe270 2024-04-12T05:27:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.261+0000 7f452b7fe700 1 --2- 172.21.15.113:0/804635400 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f452406caa0 0x7f452406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.261+0000 7f452b7fe700 1 -- 172.21.15.113:0/804635400 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f452c08b740 con 0x7f453c0fe270 2024-04-12T05:27:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.261+0000 7f453a59c700 1 --2- 172.21.15.113:0/804635400 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f452406caa0 0x7f452406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:08.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.262+0000 7f453a59c700 1 --2- 172.21.15.113:0/804635400 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f452406caa0 0x7f452406ef60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f4530004f10 tx=0x7f453000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:08.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.266+0000 7f452b7fe700 1 -- 172.21.15.113:0/804635400 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f452c0591f0 con 0x7f453c0fe270 2024-04-12T05:27:08.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:08 smithi113 ceph-mon[103371]: from='client.34840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:08.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:08 smithi113 ceph-mon[103371]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:08.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:08 smithi113 ceph-mon[103371]: pgmap v449: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:08.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:08 smithi192 ceph-mon[85304]: from='client.34840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:08.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:08 smithi192 ceph-mon[85304]: from='client.34844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:08.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:08 smithi192 ceph-mon[85304]: pgmap v449: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:08.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.543+0000 7f454099a700 1 -- 172.21.15.113:0/804635400 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f453c047540 con 0x7f453c0fe270 2024-04-12T05:27:08.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.543+0000 7f452b7fe700 1 -- 172.21.15.113:0/804635400 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f452c05f070 con 0x7f453c0fe270 2024-04-12T05:27:08.546 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:27:08.546 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:27:08.546 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:27:08.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:27:08.546 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:27:08.546 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:27:08.546 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:27:08.547 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:27:08.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.545+0000 7f454099a700 1 -- 172.21.15.113:0/804635400 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f452406caa0 msgr2=0x7f452406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.545+0000 7f454099a700 1 --2- 172.21.15.113:0/804635400 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f452406caa0 0x7f452406ef60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f4530004f10 tx=0x7f453000b040).stop 2024-04-12T05:27:08.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.545+0000 7f454099a700 1 -- 172.21.15.113:0/804635400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f453c0fe270 msgr2=0x7f453c10e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.545+0000 7f454099a700 1 --2- 172.21.15.113:0/804635400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f453c0fe270 0x7f453c10e510 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f452c00c790 tx=0x7f452c007f50).stop 2024-04-12T05:27:08.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.545+0000 7f454099a700 1 -- 172.21.15.113:0/804635400 shutdown_connections 2024-04-12T05:27:08.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.545+0000 7f454099a700 1 --2- 172.21.15.113:0/804635400 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f452406caa0 0x7f452406ef60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.545+0000 7f454099a700 1 --2- 172.21.15.113:0/804635400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f453c0fd910 0x7f453c10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.545+0000 7f454099a700 1 --2- 172.21.15.113:0/804635400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f453c0fe270 0x7f453c10e510 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.545+0000 7f454099a700 1 -- 172.21.15.113:0/804635400 >> 172.21.15.113:0/804635400 conn(0x7f453c0f3510 msgr2=0x7f453c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:08.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.545+0000 7f454099a700 1 -- 172.21.15.113:0/804635400 shutdown_connections 2024-04-12T05:27:08.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.545+0000 7f454099a700 1 -- 172.21.15.113:0/804635400 wait complete. 2024-04-12T05:27:08.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.643+0000 7f029ff70700 1 -- 172.21.15.113:0/316536293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02980fd150 msgr2=0x7f02980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.643+0000 7f029ff70700 1 --2- 172.21.15.113:0/316536293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02980fd150 0x7f02980f6930 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f02880047f0 tx=0x7f0288009cb0).stop 2024-04-12T05:27:08.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.643+0000 7f029ff70700 1 -- 172.21.15.113:0/316536293 shutdown_connections 2024-04-12T05:27:08.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.643+0000 7f029ff70700 1 --2- 172.21.15.113:0/316536293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02980fd150 0x7f02980f6930 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.643+0000 7f029ff70700 1 --2- 172.21.15.113:0/316536293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02980fc7f0 0x7f02980fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.643+0000 7f029ff70700 1 -- 172.21.15.113:0/316536293 >> 172.21.15.113:0/316536293 conn(0x7f0298070590 msgr2=0x7f02980f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:08.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.643+0000 7f029ff70700 1 -- 172.21.15.113:0/316536293 shutdown_connections 2024-04-12T05:27:08.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.643+0000 7f029ff70700 1 -- 172.21.15.113:0/316536293 wait complete. 2024-04-12T05:27:08.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029ff70700 1 Processor -- start 2024-04-12T05:27:08.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029ff70700 1 -- start start 2024-04-12T05:27:08.650 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029ff70700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02980fc7f0 0x7f02981090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029ff70700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02980fd150 0x7f02981095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029ff70700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0298109aa0 con 0x7f02980fc7f0 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029ff70700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0298109be0 con 0x7f02980fd150 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029dd0c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02980fc7f0 0x7f02981090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029dd0c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02980fc7f0 0x7f02981090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40046/0 (socket says 172.21.15.113:40046) 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029dd0c700 1 -- 172.21.15.113:0/1639639209 learned_addr learned my addr 172.21.15.113:0/1639639209 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029d50b700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02980fd150 0x7f02981095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029dd0c700 1 -- 172.21.15.113:0/1639639209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02980fd150 msgr2=0x7f02981095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029dd0c700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02980fd150 0x7f02981095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029dd0c700 1 -- 172.21.15.113:0/1639639209 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0288005040 con 0x7f02980fc7f0 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.644+0000 7f029d50b700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02980fd150 0x7f02981095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.645+0000 7f029dd0c700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02980fc7f0 0x7f02981090c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f029400c790 tx=0x7f0294004330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:08.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.645+0000 7f028effd700 1 -- 172.21.15.113:0/1639639209 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02940106f0 con 0x7f02980fc7f0 2024-04-12T05:27:08.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.645+0000 7f028effd700 1 -- 172.21.15.113:0/1639639209 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0294010d00 con 0x7f02980fc7f0 2024-04-12T05:27:08.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.645+0000 7f029ff70700 1 -- 172.21.15.113:0/1639639209 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0298109e90 con 0x7f02980fc7f0 2024-04-12T05:27:08.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.645+0000 7f028effd700 1 -- 172.21.15.113:0/1639639209 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0294012da0 con 0x7f02980fc7f0 2024-04-12T05:27:08.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.645+0000 7f029ff70700 1 -- 172.21.15.113:0/1639639209 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0298112480 con 0x7f02980fc7f0 2024-04-12T05:27:08.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.646+0000 7f029ff70700 1 -- 172.21.15.113:0/1639639209 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029805f590 con 0x7f02980fc7f0 2024-04-12T05:27:08.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.646+0000 7f028effd700 1 -- 172.21.15.113:0/1639639209 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0294012400 con 0x7f02980fc7f0 2024-04-12T05:27:08.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.647+0000 7f028effd700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f028406caa0 0x7f028406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:08.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.647+0000 7f028effd700 1 -- 172.21.15.113:0/1639639209 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f029408b6d0 con 0x7f02980fc7f0 2024-04-12T05:27:08.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.647+0000 7f029d50b700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f028406caa0 0x7f028406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:08.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.647+0000 7f029d50b700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f028406caa0 0x7f028406ef60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f0288004f10 tx=0x7f028800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:08.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.651+0000 7f028effd700 1 -- 172.21.15.113:0/1639639209 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f02940556f0 con 0x7f02980fc7f0 2024-04-12T05:27:08.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.882+0000 7f029ff70700 1 -- 172.21.15.113:0/1639639209 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0298047540 con 0x7f02980fc7f0 2024-04-12T05:27:08.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.882+0000 7f028effd700 1 -- 172.21.15.113:0/1639639209 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f0294058d10 con 0x7f02980fc7f0 2024-04-12T05:27:08.885 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:27:08.885 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:27:08.885 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:27:08.885 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:27:08.885 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:27:08.886 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:27:08.887 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:27:08.888 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:27:08.888 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:27:08.888 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:27:08.888 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:27:08.888 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:27:08.888 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:27:08.888 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:27:08.888 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:27:08.888 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:27:08.888 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:27:08.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.884+0000 7f029ff70700 1 -- 172.21.15.113:0/1639639209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f028406caa0 msgr2=0x7f028406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.884+0000 7f029ff70700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f028406caa0 0x7f028406ef60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f0288004f10 tx=0x7f028800b040).stop 2024-04-12T05:27:08.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.884+0000 7f029ff70700 1 -- 172.21.15.113:0/1639639209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02980fc7f0 msgr2=0x7f02981090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.884+0000 7f029ff70700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02980fc7f0 0x7f02981090c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f029400c790 tx=0x7f0294004330).stop 2024-04-12T05:27:08.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.884+0000 7f029ff70700 1 -- 172.21.15.113:0/1639639209 shutdown_connections 2024-04-12T05:27:08.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.885+0000 7f029ff70700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f028406caa0 0x7f028406ef60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.885+0000 7f029ff70700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f02980fc7f0 0x7f02981090c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.885+0000 7f029ff70700 1 --2- 172.21.15.113:0/1639639209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02980fd150 0x7f02981095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.885+0000 7f029ff70700 1 -- 172.21.15.113:0/1639639209 >> 172.21.15.113:0/1639639209 conn(0x7f0298070590 msgr2=0x7f02980fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:08.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.885+0000 7f029ff70700 1 -- 172.21.15.113:0/1639639209 shutdown_connections 2024-04-12T05:27:08.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.885+0000 7f029ff70700 1 -- 172.21.15.113:0/1639639209 wait complete. 2024-04-12T05:27:08.890 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:27:08.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.984+0000 7f872f1af700 1 -- 172.21.15.113:0/2704268133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87280fd150 msgr2=0x7f87280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.984+0000 7f872f1af700 1 --2- 172.21.15.113:0/2704268133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87280fd150 0x7f87280f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f87140047f0 tx=0x7f8714009cb0).stop 2024-04-12T05:27:08.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.984+0000 7f872f1af700 1 -- 172.21.15.113:0/2704268133 shutdown_connections 2024-04-12T05:27:08.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.984+0000 7f872f1af700 1 --2- 172.21.15.113:0/2704268133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87280fd150 0x7f87280f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.984+0000 7f872f1af700 1 --2- 172.21.15.113:0/2704268133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87280fc7f0 0x7f87280fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.984+0000 7f872f1af700 1 -- 172.21.15.113:0/2704268133 >> 172.21.15.113:0/2704268133 conn(0x7f8728070590 msgr2=0x7f87280f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:08.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.984+0000 7f872f1af700 1 -- 172.21.15.113:0/2704268133 shutdown_connections 2024-04-12T05:27:08.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.984+0000 7f872f1af700 1 -- 172.21.15.113:0/2704268133 wait complete. 2024-04-12T05:27:08.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.984+0000 7f872f1af700 1 Processor -- start 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.984+0000 7f872f1af700 1 -- start start 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f872f1af700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87280fc7f0 0x7f872810cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f872f1af700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87280fd150 0x7f872810d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f872f1af700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872810d9e0 con 0x7f87280fd150 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f872f1af700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872810db20 con 0x7f87280fc7f0 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f8727fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87280fd150 0x7f872810d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f8727fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87280fd150 0x7f872810d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40050/0 (socket says 172.21.15.113:40050) 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f8727fff700 1 -- 172.21.15.113:0/2966045899 learned_addr learned my addr 172.21.15.113:0/2966045899 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f8727fff700 1 -- 172.21.15.113:0/2966045899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87280fc7f0 msgr2=0x7f872810cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f872cf4b700 1 --2- 172.21.15.113:0/2966045899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87280fc7f0 0x7f872810cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f8727fff700 1 --2- 172.21.15.113:0/2966045899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87280fc7f0 0x7f872810cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f8727fff700 1 -- 172.21.15.113:0/2966045899 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8714005040 con 0x7f87280fd150 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f8727fff700 1 --2- 172.21.15.113:0/2966045899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87280fd150 0x7f872810d3e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f87140099f0 tx=0x7f8714009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f8725ffb700 1 -- 172.21.15.113:0/2966045899 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f871400e030 con 0x7f87280fd150 2024-04-12T05:27:08.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f8725ffb700 1 -- 172.21.15.113:0/2966045899 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8714007e90 con 0x7f87280fd150 2024-04-12T05:27:08.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f872f1af700 1 -- 172.21.15.113:0/2966045899 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8728107150 con 0x7f87280fd150 2024-04-12T05:27:08.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.985+0000 7f8725ffb700 1 -- 172.21.15.113:0/2966045899 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8714018070 con 0x7f87280fd150 2024-04-12T05:27:08.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.986+0000 7f872f1af700 1 -- 172.21.15.113:0/2966045899 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87281076b0 con 0x7f87280fd150 2024-04-12T05:27:08.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.986+0000 7f872f1af700 1 -- 172.21.15.113:0/2966045899 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f872805f590 con 0x7f87280fd150 2024-04-12T05:27:08.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.987+0000 7f8725ffb700 1 -- 172.21.15.113:0/2966045899 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f871401d420 con 0x7f87280fd150 2024-04-12T05:27:08.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.987+0000 7f8725ffb700 1 --2- 172.21.15.113:0/2966045899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f871806caa0 0x7f871806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:08.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.988+0000 7f872cf4b700 1 --2- 172.21.15.113:0/2966045899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f871806caa0 0x7f871806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:08.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.988+0000 7f8725ffb700 1 -- 172.21.15.113:0/2966045899 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f871408d3b0 con 0x7f87280fd150 2024-04-12T05:27:08.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.988+0000 7f872cf4b700 1 --2- 172.21.15.113:0/2966045899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f871806caa0 0x7f871806ef60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f871c0041d0 tx=0x7f871c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:08.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:08.992+0000 7f8725ffb700 1 -- 172.21.15.113:0/2966045899 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f871405ada0 con 0x7f87280fd150 2024-04-12T05:27:09.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.196+0000 7f872f1af700 1 -- 172.21.15.113:0/2966045899 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f87280feb40 con 0x7f871806caa0 2024-04-12T05:27:09.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.197+0000 7f8725ffb700 1 -- 172.21.15.113:0/2966045899 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f87280feb40 con 0x7f871806caa0 2024-04-12T05:27:09.200 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:27:09.200 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:27:09.200 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:27:09.200 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:27:09.200 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:27:09.200 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:27:09.200 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:27:09.200 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:27:09.200 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:27:09.201 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:27:09.201 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:27:09.201 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:27:09.201 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:27:09.201 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:27:09.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.199+0000 7f872f1af700 1 -- 172.21.15.113:0/2966045899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f871806caa0 msgr2=0x7f871806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:09.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.199+0000 7f872f1af700 1 --2- 172.21.15.113:0/2966045899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f871806caa0 0x7f871806ef60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f871c0041d0 tx=0x7f871c009040).stop 2024-04-12T05:27:09.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.199+0000 7f872f1af700 1 -- 172.21.15.113:0/2966045899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87280fd150 msgr2=0x7f872810d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:09.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.199+0000 7f872f1af700 1 --2- 172.21.15.113:0/2966045899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87280fd150 0x7f872810d3e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f87140099f0 tx=0x7f8714009cb0).stop 2024-04-12T05:27:09.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.199+0000 7f872f1af700 1 -- 172.21.15.113:0/2966045899 shutdown_connections 2024-04-12T05:27:09.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.199+0000 7f872f1af700 1 --2- 172.21.15.113:0/2966045899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f871806caa0 0x7f871806ef60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:09.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.199+0000 7f872f1af700 1 --2- 172.21.15.113:0/2966045899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87280fc7f0 0x7f872810cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:09.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.199+0000 7f872f1af700 1 --2- 172.21.15.113:0/2966045899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87280fd150 0x7f872810d3e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:09.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.200+0000 7f872f1af700 1 -- 172.21.15.113:0/2966045899 >> 172.21.15.113:0/2966045899 conn(0x7f8728070590 msgr2=0x7f87280fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:09.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.200+0000 7f872f1af700 1 -- 172.21.15.113:0/2966045899 shutdown_connections 2024-04-12T05:27:09.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.200+0000 7f872f1af700 1 -- 172.21.15.113:0/2966045899 wait complete. 2024-04-12T05:27:09.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.297+0000 7fa41dfcb700 1 -- 172.21.15.113:0/183768766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f9200 msgr2=0x7fa418101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:09.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.297+0000 7fa41dfcb700 1 --2- 172.21.15.113:0/183768766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f9200 0x7fa418101910 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fa40c0047f0 tx=0x7fa40c009cb0).stop 2024-04-12T05:27:09.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.297+0000 7fa41dfcb700 1 -- 172.21.15.113:0/183768766 shutdown_connections 2024-04-12T05:27:09.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.297+0000 7fa41dfcb700 1 --2- 172.21.15.113:0/183768766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f9200 0x7fa418101910 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:09.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.297+0000 7fa41dfcb700 1 --2- 172.21.15.113:0/183768766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f88a0 0x7fa4180f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:09.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.297+0000 7fa41dfcb700 1 -- 172.21.15.113:0/183768766 >> 172.21.15.113:0/183768766 conn(0x7fa4180f23b0 msgr2=0x7fa4180f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:09.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.297+0000 7fa41dfcb700 1 -- 172.21.15.113:0/183768766 shutdown_connections 2024-04-12T05:27:09.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.297+0000 7fa41dfcb700 1 -- 172.21.15.113:0/183768766 wait complete. 2024-04-12T05:27:09.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.298+0000 7fa41dfcb700 1 Processor -- start 2024-04-12T05:27:09.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.298+0000 7fa41dfcb700 1 -- start start 2024-04-12T05:27:09.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.298+0000 7fa41dfcb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f88a0 0x7fa41810e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.298+0000 7fa41dfcb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f9200 0x7fa41810e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.298+0000 7fa41dfcb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41810ec40 con 0x7fa4180f88a0 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.298+0000 7fa41dfcb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41810ed80 con 0x7fa4180f9200 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.298+0000 7fa4177fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f88a0 0x7fa41810e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa4177fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f88a0 0x7fa41810e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40070/0 (socket says 172.21.15.113:40070) 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa4177fe700 1 -- 172.21.15.113:0/234465995 learned_addr learned my addr 172.21.15.113:0/234465995 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa416ffd700 1 --2- 172.21.15.113:0/234465995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f9200 0x7fa41810e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa4177fe700 1 -- 172.21.15.113:0/234465995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f9200 msgr2=0x7fa41810e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa4177fe700 1 --2- 172.21.15.113:0/234465995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f9200 0x7fa41810e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa4177fe700 1 -- 172.21.15.113:0/234465995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40c005040 con 0x7fa4180f88a0 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa4177fe700 1 --2- 172.21.15.113:0/234465995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f88a0 0x7fa41810e150 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fa40800e8f0 tx=0x7fa40800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa414ff9700 1 -- 172.21.15.113:0/234465995 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa40800ee90 con 0x7fa4180f88a0 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa41dfcb700 1 -- 172.21.15.113:0/234465995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa418108410 con 0x7fa4180f88a0 2024-04-12T05:27:09.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa414ff9700 1 -- 172.21.15.113:0/234465995 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa40800f040 con 0x7fa4180f88a0 2024-04-12T05:27:09.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa414ff9700 1 -- 172.21.15.113:0/234465995 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa40800c8a0 con 0x7fa4180f88a0 2024-04-12T05:27:09.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.299+0000 7fa41dfcb700 1 -- 172.21.15.113:0/234465995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa418108950 con 0x7fa4180f88a0 2024-04-12T05:27:09.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.300+0000 7fa41dfcb700 1 -- 172.21.15.113:0/234465995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa418060390 con 0x7fa4180f88a0 2024-04-12T05:27:09.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.300+0000 7fa414ff9700 1 -- 172.21.15.113:0/234465995 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa408007b40 con 0x7fa4180f88a0 2024-04-12T05:27:09.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.301+0000 7fa414ff9700 1 --2- 172.21.15.113:0/234465995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa40006caa0 0x7fa40006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:09.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.301+0000 7fa414ff9700 1 -- 172.21.15.113:0/234465995 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa40808bb10 con 0x7fa4180f88a0 2024-04-12T05:27:09.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.301+0000 7fa416ffd700 1 --2- 172.21.15.113:0/234465995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa40006caa0 0x7fa40006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:09.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.301+0000 7fa416ffd700 1 --2- 172.21.15.113:0/234465995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa40006caa0 0x7fa40006ef60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fa40c000c00 tx=0x7fa40c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:09.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.305+0000 7fa414ff9700 1 -- 172.21.15.113:0/234465995 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa4080595c0 con 0x7fa4180f88a0 2024-04-12T05:27:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:09 smithi113 ceph-mon[103371]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:09 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/804635400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:27:09.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:09 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1639639209' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:27:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:09 smithi192 ceph-mon[85304]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:09.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:09 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/804635400' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:27:09.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:09 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1639639209' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:27:09.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.570+0000 7fa41dfcb700 1 -- 172.21.15.113:0/234465995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa418047540 con 0x7fa4180f88a0 2024-04-12T05:27:09.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.571+0000 7fa414ff9700 1 -- 172.21.15.113:0/234465995 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa408059150 con 0x7fa4180f88a0 2024-04-12T05:27:09.574 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:27:09.574 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:27:09.574 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:27:09.574 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:27:09.574 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:27:09.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.573+0000 7fa41dfcb700 1 -- 172.21.15.113:0/234465995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa40006caa0 msgr2=0x7fa40006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:09.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.573+0000 7fa41dfcb700 1 --2- 172.21.15.113:0/234465995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa40006caa0 0x7fa40006ef60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fa40c000c00 tx=0x7fa40c00b040).stop 2024-04-12T05:27:09.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.573+0000 7fa41dfcb700 1 -- 172.21.15.113:0/234465995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f88a0 msgr2=0x7fa41810e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:09.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.573+0000 7fa41dfcb700 1 --2- 172.21.15.113:0/234465995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f88a0 0x7fa41810e150 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fa40800e8f0 tx=0x7fa40800b040).stop 2024-04-12T05:27:09.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.573+0000 7fa41dfcb700 1 -- 172.21.15.113:0/234465995 shutdown_connections 2024-04-12T05:27:09.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.573+0000 7fa41dfcb700 1 --2- 172.21.15.113:0/234465995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa40006caa0 0x7fa40006ef60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:09.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.573+0000 7fa41dfcb700 1 --2- 172.21.15.113:0/234465995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4180f88a0 0x7fa41810e150 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:09.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.573+0000 7fa41dfcb700 1 --2- 172.21.15.113:0/234465995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4180f9200 0x7fa41810e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:09.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.573+0000 7fa41dfcb700 1 -- 172.21.15.113:0/234465995 >> 172.21.15.113:0/234465995 conn(0x7fa4180f23b0 msgr2=0x7fa418100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:09.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.573+0000 7fa41dfcb700 1 -- 172.21.15.113:0/234465995 shutdown_connections 2024-04-12T05:27:09.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:09.573+0000 7fa41dfcb700 1 -- 172.21.15.113:0/234465995 wait complete. 2024-04-12T05:27:10.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:10 smithi113 ceph-mon[103371]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:10.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:10 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/234465995' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:27:10.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:10 smithi113 ceph-mon[103371]: pgmap v450: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:10.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:10 smithi192 ceph-mon[85304]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:10.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:10 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/234465995' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:27:10.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:10 smithi192 ceph-mon[85304]: pgmap v450: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:13 smithi113 ceph-mon[103371]: pgmap v451: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:13.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:13 smithi192 ceph-mon[85304]: pgmap v451: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:15 smithi113 ceph-mon[103371]: pgmap v452: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:15.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:15 smithi192 ceph-mon[85304]: pgmap v452: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:17 smithi113 ceph-mon[103371]: pgmap v453: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:17.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:17 smithi192 ceph-mon[85304]: pgmap v453: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:19 smithi113 ceph-mon[103371]: pgmap v454: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:19.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:19 smithi192 ceph-mon[85304]: pgmap v454: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:21 smithi113 ceph-mon[103371]: pgmap v455: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:21.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:21 smithi192 ceph-mon[85304]: pgmap v455: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:27:22.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:27:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:23 smithi113 ceph-mon[103371]: pgmap v456: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:23 smithi192 ceph-mon[85304]: pgmap v456: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:25 smithi113 ceph-mon[103371]: pgmap v457: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:25 smithi192 ceph-mon[85304]: pgmap v457: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:27.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:27 smithi113 ceph-mon[103371]: pgmap v458: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:27.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:27 smithi192 ceph-mon[85304]: pgmap v458: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:29.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:29 smithi113 ceph-mon[103371]: pgmap v459: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:29 smithi192 ceph-mon[85304]: pgmap v459: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:31 smithi113 ceph-mon[103371]: pgmap v460: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:31.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:31 smithi192 ceph-mon[85304]: pgmap v460: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:33.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:33 smithi113 ceph-mon[103371]: pgmap v461: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:33.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:33 smithi192 ceph-mon[85304]: pgmap v461: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:35.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:35 smithi192 ceph-mon[85304]: pgmap v462: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:35.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:35 smithi113 ceph-mon[103371]: pgmap v462: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:37 smithi113 ceph-mon[103371]: pgmap v463: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:37.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:27:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:37 smithi192 ceph-mon[85304]: pgmap v463: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:37.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:27:39.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:39 smithi113 ceph-mon[103371]: pgmap v464: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:39.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:39 smithi192 ceph-mon[85304]: pgmap v464: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:39.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.683+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1453852051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a480fe270 msgr2=0x7f9a480f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:39.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.683+0000 7f9a4ea96700 1 --2- 172.21.15.113:0/1453852051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a480fe270 0x7f9a480f7a50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f9a340047f0 tx=0x7f9a34009cb0).stop 2024-04-12T05:27:39.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.684+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1453852051 shutdown_connections 2024-04-12T05:27:39.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.684+0000 7f9a4ea96700 1 --2- 172.21.15.113:0/1453852051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a480fe270 0x7f9a480f7a50 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:39.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.684+0000 7f9a4ea96700 1 --2- 172.21.15.113:0/1453852051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a480fd910 0x7f9a480fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:39.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.684+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1453852051 >> 172.21.15.113:0/1453852051 conn(0x7f9a480f3530 msgr2=0x7f9a480f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:39.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.684+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1453852051 shutdown_connections 2024-04-12T05:27:39.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.684+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1453852051 wait complete. 2024-04-12T05:27:39.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.684+0000 7f9a4ea96700 1 Processor -- start 2024-04-12T05:27:39.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.684+0000 7f9a4ea96700 1 -- start start 2024-04-12T05:27:39.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.684+0000 7f9a4ea96700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a480fd910 0x7f9a4806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:39.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a4ea96700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a480fe270 0x7f9a4806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:39.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a4ea96700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4806bab0 con 0x7f9a480fe270 2024-04-12T05:27:39.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a4ea96700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4806bbf0 con 0x7f9a480fd910 2024-04-12T05:27:39.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a47fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a480fe270 0x7f9a4806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:39.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a47fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a480fe270 0x7f9a4806b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41220/0 (socket says 172.21.15.113:41220) 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a47fff700 1 -- 172.21.15.113:0/1854806661 learned_addr learned my addr 172.21.15.113:0/1854806661 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a4c832700 1 --2- 172.21.15.113:0/1854806661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a480fd910 0x7f9a4806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a47fff700 1 -- 172.21.15.113:0/1854806661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a480fd910 msgr2=0x7f9a4806b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a47fff700 1 --2- 172.21.15.113:0/1854806661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a480fd910 0x7f9a4806b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a47fff700 1 -- 172.21.15.113:0/1854806661 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a34005040 con 0x7f9a480fe270 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a47fff700 1 --2- 172.21.15.113:0/1854806661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a480fe270 0x7f9a4806b5c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f9a340099f0 tx=0x7f9a34009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a45f76700 1 -- 172.21.15.113:0/1854806661 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a3400e030 con 0x7f9a480fe270 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a45f76700 1 -- 172.21.15.113:0/1854806661 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a34007e90 con 0x7f9a480fe270 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1854806661 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4806be40 con 0x7f9a480fe270 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.685+0000 7f9a45f76700 1 -- 172.21.15.113:0/1854806661 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a34018070 con 0x7f9a480fe270 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.686+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1854806661 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a48114600 con 0x7f9a480fe270 2024-04-12T05:27:39.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.687+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1854806661 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4805f570 con 0x7f9a480fe270 2024-04-12T05:27:39.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.687+0000 7f9a45f76700 1 -- 172.21.15.113:0/1854806661 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9a3401d420 con 0x7f9a480fe270 2024-04-12T05:27:39.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.687+0000 7f9a45f76700 1 --2- 172.21.15.113:0/1854806661 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a3806caa0 0x7f9a3806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:39.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.687+0000 7f9a45f76700 1 -- 172.21.15.113:0/1854806661 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9a3408d2f0 con 0x7f9a480fe270 2024-04-12T05:27:39.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.687+0000 7f9a4c832700 1 --2- 172.21.15.113:0/1854806661 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a3806caa0 0x7f9a3806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:39.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.688+0000 7f9a4c832700 1 --2- 172.21.15.113:0/1854806661 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a3806caa0 0x7f9a3806ef60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f9a3c004b60 tx=0x7f9a3c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:39.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.692+0000 7f9a45f76700 1 -- 172.21.15.113:0/1854806661 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9a34057310 con 0x7f9a480fe270 2024-04-12T05:27:39.899 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.897+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1854806661 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a48059120 con 0x7f9a3806caa0 2024-04-12T05:27:39.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.898+0000 7f9a45f76700 1 -- 172.21.15.113:0/1854806661 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9a48059120 con 0x7f9a3806caa0 2024-04-12T05:27:39.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.900+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1854806661 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a3806caa0 msgr2=0x7f9a3806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:39.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.900+0000 7f9a4ea96700 1 --2- 172.21.15.113:0/1854806661 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a3806caa0 0x7f9a3806ef60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f9a3c004b60 tx=0x7f9a3c009040).stop 2024-04-12T05:27:39.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.900+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1854806661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a480fe270 msgr2=0x7f9a4806b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:39.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.900+0000 7f9a4ea96700 1 --2- 172.21.15.113:0/1854806661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a480fe270 0x7f9a4806b5c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f9a340099f0 tx=0x7f9a34009cb0).stop 2024-04-12T05:27:39.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.901+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1854806661 shutdown_connections 2024-04-12T05:27:39.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.901+0000 7f9a4ea96700 1 --2- 172.21.15.113:0/1854806661 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a3806caa0 0x7f9a3806ef60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:39.903 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.901+0000 7f9a4ea96700 1 --2- 172.21.15.113:0/1854806661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a480fd910 0x7f9a4806b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:39.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.901+0000 7f9a4ea96700 1 --2- 172.21.15.113:0/1854806661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a480fe270 0x7f9a4806b5c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:39.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.901+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1854806661 >> 172.21.15.113:0/1854806661 conn(0x7f9a480f3530 msgr2=0x7f9a480fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:39.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.901+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1854806661 shutdown_connections 2024-04-12T05:27:39.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:39.901+0000 7f9a4ea96700 1 -- 172.21.15.113:0/1854806661 wait complete. 2024-04-12T05:27:39.913 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:27:40.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.000+0000 7ff67a9c4700 1 -- 172.21.15.113:0/3033384073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6740fda50 msgr2=0x7ff6740fde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.000+0000 7ff67a9c4700 1 --2- 172.21.15.113:0/3033384073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6740fda50 0x7ff6740fde30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7ff6640047f0 tx=0x7ff664009cb0).stop 2024-04-12T05:27:40.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.000+0000 7ff67a9c4700 1 -- 172.21.15.113:0/3033384073 shutdown_connections 2024-04-12T05:27:40.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.000+0000 7ff67a9c4700 1 --2- 172.21.15.113:0/3033384073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6740fe3b0 0x7ff6740f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.000+0000 7ff67a9c4700 1 --2- 172.21.15.113:0/3033384073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6740fda50 0x7ff6740fde30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.000+0000 7ff67a9c4700 1 -- 172.21.15.113:0/3033384073 >> 172.21.15.113:0/3033384073 conn(0x7ff67406ce50 msgr2=0x7ff67406d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:40.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.000+0000 7ff67a9c4700 1 -- 172.21.15.113:0/3033384073 shutdown_connections 2024-04-12T05:27:40.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.000+0000 7ff67a9c4700 1 -- 172.21.15.113:0/3033384073 wait complete. 2024-04-12T05:27:40.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff67a9c4700 1 Processor -- start 2024-04-12T05:27:40.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff67a9c4700 1 -- start start 2024-04-12T05:27:40.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff67a9c4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6740fda50 0x7ff67410e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:40.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff67a9c4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6740fe3b0 0x7ff67410e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:40.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff67a9c4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67410eb70 con 0x7ff6740fe3b0 2024-04-12T05:27:40.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff67a9c4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67410ecb0 con 0x7ff6740fda50 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff6737fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6740fe3b0 0x7ff67410e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff6737fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6740fe3b0 0x7ff67410e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41230/0 (socket says 172.21.15.113:41230) 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff6737fe700 1 -- 172.21.15.113:0/828242409 learned_addr learned my addr 172.21.15.113:0/828242409 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff673fff700 1 --2- 172.21.15.113:0/828242409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6740fda50 0x7ff67410e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff6737fe700 1 -- 172.21.15.113:0/828242409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6740fda50 msgr2=0x7ff67410e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff6737fe700 1 --2- 172.21.15.113:0/828242409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6740fda50 0x7ff67410e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.001+0000 7ff6737fe700 1 -- 172.21.15.113:0/828242409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664005040 con 0x7ff6740fe3b0 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.002+0000 7ff6737fe700 1 --2- 172.21.15.113:0/828242409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6740fe3b0 0x7ff67410e570 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff65c0041d0 tx=0x7ff65c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.002+0000 7ff6717fa700 1 -- 172.21.15.113:0/828242409 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff65c007b40 con 0x7ff6740fe3b0 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.002+0000 7ff6717fa700 1 -- 172.21.15.113:0/828242409 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff65c00f040 con 0x7ff6740fe3b0 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.002+0000 7ff67a9c4700 1 -- 172.21.15.113:0/828242409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff674108340 con 0x7ff6740fe3b0 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.002+0000 7ff6717fa700 1 -- 172.21.15.113:0/828242409 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff65c0137d0 con 0x7ff6740fe3b0 2024-04-12T05:27:40.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.002+0000 7ff67a9c4700 1 -- 172.21.15.113:0/828242409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff674108880 con 0x7ff6740fe3b0 2024-04-12T05:27:40.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.003+0000 7ff67a9c4700 1 -- 172.21.15.113:0/828242409 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff674047540 con 0x7ff6740fe3b0 2024-04-12T05:27:40.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.003+0000 7ff6717fa700 1 -- 172.21.15.113:0/828242409 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff65c007ca0 con 0x7ff6740fe3b0 2024-04-12T05:27:40.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.003+0000 7ff6717fa700 1 --2- 172.21.15.113:0/828242409 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff66006caa0 0x7ff66006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:40.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.004+0000 7ff6717fa700 1 -- 172.21.15.113:0/828242409 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff65c08b860 con 0x7ff6740fe3b0 2024-04-12T05:27:40.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.004+0000 7ff673fff700 1 --2- 172.21.15.113:0/828242409 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff66006caa0 0x7ff66006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:40.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.004+0000 7ff673fff700 1 --2- 172.21.15.113:0/828242409 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff66006caa0 0x7ff66006ef60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff664004f10 tx=0x7ff66400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:40.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.008+0000 7ff6717fa700 1 -- 172.21.15.113:0/828242409 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff65c059310 con 0x7ff6740fe3b0 2024-04-12T05:27:40.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.213+0000 7ff67a9c4700 1 -- 172.21.15.113:0/828242409 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff674108b00 con 0x7ff66006caa0 2024-04-12T05:27:40.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.214+0000 7ff6717fa700 1 -- 172.21.15.113:0/828242409 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff674108b00 con 0x7ff66006caa0 2024-04-12T05:27:40.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.216+0000 7ff67a9c4700 1 -- 172.21.15.113:0/828242409 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff66006caa0 msgr2=0x7ff66006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.216+0000 7ff67a9c4700 1 --2- 172.21.15.113:0/828242409 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff66006caa0 0x7ff66006ef60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff664004f10 tx=0x7ff66400b040).stop 2024-04-12T05:27:40.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.216+0000 7ff67a9c4700 1 -- 172.21.15.113:0/828242409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6740fe3b0 msgr2=0x7ff67410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.216+0000 7ff67a9c4700 1 --2- 172.21.15.113:0/828242409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6740fe3b0 0x7ff67410e570 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff65c0041d0 tx=0x7ff65c009f60).stop 2024-04-12T05:27:40.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.216+0000 7ff67a9c4700 1 -- 172.21.15.113:0/828242409 shutdown_connections 2024-04-12T05:27:40.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.216+0000 7ff67a9c4700 1 --2- 172.21.15.113:0/828242409 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff66006caa0 0x7ff66006ef60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.216+0000 7ff67a9c4700 1 --2- 172.21.15.113:0/828242409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6740fda50 0x7ff67410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.216+0000 7ff67a9c4700 1 --2- 172.21.15.113:0/828242409 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6740fe3b0 0x7ff67410e570 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.216+0000 7ff67a9c4700 1 -- 172.21.15.113:0/828242409 >> 172.21.15.113:0/828242409 conn(0x7ff67406ce50 msgr2=0x7ff674101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:40.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.216+0000 7ff67a9c4700 1 -- 172.21.15.113:0/828242409 shutdown_connections 2024-04-12T05:27:40.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.216+0000 7ff67a9c4700 1 -- 172.21.15.113:0/828242409 wait complete. 2024-04-12T05:27:40.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.314+0000 7f79ee2f4700 1 -- 172.21.15.113:0/2507804235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79e80fc7f0 msgr2=0x7f79e80fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.314+0000 7f79ee2f4700 1 --2- 172.21.15.113:0/2507804235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79e80fc7f0 0x7f79e80fcbd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f79d80047f0 tx=0x7f79d8009cb0).stop 2024-04-12T05:27:40.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.314+0000 7f79ee2f4700 1 -- 172.21.15.113:0/2507804235 shutdown_connections 2024-04-12T05:27:40.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.314+0000 7f79ee2f4700 1 --2- 172.21.15.113:0/2507804235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79e80fd150 0x7f79e80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.314+0000 7f79ee2f4700 1 --2- 172.21.15.113:0/2507804235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79e80fc7f0 0x7f79e80fcbd0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.314+0000 7f79ee2f4700 1 -- 172.21.15.113:0/2507804235 >> 172.21.15.113:0/2507804235 conn(0x7f79e80f23b0 msgr2=0x7f79e80f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:40.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.314+0000 7f79ee2f4700 1 -- 172.21.15.113:0/2507804235 shutdown_connections 2024-04-12T05:27:40.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.314+0000 7f79ee2f4700 1 -- 172.21.15.113:0/2507804235 wait complete. 2024-04-12T05:27:40.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.315+0000 7f79ee2f4700 1 Processor -- start 2024-04-12T05:27:40.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.315+0000 7f79ee2f4700 1 -- start start 2024-04-12T05:27:40.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.315+0000 7f79ee2f4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79e80fc7f0 0x7f79e810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:40.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.315+0000 7f79ee2f4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79e80fd150 0x7f79e810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:40.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.315+0000 7f79ee2f4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e810eb20 con 0x7f79e80fd150 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.315+0000 7f79ee2f4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e810ec60 con 0x7f79e80fc7f0 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.315+0000 7f79e77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79e80fd150 0x7f79e810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.315+0000 7f79e77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79e80fd150 0x7f79e810e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41242/0 (socket says 172.21.15.113:41242) 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.315+0000 7f79e77fe700 1 -- 172.21.15.113:0/4079271179 learned_addr learned my addr 172.21.15.113:0/4079271179 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.316+0000 7f79e7fff700 1 --2- 172.21.15.113:0/4079271179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79e80fc7f0 0x7f79e810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.316+0000 7f79e77fe700 1 -- 172.21.15.113:0/4079271179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79e80fc7f0 msgr2=0x7f79e810e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.316+0000 7f79e77fe700 1 --2- 172.21.15.113:0/4079271179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79e80fc7f0 0x7f79e810e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.316+0000 7f79e77fe700 1 -- 172.21.15.113:0/4079271179 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d8005040 con 0x7f79e80fd150 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.316+0000 7f79e77fe700 1 --2- 172.21.15.113:0/4079271179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79e80fd150 0x7f79e810e520 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f79dc00c790 tx=0x7f79dc007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.316+0000 7f79e57fa700 1 -- 172.21.15.113:0/4079271179 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79dc0106f0 con 0x7f79e80fd150 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.316+0000 7f79ee2f4700 1 -- 172.21.15.113:0/4079271179 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e81082f0 con 0x7f79e80fd150 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.316+0000 7f79e57fa700 1 -- 172.21.15.113:0/4079271179 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79dc010d00 con 0x7f79e80fd150 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.316+0000 7f79e57fa700 1 -- 172.21.15.113:0/4079271179 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79dc012d50 con 0x7f79e80fd150 2024-04-12T05:27:40.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.316+0000 7f79ee2f4700 1 -- 172.21.15.113:0/4079271179 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e81088b0 con 0x7f79e80fd150 2024-04-12T05:27:40.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.317+0000 7f79ee2f4700 1 -- 172.21.15.113:0/4079271179 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e8060370 con 0x7f79e80fd150 2024-04-12T05:27:40.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.318+0000 7f79e57fa700 1 -- 172.21.15.113:0/4079271179 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f79dc012570 con 0x7f79e80fd150 2024-04-12T05:27:40.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.318+0000 7f79e57fa700 1 --2- 172.21.15.113:0/4079271179 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f79d006caa0 0x7f79d006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:40.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.318+0000 7f79e57fa700 1 -- 172.21.15.113:0/4079271179 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f79dc08b630 con 0x7f79e80fd150 2024-04-12T05:27:40.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.319+0000 7f79e7fff700 1 --2- 172.21.15.113:0/4079271179 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f79d006caa0 0x7f79d006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:40.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.319+0000 7f79e7fff700 1 --2- 172.21.15.113:0/4079271179 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f79d006caa0 0x7f79d006ef60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f79d8004f10 tx=0x7f79d800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:40.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.323+0000 7f79e57fa700 1 -- 172.21.15.113:0/4079271179 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f79dc0590e0 con 0x7f79e80fd150 2024-04-12T05:27:40.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.520+0000 7f79ee2f4700 1 -- 172.21.15.113:0/4079271179 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f79e8108fa0 con 0x7f79d006caa0 2024-04-12T05:27:40.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.529+0000 7f79e57fa700 1 -- 172.21.15.113:0/4079271179 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f79e8108fa0 con 0x7f79d006caa0 2024-04-12T05:27:40.532 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:27:40.532 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (15m) 8m ago 26m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (12m) 8m ago 26m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (12m) 8m ago 26m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (15m) 8m ago 26m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (8m) 8m ago 21m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (8m) 8m ago 21m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (8m) 8m ago 21m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (21m) 8m ago 21m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (16m) 8m ago 27m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (16m) 8m ago 26m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (13m) 8m ago 28m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (13m) 8m ago 25m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:27:40.533 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (16m) 8m ago 26m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:27:40.534 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (16m) 8m ago 25m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:27:40.534 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (12m) 8m ago 25m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:27:40.534 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (11m) 8m ago 24m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:27:40.534 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (11m) 8m ago 24m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:27:40.534 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (10m) 8m ago 23m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:27:40.534 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (10m) 8m ago 23m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:27:40.534 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (9m) 8m ago 22m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:27:40.534 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (15m) 8m ago 25m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:27:40.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.532+0000 7f79ee2f4700 1 -- 172.21.15.113:0/4079271179 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f79d006caa0 msgr2=0x7f79d006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.532+0000 7f79ee2f4700 1 --2- 172.21.15.113:0/4079271179 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f79d006caa0 0x7f79d006ef60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f79d8004f10 tx=0x7f79d800b040).stop 2024-04-12T05:27:40.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.532+0000 7f79ee2f4700 1 -- 172.21.15.113:0/4079271179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79e80fd150 msgr2=0x7f79e810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.532+0000 7f79ee2f4700 1 --2- 172.21.15.113:0/4079271179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79e80fd150 0x7f79e810e520 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f79dc00c790 tx=0x7f79dc007f50).stop 2024-04-12T05:27:40.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.532+0000 7f79ee2f4700 1 -- 172.21.15.113:0/4079271179 shutdown_connections 2024-04-12T05:27:40.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.532+0000 7f79ee2f4700 1 --2- 172.21.15.113:0/4079271179 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f79d006caa0 0x7f79d006ef60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.532+0000 7f79ee2f4700 1 --2- 172.21.15.113:0/4079271179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79e80fc7f0 0x7f79e810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.532+0000 7f79ee2f4700 1 --2- 172.21.15.113:0/4079271179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79e80fd150 0x7f79e810e520 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.532+0000 7f79ee2f4700 1 -- 172.21.15.113:0/4079271179 >> 172.21.15.113:0/4079271179 conn(0x7f79e80f23b0 msgr2=0x7f79e8060b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:40.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.532+0000 7f79ee2f4700 1 -- 172.21.15.113:0/4079271179 shutdown_connections 2024-04-12T05:27:40.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.532+0000 7f79ee2f4700 1 -- 172.21.15.113:0/4079271179 wait complete. 2024-04-12T05:27:40.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.628+0000 7fcc78b02700 1 -- 172.21.15.113:0/3645604734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc740fd150 msgr2=0x7fcc740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.628+0000 7fcc78b02700 1 --2- 172.21.15.113:0/3645604734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc740fd150 0x7fcc740f6930 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fcc640047f0 tx=0x7fcc64009cb0).stop 2024-04-12T05:27:40.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.628+0000 7fcc78b02700 1 -- 172.21.15.113:0/3645604734 shutdown_connections 2024-04-12T05:27:40.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.628+0000 7fcc78b02700 1 --2- 172.21.15.113:0/3645604734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc740fd150 0x7fcc740f6930 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.628+0000 7fcc78b02700 1 --2- 172.21.15.113:0/3645604734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc740fc7f0 0x7fcc740fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.628+0000 7fcc78b02700 1 -- 172.21.15.113:0/3645604734 >> 172.21.15.113:0/3645604734 conn(0x7fcc740f23b0 msgr2=0x7fcc740f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:40.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.628+0000 7fcc78b02700 1 -- 172.21.15.113:0/3645604734 shutdown_connections 2024-04-12T05:27:40.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.628+0000 7fcc78b02700 1 -- 172.21.15.113:0/3645604734 wait complete. 2024-04-12T05:27:40.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.629+0000 7fcc78b02700 1 Processor -- start 2024-04-12T05:27:40.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.629+0000 7fcc78b02700 1 -- start start 2024-04-12T05:27:40.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.629+0000 7fcc78b02700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc740fc7f0 0x7fcc74109190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:40.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.629+0000 7fcc78b02700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc740fd150 0x7fcc74109680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:40.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.629+0000 7fcc78b02700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc74109b70 con 0x7fcc740fc7f0 2024-04-12T05:27:40.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.629+0000 7fcc78b02700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc74109cb0 con 0x7fcc740fd150 2024-04-12T05:27:40.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.629+0000 7fcc7259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc740fc7f0 0x7fcc74109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:40.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.629+0000 7fcc7259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc740fc7f0 0x7fcc74109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58794/0 (socket says 172.21.15.113:58794) 2024-04-12T05:27:40.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.629+0000 7fcc7259c700 1 -- 172.21.15.113:0/635072277 learned_addr learned my addr 172.21.15.113:0/635072277 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:40.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.630+0000 7fcc71d9b700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc740fd150 0x7fcc74109680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:40.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.630+0000 7fcc7259c700 1 -- 172.21.15.113:0/635072277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc740fd150 msgr2=0x7fcc74109680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.630+0000 7fcc7259c700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc740fd150 0x7fcc74109680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.630+0000 7fcc7259c700 1 -- 172.21.15.113:0/635072277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc64005040 con 0x7fcc740fc7f0 2024-04-12T05:27:40.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.630+0000 7fcc71d9b700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc740fd150 0x7fcc74109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:27:40.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.630+0000 7fcc7259c700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc740fc7f0 0x7fcc74109190 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fcc6800c790 tx=0x7fcc6800cb50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:40.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.630+0000 7fcc637fe700 1 -- 172.21.15.113:0/635072277 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc680106f0 con 0x7fcc740fc7f0 2024-04-12T05:27:40.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.630+0000 7fcc637fe700 1 -- 172.21.15.113:0/635072277 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc68010d00 con 0x7fcc740fc7f0 2024-04-12T05:27:40.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.630+0000 7fcc78b02700 1 -- 172.21.15.113:0/635072277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc74109f60 con 0x7fcc740fc7f0 2024-04-12T05:27:40.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.630+0000 7fcc637fe700 1 -- 172.21.15.113:0/635072277 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc68012e50 con 0x7fcc740fc7f0 2024-04-12T05:27:40.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.630+0000 7fcc78b02700 1 -- 172.21.15.113:0/635072277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc741124d0 con 0x7fcc740fc7f0 2024-04-12T05:27:40.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.631+0000 7fcc78b02700 1 -- 172.21.15.113:0/635072277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc74060390 con 0x7fcc740fc7f0 2024-04-12T05:27:40.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.632+0000 7fcc637fe700 1 -- 172.21.15.113:0/635072277 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcc68010850 con 0x7fcc740fc7f0 2024-04-12T05:27:40.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.632+0000 7fcc637fe700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc5c06caa0 0x7fcc5c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:40.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.632+0000 7fcc637fe700 1 -- 172.21.15.113:0/635072277 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fcc6808b790 con 0x7fcc740fc7f0 2024-04-12T05:27:40.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.632+0000 7fcc71d9b700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc5c06caa0 0x7fcc5c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:40.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.632+0000 7fcc71d9b700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc5c06caa0 0x7fcc5c06ef60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fcc640043d0 tx=0x7fcc6400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:40.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.636+0000 7fcc637fe700 1 -- 172.21.15.113:0/635072277 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fcc680557b0 con 0x7fcc740fc7f0 2024-04-12T05:27:40.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.905+0000 7fcc78b02700 1 -- 172.21.15.113:0/635072277 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcc74047540 con 0x7fcc740fc7f0 2024-04-12T05:27:40.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.906+0000 7fcc637fe700 1 -- 172.21.15.113:0/635072277 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fcc68058dd0 con 0x7fcc740fc7f0 2024-04-12T05:27:40.908 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:27:40.909 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:27:40.910 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:27:40.910 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:27:40.910 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:27:40.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.908+0000 7fcc78b02700 1 -- 172.21.15.113:0/635072277 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc5c06caa0 msgr2=0x7fcc5c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.908+0000 7fcc78b02700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc5c06caa0 0x7fcc5c06ef60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fcc640043d0 tx=0x7fcc6400b040).stop 2024-04-12T05:27:40.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.908+0000 7fcc78b02700 1 -- 172.21.15.113:0/635072277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc740fc7f0 msgr2=0x7fcc74109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:40.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.908+0000 7fcc78b02700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc740fc7f0 0x7fcc74109190 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fcc6800c790 tx=0x7fcc6800cb50).stop 2024-04-12T05:27:40.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.908+0000 7fcc78b02700 1 -- 172.21.15.113:0/635072277 shutdown_connections 2024-04-12T05:27:40.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.908+0000 7fcc78b02700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc5c06caa0 0x7fcc5c06ef60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.908+0000 7fcc78b02700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc740fc7f0 0x7fcc74109190 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.908+0000 7fcc78b02700 1 --2- 172.21.15.113:0/635072277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc740fd150 0x7fcc74109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:40.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.908+0000 7fcc78b02700 1 -- 172.21.15.113:0/635072277 >> 172.21.15.113:0/635072277 conn(0x7fcc740f23b0 msgr2=0x7fcc740fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:40.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.908+0000 7fcc78b02700 1 -- 172.21.15.113:0/635072277 shutdown_connections 2024-04-12T05:27:40.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:40.908+0000 7fcc78b02700 1 -- 172.21.15.113:0/635072277 wait complete. 2024-04-12T05:27:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.006+0000 7f3259fc4700 1 -- 172.21.15.113:0/3485705474 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32540fd150 msgr2=0x7f32540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.006+0000 7f3259fc4700 1 --2- 172.21.15.113:0/3485705474 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32540fd150 0x7f32540f6930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f32480047f0 tx=0x7f3248009cb0).stop 2024-04-12T05:27:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.006+0000 7f3259fc4700 1 -- 172.21.15.113:0/3485705474 shutdown_connections 2024-04-12T05:27:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.006+0000 7f3259fc4700 1 --2- 172.21.15.113:0/3485705474 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32540fd150 0x7f32540f6930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.006+0000 7f3259fc4700 1 --2- 172.21.15.113:0/3485705474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32540fc7f0 0x7f32540fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.006+0000 7f3259fc4700 1 -- 172.21.15.113:0/3485705474 >> 172.21.15.113:0/3485705474 conn(0x7f32540f23b0 msgr2=0x7f32540f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:41.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.006+0000 7f3259fc4700 1 -- 172.21.15.113:0/3485705474 shutdown_connections 2024-04-12T05:27:41.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.006+0000 7f3259fc4700 1 -- 172.21.15.113:0/3485705474 wait complete. 2024-04-12T05:27:41.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.007+0000 7f3259fc4700 1 Processor -- start 2024-04-12T05:27:41.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.007+0000 7f3259fc4700 1 -- start start 2024-04-12T05:27:41.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f3259fc4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32540fc7f0 0x7f3254109140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:41.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f3259fc4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32540fd150 0x7f3254109630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:41.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f3259fc4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3254109b20 con 0x7f32540fc7f0 2024-04-12T05:27:41.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f3259fc4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3254109c60 con 0x7f32540fd150 2024-04-12T05:27:41.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f32537fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32540fc7f0 0x7f3254109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:41.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f32537fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32540fc7f0 0x7f3254109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58800/0 (socket says 172.21.15.113:58800) 2024-04-12T05:27:41.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f32537fe700 1 -- 172.21.15.113:0/3203927606 learned_addr learned my addr 172.21.15.113:0/3203927606 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:41.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f3252ffd700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32540fd150 0x7f3254109630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:41.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f32537fe700 1 -- 172.21.15.113:0/3203927606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32540fd150 msgr2=0x7f3254109630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f32537fe700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32540fd150 0x7f3254109630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f32537fe700 1 -- 172.21.15.113:0/3203927606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3248005040 con 0x7f32540fc7f0 2024-04-12T05:27:41.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f3252ffd700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32540fd150 0x7f3254109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:27:41.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f32537fe700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32540fc7f0 0x7f3254109140 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f324400c740 tx=0x7f3244007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:41.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f3250ff9700 1 -- 172.21.15.113:0/3203927606 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f324400e680 con 0x7f32540fc7f0 2024-04-12T05:27:41.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f3259fc4700 1 -- 172.21.15.113:0/3203927606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3254109f10 con 0x7f32540fc7f0 2024-04-12T05:27:41.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f3250ff9700 1 -- 172.21.15.113:0/3203927606 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f324400ec90 con 0x7f32540fc7f0 2024-04-12T05:27:41.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.008+0000 7f3250ff9700 1 -- 172.21.15.113:0/3203927606 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3244013d60 con 0x7f32540fc7f0 2024-04-12T05:27:41.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.009+0000 7f3259fc4700 1 -- 172.21.15.113:0/3203927606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32541124b0 con 0x7f32540fc7f0 2024-04-12T05:27:41.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.009+0000 7f3259fc4700 1 -- 172.21.15.113:0/3203927606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3254060370 con 0x7f32540fc7f0 2024-04-12T05:27:41.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.010+0000 7f3250ff9700 1 -- 172.21.15.113:0/3203927606 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f324400e7e0 con 0x7f32540fc7f0 2024-04-12T05:27:41.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.010+0000 7f3250ff9700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f323c06caa0 0x7f323c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:41.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.011+0000 7f3252ffd700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f323c06caa0 0x7f323c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:41.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.011+0000 7f3250ff9700 1 -- 172.21.15.113:0/3203927606 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f324408c150 con 0x7f32540fc7f0 2024-04-12T05:27:41.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.011+0000 7f3252ffd700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f323c06caa0 0x7f323c06ef60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f3248004f10 tx=0x7f324800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:41.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.015+0000 7f3250ff9700 1 -- 172.21.15.113:0/3203927606 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3244056170 con 0x7f32540fc7f0 2024-04-12T05:27:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.244+0000 7f3259fc4700 1 -- 172.21.15.113:0/3203927606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3254047540 con 0x7f32540fc7f0 2024-04-12T05:27:41.247 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.245+0000 7f3250ff9700 1 -- 172.21.15.113:0/3203927606 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f3244059790 con 0x7f32540fc7f0 2024-04-12T05:27:41.247 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:27:41.247 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:27:41.247 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:27:41.247 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:27:41.248 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:27:41.249 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:27:41.250 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:27:41.250 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:27:41.250 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:27:41.250 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:27:41.250 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:27:41.250 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:27:41.250 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:27:41.250 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:27:41.250 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:27:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.247+0000 7f3259fc4700 1 -- 172.21.15.113:0/3203927606 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f323c06caa0 msgr2=0x7f323c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.247+0000 7f3259fc4700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f323c06caa0 0x7f323c06ef60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f3248004f10 tx=0x7f324800b040).stop 2024-04-12T05:27:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.247+0000 7f3259fc4700 1 -- 172.21.15.113:0/3203927606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32540fc7f0 msgr2=0x7f3254109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.247+0000 7f3259fc4700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32540fc7f0 0x7f3254109140 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f324400c740 tx=0x7f3244007ef0).stop 2024-04-12T05:27:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.247+0000 7f3259fc4700 1 -- 172.21.15.113:0/3203927606 shutdown_connections 2024-04-12T05:27:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.247+0000 7f3259fc4700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f323c06caa0 0x7f323c06ef60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.247+0000 7f3259fc4700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32540fc7f0 0x7f3254109140 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.247+0000 7f3259fc4700 1 --2- 172.21.15.113:0/3203927606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32540fd150 0x7f3254109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.247+0000 7f3259fc4700 1 -- 172.21.15.113:0/3203927606 >> 172.21.15.113:0/3203927606 conn(0x7f32540f23b0 msgr2=0x7f32540fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:41.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.247+0000 7f3259fc4700 1 -- 172.21.15.113:0/3203927606 shutdown_connections 2024-04-12T05:27:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.247+0000 7f3259fc4700 1 -- 172.21.15.113:0/3203927606 wait complete. 2024-04-12T05:27:41.252 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:27:41.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.346+0000 7f4ee718a700 1 -- 172.21.15.113:0/557235150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ee00fd150 msgr2=0x7f4ee00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.346+0000 7f4ee718a700 1 --2- 172.21.15.113:0/557235150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ee00fd150 0x7f4ee00f6930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f4ecc0047f0 tx=0x7f4ecc009cb0).stop 2024-04-12T05:27:41.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.346+0000 7f4ee718a700 1 -- 172.21.15.113:0/557235150 shutdown_connections 2024-04-12T05:27:41.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.346+0000 7f4ee718a700 1 --2- 172.21.15.113:0/557235150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ee00fd150 0x7f4ee00f6930 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.346+0000 7f4ee718a700 1 --2- 172.21.15.113:0/557235150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ee00fc7f0 0x7f4ee00fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.346+0000 7f4ee718a700 1 -- 172.21.15.113:0/557235150 >> 172.21.15.113:0/557235150 conn(0x7f4ee0070590 msgr2=0x7f4ee00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:41.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.346+0000 7f4ee718a700 1 -- 172.21.15.113:0/557235150 shutdown_connections 2024-04-12T05:27:41.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.346+0000 7f4ee718a700 1 -- 172.21.15.113:0/557235150 wait complete. 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee718a700 1 Processor -- start 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee718a700 1 -- start start 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee718a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ee00fc7f0 0x7f4ee010cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee718a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ee00fd150 0x7f4ee010d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee718a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee010d9e0 con 0x7f4ee00fc7f0 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee718a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee010db20 con 0x7f4ee00fd150 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee4f26700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ee00fc7f0 0x7f4ee010cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee4f26700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ee00fc7f0 0x7f4ee010cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58822/0 (socket says 172.21.15.113:58822) 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee4f26700 1 -- 172.21.15.113:0/1069649687 learned_addr learned my addr 172.21.15.113:0/1069649687 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4edffff700 1 --2- 172.21.15.113:0/1069649687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ee00fd150 0x7f4ee010d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee4f26700 1 -- 172.21.15.113:0/1069649687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ee00fd150 msgr2=0x7f4ee010d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee4f26700 1 --2- 172.21.15.113:0/1069649687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ee00fd150 0x7f4ee010d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee4f26700 1 -- 172.21.15.113:0/1069649687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ecc005040 con 0x7f4ee00fc7f0 2024-04-12T05:27:41.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.347+0000 7f4ee4f26700 1 --2- 172.21.15.113:0/1069649687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ee00fc7f0 0x7f4ee010cef0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f4ed40041d0 tx=0x7f4ed4009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:41.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.348+0000 7f4eddffb700 1 -- 172.21.15.113:0/1069649687 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ed4016070 con 0x7f4ee00fc7f0 2024-04-12T05:27:41.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.348+0000 7f4ee718a700 1 -- 172.21.15.113:0/1069649687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee01071b0 con 0x7f4ee00fc7f0 2024-04-12T05:27:41.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.348+0000 7f4eddffb700 1 -- 172.21.15.113:0/1069649687 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ed4007d20 con 0x7f4ee00fc7f0 2024-04-12T05:27:41.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.348+0000 7f4eddffb700 1 -- 172.21.15.113:0/1069649687 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ed400de60 con 0x7f4ee00fc7f0 2024-04-12T05:27:41.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.348+0000 7f4ee718a700 1 -- 172.21.15.113:0/1069649687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee0107770 con 0x7f4ee00fc7f0 2024-04-12T05:27:41.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.349+0000 7f4ee718a700 1 -- 172.21.15.113:0/1069649687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee005f590 con 0x7f4ee00fc7f0 2024-04-12T05:27:41.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.349+0000 7f4eddffb700 1 -- 172.21.15.113:0/1069649687 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4ed4009110 con 0x7f4ee00fc7f0 2024-04-12T05:27:41.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.350+0000 7f4eddffb700 1 --2- 172.21.15.113:0/1069649687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ed006caa0 0x7f4ed006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:41.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.350+0000 7f4eddffb700 1 -- 172.21.15.113:0/1069649687 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f4ed408e1c0 con 0x7f4ee00fc7f0 2024-04-12T05:27:41.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.350+0000 7f4edffff700 1 --2- 172.21.15.113:0/1069649687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ed006caa0 0x7f4ed006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:41.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.350+0000 7f4edffff700 1 --2- 172.21.15.113:0/1069649687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ed006caa0 0x7f4ed006ef60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f4ee0108830 tx=0x7f4ecc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:41.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.354+0000 7f4eddffb700 1 -- 172.21.15.113:0/1069649687 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4ed405bc70 con 0x7f4ee00fc7f0 2024-04-12T05:27:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:41 smithi113 ceph-mon[103371]: from='client.34868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:41 smithi113 ceph-mon[103371]: pgmap v465: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:41.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:41 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/635072277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:27:41.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:41 smithi192 ceph-mon[85304]: from='client.34868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:41.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:41 smithi192 ceph-mon[85304]: pgmap v465: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:41.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:41 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/635072277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:27:41.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.558+0000 7f4ee718a700 1 -- 172.21.15.113:0/1069649687 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ee0107db0 con 0x7f4ed006caa0 2024-04-12T05:27:41.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.560+0000 7f4eddffb700 1 -- 172.21.15.113:0/1069649687 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4ee0107db0 con 0x7f4ed006caa0 2024-04-12T05:27:41.562 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:27:41.562 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:27:41.562 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:27:41.562 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:27:41.562 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:27:41.563 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:27:41.563 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:27:41.563 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:27:41.563 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:27:41.563 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:27:41.563 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:27:41.563 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:27:41.563 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:27:41.563 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:27:41.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.562+0000 7f4ee718a700 1 -- 172.21.15.113:0/1069649687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ed006caa0 msgr2=0x7f4ed006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.562+0000 7f4ee718a700 1 --2- 172.21.15.113:0/1069649687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ed006caa0 0x7f4ed006ef60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f4ee0108830 tx=0x7f4ecc00b040).stop 2024-04-12T05:27:41.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.562+0000 7f4ee718a700 1 -- 172.21.15.113:0/1069649687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ee00fc7f0 msgr2=0x7f4ee010cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.562+0000 7f4ee718a700 1 --2- 172.21.15.113:0/1069649687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ee00fc7f0 0x7f4ee010cef0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f4ed40041d0 tx=0x7f4ed4009f60).stop 2024-04-12T05:27:41.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.562+0000 7f4ee718a700 1 -- 172.21.15.113:0/1069649687 shutdown_connections 2024-04-12T05:27:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.562+0000 7f4ee718a700 1 --2- 172.21.15.113:0/1069649687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4ed006caa0 0x7f4ed006ef60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.562+0000 7f4ee718a700 1 --2- 172.21.15.113:0/1069649687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ee00fc7f0 0x7f4ee010cef0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.562+0000 7f4ee718a700 1 --2- 172.21.15.113:0/1069649687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ee00fd150 0x7f4ee010d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.562+0000 7f4ee718a700 1 -- 172.21.15.113:0/1069649687 >> 172.21.15.113:0/1069649687 conn(0x7f4ee0070590 msgr2=0x7f4ee00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.562+0000 7f4ee718a700 1 -- 172.21.15.113:0/1069649687 shutdown_connections 2024-04-12T05:27:41.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.562+0000 7f4ee718a700 1 -- 172.21.15.113:0/1069649687 wait complete. 2024-04-12T05:27:41.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.660+0000 7f4e7343d700 1 -- 172.21.15.113:0/3316554758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e6c0fc7f0 msgr2=0x7f4e6c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.660+0000 7f4e7343d700 1 --2- 172.21.15.113:0/3316554758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e6c0fc7f0 0x7f4e6c0fcbd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f4e5c0047f0 tx=0x7f4e5c009cb0).stop 2024-04-12T05:27:41.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.660+0000 7f4e7343d700 1 -- 172.21.15.113:0/3316554758 shutdown_connections 2024-04-12T05:27:41.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.660+0000 7f4e7343d700 1 --2- 172.21.15.113:0/3316554758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6c0fd150 0x7f4e6c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.660+0000 7f4e7343d700 1 --2- 172.21.15.113:0/3316554758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e6c0fc7f0 0x7f4e6c0fcbd0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.660+0000 7f4e7343d700 1 -- 172.21.15.113:0/3316554758 >> 172.21.15.113:0/3316554758 conn(0x7f4e6c070590 msgr2=0x7f4e6c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:41.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.660+0000 7f4e7343d700 1 -- 172.21.15.113:0/3316554758 shutdown_connections 2024-04-12T05:27:41.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.660+0000 7f4e7343d700 1 -- 172.21.15.113:0/3316554758 wait complete. 2024-04-12T05:27:41.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e7343d700 1 Processor -- start 2024-04-12T05:27:41.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e7343d700 1 -- start start 2024-04-12T05:27:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e7343d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e6c0fc7f0 0x7f4e6c109120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e7343d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6c0fd150 0x7f4e6c109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e7343d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c109b00 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e7343d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c109c40 con 0x7f4e6c0fd150 2024-04-12T05:27:41.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e711d9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e6c0fc7f0 0x7f4e6c109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e711d9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e6c0fc7f0 0x7f4e6c109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58832/0 (socket says 172.21.15.113:58832) 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e711d9700 1 -- 172.21.15.113:0/2591633090 learned_addr learned my addr 172.21.15.113:0/2591633090 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e709d8700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6c0fd150 0x7f4e6c109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e711d9700 1 -- 172.21.15.113:0/2591633090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6c0fd150 msgr2=0x7f4e6c109610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e711d9700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6c0fd150 0x7f4e6c109610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e711d9700 1 -- 172.21.15.113:0/2591633090 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c005040 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e709d8700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6c0fd150 0x7f4e6c109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e711d9700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e6c0fc7f0 0x7f4e6c109120 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f4e5c005010 tx=0x7f4e5c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e627fc700 1 -- 172.21.15.113:0/2591633090 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e5c00e030 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.661+0000 7f4e627fc700 1 -- 172.21.15.113:0/2591633090 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e5c007e90 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.662+0000 7f4e7343d700 1 -- 172.21.15.113:0/2591633090 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6c109e90 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.662+0000 7f4e627fc700 1 -- 172.21.15.113:0/2591633090 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e5c018070 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.662+0000 7f4e7343d700 1 -- 172.21.15.113:0/2591633090 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c112480 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.663+0000 7f4e7343d700 1 -- 172.21.15.113:0/2591633090 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6c05f590 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.663+0000 7f4e627fc700 1 -- 172.21.15.113:0/2591633090 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4e5c01d6e0 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.663+0000 7f4e627fc700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e5806ca30 0x7f4e5806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:27:41.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.664+0000 7f4e627fc700 1 -- 172.21.15.113:0/2591633090 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f4e5c08d100 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.664+0000 7f4e709d8700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e5806ca30 0x7f4e5806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:27:41.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.664+0000 7f4e709d8700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e5806ca30 0x7f4e5806eef0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f4e6c060ee0 tx=0x7f4e6800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:27:41.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.668+0000 7f4e627fc700 1 -- 172.21.15.113:0/2591633090 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4e5c05abb0 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.931+0000 7f4e7343d700 1 -- 172.21.15.113:0/2591633090 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4e6c047540 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.932+0000 7f4e627fc700 1 -- 172.21.15.113:0/2591633090 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4e5c05a740 con 0x7f4e6c0fc7f0 2024-04-12T05:27:41.934 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:27:41.934 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:27:41.934 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:27:41.935 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:27:41.935 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:27:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.934+0000 7f4e7343d700 1 -- 172.21.15.113:0/2591633090 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e5806ca30 msgr2=0x7f4e5806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.934+0000 7f4e7343d700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e5806ca30 0x7f4e5806eef0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f4e6c060ee0 tx=0x7f4e6800b040).stop 2024-04-12T05:27:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.934+0000 7f4e7343d700 1 -- 172.21.15.113:0/2591633090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e6c0fc7f0 msgr2=0x7f4e6c109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:27:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.934+0000 7f4e7343d700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e6c0fc7f0 0x7f4e6c109120 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f4e5c005010 tx=0x7f4e5c009cb0).stop 2024-04-12T05:27:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.934+0000 7f4e7343d700 1 -- 172.21.15.113:0/2591633090 shutdown_connections 2024-04-12T05:27:41.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.934+0000 7f4e7343d700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e5806ca30 0x7f4e5806eef0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.934+0000 7f4e7343d700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e6c0fc7f0 0x7f4e6c109120 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.934+0000 7f4e7343d700 1 --2- 172.21.15.113:0/2591633090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e6c0fd150 0x7f4e6c109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:27:41.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.934+0000 7f4e7343d700 1 -- 172.21.15.113:0/2591633090 >> 172.21.15.113:0/2591633090 conn(0x7f4e6c070590 msgr2=0x7f4e6c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:27:41.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.934+0000 7f4e7343d700 1 -- 172.21.15.113:0/2591633090 shutdown_connections 2024-04-12T05:27:41.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:27:41.934+0000 7f4e7343d700 1 -- 172.21.15.113:0/2591633090 wait complete. 2024-04-12T05:27:42.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:42 smithi113 ceph-mon[103371]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:42.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:42 smithi113 ceph-mon[103371]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:42.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:42 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3203927606' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:27:42.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:42 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2591633090' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:27:42.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:42 smithi192 ceph-mon[85304]: from='client.34872 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:42.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:42 smithi192 ceph-mon[85304]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:42.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:42 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3203927606' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:27:42.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:42 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2591633090' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:27:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:43 smithi113 ceph-mon[103371]: from='client.34888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:43.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:43 smithi113 ceph-mon[103371]: pgmap v466: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:43.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:43 smithi192 ceph-mon[85304]: from='client.34888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:27:43.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:43 smithi192 ceph-mon[85304]: pgmap v466: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:45.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:45 smithi113 ceph-mon[103371]: pgmap v467: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:45.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:45 smithi192 ceph-mon[85304]: pgmap v467: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:47 smithi113 ceph-mon[103371]: pgmap v468: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:47 smithi192 ceph-mon[85304]: pgmap v468: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:49.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:49 smithi113 ceph-mon[103371]: pgmap v469: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:49.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:49 smithi192 ceph-mon[85304]: pgmap v469: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:51.427 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:51 smithi113 ceph-mon[103371]: pgmap v470: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:51 smithi192 ceph-mon[85304]: pgmap v470: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:52.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:27:52.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:27:53.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:53 smithi113 ceph-mon[103371]: pgmap v471: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:53 smithi192 ceph-mon[85304]: pgmap v471: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:55.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:55 smithi113 ceph-mon[103371]: pgmap v472: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:55.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:55 smithi192 ceph-mon[85304]: pgmap v472: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:57.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:57 smithi113 ceph-mon[103371]: pgmap v473: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:57.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:57 smithi192 ceph-mon[85304]: pgmap v473: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:59.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:27:59 smithi113 ceph-mon[103371]: pgmap v474: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:27:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:27:59 smithi192 ceph-mon[85304]: pgmap v474: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:01.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:01 smithi113 ceph-mon[103371]: pgmap v475: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:01.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:01 smithi192 ceph-mon[85304]: pgmap v475: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:03.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:03 smithi113 ceph-mon[103371]: pgmap v476: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:03 smithi192 ceph-mon[85304]: pgmap v476: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:05.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:05 smithi113 ceph-mon[103371]: pgmap v477: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:05.451 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:05 smithi192 ceph-mon[85304]: pgmap v477: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:07.425 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:07 smithi113 ceph-mon[103371]: pgmap v478: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:07.426 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:28:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:07 smithi192 ceph-mon[85304]: pgmap v478: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:28:09.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:09 smithi113 ceph-mon[103371]: pgmap v479: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:09.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:09 smithi192 ceph-mon[85304]: pgmap v479: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:11.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:11 smithi113 ceph-mon[103371]: pgmap v480: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:11 smithi192 ceph-mon[85304]: pgmap v480: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:12.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.040+0000 7f8d7f185700 1 -- 172.21.15.113:0/2203477296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d780f88a0 msgr2=0x7f8d780f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.040+0000 7f8d7f185700 1 --2- 172.21.15.113:0/2203477296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d780f88a0 0x7f8d780f8c80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f8d6c0047f0 tx=0x7f8d6c009cb0).stop 2024-04-12T05:28:12.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.040+0000 7f8d7f185700 1 -- 172.21.15.113:0/2203477296 shutdown_connections 2024-04-12T05:28:12.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.040+0000 7f8d7f185700 1 --2- 172.21.15.113:0/2203477296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d780f9200 0x7f8d78101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.040+0000 7f8d7f185700 1 --2- 172.21.15.113:0/2203477296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d780f88a0 0x7f8d780f8c80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.040+0000 7f8d7f185700 1 -- 172.21.15.113:0/2203477296 >> 172.21.15.113:0/2203477296 conn(0x7f8d78070590 msgr2=0x7f8d780f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:12.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.040+0000 7f8d7f185700 1 -- 172.21.15.113:0/2203477296 shutdown_connections 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.040+0000 7f8d7f185700 1 -- 172.21.15.113:0/2203477296 wait complete. 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d7f185700 1 Processor -- start 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d7f185700 1 -- start start 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d7f185700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d780f88a0 0x7f8d781102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d7f185700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d780f9200 0x7f8d78110790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d7f185700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d78110d90 con 0x7f8d780f9200 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d7f185700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d78110ed0 con 0x7f8d780f88a0 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d77fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d780f9200 0x7f8d78110790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d77fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d780f9200 0x7f8d78110790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50190/0 (socket says 172.21.15.113:50190) 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d77fff700 1 -- 172.21.15.113:0/4137566704 learned_addr learned my addr 172.21.15.113:0/4137566704 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d7cf21700 1 --2- 172.21.15.113:0/4137566704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d780f88a0 0x7f8d781102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d77fff700 1 -- 172.21.15.113:0/4137566704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d780f88a0 msgr2=0x7f8d781102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d77fff700 1 --2- 172.21.15.113:0/4137566704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d780f88a0 0x7f8d781102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.041+0000 7f8d77fff700 1 -- 172.21.15.113:0/4137566704 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c005040 con 0x7f8d780f9200 2024-04-12T05:28:12.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.042+0000 7f8d77fff700 1 --2- 172.21.15.113:0/4137566704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d780f9200 0x7f8d78110790 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f8d640041d0 tx=0x7f8d64009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:12.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.042+0000 7f8d75ffb700 1 -- 172.21.15.113:0/4137566704 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d64007b40 con 0x7f8d780f9200 2024-04-12T05:28:12.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.042+0000 7f8d75ffb700 1 -- 172.21.15.113:0/4137566704 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d6400f040 con 0x7f8d780f9200 2024-04-12T05:28:12.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.042+0000 7f8d7f185700 1 -- 172.21.15.113:0/4137566704 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7810a560 con 0x7f8d780f9200 2024-04-12T05:28:12.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.042+0000 7f8d75ffb700 1 -- 172.21.15.113:0/4137566704 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d640137d0 con 0x7f8d780f9200 2024-04-12T05:28:12.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.042+0000 7f8d7f185700 1 -- 172.21.15.113:0/4137566704 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7810aaa0 con 0x7f8d780f9200 2024-04-12T05:28:12.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.043+0000 7f8d7f185700 1 -- 172.21.15.113:0/4137566704 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7805f590 con 0x7f8d780f9200 2024-04-12T05:28:12.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.043+0000 7f8d75ffb700 1 -- 172.21.15.113:0/4137566704 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8d64007ca0 con 0x7f8d780f9200 2024-04-12T05:28:12.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.044+0000 7f8d75ffb700 1 --2- 172.21.15.113:0/4137566704 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d6806caa0 0x7f8d6806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:12.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.044+0000 7f8d75ffb700 1 -- 172.21.15.113:0/4137566704 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f8d6408b7d0 con 0x7f8d780f9200 2024-04-12T05:28:12.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.044+0000 7f8d7cf21700 1 --2- 172.21.15.113:0/4137566704 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d6806caa0 0x7f8d6806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:12.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.044+0000 7f8d7cf21700 1 --2- 172.21.15.113:0/4137566704 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d6806caa0 0x7f8d6806ef60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f8d6c004f10 tx=0x7f8d6c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:12.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.048+0000 7f8d75ffb700 1 -- 172.21.15.113:0/4137566704 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8d64059280 con 0x7f8d780f9200 2024-04-12T05:28:12.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.254+0000 7f8d7f185700 1 -- 172.21.15.113:0/4137566704 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d7810b9c0 con 0x7f8d6806caa0 2024-04-12T05:28:12.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.255+0000 7f8d75ffb700 1 -- 172.21.15.113:0/4137566704 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8d7810b9c0 con 0x7f8d6806caa0 2024-04-12T05:28:12.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.257+0000 7f8d7f185700 1 -- 172.21.15.113:0/4137566704 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d6806caa0 msgr2=0x7f8d6806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.257+0000 7f8d7f185700 1 --2- 172.21.15.113:0/4137566704 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d6806caa0 0x7f8d6806ef60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f8d6c004f10 tx=0x7f8d6c00b040).stop 2024-04-12T05:28:12.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.257+0000 7f8d7f185700 1 -- 172.21.15.113:0/4137566704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d780f9200 msgr2=0x7f8d78110790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.257+0000 7f8d7f185700 1 --2- 172.21.15.113:0/4137566704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d780f9200 0x7f8d78110790 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f8d640041d0 tx=0x7f8d64009f60).stop 2024-04-12T05:28:12.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.257+0000 7f8d7f185700 1 -- 172.21.15.113:0/4137566704 shutdown_connections 2024-04-12T05:28:12.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.257+0000 7f8d7f185700 1 --2- 172.21.15.113:0/4137566704 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d6806caa0 0x7f8d6806ef60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.257+0000 7f8d7f185700 1 --2- 172.21.15.113:0/4137566704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d780f88a0 0x7f8d781102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.257+0000 7f8d7f185700 1 --2- 172.21.15.113:0/4137566704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d780f9200 0x7f8d78110790 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.257+0000 7f8d7f185700 1 -- 172.21.15.113:0/4137566704 >> 172.21.15.113:0/4137566704 conn(0x7f8d78070590 msgr2=0x7f8d780f6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:12.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.257+0000 7f8d7f185700 1 -- 172.21.15.113:0/4137566704 shutdown_connections 2024-04-12T05:28:12.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.257+0000 7f8d7f185700 1 -- 172.21.15.113:0/4137566704 wait complete. 2024-04-12T05:28:12.270 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:28:12.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.359+0000 7f7c76946700 1 -- 172.21.15.113:0/527094468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c700fe250 msgr2=0x7f7c700f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.359+0000 7f7c76946700 1 --2- 172.21.15.113:0/527094468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c700fe250 0x7f7c700f7a30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f7c580047f0 tx=0x7f7c58009cb0).stop 2024-04-12T05:28:12.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.359+0000 7f7c76946700 1 -- 172.21.15.113:0/527094468 shutdown_connections 2024-04-12T05:28:12.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.359+0000 7f7c76946700 1 --2- 172.21.15.113:0/527094468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c700fe250 0x7f7c700f7a30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.359+0000 7f7c76946700 1 --2- 172.21.15.113:0/527094468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c700fd8f0 0x7f7c700fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.359+0000 7f7c76946700 1 -- 172.21.15.113:0/527094468 >> 172.21.15.113:0/527094468 conn(0x7f7c700f34b0 msgr2=0x7f7c700f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:12.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.359+0000 7f7c76946700 1 -- 172.21.15.113:0/527094468 shutdown_connections 2024-04-12T05:28:12.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.359+0000 7f7c76946700 1 -- 172.21.15.113:0/527094468 wait complete. 2024-04-12T05:28:12.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.359+0000 7f7c76946700 1 Processor -- start 2024-04-12T05:28:12.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c76946700 1 -- start start 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c76946700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c700fd8f0 0x7f7c7010dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c76946700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c700fe250 0x7f7c7010e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c76946700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7010eae0 con 0x7f7c700fd8f0 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c76946700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7010ec20 con 0x7f7c700fe250 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c6ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c700fd8f0 0x7f7c7010dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c6ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c700fd8f0 0x7f7c7010dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50210/0 (socket says 172.21.15.113:50210) 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c6ffff700 1 -- 172.21.15.113:0/3740669818 learned_addr learned my addr 172.21.15.113:0/3740669818 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c67fff700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c700fe250 0x7f7c7010e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c6ffff700 1 -- 172.21.15.113:0/3740669818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c700fe250 msgr2=0x7f7c7010e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c6ffff700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c700fe250 0x7f7c7010e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c6ffff700 1 -- 172.21.15.113:0/3740669818 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c58005040 con 0x7f7c700fd8f0 2024-04-12T05:28:12.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c67fff700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c700fe250 0x7f7c7010e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:28:12.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.360+0000 7f7c6ffff700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c700fd8f0 0x7f7c7010dff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f7c600041d0 tx=0x7f7c60009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:12.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.361+0000 7f7c6dffb700 1 -- 172.21.15.113:0/3740669818 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c60007b40 con 0x7f7c700fd8f0 2024-04-12T05:28:12.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.361+0000 7f7c6dffb700 1 -- 172.21.15.113:0/3740669818 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c6000f040 con 0x7f7c700fd8f0 2024-04-12T05:28:12.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.361+0000 7f7c76946700 1 -- 172.21.15.113:0/3740669818 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c701082b0 con 0x7f7c700fd8f0 2024-04-12T05:28:12.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.361+0000 7f7c6dffb700 1 -- 172.21.15.113:0/3740669818 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c6000d800 con 0x7f7c700fd8f0 2024-04-12T05:28:12.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.361+0000 7f7c76946700 1 -- 172.21.15.113:0/3740669818 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c70108870 con 0x7f7c700fd8f0 2024-04-12T05:28:12.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.362+0000 7f7c76946700 1 -- 172.21.15.113:0/3740669818 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7005f590 con 0x7f7c700fd8f0 2024-04-12T05:28:12.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.362+0000 7f7c6dffb700 1 -- 172.21.15.113:0/3740669818 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7c60007ca0 con 0x7f7c700fd8f0 2024-04-12T05:28:12.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.362+0000 7f7c6dffb700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c5c06caa0 0x7f7c5c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:12.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.362+0000 7f7c6dffb700 1 -- 172.21.15.113:0/3740669818 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7c6008b890 con 0x7f7c700fd8f0 2024-04-12T05:28:12.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.362+0000 7f7c67fff700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c5c06caa0 0x7f7c5c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:12.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.363+0000 7f7c67fff700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c5c06caa0 0x7f7c5c06ef60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f7c58004f10 tx=0x7f7c5800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:12.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.367+0000 7f7c6dffb700 1 -- 172.21.15.113:0/3740669818 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7c60059340 con 0x7f7c700fd8f0 2024-04-12T05:28:12.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.573+0000 7f7c76946700 1 -- 172.21.15.113:0/3740669818 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c70108b40 con 0x7f7c5c06caa0 2024-04-12T05:28:12.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.574+0000 7f7c6dffb700 1 -- 172.21.15.113:0/3740669818 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7c70108b40 con 0x7f7c5c06caa0 2024-04-12T05:28:12.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.576+0000 7f7c76946700 1 -- 172.21.15.113:0/3740669818 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c5c06caa0 msgr2=0x7f7c5c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.576+0000 7f7c76946700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c5c06caa0 0x7f7c5c06ef60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f7c58004f10 tx=0x7f7c5800b040).stop 2024-04-12T05:28:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.576+0000 7f7c76946700 1 -- 172.21.15.113:0/3740669818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c700fd8f0 msgr2=0x7f7c7010dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.576+0000 7f7c76946700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c700fd8f0 0x7f7c7010dff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f7c600041d0 tx=0x7f7c60009f60).stop 2024-04-12T05:28:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.576+0000 7f7c76946700 1 -- 172.21.15.113:0/3740669818 shutdown_connections 2024-04-12T05:28:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.576+0000 7f7c76946700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7c5c06caa0 0x7f7c5c06ef60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.576+0000 7f7c76946700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c700fd8f0 0x7f7c7010dff0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.576+0000 7f7c76946700 1 --2- 172.21.15.113:0/3740669818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c700fe250 0x7f7c7010e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.576+0000 7f7c76946700 1 -- 172.21.15.113:0/3740669818 >> 172.21.15.113:0/3740669818 conn(0x7f7c700f34b0 msgr2=0x7f7c700fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.576+0000 7f7c76946700 1 -- 172.21.15.113:0/3740669818 shutdown_connections 2024-04-12T05:28:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.577+0000 7f7c76946700 1 -- 172.21.15.113:0/3740669818 wait complete. 2024-04-12T05:28:12.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.674+0000 7f72e2920700 1 -- 172.21.15.113:0/4022759910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72dc0fe270 msgr2=0x7f72dc0f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.674+0000 7f72e2920700 1 --2- 172.21.15.113:0/4022759910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72dc0fe270 0x7f72dc0f7a50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f72d00047f0 tx=0x7f72d0009cb0).stop 2024-04-12T05:28:12.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.674+0000 7f72e2920700 1 -- 172.21.15.113:0/4022759910 shutdown_connections 2024-04-12T05:28:12.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.674+0000 7f72e2920700 1 --2- 172.21.15.113:0/4022759910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72dc0fe270 0x7f72dc0f7a50 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.674+0000 7f72e2920700 1 --2- 172.21.15.113:0/4022759910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72dc0fd910 0x7f72dc0fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:28:12.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.674+0000 7f72e2920700 1 -- 172.21.15.113:0/4022759910 >> 172.21.15.113:0/4022759910 conn(0x7f72dc0f3530 msgr2=0x7f72dc0f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:12.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.674+0000 7f72e2920700 1 -- 172.21.15.113:0/4022759910 shutdown_connections 2024-04-12T05:28:12.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.674+0000 7f72e2920700 1 -- 172.21.15.113:0/4022759910 wait complete. 2024-04-12T05:28:12.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.675+0000 7f72e2920700 1 Processor -- start 2024-04-12T05:28:12.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.675+0000 7f72e2920700 1 -- start start 2024-04-12T05:28:12.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72e2920700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72dc0fd910 0x7f72dc10e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:12.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72e2920700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72dc0fe270 0x7f72dc10e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:12.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72e2920700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc10ebe0 con 0x7f72dc0fe270 2024-04-12T05:28:12.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72e2920700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc108200 con 0x7f72dc0fd910 2024-04-12T05:28:12.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72db7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72dc0fe270 0x7f72dc10e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:12.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72db7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72dc0fe270 0x7f72dc10e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50236/0 (socket says 172.21.15.113:50236) 2024-04-12T05:28:12.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72db7fe700 1 -- 172.21.15.113:0/3781694209 learned_addr learned my addr 172.21.15.113:0/3781694209 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:12.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72dbfff700 1 --2- 172.21.15.113:0/3781694209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72dc0fd910 0x7f72dc10e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:12.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72db7fe700 1 -- 172.21.15.113:0/3781694209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72dc0fd910 msgr2=0x7f72dc10e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72db7fe700 1 --2- 172.21.15.113:0/3781694209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72dc0fd910 0x7f72dc10e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72db7fe700 1 -- 172.21.15.113:0/3781694209 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d0005040 con 0x7f72dc0fe270 2024-04-12T05:28:12.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72db7fe700 1 --2- 172.21.15.113:0/3781694209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72dc0fe270 0x7f72dc10e550 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f72d00099f0 tx=0x7f72d0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:12.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72d97fa700 1 -- 172.21.15.113:0/3781694209 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72d000e030 con 0x7f72dc0fe270 2024-04-12T05:28:12.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72d97fa700 1 -- 172.21.15.113:0/3781694209 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72d0007e90 con 0x7f72dc0fe270 2024-04-12T05:28:12.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72d97fa700 1 -- 172.21.15.113:0/3781694209 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72d0018070 con 0x7f72dc0fe270 2024-04-12T05:28:12.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72e2920700 1 -- 172.21.15.113:0/3781694209 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72dc108450 con 0x7f72dc0fe270 2024-04-12T05:28:12.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.676+0000 7f72e2920700 1 -- 172.21.15.113:0/3781694209 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72dc108930 con 0x7f72dc0fe270 2024-04-12T05:28:12.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.677+0000 7f72e2920700 1 -- 172.21.15.113:0/3781694209 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72dc05f570 con 0x7f72dc0fe270 2024-04-12T05:28:12.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.678+0000 7f72d97fa700 1 -- 172.21.15.113:0/3781694209 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f72d001d420 con 0x7f72dc0fe270 2024-04-12T05:28:12.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.678+0000 7f72d97fa700 1 --2- 172.21.15.113:0/3781694209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f72c406caa0 0x7f72c406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:12.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.678+0000 7f72d97fa700 1 -- 172.21.15.113:0/3781694209 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f72d008d3b0 con 0x7f72dc0fe270 2024-04-12T05:28:12.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.678+0000 7f72dbfff700 1 --2- 172.21.15.113:0/3781694209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f72c406caa0 0x7f72c406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:12.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.679+0000 7f72dbfff700 1 --2- 172.21.15.113:0/3781694209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f72c406caa0 0x7f72c406ef60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f72cc00a880 tx=0x7f72cc008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:12.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.683+0000 7f72d97fa700 1 -- 172.21.15.113:0/3781694209 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f72d005ada0 con 0x7f72dc0fe270 2024-04-12T05:28:12.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.883+0000 7f72e2920700 1 -- 172.21.15.113:0/3781694209 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f72dc109780 con 0x7f72c406caa0 2024-04-12T05:28:12.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.893+0000 7f72d97fa700 1 -- 172.21.15.113:0/3781694209 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f72dc109780 con 0x7f72c406caa0 2024-04-12T05:28:12.896 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:28:12.897 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (16m) 8m ago 27m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:28:12.897 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (13m) 8m ago 27m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:28:12.897 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (13m) 8m ago 26m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:28:12.897 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (15m) 8m ago 26m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:28:12.897 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (9m) 8m ago 21m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:28:12.897 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (9m) 8m ago 21m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:28:12.897 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (8m) 8m ago 21m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:28:12.898 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (21m) 8m ago 21m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:28:12.898 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (17m) 8m ago 28m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:28:12.898 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (16m) 8m ago 26m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:28:12.898 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (14m) 8m ago 28m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:28:12.898 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (14m) 8m ago 26m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:28:12.899 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (16m) 8m ago 26m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:28:12.899 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (16m) 8m ago 26m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:28:12.899 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (13m) 8m ago 25m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:28:12.899 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (12m) 8m ago 25m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:28:12.899 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (11m) 8m ago 24m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:28:12.899 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (11m) 8m ago 24m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:28:12.899 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (10m) 8m ago 23m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:28:12.899 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (9m) 8m ago 23m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:28:12.899 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (16m) 8m ago 26m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:28:12.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.895+0000 7f72e2920700 1 -- 172.21.15.113:0/3781694209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f72c406caa0 msgr2=0x7f72c406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.895+0000 7f72e2920700 1 --2- 172.21.15.113:0/3781694209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f72c406caa0 0x7f72c406ef60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f72cc00a880 tx=0x7f72cc008040).stop 2024-04-12T05:28:12.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.895+0000 7f72e2920700 1 -- 172.21.15.113:0/3781694209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72dc0fe270 msgr2=0x7f72dc10e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.895+0000 7f72e2920700 1 --2- 172.21.15.113:0/3781694209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72dc0fe270 0x7f72dc10e550 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f72d00099f0 tx=0x7f72d0009cb0).stop 2024-04-12T05:28:12.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.896+0000 7f72e2920700 1 -- 172.21.15.113:0/3781694209 shutdown_connections 2024-04-12T05:28:12.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.896+0000 7f72e2920700 1 --2- 172.21.15.113:0/3781694209 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f72c406caa0 0x7f72c406ef60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.896+0000 7f72e2920700 1 --2- 172.21.15.113:0/3781694209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72dc0fd910 0x7f72dc10e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.896+0000 7f72e2920700 1 --2- 172.21.15.113:0/3781694209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72dc0fe270 0x7f72dc10e550 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.896+0000 7f72e2920700 1 -- 172.21.15.113:0/3781694209 >> 172.21.15.113:0/3781694209 conn(0x7f72dc0f3530 msgr2=0x7f72dc0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:12.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.896+0000 7f72e2920700 1 -- 172.21.15.113:0/3781694209 shutdown_connections 2024-04-12T05:28:12.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.896+0000 7f72e2920700 1 -- 172.21.15.113:0/3781694209 wait complete. 2024-04-12T05:28:12.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.992+0000 7f80cc3f6700 1 -- 172.21.15.113:0/657905529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80c40f8a10 msgr2=0x7f80c40f8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.992+0000 7f80cc3f6700 1 --2- 172.21.15.113:0/657905529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80c40f8a10 0x7f80c40f8df0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f80c00047f0 tx=0x7f80c0009cb0).stop 2024-04-12T05:28:12.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.992+0000 7f80cc3f6700 1 -- 172.21.15.113:0/657905529 shutdown_connections 2024-04-12T05:28:12.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.993+0000 7f80cc3f6700 1 --2- 172.21.15.113:0/657905529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80c40f92e0 0x7f80c4101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.993+0000 7f80cc3f6700 1 --2- 172.21.15.113:0/657905529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80c40f8a10 0x7f80c40f8df0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.993+0000 7f80cc3f6700 1 -- 172.21.15.113:0/657905529 >> 172.21.15.113:0/657905529 conn(0x7f80c406c760 msgr2=0x7f80c406cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:12.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.993+0000 7f80cc3f6700 1 -- 172.21.15.113:0/657905529 shutdown_connections 2024-04-12T05:28:12.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.993+0000 7f80cc3f6700 1 -- 172.21.15.113:0/657905529 wait complete. 2024-04-12T05:28:12.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.993+0000 7f80cc3f6700 1 Processor -- start 2024-04-12T05:28:12.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.993+0000 7f80cc3f6700 1 -- start start 2024-04-12T05:28:12.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80cc3f6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80c40f8a10 0x7f80c410e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:12.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80cc3f6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80c40f92e0 0x7f80c410e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:12.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80cc3f6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c410eb50 con 0x7f80c40f8a10 2024-04-12T05:28:12.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80cc3f6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c410ec90 con 0x7f80c40f92e0 2024-04-12T05:28:12.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80ca192700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80c40f8a10 0x7f80c410e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:12.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80ca192700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80c40f8a10 0x7f80c410e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50254/0 (socket says 172.21.15.113:50254) 2024-04-12T05:28:12.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80ca192700 1 -- 172.21.15.113:0/791598221 learned_addr learned my addr 172.21.15.113:0/791598221 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:12.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80c9991700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80c40f92e0 0x7f80c410e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:12.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80ca192700 1 -- 172.21.15.113:0/791598221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80c40f92e0 msgr2=0x7f80c410e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:12.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80ca192700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80c40f92e0 0x7f80c410e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:12.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80ca192700 1 -- 172.21.15.113:0/791598221 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c0005040 con 0x7f80c40f8a10 2024-04-12T05:28:12.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80c9991700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80c40f92e0 0x7f80c410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:28:12.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80ca192700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80c40f8a10 0x7f80c410e060 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f80c000b120 tx=0x7f80c0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:12.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80bb7fe700 1 -- 172.21.15.113:0/791598221 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80c000e030 con 0x7f80c40f8a10 2024-04-12T05:28:12.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80bb7fe700 1 -- 172.21.15.113:0/791598221 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80c0007e90 con 0x7f80c40f8a10 2024-04-12T05:28:12.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80cc3f6700 1 -- 172.21.15.113:0/791598221 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c41082c0 con 0x7f80c40f8a10 2024-04-12T05:28:12.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80bb7fe700 1 -- 172.21.15.113:0/791598221 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80c0018070 con 0x7f80c40f8a10 2024-04-12T05:28:12.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.994+0000 7f80cc3f6700 1 -- 172.21.15.113:0/791598221 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c41087a0 con 0x7f80c40f8a10 2024-04-12T05:28:13.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.995+0000 7f80cc3f6700 1 -- 172.21.15.113:0/791598221 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c405f5a0 con 0x7f80c40f8a10 2024-04-12T05:28:13.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.996+0000 7f80bb7fe700 1 -- 172.21.15.113:0/791598221 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f80c001d420 con 0x7f80c40f8a10 2024-04-12T05:28:13.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.996+0000 7f80bb7fe700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80b006caa0 0x7f80b006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:13.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.996+0000 7f80bb7fe700 1 -- 172.21.15.113:0/791598221 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f80c008d300 con 0x7f80c40f8a10 2024-04-12T05:28:13.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.996+0000 7f80c9991700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80b006caa0 0x7f80b006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:13.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:12.997+0000 7f80c9991700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80b006caa0 0x7f80b006ef60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f80c41099a0 tx=0x7f80b4009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:13.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.001+0000 7f80bb7fe700 1 -- 172.21.15.113:0/791598221 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f80c005adb0 con 0x7f80c40f8a10 2024-04-12T05:28:13.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.267+0000 7f80cc3f6700 1 -- 172.21.15.113:0/791598221 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f80c4109780 con 0x7f80c40f8a10 2024-04-12T05:28:13.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.267+0000 7f80bb7fe700 1 -- 172.21.15.113:0/791598221 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f80c005a940 con 0x7f80c40f8a10 2024-04-12T05:28:13.270 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:28:13.270 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:28:13.270 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:28:13.270 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:28:13.271 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:28:13.272 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:28:13.272 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:28:13.272 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:28:13.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.269+0000 7f80cc3f6700 1 -- 172.21.15.113:0/791598221 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80b006caa0 msgr2=0x7f80b006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:13.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.269+0000 7f80cc3f6700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80b006caa0 0x7f80b006ef60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f80c41099a0 tx=0x7f80b4009040).stop 2024-04-12T05:28:13.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.269+0000 7f80cc3f6700 1 -- 172.21.15.113:0/791598221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80c40f8a10 msgr2=0x7f80c410e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:13.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.269+0000 7f80cc3f6700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80c40f8a10 0x7f80c410e060 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f80c000b120 tx=0x7f80c0009cb0).stop 2024-04-12T05:28:13.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.269+0000 7f80cc3f6700 1 -- 172.21.15.113:0/791598221 shutdown_connections 2024-04-12T05:28:13.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.269+0000 7f80cc3f6700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f80b006caa0 0x7f80b006ef60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.269+0000 7f80cc3f6700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80c40f8a10 0x7f80c410e060 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.269+0000 7f80cc3f6700 1 --2- 172.21.15.113:0/791598221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80c40f92e0 0x7f80c410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.269+0000 7f80cc3f6700 1 -- 172.21.15.113:0/791598221 >> 172.21.15.113:0/791598221 conn(0x7f80c406c760 msgr2=0x7f80c40fc250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:13.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.269+0000 7f80cc3f6700 1 -- 172.21.15.113:0/791598221 shutdown_connections 2024-04-12T05:28:13.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.269+0000 7f80cc3f6700 1 -- 172.21.15.113:0/791598221 wait complete. 2024-04-12T05:28:13.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.366+0000 7f116630c700 1 -- 172.21.15.113:0/223167035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11600f8940 msgr2=0x7f11600f8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:13.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.366+0000 7f116630c700 1 --2- 172.21.15.113:0/223167035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11600f8940 0x7f11600f8d20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f11500047f0 tx=0x7f1150009cb0).stop 2024-04-12T05:28:13.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.366+0000 7f116630c700 1 -- 172.21.15.113:0/223167035 shutdown_connections 2024-04-12T05:28:13.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.366+0000 7f116630c700 1 --2- 172.21.15.113:0/223167035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11600f9210 0x7f11601019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:28:13.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.366+0000 7f116630c700 1 --2- 172.21.15.113:0/223167035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11600f8940 0x7f11600f8d20 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.366+0000 7f116630c700 1 -- 172.21.15.113:0/223167035 >> 172.21.15.113:0/223167035 conn(0x7f1160070590 msgr2=0x7f11600f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:13.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.366+0000 7f116630c700 1 -- 172.21.15.113:0/223167035 shutdown_connections 2024-04-12T05:28:13.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.366+0000 7f116630c700 1 -- 172.21.15.113:0/223167035 wait complete. 2024-04-12T05:28:13.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.367+0000 7f116630c700 1 Processor -- start 2024-04-12T05:28:13.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.367+0000 7f116630c700 1 -- start start 2024-04-12T05:28:13.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.367+0000 7f116630c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11600f8940 0x7f116010e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:13.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.367+0000 7f116630c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11600f9210 0x7f116010e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:13.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.367+0000 7f116630c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116010ec20 con 0x7f11600f9210 2024-04-12T05:28:13.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.367+0000 7f116630c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1160108240 con 0x7f11600f8940 2024-04-12T05:28:13.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.367+0000 7f115f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11600f9210 0x7f116010e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:13.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.367+0000 7f115f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11600f9210 0x7f116010e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50278/0 (socket says 172.21.15.113:50278) 2024-04-12T05:28:13.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.367+0000 7f115f7fe700 1 -- 172.21.15.113:0/3926527910 learned_addr learned my addr 172.21.15.113:0/3926527910 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:13.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.368+0000 7f115ffff700 1 --2- 172.21.15.113:0/3926527910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11600f8940 0x7f116010e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:13.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.368+0000 7f115f7fe700 1 -- 172.21.15.113:0/3926527910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11600f8940 msgr2=0x7f116010e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:13.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.368+0000 7f115f7fe700 1 --2- 172.21.15.113:0/3926527910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11600f8940 0x7f116010e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.368+0000 7f115f7fe700 1 -- 172.21.15.113:0/3926527910 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150005040 con 0x7f11600f9210 2024-04-12T05:28:13.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.368+0000 7f115f7fe700 1 --2- 172.21.15.113:0/3926527910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11600f9210 0x7f116010e590 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f115400b4b0 tx=0x7f11540090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:13.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.368+0000 7f115d7fa700 1 -- 172.21.15.113:0/3926527910 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1154002500 con 0x7f11600f9210 2024-04-12T05:28:13.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.368+0000 7f116630c700 1 -- 172.21.15.113:0/3926527910 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11601084f0 con 0x7f11600f9210 2024-04-12T05:28:13.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.368+0000 7f115d7fa700 1 -- 172.21.15.113:0/3926527910 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1154002b10 con 0x7f11600f9210 2024-04-12T05:28:13.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.368+0000 7f115d7fa700 1 -- 172.21.15.113:0/3926527910 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1154012cf0 con 0x7f11600f9210 2024-04-12T05:28:13.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.368+0000 7f116630c700 1 -- 172.21.15.113:0/3926527910 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1160108a30 con 0x7f11600f9210 2024-04-12T05:28:13.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.369+0000 7f116630c700 1 -- 172.21.15.113:0/3926527910 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116005f590 con 0x7f11600f9210 2024-04-12T05:28:13.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.369+0000 7f115d7fa700 1 -- 172.21.15.113:0/3926527910 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1154002660 con 0x7f11600f9210 2024-04-12T05:28:13.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.370+0000 7f115d7fa700 1 --2- 172.21.15.113:0/3926527910 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f114806cab0 0x7f114806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:13.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.370+0000 7f115d7fa700 1 -- 172.21.15.113:0/3926527910 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f115408bf30 con 0x7f11600f9210 2024-04-12T05:28:13.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.370+0000 7f115ffff700 1 --2- 172.21.15.113:0/3926527910 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f114806cab0 0x7f114806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:13.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.370+0000 7f115ffff700 1 --2- 172.21.15.113:0/3926527910 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f114806cab0 0x7f114806ef70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f11500047c0 tx=0x7f1150009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:13.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.375+0000 7f115d7fa700 1 -- 172.21.15.113:0/3926527910 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f11540bf2c0 con 0x7f11600f9210 2024-04-12T05:28:13.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:13 smithi113 ceph-mon[103371]: pgmap v481: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:13.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:13 smithi192 ceph-mon[85304]: pgmap v481: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:13.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.603+0000 7f116630c700 1 -- 172.21.15.113:0/3926527910 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1160047540 con 0x7f11600f9210 2024-04-12T05:28:13.606 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.604+0000 7f115d7fa700 1 -- 172.21.15.113:0/3926527910 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f11540599e0 con 0x7f11600f9210 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:28:13.607 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:28:13.608 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:28:13.608 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:28:13.608 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:28:13.608 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:28:13.608 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:28:13.608 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:28:13.608 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:28:13.608 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:28:13.608 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:28:13.608 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:28:13.608 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:28:13.609 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:28:13.610 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:28:13.610 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:28:13.610 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:28:13.610 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:28:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.605+0000 7f116630c700 1 -- 172.21.15.113:0/3926527910 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f114806cab0 msgr2=0x7f114806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.605+0000 7f116630c700 1 --2- 172.21.15.113:0/3926527910 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f114806cab0 0x7f114806ef70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f11500047c0 tx=0x7f1150009cb0).stop 2024-04-12T05:28:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.606+0000 7f116630c700 1 -- 172.21.15.113:0/3926527910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11600f9210 msgr2=0x7f116010e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.606+0000 7f116630c700 1 --2- 172.21.15.113:0/3926527910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11600f9210 0x7f116010e590 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f115400b4b0 tx=0x7f11540090f0).stop 2024-04-12T05:28:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.606+0000 7f116630c700 1 -- 172.21.15.113:0/3926527910 shutdown_connections 2024-04-12T05:28:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.606+0000 7f116630c700 1 --2- 172.21.15.113:0/3926527910 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f114806cab0 0x7f114806ef70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.606+0000 7f116630c700 1 --2- 172.21.15.113:0/3926527910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11600f8940 0x7f116010e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.606+0000 7f116630c700 1 --2- 172.21.15.113:0/3926527910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f11600f9210 0x7f116010e590 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.606+0000 7f116630c700 1 -- 172.21.15.113:0/3926527910 >> 172.21.15.113:0/3926527910 conn(0x7f1160070590 msgr2=0x7f11600fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.606+0000 7f116630c700 1 -- 172.21.15.113:0/3926527910 shutdown_connections 2024-04-12T05:28:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.606+0000 7f116630c700 1 -- 172.21.15.113:0/3926527910 wait complete. 2024-04-12T05:28:13.612 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:28:13.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.705+0000 7f986414f700 1 -- 172.21.15.113:0/1063822063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985c0fd150 msgr2=0x7f985c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:13.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.705+0000 7f986414f700 1 --2- 172.21.15.113:0/1063822063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985c0fd150 0x7f985c0f6930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f98580047f0 tx=0x7f9858009cb0).stop 2024-04-12T05:28:13.708 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.705+0000 7f986414f700 1 -- 172.21.15.113:0/1063822063 shutdown_connections 2024-04-12T05:28:13.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.705+0000 7f986414f700 1 --2- 172.21.15.113:0/1063822063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985c0fd150 0x7f985c0f6930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.705+0000 7f986414f700 1 --2- 172.21.15.113:0/1063822063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f985c0fc7f0 0x7f985c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.705+0000 7f986414f700 1 -- 172.21.15.113:0/1063822063 >> 172.21.15.113:0/1063822063 conn(0x7f985c070590 msgr2=0x7f985c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:13.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.705+0000 7f986414f700 1 -- 172.21.15.113:0/1063822063 shutdown_connections 2024-04-12T05:28:13.709 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.705+0000 7f986414f700 1 -- 172.21.15.113:0/1063822063 wait complete. 2024-04-12T05:28:13.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f986414f700 1 Processor -- start 2024-04-12T05:28:13.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f986414f700 1 -- start start 2024-04-12T05:28:13.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f986414f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985c0fc7f0 0x7f985c10dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:13.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f986414f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f985c0fd150 0x7f985c10e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:13.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f986414f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c10eab0 con 0x7f985c0fc7f0 2024-04-12T05:28:13.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f986414f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c10ebf0 con 0x7f985c0fd150 2024-04-12T05:28:13.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f9861eeb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985c0fc7f0 0x7f985c10dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:13.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f9861eeb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985c0fc7f0 0x7f985c10dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50298/0 (socket says 172.21.15.113:50298) 2024-04-12T05:28:13.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f9861eeb700 1 -- 172.21.15.113:0/3969474819 learned_addr learned my addr 172.21.15.113:0/3969474819 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:13.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f98616ea700 1 --2- 172.21.15.113:0/3969474819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f985c0fd150 0x7f985c10e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:13.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f9861eeb700 1 -- 172.21.15.113:0/3969474819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f985c0fd150 msgr2=0x7f985c10e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:13.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f9861eeb700 1 --2- 172.21.15.113:0/3969474819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f985c0fd150 0x7f985c10e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.706+0000 7f9861eeb700 1 -- 172.21.15.113:0/3969474819 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9858005040 con 0x7f985c0fc7f0 2024-04-12T05:28:13.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.707+0000 7f9861eeb700 1 --2- 172.21.15.113:0/3969474819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985c0fc7f0 0x7f985c10dfc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f984c0041d0 tx=0x7f984c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:13.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.707+0000 7f9852ffd700 1 -- 172.21.15.113:0/3969474819 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f984c002d90 con 0x7f985c0fc7f0 2024-04-12T05:28:13.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.707+0000 7f9852ffd700 1 -- 172.21.15.113:0/3969474819 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f984c00f040 con 0x7f985c0fc7f0 2024-04-12T05:28:13.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.707+0000 7f986414f700 1 -- 172.21.15.113:0/3969474819 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985c108280 con 0x7f985c0fc7f0 2024-04-12T05:28:13.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.707+0000 7f9852ffd700 1 -- 172.21.15.113:0/3969474819 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f984c00d800 con 0x7f985c0fc7f0 2024-04-12T05:28:13.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.707+0000 7f986414f700 1 -- 172.21.15.113:0/3969474819 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985c108840 con 0x7f985c0fc7f0 2024-04-12T05:28:13.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.708+0000 7f986414f700 1 -- 172.21.15.113:0/3969474819 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985c05f590 con 0x7f985c0fc7f0 2024-04-12T05:28:13.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.708+0000 7f9852ffd700 1 -- 172.21.15.113:0/3969474819 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f984c00da70 con 0x7f985c0fc7f0 2024-04-12T05:28:13.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.709+0000 7f9852ffd700 1 --2- 172.21.15.113:0/3969474819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f984806caa0 0x7f984806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:13.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.709+0000 7f9852ffd700 1 -- 172.21.15.113:0/3969474819 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f984c08bdd0 con 0x7f985c0fc7f0 2024-04-12T05:28:13.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.709+0000 7f98616ea700 1 --2- 172.21.15.113:0/3969474819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f984806caa0 0x7f984806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:13.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.709+0000 7f98616ea700 1 --2- 172.21.15.113:0/3969474819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f984806caa0 0x7f984806ef60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f9858004f10 tx=0x7f985800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.714+0000 7f9852ffd700 1 -- 172.21.15.113:0/3969474819 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f984c059880 con 0x7f985c0fc7f0 2024-04-12T05:28:13.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.918+0000 7f986414f700 1 -- 172.21.15.113:0/3969474819 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f985c109640 con 0x7f984806caa0 2024-04-12T05:28:13.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.919+0000 7f9852ffd700 1 -- 172.21.15.113:0/3969474819 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f985c109640 con 0x7f984806caa0 2024-04-12T05:28:13.922 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:28:13.922 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:28:13.922 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:28:13.922 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:28:13.922 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:28:13.923 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:28:13.923 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:28:13.923 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:28:13.923 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:28:13.923 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:28:13.923 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:28:13.923 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:28:13.923 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:28:13.923 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:28:13.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.922+0000 7f986414f700 1 -- 172.21.15.113:0/3969474819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f984806caa0 msgr2=0x7f984806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:13.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.922+0000 7f986414f700 1 --2- 172.21.15.113:0/3969474819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f984806caa0 0x7f984806ef60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f9858004f10 tx=0x7f985800b040).stop 2024-04-12T05:28:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.922+0000 7f986414f700 1 -- 172.21.15.113:0/3969474819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985c0fc7f0 msgr2=0x7f985c10dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.922+0000 7f986414f700 1 --2- 172.21.15.113:0/3969474819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985c0fc7f0 0x7f985c10dfc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f984c0041d0 tx=0x7f984c009f60).stop 2024-04-12T05:28:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.922+0000 7f986414f700 1 -- 172.21.15.113:0/3969474819 shutdown_connections 2024-04-12T05:28:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.922+0000 7f986414f700 1 --2- 172.21.15.113:0/3969474819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f984806caa0 0x7f984806ef60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.922+0000 7f986414f700 1 --2- 172.21.15.113:0/3969474819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f985c0fc7f0 0x7f985c10dfc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.922+0000 7f986414f700 1 --2- 172.21.15.113:0/3969474819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f985c0fd150 0x7f985c10e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.922+0000 7f986414f700 1 -- 172.21.15.113:0/3969474819 >> 172.21.15.113:0/3969474819 conn(0x7f985c070590 msgr2=0x7f985c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.922+0000 7f986414f700 1 -- 172.21.15.113:0/3969474819 shutdown_connections 2024-04-12T05:28:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:13.922+0000 7f986414f700 1 -- 172.21.15.113:0/3969474819 wait complete. 2024-04-12T05:28:14.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.020+0000 7fccb0b59700 1 -- 172.21.15.113:0/3510592320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccac0fd150 msgr2=0x7fccac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:14.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.020+0000 7fccb0b59700 1 --2- 172.21.15.113:0/3510592320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccac0fd150 0x7fccac0f6930 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fcca00047f0 tx=0x7fcca0009cb0).stop 2024-04-12T05:28:14.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.020+0000 7fccb0b59700 1 -- 172.21.15.113:0/3510592320 shutdown_connections 2024-04-12T05:28:14.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.020+0000 7fccb0b59700 1 --2- 172.21.15.113:0/3510592320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccac0fd150 0x7fccac0f6930 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:14.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.020+0000 7fccb0b59700 1 --2- 172.21.15.113:0/3510592320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccac0fc7f0 0x7fccac0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:14.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.020+0000 7fccb0b59700 1 -- 172.21.15.113:0/3510592320 >> 172.21.15.113:0/3510592320 conn(0x7fccac070590 msgr2=0x7fccac0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:14.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.020+0000 7fccb0b59700 1 -- 172.21.15.113:0/3510592320 shutdown_connections 2024-04-12T05:28:14.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.020+0000 7fccb0b59700 1 -- 172.21.15.113:0/3510592320 wait complete. 2024-04-12T05:28:14.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccb0b59700 1 Processor -- start 2024-04-12T05:28:14.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccb0b59700 1 -- start start 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccb0b59700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccac0fc7f0 0x7fccac0f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccb0b59700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccac0fd150 0x7fccac0f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccb0b59700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0f8f20 con 0x7fccac0fc7f0 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccb0b59700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0f9060 con 0x7fccac0fd150 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccaa59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccac0fc7f0 0x7fccac0f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccaa59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccac0fc7f0 0x7fccac0f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50314/0 (socket says 172.21.15.113:50314) 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccaa59c700 1 -- 172.21.15.113:0/1447947782 learned_addr learned my addr 172.21.15.113:0/1447947782 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fcca9d9b700 1 --2- 172.21.15.113:0/1447947782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccac0fd150 0x7fccac0f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccaa59c700 1 -- 172.21.15.113:0/1447947782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccac0fd150 msgr2=0x7fccac0f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccaa59c700 1 --2- 172.21.15.113:0/1447947782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccac0fd150 0x7fccac0f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccaa59c700 1 -- 172.21.15.113:0/1447947782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca0005040 con 0x7fccac0fc7f0 2024-04-12T05:28:14.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.021+0000 7fccaa59c700 1 --2- 172.21.15.113:0/1447947782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccac0fc7f0 0x7fccac0f8540 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fcc9c0041d0 tx=0x7fcc9c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:14.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.022+0000 7fcc9b7fe700 1 -- 172.21.15.113:0/1447947782 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc9c007b40 con 0x7fccac0fc7f0 2024-04-12T05:28:14.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.022+0000 7fcc9b7fe700 1 -- 172.21.15.113:0/1447947782 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc9c00f040 con 0x7fccac0fc7f0 2024-04-12T05:28:14.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.022+0000 7fccb0b59700 1 -- 172.21.15.113:0/1447947782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac0f9310 con 0x7fccac0fc7f0 2024-04-12T05:28:14.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.022+0000 7fcc9b7fe700 1 -- 172.21.15.113:0/1447947782 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc9c0147b0 con 0x7fccac0fc7f0 2024-04-12T05:28:14.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.022+0000 7fccb0b59700 1 -- 172.21.15.113:0/1447947782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccac1146a0 con 0x7fccac0fc7f0 2024-04-12T05:28:14.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.023+0000 7fccb0b59700 1 -- 172.21.15.113:0/1447947782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccac05f590 con 0x7fccac0fc7f0 2024-04-12T05:28:14.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.023+0000 7fcc9b7fe700 1 -- 172.21.15.113:0/1447947782 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcc9c009110 con 0x7fccac0fc7f0 2024-04-12T05:28:14.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.023+0000 7fcc9b7fe700 1 --2- 172.21.15.113:0/1447947782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc9406caa0 0x7fcc9406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:14.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.024+0000 7fcc9b7fe700 1 -- 172.21.15.113:0/1447947782 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fcc9c08be40 con 0x7fccac0fc7f0 2024-04-12T05:28:14.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.024+0000 7fcca9d9b700 1 --2- 172.21.15.113:0/1447947782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc9406caa0 0x7fcc9406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:14.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.024+0000 7fcca9d9b700 1 --2- 172.21.15.113:0/1447947782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc9406caa0 0x7fcc9406ef60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fccac0f9ce0 tx=0x7fcca000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:14.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.028+0000 7fcc9b7fe700 1 -- 172.21.15.113:0/1447947782 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fcc9c055e60 con 0x7fccac0fc7f0 2024-04-12T05:28:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.299+0000 7fccb0b59700 1 -- 172.21.15.113:0/1447947782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fccac047540 con 0x7fccac0fc7f0 2024-04-12T05:28:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.299+0000 7fcc9b7fe700 1 -- 172.21.15.113:0/1447947782 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fcc9c059480 con 0x7fccac0fc7f0 2024-04-12T05:28:14.301 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:28:14.301 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:28:14.301 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:28:14.301 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:28:14.301 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:28:14.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.301+0000 7fccb0b59700 1 -- 172.21.15.113:0/1447947782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc9406caa0 msgr2=0x7fcc9406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:14.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.301+0000 7fccb0b59700 1 --2- 172.21.15.113:0/1447947782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc9406caa0 0x7fcc9406ef60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fccac0f9ce0 tx=0x7fcca000b040).stop 2024-04-12T05:28:14.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.301+0000 7fccb0b59700 1 -- 172.21.15.113:0/1447947782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccac0fc7f0 msgr2=0x7fccac0f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:14.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.301+0000 7fccb0b59700 1 --2- 172.21.15.113:0/1447947782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccac0fc7f0 0x7fccac0f8540 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fcc9c0041d0 tx=0x7fcc9c009f60).stop 2024-04-12T05:28:14.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.301+0000 7fccb0b59700 1 -- 172.21.15.113:0/1447947782 shutdown_connections 2024-04-12T05:28:14.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.301+0000 7fccb0b59700 1 --2- 172.21.15.113:0/1447947782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc9406caa0 0x7fcc9406ef60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:14.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.301+0000 7fccb0b59700 1 --2- 172.21.15.113:0/1447947782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccac0fc7f0 0x7fccac0f8540 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:14.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.301+0000 7fccb0b59700 1 --2- 172.21.15.113:0/1447947782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccac0fd150 0x7fccac0f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:14.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.301+0000 7fccb0b59700 1 -- 172.21.15.113:0/1447947782 >> 172.21.15.113:0/1447947782 conn(0x7fccac070590 msgr2=0x7fccac060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:14.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.301+0000 7fccb0b59700 1 -- 172.21.15.113:0/1447947782 shutdown_connections 2024-04-12T05:28:14.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:14.301+0000 7fccb0b59700 1 -- 172.21.15.113:0/1447947782 wait complete. 2024-04-12T05:28:14.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:14 smithi113 ceph-mon[103371]: from='client.34896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:14.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:14 smithi113 ceph-mon[103371]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:14.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:14 smithi113 ceph-mon[103371]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:14.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:14 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/791598221' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:28:14.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:14 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3926527910' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:28:14.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:14 smithi192 ceph-mon[85304]: from='client.34896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:14.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:14 smithi192 ceph-mon[85304]: from='client.34900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:14.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:14 smithi192 ceph-mon[85304]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:14.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:14 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/791598221' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:28:14.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:14 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3926527910' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:28:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:15 smithi113 ceph-mon[103371]: from='client.34916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:15 smithi113 ceph-mon[103371]: pgmap v482: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:15.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:15 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1447947782' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:28:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:15 smithi192 ceph-mon[85304]: from='client.34916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:15 smithi192 ceph-mon[85304]: pgmap v482: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:15 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1447947782' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:28:17.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:17 smithi113 ceph-mon[103371]: pgmap v483: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:17.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:17 smithi192 ceph-mon[85304]: pgmap v483: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:19 smithi113 ceph-mon[103371]: pgmap v484: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:19 smithi192 ceph-mon[85304]: pgmap v484: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:21.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:21 smithi113 ceph-mon[103371]: pgmap v485: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:21.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:21 smithi192 ceph-mon[85304]: pgmap v485: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:22.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:28:22.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:28:23.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:23 smithi113 ceph-mon[103371]: pgmap v486: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:23.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:23 smithi192 ceph-mon[85304]: pgmap v486: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:25.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:25 smithi113 ceph-mon[103371]: pgmap v487: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:25.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:25 smithi192 ceph-mon[85304]: pgmap v487: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:26.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:26 smithi113 ceph-mon[103371]: pgmap v488: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:26.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:26 smithi192 ceph-mon[85304]: pgmap v488: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:29.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:29 smithi113 ceph-mon[103371]: pgmap v489: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:29.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:29 smithi192 ceph-mon[85304]: pgmap v489: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:31.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:31 smithi113 ceph-mon[103371]: pgmap v490: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:31.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:31 smithi192 ceph-mon[85304]: pgmap v490: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:33.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:33 smithi113 ceph-mon[103371]: pgmap v491: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:33.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:33 smithi192 ceph-mon[85304]: pgmap v491: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:35.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:35 smithi113 ceph-mon[103371]: pgmap v492: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:35.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:35 smithi192 ceph-mon[85304]: pgmap v492: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:37.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:37 smithi113 ceph-mon[103371]: pgmap v493: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:37.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:28:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:37 smithi192 ceph-mon[85304]: pgmap v493: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:37.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:28:39.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:39 smithi113 ceph-mon[103371]: pgmap v494: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:39.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:39 smithi192 ceph-mon[85304]: pgmap v494: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:41.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:41 smithi113 ceph-mon[103371]: pgmap v495: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:41.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:41 smithi192 ceph-mon[85304]: pgmap v495: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:43.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:43 smithi113 ceph-mon[103371]: pgmap v496: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:43.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:43 smithi192 ceph-mon[85304]: pgmap v496: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:44.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.413+0000 7fb64feef700 1 -- 172.21.15.113:0/1551802239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6480fd8f0 msgr2=0x7fb6480fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:44.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.413+0000 7fb64feef700 1 --2- 172.21.15.113:0/1551802239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6480fd8f0 0x7fb6480fdcd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fb6380047f0 tx=0x7fb638009cb0).stop 2024-04-12T05:28:44.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.413+0000 7fb64feef700 1 -- 172.21.15.113:0/1551802239 shutdown_connections 2024-04-12T05:28:44.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.413+0000 7fb64feef700 1 --2- 172.21.15.113:0/1551802239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6480fe250 0x7fb6480f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.413+0000 7fb64feef700 1 --2- 172.21.15.113:0/1551802239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6480fd8f0 0x7fb6480fdcd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.413+0000 7fb64feef700 1 -- 172.21.15.113:0/1551802239 >> 172.21.15.113:0/1551802239 conn(0x7fb6480f34b0 msgr2=0x7fb6480f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:44.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.413+0000 7fb64feef700 1 -- 172.21.15.113:0/1551802239 shutdown_connections 2024-04-12T05:28:44.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.413+0000 7fb64feef700 1 -- 172.21.15.113:0/1551802239 wait complete. 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.414+0000 7fb64feef700 1 Processor -- start 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.414+0000 7fb64feef700 1 -- start start 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.414+0000 7fb64feef700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6480fd8f0 0x7fb64810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.414+0000 7fb64feef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6480fe250 0x7fb64810e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.414+0000 7fb64feef700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64810ead0 con 0x7fb6480fd8f0 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.414+0000 7fb64feef700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64810ec10 con 0x7fb6480fe250 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.414+0000 7fb64dc8b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6480fd8f0 0x7fb64810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.414+0000 7fb64dc8b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6480fd8f0 0x7fb64810dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42730/0 (socket says 172.21.15.113:42730) 2024-04-12T05:28:44.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.414+0000 7fb64dc8b700 1 -- 172.21.15.113:0/1107571115 learned_addr learned my addr 172.21.15.113:0/1107571115 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.414+0000 7fb64d48a700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6480fe250 0x7fb64810e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.415+0000 7fb64dc8b700 1 -- 172.21.15.113:0/1107571115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6480fe250 msgr2=0x7fb64810e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.415+0000 7fb64dc8b700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6480fe250 0x7fb64810e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.415+0000 7fb64dc8b700 1 -- 172.21.15.113:0/1107571115 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb638005040 con 0x7fb6480fd8f0 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.415+0000 7fb64d48a700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6480fe250 0x7fb64810e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.415+0000 7fb64dc8b700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6480fd8f0 0x7fb64810dfe0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fb638005010 tx=0x7fb638009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.415+0000 7fb63effd700 1 -- 172.21.15.113:0/1107571115 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb63800e030 con 0x7fb6480fd8f0 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.415+0000 7fb63effd700 1 -- 172.21.15.113:0/1107571115 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb638007e90 con 0x7fb6480fd8f0 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.415+0000 7fb64feef700 1 -- 172.21.15.113:0/1107571115 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb648108240 con 0x7fb6480fd8f0 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.415+0000 7fb63effd700 1 -- 172.21.15.113:0/1107571115 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb638018070 con 0x7fb6480fd8f0 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.415+0000 7fb64feef700 1 -- 172.21.15.113:0/1107571115 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6481087a0 con 0x7fb6480fd8f0 2024-04-12T05:28:44.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.416+0000 7fb64feef700 1 -- 172.21.15.113:0/1107571115 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64805f590 con 0x7fb6480fd8f0 2024-04-12T05:28:44.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.417+0000 7fb63effd700 1 -- 172.21.15.113:0/1107571115 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb63801d6e0 con 0x7fb6480fd8f0 2024-04-12T05:28:44.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.417+0000 7fb63effd700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb63406ca30 0x7fb63406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:44.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.417+0000 7fb63effd700 1 -- 172.21.15.113:0/1107571115 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb63808d100 con 0x7fb6480fd8f0 2024-04-12T05:28:44.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.417+0000 7fb64d48a700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb63406ca30 0x7fb63406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:44.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.417+0000 7fb64d48a700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb63406ca30 0x7fb63406eef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fb648109920 tx=0x7fb644009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:44.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.422+0000 7fb63effd700 1 -- 172.21.15.113:0/1107571115 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb63805a920 con 0x7fb6480fd8f0 2024-04-12T05:28:44.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.628+0000 7fb64feef700 1 -- 172.21.15.113:0/1107571115 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6480ffc40 con 0x7fb63406ca30 2024-04-12T05:28:44.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.629+0000 7fb63effd700 1 -- 172.21.15.113:0/1107571115 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb6480ffc40 con 0x7fb63406ca30 2024-04-12T05:28:44.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.631+0000 7fb64feef700 1 -- 172.21.15.113:0/1107571115 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb63406ca30 msgr2=0x7fb63406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:44.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.631+0000 7fb64feef700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb63406ca30 0x7fb63406eef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fb648109920 tx=0x7fb644009d60).stop 2024-04-12T05:28:44.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.631+0000 7fb64feef700 1 -- 172.21.15.113:0/1107571115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6480fd8f0 msgr2=0x7fb64810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:44.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.631+0000 7fb64feef700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6480fd8f0 0x7fb64810dfe0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fb638005010 tx=0x7fb638009cb0).stop 2024-04-12T05:28:44.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.631+0000 7fb64feef700 1 -- 172.21.15.113:0/1107571115 shutdown_connections 2024-04-12T05:28:44.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.631+0000 7fb64feef700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb63406ca30 0x7fb63406eef0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.631+0000 7fb64feef700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6480fd8f0 0x7fb64810dfe0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.631+0000 7fb64feef700 1 --2- 172.21.15.113:0/1107571115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6480fe250 0x7fb64810e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.631+0000 7fb64feef700 1 -- 172.21.15.113:0/1107571115 >> 172.21.15.113:0/1107571115 conn(0x7fb6480f34b0 msgr2=0x7fb6480fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:44.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.631+0000 7fb64feef700 1 -- 172.21.15.113:0/1107571115 shutdown_connections 2024-04-12T05:28:44.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.631+0000 7fb64feef700 1 -- 172.21.15.113:0/1107571115 wait complete. 2024-04-12T05:28:44.643 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:28:44.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.730+0000 7fdad0b60700 1 -- 172.21.15.113:0/3181506656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdacc0fd8f0 msgr2=0x7fdacc0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:44.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.730+0000 7fdad0b60700 1 --2- 172.21.15.113:0/3181506656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdacc0fd8f0 0x7fdacc0fdcd0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fdac00047f0 tx=0x7fdac0009cb0).stop 2024-04-12T05:28:44.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.730+0000 7fdad0b60700 1 -- 172.21.15.113:0/3181506656 shutdown_connections 2024-04-12T05:28:44.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.730+0000 7fdad0b60700 1 --2- 172.21.15.113:0/3181506656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdacc0fe250 0x7fdacc0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.730+0000 7fdad0b60700 1 --2- 172.21.15.113:0/3181506656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdacc0fd8f0 0x7fdacc0fdcd0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.730+0000 7fdad0b60700 1 -- 172.21.15.113:0/3181506656 >> 172.21.15.113:0/3181506656 conn(0x7fdacc0f34b0 msgr2=0x7fdacc0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:44.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.730+0000 7fdad0b60700 1 -- 172.21.15.113:0/3181506656 shutdown_connections 2024-04-12T05:28:44.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.730+0000 7fdad0b60700 1 -- 172.21.15.113:0/3181506656 wait complete. 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.731+0000 7fdad0b60700 1 Processor -- start 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.731+0000 7fdad0b60700 1 -- start start 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.731+0000 7fdad0b60700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdacc0fd8f0 0x7fdacc10dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.731+0000 7fdad0b60700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdacc0fe250 0x7fdacc10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.731+0000 7fdad0b60700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc10ead0 con 0x7fdacc0fd8f0 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.731+0000 7fdad0b60700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc10ec10 con 0x7fdacc0fe250 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.731+0000 7fdaca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdacc0fd8f0 0x7fdacc10dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.731+0000 7fdaca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdacc0fd8f0 0x7fdacc10dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42750/0 (socket says 172.21.15.113:42750) 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.731+0000 7fdaca59c700 1 -- 172.21.15.113:0/4153319343 learned_addr learned my addr 172.21.15.113:0/4153319343 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.731+0000 7fdac9d9b700 1 --2- 172.21.15.113:0/4153319343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdacc0fe250 0x7fdacc10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.732+0000 7fdaca59c700 1 -- 172.21.15.113:0/4153319343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdacc0fe250 msgr2=0x7fdacc10e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.732+0000 7fdaca59c700 1 --2- 172.21.15.113:0/4153319343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdacc0fe250 0x7fdacc10e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.732+0000 7fdaca59c700 1 -- 172.21.15.113:0/4153319343 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac0005040 con 0x7fdacc0fd8f0 2024-04-12T05:28:44.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.732+0000 7fdaca59c700 1 --2- 172.21.15.113:0/4153319343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdacc0fd8f0 0x7fdacc10dfe0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fdac0005010 tx=0x7fdac0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:44.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.732+0000 7fdabb7fe700 1 -- 172.21.15.113:0/4153319343 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdac000e030 con 0x7fdacc0fd8f0 2024-04-12T05:28:44.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.732+0000 7fdabb7fe700 1 -- 172.21.15.113:0/4153319343 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdac0007e90 con 0x7fdacc0fd8f0 2024-04-12T05:28:44.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.732+0000 7fdad0b60700 1 -- 172.21.15.113:0/4153319343 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdacc108240 con 0x7fdacc0fd8f0 2024-04-12T05:28:44.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.732+0000 7fdabb7fe700 1 -- 172.21.15.113:0/4153319343 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdac0018070 con 0x7fdacc0fd8f0 2024-04-12T05:28:44.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.732+0000 7fdad0b60700 1 -- 172.21.15.113:0/4153319343 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdacc108720 con 0x7fdacc0fd8f0 2024-04-12T05:28:44.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.733+0000 7fdad0b60700 1 -- 172.21.15.113:0/4153319343 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdacc05f590 con 0x7fdacc0fd8f0 2024-04-12T05:28:44.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.733+0000 7fdabb7fe700 1 -- 172.21.15.113:0/4153319343 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdac001d670 con 0x7fdacc0fd8f0 2024-04-12T05:28:44.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.734+0000 7fdabb7fe700 1 --2- 172.21.15.113:0/4153319343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdab406caa0 0x7fdab406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:44.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.734+0000 7fdabb7fe700 1 -- 172.21.15.113:0/4153319343 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fdac008d0d0 con 0x7fdacc0fd8f0 2024-04-12T05:28:44.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.734+0000 7fdac9d9b700 1 --2- 172.21.15.113:0/4153319343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdab406caa0 0x7fdab406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:44.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.734+0000 7fdac9d9b700 1 --2- 172.21.15.113:0/4153319343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdab406caa0 0x7fdab406ef60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fdacc109920 tx=0x7fdabc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:44.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.738+0000 7fdabb7fe700 1 -- 172.21.15.113:0/4153319343 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fdac005ab80 con 0x7fdacc0fd8f0 2024-04-12T05:28:44.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.947+0000 7fdad0b60700 1 -- 172.21.15.113:0/4153319343 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdacc0ffc40 con 0x7fdab406caa0 2024-04-12T05:28:44.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.948+0000 7fdabb7fe700 1 -- 172.21.15.113:0/4153319343 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdacc0ffc40 con 0x7fdab406caa0 2024-04-12T05:28:44.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.950+0000 7fdad0b60700 1 -- 172.21.15.113:0/4153319343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdab406caa0 msgr2=0x7fdab406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:44.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.950+0000 7fdad0b60700 1 --2- 172.21.15.113:0/4153319343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdab406caa0 0x7fdab406ef60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fdacc109920 tx=0x7fdabc00b040).stop 2024-04-12T05:28:44.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.950+0000 7fdad0b60700 1 -- 172.21.15.113:0/4153319343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdacc0fd8f0 msgr2=0x7fdacc10dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:44.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.950+0000 7fdad0b60700 1 --2- 172.21.15.113:0/4153319343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdacc0fd8f0 0x7fdacc10dfe0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fdac0005010 tx=0x7fdac0009cb0).stop 2024-04-12T05:28:44.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.950+0000 7fdad0b60700 1 -- 172.21.15.113:0/4153319343 shutdown_connections 2024-04-12T05:28:44.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.950+0000 7fdad0b60700 1 --2- 172.21.15.113:0/4153319343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdab406caa0 0x7fdab406ef60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.950+0000 7fdad0b60700 1 --2- 172.21.15.113:0/4153319343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdacc0fd8f0 0x7fdacc10dfe0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.950+0000 7fdad0b60700 1 --2- 172.21.15.113:0/4153319343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdacc0fe250 0x7fdacc10e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:44.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.950+0000 7fdad0b60700 1 -- 172.21.15.113:0/4153319343 >> 172.21.15.113:0/4153319343 conn(0x7fdacc0f34b0 msgr2=0x7fdacc0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:44.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.950+0000 7fdad0b60700 1 -- 172.21.15.113:0/4153319343 shutdown_connections 2024-04-12T05:28:44.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:44.950+0000 7fdad0b60700 1 -- 172.21.15.113:0/4153319343 wait complete. 2024-04-12T05:28:45.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.047+0000 7f6734568700 1 -- 172.21.15.113:0/1296079964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f672c0f9260 msgr2=0x7f672c101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.048+0000 7f6734568700 1 --2- 172.21.15.113:0/1296079964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f672c0f9260 0x7f672c101970 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f67280047f0 tx=0x7f6728009cb0).stop 2024-04-12T05:28:45.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.048+0000 7f6734568700 1 -- 172.21.15.113:0/1296079964 shutdown_connections 2024-04-12T05:28:45.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.048+0000 7f6734568700 1 --2- 172.21.15.113:0/1296079964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f672c0f9260 0x7f672c101970 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.048+0000 7f6734568700 1 --2- 172.21.15.113:0/1296079964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f672c0f8900 0x7f672c0f8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.048+0000 7f6734568700 1 -- 172.21.15.113:0/1296079964 >> 172.21.15.113:0/1296079964 conn(0x7f672c070590 msgr2=0x7f672c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.048+0000 7f6734568700 1 -- 172.21.15.113:0/1296079964 shutdown_connections 2024-04-12T05:28:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.048+0000 7f6734568700 1 -- 172.21.15.113:0/1296079964 wait complete. 2024-04-12T05:28:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.048+0000 7f6734568700 1 Processor -- start 2024-04-12T05:28:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.048+0000 7f6734568700 1 -- start start 2024-04-12T05:28:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.048+0000 7f6734568700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f672c0f8900 0x7f672c10e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6734568700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f672c0f9260 0x7f672c10e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6734568700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c10eb00 con 0x7f672c0f9260 2024-04-12T05:28:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6734568700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c10ec40 con 0x7f672c0f8900 2024-04-12T05:28:45.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6731b03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f672c0f9260 0x7f672c10e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6731b03700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f672c0f9260 0x7f672c10e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42764/0 (socket says 172.21.15.113:42764) 2024-04-12T05:28:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6731b03700 1 -- 172.21.15.113:0/3908440139 learned_addr learned my addr 172.21.15.113:0/3908440139 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6732304700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f672c0f8900 0x7f672c10e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6731b03700 1 -- 172.21.15.113:0/3908440139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f672c0f8900 msgr2=0x7f672c10e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6731b03700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f672c0f8900 0x7f672c10e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6731b03700 1 -- 172.21.15.113:0/3908440139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6728005040 con 0x7f672c0f9260 2024-04-12T05:28:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6732304700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f672c0f8900 0x7f672c10e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:28:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6731b03700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f672c0f9260 0x7f672c10e500 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f6728009310 tx=0x7f6728009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f67237fe700 1 -- 172.21.15.113:0/3908440139 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f672800e030 con 0x7f672c0f9260 2024-04-12T05:28:45.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f67237fe700 1 -- 172.21.15.113:0/3908440139 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6728007e90 con 0x7f672c0f9260 2024-04-12T05:28:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f67237fe700 1 -- 172.21.15.113:0/3908440139 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6728018070 con 0x7f672c0f9260 2024-04-12T05:28:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6734568700 1 -- 172.21.15.113:0/3908440139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c108270 con 0x7f672c0f9260 2024-04-12T05:28:45.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.049+0000 7f6734568700 1 -- 172.21.15.113:0/3908440139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672c108750 con 0x7f672c0f9260 2024-04-12T05:28:45.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.050+0000 7f6734568700 1 -- 172.21.15.113:0/3908440139 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f672c05f590 con 0x7f672c0f9260 2024-04-12T05:28:45.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.051+0000 7f67237fe700 1 -- 172.21.15.113:0/3908440139 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f672801d420 con 0x7f672c0f9260 2024-04-12T05:28:45.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.051+0000 7f67237fe700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f671806caa0 0x7f671806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:45.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.051+0000 7f67237fe700 1 -- 172.21.15.113:0/3908440139 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f672808d390 con 0x7f672c0f9260 2024-04-12T05:28:45.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.051+0000 7f6732304700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f671806caa0 0x7f671806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:45.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.051+0000 7f6732304700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f671806caa0 0x7f671806ef60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f671c0041d0 tx=0x7f671c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:45.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.055+0000 7f67237fe700 1 -- 172.21.15.113:0/3908440139 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f672805ae40 con 0x7f672c0f9260 2024-04-12T05:28:45.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.253+0000 7f6734568700 1 -- 172.21.15.113:0/3908440139 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f672c1097a0 con 0x7f671806caa0 2024-04-12T05:28:45.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.263+0000 7f67237fe700 1 -- 172.21.15.113:0/3908440139 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f672c1097a0 con 0x7f671806caa0 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (16m) 9m ago 27m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (13m) 9m ago 27m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (13m) 9m ago 27m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (16m) 9m ago 27m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (9m) 9m ago 22m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (9m) 9m ago 22m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (9m) 9m ago 22m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (22m) 9m ago 22m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (18m) 9m ago 29m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (17m) 9m ago 27m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:28:45.265 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (15m) 9m ago 29m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:28:45.266 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (14m) 9m ago 27m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:28:45.266 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (17m) 9m ago 27m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:28:45.266 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (17m) 9m ago 26m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:28:45.266 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (13m) 9m ago 26m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:28:45.266 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (13m) 9m ago 25m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:28:45.266 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (12m) 9m ago 25m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:28:45.266 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (11m) 9m ago 24m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:28:45.266 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (11m) 9m ago 24m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:28:45.266 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (10m) 9m ago 24m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:28:45.266 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (16m) 9m ago 26m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:28:45.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.265+0000 7f6734568700 1 -- 172.21.15.113:0/3908440139 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f671806caa0 msgr2=0x7f671806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.265+0000 7f6734568700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f671806caa0 0x7f671806ef60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f671c0041d0 tx=0x7f671c009040).stop 2024-04-12T05:28:45.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.265+0000 7f6734568700 1 -- 172.21.15.113:0/3908440139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f672c0f9260 msgr2=0x7f672c10e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.265+0000 7f6734568700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f672c0f9260 0x7f672c10e500 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f6728009310 tx=0x7f6728009cb0).stop 2024-04-12T05:28:45.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.265+0000 7f6734568700 1 -- 172.21.15.113:0/3908440139 shutdown_connections 2024-04-12T05:28:45.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.265+0000 7f6734568700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f671806caa0 0x7f671806ef60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.265+0000 7f6734568700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f672c0f8900 0x7f672c10e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.265+0000 7f6734568700 1 --2- 172.21.15.113:0/3908440139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f672c0f9260 0x7f672c10e500 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.265+0000 7f6734568700 1 -- 172.21.15.113:0/3908440139 >> 172.21.15.113:0/3908440139 conn(0x7f672c070590 msgr2=0x7f672c0fc170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:45.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.265+0000 7f6734568700 1 -- 172.21.15.113:0/3908440139 shutdown_connections 2024-04-12T05:28:45.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.265+0000 7f6734568700 1 -- 172.21.15.113:0/3908440139 wait complete. 2024-04-12T05:28:45.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.361+0000 7f068f229700 1 -- 172.21.15.113:0/1199688752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06880fc830 msgr2=0x7f06880fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.361+0000 7f068f229700 1 --2- 172.21.15.113:0/1199688752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06880fc830 0x7f06880fcc10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f067c0047f0 tx=0x7f067c009cb0).stop 2024-04-12T05:28:45.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 -- 172.21.15.113:0/1199688752 shutdown_connections 2024-04-12T05:28:45.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 --2- 172.21.15.113:0/1199688752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06880fd190 0x7f06880f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 --2- 172.21.15.113:0/1199688752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06880fc830 0x7f06880fcc10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 -- 172.21.15.113:0/1199688752 >> 172.21.15.113:0/1199688752 conn(0x7f068806c760 msgr2=0x7f068806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:45.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 -- 172.21.15.113:0/1199688752 shutdown_connections 2024-04-12T05:28:45.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 -- 172.21.15.113:0/1199688752 wait complete. 2024-04-12T05:28:45.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 Processor -- start 2024-04-12T05:28:45.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 -- start start 2024-04-12T05:28:45.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06880fc830 0x7f06880f8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:45.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06880fd190 0x7f06880f8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:45.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06880f8f60 con 0x7f06880fc830 2024-04-12T05:28:45.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068f229700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06880f90a0 con 0x7f06880fd190 2024-04-12T05:28:45.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.362+0000 7f068cfc5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06880fc830 0x7f06880f8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:45.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f068cfc5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06880fc830 0x7f06880f8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42782/0 (socket says 172.21.15.113:42782) 2024-04-12T05:28:45.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f068cfc5700 1 -- 172.21.15.113:0/3649605639 learned_addr learned my addr 172.21.15.113:0/3649605639 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:45.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f0687fff700 1 --2- 172.21.15.113:0/3649605639 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06880fd190 0x7f06880f8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:45.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f068cfc5700 1 -- 172.21.15.113:0/3649605639 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06880fd190 msgr2=0x7f06880f8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f068cfc5700 1 --2- 172.21.15.113:0/3649605639 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06880fd190 0x7f06880f8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f068cfc5700 1 -- 172.21.15.113:0/3649605639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c005040 con 0x7f06880fc830 2024-04-12T05:28:45.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f068cfc5700 1 --2- 172.21.15.113:0/3649605639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06880fc830 0x7f06880f8580 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f067c009380 tx=0x7f067c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:45.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f0685ffb700 1 -- 172.21.15.113:0/3649605639 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f067c018070 con 0x7f06880fc830 2024-04-12T05:28:45.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f068f229700 1 -- 172.21.15.113:0/3649605639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06880f92f0 con 0x7f06880fc830 2024-04-12T05:28:45.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f0685ffb700 1 -- 172.21.15.113:0/3649605639 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f067c007e90 con 0x7f06880fc830 2024-04-12T05:28:45.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f0685ffb700 1 -- 172.21.15.113:0/3649605639 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f067c0123c0 con 0x7f06880fc830 2024-04-12T05:28:45.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.363+0000 7f068f229700 1 -- 172.21.15.113:0/3649605639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0688114720 con 0x7f06880fc830 2024-04-12T05:28:45.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.364+0000 7f068f229700 1 -- 172.21.15.113:0/3649605639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068805f550 con 0x7f06880fc830 2024-04-12T05:28:45.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.365+0000 7f0685ffb700 1 -- 172.21.15.113:0/3649605639 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f067c01d420 con 0x7f06880fc830 2024-04-12T05:28:45.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.365+0000 7f0685ffb700 1 --2- 172.21.15.113:0/3649605639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f067806caa0 0x7f067806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:45.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.365+0000 7f0685ffb700 1 -- 172.21.15.113:0/3649605639 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f067c08caa0 con 0x7f06880fc830 2024-04-12T05:28:45.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.365+0000 7f0687fff700 1 --2- 172.21.15.113:0/3649605639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f067806caa0 0x7f067806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:45.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.366+0000 7f0687fff700 1 --2- 172.21.15.113:0/3649605639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f067806caa0 0x7f067806ef60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f06880f9d20 tx=0x7f067400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:45.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.370+0000 7f0685ffb700 1 -- 172.21.15.113:0/3649605639 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f067c056a00 con 0x7f06880fc830 2024-04-12T05:28:45.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:45 smithi113 ceph-mon[103371]: pgmap v497: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:45.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:45 smithi192 ceph-mon[85304]: pgmap v497: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:45.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.636+0000 7f068f229700 1 -- 172.21.15.113:0/3649605639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0688114d30 con 0x7f06880fc830 2024-04-12T05:28:45.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.637+0000 7f0685ffb700 1 -- 172.21.15.113:0/3649605639 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f067c05a020 con 0x7f06880fc830 2024-04-12T05:28:45.639 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:28:45.639 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:28:45.639 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:28:45.639 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:28:45.639 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:28:45.639 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:28:45.639 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:28:45.639 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:28:45.639 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:28:45.639 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:28:45.640 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:28:45.640 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:28:45.640 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:28:45.640 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:28:45.640 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:28:45.640 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:28:45.640 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:28:45.640 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:28:45.640 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:28:45.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.639+0000 7f068f229700 1 -- 172.21.15.113:0/3649605639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f067806caa0 msgr2=0x7f067806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.639+0000 7f068f229700 1 --2- 172.21.15.113:0/3649605639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f067806caa0 0x7f067806ef60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f06880f9d20 tx=0x7f067400b040).stop 2024-04-12T05:28:45.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.639+0000 7f068f229700 1 -- 172.21.15.113:0/3649605639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06880fc830 msgr2=0x7f06880f8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.639+0000 7f068f229700 1 --2- 172.21.15.113:0/3649605639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06880fc830 0x7f06880f8580 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f067c009380 tx=0x7f067c009cb0).stop 2024-04-12T05:28:45.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.639+0000 7f068f229700 1 -- 172.21.15.113:0/3649605639 shutdown_connections 2024-04-12T05:28:45.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.639+0000 7f068f229700 1 --2- 172.21.15.113:0/3649605639 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f067806caa0 0x7f067806ef60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.639+0000 7f068f229700 1 --2- 172.21.15.113:0/3649605639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f06880fc830 0x7f06880f8580 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.639+0000 7f068f229700 1 --2- 172.21.15.113:0/3649605639 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06880fd190 0x7f06880f8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.639+0000 7f068f229700 1 -- 172.21.15.113:0/3649605639 >> 172.21.15.113:0/3649605639 conn(0x7f068806c760 msgr2=0x7f0688060dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:45.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.639+0000 7f068f229700 1 -- 172.21.15.113:0/3649605639 shutdown_connections 2024-04-12T05:28:45.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.639+0000 7f068f229700 1 -- 172.21.15.113:0/3649605639 wait complete. 2024-04-12T05:28:45.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.736+0000 7efd287bc700 1 -- 172.21.15.113:0/1616692179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd200fc7f0 msgr2=0x7efd200fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.736+0000 7efd287bc700 1 --2- 172.21.15.113:0/1616692179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd200fc7f0 0x7efd200fcbd0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7efd1c0047f0 tx=0x7efd1c009cb0).stop 2024-04-12T05:28:45.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.736+0000 7efd287bc700 1 -- 172.21.15.113:0/1616692179 shutdown_connections 2024-04-12T05:28:45.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.736+0000 7efd287bc700 1 --2- 172.21.15.113:0/1616692179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd200fd150 0x7efd200f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:28:45.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.736+0000 7efd287bc700 1 --2- 172.21.15.113:0/1616692179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd200fc7f0 0x7efd200fcbd0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.736+0000 7efd287bc700 1 -- 172.21.15.113:0/1616692179 >> 172.21.15.113:0/1616692179 conn(0x7efd20070590 msgr2=0x7efd200f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:45.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.736+0000 7efd287bc700 1 -- 172.21.15.113:0/1616692179 shutdown_connections 2024-04-12T05:28:45.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.737+0000 7efd287bc700 1 -- 172.21.15.113:0/1616692179 wait complete. 2024-04-12T05:28:45.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.737+0000 7efd287bc700 1 Processor -- start 2024-04-12T05:28:45.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.737+0000 7efd287bc700 1 -- start start 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.737+0000 7efd287bc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd200fc7f0 0x7efd2010e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.737+0000 7efd287bc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd200fd150 0x7efd2010e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.737+0000 7efd287bc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2010ebe0 con 0x7efd200fc7f0 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.737+0000 7efd287bc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd20108200 con 0x7efd200fd150 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.737+0000 7efd26558700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd200fc7f0 0x7efd2010e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.737+0000 7efd26558700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd200fc7f0 0x7efd2010e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42796/0 (socket says 172.21.15.113:42796) 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.737+0000 7efd26558700 1 -- 172.21.15.113:0/102821543 learned_addr learned my addr 172.21.15.113:0/102821543 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.738+0000 7efd25d57700 1 --2- 172.21.15.113:0/102821543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd200fd150 0x7efd2010e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.738+0000 7efd26558700 1 -- 172.21.15.113:0/102821543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd200fd150 msgr2=0x7efd2010e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.738+0000 7efd26558700 1 --2- 172.21.15.113:0/102821543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd200fd150 0x7efd2010e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.738+0000 7efd26558700 1 -- 172.21.15.113:0/102821543 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1c005040 con 0x7efd200fc7f0 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.738+0000 7efd26558700 1 --2- 172.21.15.113:0/102821543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd200fc7f0 0x7efd2010e060 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7efd1c00b120 tx=0x7efd1c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.738+0000 7efd177fe700 1 -- 172.21.15.113:0/102821543 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd1c00e030 con 0x7efd200fc7f0 2024-04-12T05:28:45.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.738+0000 7efd177fe700 1 -- 172.21.15.113:0/102821543 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd1c007e90 con 0x7efd200fc7f0 2024-04-12T05:28:45.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.738+0000 7efd287bc700 1 -- 172.21.15.113:0/102821543 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd20108450 con 0x7efd200fc7f0 2024-04-12T05:28:45.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.738+0000 7efd177fe700 1 -- 172.21.15.113:0/102821543 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd1c018070 con 0x7efd200fc7f0 2024-04-12T05:28:45.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.738+0000 7efd287bc700 1 -- 172.21.15.113:0/102821543 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd20108930 con 0x7efd200fc7f0 2024-04-12T05:28:45.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.739+0000 7efd287bc700 1 -- 172.21.15.113:0/102821543 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd2005f590 con 0x7efd200fc7f0 2024-04-12T05:28:45.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.739+0000 7efd177fe700 1 -- 172.21.15.113:0/102821543 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7efd1c01d420 con 0x7efd200fc7f0 2024-04-12T05:28:45.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.740+0000 7efd177fe700 1 --2- 172.21.15.113:0/102821543 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efd0c06caa0 0x7efd0c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:45.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.740+0000 7efd177fe700 1 -- 172.21.15.113:0/102821543 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7efd1c08d250 con 0x7efd200fc7f0 2024-04-12T05:28:45.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.740+0000 7efd25d57700 1 --2- 172.21.15.113:0/102821543 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efd0c06caa0 0x7efd0c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:45.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.740+0000 7efd25d57700 1 --2- 172.21.15.113:0/102821543 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efd0c06caa0 0x7efd0c06ef60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7efd201099a0 tx=0x7efd10008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:45.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.744+0000 7efd177fe700 1 -- 172.21.15.113:0/102821543 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7efd1c05ad00 con 0x7efd200fc7f0 2024-04-12T05:28:45.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.974+0000 7efd287bc700 1 -- 172.21.15.113:0/102821543 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efd20108fc0 con 0x7efd200fc7f0 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.975+0000 7efd177fe700 1 -- 172.21.15.113:0/102821543 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7efd1c05a890 con 0x7efd200fc7f0 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:28:45.977 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:28:45.978 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:28:45.979 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:28:45.980 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:28:45.980 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:28:45.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.977+0000 7efd287bc700 1 -- 172.21.15.113:0/102821543 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efd0c06caa0 msgr2=0x7efd0c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.977+0000 7efd287bc700 1 --2- 172.21.15.113:0/102821543 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efd0c06caa0 0x7efd0c06ef60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7efd201099a0 tx=0x7efd10008040).stop 2024-04-12T05:28:45.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.977+0000 7efd287bc700 1 -- 172.21.15.113:0/102821543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd200fc7f0 msgr2=0x7efd2010e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:45.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.977+0000 7efd287bc700 1 --2- 172.21.15.113:0/102821543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd200fc7f0 0x7efd2010e060 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7efd1c00b120 tx=0x7efd1c009cb0).stop 2024-04-12T05:28:45.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.977+0000 7efd287bc700 1 -- 172.21.15.113:0/102821543 shutdown_connections 2024-04-12T05:28:45.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.977+0000 7efd287bc700 1 --2- 172.21.15.113:0/102821543 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efd0c06caa0 0x7efd0c06ef60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.977+0000 7efd287bc700 1 --2- 172.21.15.113:0/102821543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd200fc7f0 0x7efd2010e060 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.977+0000 7efd287bc700 1 --2- 172.21.15.113:0/102821543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd200fd150 0x7efd2010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:45.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.977+0000 7efd287bc700 1 -- 172.21.15.113:0/102821543 >> 172.21.15.113:0/102821543 conn(0x7efd20070590 msgr2=0x7efd20060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:45.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.977+0000 7efd287bc700 1 -- 172.21.15.113:0/102821543 shutdown_connections 2024-04-12T05:28:45.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:45.977+0000 7efd287bc700 1 -- 172.21.15.113:0/102821543 wait complete. 2024-04-12T05:28:45.981 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:28:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.075+0000 7f0568831700 1 -- 172.21.15.113:0/3714431942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05640fd150 msgr2=0x7f05640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.075+0000 7f0568831700 1 --2- 172.21.15.113:0/3714431942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05640fd150 0x7f05640f6930 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f05580047f0 tx=0x7f0558009cb0).stop 2024-04-12T05:28:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.076+0000 7f0568831700 1 -- 172.21.15.113:0/3714431942 shutdown_connections 2024-04-12T05:28:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.076+0000 7f0568831700 1 --2- 172.21.15.113:0/3714431942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05640fd150 0x7f05640f6930 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:46.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.076+0000 7f0568831700 1 --2- 172.21.15.113:0/3714431942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05640fc7f0 0x7f05640fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:28:46.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.076+0000 7f0568831700 1 -- 172.21.15.113:0/3714431942 >> 172.21.15.113:0/3714431942 conn(0x7f05640f23b0 msgr2=0x7f05640f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:46.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.076+0000 7f0568831700 1 -- 172.21.15.113:0/3714431942 shutdown_connections 2024-04-12T05:28:46.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.076+0000 7f0568831700 1 -- 172.21.15.113:0/3714431942 wait complete. 2024-04-12T05:28:46.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.076+0000 7f0568831700 1 Processor -- start 2024-04-12T05:28:46.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.076+0000 7f0568831700 1 -- start start 2024-04-12T05:28:46.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f0568831700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05640fc7f0 0x7f056410e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f0568831700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05640fd150 0x7f056410e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f0568831700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056410ebf0 con 0x7f05640fc7f0 2024-04-12T05:28:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f0568831700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0564108210 con 0x7f05640fd150 2024-04-12T05:28:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f056259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05640fc7f0 0x7f056410e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f056259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05640fc7f0 0x7f056410e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42804/0 (socket says 172.21.15.113:42804) 2024-04-12T05:28:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f056259c700 1 -- 172.21.15.113:0/2184411175 learned_addr learned my addr 172.21.15.113:0/2184411175 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f0561d9b700 1 --2- 172.21.15.113:0/2184411175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05640fd150 0x7f056410e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f056259c700 1 -- 172.21.15.113:0/2184411175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05640fd150 msgr2=0x7f056410e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:46.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f056259c700 1 --2- 172.21.15.113:0/2184411175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05640fd150 0x7f056410e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:46.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f056259c700 1 -- 172.21.15.113:0/2184411175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0558005040 con 0x7f05640fc7f0 2024-04-12T05:28:46.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f056259c700 1 --2- 172.21.15.113:0/2184411175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05640fc7f0 0x7f056410e070 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f055400b4b0 tx=0x7f05540090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:46.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f05537fe700 1 -- 172.21.15.113:0/2184411175 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0554002500 con 0x7f05640fc7f0 2024-04-12T05:28:46.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f05537fe700 1 -- 172.21.15.113:0/2184411175 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0554002b10 con 0x7f05640fc7f0 2024-04-12T05:28:46.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f0568831700 1 -- 172.21.15.113:0/2184411175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05641084c0 con 0x7f05640fc7f0 2024-04-12T05:28:46.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f05537fe700 1 -- 172.21.15.113:0/2184411175 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0554012cb0 con 0x7f05640fc7f0 2024-04-12T05:28:46.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.077+0000 7f0568831700 1 -- 172.21.15.113:0/2184411175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05641089d0 con 0x7f05640fc7f0 2024-04-12T05:28:46.084 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.078+0000 7f0568831700 1 -- 172.21.15.113:0/2184411175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0564060370 con 0x7f05640fc7f0 2024-04-12T05:28:46.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.079+0000 7f05537fe700 1 -- 172.21.15.113:0/2184411175 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0554002660 con 0x7f05640fc7f0 2024-04-12T05:28:46.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.079+0000 7f05537fe700 1 --2- 172.21.15.113:0/2184411175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054c06caa0 0x7f054c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:46.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.079+0000 7f05537fe700 1 -- 172.21.15.113:0/2184411175 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f055408bf50 con 0x7f05640fc7f0 2024-04-12T05:28:46.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.079+0000 7f0561d9b700 1 --2- 172.21.15.113:0/2184411175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054c06caa0 0x7f054c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:46.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.080+0000 7f0561d9b700 1 --2- 172.21.15.113:0/2184411175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054c06caa0 0x7f054c06ef60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f0558004f10 tx=0x7f055800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:46.085 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.084+0000 7f05537fe700 1 -- 172.21.15.113:0/2184411175 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0554059940 con 0x7f05640fc7f0 2024-04-12T05:28:46.291 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.289+0000 7f0568831700 1 -- 172.21.15.113:0/2184411175 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0564103660 con 0x7f054c06caa0 2024-04-12T05:28:46.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.290+0000 7f05537fe700 1 -- 172.21.15.113:0/2184411175 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0564103660 con 0x7f054c06caa0 2024-04-12T05:28:46.292 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:28:46.292 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:28:46.292 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:28:46.292 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:28:46.292 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:28:46.293 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:28:46.293 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:28:46.293 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:28:46.293 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:28:46.293 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:28:46.293 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:28:46.293 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:28:46.293 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:28:46.293 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:28:46.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.292+0000 7f0568831700 1 -- 172.21.15.113:0/2184411175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054c06caa0 msgr2=0x7f054c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:46.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.292+0000 7f0568831700 1 --2- 172.21.15.113:0/2184411175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054c06caa0 0x7f054c06ef60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f0558004f10 tx=0x7f055800b040).stop 2024-04-12T05:28:46.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.292+0000 7f0568831700 1 -- 172.21.15.113:0/2184411175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05640fc7f0 msgr2=0x7f056410e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:46.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.292+0000 7f0568831700 1 --2- 172.21.15.113:0/2184411175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05640fc7f0 0x7f056410e070 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f055400b4b0 tx=0x7f05540090f0).stop 2024-04-12T05:28:46.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.292+0000 7f0568831700 1 -- 172.21.15.113:0/2184411175 shutdown_connections 2024-04-12T05:28:46.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.292+0000 7f0568831700 1 --2- 172.21.15.113:0/2184411175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054c06caa0 0x7f054c06ef60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:46.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.292+0000 7f0568831700 1 --2- 172.21.15.113:0/2184411175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05640fc7f0 0x7f056410e070 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:46.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.292+0000 7f0568831700 1 --2- 172.21.15.113:0/2184411175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05640fd150 0x7f056410e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:46.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.292+0000 7f0568831700 1 -- 172.21.15.113:0/2184411175 >> 172.21.15.113:0/2184411175 conn(0x7f05640f23b0 msgr2=0x7f0564061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:46.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.292+0000 7f0568831700 1 -- 172.21.15.113:0/2184411175 shutdown_connections 2024-04-12T05:28:46.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.292+0000 7f0568831700 1 -- 172.21.15.113:0/2184411175 wait complete. 2024-04-12T05:28:46.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.390+0000 7f49928c0700 1 -- 172.21.15.113:0/2733982103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0fd910 msgr2=0x7f498c0fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:46.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.390+0000 7f49928c0700 1 --2- 172.21.15.113:0/2733982103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0fd910 0x7f498c0fdcf0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f49800047f0 tx=0x7f4980009cb0).stop 2024-04-12T05:28:46.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.390+0000 7f49928c0700 1 -- 172.21.15.113:0/2733982103 shutdown_connections 2024-04-12T05:28:46.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.390+0000 7f49928c0700 1 --2- 172.21.15.113:0/2733982103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f498c0fe270 0x7f498c0f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:46.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.390+0000 7f49928c0700 1 --2- 172.21.15.113:0/2733982103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0fd910 0x7f498c0fdcf0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:46.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.390+0000 7f49928c0700 1 -- 172.21.15.113:0/2733982103 >> 172.21.15.113:0/2733982103 conn(0x7f498c0f3530 msgr2=0x7f498c0f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:46.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.390+0000 7f49928c0700 1 -- 172.21.15.113:0/2733982103 shutdown_connections 2024-04-12T05:28:46.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.390+0000 7f49928c0700 1 -- 172.21.15.113:0/2733982103 wait complete. 2024-04-12T05:28:46.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.390+0000 7f49928c0700 1 Processor -- start 2024-04-12T05:28:46.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.390+0000 7f49928c0700 1 -- start start 2024-04-12T05:28:46.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f49928c0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f498c0fd910 0x7f498c10e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:46.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f49928c0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0fe270 0x7f498c10e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:46.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f49928c0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c10eba0 con 0x7f498c0fe270 2024-04-12T05:28:46.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f49928c0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c10ece0 con 0x7f498c0fd910 2024-04-12T05:28:46.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f498b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0fe270 0x7f498c10e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:46.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f498b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0fe270 0x7f498c10e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42830/0 (socket says 172.21.15.113:42830) 2024-04-12T05:28:46.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f498b7fe700 1 -- 172.21.15.113:0/3319909987 learned_addr learned my addr 172.21.15.113:0/3319909987 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:28:46.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f498bfff700 1 --2- 172.21.15.113:0/3319909987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f498c0fd910 0x7f498c10e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f498b7fe700 1 -- 172.21.15.113:0/3319909987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f498c0fd910 msgr2=0x7f498c10e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f498b7fe700 1 --2- 172.21.15.113:0/3319909987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f498c0fd910 0x7f498c10e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f498b7fe700 1 -- 172.21.15.113:0/3319909987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4980005040 con 0x7f498c0fe270 2024-04-12T05:28:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f498b7fe700 1 --2- 172.21.15.113:0/3319909987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0fe270 0x7f498c10e5a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f497c002820 tx=0x7f497c009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f49897fa700 1 -- 172.21.15.113:0/3319909987 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f497c007b40 con 0x7f498c0fe270 2024-04-12T05:28:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f49897fa700 1 -- 172.21.15.113:0/3319909987 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f497c00f040 con 0x7f498c0fe270 2024-04-12T05:28:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f49928c0700 1 -- 172.21.15.113:0/3319909987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498c108370 con 0x7f498c0fe270 2024-04-12T05:28:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f49897fa700 1 -- 172.21.15.113:0/3319909987 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f497c0148b0 con 0x7f498c0fe270 2024-04-12T05:28:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.391+0000 7f49928c0700 1 -- 172.21.15.113:0/3319909987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498c1088b0 con 0x7f498c0fe270 2024-04-12T05:28:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.392+0000 7f49928c0700 1 -- 172.21.15.113:0/3319909987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498c05f570 con 0x7f498c0fe270 2024-04-12T05:28:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.394+0000 7f49897fa700 1 -- 172.21.15.113:0/3319909987 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f497c007ca0 con 0x7f498c0fe270 2024-04-12T05:28:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.394+0000 7f49897fa700 1 --2- 172.21.15.113:0/3319909987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f497406caa0 0x7f497406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:28:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.394+0000 7f49897fa700 1 -- 172.21.15.113:0/3319909987 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f497c08bb20 con 0x7f498c0fe270 2024-04-12T05:28:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.394+0000 7f498bfff700 1 --2- 172.21.15.113:0/3319909987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f497406caa0 0x7f497406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:28:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.394+0000 7f498bfff700 1 --2- 172.21.15.113:0/3319909987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f497406caa0 0x7f497406ef60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4980004f10 tx=0x7f498000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:28:46.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.398+0000 7f49897fa700 1 -- 172.21.15.113:0/3319909987 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f497c0595d0 con 0x7f498c0fe270 2024-04-12T05:28:46.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:46 smithi113 ceph-mon[103371]: from='client.34924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:46.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:46 smithi113 ceph-mon[103371]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:46.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:46 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3649605639' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:28:46.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:46 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/102821543' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:28:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:46 smithi192 ceph-mon[85304]: from='client.34924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:46 smithi192 ceph-mon[85304]: from='client.34928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:46 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3649605639' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:28:46.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:46 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/102821543' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:28:46.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.661+0000 7f49928c0700 1 -- 172.21.15.113:0/3319909987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f498c047540 con 0x7f498c0fe270 2024-04-12T05:28:46.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.661+0000 7f49897fa700 1 -- 172.21.15.113:0/3319909987 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f497c059160 con 0x7f498c0fe270 2024-04-12T05:28:46.663 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:28:46.663 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:28:46.663 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:28:46.663 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:28:46.663 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:28:46.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.663+0000 7f49928c0700 1 -- 172.21.15.113:0/3319909987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f497406caa0 msgr2=0x7f497406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:46.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.663+0000 7f49928c0700 1 --2- 172.21.15.113:0/3319909987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f497406caa0 0x7f497406ef60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4980004f10 tx=0x7f498000b040).stop 2024-04-12T05:28:46.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.663+0000 7f49928c0700 1 -- 172.21.15.113:0/3319909987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0fe270 msgr2=0x7f498c10e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:28:46.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.663+0000 7f49928c0700 1 --2- 172.21.15.113:0/3319909987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0fe270 0x7f498c10e5a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f497c002820 tx=0x7f497c009f20).stop 2024-04-12T05:28:46.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.663+0000 7f49928c0700 1 -- 172.21.15.113:0/3319909987 shutdown_connections 2024-04-12T05:28:46.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.663+0000 7f49928c0700 1 --2- 172.21.15.113:0/3319909987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f497406caa0 0x7f497406ef60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:46.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.663+0000 7f49928c0700 1 --2- 172.21.15.113:0/3319909987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f498c0fd910 0x7f498c10e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:46.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.663+0000 7f49928c0700 1 --2- 172.21.15.113:0/3319909987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f498c0fe270 0x7f498c10e5a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:28:46.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.663+0000 7f49928c0700 1 -- 172.21.15.113:0/3319909987 >> 172.21.15.113:0/3319909987 conn(0x7f498c0f3530 msgr2=0x7f498c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:28:46.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.663+0000 7f49928c0700 1 -- 172.21.15.113:0/3319909987 shutdown_connections 2024-04-12T05:28:46.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:28:46.663+0000 7f49928c0700 1 -- 172.21.15.113:0/3319909987 wait complete. 2024-04-12T05:28:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:47 smithi113 ceph-mon[103371]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:47 smithi113 ceph-mon[103371]: pgmap v498: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:47.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:47 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3319909987' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:28:47.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:47 smithi192 ceph-mon[85304]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:47 smithi192 ceph-mon[85304]: pgmap v498: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:47 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3319909987' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:28:48.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:48 smithi113 ceph-mon[103371]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:48.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:48 smithi192 ceph-mon[85304]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:28:49.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:49 smithi113 ceph-mon[103371]: pgmap v499: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:49.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:49 smithi192 ceph-mon[85304]: pgmap v499: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:51.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:51 smithi113 ceph-mon[103371]: pgmap v500: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:51 smithi192 ceph-mon[85304]: pgmap v500: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:52.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:28:52.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:28:53.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:53 smithi113 ceph-mon[103371]: pgmap v501: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:53.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:53 smithi192 ceph-mon[85304]: pgmap v501: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:55.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:55 smithi113 ceph-mon[103371]: pgmap v502: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:55.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:55 smithi192 ceph-mon[85304]: pgmap v502: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:57.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:57 smithi113 ceph-mon[103371]: pgmap v503: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:57.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:57 smithi192 ceph-mon[85304]: pgmap v503: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:59.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:28:59 smithi113 ceph-mon[103371]: pgmap v504: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:28:59.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:28:59 smithi192 ceph-mon[85304]: pgmap v504: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:01.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:01 smithi113 ceph-mon[103371]: pgmap v505: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:01.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:01 smithi192 ceph-mon[85304]: pgmap v505: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:03.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:03 smithi113 ceph-mon[103371]: pgmap v506: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:03.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:03 smithi192 ceph-mon[85304]: pgmap v506: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:05.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:05 smithi113 ceph-mon[103371]: pgmap v507: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:05.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:05 smithi192 ceph-mon[85304]: pgmap v507: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:07.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:07 smithi113 ceph-mon[103371]: pgmap v508: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:07.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:07.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:07 smithi192 ceph-mon[85304]: pgmap v508: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:07.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:09.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:09 smithi113 ceph-mon[103371]: pgmap v509: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:09.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:09 smithi192 ceph-mon[85304]: pgmap v509: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:11.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:11 smithi113 ceph-mon[103371]: pgmap v510: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:11.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:11 smithi192 ceph-mon[85304]: pgmap v510: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:13.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:13 smithi113 ceph-mon[103371]: pgmap v511: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:13.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:13 smithi192 ceph-mon[85304]: pgmap v511: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:15.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:15 smithi113 ceph-mon[103371]: pgmap v512: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:15.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:15 smithi192 ceph-mon[85304]: pgmap v512: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:16.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.773+0000 7fd443ca3700 1 -- 172.21.15.113:0/3582328705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd43c0fd150 msgr2=0x7fd43c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:16.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.773+0000 7fd443ca3700 1 --2- 172.21.15.113:0/3582328705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd43c0fd150 0x7fd43c0f6930 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fd4300047f0 tx=0x7fd430009cb0).stop 2024-04-12T05:29:16.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.773+0000 7fd443ca3700 1 -- 172.21.15.113:0/3582328705 shutdown_connections 2024-04-12T05:29:16.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.773+0000 7fd443ca3700 1 --2- 172.21.15.113:0/3582328705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd43c0fd150 0x7fd43c0f6930 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:16.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.773+0000 7fd443ca3700 1 --2- 172.21.15.113:0/3582328705 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd43c0fc7f0 0x7fd43c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:16.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.773+0000 7fd443ca3700 1 -- 172.21.15.113:0/3582328705 >> 172.21.15.113:0/3582328705 conn(0x7fd43c070590 msgr2=0x7fd43c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:16.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.773+0000 7fd443ca3700 1 -- 172.21.15.113:0/3582328705 shutdown_connections 2024-04-12T05:29:16.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.773+0000 7fd443ca3700 1 -- 172.21.15.113:0/3582328705 wait complete. 2024-04-12T05:29:16.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd443ca3700 1 Processor -- start 2024-04-12T05:29:16.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd443ca3700 1 -- start start 2024-04-12T05:29:16.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd443ca3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd43c0fc7f0 0x7fd43c1090e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:16.777 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd443ca3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd43c0fd150 0x7fd43c1095d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd443ca3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c109ac0 con 0x7fd43c0fd150 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd443ca3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c109c00 con 0x7fd43c0fc7f0 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd44123e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd43c0fd150 0x7fd43c1095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd44123e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd43c0fd150 0x7fd43c1095d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53618/0 (socket says 172.21.15.113:53618) 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd44123e700 1 -- 172.21.15.113:0/3900709068 learned_addr learned my addr 172.21.15.113:0/3900709068 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd441a3f700 1 --2- 172.21.15.113:0/3900709068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd43c0fc7f0 0x7fd43c1090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd44123e700 1 -- 172.21.15.113:0/3900709068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd43c0fc7f0 msgr2=0x7fd43c1090e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd44123e700 1 --2- 172.21.15.113:0/3900709068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd43c0fc7f0 0x7fd43c1090e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.774+0000 7fd44123e700 1 -- 172.21.15.113:0/3900709068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd430005040 con 0x7fd43c0fd150 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.775+0000 7fd44123e700 1 --2- 172.21.15.113:0/3900709068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd43c0fd150 0x7fd43c1095d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fd4300099f0 tx=0x7fd430009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.775+0000 7fd43affd700 1 -- 172.21.15.113:0/3900709068 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd43000e030 con 0x7fd43c0fd150 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.775+0000 7fd43affd700 1 -- 172.21.15.113:0/3900709068 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd430007e90 con 0x7fd43c0fd150 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.775+0000 7fd443ca3700 1 -- 172.21.15.113:0/3900709068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd43c109e50 con 0x7fd43c0fd150 2024-04-12T05:29:16.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.775+0000 7fd43affd700 1 -- 172.21.15.113:0/3900709068 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd430018070 con 0x7fd43c0fd150 2024-04-12T05:29:16.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.775+0000 7fd443ca3700 1 -- 172.21.15.113:0/3900709068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43c112440 con 0x7fd43c0fd150 2024-04-12T05:29:16.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.776+0000 7fd443ca3700 1 -- 172.21.15.113:0/3900709068 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43c05f590 con 0x7fd43c0fd150 2024-04-12T05:29:16.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.777+0000 7fd43affd700 1 -- 172.21.15.113:0/3900709068 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd4300094c0 con 0x7fd43c0fd150 2024-04-12T05:29:16.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.777+0000 7fd43affd700 1 --2- 172.21.15.113:0/3900709068 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd42406caa0 0x7fd42406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:16.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.777+0000 7fd43affd700 1 -- 172.21.15.113:0/3900709068 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd43008d430 con 0x7fd43c0fd150 2024-04-12T05:29:16.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.777+0000 7fd441a3f700 1 --2- 172.21.15.113:0/3900709068 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd42406caa0 0x7fd42406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:16.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.778+0000 7fd441a3f700 1 --2- 172.21.15.113:0/3900709068 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd42406caa0 0x7fd42406ef60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fd42c004fd0 tx=0x7fd42c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:16.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.781+0000 7fd43affd700 1 -- 172.21.15.113:0/3900709068 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd430057450 con 0x7fd43c0fd150 2024-04-12T05:29:16.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.992+0000 7fd443ca3700 1 -- 172.21.15.113:0/3900709068 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd43c059120 con 0x7fd42406caa0 2024-04-12T05:29:16.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.993+0000 7fd43affd700 1 -- 172.21.15.113:0/3900709068 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd43c059120 con 0x7fd42406caa0 2024-04-12T05:29:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.996+0000 7fd443ca3700 1 -- 172.21.15.113:0/3900709068 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd42406caa0 msgr2=0x7fd42406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.996+0000 7fd443ca3700 1 --2- 172.21.15.113:0/3900709068 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd42406caa0 0x7fd42406ef60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fd42c004fd0 tx=0x7fd42c009040).stop 2024-04-12T05:29:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.996+0000 7fd443ca3700 1 -- 172.21.15.113:0/3900709068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd43c0fd150 msgr2=0x7fd43c1095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.996+0000 7fd443ca3700 1 --2- 172.21.15.113:0/3900709068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd43c0fd150 0x7fd43c1095d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fd4300099f0 tx=0x7fd430009cb0).stop 2024-04-12T05:29:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.996+0000 7fd443ca3700 1 -- 172.21.15.113:0/3900709068 shutdown_connections 2024-04-12T05:29:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.996+0000 7fd443ca3700 1 --2- 172.21.15.113:0/3900709068 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd42406caa0 0x7fd42406ef60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:16.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.996+0000 7fd443ca3700 1 --2- 172.21.15.113:0/3900709068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd43c0fc7f0 0x7fd43c1090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.996+0000 7fd443ca3700 1 --2- 172.21.15.113:0/3900709068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd43c0fd150 0x7fd43c1095d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.996+0000 7fd443ca3700 1 -- 172.21.15.113:0/3900709068 >> 172.21.15.113:0/3900709068 conn(0x7fd43c070590 msgr2=0x7fd43c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.996+0000 7fd443ca3700 1 -- 172.21.15.113:0/3900709068 shutdown_connections 2024-04-12T05:29:16.998 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:16.996+0000 7fd443ca3700 1 -- 172.21.15.113:0/3900709068 wait complete. 2024-04-12T05:29:17.007 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:29:17.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.094+0000 7fbbf771e700 1 -- 172.21.15.113:0/517746477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf00f9210 msgr2=0x7fbbf01019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.094+0000 7fbbf771e700 1 --2- 172.21.15.113:0/517746477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf00f9210 0x7fbbf01019b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fbbe00047f0 tx=0x7fbbe0009cb0).stop 2024-04-12T05:29:17.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.094+0000 7fbbf771e700 1 -- 172.21.15.113:0/517746477 shutdown_connections 2024-04-12T05:29:17.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.094+0000 7fbbf771e700 1 --2- 172.21.15.113:0/517746477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf00f9210 0x7fbbf01019b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.094+0000 7fbbf771e700 1 --2- 172.21.15.113:0/517746477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf00f8940 0x7fbbf00f8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.094+0000 7fbbf771e700 1 -- 172.21.15.113:0/517746477 >> 172.21.15.113:0/517746477 conn(0x7fbbf0070590 msgr2=0x7fbbf00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.094+0000 7fbbf771e700 1 -- 172.21.15.113:0/517746477 shutdown_connections 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.094+0000 7fbbf771e700 1 -- 172.21.15.113:0/517746477 wait complete. 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.095+0000 7fbbf771e700 1 Processor -- start 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.095+0000 7fbbf771e700 1 -- start start 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.095+0000 7fbbf771e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf00f8940 0x7fbbf010e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.095+0000 7fbbf771e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf00f9210 0x7fbbf010e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.095+0000 7fbbf771e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf010ec10 con 0x7fbbf00f9210 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.095+0000 7fbbf771e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf0108230 con 0x7fbbf00f8940 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.095+0000 7fbbf4cb9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf00f9210 0x7fbbf010e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.095+0000 7fbbf4cb9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf00f9210 0x7fbbf010e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53626/0 (socket says 172.21.15.113:53626) 2024-04-12T05:29:17.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.095+0000 7fbbf4cb9700 1 -- 172.21.15.113:0/3845932235 learned_addr learned my addr 172.21.15.113:0/3845932235 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:17.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.095+0000 7fbbf54ba700 1 --2- 172.21.15.113:0/3845932235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf00f8940 0x7fbbf010e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:17.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.096+0000 7fbbf4cb9700 1 -- 172.21.15.113:0/3845932235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf00f8940 msgr2=0x7fbbf010e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.096+0000 7fbbf4cb9700 1 --2- 172.21.15.113:0/3845932235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf00f8940 0x7fbbf010e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.096+0000 7fbbf4cb9700 1 -- 172.21.15.113:0/3845932235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe0005040 con 0x7fbbf00f9210 2024-04-12T05:29:17.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.096+0000 7fbbf4cb9700 1 --2- 172.21.15.113:0/3845932235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf00f9210 0x7fbbf010e580 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fbbe000b120 tx=0x7fbbe0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:17.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.096+0000 7fbbe67fc700 1 -- 172.21.15.113:0/3845932235 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe000e030 con 0x7fbbf00f9210 2024-04-12T05:29:17.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.096+0000 7fbbe67fc700 1 -- 172.21.15.113:0/3845932235 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbe0007e90 con 0x7fbbf00f9210 2024-04-12T05:29:17.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.096+0000 7fbbf771e700 1 -- 172.21.15.113:0/3845932235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf0108480 con 0x7fbbf00f9210 2024-04-12T05:29:17.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.096+0000 7fbbe67fc700 1 -- 172.21.15.113:0/3845932235 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe0018070 con 0x7fbbf00f9210 2024-04-12T05:29:17.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.096+0000 7fbbf771e700 1 -- 172.21.15.113:0/3845932235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf0108960 con 0x7fbbf00f9210 2024-04-12T05:29:17.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.097+0000 7fbbf771e700 1 -- 172.21.15.113:0/3845932235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf005f590 con 0x7fbbf00f9210 2024-04-12T05:29:17.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.097+0000 7fbbe67fc700 1 -- 172.21.15.113:0/3845932235 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbbe001d420 con 0x7fbbf00f9210 2024-04-12T05:29:17.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.098+0000 7fbbe67fc700 1 --2- 172.21.15.113:0/3845932235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06caa0 0x7fbbdc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:17.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.098+0000 7fbbe67fc700 1 -- 172.21.15.113:0/3845932235 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbbe008d1b0 con 0x7fbbf00f9210 2024-04-12T05:29:17.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.098+0000 7fbbf54ba700 1 --2- 172.21.15.113:0/3845932235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06caa0 0x7fbbdc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:17.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.098+0000 7fbbf54ba700 1 --2- 172.21.15.113:0/3845932235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06caa0 0x7fbbdc06ef60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fbbec00b910 tx=0x7fbbec008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:17.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.102+0000 7fbbe67fc700 1 -- 172.21.15.113:0/3845932235 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbbe005ac60 con 0x7fbbf00f9210 2024-04-12T05:29:17.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.312+0000 7fbbf771e700 1 -- 172.21.15.113:0/3845932235 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbf0109780 con 0x7fbbdc06caa0 2024-04-12T05:29:17.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.314+0000 7fbbe67fc700 1 -- 172.21.15.113:0/3845932235 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbbf0109780 con 0x7fbbdc06caa0 2024-04-12T05:29:17.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.316+0000 7fbbf771e700 1 -- 172.21.15.113:0/3845932235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06caa0 msgr2=0x7fbbdc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.316+0000 7fbbf771e700 1 --2- 172.21.15.113:0/3845932235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06caa0 0x7fbbdc06ef60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fbbec00b910 tx=0x7fbbec008040).stop 2024-04-12T05:29:17.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.316+0000 7fbbf771e700 1 -- 172.21.15.113:0/3845932235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf00f9210 msgr2=0x7fbbf010e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.316+0000 7fbbf771e700 1 --2- 172.21.15.113:0/3845932235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf00f9210 0x7fbbf010e580 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fbbe000b120 tx=0x7fbbe0009cb0).stop 2024-04-12T05:29:17.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.316+0000 7fbbf771e700 1 -- 172.21.15.113:0/3845932235 shutdown_connections 2024-04-12T05:29:17.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.316+0000 7fbbf771e700 1 --2- 172.21.15.113:0/3845932235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06caa0 0x7fbbdc06ef60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.316+0000 7fbbf771e700 1 --2- 172.21.15.113:0/3845932235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf00f8940 0x7fbbf010e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.316+0000 7fbbf771e700 1 --2- 172.21.15.113:0/3845932235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf00f9210 0x7fbbf010e580 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.316+0000 7fbbf771e700 1 -- 172.21.15.113:0/3845932235 >> 172.21.15.113:0/3845932235 conn(0x7fbbf0070590 msgr2=0x7fbbf00fc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:17.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.316+0000 7fbbf771e700 1 -- 172.21.15.113:0/3845932235 shutdown_connections 2024-04-12T05:29:17.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.316+0000 7fbbf771e700 1 -- 172.21.15.113:0/3845932235 wait complete. 2024-04-12T05:29:17.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.413+0000 7fd3e7115700 1 -- 172.21.15.113:0/4275822115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3e00fc7f0 msgr2=0x7fd3e00fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.413+0000 7fd3e7115700 1 --2- 172.21.15.113:0/4275822115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3e00fc7f0 0x7fd3e00fcbd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fd3d40047f0 tx=0x7fd3d4009cb0).stop 2024-04-12T05:29:17.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 -- 172.21.15.113:0/4275822115 shutdown_connections 2024-04-12T05:29:17.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 --2- 172.21.15.113:0/4275822115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e00fd150 0x7fd3e00f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 --2- 172.21.15.113:0/4275822115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3e00fc7f0 0x7fd3e00fcbd0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 -- 172.21.15.113:0/4275822115 >> 172.21.15.113:0/4275822115 conn(0x7fd3e0070590 msgr2=0x7fd3e00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:17.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 -- 172.21.15.113:0/4275822115 shutdown_connections 2024-04-12T05:29:17.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 -- 172.21.15.113:0/4275822115 wait complete. 2024-04-12T05:29:17.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 Processor -- start 2024-04-12T05:29:17.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 -- start start 2024-04-12T05:29:17.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e00fc7f0 0x7fd3e010cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:17.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3e00fd150 0x7fd3e010d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e010d9f0 con 0x7fd3e00fd150 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3e7115700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e010db30 con 0x7fd3e00fc7f0 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3dffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3e00fd150 0x7fd3e010d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.414+0000 7fd3dffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3e00fd150 0x7fd3e010d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53636/0 (socket says 172.21.15.113:53636) 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.415+0000 7fd3dffff700 1 -- 172.21.15.113:0/3787169587 learned_addr learned my addr 172.21.15.113:0/3787169587 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.415+0000 7fd3dffff700 1 -- 172.21.15.113:0/3787169587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e00fc7f0 msgr2=0x7fd3e010cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.415+0000 7fd3e4eb1700 1 --2- 172.21.15.113:0/3787169587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e00fc7f0 0x7fd3e010cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.415+0000 7fd3dffff700 1 --2- 172.21.15.113:0/3787169587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e00fc7f0 0x7fd3e010cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.415+0000 7fd3dffff700 1 -- 172.21.15.113:0/3787169587 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d4005040 con 0x7fd3e00fd150 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.415+0000 7fd3dffff700 1 --2- 172.21.15.113:0/3787169587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3e00fd150 0x7fd3e010d3f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fd3cc00c790 tx=0x7fd3cc007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.415+0000 7fd3ddffb700 1 -- 172.21.15.113:0/3787169587 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3cc0106f0 con 0x7fd3e00fd150 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.415+0000 7fd3e7115700 1 -- 172.21.15.113:0/3787169587 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e01071c0 con 0x7fd3e00fd150 2024-04-12T05:29:17.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.415+0000 7fd3ddffb700 1 -- 172.21.15.113:0/3787169587 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3cc010d00 con 0x7fd3e00fd150 2024-04-12T05:29:17.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.415+0000 7fd3ddffb700 1 -- 172.21.15.113:0/3787169587 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3cc012cf0 con 0x7fd3e00fd150 2024-04-12T05:29:17.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.415+0000 7fd3e7115700 1 -- 172.21.15.113:0/3787169587 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e0107780 con 0x7fd3e00fd150 2024-04-12T05:29:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.416+0000 7fd3e7115700 1 -- 172.21.15.113:0/3787169587 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e005f590 con 0x7fd3e00fd150 2024-04-12T05:29:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.416+0000 7fd3ddffb700 1 -- 172.21.15.113:0/3787169587 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd3cc012400 con 0x7fd3e00fd150 2024-04-12T05:29:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.417+0000 7fd3ddffb700 1 --2- 172.21.15.113:0/3787169587 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd3d006caa0 0x7fd3d006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.417+0000 7fd3ddffb700 1 -- 172.21.15.113:0/3787169587 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd3cc08b720 con 0x7fd3e00fd150 2024-04-12T05:29:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.417+0000 7fd3e4eb1700 1 --2- 172.21.15.113:0/3787169587 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd3d006caa0 0x7fd3d006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:17.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.417+0000 7fd3e4eb1700 1 --2- 172.21.15.113:0/3787169587 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd3d006caa0 0x7fd3d006ef60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fd3d4000c00 tx=0x7fd3d400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:17.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.421+0000 7fd3ddffb700 1 -- 172.21.15.113:0/3787169587 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd3cc0591d0 con 0x7fd3e00fd150 2024-04-12T05:29:17.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:17 smithi113 ceph-mon[103371]: pgmap v513: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:17.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:17 smithi192 ceph-mon[85304]: pgmap v513: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:17.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.620+0000 7fd3e7115700 1 -- 172.21.15.113:0/3787169587 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3e0107a00 con 0x7fd3d006caa0 2024-04-12T05:29:17.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.629+0000 7fd3ddffb700 1 -- 172.21.15.113:0/3787169587 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd3e0107a00 con 0x7fd3d006caa0 2024-04-12T05:29:17.631 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:29:17.631 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (17m) 10m ago 28m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:29:17.631 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (14m) 10m ago 28m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:29:17.631 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (14m) 9m ago 27m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:29:17.631 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (16m) 10m ago 28m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:29:17.631 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (10m) 10m ago 23m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:29:17.631 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (10m) 10m ago 22m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:29:17.631 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (9m) 9m ago 22m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:29:17.631 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (22m) 9m ago 22m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (18m) 10m ago 29m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (18m) 9m ago 27m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (15m) 10m ago 29m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (15m) 9m ago 27m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (17m) 10m ago 28m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (17m) 9m ago 27m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (14m) 10m ago 26m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (13m) 10m ago 26m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (12m) 10m ago 25m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (12m) 9m ago 25m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (11m) 9m ago 25m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (10m) 9m ago 24m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:29:17.632 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (17m) 9m ago 27m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:29:17.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.632+0000 7fd3e7115700 1 -- 172.21.15.113:0/3787169587 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd3d006caa0 msgr2=0x7fd3d006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.632+0000 7fd3e7115700 1 --2- 172.21.15.113:0/3787169587 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd3d006caa0 0x7fd3d006ef60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fd3d4000c00 tx=0x7fd3d400b040).stop 2024-04-12T05:29:17.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.632+0000 7fd3e7115700 1 -- 172.21.15.113:0/3787169587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3e00fd150 msgr2=0x7fd3e010d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.632+0000 7fd3e7115700 1 --2- 172.21.15.113:0/3787169587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3e00fd150 0x7fd3e010d3f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fd3cc00c790 tx=0x7fd3cc007f70).stop 2024-04-12T05:29:17.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.632+0000 7fd3e7115700 1 -- 172.21.15.113:0/3787169587 shutdown_connections 2024-04-12T05:29:17.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.632+0000 7fd3e7115700 1 --2- 172.21.15.113:0/3787169587 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd3d006caa0 0x7fd3d006ef60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.632+0000 7fd3e7115700 1 --2- 172.21.15.113:0/3787169587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e00fc7f0 0x7fd3e010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.632+0000 7fd3e7115700 1 --2- 172.21.15.113:0/3787169587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3e00fd150 0x7fd3e010d3f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.632+0000 7fd3e7115700 1 -- 172.21.15.113:0/3787169587 >> 172.21.15.113:0/3787169587 conn(0x7fd3e0070590 msgr2=0x7fd3e00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:17.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.632+0000 7fd3e7115700 1 -- 172.21.15.113:0/3787169587 shutdown_connections 2024-04-12T05:29:17.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.632+0000 7fd3e7115700 1 -- 172.21.15.113:0/3787169587 wait complete. 2024-04-12T05:29:17.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.728+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2109528434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fc40fc7f0 msgr2=0x7f1fc40fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.728+0000 7f1fc9c32700 1 --2- 172.21.15.113:0/2109528434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fc40fc7f0 0x7f1fc40fcbd0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1fb40047f0 tx=0x7f1fb4009cb0).stop 2024-04-12T05:29:17.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.728+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2109528434 shutdown_connections 2024-04-12T05:29:17.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.728+0000 7f1fc9c32700 1 --2- 172.21.15.113:0/2109528434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fc40fd150 0x7f1fc40f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:29:17.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.728+0000 7f1fc9c32700 1 --2- 172.21.15.113:0/2109528434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fc40fc7f0 0x7f1fc40fcbd0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.728+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2109528434 >> 172.21.15.113:0/2109528434 conn(0x7f1fc40f23b0 msgr2=0x7f1fc40f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:17.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.728+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2109528434 shutdown_connections 2024-04-12T05:29:17.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.728+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2109528434 wait complete. 2024-04-12T05:29:17.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.729+0000 7f1fc9c32700 1 Processor -- start 2024-04-12T05:29:17.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.729+0000 7f1fc9c32700 1 -- start start 2024-04-12T05:29:17.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.729+0000 7f1fc9c32700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fc40fc7f0 0x7f1fc410e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:17.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.729+0000 7f1fc9c32700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fc40fd150 0x7f1fc410e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:17.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.729+0000 7f1fc9c32700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc410ec10 con 0x7f1fc40fd150 2024-04-12T05:29:17.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.729+0000 7f1fc9c32700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc4108230 con 0x7f1fc40fc7f0 2024-04-12T05:29:17.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.729+0000 7f1fc2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fc40fd150 0x7f1fc410e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:17.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.729+0000 7f1fc2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fc40fd150 0x7f1fc410e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53652/0 (socket says 172.21.15.113:53652) 2024-04-12T05:29:17.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.729+0000 7f1fc2ffd700 1 -- 172.21.15.113:0/2500014433 learned_addr learned my addr 172.21.15.113:0/2500014433 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:17.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.730+0000 7f1fc37fe700 1 --2- 172.21.15.113:0/2500014433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fc40fc7f0 0x7f1fc410e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:17.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.730+0000 7f1fc2ffd700 1 -- 172.21.15.113:0/2500014433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fc40fc7f0 msgr2=0x7f1fc410e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:17.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.730+0000 7f1fc2ffd700 1 --2- 172.21.15.113:0/2500014433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fc40fc7f0 0x7f1fc410e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:17.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.730+0000 7f1fc2ffd700 1 -- 172.21.15.113:0/2500014433 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4005040 con 0x7f1fc40fd150 2024-04-12T05:29:17.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.730+0000 7f1fc2ffd700 1 --2- 172.21.15.113:0/2500014433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fc40fd150 0x7f1fc410e580 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f1fb800b480 tx=0x7f1fb80090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:17.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.730+0000 7f1fc0ff9700 1 -- 172.21.15.113:0/2500014433 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fb8002500 con 0x7f1fc40fd150 2024-04-12T05:29:17.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.730+0000 7f1fc0ff9700 1 -- 172.21.15.113:0/2500014433 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fb8002b10 con 0x7f1fc40fd150 2024-04-12T05:29:17.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.730+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2500014433 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc41084e0 con 0x7f1fc40fd150 2024-04-12T05:29:17.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.730+0000 7f1fc0ff9700 1 -- 172.21.15.113:0/2500014433 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fb8012c60 con 0x7f1fc40fd150 2024-04-12T05:29:17.733 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.730+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2500014433 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc4108a20 con 0x7f1fc40fd150 2024-04-12T05:29:17.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.731+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2500014433 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc4060370 con 0x7f1fc40fd150 2024-04-12T05:29:17.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.731+0000 7f1fc0ff9700 1 -- 172.21.15.113:0/2500014433 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1fb8002660 con 0x7f1fc40fd150 2024-04-12T05:29:17.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.732+0000 7f1fc0ff9700 1 --2- 172.21.15.113:0/2500014433 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fac06caa0 0x7f1fac06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:17.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.732+0000 7f1fc0ff9700 1 -- 172.21.15.113:0/2500014433 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1fb808be50 con 0x7f1fc40fd150 2024-04-12T05:29:17.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.732+0000 7f1fc37fe700 1 --2- 172.21.15.113:0/2500014433 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fac06caa0 0x7f1fac06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:17.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.732+0000 7f1fc37fe700 1 --2- 172.21.15.113:0/2500014433 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fac06caa0 0x7f1fac06ef60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f1fb4007cf0 tx=0x7f1fb400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:17.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:17.736+0000 7f1fc0ff9700 1 -- 172.21.15.113:0/2500014433 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1fb8059900 con 0x7f1fc40fd150 2024-04-12T05:29:18.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.003+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2500014433 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1fc41090c0 con 0x7f1fc40fd150 2024-04-12T05:29:18.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.003+0000 7f1fc0ff9700 1 -- 172.21.15.113:0/2500014433 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f1fb8059490 con 0x7f1fc40fd150 2024-04-12T05:29:18.005 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:29:18.005 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:29:18.005 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:29:18.005 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:29:18.005 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:29:18.005 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:29:18.005 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:29:18.005 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:29:18.006 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:29:18.006 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:29:18.006 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:29:18.006 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:29:18.006 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:29:18.006 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:29:18.006 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:29:18.006 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:29:18.006 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:29:18.006 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:29:18.006 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:29:18.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.005+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2500014433 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fac06caa0 msgr2=0x7f1fac06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.005+0000 7f1fc9c32700 1 --2- 172.21.15.113:0/2500014433 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fac06caa0 0x7f1fac06ef60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f1fb4007cf0 tx=0x7f1fb400b040).stop 2024-04-12T05:29:18.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.005+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2500014433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fc40fd150 msgr2=0x7f1fc410e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.005+0000 7f1fc9c32700 1 --2- 172.21.15.113:0/2500014433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fc40fd150 0x7f1fc410e580 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f1fb800b480 tx=0x7f1fb80090f0).stop 2024-04-12T05:29:18.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.006+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2500014433 shutdown_connections 2024-04-12T05:29:18.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.006+0000 7f1fc9c32700 1 --2- 172.21.15.113:0/2500014433 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fac06caa0 0x7f1fac06ef60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.006+0000 7f1fc9c32700 1 --2- 172.21.15.113:0/2500014433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fc40fc7f0 0x7f1fc410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.006+0000 7f1fc9c32700 1 --2- 172.21.15.113:0/2500014433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fc40fd150 0x7f1fc410e580 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.006+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2500014433 >> 172.21.15.113:0/2500014433 conn(0x7f1fc40f23b0 msgr2=0x7f1fc4060b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:18.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.006+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2500014433 shutdown_connections 2024-04-12T05:29:18.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.006+0000 7f1fc9c32700 1 -- 172.21.15.113:0/2500014433 wait complete. 2024-04-12T05:29:18.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.103+0000 7f1fe3645700 1 -- 172.21.15.113:0/1565620980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fdc0fc7f0 msgr2=0x7f1fdc0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.104+0000 7f1fe3645700 1 --2- 172.21.15.113:0/1565620980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fdc0fc7f0 0x7f1fdc0fcbd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f1fd80047f0 tx=0x7f1fd8009cb0).stop 2024-04-12T05:29:18.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.104+0000 7f1fe3645700 1 -- 172.21.15.113:0/1565620980 shutdown_connections 2024-04-12T05:29:18.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.104+0000 7f1fe3645700 1 --2- 172.21.15.113:0/1565620980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fdc0fd150 0x7f1fdc0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.104+0000 7f1fe3645700 1 --2- 172.21.15.113:0/1565620980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fdc0fc7f0 0x7f1fdc0fcbd0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.104+0000 7f1fe3645700 1 -- 172.21.15.113:0/1565620980 >> 172.21.15.113:0/1565620980 conn(0x7f1fdc070590 msgr2=0x7f1fdc0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:18.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.104+0000 7f1fe3645700 1 -- 172.21.15.113:0/1565620980 shutdown_connections 2024-04-12T05:29:18.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.104+0000 7f1fe3645700 1 -- 172.21.15.113:0/1565620980 wait complete. 2024-04-12T05:29:18.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.104+0000 7f1fe3645700 1 Processor -- start 2024-04-12T05:29:18.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.104+0000 7f1fe3645700 1 -- start start 2024-04-12T05:29:18.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.104+0000 7f1fe3645700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fdc0fc7f0 0x7f1fdc10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:18.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe3645700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fdc0fd150 0x7f1fdc10e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:18.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe3645700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc10eb20 con 0x7f1fdc0fd150 2024-04-12T05:29:18.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe3645700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc10ec60 con 0x7f1fdc0fc7f0 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe0be0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fdc0fd150 0x7f1fdc10e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe0be0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fdc0fd150 0x7f1fdc10e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53662/0 (socket says 172.21.15.113:53662) 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe0be0700 1 -- 172.21.15.113:0/1110234078 learned_addr learned my addr 172.21.15.113:0/1110234078 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe13e1700 1 --2- 172.21.15.113:0/1110234078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fdc0fc7f0 0x7f1fdc10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe0be0700 1 -- 172.21.15.113:0/1110234078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fdc0fc7f0 msgr2=0x7f1fdc10e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe0be0700 1 --2- 172.21.15.113:0/1110234078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fdc0fc7f0 0x7f1fdc10e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe0be0700 1 -- 172.21.15.113:0/1110234078 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd8005040 con 0x7f1fdc0fd150 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe0be0700 1 --2- 172.21.15.113:0/1110234078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fdc0fd150 0x7f1fdc10e520 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f1fcc002820 tx=0x7f1fcc009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fd27fc700 1 -- 172.21.15.113:0/1110234078 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fcc007b40 con 0x7f1fdc0fd150 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe3645700 1 -- 172.21.15.113:0/1110234078 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fdc1082f0 con 0x7f1fdc0fd150 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fd27fc700 1 -- 172.21.15.113:0/1110234078 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fcc00f040 con 0x7f1fdc0fd150 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fd27fc700 1 -- 172.21.15.113:0/1110234078 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fcc0148b0 con 0x7f1fdc0fd150 2024-04-12T05:29:18.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.105+0000 7f1fe3645700 1 -- 172.21.15.113:0/1110234078 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fdc1088b0 con 0x7f1fdc0fd150 2024-04-12T05:29:18.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.106+0000 7f1fe3645700 1 -- 172.21.15.113:0/1110234078 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fdc05f590 con 0x7f1fdc0fd150 2024-04-12T05:29:18.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.107+0000 7f1fd27fc700 1 -- 172.21.15.113:0/1110234078 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1fcc009110 con 0x7f1fdc0fd150 2024-04-12T05:29:18.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.107+0000 7f1fd27fc700 1 --2- 172.21.15.113:0/1110234078 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fc806cab0 0x7f1fc806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:18.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.107+0000 7f1fd27fc700 1 -- 172.21.15.113:0/1110234078 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1fcc08b910 con 0x7f1fdc0fd150 2024-04-12T05:29:18.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.107+0000 7f1fe13e1700 1 --2- 172.21.15.113:0/1110234078 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fc806cab0 0x7f1fc806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:18.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.107+0000 7f1fe13e1700 1 --2- 172.21.15.113:0/1110234078 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fc806cab0 0x7f1fc806ef70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f1fd8004f10 tx=0x7f1fd8009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:18.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.111+0000 7f1fd27fc700 1 -- 172.21.15.113:0/1110234078 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1fcc0593c0 con 0x7f1fdc0fd150 2024-04-12T05:29:18.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.343+0000 7f1fe3645700 1 -- 172.21.15.113:0/1110234078 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1fdc047540 con 0x7f1fdc0fd150 2024-04-12T05:29:18.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.344+0000 7f1fd27fc700 1 -- 172.21.15.113:0/1110234078 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f1fcc058f50 con 0x7f1fdc0fd150 2024-04-12T05:29:18.345 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:29:18.345 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:29:18.345 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:29:18.345 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:29:18.345 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:29:18.346 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:29:18.347 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:29:18.348 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:29:18.348 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:29:18.348 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:29:18.348 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:29:18.348 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:29:18.348 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:29:18.348 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:29:18.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.346+0000 7f1fe3645700 1 -- 172.21.15.113:0/1110234078 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fc806cab0 msgr2=0x7f1fc806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.346+0000 7f1fe3645700 1 --2- 172.21.15.113:0/1110234078 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fc806cab0 0x7f1fc806ef70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f1fd8004f10 tx=0x7f1fd8009cb0).stop 2024-04-12T05:29:18.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.346+0000 7f1fe3645700 1 -- 172.21.15.113:0/1110234078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fdc0fd150 msgr2=0x7f1fdc10e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.346+0000 7f1fe3645700 1 --2- 172.21.15.113:0/1110234078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fdc0fd150 0x7f1fdc10e520 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f1fcc002820 tx=0x7f1fcc009f20).stop 2024-04-12T05:29:18.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.346+0000 7f1fe3645700 1 -- 172.21.15.113:0/1110234078 shutdown_connections 2024-04-12T05:29:18.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.346+0000 7f1fe3645700 1 --2- 172.21.15.113:0/1110234078 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1fc806cab0 0x7f1fc806ef70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.346+0000 7f1fe3645700 1 --2- 172.21.15.113:0/1110234078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1fdc0fc7f0 0x7f1fdc10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.346+0000 7f1fe3645700 1 --2- 172.21.15.113:0/1110234078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fdc0fd150 0x7f1fdc10e520 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.346+0000 7f1fe3645700 1 -- 172.21.15.113:0/1110234078 >> 172.21.15.113:0/1110234078 conn(0x7f1fdc070590 msgr2=0x7f1fdc060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:18.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.346+0000 7f1fe3645700 1 -- 172.21.15.113:0/1110234078 shutdown_connections 2024-04-12T05:29:18.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.346+0000 7f1fe3645700 1 -- 172.21.15.113:0/1110234078 wait complete. 2024-04-12T05:29:18.350 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:29:18.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:18 smithi113 ceph-mon[103371]: from='client.34952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:18.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:18 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2500014433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:29:18.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.446+0000 7f1c9733f700 1 -- 172.21.15.113:0/3602652797 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c900fd140 msgr2=0x7f1c900f7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.446+0000 7f1c9733f700 1 --2- 172.21.15.113:0/3602652797 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c900fd140 0x7f1c900f7090 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f1c8c0047f0 tx=0x7f1c8c009cb0).stop 2024-04-12T05:29:18.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.446+0000 7f1c9733f700 1 -- 172.21.15.113:0/3602652797 shutdown_connections 2024-04-12T05:29:18.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.446+0000 7f1c9733f700 1 --2- 172.21.15.113:0/3602652797 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c900fd140 0x7f1c900f7090 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.446+0000 7f1c9733f700 1 --2- 172.21.15.113:0/3602652797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c900fc870 0x7f1c900fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:29:18.447 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.446+0000 7f1c9733f700 1 -- 172.21.15.113:0/3602652797 >> 172.21.15.113:0/3602652797 conn(0x7f1c9006ce40 msgr2=0x7f1c9006d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:18.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.446+0000 7f1c9733f700 1 -- 172.21.15.113:0/3602652797 shutdown_connections 2024-04-12T05:29:18.448 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.446+0000 7f1c9733f700 1 -- 172.21.15.113:0/3602652797 wait complete. 2024-04-12T05:29:18.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:18 smithi192 ceph-mon[85304]: from='client.34952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:18.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:18 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2500014433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:29:18.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.446+0000 7f1c9733f700 1 Processor -- start 2024-04-12T05:29:18.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c9733f700 1 -- start start 2024-04-12T05:29:18.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c9733f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c900fc870 0x7f1c901102f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:18.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c9733f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c900fd140 0x7f1c901107e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:18.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c9733f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c90110e70 con 0x7f1c900fd140 2024-04-12T05:29:18.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c9733f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9010a490 con 0x7f1c900fc870 2024-04-12T05:29:18.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c948da700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c900fd140 0x7f1c901107e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:18.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c948da700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c900fd140 0x7f1c901107e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53678/0 (socket says 172.21.15.113:53678) 2024-04-12T05:29:18.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c948da700 1 -- 172.21.15.113:0/835757819 learned_addr learned my addr 172.21.15.113:0/835757819 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:18.450 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c948da700 1 -- 172.21.15.113:0/835757819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c900fc870 msgr2=0x7f1c901102f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c948da700 1 --2- 172.21.15.113:0/835757819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c900fc870 0x7f1c901102f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:29:18.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c948da700 1 -- 172.21.15.113:0/835757819 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c005040 con 0x7f1c900fd140 2024-04-12T05:29:18.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c948da700 1 --2- 172.21.15.113:0/835757819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c900fd140 0x7f1c901107e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f1c8c000c00 tx=0x7f1c8c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:18.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c867fc700 1 -- 172.21.15.113:0/835757819 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c8c00e030 con 0x7f1c900fd140 2024-04-12T05:29:18.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c867fc700 1 -- 172.21.15.113:0/835757819 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c8c007e90 con 0x7f1c900fd140 2024-04-12T05:29:18.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c9733f700 1 -- 172.21.15.113:0/835757819 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c9010a6e0 con 0x7f1c900fd140 2024-04-12T05:29:18.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.447+0000 7f1c867fc700 1 -- 172.21.15.113:0/835757819 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c8c018070 con 0x7f1c900fd140 2024-04-12T05:29:18.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.448+0000 7f1c9733f700 1 -- 172.21.15.113:0/835757819 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9010abc0 con 0x7f1c900fd140 2024-04-12T05:29:18.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.448+0000 7f1c9733f700 1 -- 172.21.15.113:0/835757819 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c90060670 con 0x7f1c900fd140 2024-04-12T05:29:18.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.449+0000 7f1c867fc700 1 -- 172.21.15.113:0/835757819 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1c8c01d420 con 0x7f1c900fd140 2024-04-12T05:29:18.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.449+0000 7f1c867fc700 1 --2- 172.21.15.113:0/835757819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c7c06cab0 0x7f1c7c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:18.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.449+0000 7f1c867fc700 1 -- 172.21.15.113:0/835757819 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1c8c08d280 con 0x7f1c900fd140 2024-04-12T05:29:18.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.449+0000 7f1c950db700 1 --2- 172.21.15.113:0/835757819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c7c06cab0 0x7f1c7c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:18.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.450+0000 7f1c950db700 1 --2- 172.21.15.113:0/835757819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c7c06cab0 0x7f1c7c06ef70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f1c80009700 tx=0x7f1c80006c30).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:18.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.454+0000 7f1c867fc700 1 -- 172.21.15.113:0/835757819 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1c8c05ad30 con 0x7f1c900fd140 2024-04-12T05:29:18.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.659+0000 7f1c9733f700 1 -- 172.21.15.113:0/835757819 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c9010ba80 con 0x7f1c7c06cab0 2024-04-12T05:29:18.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.660+0000 7f1c867fc700 1 -- 172.21.15.113:0/835757819 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1c9010ba80 con 0x7f1c7c06cab0 2024-04-12T05:29:18.661 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:29:18.661 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:29:18.662 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:29:18.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.662+0000 7f1c9733f700 1 -- 172.21.15.113:0/835757819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c7c06cab0 msgr2=0x7f1c7c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.662+0000 7f1c9733f700 1 --2- 172.21.15.113:0/835757819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c7c06cab0 0x7f1c7c06ef70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f1c80009700 tx=0x7f1c80006c30).stop 2024-04-12T05:29:18.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.662+0000 7f1c9733f700 1 -- 172.21.15.113:0/835757819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c900fd140 msgr2=0x7f1c901107e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.662+0000 7f1c9733f700 1 --2- 172.21.15.113:0/835757819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c900fd140 0x7f1c901107e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f1c8c000c00 tx=0x7f1c8c009cb0).stop 2024-04-12T05:29:18.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.662+0000 7f1c9733f700 1 -- 172.21.15.113:0/835757819 shutdown_connections 2024-04-12T05:29:18.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.662+0000 7f1c9733f700 1 --2- 172.21.15.113:0/835757819 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c7c06cab0 0x7f1c7c06ef70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.662+0000 7f1c9733f700 1 --2- 172.21.15.113:0/835757819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c900fc870 0x7f1c901102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:29:18.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.662+0000 7f1c9733f700 1 --2- 172.21.15.113:0/835757819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c900fd140 0x7f1c901107e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.662+0000 7f1c9733f700 1 -- 172.21.15.113:0/835757819 >> 172.21.15.113:0/835757819 conn(0x7f1c9006ce40 msgr2=0x7f1c900f5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:18.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.662+0000 7f1c9733f700 1 -- 172.21.15.113:0/835757819 shutdown_connections 2024-04-12T05:29:18.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.662+0000 7f1c9733f700 1 -- 172.21.15.113:0/835757819 wait complete. 2024-04-12T05:29:18.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.760+0000 7fa02eec6700 1 -- 172.21.15.113:0/2424737457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0280fe250 msgr2=0x7fa0280f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.760+0000 7fa02eec6700 1 --2- 172.21.15.113:0/2424737457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0280fe250 0x7fa0280f7a30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fa0140047f0 tx=0x7fa014009cb0).stop 2024-04-12T05:29:18.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.760+0000 7fa02eec6700 1 -- 172.21.15.113:0/2424737457 shutdown_connections 2024-04-12T05:29:18.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.760+0000 7fa02eec6700 1 --2- 172.21.15.113:0/2424737457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0280fe250 0x7fa0280f7a30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.760+0000 7fa02eec6700 1 --2- 172.21.15.113:0/2424737457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0280fd8f0 0x7fa0280fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.760+0000 7fa02eec6700 1 -- 172.21.15.113:0/2424737457 >> 172.21.15.113:0/2424737457 conn(0x7fa0280f34b0 msgr2=0x7fa0280f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.760+0000 7fa02eec6700 1 -- 172.21.15.113:0/2424737457 shutdown_connections 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.760+0000 7fa02eec6700 1 -- 172.21.15.113:0/2424737457 wait complete. 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02eec6700 1 Processor -- start 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02eec6700 1 -- start start 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02eec6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0280fd8f0 0x7fa028110220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02eec6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0280fe250 0x7fa028110710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02eec6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa028110d10 con 0x7fa0280fd8f0 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02eec6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa028110e50 con 0x7fa0280fe250 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02cc62700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0280fd8f0 0x7fa028110220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02cc62700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0280fd8f0 0x7fa028110220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53694/0 (socket says 172.21.15.113:53694) 2024-04-12T05:29:18.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02cc62700 1 -- 172.21.15.113:0/2412036420 learned_addr learned my addr 172.21.15.113:0/2412036420 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:18.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa027fff700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0280fe250 0x7fa028110710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:18.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02cc62700 1 -- 172.21.15.113:0/2412036420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0280fe250 msgr2=0x7fa028110710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:18.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02cc62700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0280fe250 0x7fa028110710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:18.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa02cc62700 1 -- 172.21.15.113:0/2412036420 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa014005040 con 0x7fa0280fd8f0 2024-04-12T05:29:18.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.761+0000 7fa027fff700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0280fe250 0x7fa028110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:29:18.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.762+0000 7fa02cc62700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0280fd8f0 0x7fa028110220 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa01c00c790 tx=0x7fa01c00cb50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:18.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.762+0000 7fa025ffb700 1 -- 172.21.15.113:0/2412036420 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa01c0106f0 con 0x7fa0280fd8f0 2024-04-12T05:29:18.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.762+0000 7fa025ffb700 1 -- 172.21.15.113:0/2412036420 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa01c010d00 con 0x7fa0280fd8f0 2024-04-12T05:29:18.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.762+0000 7fa025ffb700 1 -- 172.21.15.113:0/2412036420 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa01c012e50 con 0x7fa0280fd8f0 2024-04-12T05:29:18.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.762+0000 7fa02eec6700 1 -- 172.21.15.113:0/2412036420 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02810a4e0 con 0x7fa0280fd8f0 2024-04-12T05:29:18.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.762+0000 7fa02eec6700 1 -- 172.21.15.113:0/2412036420 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02810aa20 con 0x7fa0280fd8f0 2024-04-12T05:29:18.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.763+0000 7fa02eec6700 1 -- 172.21.15.113:0/2412036420 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa02805f590 con 0x7fa0280fd8f0 2024-04-12T05:29:18.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.763+0000 7fa025ffb700 1 -- 172.21.15.113:0/2412036420 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa01c010850 con 0x7fa0280fd8f0 2024-04-12T05:29:18.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.764+0000 7fa025ffb700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa01806caa0 0x7fa01806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:18.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.764+0000 7fa025ffb700 1 -- 172.21.15.113:0/2412036420 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa01c08b700 con 0x7fa0280fd8f0 2024-04-12T05:29:18.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.764+0000 7fa027fff700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa01806caa0 0x7fa01806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:18.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.764+0000 7fa027fff700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa01806caa0 0x7fa01806ef60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa0140043d0 tx=0x7fa01400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:18.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:18.768+0000 7fa025ffb700 1 -- 172.21.15.113:0/2412036420 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa01c0591b0 con 0x7fa0280fd8f0 2024-04-12T05:29:19.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.034+0000 7fa02eec6700 1 -- 172.21.15.113:0/2412036420 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa028047540 con 0x7fa0280fd8f0 2024-04-12T05:29:19.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.034+0000 7fa025ffb700 1 -- 172.21.15.113:0/2412036420 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa01c058d40 con 0x7fa0280fd8f0 2024-04-12T05:29:19.035 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:29:19.036 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:29:19.036 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:29:19.036 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:29:19.036 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:29:19.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.036+0000 7fa02eec6700 1 -- 172.21.15.113:0/2412036420 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa01806caa0 msgr2=0x7fa01806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:19.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.036+0000 7fa02eec6700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa01806caa0 0x7fa01806ef60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa0140043d0 tx=0x7fa01400b040).stop 2024-04-12T05:29:19.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.036+0000 7fa02eec6700 1 -- 172.21.15.113:0/2412036420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0280fd8f0 msgr2=0x7fa028110220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:19.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.036+0000 7fa02eec6700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0280fd8f0 0x7fa028110220 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa01c00c790 tx=0x7fa01c00cb50).stop 2024-04-12T05:29:19.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.036+0000 7fa02eec6700 1 -- 172.21.15.113:0/2412036420 shutdown_connections 2024-04-12T05:29:19.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.036+0000 7fa02eec6700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa01806caa0 0x7fa01806ef60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:19.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.036+0000 7fa02eec6700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0280fd8f0 0x7fa028110220 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:19.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.036+0000 7fa02eec6700 1 --2- 172.21.15.113:0/2412036420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0280fe250 0x7fa028110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:19.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.036+0000 7fa02eec6700 1 -- 172.21.15.113:0/2412036420 >> 172.21.15.113:0/2412036420 conn(0x7fa0280f34b0 msgr2=0x7fa02806b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:19.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.036+0000 7fa02eec6700 1 -- 172.21.15.113:0/2412036420 shutdown_connections 2024-04-12T05:29:19.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:19.036+0000 7fa02eec6700 1 -- 172.21.15.113:0/2412036420 wait complete. 2024-04-12T05:29:19.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:19 smithi113 ceph-mon[103371]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:19.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:19 smithi113 ceph-mon[103371]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:19.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:19 smithi113 ceph-mon[103371]: pgmap v514: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:19.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:19 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1110234078' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:29:19.424 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:19 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2412036420' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:29:19.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:19 smithi192 ceph-mon[85304]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:19.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:19 smithi192 ceph-mon[85304]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:19.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:19 smithi192 ceph-mon[85304]: pgmap v514: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:19.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:19 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1110234078' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:29:19.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:19 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2412036420' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:29:20.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:20 smithi113 ceph-mon[103371]: from='client.34972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:20.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:20 smithi192 ceph-mon[85304]: from='client.34972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:21.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:21 smithi113 ceph-mon[103371]: pgmap v515: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:21.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:21 smithi192 ceph-mon[85304]: pgmap v515: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:22.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:22.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:23.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:23 smithi113 ceph-mon[103371]: pgmap v516: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:23.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:23 smithi192 ceph-mon[85304]: pgmap v516: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:25.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:25 smithi113 ceph-mon[103371]: pgmap v517: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:25.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:25 smithi192 ceph-mon[85304]: pgmap v517: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:27.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:27 smithi113 ceph-mon[103371]: pgmap v518: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:27.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:27 smithi192 ceph-mon[85304]: pgmap v518: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:29.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:29 smithi113 ceph-mon[103371]: pgmap v519: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:29.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:29 smithi192 ceph-mon[85304]: pgmap v519: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:31.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:31 smithi113 ceph-mon[103371]: pgmap v520: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:31.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:31 smithi192 ceph-mon[85304]: pgmap v520: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:33.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:33 smithi113 ceph-mon[103371]: pgmap v521: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:33.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:33 smithi192 ceph-mon[85304]: pgmap v521: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:35.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:35 smithi113 ceph-mon[103371]: pgmap v522: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:35.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:35 smithi192 ceph-mon[85304]: pgmap v522: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:37.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:37 smithi113 ceph-mon[103371]: pgmap v523: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:37.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:37.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:37 smithi192 ceph-mon[85304]: pgmap v523: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:37.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:39.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:39 smithi113 ceph-mon[103371]: pgmap v524: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:39.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:39 smithi192 ceph-mon[85304]: pgmap v524: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:41.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:41 smithi113 ceph-mon[103371]: pgmap v525: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:41.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:41 smithi192 ceph-mon[85304]: pgmap v525: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:43.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:43 smithi113 ceph-mon[103371]: pgmap v526: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:43.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:43 smithi192 ceph-mon[85304]: pgmap v526: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:45.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:45 smithi113 ceph-mon[103371]: pgmap v527: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:45.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:45 smithi192 ceph-mon[85304]: pgmap v527: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:47.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:47 smithi192 ceph-mon[85304]: pgmap v528: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:47.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:47 smithi113 ceph-mon[103371]: pgmap v528: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:48.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:48 smithi113 ceph-mon[103371]: pgmap v529: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:48.697 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:48 smithi192 ceph-mon[85304]: pgmap v529: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:49.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 -- 172.21.15.113:0/1303126140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c340fe270 msgr2=0x7f1c340f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 --2- 172.21.15.113:0/1303126140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c340fe270 0x7f1c340f7a50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f1c240047f0 tx=0x7f1c24009cb0).stop 2024-04-12T05:29:49.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 -- 172.21.15.113:0/1303126140 shutdown_connections 2024-04-12T05:29:49.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 --2- 172.21.15.113:0/1303126140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c340fe270 0x7f1c340f7a50 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 --2- 172.21.15.113:0/1303126140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c340fd910 0x7f1c340fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 -- 172.21.15.113:0/1303126140 >> 172.21.15.113:0/1303126140 conn(0x7f1c340f3530 msgr2=0x7f1c340f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:49.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 -- 172.21.15.113:0/1303126140 shutdown_connections 2024-04-12T05:29:49.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 -- 172.21.15.113:0/1303126140 wait complete. 2024-04-12T05:29:49.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 Processor -- start 2024-04-12T05:29:49.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 -- start start 2024-04-12T05:29:49.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c340fd910 0x7f1c3406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:49.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c340fe270 0x7f1c3406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:49.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3406bab0 con 0x7f1c340fe270 2024-04-12T05:29:49.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.146+0000 7f1c38b1a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3406bbf0 con 0x7f1c340fd910 2024-04-12T05:29:49.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c31d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c340fe270 0x7f1c3406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:49.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c31d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c340fe270 0x7f1c3406b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58778/0 (socket says 172.21.15.113:58778) 2024-04-12T05:29:49.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c31d9b700 1 -- 172.21.15.113:0/112959342 learned_addr learned my addr 172.21.15.113:0/112959342 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:49.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c3259c700 1 --2- 172.21.15.113:0/112959342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c340fd910 0x7f1c3406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:49.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c31d9b700 1 -- 172.21.15.113:0/112959342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c340fd910 msgr2=0x7f1c3406b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c31d9b700 1 --2- 172.21.15.113:0/112959342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c340fd910 0x7f1c3406b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c31d9b700 1 -- 172.21.15.113:0/112959342 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c24005040 con 0x7f1c340fe270 2024-04-12T05:29:49.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c31d9b700 1 --2- 172.21.15.113:0/112959342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c340fe270 0x7f1c3406b5c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f1c24000c00 tx=0x7f1c24009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:49.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c2b7fe700 1 -- 172.21.15.113:0/112959342 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c2400e030 con 0x7f1c340fe270 2024-04-12T05:29:49.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c2b7fe700 1 -- 172.21.15.113:0/112959342 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c24007e90 con 0x7f1c340fe270 2024-04-12T05:29:49.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c38b1a700 1 -- 172.21.15.113:0/112959342 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3406be40 con 0x7f1c340fe270 2024-04-12T05:29:49.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c2b7fe700 1 -- 172.21.15.113:0/112959342 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c24018070 con 0x7f1c340fe270 2024-04-12T05:29:49.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.148+0000 7f1c38b1a700 1 -- 172.21.15.113:0/112959342 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c341145d0 con 0x7f1c340fe270 2024-04-12T05:29:49.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.149+0000 7f1c38b1a700 1 -- 172.21.15.113:0/112959342 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3405f570 con 0x7f1c340fe270 2024-04-12T05:29:49.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.150+0000 7f1c2b7fe700 1 -- 172.21.15.113:0/112959342 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1c2401d420 con 0x7f1c340fe270 2024-04-12T05:29:49.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.150+0000 7f1c2b7fe700 1 --2- 172.21.15.113:0/112959342 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c2006cab0 0x7f1c2006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:49.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.150+0000 7f1c2b7fe700 1 -- 172.21.15.113:0/112959342 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1c2408cfe0 con 0x7f1c340fe270 2024-04-12T05:29:49.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.150+0000 7f1c3259c700 1 --2- 172.21.15.113:0/112959342 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c2006cab0 0x7f1c2006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:49.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.151+0000 7f1c3259c700 1 --2- 172.21.15.113:0/112959342 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c2006cab0 0x7f1c2006ef70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f1c1c004de0 tx=0x7f1c1c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:49.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.154+0000 7f1c2b7fe700 1 -- 172.21.15.113:0/112959342 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1c24057000 con 0x7f1c340fe270 2024-04-12T05:29:49.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.361+0000 7f1c38b1a700 1 -- 172.21.15.113:0/112959342 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c34059120 con 0x7f1c2006cab0 2024-04-12T05:29:49.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.363+0000 7f1c2b7fe700 1 -- 172.21.15.113:0/112959342 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1c34059120 con 0x7f1c2006cab0 2024-04-12T05:29:49.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.365+0000 7f1c38b1a700 1 -- 172.21.15.113:0/112959342 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c2006cab0 msgr2=0x7f1c2006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.365+0000 7f1c38b1a700 1 --2- 172.21.15.113:0/112959342 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c2006cab0 0x7f1c2006ef70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f1c1c004de0 tx=0x7f1c1c00b040).stop 2024-04-12T05:29:49.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.365+0000 7f1c38b1a700 1 -- 172.21.15.113:0/112959342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c340fe270 msgr2=0x7f1c3406b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.365+0000 7f1c38b1a700 1 --2- 172.21.15.113:0/112959342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c340fe270 0x7f1c3406b5c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f1c24000c00 tx=0x7f1c24009cb0).stop 2024-04-12T05:29:49.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.365+0000 7f1c38b1a700 1 -- 172.21.15.113:0/112959342 shutdown_connections 2024-04-12T05:29:49.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.365+0000 7f1c38b1a700 1 --2- 172.21.15.113:0/112959342 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1c2006cab0 0x7f1c2006ef70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.365+0000 7f1c38b1a700 1 --2- 172.21.15.113:0/112959342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c340fd910 0x7f1c3406b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.365+0000 7f1c38b1a700 1 --2- 172.21.15.113:0/112959342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c340fe270 0x7f1c3406b5c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.365+0000 7f1c38b1a700 1 -- 172.21.15.113:0/112959342 >> 172.21.15.113:0/112959342 conn(0x7f1c340f3530 msgr2=0x7f1c340fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:49.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.365+0000 7f1c38b1a700 1 -- 172.21.15.113:0/112959342 shutdown_connections 2024-04-12T05:29:49.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.365+0000 7f1c38b1a700 1 -- 172.21.15.113:0/112959342 wait complete. 2024-04-12T05:29:49.376 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:29:49.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.463+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2674747045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a680fe250 msgr2=0x7f9a680f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.463+0000 7f9a6dac9700 1 --2- 172.21.15.113:0/2674747045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a680fe250 0x7f9a680f7a30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f9a5c0047f0 tx=0x7f9a5c009cb0).stop 2024-04-12T05:29:49.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.463+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2674747045 shutdown_connections 2024-04-12T05:29:49.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.463+0000 7f9a6dac9700 1 --2- 172.21.15.113:0/2674747045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a680fe250 0x7f9a680f7a30 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.463+0000 7f9a6dac9700 1 --2- 172.21.15.113:0/2674747045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a680fd8f0 0x7f9a680fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.463+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2674747045 >> 172.21.15.113:0/2674747045 conn(0x7f9a680f34b0 msgr2=0x7f9a680f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:49.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.463+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2674747045 shutdown_connections 2024-04-12T05:29:49.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.463+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2674747045 wait complete. 2024-04-12T05:29:49.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.464+0000 7f9a6dac9700 1 Processor -- start 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.464+0000 7f9a6dac9700 1 -- start start 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.464+0000 7f9a6dac9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a680fd8f0 0x7f9a6810dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.464+0000 7f9a6dac9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a680fe250 0x7f9a6810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.464+0000 7f9a6dac9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6810eae0 con 0x7f9a680fe250 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.464+0000 7f9a6dac9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6810ec20 con 0x7f9a680fd8f0 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a66ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a680fe250 0x7f9a6810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a66ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a680fe250 0x7f9a6810e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58796/0 (socket says 172.21.15.113:58796) 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a66ffd700 1 -- 172.21.15.113:0/2467380393 learned_addr learned my addr 172.21.15.113:0/2467380393 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a677fe700 1 --2- 172.21.15.113:0/2467380393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a680fd8f0 0x7f9a6810dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a66ffd700 1 -- 172.21.15.113:0/2467380393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a680fd8f0 msgr2=0x7f9a6810dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a66ffd700 1 --2- 172.21.15.113:0/2467380393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a680fd8f0 0x7f9a6810dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a66ffd700 1 -- 172.21.15.113:0/2467380393 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c005040 con 0x7f9a680fe250 2024-04-12T05:29:49.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a66ffd700 1 --2- 172.21.15.113:0/2467380393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a680fe250 0x7f9a6810e4e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f9a5c000c00 tx=0x7f9a5c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:49.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a64f74700 1 -- 172.21.15.113:0/2467380393 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a5c00e030 con 0x7f9a680fe250 2024-04-12T05:29:49.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2467380393 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a68108250 con 0x7f9a680fe250 2024-04-12T05:29:49.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a64f74700 1 -- 172.21.15.113:0/2467380393 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a5c007e90 con 0x7f9a680fe250 2024-04-12T05:29:49.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a64f74700 1 -- 172.21.15.113:0/2467380393 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a5c018070 con 0x7f9a680fe250 2024-04-12T05:29:49.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.465+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2467380393 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a681087b0 con 0x7f9a680fe250 2024-04-12T05:29:49.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.466+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2467380393 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6805f590 con 0x7f9a680fe250 2024-04-12T05:29:49.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.467+0000 7f9a64f74700 1 -- 172.21.15.113:0/2467380393 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9a5c01d420 con 0x7f9a680fe250 2024-04-12T05:29:49.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.467+0000 7f9a64f74700 1 --2- 172.21.15.113:0/2467380393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a5006cab0 0x7f9a5006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:49.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.467+0000 7f9a64f74700 1 -- 172.21.15.113:0/2467380393 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9a5c08cfe0 con 0x7f9a680fe250 2024-04-12T05:29:49.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.467+0000 7f9a677fe700 1 --2- 172.21.15.113:0/2467380393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a5006cab0 0x7f9a5006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:49.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.468+0000 7f9a677fe700 1 --2- 172.21.15.113:0/2467380393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a5006cab0 0x7f9a5006ef70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f9a58004de0 tx=0x7f9a5800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:49.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.472+0000 7f9a64f74700 1 -- 172.21.15.113:0/2467380393 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9a5c05aa90 con 0x7f9a680fe250 2024-04-12T05:29:49.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.676+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2467380393 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a680ffc40 con 0x7f9a5006cab0 2024-04-12T05:29:49.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.677+0000 7f9a64f74700 1 -- 172.21.15.113:0/2467380393 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9a680ffc40 con 0x7f9a5006cab0 2024-04-12T05:29:49.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.679+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2467380393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a5006cab0 msgr2=0x7f9a5006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.679+0000 7f9a6dac9700 1 --2- 172.21.15.113:0/2467380393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a5006cab0 0x7f9a5006ef70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f9a58004de0 tx=0x7f9a5800b040).stop 2024-04-12T05:29:49.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.679+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2467380393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a680fe250 msgr2=0x7f9a6810e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.679+0000 7f9a6dac9700 1 --2- 172.21.15.113:0/2467380393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a680fe250 0x7f9a6810e4e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f9a5c000c00 tx=0x7f9a5c009cb0).stop 2024-04-12T05:29:49.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.679+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2467380393 shutdown_connections 2024-04-12T05:29:49.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.679+0000 7f9a6dac9700 1 --2- 172.21.15.113:0/2467380393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a5006cab0 0x7f9a5006ef70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.679+0000 7f9a6dac9700 1 --2- 172.21.15.113:0/2467380393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a680fd8f0 0x7f9a6810dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.679+0000 7f9a6dac9700 1 --2- 172.21.15.113:0/2467380393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a680fe250 0x7f9a6810e4e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.679+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2467380393 >> 172.21.15.113:0/2467380393 conn(0x7f9a680f34b0 msgr2=0x7f9a680fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:49.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.679+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2467380393 shutdown_connections 2024-04-12T05:29:49.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.679+0000 7f9a6dac9700 1 -- 172.21.15.113:0/2467380393 wait complete. 2024-04-12T05:29:49.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.777+0000 7fc9fe523700 1 -- 172.21.15.113:0/805778070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9f80fc7f0 msgr2=0x7fc9f80fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.778 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.777+0000 7fc9fe523700 1 --2- 172.21.15.113:0/805778070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9f80fc7f0 0x7fc9f80fcbd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7fc9e00047f0 tx=0x7fc9e0009cb0).stop 2024-04-12T05:29:49.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.777+0000 7fc9fe523700 1 -- 172.21.15.113:0/805778070 shutdown_connections 2024-04-12T05:29:49.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.777+0000 7fc9fe523700 1 --2- 172.21.15.113:0/805778070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9f80fd150 0x7fc9f80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.777+0000 7fc9fe523700 1 --2- 172.21.15.113:0/805778070 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9f80fc7f0 0x7fc9f80fcbd0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.777+0000 7fc9fe523700 1 -- 172.21.15.113:0/805778070 >> 172.21.15.113:0/805778070 conn(0x7fc9f8070590 msgr2=0x7fc9f80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:49.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.777+0000 7fc9fe523700 1 -- 172.21.15.113:0/805778070 shutdown_connections 2024-04-12T05:29:49.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.777+0000 7fc9fe523700 1 -- 172.21.15.113:0/805778070 wait complete. 2024-04-12T05:29:49.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9fe523700 1 Processor -- start 2024-04-12T05:29:49.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9fe523700 1 -- start start 2024-04-12T05:29:49.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9fe523700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9f80fc7f0 0x7fc9f8109100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:49.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9fe523700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9f80fd150 0x7fc9f81095f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:49.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9fe523700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f8109ae0 con 0x7fc9f80fd150 2024-04-12T05:29:49.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9fe523700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f8109c20 con 0x7fc9f80fc7f0 2024-04-12T05:29:49.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9f77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9f80fd150 0x7fc9f81095f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:49.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9f77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9f80fd150 0x7fc9f81095f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58810/0 (socket says 172.21.15.113:58810) 2024-04-12T05:29:49.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9f77fe700 1 -- 172.21.15.113:0/858121877 learned_addr learned my addr 172.21.15.113:0/858121877 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:49.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9f7fff700 1 --2- 172.21.15.113:0/858121877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9f80fc7f0 0x7fc9f8109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:49.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9f77fe700 1 -- 172.21.15.113:0/858121877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9f80fc7f0 msgr2=0x7fc9f8109100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9f77fe700 1 --2- 172.21.15.113:0/858121877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9f80fc7f0 0x7fc9f8109100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.778+0000 7fc9f77fe700 1 -- 172.21.15.113:0/858121877 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e0005040 con 0x7fc9f80fd150 2024-04-12T05:29:49.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.779+0000 7fc9f77fe700 1 --2- 172.21.15.113:0/858121877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9f80fd150 0x7fc9f81095f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7fc9e8002820 tx=0x7fc9e8009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:49.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.779+0000 7fc9f57fa700 1 -- 172.21.15.113:0/858121877 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9e8007b40 con 0x7fc9f80fd150 2024-04-12T05:29:49.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.779+0000 7fc9f57fa700 1 -- 172.21.15.113:0/858121877 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9e800f040 con 0x7fc9f80fd150 2024-04-12T05:29:49.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.779+0000 7fc9fe523700 1 -- 172.21.15.113:0/858121877 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f8109ed0 con 0x7fc9f80fd150 2024-04-12T05:29:49.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.779+0000 7fc9f57fa700 1 -- 172.21.15.113:0/858121877 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9e8014910 con 0x7fc9f80fd150 2024-04-12T05:29:49.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.779+0000 7fc9fe523700 1 -- 172.21.15.113:0/858121877 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f81124f0 con 0x7fc9f80fd150 2024-04-12T05:29:49.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.780+0000 7fc9fe523700 1 -- 172.21.15.113:0/858121877 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f805f590 con 0x7fc9f80fd150 2024-04-12T05:29:49.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.780+0000 7fc9f57fa700 1 -- 172.21.15.113:0/858121877 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc9e8007ca0 con 0x7fc9f80fd150 2024-04-12T05:29:49.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.781+0000 7fc9f57fa700 1 --2- 172.21.15.113:0/858121877 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9e406caa0 0x7fc9e406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:49.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.781+0000 7fc9f57fa700 1 -- 172.21.15.113:0/858121877 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc9e808b750 con 0x7fc9f80fd150 2024-04-12T05:29:49.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.781+0000 7fc9f7fff700 1 --2- 172.21.15.113:0/858121877 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9e406caa0 0x7fc9e406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:49.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.781+0000 7fc9f7fff700 1 --2- 172.21.15.113:0/858121877 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9e406caa0 0x7fc9e406ef60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fc9e0004f10 tx=0x7fc9e000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:49.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.785+0000 7fc9f57fa700 1 -- 172.21.15.113:0/858121877 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc9e8055770 con 0x7fc9f80fd150 2024-04-12T05:29:49.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.984+0000 7fc9fe523700 1 -- 172.21.15.113:0/858121877 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9f8060d50 con 0x7fc9e406caa0 2024-04-12T05:29:49.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.993+0000 7fc9f57fa700 1 -- 172.21.15.113:0/858121877 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc9f8060d50 con 0x7fc9e406caa0 2024-04-12T05:29:49.994 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:29:49.994 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (17m) 10m ago 28m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:29:49.994 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (15m) 10m ago 28m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:29:49.994 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (14m) 10m ago 28m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:29:49.994 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (17m) 10m ago 28m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (10m) 10m ago 23m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (10m) 10m ago 23m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (10m) 10m ago 23m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (23m) 10m ago 23m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (19m) 10m ago 30m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (18m) 10m ago 28m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (16m) 10m ago 30m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (15m) 10m ago 28m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (18m) 10m ago 28m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (18m) 10m ago 27m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (14m) 10m ago 27m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (14m) 10m ago 26m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (13m) 10m ago 26m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:29:49.995 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (12m) 10m ago 26m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:29:49.996 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (12m) 10m ago 25m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:29:49.996 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (11m) 10m ago 25m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:29:49.996 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (17m) 10m ago 27m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:29:49.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.995+0000 7fc9fe523700 1 -- 172.21.15.113:0/858121877 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9e406caa0 msgr2=0x7fc9e406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.995+0000 7fc9fe523700 1 --2- 172.21.15.113:0/858121877 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9e406caa0 0x7fc9e406ef60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fc9e0004f10 tx=0x7fc9e000b040).stop 2024-04-12T05:29:49.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.995+0000 7fc9fe523700 1 -- 172.21.15.113:0/858121877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9f80fd150 msgr2=0x7fc9f81095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:49.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.995+0000 7fc9fe523700 1 --2- 172.21.15.113:0/858121877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9f80fd150 0x7fc9f81095f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7fc9e8002820 tx=0x7fc9e8009f20).stop 2024-04-12T05:29:49.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.995+0000 7fc9fe523700 1 -- 172.21.15.113:0/858121877 shutdown_connections 2024-04-12T05:29:49.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.995+0000 7fc9fe523700 1 --2- 172.21.15.113:0/858121877 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc9e406caa0 0x7fc9e406ef60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.995+0000 7fc9fe523700 1 --2- 172.21.15.113:0/858121877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9f80fc7f0 0x7fc9f8109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.995+0000 7fc9fe523700 1 --2- 172.21.15.113:0/858121877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9f80fd150 0x7fc9f81095f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:49.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.995+0000 7fc9fe523700 1 -- 172.21.15.113:0/858121877 >> 172.21.15.113:0/858121877 conn(0x7fc9f8070590 msgr2=0x7fc9f80fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:49.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.995+0000 7fc9fe523700 1 -- 172.21.15.113:0/858121877 shutdown_connections 2024-04-12T05:29:49.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:49.995+0000 7fc9fe523700 1 -- 172.21.15.113:0/858121877 wait complete. 2024-04-12T05:29:50.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.093+0000 7fac39334700 1 -- 172.21.15.113:0/2691715420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac340fd1f0 msgr2=0x7fac340f69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:50.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.093+0000 7fac39334700 1 --2- 172.21.15.113:0/2691715420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac340fd1f0 0x7fac340f69d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7fac280047f0 tx=0x7fac28009cb0).stop 2024-04-12T05:29:50.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.093+0000 7fac39334700 1 -- 172.21.15.113:0/2691715420 shutdown_connections 2024-04-12T05:29:50.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.093+0000 7fac39334700 1 --2- 172.21.15.113:0/2691715420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac340fd1f0 0x7fac340f69d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.093+0000 7fac39334700 1 --2- 172.21.15.113:0/2691715420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac340fc890 0x7fac340fcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.093+0000 7fac39334700 1 -- 172.21.15.113:0/2691715420 >> 172.21.15.113:0/2691715420 conn(0x7fac340f2490 msgr2=0x7fac340f48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.093+0000 7fac39334700 1 -- 172.21.15.113:0/2691715420 shutdown_connections 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.093+0000 7fac39334700 1 -- 172.21.15.113:0/2691715420 wait complete. 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.094+0000 7fac39334700 1 Processor -- start 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.094+0000 7fac39334700 1 -- start start 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.094+0000 7fac39334700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac340fc890 0x7fac34106f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.094+0000 7fac39334700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac340fd1f0 0x7fac34107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.094+0000 7fac39334700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac34061a90 con 0x7fac340fc890 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.094+0000 7fac39334700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac34107940 con 0x7fac340fd1f0 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.094+0000 7fac32ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac340fc890 0x7fac34106f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.094+0000 7fac32ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac340fc890 0x7fac34106f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58830/0 (socket says 172.21.15.113:58830) 2024-04-12T05:29:50.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.094+0000 7fac32ffd700 1 -- 172.21.15.113:0/757411430 learned_addr learned my addr 172.21.15.113:0/757411430 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:50.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.095+0000 7fac327fc700 1 --2- 172.21.15.113:0/757411430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac340fd1f0 0x7fac34107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:50.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.095+0000 7fac32ffd700 1 -- 172.21.15.113:0/757411430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac340fd1f0 msgr2=0x7fac34107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:50.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.095+0000 7fac32ffd700 1 --2- 172.21.15.113:0/757411430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac340fd1f0 0x7fac34107450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.095+0000 7fac32ffd700 1 -- 172.21.15.113:0/757411430 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac28005040 con 0x7fac340fc890 2024-04-12T05:29:50.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.095+0000 7fac32ffd700 1 --2- 172.21.15.113:0/757411430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac340fc890 0x7fac34106f60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fac2400c740 tx=0x7fac24007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:50.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.095+0000 7fac1bfff700 1 -- 172.21.15.113:0/757411430 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac2400e680 con 0x7fac340fc890 2024-04-12T05:29:50.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.095+0000 7fac1bfff700 1 -- 172.21.15.113:0/757411430 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac2400ec90 con 0x7fac340fc890 2024-04-12T05:29:50.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.095+0000 7fac39334700 1 -- 172.21.15.113:0/757411430 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac34107c20 con 0x7fac340fc890 2024-04-12T05:29:50.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.095+0000 7fac1bfff700 1 -- 172.21.15.113:0/757411430 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac24013d30 con 0x7fac340fc890 2024-04-12T05:29:50.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.095+0000 7fac39334700 1 -- 172.21.15.113:0/757411430 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac34114790 con 0x7fac340fc890 2024-04-12T05:29:50.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.096+0000 7fac39334700 1 -- 172.21.15.113:0/757411430 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac34060330 con 0x7fac340fc890 2024-04-12T05:29:50.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.096+0000 7fac1bfff700 1 -- 172.21.15.113:0/757411430 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fac24013410 con 0x7fac340fc890 2024-04-12T05:29:50.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.097+0000 7fac1bfff700 1 --2- 172.21.15.113:0/757411430 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac1c06caa0 0x7fac1c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:50.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.097+0000 7fac1bfff700 1 -- 172.21.15.113:0/757411430 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fac2408c090 con 0x7fac340fc890 2024-04-12T05:29:50.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.097+0000 7fac327fc700 1 --2- 172.21.15.113:0/757411430 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac1c06caa0 0x7fac1c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:50.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.097+0000 7fac327fc700 1 --2- 172.21.15.113:0/757411430 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac1c06caa0 0x7fac1c06ef60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fac34061540 tx=0x7fac2800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:50.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.101+0000 7fac1bfff700 1 -- 172.21.15.113:0/757411430 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fac24059b40 con 0x7fac340fc890 2024-04-12T05:29:50.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.372+0000 7fac39334700 1 -- 172.21.15.113:0/757411430 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fac34061270 con 0x7fac340fc890 2024-04-12T05:29:50.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.373+0000 7fac1bfff700 1 -- 172.21.15.113:0/757411430 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fac240596d0 con 0x7fac340fc890 2024-04-12T05:29:50.374 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:29:50.374 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:29:50.374 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:29:50.374 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:29:50.374 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:29:50.375 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:29:50.376 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:29:50.376 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:29:50.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.375+0000 7fac39334700 1 -- 172.21.15.113:0/757411430 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac1c06caa0 msgr2=0x7fac1c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:50.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.375+0000 7fac39334700 1 --2- 172.21.15.113:0/757411430 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac1c06caa0 0x7fac1c06ef60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fac34061540 tx=0x7fac2800b040).stop 2024-04-12T05:29:50.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.375+0000 7fac39334700 1 -- 172.21.15.113:0/757411430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac340fc890 msgr2=0x7fac34106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:50.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.375+0000 7fac39334700 1 --2- 172.21.15.113:0/757411430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac340fc890 0x7fac34106f60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fac2400c740 tx=0x7fac24007ef0).stop 2024-04-12T05:29:50.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.375+0000 7fac39334700 1 -- 172.21.15.113:0/757411430 shutdown_connections 2024-04-12T05:29:50.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.375+0000 7fac39334700 1 --2- 172.21.15.113:0/757411430 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac1c06caa0 0x7fac1c06ef60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.375+0000 7fac39334700 1 --2- 172.21.15.113:0/757411430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac340fc890 0x7fac34106f60 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.375+0000 7fac39334700 1 --2- 172.21.15.113:0/757411430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac340fd1f0 0x7fac34107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.375+0000 7fac39334700 1 -- 172.21.15.113:0/757411430 >> 172.21.15.113:0/757411430 conn(0x7fac340f2490 msgr2=0x7fac340fac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:50.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.375+0000 7fac39334700 1 -- 172.21.15.113:0/757411430 shutdown_connections 2024-04-12T05:29:50.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.375+0000 7fac39334700 1 -- 172.21.15.113:0/757411430 wait complete. 2024-04-12T05:29:50.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.473+0000 7f424496e700 1 -- 172.21.15.113:0/174257048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42400fd910 msgr2=0x7f42400fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:50.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.473+0000 7f424496e700 1 --2- 172.21.15.113:0/174257048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42400fd910 0x7f42400fdcf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f42340047f0 tx=0x7f4234009cb0).stop 2024-04-12T05:29:50.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.473+0000 7f424496e700 1 -- 172.21.15.113:0/174257048 shutdown_connections 2024-04-12T05:29:50.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.473+0000 7f424496e700 1 --2- 172.21.15.113:0/174257048 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42400fe270 0x7f42400f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:29:50.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.473+0000 7f424496e700 1 --2- 172.21.15.113:0/174257048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42400fd910 0x7f42400fdcf0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.473+0000 7f424496e700 1 -- 172.21.15.113:0/174257048 >> 172.21.15.113:0/174257048 conn(0x7f42400f3530 msgr2=0x7f42400f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:50.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.473+0000 7f424496e700 1 -- 172.21.15.113:0/174257048 shutdown_connections 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.473+0000 7f424496e700 1 -- 172.21.15.113:0/174257048 wait complete. 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f424496e700 1 Processor -- start 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f424496e700 1 -- start start 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f424496e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42400fd910 0x7f4240110280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f424496e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42400fe270 0x7f4240110770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f424496e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4240110e00 con 0x7f42400fe270 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f424496e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424010a420 con 0x7f42400fd910 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f423dd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42400fe270 0x7f4240110770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f423dd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42400fe270 0x7f4240110770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53776/0 (socket says 172.21.15.113:53776) 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f423dd9b700 1 -- 172.21.15.113:0/163050247 learned_addr learned my addr 172.21.15.113:0/163050247 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f423e59c700 1 --2- 172.21.15.113:0/163050247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42400fd910 0x7f4240110280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f423dd9b700 1 -- 172.21.15.113:0/163050247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42400fd910 msgr2=0x7f4240110280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f423dd9b700 1 --2- 172.21.15.113:0/163050247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42400fd910 0x7f4240110280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f423dd9b700 1 -- 172.21.15.113:0/163050247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4234005040 con 0x7f42400fe270 2024-04-12T05:29:50.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.474+0000 7f423dd9b700 1 --2- 172.21.15.113:0/163050247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42400fe270 0x7f4240110770 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f423000b480 tx=0x7f42300090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:50.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.475+0000 7f422f7fe700 1 -- 172.21.15.113:0/163050247 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4230002500 con 0x7f42400fe270 2024-04-12T05:29:50.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.475+0000 7f422f7fe700 1 -- 172.21.15.113:0/163050247 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4230002b10 con 0x7f42400fe270 2024-04-12T05:29:50.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.475+0000 7f422f7fe700 1 -- 172.21.15.113:0/163050247 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4230012c90 con 0x7f42400fe270 2024-04-12T05:29:50.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.475+0000 7f424496e700 1 -- 172.21.15.113:0/163050247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424010a6d0 con 0x7f42400fe270 2024-04-12T05:29:50.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.475+0000 7f424496e700 1 -- 172.21.15.113:0/163050247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424010ac10 con 0x7f42400fe270 2024-04-12T05:29:50.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.476+0000 7f424496e700 1 -- 172.21.15.113:0/163050247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424005f570 con 0x7f42400fe270 2024-04-12T05:29:50.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.476+0000 7f422f7fe700 1 -- 172.21.15.113:0/163050247 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4230002660 con 0x7f42400fe270 2024-04-12T05:29:50.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.476+0000 7f422f7fe700 1 --2- 172.21.15.113:0/163050247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f422806caa0 0x7f422806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:50.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.477+0000 7f422f7fe700 1 -- 172.21.15.113:0/163050247 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f423008bf20 con 0x7f42400fe270 2024-04-12T05:29:50.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.477+0000 7f423e59c700 1 --2- 172.21.15.113:0/163050247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f422806caa0 0x7f422806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:50.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.477+0000 7f423e59c700 1 --2- 172.21.15.113:0/163050247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f422806caa0 0x7f422806ef60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f4234004f10 tx=0x7f423400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:50.481 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.481+0000 7f422f7fe700 1 -- 172.21.15.113:0/163050247 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f42300bf2c0 con 0x7f42400fe270 2024-04-12T05:29:50.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.709+0000 7f424496e700 1 -- 172.21.15.113:0/163050247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4240047540 con 0x7f42400fe270 2024-04-12T05:29:50.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.709+0000 7f422f7fe700 1 -- 172.21.15.113:0/163050247 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f42300599d0 con 0x7f42400fe270 2024-04-12T05:29:50.710 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:29:50.710 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:29:50.711 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:29:50.712 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:29:50.713 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:29:50.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.711+0000 7f424496e700 1 -- 172.21.15.113:0/163050247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f422806caa0 msgr2=0x7f422806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:50.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.711+0000 7f424496e700 1 --2- 172.21.15.113:0/163050247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f422806caa0 0x7f422806ef60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f4234004f10 tx=0x7f423400b040).stop 2024-04-12T05:29:50.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.711+0000 7f424496e700 1 -- 172.21.15.113:0/163050247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42400fe270 msgr2=0x7f4240110770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:50.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.711+0000 7f424496e700 1 --2- 172.21.15.113:0/163050247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42400fe270 0x7f4240110770 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f423000b480 tx=0x7f42300090f0).stop 2024-04-12T05:29:50.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.711+0000 7f424496e700 1 -- 172.21.15.113:0/163050247 shutdown_connections 2024-04-12T05:29:50.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.711+0000 7f424496e700 1 --2- 172.21.15.113:0/163050247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f422806caa0 0x7f422806ef60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.711+0000 7f424496e700 1 --2- 172.21.15.113:0/163050247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42400fd910 0x7f4240110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.711+0000 7f424496e700 1 --2- 172.21.15.113:0/163050247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f42400fe270 0x7f4240110770 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.711+0000 7f424496e700 1 -- 172.21.15.113:0/163050247 >> 172.21.15.113:0/163050247 conn(0x7f42400f3530 msgr2=0x7f424006b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:50.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.711+0000 7f424496e700 1 -- 172.21.15.113:0/163050247 shutdown_connections 2024-04-12T05:29:50.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.711+0000 7f424496e700 1 -- 172.21.15.113:0/163050247 wait complete. 2024-04-12T05:29:50.715 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:29:50.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.810+0000 7f7dddec9700 1 -- 172.21.15.113:0/3285215710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7dd80fe250 msgr2=0x7f7dd80f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:50.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.810+0000 7f7dddec9700 1 --2- 172.21.15.113:0/3285215710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7dd80fe250 0x7f7dd80f7a30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f7dc80047f0 tx=0x7f7dc8009cb0).stop 2024-04-12T05:29:50.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.810+0000 7f7dddec9700 1 -- 172.21.15.113:0/3285215710 shutdown_connections 2024-04-12T05:29:50.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.810+0000 7f7dddec9700 1 --2- 172.21.15.113:0/3285215710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7dd80fe250 0x7f7dd80f7a30 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.810+0000 7f7dddec9700 1 --2- 172.21.15.113:0/3285215710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dd80fd8f0 0x7f7dd80fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.810+0000 7f7dddec9700 1 -- 172.21.15.113:0/3285215710 >> 172.21.15.113:0/3285215710 conn(0x7f7dd80f34b0 msgr2=0x7f7dd80f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:50.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.810+0000 7f7dddec9700 1 -- 172.21.15.113:0/3285215710 shutdown_connections 2024-04-12T05:29:50.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.810+0000 7f7dddec9700 1 -- 172.21.15.113:0/3285215710 wait complete. 2024-04-12T05:29:50.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dddec9700 1 Processor -- start 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dddec9700 1 -- start start 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dddec9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7dd80fd8f0 0x7f7dd806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dddec9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dd80fe250 0x7f7dd806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dddec9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd806bab0 con 0x7f7dd80fd8f0 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dddec9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd806bbf0 con 0x7f7dd80fe250 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dd77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7dd80fd8f0 0x7f7dd806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dd77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7dd80fd8f0 0x7f7dd806b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53792/0 (socket says 172.21.15.113:53792) 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dd77fe700 1 -- 172.21.15.113:0/3700278178 learned_addr learned my addr 172.21.15.113:0/3700278178 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dd6ffd700 1 --2- 172.21.15.113:0/3700278178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dd80fe250 0x7f7dd806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dd77fe700 1 -- 172.21.15.113:0/3700278178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dd80fe250 msgr2=0x7f7dd806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dd77fe700 1 --2- 172.21.15.113:0/3700278178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dd80fe250 0x7f7dd806b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dd77fe700 1 -- 172.21.15.113:0/3700278178 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc8005040 con 0x7f7dd80fd8f0 2024-04-12T05:29:50.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.811+0000 7f7dd77fe700 1 --2- 172.21.15.113:0/3700278178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7dd80fd8f0 0x7f7dd806b0d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f7dcc002770 tx=0x7f7dcc002b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:50.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.812+0000 7f7dd4ff9700 1 -- 172.21.15.113:0/3700278178 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dcc002d80 con 0x7f7dd80fd8f0 2024-04-12T05:29:50.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.812+0000 7f7dd4ff9700 1 -- 172.21.15.113:0/3700278178 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7dcc00f040 con 0x7f7dd80fd8f0 2024-04-12T05:29:50.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.812+0000 7f7dddec9700 1 -- 172.21.15.113:0/3700278178 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd806bea0 con 0x7f7dd80fd8f0 2024-04-12T05:29:50.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.812+0000 7f7dd4ff9700 1 -- 172.21.15.113:0/3700278178 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dcc00d800 con 0x7f7dd80fd8f0 2024-04-12T05:29:50.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.812+0000 7f7dddec9700 1 -- 172.21.15.113:0/3700278178 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd81146e0 con 0x7f7dd80fd8f0 2024-04-12T05:29:50.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.813+0000 7f7dddec9700 1 -- 172.21.15.113:0/3700278178 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd805f590 con 0x7f7dd80fd8f0 2024-04-12T05:29:50.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.813+0000 7f7dd4ff9700 1 -- 172.21.15.113:0/3700278178 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7dcc009110 con 0x7f7dd80fd8f0 2024-04-12T05:29:50.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.813+0000 7f7dd4ff9700 1 --2- 172.21.15.113:0/3700278178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7dc006caa0 0x7f7dc006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:50.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.814+0000 7f7dd4ff9700 1 -- 172.21.15.113:0/3700278178 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7dcc08ba80 con 0x7f7dd80fd8f0 2024-04-12T05:29:50.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.814+0000 7f7dd6ffd700 1 --2- 172.21.15.113:0/3700278178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7dc006caa0 0x7f7dc006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:50.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.814+0000 7f7dd6ffd700 1 --2- 172.21.15.113:0/3700278178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7dc006caa0 0x7f7dc006ef60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f7dd806c870 tx=0x7f7dc800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:50.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:50.818+0000 7f7dd4ff9700 1 -- 172.21.15.113:0/3700278178 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7dcc055aa0 con 0x7f7dd80fd8f0 2024-04-12T05:29:51.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.024+0000 7f7dddec9700 1 -- 172.21.15.113:0/3700278178 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7dd8059120 con 0x7f7dc006caa0 2024-04-12T05:29:51.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.025+0000 7f7dd4ff9700 1 -- 172.21.15.113:0/3700278178 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7dd8059120 con 0x7f7dc006caa0 2024-04-12T05:29:51.026 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:29:51.026 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:29:51.026 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:29:51.027 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:29:51.027 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:29:51.027 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:29:51.027 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:29:51.027 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:29:51.027 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:29:51.027 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:29:51.027 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:29:51.027 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:29:51.027 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:29:51.027 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:29:51.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.027+0000 7f7dddec9700 1 -- 172.21.15.113:0/3700278178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7dc006caa0 msgr2=0x7f7dc006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:51.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.027+0000 7f7dddec9700 1 --2- 172.21.15.113:0/3700278178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7dc006caa0 0x7f7dc006ef60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f7dd806c870 tx=0x7f7dc800b040).stop 2024-04-12T05:29:51.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.027+0000 7f7dddec9700 1 -- 172.21.15.113:0/3700278178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7dd80fd8f0 msgr2=0x7f7dd806b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:51.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.027+0000 7f7dddec9700 1 --2- 172.21.15.113:0/3700278178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7dd80fd8f0 0x7f7dd806b0d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f7dcc002770 tx=0x7f7dcc002b30).stop 2024-04-12T05:29:51.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.027+0000 7f7dddec9700 1 -- 172.21.15.113:0/3700278178 shutdown_connections 2024-04-12T05:29:51.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.027+0000 7f7dddec9700 1 --2- 172.21.15.113:0/3700278178 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7dc006caa0 0x7f7dc006ef60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:51.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.027+0000 7f7dddec9700 1 --2- 172.21.15.113:0/3700278178 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7dd80fd8f0 0x7f7dd806b0d0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:51.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.027+0000 7f7dddec9700 1 --2- 172.21.15.113:0/3700278178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dd80fe250 0x7f7dd806b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:51.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.027+0000 7f7dddec9700 1 -- 172.21.15.113:0/3700278178 >> 172.21.15.113:0/3700278178 conn(0x7f7dd80f34b0 msgr2=0x7f7dd80fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:51.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.027+0000 7f7dddec9700 1 -- 172.21.15.113:0/3700278178 shutdown_connections 2024-04-12T05:29:51.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.027+0000 7f7dddec9700 1 -- 172.21.15.113:0/3700278178 wait complete. 2024-04-12T05:29:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.125+0000 7f927de75700 1 -- 172.21.15.113:0/2935730135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92780fe250 msgr2=0x7f92780f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.125+0000 7f927de75700 1 --2- 172.21.15.113:0/2935730135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92780fe250 0x7f92780f7a30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f92680047f0 tx=0x7f9268009cb0).stop 2024-04-12T05:29:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.125+0000 7f927de75700 1 -- 172.21.15.113:0/2935730135 shutdown_connections 2024-04-12T05:29:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.125+0000 7f927de75700 1 --2- 172.21.15.113:0/2935730135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92780fe250 0x7f92780f7a30 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.125+0000 7f927de75700 1 --2- 172.21.15.113:0/2935730135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92780fd8f0 0x7f92780fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:29:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.125+0000 7f927de75700 1 -- 172.21.15.113:0/2935730135 >> 172.21.15.113:0/2935730135 conn(0x7f92780f34b0 msgr2=0x7f92780f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:51.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.125+0000 7f927de75700 1 -- 172.21.15.113:0/2935730135 shutdown_connections 2024-04-12T05:29:51.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.125+0000 7f927de75700 1 -- 172.21.15.113:0/2935730135 wait complete. 2024-04-12T05:29:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.126+0000 7f927de75700 1 Processor -- start 2024-04-12T05:29:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.126+0000 7f927de75700 1 -- start start 2024-04-12T05:29:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.126+0000 7f927de75700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92780fd8f0 0x7f927810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.126+0000 7f927de75700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92780fe250 0x7f927810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.126+0000 7f927de75700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927810ebb0 con 0x7f92780fe250 2024-04-12T05:29:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.126+0000 7f927de75700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92781081d0 con 0x7f92780fd8f0 2024-04-12T05:29:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f9276ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92780fe250 0x7f927810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f9276ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92780fe250 0x7f927810e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53802/0 (socket says 172.21.15.113:53802) 2024-04-12T05:29:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f9276ffd700 1 -- 172.21.15.113:0/3102379956 learned_addr learned my addr 172.21.15.113:0/3102379956 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:29:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f92777fe700 1 --2- 172.21.15.113:0/3102379956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92780fd8f0 0x7f927810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:51.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f9276ffd700 1 -- 172.21.15.113:0/3102379956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92780fd8f0 msgr2=0x7f927810e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f9276ffd700 1 --2- 172.21.15.113:0/3102379956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92780fd8f0 0x7f927810e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f9276ffd700 1 -- 172.21.15.113:0/3102379956 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9268005040 con 0x7f92780fe250 2024-04-12T05:29:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f9276ffd700 1 --2- 172.21.15.113:0/3102379956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92780fe250 0x7f927810e520 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f92680099f0 tx=0x7f9268009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f9274ff9700 1 -- 172.21.15.113:0/3102379956 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f926800e030 con 0x7f92780fe250 2024-04-12T05:29:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f927de75700 1 -- 172.21.15.113:0/3102379956 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9278108420 con 0x7f92780fe250 2024-04-12T05:29:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f9274ff9700 1 -- 172.21.15.113:0/3102379956 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9268007e90 con 0x7f92780fe250 2024-04-12T05:29:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f9274ff9700 1 -- 172.21.15.113:0/3102379956 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9268018070 con 0x7f92780fe250 2024-04-12T05:29:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.127+0000 7f927de75700 1 -- 172.21.15.113:0/3102379956 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9278108900 con 0x7f92780fe250 2024-04-12T05:29:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.128+0000 7f927de75700 1 -- 172.21.15.113:0/3102379956 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927805f590 con 0x7f92780fe250 2024-04-12T05:29:51.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.129+0000 7f9274ff9700 1 -- 172.21.15.113:0/3102379956 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f926801d420 con 0x7f92780fe250 2024-04-12T05:29:51.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.129+0000 7f9274ff9700 1 --2- 172.21.15.113:0/3102379956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f926006caa0 0x7f926006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:29:51.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.129+0000 7f9274ff9700 1 -- 172.21.15.113:0/3102379956 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f926808d240 con 0x7f92780fe250 2024-04-12T05:29:51.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.129+0000 7f92777fe700 1 --2- 172.21.15.113:0/3102379956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f926006caa0 0x7f926006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:29:51.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.130+0000 7f92777fe700 1 --2- 172.21.15.113:0/3102379956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f926006caa0 0x7f926006ef60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f926c006fd0 tx=0x7f926c009060).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:29:51.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.134+0000 7f9274ff9700 1 -- 172.21.15.113:0/3102379956 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f926805acf0 con 0x7f92780fe250 2024-04-12T05:29:51.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.399+0000 7f927de75700 1 -- 172.21.15.113:0/3102379956 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f92781096b0 con 0x7f92780fe250 2024-04-12T05:29:51.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.400+0000 7f9274ff9700 1 -- 172.21.15.113:0/3102379956 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f926805a880 con 0x7f92780fe250 2024-04-12T05:29:51.401 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:29:51.401 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:29:51.401 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:29:51.402 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:29:51.402 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:29:51.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.402+0000 7f927de75700 1 -- 172.21.15.113:0/3102379956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f926006caa0 msgr2=0x7f926006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:51.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.402+0000 7f927de75700 1 --2- 172.21.15.113:0/3102379956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f926006caa0 0x7f926006ef60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f926c006fd0 tx=0x7f926c009060).stop 2024-04-12T05:29:51.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.402+0000 7f927de75700 1 -- 172.21.15.113:0/3102379956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92780fe250 msgr2=0x7f927810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:29:51.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.402+0000 7f927de75700 1 --2- 172.21.15.113:0/3102379956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92780fe250 0x7f927810e520 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f92680099f0 tx=0x7f9268009cb0).stop 2024-04-12T05:29:51.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.402+0000 7f927de75700 1 -- 172.21.15.113:0/3102379956 shutdown_connections 2024-04-12T05:29:51.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.402+0000 7f927de75700 1 --2- 172.21.15.113:0/3102379956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f926006caa0 0x7f926006ef60 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:51.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.402+0000 7f927de75700 1 --2- 172.21.15.113:0/3102379956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92780fd8f0 0x7f927810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:51.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.402+0000 7f927de75700 1 --2- 172.21.15.113:0/3102379956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92780fe250 0x7f927810e520 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:29:51.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.402+0000 7f927de75700 1 -- 172.21.15.113:0/3102379956 >> 172.21.15.113:0/3102379956 conn(0x7f92780f34b0 msgr2=0x7f92780fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:29:51.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.402+0000 7f927de75700 1 -- 172.21.15.113:0/3102379956 shutdown_connections 2024-04-12T05:29:51.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:29:51.402+0000 7f927de75700 1 -- 172.21.15.113:0/3102379956 wait complete. 2024-04-12T05:29:51.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:51 smithi113 ceph-mon[103371]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:51.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:51 smithi113 ceph-mon[103371]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:51.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:51 smithi113 ceph-mon[103371]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:51.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:51 smithi113 ceph-mon[103371]: pgmap v530: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:51.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:51 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/757411430' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:29:51.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:51 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/163050247' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:29:51.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:51 smithi192 ceph-mon[85304]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:51.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:51 smithi192 ceph-mon[85304]: from='client.34984 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:51.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:51 smithi192 ceph-mon[85304]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:51.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:51 smithi192 ceph-mon[85304]: pgmap v530: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:51 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/757411430' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:29:51.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:51 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/163050247' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:29:52.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:52 smithi113 ceph-mon[103371]: from='client.35000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:52.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:52 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3102379956' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:29:52.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:52.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:52 smithi192 ceph-mon[85304]: from='client.35000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:29:52.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:52 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3102379956' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:29:52.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:29:53.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:53 smithi113 ceph-mon[103371]: pgmap v531: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:53.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:53 smithi192 ceph-mon[85304]: pgmap v531: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:55.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:55 smithi113 ceph-mon[103371]: pgmap v532: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:55.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:55 smithi192 ceph-mon[85304]: pgmap v532: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:57.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:57 smithi113 ceph-mon[103371]: pgmap v533: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:57.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:57 smithi192 ceph-mon[85304]: pgmap v533: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:59.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:29:59 smithi113 ceph-mon[103371]: pgmap v534: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:29:59.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:29:59 smithi192 ceph-mon[85304]: pgmap v534: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:29:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:30:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:00 smithi113 ceph-mon[103371]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:30:00.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:00 smithi192 ceph-mon[85304]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:30:01.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:01 smithi113 ceph-mon[103371]: pgmap v535: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:01.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:01 smithi192 ceph-mon[85304]: pgmap v535: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:03.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:03 smithi113 ceph-mon[103371]: pgmap v536: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:03.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:03 smithi192 ceph-mon[85304]: pgmap v536: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:05.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:05 smithi113 ceph-mon[103371]: pgmap v537: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:05.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:05 smithi192 ceph-mon[85304]: pgmap v537: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:07.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:07 smithi113 ceph-mon[103371]: pgmap v538: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:07.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:07.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:07 smithi192 ceph-mon[85304]: pgmap v538: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:07.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:09.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:09 smithi113 ceph-mon[103371]: pgmap v539: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:09.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:09 smithi192 ceph-mon[85304]: pgmap v539: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:11.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:11 smithi113 ceph-mon[103371]: pgmap v540: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:11.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:11 smithi192 ceph-mon[85304]: pgmap v540: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:13.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:13 smithi113 ceph-mon[103371]: pgmap v541: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:13.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:13 smithi192 ceph-mon[85304]: pgmap v541: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:15.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:15 smithi113 ceph-mon[103371]: pgmap v542: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:15.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:15 smithi192 ceph-mon[85304]: pgmap v542: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:17.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:17 smithi113 ceph-mon[103371]: pgmap v543: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:17.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:17 smithi192 ceph-mon[85304]: pgmap v543: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:19.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:19 smithi113 ceph-mon[103371]: pgmap v544: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:19.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:19 smithi192 ceph-mon[85304]: pgmap v544: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:21.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:21 smithi113 ceph-mon[103371]: pgmap v545: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:21.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:21 smithi192 ceph-mon[85304]: pgmap v545: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:21.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.508+0000 7f1b5c32e700 1 -- 172.21.15.113:0/1859455822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b540fd150 msgr2=0x7f1b540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:21.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.508+0000 7f1b5c32e700 1 --2- 172.21.15.113:0/1859455822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b540fd150 0x7f1b540f6930 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f1b440047f0 tx=0x7f1b44009cb0).stop 2024-04-12T05:30:21.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.508+0000 7f1b5c32e700 1 -- 172.21.15.113:0/1859455822 shutdown_connections 2024-04-12T05:30:21.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.508+0000 7f1b5c32e700 1 --2- 172.21.15.113:0/1859455822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b540fd150 0x7f1b540f6930 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:21.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.508+0000 7f1b5c32e700 1 --2- 172.21.15.113:0/1859455822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b540fc7f0 0x7f1b540fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:21.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.508+0000 7f1b5c32e700 1 -- 172.21.15.113:0/1859455822 >> 172.21.15.113:0/1859455822 conn(0x7f1b54070590 msgr2=0x7f1b540f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:21.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.508+0000 7f1b5c32e700 1 -- 172.21.15.113:0/1859455822 shutdown_connections 2024-04-12T05:30:21.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.509+0000 7f1b5c32e700 1 -- 172.21.15.113:0/1859455822 wait complete. 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.509+0000 7f1b5c32e700 1 Processor -- start 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.509+0000 7f1b5c32e700 1 -- start start 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.509+0000 7f1b5c32e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b540fc7f0 0x7f1b541090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.509+0000 7f1b5c32e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b540fd150 0x7f1b541095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.509+0000 7f1b5c32e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b54109aa0 con 0x7f1b540fc7f0 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.509+0000 7f1b5c32e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b54109be0 con 0x7f1b540fd150 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.509+0000 7f1b5a0ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b540fc7f0 0x7f1b541090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.509+0000 7f1b5a0ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b540fc7f0 0x7f1b541090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37484/0 (socket says 172.21.15.113:37484) 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.509+0000 7f1b5a0ca700 1 -- 172.21.15.113:0/4200710848 learned_addr learned my addr 172.21.15.113:0/4200710848 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.510+0000 7f1b598c9700 1 --2- 172.21.15.113:0/4200710848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b540fd150 0x7f1b541095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.510+0000 7f1b5a0ca700 1 -- 172.21.15.113:0/4200710848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b540fd150 msgr2=0x7f1b541095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.510+0000 7f1b5a0ca700 1 --2- 172.21.15.113:0/4200710848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b540fd150 0x7f1b541095b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.510+0000 7f1b5a0ca700 1 -- 172.21.15.113:0/4200710848 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44005040 con 0x7f1b540fc7f0 2024-04-12T05:30:21.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.510+0000 7f1b5a0ca700 1 --2- 172.21.15.113:0/4200710848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b540fc7f0 0x7f1b541090c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f1b5000dad0 tx=0x7f1b50004ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:21.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.510+0000 7f1b4b7fe700 1 -- 172.21.15.113:0/4200710848 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b5000b750 con 0x7f1b540fc7f0 2024-04-12T05:30:21.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.510+0000 7f1b4b7fe700 1 -- 172.21.15.113:0/4200710848 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b5000f040 con 0x7f1b540fc7f0 2024-04-12T05:30:21.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.510+0000 7f1b5c32e700 1 -- 172.21.15.113:0/4200710848 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b54109e90 con 0x7f1b540fc7f0 2024-04-12T05:30:21.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.510+0000 7f1b4b7fe700 1 -- 172.21.15.113:0/4200710848 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b50013ce0 con 0x7f1b540fc7f0 2024-04-12T05:30:21.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.510+0000 7f1b5c32e700 1 -- 172.21.15.113:0/4200710848 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b54112480 con 0x7f1b540fc7f0 2024-04-12T05:30:21.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.511+0000 7f1b5c32e700 1 -- 172.21.15.113:0/4200710848 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5405f590 con 0x7f1b540fc7f0 2024-04-12T05:30:21.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.511+0000 7f1b4b7fe700 1 -- 172.21.15.113:0/4200710848 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1b5000b8b0 con 0x7f1b540fc7f0 2024-04-12T05:30:21.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.512+0000 7f1b4b7fe700 1 --2- 172.21.15.113:0/4200710848 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b4006caa0 0x7f1b4006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:21.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.512+0000 7f1b4b7fe700 1 -- 172.21.15.113:0/4200710848 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1b5008b840 con 0x7f1b540fc7f0 2024-04-12T05:30:21.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.512+0000 7f1b598c9700 1 --2- 172.21.15.113:0/4200710848 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b4006caa0 0x7f1b4006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:21.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.512+0000 7f1b598c9700 1 --2- 172.21.15.113:0/4200710848 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b4006caa0 0x7f1b4006ef60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f1b44000c00 tx=0x7f1b4400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:21.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.517+0000 7f1b4b7fe700 1 -- 172.21.15.113:0/4200710848 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1b50055910 con 0x7f1b540fc7f0 2024-04-12T05:30:21.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.721+0000 7f1b5c32e700 1 -- 172.21.15.113:0/4200710848 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b54059120 con 0x7f1b4006caa0 2024-04-12T05:30:21.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.722+0000 7f1b4b7fe700 1 -- 172.21.15.113:0/4200710848 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1b54059120 con 0x7f1b4006caa0 2024-04-12T05:30:21.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.724+0000 7f1b5c32e700 1 -- 172.21.15.113:0/4200710848 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b4006caa0 msgr2=0x7f1b4006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:21.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.724+0000 7f1b5c32e700 1 --2- 172.21.15.113:0/4200710848 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b4006caa0 0x7f1b4006ef60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f1b44000c00 tx=0x7f1b4400b040).stop 2024-04-12T05:30:21.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.724+0000 7f1b5c32e700 1 -- 172.21.15.113:0/4200710848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b540fc7f0 msgr2=0x7f1b541090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:21.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.724+0000 7f1b5c32e700 1 --2- 172.21.15.113:0/4200710848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b540fc7f0 0x7f1b541090c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f1b5000dad0 tx=0x7f1b50004ee0).stop 2024-04-12T05:30:21.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.724+0000 7f1b5c32e700 1 -- 172.21.15.113:0/4200710848 shutdown_connections 2024-04-12T05:30:21.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.724+0000 7f1b5c32e700 1 --2- 172.21.15.113:0/4200710848 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b4006caa0 0x7f1b4006ef60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:21.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.724+0000 7f1b5c32e700 1 --2- 172.21.15.113:0/4200710848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b540fc7f0 0x7f1b541090c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:21.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.724+0000 7f1b5c32e700 1 --2- 172.21.15.113:0/4200710848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b540fd150 0x7f1b541095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:21.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.724+0000 7f1b5c32e700 1 -- 172.21.15.113:0/4200710848 >> 172.21.15.113:0/4200710848 conn(0x7f1b54070590 msgr2=0x7f1b540fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:21.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.724+0000 7f1b5c32e700 1 -- 172.21.15.113:0/4200710848 shutdown_connections 2024-04-12T05:30:21.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.724+0000 7f1b5c32e700 1 -- 172.21.15.113:0/4200710848 wait complete. 2024-04-12T05:30:21.735 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:30:21.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.823+0000 7f2ce1b28700 1 -- 172.21.15.113:0/209523379 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc0fe270 msgr2=0x7f2cdc0f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:21.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.823+0000 7f2ce1b28700 1 --2- 172.21.15.113:0/209523379 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc0fe270 0x7f2cdc0f7a50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f2cd00047f0 tx=0x7f2cd0009cb0).stop 2024-04-12T05:30:21.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.823+0000 7f2ce1b28700 1 -- 172.21.15.113:0/209523379 shutdown_connections 2024-04-12T05:30:21.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.823+0000 7f2ce1b28700 1 --2- 172.21.15.113:0/209523379 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc0fe270 0x7f2cdc0f7a50 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.823+0000 7f2ce1b28700 1 --2- 172.21.15.113:0/209523379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cdc0fd910 0x7f2cdc0fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.823+0000 7f2ce1b28700 1 -- 172.21.15.113:0/209523379 >> 172.21.15.113:0/209523379 conn(0x7f2cdc0f3530 msgr2=0x7f2cdc0f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.823+0000 7f2ce1b28700 1 -- 172.21.15.113:0/209523379 shutdown_connections 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.823+0000 7f2ce1b28700 1 -- 172.21.15.113:0/209523379 wait complete. 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.823+0000 7f2ce1b28700 1 Processor -- start 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2ce1b28700 1 -- start start 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2ce1b28700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cdc0fd910 0x7f2cdc10dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2ce1b28700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc0fe270 0x7f2cdc10e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2ce1b28700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc10eab0 con 0x7f2cdc0fe270 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2ce1b28700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc10ebf0 con 0x7f2cdc0fd910 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2cdaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc0fe270 0x7f2cdc10e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2cdaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc0fe270 0x7f2cdc10e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37510/0 (socket says 172.21.15.113:37510) 2024-04-12T05:30:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2cdaffd700 1 -- 172.21.15.113:0/1542349716 learned_addr learned my addr 172.21.15.113:0/1542349716 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:21.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2cdb7fe700 1 --2- 172.21.15.113:0/1542349716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cdc0fd910 0x7f2cdc10dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:21.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2cdaffd700 1 -- 172.21.15.113:0/1542349716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cdc0fd910 msgr2=0x7f2cdc10dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:21.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2cdaffd700 1 --2- 172.21.15.113:0/1542349716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cdc0fd910 0x7f2cdc10dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:21.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2cdaffd700 1 -- 172.21.15.113:0/1542349716 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd0005040 con 0x7f2cdc0fe270 2024-04-12T05:30:21.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.824+0000 7f2cdaffd700 1 --2- 172.21.15.113:0/1542349716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc0fe270 0x7f2cdc10e4b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f2cd00099f0 tx=0x7f2cd0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:21.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.825+0000 7f2cd8ff9700 1 -- 172.21.15.113:0/1542349716 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cd000e030 con 0x7f2cdc0fe270 2024-04-12T05:30:21.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.825+0000 7f2ce1b28700 1 -- 172.21.15.113:0/1542349716 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc108220 con 0x7f2cdc0fe270 2024-04-12T05:30:21.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.825+0000 7f2cd8ff9700 1 -- 172.21.15.113:0/1542349716 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2cd0007e90 con 0x7f2cdc0fe270 2024-04-12T05:30:21.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.825+0000 7f2cd8ff9700 1 -- 172.21.15.113:0/1542349716 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cd0018070 con 0x7f2cdc0fe270 2024-04-12T05:30:21.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.825+0000 7f2ce1b28700 1 -- 172.21.15.113:0/1542349716 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc108780 con 0x7f2cdc0fe270 2024-04-12T05:30:21.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.826+0000 7f2ce1b28700 1 -- 172.21.15.113:0/1542349716 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc05f570 con 0x7f2cdc0fe270 2024-04-12T05:30:21.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.826+0000 7f2cd8ff9700 1 -- 172.21.15.113:0/1542349716 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2cd0009590 con 0x7f2cdc0fe270 2024-04-12T05:30:21.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.826+0000 7f2cd8ff9700 1 --2- 172.21.15.113:0/1542349716 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cc406ca30 0x7f2cc406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:21.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.827+0000 7f2cd8ff9700 1 -- 172.21.15.113:0/1542349716 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2cd008d0f0 con 0x7f2cdc0fe270 2024-04-12T05:30:21.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.827+0000 7f2cdb7fe700 1 --2- 172.21.15.113:0/1542349716 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cc406ca30 0x7f2cc406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:21.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.827+0000 7f2cdb7fe700 1 --2- 172.21.15.113:0/1542349716 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cc406ca30 0x7f2cc406eef0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f2ccc009170 tx=0x7f2ccc009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:21.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:21.831+0000 7f2cd8ff9700 1 -- 172.21.15.113:0/1542349716 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2cd005aba0 con 0x7f2cdc0fe270 2024-04-12T05:30:22.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.042+0000 7f2ce1b28700 1 -- 172.21.15.113:0/1542349716 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cdc0ffc60 con 0x7f2cc406ca30 2024-04-12T05:30:22.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.043+0000 7f2cd8ff9700 1 -- 172.21.15.113:0/1542349716 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2cdc0ffc60 con 0x7f2cc406ca30 2024-04-12T05:30:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.045+0000 7f2ce1b28700 1 -- 172.21.15.113:0/1542349716 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cc406ca30 msgr2=0x7f2cc406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.045+0000 7f2ce1b28700 1 --2- 172.21.15.113:0/1542349716 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cc406ca30 0x7f2cc406eef0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f2ccc009170 tx=0x7f2ccc009da0).stop 2024-04-12T05:30:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.045+0000 7f2ce1b28700 1 -- 172.21.15.113:0/1542349716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc0fe270 msgr2=0x7f2cdc10e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.045+0000 7f2ce1b28700 1 --2- 172.21.15.113:0/1542349716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc0fe270 0x7f2cdc10e4b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f2cd00099f0 tx=0x7f2cd0009cb0).stop 2024-04-12T05:30:22.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.046+0000 7f2ce1b28700 1 -- 172.21.15.113:0/1542349716 shutdown_connections 2024-04-12T05:30:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.046+0000 7f2ce1b28700 1 --2- 172.21.15.113:0/1542349716 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cc406ca30 0x7f2cc406eef0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.046+0000 7f2ce1b28700 1 --2- 172.21.15.113:0/1542349716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cdc0fd910 0x7f2cdc10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.046+0000 7f2ce1b28700 1 --2- 172.21.15.113:0/1542349716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cdc0fe270 0x7f2cdc10e4b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.046+0000 7f2ce1b28700 1 -- 172.21.15.113:0/1542349716 >> 172.21.15.113:0/1542349716 conn(0x7f2cdc0f3530 msgr2=0x7f2cdc0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.046+0000 7f2ce1b28700 1 -- 172.21.15.113:0/1542349716 shutdown_connections 2024-04-12T05:30:22.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.046+0000 7f2ce1b28700 1 -- 172.21.15.113:0/1542349716 wait complete. 2024-04-12T05:30:22.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.145+0000 7f37f883d700 1 -- 172.21.15.113:0/2535510133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37f40fd8f0 msgr2=0x7f37f40fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.145+0000 7f37f883d700 1 --2- 172.21.15.113:0/2535510133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37f40fd8f0 0x7f37f40fdcd0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f37e40047f0 tx=0x7f37e4009cb0).stop 2024-04-12T05:30:22.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.145+0000 7f37f883d700 1 -- 172.21.15.113:0/2535510133 shutdown_connections 2024-04-12T05:30:22.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.145+0000 7f37f883d700 1 --2- 172.21.15.113:0/2535510133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37f40fe250 0x7f37f40f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.145+0000 7f37f883d700 1 --2- 172.21.15.113:0/2535510133 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37f40fd8f0 0x7f37f40fdcd0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.145+0000 7f37f883d700 1 -- 172.21.15.113:0/2535510133 >> 172.21.15.113:0/2535510133 conn(0x7f37f40f34b0 msgr2=0x7f37f40f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:22.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.145+0000 7f37f883d700 1 -- 172.21.15.113:0/2535510133 shutdown_connections 2024-04-12T05:30:22.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.146+0000 7f37f883d700 1 -- 172.21.15.113:0/2535510133 wait complete. 2024-04-12T05:30:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.146+0000 7f37f883d700 1 Processor -- start 2024-04-12T05:30:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.146+0000 7f37f883d700 1 -- start start 2024-04-12T05:30:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.146+0000 7f37f883d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37f40fd8f0 0x7f37f410dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.146+0000 7f37f883d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37f40fe250 0x7f37f410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.146+0000 7f37f883d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f410ead0 con 0x7f37f40fe250 2024-04-12T05:30:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.146+0000 7f37f883d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f410ec10 con 0x7f37f40fd8f0 2024-04-12T05:30:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37f1d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37f40fe250 0x7f37f410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37f1d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37f40fe250 0x7f37f410e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37516/0 (socket says 172.21.15.113:37516) 2024-04-12T05:30:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37f1d9b700 1 -- 172.21.15.113:0/994340574 learned_addr learned my addr 172.21.15.113:0/994340574 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37f259c700 1 --2- 172.21.15.113:0/994340574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37f40fd8f0 0x7f37f410dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:22.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37f1d9b700 1 -- 172.21.15.113:0/994340574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37f40fd8f0 msgr2=0x7f37f410dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37f1d9b700 1 --2- 172.21.15.113:0/994340574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37f40fd8f0 0x7f37f410dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37f1d9b700 1 -- 172.21.15.113:0/994340574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e4005040 con 0x7f37f40fe250 2024-04-12T05:30:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37f1d9b700 1 --2- 172.21.15.113:0/994340574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37f40fe250 0x7f37f410e4d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f37dc002780 tx=0x7f37dc009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37eb7fe700 1 -- 172.21.15.113:0/994340574 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37dc007b40 con 0x7f37f40fe250 2024-04-12T05:30:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37f883d700 1 -- 172.21.15.113:0/994340574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f41082a0 con 0x7f37f40fe250 2024-04-12T05:30:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37eb7fe700 1 -- 172.21.15.113:0/994340574 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37dc00f040 con 0x7f37f40fe250 2024-04-12T05:30:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37eb7fe700 1 -- 172.21.15.113:0/994340574 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37dc014780 con 0x7f37f40fe250 2024-04-12T05:30:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.147+0000 7f37f883d700 1 -- 172.21.15.113:0/994340574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f4108860 con 0x7f37f40fe250 2024-04-12T05:30:22.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.148+0000 7f37f883d700 1 -- 172.21.15.113:0/994340574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f405f590 con 0x7f37f40fe250 2024-04-12T05:30:22.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.149+0000 7f37eb7fe700 1 -- 172.21.15.113:0/994340574 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f37dc007ca0 con 0x7f37f40fe250 2024-04-12T05:30:22.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.149+0000 7f37eb7fe700 1 --2- 172.21.15.113:0/994340574 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f37e006caa0 0x7f37e006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:22.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.149+0000 7f37eb7fe700 1 -- 172.21.15.113:0/994340574 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f37dc08bde0 con 0x7f37f40fe250 2024-04-12T05:30:22.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.149+0000 7f37f259c700 1 --2- 172.21.15.113:0/994340574 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f37e006caa0 0x7f37e006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:22.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.150+0000 7f37f259c700 1 --2- 172.21.15.113:0/994340574 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f37e006caa0 0x7f37e006ef60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f37e4004f10 tx=0x7f37e400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:22.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.154+0000 7f37eb7fe700 1 -- 172.21.15.113:0/994340574 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f37dc059890 con 0x7f37f40fe250 2024-04-12T05:30:22.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.352+0000 7f37f883d700 1 -- 172.21.15.113:0/994340574 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f37f4108ae0 con 0x7f37e006caa0 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.362+0000 7f37eb7fe700 1 -- 172.21.15.113:0/994340574 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f37f4108ae0 con 0x7f37e006caa0 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (18m) 11m ago 29m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (15m) 11m ago 29m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (15m) 10m ago 28m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (17m) 11m ago 29m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (11m) 11m ago 24m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (11m) 11m ago 24m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (10m) 10m ago 24m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (23m) 10m ago 23m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (19m) 11m ago 30m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (19m) 10m ago 28m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:30:22.363 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (16m) 11m ago 30m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:30:22.364 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (16m) 10m ago 28m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:30:22.364 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (18m) 11m ago 29m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:30:22.364 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (18m) 10m ago 28m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:30:22.364 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (15m) 11m ago 27m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:30:22.364 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (14m) 11m ago 27m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:30:22.364 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (14m) 11m ago 26m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:30:22.364 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (13m) 10m ago 26m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:30:22.364 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (12m) 10m ago 26m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:30:22.364 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (11m) 10m ago 25m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:30:22.364 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (18m) 10m ago 28m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:30:22.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.364+0000 7f37f883d700 1 -- 172.21.15.113:0/994340574 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f37e006caa0 msgr2=0x7f37e006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.364+0000 7f37f883d700 1 --2- 172.21.15.113:0/994340574 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f37e006caa0 0x7f37e006ef60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f37e4004f10 tx=0x7f37e400b040).stop 2024-04-12T05:30:22.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.364+0000 7f37f883d700 1 -- 172.21.15.113:0/994340574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37f40fe250 msgr2=0x7f37f410e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.364+0000 7f37f883d700 1 --2- 172.21.15.113:0/994340574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37f40fe250 0x7f37f410e4d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f37dc002780 tx=0x7f37dc009ee0).stop 2024-04-12T05:30:22.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.364+0000 7f37f883d700 1 -- 172.21.15.113:0/994340574 shutdown_connections 2024-04-12T05:30:22.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.364+0000 7f37f883d700 1 --2- 172.21.15.113:0/994340574 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f37e006caa0 0x7f37e006ef60 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.364+0000 7f37f883d700 1 --2- 172.21.15.113:0/994340574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37f40fd8f0 0x7f37f410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.364+0000 7f37f883d700 1 --2- 172.21.15.113:0/994340574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37f40fe250 0x7f37f410e4d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.364+0000 7f37f883d700 1 -- 172.21.15.113:0/994340574 >> 172.21.15.113:0/994340574 conn(0x7f37f40f34b0 msgr2=0x7f37f40fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:22.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.364+0000 7f37f883d700 1 -- 172.21.15.113:0/994340574 shutdown_connections 2024-04-12T05:30:22.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.364+0000 7f37f883d700 1 -- 172.21.15.113:0/994340574 wait complete. 2024-04-12T05:30:22.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:22.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:22.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.461+0000 7f3cc2757700 1 -- 172.21.15.113:0/4067633621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cbc0fe250 msgr2=0x7f3cbc0f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.461+0000 7f3cc2757700 1 --2- 172.21.15.113:0/4067633621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cbc0fe250 0x7f3cbc0f7a30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f3cb00047f0 tx=0x7f3cb0009cb0).stop 2024-04-12T05:30:22.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.461+0000 7f3cc2757700 1 -- 172.21.15.113:0/4067633621 shutdown_connections 2024-04-12T05:30:22.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.461+0000 7f3cc2757700 1 --2- 172.21.15.113:0/4067633621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cbc0fe250 0x7f3cbc0f7a30 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.461+0000 7f3cc2757700 1 --2- 172.21.15.113:0/4067633621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cbc0fd8f0 0x7f3cbc0fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.461+0000 7f3cc2757700 1 -- 172.21.15.113:0/4067633621 >> 172.21.15.113:0/4067633621 conn(0x7f3cbc0f34b0 msgr2=0x7f3cbc0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:22.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.461+0000 7f3cc2757700 1 -- 172.21.15.113:0/4067633621 shutdown_connections 2024-04-12T05:30:22.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.461+0000 7f3cc2757700 1 -- 172.21.15.113:0/4067633621 wait complete. 2024-04-12T05:30:22.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.462+0000 7f3cc2757700 1 Processor -- start 2024-04-12T05:30:22.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.462+0000 7f3cc2757700 1 -- start start 2024-04-12T05:30:22.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.462+0000 7f3cc2757700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cbc0fd8f0 0x7f3cbc10e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:22.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.462+0000 7f3cc2757700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cbc0fe250 0x7f3cbc10e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:22.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.462+0000 7f3cc2757700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc10eb70 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.462+0000 7f3cc2757700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc10ecb0 con 0x7f3cbc0fe250 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.462+0000 7f3cbbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cbc0fd8f0 0x7f3cbc10e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cbbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cbc0fd8f0 0x7f3cbc10e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37532/0 (socket says 172.21.15.113:37532) 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cbbfff700 1 -- 172.21.15.113:0/3650272973 learned_addr learned my addr 172.21.15.113:0/3650272973 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cbb7fe700 1 --2- 172.21.15.113:0/3650272973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cbc0fe250 0x7f3cbc10e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cbbfff700 1 -- 172.21.15.113:0/3650272973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cbc0fe250 msgr2=0x7f3cbc10e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cbbfff700 1 --2- 172.21.15.113:0/3650272973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cbc0fe250 0x7f3cbc10e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cbbfff700 1 -- 172.21.15.113:0/3650272973 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb0005040 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cbbfff700 1 --2- 172.21.15.113:0/3650272973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cbc0fd8f0 0x7f3cbc10e080 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f3cac00c740 tx=0x7f3cac007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cb97fa700 1 -- 172.21.15.113:0/3650272973 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cac00e680 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cb97fa700 1 -- 172.21.15.113:0/3650272973 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cac00ec90 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cb97fa700 1 -- 172.21.15.113:0/3650272973 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cac013d30 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cc2757700 1 -- 172.21.15.113:0/3650272973 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cbc108340 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.463+0000 7f3cc2757700 1 -- 172.21.15.113:0/3650272973 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cbc108880 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.464+0000 7f3cc2757700 1 -- 172.21.15.113:0/3650272973 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cbc05f590 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.465+0000 7f3cb97fa700 1 -- 172.21.15.113:0/3650272973 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3cac013410 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.465+0000 7f3cb97fa700 1 --2- 172.21.15.113:0/3650272973 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3ca406caa0 0x7f3ca406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:22.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.465+0000 7f3cb97fa700 1 -- 172.21.15.113:0/3650272973 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3cac08c050 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.465+0000 7f3cbb7fe700 1 --2- 172.21.15.113:0/3650272973 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3ca406caa0 0x7f3ca406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:22.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.465+0000 7f3cbb7fe700 1 --2- 172.21.15.113:0/3650272973 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3ca406caa0 0x7f3ca406ef60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f3cbc1099c0 tx=0x7f3cb000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:22.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.469+0000 7f3cb97fa700 1 -- 172.21.15.113:0/3650272973 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3cac059b00 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.733+0000 7f3cc2757700 1 -- 172.21.15.113:0/3650272973 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3cbc1097a0 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.734+0000 7f3cb97fa700 1 -- 172.21.15.113:0/3650272973 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f3cac059690 con 0x7f3cbc0fd8f0 2024-04-12T05:30:22.734 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:30:22.735 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:30:22.736 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:30:22.736 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:30:22.736 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:30:22.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.736+0000 7f3cc2757700 1 -- 172.21.15.113:0/3650272973 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3ca406caa0 msgr2=0x7f3ca406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.736+0000 7f3cc2757700 1 --2- 172.21.15.113:0/3650272973 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3ca406caa0 0x7f3ca406ef60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f3cbc1099c0 tx=0x7f3cb000b040).stop 2024-04-12T05:30:22.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.736+0000 7f3cc2757700 1 -- 172.21.15.113:0/3650272973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cbc0fd8f0 msgr2=0x7f3cbc10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.736+0000 7f3cc2757700 1 --2- 172.21.15.113:0/3650272973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cbc0fd8f0 0x7f3cbc10e080 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f3cac00c740 tx=0x7f3cac007ef0).stop 2024-04-12T05:30:22.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.736+0000 7f3cc2757700 1 -- 172.21.15.113:0/3650272973 shutdown_connections 2024-04-12T05:30:22.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.736+0000 7f3cc2757700 1 --2- 172.21.15.113:0/3650272973 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3ca406caa0 0x7f3ca406ef60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.736+0000 7f3cc2757700 1 --2- 172.21.15.113:0/3650272973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cbc0fd8f0 0x7f3cbc10e080 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.736+0000 7f3cc2757700 1 --2- 172.21.15.113:0/3650272973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cbc0fe250 0x7f3cbc10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.736+0000 7f3cc2757700 1 -- 172.21.15.113:0/3650272973 >> 172.21.15.113:0/3650272973 conn(0x7f3cbc0f34b0 msgr2=0x7f3cbc0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:22.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.736+0000 7f3cc2757700 1 -- 172.21.15.113:0/3650272973 shutdown_connections 2024-04-12T05:30:22.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.736+0000 7f3cc2757700 1 -- 172.21.15.113:0/3650272973 wait complete. 2024-04-12T05:30:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.835+0000 7fe7b28a9700 1 -- 172.21.15.113:0/3887389179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7ac0fd8f0 msgr2=0x7fe7ac0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.835+0000 7fe7b28a9700 1 --2- 172.21.15.113:0/3887389179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7ac0fd8f0 0x7fe7ac0fdcd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fe79c0047f0 tx=0x7fe79c009cb0).stop 2024-04-12T05:30:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.835+0000 7fe7b28a9700 1 -- 172.21.15.113:0/3887389179 shutdown_connections 2024-04-12T05:30:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.835+0000 7fe7b28a9700 1 --2- 172.21.15.113:0/3887389179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7ac0fe250 0x7fe7ac0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:30:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.835+0000 7fe7b28a9700 1 --2- 172.21.15.113:0/3887389179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7ac0fd8f0 0x7fe7ac0fdcd0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.835+0000 7fe7b28a9700 1 -- 172.21.15.113:0/3887389179 >> 172.21.15.113:0/3887389179 conn(0x7fe7ac0f34b0 msgr2=0x7fe7ac0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.835+0000 7fe7b28a9700 1 -- 172.21.15.113:0/3887389179 shutdown_connections 2024-04-12T05:30:22.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.835+0000 7fe7b28a9700 1 -- 172.21.15.113:0/3887389179 wait complete. 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.836+0000 7fe7b28a9700 1 Processor -- start 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.836+0000 7fe7b28a9700 1 -- start start 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.836+0000 7fe7b28a9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7ac0fd8f0 0x7fe7ac10e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.836+0000 7fe7b28a9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7ac0fe250 0x7fe7ac10e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.836+0000 7fe7b28a9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac10ec80 con 0x7fe7ac0fd8f0 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.836+0000 7fe7b28a9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac1082a0 con 0x7fe7ac0fe250 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.836+0000 7fe7abfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7ac0fd8f0 0x7fe7ac10e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.836+0000 7fe7abfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7ac0fd8f0 0x7fe7ac10e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37558/0 (socket says 172.21.15.113:37558) 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.836+0000 7fe7abfff700 1 -- 172.21.15.113:0/1780188200 learned_addr learned my addr 172.21.15.113:0/1780188200 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.836+0000 7fe7ab7fe700 1 --2- 172.21.15.113:0/1780188200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7ac0fe250 0x7fe7ac10e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.837+0000 7fe7abfff700 1 -- 172.21.15.113:0/1780188200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7ac0fe250 msgr2=0x7fe7ac10e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.837+0000 7fe7abfff700 1 --2- 172.21.15.113:0/1780188200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7ac0fe250 0x7fe7ac10e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:22.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.837+0000 7fe7abfff700 1 -- 172.21.15.113:0/1780188200 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79c005040 con 0x7fe7ac0fd8f0 2024-04-12T05:30:22.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.837+0000 7fe7abfff700 1 --2- 172.21.15.113:0/1780188200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7ac0fd8f0 0x7fe7ac10e100 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7fe79c013640 tx=0x7fe79c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:22.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.837+0000 7fe7a97fa700 1 -- 172.21.15.113:0/1780188200 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe79c00e030 con 0x7fe7ac0fd8f0 2024-04-12T05:30:22.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.837+0000 7fe7a97fa700 1 -- 172.21.15.113:0/1780188200 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe79c007e90 con 0x7fe7ac0fd8f0 2024-04-12T05:30:22.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.837+0000 7fe7a97fa700 1 -- 172.21.15.113:0/1780188200 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe79c018070 con 0x7fe7ac0fd8f0 2024-04-12T05:30:22.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.837+0000 7fe7b28a9700 1 -- 172.21.15.113:0/1780188200 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ac1084f0 con 0x7fe7ac0fd8f0 2024-04-12T05:30:22.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.837+0000 7fe7b28a9700 1 -- 172.21.15.113:0/1780188200 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ac1089d0 con 0x7fe7ac0fd8f0 2024-04-12T05:30:22.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.838+0000 7fe7b28a9700 1 -- 172.21.15.113:0/1780188200 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ac05f590 con 0x7fe7ac0fd8f0 2024-04-12T05:30:22.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.838+0000 7fe7a97fa700 1 -- 172.21.15.113:0/1780188200 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe79c01d420 con 0x7fe7ac0fd8f0 2024-04-12T05:30:22.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.839+0000 7fe7a97fa700 1 --2- 172.21.15.113:0/1780188200 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79406cab0 0x7fe79406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:22.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.839+0000 7fe7a97fa700 1 -- 172.21.15.113:0/1780188200 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe79c08d0f0 con 0x7fe7ac0fd8f0 2024-04-12T05:30:22.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.839+0000 7fe7ab7fe700 1 --2- 172.21.15.113:0/1780188200 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79406cab0 0x7fe79406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:22.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.839+0000 7fe7ab7fe700 1 --2- 172.21.15.113:0/1780188200 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79406cab0 0x7fe79406ef70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fe7ac109a40 tx=0x7fe7a0009060).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:22.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:22.843+0000 7fe7a97fa700 1 -- 172.21.15.113:0/1780188200 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe79c05aba0 con 0x7fe7ac0fd8f0 2024-04-12T05:30:23.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.073+0000 7fe7b28a9700 1 -- 172.21.15.113:0/1780188200 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe7ac047540 con 0x7fe7ac0fd8f0 2024-04-12T05:30:23.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.073+0000 7fe7a97fa700 1 -- 172.21.15.113:0/1780188200 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7fe79c05a730 con 0x7fe7ac0fd8f0 2024-04-12T05:30:23.074 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:30:23.074 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:30:23.074 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:30:23.074 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:30:23.075 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:30:23.076 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:30:23.077 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:30:23.077 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:30:23.077 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:30:23.077 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:30:23.077 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:30:23.077 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:30:23.077 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:30:23.077 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:30:23.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.075+0000 7fe7b28a9700 1 -- 172.21.15.113:0/1780188200 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79406cab0 msgr2=0x7fe79406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:23.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.075+0000 7fe7b28a9700 1 --2- 172.21.15.113:0/1780188200 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79406cab0 0x7fe79406ef70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fe7ac109a40 tx=0x7fe7a0009060).stop 2024-04-12T05:30:23.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.075+0000 7fe7b28a9700 1 -- 172.21.15.113:0/1780188200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7ac0fd8f0 msgr2=0x7fe7ac10e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:23.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.075+0000 7fe7b28a9700 1 --2- 172.21.15.113:0/1780188200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7ac0fd8f0 0x7fe7ac10e100 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7fe79c013640 tx=0x7fe79c009cb0).stop 2024-04-12T05:30:23.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.076+0000 7fe7b28a9700 1 -- 172.21.15.113:0/1780188200 shutdown_connections 2024-04-12T05:30:23.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.076+0000 7fe7b28a9700 1 --2- 172.21.15.113:0/1780188200 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79406cab0 0x7fe79406ef70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.076+0000 7fe7b28a9700 1 --2- 172.21.15.113:0/1780188200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7ac0fd8f0 0x7fe7ac10e100 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.076+0000 7fe7b28a9700 1 --2- 172.21.15.113:0/1780188200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7ac0fe250 0x7fe7ac10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.076+0000 7fe7b28a9700 1 -- 172.21.15.113:0/1780188200 >> 172.21.15.113:0/1780188200 conn(0x7fe7ac0f34b0 msgr2=0x7fe7ac0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:23.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.076+0000 7fe7b28a9700 1 -- 172.21.15.113:0/1780188200 shutdown_connections 2024-04-12T05:30:23.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.076+0000 7fe7b28a9700 1 -- 172.21.15.113:0/1780188200 wait complete. 2024-04-12T05:30:23.079 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:30:23.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.175+0000 7eff4f750700 1 -- 172.21.15.113:0/1704185135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff480fd8f0 msgr2=0x7eff480fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:23.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.175+0000 7eff4f750700 1 --2- 172.21.15.113:0/1704185135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff480fd8f0 0x7eff480fdcd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7eff380047f0 tx=0x7eff38009cb0).stop 2024-04-12T05:30:23.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.175+0000 7eff4f750700 1 -- 172.21.15.113:0/1704185135 shutdown_connections 2024-04-12T05:30:23.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.175+0000 7eff4f750700 1 --2- 172.21.15.113:0/1704185135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff480fe250 0x7eff480f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.175+0000 7eff4f750700 1 --2- 172.21.15.113:0/1704185135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff480fd8f0 0x7eff480fdcd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.175+0000 7eff4f750700 1 -- 172.21.15.113:0/1704185135 >> 172.21.15.113:0/1704185135 conn(0x7eff480f34b0 msgr2=0x7eff480f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:23.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.175+0000 7eff4f750700 1 -- 172.21.15.113:0/1704185135 shutdown_connections 2024-04-12T05:30:23.176 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.175+0000 7eff4f750700 1 -- 172.21.15.113:0/1704185135 wait complete. 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.176+0000 7eff4f750700 1 Processor -- start 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.176+0000 7eff4f750700 1 -- start start 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.176+0000 7eff4f750700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff480fd8f0 0x7eff4810e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.176+0000 7eff4f750700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff480fe250 0x7eff4810e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.176+0000 7eff4f750700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4810eaf0 con 0x7eff480fd8f0 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.176+0000 7eff4f750700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4810ec30 con 0x7eff480fe250 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.176+0000 7eff4d4ec700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff480fd8f0 0x7eff4810e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.176+0000 7eff4d4ec700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff480fd8f0 0x7eff4810e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37580/0 (socket says 172.21.15.113:37580) 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff4d4ec700 1 -- 172.21.15.113:0/352318464 learned_addr learned my addr 172.21.15.113:0/352318464 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff4cceb700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff480fe250 0x7eff4810e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff4d4ec700 1 -- 172.21.15.113:0/352318464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff480fe250 msgr2=0x7eff4810e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff4d4ec700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff480fe250 0x7eff4810e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff4d4ec700 1 -- 172.21.15.113:0/352318464 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff38005040 con 0x7eff480fd8f0 2024-04-12T05:30:23.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff4cceb700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff480fe250 0x7eff4810e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:30:23.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff4d4ec700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff480fd8f0 0x7eff4810e000 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7eff38005010 tx=0x7eff38009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:23.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff3e7fc700 1 -- 172.21.15.113:0/352318464 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff3800e030 con 0x7eff480fd8f0 2024-04-12T05:30:23.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff3e7fc700 1 -- 172.21.15.113:0/352318464 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff38007e90 con 0x7eff480fd8f0 2024-04-12T05:30:23.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff4f750700 1 -- 172.21.15.113:0/352318464 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff48108260 con 0x7eff480fd8f0 2024-04-12T05:30:23.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff3e7fc700 1 -- 172.21.15.113:0/352318464 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff38018070 con 0x7eff480fd8f0 2024-04-12T05:30:23.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.177+0000 7eff4f750700 1 -- 172.21.15.113:0/352318464 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff481087c0 con 0x7eff480fd8f0 2024-04-12T05:30:23.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.178+0000 7eff4f750700 1 -- 172.21.15.113:0/352318464 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4805f590 con 0x7eff480fd8f0 2024-04-12T05:30:23.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.179+0000 7eff3e7fc700 1 -- 172.21.15.113:0/352318464 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7eff3801d6e0 con 0x7eff480fd8f0 2024-04-12T05:30:23.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.179+0000 7eff3e7fc700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff3406ca30 0x7eff3406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:23.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.179+0000 7eff3e7fc700 1 -- 172.21.15.113:0/352318464 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7eff3808d100 con 0x7eff480fd8f0 2024-04-12T05:30:23.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.179+0000 7eff4cceb700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff3406ca30 0x7eff3406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:23.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.179+0000 7eff4cceb700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff3406ca30 0x7eff3406eef0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7eff48109940 tx=0x7eff44009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:23.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.183+0000 7eff3e7fc700 1 -- 172.21.15.113:0/352318464 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7eff3805a920 con 0x7eff480fd8f0 2024-04-12T05:30:23.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.392+0000 7eff4f750700 1 -- 172.21.15.113:0/352318464 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7eff480ffc40 con 0x7eff3406ca30 2024-04-12T05:30:23.394 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.393+0000 7eff3e7fc700 1 -- 172.21.15.113:0/352318464 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7eff480ffc40 con 0x7eff3406ca30 2024-04-12T05:30:23.394 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:30:23.394 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:30:23.394 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:30:23.394 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:30:23.394 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:30:23.395 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:30:23.395 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:30:23.395 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:30:23.395 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:30:23.395 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:30:23.395 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:30:23.395 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:30:23.395 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:30:23.395 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:30:23.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.395+0000 7eff4f750700 1 -- 172.21.15.113:0/352318464 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff3406ca30 msgr2=0x7eff3406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:23.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.395+0000 7eff4f750700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff3406ca30 0x7eff3406eef0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7eff48109940 tx=0x7eff44009da0).stop 2024-04-12T05:30:23.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.395+0000 7eff4f750700 1 -- 172.21.15.113:0/352318464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff480fd8f0 msgr2=0x7eff4810e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:23.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.395+0000 7eff4f750700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff480fd8f0 0x7eff4810e000 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7eff38005010 tx=0x7eff38009cb0).stop 2024-04-12T05:30:23.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.396+0000 7eff4f750700 1 -- 172.21.15.113:0/352318464 shutdown_connections 2024-04-12T05:30:23.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.396+0000 7eff4f750700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff3406ca30 0x7eff3406eef0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.396+0000 7eff4f750700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff480fd8f0 0x7eff4810e000 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.396+0000 7eff4f750700 1 --2- 172.21.15.113:0/352318464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff480fe250 0x7eff4810e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.396+0000 7eff4f750700 1 -- 172.21.15.113:0/352318464 >> 172.21.15.113:0/352318464 conn(0x7eff480f34b0 msgr2=0x7eff480fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:23.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.396+0000 7eff4f750700 1 -- 172.21.15.113:0/352318464 shutdown_connections 2024-04-12T05:30:23.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.396+0000 7eff4f750700 1 -- 172.21.15.113:0/352318464 wait complete. 2024-04-12T05:30:23.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:23 smithi113 ceph-mon[103371]: from='client.35008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:23.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:23 smithi113 ceph-mon[103371]: pgmap v546: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:23.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:23 smithi113 ceph-mon[103371]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:23.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:23 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3650272973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:30:23.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:23 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1780188200' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:30:23.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:23 smithi192 ceph-mon[85304]: from='client.35008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:23.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:23 smithi192 ceph-mon[85304]: pgmap v546: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:23.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:23 smithi192 ceph-mon[85304]: from='client.35012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:23.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:23 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3650272973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:30:23.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:23 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1780188200' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:30:23.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.493+0000 7f5b44fb9700 1 -- 172.21.15.113:0/12740999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b400f9200 msgr2=0x7f5b40101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:23.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.493+0000 7f5b44fb9700 1 --2- 172.21.15.113:0/12740999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b400f9200 0x7f5b40101910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f5b340047f0 tx=0x7f5b34009cb0).stop 2024-04-12T05:30:23.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.494+0000 7f5b44fb9700 1 -- 172.21.15.113:0/12740999 shutdown_connections 2024-04-12T05:30:23.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.494+0000 7f5b44fb9700 1 --2- 172.21.15.113:0/12740999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b400f9200 0x7f5b40101910 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.494+0000 7f5b44fb9700 1 --2- 172.21.15.113:0/12740999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b400f88a0 0x7f5b400f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.494+0000 7f5b44fb9700 1 -- 172.21.15.113:0/12740999 >> 172.21.15.113:0/12740999 conn(0x7f5b40070590 msgr2=0x7f5b400f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:23.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.494+0000 7f5b44fb9700 1 -- 172.21.15.113:0/12740999 shutdown_connections 2024-04-12T05:30:23.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.494+0000 7f5b44fb9700 1 -- 172.21.15.113:0/12740999 wait complete. 2024-04-12T05:30:23.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.494+0000 7f5b44fb9700 1 Processor -- start 2024-04-12T05:30:23.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.494+0000 7f5b44fb9700 1 -- start start 2024-04-12T05:30:23.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b44fb9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b400f88a0 0x7f5b4010e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b44fb9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b400f9200 0x7f5b4010e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b44fb9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4010eb00 con 0x7f5b400f88a0 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b44fb9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4010ec40 con 0x7f5b400f9200 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b3e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b400f88a0 0x7f5b4010e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b3e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b400f88a0 0x7f5b4010e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37604/0 (socket says 172.21.15.113:37604) 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b3e59c700 1 -- 172.21.15.113:0/1930322110 learned_addr learned my addr 172.21.15.113:0/1930322110 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b3dd9b700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b400f9200 0x7f5b4010e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b3e59c700 1 -- 172.21.15.113:0/1930322110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b400f9200 msgr2=0x7f5b4010e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b3e59c700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b400f9200 0x7f5b4010e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b3e59c700 1 -- 172.21.15.113:0/1930322110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b34005040 con 0x7f5b400f88a0 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b3dd9b700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b400f9200 0x7f5b4010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b3e59c700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b400f88a0 0x7f5b4010e010 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f5b3000c740 tx=0x7f5b30007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b2f7fe700 1 -- 172.21.15.113:0/1930322110 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b3000e680 con 0x7f5b400f88a0 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b44fb9700 1 -- 172.21.15.113:0/1930322110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b40108340 con 0x7f5b400f88a0 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b2f7fe700 1 -- 172.21.15.113:0/1930322110 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b3000ec90 con 0x7f5b400f88a0 2024-04-12T05:30:23.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b2f7fe700 1 -- 172.21.15.113:0/1930322110 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b30013d60 con 0x7f5b400f88a0 2024-04-12T05:30:23.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.495+0000 7f5b44fb9700 1 -- 172.21.15.113:0/1930322110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b40108880 con 0x7f5b400f88a0 2024-04-12T05:30:23.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.496+0000 7f5b44fb9700 1 -- 172.21.15.113:0/1930322110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4005f590 con 0x7f5b400f88a0 2024-04-12T05:30:23.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.497+0000 7f5b2f7fe700 1 -- 172.21.15.113:0/1930322110 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5b3000e7e0 con 0x7f5b400f88a0 2024-04-12T05:30:23.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.497+0000 7f5b2f7fe700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b2806caa0 0x7f5b2806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:23.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.497+0000 7f5b2f7fe700 1 -- 172.21.15.113:0/1930322110 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5b3008c1a0 con 0x7f5b400f88a0 2024-04-12T05:30:23.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.497+0000 7f5b3dd9b700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b2806caa0 0x7f5b2806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:23.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.498+0000 7f5b3dd9b700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b2806caa0 0x7f5b2806ef60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f5b34007cf0 tx=0x7f5b3400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:23.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.502+0000 7f5b2f7fe700 1 -- 172.21.15.113:0/1930322110 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5b30059c50 con 0x7f5b400f88a0 2024-04-12T05:30:23.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.771+0000 7f5b44fb9700 1 -- 172.21.15.113:0/1930322110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5b40047540 con 0x7f5b400f88a0 2024-04-12T05:30:23.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.771+0000 7f5b2f7fe700 1 -- 172.21.15.113:0/1930322110 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f5b300597e0 con 0x7f5b400f88a0 2024-04-12T05:30:23.772 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:30:23.772 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:30:23.772 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:30:23.772 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:30:23.772 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:30:23.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.773+0000 7f5b44fb9700 1 -- 172.21.15.113:0/1930322110 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b2806caa0 msgr2=0x7f5b2806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:23.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.773+0000 7f5b44fb9700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b2806caa0 0x7f5b2806ef60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f5b34007cf0 tx=0x7f5b3400b040).stop 2024-04-12T05:30:23.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.773+0000 7f5b44fb9700 1 -- 172.21.15.113:0/1930322110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b400f88a0 msgr2=0x7f5b4010e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:23.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.773+0000 7f5b44fb9700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b400f88a0 0x7f5b4010e010 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f5b3000c740 tx=0x7f5b30007ef0).stop 2024-04-12T05:30:23.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.773+0000 7f5b44fb9700 1 -- 172.21.15.113:0/1930322110 shutdown_connections 2024-04-12T05:30:23.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.773+0000 7f5b44fb9700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b2806caa0 0x7f5b2806ef60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.773+0000 7f5b44fb9700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b400f88a0 0x7f5b4010e010 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.773+0000 7f5b44fb9700 1 --2- 172.21.15.113:0/1930322110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b400f9200 0x7f5b4010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:23.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.773+0000 7f5b44fb9700 1 -- 172.21.15.113:0/1930322110 >> 172.21.15.113:0/1930322110 conn(0x7f5b40070590 msgr2=0x7f5b400fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:23.774 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.773+0000 7f5b44fb9700 1 -- 172.21.15.113:0/1930322110 shutdown_connections 2024-04-12T05:30:23.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:23.773+0000 7f5b44fb9700 1 -- 172.21.15.113:0/1930322110 wait complete. 2024-04-12T05:30:24.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:24 smithi113 ceph-mon[103371]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:24.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:24 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1930322110' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:30:24.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:24 smithi192 ceph-mon[85304]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:24.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:24 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1930322110' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:30:25.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:25 smithi113 ceph-mon[103371]: from='client.35028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:25.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:25 smithi113 ceph-mon[103371]: pgmap v547: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:25.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:25 smithi192 ceph-mon[85304]: from='client.35028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:25.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:25 smithi192 ceph-mon[85304]: pgmap v547: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:27.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:27 smithi113 ceph-mon[103371]: pgmap v548: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:27.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:27 smithi192 ceph-mon[85304]: pgmap v548: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:29.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:29 smithi113 ceph-mon[103371]: pgmap v549: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:29.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:29 smithi192 ceph-mon[85304]: pgmap v549: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:31.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:31 smithi113 ceph-mon[103371]: pgmap v550: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:31.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:31 smithi192 ceph-mon[85304]: pgmap v550: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:33.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:33 smithi113 ceph-mon[103371]: pgmap v551: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:33.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:33 smithi192 ceph-mon[85304]: pgmap v551: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:35.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:35 smithi113 ceph-mon[103371]: pgmap v552: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:35.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:35 smithi192 ceph-mon[85304]: pgmap v552: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:37.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:37 smithi113 ceph-mon[103371]: pgmap v553: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:37.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:37.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:37 smithi192 ceph-mon[85304]: pgmap v553: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:37.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:39.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:39 smithi113 ceph-mon[103371]: pgmap v554: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:39.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:39 smithi192 ceph-mon[85304]: pgmap v554: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:41.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:41 smithi113 ceph-mon[103371]: pgmap v555: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:41.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:41 smithi192 ceph-mon[85304]: pgmap v555: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:43.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:43 smithi113 ceph-mon[103371]: pgmap v556: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:43.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:43 smithi192 ceph-mon[85304]: pgmap v556: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:45.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:45 smithi113 ceph-mon[103371]: pgmap v557: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:45.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:45 smithi192 ceph-mon[85304]: pgmap v557: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:47.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:47 smithi113 ceph-mon[103371]: pgmap v558: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:47.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:47 smithi192 ceph-mon[85304]: pgmap v558: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:49.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:49 smithi113 ceph-mon[103371]: pgmap v559: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:49.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:49 smithi192 ceph-mon[85304]: pgmap v559: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:51.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:51 smithi113 ceph-mon[103371]: pgmap v560: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:51.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:51 smithi192 ceph-mon[85304]: pgmap v560: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:52.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:52.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:30:53.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:53 smithi113 ceph-mon[103371]: pgmap v561: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:53.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:53 smithi192 ceph-mon[85304]: pgmap v561: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:53.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.884+0000 7ff4308ef700 1 -- 172.21.15.113:0/4054362837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff42c0f9200 msgr2=0x7ff42c101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:53.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.884+0000 7ff4308ef700 1 --2- 172.21.15.113:0/4054362837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff42c0f9200 0x7ff42c101910 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7ff4200047f0 tx=0x7ff420009cb0).stop 2024-04-12T05:30:53.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.885+0000 7ff4308ef700 1 -- 172.21.15.113:0/4054362837 shutdown_connections 2024-04-12T05:30:53.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.885+0000 7ff4308ef700 1 --2- 172.21.15.113:0/4054362837 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff42c0f9200 0x7ff42c101910 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:53.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.885+0000 7ff4308ef700 1 --2- 172.21.15.113:0/4054362837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff42c0f88a0 0x7ff42c0f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:53.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.885+0000 7ff4308ef700 1 -- 172.21.15.113:0/4054362837 >> 172.21.15.113:0/4054362837 conn(0x7ff42c070590 msgr2=0x7ff42c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:53.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.885+0000 7ff4308ef700 1 -- 172.21.15.113:0/4054362837 shutdown_connections 2024-04-12T05:30:53.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.885+0000 7ff4308ef700 1 -- 172.21.15.113:0/4054362837 wait complete. 2024-04-12T05:30:53.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.885+0000 7ff4308ef700 1 Processor -- start 2024-04-12T05:30:53.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.885+0000 7ff4308ef700 1 -- start start 2024-04-12T05:30:53.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff4308ef700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff42c0f88a0 0x7ff42c10e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:53.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff4308ef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff42c0f9200 0x7ff42c10e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:53.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff4308ef700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c10eb80 con 0x7ff42c0f88a0 2024-04-12T05:30:53.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff4308ef700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c10ecc0 con 0x7ff42c0f9200 2024-04-12T05:30:53.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff42a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff42c0f88a0 0x7ff42c10e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:53.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff42a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff42c0f88a0 0x7ff42c10e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39910/0 (socket says 172.21.15.113:39910) 2024-04-12T05:30:53.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff42a59c700 1 -- 172.21.15.113:0/1459623419 learned_addr learned my addr 172.21.15.113:0/1459623419 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:53.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff429d9b700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff42c0f9200 0x7ff42c10e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:53.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff42a59c700 1 -- 172.21.15.113:0/1459623419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff42c0f9200 msgr2=0x7ff42c10e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:53.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff42a59c700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff42c0f9200 0x7ff42c10e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:53.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff42a59c700 1 -- 172.21.15.113:0/1459623419 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff420005040 con 0x7ff42c0f88a0 2024-04-12T05:30:53.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff429d9b700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff42c0f9200 0x7ff42c10e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:30:53.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff42a59c700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff42c0f88a0 0x7ff42c10e090 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7ff41c0041d0 tx=0x7ff41c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:53.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.886+0000 7ff41b7fe700 1 -- 172.21.15.113:0/1459623419 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41c007b40 con 0x7ff42c0f88a0 2024-04-12T05:30:53.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.887+0000 7ff4308ef700 1 -- 172.21.15.113:0/1459623419 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42c108350 con 0x7ff42c0f88a0 2024-04-12T05:30:53.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.887+0000 7ff41b7fe700 1 -- 172.21.15.113:0/1459623419 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff41c00f040 con 0x7ff42c0f88a0 2024-04-12T05:30:53.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.887+0000 7ff41b7fe700 1 -- 172.21.15.113:0/1459623419 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41c00d800 con 0x7ff42c0f88a0 2024-04-12T05:30:53.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.887+0000 7ff4308ef700 1 -- 172.21.15.113:0/1459623419 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42c108890 con 0x7ff42c0f88a0 2024-04-12T05:30:53.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.887+0000 7ff4308ef700 1 -- 172.21.15.113:0/1459623419 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42c05f590 con 0x7ff42c0f88a0 2024-04-12T05:30:53.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.888+0000 7ff41b7fe700 1 -- 172.21.15.113:0/1459623419 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff41c007ca0 con 0x7ff42c0f88a0 2024-04-12T05:30:53.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.888+0000 7ff41b7fe700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff41406caa0 0x7ff41406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:53.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.889+0000 7ff41b7fe700 1 -- 172.21.15.113:0/1459623419 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff41c08ba10 con 0x7ff42c0f88a0 2024-04-12T05:30:53.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.889+0000 7ff429d9b700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff41406caa0 0x7ff41406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:53.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.889+0000 7ff429d9b700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff41406caa0 0x7ff41406ef60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7ff4200043d0 tx=0x7ff42000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:53.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:53.893+0000 7ff41b7fe700 1 -- 172.21.15.113:0/1459623419 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff41c059400 con 0x7ff42c0f88a0 2024-04-12T05:30:54.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.104+0000 7ff4308ef700 1 -- 172.21.15.113:0/1459623419 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff42c108b10 con 0x7ff41406caa0 2024-04-12T05:30:54.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.105+0000 7ff41b7fe700 1 -- 172.21.15.113:0/1459623419 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff42c108b10 con 0x7ff41406caa0 2024-04-12T05:30:54.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.107+0000 7ff4308ef700 1 -- 172.21.15.113:0/1459623419 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff41406caa0 msgr2=0x7ff41406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.107+0000 7ff4308ef700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff41406caa0 0x7ff41406ef60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7ff4200043d0 tx=0x7ff42000b040).stop 2024-04-12T05:30:54.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.107+0000 7ff4308ef700 1 -- 172.21.15.113:0/1459623419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff42c0f88a0 msgr2=0x7ff42c10e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.107+0000 7ff4308ef700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff42c0f88a0 0x7ff42c10e090 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7ff41c0041d0 tx=0x7ff41c009f60).stop 2024-04-12T05:30:54.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.107+0000 7ff4308ef700 1 -- 172.21.15.113:0/1459623419 shutdown_connections 2024-04-12T05:30:54.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.107+0000 7ff4308ef700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff41406caa0 0x7ff41406ef60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.107+0000 7ff4308ef700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff42c0f88a0 0x7ff42c10e090 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.107+0000 7ff4308ef700 1 --2- 172.21.15.113:0/1459623419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff42c0f9200 0x7ff42c10e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.107+0000 7ff4308ef700 1 -- 172.21.15.113:0/1459623419 >> 172.21.15.113:0/1459623419 conn(0x7ff42c070590 msgr2=0x7ff42c0fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:54.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.107+0000 7ff4308ef700 1 -- 172.21.15.113:0/1459623419 shutdown_connections 2024-04-12T05:30:54.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.107+0000 7ff4308ef700 1 -- 172.21.15.113:0/1459623419 wait complete. 2024-04-12T05:30:54.118 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:30:54.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.207+0000 7fccd46e0700 1 -- 172.21.15.113:0/3709158138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcccc0fc7f0 msgr2=0x7fcccc0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.207+0000 7fccd46e0700 1 --2- 172.21.15.113:0/3709158138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcccc0fc7f0 0x7fcccc0fcbd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fccc80047f0 tx=0x7fccc8009cb0).stop 2024-04-12T05:30:54.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.207+0000 7fccd46e0700 1 -- 172.21.15.113:0/3709158138 shutdown_connections 2024-04-12T05:30:54.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.207+0000 7fccd46e0700 1 --2- 172.21.15.113:0/3709158138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcccc0fd150 0x7fcccc0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:30:54.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.207+0000 7fccd46e0700 1 --2- 172.21.15.113:0/3709158138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcccc0fc7f0 0x7fcccc0fcbd0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.207+0000 7fccd46e0700 1 -- 172.21.15.113:0/3709158138 >> 172.21.15.113:0/3709158138 conn(0x7fcccc070590 msgr2=0x7fcccc0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:54.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.207+0000 7fccd46e0700 1 -- 172.21.15.113:0/3709158138 shutdown_connections 2024-04-12T05:30:54.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.207+0000 7fccd46e0700 1 -- 172.21.15.113:0/3709158138 wait complete. 2024-04-12T05:30:54.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.208+0000 7fccd46e0700 1 Processor -- start 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.208+0000 7fccd46e0700 1 -- start start 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.208+0000 7fccd46e0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcccc0fc7f0 0x7fcccc10e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.208+0000 7fccd46e0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcccc0fd150 0x7fcccc10e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.208+0000 7fccd46e0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc10ebe0 con 0x7fcccc0fd150 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.208+0000 7fccd46e0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc108200 con 0x7fcccc0fc7f0 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.208+0000 7fccd1c7b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcccc0fd150 0x7fcccc10e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.208+0000 7fccd1c7b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcccc0fd150 0x7fcccc10e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39934/0 (socket says 172.21.15.113:39934) 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.208+0000 7fccd1c7b700 1 -- 172.21.15.113:0/3487859138 learned_addr learned my addr 172.21.15.113:0/3487859138 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.208+0000 7fccd247c700 1 --2- 172.21.15.113:0/3487859138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcccc0fc7f0 0x7fcccc10e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.209+0000 7fccd1c7b700 1 -- 172.21.15.113:0/3487859138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcccc0fc7f0 msgr2=0x7fcccc10e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.209+0000 7fccd1c7b700 1 --2- 172.21.15.113:0/3487859138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcccc0fc7f0 0x7fcccc10e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.209+0000 7fccd1c7b700 1 -- 172.21.15.113:0/3487859138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc8005040 con 0x7fcccc0fd150 2024-04-12T05:30:54.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.209+0000 7fccd1c7b700 1 --2- 172.21.15.113:0/3487859138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcccc0fd150 0x7fcccc10e550 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fccbc00b480 tx=0x7fccbc0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:54.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.209+0000 7fccc37fe700 1 -- 172.21.15.113:0/3487859138 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccbc002500 con 0x7fcccc0fd150 2024-04-12T05:30:54.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.209+0000 7fccc37fe700 1 -- 172.21.15.113:0/3487859138 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccbc002b10 con 0x7fcccc0fd150 2024-04-12T05:30:54.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.209+0000 7fccc37fe700 1 -- 172.21.15.113:0/3487859138 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccbc012c90 con 0x7fcccc0fd150 2024-04-12T05:30:54.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.209+0000 7fccd46e0700 1 -- 172.21.15.113:0/3487859138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc1084b0 con 0x7fcccc0fd150 2024-04-12T05:30:54.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.209+0000 7fccd46e0700 1 -- 172.21.15.113:0/3487859138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc1089f0 con 0x7fcccc0fd150 2024-04-12T05:30:54.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.210+0000 7fccd46e0700 1 -- 172.21.15.113:0/3487859138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc05f590 con 0x7fcccc0fd150 2024-04-12T05:30:54.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.210+0000 7fccc37fe700 1 -- 172.21.15.113:0/3487859138 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fccbc002660 con 0x7fcccc0fd150 2024-04-12T05:30:54.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.211+0000 7fccc37fe700 1 --2- 172.21.15.113:0/3487859138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fccb806caa0 0x7fccb806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:54.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.211+0000 7fccc37fe700 1 -- 172.21.15.113:0/3487859138 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fccbc08be50 con 0x7fcccc0fd150 2024-04-12T05:30:54.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.211+0000 7fccd247c700 1 --2- 172.21.15.113:0/3487859138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fccb806caa0 0x7fccb806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:54.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.211+0000 7fccd247c700 1 --2- 172.21.15.113:0/3487859138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fccb806caa0 0x7fccb806ef60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fccc8004f10 tx=0x7fccc800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:54.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.215+0000 7fccc37fe700 1 -- 172.21.15.113:0/3487859138 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fccbc059900 con 0x7fcccc0fd150 2024-04-12T05:30:54.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.419+0000 7fccd46e0700 1 -- 172.21.15.113:0/3487859138 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcccc1096e0 con 0x7fccb806caa0 2024-04-12T05:30:54.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.420+0000 7fccc37fe700 1 -- 172.21.15.113:0/3487859138 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcccc1096e0 con 0x7fccb806caa0 2024-04-12T05:30:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.422+0000 7fccd46e0700 1 -- 172.21.15.113:0/3487859138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fccb806caa0 msgr2=0x7fccb806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.422+0000 7fccd46e0700 1 --2- 172.21.15.113:0/3487859138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fccb806caa0 0x7fccb806ef60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fccc8004f10 tx=0x7fccc800b040).stop 2024-04-12T05:30:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.422+0000 7fccd46e0700 1 -- 172.21.15.113:0/3487859138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcccc0fd150 msgr2=0x7fcccc10e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.422+0000 7fccd46e0700 1 --2- 172.21.15.113:0/3487859138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcccc0fd150 0x7fcccc10e550 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fccbc00b480 tx=0x7fccbc0090f0).stop 2024-04-12T05:30:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.422+0000 7fccd46e0700 1 -- 172.21.15.113:0/3487859138 shutdown_connections 2024-04-12T05:30:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.422+0000 7fccd46e0700 1 --2- 172.21.15.113:0/3487859138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fccb806caa0 0x7fccb806ef60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.422+0000 7fccd46e0700 1 --2- 172.21.15.113:0/3487859138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcccc0fc7f0 0x7fcccc10e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.422+0000 7fccd46e0700 1 --2- 172.21.15.113:0/3487859138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcccc0fd150 0x7fcccc10e550 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.422+0000 7fccd46e0700 1 -- 172.21.15.113:0/3487859138 >> 172.21.15.113:0/3487859138 conn(0x7fcccc070590 msgr2=0x7fcccc060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.423+0000 7fccd46e0700 1 -- 172.21.15.113:0/3487859138 shutdown_connections 2024-04-12T05:30:54.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.423+0000 7fccd46e0700 1 -- 172.21.15.113:0/3487859138 wait complete. 2024-04-12T05:30:54.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.520+0000 7f350e0aa700 1 -- 172.21.15.113:0/980410256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35080fd910 msgr2=0x7f35080fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.520+0000 7f350e0aa700 1 --2- 172.21.15.113:0/980410256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35080fd910 0x7f35080fdcf0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f34fc0047f0 tx=0x7f34fc009cb0).stop 2024-04-12T05:30:54.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.520+0000 7f350e0aa700 1 -- 172.21.15.113:0/980410256 shutdown_connections 2024-04-12T05:30:54.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.520+0000 7f350e0aa700 1 --2- 172.21.15.113:0/980410256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35080fe270 0x7f35080f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.520+0000 7f350e0aa700 1 --2- 172.21.15.113:0/980410256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35080fd910 0x7f35080fdcf0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.520+0000 7f350e0aa700 1 -- 172.21.15.113:0/980410256 >> 172.21.15.113:0/980410256 conn(0x7f35080f3530 msgr2=0x7f35080f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:54.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.520+0000 7f350e0aa700 1 -- 172.21.15.113:0/980410256 shutdown_connections 2024-04-12T05:30:54.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.520+0000 7f350e0aa700 1 -- 172.21.15.113:0/980410256 wait complete. 2024-04-12T05:30:54.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.521+0000 7f350e0aa700 1 Processor -- start 2024-04-12T05:30:54.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.521+0000 7f350e0aa700 1 -- start start 2024-04-12T05:30:54.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.521+0000 7f350e0aa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35080fe270 0x7f350810e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:54.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.521+0000 7f350e0aa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f350810e6c0 0x7f3508108370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:54.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.521+0000 7f350e0aa700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350810ec50 con 0x7f350810e6c0 2024-04-12T05:30:54.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.521+0000 7f350e0aa700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350810ed90 con 0x7f35080fe270 2024-04-12T05:30:54.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.521+0000 7f3506ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f350810e6c0 0x7f3508108370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.521+0000 7f3506ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f350810e6c0 0x7f3508108370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39948/0 (socket says 172.21.15.113:39948) 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.521+0000 7f3506ffd700 1 -- 172.21.15.113:0/1315526343 learned_addr learned my addr 172.21.15.113:0/1315526343 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.521+0000 7f35077fe700 1 --2- 172.21.15.113:0/1315526343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35080fe270 0x7f350810e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.522+0000 7f3506ffd700 1 -- 172.21.15.113:0/1315526343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35080fe270 msgr2=0x7f350810e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.522+0000 7f3506ffd700 1 --2- 172.21.15.113:0/1315526343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35080fe270 0x7f350810e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.522+0000 7f3506ffd700 1 -- 172.21.15.113:0/1315526343 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc005040 con 0x7f350810e6c0 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.522+0000 7f3506ffd700 1 --2- 172.21.15.113:0/1315526343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f350810e6c0 0x7f3508108370 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f34f8002820 tx=0x7f34f8009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.522+0000 7f3504ff9700 1 -- 172.21.15.113:0/1315526343 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34f8007b40 con 0x7f350810e6c0 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.522+0000 7f3504ff9700 1 -- 172.21.15.113:0/1315526343 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34f800f040 con 0x7f350810e6c0 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.522+0000 7f350e0aa700 1 -- 172.21.15.113:0/1315526343 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35081088c0 con 0x7f350810e6c0 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.522+0000 7f3504ff9700 1 -- 172.21.15.113:0/1315526343 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34f80148b0 con 0x7f350810e6c0 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.522+0000 7f350e0aa700 1 -- 172.21.15.113:0/1315526343 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3508108e80 con 0x7f350810e6c0 2024-04-12T05:30:54.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.523+0000 7f350e0aa700 1 -- 172.21.15.113:0/1315526343 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350805f570 con 0x7f350810e6c0 2024-04-12T05:30:54.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.524+0000 7f3504ff9700 1 -- 172.21.15.113:0/1315526343 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f34f8007ca0 con 0x7f350810e6c0 2024-04-12T05:30:54.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.524+0000 7f3504ff9700 1 --2- 172.21.15.113:0/1315526343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f34f006cab0 0x7f34f006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:54.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.524+0000 7f3504ff9700 1 -- 172.21.15.113:0/1315526343 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f34f808bb70 con 0x7f350810e6c0 2024-04-12T05:30:54.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.525+0000 7f35077fe700 1 --2- 172.21.15.113:0/1315526343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f34f006cab0 0x7f34f006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:54.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.525+0000 7f35077fe700 1 --2- 172.21.15.113:0/1315526343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f34f006cab0 0x7f34f006ef70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f34fc000c00 tx=0x7f34fc009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:54.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.528+0000 7f3504ff9700 1 -- 172.21.15.113:0/1315526343 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f34f8059620 con 0x7f350810e6c0 2024-04-12T05:30:54.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.726+0000 7f350e0aa700 1 -- 172.21.15.113:0/1315526343 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3508109850 con 0x7f34f006cab0 2024-04-12T05:30:54.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.735+0000 7f3504ff9700 1 -- 172.21.15.113:0/1315526343 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3508109850 con 0x7f34f006cab0 2024-04-12T05:30:54.736 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:30:54.736 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (18m) 11m ago 29m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:30:54.736 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (16m) 11m ago 29m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:30:54.736 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (16m) 11m ago 29m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:30:54.736 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (18m) 11m ago 29m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:30:54.736 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (12m) 11m ago 24m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:30:54.736 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (11m) 11m ago 24m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:30:54.736 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (11m) 11m ago 24m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:30:54.736 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (24m) 11m ago 24m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:30:54.736 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (20m) 11m ago 31m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:30:54.736 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (19m) 11m ago 29m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:30:54.737 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (17m) 11m ago 31m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:30:54.737 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (16m) 11m ago 29m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:30:54.737 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (19m) 11m ago 29m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:30:54.737 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (19m) 11m ago 29m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:30:54.737 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (15m) 11m ago 28m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:30:54.737 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (15m) 11m ago 27m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:30:54.737 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (14m) 11m ago 27m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:30:54.737 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (13m) 11m ago 27m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:30:54.737 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (13m) 11m ago 26m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:30:54.737 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (12m) 11m ago 26m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:30:54.737 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (19m) 11m ago 28m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:30:54.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.737+0000 7f350e0aa700 1 -- 172.21.15.113:0/1315526343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f34f006cab0 msgr2=0x7f34f006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.737+0000 7f350e0aa700 1 --2- 172.21.15.113:0/1315526343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f34f006cab0 0x7f34f006ef70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f34fc000c00 tx=0x7f34fc009cb0).stop 2024-04-12T05:30:54.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.737+0000 7f350e0aa700 1 -- 172.21.15.113:0/1315526343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f350810e6c0 msgr2=0x7f3508108370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.737+0000 7f350e0aa700 1 --2- 172.21.15.113:0/1315526343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f350810e6c0 0x7f3508108370 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f34f8002820 tx=0x7f34f8009f20).stop 2024-04-12T05:30:54.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.738+0000 7f350e0aa700 1 -- 172.21.15.113:0/1315526343 shutdown_connections 2024-04-12T05:30:54.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.738+0000 7f350e0aa700 1 --2- 172.21.15.113:0/1315526343 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f34f006cab0 0x7f34f006ef70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.738+0000 7f350e0aa700 1 --2- 172.21.15.113:0/1315526343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35080fe270 0x7f350810e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.738+0000 7f350e0aa700 1 --2- 172.21.15.113:0/1315526343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f350810e6c0 0x7f3508108370 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.738+0000 7f350e0aa700 1 -- 172.21.15.113:0/1315526343 >> 172.21.15.113:0/1315526343 conn(0x7f35080f3530 msgr2=0x7f35080fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:54.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.738+0000 7f350e0aa700 1 -- 172.21.15.113:0/1315526343 shutdown_connections 2024-04-12T05:30:54.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.738+0000 7f350e0aa700 1 -- 172.21.15.113:0/1315526343 wait complete. 2024-04-12T05:30:54.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.833+0000 7f06a3b02700 1 -- 172.21.15.113:0/1135561011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c0fc7f0 msgr2=0x7f069c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.833+0000 7f06a3b02700 1 --2- 172.21.15.113:0/1135561011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c0fc7f0 0x7f069c0fcbd0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f06900047f0 tx=0x7f0690009cb0).stop 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.833+0000 7f06a3b02700 1 -- 172.21.15.113:0/1135561011 shutdown_connections 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.833+0000 7f06a3b02700 1 --2- 172.21.15.113:0/1135561011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f069c0fd150 0x7f069c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.833+0000 7f06a3b02700 1 --2- 172.21.15.113:0/1135561011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c0fc7f0 0x7f069c0fcbd0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.833+0000 7f06a3b02700 1 -- 172.21.15.113:0/1135561011 >> 172.21.15.113:0/1135561011 conn(0x7f069c070590 msgr2=0x7f069c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.833+0000 7f06a3b02700 1 -- 172.21.15.113:0/1135561011 shutdown_connections 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.833+0000 7f06a3b02700 1 -- 172.21.15.113:0/1135561011 wait complete. 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.834+0000 7f06a3b02700 1 Processor -- start 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.834+0000 7f06a3b02700 1 -- start start 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.834+0000 7f06a3b02700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f069c0fc7f0 0x7f069c109100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.834+0000 7f06a3b02700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c0fd150 0x7f069c1095f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.834+0000 7f06a3b02700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c109ae0 con 0x7f069c0fd150 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.834+0000 7f06a3b02700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c109c20 con 0x7f069c0fc7f0 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.834+0000 7f06a109d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c0fd150 0x7f069c1095f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:54.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.834+0000 7f06a109d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c0fd150 0x7f069c1095f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39962/0 (socket says 172.21.15.113:39962) 2024-04-12T05:30:54.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.834+0000 7f06a109d700 1 -- 172.21.15.113:0/3936696919 learned_addr learned my addr 172.21.15.113:0/3936696919 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:54.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.835+0000 7f06a189e700 1 --2- 172.21.15.113:0/3936696919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f069c0fc7f0 0x7f069c109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:54.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.835+0000 7f06a109d700 1 -- 172.21.15.113:0/3936696919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f069c0fc7f0 msgr2=0x7f069c109100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:54.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.835+0000 7f06a109d700 1 --2- 172.21.15.113:0/3936696919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f069c0fc7f0 0x7f069c109100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:54.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.835+0000 7f06a109d700 1 -- 172.21.15.113:0/3936696919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0690005040 con 0x7f069c0fd150 2024-04-12T05:30:54.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.835+0000 7f06a109d700 1 --2- 172.21.15.113:0/3936696919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c0fd150 0x7f069c1095f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f0698002820 tx=0x7f0698009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:54.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.835+0000 7f068effd700 1 -- 172.21.15.113:0/3936696919 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0698007b40 con 0x7f069c0fd150 2024-04-12T05:30:54.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.835+0000 7f068effd700 1 -- 172.21.15.113:0/3936696919 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f069800f040 con 0x7f069c0fd150 2024-04-12T05:30:54.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.835+0000 7f06a3b02700 1 -- 172.21.15.113:0/3936696919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c109ed0 con 0x7f069c0fd150 2024-04-12T05:30:54.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.835+0000 7f068effd700 1 -- 172.21.15.113:0/3936696919 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06980148b0 con 0x7f069c0fd150 2024-04-12T05:30:54.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.835+0000 7f06a3b02700 1 -- 172.21.15.113:0/3936696919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c1124f0 con 0x7f069c0fd150 2024-04-12T05:30:54.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.836+0000 7f06a3b02700 1 -- 172.21.15.113:0/3936696919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c05f590 con 0x7f069c0fd150 2024-04-12T05:30:54.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.836+0000 7f068effd700 1 -- 172.21.15.113:0/3936696919 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0698007ca0 con 0x7f069c0fd150 2024-04-12T05:30:54.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.837+0000 7f068effd700 1 --2- 172.21.15.113:0/3936696919 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f068806caa0 0x7f068806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:54.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.837+0000 7f068effd700 1 -- 172.21.15.113:0/3936696919 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f069808b8f0 con 0x7f069c0fd150 2024-04-12T05:30:54.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.837+0000 7f06a189e700 1 --2- 172.21.15.113:0/3936696919 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f068806caa0 0x7f068806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:54.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.837+0000 7f06a189e700 1 --2- 172.21.15.113:0/3936696919 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f068806caa0 0x7f068806ef60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f0690004f10 tx=0x7f069000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:54.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:54.841+0000 7f068effd700 1 -- 172.21.15.113:0/3936696919 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0698055910 con 0x7f069c0fd150 2024-04-12T05:30:55.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.112+0000 7f06a3b02700 1 -- 172.21.15.113:0/3936696919 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f069c047540 con 0x7f069c0fd150 2024-04-12T05:30:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.112+0000 7f068effd700 1 -- 172.21.15.113:0/3936696919 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f0698058f30 con 0x7f069c0fd150 2024-04-12T05:30:55.113 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:30:55.113 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:30:55.113 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:30:55.113 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:30:55.113 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:30:55.113 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:30:55.113 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:30:55.113 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:30:55.113 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:30:55.113 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:30:55.114 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:30:55.114 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:30:55.114 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:30:55.114 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:30:55.114 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:30:55.114 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:30:55.114 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:30:55.114 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:30:55.114 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:30:55.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.114+0000 7f06a3b02700 1 -- 172.21.15.113:0/3936696919 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f068806caa0 msgr2=0x7f068806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.114+0000 7f06a3b02700 1 --2- 172.21.15.113:0/3936696919 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f068806caa0 0x7f068806ef60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f0690004f10 tx=0x7f069000b040).stop 2024-04-12T05:30:55.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.114+0000 7f06a3b02700 1 -- 172.21.15.113:0/3936696919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c0fd150 msgr2=0x7f069c1095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.114+0000 7f06a3b02700 1 --2- 172.21.15.113:0/3936696919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c0fd150 0x7f069c1095f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f0698002820 tx=0x7f0698009f20).stop 2024-04-12T05:30:55.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.114+0000 7f06a3b02700 1 -- 172.21.15.113:0/3936696919 shutdown_connections 2024-04-12T05:30:55.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.114+0000 7f06a3b02700 1 --2- 172.21.15.113:0/3936696919 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f068806caa0 0x7f068806ef60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.114+0000 7f06a3b02700 1 --2- 172.21.15.113:0/3936696919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f069c0fc7f0 0x7f069c109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.114+0000 7f06a3b02700 1 --2- 172.21.15.113:0/3936696919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c0fd150 0x7f069c1095f0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.114+0000 7f06a3b02700 1 -- 172.21.15.113:0/3936696919 >> 172.21.15.113:0/3936696919 conn(0x7f069c070590 msgr2=0x7f069c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:55.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.114+0000 7f06a3b02700 1 -- 172.21.15.113:0/3936696919 shutdown_connections 2024-04-12T05:30:55.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.114+0000 7f06a3b02700 1 -- 172.21.15.113:0/3936696919 wait complete. 2024-04-12T05:30:55.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.214+0000 7f925550c700 1 -- 172.21.15.113:0/2371321494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500fd970 msgr2=0x7f92500fdd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.214+0000 7f925550c700 1 --2- 172.21.15.113:0/2371321494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500fd970 0x7f92500fdd50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f92400047f0 tx=0x7f9240009cb0).stop 2024-04-12T05:30:55.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.214+0000 7f925550c700 1 -- 172.21.15.113:0/2371321494 shutdown_connections 2024-04-12T05:30:55.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.214+0000 7f925550c700 1 --2- 172.21.15.113:0/2371321494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92500fe2d0 0x7f92500f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.214+0000 7f925550c700 1 --2- 172.21.15.113:0/2371321494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500fd970 0x7f92500fdd50 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.214+0000 7f925550c700 1 -- 172.21.15.113:0/2371321494 >> 172.21.15.113:0/2371321494 conn(0x7f92500f3550 msgr2=0x7f92500f5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:55.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.214+0000 7f925550c700 1 -- 172.21.15.113:0/2371321494 shutdown_connections 2024-04-12T05:30:55.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.214+0000 7f925550c700 1 -- 172.21.15.113:0/2371321494 wait complete. 2024-04-12T05:30:55.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.214+0000 7f925550c700 1 Processor -- start 2024-04-12T05:30:55.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f925550c700 1 -- start start 2024-04-12T05:30:55.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f925550c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500fd970 0x7f925010e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:55.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f925550c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92500fe2d0 0x7f925010e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:55.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f925550c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925010eba0 con 0x7f92500fd970 2024-04-12T05:30:55.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f925550c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925010ece0 con 0x7f92500fe2d0 2024-04-12T05:30:55.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f924effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500fd970 0x7f925010e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:55.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f924effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500fd970 0x7f925010e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39976/0 (socket says 172.21.15.113:39976) 2024-04-12T05:30:55.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f924effd700 1 -- 172.21.15.113:0/2604125214 learned_addr learned my addr 172.21.15.113:0/2604125214 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:55.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f924e7fc700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92500fe2d0 0x7f925010e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:55.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f924effd700 1 -- 172.21.15.113:0/2604125214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92500fe2d0 msgr2=0x7f925010e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f924effd700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92500fe2d0 0x7f925010e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f924effd700 1 -- 172.21.15.113:0/2604125214 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9240005040 con 0x7f92500fd970 2024-04-12T05:30:55.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f924e7fc700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92500fe2d0 0x7f925010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:30:55.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.215+0000 7f924effd700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500fd970 0x7f925010e0b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f9240005010 tx=0x7f9240009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:55.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.216+0000 7f9237fff700 1 -- 172.21.15.113:0/2604125214 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f924000e030 con 0x7f92500fd970 2024-04-12T05:30:55.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.216+0000 7f925550c700 1 -- 172.21.15.113:0/2604125214 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9250108310 con 0x7f92500fd970 2024-04-12T05:30:55.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.216+0000 7f9237fff700 1 -- 172.21.15.113:0/2604125214 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9240007e90 con 0x7f92500fd970 2024-04-12T05:30:55.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.216+0000 7f9237fff700 1 -- 172.21.15.113:0/2604125214 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9240018070 con 0x7f92500fd970 2024-04-12T05:30:55.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.216+0000 7f925550c700 1 -- 172.21.15.113:0/2604125214 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92501087f0 con 0x7f92500fd970 2024-04-12T05:30:55.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.217+0000 7f925550c700 1 -- 172.21.15.113:0/2604125214 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9250060370 con 0x7f92500fd970 2024-04-12T05:30:55.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.217+0000 7f9237fff700 1 -- 172.21.15.113:0/2604125214 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9240009610 con 0x7f92500fd970 2024-04-12T05:30:55.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.217+0000 7f9237fff700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f923806ca30 0x7f923806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:55.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.217+0000 7f9237fff700 1 -- 172.21.15.113:0/2604125214 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f924008d0f0 con 0x7f92500fd970 2024-04-12T05:30:55.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.217+0000 7f924e7fc700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f923806ca30 0x7f923806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:55.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.218+0000 7f924e7fc700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f923806ca30 0x7f923806eef0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f92501099f0 tx=0x7f9244009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:55.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.222+0000 7f9237fff700 1 -- 172.21.15.113:0/2604125214 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f924005aba0 con 0x7f92500fd970 2024-04-12T05:30:55.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:55 smithi113 ceph-mon[103371]: pgmap v562: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:55.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:55 smithi113 ceph-mon[103371]: from='client.35036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:55.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:55 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3936696919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:30:55.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:55 smithi192 ceph-mon[85304]: pgmap v562: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:55.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:55 smithi192 ceph-mon[85304]: from='client.35036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:55.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:55 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3936696919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:30:55.451 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.451+0000 7f925550c700 1 -- 172.21.15.113:0/2604125214 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9250047540 con 0x7f92500fd970 2024-04-12T05:30:55.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.452+0000 7f9237fff700 1 -- 172.21.15.113:0/2604125214 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f924005a730 con 0x7f92500fd970 2024-04-12T05:30:55.452 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:30:55.452 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:30:55.453 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:30:55.453 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:30:55.453 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:30:55.453 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:30:55.453 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:30:55.453 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:30:55.453 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:30:55.453 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:30:55.453 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:30:55.453 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:30:55.454 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:30:55.454 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:30:55.454 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:30:55.454 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:30:55.454 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:30:55.454 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:30:55.454 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:30:55.454 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:30:55.454 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:30:55.454 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:30:55.454 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:30:55.455 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:30:55.455 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:30:55.455 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:30:55.455 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:30:55.455 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:30:55.455 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:30:55.455 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:30:55.455 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:30:55.455 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:30:55.455 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:30:55.455 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:30:55.456 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:30:55.456 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:30:55.456 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:30:55.456 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:30:55.456 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:30:55.456 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:30:55.456 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:30:55.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.454+0000 7f925550c700 1 -- 172.21.15.113:0/2604125214 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f923806ca30 msgr2=0x7f923806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.454+0000 7f925550c700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f923806ca30 0x7f923806eef0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f92501099f0 tx=0x7f9244009da0).stop 2024-04-12T05:30:55.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.454+0000 7f925550c700 1 -- 172.21.15.113:0/2604125214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500fd970 msgr2=0x7f925010e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.454+0000 7f925550c700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500fd970 0x7f925010e0b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f9240005010 tx=0x7f9240009cb0).stop 2024-04-12T05:30:55.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.454+0000 7f925550c700 1 -- 172.21.15.113:0/2604125214 shutdown_connections 2024-04-12T05:30:55.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.454+0000 7f925550c700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f923806ca30 0x7f923806eef0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.454+0000 7f925550c700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f92500fd970 0x7f925010e0b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.454+0000 7f925550c700 1 --2- 172.21.15.113:0/2604125214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92500fe2d0 0x7f925010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.454+0000 7f925550c700 1 -- 172.21.15.113:0/2604125214 >> 172.21.15.113:0/2604125214 conn(0x7f92500f3550 msgr2=0x7f92500fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:55.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.454+0000 7f925550c700 1 -- 172.21.15.113:0/2604125214 shutdown_connections 2024-04-12T05:30:55.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.454+0000 7f925550c700 1 -- 172.21.15.113:0/2604125214 wait complete. 2024-04-12T05:30:55.458 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:30:55.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.553+0000 7f0ba21cd700 1 -- 172.21.15.113:0/954417719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b9c0fd8f0 msgr2=0x7f0b9c0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.553+0000 7f0ba21cd700 1 --2- 172.21.15.113:0/954417719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b9c0fd8f0 0x7f0b9c0fdcd0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f0b900047f0 tx=0x7f0b90009cb0).stop 2024-04-12T05:30:55.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.553+0000 7f0ba21cd700 1 -- 172.21.15.113:0/954417719 shutdown_connections 2024-04-12T05:30:55.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.553+0000 7f0ba21cd700 1 --2- 172.21.15.113:0/954417719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b9c0fe250 0x7f0b9c0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.553+0000 7f0ba21cd700 1 --2- 172.21.15.113:0/954417719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b9c0fd8f0 0x7f0b9c0fdcd0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.553+0000 7f0ba21cd700 1 -- 172.21.15.113:0/954417719 >> 172.21.15.113:0/954417719 conn(0x7f0b9c0f34b0 msgr2=0x7f0b9c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:55.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.553+0000 7f0ba21cd700 1 -- 172.21.15.113:0/954417719 shutdown_connections 2024-04-12T05:30:55.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.553+0000 7f0ba21cd700 1 -- 172.21.15.113:0/954417719 wait complete. 2024-04-12T05:30:55.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.553+0000 7f0ba21cd700 1 Processor -- start 2024-04-12T05:30:55.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0ba21cd700 1 -- start start 2024-04-12T05:30:55.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0ba21cd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b9c0fd8f0 0x7f0b9c06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:55.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0ba21cd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b9c0fe250 0x7f0b9c06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:55.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0ba21cd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c06bab0 con 0x7f0b9c0fd8f0 2024-04-12T05:30:55.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0ba21cd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c06bbf0 con 0x7f0b9c0fe250 2024-04-12T05:30:55.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0b9b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b9c0fd8f0 0x7f0b9c06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:55.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0b9b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b9c0fd8f0 0x7f0b9c06b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39994/0 (socket says 172.21.15.113:39994) 2024-04-12T05:30:55.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0b9b7fe700 1 -- 172.21.15.113:0/586683485 learned_addr learned my addr 172.21.15.113:0/586683485 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:55.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0b9b7fe700 1 -- 172.21.15.113:0/586683485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b9c0fe250 msgr2=0x7f0b9c06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0b9b7fe700 1 --2- 172.21.15.113:0/586683485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b9c0fe250 0x7f0b9c06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:30:55.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0b9b7fe700 1 -- 172.21.15.113:0/586683485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b90005040 con 0x7f0b9c0fd8f0 2024-04-12T05:30:55.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0b9b7fe700 1 --2- 172.21.15.113:0/586683485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b9c0fd8f0 0x7f0b9c06b0d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f0b90005010 tx=0x7f0b90009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:55.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.554+0000 7f0b98ff9700 1 -- 172.21.15.113:0/586683485 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b9000e030 con 0x7f0b9c0fd8f0 2024-04-12T05:30:55.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.555+0000 7f0ba21cd700 1 -- 172.21.15.113:0/586683485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c06be40 con 0x7f0b9c0fd8f0 2024-04-12T05:30:55.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.555+0000 7f0b98ff9700 1 -- 172.21.15.113:0/586683485 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b90007e90 con 0x7f0b9c0fd8f0 2024-04-12T05:30:55.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.555+0000 7f0b98ff9700 1 -- 172.21.15.113:0/586683485 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b90018070 con 0x7f0b9c0fd8f0 2024-04-12T05:30:55.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.555+0000 7f0ba21cd700 1 -- 172.21.15.113:0/586683485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c1145f0 con 0x7f0b9c0fd8f0 2024-04-12T05:30:55.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.555+0000 7f0ba21cd700 1 -- 172.21.15.113:0/586683485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9c05f590 con 0x7f0b9c0fd8f0 2024-04-12T05:30:55.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.556+0000 7f0b98ff9700 1 -- 172.21.15.113:0/586683485 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0b9001d6e0 con 0x7f0b9c0fd8f0 2024-04-12T05:30:55.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.557+0000 7f0b98ff9700 1 --2- 172.21.15.113:0/586683485 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8406ca30 0x7f0b8406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:55.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.557+0000 7f0b98ff9700 1 -- 172.21.15.113:0/586683485 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0b9008d420 con 0x7f0b9c0fd8f0 2024-04-12T05:30:55.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.557+0000 7f0b9affd700 1 --2- 172.21.15.113:0/586683485 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8406ca30 0x7f0b8406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:55.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.557+0000 7f0b9affd700 1 --2- 172.21.15.113:0/586683485 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8406ca30 0x7f0b8406eef0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f0b9c06c870 tx=0x7f0b8c009b60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:55.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.561+0000 7f0b98ff9700 1 -- 172.21.15.113:0/586683485 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0b90057500 con 0x7f0b9c0fd8f0 2024-04-12T05:30:55.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.764+0000 7f0ba21cd700 1 -- 172.21.15.113:0/586683485 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b9c0ffcf0 con 0x7f0b8406ca30 2024-04-12T05:30:55.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.765+0000 7f0b98ff9700 1 -- 172.21.15.113:0/586683485 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0b9c0ffcf0 con 0x7f0b8406ca30 2024-04-12T05:30:55.765 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:30:55.765 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:30:55.765 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:30:55.765 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:30:55.765 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:30:55.766 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:30:55.766 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:30:55.766 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:30:55.766 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:30:55.766 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:30:55.766 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:30:55.766 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:30:55.766 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:30:55.766 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:30:55.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.767+0000 7f0ba21cd700 1 -- 172.21.15.113:0/586683485 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8406ca30 msgr2=0x7f0b8406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.767+0000 7f0ba21cd700 1 --2- 172.21.15.113:0/586683485 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8406ca30 0x7f0b8406eef0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f0b9c06c870 tx=0x7f0b8c009b60).stop 2024-04-12T05:30:55.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.767+0000 7f0ba21cd700 1 -- 172.21.15.113:0/586683485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b9c0fd8f0 msgr2=0x7f0b9c06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.767+0000 7f0ba21cd700 1 --2- 172.21.15.113:0/586683485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b9c0fd8f0 0x7f0b9c06b0d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f0b90005010 tx=0x7f0b90009cb0).stop 2024-04-12T05:30:55.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.767+0000 7f0ba21cd700 1 -- 172.21.15.113:0/586683485 shutdown_connections 2024-04-12T05:30:55.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.767+0000 7f0ba21cd700 1 --2- 172.21.15.113:0/586683485 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8406ca30 0x7f0b8406eef0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.767+0000 7f0ba21cd700 1 --2- 172.21.15.113:0/586683485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b9c0fd8f0 0x7f0b9c06b0d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.767+0000 7f0ba21cd700 1 --2- 172.21.15.113:0/586683485 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b9c0fe250 0x7f0b9c06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:30:55.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.767+0000 7f0ba21cd700 1 -- 172.21.15.113:0/586683485 >> 172.21.15.113:0/586683485 conn(0x7f0b9c0f34b0 msgr2=0x7f0b9c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:55.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.767+0000 7f0ba21cd700 1 -- 172.21.15.113:0/586683485 shutdown_connections 2024-04-12T05:30:55.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.767+0000 7f0ba21cd700 1 -- 172.21.15.113:0/586683485 wait complete. 2024-04-12T05:30:55.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.864+0000 7f8cd702e700 1 -- 172.21.15.113:0/2502833887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cd00fd910 msgr2=0x7f8cd00fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.864+0000 7f8cd702e700 1 --2- 172.21.15.113:0/2502833887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cd00fd910 0x7f8cd00fdcf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f8cc40047f0 tx=0x7f8cc4009cb0).stop 2024-04-12T05:30:55.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.864+0000 7f8cd702e700 1 -- 172.21.15.113:0/2502833887 shutdown_connections 2024-04-12T05:30:55.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.864+0000 7f8cd702e700 1 --2- 172.21.15.113:0/2502833887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cd00fe270 0x7f8cd00f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.864+0000 7f8cd702e700 1 --2- 172.21.15.113:0/2502833887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cd00fd910 0x7f8cd00fdcf0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:55.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.864+0000 7f8cd702e700 1 -- 172.21.15.113:0/2502833887 >> 172.21.15.113:0/2502833887 conn(0x7f8cd00f3530 msgr2=0x7f8cd00f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:55.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.864+0000 7f8cd702e700 1 -- 172.21.15.113:0/2502833887 shutdown_connections 2024-04-12T05:30:55.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.864+0000 7f8cd702e700 1 -- 172.21.15.113:0/2502833887 wait complete. 2024-04-12T05:30:55.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.864+0000 7f8cd702e700 1 Processor -- start 2024-04-12T05:30:55.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.864+0000 7f8cd702e700 1 -- start start 2024-04-12T05:30:55.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd702e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cd00fd910 0x7f8cd010e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd702e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cd00fe270 0x7f8cd010e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd702e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd010eba0 con 0x7f8cd00fd910 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd702e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd010ece0 con 0x7f8cd00fe270 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd4dca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cd00fd910 0x7f8cd010e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd4dca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cd00fd910 0x7f8cd010e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40018/0 (socket says 172.21.15.113:40018) 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd4dca700 1 -- 172.21.15.113:0/115213476 learned_addr learned my addr 172.21.15.113:0/115213476 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd4dca700 1 -- 172.21.15.113:0/115213476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cd00fe270 msgr2=0x7f8cd010e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd4dca700 1 --2- 172.21.15.113:0/115213476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cd00fe270 0x7f8cd010e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd4dca700 1 -- 172.21.15.113:0/115213476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc4005040 con 0x7f8cd00fd910 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd4dca700 1 --2- 172.21.15.113:0/115213476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cd00fd910 0x7f8cd010e0b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f8cc400b120 tx=0x7f8cc4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8ccdffb700 1 -- 172.21.15.113:0/115213476 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cc400e030 con 0x7f8cd00fd910 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8cd702e700 1 -- 172.21.15.113:0/115213476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd0108310 con 0x7f8cd00fd910 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8ccdffb700 1 -- 172.21.15.113:0/115213476 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cc4007e90 con 0x7f8cd00fd910 2024-04-12T05:30:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.865+0000 7f8ccdffb700 1 -- 172.21.15.113:0/115213476 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cc4018070 con 0x7f8cd00fd910 2024-04-12T05:30:55.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.866+0000 7f8cd702e700 1 -- 172.21.15.113:0/115213476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd01087f0 con 0x7f8cd00fd910 2024-04-12T05:30:55.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.866+0000 7f8cd702e700 1 -- 172.21.15.113:0/115213476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd005f570 con 0x7f8cd00fd910 2024-04-12T05:30:55.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.867+0000 7f8ccdffb700 1 -- 172.21.15.113:0/115213476 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8cc401d420 con 0x7f8cd00fd910 2024-04-12T05:30:55.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.867+0000 7f8ccdffb700 1 --2- 172.21.15.113:0/115213476 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8cc006caa0 0x7f8cc006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:30:55.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.867+0000 7f8ccdffb700 1 -- 172.21.15.113:0/115213476 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f8cc408d300 con 0x7f8cd00fd910 2024-04-12T05:30:55.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.868+0000 7f8ccffff700 1 --2- 172.21.15.113:0/115213476 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8cc006caa0 0x7f8cc006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:30:55.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.868+0000 7f8ccffff700 1 --2- 172.21.15.113:0/115213476 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8cc006caa0 0x7f8cc006ef60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f8cd01099f0 tx=0x7f8cbc007bc0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:30:55.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:55.872+0000 7f8ccdffb700 1 -- 172.21.15.113:0/115213476 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8cc405adb0 con 0x7f8cd00fd910 2024-04-12T05:30:56.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.135+0000 7f8cd702e700 1 -- 172.21.15.113:0/115213476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8cd0047540 con 0x7f8cd00fd910 2024-04-12T05:30:56.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.135+0000 7f8ccdffb700 1 -- 172.21.15.113:0/115213476 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f8cc405a940 con 0x7f8cd00fd910 2024-04-12T05:30:56.136 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:30:56.136 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:30:56.136 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:30:56.136 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:30:56.136 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:30:56.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.137+0000 7f8cd702e700 1 -- 172.21.15.113:0/115213476 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8cc006caa0 msgr2=0x7f8cc006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:56.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.137+0000 7f8cd702e700 1 --2- 172.21.15.113:0/115213476 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8cc006caa0 0x7f8cc006ef60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f8cd01099f0 tx=0x7f8cbc007bc0).stop 2024-04-12T05:30:56.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.137+0000 7f8cd702e700 1 -- 172.21.15.113:0/115213476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cd00fd910 msgr2=0x7f8cd010e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:30:56.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.137+0000 7f8cd702e700 1 --2- 172.21.15.113:0/115213476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cd00fd910 0x7f8cd010e0b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f8cc400b120 tx=0x7f8cc4009cb0).stop 2024-04-12T05:30:56.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.137+0000 7f8cd702e700 1 -- 172.21.15.113:0/115213476 shutdown_connections 2024-04-12T05:30:56.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.137+0000 7f8cd702e700 1 --2- 172.21.15.113:0/115213476 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8cc006caa0 0x7f8cc006ef60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:56.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.137+0000 7f8cd702e700 1 --2- 172.21.15.113:0/115213476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8cd00fd910 0x7f8cd010e0b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:30:56.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.137+0000 7f8cd702e700 1 --2- 172.21.15.113:0/115213476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cd00fe270 0x7f8cd010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:30:56.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.137+0000 7f8cd702e700 1 -- 172.21.15.113:0/115213476 >> 172.21.15.113:0/115213476 conn(0x7f8cd00f3530 msgr2=0x7f8cd00fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:30:56.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.137+0000 7f8cd702e700 1 -- 172.21.15.113:0/115213476 shutdown_connections 2024-04-12T05:30:56.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:30:56.137+0000 7f8cd702e700 1 -- 172.21.15.113:0/115213476 wait complete. 2024-04-12T05:30:56.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:56 smithi192 ceph-mon[85304]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:56.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:56 smithi192 ceph-mon[85304]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:56.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:56 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2604125214' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:30:56.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:56 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/115213476' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:30:56.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:56 smithi113 ceph-mon[103371]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:56.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:56 smithi113 ceph-mon[103371]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:56.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:56 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2604125214' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:30:56.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:56 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/115213476' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:30:57.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:57 smithi192 ceph-mon[85304]: from='client.35056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:57.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:57 smithi192 ceph-mon[85304]: pgmap v563: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:57.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:57 smithi113 ceph-mon[103371]: from='client.35056 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:30:57.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:57 smithi113 ceph-mon[103371]: pgmap v563: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:59.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:30:59 smithi192 ceph-mon[85304]: pgmap v564: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:30:59.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:30:59 smithi113 ceph-mon[103371]: pgmap v564: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:01.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:01 smithi192 ceph-mon[85304]: pgmap v565: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:01.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:01 smithi113 ceph-mon[103371]: pgmap v565: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:03.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:03 smithi192 ceph-mon[85304]: pgmap v566: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:03.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:03 smithi113 ceph-mon[103371]: pgmap v566: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:04.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:04 smithi113 ceph-mon[103371]: pgmap v567: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:04.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:04 smithi192 ceph-mon[85304]: pgmap v567: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:07.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:07 smithi113 ceph-mon[103371]: pgmap v568: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:07.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:07.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:07 smithi192 ceph-mon[85304]: pgmap v568: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:07.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:09.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:09 smithi113 ceph-mon[103371]: pgmap v569: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:09.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:09 smithi192 ceph-mon[85304]: pgmap v569: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:11.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:11 smithi113 ceph-mon[103371]: pgmap v570: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:11.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:11 smithi192 ceph-mon[85304]: pgmap v570: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:13.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:13 smithi113 ceph-mon[103371]: pgmap v571: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:13.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:13 smithi192 ceph-mon[85304]: pgmap v571: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:15.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:15 smithi113 ceph-mon[103371]: pgmap v572: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:15.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:15 smithi192 ceph-mon[85304]: pgmap v572: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:17.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:17 smithi113 ceph-mon[103371]: pgmap v573: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:17.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:17 smithi192 ceph-mon[85304]: pgmap v573: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:19.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:19 smithi113 ceph-mon[103371]: pgmap v574: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:19.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:19 smithi192 ceph-mon[85304]: pgmap v574: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:21.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:21 smithi113 ceph-mon[103371]: pgmap v575: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:21.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:21 smithi192 ceph-mon[85304]: pgmap v575: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:22.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:22.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:23.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:23 smithi113 ceph-mon[103371]: pgmap v576: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:23.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:23 smithi192 ceph-mon[85304]: pgmap v576: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:25.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:25 smithi113 ceph-mon[103371]: pgmap v577: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:25.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:25 smithi192 ceph-mon[85304]: pgmap v577: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:26.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.249+0000 7f716a332700 1 -- 172.21.15.113:0/839642111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71640f88e0 msgr2=0x7f71640f8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:26.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.249+0000 7f716a332700 1 --2- 172.21.15.113:0/839642111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71640f88e0 0x7f71640f8cc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f71580047f0 tx=0x7f7158009cb0).stop 2024-04-12T05:31:26.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.249+0000 7f716a332700 1 -- 172.21.15.113:0/839642111 shutdown_connections 2024-04-12T05:31:26.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.249+0000 7f716a332700 1 --2- 172.21.15.113:0/839642111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71640f9240 0x7f7164101950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.249+0000 7f716a332700 1 --2- 172.21.15.113:0/839642111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71640f88e0 0x7f71640f8cc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.249+0000 7f716a332700 1 -- 172.21.15.113:0/839642111 >> 172.21.15.113:0/839642111 conn(0x7f71640f2430 msgr2=0x7f71640f4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:26.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.249+0000 7f716a332700 1 -- 172.21.15.113:0/839642111 shutdown_connections 2024-04-12T05:31:26.250 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.249+0000 7f716a332700 1 -- 172.21.15.113:0/839642111 wait complete. 2024-04-12T05:31:26.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.249+0000 7f716a332700 1 Processor -- start 2024-04-12T05:31:26.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.249+0000 7f716a332700 1 -- start start 2024-04-12T05:31:26.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f716a332700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71640f88e0 0x7f71640f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:26.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f716a332700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71640f9240 0x7f71640f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:26.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f716a332700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71640f7270 con 0x7f71640f9240 2024-04-12T05:31:26.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f716a332700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71640f73b0 con 0x7f71640f88e0 2024-04-12T05:31:26.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f71637fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71640f9240 0x7f71640f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f71637fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71640f9240 0x7f71640f6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35814/0 (socket says 172.21.15.113:35814) 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f71637fe700 1 -- 172.21.15.113:0/2742424499 learned_addr learned my addr 172.21.15.113:0/2742424499 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f7163fff700 1 --2- 172.21.15.113:0/2742424499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71640f88e0 0x7f71640f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f71637fe700 1 -- 172.21.15.113:0/2742424499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71640f88e0 msgr2=0x7f71640f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f71637fe700 1 --2- 172.21.15.113:0/2742424499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71640f88e0 0x7f71640f6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f71637fe700 1 -- 172.21.15.113:0/2742424499 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7158005040 con 0x7f71640f9240 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.250+0000 7f71637fe700 1 --2- 172.21.15.113:0/2742424499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71640f9240 0x7f71640f6d80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f715400e8f0 tx=0x7f715400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.251+0000 7f71617fa700 1 -- 172.21.15.113:0/2742424499 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7154007b40 con 0x7f71640f9240 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.251+0000 7f716a332700 1 -- 172.21.15.113:0/2742424499 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71640f7660 con 0x7f71640f9240 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.251+0000 7f71617fa700 1 -- 172.21.15.113:0/2742424499 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f715400f040 con 0x7f71640f9240 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.251+0000 7f71617fa700 1 -- 172.21.15.113:0/2742424499 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7154014880 con 0x7f71640f9240 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.251+0000 7f716a332700 1 -- 172.21.15.113:0/2742424499 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7164114810 con 0x7f71640f9240 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.252+0000 7f716a332700 1 -- 172.21.15.113:0/2742424499 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7164047540 con 0x7f71640f9240 2024-04-12T05:31:26.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.252+0000 7f71617fa700 1 -- 172.21.15.113:0/2742424499 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7154014b00 con 0x7f71640f9240 2024-04-12T05:31:26.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.253+0000 7f71617fa700 1 --2- 172.21.15.113:0/2742424499 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714c06caa0 0x7f714c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:26.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.253+0000 7f71617fa700 1 -- 172.21.15.113:0/2742424499 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f715408ba90 con 0x7f71640f9240 2024-04-12T05:31:26.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.253+0000 7f7163fff700 1 --2- 172.21.15.113:0/2742424499 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714c06caa0 0x7f714c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:26.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.253+0000 7f7163fff700 1 --2- 172.21.15.113:0/2742424499 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714c06caa0 0x7f714c06ef60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f71580047c0 tx=0x7f7158009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:26.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.257+0000 7f71617fa700 1 -- 172.21.15.113:0/2742424499 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7154055ab0 con 0x7f71640f9240 2024-04-12T05:31:26.465 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.465+0000 7f716a332700 1 -- 172.21.15.113:0/2742424499 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f71640f7e50 con 0x7f714c06caa0 2024-04-12T05:31:26.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.467+0000 7f71617fa700 1 -- 172.21.15.113:0/2742424499 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f71640f7e50 con 0x7f714c06caa0 2024-04-12T05:31:26.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.469+0000 7f716a332700 1 -- 172.21.15.113:0/2742424499 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714c06caa0 msgr2=0x7f714c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:26.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.469+0000 7f716a332700 1 --2- 172.21.15.113:0/2742424499 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714c06caa0 0x7f714c06ef60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f71580047c0 tx=0x7f7158009cb0).stop 2024-04-12T05:31:26.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.469+0000 7f716a332700 1 -- 172.21.15.113:0/2742424499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71640f9240 msgr2=0x7f71640f6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:26.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.469+0000 7f716a332700 1 --2- 172.21.15.113:0/2742424499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71640f9240 0x7f71640f6d80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f715400e8f0 tx=0x7f715400b040).stop 2024-04-12T05:31:26.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.469+0000 7f716a332700 1 -- 172.21.15.113:0/2742424499 shutdown_connections 2024-04-12T05:31:26.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.469+0000 7f716a332700 1 --2- 172.21.15.113:0/2742424499 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f714c06caa0 0x7f714c06ef60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.469+0000 7f716a332700 1 --2- 172.21.15.113:0/2742424499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71640f88e0 0x7f71640f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.469+0000 7f716a332700 1 --2- 172.21.15.113:0/2742424499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71640f9240 0x7f71640f6d80 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.469+0000 7f716a332700 1 -- 172.21.15.113:0/2742424499 >> 172.21.15.113:0/2742424499 conn(0x7f71640f2430 msgr2=0x7f7164101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:26.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.469+0000 7f716a332700 1 -- 172.21.15.113:0/2742424499 shutdown_connections 2024-04-12T05:31:26.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.469+0000 7f716a332700 1 -- 172.21.15.113:0/2742424499 wait complete. 2024-04-12T05:31:26.479 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:31:26.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.568+0000 7f27e8248700 1 -- 172.21.15.113:0/528749679 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00fd150 msgr2=0x7f27e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:26.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.568+0000 7f27e8248700 1 --2- 172.21.15.113:0/528749679 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00fd150 0x7f27e00f6930 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f27dc0047f0 tx=0x7f27dc009cb0).stop 2024-04-12T05:31:26.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.568+0000 7f27e8248700 1 -- 172.21.15.113:0/528749679 shutdown_connections 2024-04-12T05:31:26.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.568+0000 7f27e8248700 1 --2- 172.21.15.113:0/528749679 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00fd150 0x7f27e00f6930 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.568+0000 7f27e8248700 1 --2- 172.21.15.113:0/528749679 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27e00fc7f0 0x7f27e00fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.568+0000 7f27e8248700 1 -- 172.21.15.113:0/528749679 >> 172.21.15.113:0/528749679 conn(0x7f27e0070590 msgr2=0x7f27e00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:26.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.568+0000 7f27e8248700 1 -- 172.21.15.113:0/528749679 shutdown_connections 2024-04-12T05:31:26.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.568+0000 7f27e8248700 1 -- 172.21.15.113:0/528749679 wait complete. 2024-04-12T05:31:26.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.569+0000 7f27e8248700 1 Processor -- start 2024-04-12T05:31:26.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.569+0000 7f27e8248700 1 -- start start 2024-04-12T05:31:26.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.569+0000 7f27e8248700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00fc7f0 0x7f27e010ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:26.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.569+0000 7f27e8248700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27e00fd150 0x7f27e010d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:26.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.569+0000 7f27e8248700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e010d9c0 con 0x7f27e00fc7f0 2024-04-12T05:31:26.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.569+0000 7f27e8248700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e010db00 con 0x7f27e00fd150 2024-04-12T05:31:26.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.569+0000 7f27e5fe4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00fc7f0 0x7f27e010ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:26.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.569+0000 7f27e5fe4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00fc7f0 0x7f27e010ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35832/0 (socket says 172.21.15.113:35832) 2024-04-12T05:31:26.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.569+0000 7f27e5fe4700 1 -- 172.21.15.113:0/4258118383 learned_addr learned my addr 172.21.15.113:0/4258118383 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:26.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.569+0000 7f27e57e3700 1 --2- 172.21.15.113:0/4258118383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27e00fd150 0x7f27e010d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:26.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.570+0000 7f27e5fe4700 1 -- 172.21.15.113:0/4258118383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27e00fd150 msgr2=0x7f27e010d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:26.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.570+0000 7f27e5fe4700 1 --2- 172.21.15.113:0/4258118383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27e00fd150 0x7f27e010d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.570+0000 7f27e5fe4700 1 -- 172.21.15.113:0/4258118383 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc005040 con 0x7f27e00fc7f0 2024-04-12T05:31:26.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.570+0000 7f27e5fe4700 1 --2- 172.21.15.113:0/4258118383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00fc7f0 0x7f27e010ced0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f27d0002770 tx=0x7f27d0002a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:26.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.570+0000 7f27d6ffd700 1 -- 172.21.15.113:0/4258118383 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27d0007b40 con 0x7f27e00fc7f0 2024-04-12T05:31:26.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.570+0000 7f27e8248700 1 -- 172.21.15.113:0/4258118383 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e0107190 con 0x7f27e00fc7f0 2024-04-12T05:31:26.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.570+0000 7f27d6ffd700 1 -- 172.21.15.113:0/4258118383 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27d000f040 con 0x7f27e00fc7f0 2024-04-12T05:31:26.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.570+0000 7f27d6ffd700 1 -- 172.21.15.113:0/4258118383 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27d0014820 con 0x7f27e00fc7f0 2024-04-12T05:31:26.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.570+0000 7f27e8248700 1 -- 172.21.15.113:0/4258118383 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e01076d0 con 0x7f27e00fc7f0 2024-04-12T05:31:26.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.571+0000 7f27e8248700 1 -- 172.21.15.113:0/4258118383 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e005f590 con 0x7f27e00fc7f0 2024-04-12T05:31:26.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.571+0000 7f27d6ffd700 1 -- 172.21.15.113:0/4258118383 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f27d0007ca0 con 0x7f27e00fc7f0 2024-04-12T05:31:26.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.572+0000 7f27d6ffd700 1 --2- 172.21.15.113:0/4258118383 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27cc06caa0 0x7f27cc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:26.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.572+0000 7f27d6ffd700 1 -- 172.21.15.113:0/4258118383 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f27d008ba10 con 0x7f27e00fc7f0 2024-04-12T05:31:26.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.572+0000 7f27e57e3700 1 --2- 172.21.15.113:0/4258118383 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27cc06caa0 0x7f27cc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:26.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.572+0000 7f27e57e3700 1 --2- 172.21.15.113:0/4258118383 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27cc06caa0 0x7f27cc06ef60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f27e0108810 tx=0x7f27dc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:26.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.576+0000 7f27d6ffd700 1 -- 172.21.15.113:0/4258118383 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f27d00594c0 con 0x7f27e00fc7f0 2024-04-12T05:31:26.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.779+0000 7f27e8248700 1 -- 172.21.15.113:0/4258118383 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f27e0107950 con 0x7f27cc06caa0 2024-04-12T05:31:26.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.780+0000 7f27d6ffd700 1 -- 172.21.15.113:0/4258118383 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f27e0107950 con 0x7f27cc06caa0 2024-04-12T05:31:26.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.782+0000 7f27e8248700 1 -- 172.21.15.113:0/4258118383 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27cc06caa0 msgr2=0x7f27cc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:26.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.782+0000 7f27e8248700 1 --2- 172.21.15.113:0/4258118383 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27cc06caa0 0x7f27cc06ef60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f27e0108810 tx=0x7f27dc00b040).stop 2024-04-12T05:31:26.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.782+0000 7f27e8248700 1 -- 172.21.15.113:0/4258118383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00fc7f0 msgr2=0x7f27e010ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:26.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.782+0000 7f27e8248700 1 --2- 172.21.15.113:0/4258118383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00fc7f0 0x7f27e010ced0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f27d0002770 tx=0x7f27d0002a80).stop 2024-04-12T05:31:26.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.782+0000 7f27e8248700 1 -- 172.21.15.113:0/4258118383 shutdown_connections 2024-04-12T05:31:26.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.782+0000 7f27e8248700 1 --2- 172.21.15.113:0/4258118383 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27cc06caa0 0x7f27cc06ef60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.782+0000 7f27e8248700 1 --2- 172.21.15.113:0/4258118383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27e00fc7f0 0x7f27e010ced0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.782+0000 7f27e8248700 1 --2- 172.21.15.113:0/4258118383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27e00fd150 0x7f27e010d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.782+0000 7f27e8248700 1 -- 172.21.15.113:0/4258118383 >> 172.21.15.113:0/4258118383 conn(0x7f27e0070590 msgr2=0x7f27e00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:26.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.782+0000 7f27e8248700 1 -- 172.21.15.113:0/4258118383 shutdown_connections 2024-04-12T05:31:26.783 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.782+0000 7f27e8248700 1 -- 172.21.15.113:0/4258118383 wait complete. 2024-04-12T05:31:26.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.880+0000 7fa373071700 1 -- 172.21.15.113:0/2062600684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36c0fe2d0 msgr2=0x7fa36c0f7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.880+0000 7fa373071700 1 --2- 172.21.15.113:0/2062600684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36c0fe2d0 0x7fa36c0f7ab0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fa3580047f0 tx=0x7fa358009cb0).stop 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.880+0000 7fa373071700 1 -- 172.21.15.113:0/2062600684 shutdown_connections 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.880+0000 7fa373071700 1 --2- 172.21.15.113:0/2062600684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36c0fe2d0 0x7fa36c0f7ab0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.880+0000 7fa373071700 1 --2- 172.21.15.113:0/2062600684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c0fd970 0x7fa36c0fdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.880+0000 7fa373071700 1 -- 172.21.15.113:0/2062600684 >> 172.21.15.113:0/2062600684 conn(0x7fa36c0f3550 msgr2=0x7fa36c0f5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.880+0000 7fa373071700 1 -- 172.21.15.113:0/2062600684 shutdown_connections 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.880+0000 7fa373071700 1 -- 172.21.15.113:0/2062600684 wait complete. 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.880+0000 7fa373071700 1 Processor -- start 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.880+0000 7fa373071700 1 -- start start 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa373071700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c0fd970 0x7fa36c10e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa373071700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36c0fe2d0 0x7fa36c10e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa373071700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c10ebf0 con 0x7fa36c0fe2d0 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa373071700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c108210 con 0x7fa36c0fd970 2024-04-12T05:31:26.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa36bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36c0fe2d0 0x7fa36c10e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa36bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36c0fe2d0 0x7fa36c10e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35852/0 (socket says 172.21.15.113:35852) 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa36bfff700 1 -- 172.21.15.113:0/1846586805 learned_addr learned my addr 172.21.15.113:0/1846586805 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa36bfff700 1 -- 172.21.15.113:0/1846586805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c0fd970 msgr2=0x7fa36c10e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa36bfff700 1 --2- 172.21.15.113:0/1846586805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c0fd970 0x7fa36c10e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa36bfff700 1 -- 172.21.15.113:0/1846586805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa358005040 con 0x7fa36c0fe2d0 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa36bfff700 1 --2- 172.21.15.113:0/1846586805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36c0fe2d0 0x7fa36c10e560 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fa3580099f0 tx=0x7fa358009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa369ffb700 1 -- 172.21.15.113:0/1846586805 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa35800e030 con 0x7fa36c0fe2d0 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa369ffb700 1 -- 172.21.15.113:0/1846586805 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa358007e90 con 0x7fa36c0fe2d0 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa373071700 1 -- 172.21.15.113:0/1846586805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36c108460 con 0x7fa36c0fe2d0 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.881+0000 7fa369ffb700 1 -- 172.21.15.113:0/1846586805 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa358018070 con 0x7fa36c0fe2d0 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.882+0000 7fa373071700 1 -- 172.21.15.113:0/1846586805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36c108940 con 0x7fa36c0fe2d0 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.882+0000 7fa373071700 1 -- 172.21.15.113:0/1846586805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36c060370 con 0x7fa36c0fe2d0 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.883+0000 7fa369ffb700 1 -- 172.21.15.113:0/1846586805 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa35801d420 con 0x7fa36c0fe2d0 2024-04-12T05:31:26.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.883+0000 7fa369ffb700 1 --2- 172.21.15.113:0/1846586805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa35c06caa0 0x7fa35c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:26.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.883+0000 7fa369ffb700 1 -- 172.21.15.113:0/1846586805 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa35808d2e0 con 0x7fa36c0fe2d0 2024-04-12T05:31:26.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.883+0000 7fa370e0d700 1 --2- 172.21.15.113:0/1846586805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa35c06caa0 0x7fa35c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:26.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.884+0000 7fa370e0d700 1 --2- 172.21.15.113:0/1846586805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa35c06caa0 0x7fa35c06ef60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fa360009700 tx=0x7fa360006c30).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:26.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:26.888+0000 7fa369ffb700 1 -- 172.21.15.113:0/1846586805 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa35805ad90 con 0x7fa36c0fe2d0 2024-04-12T05:31:27.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.085+0000 7fa373071700 1 -- 172.21.15.113:0/1846586805 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa36c1096f0 con 0x7fa35c06caa0 2024-04-12T05:31:27.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.095+0000 7fa369ffb700 1 -- 172.21.15.113:0/1846586805 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa36c1096f0 con 0x7fa35c06caa0 2024-04-12T05:31:27.096 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:31:27.096 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (19m) 12m ago 30m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:31:27.096 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (16m) 12m ago 30m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:31:27.096 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (16m) 11m ago 29m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:31:27.096 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (19m) 12m ago 30m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:31:27.096 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (12m) 12m ago 25m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:31:27.096 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (12m) 12m ago 25m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:31:27.096 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (11m) 11m ago 25m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:31:27.096 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (25m) 11m ago 25m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:31:27.097 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (20m) 12m ago 31m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:31:27.097 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (20m) 11m ago 29m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:31:27.097 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (17m) 12m ago 31m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:31:27.097 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (17m) 11m ago 29m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:31:27.097 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (19m) 12m ago 30m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:31:27.097 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (19m) 11m ago 29m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:31:27.097 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (16m) 12m ago 29m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:31:27.097 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (15m) 12m ago 28m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:31:27.097 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (15m) 12m ago 28m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:31:27.097 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (14m) 11m ago 27m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:31:27.097 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (13m) 11m ago 27m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:31:27.098 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (12m) 11m ago 26m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:31:27.098 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (19m) 11m ago 29m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:31:27.098 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.097+0000 7fa373071700 1 -- 172.21.15.113:0/1846586805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa35c06caa0 msgr2=0x7fa35c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.097+0000 7fa373071700 1 --2- 172.21.15.113:0/1846586805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa35c06caa0 0x7fa35c06ef60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fa360009700 tx=0x7fa360006c30).stop 2024-04-12T05:31:27.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.097+0000 7fa373071700 1 -- 172.21.15.113:0/1846586805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36c0fe2d0 msgr2=0x7fa36c10e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.097+0000 7fa373071700 1 --2- 172.21.15.113:0/1846586805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36c0fe2d0 0x7fa36c10e560 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fa3580099f0 tx=0x7fa358009cb0).stop 2024-04-12T05:31:27.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.097+0000 7fa373071700 1 -- 172.21.15.113:0/1846586805 shutdown_connections 2024-04-12T05:31:27.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.097+0000 7fa373071700 1 --2- 172.21.15.113:0/1846586805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa35c06caa0 0x7fa35c06ef60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.097+0000 7fa373071700 1 --2- 172.21.15.113:0/1846586805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c0fd970 0x7fa36c10e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:31:27.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.097+0000 7fa373071700 1 --2- 172.21.15.113:0/1846586805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa36c0fe2d0 0x7fa36c10e560 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.097+0000 7fa373071700 1 -- 172.21.15.113:0/1846586805 >> 172.21.15.113:0/1846586805 conn(0x7fa36c0f3550 msgr2=0x7fa36c0fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:27.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.097+0000 7fa373071700 1 -- 172.21.15.113:0/1846586805 shutdown_connections 2024-04-12T05:31:27.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.097+0000 7fa373071700 1 -- 172.21.15.113:0/1846586805 wait complete. 2024-04-12T05:31:27.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.195+0000 7f2f98e4a700 1 -- 172.21.15.113:0/801743932 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f940fc7f0 msgr2=0x7f2f940fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.195+0000 7f2f98e4a700 1 --2- 172.21.15.113:0/801743932 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f940fc7f0 0x7f2f940fcbd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f2f880047f0 tx=0x7f2f88009cb0).stop 2024-04-12T05:31:27.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.195+0000 7f2f98e4a700 1 -- 172.21.15.113:0/801743932 shutdown_connections 2024-04-12T05:31:27.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.195+0000 7f2f98e4a700 1 --2- 172.21.15.113:0/801743932 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f940fd150 0x7f2f940f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.195+0000 7f2f98e4a700 1 --2- 172.21.15.113:0/801743932 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f940fc7f0 0x7f2f940fcbd0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.195+0000 7f2f98e4a700 1 -- 172.21.15.113:0/801743932 >> 172.21.15.113:0/801743932 conn(0x7f2f94070590 msgr2=0x7f2f940f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:27.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.195+0000 7f2f98e4a700 1 -- 172.21.15.113:0/801743932 shutdown_connections 2024-04-12T05:31:27.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.195+0000 7f2f98e4a700 1 -- 172.21.15.113:0/801743932 wait complete. 2024-04-12T05:31:27.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f98e4a700 1 Processor -- start 2024-04-12T05:31:27.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f98e4a700 1 -- start start 2024-04-12T05:31:27.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f98e4a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f940fc7f0 0x7f2f940f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:27.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f98e4a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f940fd150 0x7f2f940f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f98e4a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f940f8f20 con 0x7f2f940fc7f0 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f98e4a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f940f9060 con 0x7f2f940fd150 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f9259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f940fc7f0 0x7f2f940f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f9259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f940fc7f0 0x7f2f940f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35872/0 (socket says 172.21.15.113:35872) 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f9259c700 1 -- 172.21.15.113:0/3270920137 learned_addr learned my addr 172.21.15.113:0/3270920137 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f91d9b700 1 --2- 172.21.15.113:0/3270920137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f940fd150 0x7f2f940f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f9259c700 1 -- 172.21.15.113:0/3270920137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f940fd150 msgr2=0x7f2f940f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f9259c700 1 --2- 172.21.15.113:0/3270920137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f940fd150 0x7f2f940f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.196+0000 7f2f9259c700 1 -- 172.21.15.113:0/3270920137 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f88005040 con 0x7f2f940fc7f0 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.197+0000 7f2f9259c700 1 --2- 172.21.15.113:0/3270920137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f940fc7f0 0x7f2f940f8540 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f2f8800b120 tx=0x7f2f88009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.197+0000 7f2f837fe700 1 -- 172.21.15.113:0/3270920137 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f8800e030 con 0x7f2f940fc7f0 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.197+0000 7f2f98e4a700 1 -- 172.21.15.113:0/3270920137 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f940f92b0 con 0x7f2f940fc7f0 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.197+0000 7f2f837fe700 1 -- 172.21.15.113:0/3270920137 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f88007e90 con 0x7f2f940fc7f0 2024-04-12T05:31:27.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.197+0000 7f2f837fe700 1 -- 172.21.15.113:0/3270920137 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f88018070 con 0x7f2f940fc7f0 2024-04-12T05:31:27.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.197+0000 7f2f98e4a700 1 -- 172.21.15.113:0/3270920137 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f94114600 con 0x7f2f940fc7f0 2024-04-12T05:31:27.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.198+0000 7f2f98e4a700 1 -- 172.21.15.113:0/3270920137 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9405f590 con 0x7f2f940fc7f0 2024-04-12T05:31:27.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.198+0000 7f2f837fe700 1 -- 172.21.15.113:0/3270920137 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2f8801d420 con 0x7f2f940fc7f0 2024-04-12T05:31:27.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.198+0000 7f2f837fe700 1 --2- 172.21.15.113:0/3270920137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f7c06caa0 0x7f2f7c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:27.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.199+0000 7f2f837fe700 1 -- 172.21.15.113:0/3270920137 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2f8808d160 con 0x7f2f940fc7f0 2024-04-12T05:31:27.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.199+0000 7f2f91d9b700 1 --2- 172.21.15.113:0/3270920137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f7c06caa0 0x7f2f7c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:27.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.199+0000 7f2f91d9b700 1 --2- 172.21.15.113:0/3270920137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f7c06caa0 0x7f2f7c06ef60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f2f940f9ce0 tx=0x7f2f84009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:27.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.203+0000 7f2f837fe700 1 -- 172.21.15.113:0/3270920137 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2f88057180 con 0x7f2f940fc7f0 2024-04-12T05:31:27.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:27 smithi113 ceph-mon[103371]: pgmap v578: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:27.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:27 smithi192 ceph-mon[85304]: pgmap v578: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:27.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.468+0000 7f2f98e4a700 1 -- 172.21.15.113:0/3270920137 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2f94114c60 con 0x7f2f940fc7f0 2024-04-12T05:31:27.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.469+0000 7f2f837fe700 1 -- 172.21.15.113:0/3270920137 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f2f8805a7a0 con 0x7f2f940fc7f0 2024-04-12T05:31:27.469 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:31:27.469 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:31:27.469 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:31:27.469 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:31:27.470 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:31:27.471 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:31:27.471 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:31:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.471+0000 7f2f98e4a700 1 -- 172.21.15.113:0/3270920137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f7c06caa0 msgr2=0x7f2f7c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.471+0000 7f2f98e4a700 1 --2- 172.21.15.113:0/3270920137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f7c06caa0 0x7f2f7c06ef60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f2f940f9ce0 tx=0x7f2f84009040).stop 2024-04-12T05:31:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.471+0000 7f2f98e4a700 1 -- 172.21.15.113:0/3270920137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f940fc7f0 msgr2=0x7f2f940f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.471+0000 7f2f98e4a700 1 --2- 172.21.15.113:0/3270920137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f940fc7f0 0x7f2f940f8540 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f2f8800b120 tx=0x7f2f88009cb0).stop 2024-04-12T05:31:27.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.471+0000 7f2f98e4a700 1 -- 172.21.15.113:0/3270920137 shutdown_connections 2024-04-12T05:31:27.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.471+0000 7f2f98e4a700 1 --2- 172.21.15.113:0/3270920137 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2f7c06caa0 0x7f2f7c06ef60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.471+0000 7f2f98e4a700 1 --2- 172.21.15.113:0/3270920137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f940fc7f0 0x7f2f940f8540 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.471+0000 7f2f98e4a700 1 --2- 172.21.15.113:0/3270920137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f940fd150 0x7f2f940f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.471+0000 7f2f98e4a700 1 -- 172.21.15.113:0/3270920137 >> 172.21.15.113:0/3270920137 conn(0x7f2f94070590 msgr2=0x7f2f94060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:27.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.471+0000 7f2f98e4a700 1 -- 172.21.15.113:0/3270920137 shutdown_connections 2024-04-12T05:31:27.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.471+0000 7f2f98e4a700 1 -- 172.21.15.113:0/3270920137 wait complete. 2024-04-12T05:31:27.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.569+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/2015472056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5a40fcef0 msgr2=0x7ff5a40f66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.569+0000 7ff5a9f9f700 1 --2- 172.21.15.113:0/2015472056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5a40fcef0 0x7ff5a40f66d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7ff5900047f0 tx=0x7ff590009cb0).stop 2024-04-12T05:31:27.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.569+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/2015472056 shutdown_connections 2024-04-12T05:31:27.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.569+0000 7ff5a9f9f700 1 --2- 172.21.15.113:0/2015472056 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5a40fcef0 0x7ff5a40f66d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.569+0000 7ff5a9f9f700 1 --2- 172.21.15.113:0/2015472056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5a40fc590 0x7ff5a40fc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.569+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/2015472056 >> 172.21.15.113:0/2015472056 conn(0x7ff5a40f2150 msgr2=0x7ff5a40f4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:27.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.569+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/2015472056 shutdown_connections 2024-04-12T05:31:27.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.569+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/2015472056 wait complete. 2024-04-12T05:31:27.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.569+0000 7ff5a9f9f700 1 Processor -- start 2024-04-12T05:31:27.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a9f9f700 1 -- start start 2024-04-12T05:31:27.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a9f9f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5a40fc590 0x7ff5a4106d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:27.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a9f9f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5a40fcef0 0x7ff5a4107230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:27.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a9f9f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4107720 con 0x7ff5a40fc590 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a9f9f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4107860 con 0x7ff5a40fcef0 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a8f9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5a40fc590 0x7ff5a4106d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a8f9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5a40fc590 0x7ff5a4106d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35892/0 (socket says 172.21.15.113:35892) 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a8f9d700 1 -- 172.21.15.113:0/3587353273 learned_addr learned my addr 172.21.15.113:0/3587353273 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a3fff700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5a40fcef0 0x7ff5a4107230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a8f9d700 1 -- 172.21.15.113:0/3587353273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5a40fcef0 msgr2=0x7ff5a4107230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a8f9d700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5a40fcef0 0x7ff5a4107230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a8f9d700 1 -- 172.21.15.113:0/3587353273 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590005040 con 0x7ff5a40fc590 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a3fff700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5a40fcef0 0x7ff5a4107230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a8f9d700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5a40fc590 0x7ff5a4106d40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7ff59800e8f0 tx=0x7ff59800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a1ffb700 1 -- 172.21.15.113:0/3587353273 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff59800ee90 con 0x7ff5a40fc590 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a1ffb700 1 -- 172.21.15.113:0/3587353273 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff59800f040 con 0x7ff5a40fc590 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/3587353273 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a4107b40 con 0x7ff5a40fc590 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.570+0000 7ff5a1ffb700 1 -- 172.21.15.113:0/3587353273 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff59800c8a0 con 0x7ff5a40fc590 2024-04-12T05:31:27.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.571+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/3587353273 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a4114780 con 0x7ff5a40fc590 2024-04-12T05:31:27.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.572+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/3587353273 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a405ffc0 con 0x7ff5a40fc590 2024-04-12T05:31:27.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.572+0000 7ff5a1ffb700 1 -- 172.21.15.113:0/3587353273 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff598007b40 con 0x7ff5a40fc590 2024-04-12T05:31:27.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.572+0000 7ff5a1ffb700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff59406caa0 0x7ff59406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:27.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.572+0000 7ff5a1ffb700 1 -- 172.21.15.113:0/3587353273 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff59808bb90 con 0x7ff5a40fc590 2024-04-12T05:31:27.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.572+0000 7ff5a3fff700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff59406caa0 0x7ff59406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:27.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.573+0000 7ff5a3fff700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff59406caa0 0x7ff59406ef60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7ff5a4061310 tx=0x7ff59000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:27.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.577+0000 7ff5a1ffb700 1 -- 172.21.15.113:0/3587353273 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff5980595c0 con 0x7ff5a40fc590 2024-04-12T05:31:27.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.805+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/3587353273 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff5a4061050 con 0x7ff5a40fc590 2024-04-12T05:31:27.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.805+0000 7ff5a1ffb700 1 -- 172.21.15.113:0/3587353273 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7ff598059150 con 0x7ff5a40fc590 2024-04-12T05:31:27.806 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:31:27.806 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:31:27.806 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:31:27.806 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:31:27.806 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:31:27.806 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:31:27.806 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:31:27.806 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:31:27.806 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:31:27.806 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:31:27.806 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:31:27.807 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:31:27.808 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:31:27.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.807+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/3587353273 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff59406caa0 msgr2=0x7ff59406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.807+0000 7ff5a9f9f700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff59406caa0 0x7ff59406ef60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7ff5a4061310 tx=0x7ff59000b040).stop 2024-04-12T05:31:27.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.807+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/3587353273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5a40fc590 msgr2=0x7ff5a4106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.807+0000 7ff5a9f9f700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5a40fc590 0x7ff5a4106d40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7ff59800e8f0 tx=0x7ff59800b040).stop 2024-04-12T05:31:27.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.807+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/3587353273 shutdown_connections 2024-04-12T05:31:27.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.807+0000 7ff5a9f9f700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff59406caa0 0x7ff59406ef60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.807+0000 7ff5a9f9f700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5a40fc590 0x7ff5a4106d40 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.807+0000 7ff5a9f9f700 1 --2- 172.21.15.113:0/3587353273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5a40fcef0 0x7ff5a4107230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.807+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/3587353273 >> 172.21.15.113:0/3587353273 conn(0x7ff5a40f2150 msgr2=0x7ff5a40fa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:27.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.807+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/3587353273 shutdown_connections 2024-04-12T05:31:27.810 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.807+0000 7ff5a9f9f700 1 -- 172.21.15.113:0/3587353273 wait complete. 2024-04-12T05:31:27.810 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:31:27.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.906+0000 7ff2927f4700 1 -- 172.21.15.113:0/1679567284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28c0fe270 msgr2=0x7ff28c0f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.906+0000 7ff2927f4700 1 --2- 172.21.15.113:0/1679567284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28c0fe270 0x7ff28c0f7a50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7ff27c0047f0 tx=0x7ff27c009cb0).stop 2024-04-12T05:31:27.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.906+0000 7ff2927f4700 1 -- 172.21.15.113:0/1679567284 shutdown_connections 2024-04-12T05:31:27.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.906+0000 7ff2927f4700 1 --2- 172.21.15.113:0/1679567284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28c0fe270 0x7ff28c0f7a50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.906+0000 7ff2927f4700 1 --2- 172.21.15.113:0/1679567284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff28c0fd910 0x7ff28c0fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.906+0000 7ff2927f4700 1 -- 172.21.15.113:0/1679567284 >> 172.21.15.113:0/1679567284 conn(0x7ff28c0f3530 msgr2=0x7ff28c0f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:27.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.906+0000 7ff2927f4700 1 -- 172.21.15.113:0/1679567284 shutdown_connections 2024-04-12T05:31:27.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.906+0000 7ff2927f4700 1 -- 172.21.15.113:0/1679567284 wait complete. 2024-04-12T05:31:27.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.906+0000 7ff2927f4700 1 Processor -- start 2024-04-12T05:31:27.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.906+0000 7ff2927f4700 1 -- start start 2024-04-12T05:31:27.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff2927f4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff28c0fd910 0x7ff28c10dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:27.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff2927f4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28c0fe270 0x7ff28c10e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:27.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff2927f4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c10eab0 con 0x7ff28c0fe270 2024-04-12T05:31:27.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff2927f4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c10ebf0 con 0x7ff28c0fd910 2024-04-12T05:31:27.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff28b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28c0fe270 0x7ff28c10e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff28b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28c0fe270 0x7ff28c10e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35908/0 (socket says 172.21.15.113:35908) 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff28b7fe700 1 -- 172.21.15.113:0/2617790427 learned_addr learned my addr 172.21.15.113:0/2617790427 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff28bfff700 1 --2- 172.21.15.113:0/2617790427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff28c0fd910 0x7ff28c10dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff28b7fe700 1 -- 172.21.15.113:0/2617790427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff28c0fd910 msgr2=0x7ff28c10dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff28b7fe700 1 --2- 172.21.15.113:0/2617790427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff28c0fd910 0x7ff28c10dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff28b7fe700 1 -- 172.21.15.113:0/2617790427 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c005040 con 0x7ff28c0fe270 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff28b7fe700 1 --2- 172.21.15.113:0/2617790427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28c0fe270 0x7ff28c10e4b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7ff27c0099f0 tx=0x7ff27c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff2897fa700 1 -- 172.21.15.113:0/2617790427 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff27c00e030 con 0x7ff28c0fe270 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff2897fa700 1 -- 172.21.15.113:0/2617790427 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff27c007e90 con 0x7ff28c0fe270 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff2927f4700 1 -- 172.21.15.113:0/2617790427 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28c108220 con 0x7ff28c0fe270 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff2897fa700 1 -- 172.21.15.113:0/2617790427 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff27c018070 con 0x7ff28c0fe270 2024-04-12T05:31:27.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.907+0000 7ff2927f4700 1 -- 172.21.15.113:0/2617790427 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28c108780 con 0x7ff28c0fe270 2024-04-12T05:31:27.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.908+0000 7ff2927f4700 1 -- 172.21.15.113:0/2617790427 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28c05f570 con 0x7ff28c0fe270 2024-04-12T05:31:27.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.909+0000 7ff2897fa700 1 -- 172.21.15.113:0/2617790427 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff27c009590 con 0x7ff28c0fe270 2024-04-12T05:31:27.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.909+0000 7ff2897fa700 1 --2- 172.21.15.113:0/2617790427 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff27406ca30 0x7ff27406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:27.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.909+0000 7ff2897fa700 1 -- 172.21.15.113:0/2617790427 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff27c08d0f0 con 0x7ff28c0fe270 2024-04-12T05:31:27.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.909+0000 7ff28bfff700 1 --2- 172.21.15.113:0/2617790427 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff27406ca30 0x7ff27406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:27.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.910+0000 7ff28bfff700 1 --2- 172.21.15.113:0/2617790427 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff27406ca30 0x7ff27406eef0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff280009170 tx=0x7ff280009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:27.914+0000 7ff2897fa700 1 -- 172.21.15.113:0/2617790427 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff27c05aba0 con 0x7ff28c0fe270 2024-04-12T05:31:28.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.120+0000 7ff2927f4700 1 -- 172.21.15.113:0/2617790427 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff28c0ffc60 con 0x7ff27406ca30 2024-04-12T05:31:28.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.121+0000 7ff2897fa700 1 -- 172.21.15.113:0/2617790427 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff28c0ffc60 con 0x7ff27406ca30 2024-04-12T05:31:28.121 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:31:28.121 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:31:28.121 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:31:28.121 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:31:28.121 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:31:28.121 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:31:28.121 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:31:28.121 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:31:28.121 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:31:28.122 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:31:28.122 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:31:28.122 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:31:28.122 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:31:28.122 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:31:28.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.123+0000 7ff2927f4700 1 -- 172.21.15.113:0/2617790427 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff27406ca30 msgr2=0x7ff27406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:28.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.123+0000 7ff2927f4700 1 --2- 172.21.15.113:0/2617790427 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff27406ca30 0x7ff27406eef0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff280009170 tx=0x7ff280009da0).stop 2024-04-12T05:31:28.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.123+0000 7ff2927f4700 1 -- 172.21.15.113:0/2617790427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28c0fe270 msgr2=0x7ff28c10e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:28.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.123+0000 7ff2927f4700 1 --2- 172.21.15.113:0/2617790427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28c0fe270 0x7ff28c10e4b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7ff27c0099f0 tx=0x7ff27c009cb0).stop 2024-04-12T05:31:28.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.123+0000 7ff2927f4700 1 -- 172.21.15.113:0/2617790427 shutdown_connections 2024-04-12T05:31:28.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.123+0000 7ff2927f4700 1 --2- 172.21.15.113:0/2617790427 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff27406ca30 0x7ff27406eef0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:28.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.123+0000 7ff2927f4700 1 --2- 172.21.15.113:0/2617790427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff28c0fd910 0x7ff28c10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:28.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.123+0000 7ff2927f4700 1 --2- 172.21.15.113:0/2617790427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff28c0fe270 0x7ff28c10e4b0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:28.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.123+0000 7ff2927f4700 1 -- 172.21.15.113:0/2617790427 >> 172.21.15.113:0/2617790427 conn(0x7ff28c0f3530 msgr2=0x7ff28c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:28.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.123+0000 7ff2927f4700 1 -- 172.21.15.113:0/2617790427 shutdown_connections 2024-04-12T05:31:28.124 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.123+0000 7ff2927f4700 1 -- 172.21.15.113:0/2617790427 wait complete. 2024-04-12T05:31:28.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.220+0000 7efd15506700 1 -- 172.21.15.113:0/18103383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd100fd050 msgr2=0x7efd100f6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:28.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.220+0000 7efd15506700 1 --2- 172.21.15.113:0/18103383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd100fd050 0x7efd100f6830 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7efd000047f0 tx=0x7efd00009cb0).stop 2024-04-12T05:31:28.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.220+0000 7efd15506700 1 -- 172.21.15.113:0/18103383 shutdown_connections 2024-04-12T05:31:28.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.220+0000 7efd15506700 1 --2- 172.21.15.113:0/18103383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd100fd050 0x7efd100f6830 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:28.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.220+0000 7efd15506700 1 --2- 172.21.15.113:0/18103383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd100fc6f0 0x7efd100fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:31:28.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.220+0000 7efd15506700 1 -- 172.21.15.113:0/18103383 >> 172.21.15.113:0/18103383 conn(0x7efd10070470 msgr2=0x7efd100f47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:28.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.220+0000 7efd15506700 1 -- 172.21.15.113:0/18103383 shutdown_connections 2024-04-12T05:31:28.221 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.220+0000 7efd15506700 1 -- 172.21.15.113:0/18103383 wait complete. 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.221+0000 7efd15506700 1 Processor -- start 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.221+0000 7efd15506700 1 -- start start 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.221+0000 7efd15506700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd100fc6f0 0x7efd1010e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.221+0000 7efd15506700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd100fd050 0x7efd1010e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.221+0000 7efd15506700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1010ec40 con 0x7efd100fd050 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.221+0000 7efd15506700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd10108050 con 0x7efd100fc6f0 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.221+0000 7efd0e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd100fd050 0x7efd1010e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.221+0000 7efd0e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd100fd050 0x7efd1010e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35932/0 (socket says 172.21.15.113:35932) 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.221+0000 7efd0e7fc700 1 -- 172.21.15.113:0/3336801526 learned_addr learned my addr 172.21.15.113:0/3336801526 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.222+0000 7efd0effd700 1 --2- 172.21.15.113:0/3336801526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd100fc6f0 0x7efd1010e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.222+0000 7efd0e7fc700 1 -- 172.21.15.113:0/3336801526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd100fc6f0 msgr2=0x7efd1010e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:28.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.222+0000 7efd0e7fc700 1 --2- 172.21.15.113:0/3336801526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd100fc6f0 0x7efd1010e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:28.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.222+0000 7efd0e7fc700 1 -- 172.21.15.113:0/3336801526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd00005040 con 0x7efd100fd050 2024-04-12T05:31:28.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.222+0000 7efd0e7fc700 1 --2- 172.21.15.113:0/3336801526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd100fd050 0x7efd1010e5b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7efd000099f0 tx=0x7efd00009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:28.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.222+0000 7efcf7fff700 1 -- 172.21.15.113:0/3336801526 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd0000e030 con 0x7efd100fd050 2024-04-12T05:31:28.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.222+0000 7efcf7fff700 1 -- 172.21.15.113:0/3336801526 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd00007e90 con 0x7efd100fd050 2024-04-12T05:31:28.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.222+0000 7efcf7fff700 1 -- 172.21.15.113:0/3336801526 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd00018070 con 0x7efd100fd050 2024-04-12T05:31:28.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.222+0000 7efd15506700 1 -- 172.21.15.113:0/3336801526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd101082a0 con 0x7efd100fd050 2024-04-12T05:31:28.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.222+0000 7efd15506700 1 -- 172.21.15.113:0/3336801526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd10108780 con 0x7efd100fd050 2024-04-12T05:31:28.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.223+0000 7efd15506700 1 -- 172.21.15.113:0/3336801526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1005f510 con 0x7efd100fd050 2024-04-12T05:31:28.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.224+0000 7efcf7fff700 1 -- 172.21.15.113:0/3336801526 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7efd0001d420 con 0x7efd100fd050 2024-04-12T05:31:28.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.224+0000 7efcf7fff700 1 --2- 172.21.15.113:0/3336801526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efcf806caf0 0x7efcf806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:28.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.224+0000 7efcf7fff700 1 -- 172.21.15.113:0/3336801526 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7efd0001de40 con 0x7efd100fd050 2024-04-12T05:31:28.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.224+0000 7efd0effd700 1 --2- 172.21.15.113:0/3336801526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efcf806caf0 0x7efcf806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:28.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.224+0000 7efd0effd700 1 --2- 172.21.15.113:0/3336801526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efcf806caf0 0x7efcf806efb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7efd0400a880 tx=0x7efd04008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:28.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.228+0000 7efcf7fff700 1 -- 172.21.15.113:0/3336801526 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7efd0005b1f0 con 0x7efd100fd050 2024-04-12T05:31:28.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:28 smithi113 ceph-mon[103371]: from='client.35064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:28.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:28 smithi113 ceph-mon[103371]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:28.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:28 smithi113 ceph-mon[103371]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:28.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:28 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3270920137' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:31:28.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:28 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3587353273' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:31:28.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:28 smithi192 ceph-mon[85304]: from='client.35064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:28.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:28 smithi192 ceph-mon[85304]: from='client.35068 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:28.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:28 smithi192 ceph-mon[85304]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:28.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:28 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3270920137' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:31:28.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:28 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3587353273' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:31:28.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.494+0000 7efd15506700 1 -- 172.21.15.113:0/3336801526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efd100f5ec0 con 0x7efd100fd050 2024-04-12T05:31:28.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.495+0000 7efcf7fff700 1 -- 172.21.15.113:0/3336801526 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7efd00061020 con 0x7efd100fd050 2024-04-12T05:31:28.495 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:31:28.495 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:31:28.495 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:31:28.495 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:31:28.495 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:31:28.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.497+0000 7efd15506700 1 -- 172.21.15.113:0/3336801526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efcf806caf0 msgr2=0x7efcf806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:28.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.497+0000 7efd15506700 1 --2- 172.21.15.113:0/3336801526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efcf806caf0 0x7efcf806efb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7efd0400a880 tx=0x7efd04008040).stop 2024-04-12T05:31:28.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.497+0000 7efd15506700 1 -- 172.21.15.113:0/3336801526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd100fd050 msgr2=0x7efd1010e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:28.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.497+0000 7efd15506700 1 --2- 172.21.15.113:0/3336801526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd100fd050 0x7efd1010e5b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7efd000099f0 tx=0x7efd00009cb0).stop 2024-04-12T05:31:28.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.497+0000 7efd15506700 1 -- 172.21.15.113:0/3336801526 shutdown_connections 2024-04-12T05:31:28.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.497+0000 7efd15506700 1 --2- 172.21.15.113:0/3336801526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efcf806caf0 0x7efcf806efb0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:28.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.497+0000 7efd15506700 1 --2- 172.21.15.113:0/3336801526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd100fc6f0 0x7efd1010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:28.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.497+0000 7efd15506700 1 --2- 172.21.15.113:0/3336801526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd100fd050 0x7efd1010e5b0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:28.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.497+0000 7efd15506700 1 -- 172.21.15.113:0/3336801526 >> 172.21.15.113:0/3336801526 conn(0x7efd10070470 msgr2=0x7efd10060d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:28.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.497+0000 7efd15506700 1 -- 172.21.15.113:0/3336801526 shutdown_connections 2024-04-12T05:31:28.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:28.497+0000 7efd15506700 1 -- 172.21.15.113:0/3336801526 wait complete. 2024-04-12T05:31:29.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:29 smithi113 ceph-mon[103371]: pgmap v579: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:29.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:29 smithi113 ceph-mon[103371]: from='client.35084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:29.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:29 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3336801526' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:31:29.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:29 smithi192 ceph-mon[85304]: pgmap v579: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:29.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:29 smithi192 ceph-mon[85304]: from='client.35084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:31:29.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:29 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3336801526' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:31:31.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:31 smithi113 ceph-mon[103371]: pgmap v580: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:31.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:31 smithi192 ceph-mon[85304]: pgmap v580: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:33.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:33 smithi113 ceph-mon[103371]: pgmap v581: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:33.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:33 smithi192 ceph-mon[85304]: pgmap v581: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:35.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:35 smithi113 ceph-mon[103371]: pgmap v582: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:35.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:35 smithi192 ceph-mon[85304]: pgmap v582: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:37.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:37 smithi113 ceph-mon[103371]: pgmap v583: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:37.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:37.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:37 smithi192 ceph-mon[85304]: pgmap v583: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:37.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:39.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:39 smithi113 ceph-mon[103371]: pgmap v584: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:39.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:39 smithi192 ceph-mon[85304]: pgmap v584: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:41.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:41 smithi113 ceph-mon[103371]: pgmap v585: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:41.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:41 smithi192 ceph-mon[85304]: pgmap v585: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:43.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:43 smithi113 ceph-mon[103371]: pgmap v586: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:43.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:43 smithi192 ceph-mon[85304]: pgmap v586: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:45.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:45 smithi113 ceph-mon[103371]: pgmap v587: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:45.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:45 smithi192 ceph-mon[85304]: pgmap v587: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:47.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:47 smithi113 ceph-mon[103371]: pgmap v588: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:47.449 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:47 smithi192 ceph-mon[85304]: pgmap v588: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:49.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:49 smithi113 ceph-mon[103371]: pgmap v589: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:49.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:49 smithi192 ceph-mon[85304]: pgmap v589: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:51.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:51 smithi113 ceph-mon[103371]: pgmap v590: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:51.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:51 smithi192 ceph-mon[85304]: pgmap v590: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:52.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:52.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:31:53.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:53 smithi113 ceph-mon[103371]: pgmap v591: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:53.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:53 smithi192 ceph-mon[85304]: pgmap v591: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:55.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:55 smithi113 ceph-mon[103371]: pgmap v592: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:55.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:55 smithi192 ceph-mon[85304]: pgmap v592: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:57.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:57 smithi113 ceph-mon[103371]: pgmap v593: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:57.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:57 smithi192 ceph-mon[85304]: pgmap v593: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:58.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.608+0000 7f76a17fc700 1 -- 172.21.15.113:0/2438350668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f769c0fd150 msgr2=0x7f769c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:58.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.608+0000 7f76a17fc700 1 --2- 172.21.15.113:0/2438350668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f769c0fd150 0x7f769c0f6930 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f76900047f0 tx=0x7f7690009cb0).stop 2024-04-12T05:31:58.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.608+0000 7f76a17fc700 1 -- 172.21.15.113:0/2438350668 shutdown_connections 2024-04-12T05:31:58.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.608+0000 7f76a17fc700 1 --2- 172.21.15.113:0/2438350668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f769c0fd150 0x7f769c0f6930 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:58.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.608+0000 7f76a17fc700 1 --2- 172.21.15.113:0/2438350668 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f769c0fc7f0 0x7f769c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:58.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.608+0000 7f76a17fc700 1 -- 172.21.15.113:0/2438350668 >> 172.21.15.113:0/2438350668 conn(0x7f769c070590 msgr2=0x7f769c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:58.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.608+0000 7f76a17fc700 1 -- 172.21.15.113:0/2438350668 shutdown_connections 2024-04-12T05:31:58.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.608+0000 7f76a17fc700 1 -- 172.21.15.113:0/2438350668 wait complete. 2024-04-12T05:31:58.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.609+0000 7f76a17fc700 1 Processor -- start 2024-04-12T05:31:58.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.609+0000 7f76a17fc700 1 -- start start 2024-04-12T05:31:58.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.609+0000 7f76a17fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f769c0fc7f0 0x7f769c10cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:58.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.609+0000 7f76a17fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f769c0fd150 0x7f769c10d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:58.610 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.609+0000 7f76a17fc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c10d9e0 con 0x7f769c0fd150 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.609+0000 7f76a17fc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c10db20 con 0x7f769c0fc7f0 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.609+0000 7f769a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f769c0fd150 0x7f769c10d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.609+0000 7f769a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f769c0fd150 0x7f769c10d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40430/0 (socket says 172.21.15.113:40430) 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.609+0000 7f769a7fc700 1 -- 172.21.15.113:0/3451873488 learned_addr learned my addr 172.21.15.113:0/3451873488 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.610+0000 7f769affd700 1 --2- 172.21.15.113:0/3451873488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f769c0fc7f0 0x7f769c10cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.610+0000 7f769a7fc700 1 -- 172.21.15.113:0/3451873488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f769c0fc7f0 msgr2=0x7f769c10cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.610+0000 7f769a7fc700 1 --2- 172.21.15.113:0/3451873488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f769c0fc7f0 0x7f769c10cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.610+0000 7f769a7fc700 1 -- 172.21.15.113:0/3451873488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7690005040 con 0x7f769c0fd150 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.610+0000 7f769a7fc700 1 --2- 172.21.15.113:0/3451873488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f769c0fd150 0x7f769c10d3e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f76900099f0 tx=0x7f7690009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.610+0000 7f7683fff700 1 -- 172.21.15.113:0/3451873488 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f769000e030 con 0x7f769c0fd150 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.610+0000 7f76a17fc700 1 -- 172.21.15.113:0/3451873488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769c107150 con 0x7f769c0fd150 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.610+0000 7f7683fff700 1 -- 172.21.15.113:0/3451873488 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7690007e90 con 0x7f769c0fd150 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.610+0000 7f7683fff700 1 -- 172.21.15.113:0/3451873488 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7690018070 con 0x7f769c0fd150 2024-04-12T05:31:58.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.610+0000 7f76a17fc700 1 -- 172.21.15.113:0/3451873488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769c1076b0 con 0x7f769c0fd150 2024-04-12T05:31:58.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.611+0000 7f76a17fc700 1 -- 172.21.15.113:0/3451873488 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769c05f590 con 0x7f769c0fd150 2024-04-12T05:31:58.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.612+0000 7f7683fff700 1 -- 172.21.15.113:0/3451873488 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f769001d420 con 0x7f769c0fd150 2024-04-12T05:31:58.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.612+0000 7f7683fff700 1 --2- 172.21.15.113:0/3451873488 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f768406caa0 0x7f768406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:58.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.612+0000 7f7683fff700 1 -- 172.21.15.113:0/3451873488 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f769008d230 con 0x7f769c0fd150 2024-04-12T05:31:58.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.612+0000 7f769affd700 1 --2- 172.21.15.113:0/3451873488 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f768406caa0 0x7f768406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:58.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.613+0000 7f769affd700 1 --2- 172.21.15.113:0/3451873488 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f768406caa0 0x7f768406ef60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f768c0041d0 tx=0x7f768c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:58.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.617+0000 7f7683fff700 1 -- 172.21.15.113:0/3451873488 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f769005ace0 con 0x7f769c0fd150 2024-04-12T05:31:58.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.820+0000 7f76a17fc700 1 -- 172.21.15.113:0/3451873488 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f769c1085e0 con 0x7f768406caa0 2024-04-12T05:31:58.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.821+0000 7f7683fff700 1 -- 172.21.15.113:0/3451873488 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f769c1085e0 con 0x7f768406caa0 2024-04-12T05:31:58.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.823+0000 7f76a17fc700 1 -- 172.21.15.113:0/3451873488 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f768406caa0 msgr2=0x7f768406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:58.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.823+0000 7f76a17fc700 1 --2- 172.21.15.113:0/3451873488 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f768406caa0 0x7f768406ef60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f768c0041d0 tx=0x7f768c009040).stop 2024-04-12T05:31:58.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.823+0000 7f76a17fc700 1 -- 172.21.15.113:0/3451873488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f769c0fd150 msgr2=0x7f769c10d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:58.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.823+0000 7f76a17fc700 1 --2- 172.21.15.113:0/3451873488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f769c0fd150 0x7f769c10d3e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f76900099f0 tx=0x7f7690009cb0).stop 2024-04-12T05:31:58.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.823+0000 7f76a17fc700 1 -- 172.21.15.113:0/3451873488 shutdown_connections 2024-04-12T05:31:58.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.823+0000 7f76a17fc700 1 --2- 172.21.15.113:0/3451873488 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f768406caa0 0x7f768406ef60 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:58.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.823+0000 7f76a17fc700 1 --2- 172.21.15.113:0/3451873488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f769c0fc7f0 0x7f769c10cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:58.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.823+0000 7f76a17fc700 1 --2- 172.21.15.113:0/3451873488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f769c0fd150 0x7f769c10d3e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:58.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.823+0000 7f76a17fc700 1 -- 172.21.15.113:0/3451873488 >> 172.21.15.113:0/3451873488 conn(0x7f769c070590 msgr2=0x7f769c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:58.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.823+0000 7f76a17fc700 1 -- 172.21.15.113:0/3451873488 shutdown_connections 2024-04-12T05:31:58.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.824+0000 7f76a17fc700 1 -- 172.21.15.113:0/3451873488 wait complete. 2024-04-12T05:31:58.833 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:31:58.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.922+0000 7fa57945f700 1 -- 172.21.15.113:0/288479565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5740fd970 msgr2=0x7fa5740fdd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:58.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.922+0000 7fa57945f700 1 --2- 172.21.15.113:0/288479565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5740fd970 0x7fa5740fdd50 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7fa5640047f0 tx=0x7fa564009cb0).stop 2024-04-12T05:31:58.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.922+0000 7fa57945f700 1 -- 172.21.15.113:0/288479565 shutdown_connections 2024-04-12T05:31:58.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.922+0000 7fa57945f700 1 --2- 172.21.15.113:0/288479565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5740fe2d0 0x7fa5740f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:58.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.922+0000 7fa57945f700 1 --2- 172.21.15.113:0/288479565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5740fd970 0x7fa5740fdd50 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:58.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.922+0000 7fa57945f700 1 -- 172.21.15.113:0/288479565 >> 172.21.15.113:0/288479565 conn(0x7fa5740f3550 msgr2=0x7fa5740f5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.922+0000 7fa57945f700 1 -- 172.21.15.113:0/288479565 shutdown_connections 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.922+0000 7fa57945f700 1 -- 172.21.15.113:0/288479565 wait complete. 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.923+0000 7fa57945f700 1 Processor -- start 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.923+0000 7fa57945f700 1 -- start start 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.923+0000 7fa57945f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5740fd970 0x7fa57410dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.923+0000 7fa57945f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5740fe2d0 0x7fa57410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.923+0000 7fa57945f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57410ead0 con 0x7fa5740fe2d0 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.923+0000 7fa57945f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57410ec10 con 0x7fa5740fd970 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.923+0000 7fa5727fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5740fe2d0 0x7fa57410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa5727fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5740fe2d0 0x7fa57410e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40450/0 (socket says 172.21.15.113:40450) 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa5727fc700 1 -- 172.21.15.113:0/2844187911 learned_addr learned my addr 172.21.15.113:0/2844187911 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:58.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa572ffd700 1 --2- 172.21.15.113:0/2844187911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5740fd970 0x7fa57410dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:58.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa5727fc700 1 -- 172.21.15.113:0/2844187911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5740fd970 msgr2=0x7fa57410dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:58.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa5727fc700 1 --2- 172.21.15.113:0/2844187911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5740fd970 0x7fa57410dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:58.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa5727fc700 1 -- 172.21.15.113:0/2844187911 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa564005040 con 0x7fa5740fe2d0 2024-04-12T05:31:58.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa5727fc700 1 --2- 172.21.15.113:0/2844187911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5740fe2d0 0x7fa57410e4d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7fa568002820 tx=0x7fa568009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:58.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa55bfff700 1 -- 172.21.15.113:0/2844187911 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa568007b40 con 0x7fa5740fe2d0 2024-04-12T05:31:58.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa55bfff700 1 -- 172.21.15.113:0/2844187911 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa56800f040 con 0x7fa5740fe2d0 2024-04-12T05:31:58.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa57945f700 1 -- 172.21.15.113:0/2844187911 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5741082a0 con 0x7fa5740fe2d0 2024-04-12T05:31:58.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa55bfff700 1 -- 172.21.15.113:0/2844187911 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5680148b0 con 0x7fa5740fe2d0 2024-04-12T05:31:58.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.924+0000 7fa57945f700 1 -- 172.21.15.113:0/2844187911 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa574108860 con 0x7fa5740fe2d0 2024-04-12T05:31:58.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.925+0000 7fa57945f700 1 -- 172.21.15.113:0/2844187911 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa574060370 con 0x7fa5740fe2d0 2024-04-12T05:31:58.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.925+0000 7fa55bfff700 1 -- 172.21.15.113:0/2844187911 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa568007ca0 con 0x7fa5740fe2d0 2024-04-12T05:31:58.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.926+0000 7fa55bfff700 1 --2- 172.21.15.113:0/2844187911 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa55c06caa0 0x7fa55c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:58.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.926+0000 7fa55bfff700 1 -- 172.21.15.113:0/2844187911 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa56808b9b0 con 0x7fa5740fe2d0 2024-04-12T05:31:58.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.926+0000 7fa572ffd700 1 --2- 172.21.15.113:0/2844187911 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa55c06caa0 0x7fa55c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:58.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.926+0000 7fa572ffd700 1 --2- 172.21.15.113:0/2844187911 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa55c06caa0 0x7fa55c06ef60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fa564004f10 tx=0x7fa56400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:58.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:58.930+0000 7fa55bfff700 1 -- 172.21.15.113:0/2844187911 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa5680593a0 con 0x7fa5740fe2d0 2024-04-12T05:31:59.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.138+0000 7fa57945f700 1 -- 172.21.15.113:0/2844187911 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa574108ae0 con 0x7fa55c06caa0 2024-04-12T05:31:59.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.139+0000 7fa55bfff700 1 -- 172.21.15.113:0/2844187911 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa574108ae0 con 0x7fa55c06caa0 2024-04-12T05:31:59.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.141+0000 7fa57945f700 1 -- 172.21.15.113:0/2844187911 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa55c06caa0 msgr2=0x7fa55c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.141+0000 7fa57945f700 1 --2- 172.21.15.113:0/2844187911 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa55c06caa0 0x7fa55c06ef60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fa564004f10 tx=0x7fa56400b040).stop 2024-04-12T05:31:59.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.141+0000 7fa57945f700 1 -- 172.21.15.113:0/2844187911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5740fe2d0 msgr2=0x7fa57410e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.141+0000 7fa57945f700 1 --2- 172.21.15.113:0/2844187911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5740fe2d0 0x7fa57410e4d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7fa568002820 tx=0x7fa568009f20).stop 2024-04-12T05:31:59.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.141+0000 7fa57945f700 1 -- 172.21.15.113:0/2844187911 shutdown_connections 2024-04-12T05:31:59.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.141+0000 7fa57945f700 1 --2- 172.21.15.113:0/2844187911 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa55c06caa0 0x7fa55c06ef60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.141+0000 7fa57945f700 1 --2- 172.21.15.113:0/2844187911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5740fd970 0x7fa57410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.141+0000 7fa57945f700 1 --2- 172.21.15.113:0/2844187911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5740fe2d0 0x7fa57410e4d0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.141+0000 7fa57945f700 1 -- 172.21.15.113:0/2844187911 >> 172.21.15.113:0/2844187911 conn(0x7fa5740f3550 msgr2=0x7fa5740fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:59.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.141+0000 7fa57945f700 1 -- 172.21.15.113:0/2844187911 shutdown_connections 2024-04-12T05:31:59.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.141+0000 7fa57945f700 1 -- 172.21.15.113:0/2844187911 wait complete. 2024-04-12T05:31:59.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.240+0000 7f2a42f48700 1 -- 172.21.15.113:0/1309713910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a3c0fc8a0 msgr2=0x7f2a3c0f60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.240+0000 7f2a42f48700 1 --2- 172.21.15.113:0/1309713910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a3c0fc8a0 0x7f2a3c0f60d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f2a280047f0 tx=0x7f2a28009cb0).stop 2024-04-12T05:31:59.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.240+0000 7f2a42f48700 1 -- 172.21.15.113:0/1309713910 shutdown_connections 2024-04-12T05:31:59.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.240+0000 7f2a42f48700 1 --2- 172.21.15.113:0/1309713910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a3c0fc8a0 0x7f2a3c0f60d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.240+0000 7f2a42f48700 1 --2- 172.21.15.113:0/1309713910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a3c0fbf40 0x7f2a3c0fc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.240+0000 7f2a42f48700 1 -- 172.21.15.113:0/1309713910 >> 172.21.15.113:0/1309713910 conn(0x7f2a3c06cea0 msgr2=0x7f2a3c06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:59.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.240+0000 7f2a42f48700 1 -- 172.21.15.113:0/1309713910 shutdown_connections 2024-04-12T05:31:59.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.240+0000 7f2a42f48700 1 -- 172.21.15.113:0/1309713910 wait complete. 2024-04-12T05:31:59.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.241+0000 7f2a42f48700 1 Processor -- start 2024-04-12T05:31:59.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.241+0000 7f2a42f48700 1 -- start start 2024-04-12T05:31:59.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.241+0000 7f2a42f48700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a3c0fbf40 0x7f2a3c109080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:59.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.241+0000 7f2a42f48700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a3c0fc8a0 0x7f2a3c109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:59.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.241+0000 7f2a42f48700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c109a60 con 0x7f2a3c0fc8a0 2024-04-12T05:31:59.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.241+0000 7f2a42f48700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c109ba0 con 0x7f2a3c0fbf40 2024-04-12T05:31:59.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a3bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a3c0fc8a0 0x7f2a3c109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:59.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a3bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a3c0fc8a0 0x7f2a3c109570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40468/0 (socket says 172.21.15.113:40468) 2024-04-12T05:31:59.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a3bfff700 1 -- 172.21.15.113:0/795411279 learned_addr learned my addr 172.21.15.113:0/795411279 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:59.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a40ce4700 1 --2- 172.21.15.113:0/795411279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a3c0fbf40 0x7f2a3c109080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:59.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a3bfff700 1 -- 172.21.15.113:0/795411279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a3c0fbf40 msgr2=0x7f2a3c109080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a3bfff700 1 --2- 172.21.15.113:0/795411279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a3c0fbf40 0x7f2a3c109080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a3bfff700 1 -- 172.21.15.113:0/795411279 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a28005040 con 0x7f2a3c0fc8a0 2024-04-12T05:31:59.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a3bfff700 1 --2- 172.21.15.113:0/795411279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a3c0fc8a0 0x7f2a3c109570 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f2a28005010 tx=0x7f2a28009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:59.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a39ffb700 1 -- 172.21.15.113:0/795411279 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2800e030 con 0x7f2a3c0fc8a0 2024-04-12T05:31:59.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a39ffb700 1 -- 172.21.15.113:0/795411279 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a28007e90 con 0x7f2a3c0fc8a0 2024-04-12T05:31:59.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a42f48700 1 -- 172.21.15.113:0/795411279 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3c109df0 con 0x7f2a3c0fc8a0 2024-04-12T05:31:59.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a39ffb700 1 -- 172.21.15.113:0/795411279 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a28018070 con 0x7f2a3c0fc8a0 2024-04-12T05:31:59.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.242+0000 7f2a42f48700 1 -- 172.21.15.113:0/795411279 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3c112360 con 0x7f2a3c0fc8a0 2024-04-12T05:31:59.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.243+0000 7f2a42f48700 1 -- 172.21.15.113:0/795411279 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3c047540 con 0x7f2a3c0fc8a0 2024-04-12T05:31:59.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.244+0000 7f2a39ffb700 1 -- 172.21.15.113:0/795411279 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2a2801d6e0 con 0x7f2a3c0fc8a0 2024-04-12T05:31:59.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.244+0000 7f2a39ffb700 1 --2- 172.21.15.113:0/795411279 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a2c06caa0 0x7f2a2c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:59.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.244+0000 7f2a39ffb700 1 -- 172.21.15.113:0/795411279 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2a2808d0d0 con 0x7f2a3c0fc8a0 2024-04-12T05:31:59.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.244+0000 7f2a40ce4700 1 --2- 172.21.15.113:0/795411279 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a2c06caa0 0x7f2a2c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:59.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.245+0000 7f2a40ce4700 1 --2- 172.21.15.113:0/795411279 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a2c06caa0 0x7f2a2c06ef60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f2a30009250 tx=0x7f2a30009de0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:59.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.248+0000 7f2a39ffb700 1 -- 172.21.15.113:0/795411279 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2a2805ab80 con 0x7f2a3c0fc8a0 2024-04-12T05:31:59.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:31:59 smithi113 ceph-mon[103371]: pgmap v594: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:59.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:31:59 smithi192 ceph-mon[85304]: pgmap v594: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:31:59.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.454+0000 7f2a42f48700 1 -- 172.21.15.113:0/795411279 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a3c0fd8b0 con 0x7f2a2c06caa0 2024-04-12T05:31:59.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.464+0000 7f2a39ffb700 1 -- 172.21.15.113:0/795411279 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2a3c0fd8b0 con 0x7f2a2c06caa0 2024-04-12T05:31:59.463 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (19m) 12m ago 31m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (17m) 12m ago 30m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (17m) 12m ago 30m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (19m) 12m ago 30m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (13m) 12m ago 25m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (12m) 12m ago 25m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (12m) 12m ago 25m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (25m) 12m ago 25m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (21m) 12m ago 32m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (20m) 12m ago 30m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (18m) 12m ago 32m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (17m) 12m ago 30m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:31:59.464 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (20m) 12m ago 30m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:31:59.465 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (20m) 12m ago 30m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:31:59.465 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (16m) 12m ago 29m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:31:59.465 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (16m) 12m ago 29m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:31:59.465 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (15m) 12m ago 28m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:31:59.465 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (15m) 12m ago 28m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:31:59.465 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (14m) 12m ago 27m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:31:59.465 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (13m) 12m ago 27m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:31:59.465 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (20m) 12m ago 29m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:31:59.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.466+0000 7f2a42f48700 1 -- 172.21.15.113:0/795411279 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a2c06caa0 msgr2=0x7f2a2c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.466+0000 7f2a42f48700 1 --2- 172.21.15.113:0/795411279 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a2c06caa0 0x7f2a2c06ef60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f2a30009250 tx=0x7f2a30009de0).stop 2024-04-12T05:31:59.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.466+0000 7f2a42f48700 1 -- 172.21.15.113:0/795411279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a3c0fc8a0 msgr2=0x7f2a3c109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.466+0000 7f2a42f48700 1 --2- 172.21.15.113:0/795411279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a3c0fc8a0 0x7f2a3c109570 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f2a28005010 tx=0x7f2a28009cb0).stop 2024-04-12T05:31:59.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.466+0000 7f2a42f48700 1 -- 172.21.15.113:0/795411279 shutdown_connections 2024-04-12T05:31:59.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.466+0000 7f2a42f48700 1 --2- 172.21.15.113:0/795411279 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2a2c06caa0 0x7f2a2c06ef60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.466+0000 7f2a42f48700 1 --2- 172.21.15.113:0/795411279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a3c0fbf40 0x7f2a3c109080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.466+0000 7f2a42f48700 1 --2- 172.21.15.113:0/795411279 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a3c0fc8a0 0x7f2a3c109570 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.466+0000 7f2a42f48700 1 -- 172.21.15.113:0/795411279 >> 172.21.15.113:0/795411279 conn(0x7f2a3c06cea0 msgr2=0x7f2a3c0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:59.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.466+0000 7f2a42f48700 1 -- 172.21.15.113:0/795411279 shutdown_connections 2024-04-12T05:31:59.466 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.466+0000 7f2a42f48700 1 -- 172.21.15.113:0/795411279 wait complete. 2024-04-12T05:31:59.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.563+0000 7fc91bd73700 1 -- 172.21.15.113:0/3636779166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9140fd150 msgr2=0x7fc9140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.563+0000 7fc91bd73700 1 --2- 172.21.15.113:0/3636779166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9140fd150 0x7fc9140f6930 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7fc9100047f0 tx=0x7fc910009cb0).stop 2024-04-12T05:31:59.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.563+0000 7fc91bd73700 1 -- 172.21.15.113:0/3636779166 shutdown_connections 2024-04-12T05:31:59.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.563+0000 7fc91bd73700 1 --2- 172.21.15.113:0/3636779166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9140fd150 0x7fc9140f6930 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.563+0000 7fc91bd73700 1 --2- 172.21.15.113:0/3636779166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9140fc7f0 0x7fc9140fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.563+0000 7fc91bd73700 1 -- 172.21.15.113:0/3636779166 >> 172.21.15.113:0/3636779166 conn(0x7fc914070590 msgr2=0x7fc9140f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:59.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.563+0000 7fc91bd73700 1 -- 172.21.15.113:0/3636779166 shutdown_connections 2024-04-12T05:31:59.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.563+0000 7fc91bd73700 1 -- 172.21.15.113:0/3636779166 wait complete. 2024-04-12T05:31:59.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91bd73700 1 Processor -- start 2024-04-12T05:31:59.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91bd73700 1 -- start start 2024-04-12T05:31:59.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91bd73700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9140fc7f0 0x7fc9141090a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:59.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91bd73700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9140fd150 0x7fc914109590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:59.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91bd73700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc914109a80 con 0x7fc9140fd150 2024-04-12T05:31:59.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91bd73700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc914109bc0 con 0x7fc9140fc7f0 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91930e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9140fd150 0x7fc914109590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91930e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9140fd150 0x7fc914109590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40482/0 (socket says 172.21.15.113:40482) 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91930e700 1 -- 172.21.15.113:0/356387131 learned_addr learned my addr 172.21.15.113:0/356387131 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc919b0f700 1 --2- 172.21.15.113:0/356387131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9140fc7f0 0x7fc9141090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91930e700 1 -- 172.21.15.113:0/356387131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9140fc7f0 msgr2=0x7fc9141090a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91930e700 1 --2- 172.21.15.113:0/356387131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9140fc7f0 0x7fc9141090a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.564+0000 7fc91930e700 1 -- 172.21.15.113:0/356387131 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc910005040 con 0x7fc9140fd150 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.565+0000 7fc91930e700 1 --2- 172.21.15.113:0/356387131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9140fd150 0x7fc914109590 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7fc9100099f0 tx=0x7fc910009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.565+0000 7fc906ffd700 1 -- 172.21.15.113:0/356387131 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc91000e030 con 0x7fc9140fd150 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.565+0000 7fc906ffd700 1 -- 172.21.15.113:0/356387131 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc910007e90 con 0x7fc9140fd150 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.565+0000 7fc91bd73700 1 -- 172.21.15.113:0/356387131 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc914109e10 con 0x7fc9140fd150 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.565+0000 7fc906ffd700 1 -- 172.21.15.113:0/356387131 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc910018070 con 0x7fc9140fd150 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.565+0000 7fc91bd73700 1 -- 172.21.15.113:0/356387131 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc914112400 con 0x7fc9140fd150 2024-04-12T05:31:59.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.566+0000 7fc91bd73700 1 -- 172.21.15.113:0/356387131 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc91405f590 con 0x7fc9140fd150 2024-04-12T05:31:59.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.566+0000 7fc906ffd700 1 -- 172.21.15.113:0/356387131 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc91001d420 con 0x7fc9140fd150 2024-04-12T05:31:59.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.567+0000 7fc906ffd700 1 --2- 172.21.15.113:0/356387131 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc90006caa0 0x7fc90006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:59.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.567+0000 7fc906ffd700 1 -- 172.21.15.113:0/356387131 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc91008d2f0 con 0x7fc9140fd150 2024-04-12T05:31:59.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.567+0000 7fc919b0f700 1 --2- 172.21.15.113:0/356387131 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc90006caa0 0x7fc90006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:59.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.567+0000 7fc919b0f700 1 --2- 172.21.15.113:0/356387131 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc90006caa0 0x7fc90006ef60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fc908004b60 tx=0x7fc908009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:59.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.571+0000 7fc906ffd700 1 -- 172.21.15.113:0/356387131 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc910057310 con 0x7fc9140fd150 2024-04-12T05:31:59.837 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.837+0000 7fc91bd73700 1 -- 172.21.15.113:0/356387131 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc914047540 con 0x7fc9140fd150 2024-04-12T05:31:59.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.837+0000 7fc906ffd700 1 -- 172.21.15.113:0/356387131 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7fc91005a930 con 0x7fc9140fd150 2024-04-12T05:31:59.838 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:31:59.838 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:31:59.838 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:31:59.838 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:31:59.838 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:31:59.838 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:31:59.838 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:31:59.838 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:31:59.838 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:31:59.838 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:31:59.839 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:31:59.839 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:31:59.839 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:31:59.839 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:31:59.839 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:31:59.839 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:31:59.839 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:31:59.839 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:31:59.839 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:31:59.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.839+0000 7fc91bd73700 1 -- 172.21.15.113:0/356387131 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc90006caa0 msgr2=0x7fc90006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.839+0000 7fc91bd73700 1 --2- 172.21.15.113:0/356387131 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc90006caa0 0x7fc90006ef60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fc908004b60 tx=0x7fc908009040).stop 2024-04-12T05:31:59.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.839+0000 7fc91bd73700 1 -- 172.21.15.113:0/356387131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9140fd150 msgr2=0x7fc914109590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.839+0000 7fc91bd73700 1 --2- 172.21.15.113:0/356387131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9140fd150 0x7fc914109590 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7fc9100099f0 tx=0x7fc910009cb0).stop 2024-04-12T05:31:59.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.839+0000 7fc91bd73700 1 -- 172.21.15.113:0/356387131 shutdown_connections 2024-04-12T05:31:59.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.839+0000 7fc91bd73700 1 --2- 172.21.15.113:0/356387131 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc90006caa0 0x7fc90006ef60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.839+0000 7fc91bd73700 1 --2- 172.21.15.113:0/356387131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9140fc7f0 0x7fc9141090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.839+0000 7fc91bd73700 1 --2- 172.21.15.113:0/356387131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9140fd150 0x7fc914109590 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.839+0000 7fc91bd73700 1 -- 172.21.15.113:0/356387131 >> 172.21.15.113:0/356387131 conn(0x7fc914070590 msgr2=0x7fc9140fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:59.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.839+0000 7fc91bd73700 1 -- 172.21.15.113:0/356387131 shutdown_connections 2024-04-12T05:31:59.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.839+0000 7fc91bd73700 1 -- 172.21.15.113:0/356387131 wait complete. 2024-04-12T05:31:59.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.937+0000 7f480da2f700 1 -- 172.21.15.113:0/1004098183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48080fd990 msgr2=0x7f48080fdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.937+0000 7f480da2f700 1 --2- 172.21.15.113:0/1004098183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48080fd990 0x7f48080fdd70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f47fc0047f0 tx=0x7f47fc009cb0).stop 2024-04-12T05:31:59.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.937+0000 7f480da2f700 1 -- 172.21.15.113:0/1004098183 shutdown_connections 2024-04-12T05:31:59.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.937+0000 7f480da2f700 1 --2- 172.21.15.113:0/1004098183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48080fe2f0 0x7f48080f7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.937+0000 7f480da2f700 1 --2- 172.21.15.113:0/1004098183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48080fd990 0x7f48080fdd70 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.937+0000 7f480da2f700 1 -- 172.21.15.113:0/1004098183 >> 172.21.15.113:0/1004098183 conn(0x7f48080f3570 msgr2=0x7f48080f5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:31:59.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.937+0000 7f480da2f700 1 -- 172.21.15.113:0/1004098183 shutdown_connections 2024-04-12T05:31:59.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.937+0000 7f480da2f700 1 -- 172.21.15.113:0/1004098183 wait complete. 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.937+0000 7f480da2f700 1 Processor -- start 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f480da2f700 1 -- start start 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f480da2f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48080fd990 0x7f480810e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f480da2f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48080fe2f0 0x7f480810e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f480da2f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480810eb90 con 0x7f48080fe2f0 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f480da2f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480810ecd0 con 0x7f48080fd990 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f48067fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48080fe2f0 0x7f480810e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f48067fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48080fe2f0 0x7f480810e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40506/0 (socket says 172.21.15.113:40506) 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f48067fc700 1 -- 172.21.15.113:0/3323296461 learned_addr learned my addr 172.21.15.113:0/3323296461 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f4806ffd700 1 --2- 172.21.15.113:0/3323296461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48080fd990 0x7f480810e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f48067fc700 1 -- 172.21.15.113:0/3323296461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48080fd990 msgr2=0x7f480810e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f48067fc700 1 --2- 172.21.15.113:0/3323296461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48080fd990 0x7f480810e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f48067fc700 1 -- 172.21.15.113:0/3323296461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc005040 con 0x7f48080fe2f0 2024-04-12T05:31:59.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.938+0000 7f48067fc700 1 --2- 172.21.15.113:0/3323296461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48080fe2f0 0x7f480810e590 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f47f800c790 tx=0x7f47f8007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:59.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.939+0000 7f47effff700 1 -- 172.21.15.113:0/3323296461 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47f800e680 con 0x7f48080fe2f0 2024-04-12T05:31:59.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.939+0000 7f47effff700 1 -- 172.21.15.113:0/3323296461 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47f800ec90 con 0x7f48080fe2f0 2024-04-12T05:31:59.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.939+0000 7f480da2f700 1 -- 172.21.15.113:0/3323296461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4808108360 con 0x7f48080fe2f0 2024-04-12T05:31:59.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.939+0000 7f47effff700 1 -- 172.21.15.113:0/3323296461 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47f8012c70 con 0x7f48080fe2f0 2024-04-12T05:31:59.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.939+0000 7f480da2f700 1 -- 172.21.15.113:0/3323296461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48081088a0 con 0x7f48080fe2f0 2024-04-12T05:31:59.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.940+0000 7f480da2f700 1 -- 172.21.15.113:0/3323296461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4808060330 con 0x7f48080fe2f0 2024-04-12T05:31:59.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.940+0000 7f47effff700 1 -- 172.21.15.113:0/3323296461 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f47f8012490 con 0x7f48080fe2f0 2024-04-12T05:31:59.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.941+0000 7f47effff700 1 --2- 172.21.15.113:0/3323296461 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f47f006caf0 0x7f47f006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:31:59.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.941+0000 7f47effff700 1 -- 172.21.15.113:0/3323296461 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f47f808b7f0 con 0x7f48080fe2f0 2024-04-12T05:31:59.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.941+0000 7f4806ffd700 1 --2- 172.21.15.113:0/3323296461 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f47f006caf0 0x7f47f006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:31:59.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.941+0000 7f4806ffd700 1 --2- 172.21.15.113:0/3323296461 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f47f006caf0 0x7f47f006efb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f47fc007cf0 tx=0x7f47fc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:31:59.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:31:59.945+0000 7f47effff700 1 -- 172.21.15.113:0/3323296461 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f47f80591e0 con 0x7f48080fe2f0 2024-04-12T05:32:00.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.173+0000 7f480da2f700 1 -- 172.21.15.113:0/3323296461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f48080ffce0 con 0x7f48080fe2f0 2024-04-12T05:32:00.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.174+0000 7f47effff700 1 -- 172.21.15.113:0/3323296461 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f47f8058d70 con 0x7f48080fe2f0 2024-04-12T05:32:00.173 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:32:00.174 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:32:00.175 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:32:00.176 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:32:00.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.176+0000 7f480da2f700 1 -- 172.21.15.113:0/3323296461 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f47f006caf0 msgr2=0x7f47f006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:00.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.176+0000 7f480da2f700 1 --2- 172.21.15.113:0/3323296461 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f47f006caf0 0x7f47f006efb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f47fc007cf0 tx=0x7f47fc00b040).stop 2024-04-12T05:32:00.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.176+0000 7f480da2f700 1 -- 172.21.15.113:0/3323296461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48080fe2f0 msgr2=0x7f480810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:00.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.176+0000 7f480da2f700 1 --2- 172.21.15.113:0/3323296461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48080fe2f0 0x7f480810e590 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f47f800c790 tx=0x7f47f8007f70).stop 2024-04-12T05:32:00.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.176+0000 7f480da2f700 1 -- 172.21.15.113:0/3323296461 shutdown_connections 2024-04-12T05:32:00.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.176+0000 7f480da2f700 1 --2- 172.21.15.113:0/3323296461 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f47f006caf0 0x7f47f006efb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.176+0000 7f480da2f700 1 --2- 172.21.15.113:0/3323296461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48080fd990 0x7f480810e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.176+0000 7f480da2f700 1 --2- 172.21.15.113:0/3323296461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48080fe2f0 0x7f480810e590 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.176+0000 7f480da2f700 1 -- 172.21.15.113:0/3323296461 >> 172.21.15.113:0/3323296461 conn(0x7f48080f3570 msgr2=0x7f48080fbd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:00.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.176+0000 7f480da2f700 1 -- 172.21.15.113:0/3323296461 shutdown_connections 2024-04-12T05:32:00.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.176+0000 7f480da2f700 1 -- 172.21.15.113:0/3323296461 wait complete. 2024-04-12T05:32:00.178 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:32:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.275+0000 7fac6e685700 1 -- 172.21.15.113:0/626162233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac680fc7f0 msgr2=0x7fac680fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.275+0000 7fac6e685700 1 --2- 172.21.15.113:0/626162233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac680fc7f0 0x7fac680fcbd0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fac5c0047f0 tx=0x7fac5c009cb0).stop 2024-04-12T05:32:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.275+0000 7fac6e685700 1 -- 172.21.15.113:0/626162233 shutdown_connections 2024-04-12T05:32:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.275+0000 7fac6e685700 1 --2- 172.21.15.113:0/626162233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac680fd150 0x7fac680f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.275+0000 7fac6e685700 1 --2- 172.21.15.113:0/626162233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac680fc7f0 0x7fac680fcbd0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.275+0000 7fac6e685700 1 -- 172.21.15.113:0/626162233 >> 172.21.15.113:0/626162233 conn(0x7fac68070590 msgr2=0x7fac680f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.275+0000 7fac6e685700 1 -- 172.21.15.113:0/626162233 shutdown_connections 2024-04-12T05:32:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.275+0000 7fac6e685700 1 -- 172.21.15.113:0/626162233 wait complete. 2024-04-12T05:32:00.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.275+0000 7fac6e685700 1 Processor -- start 2024-04-12T05:32:00.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.275+0000 7fac6e685700 1 -- start start 2024-04-12T05:32:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac6e685700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac680fc7f0 0x7fac68109120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac6e685700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac680fd150 0x7fac68109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac6e685700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac68109b00 con 0x7fac680fd150 2024-04-12T05:32:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac6e685700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac68109c40 con 0x7fac680fc7f0 2024-04-12T05:32:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac677fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac680fd150 0x7fac68109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac677fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac680fd150 0x7fac68109610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40532/0 (socket says 172.21.15.113:40532) 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac677fe700 1 -- 172.21.15.113:0/550342608 learned_addr learned my addr 172.21.15.113:0/550342608 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac67fff700 1 --2- 172.21.15.113:0/550342608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac680fc7f0 0x7fac68109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac677fe700 1 -- 172.21.15.113:0/550342608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac680fc7f0 msgr2=0x7fac68109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac677fe700 1 --2- 172.21.15.113:0/550342608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac680fc7f0 0x7fac68109120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac677fe700 1 -- 172.21.15.113:0/550342608 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c005040 con 0x7fac680fd150 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac677fe700 1 --2- 172.21.15.113:0/550342608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac680fd150 0x7fac68109610 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7fac5800c790 tx=0x7fac58007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac657fa700 1 -- 172.21.15.113:0/550342608 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac580106f0 con 0x7fac680fd150 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac6e685700 1 -- 172.21.15.113:0/550342608 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac68109ef0 con 0x7fac680fd150 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac657fa700 1 -- 172.21.15.113:0/550342608 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac58010d00 con 0x7fac680fd150 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.276+0000 7fac657fa700 1 -- 172.21.15.113:0/550342608 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac58012d50 con 0x7fac680fd150 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.277+0000 7fac6e685700 1 -- 172.21.15.113:0/550342608 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac681124e0 con 0x7fac680fd150 2024-04-12T05:32:00.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.277+0000 7fac6e685700 1 -- 172.21.15.113:0/550342608 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6805f590 con 0x7fac680fd150 2024-04-12T05:32:00.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.278+0000 7fac657fa700 1 -- 172.21.15.113:0/550342608 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fac58012570 con 0x7fac680fd150 2024-04-12T05:32:00.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.279+0000 7fac657fa700 1 --2- 172.21.15.113:0/550342608 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac5006caa0 0x7fac5006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:00.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.279+0000 7fac657fa700 1 -- 172.21.15.113:0/550342608 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fac5808c070 con 0x7fac680fd150 2024-04-12T05:32:00.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.279+0000 7fac67fff700 1 --2- 172.21.15.113:0/550342608 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac5006caa0 0x7fac5006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:00.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.279+0000 7fac67fff700 1 --2- 172.21.15.113:0/550342608 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac5006caa0 0x7fac5006ef60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fac5c004f10 tx=0x7fac5c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:00.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.282+0000 7fac657fa700 1 -- 172.21.15.113:0/550342608 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fac58056090 con 0x7fac680fd150 2024-04-12T05:32:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:00 smithi113 ceph-mon[103371]: from='client.35092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:00 smithi113 ceph-mon[103371]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:00 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/356387131' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:32:00.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:00 smithi192 ceph-mon[85304]: from='client.35092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:00.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:00 smithi192 ceph-mon[85304]: from='client.35096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:00.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:00 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/356387131' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:32:00.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.488+0000 7fac6e685700 1 -- 172.21.15.113:0/550342608 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fac68059120 con 0x7fac5006caa0 2024-04-12T05:32:00.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.489+0000 7fac657fa700 1 -- 172.21.15.113:0/550342608 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fac68059120 con 0x7fac5006caa0 2024-04-12T05:32:00.489 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:32:00.489 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:32:00.489 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:32:00.489 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:32:00.489 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:32:00.489 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:32:00.489 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:32:00.489 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:32:00.489 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:32:00.489 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:32:00.489 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:32:00.490 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:32:00.490 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:32:00.490 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:32:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.491+0000 7fac6e685700 1 -- 172.21.15.113:0/550342608 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac5006caa0 msgr2=0x7fac5006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.491+0000 7fac6e685700 1 --2- 172.21.15.113:0/550342608 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac5006caa0 0x7fac5006ef60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fac5c004f10 tx=0x7fac5c00b040).stop 2024-04-12T05:32:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.491+0000 7fac6e685700 1 -- 172.21.15.113:0/550342608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac680fd150 msgr2=0x7fac68109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.491+0000 7fac6e685700 1 --2- 172.21.15.113:0/550342608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac680fd150 0x7fac68109610 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7fac5800c790 tx=0x7fac58007f50).stop 2024-04-12T05:32:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.491+0000 7fac6e685700 1 -- 172.21.15.113:0/550342608 shutdown_connections 2024-04-12T05:32:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.491+0000 7fac6e685700 1 --2- 172.21.15.113:0/550342608 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fac5006caa0 0x7fac5006ef60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.491+0000 7fac6e685700 1 --2- 172.21.15.113:0/550342608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac680fc7f0 0x7fac68109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.491+0000 7fac6e685700 1 --2- 172.21.15.113:0/550342608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac680fd150 0x7fac68109610 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.491+0000 7fac6e685700 1 -- 172.21.15.113:0/550342608 >> 172.21.15.113:0/550342608 conn(0x7fac68070590 msgr2=0x7fac680fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:00.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.491+0000 7fac6e685700 1 -- 172.21.15.113:0/550342608 shutdown_connections 2024-04-12T05:32:00.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.491+0000 7fac6e685700 1 -- 172.21.15.113:0/550342608 wait complete. 2024-04-12T05:32:00.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.589+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3358690105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8a40fc7f0 msgr2=0x7fe8a40fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:00.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.589+0000 7fe8ac43a700 1 --2- 172.21.15.113:0/3358690105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8a40fc7f0 0x7fe8a40fcbd0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fe8a00047f0 tx=0x7fe8a0009cb0).stop 2024-04-12T05:32:00.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3358690105 shutdown_connections 2024-04-12T05:32:00.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 --2- 172.21.15.113:0/3358690105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a40fd150 0x7fe8a40f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 --2- 172.21.15.113:0/3358690105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8a40fc7f0 0x7fe8a40fcbd0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3358690105 >> 172.21.15.113:0/3358690105 conn(0x7fe8a4070590 msgr2=0x7fe8a40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:00.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3358690105 shutdown_connections 2024-04-12T05:32:00.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3358690105 wait complete. 2024-04-12T05:32:00.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 Processor -- start 2024-04-12T05:32:00.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 -- start start 2024-04-12T05:32:00.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a40fc7f0 0x7fe8a40f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:00.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8a40fd150 0x7fe8a40f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:00.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40f8f20 con 0x7fe8a40fd150 2024-04-12T05:32:00.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.590+0000 7fe8ac43a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40f9060 con 0x7fe8a40fc7f0 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe8a99d5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8a40fd150 0x7fe8a40f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe8a99d5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8a40fd150 0x7fe8a40f8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56122/0 (socket says 172.21.15.113:56122) 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe8a99d5700 1 -- 172.21.15.113:0/3452541991 learned_addr learned my addr 172.21.15.113:0/3452541991 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe8aa1d6700 1 --2- 172.21.15.113:0/3452541991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a40fc7f0 0x7fe8a40f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe8a99d5700 1 -- 172.21.15.113:0/3452541991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a40fc7f0 msgr2=0x7fe8a40f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe8a99d5700 1 --2- 172.21.15.113:0/3452541991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a40fc7f0 0x7fe8a40f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe8a99d5700 1 -- 172.21.15.113:0/3452541991 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a0005040 con 0x7fe8a40fd150 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe8a99d5700 1 --2- 172.21.15.113:0/3452541991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8a40fd150 0x7fe8a40f8a30 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fe89400c710 tx=0x7fe894007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe89b7fe700 1 -- 172.21.15.113:0/3452541991 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe89400e680 con 0x7fe8a40fd150 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe89b7fe700 1 -- 172.21.15.113:0/3452541991 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe89400ec90 con 0x7fe8a40fd150 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3452541991 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a40f9310 con 0x7fe8a40fd150 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe89b7fe700 1 -- 172.21.15.113:0/3452541991 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe894013cd0 con 0x7fe8a40fd150 2024-04-12T05:32:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.591+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3452541991 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a41146a0 con 0x7fe8a40fd150 2024-04-12T05:32:00.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.592+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3452541991 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a405f590 con 0x7fe8a40fd150 2024-04-12T05:32:00.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.592+0000 7fe89b7fe700 1 -- 172.21.15.113:0/3452541991 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe894013410 con 0x7fe8a40fd150 2024-04-12T05:32:00.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.593+0000 7fe89b7fe700 1 --2- 172.21.15.113:0/3452541991 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe89006caa0 0x7fe89006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:00.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.593+0000 7fe89b7fe700 1 -- 172.21.15.113:0/3452541991 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe89408c080 con 0x7fe8a40fd150 2024-04-12T05:32:00.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.593+0000 7fe8aa1d6700 1 --2- 172.21.15.113:0/3452541991 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe89006caa0 0x7fe89006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:00.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.593+0000 7fe8aa1d6700 1 --2- 172.21.15.113:0/3452541991 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe89006caa0 0x7fe89006ef60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fe8a0000c00 tx=0x7fe8a000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:00.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.597+0000 7fe89b7fe700 1 -- 172.21.15.113:0/3452541991 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe8940560a0 con 0x7fe8a40fd150 2024-04-12T05:32:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.865+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3452541991 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe8a40f9ac0 con 0x7fe8a40fd150 2024-04-12T05:32:00.865 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.865+0000 7fe89b7fe700 1 -- 172.21.15.113:0/3452541991 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fe8940596c0 con 0x7fe8a40fd150 2024-04-12T05:32:00.865 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:32:00.865 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:32:00.865 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:32:00.865 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:32:00.865 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:32:00.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.867+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3452541991 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe89006caa0 msgr2=0x7fe89006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:00.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.867+0000 7fe8ac43a700 1 --2- 172.21.15.113:0/3452541991 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe89006caa0 0x7fe89006ef60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fe8a0000c00 tx=0x7fe8a000b040).stop 2024-04-12T05:32:00.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.867+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3452541991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8a40fd150 msgr2=0x7fe8a40f8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:00.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.867+0000 7fe8ac43a700 1 --2- 172.21.15.113:0/3452541991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8a40fd150 0x7fe8a40f8a30 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fe89400c710 tx=0x7fe894007ec0).stop 2024-04-12T05:32:00.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.867+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3452541991 shutdown_connections 2024-04-12T05:32:00.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.867+0000 7fe8ac43a700 1 --2- 172.21.15.113:0/3452541991 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe89006caa0 0x7fe89006ef60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.867+0000 7fe8ac43a700 1 --2- 172.21.15.113:0/3452541991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a40fc7f0 0x7fe8a40f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.867+0000 7fe8ac43a700 1 --2- 172.21.15.113:0/3452541991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8a40fd150 0x7fe8a40f8a30 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:00.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.867+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3452541991 >> 172.21.15.113:0/3452541991 conn(0x7fe8a4070590 msgr2=0x7fe8a4060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:00.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.867+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3452541991 shutdown_connections 2024-04-12T05:32:00.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:00.867+0000 7fe8ac43a700 1 -- 172.21.15.113:0/3452541991 wait complete. 2024-04-12T05:32:01.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:01 smithi192 ceph-mon[85304]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:01.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:01 smithi192 ceph-mon[85304]: pgmap v595: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:01.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:01 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3323296461' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:32:01.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:01 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3452541991' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:32:01.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:01 smithi113 ceph-mon[103371]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:01.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:01 smithi113 ceph-mon[103371]: pgmap v595: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:01.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:01 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3323296461' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:32:01.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:01 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3452541991' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:32:02.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:02 smithi192 ceph-mon[85304]: from='client.35112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:02.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:02 smithi113 ceph-mon[103371]: from='client.35112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:03.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:03 smithi192 ceph-mon[85304]: pgmap v596: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:03.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:03 smithi113 ceph-mon[103371]: pgmap v596: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:05.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:05 smithi192 ceph-mon[85304]: pgmap v597: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:05.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:05 smithi113 ceph-mon[103371]: pgmap v597: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:07.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:07 smithi192 ceph-mon[85304]: pgmap v598: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:07.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:07.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:07 smithi113 ceph-mon[103371]: pgmap v598: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:07.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:09.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:09 smithi192 ceph-mon[85304]: pgmap v599: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:09.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:09 smithi113 ceph-mon[103371]: pgmap v599: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:11.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:11 smithi192 ceph-mon[85304]: pgmap v600: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:11.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:11 smithi113 ceph-mon[103371]: pgmap v600: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:13.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:13 smithi192 ceph-mon[85304]: pgmap v601: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:13.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:13 smithi113 ceph-mon[103371]: pgmap v601: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:15.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:15 smithi113 ceph-mon[103371]: pgmap v602: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:15.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:15 smithi192 ceph-mon[85304]: pgmap v602: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:16.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:16 smithi113 ceph-mon[103371]: pgmap v603: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:16.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:16 smithi192 ceph-mon[85304]: pgmap v603: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:19.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:19 smithi113 ceph-mon[103371]: pgmap v604: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:19.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:19 smithi192 ceph-mon[85304]: pgmap v604: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:21.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:21 smithi113 ceph-mon[103371]: pgmap v605: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:21.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:21 smithi192 ceph-mon[85304]: pgmap v605: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:22.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:22.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:23.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:23 smithi113 ceph-mon[103371]: pgmap v606: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:23.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:23 smithi192 ceph-mon[85304]: pgmap v606: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:25.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:25 smithi113 ceph-mon[103371]: pgmap v607: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:25.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:25 smithi192 ceph-mon[85304]: pgmap v607: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:27.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:27 smithi113 ceph-mon[103371]: pgmap v608: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:27.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:27 smithi192 ceph-mon[85304]: pgmap v608: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:29.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:29 smithi113 ceph-mon[103371]: pgmap v609: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:29.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:29 smithi192 ceph-mon[85304]: pgmap v609: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:30.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.978+0000 7fbe32794700 1 -- 172.21.15.113:0/2458912127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe2c0fda50 msgr2=0x7fbe2c0fde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:30.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.978+0000 7fbe32794700 1 --2- 172.21.15.113:0/2458912127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe2c0fda50 0x7fbe2c0fde30 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7fbe200047f0 tx=0x7fbe20009cb0).stop 2024-04-12T05:32:30.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.978+0000 7fbe32794700 1 -- 172.21.15.113:0/2458912127 shutdown_connections 2024-04-12T05:32:30.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.978+0000 7fbe32794700 1 --2- 172.21.15.113:0/2458912127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe2c0fe3b0 0x7fbe2c0f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:30.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.978+0000 7fbe32794700 1 --2- 172.21.15.113:0/2458912127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe2c0fda50 0x7fbe2c0fde30 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:30.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.978+0000 7fbe32794700 1 -- 172.21.15.113:0/2458912127 >> 172.21.15.113:0/2458912127 conn(0x7fbe2c06ce50 msgr2=0x7fbe2c06d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:30.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.978+0000 7fbe32794700 1 -- 172.21.15.113:0/2458912127 shutdown_connections 2024-04-12T05:32:30.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.978+0000 7fbe32794700 1 -- 172.21.15.113:0/2458912127 wait complete. 2024-04-12T05:32:30.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.979+0000 7fbe32794700 1 Processor -- start 2024-04-12T05:32:30.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.979+0000 7fbe32794700 1 -- start start 2024-04-12T05:32:30.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.979+0000 7fbe32794700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe2c0fda50 0x7fbe2c10e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:30.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.979+0000 7fbe32794700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe2c0fe3b0 0x7fbe2c10e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:30.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.979+0000 7fbe32794700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c10eb70 con 0x7fbe2c0fe3b0 2024-04-12T05:32:30.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.979+0000 7fbe32794700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c10ecb0 con 0x7fbe2c0fda50 2024-04-12T05:32:30.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe2b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe2c0fe3b0 0x7fbe2c10e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:30.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe2b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe2c0fe3b0 0x7fbe2c10e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38024/0 (socket says 172.21.15.113:38024) 2024-04-12T05:32:30.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe2b7fe700 1 -- 172.21.15.113:0/1989143001 learned_addr learned my addr 172.21.15.113:0/1989143001 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:32:30.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe2bfff700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe2c0fda50 0x7fbe2c10e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:30.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe2b7fe700 1 -- 172.21.15.113:0/1989143001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe2c0fda50 msgr2=0x7fbe2c10e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:30.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe2b7fe700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe2c0fda50 0x7fbe2c10e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:30.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe2b7fe700 1 -- 172.21.15.113:0/1989143001 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe20005040 con 0x7fbe2c0fe3b0 2024-04-12T05:32:30.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe2bfff700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe2c0fda50 0x7fbe2c10e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:32:30.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe2b7fe700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe2c0fe3b0 0x7fbe2c10e570 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7fbe1c0041d0 tx=0x7fbe1c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:30.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe297fa700 1 -- 172.21.15.113:0/1989143001 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe1c007b40 con 0x7fbe2c0fe3b0 2024-04-12T05:32:30.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe297fa700 1 -- 172.21.15.113:0/1989143001 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe1c00f040 con 0x7fbe2c0fe3b0 2024-04-12T05:32:30.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe32794700 1 -- 172.21.15.113:0/1989143001 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe2c108340 con 0x7fbe2c0fe3b0 2024-04-12T05:32:30.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe297fa700 1 -- 172.21.15.113:0/1989143001 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe1c00d800 con 0x7fbe2c0fe3b0 2024-04-12T05:32:30.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.980+0000 7fbe32794700 1 -- 172.21.15.113:0/1989143001 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe2c108880 con 0x7fbe2c0fe3b0 2024-04-12T05:32:30.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.981+0000 7fbe32794700 1 -- 172.21.15.113:0/1989143001 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2c047540 con 0x7fbe2c0fe3b0 2024-04-12T05:32:30.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.982+0000 7fbe297fa700 1 -- 172.21.15.113:0/1989143001 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbe1c007ca0 con 0x7fbe2c0fe3b0 2024-04-12T05:32:30.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.982+0000 7fbe297fa700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe1406caa0 0x7fbe1406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:30.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.982+0000 7fbe2bfff700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe1406caa0 0x7fbe1406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:30.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.982+0000 7fbe297fa700 1 -- 172.21.15.113:0/1989143001 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbe1c08b950 con 0x7fbe2c0fe3b0 2024-04-12T05:32:30.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.983+0000 7fbe2bfff700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe1406caa0 0x7fbe1406ef60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fbe20004540 tx=0x7fbe2000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:30.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:30.987+0000 7fbe297fa700 1 -- 172.21.15.113:0/1989143001 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbe1c059400 con 0x7fbe2c0fe3b0 2024-04-12T05:32:31.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.194+0000 7fbe32794700 1 -- 172.21.15.113:0/1989143001 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe2c108b00 con 0x7fbe1406caa0 2024-04-12T05:32:31.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.195+0000 7fbe297fa700 1 -- 172.21.15.113:0/1989143001 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbe2c108b00 con 0x7fbe1406caa0 2024-04-12T05:32:31.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.197+0000 7fbe32794700 1 -- 172.21.15.113:0/1989143001 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe1406caa0 msgr2=0x7fbe1406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.197+0000 7fbe32794700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe1406caa0 0x7fbe1406ef60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fbe20004540 tx=0x7fbe2000b040).stop 2024-04-12T05:32:31.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.197+0000 7fbe32794700 1 -- 172.21.15.113:0/1989143001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe2c0fe3b0 msgr2=0x7fbe2c10e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.197+0000 7fbe32794700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe2c0fe3b0 0x7fbe2c10e570 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7fbe1c0041d0 tx=0x7fbe1c009f60).stop 2024-04-12T05:32:31.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.197+0000 7fbe32794700 1 -- 172.21.15.113:0/1989143001 shutdown_connections 2024-04-12T05:32:31.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.197+0000 7fbe32794700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe1406caa0 0x7fbe1406ef60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.197+0000 7fbe32794700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe2c0fda50 0x7fbe2c10e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.197+0000 7fbe32794700 1 --2- 172.21.15.113:0/1989143001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe2c0fe3b0 0x7fbe2c10e570 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.197+0000 7fbe32794700 1 -- 172.21.15.113:0/1989143001 >> 172.21.15.113:0/1989143001 conn(0x7fbe2c06ce50 msgr2=0x7fbe2c101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:31.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.198+0000 7fbe32794700 1 -- 172.21.15.113:0/1989143001 shutdown_connections 2024-04-12T05:32:31.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.198+0000 7fbe32794700 1 -- 172.21.15.113:0/1989143001 wait complete. 2024-04-12T05:32:31.207 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:32:31.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.296+0000 7f469032c700 1 -- 172.21.15.113:0/4049800944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880fd150 msgr2=0x7f46880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.296+0000 7f469032c700 1 --2- 172.21.15.113:0/4049800944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880fd150 0x7f46880f6930 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f46840047f0 tx=0x7f4684009cb0).stop 2024-04-12T05:32:31.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.296+0000 7f469032c700 1 -- 172.21.15.113:0/4049800944 shutdown_connections 2024-04-12T05:32:31.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.296+0000 7f469032c700 1 --2- 172.21.15.113:0/4049800944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880fd150 0x7f46880f6930 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.296+0000 7f469032c700 1 --2- 172.21.15.113:0/4049800944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46880fc7f0 0x7f46880fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.296+0000 7f469032c700 1 -- 172.21.15.113:0/4049800944 >> 172.21.15.113:0/4049800944 conn(0x7f4688070590 msgr2=0x7f46880f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.296+0000 7f469032c700 1 -- 172.21.15.113:0/4049800944 shutdown_connections 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.296+0000 7f469032c700 1 -- 172.21.15.113:0/4049800944 wait complete. 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.296+0000 7f469032c700 1 Processor -- start 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f469032c700 1 -- start start 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f469032c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46880fc7f0 0x7f46881101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f469032c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880fd150 0x7f46881106c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f469032c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4688110cc0 con 0x7f46880fd150 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f469032c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4688110e00 con 0x7f46880fc7f0 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f468d8c7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880fd150 0x7f46881106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f468d8c7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880fd150 0x7f46881106c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38040/0 (socket says 172.21.15.113:38040) 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f468d8c7700 1 -- 172.21.15.113:0/1681028475 learned_addr learned my addr 172.21.15.113:0/1681028475 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f468d8c7700 1 -- 172.21.15.113:0/1681028475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46880fc7f0 msgr2=0x7f46881101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f468d8c7700 1 --2- 172.21.15.113:0/1681028475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46880fc7f0 0x7f46881101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:32:31.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f468d8c7700 1 -- 172.21.15.113:0/1681028475 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4684005040 con 0x7f46880fd150 2024-04-12T05:32:31.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f468d8c7700 1 --2- 172.21.15.113:0/1681028475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880fd150 0x7f46881106c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f46840099f0 tx=0x7f4684009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:31.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f467f7fe700 1 -- 172.21.15.113:0/1681028475 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f468400e030 con 0x7f46880fd150 2024-04-12T05:32:31.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f467f7fe700 1 -- 172.21.15.113:0/1681028475 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4684007e90 con 0x7f46880fd150 2024-04-12T05:32:31.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f467f7fe700 1 -- 172.21.15.113:0/1681028475 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4684018070 con 0x7f46880fd150 2024-04-12T05:32:31.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f469032c700 1 -- 172.21.15.113:0/1681028475 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468810a430 con 0x7f46880fd150 2024-04-12T05:32:31.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.297+0000 7f469032c700 1 -- 172.21.15.113:0/1681028475 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468810a990 con 0x7f46880fd150 2024-04-12T05:32:31.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.298+0000 7f469032c700 1 -- 172.21.15.113:0/1681028475 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468805f590 con 0x7f46880fd150 2024-04-12T05:32:31.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.299+0000 7f467f7fe700 1 -- 172.21.15.113:0/1681028475 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4684009590 con 0x7f46880fd150 2024-04-12T05:32:31.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.299+0000 7f467f7fe700 1 --2- 172.21.15.113:0/1681028475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f467406caa0 0x7f467406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:31.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.299+0000 7f467f7fe700 1 -- 172.21.15.113:0/1681028475 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f468408d050 con 0x7f46880fd150 2024-04-12T05:32:31.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.299+0000 7f468e0c8700 1 --2- 172.21.15.113:0/1681028475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f467406caa0 0x7f467406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:31.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.300+0000 7f468e0c8700 1 --2- 172.21.15.113:0/1681028475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f467406caa0 0x7f467406ef60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f4678004b90 tx=0x7f4678009b60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:31.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.304+0000 7f467f7fe700 1 -- 172.21.15.113:0/1681028475 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f468405ab00 con 0x7f46880fd150 2024-04-12T05:32:31.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:31 smithi113 ceph-mon[103371]: pgmap v610: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:31.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:31 smithi192 ceph-mon[85304]: pgmap v610: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:31.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.508+0000 7f469032c700 1 -- 172.21.15.113:0/1681028475 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f468810b080 con 0x7f467406caa0 2024-04-12T05:32:31.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.509+0000 7f467f7fe700 1 -- 172.21.15.113:0/1681028475 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f468810b080 con 0x7f467406caa0 2024-04-12T05:32:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.511+0000 7f469032c700 1 -- 172.21.15.113:0/1681028475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f467406caa0 msgr2=0x7f467406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.511+0000 7f469032c700 1 --2- 172.21.15.113:0/1681028475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f467406caa0 0x7f467406ef60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f4678004b90 tx=0x7f4678009b60).stop 2024-04-12T05:32:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.511+0000 7f469032c700 1 -- 172.21.15.113:0/1681028475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880fd150 msgr2=0x7f46881106c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.511+0000 7f469032c700 1 --2- 172.21.15.113:0/1681028475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880fd150 0x7f46881106c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f46840099f0 tx=0x7f4684009cb0).stop 2024-04-12T05:32:31.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.511+0000 7f469032c700 1 -- 172.21.15.113:0/1681028475 shutdown_connections 2024-04-12T05:32:31.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.511+0000 7f469032c700 1 --2- 172.21.15.113:0/1681028475 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f467406caa0 0x7f467406ef60 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.511+0000 7f469032c700 1 --2- 172.21.15.113:0/1681028475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46880fc7f0 0x7f46881101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:32:31.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.511+0000 7f469032c700 1 --2- 172.21.15.113:0/1681028475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880fd150 0x7f46881106c0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.511+0000 7f469032c700 1 -- 172.21.15.113:0/1681028475 >> 172.21.15.113:0/1681028475 conn(0x7f4688070590 msgr2=0x7f4688060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:31.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.512+0000 7f469032c700 1 -- 172.21.15.113:0/1681028475 shutdown_connections 2024-04-12T05:32:31.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.512+0000 7f469032c700 1 -- 172.21.15.113:0/1681028475 wait complete. 2024-04-12T05:32:31.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.609+0000 7f056552b700 1 -- 172.21.15.113:0/3469946195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05600fe250 msgr2=0x7f05600f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.609+0000 7f056552b700 1 --2- 172.21.15.113:0/3469946195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05600fe250 0x7f05600f7a30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f05500047f0 tx=0x7f0550009cb0).stop 2024-04-12T05:32:31.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.609+0000 7f056552b700 1 -- 172.21.15.113:0/3469946195 shutdown_connections 2024-04-12T05:32:31.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.609+0000 7f056552b700 1 --2- 172.21.15.113:0/3469946195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05600fe250 0x7f05600f7a30 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.609+0000 7f056552b700 1 --2- 172.21.15.113:0/3469946195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05600fd8f0 0x7f05600fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.609+0000 7f056552b700 1 -- 172.21.15.113:0/3469946195 >> 172.21.15.113:0/3469946195 conn(0x7f05600f34b0 msgr2=0x7f05600f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:31.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.609+0000 7f056552b700 1 -- 172.21.15.113:0/3469946195 shutdown_connections 2024-04-12T05:32:31.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.609+0000 7f056552b700 1 -- 172.21.15.113:0/3469946195 wait complete. 2024-04-12T05:32:31.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.609+0000 7f056552b700 1 Processor -- start 2024-04-12T05:32:31.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f056552b700 1 -- start start 2024-04-12T05:32:31.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f056552b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05600fd8f0 0x7f056010e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:31.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f056552b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05600fe250 0x7f056010e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:31.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f056552b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056010eb70 con 0x7f05600fe250 2024-04-12T05:32:31.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f056552b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056010ecb0 con 0x7f05600fd8f0 2024-04-12T05:32:31.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f055e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05600fe250 0x7f056010e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:31.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f055e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05600fe250 0x7f056010e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38052/0 (socket says 172.21.15.113:38052) 2024-04-12T05:32:31.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f055e7fc700 1 -- 172.21.15.113:0/2493645034 learned_addr learned my addr 172.21.15.113:0/2493645034 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:32:31.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f055effd700 1 --2- 172.21.15.113:0/2493645034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05600fd8f0 0x7f056010e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:31.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f055e7fc700 1 -- 172.21.15.113:0/2493645034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05600fd8f0 msgr2=0x7f056010e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f055e7fc700 1 --2- 172.21.15.113:0/2493645034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05600fd8f0 0x7f056010e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f055e7fc700 1 -- 172.21.15.113:0/2493645034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0550005040 con 0x7f05600fe250 2024-04-12T05:32:31.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.610+0000 7f055e7fc700 1 --2- 172.21.15.113:0/2493645034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05600fe250 0x7f056010e570 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f05500099f0 tx=0x7f0550009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:31.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.611+0000 7f0547fff700 1 -- 172.21.15.113:0/2493645034 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f055000e030 con 0x7f05600fe250 2024-04-12T05:32:31.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.611+0000 7f0547fff700 1 -- 172.21.15.113:0/2493645034 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0550007e90 con 0x7f05600fe250 2024-04-12T05:32:31.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.611+0000 7f056552b700 1 -- 172.21.15.113:0/2493645034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05601082e0 con 0x7f05600fe250 2024-04-12T05:32:31.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.611+0000 7f0547fff700 1 -- 172.21.15.113:0/2493645034 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0550018070 con 0x7f05600fe250 2024-04-12T05:32:31.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.611+0000 7f056552b700 1 -- 172.21.15.113:0/2493645034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05601087c0 con 0x7f05600fe250 2024-04-12T05:32:31.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.611+0000 7f056552b700 1 -- 172.21.15.113:0/2493645034 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056005f590 con 0x7f05600fe250 2024-04-12T05:32:31.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.612+0000 7f0547fff700 1 -- 172.21.15.113:0/2493645034 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f055001d420 con 0x7f05600fe250 2024-04-12T05:32:31.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.612+0000 7f0547fff700 1 --2- 172.21.15.113:0/2493645034 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054806caa0 0x7f054806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:31.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.613+0000 7f055effd700 1 --2- 172.21.15.113:0/2493645034 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054806caa0 0x7f054806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:31.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.613+0000 7f0547fff700 1 -- 172.21.15.113:0/2493645034 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f055008d2f0 con 0x7f05600fe250 2024-04-12T05:32:31.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.613+0000 7f055effd700 1 --2- 172.21.15.113:0/2493645034 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054806caa0 0x7f054806ef60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f0554004b60 tx=0x7f0554009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:31.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.617+0000 7f0547fff700 1 -- 172.21.15.113:0/2493645034 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f055005ada0 con 0x7f05600fe250 2024-04-12T05:32:31.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.813+0000 7f056552b700 1 -- 172.21.15.113:0/2493645034 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0560109700 con 0x7f054806caa0 2024-04-12T05:32:31.822 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.823+0000 7f0547fff700 1 -- 172.21.15.113:0/2493645034 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0560109700 con 0x7f054806caa0 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (20m) 13m ago 31m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (17m) 13m ago 31m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (17m) 12m ago 30m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (20m) 13m ago 31m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (13m) 13m ago 26m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (13m) 13m ago 26m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (13m) 12m ago 26m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (26m) 12m ago 26m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (21m) 13m ago 32m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (21m) 12m ago 30m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:32:31.823 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (18m) 13m ago 33m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:32:31.824 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (18m) 12m ago 30m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:32:31.824 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (21m) 13m ago 31m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:32:31.824 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (20m) 12m ago 30m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:32:31.824 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (17m) 13m ago 30m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:32:31.824 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (16m) 13m ago 29m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:32:31.824 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (16m) 13m ago 29m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:32:31.824 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (15m) 12m ago 28m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:32:31.824 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (15m) 12m ago 28m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:32:31.824 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (13m) 12m ago 27m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:32:31.824 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (20m) 12m ago 30m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:32:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.825+0000 7f056552b700 1 -- 172.21.15.113:0/2493645034 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054806caa0 msgr2=0x7f054806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.825+0000 7f056552b700 1 --2- 172.21.15.113:0/2493645034 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054806caa0 0x7f054806ef60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f0554004b60 tx=0x7f0554009040).stop 2024-04-12T05:32:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.825+0000 7f056552b700 1 -- 172.21.15.113:0/2493645034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05600fe250 msgr2=0x7f056010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.825+0000 7f056552b700 1 --2- 172.21.15.113:0/2493645034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05600fe250 0x7f056010e570 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f05500099f0 tx=0x7f0550009cb0).stop 2024-04-12T05:32:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.825+0000 7f056552b700 1 -- 172.21.15.113:0/2493645034 shutdown_connections 2024-04-12T05:32:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.825+0000 7f056552b700 1 --2- 172.21.15.113:0/2493645034 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f054806caa0 0x7f054806ef60 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.825+0000 7f056552b700 1 --2- 172.21.15.113:0/2493645034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05600fd8f0 0x7f056010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.825+0000 7f056552b700 1 --2- 172.21.15.113:0/2493645034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05600fe250 0x7f056010e570 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.825+0000 7f056552b700 1 -- 172.21.15.113:0/2493645034 >> 172.21.15.113:0/2493645034 conn(0x7f05600f34b0 msgr2=0x7f05600fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.825+0000 7f056552b700 1 -- 172.21.15.113:0/2493645034 shutdown_connections 2024-04-12T05:32:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.825+0000 7f056552b700 1 -- 172.21.15.113:0/2493645034 wait complete. 2024-04-12T05:32:31.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.921+0000 7f54a0ef3700 1 -- 172.21.15.113:0/784472542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0fe220 msgr2=0x7f549c0f7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.921+0000 7f54a0ef3700 1 --2- 172.21.15.113:0/784472542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0fe220 0x7f549c0f7a00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f54900047f0 tx=0x7f5490009cb0).stop 2024-04-12T05:32:31.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.922+0000 7f54a0ef3700 1 -- 172.21.15.113:0/784472542 shutdown_connections 2024-04-12T05:32:31.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.922+0000 7f54a0ef3700 1 --2- 172.21.15.113:0/784472542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0fe220 0x7f549c0f7a00 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.922+0000 7f54a0ef3700 1 --2- 172.21.15.113:0/784472542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f549c0fd8c0 0x7f549c0fdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.922+0000 7f54a0ef3700 1 -- 172.21.15.113:0/784472542 >> 172.21.15.113:0/784472542 conn(0x7f549c0f34a0 msgr2=0x7f549c0f58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:31.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.922+0000 7f54a0ef3700 1 -- 172.21.15.113:0/784472542 shutdown_connections 2024-04-12T05:32:31.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.922+0000 7f54a0ef3700 1 -- 172.21.15.113:0/784472542 wait complete. 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.922+0000 7f54a0ef3700 1 Processor -- start 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.922+0000 7f54a0ef3700 1 -- start start 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f54a0ef3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0fd8c0 0x7f549c10e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f54a0ef3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f549c0fe220 0x7f549c10e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f54a0ef3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c10eb60 con 0x7f549c0fd8c0 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f54a0ef3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c10eca0 con 0x7f549c0fe220 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f549a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0fd8c0 0x7f549c10e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f549a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0fd8c0 0x7f549c10e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38060/0 (socket says 172.21.15.113:38060) 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f549a59c700 1 -- 172.21.15.113:0/633536702 learned_addr learned my addr 172.21.15.113:0/633536702 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f5499d9b700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f549c0fe220 0x7f549c10e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f549a59c700 1 -- 172.21.15.113:0/633536702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f549c0fe220 msgr2=0x7f549c10e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:31.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f549a59c700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f549c0fe220 0x7f549c10e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:31.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f549a59c700 1 -- 172.21.15.113:0/633536702 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5490005040 con 0x7f549c0fd8c0 2024-04-12T05:32:31.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f5499d9b700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f549c0fe220 0x7f549c10e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:32:31.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f549a59c700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0fd8c0 0x7f549c10e070 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f548c0041d0 tx=0x7f548c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:31.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f548b7fe700 1 -- 172.21.15.113:0/633536702 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f548c007b40 con 0x7f549c0fd8c0 2024-04-12T05:32:31.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f548b7fe700 1 -- 172.21.15.113:0/633536702 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f548c00f040 con 0x7f549c0fd8c0 2024-04-12T05:32:31.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f54a0ef3700 1 -- 172.21.15.113:0/633536702 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549c108330 con 0x7f549c0fd8c0 2024-04-12T05:32:31.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f548b7fe700 1 -- 172.21.15.113:0/633536702 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f548c00d800 con 0x7f549c0fd8c0 2024-04-12T05:32:31.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.923+0000 7f54a0ef3700 1 -- 172.21.15.113:0/633536702 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549c108870 con 0x7f549c0fd8c0 2024-04-12T05:32:31.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.924+0000 7f54a0ef3700 1 -- 172.21.15.113:0/633536702 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549c060390 con 0x7f549c0fd8c0 2024-04-12T05:32:31.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.925+0000 7f548b7fe700 1 -- 172.21.15.113:0/633536702 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f548c007ca0 con 0x7f549c0fd8c0 2024-04-12T05:32:31.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.926+0000 7f548b7fe700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f548406caa0 0x7f548406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:31.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.926+0000 7f548b7fe700 1 -- 172.21.15.113:0/633536702 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f548c08b970 con 0x7f549c0fd8c0 2024-04-12T05:32:31.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.926+0000 7f5499d9b700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f548406caa0 0x7f548406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:31.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.926+0000 7f5499d9b700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f548406caa0 0x7f548406ef60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f54900043d0 tx=0x7f549000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:31.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:31.929+0000 7f548b7fe700 1 -- 172.21.15.113:0/633536702 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f548c059420 con 0x7f549c0fd8c0 2024-04-12T05:32:32.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.195+0000 7f54a0ef3700 1 -- 172.21.15.113:0/633536702 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f549c0ffc50 con 0x7f549c0fd8c0 2024-04-12T05:32:32.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.196+0000 7f548b7fe700 1 -- 172.21.15.113:0/633536702 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f548c058fb0 con 0x7f549c0fd8c0 2024-04-12T05:32:32.195 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:32:32.196 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:32:32.197 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:32:32.197 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:32:32.197 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:32:32.197 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:32:32.197 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:32:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.198+0000 7f54a0ef3700 1 -- 172.21.15.113:0/633536702 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f548406caa0 msgr2=0x7f548406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.198+0000 7f54a0ef3700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f548406caa0 0x7f548406ef60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f54900043d0 tx=0x7f549000b040).stop 2024-04-12T05:32:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.198+0000 7f54a0ef3700 1 -- 172.21.15.113:0/633536702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0fd8c0 msgr2=0x7f549c10e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.198+0000 7f54a0ef3700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0fd8c0 0x7f549c10e070 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f548c0041d0 tx=0x7f548c009f60).stop 2024-04-12T05:32:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.198+0000 7f54a0ef3700 1 -- 172.21.15.113:0/633536702 shutdown_connections 2024-04-12T05:32:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.198+0000 7f54a0ef3700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f548406caa0 0x7f548406ef60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.198+0000 7f54a0ef3700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f549c0fd8c0 0x7f549c10e070 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.198+0000 7f54a0ef3700 1 --2- 172.21.15.113:0/633536702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f549c0fe220 0x7f549c10e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.198+0000 7f54a0ef3700 1 -- 172.21.15.113:0/633536702 >> 172.21.15.113:0/633536702 conn(0x7f549c0f34a0 msgr2=0x7f549c0fbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.198+0000 7f54a0ef3700 1 -- 172.21.15.113:0/633536702 shutdown_connections 2024-04-12T05:32:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.198+0000 7f54a0ef3700 1 -- 172.21.15.113:0/633536702 wait complete. 2024-04-12T05:32:32.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.296+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1181430611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8a80fd140 msgr2=0x7fb8a80f6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.296+0000 7fb8ae3bd700 1 --2- 172.21.15.113:0/1181430611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8a80fd140 0x7fb8a80f6920 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7fb89c0047f0 tx=0x7fb89c009cb0).stop 2024-04-12T05:32:32.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.296+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1181430611 shutdown_connections 2024-04-12T05:32:32.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.296+0000 7fb8ae3bd700 1 --2- 172.21.15.113:0/1181430611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8a80fd140 0x7fb8a80f6920 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.296+0000 7fb8ae3bd700 1 --2- 172.21.15.113:0/1181430611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a80fc7e0 0x7fb8a80fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.296+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1181430611 >> 172.21.15.113:0/1181430611 conn(0x7fb8a80f23c0 msgr2=0x7fb8a80f47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:32.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.296+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1181430611 shutdown_connections 2024-04-12T05:32:32.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.296+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1181430611 wait complete. 2024-04-12T05:32:32.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.297+0000 7fb8ae3bd700 1 Processor -- start 2024-04-12T05:32:32.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.297+0000 7fb8ae3bd700 1 -- start start 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.297+0000 7fb8ae3bd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a80fc7e0 0x7fb8a810e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.297+0000 7fb8ae3bd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8a80fd140 0x7fb8a810e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.297+0000 7fb8ae3bd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a810eb90 con 0x7fb8a80fd140 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.297+0000 7fb8ae3bd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a810ecd0 con 0x7fb8a80fc7e0 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.297+0000 7fb8a77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8a80fd140 0x7fb8a810e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.297+0000 7fb8a77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8a80fd140 0x7fb8a810e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38078/0 (socket says 172.21.15.113:38078) 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.297+0000 7fb8a77fe700 1 -- 172.21.15.113:0/1974990781 learned_addr learned my addr 172.21.15.113:0/1974990781 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.297+0000 7fb8a7fff700 1 --2- 172.21.15.113:0/1974990781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a80fc7e0 0x7fb8a810e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.298+0000 7fb8a77fe700 1 -- 172.21.15.113:0/1974990781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a80fc7e0 msgr2=0x7fb8a810e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.298+0000 7fb8a77fe700 1 --2- 172.21.15.113:0/1974990781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a80fc7e0 0x7fb8a810e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.298+0000 7fb8a77fe700 1 -- 172.21.15.113:0/1974990781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c005040 con 0x7fb8a80fd140 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.298+0000 7fb8a77fe700 1 --2- 172.21.15.113:0/1974990781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8a80fd140 0x7fb8a810e590 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7fb89c0099f0 tx=0x7fb89c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.298+0000 7fb8a57fa700 1 -- 172.21.15.113:0/1974990781 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb89c00e030 con 0x7fb8a80fd140 2024-04-12T05:32:32.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.298+0000 7fb8a57fa700 1 -- 172.21.15.113:0/1974990781 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb89c007e90 con 0x7fb8a80fd140 2024-04-12T05:32:32.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.298+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1974990781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a8108300 con 0x7fb8a80fd140 2024-04-12T05:32:32.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.298+0000 7fb8a57fa700 1 -- 172.21.15.113:0/1974990781 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb89c018070 con 0x7fb8a80fd140 2024-04-12T05:32:32.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.298+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1974990781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a81087e0 con 0x7fb8a80fd140 2024-04-12T05:32:32.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.299+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1974990781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a8060370 con 0x7fb8a80fd140 2024-04-12T05:32:32.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.299+0000 7fb8a57fa700 1 -- 172.21.15.113:0/1974990781 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb89c009590 con 0x7fb8a80fd140 2024-04-12T05:32:32.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.300+0000 7fb8a57fa700 1 --2- 172.21.15.113:0/1974990781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb89006caa0 0x7fb89006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:32.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.300+0000 7fb8a57fa700 1 -- 172.21.15.113:0/1974990781 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb89c08d010 con 0x7fb8a80fd140 2024-04-12T05:32:32.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.300+0000 7fb8a7fff700 1 --2- 172.21.15.113:0/1974990781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb89006caa0 0x7fb89006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:32.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.300+0000 7fb8a7fff700 1 --2- 172.21.15.113:0/1974990781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb89006caa0 0x7fb89006ef60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fb898009170 tx=0x7fb898009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:32.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.304+0000 7fb8a57fa700 1 -- 172.21.15.113:0/1974990781 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb89c05aac0 con 0x7fb8a80fd140 2024-04-12T05:32:32.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:32 smithi113 ceph-mon[103371]: from='client.35120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:32.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:32 smithi192 ceph-mon[85304]: from='client.35120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:32.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.531+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1974990781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb8a8047540 con 0x7fb8a80fd140 2024-04-12T05:32:32.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.531+0000 7fb8a57fa700 1 -- 172.21.15.113:0/1974990781 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7fb89c05a650 con 0x7fb8a80fd140 2024-04-12T05:32:32.531 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:32:32.531 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:32:32.531 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:32:32.531 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:32:32.531 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:32:32.531 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:32:32.531 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:32:32.531 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:32:32.532 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:32:32.533 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:32:32.534 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:32:32.534 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:32:32.534 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:32:32.534 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.533+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1974990781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb89006caa0 msgr2=0x7fb89006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.533+0000 7fb8ae3bd700 1 --2- 172.21.15.113:0/1974990781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb89006caa0 0x7fb89006ef60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fb898009170 tx=0x7fb898009da0).stop 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.533+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1974990781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8a80fd140 msgr2=0x7fb8a810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.533+0000 7fb8ae3bd700 1 --2- 172.21.15.113:0/1974990781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8a80fd140 0x7fb8a810e590 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7fb89c0099f0 tx=0x7fb89c009cb0).stop 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.533+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1974990781 shutdown_connections 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.533+0000 7fb8ae3bd700 1 --2- 172.21.15.113:0/1974990781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb89006caa0 0x7fb89006ef60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.533+0000 7fb8ae3bd700 1 --2- 172.21.15.113:0/1974990781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8a80fc7e0 0x7fb8a810e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.533+0000 7fb8ae3bd700 1 --2- 172.21.15.113:0/1974990781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8a80fd140 0x7fb8a810e590 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.533+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1974990781 >> 172.21.15.113:0/1974990781 conn(0x7fb8a80f23c0 msgr2=0x7fb8a8061380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.533+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1974990781 shutdown_connections 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.533+0000 7fb8ae3bd700 1 -- 172.21.15.113:0/1974990781 wait complete. 2024-04-12T05:32:32.535 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:32:32.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.632+0000 7feb8e09f700 1 -- 172.21.15.113:0/3207540361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb880f9200 msgr2=0x7feb88101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.632+0000 7feb8e09f700 1 --2- 172.21.15.113:0/3207540361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb880f9200 0x7feb88101910 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7feb7c009b60 tx=0x7feb7c004d10).stop 2024-04-12T05:32:32.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.632+0000 7feb8e09f700 1 -- 172.21.15.113:0/3207540361 shutdown_connections 2024-04-12T05:32:32.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.632+0000 7feb8e09f700 1 --2- 172.21.15.113:0/3207540361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb880f9200 0x7feb88101910 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.632+0000 7feb8e09f700 1 --2- 172.21.15.113:0/3207540361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb880f88a0 0x7feb880f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.632+0000 7feb8e09f700 1 -- 172.21.15.113:0/3207540361 >> 172.21.15.113:0/3207540361 conn(0x7feb88070590 msgr2=0x7feb880f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:32.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.633+0000 7feb8e09f700 1 -- 172.21.15.113:0/3207540361 shutdown_connections 2024-04-12T05:32:32.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.633+0000 7feb8e09f700 1 -- 172.21.15.113:0/3207540361 wait complete. 2024-04-12T05:32:32.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.633+0000 7feb8e09f700 1 Processor -- start 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.633+0000 7feb8e09f700 1 -- start start 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.633+0000 7feb8e09f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb880f88a0 0x7feb8810e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.633+0000 7feb8e09f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb880f9200 0x7feb8810e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb8e09f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8810eb70 con 0x7feb880f9200 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb8e09f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8810ecb0 con 0x7feb880f88a0 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb86ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb880f9200 0x7feb8810e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb86ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb880f9200 0x7feb8810e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38092/0 (socket says 172.21.15.113:38092) 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb86ffd700 1 -- 172.21.15.113:0/3565171241 learned_addr learned my addr 172.21.15.113:0/3565171241 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb86ffd700 1 -- 172.21.15.113:0/3565171241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb880f88a0 msgr2=0x7feb8810e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb86ffd700 1 --2- 172.21.15.113:0/3565171241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb880f88a0 0x7feb8810e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb86ffd700 1 -- 172.21.15.113:0/3565171241 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c0097c0 con 0x7feb880f9200 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb86ffd700 1 --2- 172.21.15.113:0/3565171241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb880f9200 0x7feb8810e570 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7feb7c009040 tx=0x7feb7c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:32.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb84ff9700 1 -- 172.21.15.113:0/3565171241 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb7c00e070 con 0x7feb880f9200 2024-04-12T05:32:32.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb84ff9700 1 -- 172.21.15.113:0/3565171241 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb7c00bec0 con 0x7feb880f9200 2024-04-12T05:32:32.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb8e09f700 1 -- 172.21.15.113:0/3565171241 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb881082e0 con 0x7feb880f9200 2024-04-12T05:32:32.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb84ff9700 1 -- 172.21.15.113:0/3565171241 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb7c019070 con 0x7feb880f9200 2024-04-12T05:32:32.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.634+0000 7feb8e09f700 1 -- 172.21.15.113:0/3565171241 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb881087c0 con 0x7feb880f9200 2024-04-12T05:32:32.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.635+0000 7feb8e09f700 1 -- 172.21.15.113:0/3565171241 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8805f590 con 0x7feb880f9200 2024-04-12T05:32:32.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.636+0000 7feb84ff9700 1 -- 172.21.15.113:0/3565171241 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7feb7c0139d0 con 0x7feb880f9200 2024-04-12T05:32:32.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.636+0000 7feb84ff9700 1 --2- 172.21.15.113:0/3565171241 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb7006caa0 0x7feb7006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:32.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.636+0000 7feb84ff9700 1 -- 172.21.15.113:0/3565171241 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7feb7c08d140 con 0x7feb880f9200 2024-04-12T05:32:32.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.636+0000 7feb877fe700 1 --2- 172.21.15.113:0/3565171241 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb7006caa0 0x7feb7006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:32.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.637+0000 7feb877fe700 1 --2- 172.21.15.113:0/3565171241 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb7006caa0 0x7feb7006ef60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7feb80004b90 tx=0x7feb80009b60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:32.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.641+0000 7feb84ff9700 1 -- 172.21.15.113:0/3565171241 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7feb7c05abf0 con 0x7feb880f9200 2024-04-12T05:32:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.846+0000 7feb8e09f700 1 -- 172.21.15.113:0/3565171241 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feb881097a0 con 0x7feb7006caa0 2024-04-12T05:32:32.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.848+0000 7feb84ff9700 1 -- 172.21.15.113:0/3565171241 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7feb881097a0 con 0x7feb7006caa0 2024-04-12T05:32:32.847 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:32:32.847 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:32:32.847 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:32:32.848 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:32:32.848 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:32:32.848 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:32:32.848 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:32:32.848 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:32:32.848 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:32:32.848 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:32:32.848 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:32:32.848 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:32:32.848 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:32:32.848 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:32:32.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.850+0000 7feb8e09f700 1 -- 172.21.15.113:0/3565171241 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb7006caa0 msgr2=0x7feb7006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.850+0000 7feb8e09f700 1 --2- 172.21.15.113:0/3565171241 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb7006caa0 0x7feb7006ef60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7feb80004b90 tx=0x7feb80009b60).stop 2024-04-12T05:32:32.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.850+0000 7feb8e09f700 1 -- 172.21.15.113:0/3565171241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb880f9200 msgr2=0x7feb8810e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.850+0000 7feb8e09f700 1 --2- 172.21.15.113:0/3565171241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb880f9200 0x7feb8810e570 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7feb7c009040 tx=0x7feb7c004d10).stop 2024-04-12T05:32:32.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.850+0000 7feb8e09f700 1 -- 172.21.15.113:0/3565171241 shutdown_connections 2024-04-12T05:32:32.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.850+0000 7feb8e09f700 1 --2- 172.21.15.113:0/3565171241 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb7006caa0 0x7feb7006ef60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.850+0000 7feb8e09f700 1 --2- 172.21.15.113:0/3565171241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb880f88a0 0x7feb8810e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:32:32.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.850+0000 7feb8e09f700 1 --2- 172.21.15.113:0/3565171241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb880f9200 0x7feb8810e570 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.850+0000 7feb8e09f700 1 -- 172.21.15.113:0/3565171241 >> 172.21.15.113:0/3565171241 conn(0x7feb88070590 msgr2=0x7feb880fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:32.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.850+0000 7feb8e09f700 1 -- 172.21.15.113:0/3565171241 shutdown_connections 2024-04-12T05:32:32.850 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.850+0000 7feb8e09f700 1 -- 172.21.15.113:0/3565171241 wait complete. 2024-04-12T05:32:32.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.948+0000 7f39687e2700 1 -- 172.21.15.113:0/4049332175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39600fd8c0 msgr2=0x7f39600fdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.948+0000 7f39687e2700 1 --2- 172.21.15.113:0/4049332175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39600fd8c0 0x7f39600fdca0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f39500047f0 tx=0x7f3950009cb0).stop 2024-04-12T05:32:32.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.948+0000 7f39687e2700 1 -- 172.21.15.113:0/4049332175 shutdown_connections 2024-04-12T05:32:32.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.948+0000 7f39687e2700 1 --2- 172.21.15.113:0/4049332175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39600fe220 0x7f39600f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.948+0000 7f39687e2700 1 --2- 172.21.15.113:0/4049332175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39600fd8c0 0x7f39600fdca0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.948+0000 7f39687e2700 1 -- 172.21.15.113:0/4049332175 >> 172.21.15.113:0/4049332175 conn(0x7f39600f34a0 msgr2=0x7f39600f58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:32.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.948+0000 7f39687e2700 1 -- 172.21.15.113:0/4049332175 shutdown_connections 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.948+0000 7f39687e2700 1 -- 172.21.15.113:0/4049332175 wait complete. 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.948+0000 7f39687e2700 1 Processor -- start 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.948+0000 7f39687e2700 1 -- start start 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f39687e2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39600fd8c0 0x7f396010dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f39687e2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39600fe220 0x7f396010e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f39687e2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396010eae0 con 0x7f39600fe220 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f39687e2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396010ec20 con 0x7f39600fd8c0 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f3965d7d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39600fe220 0x7f396010e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f3965d7d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39600fe220 0x7f396010e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38122/0 (socket says 172.21.15.113:38122) 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f3965d7d700 1 -- 172.21.15.113:0/4245194344 learned_addr learned my addr 172.21.15.113:0/4245194344 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f396657e700 1 --2- 172.21.15.113:0/4245194344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39600fd8c0 0x7f396010dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f3965d7d700 1 -- 172.21.15.113:0/4245194344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39600fd8c0 msgr2=0x7f396010dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:32.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f3965d7d700 1 --2- 172.21.15.113:0/4245194344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39600fd8c0 0x7f396010dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:32.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f3965d7d700 1 -- 172.21.15.113:0/4245194344 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3950005040 con 0x7f39600fe220 2024-04-12T05:32:32.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f3965d7d700 1 --2- 172.21.15.113:0/4245194344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39600fe220 0x7f396010e4e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f395c002780 tx=0x7f395c009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:32.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f39577fe700 1 -- 172.21.15.113:0/4245194344 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f395c007b40 con 0x7f39600fe220 2024-04-12T05:32:32.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f39577fe700 1 -- 172.21.15.113:0/4245194344 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f395c00f040 con 0x7f39600fe220 2024-04-12T05:32:32.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f39687e2700 1 -- 172.21.15.113:0/4245194344 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39601082b0 con 0x7f39600fe220 2024-04-12T05:32:32.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f39577fe700 1 -- 172.21.15.113:0/4245194344 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f395c014780 con 0x7f39600fe220 2024-04-12T05:32:32.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.949+0000 7f39687e2700 1 -- 172.21.15.113:0/4245194344 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3960108870 con 0x7f39600fe220 2024-04-12T05:32:32.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.950+0000 7f39687e2700 1 -- 172.21.15.113:0/4245194344 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3960060390 con 0x7f39600fe220 2024-04-12T05:32:32.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.951+0000 7f39577fe700 1 -- 172.21.15.113:0/4245194344 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f395c007ca0 con 0x7f39600fe220 2024-04-12T05:32:32.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.951+0000 7f39577fe700 1 --2- 172.21.15.113:0/4245194344 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394c06caa0 0x7f394c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:32:32.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.951+0000 7f39577fe700 1 -- 172.21.15.113:0/4245194344 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f395c08bd50 con 0x7f39600fe220 2024-04-12T05:32:32.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.951+0000 7f396657e700 1 --2- 172.21.15.113:0/4245194344 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394c06caa0 0x7f394c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:32:32.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.952+0000 7f396657e700 1 --2- 172.21.15.113:0/4245194344 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394c06caa0 0x7f394c06ef60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f3950004f10 tx=0x7f395000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:32:32.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:32.956+0000 7f39577fe700 1 -- 172.21.15.113:0/4245194344 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f395c059800 con 0x7f39600fe220 2024-04-12T05:32:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.224+0000 7f39687e2700 1 -- 172.21.15.113:0/4245194344 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3960047540 con 0x7f39600fe220 2024-04-12T05:32:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.225+0000 7f39577fe700 1 -- 172.21.15.113:0/4245194344 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f395c059390 con 0x7f39600fe220 2024-04-12T05:32:33.225 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:32:33.225 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:32:33.225 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:32:33.225 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:32:33.225 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:32:33.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.227+0000 7f39687e2700 1 -- 172.21.15.113:0/4245194344 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394c06caa0 msgr2=0x7f394c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:33.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.227+0000 7f39687e2700 1 --2- 172.21.15.113:0/4245194344 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394c06caa0 0x7f394c06ef60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f3950004f10 tx=0x7f395000b040).stop 2024-04-12T05:32:33.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.227+0000 7f39687e2700 1 -- 172.21.15.113:0/4245194344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39600fe220 msgr2=0x7f396010e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:32:33.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.227+0000 7f39687e2700 1 --2- 172.21.15.113:0/4245194344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39600fe220 0x7f396010e4e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f395c002780 tx=0x7f395c009ee0).stop 2024-04-12T05:32:33.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.227+0000 7f39687e2700 1 -- 172.21.15.113:0/4245194344 shutdown_connections 2024-04-12T05:32:33.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.227+0000 7f39687e2700 1 --2- 172.21.15.113:0/4245194344 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f394c06caa0 0x7f394c06ef60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:33.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.227+0000 7f39687e2700 1 --2- 172.21.15.113:0/4245194344 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39600fd8c0 0x7f396010dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:33.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.227+0000 7f39687e2700 1 --2- 172.21.15.113:0/4245194344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39600fe220 0x7f396010e4e0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:32:33.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.227+0000 7f39687e2700 1 -- 172.21.15.113:0/4245194344 >> 172.21.15.113:0/4245194344 conn(0x7f39600f34a0 msgr2=0x7f39600fbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:32:33.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.227+0000 7f39687e2700 1 -- 172.21.15.113:0/4245194344 shutdown_connections 2024-04-12T05:32:33.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:32:33.227+0000 7f39687e2700 1 -- 172.21.15.113:0/4245194344 wait complete. 2024-04-12T05:32:33.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:33 smithi113 ceph-mon[103371]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:33.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:33 smithi113 ceph-mon[103371]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:33.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:33 smithi113 ceph-mon[103371]: pgmap v611: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:33.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:33 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/633536702' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:32:33.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:33 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1974990781' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:32:33.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:33 smithi192 ceph-mon[85304]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:33.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:33 smithi192 ceph-mon[85304]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:33.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:33 smithi192 ceph-mon[85304]: pgmap v611: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:33.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:33 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/633536702' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:32:33.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:33 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1974990781' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:32:34.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:34 smithi113 ceph-mon[103371]: from='client.35140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:34.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:34 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/4245194344' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:32:34.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:34 smithi192 ceph-mon[85304]: from='client.35140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:32:34.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:34 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/4245194344' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:32:35.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:35 smithi113 ceph-mon[103371]: pgmap v612: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:35.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:35 smithi192 ceph-mon[85304]: pgmap v612: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:37.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:37 smithi113 ceph-mon[103371]: pgmap v613: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:37.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:37.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:37 smithi192 ceph-mon[85304]: pgmap v613: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:37.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:39.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:39 smithi113 ceph-mon[103371]: pgmap v614: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:39.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:39 smithi192 ceph-mon[85304]: pgmap v614: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:41.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:41 smithi113 ceph-mon[103371]: pgmap v615: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:41.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:41 smithi192 ceph-mon[85304]: pgmap v615: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:43.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:43 smithi113 ceph-mon[103371]: pgmap v616: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:43.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:43 smithi192 ceph-mon[85304]: pgmap v616: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:45.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:45 smithi113 ceph-mon[103371]: pgmap v617: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:45.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:45 smithi192 ceph-mon[85304]: pgmap v617: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:47.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:47 smithi113 ceph-mon[103371]: pgmap v618: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:47.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:47 smithi192 ceph-mon[85304]: pgmap v618: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:49.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:49 smithi113 ceph-mon[103371]: pgmap v619: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:49.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:49 smithi192 ceph-mon[85304]: pgmap v619: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:51.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:51 smithi113 ceph-mon[103371]: pgmap v620: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:51.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:51 smithi192 ceph-mon[85304]: pgmap v620: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:52.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:52.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:32:53.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:53 smithi113 ceph-mon[103371]: pgmap v621: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:53.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:53 smithi192 ceph-mon[85304]: pgmap v621: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:55.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:55 smithi113 ceph-mon[103371]: pgmap v622: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:55.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:55 smithi192 ceph-mon[85304]: pgmap v622: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:57.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:57 smithi113 ceph-mon[103371]: pgmap v623: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:57.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:57 smithi192 ceph-mon[85304]: pgmap v623: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:59.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:32:59 smithi192 ceph-mon[85304]: pgmap v624: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:32:59.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:32:59 smithi113 ceph-mon[103371]: pgmap v624: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:01.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:01 smithi192 ceph-mon[85304]: pgmap v625: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:01.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:01 smithi113 ceph-mon[103371]: pgmap v625: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:03.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.337+0000 7f04d8716700 1 -- 172.21.15.113:0/977600407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04d00fe270 msgr2=0x7f04d00f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.337+0000 7f04d8716700 1 --2- 172.21.15.113:0/977600407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04d00fe270 0x7f04d00f7a50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f04c00047f0 tx=0x7f04c0009cb0).stop 2024-04-12T05:33:03.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 -- 172.21.15.113:0/977600407 shutdown_connections 2024-04-12T05:33:03.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 --2- 172.21.15.113:0/977600407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04d00fe270 0x7f04d00f7a50 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 --2- 172.21.15.113:0/977600407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04d00fd910 0x7f04d00fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:33:03.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 -- 172.21.15.113:0/977600407 >> 172.21.15.113:0/977600407 conn(0x7f04d00f3510 msgr2=0x7f04d00f5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:03.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 -- 172.21.15.113:0/977600407 shutdown_connections 2024-04-12T05:33:03.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 -- 172.21.15.113:0/977600407 wait complete. 2024-04-12T05:33:03.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 Processor -- start 2024-04-12T05:33:03.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 -- start start 2024-04-12T05:33:03.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04d00fd910 0x7f04d010e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:03.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04d00fe270 0x7f04d010e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d010ebe0 con 0x7f04d00fd910 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.338+0000 7f04d8716700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d0108200 con 0x7f04d00fe270 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04d64b2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04d00fd910 0x7f04d010e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04d64b2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04d00fd910 0x7f04d010e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60150/0 (socket says 172.21.15.113:60150) 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04d64b2700 1 -- 172.21.15.113:0/4290715231 learned_addr learned my addr 172.21.15.113:0/4290715231 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04d64b2700 1 -- 172.21.15.113:0/4290715231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04d00fe270 msgr2=0x7f04d010e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04d64b2700 1 --2- 172.21.15.113:0/4290715231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04d00fe270 0x7f04d010e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04d64b2700 1 -- 172.21.15.113:0/4290715231 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c0005040 con 0x7f04d00fd910 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04d64b2700 1 --2- 172.21.15.113:0/4290715231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04d00fd910 0x7f04d010e060 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f04cc00b4b0 tx=0x7f04cc0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04c77fe700 1 -- 172.21.15.113:0/4290715231 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04cc002500 con 0x7f04d00fd910 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04c77fe700 1 -- 172.21.15.113:0/4290715231 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04cc002b10 con 0x7f04d00fd910 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04d8716700 1 -- 172.21.15.113:0/4290715231 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d01084b0 con 0x7f04d00fd910 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04c77fe700 1 -- 172.21.15.113:0/4290715231 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04cc012cf0 con 0x7f04d00fd910 2024-04-12T05:33:03.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.339+0000 7f04d8716700 1 -- 172.21.15.113:0/4290715231 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d01089f0 con 0x7f04d00fd910 2024-04-12T05:33:03.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.340+0000 7f04d8716700 1 -- 172.21.15.113:0/4290715231 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d0060650 con 0x7f04d00fd910 2024-04-12T05:33:03.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.341+0000 7f04c77fe700 1 -- 172.21.15.113:0/4290715231 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f04cc002660 con 0x7f04d00fd910 2024-04-12T05:33:03.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.341+0000 7f04c77fe700 1 --2- 172.21.15.113:0/4290715231 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04bc06caf0 0x7f04bc06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:03.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.341+0000 7f04c77fe700 1 -- 172.21.15.113:0/4290715231 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f04cc08bff0 con 0x7f04d00fd910 2024-04-12T05:33:03.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.341+0000 7f04d5cb1700 1 --2- 172.21.15.113:0/4290715231 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04bc06caf0 0x7f04bc06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:03.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.342+0000 7f04d5cb1700 1 --2- 172.21.15.113:0/4290715231 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04bc06caf0 0x7f04bc06efb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f04d01099a0 tx=0x7f04c0009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:03.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.346+0000 7f04c77fe700 1 -- 172.21.15.113:0/4290715231 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f04cc0599e0 con 0x7f04d00fd910 2024-04-12T05:33:03.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:03 smithi192 ceph-mon[85304]: pgmap v626: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:03.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.551+0000 7f04d8716700 1 -- 172.21.15.113:0/4290715231 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f04d0109750 con 0x7f04bc06caf0 2024-04-12T05:33:03.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.552+0000 7f04c77fe700 1 -- 172.21.15.113:0/4290715231 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f04d0109750 con 0x7f04bc06caf0 2024-04-12T05:33:03.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.554+0000 7f04d8716700 1 -- 172.21.15.113:0/4290715231 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04bc06caf0 msgr2=0x7f04bc06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.554+0000 7f04d8716700 1 --2- 172.21.15.113:0/4290715231 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04bc06caf0 0x7f04bc06efb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f04d01099a0 tx=0x7f04c0009cb0).stop 2024-04-12T05:33:03.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.554+0000 7f04d8716700 1 -- 172.21.15.113:0/4290715231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04d00fd910 msgr2=0x7f04d010e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.554+0000 7f04d8716700 1 --2- 172.21.15.113:0/4290715231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04d00fd910 0x7f04d010e060 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f04cc00b4b0 tx=0x7f04cc0090f0).stop 2024-04-12T05:33:03.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.555+0000 7f04d8716700 1 -- 172.21.15.113:0/4290715231 shutdown_connections 2024-04-12T05:33:03.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.555+0000 7f04d8716700 1 --2- 172.21.15.113:0/4290715231 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04bc06caf0 0x7f04bc06efb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.555+0000 7f04d8716700 1 --2- 172.21.15.113:0/4290715231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04d00fd910 0x7f04d010e060 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.555+0000 7f04d8716700 1 --2- 172.21.15.113:0/4290715231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04d00fe270 0x7f04d010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:33:03.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.555+0000 7f04d8716700 1 -- 172.21.15.113:0/4290715231 >> 172.21.15.113:0/4290715231 conn(0x7f04d00f3510 msgr2=0x7f04d00fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:03.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.555+0000 7f04d8716700 1 -- 172.21.15.113:0/4290715231 shutdown_connections 2024-04-12T05:33:03.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.555+0000 7f04d8716700 1 -- 172.21.15.113:0/4290715231 wait complete. 2024-04-12T05:33:03.564 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:33:03.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.653+0000 7f25aad37700 1 -- 172.21.15.113:0/3176603862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a40fd910 msgr2=0x7f25a40fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.653+0000 7f25aad37700 1 --2- 172.21.15.113:0/3176603862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a40fd910 0x7f25a40fdcf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f25980047f0 tx=0x7f2598009cb0).stop 2024-04-12T05:33:03.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.653+0000 7f25aad37700 1 -- 172.21.15.113:0/3176603862 shutdown_connections 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.653+0000 7f25aad37700 1 --2- 172.21.15.113:0/3176603862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40fe270 0x7f25a40f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.653+0000 7f25aad37700 1 --2- 172.21.15.113:0/3176603862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a40fd910 0x7f25a40fdcf0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.653+0000 7f25aad37700 1 -- 172.21.15.113:0/3176603862 >> 172.21.15.113:0/3176603862 conn(0x7f25a40f3530 msgr2=0x7f25a40f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.653+0000 7f25aad37700 1 -- 172.21.15.113:0/3176603862 shutdown_connections 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.653+0000 7f25aad37700 1 -- 172.21.15.113:0/3176603862 wait complete. 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25aad37700 1 Processor -- start 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25aad37700 1 -- start start 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25aad37700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a40fd910 0x7f25a406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25aad37700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40fe270 0x7f25a406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25aad37700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a406bab0 con 0x7f25a40fd910 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25aad37700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a406bbf0 con 0x7f25a40fe270 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25a8ad3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a40fd910 0x7f25a406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25a8ad3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a40fd910 0x7f25a406b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60160/0 (socket says 172.21.15.113:60160) 2024-04-12T05:33:03.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25a8ad3700 1 -- 172.21.15.113:0/785293768 learned_addr learned my addr 172.21.15.113:0/785293768 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:03.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25a3fff700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40fe270 0x7f25a406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:03.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25a8ad3700 1 -- 172.21.15.113:0/785293768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40fe270 msgr2=0x7f25a406b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25a8ad3700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40fe270 0x7f25a406b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25a8ad3700 1 -- 172.21.15.113:0/785293768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598005040 con 0x7f25a40fd910 2024-04-12T05:33:03.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25a3fff700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40fe270 0x7f25a406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:33:03.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.654+0000 7f25a8ad3700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a40fd910 0x7f25a406b0d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f2598005010 tx=0x7f2598009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:03.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.655+0000 7f25a1ffb700 1 -- 172.21.15.113:0/785293768 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f259800e030 con 0x7f25a40fd910 2024-04-12T05:33:03.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.655+0000 7f25a1ffb700 1 -- 172.21.15.113:0/785293768 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2598007e90 con 0x7f25a40fd910 2024-04-12T05:33:03.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.655+0000 7f25a1ffb700 1 -- 172.21.15.113:0/785293768 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2598018070 con 0x7f25a40fd910 2024-04-12T05:33:03.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.655+0000 7f25aad37700 1 -- 172.21.15.113:0/785293768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a406be40 con 0x7f25a40fd910 2024-04-12T05:33:03.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.655+0000 7f25aad37700 1 -- 172.21.15.113:0/785293768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a41146c0 con 0x7f25a40fd910 2024-04-12T05:33:03.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.656+0000 7f25aad37700 1 -- 172.21.15.113:0/785293768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a405f570 con 0x7f25a40fd910 2024-04-12T05:33:03.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.656+0000 7f25a1ffb700 1 -- 172.21.15.113:0/785293768 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f259801d6e0 con 0x7f25a40fd910 2024-04-12T05:33:03.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.657+0000 7f25a1ffb700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f259406cb70 0x7f259406f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:03.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.657+0000 7f25a1ffb700 1 -- 172.21.15.113:0/785293768 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f259808d120 con 0x7f25a40fd910 2024-04-12T05:33:03.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.657+0000 7f25a3fff700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f259406cb70 0x7f259406f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:03.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.657+0000 7f25a3fff700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f259406cb70 0x7f259406f030 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f25a406c870 tx=0x7f259000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:03.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.661+0000 7f25a1ffb700 1 -- 172.21.15.113:0/785293768 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2598057140 con 0x7f25a40fd910 2024-04-12T05:33:03.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:03 smithi113 ceph-mon[103371]: pgmap v626: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:03.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.864+0000 7f25aad37700 1 -- 172.21.15.113:0/785293768 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f25a4059120 con 0x7f259406cb70 2024-04-12T05:33:03.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.865+0000 7f25a1ffb700 1 -- 172.21.15.113:0/785293768 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f25a4059120 con 0x7f259406cb70 2024-04-12T05:33:03.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.867+0000 7f25aad37700 1 -- 172.21.15.113:0/785293768 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f259406cb70 msgr2=0x7f259406f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.867+0000 7f25aad37700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f259406cb70 0x7f259406f030 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f25a406c870 tx=0x7f259000b040).stop 2024-04-12T05:33:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.867+0000 7f25aad37700 1 -- 172.21.15.113:0/785293768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a40fd910 msgr2=0x7f25a406b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.867+0000 7f25aad37700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a40fd910 0x7f25a406b0d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f2598005010 tx=0x7f2598009cb0).stop 2024-04-12T05:33:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.867+0000 7f25aad37700 1 -- 172.21.15.113:0/785293768 shutdown_connections 2024-04-12T05:33:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.867+0000 7f25aad37700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f259406cb70 0x7f259406f030 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.867+0000 7f25aad37700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25a40fd910 0x7f25a406b0d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.867+0000 7f25aad37700 1 --2- 172.21.15.113:0/785293768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40fe270 0x7f25a406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.867+0000 7f25aad37700 1 -- 172.21.15.113:0/785293768 >> 172.21.15.113:0/785293768 conn(0x7f25a40f3530 msgr2=0x7f25a40fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.867+0000 7f25aad37700 1 -- 172.21.15.113:0/785293768 shutdown_connections 2024-04-12T05:33:03.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.867+0000 7f25aad37700 1 -- 172.21.15.113:0/785293768 wait complete. 2024-04-12T05:33:03.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.965+0000 7fb74460a700 1 -- 172.21.15.113:0/3050001582 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb73c0fd150 msgr2=0x7fb73c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.965+0000 7fb74460a700 1 --2- 172.21.15.113:0/3050001582 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb73c0fd150 0x7fb73c0f6930 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fb7380047f0 tx=0x7fb738009cb0).stop 2024-04-12T05:33:03.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.965+0000 7fb74460a700 1 -- 172.21.15.113:0/3050001582 shutdown_connections 2024-04-12T05:33:03.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.965+0000 7fb74460a700 1 --2- 172.21.15.113:0/3050001582 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb73c0fd150 0x7fb73c0f6930 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.965+0000 7fb74460a700 1 --2- 172.21.15.113:0/3050001582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c0fc7f0 0x7fb73c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.965+0000 7fb74460a700 1 -- 172.21.15.113:0/3050001582 >> 172.21.15.113:0/3050001582 conn(0x7fb73c070590 msgr2=0x7fb73c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:03.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.965+0000 7fb74460a700 1 -- 172.21.15.113:0/3050001582 shutdown_connections 2024-04-12T05:33:03.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.965+0000 7fb74460a700 1 -- 172.21.15.113:0/3050001582 wait complete. 2024-04-12T05:33:03.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb74460a700 1 Processor -- start 2024-04-12T05:33:03.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb74460a700 1 -- start start 2024-04-12T05:33:03.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb74460a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb73c0fc7f0 0x7fb73c10e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:03.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb74460a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c0fd150 0x7fb73c10e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:03.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb74460a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c10eaf0 con 0x7fb73c0fc7f0 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb74460a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c10ec30 con 0x7fb73c0fd150 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb7423a6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb73c0fc7f0 0x7fb73c10e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb7423a6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb73c0fc7f0 0x7fb73c10e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60184/0 (socket says 172.21.15.113:60184) 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb7423a6700 1 -- 172.21.15.113:0/3235007667 learned_addr learned my addr 172.21.15.113:0/3235007667 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb741ba5700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c0fd150 0x7fb73c10e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb7423a6700 1 -- 172.21.15.113:0/3235007667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c0fd150 msgr2=0x7fb73c10e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb7423a6700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c0fd150 0x7fb73c10e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb7423a6700 1 -- 172.21.15.113:0/3235007667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb738005040 con 0x7fb73c0fc7f0 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb741ba5700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c0fd150 0x7fb73c10e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.966+0000 7fb7423a6700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb73c0fc7f0 0x7fb73c10e000 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7fb7300041d0 tx=0x7fb730009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.967+0000 7fb72f7fe700 1 -- 172.21.15.113:0/3235007667 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb730007b40 con 0x7fb73c0fc7f0 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.967+0000 7fb72f7fe700 1 -- 172.21.15.113:0/3235007667 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb73000f040 con 0x7fb73c0fc7f0 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.967+0000 7fb74460a700 1 -- 172.21.15.113:0/3235007667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73c1082c0 con 0x7fb73c0fc7f0 2024-04-12T05:33:03.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.967+0000 7fb72f7fe700 1 -- 172.21.15.113:0/3235007667 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb73000d800 con 0x7fb73c0fc7f0 2024-04-12T05:33:03.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.967+0000 7fb74460a700 1 -- 172.21.15.113:0/3235007667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73c108880 con 0x7fb73c0fc7f0 2024-04-12T05:33:03.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.967+0000 7fb74460a700 1 -- 172.21.15.113:0/3235007667 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73c05f590 con 0x7fb73c0fc7f0 2024-04-12T05:33:03.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.968+0000 7fb72f7fe700 1 -- 172.21.15.113:0/3235007667 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb730007ca0 con 0x7fb73c0fc7f0 2024-04-12T05:33:03.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.969+0000 7fb72f7fe700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb72806caa0 0x7fb72806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:03.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.969+0000 7fb741ba5700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb72806caa0 0x7fb72806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:03.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.969+0000 7fb72f7fe700 1 -- 172.21.15.113:0/3235007667 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb73008b980 con 0x7fb73c0fc7f0 2024-04-12T05:33:03.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.969+0000 7fb741ba5700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb72806caa0 0x7fb72806ef60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fb738004f10 tx=0x7fb73800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:03.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:03.973+0000 7fb72f7fe700 1 -- 172.21.15.113:0/3235007667 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb730059430 con 0x7fb73c0fc7f0 2024-04-12T05:33:04.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.168+0000 7fb74460a700 1 -- 172.21.15.113:0/3235007667 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb73c108f20 con 0x7fb72806caa0 2024-04-12T05:33:04.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.178+0000 7fb72f7fe700 1 -- 172.21.15.113:0/3235007667 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb73c108f20 con 0x7fb72806caa0 2024-04-12T05:33:04.177 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:33:04.177 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (20m) 13m ago 32m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:33:04.177 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (18m) 13m ago 32m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:33:04.177 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (18m) 13m ago 31m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:33:04.177 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (20m) 13m ago 31m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:33:04.177 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (14m) 13m ago 26m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:33:04.177 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (13m) 13m ago 26m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (13m) 13m ago 26m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (26m) 13m ago 26m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (22m) 13m ago 33m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (21m) 13m ago 31m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (19m) 13m ago 33m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (18m) 13m ago 31m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (21m) 13m ago 31m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (21m) 13m ago 31m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (17m) 13m ago 30m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (17m) 13m ago 30m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (16m) 13m ago 29m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (16m) 13m ago 29m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (15m) 13m ago 28m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:33:04.178 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (14m) 13m ago 28m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:33:04.179 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (21m) 13m ago 31m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:33:04.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.180+0000 7fb74460a700 1 -- 172.21.15.113:0/3235007667 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb72806caa0 msgr2=0x7fb72806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.180+0000 7fb74460a700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb72806caa0 0x7fb72806ef60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fb738004f10 tx=0x7fb73800b040).stop 2024-04-12T05:33:04.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.180+0000 7fb74460a700 1 -- 172.21.15.113:0/3235007667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb73c0fc7f0 msgr2=0x7fb73c10e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.180+0000 7fb74460a700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb73c0fc7f0 0x7fb73c10e000 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7fb7300041d0 tx=0x7fb730009f60).stop 2024-04-12T05:33:04.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.180+0000 7fb74460a700 1 -- 172.21.15.113:0/3235007667 shutdown_connections 2024-04-12T05:33:04.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.180+0000 7fb74460a700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb72806caa0 0x7fb72806ef60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.180+0000 7fb74460a700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb73c0fc7f0 0x7fb73c10e000 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.180+0000 7fb74460a700 1 --2- 172.21.15.113:0/3235007667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c0fd150 0x7fb73c10e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.180+0000 7fb74460a700 1 -- 172.21.15.113:0/3235007667 >> 172.21.15.113:0/3235007667 conn(0x7fb73c070590 msgr2=0x7fb73c060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:04.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.180+0000 7fb74460a700 1 -- 172.21.15.113:0/3235007667 shutdown_connections 2024-04-12T05:33:04.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.180+0000 7fb74460a700 1 -- 172.21.15.113:0/3235007667 wait complete. 2024-04-12T05:33:04.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.276+0000 7f833af8e700 1 -- 172.21.15.113:0/1529391447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83340fc7f0 msgr2=0x7f83340fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.276+0000 7f833af8e700 1 --2- 172.21.15.113:0/1529391447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83340fc7f0 0x7f83340fcbd0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f83280047f0 tx=0x7f8328009cb0).stop 2024-04-12T05:33:04.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.276+0000 7f833af8e700 1 -- 172.21.15.113:0/1529391447 shutdown_connections 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.276+0000 7f833af8e700 1 --2- 172.21.15.113:0/1529391447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83340fd150 0x7f83340f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.276+0000 7f833af8e700 1 --2- 172.21.15.113:0/1529391447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83340fc7f0 0x7f83340fcbd0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.276+0000 7f833af8e700 1 -- 172.21.15.113:0/1529391447 >> 172.21.15.113:0/1529391447 conn(0x7f8334070590 msgr2=0x7f83340f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.276+0000 7f833af8e700 1 -- 172.21.15.113:0/1529391447 shutdown_connections 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.276+0000 7f833af8e700 1 -- 172.21.15.113:0/1529391447 wait complete. 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.277+0000 7f833af8e700 1 Processor -- start 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.277+0000 7f833af8e700 1 -- start start 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.277+0000 7f833af8e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83340fc7f0 0x7f833410e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.277+0000 7f833af8e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83340fd150 0x7f833410e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.277+0000 7f833af8e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833410ec00 con 0x7f83340fc7f0 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.277+0000 7f833af8e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8334108220 con 0x7f83340fd150 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.277+0000 7f8338d2a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83340fc7f0 0x7f833410e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:04.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.277+0000 7f8338d2a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83340fc7f0 0x7f833410e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60194/0 (socket says 172.21.15.113:60194) 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.277+0000 7f8338d2a700 1 -- 172.21.15.113:0/3830092566 learned_addr learned my addr 172.21.15.113:0/3830092566 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.278+0000 7f8333fff700 1 --2- 172.21.15.113:0/3830092566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83340fd150 0x7f833410e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.278+0000 7f8338d2a700 1 -- 172.21.15.113:0/3830092566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83340fd150 msgr2=0x7f833410e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.278+0000 7f8338d2a700 1 --2- 172.21.15.113:0/3830092566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83340fd150 0x7f833410e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.278+0000 7f8338d2a700 1 -- 172.21.15.113:0/3830092566 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8328005040 con 0x7f83340fc7f0 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.278+0000 7f8338d2a700 1 --2- 172.21.15.113:0/3830092566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83340fc7f0 0x7f833410e080 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f832800b120 tx=0x7f8328009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.278+0000 7f8331ffb700 1 -- 172.21.15.113:0/3830092566 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f832800e030 con 0x7f83340fc7f0 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.278+0000 7f8331ffb700 1 -- 172.21.15.113:0/3830092566 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8328007e90 con 0x7f83340fc7f0 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.278+0000 7f833af8e700 1 -- 172.21.15.113:0/3830092566 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8334108470 con 0x7f83340fc7f0 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.278+0000 7f8331ffb700 1 -- 172.21.15.113:0/3830092566 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8328018070 con 0x7f83340fc7f0 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.278+0000 7f833af8e700 1 -- 172.21.15.113:0/3830092566 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8334108950 con 0x7f83340fc7f0 2024-04-12T05:33:04.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.279+0000 7f833af8e700 1 -- 172.21.15.113:0/3830092566 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833405f590 con 0x7f83340fc7f0 2024-04-12T05:33:04.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.279+0000 7f8331ffb700 1 -- 172.21.15.113:0/3830092566 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f832801d420 con 0x7f83340fc7f0 2024-04-12T05:33:04.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.280+0000 7f8331ffb700 1 --2- 172.21.15.113:0/3830092566 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832406caa0 0x7f832406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:04.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.280+0000 7f8331ffb700 1 -- 172.21.15.113:0/3830092566 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f832808d320 con 0x7f83340fc7f0 2024-04-12T05:33:04.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.280+0000 7f8333fff700 1 --2- 172.21.15.113:0/3830092566 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832406caa0 0x7f832406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:04.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.280+0000 7f8333fff700 1 --2- 172.21.15.113:0/3830092566 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832406caa0 0x7f832406ef60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f83341099c0 tx=0x7f8320008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:04.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.284+0000 7f8331ffb700 1 -- 172.21.15.113:0/3830092566 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f832805add0 con 0x7f83340fc7f0 2024-04-12T05:33:04.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.551+0000 7f833af8e700 1 -- 172.21.15.113:0/3830092566 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8334109050 con 0x7f83340fc7f0 2024-04-12T05:33:04.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.552+0000 7f8331ffb700 1 -- 172.21.15.113:0/3830092566 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f832805a960 con 0x7f83340fc7f0 2024-04-12T05:33:04.551 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:33:04.551 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:33:04.551 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:33:04.551 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:33:04.552 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:33:04.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.554+0000 7f833af8e700 1 -- 172.21.15.113:0/3830092566 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832406caa0 msgr2=0x7f832406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.554+0000 7f833af8e700 1 --2- 172.21.15.113:0/3830092566 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832406caa0 0x7f832406ef60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f83341099c0 tx=0x7f8320008040).stop 2024-04-12T05:33:04.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.554+0000 7f833af8e700 1 -- 172.21.15.113:0/3830092566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83340fc7f0 msgr2=0x7f833410e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.554+0000 7f833af8e700 1 --2- 172.21.15.113:0/3830092566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83340fc7f0 0x7f833410e080 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f832800b120 tx=0x7f8328009cb0).stop 2024-04-12T05:33:04.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.554+0000 7f833af8e700 1 -- 172.21.15.113:0/3830092566 shutdown_connections 2024-04-12T05:33:04.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.554+0000 7f833af8e700 1 --2- 172.21.15.113:0/3830092566 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f832406caa0 0x7f832406ef60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.554+0000 7f833af8e700 1 --2- 172.21.15.113:0/3830092566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f83340fc7f0 0x7f833410e080 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.554+0000 7f833af8e700 1 --2- 172.21.15.113:0/3830092566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83340fd150 0x7f833410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.554+0000 7f833af8e700 1 -- 172.21.15.113:0/3830092566 >> 172.21.15.113:0/3830092566 conn(0x7f8334070590 msgr2=0x7f8334060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:04.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.554+0000 7f833af8e700 1 -- 172.21.15.113:0/3830092566 shutdown_connections 2024-04-12T05:33:04.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.554+0000 7f833af8e700 1 -- 172.21.15.113:0/3830092566 wait complete. 2024-04-12T05:33:04.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.652+0000 7f5926b45700 1 -- 172.21.15.113:0/1321019006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59200fe250 msgr2=0x7f59200f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.652+0000 7f5926b45700 1 --2- 172.21.15.113:0/1321019006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59200fe250 0x7f59200f7a30 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f590c0047f0 tx=0x7f590c009cb0).stop 2024-04-12T05:33:04.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.652+0000 7f5926b45700 1 -- 172.21.15.113:0/1321019006 shutdown_connections 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.652+0000 7f5926b45700 1 --2- 172.21.15.113:0/1321019006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59200fe250 0x7f59200f7a30 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.652+0000 7f5926b45700 1 --2- 172.21.15.113:0/1321019006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59200fd8f0 0x7f59200fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.652+0000 7f5926b45700 1 -- 172.21.15.113:0/1321019006 >> 172.21.15.113:0/1321019006 conn(0x7f59200f34b0 msgr2=0x7f59200f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.652+0000 7f5926b45700 1 -- 172.21.15.113:0/1321019006 shutdown_connections 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.652+0000 7f5926b45700 1 -- 172.21.15.113:0/1321019006 wait complete. 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.652+0000 7f5926b45700 1 Processor -- start 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f5926b45700 1 -- start start 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f5926b45700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59200fd8f0 0x7f592010e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f5926b45700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59200fe250 0x7f592010e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f5926b45700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592010eb70 con 0x7f59200fd8f0 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f5926b45700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592010ecb0 con 0x7f59200fe250 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f59248e1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59200fd8f0 0x7f592010e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:04.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f59248e1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59200fd8f0 0x7f592010e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60214/0 (socket says 172.21.15.113:60214) 2024-04-12T05:33:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f59248e1700 1 -- 172.21.15.113:0/736015647 learned_addr learned my addr 172.21.15.113:0/736015647 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f591ffff700 1 --2- 172.21.15.113:0/736015647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59200fe250 0x7f592010e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f59248e1700 1 -- 172.21.15.113:0/736015647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59200fe250 msgr2=0x7f592010e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f59248e1700 1 --2- 172.21.15.113:0/736015647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59200fe250 0x7f592010e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f59248e1700 1 -- 172.21.15.113:0/736015647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590c005040 con 0x7f59200fd8f0 2024-04-12T05:33:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f59248e1700 1 --2- 172.21.15.113:0/736015647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59200fd8f0 0x7f592010e080 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f59140041d0 tx=0x7f5914009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f591dffb700 1 -- 172.21.15.113:0/736015647 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5914002d90 con 0x7f59200fd8f0 2024-04-12T05:33:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f591dffb700 1 -- 172.21.15.113:0/736015647 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f591400f040 con 0x7f59200fd8f0 2024-04-12T05:33:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f5926b45700 1 -- 172.21.15.113:0/736015647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5920108340 con 0x7f59200fd8f0 2024-04-12T05:33:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.653+0000 7f591dffb700 1 -- 172.21.15.113:0/736015647 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f591400d800 con 0x7f59200fd8f0 2024-04-12T05:33:04.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.654+0000 7f5926b45700 1 -- 172.21.15.113:0/736015647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5920108880 con 0x7f59200fd8f0 2024-04-12T05:33:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.655+0000 7f5926b45700 1 -- 172.21.15.113:0/736015647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592005f590 con 0x7f59200fd8f0 2024-04-12T05:33:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.655+0000 7f591dffb700 1 -- 172.21.15.113:0/736015647 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f591400da70 con 0x7f59200fd8f0 2024-04-12T05:33:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.655+0000 7f591dffb700 1 --2- 172.21.15.113:0/736015647 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f591006caa0 0x7f591006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:04.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.655+0000 7f591dffb700 1 -- 172.21.15.113:0/736015647 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f591408bdd0 con 0x7f59200fd8f0 2024-04-12T05:33:04.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.655+0000 7f591ffff700 1 --2- 172.21.15.113:0/736015647 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f591006caa0 0x7f591006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:04.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.656+0000 7f591ffff700 1 --2- 172.21.15.113:0/736015647 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f591006caa0 0x7f591006ef60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f590c004f10 tx=0x7f590c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:04.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.660+0000 7f591dffb700 1 -- 172.21.15.113:0/736015647 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5914059880 con 0x7f59200fd8f0 2024-04-12T05:33:04.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.888+0000 7f5926b45700 1 -- 172.21.15.113:0/736015647 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5920109800 con 0x7f59200fd8f0 2024-04-12T05:33:04.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.889+0000 7f591dffb700 1 -- 172.21.15.113:0/736015647 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f5914059410 con 0x7f59200fd8f0 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:33:04.889 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:33:04.890 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:33:04.891 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:33:04.891 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:33:04.891 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:33:04.891 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:33:04.891 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:33:04.891 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:33:04.891 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:33:04.891 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:33:04.892 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:33:04.892 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:33:04.892 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:33:04.892 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:33:04.892 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:33:04.892 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.891+0000 7f5926b45700 1 -- 172.21.15.113:0/736015647 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f591006caa0 msgr2=0x7f591006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.891+0000 7f5926b45700 1 --2- 172.21.15.113:0/736015647 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f591006caa0 0x7f591006ef60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f590c004f10 tx=0x7f590c00b040).stop 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.891+0000 7f5926b45700 1 -- 172.21.15.113:0/736015647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59200fd8f0 msgr2=0x7f592010e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.891+0000 7f5926b45700 1 --2- 172.21.15.113:0/736015647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59200fd8f0 0x7f592010e080 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f59140041d0 tx=0x7f5914009f60).stop 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.891+0000 7f5926b45700 1 -- 172.21.15.113:0/736015647 shutdown_connections 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.891+0000 7f5926b45700 1 --2- 172.21.15.113:0/736015647 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f591006caa0 0x7f591006ef60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.891+0000 7f5926b45700 1 --2- 172.21.15.113:0/736015647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59200fd8f0 0x7f592010e080 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.891+0000 7f5926b45700 1 --2- 172.21.15.113:0/736015647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59200fe250 0x7f592010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.891+0000 7f5926b45700 1 -- 172.21.15.113:0/736015647 >> 172.21.15.113:0/736015647 conn(0x7f59200f34b0 msgr2=0x7f59200fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.891+0000 7f5926b45700 1 -- 172.21.15.113:0/736015647 shutdown_connections 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.891+0000 7f5926b45700 1 -- 172.21.15.113:0/736015647 wait complete. 2024-04-12T05:33:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:33:04.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.990+0000 7f4d3c836700 1 -- 172.21.15.113:0/1003712722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d380fe250 msgr2=0x7f4d380f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.990+0000 7f4d3c836700 1 --2- 172.21.15.113:0/1003712722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d380fe250 0x7f4d380f7a30 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f4d280047f0 tx=0x7f4d28009cb0).stop 2024-04-12T05:33:04.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.990+0000 7f4d3c836700 1 -- 172.21.15.113:0/1003712722 shutdown_connections 2024-04-12T05:33:04.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.990+0000 7f4d3c836700 1 --2- 172.21.15.113:0/1003712722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d380fe250 0x7f4d380f7a30 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.990+0000 7f4d3c836700 1 --2- 172.21.15.113:0/1003712722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d380fd8f0 0x7f4d380fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:33:04.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.990+0000 7f4d3c836700 1 -- 172.21.15.113:0/1003712722 >> 172.21.15.113:0/1003712722 conn(0x7f4d380f34b0 msgr2=0x7f4d380f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:04.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.990+0000 7f4d3c836700 1 -- 172.21.15.113:0/1003712722 shutdown_connections 2024-04-12T05:33:04.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.990+0000 7f4d3c836700 1 -- 172.21.15.113:0/1003712722 wait complete. 2024-04-12T05:33:04.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d3c836700 1 Processor -- start 2024-04-12T05:33:04.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d3c836700 1 -- start start 2024-04-12T05:33:04.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d3c836700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d380fd8f0 0x7f4d3810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:04.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d3c836700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d380fe250 0x7f4d3810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:04.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d3c836700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3810ebb0 con 0x7f4d380fe250 2024-04-12T05:33:04.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d3c836700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d381081d0 con 0x7f4d380fd8f0 2024-04-12T05:33:04.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d35d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d380fe250 0x7f4d3810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:04.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d35d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d380fe250 0x7f4d3810e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60238/0 (socket says 172.21.15.113:60238) 2024-04-12T05:33:04.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d35d9b700 1 -- 172.21.15.113:0/2404299739 learned_addr learned my addr 172.21.15.113:0/2404299739 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d3659c700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d380fd8f0 0x7f4d3810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d35d9b700 1 -- 172.21.15.113:0/2404299739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d380fd8f0 msgr2=0x7f4d3810e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.991+0000 7f4d35d9b700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d380fd8f0 0x7f4d3810e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.992+0000 7f4d35d9b700 1 -- 172.21.15.113:0/2404299739 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d28005040 con 0x7f4d380fe250 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.992+0000 7f4d3659c700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d380fd8f0 0x7f4d3810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.992+0000 7f4d35d9b700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d380fe250 0x7f4d3810e520 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f4d280099f0 tx=0x7f4d28009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.992+0000 7f4d277fe700 1 -- 172.21.15.113:0/2404299739 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d2800e030 con 0x7f4d380fe250 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.992+0000 7f4d277fe700 1 -- 172.21.15.113:0/2404299739 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d28007e90 con 0x7f4d380fe250 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.992+0000 7f4d3c836700 1 -- 172.21.15.113:0/2404299739 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d38108420 con 0x7f4d380fe250 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.992+0000 7f4d277fe700 1 -- 172.21.15.113:0/2404299739 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d28018070 con 0x7f4d380fe250 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.992+0000 7f4d3c836700 1 -- 172.21.15.113:0/2404299739 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d38108900 con 0x7f4d380fe250 2024-04-12T05:33:04.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.993+0000 7f4d3c836700 1 -- 172.21.15.113:0/2404299739 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d3805f590 con 0x7f4d380fe250 2024-04-12T05:33:04.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.993+0000 7f4d277fe700 1 -- 172.21.15.113:0/2404299739 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4d2801d420 con 0x7f4d380fe250 2024-04-12T05:33:04.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.994+0000 7f4d277fe700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4d2006caa0 0x7f4d2006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:04.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.994+0000 7f4d277fe700 1 -- 172.21.15.113:0/2404299739 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f4d2808d320 con 0x7f4d380fe250 2024-04-12T05:33:04.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.994+0000 7f4d3659c700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4d2006caa0 0x7f4d2006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:04.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.994+0000 7f4d3659c700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4d2006caa0 0x7f4d2006ef60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f4d2c009990 tx=0x7f4d2c008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:04.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:04.998+0000 7f4d277fe700 1 -- 172.21.15.113:0/2404299739 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4d2805add0 con 0x7f4d380fe250 2024-04-12T05:33:05.204 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.204+0000 7f4d3c836700 1 -- 172.21.15.113:0/2404299739 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d381096b0 con 0x7f4d2006caa0 2024-04-12T05:33:05.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.206+0000 7f4d277fe700 1 -- 172.21.15.113:0/2404299739 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4d381096b0 con 0x7f4d2006caa0 2024-04-12T05:33:05.205 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:33:05.205 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:33:05.205 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:33:05.206 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:33:05.206 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:33:05.206 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:33:05.206 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:33:05.206 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:33:05.206 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:33:05.206 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:33:05.206 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:33:05.206 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:33:05.206 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:33:05.207 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:33:05.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.208+0000 7f4d3c836700 1 -- 172.21.15.113:0/2404299739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4d2006caa0 msgr2=0x7f4d2006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:05.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.208+0000 7f4d3c836700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4d2006caa0 0x7f4d2006ef60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f4d2c009990 tx=0x7f4d2c008040).stop 2024-04-12T05:33:05.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.208+0000 7f4d3c836700 1 -- 172.21.15.113:0/2404299739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d380fe250 msgr2=0x7f4d3810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:05.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.208+0000 7f4d3c836700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d380fe250 0x7f4d3810e520 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f4d280099f0 tx=0x7f4d28009cb0).stop 2024-04-12T05:33:05.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.208+0000 7f4d3c836700 1 -- 172.21.15.113:0/2404299739 shutdown_connections 2024-04-12T05:33:05.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.208+0000 7f4d3c836700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4d2006caa0 0x7f4d2006ef60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:05.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.208+0000 7f4d3c836700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d380fd8f0 0x7f4d3810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:05.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.208+0000 7f4d3c836700 1 --2- 172.21.15.113:0/2404299739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d380fe250 0x7f4d3810e520 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:05.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.208+0000 7f4d3c836700 1 -- 172.21.15.113:0/2404299739 >> 172.21.15.113:0/2404299739 conn(0x7f4d380f34b0 msgr2=0x7f4d380fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:05.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.208+0000 7f4d3c836700 1 -- 172.21.15.113:0/2404299739 shutdown_connections 2024-04-12T05:33:05.208 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.208+0000 7f4d3c836700 1 -- 172.21.15.113:0/2404299739 wait complete. 2024-04-12T05:33:05.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.306+0000 7fc23d185700 1 -- 172.21.15.113:0/2724343896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2380fd8f0 msgr2=0x7fc2380fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:05.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.306+0000 7fc23d185700 1 --2- 172.21.15.113:0/2724343896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2380fd8f0 0x7fc2380fdcd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7fc2280047f0 tx=0x7fc228009cb0).stop 2024-04-12T05:33:05.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.306+0000 7fc23d185700 1 -- 172.21.15.113:0/2724343896 shutdown_connections 2024-04-12T05:33:05.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.306+0000 7fc23d185700 1 --2- 172.21.15.113:0/2724343896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2380fe250 0x7fc2380f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:05.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.306+0000 7fc23d185700 1 --2- 172.21.15.113:0/2724343896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2380fd8f0 0x7fc2380fdcd0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:05.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.306+0000 7fc23d185700 1 -- 172.21.15.113:0/2724343896 >> 172.21.15.113:0/2724343896 conn(0x7fc2380f34b0 msgr2=0x7fc2380f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:05.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.306+0000 7fc23d185700 1 -- 172.21.15.113:0/2724343896 shutdown_connections 2024-04-12T05:33:05.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.306+0000 7fc23d185700 1 -- 172.21.15.113:0/2724343896 wait complete. 2024-04-12T05:33:05.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.307+0000 7fc23d185700 1 Processor -- start 2024-04-12T05:33:05.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.307+0000 7fc23d185700 1 -- start start 2024-04-12T05:33:05.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.307+0000 7fc23d185700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2380fd8f0 0x7fc23810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:05.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.307+0000 7fc23d185700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2380fe250 0x7fc23810e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:05.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.307+0000 7fc23d185700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23810ead0 con 0x7fc2380fd8f0 2024-04-12T05:33:05.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.307+0000 7fc23d185700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23810ec10 con 0x7fc2380fe250 2024-04-12T05:33:05.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc236d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2380fd8f0 0x7fc23810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:05.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc236d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2380fd8f0 0x7fc23810dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:60254/0 (socket says 172.21.15.113:60254) 2024-04-12T05:33:05.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc236d9d700 1 -- 172.21.15.113:0/4285289781 learned_addr learned my addr 172.21.15.113:0/4285289781 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:05.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc23659c700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2380fe250 0x7fc23810e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:05.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc236d9d700 1 -- 172.21.15.113:0/4285289781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2380fe250 msgr2=0x7fc23810e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:05.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc236d9d700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2380fe250 0x7fc23810e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:05.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc236d9d700 1 -- 172.21.15.113:0/4285289781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc228005040 con 0x7fc2380fd8f0 2024-04-12T05:33:05.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc23659c700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2380fe250 0x7fc23810e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:33:05.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc236d9d700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2380fd8f0 0x7fc23810dfe0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7fc22800b120 tx=0x7fc228009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:05.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc21ffff700 1 -- 172.21.15.113:0/4285289781 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc22800e030 con 0x7fc2380fd8f0 2024-04-12T05:33:05.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc21ffff700 1 -- 172.21.15.113:0/4285289781 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc228007e90 con 0x7fc2380fd8f0 2024-04-12T05:33:05.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc23d185700 1 -- 172.21.15.113:0/4285289781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc238108240 con 0x7fc2380fd8f0 2024-04-12T05:33:05.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc21ffff700 1 -- 172.21.15.113:0/4285289781 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc228018070 con 0x7fc2380fd8f0 2024-04-12T05:33:05.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.308+0000 7fc23d185700 1 -- 172.21.15.113:0/4285289781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2381087a0 con 0x7fc2380fd8f0 2024-04-12T05:33:05.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.309+0000 7fc23d185700 1 -- 172.21.15.113:0/4285289781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23805f590 con 0x7fc2380fd8f0 2024-04-12T05:33:05.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.310+0000 7fc21ffff700 1 -- 172.21.15.113:0/4285289781 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc22801d420 con 0x7fc2380fd8f0 2024-04-12T05:33:05.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.310+0000 7fc21ffff700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 0x7fc22006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:05.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.310+0000 7fc21ffff700 1 -- 172.21.15.113:0/4285289781 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc22808d300 con 0x7fc2380fd8f0 2024-04-12T05:33:05.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.310+0000 7fc23659c700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 0x7fc22006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:05.310 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.310+0000 7fc23659c700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 0x7fc22006ef60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fc238109920 tx=0x7fc22c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:05.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.314+0000 7fc21ffff700 1 -- 172.21.15.113:0/4285289781 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc22805adb0 con 0x7fc2380fd8f0 2024-04-12T05:33:05.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:05 smithi192 ceph-mon[85304]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:05.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:05 smithi192 ceph-mon[85304]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:05.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:05 smithi192 ceph-mon[85304]: pgmap v627: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:05.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:05 smithi192 ceph-mon[85304]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:05.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:05 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3830092566' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:33:05.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:05 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/736015647' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:33:05.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.580+0000 7fc23d185700 1 -- 172.21.15.113:0/4285289781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc238109700 con 0x7fc2380fd8f0 2024-04-12T05:33:05.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.580+0000 7fc21ffff700 1 -- 172.21.15.113:0/4285289781 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc22805a940 con 0x7fc2380fd8f0 2024-04-12T05:33:05.580 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:33:05.580 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:33:05.580 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:33:05.581 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:33:05.581 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:33:05.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.582+0000 7fc23d185700 1 -- 172.21.15.113:0/4285289781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 msgr2=0x7fc22006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:05.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.582+0000 7fc23d185700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 0x7fc22006ef60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fc238109920 tx=0x7fc22c009040).stop 2024-04-12T05:33:05.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.582+0000 7fc23d185700 1 -- 172.21.15.113:0/4285289781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2380fd8f0 msgr2=0x7fc23810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:05.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.582+0000 7fc23d185700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2380fd8f0 0x7fc23810dfe0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7fc22800b120 tx=0x7fc228009cb0).stop 2024-04-12T05:33:05.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.583+0000 7fc23d185700 1 -- 172.21.15.113:0/4285289781 shutdown_connections 2024-04-12T05:33:05.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.583+0000 7fc23d185700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 0x7fc22006ef60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:05.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.583+0000 7fc23d185700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2380fd8f0 0x7fc23810dfe0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:05.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.583+0000 7fc23d185700 1 --2- 172.21.15.113:0/4285289781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2380fe250 0x7fc23810e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:05.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.583+0000 7fc23d185700 1 -- 172.21.15.113:0/4285289781 >> 172.21.15.113:0/4285289781 conn(0x7fc2380f34b0 msgr2=0x7fc2380fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:05.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.583+0000 7fc23d185700 1 -- 172.21.15.113:0/4285289781 shutdown_connections 2024-04-12T05:33:05.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:05.583+0000 7fc23d185700 1 -- 172.21.15.113:0/4285289781 wait complete. 2024-04-12T05:33:05.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:05 smithi113 ceph-mon[103371]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:05.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:05 smithi113 ceph-mon[103371]: from='client.35152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:05.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:05 smithi113 ceph-mon[103371]: pgmap v627: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:05.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:05 smithi113 ceph-mon[103371]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:05.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:05 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3830092566' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:33:05.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:05 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/736015647' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:33:06.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:06 smithi192 ceph-mon[85304]: from='client.35168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:06.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:06 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/4285289781' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:33:06.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:06 smithi113 ceph-mon[103371]: from='client.35168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:06.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:06 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/4285289781' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:33:07.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:07 smithi192 ceph-mon[85304]: pgmap v628: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:07.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:07.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:07 smithi113 ceph-mon[103371]: pgmap v628: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:07.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:09.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:09 smithi192 ceph-mon[85304]: pgmap v629: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:09.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:09 smithi113 ceph-mon[103371]: pgmap v629: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:11.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:11 smithi113 ceph-mon[103371]: pgmap v630: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:11.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:11 smithi192 ceph-mon[85304]: pgmap v630: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:13.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:13 smithi113 ceph-mon[103371]: pgmap v631: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:13.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:13 smithi192 ceph-mon[85304]: pgmap v631: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:15.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:15 smithi113 ceph-mon[103371]: pgmap v632: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:15.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:15 smithi192 ceph-mon[85304]: pgmap v632: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:17.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:17 smithi113 ceph-mon[103371]: pgmap v633: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:17.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:17 smithi192 ceph-mon[85304]: pgmap v633: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:19.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:19 smithi113 ceph-mon[103371]: pgmap v634: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:19.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:19 smithi192 ceph-mon[85304]: pgmap v634: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:20.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:20 smithi113 ceph-mon[103371]: pgmap v635: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:20.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:20 smithi192 ceph-mon[85304]: pgmap v635: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:22.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:22.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:23.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:22 smithi113 ceph-mon[103371]: pgmap v636: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:23.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:22 smithi192 ceph-mon[85304]: pgmap v636: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:25.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:25 smithi113 ceph-mon[103371]: pgmap v637: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:25.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:25 smithi192 ceph-mon[85304]: pgmap v637: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:27.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:27 smithi113 ceph-mon[103371]: pgmap v638: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:27.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:27 smithi192 ceph-mon[85304]: pgmap v638: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:29.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:29 smithi113 ceph-mon[103371]: pgmap v639: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:29.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:29 smithi192 ceph-mon[85304]: pgmap v639: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:31.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:31 smithi113 ceph-mon[103371]: pgmap v640: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:31.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:31 smithi192 ceph-mon[85304]: pgmap v640: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:33.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:33 smithi113 ceph-mon[103371]: pgmap v641: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:33.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:33 smithi192 ceph-mon[85304]: pgmap v641: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:35.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:35 smithi113 ceph-mon[103371]: pgmap v642: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:35.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:35 smithi192 ceph-mon[85304]: pgmap v642: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:35.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.693+0000 7f0068378700 1 -- 172.21.15.113:0/3177959568 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00600fd910 msgr2=0x7f00600fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:35.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.693+0000 7f0068378700 1 --2- 172.21.15.113:0/3177959568 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00600fd910 0x7f00600fdcf0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f00500047f0 tx=0x7f0050009cb0).stop 2024-04-12T05:33:35.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.693+0000 7f0068378700 1 -- 172.21.15.113:0/3177959568 shutdown_connections 2024-04-12T05:33:35.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.693+0000 7f0068378700 1 --2- 172.21.15.113:0/3177959568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00600fe270 0x7f00600f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:35.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.693+0000 7f0068378700 1 --2- 172.21.15.113:0/3177959568 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00600fd910 0x7f00600fdcf0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:35.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.693+0000 7f0068378700 1 -- 172.21.15.113:0/3177959568 >> 172.21.15.113:0/3177959568 conn(0x7f00600f3530 msgr2=0x7f00600f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:35.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.693+0000 7f0068378700 1 -- 172.21.15.113:0/3177959568 shutdown_connections 2024-04-12T05:33:35.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.693+0000 7f0068378700 1 -- 172.21.15.113:0/3177959568 wait complete. 2024-04-12T05:33:35.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.694+0000 7f0068378700 1 Processor -- start 2024-04-12T05:33:35.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.694+0000 7f0068378700 1 -- start start 2024-04-12T05:33:35.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.694+0000 7f0068378700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00600fd910 0x7f006006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:35.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.694+0000 7f0068378700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00600fe270 0x7f006006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:35.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.694+0000 7f0068378700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006006bab0 con 0x7f00600fe270 2024-04-12T05:33:35.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.694+0000 7f0068378700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006006bbf0 con 0x7f00600fd910 2024-04-12T05:33:35.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.694+0000 7f0065913700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00600fe270 0x7f006006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:35.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.694+0000 7f0065913700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00600fe270 0x7f006006b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34200/0 (socket says 172.21.15.113:34200) 2024-04-12T05:33:35.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.694+0000 7f0065913700 1 -- 172.21.15.113:0/2221813732 learned_addr learned my addr 172.21.15.113:0/2221813732 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:35.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.694+0000 7f0066114700 1 --2- 172.21.15.113:0/2221813732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00600fd910 0x7f006006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:35.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.695+0000 7f0065913700 1 -- 172.21.15.113:0/2221813732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00600fd910 msgr2=0x7f006006b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:35.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.695+0000 7f0065913700 1 --2- 172.21.15.113:0/2221813732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00600fd910 0x7f006006b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.695+0000 7f0065913700 1 -- 172.21.15.113:0/2221813732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0050005040 con 0x7f00600fe270 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.695+0000 7f0065913700 1 --2- 172.21.15.113:0/2221813732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00600fe270 0x7f006006b5c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f005c002820 tx=0x7f005c009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.695+0000 7f00577fe700 1 -- 172.21.15.113:0/2221813732 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f005c007b40 con 0x7f00600fe270 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.695+0000 7f00577fe700 1 -- 172.21.15.113:0/2221813732 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f005c00f040 con 0x7f00600fe270 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.695+0000 7f0068378700 1 -- 172.21.15.113:0/2221813732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006006bea0 con 0x7f00600fe270 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.695+0000 7f00577fe700 1 -- 172.21.15.113:0/2221813732 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f005c0148b0 con 0x7f00600fe270 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.695+0000 7f0068378700 1 -- 172.21.15.113:0/2221813732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0060114710 con 0x7f00600fe270 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.696+0000 7f0068378700 1 -- 172.21.15.113:0/2221813732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006005f570 con 0x7f00600fe270 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.696+0000 7f00577fe700 1 -- 172.21.15.113:0/2221813732 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f005c007ca0 con 0x7f00600fe270 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.697+0000 7f00577fe700 1 --2- 172.21.15.113:0/2221813732 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f004c06caa0 0x7f004c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.697+0000 7f00577fe700 1 -- 172.21.15.113:0/2221813732 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f005c08b7e0 con 0x7f00600fe270 2024-04-12T05:33:35.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.697+0000 7f0066114700 1 --2- 172.21.15.113:0/2221813732 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f004c06caa0 0x7f004c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:35.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.697+0000 7f0066114700 1 --2- 172.21.15.113:0/2221813732 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f004c06caa0 0x7f004c06ef60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f0050004f10 tx=0x7f005000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:35.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.701+0000 7f00577fe700 1 -- 172.21.15.113:0/2221813732 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f005c055800 con 0x7f00600fe270 2024-04-12T05:33:35.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.906+0000 7f0068378700 1 -- 172.21.15.113:0/2221813732 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f006006c6c0 con 0x7f004c06caa0 2024-04-12T05:33:35.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.907+0000 7f00577fe700 1 -- 172.21.15.113:0/2221813732 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f006006c6c0 con 0x7f004c06caa0 2024-04-12T05:33:35.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.909+0000 7f0068378700 1 -- 172.21.15.113:0/2221813732 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f004c06caa0 msgr2=0x7f004c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:35.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.909+0000 7f0068378700 1 --2- 172.21.15.113:0/2221813732 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f004c06caa0 0x7f004c06ef60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f0050004f10 tx=0x7f005000b040).stop 2024-04-12T05:33:35.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.909+0000 7f0068378700 1 -- 172.21.15.113:0/2221813732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00600fe270 msgr2=0x7f006006b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:35.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.909+0000 7f0068378700 1 --2- 172.21.15.113:0/2221813732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00600fe270 0x7f006006b5c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f005c002820 tx=0x7f005c009f20).stop 2024-04-12T05:33:35.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.909+0000 7f0068378700 1 -- 172.21.15.113:0/2221813732 shutdown_connections 2024-04-12T05:33:35.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.909+0000 7f0068378700 1 --2- 172.21.15.113:0/2221813732 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f004c06caa0 0x7f004c06ef60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:35.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.909+0000 7f0068378700 1 --2- 172.21.15.113:0/2221813732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00600fd910 0x7f006006b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:35.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.909+0000 7f0068378700 1 --2- 172.21.15.113:0/2221813732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00600fe270 0x7f006006b5c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:35.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.909+0000 7f0068378700 1 -- 172.21.15.113:0/2221813732 >> 172.21.15.113:0/2221813732 conn(0x7f00600f3530 msgr2=0x7f00600fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:35.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.909+0000 7f0068378700 1 -- 172.21.15.113:0/2221813732 shutdown_connections 2024-04-12T05:33:35.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:35.909+0000 7f0068378700 1 -- 172.21.15.113:0/2221813732 wait complete. 2024-04-12T05:33:35.918 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:33:36.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.008+0000 7fe4d58c0700 1 -- 172.21.15.113:0/2228099612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d00fc0e0 msgr2=0x7fe4d00fc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.008+0000 7fe4d58c0700 1 --2- 172.21.15.113:0/2228099612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d00fc0e0 0x7fe4d00fc4c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fe4c40047f0 tx=0x7fe4c4009cb0).stop 2024-04-12T05:33:36.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.008+0000 7fe4d58c0700 1 -- 172.21.15.113:0/2228099612 shutdown_connections 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.008+0000 7fe4d58c0700 1 --2- 172.21.15.113:0/2228099612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4d00fc9b0 0x7fe4d00f61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.008+0000 7fe4d58c0700 1 --2- 172.21.15.113:0/2228099612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d00fc0e0 0x7fe4d00fc4c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.008+0000 7fe4d58c0700 1 -- 172.21.15.113:0/2228099612 >> 172.21.15.113:0/2228099612 conn(0x7fe4d006cea0 msgr2=0x7fe4d006d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.008+0000 7fe4d58c0700 1 -- 172.21.15.113:0/2228099612 shutdown_connections 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.008+0000 7fe4d58c0700 1 -- 172.21.15.113:0/2228099612 wait complete. 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4d58c0700 1 Processor -- start 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4d58c0700 1 -- start start 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4d58c0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d00fc0e0 0x7fe4d010e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4d58c0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4d00fc9b0 0x7fe4d010e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4d58c0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d010ebb0 con 0x7fe4d00fc0e0 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4d58c0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d010ecf0 con 0x7fe4d00fc9b0 2024-04-12T05:33:36.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4ceffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d00fc0e0 0x7fe4d010e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:36.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4ceffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d00fc0e0 0x7fe4d010e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34222/0 (socket says 172.21.15.113:34222) 2024-04-12T05:33:36.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4ceffd700 1 -- 172.21.15.113:0/4023973532 learned_addr learned my addr 172.21.15.113:0/4023973532 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:36.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4ce7fc700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4d00fc9b0 0x7fe4d010e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:36.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4ceffd700 1 -- 172.21.15.113:0/4023973532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4d00fc9b0 msgr2=0x7fe4d010e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4ceffd700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4d00fc9b0 0x7fe4d010e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4ceffd700 1 -- 172.21.15.113:0/4023973532 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c4005040 con 0x7fe4d00fc0e0 2024-04-12T05:33:36.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.009+0000 7fe4ce7fc700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4d00fc9b0 0x7fe4d010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:33:36.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.010+0000 7fe4ceffd700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d00fc0e0 0x7fe4d010e0c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fe4c4009380 tx=0x7fe4c4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:36.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.010+0000 7fe4b7fff700 1 -- 172.21.15.113:0/4023973532 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4c4018070 con 0x7fe4d00fc0e0 2024-04-12T05:33:36.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.010+0000 7fe4b7fff700 1 -- 172.21.15.113:0/4023973532 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4c4007e90 con 0x7fe4d00fc0e0 2024-04-12T05:33:36.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.010+0000 7fe4d58c0700 1 -- 172.21.15.113:0/4023973532 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d0108320 con 0x7fe4d00fc0e0 2024-04-12T05:33:36.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.010+0000 7fe4b7fff700 1 -- 172.21.15.113:0/4023973532 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4c40123c0 con 0x7fe4d00fc0e0 2024-04-12T05:33:36.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.010+0000 7fe4d58c0700 1 -- 172.21.15.113:0/4023973532 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d0108880 con 0x7fe4d00fc0e0 2024-04-12T05:33:36.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.011+0000 7fe4d58c0700 1 -- 172.21.15.113:0/4023973532 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d0047540 con 0x7fe4d00fc0e0 2024-04-12T05:33:36.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.011+0000 7fe4b7fff700 1 -- 172.21.15.113:0/4023973532 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe4c401d420 con 0x7fe4d00fc0e0 2024-04-12T05:33:36.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.012+0000 7fe4b7fff700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4b806caa0 0x7fe4b806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:36.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.012+0000 7fe4b7fff700 1 -- 172.21.15.113:0/4023973532 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe4c408cc50 con 0x7fe4d00fc0e0 2024-04-12T05:33:36.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.012+0000 7fe4ce7fc700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4b806caa0 0x7fe4b806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:36.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.012+0000 7fe4ce7fc700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4b806caa0 0x7fe4b806ef60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fe4c0007fd0 tx=0x7fe4c000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:36.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.016+0000 7fe4b7fff700 1 -- 172.21.15.113:0/4023973532 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe4c405a640 con 0x7fe4d00fc0e0 2024-04-12T05:33:36.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.221+0000 7fe4d58c0700 1 -- 172.21.15.113:0/4023973532 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4d00fa220 con 0x7fe4b806caa0 2024-04-12T05:33:36.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.223+0000 7fe4b7fff700 1 -- 172.21.15.113:0/4023973532 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe4d00fa220 con 0x7fe4b806caa0 2024-04-12T05:33:36.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.225+0000 7fe4d58c0700 1 -- 172.21.15.113:0/4023973532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4b806caa0 msgr2=0x7fe4b806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.225+0000 7fe4d58c0700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4b806caa0 0x7fe4b806ef60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fe4c0007fd0 tx=0x7fe4c000b040).stop 2024-04-12T05:33:36.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.225+0000 7fe4d58c0700 1 -- 172.21.15.113:0/4023973532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d00fc0e0 msgr2=0x7fe4d010e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.225+0000 7fe4d58c0700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d00fc0e0 0x7fe4d010e0c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fe4c4009380 tx=0x7fe4c4009cb0).stop 2024-04-12T05:33:36.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.225+0000 7fe4d58c0700 1 -- 172.21.15.113:0/4023973532 shutdown_connections 2024-04-12T05:33:36.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.225+0000 7fe4d58c0700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe4b806caa0 0x7fe4b806ef60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.225+0000 7fe4d58c0700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4d00fc0e0 0x7fe4d010e0c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.225+0000 7fe4d58c0700 1 --2- 172.21.15.113:0/4023973532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4d00fc9b0 0x7fe4d010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.225+0000 7fe4d58c0700 1 -- 172.21.15.113:0/4023973532 >> 172.21.15.113:0/4023973532 conn(0x7fe4d006cea0 msgr2=0x7fe4d0101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:36.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.225+0000 7fe4d58c0700 1 -- 172.21.15.113:0/4023973532 shutdown_connections 2024-04-12T05:33:36.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.225+0000 7fe4d58c0700 1 -- 172.21.15.113:0/4023973532 wait complete. 2024-04-12T05:33:36.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.322+0000 7f576b851700 1 -- 172.21.15.113:0/4078917222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57640f9200 msgr2=0x7f5764101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.322+0000 7f576b851700 1 --2- 172.21.15.113:0/4078917222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57640f9200 0x7f5764101910 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f57540047f0 tx=0x7f5754009cb0).stop 2024-04-12T05:33:36.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 -- 172.21.15.113:0/4078917222 shutdown_connections 2024-04-12T05:33:36.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 --2- 172.21.15.113:0/4078917222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57640f9200 0x7f5764101910 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 --2- 172.21.15.113:0/4078917222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57640f88a0 0x7f57640f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 -- 172.21.15.113:0/4078917222 >> 172.21.15.113:0/4078917222 conn(0x7f5764070590 msgr2=0x7f57640f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:36.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 -- 172.21.15.113:0/4078917222 shutdown_connections 2024-04-12T05:33:36.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 -- 172.21.15.113:0/4078917222 wait complete. 2024-04-12T05:33:36.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 Processor -- start 2024-04-12T05:33:36.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 -- start start 2024-04-12T05:33:36.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57640f88a0 0x7f576410e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:36.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57640f9200 0x7f576410e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:36.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576410eb70 con 0x7f57640f9200 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.323+0000 7f576b851700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576410ecb0 con 0x7f57640f88a0 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f5768dec700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57640f9200 0x7f576410e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f5768dec700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57640f9200 0x7f576410e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34228/0 (socket says 172.21.15.113:34228) 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f5768dec700 1 -- 172.21.15.113:0/1388336849 learned_addr learned my addr 172.21.15.113:0/1388336849 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f57695ed700 1 --2- 172.21.15.113:0/1388336849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57640f88a0 0x7f576410e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f5768dec700 1 -- 172.21.15.113:0/1388336849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57640f88a0 msgr2=0x7f576410e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f5768dec700 1 --2- 172.21.15.113:0/1388336849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57640f88a0 0x7f576410e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f5768dec700 1 -- 172.21.15.113:0/1388336849 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5754005040 con 0x7f57640f9200 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f5768dec700 1 --2- 172.21.15.113:0/1388336849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57640f9200 0x7f576410e570 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f5754009ee0 tx=0x7f5754009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f575a7fc700 1 -- 172.21.15.113:0/1388336849 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f575400e030 con 0x7f57640f9200 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f575a7fc700 1 -- 172.21.15.113:0/1388336849 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5754007e90 con 0x7f57640f9200 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f576b851700 1 -- 172.21.15.113:0/1388336849 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57641082e0 con 0x7f57640f9200 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f575a7fc700 1 -- 172.21.15.113:0/1388336849 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5754018070 con 0x7f57640f9200 2024-04-12T05:33:36.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.324+0000 7f576b851700 1 -- 172.21.15.113:0/1388336849 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57641087c0 con 0x7f57640f9200 2024-04-12T05:33:36.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.325+0000 7f576b851700 1 -- 172.21.15.113:0/1388336849 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576405f590 con 0x7f57640f9200 2024-04-12T05:33:36.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.326+0000 7f575a7fc700 1 -- 172.21.15.113:0/1388336849 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f575401c900 con 0x7f57640f9200 2024-04-12T05:33:36.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.326+0000 7f575a7fc700 1 --2- 172.21.15.113:0/1388336849 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f575006caa0 0x7f575006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:36.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.326+0000 7f575a7fc700 1 -- 172.21.15.113:0/1388336849 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f575408d5a0 con 0x7f57640f9200 2024-04-12T05:33:36.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.326+0000 7f57695ed700 1 --2- 172.21.15.113:0/1388336849 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f575006caa0 0x7f575006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:36.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.327+0000 7f57695ed700 1 --2- 172.21.15.113:0/1388336849 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f575006caa0 0x7f575006ef60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f5760004be0 tx=0x7f576000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:36.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.330+0000 7f575a7fc700 1 -- 172.21.15.113:0/1388336849 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f575405b050 con 0x7f57640f9200 2024-04-12T05:33:36.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.528+0000 7f576b851700 1 -- 172.21.15.113:0/1388336849 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5764109810 con 0x7f575006caa0 2024-04-12T05:33:36.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.538+0000 7f575a7fc700 1 -- 172.21.15.113:0/1388336849 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5764109810 con 0x7f575006caa0 2024-04-12T05:33:36.538 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:33:36.539 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (21m) 14m ago 32m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:33:36.539 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (18m) 14m ago 32m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:33:36.539 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (18m) 14m ago 32m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:33:36.539 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (21m) 14m ago 32m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:33:36.539 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (14m) 14m ago 27m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:33:36.539 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (14m) 14m ago 27m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:33:36.539 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (14m) 14m ago 27m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:33:36.539 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (27m) 14m ago 27m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:33:36.539 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (22m) 14m ago 33m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:33:36.539 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (22m) 14m ago 31m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:33:36.540 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (19m) 14m ago 34m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:33:36.540 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (19m) 14m ago 31m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:33:36.540 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (22m) 14m ago 32m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:33:36.540 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (21m) 14m ago 31m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:33:36.540 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (18m) 14m ago 31m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:33:36.540 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (17m) 14m ago 30m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:33:36.540 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (17m) 14m ago 30m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:33:36.540 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (16m) 14m ago 29m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:33:36.540 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (16m) 14m ago 29m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:33:36.540 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (15m) 14m ago 28m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:33:36.540 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (21m) 14m ago 31m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:33:36.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.540+0000 7f576b851700 1 -- 172.21.15.113:0/1388336849 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f575006caa0 msgr2=0x7f575006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.540+0000 7f576b851700 1 --2- 172.21.15.113:0/1388336849 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f575006caa0 0x7f575006ef60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f5760004be0 tx=0x7f576000b040).stop 2024-04-12T05:33:36.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.540+0000 7f576b851700 1 -- 172.21.15.113:0/1388336849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57640f9200 msgr2=0x7f576410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.540+0000 7f576b851700 1 --2- 172.21.15.113:0/1388336849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57640f9200 0x7f576410e570 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f5754009ee0 tx=0x7f5754009cb0).stop 2024-04-12T05:33:36.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.540+0000 7f576b851700 1 -- 172.21.15.113:0/1388336849 shutdown_connections 2024-04-12T05:33:36.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.540+0000 7f576b851700 1 --2- 172.21.15.113:0/1388336849 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f575006caa0 0x7f575006ef60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.540+0000 7f576b851700 1 --2- 172.21.15.113:0/1388336849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57640f88a0 0x7f576410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.540+0000 7f576b851700 1 --2- 172.21.15.113:0/1388336849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57640f9200 0x7f576410e570 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.540+0000 7f576b851700 1 -- 172.21.15.113:0/1388336849 >> 172.21.15.113:0/1388336849 conn(0x7f5764070590 msgr2=0x7f57640fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:36.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.540+0000 7f576b851700 1 -- 172.21.15.113:0/1388336849 shutdown_connections 2024-04-12T05:33:36.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.540+0000 7f576b851700 1 -- 172.21.15.113:0/1388336849 wait complete. 2024-04-12T05:33:36.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.637+0000 7f09027bc700 1 -- 172.21.15.113:0/4065551337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08fc0fc7f0 msgr2=0x7f08fc0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.637+0000 7f09027bc700 1 --2- 172.21.15.113:0/4065551337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08fc0fc7f0 0x7f08fc0fcbd0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f08ec0047f0 tx=0x7f08ec009cb0).stop 2024-04-12T05:33:36.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.637+0000 7f09027bc700 1 -- 172.21.15.113:0/4065551337 shutdown_connections 2024-04-12T05:33:36.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.637+0000 7f09027bc700 1 --2- 172.21.15.113:0/4065551337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08fc0fd150 0x7f08fc0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.637+0000 7f09027bc700 1 --2- 172.21.15.113:0/4065551337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08fc0fc7f0 0x7f08fc0fcbd0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.637+0000 7f09027bc700 1 -- 172.21.15.113:0/4065551337 >> 172.21.15.113:0/4065551337 conn(0x7f08fc070590 msgr2=0x7f08fc0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:36.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.637+0000 7f09027bc700 1 -- 172.21.15.113:0/4065551337 shutdown_connections 2024-04-12T05:33:36.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.637+0000 7f09027bc700 1 -- 172.21.15.113:0/4065551337 wait complete. 2024-04-12T05:33:36.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.637+0000 7f09027bc700 1 Processor -- start 2024-04-12T05:33:36.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f09027bc700 1 -- start start 2024-04-12T05:33:36.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f09027bc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08fc0fc7f0 0x7f08fc109120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f09027bc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08fc0fd150 0x7f08fc109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f09027bc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc109b00 con 0x7f08fc0fd150 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f09027bc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc109c40 con 0x7f08fc0fc7f0 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f08fb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08fc0fd150 0x7f08fc109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f08fb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08fc0fd150 0x7f08fc109610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34246/0 (socket says 172.21.15.113:34246) 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f08fb7fe700 1 -- 172.21.15.113:0/774599887 learned_addr learned my addr 172.21.15.113:0/774599887 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f08fbfff700 1 --2- 172.21.15.113:0/774599887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08fc0fc7f0 0x7f08fc109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f08fb7fe700 1 -- 172.21.15.113:0/774599887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08fc0fc7f0 msgr2=0x7f08fc109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f08fb7fe700 1 --2- 172.21.15.113:0/774599887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08fc0fc7f0 0x7f08fc109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f08fb7fe700 1 -- 172.21.15.113:0/774599887 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec005040 con 0x7f08fc0fd150 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.638+0000 7f08fb7fe700 1 --2- 172.21.15.113:0/774599887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08fc0fd150 0x7f08fc109610 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f08e400c790 tx=0x7f08e4007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.639+0000 7f08f97fa700 1 -- 172.21.15.113:0/774599887 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08e40106f0 con 0x7f08fc0fd150 2024-04-12T05:33:36.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.639+0000 7f08f97fa700 1 -- 172.21.15.113:0/774599887 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08e4010d00 con 0x7f08fc0fd150 2024-04-12T05:33:36.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.639+0000 7f08f97fa700 1 -- 172.21.15.113:0/774599887 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08e4012d50 con 0x7f08fc0fd150 2024-04-12T05:33:36.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.639+0000 7f09027bc700 1 -- 172.21.15.113:0/774599887 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08fc109ef0 con 0x7f08fc0fd150 2024-04-12T05:33:36.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.639+0000 7f09027bc700 1 -- 172.21.15.113:0/774599887 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08fc1124e0 con 0x7f08fc0fd150 2024-04-12T05:33:36.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.640+0000 7f09027bc700 1 -- 172.21.15.113:0/774599887 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08fc05f590 con 0x7f08fc0fd150 2024-04-12T05:33:36.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.640+0000 7f08f97fa700 1 -- 172.21.15.113:0/774599887 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f08e4012570 con 0x7f08fc0fd150 2024-04-12T05:33:36.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.640+0000 7f08f97fa700 1 --2- 172.21.15.113:0/774599887 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08e806caa0 0x7f08e806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:36.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.640+0000 7f08f97fa700 1 -- 172.21.15.113:0/774599887 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f08e408b760 con 0x7f08fc0fd150 2024-04-12T05:33:36.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.641+0000 7f08fbfff700 1 --2- 172.21.15.113:0/774599887 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08e806caa0 0x7f08e806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:36.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.641+0000 7f08fbfff700 1 --2- 172.21.15.113:0/774599887 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08e806caa0 0x7f08e806ef60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f08ec000c00 tx=0x7f08ec00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:36.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.645+0000 7f08f97fa700 1 -- 172.21.15.113:0/774599887 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f08e4055780 con 0x7f08fc0fd150 2024-04-12T05:33:36.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.909+0000 7f09027bc700 1 -- 172.21.15.113:0/774599887 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f08fc060cf0 con 0x7f08fc0fd150 2024-04-12T05:33:36.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.910+0000 7f08f97fa700 1 -- 172.21.15.113:0/774599887 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f08e4058da0 con 0x7f08fc0fd150 2024-04-12T05:33:36.910 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:33:36.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:33:36.912 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:33:36.912 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:33:36.912 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:33:36.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.912+0000 7f09027bc700 1 -- 172.21.15.113:0/774599887 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08e806caa0 msgr2=0x7f08e806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.912+0000 7f09027bc700 1 --2- 172.21.15.113:0/774599887 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08e806caa0 0x7f08e806ef60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f08ec000c00 tx=0x7f08ec00b040).stop 2024-04-12T05:33:36.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.912+0000 7f09027bc700 1 -- 172.21.15.113:0/774599887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08fc0fd150 msgr2=0x7f08fc109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:36.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.912+0000 7f09027bc700 1 --2- 172.21.15.113:0/774599887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08fc0fd150 0x7f08fc109610 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f08e400c790 tx=0x7f08e4007f50).stop 2024-04-12T05:33:36.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.912+0000 7f09027bc700 1 -- 172.21.15.113:0/774599887 shutdown_connections 2024-04-12T05:33:36.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.912+0000 7f09027bc700 1 --2- 172.21.15.113:0/774599887 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08e806caa0 0x7f08e806ef60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.912+0000 7f09027bc700 1 --2- 172.21.15.113:0/774599887 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08fc0fc7f0 0x7f08fc109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.912+0000 7f09027bc700 1 --2- 172.21.15.113:0/774599887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08fc0fd150 0x7f08fc109610 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:36.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.912+0000 7f09027bc700 1 -- 172.21.15.113:0/774599887 >> 172.21.15.113:0/774599887 conn(0x7f08fc070590 msgr2=0x7f08fc0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:36.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.912+0000 7f09027bc700 1 -- 172.21.15.113:0/774599887 shutdown_connections 2024-04-12T05:33:36.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:36.912+0000 7f09027bc700 1 -- 172.21.15.113:0/774599887 wait complete. 2024-04-12T05:33:37.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.010+0000 7fb28eed0700 1 -- 172.21.15.113:0/1438443514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880f9230 msgr2=0x7fb2881019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.010+0000 7fb28eed0700 1 --2- 172.21.15.113:0/1438443514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880f9230 0x7fb2881019d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fb274004820 tx=0x7fb274009cb0).stop 2024-04-12T05:33:37.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.010+0000 7fb28eed0700 1 -- 172.21.15.113:0/1438443514 shutdown_connections 2024-04-12T05:33:37.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.010+0000 7fb28eed0700 1 --2- 172.21.15.113:0/1438443514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880f9230 0x7fb2881019d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.010+0000 7fb28eed0700 1 --2- 172.21.15.113:0/1438443514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880f8960 0x7fb2880f8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.010+0000 7fb28eed0700 1 -- 172.21.15.113:0/1438443514 >> 172.21.15.113:0/1438443514 conn(0x7fb288070590 msgr2=0x7fb2880f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:37.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.011+0000 7fb28eed0700 1 -- 172.21.15.113:0/1438443514 shutdown_connections 2024-04-12T05:33:37.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.011+0000 7fb28eed0700 1 -- 172.21.15.113:0/1438443514 wait complete. 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.011+0000 7fb28eed0700 1 Processor -- start 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.011+0000 7fb28eed0700 1 -- start start 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.011+0000 7fb28eed0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880f8960 0x7fb28810e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.011+0000 7fb28eed0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880f9230 0x7fb28810e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.011+0000 7fb28eed0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28810eb00 con 0x7fb2880f9230 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.011+0000 7fb28eed0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28810ec40 con 0x7fb2880f8960 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.011+0000 7fb287fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880f9230 0x7fb28810e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.011+0000 7fb287fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880f9230 0x7fb28810e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34262/0 (socket says 172.21.15.113:34262) 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.011+0000 7fb287fff700 1 -- 172.21.15.113:0/1263815220 learned_addr learned my addr 172.21.15.113:0/1263815220 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.012+0000 7fb28cc6c700 1 --2- 172.21.15.113:0/1263815220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880f8960 0x7fb28810e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.012+0000 7fb287fff700 1 -- 172.21.15.113:0/1263815220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880f8960 msgr2=0x7fb28810e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.012+0000 7fb287fff700 1 --2- 172.21.15.113:0/1263815220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880f8960 0x7fb28810e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.012+0000 7fb287fff700 1 -- 172.21.15.113:0/1263815220 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274005040 con 0x7fb2880f9230 2024-04-12T05:33:37.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.012+0000 7fb287fff700 1 --2- 172.21.15.113:0/1263815220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880f9230 0x7fb28810e500 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7fb274005010 tx=0x7fb274009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:37.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.012+0000 7fb285ffb700 1 -- 172.21.15.113:0/1263815220 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb27400e030 con 0x7fb2880f9230 2024-04-12T05:33:37.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.012+0000 7fb285ffb700 1 -- 172.21.15.113:0/1263815220 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb274007e90 con 0x7fb2880f9230 2024-04-12T05:33:37.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.012+0000 7fb285ffb700 1 -- 172.21.15.113:0/1263815220 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb274018070 con 0x7fb2880f9230 2024-04-12T05:33:37.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.012+0000 7fb28eed0700 1 -- 172.21.15.113:0/1263815220 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb288108270 con 0x7fb2880f9230 2024-04-12T05:33:37.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.012+0000 7fb28eed0700 1 -- 172.21.15.113:0/1263815220 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb288108750 con 0x7fb2880f9230 2024-04-12T05:33:37.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.013+0000 7fb28eed0700 1 -- 172.21.15.113:0/1263815220 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28805f590 con 0x7fb2880f9230 2024-04-12T05:33:37.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.014+0000 7fb285ffb700 1 -- 172.21.15.113:0/1263815220 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb27401d420 con 0x7fb2880f9230 2024-04-12T05:33:37.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.014+0000 7fb285ffb700 1 --2- 172.21.15.113:0/1263815220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb27806caa0 0x7fb27806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:37.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.014+0000 7fb285ffb700 1 -- 172.21.15.113:0/1263815220 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb27408d0d0 con 0x7fb2880f9230 2024-04-12T05:33:37.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.014+0000 7fb28cc6c700 1 --2- 172.21.15.113:0/1263815220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb27806caa0 0x7fb27806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:37.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.014+0000 7fb28cc6c700 1 --2- 172.21.15.113:0/1263815220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb27806caa0 0x7fb27806ef60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fb27c004be0 tx=0x7fb27c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:37.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.018+0000 7fb285ffb700 1 -- 172.21.15.113:0/1263815220 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb27405ab80 con 0x7fb2880f9230 2024-04-12T05:33:37.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.250+0000 7fb28eed0700 1 -- 172.21.15.113:0/1263815220 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb288047540 con 0x7fb2880f9230 2024-04-12T05:33:37.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.251+0000 7fb285ffb700 1 -- 172.21.15.113:0/1263815220 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7fb27401d6d0 con 0x7fb2880f9230 2024-04-12T05:33:37.251 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:33:37.251 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:33:37.251 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:33:37.251 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:33:37.251 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:33:37.251 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:33:37.251 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:33:37.252 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:33:37.253 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:33:37.254 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:33:37.254 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:33:37.254 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:33:37.254 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:33:37.254 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:33:37.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.253+0000 7fb28eed0700 1 -- 172.21.15.113:0/1263815220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb27806caa0 msgr2=0x7fb27806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.253+0000 7fb28eed0700 1 --2- 172.21.15.113:0/1263815220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb27806caa0 0x7fb27806ef60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fb27c004be0 tx=0x7fb27c00b040).stop 2024-04-12T05:33:37.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.253+0000 7fb28eed0700 1 -- 172.21.15.113:0/1263815220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880f9230 msgr2=0x7fb28810e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.253+0000 7fb28eed0700 1 --2- 172.21.15.113:0/1263815220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880f9230 0x7fb28810e500 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7fb274005010 tx=0x7fb274009cb0).stop 2024-04-12T05:33:37.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.253+0000 7fb28eed0700 1 -- 172.21.15.113:0/1263815220 shutdown_connections 2024-04-12T05:33:37.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.253+0000 7fb28eed0700 1 --2- 172.21.15.113:0/1263815220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb27806caa0 0x7fb27806ef60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.253+0000 7fb28eed0700 1 --2- 172.21.15.113:0/1263815220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb2880f8960 0x7fb28810e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.253+0000 7fb28eed0700 1 --2- 172.21.15.113:0/1263815220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2880f9230 0x7fb28810e500 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.253+0000 7fb28eed0700 1 -- 172.21.15.113:0/1263815220 >> 172.21.15.113:0/1263815220 conn(0x7fb288070590 msgr2=0x7fb2880fc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:37.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.253+0000 7fb28eed0700 1 -- 172.21.15.113:0/1263815220 shutdown_connections 2024-04-12T05:33:37.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.253+0000 7fb28eed0700 1 -- 172.21.15.113:0/1263815220 wait complete. 2024-04-12T05:33:37.256 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:33:37.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.352+0000 7f086c313700 1 -- 172.21.15.113:0/590561766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08640fc7f0 msgr2=0x7f08640fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.352+0000 7f086c313700 1 --2- 172.21.15.113:0/590561766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08640fc7f0 0x7f08640fcbd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f08580047f0 tx=0x7f0858009cb0).stop 2024-04-12T05:33:37.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.352+0000 7f086c313700 1 -- 172.21.15.113:0/590561766 shutdown_connections 2024-04-12T05:33:37.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.352+0000 7f086c313700 1 --2- 172.21.15.113:0/590561766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08640fd150 0x7f08640f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.352+0000 7f086c313700 1 --2- 172.21.15.113:0/590561766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08640fc7f0 0x7f08640fcbd0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.352+0000 7f086c313700 1 -- 172.21.15.113:0/590561766 >> 172.21.15.113:0/590561766 conn(0x7f08640f23b0 msgr2=0x7f08640f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:37.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.352+0000 7f086c313700 1 -- 172.21.15.113:0/590561766 shutdown_connections 2024-04-12T05:33:37.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.352+0000 7f086c313700 1 -- 172.21.15.113:0/590561766 wait complete. 2024-04-12T05:33:37.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086c313700 1 Processor -- start 2024-04-12T05:33:37.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086c313700 1 -- start start 2024-04-12T05:33:37.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086c313700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08640fc7f0 0x7f08641090d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:37.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086c313700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08640fd150 0x7f08641095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:37.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086c313700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0864109b00 con 0x7f08640fc7f0 2024-04-12T05:33:37.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086c313700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0864109c40 con 0x7f08640fd150 2024-04-12T05:33:37.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086a0af700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08640fc7f0 0x7f08641090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:37.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086a0af700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08640fc7f0 0x7f08641090d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34290/0 (socket says 172.21.15.113:34290) 2024-04-12T05:33:37.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086a0af700 1 -- 172.21.15.113:0/1543809891 learned_addr learned my addr 172.21.15.113:0/1543809891 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:37.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f08698ae700 1 --2- 172.21.15.113:0/1543809891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08640fd150 0x7f08641095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:37.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086a0af700 1 -- 172.21.15.113:0/1543809891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08640fd150 msgr2=0x7f08641095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086a0af700 1 --2- 172.21.15.113:0/1543809891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08640fd150 0x7f08641095c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086a0af700 1 -- 172.21.15.113:0/1543809891 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0858005040 con 0x7f08640fc7f0 2024-04-12T05:33:37.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.353+0000 7f086a0af700 1 --2- 172.21.15.113:0/1543809891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08640fc7f0 0x7f08641090d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f0858005010 tx=0x7f0858009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:37.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.354+0000 7f08577fe700 1 -- 172.21.15.113:0/1543809891 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f085800e030 con 0x7f08640fc7f0 2024-04-12T05:33:37.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.354+0000 7f08577fe700 1 -- 172.21.15.113:0/1543809891 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0858007e90 con 0x7f08640fc7f0 2024-04-12T05:33:37.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.354+0000 7f086c313700 1 -- 172.21.15.113:0/1543809891 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0864109e90 con 0x7f08640fc7f0 2024-04-12T05:33:37.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.354+0000 7f08577fe700 1 -- 172.21.15.113:0/1543809891 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0858018070 con 0x7f08640fc7f0 2024-04-12T05:33:37.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.354+0000 7f086c313700 1 -- 172.21.15.113:0/1543809891 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0864112430 con 0x7f08640fc7f0 2024-04-12T05:33:37.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.355+0000 7f086c313700 1 -- 172.21.15.113:0/1543809891 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0864060390 con 0x7f08640fc7f0 2024-04-12T05:33:37.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.355+0000 7f08577fe700 1 -- 172.21.15.113:0/1543809891 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f085801d6e0 con 0x7f08640fc7f0 2024-04-12T05:33:37.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.355+0000 7f08577fe700 1 --2- 172.21.15.113:0/1543809891 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f085006ca30 0x7f085006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:37.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.355+0000 7f08577fe700 1 -- 172.21.15.113:0/1543809891 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f085808d170 con 0x7f08640fc7f0 2024-04-12T05:33:37.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.355+0000 7f08698ae700 1 --2- 172.21.15.113:0/1543809891 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f085006ca30 0x7f085006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:37.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.356+0000 7f08698ae700 1 --2- 172.21.15.113:0/1543809891 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f085006ca30 0x7f085006eef0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f08640614f0 tx=0x7f0860009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:37.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.360+0000 7f08577fe700 1 -- 172.21.15.113:0/1543809891 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0858057190 con 0x7f08640fc7f0 2024-04-12T05:33:37.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:37 smithi113 ceph-mon[103371]: from='client.35176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:37.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:37 smithi113 ceph-mon[103371]: pgmap v643: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:37.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:37.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:37 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/774599887' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:33:37.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:37 smithi192 ceph-mon[85304]: from='client.35176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:37.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:37 smithi192 ceph-mon[85304]: pgmap v643: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:37.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:37.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:37 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/774599887' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:33:37.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.565+0000 7f086c313700 1 -- 172.21.15.113:0/1543809891 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0864061340 con 0x7f085006ca30 2024-04-12T05:33:37.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.566+0000 7f08577fe700 1 -- 172.21.15.113:0/1543809891 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0864061340 con 0x7f085006ca30 2024-04-12T05:33:37.566 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:33:37.566 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:33:37.566 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:33:37.567 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:33:37.567 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:33:37.567 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:33:37.567 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:33:37.567 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:33:37.567 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:33:37.567 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:33:37.567 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:33:37.567 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:33:37.567 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:33:37.567 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:33:37.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.568+0000 7f086c313700 1 -- 172.21.15.113:0/1543809891 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f085006ca30 msgr2=0x7f085006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.568+0000 7f086c313700 1 --2- 172.21.15.113:0/1543809891 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f085006ca30 0x7f085006eef0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f08640614f0 tx=0x7f0860009da0).stop 2024-04-12T05:33:37.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.568+0000 7f086c313700 1 -- 172.21.15.113:0/1543809891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08640fc7f0 msgr2=0x7f08641090d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.568+0000 7f086c313700 1 --2- 172.21.15.113:0/1543809891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08640fc7f0 0x7f08641090d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f0858005010 tx=0x7f0858009cb0).stop 2024-04-12T05:33:37.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.568+0000 7f086c313700 1 -- 172.21.15.113:0/1543809891 shutdown_connections 2024-04-12T05:33:37.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.568+0000 7f086c313700 1 --2- 172.21.15.113:0/1543809891 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f085006ca30 0x7f085006eef0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.568+0000 7f086c313700 1 --2- 172.21.15.113:0/1543809891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08640fc7f0 0x7f08641090d0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.568+0000 7f086c313700 1 --2- 172.21.15.113:0/1543809891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08640fd150 0x7f08641095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.568+0000 7f086c313700 1 -- 172.21.15.113:0/1543809891 >> 172.21.15.113:0/1543809891 conn(0x7f08640f23b0 msgr2=0x7f08640fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:37.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.568+0000 7f086c313700 1 -- 172.21.15.113:0/1543809891 shutdown_connections 2024-04-12T05:33:37.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.568+0000 7f086c313700 1 -- 172.21.15.113:0/1543809891 wait complete. 2024-04-12T05:33:37.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.666+0000 7f43624a8700 1 -- 172.21.15.113:0/1557721419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f435c0fd910 msgr2=0x7f435c0fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.666+0000 7f43624a8700 1 --2- 172.21.15.113:0/1557721419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f435c0fd910 0x7f435c0fdcf0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f434c0047f0 tx=0x7f434c009cb0).stop 2024-04-12T05:33:37.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.666+0000 7f43624a8700 1 -- 172.21.15.113:0/1557721419 shutdown_connections 2024-04-12T05:33:37.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.666+0000 7f43624a8700 1 --2- 172.21.15.113:0/1557721419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0fe270 0x7f435c0f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.666+0000 7f43624a8700 1 --2- 172.21.15.113:0/1557721419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f435c0fd910 0x7f435c0fdcf0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.666+0000 7f43624a8700 1 -- 172.21.15.113:0/1557721419 >> 172.21.15.113:0/1557721419 conn(0x7f435c0f3530 msgr2=0x7f435c0f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:37.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.666+0000 7f43624a8700 1 -- 172.21.15.113:0/1557721419 shutdown_connections 2024-04-12T05:33:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.666+0000 7f43624a8700 1 -- 172.21.15.113:0/1557721419 wait complete. 2024-04-12T05:33:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.667+0000 7f43624a8700 1 Processor -- start 2024-04-12T05:33:37.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.667+0000 7f43624a8700 1 -- start start 2024-04-12T05:33:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.667+0000 7f43624a8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f435c0fd910 0x7f435c06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.667+0000 7f43624a8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0fe270 0x7f435c06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:37.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.667+0000 7f43624a8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c06bab0 con 0x7f435c0fd910 2024-04-12T05:33:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.667+0000 7f43624a8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c06bbf0 con 0x7f435c0fe270 2024-04-12T05:33:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f435bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f435c0fd910 0x7f435c06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f435bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f435c0fd910 0x7f435c06b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34302/0 (socket says 172.21.15.113:34302) 2024-04-12T05:33:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f435bfff700 1 -- 172.21.15.113:0/2442861721 learned_addr learned my addr 172.21.15.113:0/2442861721 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:33:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f435b7fe700 1 --2- 172.21.15.113:0/2442861721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0fe270 0x7f435c06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f435bfff700 1 -- 172.21.15.113:0/2442861721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0fe270 msgr2=0x7f435c06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f435bfff700 1 --2- 172.21.15.113:0/2442861721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0fe270 0x7f435c06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f435bfff700 1 -- 172.21.15.113:0/2442861721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c005040 con 0x7f435c0fd910 2024-04-12T05:33:37.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f435bfff700 1 --2- 172.21.15.113:0/2442861721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f435c0fd910 0x7f435c06b0d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f434c009380 tx=0x7f434c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f43597fa700 1 -- 172.21.15.113:0/2442861721 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f434c018070 con 0x7f435c0fd910 2024-04-12T05:33:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f43597fa700 1 -- 172.21.15.113:0/2442861721 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f434c007e90 con 0x7f435c0fd910 2024-04-12T05:33:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f43624a8700 1 -- 172.21.15.113:0/2442861721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c06be40 con 0x7f435c0fd910 2024-04-12T05:33:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f43597fa700 1 -- 172.21.15.113:0/2442861721 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f434c01c380 con 0x7f435c0fd910 2024-04-12T05:33:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.668+0000 7f43624a8700 1 -- 172.21.15.113:0/2442861721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c114740 con 0x7f435c0fd910 2024-04-12T05:33:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.669+0000 7f43624a8700 1 -- 172.21.15.113:0/2442861721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435c05f570 con 0x7f435c0fd910 2024-04-12T05:33:37.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.670+0000 7f43597fa700 1 -- 172.21.15.113:0/2442861721 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f434c0123c0 con 0x7f435c0fd910 2024-04-12T05:33:37.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.670+0000 7f43597fa700 1 --2- 172.21.15.113:0/2442861721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f434406cab0 0x7f434406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:33:37.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.670+0000 7f43597fa700 1 -- 172.21.15.113:0/2442861721 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f434c08cc60 con 0x7f435c0fd910 2024-04-12T05:33:37.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.670+0000 7f435b7fe700 1 --2- 172.21.15.113:0/2442861721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f434406cab0 0x7f434406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:33:37.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.670+0000 7f435b7fe700 1 --2- 172.21.15.113:0/2442861721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f434406cab0 0x7f434406ef70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f435c06c870 tx=0x7f435000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:33:37.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.674+0000 7f43597fa700 1 -- 172.21.15.113:0/2442861721 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f434c056c80 con 0x7f435c0fd910 2024-04-12T05:33:37.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.938+0000 7f43624a8700 1 -- 172.21.15.113:0/2442861721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f435c047540 con 0x7f435c0fd910 2024-04-12T05:33:37.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.939+0000 7f43597fa700 1 -- 172.21.15.113:0/2442861721 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f434c05a2a0 con 0x7f435c0fd910 2024-04-12T05:33:37.939 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:33:37.939 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:33:37.939 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:33:37.939 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:33:37.939 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:33:37.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.941+0000 7f43624a8700 1 -- 172.21.15.113:0/2442861721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f434406cab0 msgr2=0x7f434406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.941+0000 7f43624a8700 1 --2- 172.21.15.113:0/2442861721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f434406cab0 0x7f434406ef70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f435c06c870 tx=0x7f435000b040).stop 2024-04-12T05:33:37.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.941+0000 7f43624a8700 1 -- 172.21.15.113:0/2442861721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f435c0fd910 msgr2=0x7f435c06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:33:37.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.941+0000 7f43624a8700 1 --2- 172.21.15.113:0/2442861721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f435c0fd910 0x7f435c06b0d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f434c009380 tx=0x7f434c009cb0).stop 2024-04-12T05:33:37.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.941+0000 7f43624a8700 1 -- 172.21.15.113:0/2442861721 shutdown_connections 2024-04-12T05:33:37.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.941+0000 7f43624a8700 1 --2- 172.21.15.113:0/2442861721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f434406cab0 0x7f434406ef70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.941+0000 7f43624a8700 1 --2- 172.21.15.113:0/2442861721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f435c0fd910 0x7f435c06b0d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.941+0000 7f43624a8700 1 --2- 172.21.15.113:0/2442861721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435c0fe270 0x7f435c06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:33:37.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.941+0000 7f43624a8700 1 -- 172.21.15.113:0/2442861721 >> 172.21.15.113:0/2442861721 conn(0x7f435c0f3530 msgr2=0x7f435c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:33:37.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.941+0000 7f43624a8700 1 -- 172.21.15.113:0/2442861721 shutdown_connections 2024-04-12T05:33:37.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:33:37.941+0000 7f43624a8700 1 -- 172.21.15.113:0/2442861721 wait complete. 2024-04-12T05:33:38.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:38 smithi113 ceph-mon[103371]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:38.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:38 smithi113 ceph-mon[103371]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:38.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:38 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1263815220' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:33:38.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:38 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2442861721' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:33:38.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:38 smithi192 ceph-mon[85304]: from='client.35180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:38.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:38 smithi192 ceph-mon[85304]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:38.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:38 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1263815220' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:33:38.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:38 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2442861721' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:33:39.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:39 smithi113 ceph-mon[103371]: from='client.35196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:39.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:39 smithi113 ceph-mon[103371]: pgmap v644: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:39.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:39 smithi192 ceph-mon[85304]: from='client.35196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:33:39.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:39 smithi192 ceph-mon[85304]: pgmap v644: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:41.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:41 smithi113 ceph-mon[103371]: pgmap v645: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:41.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:41 smithi192 ceph-mon[85304]: pgmap v645: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:43.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:43 smithi113 ceph-mon[103371]: pgmap v646: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:43.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:43 smithi192 ceph-mon[85304]: pgmap v646: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:45.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:45 smithi113 ceph-mon[103371]: pgmap v647: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:45.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:45 smithi192 ceph-mon[85304]: pgmap v647: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:47.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:47 smithi113 ceph-mon[103371]: pgmap v648: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:47.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:47 smithi192 ceph-mon[85304]: pgmap v648: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:49.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:49 smithi113 ceph-mon[103371]: pgmap v649: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:49.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:49 smithi192 ceph-mon[85304]: pgmap v649: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:51.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:51 smithi113 ceph-mon[103371]: pgmap v650: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:51.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:51 smithi192 ceph-mon[85304]: pgmap v650: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:52.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:52.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:33:53.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:53 smithi113 ceph-mon[103371]: pgmap v651: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:53.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:53 smithi192 ceph-mon[85304]: pgmap v651: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:55.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:55 smithi113 ceph-mon[103371]: pgmap v652: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:55.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:55 smithi192 ceph-mon[85304]: pgmap v652: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:57.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:57 smithi113 ceph-mon[103371]: pgmap v653: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:57.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:57 smithi192 ceph-mon[85304]: pgmap v653: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:59.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:33:59 smithi113 ceph-mon[103371]: pgmap v654: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:33:59.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:33:59 smithi192 ceph-mon[85304]: pgmap v654: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:01.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:01 smithi192 ceph-mon[85304]: pgmap v655: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:01.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:01 smithi113 ceph-mon[103371]: pgmap v655: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:03.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:03 smithi192 ceph-mon[85304]: pgmap v656: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:03.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:03 smithi113 ceph-mon[103371]: pgmap v656: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:05.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:05 smithi192 ceph-mon[85304]: pgmap v657: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:05.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:05 smithi113 ceph-mon[103371]: pgmap v657: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:07.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:07 smithi192 ceph-mon[85304]: pgmap v658: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:07.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:07.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:07 smithi113 ceph-mon[103371]: pgmap v658: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:07.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:08.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.053+0000 7f0f92207700 1 -- 172.21.15.113:0/3955345680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0f9200 msgr2=0x7f0f8c101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.053+0000 7f0f92207700 1 --2- 172.21.15.113:0/3955345680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0f9200 0x7f0f8c101910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f0f800047f0 tx=0x7f0f80009cb0).stop 2024-04-12T05:34:08.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.053+0000 7f0f92207700 1 -- 172.21.15.113:0/3955345680 shutdown_connections 2024-04-12T05:34:08.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.053+0000 7f0f92207700 1 --2- 172.21.15.113:0/3955345680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0f9200 0x7f0f8c101910 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.053+0000 7f0f92207700 1 --2- 172.21.15.113:0/3955345680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0f88a0 0x7f0f8c0f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:34:08.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.053+0000 7f0f92207700 1 -- 172.21.15.113:0/3955345680 >> 172.21.15.113:0/3955345680 conn(0x7f0f8c070590 msgr2=0x7f0f8c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:08.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.053+0000 7f0f92207700 1 -- 172.21.15.113:0/3955345680 shutdown_connections 2024-04-12T05:34:08.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.053+0000 7f0f92207700 1 -- 172.21.15.113:0/3955345680 wait complete. 2024-04-12T05:34:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.054+0000 7f0f92207700 1 Processor -- start 2024-04-12T05:34:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.054+0000 7f0f92207700 1 -- start start 2024-04-12T05:34:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.054+0000 7f0f92207700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0f88a0 0x7f0f8c10e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.054+0000 7f0f92207700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0f9200 0x7f0f8c10e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.054+0000 7f0f92207700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c10ec70 con 0x7f0f8c0f9200 2024-04-12T05:34:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.054+0000 7f0f92207700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c108290 con 0x7f0f8c0f88a0 2024-04-12T05:34:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f8affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0f9200 0x7f0f8c10e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f8affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0f9200 0x7f0f8c10e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59154/0 (socket says 172.21.15.113:59154) 2024-04-12T05:34:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f8affd700 1 -- 172.21.15.113:0/3998333076 learned_addr learned my addr 172.21.15.113:0/3998333076 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f8b7fe700 1 --2- 172.21.15.113:0/3998333076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0f88a0 0x7f0f8c10e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:08.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f8affd700 1 -- 172.21.15.113:0/3998333076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0f88a0 msgr2=0x7f0f8c10e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f8affd700 1 --2- 172.21.15.113:0/3998333076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0f88a0 0x7f0f8c10e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f8affd700 1 -- 172.21.15.113:0/3998333076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f80005040 con 0x7f0f8c0f9200 2024-04-12T05:34:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f8affd700 1 --2- 172.21.15.113:0/3998333076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0f9200 0x7f0f8c10e5e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f0f8000b120 tx=0x7f0f80009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f88ff9700 1 -- 172.21.15.113:0/3998333076 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f8000e030 con 0x7f0f8c0f9200 2024-04-12T05:34:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f88ff9700 1 -- 172.21.15.113:0/3998333076 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f80007e90 con 0x7f0f8c0f9200 2024-04-12T05:34:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f92207700 1 -- 172.21.15.113:0/3998333076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8c1084e0 con 0x7f0f8c0f9200 2024-04-12T05:34:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f88ff9700 1 -- 172.21.15.113:0/3998333076 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f80018070 con 0x7f0f8c0f9200 2024-04-12T05:34:08.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.055+0000 7f0f92207700 1 -- 172.21.15.113:0/3998333076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8c1089c0 con 0x7f0f8c0f9200 2024-04-12T05:34:08.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.056+0000 7f0f92207700 1 -- 172.21.15.113:0/3998333076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8c05f590 con 0x7f0f8c0f9200 2024-04-12T05:34:08.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.057+0000 7f0f88ff9700 1 -- 172.21.15.113:0/3998333076 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0f8001d420 con 0x7f0f8c0f9200 2024-04-12T05:34:08.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.057+0000 7f0f88ff9700 1 --2- 172.21.15.113:0/3998333076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 0x7f0f7406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:08.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.057+0000 7f0f88ff9700 1 -- 172.21.15.113:0/3998333076 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0f8008d260 con 0x7f0f8c0f9200 2024-04-12T05:34:08.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.057+0000 7f0f8b7fe700 1 --2- 172.21.15.113:0/3998333076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 0x7f0f7406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:08.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.058+0000 7f0f8b7fe700 1 --2- 172.21.15.113:0/3998333076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 0x7f0f7406ef60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f0f7c00b910 tx=0x7f0f7c008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:08.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.062+0000 7f0f88ff9700 1 -- 172.21.15.113:0/3998333076 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0f8005ad10 con 0x7f0f8c0f9200 2024-04-12T05:34:08.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.268+0000 7f0f92207700 1 -- 172.21.15.113:0/3998333076 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f8c109770 con 0x7f0f7406caa0 2024-04-12T05:34:08.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.270+0000 7f0f88ff9700 1 -- 172.21.15.113:0/3998333076 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0f8c109770 con 0x7f0f7406caa0 2024-04-12T05:34:08.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.271+0000 7f0f92207700 1 -- 172.21.15.113:0/3998333076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 msgr2=0x7f0f7406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.271+0000 7f0f92207700 1 --2- 172.21.15.113:0/3998333076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 0x7f0f7406ef60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f0f7c00b910 tx=0x7f0f7c008040).stop 2024-04-12T05:34:08.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.272+0000 7f0f92207700 1 -- 172.21.15.113:0/3998333076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0f9200 msgr2=0x7f0f8c10e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.272+0000 7f0f92207700 1 --2- 172.21.15.113:0/3998333076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0f9200 0x7f0f8c10e5e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f0f8000b120 tx=0x7f0f80009cb0).stop 2024-04-12T05:34:08.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.272+0000 7f0f92207700 1 -- 172.21.15.113:0/3998333076 shutdown_connections 2024-04-12T05:34:08.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.272+0000 7f0f92207700 1 --2- 172.21.15.113:0/3998333076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 0x7f0f7406ef60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.272+0000 7f0f92207700 1 --2- 172.21.15.113:0/3998333076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0f88a0 0x7f0f8c10e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.272+0000 7f0f92207700 1 --2- 172.21.15.113:0/3998333076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0f9200 0x7f0f8c10e5e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.272+0000 7f0f92207700 1 -- 172.21.15.113:0/3998333076 >> 172.21.15.113:0/3998333076 conn(0x7f0f8c070590 msgr2=0x7f0f8c0fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:08.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.272+0000 7f0f92207700 1 -- 172.21.15.113:0/3998333076 shutdown_connections 2024-04-12T05:34:08.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.272+0000 7f0f92207700 1 -- 172.21.15.113:0/3998333076 wait complete. 2024-04-12T05:34:08.284 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:34:08.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 -- 172.21.15.113:0/3599227352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72800fc7f0 msgr2=0x7f72800fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 --2- 172.21.15.113:0/3599227352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72800fc7f0 0x7f72800fcbd0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f72700047f0 tx=0x7f7270009cb0).stop 2024-04-12T05:34:08.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 -- 172.21.15.113:0/3599227352 shutdown_connections 2024-04-12T05:34:08.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 --2- 172.21.15.113:0/3599227352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72800fd150 0x7f72800f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 --2- 172.21.15.113:0/3599227352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72800fc7f0 0x7f72800fcbd0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 -- 172.21.15.113:0/3599227352 >> 172.21.15.113:0/3599227352 conn(0x7f7280070590 msgr2=0x7f72800f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 -- 172.21.15.113:0/3599227352 shutdown_connections 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 -- 172.21.15.113:0/3599227352 wait complete. 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 Processor -- start 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 -- start start 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72800fc7f0 0x7f728010e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.374+0000 7f728816b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72800fd150 0x7f728010e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f728816b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728010eb20 con 0x7f72800fd150 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f728816b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728010ec60 con 0x7f72800fc7f0 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f7285706700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72800fd150 0x7f728010e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f7285706700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72800fd150 0x7f728010e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59182/0 (socket says 172.21.15.113:59182) 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f7285706700 1 -- 172.21.15.113:0/1057157459 learned_addr learned my addr 172.21.15.113:0/1057157459 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f7285706700 1 -- 172.21.15.113:0/1057157459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72800fc7f0 msgr2=0x7f728010e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f7285f07700 1 --2- 172.21.15.113:0/1057157459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72800fc7f0 0x7f728010e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:08.375 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f7285706700 1 --2- 172.21.15.113:0/1057157459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72800fc7f0 0x7f728010e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f7285706700 1 -- 172.21.15.113:0/1057157459 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7270005040 con 0x7f72800fd150 2024-04-12T05:34:08.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f7285706700 1 --2- 172.21.15.113:0/1057157459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72800fd150 0x7f728010e520 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f727c002820 tx=0x7f727c009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:08.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f7276ffd700 1 -- 172.21.15.113:0/1057157459 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f727c007b40 con 0x7f72800fd150 2024-04-12T05:34:08.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f7276ffd700 1 -- 172.21.15.113:0/1057157459 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f727c00f040 con 0x7f72800fd150 2024-04-12T05:34:08.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f7276ffd700 1 -- 172.21.15.113:0/1057157459 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f727c014910 con 0x7f72800fd150 2024-04-12T05:34:08.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f728816b700 1 -- 172.21.15.113:0/1057157459 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72801082f0 con 0x7f72800fd150 2024-04-12T05:34:08.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.375+0000 7f728816b700 1 -- 172.21.15.113:0/1057157459 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72801088b0 con 0x7f72800fd150 2024-04-12T05:34:08.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.376+0000 7f728816b700 1 -- 172.21.15.113:0/1057157459 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728005f590 con 0x7f72800fd150 2024-04-12T05:34:08.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.377+0000 7f7276ffd700 1 -- 172.21.15.113:0/1057157459 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f727c007ca0 con 0x7f72800fd150 2024-04-12T05:34:08.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.377+0000 7f7276ffd700 1 --2- 172.21.15.113:0/1057157459 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f726c06cab0 0x7f726c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:08.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.377+0000 7f7276ffd700 1 -- 172.21.15.113:0/1057157459 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f727c08b7f0 con 0x7f72800fd150 2024-04-12T05:34:08.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.377+0000 7f7285f07700 1 --2- 172.21.15.113:0/1057157459 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f726c06cab0 0x7f726c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:08.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.377+0000 7f7285f07700 1 --2- 172.21.15.113:0/1057157459 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f726c06cab0 0x7f726c06ef70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f7270000c00 tx=0x7f7270009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:08.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.381+0000 7f7276ffd700 1 -- 172.21.15.113:0/1057157459 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f727c0592a0 con 0x7f72800fd150 2024-04-12T05:34:08.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.585+0000 7f728816b700 1 -- 172.21.15.113:0/1057157459 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7280108f50 con 0x7f726c06cab0 2024-04-12T05:34:08.585 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.587+0000 7f7276ffd700 1 -- 172.21.15.113:0/1057157459 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7280108f50 con 0x7f726c06cab0 2024-04-12T05:34:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.589+0000 7f728816b700 1 -- 172.21.15.113:0/1057157459 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f726c06cab0 msgr2=0x7f726c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.589+0000 7f728816b700 1 --2- 172.21.15.113:0/1057157459 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f726c06cab0 0x7f726c06ef70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f7270000c00 tx=0x7f7270009cb0).stop 2024-04-12T05:34:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.589+0000 7f728816b700 1 -- 172.21.15.113:0/1057157459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72800fd150 msgr2=0x7f728010e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.589+0000 7f728816b700 1 --2- 172.21.15.113:0/1057157459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72800fd150 0x7f728010e520 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f727c002820 tx=0x7f727c009f20).stop 2024-04-12T05:34:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.589+0000 7f728816b700 1 -- 172.21.15.113:0/1057157459 shutdown_connections 2024-04-12T05:34:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.589+0000 7f728816b700 1 --2- 172.21.15.113:0/1057157459 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f726c06cab0 0x7f726c06ef70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.589+0000 7f728816b700 1 --2- 172.21.15.113:0/1057157459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72800fc7f0 0x7f728010e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.589+0000 7f728816b700 1 --2- 172.21.15.113:0/1057157459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72800fd150 0x7f728010e520 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.589+0000 7f728816b700 1 -- 172.21.15.113:0/1057157459 >> 172.21.15.113:0/1057157459 conn(0x7f7280070590 msgr2=0x7f7280060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.589+0000 7f728816b700 1 -- 172.21.15.113:0/1057157459 shutdown_connections 2024-04-12T05:34:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.589+0000 7f728816b700 1 -- 172.21.15.113:0/1057157459 wait complete. 2024-04-12T05:34:08.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.687+0000 7f392f82b700 1 -- 172.21.15.113:0/1405289782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280fd150 msgr2=0x7f39280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.687+0000 7f392f82b700 1 --2- 172.21.15.113:0/1405289782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280fd150 0x7f39280f6930 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7f39240047f0 tx=0x7f3924009cb0).stop 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.687+0000 7f392f82b700 1 -- 172.21.15.113:0/1405289782 shutdown_connections 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.687+0000 7f392f82b700 1 --2- 172.21.15.113:0/1405289782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280fd150 0x7f39280f6930 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.687+0000 7f392f82b700 1 --2- 172.21.15.113:0/1405289782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39280fc7f0 0x7f39280fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.687+0000 7f392f82b700 1 -- 172.21.15.113:0/1405289782 >> 172.21.15.113:0/1405289782 conn(0x7f3928070590 msgr2=0x7f39280f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.687+0000 7f392f82b700 1 -- 172.21.15.113:0/1405289782 shutdown_connections 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.687+0000 7f392f82b700 1 -- 172.21.15.113:0/1405289782 wait complete. 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392f82b700 1 Processor -- start 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392f82b700 1 -- start start 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392f82b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39280fc7f0 0x7f39281090e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392f82b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280fd150 0x7f39281095d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:08.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392f82b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3928109ac0 con 0x7f39280fd150 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392f82b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3928109c00 con 0x7f39280fc7f0 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392cdc6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280fd150 0x7f39281095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392cdc6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280fd150 0x7f39281095d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59192/0 (socket says 172.21.15.113:59192) 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392cdc6700 1 -- 172.21.15.113:0/512565781 learned_addr learned my addr 172.21.15.113:0/512565781 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392d5c7700 1 --2- 172.21.15.113:0/512565781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39280fc7f0 0x7f39281090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392cdc6700 1 -- 172.21.15.113:0/512565781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39280fc7f0 msgr2=0x7f39281090e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392cdc6700 1 --2- 172.21.15.113:0/512565781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39280fc7f0 0x7f39281090e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392cdc6700 1 -- 172.21.15.113:0/512565781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3924005040 con 0x7f39280fd150 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.688+0000 7f392cdc6700 1 --2- 172.21.15.113:0/512565781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280fd150 0x7f39281095d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f39240099f0 tx=0x7f3924009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.689+0000 7f391a7fc700 1 -- 172.21.15.113:0/512565781 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f392400e030 con 0x7f39280fd150 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.689+0000 7f391a7fc700 1 -- 172.21.15.113:0/512565781 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3924007e90 con 0x7f39280fd150 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.689+0000 7f392f82b700 1 -- 172.21.15.113:0/512565781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3928109e50 con 0x7f39280fd150 2024-04-12T05:34:08.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.689+0000 7f391a7fc700 1 -- 172.21.15.113:0/512565781 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3924018070 con 0x7f39280fd150 2024-04-12T05:34:08.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.689+0000 7f392f82b700 1 -- 172.21.15.113:0/512565781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3928112440 con 0x7f39280fd150 2024-04-12T05:34:08.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.690+0000 7f392f82b700 1 -- 172.21.15.113:0/512565781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392805f590 con 0x7f39280fd150 2024-04-12T05:34:08.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.690+0000 7f391a7fc700 1 -- 172.21.15.113:0/512565781 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3924009590 con 0x7f39280fd150 2024-04-12T05:34:08.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.690+0000 7f391a7fc700 1 --2- 172.21.15.113:0/512565781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f391406ca30 0x7f391406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:08.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.691+0000 7f391a7fc700 1 -- 172.21.15.113:0/512565781 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f392408d0f0 con 0x7f39280fd150 2024-04-12T05:34:08.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.691+0000 7f392d5c7700 1 --2- 172.21.15.113:0/512565781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f391406ca30 0x7f391406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:08.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.691+0000 7f392d5c7700 1 --2- 172.21.15.113:0/512565781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f391406ca30 0x7f391406eef0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f391c004be0 tx=0x7f391c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:08.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.695+0000 7f391a7fc700 1 -- 172.21.15.113:0/512565781 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3924057110 con 0x7f39280fd150 2024-04-12T05:34:08.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.893+0000 7f392f82b700 1 -- 172.21.15.113:0/512565781 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3928059120 con 0x7f391406ca30 2024-04-12T05:34:08.902 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.903+0000 7f391a7fc700 1 -- 172.21.15.113:0/512565781 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3928059120 con 0x7f391406ca30 2024-04-12T05:34:08.902 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:34:08.902 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (22m) 14m ago 33m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:34:08.902 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (19m) 14m ago 33m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:34:08.902 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (19m) 14m ago 32m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:34:08.902 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (21m) 14m ago 32m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:34:08.902 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (15m) 14m ago 27m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:34:08.902 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (14m) 14m ago 27m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:34:08.902 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (14m) 14m ago 27m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:34:08.902 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (27m) 14m ago 27m 16.5M - 16.2.15-1-g8963e9de 727f386679cb ce2c25ba6c58 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (23m) 14m ago 34m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (22m) 14m ago 32m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (20m) 14m ago 34m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (20m) 14m ago 32m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (22m) 14m ago 32m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (22m) 14m ago 32m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (18m) 14m ago 31m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (18m) 14m ago 31m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (17m) 14m ago 30m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (17m) 14m ago 30m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (16m) 14m ago 29m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (15m) 14m ago 29m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:34:08.903 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (22m) 14m ago 32m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:34:08.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.905+0000 7f392f82b700 1 -- 172.21.15.113:0/512565781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f391406ca30 msgr2=0x7f391406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.905+0000 7f392f82b700 1 --2- 172.21.15.113:0/512565781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f391406ca30 0x7f391406eef0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f391c004be0 tx=0x7f391c00b040).stop 2024-04-12T05:34:08.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.905+0000 7f392f82b700 1 -- 172.21.15.113:0/512565781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280fd150 msgr2=0x7f39281095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:08.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.905+0000 7f392f82b700 1 --2- 172.21.15.113:0/512565781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280fd150 0x7f39281095d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f39240099f0 tx=0x7f3924009cb0).stop 2024-04-12T05:34:08.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.905+0000 7f392f82b700 1 -- 172.21.15.113:0/512565781 shutdown_connections 2024-04-12T05:34:08.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.905+0000 7f392f82b700 1 --2- 172.21.15.113:0/512565781 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f391406ca30 0x7f391406eef0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.905+0000 7f392f82b700 1 --2- 172.21.15.113:0/512565781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39280fc7f0 0x7f39281090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.904 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.905+0000 7f392f82b700 1 --2- 172.21.15.113:0/512565781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39280fd150 0x7f39281095d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:08.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.905+0000 7f392f82b700 1 -- 172.21.15.113:0/512565781 >> 172.21.15.113:0/512565781 conn(0x7f3928070590 msgr2=0x7f39280fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:08.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.905+0000 7f392f82b700 1 -- 172.21.15.113:0/512565781 shutdown_connections 2024-04-12T05:34:08.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:08.905+0000 7f392f82b700 1 -- 172.21.15.113:0/512565781 wait complete. 2024-04-12T05:34:09.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.001+0000 7f6f47a52700 1 -- 172.21.15.113:0/2085678012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f400fd8d0 msgr2=0x7f6f400fdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.001+0000 7f6f47a52700 1 --2- 172.21.15.113:0/2085678012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f400fd8d0 0x7f6f400fdcb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f6f3c0047f0 tx=0x7f6f3c009cb0).stop 2024-04-12T05:34:09.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.002+0000 7f6f47a52700 1 -- 172.21.15.113:0/2085678012 shutdown_connections 2024-04-12T05:34:09.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.002+0000 7f6f47a52700 1 --2- 172.21.15.113:0/2085678012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f400fe230 0x7f6f400f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.002+0000 7f6f47a52700 1 --2- 172.21.15.113:0/2085678012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f400fd8d0 0x7f6f400fdcb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.002+0000 7f6f47a52700 1 -- 172.21.15.113:0/2085678012 >> 172.21.15.113:0/2085678012 conn(0x7f6f4006cea0 msgr2=0x7f6f4006d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:09.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.002+0000 7f6f47a52700 1 -- 172.21.15.113:0/2085678012 shutdown_connections 2024-04-12T05:34:09.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.002+0000 7f6f47a52700 1 -- 172.21.15.113:0/2085678012 wait complete. 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.002+0000 7f6f47a52700 1 Processor -- start 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.002+0000 7f6f47a52700 1 -- start start 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f47a52700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f400fd8d0 0x7f6f4010e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f47a52700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f400fe230 0x7f6f4010e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f47a52700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4010eb40 con 0x7f6f400fd8d0 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f47a52700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4010ec80 con 0x7f6f400fe230 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f457ee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f400fd8d0 0x7f6f4010e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f457ee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f400fd8d0 0x7f6f4010e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59210/0 (socket says 172.21.15.113:59210) 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f457ee700 1 -- 172.21.15.113:0/3983956836 learned_addr learned my addr 172.21.15.113:0/3983956836 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f44fed700 1 --2- 172.21.15.113:0/3983956836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f400fe230 0x7f6f4010e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f457ee700 1 -- 172.21.15.113:0/3983956836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f400fe230 msgr2=0x7f6f4010e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f457ee700 1 --2- 172.21.15.113:0/3983956836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f400fe230 0x7f6f4010e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f457ee700 1 -- 172.21.15.113:0/3983956836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3c005040 con 0x7f6f400fd8d0 2024-04-12T05:34:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f457ee700 1 --2- 172.21.15.113:0/3983956836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f400fd8d0 0x7f6f4010e050 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f6f3c005010 tx=0x7f6f3c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f367fc700 1 -- 172.21.15.113:0/3983956836 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f3c00e030 con 0x7f6f400fd8d0 2024-04-12T05:34:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.003+0000 7f6f367fc700 1 -- 172.21.15.113:0/3983956836 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f3c007e90 con 0x7f6f400fd8d0 2024-04-12T05:34:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.004+0000 7f6f47a52700 1 -- 172.21.15.113:0/3983956836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f401082b0 con 0x7f6f400fd8d0 2024-04-12T05:34:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.004+0000 7f6f367fc700 1 -- 172.21.15.113:0/3983956836 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f3c018070 con 0x7f6f400fd8d0 2024-04-12T05:34:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.004+0000 7f6f47a52700 1 -- 172.21.15.113:0/3983956836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f40108790 con 0x7f6f400fd8d0 2024-04-12T05:34:09.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.005+0000 7f6f47a52700 1 -- 172.21.15.113:0/3983956836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f40047540 con 0x7f6f400fd8d0 2024-04-12T05:34:09.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.005+0000 7f6f367fc700 1 -- 172.21.15.113:0/3983956836 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6f3c01d6e0 con 0x7f6f400fd8d0 2024-04-12T05:34:09.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.005+0000 7f6f367fc700 1 --2- 172.21.15.113:0/3983956836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f2c06caa0 0x7f6f2c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:09.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.005+0000 7f6f367fc700 1 -- 172.21.15.113:0/3983956836 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6f3c08d060 con 0x7f6f400fd8d0 2024-04-12T05:34:09.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.005+0000 7f6f44fed700 1 --2- 172.21.15.113:0/3983956836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f2c06caa0 0x7f6f2c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:09.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.006+0000 7f6f44fed700 1 --2- 172.21.15.113:0/3983956836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f2c06caa0 0x7f6f2c06ef60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f6f40109990 tx=0x7f6f30009dc0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:09.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.010+0000 7f6f367fc700 1 -- 172.21.15.113:0/3983956836 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6f3c05abd0 con 0x7f6f400fd8d0 2024-04-12T05:34:09.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.277+0000 7f6f47a52700 1 -- 172.21.15.113:0/3983956836 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6f401097e0 con 0x7f6f400fd8d0 2024-04-12T05:34:09.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.278+0000 7f6f367fc700 1 -- 172.21.15.113:0/3983956836 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+847 (secure 0 0 0) 0x7f6f3c05a760 con 0x7f6f400fd8d0 2024-04-12T05:34:09.277 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:34:09.277 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:34:09.277 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:34:09.277 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:34:09.277 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 3 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 16.2.15-1-g8963e9de (8963e9dec9df5145388f0d6f6f32af0a74591caf) pacific (stable)": 1, 2024-04-12T05:34:09.278 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 13 2024-04-12T05:34:09.279 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:34:09.279 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:34:09.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.280+0000 7f6f47a52700 1 -- 172.21.15.113:0/3983956836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f2c06caa0 msgr2=0x7f6f2c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.280+0000 7f6f47a52700 1 --2- 172.21.15.113:0/3983956836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f2c06caa0 0x7f6f2c06ef60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f6f40109990 tx=0x7f6f30009dc0).stop 2024-04-12T05:34:09.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.280+0000 7f6f47a52700 1 -- 172.21.15.113:0/3983956836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f400fd8d0 msgr2=0x7f6f4010e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.280+0000 7f6f47a52700 1 --2- 172.21.15.113:0/3983956836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f400fd8d0 0x7f6f4010e050 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f6f3c005010 tx=0x7f6f3c009cb0).stop 2024-04-12T05:34:09.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.280+0000 7f6f47a52700 1 -- 172.21.15.113:0/3983956836 shutdown_connections 2024-04-12T05:34:09.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.280+0000 7f6f47a52700 1 --2- 172.21.15.113:0/3983956836 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f2c06caa0 0x7f6f2c06ef60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.280+0000 7f6f47a52700 1 --2- 172.21.15.113:0/3983956836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f400fd8d0 0x7f6f4010e050 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.280+0000 7f6f47a52700 1 --2- 172.21.15.113:0/3983956836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f400fe230 0x7f6f4010e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.280+0000 7f6f47a52700 1 -- 172.21.15.113:0/3983956836 >> 172.21.15.113:0/3983956836 conn(0x7f6f4006cea0 msgr2=0x7f6f40100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:09.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.280+0000 7f6f47a52700 1 -- 172.21.15.113:0/3983956836 shutdown_connections 2024-04-12T05:34:09.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.280+0000 7f6f47a52700 1 -- 172.21.15.113:0/3983956836 wait complete. 2024-04-12T05:34:09.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.378+0000 7f076c70d700 1 -- 172.21.15.113:0/4062697461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07640fd8f0 msgr2=0x7f07640fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.378+0000 7f076c70d700 1 --2- 172.21.15.113:0/4062697461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07640fd8f0 0x7f07640fdcd0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f07540047f0 tx=0x7f0754009cb0).stop 2024-04-12T05:34:09.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.378+0000 7f076c70d700 1 -- 172.21.15.113:0/4062697461 shutdown_connections 2024-04-12T05:34:09.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.378+0000 7f076c70d700 1 --2- 172.21.15.113:0/4062697461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07640fe250 0x7f07640f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.378+0000 7f076c70d700 1 --2- 172.21.15.113:0/4062697461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07640fd8f0 0x7f07640fdcd0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.378+0000 7f076c70d700 1 -- 172.21.15.113:0/4062697461 >> 172.21.15.113:0/4062697461 conn(0x7f07640f34b0 msgr2=0x7f07640f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.378+0000 7f076c70d700 1 -- 172.21.15.113:0/4062697461 shutdown_connections 2024-04-12T05:34:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.378+0000 7f076c70d700 1 -- 172.21.15.113:0/4062697461 wait complete. 2024-04-12T05:34:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.378+0000 7f076c70d700 1 Processor -- start 2024-04-12T05:34:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.378+0000 7f076c70d700 1 -- start start 2024-04-12T05:34:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076c70d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07640fd8f0 0x7f076410e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076c70d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07640fe250 0x7f076410e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:09.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076c70d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076410eaf0 con 0x7f07640fd8f0 2024-04-12T05:34:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076c70d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076410ec30 con 0x7f07640fe250 2024-04-12T05:34:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076a4a9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07640fd8f0 0x7f076410e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076a4a9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07640fd8f0 0x7f076410e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59230/0 (socket says 172.21.15.113:59230) 2024-04-12T05:34:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076a4a9700 1 -- 172.21.15.113:0/1429727197 learned_addr learned my addr 172.21.15.113:0/1429727197 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076a4a9700 1 -- 172.21.15.113:0/1429727197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07640fe250 msgr2=0x7f076410e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076a4a9700 1 --2- 172.21.15.113:0/1429727197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07640fe250 0x7f076410e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:34:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076a4a9700 1 -- 172.21.15.113:0/1429727197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0754005040 con 0x7f07640fd8f0 2024-04-12T05:34:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076a4a9700 1 --2- 172.21.15.113:0/1429727197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07640fd8f0 0x7f076410e000 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f075400b120 tx=0x7f0754009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:09.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f075b7fe700 1 -- 172.21.15.113:0/1429727197 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f075400e030 con 0x7f07640fd8f0 2024-04-12T05:34:09.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076c70d700 1 -- 172.21.15.113:0/1429727197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0764108260 con 0x7f07640fd8f0 2024-04-12T05:34:09.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f075b7fe700 1 -- 172.21.15.113:0/1429727197 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0754007e90 con 0x7f07640fd8f0 2024-04-12T05:34:09.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f075b7fe700 1 -- 172.21.15.113:0/1429727197 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0754018070 con 0x7f07640fd8f0 2024-04-12T05:34:09.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.379+0000 7f076c70d700 1 -- 172.21.15.113:0/1429727197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07641087c0 con 0x7f07640fd8f0 2024-04-12T05:34:09.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.380+0000 7f076c70d700 1 -- 172.21.15.113:0/1429727197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076405f590 con 0x7f07640fd8f0 2024-04-12T05:34:09.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.381+0000 7f075b7fe700 1 -- 172.21.15.113:0/1429727197 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f075401d420 con 0x7f07640fd8f0 2024-04-12T05:34:09.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.381+0000 7f075b7fe700 1 --2- 172.21.15.113:0/1429727197 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f075006caa0 0x7f075006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:09.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.381+0000 7f075b7fe700 1 -- 172.21.15.113:0/1429727197 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f075408d6c0 con 0x7f07640fd8f0 2024-04-12T05:34:09.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.381+0000 7f0769ca8700 1 --2- 172.21.15.113:0/1429727197 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f075006caa0 0x7f075006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:09.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.382+0000 7f0769ca8700 1 --2- 172.21.15.113:0/1429727197 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f075006caa0 0x7f075006ef60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f0764109940 tx=0x7f0760007bc0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:09.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.385+0000 7f075b7fe700 1 -- 172.21.15.113:0/1429727197 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f075405b150 con 0x7f07640fd8f0 2024-04-12T05:34:09.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:09 smithi192 ceph-mon[85304]: pgmap v659: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:09.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.614+0000 7f076c70d700 1 -- 172.21.15.113:0/1429727197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0764109720 con 0x7f07640fd8f0 2024-04-12T05:34:09.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.614+0000 7f075b7fe700 1 -- 172.21.15.113:0/1429727197 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+1841 (secure 0 0 0) 0x7f075405ace0 con 0x7f07640fd8f0 2024-04-12T05:34:09.614 INFO:teuthology.orchestra.run.smithi113.stdout:e26 2024-04-12T05:34:09.614 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:34:09.614 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:34:09.614 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:34:09.614 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:34:09.614 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:34:09.614 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:34:09.614 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:34:09.614 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:34:09.614 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:34:09.614 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:34:09.615 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:34:09.616 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:34:09.617 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:24271} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/3401306304,v1:172.21.15.192:6827/3401306304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:09.617 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:09.617 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:09.617 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:09.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.616+0000 7f076c70d700 1 -- 172.21.15.113:0/1429727197 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f075006caa0 msgr2=0x7f075006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.616+0000 7f076c70d700 1 --2- 172.21.15.113:0/1429727197 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f075006caa0 0x7f075006ef60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f0764109940 tx=0x7f0760007bc0).stop 2024-04-12T05:34:09.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.616+0000 7f076c70d700 1 -- 172.21.15.113:0/1429727197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07640fd8f0 msgr2=0x7f076410e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.616+0000 7f076c70d700 1 --2- 172.21.15.113:0/1429727197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07640fd8f0 0x7f076410e000 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f075400b120 tx=0x7f0754009cb0).stop 2024-04-12T05:34:09.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.617+0000 7f076c70d700 1 -- 172.21.15.113:0/1429727197 shutdown_connections 2024-04-12T05:34:09.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.617+0000 7f076c70d700 1 --2- 172.21.15.113:0/1429727197 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f075006caa0 0x7f075006ef60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.617+0000 7f076c70d700 1 --2- 172.21.15.113:0/1429727197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07640fd8f0 0x7f076410e000 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.617+0000 7f076c70d700 1 --2- 172.21.15.113:0/1429727197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07640fe250 0x7f076410e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:34:09.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.617+0000 7f076c70d700 1 -- 172.21.15.113:0/1429727197 >> 172.21.15.113:0/1429727197 conn(0x7f07640f34b0 msgr2=0x7f07640fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:09.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.617+0000 7f076c70d700 1 -- 172.21.15.113:0/1429727197 shutdown_connections 2024-04-12T05:34:09.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.617+0000 7f076c70d700 1 -- 172.21.15.113:0/1429727197 wait complete. 2024-04-12T05:34:09.619 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:34:09.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:09 smithi113 ceph-mon[103371]: pgmap v659: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:09.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.716+0000 7fe7ae857700 1 -- 172.21.15.113:0/404054425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a80fd150 msgr2=0x7fe7a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.716+0000 7fe7ae857700 1 --2- 172.21.15.113:0/404054425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a80fd150 0x7fe7a80f6930 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7fe7980047f0 tx=0x7fe798009cb0).stop 2024-04-12T05:34:09.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.716+0000 7fe7ae857700 1 -- 172.21.15.113:0/404054425 shutdown_connections 2024-04-12T05:34:09.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.716+0000 7fe7ae857700 1 --2- 172.21.15.113:0/404054425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a80fd150 0x7fe7a80f6930 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.716+0000 7fe7ae857700 1 --2- 172.21.15.113:0/404054425 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a80fc7f0 0x7fe7a80fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.716+0000 7fe7ae857700 1 -- 172.21.15.113:0/404054425 >> 172.21.15.113:0/404054425 conn(0x7fe7a8070590 msgr2=0x7fe7a80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:09.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.716+0000 7fe7ae857700 1 -- 172.21.15.113:0/404054425 shutdown_connections 2024-04-12T05:34:09.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.716+0000 7fe7ae857700 1 -- 172.21.15.113:0/404054425 wait complete. 2024-04-12T05:34:09.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7ae857700 1 Processor -- start 2024-04-12T05:34:09.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7ae857700 1 -- start start 2024-04-12T05:34:09.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7ae857700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a80fc7f0 0x7fe7a8109080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:09.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7ae857700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a80fd150 0x7fe7a8109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:09.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7ae857700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a8109a60 con 0x7fe7a80fc7f0 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7ae857700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a8109ba0 con 0x7fe7a80fd150 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7a7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a80fc7f0 0x7fe7a8109080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7a7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a80fc7f0 0x7fe7a8109080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59250/0 (socket says 172.21.15.113:59250) 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7a7fff700 1 -- 172.21.15.113:0/972427747 learned_addr learned my addr 172.21.15.113:0/972427747 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7a77fe700 1 --2- 172.21.15.113:0/972427747 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a80fd150 0x7fe7a8109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7a7fff700 1 -- 172.21.15.113:0/972427747 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a80fd150 msgr2=0x7fe7a8109570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7a7fff700 1 --2- 172.21.15.113:0/972427747 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a80fd150 0x7fe7a8109570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.717+0000 7fe7a7fff700 1 -- 172.21.15.113:0/972427747 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe798005040 con 0x7fe7a80fc7f0 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.718+0000 7fe7a7fff700 1 --2- 172.21.15.113:0/972427747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a80fc7f0 0x7fe7a8109080 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fe79c0041d0 tx=0x7fe79c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.718+0000 7fe7a57fa700 1 -- 172.21.15.113:0/972427747 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe79c007b40 con 0x7fe7a80fc7f0 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.718+0000 7fe7ae857700 1 -- 172.21.15.113:0/972427747 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a8109e80 con 0x7fe7a80fc7f0 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.718+0000 7fe7a57fa700 1 -- 172.21.15.113:0/972427747 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe79c00f040 con 0x7fe7a80fc7f0 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.718+0000 7fe7a57fa700 1 -- 172.21.15.113:0/972427747 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe79c0137d0 con 0x7fe7a80fc7f0 2024-04-12T05:34:09.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.718+0000 7fe7ae857700 1 -- 172.21.15.113:0/972427747 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a8112440 con 0x7fe7a80fc7f0 2024-04-12T05:34:09.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.719+0000 7fe7ae857700 1 -- 172.21.15.113:0/972427747 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a805f590 con 0x7fe7a80fc7f0 2024-04-12T05:34:09.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.719+0000 7fe7a57fa700 1 -- 172.21.15.113:0/972427747 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe79c007ca0 con 0x7fe7a80fc7f0 2024-04-12T05:34:09.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.720+0000 7fe7a57fa700 1 --2- 172.21.15.113:0/972427747 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 0x7fe79006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:09.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.720+0000 7fe7a57fa700 1 -- 172.21.15.113:0/972427747 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe79c08b860 con 0x7fe7a80fc7f0 2024-04-12T05:34:09.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.720+0000 7fe7a77fe700 1 --2- 172.21.15.113:0/972427747 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 0x7fe79006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:09.720 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.720+0000 7fe7a77fe700 1 --2- 172.21.15.113:0/972427747 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 0x7fe79006ef60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fe7a8060ed0 tx=0x7fe79800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:09.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.724+0000 7fe7a57fa700 1 -- 172.21.15.113:0/972427747 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe79c055880 con 0x7fe7a80fc7f0 2024-04-12T05:34:09.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.930+0000 7fe7ae857700 1 -- 172.21.15.113:0/972427747 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7a8059120 con 0x7fe79006caa0 2024-04-12T05:34:09.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.931+0000 7fe7a57fa700 1 -- 172.21.15.113:0/972427747 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe7a8059120 con 0x7fe79006caa0 2024-04-12T05:34:09.930 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:34:09.930 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:34:09.930 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:34:09.930 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:34:09.930 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:34:09.930 INFO:teuthology.orchestra.run.smithi113.stdout: "mon", 2024-04-12T05:34:09.931 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:34:09.931 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:34:09.931 INFO:teuthology.orchestra.run.smithi113.stdout: "crash" 2024-04-12T05:34:09.931 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:34:09.931 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "15/21 daemons upgraded", 2024-04-12T05:34:09.931 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:34:09.931 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:34:09.931 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:34:09.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.933+0000 7fe7ae857700 1 -- 172.21.15.113:0/972427747 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 msgr2=0x7fe79006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.933+0000 7fe7ae857700 1 --2- 172.21.15.113:0/972427747 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 0x7fe79006ef60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fe7a8060ed0 tx=0x7fe79800b040).stop 2024-04-12T05:34:09.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.933+0000 7fe7ae857700 1 -- 172.21.15.113:0/972427747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a80fc7f0 msgr2=0x7fe7a8109080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:09.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.933+0000 7fe7ae857700 1 --2- 172.21.15.113:0/972427747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a80fc7f0 0x7fe7a8109080 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fe79c0041d0 tx=0x7fe79c009f60).stop 2024-04-12T05:34:09.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.933+0000 7fe7ae857700 1 -- 172.21.15.113:0/972427747 shutdown_connections 2024-04-12T05:34:09.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.933+0000 7fe7ae857700 1 --2- 172.21.15.113:0/972427747 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 0x7fe79006ef60 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.933+0000 7fe7ae857700 1 --2- 172.21.15.113:0/972427747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a80fc7f0 0x7fe7a8109080 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.933+0000 7fe7ae857700 1 --2- 172.21.15.113:0/972427747 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a80fd150 0x7fe7a8109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:09.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.933+0000 7fe7ae857700 1 -- 172.21.15.113:0/972427747 >> 172.21.15.113:0/972427747 conn(0x7fe7a8070590 msgr2=0x7fe7a80fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:09.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.933+0000 7fe7ae857700 1 -- 172.21.15.113:0/972427747 shutdown_connections 2024-04-12T05:34:09.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:09.933+0000 7fe7ae857700 1 -- 172.21.15.113:0/972427747 wait complete. 2024-04-12T05:34:10.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.031+0000 7f787411c700 1 -- 172.21.15.113:0/472904117 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f786c0fd8f0 msgr2=0x7f786c0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:10.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.031+0000 7f787411c700 1 --2- 172.21.15.113:0/472904117 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f786c0fd8f0 0x7f786c0fdcd0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f78680047f0 tx=0x7f7868009cb0).stop 2024-04-12T05:34:10.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.031+0000 7f787411c700 1 -- 172.21.15.113:0/472904117 shutdown_connections 2024-04-12T05:34:10.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.031+0000 7f787411c700 1 --2- 172.21.15.113:0/472904117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f786c0fe250 0x7f786c0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:10.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.031+0000 7f787411c700 1 --2- 172.21.15.113:0/472904117 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f786c0fd8f0 0x7f786c0fdcd0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:10.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.031+0000 7f787411c700 1 -- 172.21.15.113:0/472904117 >> 172.21.15.113:0/472904117 conn(0x7f786c0f34b0 msgr2=0x7f786c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:10.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.031+0000 7f787411c700 1 -- 172.21.15.113:0/472904117 shutdown_connections 2024-04-12T05:34:10.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.031+0000 7f787411c700 1 -- 172.21.15.113:0/472904117 wait complete. 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f787411c700 1 Processor -- start 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f787411c700 1 -- start start 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f787411c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f786c0fd8f0 0x7f786c10e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f787411c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f786c0fe250 0x7f786c10e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f787411c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c10eaf0 con 0x7f786c0fe250 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f787411c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c10ec30 con 0x7f786c0fd8f0 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f78716b7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f786c0fe250 0x7f786c10e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f78716b7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f786c0fe250 0x7f786c10e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59264/0 (socket says 172.21.15.113:59264) 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f78716b7700 1 -- 172.21.15.113:0/246248045 learned_addr learned my addr 172.21.15.113:0/246248045 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f78716b7700 1 -- 172.21.15.113:0/246248045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f786c0fd8f0 msgr2=0x7f786c10e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f78716b7700 1 --2- 172.21.15.113:0/246248045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f786c0fd8f0 0x7f786c10e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:34:10.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f78716b7700 1 -- 172.21.15.113:0/246248045 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7868005040 con 0x7f786c0fe250 2024-04-12T05:34:10.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f78716b7700 1 --2- 172.21.15.113:0/246248045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f786c0fe250 0x7f786c10e4f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f785c00c790 tx=0x7f785c007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:10.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.032+0000 7f7862ffd700 1 -- 172.21.15.113:0/246248045 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f785c0106f0 con 0x7f786c0fe250 2024-04-12T05:34:10.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.033+0000 7f7862ffd700 1 -- 172.21.15.113:0/246248045 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f785c010d00 con 0x7f786c0fe250 2024-04-12T05:34:10.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.033+0000 7f7862ffd700 1 -- 172.21.15.113:0/246248045 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f785c012d50 con 0x7f786c0fe250 2024-04-12T05:34:10.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.033+0000 7f787411c700 1 -- 172.21.15.113:0/246248045 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786c1082c0 con 0x7f786c0fe250 2024-04-12T05:34:10.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.033+0000 7f787411c700 1 -- 172.21.15.113:0/246248045 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786c108880 con 0x7f786c0fe250 2024-04-12T05:34:10.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.034+0000 7f787411c700 1 -- 172.21.15.113:0/246248045 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786c05f590 con 0x7f786c0fe250 2024-04-12T05:34:10.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.034+0000 7f7862ffd700 1 -- 172.21.15.113:0/246248045 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f785c012570 con 0x7f786c0fe250 2024-04-12T05:34:10.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.034+0000 7f7862ffd700 1 --2- 172.21.15.113:0/246248045 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f785806caa0 0x7f785806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:10.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.034+0000 7f7862ffd700 1 -- 172.21.15.113:0/246248045 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f785c08b790 con 0x7f786c0fe250 2024-04-12T05:34:10.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.034+0000 7f7871eb8700 1 --2- 172.21.15.113:0/246248045 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f785806caa0 0x7f785806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:10.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.035+0000 7f7871eb8700 1 --2- 172.21.15.113:0/246248045 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f785806caa0 0x7f785806ef60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f78680047f0 tx=0x7f7868009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:10.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.039+0000 7f7862ffd700 1 -- 172.21.15.113:0/246248045 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f785c059240 con 0x7f786c0fe250 2024-04-12T05:34:10.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.307+0000 7f787411c700 1 -- 172.21.15.113:0/246248045 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f786c047540 con 0x7f786c0fe250 2024-04-12T05:34:10.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.307+0000 7f7862ffd700 1 -- 172.21.15.113:0/246248045 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f785c058dd0 con 0x7f786c0fe250 2024-04-12T05:34:10.306 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:34:10.307 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:34:10.307 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:34:10.307 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:34:10.307 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:34:10.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.309+0000 7f787411c700 1 -- 172.21.15.113:0/246248045 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f785806caa0 msgr2=0x7f785806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:10.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.309+0000 7f787411c700 1 --2- 172.21.15.113:0/246248045 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f785806caa0 0x7f785806ef60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f78680047f0 tx=0x7f7868009cb0).stop 2024-04-12T05:34:10.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.309+0000 7f787411c700 1 -- 172.21.15.113:0/246248045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f786c0fe250 msgr2=0x7f786c10e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:10.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.309+0000 7f787411c700 1 --2- 172.21.15.113:0/246248045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f786c0fe250 0x7f786c10e4f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f785c00c790 tx=0x7f785c007f50).stop 2024-04-12T05:34:10.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.309+0000 7f787411c700 1 -- 172.21.15.113:0/246248045 shutdown_connections 2024-04-12T05:34:10.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.309+0000 7f787411c700 1 --2- 172.21.15.113:0/246248045 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f785806caa0 0x7f785806ef60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:10.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.309+0000 7f787411c700 1 --2- 172.21.15.113:0/246248045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f786c0fd8f0 0x7f786c10e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:34:10.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.309+0000 7f787411c700 1 --2- 172.21.15.113:0/246248045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f786c0fe250 0x7f786c10e4f0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:10.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.309+0000 7f787411c700 1 -- 172.21.15.113:0/246248045 >> 172.21.15.113:0/246248045 conn(0x7f786c0f34b0 msgr2=0x7f786c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:10.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.309+0000 7f787411c700 1 -- 172.21.15.113:0/246248045 shutdown_connections 2024-04-12T05:34:10.309 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:10.309+0000 7f787411c700 1 -- 172.21.15.113:0/246248045 wait complete. 2024-04-12T05:34:10.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:10 smithi192 ceph-mon[85304]: from='client.35204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:10.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:10 smithi192 ceph-mon[85304]: from='client.35208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:10.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:10 smithi192 ceph-mon[85304]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:10.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:10 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3983956836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:10.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:10 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1429727197' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:34:10.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:10 smithi113 ceph-mon[103371]: from='client.35204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:10.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:10 smithi113 ceph-mon[103371]: from='client.35208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:10.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:10 smithi113 ceph-mon[103371]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:10.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:10 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3983956836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:10.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:10 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1429727197' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:34:11.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:11 smithi192 ceph-mon[85304]: from='client.35224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:11.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:11 smithi192 ceph-mon[85304]: pgmap v660: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:11.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:11 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/246248045' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:34:11.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:11 smithi113 ceph-mon[103371]: from='client.35224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:11.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:11 smithi113 ceph-mon[103371]: pgmap v660: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:11.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:11 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/246248045' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:34:13.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:13 smithi192 ceph-mon[85304]: pgmap v661: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:13.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:13 smithi113 ceph-mon[103371]: pgmap v661: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:15.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:15 smithi192 ceph-mon[85304]: pgmap v662: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:15.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:15 smithi113 ceph-mon[103371]: pgmap v662: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:17.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:17 smithi113 ceph-mon[103371]: pgmap v663: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:17.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:17 smithi192 ceph-mon[85304]: pgmap v663: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:19.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:19 smithi113 ceph-mon[103371]: pgmap v664: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:19.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:19 smithi192 ceph-mon[85304]: pgmap v664: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:21.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:21 smithi113 ceph-mon[103371]: pgmap v665: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:21.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:21 smithi192 ceph-mon[85304]: pgmap v665: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:22.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:22.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:23.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:23 smithi113 ceph-mon[103371]: pgmap v666: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:23.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:23 smithi192 ceph-mon[85304]: pgmap v666: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:25.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:25 smithi113 ceph-mon[103371]: pgmap v667: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:25.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:25 smithi192 ceph-mon[85304]: pgmap v667: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:27.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:27 smithi113 ceph-mon[103371]: pgmap v668: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:27.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:27 smithi192 ceph-mon[85304]: pgmap v668: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:29.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:29 smithi113 ceph-mon[103371]: pgmap v669: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:29.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:29 smithi192 ceph-mon[85304]: pgmap v669: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:30.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:30.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:30.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:30.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:30.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:30.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:30.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:30.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:30 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:30.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:30.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:34:30.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:30.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:30.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:30.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:30.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:30.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:30 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:31.417 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:31 smithi192 ceph-mon[85304]: pgmap v670: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:31.417 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:31 smithi192 ceph-mon[85304]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-12T05:34:31.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:31 smithi113 ceph-mon[103371]: pgmap v670: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:31.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:31 smithi113 ceph-mon[103371]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-12T05:34:32.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:32 smithi113 ceph-mon[103371]: Upgrade: Updating mds.cephfs.smithi192.dtnaoi 2024-04-12T05:34:32.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:32 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:32.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:32 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.dtnaoi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:34:32.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:32 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:32.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:32 smithi113 ceph-mon[103371]: Deploying daemon mds.cephfs.smithi192.dtnaoi on smithi192 2024-04-12T05:34:32.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:32 smithi113 ceph-mon[103371]: pgmap v671: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:32.917 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:32 smithi192 ceph-mon[85304]: Upgrade: Updating mds.cephfs.smithi192.dtnaoi 2024-04-12T05:34:32.917 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:32 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:32.917 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:32 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi192.dtnaoi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T05:34:32.917 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:32 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:34:32.917 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:32 smithi192 ceph-mon[85304]: Deploying daemon mds.cephfs.smithi192.dtnaoi on smithi192 2024-04-12T05:34:32.918 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:32 smithi192 ceph-mon[85304]: pgmap v671: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:34.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:34 smithi113 ceph-mon[103371]: osdmap e98: 6 total, 6 up, 6 in 2024-04-12T05:34:34.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:34 smithi113 ceph-mon[103371]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:34:34.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:34 smithi113 ceph-mon[103371]: pgmap v673: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:34.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:34 smithi192 ceph-mon[85304]: osdmap e98: 6 total, 6 up, 6 in 2024-04-12T05:34:34.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:34 smithi192 ceph-mon[85304]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-12T05:34:34.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:34 smithi192 ceph-mon[85304]: pgmap v673: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:37.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:36 smithi113 ceph-mon[103371]: pgmap v674: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:37.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:36 smithi192 ceph-mon[85304]: pgmap v674: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:38.168 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:38.169 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:38.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:38.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:39.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:38 smithi113 ceph-mon[103371]: pgmap v675: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:39.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:39.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:39.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:39.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:39.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:39.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:39.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:38 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:39.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:38 smithi192 ceph-mon[85304]: pgmap v675: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:39.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:39.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:39.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:34:39.196 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:39.196 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:39.196 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:39.196 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:38 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:40.125 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:39 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] up:boot 2024-04-12T05:34:40.125 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:39 smithi192 ceph-mon[85304]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:34:40.125 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:34:40.129 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:39 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] up:boot 2024-04-12T05:34:40.129 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:39 smithi113 ceph-mon[103371]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:34:40.129 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:34:40.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.412+0000 7fccbcb18700 1 -- 172.21.15.113:0/852926264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb80fe2b0 msgr2=0x7fccb8060680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:40.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.412+0000 7fccbcb18700 1 --2- 172.21.15.113:0/852926264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb80fe2b0 0x7fccb8060680 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fcca80047f0 tx=0x7fcca8009cb0).stop 2024-04-12T05:34:40.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.412+0000 7fccbcb18700 1 -- 172.21.15.113:0/852926264 shutdown_connections 2024-04-12T05:34:40.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.412+0000 7fccbcb18700 1 --2- 172.21.15.113:0/852926264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb80fe2b0 0x7fccb8060680 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.412+0000 7fccbcb18700 1 --2- 172.21.15.113:0/852926264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb80fd950 0x7fccb80fdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.412+0000 7fccbcb18700 1 -- 172.21.15.113:0/852926264 >> 172.21.15.113:0/852926264 conn(0x7fccb806c760 msgr2=0x7fccb806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:40.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.412+0000 7fccbcb18700 1 -- 172.21.15.113:0/852926264 shutdown_connections 2024-04-12T05:34:40.412 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.412+0000 7fccbcb18700 1 -- 172.21.15.113:0/852926264 wait complete. 2024-04-12T05:34:40.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.413+0000 7fccbcb18700 1 Processor -- start 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.413+0000 7fccbcb18700 1 -- start start 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.413+0000 7fccbcb18700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb80fd950 0x7fccb810e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.413+0000 7fccbcb18700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb80fe2b0 0x7fccb810e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.413+0000 7fccbcb18700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb810ec50 con 0x7fccb80fd950 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.413+0000 7fccbcb18700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb810ed90 con 0x7fccb80fe2b0 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.413+0000 7fccb659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb80fd950 0x7fccb810e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.413+0000 7fccb659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb80fd950 0x7fccb810e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48940/0 (socket says 172.21.15.113:48940) 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.413+0000 7fccb659c700 1 -- 172.21.15.113:0/4262098004 learned_addr learned my addr 172.21.15.113:0/4262098004 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.414+0000 7fccb5d9b700 1 --2- 172.21.15.113:0/4262098004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb80fe2b0 0x7fccb810e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.414+0000 7fccb659c700 1 -- 172.21.15.113:0/4262098004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb80fe2b0 msgr2=0x7fccb810e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.414+0000 7fccb659c700 1 --2- 172.21.15.113:0/4262098004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb80fe2b0 0x7fccb810e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.414+0000 7fccb659c700 1 -- 172.21.15.113:0/4262098004 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca8005040 con 0x7fccb80fd950 2024-04-12T05:34:40.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.414+0000 7fccb659c700 1 --2- 172.21.15.113:0/4262098004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb80fd950 0x7fccb810e160 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7fccac00e910 tx=0x7fccac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:40.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.414+0000 7fcca77fe700 1 -- 172.21.15.113:0/4262098004 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccac007b40 con 0x7fccb80fd950 2024-04-12T05:34:40.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.414+0000 7fccbcb18700 1 -- 172.21.15.113:0/4262098004 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb81083c0 con 0x7fccb80fd950 2024-04-12T05:34:40.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.414+0000 7fcca77fe700 1 -- 172.21.15.113:0/4262098004 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccac00f040 con 0x7fccb80fd950 2024-04-12T05:34:40.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.414+0000 7fccbcb18700 1 -- 172.21.15.113:0/4262098004 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb8108980 con 0x7fccb80fd950 2024-04-12T05:34:40.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.414+0000 7fcca77fe700 1 -- 172.21.15.113:0/4262098004 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccac0148f0 con 0x7fccb80fd950 2024-04-12T05:34:40.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.415+0000 7fccbcb18700 1 -- 172.21.15.113:0/4262098004 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb805f570 con 0x7fccb80fd950 2024-04-12T05:34:40.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.415+0000 7fcca77fe700 1 -- 172.21.15.113:0/4262098004 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fccac007ca0 con 0x7fccb80fd950 2024-04-12T05:34:40.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.416+0000 7fcca77fe700 1 --2- 172.21.15.113:0/4262098004 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcca006caa0 0x7fcca006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:40.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.416+0000 7fcca77fe700 1 -- 172.21.15.113:0/4262098004 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fccac08b840 con 0x7fccb80fd950 2024-04-12T05:34:40.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.416+0000 7fccb5d9b700 1 --2- 172.21.15.113:0/4262098004 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcca006caa0 0x7fcca006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:40.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.416+0000 7fccb5d9b700 1 --2- 172.21.15.113:0/4262098004 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcca006caa0 0x7fcca006ef60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fccb8109aa0 tx=0x7fcca8009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:40.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.420+0000 7fcca77fe700 1 -- 172.21.15.113:0/4262098004 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fccac0592f0 con 0x7fccb80fd950 2024-04-12T05:34:40.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.644+0000 7fccbcb18700 1 -- 172.21.15.113:0/4262098004 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fccb8108bf0 con 0x7fcca006caa0 2024-04-12T05:34:40.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.645+0000 7fcca77fe700 1 -- 172.21.15.113:0/4262098004 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fccb8108bf0 con 0x7fcca006caa0 2024-04-12T05:34:40.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.649+0000 7fccbcb18700 1 -- 172.21.15.113:0/4262098004 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcca006caa0 msgr2=0x7fcca006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:40.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.649+0000 7fccbcb18700 1 --2- 172.21.15.113:0/4262098004 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcca006caa0 0x7fcca006ef60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fccb8109aa0 tx=0x7fcca8009cb0).stop 2024-04-12T05:34:40.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.649+0000 7fccbcb18700 1 -- 172.21.15.113:0/4262098004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb80fd950 msgr2=0x7fccb810e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:40.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.649+0000 7fccbcb18700 1 --2- 172.21.15.113:0/4262098004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb80fd950 0x7fccb810e160 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7fccac00e910 tx=0x7fccac00b040).stop 2024-04-12T05:34:40.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.649+0000 7fccbcb18700 1 -- 172.21.15.113:0/4262098004 shutdown_connections 2024-04-12T05:34:40.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.649+0000 7fccbcb18700 1 --2- 172.21.15.113:0/4262098004 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcca006caa0 0x7fcca006ef60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.649+0000 7fccbcb18700 1 --2- 172.21.15.113:0/4262098004 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb80fd950 0x7fccb810e160 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.649+0000 7fccbcb18700 1 --2- 172.21.15.113:0/4262098004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fccb80fe2b0 0x7fccb810e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.649+0000 7fccbcb18700 1 -- 172.21.15.113:0/4262098004 >> 172.21.15.113:0/4262098004 conn(0x7fccb806c760 msgr2=0x7fccb80fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:40.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.649+0000 7fccbcb18700 1 -- 172.21.15.113:0/4262098004 shutdown_connections 2024-04-12T05:34:40.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.649+0000 7fccbcb18700 1 -- 172.21.15.113:0/4262098004 wait complete. 2024-04-12T05:34:40.658 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:34:40.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.749+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2775994323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d640fd8f0 msgr2=0x7f2d640fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:40.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.749+0000 7f2d6bde1700 1 --2- 172.21.15.113:0/2775994323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d640fd8f0 0x7f2d640fdcd0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f2d600047f0 tx=0x7f2d60009cb0).stop 2024-04-12T05:34:40.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.749+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2775994323 shutdown_connections 2024-04-12T05:34:40.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.749+0000 7f2d6bde1700 1 --2- 172.21.15.113:0/2775994323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d640fe250 0x7f2d640f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.749+0000 7f2d6bde1700 1 --2- 172.21.15.113:0/2775994323 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d640fd8f0 0x7f2d640fdcd0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.749+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2775994323 >> 172.21.15.113:0/2775994323 conn(0x7f2d640f34b0 msgr2=0x7f2d640f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:40.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.749+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2775994323 shutdown_connections 2024-04-12T05:34:40.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.750+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2775994323 wait complete. 2024-04-12T05:34:40.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.750+0000 7f2d6bde1700 1 Processor -- start 2024-04-12T05:34:40.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.750+0000 7f2d6bde1700 1 -- start start 2024-04-12T05:34:40.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6bde1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d640fd8f0 0x7f2d6410dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:40.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6bde1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d640fe250 0x7f2d6410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:40.751 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6bde1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6410ead0 con 0x7f2d640fe250 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6bde1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6410ec10 con 0x7f2d640fd8f0 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6937c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d640fe250 0x7f2d6410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6937c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d640fe250 0x7f2d6410e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48958/0 (socket says 172.21.15.113:48958) 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6937c700 1 -- 172.21.15.113:0/2713396337 learned_addr learned my addr 172.21.15.113:0/2713396337 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d69b7d700 1 --2- 172.21.15.113:0/2713396337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d640fd8f0 0x7f2d6410dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6937c700 1 -- 172.21.15.113:0/2713396337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d640fd8f0 msgr2=0x7f2d6410dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6937c700 1 --2- 172.21.15.113:0/2713396337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d640fd8f0 0x7f2d6410dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6937c700 1 -- 172.21.15.113:0/2713396337 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d60005040 con 0x7f2d640fe250 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6937c700 1 --2- 172.21.15.113:0/2713396337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d640fe250 0x7f2d6410e4d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f2d5400c790 tx=0x7f2d54007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d5affd700 1 -- 172.21.15.113:0/2713396337 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d5400e680 con 0x7f2d640fe250 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d5affd700 1 -- 172.21.15.113:0/2713396337 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d5400ec90 con 0x7f2d640fe250 2024-04-12T05:34:40.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2713396337 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d641082a0 con 0x7f2d640fe250 2024-04-12T05:34:40.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d5affd700 1 -- 172.21.15.113:0/2713396337 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d54012c70 con 0x7f2d640fe250 2024-04-12T05:34:40.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.751+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2713396337 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d64108860 con 0x7f2d640fe250 2024-04-12T05:34:40.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.752+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2713396337 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6405f590 con 0x7f2d640fe250 2024-04-12T05:34:40.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.753+0000 7f2d5affd700 1 -- 172.21.15.113:0/2713396337 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2d54012490 con 0x7f2d640fe250 2024-04-12T05:34:40.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.753+0000 7f2d5affd700 1 --2- 172.21.15.113:0/2713396337 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2d5006caa0 0x7f2d5006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:40.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.753+0000 7f2d5affd700 1 -- 172.21.15.113:0/2713396337 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2d5408b730 con 0x7f2d640fe250 2024-04-12T05:34:40.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.753+0000 7f2d69b7d700 1 --2- 172.21.15.113:0/2713396337 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2d5006caa0 0x7f2d5006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:40.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.754+0000 7f2d69b7d700 1 --2- 172.21.15.113:0/2713396337 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2d5006caa0 0x7f2d5006ef60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f2d60007cf0 tx=0x7f2d6000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:40.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.758+0000 7f2d5affd700 1 -- 172.21.15.113:0/2713396337 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2d540591e0 con 0x7f2d640fe250 2024-04-12T05:34:40.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.979+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2713396337 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d64108a80 con 0x7f2d5006caa0 2024-04-12T05:34:40.979 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.981+0000 7f2d5affd700 1 -- 172.21.15.113:0/2713396337 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2d64108a80 con 0x7f2d5006caa0 2024-04-12T05:34:40.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.983+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2713396337 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2d5006caa0 msgr2=0x7f2d5006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:40.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.983+0000 7f2d6bde1700 1 --2- 172.21.15.113:0/2713396337 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2d5006caa0 0x7f2d5006ef60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f2d60007cf0 tx=0x7f2d6000b040).stop 2024-04-12T05:34:40.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.983+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2713396337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d640fe250 msgr2=0x7f2d6410e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:40.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.983+0000 7f2d6bde1700 1 --2- 172.21.15.113:0/2713396337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d640fe250 0x7f2d6410e4d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f2d5400c790 tx=0x7f2d54007f70).stop 2024-04-12T05:34:40.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.983+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2713396337 shutdown_connections 2024-04-12T05:34:40.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.983+0000 7f2d6bde1700 1 --2- 172.21.15.113:0/2713396337 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2d5006caa0 0x7f2d5006ef60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.983+0000 7f2d6bde1700 1 --2- 172.21.15.113:0/2713396337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d640fd8f0 0x7f2d6410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.983+0000 7f2d6bde1700 1 --2- 172.21.15.113:0/2713396337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d640fe250 0x7f2d6410e4d0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:40.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.983+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2713396337 >> 172.21.15.113:0/2713396337 conn(0x7f2d640f34b0 msgr2=0x7f2d640fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:40.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.983+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2713396337 shutdown_connections 2024-04-12T05:34:40.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:40.983+0000 7f2d6bde1700 1 -- 172.21.15.113:0/2713396337 wait complete. 2024-04-12T05:34:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.088+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/2814909055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f480fd150 msgr2=0x7f6f480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.088+0000 7f6f4cb8c700 1 --2- 172.21.15.113:0/2814909055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f480fd150 0x7f6f480f6930 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f6f380047f0 tx=0x7f6f38009cb0).stop 2024-04-12T05:34:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.088+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/2814909055 shutdown_connections 2024-04-12T05:34:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.088+0000 7f6f4cb8c700 1 --2- 172.21.15.113:0/2814909055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f480fd150 0x7f6f480f6930 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.088+0000 7f6f4cb8c700 1 --2- 172.21.15.113:0/2814909055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f480fc7f0 0x7f6f480fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.088+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/2814909055 >> 172.21.15.113:0/2814909055 conn(0x7f6f48070590 msgr2=0x7f6f480f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:41.087 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.088+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/2814909055 shutdown_connections 2024-04-12T05:34:41.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.088+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/2814909055 wait complete. 2024-04-12T05:34:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.089+0000 7f6f4cb8c700 1 Processor -- start 2024-04-12T05:34:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.089+0000 7f6f4cb8c700 1 -- start start 2024-04-12T05:34:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.089+0000 7f6f4cb8c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f480fc7f0 0x7f6f4810e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.089+0000 7f6f4cb8c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f480fd150 0x7f6f4810e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.089+0000 7f6f4cb8c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4810eb10 con 0x7f6f480fd150 2024-04-12T05:34:41.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.089+0000 7f6f4cb8c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4810ec50 con 0x7f6f480fc7f0 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.089+0000 7f6f45d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f480fd150 0x7f6f4810e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.089+0000 7f6f45d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f480fd150 0x7f6f4810e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:48990/0 (socket says 172.21.15.113:48990) 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.089+0000 7f6f45d9b700 1 -- 172.21.15.113:0/303510237 learned_addr learned my addr 172.21.15.113:0/303510237 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.089+0000 7f6f4659c700 1 --2- 172.21.15.113:0/303510237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f480fc7f0 0x7f6f4810e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.090+0000 7f6f45d9b700 1 -- 172.21.15.113:0/303510237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f480fc7f0 msgr2=0x7f6f4810e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.090+0000 7f6f45d9b700 1 --2- 172.21.15.113:0/303510237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f480fc7f0 0x7f6f4810e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.090+0000 7f6f45d9b700 1 -- 172.21.15.113:0/303510237 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f38005040 con 0x7f6f480fd150 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.090+0000 7f6f45d9b700 1 --2- 172.21.15.113:0/303510237 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f480fd150 0x7f6f4810e510 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f6f380099f0 tx=0x7f6f38009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.090+0000 7f6f377fe700 1 -- 172.21.15.113:0/303510237 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f3800e030 con 0x7f6f480fd150 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.090+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/303510237 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f48108280 con 0x7f6f480fd150 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.090+0000 7f6f377fe700 1 -- 172.21.15.113:0/303510237 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f38007e90 con 0x7f6f480fd150 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.090+0000 7f6f377fe700 1 -- 172.21.15.113:0/303510237 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f38018070 con 0x7f6f480fd150 2024-04-12T05:34:41.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.090+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/303510237 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f481087e0 con 0x7f6f480fd150 2024-04-12T05:34:41.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.091+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/303510237 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4805f590 con 0x7f6f480fd150 2024-04-12T05:34:41.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.092+0000 7f6f377fe700 1 -- 172.21.15.113:0/303510237 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6f38009590 con 0x7f6f480fd150 2024-04-12T05:34:41.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.092+0000 7f6f377fe700 1 --2- 172.21.15.113:0/303510237 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f3006ca30 0x7f6f3006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:41.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.092+0000 7f6f377fe700 1 -- 172.21.15.113:0/303510237 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6f3808d0f0 con 0x7f6f480fd150 2024-04-12T05:34:41.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.092+0000 7f6f4659c700 1 --2- 172.21.15.113:0/303510237 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f3006ca30 0x7f6f3006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:41.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.092+0000 7f6f4659c700 1 --2- 172.21.15.113:0/303510237 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f3006ca30 0x7f6f3006eef0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f6f3c009170 tx=0x7f6f3c009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:41.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.096+0000 7f6f377fe700 1 -- 172.21.15.113:0/303510237 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6f3805aba0 con 0x7f6f480fd150 2024-04-12T05:34:41.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:40 smithi113 ceph-mon[103371]: pgmap v676: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:41.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:40 smithi192 ceph-mon[85304]: pgmap v676: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:41.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.294+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/303510237 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f480feb40 con 0x7f6f3006ca30 2024-04-12T05:34:41.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.304+0000 7f6f377fe700 1 -- 172.21.15.113:0/303510237 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6f480feb40 con 0x7f6f3006ca30 2024-04-12T05:34:41.302 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (22m) 15m ago 33m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (19m) 15m ago 33m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (19m) 15m ago 33m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (22m) 15m ago 33m 90.7M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (15m) 15m ago 28m 14.7M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (15m) 15m ago 28m 12.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (15m) 15m ago 28m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 starting - - - - 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (23m) 15m ago 34m 506M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (23m) 15m ago 33m 450M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (20m) 15m ago 35m 59.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (20m) 15m ago 32m 41.8M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:34:41.303 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (23m) 15m ago 33m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:34:41.304 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (23m) 15m ago 32m 18.2M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:34:41.304 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (19m) 15m ago 32m 118M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:34:41.304 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (18m) 15m ago 31m 132M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:34:41.304 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (18m) 15m ago 31m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:34:41.304 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (17m) 15m ago 30m 156M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:34:41.304 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (17m) 15m ago 30m 101M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:34:41.304 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (16m) 15m ago 29m 98.5M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:34:41.304 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (22m) 15m ago 32m 60.3M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:34:41.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.306+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/303510237 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f3006ca30 msgr2=0x7f6f3006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:41.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.306+0000 7f6f4cb8c700 1 --2- 172.21.15.113:0/303510237 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f3006ca30 0x7f6f3006eef0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f6f3c009170 tx=0x7f6f3c009da0).stop 2024-04-12T05:34:41.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.306+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/303510237 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f480fd150 msgr2=0x7f6f4810e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:41.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.306+0000 7f6f4cb8c700 1 --2- 172.21.15.113:0/303510237 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f480fd150 0x7f6f4810e510 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f6f380099f0 tx=0x7f6f38009cb0).stop 2024-04-12T05:34:41.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.306+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/303510237 shutdown_connections 2024-04-12T05:34:41.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.306+0000 7f6f4cb8c700 1 --2- 172.21.15.113:0/303510237 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f3006ca30 0x7f6f3006eef0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.306+0000 7f6f4cb8c700 1 --2- 172.21.15.113:0/303510237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f480fc7f0 0x7f6f4810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.306+0000 7f6f4cb8c700 1 --2- 172.21.15.113:0/303510237 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f480fd150 0x7f6f4810e510 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.306+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/303510237 >> 172.21.15.113:0/303510237 conn(0x7f6f48070590 msgr2=0x7f6f480fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:41.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.306+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/303510237 shutdown_connections 2024-04-12T05:34:41.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.306+0000 7f6f4cb8c700 1 -- 172.21.15.113:0/303510237 wait complete. 2024-04-12T05:34:41.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.415+0000 7fa404820700 1 -- 172.21.15.113:0/3851658566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4000fe270 msgr2=0x7fa4000f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:41.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.415+0000 7fa404820700 1 --2- 172.21.15.113:0/3851658566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4000fe270 0x7fa4000f7a50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7fa3f40047f0 tx=0x7fa3f4009cb0).stop 2024-04-12T05:34:41.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.415+0000 7fa404820700 1 -- 172.21.15.113:0/3851658566 shutdown_connections 2024-04-12T05:34:41.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.415+0000 7fa404820700 1 --2- 172.21.15.113:0/3851658566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4000fe270 0x7fa4000f7a50 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.415+0000 7fa404820700 1 --2- 172.21.15.113:0/3851658566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4000fd910 0x7fa4000fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.415+0000 7fa404820700 1 -- 172.21.15.113:0/3851658566 >> 172.21.15.113:0/3851658566 conn(0x7fa4000f3530 msgr2=0x7fa4000f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:41.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.415+0000 7fa404820700 1 -- 172.21.15.113:0/3851658566 shutdown_connections 2024-04-12T05:34:41.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.415+0000 7fa404820700 1 -- 172.21.15.113:0/3851658566 wait complete. 2024-04-12T05:34:41.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.416+0000 7fa404820700 1 Processor -- start 2024-04-12T05:34:41.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.416+0000 7fa404820700 1 -- start start 2024-04-12T05:34:41.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.416+0000 7fa404820700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4000fd910 0x7fa40010dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:41.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.416+0000 7fa404820700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4000fe270 0x7fa40010e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:41.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.416+0000 7fa404820700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40010eae0 con 0x7fa4000fe270 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.416+0000 7fa404820700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40010ec20 con 0x7fa4000fd910 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.416+0000 7fa3fdd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4000fe270 0x7fa40010e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.416+0000 7fa3fdd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4000fe270 0x7fa40010e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49010/0 (socket says 172.21.15.113:49010) 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.416+0000 7fa3fdd9b700 1 -- 172.21.15.113:0/3180152030 learned_addr learned my addr 172.21.15.113:0/3180152030 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.416+0000 7fa3fe59c700 1 --2- 172.21.15.113:0/3180152030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4000fd910 0x7fa40010dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.417+0000 7fa3fdd9b700 1 -- 172.21.15.113:0/3180152030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4000fd910 msgr2=0x7fa40010dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.417+0000 7fa3fdd9b700 1 --2- 172.21.15.113:0/3180152030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4000fd910 0x7fa40010dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.417+0000 7fa3fdd9b700 1 -- 172.21.15.113:0/3180152030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f4005040 con 0x7fa4000fe270 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.417+0000 7fa3fdd9b700 1 --2- 172.21.15.113:0/3180152030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4000fe270 0x7fa40010e4e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7fa3f40099f0 tx=0x7fa3f4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.417+0000 7fa3ef7fe700 1 -- 172.21.15.113:0/3180152030 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3f400e030 con 0x7fa4000fe270 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.417+0000 7fa3ef7fe700 1 -- 172.21.15.113:0/3180152030 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3f4007e90 con 0x7fa4000fe270 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.417+0000 7fa404820700 1 -- 172.21.15.113:0/3180152030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa400108250 con 0x7fa4000fe270 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.417+0000 7fa3ef7fe700 1 -- 172.21.15.113:0/3180152030 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3f4018070 con 0x7fa4000fe270 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.417+0000 7fa404820700 1 -- 172.21.15.113:0/3180152030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa400108730 con 0x7fa4000fe270 2024-04-12T05:34:41.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.418+0000 7fa404820700 1 -- 172.21.15.113:0/3180152030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40005f570 con 0x7fa4000fe270 2024-04-12T05:34:41.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.418+0000 7fa3ef7fe700 1 -- 172.21.15.113:0/3180152030 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa3f40094c0 con 0x7fa4000fe270 2024-04-12T05:34:41.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.419+0000 7fa3ef7fe700 1 --2- 172.21.15.113:0/3180152030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa3e806caa0 0x7fa3e806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:41.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.419+0000 7fa3ef7fe700 1 -- 172.21.15.113:0/3180152030 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa3f408d0c0 con 0x7fa4000fe270 2024-04-12T05:34:41.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.419+0000 7fa3fe59c700 1 --2- 172.21.15.113:0/3180152030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa3e806caa0 0x7fa3e806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:41.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.419+0000 7fa3fe59c700 1 --2- 172.21.15.113:0/3180152030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa3e806caa0 0x7fa3e806ef60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fa3f0004e60 tx=0x7fa3f000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:41.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.423+0000 7fa3ef7fe700 1 -- 172.21.15.113:0/3180152030 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa3f405ab70 con 0x7fa4000fe270 2024-04-12T05:34:41.722 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.724+0000 7fa404820700 1 -- 172.21.15.113:0/3180152030 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa400109670 con 0x7fa4000fe270 2024-04-12T05:34:41.723 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.724+0000 7fa3ef7fe700 1 -- 172.21.15.113:0/3180152030 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fa3f405a700 con 0x7fa4000fe270 2024-04-12T05:34:41.723 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:34:41.723 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:34:41.723 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:34:41.723 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:34:41.724 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:34:41.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.726+0000 7fa404820700 1 -- 172.21.15.113:0/3180152030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa3e806caa0 msgr2=0x7fa3e806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:41.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.726+0000 7fa404820700 1 --2- 172.21.15.113:0/3180152030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa3e806caa0 0x7fa3e806ef60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fa3f0004e60 tx=0x7fa3f000b040).stop 2024-04-12T05:34:41.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.726+0000 7fa404820700 1 -- 172.21.15.113:0/3180152030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4000fe270 msgr2=0x7fa40010e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:41.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.726+0000 7fa404820700 1 --2- 172.21.15.113:0/3180152030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4000fe270 0x7fa40010e4e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7fa3f40099f0 tx=0x7fa3f4009cb0).stop 2024-04-12T05:34:41.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.727+0000 7fa404820700 1 -- 172.21.15.113:0/3180152030 shutdown_connections 2024-04-12T05:34:41.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.727+0000 7fa404820700 1 --2- 172.21.15.113:0/3180152030 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa3e806caa0 0x7fa3e806ef60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.727+0000 7fa404820700 1 --2- 172.21.15.113:0/3180152030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4000fd910 0x7fa40010dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.727+0000 7fa404820700 1 --2- 172.21.15.113:0/3180152030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4000fe270 0x7fa40010e4e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.727+0000 7fa404820700 1 -- 172.21.15.113:0/3180152030 >> 172.21.15.113:0/3180152030 conn(0x7fa4000f3530 msgr2=0x7fa4000fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:41.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.727+0000 7fa404820700 1 -- 172.21.15.113:0/3180152030 shutdown_connections 2024-04-12T05:34:41.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.727+0000 7fa404820700 1 -- 172.21.15.113:0/3180152030 wait complete. 2024-04-12T05:34:41.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.826+0000 7fc1bf86f700 1 -- 172.21.15.113:0/4157074204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b80fc7f0 msgr2=0x7fc1b80fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:41.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.826+0000 7fc1bf86f700 1 --2- 172.21.15.113:0/4157074204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b80fc7f0 0x7fc1b80fcbd0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fc1ac0047f0 tx=0x7fc1ac009cb0).stop 2024-04-12T05:34:41.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.826+0000 7fc1bf86f700 1 -- 172.21.15.113:0/4157074204 shutdown_connections 2024-04-12T05:34:41.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.826+0000 7fc1bf86f700 1 --2- 172.21.15.113:0/4157074204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b80fd150 0x7fc1b80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.826+0000 7fc1bf86f700 1 --2- 172.21.15.113:0/4157074204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b80fc7f0 0x7fc1b80fcbd0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.826+0000 7fc1bf86f700 1 -- 172.21.15.113:0/4157074204 >> 172.21.15.113:0/4157074204 conn(0x7fc1b8070590 msgr2=0x7fc1b80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:41.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.826+0000 7fc1bf86f700 1 -- 172.21.15.113:0/4157074204 shutdown_connections 2024-04-12T05:34:41.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.826+0000 7fc1bf86f700 1 -- 172.21.15.113:0/4157074204 wait complete. 2024-04-12T05:34:41.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bf86f700 1 Processor -- start 2024-04-12T05:34:41.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bf86f700 1 -- start start 2024-04-12T05:34:41.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bf86f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b80fc7f0 0x7fc1b810d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:41.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bf86f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b80fd150 0x7fc1b810d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:41.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bf86f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b810db10 con 0x7fc1b80fd150 2024-04-12T05:34:41.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bf86f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b810dc50 con 0x7fc1b80fc7f0 2024-04-12T05:34:41.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bce0a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b80fd150 0x7fc1b810d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:41.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bce0a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b80fd150 0x7fc1b810d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49020/0 (socket says 172.21.15.113:49020) 2024-04-12T05:34:41.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bce0a700 1 -- 172.21.15.113:0/539884763 learned_addr learned my addr 172.21.15.113:0/539884763 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bd60b700 1 --2- 172.21.15.113:0/539884763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b80fc7f0 0x7fc1b810d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bce0a700 1 -- 172.21.15.113:0/539884763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b80fc7f0 msgr2=0x7fc1b810d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bce0a700 1 --2- 172.21.15.113:0/539884763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b80fc7f0 0x7fc1b810d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.827+0000 7fc1bce0a700 1 -- 172.21.15.113:0/539884763 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac005040 con 0x7fc1b80fd150 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.828+0000 7fc1bce0a700 1 --2- 172.21.15.113:0/539884763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b80fd150 0x7fc1b810d510 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7fc1b400e940 tx=0x7fc1b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.828+0000 7fc1aa7fc700 1 -- 172.21.15.113:0/539884763 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1b4007b40 con 0x7fc1b80fd150 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.828+0000 7fc1bf86f700 1 -- 172.21.15.113:0/539884763 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b81072c0 con 0x7fc1b80fd150 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.828+0000 7fc1aa7fc700 1 -- 172.21.15.113:0/539884763 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1b400f040 con 0x7fc1b80fd150 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.828+0000 7fc1aa7fc700 1 -- 172.21.15.113:0/539884763 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1b40148e0 con 0x7fc1b80fd150 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.828+0000 7fc1bf86f700 1 -- 172.21.15.113:0/539884763 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b8107880 con 0x7fc1b80fd150 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.829+0000 7fc1bf86f700 1 -- 172.21.15.113:0/539884763 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b805f590 con 0x7fc1b80fd150 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.829+0000 7fc1aa7fc700 1 -- 172.21.15.113:0/539884763 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc1b4007ca0 con 0x7fc1b80fd150 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.829+0000 7fc1aa7fc700 1 --2- 172.21.15.113:0/539884763 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc1a406cab0 0x7fc1a406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:41.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.830+0000 7fc1aa7fc700 1 -- 172.21.15.113:0/539884763 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc1b408b8b0 con 0x7fc1b80fd150 2024-04-12T05:34:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.830+0000 7fc1bd60b700 1 --2- 172.21.15.113:0/539884763 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc1a406cab0 0x7fc1a406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:41.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.830+0000 7fc1bd60b700 1 --2- 172.21.15.113:0/539884763 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc1a406cab0 0x7fc1a406ef70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fc1ac004540 tx=0x7fc1ac009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:41.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:41.834+0000 7fc1aa7fc700 1 -- 172.21.15.113:0/539884763 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc1b40558d0 con 0x7fc1b80fd150 2024-04-12T05:34:42.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.069+0000 7fc1bf86f700 1 -- 172.21.15.113:0/539884763 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc1b8047540 con 0x7fc1b80fd150 2024-04-12T05:34:42.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.069+0000 7fc1aa7fc700 1 -- 172.21.15.113:0/539884763 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fc1b4058ef0 con 0x7fc1b80fd150 2024-04-12T05:34:42.068 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:34:42.068 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:34:42.068 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:34:42.068 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:34:42.069 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:34:42.070 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:34:42.071 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:34:42.071 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:34:42.071 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:34:42.071 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:34:42.071 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:34:42.071 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:34:42.071 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:42.071 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:42.071 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:42.071 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:34:42.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.071+0000 7fc1bf86f700 1 -- 172.21.15.113:0/539884763 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc1a406cab0 msgr2=0x7fc1a406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:42.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.071+0000 7fc1bf86f700 1 --2- 172.21.15.113:0/539884763 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc1a406cab0 0x7fc1a406ef70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fc1ac004540 tx=0x7fc1ac009cb0).stop 2024-04-12T05:34:42.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.071+0000 7fc1bf86f700 1 -- 172.21.15.113:0/539884763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b80fd150 msgr2=0x7fc1b810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:42.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.071+0000 7fc1bf86f700 1 --2- 172.21.15.113:0/539884763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b80fd150 0x7fc1b810d510 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7fc1b400e940 tx=0x7fc1b400b040).stop 2024-04-12T05:34:42.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.071+0000 7fc1bf86f700 1 -- 172.21.15.113:0/539884763 shutdown_connections 2024-04-12T05:34:42.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.071+0000 7fc1bf86f700 1 --2- 172.21.15.113:0/539884763 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc1a406cab0 0x7fc1a406ef70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.071+0000 7fc1bf86f700 1 --2- 172.21.15.113:0/539884763 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1b80fc7f0 0x7fc1b810d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.071+0000 7fc1bf86f700 1 --2- 172.21.15.113:0/539884763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc1b80fd150 0x7fc1b810d510 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.071+0000 7fc1bf86f700 1 -- 172.21.15.113:0/539884763 >> 172.21.15.113:0/539884763 conn(0x7fc1b8070590 msgr2=0x7fc1b80fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:42.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.071+0000 7fc1bf86f700 1 -- 172.21.15.113:0/539884763 shutdown_connections 2024-04-12T05:34:42.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.071+0000 7fc1bf86f700 1 -- 172.21.15.113:0/539884763 wait complete. 2024-04-12T05:34:42.073 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:34:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.171+0000 7f2412b86700 1 -- 172.21.15.113:0/404107355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f240c0fe270 msgr2=0x7f240c0f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.171+0000 7f2412b86700 1 --2- 172.21.15.113:0/404107355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f240c0fe270 0x7f240c0f7a50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f23f80047f0 tx=0x7f23f8009cb0).stop 2024-04-12T05:34:42.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.171+0000 7f2412b86700 1 -- 172.21.15.113:0/404107355 shutdown_connections 2024-04-12T05:34:42.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.171+0000 7f2412b86700 1 --2- 172.21.15.113:0/404107355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f240c0fe270 0x7f240c0f7a50 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.171+0000 7f2412b86700 1 --2- 172.21.15.113:0/404107355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240c0fd910 0x7f240c0fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.171+0000 7f2412b86700 1 -- 172.21.15.113:0/404107355 >> 172.21.15.113:0/404107355 conn(0x7f240c0f3530 msgr2=0x7f240c0f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:42.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.171+0000 7f2412b86700 1 -- 172.21.15.113:0/404107355 shutdown_connections 2024-04-12T05:34:42.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.171+0000 7f2412b86700 1 -- 172.21.15.113:0/404107355 wait complete. 2024-04-12T05:34:42.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:41 smithi113 ceph-mon[103371]: from='client.35234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:42.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:41 smithi113 ceph-mon[103371]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:42.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:41 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:42.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:41 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:42.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:41 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3180152030' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:42.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.172+0000 7f2412b86700 1 Processor -- start 2024-04-12T05:34:42.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.172+0000 7f2412b86700 1 -- start start 2024-04-12T05:34:42.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.172+0000 7f2412b86700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f240c0fd910 0x7f240c10e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:42.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.172+0000 7f2412b86700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240c0fe270 0x7f240c10e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:42.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.172+0000 7f2412b86700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c10ebf0 con 0x7f240c0fd910 2024-04-12T05:34:42.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.172+0000 7f2412b86700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c10ed30 con 0x7f240c0fe270 2024-04-12T05:34:42.173 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.172+0000 7f2410922700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f240c0fd910 0x7f240c10e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.172+0000 7f2410922700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f240c0fd910 0x7f240c10e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49048/0 (socket says 172.21.15.113:49048) 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.172+0000 7f2410922700 1 -- 172.21.15.113:0/2310164725 learned_addr learned my addr 172.21.15.113:0/2310164725 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.172+0000 7f240bfff700 1 --2- 172.21.15.113:0/2310164725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240c0fe270 0x7f240c10e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.173+0000 7f2410922700 1 -- 172.21.15.113:0/2310164725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240c0fe270 msgr2=0x7f240c10e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.173+0000 7f2410922700 1 --2- 172.21.15.113:0/2310164725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240c0fe270 0x7f240c10e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.173+0000 7f2410922700 1 -- 172.21.15.113:0/2310164725 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f8005040 con 0x7f240c0fd910 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.173+0000 7f2410922700 1 --2- 172.21.15.113:0/2310164725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f240c0fd910 0x7f240c10e100 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f240000e910 tx=0x7f240000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.173+0000 7f2409ffb700 1 -- 172.21.15.113:0/2310164725 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2400007b40 con 0x7f240c0fd910 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.173+0000 7f2412b86700 1 -- 172.21.15.113:0/2310164725 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c108360 con 0x7f240c0fd910 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.173+0000 7f2409ffb700 1 -- 172.21.15.113:0/2310164725 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f240000f040 con 0x7f240c0fd910 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.173+0000 7f2409ffb700 1 -- 172.21.15.113:0/2310164725 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2400014920 con 0x7f240c0fd910 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.173+0000 7f2412b86700 1 -- 172.21.15.113:0/2310164725 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c108920 con 0x7f240c0fd910 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.174+0000 7f2412b86700 1 -- 172.21.15.113:0/2310164725 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240c05f570 con 0x7f240c0fd910 2024-04-12T05:34:42.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.175+0000 7f2409ffb700 1 -- 172.21.15.113:0/2310164725 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2400007ca0 con 0x7f240c0fd910 2024-04-12T05:34:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.175+0000 7f2409ffb700 1 --2- 172.21.15.113:0/2310164725 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f23fc06caa0 0x7f23fc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.175+0000 7f2409ffb700 1 -- 172.21.15.113:0/2310164725 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f240008b970 con 0x7f240c0fd910 2024-04-12T05:34:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.175+0000 7f240bfff700 1 --2- 172.21.15.113:0/2310164725 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f23fc06caa0 0x7f23fc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:42.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.176+0000 7f240bfff700 1 --2- 172.21.15.113:0/2310164725 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f23fc06caa0 0x7f23fc06ef60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f240c109a40 tx=0x7f23f8009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:42.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.179+0000 7f2409ffb700 1 -- 172.21.15.113:0/2310164725 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2400059420 con 0x7f240c0fd910 2024-04-12T05:34:42.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:41 smithi192 ceph-mon[85304]: from='client.35234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:42.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:41 smithi192 ceph-mon[85304]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:42.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:41 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:42.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:41 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:42.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:41 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3180152030' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:34:42.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.411+0000 7f2412b86700 1 -- 172.21.15.113:0/2310164725 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f240c0ffc60 con 0x7f23fc06caa0 2024-04-12T05:34:42.411 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.412+0000 7f2409ffb700 1 -- 172.21.15.113:0/2310164725 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f240c0ffc60 con 0x7f23fc06caa0 2024-04-12T05:34:42.411 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:34:42.411 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:34:42.411 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:34:42.411 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:34:42.411 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:34:42.411 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:34:42.411 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:34:42.411 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:34:42.411 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:34:42.411 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:34:42.412 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:34:42.412 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:34:42.412 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:34:42.412 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:34:42.412 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:34:42.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.414+0000 7f2412b86700 1 -- 172.21.15.113:0/2310164725 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f23fc06caa0 msgr2=0x7f23fc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:42.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.414+0000 7f2412b86700 1 --2- 172.21.15.113:0/2310164725 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f23fc06caa0 0x7f23fc06ef60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f240c109a40 tx=0x7f23f8009cb0).stop 2024-04-12T05:34:42.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.414+0000 7f2412b86700 1 -- 172.21.15.113:0/2310164725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f240c0fd910 msgr2=0x7f240c10e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:42.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.414+0000 7f2412b86700 1 --2- 172.21.15.113:0/2310164725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f240c0fd910 0x7f240c10e100 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f240000e910 tx=0x7f240000b040).stop 2024-04-12T05:34:42.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.414+0000 7f2412b86700 1 -- 172.21.15.113:0/2310164725 shutdown_connections 2024-04-12T05:34:42.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.414+0000 7f2412b86700 1 --2- 172.21.15.113:0/2310164725 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f23fc06caa0 0x7f23fc06ef60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.415+0000 7f2412b86700 1 --2- 172.21.15.113:0/2310164725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f240c0fd910 0x7f240c10e100 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.415+0000 7f2412b86700 1 --2- 172.21.15.113:0/2310164725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240c0fe270 0x7f240c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.415+0000 7f2412b86700 1 -- 172.21.15.113:0/2310164725 >> 172.21.15.113:0/2310164725 conn(0x7f240c0f3530 msgr2=0x7f240c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:42.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.415+0000 7f2412b86700 1 -- 172.21.15.113:0/2310164725 shutdown_connections 2024-04-12T05:34:42.414 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.415+0000 7f2412b86700 1 -- 172.21.15.113:0/2310164725 wait complete. 2024-04-12T05:34:42.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.526+0000 7fd0d8f25700 1 -- 172.21.15.113:0/811742677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0d40f9200 msgr2=0x7fd0d4101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:42.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.526+0000 7fd0d8f25700 1 --2- 172.21.15.113:0/811742677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0d40f9200 0x7fd0d4101910 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7fd0c80047f0 tx=0x7fd0c8009cb0).stop 2024-04-12T05:34:42.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.526+0000 7fd0d8f25700 1 -- 172.21.15.113:0/811742677 shutdown_connections 2024-04-12T05:34:42.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.526+0000 7fd0d8f25700 1 --2- 172.21.15.113:0/811742677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0d40f9200 0x7fd0d4101910 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.526+0000 7fd0d8f25700 1 --2- 172.21.15.113:0/811742677 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0d40f88a0 0x7fd0d40f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.526+0000 7fd0d8f25700 1 -- 172.21.15.113:0/811742677 >> 172.21.15.113:0/811742677 conn(0x7fd0d4070590 msgr2=0x7fd0d40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:42.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.526+0000 7fd0d8f25700 1 -- 172.21.15.113:0/811742677 shutdown_connections 2024-04-12T05:34:42.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.526+0000 7fd0d8f25700 1 -- 172.21.15.113:0/811742677 wait complete. 2024-04-12T05:34:42.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.527+0000 7fd0d8f25700 1 Processor -- start 2024-04-12T05:34:42.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.527+0000 7fd0d8f25700 1 -- start start 2024-04-12T05:34:42.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.527+0000 7fd0d8f25700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0d40f88a0 0x7fd0d410e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:42.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.527+0000 7fd0d8f25700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0d40f9200 0x7fd0d410e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:42.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.527+0000 7fd0d8f25700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d410ec80 con 0x7fd0d40f88a0 2024-04-12T05:34:42.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.527+0000 7fd0d8f25700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d410edc0 con 0x7fd0d40f9200 2024-04-12T05:34:42.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.527+0000 7fd0d259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0d40f88a0 0x7fd0d410e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:42.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.527+0000 7fd0d259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0d40f88a0 0x7fd0d410e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49076/0 (socket says 172.21.15.113:49076) 2024-04-12T05:34:42.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.527+0000 7fd0d259c700 1 -- 172.21.15.113:0/1465411163 learned_addr learned my addr 172.21.15.113:0/1465411163 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:34:42.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.527+0000 7fd0d1d9b700 1 --2- 172.21.15.113:0/1465411163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0d40f9200 0x7fd0d410e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:42.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.528+0000 7fd0d259c700 1 -- 172.21.15.113:0/1465411163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0d40f9200 msgr2=0x7fd0d410e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:42.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.528+0000 7fd0d259c700 1 --2- 172.21.15.113:0/1465411163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0d40f9200 0x7fd0d410e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.528+0000 7fd0d259c700 1 -- 172.21.15.113:0/1465411163 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c8005040 con 0x7fd0d40f88a0 2024-04-12T05:34:42.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.528+0000 7fd0d259c700 1 --2- 172.21.15.113:0/1465411163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0d40f88a0 0x7fd0d410e190 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7fd0c400e8f0 tx=0x7fd0c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:42.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.528+0000 7fd0c37fe700 1 -- 172.21.15.113:0/1465411163 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0c4007b40 con 0x7fd0d40f88a0 2024-04-12T05:34:42.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.528+0000 7fd0d8f25700 1 -- 172.21.15.113:0/1465411163 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d4108450 con 0x7fd0d40f88a0 2024-04-12T05:34:42.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.528+0000 7fd0c37fe700 1 -- 172.21.15.113:0/1465411163 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0c400f040 con 0x7fd0d40f88a0 2024-04-12T05:34:42.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.528+0000 7fd0c37fe700 1 -- 172.21.15.113:0/1465411163 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0c4014880 con 0x7fd0d40f88a0 2024-04-12T05:34:42.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.528+0000 7fd0d8f25700 1 -- 172.21.15.113:0/1465411163 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d4108990 con 0x7fd0d40f88a0 2024-04-12T05:34:42.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.529+0000 7fd0d8f25700 1 -- 172.21.15.113:0/1465411163 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d405f590 con 0x7fd0d40f88a0 2024-04-12T05:34:42.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.530+0000 7fd0c37fe700 1 -- 172.21.15.113:0/1465411163 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd0c4009090 con 0x7fd0d40f88a0 2024-04-12T05:34:42.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.531+0000 7fd0c37fe700 1 --2- 172.21.15.113:0/1465411163 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0bc070eb0 0x7fd0bc073370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:34:42.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.531+0000 7fd0d1d9b700 1 --2- 172.21.15.113:0/1465411163 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0bc070eb0 0x7fd0bc073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:34:42.529 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.531+0000 7fd0c37fe700 1 -- 172.21.15.113:0/1465411163 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd0c408c180 con 0x7fd0d40f88a0 2024-04-12T05:34:42.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.531+0000 7fd0d1d9b700 1 --2- 172.21.15.113:0/1465411163 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0bc070eb0 0x7fd0bc073370 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fd0d4109ad0 tx=0x7fd0c8009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:34:42.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.534+0000 7fd0c37fe700 1 -- 172.21.15.113:0/1465411163 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd0c4059c30 con 0x7fd0d40f88a0 2024-04-12T05:34:42.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.868+0000 7fd0d8f25700 1 -- 172.21.15.113:0/1465411163 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd0d41098b0 con 0x7fd0d40f88a0 2024-04-12T05:34:42.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.868+0000 7fd0c37fe700 1 -- 172.21.15.113:0/1465411163 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fd0c4019e10 con 0x7fd0d40f88a0 2024-04-12T05:34:42.867 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:34:42.867 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:34:42.867 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:34:42.867 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:34:42.867 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:34:42.867 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:34:42.867 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:34:42.868 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:34:42.869 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:34:42.869 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:34:42.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.870+0000 7fd0d8f25700 1 -- 172.21.15.113:0/1465411163 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0bc070eb0 msgr2=0x7fd0bc073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:42.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.870+0000 7fd0d8f25700 1 --2- 172.21.15.113:0/1465411163 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0bc070eb0 0x7fd0bc073370 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fd0d4109ad0 tx=0x7fd0c8009cb0).stop 2024-04-12T05:34:42.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.870+0000 7fd0d8f25700 1 -- 172.21.15.113:0/1465411163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0d40f88a0 msgr2=0x7fd0d410e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:34:42.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.870+0000 7fd0d8f25700 1 --2- 172.21.15.113:0/1465411163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0d40f88a0 0x7fd0d410e190 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7fd0c400e8f0 tx=0x7fd0c400b040).stop 2024-04-12T05:34:42.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.870+0000 7fd0d8f25700 1 -- 172.21.15.113:0/1465411163 shutdown_connections 2024-04-12T05:34:42.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.870+0000 7fd0d8f25700 1 --2- 172.21.15.113:0/1465411163 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0bc070eb0 0x7fd0bc073370 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.870+0000 7fd0d8f25700 1 --2- 172.21.15.113:0/1465411163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0d40f88a0 0x7fd0d410e190 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.870+0000 7fd0d8f25700 1 --2- 172.21.15.113:0/1465411163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0d40f9200 0x7fd0d410e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:34:42.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.870+0000 7fd0d8f25700 1 -- 172.21.15.113:0/1465411163 >> 172.21.15.113:0/1465411163 conn(0x7fd0d4070590 msgr2=0x7fd0d41012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:34:42.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.870+0000 7fd0d8f25700 1 -- 172.21.15.113:0/1465411163 shutdown_connections 2024-04-12T05:34:42.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:34:42.870+0000 7fd0d8f25700 1 -- 172.21.15.113:0/1465411163 wait complete. 2024-04-12T05:34:43.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:42 smithi113 ceph-mon[103371]: from='client.35242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:43.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:42 smithi113 ceph-mon[103371]: pgmap v677: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:43.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:42 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/539884763' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:34:43.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:43.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:43.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:43.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:43.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:42 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1465411163' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:34:43.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:42 smithi192 ceph-mon[85304]: from='client.35242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:43.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:42 smithi192 ceph-mon[85304]: pgmap v677: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:43.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:42 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/539884763' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:34:43.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:43.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:43.196 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:43.196 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:34:43.196 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:42 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1465411163' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:34:43.945 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:43 smithi192 ceph-mon[85304]: from='client.35254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:44.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:43 smithi113 ceph-mon[103371]: from='client.35254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:34:45.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:44 smithi113 ceph-mon[103371]: pgmap v678: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:45.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:44 smithi192 ceph-mon[85304]: pgmap v678: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:47.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:46 smithi113 ceph-mon[103371]: pgmap v679: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:47.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:46 smithi192 ceph-mon[85304]: pgmap v679: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:49.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:48 smithi113 ceph-mon[103371]: pgmap v680: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:49.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:48 smithi192 ceph-mon[85304]: pgmap v680: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:51.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:50 smithi113 ceph-mon[103371]: pgmap v681: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:51.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:50 smithi192 ceph-mon[85304]: pgmap v681: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:52.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:51 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:52.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:51 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:34:53.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:52 smithi113 ceph-mon[103371]: pgmap v682: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:53.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:52 smithi192 ceph-mon[85304]: pgmap v682: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:55.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:54 smithi113 ceph-mon[103371]: pgmap v683: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:55.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:54 smithi192 ceph-mon[85304]: pgmap v683: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:57.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:56 smithi113 ceph-mon[103371]: pgmap v684: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:57.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:56 smithi192 ceph-mon[85304]: pgmap v684: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:59.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:34:59 smithi113 ceph-mon[103371]: pgmap v685: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:34:59.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:34:59 smithi192 ceph-mon[85304]: pgmap v685: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:01.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:01 smithi113 ceph-mon[103371]: pgmap v686: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:01.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:01 smithi192 ceph-mon[85304]: pgmap v686: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:03.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:03 smithi113 ceph-mon[103371]: pgmap v687: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:03.446 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:03 smithi192 ceph-mon[85304]: pgmap v687: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:05.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:05 smithi113 ceph-mon[103371]: pgmap v688: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:05.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:05 smithi192 ceph-mon[85304]: pgmap v688: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:07.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:07 smithi113 ceph-mon[103371]: pgmap v689: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:07.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:07.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:07 smithi192 ceph-mon[85304]: pgmap v689: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:07.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:09.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:09 smithi113 ceph-mon[103371]: pgmap v690: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:09.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:09 smithi192 ceph-mon[85304]: pgmap v690: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:11.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:11 smithi113 ceph-mon[103371]: pgmap v691: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:11.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:11 smithi192 ceph-mon[85304]: pgmap v691: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:12.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.980+0000 7fbaae59c700 1 -- 172.21.15.113:0/100088954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa80fe250 msgr2=0x7fbaa80f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:12.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.980+0000 7fbaae59c700 1 --2- 172.21.15.113:0/100088954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa80fe250 0x7fbaa80f7a30 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7fba9c0047f0 tx=0x7fba9c009cb0).stop 2024-04-12T05:35:12.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.980+0000 7fbaae59c700 1 -- 172.21.15.113:0/100088954 shutdown_connections 2024-04-12T05:35:12.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.980+0000 7fbaae59c700 1 --2- 172.21.15.113:0/100088954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa80fe250 0x7fbaa80f7a30 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:12.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.980+0000 7fbaae59c700 1 --2- 172.21.15.113:0/100088954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbaa80fd8f0 0x7fbaa80fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:35:12.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.980+0000 7fbaae59c700 1 -- 172.21.15.113:0/100088954 >> 172.21.15.113:0/100088954 conn(0x7fbaa80f34b0 msgr2=0x7fbaa80f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:12.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.980+0000 7fbaae59c700 1 -- 172.21.15.113:0/100088954 shutdown_connections 2024-04-12T05:35:12.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.980+0000 7fbaae59c700 1 -- 172.21.15.113:0/100088954 wait complete. 2024-04-12T05:35:12.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.981+0000 7fbaae59c700 1 Processor -- start 2024-04-12T05:35:12.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.981+0000 7fbaae59c700 1 -- start start 2024-04-12T05:35:12.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.981+0000 7fbaae59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa80fd8f0 0x7fbaa810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:12.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.981+0000 7fbaae59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbaa80fe250 0x7fbaa810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.981+0000 7fbaae59c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa810ebb0 con 0x7fbaa80fd8f0 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.981+0000 7fbaae59c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa81081d0 con 0x7fbaa80fe250 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.981+0000 7fbaa7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa80fd8f0 0x7fbaa810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.981+0000 7fbaa7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa80fd8f0 0x7fbaa810e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50140/0 (socket says 172.21.15.113:50140) 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.981+0000 7fbaa7fff700 1 -- 172.21.15.113:0/1244957569 learned_addr learned my addr 172.21.15.113:0/1244957569 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.981+0000 7fbaa77fe700 1 --2- 172.21.15.113:0/1244957569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbaa80fe250 0x7fbaa810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.982+0000 7fbaa7fff700 1 -- 172.21.15.113:0/1244957569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbaa80fe250 msgr2=0x7fbaa810e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.982+0000 7fbaa7fff700 1 --2- 172.21.15.113:0/1244957569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbaa80fe250 0x7fbaa810e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.982+0000 7fbaa7fff700 1 -- 172.21.15.113:0/1244957569 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9c005040 con 0x7fbaa80fd8f0 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.982+0000 7fbaa7fff700 1 --2- 172.21.15.113:0/1244957569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa80fd8f0 0x7fbaa810e030 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fba9800b4b0 tx=0x7fba980090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.982+0000 7fbaa57fa700 1 -- 172.21.15.113:0/1244957569 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba98002500 con 0x7fbaa80fd8f0 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.982+0000 7fbaa57fa700 1 -- 172.21.15.113:0/1244957569 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba98002b10 con 0x7fbaa80fd8f0 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.982+0000 7fbaa57fa700 1 -- 172.21.15.113:0/1244957569 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba98012cb0 con 0x7fbaa80fd8f0 2024-04-12T05:35:12.983 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.982+0000 7fbaae59c700 1 -- 172.21.15.113:0/1244957569 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa8108480 con 0x7fbaa80fd8f0 2024-04-12T05:35:12.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.982+0000 7fbaae59c700 1 -- 172.21.15.113:0/1244957569 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa81089c0 con 0x7fbaa80fd8f0 2024-04-12T05:35:12.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.983+0000 7fbaae59c700 1 -- 172.21.15.113:0/1244957569 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa805f590 con 0x7fbaa80fd8f0 2024-04-12T05:35:12.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.983+0000 7fbaa57fa700 1 -- 172.21.15.113:0/1244957569 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fba98002660 con 0x7fbaa80fd8f0 2024-04-12T05:35:12.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.984+0000 7fbaa57fa700 1 --2- 172.21.15.113:0/1244957569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba9006caa0 0x7fba9006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:12.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.984+0000 7fbaa57fa700 1 -- 172.21.15.113:0/1244957569 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fba9808be90 con 0x7fbaa80fd8f0 2024-04-12T05:35:12.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.984+0000 7fbaa77fe700 1 --2- 172.21.15.113:0/1244957569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba9006caa0 0x7fba9006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:12.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.984+0000 7fbaa77fe700 1 --2- 172.21.15.113:0/1244957569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba9006caa0 0x7fba9006ef60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fbaa8109970 tx=0x7fba9c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:12.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:12.988+0000 7fbaa57fa700 1 -- 172.21.15.113:0/1244957569 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fba98059940 con 0x7fbaa80fd8f0 2024-04-12T05:35:13.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.195+0000 7fbaae59c700 1 -- 172.21.15.113:0/1244957569 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbaa8109720 con 0x7fba9006caa0 2024-04-12T05:35:13.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.196+0000 7fbaa57fa700 1 -- 172.21.15.113:0/1244957569 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbaa8109720 con 0x7fba9006caa0 2024-04-12T05:35:13.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.198+0000 7fbaae59c700 1 -- 172.21.15.113:0/1244957569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba9006caa0 msgr2=0x7fba9006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.198+0000 7fbaae59c700 1 --2- 172.21.15.113:0/1244957569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba9006caa0 0x7fba9006ef60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fbaa8109970 tx=0x7fba9c00b040).stop 2024-04-12T05:35:13.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.198+0000 7fbaae59c700 1 -- 172.21.15.113:0/1244957569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa80fd8f0 msgr2=0x7fbaa810e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.198+0000 7fbaae59c700 1 --2- 172.21.15.113:0/1244957569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa80fd8f0 0x7fbaa810e030 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fba9800b4b0 tx=0x7fba980090f0).stop 2024-04-12T05:35:13.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.198+0000 7fbaae59c700 1 -- 172.21.15.113:0/1244957569 shutdown_connections 2024-04-12T05:35:13.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.198+0000 7fbaae59c700 1 --2- 172.21.15.113:0/1244957569 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba9006caa0 0x7fba9006ef60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.198+0000 7fbaae59c700 1 --2- 172.21.15.113:0/1244957569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbaa80fd8f0 0x7fbaa810e030 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.198+0000 7fbaae59c700 1 --2- 172.21.15.113:0/1244957569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbaa80fe250 0x7fbaa810e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.198+0000 7fbaae59c700 1 -- 172.21.15.113:0/1244957569 >> 172.21.15.113:0/1244957569 conn(0x7fbaa80f34b0 msgr2=0x7fbaa80fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:13.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.198+0000 7fbaae59c700 1 -- 172.21.15.113:0/1244957569 shutdown_connections 2024-04-12T05:35:13.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.198+0000 7fbaae59c700 1 -- 172.21.15.113:0/1244957569 wait complete. 2024-04-12T05:35:13.208 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.296+0000 7f9bff1c3700 1 -- 172.21.15.113:0/593329221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bf80f9210 msgr2=0x7f9bf81019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.296+0000 7f9bff1c3700 1 --2- 172.21.15.113:0/593329221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bf80f9210 0x7f9bf81019b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f9be40047f0 tx=0x7f9be4009cb0).stop 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.296+0000 7f9bff1c3700 1 -- 172.21.15.113:0/593329221 shutdown_connections 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.296+0000 7f9bff1c3700 1 --2- 172.21.15.113:0/593329221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bf80f9210 0x7f9bf81019b0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.296+0000 7f9bff1c3700 1 --2- 172.21.15.113:0/593329221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bf80f8940 0x7f9bf80f8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.296+0000 7f9bff1c3700 1 -- 172.21.15.113:0/593329221 >> 172.21.15.113:0/593329221 conn(0x7f9bf8070590 msgr2=0x7f9bf80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.296+0000 7f9bff1c3700 1 -- 172.21.15.113:0/593329221 shutdown_connections 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.296+0000 7f9bff1c3700 1 -- 172.21.15.113:0/593329221 wait complete. 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.297+0000 7f9bff1c3700 1 Processor -- start 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.297+0000 7f9bff1c3700 1 -- start start 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.297+0000 7f9bff1c3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bf80f8940 0x7f9bf810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:13.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.297+0000 7f9bff1c3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bf80f9210 0x7f9bf810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.297+0000 7f9bff1c3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf810eb20 con 0x7f9bf80f8940 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.297+0000 7f9bff1c3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf810ec60 con 0x7f9bf80f9210 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.297+0000 7f9bfcf5f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bf80f8940 0x7f9bf810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.297+0000 7f9bfcf5f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bf80f8940 0x7f9bf810e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50150/0 (socket says 172.21.15.113:50150) 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.297+0000 7f9bfcf5f700 1 -- 172.21.15.113:0/3485276987 learned_addr learned my addr 172.21.15.113:0/3485276987 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.297+0000 7f9bf7fff700 1 --2- 172.21.15.113:0/3485276987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bf80f9210 0x7f9bf810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.298+0000 7f9bfcf5f700 1 -- 172.21.15.113:0/3485276987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bf80f9210 msgr2=0x7f9bf810e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.298+0000 7f9bfcf5f700 1 --2- 172.21.15.113:0/3485276987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bf80f9210 0x7f9bf810e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.298+0000 7f9bfcf5f700 1 -- 172.21.15.113:0/3485276987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4005040 con 0x7f9bf80f8940 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.298+0000 7f9bfcf5f700 1 --2- 172.21.15.113:0/3485276987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bf80f8940 0x7f9bf810e030 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f9bec0041d0 tx=0x7f9bec009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.298+0000 7f9bf5ffb700 1 -- 172.21.15.113:0/3485276987 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bec002d90 con 0x7f9bf80f8940 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.298+0000 7f9bf5ffb700 1 -- 172.21.15.113:0/3485276987 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bec00f040 con 0x7f9bf80f8940 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.298+0000 7f9bff1c3700 1 -- 172.21.15.113:0/3485276987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf81082f0 con 0x7f9bf80f8940 2024-04-12T05:35:13.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.298+0000 7f9bf5ffb700 1 -- 172.21.15.113:0/3485276987 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bec00d800 con 0x7f9bf80f8940 2024-04-12T05:35:13.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.298+0000 7f9bff1c3700 1 -- 172.21.15.113:0/3485276987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf8108830 con 0x7f9bf80f8940 2024-04-12T05:35:13.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.299+0000 7f9bff1c3700 1 -- 172.21.15.113:0/3485276987 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf805f590 con 0x7f9bf80f8940 2024-04-12T05:35:13.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.299+0000 7f9bf5ffb700 1 -- 172.21.15.113:0/3485276987 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9bec00dae0 con 0x7f9bf80f8940 2024-04-12T05:35:13.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.300+0000 7f9bf5ffb700 1 --2- 172.21.15.113:0/3485276987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9be806caa0 0x7f9be806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:13.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.300+0000 7f9bf5ffb700 1 -- 172.21.15.113:0/3485276987 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9bec08be80 con 0x7f9bf80f8940 2024-04-12T05:35:13.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.300+0000 7f9bf7fff700 1 --2- 172.21.15.113:0/3485276987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9be806caa0 0x7f9be806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.300+0000 7f9bf7fff700 1 --2- 172.21.15.113:0/3485276987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9be806caa0 0x7f9be806ef60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f9be4004f10 tx=0x7f9be400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:13.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.304+0000 7f9bf5ffb700 1 -- 172.21.15.113:0/3485276987 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9bec059930 con 0x7f9bf80f8940 2024-04-12T05:35:13.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:13 smithi192 ceph-mon[85304]: pgmap v692: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:13.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.507+0000 7f9bff1c3700 1 -- 172.21.15.113:0/3485276987 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bf8108ab0 con 0x7f9be806caa0 2024-04-12T05:35:13.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.508+0000 7f9bf5ffb700 1 -- 172.21.15.113:0/3485276987 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9bf8108ab0 con 0x7f9be806caa0 2024-04-12T05:35:13.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.510+0000 7f9bff1c3700 1 -- 172.21.15.113:0/3485276987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9be806caa0 msgr2=0x7f9be806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.510+0000 7f9bff1c3700 1 --2- 172.21.15.113:0/3485276987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9be806caa0 0x7f9be806ef60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f9be4004f10 tx=0x7f9be400b040).stop 2024-04-12T05:35:13.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.510+0000 7f9bff1c3700 1 -- 172.21.15.113:0/3485276987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bf80f8940 msgr2=0x7f9bf810e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.510+0000 7f9bff1c3700 1 --2- 172.21.15.113:0/3485276987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bf80f8940 0x7f9bf810e030 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f9bec0041d0 tx=0x7f9bec009f60).stop 2024-04-12T05:35:13.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.510+0000 7f9bff1c3700 1 -- 172.21.15.113:0/3485276987 shutdown_connections 2024-04-12T05:35:13.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.510+0000 7f9bff1c3700 1 --2- 172.21.15.113:0/3485276987 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9be806caa0 0x7f9be806ef60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.510+0000 7f9bff1c3700 1 --2- 172.21.15.113:0/3485276987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bf80f8940 0x7f9bf810e030 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.510+0000 7f9bff1c3700 1 --2- 172.21.15.113:0/3485276987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9bf80f9210 0x7f9bf810e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.510+0000 7f9bff1c3700 1 -- 172.21.15.113:0/3485276987 >> 172.21.15.113:0/3485276987 conn(0x7f9bf8070590 msgr2=0x7f9bf80fc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:13.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.510+0000 7f9bff1c3700 1 -- 172.21.15.113:0/3485276987 shutdown_connections 2024-04-12T05:35:13.510 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.510+0000 7f9bff1c3700 1 -- 172.21.15.113:0/3485276987 wait complete. 2024-04-12T05:35:13.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.608+0000 7f9dd41f6700 1 -- 172.21.15.113:0/1500816822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9dcc0fe210 msgr2=0x7f9dcc0f5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.608+0000 7f9dd41f6700 1 --2- 172.21.15.113:0/1500816822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9dcc0fe210 0x7f9dcc0f5f80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f9dc80047f0 tx=0x7f9dc8009cb0).stop 2024-04-12T05:35:13.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.608+0000 7f9dd41f6700 1 -- 172.21.15.113:0/1500816822 shutdown_connections 2024-04-12T05:35:13.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.608+0000 7f9dd41f6700 1 --2- 172.21.15.113:0/1500816822 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9dcc0fe210 0x7f9dcc0f5f80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.608+0000 7f9dd41f6700 1 --2- 172.21.15.113:0/1500816822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dcc0fd8b0 0x7f9dcc0fdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.608+0000 7f9dd41f6700 1 -- 172.21.15.113:0/1500816822 >> 172.21.15.113:0/1500816822 conn(0x7f9dcc06ce40 msgr2=0x7f9dcc06d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:13.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.608+0000 7f9dd41f6700 1 -- 172.21.15.113:0/1500816822 shutdown_connections 2024-04-12T05:35:13.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.608+0000 7f9dd41f6700 1 -- 172.21.15.113:0/1500816822 wait complete. 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.609+0000 7f9dd41f6700 1 Processor -- start 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.609+0000 7f9dd41f6700 1 -- start start 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.609+0000 7f9dd41f6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dcc0fd8b0 0x7f9dcc10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.609+0000 7f9dd41f6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9dcc0fe210 0x7f9dcc10e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.609+0000 7f9dd41f6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc10eb20 con 0x7f9dcc0fe210 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.609+0000 7f9dd41f6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc10ec60 con 0x7f9dcc0fd8b0 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.609+0000 7f9dd1791700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9dcc0fe210 0x7f9dcc10e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.609+0000 7f9dd1791700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9dcc0fe210 0x7f9dcc10e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50172/0 (socket says 172.21.15.113:50172) 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.609+0000 7f9dd1791700 1 -- 172.21.15.113:0/3124191701 learned_addr learned my addr 172.21.15.113:0/3124191701 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.609+0000 7f9dd1f92700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dcc0fd8b0 0x7f9dcc10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.610+0000 7f9dd1791700 1 -- 172.21.15.113:0/3124191701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dcc0fd8b0 msgr2=0x7f9dcc10e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.610+0000 7f9dd1791700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dcc0fd8b0 0x7f9dcc10e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.610+0000 7f9dd1791700 1 -- 172.21.15.113:0/3124191701 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc8005040 con 0x7f9dcc0fe210 2024-04-12T05:35:13.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.610+0000 7f9dd1f92700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dcc0fd8b0 0x7f9dcc10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:35:13.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.610+0000 7f9dd1791700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9dcc0fe210 0x7f9dcc10e520 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f9dc800b120 tx=0x7f9dc8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:13.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.610+0000 7f9dbeffd700 1 -- 172.21.15.113:0/3124191701 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dc800e030 con 0x7f9dcc0fe210 2024-04-12T05:35:13.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.610+0000 7f9dbeffd700 1 -- 172.21.15.113:0/3124191701 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9dc8007e90 con 0x7f9dcc0fe210 2024-04-12T05:35:13.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.610+0000 7f9dd41f6700 1 -- 172.21.15.113:0/3124191701 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dcc108290 con 0x7f9dcc0fe210 2024-04-12T05:35:13.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.610+0000 7f9dbeffd700 1 -- 172.21.15.113:0/3124191701 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dc8018070 con 0x7f9dcc0fe210 2024-04-12T05:35:13.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.610+0000 7f9dd41f6700 1 -- 172.21.15.113:0/3124191701 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dcc108770 con 0x7f9dcc0fe210 2024-04-12T05:35:13.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.611+0000 7f9dd41f6700 1 -- 172.21.15.113:0/3124191701 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dcc060670 con 0x7f9dcc0fe210 2024-04-12T05:35:13.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.611+0000 7f9dbeffd700 1 -- 172.21.15.113:0/3124191701 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9dc801d420 con 0x7f9dcc0fe210 2024-04-12T05:35:13.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.612+0000 7f9dbeffd700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9db806caa0 0x7f9db806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:13.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.612+0000 7f9dbeffd700 1 -- 172.21.15.113:0/3124191701 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9dc808d300 con 0x7f9dcc0fe210 2024-04-12T05:35:13.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.612+0000 7f9dd1f92700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9db806caa0 0x7f9db806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.613 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.612+0000 7f9dd1f92700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9db806caa0 0x7f9db806ef60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f9dc0007fd0 tx=0x7f9dc0009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:13.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.616+0000 7f9dbeffd700 1 -- 172.21.15.113:0/3124191701 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9dc805adb0 con 0x7f9dcc0fe210 2024-04-12T05:35:13.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:13 smithi113 ceph-mon[103371]: pgmap v692: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:13.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.814+0000 7f9dd41f6700 1 -- 172.21.15.113:0/3124191701 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9dcc109750 con 0x7f9db806caa0 2024-04-12T05:35:13.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.823+0000 7f9dbeffd700 1 -- 172.21.15.113:0/3124191701 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9dcc109750 con 0x7f9db806caa0 2024-04-12T05:35:13.823 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:35:13.823 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (23m) 31s ago 34m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:35:13.823 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (20m) 31s ago 34m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:35:13.823 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (20m) 32s ago 33m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:35:13.823 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (22m) 31s ago 33m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:35:13.824 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (16m) 31s ago 28m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:35:13.824 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (16m) 31s ago 28m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:35:13.824 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (15m) 32s ago 28m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:35:13.824 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (36s) 32s ago 28m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:35:13.824 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (24m) 31s ago 35m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:35:13.824 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (23m) 32s ago 33m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:35:13.824 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (21m) 31s ago 35m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:35:13.824 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (21m) 32s ago 33m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:35:13.824 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (23m) 31s ago 33m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:35:13.824 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (23m) 32s ago 33m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:35:13.825 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (20m) 31s ago 32m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:35:13.825 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (19m) 31s ago 32m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:35:13.825 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (18m) 31s ago 31m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:35:13.825 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (18m) 32s ago 31m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:35:13.825 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (17m) 32s ago 30m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:35:13.825 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (16m) 32s ago 30m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:35:13.825 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (23m) 32s ago 33m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:35:13.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.825+0000 7f9dd41f6700 1 -- 172.21.15.113:0/3124191701 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9db806caa0 msgr2=0x7f9db806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.825+0000 7f9dd41f6700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9db806caa0 0x7f9db806ef60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f9dc0007fd0 tx=0x7f9dc0009040).stop 2024-04-12T05:35:13.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.825+0000 7f9dd41f6700 1 -- 172.21.15.113:0/3124191701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9dcc0fe210 msgr2=0x7f9dcc10e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.825+0000 7f9dd41f6700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9dcc0fe210 0x7f9dcc10e520 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f9dc800b120 tx=0x7f9dc8009cb0).stop 2024-04-12T05:35:13.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.826+0000 7f9dd41f6700 1 -- 172.21.15.113:0/3124191701 shutdown_connections 2024-04-12T05:35:13.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.826+0000 7f9dd41f6700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9db806caa0 0x7f9db806ef60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.826+0000 7f9dd41f6700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9dcc0fd8b0 0x7f9dcc10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.826+0000 7f9dd41f6700 1 --2- 172.21.15.113:0/3124191701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9dcc0fe210 0x7f9dcc10e520 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.826+0000 7f9dd41f6700 1 -- 172.21.15.113:0/3124191701 >> 172.21.15.113:0/3124191701 conn(0x7f9dcc06ce40 msgr2=0x7f9dcc100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:13.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.826+0000 7f9dd41f6700 1 -- 172.21.15.113:0/3124191701 shutdown_connections 2024-04-12T05:35:13.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.826+0000 7f9dd41f6700 1 -- 172.21.15.113:0/3124191701 wait complete. 2024-04-12T05:35:13.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.922+0000 7f6a102e2700 1 -- 172.21.15.113:0/1597616917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a080fc7f0 msgr2=0x7f6a080fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.922+0000 7f6a102e2700 1 --2- 172.21.15.113:0/1597616917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a080fc7f0 0x7f6a080fcbd0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f69f80047f0 tx=0x7f69f8009cb0).stop 2024-04-12T05:35:13.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.922+0000 7f6a102e2700 1 -- 172.21.15.113:0/1597616917 shutdown_connections 2024-04-12T05:35:13.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.922+0000 7f6a102e2700 1 --2- 172.21.15.113:0/1597616917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a080fd150 0x7f6a080f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.922+0000 7f6a102e2700 1 --2- 172.21.15.113:0/1597616917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a080fc7f0 0x7f6a080fcbd0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.922+0000 7f6a102e2700 1 -- 172.21.15.113:0/1597616917 >> 172.21.15.113:0/1597616917 conn(0x7f6a08070590 msgr2=0x7f6a080f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:13.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.922+0000 7f6a102e2700 1 -- 172.21.15.113:0/1597616917 shutdown_connections 2024-04-12T05:35:13.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.922+0000 7f6a102e2700 1 -- 172.21.15.113:0/1597616917 wait complete. 2024-04-12T05:35:13.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a102e2700 1 Processor -- start 2024-04-12T05:35:13.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a102e2700 1 -- start start 2024-04-12T05:35:13.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a102e2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a080fc7f0 0x7f6a08109120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:13.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a102e2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a080fd150 0x7f6a08109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:13.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a102e2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a08109b00 con 0x7f6a080fd150 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a102e2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a08109c40 con 0x7f6a080fc7f0 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a0d87d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a080fd150 0x7f6a08109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a0d87d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a080fd150 0x7f6a08109610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50184/0 (socket says 172.21.15.113:50184) 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a0d87d700 1 -- 172.21.15.113:0/4251890558 learned_addr learned my addr 172.21.15.113:0/4251890558 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a0e07e700 1 --2- 172.21.15.113:0/4251890558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a080fc7f0 0x7f6a08109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a0d87d700 1 -- 172.21.15.113:0/4251890558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a080fc7f0 msgr2=0x7f6a08109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a0d87d700 1 --2- 172.21.15.113:0/4251890558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a080fc7f0 0x7f6a08109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a0d87d700 1 -- 172.21.15.113:0/4251890558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f8005040 con 0x7f6a080fd150 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.923+0000 7f6a0d87d700 1 --2- 172.21.15.113:0/4251890558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a080fd150 0x7f6a08109610 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f6a0400c790 tx=0x7f6a04007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.924+0000 7f69ff779700 1 -- 172.21.15.113:0/4251890558 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a040106f0 con 0x7f6a080fd150 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.924+0000 7f69ff779700 1 -- 172.21.15.113:0/4251890558 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a04010d00 con 0x7f6a080fd150 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.924+0000 7f6a102e2700 1 -- 172.21.15.113:0/4251890558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a08109ef0 con 0x7f6a080fd150 2024-04-12T05:35:13.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.924+0000 7f69ff779700 1 -- 172.21.15.113:0/4251890558 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a04012d50 con 0x7f6a080fd150 2024-04-12T05:35:13.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.924+0000 7f6a102e2700 1 -- 172.21.15.113:0/4251890558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a081124e0 con 0x7f6a080fd150 2024-04-12T05:35:13.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.924+0000 7f6a102e2700 1 -- 172.21.15.113:0/4251890558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0805f590 con 0x7f6a080fd150 2024-04-12T05:35:13.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.925+0000 7f69ff779700 1 -- 172.21.15.113:0/4251890558 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6a04012570 con 0x7f6a080fd150 2024-04-12T05:35:13.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.926+0000 7f69ff779700 1 --2- 172.21.15.113:0/4251890558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f69f406caa0 0x7f69f406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:13.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.926+0000 7f69ff779700 1 -- 172.21.15.113:0/4251890558 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6a0408bed0 con 0x7f6a080fd150 2024-04-12T05:35:13.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.926+0000 7f6a0e07e700 1 --2- 172.21.15.113:0/4251890558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f69f406caa0 0x7f69f406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:13.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.926+0000 7f6a0e07e700 1 --2- 172.21.15.113:0/4251890558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f69f406caa0 0x7f69f406ef60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f69f8000c00 tx=0x7f69f800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:13.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:13.930+0000 7f69ff779700 1 -- 172.21.15.113:0/4251890558 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6a04056030 con 0x7f6a080fd150 2024-04-12T05:35:14.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.196+0000 7f6a102e2700 1 -- 172.21.15.113:0/4251890558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6a080feb80 con 0x7f6a080fd150 2024-04-12T05:35:14.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.196+0000 7f69ff779700 1 -- 172.21.15.113:0/4251890558 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f6a04016290 con 0x7f6a080fd150 2024-04-12T05:35:14.196 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:35:14.196 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:35:14.196 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:35:14.196 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:35:14.196 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:35:14.196 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:35:14.197 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:35:14.197 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:35:14.197 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:35:14.197 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:35:14.197 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:35:14.197 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:35:14.197 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:35:14.197 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:35:14.197 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:35:14.197 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:35:14.197 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:35:14.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.198+0000 7f6a102e2700 1 -- 172.21.15.113:0/4251890558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f69f406caa0 msgr2=0x7f69f406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.198+0000 7f6a102e2700 1 --2- 172.21.15.113:0/4251890558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f69f406caa0 0x7f69f406ef60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f69f8000c00 tx=0x7f69f800b040).stop 2024-04-12T05:35:14.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.198+0000 7f6a102e2700 1 -- 172.21.15.113:0/4251890558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a080fd150 msgr2=0x7f6a08109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.198+0000 7f6a102e2700 1 --2- 172.21.15.113:0/4251890558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a080fd150 0x7f6a08109610 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f6a0400c790 tx=0x7f6a04007f50).stop 2024-04-12T05:35:14.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.199+0000 7f6a102e2700 1 -- 172.21.15.113:0/4251890558 shutdown_connections 2024-04-12T05:35:14.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.199+0000 7f6a102e2700 1 --2- 172.21.15.113:0/4251890558 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f69f406caa0 0x7f69f406ef60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.199+0000 7f6a102e2700 1 --2- 172.21.15.113:0/4251890558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a080fc7f0 0x7f6a08109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.199+0000 7f6a102e2700 1 --2- 172.21.15.113:0/4251890558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a080fd150 0x7f6a08109610 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.199+0000 7f6a102e2700 1 -- 172.21.15.113:0/4251890558 >> 172.21.15.113:0/4251890558 conn(0x7f6a08070590 msgr2=0x7f6a080fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:14.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.199+0000 7f6a102e2700 1 -- 172.21.15.113:0/4251890558 shutdown_connections 2024-04-12T05:35:14.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.199+0000 7f6a102e2700 1 -- 172.21.15.113:0/4251890558 wait complete. 2024-04-12T05:35:14.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.298+0000 7f835090f700 1 -- 172.21.15.113:0/3109495839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f834c0fd150 msgr2=0x7f834c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.298+0000 7f835090f700 1 --2- 172.21.15.113:0/3109495839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f834c0fd150 0x7f834c0f6930 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f833c0047f0 tx=0x7f833c009cb0).stop 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.298+0000 7f835090f700 1 -- 172.21.15.113:0/3109495839 shutdown_connections 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.298+0000 7f835090f700 1 --2- 172.21.15.113:0/3109495839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f834c0fd150 0x7f834c0f6930 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.298+0000 7f835090f700 1 --2- 172.21.15.113:0/3109495839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0fc7f0 0x7f834c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.298+0000 7f835090f700 1 -- 172.21.15.113:0/3109495839 >> 172.21.15.113:0/3109495839 conn(0x7f834c070590 msgr2=0x7f834c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.298+0000 7f835090f700 1 -- 172.21.15.113:0/3109495839 shutdown_connections 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.298+0000 7f835090f700 1 -- 172.21.15.113:0/3109495839 wait complete. 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.298+0000 7f835090f700 1 Processor -- start 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f835090f700 1 -- start start 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f835090f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f834c0fc7f0 0x7f834c1090e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f835090f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0fd150 0x7f834c1095d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f835090f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c109ac0 con 0x7f834c0fc7f0 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f835090f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834c109c00 con 0x7f834c0fd150 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f834a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f834c0fc7f0 0x7f834c1090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:14.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f834a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f834c0fc7f0 0x7f834c1090e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50204/0 (socket says 172.21.15.113:50204) 2024-04-12T05:35:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f834a59c700 1 -- 172.21.15.113:0/1531661909 learned_addr learned my addr 172.21.15.113:0/1531661909 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f8349d9b700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0fd150 0x7f834c1095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f834a59c700 1 -- 172.21.15.113:0/1531661909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0fd150 msgr2=0x7f834c1095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f834a59c700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0fd150 0x7f834c1095d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f834a59c700 1 -- 172.21.15.113:0/1531661909 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c005040 con 0x7f834c0fc7f0 2024-04-12T05:35:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f8349d9b700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0fd150 0x7f834c1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:35:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f834a59c700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f834c0fc7f0 0x7f834c1090e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f83400041d0 tx=0x7f8340009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.299+0000 7f833b7fe700 1 -- 172.21.15.113:0/1531661909 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8340007b40 con 0x7f834c0fc7f0 2024-04-12T05:35:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.300+0000 7f833b7fe700 1 -- 172.21.15.113:0/1531661909 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f834000f040 con 0x7f834c0fc7f0 2024-04-12T05:35:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.300+0000 7f835090f700 1 -- 172.21.15.113:0/1531661909 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f834c109eb0 con 0x7f834c0fc7f0 2024-04-12T05:35:14.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.300+0000 7f833b7fe700 1 -- 172.21.15.113:0/1531661909 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f834000d800 con 0x7f834c0fc7f0 2024-04-12T05:35:14.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.300+0000 7f835090f700 1 -- 172.21.15.113:0/1531661909 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834c1124a0 con 0x7f834c0fc7f0 2024-04-12T05:35:14.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.300+0000 7f835090f700 1 -- 172.21.15.113:0/1531661909 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834c05f590 con 0x7f834c0fc7f0 2024-04-12T05:35:14.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.301+0000 7f833b7fe700 1 -- 172.21.15.113:0/1531661909 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8340007ca0 con 0x7f834c0fc7f0 2024-04-12T05:35:14.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.301+0000 7f833b7fe700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f833406caa0 0x7f833406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:14.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.301+0000 7f833b7fe700 1 -- 172.21.15.113:0/1531661909 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f834008b950 con 0x7f834c0fc7f0 2024-04-12T05:35:14.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.301+0000 7f8349d9b700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f833406caa0 0x7f833406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:14.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.302+0000 7f8349d9b700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f833406caa0 0x7f833406ef60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f833c0043d0 tx=0x7f833c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:14.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.306+0000 7f833b7fe700 1 -- 172.21.15.113:0/1531661909 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8340055970 con 0x7f834c0fc7f0 2024-04-12T05:35:14.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:14 smithi192 ceph-mon[85304]: from='client.35262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:14.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.536+0000 7f835090f700 1 -- 172.21.15.113:0/1531661909 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f834c047540 con 0x7f834c0fc7f0 2024-04-12T05:35:14.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.536+0000 7f833b7fe700 1 -- 172.21.15.113:0/1531661909 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f8340058f90 con 0x7f834c0fc7f0 2024-04-12T05:35:14.536 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:35:14.536 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:35:14.536 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:35:14.536 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:35:14.536 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:35:14.536 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:35:14.536 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:35:14.536 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:35:14.537 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:35:14.538 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:35:14.539 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:35:14.539 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:35:14.539 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:35:14.539 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:35:14.539 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.538+0000 7f835090f700 1 -- 172.21.15.113:0/1531661909 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f833406caa0 msgr2=0x7f833406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.538+0000 7f835090f700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f833406caa0 0x7f833406ef60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f833c0043d0 tx=0x7f833c00b040).stop 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.538+0000 7f835090f700 1 -- 172.21.15.113:0/1531661909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f834c0fc7f0 msgr2=0x7f834c1090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.538+0000 7f835090f700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f834c0fc7f0 0x7f834c1090e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f83400041d0 tx=0x7f8340009f60).stop 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.538+0000 7f835090f700 1 -- 172.21.15.113:0/1531661909 shutdown_connections 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.538+0000 7f835090f700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f833406caa0 0x7f833406ef60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.538+0000 7f835090f700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f834c0fc7f0 0x7f834c1090e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.538+0000 7f835090f700 1 --2- 172.21.15.113:0/1531661909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f834c0fd150 0x7f834c1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.538+0000 7f835090f700 1 -- 172.21.15.113:0/1531661909 >> 172.21.15.113:0/1531661909 conn(0x7f834c070590 msgr2=0x7f834c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.538+0000 7f835090f700 1 -- 172.21.15.113:0/1531661909 shutdown_connections 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.538+0000 7f835090f700 1 -- 172.21.15.113:0/1531661909 wait complete. 2024-04-12T05:35:14.540 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:35:14.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.637+0000 7fd88500c700 1 -- 172.21.15.113:0/572232864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8800fc820 msgr2=0x7fd8800fcc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.637+0000 7fd88500c700 1 --2- 172.21.15.113:0/572232864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8800fc820 0x7fd8800fcc00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7fd8700047f0 tx=0x7fd870009cb0).stop 2024-04-12T05:35:14.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.637+0000 7fd88500c700 1 -- 172.21.15.113:0/572232864 shutdown_connections 2024-04-12T05:35:14.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.637+0000 7fd88500c700 1 --2- 172.21.15.113:0/572232864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8800fd180 0x7fd8800f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.637+0000 7fd88500c700 1 --2- 172.21.15.113:0/572232864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8800fc820 0x7fd8800fcc00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.637+0000 7fd88500c700 1 -- 172.21.15.113:0/572232864 >> 172.21.15.113:0/572232864 conn(0x7fd8800f2400 msgr2=0x7fd8800f4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:14.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.637+0000 7fd88500c700 1 -- 172.21.15.113:0/572232864 shutdown_connections 2024-04-12T05:35:14.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.637+0000 7fd88500c700 1 -- 172.21.15.113:0/572232864 wait complete. 2024-04-12T05:35:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.638+0000 7fd88500c700 1 Processor -- start 2024-04-12T05:35:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.638+0000 7fd88500c700 1 -- start start 2024-04-12T05:35:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.638+0000 7fd88500c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8800fc820 0x7fd8800f8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.638+0000 7fd88500c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8800fd180 0x7fd8800f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.638+0000 7fd88500c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800f8f50 con 0x7fd8800fc820 2024-04-12T05:35:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.638+0000 7fd88500c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800f9090 con 0x7fd8800fd180 2024-04-12T05:35:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.638+0000 7fd87ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8800fc820 0x7fd8800f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:14.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd87ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8800fc820 0x7fd8800f8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50218/0 (socket says 172.21.15.113:50218) 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd87ed9d700 1 -- 172.21.15.113:0/3245775147 learned_addr learned my addr 172.21.15.113:0/3245775147 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd87e59c700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8800fd180 0x7fd8800f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd87ed9d700 1 -- 172.21.15.113:0/3245775147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8800fd180 msgr2=0x7fd8800f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd87ed9d700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8800fd180 0x7fd8800f8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd87ed9d700 1 -- 172.21.15.113:0/3245775147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd870005040 con 0x7fd8800fc820 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd87e59c700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8800fd180 0x7fd8800f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd87ed9d700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8800fc820 0x7fd8800f8570 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7fd87000b120 tx=0x7fd870009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd867fff700 1 -- 172.21.15.113:0/3245775147 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd87000e030 con 0x7fd8800fc820 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd867fff700 1 -- 172.21.15.113:0/3245775147 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd870007e90 con 0x7fd8800fc820 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd88500c700 1 -- 172.21.15.113:0/3245775147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8800f92e0 con 0x7fd8800fc820 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd867fff700 1 -- 172.21.15.113:0/3245775147 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd870018070 con 0x7fd8800fc820 2024-04-12T05:35:14.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.639+0000 7fd88500c700 1 -- 172.21.15.113:0/3245775147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd880114640 con 0x7fd8800fc820 2024-04-12T05:35:14.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.640+0000 7fd88500c700 1 -- 172.21.15.113:0/3245775147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd88005f510 con 0x7fd8800fc820 2024-04-12T05:35:14.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.641+0000 7fd867fff700 1 -- 172.21.15.113:0/3245775147 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd87001d420 con 0x7fd8800fc820 2024-04-12T05:35:14.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.641+0000 7fd867fff700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd86806caa0 0x7fd86806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:14.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.641+0000 7fd867fff700 1 -- 172.21.15.113:0/3245775147 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd87008d300 con 0x7fd8800fc820 2024-04-12T05:35:14.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.641+0000 7fd87e59c700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd86806caa0 0x7fd86806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:14.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.641+0000 7fd87e59c700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd86806caa0 0x7fd86806ef60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fd8800f9d10 tx=0x7fd874009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:14.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.645+0000 7fd867fff700 1 -- 172.21.15.113:0/3245775147 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd870057320 con 0x7fd8800fc820 2024-04-12T05:35:14.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:14 smithi113 ceph-mon[103371]: from='client.35262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:14.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.852+0000 7fd88500c700 1 -- 172.21.15.113:0/3245775147 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8800fb5d0 con 0x7fd86806caa0 2024-04-12T05:35:14.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.853+0000 7fd867fff700 1 -- 172.21.15.113:0/3245775147 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd8800fb5d0 con 0x7fd86806caa0 2024-04-12T05:35:14.853 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:35:14.853 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:35:14.853 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:35:14.854 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:35:14.854 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:35:14.854 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:35:14.854 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:35:14.854 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:35:14.854 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:35:14.854 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:35:14.854 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:35:14.854 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:35:14.854 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:35:14.855 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:35:14.855 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:35:14.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.855+0000 7fd88500c700 1 -- 172.21.15.113:0/3245775147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd86806caa0 msgr2=0x7fd86806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.855+0000 7fd88500c700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd86806caa0 0x7fd86806ef60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fd8800f9d10 tx=0x7fd874009040).stop 2024-04-12T05:35:14.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.855+0000 7fd88500c700 1 -- 172.21.15.113:0/3245775147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8800fc820 msgr2=0x7fd8800f8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.855+0000 7fd88500c700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8800fc820 0x7fd8800f8570 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7fd87000b120 tx=0x7fd870009cb0).stop 2024-04-12T05:35:14.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.855+0000 7fd88500c700 1 -- 172.21.15.113:0/3245775147 shutdown_connections 2024-04-12T05:35:14.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.855+0000 7fd88500c700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd86806caa0 0x7fd86806ef60 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.855+0000 7fd88500c700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8800fc820 0x7fd8800f8570 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.855+0000 7fd88500c700 1 --2- 172.21.15.113:0/3245775147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8800fd180 0x7fd8800f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.856+0000 7fd88500c700 1 -- 172.21.15.113:0/3245775147 >> 172.21.15.113:0/3245775147 conn(0x7fd8800f2400 msgr2=0x7fd880060e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.856+0000 7fd88500c700 1 -- 172.21.15.113:0/3245775147 shutdown_connections 2024-04-12T05:35:14.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.856+0000 7fd88500c700 1 -- 172.21.15.113:0/3245775147 wait complete. 2024-04-12T05:35:14.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.953+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/536283983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e00f9200 msgr2=0x7fa6e0101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.953+0000 7fa6e5bb1700 1 --2- 172.21.15.113:0/536283983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e00f9200 0x7fa6e0101910 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fa6d00047f0 tx=0x7fa6d0009cb0).stop 2024-04-12T05:35:14.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.953+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/536283983 shutdown_connections 2024-04-12T05:35:14.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.953+0000 7fa6e5bb1700 1 --2- 172.21.15.113:0/536283983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e00f9200 0x7fa6e0101910 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.953+0000 7fa6e5bb1700 1 --2- 172.21.15.113:0/536283983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6e00f88a0 0x7fa6e00f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.953+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/536283983 >> 172.21.15.113:0/536283983 conn(0x7fa6e00f2390 msgr2=0x7fa6e00f47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:14.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.953+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/536283983 shutdown_connections 2024-04-12T05:35:14.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.953+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/536283983 wait complete. 2024-04-12T05:35:14.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6e5bb1700 1 Processor -- start 2024-04-12T05:35:14.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6e5bb1700 1 -- start start 2024-04-12T05:35:14.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6e5bb1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6e00f88a0 0x7fa6e010e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:14.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6e5bb1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e00f9200 0x7fa6e010e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:14.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6e5bb1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e010eb50 con 0x7fa6e00f9200 2024-04-12T05:35:14.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6e5bb1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e010ec90 con 0x7fa6e00f88a0 2024-04-12T05:35:14.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e00f9200 0x7fa6e010e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:14.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6deffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e00f9200 0x7fa6e010e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50240/0 (socket says 172.21.15.113:50240) 2024-04-12T05:35:14.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6deffd700 1 -- 172.21.15.113:0/1065011782 learned_addr learned my addr 172.21.15.113:0/1065011782 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:14.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6df7fe700 1 --2- 172.21.15.113:0/1065011782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6e00f88a0 0x7fa6e010e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:14.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6deffd700 1 -- 172.21.15.113:0/1065011782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6e00f88a0 msgr2=0x7fa6e010e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:14.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.954+0000 7fa6deffd700 1 --2- 172.21.15.113:0/1065011782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6e00f88a0 0x7fa6e010e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:14.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.955+0000 7fa6deffd700 1 -- 172.21.15.113:0/1065011782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d0005040 con 0x7fa6e00f9200 2024-04-12T05:35:14.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.955+0000 7fa6deffd700 1 --2- 172.21.15.113:0/1065011782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e00f9200 0x7fa6e010e550 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fa6d0005010 tx=0x7fa6d0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:14.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.955+0000 7fa6dcff9700 1 -- 172.21.15.113:0/1065011782 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d000e030 con 0x7fa6e00f9200 2024-04-12T05:35:14.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.955+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/1065011782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e01082c0 con 0x7fa6e00f9200 2024-04-12T05:35:14.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.955+0000 7fa6dcff9700 1 -- 172.21.15.113:0/1065011782 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6d0007e90 con 0x7fa6e00f9200 2024-04-12T05:35:14.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.955+0000 7fa6dcff9700 1 -- 172.21.15.113:0/1065011782 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d0018070 con 0x7fa6e00f9200 2024-04-12T05:35:14.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.955+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/1065011782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e01087a0 con 0x7fa6e00f9200 2024-04-12T05:35:14.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.956+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/1065011782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e0060390 con 0x7fa6e00f9200 2024-04-12T05:35:14.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.956+0000 7fa6dcff9700 1 -- 172.21.15.113:0/1065011782 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa6d001d6e0 con 0x7fa6e00f9200 2024-04-12T05:35:14.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.957+0000 7fa6dcff9700 1 --2- 172.21.15.113:0/1065011782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa6cc06ca80 0x7fa6cc06ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:14.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.957+0000 7fa6dcff9700 1 -- 172.21.15.113:0/1065011782 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa6d008d1c0 con 0x7fa6e00f9200 2024-04-12T05:35:14.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.957+0000 7fa6df7fe700 1 --2- 172.21.15.113:0/1065011782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa6cc06ca80 0x7fa6cc06ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:14.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.957+0000 7fa6df7fe700 1 --2- 172.21.15.113:0/1065011782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa6cc06ca80 0x7fa6cc06ef40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fa6c8009170 tx=0x7fa6c8009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:14.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:14.961+0000 7fa6dcff9700 1 -- 172.21.15.113:0/1065011782 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa6d005a920 con 0x7fa6e00f9200 2024-04-12T05:35:15.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.233+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/1065011782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa6e0047540 con 0x7fa6e00f9200 2024-04-12T05:35:15.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.234+0000 7fa6dcff9700 1 -- 172.21.15.113:0/1065011782 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fa6d005a740 con 0x7fa6e00f9200 2024-04-12T05:35:15.234 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:35:15.235 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:35:15.236 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:35:15.236 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:35:15.236 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:35:15.236 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:35:15.236 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:35:15.236 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:35:15.236 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:35:15.236 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:35:15.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.236+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/1065011782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa6cc06ca80 msgr2=0x7fa6cc06ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:15.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.236+0000 7fa6e5bb1700 1 --2- 172.21.15.113:0/1065011782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa6cc06ca80 0x7fa6cc06ef40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fa6c8009170 tx=0x7fa6c8009da0).stop 2024-04-12T05:35:15.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.236+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/1065011782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e00f9200 msgr2=0x7fa6e010e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:15.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.236+0000 7fa6e5bb1700 1 --2- 172.21.15.113:0/1065011782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e00f9200 0x7fa6e010e550 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fa6d0005010 tx=0x7fa6d0009cb0).stop 2024-04-12T05:35:15.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.236+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/1065011782 shutdown_connections 2024-04-12T05:35:15.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.236+0000 7fa6e5bb1700 1 --2- 172.21.15.113:0/1065011782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa6cc06ca80 0x7fa6cc06ef40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:15.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.236+0000 7fa6e5bb1700 1 --2- 172.21.15.113:0/1065011782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa6e00f88a0 0x7fa6e010e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:15.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.236+0000 7fa6e5bb1700 1 --2- 172.21.15.113:0/1065011782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6e00f9200 0x7fa6e010e550 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:15.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.236+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/1065011782 >> 172.21.15.113:0/1065011782 conn(0x7fa6e00f2390 msgr2=0x7fa6e00fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:15.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.236+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/1065011782 shutdown_connections 2024-04-12T05:35:15.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:15.236+0000 7fa6e5bb1700 1 -- 172.21.15.113:0/1065011782 wait complete. 2024-04-12T05:35:15.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:15 smithi192 ceph-mon[85304]: from='client.35266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:15 smithi192 ceph-mon[85304]: pgmap v693: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:15 smithi192 ceph-mon[85304]: from='client.35270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:15 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/4251890558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:35:15.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:15 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1531661909' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:35:15.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:15 smithi113 ceph-mon[103371]: from='client.35266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:15.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:15 smithi113 ceph-mon[103371]: pgmap v693: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:15.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:15 smithi113 ceph-mon[103371]: from='client.35270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:15.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:15 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/4251890558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:35:15.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:15 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1531661909' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:35:16.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:16 smithi192 ceph-mon[85304]: from='client.35282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:16.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:16 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1065011782' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:35:16.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:16 smithi113 ceph-mon[103371]: from='client.35282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:16.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:16 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1065011782' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:35:17.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:17 smithi192 ceph-mon[85304]: pgmap v694: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:17.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:17 smithi113 ceph-mon[103371]: pgmap v694: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:19.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:19 smithi192 ceph-mon[85304]: pgmap v695: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:19.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:19 smithi113 ceph-mon[103371]: pgmap v695: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:21.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:21 smithi192 ceph-mon[85304]: pgmap v696: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:21.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:21 smithi113 ceph-mon[103371]: pgmap v696: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:22.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:22.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:23.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:23 smithi113 ceph-mon[103371]: pgmap v697: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:23.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:23 smithi192 ceph-mon[85304]: pgmap v697: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:25.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:25 smithi113 ceph-mon[103371]: pgmap v698: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:25.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:25 smithi192 ceph-mon[85304]: pgmap v698: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:27.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:27 smithi113 ceph-mon[103371]: pgmap v699: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:27.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:27 smithi192 ceph-mon[85304]: pgmap v699: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:29.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:29 smithi113 ceph-mon[103371]: pgmap v700: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:29.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:29 smithi192 ceph-mon[85304]: pgmap v700: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:31.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:31 smithi113 ceph-mon[103371]: pgmap v701: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:31.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:31 smithi192 ceph-mon[85304]: pgmap v701: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:33.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:33 smithi113 ceph-mon[103371]: pgmap v702: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:33.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:33 smithi192 ceph-mon[85304]: pgmap v702: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:35.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:35 smithi113 ceph-mon[103371]: pgmap v703: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:35.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:35 smithi192 ceph-mon[85304]: pgmap v703: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:37.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:37 smithi113 ceph-mon[103371]: pgmap v704: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:37.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:37.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:37 smithi192 ceph-mon[85304]: pgmap v704: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:37.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:39.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:39 smithi113 ceph-mon[103371]: pgmap v705: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:39.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:39 smithi192 ceph-mon[85304]: pgmap v705: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:41.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:41 smithi113 ceph-mon[103371]: pgmap v706: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:41.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:41 smithi192 ceph-mon[85304]: pgmap v706: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:42.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:42 smithi113 ceph-mon[103371]: pgmap v707: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:42.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:42 smithi192 ceph-mon[85304]: pgmap v707: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:45.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:44 smithi113 ceph-mon[103371]: pgmap v708: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:45.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:44 smithi192 ceph-mon[85304]: pgmap v708: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:45.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.348+0000 7f8da68ef700 1 -- 172.21.15.113:0/3106590764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8da00fc7f0 msgr2=0x7f8da00fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:45.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.348+0000 7f8da68ef700 1 --2- 172.21.15.113:0/3106590764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8da00fc7f0 0x7f8da00fcbd0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f8d900047f0 tx=0x7f8d90009cb0).stop 2024-04-12T05:35:45.347 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.348+0000 7f8da68ef700 1 -- 172.21.15.113:0/3106590764 shutdown_connections 2024-04-12T05:35:45.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.348+0000 7f8da68ef700 1 --2- 172.21.15.113:0/3106590764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8da00fd150 0x7f8da00f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.349+0000 7f8da68ef700 1 --2- 172.21.15.113:0/3106590764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8da00fc7f0 0x7f8da00fcbd0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.349+0000 7f8da68ef700 1 -- 172.21.15.113:0/3106590764 >> 172.21.15.113:0/3106590764 conn(0x7f8da0070590 msgr2=0x7f8da00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:45.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.349+0000 7f8da68ef700 1 -- 172.21.15.113:0/3106590764 shutdown_connections 2024-04-12T05:35:45.348 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.349+0000 7f8da68ef700 1 -- 172.21.15.113:0/3106590764 wait complete. 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.350+0000 7f8da68ef700 1 Processor -- start 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.350+0000 7f8da68ef700 1 -- start start 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.350+0000 7f8da68ef700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8da00fc7f0 0x7f8da0109100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.350+0000 7f8da68ef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8da00fd150 0x7f8da01095f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.350+0000 7f8da68ef700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0109ae0 con 0x7f8da00fc7f0 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.350+0000 7f8da68ef700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0109c20 con 0x7f8da00fd150 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.350+0000 7f8d9ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8da00fc7f0 0x7f8da0109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.350+0000 7f8d9ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8da00fc7f0 0x7f8da0109100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43578/0 (socket says 172.21.15.113:43578) 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.350+0000 7f8d9ffff700 1 -- 172.21.15.113:0/3493317018 learned_addr learned my addr 172.21.15.113:0/3493317018 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.350+0000 7f8d9f7fe700 1 --2- 172.21.15.113:0/3493317018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8da00fd150 0x7f8da01095f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.351+0000 7f8d9ffff700 1 -- 172.21.15.113:0/3493317018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8da00fd150 msgr2=0x7f8da01095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.351+0000 7f8d9ffff700 1 --2- 172.21.15.113:0/3493317018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8da00fd150 0x7f8da01095f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.351+0000 7f8d9ffff700 1 -- 172.21.15.113:0/3493317018 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d90005040 con 0x7f8da00fc7f0 2024-04-12T05:35:45.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.351+0000 7f8d9ffff700 1 --2- 172.21.15.113:0/3493317018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8da00fc7f0 0x7f8da0109100 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f8d90005010 tx=0x7f8d90009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:45.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.351+0000 7f8d9d7fa700 1 -- 172.21.15.113:0/3493317018 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d9000e030 con 0x7f8da00fc7f0 2024-04-12T05:35:45.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.351+0000 7f8d9d7fa700 1 -- 172.21.15.113:0/3493317018 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d90007e90 con 0x7f8da00fc7f0 2024-04-12T05:35:45.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.351+0000 7f8da68ef700 1 -- 172.21.15.113:0/3493317018 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da0109e70 con 0x7f8da00fc7f0 2024-04-12T05:35:45.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.351+0000 7f8d9d7fa700 1 -- 172.21.15.113:0/3493317018 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d90018070 con 0x7f8da00fc7f0 2024-04-12T05:35:45.351 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.351+0000 7f8da68ef700 1 -- 172.21.15.113:0/3493317018 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da0112460 con 0x7f8da00fc7f0 2024-04-12T05:35:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.352+0000 7f8da68ef700 1 -- 172.21.15.113:0/3493317018 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da005f590 con 0x7f8da00fc7f0 2024-04-12T05:35:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.353+0000 7f8d9d7fa700 1 -- 172.21.15.113:0/3493317018 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8d9001d670 con 0x7f8da00fc7f0 2024-04-12T05:35:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.353+0000 7f8d9d7fa700 1 --2- 172.21.15.113:0/3493317018 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d8806caa0 0x7f8d8806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.353+0000 7f8d9d7fa700 1 -- 172.21.15.113:0/3493317018 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f8d9008d000 con 0x7f8da00fc7f0 2024-04-12T05:35:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.353+0000 7f8d9f7fe700 1 --2- 172.21.15.113:0/3493317018 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d8806caa0 0x7f8d8806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:45.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.353+0000 7f8d9f7fe700 1 --2- 172.21.15.113:0/3493317018 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d8806caa0 0x7f8d8806ef60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f8da0060f00 tx=0x7f8d9400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:45.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.358+0000 7f8d9d7fa700 1 -- 172.21.15.113:0/3493317018 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8d90057020 con 0x7f8da00fc7f0 2024-04-12T05:35:45.561 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.563+0000 7f8da68ef700 1 -- 172.21.15.113:0/3493317018 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8da0059120 con 0x7f8d8806caa0 2024-04-12T05:35:45.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.564+0000 7f8d9d7fa700 1 -- 172.21.15.113:0/3493317018 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8da0059120 con 0x7f8d8806caa0 2024-04-12T05:35:45.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.567+0000 7f8da68ef700 1 -- 172.21.15.113:0/3493317018 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d8806caa0 msgr2=0x7f8d8806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:45.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.567+0000 7f8da68ef700 1 --2- 172.21.15.113:0/3493317018 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d8806caa0 0x7f8d8806ef60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f8da0060f00 tx=0x7f8d9400b040).stop 2024-04-12T05:35:45.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.567+0000 7f8da68ef700 1 -- 172.21.15.113:0/3493317018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8da00fc7f0 msgr2=0x7f8da0109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:45.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.567+0000 7f8da68ef700 1 --2- 172.21.15.113:0/3493317018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8da00fc7f0 0x7f8da0109100 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f8d90005010 tx=0x7f8d90009cb0).stop 2024-04-12T05:35:45.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.567+0000 7f8da68ef700 1 -- 172.21.15.113:0/3493317018 shutdown_connections 2024-04-12T05:35:45.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.567+0000 7f8da68ef700 1 --2- 172.21.15.113:0/3493317018 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8d8806caa0 0x7f8d8806ef60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.567+0000 7f8da68ef700 1 --2- 172.21.15.113:0/3493317018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8da00fc7f0 0x7f8da0109100 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.567+0000 7f8da68ef700 1 --2- 172.21.15.113:0/3493317018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8da00fd150 0x7f8da01095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.567+0000 7f8da68ef700 1 -- 172.21.15.113:0/3493317018 >> 172.21.15.113:0/3493317018 conn(0x7f8da0070590 msgr2=0x7f8da00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:45.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.567+0000 7f8da68ef700 1 -- 172.21.15.113:0/3493317018 shutdown_connections 2024-04-12T05:35:45.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.567+0000 7f8da68ef700 1 -- 172.21.15.113:0/3493317018 wait complete. 2024-04-12T05:35:45.575 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:35:45.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.666+0000 7fa861294700 1 -- 172.21.15.113:0/424401742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85c0f6fb0 msgr2=0x7fa85c103d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:45.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.666+0000 7fa861294700 1 --2- 172.21.15.113:0/424401742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85c0f6fb0 0x7fa85c103d50 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fa8500047f0 tx=0x7fa850009cb0).stop 2024-04-12T05:35:45.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.666+0000 7fa861294700 1 -- 172.21.15.113:0/424401742 shutdown_connections 2024-04-12T05:35:45.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.666+0000 7fa861294700 1 --2- 172.21.15.113:0/424401742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85c0f6fb0 0x7fa85c103d50 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.666+0000 7fa861294700 1 --2- 172.21.15.113:0/424401742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa85c0f66e0 0x7fa85c0f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.666+0000 7fa861294700 1 -- 172.21.15.113:0/424401742 >> 172.21.15.113:0/424401742 conn(0x7fa85c0f23c0 msgr2=0x7fa85c0f47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:45.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.666+0000 7fa861294700 1 -- 172.21.15.113:0/424401742 shutdown_connections 2024-04-12T05:35:45.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.667+0000 7fa861294700 1 -- 172.21.15.113:0/424401742 wait complete. 2024-04-12T05:35:45.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.667+0000 7fa861294700 1 Processor -- start 2024-04-12T05:35:45.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.667+0000 7fa861294700 1 -- start start 2024-04-12T05:35:45.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.667+0000 7fa861294700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa85c0f66e0 0x7fa85c110350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:45.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.667+0000 7fa861294700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85c0f6fb0 0x7fa85c110840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:45.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.667+0000 7fa861294700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c110e40 con 0x7fa85c0f6fb0 2024-04-12T05:35:45.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.667+0000 7fa861294700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c110f80 con 0x7fa85c0f66e0 2024-04-12T05:35:45.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa85a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85c0f6fb0 0x7fa85c110840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:45.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa85a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85c0f6fb0 0x7fa85c110840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43596/0 (socket says 172.21.15.113:43596) 2024-04-12T05:35:45.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa85a7fc700 1 -- 172.21.15.113:0/913902171 learned_addr learned my addr 172.21.15.113:0/913902171 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:45.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa85affd700 1 --2- 172.21.15.113:0/913902171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa85c0f66e0 0x7fa85c110350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:45.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa85a7fc700 1 -- 172.21.15.113:0/913902171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa85c0f66e0 msgr2=0x7fa85c110350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:45.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa85a7fc700 1 --2- 172.21.15.113:0/913902171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa85c0f66e0 0x7fa85c110350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa85a7fc700 1 -- 172.21.15.113:0/913902171 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa850005040 con 0x7fa85c0f6fb0 2024-04-12T05:35:45.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa85a7fc700 1 --2- 172.21.15.113:0/913902171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85c0f6fb0 0x7fa85c110840 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fa8500047c0 tx=0x7fa850009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:45.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa843fff700 1 -- 172.21.15.113:0/913902171 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa85000e030 con 0x7fa85c0f6fb0 2024-04-12T05:35:45.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa843fff700 1 -- 172.21.15.113:0/913902171 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa850007e90 con 0x7fa85c0f6fb0 2024-04-12T05:35:45.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa843fff700 1 -- 172.21.15.113:0/913902171 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa850018070 con 0x7fa85c0f6fb0 2024-04-12T05:35:45.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa861294700 1 -- 172.21.15.113:0/913902171 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85c10a5b0 con 0x7fa85c0f6fb0 2024-04-12T05:35:45.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.668+0000 7fa861294700 1 -- 172.21.15.113:0/913902171 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85c10aa90 con 0x7fa85c0f6fb0 2024-04-12T05:35:45.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.669+0000 7fa861294700 1 -- 172.21.15.113:0/913902171 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85c060370 con 0x7fa85c0f6fb0 2024-04-12T05:35:45.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.670+0000 7fa843fff700 1 -- 172.21.15.113:0/913902171 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa85001d420 con 0x7fa85c0f6fb0 2024-04-12T05:35:45.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.670+0000 7fa843fff700 1 --2- 172.21.15.113:0/913902171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa84406caa0 0x7fa84406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:45.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.670+0000 7fa843fff700 1 -- 172.21.15.113:0/913902171 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa85008d260 con 0x7fa85c0f6fb0 2024-04-12T05:35:45.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.670+0000 7fa85affd700 1 --2- 172.21.15.113:0/913902171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa84406caa0 0x7fa84406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:45.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.670+0000 7fa85affd700 1 --2- 172.21.15.113:0/913902171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa84406caa0 0x7fa84406ef60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fa84c009170 tx=0x7fa84c009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:45.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.675+0000 7fa843fff700 1 -- 172.21.15.113:0/913902171 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa85005add0 con 0x7fa85c0f6fb0 2024-04-12T05:35:45.880 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.882+0000 7fa861294700 1 -- 172.21.15.113:0/913902171 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa85c10b9d0 con 0x7fa84406caa0 2024-04-12T05:35:45.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.883+0000 7fa843fff700 1 -- 172.21.15.113:0/913902171 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa85c10b9d0 con 0x7fa84406caa0 2024-04-12T05:35:45.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.885+0000 7fa861294700 1 -- 172.21.15.113:0/913902171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa84406caa0 msgr2=0x7fa84406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:45.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.885+0000 7fa861294700 1 --2- 172.21.15.113:0/913902171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa84406caa0 0x7fa84406ef60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fa84c009170 tx=0x7fa84c009da0).stop 2024-04-12T05:35:45.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.885+0000 7fa861294700 1 -- 172.21.15.113:0/913902171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85c0f6fb0 msgr2=0x7fa85c110840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:45.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.885+0000 7fa861294700 1 --2- 172.21.15.113:0/913902171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85c0f6fb0 0x7fa85c110840 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fa8500047c0 tx=0x7fa850009cb0).stop 2024-04-12T05:35:45.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.885+0000 7fa861294700 1 -- 172.21.15.113:0/913902171 shutdown_connections 2024-04-12T05:35:45.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.885+0000 7fa861294700 1 --2- 172.21.15.113:0/913902171 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa84406caa0 0x7fa84406ef60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.885+0000 7fa861294700 1 --2- 172.21.15.113:0/913902171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa85c0f66e0 0x7fa85c110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.885+0000 7fa861294700 1 --2- 172.21.15.113:0/913902171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa85c0f6fb0 0x7fa85c110840 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.885+0000 7fa861294700 1 -- 172.21.15.113:0/913902171 >> 172.21.15.113:0/913902171 conn(0x7fa85c0f23c0 msgr2=0x7fa85c0f40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:45.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.885+0000 7fa861294700 1 -- 172.21.15.113:0/913902171 shutdown_connections 2024-04-12T05:35:45.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.885+0000 7fa861294700 1 -- 172.21.15.113:0/913902171 wait complete. 2024-04-12T05:35:45.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.983+0000 7f7649b72700 1 -- 172.21.15.113:0/533671892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76440f9200 msgr2=0x7f7644101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:45.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.983+0000 7f7649b72700 1 --2- 172.21.15.113:0/533671892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76440f9200 0x7f7644101910 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f76340047f0 tx=0x7f7634009cb0).stop 2024-04-12T05:35:45.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.983+0000 7f7649b72700 1 -- 172.21.15.113:0/533671892 shutdown_connections 2024-04-12T05:35:45.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.983+0000 7f7649b72700 1 --2- 172.21.15.113:0/533671892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76440f9200 0x7f7644101910 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.983+0000 7f7649b72700 1 --2- 172.21.15.113:0/533671892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76440f88a0 0x7f76440f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.983+0000 7f7649b72700 1 -- 172.21.15.113:0/533671892 >> 172.21.15.113:0/533671892 conn(0x7f7644070590 msgr2=0x7f76440f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:45.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.983+0000 7f7649b72700 1 -- 172.21.15.113:0/533671892 shutdown_connections 2024-04-12T05:35:45.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.983+0000 7f7649b72700 1 -- 172.21.15.113:0/533671892 wait complete. 2024-04-12T05:35:45.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.983+0000 7f7649b72700 1 Processor -- start 2024-04-12T05:35:45.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7649b72700 1 -- start start 2024-04-12T05:35:45.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7649b72700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76440f88a0 0x7f764410e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:45.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7649b72700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76440f9200 0x7f764410e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:45.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7649b72700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764410eb70 con 0x7f76440f9200 2024-04-12T05:35:45.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7649b72700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764410ecb0 con 0x7f76440f88a0 2024-04-12T05:35:45.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7642ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76440f9200 0x7f764410e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:45.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7642ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76440f9200 0x7f764410e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43616/0 (socket says 172.21.15.113:43616) 2024-04-12T05:35:45.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7642ffd700 1 -- 172.21.15.113:0/2315467956 learned_addr learned my addr 172.21.15.113:0/2315467956 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:45.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f76437fe700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76440f88a0 0x7f764410e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:45.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7642ffd700 1 -- 172.21.15.113:0/2315467956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76440f88a0 msgr2=0x7f764410e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:45.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7642ffd700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76440f88a0 0x7f764410e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:45.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7642ffd700 1 -- 172.21.15.113:0/2315467956 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7634005040 con 0x7f76440f9200 2024-04-12T05:35:45.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f76437fe700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76440f88a0 0x7f764410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:35:45.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.984+0000 7f7642ffd700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76440f9200 0x7f764410e570 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f763400b120 tx=0x7f7634009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:45.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.985+0000 7f7640ff9700 1 -- 172.21.15.113:0/2315467956 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f763400e030 con 0x7f76440f9200 2024-04-12T05:35:45.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.985+0000 7f7649b72700 1 -- 172.21.15.113:0/2315467956 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76441082e0 con 0x7f76440f9200 2024-04-12T05:35:45.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.985+0000 7f7640ff9700 1 -- 172.21.15.113:0/2315467956 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7634007e90 con 0x7f76440f9200 2024-04-12T05:35:45.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.985+0000 7f7640ff9700 1 -- 172.21.15.113:0/2315467956 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7634018070 con 0x7f76440f9200 2024-04-12T05:35:45.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.985+0000 7f7649b72700 1 -- 172.21.15.113:0/2315467956 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76441087c0 con 0x7f76440f9200 2024-04-12T05:35:45.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.985+0000 7f7649b72700 1 -- 172.21.15.113:0/2315467956 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764405f590 con 0x7f76440f9200 2024-04-12T05:35:45.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.986+0000 7f7640ff9700 1 -- 172.21.15.113:0/2315467956 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f763401d420 con 0x7f76440f9200 2024-04-12T05:35:45.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.986+0000 7f7640ff9700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f763006caa0 0x7f763006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:45.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.987+0000 7f7640ff9700 1 -- 172.21.15.113:0/2315467956 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f763408d320 con 0x7f76440f9200 2024-04-12T05:35:45.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.987+0000 7f76437fe700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f763006caa0 0x7f763006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:45.986 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.987+0000 7f76437fe700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f763006caa0 0x7f763006ef60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f762c004ba0 tx=0x7f762c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:45.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:45.991+0000 7f7640ff9700 1 -- 172.21.15.113:0/2315467956 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f763405add0 con 0x7f76440f9200 2024-04-12T05:35:46.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.188+0000 7f7649b72700 1 -- 172.21.15.113:0/2315467956 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7644109770 con 0x7f763006caa0 2024-04-12T05:35:46.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.197+0000 7f7640ff9700 1 -- 172.21.15.113:0/2315467956 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7644109770 con 0x7f763006caa0 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (23m) 63s ago 34m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (21m) 63s ago 34m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (20m) 64s ago 34m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (23m) 63s ago 34m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (16m) 63s ago 29m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (16m) 63s ago 29m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (16m) 64s ago 29m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (68s) 64s ago 29m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (25m) 63s ago 36m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (24m) 64s ago 34m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:35:46.196 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (22m) 63s ago 36m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:35:46.197 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (21m) 64s ago 34m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:35:46.197 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (24m) 63s ago 34m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:35:46.197 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (24m) 64s ago 33m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:35:46.197 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (20m) 63s ago 33m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:35:46.197 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (20m) 63s ago 32m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:35:46.197 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (19m) 63s ago 32m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:35:46.197 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (18m) 64s ago 31m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:35:46.197 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (18m) 64s ago 31m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:35:46.197 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (17m) 64s ago 31m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:35:46.197 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (23m) 64s ago 33m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:35:46.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.199+0000 7f7649b72700 1 -- 172.21.15.113:0/2315467956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f763006caa0 msgr2=0x7f763006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:46.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.199+0000 7f7649b72700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f763006caa0 0x7f763006ef60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f762c004ba0 tx=0x7f762c009040).stop 2024-04-12T05:35:46.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.199+0000 7f7649b72700 1 -- 172.21.15.113:0/2315467956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76440f9200 msgr2=0x7f764410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:46.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.199+0000 7f7649b72700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76440f9200 0x7f764410e570 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f763400b120 tx=0x7f7634009cb0).stop 2024-04-12T05:35:46.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.199+0000 7f7649b72700 1 -- 172.21.15.113:0/2315467956 shutdown_connections 2024-04-12T05:35:46.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.199+0000 7f7649b72700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f763006caa0 0x7f763006ef60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.199+0000 7f7649b72700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76440f88a0 0x7f764410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.199+0000 7f7649b72700 1 --2- 172.21.15.113:0/2315467956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76440f9200 0x7f764410e570 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.199+0000 7f7649b72700 1 -- 172.21.15.113:0/2315467956 >> 172.21.15.113:0/2315467956 conn(0x7f7644070590 msgr2=0x7f76440fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:46.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.199+0000 7f7649b72700 1 -- 172.21.15.113:0/2315467956 shutdown_connections 2024-04-12T05:35:46.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.199+0000 7f7649b72700 1 -- 172.21.15.113:0/2315467956 wait complete. 2024-04-12T05:35:46.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.295+0000 7f182bcf6700 1 -- 172.21.15.113:0/4012532515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240fc820 msgr2=0x7f18240fcc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:46.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.295+0000 7f182bcf6700 1 --2- 172.21.15.113:0/4012532515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240fc820 0x7f18240fcc00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f18200047f0 tx=0x7f1820009cb0).stop 2024-04-12T05:35:46.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.295+0000 7f182bcf6700 1 -- 172.21.15.113:0/4012532515 shutdown_connections 2024-04-12T05:35:46.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.295+0000 7f182bcf6700 1 --2- 172.21.15.113:0/4012532515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18240fd180 0x7f18240f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.295+0000 7f182bcf6700 1 --2- 172.21.15.113:0/4012532515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240fc820 0x7f18240fcc00 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.295+0000 7f182bcf6700 1 -- 172.21.15.113:0/4012532515 >> 172.21.15.113:0/4012532515 conn(0x7f18240f2400 msgr2=0x7f18240f4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.295+0000 7f182bcf6700 1 -- 172.21.15.113:0/4012532515 shutdown_connections 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.295+0000 7f182bcf6700 1 -- 172.21.15.113:0/4012532515 wait complete. 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.296+0000 7f182bcf6700 1 Processor -- start 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.296+0000 7f182bcf6700 1 -- start start 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.296+0000 7f182bcf6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18240fc820 0x7f182410dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.296+0000 7f182bcf6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240fd180 0x7f182410e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.296+0000 7f182bcf6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182410eae0 con 0x7f18240fd180 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.296+0000 7f182bcf6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182410ec20 con 0x7f18240fc820 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.296+0000 7f1829291700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240fd180 0x7f182410e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.296+0000 7f1829291700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240fd180 0x7f182410e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43632/0 (socket says 172.21.15.113:43632) 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.296+0000 7f1829291700 1 -- 172.21.15.113:0/3848721714 learned_addr learned my addr 172.21.15.113:0/3848721714 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:46.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.296+0000 7f1829a92700 1 --2- 172.21.15.113:0/3848721714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18240fc820 0x7f182410dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.297+0000 7f1829291700 1 -- 172.21.15.113:0/3848721714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18240fc820 msgr2=0x7f182410dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.297+0000 7f1829291700 1 --2- 172.21.15.113:0/3848721714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18240fc820 0x7f182410dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.297+0000 7f1829291700 1 -- 172.21.15.113:0/3848721714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1820005040 con 0x7f18240fd180 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.297+0000 7f1829291700 1 --2- 172.21.15.113:0/3848721714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240fd180 0x7f182410e4e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f1814002780 tx=0x7f1814009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.297+0000 7f181affd700 1 -- 172.21.15.113:0/3848721714 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1814007b40 con 0x7f18240fd180 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.297+0000 7f181affd700 1 -- 172.21.15.113:0/3848721714 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f181400f040 con 0x7f18240fd180 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.297+0000 7f182bcf6700 1 -- 172.21.15.113:0/3848721714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18241082b0 con 0x7f18240fd180 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.297+0000 7f181affd700 1 -- 172.21.15.113:0/3848721714 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1814014780 con 0x7f18240fd180 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.297+0000 7f182bcf6700 1 -- 172.21.15.113:0/3848721714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1824108870 con 0x7f18240fd180 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.298+0000 7f182bcf6700 1 -- 172.21.15.113:0/3848721714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182405f510 con 0x7f18240fd180 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.298+0000 7f181affd700 1 -- 172.21.15.113:0/3848721714 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1814007ca0 con 0x7f18240fd180 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.299+0000 7f181affd700 1 --2- 172.21.15.113:0/3848721714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f181006caa0 0x7f181006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.299+0000 7f181affd700 1 -- 172.21.15.113:0/3848721714 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f181408bd50 con 0x7f18240fd180 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.299+0000 7f1829a92700 1 --2- 172.21.15.113:0/3848721714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f181006caa0 0x7f181006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:46.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.299+0000 7f1829a92700 1 --2- 172.21.15.113:0/3848721714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f181006caa0 0x7f181006ef60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f1820004f10 tx=0x7f182000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:46.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.303+0000 7f181affd700 1 -- 172.21.15.113:0/3848721714 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1814059800 con 0x7f18240fd180 2024-04-12T05:35:46.565 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.566+0000 7f182bcf6700 1 -- 172.21.15.113:0/3848721714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1824047540 con 0x7f18240fd180 2024-04-12T05:35:46.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.567+0000 7f181affd700 1 -- 172.21.15.113:0/3848721714 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f1814059390 con 0x7f18240fd180 2024-04-12T05:35:46.566 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:35:46.566 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:35:46.566 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:35:46.566 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:35:46.566 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:35:46.566 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:35:46.566 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:35:46.566 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:35:46.566 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:35:46.567 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:35:46.567 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:35:46.567 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:35:46.567 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:35:46.567 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:35:46.567 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:35:46.567 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:35:46.567 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:35:46.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.569+0000 7f182bcf6700 1 -- 172.21.15.113:0/3848721714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f181006caa0 msgr2=0x7f181006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:46.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.569+0000 7f182bcf6700 1 --2- 172.21.15.113:0/3848721714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f181006caa0 0x7f181006ef60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f1820004f10 tx=0x7f182000b040).stop 2024-04-12T05:35:46.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.569+0000 7f182bcf6700 1 -- 172.21.15.113:0/3848721714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240fd180 msgr2=0x7f182410e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:46.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.569+0000 7f182bcf6700 1 --2- 172.21.15.113:0/3848721714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240fd180 0x7f182410e4e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f1814002780 tx=0x7f1814009ee0).stop 2024-04-12T05:35:46.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.569+0000 7f182bcf6700 1 -- 172.21.15.113:0/3848721714 shutdown_connections 2024-04-12T05:35:46.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.569+0000 7f182bcf6700 1 --2- 172.21.15.113:0/3848721714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f181006caa0 0x7f181006ef60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.569+0000 7f182bcf6700 1 --2- 172.21.15.113:0/3848721714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18240fc820 0x7f182410dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.569+0000 7f182bcf6700 1 --2- 172.21.15.113:0/3848721714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240fd180 0x7f182410e4e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.569+0000 7f182bcf6700 1 -- 172.21.15.113:0/3848721714 >> 172.21.15.113:0/3848721714 conn(0x7f18240f2400 msgr2=0x7f1824060e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:46.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.569+0000 7f182bcf6700 1 -- 172.21.15.113:0/3848721714 shutdown_connections 2024-04-12T05:35:46.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.569+0000 7f182bcf6700 1 -- 172.21.15.113:0/3848721714 wait complete. 2024-04-12T05:35:46.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.668+0000 7f0f91594700 1 -- 172.21.15.113:0/1703685349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0fc890 msgr2=0x7f0f8c0fcc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:46.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.668+0000 7f0f91594700 1 --2- 172.21.15.113:0/1703685349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0fc890 0x7f0f8c0fcc70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f0f800047f0 tx=0x7f0f80009cb0).stop 2024-04-12T05:35:46.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.668+0000 7f0f91594700 1 -- 172.21.15.113:0/1703685349 shutdown_connections 2024-04-12T05:35:46.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.668+0000 7f0f91594700 1 --2- 172.21.15.113:0/1703685349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0fd1f0 0x7f0f8c0f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.668+0000 7f0f91594700 1 --2- 172.21.15.113:0/1703685349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0fc890 0x7f0f8c0fcc70 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.668+0000 7f0f91594700 1 -- 172.21.15.113:0/1703685349 >> 172.21.15.113:0/1703685349 conn(0x7f0f8c0f2490 msgr2=0x7f0f8c0f48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:46.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.668+0000 7f0f91594700 1 -- 172.21.15.113:0/1703685349 shutdown_connections 2024-04-12T05:35:46.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.668+0000 7f0f91594700 1 -- 172.21.15.113:0/1703685349 wait complete. 2024-04-12T05:35:46.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.668+0000 7f0f91594700 1 Processor -- start 2024-04-12T05:35:46.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f91594700 1 -- start start 2024-04-12T05:35:46.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f91594700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0fc890 0x7f0f8c10d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:46.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f91594700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0fd1f0 0x7f0f8c10d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:46.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f91594700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c10db40 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f91594700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c10dc80 con 0x7f0f8c0fc890 2024-04-12T05:35:46.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f8a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0fd1f0 0x7f0f8c10d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:46.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f8a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0fd1f0 0x7f0f8c10d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43646/0 (socket says 172.21.15.113:43646) 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f8a7fc700 1 -- 172.21.15.113:0/1060963026 learned_addr learned my addr 172.21.15.113:0/1060963026 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f8affd700 1 --2- 172.21.15.113:0/1060963026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0fc890 0x7f0f8c10d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f8a7fc700 1 -- 172.21.15.113:0/1060963026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0fc890 msgr2=0x7f0f8c10d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f8a7fc700 1 --2- 172.21.15.113:0/1060963026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0fc890 0x7f0f8c10d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f8a7fc700 1 -- 172.21.15.113:0/1060963026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f80005040 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f8a7fc700 1 --2- 172.21.15.113:0/1060963026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0fd1f0 0x7f0f8c10d540 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f0f7c00e940 tx=0x7f0f7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f73fff700 1 -- 172.21.15.113:0/1060963026 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f7c007b40 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f73fff700 1 -- 172.21.15.113:0/1060963026 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f7c00f040 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f91594700 1 -- 172.21.15.113:0/1060963026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8c1072b0 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.669+0000 7f0f73fff700 1 -- 172.21.15.113:0/1060963026 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f7c0148e0 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.670+0000 7f0f91594700 1 -- 172.21.15.113:0/1060963026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8c107870 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.670+0000 7f0f91594700 1 -- 172.21.15.113:0/1060963026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8c060330 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.671+0000 7f0f73fff700 1 -- 172.21.15.113:0/1060963026 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0f7c007ca0 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.671+0000 7f0f73fff700 1 --2- 172.21.15.113:0/1060963026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 0x7f0f7406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:46.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.671+0000 7f0f73fff700 1 -- 172.21.15.113:0/1060963026 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0f7c08b860 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.671+0000 7f0f8affd700 1 --2- 172.21.15.113:0/1060963026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 0x7f0f7406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:46.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.672+0000 7f0f8affd700 1 --2- 172.21.15.113:0/1060963026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 0x7f0f7406ef60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f0f800047c0 tx=0x7f0f80009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:46.674 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.676+0000 7f0f73fff700 1 -- 172.21.15.113:0/1060963026 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0f7c059310 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.906+0000 7f0f91594700 1 -- 172.21.15.113:0/1060963026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0f8c0febe0 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.907+0000 7f0f73fff700 1 -- 172.21.15.113:0/1060963026 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f0f7c058ea0 con 0x7f0f8c0fd1f0 2024-04-12T05:35:46.905 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:35:46.905 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:35:46.905 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:35:46.905 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:35:46.906 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:35:46.907 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:35:46.908 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:35:46.908 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:35:46.908 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:35:46.908 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:35:46.908 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:35:46.908 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:35:46.908 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:35:46.908 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:35:46.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.909+0000 7f0f91594700 1 -- 172.21.15.113:0/1060963026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 msgr2=0x7f0f7406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:46.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.909+0000 7f0f91594700 1 --2- 172.21.15.113:0/1060963026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 0x7f0f7406ef60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f0f800047c0 tx=0x7f0f80009cb0).stop 2024-04-12T05:35:46.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.909+0000 7f0f91594700 1 -- 172.21.15.113:0/1060963026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0fd1f0 msgr2=0x7f0f8c10d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:46.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.909+0000 7f0f91594700 1 --2- 172.21.15.113:0/1060963026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0fd1f0 0x7f0f8c10d540 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f0f7c00e940 tx=0x7f0f7c00b040).stop 2024-04-12T05:35:46.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.909+0000 7f0f91594700 1 -- 172.21.15.113:0/1060963026 shutdown_connections 2024-04-12T05:35:46.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.909+0000 7f0f91594700 1 --2- 172.21.15.113:0/1060963026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0f7406caa0 0x7f0f7406ef60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.909+0000 7f0f91594700 1 --2- 172.21.15.113:0/1060963026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0fc890 0x7f0f8c10d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.909+0000 7f0f91594700 1 --2- 172.21.15.113:0/1060963026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f8c0fd1f0 0x7f0f8c10d540 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:46.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.909+0000 7f0f91594700 1 -- 172.21.15.113:0/1060963026 >> 172.21.15.113:0/1060963026 conn(0x7f0f8c0f2490 msgr2=0x7f0f8c0fac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:46.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.909+0000 7f0f91594700 1 -- 172.21.15.113:0/1060963026 shutdown_connections 2024-04-12T05:35:46.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:46.909+0000 7f0f91594700 1 -- 172.21.15.113:0/1060963026 wait complete. 2024-04-12T05:35:46.910 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:35:47.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.009+0000 7f850174d700 1 -- 172.21.15.113:0/4288819776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84fc0f66f0 msgr2=0x7f84fc0f6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:47.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.009+0000 7f850174d700 1 --2- 172.21.15.113:0/4288819776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84fc0f66f0 0x7f84fc0f6ad0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f84f0004820 tx=0x7f84f0009cb0).stop 2024-04-12T05:35:47.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.009+0000 7f850174d700 1 -- 172.21.15.113:0/4288819776 shutdown_connections 2024-04-12T05:35:47.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.009+0000 7f850174d700 1 --2- 172.21.15.113:0/4288819776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84fc0f6fc0 0x7f84fc103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.009+0000 7f850174d700 1 --2- 172.21.15.113:0/4288819776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84fc0f66f0 0x7f84fc0f6ad0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.009+0000 7f850174d700 1 -- 172.21.15.113:0/4288819776 >> 172.21.15.113:0/4288819776 conn(0x7f84fc070590 msgr2=0x7f84fc0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:47.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.009+0000 7f850174d700 1 -- 172.21.15.113:0/4288819776 shutdown_connections 2024-04-12T05:35:47.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.009+0000 7f850174d700 1 -- 172.21.15.113:0/4288819776 wait complete. 2024-04-12T05:35:47.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.010+0000 7f850174d700 1 Processor -- start 2024-04-12T05:35:47.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.010+0000 7f850174d700 1 -- start start 2024-04-12T05:35:47.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.010+0000 7f850174d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84fc0f66f0 0x7f84fc1102b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:47.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.010+0000 7f850174d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84fc0f6fc0 0x7f84fc1107a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:47.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.010+0000 7f850174d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc110da0 con 0x7f84fc0f6fc0 2024-04-12T05:35:47.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.010+0000 7f850174d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc110ee0 con 0x7f84fc0f66f0 2024-04-12T05:35:47.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.010+0000 7f84fa7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84fc0f6fc0 0x7f84fc1107a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:47.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.010+0000 7f84fa7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84fc0f6fc0 0x7f84fc1107a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43676/0 (socket says 172.21.15.113:43676) 2024-04-12T05:35:47.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.010+0000 7f84fa7fc700 1 -- 172.21.15.113:0/2892296263 learned_addr learned my addr 172.21.15.113:0/2892296263 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:47.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.010+0000 7f84faffd700 1 --2- 172.21.15.113:0/2892296263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84fc0f66f0 0x7f84fc1102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:47.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.011+0000 7f84fa7fc700 1 -- 172.21.15.113:0/2892296263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84fc0f66f0 msgr2=0x7f84fc1102b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:47.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.011+0000 7f84fa7fc700 1 --2- 172.21.15.113:0/2892296263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84fc0f66f0 0x7f84fc1102b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.011+0000 7f84fa7fc700 1 -- 172.21.15.113:0/2892296263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f0005040 con 0x7f84fc0f6fc0 2024-04-12T05:35:47.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.011+0000 7f84fa7fc700 1 --2- 172.21.15.113:0/2892296263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84fc0f6fc0 0x7f84fc1107a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f84ec002770 tx=0x7f84ec002a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:47.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.011+0000 7f84e3fff700 1 -- 172.21.15.113:0/2892296263 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84ec007b40 con 0x7f84fc0f6fc0 2024-04-12T05:35:47.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.011+0000 7f850174d700 1 -- 172.21.15.113:0/2892296263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84fc10a570 con 0x7f84fc0f6fc0 2024-04-12T05:35:47.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.011+0000 7f84e3fff700 1 -- 172.21.15.113:0/2892296263 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84ec00f040 con 0x7f84fc0f6fc0 2024-04-12T05:35:47.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.011+0000 7f84e3fff700 1 -- 172.21.15.113:0/2892296263 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84ec014820 con 0x7f84fc0f6fc0 2024-04-12T05:35:47.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.011+0000 7f850174d700 1 -- 172.21.15.113:0/2892296263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84fc10aab0 con 0x7f84fc0f6fc0 2024-04-12T05:35:47.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.012+0000 7f850174d700 1 -- 172.21.15.113:0/2892296263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84fc05f590 con 0x7f84fc0f6fc0 2024-04-12T05:35:47.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.012+0000 7f84e3fff700 1 -- 172.21.15.113:0/2892296263 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f84ec007ca0 con 0x7f84fc0f6fc0 2024-04-12T05:35:47.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.013+0000 7f84e3fff700 1 --2- 172.21.15.113:0/2892296263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84e406caa0 0x7f84e406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:47.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.013+0000 7f84e3fff700 1 -- 172.21.15.113:0/2892296263 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f84ec08ba40 con 0x7f84fc0f6fc0 2024-04-12T05:35:47.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.013+0000 7f84faffd700 1 --2- 172.21.15.113:0/2892296263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84e406caa0 0x7f84e406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:47.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.013+0000 7f84faffd700 1 --2- 172.21.15.113:0/2892296263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84e406caa0 0x7f84e406ef60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f84f0005010 tx=0x7f84f000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:47.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.017+0000 7f84e3fff700 1 -- 172.21.15.113:0/2892296263 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f84ec0594f0 con 0x7f84fc0f6fc0 2024-04-12T05:35:47.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:46 smithi113 ceph-mon[103371]: from='client.35290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:47.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:46 smithi113 ceph-mon[103371]: pgmap v709: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:47.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:46 smithi113 ceph-mon[103371]: from='client.35294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:47.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:46 smithi113 ceph-mon[103371]: from='client.35298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:47.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:46 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3848721714' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:35:47.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:46 smithi192 ceph-mon[85304]: from='client.35290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:47.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:46 smithi192 ceph-mon[85304]: pgmap v709: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:47.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:46 smithi192 ceph-mon[85304]: from='client.35294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:47.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:46 smithi192 ceph-mon[85304]: from='client.35298 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:47.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:46 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3848721714' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:35:47.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.225+0000 7f850174d700 1 -- 172.21.15.113:0/2892296263 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f84fc10b9a0 con 0x7f84e406caa0 2024-04-12T05:35:47.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.226+0000 7f84e3fff700 1 -- 172.21.15.113:0/2892296263 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f84fc10b9a0 con 0x7f84e406caa0 2024-04-12T05:35:47.224 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:35:47.224 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:35:47.225 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:35:47.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.228+0000 7f850174d700 1 -- 172.21.15.113:0/2892296263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84e406caa0 msgr2=0x7f84e406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:47.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.228+0000 7f850174d700 1 --2- 172.21.15.113:0/2892296263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84e406caa0 0x7f84e406ef60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f84f0005010 tx=0x7f84f000b040).stop 2024-04-12T05:35:47.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.228+0000 7f850174d700 1 -- 172.21.15.113:0/2892296263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84fc0f6fc0 msgr2=0x7f84fc1107a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:47.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.228+0000 7f850174d700 1 --2- 172.21.15.113:0/2892296263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84fc0f6fc0 0x7f84fc1107a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f84ec002770 tx=0x7f84ec002a80).stop 2024-04-12T05:35:47.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.228+0000 7f850174d700 1 -- 172.21.15.113:0/2892296263 shutdown_connections 2024-04-12T05:35:47.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.228+0000 7f850174d700 1 --2- 172.21.15.113:0/2892296263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84e406caa0 0x7f84e406ef60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.228+0000 7f850174d700 1 --2- 172.21.15.113:0/2892296263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84fc0f66f0 0x7f84fc1102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.228+0000 7f850174d700 1 --2- 172.21.15.113:0/2892296263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f84fc0f6fc0 0x7f84fc1107a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.228+0000 7f850174d700 1 -- 172.21.15.113:0/2892296263 >> 172.21.15.113:0/2892296263 conn(0x7f84fc070590 msgr2=0x7f84fc06d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:47.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.228+0000 7f850174d700 1 -- 172.21.15.113:0/2892296263 shutdown_connections 2024-04-12T05:35:47.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.228+0000 7f850174d700 1 -- 172.21.15.113:0/2892296263 wait complete. 2024-04-12T05:35:47.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.327+0000 7fad1ad91700 1 -- 172.21.15.113:0/1236139213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad140fd150 msgr2=0x7fad140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.327+0000 7fad1ad91700 1 --2- 172.21.15.113:0/1236139213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad140fd150 0x7fad140f6930 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7fad000047f0 tx=0x7fad00009cb0).stop 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 -- 172.21.15.113:0/1236139213 shutdown_connections 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 --2- 172.21.15.113:0/1236139213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad140fd150 0x7fad140f6930 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 --2- 172.21.15.113:0/1236139213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad140fc7f0 0x7fad140fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 -- 172.21.15.113:0/1236139213 >> 172.21.15.113:0/1236139213 conn(0x7fad140f23b0 msgr2=0x7fad140f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 -- 172.21.15.113:0/1236139213 shutdown_connections 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 -- 172.21.15.113:0/1236139213 wait complete. 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 Processor -- start 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 -- start start 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad140fc7f0 0x7fad1410cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad140fd150 0x7fad1410d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1410da00 con 0x7fad140fd150 2024-04-12T05:35:47.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.328+0000 7fad1ad91700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1410db40 con 0x7fad140fc7f0 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad13fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad140fd150 0x7fad1410d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad13fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad140fd150 0x7fad1410d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43694/0 (socket says 172.21.15.113:43694) 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad13fff700 1 -- 172.21.15.113:0/3304447333 learned_addr learned my addr 172.21.15.113:0/3304447333 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad18b2d700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad140fc7f0 0x7fad1410cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad13fff700 1 -- 172.21.15.113:0/3304447333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad140fc7f0 msgr2=0x7fad1410cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad13fff700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad140fc7f0 0x7fad1410cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad13fff700 1 -- 172.21.15.113:0/3304447333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad00005040 con 0x7fad140fd150 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad18b2d700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad140fc7f0 0x7fad1410cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad13fff700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad140fd150 0x7fad1410d400 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fad000099f0 tx=0x7fad00009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad11ffb700 1 -- 172.21.15.113:0/3304447333 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad0000e030 con 0x7fad140fd150 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad11ffb700 1 -- 172.21.15.113:0/3304447333 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad00007e90 con 0x7fad140fd150 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad1ad91700 1 -- 172.21.15.113:0/3304447333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad14107170 con 0x7fad140fd150 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad11ffb700 1 -- 172.21.15.113:0/3304447333 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad00018070 con 0x7fad140fd150 2024-04-12T05:35:47.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.329+0000 7fad1ad91700 1 -- 172.21.15.113:0/3304447333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad141076d0 con 0x7fad140fd150 2024-04-12T05:35:47.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.330+0000 7fad1ad91700 1 -- 172.21.15.113:0/3304447333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad14060390 con 0x7fad140fd150 2024-04-12T05:35:47.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.331+0000 7fad11ffb700 1 -- 172.21.15.113:0/3304447333 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fad00009590 con 0x7fad140fd150 2024-04-12T05:35:47.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.331+0000 7fad11ffb700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad0406ca30 0x7fad0406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:35:47.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.331+0000 7fad11ffb700 1 -- 172.21.15.113:0/3304447333 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fad0008d0d0 con 0x7fad140fd150 2024-04-12T05:35:47.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.331+0000 7fad18b2d700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad0406ca30 0x7fad0406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:35:47.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.332+0000 7fad18b2d700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad0406ca30 0x7fad0406eef0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fad08004c80 tx=0x7fad0800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:35:47.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.336+0000 7fad11ffb700 1 -- 172.21.15.113:0/3304447333 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fad0005ab80 con 0x7fad140fd150 2024-04-12T05:35:47.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.599+0000 7fad1ad91700 1 -- 172.21.15.113:0/3304447333 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fad14047540 con 0x7fad140fd150 2024-04-12T05:35:47.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.600+0000 7fad11ffb700 1 -- 172.21.15.113:0/3304447333 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fad0005a710 con 0x7fad140fd150 2024-04-12T05:35:47.598 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:35:47.598 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:35:47.599 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:35:47.600 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:35:47.600 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:35:47.600 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:35:47.600 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:35:47.600 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:35:47.600 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:35:47.600 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:35:47.600 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:35:47.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.602+0000 7fad1ad91700 1 -- 172.21.15.113:0/3304447333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad0406ca30 msgr2=0x7fad0406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:47.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.602+0000 7fad1ad91700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad0406ca30 0x7fad0406eef0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fad08004c80 tx=0x7fad0800b040).stop 2024-04-12T05:35:47.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.602+0000 7fad1ad91700 1 -- 172.21.15.113:0/3304447333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad140fd150 msgr2=0x7fad1410d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:35:47.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.602+0000 7fad1ad91700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad140fd150 0x7fad1410d400 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fad000099f0 tx=0x7fad00009cb0).stop 2024-04-12T05:35:47.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.602+0000 7fad1ad91700 1 -- 172.21.15.113:0/3304447333 shutdown_connections 2024-04-12T05:35:47.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.602+0000 7fad1ad91700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad0406ca30 0x7fad0406eef0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.602+0000 7fad1ad91700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad140fc7f0 0x7fad1410cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.602+0000 7fad1ad91700 1 --2- 172.21.15.113:0/3304447333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad140fd150 0x7fad1410d400 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:35:47.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.602+0000 7fad1ad91700 1 -- 172.21.15.113:0/3304447333 >> 172.21.15.113:0/3304447333 conn(0x7fad140f23b0 msgr2=0x7fad140fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:35:47.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.602+0000 7fad1ad91700 1 -- 172.21.15.113:0/3304447333 shutdown_connections 2024-04-12T05:35:47.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:35:47.602+0000 7fad1ad91700 1 -- 172.21.15.113:0/3304447333 wait complete. 2024-04-12T05:35:48.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:47 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1060963026' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:35:48.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:47 smithi113 ceph-mon[103371]: from='client.35310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:48.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:47 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3304447333' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:35:48.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:47 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1060963026' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:35:48.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:47 smithi192 ceph-mon[85304]: from='client.35310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:35:48.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:47 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3304447333' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:35:49.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:48 smithi113 ceph-mon[103371]: pgmap v710: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:49.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:48 smithi192 ceph-mon[85304]: pgmap v710: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:51.171 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:50 smithi113 ceph-mon[103371]: pgmap v711: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:51.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:50 smithi192 ceph-mon[85304]: pgmap v711: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:52.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:51 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:52.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:51 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:35:53.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:52 smithi113 ceph-mon[103371]: pgmap v712: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:53.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:52 smithi192 ceph-mon[85304]: pgmap v712: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:55.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:54 smithi113 ceph-mon[103371]: pgmap v713: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:55.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:54 smithi192 ceph-mon[85304]: pgmap v713: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:57.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:56 smithi113 ceph-mon[103371]: pgmap v714: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:57.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:56 smithi192 ceph-mon[85304]: pgmap v714: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:59.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:35:58 smithi113 ceph-mon[103371]: pgmap v715: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:35:59.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:35:58 smithi192 ceph-mon[85304]: pgmap v715: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:01.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:00 smithi113 ceph-mon[103371]: pgmap v716: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:01.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:00 smithi192 ceph-mon[85304]: pgmap v716: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:03.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:02 smithi113 ceph-mon[103371]: pgmap v717: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:03.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:02 smithi192 ceph-mon[85304]: pgmap v717: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:05.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:04 smithi113 ceph-mon[103371]: pgmap v718: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:05.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:04 smithi192 ceph-mon[85304]: pgmap v718: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:07.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:06 smithi113 ceph-mon[103371]: pgmap v719: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:07.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:07.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:06 smithi192 ceph-mon[85304]: pgmap v719: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:07.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:09.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:08 smithi113 ceph-mon[103371]: pgmap v720: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:09.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:08 smithi192 ceph-mon[85304]: pgmap v720: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:11.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:10 smithi113 ceph-mon[103371]: pgmap v721: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:11.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:10 smithi192 ceph-mon[85304]: pgmap v721: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:13.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:12 smithi113 ceph-mon[103371]: pgmap v722: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:13.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:12 smithi192 ceph-mon[85304]: pgmap v722: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:15.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:14 smithi113 ceph-mon[103371]: pgmap v723: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:15.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:14 smithi192 ceph-mon[85304]: pgmap v723: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:17.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:16 smithi113 ceph-mon[103371]: pgmap v724: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:17.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:16 smithi192 ceph-mon[85304]: pgmap v724: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:17.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.713+0000 7fa520393700 1 -- 172.21.15.113:0/3280872336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5180fd150 msgr2=0x7fa5180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:17.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.713+0000 7fa520393700 1 --2- 172.21.15.113:0/3280872336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5180fd150 0x7fa5180f6930 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fa5140047f0 tx=0x7fa514009cb0).stop 2024-04-12T05:36:17.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.713+0000 7fa520393700 1 -- 172.21.15.113:0/3280872336 shutdown_connections 2024-04-12T05:36:17.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.713+0000 7fa520393700 1 --2- 172.21.15.113:0/3280872336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5180fd150 0x7fa5180f6930 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:17.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.713+0000 7fa520393700 1 --2- 172.21.15.113:0/3280872336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5180fc7f0 0x7fa5180fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:17.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.713+0000 7fa520393700 1 -- 172.21.15.113:0/3280872336 >> 172.21.15.113:0/3280872336 conn(0x7fa5180f23b0 msgr2=0x7fa5180f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:17.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.713+0000 7fa520393700 1 -- 172.21.15.113:0/3280872336 shutdown_connections 2024-04-12T05:36:17.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.713+0000 7fa520393700 1 -- 172.21.15.113:0/3280872336 wait complete. 2024-04-12T05:36:17.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.714+0000 7fa520393700 1 Processor -- start 2024-04-12T05:36:17.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.714+0000 7fa520393700 1 -- start start 2024-04-12T05:36:17.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.714+0000 7fa520393700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5180fc7f0 0x7fa51810e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:17.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.714+0000 7fa520393700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5180fd150 0x7fa51810e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:17.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.714+0000 7fa520393700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51810eaf0 con 0x7fa5180fc7f0 2024-04-12T05:36:17.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.714+0000 7fa520393700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51810ec30 con 0x7fa5180fd150 2024-04-12T05:36:17.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.714+0000 7fa51e12f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5180fc7f0 0x7fa51810e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.714+0000 7fa51e12f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5180fc7f0 0x7fa51810e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53728/0 (socket says 172.21.15.113:53728) 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.714+0000 7fa51e12f700 1 -- 172.21.15.113:0/27575136 learned_addr learned my addr 172.21.15.113:0/27575136 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.714+0000 7fa51d92e700 1 --2- 172.21.15.113:0/27575136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5180fd150 0x7fa51810e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.715+0000 7fa51e12f700 1 -- 172.21.15.113:0/27575136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5180fd150 msgr2=0x7fa51810e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.715+0000 7fa51e12f700 1 --2- 172.21.15.113:0/27575136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5180fd150 0x7fa51810e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.715+0000 7fa51e12f700 1 -- 172.21.15.113:0/27575136 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa514005040 con 0x7fa5180fc7f0 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.715+0000 7fa51e12f700 1 --2- 172.21.15.113:0/27575136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5180fc7f0 0x7fa51810e000 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7fa50c0041d0 tx=0x7fa50c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.715+0000 7fa50b7fe700 1 -- 172.21.15.113:0/27575136 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa50c007b40 con 0x7fa5180fc7f0 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.715+0000 7fa520393700 1 -- 172.21.15.113:0/27575136 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5181082c0 con 0x7fa5180fc7f0 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.715+0000 7fa50b7fe700 1 -- 172.21.15.113:0/27575136 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa50c00f040 con 0x7fa5180fc7f0 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.715+0000 7fa50b7fe700 1 -- 172.21.15.113:0/27575136 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa50c0147b0 con 0x7fa5180fc7f0 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.715+0000 7fa520393700 1 -- 172.21.15.113:0/27575136 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa518108880 con 0x7fa5180fc7f0 2024-04-12T05:36:17.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.716+0000 7fa520393700 1 -- 172.21.15.113:0/27575136 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa518060390 con 0x7fa5180fc7f0 2024-04-12T05:36:17.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.717+0000 7fa50b7fe700 1 -- 172.21.15.113:0/27575136 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa50c009110 con 0x7fa5180fc7f0 2024-04-12T05:36:17.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.717+0000 7fa50b7fe700 1 --2- 172.21.15.113:0/27575136 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa50406caa0 0x7fa50406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:17.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.717+0000 7fa50b7fe700 1 -- 172.21.15.113:0/27575136 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa50c08c190 con 0x7fa5180fc7f0 2024-04-12T05:36:17.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.717+0000 7fa51d92e700 1 --2- 172.21.15.113:0/27575136 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa50406caa0 0x7fa50406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:17.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.718+0000 7fa51d92e700 1 --2- 172.21.15.113:0/27575136 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa50406caa0 0x7fa50406ef60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fa518109940 tx=0x7fa51400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:17.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.721+0000 7fa50b7fe700 1 -- 172.21.15.113:0/27575136 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa50c059c40 con 0x7fa5180fc7f0 2024-04-12T05:36:17.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.926+0000 7fa520393700 1 -- 172.21.15.113:0/27575136 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa518108f20 con 0x7fa50406caa0 2024-04-12T05:36:17.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.927+0000 7fa50b7fe700 1 -- 172.21.15.113:0/27575136 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa518108f20 con 0x7fa50406caa0 2024-04-12T05:36:17.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.930+0000 7fa520393700 1 -- 172.21.15.113:0/27575136 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa50406caa0 msgr2=0x7fa50406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:17.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.930+0000 7fa520393700 1 --2- 172.21.15.113:0/27575136 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa50406caa0 0x7fa50406ef60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fa518109940 tx=0x7fa51400b040).stop 2024-04-12T05:36:17.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.930+0000 7fa520393700 1 -- 172.21.15.113:0/27575136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5180fc7f0 msgr2=0x7fa51810e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:17.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.930+0000 7fa520393700 1 --2- 172.21.15.113:0/27575136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5180fc7f0 0x7fa51810e000 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7fa50c0041d0 tx=0x7fa50c009f60).stop 2024-04-12T05:36:17.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.930+0000 7fa520393700 1 -- 172.21.15.113:0/27575136 shutdown_connections 2024-04-12T05:36:17.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.930+0000 7fa520393700 1 --2- 172.21.15.113:0/27575136 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa50406caa0 0x7fa50406ef60 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:17.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.930+0000 7fa520393700 1 --2- 172.21.15.113:0/27575136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5180fc7f0 0x7fa51810e000 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:17.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.930+0000 7fa520393700 1 --2- 172.21.15.113:0/27575136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5180fd150 0x7fa51810e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:17.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.930+0000 7fa520393700 1 -- 172.21.15.113:0/27575136 >> 172.21.15.113:0/27575136 conn(0x7fa5180f23b0 msgr2=0x7fa5180613a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:17.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.930+0000 7fa520393700 1 -- 172.21.15.113:0/27575136 shutdown_connections 2024-04-12T05:36:17.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:17.930+0000 7fa520393700 1 -- 172.21.15.113:0/27575136 wait complete. 2024-04-12T05:36:17.938 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:36:18.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.028+0000 7f98d2097700 1 -- 172.21.15.113:0/3441520047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc0fe250 msgr2=0x7f98cc0f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.028+0000 7f98d2097700 1 --2- 172.21.15.113:0/3441520047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc0fe250 0x7f98cc0f7a30 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f98c00047f0 tx=0x7f98c0009cb0).stop 2024-04-12T05:36:18.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.028+0000 7f98d2097700 1 -- 172.21.15.113:0/3441520047 shutdown_connections 2024-04-12T05:36:18.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.028+0000 7f98d2097700 1 --2- 172.21.15.113:0/3441520047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc0fe250 0x7f98cc0f7a30 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.028+0000 7f98d2097700 1 --2- 172.21.15.113:0/3441520047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98cc0fd8f0 0x7f98cc0fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.028+0000 7f98d2097700 1 -- 172.21.15.113:0/3441520047 >> 172.21.15.113:0/3441520047 conn(0x7f98cc0f34b0 msgr2=0x7f98cc0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:18.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.028+0000 7f98d2097700 1 -- 172.21.15.113:0/3441520047 shutdown_connections 2024-04-12T05:36:18.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.028+0000 7f98d2097700 1 -- 172.21.15.113:0/3441520047 wait complete. 2024-04-12T05:36:18.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.029+0000 7f98d2097700 1 Processor -- start 2024-04-12T05:36:18.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.029+0000 7f98d2097700 1 -- start start 2024-04-12T05:36:18.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.029+0000 7f98d2097700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc0fd8f0 0x7f98cc06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:18.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.029+0000 7f98d2097700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98cc0fe250 0x7f98cc06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:18.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.029+0000 7f98d2097700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc06bab0 con 0x7f98cc0fd8f0 2024-04-12T05:36:18.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.029+0000 7f98d2097700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc06bbf0 con 0x7f98cc0fe250 2024-04-12T05:36:18.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.029+0000 7f98cb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc0fd8f0 0x7f98cc06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:18.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.029+0000 7f98cb7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc0fd8f0 0x7f98cc06b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53742/0 (socket says 172.21.15.113:53742) 2024-04-12T05:36:18.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.029+0000 7f98cb7fe700 1 -- 172.21.15.113:0/3527118096 learned_addr learned my addr 172.21.15.113:0/3527118096 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:18.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.029+0000 7f98caffd700 1 --2- 172.21.15.113:0/3527118096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98cc0fe250 0x7f98cc06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:18.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.030+0000 7f98cb7fe700 1 -- 172.21.15.113:0/3527118096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98cc0fe250 msgr2=0x7f98cc06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.030+0000 7f98cb7fe700 1 --2- 172.21.15.113:0/3527118096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98cc0fe250 0x7f98cc06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.030+0000 7f98cb7fe700 1 -- 172.21.15.113:0/3527118096 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0005040 con 0x7f98cc0fd8f0 2024-04-12T05:36:18.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.030+0000 7f98cb7fe700 1 --2- 172.21.15.113:0/3527118096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc0fd8f0 0x7f98cc06b0d0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f98bc00e8f0 tx=0x7f98bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:18.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.030+0000 7f98c8ff9700 1 -- 172.21.15.113:0/3527118096 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98bc007b40 con 0x7f98cc0fd8f0 2024-04-12T05:36:18.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.030+0000 7f98c8ff9700 1 -- 172.21.15.113:0/3527118096 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98bc00f040 con 0x7f98cc0fd8f0 2024-04-12T05:36:18.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.030+0000 7f98d2097700 1 -- 172.21.15.113:0/3527118096 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc06bea0 con 0x7f98cc0fd8f0 2024-04-12T05:36:18.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.030+0000 7f98c8ff9700 1 -- 172.21.15.113:0/3527118096 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98bc014880 con 0x7f98cc0fd8f0 2024-04-12T05:36:18.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.030+0000 7f98d2097700 1 -- 172.21.15.113:0/3527118096 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc114810 con 0x7f98cc0fd8f0 2024-04-12T05:36:18.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.031+0000 7f98d2097700 1 -- 172.21.15.113:0/3527118096 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc05f590 con 0x7f98cc0fd8f0 2024-04-12T05:36:18.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.031+0000 7f98c8ff9700 1 -- 172.21.15.113:0/3527118096 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f98bc014b00 con 0x7f98cc0fd8f0 2024-04-12T05:36:18.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.032+0000 7f98c8ff9700 1 --2- 172.21.15.113:0/3527118096 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f98b406caa0 0x7f98b406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:18.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.032+0000 7f98c8ff9700 1 -- 172.21.15.113:0/3527118096 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f98bc08ba50 con 0x7f98cc0fd8f0 2024-04-12T05:36:18.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.032+0000 7f98caffd700 1 --2- 172.21.15.113:0/3527118096 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f98b406caa0 0x7f98b406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:18.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.032+0000 7f98caffd700 1 --2- 172.21.15.113:0/3527118096 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f98b406caa0 0x7f98b406ef60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f98cc06c870 tx=0x7f98c0009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:18.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.036+0000 7f98c8ff9700 1 -- 172.21.15.113:0/3527118096 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f98bc055a70 con 0x7f98cc0fd8f0 2024-04-12T05:36:18.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.239+0000 7f98d2097700 1 -- 172.21.15.113:0/3527118096 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f98cc059120 con 0x7f98b406caa0 2024-04-12T05:36:18.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.240+0000 7f98c8ff9700 1 -- 172.21.15.113:0/3527118096 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f98cc059120 con 0x7f98b406caa0 2024-04-12T05:36:18.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.242+0000 7f98d2097700 1 -- 172.21.15.113:0/3527118096 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f98b406caa0 msgr2=0x7f98b406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.242+0000 7f98d2097700 1 --2- 172.21.15.113:0/3527118096 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f98b406caa0 0x7f98b406ef60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f98cc06c870 tx=0x7f98c0009cb0).stop 2024-04-12T05:36:18.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.242+0000 7f98d2097700 1 -- 172.21.15.113:0/3527118096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc0fd8f0 msgr2=0x7f98cc06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.242+0000 7f98d2097700 1 --2- 172.21.15.113:0/3527118096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc0fd8f0 0x7f98cc06b0d0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f98bc00e8f0 tx=0x7f98bc00b040).stop 2024-04-12T05:36:18.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.242+0000 7f98d2097700 1 -- 172.21.15.113:0/3527118096 shutdown_connections 2024-04-12T05:36:18.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.242+0000 7f98d2097700 1 --2- 172.21.15.113:0/3527118096 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f98b406caa0 0x7f98b406ef60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.242+0000 7f98d2097700 1 --2- 172.21.15.113:0/3527118096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98cc0fd8f0 0x7f98cc06b0d0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.242+0000 7f98d2097700 1 --2- 172.21.15.113:0/3527118096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98cc0fe250 0x7f98cc06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.242+0000 7f98d2097700 1 -- 172.21.15.113:0/3527118096 >> 172.21.15.113:0/3527118096 conn(0x7f98cc0f34b0 msgr2=0x7f98cc0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:18.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.242+0000 7f98d2097700 1 -- 172.21.15.113:0/3527118096 shutdown_connections 2024-04-12T05:36:18.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.242+0000 7f98d2097700 1 -- 172.21.15.113:0/3527118096 wait complete. 2024-04-12T05:36:18.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.339+0000 7efc29113700 1 -- 172.21.15.113:0/4160174268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc240fd8f0 msgr2=0x7efc240fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.339+0000 7efc29113700 1 --2- 172.21.15.113:0/4160174268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc240fd8f0 0x7efc240fdcd0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7efc180047f0 tx=0x7efc18009cb0).stop 2024-04-12T05:36:18.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.340+0000 7efc29113700 1 -- 172.21.15.113:0/4160174268 shutdown_connections 2024-04-12T05:36:18.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.340+0000 7efc29113700 1 --2- 172.21.15.113:0/4160174268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc240fe250 0x7efc240f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.340+0000 7efc29113700 1 --2- 172.21.15.113:0/4160174268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc240fd8f0 0x7efc240fdcd0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.340+0000 7efc29113700 1 -- 172.21.15.113:0/4160174268 >> 172.21.15.113:0/4160174268 conn(0x7efc240f34b0 msgr2=0x7efc240f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:18.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.340+0000 7efc29113700 1 -- 172.21.15.113:0/4160174268 shutdown_connections 2024-04-12T05:36:18.339 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.340+0000 7efc29113700 1 -- 172.21.15.113:0/4160174268 wait complete. 2024-04-12T05:36:18.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.340+0000 7efc29113700 1 Processor -- start 2024-04-12T05:36:18.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc29113700 1 -- start start 2024-04-12T05:36:18.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc29113700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc240fd8f0 0x7efc2410e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:18.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc29113700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc240fe250 0x7efc2410e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:18.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc29113700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2410ec00 con 0x7efc240fe250 2024-04-12T05:36:18.340 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc29113700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2410ed40 con 0x7efc240fd8f0 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc2259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc240fe250 0x7efc2410e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc2259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc240fe250 0x7efc2410e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53768/0 (socket says 172.21.15.113:53768) 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc2259c700 1 -- 172.21.15.113:0/246496625 learned_addr learned my addr 172.21.15.113:0/246496625 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc22d9d700 1 --2- 172.21.15.113:0/246496625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc240fd8f0 0x7efc2410e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc2259c700 1 -- 172.21.15.113:0/246496625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc240fd8f0 msgr2=0x7efc2410e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc2259c700 1 --2- 172.21.15.113:0/246496625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc240fd8f0 0x7efc2410e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc2259c700 1 -- 172.21.15.113:0/246496625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc18005040 con 0x7efc240fe250 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc2259c700 1 --2- 172.21.15.113:0/246496625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc240fe250 0x7efc2410e600 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7efc1400e940 tx=0x7efc1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc0bfff700 1 -- 172.21.15.113:0/246496625 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc14007b40 con 0x7efc240fe250 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc0bfff700 1 -- 172.21.15.113:0/246496625 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc1400f040 con 0x7efc240fe250 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc29113700 1 -- 172.21.15.113:0/246496625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc24108370 con 0x7efc240fe250 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.341+0000 7efc0bfff700 1 -- 172.21.15.113:0/246496625 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc140148e0 con 0x7efc240fe250 2024-04-12T05:36:18.341 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.342+0000 7efc29113700 1 -- 172.21.15.113:0/246496625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc24108930 con 0x7efc240fe250 2024-04-12T05:36:18.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.343+0000 7efc29113700 1 -- 172.21.15.113:0/246496625 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc2405f590 con 0x7efc240fe250 2024-04-12T05:36:18.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.343+0000 7efc0bfff700 1 -- 172.21.15.113:0/246496625 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7efc14007ca0 con 0x7efc240fe250 2024-04-12T05:36:18.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.343+0000 7efc0bfff700 1 --2- 172.21.15.113:0/246496625 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc0c06caa0 0x7efc0c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:18.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.343+0000 7efc0bfff700 1 -- 172.21.15.113:0/246496625 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7efc1408b860 con 0x7efc240fe250 2024-04-12T05:36:18.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.343+0000 7efc22d9d700 1 --2- 172.21.15.113:0/246496625 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc0c06caa0 0x7efc0c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:18.343 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.344+0000 7efc22d9d700 1 --2- 172.21.15.113:0/246496625 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc0c06caa0 0x7efc0c06ef60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7efc180047c0 tx=0x7efc18009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:18.346 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.348+0000 7efc0bfff700 1 -- 172.21.15.113:0/246496625 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7efc14059310 con 0x7efc240fe250 2024-04-12T05:36:18.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.542+0000 7efc29113700 1 -- 172.21.15.113:0/246496625 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efc240ffc40 con 0x7efc0c06caa0 2024-04-12T05:36:18.549 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.551+0000 7efc0bfff700 1 -- 172.21.15.113:0/246496625 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7efc240ffc40 con 0x7efc0c06caa0 2024-04-12T05:36:18.550 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:36:18.550 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (24m) 96s ago 35m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:36:18.550 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (21m) 96s ago 35m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:36:18.550 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (21m) 96s ago 34m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:36:18.550 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (23m) 96s ago 35m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:36:18.550 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (17m) 96s ago 30m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:36:18.550 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (17m) 96s ago 29m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:36:18.550 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (16m) 96s ago 29m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:36:18.550 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (100s) 96s ago 29m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:36:18.550 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (25m) 96s ago 36m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (25m) 96s ago 34m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (22m) 96s ago 36m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (22m) 96s ago 34m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (24m) 96s ago 35m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (24m) 96s ago 34m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (21m) 96s ago 33m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (20m) 96s ago 33m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (19m) 96s ago 32m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (19m) 96s ago 32m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (18m) 96s ago 32m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (17m) 96s ago 31m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:36:18.551 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (24m) 96s ago 34m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:36:18.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.553+0000 7efc29113700 1 -- 172.21.15.113:0/246496625 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc0c06caa0 msgr2=0x7efc0c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.553+0000 7efc29113700 1 --2- 172.21.15.113:0/246496625 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc0c06caa0 0x7efc0c06ef60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7efc180047c0 tx=0x7efc18009cb0).stop 2024-04-12T05:36:18.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.553+0000 7efc29113700 1 -- 172.21.15.113:0/246496625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc240fe250 msgr2=0x7efc2410e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.553+0000 7efc29113700 1 --2- 172.21.15.113:0/246496625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc240fe250 0x7efc2410e600 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7efc1400e940 tx=0x7efc1400b040).stop 2024-04-12T05:36:18.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.554+0000 7efc29113700 1 -- 172.21.15.113:0/246496625 shutdown_connections 2024-04-12T05:36:18.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.554+0000 7efc29113700 1 --2- 172.21.15.113:0/246496625 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc0c06caa0 0x7efc0c06ef60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.554+0000 7efc29113700 1 --2- 172.21.15.113:0/246496625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc240fd8f0 0x7efc2410e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.554+0000 7efc29113700 1 --2- 172.21.15.113:0/246496625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc240fe250 0x7efc2410e600 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.554+0000 7efc29113700 1 -- 172.21.15.113:0/246496625 >> 172.21.15.113:0/246496625 conn(0x7efc240f34b0 msgr2=0x7efc240fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:18.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.554+0000 7efc29113700 1 -- 172.21.15.113:0/246496625 shutdown_connections 2024-04-12T05:36:18.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.554+0000 7efc29113700 1 -- 172.21.15.113:0/246496625 wait complete. 2024-04-12T05:36:18.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.650+0000 7f365882b700 1 -- 172.21.15.113:0/966698478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36540fc7f0 msgr2=0x7f36540fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.650+0000 7f365882b700 1 --2- 172.21.15.113:0/966698478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36540fc7f0 0x7f36540fcbd0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f36440047f0 tx=0x7f3644009cb0).stop 2024-04-12T05:36:18.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.650+0000 7f365882b700 1 -- 172.21.15.113:0/966698478 shutdown_connections 2024-04-12T05:36:18.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.650+0000 7f365882b700 1 --2- 172.21.15.113:0/966698478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36540fd150 0x7f36540f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.650+0000 7f365882b700 1 --2- 172.21.15.113:0/966698478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36540fc7f0 0x7f36540fcbd0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.650+0000 7f365882b700 1 -- 172.21.15.113:0/966698478 >> 172.21.15.113:0/966698478 conn(0x7f36540f23b0 msgr2=0x7f36540f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:18.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.650+0000 7f365882b700 1 -- 172.21.15.113:0/966698478 shutdown_connections 2024-04-12T05:36:18.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.650+0000 7f365882b700 1 -- 172.21.15.113:0/966698478 wait complete. 2024-04-12T05:36:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.651+0000 7f365882b700 1 Processor -- start 2024-04-12T05:36:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.651+0000 7f365882b700 1 -- start start 2024-04-12T05:36:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.651+0000 7f365882b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36540fc7f0 0x7f36540f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.651+0000 7f365882b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36540fd150 0x7f36540f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.651+0000 7f365882b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36540f8f20 con 0x7f36540fd150 2024-04-12T05:36:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.651+0000 7f365882b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36540f9060 con 0x7f36540fc7f0 2024-04-12T05:36:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f3651d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36540fd150 0x7f36540f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f3651d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36540fd150 0x7f36540f8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53786/0 (socket says 172.21.15.113:53786) 2024-04-12T05:36:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f3651d9b700 1 -- 172.21.15.113:0/1385988066 learned_addr learned my addr 172.21.15.113:0/1385988066 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f365259c700 1 --2- 172.21.15.113:0/1385988066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36540fc7f0 0x7f36540f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f3651d9b700 1 -- 172.21.15.113:0/1385988066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36540fc7f0 msgr2=0x7f36540f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f3651d9b700 1 --2- 172.21.15.113:0/1385988066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36540fc7f0 0x7f36540f8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f3651d9b700 1 -- 172.21.15.113:0/1385988066 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3644005040 con 0x7f36540fd150 2024-04-12T05:36:18.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f3651d9b700 1 --2- 172.21.15.113:0/1385988066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36540fd150 0x7f36540f8a30 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f363c002780 tx=0x7f363c009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:18.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f364b7fe700 1 -- 172.21.15.113:0/1385988066 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f363c007b40 con 0x7f36540fd150 2024-04-12T05:36:18.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f365882b700 1 -- 172.21.15.113:0/1385988066 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36540f9310 con 0x7f36540fd150 2024-04-12T05:36:18.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f364b7fe700 1 -- 172.21.15.113:0/1385988066 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f363c00f040 con 0x7f36540fd150 2024-04-12T05:36:18.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f364b7fe700 1 -- 172.21.15.113:0/1385988066 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f363c014780 con 0x7f36540fd150 2024-04-12T05:36:18.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.652+0000 7f365882b700 1 -- 172.21.15.113:0/1385988066 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36541146f0 con 0x7f36540fd150 2024-04-12T05:36:18.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.653+0000 7f365882b700 1 -- 172.21.15.113:0/1385988066 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3654060370 con 0x7f36540fd150 2024-04-12T05:36:18.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.654+0000 7f364b7fe700 1 -- 172.21.15.113:0/1385988066 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f363c007ca0 con 0x7f36540fd150 2024-04-12T05:36:18.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.654+0000 7f364b7fe700 1 --2- 172.21.15.113:0/1385988066 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f364006caa0 0x7f364006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:18.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.654+0000 7f364b7fe700 1 -- 172.21.15.113:0/1385988066 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f363c08bfc0 con 0x7f36540fd150 2024-04-12T05:36:18.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.654+0000 7f365259c700 1 --2- 172.21.15.113:0/1385988066 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f364006caa0 0x7f364006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:18.653 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.655+0000 7f365259c700 1 --2- 172.21.15.113:0/1385988066 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f364006caa0 0x7f364006ef60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f3644004f10 tx=0x7f364400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:18.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.658+0000 7f364b7fe700 1 -- 172.21.15.113:0/1385988066 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f363c055fe0 con 0x7f36540fd150 2024-04-12T05:36:18.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.926+0000 7f365882b700 1 -- 172.21.15.113:0/1385988066 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3654047540 con 0x7f36540fd150 2024-04-12T05:36:18.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.927+0000 7f364b7fe700 1 -- 172.21.15.113:0/1385988066 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f363c059600 con 0x7f36540fd150 2024-04-12T05:36:18.925 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:36:18.925 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:36:18.925 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:36:18.925 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:36:18.925 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:36:18.926 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:36:18.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.929+0000 7f365882b700 1 -- 172.21.15.113:0/1385988066 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f364006caa0 msgr2=0x7f364006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.929+0000 7f365882b700 1 --2- 172.21.15.113:0/1385988066 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f364006caa0 0x7f364006ef60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f3644004f10 tx=0x7f364400b040).stop 2024-04-12T05:36:18.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.929+0000 7f365882b700 1 -- 172.21.15.113:0/1385988066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36540fd150 msgr2=0x7f36540f8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:18.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.929+0000 7f365882b700 1 --2- 172.21.15.113:0/1385988066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36540fd150 0x7f36540f8a30 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f363c002780 tx=0x7f363c009ee0).stop 2024-04-12T05:36:18.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.929+0000 7f365882b700 1 -- 172.21.15.113:0/1385988066 shutdown_connections 2024-04-12T05:36:18.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.929+0000 7f365882b700 1 --2- 172.21.15.113:0/1385988066 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f364006caa0 0x7f364006ef60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.929+0000 7f365882b700 1 --2- 172.21.15.113:0/1385988066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36540fc7f0 0x7f36540f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.929+0000 7f365882b700 1 --2- 172.21.15.113:0/1385988066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36540fd150 0x7f36540f8a30 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:18.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.929+0000 7f365882b700 1 -- 172.21.15.113:0/1385988066 >> 172.21.15.113:0/1385988066 conn(0x7f36540f23b0 msgr2=0x7f3654061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:18.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.929+0000 7f365882b700 1 -- 172.21.15.113:0/1385988066 shutdown_connections 2024-04-12T05:36:18.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:18.929+0000 7f365882b700 1 -- 172.21.15.113:0/1385988066 wait complete. 2024-04-12T05:36:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.027+0000 7f66a1153700 1 -- 172.21.15.113:0/580693978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f669c0fd150 msgr2=0x7f669c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.027+0000 7f66a1153700 1 --2- 172.21.15.113:0/580693978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f669c0fd150 0x7f669c0f6930 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f668c0047f0 tx=0x7f668c009cb0).stop 2024-04-12T05:36:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.027+0000 7f66a1153700 1 -- 172.21.15.113:0/580693978 shutdown_connections 2024-04-12T05:36:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.027+0000 7f66a1153700 1 --2- 172.21.15.113:0/580693978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f669c0fd150 0x7f669c0f6930 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.027+0000 7f66a1153700 1 --2- 172.21.15.113:0/580693978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f669c0fc7f0 0x7f669c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.027+0000 7f66a1153700 1 -- 172.21.15.113:0/580693978 >> 172.21.15.113:0/580693978 conn(0x7f669c0f23b0 msgr2=0x7f669c0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.027+0000 7f66a1153700 1 -- 172.21.15.113:0/580693978 shutdown_connections 2024-04-12T05:36:19.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.027+0000 7f66a1153700 1 -- 172.21.15.113:0/580693978 wait complete. 2024-04-12T05:36:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.028+0000 7f66a1153700 1 Processor -- start 2024-04-12T05:36:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.028+0000 7f66a1153700 1 -- start start 2024-04-12T05:36:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.028+0000 7f66a1153700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f669c0fc7f0 0x7f669c10e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.028+0000 7f66a1153700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f669c0fd150 0x7f669c10e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:19.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.028+0000 7f66a1153700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c10eb60 con 0x7f669c0fd150 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.028+0000 7f66a1153700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c10eca0 con 0x7f669c0fc7f0 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f669a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f669c0fd150 0x7f669c10e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f669a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f669c0fd150 0x7f669c10e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53806/0 (socket says 172.21.15.113:53806) 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f669a59c700 1 -- 172.21.15.113:0/1690826235 learned_addr learned my addr 172.21.15.113:0/1690826235 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f669ad9d700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f669c0fc7f0 0x7f669c10e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f669a59c700 1 -- 172.21.15.113:0/1690826235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f669c0fc7f0 msgr2=0x7f669c10e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f669a59c700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f669c0fc7f0 0x7f669c10e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f669a59c700 1 -- 172.21.15.113:0/1690826235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c005040 con 0x7f669c0fd150 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f669ad9d700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f669c0fc7f0 0x7f669c10e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f669a59c700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f669c0fd150 0x7f669c10e560 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f668c0099f0 tx=0x7f668c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f6683fff700 1 -- 172.21.15.113:0/1690826235 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f668c00e030 con 0x7f669c0fd150 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f6683fff700 1 -- 172.21.15.113:0/1690826235 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f668c007e90 con 0x7f669c0fd150 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f66a1153700 1 -- 172.21.15.113:0/1690826235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669c1082d0 con 0x7f669c0fd150 2024-04-12T05:36:19.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f6683fff700 1 -- 172.21.15.113:0/1690826235 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f668c018070 con 0x7f669c0fd150 2024-04-12T05:36:19.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.029+0000 7f66a1153700 1 -- 172.21.15.113:0/1690826235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669c1087b0 con 0x7f669c0fd150 2024-04-12T05:36:19.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.030+0000 7f66a1153700 1 -- 172.21.15.113:0/1690826235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669c060390 con 0x7f669c0fd150 2024-04-12T05:36:19.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.031+0000 7f6683fff700 1 -- 172.21.15.113:0/1690826235 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f668c01d420 con 0x7f669c0fd150 2024-04-12T05:36:19.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.031+0000 7f6683fff700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f668406caa0 0x7f668406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:19.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.031+0000 7f6683fff700 1 -- 172.21.15.113:0/1690826235 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f668c08d2f0 con 0x7f669c0fd150 2024-04-12T05:36:19.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.031+0000 7f669ad9d700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f668406caa0 0x7f668406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:19.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.031+0000 7f669ad9d700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f668406caa0 0x7f668406ef60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f66900041d0 tx=0x7f6690009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:19.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.035+0000 7f6683fff700 1 -- 172.21.15.113:0/1690826235 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f668c05ada0 con 0x7f669c0fd150 2024-04-12T05:36:19.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:18 smithi113 ceph-mon[103371]: pgmap v725: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:19.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:18 smithi113 ceph-mon[103371]: from='client.35318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:19.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:18 smithi192 ceph-mon[85304]: pgmap v725: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:19.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:18 smithi192 ceph-mon[85304]: from='client.35318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:19.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.268+0000 7f66a1153700 1 -- 172.21.15.113:0/1690826235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f669c108df0 con 0x7f669c0fd150 2024-04-12T05:36:19.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.268+0000 7f6683fff700 1 -- 172.21.15.113:0/1690826235 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f668c05a930 con 0x7f669c0fd150 2024-04-12T05:36:19.266 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:36:19.267 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:36:19.268 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:36:19.269 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:36:19.269 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:36:19.269 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:36:19.269 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:36:19.269 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:36:19.269 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:36:19.269 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:36:19.269 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:36:19.269 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:36:19.269 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:36:19.269 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:36:19.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.270+0000 7f66a1153700 1 -- 172.21.15.113:0/1690826235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f668406caa0 msgr2=0x7f668406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.270+0000 7f66a1153700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f668406caa0 0x7f668406ef60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f66900041d0 tx=0x7f6690009040).stop 2024-04-12T05:36:19.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.270+0000 7f66a1153700 1 -- 172.21.15.113:0/1690826235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f669c0fd150 msgr2=0x7f669c10e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.270+0000 7f66a1153700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f669c0fd150 0x7f669c10e560 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f668c0099f0 tx=0x7f668c009cb0).stop 2024-04-12T05:36:19.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.270+0000 7f66a1153700 1 -- 172.21.15.113:0/1690826235 shutdown_connections 2024-04-12T05:36:19.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.270+0000 7f66a1153700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f668406caa0 0x7f668406ef60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.270+0000 7f66a1153700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f669c0fc7f0 0x7f669c10e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.270+0000 7f66a1153700 1 --2- 172.21.15.113:0/1690826235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f669c0fd150 0x7f669c10e560 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.270+0000 7f66a1153700 1 -- 172.21.15.113:0/1690826235 >> 172.21.15.113:0/1690826235 conn(0x7f669c0f23b0 msgr2=0x7f669c0613a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:19.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.270+0000 7f66a1153700 1 -- 172.21.15.113:0/1690826235 shutdown_connections 2024-04-12T05:36:19.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.270+0000 7f66a1153700 1 -- 172.21.15.113:0/1690826235 wait complete. 2024-04-12T05:36:19.271 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:36:19.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.368+0000 7ff42922e700 1 -- 172.21.15.113:0/3564768708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4240fd150 msgr2=0x7ff4240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.368+0000 7ff42922e700 1 --2- 172.21.15.113:0/3564768708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4240fd150 0x7ff4240f6930 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7ff4140047f0 tx=0x7ff414009cb0).stop 2024-04-12T05:36:19.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.369+0000 7ff42922e700 1 -- 172.21.15.113:0/3564768708 shutdown_connections 2024-04-12T05:36:19.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.369+0000 7ff42922e700 1 --2- 172.21.15.113:0/3564768708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4240fd150 0x7ff4240f6930 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.369+0000 7ff42922e700 1 --2- 172.21.15.113:0/3564768708 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4240fc7f0 0x7ff4240fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.369+0000 7ff42922e700 1 -- 172.21.15.113:0/3564768708 >> 172.21.15.113:0/3564768708 conn(0x7ff424070590 msgr2=0x7ff4240f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:19.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.369+0000 7ff42922e700 1 -- 172.21.15.113:0/3564768708 shutdown_connections 2024-04-12T05:36:19.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.369+0000 7ff42922e700 1 -- 172.21.15.113:0/3564768708 wait complete. 2024-04-12T05:36:19.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.369+0000 7ff42922e700 1 Processor -- start 2024-04-12T05:36:19.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff42922e700 1 -- start start 2024-04-12T05:36:19.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff42922e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4240fc7f0 0x7ff42410ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff42922e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4240fd150 0x7ff42410d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff42922e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42410d9c0 con 0x7ff4240fc7f0 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff42922e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42410db00 con 0x7ff4240fd150 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff422d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4240fc7f0 0x7ff42410ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff422d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4240fc7f0 0x7ff42410ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53830/0 (socket says 172.21.15.113:53830) 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff422d9d700 1 -- 172.21.15.113:0/2096386486 learned_addr learned my addr 172.21.15.113:0/2096386486 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff42259c700 1 --2- 172.21.15.113:0/2096386486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4240fd150 0x7ff42410d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff422d9d700 1 -- 172.21.15.113:0/2096386486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4240fd150 msgr2=0x7ff42410d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff422d9d700 1 --2- 172.21.15.113:0/2096386486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4240fd150 0x7ff42410d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff422d9d700 1 -- 172.21.15.113:0/2096386486 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff414005040 con 0x7ff4240fc7f0 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.370+0000 7ff422d9d700 1 --2- 172.21.15.113:0/2096386486 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4240fc7f0 0x7ff42410ced0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7ff4180041d0 tx=0x7ff418009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.371+0000 7ff40bfff700 1 -- 172.21.15.113:0/2096386486 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff418002d90 con 0x7ff4240fc7f0 2024-04-12T05:36:19.370 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.371+0000 7ff40bfff700 1 -- 172.21.15.113:0/2096386486 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff41800f040 con 0x7ff4240fc7f0 2024-04-12T05:36:19.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.371+0000 7ff42922e700 1 -- 172.21.15.113:0/2096386486 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff424107190 con 0x7ff4240fc7f0 2024-04-12T05:36:19.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.371+0000 7ff40bfff700 1 -- 172.21.15.113:0/2096386486 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41800d800 con 0x7ff4240fc7f0 2024-04-12T05:36:19.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.371+0000 7ff42922e700 1 -- 172.21.15.113:0/2096386486 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff424107750 con 0x7ff4240fc7f0 2024-04-12T05:36:19.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.372+0000 7ff42922e700 1 -- 172.21.15.113:0/2096386486 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42405f590 con 0x7ff4240fc7f0 2024-04-12T05:36:19.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.372+0000 7ff40bfff700 1 -- 172.21.15.113:0/2096386486 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff41800dac0 con 0x7ff4240fc7f0 2024-04-12T05:36:19.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.373+0000 7ff40bfff700 1 --2- 172.21.15.113:0/2096386486 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff40c06caa0 0x7ff40c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:19.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.373+0000 7ff42259c700 1 --2- 172.21.15.113:0/2096386486 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff40c06caa0 0x7ff40c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:19.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.373+0000 7ff40bfff700 1 -- 172.21.15.113:0/2096386486 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff41808bea0 con 0x7ff4240fc7f0 2024-04-12T05:36:19.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.373+0000 7ff42259c700 1 --2- 172.21.15.113:0/2096386486 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff40c06caa0 0x7ff40c06ef60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ff414004f10 tx=0x7ff41400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:19.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.377+0000 7ff40bfff700 1 -- 172.21.15.113:0/2096386486 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff418059950 con 0x7ff4240fc7f0 2024-04-12T05:36:19.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.582+0000 7ff42922e700 1 -- 172.21.15.113:0/2096386486 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4241079d0 con 0x7ff40c06caa0 2024-04-12T05:36:19.581 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.583+0000 7ff40bfff700 1 -- 172.21.15.113:0/2096386486 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff4241079d0 con 0x7ff40c06caa0 2024-04-12T05:36:19.581 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:36:19.581 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:36:19.581 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:36:19.582 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:36:19.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.585+0000 7ff42922e700 1 -- 172.21.15.113:0/2096386486 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff40c06caa0 msgr2=0x7ff40c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.585+0000 7ff42922e700 1 --2- 172.21.15.113:0/2096386486 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff40c06caa0 0x7ff40c06ef60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ff414004f10 tx=0x7ff41400b040).stop 2024-04-12T05:36:19.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.585+0000 7ff42922e700 1 -- 172.21.15.113:0/2096386486 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4240fc7f0 msgr2=0x7ff42410ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.585+0000 7ff42922e700 1 --2- 172.21.15.113:0/2096386486 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4240fc7f0 0x7ff42410ced0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7ff4180041d0 tx=0x7ff418009f60).stop 2024-04-12T05:36:19.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.585+0000 7ff42922e700 1 -- 172.21.15.113:0/2096386486 shutdown_connections 2024-04-12T05:36:19.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.585+0000 7ff42922e700 1 --2- 172.21.15.113:0/2096386486 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff40c06caa0 0x7ff40c06ef60 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.585+0000 7ff42922e700 1 --2- 172.21.15.113:0/2096386486 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4240fc7f0 0x7ff42410ced0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.585+0000 7ff42922e700 1 --2- 172.21.15.113:0/2096386486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4240fd150 0x7ff42410d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.585+0000 7ff42922e700 1 -- 172.21.15.113:0/2096386486 >> 172.21.15.113:0/2096386486 conn(0x7ff424070590 msgr2=0x7ff4240fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:19.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.585+0000 7ff42922e700 1 -- 172.21.15.113:0/2096386486 shutdown_connections 2024-04-12T05:36:19.584 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.585+0000 7ff42922e700 1 -- 172.21.15.113:0/2096386486 wait complete. 2024-04-12T05:36:19.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.682+0000 7f28564d0700 1 -- 172.21.15.113:0/679746320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500fe210 msgr2=0x7f28500f5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.682+0000 7f28564d0700 1 --2- 172.21.15.113:0/679746320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500fe210 0x7f28500f5f80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f28440047f0 tx=0x7f2844009cb0).stop 2024-04-12T05:36:19.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.682+0000 7f28564d0700 1 -- 172.21.15.113:0/679746320 shutdown_connections 2024-04-12T05:36:19.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.682+0000 7f28564d0700 1 --2- 172.21.15.113:0/679746320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500fe210 0x7f28500f5f80 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.682+0000 7f28564d0700 1 --2- 172.21.15.113:0/679746320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28500fd8b0 0x7f28500fdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.682+0000 7f28564d0700 1 -- 172.21.15.113:0/679746320 >> 172.21.15.113:0/679746320 conn(0x7f285006ce40 msgr2=0x7f285006d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:19.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.683+0000 7f28564d0700 1 -- 172.21.15.113:0/679746320 shutdown_connections 2024-04-12T05:36:19.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.683+0000 7f28564d0700 1 -- 172.21.15.113:0/679746320 wait complete. 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.683+0000 7f28564d0700 1 Processor -- start 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.683+0000 7f28564d0700 1 -- start start 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f28564d0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28500fd8b0 0x7f285010e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f28564d0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500fe210 0x7f285010e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f28564d0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285010eb20 con 0x7f28500fe210 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f28564d0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285010ec60 con 0x7f28500fd8b0 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f284f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500fe210 0x7f285010e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f284f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500fe210 0x7f285010e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53842/0 (socket says 172.21.15.113:53842) 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f284f7fe700 1 -- 172.21.15.113:0/3437812576 learned_addr learned my addr 172.21.15.113:0/3437812576 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f284ffff700 1 --2- 172.21.15.113:0/3437812576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28500fd8b0 0x7f285010e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f284f7fe700 1 -- 172.21.15.113:0/3437812576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28500fd8b0 msgr2=0x7f285010e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f284f7fe700 1 --2- 172.21.15.113:0/3437812576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28500fd8b0 0x7f285010e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f284f7fe700 1 -- 172.21.15.113:0/3437812576 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2844005040 con 0x7f28500fe210 2024-04-12T05:36:19.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.684+0000 7f284f7fe700 1 --2- 172.21.15.113:0/3437812576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500fe210 0x7f285010e520 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f2844005010 tx=0x7f2844009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:19.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.685+0000 7f284d7fa700 1 -- 172.21.15.113:0/3437812576 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f284400e030 con 0x7f28500fe210 2024-04-12T05:36:19.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.685+0000 7f284d7fa700 1 -- 172.21.15.113:0/3437812576 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2844007e90 con 0x7f28500fe210 2024-04-12T05:36:19.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.685+0000 7f28564d0700 1 -- 172.21.15.113:0/3437812576 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2850108290 con 0x7f28500fe210 2024-04-12T05:36:19.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.685+0000 7f284d7fa700 1 -- 172.21.15.113:0/3437812576 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2844018070 con 0x7f28500fe210 2024-04-12T05:36:19.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.685+0000 7f28564d0700 1 -- 172.21.15.113:0/3437812576 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2850108770 con 0x7f28500fe210 2024-04-12T05:36:19.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.685+0000 7f28564d0700 1 -- 172.21.15.113:0/3437812576 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2850060670 con 0x7f28500fe210 2024-04-12T05:36:19.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.686+0000 7f284d7fa700 1 -- 172.21.15.113:0/3437812576 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f284401d6e0 con 0x7f28500fe210 2024-04-12T05:36:19.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.686+0000 7f284d7fa700 1 --2- 172.21.15.113:0/3437812576 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f283806ca30 0x7f283806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:19.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.686+0000 7f284d7fa700 1 -- 172.21.15.113:0/3437812576 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f284408d100 con 0x7f28500fe210 2024-04-12T05:36:19.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.686+0000 7f284ffff700 1 --2- 172.21.15.113:0/3437812576 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f283806ca30 0x7f283806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:19.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.687+0000 7f284ffff700 1 --2- 172.21.15.113:0/3437812576 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f283806ca30 0x7f283806eef0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f2840009170 tx=0x7f2840009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:19.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.691+0000 7f284d7fa700 1 -- 172.21.15.113:0/3437812576 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f284405a920 con 0x7f28500fe210 2024-04-12T05:36:19.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.956+0000 7f28564d0700 1 -- 172.21.15.113:0/3437812576 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2850047540 con 0x7f28500fe210 2024-04-12T05:36:19.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.956+0000 7f284d7fa700 1 -- 172.21.15.113:0/3437812576 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f284405a740 con 0x7f28500fe210 2024-04-12T05:36:19.955 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:36:19.955 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:36:19.956 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:36:19.957 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:36:19.957 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:36:19.957 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:36:19.957 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:36:19.957 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:36:19.957 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:36:19.957 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:36:19.957 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:36:19.957 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:36:19.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.958+0000 7f28564d0700 1 -- 172.21.15.113:0/3437812576 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f283806ca30 msgr2=0x7f283806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.959+0000 7f28564d0700 1 --2- 172.21.15.113:0/3437812576 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f283806ca30 0x7f283806eef0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f2840009170 tx=0x7f2840009da0).stop 2024-04-12T05:36:19.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.959+0000 7f28564d0700 1 -- 172.21.15.113:0/3437812576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500fe210 msgr2=0x7f285010e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:19.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.959+0000 7f28564d0700 1 --2- 172.21.15.113:0/3437812576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500fe210 0x7f285010e520 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f2844005010 tx=0x7f2844009cb0).stop 2024-04-12T05:36:19.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.959+0000 7f28564d0700 1 -- 172.21.15.113:0/3437812576 shutdown_connections 2024-04-12T05:36:19.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.959+0000 7f28564d0700 1 --2- 172.21.15.113:0/3437812576 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f283806ca30 0x7f283806eef0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.959+0000 7f28564d0700 1 --2- 172.21.15.113:0/3437812576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28500fd8b0 0x7f285010e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.959+0000 7f28564d0700 1 --2- 172.21.15.113:0/3437812576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500fe210 0x7f285010e520 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:19.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.959+0000 7f28564d0700 1 -- 172.21.15.113:0/3437812576 >> 172.21.15.113:0/3437812576 conn(0x7f285006ce40 msgr2=0x7f2850100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:19.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.959+0000 7f28564d0700 1 -- 172.21.15.113:0/3437812576 shutdown_connections 2024-04-12T05:36:19.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:19.959+0000 7f28564d0700 1 -- 172.21.15.113:0/3437812576 wait complete. 2024-04-12T05:36:20.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:19 smithi113 ceph-mon[103371]: from='client.35322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:20.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:19 smithi113 ceph-mon[103371]: from='client.35326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:20.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:19 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1385988066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:36:20.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:19 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1690826235' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:36:20.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:19 smithi192 ceph-mon[85304]: from='client.35322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:20.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:19 smithi192 ceph-mon[85304]: from='client.35326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:20.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:19 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1385988066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:36:20.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:19 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1690826235' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:36:21.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:20 smithi113 ceph-mon[103371]: from='client.35338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:21.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:20 smithi113 ceph-mon[103371]: pgmap v726: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:21.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:20 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3437812576' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:36:21.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:20 smithi192 ceph-mon[85304]: from='client.35338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:21.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:20 smithi192 ceph-mon[85304]: pgmap v726: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:21.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:20 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3437812576' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:36:22.170 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:22.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:23.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:22 smithi192 ceph-mon[85304]: pgmap v727: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:23.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:22 smithi113 ceph-mon[103371]: pgmap v727: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:25.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:24 smithi192 ceph-mon[85304]: pgmap v728: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:25.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:24 smithi113 ceph-mon[103371]: pgmap v728: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:27.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:26 smithi192 ceph-mon[85304]: pgmap v729: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:27.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:26 smithi113 ceph-mon[103371]: pgmap v729: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:29.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:28 smithi192 ceph-mon[85304]: pgmap v730: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:29.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:28 smithi113 ceph-mon[103371]: pgmap v730: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:31.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:30 smithi192 ceph-mon[85304]: pgmap v731: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:31.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:30 smithi113 ceph-mon[103371]: pgmap v731: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:33.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:32 smithi192 ceph-mon[85304]: pgmap v732: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:33.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:32 smithi113 ceph-mon[103371]: pgmap v732: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:35.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:34 smithi192 ceph-mon[85304]: pgmap v733: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:35.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:34 smithi113 ceph-mon[103371]: pgmap v733: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:37.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:36 smithi192 ceph-mon[85304]: pgmap v734: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:37.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:37.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:36 smithi113 ceph-mon[103371]: pgmap v734: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:37.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:39.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:38 smithi113 ceph-mon[103371]: pgmap v735: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:39.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:38 smithi192 ceph-mon[85304]: pgmap v735: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:41.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:40 smithi113 ceph-mon[103371]: pgmap v736: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:41.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:40 smithi192 ceph-mon[85304]: pgmap v736: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:43.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:42 smithi113 ceph-mon[103371]: pgmap v737: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:43.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:42 smithi192 ceph-mon[85304]: pgmap v737: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:45.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:44 smithi113 ceph-mon[103371]: pgmap v738: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:45.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:44 smithi192 ceph-mon[85304]: pgmap v738: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:47.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:46 smithi113 ceph-mon[103371]: pgmap v739: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:47.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:46 smithi192 ceph-mon[85304]: pgmap v739: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:49.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:48 smithi113 ceph-mon[103371]: pgmap v740: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:49.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:48 smithi192 ceph-mon[85304]: pgmap v740: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:50.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.069+0000 7fbbf907c700 1 -- 172.21.15.113:0/3098011494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf40fe250 msgr2=0x7fbbf40f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.069+0000 7fbbf907c700 1 --2- 172.21.15.113:0/3098011494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf40fe250 0x7fbbf40f7a30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7fbbe40047f0 tx=0x7fbbe4009cb0).stop 2024-04-12T05:36:50.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.069+0000 7fbbf907c700 1 -- 172.21.15.113:0/3098011494 shutdown_connections 2024-04-12T05:36:50.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.069+0000 7fbbf907c700 1 --2- 172.21.15.113:0/3098011494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf40fe250 0x7fbbf40f7a30 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.069+0000 7fbbf907c700 1 --2- 172.21.15.113:0/3098011494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf40fd8f0 0x7fbbf40fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.069+0000 7fbbf907c700 1 -- 172.21.15.113:0/3098011494 >> 172.21.15.113:0/3098011494 conn(0x7fbbf40f34b0 msgr2=0x7fbbf40f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:50.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.069+0000 7fbbf907c700 1 -- 172.21.15.113:0/3098011494 shutdown_connections 2024-04-12T05:36:50.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.069+0000 7fbbf907c700 1 -- 172.21.15.113:0/3098011494 wait complete. 2024-04-12T05:36:50.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.070+0000 7fbbf907c700 1 Processor -- start 2024-04-12T05:36:50.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.070+0000 7fbbf907c700 1 -- start start 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.070+0000 7fbbf907c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf40fd8f0 0x7fbbf410e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.070+0000 7fbbf907c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf40fe250 0x7fbbf410e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.070+0000 7fbbf907c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf410eb70 con 0x7fbbf40fe250 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.070+0000 7fbbf907c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf410ecb0 con 0x7fbbf40fd8f0 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.070+0000 7fbbf259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf40fe250 0x7fbbf410e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.070+0000 7fbbf259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf40fe250 0x7fbbf410e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55296/0 (socket says 172.21.15.113:55296) 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.070+0000 7fbbf259c700 1 -- 172.21.15.113:0/3241222288 learned_addr learned my addr 172.21.15.113:0/3241222288 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.071+0000 7fbbf2d9d700 1 --2- 172.21.15.113:0/3241222288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf40fd8f0 0x7fbbf410e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.071+0000 7fbbf259c700 1 -- 172.21.15.113:0/3241222288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf40fd8f0 msgr2=0x7fbbf410e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.071+0000 7fbbf259c700 1 --2- 172.21.15.113:0/3241222288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf40fd8f0 0x7fbbf410e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.071+0000 7fbbf259c700 1 -- 172.21.15.113:0/3241222288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe4005040 con 0x7fbbf40fe250 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.071+0000 7fbbf259c700 1 --2- 172.21.15.113:0/3241222288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf40fe250 0x7fbbf410e570 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7fbbe4000c00 tx=0x7fbbe4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.071+0000 7fbbdbfff700 1 -- 172.21.15.113:0/3241222288 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe400e030 con 0x7fbbf40fe250 2024-04-12T05:36:50.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.071+0000 7fbbdbfff700 1 -- 172.21.15.113:0/3241222288 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbe4007e90 con 0x7fbbf40fe250 2024-04-12T05:36:50.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.071+0000 7fbbf907c700 1 -- 172.21.15.113:0/3241222288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf41082e0 con 0x7fbbf40fe250 2024-04-12T05:36:50.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.071+0000 7fbbdbfff700 1 -- 172.21.15.113:0/3241222288 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe4018070 con 0x7fbbf40fe250 2024-04-12T05:36:50.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.071+0000 7fbbf907c700 1 -- 172.21.15.113:0/3241222288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf41087c0 con 0x7fbbf40fe250 2024-04-12T05:36:50.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.072+0000 7fbbf907c700 1 -- 172.21.15.113:0/3241222288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf405f590 con 0x7fbbf40fe250 2024-04-12T05:36:50.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.072+0000 7fbbdbfff700 1 -- 172.21.15.113:0/3241222288 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbbe4027440 con 0x7fbbf40fe250 2024-04-12T05:36:50.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.073+0000 7fbbdbfff700 1 --2- 172.21.15.113:0/3241222288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06cab0 0x7fbbdc06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:50.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.073+0000 7fbbdbfff700 1 -- 172.21.15.113:0/3241222288 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbbe408cef0 con 0x7fbbf40fe250 2024-04-12T05:36:50.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.073+0000 7fbbf2d9d700 1 --2- 172.21.15.113:0/3241222288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06cab0 0x7fbbdc06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:50.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.073+0000 7fbbf2d9d700 1 --2- 172.21.15.113:0/3241222288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06cab0 0x7fbbdc06ef70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fbbe8004e60 tx=0x7fbbe800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:50.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.077+0000 7fbbdbfff700 1 -- 172.21.15.113:0/3241222288 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbbe405aa60 con 0x7fbbf40fe250 2024-04-12T05:36:50.282 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.284+0000 7fbbf907c700 1 -- 172.21.15.113:0/3241222288 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbf41097e0 con 0x7fbbdc06cab0 2024-04-12T05:36:50.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.285+0000 7fbbdbfff700 1 -- 172.21.15.113:0/3241222288 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbbf41097e0 con 0x7fbbdc06cab0 2024-04-12T05:36:50.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.288+0000 7fbbf907c700 1 -- 172.21.15.113:0/3241222288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06cab0 msgr2=0x7fbbdc06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.288+0000 7fbbf907c700 1 --2- 172.21.15.113:0/3241222288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06cab0 0x7fbbdc06ef70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fbbe8004e60 tx=0x7fbbe800b040).stop 2024-04-12T05:36:50.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.288+0000 7fbbf907c700 1 -- 172.21.15.113:0/3241222288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf40fe250 msgr2=0x7fbbf410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.288+0000 7fbbf907c700 1 --2- 172.21.15.113:0/3241222288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf40fe250 0x7fbbf410e570 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7fbbe4000c00 tx=0x7fbbe4009cb0).stop 2024-04-12T05:36:50.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.288+0000 7fbbf907c700 1 -- 172.21.15.113:0/3241222288 shutdown_connections 2024-04-12T05:36:50.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.288+0000 7fbbf907c700 1 --2- 172.21.15.113:0/3241222288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbbdc06cab0 0x7fbbdc06ef70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.288+0000 7fbbf907c700 1 --2- 172.21.15.113:0/3241222288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbf40fd8f0 0x7fbbf410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.288+0000 7fbbf907c700 1 --2- 172.21.15.113:0/3241222288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbf40fe250 0x7fbbf410e570 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.286 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.288+0000 7fbbf907c700 1 -- 172.21.15.113:0/3241222288 >> 172.21.15.113:0/3241222288 conn(0x7fbbf40f34b0 msgr2=0x7fbbf40fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:50.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.288+0000 7fbbf907c700 1 -- 172.21.15.113:0/3241222288 shutdown_connections 2024-04-12T05:36:50.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.288+0000 7fbbf907c700 1 -- 172.21.15.113:0/3241222288 wait complete. 2024-04-12T05:36:50.296 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:36:50.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.387+0000 7f70223da700 1 -- 172.21.15.113:0/2484620958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f701c0f9220 msgr2=0x7f701c101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.387+0000 7f70223da700 1 --2- 172.21.15.113:0/2484620958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f701c0f9220 0x7f701c101930 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f700c0047f0 tx=0x7f700c009cb0).stop 2024-04-12T05:36:50.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.387+0000 7f70223da700 1 -- 172.21.15.113:0/2484620958 shutdown_connections 2024-04-12T05:36:50.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.387+0000 7f70223da700 1 --2- 172.21.15.113:0/2484620958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f701c0f9220 0x7f701c101930 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.387+0000 7f70223da700 1 --2- 172.21.15.113:0/2484620958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f701c0f88c0 0x7f701c0f8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.387+0000 7f70223da700 1 -- 172.21.15.113:0/2484620958 >> 172.21.15.113:0/2484620958 conn(0x7f701c0f2410 msgr2=0x7f701c0f4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:50.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.387+0000 7f70223da700 1 -- 172.21.15.113:0/2484620958 shutdown_connections 2024-04-12T05:36:50.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.387+0000 7f70223da700 1 -- 172.21.15.113:0/2484620958 wait complete. 2024-04-12T05:36:50.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.387+0000 7f70223da700 1 Processor -- start 2024-04-12T05:36:50.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.387+0000 7f70223da700 1 -- start start 2024-04-12T05:36:50.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f70223da700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f701c0f88c0 0x7f701c10e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:50.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f70223da700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f701c0f9220 0x7f701c10e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:50.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f70223da700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c10eb60 con 0x7f701c0f9220 2024-04-12T05:36:50.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f70223da700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c10eca0 con 0x7f701c0f88c0 2024-04-12T05:36:50.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f701b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f701c0f9220 0x7f701c10e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:50.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f701b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f701c0f9220 0x7f701c10e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55032/0 (socket says 172.21.15.113:55032) 2024-04-12T05:36:50.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f701b7fe700 1 -- 172.21.15.113:0/754402438 learned_addr learned my addr 172.21.15.113:0/754402438 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:50.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f701bfff700 1 --2- 172.21.15.113:0/754402438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f701c0f88c0 0x7f701c10e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:50.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f701b7fe700 1 -- 172.21.15.113:0/754402438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f701c0f88c0 msgr2=0x7f701c10e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f701b7fe700 1 --2- 172.21.15.113:0/754402438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f701c0f88c0 0x7f701c10e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f701b7fe700 1 -- 172.21.15.113:0/754402438 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c005040 con 0x7f701c0f9220 2024-04-12T05:36:50.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f701b7fe700 1 --2- 172.21.15.113:0/754402438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f701c0f9220 0x7f701c10e560 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f700c005010 tx=0x7f700c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:50.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f70197fa700 1 -- 172.21.15.113:0/754402438 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f700c00e030 con 0x7f701c0f9220 2024-04-12T05:36:50.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f70223da700 1 -- 172.21.15.113:0/754402438 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c1082d0 con 0x7f701c0f9220 2024-04-12T05:36:50.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f70197fa700 1 -- 172.21.15.113:0/754402438 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f700c007e90 con 0x7f701c0f9220 2024-04-12T05:36:50.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f70197fa700 1 -- 172.21.15.113:0/754402438 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f700c018070 con 0x7f701c0f9220 2024-04-12T05:36:50.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.388+0000 7f70223da700 1 -- 172.21.15.113:0/754402438 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701c1087b0 con 0x7f701c0f9220 2024-04-12T05:36:50.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.389+0000 7f70223da700 1 -- 172.21.15.113:0/754402438 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701c05f4f0 con 0x7f701c0f9220 2024-04-12T05:36:50.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.390+0000 7f70197fa700 1 -- 172.21.15.113:0/754402438 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f700c01d6e0 con 0x7f701c0f9220 2024-04-12T05:36:50.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.390+0000 7f70197fa700 1 --2- 172.21.15.113:0/754402438 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f700406ca30 0x7f700406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:50.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.390+0000 7f70197fa700 1 -- 172.21.15.113:0/754402438 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f700c08d100 con 0x7f701c0f9220 2024-04-12T05:36:50.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.390+0000 7f701bfff700 1 --2- 172.21.15.113:0/754402438 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f700406ca30 0x7f700406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:50.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.391+0000 7f701bfff700 1 --2- 172.21.15.113:0/754402438 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f700406ca30 0x7f700406eef0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f7010009170 tx=0x7f7010009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:50.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.394+0000 7f70197fa700 1 -- 172.21.15.113:0/754402438 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f700c05a920 con 0x7f701c0f9220 2024-04-12T05:36:50.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.599+0000 7f70223da700 1 -- 172.21.15.113:0/754402438 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f701c0fedf0 con 0x7f700406ca30 2024-04-12T05:36:50.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.600+0000 7f70197fa700 1 -- 172.21.15.113:0/754402438 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f701c0fedf0 con 0x7f700406ca30 2024-04-12T05:36:50.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.602+0000 7f70223da700 1 -- 172.21.15.113:0/754402438 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f700406ca30 msgr2=0x7f700406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.602+0000 7f70223da700 1 --2- 172.21.15.113:0/754402438 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f700406ca30 0x7f700406eef0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f7010009170 tx=0x7f7010009da0).stop 2024-04-12T05:36:50.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.602+0000 7f70223da700 1 -- 172.21.15.113:0/754402438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f701c0f9220 msgr2=0x7f701c10e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.602+0000 7f70223da700 1 --2- 172.21.15.113:0/754402438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f701c0f9220 0x7f701c10e560 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f700c005010 tx=0x7f700c009cb0).stop 2024-04-12T05:36:50.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.602+0000 7f70223da700 1 -- 172.21.15.113:0/754402438 shutdown_connections 2024-04-12T05:36:50.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.602+0000 7f70223da700 1 --2- 172.21.15.113:0/754402438 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f700406ca30 0x7f700406eef0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.602+0000 7f70223da700 1 --2- 172.21.15.113:0/754402438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f701c0f88c0 0x7f701c10e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.602+0000 7f70223da700 1 --2- 172.21.15.113:0/754402438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f701c0f9220 0x7f701c10e560 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.602+0000 7f70223da700 1 -- 172.21.15.113:0/754402438 >> 172.21.15.113:0/754402438 conn(0x7f701c0f2410 msgr2=0x7f701c0fc130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:50.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.602+0000 7f70223da700 1 -- 172.21.15.113:0/754402438 shutdown_connections 2024-04-12T05:36:50.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.602+0000 7f70223da700 1 -- 172.21.15.113:0/754402438 wait complete. 2024-04-12T05:36:50.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.700+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/763592140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f480fd910 msgr2=0x7f9f480fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.700+0000 7f9f4fb8c700 1 --2- 172.21.15.113:0/763592140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f480fd910 0x7f9f480fdcf0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f9f440047f0 tx=0x7f9f44009cb0).stop 2024-04-12T05:36:50.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.700+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/763592140 shutdown_connections 2024-04-12T05:36:50.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.700+0000 7f9f4fb8c700 1 --2- 172.21.15.113:0/763592140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f480fe270 0x7f9f480f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.700+0000 7f9f4fb8c700 1 --2- 172.21.15.113:0/763592140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f480fd910 0x7f9f480fdcf0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.700+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/763592140 >> 172.21.15.113:0/763592140 conn(0x7f9f480f3510 msgr2=0x7f9f480f5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.700+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/763592140 shutdown_connections 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.700+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/763592140 wait complete. 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.701+0000 7f9f4fb8c700 1 Processor -- start 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.701+0000 7f9f4fb8c700 1 -- start start 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.701+0000 7f9f4fb8c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f480fd910 0x7f9f4806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.701+0000 7f9f4fb8c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f480fe270 0x7f9f4806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.701+0000 7f9f4fb8c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4806bab0 con 0x7f9f480fe270 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.701+0000 7f9f4fb8c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4806bbf0 con 0x7f9f480fd910 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.701+0000 7f9f4d127700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f480fe270 0x7f9f4806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.701+0000 7f9f4d127700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f480fe270 0x7f9f4806b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55060/0 (socket says 172.21.15.113:55060) 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.701+0000 7f9f4d127700 1 -- 172.21.15.113:0/841427934 learned_addr learned my addr 172.21.15.113:0/841427934 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:50.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.702+0000 7f9f4d928700 1 --2- 172.21.15.113:0/841427934 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f480fd910 0x7f9f4806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:50.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.702+0000 7f9f4d127700 1 -- 172.21.15.113:0/841427934 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f480fd910 msgr2=0x7f9f4806b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.702+0000 7f9f4d127700 1 --2- 172.21.15.113:0/841427934 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f480fd910 0x7f9f4806b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.702+0000 7f9f4d127700 1 -- 172.21.15.113:0/841427934 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f44005040 con 0x7f9f480fe270 2024-04-12T05:36:50.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.702+0000 7f9f4d127700 1 --2- 172.21.15.113:0/841427934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f480fe270 0x7f9f4806b5c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f9f3800c710 tx=0x7f9f38007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:50.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.702+0000 7f9f3effd700 1 -- 172.21.15.113:0/841427934 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f3800e680 con 0x7f9f480fe270 2024-04-12T05:36:50.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.702+0000 7f9f3effd700 1 -- 172.21.15.113:0/841427934 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f3800ec90 con 0x7f9f480fe270 2024-04-12T05:36:50.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.702+0000 7f9f3effd700 1 -- 172.21.15.113:0/841427934 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f38013cd0 con 0x7f9f480fe270 2024-04-12T05:36:50.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.702+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/841427934 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f4806bea0 con 0x7f9f480fe270 2024-04-12T05:36:50.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.702+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/841427934 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f48114690 con 0x7f9f480fe270 2024-04-12T05:36:50.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.703+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/841427934 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f48060650 con 0x7f9f480fe270 2024-04-12T05:36:50.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.703+0000 7f9f3effd700 1 -- 172.21.15.113:0/841427934 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9f38013410 con 0x7f9f480fe270 2024-04-12T05:36:50.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.704+0000 7f9f3effd700 1 --2- 172.21.15.113:0/841427934 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3406caa0 0x7f9f3406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:50.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.704+0000 7f9f3effd700 1 -- 172.21.15.113:0/841427934 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9f3808c140 con 0x7f9f480fe270 2024-04-12T05:36:50.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.704+0000 7f9f4d928700 1 --2- 172.21.15.113:0/841427934 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3406caa0 0x7f9f3406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:50.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.704+0000 7f9f4d928700 1 --2- 172.21.15.113:0/841427934 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3406caa0 0x7f9f3406ef60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f9f44004f10 tx=0x7f9f4400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:50.706 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.708+0000 7f9f3effd700 1 -- 172.21.15.113:0/841427934 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9f38056160 con 0x7f9f480fe270 2024-04-12T05:36:50.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.902+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/841427934 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f4806c690 con 0x7f9f3406caa0 2024-04-12T05:36:50.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.911+0000 7f9f3effd700 1 -- 172.21.15.113:0/841427934 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9f4806c690 con 0x7f9f3406caa0 2024-04-12T05:36:50.910 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:36:50.910 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (24m) 2m ago 35m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:36:50.910 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (22m) 2m ago 35m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:36:50.910 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (22m) 2m ago 35m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:36:50.910 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (24m) 2m ago 35m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:36:50.910 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (17m) 2m ago 30m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:36:50.910 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (17m) 2m ago 30m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:36:50.910 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (17m) 2m ago 30m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:36:50.910 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (2m) 2m ago 30m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (26m) 2m ago 37m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (25m) 2m ago 35m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (23m) 2m ago 37m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (22m) 2m ago 35m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (25m) 2m ago 35m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (25m) 2m ago 35m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (21m) 2m ago 34m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (21m) 2m ago 33m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (20m) 2m ago 33m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (19m) 2m ago 33m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (19m) 2m ago 32m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (18m) 2m ago 32m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:36:50.911 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (24m) 2m ago 34m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:36:50.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.913+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/841427934 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3406caa0 msgr2=0x7f9f3406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.913+0000 7f9f4fb8c700 1 --2- 172.21.15.113:0/841427934 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3406caa0 0x7f9f3406ef60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f9f44004f10 tx=0x7f9f4400b040).stop 2024-04-12T05:36:50.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.913+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/841427934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f480fe270 msgr2=0x7f9f4806b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:50.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.913+0000 7f9f4fb8c700 1 --2- 172.21.15.113:0/841427934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f480fe270 0x7f9f4806b5c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f9f3800c710 tx=0x7f9f38007ec0).stop 2024-04-12T05:36:50.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.914+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/841427934 shutdown_connections 2024-04-12T05:36:50.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.914+0000 7f9f4fb8c700 1 --2- 172.21.15.113:0/841427934 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3406caa0 0x7f9f3406ef60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.914+0000 7f9f4fb8c700 1 --2- 172.21.15.113:0/841427934 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f480fd910 0x7f9f4806b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.914+0000 7f9f4fb8c700 1 --2- 172.21.15.113:0/841427934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f480fe270 0x7f9f4806b5c0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:50.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.914+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/841427934 >> 172.21.15.113:0/841427934 conn(0x7f9f480f3510 msgr2=0x7f9f480fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:50.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.914+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/841427934 shutdown_connections 2024-04-12T05:36:50.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:50.914+0000 7f9f4fb8c700 1 -- 172.21.15.113:0/841427934 wait complete. 2024-04-12T05:36:51.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.012+0000 7f9825166700 1 -- 172.21.15.113:0/265086448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fd8f0 msgr2=0x7f98200fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.012+0000 7f9825166700 1 --2- 172.21.15.113:0/265086448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fd8f0 0x7f98200fdcd0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f98140047f0 tx=0x7f9814009cb0).stop 2024-04-12T05:36:51.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.012+0000 7f9825166700 1 -- 172.21.15.113:0/265086448 shutdown_connections 2024-04-12T05:36:51.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.012+0000 7f9825166700 1 --2- 172.21.15.113:0/265086448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fe250 0x7f98200f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.012+0000 7f9825166700 1 --2- 172.21.15.113:0/265086448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fd8f0 0x7f98200fdcd0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.012+0000 7f9825166700 1 -- 172.21.15.113:0/265086448 >> 172.21.15.113:0/265086448 conn(0x7f98200f34b0 msgr2=0x7f98200f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:51.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.012+0000 7f9825166700 1 -- 172.21.15.113:0/265086448 shutdown_connections 2024-04-12T05:36:51.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.012+0000 7f9825166700 1 -- 172.21.15.113:0/265086448 wait complete. 2024-04-12T05:36:51.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.013+0000 7f9825166700 1 Processor -- start 2024-04-12T05:36:51.012 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.013+0000 7f9825166700 1 -- start start 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.013+0000 7f9825166700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fd8f0 0x7f982010e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.013+0000 7f9825166700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fe250 0x7f982010e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.013+0000 7f9825166700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982010ebe0 con 0x7f98200fd8f0 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.013+0000 7f9825166700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982010ed20 con 0x7f98200fe250 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.013+0000 7f981ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fd8f0 0x7f982010e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.013+0000 7f981ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fd8f0 0x7f982010e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55086/0 (socket says 172.21.15.113:55086) 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.013+0000 7f981ed9d700 1 -- 172.21.15.113:0/3966389390 learned_addr learned my addr 172.21.15.113:0/3966389390 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.013+0000 7f981e59c700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fe250 0x7f982010e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.014+0000 7f981ed9d700 1 -- 172.21.15.113:0/3966389390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fe250 msgr2=0x7f982010e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.014+0000 7f981ed9d700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fe250 0x7f982010e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.014+0000 7f981ed9d700 1 -- 172.21.15.113:0/3966389390 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9814005040 con 0x7f98200fd8f0 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.014+0000 7f981e59c700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fe250 0x7f982010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.014+0000 7f981ed9d700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fd8f0 0x7f982010e0f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f9814009380 tx=0x7f9814009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.014+0000 7f9807fff700 1 -- 172.21.15.113:0/3966389390 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9814018070 con 0x7f98200fd8f0 2024-04-12T05:36:51.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.014+0000 7f9825166700 1 -- 172.21.15.113:0/3966389390 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9820108350 con 0x7f98200fd8f0 2024-04-12T05:36:51.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.014+0000 7f9807fff700 1 -- 172.21.15.113:0/3966389390 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9814007e90 con 0x7f98200fd8f0 2024-04-12T05:36:51.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.014+0000 7f9807fff700 1 -- 172.21.15.113:0/3966389390 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98140123c0 con 0x7f98200fd8f0 2024-04-12T05:36:51.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.014+0000 7f9825166700 1 -- 172.21.15.113:0/3966389390 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98201088b0 con 0x7f98200fd8f0 2024-04-12T05:36:51.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.015+0000 7f9807fff700 1 -- 172.21.15.113:0/3966389390 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f981401d420 con 0x7f98200fd8f0 2024-04-12T05:36:51.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.015+0000 7f9825166700 1 -- 172.21.15.113:0/3966389390 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982005f590 con 0x7f98200fd8f0 2024-04-12T05:36:51.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.016+0000 7f9807fff700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 0x7f980806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:51.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.016+0000 7f9807fff700 1 -- 172.21.15.113:0/3966389390 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f981408cba0 con 0x7f98200fd8f0 2024-04-12T05:36:51.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.016+0000 7f981e59c700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 0x7f980806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:51.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.016+0000 7f981e59c700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 0x7f980806ef60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f9810007fd0 tx=0x7f981000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:51.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.020+0000 7f9807fff700 1 -- 172.21.15.113:0/3966389390 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f981405a5d0 con 0x7f98200fd8f0 2024-04-12T05:36:51.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.286+0000 7f9825166700 1 -- 172.21.15.113:0/3966389390 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9820109870 con 0x7f98200fd8f0 2024-04-12T05:36:51.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.287+0000 7f9807fff700 1 -- 172.21.15.113:0/3966389390 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f981405a160 con 0x7f98200fd8f0 2024-04-12T05:36:51.285 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:36:51.285 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:36:51.285 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:36:51.285 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:36:51.285 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:36:51.286 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:36:51.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.289+0000 7f9825166700 1 -- 172.21.15.113:0/3966389390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 msgr2=0x7f980806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.289+0000 7f9825166700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 0x7f980806ef60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f9810007fd0 tx=0x7f981000b040).stop 2024-04-12T05:36:51.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.289+0000 7f9825166700 1 -- 172.21.15.113:0/3966389390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fd8f0 msgr2=0x7f982010e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.289+0000 7f9825166700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fd8f0 0x7f982010e0f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f9814009380 tx=0x7f9814009cb0).stop 2024-04-12T05:36:51.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.289+0000 7f9825166700 1 -- 172.21.15.113:0/3966389390 shutdown_connections 2024-04-12T05:36:51.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.289+0000 7f9825166700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 0x7f980806ef60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.289+0000 7f9825166700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fd8f0 0x7f982010e0f0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.289+0000 7f9825166700 1 --2- 172.21.15.113:0/3966389390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fe250 0x7f982010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.289+0000 7f9825166700 1 -- 172.21.15.113:0/3966389390 >> 172.21.15.113:0/3966389390 conn(0x7f98200f34b0 msgr2=0x7f98200fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:51.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.289+0000 7f9825166700 1 -- 172.21.15.113:0/3966389390 shutdown_connections 2024-04-12T05:36:51.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.289+0000 7f9825166700 1 -- 172.21.15.113:0/3966389390 wait complete. 2024-04-12T05:36:51.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.386+0000 7fc5bf715700 1 -- 172.21.15.113:0/2826093495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80fc7f0 msgr2=0x7fc5b80fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.386+0000 7fc5bf715700 1 --2- 172.21.15.113:0/2826093495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80fc7f0 0x7fc5b80fcbd0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7fc5b40047f0 tx=0x7fc5b4009cb0).stop 2024-04-12T05:36:51.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.386+0000 7fc5bf715700 1 -- 172.21.15.113:0/2826093495 shutdown_connections 2024-04-12T05:36:51.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.386+0000 7fc5bf715700 1 --2- 172.21.15.113:0/2826093495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b80fd150 0x7fc5b80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.386+0000 7fc5bf715700 1 --2- 172.21.15.113:0/2826093495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80fc7f0 0x7fc5b80fcbd0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.386+0000 7fc5bf715700 1 -- 172.21.15.113:0/2826093495 >> 172.21.15.113:0/2826093495 conn(0x7fc5b8070590 msgr2=0x7fc5b80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:51.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.387+0000 7fc5bf715700 1 -- 172.21.15.113:0/2826093495 shutdown_connections 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.387+0000 7fc5bf715700 1 -- 172.21.15.113:0/2826093495 wait complete. 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.387+0000 7fc5bf715700 1 Processor -- start 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.387+0000 7fc5bf715700 1 -- start start 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.387+0000 7fc5bf715700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b80fc7f0 0x7fc5b810dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.387+0000 7fc5bf715700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80fd150 0x7fc5b810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.387+0000 7fc5bf715700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b810eae0 con 0x7fc5b80fd150 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.387+0000 7fc5bf715700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b810ec20 con 0x7fc5b80fc7f0 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.387+0000 7fc5bccb0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80fd150 0x7fc5b810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.387+0000 7fc5bccb0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80fd150 0x7fc5b810e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55100/0 (socket says 172.21.15.113:55100) 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.387+0000 7fc5bccb0700 1 -- 172.21.15.113:0/567554190 learned_addr learned my addr 172.21.15.113:0/567554190 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.388+0000 7fc5bd4b1700 1 --2- 172.21.15.113:0/567554190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b80fc7f0 0x7fc5b810dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.388+0000 7fc5bccb0700 1 -- 172.21.15.113:0/567554190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b80fc7f0 msgr2=0x7fc5b810dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.388+0000 7fc5bccb0700 1 --2- 172.21.15.113:0/567554190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b80fc7f0 0x7fc5b810dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.388+0000 7fc5bccb0700 1 -- 172.21.15.113:0/567554190 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b4005040 con 0x7fc5b80fd150 2024-04-12T05:36:51.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.388+0000 7fc5bccb0700 1 --2- 172.21.15.113:0/567554190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80fd150 0x7fc5b810e4e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7fc5a8002780 tx=0x7fc5a8009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:51.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.388+0000 7fc5ae7fc700 1 -- 172.21.15.113:0/567554190 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5a8007b40 con 0x7fc5b80fd150 2024-04-12T05:36:51.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.388+0000 7fc5ae7fc700 1 -- 172.21.15.113:0/567554190 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5a800f040 con 0x7fc5b80fd150 2024-04-12T05:36:51.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.388+0000 7fc5ae7fc700 1 -- 172.21.15.113:0/567554190 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5a8014780 con 0x7fc5b80fd150 2024-04-12T05:36:51.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.388+0000 7fc5bf715700 1 -- 172.21.15.113:0/567554190 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b81082b0 con 0x7fc5b80fd150 2024-04-12T05:36:51.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.388+0000 7fc5bf715700 1 -- 172.21.15.113:0/567554190 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b8108820 con 0x7fc5b80fd150 2024-04-12T05:36:51.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.389+0000 7fc5bf715700 1 -- 172.21.15.113:0/567554190 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b805f590 con 0x7fc5b80fd150 2024-04-12T05:36:51.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.389+0000 7fc5ae7fc700 1 -- 172.21.15.113:0/567554190 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc5a8007ca0 con 0x7fc5b80fd150 2024-04-12T05:36:51.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.390+0000 7fc5ae7fc700 1 --2- 172.21.15.113:0/567554190 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc5a406caf0 0x7fc5a406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:51.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.390+0000 7fc5ae7fc700 1 -- 172.21.15.113:0/567554190 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc5a808be10 con 0x7fc5b80fd150 2024-04-12T05:36:51.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.390+0000 7fc5bd4b1700 1 --2- 172.21.15.113:0/567554190 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc5a406caf0 0x7fc5a406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:51.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.390+0000 7fc5bd4b1700 1 --2- 172.21.15.113:0/567554190 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc5a406caf0 0x7fc5a406efb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fc5b4004f10 tx=0x7fc5b400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:51.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.394+0000 7fc5ae7fc700 1 -- 172.21.15.113:0/567554190 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc5a8059800 con 0x7fc5b80fd150 2024-04-12T05:36:51.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:50 smithi113 ceph-mon[103371]: pgmap v741: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:51.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:50 smithi192 ceph-mon[85304]: pgmap v741: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.623+0000 7fc5bf715700 1 -- 172.21.15.113:0/567554190 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc5b8108f10 con 0x7fc5b80fd150 2024-04-12T05:36:51.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.623+0000 7fc5ae7fc700 1 -- 172.21.15.113:0/567554190 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fc5a8059390 con 0x7fc5b80fd150 2024-04-12T05:36:51.621 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:36:51.622 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:36:51.622 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:36:51.622 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:36:51.622 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:36:51.622 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:36:51.622 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:36:51.622 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:36:51.622 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:36:51.622 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:36:51.622 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:36:51.623 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:36:51.623 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:36:51.623 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:36:51.623 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:36:51.623 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:36:51.623 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:36:51.623 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:36:51.623 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:36:51.623 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:36:51.623 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:36:51.623 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:36:51.624 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:36:51.625 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:36:51.625 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:36:51.625 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:36:51.625 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:36:51.625 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:36:51.625 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:36:51.625 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:36:51.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.625+0000 7fc5bf715700 1 -- 172.21.15.113:0/567554190 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc5a406caf0 msgr2=0x7fc5a406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.625+0000 7fc5bf715700 1 --2- 172.21.15.113:0/567554190 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc5a406caf0 0x7fc5a406efb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fc5b4004f10 tx=0x7fc5b400b040).stop 2024-04-12T05:36:51.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.625+0000 7fc5bf715700 1 -- 172.21.15.113:0/567554190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80fd150 msgr2=0x7fc5b810e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.625+0000 7fc5bf715700 1 --2- 172.21.15.113:0/567554190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80fd150 0x7fc5b810e4e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7fc5a8002780 tx=0x7fc5a8009ee0).stop 2024-04-12T05:36:51.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.625+0000 7fc5bf715700 1 -- 172.21.15.113:0/567554190 shutdown_connections 2024-04-12T05:36:51.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.625+0000 7fc5bf715700 1 --2- 172.21.15.113:0/567554190 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc5a406caf0 0x7fc5a406efb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.625+0000 7fc5bf715700 1 --2- 172.21.15.113:0/567554190 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b80fc7f0 0x7fc5b810dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.625+0000 7fc5bf715700 1 --2- 172.21.15.113:0/567554190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5b80fd150 0x7fc5b810e4e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.625+0000 7fc5bf715700 1 -- 172.21.15.113:0/567554190 >> 172.21.15.113:0/567554190 conn(0x7fc5b8070590 msgr2=0x7fc5b8060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:51.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.625+0000 7fc5bf715700 1 -- 172.21.15.113:0/567554190 shutdown_connections 2024-04-12T05:36:51.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.625+0000 7fc5bf715700 1 -- 172.21.15.113:0/567554190 wait complete. 2024-04-12T05:36:51.628 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:36:51.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.723+0000 7efc3b512700 1 -- 172.21.15.113:0/1411247723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc340f9200 msgr2=0x7efc34101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.723+0000 7efc3b512700 1 --2- 172.21.15.113:0/1411247723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc340f9200 0x7efc34101910 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7efc300047f0 tx=0x7efc30009cb0).stop 2024-04-12T05:36:51.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.723+0000 7efc3b512700 1 -- 172.21.15.113:0/1411247723 shutdown_connections 2024-04-12T05:36:51.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.723+0000 7efc3b512700 1 --2- 172.21.15.113:0/1411247723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc340f9200 0x7efc34101910 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.724 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.723+0000 7efc3b512700 1 --2- 172.21.15.113:0/1411247723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc340f88a0 0x7efc340f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.723+0000 7efc3b512700 1 -- 172.21.15.113:0/1411247723 >> 172.21.15.113:0/1411247723 conn(0x7efc34070590 msgr2=0x7efc340f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.723+0000 7efc3b512700 1 -- 172.21.15.113:0/1411247723 shutdown_connections 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.723+0000 7efc3b512700 1 -- 172.21.15.113:0/1411247723 wait complete. 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc3b512700 1 Processor -- start 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc3b512700 1 -- start start 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc3b512700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc340f88a0 0x7efc340f6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc3b512700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc340f9200 0x7efc340f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc3b512700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc340f7230 con 0x7efc340f9200 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc3b512700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc340f7370 con 0x7efc340f88a0 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc38aad700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc340f9200 0x7efc340f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc38aad700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc340f9200 0x7efc340f6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55124/0 (socket says 172.21.15.113:55124) 2024-04-12T05:36:51.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc38aad700 1 -- 172.21.15.113:0/3200439564 learned_addr learned my addr 172.21.15.113:0/3200439564 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:51.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc392ae700 1 --2- 172.21.15.113:0/3200439564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc340f88a0 0x7efc340f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:51.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc38aad700 1 -- 172.21.15.113:0/3200439564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc340f88a0 msgr2=0x7efc340f6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc38aad700 1 --2- 172.21.15.113:0/3200439564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc340f88a0 0x7efc340f6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.724+0000 7efc38aad700 1 -- 172.21.15.113:0/3200439564 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc30005040 con 0x7efc340f9200 2024-04-12T05:36:51.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.725+0000 7efc38aad700 1 --2- 172.21.15.113:0/3200439564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc340f9200 0x7efc340f6d40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7efc30005010 tx=0x7efc30009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:51.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.725+0000 7efc2a7fc700 1 -- 172.21.15.113:0/3200439564 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc3000e030 con 0x7efc340f9200 2024-04-12T05:36:51.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.725+0000 7efc2a7fc700 1 -- 172.21.15.113:0/3200439564 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc30007e90 con 0x7efc340f9200 2024-04-12T05:36:51.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.725+0000 7efc2a7fc700 1 -- 172.21.15.113:0/3200439564 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc30018070 con 0x7efc340f9200 2024-04-12T05:36:51.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.725+0000 7efc3b512700 1 -- 172.21.15.113:0/3200439564 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc340f75c0 con 0x7efc340f9200 2024-04-12T05:36:51.726 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.725+0000 7efc3b512700 1 -- 172.21.15.113:0/3200439564 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc341146a0 con 0x7efc340f9200 2024-04-12T05:36:51.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.726+0000 7efc3b512700 1 -- 172.21.15.113:0/3200439564 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3405f590 con 0x7efc340f9200 2024-04-12T05:36:51.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.726+0000 7efc2a7fc700 1 -- 172.21.15.113:0/3200439564 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7efc3001d670 con 0x7efc340f9200 2024-04-12T05:36:51.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.726+0000 7efc2a7fc700 1 --2- 172.21.15.113:0/3200439564 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc2006caa0 0x7efc2006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:51.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.727+0000 7efc2a7fc700 1 -- 172.21.15.113:0/3200439564 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7efc3008d0d0 con 0x7efc340f9200 2024-04-12T05:36:51.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.727+0000 7efc392ae700 1 --2- 172.21.15.113:0/3200439564 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc2006caa0 0x7efc2006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:51.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.727+0000 7efc392ae700 1 --2- 172.21.15.113:0/3200439564 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc2006caa0 0x7efc2006ef60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7efc24004de0 tx=0x7efc2400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:51.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.731+0000 7efc2a7fc700 1 -- 172.21.15.113:0/3200439564 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7efc300570f0 con 0x7efc340f9200 2024-04-12T05:36:51.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.939+0000 7efc3b512700 1 -- 172.21.15.113:0/3200439564 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efc34059120 con 0x7efc2006caa0 2024-04-12T05:36:51.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.940+0000 7efc2a7fc700 1 -- 172.21.15.113:0/3200439564 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7efc34059120 con 0x7efc2006caa0 2024-04-12T05:36:51.938 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:36:51.938 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:36:51.938 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:36:51.938 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:36:51.938 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:36:51.938 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:36:51.938 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:36:51.939 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:36:51.939 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:36:51.939 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:36:51.939 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:36:51.939 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:36:51.939 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:36:51.939 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:36:51.939 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:36:51.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.942+0000 7efc3b512700 1 -- 172.21.15.113:0/3200439564 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc2006caa0 msgr2=0x7efc2006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.942+0000 7efc3b512700 1 --2- 172.21.15.113:0/3200439564 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc2006caa0 0x7efc2006ef60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7efc24004de0 tx=0x7efc2400b040).stop 2024-04-12T05:36:51.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.942+0000 7efc3b512700 1 -- 172.21.15.113:0/3200439564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc340f9200 msgr2=0x7efc340f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:51.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.942+0000 7efc3b512700 1 --2- 172.21.15.113:0/3200439564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc340f9200 0x7efc340f6d40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7efc30005010 tx=0x7efc30009cb0).stop 2024-04-12T05:36:51.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.942+0000 7efc3b512700 1 -- 172.21.15.113:0/3200439564 shutdown_connections 2024-04-12T05:36:51.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.942+0000 7efc3b512700 1 --2- 172.21.15.113:0/3200439564 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7efc2006caa0 0x7efc2006ef60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.942+0000 7efc3b512700 1 --2- 172.21.15.113:0/3200439564 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc340f88a0 0x7efc340f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.942+0000 7efc3b512700 1 --2- 172.21.15.113:0/3200439564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc340f9200 0x7efc340f6d40 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:51.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.942+0000 7efc3b512700 1 -- 172.21.15.113:0/3200439564 >> 172.21.15.113:0/3200439564 conn(0x7efc34070590 msgr2=0x7efc340fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:51.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.942+0000 7efc3b512700 1 -- 172.21.15.113:0/3200439564 shutdown_connections 2024-04-12T05:36:51.941 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:51.942+0000 7efc3b512700 1 -- 172.21.15.113:0/3200439564 wait complete. 2024-04-12T05:36:52.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.043+0000 7faed0cf3700 1 -- 172.21.15.113:0/2963273873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faecc0fd1f0 msgr2=0x7faecc0f69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:52.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.043+0000 7faed0cf3700 1 --2- 172.21.15.113:0/2963273873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faecc0fd1f0 0x7faecc0f69d0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7faebc0047f0 tx=0x7faebc009cb0).stop 2024-04-12T05:36:52.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.043+0000 7faed0cf3700 1 -- 172.21.15.113:0/2963273873 shutdown_connections 2024-04-12T05:36:52.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.043+0000 7faed0cf3700 1 --2- 172.21.15.113:0/2963273873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faecc0fd1f0 0x7faecc0f69d0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:52.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.043+0000 7faed0cf3700 1 --2- 172.21.15.113:0/2963273873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faecc0fc890 0x7faecc0fcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:52.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.043+0000 7faed0cf3700 1 -- 172.21.15.113:0/2963273873 >> 172.21.15.113:0/2963273873 conn(0x7faecc0f2490 msgr2=0x7faecc0f48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:52.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.043+0000 7faed0cf3700 1 -- 172.21.15.113:0/2963273873 shutdown_connections 2024-04-12T05:36:52.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.043+0000 7faed0cf3700 1 -- 172.21.15.113:0/2963273873 wait complete. 2024-04-12T05:36:52.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faed0cf3700 1 Processor -- start 2024-04-12T05:36:52.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faed0cf3700 1 -- start start 2024-04-12T05:36:52.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faed0cf3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faecc0fc890 0x7faecc10e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:52.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faed0cf3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faecc0fd1f0 0x7faecc10e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:52.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faed0cf3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc10eb00 con 0x7faecc0fc890 2024-04-12T05:36:52.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faed0cf3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc10ec40 con 0x7faecc0fd1f0 2024-04-12T05:36:52.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faeca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faecc0fc890 0x7faecc10e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:52.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faeca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faecc0fc890 0x7faecc10e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55144/0 (socket says 172.21.15.113:55144) 2024-04-12T05:36:52.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faeca59c700 1 -- 172.21.15.113:0/582079132 learned_addr learned my addr 172.21.15.113:0/582079132 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:36:52.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faec9d9b700 1 --2- 172.21.15.113:0/582079132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faecc0fd1f0 0x7faecc10e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:52.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faeca59c700 1 -- 172.21.15.113:0/582079132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faecc0fd1f0 msgr2=0x7faecc10e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:52.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faeca59c700 1 --2- 172.21.15.113:0/582079132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faecc0fd1f0 0x7faecc10e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:52.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faeca59c700 1 -- 172.21.15.113:0/582079132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc005040 con 0x7faecc0fc890 2024-04-12T05:36:52.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faeca59c700 1 --2- 172.21.15.113:0/582079132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faecc0fc890 0x7faecc10e010 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7faec00041d0 tx=0x7faec0009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:52.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.044+0000 7faebb7fe700 1 -- 172.21.15.113:0/582079132 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faec0002d90 con 0x7faecc0fc890 2024-04-12T05:36:52.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.045+0000 7faebb7fe700 1 -- 172.21.15.113:0/582079132 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faec000f040 con 0x7faecc0fc890 2024-04-12T05:36:52.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.045+0000 7faed0cf3700 1 -- 172.21.15.113:0/582079132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc1082d0 con 0x7faecc0fc890 2024-04-12T05:36:52.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.045+0000 7faebb7fe700 1 -- 172.21.15.113:0/582079132 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faec000d800 con 0x7faecc0fc890 2024-04-12T05:36:52.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.045+0000 7faed0cf3700 1 -- 172.21.15.113:0/582079132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc108890 con 0x7faecc0fc890 2024-04-12T05:36:52.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.045+0000 7faed0cf3700 1 -- 172.21.15.113:0/582079132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faecc060330 con 0x7faecc0fc890 2024-04-12T05:36:52.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.046+0000 7faebb7fe700 1 -- 172.21.15.113:0/582079132 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faec000dac0 con 0x7faecc0fc890 2024-04-12T05:36:52.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.046+0000 7faebb7fe700 1 --2- 172.21.15.113:0/582079132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeb406caa0 0x7faeb406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:36:52.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.047+0000 7faebb7fe700 1 -- 172.21.15.113:0/582079132 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7faec008bea0 con 0x7faecc0fc890 2024-04-12T05:36:52.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.047+0000 7faec9d9b700 1 --2- 172.21.15.113:0/582079132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeb406caa0 0x7faeb406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:36:52.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.047+0000 7faec9d9b700 1 --2- 172.21.15.113:0/582079132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeb406caa0 0x7faeb406ef60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7faebc004f10 tx=0x7faebc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:36:52.049 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.051+0000 7faebb7fe700 1 -- 172.21.15.113:0/582079132 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7faec0059950 con 0x7faecc0fc890 2024-04-12T05:36:52.265 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:51 smithi113 ceph-mon[103371]: from='client.35346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:52.265 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:51 smithi113 ceph-mon[103371]: from='client.35350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:52.265 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:51 smithi113 ceph-mon[103371]: from='client.35354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:52.265 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:51 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3966389390' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:36:52.265 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:51 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/567554190' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:36:52.265 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:51 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:52.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.315+0000 7faed0cf3700 1 -- 172.21.15.113:0/582079132 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faecc047540 con 0x7faecc0fc890 2024-04-12T05:36:52.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.315+0000 7faebb7fe700 1 -- 172.21.15.113:0/582079132 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7faec00594e0 con 0x7faecc0fc890 2024-04-12T05:36:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:36:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:36:52.314 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:36:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:36:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:36:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:36:52.314 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:36:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:36:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:36:52.316 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:36:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:36:52.316 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:36:52.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.317+0000 7faed0cf3700 1 -- 172.21.15.113:0/582079132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeb406caa0 msgr2=0x7faeb406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:52.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.317+0000 7faed0cf3700 1 --2- 172.21.15.113:0/582079132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeb406caa0 0x7faeb406ef60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7faebc004f10 tx=0x7faebc00b040).stop 2024-04-12T05:36:52.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.317+0000 7faed0cf3700 1 -- 172.21.15.113:0/582079132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faecc0fc890 msgr2=0x7faecc10e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:36:52.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.317+0000 7faed0cf3700 1 --2- 172.21.15.113:0/582079132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faecc0fc890 0x7faecc10e010 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7faec00041d0 tx=0x7faec0009f60).stop 2024-04-12T05:36:52.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.317+0000 7faed0cf3700 1 -- 172.21.15.113:0/582079132 shutdown_connections 2024-04-12T05:36:52.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.317+0000 7faed0cf3700 1 --2- 172.21.15.113:0/582079132 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faeb406caa0 0x7faeb406ef60 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:52.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.317+0000 7faed0cf3700 1 --2- 172.21.15.113:0/582079132 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faecc0fc890 0x7faecc10e010 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:52.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.317+0000 7faed0cf3700 1 --2- 172.21.15.113:0/582079132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faecc0fd1f0 0x7faecc10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:36:52.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.318+0000 7faed0cf3700 1 -- 172.21.15.113:0/582079132 >> 172.21.15.113:0/582079132 conn(0x7faecc0f2490 msgr2=0x7faecc061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:36:52.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.318+0000 7faed0cf3700 1 -- 172.21.15.113:0/582079132 shutdown_connections 2024-04-12T05:36:52.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:36:52.318+0000 7faed0cf3700 1 -- 172.21.15.113:0/582079132 wait complete. 2024-04-12T05:36:52.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:51 smithi192 ceph-mon[85304]: from='client.35346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:52.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:51 smithi192 ceph-mon[85304]: from='client.35350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:52.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:51 smithi192 ceph-mon[85304]: from='client.35354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:52.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:51 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3966389390' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:36:52.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:51 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/567554190' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:36:52.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:51 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:36:53.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:52 smithi113 ceph-mon[103371]: pgmap v742: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:53.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:52 smithi113 ceph-mon[103371]: from='client.35366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:53.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:52 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/582079132' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:36:53.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:52 smithi192 ceph-mon[85304]: pgmap v742: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:53.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:52 smithi192 ceph-mon[85304]: from='client.35366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:36:53.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:52 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/582079132' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:36:55.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:54 smithi113 ceph-mon[103371]: pgmap v743: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:55.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:54 smithi192 ceph-mon[85304]: pgmap v743: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:57.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:56 smithi113 ceph-mon[103371]: pgmap v744: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:57.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:56 smithi192 ceph-mon[85304]: pgmap v744: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:59.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:36:58 smithi113 ceph-mon[103371]: pgmap v745: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:36:59.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:36:58 smithi192 ceph-mon[85304]: pgmap v745: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:01.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:00 smithi113 ceph-mon[103371]: pgmap v746: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:01.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:00 smithi192 ceph-mon[85304]: pgmap v746: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:03.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:02 smithi113 ceph-mon[103371]: pgmap v747: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:03.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:02 smithi192 ceph-mon[85304]: pgmap v747: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:05.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:04 smithi113 ceph-mon[103371]: pgmap v748: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:05.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:04 smithi192 ceph-mon[85304]: pgmap v748: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:07.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:06 smithi113 ceph-mon[103371]: pgmap v749: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:07.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:07.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:06 smithi192 ceph-mon[85304]: pgmap v749: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:07.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:09.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:08 smithi113 ceph-mon[103371]: pgmap v750: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:09.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:08 smithi192 ceph-mon[85304]: pgmap v750: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:11.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:11 smithi113 ceph-mon[103371]: pgmap v751: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:11.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:11 smithi192 ceph-mon[85304]: pgmap v751: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:13.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:13 smithi113 ceph-mon[103371]: pgmap v752: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:13.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:13 smithi192 ceph-mon[85304]: pgmap v752: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:15.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:15 smithi113 ceph-mon[103371]: pgmap v753: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:15.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:15 smithi192 ceph-mon[85304]: pgmap v753: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:17.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:17 smithi113 ceph-mon[103371]: pgmap v754: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:17.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:17 smithi192 ceph-mon[85304]: pgmap v754: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:19.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:19 smithi113 ceph-mon[103371]: pgmap v755: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:19.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:19 smithi192 ceph-mon[85304]: pgmap v755: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:21.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:21 smithi113 ceph-mon[103371]: pgmap v756: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:21.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:21 smithi192 ceph-mon[85304]: pgmap v756: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:22.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:22.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.428+0000 7f4b43b7d700 1 -- 172.21.15.113:0/1544243859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fd8f0 msgr2=0x7f4b3c0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:22.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.428+0000 7f4b43b7d700 1 --2- 172.21.15.113:0/1544243859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fd8f0 0x7f4b3c0fdcd0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f4b300047f0 tx=0x7f4b30009cb0).stop 2024-04-12T05:37:22.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.428+0000 7f4b43b7d700 1 -- 172.21.15.113:0/1544243859 shutdown_connections 2024-04-12T05:37:22.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.428+0000 7f4b43b7d700 1 --2- 172.21.15.113:0/1544243859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c0fe250 0x7f4b3c0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.428+0000 7f4b43b7d700 1 --2- 172.21.15.113:0/1544243859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fd8f0 0x7f4b3c0fdcd0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.428+0000 7f4b43b7d700 1 -- 172.21.15.113:0/1544243859 >> 172.21.15.113:0/1544243859 conn(0x7f4b3c0f34b0 msgr2=0x7f4b3c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:22.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.428+0000 7f4b43b7d700 1 -- 172.21.15.113:0/1544243859 shutdown_connections 2024-04-12T05:37:22.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.428+0000 7f4b43b7d700 1 -- 172.21.15.113:0/1544243859 wait complete. 2024-04-12T05:37:22.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.429+0000 7f4b43b7d700 1 Processor -- start 2024-04-12T05:37:22.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.429+0000 7f4b43b7d700 1 -- start start 2024-04-12T05:37:22.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.429+0000 7f4b43b7d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fd8f0 0x7f4b3c10dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:22.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.429+0000 7f4b43b7d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c0fe250 0x7f4b3c10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:22.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.429+0000 7f4b43b7d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c10ead0 con 0x7f4b3c0fd8f0 2024-04-12T05:37:22.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.429+0000 7f4b43b7d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c10ec10 con 0x7f4b3c0fe250 2024-04-12T05:37:22.429 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.429+0000 7f4b41919700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fd8f0 0x7f4b3c10dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:22.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.429+0000 7f4b41919700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fd8f0 0x7f4b3c10dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50534/0 (socket says 172.21.15.113:50534) 2024-04-12T05:37:22.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.429+0000 7f4b41919700 1 -- 172.21.15.113:0/629892676 learned_addr learned my addr 172.21.15.113:0/629892676 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:22.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.429+0000 7f4b41118700 1 --2- 172.21.15.113:0/629892676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c0fe250 0x7f4b3c10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:22.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.430+0000 7f4b41919700 1 -- 172.21.15.113:0/629892676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c0fe250 msgr2=0x7f4b3c10e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:22.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.430+0000 7f4b41919700 1 --2- 172.21.15.113:0/629892676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c0fe250 0x7f4b3c10e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.430+0000 7f4b41919700 1 -- 172.21.15.113:0/629892676 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b30005040 con 0x7f4b3c0fd8f0 2024-04-12T05:37:22.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.430+0000 7f4b41919700 1 --2- 172.21.15.113:0/629892676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fd8f0 0x7f4b3c10dfe0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f4b30005010 tx=0x7f4b30009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:22.430 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.430+0000 7f4b2effd700 1 -- 172.21.15.113:0/629892676 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b3000e030 con 0x7f4b3c0fd8f0 2024-04-12T05:37:22.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.430+0000 7f4b2effd700 1 -- 172.21.15.113:0/629892676 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b30007e90 con 0x7f4b3c0fd8f0 2024-04-12T05:37:22.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.430+0000 7f4b43b7d700 1 -- 172.21.15.113:0/629892676 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c108240 con 0x7f4b3c0fd8f0 2024-04-12T05:37:22.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.430+0000 7f4b2effd700 1 -- 172.21.15.113:0/629892676 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b30018070 con 0x7f4b3c0fd8f0 2024-04-12T05:37:22.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.430+0000 7f4b43b7d700 1 -- 172.21.15.113:0/629892676 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c108720 con 0x7f4b3c0fd8f0 2024-04-12T05:37:22.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.431+0000 7f4b43b7d700 1 -- 172.21.15.113:0/629892676 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c05f590 con 0x7f4b3c0fd8f0 2024-04-12T05:37:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.431+0000 7f4b2effd700 1 -- 172.21.15.113:0/629892676 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4b3001d670 con 0x7f4b3c0fd8f0 2024-04-12T05:37:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.432+0000 7f4b2effd700 1 --2- 172.21.15.113:0/629892676 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4b2806caa0 0x7f4b2806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.432+0000 7f4b2effd700 1 -- 172.21.15.113:0/629892676 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f4b3008d0d0 con 0x7f4b3c0fd8f0 2024-04-12T05:37:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.432+0000 7f4b41118700 1 --2- 172.21.15.113:0/629892676 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4b2806caa0 0x7f4b2806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:22.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.432+0000 7f4b41118700 1 --2- 172.21.15.113:0/629892676 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4b2806caa0 0x7f4b2806ef60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f4b3c109920 tx=0x7f4b3800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:22.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.436+0000 7f4b2effd700 1 -- 172.21.15.113:0/629892676 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4b3005ab80 con 0x7f4b3c0fd8f0 2024-04-12T05:37:22.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:22.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.639+0000 7f4b43b7d700 1 -- 172.21.15.113:0/629892676 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b3c0ffc40 con 0x7f4b2806caa0 2024-04-12T05:37:22.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.640+0000 7f4b2effd700 1 -- 172.21.15.113:0/629892676 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4b3c0ffc40 con 0x7f4b2806caa0 2024-04-12T05:37:22.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.642+0000 7f4b43b7d700 1 -- 172.21.15.113:0/629892676 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4b2806caa0 msgr2=0x7f4b2806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:22.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.642+0000 7f4b43b7d700 1 --2- 172.21.15.113:0/629892676 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4b2806caa0 0x7f4b2806ef60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f4b3c109920 tx=0x7f4b3800b040).stop 2024-04-12T05:37:22.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.642+0000 7f4b43b7d700 1 -- 172.21.15.113:0/629892676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fd8f0 msgr2=0x7f4b3c10dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:22.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.642+0000 7f4b43b7d700 1 --2- 172.21.15.113:0/629892676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fd8f0 0x7f4b3c10dfe0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f4b30005010 tx=0x7f4b30009cb0).stop 2024-04-12T05:37:22.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.643+0000 7f4b43b7d700 1 -- 172.21.15.113:0/629892676 shutdown_connections 2024-04-12T05:37:22.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.643+0000 7f4b43b7d700 1 --2- 172.21.15.113:0/629892676 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4b2806caa0 0x7f4b2806ef60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.643+0000 7f4b43b7d700 1 --2- 172.21.15.113:0/629892676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b3c0fd8f0 0x7f4b3c10dfe0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.643+0000 7f4b43b7d700 1 --2- 172.21.15.113:0/629892676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b3c0fe250 0x7f4b3c10e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.643+0000 7f4b43b7d700 1 -- 172.21.15.113:0/629892676 >> 172.21.15.113:0/629892676 conn(0x7f4b3c0f34b0 msgr2=0x7f4b3c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:22.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.643+0000 7f4b43b7d700 1 -- 172.21.15.113:0/629892676 shutdown_connections 2024-04-12T05:37:22.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.643+0000 7f4b43b7d700 1 -- 172.21.15.113:0/629892676 wait complete. 2024-04-12T05:37:22.650 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:37:22.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.741+0000 7fa0d0d43700 1 -- 172.21.15.113:0/3469002309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0fd8f0 msgr2=0x7fa0cc0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:22.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.741+0000 7fa0d0d43700 1 --2- 172.21.15.113:0/3469002309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0fd8f0 0x7fa0cc0fdcd0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7fa0c00047f0 tx=0x7fa0c0009cb0).stop 2024-04-12T05:37:22.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.741+0000 7fa0d0d43700 1 -- 172.21.15.113:0/3469002309 shutdown_connections 2024-04-12T05:37:22.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.741+0000 7fa0d0d43700 1 --2- 172.21.15.113:0/3469002309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0fe250 0x7fa0cc0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.741+0000 7fa0d0d43700 1 --2- 172.21.15.113:0/3469002309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0fd8f0 0x7fa0cc0fdcd0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.741+0000 7fa0d0d43700 1 -- 172.21.15.113:0/3469002309 >> 172.21.15.113:0/3469002309 conn(0x7fa0cc0f34b0 msgr2=0x7fa0cc0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:22.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.741+0000 7fa0d0d43700 1 -- 172.21.15.113:0/3469002309 shutdown_connections 2024-04-12T05:37:22.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.741+0000 7fa0d0d43700 1 -- 172.21.15.113:0/3469002309 wait complete. 2024-04-12T05:37:22.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.742+0000 7fa0d0d43700 1 Processor -- start 2024-04-12T05:37:22.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.742+0000 7fa0d0d43700 1 -- start start 2024-04-12T05:37:22.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.742+0000 7fa0d0d43700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0fd8f0 0x7fa0cc06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:22.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.742+0000 7fa0d0d43700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0fe250 0x7fa0cc06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:22.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.742+0000 7fa0d0d43700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc06bab0 con 0x7fa0cc0fd8f0 2024-04-12T05:37:22.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.742+0000 7fa0d0d43700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc06bbf0 con 0x7fa0cc0fe250 2024-04-12T05:37:22.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.742+0000 7fa0ca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0fd8f0 0x7fa0cc06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:22.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.742+0000 7fa0ca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0fd8f0 0x7fa0cc06b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50550/0 (socket says 172.21.15.113:50550) 2024-04-12T05:37:22.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.742+0000 7fa0ca59c700 1 -- 172.21.15.113:0/2406133565 learned_addr learned my addr 172.21.15.113:0/2406133565 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:22.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.743+0000 7fa0c9d9b700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0fe250 0x7fa0cc06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:22.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.743+0000 7fa0ca59c700 1 -- 172.21.15.113:0/2406133565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0fe250 msgr2=0x7fa0cc06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:22.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.743+0000 7fa0ca59c700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0fe250 0x7fa0cc06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.743+0000 7fa0ca59c700 1 -- 172.21.15.113:0/2406133565 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c0005040 con 0x7fa0cc0fd8f0 2024-04-12T05:37:22.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.743+0000 7fa0c9d9b700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0fe250 0x7fa0cc06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:37:22.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.743+0000 7fa0ca59c700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0fd8f0 0x7fa0cc06b0d0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7fa0c0005010 tx=0x7fa0c0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:22.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.743+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2406133565 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0c000e030 con 0x7fa0cc0fd8f0 2024-04-12T05:37:22.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.743+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2406133565 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0c0007e90 con 0x7fa0cc0fd8f0 2024-04-12T05:37:22.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.743+0000 7fa0d0d43700 1 -- 172.21.15.113:0/2406133565 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0cc06be40 con 0x7fa0cc0fd8f0 2024-04-12T05:37:22.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.743+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2406133565 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0c0018070 con 0x7fa0cc0fd8f0 2024-04-12T05:37:22.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.743+0000 7fa0d0d43700 1 -- 172.21.15.113:0/2406133565 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc114610 con 0x7fa0cc0fd8f0 2024-04-12T05:37:22.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.744+0000 7fa0d0d43700 1 -- 172.21.15.113:0/2406133565 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0cc05f590 con 0x7fa0cc0fd8f0 2024-04-12T05:37:22.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.745+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2406133565 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa0c001d6e0 con 0x7fa0cc0fd8f0 2024-04-12T05:37:22.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.745+0000 7fa0bb7fe700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa0b406ca30 0x7fa0b406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:22.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.745+0000 7fa0c9d9b700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa0b406ca30 0x7fa0b406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:22.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.745+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2406133565 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa0c008d100 con 0x7fa0cc0fd8f0 2024-04-12T05:37:22.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.745+0000 7fa0c9d9b700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa0b406ca30 0x7fa0b406eef0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fa0cc06c870 tx=0x7fa0bc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:22.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.750+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2406133565 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa0c005abb0 con 0x7fa0cc0fd8f0 2024-04-12T05:37:22.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.959+0000 7fa0d0d43700 1 -- 172.21.15.113:0/2406133565 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0cc059120 con 0x7fa0b406ca30 2024-04-12T05:37:22.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.960+0000 7fa0bb7fe700 1 -- 172.21.15.113:0/2406133565 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa0cc059120 con 0x7fa0b406ca30 2024-04-12T05:37:22.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.962+0000 7fa0d0d43700 1 -- 172.21.15.113:0/2406133565 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa0b406ca30 msgr2=0x7fa0b406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:22.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.962+0000 7fa0d0d43700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa0b406ca30 0x7fa0b406eef0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fa0cc06c870 tx=0x7fa0bc00b040).stop 2024-04-12T05:37:22.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.962+0000 7fa0d0d43700 1 -- 172.21.15.113:0/2406133565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0fd8f0 msgr2=0x7fa0cc06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:22.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.962+0000 7fa0d0d43700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0fd8f0 0x7fa0cc06b0d0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7fa0c0005010 tx=0x7fa0c0009cb0).stop 2024-04-12T05:37:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.962+0000 7fa0d0d43700 1 -- 172.21.15.113:0/2406133565 shutdown_connections 2024-04-12T05:37:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.962+0000 7fa0d0d43700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa0b406ca30 0x7fa0b406eef0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.962+0000 7fa0d0d43700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0cc0fd8f0 0x7fa0cc06b0d0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.962+0000 7fa0d0d43700 1 --2- 172.21.15.113:0/2406133565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0cc0fe250 0x7fa0cc06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.962+0000 7fa0d0d43700 1 -- 172.21.15.113:0/2406133565 >> 172.21.15.113:0/2406133565 conn(0x7fa0cc0f34b0 msgr2=0x7fa0cc0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.962+0000 7fa0d0d43700 1 -- 172.21.15.113:0/2406133565 shutdown_connections 2024-04-12T05:37:22.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:22.962+0000 7fa0d0d43700 1 -- 172.21.15.113:0/2406133565 wait complete. 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.062+0000 7f0518ad8700 1 -- 172.21.15.113:0/3371492035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05140fd8f0 msgr2=0x7f05140fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.062+0000 7f0518ad8700 1 --2- 172.21.15.113:0/3371492035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05140fd8f0 0x7f05140fdcd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f05080047f0 tx=0x7f0508009cb0).stop 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.062+0000 7f0518ad8700 1 -- 172.21.15.113:0/3371492035 shutdown_connections 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.062+0000 7f0518ad8700 1 --2- 172.21.15.113:0/3371492035 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05140fe250 0x7f05140f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.062+0000 7f0518ad8700 1 --2- 172.21.15.113:0/3371492035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05140fd8f0 0x7f05140fdcd0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.062+0000 7f0518ad8700 1 -- 172.21.15.113:0/3371492035 >> 172.21.15.113:0/3371492035 conn(0x7f05140f34b0 msgr2=0x7f05140f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.062+0000 7f0518ad8700 1 -- 172.21.15.113:0/3371492035 shutdown_connections 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.062+0000 7f0518ad8700 1 -- 172.21.15.113:0/3371492035 wait complete. 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0518ad8700 1 Processor -- start 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0518ad8700 1 -- start start 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0518ad8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05140fd8f0 0x7f051410e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:23.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0518ad8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05140fe250 0x7f051410e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0518ad8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051410eaf0 con 0x7f05140fe250 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0518ad8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051410ec30 con 0x7f05140fd8f0 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0511d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05140fe250 0x7f051410e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0511d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05140fe250 0x7f051410e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50566/0 (socket says 172.21.15.113:50566) 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0511d9b700 1 -- 172.21.15.113:0/2116004162 learned_addr learned my addr 172.21.15.113:0/2116004162 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f051259c700 1 --2- 172.21.15.113:0/2116004162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05140fd8f0 0x7f051410e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0511d9b700 1 -- 172.21.15.113:0/2116004162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05140fd8f0 msgr2=0x7f051410e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0511d9b700 1 --2- 172.21.15.113:0/2116004162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05140fd8f0 0x7f051410e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0511d9b700 1 -- 172.21.15.113:0/2116004162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0508005040 con 0x7f05140fe250 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f0511d9b700 1 --2- 172.21.15.113:0/2116004162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05140fe250 0x7f051410e4f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f0504002820 tx=0x7f0504009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f05037fe700 1 -- 172.21.15.113:0/2116004162 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0504007b40 con 0x7f05140fe250 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.063+0000 7f05037fe700 1 -- 172.21.15.113:0/2116004162 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f050400f040 con 0x7f05140fe250 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.064+0000 7f0518ad8700 1 -- 172.21.15.113:0/2116004162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05141082c0 con 0x7f05140fe250 2024-04-12T05:37:23.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.064+0000 7f05037fe700 1 -- 172.21.15.113:0/2116004162 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05040148b0 con 0x7f05140fe250 2024-04-12T05:37:23.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.064+0000 7f0518ad8700 1 -- 172.21.15.113:0/2116004162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0514108880 con 0x7f05140fe250 2024-04-12T05:37:23.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.064+0000 7f0518ad8700 1 -- 172.21.15.113:0/2116004162 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f051405f590 con 0x7f05140fe250 2024-04-12T05:37:23.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.065+0000 7f05037fe700 1 -- 172.21.15.113:0/2116004162 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0504007ca0 con 0x7f05140fe250 2024-04-12T05:37:23.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.065+0000 7f05037fe700 1 --2- 172.21.15.113:0/2116004162 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04fc06caa0 0x7f04fc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:23.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.065+0000 7f05037fe700 1 -- 172.21.15.113:0/2116004162 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f050408b8f0 con 0x7f05140fe250 2024-04-12T05:37:23.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.065+0000 7f051259c700 1 --2- 172.21.15.113:0/2116004162 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04fc06caa0 0x7f04fc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:23.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.066+0000 7f051259c700 1 --2- 172.21.15.113:0/2116004162 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04fc06caa0 0x7f04fc06ef60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f0508004f10 tx=0x7f050800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:23.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.070+0000 7f05037fe700 1 -- 172.21.15.113:0/2116004162 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f05040593a0 con 0x7f05140fe250 2024-04-12T05:37:23.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.269+0000 7f0518ad8700 1 -- 172.21.15.113:0/2116004162 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0514108b00 con 0x7f04fc06caa0 2024-04-12T05:37:23.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.279+0000 7f05037fe700 1 -- 172.21.15.113:0/2116004162 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0514108b00 con 0x7f04fc06caa0 2024-04-12T05:37:23.277 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:37:23.277 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (25m) 2m ago 36m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:37:23.277 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (22m) 2m ago 36m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:37:23.277 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (22m) 2m ago 35m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:37:23.277 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (24m) 2m ago 36m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:37:23.277 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (18m) 2m ago 31m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (18m) 2m ago 31m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (17m) 2m ago 31m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (2m) 2m ago 30m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (26m) 2m ago 37m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (26m) 2m ago 35m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (23m) 2m ago 37m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (23m) 2m ago 35m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (25m) 2m ago 36m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (25m) 2m ago 35m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (22m) 2m ago 35m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (21m) 2m ago 34m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (21m) 2m ago 33m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:37:23.278 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (20m) 2m ago 33m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:37:23.279 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (19m) 2m ago 33m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:37:23.279 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (18m) 2m ago 32m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:37:23.279 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (25m) 2m ago 35m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:37:23.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.281+0000 7f0518ad8700 1 -- 172.21.15.113:0/2116004162 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04fc06caa0 msgr2=0x7f04fc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.281+0000 7f0518ad8700 1 --2- 172.21.15.113:0/2116004162 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04fc06caa0 0x7f04fc06ef60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f0508004f10 tx=0x7f050800b040).stop 2024-04-12T05:37:23.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.281+0000 7f0518ad8700 1 -- 172.21.15.113:0/2116004162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05140fe250 msgr2=0x7f051410e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.281+0000 7f0518ad8700 1 --2- 172.21.15.113:0/2116004162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05140fe250 0x7f051410e4f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f0504002820 tx=0x7f0504009f20).stop 2024-04-12T05:37:23.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.281+0000 7f0518ad8700 1 -- 172.21.15.113:0/2116004162 shutdown_connections 2024-04-12T05:37:23.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.281+0000 7f0518ad8700 1 --2- 172.21.15.113:0/2116004162 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04fc06caa0 0x7f04fc06ef60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.281+0000 7f0518ad8700 1 --2- 172.21.15.113:0/2116004162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05140fd8f0 0x7f051410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.281+0000 7f0518ad8700 1 --2- 172.21.15.113:0/2116004162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05140fe250 0x7f051410e4f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.281+0000 7f0518ad8700 1 -- 172.21.15.113:0/2116004162 >> 172.21.15.113:0/2116004162 conn(0x7f05140f34b0 msgr2=0x7f05140fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:23.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.281+0000 7f0518ad8700 1 -- 172.21.15.113:0/2116004162 shutdown_connections 2024-04-12T05:37:23.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.281+0000 7f0518ad8700 1 -- 172.21.15.113:0/2116004162 wait complete. 2024-04-12T05:37:23.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.378+0000 7fbeb8359700 1 -- 172.21.15.113:0/3162928536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb00fd150 msgr2=0x7fbeb00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.378+0000 7fbeb8359700 1 --2- 172.21.15.113:0/3162928536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb00fd150 0x7fbeb00f6930 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fbea40047f0 tx=0x7fbea4009cb0).stop 2024-04-12T05:37:23.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.378+0000 7fbeb8359700 1 -- 172.21.15.113:0/3162928536 shutdown_connections 2024-04-12T05:37:23.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.378+0000 7fbeb8359700 1 --2- 172.21.15.113:0/3162928536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb00fd150 0x7fbeb00f6930 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.378+0000 7fbeb8359700 1 --2- 172.21.15.113:0/3162928536 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbeb00fc7f0 0x7fbeb00fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.378+0000 7fbeb8359700 1 -- 172.21.15.113:0/3162928536 >> 172.21.15.113:0/3162928536 conn(0x7fbeb00f23b0 msgr2=0x7fbeb00f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:23.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.378+0000 7fbeb8359700 1 -- 172.21.15.113:0/3162928536 shutdown_connections 2024-04-12T05:37:23.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.378+0000 7fbeb8359700 1 -- 172.21.15.113:0/3162928536 wait complete. 2024-04-12T05:37:23.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.379+0000 7fbeb8359700 1 Processor -- start 2024-04-12T05:37:23.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.379+0000 7fbeb8359700 1 -- start start 2024-04-12T05:37:23.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.379+0000 7fbeb8359700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb00fc7f0 0x7fbeb0109160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:23.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.379+0000 7fbeb8359700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbeb00fd150 0x7fbeb0109650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:23.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.379+0000 7fbeb8359700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0109b40 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.379+0000 7fbeb8359700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0109c80 con 0x7fbeb00fd150 2024-04-12T05:37:23.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.379+0000 7fbeb60f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb00fc7f0 0x7fbeb0109160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:23.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.379+0000 7fbeb60f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb00fc7f0 0x7fbeb0109160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50582/0 (socket says 172.21.15.113:50582) 2024-04-12T05:37:23.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.379+0000 7fbeb60f5700 1 -- 172.21.15.113:0/2127110094 learned_addr learned my addr 172.21.15.113:0/2127110094 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:23.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.380+0000 7fbeb58f4700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbeb00fd150 0x7fbeb0109650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:23.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.380+0000 7fbeb60f5700 1 -- 172.21.15.113:0/2127110094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbeb00fd150 msgr2=0x7fbeb0109650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.380+0000 7fbeb60f5700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbeb00fd150 0x7fbeb0109650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.380+0000 7fbeb60f5700 1 -- 172.21.15.113:0/2127110094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea4005040 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.380+0000 7fbeb58f4700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbeb00fd150 0x7fbeb0109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:37:23.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.380+0000 7fbeb60f5700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb00fc7f0 0x7fbeb0109160 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7fbeac00c740 tx=0x7fbeac007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:23.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.380+0000 7fbea37fe700 1 -- 172.21.15.113:0/2127110094 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeac00e680 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.380+0000 7fbea37fe700 1 -- 172.21.15.113:0/2127110094 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbeac00ec90 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.380+0000 7fbeb8359700 1 -- 172.21.15.113:0/2127110094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb0109f30 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.380+0000 7fbea37fe700 1 -- 172.21.15.113:0/2127110094 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeac013d60 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.380+0000 7fbeb8359700 1 -- 172.21.15.113:0/2127110094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb01124a0 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.381+0000 7fbeb8359700 1 -- 172.21.15.113:0/2127110094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb0060370 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.381+0000 7fbea37fe700 1 -- 172.21.15.113:0/2127110094 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbeac00e7e0 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.382+0000 7fbea37fe700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe9c06caa0 0x7fbe9c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:23.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.382+0000 7fbea37fe700 1 -- 172.21.15.113:0/2127110094 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbeac08c0d0 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.382+0000 7fbeb58f4700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe9c06caa0 0x7fbe9c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:23.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.382+0000 7fbeb58f4700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe9c06caa0 0x7fbe9c06ef60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fbea40043d0 tx=0x7fbea400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:23.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.386+0000 7fbea37fe700 1 -- 172.21.15.113:0/2127110094 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbeac0560f0 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:23 smithi113 ceph-mon[103371]: pgmap v757: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:23.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:23 smithi192 ceph-mon[85304]: pgmap v757: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:23.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.653+0000 7fbeb8359700 1 -- 172.21.15.113:0/2127110094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbeb0047540 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.654+0000 7fbea37fe700 1 -- 172.21.15.113:0/2127110094 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fbeac059710 con 0x7fbeb00fc7f0 2024-04-12T05:37:23.652 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:37:23.652 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:37:23.652 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:37:23.652 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:37:23.652 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:37:23.652 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:37:23.652 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:37:23.653 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:37:23.653 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:37:23.653 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:37:23.653 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:37:23.653 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:37:23.653 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:37:23.653 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:37:23.653 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:37:23.653 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:37:23.653 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:37:23.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.656+0000 7fbeb8359700 1 -- 172.21.15.113:0/2127110094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe9c06caa0 msgr2=0x7fbe9c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.656+0000 7fbeb8359700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe9c06caa0 0x7fbe9c06ef60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fbea40043d0 tx=0x7fbea400b040).stop 2024-04-12T05:37:23.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.656+0000 7fbeb8359700 1 -- 172.21.15.113:0/2127110094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb00fc7f0 msgr2=0x7fbeb0109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.656+0000 7fbeb8359700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb00fc7f0 0x7fbeb0109160 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7fbeac00c740 tx=0x7fbeac007ef0).stop 2024-04-12T05:37:23.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.656+0000 7fbeb8359700 1 -- 172.21.15.113:0/2127110094 shutdown_connections 2024-04-12T05:37:23.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.656+0000 7fbeb8359700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe9c06caa0 0x7fbe9c06ef60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.656+0000 7fbeb8359700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb00fc7f0 0x7fbeb0109160 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.656+0000 7fbeb8359700 1 --2- 172.21.15.113:0/2127110094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbeb00fd150 0x7fbeb0109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.656+0000 7fbeb8359700 1 -- 172.21.15.113:0/2127110094 >> 172.21.15.113:0/2127110094 conn(0x7fbeb00f23b0 msgr2=0x7fbeb00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:23.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.656+0000 7fbeb8359700 1 -- 172.21.15.113:0/2127110094 shutdown_connections 2024-04-12T05:37:23.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.656+0000 7fbeb8359700 1 -- 172.21.15.113:0/2127110094 wait complete. 2024-04-12T05:37:23.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.754+0000 7fd7a9be4700 1 -- 172.21.15.113:0/3758576860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7a40f88d0 msgr2=0x7fd7a40f8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.754+0000 7fd7a9be4700 1 --2- 172.21.15.113:0/3758576860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7a40f88d0 0x7fd7a40f8cb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fd7940047f0 tx=0x7fd794009cb0).stop 2024-04-12T05:37:23.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.754+0000 7fd7a9be4700 1 -- 172.21.15.113:0/3758576860 shutdown_connections 2024-04-12T05:37:23.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.754+0000 7fd7a9be4700 1 --2- 172.21.15.113:0/3758576860 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a40f9230 0x7fd7a4101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.754+0000 7fd7a9be4700 1 --2- 172.21.15.113:0/3758576860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7a40f88d0 0x7fd7a40f8cb0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.754+0000 7fd7a9be4700 1 -- 172.21.15.113:0/3758576860 >> 172.21.15.113:0/3758576860 conn(0x7fd7a40f2400 msgr2=0x7fd7a40f4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:23.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.754+0000 7fd7a9be4700 1 -- 172.21.15.113:0/3758576860 shutdown_connections 2024-04-12T05:37:23.752 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.754+0000 7fd7a9be4700 1 -- 172.21.15.113:0/3758576860 wait complete. 2024-04-12T05:37:23.755 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.754+0000 7fd7a9be4700 1 Processor -- start 2024-04-12T05:37:23.755 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a9be4700 1 -- start start 2024-04-12T05:37:23.755 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a9be4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7a40f88d0 0x7fd7a410e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:23.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a9be4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a40f9230 0x7fd7a410e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:23.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a9be4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a410eb30 con 0x7fd7a40f88d0 2024-04-12T05:37:23.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a9be4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a410ec70 con 0x7fd7a40f9230 2024-04-12T05:37:23.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7a40f88d0 0x7fd7a410e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:23.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7a40f88d0 0x7fd7a410e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50596/0 (socket says 172.21.15.113:50596) 2024-04-12T05:37:23.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a37fe700 1 -- 172.21.15.113:0/585565125 learned_addr learned my addr 172.21.15.113:0/585565125 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:23.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a2ffd700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a40f9230 0x7fd7a410e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:23.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a37fe700 1 -- 172.21.15.113:0/585565125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a40f9230 msgr2=0x7fd7a410e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a37fe700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a40f9230 0x7fd7a410e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a37fe700 1 -- 172.21.15.113:0/585565125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd794005040 con 0x7fd7a40f88d0 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a2ffd700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a40f9230 0x7fd7a410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.755+0000 7fd7a37fe700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7a40f88d0 0x7fd7a410e040 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fd794005010 tx=0x7fd794009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.756+0000 7fd7a0ff9700 1 -- 172.21.15.113:0/585565125 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd79400e030 con 0x7fd7a40f88d0 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.756+0000 7fd7a9be4700 1 -- 172.21.15.113:0/585565125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a41082a0 con 0x7fd7a40f88d0 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.756+0000 7fd7a0ff9700 1 -- 172.21.15.113:0/585565125 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd794007e90 con 0x7fd7a40f88d0 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.756+0000 7fd7a0ff9700 1 -- 172.21.15.113:0/585565125 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd794018070 con 0x7fd7a40f88d0 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.756+0000 7fd7a9be4700 1 -- 172.21.15.113:0/585565125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a4108780 con 0x7fd7a40f88d0 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.757+0000 7fd7a9be4700 1 -- 172.21.15.113:0/585565125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a4060330 con 0x7fd7a40f88d0 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.757+0000 7fd7a0ff9700 1 -- 172.21.15.113:0/585565125 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd79401d6e0 con 0x7fd7a40f88d0 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.757+0000 7fd7a0ff9700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd79006ca30 0x7fd79006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.758+0000 7fd7a0ff9700 1 -- 172.21.15.113:0/585565125 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd79408d080 con 0x7fd7a40f88d0 2024-04-12T05:37:23.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.758+0000 7fd7a2ffd700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd79006ca30 0x7fd79006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:23.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.758+0000 7fd7a2ffd700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd79006ca30 0x7fd79006eef0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fd78c009a00 tx=0x7fd78c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:23.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.762+0000 7fd7a0ff9700 1 -- 172.21.15.113:0/585565125 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd79405ab30 con 0x7fd7a40f88d0 2024-04-12T05:37:23.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.991+0000 7fd7a9be4700 1 -- 172.21.15.113:0/585565125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd7a41097c0 con 0x7fd7a40f88d0 2024-04-12T05:37:23.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.991+0000 7fd7a0ff9700 1 -- 172.21.15.113:0/585565125 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fd79405a6c0 con 0x7fd7a40f88d0 2024-04-12T05:37:23.990 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:37:23.990 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:37:23.990 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:37:23.990 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:37:23.990 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:37:23.990 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:37:23.990 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:37:23.990 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:37:23.990 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:37:23.990 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:37:23.991 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:37:23.991 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:37:23.991 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:37:23.991 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:37:23.991 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:37:23.991 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:37:23.991 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:37:23.991 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:37:23.991 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:37:23.991 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:37:23.991 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:37:23.992 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:37:23.992 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:37:23.992 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:37:23.992 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:37:23.992 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:37:23.992 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:37:23.992 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:37:23.993 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:37:23.994 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:37:23.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.993+0000 7fd7a9be4700 1 -- 172.21.15.113:0/585565125 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd79006ca30 msgr2=0x7fd79006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.993+0000 7fd7a9be4700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd79006ca30 0x7fd79006eef0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fd78c009a00 tx=0x7fd78c00b040).stop 2024-04-12T05:37:23.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.993+0000 7fd7a9be4700 1 -- 172.21.15.113:0/585565125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7a40f88d0 msgr2=0x7fd7a410e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:23.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.993+0000 7fd7a9be4700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7a40f88d0 0x7fd7a410e040 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fd794005010 tx=0x7fd794009cb0).stop 2024-04-12T05:37:23.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.993+0000 7fd7a9be4700 1 -- 172.21.15.113:0/585565125 shutdown_connections 2024-04-12T05:37:23.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.993+0000 7fd7a9be4700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd79006ca30 0x7fd79006eef0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.993+0000 7fd7a9be4700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7a40f88d0 0x7fd7a410e040 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.993+0000 7fd7a9be4700 1 --2- 172.21.15.113:0/585565125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a40f9230 0x7fd7a410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:23.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.993+0000 7fd7a9be4700 1 -- 172.21.15.113:0/585565125 >> 172.21.15.113:0/585565125 conn(0x7fd7a40f2400 msgr2=0x7fd7a40fc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:23.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.993+0000 7fd7a9be4700 1 -- 172.21.15.113:0/585565125 shutdown_connections 2024-04-12T05:37:23.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:23.993+0000 7fd7a9be4700 1 -- 172.21.15.113:0/585565125 wait complete. 2024-04-12T05:37:23.996 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:37:24.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.094+0000 7f0a5474c700 1 -- 172.21.15.113:0/2948753752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a4c0fd8f0 msgr2=0x7f0a4c0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:24.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.094+0000 7f0a5474c700 1 --2- 172.21.15.113:0/2948753752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a4c0fd8f0 0x7f0a4c0fdcd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f0a3c0047f0 tx=0x7f0a3c009cb0).stop 2024-04-12T05:37:24.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.094+0000 7f0a5474c700 1 -- 172.21.15.113:0/2948753752 shutdown_connections 2024-04-12T05:37:24.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.094+0000 7f0a5474c700 1 --2- 172.21.15.113:0/2948753752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a4c0fe250 0x7f0a4c0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:24.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.094+0000 7f0a5474c700 1 --2- 172.21.15.113:0/2948753752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a4c0fd8f0 0x7f0a4c0fdcd0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:24.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.094+0000 7f0a5474c700 1 -- 172.21.15.113:0/2948753752 >> 172.21.15.113:0/2948753752 conn(0x7f0a4c0f34b0 msgr2=0x7f0a4c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:24.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.094+0000 7f0a5474c700 1 -- 172.21.15.113:0/2948753752 shutdown_connections 2024-04-12T05:37:24.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.094+0000 7f0a5474c700 1 -- 172.21.15.113:0/2948753752 wait complete. 2024-04-12T05:37:24.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.094+0000 7f0a5474c700 1 Processor -- start 2024-04-12T05:37:24.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.094+0000 7f0a5474c700 1 -- start start 2024-04-12T05:37:24.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a5474c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a4c0fd8f0 0x7f0a4c110210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:24.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a5474c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a4c0fe250 0x7f0a4c110700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:24.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a5474c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c110d00 con 0x7f0a4c0fd8f0 2024-04-12T05:37:24.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a5474c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c110e40 con 0x7f0a4c0fe250 2024-04-12T05:37:24.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a524e8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a4c0fd8f0 0x7f0a4c110210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:24.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a524e8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a4c0fd8f0 0x7f0a4c110210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50624/0 (socket says 172.21.15.113:50624) 2024-04-12T05:37:24.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a524e8700 1 -- 172.21.15.113:0/3263516452 learned_addr learned my addr 172.21.15.113:0/3263516452 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:24.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a524e8700 1 -- 172.21.15.113:0/3263516452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a4c0fe250 msgr2=0x7f0a4c110700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:24.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a524e8700 1 --2- 172.21.15.113:0/3263516452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a4c0fe250 0x7f0a4c110700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a524e8700 1 -- 172.21.15.113:0/3263516452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3c005040 con 0x7f0a4c0fd8f0 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a524e8700 1 --2- 172.21.15.113:0/3263516452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a4c0fd8f0 0x7f0a4c110210 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f0a3c00b120 tx=0x7f0a3c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a437fe700 1 -- 172.21.15.113:0/3263516452 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a3c00e030 con 0x7f0a4c0fd8f0 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a437fe700 1 -- 172.21.15.113:0/3263516452 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a3c007e90 con 0x7f0a4c0fd8f0 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a437fe700 1 -- 172.21.15.113:0/3263516452 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a3c018070 con 0x7f0a4c0fd8f0 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a5474c700 1 -- 172.21.15.113:0/3263516452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4c10a4c0 con 0x7f0a4c0fd8f0 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.095+0000 7f0a5474c700 1 -- 172.21.15.113:0/3263516452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4c10aa20 con 0x7f0a4c0fd8f0 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.096+0000 7f0a5474c700 1 -- 172.21.15.113:0/3263516452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4c05f590 con 0x7f0a4c0fd8f0 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.097+0000 7f0a437fe700 1 -- 172.21.15.113:0/3263516452 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0a3c01d420 con 0x7f0a4c0fd8f0 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.097+0000 7f0a437fe700 1 --2- 172.21.15.113:0/3263516452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0a3806caa0 0x7f0a3806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.097+0000 7f0a437fe700 1 -- 172.21.15.113:0/3263516452 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0a3c08d300 con 0x7f0a4c0fd8f0 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.097+0000 7f0a51ce7700 1 --2- 172.21.15.113:0/3263516452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0a3806caa0 0x7f0a3806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:24.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.097+0000 7f0a51ce7700 1 --2- 172.21.15.113:0/3263516452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0a3806caa0 0x7f0a3806ef60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f0a4c10bb50 tx=0x7f0a48007bc0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:24.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.101+0000 7f0a437fe700 1 -- 172.21.15.113:0/3263516452 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0a3c05adb0 con 0x7f0a4c0fd8f0 2024-04-12T05:37:24.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.306+0000 7f0a5474c700 1 -- 172.21.15.113:0/3263516452 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a4c10b9a0 con 0x7f0a3806caa0 2024-04-12T05:37:24.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.307+0000 7f0a437fe700 1 -- 172.21.15.113:0/3263516452 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0a4c10b9a0 con 0x7f0a3806caa0 2024-04-12T05:37:24.305 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:37:24.305 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:37:24.305 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:37:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:37:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:37:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:37:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:37:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:37:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:37:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:37:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:37:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:37:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:37:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:37:24.307 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:37:24.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.309+0000 7f0a5474c700 1 -- 172.21.15.113:0/3263516452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0a3806caa0 msgr2=0x7f0a3806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:24.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.309+0000 7f0a5474c700 1 --2- 172.21.15.113:0/3263516452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0a3806caa0 0x7f0a3806ef60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f0a4c10bb50 tx=0x7f0a48007bc0).stop 2024-04-12T05:37:24.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.309+0000 7f0a5474c700 1 -- 172.21.15.113:0/3263516452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a4c0fd8f0 msgr2=0x7f0a4c110210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:24.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.309+0000 7f0a5474c700 1 --2- 172.21.15.113:0/3263516452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a4c0fd8f0 0x7f0a4c110210 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f0a3c00b120 tx=0x7f0a3c009cb0).stop 2024-04-12T05:37:24.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.309+0000 7f0a5474c700 1 -- 172.21.15.113:0/3263516452 shutdown_connections 2024-04-12T05:37:24.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.309+0000 7f0a5474c700 1 --2- 172.21.15.113:0/3263516452 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0a3806caa0 0x7f0a3806ef60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:24.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.309+0000 7f0a5474c700 1 --2- 172.21.15.113:0/3263516452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a4c0fd8f0 0x7f0a4c110210 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:24.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.309+0000 7f0a5474c700 1 --2- 172.21.15.113:0/3263516452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a4c0fe250 0x7f0a4c110700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:37:24.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.309+0000 7f0a5474c700 1 -- 172.21.15.113:0/3263516452 >> 172.21.15.113:0/3263516452 conn(0x7f0a4c0f34b0 msgr2=0x7f0a4c06b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:24.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.309+0000 7f0a5474c700 1 -- 172.21.15.113:0/3263516452 shutdown_connections 2024-04-12T05:37:24.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.309+0000 7f0a5474c700 1 -- 172.21.15.113:0/3263516452 wait complete. 2024-04-12T05:37:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.408+0000 7f98019d0700 1 -- 172.21.15.113:0/2381123128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97fc0f9200 msgr2=0x7f97fc101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.408+0000 7f98019d0700 1 --2- 172.21.15.113:0/2381123128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97fc0f9200 0x7f97fc101910 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f97ec0047f0 tx=0x7f97ec009cb0).stop 2024-04-12T05:37:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.408+0000 7f98019d0700 1 -- 172.21.15.113:0/2381123128 shutdown_connections 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.408+0000 7f98019d0700 1 --2- 172.21.15.113:0/2381123128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97fc0f9200 0x7f97fc101910 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.408+0000 7f98019d0700 1 --2- 172.21.15.113:0/2381123128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97fc0f88a0 0x7f97fc0f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.408+0000 7f98019d0700 1 -- 172.21.15.113:0/2381123128 >> 172.21.15.113:0/2381123128 conn(0x7f97fc070590 msgr2=0x7f97fc0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.408+0000 7f98019d0700 1 -- 172.21.15.113:0/2381123128 shutdown_connections 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.408+0000 7f98019d0700 1 -- 172.21.15.113:0/2381123128 wait complete. 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.408+0000 7f98019d0700 1 Processor -- start 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.408+0000 7f98019d0700 1 -- start start 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f98019d0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97fc0f88a0 0x7f97fc10e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f98019d0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97fc0f9200 0x7f97fc10e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f98019d0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc10eb70 con 0x7f97fc0f88a0 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f98019d0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc10ecb0 con 0x7f97fc0f9200 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f97faffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97fc0f88a0 0x7f97fc10e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f97faffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97fc0f88a0 0x7f97fc10e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50646/0 (socket says 172.21.15.113:50646) 2024-04-12T05:37:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f97faffd700 1 -- 172.21.15.113:0/2210293445 learned_addr learned my addr 172.21.15.113:0/2210293445 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:24.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f97fa7fc700 1 --2- 172.21.15.113:0/2210293445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97fc0f9200 0x7f97fc10e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:24.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f97faffd700 1 -- 172.21.15.113:0/2210293445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97fc0f9200 msgr2=0x7f97fc10e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:24.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f97faffd700 1 --2- 172.21.15.113:0/2210293445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97fc0f9200 0x7f97fc10e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:24.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f97faffd700 1 -- 172.21.15.113:0/2210293445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec005040 con 0x7f97fc0f88a0 2024-04-12T05:37:24.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f97faffd700 1 --2- 172.21.15.113:0/2210293445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97fc0f88a0 0x7f97fc10e080 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f97f00041d0 tx=0x7f97f0009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:24.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f97e3fff700 1 -- 172.21.15.113:0/2210293445 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97f0007b40 con 0x7f97fc0f88a0 2024-04-12T05:37:24.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f97e3fff700 1 -- 172.21.15.113:0/2210293445 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97f000f040 con 0x7f97fc0f88a0 2024-04-12T05:37:24.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f98019d0700 1 -- 172.21.15.113:0/2210293445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97fc108340 con 0x7f97fc0f88a0 2024-04-12T05:37:24.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f97e3fff700 1 -- 172.21.15.113:0/2210293445 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97f00147b0 con 0x7f97fc0f88a0 2024-04-12T05:37:24.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.409+0000 7f98019d0700 1 -- 172.21.15.113:0/2210293445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97fc108880 con 0x7f97fc0f88a0 2024-04-12T05:37:24.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.410+0000 7f98019d0700 1 -- 172.21.15.113:0/2210293445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97fc05f590 con 0x7f97fc0f88a0 2024-04-12T05:37:24.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.411+0000 7f97e3fff700 1 -- 172.21.15.113:0/2210293445 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f97f0009110 con 0x7f97fc0f88a0 2024-04-12T05:37:24.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.411+0000 7f97e3fff700 1 --2- 172.21.15.113:0/2210293445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97e406caa0 0x7f97e406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:24.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.411+0000 7f97fa7fc700 1 --2- 172.21.15.113:0/2210293445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97e406caa0 0x7f97e406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:24.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.411+0000 7f97e3fff700 1 -- 172.21.15.113:0/2210293445 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f97f008be50 con 0x7f97fc0f88a0 2024-04-12T05:37:24.410 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.412+0000 7f97fa7fc700 1 --2- 172.21.15.113:0/2210293445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97e406caa0 0x7f97e406ef60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f97fc1099c0 tx=0x7f97ec00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:24.413 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.416+0000 7f97e3fff700 1 -- 172.21.15.113:0/2210293445 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f97f0059900 con 0x7f97fc0f88a0 2024-04-12T05:37:24.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:24 smithi113 ceph-mon[103371]: from='client.35374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:24.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:24 smithi113 ceph-mon[103371]: from='client.35378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:24.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:24 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2127110094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:37:24.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:24 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/585565125' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:37:24.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:24 smithi192 ceph-mon[85304]: from='client.35374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:24.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:24 smithi192 ceph-mon[85304]: from='client.35378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:24.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:24 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2127110094' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:37:24.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:24 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/585565125' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:37:24.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.679+0000 7f98019d0700 1 -- 172.21.15.113:0/2210293445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f97fc047540 con 0x7f97fc0f88a0 2024-04-12T05:37:24.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.680+0000 7f97e3fff700 1 -- 172.21.15.113:0/2210293445 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f97f0059490 con 0x7f97fc0f88a0 2024-04-12T05:37:24.678 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:37:24.678 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:37:24.679 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:37:24.679 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:37:24.679 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:37:24.679 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:37:24.679 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:37:24.679 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:37:24.679 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:37:24.679 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:37:24.679 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:37:24.680 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:37:24.680 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:37:24.680 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:37:24.680 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:37:24.680 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:37:24.680 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:37:24.680 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:37:24.680 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:37:24.680 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:37:24.680 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:37:24.681 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:37:24.681 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:37:24.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.682+0000 7f98019d0700 1 -- 172.21.15.113:0/2210293445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97e406caa0 msgr2=0x7f97e406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:24.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.682+0000 7f98019d0700 1 --2- 172.21.15.113:0/2210293445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97e406caa0 0x7f97e406ef60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f97fc1099c0 tx=0x7f97ec00b040).stop 2024-04-12T05:37:24.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.682+0000 7f98019d0700 1 -- 172.21.15.113:0/2210293445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97fc0f88a0 msgr2=0x7f97fc10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:24.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.682+0000 7f98019d0700 1 --2- 172.21.15.113:0/2210293445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97fc0f88a0 0x7f97fc10e080 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f97f00041d0 tx=0x7f97f0009f60).stop 2024-04-12T05:37:24.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.682+0000 7f98019d0700 1 -- 172.21.15.113:0/2210293445 shutdown_connections 2024-04-12T05:37:24.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.682+0000 7f98019d0700 1 --2- 172.21.15.113:0/2210293445 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97e406caa0 0x7f97e406ef60 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:24.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.682+0000 7f98019d0700 1 --2- 172.21.15.113:0/2210293445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97fc0f88a0 0x7f97fc10e080 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:24.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.682+0000 7f98019d0700 1 --2- 172.21.15.113:0/2210293445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97fc0f9200 0x7f97fc10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:24.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.682+0000 7f98019d0700 1 -- 172.21.15.113:0/2210293445 >> 172.21.15.113:0/2210293445 conn(0x7f97fc070590 msgr2=0x7f97fc0fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:24.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.682+0000 7f98019d0700 1 -- 172.21.15.113:0/2210293445 shutdown_connections 2024-04-12T05:37:24.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:24.682+0000 7f98019d0700 1 -- 172.21.15.113:0/2210293445 wait complete. 2024-04-12T05:37:25.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:25 smithi113 ceph-mon[103371]: from='client.35382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:25.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:25 smithi113 ceph-mon[103371]: pgmap v758: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:25.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:25 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2210293445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:37:25.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:25 smithi192 ceph-mon[85304]: from='client.35382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:25.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:25 smithi192 ceph-mon[85304]: pgmap v758: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:25.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:25 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2210293445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:37:26.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:26 smithi113 ceph-mon[103371]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:26.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:26 smithi192 ceph-mon[85304]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:27.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:27 smithi113 ceph-mon[103371]: pgmap v759: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:27.447 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:27 smithi192 ceph-mon[85304]: pgmap v759: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:29.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:29 smithi113 ceph-mon[103371]: pgmap v760: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:29.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:29 smithi192 ceph-mon[85304]: pgmap v760: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:31.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:31 smithi113 ceph-mon[103371]: pgmap v761: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:31.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:31 smithi192 ceph-mon[85304]: pgmap v761: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:33.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:33 smithi113 ceph-mon[103371]: pgmap v762: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:33.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:33 smithi192 ceph-mon[85304]: pgmap v762: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:35.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:35 smithi113 ceph-mon[103371]: pgmap v763: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:35.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:35 smithi192 ceph-mon[85304]: pgmap v763: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:37.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:37 smithi113 ceph-mon[103371]: pgmap v764: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:37.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:37.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:37 smithi192 ceph-mon[85304]: pgmap v764: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:37.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:39.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:39 smithi113 ceph-mon[103371]: pgmap v765: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:39.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:39 smithi192 ceph-mon[85304]: pgmap v765: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:41.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:41 smithi113 ceph-mon[103371]: pgmap v766: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:41.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:41 smithi192 ceph-mon[85304]: pgmap v766: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:43.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:43 smithi113 ceph-mon[103371]: pgmap v767: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:43.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:43 smithi192 ceph-mon[85304]: pgmap v767: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:45.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:45 smithi113 ceph-mon[103371]: pgmap v768: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:45.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:45 smithi192 ceph-mon[85304]: pgmap v768: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:47.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:47 smithi113 ceph-mon[103371]: pgmap v769: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:47.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:47 smithi192 ceph-mon[85304]: pgmap v769: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:49.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:49 smithi113 ceph-mon[103371]: pgmap v770: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:49.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:49 smithi192 ceph-mon[85304]: pgmap v770: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:51.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:51 smithi113 ceph-mon[103371]: pgmap v771: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:51.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:51 smithi192 ceph-mon[85304]: pgmap v771: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:52.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:52.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:37:53.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:53 smithi113 ceph-mon[103371]: pgmap v772: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:53.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:53 smithi192 ceph-mon[85304]: pgmap v772: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.793+0000 7f804eb6e700 1 -- 172.21.15.113:0/2257385174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80480fc7f0 msgr2=0x7f80480fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:54.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.793+0000 7f804eb6e700 1 --2- 172.21.15.113:0/2257385174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80480fc7f0 0x7f80480fcbd0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f803c0047f0 tx=0x7f803c009cb0).stop 2024-04-12T05:37:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.793+0000 7f804eb6e700 1 -- 172.21.15.113:0/2257385174 shutdown_connections 2024-04-12T05:37:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.793+0000 7f804eb6e700 1 --2- 172.21.15.113:0/2257385174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80480fd150 0x7f80480f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.793+0000 7f804eb6e700 1 --2- 172.21.15.113:0/2257385174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80480fc7f0 0x7f80480fcbd0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.793+0000 7f804eb6e700 1 -- 172.21.15.113:0/2257385174 >> 172.21.15.113:0/2257385174 conn(0x7f8048070590 msgr2=0x7f80480f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.793+0000 7f804eb6e700 1 -- 172.21.15.113:0/2257385174 shutdown_connections 2024-04-12T05:37:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.793+0000 7f804eb6e700 1 -- 172.21.15.113:0/2257385174 wait complete. 2024-04-12T05:37:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804eb6e700 1 Processor -- start 2024-04-12T05:37:54.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804eb6e700 1 -- start start 2024-04-12T05:37:54.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804eb6e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80480fc7f0 0x7f80481090d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:54.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804eb6e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80480fd150 0x7f80481095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804eb6e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8048109ab0 con 0x7f80480fc7f0 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804eb6e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8048109bf0 con 0x7f80480fd150 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804c90a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80480fc7f0 0x7f80481090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804c90a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80480fc7f0 0x7f80481090d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54818/0 (socket says 172.21.15.113:54818) 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804c90a700 1 -- 172.21.15.113:0/873816693 learned_addr learned my addr 172.21.15.113:0/873816693 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f8047fff700 1 --2- 172.21.15.113:0/873816693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80480fd150 0x7f80481095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804c90a700 1 -- 172.21.15.113:0/873816693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80480fd150 msgr2=0x7f80481095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804c90a700 1 --2- 172.21.15.113:0/873816693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80480fd150 0x7f80481095c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804c90a700 1 -- 172.21.15.113:0/873816693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c005040 con 0x7f80480fc7f0 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.794+0000 7f804c90a700 1 --2- 172.21.15.113:0/873816693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80480fc7f0 0x7f80481090d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f803c00b120 tx=0x7f803c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.795+0000 7f8045ffb700 1 -- 172.21.15.113:0/873816693 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f803c00e030 con 0x7f80480fc7f0 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.795+0000 7f8045ffb700 1 -- 172.21.15.113:0/873816693 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f803c007e90 con 0x7f80480fc7f0 2024-04-12T05:37:54.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.795+0000 7f8045ffb700 1 -- 172.21.15.113:0/873816693 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f803c018070 con 0x7f80480fc7f0 2024-04-12T05:37:54.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.795+0000 7f804eb6e700 1 -- 172.21.15.113:0/873816693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8048109e40 con 0x7f80480fc7f0 2024-04-12T05:37:54.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.795+0000 7f804eb6e700 1 -- 172.21.15.113:0/873816693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8048112430 con 0x7f80480fc7f0 2024-04-12T05:37:54.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.796+0000 7f804eb6e700 1 -- 172.21.15.113:0/873816693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f804805f590 con 0x7f80480fc7f0 2024-04-12T05:37:54.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.796+0000 7f8045ffb700 1 -- 172.21.15.113:0/873816693 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f803c01d420 con 0x7f80480fc7f0 2024-04-12T05:37:54.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.797+0000 7f8045ffb700 1 --2- 172.21.15.113:0/873816693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f803806caa0 0x7f803806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:54.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.797+0000 7f8045ffb700 1 -- 172.21.15.113:0/873816693 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f803c08d300 con 0x7f80480fc7f0 2024-04-12T05:37:54.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.797+0000 7f8047fff700 1 --2- 172.21.15.113:0/873816693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f803806caa0 0x7f803806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:54.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.798+0000 7f8047fff700 1 --2- 172.21.15.113:0/873816693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f803806caa0 0x7f803806ef60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f8048060f00 tx=0x7f8034009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:54.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:54.802+0000 7f8045ffb700 1 -- 172.21.15.113:0/873816693 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f803c057320 con 0x7f80480fc7f0 2024-04-12T05:37:55.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.008+0000 7f804eb6e700 1 -- 172.21.15.113:0/873816693 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8048059120 con 0x7f803806caa0 2024-04-12T05:37:55.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.009+0000 7f8045ffb700 1 -- 172.21.15.113:0/873816693 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8048059120 con 0x7f803806caa0 2024-04-12T05:37:55.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.011+0000 7f804eb6e700 1 -- 172.21.15.113:0/873816693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f803806caa0 msgr2=0x7f803806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.011+0000 7f804eb6e700 1 --2- 172.21.15.113:0/873816693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f803806caa0 0x7f803806ef60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f8048060f00 tx=0x7f8034009040).stop 2024-04-12T05:37:55.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.011+0000 7f804eb6e700 1 -- 172.21.15.113:0/873816693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80480fc7f0 msgr2=0x7f80481090d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.011+0000 7f804eb6e700 1 --2- 172.21.15.113:0/873816693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80480fc7f0 0x7f80481090d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f803c00b120 tx=0x7f803c009cb0).stop 2024-04-12T05:37:55.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.011+0000 7f804eb6e700 1 -- 172.21.15.113:0/873816693 shutdown_connections 2024-04-12T05:37:55.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.011+0000 7f804eb6e700 1 --2- 172.21.15.113:0/873816693 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f803806caa0 0x7f803806ef60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.011+0000 7f804eb6e700 1 --2- 172.21.15.113:0/873816693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80480fc7f0 0x7f80481090d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.011+0000 7f804eb6e700 1 --2- 172.21.15.113:0/873816693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80480fd150 0x7f80481095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.011+0000 7f804eb6e700 1 -- 172.21.15.113:0/873816693 >> 172.21.15.113:0/873816693 conn(0x7f8048070590 msgr2=0x7f80480fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:55.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.011+0000 7f804eb6e700 1 -- 172.21.15.113:0/873816693 shutdown_connections 2024-04-12T05:37:55.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.011+0000 7f804eb6e700 1 -- 172.21.15.113:0/873816693 wait complete. 2024-04-12T05:37:55.019 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:37:55.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.112+0000 7feb2f9e0700 1 -- 172.21.15.113:0/874287389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb280fc7f0 msgr2=0x7feb280fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.112+0000 7feb2f9e0700 1 --2- 172.21.15.113:0/874287389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb280fc7f0 0x7feb280fcbd0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7feb18009b60 tx=0x7feb18004d10).stop 2024-04-12T05:37:55.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.112+0000 7feb2f9e0700 1 -- 172.21.15.113:0/874287389 shutdown_connections 2024-04-12T05:37:55.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.112+0000 7feb2f9e0700 1 --2- 172.21.15.113:0/874287389 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb280fd150 0x7feb280f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.112+0000 7feb2f9e0700 1 --2- 172.21.15.113:0/874287389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb280fc7f0 0x7feb280fcbd0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.112+0000 7feb2f9e0700 1 -- 172.21.15.113:0/874287389 >> 172.21.15.113:0/874287389 conn(0x7feb280f23b0 msgr2=0x7feb280f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:55.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.112+0000 7feb2f9e0700 1 -- 172.21.15.113:0/874287389 shutdown_connections 2024-04-12T05:37:55.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.112+0000 7feb2f9e0700 1 -- 172.21.15.113:0/874287389 wait complete. 2024-04-12T05:37:55.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.113+0000 7feb2f9e0700 1 Processor -- start 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.113+0000 7feb2f9e0700 1 -- start start 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.113+0000 7feb2f9e0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb280fc7f0 0x7feb2810cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.113+0000 7feb2f9e0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb280fd150 0x7feb2810d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.113+0000 7feb2f9e0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2810da10 con 0x7feb280fd150 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.113+0000 7feb2f9e0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2810db50 con 0x7feb280fc7f0 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.113+0000 7feb2cf7b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb280fd150 0x7feb2810d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.113+0000 7feb2cf7b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb280fd150 0x7feb2810d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54846/0 (socket says 172.21.15.113:54846) 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.113+0000 7feb2cf7b700 1 -- 172.21.15.113:0/3174781290 learned_addr learned my addr 172.21.15.113:0/3174781290 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.114+0000 7feb2d77c700 1 --2- 172.21.15.113:0/3174781290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb280fc7f0 0x7feb2810cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.114+0000 7feb2cf7b700 1 -- 172.21.15.113:0/3174781290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb280fc7f0 msgr2=0x7feb2810cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.114+0000 7feb2cf7b700 1 --2- 172.21.15.113:0/3174781290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb280fc7f0 0x7feb2810cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.114+0000 7feb2cf7b700 1 -- 172.21.15.113:0/3174781290 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb180097c0 con 0x7feb280fd150 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.114+0000 7feb2cf7b700 1 --2- 172.21.15.113:0/3174781290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb280fd150 0x7feb2810d410 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7feb1c002780 tx=0x7feb1c009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:55.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.114+0000 7feb267fc700 1 -- 172.21.15.113:0/3174781290 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb1c007b40 con 0x7feb280fd150 2024-04-12T05:37:55.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.114+0000 7feb2f9e0700 1 -- 172.21.15.113:0/3174781290 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb281071e0 con 0x7feb280fd150 2024-04-12T05:37:55.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.114+0000 7feb267fc700 1 -- 172.21.15.113:0/3174781290 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb1c00f040 con 0x7feb280fd150 2024-04-12T05:37:55.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.114+0000 7feb267fc700 1 -- 172.21.15.113:0/3174781290 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb1c014780 con 0x7feb280fd150 2024-04-12T05:37:55.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.114+0000 7feb2f9e0700 1 -- 172.21.15.113:0/3174781290 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb281077a0 con 0x7feb280fd150 2024-04-12T05:37:55.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.115+0000 7feb2f9e0700 1 -- 172.21.15.113:0/3174781290 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb28060390 con 0x7feb280fd150 2024-04-12T05:37:55.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.115+0000 7feb267fc700 1 -- 172.21.15.113:0/3174781290 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7feb1c007ca0 con 0x7feb280fd150 2024-04-12T05:37:55.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.116+0000 7feb267fc700 1 --2- 172.21.15.113:0/3174781290 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb1006caa0 0x7feb1006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:55.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.116+0000 7feb267fc700 1 -- 172.21.15.113:0/3174781290 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7feb1c08bd50 con 0x7feb280fd150 2024-04-12T05:37:55.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.116+0000 7feb2d77c700 1 --2- 172.21.15.113:0/3174781290 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb1006caa0 0x7feb1006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:55.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.116+0000 7feb2d77c700 1 --2- 172.21.15.113:0/3174781290 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb1006caa0 0x7feb1006ef60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7feb18009750 tx=0x7feb18004d10).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:55.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.120+0000 7feb267fc700 1 -- 172.21.15.113:0/3174781290 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7feb1c059800 con 0x7feb280fd150 2024-04-12T05:37:55.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.324+0000 7feb2f9e0700 1 -- 172.21.15.113:0/3174781290 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feb28107a20 con 0x7feb1006caa0 2024-04-12T05:37:55.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.326+0000 7feb267fc700 1 -- 172.21.15.113:0/3174781290 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7feb28107a20 con 0x7feb1006caa0 2024-04-12T05:37:55.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.328+0000 7feb2f9e0700 1 -- 172.21.15.113:0/3174781290 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb1006caa0 msgr2=0x7feb1006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.328+0000 7feb2f9e0700 1 --2- 172.21.15.113:0/3174781290 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb1006caa0 0x7feb1006ef60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7feb18009750 tx=0x7feb18004d10).stop 2024-04-12T05:37:55.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.328+0000 7feb2f9e0700 1 -- 172.21.15.113:0/3174781290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb280fd150 msgr2=0x7feb2810d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.328+0000 7feb2f9e0700 1 --2- 172.21.15.113:0/3174781290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb280fd150 0x7feb2810d410 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7feb1c002780 tx=0x7feb1c009ee0).stop 2024-04-12T05:37:55.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.328+0000 7feb2f9e0700 1 -- 172.21.15.113:0/3174781290 shutdown_connections 2024-04-12T05:37:55.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.328+0000 7feb2f9e0700 1 --2- 172.21.15.113:0/3174781290 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7feb1006caa0 0x7feb1006ef60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.328+0000 7feb2f9e0700 1 --2- 172.21.15.113:0/3174781290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb280fc7f0 0x7feb2810cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.328+0000 7feb2f9e0700 1 --2- 172.21.15.113:0/3174781290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb280fd150 0x7feb2810d410 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.328+0000 7feb2f9e0700 1 -- 172.21.15.113:0/3174781290 >> 172.21.15.113:0/3174781290 conn(0x7feb280f23b0 msgr2=0x7feb280fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:55.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.328+0000 7feb2f9e0700 1 -- 172.21.15.113:0/3174781290 shutdown_connections 2024-04-12T05:37:55.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.328+0000 7feb2f9e0700 1 -- 172.21.15.113:0/3174781290 wait complete. 2024-04-12T05:37:55.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:55 smithi113 ceph-mon[103371]: pgmap v773: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:55.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.425+0000 7f5b8908c700 1 -- 172.21.15.113:0/3130401079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd150 msgr2=0x7f5b840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.425+0000 7f5b8908c700 1 --2- 172.21.15.113:0/3130401079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd150 0x7f5b840f6930 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f5b740047f0 tx=0x7f5b74009cb0).stop 2024-04-12T05:37:55.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.425+0000 7f5b8908c700 1 -- 172.21.15.113:0/3130401079 shutdown_connections 2024-04-12T05:37:55.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.425+0000 7f5b8908c700 1 --2- 172.21.15.113:0/3130401079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd150 0x7f5b840f6930 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.425+0000 7f5b8908c700 1 --2- 172.21.15.113:0/3130401079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fc7f0 0x7f5b840fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.425+0000 7f5b8908c700 1 -- 172.21.15.113:0/3130401079 >> 172.21.15.113:0/3130401079 conn(0x7f5b84070590 msgr2=0x7f5b840f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:55.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.425+0000 7f5b8908c700 1 -- 172.21.15.113:0/3130401079 shutdown_connections 2024-04-12T05:37:55.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.425+0000 7f5b8908c700 1 -- 172.21.15.113:0/3130401079 wait complete. 2024-04-12T05:37:55.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.426+0000 7f5b8908c700 1 Processor -- start 2024-04-12T05:37:55.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.426+0000 7f5b8908c700 1 -- start start 2024-04-12T05:37:55.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.426+0000 7f5b8908c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fc7f0 0x7f5b840f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:55.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.426+0000 7f5b8908c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd150 0x7f5b840f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:55.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.426+0000 7f5b8908c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b840f8f20 con 0x7f5b840fd150 2024-04-12T05:37:55.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.426+0000 7f5b8908c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b840f9060 con 0x7f5b840fc7f0 2024-04-12T05:37:55.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.426+0000 7f5b8259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd150 0x7f5b840f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:55.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.426+0000 7f5b8259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd150 0x7f5b840f8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54862/0 (socket says 172.21.15.113:54862) 2024-04-12T05:37:55.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.426+0000 7f5b8259c700 1 -- 172.21.15.113:0/4026646466 learned_addr learned my addr 172.21.15.113:0/4026646466 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:55.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.427+0000 7f5b82d9d700 1 --2- 172.21.15.113:0/4026646466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fc7f0 0x7f5b840f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:55.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.427+0000 7f5b8259c700 1 -- 172.21.15.113:0/4026646466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fc7f0 msgr2=0x7f5b840f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.427+0000 7f5b8259c700 1 --2- 172.21.15.113:0/4026646466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fc7f0 0x7f5b840f8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.427+0000 7f5b8259c700 1 -- 172.21.15.113:0/4026646466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b74005040 con 0x7f5b840fd150 2024-04-12T05:37:55.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.427+0000 7f5b8259c700 1 --2- 172.21.15.113:0/4026646466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd150 0x7f5b840f8a30 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f5b74000c00 tx=0x7f5b74009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:55.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.427+0000 7f5b6bfff700 1 -- 172.21.15.113:0/4026646466 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b7400e030 con 0x7f5b840fd150 2024-04-12T05:37:55.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.427+0000 7f5b6bfff700 1 -- 172.21.15.113:0/4026646466 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b74007e90 con 0x7f5b840fd150 2024-04-12T05:37:55.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.427+0000 7f5b8908c700 1 -- 172.21.15.113:0/4026646466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b840f92b0 con 0x7f5b840fd150 2024-04-12T05:37:55.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.427+0000 7f5b6bfff700 1 -- 172.21.15.113:0/4026646466 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b74018070 con 0x7f5b840fd150 2024-04-12T05:37:55.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.427+0000 7f5b8908c700 1 -- 172.21.15.113:0/4026646466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b841145f0 con 0x7f5b840fd150 2024-04-12T05:37:55.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.428+0000 7f5b8908c700 1 -- 172.21.15.113:0/4026646466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8405f590 con 0x7f5b840fd150 2024-04-12T05:37:55.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.429+0000 7f5b6bfff700 1 -- 172.21.15.113:0/4026646466 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5b7401d420 con 0x7f5b840fd150 2024-04-12T05:37:55.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.430+0000 7f5b6bfff700 1 --2- 172.21.15.113:0/4026646466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b6c06cab0 0x7f5b6c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:55.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.430+0000 7f5b6bfff700 1 -- 172.21.15.113:0/4026646466 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5b7408d720 con 0x7f5b840fd150 2024-04-12T05:37:55.427 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.430+0000 7f5b82d9d700 1 --2- 172.21.15.113:0/4026646466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b6c06cab0 0x7f5b6c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:55.428 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.430+0000 7f5b82d9d700 1 --2- 172.21.15.113:0/4026646466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b6c06cab0 0x7f5b6c06ef70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f5b780041d0 tx=0x7f5b78009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:55.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.433+0000 7f5b6bfff700 1 -- 172.21.15.113:0/4026646466 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5b74057740 con 0x7f5b840fd150 2024-04-12T05:37:55.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:55 smithi192 ceph-mon[85304]: pgmap v773: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:55.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.631+0000 7f5b8908c700 1 -- 172.21.15.113:0/4026646466 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b840f9ec0 con 0x7f5b6c06cab0 2024-04-12T05:37:55.638 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.640+0000 7f5b6bfff700 1 -- 172.21.15.113:0/4026646466 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5b840f9ec0 con 0x7f5b6c06cab0 2024-04-12T05:37:55.638 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:37:55.638 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (25m) 3m ago 36m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (23m) 3m ago 36m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (23m) 3m ago 36m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (25m) 3m ago 36m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (19m) 3m ago 31m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (18m) 3m ago 31m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (18m) 3m ago 31m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (3m) 3m ago 31m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (27m) 3m ago 38m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (26m) 3m ago 36m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (24m) 3m ago 38m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (23m) 3m ago 36m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (26m) 3m ago 36m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:37:55.639 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (26m) 3m ago 36m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:37:55.640 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (22m) 3m ago 35m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:37:55.640 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (22m) 3m ago 34m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:37:55.640 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (21m) 3m ago 34m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:37:55.640 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (21m) 3m ago 34m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:37:55.640 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (20m) 3m ago 33m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:37:55.640 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (19m) 3m ago 33m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:37:55.640 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (26m) 3m ago 35m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:37:55.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.642+0000 7f5b8908c700 1 -- 172.21.15.113:0/4026646466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b6c06cab0 msgr2=0x7f5b6c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.642+0000 7f5b8908c700 1 --2- 172.21.15.113:0/4026646466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b6c06cab0 0x7f5b6c06ef70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f5b780041d0 tx=0x7f5b78009040).stop 2024-04-12T05:37:55.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.642+0000 7f5b8908c700 1 -- 172.21.15.113:0/4026646466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd150 msgr2=0x7f5b840f8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.642+0000 7f5b8908c700 1 --2- 172.21.15.113:0/4026646466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd150 0x7f5b840f8a30 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f5b74000c00 tx=0x7f5b74009cb0).stop 2024-04-12T05:37:55.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.642+0000 7f5b8908c700 1 -- 172.21.15.113:0/4026646466 shutdown_connections 2024-04-12T05:37:55.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.642+0000 7f5b8908c700 1 --2- 172.21.15.113:0/4026646466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b6c06cab0 0x7f5b6c06ef70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.642+0000 7f5b8908c700 1 --2- 172.21.15.113:0/4026646466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fc7f0 0x7f5b840f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.642+0000 7f5b8908c700 1 --2- 172.21.15.113:0/4026646466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd150 0x7f5b840f8a30 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.642+0000 7f5b8908c700 1 -- 172.21.15.113:0/4026646466 >> 172.21.15.113:0/4026646466 conn(0x7f5b84070590 msgr2=0x7f5b84060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:55.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.642+0000 7f5b8908c700 1 -- 172.21.15.113:0/4026646466 shutdown_connections 2024-04-12T05:37:55.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.642+0000 7f5b8908c700 1 -- 172.21.15.113:0/4026646466 wait complete. 2024-04-12T05:37:55.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.739+0000 7fe91a449700 1 -- 172.21.15.113:0/4264676982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9140fd150 msgr2=0x7fe9140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.739+0000 7fe91a449700 1 --2- 172.21.15.113:0/4264676982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9140fd150 0x7fe9140f6930 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fe9080047f0 tx=0x7fe908009cb0).stop 2024-04-12T05:37:55.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.739+0000 7fe91a449700 1 -- 172.21.15.113:0/4264676982 shutdown_connections 2024-04-12T05:37:55.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.739+0000 7fe91a449700 1 --2- 172.21.15.113:0/4264676982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9140fd150 0x7fe9140f6930 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.739+0000 7fe91a449700 1 --2- 172.21.15.113:0/4264676982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9140fc7f0 0x7fe9140fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.739+0000 7fe91a449700 1 -- 172.21.15.113:0/4264676982 >> 172.21.15.113:0/4264676982 conn(0x7fe914070590 msgr2=0x7fe9140f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:55.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.740+0000 7fe91a449700 1 -- 172.21.15.113:0/4264676982 shutdown_connections 2024-04-12T05:37:55.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.740+0000 7fe91a449700 1 -- 172.21.15.113:0/4264676982 wait complete. 2024-04-12T05:37:55.738 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.740+0000 7fe91a449700 1 Processor -- start 2024-04-12T05:37:55.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.740+0000 7fe91a449700 1 -- start start 2024-04-12T05:37:55.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.740+0000 7fe91a449700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9140fc7f0 0x7fe9141090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:55.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.740+0000 7fe91a449700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9140fd150 0x7fe9141095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:55.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.740+0000 7fe91a449700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe914109aa0 con 0x7fe9140fd150 2024-04-12T05:37:55.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.740+0000 7fe91a449700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe914109be0 con 0x7fe9140fc7f0 2024-04-12T05:37:55.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe9137fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9140fd150 0x7fe9141095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe9137fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9140fd150 0x7fe9141095b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54880/0 (socket says 172.21.15.113:54880) 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe9137fe700 1 -- 172.21.15.113:0/2141061271 learned_addr learned my addr 172.21.15.113:0/2141061271 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe913fff700 1 --2- 172.21.15.113:0/2141061271 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9140fc7f0 0x7fe9141090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe9137fe700 1 -- 172.21.15.113:0/2141061271 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9140fc7f0 msgr2=0x7fe9141090c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe9137fe700 1 --2- 172.21.15.113:0/2141061271 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9140fc7f0 0x7fe9141090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe9137fe700 1 -- 172.21.15.113:0/2141061271 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe908005040 con 0x7fe9140fd150 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe9137fe700 1 --2- 172.21.15.113:0/2141061271 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9140fd150 0x7fe9141095b0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7fe9080099f0 tx=0x7fe908009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe9117fa700 1 -- 172.21.15.113:0/2141061271 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe90800e030 con 0x7fe9140fd150 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe91a449700 1 -- 172.21.15.113:0/2141061271 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe914109e30 con 0x7fe9140fd150 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe9117fa700 1 -- 172.21.15.113:0/2141061271 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe908007e90 con 0x7fe9140fd150 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe9117fa700 1 -- 172.21.15.113:0/2141061271 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe908018070 con 0x7fe9140fd150 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.741+0000 7fe91a449700 1 -- 172.21.15.113:0/2141061271 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe914112420 con 0x7fe9140fd150 2024-04-12T05:37:55.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.742+0000 7fe91a449700 1 -- 172.21.15.113:0/2141061271 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91405f590 con 0x7fe9140fd150 2024-04-12T05:37:55.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.743+0000 7fe9117fa700 1 -- 172.21.15.113:0/2141061271 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe90801d420 con 0x7fe9140fd150 2024-04-12T05:37:55.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.743+0000 7fe9117fa700 1 --2- 172.21.15.113:0/2141061271 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe8fc06caa0 0x7fe8fc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:55.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.743+0000 7fe9117fa700 1 -- 172.21.15.113:0/2141061271 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe90808d2f0 con 0x7fe9140fd150 2024-04-12T05:37:55.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.743+0000 7fe913fff700 1 --2- 172.21.15.113:0/2141061271 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe8fc06caa0 0x7fe8fc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:55.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.743+0000 7fe913fff700 1 --2- 172.21.15.113:0/2141061271 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe8fc06caa0 0x7fe8fc06ef60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fe9040041d0 tx=0x7fe904009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:55.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:55.747+0000 7fe9117fa700 1 -- 172.21.15.113:0/2141061271 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe908057310 con 0x7fe9140fd150 2024-04-12T05:37:56.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.015+0000 7fe91a449700 1 -- 172.21.15.113:0/2141061271 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe9140feb80 con 0x7fe9140fd150 2024-04-12T05:37:56.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.015+0000 7fe9117fa700 1 -- 172.21.15.113:0/2141061271 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fe90805a930 con 0x7fe9140fd150 2024-04-12T05:37:56.013 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:37:56.013 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:37:56.013 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:37:56.013 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:37:56.013 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:37:56.014 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:37:56.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.017+0000 7fe91a449700 1 -- 172.21.15.113:0/2141061271 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe8fc06caa0 msgr2=0x7fe8fc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.017+0000 7fe91a449700 1 --2- 172.21.15.113:0/2141061271 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe8fc06caa0 0x7fe8fc06ef60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fe9040041d0 tx=0x7fe904009040).stop 2024-04-12T05:37:56.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.017+0000 7fe91a449700 1 -- 172.21.15.113:0/2141061271 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9140fd150 msgr2=0x7fe9141095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.017+0000 7fe91a449700 1 --2- 172.21.15.113:0/2141061271 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9140fd150 0x7fe9141095b0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7fe9080099f0 tx=0x7fe908009cb0).stop 2024-04-12T05:37:56.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.017+0000 7fe91a449700 1 -- 172.21.15.113:0/2141061271 shutdown_connections 2024-04-12T05:37:56.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.017+0000 7fe91a449700 1 --2- 172.21.15.113:0/2141061271 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe8fc06caa0 0x7fe8fc06ef60 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.017+0000 7fe91a449700 1 --2- 172.21.15.113:0/2141061271 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9140fc7f0 0x7fe9141090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.017+0000 7fe91a449700 1 --2- 172.21.15.113:0/2141061271 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9140fd150 0x7fe9141095b0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.017+0000 7fe91a449700 1 -- 172.21.15.113:0/2141061271 >> 172.21.15.113:0/2141061271 conn(0x7fe914070590 msgr2=0x7fe9140fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:56.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.017+0000 7fe91a449700 1 -- 172.21.15.113:0/2141061271 shutdown_connections 2024-04-12T05:37:56.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.017+0000 7fe91a449700 1 -- 172.21.15.113:0/2141061271 wait complete. 2024-04-12T05:37:56.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.115+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/444559692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c780fd150 msgr2=0x7f0c780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.115+0000 7f0c7ecfd700 1 --2- 172.21.15.113:0/444559692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c780fd150 0x7f0c780f6930 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f0c640047f0 tx=0x7f0c64009cb0).stop 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.115+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/444559692 shutdown_connections 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.115+0000 7f0c7ecfd700 1 --2- 172.21.15.113:0/444559692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c780fd150 0x7f0c780f6930 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.115+0000 7f0c7ecfd700 1 --2- 172.21.15.113:0/444559692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c780fc7f0 0x7f0c780fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.115+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/444559692 >> 172.21.15.113:0/444559692 conn(0x7f0c78070590 msgr2=0x7f0c780f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.115+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/444559692 shutdown_connections 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.115+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/444559692 wait complete. 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ecfd700 1 Processor -- start 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ecfd700 1 -- start start 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ecfd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c780fc7f0 0x7f0c7810e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ecfd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c780fd150 0x7f0c7810e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:56.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ecfd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7810eaf0 con 0x7f0c780fc7f0 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ecfd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7810ec30 con 0x7f0c780fd150 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ca99700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c780fc7f0 0x7f0c7810e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ca99700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c780fc7f0 0x7f0c7810e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54892/0 (socket says 172.21.15.113:54892) 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ca99700 1 -- 172.21.15.113:0/1043066207 learned_addr learned my addr 172.21.15.113:0/1043066207 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c77fff700 1 --2- 172.21.15.113:0/1043066207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c780fd150 0x7f0c7810e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ca99700 1 -- 172.21.15.113:0/1043066207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c780fd150 msgr2=0x7f0c7810e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ca99700 1 --2- 172.21.15.113:0/1043066207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c780fd150 0x7f0c7810e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ca99700 1 -- 172.21.15.113:0/1043066207 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c64005040 con 0x7f0c780fc7f0 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.116+0000 7f0c7ca99700 1 --2- 172.21.15.113:0/1043066207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c780fc7f0 0x7f0c7810e000 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f0c6c0041d0 tx=0x7f0c6c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.117+0000 7f0c75ffb700 1 -- 172.21.15.113:0/1043066207 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c6c007b40 con 0x7f0c780fc7f0 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.117+0000 7f0c75ffb700 1 -- 172.21.15.113:0/1043066207 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c6c00f040 con 0x7f0c780fc7f0 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.117+0000 7f0c75ffb700 1 -- 172.21.15.113:0/1043066207 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c6c0137d0 con 0x7f0c780fc7f0 2024-04-12T05:37:56.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.117+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/1043066207 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c781082c0 con 0x7f0c780fc7f0 2024-04-12T05:37:56.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.117+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/1043066207 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c78108880 con 0x7f0c780fc7f0 2024-04-12T05:37:56.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.118+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/1043066207 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7805f590 con 0x7f0c780fc7f0 2024-04-12T05:37:56.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.118+0000 7f0c75ffb700 1 -- 172.21.15.113:0/1043066207 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0c6c007ca0 con 0x7f0c780fc7f0 2024-04-12T05:37:56.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.118+0000 7f0c75ffb700 1 --2- 172.21.15.113:0/1043066207 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0c6806caa0 0x7f0c6806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:56.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.119+0000 7f0c75ffb700 1 -- 172.21.15.113:0/1043066207 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0c6c08b7e0 con 0x7f0c780fc7f0 2024-04-12T05:37:56.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.119+0000 7f0c77fff700 1 --2- 172.21.15.113:0/1043066207 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0c6806caa0 0x7f0c6806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:56.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.119+0000 7f0c77fff700 1 --2- 172.21.15.113:0/1043066207 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0c6806caa0 0x7f0c6806ef60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f0c78109940 tx=0x7f0c6400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:56.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.123+0000 7f0c75ffb700 1 -- 172.21.15.113:0/1043066207 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0c6c059290 con 0x7f0c780fc7f0 2024-04-12T05:37:56.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.351+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/1043066207 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0c78109070 con 0x7f0c780fc7f0 2024-04-12T05:37:56.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.352+0000 7f0c75ffb700 1 -- 172.21.15.113:0/1043066207 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f0c78109070 con 0x7f0c780fc7f0 2024-04-12T05:37:56.350 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:37:56.350 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:37:56.350 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:37:56.350 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:37:56.350 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:37:56.350 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:37:56.350 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:37:56.351 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:37:56.352 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:37:56.353 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:37:56.353 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:37:56.353 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:37:56.353 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:37:56.353 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:37:56.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.354+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/1043066207 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0c6806caa0 msgr2=0x7f0c6806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.354+0000 7f0c7ecfd700 1 --2- 172.21.15.113:0/1043066207 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0c6806caa0 0x7f0c6806ef60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f0c78109940 tx=0x7f0c6400b040).stop 2024-04-12T05:37:56.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.354+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/1043066207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c780fc7f0 msgr2=0x7f0c7810e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.354+0000 7f0c7ecfd700 1 --2- 172.21.15.113:0/1043066207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c780fc7f0 0x7f0c7810e000 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f0c6c0041d0 tx=0x7f0c6c009f60).stop 2024-04-12T05:37:56.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.354+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/1043066207 shutdown_connections 2024-04-12T05:37:56.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.354+0000 7f0c7ecfd700 1 --2- 172.21.15.113:0/1043066207 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0c6806caa0 0x7f0c6806ef60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.354+0000 7f0c7ecfd700 1 --2- 172.21.15.113:0/1043066207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c780fc7f0 0x7f0c7810e000 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.354+0000 7f0c7ecfd700 1 --2- 172.21.15.113:0/1043066207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c780fd150 0x7f0c7810e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.355+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/1043066207 >> 172.21.15.113:0/1043066207 conn(0x7f0c78070590 msgr2=0x7f0c78060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:56.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.355+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/1043066207 shutdown_connections 2024-04-12T05:37:56.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.355+0000 7f0c7ecfd700 1 -- 172.21.15.113:0/1043066207 wait complete. 2024-04-12T05:37:56.355 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:37:56.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:56 smithi113 ceph-mon[103371]: from='client.35402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:56.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:56 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2141061271' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:37:56.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:56 smithi192 ceph-mon[85304]: from='client.35402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:56.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:56 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2141061271' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:37:56.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.454+0000 7f1457288700 1 -- 172.21.15.113:0/741429102 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14500fd8f0 msgr2=0x7f14500fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.454+0000 7f1457288700 1 --2- 172.21.15.113:0/741429102 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14500fd8f0 0x7f14500fdcd0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f14400047f0 tx=0x7f1440009cb0).stop 2024-04-12T05:37:56.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.454+0000 7f1457288700 1 -- 172.21.15.113:0/741429102 shutdown_connections 2024-04-12T05:37:56.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.454+0000 7f1457288700 1 --2- 172.21.15.113:0/741429102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14500fe250 0x7f14500f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.454+0000 7f1457288700 1 --2- 172.21.15.113:0/741429102 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14500fd8f0 0x7f14500fdcd0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.454+0000 7f1457288700 1 -- 172.21.15.113:0/741429102 >> 172.21.15.113:0/741429102 conn(0x7f14500f34b0 msgr2=0x7f14500f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:56.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.454+0000 7f1457288700 1 -- 172.21.15.113:0/741429102 shutdown_connections 2024-04-12T05:37:56.452 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.454+0000 7f1457288700 1 -- 172.21.15.113:0/741429102 wait complete. 2024-04-12T05:37:56.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1457288700 1 Processor -- start 2024-04-12T05:37:56.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1457288700 1 -- start start 2024-04-12T05:37:56.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1457288700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14500fd8f0 0x7f145010e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:56.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1457288700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14500fe250 0x7f145010e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:56.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1457288700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145010eaf0 con 0x7f14500fe250 2024-04-12T05:37:56.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1457288700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145010ec30 con 0x7f14500fd8f0 2024-04-12T05:37:56.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1454823700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14500fe250 0x7f145010e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1454823700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14500fe250 0x7f145010e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54908/0 (socket says 172.21.15.113:54908) 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1454823700 1 -- 172.21.15.113:0/2078446828 learned_addr learned my addr 172.21.15.113:0/2078446828 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1455024700 1 --2- 172.21.15.113:0/2078446828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14500fd8f0 0x7f145010e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1454823700 1 -- 172.21.15.113:0/2078446828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14500fd8f0 msgr2=0x7f145010e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1454823700 1 --2- 172.21.15.113:0/2078446828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14500fd8f0 0x7f145010e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1454823700 1 -- 172.21.15.113:0/2078446828 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1440005040 con 0x7f14500fe250 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1454823700 1 --2- 172.21.15.113:0/2078446828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14500fe250 0x7f145010e4f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f144c00c790 tx=0x7f144c007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1446777700 1 -- 172.21.15.113:0/2078446828 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f144c00e680 con 0x7f14500fe250 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1446777700 1 -- 172.21.15.113:0/2078446828 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f144c00ec90 con 0x7f14500fe250 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.455+0000 7f1457288700 1 -- 172.21.15.113:0/2078446828 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14501082c0 con 0x7f14500fe250 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.456+0000 7f1446777700 1 -- 172.21.15.113:0/2078446828 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f144c012cf0 con 0x7f14500fe250 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.456+0000 7f1457288700 1 -- 172.21.15.113:0/2078446828 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1450108880 con 0x7f14500fe250 2024-04-12T05:37:56.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.456+0000 7f1457288700 1 -- 172.21.15.113:0/2078446828 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145005f590 con 0x7f14500fe250 2024-04-12T05:37:56.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.457+0000 7f1446777700 1 -- 172.21.15.113:0/2078446828 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f144c012500 con 0x7f14500fe250 2024-04-12T05:37:56.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.458+0000 7f1446777700 1 --2- 172.21.15.113:0/2078446828 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f143c06caa0 0x7f143c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:56.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.458+0000 7f1446777700 1 -- 172.21.15.113:0/2078446828 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f144c08bf60 con 0x7f14500fe250 2024-04-12T05:37:56.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.458+0000 7f1455024700 1 --2- 172.21.15.113:0/2078446828 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f143c06caa0 0x7f143c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:56.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.458+0000 7f1455024700 1 --2- 172.21.15.113:0/2078446828 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f143c06caa0 0x7f143c06ef60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f1440007cf0 tx=0x7f144000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:56.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.462+0000 7f1446777700 1 -- 172.21.15.113:0/2078446828 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f144c059a10 con 0x7f14500fe250 2024-04-12T05:37:56.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.669+0000 7f1457288700 1 -- 172.21.15.113:0/2078446828 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1450109790 con 0x7f143c06caa0 2024-04-12T05:37:56.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.670+0000 7f1446777700 1 -- 172.21.15.113:0/2078446828 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1450109790 con 0x7f143c06caa0 2024-04-12T05:37:56.668 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:37:56.668 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:37:56.668 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:37:56.668 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:37:56.668 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:37:56.668 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:37:56.669 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:37:56.669 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:37:56.669 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:37:56.669 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:37:56.669 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:37:56.669 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:37:56.669 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:37:56.669 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:37:56.669 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:37:56.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.672+0000 7f1457288700 1 -- 172.21.15.113:0/2078446828 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f143c06caa0 msgr2=0x7f143c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.672+0000 7f1457288700 1 --2- 172.21.15.113:0/2078446828 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f143c06caa0 0x7f143c06ef60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f1440007cf0 tx=0x7f144000b040).stop 2024-04-12T05:37:56.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.672+0000 7f1457288700 1 -- 172.21.15.113:0/2078446828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14500fe250 msgr2=0x7f145010e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.672+0000 7f1457288700 1 --2- 172.21.15.113:0/2078446828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14500fe250 0x7f145010e4f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f144c00c790 tx=0x7f144c007f50).stop 2024-04-12T05:37:56.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.672+0000 7f1457288700 1 -- 172.21.15.113:0/2078446828 shutdown_connections 2024-04-12T05:37:56.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.672+0000 7f1457288700 1 --2- 172.21.15.113:0/2078446828 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f143c06caa0 0x7f143c06ef60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.672+0000 7f1457288700 1 --2- 172.21.15.113:0/2078446828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14500fd8f0 0x7f145010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.672+0000 7f1457288700 1 --2- 172.21.15.113:0/2078446828 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f14500fe250 0x7f145010e4f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.672+0000 7f1457288700 1 -- 172.21.15.113:0/2078446828 >> 172.21.15.113:0/2078446828 conn(0x7f14500f34b0 msgr2=0x7f14500fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:56.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.672+0000 7f1457288700 1 -- 172.21.15.113:0/2078446828 shutdown_connections 2024-04-12T05:37:56.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.672+0000 7f1457288700 1 -- 172.21.15.113:0/2078446828 wait complete. 2024-04-12T05:37:56.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 -- 172.21.15.113:0/3322087235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf140fd8f0 msgr2=0x7faf140fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 --2- 172.21.15.113:0/3322087235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf140fd8f0 0x7faf140fdcd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7faf080047f0 tx=0x7faf08009cb0).stop 2024-04-12T05:37:56.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 -- 172.21.15.113:0/3322087235 shutdown_connections 2024-04-12T05:37:56.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 --2- 172.21.15.113:0/3322087235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf140fe250 0x7faf140f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 --2- 172.21.15.113:0/3322087235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf140fd8f0 0x7faf140fdcd0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 -- 172.21.15.113:0/3322087235 >> 172.21.15.113:0/3322087235 conn(0x7faf140f34b0 msgr2=0x7faf140f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:56.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 -- 172.21.15.113:0/3322087235 shutdown_connections 2024-04-12T05:37:56.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 -- 172.21.15.113:0/3322087235 wait complete. 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 Processor -- start 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 -- start start 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf140fd8f0 0x7faf14110210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf140fe250 0x7faf14110700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf14110d00 con 0x7faf140fe250 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.770+0000 7faf1ab9f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf14110e40 con 0x7faf140fd8f0 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf13fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf140fe250 0x7faf14110700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf13fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf140fe250 0x7faf14110700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54932/0 (socket says 172.21.15.113:54932) 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf13fff700 1 -- 172.21.15.113:0/2160645247 learned_addr learned my addr 172.21.15.113:0/2160645247 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf1893b700 1 --2- 172.21.15.113:0/2160645247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf140fd8f0 0x7faf14110210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf13fff700 1 -- 172.21.15.113:0/2160645247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf140fd8f0 msgr2=0x7faf14110210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf13fff700 1 --2- 172.21.15.113:0/2160645247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf140fd8f0 0x7faf14110210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf13fff700 1 -- 172.21.15.113:0/2160645247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf08005040 con 0x7faf140fe250 2024-04-12T05:37:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf13fff700 1 --2- 172.21.15.113:0/2160645247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf140fe250 0x7faf14110700 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7faf00002820 tx=0x7faf00009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:56.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf11ffb700 1 -- 172.21.15.113:0/2160645247 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf00007b40 con 0x7faf140fe250 2024-04-12T05:37:56.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf11ffb700 1 -- 172.21.15.113:0/2160645247 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf0000f040 con 0x7faf140fe250 2024-04-12T05:37:56.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf1ab9f700 1 -- 172.21.15.113:0/2160645247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1410a520 con 0x7faf140fe250 2024-04-12T05:37:56.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf11ffb700 1 -- 172.21.15.113:0/2160645247 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf000148b0 con 0x7faf140fe250 2024-04-12T05:37:56.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.771+0000 7faf1ab9f700 1 -- 172.21.15.113:0/2160645247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1410aae0 con 0x7faf140fe250 2024-04-12T05:37:56.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.772+0000 7faf1ab9f700 1 -- 172.21.15.113:0/2160645247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1405f590 con 0x7faf140fe250 2024-04-12T05:37:56.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.773+0000 7faf11ffb700 1 -- 172.21.15.113:0/2160645247 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faf00007ca0 con 0x7faf140fe250 2024-04-12T05:37:56.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.773+0000 7faf11ffb700 1 --2- 172.21.15.113:0/2160645247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faf0406caa0 0x7faf0406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:37:56.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.773+0000 7faf11ffb700 1 -- 172.21.15.113:0/2160645247 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7faf0008b8f0 con 0x7faf140fe250 2024-04-12T05:37:56.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.773+0000 7faf1893b700 1 --2- 172.21.15.113:0/2160645247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faf0406caa0 0x7faf0406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:37:56.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.773+0000 7faf1893b700 1 --2- 172.21.15.113:0/2160645247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faf0406caa0 0x7faf0406ef60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7faf08004f10 tx=0x7faf0800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:37:56.775 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:56.777+0000 7faf11ffb700 1 -- 172.21.15.113:0/2160645247 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7faf000593a0 con 0x7faf140fe250 2024-04-12T05:37:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.046+0000 7faf1ab9f700 1 -- 172.21.15.113:0/2160645247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faf14047540 con 0x7faf140fe250 2024-04-12T05:37:57.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.046+0000 7faf11ffb700 1 -- 172.21.15.113:0/2160645247 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7faf00019e10 con 0x7faf140fe250 2024-04-12T05:37:57.044 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:37:57.044 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:37:57.044 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:37:57.044 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:37:57.045 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:37:57.046 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:37:57.046 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:37:57.046 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:37:57.046 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:37:57.046 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:37:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.048+0000 7faf1ab9f700 1 -- 172.21.15.113:0/2160645247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faf0406caa0 msgr2=0x7faf0406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.048+0000 7faf1ab9f700 1 --2- 172.21.15.113:0/2160645247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faf0406caa0 0x7faf0406ef60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7faf08004f10 tx=0x7faf0800b040).stop 2024-04-12T05:37:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.048+0000 7faf1ab9f700 1 -- 172.21.15.113:0/2160645247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf140fe250 msgr2=0x7faf14110700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:37:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.048+0000 7faf1ab9f700 1 --2- 172.21.15.113:0/2160645247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf140fe250 0x7faf14110700 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7faf00002820 tx=0x7faf00009f20).stop 2024-04-12T05:37:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.048+0000 7faf1ab9f700 1 -- 172.21.15.113:0/2160645247 shutdown_connections 2024-04-12T05:37:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.048+0000 7faf1ab9f700 1 --2- 172.21.15.113:0/2160645247 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faf0406caa0 0x7faf0406ef60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.048+0000 7faf1ab9f700 1 --2- 172.21.15.113:0/2160645247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf140fd8f0 0x7faf14110210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.048+0000 7faf1ab9f700 1 --2- 172.21.15.113:0/2160645247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf140fe250 0x7faf14110700 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:37:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.048+0000 7faf1ab9f700 1 -- 172.21.15.113:0/2160645247 >> 172.21.15.113:0/2160645247 conn(0x7faf140f34b0 msgr2=0x7faf1406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:37:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.048+0000 7faf1ab9f700 1 -- 172.21.15.113:0/2160645247 shutdown_connections 2024-04-12T05:37:57.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:37:57.048+0000 7faf1ab9f700 1 -- 172.21.15.113:0/2160645247 wait complete. 2024-04-12T05:37:57.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:57 smithi113 ceph-mon[103371]: from='client.35406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:57.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:57 smithi113 ceph-mon[103371]: from='client.35410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:57.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:57 smithi113 ceph-mon[103371]: pgmap v774: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:57.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:57 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1043066207' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:37:57.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:57 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2160645247' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:37:57.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:57 smithi192 ceph-mon[85304]: from='client.35406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:57.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:57 smithi192 ceph-mon[85304]: from='client.35410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:57.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:57 smithi192 ceph-mon[85304]: pgmap v774: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:57.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:57 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1043066207' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:37:57.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:57 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2160645247' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:37:58.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:58 smithi113 ceph-mon[103371]: from='client.35422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:58.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:58 smithi192 ceph-mon[85304]: from='client.35422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:37:59.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:37:59 smithi113 ceph-mon[103371]: pgmap v775: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:37:59.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:37:59 smithi192 ceph-mon[85304]: pgmap v775: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:01.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:01 smithi113 ceph-mon[103371]: pgmap v776: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:01.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:01 smithi192 ceph-mon[85304]: pgmap v776: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:03.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:03 smithi113 ceph-mon[103371]: pgmap v777: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:03.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:03 smithi192 ceph-mon[85304]: pgmap v777: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:05.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:05 smithi113 ceph-mon[103371]: pgmap v778: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:05.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:05 smithi192 ceph-mon[85304]: pgmap v778: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:07.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:07 smithi113 ceph-mon[103371]: pgmap v779: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:07.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:07.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:07 smithi192 ceph-mon[85304]: pgmap v779: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:07.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:09.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:09 smithi113 ceph-mon[103371]: pgmap v780: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:09.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:09 smithi192 ceph-mon[85304]: pgmap v780: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:11.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:11 smithi113 ceph-mon[103371]: pgmap v781: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:11.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:11 smithi192 ceph-mon[85304]: pgmap v781: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:13.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:13 smithi113 ceph-mon[103371]: pgmap v782: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:13.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:13 smithi192 ceph-mon[85304]: pgmap v782: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:15.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:15 smithi113 ceph-mon[103371]: pgmap v783: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:15.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:15 smithi192 ceph-mon[85304]: pgmap v783: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:17.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:17 smithi113 ceph-mon[103371]: pgmap v784: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:17.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:17 smithi192 ceph-mon[85304]: pgmap v784: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:19.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:19 smithi113 ceph-mon[103371]: pgmap v785: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:19.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:19 smithi192 ceph-mon[85304]: pgmap v785: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:21.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:21 smithi113 ceph-mon[103371]: pgmap v786: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:21.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:21 smithi192 ceph-mon[85304]: pgmap v786: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:22.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:22.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:23.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:23 smithi113 ceph-mon[103371]: pgmap v787: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:23.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:23 smithi192 ceph-mon[85304]: pgmap v787: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:25.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:25 smithi113 ceph-mon[103371]: pgmap v788: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:25.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:25 smithi192 ceph-mon[85304]: pgmap v788: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:27.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.156+0000 7f3b68e41700 1 -- 172.21.15.113:0/3008673414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b640f88a0 msgr2=0x7f3b640f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:27.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.156+0000 7f3b68e41700 1 --2- 172.21.15.113:0/3008673414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b640f88a0 0x7f3b640f8c80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f3b540047f0 tx=0x7f3b54009cb0).stop 2024-04-12T05:38:27.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.156+0000 7f3b68e41700 1 -- 172.21.15.113:0/3008673414 shutdown_connections 2024-04-12T05:38:27.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.156+0000 7f3b68e41700 1 --2- 172.21.15.113:0/3008673414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b640f9200 0x7f3b64101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.156+0000 7f3b68e41700 1 --2- 172.21.15.113:0/3008673414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b640f88a0 0x7f3b640f8c80 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.156+0000 7f3b68e41700 1 -- 172.21.15.113:0/3008673414 >> 172.21.15.113:0/3008673414 conn(0x7f3b64070590 msgr2=0x7f3b640f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:27.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.156+0000 7f3b68e41700 1 -- 172.21.15.113:0/3008673414 shutdown_connections 2024-04-12T05:38:27.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.156+0000 7f3b68e41700 1 -- 172.21.15.113:0/3008673414 wait complete. 2024-04-12T05:38:27.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.157+0000 7f3b68e41700 1 Processor -- start 2024-04-12T05:38:27.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b68e41700 1 -- start start 2024-04-12T05:38:27.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b68e41700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b640f88a0 0x7f3b6410e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:27.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b68e41700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b640f9200 0x7f3b6410e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:27.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b68e41700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6410ec20 con 0x7f3b640f88a0 2024-04-12T05:38:27.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b68e41700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6410ed60 con 0x7f3b640f9200 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b6259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b640f88a0 0x7f3b6410e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b6259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b640f88a0 0x7f3b6410e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56372/0 (socket says 172.21.15.113:56372) 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b6259c700 1 -- 172.21.15.113:0/693313031 learned_addr learned my addr 172.21.15.113:0/693313031 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b61d9b700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b640f9200 0x7f3b6410e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b6259c700 1 -- 172.21.15.113:0/693313031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b640f9200 msgr2=0x7f3b6410e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b6259c700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b640f9200 0x7f3b6410e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b6259c700 1 -- 172.21.15.113:0/693313031 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b54005040 con 0x7f3b640f88a0 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b61d9b700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b640f9200 0x7f3b6410e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.158+0000 7f3b6259c700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b640f88a0 0x7f3b6410e130 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f3b54009380 tx=0x7f3b54009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.159+0000 7f3b537fe700 1 -- 172.21.15.113:0/693313031 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b54018070 con 0x7f3b640f88a0 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.159+0000 7f3b68e41700 1 -- 172.21.15.113:0/693313031 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b64108390 con 0x7f3b640f88a0 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.159+0000 7f3b537fe700 1 -- 172.21.15.113:0/693313031 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b54007e90 con 0x7f3b640f88a0 2024-04-12T05:38:27.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.159+0000 7f3b537fe700 1 -- 172.21.15.113:0/693313031 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b5401c380 con 0x7f3b640f88a0 2024-04-12T05:38:27.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.159+0000 7f3b68e41700 1 -- 172.21.15.113:0/693313031 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b64108870 con 0x7f3b640f88a0 2024-04-12T05:38:27.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.159+0000 7f3b68e41700 1 -- 172.21.15.113:0/693313031 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6405f590 con 0x7f3b640f88a0 2024-04-12T05:38:27.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.160+0000 7f3b537fe700 1 -- 172.21.15.113:0/693313031 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3b540123c0 con 0x7f3b640f88a0 2024-04-12T05:38:27.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.160+0000 7f3b537fe700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3b4c06cab0 0x7f3b4c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:27.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.160+0000 7f3b537fe700 1 -- 172.21.15.113:0/693313031 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3b5408cc60 con 0x7f3b640f88a0 2024-04-12T05:38:27.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.161+0000 7f3b61d9b700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3b4c06cab0 0x7f3b4c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:27.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.161+0000 7f3b61d9b700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3b4c06cab0 0x7f3b4c06ef70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f3b58007fd0 tx=0x7f3b5800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:27.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.165+0000 7f3b537fe700 1 -- 172.21.15.113:0/693313031 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3b5405a710 con 0x7f3b640f88a0 2024-04-12T05:38:27.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.370+0000 7f3b68e41700 1 -- 172.21.15.113:0/693313031 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b640fef10 con 0x7f3b4c06cab0 2024-04-12T05:38:27.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.371+0000 7f3b537fe700 1 -- 172.21.15.113:0/693313031 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3b640fef10 con 0x7f3b4c06cab0 2024-04-12T05:38:27.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.373+0000 7f3b68e41700 1 -- 172.21.15.113:0/693313031 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3b4c06cab0 msgr2=0x7f3b4c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:27.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.373+0000 7f3b68e41700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3b4c06cab0 0x7f3b4c06ef70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f3b58007fd0 tx=0x7f3b5800b040).stop 2024-04-12T05:38:27.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.373+0000 7f3b68e41700 1 -- 172.21.15.113:0/693313031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b640f88a0 msgr2=0x7f3b6410e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:27.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.373+0000 7f3b68e41700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b640f88a0 0x7f3b6410e130 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f3b54009380 tx=0x7f3b54009cb0).stop 2024-04-12T05:38:27.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.373+0000 7f3b68e41700 1 -- 172.21.15.113:0/693313031 shutdown_connections 2024-04-12T05:38:27.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.373+0000 7f3b68e41700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3b4c06cab0 0x7f3b4c06ef70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.374+0000 7f3b68e41700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b640f88a0 0x7f3b6410e130 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.374+0000 7f3b68e41700 1 --2- 172.21.15.113:0/693313031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b640f9200 0x7f3b6410e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.374+0000 7f3b68e41700 1 -- 172.21.15.113:0/693313031 >> 172.21.15.113:0/693313031 conn(0x7f3b64070590 msgr2=0x7f3b640fc250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:27.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.374+0000 7f3b68e41700 1 -- 172.21.15.113:0/693313031 shutdown_connections 2024-04-12T05:38:27.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.374+0000 7f3b68e41700 1 -- 172.21.15.113:0/693313031 wait complete. 2024-04-12T05:38:27.381 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:38:27.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:27 smithi113 ceph-mon[103371]: pgmap v789: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:27.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:27 smithi192 ceph-mon[85304]: pgmap v789: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.473+0000 7fa3a2f94700 1 -- 172.21.15.113:0/2587805475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa39c0fd8f0 msgr2=0x7fa39c0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.473+0000 7fa3a2f94700 1 --2- 172.21.15.113:0/2587805475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa39c0fd8f0 0x7fa39c0fdcd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7fa3900047f0 tx=0x7fa390009cb0).stop 2024-04-12T05:38:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.473+0000 7fa3a2f94700 1 -- 172.21.15.113:0/2587805475 shutdown_connections 2024-04-12T05:38:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.473+0000 7fa3a2f94700 1 --2- 172.21.15.113:0/2587805475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa39c0fe250 0x7fa39c0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.473+0000 7fa3a2f94700 1 --2- 172.21.15.113:0/2587805475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa39c0fd8f0 0x7fa39c0fdcd0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.473+0000 7fa3a2f94700 1 -- 172.21.15.113:0/2587805475 >> 172.21.15.113:0/2587805475 conn(0x7fa39c0f34b0 msgr2=0x7fa39c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.473+0000 7fa3a2f94700 1 -- 172.21.15.113:0/2587805475 shutdown_connections 2024-04-12T05:38:27.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.473+0000 7fa3a2f94700 1 -- 172.21.15.113:0/2587805475 wait complete. 2024-04-12T05:38:27.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa3a2f94700 1 Processor -- start 2024-04-12T05:38:27.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa3a2f94700 1 -- start start 2024-04-12T05:38:27.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa3a2f94700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa39c0fd8f0 0x7fa39c10dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:27.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa3a2f94700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa39c0fe250 0x7fa39c10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:27.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa3a2f94700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c10ead0 con 0x7fa39c0fe250 2024-04-12T05:38:27.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa3a2f94700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c10ec10 con 0x7fa39c0fd8f0 2024-04-12T05:38:27.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa39bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa39c0fe250 0x7fa39c10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:27.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa39bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa39c0fe250 0x7fa39c10e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56380/0 (socket says 172.21.15.113:56380) 2024-04-12T05:38:27.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa39bfff700 1 -- 172.21.15.113:0/963425673 learned_addr learned my addr 172.21.15.113:0/963425673 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:38:27.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa3a0d30700 1 --2- 172.21.15.113:0/963425673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa39c0fd8f0 0x7fa39c10dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:27.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa39bfff700 1 -- 172.21.15.113:0/963425673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa39c0fd8f0 msgr2=0x7fa39c10dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:27.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa39bfff700 1 --2- 172.21.15.113:0/963425673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa39c0fd8f0 0x7fa39c10dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa39bfff700 1 -- 172.21.15.113:0/963425673 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa390005040 con 0x7fa39c0fe250 2024-04-12T05:38:27.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.474+0000 7fa39bfff700 1 --2- 172.21.15.113:0/963425673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa39c0fe250 0x7fa39c10e4d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7fa388002780 tx=0x7fa388009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:27.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.475+0000 7fa399ffb700 1 -- 172.21.15.113:0/963425673 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa388007b40 con 0x7fa39c0fe250 2024-04-12T05:38:27.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.475+0000 7fa399ffb700 1 -- 172.21.15.113:0/963425673 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa38800f040 con 0x7fa39c0fe250 2024-04-12T05:38:27.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.475+0000 7fa3a2f94700 1 -- 172.21.15.113:0/963425673 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c1082a0 con 0x7fa39c0fe250 2024-04-12T05:38:27.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.475+0000 7fa399ffb700 1 -- 172.21.15.113:0/963425673 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa388014780 con 0x7fa39c0fe250 2024-04-12T05:38:27.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.475+0000 7fa3a2f94700 1 -- 172.21.15.113:0/963425673 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c108860 con 0x7fa39c0fe250 2024-04-12T05:38:27.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.476+0000 7fa3a2f94700 1 -- 172.21.15.113:0/963425673 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c05f590 con 0x7fa39c0fe250 2024-04-12T05:38:27.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.476+0000 7fa399ffb700 1 -- 172.21.15.113:0/963425673 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa388007ca0 con 0x7fa39c0fe250 2024-04-12T05:38:27.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.477+0000 7fa399ffb700 1 --2- 172.21.15.113:0/963425673 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa38c06caa0 0x7fa38c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:27.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.477+0000 7fa399ffb700 1 -- 172.21.15.113:0/963425673 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa38808bc70 con 0x7fa39c0fe250 2024-04-12T05:38:27.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.477+0000 7fa3a0d30700 1 --2- 172.21.15.113:0/963425673 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa38c06caa0 0x7fa38c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:27.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.477+0000 7fa3a0d30700 1 --2- 172.21.15.113:0/963425673 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa38c06caa0 0x7fa38c06ef60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa390004f10 tx=0x7fa39000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:27.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.481+0000 7fa399ffb700 1 -- 172.21.15.113:0/963425673 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa388059720 con 0x7fa39c0fe250 2024-04-12T05:38:27.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.686+0000 7fa3a2f94700 1 -- 172.21.15.113:0/963425673 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa39c108b30 con 0x7fa38c06caa0 2024-04-12T05:38:27.685 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.687+0000 7fa399ffb700 1 -- 172.21.15.113:0/963425673 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa39c108b30 con 0x7fa38c06caa0 2024-04-12T05:38:27.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.689+0000 7fa3a2f94700 1 -- 172.21.15.113:0/963425673 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa38c06caa0 msgr2=0x7fa38c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:27.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.689+0000 7fa3a2f94700 1 --2- 172.21.15.113:0/963425673 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa38c06caa0 0x7fa38c06ef60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa390004f10 tx=0x7fa39000b040).stop 2024-04-12T05:38:27.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.689+0000 7fa3a2f94700 1 -- 172.21.15.113:0/963425673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa39c0fe250 msgr2=0x7fa39c10e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:27.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.689+0000 7fa3a2f94700 1 --2- 172.21.15.113:0/963425673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa39c0fe250 0x7fa39c10e4d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7fa388002780 tx=0x7fa388009ee0).stop 2024-04-12T05:38:27.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.689+0000 7fa3a2f94700 1 -- 172.21.15.113:0/963425673 shutdown_connections 2024-04-12T05:38:27.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.689+0000 7fa3a2f94700 1 --2- 172.21.15.113:0/963425673 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa38c06caa0 0x7fa38c06ef60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.689+0000 7fa3a2f94700 1 --2- 172.21.15.113:0/963425673 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa39c0fd8f0 0x7fa39c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.689+0000 7fa3a2f94700 1 --2- 172.21.15.113:0/963425673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa39c0fe250 0x7fa39c10e4d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.689+0000 7fa3a2f94700 1 -- 172.21.15.113:0/963425673 >> 172.21.15.113:0/963425673 conn(0x7fa39c0f34b0 msgr2=0x7fa39c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:27.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.689+0000 7fa3a2f94700 1 -- 172.21.15.113:0/963425673 shutdown_connections 2024-04-12T05:38:27.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.689+0000 7fa3a2f94700 1 -- 172.21.15.113:0/963425673 wait complete. 2024-04-12T05:38:27.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.786+0000 7f4608a52700 1 -- 172.21.15.113:0/1544541839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46040fc8b0 msgr2=0x7f46040f60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:27.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.786+0000 7f4608a52700 1 --2- 172.21.15.113:0/1544541839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46040fc8b0 0x7f46040f60f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f45f80047f0 tx=0x7f45f8009cb0).stop 2024-04-12T05:38:27.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.787+0000 7f4608a52700 1 -- 172.21.15.113:0/1544541839 shutdown_connections 2024-04-12T05:38:27.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.787+0000 7f4608a52700 1 --2- 172.21.15.113:0/1544541839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46040fc8b0 0x7f46040f60f0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.787+0000 7f4608a52700 1 --2- 172.21.15.113:0/1544541839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46040fbf50 0x7f46040fc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.787+0000 7f4608a52700 1 -- 172.21.15.113:0/1544541839 >> 172.21.15.113:0/1544541839 conn(0x7f460406ce40 msgr2=0x7f460406d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:27.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.787+0000 7f4608a52700 1 -- 172.21.15.113:0/1544541839 shutdown_connections 2024-04-12T05:38:27.785 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.787+0000 7f4608a52700 1 -- 172.21.15.113:0/1544541839 wait complete. 2024-04-12T05:38:27.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f4608a52700 1 Processor -- start 2024-04-12T05:38:27.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f4608a52700 1 -- start start 2024-04-12T05:38:27.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f4608a52700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46040fbf50 0x7f46041090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:27.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f4608a52700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46040fc8b0 0x7f46041095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:27.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f4608a52700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4604109aa0 con 0x7f46040fbf50 2024-04-12T05:38:27.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f4608a52700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4604109be0 con 0x7f46040fc8b0 2024-04-12T05:38:27.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f460259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46040fbf50 0x7f46041090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:27.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f460259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46040fbf50 0x7f46041090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56388/0 (socket says 172.21.15.113:56388) 2024-04-12T05:38:27.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f460259c700 1 -- 172.21.15.113:0/1218104925 learned_addr learned my addr 172.21.15.113:0/1218104925 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:38:27.787 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f4601d9b700 1 --2- 172.21.15.113:0/1218104925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46040fc8b0 0x7f46041095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:27.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f460259c700 1 -- 172.21.15.113:0/1218104925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46040fc8b0 msgr2=0x7f46041095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:27.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f460259c700 1 --2- 172.21.15.113:0/1218104925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46040fc8b0 0x7f46041095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:27.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f460259c700 1 -- 172.21.15.113:0/1218104925 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f8005040 con 0x7f46040fbf50 2024-04-12T05:38:27.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.788+0000 7f460259c700 1 --2- 172.21.15.113:0/1218104925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46040fbf50 0x7f46041090c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f45f40041d0 tx=0x7f45f4009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:27.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.789+0000 7f45f37fe700 1 -- 172.21.15.113:0/1218104925 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45f4007b40 con 0x7f46040fbf50 2024-04-12T05:38:27.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.789+0000 7f45f37fe700 1 -- 172.21.15.113:0/1218104925 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45f400f040 con 0x7f46040fbf50 2024-04-12T05:38:27.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.789+0000 7f4608a52700 1 -- 172.21.15.113:0/1218104925 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4604109e90 con 0x7f46040fbf50 2024-04-12T05:38:27.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.789+0000 7f45f37fe700 1 -- 172.21.15.113:0/1218104925 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45f40137d0 con 0x7f46040fbf50 2024-04-12T05:38:27.788 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.789+0000 7f4608a52700 1 -- 172.21.15.113:0/1218104925 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4604112480 con 0x7f46040fbf50 2024-04-12T05:38:27.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.790+0000 7f4608a52700 1 -- 172.21.15.113:0/1218104925 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460405f090 con 0x7f46040fbf50 2024-04-12T05:38:27.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.790+0000 7f45f37fe700 1 -- 172.21.15.113:0/1218104925 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f45f4007ca0 con 0x7f46040fbf50 2024-04-12T05:38:27.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.790+0000 7f45f37fe700 1 --2- 172.21.15.113:0/1218104925 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f45ec06caa0 0x7f45ec06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:27.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.790+0000 7f45f37fe700 1 -- 172.21.15.113:0/1218104925 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f45f408b860 con 0x7f46040fbf50 2024-04-12T05:38:27.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.790+0000 7f4601d9b700 1 --2- 172.21.15.113:0/1218104925 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f45ec06caa0 0x7f45ec06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:27.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.791+0000 7f4601d9b700 1 --2- 172.21.15.113:0/1218104925 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f45ec06caa0 0x7f45ec06ef60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f4604101700 tx=0x7f45f800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:27.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.795+0000 7f45f37fe700 1 -- 172.21.15.113:0/1218104925 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f45f4055880 con 0x7f46040fbf50 2024-04-12T05:38:27.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:27.992+0000 7f4608a52700 1 -- 172.21.15.113:0/1218104925 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4604059120 con 0x7f45ec06caa0 2024-04-12T05:38:28.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.002+0000 7f45f37fe700 1 -- 172.21.15.113:0/1218104925 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4604059120 con 0x7f45ec06caa0 2024-04-12T05:38:28.000 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:38:28.000 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (26m) 3m ago 37m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:38:28.000 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (23m) 3m ago 37m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:38:28.000 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (23m) 3m ago 36m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:38:28.000 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (26m) 3m ago 37m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:38:28.000 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (19m) 3m ago 32m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:38:28.000 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (19m) 3m ago 32m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:38:28.000 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (18m) 3m ago 32m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:38:28.000 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (3m) 3m ago 32m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (27m) 3m ago 38m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (27m) 3m ago 36m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (24m) 3m ago 38m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (24m) 3m ago 36m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (26m) 3m ago 37m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (26m) 3m ago 36m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (23m) 3m ago 36m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (22m) 3m ago 35m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (22m) 3m ago 35m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (21m) 3m ago 34m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (20m) 3m ago 34m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (19m) 3m ago 33m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:38:28.001 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (26m) 3m ago 36m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:38:28.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.004+0000 7f4608a52700 1 -- 172.21.15.113:0/1218104925 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f45ec06caa0 msgr2=0x7f45ec06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.004+0000 7f4608a52700 1 --2- 172.21.15.113:0/1218104925 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f45ec06caa0 0x7f45ec06ef60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f4604101700 tx=0x7f45f800b040).stop 2024-04-12T05:38:28.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.004+0000 7f4608a52700 1 -- 172.21.15.113:0/1218104925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46040fbf50 msgr2=0x7f46041090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.004+0000 7f4608a52700 1 --2- 172.21.15.113:0/1218104925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46040fbf50 0x7f46041090c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f45f40041d0 tx=0x7f45f4009f60).stop 2024-04-12T05:38:28.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.005+0000 7f4608a52700 1 -- 172.21.15.113:0/1218104925 shutdown_connections 2024-04-12T05:38:28.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.005+0000 7f4608a52700 1 --2- 172.21.15.113:0/1218104925 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f45ec06caa0 0x7f45ec06ef60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.005+0000 7f4608a52700 1 --2- 172.21.15.113:0/1218104925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46040fbf50 0x7f46041090c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.005+0000 7f4608a52700 1 --2- 172.21.15.113:0/1218104925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46040fc8b0 0x7f46041095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.005+0000 7f4608a52700 1 -- 172.21.15.113:0/1218104925 >> 172.21.15.113:0/1218104925 conn(0x7f460406ce40 msgr2=0x7f46040fa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:28.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.005+0000 7f4608a52700 1 -- 172.21.15.113:0/1218104925 shutdown_connections 2024-04-12T05:38:28.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.005+0000 7f4608a52700 1 -- 172.21.15.113:0/1218104925 wait complete. 2024-04-12T05:38:28.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.101+0000 7fb540fc6700 1 -- 172.21.15.113:0/1580761197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb53c0fd150 msgr2=0x7fb53c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.101+0000 7fb540fc6700 1 --2- 172.21.15.113:0/1580761197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb53c0fd150 0x7fb53c0f6930 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7fb52c0047f0 tx=0x7fb52c009cb0).stop 2024-04-12T05:38:28.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.101+0000 7fb540fc6700 1 -- 172.21.15.113:0/1580761197 shutdown_connections 2024-04-12T05:38:28.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.101+0000 7fb540fc6700 1 --2- 172.21.15.113:0/1580761197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb53c0fd150 0x7fb53c0f6930 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.101+0000 7fb540fc6700 1 --2- 172.21.15.113:0/1580761197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb53c0fc7f0 0x7fb53c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.101+0000 7fb540fc6700 1 -- 172.21.15.113:0/1580761197 >> 172.21.15.113:0/1580761197 conn(0x7fb53c070590 msgr2=0x7fb53c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:28.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.102+0000 7fb540fc6700 1 -- 172.21.15.113:0/1580761197 shutdown_connections 2024-04-12T05:38:28.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.102+0000 7fb540fc6700 1 -- 172.21.15.113:0/1580761197 wait complete. 2024-04-12T05:38:28.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.102+0000 7fb540fc6700 1 Processor -- start 2024-04-12T05:38:28.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.102+0000 7fb540fc6700 1 -- start start 2024-04-12T05:38:28.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.102+0000 7fb540fc6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb53c0fc7f0 0x7fb53c1101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:28.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.102+0000 7fb540fc6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb53c0fd150 0x7fb53c1106c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:28.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.102+0000 7fb540fc6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c110cc0 con 0x7fb53c0fd150 2024-04-12T05:38:28.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.102+0000 7fb540fc6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c110e00 con 0x7fb53c0fc7f0 2024-04-12T05:38:28.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb539d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb53c0fd150 0x7fb53c1106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:28.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb539d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb53c0fd150 0x7fb53c1106c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56412/0 (socket says 172.21.15.113:56412) 2024-04-12T05:38:28.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb539d9b700 1 -- 172.21.15.113:0/2659683325 learned_addr learned my addr 172.21.15.113:0/2659683325 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:38:28.101 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb53a59c700 1 --2- 172.21.15.113:0/2659683325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb53c0fc7f0 0x7fb53c1101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:28.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb539d9b700 1 -- 172.21.15.113:0/2659683325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb53c0fc7f0 msgr2=0x7fb53c1101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb539d9b700 1 --2- 172.21.15.113:0/2659683325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb53c0fc7f0 0x7fb53c1101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb539d9b700 1 -- 172.21.15.113:0/2659683325 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c005040 con 0x7fb53c0fd150 2024-04-12T05:38:28.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb539d9b700 1 --2- 172.21.15.113:0/2659683325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb53c0fd150 0x7fb53c1106c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fb52c0099f0 tx=0x7fb52c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:28.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb52b7fe700 1 -- 172.21.15.113:0/2659683325 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb52c00e030 con 0x7fb53c0fd150 2024-04-12T05:38:28.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb540fc6700 1 -- 172.21.15.113:0/2659683325 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53c10a430 con 0x7fb53c0fd150 2024-04-12T05:38:28.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb52b7fe700 1 -- 172.21.15.113:0/2659683325 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb52c007e90 con 0x7fb53c0fd150 2024-04-12T05:38:28.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb52b7fe700 1 -- 172.21.15.113:0/2659683325 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb52c018070 con 0x7fb53c0fd150 2024-04-12T05:38:28.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.103+0000 7fb540fc6700 1 -- 172.21.15.113:0/2659683325 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53c10a990 con 0x7fb53c0fd150 2024-04-12T05:38:28.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.104+0000 7fb540fc6700 1 -- 172.21.15.113:0/2659683325 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53c05f590 con 0x7fb53c0fd150 2024-04-12T05:38:28.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.105+0000 7fb52b7fe700 1 -- 172.21.15.113:0/2659683325 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb52c009590 con 0x7fb53c0fd150 2024-04-12T05:38:28.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.105+0000 7fb52b7fe700 1 --2- 172.21.15.113:0/2659683325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb52406ca30 0x7fb52406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:28.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.105+0000 7fb52b7fe700 1 -- 172.21.15.113:0/2659683325 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb52c08d130 con 0x7fb53c0fd150 2024-04-12T05:38:28.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.105+0000 7fb53a59c700 1 --2- 172.21.15.113:0/2659683325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb52406ca30 0x7fb52406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:28.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.106+0000 7fb53a59c700 1 --2- 172.21.15.113:0/2659683325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb52406ca30 0x7fb52406eef0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb530009170 tx=0x7fb530009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:28.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.110+0000 7fb52b7fe700 1 -- 172.21.15.113:0/2659683325 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb52c05abe0 con 0x7fb53c0fd150 2024-04-12T05:38:28.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.376+0000 7fb540fc6700 1 -- 172.21.15.113:0/2659683325 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb53c047540 con 0x7fb53c0fd150 2024-04-12T05:38:28.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.376+0000 7fb52b7fe700 1 -- 172.21.15.113:0/2659683325 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fb52c05a770 con 0x7fb53c0fd150 2024-04-12T05:38:28.374 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:38:28.374 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:38:28.374 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:38:28.374 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:38:28.375 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:38:28.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.378+0000 7fb540fc6700 1 -- 172.21.15.113:0/2659683325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb52406ca30 msgr2=0x7fb52406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.378+0000 7fb540fc6700 1 --2- 172.21.15.113:0/2659683325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb52406ca30 0x7fb52406eef0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb530009170 tx=0x7fb530009d60).stop 2024-04-12T05:38:28.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.379+0000 7fb540fc6700 1 -- 172.21.15.113:0/2659683325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb53c0fd150 msgr2=0x7fb53c1106c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.379+0000 7fb540fc6700 1 --2- 172.21.15.113:0/2659683325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb53c0fd150 0x7fb53c1106c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fb52c0099f0 tx=0x7fb52c009cb0).stop 2024-04-12T05:38:28.376 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.379+0000 7fb540fc6700 1 -- 172.21.15.113:0/2659683325 shutdown_connections 2024-04-12T05:38:28.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.379+0000 7fb540fc6700 1 --2- 172.21.15.113:0/2659683325 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb52406ca30 0x7fb52406eef0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.379+0000 7fb540fc6700 1 --2- 172.21.15.113:0/2659683325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb53c0fc7f0 0x7fb53c1101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.379+0000 7fb540fc6700 1 --2- 172.21.15.113:0/2659683325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb53c0fd150 0x7fb53c1106c0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.379+0000 7fb540fc6700 1 -- 172.21.15.113:0/2659683325 >> 172.21.15.113:0/2659683325 conn(0x7fb53c070590 msgr2=0x7fb53c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:28.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.379+0000 7fb540fc6700 1 -- 172.21.15.113:0/2659683325 shutdown_connections 2024-04-12T05:38:28.377 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.379+0000 7fb540fc6700 1 -- 172.21.15.113:0/2659683325 wait complete. 2024-04-12T05:38:28.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.476+0000 7f79809c9700 1 -- 172.21.15.113:0/650293731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f797c0fd150 msgr2=0x7f797c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.476+0000 7f79809c9700 1 --2- 172.21.15.113:0/650293731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f797c0fd150 0x7f797c0f6930 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f796c0047f0 tx=0x7f796c009cb0).stop 2024-04-12T05:38:28.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.476+0000 7f79809c9700 1 -- 172.21.15.113:0/650293731 shutdown_connections 2024-04-12T05:38:28.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.476+0000 7f79809c9700 1 --2- 172.21.15.113:0/650293731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f797c0fd150 0x7f797c0f6930 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.476+0000 7f79809c9700 1 --2- 172.21.15.113:0/650293731 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c0fc7f0 0x7f797c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.476+0000 7f79809c9700 1 -- 172.21.15.113:0/650293731 >> 172.21.15.113:0/650293731 conn(0x7f797c070590 msgr2=0x7f797c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:28.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.476+0000 7f79809c9700 1 -- 172.21.15.113:0/650293731 shutdown_connections 2024-04-12T05:38:28.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.476+0000 7f79809c9700 1 -- 172.21.15.113:0/650293731 wait complete. 2024-04-12T05:38:28.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.476+0000 7f79809c9700 1 Processor -- start 2024-04-12T05:38:28.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f79809c9700 1 -- start start 2024-04-12T05:38:28.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f79809c9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c0fc7f0 0x7f797c10cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:28.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f79809c9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f797c0fd150 0x7f797c10d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:28.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f79809c9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c10d9e0 con 0x7f797c0fd150 2024-04-12T05:38:28.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f79809c9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c10db20 con 0x7f797c0fc7f0 2024-04-12T05:38:28.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f7979d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f797c0fd150 0x7f797c10d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f7979d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f797c0fd150 0x7f797c10d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56426/0 (socket says 172.21.15.113:56426) 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f7979d9b700 1 -- 172.21.15.113:0/3484194782 learned_addr learned my addr 172.21.15.113:0/3484194782 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f797a59c700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c0fc7f0 0x7f797c10cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f7979d9b700 1 -- 172.21.15.113:0/3484194782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c0fc7f0 msgr2=0x7f797c10cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f7979d9b700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c0fc7f0 0x7f797c10cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f7979d9b700 1 -- 172.21.15.113:0/3484194782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c005040 con 0x7f797c0fd150 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f797a59c700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c0fc7f0 0x7f797c10cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f7979d9b700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f797c0fd150 0x7f797c10d3e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f796c0099f0 tx=0x7f796c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f79737fe700 1 -- 172.21.15.113:0/3484194782 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f796c00e030 con 0x7f797c0fd150 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.477+0000 7f79737fe700 1 -- 172.21.15.113:0/3484194782 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f796c007e90 con 0x7f797c0fd150 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.478+0000 7f79809c9700 1 -- 172.21.15.113:0/3484194782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797c107150 con 0x7f797c0fd150 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.478+0000 7f79737fe700 1 -- 172.21.15.113:0/3484194782 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f796c018070 con 0x7f797c0fd150 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.478+0000 7f79809c9700 1 -- 172.21.15.113:0/3484194782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797c1076b0 con 0x7f797c0fd150 2024-04-12T05:38:28.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.478+0000 7f79809c9700 1 -- 172.21.15.113:0/3484194782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797c05f590 con 0x7f797c0fd150 2024-04-12T05:38:28.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.479+0000 7f79737fe700 1 -- 172.21.15.113:0/3484194782 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f796c009590 con 0x7f797c0fd150 2024-04-12T05:38:28.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.479+0000 7f79737fe700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f796806caa0 0x7f796806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:28.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.479+0000 7f79737fe700 1 -- 172.21.15.113:0/3484194782 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f796c08d010 con 0x7f797c0fd150 2024-04-12T05:38:28.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.479+0000 7f797a59c700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f796806caa0 0x7f796806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:28.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.480+0000 7f797a59c700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f796806caa0 0x7f796806ef60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f7964009250 tx=0x7f7964009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:28.482 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.484+0000 7f79737fe700 1 -- 172.21.15.113:0/3484194782 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f796c05aac0 con 0x7f797c0fd150 2024-04-12T05:38:28.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.714+0000 7f79809c9700 1 -- 172.21.15.113:0/3484194782 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f797c047540 con 0x7f797c0fd150 2024-04-12T05:38:28.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.714+0000 7f79737fe700 1 -- 172.21.15.113:0/3484194782 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f796c05a650 con 0x7f797c0fd150 2024-04-12T05:38:28.712 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:38:28.712 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:38:28.713 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:38:28.714 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:38:28.715 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:38:28.715 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:38:28.715 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:38:28.715 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:38:28.715 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:38:28.715 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:38:28.715 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:38:28.715 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:38:28.715 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:38:28.715 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:38:28.715 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:38:28.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.716+0000 7f79809c9700 1 -- 172.21.15.113:0/3484194782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f796806caa0 msgr2=0x7f796806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.716+0000 7f79809c9700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f796806caa0 0x7f796806ef60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f7964009250 tx=0x7f7964009da0).stop 2024-04-12T05:38:28.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.716+0000 7f79809c9700 1 -- 172.21.15.113:0/3484194782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f797c0fd150 msgr2=0x7f797c10d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.716+0000 7f79809c9700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f797c0fd150 0x7f797c10d3e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f796c0099f0 tx=0x7f796c009cb0).stop 2024-04-12T05:38:28.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.717+0000 7f79809c9700 1 -- 172.21.15.113:0/3484194782 shutdown_connections 2024-04-12T05:38:28.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.717+0000 7f79809c9700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f796806caa0 0x7f796806ef60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.717+0000 7f79809c9700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f797c0fc7f0 0x7f797c10cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.717+0000 7f79809c9700 1 --2- 172.21.15.113:0/3484194782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f797c0fd150 0x7f797c10d3e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.717+0000 7f79809c9700 1 -- 172.21.15.113:0/3484194782 >> 172.21.15.113:0/3484194782 conn(0x7f797c070590 msgr2=0x7f797c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:28.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.717+0000 7f79809c9700 1 -- 172.21.15.113:0/3484194782 shutdown_connections 2024-04-12T05:38:28.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.717+0000 7f79809c9700 1 -- 172.21.15.113:0/3484194782 wait complete. 2024-04-12T05:38:28.717 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:38:28.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.815+0000 7f63a1fd6700 1 -- 172.21.15.113:0/2910671649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f639c0fd8f0 msgr2=0x7f639c0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.815+0000 7f63a1fd6700 1 --2- 172.21.15.113:0/2910671649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f639c0fd8f0 0x7f639c0fdcd0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f638c009b60 tx=0x7f638c004d10).stop 2024-04-12T05:38:28.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.815+0000 7f63a1fd6700 1 -- 172.21.15.113:0/2910671649 shutdown_connections 2024-04-12T05:38:28.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.815+0000 7f63a1fd6700 1 --2- 172.21.15.113:0/2910671649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f639c0fe250 0x7f639c0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.815+0000 7f63a1fd6700 1 --2- 172.21.15.113:0/2910671649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f639c0fd8f0 0x7f639c0fdcd0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.815+0000 7f63a1fd6700 1 -- 172.21.15.113:0/2910671649 >> 172.21.15.113:0/2910671649 conn(0x7f639c0f34b0 msgr2=0x7f639c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:28.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.815+0000 7f63a1fd6700 1 -- 172.21.15.113:0/2910671649 shutdown_connections 2024-04-12T05:38:28.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.815+0000 7f63a1fd6700 1 -- 172.21.15.113:0/2910671649 wait complete. 2024-04-12T05:38:28.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f63a1fd6700 1 Processor -- start 2024-04-12T05:38:28.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f63a1fd6700 1 -- start start 2024-04-12T05:38:28.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f63a1fd6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f639c0fd8f0 0x7f639c10e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:28.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f63a1fd6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f639c0fe250 0x7f639c10e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:28.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f63a1fd6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639c10eaf0 con 0x7f639c0fd8f0 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f63a1fd6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639c10ec30 con 0x7f639c0fe250 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f639b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f639c0fd8f0 0x7f639c10e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f639b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f639c0fd8f0 0x7f639c10e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56432/0 (socket says 172.21.15.113:56432) 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f639b7fe700 1 -- 172.21.15.113:0/3463808942 learned_addr learned my addr 172.21.15.113:0/3463808942 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f639affd700 1 --2- 172.21.15.113:0/3463808942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f639c0fe250 0x7f639c10e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f639b7fe700 1 -- 172.21.15.113:0/3463808942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f639c0fe250 msgr2=0x7f639c10e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f639b7fe700 1 --2- 172.21.15.113:0/3463808942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f639c0fe250 0x7f639c10e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f639b7fe700 1 -- 172.21.15.113:0/3463808942 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c0097c0 con 0x7f639c0fd8f0 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.816+0000 7f639b7fe700 1 --2- 172.21.15.113:0/3463808942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f639c0fd8f0 0x7f639c10e000 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f638c009040 tx=0x7f638c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.817+0000 7f6398ff9700 1 -- 172.21.15.113:0/3463808942 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f638c00e070 con 0x7f639c0fd8f0 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.817+0000 7f6398ff9700 1 -- 172.21.15.113:0/3463808942 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f638c00bec0 con 0x7f639c0fd8f0 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.817+0000 7f6398ff9700 1 -- 172.21.15.113:0/3463808942 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f638c019070 con 0x7f639c0fd8f0 2024-04-12T05:38:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.817+0000 7f63a1fd6700 1 -- 172.21.15.113:0/3463808942 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639c108260 con 0x7f639c0fd8f0 2024-04-12T05:38:28.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.817+0000 7f63a1fd6700 1 -- 172.21.15.113:0/3463808942 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639c1087c0 con 0x7f639c0fd8f0 2024-04-12T05:38:28.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.818+0000 7f63a1fd6700 1 -- 172.21.15.113:0/3463808942 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f639c05f590 con 0x7f639c0fd8f0 2024-04-12T05:38:28.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.819+0000 7f6398ff9700 1 -- 172.21.15.113:0/3463808942 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f638c0139d0 con 0x7f639c0fd8f0 2024-04-12T05:38:28.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.819+0000 7f6398ff9700 1 --2- 172.21.15.113:0/3463808942 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638806caa0 0x7f638806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:28.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.819+0000 7f6398ff9700 1 -- 172.21.15.113:0/3463808942 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f638c08d160 con 0x7f639c0fd8f0 2024-04-12T05:38:28.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.819+0000 7f639affd700 1 --2- 172.21.15.113:0/3463808942 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638806caa0 0x7f638806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:28.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.819+0000 7f639affd700 1 --2- 172.21.15.113:0/3463808942 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638806caa0 0x7f638806ef60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f639c109940 tx=0x7f6384009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:28.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:28.823+0000 7f6398ff9700 1 -- 172.21.15.113:0/3463808942 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f638c05ac10 con 0x7f639c0fd8f0 2024-04-12T05:38:29.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.028+0000 7f63a1fd6700 1 -- 172.21.15.113:0/3463808942 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f639c0ffc40 con 0x7f638806caa0 2024-04-12T05:38:29.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.029+0000 7f6398ff9700 1 -- 172.21.15.113:0/3463808942 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f639c0ffc40 con 0x7f638806caa0 2024-04-12T05:38:29.027 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:38:29.027 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:38:29.027 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:38:29.027 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:38:29.027 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:38:29.028 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:38:29.028 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:38:29.028 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:38:29.028 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:38:29.028 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:38:29.028 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:38:29.028 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:38:29.028 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:38:29.028 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:38:29.028 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:38:29.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.031+0000 7f63a1fd6700 1 -- 172.21.15.113:0/3463808942 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638806caa0 msgr2=0x7f638806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:29.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.031+0000 7f63a1fd6700 1 --2- 172.21.15.113:0/3463808942 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638806caa0 0x7f638806ef60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f639c109940 tx=0x7f6384009da0).stop 2024-04-12T05:38:29.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.031+0000 7f63a1fd6700 1 -- 172.21.15.113:0/3463808942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f639c0fd8f0 msgr2=0x7f639c10e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:29.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.031+0000 7f63a1fd6700 1 --2- 172.21.15.113:0/3463808942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f639c0fd8f0 0x7f639c10e000 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f638c009040 tx=0x7f638c004d10).stop 2024-04-12T05:38:29.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.032+0000 7f63a1fd6700 1 -- 172.21.15.113:0/3463808942 shutdown_connections 2024-04-12T05:38:29.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.032+0000 7f63a1fd6700 1 --2- 172.21.15.113:0/3463808942 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f638806caa0 0x7f638806ef60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:29.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.032+0000 7f63a1fd6700 1 --2- 172.21.15.113:0/3463808942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f639c0fd8f0 0x7f639c10e000 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:29.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.032+0000 7f63a1fd6700 1 --2- 172.21.15.113:0/3463808942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f639c0fe250 0x7f639c10e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:29.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.032+0000 7f63a1fd6700 1 -- 172.21.15.113:0/3463808942 >> 172.21.15.113:0/3463808942 conn(0x7f639c0f34b0 msgr2=0x7f639c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:29.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.032+0000 7f63a1fd6700 1 -- 172.21.15.113:0/3463808942 shutdown_connections 2024-04-12T05:38:29.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.032+0000 7f63a1fd6700 1 -- 172.21.15.113:0/3463808942 wait complete. 2024-04-12T05:38:29.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.129+0000 7f5cd071b700 1 -- 172.21.15.113:0/2368353927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc80fd150 msgr2=0x7f5cc80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:29.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.129+0000 7f5cd071b700 1 --2- 172.21.15.113:0/2368353927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc80fd150 0x7f5cc80f6930 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f5cc40047f0 tx=0x7f5cc4009cb0).stop 2024-04-12T05:38:29.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.129+0000 7f5cd071b700 1 -- 172.21.15.113:0/2368353927 shutdown_connections 2024-04-12T05:38:29.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.129+0000 7f5cd071b700 1 --2- 172.21.15.113:0/2368353927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc80fd150 0x7f5cc80f6930 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:29.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.129+0000 7f5cd071b700 1 --2- 172.21.15.113:0/2368353927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cc80fc7f0 0x7f5cc80fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:29.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.129+0000 7f5cd071b700 1 -- 172.21.15.113:0/2368353927 >> 172.21.15.113:0/2368353927 conn(0x7f5cc8070590 msgr2=0x7f5cc80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:29.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.129+0000 7f5cd071b700 1 -- 172.21.15.113:0/2368353927 shutdown_connections 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.129+0000 7f5cd071b700 1 -- 172.21.15.113:0/2368353927 wait complete. 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.129+0000 7f5cd071b700 1 Processor -- start 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cd071b700 1 -- start start 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cd071b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc80fc7f0 0x7f5cc810cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cd071b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cc80fd150 0x7f5cc810d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cd071b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc810d9e0 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cd071b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc810db20 con 0x7f5cc80fd150 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cce4b7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc80fc7f0 0x7f5cc810cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cce4b7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc80fc7f0 0x7f5cc810cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56444/0 (socket says 172.21.15.113:56444) 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cce4b7700 1 -- 172.21.15.113:0/3521689345 learned_addr learned my addr 172.21.15.113:0/3521689345 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cce4b7700 1 -- 172.21.15.113:0/3521689345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cc80fd150 msgr2=0x7f5cc810d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cce4b7700 1 --2- 172.21.15.113:0/3521689345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cc80fd150 0x7f5cc810d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cce4b7700 1 -- 172.21.15.113:0/3521689345 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc4005040 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cce4b7700 1 --2- 172.21.15.113:0/3521689345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc80fc7f0 0x7f5cc810cef0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f5cbc00c790 tx=0x7f5cbc00cb50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:29.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cbb7fe700 1 -- 172.21.15.113:0/3521689345 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cbc0106f0 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cbb7fe700 1 -- 172.21.15.113:0/3521689345 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cbc010d00 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cd071b700 1 -- 172.21.15.113:0/3521689345 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc81071b0 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cbb7fe700 1 -- 172.21.15.113:0/3521689345 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cbc012e50 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.130+0000 7f5cd071b700 1 -- 172.21.15.113:0/3521689345 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc8107770 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.131+0000 7f5cd071b700 1 -- 172.21.15.113:0/3521689345 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc805f590 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.132+0000 7f5cbb7fe700 1 -- 172.21.15.113:0/3521689345 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5cbc010850 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.132+0000 7f5cbb7fe700 1 --2- 172.21.15.113:0/3521689345 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5cb406caa0 0x7f5cb406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:29.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.132+0000 7f5cbb7fe700 1 -- 172.21.15.113:0/3521689345 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5cbc08b700 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.132+0000 7f5ccdcb6700 1 --2- 172.21.15.113:0/3521689345 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5cb406caa0 0x7f5cb406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:29.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.133+0000 7f5ccdcb6700 1 --2- 172.21.15.113:0/3521689345 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5cb406caa0 0x7f5cb406ef60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f5cc8108830 tx=0x7f5cc4009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:29.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.137+0000 7f5cbb7fe700 1 -- 172.21.15.113:0/3521689345 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5cbc0591b0 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.403+0000 7f5cd071b700 1 -- 172.21.15.113:0/3521689345 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5cc8047540 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.403+0000 7f5cbb7fe700 1 -- 172.21.15.113:0/3521689345 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f5cbc016e10 con 0x7f5cc80fc7f0 2024-04-12T05:38:29.401 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:38:29.401 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:38:29.401 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:38:29.401 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:38:29.401 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:38:29.402 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:38:29.403 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:38:29.403 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:38:29.403 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:38:29.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.405+0000 7f5cd071b700 1 -- 172.21.15.113:0/3521689345 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5cb406caa0 msgr2=0x7f5cb406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:29.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.405+0000 7f5cd071b700 1 --2- 172.21.15.113:0/3521689345 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5cb406caa0 0x7f5cb406ef60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f5cc8108830 tx=0x7f5cc4009cb0).stop 2024-04-12T05:38:29.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.405+0000 7f5cd071b700 1 -- 172.21.15.113:0/3521689345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc80fc7f0 msgr2=0x7f5cc810cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:29.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.405+0000 7f5cd071b700 1 --2- 172.21.15.113:0/3521689345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc80fc7f0 0x7f5cc810cef0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f5cbc00c790 tx=0x7f5cbc00cb50).stop 2024-04-12T05:38:29.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.405+0000 7f5cd071b700 1 -- 172.21.15.113:0/3521689345 shutdown_connections 2024-04-12T05:38:29.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.405+0000 7f5cd071b700 1 --2- 172.21.15.113:0/3521689345 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5cb406caa0 0x7f5cb406ef60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:29.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.405+0000 7f5cd071b700 1 --2- 172.21.15.113:0/3521689345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5cc80fc7f0 0x7f5cc810cef0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:29.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.405+0000 7f5cd071b700 1 --2- 172.21.15.113:0/3521689345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cc80fd150 0x7f5cc810d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:38:29.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.405+0000 7f5cd071b700 1 -- 172.21.15.113:0/3521689345 >> 172.21.15.113:0/3521689345 conn(0x7f5cc8070590 msgr2=0x7f5cc80fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:29.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.406+0000 7f5cd071b700 1 -- 172.21.15.113:0/3521689345 shutdown_connections 2024-04-12T05:38:29.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:29.406+0000 7f5cd071b700 1 -- 172.21.15.113:0/3521689345 wait complete. 2024-04-12T05:38:29.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:29 smithi113 ceph-mon[103371]: from='client.35430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:29.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:29 smithi113 ceph-mon[103371]: from='client.35434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:29.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:29 smithi113 ceph-mon[103371]: pgmap v790: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:29.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:29 smithi113 ceph-mon[103371]: from='client.35438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:29.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:29 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2659683325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:38:29.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:29 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3484194782' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:38:29.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:29 smithi192 ceph-mon[85304]: from='client.35430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:29.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:29 smithi192 ceph-mon[85304]: from='client.35434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:29.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:29 smithi192 ceph-mon[85304]: pgmap v790: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:29.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:29 smithi192 ceph-mon[85304]: from='client.35438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:29.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:29 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2659683325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:38:29.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:29 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3484194782' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:38:30.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:30 smithi113 ceph-mon[103371]: from='client.35450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:30.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:30 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3521689345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:38:30.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:30 smithi192 ceph-mon[85304]: from='client.35450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:38:30.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:30 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3521689345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:38:31.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:31 smithi113 ceph-mon[103371]: pgmap v791: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:31.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:31 smithi192 ceph-mon[85304]: pgmap v791: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:33.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:33 smithi113 ceph-mon[103371]: pgmap v792: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:33.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:33 smithi192 ceph-mon[85304]: pgmap v792: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:35.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:35 smithi113 ceph-mon[103371]: pgmap v793: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:35.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:35 smithi192 ceph-mon[85304]: pgmap v793: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:37.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:37 smithi113 ceph-mon[103371]: pgmap v794: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:37.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:37.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:37 smithi192 ceph-mon[85304]: pgmap v794: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:37.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:39.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:39 smithi192 ceph-mon[85304]: pgmap v795: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:39 smithi113 ceph-mon[103371]: pgmap v795: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:41.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:41 smithi192 ceph-mon[85304]: pgmap v796: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:41.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:41 smithi113 ceph-mon[103371]: pgmap v796: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:43.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:43 smithi192 ceph-mon[85304]: pgmap v797: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:43.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:43 smithi113 ceph-mon[103371]: pgmap v797: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:45.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:45 smithi192 ceph-mon[85304]: pgmap v798: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:45.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:45 smithi113 ceph-mon[103371]: pgmap v798: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:47.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:47 smithi192 ceph-mon[85304]: pgmap v799: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:47.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:47 smithi113 ceph-mon[103371]: pgmap v799: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:49.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:49 smithi192 ceph-mon[85304]: pgmap v800: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:49.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:49 smithi113 ceph-mon[103371]: pgmap v800: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:51.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:51 smithi192 ceph-mon[85304]: pgmap v801: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:51.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:51 smithi113 ceph-mon[103371]: pgmap v801: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:52.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:52.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:38:53.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:53 smithi113 ceph-mon[103371]: pgmap v802: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:53.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:53 smithi192 ceph-mon[85304]: pgmap v802: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:55.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:55 smithi113 ceph-mon[103371]: pgmap v803: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:55.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:55 smithi192 ceph-mon[85304]: pgmap v803: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:57.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:57 smithi113 ceph-mon[103371]: pgmap v804: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:57.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:57 smithi192 ceph-mon[85304]: pgmap v804: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:59.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.516+0000 7f6fcb372700 1 -- 172.21.15.113:0/901429552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fc40fc7f0 msgr2=0x7f6fc40fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:59.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.516+0000 7f6fcb372700 1 --2- 172.21.15.113:0/901429552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fc40fc7f0 0x7f6fc40fcbd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7f6fc0009b60 tx=0x7f6fc0004d10).stop 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.516+0000 7f6fcb372700 1 -- 172.21.15.113:0/901429552 shutdown_connections 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.516+0000 7f6fcb372700 1 --2- 172.21.15.113:0/901429552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc40fd150 0x7f6fc40f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.516+0000 7f6fcb372700 1 --2- 172.21.15.113:0/901429552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fc40fc7f0 0x7f6fc40fcbd0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.516+0000 7f6fcb372700 1 -- 172.21.15.113:0/901429552 >> 172.21.15.113:0/901429552 conn(0x7f6fc40f23b0 msgr2=0x7f6fc40f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.516+0000 7f6fcb372700 1 -- 172.21.15.113:0/901429552 shutdown_connections 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.516+0000 7f6fcb372700 1 -- 172.21.15.113:0/901429552 wait complete. 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fcb372700 1 Processor -- start 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fcb372700 1 -- start start 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fcb372700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fc40fc7f0 0x7f6fc410e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fcb372700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc40fd150 0x7f6fc410e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fcb372700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc410ec10 con 0x7f6fc40fc7f0 2024-04-12T05:38:59.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fcb372700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc4108230 con 0x7f6fc40fd150 2024-04-12T05:38:59.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fc910e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fc40fc7f0 0x7f6fc410e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:59.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fc910e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fc40fc7f0 0x7f6fc410e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42590/0 (socket says 172.21.15.113:42590) 2024-04-12T05:38:59.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fc910e700 1 -- 172.21.15.113:0/4174978615 learned_addr learned my addr 172.21.15.113:0/4174978615 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:38:59.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fc890d700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc40fd150 0x7f6fc410e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:59.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fc910e700 1 -- 172.21.15.113:0/4174978615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc40fd150 msgr2=0x7f6fc410e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:59.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fc910e700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc40fd150 0x7f6fc410e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:59.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fc910e700 1 -- 172.21.15.113:0/4174978615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc00097c0 con 0x7f6fc40fc7f0 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.517+0000 7f6fc890d700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc40fd150 0x7f6fc410e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.518+0000 7f6fc910e700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fc40fc7f0 0x7f6fc410e090 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f6fc0009040 tx=0x7f6fc0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.518+0000 7f6fba7fc700 1 -- 172.21.15.113:0/4174978615 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fc000e070 con 0x7f6fc40fc7f0 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.518+0000 7f6fba7fc700 1 -- 172.21.15.113:0/4174978615 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fc000bec0 con 0x7f6fc40fc7f0 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.518+0000 7f6fcb372700 1 -- 172.21.15.113:0/4174978615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc4108480 con 0x7f6fc40fc7f0 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.518+0000 7f6fba7fc700 1 -- 172.21.15.113:0/4174978615 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fc0019070 con 0x7f6fc40fc7f0 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.518+0000 7f6fcb372700 1 -- 172.21.15.113:0/4174978615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc4108960 con 0x7f6fc40fc7f0 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.519+0000 7f6fcb372700 1 -- 172.21.15.113:0/4174978615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc4060370 con 0x7f6fc40fc7f0 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.519+0000 7f6fba7fc700 1 -- 172.21.15.113:0/4174978615 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6fc001d6b0 con 0x7f6fc40fc7f0 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.520+0000 7f6fba7fc700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6fb006caa0 0x7f6fb006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.520+0000 7f6fba7fc700 1 -- 172.21.15.113:0/4174978615 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6fc008d240 con 0x7f6fc40fc7f0 2024-04-12T05:38:59.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.520+0000 7f6fc890d700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6fb006caa0 0x7f6fb006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:59.519 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.520+0000 7f6fc890d700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6fb006caa0 0x7f6fb006ef60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f6fc41099d0 tx=0x7f6fb4008040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:59.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.524+0000 7f6fba7fc700 1 -- 172.21.15.113:0/4174978615 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6fc005acf0 con 0x7f6fc40fc7f0 2024-04-12T05:38:59.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:38:59 smithi113 ceph-mon[103371]: pgmap v805: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:59.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:38:59 smithi192 ceph-mon[85304]: pgmap v805: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:38:59.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.730+0000 7f6fcb372700 1 -- 172.21.15.113:0/4174978615 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fc4109060 con 0x7f6fb006caa0 2024-04-12T05:38:59.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.732+0000 7f6fba7fc700 1 -- 172.21.15.113:0/4174978615 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6fc4109060 con 0x7f6fb006caa0 2024-04-12T05:38:59.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.734+0000 7f6fcb372700 1 -- 172.21.15.113:0/4174978615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6fb006caa0 msgr2=0x7f6fb006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:59.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.734+0000 7f6fcb372700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6fb006caa0 0x7f6fb006ef60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f6fc41099d0 tx=0x7f6fb4008040).stop 2024-04-12T05:38:59.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.734+0000 7f6fcb372700 1 -- 172.21.15.113:0/4174978615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fc40fc7f0 msgr2=0x7f6fc410e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:59.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.734+0000 7f6fcb372700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fc40fc7f0 0x7f6fc410e090 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f6fc0009040 tx=0x7f6fc0004d10).stop 2024-04-12T05:38:59.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.734+0000 7f6fcb372700 1 -- 172.21.15.113:0/4174978615 shutdown_connections 2024-04-12T05:38:59.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.734+0000 7f6fcb372700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6fb006caa0 0x7f6fb006ef60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:59.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.734+0000 7f6fcb372700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fc40fc7f0 0x7f6fc410e090 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:59.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.734+0000 7f6fcb372700 1 --2- 172.21.15.113:0/4174978615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc40fd150 0x7f6fc410e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:59.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.734+0000 7f6fcb372700 1 -- 172.21.15.113:0/4174978615 >> 172.21.15.113:0/4174978615 conn(0x7f6fc40f23b0 msgr2=0x7f6fc4060b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:59.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.734+0000 7f6fcb372700 1 -- 172.21.15.113:0/4174978615 shutdown_connections 2024-04-12T05:38:59.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.734+0000 7f6fcb372700 1 -- 172.21.15.113:0/4174978615 wait complete. 2024-04-12T05:38:59.741 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:38:59.831 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.831+0000 7faefc4f1700 1 -- 172.21.15.113:0/614067201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef40fd070 msgr2=0x7faef40f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.831+0000 7faefc4f1700 1 --2- 172.21.15.113:0/614067201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef40fd070 0x7faef40f6850 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7faee40047f0 tx=0x7faee4009cb0).stop 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.831+0000 7faefc4f1700 1 -- 172.21.15.113:0/614067201 shutdown_connections 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.831+0000 7faefc4f1700 1 --2- 172.21.15.113:0/614067201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef40fd070 0x7faef40f6850 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.831+0000 7faefc4f1700 1 --2- 172.21.15.113:0/614067201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef40fc710 0x7faef40fcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.831+0000 7faefc4f1700 1 -- 172.21.15.113:0/614067201 >> 172.21.15.113:0/614067201 conn(0x7faef40f22f0 msgr2=0x7faef40f4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.831+0000 7faefc4f1700 1 -- 172.21.15.113:0/614067201 shutdown_connections 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.831+0000 7faefc4f1700 1 -- 172.21.15.113:0/614067201 wait complete. 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.832+0000 7faefc4f1700 1 Processor -- start 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.832+0000 7faefc4f1700 1 -- start start 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.832+0000 7faefc4f1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef40fc710 0x7faef410e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.832+0000 7faefc4f1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef40fd070 0x7faef410e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.832+0000 7faefc4f1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef410eb70 con 0x7faef40fc710 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.832+0000 7faefc4f1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef410ecb0 con 0x7faef40fd070 2024-04-12T05:38:59.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.832+0000 7faefa28d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef40fc710 0x7faef410e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.832+0000 7faefa28d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef40fc710 0x7faef410e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42608/0 (socket says 172.21.15.113:42608) 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.832+0000 7faefa28d700 1 -- 172.21.15.113:0/940894292 learned_addr learned my addr 172.21.15.113:0/940894292 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.833+0000 7faef9a8c700 1 --2- 172.21.15.113:0/940894292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef40fd070 0x7faef410e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.833+0000 7faefa28d700 1 -- 172.21.15.113:0/940894292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef40fd070 msgr2=0x7faef410e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.833+0000 7faefa28d700 1 --2- 172.21.15.113:0/940894292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef40fd070 0x7faef410e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.833+0000 7faefa28d700 1 -- 172.21.15.113:0/940894292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee4005040 con 0x7faef40fc710 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.833+0000 7faefa28d700 1 --2- 172.21.15.113:0/940894292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef40fc710 0x7faef410e080 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7faef00041d0 tx=0x7faef0009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.833+0000 7faeeb7fe700 1 -- 172.21.15.113:0/940894292 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faef0002d90 con 0x7faef40fc710 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.833+0000 7faefc4f1700 1 -- 172.21.15.113:0/940894292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef4108130 con 0x7faef40fc710 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.833+0000 7faeeb7fe700 1 -- 172.21.15.113:0/940894292 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faef000f040 con 0x7faef40fc710 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.833+0000 7faeeb7fe700 1 -- 172.21.15.113:0/940894292 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faef000d800 con 0x7faef40fc710 2024-04-12T05:38:59.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.833+0000 7faefc4f1700 1 -- 172.21.15.113:0/940894292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef41086f0 con 0x7faef40fc710 2024-04-12T05:38:59.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.834+0000 7faefc4f1700 1 -- 172.21.15.113:0/940894292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef4060180 con 0x7faef40fc710 2024-04-12T05:38:59.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.834+0000 7faeeb7fe700 1 -- 172.21.15.113:0/940894292 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faef000dac0 con 0x7faef40fc710 2024-04-12T05:38:59.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.835+0000 7faeeb7fe700 1 --2- 172.21.15.113:0/940894292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee006caf0 0x7faee006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:38:59.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.835+0000 7faef9a8c700 1 --2- 172.21.15.113:0/940894292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee006caf0 0x7faee006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:38:59.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.835+0000 7faeeb7fe700 1 -- 172.21.15.113:0/940894292 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7faef008bea0 con 0x7faef40fc710 2024-04-12T05:38:59.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.835+0000 7faef9a8c700 1 --2- 172.21.15.113:0/940894292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee006caf0 0x7faee006efb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7faee4004f10 tx=0x7faee400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:38:59.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:38:59.839+0000 7faeeb7fe700 1 -- 172.21.15.113:0/940894292 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7faef0059950 con 0x7faef40fc710 2024-04-12T05:39:00.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.045+0000 7faefc4f1700 1 -- 172.21.15.113:0/940894292 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faef4109700 con 0x7faee006caf0 2024-04-12T05:39:00.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.046+0000 7faeeb7fe700 1 -- 172.21.15.113:0/940894292 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7faef4109700 con 0x7faee006caf0 2024-04-12T05:39:00.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.048+0000 7faefc4f1700 1 -- 172.21.15.113:0/940894292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee006caf0 msgr2=0x7faee006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.048+0000 7faefc4f1700 1 --2- 172.21.15.113:0/940894292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee006caf0 0x7faee006efb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7faee4004f10 tx=0x7faee400b040).stop 2024-04-12T05:39:00.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.048+0000 7faefc4f1700 1 -- 172.21.15.113:0/940894292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef40fc710 msgr2=0x7faef410e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.048+0000 7faefc4f1700 1 --2- 172.21.15.113:0/940894292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef40fc710 0x7faef410e080 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7faef00041d0 tx=0x7faef0009f60).stop 2024-04-12T05:39:00.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.048+0000 7faefc4f1700 1 -- 172.21.15.113:0/940894292 shutdown_connections 2024-04-12T05:39:00.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.048+0000 7faefc4f1700 1 --2- 172.21.15.113:0/940894292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee006caf0 0x7faee006efb0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.048+0000 7faefc4f1700 1 --2- 172.21.15.113:0/940894292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef40fc710 0x7faef410e080 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.048+0000 7faefc4f1700 1 --2- 172.21.15.113:0/940894292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef40fd070 0x7faef410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.048+0000 7faefc4f1700 1 -- 172.21.15.113:0/940894292 >> 172.21.15.113:0/940894292 conn(0x7faef40f22f0 msgr2=0x7faef4061150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:00.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.048+0000 7faefc4f1700 1 -- 172.21.15.113:0/940894292 shutdown_connections 2024-04-12T05:39:00.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.048+0000 7faefc4f1700 1 -- 172.21.15.113:0/940894292 wait complete. 2024-04-12T05:39:00.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.146+0000 7f2ba622c700 1 -- 172.21.15.113:0/4183444538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ba00f88a0 msgr2=0x7f2ba00f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.146+0000 7f2ba622c700 1 --2- 172.21.15.113:0/4183444538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ba00f88a0 0x7f2ba00f8c80 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f2b940047f0 tx=0x7f2b94009cb0).stop 2024-04-12T05:39:00.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.146+0000 7f2ba622c700 1 -- 172.21.15.113:0/4183444538 shutdown_connections 2024-04-12T05:39:00.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.146+0000 7f2ba622c700 1 --2- 172.21.15.113:0/4183444538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ba00f9200 0x7f2ba0101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.146+0000 7f2ba622c700 1 --2- 172.21.15.113:0/4183444538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ba00f88a0 0x7f2ba00f8c80 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.146+0000 7f2ba622c700 1 -- 172.21.15.113:0/4183444538 >> 172.21.15.113:0/4183444538 conn(0x7f2ba0070590 msgr2=0x7f2ba00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:00.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.146+0000 7f2ba622c700 1 -- 172.21.15.113:0/4183444538 shutdown_connections 2024-04-12T05:39:00.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.146+0000 7f2ba622c700 1 -- 172.21.15.113:0/4183444538 wait complete. 2024-04-12T05:39:00.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.146+0000 7f2ba622c700 1 Processor -- start 2024-04-12T05:39:00.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.146+0000 7f2ba622c700 1 -- start start 2024-04-12T05:39:00.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2ba622c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ba00f88a0 0x7f2ba010e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:00.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2ba622c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ba00f9200 0x7f2ba010e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:00.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2ba622c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba010eb10 con 0x7f2ba00f88a0 2024-04-12T05:39:00.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2ba622c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba010ec50 con 0x7f2ba00f9200 2024-04-12T05:39:00.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2b9f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ba00f88a0 0x7f2ba010e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:00.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2b9f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ba00f88a0 0x7f2ba010e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42632/0 (socket says 172.21.15.113:42632) 2024-04-12T05:39:00.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2b9f7fe700 1 -- 172.21.15.113:0/2143447755 learned_addr learned my addr 172.21.15.113:0/2143447755 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:00.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2b9f7fe700 1 -- 172.21.15.113:0/2143447755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ba00f9200 msgr2=0x7f2ba010e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2b9f7fe700 1 --2- 172.21.15.113:0/2143447755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ba00f9200 0x7f2ba010e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:39:00.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2b9f7fe700 1 -- 172.21.15.113:0/2143447755 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b94005040 con 0x7f2ba00f88a0 2024-04-12T05:39:00.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2b9f7fe700 1 --2- 172.21.15.113:0/2143447755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ba00f88a0 0x7f2ba010e020 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f2b9400b120 tx=0x7f2b94009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:00.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2b9cff9700 1 -- 172.21.15.113:0/2143447755 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b9400e030 con 0x7f2ba00f88a0 2024-04-12T05:39:00.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2b9cff9700 1 -- 172.21.15.113:0/2143447755 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b94007e90 con 0x7f2ba00f88a0 2024-04-12T05:39:00.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2b9cff9700 1 -- 172.21.15.113:0/2143447755 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b94018070 con 0x7f2ba00f88a0 2024-04-12T05:39:00.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2ba622c700 1 -- 172.21.15.113:0/2143447755 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba0108280 con 0x7f2ba00f88a0 2024-04-12T05:39:00.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.147+0000 7f2ba622c700 1 -- 172.21.15.113:0/2143447755 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba0108760 con 0x7f2ba00f88a0 2024-04-12T05:39:00.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.148+0000 7f2ba622c700 1 -- 172.21.15.113:0/2143447755 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba005f590 con 0x7f2ba00f88a0 2024-04-12T05:39:00.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.149+0000 7f2b9cff9700 1 -- 172.21.15.113:0/2143447755 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2b9401d420 con 0x7f2ba00f88a0 2024-04-12T05:39:00.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.149+0000 7f2b9cff9700 1 --2- 172.21.15.113:0/2143447755 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b8806caa0 0x7f2b8806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:00.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.149+0000 7f2b9cff9700 1 -- 172.21.15.113:0/2143447755 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2b9408d250 con 0x7f2ba00f88a0 2024-04-12T05:39:00.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.149+0000 7f2b9effd700 1 --2- 172.21.15.113:0/2143447755 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b8806caa0 0x7f2b8806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:00.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.150+0000 7f2b9effd700 1 --2- 172.21.15.113:0/2143447755 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b8806caa0 0x7f2b8806ef60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f2ba0109960 tx=0x7f2b90007b40).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:00.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.154+0000 7f2b9cff9700 1 -- 172.21.15.113:0/2143447755 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2b9405ad00 con 0x7f2ba00f88a0 2024-04-12T05:39:00.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.351+0000 7f2ba622c700 1 -- 172.21.15.113:0/2143447755 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ba0109740 con 0x7f2b8806caa0 2024-04-12T05:39:00.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.362+0000 7f2b9cff9700 1 -- 172.21.15.113:0/2143447755 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2ba0109740 con 0x7f2b8806caa0 2024-04-12T05:39:00.360 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:39:00.360 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (26m) 4m ago 38m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:39:00.360 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (24m) 4m ago 37m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:39:00.360 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (24m) 4m ago 37m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (26m) 4m ago 37m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (20m) 4m ago 32m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (19m) 4m ago 32m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (19m) 4m ago 32m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (4m) 4m ago 32m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (28m) 4m ago 39m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (27m) 4m ago 37m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (25m) 4m ago 39m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (24m) 4m ago 37m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (27m) 4m ago 37m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (27m) 4m ago 37m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (23m) 4m ago 36m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (23m) 4m ago 36m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (22m) 4m ago 35m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:39:00.361 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (22m) 4m ago 35m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:39:00.362 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (21m) 4m ago 34m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:39:00.362 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (20m) 4m ago 34m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:39:00.362 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (27m) 4m ago 36m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:39:00.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.364+0000 7f2ba622c700 1 -- 172.21.15.113:0/2143447755 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b8806caa0 msgr2=0x7f2b8806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.364+0000 7f2ba622c700 1 --2- 172.21.15.113:0/2143447755 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b8806caa0 0x7f2b8806ef60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f2ba0109960 tx=0x7f2b90007b40).stop 2024-04-12T05:39:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.364+0000 7f2ba622c700 1 -- 172.21.15.113:0/2143447755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ba00f88a0 msgr2=0x7f2ba010e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.364+0000 7f2ba622c700 1 --2- 172.21.15.113:0/2143447755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ba00f88a0 0x7f2ba010e020 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f2b9400b120 tx=0x7f2b94009cb0).stop 2024-04-12T05:39:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.364+0000 7f2ba622c700 1 -- 172.21.15.113:0/2143447755 shutdown_connections 2024-04-12T05:39:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.364+0000 7f2ba622c700 1 --2- 172.21.15.113:0/2143447755 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b8806caa0 0x7f2b8806ef60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.364+0000 7f2ba622c700 1 --2- 172.21.15.113:0/2143447755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2ba00f88a0 0x7f2ba010e020 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.364+0000 7f2ba622c700 1 --2- 172.21.15.113:0/2143447755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ba00f9200 0x7f2ba010e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:39:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.364+0000 7f2ba622c700 1 -- 172.21.15.113:0/2143447755 >> 172.21.15.113:0/2143447755 conn(0x7f2ba0070590 msgr2=0x7f2ba00fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.364+0000 7f2ba622c700 1 -- 172.21.15.113:0/2143447755 shutdown_connections 2024-04-12T05:39:00.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.364+0000 7f2ba622c700 1 -- 172.21.15.113:0/2143447755 wait complete. 2024-04-12T05:39:00.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.461+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1477227038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fdc0fe360 msgr2=0x7f2fdc0f5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.461+0000 7f2fe41f2700 1 --2- 172.21.15.113:0/1477227038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fdc0fe360 0x7f2fdc0f5fc0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f2fcc0047f0 tx=0x7f2fcc009cb0).stop 2024-04-12T05:39:00.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.461+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1477227038 shutdown_connections 2024-04-12T05:39:00.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.461+0000 7f2fe41f2700 1 --2- 172.21.15.113:0/1477227038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fdc0fe360 0x7f2fdc0f5fc0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.461+0000 7f2fe41f2700 1 --2- 172.21.15.113:0/1477227038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fdc0fda00 0x7f2fdc0fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.461 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.461+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1477227038 >> 172.21.15.113:0/1477227038 conn(0x7f2fdc06cea0 msgr2=0x7f2fdc06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.461+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1477227038 shutdown_connections 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.461+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1477227038 wait complete. 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.461+0000 7f2fe41f2700 1 Processor -- start 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.461+0000 7f2fe41f2700 1 -- start start 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe41f2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fdc0fda00 0x7f2fdc10e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe41f2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fdc0fe360 0x7f2fdc10e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe41f2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc10ec20 con 0x7f2fdc0fda00 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe41f2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc10ed60 con 0x7f2fdc0fe360 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe1f8e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fdc0fda00 0x7f2fdc10e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe1f8e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fdc0fda00 0x7f2fdc10e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56582/0 (socket says 172.21.15.113:56582) 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe1f8e700 1 -- 172.21.15.113:0/1303694806 learned_addr learned my addr 172.21.15.113:0/1303694806 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe178d700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fdc0fe360 0x7f2fdc10e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe1f8e700 1 -- 172.21.15.113:0/1303694806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fdc0fe360 msgr2=0x7f2fdc10e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.462 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe1f8e700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fdc0fe360 0x7f2fdc10e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe1f8e700 1 -- 172.21.15.113:0/1303694806 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc005040 con 0x7f2fdc0fda00 2024-04-12T05:39:00.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe178d700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fdc0fe360 0x7f2fdc10e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:39:00.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe1f8e700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fdc0fda00 0x7f2fdc10e130 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f2fd800e910 tx=0x7f2fd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:00.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fd2ffd700 1 -- 172.21.15.113:0/1303694806 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fd800eeb0 con 0x7f2fdc0fda00 2024-04-12T05:39:00.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fd2ffd700 1 -- 172.21.15.113:0/1303694806 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fd800f040 con 0x7f2fdc0fda00 2024-04-12T05:39:00.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fd2ffd700 1 -- 172.21.15.113:0/1303694806 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fd800c8a0 con 0x7f2fdc0fda00 2024-04-12T05:39:00.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1303694806 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fdc108390 con 0x7f2fdc0fda00 2024-04-12T05:39:00.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.462+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1303694806 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fdc108950 con 0x7f2fdc0fda00 2024-04-12T05:39:00.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.463+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1303694806 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fdc047540 con 0x7f2fdc0fda00 2024-04-12T05:39:00.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.464+0000 7f2fd2ffd700 1 -- 172.21.15.113:0/1303694806 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2fd80090d0 con 0x7f2fdc0fda00 2024-04-12T05:39:00.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.464+0000 7f2fd2ffd700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2fc806caa0 0x7f2fc806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:00.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.464+0000 7f2fd2ffd700 1 -- 172.21.15.113:0/1303694806 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2fd808b880 con 0x7f2fdc0fda00 2024-04-12T05:39:00.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.464+0000 7f2fe178d700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2fc806caa0 0x7f2fc806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:00.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.465+0000 7f2fe178d700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2fc806caa0 0x7f2fc806ef60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f2fcc004540 tx=0x7f2fcc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:00.467 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.469+0000 7f2fd2ffd700 1 -- 172.21.15.113:0/1303694806 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2fd8059330 con 0x7f2fdc0fda00 2024-04-12T05:39:00.735 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.736+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1303694806 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2fdc0ffdc0 con 0x7f2fdc0fda00 2024-04-12T05:39:00.736 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.737+0000 7f2fd2ffd700 1 -- 172.21.15.113:0/1303694806 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f2fd8055070 con 0x7f2fdc0fda00 2024-04-12T05:39:00.736 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:39:00.736 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:39:00.736 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:39:00.736 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:39:00.736 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:39:00.737 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:39:00.737 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:39:00.737 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:39:00.737 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:39:00.737 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:39:00.737 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:39:00.737 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:39:00.738 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:39:00.738 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:39:00.738 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:39:00.738 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:39:00.738 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:39:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.739+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1303694806 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2fc806caa0 msgr2=0x7f2fc806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.739+0000 7f2fe41f2700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2fc806caa0 0x7f2fc806ef60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f2fcc004540 tx=0x7f2fcc00b040).stop 2024-04-12T05:39:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.739+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1303694806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fdc0fda00 msgr2=0x7f2fdc10e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.739+0000 7f2fe41f2700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fdc0fda00 0x7f2fdc10e130 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f2fd800e910 tx=0x7f2fd800b040).stop 2024-04-12T05:39:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.739+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1303694806 shutdown_connections 2024-04-12T05:39:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.739+0000 7f2fe41f2700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2fc806caa0 0x7f2fc806ef60 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.739+0000 7f2fe41f2700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fdc0fda00 0x7f2fdc10e130 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.739+0000 7f2fe41f2700 1 --2- 172.21.15.113:0/1303694806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fdc0fe360 0x7f2fdc10e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.739+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1303694806 >> 172.21.15.113:0/1303694806 conn(0x7f2fdc06cea0 msgr2=0x7f2fdc0fbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:00.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.739+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1303694806 shutdown_connections 2024-04-12T05:39:00.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.739+0000 7f2fe41f2700 1 -- 172.21.15.113:0/1303694806 wait complete. 2024-04-12T05:39:00.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.837+0000 7f0fed820700 1 -- 172.21.15.113:0/2017848066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fe80fd150 msgr2=0x7f0fe80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.837+0000 7f0fed820700 1 --2- 172.21.15.113:0/2017848066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fe80fd150 0x7f0fe80f6930 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f0fd80047f0 tx=0x7f0fd8009cb0).stop 2024-04-12T05:39:00.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.837+0000 7f0fed820700 1 -- 172.21.15.113:0/2017848066 shutdown_connections 2024-04-12T05:39:00.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.837+0000 7f0fed820700 1 --2- 172.21.15.113:0/2017848066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fe80fd150 0x7f0fe80f6930 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.837+0000 7f0fed820700 1 --2- 172.21.15.113:0/2017848066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fe80fc7f0 0x7f0fe80fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.837+0000 7f0fed820700 1 -- 172.21.15.113:0/2017848066 >> 172.21.15.113:0/2017848066 conn(0x7f0fe8070590 msgr2=0x7f0fe80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:00.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.837+0000 7f0fed820700 1 -- 172.21.15.113:0/2017848066 shutdown_connections 2024-04-12T05:39:00.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.837+0000 7f0fed820700 1 -- 172.21.15.113:0/2017848066 wait complete. 2024-04-12T05:39:00.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.837+0000 7f0fed820700 1 Processor -- start 2024-04-12T05:39:00.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.837+0000 7f0fed820700 1 -- start start 2024-04-12T05:39:00.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fed820700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fe80fc7f0 0x7f0fe81090e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:00.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fed820700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fe80fd150 0x7f0fe81095d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:00.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fed820700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8109ac0 con 0x7f0fe80fc7f0 2024-04-12T05:39:00.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fed820700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8109c00 con 0x7f0fe80fd150 2024-04-12T05:39:00.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fe6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fe80fc7f0 0x7f0fe81090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:00.839 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fe6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fe80fc7f0 0x7f0fe81090e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56600/0 (socket says 172.21.15.113:56600) 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fe6ffd700 1 -- 172.21.15.113:0/1674755714 learned_addr learned my addr 172.21.15.113:0/1674755714 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fe67fc700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fe80fd150 0x7f0fe81095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fe6ffd700 1 -- 172.21.15.113:0/1674755714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fe80fd150 msgr2=0x7f0fe81095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fe6ffd700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fe80fd150 0x7f0fe81095d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fe6ffd700 1 -- 172.21.15.113:0/1674755714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd8005040 con 0x7f0fe80fc7f0 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fe67fc700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fe80fd150 0x7f0fe81095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fe6ffd700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fe80fc7f0 0x7f0fe81090e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f0fdc00c790 tx=0x7f0fdc00cb50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fcffff700 1 -- 172.21.15.113:0/1674755714 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fdc0106f0 con 0x7f0fe80fc7f0 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fcffff700 1 -- 172.21.15.113:0/1674755714 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fdc010d00 con 0x7f0fe80fc7f0 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fed820700 1 -- 172.21.15.113:0/1674755714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe8109eb0 con 0x7f0fe80fc7f0 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fcffff700 1 -- 172.21.15.113:0/1674755714 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fdc012e50 con 0x7f0fe80fc7f0 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.838+0000 7f0fed820700 1 -- 172.21.15.113:0/1674755714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe81124a0 con 0x7f0fe80fc7f0 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.839+0000 7f0fed820700 1 -- 172.21.15.113:0/1674755714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe805f590 con 0x7f0fe80fc7f0 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.840+0000 7f0fcffff700 1 -- 172.21.15.113:0/1674755714 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0fdc010850 con 0x7f0fe80fc7f0 2024-04-12T05:39:00.840 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.840+0000 7f0fcffff700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0fd006caa0 0x7f0fd006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:00.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.840+0000 7f0fcffff700 1 -- 172.21.15.113:0/1674755714 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0fdc08b790 con 0x7f0fe80fc7f0 2024-04-12T05:39:00.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.840+0000 7f0fe67fc700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0fd006caa0 0x7f0fd006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:00.841 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.841+0000 7f0fe67fc700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0fd006caa0 0x7f0fd006ef60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0fd80043d0 tx=0x7f0fd800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:00.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:00.844+0000 7f0fcffff700 1 -- 172.21.15.113:0/1674755714 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0fdc0557b0 con 0x7f0fe80fc7f0 2024-04-12T05:39:01.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.074+0000 7f0fed820700 1 -- 172.21.15.113:0/1674755714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0fe8047540 con 0x7f0fe80fc7f0 2024-04-12T05:39:01.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.075+0000 7f0fcffff700 1 -- 172.21.15.113:0/1674755714 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f0fdc058dd0 con 0x7f0fe80fc7f0 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:39:01.074 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:39:01.075 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:39:01.076 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:39:01.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.077+0000 7f0fed820700 1 -- 172.21.15.113:0/1674755714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0fd006caa0 msgr2=0x7f0fd006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:01.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.077+0000 7f0fed820700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0fd006caa0 0x7f0fd006ef60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0fd80043d0 tx=0x7f0fd800b040).stop 2024-04-12T05:39:01.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.077+0000 7f0fed820700 1 -- 172.21.15.113:0/1674755714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fe80fc7f0 msgr2=0x7f0fe81090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:01.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.077+0000 7f0fed820700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fe80fc7f0 0x7f0fe81090e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f0fdc00c790 tx=0x7f0fdc00cb50).stop 2024-04-12T05:39:01.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.077+0000 7f0fed820700 1 -- 172.21.15.113:0/1674755714 shutdown_connections 2024-04-12T05:39:01.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.077+0000 7f0fed820700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0fd006caa0 0x7f0fd006ef60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.077+0000 7f0fed820700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0fe80fc7f0 0x7f0fe81090e0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.077+0000 7f0fed820700 1 --2- 172.21.15.113:0/1674755714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fe80fd150 0x7f0fe81095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.077+0000 7f0fed820700 1 -- 172.21.15.113:0/1674755714 >> 172.21.15.113:0/1674755714 conn(0x7f0fe8070590 msgr2=0x7f0fe80fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:01.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.077+0000 7f0fed820700 1 -- 172.21.15.113:0/1674755714 shutdown_connections 2024-04-12T05:39:01.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.077+0000 7f0fed820700 1 -- 172.21.15.113:0/1674755714 wait complete. 2024-04-12T05:39:01.078 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:39:01.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.175+0000 7fad61a6a700 1 -- 172.21.15.113:0/1883477563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad5c0fc7f0 msgr2=0x7fad5c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:01.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.175+0000 7fad61a6a700 1 --2- 172.21.15.113:0/1883477563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad5c0fc7f0 0x7fad5c0fcbd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7fad4c0047f0 tx=0x7fad4c009cb0).stop 2024-04-12T05:39:01.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.175+0000 7fad61a6a700 1 -- 172.21.15.113:0/1883477563 shutdown_connections 2024-04-12T05:39:01.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.175+0000 7fad61a6a700 1 --2- 172.21.15.113:0/1883477563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad5c0fd150 0x7fad5c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.175+0000 7fad61a6a700 1 --2- 172.21.15.113:0/1883477563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad5c0fc7f0 0x7fad5c0fcbd0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.175+0000 7fad61a6a700 1 -- 172.21.15.113:0/1883477563 >> 172.21.15.113:0/1883477563 conn(0x7fad5c070590 msgr2=0x7fad5c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:01.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.175+0000 7fad61a6a700 1 -- 172.21.15.113:0/1883477563 shutdown_connections 2024-04-12T05:39:01.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.175+0000 7fad61a6a700 1 -- 172.21.15.113:0/1883477563 wait complete. 2024-04-12T05:39:01.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.176+0000 7fad61a6a700 1 Processor -- start 2024-04-12T05:39:01.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.176+0000 7fad61a6a700 1 -- start start 2024-04-12T05:39:01.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.176+0000 7fad61a6a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad5c0fc7f0 0x7fad5c10e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:01.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.176+0000 7fad61a6a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad5c0fd150 0x7fad5c10e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:01.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.176+0000 7fad61a6a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c10eb30 con 0x7fad5c0fd150 2024-04-12T05:39:01.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.176+0000 7fad61a6a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c10ec70 con 0x7fad5c0fc7f0 2024-04-12T05:39:01.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad5affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad5c0fd150 0x7fad5c10e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:01.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad5affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad5c0fd150 0x7fad5c10e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56608/0 (socket says 172.21.15.113:56608) 2024-04-12T05:39:01.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad5affd700 1 -- 172.21.15.113:0/701197124 learned_addr learned my addr 172.21.15.113:0/701197124 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:01.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad5b7fe700 1 --2- 172.21.15.113:0/701197124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad5c0fc7f0 0x7fad5c10e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:01.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad5affd700 1 -- 172.21.15.113:0/701197124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad5c0fc7f0 msgr2=0x7fad5c10e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:01.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad5affd700 1 --2- 172.21.15.113:0/701197124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad5c0fc7f0 0x7fad5c10e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad5affd700 1 -- 172.21.15.113:0/701197124 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c005040 con 0x7fad5c0fd150 2024-04-12T05:39:01.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad5affd700 1 --2- 172.21.15.113:0/701197124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad5c0fd150 0x7fad5c10e530 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7fad5000c790 tx=0x7fad50007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:01.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad58f74700 1 -- 172.21.15.113:0/701197124 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad500106f0 con 0x7fad5c0fd150 2024-04-12T05:39:01.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad61a6a700 1 -- 172.21.15.113:0/701197124 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5c108300 con 0x7fad5c0fd150 2024-04-12T05:39:01.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad58f74700 1 -- 172.21.15.113:0/701197124 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad50010d00 con 0x7fad5c0fd150 2024-04-12T05:39:01.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad58f74700 1 -- 172.21.15.113:0/701197124 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad50012d50 con 0x7fad5c0fd150 2024-04-12T05:39:01.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.177+0000 7fad61a6a700 1 -- 172.21.15.113:0/701197124 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5c1088c0 con 0x7fad5c0fd150 2024-04-12T05:39:01.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.178+0000 7fad61a6a700 1 -- 172.21.15.113:0/701197124 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5c05f590 con 0x7fad5c0fd150 2024-04-12T05:39:01.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.179+0000 7fad58f74700 1 -- 172.21.15.113:0/701197124 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fad50012570 con 0x7fad5c0fd150 2024-04-12T05:39:01.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.179+0000 7fad58f74700 1 --2- 172.21.15.113:0/701197124 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad4406caa0 0x7fad4406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:01.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.179+0000 7fad58f74700 1 -- 172.21.15.113:0/701197124 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fad5008b740 con 0x7fad5c0fd150 2024-04-12T05:39:01.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.179+0000 7fad5b7fe700 1 --2- 172.21.15.113:0/701197124 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad4406caa0 0x7fad4406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:01.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.179+0000 7fad5b7fe700 1 --2- 172.21.15.113:0/701197124 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad4406caa0 0x7fad4406ef60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fad4c004f10 tx=0x7fad4c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:01.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.183+0000 7fad58f74700 1 -- 172.21.15.113:0/701197124 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fad500591f0 con 0x7fad5c0fd150 2024-04-12T05:39:01.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.390+0000 7fad61a6a700 1 -- 172.21.15.113:0/701197124 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fad5c108f60 con 0x7fad4406caa0 2024-04-12T05:39:01.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.391+0000 7fad58f74700 1 -- 172.21.15.113:0/701197124 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fad5c108f60 con 0x7fad4406caa0 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:39:01.390 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:39:01.391 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:39:01.391 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:39:01.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.393+0000 7fad61a6a700 1 -- 172.21.15.113:0/701197124 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad4406caa0 msgr2=0x7fad4406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:01.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.393+0000 7fad61a6a700 1 --2- 172.21.15.113:0/701197124 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad4406caa0 0x7fad4406ef60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fad4c004f10 tx=0x7fad4c00b040).stop 2024-04-12T05:39:01.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.393+0000 7fad61a6a700 1 -- 172.21.15.113:0/701197124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad5c0fd150 msgr2=0x7fad5c10e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:01.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.393+0000 7fad61a6a700 1 --2- 172.21.15.113:0/701197124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad5c0fd150 0x7fad5c10e530 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7fad5000c790 tx=0x7fad50007f50).stop 2024-04-12T05:39:01.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.393+0000 7fad61a6a700 1 -- 172.21.15.113:0/701197124 shutdown_connections 2024-04-12T05:39:01.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.393+0000 7fad61a6a700 1 --2- 172.21.15.113:0/701197124 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fad4406caa0 0x7fad4406ef60 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.393+0000 7fad61a6a700 1 --2- 172.21.15.113:0/701197124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad5c0fc7f0 0x7fad5c10e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.393+0000 7fad61a6a700 1 --2- 172.21.15.113:0/701197124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad5c0fd150 0x7fad5c10e530 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.393+0000 7fad61a6a700 1 -- 172.21.15.113:0/701197124 >> 172.21.15.113:0/701197124 conn(0x7fad5c070590 msgr2=0x7fad5c060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:01.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.393+0000 7fad61a6a700 1 -- 172.21.15.113:0/701197124 shutdown_connections 2024-04-12T05:39:01.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.393+0000 7fad61a6a700 1 -- 172.21.15.113:0/701197124 wait complete. 2024-04-12T05:39:01.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.491+0000 7f9159e33700 1 -- 172.21.15.113:0/4233360291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540fc7f0 msgr2=0x7f91540fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:01.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.491+0000 7f9159e33700 1 --2- 172.21.15.113:0/4233360291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540fc7f0 0x7f91540fcbd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f91440047f0 tx=0x7f9144009cb0).stop 2024-04-12T05:39:01.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.491+0000 7f9159e33700 1 -- 172.21.15.113:0/4233360291 shutdown_connections 2024-04-12T05:39:01.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.491+0000 7f9159e33700 1 --2- 172.21.15.113:0/4233360291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540fd150 0x7f91540f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.491+0000 7f9159e33700 1 --2- 172.21.15.113:0/4233360291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540fc7f0 0x7f91540fcbd0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.491+0000 7f9159e33700 1 -- 172.21.15.113:0/4233360291 >> 172.21.15.113:0/4233360291 conn(0x7f9154070590 msgr2=0x7f91540f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:01.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.491+0000 7f9159e33700 1 -- 172.21.15.113:0/4233360291 shutdown_connections 2024-04-12T05:39:01.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.491+0000 7f9159e33700 1 -- 172.21.15.113:0/4233360291 wait complete. 2024-04-12T05:39:01.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.492+0000 7f9159e33700 1 Processor -- start 2024-04-12T05:39:01.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.492+0000 7f9159e33700 1 -- start start 2024-04-12T05:39:01.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.492+0000 7f9159e33700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540fc7f0 0x7f915410cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:01.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.492+0000 7f9159e33700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540fd150 0x7f915410d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:01.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.492+0000 7f9159e33700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915410d9f0 con 0x7f91540fc7f0 2024-04-12T05:39:01.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.492+0000 7f9159e33700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915410db30 con 0x7f91540fd150 2024-04-12T05:39:01.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.492+0000 7f91537fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540fc7f0 0x7f915410cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.492+0000 7f91537fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540fc7f0 0x7f915410cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56640/0 (socket says 172.21.15.113:56640) 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.492+0000 7f91537fe700 1 -- 172.21.15.113:0/2919455414 learned_addr learned my addr 172.21.15.113:0/2919455414 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.493+0000 7f9152ffd700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540fd150 0x7f915410d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.493+0000 7f91537fe700 1 -- 172.21.15.113:0/2919455414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540fd150 msgr2=0x7f915410d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.493+0000 7f91537fe700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540fd150 0x7f915410d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.493+0000 7f91537fe700 1 -- 172.21.15.113:0/2919455414 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9144005040 con 0x7f91540fc7f0 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.493+0000 7f9152ffd700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540fd150 0x7f915410d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.493+0000 7f91537fe700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540fc7f0 0x7f915410cf00 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f9144005010 tx=0x7f9144009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.493+0000 7f9150ff9700 1 -- 172.21.15.113:0/2919455414 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f914400e030 con 0x7f91540fc7f0 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.493+0000 7f9150ff9700 1 -- 172.21.15.113:0/2919455414 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9144007e90 con 0x7f91540fc7f0 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.493+0000 7f9159e33700 1 -- 172.21.15.113:0/2919455414 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9154107160 con 0x7f91540fc7f0 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.493+0000 7f9150ff9700 1 -- 172.21.15.113:0/2919455414 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9144018070 con 0x7f91540fc7f0 2024-04-12T05:39:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.493+0000 7f9159e33700 1 -- 172.21.15.113:0/2919455414 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91541076c0 con 0x7f91540fc7f0 2024-04-12T05:39:01.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.494+0000 7f9159e33700 1 -- 172.21.15.113:0/2919455414 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915405f590 con 0x7f91540fc7f0 2024-04-12T05:39:01.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.495+0000 7f9150ff9700 1 -- 172.21.15.113:0/2919455414 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f914401d6e0 con 0x7f91540fc7f0 2024-04-12T05:39:01.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.495+0000 7f9150ff9700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913c06ca30 0x7f913c06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:01.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.495+0000 7f9150ff9700 1 -- 172.21.15.113:0/2919455414 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f914408d100 con 0x7f91540fc7f0 2024-04-12T05:39:01.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.495+0000 7f9152ffd700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913c06ca30 0x7f913c06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:01.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.495+0000 7f9152ffd700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913c06ca30 0x7f913c06eef0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f9154108840 tx=0x7f9148009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:01.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.499+0000 7f9150ff9700 1 -- 172.21.15.113:0/2919455414 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f914405a920 con 0x7f91540fc7f0 2024-04-12T05:39:01.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:01 smithi113 ceph-mon[103371]: from='client.35458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:01.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:01 smithi113 ceph-mon[103371]: pgmap v806: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:01.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:01 smithi113 ceph-mon[103371]: from='client.35462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:01.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:01 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1303694806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:01.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:01 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1674755714' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:39:01.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:01 smithi192 ceph-mon[85304]: from='client.35458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:01.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:01 smithi192 ceph-mon[85304]: pgmap v806: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:01.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:01 smithi192 ceph-mon[85304]: from='client.35462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:01.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:01 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1303694806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:01.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:01 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1674755714' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:39:01.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.765+0000 7f9159e33700 1 -- 172.21.15.113:0/2919455414 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9154047540 con 0x7f91540fc7f0 2024-04-12T05:39:01.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.765+0000 7f9150ff9700 1 -- 172.21.15.113:0/2919455414 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f914405a740 con 0x7f91540fc7f0 2024-04-12T05:39:01.764 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:39:01.764 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:39:01.765 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:39:01.766 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:39:01.766 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:39:01.766 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:39:01.766 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:39:01.766 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:39:01.766 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:39:01.766 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:39:01.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.767+0000 7f9159e33700 1 -- 172.21.15.113:0/2919455414 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913c06ca30 msgr2=0x7f913c06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:01.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.767+0000 7f9159e33700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913c06ca30 0x7f913c06eef0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f9154108840 tx=0x7f9148009d60).stop 2024-04-12T05:39:01.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.767+0000 7f9159e33700 1 -- 172.21.15.113:0/2919455414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540fc7f0 msgr2=0x7f915410cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:01.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.767+0000 7f9159e33700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540fc7f0 0x7f915410cf00 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f9144005010 tx=0x7f9144009cb0).stop 2024-04-12T05:39:01.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.768+0000 7f9159e33700 1 -- 172.21.15.113:0/2919455414 shutdown_connections 2024-04-12T05:39:01.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.768+0000 7f9159e33700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f913c06ca30 0x7f913c06eef0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.768+0000 7f9159e33700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91540fc7f0 0x7f915410cf00 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.768+0000 7f9159e33700 1 --2- 172.21.15.113:0/2919455414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91540fd150 0x7f915410d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:01.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.768+0000 7f9159e33700 1 -- 172.21.15.113:0/2919455414 >> 172.21.15.113:0/2919455414 conn(0x7f9154070590 msgr2=0x7f91540fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:01.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.768+0000 7f9159e33700 1 -- 172.21.15.113:0/2919455414 shutdown_connections 2024-04-12T05:39:01.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:01.768+0000 7f9159e33700 1 -- 172.21.15.113:0/2919455414 wait complete. 2024-04-12T05:39:02.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:02 smithi113 ceph-mon[103371]: from='client.35466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:02.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:02 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2919455414' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:39:02.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:02 smithi192 ceph-mon[85304]: from='client.35466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:02.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:02 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2919455414' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:39:03.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:03 smithi113 ceph-mon[103371]: from='client.35478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:03.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:03 smithi113 ceph-mon[103371]: pgmap v807: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:03.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:03 smithi192 ceph-mon[85304]: from='client.35478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:03.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:03 smithi192 ceph-mon[85304]: pgmap v807: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:05.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:05 smithi113 ceph-mon[103371]: pgmap v808: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:05.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:05 smithi192 ceph-mon[85304]: pgmap v808: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:07.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:07 smithi113 ceph-mon[103371]: pgmap v809: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:07.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:07.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:07 smithi192 ceph-mon[85304]: pgmap v809: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:07.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:09.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:09 smithi113 ceph-mon[103371]: pgmap v810: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:09.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:09 smithi192 ceph-mon[85304]: pgmap v810: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:11.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:11 smithi113 ceph-mon[103371]: pgmap v811: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:11.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:11 smithi192 ceph-mon[85304]: pgmap v811: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:13.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:13 smithi113 ceph-mon[103371]: pgmap v812: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:13.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:13 smithi192 ceph-mon[85304]: pgmap v812: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:15.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:15 smithi113 ceph-mon[103371]: pgmap v813: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:15.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:15 smithi192 ceph-mon[85304]: pgmap v813: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:17.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:17 smithi113 ceph-mon[103371]: pgmap v814: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:17.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:17 smithi192 ceph-mon[85304]: pgmap v814: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:19.675 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:19 smithi113 ceph-mon[103371]: pgmap v815: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:19.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:19 smithi192 ceph-mon[85304]: pgmap v815: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:21.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:21 smithi113 ceph-mon[103371]: pgmap v816: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:21.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:21 smithi192 ceph-mon[85304]: pgmap v816: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:22.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:22.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:23.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:23 smithi113 ceph-mon[103371]: pgmap v817: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:23.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:23 smithi192 ceph-mon[85304]: pgmap v817: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:25.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:25 smithi113 ceph-mon[103371]: pgmap v818: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:25.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:25 smithi192 ceph-mon[85304]: pgmap v818: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:27.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:27 smithi113 ceph-mon[103371]: pgmap v819: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:27.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:27 smithi192 ceph-mon[85304]: pgmap v819: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:29.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:29 smithi113 ceph-mon[103371]: pgmap v820: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:29.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:29 smithi192 ceph-mon[85304]: pgmap v820: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:31.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:31 smithi113 ceph-mon[103371]: pgmap v821: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:31.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:31 smithi192 ceph-mon[85304]: pgmap v821: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:31.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.880+0000 7f47359dd700 1 -- 172.21.15.113:0/321247310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47300fda00 msgr2=0x7f47300fdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:31.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.880+0000 7f47359dd700 1 --2- 172.21.15.113:0/321247310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47300fda00 0x7f47300fdde0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f47240047f0 tx=0x7f4724009cb0).stop 2024-04-12T05:39:31.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.880+0000 7f47359dd700 1 -- 172.21.15.113:0/321247310 shutdown_connections 2024-04-12T05:39:31.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.880+0000 7f47359dd700 1 --2- 172.21.15.113:0/321247310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47300fe360 0x7f47300f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:31.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.880+0000 7f47359dd700 1 --2- 172.21.15.113:0/321247310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47300fda00 0x7f47300fdde0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:31.878 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.880+0000 7f47359dd700 1 -- 172.21.15.113:0/321247310 >> 172.21.15.113:0/321247310 conn(0x7f473006cea0 msgr2=0x7f473006d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:31.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.880+0000 7f47359dd700 1 -- 172.21.15.113:0/321247310 shutdown_connections 2024-04-12T05:39:31.879 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.880+0000 7f47359dd700 1 -- 172.21.15.113:0/321247310 wait complete. 2024-04-12T05:39:31.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.881+0000 7f47359dd700 1 Processor -- start 2024-04-12T05:39:31.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.881+0000 7f47359dd700 1 -- start start 2024-04-12T05:39:31.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.881+0000 7f47359dd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47300fda00 0x7f473010df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:31.881 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.881+0000 7f47359dd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47300fe360 0x7f473010e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.881+0000 7f47359dd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473010ea60 con 0x7f47300fda00 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.881+0000 7f47359dd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473010eba0 con 0x7f47300fe360 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.881+0000 7f472effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47300fda00 0x7f473010df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.881+0000 7f472effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47300fda00 0x7f473010df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36954/0 (socket says 172.21.15.113:36954) 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.881+0000 7f472effd700 1 -- 172.21.15.113:0/1954256840 learned_addr learned my addr 172.21.15.113:0/1954256840 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.881+0000 7f472e7fc700 1 --2- 172.21.15.113:0/1954256840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47300fe360 0x7f473010e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.882+0000 7f472effd700 1 -- 172.21.15.113:0/1954256840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47300fe360 msgr2=0x7f473010e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.882+0000 7f472effd700 1 --2- 172.21.15.113:0/1954256840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47300fe360 0x7f473010e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.882+0000 7f472effd700 1 -- 172.21.15.113:0/1954256840 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4724005040 con 0x7f47300fda00 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.882+0000 7f472effd700 1 --2- 172.21.15.113:0/1954256840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47300fda00 0x7f473010df70 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f4724005010 tx=0x7f4724009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.882+0000 7f4717fff700 1 -- 172.21.15.113:0/1954256840 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f472400e030 con 0x7f47300fda00 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.882+0000 7f47359dd700 1 -- 172.21.15.113:0/1954256840 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47301081d0 con 0x7f47300fda00 2024-04-12T05:39:31.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.882+0000 7f4717fff700 1 -- 172.21.15.113:0/1954256840 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4724007e90 con 0x7f47300fda00 2024-04-12T05:39:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.882+0000 7f4717fff700 1 -- 172.21.15.113:0/1954256840 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4724018070 con 0x7f47300fda00 2024-04-12T05:39:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.882+0000 7f47359dd700 1 -- 172.21.15.113:0/1954256840 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4730108730 con 0x7f47300fda00 2024-04-12T05:39:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.883+0000 7f47359dd700 1 -- 172.21.15.113:0/1954256840 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4730047540 con 0x7f47300fda00 2024-04-12T05:39:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.883+0000 7f4717fff700 1 -- 172.21.15.113:0/1954256840 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f472401d670 con 0x7f47300fda00 2024-04-12T05:39:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.884+0000 7f4717fff700 1 --2- 172.21.15.113:0/1954256840 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471806caa0 0x7f471806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.884+0000 7f4717fff700 1 -- 172.21.15.113:0/1954256840 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f472408d0d0 con 0x7f47300fda00 2024-04-12T05:39:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.884+0000 7f472e7fc700 1 --2- 172.21.15.113:0/1954256840 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471806caa0 0x7f471806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:31.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.884+0000 7f472e7fc700 1 --2- 172.21.15.113:0/1954256840 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471806caa0 0x7f471806ef60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f47301098b0 tx=0x7f472000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:31.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:31.889+0000 7f4717fff700 1 -- 172.21.15.113:0/1954256840 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f472405ab80 con 0x7f47300fda00 2024-04-12T05:39:32.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.093+0000 7f47359dd700 1 -- 172.21.15.113:0/1954256840 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f47300ffd80 con 0x7f471806caa0 2024-04-12T05:39:32.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.094+0000 7f4717fff700 1 -- 172.21.15.113:0/1954256840 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f47300ffd80 con 0x7f471806caa0 2024-04-12T05:39:32.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.097+0000 7f47359dd700 1 -- 172.21.15.113:0/1954256840 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471806caa0 msgr2=0x7f471806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.097+0000 7f47359dd700 1 --2- 172.21.15.113:0/1954256840 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471806caa0 0x7f471806ef60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f47301098b0 tx=0x7f472000b040).stop 2024-04-12T05:39:32.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.097+0000 7f47359dd700 1 -- 172.21.15.113:0/1954256840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47300fda00 msgr2=0x7f473010df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.097+0000 7f47359dd700 1 --2- 172.21.15.113:0/1954256840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47300fda00 0x7f473010df70 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f4724005010 tx=0x7f4724009cb0).stop 2024-04-12T05:39:32.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.097+0000 7f47359dd700 1 -- 172.21.15.113:0/1954256840 shutdown_connections 2024-04-12T05:39:32.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.097+0000 7f47359dd700 1 --2- 172.21.15.113:0/1954256840 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f471806caa0 0x7f471806ef60 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.097+0000 7f47359dd700 1 --2- 172.21.15.113:0/1954256840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47300fda00 0x7f473010df70 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.097+0000 7f47359dd700 1 --2- 172.21.15.113:0/1954256840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47300fe360 0x7f473010e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.097+0000 7f47359dd700 1 -- 172.21.15.113:0/1954256840 >> 172.21.15.113:0/1954256840 conn(0x7f473006cea0 msgr2=0x7f47300fbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:32.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.097+0000 7f47359dd700 1 -- 172.21.15.113:0/1954256840 shutdown_connections 2024-04-12T05:39:32.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.097+0000 7f47359dd700 1 -- 172.21.15.113:0/1954256840 wait complete. 2024-04-12T05:39:32.104 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:39:32.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.196+0000 7fcc444f9700 1 -- 172.21.15.113:0/511310923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc3c0fc7f0 msgr2=0x7fcc3c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.196+0000 7fcc444f9700 1 --2- 172.21.15.113:0/511310923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc3c0fc7f0 0x7fcc3c0fcbd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7fcc380047f0 tx=0x7fcc38009cb0).stop 2024-04-12T05:39:32.195 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.196+0000 7fcc444f9700 1 -- 172.21.15.113:0/511310923 shutdown_connections 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.196+0000 7fcc444f9700 1 --2- 172.21.15.113:0/511310923 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc3c0fd150 0x7fcc3c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.196+0000 7fcc444f9700 1 --2- 172.21.15.113:0/511310923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc3c0fc7f0 0x7fcc3c0fcbd0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.196+0000 7fcc444f9700 1 -- 172.21.15.113:0/511310923 >> 172.21.15.113:0/511310923 conn(0x7fcc3c070590 msgr2=0x7fcc3c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.196+0000 7fcc444f9700 1 -- 172.21.15.113:0/511310923 shutdown_connections 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.196+0000 7fcc444f9700 1 -- 172.21.15.113:0/511310923 wait complete. 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.197+0000 7fcc444f9700 1 Processor -- start 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.197+0000 7fcc444f9700 1 -- start start 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.197+0000 7fcc444f9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc3c0fc7f0 0x7fcc3c10dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.197+0000 7fcc444f9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc3c0fd150 0x7fcc3c10e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.197+0000 7fcc444f9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c10eab0 con 0x7fcc3c0fc7f0 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.197+0000 7fcc444f9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c10ebf0 con 0x7fcc3c0fd150 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.197+0000 7fcc42295700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc3c0fc7f0 0x7fcc3c10dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:32.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.197+0000 7fcc42295700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc3c0fc7f0 0x7fcc3c10dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36972/0 (socket says 172.21.15.113:36972) 2024-04-12T05:39:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.197+0000 7fcc42295700 1 -- 172.21.15.113:0/341882853 learned_addr learned my addr 172.21.15.113:0/341882853 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.197+0000 7fcc41a94700 1 --2- 172.21.15.113:0/341882853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc3c0fd150 0x7fcc3c10e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.198+0000 7fcc42295700 1 -- 172.21.15.113:0/341882853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc3c0fd150 msgr2=0x7fcc3c10e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.198+0000 7fcc42295700 1 --2- 172.21.15.113:0/341882853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc3c0fd150 0x7fcc3c10e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.198+0000 7fcc42295700 1 -- 172.21.15.113:0/341882853 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc38005040 con 0x7fcc3c0fc7f0 2024-04-12T05:39:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.198+0000 7fcc42295700 1 --2- 172.21.15.113:0/341882853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc3c0fc7f0 0x7fcc3c10dfc0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7fcc3800b120 tx=0x7fcc38009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.198+0000 7fcc337fe700 1 -- 172.21.15.113:0/341882853 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc3800e030 con 0x7fcc3c0fc7f0 2024-04-12T05:39:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.198+0000 7fcc337fe700 1 -- 172.21.15.113:0/341882853 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc38007e90 con 0x7fcc3c0fc7f0 2024-04-12T05:39:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.198+0000 7fcc337fe700 1 -- 172.21.15.113:0/341882853 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc38018070 con 0x7fcc3c0fc7f0 2024-04-12T05:39:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.198+0000 7fcc444f9700 1 -- 172.21.15.113:0/341882853 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3c108220 con 0x7fcc3c0fc7f0 2024-04-12T05:39:32.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.198+0000 7fcc444f9700 1 -- 172.21.15.113:0/341882853 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3c108780 con 0x7fcc3c0fc7f0 2024-04-12T05:39:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.199+0000 7fcc444f9700 1 -- 172.21.15.113:0/341882853 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3c05f590 con 0x7fcc3c0fc7f0 2024-04-12T05:39:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.199+0000 7fcc337fe700 1 -- 172.21.15.113:0/341882853 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcc3801d420 con 0x7fcc3c0fc7f0 2024-04-12T05:39:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.200+0000 7fcc337fe700 1 --2- 172.21.15.113:0/341882853 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc2806cb70 0x7fcc2806f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:32.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.200+0000 7fcc337fe700 1 -- 172.21.15.113:0/341882853 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fcc3808d0b0 con 0x7fcc3c0fc7f0 2024-04-12T05:39:32.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.200+0000 7fcc41a94700 1 --2- 172.21.15.113:0/341882853 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc2806cb70 0x7fcc2806f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:32.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.200+0000 7fcc41a94700 1 --2- 172.21.15.113:0/341882853 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc2806cb70 0x7fcc2806f030 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fcc3c109900 tx=0x7fcc2c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:32.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.204+0000 7fcc337fe700 1 -- 172.21.15.113:0/341882853 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fcc3805abe0 con 0x7fcc3c0fc7f0 2024-04-12T05:39:32.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.420+0000 7fcc444f9700 1 -- 172.21.15.113:0/341882853 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc3c108e10 con 0x7fcc2806cb70 2024-04-12T05:39:32.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.421+0000 7fcc337fe700 1 -- 172.21.15.113:0/341882853 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcc3c108e10 con 0x7fcc2806cb70 2024-04-12T05:39:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.423+0000 7fcc444f9700 1 -- 172.21.15.113:0/341882853 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc2806cb70 msgr2=0x7fcc2806f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.423+0000 7fcc444f9700 1 --2- 172.21.15.113:0/341882853 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc2806cb70 0x7fcc2806f030 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fcc3c109900 tx=0x7fcc2c009040).stop 2024-04-12T05:39:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.423+0000 7fcc444f9700 1 -- 172.21.15.113:0/341882853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc3c0fc7f0 msgr2=0x7fcc3c10dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.423+0000 7fcc444f9700 1 --2- 172.21.15.113:0/341882853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc3c0fc7f0 0x7fcc3c10dfc0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7fcc3800b120 tx=0x7fcc38009cb0).stop 2024-04-12T05:39:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.423+0000 7fcc444f9700 1 -- 172.21.15.113:0/341882853 shutdown_connections 2024-04-12T05:39:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.423+0000 7fcc444f9700 1 --2- 172.21.15.113:0/341882853 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcc2806cb70 0x7fcc2806f030 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.423+0000 7fcc444f9700 1 --2- 172.21.15.113:0/341882853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc3c0fc7f0 0x7fcc3c10dfc0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.423+0000 7fcc444f9700 1 --2- 172.21.15.113:0/341882853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc3c0fd150 0x7fcc3c10e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.423+0000 7fcc444f9700 1 -- 172.21.15.113:0/341882853 >> 172.21.15.113:0/341882853 conn(0x7fcc3c070590 msgr2=0x7fcc3c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.423+0000 7fcc444f9700 1 -- 172.21.15.113:0/341882853 shutdown_connections 2024-04-12T05:39:32.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.423+0000 7fcc444f9700 1 -- 172.21.15.113:0/341882853 wait complete. 2024-04-12T05:39:32.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.521+0000 7f777c5bb700 1 -- 172.21.15.113:0/3165290477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77740fc7f0 msgr2=0x7f77740fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.521+0000 7f777c5bb700 1 --2- 172.21.15.113:0/3165290477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77740fc7f0 0x7f77740fcbd0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f77680047f0 tx=0x7f7768009cb0).stop 2024-04-12T05:39:32.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.521+0000 7f777c5bb700 1 -- 172.21.15.113:0/3165290477 shutdown_connections 2024-04-12T05:39:32.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.521+0000 7f777c5bb700 1 --2- 172.21.15.113:0/3165290477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77740fd150 0x7f77740f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.521+0000 7f777c5bb700 1 --2- 172.21.15.113:0/3165290477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77740fc7f0 0x7f77740fcbd0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.520 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.521+0000 7f777c5bb700 1 -- 172.21.15.113:0/3165290477 >> 172.21.15.113:0/3165290477 conn(0x7f7774070590 msgr2=0x7f77740f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.521+0000 7f777c5bb700 1 -- 172.21.15.113:0/3165290477 shutdown_connections 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.521+0000 7f777c5bb700 1 -- 172.21.15.113:0/3165290477 wait complete. 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777c5bb700 1 Processor -- start 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777c5bb700 1 -- start start 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777c5bb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77740fc7f0 0x7f777410e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777c5bb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77740fd150 0x7f777410e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777c5bb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777410eb30 con 0x7f77740fc7f0 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777c5bb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777410ec70 con 0x7f77740fd150 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777a357700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77740fc7f0 0x7f777410e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777a357700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77740fc7f0 0x7f777410e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36992/0 (socket says 172.21.15.113:36992) 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777a357700 1 -- 172.21.15.113:0/961298396 learned_addr learned my addr 172.21.15.113:0/961298396 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f7779b56700 1 --2- 172.21.15.113:0/961298396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77740fd150 0x7f777410e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:32.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777a357700 1 -- 172.21.15.113:0/961298396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77740fd150 msgr2=0x7f777410e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777a357700 1 --2- 172.21.15.113:0/961298396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77740fd150 0x7f777410e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777a357700 1 -- 172.21.15.113:0/961298396 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7768005040 con 0x7f77740fc7f0 2024-04-12T05:39:32.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.522+0000 7f777a357700 1 --2- 172.21.15.113:0/961298396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77740fc7f0 0x7f777410e040 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f776800b120 tx=0x7f7768009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:32.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.523+0000 7f77677fe700 1 -- 172.21.15.113:0/961298396 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f776800e030 con 0x7f77740fc7f0 2024-04-12T05:39:32.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.523+0000 7f77677fe700 1 -- 172.21.15.113:0/961298396 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7768007e90 con 0x7f77740fc7f0 2024-04-12T05:39:32.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.523+0000 7f777c5bb700 1 -- 172.21.15.113:0/961298396 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77741082a0 con 0x7f77740fc7f0 2024-04-12T05:39:32.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.523+0000 7f77677fe700 1 -- 172.21.15.113:0/961298396 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7768018070 con 0x7f77740fc7f0 2024-04-12T05:39:32.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.523+0000 7f777c5bb700 1 -- 172.21.15.113:0/961298396 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7774108800 con 0x7f77740fc7f0 2024-04-12T05:39:32.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.524+0000 7f777c5bb700 1 -- 172.21.15.113:0/961298396 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777405f590 con 0x7f77740fc7f0 2024-04-12T05:39:32.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.524+0000 7f77677fe700 1 -- 172.21.15.113:0/961298396 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f776801d420 con 0x7f77740fc7f0 2024-04-12T05:39:32.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.525+0000 7f77677fe700 1 --2- 172.21.15.113:0/961298396 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f776006caa0 0x7f776006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:32.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.525+0000 7f77677fe700 1 -- 172.21.15.113:0/961298396 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f776808d630 con 0x7f77740fc7f0 2024-04-12T05:39:32.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.525+0000 7f7779b56700 1 --2- 172.21.15.113:0/961298396 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f776006caa0 0x7f776006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:32.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.525+0000 7f7779b56700 1 --2- 172.21.15.113:0/961298396 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f776006caa0 0x7f776006ef60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f7774109980 tx=0x7f7770009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:32.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.529+0000 7f77677fe700 1 -- 172.21.15.113:0/961298396 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f776805b0e0 con 0x7f77740fc7f0 2024-04-12T05:39:32.725 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.727+0000 7f777c5bb700 1 -- 172.21.15.113:0/961298396 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7774108e40 con 0x7f776006caa0 2024-04-12T05:39:32.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.737+0000 7f77677fe700 1 -- 172.21.15.113:0/961298396 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7774108e40 con 0x7f776006caa0 2024-04-12T05:39:32.734 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:39:32.734 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (27m) 4m ago 38m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (24m) 4m ago 38m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (24m) 4m ago 38m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (27m) 4m ago 38m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (20m) 4m ago 33m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (20m) 4m ago 33m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (20m) 4m ago 33m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (4m) 4m ago 33m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (28m) 4m ago 39m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (28m) 4m ago 37m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (25m) 4m ago 40m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (25m) 4m ago 37m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (28m) 4m ago 38m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (27m) 4m ago 37m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:39:32.735 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (24m) 4m ago 37m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:39:32.736 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (23m) 4m ago 36m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:39:32.736 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (23m) 4m ago 36m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:39:32.736 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (22m) 4m ago 35m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:39:32.736 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (22m) 4m ago 35m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:39:32.736 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (20m) 4m ago 34m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:39:32.736 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (27m) 4m ago 37m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:39:32.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.739+0000 7f777c5bb700 1 -- 172.21.15.113:0/961298396 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f776006caa0 msgr2=0x7f776006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.739+0000 7f777c5bb700 1 --2- 172.21.15.113:0/961298396 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f776006caa0 0x7f776006ef60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f7774109980 tx=0x7f7770009040).stop 2024-04-12T05:39:32.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.739+0000 7f777c5bb700 1 -- 172.21.15.113:0/961298396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77740fc7f0 msgr2=0x7f777410e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.739+0000 7f777c5bb700 1 --2- 172.21.15.113:0/961298396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77740fc7f0 0x7f777410e040 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f776800b120 tx=0x7f7768009cb0).stop 2024-04-12T05:39:32.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.739+0000 7f777c5bb700 1 -- 172.21.15.113:0/961298396 shutdown_connections 2024-04-12T05:39:32.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.739+0000 7f777c5bb700 1 --2- 172.21.15.113:0/961298396 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f776006caa0 0x7f776006ef60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.739+0000 7f777c5bb700 1 --2- 172.21.15.113:0/961298396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77740fc7f0 0x7f777410e040 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.739+0000 7f777c5bb700 1 --2- 172.21.15.113:0/961298396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77740fd150 0x7f777410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.739+0000 7f777c5bb700 1 -- 172.21.15.113:0/961298396 >> 172.21.15.113:0/961298396 conn(0x7f7774070590 msgr2=0x7f7774060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:32.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.739+0000 7f777c5bb700 1 -- 172.21.15.113:0/961298396 shutdown_connections 2024-04-12T05:39:32.737 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.739+0000 7f777c5bb700 1 -- 172.21.15.113:0/961298396 wait complete. 2024-04-12T05:39:32.842 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.843+0000 7f30d15cd700 1 -- 172.21.15.113:0/3301786014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30cc0fe220 msgr2=0x7f30cc0f7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.843+0000 7f30d15cd700 1 --2- 172.21.15.113:0/3301786014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30cc0fe220 0x7f30cc0f7a00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f30b40047f0 tx=0x7f30b4009cb0).stop 2024-04-12T05:39:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.843+0000 7f30d15cd700 1 -- 172.21.15.113:0/3301786014 shutdown_connections 2024-04-12T05:39:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.843+0000 7f30d15cd700 1 --2- 172.21.15.113:0/3301786014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30cc0fe220 0x7f30cc0f7a00 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.843+0000 7f30d15cd700 1 --2- 172.21.15.113:0/3301786014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30cc0fd8c0 0x7f30cc0fdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.843+0000 7f30d15cd700 1 -- 172.21.15.113:0/3301786014 >> 172.21.15.113:0/3301786014 conn(0x7f30cc0f34a0 msgr2=0x7f30cc0f58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.843+0000 7f30d15cd700 1 -- 172.21.15.113:0/3301786014 shutdown_connections 2024-04-12T05:39:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.843+0000 7f30d15cd700 1 -- 172.21.15.113:0/3301786014 wait complete. 2024-04-12T05:39:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30d15cd700 1 Processor -- start 2024-04-12T05:39:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30d15cd700 1 -- start start 2024-04-12T05:39:32.843 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30d15cd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30cc0fd8c0 0x7f30cc06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:32.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30d15cd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30cc0fe220 0x7f30cc06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:32.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30d15cd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc06bab0 con 0x7f30cc0fe220 2024-04-12T05:39:32.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30d15cd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc06bbf0 con 0x7f30cc0fd8c0 2024-04-12T05:39:32.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30ca7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30cc0fe220 0x7f30cc06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:32.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30ca7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30cc0fe220 0x7f30cc06b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37010/0 (socket says 172.21.15.113:37010) 2024-04-12T05:39:32.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30ca7fc700 1 -- 172.21.15.113:0/2945212122 learned_addr learned my addr 172.21.15.113:0/2945212122 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:32.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30caffd700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30cc0fd8c0 0x7f30cc06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:32.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30ca7fc700 1 -- 172.21.15.113:0/2945212122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30cc0fd8c0 msgr2=0x7f30cc06b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:32.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30ca7fc700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30cc0fd8c0 0x7f30cc06b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:32.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.844+0000 7f30ca7fc700 1 -- 172.21.15.113:0/2945212122 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b4005040 con 0x7f30cc0fe220 2024-04-12T05:39:32.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.845+0000 7f30caffd700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30cc0fd8c0 0x7f30cc06b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:39:32.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.845+0000 7f30ca7fc700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30cc0fe220 0x7f30cc06b5c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f30b40099f0 tx=0x7f30b4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:32.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.845+0000 7f30c3fff700 1 -- 172.21.15.113:0/2945212122 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30b400e030 con 0x7f30cc0fe220 2024-04-12T05:39:32.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.845+0000 7f30d15cd700 1 -- 172.21.15.113:0/2945212122 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30cc06be40 con 0x7f30cc0fe220 2024-04-12T05:39:32.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.845+0000 7f30c3fff700 1 -- 172.21.15.113:0/2945212122 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30b4007e90 con 0x7f30cc0fe220 2024-04-12T05:39:32.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.845+0000 7f30c3fff700 1 -- 172.21.15.113:0/2945212122 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30b4018070 con 0x7f30cc0fe220 2024-04-12T05:39:32.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.845+0000 7f30d15cd700 1 -- 172.21.15.113:0/2945212122 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30cc114680 con 0x7f30cc0fe220 2024-04-12T05:39:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.846+0000 7f30d15cd700 1 -- 172.21.15.113:0/2945212122 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30cc060390 con 0x7f30cc0fe220 2024-04-12T05:39:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.846+0000 7f30c3fff700 1 -- 172.21.15.113:0/2945212122 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f30b4009590 con 0x7f30cc0fe220 2024-04-12T05:39:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.847+0000 7f30c3fff700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b806ca30 0x7f30b806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.847+0000 7f30c3fff700 1 -- 172.21.15.113:0/2945212122 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f30b408d0f0 con 0x7f30cc0fe220 2024-04-12T05:39:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.847+0000 7f30caffd700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b806ca30 0x7f30b806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:32.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.847+0000 7f30caffd700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b806ca30 0x7f30b806eef0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f30bc004ca0 tx=0x7f30bc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:32.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:32.851+0000 7f30c3fff700 1 -- 172.21.15.113:0/2945212122 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f30b4057110 con 0x7f30cc0fe220 2024-04-12T05:39:33.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.121+0000 7f30d15cd700 1 -- 172.21.15.113:0/2945212122 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f30cc047540 con 0x7f30cc0fe220 2024-04-12T05:39:33.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.121+0000 7f30c3fff700 1 -- 172.21.15.113:0/2945212122 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f30b405a730 con 0x7f30cc0fe220 2024-04-12T05:39:33.119 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:39:33.119 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:39:33.119 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:39:33.120 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:39:33.120 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:39:33.120 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:39:33.120 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:39:33.120 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:39:33.120 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:39:33.120 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:39:33.120 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:39:33.120 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:39:33.121 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:39:33.121 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:39:33.121 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:39:33.121 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:39:33.121 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:39:33.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.124+0000 7f30d15cd700 1 -- 172.21.15.113:0/2945212122 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b806ca30 msgr2=0x7f30b806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.124+0000 7f30d15cd700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b806ca30 0x7f30b806eef0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f30bc004ca0 tx=0x7f30bc00b040).stop 2024-04-12T05:39:33.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.124+0000 7f30d15cd700 1 -- 172.21.15.113:0/2945212122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30cc0fe220 msgr2=0x7f30cc06b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.124+0000 7f30d15cd700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30cc0fe220 0x7f30cc06b5c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f30b40099f0 tx=0x7f30b4009cb0).stop 2024-04-12T05:39:33.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.124+0000 7f30d15cd700 1 -- 172.21.15.113:0/2945212122 shutdown_connections 2024-04-12T05:39:33.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.124+0000 7f30d15cd700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b806ca30 0x7f30b806eef0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.124+0000 7f30d15cd700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30cc0fd8c0 0x7f30cc06b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.122 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.124+0000 7f30d15cd700 1 --2- 172.21.15.113:0/2945212122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30cc0fe220 0x7f30cc06b5c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.124+0000 7f30d15cd700 1 -- 172.21.15.113:0/2945212122 >> 172.21.15.113:0/2945212122 conn(0x7f30cc0f34a0 msgr2=0x7f30cc0fbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:33.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.124+0000 7f30d15cd700 1 -- 172.21.15.113:0/2945212122 shutdown_connections 2024-04-12T05:39:33.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.124+0000 7f30d15cd700 1 -- 172.21.15.113:0/2945212122 wait complete. 2024-04-12T05:39:33.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.221+0000 7fcff5695700 1 -- 172.21.15.113:0/2567478891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcff00f88c0 msgr2=0x7fcff00f8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.221+0000 7fcff5695700 1 --2- 172.21.15.113:0/2567478891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcff00f88c0 0x7fcff00f8ca0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7fcfe00047f0 tx=0x7fcfe0009cb0).stop 2024-04-12T05:39:33.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.221+0000 7fcff5695700 1 -- 172.21.15.113:0/2567478891 shutdown_connections 2024-04-12T05:39:33.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.221+0000 7fcff5695700 1 --2- 172.21.15.113:0/2567478891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff00f9220 0x7fcff0101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:39:33.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.221+0000 7fcff5695700 1 --2- 172.21.15.113:0/2567478891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcff00f88c0 0x7fcff00f8ca0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.221+0000 7fcff5695700 1 -- 172.21.15.113:0/2567478891 >> 172.21.15.113:0/2567478891 conn(0x7fcff00f2430 msgr2=0x7fcff00f4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:33.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.221+0000 7fcff5695700 1 -- 172.21.15.113:0/2567478891 shutdown_connections 2024-04-12T05:39:33.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.221+0000 7fcff5695700 1 -- 172.21.15.113:0/2567478891 wait complete. 2024-04-12T05:39:33.220 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.222+0000 7fcff5695700 1 Processor -- start 2024-04-12T05:39:33.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.222+0000 7fcff5695700 1 -- start start 2024-04-12T05:39:33.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.222+0000 7fcff5695700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff00f88c0 0x7fcff010e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:33.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.222+0000 7fcff5695700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcff00f9220 0x7fcff010e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:33.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.222+0000 7fcff5695700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff010eca0 con 0x7fcff00f9220 2024-04-12T05:39:33.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.222+0000 7fcff5695700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff01082c0 con 0x7fcff00f88c0 2024-04-12T05:39:33.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.222+0000 7fcfee7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcff00f9220 0x7fcff010e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.222+0000 7fcfee7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcff00f9220 0x7fcff010e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37022/0 (socket says 172.21.15.113:37022) 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.222+0000 7fcfee7fc700 1 -- 172.21.15.113:0/2735525293 learned_addr learned my addr 172.21.15.113:0/2735525293 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.223+0000 7fcfeeffd700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff00f88c0 0x7fcff010e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.223+0000 7fcfee7fc700 1 -- 172.21.15.113:0/2735525293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff00f88c0 msgr2=0x7fcff010e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.223+0000 7fcfee7fc700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff00f88c0 0x7fcff010e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.223+0000 7fcfee7fc700 1 -- 172.21.15.113:0/2735525293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe0005040 con 0x7fcff00f9220 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.223+0000 7fcfeeffd700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff00f88c0 0x7fcff010e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.223+0000 7fcfee7fc700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcff00f9220 0x7fcff010e610 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7fcfe400b4b0 tx=0x7fcfe40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.223+0000 7fcfd7fff700 1 -- 172.21.15.113:0/2735525293 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe4002500 con 0x7fcff00f9220 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.223+0000 7fcfd7fff700 1 -- 172.21.15.113:0/2735525293 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfe4002b10 con 0x7fcff00f9220 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.223+0000 7fcff5695700 1 -- 172.21.15.113:0/2735525293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff0108570 con 0x7fcff00f9220 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.223+0000 7fcfd7fff700 1 -- 172.21.15.113:0/2735525293 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfe4012cf0 con 0x7fcff00f9220 2024-04-12T05:39:33.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.223+0000 7fcff5695700 1 -- 172.21.15.113:0/2735525293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff0108ab0 con 0x7fcff00f9220 2024-04-12T05:39:33.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.224+0000 7fcff5695700 1 -- 172.21.15.113:0/2735525293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff0060330 con 0x7fcff00f9220 2024-04-12T05:39:33.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.224+0000 7fcfd7fff700 1 -- 172.21.15.113:0/2735525293 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcfe4002660 con 0x7fcff00f9220 2024-04-12T05:39:33.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.225+0000 7fcfd7fff700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcfd806caa0 0x7fcfd806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:33.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.225+0000 7fcfd7fff700 1 -- 172.21.15.113:0/2735525293 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fcfe408bf30 con 0x7fcff00f9220 2024-04-12T05:39:33.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.225+0000 7fcfeeffd700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcfd806caa0 0x7fcfd806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:33.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.225+0000 7fcfeeffd700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcfd806caa0 0x7fcfd806ef60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fcfe00047c0 tx=0x7fcfe000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:33.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.229+0000 7fcfd7fff700 1 -- 172.21.15.113:0/2735525293 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fcfe40bf2c0 con 0x7fcff00f9220 2024-04-12T05:39:33.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.456+0000 7fcff5695700 1 -- 172.21.15.113:0/2735525293 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcff01097a0 con 0x7fcff00f9220 2024-04-12T05:39:33.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.457+0000 7fcfd7fff700 1 -- 172.21.15.113:0/2735525293 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fcfe40599e0 con 0x7fcff00f9220 2024-04-12T05:39:33.454 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:39:33.454 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:39:33.454 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:39:33.454 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:39:33.455 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:39:33.456 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:39:33.457 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:39:33.457 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:39:33.457 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:39:33.457 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:39:33.457 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:39:33.457 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:39:33.457 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:39:33.457 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:39:33.457 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:39:33.457 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:39:33.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.459+0000 7fcff5695700 1 -- 172.21.15.113:0/2735525293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcfd806caa0 msgr2=0x7fcfd806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.459+0000 7fcff5695700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcfd806caa0 0x7fcfd806ef60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fcfe00047c0 tx=0x7fcfe000b040).stop 2024-04-12T05:39:33.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.459+0000 7fcff5695700 1 -- 172.21.15.113:0/2735525293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcff00f9220 msgr2=0x7fcff010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.459+0000 7fcff5695700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcff00f9220 0x7fcff010e610 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7fcfe400b4b0 tx=0x7fcfe40090f0).stop 2024-04-12T05:39:33.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.459+0000 7fcff5695700 1 -- 172.21.15.113:0/2735525293 shutdown_connections 2024-04-12T05:39:33.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.459+0000 7fcff5695700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcfd806caa0 0x7fcfd806ef60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.459+0000 7fcff5695700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcff00f88c0 0x7fcff010e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.459+0000 7fcff5695700 1 --2- 172.21.15.113:0/2735525293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcff00f9220 0x7fcff010e610 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.459+0000 7fcff5695700 1 -- 172.21.15.113:0/2735525293 >> 172.21.15.113:0/2735525293 conn(0x7fcff00f2430 msgr2=0x7fcff00fc170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:33.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.459+0000 7fcff5695700 1 -- 172.21.15.113:0/2735525293 shutdown_connections 2024-04-12T05:39:33.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.459+0000 7fcff5695700 1 -- 172.21.15.113:0/2735525293 wait complete. 2024-04-12T05:39:33.459 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:39:33.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.557+0000 7fab15d14700 1 -- 172.21.15.113:0/1961757440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab100fd8c0 msgr2=0x7fab100fdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.557+0000 7fab15d14700 1 --2- 172.21.15.113:0/1961757440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab100fd8c0 0x7fab100fdca0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fab000047f0 tx=0x7fab00009cb0).stop 2024-04-12T05:39:33.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.557+0000 7fab15d14700 1 -- 172.21.15.113:0/1961757440 shutdown_connections 2024-04-12T05:39:33.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.557+0000 7fab15d14700 1 --2- 172.21.15.113:0/1961757440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab100fe220 0x7fab100f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.557+0000 7fab15d14700 1 --2- 172.21.15.113:0/1961757440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab100fd8c0 0x7fab100fdca0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.557+0000 7fab15d14700 1 -- 172.21.15.113:0/1961757440 >> 172.21.15.113:0/1961757440 conn(0x7fab100f34a0 msgr2=0x7fab100f58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:33.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.557+0000 7fab15d14700 1 -- 172.21.15.113:0/1961757440 shutdown_connections 2024-04-12T05:39:33.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.557+0000 7fab15d14700 1 -- 172.21.15.113:0/1961757440 wait complete. 2024-04-12T05:39:33.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.558+0000 7fab15d14700 1 Processor -- start 2024-04-12T05:39:33.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.558+0000 7fab15d14700 1 -- start start 2024-04-12T05:39:33.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.558+0000 7fab15d14700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab100fd8c0 0x7fab1010df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:33.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.558+0000 7fab15d14700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab100fe220 0x7fab1010e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:33.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.558+0000 7fab15d14700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1010ea70 con 0x7fab100fd8c0 2024-04-12T05:39:33.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.558+0000 7fab15d14700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1010ebb0 con 0x7fab100fe220 2024-04-12T05:39:33.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab0f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab100fd8c0 0x7fab1010df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:33.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab0f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab100fd8c0 0x7fab1010df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37052/0 (socket says 172.21.15.113:37052) 2024-04-12T05:39:33.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab0f7fe700 1 -- 172.21.15.113:0/693267841 learned_addr learned my addr 172.21.15.113:0/693267841 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:33.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab0effd700 1 --2- 172.21.15.113:0/693267841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab100fe220 0x7fab1010e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:33.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab0f7fe700 1 -- 172.21.15.113:0/693267841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab100fe220 msgr2=0x7fab1010e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab0f7fe700 1 --2- 172.21.15.113:0/693267841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab100fe220 0x7fab1010e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab0f7fe700 1 -- 172.21.15.113:0/693267841 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab00005040 con 0x7fab100fd8c0 2024-04-12T05:39:33.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab0f7fe700 1 --2- 172.21.15.113:0/693267841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab100fd8c0 0x7fab1010df80 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7fab00005010 tx=0x7fab00009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:33.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab0cff9700 1 -- 172.21.15.113:0/693267841 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab0000e030 con 0x7fab100fd8c0 2024-04-12T05:39:33.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab15d14700 1 -- 172.21.15.113:0/693267841 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab10108250 con 0x7fab100fd8c0 2024-04-12T05:39:33.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab0cff9700 1 -- 172.21.15.113:0/693267841 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab00007e90 con 0x7fab100fd8c0 2024-04-12T05:39:33.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab0cff9700 1 -- 172.21.15.113:0/693267841 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab00018070 con 0x7fab100fd8c0 2024-04-12T05:39:33.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.559+0000 7fab15d14700 1 -- 172.21.15.113:0/693267841 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab101087b0 con 0x7fab100fd8c0 2024-04-12T05:39:33.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.560+0000 7fab15d14700 1 -- 172.21.15.113:0/693267841 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab10060390 con 0x7fab100fd8c0 2024-04-12T05:39:33.559 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.561+0000 7fab0cff9700 1 -- 172.21.15.113:0/693267841 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fab0001d670 con 0x7fab100fd8c0 2024-04-12T05:39:33.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.561+0000 7fab0cff9700 1 --2- 172.21.15.113:0/693267841 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faaf806caa0 0x7faaf806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:33.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.561+0000 7fab0cff9700 1 -- 172.21.15.113:0/693267841 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fab0008d150 con 0x7fab100fd8c0 2024-04-12T05:39:33.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.561+0000 7fab0effd700 1 --2- 172.21.15.113:0/693267841 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faaf806caa0 0x7faaf806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:33.560 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.561+0000 7fab0effd700 1 --2- 172.21.15.113:0/693267841 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faaf806caa0 0x7faaf806ef60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fab101098c0 tx=0x7fab0400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:33.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.565+0000 7fab0cff9700 1 -- 172.21.15.113:0/693267841 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fab0005ac00 con 0x7fab100fd8c0 2024-04-12T05:39:33.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:33 smithi113 ceph-mon[103371]: pgmap v822: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:33.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:33 smithi113 ceph-mon[103371]: from='client.35486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:33.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:33 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2945212122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:33.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:33 smithi192 ceph-mon[85304]: pgmap v822: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:33.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:33 smithi192 ceph-mon[85304]: from='client.35486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:33.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:33 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2945212122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:39:33.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.769+0000 7fab15d14700 1 -- 172.21.15.113:0/693267841 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fab10108a90 con 0x7faaf806caa0 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.770+0000 7fab0cff9700 1 -- 172.21.15.113:0/693267841 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fab10108a90 con 0x7faaf806caa0 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:39:33.768 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:39:33.769 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:39:33.769 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:39:33.769 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:39:33.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.772+0000 7fab15d14700 1 -- 172.21.15.113:0/693267841 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faaf806caa0 msgr2=0x7faaf806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.772+0000 7fab15d14700 1 --2- 172.21.15.113:0/693267841 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faaf806caa0 0x7faaf806ef60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fab101098c0 tx=0x7fab0400b040).stop 2024-04-12T05:39:33.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.772+0000 7fab15d14700 1 -- 172.21.15.113:0/693267841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab100fd8c0 msgr2=0x7fab1010df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.772+0000 7fab15d14700 1 --2- 172.21.15.113:0/693267841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab100fd8c0 0x7fab1010df80 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7fab00005010 tx=0x7fab00009cb0).stop 2024-04-12T05:39:33.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.772+0000 7fab15d14700 1 -- 172.21.15.113:0/693267841 shutdown_connections 2024-04-12T05:39:33.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.772+0000 7fab15d14700 1 --2- 172.21.15.113:0/693267841 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faaf806caa0 0x7faaf806ef60 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.772+0000 7fab15d14700 1 --2- 172.21.15.113:0/693267841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab100fd8c0 0x7fab1010df80 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.772+0000 7fab15d14700 1 --2- 172.21.15.113:0/693267841 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab100fe220 0x7fab1010e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.772+0000 7fab15d14700 1 -- 172.21.15.113:0/693267841 >> 172.21.15.113:0/693267841 conn(0x7fab100f34a0 msgr2=0x7fab100fbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:33.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.773+0000 7fab15d14700 1 -- 172.21.15.113:0/693267841 shutdown_connections 2024-04-12T05:39:33.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.773+0000 7fab15d14700 1 -- 172.21.15.113:0/693267841 wait complete. 2024-04-12T05:39:33.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.870+0000 7f27c59fd700 1 -- 172.21.15.113:0/2400221491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27c00fd150 msgr2=0x7f27c00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.870+0000 7f27c59fd700 1 --2- 172.21.15.113:0/2400221491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27c00fd150 0x7f27c00f6930 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7f27b40047f0 tx=0x7f27b4009cb0).stop 2024-04-12T05:39:33.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.871+0000 7f27c59fd700 1 -- 172.21.15.113:0/2400221491 shutdown_connections 2024-04-12T05:39:33.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.871+0000 7f27c59fd700 1 --2- 172.21.15.113:0/2400221491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27c00fd150 0x7f27c00f6930 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.871+0000 7f27c59fd700 1 --2- 172.21.15.113:0/2400221491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c00fc7f0 0x7f27c00fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.871+0000 7f27c59fd700 1 -- 172.21.15.113:0/2400221491 >> 172.21.15.113:0/2400221491 conn(0x7f27c0070590 msgr2=0x7f27c00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:33.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.871+0000 7f27c59fd700 1 -- 172.21.15.113:0/2400221491 shutdown_connections 2024-04-12T05:39:33.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.871+0000 7f27c59fd700 1 -- 172.21.15.113:0/2400221491 wait complete. 2024-04-12T05:39:33.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.871+0000 7f27c59fd700 1 Processor -- start 2024-04-12T05:39:33.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.871+0000 7f27c59fd700 1 -- start start 2024-04-12T05:39:33.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27c59fd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c00fc7f0 0x7f27c010ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:33.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27c59fd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27c00fd150 0x7f27c010d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:33.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27c59fd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c010d9c0 con 0x7f27c00fd150 2024-04-12T05:39:33.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27c59fd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c010db00 con 0x7f27c00fc7f0 2024-04-12T05:39:33.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27be7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27c00fd150 0x7f27c010d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:33.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27be7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27c00fd150 0x7f27c010d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:37068/0 (socket says 172.21.15.113:37068) 2024-04-12T05:39:33.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27be7fc700 1 -- 172.21.15.113:0/1101371595 learned_addr learned my addr 172.21.15.113:0/1101371595 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:39:33.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27beffd700 1 --2- 172.21.15.113:0/1101371595 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c00fc7f0 0x7f27c010ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:33.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27be7fc700 1 -- 172.21.15.113:0/1101371595 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c00fc7f0 msgr2=0x7f27c010ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:33.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27be7fc700 1 --2- 172.21.15.113:0/1101371595 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c00fc7f0 0x7f27c010ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:33.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27be7fc700 1 -- 172.21.15.113:0/1101371595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b4005040 con 0x7f27c00fd150 2024-04-12T05:39:33.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27be7fc700 1 --2- 172.21.15.113:0/1101371595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27c00fd150 0x7f27c010d3c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f27b40099f0 tx=0x7f27b4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:33.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27a7fff700 1 -- 172.21.15.113:0/1101371595 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27b400e030 con 0x7f27c00fd150 2024-04-12T05:39:33.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27a7fff700 1 -- 172.21.15.113:0/1101371595 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27b4007e90 con 0x7f27c00fd150 2024-04-12T05:39:33.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27a7fff700 1 -- 172.21.15.113:0/1101371595 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27b4018070 con 0x7f27c00fd150 2024-04-12T05:39:33.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27c59fd700 1 -- 172.21.15.113:0/1101371595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c0107130 con 0x7f27c00fd150 2024-04-12T05:39:33.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.872+0000 7f27c59fd700 1 -- 172.21.15.113:0/1101371595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c0107690 con 0x7f27c00fd150 2024-04-12T05:39:33.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.873+0000 7f27c59fd700 1 -- 172.21.15.113:0/1101371595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c005f590 con 0x7f27c00fd150 2024-04-12T05:39:33.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.874+0000 7f27a7fff700 1 -- 172.21.15.113:0/1101371595 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f27b4009590 con 0x7f27c00fd150 2024-04-12T05:39:33.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.875+0000 7f27a7fff700 1 --2- 172.21.15.113:0/1101371595 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27a806caa0 0x7f27a806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:39:33.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.875+0000 7f27a7fff700 1 -- 172.21.15.113:0/1101371595 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f27b408d400 con 0x7f27c00fd150 2024-04-12T05:39:33.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.875+0000 7f27beffd700 1 --2- 172.21.15.113:0/1101371595 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27a806caa0 0x7f27a806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:39:33.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.875+0000 7f27beffd700 1 --2- 172.21.15.113:0/1101371595 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27a806caa0 0x7f27a806ef60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f27b0009170 tx=0x7f27b0009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:39:33.876 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:33.878+0000 7f27a7fff700 1 -- 172.21.15.113:0/1101371595 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f27b405aeb0 con 0x7f27c00fd150 2024-04-12T05:39:34.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.145+0000 7f27c59fd700 1 -- 172.21.15.113:0/1101371595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f27c0047540 con 0x7f27c00fd150 2024-04-12T05:39:34.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.145+0000 7f27a7fff700 1 -- 172.21.15.113:0/1101371595 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f27b405aa40 con 0x7f27c00fd150 2024-04-12T05:39:34.143 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:39:34.143 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:39:34.143 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:39:34.143 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:39:34.143 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:39:34.143 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:39:34.143 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:39:34.143 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:39:34.143 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:39:34.144 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:39:34.144 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:39:34.144 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:39:34.144 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:39:34.144 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:39:34.144 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:39:34.144 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:39:34.144 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:39:34.144 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:39:34.145 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:39:34.145 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:39:34.145 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:39:34.145 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:39:34.145 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:39:34.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.147+0000 7f27c59fd700 1 -- 172.21.15.113:0/1101371595 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27a806caa0 msgr2=0x7f27a806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:34.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.147+0000 7f27c59fd700 1 --2- 172.21.15.113:0/1101371595 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27a806caa0 0x7f27a806ef60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f27b0009170 tx=0x7f27b0009d60).stop 2024-04-12T05:39:34.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.147+0000 7f27c59fd700 1 -- 172.21.15.113:0/1101371595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27c00fd150 msgr2=0x7f27c010d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:39:34.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.147+0000 7f27c59fd700 1 --2- 172.21.15.113:0/1101371595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27c00fd150 0x7f27c010d3c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f27b40099f0 tx=0x7f27b4009cb0).stop 2024-04-12T05:39:34.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.147+0000 7f27c59fd700 1 -- 172.21.15.113:0/1101371595 shutdown_connections 2024-04-12T05:39:34.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.147+0000 7f27c59fd700 1 --2- 172.21.15.113:0/1101371595 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f27a806caa0 0x7f27a806ef60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:34.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.147+0000 7f27c59fd700 1 --2- 172.21.15.113:0/1101371595 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27c00fc7f0 0x7f27c010ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:34.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.147+0000 7f27c59fd700 1 --2- 172.21.15.113:0/1101371595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27c00fd150 0x7f27c010d3c0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:39:34.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.147+0000 7f27c59fd700 1 -- 172.21.15.113:0/1101371595 >> 172.21.15.113:0/1101371595 conn(0x7f27c0070590 msgr2=0x7f27c00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:39:34.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.147+0000 7f27c59fd700 1 -- 172.21.15.113:0/1101371595 shutdown_connections 2024-04-12T05:39:34.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:39:34.147+0000 7f27c59fd700 1 -- 172.21.15.113:0/1101371595 wait complete. 2024-04-12T05:39:34.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:34 smithi113 ceph-mon[103371]: from='client.35490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:34.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:34 smithi113 ceph-mon[103371]: from='client.35494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:34.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:34 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2735525293' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:39:34.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:34 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1101371595' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:39:34.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:34 smithi192 ceph-mon[85304]: from='client.35490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:34.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:34 smithi192 ceph-mon[85304]: from='client.35494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:34.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:34 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2735525293' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:39:34.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:34 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1101371595' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:39:35.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:35 smithi113 ceph-mon[103371]: from='client.35506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:35.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:35 smithi113 ceph-mon[103371]: pgmap v823: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:35.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:35 smithi192 ceph-mon[85304]: from='client.35506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:39:35.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:35 smithi192 ceph-mon[85304]: pgmap v823: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:36.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:36 smithi113 ceph-mon[103371]: pgmap v824: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:36.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:36 smithi192 ceph-mon[85304]: pgmap v824: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:37.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:37.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:38.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:38 smithi113 ceph-mon[103371]: pgmap v825: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:38.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:38 smithi192 ceph-mon[85304]: pgmap v825: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:41.169 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:40 smithi113 ceph-mon[103371]: pgmap v826: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:41.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:40 smithi192 ceph-mon[85304]: pgmap v826: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:43.169 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:42 smithi113 ceph-mon[103371]: pgmap v827: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:43.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:42 smithi192 ceph-mon[85304]: pgmap v827: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:45.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:44 smithi192 ceph-mon[85304]: pgmap v828: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:45.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:44 smithi113 ceph-mon[103371]: pgmap v828: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:47.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:46 smithi192 ceph-mon[85304]: pgmap v829: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:47.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:46 smithi113 ceph-mon[103371]: pgmap v829: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:49.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:49 smithi113 ceph-mon[103371]: pgmap v830: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:49.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:49 smithi192 ceph-mon[85304]: pgmap v830: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:51.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:51 smithi113 ceph-mon[103371]: pgmap v831: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:51.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:51 smithi192 ceph-mon[85304]: pgmap v831: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:52.600 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:52.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:39:53.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:53 smithi113 ceph-mon[103371]: pgmap v832: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:53.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:53 smithi192 ceph-mon[85304]: pgmap v832: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:55.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:55 smithi113 ceph-mon[103371]: pgmap v833: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:55.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:55 smithi192 ceph-mon[85304]: pgmap v833: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:57.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:57 smithi113 ceph-mon[103371]: pgmap v834: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:57.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:57 smithi192 ceph-mon[85304]: pgmap v834: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:59.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:39:59 smithi113 ceph-mon[103371]: pgmap v835: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:39:59.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:39:59 smithi192 ceph-mon[85304]: pgmap v835: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:00.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:40:00.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:40:00.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:40:00.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : /usr/bin/podman: timeout after 895 seconds 2024-04-12T05:40:00.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:40:00.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : Traceback (most recent call last): 2024-04-12T05:40:00.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:40:00.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : "__main__", mod_spec) 2024-04-12T05:40:00.420 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : exec(code, run_globals) 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:40:00.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:39:59.999+0000 7f1428c33700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: /usr/bin/podman: timeout after 895 seconds 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: Traceback (most recent call last): 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: "__main__", mod_spec) 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: exec(code, run_globals) 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:40:00.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:40:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:40:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:40:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:40:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:40:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:40:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:40:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:40:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:40:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: fs cephfs is degraded 2024-04-12T05:40:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:40:00.423 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:00 smithi113 ceph-mon[103371]: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:40:00.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:40:00.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:40:00.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:40:00.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: /usr/bin/podman: timeout after 895 seconds 2024-04-12T05:40:00.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:40:00.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: Traceback (most recent call last): 2024-04-12T05:40:00.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:40:00.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: "__main__", mod_spec) 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: exec(code, run_globals) 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: fs cephfs is degraded 2024-04-12T05:40:00.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:40:00.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:00 smithi192 ceph-mon[85304]: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:40:01.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:01 smithi113 ceph-mon[103371]: pgmap v836: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:01.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:01 smithi192 ceph-mon[85304]: pgmap v836: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:03.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:03 smithi113 ceph-mon[103371]: pgmap v837: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:03.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:03 smithi192 ceph-mon[85304]: pgmap v837: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:04.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.260+0000 7f6270bda700 1 -- 172.21.15.113:0/919326625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f626c0fe270 msgr2=0x7f626c0f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:04.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.260+0000 7f6270bda700 1 --2- 172.21.15.113:0/919326625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f626c0fe270 0x7f626c0f7a50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f625c0047f0 tx=0x7f625c009cb0).stop 2024-04-12T05:40:04.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.260+0000 7f6270bda700 1 -- 172.21.15.113:0/919326625 shutdown_connections 2024-04-12T05:40:04.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.260+0000 7f6270bda700 1 --2- 172.21.15.113:0/919326625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f626c0fe270 0x7f626c0f7a50 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.260+0000 7f6270bda700 1 --2- 172.21.15.113:0/919326625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f626c0fd910 0x7f626c0fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.260+0000 7f6270bda700 1 -- 172.21.15.113:0/919326625 >> 172.21.15.113:0/919326625 conn(0x7f626c0f3510 msgr2=0x7f626c0f5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:04.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.260+0000 7f6270bda700 1 -- 172.21.15.113:0/919326625 shutdown_connections 2024-04-12T05:40:04.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.260+0000 7f6270bda700 1 -- 172.21.15.113:0/919326625 wait complete. 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.261+0000 7f6270bda700 1 Processor -- start 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.261+0000 7f6270bda700 1 -- start start 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.261+0000 7f6270bda700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f626c0fd910 0x7f626c10e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.261+0000 7f6270bda700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f626c0fe270 0x7f626c10e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.261+0000 7f6270bda700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c10eb30 con 0x7f626c0fd910 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.261+0000 7f6270bda700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c10ec70 con 0x7f626c0fe270 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.261+0000 7f626a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f626c0fd910 0x7f626c10e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.261+0000 7f626a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f626c0fd910 0x7f626c10e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44570/0 (socket says 172.21.15.113:44570) 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.261+0000 7f626a59c700 1 -- 172.21.15.113:0/326343556 learned_addr learned my addr 172.21.15.113:0/326343556 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.261+0000 7f6269d9b700 1 --2- 172.21.15.113:0/326343556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f626c0fe270 0x7f626c10e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.262+0000 7f626a59c700 1 -- 172.21.15.113:0/326343556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f626c0fe270 msgr2=0x7f626c10e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:04.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.262+0000 7f626a59c700 1 --2- 172.21.15.113:0/326343556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f626c0fe270 0x7f626c10e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.262+0000 7f626a59c700 1 -- 172.21.15.113:0/326343556 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c005040 con 0x7f626c0fd910 2024-04-12T05:40:04.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.262+0000 7f626a59c700 1 --2- 172.21.15.113:0/326343556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f626c0fd910 0x7f626c10e040 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f626000c790 tx=0x7f626000caa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:04.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.262+0000 7f625b7fe700 1 -- 172.21.15.113:0/326343556 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f626000e680 con 0x7f626c0fd910 2024-04-12T05:40:04.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.262+0000 7f625b7fe700 1 -- 172.21.15.113:0/326343556 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f626000ec90 con 0x7f626c0fd910 2024-04-12T05:40:04.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.262+0000 7f6270bda700 1 -- 172.21.15.113:0/326343556 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626c108300 con 0x7f626c0fd910 2024-04-12T05:40:04.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.262+0000 7f625b7fe700 1 -- 172.21.15.113:0/326343556 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6260012de0 con 0x7f626c0fd910 2024-04-12T05:40:04.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.262+0000 7f6270bda700 1 -- 172.21.15.113:0/326343556 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c108840 con 0x7f626c0fd910 2024-04-12T05:40:04.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.263+0000 7f6270bda700 1 -- 172.21.15.113:0/326343556 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626c060650 con 0x7f626c0fd910 2024-04-12T05:40:04.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.264+0000 7f625b7fe700 1 -- 172.21.15.113:0/326343556 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f626000e7e0 con 0x7f626c0fd910 2024-04-12T05:40:04.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.264+0000 7f625b7fe700 1 --2- 172.21.15.113:0/326343556 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f625406cab0 0x7f625406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:04.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.264+0000 7f625b7fe700 1 -- 172.21.15.113:0/326343556 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f626008b640 con 0x7f626c0fd910 2024-04-12T05:40:04.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.264+0000 7f6269d9b700 1 --2- 172.21.15.113:0/326343556 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f625406cab0 0x7f625406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:04.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.264+0000 7f6269d9b700 1 --2- 172.21.15.113:0/326343556 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f625406cab0 0x7f625406ef70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f626c109980 tx=0x7f625c009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:04.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.268+0000 7f625b7fe700 1 -- 172.21.15.113:0/326343556 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f62600590f0 con 0x7f626c0fd910 2024-04-12T05:40:04.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.476+0000 7f6270bda700 1 -- 172.21.15.113:0/326343556 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f626c108b10 con 0x7f625406cab0 2024-04-12T05:40:04.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.478+0000 7f625b7fe700 1 -- 172.21.15.113:0/326343556 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f626c108b10 con 0x7f625406cab0 2024-04-12T05:40:04.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.480+0000 7f6270bda700 1 -- 172.21.15.113:0/326343556 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f625406cab0 msgr2=0x7f625406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:04.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.480+0000 7f6270bda700 1 --2- 172.21.15.113:0/326343556 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f625406cab0 0x7f625406ef70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f626c109980 tx=0x7f625c009cb0).stop 2024-04-12T05:40:04.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.480+0000 7f6270bda700 1 -- 172.21.15.113:0/326343556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f626c0fd910 msgr2=0x7f626c10e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:04.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.480+0000 7f6270bda700 1 --2- 172.21.15.113:0/326343556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f626c0fd910 0x7f626c10e040 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f626000c790 tx=0x7f626000caa0).stop 2024-04-12T05:40:04.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.480+0000 7f6270bda700 1 -- 172.21.15.113:0/326343556 shutdown_connections 2024-04-12T05:40:04.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.480+0000 7f6270bda700 1 --2- 172.21.15.113:0/326343556 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f625406cab0 0x7f625406ef70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.480+0000 7f6270bda700 1 --2- 172.21.15.113:0/326343556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f626c0fd910 0x7f626c10e040 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.480+0000 7f6270bda700 1 --2- 172.21.15.113:0/326343556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f626c0fe270 0x7f626c10e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.480+0000 7f6270bda700 1 -- 172.21.15.113:0/326343556 >> 172.21.15.113:0/326343556 conn(0x7f626c0f3510 msgr2=0x7f626c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:04.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.480+0000 7f6270bda700 1 -- 172.21.15.113:0/326343556 shutdown_connections 2024-04-12T05:40:04.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.480+0000 7f6270bda700 1 -- 172.21.15.113:0/326343556 wait complete. 2024-04-12T05:40:04.487 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:40:04.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.578+0000 7f3dc97ca700 1 -- 172.21.15.113:0/1669338977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dc40fd150 msgr2=0x7f3dc40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:04.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.578+0000 7f3dc97ca700 1 --2- 172.21.15.113:0/1669338977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dc40fd150 0x7f3dc40f6930 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f3dac0047f0 tx=0x7f3dac009cb0).stop 2024-04-12T05:40:04.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.578+0000 7f3dc97ca700 1 -- 172.21.15.113:0/1669338977 shutdown_connections 2024-04-12T05:40:04.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.578+0000 7f3dc97ca700 1 --2- 172.21.15.113:0/1669338977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dc40fd150 0x7f3dc40f6930 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.578+0000 7f3dc97ca700 1 --2- 172.21.15.113:0/1669338977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dc40fc7f0 0x7f3dc40fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.578+0000 7f3dc97ca700 1 -- 172.21.15.113:0/1669338977 >> 172.21.15.113:0/1669338977 conn(0x7f3dc40f23b0 msgr2=0x7f3dc40f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:04.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.578+0000 7f3dc97ca700 1 -- 172.21.15.113:0/1669338977 shutdown_connections 2024-04-12T05:40:04.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.578+0000 7f3dc97ca700 1 -- 172.21.15.113:0/1669338977 wait complete. 2024-04-12T05:40:04.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.579+0000 7f3dc97ca700 1 Processor -- start 2024-04-12T05:40:04.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.579+0000 7f3dc97ca700 1 -- start start 2024-04-12T05:40:04.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.579+0000 7f3dc97ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dc40fc7f0 0x7f3dc41101e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:04.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.579+0000 7f3dc97ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dc40fd150 0x7f3dc41106d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.579+0000 7f3dc97ca700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4110cd0 con 0x7f3dc40fc7f0 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.579+0000 7f3dc97ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4110e10 con 0x7f3dc40fd150 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.579+0000 7f3dc2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dc40fc7f0 0x7f3dc41101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.579+0000 7f3dc2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dc40fc7f0 0x7f3dc41101e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44600/0 (socket says 172.21.15.113:44600) 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.579+0000 7f3dc2ffd700 1 -- 172.21.15.113:0/3042792145 learned_addr learned my addr 172.21.15.113:0/3042792145 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.580+0000 7f3dc27fc700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dc40fd150 0x7f3dc41106d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.580+0000 7f3dc2ffd700 1 -- 172.21.15.113:0/3042792145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dc40fd150 msgr2=0x7f3dc41106d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.580+0000 7f3dc2ffd700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dc40fd150 0x7f3dc41106d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.580+0000 7f3dc2ffd700 1 -- 172.21.15.113:0/3042792145 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac005040 con 0x7f3dc40fc7f0 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.580+0000 7f3dc27fc700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dc40fd150 0x7f3dc41106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.580+0000 7f3dc2ffd700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dc40fc7f0 0x7f3dc41101e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f3db40041d0 tx=0x7f3db4009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.580+0000 7f3dbbfff700 1 -- 172.21.15.113:0/3042792145 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3db4007b40 con 0x7f3dc40fc7f0 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.580+0000 7f3dbbfff700 1 -- 172.21.15.113:0/3042792145 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3db400f040 con 0x7f3dc40fc7f0 2024-04-12T05:40:04.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.580+0000 7f3dc97ca700 1 -- 172.21.15.113:0/3042792145 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc410a4a0 con 0x7f3dc40fc7f0 2024-04-12T05:40:04.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.580+0000 7f3dbbfff700 1 -- 172.21.15.113:0/3042792145 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3db400d800 con 0x7f3dc40fc7f0 2024-04-12T05:40:04.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.580+0000 7f3dc97ca700 1 -- 172.21.15.113:0/3042792145 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc410aa60 con 0x7f3dc40fc7f0 2024-04-12T05:40:04.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.581+0000 7f3dc97ca700 1 -- 172.21.15.113:0/3042792145 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc4060370 con 0x7f3dc40fc7f0 2024-04-12T05:40:04.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.581+0000 7f3dbbfff700 1 -- 172.21.15.113:0/3042792145 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3db4007ca0 con 0x7f3dc40fc7f0 2024-04-12T05:40:04.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.582+0000 7f3dbbfff700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3db006caa0 0x7f3db006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:04.580 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.582+0000 7f3dc27fc700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3db006caa0 0x7f3db006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:04.581 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.582+0000 7f3dbbfff700 1 -- 172.21.15.113:0/3042792145 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3db408b950 con 0x7f3dc40fc7f0 2024-04-12T05:40:04.581 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.582+0000 7f3dc27fc700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3db006caa0 0x7f3db006ef60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f3dac004f10 tx=0x7f3dac00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:04.583 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.586+0000 7f3dbbfff700 1 -- 172.21.15.113:0/3042792145 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3db4059400 con 0x7f3dc40fc7f0 2024-04-12T05:40:04.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.792+0000 7f3dc97ca700 1 -- 172.21.15.113:0/3042792145 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3dc410b100 con 0x7f3db006caa0 2024-04-12T05:40:04.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.793+0000 7f3dbbfff700 1 -- 172.21.15.113:0/3042792145 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3dc410b100 con 0x7f3db006caa0 2024-04-12T05:40:04.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.795+0000 7f3dc97ca700 1 -- 172.21.15.113:0/3042792145 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3db006caa0 msgr2=0x7f3db006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:04.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.795+0000 7f3dc97ca700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3db006caa0 0x7f3db006ef60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f3dac004f10 tx=0x7f3dac00b040).stop 2024-04-12T05:40:04.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.795+0000 7f3dc97ca700 1 -- 172.21.15.113:0/3042792145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dc40fc7f0 msgr2=0x7f3dc41101e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:04.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.795+0000 7f3dc97ca700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dc40fc7f0 0x7f3dc41101e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f3db40041d0 tx=0x7f3db4009f60).stop 2024-04-12T05:40:04.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.795+0000 7f3dc97ca700 1 -- 172.21.15.113:0/3042792145 shutdown_connections 2024-04-12T05:40:04.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.795+0000 7f3dc97ca700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3db006caa0 0x7f3db006ef60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.795+0000 7f3dc97ca700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dc40fc7f0 0x7f3dc41101e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.795+0000 7f3dc97ca700 1 --2- 172.21.15.113:0/3042792145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dc40fd150 0x7f3dc41106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.795+0000 7f3dc97ca700 1 -- 172.21.15.113:0/3042792145 >> 172.21.15.113:0/3042792145 conn(0x7f3dc40f23b0 msgr2=0x7f3dc4061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:04.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.795+0000 7f3dc97ca700 1 -- 172.21.15.113:0/3042792145 shutdown_connections 2024-04-12T05:40:04.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.795+0000 7f3dc97ca700 1 -- 172.21.15.113:0/3042792145 wait complete. 2024-04-12T05:40:04.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.893+0000 7fd14b327700 1 -- 172.21.15.113:0/711751907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1440fd8f0 msgr2=0x7fd1440fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:04.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.893+0000 7fd14b327700 1 --2- 172.21.15.113:0/711751907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1440fd8f0 0x7fd1440fdcd0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7fd1340047f0 tx=0x7fd134009cb0).stop 2024-04-12T05:40:04.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.893+0000 7fd14b327700 1 -- 172.21.15.113:0/711751907 shutdown_connections 2024-04-12T05:40:04.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.893+0000 7fd14b327700 1 --2- 172.21.15.113:0/711751907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1440fe250 0x7fd1440f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.893+0000 7fd14b327700 1 --2- 172.21.15.113:0/711751907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1440fd8f0 0x7fd1440fdcd0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.893+0000 7fd14b327700 1 -- 172.21.15.113:0/711751907 >> 172.21.15.113:0/711751907 conn(0x7fd1440f34b0 msgr2=0x7fd1440f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.893+0000 7fd14b327700 1 -- 172.21.15.113:0/711751907 shutdown_connections 2024-04-12T05:40:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.893+0000 7fd14b327700 1 -- 172.21.15.113:0/711751907 wait complete. 2024-04-12T05:40:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd14b327700 1 Processor -- start 2024-04-12T05:40:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd14b327700 1 -- start start 2024-04-12T05:40:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd14b327700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1440fd8f0 0x7fd14410dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd14b327700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1440fe250 0x7fd14410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd14b327700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14410ead0 con 0x7fd1440fd8f0 2024-04-12T05:40:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd14b327700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14410ec10 con 0x7fd1440fe250 2024-04-12T05:40:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd1490c3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1440fd8f0 0x7fd14410dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:04.893 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd1490c3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1440fd8f0 0x7fd14410dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44618/0 (socket says 172.21.15.113:44618) 2024-04-12T05:40:04.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd1490c3700 1 -- 172.21.15.113:0/1026566751 learned_addr learned my addr 172.21.15.113:0/1026566751 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:04.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd1488c2700 1 --2- 172.21.15.113:0/1026566751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1440fe250 0x7fd14410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:04.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd1490c3700 1 -- 172.21.15.113:0/1026566751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1440fe250 msgr2=0x7fd14410e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:04.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd1490c3700 1 --2- 172.21.15.113:0/1026566751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1440fe250 0x7fd14410e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:04.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd1490c3700 1 -- 172.21.15.113:0/1026566751 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd134005040 con 0x7fd1440fd8f0 2024-04-12T05:40:04.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.894+0000 7fd1490c3700 1 --2- 172.21.15.113:0/1026566751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1440fd8f0 0x7fd14410dfe0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fd134005010 tx=0x7fd134009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:04.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.895+0000 7fd13a7fc700 1 -- 172.21.15.113:0/1026566751 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd13400e030 con 0x7fd1440fd8f0 2024-04-12T05:40:04.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.895+0000 7fd13a7fc700 1 -- 172.21.15.113:0/1026566751 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd134007e90 con 0x7fd1440fd8f0 2024-04-12T05:40:04.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.895+0000 7fd14b327700 1 -- 172.21.15.113:0/1026566751 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd144108240 con 0x7fd1440fd8f0 2024-04-12T05:40:04.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.895+0000 7fd13a7fc700 1 -- 172.21.15.113:0/1026566751 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd134018070 con 0x7fd1440fd8f0 2024-04-12T05:40:04.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.895+0000 7fd14b327700 1 -- 172.21.15.113:0/1026566751 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1441087a0 con 0x7fd1440fd8f0 2024-04-12T05:40:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.896+0000 7fd14b327700 1 -- 172.21.15.113:0/1026566751 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14405f590 con 0x7fd1440fd8f0 2024-04-12T05:40:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.896+0000 7fd13a7fc700 1 -- 172.21.15.113:0/1026566751 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd13401d6e0 con 0x7fd1440fd8f0 2024-04-12T05:40:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.896+0000 7fd13a7fc700 1 --2- 172.21.15.113:0/1026566751 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd13006ca30 0x7fd13006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.897+0000 7fd1488c2700 1 --2- 172.21.15.113:0/1026566751 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd13006ca30 0x7fd13006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.897+0000 7fd13a7fc700 1 -- 172.21.15.113:0/1026566751 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd13408d100 con 0x7fd1440fd8f0 2024-04-12T05:40:04.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.897+0000 7fd1488c2700 1 --2- 172.21.15.113:0/1026566751 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd13006ca30 0x7fd13006eef0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd144109920 tx=0x7fd140009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:04.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:04.901+0000 7fd13a7fc700 1 -- 172.21.15.113:0/1026566751 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd13405a920 con 0x7fd1440fd8f0 2024-04-12T05:40:05.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.103+0000 7fd14b327700 1 -- 172.21.15.113:0/1026566751 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1440ffc40 con 0x7fd13006ca30 2024-04-12T05:40:05.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.112+0000 7fd13a7fc700 1 -- 172.21.15.113:0/1026566751 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd1440ffc40 con 0x7fd13006ca30 2024-04-12T05:40:05.110 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:40:05.110 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (27m) 5m ago 39m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:40:05.110 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (25m) 5m ago 39m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:40:05.110 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (25m) 5m ago 38m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:40:05.110 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (27m) 5m ago 38m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:40:05.110 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (21m) 5m ago 33m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:40:05.110 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (20m) 5m ago 33m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:40:05.110 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (20m) 5m ago 33m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:40:05.110 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (5m) 5m ago 33m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:40:05.110 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (29m) 5m ago 40m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:40:05.110 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (28m) 5m ago 38m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:40:05.111 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (26m) 5m ago 40m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:40:05.111 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (25m) 5m ago 38m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:40:05.111 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (28m) 5m ago 38m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:40:05.111 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (28m) 5m ago 38m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:40:05.111 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (24m) 5m ago 37m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:40:05.111 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (24m) 5m ago 37m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:40:05.111 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (23m) 5m ago 36m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:40:05.111 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (23m) 5m ago 36m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:40:05.111 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (22m) 5m ago 35m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:40:05.111 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (21m) 5m ago 35m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:40:05.111 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (28m) 5m ago 38m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:40:05.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.114+0000 7fd14b327700 1 -- 172.21.15.113:0/1026566751 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd13006ca30 msgr2=0x7fd13006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.114+0000 7fd14b327700 1 --2- 172.21.15.113:0/1026566751 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd13006ca30 0x7fd13006eef0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd144109920 tx=0x7fd140009d60).stop 2024-04-12T05:40:05.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.114+0000 7fd14b327700 1 -- 172.21.15.113:0/1026566751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1440fd8f0 msgr2=0x7fd14410dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.114+0000 7fd14b327700 1 --2- 172.21.15.113:0/1026566751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1440fd8f0 0x7fd14410dfe0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fd134005010 tx=0x7fd134009cb0).stop 2024-04-12T05:40:05.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.115+0000 7fd14b327700 1 -- 172.21.15.113:0/1026566751 shutdown_connections 2024-04-12T05:40:05.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.115+0000 7fd14b327700 1 --2- 172.21.15.113:0/1026566751 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd13006ca30 0x7fd13006eef0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.115+0000 7fd14b327700 1 --2- 172.21.15.113:0/1026566751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1440fd8f0 0x7fd14410dfe0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.115+0000 7fd14b327700 1 --2- 172.21.15.113:0/1026566751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1440fe250 0x7fd14410e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.115+0000 7fd14b327700 1 -- 172.21.15.113:0/1026566751 >> 172.21.15.113:0/1026566751 conn(0x7fd1440f34b0 msgr2=0x7fd1440fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:05.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.115+0000 7fd14b327700 1 -- 172.21.15.113:0/1026566751 shutdown_connections 2024-04-12T05:40:05.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.115+0000 7fd14b327700 1 -- 172.21.15.113:0/1026566751 wait complete. 2024-04-12T05:40:05.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.211+0000 7fe7abbe1700 1 -- 172.21.15.113:0/4250083618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a40fd150 msgr2=0x7fe7a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.211+0000 7fe7abbe1700 1 --2- 172.21.15.113:0/4250083618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a40fd150 0x7fe7a40f6930 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7fe7a00047f0 tx=0x7fe7a0009cb0).stop 2024-04-12T05:40:05.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.211+0000 7fe7abbe1700 1 -- 172.21.15.113:0/4250083618 shutdown_connections 2024-04-12T05:40:05.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.211+0000 7fe7abbe1700 1 --2- 172.21.15.113:0/4250083618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a40fd150 0x7fe7a40f6930 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.211+0000 7fe7abbe1700 1 --2- 172.21.15.113:0/4250083618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a40fc7f0 0x7fe7a40fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.211+0000 7fe7abbe1700 1 -- 172.21.15.113:0/4250083618 >> 172.21.15.113:0/4250083618 conn(0x7fe7a4070590 msgr2=0x7fe7a40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:05.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.211+0000 7fe7abbe1700 1 -- 172.21.15.113:0/4250083618 shutdown_connections 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.211+0000 7fe7abbe1700 1 -- 172.21.15.113:0/4250083618 wait complete. 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.212+0000 7fe7abbe1700 1 Processor -- start 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.212+0000 7fe7abbe1700 1 -- start start 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.212+0000 7fe7abbe1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a40fc7f0 0x7fe7a410ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.212+0000 7fe7abbe1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a40fd150 0x7fe7a410d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.212+0000 7fe7abbe1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a410d9c0 con 0x7fe7a40fd150 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.212+0000 7fe7abbe1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a410db00 con 0x7fe7a40fc7f0 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.212+0000 7fe7a917c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a40fd150 0x7fe7a410d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.212+0000 7fe7a917c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a40fd150 0x7fe7a410d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44646/0 (socket says 172.21.15.113:44646) 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.212+0000 7fe7a917c700 1 -- 172.21.15.113:0/955710468 learned_addr learned my addr 172.21.15.113:0/955710468 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.212+0000 7fe7a997d700 1 --2- 172.21.15.113:0/955710468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a40fc7f0 0x7fe7a410ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.213+0000 7fe7a917c700 1 -- 172.21.15.113:0/955710468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a40fc7f0 msgr2=0x7fe7a410ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.213+0000 7fe7a917c700 1 --2- 172.21.15.113:0/955710468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a40fc7f0 0x7fe7a410ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.211 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.213+0000 7fe7a917c700 1 -- 172.21.15.113:0/955710468 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a0005040 con 0x7fe7a40fd150 2024-04-12T05:40:05.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.213+0000 7fe7a917c700 1 --2- 172.21.15.113:0/955710468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a40fd150 0x7fe7a410d3c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fe7a00099f0 tx=0x7fe7a0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:05.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.213+0000 7fe796ffd700 1 -- 172.21.15.113:0/955710468 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7a000e030 con 0x7fe7a40fd150 2024-04-12T05:40:05.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.213+0000 7fe796ffd700 1 -- 172.21.15.113:0/955710468 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7a0007e90 con 0x7fe7a40fd150 2024-04-12T05:40:05.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.213+0000 7fe7abbe1700 1 -- 172.21.15.113:0/955710468 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a4107130 con 0x7fe7a40fd150 2024-04-12T05:40:05.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.213+0000 7fe796ffd700 1 -- 172.21.15.113:0/955710468 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7a0018070 con 0x7fe7a40fd150 2024-04-12T05:40:05.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.213+0000 7fe7abbe1700 1 -- 172.21.15.113:0/955710468 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a4107690 con 0x7fe7a40fd150 2024-04-12T05:40:05.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.214+0000 7fe7abbe1700 1 -- 172.21.15.113:0/955710468 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a405f590 con 0x7fe7a40fd150 2024-04-12T05:40:05.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.214+0000 7fe796ffd700 1 -- 172.21.15.113:0/955710468 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe7a001d420 con 0x7fe7a40fd150 2024-04-12T05:40:05.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.215+0000 7fe796ffd700 1 --2- 172.21.15.113:0/955710468 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 0x7fe79006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:05.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.215+0000 7fe796ffd700 1 -- 172.21.15.113:0/955710468 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe7a008d2f0 con 0x7fe7a40fd150 2024-04-12T05:40:05.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.215+0000 7fe7a997d700 1 --2- 172.21.15.113:0/955710468 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 0x7fe79006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:05.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.215+0000 7fe7a997d700 1 --2- 172.21.15.113:0/955710468 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 0x7fe79006ef60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fe7980041d0 tx=0x7fe798009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:05.216 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.219+0000 7fe796ffd700 1 -- 172.21.15.113:0/955710468 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe7a005ada0 con 0x7fe7a40fd150 2024-04-12T05:40:05.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.490+0000 7fe7abbe1700 1 -- 172.21.15.113:0/955710468 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe7a40feb40 con 0x7fe7a40fd150 2024-04-12T05:40:05.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.491+0000 7fe796ffd700 1 -- 172.21.15.113:0/955710468 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fe7a005a930 con 0x7fe7a40fd150 2024-04-12T05:40:05.488 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:40:05.488 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:40:05.488 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:40:05.488 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:40:05.488 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:40:05.488 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:40:05.488 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:40:05.489 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:40:05.489 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:40:05.489 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:40:05.489 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:40:05.489 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:40:05.489 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:40:05.489 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:40:05.489 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:40:05.489 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:40:05.489 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:40:05.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.493+0000 7fe7abbe1700 1 -- 172.21.15.113:0/955710468 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 msgr2=0x7fe79006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.493+0000 7fe7abbe1700 1 --2- 172.21.15.113:0/955710468 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 0x7fe79006ef60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fe7980041d0 tx=0x7fe798009040).stop 2024-04-12T05:40:05.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.493+0000 7fe7abbe1700 1 -- 172.21.15.113:0/955710468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a40fd150 msgr2=0x7fe7a410d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.493+0000 7fe7abbe1700 1 --2- 172.21.15.113:0/955710468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a40fd150 0x7fe7a410d3c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fe7a00099f0 tx=0x7fe7a0009cb0).stop 2024-04-12T05:40:05.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.493+0000 7fe7abbe1700 1 -- 172.21.15.113:0/955710468 shutdown_connections 2024-04-12T05:40:05.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.493+0000 7fe7abbe1700 1 --2- 172.21.15.113:0/955710468 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe79006caa0 0x7fe79006ef60 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.493+0000 7fe7abbe1700 1 --2- 172.21.15.113:0/955710468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7a40fc7f0 0x7fe7a410ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.493+0000 7fe7abbe1700 1 --2- 172.21.15.113:0/955710468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7a40fd150 0x7fe7a410d3c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.493+0000 7fe7abbe1700 1 -- 172.21.15.113:0/955710468 >> 172.21.15.113:0/955710468 conn(0x7fe7a4070590 msgr2=0x7fe7a40fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:05.490 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.493+0000 7fe7abbe1700 1 -- 172.21.15.113:0/955710468 shutdown_connections 2024-04-12T05:40:05.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.493+0000 7fe7abbe1700 1 -- 172.21.15.113:0/955710468 wait complete. 2024-04-12T05:40:05.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.590+0000 7f6a26355700 1 -- 172.21.15.113:0/1017303405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a200f9260 msgr2=0x7f6a20101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.590+0000 7f6a26355700 1 --2- 172.21.15.113:0/1017303405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a200f9260 0x7f6a20101970 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7f6a140047f0 tx=0x7f6a14009cb0).stop 2024-04-12T05:40:05.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.590+0000 7f6a26355700 1 -- 172.21.15.113:0/1017303405 shutdown_connections 2024-04-12T05:40:05.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.590+0000 7f6a26355700 1 --2- 172.21.15.113:0/1017303405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a200f9260 0x7f6a20101970 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.590+0000 7f6a26355700 1 --2- 172.21.15.113:0/1017303405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a200f8900 0x7f6a200f8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.590+0000 7f6a26355700 1 -- 172.21.15.113:0/1017303405 >> 172.21.15.113:0/1017303405 conn(0x7f6a20070590 msgr2=0x7f6a200f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.590+0000 7f6a26355700 1 -- 172.21.15.113:0/1017303405 shutdown_connections 2024-04-12T05:40:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.590+0000 7f6a26355700 1 -- 172.21.15.113:0/1017303405 wait complete. 2024-04-12T05:40:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.591+0000 7f6a26355700 1 Processor -- start 2024-04-12T05:40:05.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.591+0000 7f6a26355700 1 -- start start 2024-04-12T05:40:05.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.591+0000 7f6a26355700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a200f8900 0x7f6a2010e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:05.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.591+0000 7f6a26355700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a200f9260 0x7f6a2010e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:05.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.591+0000 7f6a26355700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2010eb00 con 0x7f6a200f8900 2024-04-12T05:40:05.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.591+0000 7f6a26355700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2010ec40 con 0x7f6a200f9260 2024-04-12T05:40:05.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a200f8900 0x7f6a2010e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:05.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a200f8900 0x7f6a2010e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44656/0 (socket says 172.21.15.113:44656) 2024-04-12T05:40:05.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1ffff700 1 -- 172.21.15.113:0/1227281422 learned_addr learned my addr 172.21.15.113:0/1227281422 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:05.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1f7fe700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a200f9260 0x7f6a2010e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:05.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1ffff700 1 -- 172.21.15.113:0/1227281422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a200f9260 msgr2=0x7f6a2010e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1ffff700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a200f9260 0x7f6a2010e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1ffff700 1 -- 172.21.15.113:0/1227281422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a14005040 con 0x7f6a200f8900 2024-04-12T05:40:05.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1f7fe700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a200f9260 0x7f6a2010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:40:05.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1ffff700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a200f8900 0x7f6a2010e010 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f6a1000c740 tx=0x7f6a10007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:05.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1d7fa700 1 -- 172.21.15.113:0/1227281422 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a1000e680 con 0x7f6a200f8900 2024-04-12T05:40:05.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1d7fa700 1 -- 172.21.15.113:0/1227281422 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a1000ec90 con 0x7f6a200f8900 2024-04-12T05:40:05.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a26355700 1 -- 172.21.15.113:0/1227281422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a201082d0 con 0x7f6a200f8900 2024-04-12T05:40:05.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a1d7fa700 1 -- 172.21.15.113:0/1227281422 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a10013d60 con 0x7f6a200f8900 2024-04-12T05:40:05.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.592+0000 7f6a26355700 1 -- 172.21.15.113:0/1227281422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a20108810 con 0x7f6a200f8900 2024-04-12T05:40:05.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.593+0000 7f6a26355700 1 -- 172.21.15.113:0/1227281422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2005f590 con 0x7f6a200f8900 2024-04-12T05:40:05.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.594+0000 7f6a1d7fa700 1 -- 172.21.15.113:0/1227281422 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6a1000e7e0 con 0x7f6a200f8900 2024-04-12T05:40:05.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.594+0000 7f6a1d7fa700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6a0806caa0 0x7f6a0806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:05.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.594+0000 7f6a1d7fa700 1 -- 172.21.15.113:0/1227281422 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6a1008c150 con 0x7f6a200f8900 2024-04-12T05:40:05.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.594+0000 7f6a1f7fe700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6a0806caa0 0x7f6a0806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:05.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.594+0000 7f6a1f7fe700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6a0806caa0 0x7f6a0806ef60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f6a14000c00 tx=0x7f6a1400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:05.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.598+0000 7f6a1d7fa700 1 -- 172.21.15.113:0/1227281422 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6a10059c00 con 0x7f6a200f8900 2024-04-12T05:40:05.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:05 smithi113 ceph-mon[103371]: pgmap v838: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:05.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:05 smithi192 ceph-mon[85304]: pgmap v838: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:05.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.824+0000 7f6a26355700 1 -- 172.21.15.113:0/1227281422 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6a20109790 con 0x7f6a200f8900 2024-04-12T05:40:05.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.824+0000 7f6a1d7fa700 1 -- 172.21.15.113:0/1227281422 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f6a10059790 con 0x7f6a200f8900 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:40:05.822 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:40:05.823 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:40:05.824 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:40:05.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.826+0000 7f6a26355700 1 -- 172.21.15.113:0/1227281422 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6a0806caa0 msgr2=0x7f6a0806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.826+0000 7f6a26355700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6a0806caa0 0x7f6a0806ef60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f6a14000c00 tx=0x7f6a1400b040).stop 2024-04-12T05:40:05.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.826+0000 7f6a26355700 1 -- 172.21.15.113:0/1227281422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a200f8900 msgr2=0x7f6a2010e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.826+0000 7f6a26355700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a200f8900 0x7f6a2010e010 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f6a1000c740 tx=0x7f6a10007ef0).stop 2024-04-12T05:40:05.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.826+0000 7f6a26355700 1 -- 172.21.15.113:0/1227281422 shutdown_connections 2024-04-12T05:40:05.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.826+0000 7f6a26355700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6a0806caa0 0x7f6a0806ef60 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.826+0000 7f6a26355700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a200f8900 0x7f6a2010e010 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.826+0000 7f6a26355700 1 --2- 172.21.15.113:0/1227281422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a200f9260 0x7f6a2010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.826+0000 7f6a26355700 1 -- 172.21.15.113:0/1227281422 >> 172.21.15.113:0/1227281422 conn(0x7f6a20070590 msgr2=0x7f6a200fc170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:05.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.826+0000 7f6a26355700 1 -- 172.21.15.113:0/1227281422 shutdown_connections 2024-04-12T05:40:05.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.827+0000 7f6a26355700 1 -- 172.21.15.113:0/1227281422 wait complete. 2024-04-12T05:40:05.826 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:40:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.926+0000 7f117422c700 1 -- 172.21.15.113:0/3292985624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f116c0fd150 msgr2=0x7f116c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.926+0000 7f117422c700 1 --2- 172.21.15.113:0/3292985624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f116c0fd150 0x7f116c0f6930 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f11680047f0 tx=0x7f1168009cb0).stop 2024-04-12T05:40:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.926+0000 7f117422c700 1 -- 172.21.15.113:0/3292985624 shutdown_connections 2024-04-12T05:40:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.926+0000 7f117422c700 1 --2- 172.21.15.113:0/3292985624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f116c0fd150 0x7f116c0f6930 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.926+0000 7f117422c700 1 --2- 172.21.15.113:0/3292985624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f116c0fc7f0 0x7f116c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.926+0000 7f117422c700 1 -- 172.21.15.113:0/3292985624 >> 172.21.15.113:0/3292985624 conn(0x7f116c070590 msgr2=0x7f116c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.926+0000 7f117422c700 1 -- 172.21.15.113:0/3292985624 shutdown_connections 2024-04-12T05:40:05.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.926+0000 7f117422c700 1 -- 172.21.15.113:0/3292985624 wait complete. 2024-04-12T05:40:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.926+0000 7f117422c700 1 Processor -- start 2024-04-12T05:40:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f117422c700 1 -- start start 2024-04-12T05:40:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f117422c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f116c0fc7f0 0x7f116c10cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f117422c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f116c0fd150 0x7f116c10d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f117422c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c10d9e0 con 0x7f116c0fc7f0 2024-04-12T05:40:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f117422c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c10db20 con 0x7f116c0fd150 2024-04-12T05:40:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f1171fc8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f116c0fc7f0 0x7f116c10cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f1171fc8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f116c0fc7f0 0x7f116c10cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44670/0 (socket says 172.21.15.113:44670) 2024-04-12T05:40:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f1171fc8700 1 -- 172.21.15.113:0/202881365 learned_addr learned my addr 172.21.15.113:0/202881365 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f11717c7700 1 --2- 172.21.15.113:0/202881365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f116c0fd150 0x7f116c10d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:05.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f1171fc8700 1 -- 172.21.15.113:0/202881365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f116c0fd150 msgr2=0x7f116c10d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:05.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f1171fc8700 1 --2- 172.21.15.113:0/202881365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f116c0fd150 0x7f116c10d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:05.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f1171fc8700 1 -- 172.21.15.113:0/202881365 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1168005040 con 0x7f116c0fc7f0 2024-04-12T05:40:05.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f1171fc8700 1 --2- 172.21.15.113:0/202881365 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f116c0fc7f0 0x7f116c10cef0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f116000c790 tx=0x7f116000caa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:05.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f115effd700 1 -- 172.21.15.113:0/202881365 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f116000e680 con 0x7f116c0fc7f0 2024-04-12T05:40:05.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f117422c700 1 -- 172.21.15.113:0/202881365 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116c1071b0 con 0x7f116c0fc7f0 2024-04-12T05:40:05.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f115effd700 1 -- 172.21.15.113:0/202881365 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f116000ec90 con 0x7f116c0fc7f0 2024-04-12T05:40:05.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.927+0000 7f115effd700 1 -- 172.21.15.113:0/202881365 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1160012de0 con 0x7f116c0fc7f0 2024-04-12T05:40:05.927 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.928+0000 7f117422c700 1 -- 172.21.15.113:0/202881365 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f116c107770 con 0x7f116c0fc7f0 2024-04-12T05:40:05.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.928+0000 7f117422c700 1 -- 172.21.15.113:0/202881365 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116c05f590 con 0x7f116c0fc7f0 2024-04-12T05:40:05.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.929+0000 7f115effd700 1 -- 172.21.15.113:0/202881365 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f116000e7e0 con 0x7f116c0fc7f0 2024-04-12T05:40:05.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.930+0000 7f115effd700 1 --2- 172.21.15.113:0/202881365 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f115806caa0 0x7f115806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:05.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.930+0000 7f11717c7700 1 --2- 172.21.15.113:0/202881365 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f115806caa0 0x7f115806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:05.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.930+0000 7f115effd700 1 -- 172.21.15.113:0/202881365 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f116008b720 con 0x7f116c0fc7f0 2024-04-12T05:40:05.928 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.930+0000 7f11717c7700 1 --2- 172.21.15.113:0/202881365 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f115806caa0 0x7f115806ef60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f116c108830 tx=0x7f116800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:05.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:05.934+0000 7f115effd700 1 -- 172.21.15.113:0/202881365 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f11600591d0 con 0x7f116c0fc7f0 2024-04-12T05:40:06.138 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.140+0000 7f117422c700 1 -- 172.21.15.113:0/202881365 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f116c1079f0 con 0x7f115806caa0 2024-04-12T05:40:06.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.142+0000 7f115effd700 1 -- 172.21.15.113:0/202881365 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f116c1079f0 con 0x7f115806caa0 2024-04-12T05:40:06.139 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:40:06.139 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:40:06.139 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:40:06.139 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:40:06.139 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:40:06.139 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:40:06.139 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:40:06.140 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:40:06.140 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:40:06.140 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:40:06.140 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:40:06.140 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:40:06.140 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:40:06.140 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:40:06.140 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:40:06.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.144+0000 7f117422c700 1 -- 172.21.15.113:0/202881365 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f115806caa0 msgr2=0x7f115806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:06.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.144+0000 7f117422c700 1 --2- 172.21.15.113:0/202881365 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f115806caa0 0x7f115806ef60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f116c108830 tx=0x7f116800b040).stop 2024-04-12T05:40:06.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.144+0000 7f117422c700 1 -- 172.21.15.113:0/202881365 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f116c0fc7f0 msgr2=0x7f116c10cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:06.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.144+0000 7f117422c700 1 --2- 172.21.15.113:0/202881365 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f116c0fc7f0 0x7f116c10cef0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f116000c790 tx=0x7f116000caa0).stop 2024-04-12T05:40:06.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.144+0000 7f117422c700 1 -- 172.21.15.113:0/202881365 shutdown_connections 2024-04-12T05:40:06.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.144+0000 7f117422c700 1 --2- 172.21.15.113:0/202881365 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f115806caa0 0x7f115806ef60 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:06.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.144+0000 7f117422c700 1 --2- 172.21.15.113:0/202881365 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f116c0fc7f0 0x7f116c10cef0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:06.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.144+0000 7f117422c700 1 --2- 172.21.15.113:0/202881365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f116c0fd150 0x7f116c10d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:06.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.144+0000 7f117422c700 1 -- 172.21.15.113:0/202881365 >> 172.21.15.113:0/202881365 conn(0x7f116c070590 msgr2=0x7f116c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:06.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.144+0000 7f117422c700 1 -- 172.21.15.113:0/202881365 shutdown_connections 2024-04-12T05:40:06.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.144+0000 7f117422c700 1 -- 172.21.15.113:0/202881365 wait complete. 2024-04-12T05:40:06.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.242+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/487274054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ed00f6710 msgr2=0x7f8ed00f6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:06.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.242+0000 7f8ed7b9a700 1 --2- 172.21.15.113:0/487274054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ed00f6710 0x7f8ed00f6af0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f8ecc0047f0 tx=0x7f8ecc009cb0).stop 2024-04-12T05:40:06.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.242+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/487274054 shutdown_connections 2024-04-12T05:40:06.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.242+0000 7f8ed7b9a700 1 --2- 172.21.15.113:0/487274054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed00f6fe0 0x7f8ed0103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:06.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.242+0000 7f8ed7b9a700 1 --2- 172.21.15.113:0/487274054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ed00f6710 0x7f8ed00f6af0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:06.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.242+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/487274054 >> 172.21.15.113:0/487274054 conn(0x7f8ed0070590 msgr2=0x7f8ed00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:06.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.242+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/487274054 shutdown_connections 2024-04-12T05:40:06.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.242+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/487274054 wait complete. 2024-04-12T05:40:06.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed7b9a700 1 Processor -- start 2024-04-12T05:40:06.240 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed7b9a700 1 -- start start 2024-04-12T05:40:06.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed7b9a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ed00f6710 0x7f8ed0110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:06.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed7b9a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed00f6fe0 0x7f8ed0110780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:06.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed7b9a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed0110d80 con 0x7f8ed00f6710 2024-04-12T05:40:06.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed7b9a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed0110ec0 con 0x7f8ed00f6fe0 2024-04-12T05:40:06.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed5936700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ed00f6710 0x7f8ed0110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:06.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed5936700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ed00f6710 0x7f8ed0110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44702/0 (socket says 172.21.15.113:44702) 2024-04-12T05:40:06.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed5936700 1 -- 172.21.15.113:0/1748791922 learned_addr learned my addr 172.21.15.113:0/1748791922 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:06.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed5135700 1 --2- 172.21.15.113:0/1748791922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed00f6fe0 0x7f8ed0110780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:06.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed5936700 1 -- 172.21.15.113:0/1748791922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed00f6fe0 msgr2=0x7f8ed0110780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:06.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed5936700 1 --2- 172.21.15.113:0/1748791922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed00f6fe0 0x7f8ed0110780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:06.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.243+0000 7f8ed5936700 1 -- 172.21.15.113:0/1748791922 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc005040 con 0x7f8ed00f6710 2024-04-12T05:40:06.242 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.244+0000 7f8ed5936700 1 --2- 172.21.15.113:0/1748791922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ed00f6710 0x7f8ed0110290 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f8ecc0047c0 tx=0x7f8ecc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:06.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.245+0000 7f8ec6ffd700 1 -- 172.21.15.113:0/1748791922 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ecc019070 con 0x7f8ed00f6710 2024-04-12T05:40:06.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.246+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/1748791922 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed010a4f0 con 0x7f8ed00f6710 2024-04-12T05:40:06.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.246+0000 7f8ec6ffd700 1 -- 172.21.15.113:0/1748791922 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ecc009790 con 0x7f8ed00f6710 2024-04-12T05:40:06.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.246+0000 7f8ec6ffd700 1 -- 172.21.15.113:0/1748791922 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ecc014630 con 0x7f8ed00f6710 2024-04-12T05:40:06.243 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.246+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/1748791922 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed010aab0 con 0x7f8ed00f6710 2024-04-12T05:40:06.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.247+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/1748791922 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed005f590 con 0x7f8ed00f6710 2024-04-12T05:40:06.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.247+0000 7f8ec6ffd700 1 -- 172.21.15.113:0/1748791922 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8ecc0092e0 con 0x7f8ed00f6710 2024-04-12T05:40:06.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.248+0000 7f8ec6ffd700 1 --2- 172.21.15.113:0/1748791922 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ebc06caa0 0x7f8ebc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:06.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.248+0000 7f8ec6ffd700 1 -- 172.21.15.113:0/1748791922 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f8ecc08cd60 con 0x7f8ed00f6710 2024-04-12T05:40:06.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.248+0000 7f8ed5135700 1 --2- 172.21.15.113:0/1748791922 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ebc06caa0 0x7f8ebc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:06.246 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.248+0000 7f8ed5135700 1 --2- 172.21.15.113:0/1748791922 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ebc06caa0 0x7f8ebc06ef60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f8ed010bbd0 tx=0x7f8ec000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:06.251 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.254+0000 7f8ec6ffd700 1 -- 172.21.15.113:0/1748791922 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8ecc05a8d0 con 0x7f8ed00f6710 2024-04-12T05:40:06.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.520+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/1748791922 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8ed0047540 con 0x7f8ed00f6710 2024-04-12T05:40:06.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.521+0000 7f8ec6ffd700 1 -- 172.21.15.113:0/1748791922 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f8ecc05a460 con 0x7f8ed00f6710 2024-04-12T05:40:06.519 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:40:06.519 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:40:06.519 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:40:06.519 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:40:06.519 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:40:06.519 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:40:06.519 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:40:06.519 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:40:06.519 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:40:06.519 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:40:06.519 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:40:06.520 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:40:06.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.523+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/1748791922 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ebc06caa0 msgr2=0x7f8ebc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:06.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.523+0000 7f8ed7b9a700 1 --2- 172.21.15.113:0/1748791922 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ebc06caa0 0x7f8ebc06ef60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f8ed010bbd0 tx=0x7f8ec000b040).stop 2024-04-12T05:40:06.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.523+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/1748791922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ed00f6710 msgr2=0x7f8ed0110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:06.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.523+0000 7f8ed7b9a700 1 --2- 172.21.15.113:0/1748791922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ed00f6710 0x7f8ed0110290 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f8ecc0047c0 tx=0x7f8ecc009cb0).stop 2024-04-12T05:40:06.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.523+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/1748791922 shutdown_connections 2024-04-12T05:40:06.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.523+0000 7f8ed7b9a700 1 --2- 172.21.15.113:0/1748791922 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ebc06caa0 0x7f8ebc06ef60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:06.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.523+0000 7f8ed7b9a700 1 --2- 172.21.15.113:0/1748791922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ed00f6710 0x7f8ed0110290 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:06.521 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.523+0000 7f8ed7b9a700 1 --2- 172.21.15.113:0/1748791922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed00f6fe0 0x7f8ed0110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:06.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.523+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/1748791922 >> 172.21.15.113:0/1748791922 conn(0x7f8ed0070590 msgr2=0x7f8ed006d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:06.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.523+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/1748791922 shutdown_connections 2024-04-12T05:40:06.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:06.523+0000 7f8ed7b9a700 1 -- 172.21.15.113:0/1748791922 wait complete. 2024-04-12T05:40:06.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:06 smithi113 ceph-mon[103371]: from='client.35514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:06.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:06 smithi113 ceph-mon[103371]: from='client.35518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:06.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:06 smithi113 ceph-mon[103371]: from='client.35522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:06.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:06 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/955710468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:06.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:06 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1227281422' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:40:06.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:06 smithi192 ceph-mon[85304]: from='client.35514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:06.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:06 smithi192 ceph-mon[85304]: from='client.35518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:06.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:06 smithi192 ceph-mon[85304]: from='client.35522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:06.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:06 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/955710468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:06.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:06 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1227281422' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:40:07.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:07 smithi113 ceph-mon[103371]: pgmap v839: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:07.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:07 smithi113 ceph-mon[103371]: from='client.35534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:07.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:07 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1748791922' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:40:07.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:07.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:07 smithi192 ceph-mon[85304]: pgmap v839: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:07.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:07 smithi192 ceph-mon[85304]: from='client.35534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:07.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:07 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1748791922' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:40:07.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:09.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:09 smithi113 ceph-mon[103371]: pgmap v840: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:09.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:09 smithi192 ceph-mon[85304]: pgmap v840: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:11.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:11 smithi113 ceph-mon[103371]: pgmap v841: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:11.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:11 smithi192 ceph-mon[85304]: pgmap v841: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:13.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:13 smithi113 ceph-mon[103371]: pgmap v842: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:13.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:13 smithi192 ceph-mon[85304]: pgmap v842: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:15.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:15 smithi113 ceph-mon[103371]: pgmap v843: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:15.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:15 smithi192 ceph-mon[85304]: pgmap v843: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:17.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:17 smithi113 ceph-mon[103371]: pgmap v844: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:17.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:17 smithi192 ceph-mon[85304]: pgmap v844: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:19.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:19 smithi113 ceph-mon[103371]: pgmap v845: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:19.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:19 smithi192 ceph-mon[85304]: pgmap v845: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:21.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:21 smithi113 ceph-mon[103371]: pgmap v846: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:21.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:21 smithi192 ceph-mon[85304]: pgmap v846: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:22.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:22.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:23.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:23 smithi113 ceph-mon[103371]: pgmap v847: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:23.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:23 smithi192 ceph-mon[85304]: pgmap v847: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:25.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:25 smithi113 ceph-mon[103371]: pgmap v848: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:25.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:25 smithi192 ceph-mon[85304]: pgmap v848: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:26.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:26 smithi113 ceph-mon[103371]: pgmap v849: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:26.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:26 smithi192 ceph-mon[85304]: pgmap v849: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:29.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:28 smithi192 ceph-mon[85304]: pgmap v850: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:29.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:28 smithi113 ceph-mon[103371]: pgmap v850: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:31.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:30 smithi192 ceph-mon[85304]: pgmap v851: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:31.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:30 smithi113 ceph-mon[103371]: pgmap v851: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:33.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:32 smithi192 ceph-mon[85304]: pgmap v852: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:33.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:32 smithi113 ceph-mon[103371]: pgmap v852: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:35.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:34 smithi192 ceph-mon[85304]: pgmap v853: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:35.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:34 smithi113 ceph-mon[103371]: pgmap v853: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:36.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.633+0000 7f21eb804700 1 -- 172.21.15.113:0/2742483248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21e40fe2d0 msgr2=0x7f21e40f7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:36.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.633+0000 7f21eb804700 1 --2- 172.21.15.113:0/2742483248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21e40fe2d0 0x7f21e40f7ab0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f21e00047f0 tx=0x7f21e0009cb0).stop 2024-04-12T05:40:36.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.633+0000 7f21eb804700 1 -- 172.21.15.113:0/2742483248 shutdown_connections 2024-04-12T05:40:36.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.633+0000 7f21eb804700 1 --2- 172.21.15.113:0/2742483248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21e40fe2d0 0x7f21e40f7ab0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:36.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.633+0000 7f21eb804700 1 --2- 172.21.15.113:0/2742483248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21e40fd970 0x7f21e40fdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:36.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.633+0000 7f21eb804700 1 -- 172.21.15.113:0/2742483248 >> 172.21.15.113:0/2742483248 conn(0x7f21e40f3550 msgr2=0x7f21e40f5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:36.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.633+0000 7f21eb804700 1 -- 172.21.15.113:0/2742483248 shutdown_connections 2024-04-12T05:40:36.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.633+0000 7f21eb804700 1 -- 172.21.15.113:0/2742483248 wait complete. 2024-04-12T05:40:36.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21eb804700 1 Processor -- start 2024-04-12T05:40:36.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21eb804700 1 -- start start 2024-04-12T05:40:36.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21eb804700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21e40fd970 0x7f21e406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:36.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21eb804700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21e40fe2d0 0x7f21e406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:36.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21eb804700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e406bab0 con 0x7f21e40fd970 2024-04-12T05:40:36.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21eb804700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e406bbf0 con 0x7f21e40fe2d0 2024-04-12T05:40:36.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21e95a0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21e40fd970 0x7f21e406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21e95a0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21e40fd970 0x7f21e406b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33572/0 (socket says 172.21.15.113:33572) 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21e95a0700 1 -- 172.21.15.113:0/3620419678 learned_addr learned my addr 172.21.15.113:0/3620419678 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21e8d9f700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21e40fe2d0 0x7f21e406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21e95a0700 1 -- 172.21.15.113:0/3620419678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21e40fe2d0 msgr2=0x7f21e406b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21e95a0700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21e40fe2d0 0x7f21e406b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21e95a0700 1 -- 172.21.15.113:0/3620419678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e0005040 con 0x7f21e40fd970 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.634+0000 7f21e8d9f700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21e40fe2d0 0x7f21e406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.635+0000 7f21e95a0700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21e40fd970 0x7f21e406b0d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f21d40041d0 tx=0x7f21d4009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.635+0000 7f21da7fc700 1 -- 172.21.15.113:0/3620419678 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d4002d90 con 0x7f21e40fd970 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.635+0000 7f21da7fc700 1 -- 172.21.15.113:0/3620419678 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21d400f040 con 0x7f21e40fd970 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.635+0000 7f21eb804700 1 -- 172.21.15.113:0/3620419678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e406bea0 con 0x7f21e40fd970 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.635+0000 7f21da7fc700 1 -- 172.21.15.113:0/3620419678 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d400d800 con 0x7f21e40fd970 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.635+0000 7f21eb804700 1 -- 172.21.15.113:0/3620419678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e41146c0 con 0x7f21e40fd970 2024-04-12T05:40:36.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.636+0000 7f21eb804700 1 -- 172.21.15.113:0/3620419678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e4060370 con 0x7f21e40fd970 2024-04-12T05:40:36.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.636+0000 7f21da7fc700 1 -- 172.21.15.113:0/3620419678 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f21d400dae0 con 0x7f21e40fd970 2024-04-12T05:40:36.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.637+0000 7f21da7fc700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21d006caa0 0x7f21d006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:36.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.637+0000 7f21da7fc700 1 -- 172.21.15.113:0/3620419678 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f21d408bf40 con 0x7f21e40fd970 2024-04-12T05:40:36.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.637+0000 7f21e8d9f700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21d006caa0 0x7f21d006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:36.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.637+0000 7f21e8d9f700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21d006caa0 0x7f21d006ef60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f21e0004f10 tx=0x7f21e000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:36.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.641+0000 7f21da7fc700 1 -- 172.21.15.113:0/3620419678 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f21d4055ea0 con 0x7f21e40fd970 2024-04-12T05:40:36.844 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.846+0000 7f21eb804700 1 -- 172.21.15.113:0/3620419678 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21e4059120 con 0x7f21d006caa0 2024-04-12T05:40:36.845 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.847+0000 7f21da7fc700 1 -- 172.21.15.113:0/3620419678 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f21e4059120 con 0x7f21d006caa0 2024-04-12T05:40:36.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.849+0000 7f21eb804700 1 -- 172.21.15.113:0/3620419678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21d006caa0 msgr2=0x7f21d006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:36.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.849+0000 7f21eb804700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21d006caa0 0x7f21d006ef60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f21e0004f10 tx=0x7f21e000b040).stop 2024-04-12T05:40:36.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.849+0000 7f21eb804700 1 -- 172.21.15.113:0/3620419678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21e40fd970 msgr2=0x7f21e406b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:36.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.849+0000 7f21eb804700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21e40fd970 0x7f21e406b0d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f21d40041d0 tx=0x7f21d4009f60).stop 2024-04-12T05:40:36.847 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.849+0000 7f21eb804700 1 -- 172.21.15.113:0/3620419678 shutdown_connections 2024-04-12T05:40:36.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.849+0000 7f21eb804700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21d006caa0 0x7f21d006ef60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:36.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.849+0000 7f21eb804700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21e40fd970 0x7f21e406b0d0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:36.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.849+0000 7f21eb804700 1 --2- 172.21.15.113:0/3620419678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21e40fe2d0 0x7f21e406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:36.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.849+0000 7f21eb804700 1 -- 172.21.15.113:0/3620419678 >> 172.21.15.113:0/3620419678 conn(0x7f21e40f3550 msgr2=0x7f21e40fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:36.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.849+0000 7f21eb804700 1 -- 172.21.15.113:0/3620419678 shutdown_connections 2024-04-12T05:40:36.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.849+0000 7f21eb804700 1 -- 172.21.15.113:0/3620419678 wait complete. 2024-04-12T05:40:36.858 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:40:36.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.950+0000 7f4ea676a700 1 -- 172.21.15.113:0/905538347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ea00fe220 msgr2=0x7f4ea00f7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:36.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.950+0000 7f4ea676a700 1 --2- 172.21.15.113:0/905538347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ea00fe220 0x7f4ea00f7a00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f4e940047f0 tx=0x7f4e94009cb0).stop 2024-04-12T05:40:36.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.950+0000 7f4ea676a700 1 -- 172.21.15.113:0/905538347 shutdown_connections 2024-04-12T05:40:36.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.950+0000 7f4ea676a700 1 --2- 172.21.15.113:0/905538347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ea00fe220 0x7f4ea00f7a00 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:36.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.950+0000 7f4ea676a700 1 --2- 172.21.15.113:0/905538347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea00fd8c0 0x7f4ea00fdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:36.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.950+0000 7f4ea676a700 1 -- 172.21.15.113:0/905538347 >> 172.21.15.113:0/905538347 conn(0x7f4ea00f34a0 msgr2=0x7f4ea00f58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:36.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.950+0000 7f4ea676a700 1 -- 172.21.15.113:0/905538347 shutdown_connections 2024-04-12T05:40:36.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.950+0000 7f4ea676a700 1 -- 172.21.15.113:0/905538347 wait complete. 2024-04-12T05:40:36.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4ea676a700 1 Processor -- start 2024-04-12T05:40:36.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4ea676a700 1 -- start start 2024-04-12T05:40:36.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4ea676a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea00fd8c0 0x7f4ea006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:36.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4ea676a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ea00fe220 0x7f4ea006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:36.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4ea676a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea006bab0 con 0x7f4ea00fe220 2024-04-12T05:40:36.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4ea676a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea006bbf0 con 0x7f4ea00fd8c0 2024-04-12T05:40:36.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4e9f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ea00fe220 0x7f4ea006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:36.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4e9f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ea00fe220 0x7f4ea006b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33588/0 (socket says 172.21.15.113:33588) 2024-04-12T05:40:36.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4e9f7fe700 1 -- 172.21.15.113:0/4008102762 learned_addr learned my addr 172.21.15.113:0/4008102762 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:36.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4e9ffff700 1 --2- 172.21.15.113:0/4008102762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea00fd8c0 0x7f4ea006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:36.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4e9f7fe700 1 -- 172.21.15.113:0/4008102762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea00fd8c0 msgr2=0x7f4ea006b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:36.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4e9f7fe700 1 --2- 172.21.15.113:0/4008102762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea00fd8c0 0x7f4ea006b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:36.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.951+0000 7f4e9f7fe700 1 -- 172.21.15.113:0/4008102762 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e94005040 con 0x7f4ea00fe220 2024-04-12T05:40:36.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.952+0000 7f4e9f7fe700 1 --2- 172.21.15.113:0/4008102762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ea00fe220 0x7f4ea006b5c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f4e94000c00 tx=0x7f4e94009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:36.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.952+0000 7f4e9d7fa700 1 -- 172.21.15.113:0/4008102762 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e94018070 con 0x7f4ea00fe220 2024-04-12T05:40:36.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.952+0000 7f4ea676a700 1 -- 172.21.15.113:0/4008102762 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea006be40 con 0x7f4ea00fe220 2024-04-12T05:40:36.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.952+0000 7f4e9d7fa700 1 -- 172.21.15.113:0/4008102762 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e94007e90 con 0x7f4ea00fe220 2024-04-12T05:40:36.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.952+0000 7f4e9d7fa700 1 -- 172.21.15.113:0/4008102762 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e9401de20 con 0x7f4ea00fe220 2024-04-12T05:40:36.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.952+0000 7f4ea676a700 1 -- 172.21.15.113:0/4008102762 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea0114710 con 0x7f4ea00fe220 2024-04-12T05:40:36.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.953+0000 7f4ea676a700 1 -- 172.21.15.113:0/4008102762 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea0060390 con 0x7f4ea00fe220 2024-04-12T05:40:36.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.953+0000 7f4e9d7fa700 1 -- 172.21.15.113:0/4008102762 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4e9401d420 con 0x7f4ea00fe220 2024-04-12T05:40:36.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.954+0000 7f4e9d7fa700 1 --2- 172.21.15.113:0/4008102762 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e8806ca30 0x7f4e8806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:36.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.954+0000 7f4e9ffff700 1 --2- 172.21.15.113:0/4008102762 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e8806ca30 0x7f4e8806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:36.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.954+0000 7f4e9d7fa700 1 -- 172.21.15.113:0/4008102762 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f4e9408c990 con 0x7f4ea00fe220 2024-04-12T05:40:36.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.954+0000 7f4e9ffff700 1 --2- 172.21.15.113:0/4008102762 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e8806ca30 0x7f4e8806eef0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f4e900048b0 tx=0x7f4e9000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:36.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:36.958+0000 7f4e9d7fa700 1 -- 172.21.15.113:0/4008102762 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4e940569b0 con 0x7f4ea00fe220 2024-04-12T05:40:37.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.165+0000 7f4ea676a700 1 -- 172.21.15.113:0/4008102762 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ea0059120 con 0x7f4e8806ca30 2024-04-12T05:40:37.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.167+0000 7f4e9d7fa700 1 -- 172.21.15.113:0/4008102762 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4ea0059120 con 0x7f4e8806ca30 2024-04-12T05:40:37.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.169+0000 7f4ea676a700 1 -- 172.21.15.113:0/4008102762 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e8806ca30 msgr2=0x7f4e8806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.169+0000 7f4ea676a700 1 --2- 172.21.15.113:0/4008102762 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e8806ca30 0x7f4e8806eef0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f4e900048b0 tx=0x7f4e9000b040).stop 2024-04-12T05:40:37.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.169+0000 7f4ea676a700 1 -- 172.21.15.113:0/4008102762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ea00fe220 msgr2=0x7f4ea006b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.169+0000 7f4ea676a700 1 --2- 172.21.15.113:0/4008102762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ea00fe220 0x7f4ea006b5c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f4e94000c00 tx=0x7f4e94009cb0).stop 2024-04-12T05:40:37.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.169+0000 7f4ea676a700 1 -- 172.21.15.113:0/4008102762 shutdown_connections 2024-04-12T05:40:37.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.169+0000 7f4ea676a700 1 --2- 172.21.15.113:0/4008102762 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4e8806ca30 0x7f4e8806eef0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.169+0000 7f4ea676a700 1 --2- 172.21.15.113:0/4008102762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea00fd8c0 0x7f4ea006b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.169+0000 7f4ea676a700 1 --2- 172.21.15.113:0/4008102762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ea00fe220 0x7f4ea006b5c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.169+0000 7f4ea676a700 1 -- 172.21.15.113:0/4008102762 >> 172.21.15.113:0/4008102762 conn(0x7f4ea00f34a0 msgr2=0x7f4ea00fbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:37.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.169+0000 7f4ea676a700 1 -- 172.21.15.113:0/4008102762 shutdown_connections 2024-04-12T05:40:37.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.169+0000 7f4ea676a700 1 -- 172.21.15.113:0/4008102762 wait complete. 2024-04-12T05:40:37.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:36 smithi192 ceph-mon[85304]: pgmap v854: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:37.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:37.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.267+0000 7f8f6fa09700 1 -- 172.21.15.113:0/874626876 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f680fd150 msgr2=0x7f8f680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.267+0000 7f8f6fa09700 1 --2- 172.21.15.113:0/874626876 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f680fd150 0x7f8f680f6930 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7f8f640047f0 tx=0x7f8f64009cb0).stop 2024-04-12T05:40:37.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.267+0000 7f8f6fa09700 1 -- 172.21.15.113:0/874626876 shutdown_connections 2024-04-12T05:40:37.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.267+0000 7f8f6fa09700 1 --2- 172.21.15.113:0/874626876 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f680fd150 0x7f8f680f6930 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.267+0000 7f8f6fa09700 1 --2- 172.21.15.113:0/874626876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f680fc7f0 0x7f8f680fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.267+0000 7f8f6fa09700 1 -- 172.21.15.113:0/874626876 >> 172.21.15.113:0/874626876 conn(0x7f8f680f23b0 msgr2=0x7f8f680f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:37.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.267+0000 7f8f6fa09700 1 -- 172.21.15.113:0/874626876 shutdown_connections 2024-04-12T05:40:37.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.267+0000 7f8f6fa09700 1 -- 172.21.15.113:0/874626876 wait complete. 2024-04-12T05:40:37.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6fa09700 1 Processor -- start 2024-04-12T05:40:37.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6fa09700 1 -- start start 2024-04-12T05:40:37.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6fa09700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f680fc7f0 0x7f8f68109160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:37.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6fa09700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f680fd150 0x7f8f68109650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:37.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6fa09700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f68109b40 con 0x7f8f680fc7f0 2024-04-12T05:40:37.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6fa09700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f68109c80 con 0x7f8f680fd150 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6d7a5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f680fc7f0 0x7f8f68109160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6d7a5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f680fc7f0 0x7f8f68109160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33610/0 (socket says 172.21.15.113:33610) 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6d7a5700 1 -- 172.21.15.113:0/2783895723 learned_addr learned my addr 172.21.15.113:0/2783895723 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6cfa4700 1 --2- 172.21.15.113:0/2783895723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f680fd150 0x7f8f68109650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6d7a5700 1 -- 172.21.15.113:0/2783895723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f680fd150 msgr2=0x7f8f68109650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6d7a5700 1 --2- 172.21.15.113:0/2783895723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f680fd150 0x7f8f68109650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.268+0000 7f8f6d7a5700 1 -- 172.21.15.113:0/2783895723 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f64005040 con 0x7f8f680fc7f0 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.269+0000 7f8f6d7a5700 1 --2- 172.21.15.113:0/2783895723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f680fc7f0 0x7f8f68109160 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f8f580041d0 tx=0x7f8f58009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.269+0000 7f8f5e7fc700 1 -- 172.21.15.113:0/2783895723 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f58002d90 con 0x7f8f680fc7f0 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.269+0000 7f8f6fa09700 1 -- 172.21.15.113:0/2783895723 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f68109f30 con 0x7f8f680fc7f0 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.269+0000 7f8f5e7fc700 1 -- 172.21.15.113:0/2783895723 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f5800f040 con 0x7f8f680fc7f0 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.269+0000 7f8f5e7fc700 1 -- 172.21.15.113:0/2783895723 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f5800d800 con 0x7f8f680fc7f0 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.269+0000 7f8f6fa09700 1 -- 172.21.15.113:0/2783895723 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f681124a0 con 0x7f8f680fc7f0 2024-04-12T05:40:37.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.270+0000 7f8f6fa09700 1 -- 172.21.15.113:0/2783895723 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f68060390 con 0x7f8f680fc7f0 2024-04-12T05:40:37.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.270+0000 7f8f5e7fc700 1 -- 172.21.15.113:0/2783895723 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8f5800dac0 con 0x7f8f680fc7f0 2024-04-12T05:40:37.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.271+0000 7f8f5e7fc700 1 --2- 172.21.15.113:0/2783895723 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8f5406caa0 0x7f8f5406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:37.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.271+0000 7f8f5e7fc700 1 -- 172.21.15.113:0/2783895723 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f8f5808bea0 con 0x7f8f680fc7f0 2024-04-12T05:40:37.270 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.271+0000 7f8f6cfa4700 1 --2- 172.21.15.113:0/2783895723 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8f5406caa0 0x7f8f5406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:37.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.271+0000 7f8f6cfa4700 1 --2- 172.21.15.113:0/2783895723 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8f5406caa0 0x7f8f5406ef60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f8f64004f10 tx=0x7f8f6400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:37.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.275+0000 7f8f5e7fc700 1 -- 172.21.15.113:0/2783895723 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8f58055ec0 con 0x7f8f680fc7f0 2024-04-12T05:40:37.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:36 smithi113 ceph-mon[103371]: pgmap v854: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:37.807 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:37.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.474+0000 7f8f6fa09700 1 -- 172.21.15.113:0/2783895723 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f68059120 con 0x7f8f5406caa0 2024-04-12T05:40:37.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.483+0000 7f8f5e7fc700 1 -- 172.21.15.113:0/2783895723 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f8f68059120 con 0x7f8f5406caa0 2024-04-12T05:40:37.809 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:40:37.809 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (28m) 5m ago 39m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:40:37.809 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (25m) 5m ago 39m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:40:37.809 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (25m) 5m ago 39m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:40:37.809 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (28m) 5m ago 39m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:40:37.809 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (21m) 5m ago 34m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:40:37.809 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (21m) 5m ago 34m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:40:37.809 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (21m) 5m ago 34m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:40:37.809 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (5m) 5m ago 34m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:40:37.809 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (29m) 5m ago 40m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:40:37.809 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (29m) 5m ago 39m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:40:37.810 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (26m) 5m ago 41m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:40:37.810 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (26m) 5m ago 38m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:40:37.810 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (29m) 5m ago 39m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:40:37.810 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (28m) 5m ago 38m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:40:37.810 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (25m) 5m ago 38m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:40:37.810 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (24m) 5m ago 37m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:40:37.810 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (24m) 5m ago 37m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:40:37.810 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (23m) 5m ago 36m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:40:37.810 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (23m) 5m ago 36m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:40:37.810 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (22m) 5m ago 35m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:40:37.810 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (28m) 5m ago 38m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:40:37.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.485+0000 7f8f6fa09700 1 -- 172.21.15.113:0/2783895723 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8f5406caa0 msgr2=0x7f8f5406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.485+0000 7f8f6fa09700 1 --2- 172.21.15.113:0/2783895723 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8f5406caa0 0x7f8f5406ef60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f8f64004f10 tx=0x7f8f6400b040).stop 2024-04-12T05:40:37.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.485+0000 7f8f6fa09700 1 -- 172.21.15.113:0/2783895723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f680fc7f0 msgr2=0x7f8f68109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.485+0000 7f8f6fa09700 1 --2- 172.21.15.113:0/2783895723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f680fc7f0 0x7f8f68109160 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f8f580041d0 tx=0x7f8f58009f60).stop 2024-04-12T05:40:37.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.486+0000 7f8f6fa09700 1 -- 172.21.15.113:0/2783895723 shutdown_connections 2024-04-12T05:40:37.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.486+0000 7f8f6fa09700 1 --2- 172.21.15.113:0/2783895723 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8f5406caa0 0x7f8f5406ef60 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.486+0000 7f8f6fa09700 1 --2- 172.21.15.113:0/2783895723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f680fc7f0 0x7f8f68109160 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.486+0000 7f8f6fa09700 1 --2- 172.21.15.113:0/2783895723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f680fd150 0x7f8f68109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.486+0000 7f8f6fa09700 1 -- 172.21.15.113:0/2783895723 >> 172.21.15.113:0/2783895723 conn(0x7f8f680f23b0 msgr2=0x7f8f680fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.486+0000 7f8f6fa09700 1 -- 172.21.15.113:0/2783895723 shutdown_connections 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.486+0000 7f8f6fa09700 1 -- 172.21.15.113:0/2783895723 wait complete. 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.582+0000 7f003bbf4700 1 -- 172.21.15.113:0/3971725287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00340fd150 msgr2=0x7f00340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.582+0000 7f003bbf4700 1 --2- 172.21.15.113:0/3971725287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00340fd150 0x7f00340f6930 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7f00300047f0 tx=0x7f0030009cb0).stop 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.582+0000 7f003bbf4700 1 -- 172.21.15.113:0/3971725287 shutdown_connections 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.582+0000 7f003bbf4700 1 --2- 172.21.15.113:0/3971725287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00340fd150 0x7f00340f6930 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.582+0000 7f003bbf4700 1 --2- 172.21.15.113:0/3971725287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00340fc7f0 0x7f00340fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.582+0000 7f003bbf4700 1 -- 172.21.15.113:0/3971725287 >> 172.21.15.113:0/3971725287 conn(0x7f0034070590 msgr2=0x7f00340f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.582+0000 7f003bbf4700 1 -- 172.21.15.113:0/3971725287 shutdown_connections 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.582+0000 7f003bbf4700 1 -- 172.21.15.113:0/3971725287 wait complete. 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003bbf4700 1 Processor -- start 2024-04-12T05:40:37.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003bbf4700 1 -- start start 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003bbf4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00340fc7f0 0x7f00341101f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003bbf4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00340fd150 0x7f00341106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003bbf4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0034110ce0 con 0x7f00340fd150 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003bbf4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0034110e20 con 0x7f00340fc7f0 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003918f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00340fd150 0x7f00341106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003918f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00340fd150 0x7f00341106e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33636/0 (socket says 172.21.15.113:33636) 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003918f700 1 -- 172.21.15.113:0/2546216114 learned_addr learned my addr 172.21.15.113:0/2546216114 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003918f700 1 -- 172.21.15.113:0/2546216114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00340fc7f0 msgr2=0x7f00341101f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003918f700 1 --2- 172.21.15.113:0/2546216114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00340fc7f0 0x7f00341101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003918f700 1 -- 172.21.15.113:0/2546216114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0030005040 con 0x7f00340fd150 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003918f700 1 --2- 172.21.15.113:0/2546216114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00340fd150 0x7f00341106e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7f00300099f0 tx=0x7f0030009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f002affd700 1 -- 172.21.15.113:0/2546216114 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f003000e030 con 0x7f00340fd150 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003bbf4700 1 -- 172.21.15.113:0/2546216114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003410a450 con 0x7f00340fd150 2024-04-12T05:40:37.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f002affd700 1 -- 172.21.15.113:0/2546216114 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0030007e90 con 0x7f00340fd150 2024-04-12T05:40:37.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f002affd700 1 -- 172.21.15.113:0/2546216114 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0030018070 con 0x7f00340fd150 2024-04-12T05:40:37.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.583+0000 7f003bbf4700 1 -- 172.21.15.113:0/2546216114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003410a9b0 con 0x7f00340fd150 2024-04-12T05:40:37.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.584+0000 7f003bbf4700 1 -- 172.21.15.113:0/2546216114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003405f590 con 0x7f00340fd150 2024-04-12T05:40:37.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.585+0000 7f002affd700 1 -- 172.21.15.113:0/2546216114 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0030009590 con 0x7f00340fd150 2024-04-12T05:40:37.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.585+0000 7f002affd700 1 --2- 172.21.15.113:0/2546216114 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f002006ca30 0x7f002006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:37.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.585+0000 7f002affd700 1 -- 172.21.15.113:0/2546216114 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f003008d0f0 con 0x7f00340fd150 2024-04-12T05:40:37.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.585+0000 7f0039990700 1 --2- 172.21.15.113:0/2546216114 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f002006ca30 0x7f002006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:37.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.586+0000 7f0039990700 1 --2- 172.21.15.113:0/2546216114 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f002006ca30 0x7f002006eef0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f0024004b90 tx=0x7f0024009b60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:37.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.590+0000 7f002affd700 1 -- 172.21.15.113:0/2546216114 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f003005aba0 con 0x7f00340fd150 2024-04-12T05:40:37.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.856+0000 7f003bbf4700 1 -- 172.21.15.113:0/2546216114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0034047540 con 0x7f00340fd150 2024-04-12T05:40:37.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.857+0000 7f002affd700 1 -- 172.21.15.113:0/2546216114 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f003005a730 con 0x7f00340fd150 2024-04-12T05:40:37.855 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:40:37.855 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:40:37.855 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:40:37.855 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:40:37.855 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:40:37.855 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:40:37.855 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:40:37.855 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:40:37.855 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:40:37.855 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:40:37.856 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:40:37.856 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:40:37.856 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:40:37.856 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:40:37.856 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:40:37.856 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:40:37.856 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:40:37.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.859+0000 7f003bbf4700 1 -- 172.21.15.113:0/2546216114 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f002006ca30 msgr2=0x7f002006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.859+0000 7f003bbf4700 1 --2- 172.21.15.113:0/2546216114 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f002006ca30 0x7f002006eef0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f0024004b90 tx=0x7f0024009b60).stop 2024-04-12T05:40:37.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.859+0000 7f003bbf4700 1 -- 172.21.15.113:0/2546216114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00340fd150 msgr2=0x7f00341106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.859+0000 7f003bbf4700 1 --2- 172.21.15.113:0/2546216114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00340fd150 0x7f00341106e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7f00300099f0 tx=0x7f0030009cb0).stop 2024-04-12T05:40:37.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.859+0000 7f003bbf4700 1 -- 172.21.15.113:0/2546216114 shutdown_connections 2024-04-12T05:40:37.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.859+0000 7f003bbf4700 1 --2- 172.21.15.113:0/2546216114 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f002006ca30 0x7f002006eef0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.859+0000 7f003bbf4700 1 --2- 172.21.15.113:0/2546216114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00340fc7f0 0x7f00341101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:40:37.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.859+0000 7f003bbf4700 1 --2- 172.21.15.113:0/2546216114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00340fd150 0x7f00341106e0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.859+0000 7f003bbf4700 1 -- 172.21.15.113:0/2546216114 >> 172.21.15.113:0/2546216114 conn(0x7f0034070590 msgr2=0x7f0034060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:37.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.859+0000 7f003bbf4700 1 -- 172.21.15.113:0/2546216114 shutdown_connections 2024-04-12T05:40:37.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.859+0000 7f003bbf4700 1 -- 172.21.15.113:0/2546216114 wait complete. 2024-04-12T05:40:37.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.957+0000 7f194174f700 1 -- 172.21.15.113:0/3436081819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f193c0f88a0 msgr2=0x7f193c0f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.957+0000 7f194174f700 1 --2- 172.21.15.113:0/3436081819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f193c0f88a0 0x7f193c0f8c80 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f192c0047f0 tx=0x7f192c009cb0).stop 2024-04-12T05:40:37.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.957+0000 7f194174f700 1 -- 172.21.15.113:0/3436081819 shutdown_connections 2024-04-12T05:40:37.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.957+0000 7f194174f700 1 --2- 172.21.15.113:0/3436081819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f193c0f9200 0x7f193c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.957+0000 7f194174f700 1 --2- 172.21.15.113:0/3436081819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f193c0f88a0 0x7f193c0f8c80 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.957+0000 7f194174f700 1 -- 172.21.15.113:0/3436081819 >> 172.21.15.113:0/3436081819 conn(0x7f193c0f23b0 msgr2=0x7f193c0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:37.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.957+0000 7f194174f700 1 -- 172.21.15.113:0/3436081819 shutdown_connections 2024-04-12T05:40:37.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.957+0000 7f194174f700 1 -- 172.21.15.113:0/3436081819 wait complete. 2024-04-12T05:40:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.958+0000 7f194174f700 1 Processor -- start 2024-04-12T05:40:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.958+0000 7f194174f700 1 -- start start 2024-04-12T05:40:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.958+0000 7f194174f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f193c0f88a0 0x7f193c10e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.958+0000 7f194174f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f193c0f9200 0x7f193c10e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.958+0000 7f194174f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c10eb90 con 0x7f193c0f88a0 2024-04-12T05:40:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.958+0000 7f194174f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c10ecd0 con 0x7f193c0f9200 2024-04-12T05:40:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.958+0000 7f193affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f193c0f88a0 0x7f193c10e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.958+0000 7f193affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f193c0f88a0 0x7f193c10e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33656/0 (socket says 172.21.15.113:33656) 2024-04-12T05:40:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.958+0000 7f193affd700 1 -- 172.21.15.113:0/3483932091 learned_addr learned my addr 172.21.15.113:0/3483932091 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.958+0000 7f193a7fc700 1 --2- 172.21.15.113:0/3483932091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f193c0f9200 0x7f193c10e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:37.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.959+0000 7f193affd700 1 -- 172.21.15.113:0/3483932091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f193c0f9200 msgr2=0x7f193c10e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:37.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.959+0000 7f193affd700 1 --2- 172.21.15.113:0/3483932091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f193c0f9200 0x7f193c10e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:37.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.959+0000 7f193affd700 1 -- 172.21.15.113:0/3483932091 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c005040 con 0x7f193c0f88a0 2024-04-12T05:40:37.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.959+0000 7f193affd700 1 --2- 172.21.15.113:0/3483932091 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f193c0f88a0 0x7f193c10e0a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f192c0043a0 tx=0x7f192c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:37.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.959+0000 7f1923fff700 1 -- 172.21.15.113:0/3483932091 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f192c00e030 con 0x7f193c0f88a0 2024-04-12T05:40:37.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.959+0000 7f194174f700 1 -- 172.21.15.113:0/3483932091 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c108350 con 0x7f193c0f88a0 2024-04-12T05:40:37.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.959+0000 7f1923fff700 1 -- 172.21.15.113:0/3483932091 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f192c007e90 con 0x7f193c0f88a0 2024-04-12T05:40:37.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.959+0000 7f1923fff700 1 -- 172.21.15.113:0/3483932091 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f192c018070 con 0x7f193c0f88a0 2024-04-12T05:40:37.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.959+0000 7f194174f700 1 -- 172.21.15.113:0/3483932091 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c108830 con 0x7f193c0f88a0 2024-04-12T05:40:37.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.960+0000 7f194174f700 1 -- 172.21.15.113:0/3483932091 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193c060370 con 0x7f193c0f88a0 2024-04-12T05:40:37.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.960+0000 7f1923fff700 1 -- 172.21.15.113:0/3483932091 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f192c0094c0 con 0x7f193c0f88a0 2024-04-12T05:40:37.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.961+0000 7f1923fff700 1 --2- 172.21.15.113:0/3483932091 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f192406cab0 0x7f192406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:37.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.961+0000 7f1923fff700 1 -- 172.21.15.113:0/3483932091 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f192c08d0a0 con 0x7f193c0f88a0 2024-04-12T05:40:37.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.961+0000 7f193a7fc700 1 --2- 172.21.15.113:0/3483932091 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f192406cab0 0x7f192406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:37.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.961+0000 7f193a7fc700 1 --2- 172.21.15.113:0/3483932091 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f192406cab0 0x7f192406ef70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f193c1099e0 tx=0x7f193000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:37.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:37.965+0000 7f1923fff700 1 -- 172.21.15.113:0/3483932091 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f192c05ab50 con 0x7f193c0f88a0 2024-04-12T05:40:38.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.195+0000 7f194174f700 1 -- 172.21.15.113:0/3483932091 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f193c109820 con 0x7f193c0f88a0 2024-04-12T05:40:38.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.196+0000 7f1923fff700 1 -- 172.21.15.113:0/3483932091 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f192c05a6e0 con 0x7f193c0f88a0 2024-04-12T05:40:38.194 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:40:38.194 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:40:38.194 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:40:38.195 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:40:38.196 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:40:38.197 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:40:38.197 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:40:38.197 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:40:38.197 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:40:38.197 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:40:38.197 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:40:38.197 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:40:38.197 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:40:38.197 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:40:38.197 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:40:38.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.198+0000 7f194174f700 1 -- 172.21.15.113:0/3483932091 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f192406cab0 msgr2=0x7f192406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:38.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.198+0000 7f194174f700 1 --2- 172.21.15.113:0/3483932091 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f192406cab0 0x7f192406ef70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f193c1099e0 tx=0x7f193000b040).stop 2024-04-12T05:40:38.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.198+0000 7f194174f700 1 -- 172.21.15.113:0/3483932091 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f193c0f88a0 msgr2=0x7f193c10e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:38.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.198+0000 7f194174f700 1 --2- 172.21.15.113:0/3483932091 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f193c0f88a0 0x7f193c10e0a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f192c0043a0 tx=0x7f192c009cb0).stop 2024-04-12T05:40:38.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.198+0000 7f194174f700 1 -- 172.21.15.113:0/3483932091 shutdown_connections 2024-04-12T05:40:38.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.198+0000 7f194174f700 1 --2- 172.21.15.113:0/3483932091 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f192406cab0 0x7f192406ef70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.198+0000 7f194174f700 1 --2- 172.21.15.113:0/3483932091 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f193c0f88a0 0x7f193c10e0a0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.198+0000 7f194174f700 1 --2- 172.21.15.113:0/3483932091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f193c0f9200 0x7f193c10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.198+0000 7f194174f700 1 -- 172.21.15.113:0/3483932091 >> 172.21.15.113:0/3483932091 conn(0x7f193c0f23b0 msgr2=0x7f193c0fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:38.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.198+0000 7f194174f700 1 -- 172.21.15.113:0/3483932091 shutdown_connections 2024-04-12T05:40:38.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.198+0000 7f194174f700 1 -- 172.21.15.113:0/3483932091 wait complete. 2024-04-12T05:40:38.199 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:40:38.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.298+0000 7f0e2c674700 1 -- 172.21.15.113:0/4226404259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e240fd910 msgr2=0x7f0e240fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.298+0000 7f0e2c674700 1 --2- 172.21.15.113:0/4226404259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e240fd910 0x7f0e240fdcf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f0e140047f0 tx=0x7f0e14009cb0).stop 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.298+0000 7f0e2c674700 1 -- 172.21.15.113:0/4226404259 shutdown_connections 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.298+0000 7f0e2c674700 1 --2- 172.21.15.113:0/4226404259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e240fe270 0x7f0e240f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.298+0000 7f0e2c674700 1 --2- 172.21.15.113:0/4226404259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e240fd910 0x7f0e240fdcf0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.298+0000 7f0e2c674700 1 -- 172.21.15.113:0/4226404259 >> 172.21.15.113:0/4226404259 conn(0x7f0e240f3530 msgr2=0x7f0e240f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.298+0000 7f0e2c674700 1 -- 172.21.15.113:0/4226404259 shutdown_connections 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.298+0000 7f0e2c674700 1 -- 172.21.15.113:0/4226404259 wait complete. 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.298+0000 7f0e2c674700 1 Processor -- start 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.298+0000 7f0e2c674700 1 -- start start 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2c674700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e240fd910 0x7f0e2410e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2c674700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e240fe270 0x7f0e2410e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:38.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2c674700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2410eba0 con 0x7f0e240fd910 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2c674700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2410ece0 con 0x7f0e240fe270 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2a410700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e240fd910 0x7f0e2410e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2a410700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e240fd910 0x7f0e2410e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33662/0 (socket says 172.21.15.113:33662) 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2a410700 1 -- 172.21.15.113:0/3776286660 learned_addr learned my addr 172.21.15.113:0/3776286660 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e29c0f700 1 --2- 172.21.15.113:0/3776286660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e240fe270 0x7f0e2410e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2a410700 1 -- 172.21.15.113:0/3776286660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e240fe270 msgr2=0x7f0e2410e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2a410700 1 --2- 172.21.15.113:0/3776286660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e240fe270 0x7f0e2410e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2a410700 1 -- 172.21.15.113:0/3776286660 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e14005040 con 0x7f0e240fd910 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2a410700 1 --2- 172.21.15.113:0/3776286660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e240fd910 0x7f0e2410e0b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7f0e1400b120 tx=0x7f0e14009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e1b7fe700 1 -- 172.21.15.113:0/3776286660 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e1400e030 con 0x7f0e240fd910 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e1b7fe700 1 -- 172.21.15.113:0/3776286660 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e14007e90 con 0x7f0e240fd910 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2c674700 1 -- 172.21.15.113:0/3776286660 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e24108310 con 0x7f0e240fd910 2024-04-12T05:40:38.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e1b7fe700 1 -- 172.21.15.113:0/3776286660 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e14018070 con 0x7f0e240fd910 2024-04-12T05:40:38.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.299+0000 7f0e2c674700 1 -- 172.21.15.113:0/3776286660 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e241087f0 con 0x7f0e240fd910 2024-04-12T05:40:38.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.300+0000 7f0e2c674700 1 -- 172.21.15.113:0/3776286660 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2405f570 con 0x7f0e240fd910 2024-04-12T05:40:38.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.301+0000 7f0e1b7fe700 1 -- 172.21.15.113:0/3776286660 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0e1401d420 con 0x7f0e240fd910 2024-04-12T05:40:38.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.301+0000 7f0e1b7fe700 1 --2- 172.21.15.113:0/3776286660 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0e1006caa0 0x7f0e1006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:38.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.301+0000 7f0e1b7fe700 1 -- 172.21.15.113:0/3776286660 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0e1408d300 con 0x7f0e240fd910 2024-04-12T05:40:38.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.301+0000 7f0e29c0f700 1 --2- 172.21.15.113:0/3776286660 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0e1006caa0 0x7f0e1006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:38.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.301+0000 7f0e29c0f700 1 --2- 172.21.15.113:0/3776286660 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0e1006caa0 0x7f0e1006ef60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f0e241099f0 tx=0x7f0e20009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:38.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.306+0000 7f0e1b7fe700 1 -- 172.21.15.113:0/3776286660 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0e1405adb0 con 0x7f0e240fd910 2024-04-12T05:40:38.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:37 smithi113 ceph-mon[103371]: from='client.35542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:38.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:37 smithi113 ceph-mon[103371]: from='client.35546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:38.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:37 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2546216114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:38.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:37 smithi192 ceph-mon[85304]: from='client.35542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:38.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:37 smithi192 ceph-mon[85304]: from='client.35546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:38.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:37 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2546216114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:40:38.511 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.513+0000 7f0e2c674700 1 -- 172.21.15.113:0/3776286660 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e240ffc60 con 0x7f0e1006caa0 2024-04-12T05:40:38.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.514+0000 7f0e1b7fe700 1 -- 172.21.15.113:0/3776286660 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0e240ffc60 con 0x7f0e1006caa0 2024-04-12T05:40:38.512 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:40:38.512 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:40:38.512 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:40:38.513 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:40:38.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.516+0000 7f0e2c674700 1 -- 172.21.15.113:0/3776286660 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0e1006caa0 msgr2=0x7f0e1006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:38.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.516+0000 7f0e2c674700 1 --2- 172.21.15.113:0/3776286660 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0e1006caa0 0x7f0e1006ef60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f0e241099f0 tx=0x7f0e20009040).stop 2024-04-12T05:40:38.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.516+0000 7f0e2c674700 1 -- 172.21.15.113:0/3776286660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e240fd910 msgr2=0x7f0e2410e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:38.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.516+0000 7f0e2c674700 1 --2- 172.21.15.113:0/3776286660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e240fd910 0x7f0e2410e0b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7f0e1400b120 tx=0x7f0e14009cb0).stop 2024-04-12T05:40:38.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.516+0000 7f0e2c674700 1 -- 172.21.15.113:0/3776286660 shutdown_connections 2024-04-12T05:40:38.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.516+0000 7f0e2c674700 1 --2- 172.21.15.113:0/3776286660 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0e1006caa0 0x7f0e1006ef60 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.516+0000 7f0e2c674700 1 --2- 172.21.15.113:0/3776286660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0e240fd910 0x7f0e2410e0b0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.516+0000 7f0e2c674700 1 --2- 172.21.15.113:0/3776286660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e240fe270 0x7f0e2410e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.516+0000 7f0e2c674700 1 -- 172.21.15.113:0/3776286660 >> 172.21.15.113:0/3776286660 conn(0x7f0e240f3530 msgr2=0x7f0e240fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:38.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.516+0000 7f0e2c674700 1 -- 172.21.15.113:0/3776286660 shutdown_connections 2024-04-12T05:40:38.515 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.516+0000 7f0e2c674700 1 -- 172.21.15.113:0/3776286660 wait complete. 2024-04-12T05:40:38.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.614+0000 7fd2a5f36700 1 -- 172.21.15.113:0/1801280374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2a00fc7f0 msgr2=0x7fd2a00fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:38.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.614+0000 7fd2a5f36700 1 --2- 172.21.15.113:0/1801280374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2a00fc7f0 0x7fd2a00fcbd0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7fd2900047f0 tx=0x7fd290009cb0).stop 2024-04-12T05:40:38.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.614+0000 7fd2a5f36700 1 -- 172.21.15.113:0/1801280374 shutdown_connections 2024-04-12T05:40:38.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.614+0000 7fd2a5f36700 1 --2- 172.21.15.113:0/1801280374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a00fd150 0x7fd2a00f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.614+0000 7fd2a5f36700 1 --2- 172.21.15.113:0/1801280374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2a00fc7f0 0x7fd2a00fcbd0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.614+0000 7fd2a5f36700 1 -- 172.21.15.113:0/1801280374 >> 172.21.15.113:0/1801280374 conn(0x7fd2a0070590 msgr2=0x7fd2a00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:38.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.614+0000 7fd2a5f36700 1 -- 172.21.15.113:0/1801280374 shutdown_connections 2024-04-12T05:40:38.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.614+0000 7fd2a5f36700 1 -- 172.21.15.113:0/1801280374 wait complete. 2024-04-12T05:40:38.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.614+0000 7fd2a5f36700 1 Processor -- start 2024-04-12T05:40:38.614 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.614+0000 7fd2a5f36700 1 -- start start 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd2a5f36700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a00fc7f0 0x7fd2a010e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd2a5f36700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2a00fd150 0x7fd2a010e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd2a5f36700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a010ec10 con 0x7fd2a00fd150 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd2a5f36700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a010ed50 con 0x7fd2a00fc7f0 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2a00fd150 0x7fd2a010e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2a00fd150 0x7fd2a010e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33684/0 (socket says 172.21.15.113:33684) 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29effd700 1 -- 172.21.15.113:0/3921794810 learned_addr learned my addr 172.21.15.113:0/3921794810 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29f7fe700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a00fc7f0 0x7fd2a010e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29effd700 1 -- 172.21.15.113:0/3921794810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a00fc7f0 msgr2=0x7fd2a010e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29effd700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a00fc7f0 0x7fd2a010e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29effd700 1 -- 172.21.15.113:0/3921794810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd290005040 con 0x7fd2a00fd150 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29f7fe700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a00fc7f0 0x7fd2a010e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29effd700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2a00fd150 0x7fd2a010e610 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7fd29400e940 tx=0x7fd29400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:38.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29cff9700 1 -- 172.21.15.113:0/3921794810 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd29400ee80 con 0x7fd2a00fd150 2024-04-12T05:40:38.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd2a5f36700 1 -- 172.21.15.113:0/3921794810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a01083c0 con 0x7fd2a00fd150 2024-04-12T05:40:38.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29cff9700 1 -- 172.21.15.113:0/3921794810 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd29400f040 con 0x7fd2a00fd150 2024-04-12T05:40:38.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd29cff9700 1 -- 172.21.15.113:0/3921794810 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd29400c810 con 0x7fd2a00fd150 2024-04-12T05:40:38.616 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.615+0000 7fd2a5f36700 1 -- 172.21.15.113:0/3921794810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a0108980 con 0x7fd2a00fd150 2024-04-12T05:40:38.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.616+0000 7fd2a5f36700 1 -- 172.21.15.113:0/3921794810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a005f590 con 0x7fd2a00fd150 2024-04-12T05:40:38.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.617+0000 7fd29cff9700 1 -- 172.21.15.113:0/3921794810 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd2940090d0 con 0x7fd2a00fd150 2024-04-12T05:40:38.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.617+0000 7fd29cff9700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd28806ca30 0x7fd28806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:40:38.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.617+0000 7fd29cff9700 1 -- 172.21.15.113:0/3921794810 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd29408b970 con 0x7fd2a00fd150 2024-04-12T05:40:38.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.617+0000 7fd29f7fe700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd28806ca30 0x7fd28806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:40:38.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.618+0000 7fd29f7fe700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd28806ca30 0x7fd28806eef0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd290004540 tx=0x7fd29000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:40:38.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.622+0000 7fd29cff9700 1 -- 172.21.15.113:0/3921794810 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd294055990 con 0x7fd2a00fd150 2024-04-12T05:40:38.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.887+0000 7fd2a5f36700 1 -- 172.21.15.113:0/3921794810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd2a0047540 con 0x7fd2a00fd150 2024-04-12T05:40:38.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.887+0000 7fd29cff9700 1 -- 172.21.15.113:0/3921794810 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fd2940093d0 con 0x7fd2a00fd150 2024-04-12T05:40:38.886 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:40:38.886 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:40:38.886 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:40:38.886 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:40:38.886 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:40:38.886 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:40:38.886 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:40:38.886 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:40:38.886 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:40:38.887 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:40:38.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.889+0000 7fd2a5f36700 1 -- 172.21.15.113:0/3921794810 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd28806ca30 msgr2=0x7fd28806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:38.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.889+0000 7fd2a5f36700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd28806ca30 0x7fd28806eef0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd290004540 tx=0x7fd29000b040).stop 2024-04-12T05:40:38.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.889+0000 7fd2a5f36700 1 -- 172.21.15.113:0/3921794810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2a00fd150 msgr2=0x7fd2a010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:40:38.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.889+0000 7fd2a5f36700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2a00fd150 0x7fd2a010e610 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7fd29400e940 tx=0x7fd29400b040).stop 2024-04-12T05:40:38.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.889+0000 7fd2a5f36700 1 -- 172.21.15.113:0/3921794810 shutdown_connections 2024-04-12T05:40:38.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.889+0000 7fd2a5f36700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd28806ca30 0x7fd28806eef0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.889+0000 7fd2a5f36700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2a00fc7f0 0x7fd2a010e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.889+0000 7fd2a5f36700 1 --2- 172.21.15.113:0/3921794810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2a00fd150 0x7fd2a010e610 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:40:38.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.889+0000 7fd2a5f36700 1 -- 172.21.15.113:0/3921794810 >> 172.21.15.113:0/3921794810 conn(0x7fd2a0070590 msgr2=0x7fd2a0060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:40:38.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.890+0000 7fd2a5f36700 1 -- 172.21.15.113:0/3921794810 shutdown_connections 2024-04-12T05:40:38.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:40:38.890+0000 7fd2a5f36700 1 -- 172.21.15.113:0/3921794810 wait complete. 2024-04-12T05:40:39.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:38 smithi113 ceph-mon[103371]: from='client.35550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:39.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:38 smithi113 ceph-mon[103371]: pgmap v855: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:39.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:38 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3483932091' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:40:39.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:38 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3921794810' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:40:39.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:38 smithi192 ceph-mon[85304]: from='client.35550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:39.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:38 smithi192 ceph-mon[85304]: pgmap v855: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:39.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:38 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3483932091' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:40:39.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:38 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3921794810' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:40:40.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:39 smithi113 ceph-mon[103371]: from='client.35562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:40.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:39 smithi192 ceph-mon[85304]: from='client.35562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:40:41.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:40 smithi113 ceph-mon[103371]: pgmap v856: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:41.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:40 smithi192 ceph-mon[85304]: pgmap v856: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:43.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:42 smithi113 ceph-mon[103371]: pgmap v857: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:43.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:42 smithi192 ceph-mon[85304]: pgmap v857: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:45.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:44 smithi113 ceph-mon[103371]: pgmap v858: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:45.445 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:44 smithi192 ceph-mon[85304]: pgmap v858: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:47.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:46 smithi113 ceph-mon[103371]: pgmap v859: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:47.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:46 smithi192 ceph-mon[85304]: pgmap v859: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:49.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:48 smithi113 ceph-mon[103371]: pgmap v860: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:49.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:48 smithi192 ceph-mon[85304]: pgmap v860: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:51.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:50 smithi113 ceph-mon[103371]: pgmap v861: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:51.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:50 smithi192 ceph-mon[85304]: pgmap v861: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:52.265 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:51 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:52.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:51 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:40:53.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:52 smithi113 ceph-mon[103371]: pgmap v862: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:53.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:52 smithi192 ceph-mon[85304]: pgmap v862: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:55.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:54 smithi113 ceph-mon[103371]: pgmap v863: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:55.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:54 smithi192 ceph-mon[85304]: pgmap v863: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:57.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:56 smithi113 ceph-mon[103371]: pgmap v864: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:57.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:56 smithi192 ceph-mon[85304]: pgmap v864: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:59.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:40:58 smithi113 ceph-mon[103371]: pgmap v865: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:40:59.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:40:58 smithi192 ceph-mon[85304]: pgmap v865: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:01.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:00 smithi113 ceph-mon[103371]: pgmap v866: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:01.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:00 smithi192 ceph-mon[85304]: pgmap v866: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:03.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:03 smithi113 ceph-mon[103371]: pgmap v867: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:03.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:03 smithi192 ceph-mon[85304]: pgmap v867: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:05.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:05 smithi113 ceph-mon[103371]: pgmap v868: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:05.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:05 smithi192 ceph-mon[85304]: pgmap v868: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:07.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:07 smithi113 ceph-mon[103371]: pgmap v869: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:07.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:07.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:07 smithi192 ceph-mon[85304]: pgmap v869: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:07.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:08.999 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.002+0000 7fb0a0930700 1 -- 172.21.15.113:0/4111888885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb09c0fd8f0 msgr2=0x7fb09c0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.002+0000 7fb0a0930700 1 --2- 172.21.15.113:0/4111888885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb09c0fd8f0 0x7fb09c0fdcd0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7fb0900047f0 tx=0x7fb090009cb0).stop 2024-04-12T05:41:09.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.002+0000 7fb0a0930700 1 -- 172.21.15.113:0/4111888885 shutdown_connections 2024-04-12T05:41:09.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.002+0000 7fb0a0930700 1 --2- 172.21.15.113:0/4111888885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb09c0fe250 0x7fb09c0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.002+0000 7fb0a0930700 1 --2- 172.21.15.113:0/4111888885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb09c0fd8f0 0x7fb09c0fdcd0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.002+0000 7fb0a0930700 1 -- 172.21.15.113:0/4111888885 >> 172.21.15.113:0/4111888885 conn(0x7fb09c0f34b0 msgr2=0x7fb09c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:09.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.002+0000 7fb0a0930700 1 -- 172.21.15.113:0/4111888885 shutdown_connections 2024-04-12T05:41:09.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.002+0000 7fb0a0930700 1 -- 172.21.15.113:0/4111888885 wait complete. 2024-04-12T05:41:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.003+0000 7fb0a0930700 1 Processor -- start 2024-04-12T05:41:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.003+0000 7fb0a0930700 1 -- start start 2024-04-12T05:41:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.003+0000 7fb0a0930700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb09c0fd8f0 0x7fb09c10dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.003+0000 7fb0a0930700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb09c0fe250 0x7fb09c10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.003+0000 7fb0a0930700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c10ead0 con 0x7fb09c0fd8f0 2024-04-12T05:41:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.003+0000 7fb0a0930700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c10ec10 con 0x7fb09c0fe250 2024-04-12T05:41:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.003+0000 7fb09a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb09c0fd8f0 0x7fb09c10dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.003+0000 7fb09a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb09c0fd8f0 0x7fb09c10dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57286/0 (socket says 172.21.15.113:57286) 2024-04-12T05:41:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.003+0000 7fb09a59c700 1 -- 172.21.15.113:0/3154529766 learned_addr learned my addr 172.21.15.113:0/3154529766 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:09.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.004+0000 7fb099d9b700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb09c0fe250 0x7fb09c10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.004+0000 7fb09a59c700 1 -- 172.21.15.113:0/3154529766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb09c0fe250 msgr2=0x7fb09c10e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.004+0000 7fb09a59c700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb09c0fe250 0x7fb09c10e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.004+0000 7fb09a59c700 1 -- 172.21.15.113:0/3154529766 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb090005040 con 0x7fb09c0fd8f0 2024-04-12T05:41:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.004+0000 7fb099d9b700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb09c0fe250 0x7fb09c10e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:41:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.004+0000 7fb09a59c700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb09c0fd8f0 0x7fb09c10dfe0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7fb09000b120 tx=0x7fb090009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.004+0000 7fb08b7fe700 1 -- 172.21.15.113:0/3154529766 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb09000e030 con 0x7fb09c0fd8f0 2024-04-12T05:41:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.004+0000 7fb08b7fe700 1 -- 172.21.15.113:0/3154529766 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb090007e90 con 0x7fb09c0fd8f0 2024-04-12T05:41:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.004+0000 7fb0a0930700 1 -- 172.21.15.113:0/3154529766 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09c108240 con 0x7fb09c0fd8f0 2024-04-12T05:41:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.004+0000 7fb08b7fe700 1 -- 172.21.15.113:0/3154529766 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb090018070 con 0x7fb09c0fd8f0 2024-04-12T05:41:09.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.004+0000 7fb0a0930700 1 -- 172.21.15.113:0/3154529766 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09c1087a0 con 0x7fb09c0fd8f0 2024-04-12T05:41:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.005+0000 7fb0a0930700 1 -- 172.21.15.113:0/3154529766 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09c05f590 con 0x7fb09c0fd8f0 2024-04-12T05:41:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.006+0000 7fb08b7fe700 1 -- 172.21.15.113:0/3154529766 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb09001d420 con 0x7fb09c0fd8f0 2024-04-12T05:41:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.006+0000 7fb08b7fe700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb08406caa0 0x7fb08406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.006+0000 7fb08b7fe700 1 -- 172.21.15.113:0/3154529766 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb09008d300 con 0x7fb09c0fd8f0 2024-04-12T05:41:09.004 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.006+0000 7fb099d9b700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb08406caa0 0x7fb08406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:09.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.006+0000 7fb099d9b700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb08406caa0 0x7fb08406ef60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb09c109920 tx=0x7fb08c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:09.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.011+0000 7fb08b7fe700 1 -- 172.21.15.113:0/3154529766 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb09005adb0 con 0x7fb09c0fd8f0 2024-04-12T05:41:09.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.217+0000 7fb0a0930700 1 -- 172.21.15.113:0/3154529766 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb09c109770 con 0x7fb08406caa0 2024-04-12T05:41:09.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.218+0000 7fb08b7fe700 1 -- 172.21.15.113:0/3154529766 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb09c109770 con 0x7fb08406caa0 2024-04-12T05:41:09.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.220+0000 7fb0a0930700 1 -- 172.21.15.113:0/3154529766 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb08406caa0 msgr2=0x7fb08406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.220+0000 7fb0a0930700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb08406caa0 0x7fb08406ef60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb09c109920 tx=0x7fb08c009040).stop 2024-04-12T05:41:09.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.220+0000 7fb0a0930700 1 -- 172.21.15.113:0/3154529766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb09c0fd8f0 msgr2=0x7fb09c10dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.220+0000 7fb0a0930700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb09c0fd8f0 0x7fb09c10dfe0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7fb09000b120 tx=0x7fb090009cb0).stop 2024-04-12T05:41:09.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.220+0000 7fb0a0930700 1 -- 172.21.15.113:0/3154529766 shutdown_connections 2024-04-12T05:41:09.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.220+0000 7fb0a0930700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb08406caa0 0x7fb08406ef60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.220+0000 7fb0a0930700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb09c0fd8f0 0x7fb09c10dfe0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.220+0000 7fb0a0930700 1 --2- 172.21.15.113:0/3154529766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb09c0fe250 0x7fb09c10e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.220+0000 7fb0a0930700 1 -- 172.21.15.113:0/3154529766 >> 172.21.15.113:0/3154529766 conn(0x7fb09c0f34b0 msgr2=0x7fb09c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:09.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.220+0000 7fb0a0930700 1 -- 172.21.15.113:0/3154529766 shutdown_connections 2024-04-12T05:41:09.218 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.220+0000 7fb0a0930700 1 -- 172.21.15.113:0/3154529766 wait complete. 2024-04-12T05:41:09.227 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:41:09.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.319+0000 7fd5faad4700 1 -- 172.21.15.113:0/1843856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f40fc7f0 msgr2=0x7fd5f40fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.319+0000 7fd5faad4700 1 --2- 172.21.15.113:0/1843856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f40fc7f0 0x7fd5f40fcbd0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7fd5e80047f0 tx=0x7fd5e8009cb0).stop 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.319+0000 7fd5faad4700 1 -- 172.21.15.113:0/1843856 shutdown_connections 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.319+0000 7fd5faad4700 1 --2- 172.21.15.113:0/1843856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5f40fd150 0x7fd5f40f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.319+0000 7fd5faad4700 1 --2- 172.21.15.113:0/1843856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f40fc7f0 0x7fd5f40fcbd0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.319+0000 7fd5faad4700 1 -- 172.21.15.113:0/1843856 >> 172.21.15.113:0/1843856 conn(0x7fd5f4070590 msgr2=0x7fd5f40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.319+0000 7fd5faad4700 1 -- 172.21.15.113:0/1843856 shutdown_connections 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.319+0000 7fd5faad4700 1 -- 172.21.15.113:0/1843856 wait complete. 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.319+0000 7fd5faad4700 1 Processor -- start 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5faad4700 1 -- start start 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5faad4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5f40fc7f0 0x7fd5f410cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5faad4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f40fd150 0x7fd5f410d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5faad4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f410d9f0 con 0x7fd5f40fd150 2024-04-12T05:41:09.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5faad4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f410db30 con 0x7fd5f40fc7f0 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5f3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f40fd150 0x7fd5f410d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5f3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f40fd150 0x7fd5f410d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57304/0 (socket says 172.21.15.113:57304) 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5f3fff700 1 -- 172.21.15.113:0/1779161721 learned_addr learned my addr 172.21.15.113:0/1779161721 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5f8870700 1 --2- 172.21.15.113:0/1779161721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5f40fc7f0 0x7fd5f410cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5f3fff700 1 -- 172.21.15.113:0/1779161721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5f40fc7f0 msgr2=0x7fd5f410cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5f3fff700 1 --2- 172.21.15.113:0/1779161721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5f40fc7f0 0x7fd5f410cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5f3fff700 1 -- 172.21.15.113:0/1779161721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e8005040 con 0x7fd5f40fd150 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5f3fff700 1 --2- 172.21.15.113:0/1779161721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f40fd150 0x7fd5f410d3f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7fd5e00044e0 tx=0x7fd5e0002a50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5f1f76700 1 -- 172.21.15.113:0/1779161721 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5e0007b40 con 0x7fd5f40fd150 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5f1f76700 1 -- 172.21.15.113:0/1779161721 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5e000f040 con 0x7fd5f40fd150 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5faad4700 1 -- 172.21.15.113:0/1779161721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f41071c0 con 0x7fd5f40fd150 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.320+0000 7fd5f1f76700 1 -- 172.21.15.113:0/1779161721 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5e0014850 con 0x7fd5f40fd150 2024-04-12T05:41:09.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.321+0000 7fd5faad4700 1 -- 172.21.15.113:0/1779161721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f4107700 con 0x7fd5f40fd150 2024-04-12T05:41:09.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.321+0000 7fd5faad4700 1 -- 172.21.15.113:0/1779161721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f405f590 con 0x7fd5f40fd150 2024-04-12T05:41:09.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.322+0000 7fd5f1f76700 1 -- 172.21.15.113:0/1779161721 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd5e0009110 con 0x7fd5f40fd150 2024-04-12T05:41:09.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.322+0000 7fd5f1f76700 1 --2- 172.21.15.113:0/1779161721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd5e406caa0 0x7fd5e406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.322+0000 7fd5f8870700 1 --2- 172.21.15.113:0/1779161721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd5e406caa0 0x7fd5e406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:09.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.322+0000 7fd5f1f76700 1 -- 172.21.15.113:0/1779161721 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd5e008b8c0 con 0x7fd5f40fd150 2024-04-12T05:41:09.321 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.323+0000 7fd5f8870700 1 --2- 172.21.15.113:0/1779161721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd5e406caa0 0x7fd5e406ef60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fd5e8004f10 tx=0x7fd5e800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:09.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.327+0000 7fd5f1f76700 1 -- 172.21.15.113:0/1779161721 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd5e0059370 con 0x7fd5f40fd150 2024-04-12T05:41:09.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:09 smithi113 ceph-mon[103371]: pgmap v870: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:09.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:09 smithi192 ceph-mon[85304]: pgmap v870: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:09.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.533+0000 7fd5faad4700 1 -- 172.21.15.113:0/1779161721 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5f41079d0 con 0x7fd5e406caa0 2024-04-12T05:41:09.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.534+0000 7fd5f1f76700 1 -- 172.21.15.113:0/1779161721 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd5f41079d0 con 0x7fd5e406caa0 2024-04-12T05:41:09.533 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.536+0000 7fd5faad4700 1 -- 172.21.15.113:0/1779161721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd5e406caa0 msgr2=0x7fd5e406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.536+0000 7fd5faad4700 1 --2- 172.21.15.113:0/1779161721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd5e406caa0 0x7fd5e406ef60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fd5e8004f10 tx=0x7fd5e800b040).stop 2024-04-12T05:41:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.536+0000 7fd5faad4700 1 -- 172.21.15.113:0/1779161721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f40fd150 msgr2=0x7fd5f410d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.536+0000 7fd5faad4700 1 --2- 172.21.15.113:0/1779161721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f40fd150 0x7fd5f410d3f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7fd5e00044e0 tx=0x7fd5e0002a50).stop 2024-04-12T05:41:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.536+0000 7fd5faad4700 1 -- 172.21.15.113:0/1779161721 shutdown_connections 2024-04-12T05:41:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.536+0000 7fd5faad4700 1 --2- 172.21.15.113:0/1779161721 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd5e406caa0 0x7fd5e406ef60 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.536+0000 7fd5faad4700 1 --2- 172.21.15.113:0/1779161721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5f40fc7f0 0x7fd5f410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.536+0000 7fd5faad4700 1 --2- 172.21.15.113:0/1779161721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f40fd150 0x7fd5f410d3f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.536+0000 7fd5faad4700 1 -- 172.21.15.113:0/1779161721 >> 172.21.15.113:0/1779161721 conn(0x7fd5f4070590 msgr2=0x7fd5f40fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.536+0000 7fd5faad4700 1 -- 172.21.15.113:0/1779161721 shutdown_connections 2024-04-12T05:41:09.534 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.536+0000 7fd5faad4700 1 -- 172.21.15.113:0/1779161721 wait complete. 2024-04-12T05:41:09.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.634+0000 7f4f8e279700 1 -- 172.21.15.113:0/313107908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f880fe250 msgr2=0x7f4f880f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.634+0000 7f4f8e279700 1 --2- 172.21.15.113:0/313107908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f880fe250 0x7f4f880f7a30 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7f4f7c0047f0 tx=0x7f4f7c009cb0).stop 2024-04-12T05:41:09.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.635+0000 7f4f8e279700 1 -- 172.21.15.113:0/313107908 shutdown_connections 2024-04-12T05:41:09.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.635+0000 7f4f8e279700 1 --2- 172.21.15.113:0/313107908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f880fe250 0x7f4f880f7a30 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.635+0000 7f4f8e279700 1 --2- 172.21.15.113:0/313107908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f880fd8f0 0x7f4f880fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.635+0000 7f4f8e279700 1 -- 172.21.15.113:0/313107908 >> 172.21.15.113:0/313107908 conn(0x7f4f880f34b0 msgr2=0x7f4f880f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:09.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.635+0000 7f4f8e279700 1 -- 172.21.15.113:0/313107908 shutdown_connections 2024-04-12T05:41:09.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.635+0000 7f4f8e279700 1 -- 172.21.15.113:0/313107908 wait complete. 2024-04-12T05:41:09.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.635+0000 7f4f8e279700 1 Processor -- start 2024-04-12T05:41:09.633 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.635+0000 7f4f8e279700 1 -- start start 2024-04-12T05:41:09.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f8e279700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f880fd8f0 0x7f4f8810e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f8e279700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f880fe250 0x7f4f8810e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f8e279700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8810ec30 con 0x7f4f880fe250 2024-04-12T05:41:09.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f8e279700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8810ed70 con 0x7f4f880fd8f0 2024-04-12T05:41:09.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f877fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f880fe250 0x7f4f8810e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:09.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f877fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f880fe250 0x7f4f8810e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57326/0 (socket says 172.21.15.113:57326) 2024-04-12T05:41:09.635 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f877fe700 1 -- 172.21.15.113:0/4210056257 learned_addr learned my addr 172.21.15.113:0/4210056257 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f87fff700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f880fd8f0 0x7f4f8810e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f877fe700 1 -- 172.21.15.113:0/4210056257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f880fd8f0 msgr2=0x7f4f8810e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f877fe700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f880fd8f0 0x7f4f8810e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f877fe700 1 -- 172.21.15.113:0/4210056257 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c005040 con 0x7f4f880fe250 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f87fff700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f880fd8f0 0x7f4f8810e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f877fe700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f880fe250 0x7f4f8810e630 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f4f7c000c00 tx=0x7f4f7c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f85775700 1 -- 172.21.15.113:0/4210056257 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f7c018070 con 0x7f4f880fe250 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f85775700 1 -- 172.21.15.113:0/4210056257 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f7c007e90 con 0x7f4f880fe250 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f8e279700 1 -- 172.21.15.113:0/4210056257 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f881083a0 con 0x7f4f880fe250 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f85775700 1 -- 172.21.15.113:0/4210056257 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f7c0123c0 con 0x7f4f880fe250 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.636+0000 7f4f8e279700 1 -- 172.21.15.113:0/4210056257 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f88108900 con 0x7f4f880fe250 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.637+0000 7f4f8e279700 1 -- 172.21.15.113:0/4210056257 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8805f590 con 0x7f4f880fe250 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.638+0000 7f4f85775700 1 -- 172.21.15.113:0/4210056257 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4f7c01d420 con 0x7f4f880fe250 2024-04-12T05:41:09.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.638+0000 7f4f85775700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4f7006caa0 0x7f4f7006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.638+0000 7f4f85775700 1 -- 172.21.15.113:0/4210056257 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f4f7c08caf0 con 0x7f4f880fe250 2024-04-12T05:41:09.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.638+0000 7f4f87fff700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4f7006caa0 0x7f4f7006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:09.637 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.638+0000 7f4f87fff700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4f7006caa0 0x7f4f7006ef60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4f78007fd0 tx=0x7f4f7800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:09.639 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.642+0000 7f4f85775700 1 -- 172.21.15.113:0/4210056257 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4f7c05a5a0 con 0x7f4f880fe250 2024-04-12T05:41:09.836 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.839+0000 7f4f8e279700 1 -- 172.21.15.113:0/4210056257 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f880ffc40 con 0x7f4f7006caa0 2024-04-12T05:41:09.846 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.849+0000 7f4f85775700 1 -- 172.21.15.113:0/4210056257 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4f880ffc40 con 0x7f4f7006caa0 2024-04-12T05:41:09.846 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:41:09.846 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (29m) 6m ago 40m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:41:09.846 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (26m) 6m ago 40m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:41:09.846 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (26m) 6m ago 39m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:41:09.846 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (28m) 6m ago 39m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:41:09.846 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (22m) 6m ago 34m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:41:09.846 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (21m) 6m ago 34m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:41:09.846 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (21m) 6m ago 34m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:41:09.846 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (6m) 6m ago 34m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (30m) 6m ago 41m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (29m) 6m ago 39m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (27m) 6m ago 41m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (27m) 6m ago 39m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (29m) 6m ago 39m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (29m) 6m ago 39m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (25m) 6m ago 38m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (25m) 6m ago 38m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (24m) 6m ago 37m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (24m) 6m ago 37m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (23m) 6m ago 36m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (22m) 6m ago 36m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:41:09.847 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (29m) 6m ago 39m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:41:09.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.851+0000 7f4f8e279700 1 -- 172.21.15.113:0/4210056257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4f7006caa0 msgr2=0x7f4f7006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.851+0000 7f4f8e279700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4f7006caa0 0x7f4f7006ef60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4f78007fd0 tx=0x7f4f7800b040).stop 2024-04-12T05:41:09.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.851+0000 7f4f8e279700 1 -- 172.21.15.113:0/4210056257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f880fe250 msgr2=0x7f4f8810e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.851+0000 7f4f8e279700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f880fe250 0x7f4f8810e630 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f4f7c000c00 tx=0x7f4f7c009cb0).stop 2024-04-12T05:41:09.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.851+0000 7f4f8e279700 1 -- 172.21.15.113:0/4210056257 shutdown_connections 2024-04-12T05:41:09.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.851+0000 7f4f8e279700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4f7006caa0 0x7f4f7006ef60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.851+0000 7f4f8e279700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f880fd8f0 0x7f4f8810e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.851+0000 7f4f8e279700 1 --2- 172.21.15.113:0/4210056257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f880fe250 0x7f4f8810e630 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.851+0000 7f4f8e279700 1 -- 172.21.15.113:0/4210056257 >> 172.21.15.113:0/4210056257 conn(0x7f4f880f34b0 msgr2=0x7f4f880fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:09.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.851+0000 7f4f8e279700 1 -- 172.21.15.113:0/4210056257 shutdown_connections 2024-04-12T05:41:09.849 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.851+0000 7f4f8e279700 1 -- 172.21.15.113:0/4210056257 wait complete. 2024-04-12T05:41:09.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.948+0000 7fddf8e1a700 1 -- 172.21.15.113:0/423819662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddf40fe250 msgr2=0x7fddf40f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.948+0000 7fddf8e1a700 1 --2- 172.21.15.113:0/423819662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddf40fe250 0x7fddf40f7a30 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7fdde40047f0 tx=0x7fdde4009cb0).stop 2024-04-12T05:41:09.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.948+0000 7fddf8e1a700 1 -- 172.21.15.113:0/423819662 shutdown_connections 2024-04-12T05:41:09.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.948+0000 7fddf8e1a700 1 --2- 172.21.15.113:0/423819662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddf40fe250 0x7fddf40f7a30 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.948+0000 7fddf8e1a700 1 --2- 172.21.15.113:0/423819662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf40fd8f0 0x7fddf40fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:09.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.948+0000 7fddf8e1a700 1 -- 172.21.15.113:0/423819662 >> 172.21.15.113:0/423819662 conn(0x7fddf40f34b0 msgr2=0x7fddf40f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:09.945 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.948+0000 7fddf8e1a700 1 -- 172.21.15.113:0/423819662 shutdown_connections 2024-04-12T05:41:09.946 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.948+0000 7fddf8e1a700 1 -- 172.21.15.113:0/423819662 wait complete. 2024-04-12T05:41:09.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.948+0000 7fddf8e1a700 1 Processor -- start 2024-04-12T05:41:09.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf8e1a700 1 -- start start 2024-04-12T05:41:09.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf8e1a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf40fd8f0 0x7fddf406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf8e1a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddf40fe250 0x7fddf406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf8e1a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf406bab0 con 0x7fddf40fe250 2024-04-12T05:41:09.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf8e1a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf406bbf0 con 0x7fddf40fd8f0 2024-04-12T05:41:09.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf1d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddf40fe250 0x7fddf406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:09.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf1d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddf40fe250 0x7fddf406b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57334/0 (socket says 172.21.15.113:57334) 2024-04-12T05:41:09.947 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf1d9b700 1 -- 172.21.15.113:0/1459484808 learned_addr learned my addr 172.21.15.113:0/1459484808 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:09.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf1d9b700 1 -- 172.21.15.113:0/1459484808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf40fd8f0 msgr2=0x7fddf406b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:09.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf1d9b700 1 --2- 172.21.15.113:0/1459484808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf40fd8f0 0x7fddf406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:41:09.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf1d9b700 1 -- 172.21.15.113:0/1459484808 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde4005040 con 0x7fddf40fe250 2024-04-12T05:41:09.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf1d9b700 1 --2- 172.21.15.113:0/1459484808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddf40fe250 0x7fddf406b5c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7fdde40099f0 tx=0x7fdde4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:09.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fdde37fe700 1 -- 172.21.15.113:0/1459484808 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdde400e030 con 0x7fddf40fe250 2024-04-12T05:41:09.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf8e1a700 1 -- 172.21.15.113:0/1459484808 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf406be40 con 0x7fddf40fe250 2024-04-12T05:41:09.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fdde37fe700 1 -- 172.21.15.113:0/1459484808 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdde4007e90 con 0x7fddf40fe250 2024-04-12T05:41:09.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fdde37fe700 1 -- 172.21.15.113:0/1459484808 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdde4018070 con 0x7fddf40fe250 2024-04-12T05:41:09.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.949+0000 7fddf8e1a700 1 -- 172.21.15.113:0/1459484808 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf4114690 con 0x7fddf40fe250 2024-04-12T05:41:09.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.950+0000 7fddf8e1a700 1 -- 172.21.15.113:0/1459484808 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf405f590 con 0x7fddf40fe250 2024-04-12T05:41:09.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.951+0000 7fdde37fe700 1 -- 172.21.15.113:0/1459484808 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdde4009590 con 0x7fddf40fe250 2024-04-12T05:41:09.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.951+0000 7fdde37fe700 1 --2- 172.21.15.113:0/1459484808 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdddc06ca30 0x7fdddc06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:09.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.951+0000 7fdde37fe700 1 -- 172.21.15.113:0/1459484808 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fdde408d130 con 0x7fddf40fe250 2024-04-12T05:41:09.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.951+0000 7fddf259c700 1 --2- 172.21.15.113:0/1459484808 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdddc06ca30 0x7fdddc06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:09.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.952+0000 7fddf259c700 1 --2- 172.21.15.113:0/1459484808 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdddc06ca30 0x7fdddc06eef0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fdde8004b90 tx=0x7fdde8009b60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:09.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:09.956+0000 7fdde37fe700 1 -- 172.21.15.113:0/1459484808 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fdde4057150 con 0x7fddf40fe250 2024-04-12T05:41:10.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.222+0000 7fddf8e1a700 1 -- 172.21.15.113:0/1459484808 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fddf40ffc80 con 0x7fddf40fe250 2024-04-12T05:41:10.219 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.222+0000 7fdde37fe700 1 -- 172.21.15.113:0/1459484808 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fdde405a770 con 0x7fddf40fe250 2024-04-12T05:41:10.220 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:41:10.220 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:41:10.220 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:41:10.220 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:41:10.220 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:41:10.220 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:41:10.220 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:41:10.220 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:41:10.220 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:41:10.221 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:41:10.221 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:41:10.221 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:41:10.221 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:41:10.221 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:41:10.221 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:41:10.221 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:41:10.221 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:41:10.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.224+0000 7fddf8e1a700 1 -- 172.21.15.113:0/1459484808 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdddc06ca30 msgr2=0x7fdddc06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.224+0000 7fddf8e1a700 1 --2- 172.21.15.113:0/1459484808 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdddc06ca30 0x7fdddc06eef0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fdde8004b90 tx=0x7fdde8009b60).stop 2024-04-12T05:41:10.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.224+0000 7fddf8e1a700 1 -- 172.21.15.113:0/1459484808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddf40fe250 msgr2=0x7fddf406b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.224+0000 7fddf8e1a700 1 --2- 172.21.15.113:0/1459484808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddf40fe250 0x7fddf406b5c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7fdde40099f0 tx=0x7fdde4009cb0).stop 2024-04-12T05:41:10.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.224+0000 7fddf8e1a700 1 -- 172.21.15.113:0/1459484808 shutdown_connections 2024-04-12T05:41:10.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.224+0000 7fddf8e1a700 1 --2- 172.21.15.113:0/1459484808 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdddc06ca30 0x7fdddc06eef0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.224+0000 7fddf8e1a700 1 --2- 172.21.15.113:0/1459484808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf40fd8f0 0x7fddf406b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:41:10.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.224+0000 7fddf8e1a700 1 --2- 172.21.15.113:0/1459484808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddf40fe250 0x7fddf406b5c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.224+0000 7fddf8e1a700 1 -- 172.21.15.113:0/1459484808 >> 172.21.15.113:0/1459484808 conn(0x7fddf40f34b0 msgr2=0x7fddf40fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:10.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.225+0000 7fddf8e1a700 1 -- 172.21.15.113:0/1459484808 shutdown_connections 2024-04-12T05:41:10.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.225+0000 7fddf8e1a700 1 -- 172.21.15.113:0/1459484808 wait complete. 2024-04-12T05:41:10.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.322+0000 7fcdc10f0700 1 -- 172.21.15.113:0/1891900158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdbc0f8940 msgr2=0x7fcdbc0f8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.322+0000 7fcdc10f0700 1 --2- 172.21.15.113:0/1891900158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdbc0f8940 0x7fcdbc0f8d20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7fcda40047f0 tx=0x7fcda4009cb0).stop 2024-04-12T05:41:10.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.322+0000 7fcdc10f0700 1 -- 172.21.15.113:0/1891900158 shutdown_connections 2024-04-12T05:41:10.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.322+0000 7fcdc10f0700 1 --2- 172.21.15.113:0/1891900158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdbc0f9210 0x7fcdbc1019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.322+0000 7fcdc10f0700 1 --2- 172.21.15.113:0/1891900158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdbc0f8940 0x7fcdbc0f8d20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.322+0000 7fcdc10f0700 1 -- 172.21.15.113:0/1891900158 >> 172.21.15.113:0/1891900158 conn(0x7fcdbc070590 msgr2=0x7fcdbc0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:10.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.322+0000 7fcdc10f0700 1 -- 172.21.15.113:0/1891900158 shutdown_connections 2024-04-12T05:41:10.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.322+0000 7fcdc10f0700 1 -- 172.21.15.113:0/1891900158 wait complete. 2024-04-12T05:41:10.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.323+0000 7fcdc10f0700 1 Processor -- start 2024-04-12T05:41:10.320 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.323+0000 7fcdc10f0700 1 -- start start 2024-04-12T05:41:10.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.323+0000 7fcdc10f0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdbc0f8940 0x7fcdbc10e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:10.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.323+0000 7fcdc10f0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdbc0f9210 0x7fcdbc10e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:10.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.323+0000 7fcdc10f0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc10eb40 con 0x7fcdbc0f9210 2024-04-12T05:41:10.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.323+0000 7fcdc10f0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc10ec80 con 0x7fcdbc0f8940 2024-04-12T05:41:10.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.323+0000 7fcdba59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdbc0f9210 0x7fcdbc10e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:10.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.323+0000 7fcdba59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdbc0f9210 0x7fcdbc10e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56096/0 (socket says 172.21.15.113:56096) 2024-04-12T05:41:10.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.323+0000 7fcdba59c700 1 -- 172.21.15.113:0/241257854 learned_addr learned my addr 172.21.15.113:0/241257854 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:10.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.324+0000 7fcdbad9d700 1 --2- 172.21.15.113:0/241257854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdbc0f8940 0x7fcdbc10e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:10.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.324+0000 7fcdba59c700 1 -- 172.21.15.113:0/241257854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdbc0f8940 msgr2=0x7fcdbc10e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.324+0000 7fcdba59c700 1 --2- 172.21.15.113:0/241257854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdbc0f8940 0x7fcdbc10e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.324+0000 7fcdba59c700 1 -- 172.21.15.113:0/241257854 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda4005040 con 0x7fcdbc0f9210 2024-04-12T05:41:10.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.324+0000 7fcdba59c700 1 --2- 172.21.15.113:0/241257854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdbc0f9210 0x7fcdbc10e540 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7fcdac0041d0 tx=0x7fcdac009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:10.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.324+0000 7fcdb3fff700 1 -- 172.21.15.113:0/241257854 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdac007b40 con 0x7fcdbc0f9210 2024-04-12T05:41:10.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.324+0000 7fcdb3fff700 1 -- 172.21.15.113:0/241257854 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdac00f040 con 0x7fcdbc0f9210 2024-04-12T05:41:10.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.324+0000 7fcdc10f0700 1 -- 172.21.15.113:0/241257854 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdbc108310 con 0x7fcdbc0f9210 2024-04-12T05:41:10.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.324+0000 7fcdb3fff700 1 -- 172.21.15.113:0/241257854 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdac00d800 con 0x7fcdbc0f9210 2024-04-12T05:41:10.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.324+0000 7fcdc10f0700 1 -- 172.21.15.113:0/241257854 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc108850 con 0x7fcdbc0f9210 2024-04-12T05:41:10.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.325+0000 7fcdc10f0700 1 -- 172.21.15.113:0/241257854 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdbc05f590 con 0x7fcdbc0f9210 2024-04-12T05:41:10.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.326+0000 7fcdb3fff700 1 -- 172.21.15.113:0/241257854 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcdac007ca0 con 0x7fcdbc0f9210 2024-04-12T05:41:10.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.326+0000 7fcdb3fff700 1 --2- 172.21.15.113:0/241257854 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcda806caa0 0x7fcda806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:10.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.326+0000 7fcdb3fff700 1 -- 172.21.15.113:0/241257854 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fcdac08b910 con 0x7fcdbc0f9210 2024-04-12T05:41:10.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.326+0000 7fcdbad9d700 1 --2- 172.21.15.113:0/241257854 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcda806caa0 0x7fcda806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:10.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.326+0000 7fcdbad9d700 1 --2- 172.21.15.113:0/241257854 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcda806caa0 0x7fcda806ef60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fcda4004f10 tx=0x7fcda400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:10.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.330+0000 7fcdb3fff700 1 -- 172.21.15.113:0/241257854 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fcdac0593c0 con 0x7fcdbc0f9210 2024-04-12T05:41:10.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:10 smithi113 ceph-mon[103371]: from='client.35570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:10.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:10 smithi192 ceph-mon[85304]: from='client.35570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:10.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.561+0000 7fcdc10f0700 1 -- 172.21.15.113:0/241257854 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcdbc047540 con 0x7fcdbc0f9210 2024-04-12T05:41:10.558 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.561+0000 7fcdb3fff700 1 -- 172.21.15.113:0/241257854 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fcdac058f50 con 0x7fcdbc0f9210 2024-04-12T05:41:10.559 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:41:10.559 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:41:10.559 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:41:10.559 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:41:10.559 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:41:10.559 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:41:10.559 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:41:10.559 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:41:10.559 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:41:10.559 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:41:10.560 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:10.561 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:10.562 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:10.562 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:10.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.563+0000 7fcdc10f0700 1 -- 172.21.15.113:0/241257854 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcda806caa0 msgr2=0x7fcda806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.562 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.563+0000 7fcdc10f0700 1 --2- 172.21.15.113:0/241257854 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcda806caa0 0x7fcda806ef60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fcda4004f10 tx=0x7fcda400b040).stop 2024-04-12T05:41:10.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.563+0000 7fcdc10f0700 1 -- 172.21.15.113:0/241257854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdbc0f9210 msgr2=0x7fcdbc10e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.563+0000 7fcdc10f0700 1 --2- 172.21.15.113:0/241257854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdbc0f9210 0x7fcdbc10e540 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7fcdac0041d0 tx=0x7fcdac009f60).stop 2024-04-12T05:41:10.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.564+0000 7fcdc10f0700 1 -- 172.21.15.113:0/241257854 shutdown_connections 2024-04-12T05:41:10.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.564+0000 7fcdc10f0700 1 --2- 172.21.15.113:0/241257854 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcda806caa0 0x7fcda806ef60 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.564+0000 7fcdc10f0700 1 --2- 172.21.15.113:0/241257854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdbc0f8940 0x7fcdbc10e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.564+0000 7fcdc10f0700 1 --2- 172.21.15.113:0/241257854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdbc0f9210 0x7fcdbc10e540 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.564+0000 7fcdc10f0700 1 -- 172.21.15.113:0/241257854 >> 172.21.15.113:0/241257854 conn(0x7fcdbc070590 msgr2=0x7fcdbc0fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:10.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.564+0000 7fcdc10f0700 1 -- 172.21.15.113:0/241257854 shutdown_connections 2024-04-12T05:41:10.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.564+0000 7fcdc10f0700 1 -- 172.21.15.113:0/241257854 wait complete. 2024-04-12T05:41:10.563 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:41:10.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.663+0000 7f1b6d529700 1 -- 172.21.15.113:0/274346703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b680fc7f0 msgr2=0x7f1b680fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.663+0000 7f1b6d529700 1 --2- 172.21.15.113:0/274346703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b680fc7f0 0x7f1b680fcbd0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f1b5c0047f0 tx=0x7f1b5c009cb0).stop 2024-04-12T05:41:10.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.663+0000 7f1b6d529700 1 -- 172.21.15.113:0/274346703 shutdown_connections 2024-04-12T05:41:10.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.663+0000 7f1b6d529700 1 --2- 172.21.15.113:0/274346703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b680fd150 0x7f1b680f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.663+0000 7f1b6d529700 1 --2- 172.21.15.113:0/274346703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b680fc7f0 0x7f1b680fcbd0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.663+0000 7f1b6d529700 1 -- 172.21.15.113:0/274346703 >> 172.21.15.113:0/274346703 conn(0x7f1b68070590 msgr2=0x7f1b680f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:10.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.663+0000 7f1b6d529700 1 -- 172.21.15.113:0/274346703 shutdown_connections 2024-04-12T05:41:10.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.663+0000 7f1b6d529700 1 -- 172.21.15.113:0/274346703 wait complete. 2024-04-12T05:41:10.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.663+0000 7f1b6d529700 1 Processor -- start 2024-04-12T05:41:10.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b6d529700 1 -- start start 2024-04-12T05:41:10.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b6d529700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b680fc7f0 0x7f1b680f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b6d529700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b680fd150 0x7f1b680f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b6d529700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b680f8f20 con 0x7f1b680fc7f0 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b6d529700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b680f9060 con 0x7f1b680fd150 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b66ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b680fc7f0 0x7f1b680f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b66ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b680fc7f0 0x7f1b680f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56122/0 (socket says 172.21.15.113:56122) 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b66ffd700 1 -- 172.21.15.113:0/3011307758 learned_addr learned my addr 172.21.15.113:0/3011307758 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b66ffd700 1 -- 172.21.15.113:0/3011307758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b680fd150 msgr2=0x7f1b680f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b66ffd700 1 --2- 172.21.15.113:0/3011307758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b680fd150 0x7f1b680f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b66ffd700 1 -- 172.21.15.113:0/3011307758 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c005040 con 0x7f1b680fc7f0 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b66ffd700 1 --2- 172.21.15.113:0/3011307758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b680fc7f0 0x7f1b680f8540 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f1b5c00b120 tx=0x7f1b5c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b4ffff700 1 -- 172.21.15.113:0/3011307758 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b5c00e030 con 0x7f1b680fc7f0 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b6d529700 1 -- 172.21.15.113:0/3011307758 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b680f92b0 con 0x7f1b680fc7f0 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b4ffff700 1 -- 172.21.15.113:0/3011307758 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b5c007e90 con 0x7f1b680fc7f0 2024-04-12T05:41:10.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b4ffff700 1 -- 172.21.15.113:0/3011307758 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b5c018070 con 0x7f1b680fc7f0 2024-04-12T05:41:10.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.664+0000 7f1b6d529700 1 -- 172.21.15.113:0/3011307758 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b681145d0 con 0x7f1b680fc7f0 2024-04-12T05:41:10.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.665+0000 7f1b6d529700 1 -- 172.21.15.113:0/3011307758 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6805f590 con 0x7f1b680fc7f0 2024-04-12T05:41:10.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.666+0000 7f1b4ffff700 1 -- 172.21.15.113:0/3011307758 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1b5c01d420 con 0x7f1b680fc7f0 2024-04-12T05:41:10.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.666+0000 7f1b4ffff700 1 --2- 172.21.15.113:0/3011307758 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b5006caa0 0x7f1b5006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:10.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.666+0000 7f1b4ffff700 1 -- 172.21.15.113:0/3011307758 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f1b5c08d3c0 con 0x7f1b680fc7f0 2024-04-12T05:41:10.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.666+0000 7f1b667fc700 1 --2- 172.21.15.113:0/3011307758 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b5006caa0 0x7f1b5006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:10.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.667+0000 7f1b667fc700 1 --2- 172.21.15.113:0/3011307758 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b5006caa0 0x7f1b5006ef60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f1b680f9ce0 tx=0x7f1b58007bc0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:10.668 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.671+0000 7f1b4ffff700 1 -- 172.21.15.113:0/3011307758 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1b5c057320 con 0x7f1b680fc7f0 2024-04-12T05:41:10.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.874+0000 7f1b6d529700 1 -- 172.21.15.113:0/3011307758 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b680fb5a0 con 0x7f1b5006caa0 2024-04-12T05:41:10.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.875+0000 7f1b4ffff700 1 -- 172.21.15.113:0/3011307758 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1b680fb5a0 con 0x7f1b5006caa0 2024-04-12T05:41:10.872 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:41:10.872 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:41:10.872 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:41:10.873 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:41:10.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.877+0000 7f1b6d529700 1 -- 172.21.15.113:0/3011307758 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b5006caa0 msgr2=0x7f1b5006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.877+0000 7f1b6d529700 1 --2- 172.21.15.113:0/3011307758 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b5006caa0 0x7f1b5006ef60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f1b680f9ce0 tx=0x7f1b58007bc0).stop 2024-04-12T05:41:10.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.877+0000 7f1b6d529700 1 -- 172.21.15.113:0/3011307758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b680fc7f0 msgr2=0x7f1b680f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.877+0000 7f1b6d529700 1 --2- 172.21.15.113:0/3011307758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b680fc7f0 0x7f1b680f8540 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f1b5c00b120 tx=0x7f1b5c009cb0).stop 2024-04-12T05:41:10.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.877+0000 7f1b6d529700 1 -- 172.21.15.113:0/3011307758 shutdown_connections 2024-04-12T05:41:10.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.877+0000 7f1b6d529700 1 --2- 172.21.15.113:0/3011307758 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f1b5006caa0 0x7f1b5006ef60 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.877+0000 7f1b6d529700 1 --2- 172.21.15.113:0/3011307758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b680fc7f0 0x7f1b680f8540 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.877+0000 7f1b6d529700 1 --2- 172.21.15.113:0/3011307758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b680fd150 0x7f1b680f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:41:10.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.877+0000 7f1b6d529700 1 -- 172.21.15.113:0/3011307758 >> 172.21.15.113:0/3011307758 conn(0x7f1b68070590 msgr2=0x7f1b68060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:10.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.877+0000 7f1b6d529700 1 -- 172.21.15.113:0/3011307758 shutdown_connections 2024-04-12T05:41:10.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.877+0000 7f1b6d529700 1 -- 172.21.15.113:0/3011307758 wait complete. 2024-04-12T05:41:10.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.975+0000 7f170597a700 1 -- 172.21.15.113:0/2115822392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17000fe220 msgr2=0x7f17000f7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.975+0000 7f170597a700 1 --2- 172.21.15.113:0/2115822392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17000fe220 0x7f17000f7a00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7f16e80047f0 tx=0x7f16e8009cb0).stop 2024-04-12T05:41:10.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.975+0000 7f170597a700 1 -- 172.21.15.113:0/2115822392 shutdown_connections 2024-04-12T05:41:10.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.975+0000 7f170597a700 1 --2- 172.21.15.113:0/2115822392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17000fe220 0x7f17000f7a00 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.975+0000 7f170597a700 1 --2- 172.21.15.113:0/2115822392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17000fd8c0 0x7f17000fdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:10.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.975+0000 7f170597a700 1 -- 172.21.15.113:0/2115822392 >> 172.21.15.113:0/2115822392 conn(0x7f17000f34a0 msgr2=0x7f17000f58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:10.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.975+0000 7f170597a700 1 -- 172.21.15.113:0/2115822392 shutdown_connections 2024-04-12T05:41:10.973 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.975+0000 7f170597a700 1 -- 172.21.15.113:0/2115822392 wait complete. 2024-04-12T05:41:10.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.976+0000 7f170597a700 1 Processor -- start 2024-04-12T05:41:10.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.976+0000 7f170597a700 1 -- start start 2024-04-12T05:41:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.976+0000 7f170597a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17000fd8c0 0x7f170010e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.976+0000 7f170597a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17000fe220 0x7f170010e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.976+0000 7f170597a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170010ebe0 con 0x7f17000fd8c0 2024-04-12T05:41:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.976+0000 7f170597a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170010ed20 con 0x7f17000fe220 2024-04-12T05:41:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.976+0000 7f16feffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17000fd8c0 0x7f170010e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.976+0000 7f16feffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17000fd8c0 0x7f170010e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56148/0 (socket says 172.21.15.113:56148) 2024-04-12T05:41:10.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.976+0000 7f16feffd700 1 -- 172.21.15.113:0/1939107630 learned_addr learned my addr 172.21.15.113:0/1939107630 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:10.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.977+0000 7f16feffd700 1 -- 172.21.15.113:0/1939107630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17000fe220 msgr2=0x7f170010e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:10.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.977+0000 7f16feffd700 1 --2- 172.21.15.113:0/1939107630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17000fe220 0x7f170010e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:41:10.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.977+0000 7f16feffd700 1 -- 172.21.15.113:0/1939107630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e8005040 con 0x7f17000fd8c0 2024-04-12T05:41:10.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.977+0000 7f16feffd700 1 --2- 172.21.15.113:0/1939107630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17000fd8c0 0x7f170010e0f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7f16f000e910 tx=0x7f16f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:10.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.977+0000 7f16f7fff700 1 -- 172.21.15.113:0/1939107630 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f000eeb0 con 0x7f17000fd8c0 2024-04-12T05:41:10.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.977+0000 7f16f7fff700 1 -- 172.21.15.113:0/1939107630 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16f000f040 con 0x7f17000fd8c0 2024-04-12T05:41:10.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.977+0000 7f170597a700 1 -- 172.21.15.113:0/1939107630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1700108350 con 0x7f17000fd8c0 2024-04-12T05:41:10.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.977+0000 7f16f7fff700 1 -- 172.21.15.113:0/1939107630 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f000c8a0 con 0x7f17000fd8c0 2024-04-12T05:41:10.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.977+0000 7f170597a700 1 -- 172.21.15.113:0/1939107630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1700108910 con 0x7f17000fd8c0 2024-04-12T05:41:10.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.978+0000 7f170597a700 1 -- 172.21.15.113:0/1939107630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1700060390 con 0x7f17000fd8c0 2024-04-12T05:41:10.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.979+0000 7f16f7fff700 1 -- 172.21.15.113:0/1939107630 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f16f00090d0 con 0x7f17000fd8c0 2024-04-12T05:41:10.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.979+0000 7f16f7fff700 1 --2- 172.21.15.113:0/1939107630 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f16ec06caa0 0x7f16ec06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:10.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.979+0000 7f16f7fff700 1 -- 172.21.15.113:0/1939107630 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f16f008b920 con 0x7f17000fd8c0 2024-04-12T05:41:10.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.979+0000 7f16fe7fc700 1 --2- 172.21.15.113:0/1939107630 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f16ec06caa0 0x7f16ec06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:10.978 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.980+0000 7f16fe7fc700 1 --2- 172.21.15.113:0/1939107630 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f16ec06caa0 0x7f16ec06ef60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f16e8004e60 tx=0x7f16e8009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:10.980 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:10.984+0000 7f16f7fff700 1 -- 172.21.15.113:0/1939107630 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f16f00593d0 con 0x7f17000fd8c0 2024-04-12T05:41:11.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.248+0000 7f170597a700 1 -- 172.21.15.113:0/1939107630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1700047540 con 0x7f17000fd8c0 2024-04-12T05:41:11.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.248+0000 7f16f7fff700 1 -- 172.21.15.113:0/1939107630 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f16f0058f60 con 0x7f17000fd8c0 2024-04-12T05:41:11.245 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:41:11.245 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:41:11.246 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:41:11.247 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:41:11.247 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:41:11.247 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:41:11.247 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:41:11.247 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:41:11.247 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:41:11.247 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:41:11.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.250+0000 7f170597a700 1 -- 172.21.15.113:0/1939107630 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f16ec06caa0 msgr2=0x7f16ec06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:11.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.250+0000 7f170597a700 1 --2- 172.21.15.113:0/1939107630 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f16ec06caa0 0x7f16ec06ef60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f16e8004e60 tx=0x7f16e8009cb0).stop 2024-04-12T05:41:11.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.250+0000 7f170597a700 1 -- 172.21.15.113:0/1939107630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17000fd8c0 msgr2=0x7f170010e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:11.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.250+0000 7f170597a700 1 --2- 172.21.15.113:0/1939107630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17000fd8c0 0x7f170010e0f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7f16f000e910 tx=0x7f16f000b040).stop 2024-04-12T05:41:11.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.250+0000 7f170597a700 1 -- 172.21.15.113:0/1939107630 shutdown_connections 2024-04-12T05:41:11.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.250+0000 7f170597a700 1 --2- 172.21.15.113:0/1939107630 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f16ec06caa0 0x7f16ec06ef60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:11.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.250+0000 7f170597a700 1 --2- 172.21.15.113:0/1939107630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17000fd8c0 0x7f170010e0f0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:11.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.250+0000 7f170597a700 1 --2- 172.21.15.113:0/1939107630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17000fe220 0x7f170010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:41:11.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.250+0000 7f170597a700 1 -- 172.21.15.113:0/1939107630 >> 172.21.15.113:0/1939107630 conn(0x7f17000f34a0 msgr2=0x7f17000fbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:11.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.250+0000 7f170597a700 1 -- 172.21.15.113:0/1939107630 shutdown_connections 2024-04-12T05:41:11.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:11.250+0000 7f170597a700 1 -- 172.21.15.113:0/1939107630 wait complete. 2024-04-12T05:41:11.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:11 smithi113 ceph-mon[103371]: from='client.35574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:11.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:11 smithi113 ceph-mon[103371]: from='client.35578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:11.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:11 smithi113 ceph-mon[103371]: pgmap v871: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:11.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:11 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1459484808' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:11.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:11 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/241257854' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:41:11.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:11 smithi192 ceph-mon[85304]: from='client.35574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:11.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:11 smithi192 ceph-mon[85304]: from='client.35578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:11.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:11 smithi192 ceph-mon[85304]: pgmap v871: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:11.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:11 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1459484808' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:11.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:11 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/241257854' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:41:12.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:12 smithi113 ceph-mon[103371]: from='client.35590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:12.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:12 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1939107630' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:41:12.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:12 smithi192 ceph-mon[85304]: from='client.35590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:12.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:12 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1939107630' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:41:13.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:13 smithi113 ceph-mon[103371]: pgmap v872: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:13.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:13 smithi192 ceph-mon[85304]: pgmap v872: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:15.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:15 smithi113 ceph-mon[103371]: pgmap v873: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:15.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:15 smithi192 ceph-mon[85304]: pgmap v873: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:17.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:17 smithi113 ceph-mon[103371]: pgmap v874: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:17.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:17 smithi192 ceph-mon[85304]: pgmap v874: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:19.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:19 smithi113 ceph-mon[103371]: pgmap v875: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:19.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:19 smithi192 ceph-mon[85304]: pgmap v875: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:21.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:21 smithi113 ceph-mon[103371]: pgmap v876: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:21.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:21 smithi192 ceph-mon[85304]: pgmap v876: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:22.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:22.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:23.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:23 smithi113 ceph-mon[103371]: pgmap v877: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:23.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:23 smithi192 ceph-mon[85304]: pgmap v877: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:25.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:25 smithi113 ceph-mon[103371]: pgmap v878: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:25.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:25 smithi192 ceph-mon[85304]: pgmap v878: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:27.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:27 smithi113 ceph-mon[103371]: pgmap v879: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:27.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:27 smithi192 ceph-mon[85304]: pgmap v879: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:29.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:29 smithi113 ceph-mon[103371]: pgmap v880: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:29.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:29 smithi192 ceph-mon[85304]: pgmap v880: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:31.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:31 smithi113 ceph-mon[103371]: pgmap v881: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:31.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:31 smithi192 ceph-mon[85304]: pgmap v881: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:33.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:33 smithi113 ceph-mon[103371]: pgmap v882: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:33.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:33 smithi192 ceph-mon[85304]: pgmap v882: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:35.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:35 smithi113 ceph-mon[103371]: pgmap v883: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:35.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:35 smithi192 ceph-mon[85304]: pgmap v883: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:37.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:37 smithi113 ceph-mon[103371]: pgmap v884: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:37.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:37.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:37 smithi192 ceph-mon[85304]: pgmap v884: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:37.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:39.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:39 smithi113 ceph-mon[103371]: pgmap v885: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:39.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:39 smithi192 ceph-mon[85304]: pgmap v885: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:41.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.362+0000 7fd38b59e700 1 -- 172.21.15.113:0/3729781857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd38c0fdff0 msgr2=0x7fd38c0f77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:41.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.362+0000 7fd38b59e700 1 --2- 172.21.15.113:0/3729781857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd38c0fdff0 0x7fd38c0f77d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7fd3800047f0 tx=0x7fd380009cb0).stop 2024-04-12T05:41:41.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.362+0000 7fd38b59e700 1 -- 172.21.15.113:0/3729781857 shutdown_connections 2024-04-12T05:41:41.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.362+0000 7fd38b59e700 1 --2- 172.21.15.113:0/3729781857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd38c0fdff0 0x7fd38c0f77d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.362+0000 7fd38b59e700 1 --2- 172.21.15.113:0/3729781857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd38c0fd690 0x7fd38c0fda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.362+0000 7fd38b59e700 1 -- 172.21.15.113:0/3729781857 >> 172.21.15.113:0/3729781857 conn(0x7fd38c0f3270 msgr2=0x7fd38c0f5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:41.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.362+0000 7fd38b59e700 1 -- 172.21.15.113:0/3729781857 shutdown_connections 2024-04-12T05:41:41.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.362+0000 7fd38b59e700 1 -- 172.21.15.113:0/3729781857 wait complete. 2024-04-12T05:41:41.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd38b59e700 1 Processor -- start 2024-04-12T05:41:41.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd38b59e700 1 -- start start 2024-04-12T05:41:41.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd38b59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd38c0fd690 0x7fd38c10e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:41.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd38b59e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd38c0fdff0 0x7fd38c10e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:41.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd38b59e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c10ebb0 con 0x7fd38c0fdff0 2024-04-12T05:41:41.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd38b59e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c10ecf0 con 0x7fd38c0fd690 2024-04-12T05:41:41.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd389d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd38c0fdff0 0x7fd38c10e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:41.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd389d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd38c0fdff0 0x7fd38c10e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52558/0 (socket says 172.21.15.113:52558) 2024-04-12T05:41:41.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd389d9b700 1 -- 172.21.15.113:0/1799891750 learned_addr learned my addr 172.21.15.113:0/1799891750 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:41.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd38a59c700 1 --2- 172.21.15.113:0/1799891750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd38c0fd690 0x7fd38c10e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:41.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd389d9b700 1 -- 172.21.15.113:0/1799891750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd38c0fd690 msgr2=0x7fd38c10e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:41.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd389d9b700 1 --2- 172.21.15.113:0/1799891750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd38c0fd690 0x7fd38c10e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.363+0000 7fd389d9b700 1 -- 172.21.15.113:0/1799891750 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd380005040 con 0x7fd38c0fdff0 2024-04-12T05:41:41.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.364+0000 7fd389d9b700 1 --2- 172.21.15.113:0/1799891750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd38c0fdff0 0x7fd38c10e5b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7fd380000c00 tx=0x7fd380009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:41.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.364+0000 7fd37b7fe700 1 -- 172.21.15.113:0/1799891750 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd380018070 con 0x7fd38c0fdff0 2024-04-12T05:41:41.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.364+0000 7fd37b7fe700 1 -- 172.21.15.113:0/1799891750 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd380007e90 con 0x7fd38c0fdff0 2024-04-12T05:41:41.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.364+0000 7fd38b59e700 1 -- 172.21.15.113:0/1799891750 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd38c108110 con 0x7fd38c0fdff0 2024-04-12T05:41:41.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.364+0000 7fd37b7fe700 1 -- 172.21.15.113:0/1799891750 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3800123c0 con 0x7fd38c0fdff0 2024-04-12T05:41:41.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.364+0000 7fd38b59e700 1 -- 172.21.15.113:0/1799891750 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c108670 con 0x7fd38c0fdff0 2024-04-12T05:41:41.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.365+0000 7fd38b59e700 1 -- 172.21.15.113:0/1799891750 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38c05f2f0 con 0x7fd38c0fdff0 2024-04-12T05:41:41.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.365+0000 7fd37b7fe700 1 -- 172.21.15.113:0/1799891750 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd38001d420 con 0x7fd38c0fdff0 2024-04-12T05:41:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.366+0000 7fd37b7fe700 1 --2- 172.21.15.113:0/1799891750 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd37406caa0 0x7fd37406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.366+0000 7fd37b7fe700 1 -- 172.21.15.113:0/1799891750 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd38008ca10 con 0x7fd38c0fdff0 2024-04-12T05:41:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.366+0000 7fd38a59c700 1 --2- 172.21.15.113:0/1799891750 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd37406caa0 0x7fd37406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:41.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.366+0000 7fd38a59c700 1 --2- 172.21.15.113:0/1799891750 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd37406caa0 0x7fd37406ef60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fd37c0092f0 tx=0x7fd37c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:41.367 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.370+0000 7fd37b7fe700 1 -- 172.21.15.113:0/1799891750 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd38005a4c0 con 0x7fd38c0fdff0 2024-04-12T05:41:41.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:41 smithi113 ceph-mon[103371]: pgmap v886: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:41.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:41 smithi192 ceph-mon[85304]: pgmap v886: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:41.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.574+0000 7fd38b59e700 1 -- 172.21.15.113:0/1799891750 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd38c1097b0 con 0x7fd37406caa0 2024-04-12T05:41:41.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.575+0000 7fd37b7fe700 1 -- 172.21.15.113:0/1799891750 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd38c1097b0 con 0x7fd37406caa0 2024-04-12T05:41:41.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.577+0000 7fd38b59e700 1 -- 172.21.15.113:0/1799891750 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd37406caa0 msgr2=0x7fd37406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:41.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.577+0000 7fd38b59e700 1 --2- 172.21.15.113:0/1799891750 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd37406caa0 0x7fd37406ef60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fd37c0092f0 tx=0x7fd37c00b040).stop 2024-04-12T05:41:41.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.577+0000 7fd38b59e700 1 -- 172.21.15.113:0/1799891750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd38c0fdff0 msgr2=0x7fd38c10e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:41.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.577+0000 7fd38b59e700 1 --2- 172.21.15.113:0/1799891750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd38c0fdff0 0x7fd38c10e5b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7fd380000c00 tx=0x7fd380009cb0).stop 2024-04-12T05:41:41.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.577+0000 7fd38b59e700 1 -- 172.21.15.113:0/1799891750 shutdown_connections 2024-04-12T05:41:41.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.577+0000 7fd38b59e700 1 --2- 172.21.15.113:0/1799891750 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd37406caa0 0x7fd37406ef60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.577+0000 7fd38b59e700 1 --2- 172.21.15.113:0/1799891750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd38c0fd690 0x7fd38c10e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.577+0000 7fd38b59e700 1 --2- 172.21.15.113:0/1799891750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd38c0fdff0 0x7fd38c10e5b0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.577+0000 7fd38b59e700 1 -- 172.21.15.113:0/1799891750 >> 172.21.15.113:0/1799891750 conn(0x7fd38c0f3270 msgr2=0x7fd38c0fba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:41.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.577+0000 7fd38b59e700 1 -- 172.21.15.113:0/1799891750 shutdown_connections 2024-04-12T05:41:41.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.577+0000 7fd38b59e700 1 -- 172.21.15.113:0/1799891750 wait complete. 2024-04-12T05:41:41.585 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:41:41.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.675+0000 7f85229df700 1 -- 172.21.15.113:0/3594871755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f851c0fc7f0 msgr2=0x7f851c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:41.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.675+0000 7f85229df700 1 --2- 172.21.15.113:0/3594871755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f851c0fc7f0 0x7f851c0fcbd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f850c0047f0 tx=0x7f850c009cb0).stop 2024-04-12T05:41:41.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.675+0000 7f85229df700 1 -- 172.21.15.113:0/3594871755 shutdown_connections 2024-04-12T05:41:41.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.675+0000 7f85229df700 1 --2- 172.21.15.113:0/3594871755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f851c0fd150 0x7f851c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:41:41.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.675+0000 7f85229df700 1 --2- 172.21.15.113:0/3594871755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f851c0fc7f0 0x7f851c0fcbd0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.675+0000 7f85229df700 1 -- 172.21.15.113:0/3594871755 >> 172.21.15.113:0/3594871755 conn(0x7f851c070590 msgr2=0x7f851c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:41.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.675+0000 7f85229df700 1 -- 172.21.15.113:0/3594871755 shutdown_connections 2024-04-12T05:41:41.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.675+0000 7f85229df700 1 -- 172.21.15.113:0/3594871755 wait complete. 2024-04-12T05:41:41.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.676+0000 7f85229df700 1 Processor -- start 2024-04-12T05:41:41.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.676+0000 7f85229df700 1 -- start start 2024-04-12T05:41:41.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.676+0000 7f85229df700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f851c0fc7f0 0x7f851c10cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:41.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.676+0000 7f85229df700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f851c0fd150 0x7f851c10d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:41.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.676+0000 7f85229df700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c10daf0 con 0x7f851c0fd150 2024-04-12T05:41:41.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.676+0000 7f85229df700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c107110 con 0x7f851c0fc7f0 2024-04-12T05:41:41.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.676+0000 7f851b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f851c0fd150 0x7f851c10d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:41.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f851b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f851c0fd150 0x7f851c10d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52572/0 (socket says 172.21.15.113:52572) 2024-04-12T05:41:41.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f851b7fe700 1 -- 172.21.15.113:0/2499852886 learned_addr learned my addr 172.21.15.113:0/2499852886 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:41.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f851bfff700 1 --2- 172.21.15.113:0/2499852886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f851c0fc7f0 0x7f851c10cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:41.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f851b7fe700 1 -- 172.21.15.113:0/2499852886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f851c0fc7f0 msgr2=0x7f851c10cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f851b7fe700 1 --2- 172.21.15.113:0/2499852886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f851c0fc7f0 0x7f851c10cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f851b7fe700 1 -- 172.21.15.113:0/2499852886 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c005040 con 0x7f851c0fd150 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f851b7fe700 1 --2- 172.21.15.113:0/2499852886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f851c0fd150 0x7f851c10d460 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f851000b480 tx=0x7f85100090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f85197fa700 1 -- 172.21.15.113:0/2499852886 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8510002500 con 0x7f851c0fd150 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f85197fa700 1 -- 172.21.15.113:0/2499852886 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8510002b10 con 0x7f851c0fd150 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f85229df700 1 -- 172.21.15.113:0/2499852886 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851c1073c0 con 0x7f851c0fd150 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f85197fa700 1 -- 172.21.15.113:0/2499852886 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8510012c60 con 0x7f851c0fd150 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.677+0000 7f85229df700 1 -- 172.21.15.113:0/2499852886 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851c107900 con 0x7f851c0fd150 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.678+0000 7f85229df700 1 -- 172.21.15.113:0/2499852886 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851c05f590 con 0x7f851c0fd150 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.679+0000 7f85197fa700 1 -- 172.21.15.113:0/2499852886 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8510002660 con 0x7f851c0fd150 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.679+0000 7f85197fa700 1 --2- 172.21.15.113:0/2499852886 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f850406caa0 0x7f850406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.679+0000 7f85197fa700 1 -- 172.21.15.113:0/2499852886 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f851008be50 con 0x7f851c0fd150 2024-04-12T05:41:41.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.679+0000 7f851bfff700 1 --2- 172.21.15.113:0/2499852886 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f850406caa0 0x7f850406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:41.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.679+0000 7f851bfff700 1 --2- 172.21.15.113:0/2499852886 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f850406caa0 0x7f850406ef60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f850c007cf0 tx=0x7f850c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:41.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.683+0000 7f85197fa700 1 -- 172.21.15.113:0/2499852886 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8510059900 con 0x7f851c0fd150 2024-04-12T05:41:41.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.889+0000 7f85229df700 1 -- 172.21.15.113:0/2499852886 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f851c108660 con 0x7f850406caa0 2024-04-12T05:41:41.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.890+0000 7f85197fa700 1 -- 172.21.15.113:0/2499852886 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f851c108660 con 0x7f850406caa0 2024-04-12T05:41:41.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.892+0000 7f85229df700 1 -- 172.21.15.113:0/2499852886 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f850406caa0 msgr2=0x7f850406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:41.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.892+0000 7f85229df700 1 --2- 172.21.15.113:0/2499852886 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f850406caa0 0x7f850406ef60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f850c007cf0 tx=0x7f850c00b040).stop 2024-04-12T05:41:41.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.892+0000 7f85229df700 1 -- 172.21.15.113:0/2499852886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f851c0fd150 msgr2=0x7f851c10d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:41.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.892+0000 7f85229df700 1 --2- 172.21.15.113:0/2499852886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f851c0fd150 0x7f851c10d460 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f851000b480 tx=0x7f85100090f0).stop 2024-04-12T05:41:41.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.892+0000 7f85229df700 1 -- 172.21.15.113:0/2499852886 shutdown_connections 2024-04-12T05:41:41.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.892+0000 7f85229df700 1 --2- 172.21.15.113:0/2499852886 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f850406caa0 0x7f850406ef60 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.892+0000 7f85229df700 1 --2- 172.21.15.113:0/2499852886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f851c0fc7f0 0x7f851c10cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.892+0000 7f85229df700 1 --2- 172.21.15.113:0/2499852886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f851c0fd150 0x7f851c10d460 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.890 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.892+0000 7f85229df700 1 -- 172.21.15.113:0/2499852886 >> 172.21.15.113:0/2499852886 conn(0x7f851c070590 msgr2=0x7f851c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:41.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.892+0000 7f85229df700 1 -- 172.21.15.113:0/2499852886 shutdown_connections 2024-04-12T05:41:41.891 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.892+0000 7f85229df700 1 -- 172.21.15.113:0/2499852886 wait complete. 2024-04-12T05:41:41.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.990+0000 7f794bd01700 1 -- 172.21.15.113:0/274308854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79440fd150 msgr2=0x7f79440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:41.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.990+0000 7f794bd01700 1 --2- 172.21.15.113:0/274308854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79440fd150 0x7f79440f6930 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7f79400047f0 tx=0x7f7940009cb0).stop 2024-04-12T05:41:41.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.990+0000 7f794bd01700 1 -- 172.21.15.113:0/274308854 shutdown_connections 2024-04-12T05:41:41.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.990+0000 7f794bd01700 1 --2- 172.21.15.113:0/274308854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79440fd150 0x7f79440f6930 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.990+0000 7f794bd01700 1 --2- 172.21.15.113:0/274308854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79440fc7f0 0x7f79440fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.990+0000 7f794bd01700 1 -- 172.21.15.113:0/274308854 >> 172.21.15.113:0/274308854 conn(0x7f7944070590 msgr2=0x7f79440f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:41.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.990+0000 7f794bd01700 1 -- 172.21.15.113:0/274308854 shutdown_connections 2024-04-12T05:41:41.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.990+0000 7f794bd01700 1 -- 172.21.15.113:0/274308854 wait complete. 2024-04-12T05:41:41.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f794bd01700 1 Processor -- start 2024-04-12T05:41:41.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f794bd01700 1 -- start start 2024-04-12T05:41:41.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f794bd01700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79440fc7f0 0x7f79441101f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:41.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f794bd01700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79440fd150 0x7f79441106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:41.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f794bd01700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7944110ce0 con 0x7f79440fc7f0 2024-04-12T05:41:41.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f794bd01700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7944110e20 con 0x7f79440fd150 2024-04-12T05:41:41.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f7949a9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79440fc7f0 0x7f79441101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:41.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f7949a9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79440fc7f0 0x7f79441101f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52594/0 (socket says 172.21.15.113:52594) 2024-04-12T05:41:41.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f7949a9d700 1 -- 172.21.15.113:0/2688681764 learned_addr learned my addr 172.21.15.113:0/2688681764 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:41.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f794929c700 1 --2- 172.21.15.113:0/2688681764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79440fd150 0x7f79441106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:41.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f7949a9d700 1 -- 172.21.15.113:0/2688681764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79440fd150 msgr2=0x7f79441106e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:41.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f7949a9d700 1 --2- 172.21.15.113:0/2688681764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79440fd150 0x7f79441106e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:41.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f7949a9d700 1 -- 172.21.15.113:0/2688681764 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7940005040 con 0x7f79440fc7f0 2024-04-12T05:41:41.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.991+0000 7f7949a9d700 1 --2- 172.21.15.113:0/2688681764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79440fc7f0 0x7f79441101f0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7f79340041d0 tx=0x7f7934009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:41.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.992+0000 7f793affd700 1 -- 172.21.15.113:0/2688681764 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7934007b40 con 0x7f79440fc7f0 2024-04-12T05:41:41.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.992+0000 7f793affd700 1 -- 172.21.15.113:0/2688681764 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f793400f040 con 0x7f79440fc7f0 2024-04-12T05:41:41.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.992+0000 7f794bd01700 1 -- 172.21.15.113:0/2688681764 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794410a4b0 con 0x7f79440fc7f0 2024-04-12T05:41:41.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.992+0000 7f793affd700 1 -- 172.21.15.113:0/2688681764 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79340137d0 con 0x7f79440fc7f0 2024-04-12T05:41:41.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.992+0000 7f794bd01700 1 -- 172.21.15.113:0/2688681764 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794410aa70 con 0x7f79440fc7f0 2024-04-12T05:41:41.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.992+0000 7f794bd01700 1 -- 172.21.15.113:0/2688681764 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794405f590 con 0x7f79440fc7f0 2024-04-12T05:41:41.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.993+0000 7f793affd700 1 -- 172.21.15.113:0/2688681764 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7934007ca0 con 0x7f79440fc7f0 2024-04-12T05:41:41.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.993+0000 7f793affd700 1 --2- 172.21.15.113:0/2688681764 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f793006caa0 0x7f793006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:41.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.994+0000 7f793affd700 1 -- 172.21.15.113:0/2688681764 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f793408b7d0 con 0x7f79440fc7f0 2024-04-12T05:41:41.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.994+0000 7f794929c700 1 --2- 172.21.15.113:0/2688681764 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f793006caa0 0x7f793006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:41.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.994+0000 7f794929c700 1 --2- 172.21.15.113:0/2688681764 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f793006caa0 0x7f793006ef60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f794410bb30 tx=0x7f794000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:41.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:41.998+0000 7f793affd700 1 -- 172.21.15.113:0/2688681764 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7934059280 con 0x7f79440fc7f0 2024-04-12T05:41:42.191 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.194+0000 7f794bd01700 1 -- 172.21.15.113:0/2688681764 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f794410b870 con 0x7f793006caa0 2024-04-12T05:41:42.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.204+0000 7f793affd700 1 -- 172.21.15.113:0/2688681764 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f794410b870 con 0x7f793006caa0 2024-04-12T05:41:42.201 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:41:42.201 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (29m) 6m ago 40m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:41:42.201 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (26m) 6m ago 40m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:41:42.202 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (26m) 7m ago 40m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:41:42.202 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (29m) 6m ago 40m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:41:42.202 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (22m) 6m ago 35m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:41:42.202 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (22m) 6m ago 35m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:41:42.202 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (22m) 7m ago 35m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:41:42.202 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (7m) 7m ago 35m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:41:42.203 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (30m) 6m ago 42m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:41:42.203 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (30m) 7m ago 40m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:41:42.203 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (28m) 6m ago 42m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:41:42.203 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (27m) 7m ago 40m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:41:42.203 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (30m) 6m ago 40m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:41:42.203 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (30m) 7m ago 39m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:41:42.204 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (26m) 6m ago 39m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:41:42.204 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (25m) 6m ago 38m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:41:42.204 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (25m) 6m ago 38m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:41:42.204 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (24m) 7m ago 37m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:41:42.204 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (24m) 7m ago 37m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:41:42.204 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (23m) 7m ago 36m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:41:42.205 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (29m) 7m ago 39m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:41:42.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.206+0000 7f794bd01700 1 -- 172.21.15.113:0/2688681764 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f793006caa0 msgr2=0x7f793006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:42.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.206+0000 7f794bd01700 1 --2- 172.21.15.113:0/2688681764 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f793006caa0 0x7f793006ef60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f794410bb30 tx=0x7f794000b040).stop 2024-04-12T05:41:42.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.206+0000 7f794bd01700 1 -- 172.21.15.113:0/2688681764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79440fc7f0 msgr2=0x7f79441101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:42.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.206+0000 7f794bd01700 1 --2- 172.21.15.113:0/2688681764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79440fc7f0 0x7f79441101f0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7f79340041d0 tx=0x7f7934009f60).stop 2024-04-12T05:41:42.206 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.206+0000 7f794bd01700 1 -- 172.21.15.113:0/2688681764 shutdown_connections 2024-04-12T05:41:42.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.206+0000 7f794bd01700 1 --2- 172.21.15.113:0/2688681764 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f793006caa0 0x7f793006ef60 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.206+0000 7f794bd01700 1 --2- 172.21.15.113:0/2688681764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79440fc7f0 0x7f79441101f0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.206+0000 7f794bd01700 1 --2- 172.21.15.113:0/2688681764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79440fd150 0x7f79441106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.206+0000 7f794bd01700 1 -- 172.21.15.113:0/2688681764 >> 172.21.15.113:0/2688681764 conn(0x7f7944070590 msgr2=0x7f7944060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:42.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.206+0000 7f794bd01700 1 -- 172.21.15.113:0/2688681764 shutdown_connections 2024-04-12T05:41:42.207 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.206+0000 7f794bd01700 1 -- 172.21.15.113:0/2688681764 wait complete. 2024-04-12T05:41:42.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.303+0000 7fc6d8036700 1 -- 172.21.15.113:0/1710221543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6d00fc7f0 msgr2=0x7fc6d00fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:42.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.303+0000 7fc6d8036700 1 --2- 172.21.15.113:0/1710221543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6d00fc7f0 0x7fc6d00fcbd0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7fc6cc0047f0 tx=0x7fc6cc009cb0).stop 2024-04-12T05:41:42.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.303+0000 7fc6d8036700 1 -- 172.21.15.113:0/1710221543 shutdown_connections 2024-04-12T05:41:42.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.303+0000 7fc6d8036700 1 --2- 172.21.15.113:0/1710221543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6d00fd150 0x7fc6d00f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.303+0000 7fc6d8036700 1 --2- 172.21.15.113:0/1710221543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6d00fc7f0 0x7fc6d00fcbd0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.303+0000 7fc6d8036700 1 -- 172.21.15.113:0/1710221543 >> 172.21.15.113:0/1710221543 conn(0x7fc6d0070590 msgr2=0x7fc6d00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:42.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.303+0000 7fc6d8036700 1 -- 172.21.15.113:0/1710221543 shutdown_connections 2024-04-12T05:41:42.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.303+0000 7fc6d8036700 1 -- 172.21.15.113:0/1710221543 wait complete. 2024-04-12T05:41:42.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.303+0000 7fc6d8036700 1 Processor -- start 2024-04-12T05:41:42.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d8036700 1 -- start start 2024-04-12T05:41:42.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d8036700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6d00fc7f0 0x7fc6d01090f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:42.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d8036700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6d00fd150 0x7fc6d01095e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:42.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d8036700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d0109ad0 con 0x7fc6d00fd150 2024-04-12T05:41:42.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d8036700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d0109c10 con 0x7fc6d00fc7f0 2024-04-12T05:41:42.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d55d1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6d00fd150 0x7fc6d01095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:42.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d55d1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6d00fd150 0x7fc6d01095e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52602/0 (socket says 172.21.15.113:52602) 2024-04-12T05:41:42.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d55d1700 1 -- 172.21.15.113:0/2722572374 learned_addr learned my addr 172.21.15.113:0/2722572374 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:42.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d5dd2700 1 --2- 172.21.15.113:0/2722572374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6d00fc7f0 0x7fc6d01090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:42.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d55d1700 1 -- 172.21.15.113:0/2722572374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6d00fc7f0 msgr2=0x7fc6d01090f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:42.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d55d1700 1 --2- 172.21.15.113:0/2722572374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6d00fc7f0 0x7fc6d01090f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d55d1700 1 -- 172.21.15.113:0/2722572374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc005040 con 0x7fc6d00fd150 2024-04-12T05:41:42.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6d55d1700 1 --2- 172.21.15.113:0/2722572374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6d00fd150 0x7fc6d01095e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7fc6c0002780 tx=0x7fc6c0009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:42.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.304+0000 7fc6c6ffd700 1 -- 172.21.15.113:0/2722572374 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c0007b40 con 0x7fc6d00fd150 2024-04-12T05:41:42.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.305+0000 7fc6c6ffd700 1 -- 172.21.15.113:0/2722572374 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6c000f040 con 0x7fc6d00fd150 2024-04-12T05:41:42.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.305+0000 7fc6d8036700 1 -- 172.21.15.113:0/2722572374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d0109ec0 con 0x7fc6d00fd150 2024-04-12T05:41:42.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.305+0000 7fc6c6ffd700 1 -- 172.21.15.113:0/2722572374 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c0014780 con 0x7fc6d00fd150 2024-04-12T05:41:42.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.305+0000 7fc6d8036700 1 -- 172.21.15.113:0/2722572374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d01124b0 con 0x7fc6d00fd150 2024-04-12T05:41:42.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.305+0000 7fc6d8036700 1 -- 172.21.15.113:0/2722572374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d005f590 con 0x7fc6d00fd150 2024-04-12T05:41:42.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.306+0000 7fc6c6ffd700 1 -- 172.21.15.113:0/2722572374 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc6c0007ca0 con 0x7fc6d00fd150 2024-04-12T05:41:42.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.307+0000 7fc6c6ffd700 1 --2- 172.21.15.113:0/2722572374 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6bc06caa0 0x7fc6bc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:42.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.307+0000 7fc6c6ffd700 1 -- 172.21.15.113:0/2722572374 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc6c008bc70 con 0x7fc6d00fd150 2024-04-12T05:41:42.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.307+0000 7fc6d5dd2700 1 --2- 172.21.15.113:0/2722572374 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6bc06caa0 0x7fc6bc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:42.307 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.307+0000 7fc6d5dd2700 1 --2- 172.21.15.113:0/2722572374 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6bc06caa0 0x7fc6bc06ef60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fc6cc004f10 tx=0x7fc6cc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:42.308 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.311+0000 7fc6c6ffd700 1 -- 172.21.15.113:0/2722572374 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc6c0055c90 con 0x7fc6d00fd150 2024-04-12T05:41:42.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.578+0000 7fc6d8036700 1 -- 172.21.15.113:0/2722572374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc6d00feb80 con 0x7fc6d00fd150 2024-04-12T05:41:42.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.578+0000 7fc6c6ffd700 1 -- 172.21.15.113:0/2722572374 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fc6c00592b0 con 0x7fc6d00fd150 2024-04-12T05:41:42.575 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:41:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:41:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:41:42.575 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:41:42.576 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:41:42.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.580+0000 7fc6d8036700 1 -- 172.21.15.113:0/2722572374 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6bc06caa0 msgr2=0x7fc6bc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:42.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.580+0000 7fc6d8036700 1 --2- 172.21.15.113:0/2722572374 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6bc06caa0 0x7fc6bc06ef60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fc6cc004f10 tx=0x7fc6cc00b040).stop 2024-04-12T05:41:42.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.580+0000 7fc6d8036700 1 -- 172.21.15.113:0/2722572374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6d00fd150 msgr2=0x7fc6d01095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:42.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.580+0000 7fc6d8036700 1 --2- 172.21.15.113:0/2722572374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6d00fd150 0x7fc6d01095e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7fc6c0002780 tx=0x7fc6c0009ee0).stop 2024-04-12T05:41:42.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.580+0000 7fc6d8036700 1 -- 172.21.15.113:0/2722572374 shutdown_connections 2024-04-12T05:41:42.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.580+0000 7fc6d8036700 1 --2- 172.21.15.113:0/2722572374 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc6bc06caa0 0x7fc6bc06ef60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.580+0000 7fc6d8036700 1 --2- 172.21.15.113:0/2722572374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6d00fc7f0 0x7fc6d01090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.580+0000 7fc6d8036700 1 --2- 172.21.15.113:0/2722572374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6d00fd150 0x7fc6d01095e0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.580+0000 7fc6d8036700 1 -- 172.21.15.113:0/2722572374 >> 172.21.15.113:0/2722572374 conn(0x7fc6d0070590 msgr2=0x7fc6d00fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:42.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.580+0000 7fc6d8036700 1 -- 172.21.15.113:0/2722572374 shutdown_connections 2024-04-12T05:41:42.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.580+0000 7fc6d8036700 1 -- 172.21.15.113:0/2722572374 wait complete. 2024-04-12T05:41:42.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.678+0000 7f2740273700 1 -- 172.21.15.113:0/4232671795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27380fd150 msgr2=0x7f27380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:42.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.678+0000 7f2740273700 1 --2- 172.21.15.113:0/4232671795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27380fd150 0x7f27380f6930 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7f27340047f0 tx=0x7f2734009cb0).stop 2024-04-12T05:41:42.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.678+0000 7f2740273700 1 -- 172.21.15.113:0/4232671795 shutdown_connections 2024-04-12T05:41:42.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.678+0000 7f2740273700 1 --2- 172.21.15.113:0/4232671795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27380fd150 0x7f27380f6930 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.678+0000 7f2740273700 1 --2- 172.21.15.113:0/4232671795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27380fc7f0 0x7f27380fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.678+0000 7f2740273700 1 -- 172.21.15.113:0/4232671795 >> 172.21.15.113:0/4232671795 conn(0x7f2738070590 msgr2=0x7f27380f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:42.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.678+0000 7f2740273700 1 -- 172.21.15.113:0/4232671795 shutdown_connections 2024-04-12T05:41:42.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.678+0000 7f2740273700 1 -- 172.21.15.113:0/4232671795 wait complete. 2024-04-12T05:41:42.677 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f2740273700 1 Processor -- start 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f2740273700 1 -- start start 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f2740273700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27380fc7f0 0x7f27380f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f2740273700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27380fd150 0x7f27380f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f2740273700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27380f8f20 con 0x7f27380fd150 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f2740273700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27380f9060 con 0x7f27380fc7f0 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f273d80e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27380fd150 0x7f27380f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f273d80e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27380fd150 0x7f27380f8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52616/0 (socket says 172.21.15.113:52616) 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f273d80e700 1 -- 172.21.15.113:0/221295805 learned_addr learned my addr 172.21.15.113:0/221295805 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f273d80e700 1 -- 172.21.15.113:0/221295805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27380fc7f0 msgr2=0x7f27380f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f273e00f700 1 --2- 172.21.15.113:0/221295805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27380fc7f0 0x7f27380f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f273d80e700 1 --2- 172.21.15.113:0/221295805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27380fc7f0 0x7f27380f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f273d80e700 1 -- 172.21.15.113:0/221295805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2734005040 con 0x7f27380fd150 2024-04-12T05:41:42.678 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.679+0000 7f273d80e700 1 --2- 172.21.15.113:0/221295805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27380fd150 0x7f27380f8a30 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7f27340099f0 tx=0x7f2734009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:42.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.680+0000 7f272f779700 1 -- 172.21.15.113:0/221295805 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f273400e030 con 0x7f27380fd150 2024-04-12T05:41:42.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.680+0000 7f2740273700 1 -- 172.21.15.113:0/221295805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27380f92b0 con 0x7f27380fd150 2024-04-12T05:41:42.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.680+0000 7f272f779700 1 -- 172.21.15.113:0/221295805 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2734007e90 con 0x7f27380fd150 2024-04-12T05:41:42.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.680+0000 7f272f779700 1 -- 172.21.15.113:0/221295805 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2734018070 con 0x7f27380fd150 2024-04-12T05:41:42.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.680+0000 7f2740273700 1 -- 172.21.15.113:0/221295805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27381145f0 con 0x7f27380fd150 2024-04-12T05:41:42.679 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.680+0000 7f2740273700 1 -- 172.21.15.113:0/221295805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273805f590 con 0x7f27380fd150 2024-04-12T05:41:42.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.681+0000 7f272f779700 1 -- 172.21.15.113:0/221295805 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f273401d420 con 0x7f27380fd150 2024-04-12T05:41:42.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.681+0000 7f272f779700 1 --2- 172.21.15.113:0/221295805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f272406caa0 0x7f272406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:42.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.682+0000 7f272f779700 1 -- 172.21.15.113:0/221295805 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f273408d2e0 con 0x7f27380fd150 2024-04-12T05:41:42.680 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.682+0000 7f273e00f700 1 --2- 172.21.15.113:0/221295805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f272406caa0 0x7f272406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:42.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.682+0000 7f273e00f700 1 --2- 172.21.15.113:0/221295805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f272406caa0 0x7f272406ef60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f27280041d0 tx=0x7f2728009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:42.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.686+0000 7f272f779700 1 -- 172.21.15.113:0/221295805 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2734057300 con 0x7f27380fd150 2024-04-12T05:41:42.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.918+0000 7f2740273700 1 -- 172.21.15.113:0/221295805 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2738047540 con 0x7f27380fd150 2024-04-12T05:41:42.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.918+0000 7f272f779700 1 -- 172.21.15.113:0/221295805 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f273405a920 con 0x7f27380fd150 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:41:42.916 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:41:42.917 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:42.918 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:41:42.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.920+0000 7f2740273700 1 -- 172.21.15.113:0/221295805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f272406caa0 msgr2=0x7f272406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:42.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.920+0000 7f2740273700 1 --2- 172.21.15.113:0/221295805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f272406caa0 0x7f272406ef60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f27280041d0 tx=0x7f2728009040).stop 2024-04-12T05:41:42.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.920+0000 7f2740273700 1 -- 172.21.15.113:0/221295805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27380fd150 msgr2=0x7f27380f8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:42.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.920+0000 7f2740273700 1 --2- 172.21.15.113:0/221295805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27380fd150 0x7f27380f8a30 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7f27340099f0 tx=0x7f2734009cb0).stop 2024-04-12T05:41:42.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.920+0000 7f2740273700 1 -- 172.21.15.113:0/221295805 shutdown_connections 2024-04-12T05:41:42.919 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.920+0000 7f2740273700 1 --2- 172.21.15.113:0/221295805 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f272406caa0 0x7f272406ef60 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.920+0000 7f2740273700 1 --2- 172.21.15.113:0/221295805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f27380fc7f0 0x7f27380f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.920+0000 7f2740273700 1 --2- 172.21.15.113:0/221295805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27380fd150 0x7f27380f8a30 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:42.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.920+0000 7f2740273700 1 -- 172.21.15.113:0/221295805 >> 172.21.15.113:0/221295805 conn(0x7f2738070590 msgr2=0x7f2738060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:42.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.920+0000 7f2740273700 1 -- 172.21.15.113:0/221295805 shutdown_connections 2024-04-12T05:41:42.920 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:42.920+0000 7f2740273700 1 -- 172.21.15.113:0/221295805 wait complete. 2024-04-12T05:41:42.920 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:41:43.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.019+0000 7fb349fa0700 1 -- 172.21.15.113:0/1497320319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3440fc7f0 msgr2=0x7fb3440fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:43.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.019+0000 7fb349fa0700 1 --2- 172.21.15.113:0/1497320319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3440fc7f0 0x7fb3440fcbd0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7fb3340047f0 tx=0x7fb334009cb0).stop 2024-04-12T05:41:43.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.019+0000 7fb349fa0700 1 -- 172.21.15.113:0/1497320319 shutdown_connections 2024-04-12T05:41:43.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.019+0000 7fb349fa0700 1 --2- 172.21.15.113:0/1497320319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3440fd150 0x7fb3440f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.019+0000 7fb349fa0700 1 --2- 172.21.15.113:0/1497320319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3440fc7f0 0x7fb3440fcbd0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.019+0000 7fb349fa0700 1 -- 172.21.15.113:0/1497320319 >> 172.21.15.113:0/1497320319 conn(0x7fb344070590 msgr2=0x7fb3440f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:43.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.019+0000 7fb349fa0700 1 -- 172.21.15.113:0/1497320319 shutdown_connections 2024-04-12T05:41:43.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.019+0000 7fb349fa0700 1 -- 172.21.15.113:0/1497320319 wait complete. 2024-04-12T05:41:43.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.020+0000 7fb349fa0700 1 Processor -- start 2024-04-12T05:41:43.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.020+0000 7fb349fa0700 1 -- start start 2024-04-12T05:41:43.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.020+0000 7fb349fa0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3440fc7f0 0x7fb34410cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:43.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.020+0000 7fb349fa0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3440fd150 0x7fb34410d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:43.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.020+0000 7fb349fa0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34410d9f0 con 0x7fb3440fd150 2024-04-12T05:41:43.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.020+0000 7fb349fa0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34410db30 con 0x7fb3440fc7f0 2024-04-12T05:41:43.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.020+0000 7fb342ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3440fd150 0x7fb34410d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:43.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.020+0000 7fb342ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3440fd150 0x7fb34410d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52644/0 (socket says 172.21.15.113:52644) 2024-04-12T05:41:43.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.020+0000 7fb342ffd700 1 -- 172.21.15.113:0/2816552821 learned_addr learned my addr 172.21.15.113:0/2816552821 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.021+0000 7fb3437fe700 1 --2- 172.21.15.113:0/2816552821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3440fc7f0 0x7fb34410cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.021+0000 7fb342ffd700 1 -- 172.21.15.113:0/2816552821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3440fc7f0 msgr2=0x7fb34410cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.021+0000 7fb342ffd700 1 --2- 172.21.15.113:0/2816552821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3440fc7f0 0x7fb34410cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.021+0000 7fb342ffd700 1 -- 172.21.15.113:0/2816552821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334005040 con 0x7fb3440fd150 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.021+0000 7fb342ffd700 1 --2- 172.21.15.113:0/2816552821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3440fd150 0x7fb34410d3f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7fb338002780 tx=0x7fb338009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.021+0000 7fb340ff9700 1 -- 172.21.15.113:0/2816552821 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb338007b40 con 0x7fb3440fd150 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.021+0000 7fb340ff9700 1 -- 172.21.15.113:0/2816552821 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb33800f040 con 0x7fb3440fd150 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.021+0000 7fb340ff9700 1 -- 172.21.15.113:0/2816552821 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb338014780 con 0x7fb3440fd150 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.021+0000 7fb349fa0700 1 -- 172.21.15.113:0/2816552821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3441071c0 con 0x7fb3440fd150 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.021+0000 7fb349fa0700 1 -- 172.21.15.113:0/2816552821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb344107780 con 0x7fb3440fd150 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.022+0000 7fb349fa0700 1 -- 172.21.15.113:0/2816552821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34405f590 con 0x7fb3440fd150 2024-04-12T05:41:43.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.022+0000 7fb340ff9700 1 -- 172.21.15.113:0/2816552821 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb338007ca0 con 0x7fb3440fd150 2024-04-12T05:41:43.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.023+0000 7fb340ff9700 1 --2- 172.21.15.113:0/2816552821 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb32c06caa0 0x7fb32c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:43.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.023+0000 7fb340ff9700 1 -- 172.21.15.113:0/2816552821 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb33808bcf0 con 0x7fb3440fd150 2024-04-12T05:41:43.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.023+0000 7fb3437fe700 1 --2- 172.21.15.113:0/2816552821 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb32c06caa0 0x7fb32c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:43.021 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.023+0000 7fb3437fe700 1 --2- 172.21.15.113:0/2816552821 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb32c06caa0 0x7fb32c06ef60 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb334004f10 tx=0x7fb33400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:43.024 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.027+0000 7fb340ff9700 1 -- 172.21.15.113:0/2816552821 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb3380597a0 con 0x7fb3440fd150 2024-04-12T05:41:43.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.233+0000 7fb349fa0700 1 -- 172.21.15.113:0/2816552821 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb344107a50 con 0x7fb32c06caa0 2024-04-12T05:41:43.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.234+0000 7fb340ff9700 1 -- 172.21.15.113:0/2816552821 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb344107a50 con 0x7fb32c06caa0 2024-04-12T05:41:43.231 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:41:43.231 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:41:43.231 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:41:43.231 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:41:43.231 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:41:43.231 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:41:43.231 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:41:43.231 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:41:43.232 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:41:43.232 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:41:43.232 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:41:43.232 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:41:43.232 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:41:43.232 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:41:43.232 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:41:43.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.236+0000 7fb349fa0700 1 -- 172.21.15.113:0/2816552821 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb32c06caa0 msgr2=0x7fb32c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:43.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.236+0000 7fb349fa0700 1 --2- 172.21.15.113:0/2816552821 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb32c06caa0 0x7fb32c06ef60 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb334004f10 tx=0x7fb33400b040).stop 2024-04-12T05:41:43.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.236+0000 7fb349fa0700 1 -- 172.21.15.113:0/2816552821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3440fd150 msgr2=0x7fb34410d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:43.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.236+0000 7fb349fa0700 1 --2- 172.21.15.113:0/2816552821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3440fd150 0x7fb34410d3f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7fb338002780 tx=0x7fb338009ee0).stop 2024-04-12T05:41:43.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.236+0000 7fb349fa0700 1 -- 172.21.15.113:0/2816552821 shutdown_connections 2024-04-12T05:41:43.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.236+0000 7fb349fa0700 1 --2- 172.21.15.113:0/2816552821 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb32c06caa0 0x7fb32c06ef60 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.236+0000 7fb349fa0700 1 --2- 172.21.15.113:0/2816552821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3440fc7f0 0x7fb34410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.236+0000 7fb349fa0700 1 --2- 172.21.15.113:0/2816552821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3440fd150 0x7fb34410d3f0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.236+0000 7fb349fa0700 1 -- 172.21.15.113:0/2816552821 >> 172.21.15.113:0/2816552821 conn(0x7fb344070590 msgr2=0x7fb3440fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:43.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.236+0000 7fb349fa0700 1 -- 172.21.15.113:0/2816552821 shutdown_connections 2024-04-12T05:41:43.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.236+0000 7fb349fa0700 1 -- 172.21.15.113:0/2816552821 wait complete. 2024-04-12T05:41:43.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.334+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/3267818569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9edc0fc7f0 msgr2=0x7f9edc0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:43.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.334+0000 7f9ee0aa6700 1 --2- 172.21.15.113:0/3267818569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9edc0fc7f0 0x7f9edc0fcbd0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f9ec40047f0 tx=0x7f9ec4009cb0).stop 2024-04-12T05:41:43.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.334+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/3267818569 shutdown_connections 2024-04-12T05:41:43.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.334+0000 7f9ee0aa6700 1 --2- 172.21.15.113:0/3267818569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9edc0fd150 0x7f9edc0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.334+0000 7f9ee0aa6700 1 --2- 172.21.15.113:0/3267818569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9edc0fc7f0 0x7f9edc0fcbd0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.334+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/3267818569 >> 172.21.15.113:0/3267818569 conn(0x7f9edc070590 msgr2=0x7f9edc0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:43.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.334+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/3267818569 shutdown_connections 2024-04-12T05:41:43.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.334+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/3267818569 wait complete. 2024-04-12T05:41:43.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ee0aa6700 1 Processor -- start 2024-04-12T05:41:43.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ee0aa6700 1 -- start start 2024-04-12T05:41:43.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ee0aa6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9edc0fc7f0 0x7f9edc106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ee0aa6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9edc0fd150 0x7f9edc1074d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ee0aa6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0614e0 con 0x7f9edc0fd150 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ee0aa6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc1079c0 con 0x7f9edc0fc7f0 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ed9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9edc0fd150 0x7f9edc1074d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ed9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9edc0fd150 0x7f9edc1074d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52660/0 (socket says 172.21.15.113:52660) 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ed9d9b700 1 -- 172.21.15.113:0/2960970917 learned_addr learned my addr 172.21.15.113:0/2960970917 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9eda59c700 1 --2- 172.21.15.113:0/2960970917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9edc0fc7f0 0x7f9edc106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ed9d9b700 1 -- 172.21.15.113:0/2960970917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9edc0fc7f0 msgr2=0x7f9edc106fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ed9d9b700 1 --2- 172.21.15.113:0/2960970917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9edc0fc7f0 0x7f9edc106fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ed9d9b700 1 -- 172.21.15.113:0/2960970917 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec4005040 con 0x7f9edc0fd150 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.335+0000 7f9ed9d9b700 1 --2- 172.21.15.113:0/2960970917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9edc0fd150 0x7f9edc1074d0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f9ecc00e8c0 tx=0x7f9ecc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.336+0000 7f9ed37fe700 1 -- 172.21.15.113:0/2960970917 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ecc007b40 con 0x7f9edc0fd150 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.336+0000 7f9ed37fe700 1 -- 172.21.15.113:0/2960970917 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ecc00f040 con 0x7f9edc0fd150 2024-04-12T05:41:43.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.336+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/2960970917 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9edc107ca0 con 0x7f9edc0fd150 2024-04-12T05:41:43.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.336+0000 7f9ed37fe700 1 -- 172.21.15.113:0/2960970917 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ecc014830 con 0x7f9edc0fd150 2024-04-12T05:41:43.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.336+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/2960970917 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9edc114810 con 0x7f9edc0fd150 2024-04-12T05:41:43.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.337+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/2960970917 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9edc05f590 con 0x7f9edc0fd150 2024-04-12T05:41:43.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.337+0000 7f9ed37fe700 1 -- 172.21.15.113:0/2960970917 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9ecc014af0 con 0x7f9edc0fd150 2024-04-12T05:41:43.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.337+0000 7f9ed37fe700 1 --2- 172.21.15.113:0/2960970917 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ec806caa0 0x7f9ec806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:41:43.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.338+0000 7f9ed37fe700 1 -- 172.21.15.113:0/2960970917 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9ecc08b920 con 0x7f9edc0fd150 2024-04-12T05:41:43.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.338+0000 7f9eda59c700 1 --2- 172.21.15.113:0/2960970917 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ec806caa0 0x7f9ec806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:41:43.336 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.338+0000 7f9eda59c700 1 --2- 172.21.15.113:0/2960970917 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ec806caa0 0x7f9ec806ef60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f9ec40047c0 tx=0x7f9ec4009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:41:43.338 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.342+0000 7f9ed37fe700 1 -- 172.21.15.113:0/2960970917 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9ecc059490 con 0x7f9edc0fd150 2024-04-12T05:41:43.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:43 smithi113 ceph-mon[103371]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:43.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:43 smithi113 ceph-mon[103371]: from='client.35602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:43.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:43 smithi113 ceph-mon[103371]: pgmap v887: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:43.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:43 smithi113 ceph-mon[103371]: from='client.35606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:43.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:43 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2722572374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:43.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:43 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/221295805' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:41:43.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:43 smithi192 ceph-mon[85304]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:43.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:43 smithi192 ceph-mon[85304]: from='client.35602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:43.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:43 smithi192 ceph-mon[85304]: pgmap v887: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:43.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:43 smithi192 ceph-mon[85304]: from='client.35606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:43.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:43 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2722572374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:41:43.444 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:43 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/221295805' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:41:43.604 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.607+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/2960970917 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9edc047540 con 0x7f9edc0fd150 2024-04-12T05:41:43.605 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.608+0000 7f9ed37fe700 1 -- 172.21.15.113:0/2960970917 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f9ecc059020 con 0x7f9edc0fd150 2024-04-12T05:41:43.605 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:41:43.605 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:41:43.605 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:41:43.605 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:41:43.605 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:41:43.605 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:41:43.605 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:41:43.605 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:41:43.605 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:41:43.606 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:41:43.607 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:41:43.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.610+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/2960970917 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ec806caa0 msgr2=0x7f9ec806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:43.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.610+0000 7f9ee0aa6700 1 --2- 172.21.15.113:0/2960970917 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ec806caa0 0x7f9ec806ef60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f9ec40047c0 tx=0x7f9ec4009cb0).stop 2024-04-12T05:41:43.607 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.610+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/2960970917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9edc0fd150 msgr2=0x7f9edc1074d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:41:43.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.610+0000 7f9ee0aa6700 1 --2- 172.21.15.113:0/2960970917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9edc0fd150 0x7f9edc1074d0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f9ecc00e8c0 tx=0x7f9ecc00b040).stop 2024-04-12T05:41:43.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.610+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/2960970917 shutdown_connections 2024-04-12T05:41:43.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.610+0000 7f9ee0aa6700 1 --2- 172.21.15.113:0/2960970917 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9ec806caa0 0x7f9ec806ef60 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.610+0000 7f9ee0aa6700 1 --2- 172.21.15.113:0/2960970917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9edc0fc7f0 0x7f9edc106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.610+0000 7f9ee0aa6700 1 --2- 172.21.15.113:0/2960970917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9edc0fd150 0x7f9edc1074d0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:41:43.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.610+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/2960970917 >> 172.21.15.113:0/2960970917 conn(0x7f9edc070590 msgr2=0x7f9edc0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:41:43.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.610+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/2960970917 shutdown_connections 2024-04-12T05:41:43.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:41:43.610+0000 7f9ee0aa6700 1 -- 172.21.15.113:0/2960970917 wait complete. 2024-04-12T05:41:44.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:44 smithi113 ceph-mon[103371]: from='client.35618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:44.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:44 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2960970917' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:41:44.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:44 smithi192 ceph-mon[85304]: from='client.35618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:41:44.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:44 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2960970917' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:41:45.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:45 smithi113 ceph-mon[103371]: pgmap v888: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:45.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:45 smithi192 ceph-mon[85304]: pgmap v888: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:47.421 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:47 smithi113 ceph-mon[103371]: pgmap v889: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:47.448 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:47 smithi192 ceph-mon[85304]: pgmap v889: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:49.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:49 smithi113 ceph-mon[103371]: pgmap v890: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:49.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:49 smithi192 ceph-mon[85304]: pgmap v890: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:51.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:51 smithi113 ceph-mon[103371]: pgmap v891: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:51.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:51 smithi192 ceph-mon[85304]: pgmap v891: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:52.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:52.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:41:53.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:53 smithi113 ceph-mon[103371]: pgmap v892: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:53.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:53 smithi192 ceph-mon[85304]: pgmap v892: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:55.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:55 smithi113 ceph-mon[103371]: pgmap v893: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:55.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:55 smithi192 ceph-mon[85304]: pgmap v893: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:57.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:57 smithi113 ceph-mon[103371]: pgmap v894: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:57.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:57 smithi192 ceph-mon[85304]: pgmap v894: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:59.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:41:59 smithi113 ceph-mon[103371]: pgmap v895: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:41:59.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:41:59 smithi192 ceph-mon[85304]: pgmap v895: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:01.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:01 smithi113 ceph-mon[103371]: pgmap v896: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:01.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:01 smithi192 ceph-mon[85304]: pgmap v896: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:03.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:03 smithi113 ceph-mon[103371]: pgmap v897: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:03.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:03 smithi192 ceph-mon[85304]: pgmap v897: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:05.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:05 smithi113 ceph-mon[103371]: pgmap v898: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:05.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:05 smithi192 ceph-mon[85304]: pgmap v898: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:07.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:07 smithi113 ceph-mon[103371]: pgmap v899: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:07.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:07.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:07 smithi192 ceph-mon[85304]: pgmap v899: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:07.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:09.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:09 smithi113 ceph-mon[103371]: pgmap v900: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:09.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:09 smithi192 ceph-mon[85304]: pgmap v900: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:11.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:11 smithi113 ceph-mon[103371]: pgmap v901: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:11.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:11 smithi192 ceph-mon[85304]: pgmap v901: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:13.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:13 smithi113 ceph-mon[103371]: pgmap v902: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:13.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:13 smithi192 ceph-mon[85304]: pgmap v902: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:13.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.715+0000 7f2cce666700 1 -- 172.21.15.113:0/3344792688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cc80fc7f0 msgr2=0x7f2cc80fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.715+0000 7f2cce666700 1 --2- 172.21.15.113:0/3344792688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cc80fc7f0 0x7f2cc80fcbd0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f2cb00047f0 tx=0x7f2cb0009cb0).stop 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.716+0000 7f2cce666700 1 -- 172.21.15.113:0/3344792688 shutdown_connections 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.716+0000 7f2cce666700 1 --2- 172.21.15.113:0/3344792688 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc80fd150 0x7f2cc80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.716+0000 7f2cce666700 1 --2- 172.21.15.113:0/3344792688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cc80fc7f0 0x7f2cc80fcbd0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.716+0000 7f2cce666700 1 -- 172.21.15.113:0/3344792688 >> 172.21.15.113:0/3344792688 conn(0x7f2cc8070590 msgr2=0x7f2cc80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.716+0000 7f2cce666700 1 -- 172.21.15.113:0/3344792688 shutdown_connections 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.716+0000 7f2cce666700 1 -- 172.21.15.113:0/3344792688 wait complete. 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.716+0000 7f2cce666700 1 Processor -- start 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.716+0000 7f2cce666700 1 -- start start 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cce666700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc80fc7f0 0x7f2cc810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cce666700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cc80fd150 0x7f2cc810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cce666700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc810eb20 con 0x7f2cc80fd150 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cce666700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc810ec60 con 0x7f2cc80fc7f0 2024-04-12T05:42:13.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cc80fd150 0x7f2cc810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cc80fd150 0x7f2cc810e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36748/0 (socket says 172.21.15.113:36748) 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc77fe700 1 -- 172.21.15.113:0/3012047678 learned_addr learned my addr 172.21.15.113:0/3012047678 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc7fff700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc80fc7f0 0x7f2cc810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc77fe700 1 -- 172.21.15.113:0/3012047678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc80fc7f0 msgr2=0x7f2cc810e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc77fe700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc80fc7f0 0x7f2cc810e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc77fe700 1 -- 172.21.15.113:0/3012047678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb0005040 con 0x7f2cc80fd150 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc7fff700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc80fc7f0 0x7f2cc810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc77fe700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cc80fd150 0x7f2cc810e520 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f2cb800c790 tx=0x7f2cb8007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc57fa700 1 -- 172.21.15.113:0/3012047678 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cb80106f0 con 0x7f2cc80fd150 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc57fa700 1 -- 172.21.15.113:0/3012047678 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2cb8010d00 con 0x7f2cc80fd150 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cc57fa700 1 -- 172.21.15.113:0/3012047678 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cb8012d50 con 0x7f2cc80fd150 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cce666700 1 -- 172.21.15.113:0/3012047678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc81082f0 con 0x7f2cc80fd150 2024-04-12T05:42:13.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.717+0000 7f2cce666700 1 -- 172.21.15.113:0/3012047678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc81088b0 con 0x7f2cc80fd150 2024-04-12T05:42:13.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.718+0000 7f2cce666700 1 -- 172.21.15.113:0/3012047678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc805f590 con 0x7f2cc80fd150 2024-04-12T05:42:13.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.719+0000 7f2cc57fa700 1 -- 172.21.15.113:0/3012047678 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2cb8012570 con 0x7f2cc80fd150 2024-04-12T05:42:13.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.719+0000 7f2cc57fa700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cb406caa0 0x7f2cb406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:13.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.719+0000 7f2cc57fa700 1 -- 172.21.15.113:0/3012047678 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2cb808b630 con 0x7f2cc80fd150 2024-04-12T05:42:13.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.719+0000 7f2cc7fff700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cb406caa0 0x7f2cb406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:13.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.720+0000 7f2cc7fff700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cb406caa0 0x7f2cb406ef60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f2cb00043d0 tx=0x7f2cb000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:13.721 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.724+0000 7f2cc57fa700 1 -- 172.21.15.113:0/3012047678 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2cb80590e0 con 0x7f2cc80fd150 2024-04-12T05:42:13.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.931+0000 7f2cce666700 1 -- 172.21.15.113:0/3012047678 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cc8108fa0 con 0x7f2cb406caa0 2024-04-12T05:42:13.930 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.932+0000 7f2cc57fa700 1 -- 172.21.15.113:0/3012047678 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2cc8108fa0 con 0x7f2cb406caa0 2024-04-12T05:42:13.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.934+0000 7f2cce666700 1 -- 172.21.15.113:0/3012047678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cb406caa0 msgr2=0x7f2cb406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:13.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.934+0000 7f2cce666700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cb406caa0 0x7f2cb406ef60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f2cb00043d0 tx=0x7f2cb000b040).stop 2024-04-12T05:42:13.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.934+0000 7f2cce666700 1 -- 172.21.15.113:0/3012047678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cc80fd150 msgr2=0x7f2cc810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:13.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.934+0000 7f2cce666700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cc80fd150 0x7f2cc810e520 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f2cb800c790 tx=0x7f2cb8007f50).stop 2024-04-12T05:42:13.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.934+0000 7f2cce666700 1 -- 172.21.15.113:0/3012047678 shutdown_connections 2024-04-12T05:42:13.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.934+0000 7f2cce666700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2cb406caa0 0x7f2cb406ef60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:13.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.934+0000 7f2cce666700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc80fc7f0 0x7f2cc810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:13.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.934+0000 7f2cce666700 1 --2- 172.21.15.113:0/3012047678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cc80fd150 0x7f2cc810e520 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:13.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.934+0000 7f2cce666700 1 -- 172.21.15.113:0/3012047678 >> 172.21.15.113:0/3012047678 conn(0x7f2cc8070590 msgr2=0x7f2cc8060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:13.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.934+0000 7f2cce666700 1 -- 172.21.15.113:0/3012047678 shutdown_connections 2024-04-12T05:42:13.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:13.935+0000 7f2cce666700 1 -- 172.21.15.113:0/3012047678 wait complete. 2024-04-12T05:42:13.942 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:42:14.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.033+0000 7f74e7d2d700 1 -- 172.21.15.113:0/878295443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74e00f9210 msgr2=0x7f74e01019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.033+0000 7f74e7d2d700 1 --2- 172.21.15.113:0/878295443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74e00f9210 0x7f74e01019b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7f74dc0047f0 tx=0x7f74dc009cb0).stop 2024-04-12T05:42:14.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.033+0000 7f74e7d2d700 1 -- 172.21.15.113:0/878295443 shutdown_connections 2024-04-12T05:42:14.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.033+0000 7f74e7d2d700 1 --2- 172.21.15.113:0/878295443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74e00f9210 0x7f74e01019b0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.033+0000 7f74e7d2d700 1 --2- 172.21.15.113:0/878295443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74e00f8940 0x7f74e00f8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.033+0000 7f74e7d2d700 1 -- 172.21.15.113:0/878295443 >> 172.21.15.113:0/878295443 conn(0x7f74e0070590 msgr2=0x7f74e00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:14.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.033+0000 7f74e7d2d700 1 -- 172.21.15.113:0/878295443 shutdown_connections 2024-04-12T05:42:14.032 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.033+0000 7f74e7d2d700 1 -- 172.21.15.113:0/878295443 wait complete. 2024-04-12T05:42:14.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e7d2d700 1 Processor -- start 2024-04-12T05:42:14.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e7d2d700 1 -- start start 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e7d2d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74e00f8940 0x7f74e010e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e7d2d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74e00f9210 0x7f74e010e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e7d2d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e010eb20 con 0x7f74e00f9210 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e7d2d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e010ec60 con 0x7f74e00f8940 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e52c8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74e00f9210 0x7f74e010e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e52c8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74e00f9210 0x7f74e010e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36780/0 (socket says 172.21.15.113:36780) 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e52c8700 1 -- 172.21.15.113:0/1935327700 learned_addr learned my addr 172.21.15.113:0/1935327700 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e5ac9700 1 --2- 172.21.15.113:0/1935327700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74e00f8940 0x7f74e010e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e52c8700 1 -- 172.21.15.113:0/1935327700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74e00f8940 msgr2=0x7f74e010e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e52c8700 1 --2- 172.21.15.113:0/1935327700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74e00f8940 0x7f74e010e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e52c8700 1 -- 172.21.15.113:0/1935327700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc005040 con 0x7f74e00f9210 2024-04-12T05:42:14.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.034+0000 7f74e52c8700 1 --2- 172.21.15.113:0/1935327700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74e00f9210 0x7f74e010e520 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7f74dc00b120 tx=0x7f74dc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:14.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.035+0000 7f74d6ffd700 1 -- 172.21.15.113:0/1935327700 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74dc00e030 con 0x7f74e00f9210 2024-04-12T05:42:14.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.035+0000 7f74d6ffd700 1 -- 172.21.15.113:0/1935327700 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74dc007e90 con 0x7f74e00f9210 2024-04-12T05:42:14.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.035+0000 7f74e7d2d700 1 -- 172.21.15.113:0/1935327700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e0108290 con 0x7f74e00f9210 2024-04-12T05:42:14.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.035+0000 7f74d6ffd700 1 -- 172.21.15.113:0/1935327700 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74dc018070 con 0x7f74e00f9210 2024-04-12T05:42:14.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.035+0000 7f74e7d2d700 1 -- 172.21.15.113:0/1935327700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e0108770 con 0x7f74e00f9210 2024-04-12T05:42:14.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.036+0000 7f74e7d2d700 1 -- 172.21.15.113:0/1935327700 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e005f590 con 0x7f74e00f9210 2024-04-12T05:42:14.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.036+0000 7f74d6ffd700 1 -- 172.21.15.113:0/1935327700 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f74dc01d420 con 0x7f74e00f9210 2024-04-12T05:42:14.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.036+0000 7f74d6ffd700 1 --2- 172.21.15.113:0/1935327700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f74cc06caa0 0x7f74cc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:14.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.036+0000 7f74d6ffd700 1 -- 172.21.15.113:0/1935327700 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f74dc08d170 con 0x7f74e00f9210 2024-04-12T05:42:14.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.036+0000 7f74e5ac9700 1 --2- 172.21.15.113:0/1935327700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f74cc06caa0 0x7f74cc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:14.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.037+0000 7f74e5ac9700 1 --2- 172.21.15.113:0/1935327700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f74cc06caa0 0x7f74cc06ef60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f74d00041d0 tx=0x7f74d0009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:14.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.041+0000 7f74d6ffd700 1 -- 172.21.15.113:0/1935327700 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f74dc05ac20 con 0x7f74e00f9210 2024-04-12T05:42:14.244 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.247+0000 7f74e7d2d700 1 -- 172.21.15.113:0/1935327700 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f74e01097c0 con 0x7f74cc06caa0 2024-04-12T05:42:14.245 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.248+0000 7f74d6ffd700 1 -- 172.21.15.113:0/1935327700 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f74e01097c0 con 0x7f74cc06caa0 2024-04-12T05:42:14.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.250+0000 7f74e7d2d700 1 -- 172.21.15.113:0/1935327700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f74cc06caa0 msgr2=0x7f74cc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.250+0000 7f74e7d2d700 1 --2- 172.21.15.113:0/1935327700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f74cc06caa0 0x7f74cc06ef60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f74d00041d0 tx=0x7f74d0009040).stop 2024-04-12T05:42:14.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.250+0000 7f74e7d2d700 1 -- 172.21.15.113:0/1935327700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74e00f9210 msgr2=0x7f74e010e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.250+0000 7f74e7d2d700 1 --2- 172.21.15.113:0/1935327700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74e00f9210 0x7f74e010e520 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7f74dc00b120 tx=0x7f74dc009cb0).stop 2024-04-12T05:42:14.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.250+0000 7f74e7d2d700 1 -- 172.21.15.113:0/1935327700 shutdown_connections 2024-04-12T05:42:14.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.250+0000 7f74e7d2d700 1 --2- 172.21.15.113:0/1935327700 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f74cc06caa0 0x7f74cc06ef60 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.250+0000 7f74e7d2d700 1 --2- 172.21.15.113:0/1935327700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74e00f8940 0x7f74e010e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.250+0000 7f74e7d2d700 1 --2- 172.21.15.113:0/1935327700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74e00f9210 0x7f74e010e520 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.248 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.250+0000 7f74e7d2d700 1 -- 172.21.15.113:0/1935327700 >> 172.21.15.113:0/1935327700 conn(0x7f74e0070590 msgr2=0x7f74e00fc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:14.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.250+0000 7f74e7d2d700 1 -- 172.21.15.113:0/1935327700 shutdown_connections 2024-04-12T05:42:14.249 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.250+0000 7f74e7d2d700 1 -- 172.21.15.113:0/1935327700 wait complete. 2024-04-12T05:42:14.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.350+0000 7fba57a1c700 1 -- 172.21.15.113:0/1250108562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba500f8940 msgr2=0x7fba500f8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.350+0000 7fba57a1c700 1 --2- 172.21.15.113:0/1250108562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba500f8940 0x7fba500f8d20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7fba400047f0 tx=0x7fba40009cb0).stop 2024-04-12T05:42:14.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.350+0000 7fba57a1c700 1 -- 172.21.15.113:0/1250108562 shutdown_connections 2024-04-12T05:42:14.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.350+0000 7fba57a1c700 1 --2- 172.21.15.113:0/1250108562 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba500f9210 0x7fba501019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.350+0000 7fba57a1c700 1 --2- 172.21.15.113:0/1250108562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba500f8940 0x7fba500f8d20 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.350+0000 7fba57a1c700 1 -- 172.21.15.113:0/1250108562 >> 172.21.15.113:0/1250108562 conn(0x7fba50070590 msgr2=0x7fba500f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:14.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.350+0000 7fba57a1c700 1 -- 172.21.15.113:0/1250108562 shutdown_connections 2024-04-12T05:42:14.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.350+0000 7fba57a1c700 1 -- 172.21.15.113:0/1250108562 wait complete. 2024-04-12T05:42:14.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.351+0000 7fba57a1c700 1 Processor -- start 2024-04-12T05:42:14.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.351+0000 7fba57a1c700 1 -- start start 2024-04-12T05:42:14.349 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.351+0000 7fba57a1c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba500f8940 0x7fba5010e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:14.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.351+0000 7fba57a1c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba500f9210 0x7fba5010e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:14.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.351+0000 7fba57a1c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5010eb40 con 0x7fba500f9210 2024-04-12T05:42:14.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.351+0000 7fba57a1c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5010ec80 con 0x7fba500f8940 2024-04-12T05:42:14.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.351+0000 7fba54fb7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba500f9210 0x7fba5010e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:14.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.351+0000 7fba54fb7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba500f9210 0x7fba5010e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36806/0 (socket says 172.21.15.113:36806) 2024-04-12T05:42:14.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.351+0000 7fba54fb7700 1 -- 172.21.15.113:0/2604310220 learned_addr learned my addr 172.21.15.113:0/2604310220 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:14.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.351+0000 7fba557b8700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba500f8940 0x7fba5010e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:14.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.352+0000 7fba54fb7700 1 -- 172.21.15.113:0/2604310220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba500f8940 msgr2=0x7fba5010e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.352+0000 7fba54fb7700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba500f8940 0x7fba5010e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.352+0000 7fba54fb7700 1 -- 172.21.15.113:0/2604310220 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba40005040 con 0x7fba500f9210 2024-04-12T05:42:14.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.352+0000 7fba557b8700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba500f8940 0x7fba5010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:42:14.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.352+0000 7fba54fb7700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba500f9210 0x7fba5010e540 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0x7fba4c0041d0 tx=0x7fba4c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:14.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.352+0000 7fba467fc700 1 -- 172.21.15.113:0/2604310220 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba4c002d90 con 0x7fba500f9210 2024-04-12T05:42:14.352 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.352+0000 7fba467fc700 1 -- 172.21.15.113:0/2604310220 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba4c00f040 con 0x7fba500f9210 2024-04-12T05:42:14.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.352+0000 7fba57a1c700 1 -- 172.21.15.113:0/2604310220 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba50108310 con 0x7fba500f9210 2024-04-12T05:42:14.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.352+0000 7fba467fc700 1 -- 172.21.15.113:0/2604310220 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba4c00d800 con 0x7fba500f9210 2024-04-12T05:42:14.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.352+0000 7fba57a1c700 1 -- 172.21.15.113:0/2604310220 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba50108850 con 0x7fba500f9210 2024-04-12T05:42:14.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.353+0000 7fba57a1c700 1 -- 172.21.15.113:0/2604310220 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5005f590 con 0x7fba500f9210 2024-04-12T05:42:14.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.353+0000 7fba467fc700 1 -- 172.21.15.113:0/2604310220 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fba4c00dac0 con 0x7fba500f9210 2024-04-12T05:42:14.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.354+0000 7fba467fc700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba3c06caa0 0x7fba3c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:14.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.354+0000 7fba467fc700 1 -- 172.21.15.113:0/2604310220 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fba4c08be60 con 0x7fba500f9210 2024-04-12T05:42:14.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.354+0000 7fba557b8700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba3c06caa0 0x7fba3c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:14.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.354+0000 7fba557b8700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba3c06caa0 0x7fba3c06ef60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fba40004540 tx=0x7fba4000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:14.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.358+0000 7fba467fc700 1 -- 172.21.15.113:0/2604310220 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fba4c059910 con 0x7fba500f9210 2024-04-12T05:42:14.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.556+0000 7fba57a1c700 1 -- 172.21.15.113:0/2604310220 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fba50108b30 con 0x7fba3c06caa0 2024-04-12T05:42:14.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.566+0000 7fba467fc700 1 -- 172.21.15.113:0/2604310220 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fba50108b30 con 0x7fba3c06caa0 2024-04-12T05:42:14.564 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:42:14.564 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (30m) 7m ago 41m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:42:14.564 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (27m) 7m ago 41m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:42:14.564 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (27m) 7m ago 40m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:42:14.564 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (29m) 7m ago 41m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (23m) 7m ago 35m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (23m) 7m ago 35m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (22m) 7m ago 35m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (7m) 7m ago 35m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (31m) 7m ago 42m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (30m) 7m ago 40m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (28m) 7m ago 42m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (28m) 7m ago 40m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (30m) 7m ago 40m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (30m) 7m ago 40m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (27m) 7m ago 39m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (26m) 7m ago 39m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (25m) 7m ago 38m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:42:14.565 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (25m) 7m ago 38m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:42:14.566 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (24m) 7m ago 37m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:42:14.566 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (23m) 7m ago 37m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:42:14.566 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (30m) 7m ago 40m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:42:14.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.568+0000 7fba57a1c700 1 -- 172.21.15.113:0/2604310220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba3c06caa0 msgr2=0x7fba3c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.568+0000 7fba57a1c700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba3c06caa0 0x7fba3c06ef60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fba40004540 tx=0x7fba4000b040).stop 2024-04-12T05:42:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.568+0000 7fba57a1c700 1 -- 172.21.15.113:0/2604310220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba500f9210 msgr2=0x7fba5010e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.568+0000 7fba57a1c700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba500f9210 0x7fba5010e540 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0x7fba4c0041d0 tx=0x7fba4c009f60).stop 2024-04-12T05:42:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.568+0000 7fba57a1c700 1 -- 172.21.15.113:0/2604310220 shutdown_connections 2024-04-12T05:42:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.568+0000 7fba57a1c700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fba3c06caa0 0x7fba3c06ef60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.568+0000 7fba57a1c700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba500f8940 0x7fba5010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.568+0000 7fba57a1c700 1 --2- 172.21.15.113:0/2604310220 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba500f9210 0x7fba5010e540 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.568+0000 7fba57a1c700 1 -- 172.21.15.113:0/2604310220 >> 172.21.15.113:0/2604310220 conn(0x7fba50070590 msgr2=0x7fba500fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.568+0000 7fba57a1c700 1 -- 172.21.15.113:0/2604310220 shutdown_connections 2024-04-12T05:42:14.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.568+0000 7fba57a1c700 1 -- 172.21.15.113:0/2604310220 wait complete. 2024-04-12T05:42:14.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.664+0000 7f6f67c23700 1 -- 172.21.15.113:0/302388703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f600fc970 msgr2=0x7f6f600fcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.664+0000 7f6f67c23700 1 --2- 172.21.15.113:0/302388703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f600fc970 0x7f6f600fcd50 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f6f5c0047f0 tx=0x7f6f5c009cb0).stop 2024-04-12T05:42:14.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.664+0000 7f6f67c23700 1 -- 172.21.15.113:0/302388703 shutdown_connections 2024-04-12T05:42:14.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f67c23700 1 --2- 172.21.15.113:0/302388703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f600fd2d0 0x7f6f600f70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f67c23700 1 --2- 172.21.15.113:0/302388703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f600fc970 0x7f6f600fcd50 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f67c23700 1 -- 172.21.15.113:0/302388703 >> 172.21.15.113:0/302388703 conn(0x7f6f6006ce50 msgr2=0x7f6f6006d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f67c23700 1 -- 172.21.15.113:0/302388703 shutdown_connections 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f67c23700 1 -- 172.21.15.113:0/302388703 wait complete. 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f67c23700 1 Processor -- start 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f67c23700 1 -- start start 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f67c23700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f600fc970 0x7f6f6010e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f67c23700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f600fd2d0 0x7f6f6010e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f67c23700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6010ebb0 con 0x7f6f600fc970 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f67c23700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6010ecf0 con 0x7f6f600fd2d0 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.665+0000 7f6f659bf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f600fc970 0x7f6f6010e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f659bf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f600fc970 0x7f6f6010e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36810/0 (socket says 172.21.15.113:36810) 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f659bf700 1 -- 172.21.15.113:0/3393940292 learned_addr learned my addr 172.21.15.113:0/3393940292 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f651be700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f600fd2d0 0x7f6f6010e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:14.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f659bf700 1 -- 172.21.15.113:0/3393940292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f600fd2d0 msgr2=0x7f6f6010e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f659bf700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f600fd2d0 0x7f6f6010e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f659bf700 1 -- 172.21.15.113:0/3393940292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c005040 con 0x7f6f600fc970 2024-04-12T05:42:14.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f651be700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f600fd2d0 0x7f6f6010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:42:14.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f659bf700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f600fc970 0x7f6f6010e0c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f6f5c0099f0 tx=0x7f6f5c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:14.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f56ffd700 1 -- 172.21.15.113:0/3393940292 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f5c00e030 con 0x7f6f600fc970 2024-04-12T05:42:14.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f56ffd700 1 -- 172.21.15.113:0/3393940292 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f5c007e90 con 0x7f6f600fc970 2024-04-12T05:42:14.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f67c23700 1 -- 172.21.15.113:0/3393940292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f60108320 con 0x7f6f600fc970 2024-04-12T05:42:14.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f56ffd700 1 -- 172.21.15.113:0/3393940292 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f5c018070 con 0x7f6f600fc970 2024-04-12T05:42:14.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.666+0000 7f6f67c23700 1 -- 172.21.15.113:0/3393940292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f60108800 con 0x7f6f600fc970 2024-04-12T05:42:14.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.667+0000 7f6f67c23700 1 -- 172.21.15.113:0/3393940292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f60047540 con 0x7f6f600fc970 2024-04-12T05:42:14.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.668+0000 7f6f56ffd700 1 -- 172.21.15.113:0/3393940292 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6f5c01d420 con 0x7f6f600fc970 2024-04-12T05:42:14.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.668+0000 7f6f56ffd700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4c06caa0 0x7f6f4c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:14.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.668+0000 7f6f56ffd700 1 -- 172.21.15.113:0/3393940292 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6f5c08d2f0 con 0x7f6f600fc970 2024-04-12T05:42:14.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.668+0000 7f6f651be700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4c06caa0 0x7f6f4c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:14.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.668+0000 7f6f651be700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4c06caa0 0x7f6f4c06ef60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f6f60109a00 tx=0x7f6f50009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:14.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.672+0000 7f6f56ffd700 1 -- 172.21.15.113:0/3393940292 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6f5c05ada0 con 0x7f6f600fc970 2024-04-12T05:42:14.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.939+0000 7f6f67c23700 1 -- 172.21.15.113:0/3393940292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6f60108e40 con 0x7f6f600fc970 2024-04-12T05:42:14.937 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.939+0000 7f6f56ffd700 1 -- 172.21.15.113:0/3393940292 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f6f5c05a930 con 0x7f6f600fc970 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:42:14.938 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:42:14.939 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:42:14.939 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:42:14.939 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:42:14.939 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:42:14.939 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:42:14.939 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.941+0000 7f6f67c23700 1 -- 172.21.15.113:0/3393940292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4c06caa0 msgr2=0x7f6f4c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.941+0000 7f6f67c23700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4c06caa0 0x7f6f4c06ef60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f6f60109a00 tx=0x7f6f50009040).stop 2024-04-12T05:42:14.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.941+0000 7f6f67c23700 1 -- 172.21.15.113:0/3393940292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f600fc970 msgr2=0x7f6f6010e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:14.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.941+0000 7f6f67c23700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f600fc970 0x7f6f6010e0c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f6f5c0099f0 tx=0x7f6f5c009cb0).stop 2024-04-12T05:42:14.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.941+0000 7f6f67c23700 1 -- 172.21.15.113:0/3393940292 shutdown_connections 2024-04-12T05:42:14.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.941+0000 7f6f67c23700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f4c06caa0 0x7f6f4c06ef60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.941+0000 7f6f67c23700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f600fc970 0x7f6f6010e0c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.941+0000 7f6f67c23700 1 --2- 172.21.15.113:0/3393940292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f600fd2d0 0x7f6f6010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:14.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.941+0000 7f6f67c23700 1 -- 172.21.15.113:0/3393940292 >> 172.21.15.113:0/3393940292 conn(0x7f6f6006ce50 msgr2=0x7f6f600f5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:14.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.942+0000 7f6f67c23700 1 -- 172.21.15.113:0/3393940292 shutdown_connections 2024-04-12T05:42:14.940 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:14.942+0000 7f6f67c23700 1 -- 172.21.15.113:0/3393940292 wait complete. 2024-04-12T05:42:15.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.038+0000 7f75fd565700 1 -- 172.21.15.113:0/3859932641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75f80f88a0 msgr2=0x7f75f80f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.038+0000 7f75fd565700 1 --2- 172.21.15.113:0/3859932641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75f80f88a0 0x7f75f80f8c80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f75e80047f0 tx=0x7f75e8009cb0).stop 2024-04-12T05:42:15.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.038+0000 7f75fd565700 1 -- 172.21.15.113:0/3859932641 shutdown_connections 2024-04-12T05:42:15.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.038+0000 7f75fd565700 1 --2- 172.21.15.113:0/3859932641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75f80f9200 0x7f75f8101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.037 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.038+0000 7f75fd565700 1 --2- 172.21.15.113:0/3859932641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75f80f88a0 0x7f75f80f8c80 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.038+0000 7f75fd565700 1 -- 172.21.15.113:0/3859932641 >> 172.21.15.113:0/3859932641 conn(0x7f75f80f2390 msgr2=0x7f75f80f47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:15.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.038+0000 7f75fd565700 1 -- 172.21.15.113:0/3859932641 shutdown_connections 2024-04-12T05:42:15.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.039+0000 7f75fd565700 1 -- 172.21.15.113:0/3859932641 wait complete. 2024-04-12T05:42:15.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.039+0000 7f75fd565700 1 Processor -- start 2024-04-12T05:42:15.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.039+0000 7f75fd565700 1 -- start start 2024-04-12T05:42:15.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.039+0000 7f75fd565700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75f80f88a0 0x7f75f810e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:15.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.039+0000 7f75fd565700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75f80f9200 0x7f75f810e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:15.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.039+0000 7f75fd565700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f810eb50 con 0x7f75f80f88a0 2024-04-12T05:42:15.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75fd565700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f810ec90 con 0x7f75f80f9200 2024-04-12T05:42:15.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75f6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75f80f88a0 0x7f75f810e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:15.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75f6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75f80f88a0 0x7f75f810e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36826/0 (socket says 172.21.15.113:36826) 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75f6ffd700 1 -- 172.21.15.113:0/1894485885 learned_addr learned my addr 172.21.15.113:0/1894485885 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75f67fc700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75f80f9200 0x7f75f810e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75f6ffd700 1 -- 172.21.15.113:0/1894485885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75f80f9200 msgr2=0x7f75f810e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75f6ffd700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75f80f9200 0x7f75f810e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75f6ffd700 1 -- 172.21.15.113:0/1894485885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e8005040 con 0x7f75f80f88a0 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75f67fc700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75f80f9200 0x7f75f810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75f6ffd700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75f80f88a0 0x7f75f810e060 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f75e8005010 tx=0x7f75e8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75dffff700 1 -- 172.21.15.113:0/1894485885 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75e800e030 con 0x7f75f80f88a0 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75fd565700 1 -- 172.21.15.113:0/1894485885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f81082c0 con 0x7f75f80f88a0 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75dffff700 1 -- 172.21.15.113:0/1894485885 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75e8007e90 con 0x7f75f80f88a0 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75dffff700 1 -- 172.21.15.113:0/1894485885 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75e8018070 con 0x7f75f80f88a0 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.040+0000 7f75fd565700 1 -- 172.21.15.113:0/1894485885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f81087a0 con 0x7f75f80f88a0 2024-04-12T05:42:15.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.041+0000 7f75fd565700 1 -- 172.21.15.113:0/1894485885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f8060390 con 0x7f75f80f88a0 2024-04-12T05:42:15.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.042+0000 7f75dffff700 1 -- 172.21.15.113:0/1894485885 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f75e801d6e0 con 0x7f75f80f88a0 2024-04-12T05:42:15.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.042+0000 7f75dffff700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f75e006ca30 0x7f75e006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:15.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.042+0000 7f75dffff700 1 -- 172.21.15.113:0/1894485885 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f75e808d170 con 0x7f75f80f88a0 2024-04-12T05:42:15.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.042+0000 7f75f67fc700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f75e006ca30 0x7f75e006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:15.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.042+0000 7f75f67fc700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f75e006ca30 0x7f75e006eef0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f75f81099a0 tx=0x7f75ec009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:15.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.047+0000 7f75dffff700 1 -- 172.21.15.113:0/1894485885 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f75e805ac20 con 0x7f75f80f88a0 2024-04-12T05:42:15.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.277+0000 7f75fd565700 1 -- 172.21.15.113:0/1894485885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f75f8047540 con 0x7f75f80f88a0 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.278+0000 7f75dffff700 1 -- 172.21.15.113:0/1894485885 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f75e805a7b0 con 0x7f75f80f88a0 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:42:15.276 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:42:15.277 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:15.278 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:15.279 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:15.279 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:15.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.280+0000 7f75fd565700 1 -- 172.21.15.113:0/1894485885 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f75e006ca30 msgr2=0x7f75e006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.280+0000 7f75fd565700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f75e006ca30 0x7f75e006eef0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f75f81099a0 tx=0x7f75ec009da0).stop 2024-04-12T05:42:15.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.280+0000 7f75fd565700 1 -- 172.21.15.113:0/1894485885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75f80f88a0 msgr2=0x7f75f810e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.280+0000 7f75fd565700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75f80f88a0 0x7f75f810e060 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f75e8005010 tx=0x7f75e8009cb0).stop 2024-04-12T05:42:15.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.280+0000 7f75fd565700 1 -- 172.21.15.113:0/1894485885 shutdown_connections 2024-04-12T05:42:15.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.280+0000 7f75fd565700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f75e006ca30 0x7f75e006eef0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.280+0000 7f75fd565700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f75f80f88a0 0x7f75f810e060 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.280+0000 7f75fd565700 1 --2- 172.21.15.113:0/1894485885 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75f80f9200 0x7f75f810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.280+0000 7f75fd565700 1 -- 172.21.15.113:0/1894485885 >> 172.21.15.113:0/1894485885 conn(0x7f75f80f2390 msgr2=0x7f75f80fc170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:15.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.280+0000 7f75fd565700 1 -- 172.21.15.113:0/1894485885 shutdown_connections 2024-04-12T05:42:15.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.280+0000 7f75fd565700 1 -- 172.21.15.113:0/1894485885 wait complete. 2024-04-12T05:42:15.280 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:42:15.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.379+0000 7fbe86a06700 1 -- 172.21.15.113:0/1001781770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe800fd150 msgr2=0x7fbe800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.379+0000 7fbe86a06700 1 --2- 172.21.15.113:0/1001781770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe800fd150 0x7fbe800f6930 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0x7fbe700047f0 tx=0x7fbe70009cb0).stop 2024-04-12T05:42:15.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.380+0000 7fbe86a06700 1 -- 172.21.15.113:0/1001781770 shutdown_connections 2024-04-12T05:42:15.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.380+0000 7fbe86a06700 1 --2- 172.21.15.113:0/1001781770 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe800fd150 0x7fbe800f6930 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.380+0000 7fbe86a06700 1 --2- 172.21.15.113:0/1001781770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe800fc7f0 0x7fbe800fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.380+0000 7fbe86a06700 1 -- 172.21.15.113:0/1001781770 >> 172.21.15.113:0/1001781770 conn(0x7fbe80070590 msgr2=0x7fbe800f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:15.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.380+0000 7fbe86a06700 1 -- 172.21.15.113:0/1001781770 shutdown_connections 2024-04-12T05:42:15.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.380+0000 7fbe86a06700 1 -- 172.21.15.113:0/1001781770 wait complete. 2024-04-12T05:42:15.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.380+0000 7fbe86a06700 1 Processor -- start 2024-04-12T05:42:15.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.380+0000 7fbe86a06700 1 -- start start 2024-04-12T05:42:15.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe86a06700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe800fc7f0 0x7fbe8010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:15.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe86a06700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe800fd150 0x7fbe8010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:15.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe86a06700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8010ead0 con 0x7fbe800fc7f0 2024-04-12T05:42:15.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe86a06700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8010ec10 con 0x7fbe800fd150 2024-04-12T05:42:15.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe7ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe800fc7f0 0x7fbe8010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:15.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe7ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe800fc7f0 0x7fbe8010dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36848/0 (socket says 172.21.15.113:36848) 2024-04-12T05:42:15.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe7ffff700 1 -- 172.21.15.113:0/2865506302 learned_addr learned my addr 172.21.15.113:0/2865506302 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:15.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe7f7fe700 1 --2- 172.21.15.113:0/2865506302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe800fd150 0x7fbe8010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe7ffff700 1 -- 172.21.15.113:0/2865506302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe800fd150 msgr2=0x7fbe8010e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe7ffff700 1 --2- 172.21.15.113:0/2865506302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe800fd150 0x7fbe8010e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe7ffff700 1 -- 172.21.15.113:0/2865506302 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70005040 con 0x7fbe800fc7f0 2024-04-12T05:42:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe7ffff700 1 --2- 172.21.15.113:0/2865506302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe800fc7f0 0x7fbe8010dfe0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7fbe740041d0 tx=0x7fbe74009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe7d7fa700 1 -- 172.21.15.113:0/2865506302 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe74007b40 con 0x7fbe800fc7f0 2024-04-12T05:42:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe7d7fa700 1 -- 172.21.15.113:0/2865506302 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe7400f040 con 0x7fbe800fc7f0 2024-04-12T05:42:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe86a06700 1 -- 172.21.15.113:0/2865506302 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe801082a0 con 0x7fbe800fc7f0 2024-04-12T05:42:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe7d7fa700 1 -- 172.21.15.113:0/2865506302 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe740137d0 con 0x7fbe800fc7f0 2024-04-12T05:42:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.381+0000 7fbe86a06700 1 -- 172.21.15.113:0/2865506302 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe80108860 con 0x7fbe800fc7f0 2024-04-12T05:42:15.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.382+0000 7fbe86a06700 1 -- 172.21.15.113:0/2865506302 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8005f590 con 0x7fbe800fc7f0 2024-04-12T05:42:15.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.383+0000 7fbe7d7fa700 1 -- 172.21.15.113:0/2865506302 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbe74007ca0 con 0x7fbe800fc7f0 2024-04-12T05:42:15.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.383+0000 7fbe7d7fa700 1 --2- 172.21.15.113:0/2865506302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe6806caa0 0x7fbe6806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:15.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.383+0000 7fbe7f7fe700 1 --2- 172.21.15.113:0/2865506302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe6806caa0 0x7fbe6806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:15.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.383+0000 7fbe7d7fa700 1 -- 172.21.15.113:0/2865506302 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbe7408b860 con 0x7fbe800fc7f0 2024-04-12T05:42:15.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.384+0000 7fbe7f7fe700 1 --2- 172.21.15.113:0/2865506302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe6806caa0 0x7fbe6806ef60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fbe80109920 tx=0x7fbe7000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:15.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.388+0000 7fbe7d7fa700 1 -- 172.21.15.113:0/2865506302 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbe74059310 con 0x7fbe800fc7f0 2024-04-12T05:42:15.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:15 smithi113 ceph-mon[103371]: pgmap v903: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:15.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:15 smithi113 ceph-mon[103371]: from='client.35626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:15.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:15 smithi113 ceph-mon[103371]: from='client.35630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:15.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:15 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3393940292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:15.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:15 smithi192 ceph-mon[85304]: pgmap v903: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:15.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:15 smithi192 ceph-mon[85304]: from='client.35626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:15.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:15 smithi192 ceph-mon[85304]: from='client.35630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:15.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:15 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3393940292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:15.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.592+0000 7fbe86a06700 1 -- 172.21.15.113:0/2865506302 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe80108f00 con 0x7fbe6806caa0 2024-04-12T05:42:15.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.593+0000 7fbe7d7fa700 1 -- 172.21.15.113:0/2865506302 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbe80108f00 con 0x7fbe6806caa0 2024-04-12T05:42:15.591 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:42:15.591 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:42:15.591 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:42:15.591 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:42:15.592 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:42:15.592 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:42:15.592 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:42:15.592 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:42:15.592 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:42:15.592 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:42:15.592 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:42:15.592 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:42:15.592 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:42:15.592 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:42:15.592 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:42:15.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.595+0000 7fbe86a06700 1 -- 172.21.15.113:0/2865506302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe6806caa0 msgr2=0x7fbe6806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.595+0000 7fbe86a06700 1 --2- 172.21.15.113:0/2865506302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe6806caa0 0x7fbe6806ef60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fbe80109920 tx=0x7fbe7000b040).stop 2024-04-12T05:42:15.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.595+0000 7fbe86a06700 1 -- 172.21.15.113:0/2865506302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe800fc7f0 msgr2=0x7fbe8010dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.595+0000 7fbe86a06700 1 --2- 172.21.15.113:0/2865506302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe800fc7f0 0x7fbe8010dfe0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7fbe740041d0 tx=0x7fbe74009f60).stop 2024-04-12T05:42:15.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.595+0000 7fbe86a06700 1 -- 172.21.15.113:0/2865506302 shutdown_connections 2024-04-12T05:42:15.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.595+0000 7fbe86a06700 1 --2- 172.21.15.113:0/2865506302 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbe6806caa0 0x7fbe6806ef60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.595+0000 7fbe86a06700 1 --2- 172.21.15.113:0/2865506302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe800fc7f0 0x7fbe8010dfe0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.595+0000 7fbe86a06700 1 --2- 172.21.15.113:0/2865506302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbe800fd150 0x7fbe8010e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.595+0000 7fbe86a06700 1 -- 172.21.15.113:0/2865506302 >> 172.21.15.113:0/2865506302 conn(0x7fbe80070590 msgr2=0x7fbe80060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:15.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.596+0000 7fbe86a06700 1 -- 172.21.15.113:0/2865506302 shutdown_connections 2024-04-12T05:42:15.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.596+0000 7fbe86a06700 1 -- 172.21.15.113:0/2865506302 wait complete. 2024-04-12T05:42:15.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.692+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/4217273372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3e00fe250 msgr2=0x7fe3e00f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.692+0000 7fe3e7cf8700 1 --2- 172.21.15.113:0/4217273372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3e00fe250 0x7fe3e00f7a30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7fe3d00047f0 tx=0x7fe3d0009cb0).stop 2024-04-12T05:42:15.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.692+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/4217273372 shutdown_connections 2024-04-12T05:42:15.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.692+0000 7fe3e7cf8700 1 --2- 172.21.15.113:0/4217273372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3e00fe250 0x7fe3e00f7a30 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.692+0000 7fe3e7cf8700 1 --2- 172.21.15.113:0/4217273372 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e00fd8f0 0x7fe3e00fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.692+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/4217273372 >> 172.21.15.113:0/4217273372 conn(0x7fe3e00f34b0 msgr2=0x7fe3e00f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:15.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.692+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/4217273372 shutdown_connections 2024-04-12T05:42:15.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.692+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/4217273372 wait complete. 2024-04-12T05:42:15.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.693+0000 7fe3e7cf8700 1 Processor -- start 2024-04-12T05:42:15.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.693+0000 7fe3e7cf8700 1 -- start start 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.693+0000 7fe3e7cf8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3e00fd8f0 0x7fe3e006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.693+0000 7fe3e7cf8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e00fe250 0x7fe3e006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.693+0000 7fe3e7cf8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e006bab0 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.693+0000 7fe3e7cf8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e006bbf0 con 0x7fe3e00fe250 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.693+0000 7fe3e5a94700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3e00fd8f0 0x7fe3e006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.693+0000 7fe3e5a94700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3e00fd8f0 0x7fe3e006b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36866/0 (socket says 172.21.15.113:36866) 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.693+0000 7fe3e5a94700 1 -- 172.21.15.113:0/26141175 learned_addr learned my addr 172.21.15.113:0/26141175 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.694+0000 7fe3e5293700 1 --2- 172.21.15.113:0/26141175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e00fe250 0x7fe3e006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.694+0000 7fe3e5a94700 1 -- 172.21.15.113:0/26141175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e00fe250 msgr2=0x7fe3e006b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.694+0000 7fe3e5a94700 1 --2- 172.21.15.113:0/26141175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e00fe250 0x7fe3e006b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.694+0000 7fe3e5a94700 1 -- 172.21.15.113:0/26141175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d0005040 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.694+0000 7fe3e5a94700 1 --2- 172.21.15.113:0/26141175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3e00fd8f0 0x7fe3e006b0d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7fe3dc0041d0 tx=0x7fe3dc009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.694+0000 7fe3d6ffd700 1 -- 172.21.15.113:0/26141175 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3dc007b40 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.694+0000 7fe3d6ffd700 1 -- 172.21.15.113:0/26141175 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3dc00f040 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.694+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/26141175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e006bea0 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.694+0000 7fe3d6ffd700 1 -- 172.21.15.113:0/26141175 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3dc0137d0 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.694+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/26141175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e01146d0 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.695+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/26141175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e005f590 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.695+0000 7fe3d6ffd700 1 -- 172.21.15.113:0/26141175 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe3dc007ca0 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.696+0000 7fe3d6ffd700 1 --2- 172.21.15.113:0/26141175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe3cc06caa0 0x7fe3cc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:15.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.696+0000 7fe3e5293700 1 --2- 172.21.15.113:0/26141175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe3cc06caa0 0x7fe3cc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:15.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.696+0000 7fe3d6ffd700 1 -- 172.21.15.113:0/26141175 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe3dc08b840 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.696+0000 7fe3e5293700 1 --2- 172.21.15.113:0/26141175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe3cc06caa0 0x7fe3cc06ef60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fe3e006c870 tx=0x7fe3d000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:15.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.700+0000 7fe3d6ffd700 1 -- 172.21.15.113:0/26141175 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe3dc055860 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.966+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/26141175 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe3e006c650 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.966+0000 7fe3d6ffd700 1 -- 172.21.15.113:0/26141175 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fe3dc058e80 con 0x7fe3e00fd8f0 2024-04-12T05:42:15.964 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:42:15.965 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:42:15.966 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:42:15.966 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:42:15.966 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:42:15.966 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:42:15.966 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:42:15.966 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:42:15.966 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:42:15.966 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:42:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.968+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/26141175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe3cc06caa0 msgr2=0x7fe3cc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.968+0000 7fe3e7cf8700 1 --2- 172.21.15.113:0/26141175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe3cc06caa0 0x7fe3cc06ef60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fe3e006c870 tx=0x7fe3d000b040).stop 2024-04-12T05:42:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.969+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/26141175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3e00fd8f0 msgr2=0x7fe3e006b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.969+0000 7fe3e7cf8700 1 --2- 172.21.15.113:0/26141175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3e00fd8f0 0x7fe3e006b0d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7fe3dc0041d0 tx=0x7fe3dc009f60).stop 2024-04-12T05:42:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.969+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/26141175 shutdown_connections 2024-04-12T05:42:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.969+0000 7fe3e7cf8700 1 --2- 172.21.15.113:0/26141175 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe3cc06caa0 0x7fe3cc06ef60 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.969+0000 7fe3e7cf8700 1 --2- 172.21.15.113:0/26141175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3e00fd8f0 0x7fe3e006b0d0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.969+0000 7fe3e7cf8700 1 --2- 172.21.15.113:0/26141175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3e00fe250 0x7fe3e006b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.969+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/26141175 >> 172.21.15.113:0/26141175 conn(0x7fe3e00f34b0 msgr2=0x7fe3e00fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.969+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/26141175 shutdown_connections 2024-04-12T05:42:15.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:15.969+0000 7fe3e7cf8700 1 -- 172.21.15.113:0/26141175 wait complete. 2024-04-12T05:42:16.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:16 smithi113 ceph-mon[103371]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:16.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:16 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1894485885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:42:16.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:16 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/26141175' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:42:16.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:16 smithi192 ceph-mon[85304]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:16.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:16 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1894485885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:42:16.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:16 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/26141175' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:42:17.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:17 smithi113 ceph-mon[103371]: from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:17.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:17 smithi113 ceph-mon[103371]: pgmap v904: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:17.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:17 smithi192 ceph-mon[85304]: from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:17.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:17 smithi192 ceph-mon[85304]: pgmap v904: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:19.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:19 smithi113 ceph-mon[103371]: pgmap v905: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:19.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:19 smithi192 ceph-mon[85304]: pgmap v905: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:21.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:21 smithi113 ceph-mon[103371]: pgmap v906: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:21.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:21 smithi192 ceph-mon[85304]: pgmap v906: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:22.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:22.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:23.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:23 smithi113 ceph-mon[103371]: pgmap v907: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:23.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:23 smithi192 ceph-mon[85304]: pgmap v907: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:25.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:25 smithi113 ceph-mon[103371]: pgmap v908: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:25.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:25 smithi192 ceph-mon[85304]: pgmap v908: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:27.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:27 smithi113 ceph-mon[103371]: pgmap v909: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:27.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:27 smithi192 ceph-mon[85304]: pgmap v909: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:29.422 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:29 smithi113 ceph-mon[103371]: pgmap v910: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:29.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:29 smithi192 ceph-mon[85304]: pgmap v910: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:31.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:31 smithi192 ceph-mon[85304]: pgmap v911: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:31.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:31 smithi113 ceph-mon[103371]: pgmap v911: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:33.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:33 smithi192 ceph-mon[85304]: pgmap v912: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:33.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:33 smithi113 ceph-mon[103371]: pgmap v912: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:35.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:35 smithi192 ceph-mon[85304]: pgmap v913: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:35.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:35 smithi113 ceph-mon[103371]: pgmap v913: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:37.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:37 smithi192 ceph-mon[85304]: pgmap v914: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:37.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:37.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:37 smithi113 ceph-mon[103371]: pgmap v914: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:37.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:39.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:39 smithi192 ceph-mon[85304]: pgmap v915: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:39 smithi113 ceph-mon[103371]: pgmap v915: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:41.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:41 smithi192 ceph-mon[85304]: pgmap v916: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:41 smithi113 ceph-mon[103371]: pgmap v916: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:43.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:43 smithi192 ceph-mon[85304]: pgmap v917: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:43.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:43 smithi113 ceph-mon[103371]: pgmap v917: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:45.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:45 smithi192 ceph-mon[85304]: pgmap v918: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:45.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:45 smithi113 ceph-mon[103371]: pgmap v918: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:46.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.081+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2487924249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5c0fc7f0 msgr2=0x7ffb5c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.081+0000 7ffb60dc4700 1 --2- 172.21.15.113:0/2487924249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5c0fc7f0 0x7ffb5c0fcbd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7ffb4c0047f0 tx=0x7ffb4c009cb0).stop 2024-04-12T05:42:46.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.081+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2487924249 shutdown_connections 2024-04-12T05:42:46.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.081+0000 7ffb60dc4700 1 --2- 172.21.15.113:0/2487924249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb5c0fd150 0x7ffb5c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.081+0000 7ffb60dc4700 1 --2- 172.21.15.113:0/2487924249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5c0fc7f0 0x7ffb5c0fcbd0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.081+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2487924249 >> 172.21.15.113:0/2487924249 conn(0x7ffb5c070590 msgr2=0x7ffb5c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:46.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.081+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2487924249 shutdown_connections 2024-04-12T05:42:46.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.081+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2487924249 wait complete. 2024-04-12T05:42:46.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.082+0000 7ffb60dc4700 1 Processor -- start 2024-04-12T05:42:46.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.082+0000 7ffb60dc4700 1 -- start start 2024-04-12T05:42:46.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.082+0000 7ffb60dc4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5c0fc7f0 0x7ffb5c10cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:46.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.082+0000 7ffb60dc4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb5c0fd150 0x7ffb5c10d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:46.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.082+0000 7ffb60dc4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c10d9f0 con 0x7ffb5c0fc7f0 2024-04-12T05:42:46.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.082+0000 7ffb60dc4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c10db30 con 0x7ffb5c0fd150 2024-04-12T05:42:46.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.082+0000 7ffb5a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5c0fc7f0 0x7ffb5c10cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.082+0000 7ffb5a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5c0fc7f0 0x7ffb5c10cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43226/0 (socket says 172.21.15.113:43226) 2024-04-12T05:42:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.082+0000 7ffb5a59c700 1 -- 172.21.15.113:0/2687082787 learned_addr learned my addr 172.21.15.113:0/2687082787 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.083+0000 7ffb59d9b700 1 --2- 172.21.15.113:0/2687082787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb5c0fd150 0x7ffb5c10d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.083+0000 7ffb5a59c700 1 -- 172.21.15.113:0/2687082787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb5c0fd150 msgr2=0x7ffb5c10d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.083+0000 7ffb5a59c700 1 --2- 172.21.15.113:0/2687082787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb5c0fd150 0x7ffb5c10d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.083+0000 7ffb5a59c700 1 -- 172.21.15.113:0/2687082787 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c005040 con 0x7ffb5c0fc7f0 2024-04-12T05:42:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.083+0000 7ffb5a59c700 1 --2- 172.21.15.113:0/2687082787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5c0fc7f0 0x7ffb5c10cf00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7ffb4c005010 tx=0x7ffb4c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.083+0000 7ffb4b7fe700 1 -- 172.21.15.113:0/2687082787 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb4c00e030 con 0x7ffb5c0fc7f0 2024-04-12T05:42:46.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.083+0000 7ffb4b7fe700 1 -- 172.21.15.113:0/2687082787 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb4c007e90 con 0x7ffb5c0fc7f0 2024-04-12T05:42:46.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.083+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2687082787 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5c107160 con 0x7ffb5c0fc7f0 2024-04-12T05:42:46.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.083+0000 7ffb4b7fe700 1 -- 172.21.15.113:0/2687082787 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb4c018070 con 0x7ffb5c0fc7f0 2024-04-12T05:42:46.082 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.083+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2687082787 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c1076c0 con 0x7ffb5c0fc7f0 2024-04-12T05:42:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.084+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2687082787 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5c05f590 con 0x7ffb5c0fc7f0 2024-04-12T05:42:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.085+0000 7ffb4b7fe700 1 -- 172.21.15.113:0/2687082787 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ffb4c01d670 con 0x7ffb5c0fc7f0 2024-04-12T05:42:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.085+0000 7ffb4b7fe700 1 --2- 172.21.15.113:0/2687082787 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb4406caa0 0x7ffb4406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.085+0000 7ffb4b7fe700 1 -- 172.21.15.113:0/2687082787 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ffb4c08d0d0 con 0x7ffb5c0fc7f0 2024-04-12T05:42:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.085+0000 7ffb59d9b700 1 --2- 172.21.15.113:0/2687082787 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb4406caa0 0x7ffb4406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:46.083 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.085+0000 7ffb59d9b700 1 --2- 172.21.15.113:0/2687082787 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb4406caa0 0x7ffb4406ef60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7ffb5c108840 tx=0x7ffb5000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:46.086 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.089+0000 7ffb4b7fe700 1 -- 172.21.15.113:0/2687082787 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ffb4c05ab80 con 0x7ffb5c0fc7f0 2024-04-12T05:42:46.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.295+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2687082787 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb5c0feb40 con 0x7ffb4406caa0 2024-04-12T05:42:46.293 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.297+0000 7ffb4b7fe700 1 -- 172.21.15.113:0/2687082787 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ffb5c0feb40 con 0x7ffb4406caa0 2024-04-12T05:42:46.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.299+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2687082787 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb4406caa0 msgr2=0x7ffb4406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.299+0000 7ffb60dc4700 1 --2- 172.21.15.113:0/2687082787 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb4406caa0 0x7ffb4406ef60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7ffb5c108840 tx=0x7ffb5000b040).stop 2024-04-12T05:42:46.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.299+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2687082787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5c0fc7f0 msgr2=0x7ffb5c10cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.299+0000 7ffb60dc4700 1 --2- 172.21.15.113:0/2687082787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5c0fc7f0 0x7ffb5c10cf00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7ffb4c005010 tx=0x7ffb4c009cb0).stop 2024-04-12T05:42:46.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.299+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2687082787 shutdown_connections 2024-04-12T05:42:46.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.299+0000 7ffb60dc4700 1 --2- 172.21.15.113:0/2687082787 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ffb4406caa0 0x7ffb4406ef60 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.299+0000 7ffb60dc4700 1 --2- 172.21.15.113:0/2687082787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb5c0fc7f0 0x7ffb5c10cf00 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.299+0000 7ffb60dc4700 1 --2- 172.21.15.113:0/2687082787 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb5c0fd150 0x7ffb5c10d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.299+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2687082787 >> 172.21.15.113:0/2687082787 conn(0x7ffb5c070590 msgr2=0x7ffb5c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:46.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.299+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2687082787 shutdown_connections 2024-04-12T05:42:46.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.299+0000 7ffb60dc4700 1 -- 172.21.15.113:0/2687082787 wait complete. 2024-04-12T05:42:46.306 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:42:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.397+0000 7fd83b283700 1 -- 172.21.15.113:0/2986709305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8340fd150 msgr2=0x7fd8340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.395 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.397+0000 7fd83b283700 1 --2- 172.21.15.113:0/2986709305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8340fd150 0x7fd8340f6930 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7fd8300047f0 tx=0x7fd830009cb0).stop 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.397+0000 7fd83b283700 1 -- 172.21.15.113:0/2986709305 shutdown_connections 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.397+0000 7fd83b283700 1 --2- 172.21.15.113:0/2986709305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8340fd150 0x7fd8340f6930 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.397+0000 7fd83b283700 1 --2- 172.21.15.113:0/2986709305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8340fc7f0 0x7fd8340fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.397+0000 7fd83b283700 1 -- 172.21.15.113:0/2986709305 >> 172.21.15.113:0/2986709305 conn(0x7fd8340f23b0 msgr2=0x7fd8340f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.397+0000 7fd83b283700 1 -- 172.21.15.113:0/2986709305 shutdown_connections 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.397+0000 7fd83b283700 1 -- 172.21.15.113:0/2986709305 wait complete. 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83b283700 1 Processor -- start 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83b283700 1 -- start start 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83b283700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8340fc7f0 0x7fd83410e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83b283700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8340fd150 0x7fd83410e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83b283700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83410eb50 con 0x7fd8340fc7f0 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83b283700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83410ec90 con 0x7fd8340fd150 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83901f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8340fc7f0 0x7fd83410e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83901f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8340fc7f0 0x7fd83410e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43250/0 (socket says 172.21.15.113:43250) 2024-04-12T05:42:46.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83901f700 1 -- 172.21.15.113:0/946208785 learned_addr learned my addr 172.21.15.113:0/946208785 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:46.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83881e700 1 --2- 172.21.15.113:0/946208785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8340fd150 0x7fd83410e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:46.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83901f700 1 -- 172.21.15.113:0/946208785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8340fd150 msgr2=0x7fd83410e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83901f700 1 --2- 172.21.15.113:0/946208785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8340fd150 0x7fd83410e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.398+0000 7fd83901f700 1 -- 172.21.15.113:0/946208785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd830005040 con 0x7fd8340fc7f0 2024-04-12T05:42:46.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.399+0000 7fd83901f700 1 --2- 172.21.15.113:0/946208785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8340fc7f0 0x7fd83410e060 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fd8240041d0 tx=0x7fd824009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:46.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.399+0000 7fd82a777700 1 -- 172.21.15.113:0/946208785 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd824007b40 con 0x7fd8340fc7f0 2024-04-12T05:42:46.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.399+0000 7fd83b283700 1 -- 172.21.15.113:0/946208785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd834108320 con 0x7fd8340fc7f0 2024-04-12T05:42:46.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.399+0000 7fd82a777700 1 -- 172.21.15.113:0/946208785 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd82400f040 con 0x7fd8340fc7f0 2024-04-12T05:42:46.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.399+0000 7fd82a777700 1 -- 172.21.15.113:0/946208785 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8240137d0 con 0x7fd8340fc7f0 2024-04-12T05:42:46.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.399+0000 7fd83b283700 1 -- 172.21.15.113:0/946208785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd834108860 con 0x7fd8340fc7f0 2024-04-12T05:42:46.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.400+0000 7fd83b283700 1 -- 172.21.15.113:0/946208785 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd834060390 con 0x7fd8340fc7f0 2024-04-12T05:42:46.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.400+0000 7fd82a777700 1 -- 172.21.15.113:0/946208785 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd824007ca0 con 0x7fd8340fc7f0 2024-04-12T05:42:46.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.400+0000 7fd82a777700 1 --2- 172.21.15.113:0/946208785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd82006caa0 0x7fd82006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:46.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.401+0000 7fd82a777700 1 -- 172.21.15.113:0/946208785 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd82408b860 con 0x7fd8340fc7f0 2024-04-12T05:42:46.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.401+0000 7fd83881e700 1 --2- 172.21.15.113:0/946208785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd82006caa0 0x7fd82006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:46.398 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.401+0000 7fd83881e700 1 --2- 172.21.15.113:0/946208785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd82006caa0 0x7fd82006ef60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd8341099a0 tx=0x7fd83000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:46.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.405+0000 7fd82a777700 1 -- 172.21.15.113:0/946208785 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd824059310 con 0x7fd8340fc7f0 2024-04-12T05:42:46.608 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.611+0000 7fd83b283700 1 -- 172.21.15.113:0/946208785 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd834108f00 con 0x7fd82006caa0 2024-04-12T05:42:46.609 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.612+0000 7fd82a777700 1 -- 172.21.15.113:0/946208785 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd834108f00 con 0x7fd82006caa0 2024-04-12T05:42:46.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.614+0000 7fd83b283700 1 -- 172.21.15.113:0/946208785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd82006caa0 msgr2=0x7fd82006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.614+0000 7fd83b283700 1 --2- 172.21.15.113:0/946208785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd82006caa0 0x7fd82006ef60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd8341099a0 tx=0x7fd83000b040).stop 2024-04-12T05:42:46.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.614+0000 7fd83b283700 1 -- 172.21.15.113:0/946208785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8340fc7f0 msgr2=0x7fd83410e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.614+0000 7fd83b283700 1 --2- 172.21.15.113:0/946208785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8340fc7f0 0x7fd83410e060 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fd8240041d0 tx=0x7fd824009f60).stop 2024-04-12T05:42:46.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.614+0000 7fd83b283700 1 -- 172.21.15.113:0/946208785 shutdown_connections 2024-04-12T05:42:46.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.614+0000 7fd83b283700 1 --2- 172.21.15.113:0/946208785 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd82006caa0 0x7fd82006ef60 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.614+0000 7fd83b283700 1 --2- 172.21.15.113:0/946208785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8340fc7f0 0x7fd83410e060 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.614+0000 7fd83b283700 1 --2- 172.21.15.113:0/946208785 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8340fd150 0x7fd83410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.614+0000 7fd83b283700 1 -- 172.21.15.113:0/946208785 >> 172.21.15.113:0/946208785 conn(0x7fd8340f23b0 msgr2=0x7fd834061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:46.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.614+0000 7fd83b283700 1 -- 172.21.15.113:0/946208785 shutdown_connections 2024-04-12T05:42:46.612 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.614+0000 7fd83b283700 1 -- 172.21.15.113:0/946208785 wait complete. 2024-04-12T05:42:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.712+0000 7f1617b6e700 1 -- 172.21.15.113:0/4014453801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100fd150 msgr2=0x7f16100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.712+0000 7f1617b6e700 1 --2- 172.21.15.113:0/4014453801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100fd150 0x7f16100f6930 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7f160c0047f0 tx=0x7f160c009cb0).stop 2024-04-12T05:42:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.712+0000 7f1617b6e700 1 -- 172.21.15.113:0/4014453801 shutdown_connections 2024-04-12T05:42:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.712+0000 7f1617b6e700 1 --2- 172.21.15.113:0/4014453801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100fd150 0x7f16100f6930 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.712+0000 7f1617b6e700 1 --2- 172.21.15.113:0/4014453801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16100fc7f0 0x7f16100fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.712+0000 7f1617b6e700 1 -- 172.21.15.113:0/4014453801 >> 172.21.15.113:0/4014453801 conn(0x7f1610070590 msgr2=0x7f16100f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.712+0000 7f1617b6e700 1 -- 172.21.15.113:0/4014453801 shutdown_connections 2024-04-12T05:42:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.712+0000 7f1617b6e700 1 -- 172.21.15.113:0/4014453801 wait complete. 2024-04-12T05:42:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f1617b6e700 1 Processor -- start 2024-04-12T05:42:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f1617b6e700 1 -- start start 2024-04-12T05:42:46.711 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f1617b6e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100fc7f0 0x7f161010dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f1617b6e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16100fd150 0x7f161010e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f1617b6e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161010eab0 con 0x7f16100fc7f0 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f1617b6e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161010ebf0 con 0x7f16100fd150 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f161590a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100fc7f0 0x7f161010dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f161590a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100fc7f0 0x7f161010dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43262/0 (socket says 172.21.15.113:43262) 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f161590a700 1 -- 172.21.15.113:0/1496448698 learned_addr learned my addr 172.21.15.113:0/1496448698 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f1615109700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16100fd150 0x7f161010e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f161590a700 1 -- 172.21.15.113:0/1496448698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16100fd150 msgr2=0x7f161010e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.713+0000 7f161590a700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16100fd150 0x7f161010e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.714+0000 7f161590a700 1 -- 172.21.15.113:0/1496448698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c005040 con 0x7f16100fc7f0 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.714+0000 7f1615109700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16100fd150 0x7f161010e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:42:46.712 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.714+0000 7f161590a700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100fc7f0 0x7f161010dfc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f160000c790 tx=0x7f1600004330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:46.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.714+0000 7f1606ffd700 1 -- 172.21.15.113:0/1496448698 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16000106f0 con 0x7f16100fc7f0 2024-04-12T05:42:46.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.714+0000 7f1606ffd700 1 -- 172.21.15.113:0/1496448698 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1600010d00 con 0x7f16100fc7f0 2024-04-12T05:42:46.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.714+0000 7f1617b6e700 1 -- 172.21.15.113:0/1496448698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1610108280 con 0x7f16100fc7f0 2024-04-12T05:42:46.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.714+0000 7f1606ffd700 1 -- 172.21.15.113:0/1496448698 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1600012da0 con 0x7f16100fc7f0 2024-04-12T05:42:46.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.714+0000 7f1617b6e700 1 -- 172.21.15.113:0/1496448698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1610108840 con 0x7f16100fc7f0 2024-04-12T05:42:46.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.715+0000 7f1617b6e700 1 -- 172.21.15.113:0/1496448698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161005f590 con 0x7f16100fc7f0 2024-04-12T05:42:46.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.715+0000 7f1606ffd700 1 -- 172.21.15.113:0/1496448698 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1600012400 con 0x7f16100fc7f0 2024-04-12T05:42:46.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.716+0000 7f1606ffd700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f15fc06caa0 0x7f15fc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:46.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.716+0000 7f1606ffd700 1 -- 172.21.15.113:0/1496448698 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f160008b6f0 con 0x7f16100fc7f0 2024-04-12T05:42:46.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.716+0000 7f1615109700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f15fc06caa0 0x7f15fc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:46.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.716+0000 7f1615109700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f15fc06caa0 0x7f15fc06ef60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f160c0043d0 tx=0x7f160c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:46.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.720+0000 7f1606ffd700 1 -- 172.21.15.113:0/1496448698 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f16000591a0 con 0x7f16100fc7f0 2024-04-12T05:42:46.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.916+0000 7f1617b6e700 1 -- 172.21.15.113:0/1496448698 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1610109640 con 0x7f15fc06caa0 2024-04-12T05:42:46.923 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.926+0000 7f1606ffd700 1 -- 172.21.15.113:0/1496448698 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1610109640 con 0x7f15fc06caa0 2024-04-12T05:42:46.923 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:42:46.923 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (30m) 8m ago 41m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:42:46.923 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (28m) 8m ago 41m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:42:46.923 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (27m) 8m ago 41m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:42:46.923 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (30m) 8m ago 41m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:42:46.923 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (23m) 8m ago 36m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (23m) 8m ago 36m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (23m) 8m ago 36m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (8m) 8m ago 36m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (32m) 8m ago 43m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (31m) 8m ago 41m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (29m) 8m ago 43m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (28m) 8m ago 41m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (31m) 8m ago 41m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (31m) 8m ago 40m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (27m) 8m ago 40m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (27m) 8m ago 39m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (26m) 8m ago 39m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (25m) 8m ago 38m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (25m) 8m ago 38m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:42:46.924 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (24m) 8m ago 38m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:42:46.925 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (30m) 8m ago 40m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:42:46.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.928+0000 7f1617b6e700 1 -- 172.21.15.113:0/1496448698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f15fc06caa0 msgr2=0x7f15fc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.928+0000 7f1617b6e700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f15fc06caa0 0x7f15fc06ef60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f160c0043d0 tx=0x7f160c00b040).stop 2024-04-12T05:42:46.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.928+0000 7f1617b6e700 1 -- 172.21.15.113:0/1496448698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100fc7f0 msgr2=0x7f161010dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:46.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.928+0000 7f1617b6e700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100fc7f0 0x7f161010dfc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f160000c790 tx=0x7f1600004330).stop 2024-04-12T05:42:46.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.928+0000 7f1617b6e700 1 -- 172.21.15.113:0/1496448698 shutdown_connections 2024-04-12T05:42:46.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.928+0000 7f1617b6e700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f15fc06caa0 0x7f15fc06ef60 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.928+0000 7f1617b6e700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16100fc7f0 0x7f161010dfc0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.928+0000 7f1617b6e700 1 --2- 172.21.15.113:0/1496448698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16100fd150 0x7f161010e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:46.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.928+0000 7f1617b6e700 1 -- 172.21.15.113:0/1496448698 >> 172.21.15.113:0/1496448698 conn(0x7f1610070590 msgr2=0x7f1610060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:46.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.928+0000 7f1617b6e700 1 -- 172.21.15.113:0/1496448698 shutdown_connections 2024-04-12T05:42:46.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:46.928+0000 7f1617b6e700 1 -- 172.21.15.113:0/1496448698 wait complete. 2024-04-12T05:42:47.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.026+0000 7ff8d7ada700 1 -- 172.21.15.113:0/2305962529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8d00f88a0 msgr2=0x7ff8d00f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.026+0000 7ff8d7ada700 1 --2- 172.21.15.113:0/2305962529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8d00f88a0 0x7ff8d00f8c80 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7ff8c00047f0 tx=0x7ff8c0009cb0).stop 2024-04-12T05:42:47.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.026+0000 7ff8d7ada700 1 -- 172.21.15.113:0/2305962529 shutdown_connections 2024-04-12T05:42:47.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.026+0000 7ff8d7ada700 1 --2- 172.21.15.113:0/2305962529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d00f9200 0x7ff8d0101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.026+0000 7ff8d7ada700 1 --2- 172.21.15.113:0/2305962529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8d00f88a0 0x7ff8d00f8c80 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.026+0000 7ff8d7ada700 1 -- 172.21.15.113:0/2305962529 >> 172.21.15.113:0/2305962529 conn(0x7ff8d0070590 msgr2=0x7ff8d00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:47.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.026+0000 7ff8d7ada700 1 -- 172.21.15.113:0/2305962529 shutdown_connections 2024-04-12T05:42:47.023 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.026+0000 7ff8d7ada700 1 -- 172.21.15.113:0/2305962529 wait complete. 2024-04-12T05:42:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d7ada700 1 Processor -- start 2024-04-12T05:42:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d7ada700 1 -- start start 2024-04-12T05:42:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d7ada700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8d00f88a0 0x7ff8d010e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d7ada700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d00f9200 0x7ff8d010e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d7ada700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d010eb30 con 0x7ff8d00f88a0 2024-04-12T05:42:47.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d7ada700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d010ec70 con 0x7ff8d00f9200 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d5876700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8d00f88a0 0x7ff8d010e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d5876700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8d00f88a0 0x7ff8d010e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43274/0 (socket says 172.21.15.113:43274) 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d5876700 1 -- 172.21.15.113:0/918914025 learned_addr learned my addr 172.21.15.113:0/918914025 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d5075700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d00f9200 0x7ff8d010e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d5876700 1 -- 172.21.15.113:0/918914025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d00f9200 msgr2=0x7ff8d010e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d5876700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d00f9200 0x7ff8d010e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d5876700 1 -- 172.21.15.113:0/918914025 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c0005040 con 0x7ff8d00f88a0 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d5075700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d00f9200 0x7ff8d010e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.027+0000 7ff8d5876700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8d00f88a0 0x7ff8d010e040 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7ff8c0005010 tx=0x7ff8c0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.028+0000 7ff8c6f78700 1 -- 172.21.15.113:0/918914025 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c000e030 con 0x7ff8d00f88a0 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.028+0000 7ff8c6f78700 1 -- 172.21.15.113:0/918914025 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8c0007e90 con 0x7ff8d00f88a0 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.028+0000 7ff8d7ada700 1 -- 172.21.15.113:0/918914025 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d01082a0 con 0x7ff8d00f88a0 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.028+0000 7ff8c6f78700 1 -- 172.21.15.113:0/918914025 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8c0018070 con 0x7ff8d00f88a0 2024-04-12T05:42:47.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.028+0000 7ff8d7ada700 1 -- 172.21.15.113:0/918914025 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d0108780 con 0x7ff8d00f88a0 2024-04-12T05:42:47.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.029+0000 7ff8d7ada700 1 -- 172.21.15.113:0/918914025 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d005f590 con 0x7ff8d00f88a0 2024-04-12T05:42:47.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.029+0000 7ff8c6f78700 1 -- 172.21.15.113:0/918914025 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff8c001d6e0 con 0x7ff8d00f88a0 2024-04-12T05:42:47.027 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.029+0000 7ff8c6f78700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff8bc06caa0 0x7ff8bc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:47.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.030+0000 7ff8c6f78700 1 -- 172.21.15.113:0/918914025 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff8c008d0d0 con 0x7ff8d00f88a0 2024-04-12T05:42:47.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.030+0000 7ff8d5075700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff8bc06caa0 0x7ff8bc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:47.028 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.030+0000 7ff8d5075700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff8bc06caa0 0x7ff8bc06ef60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7ff8d0109980 tx=0x7ff8cc009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:47.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.034+0000 7ff8c6f78700 1 -- 172.21.15.113:0/918914025 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff8c005a8f0 con 0x7ff8d00f88a0 2024-04-12T05:42:47.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.303+0000 7ff8d7ada700 1 -- 172.21.15.113:0/918914025 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff8d0047540 con 0x7ff8d00f88a0 2024-04-12T05:42:47.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.303+0000 7ff8c6f78700 1 -- 172.21.15.113:0/918914025 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7ff8c005a710 con 0x7ff8d00f88a0 2024-04-12T05:42:47.300 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:42:47.300 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:42:47.300 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:42:47.300 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:42:47.300 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:42:47.300 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:42:47.300 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:42:47.300 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:42:47.301 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:42:47.301 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:42:47.301 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:42:47.301 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:42:47.301 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:42:47.301 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:42:47.301 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:42:47.301 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:42:47.301 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:42:47.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.305+0000 7ff8d7ada700 1 -- 172.21.15.113:0/918914025 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff8bc06caa0 msgr2=0x7ff8bc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.305+0000 7ff8d7ada700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff8bc06caa0 0x7ff8bc06ef60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7ff8d0109980 tx=0x7ff8cc009da0).stop 2024-04-12T05:42:47.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.305+0000 7ff8d7ada700 1 -- 172.21.15.113:0/918914025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8d00f88a0 msgr2=0x7ff8d010e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.305+0000 7ff8d7ada700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8d00f88a0 0x7ff8d010e040 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7ff8c0005010 tx=0x7ff8c0009cb0).stop 2024-04-12T05:42:47.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.305+0000 7ff8d7ada700 1 -- 172.21.15.113:0/918914025 shutdown_connections 2024-04-12T05:42:47.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.305+0000 7ff8d7ada700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff8bc06caa0 0x7ff8bc06ef60 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.305+0000 7ff8d7ada700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8d00f88a0 0x7ff8d010e040 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.305+0000 7ff8d7ada700 1 --2- 172.21.15.113:0/918914025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d00f9200 0x7ff8d010e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.305+0000 7ff8d7ada700 1 -- 172.21.15.113:0/918914025 >> 172.21.15.113:0/918914025 conn(0x7ff8d0070590 msgr2=0x7ff8d00fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:47.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.305+0000 7ff8d7ada700 1 -- 172.21.15.113:0/918914025 shutdown_connections 2024-04-12T05:42:47.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.305+0000 7ff8d7ada700 1 -- 172.21.15.113:0/918914025 wait complete. 2024-04-12T05:42:47.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.403+0000 7f5310d02700 1 -- 172.21.15.113:0/3051886309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c0fd150 msgr2=0x7f530c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.403+0000 7f5310d02700 1 --2- 172.21.15.113:0/3051886309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c0fd150 0x7f530c0f6930 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f52fc0047f0 tx=0x7f52fc009cb0).stop 2024-04-12T05:42:47.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.403+0000 7f5310d02700 1 -- 172.21.15.113:0/3051886309 shutdown_connections 2024-04-12T05:42:47.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.403+0000 7f5310d02700 1 --2- 172.21.15.113:0/3051886309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c0fd150 0x7f530c0f6930 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.403+0000 7f5310d02700 1 --2- 172.21.15.113:0/3051886309 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f530c0fc7f0 0x7f530c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.403+0000 7f5310d02700 1 -- 172.21.15.113:0/3051886309 >> 172.21.15.113:0/3051886309 conn(0x7f530c0f23b0 msgr2=0x7f530c0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:47.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.403+0000 7f5310d02700 1 -- 172.21.15.113:0/3051886309 shutdown_connections 2024-04-12T05:42:47.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.403+0000 7f5310d02700 1 -- 172.21.15.113:0/3051886309 wait complete. 2024-04-12T05:42:47.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f5310d02700 1 Processor -- start 2024-04-12T05:42:47.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f5310d02700 1 -- start start 2024-04-12T05:42:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f5310d02700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f530c0fc7f0 0x7f530c10e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f5310d02700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c0fd150 0x7f530c10e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f5310d02700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c10eb80 con 0x7f530c0fd150 2024-04-12T05:42:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f5310d02700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c10ecc0 con 0x7f530c0fc7f0 2024-04-12T05:42:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f5309d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c0fd150 0x7f530c10e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f5309d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c0fd150 0x7f530c10e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43288/0 (socket says 172.21.15.113:43288) 2024-04-12T05:42:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f5309d9b700 1 -- 172.21.15.113:0/3179297801 learned_addr learned my addr 172.21.15.113:0/3179297801 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f530a59c700 1 --2- 172.21.15.113:0/3179297801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f530c0fc7f0 0x7f530c10e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f5309d9b700 1 -- 172.21.15.113:0/3179297801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f530c0fc7f0 msgr2=0x7f530c10e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.404+0000 7f5309d9b700 1 --2- 172.21.15.113:0/3179297801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f530c0fc7f0 0x7f530c10e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.405+0000 7f5309d9b700 1 -- 172.21.15.113:0/3179297801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc005040 con 0x7f530c0fd150 2024-04-12T05:42:47.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.405+0000 7f5309d9b700 1 --2- 172.21.15.113:0/3179297801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c0fd150 0x7f530c10e580 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f52fc0099f0 tx=0x7f52fc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:47.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.405+0000 7f53037fe700 1 -- 172.21.15.113:0/3179297801 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52fc00e030 con 0x7f530c0fd150 2024-04-12T05:42:47.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.405+0000 7f53037fe700 1 -- 172.21.15.113:0/3179297801 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52fc007e90 con 0x7f530c0fd150 2024-04-12T05:42:47.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.405+0000 7f5310d02700 1 -- 172.21.15.113:0/3179297801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530c1082f0 con 0x7f530c0fd150 2024-04-12T05:42:47.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.405+0000 7f53037fe700 1 -- 172.21.15.113:0/3179297801 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52fc018070 con 0x7f530c0fd150 2024-04-12T05:42:47.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.405+0000 7f5310d02700 1 -- 172.21.15.113:0/3179297801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530c1087d0 con 0x7f530c0fd150 2024-04-12T05:42:47.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.406+0000 7f5310d02700 1 -- 172.21.15.113:0/3179297801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530c060370 con 0x7f530c0fd150 2024-04-12T05:42:47.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.406+0000 7f53037fe700 1 -- 172.21.15.113:0/3179297801 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f52fc009590 con 0x7f530c0fd150 2024-04-12T05:42:47.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.407+0000 7f53037fe700 1 --2- 172.21.15.113:0/3179297801 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52f806ca30 0x7f52f806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:47.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.407+0000 7f53037fe700 1 -- 172.21.15.113:0/3179297801 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f52fc08d0f0 con 0x7f530c0fd150 2024-04-12T05:42:47.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.407+0000 7f530a59c700 1 --2- 172.21.15.113:0/3179297801 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52f806ca30 0x7f52f806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:47.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.407+0000 7f530a59c700 1 --2- 172.21.15.113:0/3179297801 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52f806ca30 0x7f52f806eef0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f52f4004be0 tx=0x7f52f400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:47.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.411+0000 7f53037fe700 1 -- 172.21.15.113:0/3179297801 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f52fc05aba0 con 0x7f530c0fd150 2024-04-12T05:42:47.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.639+0000 7f5310d02700 1 -- 172.21.15.113:0/3179297801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f530c047540 con 0x7f530c0fd150 2024-04-12T05:42:47.636 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.640+0000 7f53037fe700 1 -- 172.21.15.113:0/3179297801 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f52fc05a730 con 0x7f530c0fd150 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:42:47.637 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:42:47.638 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:47.639 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:42:47.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.642+0000 7f5310d02700 1 -- 172.21.15.113:0/3179297801 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52f806ca30 msgr2=0x7f52f806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.642+0000 7f5310d02700 1 --2- 172.21.15.113:0/3179297801 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52f806ca30 0x7f52f806eef0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f52f4004be0 tx=0x7f52f400b040).stop 2024-04-12T05:42:47.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.642+0000 7f5310d02700 1 -- 172.21.15.113:0/3179297801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c0fd150 msgr2=0x7f530c10e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.642+0000 7f5310d02700 1 --2- 172.21.15.113:0/3179297801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c0fd150 0x7f530c10e580 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f52fc0099f0 tx=0x7f52fc009cb0).stop 2024-04-12T05:42:47.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.642+0000 7f5310d02700 1 -- 172.21.15.113:0/3179297801 shutdown_connections 2024-04-12T05:42:47.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.642+0000 7f5310d02700 1 --2- 172.21.15.113:0/3179297801 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52f806ca30 0x7f52f806eef0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.642+0000 7f5310d02700 1 --2- 172.21.15.113:0/3179297801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f530c0fc7f0 0x7f530c10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.642+0000 7f5310d02700 1 --2- 172.21.15.113:0/3179297801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c0fd150 0x7f530c10e580 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.642+0000 7f5310d02700 1 -- 172.21.15.113:0/3179297801 >> 172.21.15.113:0/3179297801 conn(0x7f530c0f23b0 msgr2=0x7f530c061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:47.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.642+0000 7f5310d02700 1 -- 172.21.15.113:0/3179297801 shutdown_connections 2024-04-12T05:42:47.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.642+0000 7f5310d02700 1 -- 172.21.15.113:0/3179297801 wait complete. 2024-04-12T05:42:47.641 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:42:47.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:47 smithi113 ceph-mon[103371]: pgmap v919: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:47.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:47 smithi113 ceph-mon[103371]: from='client.35654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:47.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:47 smithi192 ceph-mon[85304]: pgmap v919: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:47.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:47 smithi192 ceph-mon[85304]: from='client.35654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:47.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.741+0000 7f045ed2d700 1 -- 172.21.15.113:0/4210769145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04580fd150 msgr2=0x7f04580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.741+0000 7f045ed2d700 1 --2- 172.21.15.113:0/4210769145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04580fd150 0x7f04580f6930 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f04400047f0 tx=0x7f0440009cb0).stop 2024-04-12T05:42:47.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.741+0000 7f045ed2d700 1 -- 172.21.15.113:0/4210769145 shutdown_connections 2024-04-12T05:42:47.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.741+0000 7f045ed2d700 1 --2- 172.21.15.113:0/4210769145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04580fd150 0x7f04580f6930 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.741+0000 7f045ed2d700 1 --2- 172.21.15.113:0/4210769145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04580fc7f0 0x7f04580fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.741+0000 7f045ed2d700 1 -- 172.21.15.113:0/4210769145 >> 172.21.15.113:0/4210769145 conn(0x7f0458070590 msgr2=0x7f04580f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:47.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.741+0000 7f045ed2d700 1 -- 172.21.15.113:0/4210769145 shutdown_connections 2024-04-12T05:42:47.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.741+0000 7f045ed2d700 1 -- 172.21.15.113:0/4210769145 wait complete. 2024-04-12T05:42:47.740 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.742+0000 7f045ed2d700 1 Processor -- start 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.742+0000 7f045ed2d700 1 -- start start 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.742+0000 7f045ed2d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04580fc7f0 0x7f045810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.742+0000 7f045ed2d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04580fd150 0x7f045810d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.742+0000 7f045ed2d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045810d9c0 con 0x7f04580fc7f0 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.742+0000 7f045ed2d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045810db00 con 0x7f04580fd150 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.742+0000 7f045cac9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04580fc7f0 0x7f045810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.742+0000 7f045cac9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04580fc7f0 0x7f045810ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43292/0 (socket says 172.21.15.113:43292) 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.742+0000 7f045cac9700 1 -- 172.21.15.113:0/922055874 learned_addr learned my addr 172.21.15.113:0/922055874 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.742+0000 7f0457fff700 1 --2- 172.21.15.113:0/922055874 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04580fd150 0x7f045810d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.743+0000 7f045cac9700 1 -- 172.21.15.113:0/922055874 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04580fd150 msgr2=0x7f045810d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.743+0000 7f045cac9700 1 --2- 172.21.15.113:0/922055874 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04580fd150 0x7f045810d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.743+0000 7f045cac9700 1 -- 172.21.15.113:0/922055874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0440005040 con 0x7f04580fc7f0 2024-04-12T05:42:47.741 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.743+0000 7f045cac9700 1 --2- 172.21.15.113:0/922055874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04580fc7f0 0x7f045810ced0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f044800c790 tx=0x7f0448004330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:47.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.743+0000 7f0455ffb700 1 -- 172.21.15.113:0/922055874 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f044800e680 con 0x7f04580fc7f0 2024-04-12T05:42:47.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.743+0000 7f0455ffb700 1 -- 172.21.15.113:0/922055874 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f044800ec90 con 0x7f04580fc7f0 2024-04-12T05:42:47.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.743+0000 7f045ed2d700 1 -- 172.21.15.113:0/922055874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0458107190 con 0x7f04580fc7f0 2024-04-12T05:42:47.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.743+0000 7f0455ffb700 1 -- 172.21.15.113:0/922055874 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0448012d60 con 0x7f04580fc7f0 2024-04-12T05:42:47.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.743+0000 7f045ed2d700 1 -- 172.21.15.113:0/922055874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0458107750 con 0x7f04580fc7f0 2024-04-12T05:42:47.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.744+0000 7f045ed2d700 1 -- 172.21.15.113:0/922055874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045805f590 con 0x7f04580fc7f0 2024-04-12T05:42:47.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.744+0000 7f0455ffb700 1 -- 172.21.15.113:0/922055874 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0448012400 con 0x7f04580fc7f0 2024-04-12T05:42:47.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.745+0000 7f0455ffb700 1 --2- 172.21.15.113:0/922055874 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f044406caa0 0x7f044406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:47.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.745+0000 7f0455ffb700 1 -- 172.21.15.113:0/922055874 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f044808b640 con 0x7f04580fc7f0 2024-04-12T05:42:47.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.745+0000 7f0457fff700 1 --2- 172.21.15.113:0/922055874 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f044406caa0 0x7f044406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:47.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.745+0000 7f0457fff700 1 --2- 172.21.15.113:0/922055874 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f044406caa0 0x7f044406ef60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f0458108810 tx=0x7f044000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:47.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.749+0000 7f0455ffb700 1 -- 172.21.15.113:0/922055874 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f04480590f0 con 0x7f04580fc7f0 2024-04-12T05:42:47.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.955+0000 7f045ed2d700 1 -- 172.21.15.113:0/922055874 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f04581079c0 con 0x7f044406caa0 2024-04-12T05:42:47.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.956+0000 7f0455ffb700 1 -- 172.21.15.113:0/922055874 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f04581079c0 con 0x7f044406caa0 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:42:47.953 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:42:47.954 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:42:47.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.958+0000 7f045ed2d700 1 -- 172.21.15.113:0/922055874 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f044406caa0 msgr2=0x7f044406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.958+0000 7f045ed2d700 1 --2- 172.21.15.113:0/922055874 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f044406caa0 0x7f044406ef60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f0458108810 tx=0x7f044000b040).stop 2024-04-12T05:42:47.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.958+0000 7f045ed2d700 1 -- 172.21.15.113:0/922055874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04580fc7f0 msgr2=0x7f045810ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:47.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.958+0000 7f045ed2d700 1 --2- 172.21.15.113:0/922055874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04580fc7f0 0x7f045810ced0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f044800c790 tx=0x7f0448004330).stop 2024-04-12T05:42:47.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.958+0000 7f045ed2d700 1 -- 172.21.15.113:0/922055874 shutdown_connections 2024-04-12T05:42:47.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.958+0000 7f045ed2d700 1 --2- 172.21.15.113:0/922055874 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f044406caa0 0x7f044406ef60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.958+0000 7f045ed2d700 1 --2- 172.21.15.113:0/922055874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04580fc7f0 0x7f045810ced0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.958+0000 7f045ed2d700 1 --2- 172.21.15.113:0/922055874 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04580fd150 0x7f045810d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:47.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.958+0000 7f045ed2d700 1 -- 172.21.15.113:0/922055874 >> 172.21.15.113:0/922055874 conn(0x7f0458070590 msgr2=0x7f04580fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:47.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.958+0000 7f045ed2d700 1 -- 172.21.15.113:0/922055874 shutdown_connections 2024-04-12T05:42:47.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:47.958+0000 7f045ed2d700 1 -- 172.21.15.113:0/922055874 wait complete. 2024-04-12T05:42:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.055+0000 7fe373ae2700 1 -- 172.21.15.113:0/3112553870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe36c0fd150 msgr2=0x7fe36c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.055+0000 7fe373ae2700 1 --2- 172.21.15.113:0/3112553870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe36c0fd150 0x7fe36c0f6930 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7fe35c0047f0 tx=0x7fe35c009cb0).stop 2024-04-12T05:42:48.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.055+0000 7fe373ae2700 1 -- 172.21.15.113:0/3112553870 shutdown_connections 2024-04-12T05:42:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.055+0000 7fe373ae2700 1 --2- 172.21.15.113:0/3112553870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe36c0fd150 0x7fe36c0f6930 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.055+0000 7fe373ae2700 1 --2- 172.21.15.113:0/3112553870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe36c0fc7f0 0x7fe36c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.055+0000 7fe373ae2700 1 -- 172.21.15.113:0/3112553870 >> 172.21.15.113:0/3112553870 conn(0x7fe36c070590 msgr2=0x7fe36c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.056+0000 7fe373ae2700 1 -- 172.21.15.113:0/3112553870 shutdown_connections 2024-04-12T05:42:48.053 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.056+0000 7fe373ae2700 1 -- 172.21.15.113:0/3112553870 wait complete. 2024-04-12T05:42:48.054 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.056+0000 7fe373ae2700 1 Processor -- start 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.056+0000 7fe373ae2700 1 -- start start 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.056+0000 7fe373ae2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe36c0fc7f0 0x7fe36c10e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.056+0000 7fe373ae2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe36c0fd150 0x7fe36c10e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.056+0000 7fe373ae2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c10eb90 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.056+0000 7fe373ae2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c10ecd0 con 0x7fe36c0fd150 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.056+0000 7fe37187e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe36c0fc7f0 0x7fe36c10e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe37187e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe36c0fc7f0 0x7fe36c10e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43314/0 (socket says 172.21.15.113:43314) 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe37187e700 1 -- 172.21.15.113:0/2186469888 learned_addr learned my addr 172.21.15.113:0/2186469888 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe37107d700 1 --2- 172.21.15.113:0/2186469888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe36c0fd150 0x7fe36c10e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe37187e700 1 -- 172.21.15.113:0/2186469888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe36c0fd150 msgr2=0x7fe36c10e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe37187e700 1 --2- 172.21.15.113:0/2186469888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe36c0fd150 0x7fe36c10e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe37187e700 1 -- 172.21.15.113:0/2186469888 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c005040 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe37187e700 1 --2- 172.21.15.113:0/2186469888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe36c0fc7f0 0x7fe36c10e0a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7fe36800e8f0 tx=0x7fe36800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:48.055 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe362f78700 1 -- 172.21.15.113:0/2186469888 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe368007b40 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe373ae2700 1 -- 172.21.15.113:0/2186469888 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c1083a0 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe362f78700 1 -- 172.21.15.113:0/2186469888 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe36800f040 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe362f78700 1 -- 172.21.15.113:0/2186469888 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe368014880 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.057+0000 7fe373ae2700 1 -- 172.21.15.113:0/2186469888 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c108960 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.058+0000 7fe373ae2700 1 -- 172.21.15.113:0/2186469888 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c05f590 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.058+0000 7fe362f78700 1 -- 172.21.15.113:0/2186469888 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe368014b00 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.059+0000 7fe362f78700 1 --2- 172.21.15.113:0/2186469888 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe35806caa0 0x7fe35806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:42:48.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.059+0000 7fe362f78700 1 -- 172.21.15.113:0/2186469888 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe36808ba50 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.059+0000 7fe37107d700 1 --2- 172.21.15.113:0/2186469888 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe35806caa0 0x7fe35806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:42:48.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.059+0000 7fe37107d700 1 --2- 172.21.15.113:0/2186469888 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe35806caa0 0x7fe35806ef60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fe36c109a20 tx=0x7fe35c009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:42:48.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.063+0000 7fe362f78700 1 -- 172.21.15.113:0/2186469888 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe368055a70 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.328+0000 7fe373ae2700 1 -- 172.21.15.113:0/2186469888 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe36c047540 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.328+0000 7fe362f78700 1 -- 172.21.15.113:0/2186469888 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fe368019e10 con 0x7fe36c0fc7f0 2024-04-12T05:42:48.325 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:42:48.325 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:42:48.325 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:42:48.326 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:42:48.327 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:42:48.327 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:42:48.327 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:42:48.327 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:42:48.327 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:42:48.327 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:42:48.327 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:42:48.327 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:42:48.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.330+0000 7fe373ae2700 1 -- 172.21.15.113:0/2186469888 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe35806caa0 msgr2=0x7fe35806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:48.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.330+0000 7fe373ae2700 1 --2- 172.21.15.113:0/2186469888 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe35806caa0 0x7fe35806ef60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fe36c109a20 tx=0x7fe35c009cb0).stop 2024-04-12T05:42:48.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.330+0000 7fe373ae2700 1 -- 172.21.15.113:0/2186469888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe36c0fc7f0 msgr2=0x7fe36c10e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:42:48.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.330+0000 7fe373ae2700 1 --2- 172.21.15.113:0/2186469888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe36c0fc7f0 0x7fe36c10e0a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7fe36800e8f0 tx=0x7fe36800b040).stop 2024-04-12T05:42:48.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.330+0000 7fe373ae2700 1 -- 172.21.15.113:0/2186469888 shutdown_connections 2024-04-12T05:42:48.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.330+0000 7fe373ae2700 1 --2- 172.21.15.113:0/2186469888 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe35806caa0 0x7fe35806ef60 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:48.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.330+0000 7fe373ae2700 1 --2- 172.21.15.113:0/2186469888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe36c0fc7f0 0x7fe36c10e0a0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:48.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.330+0000 7fe373ae2700 1 --2- 172.21.15.113:0/2186469888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe36c0fd150 0x7fe36c10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:42:48.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.331+0000 7fe373ae2700 1 -- 172.21.15.113:0/2186469888 >> 172.21.15.113:0/2186469888 conn(0x7fe36c070590 msgr2=0x7fe36c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:42:48.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.331+0000 7fe373ae2700 1 -- 172.21.15.113:0/2186469888 shutdown_connections 2024-04-12T05:42:48.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:42:48.331+0000 7fe373ae2700 1 -- 172.21.15.113:0/2186469888 wait complete. 2024-04-12T05:42:48.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:48 smithi113 ceph-mon[103371]: from='client.35658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:48.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:48 smithi113 ceph-mon[103371]: from='client.35662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:48.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:48 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/918914025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:48.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:48 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3179297801' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:42:48.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:48 smithi192 ceph-mon[85304]: from='client.35658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:48.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:48 smithi192 ceph-mon[85304]: from='client.35662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:48.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:48 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/918914025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:42:48.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:48 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3179297801' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:42:49.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:49 smithi113 ceph-mon[103371]: pgmap v920: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:49.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:49 smithi113 ceph-mon[103371]: from='client.35674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:49.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:49 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2186469888' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:42:49.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:49 smithi192 ceph-mon[85304]: pgmap v920: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:49.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:49 smithi192 ceph-mon[85304]: from='client.35674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:42:49.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:49 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2186469888' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:42:51.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:51 smithi113 ceph-mon[103371]: pgmap v921: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:51.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:51 smithi192 ceph-mon[85304]: pgmap v921: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:52.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:52.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:42:53.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:53 smithi113 ceph-mon[103371]: pgmap v922: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:53.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:53 smithi192 ceph-mon[85304]: pgmap v922: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:55.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:55 smithi113 ceph-mon[103371]: pgmap v923: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:55.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:55 smithi192 ceph-mon[85304]: pgmap v923: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:57.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:57 smithi113 ceph-mon[103371]: pgmap v924: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:57.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:57 smithi192 ceph-mon[85304]: pgmap v924: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:59.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:42:59 smithi113 ceph-mon[103371]: pgmap v925: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:42:59.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:42:59 smithi192 ceph-mon[85304]: pgmap v925: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:01.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:01 smithi113 ceph-mon[103371]: pgmap v926: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:01.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:01 smithi192 ceph-mon[85304]: pgmap v926: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:03.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:03 smithi113 ceph-mon[103371]: pgmap v927: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:03.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:03 smithi192 ceph-mon[85304]: pgmap v927: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:05.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:05 smithi113 ceph-mon[103371]: pgmap v928: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:05.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:05 smithi192 ceph-mon[85304]: pgmap v928: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:07.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:07 smithi113 ceph-mon[103371]: pgmap v929: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:07.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:07.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:07 smithi192 ceph-mon[85304]: pgmap v929: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:07.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:09.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:09 smithi113 ceph-mon[103371]: pgmap v930: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:09.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:09 smithi192 ceph-mon[85304]: pgmap v930: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:11.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:11 smithi113 ceph-mon[103371]: pgmap v931: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:11.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:11 smithi192 ceph-mon[85304]: pgmap v931: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:13.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:13 smithi113 ceph-mon[103371]: pgmap v932: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:13.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:13 smithi192 ceph-mon[85304]: pgmap v932: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:15.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:15 smithi113 ceph-mon[103371]: pgmap v933: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:15.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:15 smithi192 ceph-mon[85304]: pgmap v933: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:17.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:17 smithi113 ceph-mon[103371]: pgmap v934: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:17.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:17 smithi192 ceph-mon[85304]: pgmap v934: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:18.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.440+0000 7ff553e18700 1 -- 172.21.15.113:0/1457669490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff54c0fe270 msgr2=0x7ff54c0f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:18.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.440+0000 7ff553e18700 1 --2- 172.21.15.113:0/1457669490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff54c0fe270 0x7ff54c0f7a50 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7ff53c0047f0 tx=0x7ff53c009cb0).stop 2024-04-12T05:43:18.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.440+0000 7ff553e18700 1 -- 172.21.15.113:0/1457669490 shutdown_connections 2024-04-12T05:43:18.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.440+0000 7ff553e18700 1 --2- 172.21.15.113:0/1457669490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff54c0fe270 0x7ff54c0f7a50 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.440+0000 7ff553e18700 1 --2- 172.21.15.113:0/1457669490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54c0fd910 0x7ff54c0fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.440+0000 7ff553e18700 1 -- 172.21.15.113:0/1457669490 >> 172.21.15.113:0/1457669490 conn(0x7ff54c0f3530 msgr2=0x7ff54c0f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:18.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.441+0000 7ff553e18700 1 -- 172.21.15.113:0/1457669490 shutdown_connections 2024-04-12T05:43:18.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.441+0000 7ff553e18700 1 -- 172.21.15.113:0/1457669490 wait complete. 2024-04-12T05:43:18.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.441+0000 7ff553e18700 1 Processor -- start 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.441+0000 7ff553e18700 1 -- start start 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.441+0000 7ff553e18700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff54c0fd910 0x7ff54c10dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.441+0000 7ff553e18700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54c0fe270 0x7ff54c10e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.441+0000 7ff553e18700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c10eab0 con 0x7ff54c0fd910 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.441+0000 7ff553e18700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c10ebf0 con 0x7ff54c0fe270 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff551bb4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff54c0fd910 0x7ff54c10dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff551bb4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff54c0fd910 0x7ff54c10dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39736/0 (socket says 172.21.15.113:39736) 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff551bb4700 1 -- 172.21.15.113:0/2339182094 learned_addr learned my addr 172.21.15.113:0/2339182094 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff5513b3700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54c0fe270 0x7ff54c10e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff551bb4700 1 -- 172.21.15.113:0/2339182094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54c0fe270 msgr2=0x7ff54c10e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff551bb4700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54c0fe270 0x7ff54c10e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff551bb4700 1 -- 172.21.15.113:0/2339182094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c005040 con 0x7ff54c0fd910 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff5513b3700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54c0fe270 0x7ff54c10e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:43:18.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff551bb4700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff54c0fd910 0x7ff54c10dfc0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7ff5480041d0 tx=0x7ff548009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:18.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff542ffd700 1 -- 172.21.15.113:0/2339182094 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff548007b40 con 0x7ff54c0fd910 2024-04-12T05:43:18.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff542ffd700 1 -- 172.21.15.113:0/2339182094 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff54800f040 con 0x7ff54c0fd910 2024-04-12T05:43:18.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff553e18700 1 -- 172.21.15.113:0/2339182094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54c108280 con 0x7ff54c0fd910 2024-04-12T05:43:18.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff542ffd700 1 -- 172.21.15.113:0/2339182094 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff54800d800 con 0x7ff54c0fd910 2024-04-12T05:43:18.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.442+0000 7ff553e18700 1 -- 172.21.15.113:0/2339182094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54c108840 con 0x7ff54c0fd910 2024-04-12T05:43:18.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.443+0000 7ff553e18700 1 -- 172.21.15.113:0/2339182094 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54c05f570 con 0x7ff54c0fd910 2024-04-12T05:43:18.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.444+0000 7ff542ffd700 1 -- 172.21.15.113:0/2339182094 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff548007ca0 con 0x7ff54c0fd910 2024-04-12T05:43:18.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.444+0000 7ff542ffd700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff53806caa0 0x7ff53806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:18.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.444+0000 7ff542ffd700 1 -- 172.21.15.113:0/2339182094 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff54808b950 con 0x7ff54c0fd910 2024-04-12T05:43:18.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.444+0000 7ff5513b3700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff53806caa0 0x7ff53806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:18.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.444+0000 7ff5513b3700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff53806caa0 0x7ff53806ef60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7ff53c0043d0 tx=0x7ff53c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:18.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.448+0000 7ff542ffd700 1 -- 172.21.15.113:0/2339182094 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff548059400 con 0x7ff54c0fd910 2024-04-12T05:43:18.651 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.654+0000 7ff553e18700 1 -- 172.21.15.113:0/2339182094 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff54c108ac0 con 0x7ff53806caa0 2024-04-12T05:43:18.652 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.655+0000 7ff542ffd700 1 -- 172.21.15.113:0/2339182094 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff54c108ac0 con 0x7ff53806caa0 2024-04-12T05:43:18.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.657+0000 7ff553e18700 1 -- 172.21.15.113:0/2339182094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff53806caa0 msgr2=0x7ff53806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:18.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.657+0000 7ff553e18700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff53806caa0 0x7ff53806ef60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7ff53c0043d0 tx=0x7ff53c00b040).stop 2024-04-12T05:43:18.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.657+0000 7ff553e18700 1 -- 172.21.15.113:0/2339182094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff54c0fd910 msgr2=0x7ff54c10dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:18.654 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.657+0000 7ff553e18700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff54c0fd910 0x7ff54c10dfc0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7ff5480041d0 tx=0x7ff548009f60).stop 2024-04-12T05:43:18.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.657+0000 7ff553e18700 1 -- 172.21.15.113:0/2339182094 shutdown_connections 2024-04-12T05:43:18.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.657+0000 7ff553e18700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff53806caa0 0x7ff53806ef60 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.657+0000 7ff553e18700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff54c0fd910 0x7ff54c10dfc0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.657+0000 7ff553e18700 1 --2- 172.21.15.113:0/2339182094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff54c0fe270 0x7ff54c10e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.657+0000 7ff553e18700 1 -- 172.21.15.113:0/2339182094 >> 172.21.15.113:0/2339182094 conn(0x7ff54c0f3530 msgr2=0x7ff54c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:18.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.657+0000 7ff553e18700 1 -- 172.21.15.113:0/2339182094 shutdown_connections 2024-04-12T05:43:18.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.658+0000 7ff553e18700 1 -- 172.21.15.113:0/2339182094 wait complete. 2024-04-12T05:43:18.664 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:43:18.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.756+0000 7f632d902700 1 -- 172.21.15.113:0/3757480162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63280fc7f0 msgr2=0x7f63280fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:18.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.756+0000 7f632d902700 1 --2- 172.21.15.113:0/3757480162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63280fc7f0 0x7f63280fcbd0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f63180047f0 tx=0x7f6318009cb0).stop 2024-04-12T05:43:18.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.756+0000 7f632d902700 1 -- 172.21.15.113:0/3757480162 shutdown_connections 2024-04-12T05:43:18.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.756+0000 7f632d902700 1 --2- 172.21.15.113:0/3757480162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63280fd150 0x7f63280f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.756+0000 7f632d902700 1 --2- 172.21.15.113:0/3757480162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63280fc7f0 0x7f63280fcbd0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.756+0000 7f632d902700 1 -- 172.21.15.113:0/3757480162 >> 172.21.15.113:0/3757480162 conn(0x7f63280f23b0 msgr2=0x7f63280f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:18.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.756+0000 7f632d902700 1 -- 172.21.15.113:0/3757480162 shutdown_connections 2024-04-12T05:43:18.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.756+0000 7f632d902700 1 -- 172.21.15.113:0/3757480162 wait complete. 2024-04-12T05:43:18.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.757+0000 7f632d902700 1 Processor -- start 2024-04-12T05:43:18.754 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.757+0000 7f632d902700 1 -- start start 2024-04-12T05:43:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f632d902700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63280fc7f0 0x7f6328109110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f632d902700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63280fd150 0x7f6328109600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f632d902700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6328109af0 con 0x7f63280fd150 2024-04-12T05:43:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f632d902700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6328109c30 con 0x7f63280fc7f0 2024-04-12T05:43:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f63267fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63280fd150 0x7f6328109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f63267fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63280fd150 0x7f6328109600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39752/0 (socket says 172.21.15.113:39752) 2024-04-12T05:43:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f63267fc700 1 -- 172.21.15.113:0/2539699387 learned_addr learned my addr 172.21.15.113:0/2539699387 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:18.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f6326ffd700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63280fc7f0 0x7f6328109110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f63267fc700 1 -- 172.21.15.113:0/2539699387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63280fc7f0 msgr2=0x7f6328109110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f63267fc700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63280fc7f0 0x7f6328109110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f63267fc700 1 -- 172.21.15.113:0/2539699387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6318005040 con 0x7f63280fd150 2024-04-12T05:43:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f6326ffd700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63280fc7f0 0x7f6328109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:43:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f63267fc700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63280fd150 0x7f6328109600 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f631c00c710 tx=0x7f631c007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f630ffff700 1 -- 172.21.15.113:0/2539699387 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f631c00e680 con 0x7f63280fd150 2024-04-12T05:43:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f630ffff700 1 -- 172.21.15.113:0/2539699387 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f631c00ec90 con 0x7f63280fd150 2024-04-12T05:43:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f632d902700 1 -- 172.21.15.113:0/2539699387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6328109ee0 con 0x7f63280fd150 2024-04-12T05:43:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f630ffff700 1 -- 172.21.15.113:0/2539699387 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f631c013cd0 con 0x7f63280fd150 2024-04-12T05:43:18.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.758+0000 7f632d902700 1 -- 172.21.15.113:0/2539699387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63281124d0 con 0x7f63280fd150 2024-04-12T05:43:18.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.760+0000 7f632d902700 1 -- 172.21.15.113:0/2539699387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6328060370 con 0x7f63280fd150 2024-04-12T05:43:18.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.760+0000 7f630ffff700 1 -- 172.21.15.113:0/2539699387 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f631c013410 con 0x7f63280fd150 2024-04-12T05:43:18.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.760+0000 7f630ffff700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f631006caa0 0x7f631006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.760+0000 7f630ffff700 1 -- 172.21.15.113:0/2539699387 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f631c08c190 con 0x7f63280fd150 2024-04-12T05:43:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.760+0000 7f6326ffd700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f631006caa0 0x7f631006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:18.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.761+0000 7f6326ffd700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f631006caa0 0x7f631006ef60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6318007cf0 tx=0x7f631800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:18.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.765+0000 7f630ffff700 1 -- 172.21.15.113:0/2539699387 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f631c0561b0 con 0x7f63280fd150 2024-04-12T05:43:18.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.970+0000 7f632d902700 1 -- 172.21.15.113:0/2539699387 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6328061270 con 0x7f631006caa0 2024-04-12T05:43:18.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.971+0000 7f630ffff700 1 -- 172.21.15.113:0/2539699387 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6328061270 con 0x7f631006caa0 2024-04-12T05:43:18.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.973+0000 7f632d902700 1 -- 172.21.15.113:0/2539699387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f631006caa0 msgr2=0x7f631006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:18.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.973+0000 7f632d902700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f631006caa0 0x7f631006ef60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6318007cf0 tx=0x7f631800b040).stop 2024-04-12T05:43:18.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.973+0000 7f632d902700 1 -- 172.21.15.113:0/2539699387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63280fd150 msgr2=0x7f6328109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:18.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.973+0000 7f632d902700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63280fd150 0x7f6328109600 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f631c00c710 tx=0x7f631c007ec0).stop 2024-04-12T05:43:18.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.973+0000 7f632d902700 1 -- 172.21.15.113:0/2539699387 shutdown_connections 2024-04-12T05:43:18.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.973+0000 7f632d902700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f631006caa0 0x7f631006ef60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.973+0000 7f632d902700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63280fc7f0 0x7f6328109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.973+0000 7f632d902700 1 --2- 172.21.15.113:0/2539699387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63280fd150 0x7f6328109600 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:18.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.973+0000 7f632d902700 1 -- 172.21.15.113:0/2539699387 >> 172.21.15.113:0/2539699387 conn(0x7f63280f23b0 msgr2=0x7f63280fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:18.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.974+0000 7f632d902700 1 -- 172.21.15.113:0/2539699387 shutdown_connections 2024-04-12T05:43:18.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:18.974+0000 7f632d902700 1 -- 172.21.15.113:0/2539699387 wait complete. 2024-04-12T05:43:19.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.071+0000 7f0bd2804700 1 -- 172.21.15.113:0/375944977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bcc0f88d0 msgr2=0x7f0bcc0f8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:19.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.071+0000 7f0bd2804700 1 --2- 172.21.15.113:0/375944977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bcc0f88d0 0x7f0bcc0f8cb0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f0bbc0047f0 tx=0x7f0bbc009cb0).stop 2024-04-12T05:43:19.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.071+0000 7f0bd2804700 1 -- 172.21.15.113:0/375944977 shutdown_connections 2024-04-12T05:43:19.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.071+0000 7f0bd2804700 1 --2- 172.21.15.113:0/375944977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bcc0f9230 0x7f0bcc101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.071+0000 7f0bd2804700 1 --2- 172.21.15.113:0/375944977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bcc0f88d0 0x7f0bcc0f8cb0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.071+0000 7f0bd2804700 1 -- 172.21.15.113:0/375944977 >> 172.21.15.113:0/375944977 conn(0x7f0bcc0f2400 msgr2=0x7f0bcc0f4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:19.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.071+0000 7f0bd2804700 1 -- 172.21.15.113:0/375944977 shutdown_connections 2024-04-12T05:43:19.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.071+0000 7f0bd2804700 1 -- 172.21.15.113:0/375944977 wait complete. 2024-04-12T05:43:19.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.072+0000 7f0bd2804700 1 Processor -- start 2024-04-12T05:43:19.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.072+0000 7f0bd2804700 1 -- start start 2024-04-12T05:43:19.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.072+0000 7f0bd2804700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bcc0f88d0 0x7f0bcc10e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:19.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.072+0000 7f0bd2804700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bcc0f9230 0x7f0bcc10e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:19.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bd2804700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc10eb30 con 0x7f0bcc0f88d0 2024-04-12T05:43:19.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bd2804700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc10ec70 con 0x7f0bcc0f9230 2024-04-12T05:43:19.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bcbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bcc0f88d0 0x7f0bcc10e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:19.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bcbfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bcc0f88d0 0x7f0bcc10e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39764/0 (socket says 172.21.15.113:39764) 2024-04-12T05:43:19.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bcbfff700 1 -- 172.21.15.113:0/4214390957 learned_addr learned my addr 172.21.15.113:0/4214390957 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bcb7fe700 1 --2- 172.21.15.113:0/4214390957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bcc0f9230 0x7f0bcc10e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bcbfff700 1 -- 172.21.15.113:0/4214390957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bcc0f9230 msgr2=0x7f0bcc10e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bcbfff700 1 --2- 172.21.15.113:0/4214390957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bcc0f9230 0x7f0bcc10e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bcbfff700 1 -- 172.21.15.113:0/4214390957 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc005040 con 0x7f0bcc0f88d0 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bcbfff700 1 --2- 172.21.15.113:0/4214390957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bcc0f88d0 0x7f0bcc10e040 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f0bbc005010 tx=0x7f0bbc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bc97fa700 1 -- 172.21.15.113:0/4214390957 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bbc00e030 con 0x7f0bcc0f88d0 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bc97fa700 1 -- 172.21.15.113:0/4214390957 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bbc007e90 con 0x7f0bcc0f88d0 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bd2804700 1 -- 172.21.15.113:0/4214390957 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc1082a0 con 0x7f0bcc0f88d0 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bc97fa700 1 -- 172.21.15.113:0/4214390957 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bbc018070 con 0x7f0bcc0f88d0 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.073+0000 7f0bd2804700 1 -- 172.21.15.113:0/4214390957 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc108780 con 0x7f0bcc0f88d0 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.074+0000 7f0bd2804700 1 -- 172.21.15.113:0/4214390957 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc060330 con 0x7f0bcc0f88d0 2024-04-12T05:43:19.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.075+0000 7f0bc97fa700 1 -- 172.21.15.113:0/4214390957 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0bbc01d6e0 con 0x7f0bcc0f88d0 2024-04-12T05:43:19.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.075+0000 7f0bc97fa700 1 --2- 172.21.15.113:0/4214390957 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406ca30 0x7f0bb406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:19.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.075+0000 7f0bc97fa700 1 -- 172.21.15.113:0/4214390957 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0bbc08d080 con 0x7f0bcc0f88d0 2024-04-12T05:43:19.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.075+0000 7f0bcb7fe700 1 --2- 172.21.15.113:0/4214390957 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406ca30 0x7f0bb406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:19.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.075+0000 7f0bcb7fe700 1 --2- 172.21.15.113:0/4214390957 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406ca30 0x7f0bb406eef0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f0bcc109980 tx=0x7f0bc0009de0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:19.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.080+0000 7f0bc97fa700 1 -- 172.21.15.113:0/4214390957 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0bbc05ab30 con 0x7f0bcc0f88d0 2024-04-12T05:43:19.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.279+0000 7f0bd2804700 1 -- 172.21.15.113:0/4214390957 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bcc0fee00 con 0x7f0bb406ca30 2024-04-12T05:43:19.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.289+0000 7f0bc97fa700 1 -- 172.21.15.113:0/4214390957 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0bcc0fee00 con 0x7f0bb406ca30 2024-04-12T05:43:19.285 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:43:19.285 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (31m) 8m ago 42m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:43:19.285 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (28m) 8m ago 42m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (28m) 8m ago 41m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (30m) 8m ago 42m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (24m) 8m ago 37m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (24m) 8m ago 36m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (23m) 8m ago 37m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (8m) 8m ago 36m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (32m) 8m ago 43m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (32m) 8m ago 41m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (29m) 8m ago 43m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (29m) 8m ago 41m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (31m) 8m ago 42m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (31m) 8m ago 41m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:43:19.286 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (28m) 8m ago 40m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:43:19.287 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (27m) 8m ago 40m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:43:19.287 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (26m) 8m ago 39m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:43:19.287 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (26m) 8m ago 39m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:43:19.287 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (25m) 8m ago 39m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:43:19.287 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (24m) 8m ago 38m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:43:19.287 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (31m) 8m ago 41m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:43:19.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.291+0000 7f0bd2804700 1 -- 172.21.15.113:0/4214390957 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406ca30 msgr2=0x7f0bb406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:19.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.291+0000 7f0bd2804700 1 --2- 172.21.15.113:0/4214390957 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406ca30 0x7f0bb406eef0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f0bcc109980 tx=0x7f0bc0009de0).stop 2024-04-12T05:43:19.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.291+0000 7f0bd2804700 1 -- 172.21.15.113:0/4214390957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bcc0f88d0 msgr2=0x7f0bcc10e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:19.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.291+0000 7f0bd2804700 1 --2- 172.21.15.113:0/4214390957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bcc0f88d0 0x7f0bcc10e040 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f0bbc005010 tx=0x7f0bbc009cb0).stop 2024-04-12T05:43:19.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.291+0000 7f0bd2804700 1 -- 172.21.15.113:0/4214390957 shutdown_connections 2024-04-12T05:43:19.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.291+0000 7f0bd2804700 1 --2- 172.21.15.113:0/4214390957 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406ca30 0x7f0bb406eef0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.291+0000 7f0bd2804700 1 --2- 172.21.15.113:0/4214390957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bcc0f88d0 0x7f0bcc10e040 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.291+0000 7f0bd2804700 1 --2- 172.21.15.113:0/4214390957 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bcc0f9230 0x7f0bcc10e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.291+0000 7f0bd2804700 1 -- 172.21.15.113:0/4214390957 >> 172.21.15.113:0/4214390957 conn(0x7f0bcc0f2400 msgr2=0x7f0bcc0fc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:19.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.291+0000 7f0bd2804700 1 -- 172.21.15.113:0/4214390957 shutdown_connections 2024-04-12T05:43:19.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.291+0000 7f0bd2804700 1 -- 172.21.15.113:0/4214390957 wait complete. 2024-04-12T05:43:19.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.388+0000 7faefeb32700 1 -- 172.21.15.113:0/25139678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef80fd8c0 msgr2=0x7faef80fdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:19.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.388+0000 7faefeb32700 1 --2- 172.21.15.113:0/25139678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef80fd8c0 0x7faef80fdca0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7faeec0047f0 tx=0x7faeec009cb0).stop 2024-04-12T05:43:19.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.388+0000 7faefeb32700 1 -- 172.21.15.113:0/25139678 shutdown_connections 2024-04-12T05:43:19.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.388+0000 7faefeb32700 1 --2- 172.21.15.113:0/25139678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef80fe220 0x7faef80f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.388+0000 7faefeb32700 1 --2- 172.21.15.113:0/25139678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef80fd8c0 0x7faef80fdca0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.388+0000 7faefeb32700 1 -- 172.21.15.113:0/25139678 >> 172.21.15.113:0/25139678 conn(0x7faef80f34a0 msgr2=0x7faef80f58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:19.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.388+0000 7faefeb32700 1 -- 172.21.15.113:0/25139678 shutdown_connections 2024-04-12T05:43:19.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.388+0000 7faefeb32700 1 -- 172.21.15.113:0/25139678 wait complete. 2024-04-12T05:43:19.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.389+0000 7faefeb32700 1 Processor -- start 2024-04-12T05:43:19.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.389+0000 7faefeb32700 1 -- start start 2024-04-12T05:43:19.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.389+0000 7faefeb32700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef80fd8c0 0x7faef810dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:19.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.389+0000 7faefeb32700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef80fe220 0x7faef810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:19.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.389+0000 7faefeb32700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef810eae0 con 0x7faef80fe220 2024-04-12T05:43:19.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.389+0000 7faefeb32700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef810ec20 con 0x7faef80fd8c0 2024-04-12T05:43:19.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.389+0000 7faef7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef80fe220 0x7faef810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.389+0000 7faef7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef80fe220 0x7faef810e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39776/0 (socket says 172.21.15.113:39776) 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.389+0000 7faef7fff700 1 -- 172.21.15.113:0/1871569381 learned_addr learned my addr 172.21.15.113:0/1871569381 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.389+0000 7faefc8ce700 1 --2- 172.21.15.113:0/1871569381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef80fd8c0 0x7faef810dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.390+0000 7faef7fff700 1 -- 172.21.15.113:0/1871569381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef80fd8c0 msgr2=0x7faef810dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.390+0000 7faef7fff700 1 --2- 172.21.15.113:0/1871569381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef80fd8c0 0x7faef810dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.390+0000 7faef7fff700 1 -- 172.21.15.113:0/1871569381 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec005040 con 0x7faef80fe220 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.390+0000 7faef7fff700 1 --2- 172.21.15.113:0/1871569381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef80fe220 0x7faef810e4e0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7faee4002780 tx=0x7faee4009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.390+0000 7faef5ffb700 1 -- 172.21.15.113:0/1871569381 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faee4007b40 con 0x7faef80fe220 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.390+0000 7faefeb32700 1 -- 172.21.15.113:0/1871569381 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef81082b0 con 0x7faef80fe220 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.390+0000 7faef5ffb700 1 -- 172.21.15.113:0/1871569381 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faee400f040 con 0x7faef80fe220 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.390+0000 7faef5ffb700 1 -- 172.21.15.113:0/1871569381 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faee4014780 con 0x7faef80fe220 2024-04-12T05:43:19.388 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.390+0000 7faefeb32700 1 -- 172.21.15.113:0/1871569381 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef8108870 con 0x7faef80fe220 2024-04-12T05:43:19.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.391+0000 7faefeb32700 1 -- 172.21.15.113:0/1871569381 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef8060390 con 0x7faef80fe220 2024-04-12T05:43:19.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.391+0000 7faef5ffb700 1 -- 172.21.15.113:0/1871569381 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faee4007ca0 con 0x7faef80fe220 2024-04-12T05:43:19.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.392+0000 7faef5ffb700 1 --2- 172.21.15.113:0/1871569381 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee806caa0 0x7faee806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:19.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.392+0000 7faef5ffb700 1 -- 172.21.15.113:0/1871569381 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7faee408bd50 con 0x7faef80fe220 2024-04-12T05:43:19.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.392+0000 7faefc8ce700 1 --2- 172.21.15.113:0/1871569381 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee806caa0 0x7faee806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:19.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.392+0000 7faefc8ce700 1 --2- 172.21.15.113:0/1871569381 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee806caa0 0x7faee806ef60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7faeec004f10 tx=0x7faeec00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:19.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.396+0000 7faef5ffb700 1 -- 172.21.15.113:0/1871569381 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7faee4059800 con 0x7faef80fe220 2024-04-12T05:43:19.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.664+0000 7faefeb32700 1 -- 172.21.15.113:0/1871569381 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faef8047540 con 0x7faef80fe220 2024-04-12T05:43:19.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.665+0000 7faef5ffb700 1 -- 172.21.15.113:0/1871569381 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7faee4059390 con 0x7faef80fe220 2024-04-12T05:43:19.661 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:43:19.661 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:43:19.661 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:43:19.661 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:43:19.661 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:43:19.661 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:43:19.661 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:43:19.662 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:43:19.662 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:43:19.662 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:43:19.662 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:43:19.662 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:43:19.662 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:43:19.662 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:43:19.662 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:43:19.662 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:43:19.662 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:43:19.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.667+0000 7faefeb32700 1 -- 172.21.15.113:0/1871569381 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee806caa0 msgr2=0x7faee806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:19.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.667+0000 7faefeb32700 1 --2- 172.21.15.113:0/1871569381 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee806caa0 0x7faee806ef60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7faeec004f10 tx=0x7faeec00b040).stop 2024-04-12T05:43:19.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.667+0000 7faefeb32700 1 -- 172.21.15.113:0/1871569381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef80fe220 msgr2=0x7faef810e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:19.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.667+0000 7faefeb32700 1 --2- 172.21.15.113:0/1871569381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef80fe220 0x7faef810e4e0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7faee4002780 tx=0x7faee4009ee0).stop 2024-04-12T05:43:19.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.667+0000 7faefeb32700 1 -- 172.21.15.113:0/1871569381 shutdown_connections 2024-04-12T05:43:19.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.667+0000 7faefeb32700 1 --2- 172.21.15.113:0/1871569381 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faee806caa0 0x7faee806ef60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.667+0000 7faefeb32700 1 --2- 172.21.15.113:0/1871569381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef80fd8c0 0x7faef810dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.667+0000 7faefeb32700 1 --2- 172.21.15.113:0/1871569381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faef80fe220 0x7faef810e4e0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.667+0000 7faefeb32700 1 -- 172.21.15.113:0/1871569381 >> 172.21.15.113:0/1871569381 conn(0x7faef80f34a0 msgr2=0x7faef80fbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:19.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.667+0000 7faefeb32700 1 -- 172.21.15.113:0/1871569381 shutdown_connections 2024-04-12T05:43:19.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.667+0000 7faefeb32700 1 -- 172.21.15.113:0/1871569381 wait complete. 2024-04-12T05:43:19.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:19 smithi113 ceph-mon[103371]: pgmap v935: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:19.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:19 smithi192 ceph-mon[85304]: pgmap v935: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:19.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.764+0000 7f2c147ba700 1 -- 172.21.15.113:0/1898084336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c0c0fd150 msgr2=0x7f2c0c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:19.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.764+0000 7f2c147ba700 1 --2- 172.21.15.113:0/1898084336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c0c0fd150 0x7f2c0c0f6930 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7f2bfc0047f0 tx=0x7f2bfc009cb0).stop 2024-04-12T05:43:19.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.765+0000 7f2c147ba700 1 -- 172.21.15.113:0/1898084336 shutdown_connections 2024-04-12T05:43:19.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.765+0000 7f2c147ba700 1 --2- 172.21.15.113:0/1898084336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c0c0fd150 0x7f2c0c0f6930 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.765+0000 7f2c147ba700 1 --2- 172.21.15.113:0/1898084336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c0c0fc7f0 0x7f2c0c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.765+0000 7f2c147ba700 1 -- 172.21.15.113:0/1898084336 >> 172.21.15.113:0/1898084336 conn(0x7f2c0c070590 msgr2=0x7f2c0c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:19.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.765+0000 7f2c147ba700 1 -- 172.21.15.113:0/1898084336 shutdown_connections 2024-04-12T05:43:19.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.765+0000 7f2c147ba700 1 -- 172.21.15.113:0/1898084336 wait complete. 2024-04-12T05:43:19.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.765+0000 7f2c147ba700 1 Processor -- start 2024-04-12T05:43:19.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.765+0000 7f2c147ba700 1 -- start start 2024-04-12T05:43:19.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c147ba700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c0c0fc7f0 0x7f2c0c10cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:19.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c147ba700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c0c0fd150 0x7f2c0c10d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:19.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c147ba700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c10d9e0 con 0x7f2c0c0fd150 2024-04-12T05:43:19.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c147ba700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c10db20 con 0x7f2c0c0fc7f0 2024-04-12T05:43:19.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c11d55700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c0c0fd150 0x7f2c0c10d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:19.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c11d55700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c0c0fd150 0x7f2c0c10d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39798/0 (socket says 172.21.15.113:39798) 2024-04-12T05:43:19.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c11d55700 1 -- 172.21.15.113:0/3748250304 learned_addr learned my addr 172.21.15.113:0/3748250304 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:19.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c12556700 1 --2- 172.21.15.113:0/3748250304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c0c0fc7f0 0x7f2c0c10cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c11d55700 1 -- 172.21.15.113:0/3748250304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c0c0fc7f0 msgr2=0x7f2c0c10cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c11d55700 1 --2- 172.21.15.113:0/3748250304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c0c0fc7f0 0x7f2c0c10cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c11d55700 1 -- 172.21.15.113:0/3748250304 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc005040 con 0x7f2c0c0fd150 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c11d55700 1 --2- 172.21.15.113:0/3748250304 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c0c0fd150 0x7f2c0c10d3e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f2bfc009ee0 tx=0x7f2bfc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c037fe700 1 -- 172.21.15.113:0/3748250304 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bfc00e030 con 0x7f2c0c0fd150 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c037fe700 1 -- 172.21.15.113:0/3748250304 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bfc007e90 con 0x7f2c0c0fd150 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c037fe700 1 -- 172.21.15.113:0/3748250304 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bfc01a070 con 0x7f2c0c0fd150 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.766+0000 7f2c147ba700 1 -- 172.21.15.113:0/3748250304 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0c107150 con 0x7f2c0c0fd150 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.767+0000 7f2c147ba700 1 -- 172.21.15.113:0/3748250304 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c0c1076b0 con 0x7f2c0c0fd150 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.767+0000 7f2c147ba700 1 -- 172.21.15.113:0/3748250304 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0c05f590 con 0x7f2c0c0fd150 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.768+0000 7f2c037fe700 1 -- 172.21.15.113:0/3748250304 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2bfc012cf0 con 0x7f2c0c0fd150 2024-04-12T05:43:19.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.768+0000 7f2c037fe700 1 --2- 172.21.15.113:0/3748250304 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bf806caa0 0x7f2bf806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:19.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.768+0000 7f2c037fe700 1 -- 172.21.15.113:0/3748250304 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2bfc08f600 con 0x7f2c0c0fd150 2024-04-12T05:43:19.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.768+0000 7f2c12556700 1 --2- 172.21.15.113:0/3748250304 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bf806caa0 0x7f2bf806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:19.768 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.769+0000 7f2c12556700 1 --2- 172.21.15.113:0/3748250304 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bf806caa0 0x7f2bf806ef60 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f2c08004de0 tx=0x7f2c0800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:19.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:19.773+0000 7f2c037fe700 1 -- 172.21.15.113:0/3748250304 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2bfc05d0b0 con 0x7f2c0c0fd150 2024-04-12T05:43:19.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.000+0000 7f2c147ba700 1 -- 172.21.15.113:0/3748250304 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2c0c047540 con 0x7f2c0c0fd150 2024-04-12T05:43:19.997 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.000+0000 7f2c037fe700 1 -- 172.21.15.113:0/3748250304 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f2bfc05cc40 con 0x7f2c0c0fd150 2024-04-12T05:43:19.997 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:43:19.997 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:43:19.997 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:43:19.997 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:43:19.997 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:43:19.998 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:43:19.999 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:43:20.000 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:43:20.000 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:43:20.000 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:43:20.000 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:20.000 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:20.000 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:20.000 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:20.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.002+0000 7f2c147ba700 1 -- 172.21.15.113:0/3748250304 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bf806caa0 msgr2=0x7f2bf806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:20.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.002+0000 7f2c147ba700 1 --2- 172.21.15.113:0/3748250304 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bf806caa0 0x7f2bf806ef60 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f2c08004de0 tx=0x7f2c0800b040).stop 2024-04-12T05:43:20.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.002+0000 7f2c147ba700 1 -- 172.21.15.113:0/3748250304 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c0c0fd150 msgr2=0x7f2c0c10d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:20.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.002+0000 7f2c147ba700 1 --2- 172.21.15.113:0/3748250304 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c0c0fd150 0x7f2c0c10d3e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f2bfc009ee0 tx=0x7f2bfc009cb0).stop 2024-04-12T05:43:20.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.003+0000 7f2c147ba700 1 -- 172.21.15.113:0/3748250304 shutdown_connections 2024-04-12T05:43:20.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.003+0000 7f2c147ba700 1 --2- 172.21.15.113:0/3748250304 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2bf806caa0 0x7f2bf806ef60 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.003+0000 7f2c147ba700 1 --2- 172.21.15.113:0/3748250304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c0c0fc7f0 0x7f2c0c10cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.003+0000 7f2c147ba700 1 --2- 172.21.15.113:0/3748250304 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c0c0fd150 0x7f2c0c10d3e0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.003+0000 7f2c147ba700 1 -- 172.21.15.113:0/3748250304 >> 172.21.15.113:0/3748250304 conn(0x7f2c0c070590 msgr2=0x7f2c0c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:20.001 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.003+0000 7f2c147ba700 1 -- 172.21.15.113:0/3748250304 shutdown_connections 2024-04-12T05:43:20.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.003+0000 7f2c147ba700 1 -- 172.21.15.113:0/3748250304 wait complete. 2024-04-12T05:43:20.002 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:43:20.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.102+0000 7f3c893ef700 1 -- 172.21.15.113:0/1471757905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c840fc7f0 msgr2=0x7f3c840fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:20.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.102+0000 7f3c893ef700 1 --2- 172.21.15.113:0/1471757905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c840fc7f0 0x7f3c840fcbd0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7f3c740047f0 tx=0x7f3c74009cb0).stop 2024-04-12T05:43:20.099 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.102+0000 7f3c893ef700 1 -- 172.21.15.113:0/1471757905 shutdown_connections 2024-04-12T05:43:20.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.102+0000 7f3c893ef700 1 --2- 172.21.15.113:0/1471757905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c840fd150 0x7f3c840f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.102+0000 7f3c893ef700 1 --2- 172.21.15.113:0/1471757905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c840fc7f0 0x7f3c840fcbd0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.102+0000 7f3c893ef700 1 -- 172.21.15.113:0/1471757905 >> 172.21.15.113:0/1471757905 conn(0x7f3c84070590 msgr2=0x7f3c840f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:20.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.102+0000 7f3c893ef700 1 -- 172.21.15.113:0/1471757905 shutdown_connections 2024-04-12T05:43:20.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.102+0000 7f3c893ef700 1 -- 172.21.15.113:0/1471757905 wait complete. 2024-04-12T05:43:20.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c893ef700 1 Processor -- start 2024-04-12T05:43:20.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c893ef700 1 -- start start 2024-04-12T05:43:20.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c893ef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c840fc7f0 0x7f3c8410dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:20.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c893ef700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c840fd150 0x7f3c8410e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:20.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c893ef700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8410eab0 con 0x7f3c840fd150 2024-04-12T05:43:20.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c893ef700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8410ebf0 con 0x7f3c840fc7f0 2024-04-12T05:43:20.102 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c827fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c840fd150 0x7f3c8410e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c827fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c840fd150 0x7f3c8410e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:39808/0 (socket says 172.21.15.113:39808) 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c827fc700 1 -- 172.21.15.113:0/1050981633 learned_addr learned my addr 172.21.15.113:0/1050981633 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c827fc700 1 -- 172.21.15.113:0/1050981633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c840fc7f0 msgr2=0x7f3c8410dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c82ffd700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c840fc7f0 0x7f3c8410dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c827fc700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c840fc7f0 0x7f3c8410dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c827fc700 1 -- 172.21.15.113:0/1050981633 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c74005040 con 0x7f3c840fd150 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.103+0000 7f3c82ffd700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c840fc7f0 0x7f3c8410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.104+0000 7f3c827fc700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c840fd150 0x7f3c8410e4b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7f3c7800c710 tx=0x7f3c78007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.104+0000 7f3c6bfff700 1 -- 172.21.15.113:0/1050981633 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c7800e680 con 0x7f3c840fd150 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.104+0000 7f3c893ef700 1 -- 172.21.15.113:0/1050981633 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c84108280 con 0x7f3c840fd150 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.104+0000 7f3c6bfff700 1 -- 172.21.15.113:0/1050981633 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c7800ec90 con 0x7f3c840fd150 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.104+0000 7f3c6bfff700 1 -- 172.21.15.113:0/1050981633 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c78013cd0 con 0x7f3c840fd150 2024-04-12T05:43:20.103 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.104+0000 7f3c893ef700 1 -- 172.21.15.113:0/1050981633 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c84108840 con 0x7f3c840fd150 2024-04-12T05:43:20.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.105+0000 7f3c893ef700 1 -- 172.21.15.113:0/1050981633 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8405f590 con 0x7f3c840fd150 2024-04-12T05:43:20.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.105+0000 7f3c6bfff700 1 -- 172.21.15.113:0/1050981633 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3c78013410 con 0x7f3c840fd150 2024-04-12T05:43:20.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.106+0000 7f3c6bfff700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6c06caa0 0x7f3c6c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:20.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.106+0000 7f3c6bfff700 1 -- 172.21.15.113:0/1050981633 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3c7808c1f0 con 0x7f3c840fd150 2024-04-12T05:43:20.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.106+0000 7f3c82ffd700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6c06caa0 0x7f3c6c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:20.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.106+0000 7f3c82ffd700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6c06caa0 0x7f3c6c06ef60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f3c740043d0 tx=0x7f3c7400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:20.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.110+0000 7f3c6bfff700 1 -- 172.21.15.113:0/1050981633 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3c78059be0 con 0x7f3c840fd150 2024-04-12T05:43:20.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.317+0000 7f3c893ef700 1 -- 172.21.15.113:0/1050981633 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c84108ee0 con 0x7f3c6c06caa0 2024-04-12T05:43:20.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.318+0000 7f3c6bfff700 1 -- 172.21.15.113:0/1050981633 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3c84108ee0 con 0x7f3c6c06caa0 2024-04-12T05:43:20.315 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:43:20.315 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:43:20.315 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:43:20.315 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:43:20.315 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:43:20.316 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:43:20.316 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:43:20.316 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:43:20.316 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:43:20.316 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:43:20.316 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:43:20.316 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:43:20.316 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:43:20.316 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:43:20.316 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:43:20.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.321+0000 7f3c893ef700 1 -- 172.21.15.113:0/1050981633 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6c06caa0 msgr2=0x7f3c6c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:20.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.321+0000 7f3c893ef700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6c06caa0 0x7f3c6c06ef60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f3c740043d0 tx=0x7f3c7400b040).stop 2024-04-12T05:43:20.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.321+0000 7f3c893ef700 1 -- 172.21.15.113:0/1050981633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c840fd150 msgr2=0x7f3c8410e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:20.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.321+0000 7f3c893ef700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c840fd150 0x7f3c8410e4b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7f3c7800c710 tx=0x7f3c78007ec0).stop 2024-04-12T05:43:20.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.321+0000 7f3c893ef700 1 -- 172.21.15.113:0/1050981633 shutdown_connections 2024-04-12T05:43:20.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.321+0000 7f3c893ef700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6c06caa0 0x7f3c6c06ef60 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.321+0000 7f3c893ef700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c840fc7f0 0x7f3c8410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.321+0000 7f3c893ef700 1 --2- 172.21.15.113:0/1050981633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c840fd150 0x7f3c8410e4b0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.321+0000 7f3c893ef700 1 -- 172.21.15.113:0/1050981633 >> 172.21.15.113:0/1050981633 conn(0x7f3c84070590 msgr2=0x7f3c84060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:20.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.321+0000 7f3c893ef700 1 -- 172.21.15.113:0/1050981633 shutdown_connections 2024-04-12T05:43:20.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.321+0000 7f3c893ef700 1 -- 172.21.15.113:0/1050981633 wait complete. 2024-04-12T05:43:20.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.419+0000 7f8b05e62700 1 -- 172.21.15.113:0/137142612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b000fd150 msgr2=0x7f8b000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:20.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.419+0000 7f8b05e62700 1 --2- 172.21.15.113:0/137142612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b000fd150 0x7f8b000f6930 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f8af00047f0 tx=0x7f8af0009cb0).stop 2024-04-12T05:43:20.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.419+0000 7f8b05e62700 1 -- 172.21.15.113:0/137142612 shutdown_connections 2024-04-12T05:43:20.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.419+0000 7f8b05e62700 1 --2- 172.21.15.113:0/137142612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b000fd150 0x7f8b000f6930 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.419+0000 7f8b05e62700 1 --2- 172.21.15.113:0/137142612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b000fc7f0 0x7f8b000fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.419+0000 7f8b05e62700 1 -- 172.21.15.113:0/137142612 >> 172.21.15.113:0/137142612 conn(0x7f8b00070590 msgr2=0x7f8b000f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.419+0000 7f8b05e62700 1 -- 172.21.15.113:0/137142612 shutdown_connections 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.419+0000 7f8b05e62700 1 -- 172.21.15.113:0/137142612 wait complete. 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8b05e62700 1 Processor -- start 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8b05e62700 1 -- start start 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8b05e62700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b000fc7f0 0x7f8b001090e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8b05e62700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b000fd150 0x7f8b001095d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8b05e62700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b00109ac0 con 0x7f8b000fd150 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8b05e62700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b00109c00 con 0x7f8b000fc7f0 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8afeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b000fd150 0x7f8b001095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8afeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b000fd150 0x7f8b001095d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:40554/0 (socket says 172.21.15.113:40554) 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8afeffd700 1 -- 172.21.15.113:0/3755738065 learned_addr learned my addr 172.21.15.113:0/3755738065 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:20.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8aff7fe700 1 --2- 172.21.15.113:0/3755738065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b000fc7f0 0x7f8b001090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:20.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8afeffd700 1 -- 172.21.15.113:0/3755738065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b000fc7f0 msgr2=0x7f8b001090e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:20.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.420+0000 7f8afeffd700 1 --2- 172.21.15.113:0/3755738065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b000fc7f0 0x7f8b001090e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.421+0000 7f8afeffd700 1 -- 172.21.15.113:0/3755738065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af0005040 con 0x7f8b000fd150 2024-04-12T05:43:20.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.421+0000 7f8afeffd700 1 --2- 172.21.15.113:0/3755738065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b000fd150 0x7f8b001095d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f8af00099f0 tx=0x7f8af0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:20.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.421+0000 7f8afcff9700 1 -- 172.21.15.113:0/3755738065 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8af000e030 con 0x7f8b000fd150 2024-04-12T05:43:20.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.421+0000 7f8b05e62700 1 -- 172.21.15.113:0/3755738065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b00109e50 con 0x7f8b000fd150 2024-04-12T05:43:20.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.421+0000 7f8afcff9700 1 -- 172.21.15.113:0/3755738065 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8af0007e90 con 0x7f8b000fd150 2024-04-12T05:43:20.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.421+0000 7f8afcff9700 1 -- 172.21.15.113:0/3755738065 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8af0018070 con 0x7f8b000fd150 2024-04-12T05:43:20.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.421+0000 7f8b05e62700 1 -- 172.21.15.113:0/3755738065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b00112440 con 0x7f8b000fd150 2024-04-12T05:43:20.421 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.422+0000 7f8b05e62700 1 -- 172.21.15.113:0/3755738065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0005f590 con 0x7f8b000fd150 2024-04-12T05:43:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.422+0000 7f8afcff9700 1 -- 172.21.15.113:0/3755738065 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8af001d420 con 0x7f8b000fd150 2024-04-12T05:43:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.423+0000 7f8afcff9700 1 --2- 172.21.15.113:0/3755738065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ae806caa0 0x7f8ae806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.423+0000 7f8afcff9700 1 -- 172.21.15.113:0/3755738065 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f8af008d2f0 con 0x7f8b000fd150 2024-04-12T05:43:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.423+0000 7f8aff7fe700 1 --2- 172.21.15.113:0/3755738065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ae806caa0 0x7f8ae806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:20.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.423+0000 7f8aff7fe700 1 --2- 172.21.15.113:0/3755738065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ae806caa0 0x7f8ae806ef60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f8af40041d0 tx=0x7f8af4009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:20.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.427+0000 7f8afcff9700 1 -- 172.21.15.113:0/3755738065 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8af0057310 con 0x7f8b000fd150 2024-04-12T05:43:20.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:20 smithi113 ceph-mon[103371]: from='client.35682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:20.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:20 smithi113 ceph-mon[103371]: from='client.35686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:20.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:20 smithi113 ceph-mon[103371]: from='client.35690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:20.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:20 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1871569381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:20.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:20 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3748250304' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:43:20.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.694+0000 7f8b05e62700 1 -- 172.21.15.113:0/3755738065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8b00047540 con 0x7f8b000fd150 2024-04-12T05:43:20.693 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.694+0000 7f8afcff9700 1 -- 172.21.15.113:0/3755738065 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f8af005a930 con 0x7f8b000fd150 2024-04-12T05:43:20.693 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:43:20.693 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:43:20.693 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:43:20.693 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:43:20.693 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:43:20.693 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:43:20.693 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:43:20.694 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:43:20.694 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:43:20.694 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:43:20.694 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:43:20.694 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:43:20.694 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:43:20.694 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:43:20.694 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:43:20.694 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:43:20.694 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:43:20.694 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:43:20.695 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:43:20.695 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:43:20.695 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:43:20.695 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:43:20.695 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:43:20.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.696+0000 7f8b05e62700 1 -- 172.21.15.113:0/3755738065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ae806caa0 msgr2=0x7f8ae806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:20.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.696+0000 7f8b05e62700 1 --2- 172.21.15.113:0/3755738065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ae806caa0 0x7f8ae806ef60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f8af40041d0 tx=0x7f8af4009040).stop 2024-04-12T05:43:20.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.697+0000 7f8b05e62700 1 -- 172.21.15.113:0/3755738065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b000fd150 msgr2=0x7f8b001095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:20.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.697+0000 7f8b05e62700 1 --2- 172.21.15.113:0/3755738065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b000fd150 0x7f8b001095d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f8af00099f0 tx=0x7f8af0009cb0).stop 2024-04-12T05:43:20.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.697+0000 7f8b05e62700 1 -- 172.21.15.113:0/3755738065 shutdown_connections 2024-04-12T05:43:20.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.697+0000 7f8b05e62700 1 --2- 172.21.15.113:0/3755738065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8ae806caa0 0x7f8ae806ef60 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.697+0000 7f8b05e62700 1 --2- 172.21.15.113:0/3755738065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b000fc7f0 0x7f8b001090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.697+0000 7f8b05e62700 1 --2- 172.21.15.113:0/3755738065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8b000fd150 0x7f8b001095d0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:20.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.697+0000 7f8b05e62700 1 -- 172.21.15.113:0/3755738065 >> 172.21.15.113:0/3755738065 conn(0x7f8b00070590 msgr2=0x7f8b000fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:20.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.697+0000 7f8b05e62700 1 -- 172.21.15.113:0/3755738065 shutdown_connections 2024-04-12T05:43:20.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:20.697+0000 7f8b05e62700 1 -- 172.21.15.113:0/3755738065 wait complete. 2024-04-12T05:43:20.697 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:20 smithi192 ceph-mon[85304]: from='client.35682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:20.697 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:20 smithi192 ceph-mon[85304]: from='client.35686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:20.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:20 smithi192 ceph-mon[85304]: from='client.35690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:20.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:20 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1871569381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:20.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:20 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3748250304' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:43:21.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:21 smithi113 ceph-mon[103371]: pgmap v936: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:21.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:21 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3755738065' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:43:21.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:21 smithi192 ceph-mon[85304]: pgmap v936: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:21.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:21 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3755738065' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:43:22.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:22 smithi113 ceph-mon[103371]: from='client.35702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:22.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:22.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:22 smithi192 ceph-mon[85304]: from='client.35702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:22.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:23.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:23 smithi113 ceph-mon[103371]: pgmap v937: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:23.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:23 smithi192 ceph-mon[85304]: pgmap v937: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:25.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:25 smithi113 ceph-mon[103371]: pgmap v938: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:25.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:25 smithi192 ceph-mon[85304]: pgmap v938: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:27.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:27 smithi113 ceph-mon[103371]: pgmap v939: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:27.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:27 smithi192 ceph-mon[85304]: pgmap v939: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:29.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:29 smithi113 ceph-mon[103371]: pgmap v940: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:29.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:29 smithi192 ceph-mon[85304]: pgmap v940: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:31.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:31 smithi113 ceph-mon[103371]: pgmap v941: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:31.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:31 smithi192 ceph-mon[85304]: pgmap v941: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:33.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:33 smithi113 ceph-mon[103371]: pgmap v942: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:33.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:33 smithi192 ceph-mon[85304]: pgmap v942: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:35.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:35 smithi113 ceph-mon[103371]: pgmap v943: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:35.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:35 smithi192 ceph-mon[85304]: pgmap v943: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:37.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:37 smithi113 ceph-mon[103371]: pgmap v944: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:37.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:37.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:37 smithi192 ceph-mon[85304]: pgmap v944: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:37.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:39 smithi113 ceph-mon[103371]: pgmap v945: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:39 smithi192 ceph-mon[85304]: pgmap v945: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:41 smithi113 ceph-mon[103371]: pgmap v946: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:41.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:41 smithi192 ceph-mon[85304]: pgmap v946: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:43.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:43 smithi113 ceph-mon[103371]: pgmap v947: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:43.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:43 smithi192 ceph-mon[85304]: pgmap v947: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:45.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:45 smithi113 ceph-mon[103371]: pgmap v948: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:45.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:45 smithi192 ceph-mon[85304]: pgmap v948: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:46.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:46 smithi113 ceph-mon[103371]: pgmap v949: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:46.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:46 smithi192 ceph-mon[85304]: pgmap v949: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:49.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:48 smithi113 ceph-mon[103371]: pgmap v950: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:49.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:48 smithi192 ceph-mon[85304]: pgmap v950: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:50.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.803+0000 7f2299d24700 1 -- 172.21.15.113:0/1681258904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22940fd9d0 msgr2=0x7f22940fddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:50.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.803+0000 7f2299d24700 1 --2- 172.21.15.113:0/1681258904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22940fd9d0 0x7f22940fddb0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f22880047f0 tx=0x7f2288009cb0).stop 2024-04-12T05:43:50.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.803+0000 7f2299d24700 1 -- 172.21.15.113:0/1681258904 shutdown_connections 2024-04-12T05:43:50.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.803+0000 7f2299d24700 1 --2- 172.21.15.113:0/1681258904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22940fe330 0x7f22940f5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:50.801 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.803+0000 7f2299d24700 1 --2- 172.21.15.113:0/1681258904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22940fd9d0 0x7f22940fddb0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:50.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.803+0000 7f2299d24700 1 -- 172.21.15.113:0/1681258904 >> 172.21.15.113:0/1681258904 conn(0x7f229406cb70 msgr2=0x7f22940f3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:50.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.803+0000 7f2299d24700 1 -- 172.21.15.113:0/1681258904 shutdown_connections 2024-04-12T05:43:50.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.803+0000 7f2299d24700 1 -- 172.21.15.113:0/1681258904 wait complete. 2024-04-12T05:43:50.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.804+0000 7f2299d24700 1 Processor -- start 2024-04-12T05:43:50.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.804+0000 7f2299d24700 1 -- start start 2024-04-12T05:43:50.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.804+0000 7f2299d24700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22940fd9d0 0x7f22940f7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:50.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.804+0000 7f2299d24700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22940fe330 0x7f22940f7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:50.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.804+0000 7f2299d24700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22940f7b60 con 0x7f22940fe330 2024-04-12T05:43:50.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.804+0000 7f2299d24700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22940f7ca0 con 0x7f22940fd9d0 2024-04-12T05:43:50.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.804+0000 7f2292ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22940fe330 0x7f22940f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:50.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.804+0000 7f2292ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22940fe330 0x7f22940f7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50452/0 (socket says 172.21.15.113:50452) 2024-04-12T05:43:50.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.805+0000 7f2292ffd700 1 -- 172.21.15.113:0/1156484147 learned_addr learned my addr 172.21.15.113:0/1156484147 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:50.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.805+0000 7f22937fe700 1 --2- 172.21.15.113:0/1156484147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22940fd9d0 0x7f22940f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:50.805 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.805+0000 7f2292ffd700 1 -- 172.21.15.113:0/1156484147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22940fd9d0 msgr2=0x7f22940f7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.805+0000 7f2292ffd700 1 --2- 172.21.15.113:0/1156484147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22940fd9d0 0x7f22940f7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.805+0000 7f2292ffd700 1 -- 172.21.15.113:0/1156484147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2288005040 con 0x7f22940fe330 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.805+0000 7f2292ffd700 1 --2- 172.21.15.113:0/1156484147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22940fe330 0x7f22940f7670 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f228400c790 tx=0x7f2284004330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.805+0000 7f2290ff9700 1 -- 172.21.15.113:0/1156484147 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22840106f0 con 0x7f22940fe330 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.805+0000 7f2290ff9700 1 -- 172.21.15.113:0/1156484147 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2284010d00 con 0x7f22940fe330 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.805+0000 7f2299d24700 1 -- 172.21.15.113:0/1156484147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22940f7f50 con 0x7f22940fe330 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.805+0000 7f2290ff9700 1 -- 172.21.15.113:0/1156484147 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2284012da0 con 0x7f22940fe330 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.805+0000 7f2299d24700 1 -- 172.21.15.113:0/1156484147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2294114680 con 0x7f22940fe330 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.806+0000 7f2299d24700 1 -- 172.21.15.113:0/1156484147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2294047540 con 0x7f22940fe330 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.807+0000 7f2290ff9700 1 -- 172.21.15.113:0/1156484147 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2284012400 con 0x7f22940fe330 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.807+0000 7f2290ff9700 1 --2- 172.21.15.113:0/1156484147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f227c06caa0 0x7f227c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.807+0000 7f2290ff9700 1 -- 172.21.15.113:0/1156484147 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f228408b6d0 con 0x7f22940fe330 2024-04-12T05:43:50.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.807+0000 7f22937fe700 1 --2- 172.21.15.113:0/1156484147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f227c06caa0 0x7f227c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:50.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.807+0000 7f22937fe700 1 --2- 172.21.15.113:0/1156484147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f227c06caa0 0x7f227c06ef60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f2288004f10 tx=0x7f228800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:50.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:50.811+0000 7f2290ff9700 1 -- 172.21.15.113:0/1156484147 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f22840556f0 con 0x7f22940fe330 2024-04-12T05:43:51.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.016+0000 7f2299d24700 1 -- 172.21.15.113:0/1156484147 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2294059120 con 0x7f227c06caa0 2024-04-12T05:43:51.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.017+0000 7f2290ff9700 1 -- 172.21.15.113:0/1156484147 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2294059120 con 0x7f227c06caa0 2024-04-12T05:43:51.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.019+0000 7f2299d24700 1 -- 172.21.15.113:0/1156484147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f227c06caa0 msgr2=0x7f227c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.019+0000 7f2299d24700 1 --2- 172.21.15.113:0/1156484147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f227c06caa0 0x7f227c06ef60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f2288004f10 tx=0x7f228800b040).stop 2024-04-12T05:43:51.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.019+0000 7f2299d24700 1 -- 172.21.15.113:0/1156484147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22940fe330 msgr2=0x7f22940f7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.019+0000 7f2299d24700 1 --2- 172.21.15.113:0/1156484147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22940fe330 0x7f22940f7670 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f228400c790 tx=0x7f2284004330).stop 2024-04-12T05:43:51.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.019+0000 7f2299d24700 1 -- 172.21.15.113:0/1156484147 shutdown_connections 2024-04-12T05:43:51.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.019+0000 7f2299d24700 1 --2- 172.21.15.113:0/1156484147 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f227c06caa0 0x7f227c06ef60 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.019+0000 7f2299d24700 1 --2- 172.21.15.113:0/1156484147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22940fd9d0 0x7f22940f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.019+0000 7f2299d24700 1 --2- 172.21.15.113:0/1156484147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22940fe330 0x7f22940f7670 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.019+0000 7f2299d24700 1 -- 172.21.15.113:0/1156484147 >> 172.21.15.113:0/1156484147 conn(0x7f229406cb70 msgr2=0x7f2294101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:51.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.019+0000 7f2299d24700 1 -- 172.21.15.113:0/1156484147 shutdown_connections 2024-04-12T05:43:51.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.019+0000 7f2299d24700 1 -- 172.21.15.113:0/1156484147 wait complete. 2024-04-12T05:43:51.027 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:43:51.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.118+0000 7fe1512d3700 1 -- 172.21.15.113:0/145575571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe14c0fd8c0 msgr2=0x7fe14c0fdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.118+0000 7fe1512d3700 1 --2- 172.21.15.113:0/145575571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe14c0fd8c0 0x7fe14c0fdca0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fe1400047f0 tx=0x7fe140009cb0).stop 2024-04-12T05:43:51.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.118+0000 7fe1512d3700 1 -- 172.21.15.113:0/145575571 shutdown_connections 2024-04-12T05:43:51.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.118+0000 7fe1512d3700 1 --2- 172.21.15.113:0/145575571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe14c0fe220 0x7fe14c0f7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.118+0000 7fe1512d3700 1 --2- 172.21.15.113:0/145575571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe14c0fd8c0 0x7fe14c0fdca0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.118+0000 7fe1512d3700 1 -- 172.21.15.113:0/145575571 >> 172.21.15.113:0/145575571 conn(0x7fe14c0f34a0 msgr2=0x7fe14c0f58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:51.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.118+0000 7fe1512d3700 1 -- 172.21.15.113:0/145575571 shutdown_connections 2024-04-12T05:43:51.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.118+0000 7fe1512d3700 1 -- 172.21.15.113:0/145575571 wait complete. 2024-04-12T05:43:51.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.118+0000 7fe1512d3700 1 Processor -- start 2024-04-12T05:43:51.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe1512d3700 1 -- start start 2024-04-12T05:43:51.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe1512d3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe14c0fd8c0 0x7fe14c10e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:51.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe1512d3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe14c0fe220 0x7fe14c10e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:51.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe1512d3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c10eb90 con 0x7fe14c0fd8c0 2024-04-12T05:43:51.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe1512d3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c10ecd0 con 0x7fe14c0fe220 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe14affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe14c0fd8c0 0x7fe14c10e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe14affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe14c0fd8c0 0x7fe14c10e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50472/0 (socket says 172.21.15.113:50472) 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe14affd700 1 -- 172.21.15.113:0/3068180833 learned_addr learned my addr 172.21.15.113:0/3068180833 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe14a7fc700 1 --2- 172.21.15.113:0/3068180833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe14c0fe220 0x7fe14c10e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe14affd700 1 -- 172.21.15.113:0/3068180833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe14c0fe220 msgr2=0x7fe14c10e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe14affd700 1 --2- 172.21.15.113:0/3068180833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe14c0fe220 0x7fe14c10e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe14affd700 1 -- 172.21.15.113:0/3068180833 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe140005040 con 0x7fe14c0fd8c0 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe14affd700 1 --2- 172.21.15.113:0/3068180833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe14c0fd8c0 0x7fe14c10e0a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7fe14000b120 tx=0x7fe140009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe133fff700 1 -- 172.21.15.113:0/3068180833 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe14000e030 con 0x7fe14c0fd8c0 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.119+0000 7fe133fff700 1 -- 172.21.15.113:0/3068180833 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe140007e90 con 0x7fe14c0fd8c0 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.120+0000 7fe1512d3700 1 -- 172.21.15.113:0/3068180833 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe14c108300 con 0x7fe14c0fd8c0 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.120+0000 7fe133fff700 1 -- 172.21.15.113:0/3068180833 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe140018070 con 0x7fe14c0fd8c0 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.120+0000 7fe1512d3700 1 -- 172.21.15.113:0/3068180833 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14c1087e0 con 0x7fe14c0fd8c0 2024-04-12T05:43:51.119 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.120+0000 7fe1512d3700 1 -- 172.21.15.113:0/3068180833 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14c060390 con 0x7fe14c0fd8c0 2024-04-12T05:43:51.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.121+0000 7fe133fff700 1 -- 172.21.15.113:0/3068180833 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe14001d420 con 0x7fe14c0fd8c0 2024-04-12T05:43:51.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.121+0000 7fe133fff700 1 --2- 172.21.15.113:0/3068180833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406caa0 0x7fe13406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:51.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.121+0000 7fe133fff700 1 -- 172.21.15.113:0/3068180833 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe14008d160 con 0x7fe14c0fd8c0 2024-04-12T05:43:51.120 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.121+0000 7fe14a7fc700 1 --2- 172.21.15.113:0/3068180833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406caa0 0x7fe13406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:51.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.122+0000 7fe14a7fc700 1 --2- 172.21.15.113:0/3068180833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406caa0 0x7fe13406ef60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fe14c1099e0 tx=0x7fe13c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:51.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.126+0000 7fe133fff700 1 -- 172.21.15.113:0/3068180833 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe14005ac10 con 0x7fe14c0fd8c0 2024-04-12T05:43:51.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.331+0000 7fe1512d3700 1 -- 172.21.15.113:0/3068180833 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe14c109830 con 0x7fe13406caa0 2024-04-12T05:43:51.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.332+0000 7fe133fff700 1 -- 172.21.15.113:0/3068180833 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe14c109830 con 0x7fe13406caa0 2024-04-12T05:43:51.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.334+0000 7fe1512d3700 1 -- 172.21.15.113:0/3068180833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406caa0 msgr2=0x7fe13406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.334+0000 7fe1512d3700 1 --2- 172.21.15.113:0/3068180833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406caa0 0x7fe13406ef60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fe14c1099e0 tx=0x7fe13c009040).stop 2024-04-12T05:43:51.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.334+0000 7fe1512d3700 1 -- 172.21.15.113:0/3068180833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe14c0fd8c0 msgr2=0x7fe14c10e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.334+0000 7fe1512d3700 1 --2- 172.21.15.113:0/3068180833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe14c0fd8c0 0x7fe14c10e0a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7fe14000b120 tx=0x7fe140009cb0).stop 2024-04-12T05:43:51.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.334+0000 7fe1512d3700 1 -- 172.21.15.113:0/3068180833 shutdown_connections 2024-04-12T05:43:51.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.334+0000 7fe1512d3700 1 --2- 172.21.15.113:0/3068180833 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13406caa0 0x7fe13406ef60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.334+0000 7fe1512d3700 1 --2- 172.21.15.113:0/3068180833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe14c0fd8c0 0x7fe14c10e0a0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.334+0000 7fe1512d3700 1 --2- 172.21.15.113:0/3068180833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe14c0fe220 0x7fe14c10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.334+0000 7fe1512d3700 1 -- 172.21.15.113:0/3068180833 >> 172.21.15.113:0/3068180833 conn(0x7fe14c0f34a0 msgr2=0x7fe14c0fbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:51.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.335+0000 7fe1512d3700 1 -- 172.21.15.113:0/3068180833 shutdown_connections 2024-04-12T05:43:51.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.335+0000 7fe1512d3700 1 -- 172.21.15.113:0/3068180833 wait complete. 2024-04-12T05:43:51.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:50 smithi113 ceph-mon[103371]: pgmap v951: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:51.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 -- 172.21.15.113:0/3429060631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40fd190 msgr2=0x7f91d40f6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 --2- 172.21.15.113:0/3429060631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40fd190 0x7f91d40f6970 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f91c80047f0 tx=0x7f91c8009cb0).stop 2024-04-12T05:43:51.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 -- 172.21.15.113:0/3429060631 shutdown_connections 2024-04-12T05:43:51.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 --2- 172.21.15.113:0/3429060631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40fd190 0x7f91d40f6970 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 --2- 172.21.15.113:0/3429060631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91d40fc830 0x7f91d40fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 -- 172.21.15.113:0/3429060631 >> 172.21.15.113:0/3429060631 conn(0x7f91d406c760 msgr2=0x7f91d406cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:51.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 -- 172.21.15.113:0/3429060631 shutdown_connections 2024-04-12T05:43:51.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 -- 172.21.15.113:0/3429060631 wait complete. 2024-04-12T05:43:51.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 Processor -- start 2024-04-12T05:43:51.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 -- start start 2024-04-12T05:43:51.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91d40fc830 0x7f91d410cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:51.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40fd190 0x7f91d410d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:51.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d410da00 con 0x7f91d40fd190 2024-04-12T05:43:51.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91da1ce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d410db40 con 0x7f91d40fc830 2024-04-12T05:43:51.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91d2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40fd190 0x7f91d410d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:51.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91d2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40fd190 0x7f91d410d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50494/0 (socket says 172.21.15.113:50494) 2024-04-12T05:43:51.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.434+0000 7f91d2ffd700 1 -- 172.21.15.113:0/1291497532 learned_addr learned my addr 172.21.15.113:0/1291497532 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:51.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.435+0000 7f91d37fe700 1 --2- 172.21.15.113:0/1291497532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91d40fc830 0x7f91d410cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:51.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.435+0000 7f91d2ffd700 1 -- 172.21.15.113:0/1291497532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91d40fc830 msgr2=0x7f91d410cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.435+0000 7f91d2ffd700 1 --2- 172.21.15.113:0/1291497532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91d40fc830 0x7f91d410cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.435+0000 7f91d2ffd700 1 -- 172.21.15.113:0/1291497532 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c8005040 con 0x7f91d40fd190 2024-04-12T05:43:51.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.435+0000 7f91d2ffd700 1 --2- 172.21.15.113:0/1291497532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40fd190 0x7f91d410d400 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f91c8007b40 tx=0x7f91c8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:51.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.435+0000 7f91d0ff9700 1 -- 172.21.15.113:0/1291497532 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91c800e030 con 0x7f91d40fd190 2024-04-12T05:43:51.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.435+0000 7f91d0ff9700 1 -- 172.21.15.113:0/1291497532 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91c8007e90 con 0x7f91d40fd190 2024-04-12T05:43:51.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.435+0000 7f91da1ce700 1 -- 172.21.15.113:0/1291497532 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d4107170 con 0x7f91d40fd190 2024-04-12T05:43:51.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.435+0000 7f91d0ff9700 1 -- 172.21.15.113:0/1291497532 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91c801a070 con 0x7f91d40fd190 2024-04-12T05:43:51.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.435+0000 7f91da1ce700 1 -- 172.21.15.113:0/1291497532 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d41076d0 con 0x7f91d40fd190 2024-04-12T05:43:51.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.436+0000 7f91da1ce700 1 -- 172.21.15.113:0/1291497532 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d405f5a0 con 0x7f91d40fd190 2024-04-12T05:43:51.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.437+0000 7f91d0ff9700 1 -- 172.21.15.113:0/1291497532 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f91c80127b0 con 0x7f91d40fd190 2024-04-12T05:43:51.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.437+0000 7f91d0ff9700 1 --2- 172.21.15.113:0/1291497532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91bc06caa0 0x7f91bc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:51.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.437+0000 7f91d0ff9700 1 -- 172.21.15.113:0/1291497532 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f91c808f880 con 0x7f91d40fd190 2024-04-12T05:43:51.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.437+0000 7f91d37fe700 1 --2- 172.21.15.113:0/1291497532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91bc06caa0 0x7f91bc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:51.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.438+0000 7f91d37fe700 1 --2- 172.21.15.113:0/1291497532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91bc06caa0 0x7f91bc06ef60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f91c4004b60 tx=0x7f91c4009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:51.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.442+0000 7f91d0ff9700 1 -- 172.21.15.113:0/1291497532 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f91c805d3f0 con 0x7f91d40fd190 2024-04-12T05:43:51.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:50 smithi192 ceph-mon[85304]: pgmap v951: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:51.634 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.636+0000 7f91da1ce700 1 -- 172.21.15.113:0/1291497532 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f91d40feb80 con 0x7f91bc06caa0 2024-04-12T05:43:51.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.645+0000 7f91d0ff9700 1 -- 172.21.15.113:0/1291497532 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f91d40feb80 con 0x7f91bc06caa0 2024-04-12T05:43:51.643 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:43:51.643 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (31m) 9m ago 42m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:43:51.643 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (29m) 9m ago 42m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:43:51.643 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (29m) 9m ago 42m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:43:51.643 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (31m) 9m ago 42m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (24m) 9m ago 37m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (24m) 9m ago 37m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (24m) 9m ago 37m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (9m) 9m ago 37m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (33m) 9m ago 44m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (32m) 9m ago 42m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (30m) 9m ago 44m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (29m) 9m ago 42m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (32m) 9m ago 42m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (32m) 9m ago 42m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (28m) 9m ago 41m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (28m) 9m ago 40m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (27m) 9m ago 40m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:43:51.644 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (26m) 9m ago 40m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:43:51.645 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (26m) 9m ago 39m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:43:51.645 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (25m) 9m ago 39m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:43:51.645 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (31m) 9m ago 41m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:43:51.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.648+0000 7f91da1ce700 1 -- 172.21.15.113:0/1291497532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91bc06caa0 msgr2=0x7f91bc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.648+0000 7f91da1ce700 1 --2- 172.21.15.113:0/1291497532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91bc06caa0 0x7f91bc06ef60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f91c4004b60 tx=0x7f91c4009040).stop 2024-04-12T05:43:51.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.648+0000 7f91da1ce700 1 -- 172.21.15.113:0/1291497532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40fd190 msgr2=0x7f91d410d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.648+0000 7f91da1ce700 1 --2- 172.21.15.113:0/1291497532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40fd190 0x7f91d410d400 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f91c8007b40 tx=0x7f91c8009cb0).stop 2024-04-12T05:43:51.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.648+0000 7f91da1ce700 1 -- 172.21.15.113:0/1291497532 shutdown_connections 2024-04-12T05:43:51.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.648+0000 7f91da1ce700 1 --2- 172.21.15.113:0/1291497532 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f91bc06caa0 0x7f91bc06ef60 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.648+0000 7f91da1ce700 1 --2- 172.21.15.113:0/1291497532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91d40fc830 0x7f91d410cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.648+0000 7f91da1ce700 1 --2- 172.21.15.113:0/1291497532 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91d40fd190 0x7f91d410d400 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.648+0000 7f91da1ce700 1 -- 172.21.15.113:0/1291497532 >> 172.21.15.113:0/1291497532 conn(0x7f91d406c760 msgr2=0x7f91d40fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:51.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.648+0000 7f91da1ce700 1 -- 172.21.15.113:0/1291497532 shutdown_connections 2024-04-12T05:43:51.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.648+0000 7f91da1ce700 1 -- 172.21.15.113:0/1291497532 wait complete. 2024-04-12T05:43:51.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.744+0000 7f221d3fd700 1 -- 172.21.15.113:0/4228975201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22180fe2d0 msgr2=0x7f22180f7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.744+0000 7f221d3fd700 1 --2- 172.21.15.113:0/4228975201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22180fe2d0 0x7f22180f7ab0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7f220c0047f0 tx=0x7f220c009cb0).stop 2024-04-12T05:43:51.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.744+0000 7f221d3fd700 1 -- 172.21.15.113:0/4228975201 shutdown_connections 2024-04-12T05:43:51.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.744+0000 7f221d3fd700 1 --2- 172.21.15.113:0/4228975201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22180fe2d0 0x7f22180f7ab0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.744+0000 7f221d3fd700 1 --2- 172.21.15.113:0/4228975201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22180fd970 0x7f22180fdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.744+0000 7f221d3fd700 1 -- 172.21.15.113:0/4228975201 >> 172.21.15.113:0/4228975201 conn(0x7f22180f3550 msgr2=0x7f22180f5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:51.742 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.744+0000 7f221d3fd700 1 -- 172.21.15.113:0/4228975201 shutdown_connections 2024-04-12T05:43:51.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.744+0000 7f221d3fd700 1 -- 172.21.15.113:0/4228975201 wait complete. 2024-04-12T05:43:51.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.745+0000 7f221d3fd700 1 Processor -- start 2024-04-12T05:43:51.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.745+0000 7f221d3fd700 1 -- start start 2024-04-12T05:43:51.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.745+0000 7f221d3fd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22180fd970 0x7f221810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:51.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.745+0000 7f221d3fd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22180fe2d0 0x7f221810e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:51.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.745+0000 7f221d3fd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221810ead0 con 0x7f22180fe2d0 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.745+0000 7f221d3fd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221810ec10 con 0x7f22180fd970 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.745+0000 7f22167fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22180fe2d0 0x7f221810e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.745+0000 7f22167fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22180fe2d0 0x7f221810e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50506/0 (socket says 172.21.15.113:50506) 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.745+0000 7f22167fc700 1 -- 172.21.15.113:0/2661046927 learned_addr learned my addr 172.21.15.113:0/2661046927 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.745+0000 7f2216ffd700 1 --2- 172.21.15.113:0/2661046927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22180fd970 0x7f221810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.746+0000 7f22167fc700 1 -- 172.21.15.113:0/2661046927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22180fd970 msgr2=0x7f221810dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.746+0000 7f22167fc700 1 --2- 172.21.15.113:0/2661046927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22180fd970 0x7f221810dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.746+0000 7f22167fc700 1 -- 172.21.15.113:0/2661046927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c005040 con 0x7f22180fe2d0 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.746+0000 7f22167fc700 1 --2- 172.21.15.113:0/2661046927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22180fe2d0 0x7f221810e4d0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f220c0099f0 tx=0x7f220c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.746+0000 7f21fffff700 1 -- 172.21.15.113:0/2661046927 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220c00e030 con 0x7f22180fe2d0 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.746+0000 7f221d3fd700 1 -- 172.21.15.113:0/2661046927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2218108240 con 0x7f22180fe2d0 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.746+0000 7f21fffff700 1 -- 172.21.15.113:0/2661046927 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f220c007e90 con 0x7f22180fe2d0 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.746+0000 7f21fffff700 1 -- 172.21.15.113:0/2661046927 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220c018070 con 0x7f22180fe2d0 2024-04-12T05:43:51.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.746+0000 7f221d3fd700 1 -- 172.21.15.113:0/2661046927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22181087a0 con 0x7f22180fe2d0 2024-04-12T05:43:51.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.747+0000 7f221d3fd700 1 -- 172.21.15.113:0/2661046927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2218060370 con 0x7f22180fe2d0 2024-04-12T05:43:51.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.747+0000 7f21fffff700 1 -- 172.21.15.113:0/2661046927 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f220c01d420 con 0x7f22180fe2d0 2024-04-12T05:43:51.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.748+0000 7f21fffff700 1 --2- 172.21.15.113:0/2661046927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f220006caa0 0x7f220006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:51.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.748+0000 7f21fffff700 1 -- 172.21.15.113:0/2661046927 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f220c08d240 con 0x7f22180fe2d0 2024-04-12T05:43:51.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.748+0000 7f2216ffd700 1 --2- 172.21.15.113:0/2661046927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f220006caa0 0x7f220006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:51.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.748+0000 7f2216ffd700 1 --2- 172.21.15.113:0/2661046927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f220006caa0 0x7f220006ef60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f2208004c20 tx=0x7f2208009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:51.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:51.753+0000 7f21fffff700 1 -- 172.21.15.113:0/2661046927 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f220c05acf0 con 0x7f22180fe2d0 2024-04-12T05:43:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.040+0000 7f221d3fd700 1 -- 172.21.15.113:0/2661046927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2218047540 con 0x7f22180fe2d0 2024-04-12T05:43:52.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.040+0000 7f21fffff700 1 -- 172.21.15.113:0/2661046927 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f220c05a880 con 0x7f22180fe2d0 2024-04-12T05:43:52.037 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:43:52.037 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:43:52.037 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:43:52.037 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:43:52.037 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:43:52.037 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:43:52.037 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:43:52.037 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:43:52.037 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:43:52.038 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:43:52.038 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:43:52.038 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:43:52.038 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:43:52.038 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:43:52.038 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:43:52.038 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:43:52.038 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:43:52.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.042+0000 7f221d3fd700 1 -- 172.21.15.113:0/2661046927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f220006caa0 msgr2=0x7f220006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.042+0000 7f221d3fd700 1 --2- 172.21.15.113:0/2661046927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f220006caa0 0x7f220006ef60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f2208004c20 tx=0x7f2208009040).stop 2024-04-12T05:43:52.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.042+0000 7f221d3fd700 1 -- 172.21.15.113:0/2661046927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22180fe2d0 msgr2=0x7f221810e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.042+0000 7f221d3fd700 1 --2- 172.21.15.113:0/2661046927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22180fe2d0 0x7f221810e4d0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f220c0099f0 tx=0x7f220c009cb0).stop 2024-04-12T05:43:52.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.042+0000 7f221d3fd700 1 -- 172.21.15.113:0/2661046927 shutdown_connections 2024-04-12T05:43:52.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.042+0000 7f221d3fd700 1 --2- 172.21.15.113:0/2661046927 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f220006caa0 0x7f220006ef60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.042+0000 7f221d3fd700 1 --2- 172.21.15.113:0/2661046927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22180fd970 0x7f221810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.042+0000 7f221d3fd700 1 --2- 172.21.15.113:0/2661046927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22180fe2d0 0x7f221810e4d0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.042+0000 7f221d3fd700 1 -- 172.21.15.113:0/2661046927 >> 172.21.15.113:0/2661046927 conn(0x7f22180f3550 msgr2=0x7f22180fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:52.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.042+0000 7f221d3fd700 1 -- 172.21.15.113:0/2661046927 shutdown_connections 2024-04-12T05:43:52.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.042+0000 7f221d3fd700 1 -- 172.21.15.113:0/2661046927 wait complete. 2024-04-12T05:43:52.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.140+0000 7fb8432d8700 1 -- 172.21.15.113:0/2865006879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83c0fc7f0 msgr2=0x7fb83c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.140+0000 7fb8432d8700 1 --2- 172.21.15.113:0/2865006879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83c0fc7f0 0x7fb83c0fcbd0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7fb82c0047f0 tx=0x7fb82c009cb0).stop 2024-04-12T05:43:52.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.140+0000 7fb8432d8700 1 -- 172.21.15.113:0/2865006879 shutdown_connections 2024-04-12T05:43:52.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.140+0000 7fb8432d8700 1 --2- 172.21.15.113:0/2865006879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb83c0fd150 0x7fb83c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.140+0000 7fb8432d8700 1 --2- 172.21.15.113:0/2865006879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83c0fc7f0 0x7fb83c0fcbd0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.140+0000 7fb8432d8700 1 -- 172.21.15.113:0/2865006879 >> 172.21.15.113:0/2865006879 conn(0x7fb83c070590 msgr2=0x7fb83c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:52.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.140+0000 7fb8432d8700 1 -- 172.21.15.113:0/2865006879 shutdown_connections 2024-04-12T05:43:52.137 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.140+0000 7fb8432d8700 1 -- 172.21.15.113:0/2865006879 wait complete. 2024-04-12T05:43:52.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb8432d8700 1 Processor -- start 2024-04-12T05:43:52.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb8432d8700 1 -- start start 2024-04-12T05:43:52.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb8432d8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb83c0fc7f0 0x7fb83c1090d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:52.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb8432d8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83c0fd150 0x7fb83c1095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb8432d8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c109ab0 con 0x7fb83c0fd150 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb8432d8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c109bf0 con 0x7fb83c0fc7f0 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb840873700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83c0fd150 0x7fb83c1095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb840873700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83c0fd150 0x7fb83c1095c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50532/0 (socket says 172.21.15.113:50532) 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb840873700 1 -- 172.21.15.113:0/1831672552 learned_addr learned my addr 172.21.15.113:0/1831672552 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb841074700 1 --2- 172.21.15.113:0/1831672552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb83c0fc7f0 0x7fb83c1090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb840873700 1 -- 172.21.15.113:0/1831672552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb83c0fc7f0 msgr2=0x7fb83c1090d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb840873700 1 --2- 172.21.15.113:0/1831672552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb83c0fc7f0 0x7fb83c1090d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb840873700 1 -- 172.21.15.113:0/1831672552 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c005040 con 0x7fb83c0fd150 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.141+0000 7fb840873700 1 --2- 172.21.15.113:0/1831672552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83c0fd150 0x7fb83c1095c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7fb83800c710 tx=0x7fb838007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.142+0000 7fb832777700 1 -- 172.21.15.113:0/1831672552 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb83800e680 con 0x7fb83c0fd150 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.142+0000 7fb8432d8700 1 -- 172.21.15.113:0/1831672552 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83c109ea0 con 0x7fb83c0fd150 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.142+0000 7fb832777700 1 -- 172.21.15.113:0/1831672552 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb83800ec90 con 0x7fb83c0fd150 2024-04-12T05:43:52.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.142+0000 7fb832777700 1 -- 172.21.15.113:0/1831672552 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb838013d40 con 0x7fb83c0fd150 2024-04-12T05:43:52.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.142+0000 7fb8432d8700 1 -- 172.21.15.113:0/1831672552 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83c1124c0 con 0x7fb83c0fd150 2024-04-12T05:43:52.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.143+0000 7fb8432d8700 1 -- 172.21.15.113:0/1831672552 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83c05f590 con 0x7fb83c0fd150 2024-04-12T05:43:52.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.143+0000 7fb832777700 1 -- 172.21.15.113:0/1831672552 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb83801d440 con 0x7fb83c0fd150 2024-04-12T05:43:52.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.143+0000 7fb832777700 1 --2- 172.21.15.113:0/1831672552 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb82806caa0 0x7fb82806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:52.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.144+0000 7fb832777700 1 -- 172.21.15.113:0/1831672552 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb83808bcb0 con 0x7fb83c0fd150 2024-04-12T05:43:52.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.144+0000 7fb841074700 1 --2- 172.21.15.113:0/1831672552 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb82806caa0 0x7fb82806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:52.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.144+0000 7fb841074700 1 --2- 172.21.15.113:0/1831672552 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb82806caa0 0x7fb82806ef60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fb82c007cf0 tx=0x7fb82c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:52.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.148+0000 7fb832777700 1 -- 172.21.15.113:0/1831672552 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb838055cd0 con 0x7fb83c0fd150 2024-04-12T05:43:52.263 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:51 smithi113 ceph-mon[103371]: from='client.35710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:52.263 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:51 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:52.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.377+0000 7fb8432d8700 1 -- 172.21.15.113:0/1831672552 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb83c060d30 con 0x7fb83c0fd150 2024-04-12T05:43:52.374 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.377+0000 7fb832777700 1 -- 172.21.15.113:0/1831672552 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fb8380592f0 con 0x7fb83c0fd150 2024-04-12T05:43:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:43:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:43:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:43:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:43:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:43:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:43:52.377 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:43:52.377 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:43:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:43:52.377 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:43:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:43:52.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.379+0000 7fb8432d8700 1 -- 172.21.15.113:0/1831672552 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb82806caa0 msgr2=0x7fb82806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.379+0000 7fb8432d8700 1 --2- 172.21.15.113:0/1831672552 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb82806caa0 0x7fb82806ef60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fb82c007cf0 tx=0x7fb82c00b040).stop 2024-04-12T05:43:52.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.379+0000 7fb8432d8700 1 -- 172.21.15.113:0/1831672552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83c0fd150 msgr2=0x7fb83c1095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.379+0000 7fb8432d8700 1 --2- 172.21.15.113:0/1831672552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83c0fd150 0x7fb83c1095c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7fb83800c710 tx=0x7fb838007ec0).stop 2024-04-12T05:43:52.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.379+0000 7fb8432d8700 1 -- 172.21.15.113:0/1831672552 shutdown_connections 2024-04-12T05:43:52.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.379+0000 7fb8432d8700 1 --2- 172.21.15.113:0/1831672552 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb82806caa0 0x7fb82806ef60 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.379+0000 7fb8432d8700 1 --2- 172.21.15.113:0/1831672552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb83c0fc7f0 0x7fb83c1090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.379+0000 7fb8432d8700 1 --2- 172.21.15.113:0/1831672552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb83c0fd150 0x7fb83c1095c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.379+0000 7fb8432d8700 1 -- 172.21.15.113:0/1831672552 >> 172.21.15.113:0/1831672552 conn(0x7fb83c070590 msgr2=0x7fb83c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:52.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.379+0000 7fb8432d8700 1 -- 172.21.15.113:0/1831672552 shutdown_connections 2024-04-12T05:43:52.379 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.379+0000 7fb8432d8700 1 -- 172.21.15.113:0/1831672552 wait complete. 2024-04-12T05:43:52.379 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:43:52.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:52 smithi192 ceph-mon[85304]: from='client.35710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:52.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:43:52.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.478+0000 7ff2661dc700 1 -- 172.21.15.113:0/2586416170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2600fe250 msgr2=0x7ff2600f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.478+0000 7ff2661dc700 1 --2- 172.21.15.113:0/2586416170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2600fe250 0x7ff2600f7a30 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7ff2500047f0 tx=0x7ff250009cb0).stop 2024-04-12T05:43:52.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.478+0000 7ff2661dc700 1 -- 172.21.15.113:0/2586416170 shutdown_connections 2024-04-12T05:43:52.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.478+0000 7ff2661dc700 1 --2- 172.21.15.113:0/2586416170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2600fe250 0x7ff2600f7a30 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.478+0000 7ff2661dc700 1 --2- 172.21.15.113:0/2586416170 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2600fd8f0 0x7ff2600fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.478+0000 7ff2661dc700 1 -- 172.21.15.113:0/2586416170 >> 172.21.15.113:0/2586416170 conn(0x7ff2600f34b0 msgr2=0x7ff2600f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:52.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.478+0000 7ff2661dc700 1 -- 172.21.15.113:0/2586416170 shutdown_connections 2024-04-12T05:43:52.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.478+0000 7ff2661dc700 1 -- 172.21.15.113:0/2586416170 wait complete. 2024-04-12T05:43:52.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.479+0000 7ff2661dc700 1 Processor -- start 2024-04-12T05:43:52.476 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.479+0000 7ff2661dc700 1 -- start start 2024-04-12T05:43:52.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.479+0000 7ff2661dc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2600fd8f0 0x7ff26006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:52.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.479+0000 7ff2661dc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2600fe250 0x7ff26006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:52.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.479+0000 7ff2661dc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26006bab0 con 0x7ff2600fe250 2024-04-12T05:43:52.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.479+0000 7ff2661dc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26006bbf0 con 0x7ff2600fd8f0 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.479+0000 7ff25effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2600fe250 0x7ff26006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.479+0000 7ff25effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2600fe250 0x7ff26006b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50554/0 (socket says 172.21.15.113:50554) 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.479+0000 7ff25effd700 1 -- 172.21.15.113:0/4192066736 learned_addr learned my addr 172.21.15.113:0/4192066736 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.479+0000 7ff25f7fe700 1 --2- 172.21.15.113:0/4192066736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2600fd8f0 0x7ff26006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.480+0000 7ff25effd700 1 -- 172.21.15.113:0/4192066736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2600fd8f0 msgr2=0x7ff26006b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.480+0000 7ff25effd700 1 --2- 172.21.15.113:0/4192066736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2600fd8f0 0x7ff26006b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.480+0000 7ff25effd700 1 -- 172.21.15.113:0/4192066736 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff250005040 con 0x7ff2600fe250 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.480+0000 7ff25effd700 1 --2- 172.21.15.113:0/4192066736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2600fe250 0x7ff26006b5c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7ff2500099f0 tx=0x7ff250009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.480+0000 7ff25cff9700 1 -- 172.21.15.113:0/4192066736 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff25000e030 con 0x7ff2600fe250 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.480+0000 7ff2661dc700 1 -- 172.21.15.113:0/4192066736 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26006be40 con 0x7ff2600fe250 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.480+0000 7ff25cff9700 1 -- 172.21.15.113:0/4192066736 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff250007e90 con 0x7ff2600fe250 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.480+0000 7ff25cff9700 1 -- 172.21.15.113:0/4192066736 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff250018070 con 0x7ff2600fe250 2024-04-12T05:43:52.478 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.480+0000 7ff2661dc700 1 -- 172.21.15.113:0/4192066736 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff260114600 con 0x7ff2600fe250 2024-04-12T05:43:52.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.481+0000 7ff2661dc700 1 -- 172.21.15.113:0/4192066736 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26005f590 con 0x7ff2600fe250 2024-04-12T05:43:52.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.481+0000 7ff25cff9700 1 -- 172.21.15.113:0/4192066736 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff25001d420 con 0x7ff2600fe250 2024-04-12T05:43:52.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.482+0000 7ff25cff9700 1 --2- 172.21.15.113:0/4192066736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff24c06caa0 0x7ff24c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:52.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.482+0000 7ff25cff9700 1 -- 172.21.15.113:0/4192066736 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff25008d240 con 0x7ff2600fe250 2024-04-12T05:43:52.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.482+0000 7ff25f7fe700 1 --2- 172.21.15.113:0/4192066736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff24c06caa0 0x7ff24c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:52.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.482+0000 7ff25f7fe700 1 --2- 172.21.15.113:0/4192066736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff24c06caa0 0x7ff24c06ef60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7ff2480041d0 tx=0x7ff248009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:52.483 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.486+0000 7ff25cff9700 1 -- 172.21.15.113:0/4192066736 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff250057260 con 0x7ff2600fe250 2024-04-12T05:43:52.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.693+0000 7ff2661dc700 1 -- 172.21.15.113:0/4192066736 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff260059120 con 0x7ff24c06caa0 2024-04-12T05:43:52.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.695+0000 7ff25cff9700 1 -- 172.21.15.113:0/4192066736 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff260059120 con 0x7ff24c06caa0 2024-04-12T05:43:52.693 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:43:52.693 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:43:52.693 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:43:52.694 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:43:52.694 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:43:52.694 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:43:52.695 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:43:52.695 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:43:52.695 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:43:52.695 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:43:52.695 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:43:52.696 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:43:52.696 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:43:52.696 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:43:52.696 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:43:52.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.697+0000 7ff2661dc700 1 -- 172.21.15.113:0/4192066736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff24c06caa0 msgr2=0x7ff24c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.697+0000 7ff2661dc700 1 --2- 172.21.15.113:0/4192066736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff24c06caa0 0x7ff24c06ef60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7ff2480041d0 tx=0x7ff248009040).stop 2024-04-12T05:43:52.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.697+0000 7ff2661dc700 1 -- 172.21.15.113:0/4192066736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2600fe250 msgr2=0x7ff26006b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.697+0000 7ff2661dc700 1 --2- 172.21.15.113:0/4192066736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2600fe250 0x7ff26006b5c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7ff2500099f0 tx=0x7ff250009cb0).stop 2024-04-12T05:43:52.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.697+0000 7ff2661dc700 1 -- 172.21.15.113:0/4192066736 shutdown_connections 2024-04-12T05:43:52.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.697+0000 7ff2661dc700 1 --2- 172.21.15.113:0/4192066736 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff24c06caa0 0x7ff24c06ef60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.697+0000 7ff2661dc700 1 --2- 172.21.15.113:0/4192066736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2600fd8f0 0x7ff26006b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.697+0000 7ff2661dc700 1 --2- 172.21.15.113:0/4192066736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2600fe250 0x7ff26006b5c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.697+0000 7ff2661dc700 1 -- 172.21.15.113:0/4192066736 >> 172.21.15.113:0/4192066736 conn(0x7ff2600f34b0 msgr2=0x7ff2600fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:52.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.697+0000 7ff2661dc700 1 -- 172.21.15.113:0/4192066736 shutdown_connections 2024-04-12T05:43:52.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.697+0000 7ff2661dc700 1 -- 172.21.15.113:0/4192066736 wait complete. 2024-04-12T05:43:52.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.795+0000 7f5b8b89f700 1 -- 172.21.15.113:0/4006606774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fe250 msgr2=0x7f5b840f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.795+0000 7f5b8b89f700 1 --2- 172.21.15.113:0/4006606774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fe250 0x7f5b840f7a30 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f5b740047f0 tx=0x7f5b74009cb0).stop 2024-04-12T05:43:52.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.795+0000 7f5b8b89f700 1 -- 172.21.15.113:0/4006606774 shutdown_connections 2024-04-12T05:43:52.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.795+0000 7f5b8b89f700 1 --2- 172.21.15.113:0/4006606774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fe250 0x7f5b840f7a30 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.795+0000 7f5b8b89f700 1 --2- 172.21.15.113:0/4006606774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fd8f0 0x7f5b840fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.795+0000 7f5b8b89f700 1 -- 172.21.15.113:0/4006606774 >> 172.21.15.113:0/4006606774 conn(0x7f5b840f34b0 msgr2=0x7f5b840f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.795+0000 7f5b8b89f700 1 -- 172.21.15.113:0/4006606774 shutdown_connections 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.795+0000 7f5b8b89f700 1 -- 172.21.15.113:0/4006606774 wait complete. 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.796+0000 7f5b8b89f700 1 Processor -- start 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.796+0000 7f5b8b89f700 1 -- start start 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.796+0000 7f5b8b89f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd8f0 0x7f5b8410e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.796+0000 7f5b8b89f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fe250 0x7f5b8410e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.796+0000 7f5b8b89f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8410ec30 con 0x7f5b840fd8f0 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.796+0000 7f5b8b89f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8410ed70 con 0x7f5b840fe250 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.796+0000 7f5b8963b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd8f0 0x7f5b8410e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.796+0000 7f5b8963b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd8f0 0x7f5b8410e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50584/0 (socket says 172.21.15.113:50584) 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.796+0000 7f5b8963b700 1 -- 172.21.15.113:0/2764643662 learned_addr learned my addr 172.21.15.113:0/2764643662 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.796+0000 7f5b88e3a700 1 --2- 172.21.15.113:0/2764643662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fe250 0x7f5b8410e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:52.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.797+0000 7f5b8963b700 1 -- 172.21.15.113:0/2764643662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fe250 msgr2=0x7f5b8410e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:52.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.797+0000 7f5b8963b700 1 --2- 172.21.15.113:0/2764643662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fe250 0x7f5b8410e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:52.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.797+0000 7f5b8963b700 1 -- 172.21.15.113:0/2764643662 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b74005040 con 0x7f5b840fd8f0 2024-04-12T05:43:52.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.797+0000 7f5b8963b700 1 --2- 172.21.15.113:0/2764643662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd8f0 0x7f5b8410e140 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7f5b8000e910 tx=0x7f5b8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:52.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.797+0000 7f5b7a7fc700 1 -- 172.21.15.113:0/2764643662 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b80007b40 con 0x7f5b840fd8f0 2024-04-12T05:43:52.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.797+0000 7f5b7a7fc700 1 -- 172.21.15.113:0/2764643662 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b8000f040 con 0x7f5b840fd8f0 2024-04-12T05:43:52.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.797+0000 7f5b8b89f700 1 -- 172.21.15.113:0/2764643662 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b841083a0 con 0x7f5b840fd8f0 2024-04-12T05:43:52.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.797+0000 7f5b7a7fc700 1 -- 172.21.15.113:0/2764643662 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b80014920 con 0x7f5b840fd8f0 2024-04-12T05:43:52.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.797+0000 7f5b8b89f700 1 -- 172.21.15.113:0/2764643662 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b84108960 con 0x7f5b840fd8f0 2024-04-12T05:43:52.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.798+0000 7f5b8b89f700 1 -- 172.21.15.113:0/2764643662 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8405f590 con 0x7f5b840fd8f0 2024-04-12T05:43:52.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.798+0000 7f5b7a7fc700 1 -- 172.21.15.113:0/2764643662 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5b80007ca0 con 0x7f5b840fd8f0 2024-04-12T05:43:52.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.799+0000 7f5b7a7fc700 1 --2- 172.21.15.113:0/2764643662 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b7006caa0 0x7f5b7006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:43:52.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.799+0000 7f5b7a7fc700 1 -- 172.21.15.113:0/2764643662 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5b8008b880 con 0x7f5b840fd8f0 2024-04-12T05:43:52.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.799+0000 7f5b88e3a700 1 --2- 172.21.15.113:0/2764643662 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b7006caa0 0x7f5b7006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:43:52.798 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.799+0000 7f5b88e3a700 1 --2- 172.21.15.113:0/2764643662 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b7006caa0 0x7f5b7006ef60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f5b84109a80 tx=0x7f5b74009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:43:52.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:52.803+0000 7f5b7a7fc700 1 -- 172.21.15.113:0/2764643662 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5b80059330 con 0x7f5b840fd8f0 2024-04-12T05:43:53.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.074+0000 7f5b8b89f700 1 -- 172.21.15.113:0/2764643662 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5b84047540 con 0x7f5b840fd8f0 2024-04-12T05:43:53.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.075+0000 7f5b7a7fc700 1 -- 172.21.15.113:0/2764643662 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f5b80055020 con 0x7f5b840fd8f0 2024-04-12T05:43:53.072 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:43:53.072 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:43:53.073 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:43:53.073 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:43:53.073 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:43:53.073 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:43:53.073 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:43:53.073 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:43:53.073 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:43:53.073 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:43:53.074 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:43:53.074 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:43:53.074 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:43:53.074 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:43:53.074 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:43:53.074 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:43:53.075 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:43:53.075 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:43:53.075 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:43:53.075 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:43:53.075 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:43:53.075 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:43:53.075 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:43:53.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.077+0000 7f5b8b89f700 1 -- 172.21.15.113:0/2764643662 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b7006caa0 msgr2=0x7f5b7006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:53.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.077+0000 7f5b8b89f700 1 --2- 172.21.15.113:0/2764643662 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b7006caa0 0x7f5b7006ef60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f5b84109a80 tx=0x7f5b74009cb0).stop 2024-04-12T05:43:53.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.077+0000 7f5b8b89f700 1 -- 172.21.15.113:0/2764643662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd8f0 msgr2=0x7f5b8410e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:43:53.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.077+0000 7f5b8b89f700 1 --2- 172.21.15.113:0/2764643662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd8f0 0x7f5b8410e140 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7f5b8000e910 tx=0x7f5b8000b040).stop 2024-04-12T05:43:53.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.077+0000 7f5b8b89f700 1 -- 172.21.15.113:0/2764643662 shutdown_connections 2024-04-12T05:43:53.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.077+0000 7f5b8b89f700 1 --2- 172.21.15.113:0/2764643662 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b7006caa0 0x7f5b7006ef60 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:53.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.077+0000 7f5b8b89f700 1 --2- 172.21.15.113:0/2764643662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b840fd8f0 0x7f5b8410e140 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:53.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.077+0000 7f5b8b89f700 1 --2- 172.21.15.113:0/2764643662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b840fe250 0x7f5b8410e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:43:53.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.077+0000 7f5b8b89f700 1 -- 172.21.15.113:0/2764643662 >> 172.21.15.113:0/2764643662 conn(0x7f5b840f34b0 msgr2=0x7f5b840fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:43:53.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.077+0000 7f5b8b89f700 1 -- 172.21.15.113:0/2764643662 shutdown_connections 2024-04-12T05:43:53.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:43:53.077+0000 7f5b8b89f700 1 -- 172.21.15.113:0/2764643662 wait complete. 2024-04-12T05:43:53.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:53 smithi113 ceph-mon[103371]: from='client.35714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:53.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:53 smithi113 ceph-mon[103371]: from='client.35718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:53.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:53 smithi113 ceph-mon[103371]: pgmap v952: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:53.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:53 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2661046927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:53.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:53 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1831672552' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:43:53.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:53 smithi192 ceph-mon[85304]: from='client.35714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:53.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:53 smithi192 ceph-mon[85304]: from='client.35718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:53.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:53 smithi192 ceph-mon[85304]: pgmap v952: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:53.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:53 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2661046927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:43:53.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:53 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1831672552' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:43:54.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:54 smithi113 ceph-mon[103371]: from='client.35730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:54.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:54 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2764643662' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:43:54.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:54 smithi192 ceph-mon[85304]: from='client.35730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:43:54.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:54 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2764643662' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:43:55.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:55 smithi113 ceph-mon[103371]: pgmap v953: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:55.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:55 smithi192 ceph-mon[85304]: pgmap v953: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:57.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:57 smithi113 ceph-mon[103371]: pgmap v954: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:57.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:57 smithi192 ceph-mon[85304]: pgmap v954: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:59.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:43:59 smithi113 ceph-mon[103371]: pgmap v955: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:43:59.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:43:59 smithi192 ceph-mon[85304]: pgmap v955: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:01.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:01 smithi113 ceph-mon[103371]: pgmap v956: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:01.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:01 smithi192 ceph-mon[85304]: pgmap v956: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:03.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:03 smithi113 ceph-mon[103371]: pgmap v957: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:03.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:03 smithi192 ceph-mon[85304]: pgmap v957: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:05.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:05 smithi113 ceph-mon[103371]: pgmap v958: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:05.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:05 smithi192 ceph-mon[85304]: pgmap v958: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:07.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:07 smithi113 ceph-mon[103371]: pgmap v959: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:07.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:07.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:07 smithi192 ceph-mon[85304]: pgmap v959: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:07.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:09.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:09 smithi113 ceph-mon[103371]: pgmap v960: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:09.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:09 smithi192 ceph-mon[85304]: pgmap v960: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:11.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:11 smithi113 ceph-mon[103371]: pgmap v961: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:11.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:11 smithi192 ceph-mon[85304]: pgmap v961: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:13.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:13 smithi113 ceph-mon[103371]: pgmap v962: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:13.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:13 smithi192 ceph-mon[85304]: pgmap v962: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:15.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:15 smithi113 ceph-mon[103371]: pgmap v963: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:15.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:15 smithi192 ceph-mon[85304]: pgmap v963: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:17.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:17 smithi113 ceph-mon[103371]: pgmap v964: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:17.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:17 smithi192 ceph-mon[85304]: pgmap v964: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:19.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:19 smithi113 ceph-mon[103371]: pgmap v965: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:19.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:19 smithi192 ceph-mon[85304]: pgmap v965: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:21.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:21 smithi113 ceph-mon[103371]: pgmap v966: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:21.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:21 smithi192 ceph-mon[85304]: pgmap v966: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:22.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:22.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:23.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.188+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1499523106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2ec0f9210 msgr2=0x7fc2ec1019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:23.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.188+0000 7fc2f1f26700 1 --2- 172.21.15.113:0/1499523106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2ec0f9210 0x7fc2ec1019b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7fc2dc0047f0 tx=0x7fc2dc009cb0).stop 2024-04-12T05:44:23.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.188+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1499523106 shutdown_connections 2024-04-12T05:44:23.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.188+0000 7fc2f1f26700 1 --2- 172.21.15.113:0/1499523106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2ec0f9210 0x7fc2ec1019b0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.188+0000 7fc2f1f26700 1 --2- 172.21.15.113:0/1499523106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2ec0f8940 0x7fc2ec0f8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.188+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1499523106 >> 172.21.15.113:0/1499523106 conn(0x7fc2ec070590 msgr2=0x7fc2ec0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:23.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.188+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1499523106 shutdown_connections 2024-04-12T05:44:23.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.188+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1499523106 wait complete. 2024-04-12T05:44:23.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.189+0000 7fc2f1f26700 1 Processor -- start 2024-04-12T05:44:23.187 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.189+0000 7fc2f1f26700 1 -- start start 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.189+0000 7fc2f1f26700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2ec0f8940 0x7fc2ec1103c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.189+0000 7fc2f1f26700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2ec1108b0 0x7fc2ec10a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.189+0000 7fc2f1f26700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec110e40 con 0x7fc2ec1108b0 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.189+0000 7fc2f1f26700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec110f80 con 0x7fc2ec0f8940 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.189+0000 7fc2eaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2ec1108b0 0x7fc2ec10a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.189+0000 7fc2eaffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2ec1108b0 0x7fc2ec10a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56252/0 (socket says 172.21.15.113:56252) 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.189+0000 7fc2eaffd700 1 -- 172.21.15.113:0/1629716288 learned_addr learned my addr 172.21.15.113:0/1629716288 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.190+0000 7fc2eb7fe700 1 --2- 172.21.15.113:0/1629716288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2ec0f8940 0x7fc2ec1103c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.190+0000 7fc2eaffd700 1 -- 172.21.15.113:0/1629716288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2ec0f8940 msgr2=0x7fc2ec1103c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.190+0000 7fc2eaffd700 1 --2- 172.21.15.113:0/1629716288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2ec0f8940 0x7fc2ec1103c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.190+0000 7fc2eaffd700 1 -- 172.21.15.113:0/1629716288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc005040 con 0x7fc2ec1108b0 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.190+0000 7fc2eaffd700 1 --2- 172.21.15.113:0/1629716288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2ec1108b0 0x7fc2ec10a560 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7fc2dc000c00 tx=0x7fc2dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.190+0000 7fc2e8ff9700 1 -- 172.21.15.113:0/1629716288 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2dc00e030 con 0x7fc2ec1108b0 2024-04-12T05:44:23.188 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.190+0000 7fc2e8ff9700 1 -- 172.21.15.113:0/1629716288 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2dc01e420 con 0x7fc2ec1108b0 2024-04-12T05:44:23.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.190+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1629716288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ec10aab0 con 0x7fc2ec1108b0 2024-04-12T05:44:23.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.190+0000 7fc2e8ff9700 1 -- 172.21.15.113:0/1629716288 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2dc019070 con 0x7fc2ec1108b0 2024-04-12T05:44:23.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.190+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1629716288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ec10aff0 con 0x7fc2ec1108b0 2024-04-12T05:44:23.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.191+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1629716288 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ec05f590 con 0x7fc2ec1108b0 2024-04-12T05:44:23.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.192+0000 7fc2e8ff9700 1 -- 172.21.15.113:0/1629716288 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc2dc01e970 con 0x7fc2ec1108b0 2024-04-12T05:44:23.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.192+0000 7fc2e8ff9700 1 --2- 172.21.15.113:0/1629716288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc2d406caa0 0x7fc2d406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:23.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.193+0000 7fc2e8ff9700 1 -- 172.21.15.113:0/1629716288 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc2dc08d090 con 0x7fc2ec1108b0 2024-04-12T05:44:23.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.193+0000 7fc2eb7fe700 1 --2- 172.21.15.113:0/1629716288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc2d406caa0 0x7fc2d406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:23.190 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.193+0000 7fc2eb7fe700 1 --2- 172.21.15.113:0/1629716288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc2d406caa0 0x7fc2d406ef60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fc2e0004da0 tx=0x7fc2e000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:23.193 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.197+0000 7fc2e8ff9700 1 -- 172.21.15.113:0/1629716288 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc2dc05ab40 con 0x7fc2ec1108b0 2024-04-12T05:44:23.399 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.403+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1629716288 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2ec10bb90 con 0x7fc2d406caa0 2024-04-12T05:44:23.400 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.404+0000 7fc2e8ff9700 1 -- 172.21.15.113:0/1629716288 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc2ec10bb90 con 0x7fc2d406caa0 2024-04-12T05:44:23.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.406+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1629716288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc2d406caa0 msgr2=0x7fc2d406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:23.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.406+0000 7fc2f1f26700 1 --2- 172.21.15.113:0/1629716288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc2d406caa0 0x7fc2d406ef60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fc2e0004da0 tx=0x7fc2e000b040).stop 2024-04-12T05:44:23.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.406+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1629716288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2ec1108b0 msgr2=0x7fc2ec10a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:23.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.406+0000 7fc2f1f26700 1 --2- 172.21.15.113:0/1629716288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2ec1108b0 0x7fc2ec10a560 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7fc2dc000c00 tx=0x7fc2dc00b040).stop 2024-04-12T05:44:23.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.406+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1629716288 shutdown_connections 2024-04-12T05:44:23.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.406+0000 7fc2f1f26700 1 --2- 172.21.15.113:0/1629716288 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc2d406caa0 0x7fc2d406ef60 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.406+0000 7fc2f1f26700 1 --2- 172.21.15.113:0/1629716288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2ec0f8940 0x7fc2ec1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.406+0000 7fc2f1f26700 1 --2- 172.21.15.113:0/1629716288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2ec1108b0 0x7fc2ec10a560 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.406+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1629716288 >> 172.21.15.113:0/1629716288 conn(0x7fc2ec070590 msgr2=0x7fc2ec0f6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:23.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.406+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1629716288 shutdown_connections 2024-04-12T05:44:23.403 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.406+0000 7fc2f1f26700 1 -- 172.21.15.113:0/1629716288 wait complete. 2024-04-12T05:44:23.412 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:44:23.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:23 smithi113 ceph-mon[103371]: pgmap v967: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:23.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:23 smithi192 ceph-mon[85304]: pgmap v967: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:23.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.505+0000 7f9f590bf700 1 -- 172.21.15.113:0/2013276129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f540fc7f0 msgr2=0x7f9f540fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:23.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.505+0000 7f9f590bf700 1 --2- 172.21.15.113:0/2013276129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f540fc7f0 0x7f9f540fcbd0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7f9f480047f0 tx=0x7f9f48009cb0).stop 2024-04-12T05:44:23.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.505+0000 7f9f590bf700 1 -- 172.21.15.113:0/2013276129 shutdown_connections 2024-04-12T05:44:23.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.505+0000 7f9f590bf700 1 --2- 172.21.15.113:0/2013276129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f540fd150 0x7f9f540f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.505+0000 7f9f590bf700 1 --2- 172.21.15.113:0/2013276129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f540fc7f0 0x7f9f540fcbd0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.505+0000 7f9f590bf700 1 -- 172.21.15.113:0/2013276129 >> 172.21.15.113:0/2013276129 conn(0x7f9f54070590 msgr2=0x7f9f540f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:23.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.505+0000 7f9f590bf700 1 -- 172.21.15.113:0/2013276129 shutdown_connections 2024-04-12T05:44:23.502 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.505+0000 7f9f590bf700 1 -- 172.21.15.113:0/2013276129 wait complete. 2024-04-12T05:44:23.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.505+0000 7f9f590bf700 1 Processor -- start 2024-04-12T05:44:23.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f590bf700 1 -- start start 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f590bf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f540fc7f0 0x7f9f5410e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f590bf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f540fd150 0x7f9f5410e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f590bf700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5410ec10 con 0x7f9f540fc7f0 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f590bf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5410ed50 con 0x7f9f540fd150 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f52d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f540fc7f0 0x7f9f5410e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f52d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f540fc7f0 0x7f9f5410e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56278/0 (socket says 172.21.15.113:56278) 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f52d9d700 1 -- 172.21.15.113:0/1148380669 learned_addr learned my addr 172.21.15.113:0/1148380669 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f5259c700 1 --2- 172.21.15.113:0/1148380669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f540fd150 0x7f9f5410e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f52d9d700 1 -- 172.21.15.113:0/1148380669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f540fd150 msgr2=0x7f9f5410e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f52d9d700 1 --2- 172.21.15.113:0/1148380669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f540fd150 0x7f9f5410e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f52d9d700 1 -- 172.21.15.113:0/1148380669 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f48005040 con 0x7f9f540fc7f0 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f52d9d700 1 --2- 172.21.15.113:0/1148380669 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f540fc7f0 0x7f9f5410e120 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7f9f48009380 tx=0x7f9f48009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.506+0000 7f9f3bfff700 1 -- 172.21.15.113:0/1148380669 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f48018070 con 0x7f9f540fc7f0 2024-04-12T05:44:23.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.507+0000 7f9f590bf700 1 -- 172.21.15.113:0/1148380669 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f541083c0 con 0x7f9f540fc7f0 2024-04-12T05:44:23.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.507+0000 7f9f3bfff700 1 -- 172.21.15.113:0/1148380669 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f48007e90 con 0x7f9f540fc7f0 2024-04-12T05:44:23.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.507+0000 7f9f3bfff700 1 -- 172.21.15.113:0/1148380669 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f480123c0 con 0x7f9f540fc7f0 2024-04-12T05:44:23.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.507+0000 7f9f590bf700 1 -- 172.21.15.113:0/1148380669 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f54108920 con 0x7f9f540fc7f0 2024-04-12T05:44:23.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.508+0000 7f9f590bf700 1 -- 172.21.15.113:0/1148380669 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5405f590 con 0x7f9f540fc7f0 2024-04-12T05:44:23.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.508+0000 7f9f3bfff700 1 -- 172.21.15.113:0/1148380669 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9f4801d420 con 0x7f9f540fc7f0 2024-04-12T05:44:23.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.508+0000 7f9f3bfff700 1 --2- 172.21.15.113:0/1148380669 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3c06caa0 0x7f9f3c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:23.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.509+0000 7f9f3bfff700 1 -- 172.21.15.113:0/1148380669 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9f4808c9c0 con 0x7f9f540fc7f0 2024-04-12T05:44:23.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.509+0000 7f9f5259c700 1 --2- 172.21.15.113:0/1148380669 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3c06caa0 0x7f9f3c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:23.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.509+0000 7f9f5259c700 1 --2- 172.21.15.113:0/1148380669 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3c06caa0 0x7f9f3c06ef60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f9f54109aa0 tx=0x7f9f4400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:23.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.513+0000 7f9f3bfff700 1 -- 172.21.15.113:0/1148380669 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9f480569e0 con 0x7f9f540fc7f0 2024-04-12T05:44:23.714 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.717+0000 7f9f590bf700 1 -- 172.21.15.113:0/1148380669 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f54109850 con 0x7f9f3c06caa0 2024-04-12T05:44:23.715 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.719+0000 7f9f3bfff700 1 -- 172.21.15.113:0/1148380669 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9f54109850 con 0x7f9f3c06caa0 2024-04-12T05:44:23.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.721+0000 7f9f590bf700 1 -- 172.21.15.113:0/1148380669 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3c06caa0 msgr2=0x7f9f3c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:23.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.721+0000 7f9f590bf700 1 --2- 172.21.15.113:0/1148380669 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3c06caa0 0x7f9f3c06ef60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f9f54109aa0 tx=0x7f9f4400b040).stop 2024-04-12T05:44:23.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.721+0000 7f9f590bf700 1 -- 172.21.15.113:0/1148380669 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f540fc7f0 msgr2=0x7f9f5410e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:23.717 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.721+0000 7f9f590bf700 1 --2- 172.21.15.113:0/1148380669 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f540fc7f0 0x7f9f5410e120 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7f9f48009380 tx=0x7f9f48009cb0).stop 2024-04-12T05:44:23.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.721+0000 7f9f590bf700 1 -- 172.21.15.113:0/1148380669 shutdown_connections 2024-04-12T05:44:23.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.721+0000 7f9f590bf700 1 --2- 172.21.15.113:0/1148380669 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9f3c06caa0 0x7f9f3c06ef60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.721+0000 7f9f590bf700 1 --2- 172.21.15.113:0/1148380669 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9f540fc7f0 0x7f9f5410e120 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.721+0000 7f9f590bf700 1 --2- 172.21.15.113:0/1148380669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f540fd150 0x7f9f5410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.721+0000 7f9f590bf700 1 -- 172.21.15.113:0/1148380669 >> 172.21.15.113:0/1148380669 conn(0x7f9f54070590 msgr2=0x7f9f54060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:23.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.721+0000 7f9f590bf700 1 -- 172.21.15.113:0/1148380669 shutdown_connections 2024-04-12T05:44:23.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.721+0000 7f9f590bf700 1 -- 172.21.15.113:0/1148380669 wait complete. 2024-04-12T05:44:23.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.819+0000 7f07cc9dd700 1 -- 172.21.15.113:0/3865129273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07c80fc7f0 msgr2=0x7f07c80fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:23.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.819+0000 7f07cc9dd700 1 --2- 172.21.15.113:0/3865129273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07c80fc7f0 0x7f07c80fcbd0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f07bc0047f0 tx=0x7f07bc009cb0).stop 2024-04-12T05:44:23.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.819+0000 7f07cc9dd700 1 -- 172.21.15.113:0/3865129273 shutdown_connections 2024-04-12T05:44:23.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.819+0000 7f07cc9dd700 1 --2- 172.21.15.113:0/3865129273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07c80fd150 0x7f07c80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.819+0000 7f07cc9dd700 1 --2- 172.21.15.113:0/3865129273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07c80fc7f0 0x7f07c80fcbd0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.819+0000 7f07cc9dd700 1 -- 172.21.15.113:0/3865129273 >> 172.21.15.113:0/3865129273 conn(0x7f07c8070590 msgr2=0x7f07c80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:23.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.819+0000 7f07cc9dd700 1 -- 172.21.15.113:0/3865129273 shutdown_connections 2024-04-12T05:44:23.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.819+0000 7f07cc9dd700 1 -- 172.21.15.113:0/3865129273 wait complete. 2024-04-12T05:44:23.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07cc9dd700 1 Processor -- start 2024-04-12T05:44:23.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07cc9dd700 1 -- start start 2024-04-12T05:44:23.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07cc9dd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07c80fc7f0 0x7f07c810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:23.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07cc9dd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07c80fd150 0x7f07c810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:23.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07cc9dd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c810eb20 con 0x7f07c80fd150 2024-04-12T05:44:23.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07cc9dd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c810ec60 con 0x7f07c80fc7f0 2024-04-12T05:44:23.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07c5d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07c80fd150 0x7f07c810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:23.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07c5d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07c80fd150 0x7f07c810e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56296/0 (socket says 172.21.15.113:56296) 2024-04-12T05:44:23.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07c5d9b700 1 -- 172.21.15.113:0/1349045477 learned_addr learned my addr 172.21.15.113:0/1349045477 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:23.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07c659c700 1 --2- 172.21.15.113:0/1349045477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07c80fc7f0 0x7f07c810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:23.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07c5d9b700 1 -- 172.21.15.113:0/1349045477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07c80fc7f0 msgr2=0x7f07c810e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:23.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07c5d9b700 1 --2- 172.21.15.113:0/1349045477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07c80fc7f0 0x7f07c810e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:23.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07c5d9b700 1 -- 172.21.15.113:0/1349045477 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc005040 con 0x7f07c80fd150 2024-04-12T05:44:23.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.820+0000 7f07c5d9b700 1 --2- 172.21.15.113:0/1349045477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07c80fd150 0x7f07c810e520 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7f07b8002820 tx=0x7f07b8009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:23.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.821+0000 7f07b77fe700 1 -- 172.21.15.113:0/1349045477 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07b8007b40 con 0x7f07c80fd150 2024-04-12T05:44:23.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.821+0000 7f07cc9dd700 1 -- 172.21.15.113:0/1349045477 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c81082f0 con 0x7f07c80fd150 2024-04-12T05:44:23.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.821+0000 7f07b77fe700 1 -- 172.21.15.113:0/1349045477 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07b800f040 con 0x7f07c80fd150 2024-04-12T05:44:23.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.821+0000 7f07b77fe700 1 -- 172.21.15.113:0/1349045477 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07b80148b0 con 0x7f07c80fd150 2024-04-12T05:44:23.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.821+0000 7f07cc9dd700 1 -- 172.21.15.113:0/1349045477 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c81088b0 con 0x7f07c80fd150 2024-04-12T05:44:23.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.822+0000 7f07cc9dd700 1 -- 172.21.15.113:0/1349045477 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c805f590 con 0x7f07c80fd150 2024-04-12T05:44:23.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.822+0000 7f07b77fe700 1 -- 172.21.15.113:0/1349045477 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f07b8007ca0 con 0x7f07c80fd150 2024-04-12T05:44:23.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.822+0000 7f07b77fe700 1 --2- 172.21.15.113:0/1349045477 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07b006caa0 0x7f07b006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:23.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.822+0000 7f07b77fe700 1 -- 172.21.15.113:0/1349045477 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f07b808b8f0 con 0x7f07c80fd150 2024-04-12T05:44:23.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.822+0000 7f07c659c700 1 --2- 172.21.15.113:0/1349045477 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07b006caa0 0x7f07b006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:23.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.823+0000 7f07c659c700 1 --2- 172.21.15.113:0/1349045477 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07b006caa0 0x7f07b006ef60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f07bc004f10 tx=0x7f07bc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:23.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:23.827+0000 7f07b77fe700 1 -- 172.21.15.113:0/1349045477 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f07b80593a0 con 0x7f07c80fd150 2024-04-12T05:44:24.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.025+0000 7f07cc9dd700 1 -- 172.21.15.113:0/1349045477 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f07c8108f50 con 0x7f07b006caa0 2024-04-12T05:44:24.031 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.035+0000 7f07b77fe700 1 -- 172.21.15.113:0/1349045477 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f07c8108f50 con 0x7f07b006caa0 2024-04-12T05:44:24.031 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:44:24.031 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (32m) 9m ago 43m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:44:24.031 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (29m) 9m ago 43m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (29m) 9m ago 42m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (31m) 9m ago 43m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (25m) 9m ago 38m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (25m) 9m ago 38m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (24m) 9m ago 38m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (9m) 9m ago 37m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (33m) 9m ago 44m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (33m) 9m ago 42m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (30m) 9m ago 44m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (30m) 9m ago 42m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (32m) 9m ago 43m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (32m) 9m ago 42m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (29m) 9m ago 42m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:44:24.032 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (28m) 9m ago 41m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:44:24.033 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (28m) 9m ago 40m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:44:24.033 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (27m) 9m ago 40m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:44:24.033 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (26m) 9m ago 40m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:44:24.033 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (25m) 9m ago 39m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:44:24.033 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (32m) 9m ago 42m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:44:24.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.037+0000 7f07cc9dd700 1 -- 172.21.15.113:0/1349045477 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07b006caa0 msgr2=0x7f07b006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.037+0000 7f07cc9dd700 1 --2- 172.21.15.113:0/1349045477 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07b006caa0 0x7f07b006ef60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f07bc004f10 tx=0x7f07bc00b040).stop 2024-04-12T05:44:24.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.037+0000 7f07cc9dd700 1 -- 172.21.15.113:0/1349045477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07c80fd150 msgr2=0x7f07c810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.037+0000 7f07cc9dd700 1 --2- 172.21.15.113:0/1349045477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07c80fd150 0x7f07c810e520 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7f07b8002820 tx=0x7f07b8009f20).stop 2024-04-12T05:44:24.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.037+0000 7f07cc9dd700 1 -- 172.21.15.113:0/1349045477 shutdown_connections 2024-04-12T05:44:24.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.037+0000 7f07cc9dd700 1 --2- 172.21.15.113:0/1349045477 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07b006caa0 0x7f07b006ef60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.037+0000 7f07cc9dd700 1 --2- 172.21.15.113:0/1349045477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07c80fc7f0 0x7f07c810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.037+0000 7f07cc9dd700 1 --2- 172.21.15.113:0/1349045477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07c80fd150 0x7f07c810e520 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.037+0000 7f07cc9dd700 1 -- 172.21.15.113:0/1349045477 >> 172.21.15.113:0/1349045477 conn(0x7f07c8070590 msgr2=0x7f07c8060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:24.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.037+0000 7f07cc9dd700 1 -- 172.21.15.113:0/1349045477 shutdown_connections 2024-04-12T05:44:24.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.037+0000 7f07cc9dd700 1 -- 172.21.15.113:0/1349045477 wait complete. 2024-04-12T05:44:24.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.135+0000 7f3d44c0b700 1 -- 172.21.15.113:0/730877474 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d400fc7f0 msgr2=0x7f3d400fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.135+0000 7f3d44c0b700 1 --2- 172.21.15.113:0/730877474 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d400fc7f0 0x7f3d400fcbd0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f3d38009b60 tx=0x7f3d38004d10).stop 2024-04-12T05:44:24.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.135+0000 7f3d44c0b700 1 -- 172.21.15.113:0/730877474 shutdown_connections 2024-04-12T05:44:24.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.135+0000 7f3d44c0b700 1 --2- 172.21.15.113:0/730877474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d400fd150 0x7f3d400f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.135+0000 7f3d44c0b700 1 --2- 172.21.15.113:0/730877474 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d400fc7f0 0x7f3d400fcbd0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.135+0000 7f3d44c0b700 1 -- 172.21.15.113:0/730877474 >> 172.21.15.113:0/730877474 conn(0x7f3d40070590 msgr2=0x7f3d400f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:24.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.135+0000 7f3d44c0b700 1 -- 172.21.15.113:0/730877474 shutdown_connections 2024-04-12T05:44:24.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.135+0000 7f3d44c0b700 1 -- 172.21.15.113:0/730877474 wait complete. 2024-04-12T05:44:24.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.135+0000 7f3d44c0b700 1 Processor -- start 2024-04-12T05:44:24.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d44c0b700 1 -- start start 2024-04-12T05:44:24.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d44c0b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d400fc7f0 0x7f3d4010dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:24.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d44c0b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d400fd150 0x7f3d4010e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:24.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d44c0b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4010eab0 con 0x7f3d400fc7f0 2024-04-12T05:44:24.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d44c0b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4010ebf0 con 0x7f3d400fd150 2024-04-12T05:44:24.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d3e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d400fc7f0 0x7f3d4010dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:24.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d3e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d400fc7f0 0x7f3d4010dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56314/0 (socket says 172.21.15.113:56314) 2024-04-12T05:44:24.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d3e59c700 1 -- 172.21.15.113:0/867568010 learned_addr learned my addr 172.21.15.113:0/867568010 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d3dd9b700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d400fd150 0x7f3d4010e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d3e59c700 1 -- 172.21.15.113:0/867568010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d400fd150 msgr2=0x7f3d4010e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d3e59c700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d400fd150 0x7f3d4010e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d3e59c700 1 -- 172.21.15.113:0/867568010 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d380097c0 con 0x7f3d400fc7f0 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d3dd9b700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d400fd150 0x7f3d4010e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d3e59c700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d400fc7f0 0x7f3d4010dfc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f3d38004e90 tx=0x7f3d38004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d377fe700 1 -- 172.21.15.113:0/867568010 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d3800e070 con 0x7f3d400fc7f0 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d377fe700 1 -- 172.21.15.113:0/867568010 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d3801d8d0 con 0x7f3d400fc7f0 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d44c0b700 1 -- 172.21.15.113:0/867568010 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d40108220 con 0x7f3d400fc7f0 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.136+0000 7f3d377fe700 1 -- 172.21.15.113:0/867568010 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d38018070 con 0x7f3d400fc7f0 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.137+0000 7f3d44c0b700 1 -- 172.21.15.113:0/867568010 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d40108780 con 0x7f3d400fc7f0 2024-04-12T05:44:24.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.138+0000 7f3d44c0b700 1 -- 172.21.15.113:0/867568010 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4005f590 con 0x7f3d400fc7f0 2024-04-12T05:44:24.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.138+0000 7f3d377fe700 1 -- 172.21.15.113:0/867568010 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3d3801da40 con 0x7f3d400fc7f0 2024-04-12T05:44:24.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.138+0000 7f3d377fe700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d2c06cab0 0x7f3d2c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:24.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.138+0000 7f3d377fe700 1 -- 172.21.15.113:0/867568010 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3d3808d360 con 0x7f3d400fc7f0 2024-04-12T05:44:24.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.138+0000 7f3d3dd9b700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d2c06cab0 0x7f3d2c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:24.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.139+0000 7f3d3dd9b700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d2c06cab0 0x7f3d2c06ef70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f3d40109900 tx=0x7f3d28009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:24.139 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.143+0000 7f3d377fe700 1 -- 172.21.15.113:0/867568010 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3d3805ae10 con 0x7f3d400fc7f0 2024-04-12T05:44:24.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.408+0000 7f3d44c0b700 1 -- 172.21.15.113:0/867568010 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3d40047540 con 0x7f3d400fc7f0 2024-04-12T05:44:24.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.408+0000 7f3d377fe700 1 -- 172.21.15.113:0/867568010 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f3d3805a9a0 con 0x7f3d400fc7f0 2024-04-12T05:44:24.405 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:44:24.405 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:44:24.405 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:44:24.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:44:24.405 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:44:24.405 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:44:24.405 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:44:24.405 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:44:24.406 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:44:24.406 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:44:24.406 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:44:24.406 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:44:24.406 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:44:24.406 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:44:24.406 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:44:24.406 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:44:24.406 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:44:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.410+0000 7f3d44c0b700 1 -- 172.21.15.113:0/867568010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d2c06cab0 msgr2=0x7f3d2c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.410+0000 7f3d44c0b700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d2c06cab0 0x7f3d2c06ef70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f3d40109900 tx=0x7f3d28009040).stop 2024-04-12T05:44:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.410+0000 7f3d44c0b700 1 -- 172.21.15.113:0/867568010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d400fc7f0 msgr2=0x7f3d4010dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.410+0000 7f3d44c0b700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d400fc7f0 0x7f3d4010dfc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f3d38004e90 tx=0x7f3d38004d10).stop 2024-04-12T05:44:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.410+0000 7f3d44c0b700 1 -- 172.21.15.113:0/867568010 shutdown_connections 2024-04-12T05:44:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.410+0000 7f3d44c0b700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3d2c06cab0 0x7f3d2c06ef70 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.410+0000 7f3d44c0b700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d400fc7f0 0x7f3d4010dfc0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.410+0000 7f3d44c0b700 1 --2- 172.21.15.113:0/867568010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d400fd150 0x7f3d4010e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.410+0000 7f3d44c0b700 1 -- 172.21.15.113:0/867568010 >> 172.21.15.113:0/867568010 conn(0x7f3d40070590 msgr2=0x7f3d40060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.410+0000 7f3d44c0b700 1 -- 172.21.15.113:0/867568010 shutdown_connections 2024-04-12T05:44:24.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.410+0000 7f3d44c0b700 1 -- 172.21.15.113:0/867568010 wait complete. 2024-04-12T05:44:24.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.508+0000 7fe159cce700 1 -- 172.21.15.113:0/2858445129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1540fd150 msgr2=0x7fe1540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.508+0000 7fe159cce700 1 --2- 172.21.15.113:0/2858445129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1540fd150 0x7fe1540f6930 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7fe1440047f0 tx=0x7fe144009cb0).stop 2024-04-12T05:44:24.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.508+0000 7fe159cce700 1 -- 172.21.15.113:0/2858445129 shutdown_connections 2024-04-12T05:44:24.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.508+0000 7fe159cce700 1 --2- 172.21.15.113:0/2858445129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1540fd150 0x7fe1540f6930 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.508+0000 7fe159cce700 1 --2- 172.21.15.113:0/2858445129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1540fc7f0 0x7fe1540fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.508+0000 7fe159cce700 1 -- 172.21.15.113:0/2858445129 >> 172.21.15.113:0/2858445129 conn(0x7fe154070590 msgr2=0x7fe1540f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:24.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.508+0000 7fe159cce700 1 -- 172.21.15.113:0/2858445129 shutdown_connections 2024-04-12T05:44:24.505 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.508+0000 7fe159cce700 1 -- 172.21.15.113:0/2858445129 wait complete. 2024-04-12T05:44:24.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.508+0000 7fe159cce700 1 Processor -- start 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe159cce700 1 -- start start 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe159cce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1540fc7f0 0x7fe1541090a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe159cce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1540fd150 0x7fe154109590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe159cce700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe154109a80 con 0x7fe1540fc7f0 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe159cce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe154109bc0 con 0x7fe1540fd150 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe1537fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1540fc7f0 0x7fe1541090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe1537fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1540fc7f0 0x7fe1541090a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56330/0 (socket says 172.21.15.113:56330) 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe1537fe700 1 -- 172.21.15.113:0/731837698 learned_addr learned my addr 172.21.15.113:0/731837698 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe152ffd700 1 --2- 172.21.15.113:0/731837698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1540fd150 0x7fe154109590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe1537fe700 1 -- 172.21.15.113:0/731837698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1540fd150 msgr2=0x7fe154109590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe1537fe700 1 --2- 172.21.15.113:0/731837698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1540fd150 0x7fe154109590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe1537fe700 1 -- 172.21.15.113:0/731837698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe144005040 con 0x7fe1540fc7f0 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe1537fe700 1 --2- 172.21.15.113:0/731837698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1540fc7f0 0x7fe1541090a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7fe148002770 tx=0x7fe148002a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:24.507 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe150ff9700 1 -- 172.21.15.113:0/731837698 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe148007b40 con 0x7fe1540fc7f0 2024-04-12T05:44:24.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe150ff9700 1 -- 172.21.15.113:0/731837698 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe14800f040 con 0x7fe1540fc7f0 2024-04-12T05:44:24.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe150ff9700 1 -- 172.21.15.113:0/731837698 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe148014820 con 0x7fe1540fc7f0 2024-04-12T05:44:24.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.509+0000 7fe159cce700 1 -- 172.21.15.113:0/731837698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe154109e70 con 0x7fe1540fc7f0 2024-04-12T05:44:24.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.510+0000 7fe159cce700 1 -- 172.21.15.113:0/731837698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe154112410 con 0x7fe1540fc7f0 2024-04-12T05:44:24.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.511+0000 7fe159cce700 1 -- 172.21.15.113:0/731837698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15405f590 con 0x7fe1540fc7f0 2024-04-12T05:44:24.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.511+0000 7fe150ff9700 1 -- 172.21.15.113:0/731837698 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe148007ca0 con 0x7fe1540fc7f0 2024-04-12T05:44:24.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.511+0000 7fe150ff9700 1 --2- 172.21.15.113:0/731837698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13c06caa0 0x7fe13c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:24.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.511+0000 7fe150ff9700 1 -- 172.21.15.113:0/731837698 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe14808ba40 con 0x7fe1540fc7f0 2024-04-12T05:44:24.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.511+0000 7fe152ffd700 1 --2- 172.21.15.113:0/731837698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13c06caa0 0x7fe13c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:24.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.512+0000 7fe152ffd700 1 --2- 172.21.15.113:0/731837698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13c06caa0 0x7fe13c06ef60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fe154060f00 tx=0x7fe14400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:24.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.516+0000 7fe150ff9700 1 -- 172.21.15.113:0/731837698 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe148055a60 con 0x7fe1540fc7f0 2024-04-12T05:44:24.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.747+0000 7fe159cce700 1 -- 172.21.15.113:0/731837698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe154047540 con 0x7fe1540fc7f0 2024-04-12T05:44:24.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.747+0000 7fe150ff9700 1 -- 172.21.15.113:0/731837698 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fe148059080 con 0x7fe1540fc7f0 2024-04-12T05:44:24.744 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:44:24.744 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:44:24.744 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:44:24.744 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:44:24.744 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:44:24.744 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:44:24.745 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:44:24.746 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:44:24.747 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:44:24.747 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:44:24.747 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:24.747 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:24.747 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:24.747 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.750+0000 7fe159cce700 1 -- 172.21.15.113:0/731837698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13c06caa0 msgr2=0x7fe13c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.750+0000 7fe159cce700 1 --2- 172.21.15.113:0/731837698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13c06caa0 0x7fe13c06ef60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fe154060f00 tx=0x7fe14400b040).stop 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.750+0000 7fe159cce700 1 -- 172.21.15.113:0/731837698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1540fc7f0 msgr2=0x7fe1541090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.750+0000 7fe159cce700 1 --2- 172.21.15.113:0/731837698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1540fc7f0 0x7fe1541090a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7fe148002770 tx=0x7fe148002a80).stop 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.750+0000 7fe159cce700 1 -- 172.21.15.113:0/731837698 shutdown_connections 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.750+0000 7fe159cce700 1 --2- 172.21.15.113:0/731837698 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe13c06caa0 0x7fe13c06ef60 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.750+0000 7fe159cce700 1 --2- 172.21.15.113:0/731837698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1540fc7f0 0x7fe1541090a0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.750+0000 7fe159cce700 1 --2- 172.21.15.113:0/731837698 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1540fd150 0x7fe154109590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.750+0000 7fe159cce700 1 -- 172.21.15.113:0/731837698 >> 172.21.15.113:0/731837698 conn(0x7fe154070590 msgr2=0x7fe1540fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.750+0000 7fe159cce700 1 -- 172.21.15.113:0/731837698 shutdown_connections 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.750+0000 7fe159cce700 1 -- 172.21.15.113:0/731837698 wait complete. 2024-04-12T05:44:24.748 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:44:24.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.850+0000 7f0b4207b700 1 -- 172.21.15.113:0/513149213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b3c0fe250 msgr2=0x7f0b3c0f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.850+0000 7f0b4207b700 1 --2- 172.21.15.113:0/513149213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b3c0fe250 0x7f0b3c0f7a30 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f0b2c0047f0 tx=0x7f0b2c009cb0).stop 2024-04-12T05:44:24.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.850+0000 7f0b4207b700 1 -- 172.21.15.113:0/513149213 shutdown_connections 2024-04-12T05:44:24.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.850+0000 7f0b4207b700 1 --2- 172.21.15.113:0/513149213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b3c0fe250 0x7f0b3c0f7a30 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.850+0000 7f0b4207b700 1 --2- 172.21.15.113:0/513149213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b3c0fd8f0 0x7f0b3c0fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.850+0000 7f0b4207b700 1 -- 172.21.15.113:0/513149213 >> 172.21.15.113:0/513149213 conn(0x7f0b3c0f34b0 msgr2=0x7f0b3c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:24.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.850+0000 7f0b4207b700 1 -- 172.21.15.113:0/513149213 shutdown_connections 2024-04-12T05:44:24.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.850+0000 7f0b4207b700 1 -- 172.21.15.113:0/513149213 wait complete. 2024-04-12T05:44:24.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b4207b700 1 Processor -- start 2024-04-12T05:44:24.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b4207b700 1 -- start start 2024-04-12T05:44:24.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b4207b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b3c0fd8f0 0x7f0b3c10e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:24.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b4207b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b3c0fe250 0x7f0b3c10e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:24.853 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b4207b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c10eb70 con 0x7f0b3c0fd8f0 2024-04-12T05:44:24.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b4207b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c10ecb0 con 0x7f0b3c0fe250 2024-04-12T05:44:24.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b3b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b3c0fd8f0 0x7f0b3c10e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:24.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b3b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b3c0fd8f0 0x7f0b3c10e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56350/0 (socket says 172.21.15.113:56350) 2024-04-12T05:44:24.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b3b7fe700 1 -- 172.21.15.113:0/3436239156 learned_addr learned my addr 172.21.15.113:0/3436239156 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:24.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b3affd700 1 --2- 172.21.15.113:0/3436239156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b3c0fe250 0x7f0b3c10e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:24.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b3b7fe700 1 -- 172.21.15.113:0/3436239156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b3c0fe250 msgr2=0x7f0b3c10e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:24.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.851+0000 7f0b3b7fe700 1 --2- 172.21.15.113:0/3436239156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b3c0fe250 0x7f0b3c10e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:24.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.852+0000 7f0b3b7fe700 1 -- 172.21.15.113:0/3436239156 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c005040 con 0x7f0b3c0fd8f0 2024-04-12T05:44:24.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.852+0000 7f0b3b7fe700 1 --2- 172.21.15.113:0/3436239156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b3c0fd8f0 0x7f0b3c10e080 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f0b3000c790 tx=0x7f0b30004330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:24.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.852+0000 7f0b38ff9700 1 -- 172.21.15.113:0/3436239156 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b300106f0 con 0x7f0b3c0fd8f0 2024-04-12T05:44:24.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.852+0000 7f0b4207b700 1 -- 172.21.15.113:0/3436239156 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3c108340 con 0x7f0b3c0fd8f0 2024-04-12T05:44:24.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.852+0000 7f0b38ff9700 1 -- 172.21.15.113:0/3436239156 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b30010d00 con 0x7f0b3c0fd8f0 2024-04-12T05:44:24.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.852+0000 7f0b38ff9700 1 -- 172.21.15.113:0/3436239156 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b30012da0 con 0x7f0b3c0fd8f0 2024-04-12T05:44:24.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.852+0000 7f0b4207b700 1 -- 172.21.15.113:0/3436239156 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c108880 con 0x7f0b3c0fd8f0 2024-04-12T05:44:24.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.853+0000 7f0b4207b700 1 -- 172.21.15.113:0/3436239156 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c05f590 con 0x7f0b3c0fd8f0 2024-04-12T05:44:24.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.853+0000 7f0b38ff9700 1 -- 172.21.15.113:0/3436239156 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0b30012400 con 0x7f0b3c0fd8f0 2024-04-12T05:44:24.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.854+0000 7f0b38ff9700 1 --2- 172.21.15.113:0/3436239156 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b2406caa0 0x7f0b2406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:24.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.854+0000 7f0b38ff9700 1 -- 172.21.15.113:0/3436239156 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0b3008b7b0 con 0x7f0b3c0fd8f0 2024-04-12T05:44:24.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.854+0000 7f0b3affd700 1 --2- 172.21.15.113:0/3436239156 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b2406caa0 0x7f0b2406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:24.857 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.854+0000 7f0b3affd700 1 --2- 172.21.15.113:0/3436239156 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b2406caa0 0x7f0b2406ef60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f0b2c004f10 tx=0x7f0b2c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:24.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:24.858+0000 7f0b38ff9700 1 -- 172.21.15.113:0/3436239156 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0b300591a0 con 0x7f0b3c0fd8f0 2024-04-12T05:44:25.061 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.065+0000 7f0b4207b700 1 -- 172.21.15.113:0/3436239156 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b3c108b50 con 0x7f0b2406caa0 2024-04-12T05:44:25.062 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.066+0000 7f0b38ff9700 1 -- 172.21.15.113:0/3436239156 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0b3c108b50 con 0x7f0b2406caa0 2024-04-12T05:44:25.062 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:44:25.063 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:44:25.064 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.068+0000 7f0b4207b700 1 -- 172.21.15.113:0/3436239156 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b2406caa0 msgr2=0x7f0b2406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:25.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.068+0000 7f0b4207b700 1 --2- 172.21.15.113:0/3436239156 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b2406caa0 0x7f0b2406ef60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f0b2c004f10 tx=0x7f0b2c00b040).stop 2024-04-12T05:44:25.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.068+0000 7f0b4207b700 1 -- 172.21.15.113:0/3436239156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b3c0fd8f0 msgr2=0x7f0b3c10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:25.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.068+0000 7f0b4207b700 1 --2- 172.21.15.113:0/3436239156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b3c0fd8f0 0x7f0b3c10e080 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f0b3000c790 tx=0x7f0b30004330).stop 2024-04-12T05:44:25.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.068+0000 7f0b4207b700 1 -- 172.21.15.113:0/3436239156 shutdown_connections 2024-04-12T05:44:25.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.068+0000 7f0b4207b700 1 --2- 172.21.15.113:0/3436239156 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b2406caa0 0x7f0b2406ef60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:25.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.068+0000 7f0b4207b700 1 --2- 172.21.15.113:0/3436239156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b3c0fd8f0 0x7f0b3c10e080 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:25.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.068+0000 7f0b4207b700 1 --2- 172.21.15.113:0/3436239156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b3c0fe250 0x7f0b3c10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:25.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.068+0000 7f0b4207b700 1 -- 172.21.15.113:0/3436239156 >> 172.21.15.113:0/3436239156 conn(0x7f0b3c0f34b0 msgr2=0x7f0b3c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:25.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.068+0000 7f0b4207b700 1 -- 172.21.15.113:0/3436239156 shutdown_connections 2024-04-12T05:44:25.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.068+0000 7f0b4207b700 1 -- 172.21.15.113:0/3436239156 wait complete. 2024-04-12T05:44:25.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.166+0000 7f6c64245700 1 -- 172.21.15.113:0/2798025558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c5c0fe250 msgr2=0x7f6c5c0f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:25.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.166+0000 7f6c64245700 1 --2- 172.21.15.113:0/2798025558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c5c0fe250 0x7f6c5c0f7a30 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7f6c580047f0 tx=0x7f6c58009cb0).stop 2024-04-12T05:44:25.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.166+0000 7f6c64245700 1 -- 172.21.15.113:0/2798025558 shutdown_connections 2024-04-12T05:44:25.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.166+0000 7f6c64245700 1 --2- 172.21.15.113:0/2798025558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c5c0fe250 0x7f6c5c0f7a30 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:25.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.166+0000 7f6c64245700 1 --2- 172.21.15.113:0/2798025558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c5c0fd8f0 0x7f6c5c0fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:25.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.166+0000 7f6c64245700 1 -- 172.21.15.113:0/2798025558 >> 172.21.15.113:0/2798025558 conn(0x7f6c5c0f34b0 msgr2=0x7f6c5c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:25.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.167+0000 7f6c64245700 1 -- 172.21.15.113:0/2798025558 shutdown_connections 2024-04-12T05:44:25.164 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.167+0000 7f6c64245700 1 -- 172.21.15.113:0/2798025558 wait complete. 2024-04-12T05:44:25.165 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.167+0000 7f6c64245700 1 Processor -- start 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.167+0000 7f6c64245700 1 -- start start 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c64245700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c5c0fd8f0 0x7f6c5c10e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c64245700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c5c0fe250 0x7f6c5c10e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c64245700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c10ec20 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c64245700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c10ed60 con 0x7f6c5c0fe250 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c61fe1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c5c0fd8f0 0x7f6c5c10e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c61fe1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c5c0fd8f0 0x7f6c5c10e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56374/0 (socket says 172.21.15.113:56374) 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c61fe1700 1 -- 172.21.15.113:0/3664625901 learned_addr learned my addr 172.21.15.113:0/3664625901 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c617e0700 1 --2- 172.21.15.113:0/3664625901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c5c0fe250 0x7f6c5c10e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c61fe1700 1 -- 172.21.15.113:0/3664625901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c5c0fe250 msgr2=0x7f6c5c10e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c61fe1700 1 --2- 172.21.15.113:0/3664625901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c5c0fe250 0x7f6c5c10e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c61fe1700 1 -- 172.21.15.113:0/3664625901 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c58005040 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.166 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c61fe1700 1 --2- 172.21.15.113:0/3664625901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c5c0fd8f0 0x7f6c5c10e130 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f6c4c00e8f0 tx=0x7f6c4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:25.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c52ffd700 1 -- 172.21.15.113:0/3664625901 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c4c00ee90 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c64245700 1 -- 172.21.15.113:0/3664625901 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5c1083f0 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c52ffd700 1 -- 172.21.15.113:0/3664625901 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c4c00f040 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c52ffd700 1 -- 172.21.15.113:0/3664625901 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c4c00c8a0 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.168+0000 7f6c64245700 1 -- 172.21.15.113:0/3664625901 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5c1089b0 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.169+0000 7f6c64245700 1 -- 172.21.15.113:0/3664625901 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c05f590 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.170+0000 7f6c52ffd700 1 -- 172.21.15.113:0/3664625901 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6c4c007b40 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.170+0000 7f6c52ffd700 1 --2- 172.21.15.113:0/3664625901 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6c4806caa0 0x7f6c4806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:25.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.170+0000 7f6c52ffd700 1 -- 172.21.15.113:0/3664625901 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6c4c08bb10 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.170+0000 7f6c617e0700 1 --2- 172.21.15.113:0/3664625901 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6c4806caa0 0x7f6c4806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:25.168 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.171+0000 7f6c617e0700 1 --2- 172.21.15.113:0/3664625901 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6c4806caa0 0x7f6c4806ef60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f6c58000c00 tx=0x7f6c5800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:25.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.175+0000 7f6c52ffd700 1 -- 172.21.15.113:0/3664625901 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6c4c0595c0 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:25 smithi113 ceph-mon[103371]: from='client.35738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:25.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:25 smithi113 ceph-mon[103371]: from='client.35742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:25.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:25 smithi113 ceph-mon[103371]: pgmap v968: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:25.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:25 smithi113 ceph-mon[103371]: from='client.35746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:25.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:25 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/867568010' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:25.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:25 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/731837698' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:44:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.442+0000 7f6c64245700 1 -- 172.21.15.113:0/3664625901 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6c5c0ffc80 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.442+0000 7f6c52ffd700 1 -- 172.21.15.113:0/3664625901 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f6c4c019e10 con 0x7f6c5c0fd8f0 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:44:25.439 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:44:25.440 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:44:25.440 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:44:25.440 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:44:25.440 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:44:25.440 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:44:25.440 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:44:25.440 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:44:25.440 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:44:25.440 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:44:25.440 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:44:25.440 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:44:25.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.444+0000 7f6c64245700 1 -- 172.21.15.113:0/3664625901 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6c4806caa0 msgr2=0x7f6c4806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:25.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.444+0000 7f6c64245700 1 --2- 172.21.15.113:0/3664625901 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6c4806caa0 0x7f6c4806ef60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f6c58000c00 tx=0x7f6c5800b040).stop 2024-04-12T05:44:25.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.444+0000 7f6c64245700 1 -- 172.21.15.113:0/3664625901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c5c0fd8f0 msgr2=0x7f6c5c10e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:25.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.444+0000 7f6c64245700 1 --2- 172.21.15.113:0/3664625901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c5c0fd8f0 0x7f6c5c10e130 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f6c4c00e8f0 tx=0x7f6c4c00b040).stop 2024-04-12T05:44:25.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.444+0000 7f6c64245700 1 -- 172.21.15.113:0/3664625901 shutdown_connections 2024-04-12T05:44:25.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.444+0000 7f6c64245700 1 --2- 172.21.15.113:0/3664625901 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6c4806caa0 0x7f6c4806ef60 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:25.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.444+0000 7f6c64245700 1 --2- 172.21.15.113:0/3664625901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c5c0fd8f0 0x7f6c5c10e130 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:25.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.444+0000 7f6c64245700 1 --2- 172.21.15.113:0/3664625901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c5c0fe250 0x7f6c5c10e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:25.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.444+0000 7f6c64245700 1 -- 172.21.15.113:0/3664625901 >> 172.21.15.113:0/3664625901 conn(0x7f6c5c0f34b0 msgr2=0x7f6c5c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:25.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.444+0000 7f6c64245700 1 -- 172.21.15.113:0/3664625901 shutdown_connections 2024-04-12T05:44:25.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:25.444+0000 7f6c64245700 1 -- 172.21.15.113:0/3664625901 wait complete. 2024-04-12T05:44:25.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:25 smithi192 ceph-mon[85304]: from='client.35738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:25.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:25 smithi192 ceph-mon[85304]: from='client.35742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:25.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:25 smithi192 ceph-mon[85304]: pgmap v968: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:25.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:25 smithi192 ceph-mon[85304]: from='client.35746 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:25.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:25 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/867568010' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:25.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:25 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/731837698' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:44:26.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:26 smithi113 ceph-mon[103371]: from='client.35758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:26.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:26 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3664625901' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:44:26.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:26 smithi192 ceph-mon[85304]: from='client.35758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:26.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:26 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3664625901' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:44:27.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:27 smithi113 ceph-mon[103371]: pgmap v969: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:27.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:27 smithi192 ceph-mon[85304]: pgmap v969: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:29.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:29 smithi113 ceph-mon[103371]: pgmap v970: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:29.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:29 smithi192 ceph-mon[85304]: pgmap v970: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:31.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:31 smithi113 ceph-mon[103371]: pgmap v971: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:31.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:31 smithi192 ceph-mon[85304]: pgmap v971: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:33.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:33 smithi113 ceph-mon[103371]: pgmap v972: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:33.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:33 smithi192 ceph-mon[85304]: pgmap v972: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:35.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:35 smithi113 ceph-mon[103371]: pgmap v973: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:35.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:35 smithi192 ceph-mon[85304]: pgmap v973: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:37.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:37 smithi113 ceph-mon[103371]: pgmap v974: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:37.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:37.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:37 smithi192 ceph-mon[85304]: pgmap v974: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:37.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:39.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:39 smithi113 ceph-mon[103371]: pgmap v975: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:39.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:39 smithi192 ceph-mon[85304]: pgmap v975: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:41.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:41 smithi113 ceph-mon[103371]: pgmap v976: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:41.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:41 smithi192 ceph-mon[85304]: pgmap v976: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:43.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:43 smithi113 ceph-mon[103371]: pgmap v977: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:43.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:43 smithi192 ceph-mon[85304]: pgmap v977: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:45.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:45 smithi113 ceph-mon[103371]: pgmap v978: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:45.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:45 smithi192 ceph-mon[85304]: pgmap v978: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:47.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:47 smithi113 ceph-mon[103371]: pgmap v979: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:47.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:47 smithi192 ceph-mon[85304]: pgmap v979: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:49.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:49 smithi113 ceph-mon[103371]: pgmap v980: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:49.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:49 smithi192 ceph-mon[85304]: pgmap v980: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:51.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:51 smithi113 ceph-mon[103371]: pgmap v981: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:51.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:51 smithi192 ceph-mon[85304]: pgmap v981: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:52.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:52.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:44:53.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:53 smithi113 ceph-mon[103371]: pgmap v982: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:53.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:53 smithi192 ceph-mon[85304]: pgmap v982: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:55.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:55 smithi113 ceph-mon[103371]: pgmap v983: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:55.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:55 smithi192 ceph-mon[85304]: pgmap v983: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:55.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.551+0000 7f107440d700 1 -- 172.21.15.113:0/2432413225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f106c0f88a0 msgr2=0x7f106c0f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:55.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.551+0000 7f107440d700 1 --2- 172.21.15.113:0/2432413225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f106c0f88a0 0x7f106c0f8c80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f105c0047f0 tx=0x7f105c009cb0).stop 2024-04-12T05:44:55.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.551+0000 7f107440d700 1 -- 172.21.15.113:0/2432413225 shutdown_connections 2024-04-12T05:44:55.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.551+0000 7f107440d700 1 --2- 172.21.15.113:0/2432413225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f106c0f9200 0x7f106c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:55.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.551+0000 7f107440d700 1 --2- 172.21.15.113:0/2432413225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f106c0f88a0 0x7f106c0f8c80 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:55.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.551+0000 7f107440d700 1 -- 172.21.15.113:0/2432413225 >> 172.21.15.113:0/2432413225 conn(0x7f106c070590 msgr2=0x7f106c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:55.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.551+0000 7f107440d700 1 -- 172.21.15.113:0/2432413225 shutdown_connections 2024-04-12T05:44:55.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.551+0000 7f107440d700 1 -- 172.21.15.113:0/2432413225 wait complete. 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f107440d700 1 Processor -- start 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f107440d700 1 -- start start 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f107440d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f106c0f88a0 0x7f106c10e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f107440d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f106c0f9200 0x7f106c10e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f107440d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c10eb30 con 0x7f106c0f88a0 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f107440d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c10ec70 con 0x7f106c0f9200 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f10721a9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f106c0f88a0 0x7f106c10e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f10721a9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f106c0f88a0 0x7f106c10e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56170/0 (socket says 172.21.15.113:56170) 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f10721a9700 1 -- 172.21.15.113:0/640696588 learned_addr learned my addr 172.21.15.113:0/640696588 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f10719a8700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f106c0f9200 0x7f106c10e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f10721a9700 1 -- 172.21.15.113:0/640696588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f106c0f9200 msgr2=0x7f106c10e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:55.550 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f10721a9700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f106c0f9200 0x7f106c10e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:55.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f10721a9700 1 -- 172.21.15.113:0/640696588 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c005040 con 0x7f106c0f88a0 2024-04-12T05:44:55.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f10719a8700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f106c0f9200 0x7f106c10e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:44:55.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.552+0000 7f10721a9700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f106c0f88a0 0x7f106c10e040 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f105c00b120 tx=0x7f105c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:55.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.553+0000 7f10637fe700 1 -- 172.21.15.113:0/640696588 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f105c00e030 con 0x7f106c0f88a0 2024-04-12T05:44:55.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.553+0000 7f10637fe700 1 -- 172.21.15.113:0/640696588 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f105c007e90 con 0x7f106c0f88a0 2024-04-12T05:44:55.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.553+0000 7f107440d700 1 -- 172.21.15.113:0/640696588 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c1082a0 con 0x7f106c0f88a0 2024-04-12T05:44:55.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.553+0000 7f10637fe700 1 -- 172.21.15.113:0/640696588 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f105c018070 con 0x7f106c0f88a0 2024-04-12T05:44:55.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.553+0000 7f107440d700 1 -- 172.21.15.113:0/640696588 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c108780 con 0x7f106c0f88a0 2024-04-12T05:44:55.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.554+0000 7f107440d700 1 -- 172.21.15.113:0/640696588 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c05f590 con 0x7f106c0f88a0 2024-04-12T05:44:55.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.554+0000 7f10637fe700 1 -- 172.21.15.113:0/640696588 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f105c01d420 con 0x7f106c0f88a0 2024-04-12T05:44:55.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.554+0000 7f10637fe700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f105806caa0 0x7f105806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:55.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.555+0000 7f10637fe700 1 -- 172.21.15.113:0/640696588 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f105c08d300 con 0x7f106c0f88a0 2024-04-12T05:44:55.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.555+0000 7f10719a8700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f105806caa0 0x7f105806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:55.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.555+0000 7f10719a8700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f105806caa0 0x7f105806ef60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f106c109980 tx=0x7f1068009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:55.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.559+0000 7f10637fe700 1 -- 172.21.15.113:0/640696588 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f105c05adb0 con 0x7f106c0f88a0 2024-04-12T05:44:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.765+0000 7f107440d700 1 -- 172.21.15.113:0/640696588 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f106c0fedd0 con 0x7f105806caa0 2024-04-12T05:44:55.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.766+0000 7f10637fe700 1 -- 172.21.15.113:0/640696588 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f106c0fedd0 con 0x7f105806caa0 2024-04-12T05:44:55.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.769+0000 7f107440d700 1 -- 172.21.15.113:0/640696588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f105806caa0 msgr2=0x7f105806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:55.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.769+0000 7f107440d700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f105806caa0 0x7f105806ef60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f106c109980 tx=0x7f1068009040).stop 2024-04-12T05:44:55.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.769+0000 7f107440d700 1 -- 172.21.15.113:0/640696588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f106c0f88a0 msgr2=0x7f106c10e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:55.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.769+0000 7f107440d700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f106c0f88a0 0x7f106c10e040 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f105c00b120 tx=0x7f105c009cb0).stop 2024-04-12T05:44:55.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.769+0000 7f107440d700 1 -- 172.21.15.113:0/640696588 shutdown_connections 2024-04-12T05:44:55.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.769+0000 7f107440d700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f105806caa0 0x7f105806ef60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:55.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.769+0000 7f107440d700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f106c0f88a0 0x7f106c10e040 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:55.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.769+0000 7f107440d700 1 --2- 172.21.15.113:0/640696588 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f106c0f9200 0x7f106c10e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:55.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.769+0000 7f107440d700 1 -- 172.21.15.113:0/640696588 >> 172.21.15.113:0/640696588 conn(0x7f106c070590 msgr2=0x7f106c0fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:55.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.769+0000 7f107440d700 1 -- 172.21.15.113:0/640696588 shutdown_connections 2024-04-12T05:44:55.766 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.769+0000 7f107440d700 1 -- 172.21.15.113:0/640696588 wait complete. 2024-04-12T05:44:55.775 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:44:55.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.867+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/547340896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5a00fe270 msgr2=0x7fd5a00f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:55.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.867+0000 7fd5a5dbd700 1 --2- 172.21.15.113:0/547340896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5a00fe270 0x7fd5a00f7a50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fd5940047f0 tx=0x7fd594009cb0).stop 2024-04-12T05:44:55.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.867+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/547340896 shutdown_connections 2024-04-12T05:44:55.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.867+0000 7fd5a5dbd700 1 --2- 172.21.15.113:0/547340896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5a00fe270 0x7fd5a00f7a50 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:55.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.867+0000 7fd5a5dbd700 1 --2- 172.21.15.113:0/547340896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5a00fd910 0x7fd5a00fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:55.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.867+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/547340896 >> 172.21.15.113:0/547340896 conn(0x7fd5a00f3530 msgr2=0x7fd5a00f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:55.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.867+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/547340896 shutdown_connections 2024-04-12T05:44:55.864 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.867+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/547340896 wait complete. 2024-04-12T05:44:55.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.868+0000 7fd5a5dbd700 1 Processor -- start 2024-04-12T05:44:55.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.868+0000 7fd5a5dbd700 1 -- start start 2024-04-12T05:44:55.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.868+0000 7fd5a5dbd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5a00fd910 0x7fd5a010e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:55.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.868+0000 7fd5a5dbd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5a00fe270 0x7fd5a010e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:55.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.868+0000 7fd5a5dbd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a010ebf0 con 0x7fd5a00fd910 2024-04-12T05:44:55.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.868+0000 7fd5a5dbd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a010ed30 con 0x7fd5a00fe270 2024-04-12T05:44:55.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.868+0000 7fd59f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5a00fd910 0x7fd5a010e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:55.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.868+0000 7fd59f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5a00fd910 0x7fd5a010e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56190/0 (socket says 172.21.15.113:56190) 2024-04-12T05:44:55.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.868+0000 7fd59f7fe700 1 -- 172.21.15.113:0/1205570553 learned_addr learned my addr 172.21.15.113:0/1205570553 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:55.866 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.869+0000 7fd59effd700 1 --2- 172.21.15.113:0/1205570553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5a00fe270 0x7fd5a010e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.869+0000 7fd59f7fe700 1 -- 172.21.15.113:0/1205570553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5a00fe270 msgr2=0x7fd5a010e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.869+0000 7fd59f7fe700 1 --2- 172.21.15.113:0/1205570553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5a00fe270 0x7fd5a010e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.869+0000 7fd59f7fe700 1 -- 172.21.15.113:0/1205570553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd594005040 con 0x7fd5a00fd910 2024-04-12T05:44:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.869+0000 7fd59f7fe700 1 --2- 172.21.15.113:0/1205570553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5a00fd910 0x7fd5a010e100 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7fd59000e910 tx=0x7fd59000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.869+0000 7fd59cff9700 1 -- 172.21.15.113:0/1205570553 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd59000eeb0 con 0x7fd5a00fd910 2024-04-12T05:44:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.869+0000 7fd59cff9700 1 -- 172.21.15.113:0/1205570553 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd59000f040 con 0x7fd5a00fd910 2024-04-12T05:44:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.869+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/1205570553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a0108360 con 0x7fd5a00fd910 2024-04-12T05:44:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.869+0000 7fd59cff9700 1 -- 172.21.15.113:0/1205570553 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd59000c8a0 con 0x7fd5a00fd910 2024-04-12T05:44:55.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.869+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/1205570553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a0108920 con 0x7fd5a00fd910 2024-04-12T05:44:55.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.870+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/1205570553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a005f570 con 0x7fd5a00fd910 2024-04-12T05:44:55.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.870+0000 7fd59cff9700 1 -- 172.21.15.113:0/1205570553 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd5900090d0 con 0x7fd5a00fd910 2024-04-12T05:44:55.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.871+0000 7fd59cff9700 1 --2- 172.21.15.113:0/1205570553 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd58806caa0 0x7fd58806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:55.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.871+0000 7fd59cff9700 1 -- 172.21.15.113:0/1205570553 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd59008b920 con 0x7fd5a00fd910 2024-04-12T05:44:55.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.871+0000 7fd59effd700 1 --2- 172.21.15.113:0/1205570553 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd58806caa0 0x7fd58806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:55.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.871+0000 7fd59effd700 1 --2- 172.21.15.113:0/1205570553 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd58806caa0 0x7fd58806ef60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fd594004e60 tx=0x7fd59400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:55.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:55.875+0000 7fd59cff9700 1 -- 172.21.15.113:0/1205570553 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd5900593d0 con 0x7fd5a00fd910 2024-04-12T05:44:56.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.079+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/1205570553 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5a00ffc60 con 0x7fd58806caa0 2024-04-12T05:44:56.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.081+0000 7fd59cff9700 1 -- 172.21.15.113:0/1205570553 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd5a00ffc60 con 0x7fd58806caa0 2024-04-12T05:44:56.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.083+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/1205570553 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd58806caa0 msgr2=0x7fd58806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.083+0000 7fd5a5dbd700 1 --2- 172.21.15.113:0/1205570553 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd58806caa0 0x7fd58806ef60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fd594004e60 tx=0x7fd59400b040).stop 2024-04-12T05:44:56.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.083+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/1205570553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5a00fd910 msgr2=0x7fd5a010e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.079 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.083+0000 7fd5a5dbd700 1 --2- 172.21.15.113:0/1205570553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5a00fd910 0x7fd5a010e100 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7fd59000e910 tx=0x7fd59000b040).stop 2024-04-12T05:44:56.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.083+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/1205570553 shutdown_connections 2024-04-12T05:44:56.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.083+0000 7fd5a5dbd700 1 --2- 172.21.15.113:0/1205570553 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd58806caa0 0x7fd58806ef60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.083+0000 7fd5a5dbd700 1 --2- 172.21.15.113:0/1205570553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5a00fd910 0x7fd5a010e100 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.083+0000 7fd5a5dbd700 1 --2- 172.21.15.113:0/1205570553 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5a00fe270 0x7fd5a010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.083+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/1205570553 >> 172.21.15.113:0/1205570553 conn(0x7fd5a00f3530 msgr2=0x7fd5a00fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:56.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.083+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/1205570553 shutdown_connections 2024-04-12T05:44:56.080 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.083+0000 7fd5a5dbd700 1 -- 172.21.15.113:0/1205570553 wait complete. 2024-04-12T05:44:56.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.181+0000 7f9aad9db700 1 -- 172.21.15.113:0/3061402637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9aa80fe230 msgr2=0x7f9aa80f5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.181+0000 7f9aad9db700 1 --2- 172.21.15.113:0/3061402637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9aa80fe230 0x7f9aa80f5fa0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7f9a980047f0 tx=0x7f9a98009cb0).stop 2024-04-12T05:44:56.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.181+0000 7f9aad9db700 1 -- 172.21.15.113:0/3061402637 shutdown_connections 2024-04-12T05:44:56.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.181+0000 7f9aad9db700 1 --2- 172.21.15.113:0/3061402637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9aa80fe230 0x7f9aa80f5fa0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.181+0000 7f9aad9db700 1 --2- 172.21.15.113:0/3061402637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9aa80fd8d0 0x7f9aa80fdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.181+0000 7f9aad9db700 1 -- 172.21.15.113:0/3061402637 >> 172.21.15.113:0/3061402637 conn(0x7f9aa806ce40 msgr2=0x7f9aa806d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:56.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.182+0000 7f9aad9db700 1 -- 172.21.15.113:0/3061402637 shutdown_connections 2024-04-12T05:44:56.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.182+0000 7f9aad9db700 1 -- 172.21.15.113:0/3061402637 wait complete. 2024-04-12T05:44:56.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.182+0000 7f9aad9db700 1 Processor -- start 2024-04-12T05:44:56.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.182+0000 7f9aad9db700 1 -- start start 2024-04-12T05:44:56.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.182+0000 7f9aad9db700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9aa80fd8d0 0x7f9aa810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:56.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.182+0000 7f9aad9db700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9aa80fe230 0x7f9aa810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:56.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9aad9db700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa810eb20 con 0x7f9aa80fd8d0 2024-04-12T05:44:56.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9aad9db700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa810ec60 con 0x7f9aa80fe230 2024-04-12T05:44:56.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9aa6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9aa80fd8d0 0x7f9aa810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:56.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9aa6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9aa80fd8d0 0x7f9aa810e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56210/0 (socket says 172.21.15.113:56210) 2024-04-12T05:44:56.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9aa6ffd700 1 -- 172.21.15.113:0/1357839715 learned_addr learned my addr 172.21.15.113:0/1357839715 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:56.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9aa6ffd700 1 -- 172.21.15.113:0/1357839715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9aa80fe230 msgr2=0x7f9aa810e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9aa6ffd700 1 --2- 172.21.15.113:0/1357839715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9aa80fe230 0x7f9aa810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:44:56.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9aa6ffd700 1 -- 172.21.15.113:0/1357839715 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a98005040 con 0x7f9aa80fd8d0 2024-04-12T05:44:56.181 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9aa6ffd700 1 --2- 172.21.15.113:0/1357839715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9aa80fd8d0 0x7f9aa810e030 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f9a9c002770 tx=0x7f9a9c002b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:56.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9a8ffff700 1 -- 172.21.15.113:0/1357839715 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a9c002d80 con 0x7f9aa80fd8d0 2024-04-12T05:44:56.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9a8ffff700 1 -- 172.21.15.113:0/1357839715 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a9c00f040 con 0x7f9aa80fd8d0 2024-04-12T05:44:56.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9aad9db700 1 -- 172.21.15.113:0/1357839715 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa81082f0 con 0x7f9aa80fd8d0 2024-04-12T05:44:56.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9a8ffff700 1 -- 172.21.15.113:0/1357839715 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a9c00d800 con 0x7f9aa80fd8d0 2024-04-12T05:44:56.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.183+0000 7f9aad9db700 1 -- 172.21.15.113:0/1357839715 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa8108830 con 0x7f9aa80fd8d0 2024-04-12T05:44:56.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.184+0000 7f9aad9db700 1 -- 172.21.15.113:0/1357839715 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa805f090 con 0x7f9aa80fd8d0 2024-04-12T05:44:56.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.185+0000 7f9a8ffff700 1 -- 172.21.15.113:0/1357839715 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9a9c009110 con 0x7f9aa80fd8d0 2024-04-12T05:44:56.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.185+0000 7f9a8ffff700 1 --2- 172.21.15.113:0/1357839715 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a9006caa0 0x7f9a9006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:56.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.185+0000 7f9a8ffff700 1 -- 172.21.15.113:0/1357839715 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9a9c08bac0 con 0x7f9aa80fd8d0 2024-04-12T05:44:56.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.185+0000 7f9aa67fc700 1 --2- 172.21.15.113:0/1357839715 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a9006caa0 0x7f9a9006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:56.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.185+0000 7f9aa67fc700 1 --2- 172.21.15.113:0/1357839715 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a9006caa0 0x7f9a9006ef60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f9aa8109970 tx=0x7f9a98009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:56.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.189+0000 7f9a8ffff700 1 -- 172.21.15.113:0/1357839715 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9a9c059570 con 0x7f9aa80fd8d0 2024-04-12T05:44:56.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.388+0000 7f9aad9db700 1 -- 172.21.15.113:0/1357839715 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9aa8108ab0 con 0x7f9a9006caa0 2024-04-12T05:44:56.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.397+0000 7f9a8ffff700 1 -- 172.21.15.113:0/1357839715 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9aa8108ab0 con 0x7f9a9006caa0 2024-04-12T05:44:56.394 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:44:56.394 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (32m) 10m ago 43m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:44:56.394 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (30m) 10m ago 43m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:44:56.394 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (30m) 10m ago 43m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:44:56.394 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (32m) 10m ago 43m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:44:56.394 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (26m) 10m ago 38m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:44:56.394 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (25m) 10m ago 38m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:44:56.394 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (25m) 10m ago 38m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:44:56.394 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (10m) 10m ago 38m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (34m) 10m ago 45m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (33m) 10m ago 43m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (31m) 10m ago 45m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (30m) 10m ago 43m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (33m) 10m ago 43m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (33m) 10m ago 43m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (29m) 10m ago 42m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (29m) 10m ago 41m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (28m) 10m ago 41m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (28m) 10m ago 41m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (27m) 10m ago 40m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (26m) 10m ago 40m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:44:56.395 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (33m) 10m ago 42m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:44:56.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.399+0000 7f9aad9db700 1 -- 172.21.15.113:0/1357839715 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a9006caa0 msgr2=0x7f9a9006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.399+0000 7f9aad9db700 1 --2- 172.21.15.113:0/1357839715 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a9006caa0 0x7f9a9006ef60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f9aa8109970 tx=0x7f9a98009cb0).stop 2024-04-12T05:44:56.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.399+0000 7f9aad9db700 1 -- 172.21.15.113:0/1357839715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9aa80fd8d0 msgr2=0x7f9aa810e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.399+0000 7f9aad9db700 1 --2- 172.21.15.113:0/1357839715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9aa80fd8d0 0x7f9aa810e030 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f9a9c002770 tx=0x7f9a9c002b30).stop 2024-04-12T05:44:56.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.399+0000 7f9aad9db700 1 -- 172.21.15.113:0/1357839715 shutdown_connections 2024-04-12T05:44:56.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.399+0000 7f9aad9db700 1 --2- 172.21.15.113:0/1357839715 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a9006caa0 0x7f9a9006ef60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.396 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.399+0000 7f9aad9db700 1 --2- 172.21.15.113:0/1357839715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9aa80fd8d0 0x7f9aa810e030 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.399+0000 7f9aad9db700 1 --2- 172.21.15.113:0/1357839715 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9aa80fe230 0x7f9aa810e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:44:56.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.399+0000 7f9aad9db700 1 -- 172.21.15.113:0/1357839715 >> 172.21.15.113:0/1357839715 conn(0x7f9aa806ce40 msgr2=0x7f9aa8100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:56.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.400+0000 7f9aad9db700 1 -- 172.21.15.113:0/1357839715 shutdown_connections 2024-04-12T05:44:56.397 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.400+0000 7f9aad9db700 1 -- 172.21.15.113:0/1357839715 wait complete. 2024-04-12T05:44:56.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.495+0000 7f7d88780700 1 -- 172.21.15.113:0/3260175697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d800fd140 msgr2=0x7f7d800f7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.495+0000 7f7d88780700 1 --2- 172.21.15.113:0/3260175697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d800fd140 0x7f7d800f7090 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7f7d700047f0 tx=0x7f7d70009cb0).stop 2024-04-12T05:44:56.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 -- 172.21.15.113:0/3260175697 shutdown_connections 2024-04-12T05:44:56.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 --2- 172.21.15.113:0/3260175697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d800fd140 0x7f7d800f7090 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 --2- 172.21.15.113:0/3260175697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800fc870 0x7f7d800fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 -- 172.21.15.113:0/3260175697 >> 172.21.15.113:0/3260175697 conn(0x7f7d8006ce40 msgr2=0x7f7d8006d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:56.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 -- 172.21.15.113:0/3260175697 shutdown_connections 2024-04-12T05:44:56.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 -- 172.21.15.113:0/3260175697 wait complete. 2024-04-12T05:44:56.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 Processor -- start 2024-04-12T05:44:56.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 -- start start 2024-04-12T05:44:56.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d800fc870 0x7f7d8010e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:56.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800fd140 0x7f7d8010e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:56.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8010eb90 con 0x7f7d800fc870 2024-04-12T05:44:56.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.496+0000 7f7d88780700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8010ecd0 con 0x7f7d800fd140 2024-04-12T05:44:56.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d8651c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d800fc870 0x7f7d8010e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:56.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d8651c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d800fc870 0x7f7d8010e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56230/0 (socket says 172.21.15.113:56230) 2024-04-12T05:44:56.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d8651c700 1 -- 172.21.15.113:0/1834846262 learned_addr learned my addr 172.21.15.113:0/1834846262 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:56.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d85d1b700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800fd140 0x7f7d8010e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:56.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d8651c700 1 -- 172.21.15.113:0/1834846262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800fd140 msgr2=0x7f7d8010e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d8651c700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800fd140 0x7f7d8010e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d8651c700 1 -- 172.21.15.113:0/1834846262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d70005040 con 0x7f7d800fc870 2024-04-12T05:44:56.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d85d1b700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800fd140 0x7f7d8010e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:44:56.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d8651c700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d800fc870 0x7f7d8010e0a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7f7d7c0041d0 tx=0x7f7d7c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:56.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d777fe700 1 -- 172.21.15.113:0/1834846262 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d7c007b40 con 0x7f7d800fc870 2024-04-12T05:44:56.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d777fe700 1 -- 172.21.15.113:0/1834846262 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d7c00f040 con 0x7f7d800fc870 2024-04-12T05:44:56.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d88780700 1 -- 172.21.15.113:0/1834846262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d801083b0 con 0x7f7d800fc870 2024-04-12T05:44:56.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d777fe700 1 -- 172.21.15.113:0/1834846262 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d7c00d800 con 0x7f7d800fc870 2024-04-12T05:44:56.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.497+0000 7f7d88780700 1 -- 172.21.15.113:0/1834846262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d80108970 con 0x7f7d800fc870 2024-04-12T05:44:56.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.498+0000 7f7d88780700 1 -- 172.21.15.113:0/1834846262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d80060670 con 0x7f7d800fc870 2024-04-12T05:44:56.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.499+0000 7f7d777fe700 1 -- 172.21.15.113:0/1834846262 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7d7c007ca0 con 0x7f7d800fc870 2024-04-12T05:44:56.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.499+0000 7f7d777fe700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7d6c06caa0 0x7f7d6c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:56.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.499+0000 7f7d777fe700 1 -- 172.21.15.113:0/1834846262 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7d7c08b8b0 con 0x7f7d800fc870 2024-04-12T05:44:56.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.499+0000 7f7d85d1b700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7d6c06caa0 0x7f7d6c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:56.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.500+0000 7f7d85d1b700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7d6c06caa0 0x7f7d6c06ef60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f7d801099e0 tx=0x7f7d7000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:56.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.503+0000 7f7d777fe700 1 -- 172.21.15.113:0/1834846262 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7d7c0593e0 con 0x7f7d800fc870 2024-04-12T05:44:56.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.770+0000 7f7d88780700 1 -- 172.21.15.113:0/1834846262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7d80109720 con 0x7f7d800fc870 2024-04-12T05:44:56.767 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.771+0000 7f7d777fe700 1 -- 172.21.15.113:0/1834846262 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f7d7c058f70 con 0x7f7d800fc870 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:44:56.768 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:44:56.769 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:44:56.769 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:44:56.769 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:44:56.769 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.773+0000 7f7d88780700 1 -- 172.21.15.113:0/1834846262 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7d6c06caa0 msgr2=0x7f7d6c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.773+0000 7f7d88780700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7d6c06caa0 0x7f7d6c06ef60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f7d801099e0 tx=0x7f7d7000b040).stop 2024-04-12T05:44:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.773+0000 7f7d88780700 1 -- 172.21.15.113:0/1834846262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d800fc870 msgr2=0x7f7d8010e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.773+0000 7f7d88780700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d800fc870 0x7f7d8010e0a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7f7d7c0041d0 tx=0x7f7d7c009f60).stop 2024-04-12T05:44:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.773+0000 7f7d88780700 1 -- 172.21.15.113:0/1834846262 shutdown_connections 2024-04-12T05:44:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.773+0000 7f7d88780700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7d6c06caa0 0x7f7d6c06ef60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.773+0000 7f7d88780700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d800fc870 0x7f7d8010e0a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.773+0000 7f7d88780700 1 --2- 172.21.15.113:0/1834846262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d800fd140 0x7f7d8010e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.773+0000 7f7d88780700 1 -- 172.21.15.113:0/1834846262 >> 172.21.15.113:0/1834846262 conn(0x7f7d8006ce40 msgr2=0x7f7d800f5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.773+0000 7f7d88780700 1 -- 172.21.15.113:0/1834846262 shutdown_connections 2024-04-12T05:44:56.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.773+0000 7f7d88780700 1 -- 172.21.15.113:0/1834846262 wait complete. 2024-04-12T05:44:56.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.870+0000 7f8453da2700 1 -- 172.21.15.113:0/238503794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f844c0fc7f0 msgr2=0x7f844c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.870+0000 7f8453da2700 1 --2- 172.21.15.113:0/238503794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f844c0fc7f0 0x7f844c0fcbd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7f84480047f0 tx=0x7f8448009cb0).stop 2024-04-12T05:44:56.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.870+0000 7f8453da2700 1 -- 172.21.15.113:0/238503794 shutdown_connections 2024-04-12T05:44:56.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.870+0000 7f8453da2700 1 --2- 172.21.15.113:0/238503794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f844c0fd150 0x7f844c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.871+0000 7f8453da2700 1 --2- 172.21.15.113:0/238503794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f844c0fc7f0 0x7f844c0fcbd0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.871+0000 7f8453da2700 1 -- 172.21.15.113:0/238503794 >> 172.21.15.113:0/238503794 conn(0x7f844c0f23b0 msgr2=0x7f844c0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:56.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.871+0000 7f8453da2700 1 -- 172.21.15.113:0/238503794 shutdown_connections 2024-04-12T05:44:56.868 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.871+0000 7f8453da2700 1 -- 172.21.15.113:0/238503794 wait complete. 2024-04-12T05:44:56.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.871+0000 7f8453da2700 1 Processor -- start 2024-04-12T05:44:56.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.871+0000 7f8453da2700 1 -- start start 2024-04-12T05:44:56.869 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f8453da2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f844c0fc7f0 0x7f844c10cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f8453da2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f844c0fd150 0x7f844c10d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f8453da2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c10da10 con 0x7f844c0fd150 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f8453da2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c10db50 con 0x7f844c0fc7f0 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f845133d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f844c0fd150 0x7f844c10d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f845133d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f844c0fd150 0x7f844c10d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56248/0 (socket says 172.21.15.113:56248) 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f845133d700 1 -- 172.21.15.113:0/3433605575 learned_addr learned my addr 172.21.15.113:0/3433605575 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f8451b3e700 1 --2- 172.21.15.113:0/3433605575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f844c0fc7f0 0x7f844c10cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f845133d700 1 -- 172.21.15.113:0/3433605575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f844c0fc7f0 msgr2=0x7f844c10cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f845133d700 1 --2- 172.21.15.113:0/3433605575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f844c0fc7f0 0x7f844c10cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f845133d700 1 -- 172.21.15.113:0/3433605575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8448005040 con 0x7f844c0fd150 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f845133d700 1 --2- 172.21.15.113:0/3433605575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f844c0fd150 0x7f844c10d410 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f843c002780 tx=0x7f843c009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f8442ffd700 1 -- 172.21.15.113:0/3433605575 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f843c007b40 con 0x7f844c0fd150 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f8442ffd700 1 -- 172.21.15.113:0/3433605575 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f843c00f040 con 0x7f844c0fd150 2024-04-12T05:44:56.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f8453da2700 1 -- 172.21.15.113:0/3433605575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c1071e0 con 0x7f844c0fd150 2024-04-12T05:44:56.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.872+0000 7f8442ffd700 1 -- 172.21.15.113:0/3433605575 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f843c014780 con 0x7f844c0fd150 2024-04-12T05:44:56.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.873+0000 7f8453da2700 1 -- 172.21.15.113:0/3433605575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c1077a0 con 0x7f844c0fd150 2024-04-12T05:44:56.871 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.873+0000 7f8453da2700 1 -- 172.21.15.113:0/3433605575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c060390 con 0x7f844c0fd150 2024-04-12T05:44:56.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.874+0000 7f8442ffd700 1 -- 172.21.15.113:0/3433605575 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f843c007ca0 con 0x7f844c0fd150 2024-04-12T05:44:56.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.874+0000 7f8442ffd700 1 --2- 172.21.15.113:0/3433605575 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f843806caa0 0x7f843806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:56.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.874+0000 7f8442ffd700 1 -- 172.21.15.113:0/3433605575 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f843c08bd50 con 0x7f844c0fd150 2024-04-12T05:44:56.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.874+0000 7f8451b3e700 1 --2- 172.21.15.113:0/3433605575 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f843806caa0 0x7f843806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:56.872 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.875+0000 7f8451b3e700 1 --2- 172.21.15.113:0/3433605575 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f843806caa0 0x7f843806ef60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f8448004f10 tx=0x7f844800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:56.875 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:56.879+0000 7f8442ffd700 1 -- 172.21.15.113:0/3433605575 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f843c059800 con 0x7f844c0fd150 2024-04-12T05:44:57.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.111+0000 7f8453da2700 1 -- 172.21.15.113:0/3433605575 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f844c047540 con 0x7f844c0fd150 2024-04-12T05:44:57.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.111+0000 7f8442ffd700 1 -- 172.21.15.113:0/3433605575 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f843c059390 con 0x7f844c0fd150 2024-04-12T05:44:57.107 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:44:57.108 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:44:57.109 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:57.110 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:44:57.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.113+0000 7f8453da2700 1 -- 172.21.15.113:0/3433605575 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f843806caa0 msgr2=0x7f843806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:57.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.113+0000 7f8453da2700 1 --2- 172.21.15.113:0/3433605575 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f843806caa0 0x7f843806ef60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f8448004f10 tx=0x7f844800b040).stop 2024-04-12T05:44:57.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.113+0000 7f8453da2700 1 -- 172.21.15.113:0/3433605575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f844c0fd150 msgr2=0x7f844c10d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:57.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.113+0000 7f8453da2700 1 --2- 172.21.15.113:0/3433605575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f844c0fd150 0x7f844c10d410 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f843c002780 tx=0x7f843c009ee0).stop 2024-04-12T05:44:57.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.113+0000 7f8453da2700 1 -- 172.21.15.113:0/3433605575 shutdown_connections 2024-04-12T05:44:57.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.113+0000 7f8453da2700 1 --2- 172.21.15.113:0/3433605575 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f843806caa0 0x7f843806ef60 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.113+0000 7f8453da2700 1 --2- 172.21.15.113:0/3433605575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f844c0fc7f0 0x7f844c10cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.113+0000 7f8453da2700 1 --2- 172.21.15.113:0/3433605575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f844c0fd150 0x7f844c10d410 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.113+0000 7f8453da2700 1 -- 172.21.15.113:0/3433605575 >> 172.21.15.113:0/3433605575 conn(0x7f844c0f23b0 msgr2=0x7f844c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:57.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.113+0000 7f8453da2700 1 -- 172.21.15.113:0/3433605575 shutdown_connections 2024-04-12T05:44:57.112 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.113+0000 7f8453da2700 1 -- 172.21.15.113:0/3433605575 wait complete. 2024-04-12T05:44:57.112 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:44:57.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.213+0000 7f695dbee700 1 -- 172.21.15.113:0/509179581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69580fd150 msgr2=0x7f69580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:57.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.213+0000 7f695dbee700 1 --2- 172.21.15.113:0/509179581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69580fd150 0x7f69580f6930 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7f694c0047f0 tx=0x7f694c009cb0).stop 2024-04-12T05:44:57.209 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.213+0000 7f695dbee700 1 -- 172.21.15.113:0/509179581 shutdown_connections 2024-04-12T05:44:57.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.213+0000 7f695dbee700 1 --2- 172.21.15.113:0/509179581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69580fd150 0x7f69580f6930 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.213+0000 7f695dbee700 1 --2- 172.21.15.113:0/509179581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69580fc7f0 0x7f69580fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.213+0000 7f695dbee700 1 -- 172.21.15.113:0/509179581 >> 172.21.15.113:0/509179581 conn(0x7f6958070590 msgr2=0x7f69580f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:57.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.213+0000 7f695dbee700 1 -- 172.21.15.113:0/509179581 shutdown_connections 2024-04-12T05:44:57.210 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.213+0000 7f695dbee700 1 -- 172.21.15.113:0/509179581 wait complete. 2024-04-12T05:44:57.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.214+0000 7f695dbee700 1 Processor -- start 2024-04-12T05:44:57.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.214+0000 7f695dbee700 1 -- start start 2024-04-12T05:44:57.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.214+0000 7f695dbee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69580fc7f0 0x7f695810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:57.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.214+0000 7f695dbee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69580fd150 0x7f695810d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:57.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.214+0000 7f695dbee700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695810d9c0 con 0x7f69580fd150 2024-04-12T05:44:57.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.214+0000 7f695dbee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695810db00 con 0x7f69580fc7f0 2024-04-12T05:44:57.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.214+0000 7f6956ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69580fd150 0x7f695810d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:57.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.214+0000 7f6956ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69580fd150 0x7f695810d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56272/0 (socket says 172.21.15.113:56272) 2024-04-12T05:44:57.212 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.214+0000 7f6956ffd700 1 -- 172.21.15.113:0/1290705753 learned_addr learned my addr 172.21.15.113:0/1290705753 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:57.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.214+0000 7f69577fe700 1 --2- 172.21.15.113:0/1290705753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69580fc7f0 0x7f695810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:57.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.215+0000 7f6956ffd700 1 -- 172.21.15.113:0/1290705753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69580fc7f0 msgr2=0x7f695810ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:57.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.215+0000 7f6956ffd700 1 --2- 172.21.15.113:0/1290705753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69580fc7f0 0x7f695810ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.215+0000 7f6956ffd700 1 -- 172.21.15.113:0/1290705753 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c005040 con 0x7f69580fd150 2024-04-12T05:44:57.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.215+0000 7f6956ffd700 1 --2- 172.21.15.113:0/1290705753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69580fd150 0x7f695810d3c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7f694c004f10 tx=0x7f694c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:57.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.215+0000 7f6954ff9700 1 -- 172.21.15.113:0/1290705753 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f694c00e030 con 0x7f69580fd150 2024-04-12T05:44:57.213 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.215+0000 7f6954ff9700 1 -- 172.21.15.113:0/1290705753 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f694c007e90 con 0x7f69580fd150 2024-04-12T05:44:57.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.215+0000 7f695dbee700 1 -- 172.21.15.113:0/1290705753 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6958107130 con 0x7f69580fd150 2024-04-12T05:44:57.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.215+0000 7f6954ff9700 1 -- 172.21.15.113:0/1290705753 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f694c01a070 con 0x7f69580fd150 2024-04-12T05:44:57.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.215+0000 7f695dbee700 1 -- 172.21.15.113:0/1290705753 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6958107690 con 0x7f69580fd150 2024-04-12T05:44:57.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.216+0000 7f695dbee700 1 -- 172.21.15.113:0/1290705753 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695805f590 con 0x7f69580fd150 2024-04-12T05:44:57.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.216+0000 7f6954ff9700 1 -- 172.21.15.113:0/1290705753 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f694c01fc20 con 0x7f69580fd150 2024-04-12T05:44:57.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.217+0000 7f6954ff9700 1 --2- 172.21.15.113:0/1290705753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694006cab0 0x7f694006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:57.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.217+0000 7f6954ff9700 1 -- 172.21.15.113:0/1290705753 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f694c08f620 con 0x7f69580fd150 2024-04-12T05:44:57.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.217+0000 7f69577fe700 1 --2- 172.21.15.113:0/1290705753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694006cab0 0x7f694006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:57.215 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.217+0000 7f69577fe700 1 --2- 172.21.15.113:0/1290705753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694006cab0 0x7f694006ef70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f6948004dc0 tx=0x7f694800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:57.217 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.221+0000 7f6954ff9700 1 -- 172.21.15.113:0/1290705753 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f694c05d0d0 con 0x7f69580fd150 2024-04-12T05:44:57.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:57 smithi113 ceph-mon[103371]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:57.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:57 smithi113 ceph-mon[103371]: pgmap v984: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:57.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:57 smithi113 ceph-mon[103371]: from='client.35770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:57.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:57 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1834846262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:57.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:57 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3433605575' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:44:57.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.426+0000 7f695dbee700 1 -- 172.21.15.113:0/1290705753 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f69580feb40 con 0x7f694006cab0 2024-04-12T05:44:57.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.427+0000 7f6954ff9700 1 -- 172.21.15.113:0/1290705753 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f69580feb40 con 0x7f694006cab0 2024-04-12T05:44:57.423 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:44:57.423 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:44:57.423 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:44:57.423 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:44:57.424 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:44:57.424 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:44:57.424 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:44:57.424 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:44:57.424 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:44:57.424 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:44:57.424 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:44:57.424 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:44:57.424 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:44:57.424 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:44:57.424 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:44:57.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.429+0000 7f695dbee700 1 -- 172.21.15.113:0/1290705753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694006cab0 msgr2=0x7f694006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:57.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.429+0000 7f695dbee700 1 --2- 172.21.15.113:0/1290705753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694006cab0 0x7f694006ef70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f6948004dc0 tx=0x7f694800b040).stop 2024-04-12T05:44:57.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.429+0000 7f695dbee700 1 -- 172.21.15.113:0/1290705753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69580fd150 msgr2=0x7f695810d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:57.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.429+0000 7f695dbee700 1 --2- 172.21.15.113:0/1290705753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69580fd150 0x7f695810d3c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7f694c004f10 tx=0x7f694c009cb0).stop 2024-04-12T05:44:57.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.429+0000 7f695dbee700 1 -- 172.21.15.113:0/1290705753 shutdown_connections 2024-04-12T05:44:57.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.429+0000 7f695dbee700 1 --2- 172.21.15.113:0/1290705753 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694006cab0 0x7f694006ef70 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.429+0000 7f695dbee700 1 --2- 172.21.15.113:0/1290705753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69580fc7f0 0x7f695810ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.429+0000 7f695dbee700 1 --2- 172.21.15.113:0/1290705753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69580fd150 0x7f695810d3c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.429+0000 7f695dbee700 1 -- 172.21.15.113:0/1290705753 >> 172.21.15.113:0/1290705753 conn(0x7f6958070590 msgr2=0x7f69580fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:57.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.429+0000 7f695dbee700 1 -- 172.21.15.113:0/1290705753 shutdown_connections 2024-04-12T05:44:57.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.429+0000 7f695dbee700 1 -- 172.21.15.113:0/1290705753 wait complete. 2024-04-12T05:44:57.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:57 smithi192 ceph-mon[85304]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:57.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:57 smithi192 ceph-mon[85304]: pgmap v984: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:57.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:57 smithi192 ceph-mon[85304]: from='client.35770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:57.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:57 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1834846262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:44:57.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:57 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3433605575' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:44:57.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.526+0000 7f939cf6f700 1 -- 172.21.15.113:0/1311742456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93980fc7f0 msgr2=0x7f93980fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.526+0000 7f939cf6f700 1 --2- 172.21.15.113:0/1311742456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93980fc7f0 0x7f93980fcbd0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7f938c0047f0 tx=0x7f938c009cb0).stop 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.526+0000 7f939cf6f700 1 -- 172.21.15.113:0/1311742456 shutdown_connections 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.526+0000 7f939cf6f700 1 --2- 172.21.15.113:0/1311742456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93980fd150 0x7f93980f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.526+0000 7f939cf6f700 1 --2- 172.21.15.113:0/1311742456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93980fc7f0 0x7f93980fcbd0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.526+0000 7f939cf6f700 1 -- 172.21.15.113:0/1311742456 >> 172.21.15.113:0/1311742456 conn(0x7f9398070590 msgr2=0x7f93980f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.526+0000 7f939cf6f700 1 -- 172.21.15.113:0/1311742456 shutdown_connections 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.526+0000 7f939cf6f700 1 -- 172.21.15.113:0/1311742456 wait complete. 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.527+0000 7f939cf6f700 1 Processor -- start 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.527+0000 7f939cf6f700 1 -- start start 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.527+0000 7f939cf6f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93980fc7f0 0x7f939810dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.527+0000 7f939cf6f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93980fd150 0x7f939810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.527+0000 7f939cf6f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939810eae0 con 0x7f93980fd150 2024-04-12T05:44:57.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.527+0000 7f939cf6f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939810ec20 con 0x7f93980fc7f0 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.527+0000 7f9395d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93980fd150 0x7f939810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.527+0000 7f9395d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93980fd150 0x7f939810e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56292/0 (socket says 172.21.15.113:56292) 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.527+0000 7f9395d9b700 1 -- 172.21.15.113:0/1270394530 learned_addr learned my addr 172.21.15.113:0/1270394530 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.528+0000 7f939659c700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93980fc7f0 0x7f939810dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.528+0000 7f9395d9b700 1 -- 172.21.15.113:0/1270394530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93980fc7f0 msgr2=0x7f939810dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.528+0000 7f9395d9b700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93980fc7f0 0x7f939810dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.528+0000 7f9395d9b700 1 -- 172.21.15.113:0/1270394530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c005040 con 0x7f93980fd150 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.528+0000 7f939659c700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93980fc7f0 0x7f939810dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.528+0000 7f9395d9b700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93980fd150 0x7f939810e4e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7f93880044e0 tx=0x7f9388002b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.528+0000 7f93877fe700 1 -- 172.21.15.113:0/1270394530 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9388007b40 con 0x7f93980fd150 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.528+0000 7f939cf6f700 1 -- 172.21.15.113:0/1270394530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93981082b0 con 0x7f93980fd150 2024-04-12T05:44:57.526 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.528+0000 7f93877fe700 1 -- 172.21.15.113:0/1270394530 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f938800f040 con 0x7f93980fd150 2024-04-12T05:44:57.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.528+0000 7f93877fe700 1 -- 172.21.15.113:0/1270394530 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f938800d830 con 0x7f93980fd150 2024-04-12T05:44:57.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.528+0000 7f939cf6f700 1 -- 172.21.15.113:0/1270394530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93981087f0 con 0x7f93980fd150 2024-04-12T05:44:57.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.529+0000 7f939cf6f700 1 -- 172.21.15.113:0/1270394530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939805f590 con 0x7f93980fd150 2024-04-12T05:44:57.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.530+0000 7f93877fe700 1 -- 172.21.15.113:0/1270394530 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9388007ca0 con 0x7f93980fd150 2024-04-12T05:44:57.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.530+0000 7f93877fe700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f938006cab0 0x7f938006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:44:57.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.530+0000 7f93877fe700 1 -- 172.21.15.113:0/1270394530 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f938808baa0 con 0x7f93980fd150 2024-04-12T05:44:57.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.530+0000 7f939659c700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f938006cab0 0x7f938006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:44:57.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.530+0000 7f939659c700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f938006cab0 0x7f938006ef70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f938c004540 tx=0x7f938c009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:44:57.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.534+0000 7f93877fe700 1 -- 172.21.15.113:0/1270394530 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9388059550 con 0x7f93980fd150 2024-04-12T05:44:57.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.799+0000 7f939cf6f700 1 -- 172.21.15.113:0/1270394530 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9398108e90 con 0x7f93980fd150 2024-04-12T05:44:57.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.800+0000 7f93877fe700 1 -- 172.21.15.113:0/1270394530 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f93880590e0 con 0x7f93980fd150 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:44:57.797 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:44:57.798 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:44:57.798 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:44:57.798 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:44:57.798 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:44:57.798 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:44:57.798 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:44:57.798 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:44:57.798 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:44:57.798 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:44:57.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.802+0000 7f939cf6f700 1 -- 172.21.15.113:0/1270394530 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f938006cab0 msgr2=0x7f938006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:57.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.802+0000 7f939cf6f700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f938006cab0 0x7f938006ef70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f938c004540 tx=0x7f938c009cb0).stop 2024-04-12T05:44:57.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.802+0000 7f939cf6f700 1 -- 172.21.15.113:0/1270394530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93980fd150 msgr2=0x7f939810e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:44:57.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.802+0000 7f939cf6f700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93980fd150 0x7f939810e4e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7f93880044e0 tx=0x7f9388002b00).stop 2024-04-12T05:44:57.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.802+0000 7f939cf6f700 1 -- 172.21.15.113:0/1270394530 shutdown_connections 2024-04-12T05:44:57.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.802+0000 7f939cf6f700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f938006cab0 0x7f938006ef70 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.802+0000 7f939cf6f700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93980fc7f0 0x7f939810dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.802+0000 7f939cf6f700 1 --2- 172.21.15.113:0/1270394530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93980fd150 0x7f939810e4e0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:44:57.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.802+0000 7f939cf6f700 1 -- 172.21.15.113:0/1270394530 >> 172.21.15.113:0/1270394530 conn(0x7f9398070590 msgr2=0x7f9398060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:44:57.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.802+0000 7f939cf6f700 1 -- 172.21.15.113:0/1270394530 shutdown_connections 2024-04-12T05:44:57.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:44:57.802+0000 7f939cf6f700 1 -- 172.21.15.113:0/1270394530 wait complete. 2024-04-12T05:44:58.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:58 smithi113 ceph-mon[103371]: from='client.35774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:58.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:58 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1270394530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:44:58.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:58 smithi192 ceph-mon[85304]: from='client.35774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:58.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:58 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1270394530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:44:59.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:59 smithi113 ceph-mon[103371]: from='client.35786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:59.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:44:59 smithi113 ceph-mon[103371]: pgmap v985: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:44:59.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:59 smithi192 ceph-mon[85304]: from='client.35786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:44:59.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:44:59 smithi192 ceph-mon[85304]: pgmap v985: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:01.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:01 smithi113 ceph-mon[103371]: pgmap v986: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:01.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:01 smithi192 ceph-mon[85304]: pgmap v986: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:03.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:03 smithi113 ceph-mon[103371]: pgmap v987: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:03.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:03 smithi192 ceph-mon[85304]: pgmap v987: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:05.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:05 smithi113 ceph-mon[103371]: pgmap v988: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:05.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:05 smithi192 ceph-mon[85304]: pgmap v988: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:07.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:07 smithi113 ceph-mon[103371]: pgmap v989: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:07.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:07.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:07 smithi192 ceph-mon[85304]: pgmap v989: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:07.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:09.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:09 smithi113 ceph-mon[103371]: pgmap v990: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:09.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:09 smithi192 ceph-mon[85304]: pgmap v990: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:11.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:11 smithi113 ceph-mon[103371]: pgmap v991: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:11.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:11 smithi192 ceph-mon[85304]: pgmap v991: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:13.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:13 smithi113 ceph-mon[103371]: pgmap v992: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:13.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:13 smithi192 ceph-mon[85304]: pgmap v992: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:15.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:15 smithi113 ceph-mon[103371]: pgmap v993: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:15.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:15 smithi192 ceph-mon[85304]: pgmap v993: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:17.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:17 smithi113 ceph-mon[103371]: pgmap v994: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:17.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:17 smithi192 ceph-mon[85304]: pgmap v994: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:19.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:19 smithi113 ceph-mon[103371]: pgmap v995: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:19.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:19 smithi192 ceph-mon[85304]: pgmap v995: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:21.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:21 smithi113 ceph-mon[103371]: pgmap v996: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:21.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:21 smithi192 ceph-mon[85304]: pgmap v996: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:22.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:22.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:23.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:23 smithi113 ceph-mon[103371]: pgmap v997: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:23.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:23 smithi192 ceph-mon[85304]: pgmap v997: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:25.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:25 smithi113 ceph-mon[103371]: pgmap v998: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:25.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:25 smithi192 ceph-mon[85304]: pgmap v998: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:27.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:27 smithi113 ceph-mon[103371]: pgmap v999: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:27.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:27 smithi192 ceph-mon[85304]: pgmap v999: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:27.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.914+0000 7ff2dba0f700 1 -- 172.21.15.113:0/4082186234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2d40f9200 msgr2=0x7ff2d4101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:27.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.914+0000 7ff2dba0f700 1 --2- 172.21.15.113:0/4082186234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2d40f9200 0x7ff2d4101910 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7ff2d00047f0 tx=0x7ff2d0009cb0).stop 2024-04-12T05:45:27.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.914+0000 7ff2dba0f700 1 -- 172.21.15.113:0/4082186234 shutdown_connections 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.914+0000 7ff2dba0f700 1 --2- 172.21.15.113:0/4082186234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2d40f9200 0x7ff2d4101910 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.914+0000 7ff2dba0f700 1 --2- 172.21.15.113:0/4082186234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d40f88a0 0x7ff2d40f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.914+0000 7ff2dba0f700 1 -- 172.21.15.113:0/4082186234 >> 172.21.15.113:0/4082186234 conn(0x7ff2d40f23b0 msgr2=0x7ff2d40f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.914+0000 7ff2dba0f700 1 -- 172.21.15.113:0/4082186234 shutdown_connections 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.914+0000 7ff2dba0f700 1 -- 172.21.15.113:0/4082186234 wait complete. 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2dba0f700 1 Processor -- start 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2dba0f700 1 -- start start 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2dba0f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d40f88a0 0x7ff2d4110390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2dba0f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2d40f9200 0x7ff2d4110880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2dba0f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d4110e80 con 0x7ff2d40f9200 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2dba0f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d4110fc0 con 0x7ff2d40f88a0 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2d8faa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2d40f9200 0x7ff2d4110880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2d8faa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2d40f9200 0x7ff2d4110880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54744/0 (socket says 172.21.15.113:54744) 2024-04-12T05:45:27.913 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2d8faa700 1 -- 172.21.15.113:0/2770341450 learned_addr learned my addr 172.21.15.113:0/2770341450 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:45:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2d97ab700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d40f88a0 0x7ff2d4110390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2d8faa700 1 -- 172.21.15.113:0/2770341450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d40f88a0 msgr2=0x7ff2d4110390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2d8faa700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d40f88a0 0x7ff2d4110390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2d8faa700 1 -- 172.21.15.113:0/2770341450 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d0005040 con 0x7ff2d40f9200 2024-04-12T05:45:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.915+0000 7ff2d97ab700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d40f88a0 0x7ff2d4110390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:45:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.916+0000 7ff2d8faa700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2d40f9200 0x7ff2d4110880 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7ff2d0009380 tx=0x7ff2d0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.916+0000 7ff2c67fc700 1 -- 172.21.15.113:0/2770341450 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2d0018070 con 0x7ff2d40f9200 2024-04-12T05:45:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.916+0000 7ff2c67fc700 1 -- 172.21.15.113:0/2770341450 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2d0007e90 con 0x7ff2d40f9200 2024-04-12T05:45:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.916+0000 7ff2dba0f700 1 -- 172.21.15.113:0/2770341450 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d410a5f0 con 0x7ff2d40f9200 2024-04-12T05:45:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.916+0000 7ff2c67fc700 1 -- 172.21.15.113:0/2770341450 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2d00123c0 con 0x7ff2d40f9200 2024-04-12T05:45:27.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.916+0000 7ff2dba0f700 1 -- 172.21.15.113:0/2770341450 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d410aad0 con 0x7ff2d40f9200 2024-04-12T05:45:27.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.917+0000 7ff2dba0f700 1 -- 172.21.15.113:0/2770341450 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d4060370 con 0x7ff2d40f9200 2024-04-12T05:45:27.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.917+0000 7ff2c67fc700 1 -- 172.21.15.113:0/2770341450 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff2d001d420 con 0x7ff2d40f9200 2024-04-12T05:45:27.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.917+0000 7ff2c67fc700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff2c006caa0 0x7ff2c006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:27.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.918+0000 7ff2c67fc700 1 -- 172.21.15.113:0/2770341450 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff2d008cad0 con 0x7ff2d40f9200 2024-04-12T05:45:27.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.918+0000 7ff2d97ab700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff2c006caa0 0x7ff2c006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:27.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.918+0000 7ff2d97ab700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff2c006caa0 0x7ff2c006ef60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7ff2c8007fd0 tx=0x7ff2c800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:27.918 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:27.922+0000 7ff2c67fc700 1 -- 172.21.15.113:0/2770341450 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff2d005a580 con 0x7ff2d40f9200 2024-04-12T05:45:28.123 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.127+0000 7ff2dba0f700 1 -- 172.21.15.113:0/2770341450 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2d410bab0 con 0x7ff2c006caa0 2024-04-12T05:45:28.125 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.128+0000 7ff2c67fc700 1 -- 172.21.15.113:0/2770341450 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff2d410bab0 con 0x7ff2c006caa0 2024-04-12T05:45:28.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.130+0000 7ff2dba0f700 1 -- 172.21.15.113:0/2770341450 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff2c006caa0 msgr2=0x7ff2c006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.130+0000 7ff2dba0f700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff2c006caa0 0x7ff2c006ef60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7ff2c8007fd0 tx=0x7ff2c800b040).stop 2024-04-12T05:45:28.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.130+0000 7ff2dba0f700 1 -- 172.21.15.113:0/2770341450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2d40f9200 msgr2=0x7ff2d4110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.130+0000 7ff2dba0f700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2d40f9200 0x7ff2d4110880 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7ff2d0009380 tx=0x7ff2d0009cb0).stop 2024-04-12T05:45:28.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.131+0000 7ff2dba0f700 1 -- 172.21.15.113:0/2770341450 shutdown_connections 2024-04-12T05:45:28.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.131+0000 7ff2dba0f700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff2c006caa0 0x7ff2c006ef60 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.131+0000 7ff2dba0f700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2d40f88a0 0x7ff2d4110390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.127 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.131+0000 7ff2dba0f700 1 --2- 172.21.15.113:0/2770341450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2d40f9200 0x7ff2d4110880 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.131+0000 7ff2dba0f700 1 -- 172.21.15.113:0/2770341450 >> 172.21.15.113:0/2770341450 conn(0x7ff2d40f23b0 msgr2=0x7ff2d40f6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:28.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.131+0000 7ff2dba0f700 1 -- 172.21.15.113:0/2770341450 shutdown_connections 2024-04-12T05:45:28.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.131+0000 7ff2dba0f700 1 -- 172.21.15.113:0/2770341450 wait complete. 2024-04-12T05:45:28.136 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:45:28.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.230+0000 7f7fca961700 1 -- 172.21.15.113:0/3628233088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7fc40fd150 msgr2=0x7f7fc40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.230+0000 7f7fca961700 1 --2- 172.21.15.113:0/3628233088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7fc40fd150 0x7f7fc40f6930 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f7fb80047f0 tx=0x7f7fb8009cb0).stop 2024-04-12T05:45:28.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.230+0000 7f7fca961700 1 -- 172.21.15.113:0/3628233088 shutdown_connections 2024-04-12T05:45:28.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.230+0000 7f7fca961700 1 --2- 172.21.15.113:0/3628233088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7fc40fd150 0x7f7fc40f6930 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.230+0000 7f7fca961700 1 --2- 172.21.15.113:0/3628233088 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7fc40fc7f0 0x7f7fc40fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:45:28.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.230+0000 7f7fca961700 1 -- 172.21.15.113:0/3628233088 >> 172.21.15.113:0/3628233088 conn(0x7f7fc4070590 msgr2=0x7f7fc40f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:28.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.230+0000 7f7fca961700 1 -- 172.21.15.113:0/3628233088 shutdown_connections 2024-04-12T05:45:28.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.230+0000 7f7fca961700 1 -- 172.21.15.113:0/3628233088 wait complete. 2024-04-12T05:45:28.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fca961700 1 Processor -- start 2024-04-12T05:45:28.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fca961700 1 -- start start 2024-04-12T05:45:28.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fca961700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7fc40fc7f0 0x7f7fc4109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:28.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fca961700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7fc40fd150 0x7f7fc4109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:28.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fca961700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc4109be0 con 0x7f7fc40fc7f0 2024-04-12T05:45:28.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fca961700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc4109d20 con 0x7f7fc40fd150 2024-04-12T05:45:28.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fc3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7fc40fc7f0 0x7f7fc4109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:28.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fc3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7fc40fc7f0 0x7f7fc4109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54762/0 (socket says 172.21.15.113:54762) 2024-04-12T05:45:28.229 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fc3fff700 1 -- 172.21.15.113:0/263797995 learned_addr learned my addr 172.21.15.113:0/263797995 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:45:28.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fc37fe700 1 --2- 172.21.15.113:0/263797995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7fc40fd150 0x7f7fc4109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:28.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fc3fff700 1 -- 172.21.15.113:0/263797995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7fc40fd150 msgr2=0x7f7fc4109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fc3fff700 1 --2- 172.21.15.113:0/263797995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7fc40fd150 0x7f7fc4109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.231+0000 7f7fc3fff700 1 -- 172.21.15.113:0/263797995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb8005040 con 0x7f7fc40fc7f0 2024-04-12T05:45:28.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.232+0000 7f7fc3fff700 1 --2- 172.21.15.113:0/263797995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7fc40fc7f0 0x7f7fc4109170 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7f7fb400b4b0 tx=0x7f7fb40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:28.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.232+0000 7f7fc17fa700 1 -- 172.21.15.113:0/263797995 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fb4002500 con 0x7f7fc40fc7f0 2024-04-12T05:45:28.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.232+0000 7f7fc17fa700 1 -- 172.21.15.113:0/263797995 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fb4002b10 con 0x7f7fc40fc7f0 2024-04-12T05:45:28.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.232+0000 7f7fca961700 1 -- 172.21.15.113:0/263797995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc4109fd0 con 0x7f7fc40fc7f0 2024-04-12T05:45:28.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.232+0000 7f7fc17fa700 1 -- 172.21.15.113:0/263797995 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fb4012cf0 con 0x7f7fc40fc7f0 2024-04-12T05:45:28.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.232+0000 7f7fca961700 1 -- 172.21.15.113:0/263797995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc4112560 con 0x7f7fc40fc7f0 2024-04-12T05:45:28.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.233+0000 7f7fca961700 1 -- 172.21.15.113:0/263797995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc405f590 con 0x7f7fc40fc7f0 2024-04-12T05:45:28.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.233+0000 7f7fc17fa700 1 -- 172.21.15.113:0/263797995 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7fb4002660 con 0x7f7fc40fc7f0 2024-04-12T05:45:28.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.234+0000 7f7fc17fa700 1 --2- 172.21.15.113:0/263797995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7fac06caa0 0x7f7fac06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:28.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.234+0000 7f7fc17fa700 1 -- 172.21.15.113:0/263797995 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7fb408bf30 con 0x7f7fc40fc7f0 2024-04-12T05:45:28.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.234+0000 7f7fc37fe700 1 --2- 172.21.15.113:0/263797995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7fac06caa0 0x7f7fac06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:28.231 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.234+0000 7f7fc37fe700 1 --2- 172.21.15.113:0/263797995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7fac06caa0 0x7f7fac06ef60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f7fb8004f10 tx=0x7f7fb800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:28.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.238+0000 7f7fc17fa700 1 -- 172.21.15.113:0/263797995 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7fb40bf2c0 con 0x7f7fc40fc7f0 2024-04-12T05:45:28.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.443+0000 7f7fca961700 1 -- 172.21.15.113:0/263797995 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fc4059120 con 0x7f7fac06caa0 2024-04-12T05:45:28.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.444+0000 7f7fc17fa700 1 -- 172.21.15.113:0/263797995 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7fc4059120 con 0x7f7fac06caa0 2024-04-12T05:45:28.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.446+0000 7f7fca961700 1 -- 172.21.15.113:0/263797995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7fac06caa0 msgr2=0x7f7fac06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.446+0000 7f7fca961700 1 --2- 172.21.15.113:0/263797995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7fac06caa0 0x7f7fac06ef60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f7fb8004f10 tx=0x7f7fb800b040).stop 2024-04-12T05:45:28.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.446+0000 7f7fca961700 1 -- 172.21.15.113:0/263797995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7fc40fc7f0 msgr2=0x7f7fc4109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.446+0000 7f7fca961700 1 --2- 172.21.15.113:0/263797995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7fc40fc7f0 0x7f7fc4109170 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7f7fb400b4b0 tx=0x7f7fb40090f0).stop 2024-04-12T05:45:28.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.446+0000 7f7fca961700 1 -- 172.21.15.113:0/263797995 shutdown_connections 2024-04-12T05:45:28.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.446+0000 7f7fca961700 1 --2- 172.21.15.113:0/263797995 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7fac06caa0 0x7f7fac06ef60 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.446+0000 7f7fca961700 1 --2- 172.21.15.113:0/263797995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7fc40fc7f0 0x7f7fc4109170 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.446+0000 7f7fca961700 1 --2- 172.21.15.113:0/263797995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7fc40fd150 0x7f7fc4109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.446+0000 7f7fca961700 1 -- 172.21.15.113:0/263797995 >> 172.21.15.113:0/263797995 conn(0x7f7fc4070590 msgr2=0x7f7fc40fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:28.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.446+0000 7f7fca961700 1 -- 172.21.15.113:0/263797995 shutdown_connections 2024-04-12T05:45:28.443 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.446+0000 7f7fca961700 1 -- 172.21.15.113:0/263797995 wait complete. 2024-04-12T05:45:28.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.543+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1874393893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a480fd150 msgr2=0x7f7a480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.543+0000 7f7a4f60f700 1 --2- 172.21.15.113:0/1874393893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a480fd150 0x7f7a480f6930 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f7a380047f0 tx=0x7f7a38009cb0).stop 2024-04-12T05:45:28.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.543+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1874393893 shutdown_connections 2024-04-12T05:45:28.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.543+0000 7f7a4f60f700 1 --2- 172.21.15.113:0/1874393893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a480fd150 0x7f7a480f6930 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.543+0000 7f7a4f60f700 1 --2- 172.21.15.113:0/1874393893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a480fc7f0 0x7f7a480fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.543+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1874393893 >> 172.21.15.113:0/1874393893 conn(0x7f7a48070590 msgr2=0x7f7a480f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:28.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.543+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1874393893 shutdown_connections 2024-04-12T05:45:28.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.543+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1874393893 wait complete. 2024-04-12T05:45:28.542 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.544+0000 7f7a4f60f700 1 Processor -- start 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.544+0000 7f7a4f60f700 1 -- start start 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.544+0000 7f7a4f60f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a480fc7f0 0x7f7a4810e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.544+0000 7f7a4f60f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a480fd150 0x7f7a4810e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.544+0000 7f7a4f60f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4810ec20 con 0x7f7a480fc7f0 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.544+0000 7f7a4f60f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4810ed60 con 0x7f7a480fd150 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.544+0000 7f7a4d3ab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a480fc7f0 0x7f7a4810e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.544+0000 7f7a4d3ab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a480fc7f0 0x7f7a4810e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54784/0 (socket says 172.21.15.113:54784) 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.544+0000 7f7a4d3ab700 1 -- 172.21.15.113:0/1995993697 learned_addr learned my addr 172.21.15.113:0/1995993697 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.545+0000 7f7a4cbaa700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a480fd150 0x7f7a4810e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.545+0000 7f7a4d3ab700 1 -- 172.21.15.113:0/1995993697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a480fd150 msgr2=0x7f7a4810e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.545+0000 7f7a4d3ab700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a480fd150 0x7f7a4810e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.545+0000 7f7a4d3ab700 1 -- 172.21.15.113:0/1995993697 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a38005040 con 0x7f7a480fc7f0 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.545+0000 7f7a4cbaa700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a480fd150 0x7f7a4810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:45:28.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.545+0000 7f7a4d3ab700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a480fc7f0 0x7f7a4810e130 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7f7a4400e910 tx=0x7f7a4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:28.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.545+0000 7f7a3e7fc700 1 -- 172.21.15.113:0/1995993697 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a4400eeb0 con 0x7f7a480fc7f0 2024-04-12T05:45:28.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.545+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1995993697 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a481083e0 con 0x7f7a480fc7f0 2024-04-12T05:45:28.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.545+0000 7f7a3e7fc700 1 -- 172.21.15.113:0/1995993697 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a4400f040 con 0x7f7a480fc7f0 2024-04-12T05:45:28.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.545+0000 7f7a3e7fc700 1 -- 172.21.15.113:0/1995993697 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a4400c8a0 con 0x7f7a480fc7f0 2024-04-12T05:45:28.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.545+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1995993697 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a481089a0 con 0x7f7a480fc7f0 2024-04-12T05:45:28.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.546+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1995993697 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4805f590 con 0x7f7a480fc7f0 2024-04-12T05:45:28.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.546+0000 7f7a3e7fc700 1 -- 172.21.15.113:0/1995993697 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7a440090d0 con 0x7f7a480fc7f0 2024-04-12T05:45:28.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.547+0000 7f7a3e7fc700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7a3406caa0 0x7f7a3406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:28.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.547+0000 7f7a3e7fc700 1 -- 172.21.15.113:0/1995993697 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7a4408b920 con 0x7f7a480fc7f0 2024-04-12T05:45:28.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.547+0000 7f7a4cbaa700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7a3406caa0 0x7f7a3406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:28.545 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.547+0000 7f7a4cbaa700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7a3406caa0 0x7f7a3406ef60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f7a38004e60 tx=0x7f7a3800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:28.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.551+0000 7f7a3e7fc700 1 -- 172.21.15.113:0/1995993697 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7a440593d0 con 0x7f7a480fc7f0 2024-04-12T05:45:28.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.750+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1995993697 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a48108fe0 con 0x7f7a3406caa0 2024-04-12T05:45:28.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.760+0000 7f7a3e7fc700 1 -- 172.21.15.113:0/1995993697 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7a48108fe0 con 0x7f7a3406caa0 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (33m) 10m ago 44m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (30m) 10m ago 44m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (30m) 10m ago 43m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (33m) 10m ago 44m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (26m) 10m ago 39m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (26m) 10m ago 39m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (26m) 10m ago 39m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (10m) 10m ago 39m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (34m) 10m ago 45m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (34m) 10m ago 43m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (31m) 10m ago 45m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:45:28.757 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (31m) 10m ago 43m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:45:28.758 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (33m) 10m ago 44m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:45:28.758 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (33m) 10m ago 43m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:45:28.758 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (30m) 10m ago 43m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:45:28.758 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (29m) 10m ago 42m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:45:28.758 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (29m) 10m ago 42m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:45:28.758 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (28m) 10m ago 41m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:45:28.758 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (27m) 10m ago 41m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:45:28.758 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (26m) 10m ago 40m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:45:28.758 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (33m) 10m ago 43m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:45:28.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.762+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1995993697 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7a3406caa0 msgr2=0x7f7a3406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.762+0000 7f7a4f60f700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7a3406caa0 0x7f7a3406ef60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f7a38004e60 tx=0x7f7a3800b040).stop 2024-04-12T05:45:28.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.762+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1995993697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a480fc7f0 msgr2=0x7f7a4810e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.762+0000 7f7a4f60f700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a480fc7f0 0x7f7a4810e130 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7f7a4400e910 tx=0x7f7a4400b040).stop 2024-04-12T05:45:28.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.762+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1995993697 shutdown_connections 2024-04-12T05:45:28.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.762+0000 7f7a4f60f700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7a3406caa0 0x7f7a3406ef60 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.762+0000 7f7a4f60f700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a480fc7f0 0x7f7a4810e130 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.762+0000 7f7a4f60f700 1 --2- 172.21.15.113:0/1995993697 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a480fd150 0x7f7a4810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.762+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1995993697 >> 172.21.15.113:0/1995993697 conn(0x7f7a48070590 msgr2=0x7f7a48060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.763+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1995993697 shutdown_connections 2024-04-12T05:45:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.763+0000 7f7a4f60f700 1 -- 172.21.15.113:0/1995993697 wait complete. 2024-04-12T05:45:28.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.859+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1100508605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64cc0f6710 msgr2=0x7f64cc0f6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.859+0000 7f64d0ee2700 1 --2- 172.21.15.113:0/1100508605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64cc0f6710 0x7f64cc0f6af0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7f64c00047f0 tx=0x7f64c0009cb0).stop 2024-04-12T05:45:28.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.859+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1100508605 shutdown_connections 2024-04-12T05:45:28.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.859+0000 7f64d0ee2700 1 --2- 172.21.15.113:0/1100508605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64cc0f6fe0 0x7f64cc103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.859+0000 7f64d0ee2700 1 --2- 172.21.15.113:0/1100508605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64cc0f6710 0x7f64cc0f6af0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.859+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1100508605 >> 172.21.15.113:0/1100508605 conn(0x7f64cc0f23b0 msgr2=0x7f64cc0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:28.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.859+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1100508605 shutdown_connections 2024-04-12T05:45:28.856 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.859+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1100508605 wait complete. 2024-04-12T05:45:28.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64d0ee2700 1 Processor -- start 2024-04-12T05:45:28.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64d0ee2700 1 -- start start 2024-04-12T05:45:28.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64d0ee2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64cc0f6710 0x7f64cc1102e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:28.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64d0ee2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64cc0f6fe0 0x7f64cc1107d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:28.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64d0ee2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc110dd0 con 0x7f64cc0f6710 2024-04-12T05:45:28.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64d0ee2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc110f10 con 0x7f64cc0f6fe0 2024-04-12T05:45:28.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64ca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64cc0f6710 0x7f64cc1102e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:28.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64ca59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64cc0f6710 0x7f64cc1102e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54808/0 (socket says 172.21.15.113:54808) 2024-04-12T05:45:28.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64ca59c700 1 -- 172.21.15.113:0/1177295455 learned_addr learned my addr 172.21.15.113:0/1177295455 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:45:28.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64c9d9b700 1 --2- 172.21.15.113:0/1177295455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64cc0f6fe0 0x7f64cc1107d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:28.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64ca59c700 1 -- 172.21.15.113:0/1177295455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64cc0f6fe0 msgr2=0x7f64cc1107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:28.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64ca59c700 1 --2- 172.21.15.113:0/1177295455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64cc0f6fe0 0x7f64cc1107d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:28.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.860+0000 7f64ca59c700 1 -- 172.21.15.113:0/1177295455 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c0005040 con 0x7f64cc0f6710 2024-04-12T05:45:28.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.861+0000 7f64ca59c700 1 --2- 172.21.15.113:0/1177295455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64cc0f6710 0x7f64cc1102e0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f64c00047c0 tx=0x7f64c0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:28.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.861+0000 7f64bb7fe700 1 -- 172.21.15.113:0/1177295455 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64c000e030 con 0x7f64cc0f6710 2024-04-12T05:45:28.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.861+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1177295455 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc10a540 con 0x7f64cc0f6710 2024-04-12T05:45:28.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.861+0000 7f64bb7fe700 1 -- 172.21.15.113:0/1177295455 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64c0007e90 con 0x7f64cc0f6710 2024-04-12T05:45:28.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.861+0000 7f64bb7fe700 1 -- 172.21.15.113:0/1177295455 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64c0018070 con 0x7f64cc0f6710 2024-04-12T05:45:28.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.861+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1177295455 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc10aa20 con 0x7f64cc0f6710 2024-04-12T05:45:28.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.862+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1177295455 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc060370 con 0x7f64cc0f6710 2024-04-12T05:45:28.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.862+0000 7f64bb7fe700 1 -- 172.21.15.113:0/1177295455 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f64c001d420 con 0x7f64cc0f6710 2024-04-12T05:45:28.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.863+0000 7f64bb7fe700 1 --2- 172.21.15.113:0/1177295455 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64b406ca30 0x7f64b406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:28.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.863+0000 7f64bb7fe700 1 -- 172.21.15.113:0/1177295455 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f64c008cff0 con 0x7f64cc0f6710 2024-04-12T05:45:28.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.863+0000 7f64c9d9b700 1 --2- 172.21.15.113:0/1177295455 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64b406ca30 0x7f64b406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:28.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.863+0000 7f64c9d9b700 1 --2- 172.21.15.113:0/1177295455 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64b406ca30 0x7f64b406eef0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f64cc10bc20 tx=0x7f64bc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:28.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:28.867+0000 7f64bb7fe700 1 -- 172.21.15.113:0/1177295455 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f64c005aaa0 con 0x7f64cc0f6710 2024-04-12T05:45:29.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.134+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1177295455 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f64cc10ba00 con 0x7f64cc0f6710 2024-04-12T05:45:29.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.135+0000 7f64bb7fe700 1 -- 172.21.15.113:0/1177295455 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f64c001d6d0 con 0x7f64cc0f6710 2024-04-12T05:45:29.131 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:45:29.131 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:45:29.131 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:45:29.131 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:45:29.131 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:45:29.131 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:45:29.132 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:45:29.132 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:45:29.132 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:45:29.132 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:45:29.132 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:45:29.132 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:45:29.132 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:45:29.132 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:45:29.132 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:45:29.132 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:45:29.132 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:45:29.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.136+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1177295455 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64b406ca30 msgr2=0x7f64b406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.136+0000 7f64d0ee2700 1 --2- 172.21.15.113:0/1177295455 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64b406ca30 0x7f64b406eef0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f64cc10bc20 tx=0x7f64bc00b040).stop 2024-04-12T05:45:29.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.137+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1177295455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64cc0f6710 msgr2=0x7f64cc1102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.137+0000 7f64d0ee2700 1 --2- 172.21.15.113:0/1177295455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64cc0f6710 0x7f64cc1102e0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f64c00047c0 tx=0x7f64c0009cb0).stop 2024-04-12T05:45:29.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.137+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1177295455 shutdown_connections 2024-04-12T05:45:29.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.137+0000 7f64d0ee2700 1 --2- 172.21.15.113:0/1177295455 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f64b406ca30 0x7f64b406eef0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.137+0000 7f64d0ee2700 1 --2- 172.21.15.113:0/1177295455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64cc0f6710 0x7f64cc1102e0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.137+0000 7f64d0ee2700 1 --2- 172.21.15.113:0/1177295455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64cc0f6fe0 0x7f64cc1107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.137+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1177295455 >> 172.21.15.113:0/1177295455 conn(0x7f64cc0f23b0 msgr2=0x7f64cc0f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:29.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.137+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1177295455 shutdown_connections 2024-04-12T05:45:29.134 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.137+0000 7f64d0ee2700 1 -- 172.21.15.113:0/1177295455 wait complete. 2024-04-12T05:45:29.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.235+0000 7fc23c2a0700 1 -- 172.21.15.113:0/756700296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2340fc7f0 msgr2=0x7fc2340fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.235+0000 7fc23c2a0700 1 --2- 172.21.15.113:0/756700296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2340fc7f0 0x7fc2340fcbd0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7fc2240047f0 tx=0x7fc224009cb0).stop 2024-04-12T05:45:29.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.235+0000 7fc23c2a0700 1 -- 172.21.15.113:0/756700296 shutdown_connections 2024-04-12T05:45:29.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.235+0000 7fc23c2a0700 1 --2- 172.21.15.113:0/756700296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2340fd150 0x7fc2340f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.235+0000 7fc23c2a0700 1 --2- 172.21.15.113:0/756700296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2340fc7f0 0x7fc2340fcbd0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.235+0000 7fc23c2a0700 1 -- 172.21.15.113:0/756700296 >> 172.21.15.113:0/756700296 conn(0x7fc234070590 msgr2=0x7fc2340f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:29.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.235+0000 7fc23c2a0700 1 -- 172.21.15.113:0/756700296 shutdown_connections 2024-04-12T05:45:29.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.235+0000 7fc23c2a0700 1 -- 172.21.15.113:0/756700296 wait complete. 2024-04-12T05:45:29.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23c2a0700 1 Processor -- start 2024-04-12T05:45:29.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23c2a0700 1 -- start start 2024-04-12T05:45:29.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23c2a0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2340fc7f0 0x7fc2341101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23c2a0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2340fd150 0x7fc2341106c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23c2a0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc234110cc0 con 0x7fc2340fc7f0 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23c2a0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc234110e00 con 0x7fc2340fd150 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23a03c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2340fc7f0 0x7fc2341101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23a03c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2340fc7f0 0x7fc2341101d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54832/0 (socket says 172.21.15.113:54832) 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23a03c700 1 -- 172.21.15.113:0/876396223 learned_addr learned my addr 172.21.15.113:0/876396223 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23a03c700 1 -- 172.21.15.113:0/876396223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2340fd150 msgr2=0x7fc2341106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23a03c700 1 --2- 172.21.15.113:0/876396223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2340fd150 0x7fc2341106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23a03c700 1 -- 172.21.15.113:0/876396223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc224005040 con 0x7fc2340fc7f0 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23a03c700 1 --2- 172.21.15.113:0/876396223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2340fc7f0 0x7fc2341101d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fc224005010 tx=0x7fc224009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc22b779700 1 -- 172.21.15.113:0/876396223 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc22400e030 con 0x7fc2340fc7f0 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc22b779700 1 -- 172.21.15.113:0/876396223 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc224007e90 con 0x7fc2340fc7f0 2024-04-12T05:45:29.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23c2a0700 1 -- 172.21.15.113:0/876396223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23410a430 con 0x7fc2340fc7f0 2024-04-12T05:45:29.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc22b779700 1 -- 172.21.15.113:0/876396223 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc224018070 con 0x7fc2340fc7f0 2024-04-12T05:45:29.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.236+0000 7fc23c2a0700 1 -- 172.21.15.113:0/876396223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23410a910 con 0x7fc2340fc7f0 2024-04-12T05:45:29.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.237+0000 7fc23c2a0700 1 -- 172.21.15.113:0/876396223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23405f590 con 0x7fc2340fc7f0 2024-04-12T05:45:29.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.238+0000 7fc22b779700 1 -- 172.21.15.113:0/876396223 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc22401d6e0 con 0x7fc2340fc7f0 2024-04-12T05:45:29.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.238+0000 7fc22b779700 1 --2- 172.21.15.113:0/876396223 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 0x7fc22006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:29.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.238+0000 7fc22b779700 1 -- 172.21.15.113:0/876396223 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc22408d0d0 con 0x7fc2340fc7f0 2024-04-12T05:45:29.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.238+0000 7fc23983b700 1 --2- 172.21.15.113:0/876396223 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 0x7fc22006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:29.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.239+0000 7fc23983b700 1 --2- 172.21.15.113:0/876396223 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 0x7fc22006ef60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7fc230009fd0 tx=0x7fc230009be0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:29.239 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.243+0000 7fc22b779700 1 -- 172.21.15.113:0/876396223 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc22405a8f0 con 0x7fc2340fc7f0 2024-04-12T05:45:29.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:29 smithi192 ceph-mon[85304]: pgmap v1000: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:29.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:29 smithi192 ceph-mon[85304]: from='client.35794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:29.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:29 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1177295455' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:29.468 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.472+0000 7fc23c2a0700 1 -- 172.21.15.113:0/876396223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc23410b060 con 0x7fc2340fc7f0 2024-04-12T05:45:29.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.473+0000 7fc22b779700 1 -- 172.21.15.113:0/876396223 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fc22405a710 con 0x7fc2340fc7f0 2024-04-12T05:45:29.469 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:45:29.469 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:45:29.469 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:45:29.469 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:45:29.469 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:45:29.469 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:45:29.469 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:45:29.470 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:45:29.471 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:45:29.472 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:45:29.472 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:45:29.472 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:45:29.472 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:45:29.472 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:45:29.472 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:29.472 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:29.472 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:29.472 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:45:29.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.475+0000 7fc23c2a0700 1 -- 172.21.15.113:0/876396223 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 msgr2=0x7fc22006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.475+0000 7fc23c2a0700 1 --2- 172.21.15.113:0/876396223 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 0x7fc22006ef60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7fc230009fd0 tx=0x7fc230009be0).stop 2024-04-12T05:45:29.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.475+0000 7fc23c2a0700 1 -- 172.21.15.113:0/876396223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2340fc7f0 msgr2=0x7fc2341101d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.475+0000 7fc23c2a0700 1 --2- 172.21.15.113:0/876396223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2340fc7f0 0x7fc2341101d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fc224005010 tx=0x7fc224009cb0).stop 2024-04-12T05:45:29.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.475+0000 7fc23c2a0700 1 -- 172.21.15.113:0/876396223 shutdown_connections 2024-04-12T05:45:29.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.475+0000 7fc23c2a0700 1 --2- 172.21.15.113:0/876396223 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc22006caa0 0x7fc22006ef60 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.475+0000 7fc23c2a0700 1 --2- 172.21.15.113:0/876396223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2340fc7f0 0x7fc2341101d0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.475+0000 7fc23c2a0700 1 --2- 172.21.15.113:0/876396223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2340fd150 0x7fc2341106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:45:29.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.475+0000 7fc23c2a0700 1 -- 172.21.15.113:0/876396223 >> 172.21.15.113:0/876396223 conn(0x7fc234070590 msgr2=0x7fc234060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:29.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.475+0000 7fc23c2a0700 1 -- 172.21.15.113:0/876396223 shutdown_connections 2024-04-12T05:45:29.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.475+0000 7fc23c2a0700 1 -- 172.21.15.113:0/876396223 wait complete. 2024-04-12T05:45:29.474 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:45:29.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.572+0000 7fdd6ee86700 1 -- 172.21.15.113:0/1091471352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd680fd150 msgr2=0x7fdd680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.573+0000 7fdd6ee86700 1 --2- 172.21.15.113:0/1091471352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd680fd150 0x7fdd680f6930 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7fdd540047f0 tx=0x7fdd54009cb0).stop 2024-04-12T05:45:29.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.573+0000 7fdd6ee86700 1 -- 172.21.15.113:0/1091471352 shutdown_connections 2024-04-12T05:45:29.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.573+0000 7fdd6ee86700 1 --2- 172.21.15.113:0/1091471352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd680fd150 0x7fdd680f6930 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.573+0000 7fdd6ee86700 1 --2- 172.21.15.113:0/1091471352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd680fc7f0 0x7fdd680fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.573+0000 7fdd6ee86700 1 -- 172.21.15.113:0/1091471352 >> 172.21.15.113:0/1091471352 conn(0x7fdd68070590 msgr2=0x7fdd680f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:29.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.573+0000 7fdd6ee86700 1 -- 172.21.15.113:0/1091471352 shutdown_connections 2024-04-12T05:45:29.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.573+0000 7fdd6ee86700 1 -- 172.21.15.113:0/1091471352 wait complete. 2024-04-12T05:45:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.573+0000 7fdd6ee86700 1 Processor -- start 2024-04-12T05:45:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.573+0000 7fdd6ee86700 1 -- start start 2024-04-12T05:45:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd6ee86700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd680fc7f0 0x7fdd6810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd6ee86700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd680fd150 0x7fdd6810d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd6ee86700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6810d9c0 con 0x7fdd680fd150 2024-04-12T05:45:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd6ee86700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6810db00 con 0x7fdd680fc7f0 2024-04-12T05:45:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd67fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd680fd150 0x7fdd6810d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd67fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd680fd150 0x7fdd6810d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54854/0 (socket says 172.21.15.113:54854) 2024-04-12T05:45:29.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd67fff700 1 -- 172.21.15.113:0/82796894 learned_addr learned my addr 172.21.15.113:0/82796894 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:45:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd6cc22700 1 --2- 172.21.15.113:0/82796894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd680fc7f0 0x7fdd6810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd67fff700 1 -- 172.21.15.113:0/82796894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd680fc7f0 msgr2=0x7fdd6810ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd67fff700 1 --2- 172.21.15.113:0/82796894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd680fc7f0 0x7fdd6810ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd67fff700 1 -- 172.21.15.113:0/82796894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd54005040 con 0x7fdd680fd150 2024-04-12T05:45:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd67fff700 1 --2- 172.21.15.113:0/82796894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd680fd150 0x7fdd6810d3c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7fdd540099f0 tx=0x7fdd54009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd65ffb700 1 -- 172.21.15.113:0/82796894 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd5400e030 con 0x7fdd680fd150 2024-04-12T05:45:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd6ee86700 1 -- 172.21.15.113:0/82796894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd68107130 con 0x7fdd680fd150 2024-04-12T05:45:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd65ffb700 1 -- 172.21.15.113:0/82796894 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd54007e90 con 0x7fdd680fd150 2024-04-12T05:45:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.574+0000 7fdd65ffb700 1 -- 172.21.15.113:0/82796894 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd54018070 con 0x7fdd680fd150 2024-04-12T05:45:29.574 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.575+0000 7fdd6ee86700 1 -- 172.21.15.113:0/82796894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd68107690 con 0x7fdd680fd150 2024-04-12T05:45:29.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.576+0000 7fdd6ee86700 1 -- 172.21.15.113:0/82796894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6805f590 con 0x7fdd680fd150 2024-04-12T05:45:29.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.576+0000 7fdd65ffb700 1 -- 172.21.15.113:0/82796894 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdd5401d420 con 0x7fdd680fd150 2024-04-12T05:45:29.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.576+0000 7fdd65ffb700 1 --2- 172.21.15.113:0/82796894 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd5806caa0 0x7fdd5806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:29.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.577+0000 7fdd65ffb700 1 -- 172.21.15.113:0/82796894 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fdd5408d2e0 con 0x7fdd680fd150 2024-04-12T05:45:29.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.577+0000 7fdd6cc22700 1 --2- 172.21.15.113:0/82796894 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd5806caa0 0x7fdd5806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:29.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.577+0000 7fdd6cc22700 1 --2- 172.21.15.113:0/82796894 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd5806caa0 0x7fdd5806ef60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fdd5c0041d0 tx=0x7fdd5c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:29.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.581+0000 7fdd65ffb700 1 -- 172.21.15.113:0/82796894 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fdd5405ad90 con 0x7fdd680fd150 2024-04-12T05:45:29.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:29 smithi113 ceph-mon[103371]: pgmap v1000: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:29.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:29 smithi113 ceph-mon[103371]: from='client.35794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:29.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:29 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1177295455' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:45:29.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.784+0000 7fdd6ee86700 1 -- 172.21.15.113:0/82796894 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd68108640 con 0x7fdd5806caa0 2024-04-12T05:45:29.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.785+0000 7fdd65ffb700 1 -- 172.21.15.113:0/82796894 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdd68108640 con 0x7fdd5806caa0 2024-04-12T05:45:29.781 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:45:29.782 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:45:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.787+0000 7fdd6ee86700 1 -- 172.21.15.113:0/82796894 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd5806caa0 msgr2=0x7fdd5806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.787+0000 7fdd6ee86700 1 --2- 172.21.15.113:0/82796894 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd5806caa0 0x7fdd5806ef60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fdd5c0041d0 tx=0x7fdd5c009040).stop 2024-04-12T05:45:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.787+0000 7fdd6ee86700 1 -- 172.21.15.113:0/82796894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd680fd150 msgr2=0x7fdd6810d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.787+0000 7fdd6ee86700 1 --2- 172.21.15.113:0/82796894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd680fd150 0x7fdd6810d3c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7fdd540099f0 tx=0x7fdd54009cb0).stop 2024-04-12T05:45:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.787+0000 7fdd6ee86700 1 -- 172.21.15.113:0/82796894 shutdown_connections 2024-04-12T05:45:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.787+0000 7fdd6ee86700 1 --2- 172.21.15.113:0/82796894 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd5806caa0 0x7fdd5806ef60 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.787+0000 7fdd6ee86700 1 --2- 172.21.15.113:0/82796894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd680fc7f0 0x7fdd6810ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.787+0000 7fdd6ee86700 1 --2- 172.21.15.113:0/82796894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd680fd150 0x7fdd6810d3c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.787+0000 7fdd6ee86700 1 -- 172.21.15.113:0/82796894 >> 172.21.15.113:0/82796894 conn(0x7fdd68070590 msgr2=0x7fdd680fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.787+0000 7fdd6ee86700 1 -- 172.21.15.113:0/82796894 shutdown_connections 2024-04-12T05:45:29.784 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.787+0000 7fdd6ee86700 1 -- 172.21.15.113:0/82796894 wait complete. 2024-04-12T05:45:29.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.885+0000 7f050da8a700 1 -- 172.21.15.113:0/1122991161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05080fe270 msgr2=0x7f05080f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.885+0000 7f050da8a700 1 --2- 172.21.15.113:0/1122991161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05080fe270 0x7f05080f7a50 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f04f80047f0 tx=0x7f04f8009cb0).stop 2024-04-12T05:45:29.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.885+0000 7f050da8a700 1 -- 172.21.15.113:0/1122991161 shutdown_connections 2024-04-12T05:45:29.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.885+0000 7f050da8a700 1 --2- 172.21.15.113:0/1122991161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05080fe270 0x7f05080f7a50 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.885+0000 7f050da8a700 1 --2- 172.21.15.113:0/1122991161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05080fd910 0x7f05080fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.885+0000 7f050da8a700 1 -- 172.21.15.113:0/1122991161 >> 172.21.15.113:0/1122991161 conn(0x7f05080f3530 msgr2=0x7f05080f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:29.882 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.885+0000 7f050da8a700 1 -- 172.21.15.113:0/1122991161 shutdown_connections 2024-04-12T05:45:29.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.885+0000 7f050da8a700 1 -- 172.21.15.113:0/1122991161 wait complete. 2024-04-12T05:45:29.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.886+0000 7f050da8a700 1 Processor -- start 2024-04-12T05:45:29.883 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.886+0000 7f050da8a700 1 -- start start 2024-04-12T05:45:29.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.886+0000 7f050da8a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05080fd910 0x7f050810dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:29.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.886+0000 7f050da8a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05080fe270 0x7f050810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:29.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.886+0000 7f050da8a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050810eae0 con 0x7f05080fd910 2024-04-12T05:45:29.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.886+0000 7f050da8a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050810ec20 con 0x7f05080fe270 2024-04-12T05:45:29.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.886+0000 7f05077fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05080fd910 0x7f050810dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:29.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.886+0000 7f05077fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05080fd910 0x7f050810dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:54860/0 (socket says 172.21.15.113:54860) 2024-04-12T05:45:29.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.886+0000 7f05077fe700 1 -- 172.21.15.113:0/1493574026 learned_addr learned my addr 172.21.15.113:0/1493574026 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:45:29.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.886+0000 7f0506ffd700 1 --2- 172.21.15.113:0/1493574026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05080fe270 0x7f050810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.887+0000 7f05077fe700 1 -- 172.21.15.113:0/1493574026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05080fe270 msgr2=0x7f050810e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.887+0000 7f05077fe700 1 --2- 172.21.15.113:0/1493574026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05080fe270 0x7f050810e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.887+0000 7f05077fe700 1 -- 172.21.15.113:0/1493574026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f8005040 con 0x7f05080fd910 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.887+0000 7f05077fe700 1 --2- 172.21.15.113:0/1493574026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05080fd910 0x7f050810dff0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f04f00041d0 tx=0x7f04f0009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.887+0000 7f0504f74700 1 -- 172.21.15.113:0/1493574026 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04f0007b40 con 0x7f05080fd910 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.887+0000 7f0504f74700 1 -- 172.21.15.113:0/1493574026 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04f000f040 con 0x7f05080fd910 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.887+0000 7f050da8a700 1 -- 172.21.15.113:0/1493574026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05081082b0 con 0x7f05080fd910 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.887+0000 7f0504f74700 1 -- 172.21.15.113:0/1493574026 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04f00147b0 con 0x7f05080fd910 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.887+0000 7f050da8a700 1 -- 172.21.15.113:0/1493574026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0508108870 con 0x7f05080fd910 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.888+0000 7f050da8a700 1 -- 172.21.15.113:0/1493574026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050805f570 con 0x7f05080fd910 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.888+0000 7f0504f74700 1 -- 172.21.15.113:0/1493574026 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f04f0009110 con 0x7f05080fd910 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.889+0000 7f0504f74700 1 --2- 172.21.15.113:0/1493574026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04f406caa0 0x7f04f406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:45:29.886 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.889+0000 7f0506ffd700 1 --2- 172.21.15.113:0/1493574026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04f406caa0 0x7f04f406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:45:29.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.889+0000 7f0504f74700 1 -- 172.21.15.113:0/1493574026 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f04f008be40 con 0x7f05080fd910 2024-04-12T05:45:29.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.889+0000 7f0506ffd700 1 --2- 172.21.15.113:0/1493574026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04f406caa0 0x7f04f406ef60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f0508109930 tx=0x7f04f800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:45:29.889 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:29.893+0000 7f0504f74700 1 -- 172.21.15.113:0/1493574026 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f04f00598f0 con 0x7f05080fd910 2024-04-12T05:45:30.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.161+0000 7f050da8a700 1 -- 172.21.15.113:0/1493574026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0508047540 con 0x7f05080fd910 2024-04-12T05:45:30.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.162+0000 7f0504f74700 1 -- 172.21.15.113:0/1493574026 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f04f0059480 con 0x7f05080fd910 2024-04-12T05:45:30.158 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:45:30.158 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:45:30.158 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:45:30.159 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:45:30.160 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:45:30.160 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:45:30.160 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:45:30.160 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:45:30.160 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:45:30.160 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:45:30.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.164+0000 7f050da8a700 1 -- 172.21.15.113:0/1493574026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04f406caa0 msgr2=0x7f04f406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:30.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.164+0000 7f050da8a700 1 --2- 172.21.15.113:0/1493574026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04f406caa0 0x7f04f406ef60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f0508109930 tx=0x7f04f800b040).stop 2024-04-12T05:45:30.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.164+0000 7f050da8a700 1 -- 172.21.15.113:0/1493574026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05080fd910 msgr2=0x7f050810dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:45:30.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.164+0000 7f050da8a700 1 --2- 172.21.15.113:0/1493574026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05080fd910 0x7f050810dff0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f04f00041d0 tx=0x7f04f0009f60).stop 2024-04-12T05:45:30.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.164+0000 7f050da8a700 1 -- 172.21.15.113:0/1493574026 shutdown_connections 2024-04-12T05:45:30.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.164+0000 7f050da8a700 1 --2- 172.21.15.113:0/1493574026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f04f406caa0 0x7f04f406ef60 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:30.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.164+0000 7f050da8a700 1 --2- 172.21.15.113:0/1493574026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05080fd910 0x7f050810dff0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:30.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.164+0000 7f050da8a700 1 --2- 172.21.15.113:0/1493574026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05080fe270 0x7f050810e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:45:30.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.164+0000 7f050da8a700 1 -- 172.21.15.113:0/1493574026 >> 172.21.15.113:0/1493574026 conn(0x7f05080f3530 msgr2=0x7f05080fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:45:30.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.164+0000 7f050da8a700 1 -- 172.21.15.113:0/1493574026 shutdown_connections 2024-04-12T05:45:30.161 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:45:30.164+0000 7f050da8a700 1 -- 172.21.15.113:0/1493574026 wait complete. 2024-04-12T05:45:30.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:30 smithi192 ceph-mon[85304]: from='client.35798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:30.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:30 smithi192 ceph-mon[85304]: from='client.35802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:30.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:30 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/876396223' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:45:30.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:30 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1493574026' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:45:30.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:30 smithi113 ceph-mon[103371]: from='client.35798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:30.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:30 smithi113 ceph-mon[103371]: from='client.35802 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:30.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:30 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/876396223' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:45:30.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:30 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1493574026' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:45:31.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:31 smithi192 ceph-mon[85304]: from='client.35814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:31.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:31 smithi192 ceph-mon[85304]: pgmap v1001: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:31.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:31 smithi113 ceph-mon[103371]: from='client.35814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:45:31.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:31 smithi113 ceph-mon[103371]: pgmap v1001: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:33.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:33 smithi192 ceph-mon[85304]: pgmap v1002: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:33.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:33 smithi113 ceph-mon[103371]: pgmap v1002: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:35.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:35 smithi192 ceph-mon[85304]: pgmap v1003: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:35.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:35 smithi113 ceph-mon[103371]: pgmap v1003: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:37.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:37 smithi192 ceph-mon[85304]: pgmap v1004: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:37.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:37.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:37 smithi113 ceph-mon[103371]: pgmap v1004: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:37.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:39.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:39 smithi192 ceph-mon[85304]: pgmap v1005: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:39 smithi113 ceph-mon[103371]: pgmap v1005: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:41.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:41 smithi192 ceph-mon[85304]: pgmap v1006: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:41 smithi113 ceph-mon[103371]: pgmap v1006: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:43.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:43 smithi113 ceph-mon[103371]: pgmap v1007: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:43.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:43 smithi192 ceph-mon[85304]: pgmap v1007: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:45.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:45 smithi113 ceph-mon[103371]: pgmap v1008: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:45.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:45 smithi192 ceph-mon[85304]: pgmap v1008: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:47.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:47 smithi113 ceph-mon[103371]: pgmap v1009: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:47.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:47 smithi192 ceph-mon[85304]: pgmap v1009: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:49.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:49 smithi113 ceph-mon[103371]: pgmap v1010: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:49.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:49 smithi192 ceph-mon[85304]: pgmap v1010: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:51.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:51 smithi113 ceph-mon[103371]: pgmap v1011: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:51.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:51 smithi192 ceph-mon[85304]: pgmap v1011: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:52.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:52.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:45:53.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:53 smithi113 ceph-mon[103371]: pgmap v1012: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:53.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:53 smithi192 ceph-mon[85304]: pgmap v1012: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:55.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:55 smithi113 ceph-mon[103371]: pgmap v1013: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:55.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:55 smithi192 ceph-mon[85304]: pgmap v1013: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:57.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:57 smithi113 ceph-mon[103371]: pgmap v1014: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:57.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:57 smithi192 ceph-mon[85304]: pgmap v1014: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:59.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:45:59 smithi113 ceph-mon[103371]: pgmap v1015: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:45:59.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:45:59 smithi192 ceph-mon[85304]: pgmap v1015: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.276+0000 7f59a889f700 1 -- 172.21.15.113:0/2836776461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a40fe250 msgr2=0x7f59a40f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.276+0000 7f59a889f700 1 --2- 172.21.15.113:0/2836776461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a40fe250 0x7f59a40f7a30 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f59980047f0 tx=0x7f5998009cb0).stop 2024-04-12T05:46:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.277+0000 7f59a889f700 1 -- 172.21.15.113:0/2836776461 shutdown_connections 2024-04-12T05:46:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.277+0000 7f59a889f700 1 --2- 172.21.15.113:0/2836776461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a40fe250 0x7f59a40f7a30 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.277+0000 7f59a889f700 1 --2- 172.21.15.113:0/2836776461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59a40fd8f0 0x7f59a40fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.277+0000 7f59a889f700 1 -- 172.21.15.113:0/2836776461 >> 172.21.15.113:0/2836776461 conn(0x7f59a40f34b0 msgr2=0x7f59a40f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.277+0000 7f59a889f700 1 -- 172.21.15.113:0/2836776461 shutdown_connections 2024-04-12T05:46:00.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.277+0000 7f59a889f700 1 -- 172.21.15.113:0/2836776461 wait complete. 2024-04-12T05:46:00.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.277+0000 7f59a889f700 1 Processor -- start 2024-04-12T05:46:00.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a889f700 1 -- start start 2024-04-12T05:46:00.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a889f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59a40fd8f0 0x7f59a410e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:00.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a889f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a40fe250 0x7f59a410e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:00.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a889f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a410eb70 con 0x7f59a40fe250 2024-04-12T05:46:00.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a889f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a410ecb0 con 0x7f59a40fd8f0 2024-04-12T05:46:00.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a1d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a40fe250 0x7f59a410e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:00.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a1d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a40fe250 0x7f59a410e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:52916/0 (socket says 172.21.15.113:52916) 2024-04-12T05:46:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a1d9b700 1 -- 172.21.15.113:0/3312225412 learned_addr learned my addr 172.21.15.113:0/3312225412 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a1d9b700 1 -- 172.21.15.113:0/3312225412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59a40fd8f0 msgr2=0x7f59a410e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a1d9b700 1 --2- 172.21.15.113:0/3312225412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59a40fd8f0 0x7f59a410e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:46:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a1d9b700 1 -- 172.21.15.113:0/3312225412 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5998005040 con 0x7f59a40fe250 2024-04-12T05:46:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a1d9b700 1 --2- 172.21.15.113:0/3312225412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a40fe250 0x7f59a410e570 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f59980099f0 tx=0x7f5998009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59937fe700 1 -- 172.21.15.113:0/3312225412 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f599800e030 con 0x7f59a40fe250 2024-04-12T05:46:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59937fe700 1 -- 172.21.15.113:0/3312225412 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5998007e90 con 0x7f59a40fe250 2024-04-12T05:46:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59a889f700 1 -- 172.21.15.113:0/3312225412 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a41082e0 con 0x7f59a40fe250 2024-04-12T05:46:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.278+0000 7f59937fe700 1 -- 172.21.15.113:0/3312225412 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5998018070 con 0x7f59a40fe250 2024-04-12T05:46:00.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.279+0000 7f59a889f700 1 -- 172.21.15.113:0/3312225412 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a41087c0 con 0x7f59a40fe250 2024-04-12T05:46:00.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.279+0000 7f59a889f700 1 -- 172.21.15.113:0/3312225412 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a405f590 con 0x7f59a40fe250 2024-04-12T05:46:00.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.280+0000 7f59937fe700 1 -- 172.21.15.113:0/3312225412 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f599801d420 con 0x7f59a40fe250 2024-04-12T05:46:00.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.280+0000 7f59937fe700 1 --2- 172.21.15.113:0/3312225412 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f598c06caa0 0x7f598c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:00.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.280+0000 7f59937fe700 1 -- 172.21.15.113:0/3312225412 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f599808d2f0 con 0x7f59a40fe250 2024-04-12T05:46:00.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.280+0000 7f59a259c700 1 --2- 172.21.15.113:0/3312225412 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f598c06caa0 0x7f598c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:00.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.281+0000 7f59a259c700 1 --2- 172.21.15.113:0/3312225412 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f598c06caa0 0x7f598c06ef60 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f5994004a90 tx=0x7f5994007b40).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:00.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.285+0000 7f59937fe700 1 -- 172.21.15.113:0/3312225412 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f599805ada0 con 0x7f59a40fe250 2024-04-12T05:46:00.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.491+0000 7f59a889f700 1 -- 172.21.15.113:0/3312225412 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f59a4109770 con 0x7f598c06caa0 2024-04-12T05:46:00.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.492+0000 7f59937fe700 1 -- 172.21.15.113:0/3312225412 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f59a4109770 con 0x7f598c06caa0 2024-04-12T05:46:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.494+0000 7f59a889f700 1 -- 172.21.15.113:0/3312225412 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f598c06caa0 msgr2=0x7f598c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.494+0000 7f59a889f700 1 --2- 172.21.15.113:0/3312225412 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f598c06caa0 0x7f598c06ef60 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f5994004a90 tx=0x7f5994007b40).stop 2024-04-12T05:46:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.494+0000 7f59a889f700 1 -- 172.21.15.113:0/3312225412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a40fe250 msgr2=0x7f59a410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.494+0000 7f59a889f700 1 --2- 172.21.15.113:0/3312225412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a40fe250 0x7f59a410e570 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f59980099f0 tx=0x7f5998009cb0).stop 2024-04-12T05:46:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.495+0000 7f59a889f700 1 -- 172.21.15.113:0/3312225412 shutdown_connections 2024-04-12T05:46:00.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.495+0000 7f59a889f700 1 --2- 172.21.15.113:0/3312225412 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f598c06caa0 0x7f598c06ef60 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.495+0000 7f59a889f700 1 --2- 172.21.15.113:0/3312225412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59a40fd8f0 0x7f59a410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:46:00.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.495+0000 7f59a889f700 1 --2- 172.21.15.113:0/3312225412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59a40fe250 0x7f59a410e570 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.495+0000 7f59a889f700 1 -- 172.21.15.113:0/3312225412 >> 172.21.15.113:0/3312225412 conn(0x7f59a40f34b0 msgr2=0x7f59a40fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:00.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.495+0000 7f59a889f700 1 -- 172.21.15.113:0/3312225412 shutdown_connections 2024-04-12T05:46:00.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.495+0000 7f59a889f700 1 -- 172.21.15.113:0/3312225412 wait complete. 2024-04-12T05:46:00.500 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.593+0000 7f38a3aee700 1 -- 172.21.15.113:0/3061496505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f389c0fd150 msgr2=0x7f389c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.593+0000 7f38a3aee700 1 --2- 172.21.15.113:0/3061496505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f389c0fd150 0x7f389c0f6930 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7f388c0047f0 tx=0x7f388c009cb0).stop 2024-04-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.593+0000 7f38a3aee700 1 -- 172.21.15.113:0/3061496505 shutdown_connections 2024-04-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.593+0000 7f38a3aee700 1 --2- 172.21.15.113:0/3061496505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f389c0fd150 0x7f389c0f6930 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.593+0000 7f38a3aee700 1 --2- 172.21.15.113:0/3061496505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f389c0fc7f0 0x7f389c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.593+0000 7f38a3aee700 1 -- 172.21.15.113:0/3061496505 >> 172.21.15.113:0/3061496505 conn(0x7f389c070590 msgr2=0x7f389c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.594+0000 7f38a3aee700 1 -- 172.21.15.113:0/3061496505 shutdown_connections 2024-04-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.594+0000 7f38a3aee700 1 -- 172.21.15.113:0/3061496505 wait complete. 2024-04-12T05:46:00.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.594+0000 7f38a3aee700 1 Processor -- start 2024-04-12T05:46:00.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.594+0000 7f38a3aee700 1 -- start start 2024-04-12T05:46:00.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.594+0000 7f38a3aee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f389c0fc7f0 0x7f389c1101f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:00.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.594+0000 7f38a3aee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f389c0fd150 0x7f389c1106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:00.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.594+0000 7f38a3aee700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c110ce0 con 0x7f389c0fd150 2024-04-12T05:46:00.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.594+0000 7f38a3aee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c110e20 con 0x7f389c0fc7f0 2024-04-12T05:46:00.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f38a1089700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f389c0fd150 0x7f389c1106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f38a1089700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f389c0fd150 0x7f389c1106e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35920/0 (socket says 172.21.15.113:35920) 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f38a1089700 1 -- 172.21.15.113:0/3753446257 learned_addr learned my addr 172.21.15.113:0/3753446257 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f38a188a700 1 --2- 172.21.15.113:0/3753446257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f389c0fc7f0 0x7f389c1101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f38a1089700 1 -- 172.21.15.113:0/3753446257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f389c0fc7f0 msgr2=0x7f389c1101f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f38a1089700 1 --2- 172.21.15.113:0/3753446257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f389c0fc7f0 0x7f389c1101f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f38a1089700 1 -- 172.21.15.113:0/3753446257 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c005040 con 0x7f389c0fd150 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f38a1089700 1 --2- 172.21.15.113:0/3753446257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f389c0fd150 0x7f389c1106e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f388c000c00 tx=0x7f388c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f3892ffd700 1 -- 172.21.15.113:0/3753446257 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f388c00e030 con 0x7f389c0fd150 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f3892ffd700 1 -- 172.21.15.113:0/3753446257 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f388c007e90 con 0x7f389c0fd150 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f3892ffd700 1 -- 172.21.15.113:0/3753446257 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f388c018070 con 0x7f389c0fd150 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f38a3aee700 1 -- 172.21.15.113:0/3753446257 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389c10a450 con 0x7f389c0fd150 2024-04-12T05:46:00.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.595+0000 7f38a3aee700 1 -- 172.21.15.113:0/3753446257 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c10a9b0 con 0x7f389c0fd150 2024-04-12T05:46:00.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.596+0000 7f38a3aee700 1 -- 172.21.15.113:0/3753446257 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389c05f590 con 0x7f389c0fd150 2024-04-12T05:46:00.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.596+0000 7f3892ffd700 1 -- 172.21.15.113:0/3753446257 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f388c01d420 con 0x7f389c0fd150 2024-04-12T05:46:00.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.597+0000 7f3892ffd700 1 --2- 172.21.15.113:0/3753446257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f388806cab0 0x7f388806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:00.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.597+0000 7f3892ffd700 1 -- 172.21.15.113:0/3753446257 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f388c08d200 con 0x7f389c0fd150 2024-04-12T05:46:00.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.597+0000 7f38a188a700 1 --2- 172.21.15.113:0/3753446257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f388806cab0 0x7f388806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:00.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.597+0000 7f38a188a700 1 --2- 172.21.15.113:0/3753446257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f388806cab0 0x7f388806ef70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f38980041d0 tx=0x7f3898009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:00.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.601+0000 7f3892ffd700 1 -- 172.21.15.113:0/3753446257 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f388c05acb0 con 0x7f389c0fd150 2024-04-12T05:46:00.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.806+0000 7f38a3aee700 1 -- 172.21.15.113:0/3753446257 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f389c10b910 con 0x7f388806cab0 2024-04-12T05:46:00.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.807+0000 7f3892ffd700 1 -- 172.21.15.113:0/3753446257 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f389c10b910 con 0x7f388806cab0 2024-04-12T05:46:00.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.809+0000 7f38a3aee700 1 -- 172.21.15.113:0/3753446257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f388806cab0 msgr2=0x7f388806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:00.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.809+0000 7f38a3aee700 1 --2- 172.21.15.113:0/3753446257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f388806cab0 0x7f388806ef70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f38980041d0 tx=0x7f3898009040).stop 2024-04-12T05:46:00.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.809+0000 7f38a3aee700 1 -- 172.21.15.113:0/3753446257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f389c0fd150 msgr2=0x7f389c1106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:00.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.809+0000 7f38a3aee700 1 --2- 172.21.15.113:0/3753446257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f389c0fd150 0x7f389c1106e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f388c000c00 tx=0x7f388c009cb0).stop 2024-04-12T05:46:00.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.809+0000 7f38a3aee700 1 -- 172.21.15.113:0/3753446257 shutdown_connections 2024-04-12T05:46:00.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.809+0000 7f38a3aee700 1 --2- 172.21.15.113:0/3753446257 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f388806cab0 0x7f388806ef70 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.809+0000 7f38a3aee700 1 --2- 172.21.15.113:0/3753446257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f389c0fc7f0 0x7f389c1101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.809+0000 7f38a3aee700 1 --2- 172.21.15.113:0/3753446257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f389c0fd150 0x7f389c1106e0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.809+0000 7f38a3aee700 1 -- 172.21.15.113:0/3753446257 >> 172.21.15.113:0/3753446257 conn(0x7f389c070590 msgr2=0x7f389c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:00.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.809+0000 7f38a3aee700 1 -- 172.21.15.113:0/3753446257 shutdown_connections 2024-04-12T05:46:00.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.809+0000 7f38a3aee700 1 -- 172.21.15.113:0/3753446257 wait complete. 2024-04-12T05:46:00.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.907+0000 7fb530be6700 1 -- 172.21.15.113:0/252832813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb52c0fd150 msgr2=0x7fb52c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:00.905 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.907+0000 7fb530be6700 1 --2- 172.21.15.113:0/252832813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb52c0fd150 0x7fb52c0f6930 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7fb51c0047f0 tx=0x7fb51c009cb0).stop 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.907+0000 7fb530be6700 1 -- 172.21.15.113:0/252832813 shutdown_connections 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.907+0000 7fb530be6700 1 --2- 172.21.15.113:0/252832813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb52c0fd150 0x7fb52c0f6930 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.907+0000 7fb530be6700 1 --2- 172.21.15.113:0/252832813 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb52c0fc7f0 0x7fb52c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.907+0000 7fb530be6700 1 -- 172.21.15.113:0/252832813 >> 172.21.15.113:0/252832813 conn(0x7fb52c070590 msgr2=0x7fb52c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.907+0000 7fb530be6700 1 -- 172.21.15.113:0/252832813 shutdown_connections 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.907+0000 7fb530be6700 1 -- 172.21.15.113:0/252832813 wait complete. 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.908+0000 7fb530be6700 1 Processor -- start 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.908+0000 7fb530be6700 1 -- start start 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.908+0000 7fb530be6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb52c0fc7f0 0x7fb52c109080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.908+0000 7fb530be6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb52c0fd150 0x7fb52c109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.908+0000 7fb530be6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c109ab0 con 0x7fb52c0fd150 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.908+0000 7fb530be6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c109bf0 con 0x7fb52c0fc7f0 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.908+0000 7fb529d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb52c0fd150 0x7fb52c109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:00.906 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.908+0000 7fb529d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb52c0fd150 0x7fb52c109570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35926/0 (socket says 172.21.15.113:35926) 2024-04-12T05:46:00.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.908+0000 7fb529d9b700 1 -- 172.21.15.113:0/3472762266 learned_addr learned my addr 172.21.15.113:0/3472762266 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:00.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.909+0000 7fb52a59c700 1 --2- 172.21.15.113:0/3472762266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb52c0fc7f0 0x7fb52c109080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:00.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.909+0000 7fb529d9b700 1 -- 172.21.15.113:0/3472762266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb52c0fc7f0 msgr2=0x7fb52c109080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:00.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.909+0000 7fb529d9b700 1 --2- 172.21.15.113:0/3472762266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb52c0fc7f0 0x7fb52c109080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:00.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.909+0000 7fb529d9b700 1 -- 172.21.15.113:0/3472762266 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51c005040 con 0x7fb52c0fd150 2024-04-12T05:46:00.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.909+0000 7fb529d9b700 1 --2- 172.21.15.113:0/3472762266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb52c0fd150 0x7fb52c109570 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7fb51c0099f0 tx=0x7fb51c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:00.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.909+0000 7fb5237fe700 1 -- 172.21.15.113:0/3472762266 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb51c00e030 con 0x7fb52c0fd150 2024-04-12T05:46:00.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.909+0000 7fb530be6700 1 -- 172.21.15.113:0/3472762266 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52c109e40 con 0x7fb52c0fd150 2024-04-12T05:46:00.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.909+0000 7fb5237fe700 1 -- 172.21.15.113:0/3472762266 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb51c007e90 con 0x7fb52c0fd150 2024-04-12T05:46:00.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.909+0000 7fb5237fe700 1 -- 172.21.15.113:0/3472762266 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb51c018070 con 0x7fb52c0fd150 2024-04-12T05:46:00.907 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.909+0000 7fb530be6700 1 -- 172.21.15.113:0/3472762266 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52c1123e0 con 0x7fb52c0fd150 2024-04-12T05:46:00.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.910+0000 7fb530be6700 1 -- 172.21.15.113:0/3472762266 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb52c05f590 con 0x7fb52c0fd150 2024-04-12T05:46:00.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.910+0000 7fb5237fe700 1 -- 172.21.15.113:0/3472762266 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb51c01d420 con 0x7fb52c0fd150 2024-04-12T05:46:00.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.911+0000 7fb5237fe700 1 --2- 172.21.15.113:0/3472762266 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb51806caa0 0x7fb51806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:00.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.911+0000 7fb5237fe700 1 -- 172.21.15.113:0/3472762266 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb51c08d2f0 con 0x7fb52c0fd150 2024-04-12T05:46:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.911+0000 7fb52a59c700 1 --2- 172.21.15.113:0/3472762266 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb51806caa0 0x7fb51806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.911+0000 7fb52a59c700 1 --2- 172.21.15.113:0/3472762266 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb51806caa0 0x7fb51806ef60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fb5140041d0 tx=0x7fb514009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:00.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:00.915+0000 7fb5237fe700 1 -- 172.21.15.113:0/3472762266 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb51c057310 con 0x7fb52c0fd150 2024-04-12T05:46:01.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.113+0000 7fb530be6700 1 -- 172.21.15.113:0/3472762266 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb52c059120 con 0x7fb51806caa0 2024-04-12T05:46:01.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.122+0000 7fb5237fe700 1 -- 172.21.15.113:0/3472762266 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb52c059120 con 0x7fb51806caa0 2024-04-12T05:46:01.118 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (33m) 11m ago 45m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (31m) 11m ago 44m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (31m) 11m ago 44m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (33m) 11m ago 44m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (27m) 11m ago 39m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (26m) 11m ago 39m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (26m) 11m ago 39m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (11m) 11m ago 39m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (35m) 11m ago 46m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (34m) 11m ago 44m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (32m) 11m ago 46m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (31m) 11m ago 44m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (34m) 11m ago 44m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:46:01.119 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (34m) 11m ago 44m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:46:01.120 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (30m) 11m ago 43m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:46:01.120 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (30m) 11m ago 43m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:46:01.120 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (29m) 11m ago 42m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:46:01.120 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (29m) 11m ago 42m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:46:01.120 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (28m) 11m ago 41m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:46:01.120 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (27m) 11m ago 41m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:46:01.120 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (34m) 11m ago 43m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:46:01.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.124+0000 7fb530be6700 1 -- 172.21.15.113:0/3472762266 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb51806caa0 msgr2=0x7fb51806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.124+0000 7fb530be6700 1 --2- 172.21.15.113:0/3472762266 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb51806caa0 0x7fb51806ef60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fb5140041d0 tx=0x7fb514009040).stop 2024-04-12T05:46:01.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.124+0000 7fb530be6700 1 -- 172.21.15.113:0/3472762266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb52c0fd150 msgr2=0x7fb52c109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.124+0000 7fb530be6700 1 --2- 172.21.15.113:0/3472762266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb52c0fd150 0x7fb52c109570 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7fb51c0099f0 tx=0x7fb51c009cb0).stop 2024-04-12T05:46:01.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.124+0000 7fb530be6700 1 -- 172.21.15.113:0/3472762266 shutdown_connections 2024-04-12T05:46:01.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.124+0000 7fb530be6700 1 --2- 172.21.15.113:0/3472762266 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb51806caa0 0x7fb51806ef60 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.124+0000 7fb530be6700 1 --2- 172.21.15.113:0/3472762266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb52c0fc7f0 0x7fb52c109080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.124+0000 7fb530be6700 1 --2- 172.21.15.113:0/3472762266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb52c0fd150 0x7fb52c109570 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.124+0000 7fb530be6700 1 -- 172.21.15.113:0/3472762266 >> 172.21.15.113:0/3472762266 conn(0x7fb52c070590 msgr2=0x7fb52c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:01.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.124+0000 7fb530be6700 1 -- 172.21.15.113:0/3472762266 shutdown_connections 2024-04-12T05:46:01.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.124+0000 7fb530be6700 1 -- 172.21.15.113:0/3472762266 wait complete. 2024-04-12T05:46:01.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.221+0000 7fd798ab8700 1 -- 172.21.15.113:0/1258840812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7940fd8f0 msgr2=0x7fd7940fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.221+0000 7fd798ab8700 1 --2- 172.21.15.113:0/1258840812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7940fd8f0 0x7fd7940fdcd0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7fd7880047f0 tx=0x7fd788009cb0).stop 2024-04-12T05:46:01.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.221+0000 7fd798ab8700 1 -- 172.21.15.113:0/1258840812 shutdown_connections 2024-04-12T05:46:01.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.221+0000 7fd798ab8700 1 --2- 172.21.15.113:0/1258840812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7940fe250 0x7fd7940f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.221+0000 7fd798ab8700 1 --2- 172.21.15.113:0/1258840812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7940fd8f0 0x7fd7940fdcd0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.221+0000 7fd798ab8700 1 -- 172.21.15.113:0/1258840812 >> 172.21.15.113:0/1258840812 conn(0x7fd7940f34b0 msgr2=0x7fd7940f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:01.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.221+0000 7fd798ab8700 1 -- 172.21.15.113:0/1258840812 shutdown_connections 2024-04-12T05:46:01.222 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.221+0000 7fd798ab8700 1 -- 172.21.15.113:0/1258840812 wait complete. 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd798ab8700 1 Processor -- start 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd798ab8700 1 -- start start 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd798ab8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7940fd8f0 0x7fd79410dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd798ab8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7940fe250 0x7fd79410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd798ab8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79410ead0 con 0x7fd7940fd8f0 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd798ab8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79410ec10 con 0x7fd7940fe250 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd79259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7940fd8f0 0x7fd79410dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd79259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7940fd8f0 0x7fd79410dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35938/0 (socket says 172.21.15.113:35938) 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd79259c700 1 -- 172.21.15.113:0/77876307 learned_addr learned my addr 172.21.15.113:0/77876307 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd791d9b700 1 --2- 172.21.15.113:0/77876307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7940fe250 0x7fd79410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd79259c700 1 -- 172.21.15.113:0/77876307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7940fe250 msgr2=0x7fd79410e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd79259c700 1 --2- 172.21.15.113:0/77876307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7940fe250 0x7fd79410e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.222+0000 7fd79259c700 1 -- 172.21.15.113:0/77876307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788005040 con 0x7fd7940fd8f0 2024-04-12T05:46:01.223 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.223+0000 7fd79259c700 1 --2- 172.21.15.113:0/77876307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7940fd8f0 0x7fd79410dfe0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7fd788013640 tx=0x7fd788009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:01.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.223+0000 7fd7837fe700 1 -- 172.21.15.113:0/77876307 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd78800e030 con 0x7fd7940fd8f0 2024-04-12T05:46:01.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.223+0000 7fd7837fe700 1 -- 172.21.15.113:0/77876307 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd788007e90 con 0x7fd7940fd8f0 2024-04-12T05:46:01.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.223+0000 7fd7837fe700 1 -- 172.21.15.113:0/77876307 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd788018070 con 0x7fd7940fd8f0 2024-04-12T05:46:01.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.223+0000 7fd798ab8700 1 -- 172.21.15.113:0/77876307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd794108240 con 0x7fd7940fd8f0 2024-04-12T05:46:01.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.223+0000 7fd798ab8700 1 -- 172.21.15.113:0/77876307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7941087a0 con 0x7fd7940fd8f0 2024-04-12T05:46:01.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.224+0000 7fd798ab8700 1 -- 172.21.15.113:0/77876307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79405f590 con 0x7fd7940fd8f0 2024-04-12T05:46:01.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.224+0000 7fd7837fe700 1 -- 172.21.15.113:0/77876307 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd78801d420 con 0x7fd7940fd8f0 2024-04-12T05:46:01.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.225+0000 7fd7837fe700 1 --2- 172.21.15.113:0/77876307 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd77c06cab0 0x7fd77c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:01.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.225+0000 7fd7837fe700 1 -- 172.21.15.113:0/77876307 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd78808d790 con 0x7fd7940fd8f0 2024-04-12T05:46:01.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.225+0000 7fd791d9b700 1 --2- 172.21.15.113:0/77876307 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd77c06cab0 0x7fd77c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:01.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.225+0000 7fd791d9b700 1 --2- 172.21.15.113:0/77876307 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd77c06cab0 0x7fd77c06ef70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fd794109920 tx=0x7fd784009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:01.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.229+0000 7fd7837fe700 1 -- 172.21.15.113:0/77876307 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd78805b2c0 con 0x7fd7940fd8f0 2024-04-12T05:46:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.497+0000 7fd798ab8700 1 -- 172.21.15.113:0/77876307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd794047540 con 0x7fd7940fd8f0 2024-04-12T05:46:01.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.497+0000 7fd7837fe700 1 -- 172.21.15.113:0/77876307 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fd78805ae50 con 0x7fd7940fd8f0 2024-04-12T05:46:01.493 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:46:01.494 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:46:01.495 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:46:01.495 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:46:01.495 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:46:01.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.499+0000 7fd798ab8700 1 -- 172.21.15.113:0/77876307 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd77c06cab0 msgr2=0x7fd77c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.499+0000 7fd798ab8700 1 --2- 172.21.15.113:0/77876307 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd77c06cab0 0x7fd77c06ef70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fd794109920 tx=0x7fd784009040).stop 2024-04-12T05:46:01.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.499+0000 7fd798ab8700 1 -- 172.21.15.113:0/77876307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7940fd8f0 msgr2=0x7fd79410dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.499+0000 7fd798ab8700 1 --2- 172.21.15.113:0/77876307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7940fd8f0 0x7fd79410dfe0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7fd788013640 tx=0x7fd788009cb0).stop 2024-04-12T05:46:01.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.499+0000 7fd798ab8700 1 -- 172.21.15.113:0/77876307 shutdown_connections 2024-04-12T05:46:01.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.499+0000 7fd798ab8700 1 --2- 172.21.15.113:0/77876307 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd77c06cab0 0x7fd77c06ef70 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.499+0000 7fd798ab8700 1 --2- 172.21.15.113:0/77876307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7940fd8f0 0x7fd79410dfe0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.499+0000 7fd798ab8700 1 --2- 172.21.15.113:0/77876307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7940fe250 0x7fd79410e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.499+0000 7fd798ab8700 1 -- 172.21.15.113:0/77876307 >> 172.21.15.113:0/77876307 conn(0x7fd7940f34b0 msgr2=0x7fd7940fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:01.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.499+0000 7fd798ab8700 1 -- 172.21.15.113:0/77876307 shutdown_connections 2024-04-12T05:46:01.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.499+0000 7fd798ab8700 1 -- 172.21.15.113:0/77876307 wait complete. 2024-04-12T05:46:01.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.596+0000 7faafe701700 1 -- 172.21.15.113:0/3725373831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80fd8f0 msgr2=0x7faaf80fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.596+0000 7faafe701700 1 --2- 172.21.15.113:0/3725373831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80fd8f0 0x7faaf80fdcd0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7faae80047f0 tx=0x7faae8009cb0).stop 2024-04-12T05:46:01.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.596+0000 7faafe701700 1 -- 172.21.15.113:0/3725373831 shutdown_connections 2024-04-12T05:46:01.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.596+0000 7faafe701700 1 --2- 172.21.15.113:0/3725373831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaf80fe250 0x7faaf80f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.596+0000 7faafe701700 1 --2- 172.21.15.113:0/3725373831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80fd8f0 0x7faaf80fdcd0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.596+0000 7faafe701700 1 -- 172.21.15.113:0/3725373831 >> 172.21.15.113:0/3725373831 conn(0x7faaf80f34b0 msgr2=0x7faaf80f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:01.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.596+0000 7faafe701700 1 -- 172.21.15.113:0/3725373831 shutdown_connections 2024-04-12T05:46:01.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.596+0000 7faafe701700 1 -- 172.21.15.113:0/3725373831 wait complete. 2024-04-12T05:46:01.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.597+0000 7faafe701700 1 Processor -- start 2024-04-12T05:46:01.594 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.597+0000 7faafe701700 1 -- start start 2024-04-12T05:46:01.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.597+0000 7faafe701700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaf80fd8f0 0x7faaf810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:01.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.597+0000 7faafe701700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80fe250 0x7faaf810e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.597+0000 7faafe701700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf810ead0 con 0x7faaf80fe250 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.597+0000 7faafe701700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf810ec10 con 0x7faaf80fd8f0 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faaf77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80fe250 0x7faaf810e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faaf77fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80fe250 0x7faaf810e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35956/0 (socket says 172.21.15.113:35956) 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faaf77fe700 1 -- 172.21.15.113:0/3442167817 learned_addr learned my addr 172.21.15.113:0/3442167817 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faaf7fff700 1 --2- 172.21.15.113:0/3442167817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaf80fd8f0 0x7faaf810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faaf77fe700 1 -- 172.21.15.113:0/3442167817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaf80fd8f0 msgr2=0x7faaf810dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faaf77fe700 1 --2- 172.21.15.113:0/3442167817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaf80fd8f0 0x7faaf810dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faaf77fe700 1 -- 172.21.15.113:0/3442167817 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae8005040 con 0x7faaf80fe250 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faaf77fe700 1 --2- 172.21.15.113:0/3442167817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80fe250 0x7faaf810e4d0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7faaec00c710 tx=0x7faaec007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faaf57fa700 1 -- 172.21.15.113:0/3442167817 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaec00e680 con 0x7faaf80fe250 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faaf57fa700 1 -- 172.21.15.113:0/3442167817 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faaec00ec90 con 0x7faaf80fe250 2024-04-12T05:46:01.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faafe701700 1 -- 172.21.15.113:0/3442167817 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf81082a0 con 0x7faaf80fe250 2024-04-12T05:46:01.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faaf57fa700 1 -- 172.21.15.113:0/3442167817 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaec013d40 con 0x7faaf80fe250 2024-04-12T05:46:01.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.598+0000 7faafe701700 1 -- 172.21.15.113:0/3442167817 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf8108860 con 0x7faaf80fe250 2024-04-12T05:46:01.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.599+0000 7faafe701700 1 -- 172.21.15.113:0/3442167817 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf805f590 con 0x7faaf80fe250 2024-04-12T05:46:01.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.600+0000 7faaf57fa700 1 -- 172.21.15.113:0/3442167817 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faaec00e7e0 con 0x7faaf80fe250 2024-04-12T05:46:01.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.600+0000 7faaf57fa700 1 --2- 172.21.15.113:0/3442167817 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faae006caa0 0x7faae006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:01.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.600+0000 7faaf57fa700 1 -- 172.21.15.113:0/3442167817 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7faaec08bc90 con 0x7faaf80fe250 2024-04-12T05:46:01.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.600+0000 7faaf7fff700 1 --2- 172.21.15.113:0/3442167817 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faae006caa0 0x7faae006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:01.598 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.600+0000 7faaf7fff700 1 --2- 172.21.15.113:0/3442167817 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faae006caa0 0x7faae006ef60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7faae8007cf0 tx=0x7faae800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:01.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.604+0000 7faaf57fa700 1 -- 172.21.15.113:0/3442167817 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7faaec059740 con 0x7faaf80fe250 2024-04-12T05:46:01.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:01 smithi113 ceph-mon[103371]: pgmap v1016: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:01.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:01 smithi192 ceph-mon[85304]: pgmap v1016: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:01.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.833+0000 7faafe701700 1 -- 172.21.15.113:0/3442167817 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7faaf8047540 con 0x7faaf80fe250 2024-04-12T05:46:01.830 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.834+0000 7faaf57fa700 1 -- 172.21.15.113:0/3442167817 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7faaec0592d0 con 0x7faaf80fe250 2024-04-12T05:46:01.830 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:46:01.830 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:46:01.830 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:46:01.831 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:46:01.832 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:46:01.833 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:46:01.833 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:46:01.833 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:46:01.833 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:46:01.833 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:46:01.833 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:01.833 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:01.833 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:01.833 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:01.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.836+0000 7faafe701700 1 -- 172.21.15.113:0/3442167817 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faae006caa0 msgr2=0x7faae006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.836+0000 7faafe701700 1 --2- 172.21.15.113:0/3442167817 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faae006caa0 0x7faae006ef60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7faae8007cf0 tx=0x7faae800b040).stop 2024-04-12T05:46:01.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.836+0000 7faafe701700 1 -- 172.21.15.113:0/3442167817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80fe250 msgr2=0x7faaf810e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.836+0000 7faafe701700 1 --2- 172.21.15.113:0/3442167817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80fe250 0x7faaf810e4d0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7faaec00c710 tx=0x7faaec007ec0).stop 2024-04-12T05:46:01.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.836+0000 7faafe701700 1 -- 172.21.15.113:0/3442167817 shutdown_connections 2024-04-12T05:46:01.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.836+0000 7faafe701700 1 --2- 172.21.15.113:0/3442167817 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7faae006caa0 0x7faae006ef60 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.836+0000 7faafe701700 1 --2- 172.21.15.113:0/3442167817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaf80fd8f0 0x7faaf810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.836+0000 7faafe701700 1 --2- 172.21.15.113:0/3442167817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faaf80fe250 0x7faaf810e4d0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.836+0000 7faafe701700 1 -- 172.21.15.113:0/3442167817 >> 172.21.15.113:0/3442167817 conn(0x7faaf80f34b0 msgr2=0x7faaf80fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:01.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.836+0000 7faafe701700 1 -- 172.21.15.113:0/3442167817 shutdown_connections 2024-04-12T05:46:01.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.836+0000 7faafe701700 1 -- 172.21.15.113:0/3442167817 wait complete. 2024-04-12T05:46:01.835 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:46:01.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.935+0000 7f8910760700 1 -- 172.21.15.113:0/354663269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89080fc7f0 msgr2=0x7f89080fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.935+0000 7f8910760700 1 --2- 172.21.15.113:0/354663269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89080fc7f0 0x7f89080fcbd0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f88f80047f0 tx=0x7f88f8009cb0).stop 2024-04-12T05:46:01.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.935+0000 7f8910760700 1 -- 172.21.15.113:0/354663269 shutdown_connections 2024-04-12T05:46:01.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.935+0000 7f8910760700 1 --2- 172.21.15.113:0/354663269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89080fd150 0x7f89080f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.935+0000 7f8910760700 1 --2- 172.21.15.113:0/354663269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89080fc7f0 0x7f89080fcbd0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.935+0000 7f8910760700 1 -- 172.21.15.113:0/354663269 >> 172.21.15.113:0/354663269 conn(0x7f8908070590 msgr2=0x7f89080f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:01.933 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.935+0000 7f8910760700 1 -- 172.21.15.113:0/354663269 shutdown_connections 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.935+0000 7f8910760700 1 -- 172.21.15.113:0/354663269 wait complete. 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.935+0000 7f8910760700 1 Processor -- start 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.935+0000 7f8910760700 1 -- start start 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f8910760700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89080fc7f0 0x7f8908109090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f8910760700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89080fd150 0x7f8908109580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f8910760700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8908109ac0 con 0x7f89080fd150 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f8910760700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8908109c00 con 0x7f89080fc7f0 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f890dcfb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89080fd150 0x7f8908109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f890dcfb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89080fd150 0x7f8908109580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35968/0 (socket says 172.21.15.113:35968) 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f890dcfb700 1 -- 172.21.15.113:0/808898634 learned_addr learned my addr 172.21.15.113:0/808898634 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f890e4fc700 1 --2- 172.21.15.113:0/808898634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89080fc7f0 0x7f8908109090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f890dcfb700 1 -- 172.21.15.113:0/808898634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89080fc7f0 msgr2=0x7f8908109090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f890dcfb700 1 --2- 172.21.15.113:0/808898634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89080fc7f0 0x7f8908109090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:01.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f890dcfb700 1 -- 172.21.15.113:0/808898634 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8005040 con 0x7f89080fd150 2024-04-12T05:46:01.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f890dcfb700 1 --2- 172.21.15.113:0/808898634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89080fd150 0x7f8908109580 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7f890400c790 tx=0x7f8904007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:01.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f88ff7fe700 1 -- 172.21.15.113:0/808898634 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89040106f0 con 0x7f89080fd150 2024-04-12T05:46:01.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f88ff7fe700 1 -- 172.21.15.113:0/808898634 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8904010d00 con 0x7f89080fd150 2024-04-12T05:46:01.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f8910760700 1 -- 172.21.15.113:0/808898634 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8908109eb0 con 0x7f89080fd150 2024-04-12T05:46:01.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f88ff7fe700 1 -- 172.21.15.113:0/808898634 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8904012cf0 con 0x7f89080fd150 2024-04-12T05:46:01.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.936+0000 7f8910760700 1 -- 172.21.15.113:0/808898634 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89081124a0 con 0x7f89080fd150 2024-04-12T05:46:01.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.937+0000 7f8910760700 1 -- 172.21.15.113:0/808898634 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890805f590 con 0x7f89080fd150 2024-04-12T05:46:01.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.938+0000 7f88ff7fe700 1 -- 172.21.15.113:0/808898634 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8904012400 con 0x7f89080fd150 2024-04-12T05:46:01.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.938+0000 7f88ff7fe700 1 --2- 172.21.15.113:0/808898634 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88f406caa0 0x7f88f406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:01.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.938+0000 7f88ff7fe700 1 -- 172.21.15.113:0/808898634 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f890408b680 con 0x7f89080fd150 2024-04-12T05:46:01.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.938+0000 7f890e4fc700 1 --2- 172.21.15.113:0/808898634 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88f406caa0 0x7f88f406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:01.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.939+0000 7f890e4fc700 1 --2- 172.21.15.113:0/808898634 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88f406caa0 0x7f88f406ef60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f88f8000c00 tx=0x7f88f800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:01.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:01.943+0000 7f88ff7fe700 1 -- 172.21.15.113:0/808898634 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f89040556a0 con 0x7f89080fd150 2024-04-12T05:46:02.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.150+0000 7f8910760700 1 -- 172.21.15.113:0/808898634 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8908059120 con 0x7f88f406caa0 2024-04-12T05:46:02.147 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.151+0000 7f88ff7fe700 1 -- 172.21.15.113:0/808898634 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8908059120 con 0x7f88f406caa0 2024-04-12T05:46:02.148 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:46:02.148 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:46:02.148 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:46:02.148 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:46:02.148 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:46:02.148 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:46:02.148 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:46:02.148 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:46:02.148 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:46:02.148 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:46:02.148 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:46:02.149 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:46:02.149 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:46:02.149 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:46:02.149 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:46:02.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.153+0000 7f8910760700 1 -- 172.21.15.113:0/808898634 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88f406caa0 msgr2=0x7f88f406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:02.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.153+0000 7f8910760700 1 --2- 172.21.15.113:0/808898634 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88f406caa0 0x7f88f406ef60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f88f8000c00 tx=0x7f88f800b040).stop 2024-04-12T05:46:02.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.153+0000 7f8910760700 1 -- 172.21.15.113:0/808898634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89080fd150 msgr2=0x7f8908109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:02.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.153+0000 7f8910760700 1 --2- 172.21.15.113:0/808898634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89080fd150 0x7f8908109580 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7f890400c790 tx=0x7f8904007f70).stop 2024-04-12T05:46:02.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.153+0000 7f8910760700 1 -- 172.21.15.113:0/808898634 shutdown_connections 2024-04-12T05:46:02.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.153+0000 7f8910760700 1 --2- 172.21.15.113:0/808898634 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88f406caa0 0x7f88f406ef60 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:02.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.153+0000 7f8910760700 1 --2- 172.21.15.113:0/808898634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89080fc7f0 0x7f8908109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:02.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.153+0000 7f8910760700 1 --2- 172.21.15.113:0/808898634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89080fd150 0x7f8908109580 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:02.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.153+0000 7f8910760700 1 -- 172.21.15.113:0/808898634 >> 172.21.15.113:0/808898634 conn(0x7f8908070590 msgr2=0x7f89080fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:02.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.154+0000 7f8910760700 1 -- 172.21.15.113:0/808898634 shutdown_connections 2024-04-12T05:46:02.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.154+0000 7f8910760700 1 -- 172.21.15.113:0/808898634 wait complete. 2024-04-12T05:46:02.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.253+0000 7f30cd56f700 1 -- 172.21.15.113:0/2920929487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c80fd950 msgr2=0x7f30c80fdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:02.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.253+0000 7f30cd56f700 1 --2- 172.21.15.113:0/2920929487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c80fd950 0x7f30c80fdd30 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7f30b80047f0 tx=0x7f30b8009cb0).stop 2024-04-12T05:46:02.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.253+0000 7f30cd56f700 1 -- 172.21.15.113:0/2920929487 shutdown_connections 2024-04-12T05:46:02.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.253+0000 7f30cd56f700 1 --2- 172.21.15.113:0/2920929487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30c80fe2b0 0x7f30c8060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:02.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.253+0000 7f30cd56f700 1 --2- 172.21.15.113:0/2920929487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c80fd950 0x7f30c80fdd30 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:02.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.253+0000 7f30cd56f700 1 -- 172.21.15.113:0/2920929487 >> 172.21.15.113:0/2920929487 conn(0x7f30c806c760 msgr2=0x7f30c806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:02.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.253+0000 7f30cd56f700 1 -- 172.21.15.113:0/2920929487 shutdown_connections 2024-04-12T05:46:02.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.253+0000 7f30cd56f700 1 -- 172.21.15.113:0/2920929487 wait complete. 2024-04-12T05:46:02.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.254+0000 7f30cd56f700 1 Processor -- start 2024-04-12T05:46:02.252 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.254+0000 7f30cd56f700 1 -- start start 2024-04-12T05:46:02.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.254+0000 7f30cd56f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c80fd950 0x7f30c8110330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:02.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.254+0000 7f30cd56f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30c80fe2b0 0x7f30c8110820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:02.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.254+0000 7f30cd56f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c8110e20 con 0x7f30c80fd950 2024-04-12T05:46:02.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.254+0000 7f30cd56f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c8110f60 con 0x7f30c80fe2b0 2024-04-12T05:46:02.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30c6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c80fd950 0x7f30c8110330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:02.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30c6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c80fd950 0x7f30c8110330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:35982/0 (socket says 172.21.15.113:35982) 2024-04-12T05:46:02.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30c6ffd700 1 -- 172.21.15.113:0/1336588408 learned_addr learned my addr 172.21.15.113:0/1336588408 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:02.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30c67fc700 1 --2- 172.21.15.113:0/1336588408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30c80fe2b0 0x7f30c8110820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:02.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30c6ffd700 1 -- 172.21.15.113:0/1336588408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30c80fe2b0 msgr2=0x7f30c8110820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:02.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30c6ffd700 1 --2- 172.21.15.113:0/1336588408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30c80fe2b0 0x7f30c8110820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:02.253 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30c6ffd700 1 -- 172.21.15.113:0/1336588408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b8005040 con 0x7f30c80fd950 2024-04-12T05:46:02.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30c6ffd700 1 --2- 172.21.15.113:0/1336588408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c80fd950 0x7f30c8110330 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7f30b8009380 tx=0x7f30b8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:02.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30affff700 1 -- 172.21.15.113:0/1336588408 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30b8018070 con 0x7f30c80fd950 2024-04-12T05:46:02.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30affff700 1 -- 172.21.15.113:0/1336588408 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30b8007e90 con 0x7f30c80fd950 2024-04-12T05:46:02.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30cd56f700 1 -- 172.21.15.113:0/1336588408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c810a590 con 0x7f30c80fd950 2024-04-12T05:46:02.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30affff700 1 -- 172.21.15.113:0/1336588408 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30b80123c0 con 0x7f30c80fd950 2024-04-12T05:46:02.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.255+0000 7f30cd56f700 1 -- 172.21.15.113:0/1336588408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c810aaf0 con 0x7f30c80fd950 2024-04-12T05:46:02.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.256+0000 7f30cd56f700 1 -- 172.21.15.113:0/1336588408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c805f570 con 0x7f30c80fd950 2024-04-12T05:46:02.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.257+0000 7f30affff700 1 -- 172.21.15.113:0/1336588408 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f30b801d420 con 0x7f30c80fd950 2024-04-12T05:46:02.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.257+0000 7f30affff700 1 --2- 172.21.15.113:0/1336588408 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b006caa0 0x7f30b006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:02.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.258+0000 7f30affff700 1 -- 172.21.15.113:0/1336588408 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f30b808ce10 con 0x7f30c80fd950 2024-04-12T05:46:02.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.258+0000 7f30c67fc700 1 --2- 172.21.15.113:0/1336588408 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b006caa0 0x7f30b006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:02.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.258+0000 7f30c67fc700 1 --2- 172.21.15.113:0/1336588408 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b006caa0 0x7f30b006ef60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f30c810bc70 tx=0x7f30bc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:02.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.261+0000 7f30affff700 1 -- 172.21.15.113:0/1336588408 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f30b805a800 con 0x7f30c80fd950 2024-04-12T05:46:02.527 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.531+0000 7f30cd56f700 1 -- 172.21.15.113:0/1336588408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f30c8047540 con 0x7f30c80fd950 2024-04-12T05:46:02.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.531+0000 7f30affff700 1 -- 172.21.15.113:0/1336588408 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f30b805a390 con 0x7f30c80fd950 2024-04-12T05:46:02.528 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:46:02.528 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:46:02.528 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:46:02.528 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:46:02.528 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:46:02.528 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:46:02.528 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:46:02.528 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:46:02.528 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:46:02.529 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:46:02.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.534+0000 7f30cd56f700 1 -- 172.21.15.113:0/1336588408 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b006caa0 msgr2=0x7f30b006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:02.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.534+0000 7f30cd56f700 1 --2- 172.21.15.113:0/1336588408 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b006caa0 0x7f30b006ef60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f30c810bc70 tx=0x7f30bc00b040).stop 2024-04-12T05:46:02.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.534+0000 7f30cd56f700 1 -- 172.21.15.113:0/1336588408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c80fd950 msgr2=0x7f30c8110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:02.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.534+0000 7f30cd56f700 1 --2- 172.21.15.113:0/1336588408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c80fd950 0x7f30c8110330 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7f30b8009380 tx=0x7f30b8009cb0).stop 2024-04-12T05:46:02.530 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.534+0000 7f30cd56f700 1 -- 172.21.15.113:0/1336588408 shutdown_connections 2024-04-12T05:46:02.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.534+0000 7f30cd56f700 1 --2- 172.21.15.113:0/1336588408 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f30b006caa0 0x7f30b006ef60 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:02.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.534+0000 7f30cd56f700 1 --2- 172.21.15.113:0/1336588408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c80fd950 0x7f30c8110330 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:02.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.534+0000 7f30cd56f700 1 --2- 172.21.15.113:0/1336588408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30c80fe2b0 0x7f30c8110820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:02.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.534+0000 7f30cd56f700 1 -- 172.21.15.113:0/1336588408 >> 172.21.15.113:0/1336588408 conn(0x7f30c806c760 msgr2=0x7f30c80f7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:02.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.534+0000 7f30cd56f700 1 -- 172.21.15.113:0/1336588408 shutdown_connections 2024-04-12T05:46:02.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:02.534+0000 7f30cd56f700 1 -- 172.21.15.113:0/1336588408 wait complete. 2024-04-12T05:46:02.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:02 smithi113 ceph-mon[103371]: from='client.35822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:02.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:02 smithi113 ceph-mon[103371]: from='client.35826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:02.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:02 smithi113 ceph-mon[103371]: from='client.35830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:02.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:02 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/77876307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:02.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:02 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3442167817' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:46:02.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:02 smithi192 ceph-mon[85304]: from='client.35822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:02.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:02 smithi192 ceph-mon[85304]: from='client.35826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:02.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:02 smithi192 ceph-mon[85304]: from='client.35830 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:02.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:02 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/77876307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:02.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:02 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3442167817' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:46:03.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:03 smithi113 ceph-mon[103371]: pgmap v1017: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:03.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:03 smithi113 ceph-mon[103371]: from='client.35842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:03.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:03 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1336588408' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:46:03.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:03 smithi192 ceph-mon[85304]: pgmap v1017: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:03.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:03 smithi192 ceph-mon[85304]: from='client.35842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:03.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:03 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1336588408' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:46:05.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:05 smithi113 ceph-mon[103371]: pgmap v1018: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:05.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:05 smithi192 ceph-mon[85304]: pgmap v1018: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:07.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:07 smithi113 ceph-mon[103371]: pgmap v1019: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:07.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:07.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:07 smithi192 ceph-mon[85304]: pgmap v1019: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:07.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:09.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:09 smithi113 ceph-mon[103371]: pgmap v1020: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:09.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:09 smithi192 ceph-mon[85304]: pgmap v1020: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:11.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:11 smithi113 ceph-mon[103371]: pgmap v1021: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:11.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:11 smithi192 ceph-mon[85304]: pgmap v1021: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:13.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:13 smithi113 ceph-mon[103371]: pgmap v1022: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:13.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:13 smithi192 ceph-mon[85304]: pgmap v1022: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:15.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:15 smithi113 ceph-mon[103371]: pgmap v1023: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:15.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:15 smithi192 ceph-mon[85304]: pgmap v1023: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:17.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:17 smithi113 ceph-mon[103371]: pgmap v1024: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:17.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:17 smithi192 ceph-mon[85304]: pgmap v1024: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:19.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:19 smithi113 ceph-mon[103371]: pgmap v1025: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:19.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:19 smithi192 ceph-mon[85304]: pgmap v1025: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:21.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:21 smithi113 ceph-mon[103371]: pgmap v1026: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:21.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:21 smithi192 ceph-mon[85304]: pgmap v1026: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:22.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:22.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:23.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:23 smithi113 ceph-mon[103371]: pgmap v1027: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:23.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:23 smithi192 ceph-mon[85304]: pgmap v1027: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:25.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:25 smithi113 ceph-mon[103371]: pgmap v1028: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:25.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:25 smithi192 ceph-mon[85304]: pgmap v1028: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:27.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:27 smithi113 ceph-mon[103371]: pgmap v1029: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:27.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:27 smithi192 ceph-mon[85304]: pgmap v1029: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:29.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:29 smithi113 ceph-mon[103371]: pgmap v1030: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:29.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:29 smithi192 ceph-mon[85304]: pgmap v1030: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:31.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:31 smithi113 ceph-mon[103371]: pgmap v1031: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:31.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:31 smithi192 ceph-mon[85304]: pgmap v1031: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:32.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.645+0000 7fb88cfee700 1 -- 172.21.15.113:0/994629222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8880fe190 msgr2=0x7fb8880f7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:32.641 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.645+0000 7fb88cfee700 1 --2- 172.21.15.113:0/994629222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8880fe190 0x7fb8880f7970 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7fb8780047f0 tx=0x7fb878009cb0).stop 2024-04-12T05:46:32.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.645+0000 7fb88cfee700 1 -- 172.21.15.113:0/994629222 shutdown_connections 2024-04-12T05:46:32.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.645+0000 7fb88cfee700 1 --2- 172.21.15.113:0/994629222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8880fe190 0x7fb8880f7970 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:32.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.645+0000 7fb88cfee700 1 --2- 172.21.15.113:0/994629222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8880fd830 0x7fb8880fdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:32.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.645+0000 7fb88cfee700 1 -- 172.21.15.113:0/994629222 >> 172.21.15.113:0/994629222 conn(0x7fb8880f3450 msgr2=0x7fb8880f5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:32.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.645+0000 7fb88cfee700 1 -- 172.21.15.113:0/994629222 shutdown_connections 2024-04-12T05:46:32.642 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.645+0000 7fb88cfee700 1 -- 172.21.15.113:0/994629222 wait complete. 2024-04-12T05:46:32.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb88cfee700 1 Processor -- start 2024-04-12T05:46:32.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb88cfee700 1 -- start start 2024-04-12T05:46:32.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb88cfee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8880fd830 0x7fb88810e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:32.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb88cfee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8880fe190 0x7fb88810e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:32.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb88cfee700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88810eb60 con 0x7fb8880fe190 2024-04-12T05:46:32.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb88cfee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88810eca0 con 0x7fb8880fd830 2024-04-12T05:46:32.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb885d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8880fe190 0x7fb88810e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:32.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb885d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8880fe190 0x7fb88810e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43602/0 (socket says 172.21.15.113:43602) 2024-04-12T05:46:32.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb885d9b700 1 -- 172.21.15.113:0/2099505235 learned_addr learned my addr 172.21.15.113:0/2099505235 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:32.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb88659c700 1 --2- 172.21.15.113:0/2099505235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8880fd830 0x7fb88810e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:32.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb885d9b700 1 -- 172.21.15.113:0/2099505235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8880fd830 msgr2=0x7fb88810e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:32.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.646+0000 7fb885d9b700 1 --2- 172.21.15.113:0/2099505235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8880fd830 0x7fb88810e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:32.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.647+0000 7fb885d9b700 1 -- 172.21.15.113:0/2099505235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb878005040 con 0x7fb8880fe190 2024-04-12T05:46:32.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.647+0000 7fb885d9b700 1 --2- 172.21.15.113:0/2099505235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8880fe190 0x7fb88810e560 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7fb8780099f0 tx=0x7fb878009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:32.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.647+0000 7fb8777fe700 1 -- 172.21.15.113:0/2099505235 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb87800e030 con 0x7fb8880fe190 2024-04-12T05:46:32.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.647+0000 7fb8777fe700 1 -- 172.21.15.113:0/2099505235 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb878007e90 con 0x7fb8880fe190 2024-04-12T05:46:32.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.647+0000 7fb8777fe700 1 -- 172.21.15.113:0/2099505235 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb878018070 con 0x7fb8880fe190 2024-04-12T05:46:32.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.647+0000 7fb88cfee700 1 -- 172.21.15.113:0/2099505235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8881080c0 con 0x7fb8880fe190 2024-04-12T05:46:32.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.647+0000 7fb88cfee700 1 -- 172.21.15.113:0/2099505235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb888108620 con 0x7fb8880fe190 2024-04-12T05:46:32.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.648+0000 7fb88cfee700 1 -- 172.21.15.113:0/2099505235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88805f470 con 0x7fb8880fe190 2024-04-12T05:46:32.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.648+0000 7fb8777fe700 1 -- 172.21.15.113:0/2099505235 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb8780094c0 con 0x7fb8880fe190 2024-04-12T05:46:32.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.649+0000 7fb8777fe700 1 --2- 172.21.15.113:0/2099505235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb87006caf0 0x7fb87006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:32.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.649+0000 7fb8777fe700 1 -- 172.21.15.113:0/2099505235 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb87808d0c0 con 0x7fb8880fe190 2024-04-12T05:46:32.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.649+0000 7fb88659c700 1 --2- 172.21.15.113:0/2099505235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb87006caf0 0x7fb87006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:32.646 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.649+0000 7fb88659c700 1 --2- 172.21.15.113:0/2099505235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb87006caf0 0x7fb87006efb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fb87c004dc0 tx=0x7fb87c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:32.649 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.653+0000 7fb8777fe700 1 -- 172.21.15.113:0/2099505235 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb87805ab70 con 0x7fb8880fe190 2024-04-12T05:46:32.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.863+0000 7fb88cfee700 1 -- 172.21.15.113:0/2099505235 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb888109760 con 0x7fb87006caf0 2024-04-12T05:46:32.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.864+0000 7fb8777fe700 1 -- 172.21.15.113:0/2099505235 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb888109760 con 0x7fb87006caf0 2024-04-12T05:46:32.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.866+0000 7fb88cfee700 1 -- 172.21.15.113:0/2099505235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb87006caf0 msgr2=0x7fb87006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:32.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.866+0000 7fb88cfee700 1 --2- 172.21.15.113:0/2099505235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb87006caf0 0x7fb87006efb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fb87c004dc0 tx=0x7fb87c00b040).stop 2024-04-12T05:46:32.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.866+0000 7fb88cfee700 1 -- 172.21.15.113:0/2099505235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8880fe190 msgr2=0x7fb88810e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:32.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.866+0000 7fb88cfee700 1 --2- 172.21.15.113:0/2099505235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8880fe190 0x7fb88810e560 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7fb8780099f0 tx=0x7fb878009cb0).stop 2024-04-12T05:46:32.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.866+0000 7fb88cfee700 1 -- 172.21.15.113:0/2099505235 shutdown_connections 2024-04-12T05:46:32.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.866+0000 7fb88cfee700 1 --2- 172.21.15.113:0/2099505235 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb87006caf0 0x7fb87006efb0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:32.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.866+0000 7fb88cfee700 1 --2- 172.21.15.113:0/2099505235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb8880fd830 0x7fb88810e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:32.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.866+0000 7fb88cfee700 1 --2- 172.21.15.113:0/2099505235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8880fe190 0x7fb88810e560 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:32.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.866+0000 7fb88cfee700 1 -- 172.21.15.113:0/2099505235 >> 172.21.15.113:0/2099505235 conn(0x7fb8880f3450 msgr2=0x7fb8880fbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:32.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.866+0000 7fb88cfee700 1 -- 172.21.15.113:0/2099505235 shutdown_connections 2024-04-12T05:46:32.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.866+0000 7fb88cfee700 1 -- 172.21.15.113:0/2099505235 wait complete. 2024-04-12T05:46:32.872 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:46:32.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.965+0000 7f55f5559700 1 -- 172.21.15.113:0/2638143519 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f00fd150 msgr2=0x7f55f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:32.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.965+0000 7f55f5559700 1 --2- 172.21.15.113:0/2638143519 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f00fd150 0x7f55f00f6930 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f55e00047f0 tx=0x7f55e0009cb0).stop 2024-04-12T05:46:32.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.965+0000 7f55f5559700 1 -- 172.21.15.113:0/2638143519 shutdown_connections 2024-04-12T05:46:32.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.965+0000 7f55f5559700 1 --2- 172.21.15.113:0/2638143519 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f00fd150 0x7f55f00f6930 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:32.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.965+0000 7f55f5559700 1 --2- 172.21.15.113:0/2638143519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55f00fc7f0 0x7f55f00fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:32.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.965+0000 7f55f5559700 1 -- 172.21.15.113:0/2638143519 >> 172.21.15.113:0/2638143519 conn(0x7f55f00f23b0 msgr2=0x7f55f00f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:32.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.965+0000 7f55f5559700 1 -- 172.21.15.113:0/2638143519 shutdown_connections 2024-04-12T05:46:32.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.965+0000 7f55f5559700 1 -- 172.21.15.113:0/2638143519 wait complete. 2024-04-12T05:46:32.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.965+0000 7f55f5559700 1 Processor -- start 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55f5559700 1 -- start start 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55f5559700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f00fc7f0 0x7f55f010e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55f5559700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55f00fd150 0x7f55f010e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55f5559700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f010eb50 con 0x7f55f00fc7f0 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55f5559700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f010ec90 con 0x7f55f00fd150 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55eeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f00fc7f0 0x7f55f010e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55eeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f00fc7f0 0x7f55f010e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43618/0 (socket says 172.21.15.113:43618) 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55eeffd700 1 -- 172.21.15.113:0/2393758242 learned_addr learned my addr 172.21.15.113:0/2393758242 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55ee7fc700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55f00fd150 0x7f55f010e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55eeffd700 1 -- 172.21.15.113:0/2393758242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55f00fd150 msgr2=0x7f55f010e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55eeffd700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55f00fd150 0x7f55f010e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55eeffd700 1 -- 172.21.15.113:0/2393758242 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e0005040 con 0x7f55f00fc7f0 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55ee7fc700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55f00fd150 0x7f55f010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:46:32.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.966+0000 7f55eeffd700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f00fc7f0 0x7f55f010e060 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f55e40041d0 tx=0x7f55e4009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:32.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.967+0000 7f55d7fff700 1 -- 172.21.15.113:0/2393758242 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55e4007b40 con 0x7f55f00fc7f0 2024-04-12T05:46:32.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.967+0000 7f55f5559700 1 -- 172.21.15.113:0/2393758242 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f0108320 con 0x7f55f00fc7f0 2024-04-12T05:46:32.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.967+0000 7f55d7fff700 1 -- 172.21.15.113:0/2393758242 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55e400f040 con 0x7f55f00fc7f0 2024-04-12T05:46:32.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.967+0000 7f55d7fff700 1 -- 172.21.15.113:0/2393758242 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55e400d800 con 0x7f55f00fc7f0 2024-04-12T05:46:32.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.967+0000 7f55f5559700 1 -- 172.21.15.113:0/2393758242 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f0108860 con 0x7f55f00fc7f0 2024-04-12T05:46:32.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.968+0000 7f55f5559700 1 -- 172.21.15.113:0/2393758242 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f0060390 con 0x7f55f00fc7f0 2024-04-12T05:46:32.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.968+0000 7f55d7fff700 1 -- 172.21.15.113:0/2393758242 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f55e4007ca0 con 0x7f55f00fc7f0 2024-04-12T05:46:32.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.968+0000 7f55d7fff700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f55d806caa0 0x7f55d806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:32.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.969+0000 7f55d7fff700 1 -- 172.21.15.113:0/2393758242 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f55e408b950 con 0x7f55f00fc7f0 2024-04-12T05:46:32.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.969+0000 7f55ee7fc700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f55d806caa0 0x7f55d806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:32.966 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.969+0000 7f55ee7fc700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f55d806caa0 0x7f55d806ef60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f55e0004f10 tx=0x7f55e000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:32.969 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:32.973+0000 7f55d7fff700 1 -- 172.21.15.113:0/2393758242 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f55e4059400 con 0x7f55f00fc7f0 2024-04-12T05:46:33.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.178+0000 7f55f5559700 1 -- 172.21.15.113:0/2393758242 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f55f0108f00 con 0x7f55d806caa0 2024-04-12T05:46:33.175 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.179+0000 7f55d7fff700 1 -- 172.21.15.113:0/2393758242 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f55f0108f00 con 0x7f55d806caa0 2024-04-12T05:46:33.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.181+0000 7f55f5559700 1 -- 172.21.15.113:0/2393758242 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f55d806caa0 msgr2=0x7f55d806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.181+0000 7f55f5559700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f55d806caa0 0x7f55d806ef60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f55e0004f10 tx=0x7f55e000b040).stop 2024-04-12T05:46:33.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.181+0000 7f55f5559700 1 -- 172.21.15.113:0/2393758242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f00fc7f0 msgr2=0x7f55f010e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.181+0000 7f55f5559700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f00fc7f0 0x7f55f010e060 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f55e40041d0 tx=0x7f55e4009f60).stop 2024-04-12T05:46:33.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.182+0000 7f55f5559700 1 -- 172.21.15.113:0/2393758242 shutdown_connections 2024-04-12T05:46:33.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.182+0000 7f55f5559700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f55d806caa0 0x7f55d806ef60 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.182+0000 7f55f5559700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55f00fc7f0 0x7f55f010e060 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.182+0000 7f55f5559700 1 --2- 172.21.15.113:0/2393758242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55f00fd150 0x7f55f010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.178 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.182+0000 7f55f5559700 1 -- 172.21.15.113:0/2393758242 >> 172.21.15.113:0/2393758242 conn(0x7f55f00f23b0 msgr2=0x7f55f0061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:33.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.182+0000 7f55f5559700 1 -- 172.21.15.113:0/2393758242 shutdown_connections 2024-04-12T05:46:33.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.182+0000 7f55f5559700 1 -- 172.21.15.113:0/2393758242 wait complete. 2024-04-12T05:46:33.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.279+0000 7f5304187700 1 -- 172.21.15.113:0/1445296896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc0fe250 msgr2=0x7f52fc0f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.279+0000 7f5304187700 1 --2- 172.21.15.113:0/1445296896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc0fe250 0x7f52fc0f7a30 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f52f80047f0 tx=0x7f52f8009cb0).stop 2024-04-12T05:46:33.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.279+0000 7f5304187700 1 -- 172.21.15.113:0/1445296896 shutdown_connections 2024-04-12T05:46:33.275 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.279+0000 7f5304187700 1 --2- 172.21.15.113:0/1445296896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc0fe250 0x7f52fc0f7a30 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.279+0000 7f5304187700 1 --2- 172.21.15.113:0/1445296896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52fc0fd8f0 0x7f52fc0fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.279+0000 7f5304187700 1 -- 172.21.15.113:0/1445296896 >> 172.21.15.113:0/1445296896 conn(0x7f52fc0f34b0 msgr2=0x7f52fc0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:33.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.279+0000 7f5304187700 1 -- 172.21.15.113:0/1445296896 shutdown_connections 2024-04-12T05:46:33.276 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.279+0000 7f5304187700 1 -- 172.21.15.113:0/1445296896 wait complete. 2024-04-12T05:46:33.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5304187700 1 Processor -- start 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5304187700 1 -- start start 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5304187700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc0fd8f0 0x7f52fc06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5304187700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52fc0fe250 0x7f52fc06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5304187700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc06bab0 con 0x7f52fc0fd8f0 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5304187700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc06bbf0 con 0x7f52fc0fe250 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5301f23700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc0fd8f0 0x7f52fc06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5301f23700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc0fd8f0 0x7f52fc06b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43630/0 (socket says 172.21.15.113:43630) 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5301f23700 1 -- 172.21.15.113:0/2605989038 learned_addr learned my addr 172.21.15.113:0/2605989038 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5301722700 1 --2- 172.21.15.113:0/2605989038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52fc0fe250 0x7f52fc06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5301f23700 1 -- 172.21.15.113:0/2605989038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52fc0fe250 msgr2=0x7f52fc06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5301f23700 1 --2- 172.21.15.113:0/2605989038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52fc0fe250 0x7f52fc06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5301f23700 1 -- 172.21.15.113:0/2605989038 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f8005040 con 0x7f52fc0fd8f0 2024-04-12T05:46:33.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.280+0000 7f5301f23700 1 --2- 172.21.15.113:0/2605989038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc0fd8f0 0x7f52fc06b0d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f52ec00e8f0 tx=0x7f52ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:33.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.281+0000 7f52f2ffd700 1 -- 172.21.15.113:0/2605989038 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52ec007b40 con 0x7f52fc0fd8f0 2024-04-12T05:46:33.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.281+0000 7f5304187700 1 -- 172.21.15.113:0/2605989038 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52fc06bea0 con 0x7f52fc0fd8f0 2024-04-12T05:46:33.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.281+0000 7f52f2ffd700 1 -- 172.21.15.113:0/2605989038 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52ec00f040 con 0x7f52fc0fd8f0 2024-04-12T05:46:33.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.281+0000 7f52f2ffd700 1 -- 172.21.15.113:0/2605989038 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52ec014880 con 0x7f52fc0fd8f0 2024-04-12T05:46:33.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.281+0000 7f5304187700 1 -- 172.21.15.113:0/2605989038 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52fc114810 con 0x7f52fc0fd8f0 2024-04-12T05:46:33.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.281+0000 7f5304187700 1 -- 172.21.15.113:0/2605989038 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52fc05f590 con 0x7f52fc0fd8f0 2024-04-12T05:46:33.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.282+0000 7f52f2ffd700 1 -- 172.21.15.113:0/2605989038 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f52ec014b00 con 0x7f52fc0fd8f0 2024-04-12T05:46:33.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.282+0000 7f52f2ffd700 1 --2- 172.21.15.113:0/2605989038 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52e806caa0 0x7f52e806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:33.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.283+0000 7f5301722700 1 --2- 172.21.15.113:0/2605989038 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52e806caa0 0x7f52e806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:33.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.283+0000 7f52f2ffd700 1 -- 172.21.15.113:0/2605989038 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f52ec08ba50 con 0x7f52fc0fd8f0 2024-04-12T05:46:33.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.283+0000 7f5301722700 1 --2- 172.21.15.113:0/2605989038 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52e806caa0 0x7f52e806ef60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f52fc06c870 tx=0x7f52f8009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:33.283 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.287+0000 7f52f2ffd700 1 -- 172.21.15.113:0/2605989038 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f52ec055a70 con 0x7f52fc0fd8f0 2024-04-12T05:46:33.479 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.483+0000 7f5304187700 1 -- 172.21.15.113:0/2605989038 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f52fc06c6c0 con 0x7f52e806caa0 2024-04-12T05:46:33.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.493+0000 7f52f2ffd700 1 -- 172.21.15.113:0/2605989038 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f52fc06c6c0 con 0x7f52e806caa0 2024-04-12T05:46:33.489 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:46:33.489 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (34m) 11m ago 45m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:46:33.489 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (31m) 11m ago 45m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:46:33.489 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (31m) 11m ago 45m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (34m) 11m ago 45m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (27m) 11m ago 40m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (27m) 11m ago 40m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (27m) 11m ago 40m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (11m) 11m ago 40m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (35m) 11m ago 46m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (35m) 11m ago 44m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (32m) 11m ago 47m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (32m) 11m ago 44m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (35m) 11m ago 45m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (34m) 11m ago 44m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (31m) 11m ago 44m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:46:33.490 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (30m) 11m ago 43m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:46:33.491 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (30m) 11m ago 43m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:46:33.491 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (29m) 11m ago 42m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:46:33.491 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (29m) 11m ago 42m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:46:33.491 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (27m) 11m ago 41m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:46:33.491 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (34m) 11m ago 44m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:46:33.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.495+0000 7f5304187700 1 -- 172.21.15.113:0/2605989038 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52e806caa0 msgr2=0x7f52e806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.495+0000 7f5304187700 1 --2- 172.21.15.113:0/2605989038 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52e806caa0 0x7f52e806ef60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f52fc06c870 tx=0x7f52f8009cb0).stop 2024-04-12T05:46:33.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.495+0000 7f5304187700 1 -- 172.21.15.113:0/2605989038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc0fd8f0 msgr2=0x7f52fc06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.495+0000 7f5304187700 1 --2- 172.21.15.113:0/2605989038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc0fd8f0 0x7f52fc06b0d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f52ec00e8f0 tx=0x7f52ec00b040).stop 2024-04-12T05:46:33.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.495+0000 7f5304187700 1 -- 172.21.15.113:0/2605989038 shutdown_connections 2024-04-12T05:46:33.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.495+0000 7f5304187700 1 --2- 172.21.15.113:0/2605989038 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f52e806caa0 0x7f52e806ef60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.495+0000 7f5304187700 1 --2- 172.21.15.113:0/2605989038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc0fd8f0 0x7f52fc06b0d0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.495+0000 7f5304187700 1 --2- 172.21.15.113:0/2605989038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52fc0fe250 0x7f52fc06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.495+0000 7f5304187700 1 -- 172.21.15.113:0/2605989038 >> 172.21.15.113:0/2605989038 conn(0x7f52fc0f34b0 msgr2=0x7f52fc0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:33.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.495+0000 7f5304187700 1 -- 172.21.15.113:0/2605989038 shutdown_connections 2024-04-12T05:46:33.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.495+0000 7f5304187700 1 -- 172.21.15.113:0/2605989038 wait complete. 2024-04-12T05:46:33.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.591+0000 7f3cd62ef700 1 -- 172.21.15.113:0/622555630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd00fe270 msgr2=0x7f3cd00f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.591+0000 7f3cd62ef700 1 --2- 172.21.15.113:0/622555630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd00fe270 0x7f3cd00f7a50 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0x7f3cb80047f0 tx=0x7f3cb8009cb0).stop 2024-04-12T05:46:33.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.591+0000 7f3cd62ef700 1 -- 172.21.15.113:0/622555630 shutdown_connections 2024-04-12T05:46:33.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.591+0000 7f3cd62ef700 1 --2- 172.21.15.113:0/622555630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd00fe270 0x7f3cd00f7a50 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.591+0000 7f3cd62ef700 1 --2- 172.21.15.113:0/622555630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd00fd910 0x7f3cd00fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.591+0000 7f3cd62ef700 1 -- 172.21.15.113:0/622555630 >> 172.21.15.113:0/622555630 conn(0x7f3cd00f3530 msgr2=0x7f3cd00f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:33.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.591+0000 7f3cd62ef700 1 -- 172.21.15.113:0/622555630 shutdown_connections 2024-04-12T05:46:33.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.591+0000 7f3cd62ef700 1 -- 172.21.15.113:0/622555630 wait complete. 2024-04-12T05:46:33.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.591+0000 7f3cd62ef700 1 Processor -- start 2024-04-12T05:46:33.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3cd62ef700 1 -- start start 2024-04-12T05:46:33.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3cd62ef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd00fd910 0x7f3cd01101e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:33.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3cd62ef700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd00fe270 0x7f3cd01106d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3cd62ef700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0110cd0 con 0x7f3cd00fe270 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3cd62ef700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0110e10 con 0x7f3cd00fd910 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3ccf7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd00fe270 0x7f3cd01106d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3ccf7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd00fe270 0x7f3cd01106d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43638/0 (socket says 172.21.15.113:43638) 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3ccf7fe700 1 -- 172.21.15.113:0/1363387065 learned_addr learned my addr 172.21.15.113:0/1363387065 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3ccffff700 1 --2- 172.21.15.113:0/1363387065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd00fd910 0x7f3cd01101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3ccf7fe700 1 -- 172.21.15.113:0/1363387065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd00fd910 msgr2=0x7f3cd01101e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3ccf7fe700 1 --2- 172.21.15.113:0/1363387065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd00fd910 0x7f3cd01101e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3ccf7fe700 1 -- 172.21.15.113:0/1363387065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8005040 con 0x7f3cd00fe270 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.592+0000 7f3ccf7fe700 1 --2- 172.21.15.113:0/1363387065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd00fe270 0x7f3cd01106d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f3cb8000c00 tx=0x7f3cb8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.593+0000 7f3ccd7fa700 1 -- 172.21.15.113:0/1363387065 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cb800e030 con 0x7f3cd00fe270 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.593+0000 7f3cd62ef700 1 -- 172.21.15.113:0/1363387065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd010a440 con 0x7f3cd00fe270 2024-04-12T05:46:33.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.593+0000 7f3ccd7fa700 1 -- 172.21.15.113:0/1363387065 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cb8007e90 con 0x7f3cd00fe270 2024-04-12T05:46:33.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.593+0000 7f3ccd7fa700 1 -- 172.21.15.113:0/1363387065 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cb8018070 con 0x7f3cd00fe270 2024-04-12T05:46:33.591 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.593+0000 7f3cd62ef700 1 -- 172.21.15.113:0/1363387065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd010a9a0 con 0x7f3cd00fe270 2024-04-12T05:46:33.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.594+0000 7f3cd62ef700 1 -- 172.21.15.113:0/1363387065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd005f570 con 0x7f3cd00fe270 2024-04-12T05:46:33.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.594+0000 7f3ccd7fa700 1 -- 172.21.15.113:0/1363387065 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3cb801d420 con 0x7f3cd00fe270 2024-04-12T05:46:33.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.595+0000 7f3ccd7fa700 1 --2- 172.21.15.113:0/1363387065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3cbc06cab0 0x7f3cbc06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:33.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.595+0000 7f3ccffff700 1 --2- 172.21.15.113:0/1363387065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3cbc06cab0 0x7f3cbc06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:33.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.595+0000 7f3ccd7fa700 1 -- 172.21.15.113:0/1363387065 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3cb808d320 con 0x7f3cd00fe270 2024-04-12T05:46:33.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.595+0000 7f3ccffff700 1 --2- 172.21.15.113:0/1363387065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3cbc06cab0 0x7f3cbc06ef70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f3cc00041d0 tx=0x7f3cc0009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:33.595 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.599+0000 7f3ccd7fa700 1 -- 172.21.15.113:0/1363387065 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3cb805ae90 con 0x7f3cd00fe270 2024-04-12T05:46:33.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:33 smithi113 ceph-mon[103371]: pgmap v1032: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:33.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:33 smithi192 ceph-mon[85304]: pgmap v1032: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:33.859 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.863+0000 7f3cd62ef700 1 -- 172.21.15.113:0/1363387065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3cd00fca50 con 0x7f3cd00fe270 2024-04-12T05:46:33.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.863+0000 7f3ccd7fa700 1 -- 172.21.15.113:0/1363387065 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f3cb805aa20 con 0x7f3cd00fe270 2024-04-12T05:46:33.860 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:46:33.860 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:46:33.860 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:46:33.860 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:46:33.860 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:46:33.861 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:46:33.861 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:46:33.861 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:46:33.861 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:46:33.861 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:46:33.861 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:46:33.861 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:46:33.861 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:46:33.861 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:46:33.861 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:46:33.861 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:46:33.862 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:46:33.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.865+0000 7f3cd62ef700 1 -- 172.21.15.113:0/1363387065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3cbc06cab0 msgr2=0x7f3cbc06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.865+0000 7f3cd62ef700 1 --2- 172.21.15.113:0/1363387065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3cbc06cab0 0x7f3cbc06ef70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f3cc00041d0 tx=0x7f3cc0009040).stop 2024-04-12T05:46:33.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.865+0000 7f3cd62ef700 1 -- 172.21.15.113:0/1363387065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd00fe270 msgr2=0x7f3cd01106d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.865+0000 7f3cd62ef700 1 --2- 172.21.15.113:0/1363387065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd00fe270 0x7f3cd01106d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f3cb8000c00 tx=0x7f3cb8009cb0).stop 2024-04-12T05:46:33.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.866+0000 7f3cd62ef700 1 -- 172.21.15.113:0/1363387065 shutdown_connections 2024-04-12T05:46:33.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.866+0000 7f3cd62ef700 1 --2- 172.21.15.113:0/1363387065 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3cbc06cab0 0x7f3cbc06ef70 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.866+0000 7f3cd62ef700 1 --2- 172.21.15.113:0/1363387065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cd00fd910 0x7f3cd01101e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.866+0000 7f3cd62ef700 1 --2- 172.21.15.113:0/1363387065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd00fe270 0x7f3cd01106d0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.866+0000 7f3cd62ef700 1 -- 172.21.15.113:0/1363387065 >> 172.21.15.113:0/1363387065 conn(0x7f3cd00f3530 msgr2=0x7f3cd006b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:33.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.866+0000 7f3cd62ef700 1 -- 172.21.15.113:0/1363387065 shutdown_connections 2024-04-12T05:46:33.863 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.866+0000 7f3cd62ef700 1 -- 172.21.15.113:0/1363387065 wait complete. 2024-04-12T05:46:33.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.964+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1376958071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0e00fd8f0 msgr2=0x7fd0e00fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.964+0000 7fd0e7abb700 1 --2- 172.21.15.113:0/1376958071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0e00fd8f0 0x7fd0e00fdcd0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7fd0dc0047f0 tx=0x7fd0dc009cb0).stop 2024-04-12T05:46:33.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.964+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1376958071 shutdown_connections 2024-04-12T05:46:33.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.964+0000 7fd0e7abb700 1 --2- 172.21.15.113:0/1376958071 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0e00fe250 0x7fd0e00f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.964+0000 7fd0e7abb700 1 --2- 172.21.15.113:0/1376958071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0e00fd8f0 0x7fd0e00fdcd0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.964+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1376958071 >> 172.21.15.113:0/1376958071 conn(0x7fd0e00f34b0 msgr2=0x7fd0e00f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:33.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.964+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1376958071 shutdown_connections 2024-04-12T05:46:33.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.964+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1376958071 wait complete. 2024-04-12T05:46:33.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.964+0000 7fd0e7abb700 1 Processor -- start 2024-04-12T05:46:33.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.964+0000 7fd0e7abb700 1 -- start start 2024-04-12T05:46:33.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e7abb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0e00fd8f0 0x7fd0e010e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:33.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e7abb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0e00fe250 0x7fd0e010e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:33.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e7abb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e010ebe0 con 0x7fd0e00fd8f0 2024-04-12T05:46:33.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e7abb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e010ed20 con 0x7fd0e00fe250 2024-04-12T05:46:33.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e5857700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0e00fd8f0 0x7fd0e010e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:33.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e5857700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0e00fd8f0 0x7fd0e010e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43658/0 (socket says 172.21.15.113:43658) 2024-04-12T05:46:33.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e5857700 1 -- 172.21.15.113:0/1695844012 learned_addr learned my addr 172.21.15.113:0/1695844012 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:33.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e5056700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0e00fe250 0x7fd0e010e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:33.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e5857700 1 -- 172.21.15.113:0/1695844012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0e00fe250 msgr2=0x7fd0e010e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:33.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e5857700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0e00fe250 0x7fd0e010e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:33.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e5857700 1 -- 172.21.15.113:0/1695844012 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc005040 con 0x7fd0e00fd8f0 2024-04-12T05:46:33.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e5056700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0e00fe250 0x7fd0e010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:46:33.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e5857700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0e00fd8f0 0x7fd0e010e0f0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7fd0dc009380 tx=0x7fd0dc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:33.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0d6f78700 1 -- 172.21.15.113:0/1695844012 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0dc018070 con 0x7fd0e00fd8f0 2024-04-12T05:46:33.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0d6f78700 1 -- 172.21.15.113:0/1695844012 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0dc007e90 con 0x7fd0e00fd8f0 2024-04-12T05:46:33.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1695844012 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e0108350 con 0x7fd0e00fd8f0 2024-04-12T05:46:33.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0d6f78700 1 -- 172.21.15.113:0/1695844012 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0dc0123c0 con 0x7fd0e00fd8f0 2024-04-12T05:46:33.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.965+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1695844012 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e01088b0 con 0x7fd0e00fd8f0 2024-04-12T05:46:33.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.966+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1695844012 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e005f590 con 0x7fd0e00fd8f0 2024-04-12T05:46:33.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.967+0000 7fd0d6f78700 1 -- 172.21.15.113:0/1695844012 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd0dc01d420 con 0x7fd0e00fd8f0 2024-04-12T05:46:33.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.967+0000 7fd0d6f78700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0cc06caa0 0x7fd0cc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:33.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.967+0000 7fd0d6f78700 1 -- 172.21.15.113:0/1695844012 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd0dc08cad0 con 0x7fd0e00fd8f0 2024-04-12T05:46:33.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.967+0000 7fd0e5056700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0cc06caa0 0x7fd0cc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:33.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.968+0000 7fd0e5056700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0cc06caa0 0x7fd0cc06ef60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7fd0d0009f20 tx=0x7fd0d000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:33.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:33.972+0000 7fd0d6f78700 1 -- 172.21.15.113:0/1695844012 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd0dc05a580 con 0x7fd0e00fd8f0 2024-04-12T05:46:34.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.201+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1695844012 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd0e00ffc40 con 0x7fd0e00fd8f0 2024-04-12T05:46:34.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.201+0000 7fd0d6f78700 1 -- 172.21.15.113:0/1695844012 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fd0dc05a110 con 0x7fd0e00fd8f0 2024-04-12T05:46:34.198 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:46:34.198 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:46:34.198 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:46:34.198 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:46:34.198 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:46:34.198 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:46:34.198 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:46:34.198 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:46:34.199 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:46:34.200 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:46:34.201 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:46:34.201 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:46:34.201 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:46:34.201 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:46:34.201 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:46:34.201 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:34.201 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:34.201 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:34.201 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:46:34.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.203+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1695844012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0cc06caa0 msgr2=0x7fd0cc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:34.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.203+0000 7fd0e7abb700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0cc06caa0 0x7fd0cc06ef60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7fd0d0009f20 tx=0x7fd0d000b040).stop 2024-04-12T05:46:34.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.203+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1695844012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0e00fd8f0 msgr2=0x7fd0e010e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:34.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.203+0000 7fd0e7abb700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0e00fd8f0 0x7fd0e010e0f0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7fd0dc009380 tx=0x7fd0dc009cb0).stop 2024-04-12T05:46:34.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.203+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1695844012 shutdown_connections 2024-04-12T05:46:34.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.203+0000 7fd0e7abb700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0cc06caa0 0x7fd0cc06ef60 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.203+0000 7fd0e7abb700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0e00fd8f0 0x7fd0e010e0f0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.203+0000 7fd0e7abb700 1 --2- 172.21.15.113:0/1695844012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0e00fe250 0x7fd0e010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.203+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1695844012 >> 172.21.15.113:0/1695844012 conn(0x7fd0e00f34b0 msgr2=0x7fd0e00fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:34.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.203+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1695844012 shutdown_connections 2024-04-12T05:46:34.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.203+0000 7fd0e7abb700 1 -- 172.21.15.113:0/1695844012 wait complete. 2024-04-12T05:46:34.203 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:46:34.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.302+0000 7fb94878f700 1 -- 172.21.15.113:0/3148589741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9400fd8f0 msgr2=0x7fb9400fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:34.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.302+0000 7fb94878f700 1 --2- 172.21.15.113:0/3148589741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9400fd8f0 0x7fb9400fdcd0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7fb9300047f0 tx=0x7fb930009cb0).stop 2024-04-12T05:46:34.300 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.302+0000 7fb94878f700 1 -- 172.21.15.113:0/3148589741 shutdown_connections 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.302+0000 7fb94878f700 1 --2- 172.21.15.113:0/3148589741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9400fe250 0x7fb9400f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.302+0000 7fb94878f700 1 --2- 172.21.15.113:0/3148589741 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9400fd8f0 0x7fb9400fdcd0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.302+0000 7fb94878f700 1 -- 172.21.15.113:0/3148589741 >> 172.21.15.113:0/3148589741 conn(0x7fb9400f34b0 msgr2=0x7fb9400f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.302+0000 7fb94878f700 1 -- 172.21.15.113:0/3148589741 shutdown_connections 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.302+0000 7fb94878f700 1 -- 172.21.15.113:0/3148589741 wait complete. 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94878f700 1 Processor -- start 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94878f700 1 -- start start 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94878f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9400fd8f0 0x7fb94010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94878f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9400fe250 0x7fb94010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94878f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94010ead0 con 0x7fb9400fd8f0 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94878f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94010ec10 con 0x7fb9400fe250 2024-04-12T05:46:34.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94652b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9400fd8f0 0x7fb94010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:34.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94652b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9400fd8f0 0x7fb94010dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43664/0 (socket says 172.21.15.113:43664) 2024-04-12T05:46:34.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94652b700 1 -- 172.21.15.113:0/803591362 learned_addr learned my addr 172.21.15.113:0/803591362 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:34.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb945d2a700 1 --2- 172.21.15.113:0/803591362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9400fe250 0x7fb94010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:34.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94652b700 1 -- 172.21.15.113:0/803591362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9400fe250 msgr2=0x7fb94010e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:34.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94652b700 1 --2- 172.21.15.113:0/803591362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9400fe250 0x7fb94010e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.303+0000 7fb94652b700 1 -- 172.21.15.113:0/803591362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb930005040 con 0x7fb9400fd8f0 2024-04-12T05:46:34.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.304+0000 7fb94652b700 1 --2- 172.21.15.113:0/803591362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9400fd8f0 0x7fb94010dfe0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7fb9300043a0 tx=0x7fb930009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:34.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.304+0000 7fb9377fe700 1 -- 172.21.15.113:0/803591362 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb93000e030 con 0x7fb9400fd8f0 2024-04-12T05:46:34.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.304+0000 7fb94878f700 1 -- 172.21.15.113:0/803591362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb940108240 con 0x7fb9400fd8f0 2024-04-12T05:46:34.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.304+0000 7fb9377fe700 1 -- 172.21.15.113:0/803591362 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb930007e90 con 0x7fb9400fd8f0 2024-04-12T05:46:34.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.304+0000 7fb9377fe700 1 -- 172.21.15.113:0/803591362 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb930018070 con 0x7fb9400fd8f0 2024-04-12T05:46:34.303 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.304+0000 7fb94878f700 1 -- 172.21.15.113:0/803591362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9401087a0 con 0x7fb9400fd8f0 2024-04-12T05:46:34.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.305+0000 7fb94878f700 1 -- 172.21.15.113:0/803591362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94005f590 con 0x7fb9400fd8f0 2024-04-12T05:46:34.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.306+0000 7fb9377fe700 1 -- 172.21.15.113:0/803591362 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb9300094c0 con 0x7fb9400fd8f0 2024-04-12T05:46:34.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.306+0000 7fb9377fe700 1 --2- 172.21.15.113:0/803591362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb92c06cab0 0x7fb92c06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:34.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.306+0000 7fb9377fe700 1 -- 172.21.15.113:0/803591362 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fb93008d3b0 con 0x7fb9400fd8f0 2024-04-12T05:46:34.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.306+0000 7fb945d2a700 1 --2- 172.21.15.113:0/803591362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb92c06cab0 0x7fb92c06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:34.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.307+0000 7fb945d2a700 1 --2- 172.21.15.113:0/803591362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb92c06cab0 0x7fb92c06ef70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7fb940109920 tx=0x7fb93c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:34.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.310+0000 7fb9377fe700 1 -- 172.21.15.113:0/803591362 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fb93005ae60 con 0x7fb9400fd8f0 2024-04-12T05:46:34.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.516+0000 7fb94878f700 1 -- 172.21.15.113:0/803591362 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9400ffc40 con 0x7fb92c06cab0 2024-04-12T05:46:34.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.518+0000 7fb9377fe700 1 -- 172.21.15.113:0/803591362 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb9400ffc40 con 0x7fb92c06cab0 2024-04-12T05:46:34.514 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:46:34.514 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:46:34.514 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:46:34.514 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:46:34.514 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:46:34.514 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:46:34.515 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:46:34.515 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:46:34.515 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:46:34.515 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:46:34.515 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:46:34.515 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:46:34.515 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:46:34.515 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:46:34.515 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:46:34.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.520+0000 7fb94878f700 1 -- 172.21.15.113:0/803591362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb92c06cab0 msgr2=0x7fb92c06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:34.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.520+0000 7fb94878f700 1 --2- 172.21.15.113:0/803591362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb92c06cab0 0x7fb92c06ef70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7fb940109920 tx=0x7fb93c00b040).stop 2024-04-12T05:46:34.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.520+0000 7fb94878f700 1 -- 172.21.15.113:0/803591362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9400fd8f0 msgr2=0x7fb94010dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:34.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.520+0000 7fb94878f700 1 --2- 172.21.15.113:0/803591362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9400fd8f0 0x7fb94010dfe0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7fb9300043a0 tx=0x7fb930009cb0).stop 2024-04-12T05:46:34.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.520+0000 7fb94878f700 1 -- 172.21.15.113:0/803591362 shutdown_connections 2024-04-12T05:46:34.517 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.520+0000 7fb94878f700 1 --2- 172.21.15.113:0/803591362 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fb92c06cab0 0x7fb92c06ef70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.520+0000 7fb94878f700 1 --2- 172.21.15.113:0/803591362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9400fd8f0 0x7fb94010dfe0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.520+0000 7fb94878f700 1 --2- 172.21.15.113:0/803591362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9400fe250 0x7fb94010e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.520+0000 7fb94878f700 1 -- 172.21.15.113:0/803591362 >> 172.21.15.113:0/803591362 conn(0x7fb9400f34b0 msgr2=0x7fb9400fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:34.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.520+0000 7fb94878f700 1 -- 172.21.15.113:0/803591362 shutdown_connections 2024-04-12T05:46:34.518 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.520+0000 7fb94878f700 1 -- 172.21.15.113:0/803591362 wait complete. 2024-04-12T05:46:34.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.618+0000 7f08c5ccf700 1 -- 172.21.15.113:0/2728173817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08c00f91f0 msgr2=0x7f08c0101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:34.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.618+0000 7f08c5ccf700 1 --2- 172.21.15.113:0/2728173817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08c00f91f0 0x7f08c0101990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7f08b40047f0 tx=0x7f08b4009cb0).stop 2024-04-12T05:46:34.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.618+0000 7f08c5ccf700 1 -- 172.21.15.113:0/2728173817 shutdown_connections 2024-04-12T05:46:34.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.618+0000 7f08c5ccf700 1 --2- 172.21.15.113:0/2728173817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08c00f91f0 0x7f08c0101990 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.618+0000 7f08c5ccf700 1 --2- 172.21.15.113:0/2728173817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08c00f8920 0x7f08c00f8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.618+0000 7f08c5ccf700 1 -- 172.21.15.113:0/2728173817 >> 172.21.15.113:0/2728173817 conn(0x7f08c0070590 msgr2=0x7f08c00f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:34.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.618+0000 7f08c5ccf700 1 -- 172.21.15.113:0/2728173817 shutdown_connections 2024-04-12T05:46:34.615 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.618+0000 7f08c5ccf700 1 -- 172.21.15.113:0/2728173817 wait complete. 2024-04-12T05:46:34.617 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08c5ccf700 1 Processor -- start 2024-04-12T05:46:34.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08c5ccf700 1 -- start start 2024-04-12T05:46:34.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08c5ccf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08c00f8920 0x7f08c010e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:34.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08c5ccf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08c00f91f0 0x7f08c010e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:34.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08c5ccf700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c010eb00 con 0x7f08c00f8920 2024-04-12T05:46:34.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08c5ccf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c010ec40 con 0x7f08c00f91f0 2024-04-12T05:46:34.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08bf7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08c00f8920 0x7f08c010e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:34.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08bf7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08c00f8920 0x7f08c010e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43690/0 (socket says 172.21.15.113:43690) 2024-04-12T05:46:34.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08bf7fe700 1 -- 172.21.15.113:0/1442667376 learned_addr learned my addr 172.21.15.113:0/1442667376 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:46:34.618 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08beffd700 1 --2- 172.21.15.113:0/1442667376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08c00f91f0 0x7f08c010e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:34.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08bf7fe700 1 -- 172.21.15.113:0/1442667376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08c00f91f0 msgr2=0x7f08c010e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:34.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08bf7fe700 1 --2- 172.21.15.113:0/1442667376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08c00f91f0 0x7f08c010e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.619+0000 7f08bf7fe700 1 -- 172.21.15.113:0/1442667376 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b4005040 con 0x7f08c00f8920 2024-04-12T05:46:34.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.620+0000 7f08bf7fe700 1 --2- 172.21.15.113:0/1442667376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08c00f8920 0x7f08c010e010 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f08b000c740 tx=0x7f08b0007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:34.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.620+0000 7f08bcff9700 1 -- 172.21.15.113:0/1442667376 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08b000e680 con 0x7f08c00f8920 2024-04-12T05:46:34.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.620+0000 7f08c5ccf700 1 -- 172.21.15.113:0/1442667376 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c01082d0 con 0x7f08c00f8920 2024-04-12T05:46:34.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.620+0000 7f08bcff9700 1 -- 172.21.15.113:0/1442667376 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08b000ec90 con 0x7f08c00f8920 2024-04-12T05:46:34.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.620+0000 7f08bcff9700 1 -- 172.21.15.113:0/1442667376 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08b0013d30 con 0x7f08c00f8920 2024-04-12T05:46:34.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.620+0000 7f08c5ccf700 1 -- 172.21.15.113:0/1442667376 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c0108810 con 0x7f08c00f8920 2024-04-12T05:46:34.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.621+0000 7f08c5ccf700 1 -- 172.21.15.113:0/1442667376 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c005f590 con 0x7f08c00f8920 2024-04-12T05:46:34.619 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.621+0000 7f08bcff9700 1 -- 172.21.15.113:0/1442667376 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f08b0013410 con 0x7f08c00f8920 2024-04-12T05:46:34.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.622+0000 7f08bcff9700 1 --2- 172.21.15.113:0/1442667376 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08a806caa0 0x7f08a806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:46:34.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.622+0000 7f08bcff9700 1 -- 172.21.15.113:0/1442667376 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f08b008c070 con 0x7f08c00f8920 2024-04-12T05:46:34.620 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.622+0000 7f08beffd700 1 --2- 172.21.15.113:0/1442667376 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08a806caa0 0x7f08a806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:46:34.621 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.622+0000 7f08beffd700 1 --2- 172.21.15.113:0/1442667376 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08a806caa0 0x7f08a806ef60 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f08b4007cf0 tx=0x7f08b400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:46:34.622 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.626+0000 7f08bcff9700 1 -- 172.21.15.113:0/1442667376 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f08b0059b20 con 0x7f08c00f8920 2024-04-12T05:46:34.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:34 smithi113 ceph-mon[103371]: from='client.35850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:34.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:34 smithi113 ceph-mon[103371]: from='client.35854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:34.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:34 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1363387065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:34.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:34 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1695844012' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:46:34.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:34 smithi192 ceph-mon[85304]: from='client.35850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:34.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:34 smithi192 ceph-mon[85304]: from='client.35854 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:34.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:34 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1363387065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:46:34.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:34 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1695844012' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:46:34.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.895+0000 7f08c5ccf700 1 -- 172.21.15.113:0/1442667376 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f08c0047540 con 0x7f08c00f8920 2024-04-12T05:46:34.892 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.895+0000 7f08bcff9700 1 -- 172.21.15.113:0/1442667376 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f08b00596b0 con 0x7f08c00f8920 2024-04-12T05:46:34.892 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:46:34.892 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:46:34.892 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:46:34.892 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:46:34.892 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:46:34.892 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:46:34.893 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:46:34.894 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:46:34.894 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:46:34.894 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:46:34.894 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:46:34.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.897+0000 7f08c5ccf700 1 -- 172.21.15.113:0/1442667376 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08a806caa0 msgr2=0x7f08a806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:34.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.897+0000 7f08c5ccf700 1 --2- 172.21.15.113:0/1442667376 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08a806caa0 0x7f08a806ef60 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f08b4007cf0 tx=0x7f08b400b040).stop 2024-04-12T05:46:34.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.897+0000 7f08c5ccf700 1 -- 172.21.15.113:0/1442667376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08c00f8920 msgr2=0x7f08c010e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:46:34.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.897+0000 7f08c5ccf700 1 --2- 172.21.15.113:0/1442667376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08c00f8920 0x7f08c010e010 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f08b000c740 tx=0x7f08b0007ef0).stop 2024-04-12T05:46:34.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.898+0000 7f08c5ccf700 1 -- 172.21.15.113:0/1442667376 shutdown_connections 2024-04-12T05:46:34.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.898+0000 7f08c5ccf700 1 --2- 172.21.15.113:0/1442667376 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f08a806caa0 0x7f08a806ef60 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.898+0000 7f08c5ccf700 1 --2- 172.21.15.113:0/1442667376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08c00f8920 0x7f08c010e010 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.898+0000 7f08c5ccf700 1 --2- 172.21.15.113:0/1442667376 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08c00f91f0 0x7f08c010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:46:34.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.898+0000 7f08c5ccf700 1 -- 172.21.15.113:0/1442667376 >> 172.21.15.113:0/1442667376 conn(0x7f08c0070590 msgr2=0x7f08c00fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:46:34.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.898+0000 7f08c5ccf700 1 -- 172.21.15.113:0/1442667376 shutdown_connections 2024-04-12T05:46:34.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:46:34.898+0000 7f08c5ccf700 1 -- 172.21.15.113:0/1442667376 wait complete. 2024-04-12T05:46:35.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:35 smithi113 ceph-mon[103371]: from='client.35858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:35.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:35 smithi113 ceph-mon[103371]: pgmap v1033: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:35.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:35 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1442667376' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:46:35.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:35 smithi192 ceph-mon[85304]: from='client.35858 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:35.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:35 smithi192 ceph-mon[85304]: pgmap v1033: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:35.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:35 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1442667376' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:46:36.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:36 smithi113 ceph-mon[103371]: from='client.35870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:36.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:36 smithi192 ceph-mon[85304]: from='client.35870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:46:37.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:37 smithi113 ceph-mon[103371]: pgmap v1034: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:37.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:37.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:37 smithi192 ceph-mon[85304]: pgmap v1034: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:37.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:39 smithi113 ceph-mon[103371]: pgmap v1035: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:39 smithi192 ceph-mon[85304]: pgmap v1035: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:41 smithi113 ceph-mon[103371]: pgmap v1036: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:41.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:41 smithi192 ceph-mon[85304]: pgmap v1036: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:43.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:43 smithi113 ceph-mon[103371]: pgmap v1037: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:43.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:43 smithi192 ceph-mon[85304]: pgmap v1037: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:45.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:45 smithi113 ceph-mon[103371]: pgmap v1038: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:45.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:45 smithi192 ceph-mon[85304]: pgmap v1038: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:47.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:47 smithi113 ceph-mon[103371]: pgmap v1039: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:47.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:47 smithi192 ceph-mon[85304]: pgmap v1039: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:49.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:49 smithi113 ceph-mon[103371]: pgmap v1040: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:49.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:49 smithi192 ceph-mon[85304]: pgmap v1040: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:51.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:51 smithi113 ceph-mon[103371]: pgmap v1041: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:51.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:51 smithi192 ceph-mon[85304]: pgmap v1041: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:52.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:52.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:46:53.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:53 smithi113 ceph-mon[103371]: pgmap v1042: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:53.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:53 smithi192 ceph-mon[85304]: pgmap v1042: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:55.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:55 smithi113 ceph-mon[103371]: pgmap v1043: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:55.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:55 smithi192 ceph-mon[85304]: pgmap v1043: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:57.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:57 smithi113 ceph-mon[103371]: pgmap v1044: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:57.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:57 smithi192 ceph-mon[85304]: pgmap v1044: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:58.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:46:58 smithi113 ceph-mon[103371]: pgmap v1045: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:46:58.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:46:58 smithi192 ceph-mon[85304]: pgmap v1045: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:01.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:01 smithi113 ceph-mon[103371]: pgmap v1046: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:01.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:01 smithi192 ceph-mon[85304]: pgmap v1046: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:03.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:03 smithi113 ceph-mon[103371]: pgmap v1047: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:03.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:03 smithi192 ceph-mon[85304]: pgmap v1047: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:05.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.008+0000 7f7054f00700 1 -- 172.21.15.113:0/1243184246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70500fc7f0 msgr2=0x7f70500fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.008+0000 7f7054f00700 1 --2- 172.21.15.113:0/1243184246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70500fc7f0 0x7f70500fcbd0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f70380047f0 tx=0x7f7038009cb0).stop 2024-04-12T05:47:05.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.008+0000 7f7054f00700 1 -- 172.21.15.113:0/1243184246 shutdown_connections 2024-04-12T05:47:05.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.008+0000 7f7054f00700 1 --2- 172.21.15.113:0/1243184246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70500fd150 0x7f70500f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.008+0000 7f7054f00700 1 --2- 172.21.15.113:0/1243184246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70500fc7f0 0x7f70500fcbd0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.008+0000 7f7054f00700 1 -- 172.21.15.113:0/1243184246 >> 172.21.15.113:0/1243184246 conn(0x7f7050070590 msgr2=0x7f70500f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:05.007 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.008+0000 7f7054f00700 1 -- 172.21.15.113:0/1243184246 shutdown_connections 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.009+0000 7f7054f00700 1 -- 172.21.15.113:0/1243184246 wait complete. 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.009+0000 7f7054f00700 1 Processor -- start 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.009+0000 7f7054f00700 1 -- start start 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f7054f00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70500fc7f0 0x7f70501090d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f7054f00700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70500fd150 0x7f70501095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f7054f00700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7050109ab0 con 0x7f70500fd150 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f7054f00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7050109bf0 con 0x7f70500fc7f0 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f704dd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70500fd150 0x7f70501095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f704dd9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70500fd150 0x7f70501095c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49488/0 (socket says 172.21.15.113:49488) 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f704dd9b700 1 -- 172.21.15.113:0/4227520928 learned_addr learned my addr 172.21.15.113:0/4227520928 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f704e59c700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70500fc7f0 0x7f70501090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.008 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f704dd9b700 1 -- 172.21.15.113:0/4227520928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70500fc7f0 msgr2=0x7f70501090d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f704dd9b700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70500fc7f0 0x7f70501090d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f704dd9b700 1 -- 172.21.15.113:0/4227520928 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7038005040 con 0x7f70500fd150 2024-04-12T05:47:05.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f704e59c700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70500fc7f0 0x7f70501090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:47:05.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f704dd9b700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70500fd150 0x7f70501095c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f7040002780 tx=0x7f7040009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:05.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f70477fe700 1 -- 172.21.15.113:0/4227520928 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7040007b40 con 0x7f70500fd150 2024-04-12T05:47:05.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f70477fe700 1 -- 172.21.15.113:0/4227520928 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f704000f040 con 0x7f70500fd150 2024-04-12T05:47:05.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f7054f00700 1 -- 172.21.15.113:0/4227520928 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7050109ed0 con 0x7f70500fd150 2024-04-12T05:47:05.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f70477fe700 1 -- 172.21.15.113:0/4227520928 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7040014780 con 0x7f70500fd150 2024-04-12T05:47:05.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.010+0000 7f7054f00700 1 -- 172.21.15.113:0/4227520928 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7050112460 con 0x7f70500fd150 2024-04-12T05:47:05.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.012+0000 7f7054f00700 1 -- 172.21.15.113:0/4227520928 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705005f590 con 0x7f70500fd150 2024-04-12T05:47:05.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.012+0000 7f70477fe700 1 -- 172.21.15.113:0/4227520928 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7040007ca0 con 0x7f70500fd150 2024-04-12T05:47:05.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.012+0000 7f70477fe700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f703c06caa0 0x7f703c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.013+0000 7f70477fe700 1 -- 172.21.15.113:0/4227520928 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f704008bd00 con 0x7f70500fd150 2024-04-12T05:47:05.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.013+0000 7f704e59c700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f703c06caa0 0x7f703c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.011 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.013+0000 7f704e59c700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f703c06caa0 0x7f703c06ef60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f7038004540 tx=0x7f703800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:05.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.017+0000 7f70477fe700 1 -- 172.21.15.113:0/4227520928 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f704008f030 con 0x7f70500fd150 2024-04-12T05:47:05.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.228+0000 7f7054f00700 1 -- 172.21.15.113:0/4227520928 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7050059120 con 0x7f703c06caa0 2024-04-12T05:47:05.225 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.229+0000 7f70477fe700 1 -- 172.21.15.113:0/4227520928 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7050059120 con 0x7f703c06caa0 2024-04-12T05:47:05.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.231+0000 7f7054f00700 1 -- 172.21.15.113:0/4227520928 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f703c06caa0 msgr2=0x7f703c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.231+0000 7f7054f00700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f703c06caa0 0x7f703c06ef60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f7038004540 tx=0x7f703800b040).stop 2024-04-12T05:47:05.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.231+0000 7f7054f00700 1 -- 172.21.15.113:0/4227520928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70500fd150 msgr2=0x7f70501095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.231+0000 7f7054f00700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70500fd150 0x7f70501095c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f7040002780 tx=0x7f7040009ee0).stop 2024-04-12T05:47:05.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.231+0000 7f7054f00700 1 -- 172.21.15.113:0/4227520928 shutdown_connections 2024-04-12T05:47:05.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.231+0000 7f7054f00700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f703c06caa0 0x7f703c06ef60 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.231+0000 7f7054f00700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70500fc7f0 0x7f70501090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.231+0000 7f7054f00700 1 --2- 172.21.15.113:0/4227520928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70500fd150 0x7f70501095c0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.231+0000 7f7054f00700 1 -- 172.21.15.113:0/4227520928 >> 172.21.15.113:0/4227520928 conn(0x7f7050070590 msgr2=0x7f70500fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:05.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.232+0000 7f7054f00700 1 -- 172.21.15.113:0/4227520928 shutdown_connections 2024-04-12T05:47:05.228 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.232+0000 7f7054f00700 1 -- 172.21.15.113:0/4227520928 wait complete. 2024-04-12T05:47:05.237 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:47:05.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.331+0000 7fd4822d4700 1 -- 172.21.15.113:0/92102339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd47c0f9200 msgr2=0x7fd47c101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.331+0000 7fd4822d4700 1 --2- 172.21.15.113:0/92102339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd47c0f9200 0x7fd47c101910 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7fd46c0047f0 tx=0x7fd46c009cb0).stop 2024-04-12T05:47:05.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.332+0000 7fd4822d4700 1 -- 172.21.15.113:0/92102339 shutdown_connections 2024-04-12T05:47:05.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.332+0000 7fd4822d4700 1 --2- 172.21.15.113:0/92102339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd47c0f9200 0x7fd47c101910 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.332+0000 7fd4822d4700 1 --2- 172.21.15.113:0/92102339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd47c0f88a0 0x7fd47c0f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.332+0000 7fd4822d4700 1 -- 172.21.15.113:0/92102339 >> 172.21.15.113:0/92102339 conn(0x7fd47c0f23b0 msgr2=0x7fd47c0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:05.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.332+0000 7fd4822d4700 1 -- 172.21.15.113:0/92102339 shutdown_connections 2024-04-12T05:47:05.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.332+0000 7fd4822d4700 1 -- 172.21.15.113:0/92102339 wait complete. 2024-04-12T05:47:05.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.332+0000 7fd4822d4700 1 Processor -- start 2024-04-12T05:47:05.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.332+0000 7fd4822d4700 1 -- start start 2024-04-12T05:47:05.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd4822d4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd47c0f88a0 0x7fd47c10e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd4822d4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd47c0f9200 0x7fd47c10e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd4822d4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c10eb40 con 0x7fd47c0f9200 2024-04-12T05:47:05.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd4822d4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c10ec80 con 0x7fd47c0f88a0 2024-04-12T05:47:05.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd47b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd47c0f9200 0x7fd47c10e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd47b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd47c0f9200 0x7fd47c10e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49508/0 (socket says 172.21.15.113:49508) 2024-04-12T05:47:05.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd47b7fe700 1 -- 172.21.15.113:0/3676796924 learned_addr learned my addr 172.21.15.113:0/3676796924 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd47bfff700 1 --2- 172.21.15.113:0/3676796924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd47c0f88a0 0x7fd47c10e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd47b7fe700 1 -- 172.21.15.113:0/3676796924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd47c0f88a0 msgr2=0x7fd47c10e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd47b7fe700 1 --2- 172.21.15.113:0/3676796924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd47c0f88a0 0x7fd47c10e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd47b7fe700 1 -- 172.21.15.113:0/3676796924 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c005040 con 0x7fd47c0f9200 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd47b7fe700 1 --2- 172.21.15.113:0/3676796924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd47c0f9200 0x7fd47c10e540 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7fd46c00b120 tx=0x7fd46c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd479775700 1 -- 172.21.15.113:0/3676796924 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd46c00e030 con 0x7fd47c0f9200 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd4822d4700 1 -- 172.21.15.113:0/3676796924 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47c1082b0 con 0x7fd47c0f9200 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd479775700 1 -- 172.21.15.113:0/3676796924 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd46c007e90 con 0x7fd47c0f9200 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd479775700 1 -- 172.21.15.113:0/3676796924 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd46c018070 con 0x7fd47c0f9200 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.333+0000 7fd4822d4700 1 -- 172.21.15.113:0/3676796924 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47c108790 con 0x7fd47c0f9200 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.334+0000 7fd4822d4700 1 -- 172.21.15.113:0/3676796924 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47c060390 con 0x7fd47c0f9200 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.335+0000 7fd479775700 1 -- 172.21.15.113:0/3676796924 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd46c01d420 con 0x7fd47c0f9200 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.335+0000 7fd479775700 1 --2- 172.21.15.113:0/3676796924 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd46406caa0 0x7fd46406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.335+0000 7fd479775700 1 -- 172.21.15.113:0/3676796924 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd46c08d240 con 0x7fd47c0f9200 2024-04-12T05:47:05.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.335+0000 7fd47bfff700 1 --2- 172.21.15.113:0/3676796924 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd46406caa0 0x7fd46406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.335+0000 7fd47bfff700 1 --2- 172.21.15.113:0/3676796924 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd46406caa0 0x7fd46406ef60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fd470004d20 tx=0x7fd470009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:05.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.340+0000 7fd479775700 1 -- 172.21.15.113:0/3676796924 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd46c05acf0 con 0x7fd47c0f9200 2024-04-12T05:47:05.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:05 smithi113 ceph-mon[103371]: pgmap v1048: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:05.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:05 smithi192 ceph-mon[85304]: pgmap v1048: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:05.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.544+0000 7fd4822d4700 1 -- 172.21.15.113:0/3676796924 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd47c1096d0 con 0x7fd46406caa0 2024-04-12T05:47:05.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.546+0000 7fd479775700 1 -- 172.21.15.113:0/3676796924 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd47c1096d0 con 0x7fd46406caa0 2024-04-12T05:47:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.548+0000 7fd4822d4700 1 -- 172.21.15.113:0/3676796924 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd46406caa0 msgr2=0x7fd46406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.548+0000 7fd4822d4700 1 --2- 172.21.15.113:0/3676796924 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd46406caa0 0x7fd46406ef60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fd470004d20 tx=0x7fd470009040).stop 2024-04-12T05:47:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.548+0000 7fd4822d4700 1 -- 172.21.15.113:0/3676796924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd47c0f9200 msgr2=0x7fd47c10e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.548+0000 7fd4822d4700 1 --2- 172.21.15.113:0/3676796924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd47c0f9200 0x7fd47c10e540 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7fd46c00b120 tx=0x7fd46c009cb0).stop 2024-04-12T05:47:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.548+0000 7fd4822d4700 1 -- 172.21.15.113:0/3676796924 shutdown_connections 2024-04-12T05:47:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.548+0000 7fd4822d4700 1 --2- 172.21.15.113:0/3676796924 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd46406caa0 0x7fd46406ef60 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.548+0000 7fd4822d4700 1 --2- 172.21.15.113:0/3676796924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd47c0f88a0 0x7fd47c10e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.548+0000 7fd4822d4700 1 --2- 172.21.15.113:0/3676796924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd47c0f9200 0x7fd47c10e540 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.548+0000 7fd4822d4700 1 -- 172.21.15.113:0/3676796924 >> 172.21.15.113:0/3676796924 conn(0x7fd47c0f23b0 msgr2=0x7fd47c0fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.548+0000 7fd4822d4700 1 -- 172.21.15.113:0/3676796924 shutdown_connections 2024-04-12T05:47:05.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.548+0000 7fd4822d4700 1 -- 172.21.15.113:0/3676796924 wait complete. 2024-04-12T05:47:05.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.646+0000 7fbefa046700 1 -- 172.21.15.113:0/4008764749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef40fd8f0 msgr2=0x7fbef40fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.646+0000 7fbefa046700 1 --2- 172.21.15.113:0/4008764749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef40fd8f0 0x7fbef40fdcd0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7fbee80047f0 tx=0x7fbee8009cb0).stop 2024-04-12T05:47:05.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.646+0000 7fbefa046700 1 -- 172.21.15.113:0/4008764749 shutdown_connections 2024-04-12T05:47:05.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.646+0000 7fbefa046700 1 --2- 172.21.15.113:0/4008764749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbef40fe250 0x7fbef40f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.646+0000 7fbefa046700 1 --2- 172.21.15.113:0/4008764749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef40fd8f0 0x7fbef40fdcd0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.646+0000 7fbefa046700 1 -- 172.21.15.113:0/4008764749 >> 172.21.15.113:0/4008764749 conn(0x7fbef40f34b0 msgr2=0x7fbef40f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:05.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.646+0000 7fbefa046700 1 -- 172.21.15.113:0/4008764749 shutdown_connections 2024-04-12T05:47:05.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.646+0000 7fbefa046700 1 -- 172.21.15.113:0/4008764749 wait complete. 2024-04-12T05:47:05.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.647+0000 7fbefa046700 1 Processor -- start 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.647+0000 7fbefa046700 1 -- start start 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.647+0000 7fbefa046700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbef40fd8f0 0x7fbef410e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.647+0000 7fbefa046700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef40fe250 0x7fbef410e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.647+0000 7fbefa046700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef410ec00 con 0x7fbef40fe250 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.647+0000 7fbefa046700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef410ed40 con 0x7fbef40fd8f0 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.647+0000 7fbef2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef40fe250 0x7fbef410e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.647+0000 7fbef2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef40fe250 0x7fbef410e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49536/0 (socket says 172.21.15.113:49536) 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.647+0000 7fbef2ffd700 1 -- 172.21.15.113:0/971180549 learned_addr learned my addr 172.21.15.113:0/971180549 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.648+0000 7fbef37fe700 1 --2- 172.21.15.113:0/971180549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbef40fd8f0 0x7fbef410e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.648+0000 7fbef2ffd700 1 -- 172.21.15.113:0/971180549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbef40fd8f0 msgr2=0x7fbef410e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.648+0000 7fbef2ffd700 1 --2- 172.21.15.113:0/971180549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbef40fd8f0 0x7fbef410e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.648+0000 7fbef2ffd700 1 -- 172.21.15.113:0/971180549 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee8005040 con 0x7fbef40fe250 2024-04-12T05:47:05.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.648+0000 7fbef2ffd700 1 --2- 172.21.15.113:0/971180549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef40fe250 0x7fbef410e600 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7fbee400e940 tx=0x7fbee400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.648+0000 7fbef0ff9700 1 -- 172.21.15.113:0/971180549 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbee400ee80 con 0x7fbef40fe250 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.648+0000 7fbefa046700 1 -- 172.21.15.113:0/971180549 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef4108370 con 0x7fbef40fe250 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.648+0000 7fbef0ff9700 1 -- 172.21.15.113:0/971180549 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbee400f040 con 0x7fbef40fe250 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.648+0000 7fbef0ff9700 1 -- 172.21.15.113:0/971180549 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbee400c810 con 0x7fbef40fe250 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.648+0000 7fbefa046700 1 -- 172.21.15.113:0/971180549 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef4108930 con 0x7fbef40fe250 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.649+0000 7fbefa046700 1 -- 172.21.15.113:0/971180549 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef405f590 con 0x7fbef40fe250 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.649+0000 7fbef0ff9700 1 -- 172.21.15.113:0/971180549 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbee40090d0 con 0x7fbef40fe250 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.650+0000 7fbef0ff9700 1 --2- 172.21.15.113:0/971180549 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbedc06ca30 0x7fbedc06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.650+0000 7fbef0ff9700 1 -- 172.21.15.113:0/971180549 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbee408b970 con 0x7fbef40fe250 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.650+0000 7fbef37fe700 1 --2- 172.21.15.113:0/971180549 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbedc06ca30 0x7fbedc06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.650+0000 7fbef37fe700 1 --2- 172.21.15.113:0/971180549 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbedc06ca30 0x7fbedc06eef0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fbee8004e60 tx=0x7fbee800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:05.673 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.654+0000 7fbef0ff9700 1 -- 172.21.15.113:0/971180549 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbee4059420 con 0x7fbef40fe250 2024-04-12T05:47:05.848 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.852+0000 7fbefa046700 1 -- 172.21.15.113:0/971180549 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbef4108ba0 con 0x7fbedc06ca30 2024-04-12T05:47:05.858 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.862+0000 7fbef0ff9700 1 -- 172.21.15.113:0/971180549 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbef4108ba0 con 0x7fbedc06ca30 2024-04-12T05:47:05.858 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:47:05.858 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (34m) 12m ago 46m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:47:05.858 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (32m) 12m ago 46m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:47:05.858 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (32m) 12m ago 45m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:47:05.858 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (34m) 12m ago 45m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:47:05.858 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (28m) 12m ago 40m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:47:05.858 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (27m) 12m ago 40m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:47:05.858 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (27m) 12m ago 40m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:47:05.858 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (12m) 12m ago 40m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (36m) 12m ago 47m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (35m) 12m ago 45m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (33m) 12m ago 47m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (32m) 12m ago 45m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (35m) 12m ago 45m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (35m) 12m ago 45m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (31m) 12m ago 44m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (31m) 12m ago 44m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (30m) 12m ago 43m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (30m) 12m ago 43m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (29m) 12m ago 42m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:47:05.859 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (28m) 12m ago 42m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:47:05.860 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (35m) 12m ago 45m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:47:05.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.864+0000 7fbefa046700 1 -- 172.21.15.113:0/971180549 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbedc06ca30 msgr2=0x7fbedc06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.864+0000 7fbefa046700 1 --2- 172.21.15.113:0/971180549 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbedc06ca30 0x7fbedc06eef0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fbee8004e60 tx=0x7fbee800b040).stop 2024-04-12T05:47:05.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.864+0000 7fbefa046700 1 -- 172.21.15.113:0/971180549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef40fe250 msgr2=0x7fbef410e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.860 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.864+0000 7fbefa046700 1 --2- 172.21.15.113:0/971180549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef40fe250 0x7fbef410e600 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7fbee400e940 tx=0x7fbee400b040).stop 2024-04-12T05:47:05.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.864+0000 7fbefa046700 1 -- 172.21.15.113:0/971180549 shutdown_connections 2024-04-12T05:47:05.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.864+0000 7fbefa046700 1 --2- 172.21.15.113:0/971180549 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbedc06ca30 0x7fbedc06eef0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.864+0000 7fbefa046700 1 --2- 172.21.15.113:0/971180549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbef40fd8f0 0x7fbef410e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.864+0000 7fbefa046700 1 --2- 172.21.15.113:0/971180549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbef40fe250 0x7fbef410e600 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.864+0000 7fbefa046700 1 -- 172.21.15.113:0/971180549 >> 172.21.15.113:0/971180549 conn(0x7fbef40f34b0 msgr2=0x7fbef40fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:05.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.864+0000 7fbefa046700 1 -- 172.21.15.113:0/971180549 shutdown_connections 2024-04-12T05:47:05.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.864+0000 7fbefa046700 1 -- 172.21.15.113:0/971180549 wait complete. 2024-04-12T05:47:05.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.961+0000 7f01b9d5d700 1 -- 172.21.15.113:0/19700416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01b40fe250 msgr2=0x7f01b40f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.961+0000 7f01b9d5d700 1 --2- 172.21.15.113:0/19700416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01b40fe250 0x7f01b40f7a30 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f019c0047f0 tx=0x7f019c009cb0).stop 2024-04-12T05:47:05.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.961+0000 7f01b9d5d700 1 -- 172.21.15.113:0/19700416 shutdown_connections 2024-04-12T05:47:05.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.961+0000 7f01b9d5d700 1 --2- 172.21.15.113:0/19700416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01b40fe250 0x7f01b40f7a30 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.961+0000 7f01b9d5d700 1 --2- 172.21.15.113:0/19700416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01b40fd8f0 0x7f01b40fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.961+0000 7f01b9d5d700 1 -- 172.21.15.113:0/19700416 >> 172.21.15.113:0/19700416 conn(0x7f01b40f34b0 msgr2=0x7f01b40f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:05.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.961+0000 7f01b9d5d700 1 -- 172.21.15.113:0/19700416 shutdown_connections 2024-04-12T05:47:05.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.961+0000 7f01b9d5d700 1 -- 172.21.15.113:0/19700416 wait complete. 2024-04-12T05:47:05.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.962+0000 7f01b9d5d700 1 Processor -- start 2024-04-12T05:47:05.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.962+0000 7f01b9d5d700 1 -- start start 2024-04-12T05:47:05.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.962+0000 7f01b9d5d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01b40fd8f0 0x7f01b4110220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.962+0000 7f01b9d5d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01b40fe250 0x7f01b4110710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.962+0000 7f01b9d5d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b4110d10 con 0x7f01b40fe250 2024-04-12T05:47:05.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.962+0000 7f01b9d5d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b4110e50 con 0x7f01b40fd8f0 2024-04-12T05:47:05.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.962+0000 7f01b2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01b40fe250 0x7f01b4110710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.962+0000 7f01b2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01b40fe250 0x7f01b4110710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49558/0 (socket says 172.21.15.113:49558) 2024-04-12T05:47:05.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.962+0000 7f01b2ffd700 1 -- 172.21.15.113:0/1337888687 learned_addr learned my addr 172.21.15.113:0/1337888687 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:05.960 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.962+0000 7f01b37fe700 1 --2- 172.21.15.113:0/1337888687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01b40fd8f0 0x7f01b4110220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.963+0000 7f01b2ffd700 1 -- 172.21.15.113:0/1337888687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01b40fd8f0 msgr2=0x7f01b4110220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:05.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.963+0000 7f01b2ffd700 1 --2- 172.21.15.113:0/1337888687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01b40fd8f0 0x7f01b4110220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:05.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.963+0000 7f01b2ffd700 1 -- 172.21.15.113:0/1337888687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c005040 con 0x7f01b40fe250 2024-04-12T05:47:05.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.963+0000 7f01b2ffd700 1 --2- 172.21.15.113:0/1337888687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01b40fe250 0x7f01b4110710 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7f019c0099f0 tx=0x7f019c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:05.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.963+0000 7f01b0ff9700 1 -- 172.21.15.113:0/1337888687 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f019c00e030 con 0x7f01b40fe250 2024-04-12T05:47:05.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.963+0000 7f01b0ff9700 1 -- 172.21.15.113:0/1337888687 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f019c007e90 con 0x7f01b40fe250 2024-04-12T05:47:05.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.963+0000 7f01b9d5d700 1 -- 172.21.15.113:0/1337888687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b410a480 con 0x7f01b40fe250 2024-04-12T05:47:05.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.963+0000 7f01b0ff9700 1 -- 172.21.15.113:0/1337888687 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f019c018070 con 0x7f01b40fe250 2024-04-12T05:47:05.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.963+0000 7f01b9d5d700 1 -- 172.21.15.113:0/1337888687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b410a960 con 0x7f01b40fe250 2024-04-12T05:47:05.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.964+0000 7f01b9d5d700 1 -- 172.21.15.113:0/1337888687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b405f590 con 0x7f01b40fe250 2024-04-12T05:47:05.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.964+0000 7f01b0ff9700 1 -- 172.21.15.113:0/1337888687 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f019c01d420 con 0x7f01b40fe250 2024-04-12T05:47:05.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.965+0000 7f01b0ff9700 1 --2- 172.21.15.113:0/1337888687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f01a006caa0 0x7f01a006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:05.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.965+0000 7f01b0ff9700 1 -- 172.21.15.113:0/1337888687 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f019c08d240 con 0x7f01b40fe250 2024-04-12T05:47:05.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.965+0000 7f01b37fe700 1 --2- 172.21.15.113:0/1337888687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f01a006caa0 0x7f01a006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:05.963 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.965+0000 7f01b37fe700 1 --2- 172.21.15.113:0/1337888687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f01a006caa0 0x7f01a006ef60 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f01a40041d0 tx=0x7f01a4009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:05.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:05.969+0000 7f01b0ff9700 1 -- 172.21.15.113:0/1337888687 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f019c05acf0 con 0x7f01b40fe250 2024-04-12T05:47:06.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.234+0000 7f01b9d5d700 1 -- 172.21.15.113:0/1337888687 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f01b410b8a0 con 0x7f01b40fe250 2024-04-12T05:47:06.230 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.234+0000 7f01b0ff9700 1 -- 172.21.15.113:0/1337888687 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f019c05a880 con 0x7f01b40fe250 2024-04-12T05:47:06.231 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:47:06.231 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:47:06.231 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:47:06.231 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:47:06.231 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:47:06.231 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:47:06.231 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:47:06.231 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:47:06.231 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:47:06.231 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:47:06.232 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:47:06.232 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:47:06.232 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:47:06.232 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:47:06.232 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:47:06.232 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:47:06.232 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:47:06.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.237+0000 7f01b9d5d700 1 -- 172.21.15.113:0/1337888687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f01a006caa0 msgr2=0x7f01a006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.237+0000 7f01b9d5d700 1 --2- 172.21.15.113:0/1337888687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f01a006caa0 0x7f01a006ef60 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f01a40041d0 tx=0x7f01a4009040).stop 2024-04-12T05:47:06.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.237+0000 7f01b9d5d700 1 -- 172.21.15.113:0/1337888687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01b40fe250 msgr2=0x7f01b4110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.237+0000 7f01b9d5d700 1 --2- 172.21.15.113:0/1337888687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01b40fe250 0x7f01b4110710 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7f019c0099f0 tx=0x7f019c009cb0).stop 2024-04-12T05:47:06.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.237+0000 7f01b9d5d700 1 -- 172.21.15.113:0/1337888687 shutdown_connections 2024-04-12T05:47:06.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.237+0000 7f01b9d5d700 1 --2- 172.21.15.113:0/1337888687 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f01a006caa0 0x7f01a006ef60 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.237+0000 7f01b9d5d700 1 --2- 172.21.15.113:0/1337888687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01b40fd8f0 0x7f01b4110220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.233 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.237+0000 7f01b9d5d700 1 --2- 172.21.15.113:0/1337888687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01b40fe250 0x7f01b4110710 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.237+0000 7f01b9d5d700 1 -- 172.21.15.113:0/1337888687 >> 172.21.15.113:0/1337888687 conn(0x7f01b40f34b0 msgr2=0x7f01b406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:06.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.237+0000 7f01b9d5d700 1 -- 172.21.15.113:0/1337888687 shutdown_connections 2024-04-12T05:47:06.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.237+0000 7f01b9d5d700 1 -- 172.21.15.113:0/1337888687 wait complete. 2024-04-12T05:47:06.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.333+0000 7f69617c7700 1 -- 172.21.15.113:0/3940862412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f695c0fc7e0 msgr2=0x7f695c0fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.333+0000 7f69617c7700 1 --2- 172.21.15.113:0/3940862412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f695c0fc7e0 0x7f695c0fcbc0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7f69500047f0 tx=0x7f6950009cb0).stop 2024-04-12T05:47:06.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.333+0000 7f69617c7700 1 -- 172.21.15.113:0/3940862412 shutdown_connections 2024-04-12T05:47:06.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.333+0000 7f69617c7700 1 --2- 172.21.15.113:0/3940862412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f695c0fd140 0x7f695c0f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.330 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.333+0000 7f69617c7700 1 --2- 172.21.15.113:0/3940862412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f695c0fc7e0 0x7f695c0fcbc0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.333+0000 7f69617c7700 1 -- 172.21.15.113:0/3940862412 >> 172.21.15.113:0/3940862412 conn(0x7f695c0f23c0 msgr2=0x7f695c0f47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:06.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.333+0000 7f69617c7700 1 -- 172.21.15.113:0/3940862412 shutdown_connections 2024-04-12T05:47:06.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.334+0000 7f69617c7700 1 -- 172.21.15.113:0/3940862412 wait complete. 2024-04-12T05:47:06.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.334+0000 7f69617c7700 1 Processor -- start 2024-04-12T05:47:06.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.334+0000 7f69617c7700 1 -- start start 2024-04-12T05:47:06.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.334+0000 7f69617c7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f695c0fc7e0 0x7f695c10cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:06.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.334+0000 7f69617c7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f695c0fd140 0x7f695c10d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:06.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.334+0000 7f69617c7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c10d9f0 con 0x7f695c0fd140 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.334+0000 7f69617c7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c10db30 con 0x7f695c0fc7e0 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f695a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f695c0fd140 0x7f695c10d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f695a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f695c0fd140 0x7f695c10d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49572/0 (socket says 172.21.15.113:49572) 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f695a7fc700 1 -- 172.21.15.113:0/1867217675 learned_addr learned my addr 172.21.15.113:0/1867217675 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f695affd700 1 --2- 172.21.15.113:0/1867217675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f695c0fc7e0 0x7f695c10cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f695a7fc700 1 -- 172.21.15.113:0/1867217675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f695c0fc7e0 msgr2=0x7f695c10cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f695a7fc700 1 --2- 172.21.15.113:0/1867217675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f695c0fc7e0 0x7f695c10cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f695a7fc700 1 -- 172.21.15.113:0/1867217675 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6950005040 con 0x7f695c0fd140 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f695a7fc700 1 --2- 172.21.15.113:0/1867217675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f695c0fd140 0x7f695c10d3f0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f694c00c790 tx=0x7f694c007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f6943fff700 1 -- 172.21.15.113:0/1867217675 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f694c00e680 con 0x7f695c0fd140 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f6943fff700 1 -- 172.21.15.113:0/1867217675 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f694c00ec90 con 0x7f695c0fd140 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f69617c7700 1 -- 172.21.15.113:0/1867217675 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c1071c0 con 0x7f695c0fd140 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f6943fff700 1 -- 172.21.15.113:0/1867217675 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f694c012cf0 con 0x7f695c0fd140 2024-04-12T05:47:06.333 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.335+0000 7f69617c7700 1 -- 172.21.15.113:0/1867217675 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c107780 con 0x7f695c0fd140 2024-04-12T05:47:06.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.336+0000 7f69617c7700 1 -- 172.21.15.113:0/1867217675 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c060370 con 0x7f695c0fd140 2024-04-12T05:47:06.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.337+0000 7f6943fff700 1 -- 172.21.15.113:0/1867217675 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f694c012500 con 0x7f695c0fd140 2024-04-12T05:47:06.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.337+0000 7f6943fff700 1 --2- 172.21.15.113:0/1867217675 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694406caa0 0x7f694406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:06.334 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.337+0000 7f6943fff700 1 -- 172.21.15.113:0/1867217675 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f694c08b720 con 0x7f695c0fd140 2024-04-12T05:47:06.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.337+0000 7f695affd700 1 --2- 172.21.15.113:0/1867217675 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694406caa0 0x7f694406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:06.335 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.337+0000 7f695affd700 1 --2- 172.21.15.113:0/1867217675 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694406caa0 0x7f694406ef60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f6950007cf0 tx=0x7f695000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:06.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.341+0000 7f6943fff700 1 -- 172.21.15.113:0/1867217675 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f694c0591d0 con 0x7f695c0fd140 2024-04-12T05:47:06.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:06 smithi113 ceph-mon[103371]: from='client.35878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:06.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:06 smithi192 ceph-mon[85304]: from='client.35878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:06.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.570+0000 7f69617c7700 1 -- 172.21.15.113:0/1867217675 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f695c0feb70 con 0x7f695c0fd140 2024-04-12T05:47:06.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.570+0000 7f6943fff700 1 -- 172.21.15.113:0/1867217675 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f694c058d60 con 0x7f695c0fd140 2024-04-12T05:47:06.566 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:47:06.567 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:47:06.568 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:47:06.569 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:47:06.569 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:47:06.569 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:47:06.569 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:47:06.569 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:47:06.569 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:47:06.569 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:47:06.569 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:06.569 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:06.569 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:06.569 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:06.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.572+0000 7f69617c7700 1 -- 172.21.15.113:0/1867217675 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694406caa0 msgr2=0x7f694406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.572+0000 7f69617c7700 1 --2- 172.21.15.113:0/1867217675 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694406caa0 0x7f694406ef60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f6950007cf0 tx=0x7f695000b040).stop 2024-04-12T05:47:06.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.572+0000 7f69617c7700 1 -- 172.21.15.113:0/1867217675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f695c0fd140 msgr2=0x7f695c10d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.572+0000 7f69617c7700 1 --2- 172.21.15.113:0/1867217675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f695c0fd140 0x7f695c10d3f0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f694c00c790 tx=0x7f694c007f50).stop 2024-04-12T05:47:06.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.573+0000 7f69617c7700 1 -- 172.21.15.113:0/1867217675 shutdown_connections 2024-04-12T05:47:06.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.573+0000 7f69617c7700 1 --2- 172.21.15.113:0/1867217675 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f694406caa0 0x7f694406ef60 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.573+0000 7f69617c7700 1 --2- 172.21.15.113:0/1867217675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f695c0fc7e0 0x7f695c10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.573+0000 7f69617c7700 1 --2- 172.21.15.113:0/1867217675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f695c0fd140 0x7f695c10d3f0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.573+0000 7f69617c7700 1 -- 172.21.15.113:0/1867217675 >> 172.21.15.113:0/1867217675 conn(0x7f695c0f23c0 msgr2=0x7f695c0faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:06.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.573+0000 7f69617c7700 1 -- 172.21.15.113:0/1867217675 shutdown_connections 2024-04-12T05:47:06.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.573+0000 7f69617c7700 1 -- 172.21.15.113:0/1867217675 wait complete. 2024-04-12T05:47:06.571 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:47:06.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.671+0000 7f829fbe6700 1 -- 172.21.15.113:0/1966999067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82980f8940 msgr2=0x7f82980f8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.671+0000 7f829fbe6700 1 --2- 172.21.15.113:0/1966999067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82980f8940 0x7f82980f8d20 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7f828c0047f0 tx=0x7f828c009cb0).stop 2024-04-12T05:47:06.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.671+0000 7f829fbe6700 1 -- 172.21.15.113:0/1966999067 shutdown_connections 2024-04-12T05:47:06.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.671+0000 7f829fbe6700 1 --2- 172.21.15.113:0/1966999067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82980f9210 0x7f82981019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.671+0000 7f829fbe6700 1 --2- 172.21.15.113:0/1966999067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82980f8940 0x7f82980f8d20 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.671+0000 7f829fbe6700 1 -- 172.21.15.113:0/1966999067 >> 172.21.15.113:0/1966999067 conn(0x7f8298070590 msgr2=0x7f82980f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:06.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.671+0000 7f829fbe6700 1 -- 172.21.15.113:0/1966999067 shutdown_connections 2024-04-12T05:47:06.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.671+0000 7f829fbe6700 1 -- 172.21.15.113:0/1966999067 wait complete. 2024-04-12T05:47:06.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829fbe6700 1 Processor -- start 2024-04-12T05:47:06.669 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829fbe6700 1 -- start start 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829fbe6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82980f8940 0x7f829810e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829fbe6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82980f9210 0x7f829810e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829fbe6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829810eb40 con 0x7f82980f8940 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829fbe6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829810ec80 con 0x7f82980f9210 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829d982700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82980f8940 0x7f829810e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829d982700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82980f8940 0x7f829810e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49586/0 (socket says 172.21.15.113:49586) 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829d982700 1 -- 172.21.15.113:0/3100504513 learned_addr learned my addr 172.21.15.113:0/3100504513 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829d181700 1 --2- 172.21.15.113:0/3100504513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82980f9210 0x7f829810e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829d982700 1 -- 172.21.15.113:0/3100504513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82980f9210 msgr2=0x7f829810e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829d982700 1 --2- 172.21.15.113:0/3100504513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82980f9210 0x7f829810e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829d982700 1 -- 172.21.15.113:0/3100504513 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c005040 con 0x7f82980f8940 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.672+0000 7f829d982700 1 --2- 172.21.15.113:0/3100504513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82980f8940 0x7f829810e050 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f828c005010 tx=0x7f828c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.673+0000 7f828affd700 1 -- 172.21.15.113:0/3100504513 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f828c00e030 con 0x7f82980f8940 2024-04-12T05:47:06.670 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.673+0000 7f828affd700 1 -- 172.21.15.113:0/3100504513 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f828c007e90 con 0x7f82980f8940 2024-04-12T05:47:06.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.673+0000 7f829fbe6700 1 -- 172.21.15.113:0/3100504513 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82981082b0 con 0x7f82980f8940 2024-04-12T05:47:06.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.673+0000 7f828affd700 1 -- 172.21.15.113:0/3100504513 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f828c018070 con 0x7f82980f8940 2024-04-12T05:47:06.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.673+0000 7f829fbe6700 1 -- 172.21.15.113:0/3100504513 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8298108790 con 0x7f82980f8940 2024-04-12T05:47:06.671 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.674+0000 7f829fbe6700 1 -- 172.21.15.113:0/3100504513 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829805f590 con 0x7f82980f8940 2024-04-12T05:47:06.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.674+0000 7f828affd700 1 -- 172.21.15.113:0/3100504513 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f828c01d670 con 0x7f82980f8940 2024-04-12T05:47:06.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.675+0000 7f828affd700 1 --2- 172.21.15.113:0/3100504513 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f828406caa0 0x7f828406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:06.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.675+0000 7f828affd700 1 -- 172.21.15.113:0/3100504513 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f828c08d150 con 0x7f82980f8940 2024-04-12T05:47:06.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.675+0000 7f829d181700 1 --2- 172.21.15.113:0/3100504513 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f828406caa0 0x7f828406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:06.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.675+0000 7f829d181700 1 --2- 172.21.15.113:0/3100504513 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f828406caa0 0x7f828406ef60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f8298109990 tx=0x7f829400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:06.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.679+0000 7f828affd700 1 -- 172.21.15.113:0/3100504513 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f828c05ac00 con 0x7f82980f8940 2024-04-12T05:47:06.884 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.888+0000 7f829fbe6700 1 -- 172.21.15.113:0/3100504513 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f82980feee0 con 0x7f828406caa0 2024-04-12T05:47:06.885 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.889+0000 7f828affd700 1 -- 172.21.15.113:0/3100504513 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f82980feee0 con 0x7f828406caa0 2024-04-12T05:47:06.885 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:47:06.885 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:47:06.885 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:47:06.885 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:47:06.885 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:47:06.885 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:47:06.885 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:47:06.885 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:47:06.885 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:47:06.885 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:47:06.886 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:47:06.886 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:47:06.886 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:47:06.886 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:47:06.886 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:47:06.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.891+0000 7f829fbe6700 1 -- 172.21.15.113:0/3100504513 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f828406caa0 msgr2=0x7f828406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.891+0000 7f829fbe6700 1 --2- 172.21.15.113:0/3100504513 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f828406caa0 0x7f828406ef60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f8298109990 tx=0x7f829400b040).stop 2024-04-12T05:47:06.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.891+0000 7f829fbe6700 1 -- 172.21.15.113:0/3100504513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82980f8940 msgr2=0x7f829810e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.891+0000 7f829fbe6700 1 --2- 172.21.15.113:0/3100504513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82980f8940 0x7f829810e050 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f828c005010 tx=0x7f828c009cb0).stop 2024-04-12T05:47:06.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.891+0000 7f829fbe6700 1 -- 172.21.15.113:0/3100504513 shutdown_connections 2024-04-12T05:47:06.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.891+0000 7f829fbe6700 1 --2- 172.21.15.113:0/3100504513 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f828406caa0 0x7f828406ef60 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.891+0000 7f829fbe6700 1 --2- 172.21.15.113:0/3100504513 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82980f8940 0x7f829810e050 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.887 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.891+0000 7f829fbe6700 1 --2- 172.21.15.113:0/3100504513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82980f9210 0x7f829810e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.891+0000 7f829fbe6700 1 -- 172.21.15.113:0/3100504513 >> 172.21.15.113:0/3100504513 conn(0x7f8298070590 msgr2=0x7f82980fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:06.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.891+0000 7f829fbe6700 1 -- 172.21.15.113:0/3100504513 shutdown_connections 2024-04-12T05:47:06.888 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.891+0000 7f829fbe6700 1 -- 172.21.15.113:0/3100504513 wait complete. 2024-04-12T05:47:06.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.989+0000 7f34273ce700 1 -- 172.21.15.113:0/411946407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34200fd8f0 msgr2=0x7f34200fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.987 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.989+0000 7f34273ce700 1 --2- 172.21.15.113:0/411946407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34200fd8f0 0x7f34200fdcd0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7f341c0047f0 tx=0x7f341c009cb0).stop 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.989+0000 7f34273ce700 1 -- 172.21.15.113:0/411946407 shutdown_connections 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.989+0000 7f34273ce700 1 --2- 172.21.15.113:0/411946407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34200fe250 0x7f34200f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.989+0000 7f34273ce700 1 --2- 172.21.15.113:0/411946407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34200fd8f0 0x7f34200fdcd0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.989+0000 7f34273ce700 1 -- 172.21.15.113:0/411946407 >> 172.21.15.113:0/411946407 conn(0x7f34200f34b0 msgr2=0x7f34200f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.990+0000 7f34273ce700 1 -- 172.21.15.113:0/411946407 shutdown_connections 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.990+0000 7f34273ce700 1 -- 172.21.15.113:0/411946407 wait complete. 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.990+0000 7f34273ce700 1 Processor -- start 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.990+0000 7f34273ce700 1 -- start start 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.990+0000 7f34273ce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34200fd8f0 0x7f342010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.990+0000 7f34273ce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34200fe250 0x7f342010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.990+0000 7f34273ce700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342010ead0 con 0x7f34200fe250 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.990+0000 7f34273ce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342010ec10 con 0x7f34200fd8f0 2024-04-12T05:47:06.988 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f3424969700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34200fe250 0x7f342010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f3424969700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34200fe250 0x7f342010e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49600/0 (socket says 172.21.15.113:49600) 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f3424969700 1 -- 172.21.15.113:0/2313077503 learned_addr learned my addr 172.21.15.113:0/2313077503 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f342516a700 1 --2- 172.21.15.113:0/2313077503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34200fd8f0 0x7f342010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f3424969700 1 -- 172.21.15.113:0/2313077503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34200fd8f0 msgr2=0x7f342010dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f3424969700 1 --2- 172.21.15.113:0/2313077503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34200fd8f0 0x7f342010dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f3424969700 1 -- 172.21.15.113:0/2313077503 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c005040 con 0x7f34200fe250 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f3424969700 1 --2- 172.21.15.113:0/2313077503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34200fe250 0x7f342010e4d0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f3410002780 tx=0x7f3410009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f34167fc700 1 -- 172.21.15.113:0/2313077503 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3410007b40 con 0x7f34200fe250 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f34167fc700 1 -- 172.21.15.113:0/2313077503 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f341000f040 con 0x7f34200fe250 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f34273ce700 1 -- 172.21.15.113:0/2313077503 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34201082a0 con 0x7f34200fe250 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f34167fc700 1 -- 172.21.15.113:0/2313077503 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3410014780 con 0x7f34200fe250 2024-04-12T05:47:06.989 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.991+0000 7f34273ce700 1 -- 172.21.15.113:0/2313077503 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3420108860 con 0x7f34200fe250 2024-04-12T05:47:06.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.992+0000 7f34273ce700 1 -- 172.21.15.113:0/2313077503 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342005f590 con 0x7f34200fe250 2024-04-12T05:47:06.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.993+0000 7f34167fc700 1 -- 172.21.15.113:0/2313077503 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3410007ca0 con 0x7f34200fe250 2024-04-12T05:47:06.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.993+0000 7f34167fc700 1 --2- 172.21.15.113:0/2313077503 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f340c06caa0 0x7f340c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:06.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.993+0000 7f342516a700 1 --2- 172.21.15.113:0/2313077503 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f340c06caa0 0x7f340c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:06.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.993+0000 7f34167fc700 1 -- 172.21.15.113:0/2313077503 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f341008bd50 con 0x7f34200fe250 2024-04-12T05:47:06.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.994+0000 7f342516a700 1 --2- 172.21.15.113:0/2313077503 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f340c06caa0 0x7f340c06ef60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f341c004f10 tx=0x7f341c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:06.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:06.998+0000 7f34167fc700 1 -- 172.21.15.113:0/2313077503 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3410059800 con 0x7f34200fe250 2024-04-12T05:47:07.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.265+0000 7f34273ce700 1 -- 172.21.15.113:0/2313077503 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3420047540 con 0x7f34200fe250 2024-04-12T05:47:07.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.266+0000 7f34167fc700 1 -- 172.21.15.113:0/2313077503 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f3410059390 con 0x7f34200fe250 2024-04-12T05:47:07.262 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:47:07.262 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:47:07.262 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:47:07.262 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:47:07.262 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:47:07.262 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:47:07.263 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:47:07.264 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:47:07.264 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:47:07.264 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:47:07.264 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:47:07.264 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:47:07.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.268+0000 7f34273ce700 1 -- 172.21.15.113:0/2313077503 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f340c06caa0 msgr2=0x7f340c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:07.264 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.268+0000 7f34273ce700 1 --2- 172.21.15.113:0/2313077503 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f340c06caa0 0x7f340c06ef60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f341c004f10 tx=0x7f341c00b040).stop 2024-04-12T05:47:07.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.268+0000 7f34273ce700 1 -- 172.21.15.113:0/2313077503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34200fe250 msgr2=0x7f342010e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:07.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.268+0000 7f34273ce700 1 --2- 172.21.15.113:0/2313077503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34200fe250 0x7f342010e4d0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f3410002780 tx=0x7f3410009ee0).stop 2024-04-12T05:47:07.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.268+0000 7f34273ce700 1 -- 172.21.15.113:0/2313077503 shutdown_connections 2024-04-12T05:47:07.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.268+0000 7f34273ce700 1 --2- 172.21.15.113:0/2313077503 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f340c06caa0 0x7f340c06ef60 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:07.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.268+0000 7f34273ce700 1 --2- 172.21.15.113:0/2313077503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34200fd8f0 0x7f342010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:07.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.268+0000 7f34273ce700 1 --2- 172.21.15.113:0/2313077503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34200fe250 0x7f342010e4d0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:07.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.268+0000 7f34273ce700 1 -- 172.21.15.113:0/2313077503 >> 172.21.15.113:0/2313077503 conn(0x7f34200f34b0 msgr2=0x7f34200fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:07.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.268+0000 7f34273ce700 1 -- 172.21.15.113:0/2313077503 shutdown_connections 2024-04-12T05:47:07.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:07.268+0000 7f34273ce700 1 -- 172.21.15.113:0/2313077503 wait complete. 2024-04-12T05:47:07.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:07 smithi113 ceph-mon[103371]: from='client.35882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:07.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:07 smithi113 ceph-mon[103371]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:07.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:07 smithi113 ceph-mon[103371]: pgmap v1049: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:07.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:07 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1337888687' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:07.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:07 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1867217675' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:47:07.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:07.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:07 smithi192 ceph-mon[85304]: from='client.35882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:07.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:07 smithi192 ceph-mon[85304]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:07.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:07 smithi192 ceph-mon[85304]: pgmap v1049: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:07.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:07 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1337888687' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:07.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:07 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1867217675' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:47:07.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:08.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:08 smithi113 ceph-mon[103371]: from='client.35898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:08.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:08 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2313077503' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:47:08.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:08 smithi192 ceph-mon[85304]: from='client.35898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:08.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:08 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2313077503' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:47:09.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:09 smithi113 ceph-mon[103371]: pgmap v1050: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:09.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:09 smithi192 ceph-mon[85304]: pgmap v1050: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:11.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:11 smithi113 ceph-mon[103371]: pgmap v1051: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:11.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:11 smithi192 ceph-mon[85304]: pgmap v1051: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:13.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:13 smithi113 ceph-mon[103371]: pgmap v1052: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:13.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:13 smithi192 ceph-mon[85304]: pgmap v1052: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:15.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:15 smithi113 ceph-mon[103371]: pgmap v1053: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:15.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:15 smithi192 ceph-mon[85304]: pgmap v1053: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:17.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:17 smithi113 ceph-mon[103371]: pgmap v1054: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:17.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:17 smithi192 ceph-mon[85304]: pgmap v1054: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:19.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:19 smithi113 ceph-mon[103371]: pgmap v1055: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:19.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:19 smithi192 ceph-mon[85304]: pgmap v1055: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:21.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:21 smithi113 ceph-mon[103371]: pgmap v1056: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:21.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:21 smithi192 ceph-mon[85304]: pgmap v1056: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:22.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:22.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:23.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:23 smithi113 ceph-mon[103371]: pgmap v1057: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:23.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:23 smithi192 ceph-mon[85304]: pgmap v1057: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:25.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:25 smithi113 ceph-mon[103371]: pgmap v1058: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:25.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:25 smithi192 ceph-mon[85304]: pgmap v1058: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:27.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:27 smithi113 ceph-mon[103371]: pgmap v1059: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:27.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:27 smithi192 ceph-mon[85304]: pgmap v1059: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:29.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:29 smithi113 ceph-mon[103371]: pgmap v1060: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:29.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:29 smithi192 ceph-mon[85304]: pgmap v1060: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:31.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:31 smithi113 ceph-mon[103371]: pgmap v1061: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:31.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:31 smithi192 ceph-mon[85304]: pgmap v1061: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:33.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:33 smithi113 ceph-mon[103371]: pgmap v1062: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:33.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:33 smithi192 ceph-mon[85304]: pgmap v1062: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:35.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:35 smithi113 ceph-mon[103371]: pgmap v1063: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:35.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:35 smithi192 ceph-mon[85304]: pgmap v1063: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:37.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.381+0000 7f0dced26700 1 -- 172.21.15.113:0/207452861 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0dc80f88a0 msgr2=0x7f0dc80f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:37.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.381+0000 7f0dced26700 1 --2- 172.21.15.113:0/207452861 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0dc80f88a0 0x7f0dc80f8c80 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f0dbc0047f0 tx=0x7f0dbc009cb0).stop 2024-04-12T05:47:37.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.382+0000 7f0dced26700 1 -- 172.21.15.113:0/207452861 shutdown_connections 2024-04-12T05:47:37.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.382+0000 7f0dced26700 1 --2- 172.21.15.113:0/207452861 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dc80f9200 0x7f0dc8101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.382+0000 7f0dced26700 1 --2- 172.21.15.113:0/207452861 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0dc80f88a0 0x7f0dc80f8c80 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.382+0000 7f0dced26700 1 -- 172.21.15.113:0/207452861 >> 172.21.15.113:0/207452861 conn(0x7f0dc8070590 msgr2=0x7f0dc80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:37.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.382+0000 7f0dced26700 1 -- 172.21.15.113:0/207452861 shutdown_connections 2024-04-12T05:47:37.378 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.382+0000 7f0dced26700 1 -- 172.21.15.113:0/207452861 wait complete. 2024-04-12T05:47:37.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.382+0000 7f0dced26700 1 Processor -- start 2024-04-12T05:47:37.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.382+0000 7f0dced26700 1 -- start start 2024-04-12T05:47:37.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dced26700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0dc80f88a0 0x7f0dc810e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:37.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dced26700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dc80f9200 0x7f0dc810e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:37.380 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dced26700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc810eb10 con 0x7f0dc80f88a0 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dced26700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc810ec50 con 0x7f0dc80f9200 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dccac2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0dc80f88a0 0x7f0dc810e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dccac2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0dc80f88a0 0x7f0dc810e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58660/0 (socket says 172.21.15.113:58660) 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dccac2700 1 -- 172.21.15.113:0/2261981590 learned_addr learned my addr 172.21.15.113:0/2261981590 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dc7fff700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dc80f9200 0x7f0dc810e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dccac2700 1 -- 172.21.15.113:0/2261981590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dc80f9200 msgr2=0x7f0dc810e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dccac2700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dc80f9200 0x7f0dc810e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dccac2700 1 -- 172.21.15.113:0/2261981590 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc005040 con 0x7f0dc80f88a0 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dc7fff700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dc80f9200 0x7f0dc810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.383+0000 7f0dccac2700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0dc80f88a0 0x7f0dc810e020 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f0dbc005010 tx=0x7f0dbc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.384+0000 7f0dc5ffb700 1 -- 172.21.15.113:0/2261981590 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dbc00e030 con 0x7f0dc80f88a0 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.384+0000 7f0dced26700 1 -- 172.21.15.113:0/2261981590 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc8108280 con 0x7f0dc80f88a0 2024-04-12T05:47:37.381 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.384+0000 7f0dc5ffb700 1 -- 172.21.15.113:0/2261981590 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0dbc007e90 con 0x7f0dc80f88a0 2024-04-12T05:47:37.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.384+0000 7f0dc5ffb700 1 -- 172.21.15.113:0/2261981590 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dbc018070 con 0x7f0dc80f88a0 2024-04-12T05:47:37.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.384+0000 7f0dced26700 1 -- 172.21.15.113:0/2261981590 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc8108760 con 0x7f0dc80f88a0 2024-04-12T05:47:37.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.385+0000 7f0dced26700 1 -- 172.21.15.113:0/2261981590 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc805f590 con 0x7f0dc80f88a0 2024-04-12T05:47:37.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.385+0000 7f0dc5ffb700 1 -- 172.21.15.113:0/2261981590 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0dbc01d6e0 con 0x7f0dc80f88a0 2024-04-12T05:47:37.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.386+0000 7f0dc5ffb700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0db806ca30 0x7f0db806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:37.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.386+0000 7f0dc5ffb700 1 -- 172.21.15.113:0/2261981590 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0dbc08d0f0 con 0x7f0dc80f88a0 2024-04-12T05:47:37.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.386+0000 7f0dc7fff700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0db806ca30 0x7f0db806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:37.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.386+0000 7f0dc7fff700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0db806ca30 0x7f0db806eef0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f0db4009a00 tx=0x7f0db400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:37.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.390+0000 7f0dc5ffb700 1 -- 172.21.15.113:0/2261981590 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0dbc05aba0 con 0x7f0dc80f88a0 2024-04-12T05:47:37.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.596+0000 7f0dced26700 1 -- 172.21.15.113:0/2261981590 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0dc81097b0 con 0x7f0db806ca30 2024-04-12T05:47:37.593 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.597+0000 7f0dc5ffb700 1 -- 172.21.15.113:0/2261981590 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0dc81097b0 con 0x7f0db806ca30 2024-04-12T05:47:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.599+0000 7f0dced26700 1 -- 172.21.15.113:0/2261981590 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0db806ca30 msgr2=0x7f0db806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.599+0000 7f0dced26700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0db806ca30 0x7f0db806eef0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f0db4009a00 tx=0x7f0db400b040).stop 2024-04-12T05:47:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.599+0000 7f0dced26700 1 -- 172.21.15.113:0/2261981590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0dc80f88a0 msgr2=0x7f0dc810e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.599+0000 7f0dced26700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0dc80f88a0 0x7f0dc810e020 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f0dbc005010 tx=0x7f0dbc009cb0).stop 2024-04-12T05:47:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.599+0000 7f0dced26700 1 -- 172.21.15.113:0/2261981590 shutdown_connections 2024-04-12T05:47:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.599+0000 7f0dced26700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0db806ca30 0x7f0db806eef0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.599+0000 7f0dced26700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0dc80f88a0 0x7f0dc810e020 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.599+0000 7f0dced26700 1 --2- 172.21.15.113:0/2261981590 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dc80f9200 0x7f0dc810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.599+0000 7f0dced26700 1 -- 172.21.15.113:0/2261981590 >> 172.21.15.113:0/2261981590 conn(0x7f0dc8070590 msgr2=0x7f0dc80fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.600+0000 7f0dced26700 1 -- 172.21.15.113:0/2261981590 shutdown_connections 2024-04-12T05:47:37.596 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.600+0000 7f0dced26700 1 -- 172.21.15.113:0/2261981590 wait complete. 2024-04-12T05:47:37.605 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:47:37.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:37 smithi113 ceph-mon[103371]: pgmap v1064: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:37.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:37.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:37 smithi192 ceph-mon[85304]: pgmap v1064: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:37.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:37.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.699+0000 7f8aca558700 1 -- 172.21.15.113:0/3011063724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac40fd910 msgr2=0x7f8ac40fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:37.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.699+0000 7f8aca558700 1 --2- 172.21.15.113:0/3011063724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac40fd910 0x7f8ac40fdcf0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7f8ab80047f0 tx=0x7f8ab8009cb0).stop 2024-04-12T05:47:37.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.699+0000 7f8aca558700 1 -- 172.21.15.113:0/3011063724 shutdown_connections 2024-04-12T05:47:37.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.699+0000 7f8aca558700 1 --2- 172.21.15.113:0/3011063724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac40fe270 0x7f8ac40f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.699+0000 7f8aca558700 1 --2- 172.21.15.113:0/3011063724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac40fd910 0x7f8ac40fdcf0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.699+0000 7f8aca558700 1 -- 172.21.15.113:0/3011063724 >> 172.21.15.113:0/3011063724 conn(0x7f8ac40f3530 msgr2=0x7f8ac40f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:37.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.699+0000 7f8aca558700 1 -- 172.21.15.113:0/3011063724 shutdown_connections 2024-04-12T05:47:37.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.699+0000 7f8aca558700 1 -- 172.21.15.113:0/3011063724 wait complete. 2024-04-12T05:47:37.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.699+0000 7f8aca558700 1 Processor -- start 2024-04-12T05:47:37.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.699+0000 7f8aca558700 1 -- start start 2024-04-12T05:47:37.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8aca558700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac40fd910 0x7f8ac410e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:37.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8aca558700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac40fe270 0x7f8ac410e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:37.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8aca558700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac410eba0 con 0x7f8ac40fd910 2024-04-12T05:47:37.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8aca558700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac410ece0 con 0x7f8ac40fe270 2024-04-12T05:47:37.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8ac3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac40fd910 0x7f8ac410e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8ac3fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac40fd910 0x7f8ac410e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58668/0 (socket says 172.21.15.113:58668) 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8ac3fff700 1 -- 172.21.15.113:0/2529967058 learned_addr learned my addr 172.21.15.113:0/2529967058 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8ac37fe700 1 --2- 172.21.15.113:0/2529967058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac40fe270 0x7f8ac410e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8ac3fff700 1 -- 172.21.15.113:0/2529967058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac40fe270 msgr2=0x7f8ac410e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8ac3fff700 1 --2- 172.21.15.113:0/2529967058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac40fe270 0x7f8ac410e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8ac3fff700 1 -- 172.21.15.113:0/2529967058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab8005040 con 0x7f8ac40fd910 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8ac3fff700 1 --2- 172.21.15.113:0/2529967058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac40fd910 0x7f8ac410e0b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f8ab8005010 tx=0x7f8ab8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8ac17fa700 1 -- 172.21.15.113:0/2529967058 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ab800e030 con 0x7f8ac40fd910 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8ac17fa700 1 -- 172.21.15.113:0/2529967058 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ab8007e90 con 0x7f8ac40fd910 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8aca558700 1 -- 172.21.15.113:0/2529967058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac4108310 con 0x7f8ac40fd910 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8ac17fa700 1 -- 172.21.15.113:0/2529967058 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ab8018070 con 0x7f8ac40fd910 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.700+0000 7f8aca558700 1 -- 172.21.15.113:0/2529967058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac41087f0 con 0x7f8ac40fd910 2024-04-12T05:47:37.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.701+0000 7f8aca558700 1 -- 172.21.15.113:0/2529967058 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac405f570 con 0x7f8ac40fd910 2024-04-12T05:47:37.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.702+0000 7f8ac17fa700 1 -- 172.21.15.113:0/2529967058 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8ab801d6e0 con 0x7f8ac40fd910 2024-04-12T05:47:37.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.702+0000 7f8ac17fa700 1 --2- 172.21.15.113:0/2529967058 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8aac06ca30 0x7f8aac06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:37.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.703+0000 7f8ac37fe700 1 --2- 172.21.15.113:0/2529967058 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8aac06ca30 0x7f8aac06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:37.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.703+0000 7f8ac17fa700 1 -- 172.21.15.113:0/2529967058 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f8ab808d100 con 0x7f8ac40fd910 2024-04-12T05:47:37.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.703+0000 7f8ac37fe700 1 --2- 172.21.15.113:0/2529967058 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8aac06ca30 0x7f8aac06eef0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f8ac41099f0 tx=0x7f8ab4009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:37.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.707+0000 7f8ac17fa700 1 -- 172.21.15.113:0/2529967058 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8ab805a920 con 0x7f8ac40fd910 2024-04-12T05:47:37.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.915+0000 7f8aca558700 1 -- 172.21.15.113:0/2529967058 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ac4109840 con 0x7f8aac06ca30 2024-04-12T05:47:37.912 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.917+0000 7f8ac17fa700 1 -- 172.21.15.113:0/2529967058 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8ac4109840 con 0x7f8aac06ca30 2024-04-12T05:47:37.914 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.918+0000 7f8aca558700 1 -- 172.21.15.113:0/2529967058 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8aac06ca30 msgr2=0x7f8aac06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:37.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.918+0000 7f8aca558700 1 --2- 172.21.15.113:0/2529967058 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8aac06ca30 0x7f8aac06eef0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f8ac41099f0 tx=0x7f8ab4009da0).stop 2024-04-12T05:47:37.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.919+0000 7f8aca558700 1 -- 172.21.15.113:0/2529967058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac40fd910 msgr2=0x7f8ac410e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:37.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.919+0000 7f8aca558700 1 --2- 172.21.15.113:0/2529967058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac40fd910 0x7f8ac410e0b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f8ab8005010 tx=0x7f8ab8009cb0).stop 2024-04-12T05:47:37.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.919+0000 7f8aca558700 1 -- 172.21.15.113:0/2529967058 shutdown_connections 2024-04-12T05:47:37.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.919+0000 7f8aca558700 1 --2- 172.21.15.113:0/2529967058 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8aac06ca30 0x7f8aac06eef0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.919+0000 7f8aca558700 1 --2- 172.21.15.113:0/2529967058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ac40fd910 0x7f8ac410e0b0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.919+0000 7f8aca558700 1 --2- 172.21.15.113:0/2529967058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac40fe270 0x7f8ac410e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:37.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.919+0000 7f8aca558700 1 -- 172.21.15.113:0/2529967058 >> 172.21.15.113:0/2529967058 conn(0x7f8ac40f3530 msgr2=0x7f8ac40fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:37.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.919+0000 7f8aca558700 1 -- 172.21.15.113:0/2529967058 shutdown_connections 2024-04-12T05:47:37.915 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:37.919+0000 7f8aca558700 1 -- 172.21.15.113:0/2529967058 wait complete. 2024-04-12T05:47:38.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.016+0000 7f5badb5e700 1 -- 172.21.15.113:0/397274008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ba80fd150 msgr2=0x7f5ba80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.016+0000 7f5badb5e700 1 --2- 172.21.15.113:0/397274008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ba80fd150 0x7f5ba80f6930 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f5b9c0047f0 tx=0x7f5b9c009cb0).stop 2024-04-12T05:47:38.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.016+0000 7f5badb5e700 1 -- 172.21.15.113:0/397274008 shutdown_connections 2024-04-12T05:47:38.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.016+0000 7f5badb5e700 1 --2- 172.21.15.113:0/397274008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ba80fd150 0x7f5ba80f6930 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.016+0000 7f5badb5e700 1 --2- 172.21.15.113:0/397274008 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ba80fc7f0 0x7f5ba80fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.016+0000 7f5badb5e700 1 -- 172.21.15.113:0/397274008 >> 172.21.15.113:0/397274008 conn(0x7f5ba8070590 msgr2=0x7f5ba80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:38.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.016+0000 7f5badb5e700 1 -- 172.21.15.113:0/397274008 shutdown_connections 2024-04-12T05:47:38.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.016+0000 7f5badb5e700 1 -- 172.21.15.113:0/397274008 wait complete. 2024-04-12T05:47:38.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.017+0000 7f5badb5e700 1 Processor -- start 2024-04-12T05:47:38.013 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.017+0000 7f5badb5e700 1 -- start start 2024-04-12T05:47:38.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.017+0000 7f5badb5e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ba80fc7f0 0x7f5ba8106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:38.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.017+0000 7f5badb5e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ba80fd150 0x7f5ba81073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:38.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.017+0000 7f5badb5e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba80614e0 con 0x7f5ba80fd150 2024-04-12T05:47:38.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.017+0000 7f5badb5e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba8107890 con 0x7f5ba80fc7f0 2024-04-12T05:47:38.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.017+0000 7f5ba6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ba80fd150 0x7f5ba81073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.017+0000 7f5ba6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ba80fd150 0x7f5ba81073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58682/0 (socket says 172.21.15.113:58682) 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.017+0000 7f5ba6ffd700 1 -- 172.21.15.113:0/2024894286 learned_addr learned my addr 172.21.15.113:0/2024894286 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.018+0000 7f5ba77fe700 1 --2- 172.21.15.113:0/2024894286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ba80fc7f0 0x7f5ba8106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.018+0000 7f5ba6ffd700 1 -- 172.21.15.113:0/2024894286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ba80fc7f0 msgr2=0x7f5ba8106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.018+0000 7f5ba6ffd700 1 --2- 172.21.15.113:0/2024894286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ba80fc7f0 0x7f5ba8106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.018+0000 7f5ba6ffd700 1 -- 172.21.15.113:0/2024894286 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c005040 con 0x7f5ba80fd150 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.018+0000 7f5ba6ffd700 1 --2- 172.21.15.113:0/2024894286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ba80fd150 0x7f5ba81073a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7f5b9c0099f0 tx=0x7f5b9c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.018+0000 7f5ba4ff9700 1 -- 172.21.15.113:0/2024894286 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b9c00e030 con 0x7f5ba80fd150 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.018+0000 7f5ba4ff9700 1 -- 172.21.15.113:0/2024894286 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b9c007e90 con 0x7f5ba80fd150 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.018+0000 7f5badb5e700 1 -- 172.21.15.113:0/2024894286 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba8107b10 con 0x7f5ba80fd150 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.018+0000 7f5ba4ff9700 1 -- 172.21.15.113:0/2024894286 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b9c018070 con 0x7f5ba80fd150 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.018+0000 7f5badb5e700 1 -- 172.21.15.113:0/2024894286 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba81146e0 con 0x7f5ba80fd150 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.019+0000 7f5badb5e700 1 -- 172.21.15.113:0/2024894286 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba805f590 con 0x7f5ba80fd150 2024-04-12T05:47:38.016 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.020+0000 7f5ba4ff9700 1 -- 172.21.15.113:0/2024894286 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5b9c0094c0 con 0x7f5ba80fd150 2024-04-12T05:47:38.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.020+0000 7f5ba4ff9700 1 --2- 172.21.15.113:0/2024894286 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b9006caa0 0x7f5b9006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:38.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.020+0000 7f5ba4ff9700 1 -- 172.21.15.113:0/2024894286 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5b9c08d0c0 con 0x7f5ba80fd150 2024-04-12T05:47:38.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.020+0000 7f5ba77fe700 1 --2- 172.21.15.113:0/2024894286 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b9006caa0 0x7f5b9006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:38.017 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.020+0000 7f5ba77fe700 1 --2- 172.21.15.113:0/2024894286 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b9006caa0 0x7f5b9006ef60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f5b98004dc0 tx=0x7f5b9800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:38.020 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.024+0000 7f5ba4ff9700 1 -- 172.21.15.113:0/2024894286 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5b9c05ab70 con 0x7f5ba80fd150 2024-04-12T05:47:38.214 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.218+0000 7f5badb5e700 1 -- 172.21.15.113:0/2024894286 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ba8059120 con 0x7f5b9006caa0 2024-04-12T05:47:38.224 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.228+0000 7f5ba4ff9700 1 -- 172.21.15.113:0/2024894286 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5ba8059120 con 0x7f5b9006caa0 2024-04-12T05:47:38.224 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:47:38.224 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (35m) 12m ago 46m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:47:38.224 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (32m) 12m ago 46m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:47:38.224 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (32m) 12m ago 46m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (35m) 12m ago 46m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (28m) 12m ago 41m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (28m) 12m ago 41m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (28m) 12m ago 41m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (13m) 12m ago 41m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (36m) 12m ago 47m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (36m) 12m ago 46m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (33m) 12m ago 48m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (33m) 12m ago 45m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (36m) 12m ago 46m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (36m) 12m ago 45m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (32m) 12m ago 45m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (31m) 12m ago 44m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (31m) 12m ago 44m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:47:38.225 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (30m) 12m ago 43m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:47:38.226 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (30m) 12m ago 43m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:47:38.226 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (29m) 12m ago 42m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:47:38.226 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (35m) 12m ago 45m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:47:38.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.230+0000 7f5badb5e700 1 -- 172.21.15.113:0/2024894286 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b9006caa0 msgr2=0x7f5b9006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.226 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.230+0000 7f5badb5e700 1 --2- 172.21.15.113:0/2024894286 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b9006caa0 0x7f5b9006ef60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f5b98004dc0 tx=0x7f5b9800b040).stop 2024-04-12T05:47:38.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.230+0000 7f5badb5e700 1 -- 172.21.15.113:0/2024894286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ba80fd150 msgr2=0x7f5ba81073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.230+0000 7f5badb5e700 1 --2- 172.21.15.113:0/2024894286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ba80fd150 0x7f5ba81073a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7f5b9c0099f0 tx=0x7f5b9c009cb0).stop 2024-04-12T05:47:38.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.230+0000 7f5badb5e700 1 -- 172.21.15.113:0/2024894286 shutdown_connections 2024-04-12T05:47:38.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.230+0000 7f5badb5e700 1 --2- 172.21.15.113:0/2024894286 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5b9006caa0 0x7f5b9006ef60 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.230+0000 7f5badb5e700 1 --2- 172.21.15.113:0/2024894286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ba80fc7f0 0x7f5ba8106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.230+0000 7f5badb5e700 1 --2- 172.21.15.113:0/2024894286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5ba80fd150 0x7f5ba81073a0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.230+0000 7f5badb5e700 1 -- 172.21.15.113:0/2024894286 >> 172.21.15.113:0/2024894286 conn(0x7f5ba8070590 msgr2=0x7f5ba80fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:38.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.230+0000 7f5badb5e700 1 -- 172.21.15.113:0/2024894286 shutdown_connections 2024-04-12T05:47:38.227 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.230+0000 7f5badb5e700 1 -- 172.21.15.113:0/2024894286 wait complete. 2024-04-12T05:47:38.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.328+0000 7f655fae7700 1 -- 172.21.15.113:0/1963807268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65580f88a0 msgr2=0x7f65580f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.324 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.328+0000 7f655fae7700 1 --2- 172.21.15.113:0/1963807268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65580f88a0 0x7f65580f8c80 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7f654c0047f0 tx=0x7f654c009cb0).stop 2024-04-12T05:47:38.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.328+0000 7f655fae7700 1 -- 172.21.15.113:0/1963807268 shutdown_connections 2024-04-12T05:47:38.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.328+0000 7f655fae7700 1 --2- 172.21.15.113:0/1963807268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65580f9200 0x7f6558101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:47:38.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.328+0000 7f655fae7700 1 --2- 172.21.15.113:0/1963807268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65580f88a0 0x7f65580f8c80 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.328+0000 7f655fae7700 1 -- 172.21.15.113:0/1963807268 >> 172.21.15.113:0/1963807268 conn(0x7f6558070590 msgr2=0x7f65580f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:38.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.328+0000 7f655fae7700 1 -- 172.21.15.113:0/1963807268 shutdown_connections 2024-04-12T05:47:38.325 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.328+0000 7f655fae7700 1 -- 172.21.15.113:0/1963807268 wait complete. 2024-04-12T05:47:38.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.329+0000 7f655fae7700 1 Processor -- start 2024-04-12T05:47:38.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.329+0000 7f655fae7700 1 -- start start 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.329+0000 7f655fae7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65580f88a0 0x7f655810e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.329+0000 7f655fae7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65580f9200 0x7f655810e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.329+0000 7f655fae7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655810ec60 con 0x7f65580f9200 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.329+0000 7f655fae7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6558108280 con 0x7f65580f88a0 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.329+0000 7f655d082700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65580f9200 0x7f655810e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.329+0000 7f655d082700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65580f9200 0x7f655810e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58714/0 (socket says 172.21.15.113:58714) 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.329+0000 7f655d082700 1 -- 172.21.15.113:0/1906135393 learned_addr learned my addr 172.21.15.113:0/1906135393 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.329+0000 7f655d883700 1 --2- 172.21.15.113:0/1906135393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65580f88a0 0x7f655810e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.330+0000 7f655d082700 1 -- 172.21.15.113:0/1906135393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65580f88a0 msgr2=0x7f655810e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.330+0000 7f655d082700 1 --2- 172.21.15.113:0/1906135393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65580f88a0 0x7f655810e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.330+0000 7f655d082700 1 -- 172.21.15.113:0/1906135393 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654c005040 con 0x7f65580f9200 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.330+0000 7f655d082700 1 --2- 172.21.15.113:0/1906135393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65580f9200 0x7f655810e5d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f655400b4b0 tx=0x7f65540090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.330+0000 7f654af78700 1 -- 172.21.15.113:0/1906135393 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6554002500 con 0x7f65580f9200 2024-04-12T05:47:38.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.330+0000 7f654af78700 1 -- 172.21.15.113:0/1906135393 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6554002b10 con 0x7f65580f9200 2024-04-12T05:47:38.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.330+0000 7f655fae7700 1 -- 172.21.15.113:0/1906135393 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6558108530 con 0x7f65580f9200 2024-04-12T05:47:38.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.330+0000 7f654af78700 1 -- 172.21.15.113:0/1906135393 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6554012cb0 con 0x7f65580f9200 2024-04-12T05:47:38.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.330+0000 7f655fae7700 1 -- 172.21.15.113:0/1906135393 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6558108a70 con 0x7f65580f9200 2024-04-12T05:47:38.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.331+0000 7f655fae7700 1 -- 172.21.15.113:0/1906135393 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f655805f590 con 0x7f65580f9200 2024-04-12T05:47:38.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.332+0000 7f654af78700 1 -- 172.21.15.113:0/1906135393 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f655401c440 con 0x7f65580f9200 2024-04-12T05:47:38.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.332+0000 7f654af78700 1 --2- 172.21.15.113:0/1906135393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f654406caa0 0x7f654406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:38.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.332+0000 7f654af78700 1 -- 172.21.15.113:0/1906135393 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f655408bec0 con 0x7f65580f9200 2024-04-12T05:47:38.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.332+0000 7f655d883700 1 --2- 172.21.15.113:0/1906135393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f654406caa0 0x7f654406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:38.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.333+0000 7f655d883700 1 --2- 172.21.15.113:0/1906135393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f654406caa0 0x7f654406ef60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f654c004f10 tx=0x7f654c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:38.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.337+0000 7f654af78700 1 -- 172.21.15.113:0/1906135393 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f65540599e0 con 0x7f65580f9200 2024-04-12T05:47:38.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.603+0000 7f655fae7700 1 -- 172.21.15.113:0/1906135393 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6558047540 con 0x7f65580f9200 2024-04-12T05:47:38.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.603+0000 7f654af78700 1 -- 172.21.15.113:0/1906135393 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f6554059570 con 0x7f65580f9200 2024-04-12T05:47:38.599 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:47:38.599 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:47:38.600 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:47:38.601 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:47:38.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.605+0000 7f655fae7700 1 -- 172.21.15.113:0/1906135393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f654406caa0 msgr2=0x7f654406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.601 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.605+0000 7f655fae7700 1 --2- 172.21.15.113:0/1906135393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f654406caa0 0x7f654406ef60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f654c004f10 tx=0x7f654c00b040).stop 2024-04-12T05:47:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.605+0000 7f655fae7700 1 -- 172.21.15.113:0/1906135393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65580f9200 msgr2=0x7f655810e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.605+0000 7f655fae7700 1 --2- 172.21.15.113:0/1906135393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65580f9200 0x7f655810e5d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f655400b4b0 tx=0x7f65540090f0).stop 2024-04-12T05:47:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.605+0000 7f655fae7700 1 -- 172.21.15.113:0/1906135393 shutdown_connections 2024-04-12T05:47:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.605+0000 7f655fae7700 1 --2- 172.21.15.113:0/1906135393 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f654406caa0 0x7f654406ef60 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.605+0000 7f655fae7700 1 --2- 172.21.15.113:0/1906135393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65580f88a0 0x7f655810e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.605+0000 7f655fae7700 1 --2- 172.21.15.113:0/1906135393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65580f9200 0x7f655810e5d0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.606+0000 7f655fae7700 1 -- 172.21.15.113:0/1906135393 >> 172.21.15.113:0/1906135393 conn(0x7f6558070590 msgr2=0x7f65581001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.606+0000 7f655fae7700 1 -- 172.21.15.113:0/1906135393 shutdown_connections 2024-04-12T05:47:38.602 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.606+0000 7f655fae7700 1 -- 172.21.15.113:0/1906135393 wait complete. 2024-04-12T05:47:38.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.703+0000 7fe105908700 1 -- 172.21.15.113:0/1088504312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1000fc9c0 msgr2=0x7fe1000f6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.703+0000 7fe105908700 1 --2- 172.21.15.113:0/1088504312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1000fc9c0 0x7fe1000f6240 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fe0f40048f0 tx=0x7fe0f400adc0).stop 2024-04-12T05:47:38.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.703+0000 7fe105908700 1 -- 172.21.15.113:0/1088504312 shutdown_connections 2024-04-12T05:47:38.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.703+0000 7fe105908700 1 --2- 172.21.15.113:0/1088504312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1000fc9c0 0x7fe1000f6240 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.703+0000 7fe105908700 1 --2- 172.21.15.113:0/1088504312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1000fc060 0x7fe1000fc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.703+0000 7fe105908700 1 -- 172.21.15.113:0/1088504312 >> 172.21.15.113:0/1088504312 conn(0x7fe10006ce50 msgr2=0x7fe10006d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.703+0000 7fe105908700 1 -- 172.21.15.113:0/1088504312 shutdown_connections 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.703+0000 7fe105908700 1 -- 172.21.15.113:0/1088504312 wait complete. 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.704+0000 7fe105908700 1 Processor -- start 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.704+0000 7fe105908700 1 -- start start 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.704+0000 7fe105908700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1000fc060 0x7fe10010dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.704+0000 7fe105908700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1000fc9c0 0x7fe10010e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.704+0000 7fe105908700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10010eaa0 con 0x7fe1000fc9c0 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.704+0000 7fe105908700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10010ebe0 con 0x7fe1000fc060 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.704+0000 7fe0fe7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1000fc9c0 0x7fe10010e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.704+0000 7fe0fe7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1000fc9c0 0x7fe10010e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58734/0 (socket says 172.21.15.113:58734) 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.704+0000 7fe0fe7fc700 1 -- 172.21.15.113:0/3487164804 learned_addr learned my addr 172.21.15.113:0/3487164804 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.705+0000 7fe0feffd700 1 --2- 172.21.15.113:0/3487164804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1000fc060 0x7fe10010dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:38.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.705+0000 7fe0fe7fc700 1 -- 172.21.15.113:0/3487164804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1000fc060 msgr2=0x7fe10010dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.705+0000 7fe0fe7fc700 1 --2- 172.21.15.113:0/3487164804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1000fc060 0x7fe10010dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.705+0000 7fe0fe7fc700 1 -- 172.21.15.113:0/3487164804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f4005040 con 0x7fe1000fc9c0 2024-04-12T05:47:38.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.705+0000 7fe0fe7fc700 1 --2- 172.21.15.113:0/3487164804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1000fc9c0 0x7fe10010e4a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7fe0f400c3b0 tx=0x7fe0f400adc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:38.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.705+0000 7fe0e7fff700 1 -- 172.21.15.113:0/3487164804 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0f400f030 con 0x7fe1000fc9c0 2024-04-12T05:47:38.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.705+0000 7fe0e7fff700 1 -- 172.21.15.113:0/3487164804 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0f4007d20 con 0x7fe1000fc9c0 2024-04-12T05:47:38.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.705+0000 7fe105908700 1 -- 172.21.15.113:0/3487164804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe100108210 con 0x7fe1000fc9c0 2024-04-12T05:47:38.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.705+0000 7fe0e7fff700 1 -- 172.21.15.113:0/3487164804 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0f4019070 con 0x7fe1000fc9c0 2024-04-12T05:47:38.703 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.705+0000 7fe105908700 1 -- 172.21.15.113:0/3487164804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe100108770 con 0x7fe1000fc9c0 2024-04-12T05:47:38.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.706+0000 7fe105908700 1 -- 172.21.15.113:0/3487164804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe100047540 con 0x7fe1000fc9c0 2024-04-12T05:47:38.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.707+0000 7fe0e7fff700 1 -- 172.21.15.113:0/3487164804 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe0f4008510 con 0x7fe1000fc9c0 2024-04-12T05:47:38.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.707+0000 7fe0e7fff700 1 --2- 172.21.15.113:0/3487164804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe0e806cab0 0x7fe0e806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:38.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.707+0000 7fe0feffd700 1 --2- 172.21.15.113:0/3487164804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe0e806cab0 0x7fe0e806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:38.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.707+0000 7fe0e7fff700 1 -- 172.21.15.113:0/3487164804 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe0f408d130 con 0x7fe1000fc9c0 2024-04-12T05:47:38.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.708+0000 7fe0feffd700 1 --2- 172.21.15.113:0/3487164804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe0e806cab0 0x7fe0e806ef70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fe0f0004e60 tx=0x7fe0f000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:38.707 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.712+0000 7fe0e7fff700 1 -- 172.21.15.113:0/3487164804 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe0f405abe0 con 0x7fe1000fc9c0 2024-04-12T05:47:38.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.942+0000 7fe105908700 1 -- 172.21.15.113:0/3487164804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe100108db0 con 0x7fe1000fc9c0 2024-04-12T05:47:38.938 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.942+0000 7fe0e7fff700 1 -- 172.21.15.113:0/3487164804 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fe0f405a770 con 0x7fe1000fc9c0 2024-04-12T05:47:38.938 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:47:38.938 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:47:38.939 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:47:38.940 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:47:38.941 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:47:38.941 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:47:38.941 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:47:38.941 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:47:38.941 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:47:38.941 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:38.941 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:38.941 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:38.941 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:47:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.944+0000 7fe105908700 1 -- 172.21.15.113:0/3487164804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe0e806cab0 msgr2=0x7fe0e806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.944+0000 7fe105908700 1 --2- 172.21.15.113:0/3487164804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe0e806cab0 0x7fe0e806ef70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fe0f0004e60 tx=0x7fe0f000b040).stop 2024-04-12T05:47:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.944+0000 7fe105908700 1 -- 172.21.15.113:0/3487164804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1000fc9c0 msgr2=0x7fe10010e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.944+0000 7fe105908700 1 --2- 172.21.15.113:0/3487164804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1000fc9c0 0x7fe10010e4a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7fe0f400c3b0 tx=0x7fe0f400adc0).stop 2024-04-12T05:47:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.944+0000 7fe105908700 1 -- 172.21.15.113:0/3487164804 shutdown_connections 2024-04-12T05:47:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.944+0000 7fe105908700 1 --2- 172.21.15.113:0/3487164804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe0e806cab0 0x7fe0e806ef70 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.944+0000 7fe105908700 1 --2- 172.21.15.113:0/3487164804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1000fc060 0x7fe10010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.944+0000 7fe105908700 1 --2- 172.21.15.113:0/3487164804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe1000fc9c0 0x7fe10010e4a0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.944+0000 7fe105908700 1 -- 172.21.15.113:0/3487164804 >> 172.21.15.113:0/3487164804 conn(0x7fe10006ce50 msgr2=0x7fe1000fa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:38.942 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.944+0000 7fe105908700 1 -- 172.21.15.113:0/3487164804 shutdown_connections 2024-04-12T05:47:38.943 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:38.944+0000 7fe105908700 1 -- 172.21.15.113:0/3487164804 wait complete. 2024-04-12T05:47:38.943 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:47:39.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.043+0000 7f78a9dc3700 1 -- 172.21.15.113:0/3194337803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a40f91d0 msgr2=0x7f78a4101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:39.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.043+0000 7f78a9dc3700 1 --2- 172.21.15.113:0/3194337803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a40f91d0 0x7f78a4101970 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f78980047f0 tx=0x7f7898009cb0).stop 2024-04-12T05:47:39.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.043+0000 7f78a9dc3700 1 -- 172.21.15.113:0/3194337803 shutdown_connections 2024-04-12T05:47:39.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.043+0000 7f78a9dc3700 1 --2- 172.21.15.113:0/3194337803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a40f91d0 0x7f78a4101970 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.043+0000 7f78a9dc3700 1 --2- 172.21.15.113:0/3194337803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78a40f8900 0x7f78a40f8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.043+0000 7f78a9dc3700 1 -- 172.21.15.113:0/3194337803 >> 172.21.15.113:0/3194337803 conn(0x7f78a40f23b0 msgr2=0x7f78a40f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:39.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.043+0000 7f78a9dc3700 1 -- 172.21.15.113:0/3194337803 shutdown_connections 2024-04-12T05:47:39.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.043+0000 7f78a9dc3700 1 -- 172.21.15.113:0/3194337803 wait complete. 2024-04-12T05:47:39.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.044+0000 7f78a9dc3700 1 Processor -- start 2024-04-12T05:47:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.044+0000 7f78a9dc3700 1 -- start start 2024-04-12T05:47:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.044+0000 7f78a9dc3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78a40f8900 0x7f78a40f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a9dc3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a40f91d0 0x7f78a40f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a9dc3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a40f7290 con 0x7f78a40f91d0 2024-04-12T05:47:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a9dc3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a40f73d0 con 0x7f78a40f8900 2024-04-12T05:47:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a40f91d0 0x7f78a40f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:39.042 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a40f91d0 0x7f78a40f6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58752/0 (socket says 172.21.15.113:58752) 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a2ffd700 1 -- 172.21.15.113:0/1649966899 learned_addr learned my addr 172.21.15.113:0/1649966899 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a37fe700 1 --2- 172.21.15.113:0/1649966899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78a40f8900 0x7f78a40f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a2ffd700 1 -- 172.21.15.113:0/1649966899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78a40f8900 msgr2=0x7f78a40f68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a2ffd700 1 --2- 172.21.15.113:0/1649966899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78a40f8900 0x7f78a40f68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a2ffd700 1 -- 172.21.15.113:0/1649966899 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7898005040 con 0x7f78a40f91d0 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a2ffd700 1 --2- 172.21.15.113:0/1649966899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a40f91d0 0x7f78a40f6da0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f7898009380 tx=0x7f7898009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a0ff9700 1 -- 172.21.15.113:0/1649966899 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7898018070 con 0x7f78a40f91d0 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a9dc3700 1 -- 172.21.15.113:0/1649966899 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a40f7620 con 0x7f78a40f91d0 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a0ff9700 1 -- 172.21.15.113:0/1649966899 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7898007e90 con 0x7f78a40f91d0 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.045+0000 7f78a9dc3700 1 -- 172.21.15.113:0/1649966899 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a41147b0 con 0x7f78a40f91d0 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.046+0000 7f78a0ff9700 1 -- 172.21.15.113:0/1649966899 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f789801de30 con 0x7f78a40f91d0 2024-04-12T05:47:39.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.046+0000 7f78a9dc3700 1 -- 172.21.15.113:0/1649966899 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a4060370 con 0x7f78a40f91d0 2024-04-12T05:47:39.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.047+0000 7f78a0ff9700 1 -- 172.21.15.113:0/1649966899 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f789801d420 con 0x7f78a40f91d0 2024-04-12T05:47:39.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.047+0000 7f78a0ff9700 1 --2- 172.21.15.113:0/1649966899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f788c06caa0 0x7f788c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:39.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.047+0000 7f78a0ff9700 1 -- 172.21.15.113:0/1649966899 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f789808ca00 con 0x7f78a40f91d0 2024-04-12T05:47:39.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.047+0000 7f78a37fe700 1 --2- 172.21.15.113:0/1649966899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f788c06caa0 0x7f788c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:39.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.047+0000 7f78a37fe700 1 --2- 172.21.15.113:0/1649966899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f788c06caa0 0x7f788c06ef60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f7894004ee0 tx=0x7f789400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:39.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.051+0000 7f78a0ff9700 1 -- 172.21.15.113:0/1649966899 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7898056a20 con 0x7f78a40f91d0 2024-04-12T05:47:39.254 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.258+0000 7f78a9dc3700 1 -- 172.21.15.113:0/1649966899 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f78a40f7ea0 con 0x7f788c06caa0 2024-04-12T05:47:39.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.259+0000 7f78a0ff9700 1 -- 172.21.15.113:0/1649966899 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f78a40f7ea0 con 0x7f788c06caa0 2024-04-12T05:47:39.255 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:47:39.255 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:47:39.255 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:47:39.255 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:47:39.255 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:47:39.255 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:47:39.255 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:47:39.255 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:47:39.256 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:47:39.256 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:47:39.256 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:47:39.256 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:47:39.256 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:47:39.256 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:47:39.256 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:47:39.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.261+0000 7f78a9dc3700 1 -- 172.21.15.113:0/1649966899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f788c06caa0 msgr2=0x7f788c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:39.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.261+0000 7f78a9dc3700 1 --2- 172.21.15.113:0/1649966899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f788c06caa0 0x7f788c06ef60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f7894004ee0 tx=0x7f789400b040).stop 2024-04-12T05:47:39.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.261+0000 7f78a9dc3700 1 -- 172.21.15.113:0/1649966899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a40f91d0 msgr2=0x7f78a40f6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:39.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.261+0000 7f78a9dc3700 1 --2- 172.21.15.113:0/1649966899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a40f91d0 0x7f78a40f6da0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f7898009380 tx=0x7f7898009cb0).stop 2024-04-12T05:47:39.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.261+0000 7f78a9dc3700 1 -- 172.21.15.113:0/1649966899 shutdown_connections 2024-04-12T05:47:39.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.261+0000 7f78a9dc3700 1 --2- 172.21.15.113:0/1649966899 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f788c06caa0 0x7f788c06ef60 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.261+0000 7f78a9dc3700 1 --2- 172.21.15.113:0/1649966899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78a40f8900 0x7f78a40f68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.261+0000 7f78a9dc3700 1 --2- 172.21.15.113:0/1649966899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78a40f91d0 0x7f78a40f6da0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.261+0000 7f78a9dc3700 1 -- 172.21.15.113:0/1649966899 >> 172.21.15.113:0/1649966899 conn(0x7f78a40f23b0 msgr2=0x7f78a4101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:39.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.261+0000 7f78a9dc3700 1 -- 172.21.15.113:0/1649966899 shutdown_connections 2024-04-12T05:47:39.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.261+0000 7f78a9dc3700 1 -- 172.21.15.113:0/1649966899 wait complete. 2024-04-12T05:47:39.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.361+0000 7f3e77f14700 1 -- 172.21.15.113:0/1898250879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e700fc860 msgr2=0x7f3e700fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:39.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.361+0000 7f3e77f14700 1 --2- 172.21.15.113:0/1898250879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e700fc860 0x7f3e700fcc40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7f3e6c0047f0 tx=0x7f3e6c009cb0).stop 2024-04-12T05:47:39.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.361+0000 7f3e77f14700 1 -- 172.21.15.113:0/1898250879 shutdown_connections 2024-04-12T05:47:39.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.361+0000 7f3e77f14700 1 --2- 172.21.15.113:0/1898250879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e700fd1c0 0x7f3e700f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.361+0000 7f3e77f14700 1 --2- 172.21.15.113:0/1898250879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e700fc860 0x7f3e700fcc40 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.361+0000 7f3e77f14700 1 -- 172.21.15.113:0/1898250879 >> 172.21.15.113:0/1898250879 conn(0x7f3e700f2440 msgr2=0x7f3e700f4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:39.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.361+0000 7f3e77f14700 1 -- 172.21.15.113:0/1898250879 shutdown_connections 2024-04-12T05:47:39.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.361+0000 7f3e77f14700 1 -- 172.21.15.113:0/1898250879 wait complete. 2024-04-12T05:47:39.358 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.362+0000 7f3e77f14700 1 Processor -- start 2024-04-12T05:47:39.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.362+0000 7f3e77f14700 1 -- start start 2024-04-12T05:47:39.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.362+0000 7f3e77f14700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e700fc860 0x7f3e701090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:39.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.362+0000 7f3e77f14700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e700fd1c0 0x7f3e701095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:39.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.362+0000 7f3e77f14700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e70109aa0 con 0x7f3e700fd1c0 2024-04-12T05:47:39.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.362+0000 7f3e77f14700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e70109be0 con 0x7f3e700fc860 2024-04-12T05:47:39.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.362+0000 7f3e754af700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e700fd1c0 0x7f3e701095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:39.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.362+0000 7f3e754af700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e700fd1c0 0x7f3e701095b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:58768/0 (socket says 172.21.15.113:58768) 2024-04-12T05:47:39.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.362+0000 7f3e754af700 1 -- 172.21.15.113:0/1846003351 learned_addr learned my addr 172.21.15.113:0/1846003351 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:47:39.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.362+0000 7f3e75cb0700 1 --2- 172.21.15.113:0/1846003351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e700fc860 0x7f3e701090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:39.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.363+0000 7f3e754af700 1 -- 172.21.15.113:0/1846003351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e700fc860 msgr2=0x7f3e701090c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:39.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.363+0000 7f3e754af700 1 --2- 172.21.15.113:0/1846003351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e700fc860 0x7f3e701090c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.363+0000 7f3e754af700 1 -- 172.21.15.113:0/1846003351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c005040 con 0x7f3e700fd1c0 2024-04-12T05:47:39.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.363+0000 7f3e754af700 1 --2- 172.21.15.113:0/1846003351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e700fd1c0 0x7f3e701095b0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f3e6000c710 tx=0x7f3e60007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:39.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.363+0000 7f3e66ffd700 1 -- 172.21.15.113:0/1846003351 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e6000e680 con 0x7f3e700fd1c0 2024-04-12T05:47:39.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.363+0000 7f3e77f14700 1 -- 172.21.15.113:0/1846003351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e70109e90 con 0x7f3e700fd1c0 2024-04-12T05:47:39.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.363+0000 7f3e66ffd700 1 -- 172.21.15.113:0/1846003351 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e6000ec90 con 0x7f3e700fd1c0 2024-04-12T05:47:39.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.363+0000 7f3e66ffd700 1 -- 172.21.15.113:0/1846003351 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e60013d40 con 0x7f3e700fd1c0 2024-04-12T05:47:39.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.363+0000 7f3e77f14700 1 -- 172.21.15.113:0/1846003351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e70112480 con 0x7f3e700fd1c0 2024-04-12T05:47:39.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.364+0000 7f3e77f14700 1 -- 172.21.15.113:0/1846003351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e70060370 con 0x7f3e700fd1c0 2024-04-12T05:47:39.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.364+0000 7f3e66ffd700 1 -- 172.21.15.113:0/1846003351 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3e6000e7e0 con 0x7f3e700fd1c0 2024-04-12T05:47:39.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.365+0000 7f3e66ffd700 1 --2- 172.21.15.113:0/1846003351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3e5c06caa0 0x7f3e5c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:47:39.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.365+0000 7f3e66ffd700 1 -- 172.21.15.113:0/1846003351 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3e6008bc20 con 0x7f3e700fd1c0 2024-04-12T05:47:39.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.365+0000 7f3e75cb0700 1 --2- 172.21.15.113:0/1846003351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3e5c06caa0 0x7f3e5c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:47:39.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.365+0000 7f3e75cb0700 1 --2- 172.21.15.113:0/1846003351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3e5c06caa0 0x7f3e5c06ef60 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f3e6c007cf0 tx=0x7f3e6c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:47:39.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.369+0000 7f3e66ffd700 1 -- 172.21.15.113:0/1846003351 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3e60055c40 con 0x7f3e700fd1c0 2024-04-12T05:47:39.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.631+0000 7f3e77f14700 1 -- 172.21.15.113:0/1846003351 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3e70061240 con 0x7f3e700fd1c0 2024-04-12T05:47:39.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.631+0000 7f3e66ffd700 1 -- 172.21.15.113:0/1846003351 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f3e60018e10 con 0x7f3e700fd1c0 2024-04-12T05:47:39.628 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:47:39.628 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:47:39.628 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:47:39.628 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:47:39.628 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:47:39.629 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:47:39.630 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:47:39.630 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:47:39.630 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:47:39.630 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:47:39.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.633+0000 7f3e77f14700 1 -- 172.21.15.113:0/1846003351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3e5c06caa0 msgr2=0x7f3e5c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:39.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.633+0000 7f3e77f14700 1 --2- 172.21.15.113:0/1846003351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3e5c06caa0 0x7f3e5c06ef60 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f3e6c007cf0 tx=0x7f3e6c00b040).stop 2024-04-12T05:47:39.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.633+0000 7f3e77f14700 1 -- 172.21.15.113:0/1846003351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e700fd1c0 msgr2=0x7f3e701095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:47:39.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.633+0000 7f3e77f14700 1 --2- 172.21.15.113:0/1846003351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e700fd1c0 0x7f3e701095b0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f3e6000c710 tx=0x7f3e60007ec0).stop 2024-04-12T05:47:39.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.633+0000 7f3e77f14700 1 -- 172.21.15.113:0/1846003351 shutdown_connections 2024-04-12T05:47:39.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.633+0000 7f3e77f14700 1 --2- 172.21.15.113:0/1846003351 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3e5c06caa0 0x7f3e5c06ef60 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.633+0000 7f3e77f14700 1 --2- 172.21.15.113:0/1846003351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e700fc860 0x7f3e701090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.633+0000 7f3e77f14700 1 --2- 172.21.15.113:0/1846003351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e700fd1c0 0x7f3e701095b0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:47:39.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.633+0000 7f3e77f14700 1 -- 172.21.15.113:0/1846003351 >> 172.21.15.113:0/1846003351 conn(0x7f3e700f2440 msgr2=0x7f3e700fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:47:39.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.633+0000 7f3e77f14700 1 -- 172.21.15.113:0/1846003351 shutdown_connections 2024-04-12T05:47:39.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:47:39.633+0000 7f3e77f14700 1 -- 172.21.15.113:0/1846003351 wait complete. 2024-04-12T05:47:39.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:39 smithi113 ceph-mon[103371]: from='client.35906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:39 smithi113 ceph-mon[103371]: from='client.35910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:39 smithi113 ceph-mon[103371]: pgmap v1065: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:39 smithi113 ceph-mon[103371]: from='client.35914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:39 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1906135393' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:39 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3487164804' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:47:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:39 smithi192 ceph-mon[85304]: from='client.35906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:39 smithi192 ceph-mon[85304]: from='client.35910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:39 smithi192 ceph-mon[85304]: pgmap v1065: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:39 smithi192 ceph-mon[85304]: from='client.35914 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:39 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1906135393' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:47:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:39 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3487164804' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:47:40.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:40 smithi113 ceph-mon[103371]: from='client.35926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:40.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:40 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1846003351' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:47:40.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:40 smithi192 ceph-mon[85304]: from='client.35926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:47:40.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:40 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1846003351' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:47:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:41 smithi113 ceph-mon[103371]: pgmap v1066: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:41.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:41 smithi192 ceph-mon[85304]: pgmap v1066: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:43.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:43 smithi113 ceph-mon[103371]: pgmap v1067: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:43.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:43 smithi192 ceph-mon[85304]: pgmap v1067: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:45.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:45 smithi113 ceph-mon[103371]: pgmap v1068: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:45.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:45 smithi192 ceph-mon[85304]: pgmap v1068: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:47.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:47 smithi113 ceph-mon[103371]: pgmap v1069: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:47.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:47 smithi192 ceph-mon[85304]: pgmap v1069: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:49.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:49 smithi113 ceph-mon[103371]: pgmap v1070: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:49.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:49 smithi192 ceph-mon[85304]: pgmap v1070: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:51.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:51 smithi113 ceph-mon[103371]: pgmap v1071: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:51.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:51 smithi192 ceph-mon[85304]: pgmap v1071: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:52.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:52.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:47:53.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:53 smithi113 ceph-mon[103371]: pgmap v1072: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:53.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:53 smithi192 ceph-mon[85304]: pgmap v1072: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:54.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:54 smithi113 ceph-mon[103371]: pgmap v1073: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:54.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:54 smithi192 ceph-mon[85304]: pgmap v1073: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:57.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:57 smithi113 ceph-mon[103371]: pgmap v1074: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:57.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:57 smithi192 ceph-mon[85304]: pgmap v1074: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:59.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:47:59 smithi113 ceph-mon[103371]: pgmap v1075: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:47:59.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:47:59 smithi192 ceph-mon[85304]: pgmap v1075: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:01.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:01 smithi113 ceph-mon[103371]: pgmap v1076: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:01.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:01 smithi192 ceph-mon[85304]: pgmap v1076: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:03.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:03 smithi113 ceph-mon[103371]: pgmap v1077: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:03.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:03 smithi192 ceph-mon[85304]: pgmap v1077: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:05.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:05 smithi113 ceph-mon[103371]: pgmap v1078: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:05.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:05 smithi192 ceph-mon[85304]: pgmap v1078: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:07.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:07 smithi113 ceph-mon[103371]: pgmap v1079: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:07.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:07.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:07 smithi192 ceph-mon[85304]: pgmap v1079: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:07.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:09.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:09 smithi113 ceph-mon[103371]: pgmap v1080: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:09.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:09 smithi192 ceph-mon[85304]: pgmap v1080: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:09.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.745+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1971267419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb680fd150 msgr2=0x7fcb680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:09.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.745+0000 7fcb6f0ca700 1 --2- 172.21.15.113:0/1971267419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb680fd150 0x7fcb680f6930 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7fcb540047f0 tx=0x7fcb54009cb0).stop 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.745+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1971267419 shutdown_connections 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.745+0000 7fcb6f0ca700 1 --2- 172.21.15.113:0/1971267419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb680fd150 0x7fcb680f6930 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.745+0000 7fcb6f0ca700 1 --2- 172.21.15.113:0/1971267419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb680fc7f0 0x7fcb680fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.745+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1971267419 >> 172.21.15.113:0/1971267419 conn(0x7fcb68070590 msgr2=0x7fcb680f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.745+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1971267419 shutdown_connections 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.745+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1971267419 wait complete. 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.746+0000 7fcb6f0ca700 1 Processor -- start 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.746+0000 7fcb6f0ca700 1 -- start start 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.746+0000 7fcb6f0ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb680fc7f0 0x7fcb680f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.746+0000 7fcb6f0ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb680fd150 0x7fcb680f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.746+0000 7fcb6f0ca700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb680f8f20 con 0x7fcb680fc7f0 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.746+0000 7fcb6f0ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb680f9060 con 0x7fcb680fd150 2024-04-12T05:48:09.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.746+0000 7fcb6ce66700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb680fc7f0 0x7fcb680f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:09.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.746+0000 7fcb6ce66700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb680fc7f0 0x7fcb680f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49116/0 (socket says 172.21.15.113:49116) 2024-04-12T05:48:09.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.746+0000 7fcb6ce66700 1 -- 172.21.15.113:0/1524742055 learned_addr learned my addr 172.21.15.113:0/1524742055 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:09.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.747+0000 7fcb67fff700 1 --2- 172.21.15.113:0/1524742055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb680fd150 0x7fcb680f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:09.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.747+0000 7fcb6ce66700 1 -- 172.21.15.113:0/1524742055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb680fd150 msgr2=0x7fcb680f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:09.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.747+0000 7fcb6ce66700 1 --2- 172.21.15.113:0/1524742055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb680fd150 0x7fcb680f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:09.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.747+0000 7fcb6ce66700 1 -- 172.21.15.113:0/1524742055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb54005040 con 0x7fcb680fc7f0 2024-04-12T05:48:09.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.747+0000 7fcb6ce66700 1 --2- 172.21.15.113:0/1524742055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb680fc7f0 0x7fcb680f8540 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7fcb5c00e910 tx=0x7fcb5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:09.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.747+0000 7fcb65ffb700 1 -- 172.21.15.113:0/1524742055 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb5c00eeb0 con 0x7fcb680fc7f0 2024-04-12T05:48:09.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.747+0000 7fcb65ffb700 1 -- 172.21.15.113:0/1524742055 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb5c00f040 con 0x7fcb680fc7f0 2024-04-12T05:48:09.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.747+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1524742055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb680f92e0 con 0x7fcb680fc7f0 2024-04-12T05:48:09.745 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.747+0000 7fcb65ffb700 1 -- 172.21.15.113:0/1524742055 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb5c00c8a0 con 0x7fcb680fc7f0 2024-04-12T05:48:09.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.747+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1524742055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb68114760 con 0x7fcb680fc7f0 2024-04-12T05:48:09.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.748+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1524742055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6805f590 con 0x7fcb680fc7f0 2024-04-12T05:48:09.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.749+0000 7fcb65ffb700 1 -- 172.21.15.113:0/1524742055 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcb5c0090d0 con 0x7fcb680fc7f0 2024-04-12T05:48:09.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.749+0000 7fcb65ffb700 1 --2- 172.21.15.113:0/1524742055 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb5806caa0 0x7fcb5806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:09.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.749+0000 7fcb65ffb700 1 -- 172.21.15.113:0/1524742055 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fcb5c08b880 con 0x7fcb680fc7f0 2024-04-12T05:48:09.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.749+0000 7fcb67fff700 1 --2- 172.21.15.113:0/1524742055 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb5806caa0 0x7fcb5806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:09.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.749+0000 7fcb67fff700 1 --2- 172.21.15.113:0/1524742055 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb5806caa0 0x7fcb5806ef60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fcb54004e60 tx=0x7fcb5400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:09.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.754+0000 7fcb65ffb700 1 -- 172.21.15.113:0/1524742055 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fcb5c0558a0 con 0x7fcb680fc7f0 2024-04-12T05:48:09.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.965+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1524742055 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb680fb5a0 con 0x7fcb5806caa0 2024-04-12T05:48:09.962 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.966+0000 7fcb65ffb700 1 -- 172.21.15.113:0/1524742055 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcb680fb5a0 con 0x7fcb5806caa0 2024-04-12T05:48:09.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.968+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1524742055 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb5806caa0 msgr2=0x7fcb5806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.968+0000 7fcb6f0ca700 1 --2- 172.21.15.113:0/1524742055 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb5806caa0 0x7fcb5806ef60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fcb54004e60 tx=0x7fcb5400b040).stop 2024-04-12T05:48:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.968+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1524742055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb680fc7f0 msgr2=0x7fcb680f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.968+0000 7fcb6f0ca700 1 --2- 172.21.15.113:0/1524742055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb680fc7f0 0x7fcb680f8540 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7fcb5c00e910 tx=0x7fcb5c00b040).stop 2024-04-12T05:48:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.968+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1524742055 shutdown_connections 2024-04-12T05:48:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.968+0000 7fcb6f0ca700 1 --2- 172.21.15.113:0/1524742055 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb5806caa0 0x7fcb5806ef60 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.968+0000 7fcb6f0ca700 1 --2- 172.21.15.113:0/1524742055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb680fc7f0 0x7fcb680f8540 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.968+0000 7fcb6f0ca700 1 --2- 172.21.15.113:0/1524742055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb680fd150 0x7fcb680f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.968+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1524742055 >> 172.21.15.113:0/1524742055 conn(0x7fcb68070590 msgr2=0x7fcb68060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.969+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1524742055 shutdown_connections 2024-04-12T05:48:09.965 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:09.969+0000 7fcb6f0ca700 1 -- 172.21.15.113:0/1524742055 wait complete. 2024-04-12T05:48:09.974 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:48:10.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.067+0000 7fd1a8456700 1 -- 172.21.15.113:0/1006794131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a00fd940 msgr2=0x7fd1a00fdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.067+0000 7fd1a8456700 1 --2- 172.21.15.113:0/1006794131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a00fd940 0x7fd1a00fdd20 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7fd19c0047f0 tx=0x7fd19c009cb0).stop 2024-04-12T05:48:10.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.068+0000 7fd1a8456700 1 -- 172.21.15.113:0/1006794131 shutdown_connections 2024-04-12T05:48:10.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.068+0000 7fd1a8456700 1 --2- 172.21.15.113:0/1006794131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1a00fe2a0 0x7fd1a00606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.068+0000 7fd1a8456700 1 --2- 172.21.15.113:0/1006794131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a00fd940 0x7fd1a00fdd20 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.068+0000 7fd1a8456700 1 -- 172.21.15.113:0/1006794131 >> 172.21.15.113:0/1006794131 conn(0x7fd1a006c760 msgr2=0x7fd1a006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.068+0000 7fd1a8456700 1 -- 172.21.15.113:0/1006794131 shutdown_connections 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.068+0000 7fd1a8456700 1 -- 172.21.15.113:0/1006794131 wait complete. 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.068+0000 7fd1a8456700 1 Processor -- start 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.068+0000 7fd1a8456700 1 -- start start 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.068+0000 7fd1a8456700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1a00fd940 0x7fd1a010e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.068+0000 7fd1a8456700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a00fe2a0 0x7fd1a010e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a8456700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a010eb00 con 0x7fd1a00fe2a0 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a8456700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a010ec40 con 0x7fd1a00fd940 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a59f1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a00fe2a0 0x7fd1a010e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a59f1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a00fe2a0 0x7fd1a010e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:49136/0 (socket says 172.21.15.113:49136) 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a59f1700 1 -- 172.21.15.113:0/1329404424 learned_addr learned my addr 172.21.15.113:0/1329404424 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a61f2700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1a00fd940 0x7fd1a010e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:10.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a59f1700 1 -- 172.21.15.113:0/1329404424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1a00fd940 msgr2=0x7fd1a010e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a59f1700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1a00fd940 0x7fd1a010e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a59f1700 1 -- 172.21.15.113:0/1329404424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c005040 con 0x7fd1a00fe2a0 2024-04-12T05:48:10.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a61f2700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1a00fd940 0x7fd1a010e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:48:10.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a59f1700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a00fe2a0 0x7fd1a010e500 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7fd19000c790 tx=0x7fd190007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:10.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1977fe700 1 -- 172.21.15.113:0/1329404424 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1900106f0 con 0x7fd1a00fe2a0 2024-04-12T05:48:10.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1977fe700 1 -- 172.21.15.113:0/1329404424 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd190010d00 con 0x7fd1a00fe2a0 2024-04-12T05:48:10.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1977fe700 1 -- 172.21.15.113:0/1329404424 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd190012d50 con 0x7fd1a00fe2a0 2024-04-12T05:48:10.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a8456700 1 -- 172.21.15.113:0/1329404424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a01082d0 con 0x7fd1a00fe2a0 2024-04-12T05:48:10.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.069+0000 7fd1a8456700 1 -- 172.21.15.113:0/1329404424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a0108890 con 0x7fd1a00fe2a0 2024-04-12T05:48:10.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.070+0000 7fd1a8456700 1 -- 172.21.15.113:0/1329404424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a005f530 con 0x7fd1a00fe2a0 2024-04-12T05:48:10.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.071+0000 7fd1977fe700 1 -- 172.21.15.113:0/1329404424 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd190012570 con 0x7fd1a00fe2a0 2024-04-12T05:48:10.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.071+0000 7fd1977fe700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd18c06caa0 0x7fd18c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:10.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.071+0000 7fd1977fe700 1 -- 172.21.15.113:0/1329404424 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd19008b790 con 0x7fd1a00fe2a0 2024-04-12T05:48:10.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.071+0000 7fd1a61f2700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd18c06caa0 0x7fd18c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:10.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.071+0000 7fd1a61f2700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd18c06caa0 0x7fd18c06ef60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fd19c007cf0 tx=0x7fd19c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:10.072 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.076+0000 7fd1977fe700 1 -- 172.21.15.113:0/1329404424 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd190059240 con 0x7fd1a00fe2a0 2024-04-12T05:48:10.280 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.284+0000 7fd1a8456700 1 -- 172.21.15.113:0/1329404424 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1a0108b10 con 0x7fd18c06caa0 2024-04-12T05:48:10.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.286+0000 7fd1977fe700 1 -- 172.21.15.113:0/1329404424 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd1a0108b10 con 0x7fd18c06caa0 2024-04-12T05:48:10.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.288+0000 7fd1a8456700 1 -- 172.21.15.113:0/1329404424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd18c06caa0 msgr2=0x7fd18c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.288+0000 7fd1a8456700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd18c06caa0 0x7fd18c06ef60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fd19c007cf0 tx=0x7fd19c00b040).stop 2024-04-12T05:48:10.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.288+0000 7fd1a8456700 1 -- 172.21.15.113:0/1329404424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a00fe2a0 msgr2=0x7fd1a010e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.288+0000 7fd1a8456700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a00fe2a0 0x7fd1a010e500 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7fd19000c790 tx=0x7fd190007f50).stop 2024-04-12T05:48:10.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.288+0000 7fd1a8456700 1 -- 172.21.15.113:0/1329404424 shutdown_connections 2024-04-12T05:48:10.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.288+0000 7fd1a8456700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd18c06caa0 0x7fd18c06ef60 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.288+0000 7fd1a8456700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1a00fd940 0x7fd1a010e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.288+0000 7fd1a8456700 1 --2- 172.21.15.113:0/1329404424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a00fe2a0 0x7fd1a010e500 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.288+0000 7fd1a8456700 1 -- 172.21.15.113:0/1329404424 >> 172.21.15.113:0/1329404424 conn(0x7fd1a006c760 msgr2=0x7fd1a00fbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:10.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.288+0000 7fd1a8456700 1 -- 172.21.15.113:0/1329404424 shutdown_connections 2024-04-12T05:48:10.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.288+0000 7fd1a8456700 1 -- 172.21.15.113:0/1329404424 wait complete. 2024-04-12T05:48:10.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.386+0000 7f647640f700 1 -- 172.21.15.113:0/2335455774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64700fd150 msgr2=0x7f64700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.386+0000 7f647640f700 1 --2- 172.21.15.113:0/2335455774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64700fd150 0x7f64700f6930 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7f64600047f0 tx=0x7f6460009cb0).stop 2024-04-12T05:48:10.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.386+0000 7f647640f700 1 -- 172.21.15.113:0/2335455774 shutdown_connections 2024-04-12T05:48:10.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.386+0000 7f647640f700 1 --2- 172.21.15.113:0/2335455774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64700fd150 0x7f64700f6930 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.386+0000 7f647640f700 1 --2- 172.21.15.113:0/2335455774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64700fc7f0 0x7f64700fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.386+0000 7f647640f700 1 -- 172.21.15.113:0/2335455774 >> 172.21.15.113:0/2335455774 conn(0x7f6470070590 msgr2=0x7f64700f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:10.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.386+0000 7f647640f700 1 -- 172.21.15.113:0/2335455774 shutdown_connections 2024-04-12T05:48:10.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.386+0000 7f647640f700 1 -- 172.21.15.113:0/2335455774 wait complete. 2024-04-12T05:48:10.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.386+0000 7f647640f700 1 Processor -- start 2024-04-12T05:48:10.383 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.386+0000 7f647640f700 1 -- start start 2024-04-12T05:48:10.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f647640f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64700fc7f0 0x7f64701090e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:10.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f647640f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64700fd150 0x7f64701095d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:10.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f647640f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6470109ac0 con 0x7f64700fd150 2024-04-12T05:48:10.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f647640f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6470109c00 con 0x7f64700fc7f0 2024-04-12T05:48:10.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f646f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64700fd150 0x7f64701095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:10.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f646f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64700fd150 0x7f64701095d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41530/0 (socket says 172.21.15.113:41530) 2024-04-12T05:48:10.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f646f7fe700 1 -- 172.21.15.113:0/1021341392 learned_addr learned my addr 172.21.15.113:0/1021341392 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:10.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f646f7fe700 1 -- 172.21.15.113:0/1021341392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64700fc7f0 msgr2=0x7f64701090e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f646ffff700 1 --2- 172.21.15.113:0/1021341392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64700fc7f0 0x7f64701090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:10.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f646f7fe700 1 --2- 172.21.15.113:0/1021341392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64700fc7f0 0x7f64701090e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f646f7fe700 1 -- 172.21.15.113:0/1021341392 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6460005040 con 0x7f64700fd150 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f646f7fe700 1 --2- 172.21.15.113:0/1021341392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64700fd150 0x7f64701095d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7f64600099f0 tx=0x7f6460009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f646d7fa700 1 -- 172.21.15.113:0/1021341392 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f646000e030 con 0x7f64700fd150 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f646d7fa700 1 -- 172.21.15.113:0/1021341392 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6460007e90 con 0x7f64700fd150 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f647640f700 1 -- 172.21.15.113:0/1021341392 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6470109e50 con 0x7f64700fd150 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f646d7fa700 1 -- 172.21.15.113:0/1021341392 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6460018070 con 0x7f64700fd150 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.387+0000 7f647640f700 1 -- 172.21.15.113:0/1021341392 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6470112440 con 0x7f64700fd150 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.388+0000 7f647640f700 1 -- 172.21.15.113:0/1021341392 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647005f590 con 0x7f64700fd150 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.389+0000 7f646d7fa700 1 -- 172.21.15.113:0/1021341392 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6460009590 con 0x7f64700fd150 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.389+0000 7f646d7fa700 1 --2- 172.21.15.113:0/1021341392 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f645806ca30 0x7f645806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.389+0000 7f646d7fa700 1 -- 172.21.15.113:0/1021341392 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f646008d0f0 con 0x7f64700fd150 2024-04-12T05:48:10.386 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.389+0000 7f646ffff700 1 --2- 172.21.15.113:0/1021341392 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f645806ca30 0x7f645806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:10.387 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.390+0000 7f646ffff700 1 --2- 172.21.15.113:0/1021341392 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f645806ca30 0x7f645806eef0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f6464004be0 tx=0x7f646400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:10.389 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.394+0000 7f646d7fa700 1 -- 172.21.15.113:0/1021341392 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6460057110 con 0x7f64700fd150 2024-04-12T05:48:10.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.591+0000 7f647640f700 1 -- 172.21.15.113:0/1021341392 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6470059120 con 0x7f645806ca30 2024-04-12T05:48:10.597 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.601+0000 7f646d7fa700 1 -- 172.21.15.113:0/1021341392 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6470059120 con 0x7f645806ca30 2024-04-12T05:48:10.597 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:48:10.597 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (36m) 13m ago 47m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:48:10.597 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (33m) 13m ago 47m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:48:10.597 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (33m) 13m ago 46m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:48:10.597 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (35m) 13m ago 46m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (29m) 13m ago 41m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (28m) 13m ago 41m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (28m) 13m ago 41m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (13m) 13m ago 41m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (37m) 13m ago 48m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (36m) 13m ago 46m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (34m) 13m ago 48m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (34m) 13m ago 46m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (36m) 13m ago 46m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (36m) 13m ago 46m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (33m) 13m ago 45m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (32m) 13m ago 45m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:48:10.598 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (31m) 13m ago 44m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:48:10.599 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (31m) 13m ago 44m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:48:10.599 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (30m) 13m ago 43m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:48:10.599 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (29m) 13m ago 43m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:48:10.599 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (36m) 13m ago 46m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:48:10.599 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.603+0000 7f647640f700 1 -- 172.21.15.113:0/1021341392 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f645806ca30 msgr2=0x7f645806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.603+0000 7f647640f700 1 --2- 172.21.15.113:0/1021341392 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f645806ca30 0x7f645806eef0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f6464004be0 tx=0x7f646400b040).stop 2024-04-12T05:48:10.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.603+0000 7f647640f700 1 -- 172.21.15.113:0/1021341392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64700fd150 msgr2=0x7f64701095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.603+0000 7f647640f700 1 --2- 172.21.15.113:0/1021341392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64700fd150 0x7f64701095d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7f64600099f0 tx=0x7f6460009cb0).stop 2024-04-12T05:48:10.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.603+0000 7f647640f700 1 -- 172.21.15.113:0/1021341392 shutdown_connections 2024-04-12T05:48:10.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.603+0000 7f647640f700 1 --2- 172.21.15.113:0/1021341392 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f645806ca30 0x7f645806eef0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.603+0000 7f647640f700 1 --2- 172.21.15.113:0/1021341392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64700fc7f0 0x7f64701090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.603+0000 7f647640f700 1 --2- 172.21.15.113:0/1021341392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f64700fd150 0x7f64701095d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.603+0000 7f647640f700 1 -- 172.21.15.113:0/1021341392 >> 172.21.15.113:0/1021341392 conn(0x7f6470070590 msgr2=0x7f64700fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:10.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.603+0000 7f647640f700 1 -- 172.21.15.113:0/1021341392 shutdown_connections 2024-04-12T05:48:10.600 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.603+0000 7f647640f700 1 -- 172.21.15.113:0/1021341392 wait complete. 2024-04-12T05:48:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.700+0000 7f131924d700 1 -- 172.21.15.113:0/2050545045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13140fd150 msgr2=0x7f13140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.700+0000 7f131924d700 1 --2- 172.21.15.113:0/2050545045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13140fd150 0x7f13140f6930 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7f13040047f0 tx=0x7f1304009cb0).stop 2024-04-12T05:48:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.700+0000 7f131924d700 1 -- 172.21.15.113:0/2050545045 shutdown_connections 2024-04-12T05:48:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.700+0000 7f131924d700 1 --2- 172.21.15.113:0/2050545045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13140fd150 0x7f13140f6930 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.700+0000 7f131924d700 1 --2- 172.21.15.113:0/2050545045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13140fc7f0 0x7f13140fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.700+0000 7f131924d700 1 -- 172.21.15.113:0/2050545045 >> 172.21.15.113:0/2050545045 conn(0x7f1314070590 msgr2=0x7f13140f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.700+0000 7f131924d700 1 -- 172.21.15.113:0/2050545045 shutdown_connections 2024-04-12T05:48:10.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.700+0000 7f131924d700 1 -- 172.21.15.113:0/2050545045 wait complete. 2024-04-12T05:48:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.701+0000 7f131924d700 1 Processor -- start 2024-04-12T05:48:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.701+0000 7f131924d700 1 -- start start 2024-04-12T05:48:10.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.701+0000 7f131924d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13140fc7f0 0x7f131410e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.701+0000 7f131924d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13140fd150 0x7f131410e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.701+0000 7f131924d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131410eb90 con 0x7f13140fd150 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.701+0000 7f131924d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131410ecd0 con 0x7f13140fc7f0 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.701+0000 7f131259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13140fd150 0x7f131410e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.701+0000 7f131259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13140fd150 0x7f131410e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41548/0 (socket says 172.21.15.113:41548) 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.701+0000 7f131259c700 1 -- 172.21.15.113:0/2294745357 learned_addr learned my addr 172.21.15.113:0/2294745357 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.701+0000 7f1312d9d700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13140fc7f0 0x7f131410e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.702+0000 7f131259c700 1 -- 172.21.15.113:0/2294745357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13140fc7f0 msgr2=0x7f131410e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.702+0000 7f131259c700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13140fc7f0 0x7f131410e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.702+0000 7f131259c700 1 -- 172.21.15.113:0/2294745357 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304005040 con 0x7f13140fd150 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.702+0000 7f1312d9d700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13140fc7f0 0x7f131410e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.702+0000 7f131259c700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13140fd150 0x7f131410e590 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f1304009360 tx=0x7f1304009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:10.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.702+0000 7f130bfff700 1 -- 172.21.15.113:0/2294745357 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1304018070 con 0x7f13140fd150 2024-04-12T05:48:10.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.702+0000 7f130bfff700 1 -- 172.21.15.113:0/2294745357 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1304007e90 con 0x7f13140fd150 2024-04-12T05:48:10.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.702+0000 7f131924d700 1 -- 172.21.15.113:0/2294745357 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1314108340 con 0x7f13140fd150 2024-04-12T05:48:10.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.702+0000 7f130bfff700 1 -- 172.21.15.113:0/2294745357 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13040123c0 con 0x7f13140fd150 2024-04-12T05:48:10.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.702+0000 7f131924d700 1 -- 172.21.15.113:0/2294745357 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13141088a0 con 0x7f13140fd150 2024-04-12T05:48:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.704+0000 7f130bfff700 1 -- 172.21.15.113:0/2294745357 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f130401d420 con 0x7f13140fd150 2024-04-12T05:48:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.704+0000 7f131924d700 1 -- 172.21.15.113:0/2294745357 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131405f590 con 0x7f13140fd150 2024-04-12T05:48:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.704+0000 7f130bfff700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f130006cab0 0x7f130006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.704+0000 7f130bfff700 1 -- 172.21.15.113:0/2294745357 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f130408ca60 con 0x7f13140fd150 2024-04-12T05:48:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.704+0000 7f1312d9d700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f130006cab0 0x7f130006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:10.701 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.705+0000 7f1312d9d700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f130006cab0 0x7f130006ef70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f12fc007bf0 tx=0x7f12fc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:10.704 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.708+0000 7f130bfff700 1 -- 172.21.15.113:0/2294745357 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1304056a80 con 0x7f13140fd150 2024-04-12T05:48:10.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.976+0000 7f131924d700 1 -- 172.21.15.113:0/2294745357 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1314047540 con 0x7f13140fd150 2024-04-12T05:48:10.972 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.976+0000 7f130bfff700 1 -- 172.21.15.113:0/2294745357 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f130405a0a0 con 0x7f13140fd150 2024-04-12T05:48:10.972 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:48:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:48:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:48:10.972 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:48:10.973 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:48:10.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.978+0000 7f131924d700 1 -- 172.21.15.113:0/2294745357 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f130006cab0 msgr2=0x7f130006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.978+0000 7f131924d700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f130006cab0 0x7f130006ef70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f12fc007bf0 tx=0x7f12fc00b040).stop 2024-04-12T05:48:10.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.978+0000 7f131924d700 1 -- 172.21.15.113:0/2294745357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13140fd150 msgr2=0x7f131410e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:10.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.978+0000 7f131924d700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13140fd150 0x7f131410e590 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f1304009360 tx=0x7f1304009cb0).stop 2024-04-12T05:48:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.978+0000 7f131924d700 1 -- 172.21.15.113:0/2294745357 shutdown_connections 2024-04-12T05:48:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.978+0000 7f131924d700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f130006cab0 0x7f130006ef70 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.978+0000 7f131924d700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13140fc7f0 0x7f131410e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.978+0000 7f131924d700 1 --2- 172.21.15.113:0/2294745357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13140fd150 0x7f131410e590 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.978+0000 7f131924d700 1 -- 172.21.15.113:0/2294745357 >> 172.21.15.113:0/2294745357 conn(0x7f1314070590 msgr2=0x7f1314060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.978+0000 7f131924d700 1 -- 172.21.15.113:0/2294745357 shutdown_connections 2024-04-12T05:48:10.975 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:10.978+0000 7f131924d700 1 -- 172.21.15.113:0/2294745357 wait complete. 2024-04-12T05:48:11.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.077+0000 7fe351cce700 1 -- 172.21.15.113:0/1485858189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c0fd150 msgr2=0x7fe34c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:11.073 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.077+0000 7fe351cce700 1 --2- 172.21.15.113:0/1485858189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c0fd150 0x7fe34c0f6930 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7fe3400047f0 tx=0x7fe340009cb0).stop 2024-04-12T05:48:11.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.077+0000 7fe351cce700 1 -- 172.21.15.113:0/1485858189 shutdown_connections 2024-04-12T05:48:11.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.077+0000 7fe351cce700 1 --2- 172.21.15.113:0/1485858189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c0fd150 0x7fe34c0f6930 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.077+0000 7fe351cce700 1 --2- 172.21.15.113:0/1485858189 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0fc7f0 0x7fe34c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.077+0000 7fe351cce700 1 -- 172.21.15.113:0/1485858189 >> 172.21.15.113:0/1485858189 conn(0x7fe34c070590 msgr2=0x7fe34c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:11.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.077+0000 7fe351cce700 1 -- 172.21.15.113:0/1485858189 shutdown_connections 2024-04-12T05:48:11.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.077+0000 7fe351cce700 1 -- 172.21.15.113:0/1485858189 wait complete. 2024-04-12T05:48:11.074 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.078+0000 7fe351cce700 1 Processor -- start 2024-04-12T05:48:11.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.078+0000 7fe351cce700 1 -- start start 2024-04-12T05:48:11.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.078+0000 7fe351cce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0fc7f0 0x7fe34c10cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:11.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.078+0000 7fe351cce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c0fd150 0x7fe34c10d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:11.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.078+0000 7fe351cce700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c10d9e0 con 0x7fe34c0fd150 2024-04-12T05:48:11.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.078+0000 7fe351cce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c10db20 con 0x7fe34c0fc7f0 2024-04-12T05:48:11.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.078+0000 7fe34affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c0fd150 0x7fe34c10d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:11.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.078+0000 7fe34affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c0fd150 0x7fe34c10d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41576/0 (socket says 172.21.15.113:41576) 2024-04-12T05:48:11.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.078+0000 7fe34affd700 1 -- 172.21.15.113:0/235197878 learned_addr learned my addr 172.21.15.113:0/235197878 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:11.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.078+0000 7fe34b7fe700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0fc7f0 0x7fe34c10cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.079+0000 7fe34affd700 1 -- 172.21.15.113:0/235197878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0fc7f0 msgr2=0x7fe34c10cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.079+0000 7fe34affd700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0fc7f0 0x7fe34c10cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.079+0000 7fe34affd700 1 -- 172.21.15.113:0/235197878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe340005040 con 0x7fe34c0fd150 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.079+0000 7fe34b7fe700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0fc7f0 0x7fe34c10cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.079+0000 7fe34affd700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c0fd150 0x7fe34c10d3e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7fe3400099f0 tx=0x7fe340009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.079+0000 7fe348ff9700 1 -- 172.21.15.113:0/235197878 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe34000e030 con 0x7fe34c0fd150 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.079+0000 7fe348ff9700 1 -- 172.21.15.113:0/235197878 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe340007e90 con 0x7fe34c0fd150 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.079+0000 7fe351cce700 1 -- 172.21.15.113:0/235197878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34c107150 con 0x7fe34c0fd150 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.079+0000 7fe348ff9700 1 -- 172.21.15.113:0/235197878 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe340018070 con 0x7fe34c0fd150 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.079+0000 7fe351cce700 1 -- 172.21.15.113:0/235197878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34c1076b0 con 0x7fe34c0fd150 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.080+0000 7fe351cce700 1 -- 172.21.15.113:0/235197878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34c05f590 con 0x7fe34c0fd150 2024-04-12T05:48:11.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.080+0000 7fe348ff9700 1 -- 172.21.15.113:0/235197878 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe340009590 con 0x7fe34c0fd150 2024-04-12T05:48:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.081+0000 7fe348ff9700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe33406caa0 0x7fe33406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.081+0000 7fe348ff9700 1 -- 172.21.15.113:0/235197878 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe34008d010 con 0x7fe34c0fd150 2024-04-12T05:48:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.081+0000 7fe34b7fe700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe33406caa0 0x7fe33406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:11.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.081+0000 7fe34b7fe700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe33406caa0 0x7fe33406ef60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7fe33c009250 tx=0x7fe33c009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:11.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.085+0000 7fe348ff9700 1 -- 172.21.15.113:0/235197878 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe34005aac0 con 0x7fe34c0fd150 2024-04-12T05:48:11.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.317+0000 7fe351cce700 1 -- 172.21.15.113:0/235197878 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe34c047540 con 0x7fe34c0fd150 2024-04-12T05:48:11.313 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.318+0000 7fe348ff9700 1 -- 172.21.15.113:0/235197878 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fe34005a650 con 0x7fe34c0fd150 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:48:11.314 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:48:11.315 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:11.316 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:11.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.320+0000 7fe351cce700 1 -- 172.21.15.113:0/235197878 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe33406caa0 msgr2=0x7fe33406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:11.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.320+0000 7fe351cce700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe33406caa0 0x7fe33406ef60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7fe33c009250 tx=0x7fe33c009da0).stop 2024-04-12T05:48:11.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.320+0000 7fe351cce700 1 -- 172.21.15.113:0/235197878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c0fd150 msgr2=0x7fe34c10d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:11.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.320+0000 7fe351cce700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c0fd150 0x7fe34c10d3e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7fe3400099f0 tx=0x7fe340009cb0).stop 2024-04-12T05:48:11.317 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.320+0000 7fe351cce700 1 -- 172.21.15.113:0/235197878 shutdown_connections 2024-04-12T05:48:11.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.320+0000 7fe351cce700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe33406caa0 0x7fe33406ef60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.320+0000 7fe351cce700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0fc7f0 0x7fe34c10cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.320+0000 7fe351cce700 1 --2- 172.21.15.113:0/235197878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe34c0fd150 0x7fe34c10d3e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.320+0000 7fe351cce700 1 -- 172.21.15.113:0/235197878 >> 172.21.15.113:0/235197878 conn(0x7fe34c070590 msgr2=0x7fe34c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:11.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.320+0000 7fe351cce700 1 -- 172.21.15.113:0/235197878 shutdown_connections 2024-04-12T05:48:11.318 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.320+0000 7fe351cce700 1 -- 172.21.15.113:0/235197878 wait complete. 2024-04-12T05:48:11.318 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:48:11.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.418+0000 7fd1129f9700 1 -- 172.21.15.113:0/2709605977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0f9210 msgr2=0x7fd10c1019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:11.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.418+0000 7fd1129f9700 1 --2- 172.21.15.113:0/2709605977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0f9210 0x7fd10c1019b0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7fd0fc0047f0 tx=0x7fd0fc009cb0).stop 2024-04-12T05:48:11.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.419+0000 7fd1129f9700 1 -- 172.21.15.113:0/2709605977 shutdown_connections 2024-04-12T05:48:11.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.419+0000 7fd1129f9700 1 --2- 172.21.15.113:0/2709605977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0f9210 0x7fd10c1019b0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.419+0000 7fd1129f9700 1 --2- 172.21.15.113:0/2709605977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd10c0f8940 0x7fd10c0f8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.419+0000 7fd1129f9700 1 -- 172.21.15.113:0/2709605977 >> 172.21.15.113:0/2709605977 conn(0x7fd10c070590 msgr2=0x7fd10c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:11.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.419+0000 7fd1129f9700 1 -- 172.21.15.113:0/2709605977 shutdown_connections 2024-04-12T05:48:11.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.419+0000 7fd1129f9700 1 -- 172.21.15.113:0/2709605977 wait complete. 2024-04-12T05:48:11.415 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.419+0000 7fd1129f9700 1 Processor -- start 2024-04-12T05:48:11.416 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.419+0000 7fd1129f9700 1 -- start start 2024-04-12T05:48:11.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd1129f9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd10c0f8940 0x7fd10c10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:11.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd1129f9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0f9210 0x7fd10c10e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:11.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd1129f9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c10eb20 con 0x7fd10c0f9210 2024-04-12T05:48:11.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd1129f9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c10ec60 con 0x7fd10c0f8940 2024-04-12T05:48:11.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd10b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0f9210 0x7fd10c10e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:11.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd10b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0f9210 0x7fd10c10e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41600/0 (socket says 172.21.15.113:41600) 2024-04-12T05:48:11.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd10b7fe700 1 -- 172.21.15.113:0/2015613042 learned_addr learned my addr 172.21.15.113:0/2015613042 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:11.417 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd10bfff700 1 --2- 172.21.15.113:0/2015613042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd10c0f8940 0x7fd10c10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:11.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd10b7fe700 1 -- 172.21.15.113:0/2015613042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd10c0f8940 msgr2=0x7fd10c10e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:11.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd10b7fe700 1 --2- 172.21.15.113:0/2015613042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd10c0f8940 0x7fd10c10e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd10b7fe700 1 -- 172.21.15.113:0/2015613042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc005040 con 0x7fd10c0f9210 2024-04-12T05:48:11.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd10b7fe700 1 --2- 172.21.15.113:0/2015613042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0f9210 0x7fd10c10e520 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7fd0fc005010 tx=0x7fd0fc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:11.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd1097fa700 1 -- 172.21.15.113:0/2015613042 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0fc00e030 con 0x7fd10c0f9210 2024-04-12T05:48:11.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd1097fa700 1 -- 172.21.15.113:0/2015613042 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0fc007e90 con 0x7fd10c0f9210 2024-04-12T05:48:11.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd1129f9700 1 -- 172.21.15.113:0/2015613042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10c108290 con 0x7fd10c0f9210 2024-04-12T05:48:11.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd1097fa700 1 -- 172.21.15.113:0/2015613042 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0fc018070 con 0x7fd10c0f9210 2024-04-12T05:48:11.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.420+0000 7fd1129f9700 1 -- 172.21.15.113:0/2015613042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10c108770 con 0x7fd10c0f9210 2024-04-12T05:48:11.418 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.421+0000 7fd1129f9700 1 -- 172.21.15.113:0/2015613042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd10c05f590 con 0x7fd10c0f9210 2024-04-12T05:48:11.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:11 smithi113 ceph-mon[103371]: from='client.35934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:11.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:11 smithi113 ceph-mon[103371]: pgmap v1081: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:11.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:11 smithi113 ceph-mon[103371]: from='client.35938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:11.419 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:11 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2294745357' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:48:11.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.422+0000 7fd1097fa700 1 -- 172.21.15.113:0/2015613042 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd0fc01d670 con 0x7fd10c0f9210 2024-04-12T05:48:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.422+0000 7fd1097fa700 1 --2- 172.21.15.113:0/2015613042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0f406caa0 0x7fd0f406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.422+0000 7fd1097fa700 1 -- 172.21.15.113:0/2015613042 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd0fc08d0d0 con 0x7fd10c0f9210 2024-04-12T05:48:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.422+0000 7fd10bfff700 1 --2- 172.21.15.113:0/2015613042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0f406caa0 0x7fd0f406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:11.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.423+0000 7fd10bfff700 1 --2- 172.21.15.113:0/2015613042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0f406caa0 0x7fd0f406ef60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fd100004de0 tx=0x7fd10000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:11.422 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.427+0000 7fd1097fa700 1 -- 172.21.15.113:0/2015613042 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd0fc05ab80 con 0x7fd10c0f9210 2024-04-12T05:48:11.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:11 smithi192 ceph-mon[85304]: from='client.35934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:11.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:11 smithi192 ceph-mon[85304]: pgmap v1081: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:11.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:11 smithi192 ceph-mon[85304]: from='client.35938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:11.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:11 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2294745357' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:48:11.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.630+0000 7fd1129f9700 1 -- 172.21.15.113:0/2015613042 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd10c1097c0 con 0x7fd0f406caa0 2024-04-12T05:48:11.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.631+0000 7fd1097fa700 1 -- 172.21.15.113:0/2015613042 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd10c1097c0 con 0x7fd0f406caa0 2024-04-12T05:48:11.627 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:48:11.627 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:48:11.628 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:48:11.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.633+0000 7fd1129f9700 1 -- 172.21.15.113:0/2015613042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0f406caa0 msgr2=0x7fd0f406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:11.629 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.633+0000 7fd1129f9700 1 --2- 172.21.15.113:0/2015613042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0f406caa0 0x7fd0f406ef60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fd100004de0 tx=0x7fd10000b040).stop 2024-04-12T05:48:11.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.633+0000 7fd1129f9700 1 -- 172.21.15.113:0/2015613042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0f9210 msgr2=0x7fd10c10e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:11.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.633+0000 7fd1129f9700 1 --2- 172.21.15.113:0/2015613042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0f9210 0x7fd10c10e520 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7fd0fc005010 tx=0x7fd0fc009cb0).stop 2024-04-12T05:48:11.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.634+0000 7fd1129f9700 1 -- 172.21.15.113:0/2015613042 shutdown_connections 2024-04-12T05:48:11.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.634+0000 7fd1129f9700 1 --2- 172.21.15.113:0/2015613042 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd0f406caa0 0x7fd0f406ef60 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.634+0000 7fd1129f9700 1 --2- 172.21.15.113:0/2015613042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd10c0f8940 0x7fd10c10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.634+0000 7fd1129f9700 1 --2- 172.21.15.113:0/2015613042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd10c0f9210 0x7fd10c10e520 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.634+0000 7fd1129f9700 1 -- 172.21.15.113:0/2015613042 >> 172.21.15.113:0/2015613042 conn(0x7fd10c070590 msgr2=0x7fd10c0fc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:11.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.634+0000 7fd1129f9700 1 -- 172.21.15.113:0/2015613042 shutdown_connections 2024-04-12T05:48:11.630 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.634+0000 7fd1129f9700 1 -- 172.21.15.113:0/2015613042 wait complete. 2024-04-12T05:48:11.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.731+0000 7f045862b700 1 -- 172.21.15.113:0/1630257388 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04500fd8f0 msgr2=0x7f04500fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:11.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.731+0000 7f045862b700 1 --2- 172.21.15.113:0/1630257388 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04500fd8f0 0x7f04500fdcd0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f04400047f0 tx=0x7f0440009cb0).stop 2024-04-12T05:48:11.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.731+0000 7f045862b700 1 -- 172.21.15.113:0/1630257388 shutdown_connections 2024-04-12T05:48:11.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.731+0000 7f045862b700 1 --2- 172.21.15.113:0/1630257388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04500fe250 0x7f04500f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.727 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.731+0000 7f045862b700 1 --2- 172.21.15.113:0/1630257388 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04500fd8f0 0x7f04500fdcd0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.731+0000 7f045862b700 1 -- 172.21.15.113:0/1630257388 >> 172.21.15.113:0/1630257388 conn(0x7f04500f34b0 msgr2=0x7f04500f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:11.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.731+0000 7f045862b700 1 -- 172.21.15.113:0/1630257388 shutdown_connections 2024-04-12T05:48:11.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.731+0000 7f045862b700 1 -- 172.21.15.113:0/1630257388 wait complete. 2024-04-12T05:48:11.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.731+0000 7f045862b700 1 Processor -- start 2024-04-12T05:48:11.728 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f045862b700 1 -- start start 2024-04-12T05:48:11.729 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f045862b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04500fd8f0 0x7f045010e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f045862b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04500fe250 0x7f045010e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f045862b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045010eaf0 con 0x7f04500fe250 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f045862b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045010ec30 con 0x7f04500fd8f0 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f0455bc6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04500fe250 0x7f045010e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f0455bc6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04500fe250 0x7f045010e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41618/0 (socket says 172.21.15.113:41618) 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f0455bc6700 1 -- 172.21.15.113:0/1739471206 learned_addr learned my addr 172.21.15.113:0/1739471206 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f04563c7700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04500fd8f0 0x7f045010e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f0455bc6700 1 -- 172.21.15.113:0/1739471206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04500fd8f0 msgr2=0x7f045010e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f0455bc6700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04500fd8f0 0x7f045010e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f0455bc6700 1 -- 172.21.15.113:0/1739471206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0440005040 con 0x7f04500fe250 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f04563c7700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04500fd8f0 0x7f045010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f0455bc6700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04500fe250 0x7f045010e4f0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f044c002820 tx=0x7f044c009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f04477fe700 1 -- 172.21.15.113:0/1739471206 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f044c007b40 con 0x7f04500fe250 2024-04-12T05:48:11.730 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f04477fe700 1 -- 172.21.15.113:0/1739471206 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f044c00f040 con 0x7f04500fe250 2024-04-12T05:48:11.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f045862b700 1 -- 172.21.15.113:0/1739471206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04501082c0 con 0x7f04500fe250 2024-04-12T05:48:11.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.732+0000 7f04477fe700 1 -- 172.21.15.113:0/1739471206 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f044c014910 con 0x7f04500fe250 2024-04-12T05:48:11.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.733+0000 7f045862b700 1 -- 172.21.15.113:0/1739471206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0450108880 con 0x7f04500fe250 2024-04-12T05:48:11.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.733+0000 7f045862b700 1 -- 172.21.15.113:0/1739471206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045005f590 con 0x7f04500fe250 2024-04-12T05:48:11.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.734+0000 7f04477fe700 1 -- 172.21.15.113:0/1739471206 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f044c007ca0 con 0x7f04500fe250 2024-04-12T05:48:11.731 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.734+0000 7f04477fe700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f043c06caa0 0x7f043c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:11.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.734+0000 7f04477fe700 1 -- 172.21.15.113:0/1739471206 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f044c08b750 con 0x7f04500fe250 2024-04-12T05:48:11.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.734+0000 7f04563c7700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f043c06caa0 0x7f043c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:11.732 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.735+0000 7f04563c7700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f043c06caa0 0x7f043c06ef60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f0440004540 tx=0x7f044000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:11.734 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:11.739+0000 7f04477fe700 1 -- 172.21.15.113:0/1739471206 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f044c059200 con 0x7f04500fe250 2024-04-12T05:48:12.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.003+0000 7f045862b700 1 -- 172.21.15.113:0/1739471206 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0450047540 con 0x7f04500fe250 2024-04-12T05:48:12.000 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.004+0000 7f04477fe700 1 -- 172.21.15.113:0/1739471206 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f044c058d90 con 0x7f04500fe250 2024-04-12T05:48:12.000 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:48:12.000 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:48:12.000 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:48:12.000 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:48:12.000 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:48:12.000 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:48:12.000 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:48:12.000 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:48:12.001 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:48:12.002 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:48:12.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.006+0000 7f045862b700 1 -- 172.21.15.113:0/1739471206 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f043c06caa0 msgr2=0x7f043c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:12.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.006+0000 7f045862b700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f043c06caa0 0x7f043c06ef60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f0440004540 tx=0x7f044000b040).stop 2024-04-12T05:48:12.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.006+0000 7f045862b700 1 -- 172.21.15.113:0/1739471206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04500fe250 msgr2=0x7f045010e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:12.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.006+0000 7f045862b700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04500fe250 0x7f045010e4f0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f044c002820 tx=0x7f044c009f20).stop 2024-04-12T05:48:12.002 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.006+0000 7f045862b700 1 -- 172.21.15.113:0/1739471206 shutdown_connections 2024-04-12T05:48:12.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.006+0000 7f045862b700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f043c06caa0 0x7f043c06ef60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:12.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.006+0000 7f045862b700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04500fd8f0 0x7f045010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:12.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.006+0000 7f045862b700 1 --2- 172.21.15.113:0/1739471206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f04500fe250 0x7f045010e4f0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:12.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.006+0000 7f045862b700 1 -- 172.21.15.113:0/1739471206 >> 172.21.15.113:0/1739471206 conn(0x7f04500f34b0 msgr2=0x7f04500fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:12.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.006+0000 7f045862b700 1 -- 172.21.15.113:0/1739471206 shutdown_connections 2024-04-12T05:48:12.003 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:12.006+0000 7f045862b700 1 -- 172.21.15.113:0/1739471206 wait complete. 2024-04-12T05:48:12.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:12 smithi113 ceph-mon[103371]: from='client.35942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:12.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:12 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/235197878' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:48:12.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:12 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1739471206' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:48:12.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:12 smithi192 ceph-mon[85304]: from='client.35942 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:12.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:12 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/235197878' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:48:12.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:12 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1739471206' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:48:13.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:13 smithi113 ceph-mon[103371]: from='client.35954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:13.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:13 smithi113 ceph-mon[103371]: pgmap v1082: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:13.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:13 smithi192 ceph-mon[85304]: from='client.35954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:13.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:13 smithi192 ceph-mon[85304]: pgmap v1082: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:15.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:15 smithi113 ceph-mon[103371]: pgmap v1083: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:15.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:15 smithi192 ceph-mon[85304]: pgmap v1083: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:17.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:17 smithi113 ceph-mon[103371]: pgmap v1084: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:17.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:17 smithi192 ceph-mon[85304]: pgmap v1084: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:19.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:19 smithi113 ceph-mon[103371]: pgmap v1085: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:19.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:19 smithi192 ceph-mon[85304]: pgmap v1085: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:21.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:21 smithi113 ceph-mon[103371]: pgmap v1086: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:21.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:21 smithi192 ceph-mon[85304]: pgmap v1086: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:22.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:22.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:23.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:23 smithi113 ceph-mon[103371]: pgmap v1087: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:23.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:23 smithi192 ceph-mon[85304]: pgmap v1087: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:25.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:25 smithi113 ceph-mon[103371]: pgmap v1088: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:25.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:25 smithi192 ceph-mon[85304]: pgmap v1088: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:27.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:27 smithi113 ceph-mon[103371]: pgmap v1089: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:27.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:27 smithi192 ceph-mon[85304]: pgmap v1089: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:29.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:29 smithi113 ceph-mon[103371]: pgmap v1090: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:29.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:29 smithi192 ceph-mon[85304]: pgmap v1090: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:31.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:31 smithi113 ceph-mon[103371]: pgmap v1091: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:31.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:31 smithi192 ceph-mon[85304]: pgmap v1091: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:33.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:33 smithi113 ceph-mon[103371]: pgmap v1092: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:33.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:33 smithi192 ceph-mon[85304]: pgmap v1092: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:35.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:35 smithi113 ceph-mon[103371]: pgmap v1093: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:35.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:35 smithi192 ceph-mon[85304]: pgmap v1093: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:37.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:37 smithi113 ceph-mon[103371]: pgmap v1094: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:37.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:37.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:37 smithi192 ceph-mon[85304]: pgmap v1094: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:37.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:39.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:39 smithi113 ceph-mon[103371]: pgmap v1095: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:39.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:39 smithi192 ceph-mon[85304]: pgmap v1095: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:41.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:41 smithi113 ceph-mon[103371]: pgmap v1096: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:41.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:41 smithi192 ceph-mon[85304]: pgmap v1096: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:42.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.117+0000 7f279401b700 1 -- 172.21.15.113:0/531983320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f278c0fd150 msgr2=0x7f278c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.113 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.117+0000 7f279401b700 1 --2- 172.21.15.113:0/531983320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f278c0fd150 0x7f278c0f6930 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f277c0047f0 tx=0x7f277c009cb0).stop 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.117+0000 7f279401b700 1 -- 172.21.15.113:0/531983320 shutdown_connections 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.117+0000 7f279401b700 1 --2- 172.21.15.113:0/531983320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f278c0fd150 0x7f278c0f6930 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.117+0000 7f279401b700 1 --2- 172.21.15.113:0/531983320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c0fc7f0 0x7f278c0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.117+0000 7f279401b700 1 -- 172.21.15.113:0/531983320 >> 172.21.15.113:0/531983320 conn(0x7f278c070590 msgr2=0x7f278c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.117+0000 7f279401b700 1 -- 172.21.15.113:0/531983320 shutdown_connections 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.117+0000 7f279401b700 1 -- 172.21.15.113:0/531983320 wait complete. 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.117+0000 7f279401b700 1 Processor -- start 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.117+0000 7f279401b700 1 -- start start 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f279401b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c0fc7f0 0x7f278c10e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f279401b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f278c0fd150 0x7f278c10e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f279401b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c10ebc0 con 0x7f278c0fd150 2024-04-12T05:48:42.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f279401b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c10ed00 con 0x7f278c0fc7f0 2024-04-12T05:48:42.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f27915b6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f278c0fd150 0x7f278c10e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:42.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f27915b6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f278c0fd150 0x7f278c10e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33488/0 (socket says 172.21.15.113:33488) 2024-04-12T05:48:42.115 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f27915b6700 1 -- 172.21.15.113:0/2336707626 learned_addr learned my addr 172.21.15.113:0/2336707626 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:42.116 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f2791db7700 1 --2- 172.21.15.113:0/2336707626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c0fc7f0 0x7f278c10e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f27915b6700 1 -- 172.21.15.113:0/2336707626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c0fc7f0 msgr2=0x7f278c10e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f27915b6700 1 --2- 172.21.15.113:0/2336707626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c0fc7f0 0x7f278c10e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f27915b6700 1 -- 172.21.15.113:0/2336707626 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277c005040 con 0x7f278c0fd150 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f27915b6700 1 --2- 172.21.15.113:0/2336707626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f278c0fd150 0x7f278c10e5c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f277c000c00 tx=0x7f277c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f2782ffd700 1 -- 172.21.15.113:0/2336707626 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f277c018070 con 0x7f278c0fd150 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f2782ffd700 1 -- 172.21.15.113:0/2336707626 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f277c007e90 con 0x7f278c0fd150 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f279401b700 1 -- 172.21.15.113:0/2336707626 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c108330 con 0x7f278c0fd150 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f2782ffd700 1 -- 172.21.15.113:0/2336707626 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f277c0123c0 con 0x7f278c0fd150 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.118+0000 7f279401b700 1 -- 172.21.15.113:0/2336707626 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c108890 con 0x7f278c0fd150 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.119+0000 7f279401b700 1 -- 172.21.15.113:0/2336707626 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c05f590 con 0x7f278c0fd150 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.120+0000 7f2782ffd700 1 -- 172.21.15.113:0/2336707626 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f277c01d420 con 0x7f278c0fd150 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.120+0000 7f2782ffd700 1 --2- 172.21.15.113:0/2336707626 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f277806caa0 0x7f277806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.121+0000 7f2782ffd700 1 -- 172.21.15.113:0/2336707626 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f277c08ca10 con 0x7f278c0fd150 2024-04-12T05:48:42.117 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.121+0000 7f2791db7700 1 --2- 172.21.15.113:0/2336707626 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f277806caa0 0x7f277806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:42.118 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.121+0000 7f2791db7700 1 --2- 172.21.15.113:0/2336707626 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f277806caa0 0x7f277806ef60 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f27880092f0 tx=0x7f278800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:42.121 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.125+0000 7f2782ffd700 1 -- 172.21.15.113:0/2336707626 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f277c05a4c0 con 0x7f278c0fd150 2024-04-12T05:48:42.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.332+0000 7f279401b700 1 -- 172.21.15.113:0/2336707626 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f278c108f20 con 0x7f277806caa0 2024-04-12T05:48:42.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.333+0000 7f2782ffd700 1 -- 172.21.15.113:0/2336707626 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f278c108f20 con 0x7f277806caa0 2024-04-12T05:48:42.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.335+0000 7f279401b700 1 -- 172.21.15.113:0/2336707626 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f277806caa0 msgr2=0x7f277806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.335+0000 7f279401b700 1 --2- 172.21.15.113:0/2336707626 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f277806caa0 0x7f277806ef60 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f27880092f0 tx=0x7f278800b040).stop 2024-04-12T05:48:42.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.335+0000 7f279401b700 1 -- 172.21.15.113:0/2336707626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f278c0fd150 msgr2=0x7f278c10e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.331 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.335+0000 7f279401b700 1 --2- 172.21.15.113:0/2336707626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f278c0fd150 0x7f278c10e5c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f277c000c00 tx=0x7f277c009cb0).stop 2024-04-12T05:48:42.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.335+0000 7f279401b700 1 -- 172.21.15.113:0/2336707626 shutdown_connections 2024-04-12T05:48:42.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.335+0000 7f279401b700 1 --2- 172.21.15.113:0/2336707626 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f277806caa0 0x7f277806ef60 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.335+0000 7f279401b700 1 --2- 172.21.15.113:0/2336707626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c0fc7f0 0x7f278c10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.335+0000 7f279401b700 1 --2- 172.21.15.113:0/2336707626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f278c0fd150 0x7f278c10e5c0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.335+0000 7f279401b700 1 -- 172.21.15.113:0/2336707626 >> 172.21.15.113:0/2336707626 conn(0x7f278c070590 msgr2=0x7f278c060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:42.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.336+0000 7f279401b700 1 -- 172.21.15.113:0/2336707626 shutdown_connections 2024-04-12T05:48:42.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.336+0000 7f279401b700 1 -- 172.21.15.113:0/2336707626 wait complete. 2024-04-12T05:48:42.341 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:48:42.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.434+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/3361087609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cb00fd8f0 msgr2=0x7f4cb00fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.434+0000 7f4cb7ce6700 1 --2- 172.21.15.113:0/3361087609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cb00fd8f0 0x7f4cb00fdcd0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7f4ca00047f0 tx=0x7f4ca0009cb0).stop 2024-04-12T05:48:42.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.434+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/3361087609 shutdown_connections 2024-04-12T05:48:42.431 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.434+0000 7f4cb7ce6700 1 --2- 172.21.15.113:0/3361087609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb00fe250 0x7f4cb00f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.434+0000 7f4cb7ce6700 1 --2- 172.21.15.113:0/3361087609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cb00fd8f0 0x7f4cb00fdcd0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.434+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/3361087609 >> 172.21.15.113:0/3361087609 conn(0x7f4cb00f34b0 msgr2=0x7f4cb00f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.434+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/3361087609 shutdown_connections 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.434+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/3361087609 wait complete. 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.434+0000 7f4cb7ce6700 1 Processor -- start 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.434+0000 7f4cb7ce6700 1 -- start start 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb7ce6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cb00fd8f0 0x7f4cb010e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb7ce6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb00fe250 0x7f4cb010e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb7ce6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb010eaf0 con 0x7f4cb00fd8f0 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb7ce6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb010ec30 con 0x7f4cb00fe250 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb5a82700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cb00fd8f0 0x7f4cb010e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb5a82700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cb00fd8f0 0x7f4cb010e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33518/0 (socket says 172.21.15.113:33518) 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb5a82700 1 -- 172.21.15.113:0/1410714216 learned_addr learned my addr 172.21.15.113:0/1410714216 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb5281700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb00fe250 0x7f4cb010e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:42.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb5a82700 1 -- 172.21.15.113:0/1410714216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb00fe250 msgr2=0x7f4cb010e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb5a82700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb00fe250 0x7f4cb010e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb5a82700 1 -- 172.21.15.113:0/1410714216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca0005040 con 0x7f4cb00fd8f0 2024-04-12T05:48:42.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb5281700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb00fe250 0x7f4cb010e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:48:42.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb5a82700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cb00fd8f0 0x7f4cb010e000 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f4ca000b120 tx=0x7f4ca0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:42.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4ca6ffd700 1 -- 172.21.15.113:0/1410714216 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca000e030 con 0x7f4cb00fd8f0 2024-04-12T05:48:42.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4ca6ffd700 1 -- 172.21.15.113:0/1410714216 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ca0007e90 con 0x7f4cb00fd8f0 2024-04-12T05:48:42.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/1410714216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb0108260 con 0x7f4cb00fd8f0 2024-04-12T05:48:42.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4ca6ffd700 1 -- 172.21.15.113:0/1410714216 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca0018070 con 0x7f4cb00fd8f0 2024-04-12T05:48:42.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.435+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/1410714216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb01087c0 con 0x7f4cb00fd8f0 2024-04-12T05:48:42.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.436+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/1410714216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb005f590 con 0x7f4cb00fd8f0 2024-04-12T05:48:42.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.437+0000 7f4ca6ffd700 1 -- 172.21.15.113:0/1410714216 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4ca001d420 con 0x7f4cb00fd8f0 2024-04-12T05:48:42.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.437+0000 7f4ca6ffd700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c9c06caa0 0x7f4c9c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:42.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.437+0000 7f4ca6ffd700 1 -- 172.21.15.113:0/1410714216 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f4ca008d320 con 0x7f4cb00fd8f0 2024-04-12T05:48:42.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.437+0000 7f4cb5281700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c9c06caa0 0x7f4c9c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:42.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.437+0000 7f4cb5281700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c9c06caa0 0x7f4c9c06ef60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f4cb0109940 tx=0x7f4cac009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:42.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.442+0000 7f4ca6ffd700 1 -- 172.21.15.113:0/1410714216 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f4ca005add0 con 0x7f4cb00fd8f0 2024-04-12T05:48:42.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.648+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/1410714216 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cb0109720 con 0x7f4c9c06caa0 2024-04-12T05:48:42.645 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.649+0000 7f4ca6ffd700 1 -- 172.21.15.113:0/1410714216 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4cb0109720 con 0x7f4c9c06caa0 2024-04-12T05:48:42.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.651+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/1410714216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c9c06caa0 msgr2=0x7f4c9c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.651+0000 7f4cb7ce6700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c9c06caa0 0x7f4c9c06ef60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f4cb0109940 tx=0x7f4cac009040).stop 2024-04-12T05:48:42.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.651+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/1410714216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cb00fd8f0 msgr2=0x7f4cb010e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.651+0000 7f4cb7ce6700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cb00fd8f0 0x7f4cb010e000 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f4ca000b120 tx=0x7f4ca0009cb0).stop 2024-04-12T05:48:42.647 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.651+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/1410714216 shutdown_connections 2024-04-12T05:48:42.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.651+0000 7f4cb7ce6700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f4c9c06caa0 0x7f4c9c06ef60 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.651+0000 7f4cb7ce6700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cb00fd8f0 0x7f4cb010e000 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.651+0000 7f4cb7ce6700 1 --2- 172.21.15.113:0/1410714216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4cb00fe250 0x7f4cb010e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.651+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/1410714216 >> 172.21.15.113:0/1410714216 conn(0x7f4cb00f34b0 msgr2=0x7f4cb00fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:42.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.651+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/1410714216 shutdown_connections 2024-04-12T05:48:42.648 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.651+0000 7f4cb7ce6700 1 -- 172.21.15.113:0/1410714216 wait complete. 2024-04-12T05:48:42.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.749+0000 7f25033fc700 1 -- 172.21.15.113:0/2104307774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24fc0fd150 msgr2=0x7f24fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.749+0000 7f25033fc700 1 --2- 172.21.15.113:0/2104307774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24fc0fd150 0x7f24fc0f6930 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f24ec0047f0 tx=0x7f24ec009cb0).stop 2024-04-12T05:48:42.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.749+0000 7f25033fc700 1 -- 172.21.15.113:0/2104307774 shutdown_connections 2024-04-12T05:48:42.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.749+0000 7f25033fc700 1 --2- 172.21.15.113:0/2104307774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24fc0fd150 0x7f24fc0f6930 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.749+0000 7f25033fc700 1 --2- 172.21.15.113:0/2104307774 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24fc0fc7f0 0x7f24fc0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.749+0000 7f25033fc700 1 -- 172.21.15.113:0/2104307774 >> 172.21.15.113:0/2104307774 conn(0x7f24fc070590 msgr2=0x7f24fc0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:42.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.749+0000 7f25033fc700 1 -- 172.21.15.113:0/2104307774 shutdown_connections 2024-04-12T05:48:42.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.749+0000 7f25033fc700 1 -- 172.21.15.113:0/2104307774 wait complete. 2024-04-12T05:48:42.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.750+0000 7f25033fc700 1 Processor -- start 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.750+0000 7f25033fc700 1 -- start start 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.750+0000 7f25033fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24fc0fc7f0 0x7f24fc109080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.750+0000 7f25033fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24fc0fd150 0x7f24fc109570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.750+0000 7f25033fc700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc109ab0 con 0x7f24fc0fd150 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.750+0000 7f25033fc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc109bf0 con 0x7f24fc0fc7f0 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.750+0000 7f2500997700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24fc0fd150 0x7f24fc109570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.750+0000 7f2500997700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24fc0fd150 0x7f24fc109570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33544/0 (socket says 172.21.15.113:33544) 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.750+0000 7f2500997700 1 -- 172.21.15.113:0/2227265311 learned_addr learned my addr 172.21.15.113:0/2227265311 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.751+0000 7f2501198700 1 --2- 172.21.15.113:0/2227265311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24fc0fc7f0 0x7f24fc109080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.751+0000 7f2500997700 1 -- 172.21.15.113:0/2227265311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24fc0fc7f0 msgr2=0x7f24fc109080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.751+0000 7f2500997700 1 --2- 172.21.15.113:0/2227265311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24fc0fc7f0 0x7f24fc109080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.751+0000 7f2500997700 1 -- 172.21.15.113:0/2227265311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec005040 con 0x7f24fc0fd150 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.751+0000 7f2500997700 1 --2- 172.21.15.113:0/2227265311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24fc0fd150 0x7f24fc109570 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f24ec0099f0 tx=0x7f24ec009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.751+0000 7f24f27fc700 1 -- 172.21.15.113:0/2227265311 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24ec00e030 con 0x7f24fc0fd150 2024-04-12T05:48:42.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.751+0000 7f25033fc700 1 -- 172.21.15.113:0/2227265311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24fc109e40 con 0x7f24fc0fd150 2024-04-12T05:48:42.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.751+0000 7f24f27fc700 1 -- 172.21.15.113:0/2227265311 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24ec007e90 con 0x7f24fc0fd150 2024-04-12T05:48:42.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.751+0000 7f24f27fc700 1 -- 172.21.15.113:0/2227265311 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24ec018070 con 0x7f24fc0fd150 2024-04-12T05:48:42.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.751+0000 7f25033fc700 1 -- 172.21.15.113:0/2227265311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24fc1123e0 con 0x7f24fc0fd150 2024-04-12T05:48:42.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.752+0000 7f25033fc700 1 -- 172.21.15.113:0/2227265311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24fc05f590 con 0x7f24fc0fd150 2024-04-12T05:48:42.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.752+0000 7f24f27fc700 1 -- 172.21.15.113:0/2227265311 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f24ec009590 con 0x7f24fc0fd150 2024-04-12T05:48:42.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.753+0000 7f24f27fc700 1 --2- 172.21.15.113:0/2227265311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f24e806ca30 0x7f24e806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:42.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.753+0000 7f2501198700 1 --2- 172.21.15.113:0/2227265311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f24e806ca30 0x7f24e806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:42.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.753+0000 7f24f27fc700 1 -- 172.21.15.113:0/2227265311 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f24ec08d0f0 con 0x7f24fc0fd150 2024-04-12T05:48:42.750 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.753+0000 7f2501198700 1 --2- 172.21.15.113:0/2227265311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f24e806ca30 0x7f24e806eef0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f24f8009170 tx=0x7f24f8009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:42.753 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.757+0000 7f24f27fc700 1 -- 172.21.15.113:0/2227265311 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f24ec057110 con 0x7f24fc0fd150 2024-04-12T05:48:42.964 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.968+0000 7f25033fc700 1 -- 172.21.15.113:0/2227265311 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f24fc059120 con 0x7f24e806ca30 2024-04-12T05:48:42.974 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.978+0000 7f24f27fc700 1 -- 172.21.15.113:0/2227265311 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f24fc059120 con 0x7f24e806ca30 2024-04-12T05:48:42.974 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:48:42.974 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (36m) 14m ago 47m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:48:42.974 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (33m) 14m ago 47m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:48:42.974 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (33m) 14m ago 47m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:48:42.974 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (36m) 14m ago 47m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:48:42.974 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (29m) 14m ago 42m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (29m) 14m ago 42m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (29m) 14m ago 42m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (14m) 14m ago 42m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (37m) 14m ago 49m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (37m) 14m ago 47m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (35m) 14m ago 49m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (34m) 14m ago 47m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (37m) 14m ago 47m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (37m) 14m ago 46m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (33m) 14m ago 46m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (32m) 14m ago 45m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:48:42.975 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (32m) 14m ago 45m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:48:42.976 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (31m) 14m ago 44m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:48:42.976 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (31m) 14m ago 44m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:48:42.976 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (30m) 14m ago 43m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:48:42.976 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (36m) 14m ago 46m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:48:42.976 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.980+0000 7f25033fc700 1 -- 172.21.15.113:0/2227265311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f24e806ca30 msgr2=0x7f24e806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.980+0000 7f25033fc700 1 --2- 172.21.15.113:0/2227265311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f24e806ca30 0x7f24e806eef0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f24f8009170 tx=0x7f24f8009da0).stop 2024-04-12T05:48:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.980+0000 7f25033fc700 1 -- 172.21.15.113:0/2227265311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24fc0fd150 msgr2=0x7f24fc109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.980+0000 7f25033fc700 1 --2- 172.21.15.113:0/2227265311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24fc0fd150 0x7f24fc109570 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f24ec0099f0 tx=0x7f24ec009cb0).stop 2024-04-12T05:48:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.980+0000 7f25033fc700 1 -- 172.21.15.113:0/2227265311 shutdown_connections 2024-04-12T05:48:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.980+0000 7f25033fc700 1 --2- 172.21.15.113:0/2227265311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f24e806ca30 0x7f24e806eef0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.980+0000 7f25033fc700 1 --2- 172.21.15.113:0/2227265311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24fc0fc7f0 0x7f24fc109080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.980+0000 7f25033fc700 1 --2- 172.21.15.113:0/2227265311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24fc0fd150 0x7f24fc109570 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.980+0000 7f25033fc700 1 -- 172.21.15.113:0/2227265311 >> 172.21.15.113:0/2227265311 conn(0x7f24fc070590 msgr2=0x7f24fc0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.980+0000 7f25033fc700 1 -- 172.21.15.113:0/2227265311 shutdown_connections 2024-04-12T05:48:42.977 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:42.981+0000 7f25033fc700 1 -- 172.21.15.113:0/2227265311 wait complete. 2024-04-12T05:48:43.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.078+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/3939742950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1a40fd8f0 msgr2=0x7fa1a40fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:43.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.078+0000 7fa1a8b6f700 1 --2- 172.21.15.113:0/3939742950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1a40fd8f0 0x7fa1a40fdcd0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7fa1980047f0 tx=0x7fa198009cb0).stop 2024-04-12T05:48:43.075 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.078+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/3939742950 shutdown_connections 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.078+0000 7fa1a8b6f700 1 --2- 172.21.15.113:0/3939742950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a40fe250 0x7fa1a40f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.078+0000 7fa1a8b6f700 1 --2- 172.21.15.113:0/3939742950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1a40fd8f0 0x7fa1a40fdcd0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.078+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/3939742950 >> 172.21.15.113:0/3939742950 conn(0x7fa1a40f34b0 msgr2=0x7fa1a40f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.078+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/3939742950 shutdown_connections 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.078+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/3939742950 wait complete. 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a8b6f700 1 Processor -- start 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a8b6f700 1 -- start start 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a8b6f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1a40fd8f0 0x7fa1a41101f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a8b6f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a40fe250 0x7fa1a41106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a8b6f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a4110ce0 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a8b6f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a4110e20 con 0x7fa1a40fe250 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1a40fd8f0 0x7fa1a41101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1a40fd8f0 0x7fa1a41101f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33572/0 (socket says 172.21.15.113:33572) 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a259c700 1 -- 172.21.15.113:0/1557205518 learned_addr learned my addr 172.21.15.113:0/1557205518 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:43.076 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a1d9b700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a40fe250 0x7fa1a41106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a259c700 1 -- 172.21.15.113:0/1557205518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a40fe250 msgr2=0x7fa1a41106e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a259c700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a40fe250 0x7fa1a41106e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a259c700 1 -- 172.21.15.113:0/1557205518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa198005040 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a1d9b700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a40fe250 0x7fa1a41106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:48:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.079+0000 7fa1a259c700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1a40fd8f0 0x7fa1a41101f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7fa19800b120 tx=0x7fa198009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.080+0000 7fa1937fe700 1 -- 172.21.15.113:0/1557205518 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa19800e030 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.080+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/1557205518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a410a4a0 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.080+0000 7fa1937fe700 1 -- 172.21.15.113:0/1557205518 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa198007e90 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.080+0000 7fa1937fe700 1 -- 172.21.15.113:0/1557205518 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa198018070 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.077 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.080+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/1557205518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a410aa00 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.081+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/1557205518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a405f590 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.081+0000 7fa1937fe700 1 -- 172.21.15.113:0/1557205518 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa19801d420 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.081+0000 7fa1937fe700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa18c06caa0 0x7fa18c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:43.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.081+0000 7fa1937fe700 1 -- 172.21.15.113:0/1557205518 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fa19808d300 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.082+0000 7fa1a1d9b700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa18c06caa0 0x7fa18c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:43.078 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.082+0000 7fa1a1d9b700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa18c06caa0 0x7fa18c06ef60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fa1a410bb30 tx=0x7fa194009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:43.081 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.086+0000 7fa1937fe700 1 -- 172.21.15.113:0/1557205518 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fa19805adb0 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.357+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/1557205518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa1a410b910 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.358+0000 7fa1937fe700 1 -- 172.21.15.113:0/1557205518 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fa19805a940 con 0x7fa1a40fd8f0 2024-04-12T05:48:43.354 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:48:43.354 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:48:43.354 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:48:43.354 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:48:43.354 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:48:43.354 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:48:43.354 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:48:43.354 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:48:43.354 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:48:43.354 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:48:43.355 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:48:43.355 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:48:43.355 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:48:43.355 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:48:43.355 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:48:43.355 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:48:43.355 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:48:43.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.360+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/1557205518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa18c06caa0 msgr2=0x7fa18c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:43.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.360+0000 7fa1a8b6f700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa18c06caa0 0x7fa18c06ef60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fa1a410bb30 tx=0x7fa194009040).stop 2024-04-12T05:48:43.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.360+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/1557205518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1a40fd8f0 msgr2=0x7fa1a41101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:43.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.360+0000 7fa1a8b6f700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1a40fd8f0 0x7fa1a41101f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7fa19800b120 tx=0x7fa198009cb0).stop 2024-04-12T05:48:43.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.360+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/1557205518 shutdown_connections 2024-04-12T05:48:43.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.360+0000 7fa1a8b6f700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fa18c06caa0 0x7fa18c06ef60 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.360+0000 7fa1a8b6f700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1a40fd8f0 0x7fa1a41101f0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.360+0000 7fa1a8b6f700 1 --2- 172.21.15.113:0/1557205518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1a40fe250 0x7fa1a41106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.360+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/1557205518 >> 172.21.15.113:0/1557205518 conn(0x7fa1a40f34b0 msgr2=0x7fa1a406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:43.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.360+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/1557205518 shutdown_connections 2024-04-12T05:48:43.356 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.360+0000 7fa1a8b6f700 1 -- 172.21.15.113:0/1557205518 wait complete. 2024-04-12T05:48:43.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:43 smithi113 ceph-mon[103371]: pgmap v1097: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:43.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:43 smithi113 ceph-mon[103371]: from='client.35962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:43.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:43 smithi192 ceph-mon[85304]: pgmap v1097: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:43.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:43 smithi192 ceph-mon[85304]: from='client.35962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:43.454 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.458+0000 7f07be6cd700 1 -- 172.21.15.113:0/1012565182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07b80fe250 msgr2=0x7f07b80f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:43.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.458+0000 7f07be6cd700 1 --2- 172.21.15.113:0/1012565182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07b80fe250 0x7f07b80f7a30 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f07ac0047f0 tx=0x7f07ac009cb0).stop 2024-04-12T05:48:43.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.458+0000 7f07be6cd700 1 -- 172.21.15.113:0/1012565182 shutdown_connections 2024-04-12T05:48:43.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.458+0000 7f07be6cd700 1 --2- 172.21.15.113:0/1012565182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07b80fe250 0x7f07b80f7a30 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.458+0000 7f07be6cd700 1 --2- 172.21.15.113:0/1012565182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07b80fd8f0 0x7f07b80fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.458+0000 7f07be6cd700 1 -- 172.21.15.113:0/1012565182 >> 172.21.15.113:0/1012565182 conn(0x7f07b80f34b0 msgr2=0x7f07b80f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:43.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.458+0000 7f07be6cd700 1 -- 172.21.15.113:0/1012565182 shutdown_connections 2024-04-12T05:48:43.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.458+0000 7f07be6cd700 1 -- 172.21.15.113:0/1012565182 wait complete. 2024-04-12T05:48:43.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.459+0000 7f07be6cd700 1 Processor -- start 2024-04-12T05:48:43.455 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.459+0000 7f07be6cd700 1 -- start start 2024-04-12T05:48:43.456 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.459+0000 7f07be6cd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07b80fd8f0 0x7f07b806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.459+0000 7f07be6cd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07b80fe250 0x7f07b806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.459+0000 7f07be6cd700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b806bab0 con 0x7f07b80fd8f0 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.459+0000 7f07be6cd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b806bbf0 con 0x7f07b80fe250 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.459+0000 7f07b7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07b80fd8f0 0x7f07b806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07b7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07b80fd8f0 0x7f07b806b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33596/0 (socket says 172.21.15.113:33596) 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07b7fff700 1 -- 172.21.15.113:0/690988282 learned_addr learned my addr 172.21.15.113:0/690988282 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07b77fe700 1 --2- 172.21.15.113:0/690988282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07b80fe250 0x7f07b806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07b7fff700 1 -- 172.21.15.113:0/690988282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07b80fe250 msgr2=0x7f07b806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07b7fff700 1 --2- 172.21.15.113:0/690988282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07b80fe250 0x7f07b806b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07b7fff700 1 -- 172.21.15.113:0/690988282 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac005040 con 0x7f07b80fd8f0 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07b7fff700 1 --2- 172.21.15.113:0/690988282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07b80fd8f0 0x7f07b806b0d0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f07a800c790 tx=0x7f07a800cb50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:43.457 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07b57fa700 1 -- 172.21.15.113:0/690988282 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07a80106f0 con 0x7f07b80fd8f0 2024-04-12T05:48:43.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07b57fa700 1 -- 172.21.15.113:0/690988282 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07a8010d00 con 0x7f07b80fd8f0 2024-04-12T05:48:43.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07b57fa700 1 -- 172.21.15.113:0/690988282 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07a8012e50 con 0x7f07b80fd8f0 2024-04-12T05:48:43.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07be6cd700 1 -- 172.21.15.113:0/690988282 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b806bea0 con 0x7f07b80fd8f0 2024-04-12T05:48:43.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.460+0000 7f07be6cd700 1 -- 172.21.15.113:0/690988282 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b81146d0 con 0x7f07b80fd8f0 2024-04-12T05:48:43.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.461+0000 7f07be6cd700 1 -- 172.21.15.113:0/690988282 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b805f590 con 0x7f07b80fd8f0 2024-04-12T05:48:43.458 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.461+0000 7f07b57fa700 1 -- 172.21.15.113:0/690988282 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f07a8010850 con 0x7f07b80fd8f0 2024-04-12T05:48:43.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.462+0000 7f07b57fa700 1 --2- 172.21.15.113:0/690988282 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07a006caa0 0x7f07a006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:43.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.462+0000 7f07b57fa700 1 -- 172.21.15.113:0/690988282 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f07a808b790 con 0x7f07b80fd8f0 2024-04-12T05:48:43.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.462+0000 7f07b77fe700 1 --2- 172.21.15.113:0/690988282 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07a006caa0 0x7f07a006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:43.459 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.462+0000 7f07b77fe700 1 --2- 172.21.15.113:0/690988282 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07a006caa0 0x7f07a006ef60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f07ac004f10 tx=0x7f07ac00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:43.463 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.467+0000 7f07b57fa700 1 -- 172.21.15.113:0/690988282 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f07a80557b0 con 0x7f07b80fd8f0 2024-04-12T05:48:43.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.695+0000 7f07be6cd700 1 -- 172.21.15.113:0/690988282 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f07b8047540 con 0x7f07b80fd8f0 2024-04-12T05:48:43.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.696+0000 7f07b57fa700 1 -- 172.21.15.113:0/690988282 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7f07a8058dd0 con 0x7f07b80fd8f0 2024-04-12T05:48:43.692 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:48:43.692 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:48:43.692 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:48:43.692 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:48:43.692 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:48:43.692 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:48:43.692 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:48:43.693 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:48:43.694 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:48:43.695 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:48:43.695 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:48:43.695 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:48:43.695 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:43.695 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:43.695 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:43.695 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:48:43.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.698+0000 7f07be6cd700 1 -- 172.21.15.113:0/690988282 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07a006caa0 msgr2=0x7f07a006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:43.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.698+0000 7f07be6cd700 1 --2- 172.21.15.113:0/690988282 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07a006caa0 0x7f07a006ef60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f07ac004f10 tx=0x7f07ac00b040).stop 2024-04-12T05:48:43.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.698+0000 7f07be6cd700 1 -- 172.21.15.113:0/690988282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07b80fd8f0 msgr2=0x7f07b806b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:43.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.698+0000 7f07be6cd700 1 --2- 172.21.15.113:0/690988282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07b80fd8f0 0x7f07b806b0d0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f07a800c790 tx=0x7f07a800cb50).stop 2024-04-12T05:48:43.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.698+0000 7f07be6cd700 1 -- 172.21.15.113:0/690988282 shutdown_connections 2024-04-12T05:48:43.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.698+0000 7f07be6cd700 1 --2- 172.21.15.113:0/690988282 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f07a006caa0 0x7f07a006ef60 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.698+0000 7f07be6cd700 1 --2- 172.21.15.113:0/690988282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f07b80fd8f0 0x7f07b806b0d0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.698+0000 7f07be6cd700 1 --2- 172.21.15.113:0/690988282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07b80fe250 0x7f07b806b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.698+0000 7f07be6cd700 1 -- 172.21.15.113:0/690988282 >> 172.21.15.113:0/690988282 conn(0x7f07b80f34b0 msgr2=0x7f07b80fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:43.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.698+0000 7f07be6cd700 1 -- 172.21.15.113:0/690988282 shutdown_connections 2024-04-12T05:48:43.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.698+0000 7f07be6cd700 1 -- 172.21.15.113:0/690988282 wait complete. 2024-04-12T05:48:43.697 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:48:43.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.796+0000 7f9e21029700 1 -- 172.21.15.113:0/3649992324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1c0fc7f0 msgr2=0x7f9e1c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:43.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.796+0000 7f9e21029700 1 --2- 172.21.15.113:0/3649992324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1c0fc7f0 0x7f9e1c0fcbd0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f9e100047f0 tx=0x7f9e10009cb0).stop 2024-04-12T05:48:43.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.796+0000 7f9e21029700 1 -- 172.21.15.113:0/3649992324 shutdown_connections 2024-04-12T05:48:43.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.796+0000 7f9e21029700 1 --2- 172.21.15.113:0/3649992324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e1c0fd150 0x7f9e1c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.796+0000 7f9e21029700 1 --2- 172.21.15.113:0/3649992324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1c0fc7f0 0x7f9e1c0fcbd0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.796+0000 7f9e21029700 1 -- 172.21.15.113:0/3649992324 >> 172.21.15.113:0/3649992324 conn(0x7f9e1c070590 msgr2=0x7f9e1c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:43.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.796+0000 7f9e21029700 1 -- 172.21.15.113:0/3649992324 shutdown_connections 2024-04-12T05:48:43.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.796+0000 7f9e21029700 1 -- 172.21.15.113:0/3649992324 wait complete. 2024-04-12T05:48:43.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e21029700 1 Processor -- start 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e21029700 1 -- start start 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e21029700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1c0fc7f0 0x7f9e1c109100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e21029700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e1c0fd150 0x7f9e1c1095f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e21029700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c109ae0 con 0x7f9e1c0fc7f0 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e21029700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c109c20 con 0x7f9e1c0fd150 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e1ad9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1c0fc7f0 0x7f9e1c109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e1ad9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1c0fc7f0 0x7f9e1c109100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33612/0 (socket says 172.21.15.113:33612) 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e1ad9d700 1 -- 172.21.15.113:0/3546221961 learned_addr learned my addr 172.21.15.113:0/3546221961 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e1a59c700 1 --2- 172.21.15.113:0/3546221961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e1c0fd150 0x7f9e1c1095f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e1ad9d700 1 -- 172.21.15.113:0/3546221961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e1c0fd150 msgr2=0x7f9e1c1095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e1ad9d700 1 --2- 172.21.15.113:0/3546221961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e1c0fd150 0x7f9e1c1095f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.797+0000 7f9e1ad9d700 1 -- 172.21.15.113:0/3546221961 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e10005040 con 0x7f9e1c0fc7f0 2024-04-12T05:48:43.795 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.798+0000 7f9e1ad9d700 1 --2- 172.21.15.113:0/3546221961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1c0fc7f0 0x7f9e1c109100 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f9e10005010 tx=0x7f9e10009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:43.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.798+0000 7f9e03fff700 1 -- 172.21.15.113:0/3546221961 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e1000e030 con 0x7f9e1c0fc7f0 2024-04-12T05:48:43.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.798+0000 7f9e21029700 1 -- 172.21.15.113:0/3546221961 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c109e70 con 0x7f9e1c0fc7f0 2024-04-12T05:48:43.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.798+0000 7f9e03fff700 1 -- 172.21.15.113:0/3546221961 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e10007e90 con 0x7f9e1c0fc7f0 2024-04-12T05:48:43.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.798+0000 7f9e03fff700 1 -- 172.21.15.113:0/3546221961 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e10018070 con 0x7f9e1c0fc7f0 2024-04-12T05:48:43.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.798+0000 7f9e21029700 1 -- 172.21.15.113:0/3546221961 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c112460 con 0x7f9e1c0fc7f0 2024-04-12T05:48:43.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.799+0000 7f9e21029700 1 -- 172.21.15.113:0/3546221961 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1c05f590 con 0x7f9e1c0fc7f0 2024-04-12T05:48:43.796 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.799+0000 7f9e03fff700 1 -- 172.21.15.113:0/3546221961 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9e10009610 con 0x7f9e1c0fc7f0 2024-04-12T05:48:43.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.799+0000 7f9e03fff700 1 --2- 172.21.15.113:0/3546221961 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e0406ca30 0x7f9e0406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:43.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.800+0000 7f9e03fff700 1 -- 172.21.15.113:0/3546221961 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9e1008d0f0 con 0x7f9e1c0fc7f0 2024-04-12T05:48:43.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.800+0000 7f9e1a59c700 1 --2- 172.21.15.113:0/3546221961 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e0406ca30 0x7f9e0406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:43.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.800+0000 7f9e1a59c700 1 --2- 172.21.15.113:0/3546221961 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e0406ca30 0x7f9e0406eef0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f9e1c060f00 tx=0x7f9e0c009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:43.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:43.804+0000 7f9e03fff700 1 -- 172.21.15.113:0/3546221961 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9e10057110 con 0x7f9e1c0fc7f0 2024-04-12T05:48:44.005 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.009+0000 7f9e21029700 1 -- 172.21.15.113:0/3546221961 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e1c059120 con 0x7f9e0406ca30 2024-04-12T05:48:44.006 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.011+0000 7f9e03fff700 1 -- 172.21.15.113:0/3546221961 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9e1c059120 con 0x7f9e0406ca30 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:48:44.007 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:48:44.008 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:48:44.008 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:48:44.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.013+0000 7f9e21029700 1 -- 172.21.15.113:0/3546221961 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e0406ca30 msgr2=0x7f9e0406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:44.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.013+0000 7f9e21029700 1 --2- 172.21.15.113:0/3546221961 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e0406ca30 0x7f9e0406eef0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f9e1c060f00 tx=0x7f9e0c009da0).stop 2024-04-12T05:48:44.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.013+0000 7f9e21029700 1 -- 172.21.15.113:0/3546221961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1c0fc7f0 msgr2=0x7f9e1c109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:44.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.013+0000 7f9e21029700 1 --2- 172.21.15.113:0/3546221961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1c0fc7f0 0x7f9e1c109100 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f9e10005010 tx=0x7f9e10009cb0).stop 2024-04-12T05:48:44.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.013+0000 7f9e21029700 1 -- 172.21.15.113:0/3546221961 shutdown_connections 2024-04-12T05:48:44.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.013+0000 7f9e21029700 1 --2- 172.21.15.113:0/3546221961 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e0406ca30 0x7f9e0406eef0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:44.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.013+0000 7f9e21029700 1 --2- 172.21.15.113:0/3546221961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e1c0fc7f0 0x7f9e1c109100 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:44.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.013+0000 7f9e21029700 1 --2- 172.21.15.113:0/3546221961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e1c0fd150 0x7f9e1c1095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:44.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.013+0000 7f9e21029700 1 -- 172.21.15.113:0/3546221961 >> 172.21.15.113:0/3546221961 conn(0x7f9e1c070590 msgr2=0x7f9e1c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:44.009 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.013+0000 7f9e21029700 1 -- 172.21.15.113:0/3546221961 shutdown_connections 2024-04-12T05:48:44.010 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.013+0000 7f9e21029700 1 -- 172.21.15.113:0/3546221961 wait complete. 2024-04-12T05:48:44.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.110+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/3736742878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bc80fd150 msgr2=0x7f0bc80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.110+0000 7f0bcf4d1700 1 --2- 172.21.15.113:0/3736742878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bc80fd150 0x7f0bc80f6930 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f0bc40047f0 tx=0x7f0bc4009cb0).stop 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.110+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/3736742878 shutdown_connections 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.110+0000 7f0bcf4d1700 1 --2- 172.21.15.113:0/3736742878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bc80fd150 0x7f0bc80f6930 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.110+0000 7f0bcf4d1700 1 --2- 172.21.15.113:0/3736742878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bc80fc7f0 0x7f0bc80fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.110+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/3736742878 >> 172.21.15.113:0/3736742878 conn(0x7f0bc8070590 msgr2=0x7f0bc80f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.110+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/3736742878 shutdown_connections 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.110+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/3736742878 wait complete. 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcf4d1700 1 Processor -- start 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcf4d1700 1 -- start start 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcf4d1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bc80fc7f0 0x7f0bc80f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcf4d1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bc80fd150 0x7f0bc80f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcf4d1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc80f8f20 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.108 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcf4d1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc80f9060 con 0x7f0bc80fd150 2024-04-12T05:48:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcd26d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bc80fc7f0 0x7f0bc80f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcd26d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bc80fc7f0 0x7f0bc80f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33628/0 (socket says 172.21.15.113:33628) 2024-04-12T05:48:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcd26d700 1 -- 172.21.15.113:0/2151667990 learned_addr learned my addr 172.21.15.113:0/2151667990 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:48:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcca6c700 1 --2- 172.21.15.113:0/2151667990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bc80fd150 0x7f0bc80f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcd26d700 1 -- 172.21.15.113:0/2151667990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bc80fd150 msgr2=0x7f0bc80f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcd26d700 1 --2- 172.21.15.113:0/2151667990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bc80fd150 0x7f0bc80f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.111+0000 7f0bcd26d700 1 -- 172.21.15.113:0/2151667990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc4005040 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.112+0000 7f0bcd26d700 1 --2- 172.21.15.113:0/2151667990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bc80fc7f0 0x7f0bc80f8540 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f0bb800c740 tx=0x7f0bb8007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.112+0000 7f0bbe7fc700 1 -- 172.21.15.113:0/2151667990 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bb800e680 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.112+0000 7f0bbe7fc700 1 -- 172.21.15.113:0/2151667990 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bb800ec90 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.109 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.112+0000 7f0bbe7fc700 1 -- 172.21.15.113:0/2151667990 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bb8013d60 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.112+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/2151667990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc80f9310 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.112+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/2151667990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc81146a0 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.113+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/2151667990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc805f590 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.113+0000 7f0bbe7fc700 1 -- 172.21.15.113:0/2151667990 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0bb800e7e0 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.114+0000 7f0bbe7fc700 1 --2- 172.21.15.113:0/2151667990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406caa0 0x7f0bb406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:48:44.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.114+0000 7f0bbe7fc700 1 -- 172.21.15.113:0/2151667990 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0bb808c0d0 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.114+0000 7f0bcca6c700 1 --2- 172.21.15.113:0/2151667990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406caa0 0x7f0bb406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:48:44.111 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.114+0000 7f0bcca6c700 1 --2- 172.21.15.113:0/2151667990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406caa0 0x7f0bb406ef60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f0bc4004f10 tx=0x7f0bc400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:48:44.114 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.118+0000 7f0bbe7fc700 1 -- 172.21.15.113:0/2151667990 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0bb80560f0 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.386+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/2151667990 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0bc80f9ac0 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.382 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.386+0000 7f0bbe7fc700 1 -- 172.21.15.113:0/2151667990 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f0bb8059710 con 0x7f0bc80fc7f0 2024-04-12T05:48:44.382 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:48:44.382 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:48:44.382 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:48:44.382 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:48:44.382 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:48:44.382 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:48:44.383 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:48:44.384 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:48:44.384 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:48:44.384 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:48:44.384 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:48:44.384 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:48:44.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.388+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/2151667990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406caa0 msgr2=0x7f0bb406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:44.384 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.388+0000 7f0bcf4d1700 1 --2- 172.21.15.113:0/2151667990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406caa0 0x7f0bb406ef60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f0bc4004f10 tx=0x7f0bc400b040).stop 2024-04-12T05:48:44.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.388+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/2151667990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bc80fc7f0 msgr2=0x7f0bc80f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:48:44.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.388+0000 7f0bcf4d1700 1 --2- 172.21.15.113:0/2151667990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bc80fc7f0 0x7f0bc80f8540 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f0bb800c740 tx=0x7f0bb8007ef0).stop 2024-04-12T05:48:44.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.388+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/2151667990 shutdown_connections 2024-04-12T05:48:44.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.388+0000 7f0bcf4d1700 1 --2- 172.21.15.113:0/2151667990 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0bb406caa0 0x7f0bb406ef60 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:44.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.388+0000 7f0bcf4d1700 1 --2- 172.21.15.113:0/2151667990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0bc80fc7f0 0x7f0bc80f8540 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:44.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.388+0000 7f0bcf4d1700 1 --2- 172.21.15.113:0/2151667990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bc80fd150 0x7f0bc80f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:48:44.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.388+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/2151667990 >> 172.21.15.113:0/2151667990 conn(0x7f0bc8070590 msgr2=0x7f0bc8060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:48:44.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.388+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/2151667990 shutdown_connections 2024-04-12T05:48:44.385 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:48:44.388+0000 7f0bcf4d1700 1 -- 172.21.15.113:0/2151667990 wait complete. 2024-04-12T05:48:44.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:44 smithi113 ceph-mon[103371]: from='client.35966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:44.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:44 smithi113 ceph-mon[103371]: from='client.35970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:44.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:44 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1557205518' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:48:44.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:44 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/690988282' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:48:44.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:44 smithi192 ceph-mon[85304]: from='client.35966 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:44.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:44 smithi192 ceph-mon[85304]: from='client.35970 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:44.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:44 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1557205518' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:48:44.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:44 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/690988282' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:48:45.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:45 smithi113 ceph-mon[103371]: from='client.35982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:45.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:45 smithi113 ceph-mon[103371]: pgmap v1098: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:45.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:45 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2151667990' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:48:45.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:45 smithi192 ceph-mon[85304]: from='client.35982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:48:45.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:45 smithi192 ceph-mon[85304]: pgmap v1098: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:45.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:45 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2151667990' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:48:47.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:47 smithi113 ceph-mon[103371]: pgmap v1099: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:47.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:47 smithi192 ceph-mon[85304]: pgmap v1099: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:49.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:49 smithi113 ceph-mon[103371]: pgmap v1100: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:49.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:49 smithi192 ceph-mon[85304]: pgmap v1100: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:51.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:51 smithi113 ceph-mon[103371]: pgmap v1101: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:51.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:51 smithi192 ceph-mon[85304]: pgmap v1101: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:52.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:52.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:48:53.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:53 smithi192 ceph-mon[85304]: pgmap v1102: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:53.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:53 smithi113 ceph-mon[103371]: pgmap v1102: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:55.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:55 smithi192 ceph-mon[85304]: pgmap v1103: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:55.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:55 smithi113 ceph-mon[103371]: pgmap v1103: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:57.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:57 smithi192 ceph-mon[85304]: pgmap v1104: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:57.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:57 smithi113 ceph-mon[103371]: pgmap v1104: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:59.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:48:59 smithi192 ceph-mon[85304]: pgmap v1105: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:48:59.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:48:59 smithi113 ceph-mon[103371]: pgmap v1105: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:01.382 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:01 smithi192 ceph-mon[85304]: pgmap v1106: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:01.511 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:01 smithi113 ceph-mon[103371]: pgmap v1106: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:03.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:03 smithi192 ceph-mon[85304]: pgmap v1107: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:03.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:03 smithi113 ceph-mon[103371]: pgmap v1107: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:05.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:05 smithi192 ceph-mon[85304]: pgmap v1108: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:05.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:05 smithi113 ceph-mon[103371]: pgmap v1108: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:07.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:07 smithi192 ceph-mon[85304]: pgmap v1109: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:07.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:07.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:07 smithi113 ceph-mon[103371]: pgmap v1109: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:07.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:09.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:09 smithi113 ceph-mon[103371]: pgmap v1110: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:09.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:09 smithi192 ceph-mon[85304]: pgmap v1110: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:11.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:11 smithi113 ceph-mon[103371]: pgmap v1111: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:11.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:11 smithi192 ceph-mon[85304]: pgmap v1111: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:13.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:13 smithi113 ceph-mon[103371]: pgmap v1112: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:13.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:13 smithi192 ceph-mon[85304]: pgmap v1112: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:14.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.498+0000 7f6bb24c9700 1 -- 172.21.15.113:0/3449469341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac0fc7f0 msgr2=0x7f6bac0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:14.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.498+0000 7f6bb24c9700 1 --2- 172.21.15.113:0/3449469341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac0fc7f0 0x7f6bac0fcbd0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7f6b9c0047f0 tx=0x7f6b9c009cb0).stop 2024-04-12T05:49:14.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.498+0000 7f6bb24c9700 1 -- 172.21.15.113:0/3449469341 shutdown_connections 2024-04-12T05:49:14.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.498+0000 7f6bb24c9700 1 --2- 172.21.15.113:0/3449469341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bac0fd150 0x7f6bac0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:14.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.498+0000 7f6bb24c9700 1 --2- 172.21.15.113:0/3449469341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac0fc7f0 0x7f6bac0fcbd0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:14.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.498+0000 7f6bb24c9700 1 -- 172.21.15.113:0/3449469341 >> 172.21.15.113:0/3449469341 conn(0x7f6bac070590 msgr2=0x7f6bac0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:14.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.498+0000 7f6bb24c9700 1 -- 172.21.15.113:0/3449469341 shutdown_connections 2024-04-12T05:49:14.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.498+0000 7f6bb24c9700 1 -- 172.21.15.113:0/3449469341 wait complete. 2024-04-12T05:49:14.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.498+0000 7f6bb24c9700 1 Processor -- start 2024-04-12T05:49:14.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6bb24c9700 1 -- start start 2024-04-12T05:49:14.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6bb24c9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bac0fc7f0 0x7f6bac0f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:14.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6bb24c9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac0fd150 0x7f6bac0f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:14.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6bb24c9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0f8f20 con 0x7f6bac0fd150 2024-04-12T05:49:14.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6bb24c9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0f9060 con 0x7f6bac0fc7f0 2024-04-12T05:49:14.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6bab7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac0fd150 0x7f6bac0f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:14.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6bab7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac0fd150 0x7f6bac0f8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43630/0 (socket says 172.21.15.113:43630) 2024-04-12T05:49:14.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6bab7fe700 1 -- 172.21.15.113:0/1162600355 learned_addr learned my addr 172.21.15.113:0/1162600355 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:49:14.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6babfff700 1 --2- 172.21.15.113:0/1162600355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bac0fc7f0 0x7f6bac0f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:14.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6bab7fe700 1 -- 172.21.15.113:0/1162600355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bac0fc7f0 msgr2=0x7f6bac0f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:14.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6bab7fe700 1 --2- 172.21.15.113:0/1162600355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bac0fc7f0 0x7f6bac0f8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:14.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.499+0000 7f6bab7fe700 1 -- 172.21.15.113:0/1162600355 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c005040 con 0x7f6bac0fd150 2024-04-12T05:49:14.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.500+0000 7f6bab7fe700 1 --2- 172.21.15.113:0/1162600355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac0fd150 0x7f6bac0f8a30 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7f6ba000c710 tx=0x7f6ba0007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:14.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.500+0000 7f6ba97fa700 1 -- 172.21.15.113:0/1162600355 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ba000e680 con 0x7f6bac0fd150 2024-04-12T05:49:14.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.500+0000 7f6bb24c9700 1 -- 172.21.15.113:0/1162600355 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac0f9310 con 0x7f6bac0fd150 2024-04-12T05:49:14.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.500+0000 7f6ba97fa700 1 -- 172.21.15.113:0/1162600355 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ba000ec90 con 0x7f6bac0fd150 2024-04-12T05:49:14.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.500+0000 7f6ba97fa700 1 -- 172.21.15.113:0/1162600355 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ba0013d40 con 0x7f6bac0fd150 2024-04-12T05:49:14.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.500+0000 7f6bb24c9700 1 -- 172.21.15.113:0/1162600355 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac1146d0 con 0x7f6bac0fd150 2024-04-12T05:49:14.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.501+0000 7f6bb24c9700 1 -- 172.21.15.113:0/1162600355 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bac05f590 con 0x7f6bac0fd150 2024-04-12T05:49:14.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.501+0000 7f6ba97fa700 1 -- 172.21.15.113:0/1162600355 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6ba000e7e0 con 0x7f6bac0fd150 2024-04-12T05:49:14.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.502+0000 7f6ba97fa700 1 --2- 172.21.15.113:0/1162600355 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6b9406caa0 0x7f6b9406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:14.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.502+0000 7f6ba97fa700 1 -- 172.21.15.113:0/1162600355 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6ba008bd30 con 0x7f6bac0fd150 2024-04-12T05:49:14.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.502+0000 7f6babfff700 1 --2- 172.21.15.113:0/1162600355 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6b9406caa0 0x7f6b9406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:14.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.502+0000 7f6babfff700 1 --2- 172.21.15.113:0/1162600355 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6b9406caa0 0x7f6b9406ef60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f6b9c007cf0 tx=0x7f6b9c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:14.503 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.506+0000 7f6ba97fa700 1 -- 172.21.15.113:0/1162600355 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6ba0055c90 con 0x7f6bac0fd150 2024-04-12T05:49:14.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.712+0000 7f6bb24c9700 1 -- 172.21.15.113:0/1162600355 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bac0f9b10 con 0x7f6b9406caa0 2024-04-12T05:49:14.710 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.714+0000 7f6ba97fa700 1 -- 172.21.15.113:0/1162600355 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6bac0f9b10 con 0x7f6b9406caa0 2024-04-12T05:49:14.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.716+0000 7f6bb24c9700 1 -- 172.21.15.113:0/1162600355 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6b9406caa0 msgr2=0x7f6b9406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:14.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.716+0000 7f6bb24c9700 1 --2- 172.21.15.113:0/1162600355 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6b9406caa0 0x7f6b9406ef60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f6b9c007cf0 tx=0x7f6b9c00b040).stop 2024-04-12T05:49:14.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.716+0000 7f6bb24c9700 1 -- 172.21.15.113:0/1162600355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac0fd150 msgr2=0x7f6bac0f8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:14.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.716+0000 7f6bb24c9700 1 --2- 172.21.15.113:0/1162600355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac0fd150 0x7f6bac0f8a30 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7f6ba000c710 tx=0x7f6ba0007ec0).stop 2024-04-12T05:49:14.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.716+0000 7f6bb24c9700 1 -- 172.21.15.113:0/1162600355 shutdown_connections 2024-04-12T05:49:14.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.716+0000 7f6bb24c9700 1 --2- 172.21.15.113:0/1162600355 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6b9406caa0 0x7f6b9406ef60 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:14.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.716+0000 7f6bb24c9700 1 --2- 172.21.15.113:0/1162600355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bac0fc7f0 0x7f6bac0f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:14.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.716+0000 7f6bb24c9700 1 --2- 172.21.15.113:0/1162600355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac0fd150 0x7f6bac0f8a30 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:14.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.716+0000 7f6bb24c9700 1 -- 172.21.15.113:0/1162600355 >> 172.21.15.113:0/1162600355 conn(0x7f6bac070590 msgr2=0x7f6bac060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:14.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.716+0000 7f6bb24c9700 1 -- 172.21.15.113:0/1162600355 shutdown_connections 2024-04-12T05:49:14.713 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.716+0000 7f6bb24c9700 1 -- 172.21.15.113:0/1162600355 wait complete. 2024-04-12T05:49:14.722 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:49:14.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.814+0000 7f8513827700 1 -- 172.21.15.113:0/2417323041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f850c0fd8f0 msgr2=0x7f850c0fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:14.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.814+0000 7f8513827700 1 --2- 172.21.15.113:0/2417323041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f850c0fd8f0 0x7f850c0fdcd0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7f84fc0047f0 tx=0x7f84fc009cb0).stop 2024-04-12T05:49:14.811 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.814+0000 7f8513827700 1 -- 172.21.15.113:0/2417323041 shutdown_connections 2024-04-12T05:49:14.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.814+0000 7f8513827700 1 --2- 172.21.15.113:0/2417323041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f850c0fe250 0x7f850c0f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:14.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.814+0000 7f8513827700 1 --2- 172.21.15.113:0/2417323041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f850c0fd8f0 0x7f850c0fdcd0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:14.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.814+0000 7f8513827700 1 -- 172.21.15.113:0/2417323041 >> 172.21.15.113:0/2417323041 conn(0x7f850c0f34b0 msgr2=0x7f850c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:14.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.814+0000 7f8513827700 1 -- 172.21.15.113:0/2417323041 shutdown_connections 2024-04-12T05:49:14.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.814+0000 7f8513827700 1 -- 172.21.15.113:0/2417323041 wait complete. 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.815+0000 7f8513827700 1 Processor -- start 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.815+0000 7f8513827700 1 -- start start 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8513827700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f850c0fd8f0 0x7f850c10e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8513827700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f850c0fe250 0x7f850c10e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8513827700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c10eaf0 con 0x7f850c0fe250 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8513827700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c10ec30 con 0x7f850c0fd8f0 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8510dc2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f850c0fe250 0x7f850c10e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8510dc2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f850c0fe250 0x7f850c10e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43642/0 (socket says 172.21.15.113:43642) 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8510dc2700 1 -- 172.21.15.113:0/2099093436 learned_addr learned my addr 172.21.15.113:0/2099093436 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f85115c3700 1 --2- 172.21.15.113:0/2099093436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f850c0fd8f0 0x7f850c10e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8510dc2700 1 -- 172.21.15.113:0/2099093436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f850c0fd8f0 msgr2=0x7f850c10e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:14.814 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8510dc2700 1 --2- 172.21.15.113:0/2099093436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f850c0fd8f0 0x7f850c10e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:14.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8510dc2700 1 -- 172.21.15.113:0/2099093436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc005040 con 0x7f850c0fe250 2024-04-12T05:49:14.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8510dc2700 1 --2- 172.21.15.113:0/2099093436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f850c0fe250 0x7f850c10e4f0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f8508002820 tx=0x7f8508009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:14.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f85027fc700 1 -- 172.21.15.113:0/2099093436 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8508007b40 con 0x7f850c0fe250 2024-04-12T05:49:14.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f85027fc700 1 -- 172.21.15.113:0/2099093436 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f850800f040 con 0x7f850c0fe250 2024-04-12T05:49:14.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8513827700 1 -- 172.21.15.113:0/2099093436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c1082c0 con 0x7f850c0fe250 2024-04-12T05:49:14.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f85027fc700 1 -- 172.21.15.113:0/2099093436 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85080148b0 con 0x7f850c0fe250 2024-04-12T05:49:14.815 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.816+0000 7f8513827700 1 -- 172.21.15.113:0/2099093436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c108880 con 0x7f850c0fe250 2024-04-12T05:49:14.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.817+0000 7f8513827700 1 -- 172.21.15.113:0/2099093436 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c05f590 con 0x7f850c0fe250 2024-04-12T05:49:14.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.818+0000 7f85027fc700 1 -- 172.21.15.113:0/2099093436 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8508007ca0 con 0x7f850c0fe250 2024-04-12T05:49:14.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.818+0000 7f85027fc700 1 --2- 172.21.15.113:0/2099093436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84f806caa0 0x7f84f806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:14.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.818+0000 7f85027fc700 1 -- 172.21.15.113:0/2099093436 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f850808b8f0 con 0x7f850c0fe250 2024-04-12T05:49:14.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.818+0000 7f85115c3700 1 --2- 172.21.15.113:0/2099093436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84f806caa0 0x7f84f806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:14.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.819+0000 7f85115c3700 1 --2- 172.21.15.113:0/2099093436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84f806caa0 0x7f84f806ef60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f84fc004f10 tx=0x7f84fc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:14.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:14.823+0000 7f85027fc700 1 -- 172.21.15.113:0/2099093436 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f85080593a0 con 0x7f850c0fe250 2024-04-12T05:49:15.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.029+0000 7f8513827700 1 -- 172.21.15.113:0/2099093436 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f850c108b00 con 0x7f84f806caa0 2024-04-12T05:49:15.026 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.030+0000 7f85027fc700 1 -- 172.21.15.113:0/2099093436 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f850c108b00 con 0x7f84f806caa0 2024-04-12T05:49:15.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.032+0000 7f8513827700 1 -- 172.21.15.113:0/2099093436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84f806caa0 msgr2=0x7f84f806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.032+0000 7f8513827700 1 --2- 172.21.15.113:0/2099093436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84f806caa0 0x7f84f806ef60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f84fc004f10 tx=0x7f84fc00b040).stop 2024-04-12T05:49:15.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.032+0000 7f8513827700 1 -- 172.21.15.113:0/2099093436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f850c0fe250 msgr2=0x7f850c10e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.032+0000 7f8513827700 1 --2- 172.21.15.113:0/2099093436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f850c0fe250 0x7f850c10e4f0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f8508002820 tx=0x7f8508009f20).stop 2024-04-12T05:49:15.029 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.032+0000 7f8513827700 1 -- 172.21.15.113:0/2099093436 shutdown_connections 2024-04-12T05:49:15.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.032+0000 7f8513827700 1 --2- 172.21.15.113:0/2099093436 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f84f806caa0 0x7f84f806ef60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.032+0000 7f8513827700 1 --2- 172.21.15.113:0/2099093436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f850c0fd8f0 0x7f850c10e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.032+0000 7f8513827700 1 --2- 172.21.15.113:0/2099093436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f850c0fe250 0x7f850c10e4f0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.032+0000 7f8513827700 1 -- 172.21.15.113:0/2099093436 >> 172.21.15.113:0/2099093436 conn(0x7f850c0f34b0 msgr2=0x7f850c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:15.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.032+0000 7f8513827700 1 -- 172.21.15.113:0/2099093436 shutdown_connections 2024-04-12T05:49:15.030 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.032+0000 7f8513827700 1 -- 172.21.15.113:0/2099093436 wait complete. 2024-04-12T05:49:15.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.131+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/2899643145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb380fc820 msgr2=0x7fbb380fcc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.131+0000 7fbb3d2b9700 1 --2- 172.21.15.113:0/2899643145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb380fc820 0x7fbb380fcc00 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7fbb280047f0 tx=0x7fbb28009cb0).stop 2024-04-12T05:49:15.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.131+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/2899643145 shutdown_connections 2024-04-12T05:49:15.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.131+0000 7fbb3d2b9700 1 --2- 172.21.15.113:0/2899643145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb380fd180 0x7fbb380f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.131+0000 7fbb3d2b9700 1 --2- 172.21.15.113:0/2899643145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb380fc820 0x7fbb380fcc00 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.131+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/2899643145 >> 172.21.15.113:0/2899643145 conn(0x7fbb380f2400 msgr2=0x7fbb380f4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:15.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.131+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/2899643145 shutdown_connections 2024-04-12T05:49:15.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.131+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/2899643145 wait complete. 2024-04-12T05:49:15.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.132+0000 7fbb3d2b9700 1 Processor -- start 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.132+0000 7fbb3d2b9700 1 -- start start 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.132+0000 7fbb3d2b9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb380fc820 0x7fbb3810e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.132+0000 7fbb3d2b9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb380fd180 0x7fbb3810e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.132+0000 7fbb3d2b9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3810eb10 con 0x7fbb380fd180 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.132+0000 7fbb3d2b9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3810ec50 con 0x7fbb380fc820 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.132+0000 7fbb367fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb380fd180 0x7fbb3810e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.132+0000 7fbb367fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb380fd180 0x7fbb3810e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43666/0 (socket says 172.21.15.113:43666) 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.132+0000 7fbb367fc700 1 -- 172.21.15.113:0/3257058196 learned_addr learned my addr 172.21.15.113:0/3257058196 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.133+0000 7fbb36ffd700 1 --2- 172.21.15.113:0/3257058196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb380fc820 0x7fbb3810e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.133+0000 7fbb367fc700 1 -- 172.21.15.113:0/3257058196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb380fc820 msgr2=0x7fbb3810e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.133+0000 7fbb367fc700 1 --2- 172.21.15.113:0/3257058196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb380fc820 0x7fbb3810e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.133+0000 7fbb367fc700 1 -- 172.21.15.113:0/3257058196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb28005040 con 0x7fbb380fd180 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.133+0000 7fbb367fc700 1 --2- 172.21.15.113:0/3257058196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb380fd180 0x7fbb3810e510 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7fbb2c002820 tx=0x7fbb2c009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:15.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.133+0000 7fbb1ffff700 1 -- 172.21.15.113:0/3257058196 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb2c007b40 con 0x7fbb380fd180 2024-04-12T05:49:15.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.133+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/3257058196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb381082e0 con 0x7fbb380fd180 2024-04-12T05:49:15.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.133+0000 7fbb1ffff700 1 -- 172.21.15.113:0/3257058196 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb2c00f040 con 0x7fbb380fd180 2024-04-12T05:49:15.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.133+0000 7fbb1ffff700 1 -- 172.21.15.113:0/3257058196 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb2c014910 con 0x7fbb380fd180 2024-04-12T05:49:15.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.133+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/3257058196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb381088a0 con 0x7fbb380fd180 2024-04-12T05:49:15.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.134+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/3257058196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb38060310 con 0x7fbb380fd180 2024-04-12T05:49:15.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.134+0000 7fbb1ffff700 1 -- 172.21.15.113:0/3257058196 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbb2c009110 con 0x7fbb380fd180 2024-04-12T05:49:15.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.135+0000 7fbb1ffff700 1 --2- 172.21.15.113:0/3257058196 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbb2006cab0 0x7fbb2006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:15.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.135+0000 7fbb1ffff700 1 -- 172.21.15.113:0/3257058196 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fbb2c08b7f0 con 0x7fbb380fd180 2024-04-12T05:49:15.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.135+0000 7fbb36ffd700 1 --2- 172.21.15.113:0/3257058196 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbb2006cab0 0x7fbb2006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:15.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.135+0000 7fbb36ffd700 1 --2- 172.21.15.113:0/3257058196 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbb2006cab0 0x7fbb2006ef70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fbb28004f10 tx=0x7fbb28009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:15.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.139+0000 7fbb1ffff700 1 -- 172.21.15.113:0/3257058196 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fbb2c0592a0 con 0x7fbb380fd180 2024-04-12T05:49:15.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.335+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/3257058196 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb38108f40 con 0x7fbb2006cab0 2024-04-12T05:49:15.342 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.345+0000 7fbb1ffff700 1 -- 172.21.15.113:0/3257058196 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbb38108f40 con 0x7fbb2006cab0 2024-04-12T05:49:15.342 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:49:15.342 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (37m) 14m ago 48m 25.8M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:49:15.342 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (34m) 14m ago 48m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:49:15.342 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (34m) 14m ago 47m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:49:15.342 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (36m) 14m ago 48m 88.9M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:49:15.342 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (30m) 14m ago 42m 20.4M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:49:15.342 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (30m) 14m ago 42m 19.5M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (29m) 14m ago 42m 16.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (14m) 14m ago 42m 11.2M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (38m) 14m ago 49m 510M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (37m) 14m ago 47m 457M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (35m) 14m ago 49m 90.1M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (35m) 14m ago 47m 51.7M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (37m) 14m ago 47m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (37m) 14m ago 47m 20.3M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (34m) 14m ago 46m 119M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (33m) 14m ago 46m 136M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (32m) 14m ago 45m 126M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (32m) 14m ago 45m 159M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:49:15.343 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (31m) 14m ago 44m 105M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:49:15.344 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (30m) 14m ago 44m 102M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:49:15.344 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (37m) 14m ago 47m 80.9M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:49:15.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.347+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/3257058196 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbb2006cab0 msgr2=0x7fbb2006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.347+0000 7fbb3d2b9700 1 --2- 172.21.15.113:0/3257058196 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbb2006cab0 0x7fbb2006ef70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fbb28004f10 tx=0x7fbb28009cb0).stop 2024-04-12T05:49:15.344 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.347+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/3257058196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb380fd180 msgr2=0x7fbb3810e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.347+0000 7fbb3d2b9700 1 --2- 172.21.15.113:0/3257058196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb380fd180 0x7fbb3810e510 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7fbb2c002820 tx=0x7fbb2c009f20).stop 2024-04-12T05:49:15.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.347+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/3257058196 shutdown_connections 2024-04-12T05:49:15.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.347+0000 7fbb3d2b9700 1 --2- 172.21.15.113:0/3257058196 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fbb2006cab0 0x7fbb2006ef70 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.347+0000 7fbb3d2b9700 1 --2- 172.21.15.113:0/3257058196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb380fc820 0x7fbb3810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.347+0000 7fbb3d2b9700 1 --2- 172.21.15.113:0/3257058196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb380fd180 0x7fbb3810e510 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.347+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/3257058196 >> 172.21.15.113:0/3257058196 conn(0x7fbb380f2400 msgr2=0x7fbb38060b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:15.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.347+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/3257058196 shutdown_connections 2024-04-12T05:49:15.345 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.347+0000 7fbb3d2b9700 1 -- 172.21.15.113:0/3257058196 wait complete. 2024-04-12T05:49:15.446 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.444+0000 7f5322a26700 1 -- 172.21.15.113:0/2493771227 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f531c0fe250 msgr2=0x7f531c0f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.536 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.444+0000 7f5322a26700 1 --2- 172.21.15.113:0/2493771227 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f531c0fe250 0x7f531c0f7a30 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f53100047f0 tx=0x7f5310009cb0).stop 2024-04-12T05:49:15.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.444+0000 7f5322a26700 1 -- 172.21.15.113:0/2493771227 shutdown_connections 2024-04-12T05:49:15.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.444+0000 7f5322a26700 1 --2- 172.21.15.113:0/2493771227 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f531c0fe250 0x7f531c0f7a30 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.444+0000 7f5322a26700 1 --2- 172.21.15.113:0/2493771227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531c0fd8f0 0x7f531c0fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.444+0000 7f5322a26700 1 -- 172.21.15.113:0/2493771227 >> 172.21.15.113:0/2493771227 conn(0x7f531c0f34b0 msgr2=0x7f531c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:15.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.444+0000 7f5322a26700 1 -- 172.21.15.113:0/2493771227 shutdown_connections 2024-04-12T05:49:15.538 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.444+0000 7f5322a26700 1 -- 172.21.15.113:0/2493771227 wait complete. 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.445+0000 7f5322a26700 1 Processor -- start 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.445+0000 7f5322a26700 1 -- start start 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.445+0000 7f5322a26700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f531c0fd8f0 0x7f531c10e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.445+0000 7f5322a26700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531c0fe250 0x7f531c10e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.445+0000 7f5322a26700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c10ec30 con 0x7f531c0fd8f0 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.445+0000 7f5322a26700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c10ed70 con 0x7f531c0fe250 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.445+0000 7f531bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f531c0fd8f0 0x7f531c10e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.445+0000 7f531bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f531c0fd8f0 0x7f531c10e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43682/0 (socket says 172.21.15.113:43682) 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.445+0000 7f531bfff700 1 -- 172.21.15.113:0/314348233 learned_addr learned my addr 172.21.15.113:0/314348233 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.446+0000 7f531b7fe700 1 --2- 172.21.15.113:0/314348233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531c0fe250 0x7f531c10e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.446+0000 7f531bfff700 1 -- 172.21.15.113:0/314348233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531c0fe250 msgr2=0x7f531c10e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.446+0000 7f531bfff700 1 --2- 172.21.15.113:0/314348233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531c0fe250 0x7f531c10e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.446+0000 7f531bfff700 1 -- 172.21.15.113:0/314348233 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5310005040 con 0x7f531c0fd8f0 2024-04-12T05:49:15.539 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.446+0000 7f531bfff700 1 --2- 172.21.15.113:0/314348233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f531c0fd8f0 0x7f531c10e140 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f530c00e910 tx=0x7f530c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:15.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.446+0000 7f53197fa700 1 -- 172.21.15.113:0/314348233 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f530c007b40 con 0x7f531c0fd8f0 2024-04-12T05:49:15.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.446+0000 7f5322a26700 1 -- 172.21.15.113:0/314348233 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531c1083a0 con 0x7f531c0fd8f0 2024-04-12T05:49:15.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.446+0000 7f53197fa700 1 -- 172.21.15.113:0/314348233 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f530c00f040 con 0x7f531c0fd8f0 2024-04-12T05:49:15.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.446+0000 7f53197fa700 1 -- 172.21.15.113:0/314348233 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f530c014920 con 0x7f531c0fd8f0 2024-04-12T05:49:15.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.446+0000 7f5322a26700 1 -- 172.21.15.113:0/314348233 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531c108960 con 0x7f531c0fd8f0 2024-04-12T05:49:15.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.447+0000 7f5322a26700 1 -- 172.21.15.113:0/314348233 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531c05f590 con 0x7f531c0fd8f0 2024-04-12T05:49:15.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.448+0000 7f53197fa700 1 -- 172.21.15.113:0/314348233 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f530c007ca0 con 0x7f531c0fd8f0 2024-04-12T05:49:15.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.448+0000 7f53197fa700 1 --2- 172.21.15.113:0/314348233 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f530406caa0 0x7f530406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:15.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.448+0000 7f53197fa700 1 -- 172.21.15.113:0/314348233 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f530c08b880 con 0x7f531c0fd8f0 2024-04-12T05:49:15.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.448+0000 7f531b7fe700 1 --2- 172.21.15.113:0/314348233 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f530406caa0 0x7f530406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:15.540 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.448+0000 7f531b7fe700 1 --2- 172.21.15.113:0/314348233 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f530406caa0 0x7f530406ef60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f531c109a80 tx=0x7f5310009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:15.541 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.452+0000 7f53197fa700 1 -- 172.21.15.113:0/314348233 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f530c059330 con 0x7f531c0fd8f0 2024-04-12T05:49:15.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:15 smithi113 ceph-mon[103371]: pgmap v1113: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:15.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:15 smithi192 ceph-mon[85304]: pgmap v1113: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:15.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.719+0000 7f5322a26700 1 -- 172.21.15.113:0/314348233 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f531c1098c0 con 0x7f531c0fd8f0 2024-04-12T05:49:15.716 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.719+0000 7f53197fa700 1 -- 172.21.15.113:0/314348233 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f530c055030 con 0x7f531c0fd8f0 2024-04-12T05:49:15.716 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:49:15.717 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:49:15.718 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:49:15.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.721+0000 7f5322a26700 1 -- 172.21.15.113:0/314348233 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f530406caa0 msgr2=0x7f530406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.718 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.721+0000 7f5322a26700 1 --2- 172.21.15.113:0/314348233 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f530406caa0 0x7f530406ef60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f531c109a80 tx=0x7f5310009cb0).stop 2024-04-12T05:49:15.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.721+0000 7f5322a26700 1 -- 172.21.15.113:0/314348233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f531c0fd8f0 msgr2=0x7f531c10e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.721+0000 7f5322a26700 1 --2- 172.21.15.113:0/314348233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f531c0fd8f0 0x7f531c10e140 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f530c00e910 tx=0x7f530c00b040).stop 2024-04-12T05:49:15.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.722+0000 7f5322a26700 1 -- 172.21.15.113:0/314348233 shutdown_connections 2024-04-12T05:49:15.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.722+0000 7f5322a26700 1 --2- 172.21.15.113:0/314348233 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f530406caa0 0x7f530406ef60 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.722+0000 7f5322a26700 1 --2- 172.21.15.113:0/314348233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f531c0fd8f0 0x7f531c10e140 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.722+0000 7f5322a26700 1 --2- 172.21.15.113:0/314348233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f531c0fe250 0x7f531c10e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.722+0000 7f5322a26700 1 -- 172.21.15.113:0/314348233 >> 172.21.15.113:0/314348233 conn(0x7f531c0f34b0 msgr2=0x7f531c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:15.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.722+0000 7f5322a26700 1 -- 172.21.15.113:0/314348233 shutdown_connections 2024-04-12T05:49:15.719 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.722+0000 7f5322a26700 1 -- 172.21.15.113:0/314348233 wait complete. 2024-04-12T05:49:15.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.819+0000 7ff675729700 1 -- 172.21.15.113:0/3173312779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fd8f0 msgr2=0x7ff6700fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.819+0000 7ff675729700 1 --2- 172.21.15.113:0/3173312779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fd8f0 0x7ff6700fdcd0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7ff6600047f0 tx=0x7ff660009cb0).stop 2024-04-12T05:49:15.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.819+0000 7ff675729700 1 -- 172.21.15.113:0/3173312779 shutdown_connections 2024-04-12T05:49:15.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.819+0000 7ff675729700 1 --2- 172.21.15.113:0/3173312779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fe250 0x7ff6700f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.819+0000 7ff675729700 1 --2- 172.21.15.113:0/3173312779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fd8f0 0x7ff6700fdcd0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.819+0000 7ff675729700 1 -- 172.21.15.113:0/3173312779 >> 172.21.15.113:0/3173312779 conn(0x7ff6700f34b0 msgr2=0x7ff6700f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:15.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.819+0000 7ff675729700 1 -- 172.21.15.113:0/3173312779 shutdown_connections 2024-04-12T05:49:15.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.819+0000 7ff675729700 1 -- 172.21.15.113:0/3173312779 wait complete. 2024-04-12T05:49:15.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.820+0000 7ff675729700 1 Processor -- start 2024-04-12T05:49:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.820+0000 7ff675729700 1 -- start start 2024-04-12T05:49:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.820+0000 7ff675729700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fd8f0 0x7ff67010e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.820+0000 7ff675729700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fe250 0x7ff67010e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.820+0000 7ff675729700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67010eaf0 con 0x7ff6700fe250 2024-04-12T05:49:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.820+0000 7ff675729700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67010ec30 con 0x7ff6700fd8f0 2024-04-12T05:49:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.820+0000 7ff66e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fe250 0x7ff67010e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.820+0000 7ff66e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fe250 0x7ff67010e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43704/0 (socket says 172.21.15.113:43704) 2024-04-12T05:49:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.820+0000 7ff66e7fc700 1 -- 172.21.15.113:0/3152144023 learned_addr learned my addr 172.21.15.113:0/3152144023 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:49:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.820+0000 7ff66effd700 1 --2- 172.21.15.113:0/3152144023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fd8f0 0x7ff67010e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.821+0000 7ff66e7fc700 1 -- 172.21.15.113:0/3152144023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fd8f0 msgr2=0x7ff67010e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:15.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.821+0000 7ff66e7fc700 1 --2- 172.21.15.113:0/3152144023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fd8f0 0x7ff67010e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:15.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.821+0000 7ff66e7fc700 1 -- 172.21.15.113:0/3152144023 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff660005040 con 0x7ff6700fe250 2024-04-12T05:49:15.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.821+0000 7ff66e7fc700 1 --2- 172.21.15.113:0/3152144023 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fe250 0x7ff67010e4f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7ff664002820 tx=0x7ff664009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:15.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.821+0000 7ff657fff700 1 -- 172.21.15.113:0/3152144023 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff664007b40 con 0x7ff6700fe250 2024-04-12T05:49:15.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.821+0000 7ff675729700 1 -- 172.21.15.113:0/3152144023 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6701082c0 con 0x7ff6700fe250 2024-04-12T05:49:15.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.821+0000 7ff657fff700 1 -- 172.21.15.113:0/3152144023 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff66400f040 con 0x7ff6700fe250 2024-04-12T05:49:15.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.821+0000 7ff657fff700 1 -- 172.21.15.113:0/3152144023 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6640148b0 con 0x7ff6700fe250 2024-04-12T05:49:15.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.821+0000 7ff675729700 1 -- 172.21.15.113:0/3152144023 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff670108880 con 0x7ff6700fe250 2024-04-12T05:49:15.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.822+0000 7ff675729700 1 -- 172.21.15.113:0/3152144023 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67005f590 con 0x7ff6700fe250 2024-04-12T05:49:15.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.822+0000 7ff657fff700 1 -- 172.21.15.113:0/3152144023 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff664007ca0 con 0x7ff6700fe250 2024-04-12T05:49:15.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.823+0000 7ff657fff700 1 --2- 172.21.15.113:0/3152144023 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 0x7ff65806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:15.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.823+0000 7ff657fff700 1 -- 172.21.15.113:0/3152144023 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff66408b8c0 con 0x7ff6700fe250 2024-04-12T05:49:15.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.823+0000 7ff66effd700 1 --2- 172.21.15.113:0/3152144023 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 0x7ff65806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:15.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.823+0000 7ff66effd700 1 --2- 172.21.15.113:0/3152144023 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 0x7ff65806ef60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7ff660004f10 tx=0x7ff66000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:15.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:15.827+0000 7ff657fff700 1 -- 172.21.15.113:0/3152144023 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff664059370 con 0x7ff6700fe250 2024-04-12T05:49:16.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.055+0000 7ff675729700 1 -- 172.21.15.113:0/3152144023 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff670047540 con 0x7ff6700fe250 2024-04-12T05:49:16.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.056+0000 7ff657fff700 1 -- 172.21.15.113:0/3152144023 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1839 (secure 0 0 0) 0x7ff664058f00 con 0x7ff6700fe250 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout:e28 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 19 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:18:43.275880+0000 2024-04-12T05:49:16.053 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:up {} 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:failed 0,1 2024-04-12T05:49:16.054 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{-1:34444} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:16.055 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:49:16.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.058+0000 7ff675729700 1 -- 172.21.15.113:0/3152144023 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 msgr2=0x7ff65806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:16.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.058+0000 7ff675729700 1 --2- 172.21.15.113:0/3152144023 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 0x7ff65806ef60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7ff660004f10 tx=0x7ff66000b040).stop 2024-04-12T05:49:16.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.058+0000 7ff675729700 1 -- 172.21.15.113:0/3152144023 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fe250 msgr2=0x7ff67010e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:16.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.058+0000 7ff675729700 1 --2- 172.21.15.113:0/3152144023 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fe250 0x7ff67010e4f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7ff664002820 tx=0x7ff664009f20).stop 2024-04-12T05:49:16.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.058+0000 7ff675729700 1 -- 172.21.15.113:0/3152144023 shutdown_connections 2024-04-12T05:49:16.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.058+0000 7ff675729700 1 --2- 172.21.15.113:0/3152144023 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff65806caa0 0x7ff65806ef60 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.058+0000 7ff675729700 1 --2- 172.21.15.113:0/3152144023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6700fd8f0 0x7ff67010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.058+0000 7ff675729700 1 --2- 172.21.15.113:0/3152144023 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6700fe250 0x7ff67010e4f0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.058+0000 7ff675729700 1 -- 172.21.15.113:0/3152144023 >> 172.21.15.113:0/3152144023 conn(0x7ff6700f34b0 msgr2=0x7ff6700fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:16.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.058+0000 7ff675729700 1 -- 172.21.15.113:0/3152144023 shutdown_connections 2024-04-12T05:49:16.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.058+0000 7ff675729700 1 -- 172.21.15.113:0/3152144023 wait complete. 2024-04-12T05:49:16.057 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:49:16.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.157+0000 7f1031601700 1 -- 172.21.15.113:0/1218601999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f102c0fc7f0 msgr2=0x7f102c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:16.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.157+0000 7f1031601700 1 --2- 172.21.15.113:0/1218601999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f102c0fc7f0 0x7f102c0fcbd0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7f101c0047f0 tx=0x7f101c009cb0).stop 2024-04-12T05:49:16.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.157+0000 7f1031601700 1 -- 172.21.15.113:0/1218601999 shutdown_connections 2024-04-12T05:49:16.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.157+0000 7f1031601700 1 --2- 172.21.15.113:0/1218601999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f102c0fd150 0x7f102c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.157+0000 7f1031601700 1 --2- 172.21.15.113:0/1218601999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f102c0fc7f0 0x7f102c0fcbd0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.157+0000 7f1031601700 1 -- 172.21.15.113:0/1218601999 >> 172.21.15.113:0/1218601999 conn(0x7f102c0f23b0 msgr2=0x7f102c0f47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:16.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.157+0000 7f1031601700 1 -- 172.21.15.113:0/1218601999 shutdown_connections 2024-04-12T05:49:16.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.157+0000 7f1031601700 1 -- 172.21.15.113:0/1218601999 wait complete. 2024-04-12T05:49:16.156 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.158+0000 7f1031601700 1 Processor -- start 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.158+0000 7f1031601700 1 -- start start 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.158+0000 7f1031601700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f102c0fc7f0 0x7f102c10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.158+0000 7f1031601700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f102c0fd150 0x7f102c10e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f1031601700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c10eb20 con 0x7f102c0fd150 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f1031601700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c10ec60 con 0x7f102c0fc7f0 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f102a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f102c0fd150 0x7f102c10e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f102a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f102c0fd150 0x7f102c10e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43714/0 (socket says 172.21.15.113:43714) 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f102a7fc700 1 -- 172.21.15.113:0/556278526 learned_addr learned my addr 172.21.15.113:0/556278526 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f102affd700 1 --2- 172.21.15.113:0/556278526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f102c0fc7f0 0x7f102c10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f102a7fc700 1 -- 172.21.15.113:0/556278526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f102c0fc7f0 msgr2=0x7f102c10e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f102a7fc700 1 --2- 172.21.15.113:0/556278526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f102c0fc7f0 0x7f102c10e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f102a7fc700 1 -- 172.21.15.113:0/556278526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c005040 con 0x7f102c0fd150 2024-04-12T05:49:16.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f102a7fc700 1 --2- 172.21.15.113:0/556278526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f102c0fd150 0x7f102c10e520 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f1020002820 tx=0x7f1020009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:16.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f1013fff700 1 -- 172.21.15.113:0/556278526 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1020007b40 con 0x7f102c0fd150 2024-04-12T05:49:16.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f1013fff700 1 -- 172.21.15.113:0/556278526 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f102000f040 con 0x7f102c0fd150 2024-04-12T05:49:16.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f1031601700 1 -- 172.21.15.113:0/556278526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c1082f0 con 0x7f102c0fd150 2024-04-12T05:49:16.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f1013fff700 1 -- 172.21.15.113:0/556278526 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1020014910 con 0x7f102c0fd150 2024-04-12T05:49:16.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.159+0000 7f1031601700 1 -- 172.21.15.113:0/556278526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102c1088b0 con 0x7f102c0fd150 2024-04-12T05:49:16.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.160+0000 7f1031601700 1 -- 172.21.15.113:0/556278526 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102c060370 con 0x7f102c0fd150 2024-04-12T05:49:16.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.161+0000 7f1013fff700 1 -- 172.21.15.113:0/556278526 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1020007ca0 con 0x7f102c0fd150 2024-04-12T05:49:16.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.161+0000 7f1013fff700 1 --2- 172.21.15.113:0/556278526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f101406cab0 0x7f101406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:16.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.161+0000 7f1013fff700 1 -- 172.21.15.113:0/556278526 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f102008b800 con 0x7f102c0fd150 2024-04-12T05:49:16.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.161+0000 7f102affd700 1 --2- 172.21.15.113:0/556278526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f101406cab0 0x7f101406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:16.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.161+0000 7f102affd700 1 --2- 172.21.15.113:0/556278526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f101406cab0 0x7f101406ef70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f101c004f10 tx=0x7f101c009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:16.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.165+0000 7f1013fff700 1 -- 172.21.15.113:0/556278526 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f10200592b0 con 0x7f102c0fd150 2024-04-12T05:49:16.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.371+0000 7f1031601700 1 -- 172.21.15.113:0/556278526 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f102c108f50 con 0x7f101406cab0 2024-04-12T05:49:16.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.373+0000 7f1013fff700 1 -- 172.21.15.113:0/556278526 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f102c108f50 con 0x7f101406cab0 2024-04-12T05:49:16.369 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3", 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": true, 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [ 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "crash", 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr", 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "osd", 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "mds", 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "mon" 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: ], 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": "16/21 daemons upgraded", 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "Currently upgrading mds daemons", 2024-04-12T05:49:16.370 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:49:16.371 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:49:16.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.375+0000 7f1031601700 1 -- 172.21.15.113:0/556278526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f101406cab0 msgr2=0x7f101406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:16.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.375+0000 7f1031601700 1 --2- 172.21.15.113:0/556278526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f101406cab0 0x7f101406ef70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f101c004f10 tx=0x7f101c009cb0).stop 2024-04-12T05:49:16.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.375+0000 7f1031601700 1 -- 172.21.15.113:0/556278526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f102c0fd150 msgr2=0x7f102c10e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:16.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.375+0000 7f1031601700 1 --2- 172.21.15.113:0/556278526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f102c0fd150 0x7f102c10e520 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f1020002820 tx=0x7f1020009f20).stop 2024-04-12T05:49:16.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.375+0000 7f1031601700 1 -- 172.21.15.113:0/556278526 shutdown_connections 2024-04-12T05:49:16.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.375+0000 7f1031601700 1 --2- 172.21.15.113:0/556278526 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f101406cab0 0x7f101406ef70 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.375+0000 7f1031601700 1 --2- 172.21.15.113:0/556278526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f102c0fc7f0 0x7f102c10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.375+0000 7f1031601700 1 --2- 172.21.15.113:0/556278526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f102c0fd150 0x7f102c10e520 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.375+0000 7f1031601700 1 -- 172.21.15.113:0/556278526 >> 172.21.15.113:0/556278526 conn(0x7f102c0f23b0 msgr2=0x7f102c061360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:16.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.375+0000 7f1031601700 1 -- 172.21.15.113:0/556278526 shutdown_connections 2024-04-12T05:49:16.373 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.375+0000 7f1031601700 1 -- 172.21.15.113:0/556278526 wait complete. 2024-04-12T05:49:16.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.473+0000 7f3981c5e700 1 -- 172.21.15.113:0/469219834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f397c0fe250 msgr2=0x7f397c0f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:16.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.473+0000 7f3981c5e700 1 --2- 172.21.15.113:0/469219834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f397c0fe250 0x7f397c0f7a30 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7f39640047f0 tx=0x7f3964009cb0).stop 2024-04-12T05:49:16.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.473+0000 7f3981c5e700 1 -- 172.21.15.113:0/469219834 shutdown_connections 2024-04-12T05:49:16.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.473+0000 7f3981c5e700 1 --2- 172.21.15.113:0/469219834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f397c0fe250 0x7f397c0f7a30 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.473+0000 7f3981c5e700 1 --2- 172.21.15.113:0/469219834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f397c0fd8f0 0x7f397c0fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:49:16.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.473+0000 7f3981c5e700 1 -- 172.21.15.113:0/469219834 >> 172.21.15.113:0/469219834 conn(0x7f397c0f34b0 msgr2=0x7f397c0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:16.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.473+0000 7f3981c5e700 1 -- 172.21.15.113:0/469219834 shutdown_connections 2024-04-12T05:49:16.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.473+0000 7f3981c5e700 1 -- 172.21.15.113:0/469219834 wait complete. 2024-04-12T05:49:16.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f3981c5e700 1 Processor -- start 2024-04-12T05:49:16.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f3981c5e700 1 -- start start 2024-04-12T05:49:16.472 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f3981c5e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f397c0fd8f0 0x7f397c10e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f3981c5e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f397c0fe250 0x7f397c10e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f3981c5e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c10ebf0 con 0x7f397c0fd8f0 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f3981c5e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c108210 con 0x7f397c0fe250 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f397b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f397c0fd8f0 0x7f397c10e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f397b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f397c0fd8f0 0x7f397c10e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43738/0 (socket says 172.21.15.113:43738) 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f397b7fe700 1 -- 172.21.15.113:0/106133167 learned_addr learned my addr 172.21.15.113:0/106133167 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f397affd700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f397c0fe250 0x7f397c10e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f397b7fe700 1 -- 172.21.15.113:0/106133167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f397c0fe250 msgr2=0x7f397c10e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f397b7fe700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f397c0fe250 0x7f397c10e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f397b7fe700 1 -- 172.21.15.113:0/106133167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3964005040 con 0x7f397c0fd8f0 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.474+0000 7f397affd700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f397c0fe250 0x7f397c10e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.475+0000 7f397b7fe700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f397c0fd8f0 0x7f397c10e070 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7f396c00b4b0 tx=0x7f396c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:16.473 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.475+0000 7f3978ff9700 1 -- 172.21.15.113:0/106133167 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f396c002500 con 0x7f397c0fd8f0 2024-04-12T05:49:16.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.475+0000 7f3978ff9700 1 -- 172.21.15.113:0/106133167 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f396c002b10 con 0x7f397c0fd8f0 2024-04-12T05:49:16.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.475+0000 7f3981c5e700 1 -- 172.21.15.113:0/106133167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c1084c0 con 0x7f397c0fd8f0 2024-04-12T05:49:16.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.475+0000 7f3978ff9700 1 -- 172.21.15.113:0/106133167 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f396c012cf0 con 0x7f397c0fd8f0 2024-04-12T05:49:16.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.475+0000 7f3981c5e700 1 -- 172.21.15.113:0/106133167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c108a00 con 0x7f397c0fd8f0 2024-04-12T05:49:16.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.476+0000 7f3981c5e700 1 -- 172.21.15.113:0/106133167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c05fdb0 con 0x7f397c0fd8f0 2024-04-12T05:49:16.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.476+0000 7f3978ff9700 1 -- 172.21.15.113:0/106133167 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f396c002660 con 0x7f397c0fd8f0 2024-04-12T05:49:16.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.477+0000 7f3978ff9700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f396806caa0 0x7f396806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:16.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.477+0000 7f3978ff9700 1 -- 172.21.15.113:0/106133167 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f396c08bf30 con 0x7f397c0fd8f0 2024-04-12T05:49:16.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.477+0000 7f397affd700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f396806caa0 0x7f396806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:16.475 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.477+0000 7f397affd700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f396806caa0 0x7f396806ef60 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f3964004f10 tx=0x7f396400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:16.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.481+0000 7f3978ff9700 1 -- 172.21.15.113:0/106133167 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f396c0bf2c0 con 0x7f397c0fd8f0 2024-04-12T05:49:16.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:16 smithi113 ceph-mon[103371]: from='client.35990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:16.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:16 smithi113 ceph-mon[103371]: from='client.35994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:16.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:16 smithi113 ceph-mon[103371]: from='client.35998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:16.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:16 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/314348233' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:16.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:16 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3152144023' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:49:16.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:16 smithi192 ceph-mon[85304]: from='client.35990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:16.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:16 smithi192 ceph-mon[85304]: from='client.35994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:16.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:16 smithi192 ceph-mon[85304]: from='client.35998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:16.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:16 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/314348233' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:16.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:16 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3152144023' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:49:16.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.746+0000 7f3981c5e700 1 -- 172.21.15.113:0/106133167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f397c047540 con 0x7f397c0fd8f0 2024-04-12T05:49:16.743 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.746+0000 7f3978ff9700 1 -- 172.21.15.113:0/106133167 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f397c047540 con 0x7f397c0fd8f0 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout: host smithi192 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/mon.smithi192/config 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout:Traceback (most recent call last): 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout: "__main__", mod_spec) 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout: exec(code, run_globals) 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10889, in 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 10877, in main 2024-04-12T05:49:16.744 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-12T05:49:16.745 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-12T05:49:16.745 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-12T05:49:16.745 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-12T05:49:16.745 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-12T05:49:16.745 INFO:teuthology.orchestra.run.smithi113.stdout: File "/tmp/tmpxnq8s67_.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-12T05:49:16.745 INFO:teuthology.orchestra.run.smithi113.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 -e NODE_NAME=smithi192 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/run/ceph:z -v /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpftgbg6li:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:72f2afc3fa84ea067411fbf5c1562b80cbcb72ca6ea6f1e44b170e45174046f3 inventory --format=json-pretty --filter-for-batch 2024-04-12T05:49:16.745 INFO:teuthology.orchestra.run.smithi113.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T05:49:16.745 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is degraded 2024-04-12T05:49:16.745 INFO:teuthology.orchestra.run.smithi113.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-12T05:49:16.745 INFO:teuthology.orchestra.run.smithi113.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-12T05:49:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.748+0000 7f3981c5e700 1 -- 172.21.15.113:0/106133167 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f396806caa0 msgr2=0x7f396806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.748+0000 7f3981c5e700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f396806caa0 0x7f396806ef60 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f3964004f10 tx=0x7f396400b040).stop 2024-04-12T05:49:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.748+0000 7f3981c5e700 1 -- 172.21.15.113:0/106133167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f397c0fd8f0 msgr2=0x7f397c10e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.748+0000 7f3981c5e700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f397c0fd8f0 0x7f397c10e070 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7f396c00b4b0 tx=0x7f396c0090f0).stop 2024-04-12T05:49:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.748+0000 7f3981c5e700 1 -- 172.21.15.113:0/106133167 shutdown_connections 2024-04-12T05:49:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.748+0000 7f3981c5e700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f396806caa0 0x7f396806ef60 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.748+0000 7f3981c5e700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f397c0fd8f0 0x7f397c10e070 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.748+0000 7f3981c5e700 1 --2- 172.21.15.113:0/106133167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f397c0fe250 0x7f397c10e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:16.746 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.748+0000 7f3981c5e700 1 -- 172.21.15.113:0/106133167 >> 172.21.15.113:0/106133167 conn(0x7f397c0f34b0 msgr2=0x7f397c0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:16.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.749+0000 7f3981c5e700 1 -- 172.21.15.113:0/106133167 shutdown_connections 2024-04-12T05:49:16.747 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:16.749+0000 7f3981c5e700 1 -- 172.21.15.113:0/106133167 wait complete. 2024-04-12T05:49:17.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:17 smithi113 ceph-mon[103371]: pgmap v1114: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:17.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:17 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/106133167' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:49:17.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:17 smithi192 ceph-mon[85304]: pgmap v1114: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:17.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:17 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/106133167' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:49:18.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:18 smithi113 ceph-mon[103371]: from='client.36010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:18.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:18 smithi192 ceph-mon[85304]: from='client.36010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:19.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:19 smithi113 ceph-mon[103371]: pgmap v1115: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:19.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:19 smithi192 ceph-mon[85304]: pgmap v1115: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:21.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:21 smithi113 ceph-mon[103371]: pgmap v1116: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:21.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:21 smithi192 ceph-mon[85304]: pgmap v1116: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:22.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:22.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:23.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:23 smithi113 ceph-mon[103371]: pgmap v1117: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:23.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:23 smithi192 ceph-mon[85304]: pgmap v1117: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:25.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:25 smithi113 ceph-mon[103371]: pgmap v1118: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:25.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:25 smithi192 ceph-mon[85304]: pgmap v1118: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:27.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:27 smithi113 ceph-mon[103371]: pgmap v1119: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:27.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:27 smithi192 ceph-mon[85304]: pgmap v1119: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:29.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:29 smithi113 ceph-mon[103371]: pgmap v1120: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:29.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:29 smithi192 ceph-mon[85304]: pgmap v1120: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:31.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:31 smithi113 ceph-mon[103371]: pgmap v1121: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:31.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:31 smithi192 ceph-mon[85304]: pgmap v1121: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:33.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:33 smithi113 ceph-mon[103371]: pgmap v1122: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:33.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:33 smithi192 ceph-mon[85304]: pgmap v1122: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:35.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:35 smithi113 ceph-mon[103371]: pgmap v1123: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:35.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:35 smithi192 ceph-mon[85304]: pgmap v1123: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:37.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:37 smithi113 ceph-mon[103371]: pgmap v1124: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:37.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:37.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:37 smithi192 ceph-mon[85304]: pgmap v1124: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:37.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: pgmap v1125: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:49:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:49:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:49:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:39.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi113.ovmadh"}]': finished 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi113.ydsbei"}]': finished 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi192.ammjap"}]': finished 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi192.dtnaoi"}]': finished 2024-04-12T05:49:39.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:39 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-12T05:49:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: pgmap v1125: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:49:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:49:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:49:39.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi113.ovmadh"}]: dispatch 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi113.ovmadh"}]': finished 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi113.ydsbei"}]: dispatch 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi113.ydsbei"}]': finished 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi192.ammjap"}]: dispatch 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi192.ammjap"}]': finished 2024-04-12T05:49:39.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi192.dtnaoi"}]: dispatch 2024-04-12T05:49:39.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi192.dtnaoi"}]': finished 2024-04-12T05:49:39.694 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:39 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-12T05:49:40.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: pgmap v1126: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: pgmap v1127: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all mds 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: Upgrade: Setting filesystem cephfs Joinable 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: Standby daemon mds.cephfs.smithi113.ovmadh assigned to filesystem cephfs as rank 0 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: Standby daemon mds.cephfs.smithi113.ydsbei assigned to filesystem cephfs as rank 1 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: fsmap cephfs:2/2 {0=cephfs.smithi113.ovmadh=up:replay,1=cephfs.smithi113.ydsbei=up:replay} 2 up:standby 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-12T05:49:40.671 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.672 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-12T05:49:40.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-12T05:49:40.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-12T05:49:40.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-12T05:49:40.673 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:40 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:49:40.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: pgmap v1126: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:40.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: pgmap v1127: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:40.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all mds 2024-04-12T05:49:40.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: Upgrade: Setting filesystem cephfs Joinable 2024-04-12T05:49:40.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-12T05:49:40.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: Standby daemon mds.cephfs.smithi113.ovmadh assigned to filesystem cephfs as rank 0 2024-04-12T05:49:40.697 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: Standby daemon mds.cephfs.smithi113.ydsbei assigned to filesystem cephfs as rank 1 2024-04-12T05:49:40.697 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T05:49:40.697 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-12T05:49:40.697 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: fsmap cephfs:2/2 {0=cephfs.smithi113.ovmadh=up:replay,1=cephfs.smithi113.ydsbei=up:replay} 2 up:standby 2024-04-12T05:49:40.697 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.697 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.697 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.698 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:40.699 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:49:40.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-12T05:49:40.700 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-12T05:49:40.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-12T05:49:40.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-12T05:49:40.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-12T05:49:40.701 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-12T05:49:40.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-12T05:49:40.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-12T05:49:40.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-12T05:49:40.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-12T05:49:40.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-12T05:49:40.702 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-12T05:49:40.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-12T05:49:40.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-12T05:49:40.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-12T05:49:40.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-12T05:49:40.703 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-12T05:49:40.704 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-12T05:49:40.704 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-12T05:49:40.704 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-12T05:49:40.704 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-12T05:49:40.704 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.704 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.705 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.705 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.705 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.705 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-12T05:49:40.705 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-12T05:49:40.705 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-12T05:49:40.706 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-12T05:49:40.706 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-12T05:49:40.706 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:40 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:49:41.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all rgw 2024-04-12T05:49:41.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all rbd-mirror 2024-04-12T05:49:41.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all ceph-exporter 2024-04-12T05:49:41.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all iscsi 2024-04-12T05:49:41.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all nfs 2024-04-12T05:49:41.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: Upgrade: Setting container_image for all nvmeof 2024-04-12T05:49:41.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: Upgrade: Finalizing container_image settings 2024-04-12T05:49:41.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: Upgrade: Complete! 2024-04-12T05:49:41.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:41.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:41.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:41.443 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:41 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all rgw 2024-04-12T05:49:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all rbd-mirror 2024-04-12T05:49:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all ceph-exporter 2024-04-12T05:49:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all iscsi 2024-04-12T05:49:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all nfs 2024-04-12T05:49:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: Upgrade: Setting container_image for all nvmeof 2024-04-12T05:49:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: Upgrade: Finalizing container_image settings 2024-04-12T05:49:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: Upgrade: Complete! 2024-04-12T05:49:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:41.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:41.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:41.669 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:41 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:42.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:42 smithi113 ceph-mon[103371]: pgmap v1128: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:42.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:42 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:42.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:42 smithi192 ceph-mon[85304]: pgmap v1128: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:49:42.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:42 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:44.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:44 smithi113 ceph-mon[103371]: pgmap v1129: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 2.7 MiB/s rd, 1 op/s 2024-04-12T05:49:44.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:44 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:44.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:44 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:44.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:44 smithi192 ceph-mon[85304]: pgmap v1129: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 2.7 MiB/s rd, 1 op/s 2024-04-12T05:49:44.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:44 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:44.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:44 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:45.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:45 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:45.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:45 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:45.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:45 smithi113 ceph-mon[103371]: pgmap v1130: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 4 op/s 2024-04-12T05:49:45.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:45 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:45.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:45 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:45.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:45 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:45.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:45 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:45.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:45 smithi192 ceph-mon[85304]: pgmap v1130: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 4 op/s 2024-04-12T05:49:45.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:45 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:45.943 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:45 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:46.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.854+0000 7f9825353700 1 -- 172.21.15.113:0/4009421086 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fe250 msgr2=0x7f98200f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:46.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.854+0000 7f9825353700 1 --2- 172.21.15.113:0/4009421086 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fe250 0x7f98200f7a30 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f98140047f0 tx=0x7f9814009cb0).stop 2024-04-12T05:49:46.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.854+0000 7f9825353700 1 -- 172.21.15.113:0/4009421086 shutdown_connections 2024-04-12T05:49:46.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.854+0000 7f9825353700 1 --2- 172.21.15.113:0/4009421086 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fe250 0x7f98200f7a30 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:46.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.854+0000 7f9825353700 1 --2- 172.21.15.113:0/4009421086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fd8f0 0x7f98200fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:46.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.854+0000 7f9825353700 1 -- 172.21.15.113:0/4009421086 >> 172.21.15.113:0/4009421086 conn(0x7f98200f34b0 msgr2=0x7f98200f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:46.851 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.854+0000 7f9825353700 1 -- 172.21.15.113:0/4009421086 shutdown_connections 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.854+0000 7f9825353700 1 -- 172.21.15.113:0/4009421086 wait complete. 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.854+0000 7f9825353700 1 Processor -- start 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f9825353700 1 -- start start 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f9825353700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fd8f0 0x7f9820069f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f9825353700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fe250 0x7f982006a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f9825353700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982006ce70 con 0x7f98200fe250 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f9825353700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982006cfb0 con 0x7f98200fd8f0 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f981e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fe250 0x7f982006a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f981effd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fd8f0 0x7f9820069f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f981effd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fd8f0 0x7f9820069f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.113:57834/0 (socket says 172.21.15.113:57834) 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f981e7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fe250 0x7f982006a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38654/0 (socket says 172.21.15.113:38654) 2024-04-12T05:49:46.852 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f981effd700 1 -- 172.21.15.113:0/410022245 learned_addr learned my addr 172.21.15.113:0/410022245 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f981e7fc700 1 -- 172.21.15.113:0/410022245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fd8f0 msgr2=0x7f9820069f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f981e7fc700 1 --2- 172.21.15.113:0/410022245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fd8f0 0x7f9820069f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f981e7fc700 1 -- 172.21.15.113:0/410022245 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9814005040 con 0x7f98200fe250 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f981effd700 1 --2- 172.21.15.113:0/410022245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fd8f0 0x7f9820069f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.855+0000 7f981e7fc700 1 --2- 172.21.15.113:0/410022245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fe250 0x7f982006a420 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f9814000c00 tx=0x7f9814009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.856+0000 7f9807fff700 1 -- 172.21.15.113:0/410022245 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f981400e030 con 0x7f98200fe250 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.856+0000 7f9825353700 1 -- 172.21.15.113:0/410022245 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982006aa20 con 0x7f98200fe250 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.856+0000 7f9807fff700 1 -- 172.21.15.113:0/410022245 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9814007e90 con 0x7f98200fe250 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.856+0000 7f9807fff700 1 -- 172.21.15.113:0/410022245 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9814018070 con 0x7f98200fe250 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.856+0000 7f9825353700 1 -- 172.21.15.113:0/410022245 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982011ce50 con 0x7f98200fe250 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.857+0000 7f9805ffb700 1 -- 172.21.15.113:0/410022245 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982005f590 con 0x7f98200fe250 2024-04-12T05:49:46.854 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.857+0000 7f9807fff700 1 -- 172.21.15.113:0/410022245 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f981401d5b0 con 0x7f98200fe250 2024-04-12T05:49:46.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.858+0000 7f9807fff700 1 --2- 172.21.15.113:0/410022245 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 0x7f980806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:46.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.858+0000 7f9807fff700 1 -- 172.21.15.113:0/410022245 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f981408d130 con 0x7f98200fe250 2024-04-12T05:49:46.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.858+0000 7f981effd700 1 --2- 172.21.15.113:0/410022245 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 0x7f980806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:46.855 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.858+0000 7f981effd700 1 --2- 172.21.15.113:0/410022245 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 0x7f980806ef60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f9810004fd0 tx=0x7f981000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:46.862 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:46.866+0000 7f9807fff700 1 -- 172.21.15.113:0/410022245 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9814057150 con 0x7f98200fe250 2024-04-12T05:49:47.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.097+0000 7f9805ffb700 1 -- 172.21.15.113:0/410022245 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9820047540 con 0x7f980806caa0 2024-04-12T05:49:47.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.098+0000 7f9807fff700 1 -- 172.21.15.113:0/410022245 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f9820047540 con 0x7f980806caa0 2024-04-12T05:49:47.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.101+0000 7f9805ffb700 1 -- 172.21.15.113:0/410022245 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 msgr2=0x7f980806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:47.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.101+0000 7f9805ffb700 1 --2- 172.21.15.113:0/410022245 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 0x7f980806ef60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f9810004fd0 tx=0x7f981000b040).stop 2024-04-12T05:49:47.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.101+0000 7f9805ffb700 1 -- 172.21.15.113:0/410022245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fe250 msgr2=0x7f982006a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:47.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.101+0000 7f9805ffb700 1 --2- 172.21.15.113:0/410022245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fe250 0x7f982006a420 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f9814000c00 tx=0x7f9814009cb0).stop 2024-04-12T05:49:47.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.101+0000 7f9805ffb700 1 -- 172.21.15.113:0/410022245 shutdown_connections 2024-04-12T05:49:47.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.101+0000 7f9805ffb700 1 --2- 172.21.15.113:0/410022245 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f980806caa0 0x7f980806ef60 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:47.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.101+0000 7f9805ffb700 1 --2- 172.21.15.113:0/410022245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98200fd8f0 0x7f9820069f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:47.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.101+0000 7f9805ffb700 1 --2- 172.21.15.113:0/410022245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98200fe250 0x7f982006a420 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:47.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.101+0000 7f9805ffb700 1 -- 172.21.15.113:0/410022245 >> 172.21.15.113:0/410022245 conn(0x7f98200f34b0 msgr2=0x7f98200fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:47.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.101+0000 7f9805ffb700 1 -- 172.21.15.113:0/410022245 shutdown_connections 2024-04-12T05:49:47.097 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:47.101+0000 7f9805ffb700 1 -- 172.21.15.113:0/410022245 wait complete. 2024-04-12T05:49:47.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:47 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] up:resolve 2024-04-12T05:49:47.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:47 smithi113 ceph-mon[103371]: fsmap cephfs:2/2 {0=cephfs.smithi113.ovmadh=up:replay,1=cephfs.smithi113.ydsbei=up:resolve} 2 up:standby 2024-04-12T05:49:47.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:47 smithi113 ceph-mon[103371]: pgmap v1131: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 28 MiB/s rd, 9 op/s 2024-04-12T05:49:47.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:47 smithi113 ceph-mon[103371]: from='client.36018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:47.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:47 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] up:resolve 2024-04-12T05:49:47.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:47 smithi192 ceph-mon[85304]: fsmap cephfs:2/2 {0=cephfs.smithi113.ovmadh=up:replay,1=cephfs.smithi113.ydsbei=up:resolve} 2 up:standby 2024-04-12T05:49:47.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:47 smithi192 ceph-mon[85304]: pgmap v1131: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 28 MiB/s rd, 9 op/s 2024-04-12T05:49:47.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:47 smithi192 ceph-mon[85304]: from='client.36018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:48.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:48 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] up:resolve 2024-04-12T05:49:48.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:48 smithi113 ceph-mon[103371]: fsmap cephfs:2/2 {0=cephfs.smithi113.ovmadh=up:resolve,1=cephfs.smithi113.ydsbei=up:resolve} 2 up:standby 2024-04-12T05:49:48.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:48 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] up:resolve 2024-04-12T05:49:48.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:48 smithi192 ceph-mon[85304]: fsmap cephfs:2/2 {0=cephfs.smithi113.ovmadh=up:resolve,1=cephfs.smithi113.ydsbei=up:resolve} 2 up:standby 2024-04-12T05:49:49.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:49 smithi113 ceph-mon[103371]: reconnect by client.14486 192.168.0.1:0/3583258718 after 0 2024-04-12T05:49:49.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:49 smithi113 ceph-mon[103371]: reconnect by client.14486 192.168.0.1:0/3583258718 after 0 2024-04-12T05:49:49.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:49 smithi113 ceph-mon[103371]: reconnect by client.24303 192.168.0.1:0/1598147928 after 0.000999979 2024-04-12T05:49:49.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:49 smithi113 ceph-mon[103371]: reconnect by client.24303 192.168.0.1:0/1598147928 after 0.000999979 2024-04-12T05:49:49.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:49 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] up:reconnect 2024-04-12T05:49:49.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:49 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] up:reconnect 2024-04-12T05:49:49.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:49 smithi113 ceph-mon[103371]: fsmap cephfs:2/2 {0=cephfs.smithi113.ovmadh=up:reconnect,1=cephfs.smithi113.ydsbei=up:reconnect} 2 up:standby 2024-04-12T05:49:49.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:49 smithi113 ceph-mon[103371]: pgmap v1132: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 8 op/s 2024-04-12T05:49:49.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:49 smithi192 ceph-mon[85304]: reconnect by client.14486 192.168.0.1:0/3583258718 after 0 2024-04-12T05:49:49.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:49 smithi192 ceph-mon[85304]: reconnect by client.14486 192.168.0.1:0/3583258718 after 0 2024-04-12T05:49:49.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:49 smithi192 ceph-mon[85304]: reconnect by client.24303 192.168.0.1:0/1598147928 after 0.000999979 2024-04-12T05:49:49.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:49 smithi192 ceph-mon[85304]: reconnect by client.24303 192.168.0.1:0/1598147928 after 0.000999979 2024-04-12T05:49:49.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:49 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] up:reconnect 2024-04-12T05:49:49.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:49 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] up:reconnect 2024-04-12T05:49:49.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:49 smithi192 ceph-mon[85304]: fsmap cephfs:2/2 {0=cephfs.smithi113.ovmadh=up:reconnect,1=cephfs.smithi113.ydsbei=up:reconnect} 2 up:standby 2024-04-12T05:49:49.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:49 smithi192 ceph-mon[85304]: pgmap v1132: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 8 op/s 2024-04-12T05:49:50.000 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch ps' 2024-04-12T05:49:50.753 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:50 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] up:rejoin 2024-04-12T05:49:50.753 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:50 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] up:rejoin 2024-04-12T05:49:50.753 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:50 smithi113 ceph-mon[103371]: fsmap cephfs:2/2 {0=cephfs.smithi113.ovmadh=up:rejoin,1=cephfs.smithi113.ydsbei=up:rejoin} 2 up:standby 2024-04-12T05:49:50.753 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:50 smithi113 ceph-mon[103371]: daemon mds.cephfs.smithi113.ydsbei is now active in filesystem cephfs as rank 1 2024-04-12T05:49:50.753 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:50 smithi113 ceph-mon[103371]: daemon mds.cephfs.smithi113.ovmadh is now active in filesystem cephfs as rank 0 2024-04-12T05:49:50.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:50 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] up:rejoin 2024-04-12T05:49:50.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:50 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] up:rejoin 2024-04-12T05:49:50.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:50 smithi192 ceph-mon[85304]: fsmap cephfs:2/2 {0=cephfs.smithi113.ovmadh=up:rejoin,1=cephfs.smithi113.ydsbei=up:rejoin} 2 up:standby 2024-04-12T05:49:50.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:50 smithi192 ceph-mon[85304]: daemon mds.cephfs.smithi113.ydsbei is now active in filesystem cephfs as rank 1 2024-04-12T05:49:50.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:50 smithi192 ceph-mon[85304]: daemon mds.cephfs.smithi113.ovmadh is now active in filesystem cephfs as rank 0 2024-04-12T05:49:51.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:51 smithi113 ceph-mon[103371]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T05:49:51.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:51 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] up:active 2024-04-12T05:49:51.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:51 smithi113 ceph-mon[103371]: mds.? [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] up:active 2024-04-12T05:49:51.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:51 smithi113 ceph-mon[103371]: fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi113.ydsbei=up:active} 2 up:standby 2024-04-12T05:49:51.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:51 smithi113 ceph-mon[103371]: pgmap v1133: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 170 B/s wr, 7 op/s 2024-04-12T05:49:51.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:51 smithi192 ceph-mon[85304]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T05:49:51.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:51 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] up:active 2024-04-12T05:49:51.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:51 smithi192 ceph-mon[85304]: mds.? [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] up:active 2024-04-12T05:49:51.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:51 smithi192 ceph-mon[85304]: fsmap cephfs:2 {0=cephfs.smithi113.ovmadh=up:active,1=cephfs.smithi113.ydsbei=up:active} 2 up:standby 2024-04-12T05:49:51.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:51 smithi192 ceph-mon[85304]: pgmap v1133: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 170 B/s wr, 7 op/s 2024-04-12T05:49:52.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.658+0000 7f9072b59700 1 -- 172.21.15.113:0/194733737 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f906c0fc7f0 msgr2=0x7f906c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:52.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.658+0000 7f9072b59700 1 --2- 172.21.15.113:0/194733737 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f906c0fc7f0 0x7f906c0fcbd0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f90600047f0 tx=0x7f9060009cb0).stop 2024-04-12T05:49:52.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.658+0000 7f9072b59700 1 -- 172.21.15.113:0/194733737 shutdown_connections 2024-04-12T05:49:52.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.658+0000 7f9072b59700 1 --2- 172.21.15.113:0/194733737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f906c0fd150 0x7f906c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:52.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.658+0000 7f9072b59700 1 --2- 172.21.15.113:0/194733737 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f906c0fc7f0 0x7f906c0fcbd0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:52.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.658+0000 7f9072b59700 1 -- 172.21.15.113:0/194733737 >> 172.21.15.113:0/194733737 conn(0x7f906c070590 msgr2=0x7f906c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:52.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.658+0000 7f9072b59700 1 -- 172.21.15.113:0/194733737 shutdown_connections 2024-04-12T05:49:52.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.658+0000 7f9072b59700 1 -- 172.21.15.113:0/194733737 wait complete. 2024-04-12T05:49:52.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f9072b59700 1 Processor -- start 2024-04-12T05:49:52.655 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f9072b59700 1 -- start start 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f9072b59700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f906c0fc7f0 0x7f906c0f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f9072b59700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f906c0fd150 0x7f906c0f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f9072b59700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c0f8f20 con 0x7f906c0fc7f0 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f9072b59700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c0f9060 con 0x7f906c0fd150 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f90708f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f906c0fc7f0 0x7f906c0f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f90708f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f906c0fc7f0 0x7f906c0f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55152/0 (socket says 172.21.15.113:55152) 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f90708f5700 1 -- 172.21.15.113:0/57774713 learned_addr learned my addr 172.21.15.113:0/57774713 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f906bfff700 1 --2- 172.21.15.113:0/57774713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f906c0fd150 0x7f906c0f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f90708f5700 1 -- 172.21.15.113:0/57774713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f906c0fd150 msgr2=0x7f906c0f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f90708f5700 1 --2- 172.21.15.113:0/57774713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f906c0fd150 0x7f906c0f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f90708f5700 1 -- 172.21.15.113:0/57774713 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9060005040 con 0x7f906c0fc7f0 2024-04-12T05:49:52.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f90708f5700 1 --2- 172.21.15.113:0/57774713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f906c0fc7f0 0x7f906c0f8540 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f906000b120 tx=0x7f9060009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:52.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.659+0000 7f9069ffb700 1 -- 172.21.15.113:0/57774713 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f906000e030 con 0x7f906c0fc7f0 2024-04-12T05:49:52.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.660+0000 7f9072b59700 1 -- 172.21.15.113:0/57774713 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f906c0f92b0 con 0x7f906c0fc7f0 2024-04-12T05:49:52.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.660+0000 7f9069ffb700 1 -- 172.21.15.113:0/57774713 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9060007e90 con 0x7f906c0fc7f0 2024-04-12T05:49:52.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.660+0000 7f9069ffb700 1 -- 172.21.15.113:0/57774713 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9060018070 con 0x7f906c0fc7f0 2024-04-12T05:49:52.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.660+0000 7f9072b59700 1 -- 172.21.15.113:0/57774713 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f906c1145d0 con 0x7f906c0fc7f0 2024-04-12T05:49:52.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.661+0000 7f9072b59700 1 -- 172.21.15.113:0/57774713 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f906c05f590 con 0x7f906c0fc7f0 2024-04-12T05:49:52.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.661+0000 7f9069ffb700 1 -- 172.21.15.113:0/57774713 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f906001d420 con 0x7f906c0fc7f0 2024-04-12T05:49:52.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.661+0000 7f9069ffb700 1 --2- 172.21.15.113:0/57774713 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f905c06caf0 0x7f905c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:52.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.662+0000 7f9069ffb700 1 -- 172.21.15.113:0/57774713 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f906008d300 con 0x7f906c0fc7f0 2024-04-12T05:49:52.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.662+0000 7f906bfff700 1 --2- 172.21.15.113:0/57774713 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f905c06caf0 0x7f905c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:52.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.662+0000 7f906bfff700 1 --2- 172.21.15.113:0/57774713 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f905c06caf0 0x7f905c06efb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f906c0f9ce0 tx=0x7f9058009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:52.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.666+0000 7f9069ffb700 1 -- 172.21.15.113:0/57774713 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9060057260 con 0x7f906c0fc7f0 2024-04-12T05:49:52.861 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.865+0000 7f9072b59700 1 -- 172.21.15.113:0/57774713 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f906c0fb5a0 con 0x7f905c06caf0 2024-04-12T05:49:52.870 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.875+0000 7f9069ffb700 1 -- 172.21.15.113:0/57774713 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f906c0fb5a0 con 0x7f905c06caf0 2024-04-12T05:49:52.870 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (37m) 8s ago 48m 25.9M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (35m) 8s ago 48m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (35m) 9s ago 48m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (37m) 8s ago 48m 89.8M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (31m) 8s ago 43m 90.8M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (30m) 8s ago 43m 100M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (30m) 9s ago 43m 22.0M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (15m) 9s ago 43m 17.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (39m) 8s ago 50m 513M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (38m) 9s ago 48m 458M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (36m) 8s ago 50m 107M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (35m) 9s ago 48m 62.6M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:49:52.871 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (38m) 8s ago 48m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:49:52.872 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (38m) 9s ago 48m 20.5M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:49:52.872 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (34m) 8s ago 47m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:49:52.872 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (34m) 8s ago 46m 140M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:49:52.872 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (33m) 8s ago 46m 128M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:49:52.872 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (32m) 9s ago 46m 167M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:49:52.872 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (32m) 9s ago 45m 108M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:49:52.872 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (31m) 9s ago 45m 110M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:49:52.872 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (37m) 9s ago 47m 80.4M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:49:52.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.877+0000 7f9072b59700 1 -- 172.21.15.113:0/57774713 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f905c06caf0 msgr2=0x7f905c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:52.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.877+0000 7f9072b59700 1 --2- 172.21.15.113:0/57774713 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f905c06caf0 0x7f905c06efb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f906c0f9ce0 tx=0x7f9058009040).stop 2024-04-12T05:49:52.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.877+0000 7f9072b59700 1 -- 172.21.15.113:0/57774713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f906c0fc7f0 msgr2=0x7f906c0f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:52.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.877+0000 7f9072b59700 1 --2- 172.21.15.113:0/57774713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f906c0fc7f0 0x7f906c0f8540 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f906000b120 tx=0x7f9060009cb0).stop 2024-04-12T05:49:52.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.877+0000 7f9072b59700 1 -- 172.21.15.113:0/57774713 shutdown_connections 2024-04-12T05:49:52.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.877+0000 7f9072b59700 1 --2- 172.21.15.113:0/57774713 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f905c06caf0 0x7f905c06efb0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:52.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.877+0000 7f9072b59700 1 --2- 172.21.15.113:0/57774713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f906c0fc7f0 0x7f906c0f8540 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:52.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.877+0000 7f9072b59700 1 --2- 172.21.15.113:0/57774713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f906c0fd150 0x7f906c0f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:52.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.877+0000 7f9072b59700 1 -- 172.21.15.113:0/57774713 >> 172.21.15.113:0/57774713 conn(0x7f906c070590 msgr2=0x7f906c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:52.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.877+0000 7f9072b59700 1 -- 172.21.15.113:0/57774713 shutdown_connections 2024-04-12T05:49:52.873 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:52.877+0000 7f9072b59700 1 -- 172.21.15.113:0/57774713 wait complete. 2024-04-12T05:49:53.118 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.118 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.118 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.119 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:53.119 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.119 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.119 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:49:53.119 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:49:53.119 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.119 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:49:53.119 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:49:53.119 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:49:53.119 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:49:53.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:49:53.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:49:53.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:49:53.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:49:53.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:49:53.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:49:53.419 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch upgrade status' 2024-04-12T05:49:54.160 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:53 smithi113 ceph-mon[103371]: pgmap v1134: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-12T05:49:54.160 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:53 smithi113 ceph-mon[103371]: from='client.36022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:54.160 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:53 smithi113 ceph-mon[103371]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-12T05:49:54.160 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:53 smithi113 ceph-mon[103371]: Cluster is now healthy 2024-04-12T05:49:54.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:53 smithi192 ceph-mon[85304]: pgmap v1134: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-12T05:49:54.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:53 smithi192 ceph-mon[85304]: from='client.36022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:54.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:53 smithi192 ceph-mon[85304]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-12T05:49:54.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:53 smithi192 ceph-mon[85304]: Cluster is now healthy 2024-04-12T05:49:55.543 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.547+0000 7fde7d603700 1 -- 172.21.15.113:0/436033312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde780fc7f0 msgr2=0x7fde780fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:55.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.547+0000 7fde7d603700 1 --2- 172.21.15.113:0/436033312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde780fc7f0 0x7fde780fcbd0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7fde6c0047f0 tx=0x7fde6c009cb0).stop 2024-04-12T05:49:55.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.547+0000 7fde7d603700 1 -- 172.21.15.113:0/436033312 shutdown_connections 2024-04-12T05:49:55.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.547+0000 7fde7d603700 1 --2- 172.21.15.113:0/436033312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde780fd150 0x7fde780f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:55.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.547+0000 7fde7d603700 1 --2- 172.21.15.113:0/436033312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde780fc7f0 0x7fde780fcbd0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:55.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.547+0000 7fde7d603700 1 -- 172.21.15.113:0/436033312 >> 172.21.15.113:0/436033312 conn(0x7fde78070590 msgr2=0x7fde780f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:55.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.547+0000 7fde7d603700 1 -- 172.21.15.113:0/436033312 shutdown_connections 2024-04-12T05:49:55.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.547+0000 7fde7d603700 1 -- 172.21.15.113:0/436033312 wait complete. 2024-04-12T05:49:55.544 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.548+0000 7fde7d603700 1 Processor -- start 2024-04-12T05:49:55.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.548+0000 7fde7d603700 1 -- start start 2024-04-12T05:49:55.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.548+0000 7fde7d603700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde780fc7f0 0x7fde78109120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:55.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.548+0000 7fde7d603700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde780fd150 0x7fde78109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:55.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.548+0000 7fde7d603700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde78109b00 con 0x7fde780fc7f0 2024-04-12T05:49:55.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.548+0000 7fde7d603700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde78109c40 con 0x7fde780fd150 2024-04-12T05:49:55.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.548+0000 7fde76ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde780fc7f0 0x7fde78109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:55.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde76ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde780fc7f0 0x7fde78109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55170/0 (socket says 172.21.15.113:55170) 2024-04-12T05:49:55.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde76ffd700 1 -- 172.21.15.113:0/3517052057 learned_addr learned my addr 172.21.15.113:0/3517052057 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:49:55.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde767fc700 1 --2- 172.21.15.113:0/3517052057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde780fd150 0x7fde78109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:55.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde76ffd700 1 -- 172.21.15.113:0/3517052057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde780fd150 msgr2=0x7fde78109610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:55.546 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde76ffd700 1 --2- 172.21.15.113:0/3517052057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde780fd150 0x7fde78109610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:55.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde76ffd700 1 -- 172.21.15.113:0/3517052057 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c005040 con 0x7fde780fc7f0 2024-04-12T05:49:55.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde76ffd700 1 --2- 172.21.15.113:0/3517052057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde780fc7f0 0x7fde78109120 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7fde6c00b120 tx=0x7fde6c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:55.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde5ffff700 1 -- 172.21.15.113:0/3517052057 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6c00e030 con 0x7fde780fc7f0 2024-04-12T05:49:55.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde5ffff700 1 -- 172.21.15.113:0/3517052057 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde6c007e90 con 0x7fde780fc7f0 2024-04-12T05:49:55.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde7d603700 1 -- 172.21.15.113:0/3517052057 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde78109e90 con 0x7fde780fc7f0 2024-04-12T05:49:55.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde5ffff700 1 -- 172.21.15.113:0/3517052057 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6c018070 con 0x7fde780fc7f0 2024-04-12T05:49:55.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.549+0000 7fde7d603700 1 -- 172.21.15.113:0/3517052057 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde78112480 con 0x7fde780fc7f0 2024-04-12T05:49:55.547 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.550+0000 7fde7d603700 1 -- 172.21.15.113:0/3517052057 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7805f590 con 0x7fde780fc7f0 2024-04-12T05:49:55.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.551+0000 7fde5ffff700 1 -- 172.21.15.113:0/3517052057 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fde6c01d420 con 0x7fde780fc7f0 2024-04-12T05:49:55.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.551+0000 7fde5ffff700 1 --2- 172.21.15.113:0/3517052057 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fde6006caa0 0x7fde6006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:55.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.551+0000 7fde5ffff700 1 -- 172.21.15.113:0/3517052057 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fde6c08d240 con 0x7fde780fc7f0 2024-04-12T05:49:55.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.551+0000 7fde767fc700 1 --2- 172.21.15.113:0/3517052057 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fde6006caa0 0x7fde6006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:55.548 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.551+0000 7fde767fc700 1 --2- 172.21.15.113:0/3517052057 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fde6006caa0 0x7fde6006ef60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fde78060ee0 tx=0x7fde68009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:55.551 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.555+0000 7fde5ffff700 1 -- 172.21.15.113:0/3517052057 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fde6c057260 con 0x7fde780fc7f0 2024-04-12T05:49:55.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.762+0000 7fde7d603700 1 -- 172.21.15.113:0/3517052057 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde78059120 con 0x7fde6006caa0 2024-04-12T05:49:55.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.763+0000 7fde5ffff700 1 -- 172.21.15.113:0/3517052057 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fde78059120 con 0x7fde6006caa0 2024-04-12T05:49:55.759 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:49:55.759 INFO:teuthology.orchestra.run.smithi113.stdout: "target_image": null, 2024-04-12T05:49:55.759 INFO:teuthology.orchestra.run.smithi113.stdout: "in_progress": false, 2024-04-12T05:49:55.759 INFO:teuthology.orchestra.run.smithi113.stdout: "which": "", 2024-04-12T05:49:55.759 INFO:teuthology.orchestra.run.smithi113.stdout: "services_complete": [], 2024-04-12T05:49:55.759 INFO:teuthology.orchestra.run.smithi113.stdout: "progress": null, 2024-04-12T05:49:55.759 INFO:teuthology.orchestra.run.smithi113.stdout: "message": "", 2024-04-12T05:49:55.759 INFO:teuthology.orchestra.run.smithi113.stdout: "is_paused": false 2024-04-12T05:49:55.759 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:49:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.765+0000 7fde7d603700 1 -- 172.21.15.113:0/3517052057 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fde6006caa0 msgr2=0x7fde6006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.765+0000 7fde7d603700 1 --2- 172.21.15.113:0/3517052057 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fde6006caa0 0x7fde6006ef60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fde78060ee0 tx=0x7fde68009040).stop 2024-04-12T05:49:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.765+0000 7fde7d603700 1 -- 172.21.15.113:0/3517052057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde780fc7f0 msgr2=0x7fde78109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.765+0000 7fde7d603700 1 --2- 172.21.15.113:0/3517052057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde780fc7f0 0x7fde78109120 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7fde6c00b120 tx=0x7fde6c009cb0).stop 2024-04-12T05:49:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.765+0000 7fde7d603700 1 -- 172.21.15.113:0/3517052057 shutdown_connections 2024-04-12T05:49:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.765+0000 7fde7d603700 1 --2- 172.21.15.113:0/3517052057 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fde6006caa0 0x7fde6006ef60 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:55.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.765+0000 7fde7d603700 1 --2- 172.21.15.113:0/3517052057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde780fc7f0 0x7fde78109120 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:55.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.765+0000 7fde7d603700 1 --2- 172.21.15.113:0/3517052057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde780fd150 0x7fde78109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:55.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.765+0000 7fde7d603700 1 -- 172.21.15.113:0/3517052057 >> 172.21.15.113:0/3517052057 conn(0x7fde78070590 msgr2=0x7fde780fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:55.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.765+0000 7fde7d603700 1 -- 172.21.15.113:0/3517052057 shutdown_connections 2024-04-12T05:49:55.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:55.765+0000 7fde7d603700 1 -- 172.21.15.113:0/3517052057 wait complete. 2024-04-12T05:49:56.108 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:55 smithi113 ceph-mon[103371]: pgmap v1135: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-04-12T05:49:56.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:55 smithi192 ceph-mon[85304]: pgmap v1135: 65 pgs: 65 active+clean; 259 MiB data, 1003 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-04-12T05:49:56.359 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph health detail' 2024-04-12T05:49:57.168 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:56 smithi113 ceph-mon[103371]: from='client.36026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:57.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:56 smithi192 ceph-mon[85304]: from='client.36026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:49:57.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:57 smithi113 ceph-mon[103371]: pgmap v1136: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 14 MiB/s rd, 8.8 KiB/s wr, 14 op/s 2024-04-12T05:49:58.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.129+0000 7f5dd3fce700 1 -- 172.21.15.113:0/2538212368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc0fe250 msgr2=0x7f5dcc0f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:58.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.129+0000 7f5dd3fce700 1 --2- 172.21.15.113:0/2538212368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc0fe250 0x7f5dcc0f7a30 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f5dc80047f0 tx=0x7f5dc8009cb0).stop 2024-04-12T05:49:58.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.130+0000 7f5dd3fce700 1 -- 172.21.15.113:0/2538212368 shutdown_connections 2024-04-12T05:49:58.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.130+0000 7f5dd3fce700 1 --2- 172.21.15.113:0/2538212368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc0fe250 0x7f5dcc0f7a30 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:58.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.130+0000 7f5dd3fce700 1 --2- 172.21.15.113:0/2538212368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dcc0fd8f0 0x7f5dcc0fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:58.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.130+0000 7f5dd3fce700 1 -- 172.21.15.113:0/2538212368 >> 172.21.15.113:0/2538212368 conn(0x7f5dcc0f34b0 msgr2=0x7f5dcc0f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:58.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.130+0000 7f5dd3fce700 1 -- 172.21.15.113:0/2538212368 shutdown_connections 2024-04-12T05:49:58.126 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.130+0000 7f5dd3fce700 1 -- 172.21.15.113:0/2538212368 wait complete. 2024-04-12T05:49:58.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.130+0000 7f5dd3fce700 1 Processor -- start 2024-04-12T05:49:58.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.130+0000 7f5dd3fce700 1 -- start start 2024-04-12T05:49:58.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd3fce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dcc0fd8f0 0x7f5dcc10e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:58.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd3fce700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc0fe250 0x7f5dcc10e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:58.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd3fce700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc10eb70 con 0x7f5dcc0fe250 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd3fce700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc10ecb0 con 0x7f5dcc0fd8f0 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd1569700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc0fe250 0x7f5dcc10e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd1569700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc0fe250 0x7f5dcc10e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:55186/0 (socket says 172.21.15.113:55186) 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd1569700 1 -- 172.21.15.113:0/1788469471 learned_addr learned my addr 172.21.15.113:0/1788469471 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd1d6a700 1 --2- 172.21.15.113:0/1788469471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dcc0fd8f0 0x7f5dcc10e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd1569700 1 -- 172.21.15.113:0/1788469471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dcc0fd8f0 msgr2=0x7f5dcc10e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd1569700 1 --2- 172.21.15.113:0/1788469471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dcc0fd8f0 0x7f5dcc10e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd1569700 1 -- 172.21.15.113:0/1788469471 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8005040 con 0x7f5dcc0fe250 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd1569700 1 --2- 172.21.15.113:0/1788469471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc0fe250 0x7f5dcc10e570 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7f5dc8000c00 tx=0x7f5dc8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dc2ffd700 1 -- 172.21.15.113:0/1788469471 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dc800e030 con 0x7f5dcc0fe250 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dc2ffd700 1 -- 172.21.15.113:0/1788469471 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5dc8007e90 con 0x7f5dcc0fe250 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dc2ffd700 1 -- 172.21.15.113:0/1788469471 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dc8018070 con 0x7f5dcc0fe250 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.131+0000 7f5dd3fce700 1 -- 172.21.15.113:0/1788469471 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dcc1082e0 con 0x7f5dcc0fe250 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.132+0000 7f5dd3fce700 1 -- 172.21.15.113:0/1788469471 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dcc1087c0 con 0x7f5dcc0fe250 2024-04-12T05:49:58.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.132+0000 7f5dd3fce700 1 -- 172.21.15.113:0/1788469471 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dcc05f590 con 0x7f5dcc0fe250 2024-04-12T05:49:58.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.133+0000 7f5dc2ffd700 1 -- 172.21.15.113:0/1788469471 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5dc801d420 con 0x7f5dcc0fe250 2024-04-12T05:49:58.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.133+0000 7f5dc2ffd700 1 --2- 172.21.15.113:0/1788469471 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5db806cab0 0x7f5db806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:49:58.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.133+0000 7f5dc2ffd700 1 -- 172.21.15.113:0/1788469471 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5dc808cfe0 con 0x7f5dcc0fe250 2024-04-12T05:49:58.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.133+0000 7f5dd1d6a700 1 --2- 172.21.15.113:0/1788469471 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5db806cab0 0x7f5db806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:49:58.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.134+0000 7f5dd1d6a700 1 --2- 172.21.15.113:0/1788469471 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5db806cab0 0x7f5db806ef70 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f5dbc004dc0 tx=0x7f5dbc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:49:58.133 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.138+0000 7f5dc2ffd700 1 -- 172.21.15.113:0/1788469471 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5dc805aa90 con 0x7f5dcc0fe250 2024-04-12T05:49:58.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:57 smithi192 ceph-mon[85304]: pgmap v1136: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 14 MiB/s rd, 8.8 KiB/s wr, 14 op/s 2024-04-12T05:49:58.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.409+0000 7f5dd3fce700 1 -- 172.21.15.113:0/1788469471 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5dcc047540 con 0x7f5dcc0fe250 2024-04-12T05:49:58.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.409+0000 7f5dc2ffd700 1 -- 172.21.15.113:0/1788469471 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f5dc805a620 con 0x7f5dcc0fe250 2024-04-12T05:49:58.406 INFO:teuthology.orchestra.run.smithi113.stdout:HEALTH_OK 2024-04-12T05:49:58.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.411+0000 7f5dd3fce700 1 -- 172.21.15.113:0/1788469471 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5db806cab0 msgr2=0x7f5db806ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:58.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.411+0000 7f5dd3fce700 1 --2- 172.21.15.113:0/1788469471 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5db806cab0 0x7f5db806ef70 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f5dbc004dc0 tx=0x7f5dbc00b040).stop 2024-04-12T05:49:58.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.411+0000 7f5dd3fce700 1 -- 172.21.15.113:0/1788469471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc0fe250 msgr2=0x7f5dcc10e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:49:58.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.411+0000 7f5dd3fce700 1 --2- 172.21.15.113:0/1788469471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc0fe250 0x7f5dcc10e570 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7f5dc8000c00 tx=0x7f5dc8009cb0).stop 2024-04-12T05:49:58.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.411+0000 7f5dd3fce700 1 -- 172.21.15.113:0/1788469471 shutdown_connections 2024-04-12T05:49:58.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.411+0000 7f5dd3fce700 1 --2- 172.21.15.113:0/1788469471 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5db806cab0 0x7f5db806ef70 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:58.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.411+0000 7f5dd3fce700 1 --2- 172.21.15.113:0/1788469471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dcc0fd8f0 0x7f5dcc10e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:58.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.411+0000 7f5dd3fce700 1 --2- 172.21.15.113:0/1788469471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc0fe250 0x7f5dcc10e570 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:49:58.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.411+0000 7f5dd3fce700 1 -- 172.21.15.113:0/1788469471 >> 172.21.15.113:0/1788469471 conn(0x7f5dcc0f34b0 msgr2=0x7f5dcc0fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:49:58.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.412+0000 7f5dd3fce700 1 -- 172.21.15.113:0/1788469471 shutdown_connections 2024-04-12T05:49:58.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:49:58.412+0000 7f5dd3fce700 1 -- 172.21.15.113:0/1788469471 wait complete. 2024-04-12T05:49:59.045 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:58 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1788469471' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:49:59.065 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions' 2024-04-12T05:49:59.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:58 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1788469471' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T05:50:00.137 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:49:59 smithi113 ceph-mon[103371]: pgmap v1137: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 716 KiB/s rd, 8.8 KiB/s wr, 10 op/s 2024-04-12T05:50:00.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:49:59 smithi192 ceph-mon[85304]: pgmap v1137: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 716 KiB/s rd, 8.8 KiB/s wr, 10 op/s 2024-04-12T05:50:00.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.906+0000 7fae9921c700 1 -- 172.21.15.113:0/4153126105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae940fe250 msgr2=0x7fae940f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:00.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.906+0000 7fae9921c700 1 --2- 172.21.15.113:0/4153126105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae940fe250 0x7fae940f7a30 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7fae880047f0 tx=0x7fae88009cb0).stop 2024-04-12T05:50:00.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.906+0000 7fae9921c700 1 -- 172.21.15.113:0/4153126105 shutdown_connections 2024-04-12T05:50:00.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.906+0000 7fae9921c700 1 --2- 172.21.15.113:0/4153126105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae940fe250 0x7fae940f7a30 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:50:00.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.906+0000 7fae9921c700 1 --2- 172.21.15.113:0/4153126105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae940fd8f0 0x7fae940fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:50:00.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.906+0000 7fae9921c700 1 -- 172.21.15.113:0/4153126105 >> 172.21.15.113:0/4153126105 conn(0x7fae940f34b0 msgr2=0x7fae940f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:00.908 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.906+0000 7fae9921c700 1 -- 172.21.15.113:0/4153126105 shutdown_connections 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.906+0000 7fae9921c700 1 -- 172.21.15.113:0/4153126105 wait complete. 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.907+0000 7fae9921c700 1 Processor -- start 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.907+0000 7fae9921c700 1 -- start start 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.907+0000 7fae9921c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae940fd8f0 0x7fae9410e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.907+0000 7fae9921c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae940fe250 0x7fae9410e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.907+0000 7fae9921c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9410eb70 con 0x7fae940fe250 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae9921c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9410ecb0 con 0x7fae940fd8f0 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae9259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae940fe250 0x7fae9410e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae9259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae940fe250 0x7fae9410e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53878/0 (socket says 172.21.15.113:53878) 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae9259c700 1 -- 172.21.15.113:0/3522486986 learned_addr learned my addr 172.21.15.113:0/3522486986 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae92d9d700 1 --2- 172.21.15.113:0/3522486986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae940fd8f0 0x7fae9410e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:00.909 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae9259c700 1 -- 172.21.15.113:0/3522486986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae940fd8f0 msgr2=0x7fae9410e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae9259c700 1 --2- 172.21.15.113:0/3522486986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae940fd8f0 0x7fae9410e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae9259c700 1 -- 172.21.15.113:0/3522486986 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae88005040 con 0x7fae940fe250 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae9259c700 1 --2- 172.21.15.113:0/3522486986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae940fe250 0x7fae9410e570 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7fae880099f0 tx=0x7fae88009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae7bfff700 1 -- 172.21.15.113:0/3522486986 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae8800e030 con 0x7fae940fe250 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae7bfff700 1 -- 172.21.15.113:0/3522486986 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae88007e90 con 0x7fae940fe250 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae9921c700 1 -- 172.21.15.113:0/3522486986 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae941082e0 con 0x7fae940fe250 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae7bfff700 1 -- 172.21.15.113:0/3522486986 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae88018070 con 0x7fae940fe250 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.908+0000 7fae9921c700 1 -- 172.21.15.113:0/3522486986 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae941087c0 con 0x7fae940fe250 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.909+0000 7fae9921c700 1 -- 172.21.15.113:0/3522486986 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9405f590 con 0x7fae940fe250 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.910+0000 7fae7bfff700 1 -- 172.21.15.113:0/3522486986 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fae8801d420 con 0x7fae940fe250 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.910+0000 7fae7bfff700 1 --2- 172.21.15.113:0/3522486986 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fae7c06caa0 0x7fae7c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:50:00.910 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.910+0000 7fae92d9d700 1 --2- 172.21.15.113:0/3522486986 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fae7c06caa0 0x7fae7c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:50:00.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.910+0000 7fae7bfff700 1 -- 172.21.15.113:0/3522486986 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fae8808d2f0 con 0x7fae940fe250 2024-04-12T05:50:00.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.911+0000 7fae92d9d700 1 --2- 172.21.15.113:0/3522486986 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fae7c06caa0 0x7fae7c06ef60 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fae84004b60 tx=0x7fae84009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:50:00.911 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:00.915+0000 7fae7bfff700 1 -- 172.21.15.113:0/3522486986 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fae8805ada0 con 0x7fae940fe250 2024-04-12T05:50:00.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:00 smithi113 ceph-mon[103371]: overall HEALTH_OK 2024-04-12T05:50:01.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.186+0000 7fae9921c700 1 -- 172.21.15.113:0/3522486986 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fae94047540 con 0x7fae940fe250 2024-04-12T05:50:01.182 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.186+0000 7fae7bfff700 1 -- 172.21.15.113:0/3522486986 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fae8805a930 con 0x7fae940fe250 2024-04-12T05:50:01.182 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:50:01.182 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:50:01.182 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:50:01.183 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:50:01.184 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:50:01.184 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:50:01.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.189+0000 7fae9921c700 1 -- 172.21.15.113:0/3522486986 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fae7c06caa0 msgr2=0x7fae7c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:01.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.189+0000 7fae9921c700 1 --2- 172.21.15.113:0/3522486986 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fae7c06caa0 0x7fae7c06ef60 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fae84004b60 tx=0x7fae84009040).stop 2024-04-12T05:50:01.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.189+0000 7fae9921c700 1 -- 172.21.15.113:0/3522486986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae940fe250 msgr2=0x7fae9410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:50:01.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.189+0000 7fae9921c700 1 --2- 172.21.15.113:0/3522486986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae940fe250 0x7fae9410e570 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7fae880099f0 tx=0x7fae88009cb0).stop 2024-04-12T05:50:01.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.189+0000 7fae9921c700 1 -- 172.21.15.113:0/3522486986 shutdown_connections 2024-04-12T05:50:01.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.189+0000 7fae9921c700 1 --2- 172.21.15.113:0/3522486986 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fae7c06caa0 0x7fae7c06ef60 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:50:01.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.189+0000 7fae9921c700 1 --2- 172.21.15.113:0/3522486986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae940fd8f0 0x7fae9410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:50:01.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.189+0000 7fae9921c700 1 --2- 172.21.15.113:0/3522486986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae940fe250 0x7fae9410e570 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:50:01.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.189+0000 7fae9921c700 1 -- 172.21.15.113:0/3522486986 >> 172.21.15.113:0/3522486986 conn(0x7fae940f34b0 msgr2=0x7fae940fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:50:01.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.189+0000 7fae9921c700 1 -- 172.21.15.113:0/3522486986 shutdown_connections 2024-04-12T05:50:01.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:50:01.189+0000 7fae9921c700 1 -- 172.21.15.113:0/3522486986 wait complete. 2024-04-12T05:50:01.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:00 smithi192 ceph-mon[85304]: overall HEALTH_OK 2024-04-12T05:50:01.839 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-12T05:50:02.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:01 smithi113 ceph-mon[103371]: pgmap v1138: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 8.8 KiB/s wr, 10 op/s 2024-04-12T05:50:02.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:01 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3522486986' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:50:02.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:01 smithi192 ceph-mon[85304]: pgmap v1138: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 8.8 KiB/s wr, 10 op/s 2024-04-12T05:50:02.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:01 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3522486986' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:50:03.680 INFO:teuthology.orchestra.run.smithi113.stdout:wait for servicemap items w/ changing names to refresh 2024-04-12T05:50:04.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:03 smithi113 ceph-mon[103371]: pgmap v1139: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 8.7 KiB/s wr, 10 op/s 2024-04-12T05:50:04.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:03 smithi192 ceph-mon[85304]: pgmap v1139: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 8.7 KiB/s wr, 10 op/s 2024-04-12T05:50:04.259 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'sleep 60' 2024-04-12T05:50:06.168 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:05 smithi113 ceph-mon[103371]: pgmap v1140: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 5.2 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2024-04-12T05:50:06.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:05 smithi192 ceph-mon[85304]: pgmap v1140: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 5.2 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2024-04-12T05:50:08.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:07 smithi113 ceph-mon[103371]: pgmap v1141: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 3.5 KiB/s rd, 6.2 KiB/s wr, 6 op/s 2024-04-12T05:50:08.168 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:50:08.168 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:08.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:07 smithi192 ceph-mon[85304]: pgmap v1141: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail; 3.5 KiB/s rd, 6.2 KiB/s wr, 6 op/s 2024-04-12T05:50:08.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:50:08.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:10.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:09 smithi113 ceph-mon[103371]: pgmap v1142: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:10.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:09 smithi192 ceph-mon[85304]: pgmap v1142: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:12.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:11 smithi113 ceph-mon[103371]: pgmap v1143: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:12.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:11 smithi192 ceph-mon[85304]: pgmap v1143: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:14.168 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:13 smithi113 ceph-mon[103371]: pgmap v1144: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:14.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:13 smithi192 ceph-mon[85304]: pgmap v1144: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:16.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:15 smithi113 ceph-mon[103371]: pgmap v1145: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:16.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:15 smithi192 ceph-mon[85304]: pgmap v1145: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:18.168 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:17 smithi113 ceph-mon[103371]: pgmap v1146: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:18.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:17 smithi192 ceph-mon[85304]: pgmap v1146: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:20.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:19 smithi113 ceph-mon[103371]: pgmap v1147: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:20.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:19 smithi192 ceph-mon[85304]: pgmap v1147: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:22.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:21 smithi113 ceph-mon[103371]: pgmap v1148: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:22.168 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:22.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:21 smithi192 ceph-mon[85304]: pgmap v1148: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:22.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:24.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:23 smithi113 ceph-mon[103371]: pgmap v1149: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:24.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:23 smithi192 ceph-mon[85304]: pgmap v1149: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:26.168 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:25 smithi113 ceph-mon[103371]: pgmap v1150: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:26.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:25 smithi192 ceph-mon[85304]: pgmap v1150: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:28.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:27 smithi113 ceph-mon[103371]: pgmap v1151: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:28.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:27 smithi192 ceph-mon[85304]: pgmap v1151: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:30.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:29 smithi113 ceph-mon[103371]: pgmap v1152: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:30.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:29 smithi192 ceph-mon[85304]: pgmap v1152: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:32.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:31 smithi113 ceph-mon[103371]: pgmap v1153: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:32.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:31 smithi192 ceph-mon[85304]: pgmap v1153: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:34.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:33 smithi113 ceph-mon[103371]: pgmap v1154: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:34.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:33 smithi192 ceph-mon[85304]: pgmap v1154: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:36.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:35 smithi113 ceph-mon[103371]: pgmap v1155: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:36.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:35 smithi192 ceph-mon[85304]: pgmap v1155: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:37.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:36 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:37.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:36 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:38.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:37 smithi113 ceph-mon[103371]: pgmap v1156: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:38.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:37 smithi192 ceph-mon[85304]: pgmap v1156: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:40.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:39 smithi113 ceph-mon[103371]: pgmap v1157: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:40.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:39 smithi192 ceph-mon[85304]: pgmap v1157: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:42.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:41 smithi192 ceph-mon[85304]: pgmap v1158: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:42.262 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:41 smithi113 ceph-mon[103371]: pgmap v1158: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:44.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:43 smithi192 ceph-mon[85304]: pgmap v1159: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:44.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:43 smithi113 ceph-mon[103371]: pgmap v1159: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:46.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:45 smithi192 ceph-mon[85304]: pgmap v1160: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:46.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:45 smithi113 ceph-mon[103371]: pgmap v1160: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:48.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:47 smithi192 ceph-mon[85304]: pgmap v1161: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:48.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:47 smithi113 ceph-mon[103371]: pgmap v1161: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:50.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:49 smithi192 ceph-mon[85304]: pgmap v1162: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:50.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:49 smithi113 ceph-mon[103371]: pgmap v1162: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:52.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:51 smithi192 ceph-mon[85304]: pgmap v1163: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:52.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:51 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:52.262 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:51 smithi113 ceph-mon[103371]: pgmap v1163: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:52.262 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:51 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:50:53.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:50:53.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:50:54.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:53 smithi192 ceph-mon[85304]: pgmap v1164: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:54.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:50:54.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:50:54.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:50:54.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:53 smithi113 ceph-mon[103371]: pgmap v1164: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:54.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:50:54.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:50:54.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:50:56.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:55 smithi113 ceph-mon[103371]: pgmap v1165: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:56.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:55 smithi192 ceph-mon[85304]: pgmap v1165: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:58.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:57 smithi113 ceph-mon[103371]: pgmap v1166: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:50:58.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:57 smithi192 ceph-mon[85304]: pgmap v1166: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:00.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:50:59 smithi113 ceph-mon[103371]: pgmap v1167: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:00.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:50:59 smithi192 ceph-mon[85304]: pgmap v1167: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:02.264 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:01 smithi113 ceph-mon[103371]: pgmap v1168: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:02.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:01 smithi192 ceph-mon[85304]: pgmap v1168: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:04.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:03 smithi113 ceph-mon[103371]: pgmap v1169: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:04.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:03 smithi192 ceph-mon[85304]: pgmap v1169: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:06.280 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:05 smithi113 ceph-mon[103371]: pgmap v1170: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:06.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:05 smithi192 ceph-mon[85304]: pgmap v1170: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:06.593 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph orch ps' 2024-04-12T05:51:07.318 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:06 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:07.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:06 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:08.092 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:07 smithi113 ceph-mon[103371]: pgmap v1171: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:08.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:07 smithi192 ceph-mon[85304]: pgmap v1171: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:08.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.589+0000 7f3c76e3f700 1 -- 172.21.15.113:0/3294228273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f6fc0 msgr2=0x7f3c70103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:08.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.589+0000 7f3c76e3f700 1 --2- 172.21.15.113:0/3294228273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f6fc0 0x7f3c70103d60 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7f3c5c004820 tx=0x7f3c5c009cb0).stop 2024-04-12T05:51:08.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.589+0000 7f3c76e3f700 1 -- 172.21.15.113:0/3294228273 shutdown_connections 2024-04-12T05:51:08.586 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.589+0000 7f3c76e3f700 1 --2- 172.21.15.113:0/3294228273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f6fc0 0x7f3c70103d60 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.589+0000 7f3c76e3f700 1 --2- 172.21.15.113:0/3294228273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f66f0 0x7f3c700f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.589+0000 7f3c76e3f700 1 -- 172.21.15.113:0/3294228273 >> 172.21.15.113:0/3294228273 conn(0x7f3c70070590 msgr2=0x7f3c700f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.589+0000 7f3c76e3f700 1 -- 172.21.15.113:0/3294228273 shutdown_connections 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.589+0000 7f3c76e3f700 1 -- 172.21.15.113:0/3294228273 wait complete. 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.589+0000 7f3c76e3f700 1 Processor -- start 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.589+0000 7f3c76e3f700 1 -- start start 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c76e3f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f66f0 0x7f3c70110350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c76e3f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f6fc0 0x7f3c70110840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c76e3f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70110e40 con 0x7f3c700f6fc0 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c76e3f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70110f80 con 0x7f3c700f66f0 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c6ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f6fc0 0x7f3c70110840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:08.587 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c6ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f6fc0 0x7f3c70110840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36780/0 (socket says 172.21.15.113:36780) 2024-04-12T05:51:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c6ffff700 1 -- 172.21.15.113:0/724425505 learned_addr learned my addr 172.21.15.113:0/724425505 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c74bdb700 1 --2- 172.21.15.113:0/724425505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f66f0 0x7f3c70110350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c6ffff700 1 -- 172.21.15.113:0/724425505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f66f0 msgr2=0x7f3c70110350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c6ffff700 1 --2- 172.21.15.113:0/724425505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f66f0 0x7f3c70110350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c6ffff700 1 -- 172.21.15.113:0/724425505 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c005040 con 0x7f3c700f6fc0 2024-04-12T05:51:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c6ffff700 1 --2- 172.21.15.113:0/724425505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f6fc0 0x7f3c70110840 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f3c5c005010 tx=0x7f3c5c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c6dffb700 1 -- 172.21.15.113:0/724425505 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c5c00e030 con 0x7f3c700f6fc0 2024-04-12T05:51:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c6dffb700 1 -- 172.21.15.113:0/724425505 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c5c007e90 con 0x7f3c700f6fc0 2024-04-12T05:51:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c76e3f700 1 -- 172.21.15.113:0/724425505 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7010a5b0 con 0x7f3c700f6fc0 2024-04-12T05:51:08.588 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.590+0000 7f3c6dffb700 1 -- 172.21.15.113:0/724425505 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c5c018070 con 0x7f3c700f6fc0 2024-04-12T05:51:08.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.591+0000 7f3c76e3f700 1 -- 172.21.15.113:0/724425505 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7010aa90 con 0x7f3c700f6fc0 2024-04-12T05:51:08.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.591+0000 7f3c76e3f700 1 -- 172.21.15.113:0/724425505 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7005f590 con 0x7f3c700f6fc0 2024-04-12T05:51:08.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.592+0000 7f3c6dffb700 1 -- 172.21.15.113:0/724425505 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3c5c01d420 con 0x7f3c700f6fc0 2024-04-12T05:51:08.589 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.592+0000 7f3c6dffb700 1 --2- 172.21.15.113:0/724425505 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6006caa0 0x7f3c6006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:08.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.592+0000 7f3c6dffb700 1 -- 172.21.15.113:0/724425505 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f3c5c08d020 con 0x7f3c700f6fc0 2024-04-12T05:51:08.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.592+0000 7f3c74bdb700 1 --2- 172.21.15.113:0/724425505 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6006caa0 0x7f3c6006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:08.590 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.593+0000 7f3c74bdb700 1 --2- 172.21.15.113:0/724425505 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6006caa0 0x7f3c6006ef60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f3c64004de0 tx=0x7f3c6400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:08.592 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.597+0000 7f3c6dffb700 1 -- 172.21.15.113:0/724425505 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f3c5c05aad0 con 0x7f3c700f6fc0 2024-04-12T05:51:08.790 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.794+0000 7f3c76e3f700 1 -- 172.21.15.113:0/724425505 --> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c7010b9d0 con 0x7f3c6006caa0 2024-04-12T05:51:08.799 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.804+0000 7f3c6dffb700 1 -- 172.21.15.113:0/724425505 <== mgr.34104 v2:172.21.15.113:6800/2746473633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f3c7010b9d0 con 0x7f3c6006caa0 2024-04-12T05:51:08.800 INFO:teuthology.orchestra.run.smithi113.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T05:51:08.800 INFO:teuthology.orchestra.run.smithi113.stdout:alertmanager.smithi113 smithi113 *:9093,9094 running (38m) 84s ago 50m 25.9M - 0.25.0 c8568f914cd2 16d15ab83da9 2024-04-12T05:51:08.800 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi113 smithi113 running (36m) 84s ago 50m 6660k - 18.2.2-1187-gd540ebac bb2c5eac26a4 bc6c0642f0d4 2024-04-12T05:51:08.800 INFO:teuthology.orchestra.run.smithi113.stdout:crash.smithi192 smithi192 running (36m) 85s ago 49m 6656k - 18.2.2-1187-gd540ebac bb2c5eac26a4 be534db9d5b4 2024-04-12T05:51:08.800 INFO:teuthology.orchestra.run.smithi113.stdout:grafana.smithi113 smithi113 *:3000 running (38m) 84s ago 49m 89.8M - 9.4.7 954c08fa6188 6c6950680d96 2024-04-12T05:51:08.800 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ovmadh smithi113 running (32m) 84s ago 44m 90.8M - 18.2.2-1187-gd540ebac bb2c5eac26a4 e136faaf7adc 2024-04-12T05:51:08.800 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi113.ydsbei smithi113 running (31m) 84s ago 44m 100M - 18.2.2-1187-gd540ebac bb2c5eac26a4 be231d7c4af4 2024-04-12T05:51:08.800 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.ammjap smithi192 running (31m) 85s ago 44m 22.0M - 18.2.2-1187-gd540ebac bb2c5eac26a4 9734925df51e 2024-04-12T05:51:08.800 INFO:teuthology.orchestra.run.smithi113.stdout:mds.cephfs.smithi192.dtnaoi smithi192 running (16m) 85s ago 44m 17.9M - 18.2.2-1187-gd540ebac bb2c5eac26a4 0bdc0ab52a4c 2024-04-12T05:51:08.800 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi113.ctsazg smithi113 *:8443,9283,8765 running (40m) 84s ago 51m 513M - 18.2.2-1187-gd540ebac bb2c5eac26a4 3c801e7ed16f 2024-04-12T05:51:08.800 INFO:teuthology.orchestra.run.smithi113.stdout:mgr.smithi192.ocppyo smithi192 *:8443,9283,8765 running (39m) 85s ago 49m 458M - 18.2.2-1187-gd540ebac bb2c5eac26a4 f879ce427917 2024-04-12T05:51:08.801 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi113 smithi113 running (37m) 84s ago 51m 107M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 98cfc95ea472 2024-04-12T05:51:08.801 INFO:teuthology.orchestra.run.smithi113.stdout:mon.smithi192 smithi192 running (37m) 85s ago 49m 62.6M 2048M 18.2.2-1187-gd540ebac bb2c5eac26a4 e99449ab860c 2024-04-12T05:51:08.801 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi113 smithi113 *:9100 running (39m) 84s ago 49m 17.0M - 1.5.0 0da6a335fe13 c18f7dcab3e7 2024-04-12T05:51:08.801 INFO:teuthology.orchestra.run.smithi113.stdout:node-exporter.smithi192 smithi192 *:9100 running (39m) 85s ago 49m 20.5M - 1.5.0 0da6a335fe13 57f885811346 2024-04-12T05:51:08.801 INFO:teuthology.orchestra.run.smithi113.stdout:osd.0 smithi113 running (35m) 84s ago 48m 123M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 03af9c08ee59 2024-04-12T05:51:08.801 INFO:teuthology.orchestra.run.smithi113.stdout:osd.1 smithi113 running (35m) 84s ago 48m 140M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 cad088ac8df1 2024-04-12T05:51:08.801 INFO:teuthology.orchestra.run.smithi113.stdout:osd.2 smithi113 running (34m) 84s ago 47m 128M 1910M 18.2.2-1187-gd540ebac bb2c5eac26a4 70ada4b6cb09 2024-04-12T05:51:08.801 INFO:teuthology.orchestra.run.smithi113.stdout:osd.3 smithi192 running (34m) 85s ago 47m 167M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 537f5471b31d 2024-04-12T05:51:08.801 INFO:teuthology.orchestra.run.smithi113.stdout:osd.4 smithi192 running (33m) 85s ago 46m 108M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 0bcafbab30e6 2024-04-12T05:51:08.801 INFO:teuthology.orchestra.run.smithi113.stdout:osd.5 smithi192 running (32m) 85s ago 46m 110M 2251M 18.2.2-1187-gd540ebac bb2c5eac26a4 fbe8b39c4b06 2024-04-12T05:51:08.801 INFO:teuthology.orchestra.run.smithi113.stdout:prometheus.smithi192 smithi192 *:9095 running (39m) 85s ago 49m 80.4M - 2.43.0 a07b618ecd1d 5557507f7727 2024-04-12T05:51:08.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.806+0000 7f3c76e3f700 1 -- 172.21.15.113:0/724425505 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6006caa0 msgr2=0x7f3c6006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:08.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.806+0000 7f3c76e3f700 1 --2- 172.21.15.113:0/724425505 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6006caa0 0x7f3c6006ef60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f3c64004de0 tx=0x7f3c6400b040).stop 2024-04-12T05:51:08.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.806+0000 7f3c76e3f700 1 -- 172.21.15.113:0/724425505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f6fc0 msgr2=0x7f3c70110840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:08.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.806+0000 7f3c76e3f700 1 --2- 172.21.15.113:0/724425505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f6fc0 0x7f3c70110840 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f3c5c005010 tx=0x7f3c5c009cb0).stop 2024-04-12T05:51:08.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.806+0000 7f3c76e3f700 1 -- 172.21.15.113:0/724425505 shutdown_connections 2024-04-12T05:51:08.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.806+0000 7f3c76e3f700 1 --2- 172.21.15.113:0/724425505 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f3c6006caa0 0x7f3c6006ef60 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:08.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.806+0000 7f3c76e3f700 1 --2- 172.21.15.113:0/724425505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c700f66f0 0x7f3c70110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:08.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.806+0000 7f3c76e3f700 1 --2- 172.21.15.113:0/724425505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700f6fc0 0x7f3c70110840 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:08.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.806+0000 7f3c76e3f700 1 -- 172.21.15.113:0/724425505 >> 172.21.15.113:0/724425505 conn(0x7f3c70070590 msgr2=0x7f3c7006da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:08.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.807+0000 7f3c76e3f700 1 -- 172.21.15.113:0/724425505 shutdown_connections 2024-04-12T05:51:08.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:08.807+0000 7f3c76e3f700 1 -- 172.21.15.113:0/724425505 wait complete. 2024-04-12T05:51:09.410 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions' 2024-04-12T05:51:10.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:09 smithi113 ceph-mon[103371]: pgmap v1172: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:10.168 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:09 smithi113 ceph-mon[103371]: from='client.36038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:10.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:09 smithi192 ceph-mon[85304]: pgmap v1172: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:10.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:09 smithi192 ceph-mon[85304]: from='client.36038 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T05:51:11.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.353+0000 7f7b9843b700 1 -- 172.21.15.113:0/2808441959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b900fc7f0 msgr2=0x7f7b900fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:11.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.353+0000 7f7b9843b700 1 --2- 172.21.15.113:0/2808441959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b900fc7f0 0x7f7b900fcbd0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f7b8c0047f0 tx=0x7f7b8c009cb0).stop 2024-04-12T05:51:11.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.354+0000 7f7b9843b700 1 -- 172.21.15.113:0/2808441959 shutdown_connections 2024-04-12T05:51:11.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.354+0000 7f7b9843b700 1 --2- 172.21.15.113:0/2808441959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b900fd150 0x7f7b900f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:11.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.354+0000 7f7b9843b700 1 --2- 172.21.15.113:0/2808441959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b900fc7f0 0x7f7b900fcbd0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:11.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.354+0000 7f7b9843b700 1 -- 172.21.15.113:0/2808441959 >> 172.21.15.113:0/2808441959 conn(0x7f7b90070590 msgr2=0x7f7b900f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:11.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.354+0000 7f7b9843b700 1 -- 172.21.15.113:0/2808441959 shutdown_connections 2024-04-12T05:51:11.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.354+0000 7f7b9843b700 1 -- 172.21.15.113:0/2808441959 wait complete. 2024-04-12T05:51:11.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.354+0000 7f7b9843b700 1 Processor -- start 2024-04-12T05:51:11.350 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.354+0000 7f7b9843b700 1 -- start start 2024-04-12T05:51:11.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b9843b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b900fc7f0 0x7f7b9010cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:11.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b9843b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b900fd150 0x7f7b9010d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:11.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b9843b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9010da20 con 0x7f7b900fd150 2024-04-12T05:51:11.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b9843b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9010db60 con 0x7f7b900fc7f0 2024-04-12T05:51:11.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b959d6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b900fd150 0x7f7b9010d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:11.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b959d6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b900fd150 0x7f7b9010d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38118/0 (socket says 172.21.15.113:38118) 2024-04-12T05:51:11.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b959d6700 1 -- 172.21.15.113:0/3737034026 learned_addr learned my addr 172.21.15.113:0/3737034026 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:11.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b961d7700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b900fc7f0 0x7f7b9010cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:11.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b959d6700 1 -- 172.21.15.113:0/3737034026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b900fc7f0 msgr2=0x7f7b9010cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:11.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b959d6700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b900fc7f0 0x7f7b9010cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:11.353 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b959d6700 1 -- 172.21.15.113:0/3737034026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c005040 con 0x7f7b900fd150 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b961d7700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b900fc7f0 0x7f7b9010cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b959d6700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b900fd150 0x7f7b9010d420 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f7b8400c790 tx=0x7f7b84007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b837fe700 1 -- 172.21.15.113:0/3737034026 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b840106f0 con 0x7f7b900fd150 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b837fe700 1 -- 172.21.15.113:0/3737034026 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b84010d00 con 0x7f7b900fd150 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b9843b700 1 -- 172.21.15.113:0/3737034026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b901071f0 con 0x7f7b900fd150 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.355+0000 7f7b837fe700 1 -- 172.21.15.113:0/3737034026 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b84012d50 con 0x7f7b900fd150 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.356+0000 7f7b9843b700 1 -- 172.21.15.113:0/3737034026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b901077b0 con 0x7f7b900fd150 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.356+0000 7f7b9843b700 1 -- 172.21.15.113:0/3737034026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9005f590 con 0x7f7b900fd150 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.357+0000 7f7b837fe700 1 -- 172.21.15.113:0/3737034026 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7b84012570 con 0x7f7b900fd150 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.357+0000 7f7b837fe700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b7c06caa0 0x7f7b7c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.357+0000 7f7b837fe700 1 -- 172.21.15.113:0/3737034026 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f7b8408b740 con 0x7f7b900fd150 2024-04-12T05:51:11.354 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.357+0000 7f7b961d7700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b7c06caa0 0x7f7b7c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:11.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.358+0000 7f7b961d7700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b7c06caa0 0x7f7b7c06ef60 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f7b8c0043d0 tx=0x7f7b8c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:11.357 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.362+0000 7f7b837fe700 1 -- 172.21.15.113:0/3737034026 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f7b840591f0 con 0x7f7b900fd150 2024-04-12T05:51:11.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.632+0000 7f7b9843b700 1 -- 172.21.15.113:0/3737034026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7b90047540 con 0x7f7b900fd150 2024-04-12T05:51:11.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.633+0000 7f7b837fe700 1 -- 172.21.15.113:0/3737034026 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f7b8405f070 con 0x7f7b900fd150 2024-04-12T05:51:11.629 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-04-12T05:51:11.629 INFO:teuthology.orchestra.run.smithi113.stdout: "mon": { 2024-04-12T05:51:11.629 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:51:11.629 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:51:11.629 INFO:teuthology.orchestra.run.smithi113.stdout: "mgr": { 2024-04-12T05:51:11.629 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 2 2024-04-12T05:51:11.629 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:51:11.629 INFO:teuthology.orchestra.run.smithi113.stdout: "osd": { 2024-04-12T05:51:11.630 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 6 2024-04-12T05:51:11.630 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:51:11.630 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-04-12T05:51:11.630 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 4 2024-04-12T05:51:11.630 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-04-12T05:51:11.630 INFO:teuthology.orchestra.run.smithi113.stdout: "overall": { 2024-04-12T05:51:11.630 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)": 14 2024-04-12T05:51:11.630 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-04-12T05:51:11.630 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-04-12T05:51:11.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.635+0000 7f7b9843b700 1 -- 172.21.15.113:0/3737034026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b7c06caa0 msgr2=0x7f7b7c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:11.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.635+0000 7f7b9843b700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b7c06caa0 0x7f7b7c06ef60 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f7b8c0043d0 tx=0x7f7b8c00b040).stop 2024-04-12T05:51:11.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.635+0000 7f7b9843b700 1 -- 172.21.15.113:0/3737034026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b900fd150 msgr2=0x7f7b9010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:11.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.635+0000 7f7b9843b700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b900fd150 0x7f7b9010d420 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f7b8400c790 tx=0x7f7b84007f50).stop 2024-04-12T05:51:11.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.635+0000 7f7b9843b700 1 -- 172.21.15.113:0/3737034026 shutdown_connections 2024-04-12T05:51:11.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.635+0000 7f7b9843b700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f7b7c06caa0 0x7f7b7c06ef60 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:11.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.635+0000 7f7b9843b700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b900fc7f0 0x7f7b9010cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:11.631 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.635+0000 7f7b9843b700 1 --2- 172.21.15.113:0/3737034026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b900fd150 0x7f7b9010d420 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:11.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.635+0000 7f7b9843b700 1 -- 172.21.15.113:0/3737034026 >> 172.21.15.113:0/3737034026 conn(0x7f7b90070590 msgr2=0x7f7b900fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:11.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.635+0000 7f7b9843b700 1 -- 172.21.15.113:0/3737034026 shutdown_connections 2024-04-12T05:51:11.632 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:11.635+0000 7f7b9843b700 1 -- 172.21.15.113:0/3737034026 wait complete. 2024-04-12T05:51:12.238 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:11 smithi113 ceph-mon[103371]: pgmap v1173: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:12.238 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:11 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3737034026' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:51:12.260 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-12T05:51:12.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:11 smithi192 ceph-mon[85304]: pgmap v1173: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:12.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:11 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3737034026' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:51:14.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.092+0000 7f0943bed700 1 -- 172.21.15.113:0/1859211429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c0fc7f0 msgr2=0x7f093c0fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:14.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.092+0000 7f0943bed700 1 --2- 172.21.15.113:0/1859211429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c0fc7f0 0x7f093c0fcbd0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7f092c0047f0 tx=0x7f092c009cb0).stop 2024-04-12T05:51:14.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.092+0000 7f0943bed700 1 -- 172.21.15.113:0/1859211429 shutdown_connections 2024-04-12T05:51:14.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.092+0000 7f0943bed700 1 --2- 172.21.15.113:0/1859211429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c0fd150 0x7f093c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:14.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.092+0000 7f0943bed700 1 --2- 172.21.15.113:0/1859211429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c0fc7f0 0x7f093c0fcbd0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:14.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.092+0000 7f0943bed700 1 -- 172.21.15.113:0/1859211429 >> 172.21.15.113:0/1859211429 conn(0x7f093c070590 msgr2=0x7f093c0f4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:14.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.092+0000 7f0943bed700 1 -- 172.21.15.113:0/1859211429 shutdown_connections 2024-04-12T05:51:14.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.092+0000 7f0943bed700 1 -- 172.21.15.113:0/1859211429 wait complete. 2024-04-12T05:51:14.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0943bed700 1 Processor -- start 2024-04-12T05:51:14.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0943bed700 1 -- start start 2024-04-12T05:51:14.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0943bed700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c0fc7f0 0x7f093c0f8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:14.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0943bed700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c0fd150 0x7f093c0f9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0943bed700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c0f9740 con 0x7f093c0fd150 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0943bed700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c0f9880 con 0x7f093c0fc7f0 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0941188700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c0fd150 0x7f093c0f9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0941188700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c0fd150 0x7f093c0f9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38132/0 (socket says 172.21.15.113:38132) 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0941188700 1 -- 172.21.15.113:0/3278293993 learned_addr learned my addr 172.21.15.113:0/3278293993 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0941989700 1 --2- 172.21.15.113:0/3278293993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c0fc7f0 0x7f093c0f8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0941188700 1 -- 172.21.15.113:0/3278293993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c0fc7f0 msgr2=0x7f093c0f8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0941188700 1 --2- 172.21.15.113:0/3278293993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c0fc7f0 0x7f093c0f8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0941188700 1 -- 172.21.15.113:0/3278293993 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c005040 con 0x7f093c0fd150 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.093+0000 7f0941188700 1 --2- 172.21.15.113:0/3278293993 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c0fd150 0x7f093c0f9250 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7f0938002820 tx=0x7f0938009f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.094+0000 7f0932ffd700 1 -- 172.21.15.113:0/3278293993 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0938007b40 con 0x7f093c0fd150 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.094+0000 7f0932ffd700 1 -- 172.21.15.113:0/3278293993 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f093800f040 con 0x7f093c0fd150 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.094+0000 7f0943bed700 1 -- 172.21.15.113:0/3278293993 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093c0f9b30 con 0x7f093c0fd150 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.094+0000 7f0932ffd700 1 -- 172.21.15.113:0/3278293993 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0938014910 con 0x7f093c0fd150 2024-04-12T05:51:14.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.094+0000 7f0943bed700 1 -- 172.21.15.113:0/3278293993 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093c114720 con 0x7f093c0fd150 2024-04-12T05:51:14.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.095+0000 7f0943bed700 1 -- 172.21.15.113:0/3278293993 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093c05f590 con 0x7f093c0fd150 2024-04-12T05:51:14.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.095+0000 7f0932ffd700 1 -- 172.21.15.113:0/3278293993 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0938007ca0 con 0x7f093c0fd150 2024-04-12T05:51:14.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.095+0000 7f0932ffd700 1 --2- 172.21.15.113:0/3278293993 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f092806caa0 0x7f092806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:14.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.095+0000 7f0932ffd700 1 -- 172.21.15.113:0/3278293993 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f093808b7d0 con 0x7f093c0fd150 2024-04-12T05:51:14.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.095+0000 7f0941989700 1 --2- 172.21.15.113:0/3278293993 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f092806caa0 0x7f092806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:14.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.096+0000 7f0941989700 1 --2- 172.21.15.113:0/3278293993 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f092806caa0 0x7f092806ef60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f092c004f10 tx=0x7f092c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:14.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.100+0000 7f0932ffd700 1 -- 172.21.15.113:0/3278293993 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f09380557f0 con 0x7f093c0fd150 2024-04-12T05:51:14.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:13 smithi113 ceph-mon[103371]: pgmap v1174: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:14.365 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.370+0000 7f0943bed700 1 -- 172.21.15.113:0/3278293993 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f093c047540 con 0x7f093c0fd150 2024-04-12T05:51:14.366 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.370+0000 7f0932ffd700 1 -- 172.21.15.113:0/3278293993 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f0938058e10 con 0x7f093c0fd150 2024-04-12T05:51:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.372+0000 7f0943bed700 1 -- 172.21.15.113:0/3278293993 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f092806caa0 msgr2=0x7f092806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.372+0000 7f0943bed700 1 --2- 172.21.15.113:0/3278293993 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f092806caa0 0x7f092806ef60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f092c004f10 tx=0x7f092c00b040).stop 2024-04-12T05:51:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.372+0000 7f0943bed700 1 -- 172.21.15.113:0/3278293993 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c0fd150 msgr2=0x7f093c0f9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.372+0000 7f0943bed700 1 --2- 172.21.15.113:0/3278293993 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c0fd150 0x7f093c0f9250 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7f0938002820 tx=0x7f0938009f20).stop 2024-04-12T05:51:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.373+0000 7f0943bed700 1 -- 172.21.15.113:0/3278293993 shutdown_connections 2024-04-12T05:51:14.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.373+0000 7f0943bed700 1 --2- 172.21.15.113:0/3278293993 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f092806caa0 0x7f092806ef60 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:14.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.373+0000 7f0943bed700 1 --2- 172.21.15.113:0/3278293993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f093c0fc7f0 0x7f093c0f8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:14.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.373+0000 7f0943bed700 1 --2- 172.21.15.113:0/3278293993 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f093c0fd150 0x7f093c0f9250 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:14.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.373+0000 7f0943bed700 1 -- 172.21.15.113:0/3278293993 >> 172.21.15.113:0/3278293993 conn(0x7f093c070590 msgr2=0x7f093c060590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:14.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.373+0000 7f0943bed700 1 -- 172.21.15.113:0/3278293993 shutdown_connections 2024-04-12T05:51:14.369 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:14.373+0000 7f0943bed700 1 -- 172.21.15.113:0/3278293993 wait complete. 2024-04-12T05:51:14.378 INFO:teuthology.orchestra.run.smithi113.stdout:true 2024-04-12T05:51:14.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:13 smithi192 ceph-mon[85304]: pgmap v1174: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:14.985 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:14 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3278293993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:51:15.006 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -e sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-12T05:51:15.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:14 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3278293993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:51:16.336 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:15 smithi113 ceph-mon[103371]: pgmap v1175: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:16.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:15 smithi192 ceph-mon[85304]: pgmap v1175: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:16.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.819+0000 7f8c78de9700 1 -- 172.21.15.113:0/2516960530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c740f92e0 msgr2=0x7f8c74101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:16.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.819+0000 7f8c78de9700 1 --2- 172.21.15.113:0/2516960530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c740f92e0 0x7f8c74101a80 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7f8c640047f0 tx=0x7f8c64009cb0).stop 2024-04-12T05:51:16.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.819+0000 7f8c78de9700 1 -- 172.21.15.113:0/2516960530 shutdown_connections 2024-04-12T05:51:16.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.819+0000 7f8c78de9700 1 --2- 172.21.15.113:0/2516960530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c740f92e0 0x7f8c74101a80 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:16.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.819+0000 7f8c78de9700 1 --2- 172.21.15.113:0/2516960530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c740f8a10 0x7f8c740f8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:16.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.819+0000 7f8c78de9700 1 -- 172.21.15.113:0/2516960530 >> 172.21.15.113:0/2516960530 conn(0x7f8c7406c760 msgr2=0x7f8c7406cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:16.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.819+0000 7f8c78de9700 1 -- 172.21.15.113:0/2516960530 shutdown_connections 2024-04-12T05:51:16.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.819+0000 7f8c78de9700 1 -- 172.21.15.113:0/2516960530 wait complete. 2024-04-12T05:51:16.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.820+0000 7f8c78de9700 1 Processor -- start 2024-04-12T05:51:16.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.820+0000 7f8c78de9700 1 -- start start 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.820+0000 7f8c78de9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c740f8a10 0x7f8c7410e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.820+0000 7f8c78de9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c740f92e0 0x7f8c7410e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.820+0000 7f8c78de9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7410eb40 con 0x7f8c740f8a10 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.820+0000 7f8c78de9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7410ec80 con 0x7f8c740f92e0 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.820+0000 7f8c7259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c740f8a10 0x7f8c7410e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.820+0000 7f8c7259c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c740f8a10 0x7f8c7410e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38158/0 (socket says 172.21.15.113:38158) 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.820+0000 7f8c7259c700 1 -- 172.21.15.113:0/3579379311 learned_addr learned my addr 172.21.15.113:0/3579379311 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.821+0000 7f8c71d9b700 1 --2- 172.21.15.113:0/3579379311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c740f92e0 0x7f8c7410e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.821+0000 7f8c7259c700 1 -- 172.21.15.113:0/3579379311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c740f92e0 msgr2=0x7f8c7410e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.821+0000 7f8c7259c700 1 --2- 172.21.15.113:0/3579379311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c740f92e0 0x7f8c7410e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.821+0000 7f8c7259c700 1 -- 172.21.15.113:0/3579379311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c64005040 con 0x7f8c740f8a10 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.821+0000 7f8c7259c700 1 --2- 172.21.15.113:0/3579379311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c740f8a10 0x7f8c7410e050 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f8c5c0041d0 tx=0x7f8c5c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.821+0000 7f8c6b7fe700 1 -- 172.21.15.113:0/3579379311 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c5c007b40 con 0x7f8c740f8a10 2024-04-12T05:51:16.818 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.821+0000 7f8c6b7fe700 1 -- 172.21.15.113:0/3579379311 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c5c00f040 con 0x7f8c740f8a10 2024-04-12T05:51:16.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.821+0000 7f8c78de9700 1 -- 172.21.15.113:0/3579379311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c74108310 con 0x7f8c740f8a10 2024-04-12T05:51:16.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.821+0000 7f8c6b7fe700 1 -- 172.21.15.113:0/3579379311 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c5c0137d0 con 0x7f8c740f8a10 2024-04-12T05:51:16.819 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.821+0000 7f8c78de9700 1 -- 172.21.15.113:0/3579379311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c74108850 con 0x7f8c740f8a10 2024-04-12T05:51:16.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.822+0000 7f8c78de9700 1 -- 172.21.15.113:0/3579379311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7405f5a0 con 0x7f8c740f8a10 2024-04-12T05:51:16.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.822+0000 7f8c6b7fe700 1 -- 172.21.15.113:0/3579379311 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8c5c007ca0 con 0x7f8c740f8a10 2024-04-12T05:51:16.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.823+0000 7f8c6b7fe700 1 --2- 172.21.15.113:0/3579379311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8c6006caa0 0x7f8c6006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:16.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.823+0000 7f8c6b7fe700 1 -- 172.21.15.113:0/3579379311 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f8c5c08b7d0 con 0x7f8c740f8a10 2024-04-12T05:51:16.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.823+0000 7f8c71d9b700 1 --2- 172.21.15.113:0/3579379311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8c6006caa0 0x7f8c6006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:16.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.823+0000 7f8c71d9b700 1 --2- 172.21.15.113:0/3579379311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8c6006caa0 0x7f8c6006ef60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f8c74109990 tx=0x7f8c6400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:16.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:16.827+0000 7f8c6b7fe700 1 -- 172.21.15.113:0/3579379311 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f8c5c059280 con 0x7f8c740f8a10 2024-04-12T05:51:17.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.096+0000 7f8c78de9700 1 -- 172.21.15.113:0/3579379311 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8c74047540 con 0x7f8c740f8a10 2024-04-12T05:51:17.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.096+0000 7f8c6b7fe700 1 -- 172.21.15.113:0/3579379311 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f8c5c089e60 con 0x7f8c740f8a10 2024-04-12T05:51:17.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.098+0000 7f8c78de9700 1 -- 172.21.15.113:0/3579379311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8c6006caa0 msgr2=0x7f8c6006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:17.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.098+0000 7f8c78de9700 1 --2- 172.21.15.113:0/3579379311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8c6006caa0 0x7f8c6006ef60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f8c74109990 tx=0x7f8c6400b040).stop 2024-04-12T05:51:17.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.098+0000 7f8c78de9700 1 -- 172.21.15.113:0/3579379311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c740f8a10 msgr2=0x7f8c7410e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:17.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.098+0000 7f8c78de9700 1 --2- 172.21.15.113:0/3579379311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c740f8a10 0x7f8c7410e050 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f8c5c0041d0 tx=0x7f8c5c009f60).stop 2024-04-12T05:51:17.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.099+0000 7f8c78de9700 1 -- 172.21.15.113:0/3579379311 shutdown_connections 2024-04-12T05:51:17.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.099+0000 7f8c78de9700 1 --2- 172.21.15.113:0/3579379311 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f8c6006caa0 0x7f8c6006ef60 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:17.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.099+0000 7f8c78de9700 1 --2- 172.21.15.113:0/3579379311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8c740f8a10 0x7f8c7410e050 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:17.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.099+0000 7f8c78de9700 1 --2- 172.21.15.113:0/3579379311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c740f92e0 0x7f8c7410e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:17.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.099+0000 7f8c78de9700 1 -- 172.21.15.113:0/3579379311 >> 172.21.15.113:0/3579379311 conn(0x7f8c7406c760 msgr2=0x7f8c740fc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:17.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.099+0000 7f8c78de9700 1 -- 172.21.15.113:0/3579379311 shutdown_connections 2024-04-12T05:51:17.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:17.099+0000 7f8c78de9700 1 -- 172.21.15.113:0/3579379311 wait complete. 2024-04-12T05:51:17.105 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph version 18.2.2-1187-gd540ebac (d540ebaca6b131a1dd560e7f69e024b133bbaa42) reef (stable)" 2024-04-12T05:51:17.687 DEBUG:teuthology.parallel:result is None 2024-04-12T05:51:17.688 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T05:51:17.700 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T05:51:17.700 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-12T05:51:18.391 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:17 smithi113 ceph-mon[103371]: pgmap v1176: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:18.391 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:17 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3579379311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:51:18.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:17 smithi192 ceph-mon[85304]: pgmap v1176: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:18.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:17 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3579379311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T05:51:19.508 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.513+0000 7f010da7e700 1 -- 172.21.15.113:0/3636529888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01080fd910 msgr2=0x7f01080fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.513+0000 7f010da7e700 1 --2- 172.21.15.113:0/3636529888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01080fd910 0x7f01080fdcf0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7f00f80047f0 tx=0x7f00f8009cb0).stop 2024-04-12T05:51:19.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.513+0000 7f010da7e700 1 -- 172.21.15.113:0/3636529888 shutdown_connections 2024-04-12T05:51:19.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.513+0000 7f010da7e700 1 --2- 172.21.15.113:0/3636529888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01080fe270 0x7f01080f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:19.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.513+0000 7f010da7e700 1 --2- 172.21.15.113:0/3636529888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01080fd910 0x7f01080fdcf0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:19.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.513+0000 7f010da7e700 1 -- 172.21.15.113:0/3636529888 >> 172.21.15.113:0/3636529888 conn(0x7f01080f3510 msgr2=0x7f01080f5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:19.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.513+0000 7f010da7e700 1 -- 172.21.15.113:0/3636529888 shutdown_connections 2024-04-12T05:51:19.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.513+0000 7f010da7e700 1 -- 172.21.15.113:0/3636529888 wait complete. 2024-04-12T05:51:19.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.513+0000 7f010da7e700 1 Processor -- start 2024-04-12T05:51:19.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f010da7e700 1 -- start start 2024-04-12T05:51:19.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f010da7e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01080fd910 0x7f010810e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:19.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f010da7e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01080fe270 0x7f010810e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:19.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f010da7e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010810ec00 con 0x7f01080fd910 2024-04-12T05:51:19.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f010da7e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010810ed40 con 0x7f01080fe270 2024-04-12T05:51:19.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f01077fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01080fd910 0x7f010810e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f01077fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01080fd910 0x7f010810e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:38182/0 (socket says 172.21.15.113:38182) 2024-04-12T05:51:19.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f01077fe700 1 -- 172.21.15.113:0/1297988387 learned_addr learned my addr 172.21.15.113:0/1297988387 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:19.512 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f0106ffd700 1 --2- 172.21.15.113:0/1297988387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01080fe270 0x7f010810e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f01077fe700 1 -- 172.21.15.113:0/1297988387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01080fe270 msgr2=0x7f010810e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f01077fe700 1 --2- 172.21.15.113:0/1297988387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01080fe270 0x7f010810e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f01077fe700 1 -- 172.21.15.113:0/1297988387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f8005040 con 0x7f01080fd910 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.514+0000 7f01077fe700 1 --2- 172.21.15.113:0/1297988387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01080fd910 0x7f010810e110 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f00f8009380 tx=0x7f00f8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.515+0000 7f0104f74700 1 -- 172.21.15.113:0/1297988387 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00f8018070 con 0x7f01080fd910 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.515+0000 7f0104f74700 1 -- 172.21.15.113:0/1297988387 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00f8007e90 con 0x7f01080fd910 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.515+0000 7f010da7e700 1 -- 172.21.15.113:0/1297988387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0108108370 con 0x7f01080fd910 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.515+0000 7f0104f74700 1 -- 172.21.15.113:0/1297988387 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00f80123c0 con 0x7f01080fd910 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.515+0000 7f010da7e700 1 -- 172.21.15.113:0/1297988387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01081088d0 con 0x7f01080fd910 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.516+0000 7f010da7e700 1 -- 172.21.15.113:0/1297988387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0108060650 con 0x7f01080fd910 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.516+0000 7f0104f74700 1 -- 172.21.15.113:0/1297988387 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f00f8012630 con 0x7f01080fd910 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.516+0000 7f0104f74700 1 --2- 172.21.15.113:0/1297988387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f00f006caa0 0x7f00f006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.517+0000 7f0104f74700 1 -- 172.21.15.113:0/1297988387 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f00f808ca20 con 0x7f01080fd910 2024-04-12T05:51:19.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.517+0000 7f0106ffd700 1 --2- 172.21.15.113:0/1297988387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f00f006caa0 0x7f00f006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:19.514 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.517+0000 7f0106ffd700 1 --2- 172.21.15.113:0/1297988387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f00f006caa0 0x7f00f006ef60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f0108109a50 tx=0x7f00fc00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:19.516 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.521+0000 7f0104f74700 1 -- 172.21.15.113:0/1297988387 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f00f805a540 con 0x7f01080fd910 2024-04-12T05:51:19.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.748+0000 7f010da7e700 1 -- 172.21.15.113:0/1297988387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0108047540 con 0x7f01080fd910 2024-04-12T05:51:19.744 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.749+0000 7f0104f74700 1 -- 172.21.15.113:0/1297988387 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 35 v35) v1 ==== 76+0+1860 (secure 0 0 0) 0x7f00f805a0d0 con 0x7f01080fd910 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout:e35 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout:legacy client fscid: 1 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout:Filesystem 'cephfs' (1) 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout:fs_name cephfs 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout:epoch 35 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout:created 2024-04-12T05:06:12.143448+0000 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout:modified 2024-04-12T05:49:50.608720+0000 2024-04-12T05:51:19.745 INFO:teuthology.orchestra.run.smithi113.stdout:tableserver 0 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:root 0 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:session_timeout 60 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:session_autoclose 300 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:max_file_size 1099511627776 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:max_xattr_size 65536 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:required_client_features {} 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure 0 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:last_failure_osd_epoch 93 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:max_mds 2 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:in 0,1 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:up {0=34412,1=34444} 2024-04-12T05:51:19.746 INFO:teuthology.orchestra.run.smithi113.stdout:failed 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:damaged 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:stopped 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:data_pools [3] 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:metadata_pool 2 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:inline_data disabled 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:balancer 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:bal_rank_mask -1 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:standby_count_wanted 1 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ovmadh{0:34412} state up:active seq 468 join_fscid=1 addr [v2:172.21.15.113:6826/3491701432,v1:172.21.15.113:6827/3491701432] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi113.ydsbei{1:34444} state up:active seq 463 join_fscid=1 addr [v2:172.21.15.113:6828/280747625,v1:172.21.15.113:6829/280747625] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:Standby daemons: 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:19.747 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.ammjap{-1:44331} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6824/335558856,v1:172.21.15.192:6825/335558856] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:19.748 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.cephfs.smithi192.dtnaoi{-1:44817} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6826/946017279,v1:172.21.15.192:6827/946017279] compat {c=[1],r=[1],i=[7ff]}] 2024-04-12T05:51:19.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.751+0000 7f010da7e700 1 -- 172.21.15.113:0/1297988387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f00f006caa0 msgr2=0x7f00f006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.751+0000 7f010da7e700 1 --2- 172.21.15.113:0/1297988387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f00f006caa0 0x7f00f006ef60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f0108109a50 tx=0x7f00fc00b040).stop 2024-04-12T05:51:19.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.751+0000 7f010da7e700 1 -- 172.21.15.113:0/1297988387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01080fd910 msgr2=0x7f010810e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:19.748 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.751+0000 7f010da7e700 1 --2- 172.21.15.113:0/1297988387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01080fd910 0x7f010810e110 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f00f8009380 tx=0x7f00f8009cb0).stop 2024-04-12T05:51:19.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.751+0000 7f010da7e700 1 -- 172.21.15.113:0/1297988387 shutdown_connections 2024-04-12T05:51:19.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.751+0000 7f010da7e700 1 --2- 172.21.15.113:0/1297988387 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f00f006caa0 0x7f00f006ef60 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:19.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.751+0000 7f010da7e700 1 --2- 172.21.15.113:0/1297988387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01080fd910 0x7f010810e110 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:19.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.751+0000 7f010da7e700 1 --2- 172.21.15.113:0/1297988387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01080fe270 0x7f010810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:19.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.751+0000 7f010da7e700 1 -- 172.21.15.113:0/1297988387 >> 172.21.15.113:0/1297988387 conn(0x7f01080f3510 msgr2=0x7f01080fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:19.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.751+0000 7f010da7e700 1 -- 172.21.15.113:0/1297988387 shutdown_connections 2024-04-12T05:51:19.749 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:19.751+0000 7f010da7e700 1 -- 172.21.15.113:0/1297988387 wait complete. 2024-04-12T05:51:19.749 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 35 2024-04-12T05:51:20.085 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:19 smithi113 ceph-mon[103371]: pgmap v1177: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:20.085 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:19 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1297988387' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:51:20.361 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-12T05:51:20.376 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T05:51:20.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:19 smithi192 ceph-mon[85304]: pgmap v1177: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:20.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:19 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1297988387' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T05:51:20.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.500+0000 7ff4e3f16700 1 -- 172.21.15.113:0/3909307613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4dc0fd090 msgr2=0x7ff4dc0f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.500+0000 7ff4e3f16700 1 --2- 172.21.15.113:0/3909307613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4dc0fd090 0x7ff4dc0f6870 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7ff4d80047f0 tx=0x7ff4d8009cb0).stop 2024-04-12T05:51:20.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.500+0000 7ff4e3f16700 1 -- 172.21.15.113:0/3909307613 shutdown_connections 2024-04-12T05:51:20.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.500+0000 7ff4e3f16700 1 --2- 172.21.15.113:0/3909307613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4dc0fd090 0x7ff4dc0f6870 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:20.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.500+0000 7ff4e3f16700 1 --2- 172.21.15.113:0/3909307613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4dc0fc730 0x7ff4dc0fcb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:20.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.500+0000 7ff4e3f16700 1 -- 172.21.15.113:0/3909307613 >> 172.21.15.113:0/3909307613 conn(0x7ff4dc06c760 msgr2=0x7ff4dc06cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:20.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.500+0000 7ff4e3f16700 1 -- 172.21.15.113:0/3909307613 shutdown_connections 2024-04-12T05:51:20.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.500+0000 7ff4e3f16700 1 -- 172.21.15.113:0/3909307613 wait complete. 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.501+0000 7ff4e3f16700 1 Processor -- start 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.501+0000 7ff4e3f16700 1 -- start start 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.501+0000 7ff4e3f16700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4dc0fc730 0x7ff4dc10ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.501+0000 7ff4e3f16700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4dc0fd090 0x7ff4dc10d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.501+0000 7ff4e3f16700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc10d980 con 0x7ff4dc0fc730 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.501+0000 7ff4e3f16700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc10dac0 con 0x7ff4dc0fd090 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.501+0000 7ff4e1cb2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4dc0fc730 0x7ff4dc10ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.501+0000 7ff4e1cb2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4dc0fc730 0x7ff4dc10ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33810/0 (socket says 172.21.15.113:33810) 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.501+0000 7ff4e1cb2700 1 -- 172.21.15.113:0/935269263 learned_addr learned my addr 172.21.15.113:0/935269263 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.502+0000 7ff4e14b1700 1 --2- 172.21.15.113:0/935269263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4dc0fd090 0x7ff4dc10d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.502+0000 7ff4e1cb2700 1 -- 172.21.15.113:0/935269263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4dc0fd090 msgr2=0x7ff4dc10d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.502+0000 7ff4e1cb2700 1 --2- 172.21.15.113:0/935269263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4dc0fd090 0x7ff4dc10d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:20.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.502+0000 7ff4e1cb2700 1 -- 172.21.15.113:0/935269263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8005040 con 0x7ff4dc0fc730 2024-04-12T05:51:20.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.502+0000 7ff4e1cb2700 1 --2- 172.21.15.113:0/935269263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4dc0fc730 0x7ff4dc10ce90 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7ff4d000e970 tx=0x7ff4d000ed30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:20.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.502+0000 7ff4ceffd700 1 -- 172.21.15.113:0/935269263 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4d000cb90 con 0x7ff4dc0fc730 2024-04-12T05:51:20.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.502+0000 7ff4ceffd700 1 -- 172.21.15.113:0/935269263 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4d000ccf0 con 0x7ff4dc0fc730 2024-04-12T05:51:20.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.502+0000 7ff4e3f16700 1 -- 172.21.15.113:0/935269263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4dc107150 con 0x7ff4dc0fc730 2024-04-12T05:51:20.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.502+0000 7ff4ceffd700 1 -- 172.21.15.113:0/935269263 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4d0010610 con 0x7ff4dc0fc730 2024-04-12T05:51:20.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.502+0000 7ff4e3f16700 1 -- 172.21.15.113:0/935269263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4dc107690 con 0x7ff4dc0fc730 2024-04-12T05:51:20.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.503+0000 7ff4e3f16700 1 -- 172.21.15.113:0/935269263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4dc05f380 con 0x7ff4dc0fc730 2024-04-12T05:51:20.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.503+0000 7ff4ceffd700 1 -- 172.21.15.113:0/935269263 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff4d0011470 con 0x7ff4dc0fc730 2024-04-12T05:51:20.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.504+0000 7ff4ceffd700 1 --2- 172.21.15.113:0/935269263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff4c806caa0 0x7ff4c806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:20.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.504+0000 7ff4ceffd700 1 -- 172.21.15.113:0/935269263 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff4d008e7f0 con 0x7ff4dc0fc730 2024-04-12T05:51:20.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.504+0000 7ff4e14b1700 1 --2- 172.21.15.113:0/935269263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff4c806caa0 0x7ff4c806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:20.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.504+0000 7ff4e14b1700 1 --2- 172.21.15.113:0/935269263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff4c806caa0 0x7ff4c806ef60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7ff4d8004f10 tx=0x7ff4d800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:20.504 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.508+0000 7ff4ceffd700 1 -- 172.21.15.113:0/935269263 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff4d005c2a0 con 0x7ff4dc0fc730 2024-04-12T05:51:20.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.699+0000 7ff4e3f16700 1 -- 172.21.15.113:0/935269263 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4dc05eec0 con 0x7ff4dc0fc730 2024-04-12T05:51:20.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.700+0000 7ff4ceffd700 1 -- 172.21.15.113:0/935269263 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16556 (secure 0 0 0) 0x7ff4dc05eec0 con 0x7ff4dc0fc730 2024-04-12T05:51:20.696 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:20.696 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":98,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","created":"2024-04-12T04:59:30.005034+0000","modified":"2024-04-12T05:34:33.348722+0000","last_up_change":"2024-04-12T05:18:43.266143+0000","last_in_change":"2024-04-12T05:04:31.248597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T05:03:31.839630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{},"mgr_devicehealth":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-12T05:06:11.000491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-12T05:06:11.375495+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fc942a17-378a-48a6-b1d3-4f5730550810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":92,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6803","nonce":2361630947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6805","nonce":2361630947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6809","nonce":2361630947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6807","nonce":2361630947}]},"public_addr":"172.21.15.113:6803/2361630947","cluster_addr":"172.21.15.113:6805/2361630947","heartbeat_back_addr":"172.21.15.113:6809/2361630947","heartbeat_front_addr":"172.21.15.113:6807/2361630947","state":["exists","up"]},{"osd":1,"uuid":"39fec899-371b-4d48-99a8-52f90f52a71f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":92,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6811","nonce":1423470151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6813","nonce":1423470151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6817","nonce":1423470151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6815","nonce":1423470151}]},"public_addr":"172.21.15.113:6811/1423470151","cluster_addr":"172.21.15.113:6813/1423470151","heartbeat_back_addr":"172.21.15.113:6817/1423470151","heartbeat_front_addr":"172.21.15.113:6815/1423470151","state":["exists","up"]},{"osd":2,"uuid":"b135f3a9-d2f0-4737-8a30-9f6dc3f2571b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":63,"up_thru":92,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6819","nonce":2501657120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6821","nonce":2501657120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6825","nonce":2501657120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6823","nonce":2501657120}]},"public_addr":"172.21.15.113:6819/2501657120","cluster_addr":"172.21.15.113:6821/2501657120","heartbeat_back_addr":"172.21.15.113:6825/2501657120","heartbeat_front_addr":"172.21.15.113:6823/2501657120","state":["exists","up"]},{"osd":3,"uuid":"49fd856c-c351-4e90-9088-ece0024cd2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":72,"up_thru":92,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6801","nonce":2734551320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6803","nonce":2734551320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6807","nonce":2734551320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6805","nonce":2734551320}]},"public_addr":"172.21.15.192:6801/2734551320","cluster_addr":"172.21.15.192:6803/2734551320","heartbeat_back_addr":"172.21.15.192:6807/2734551320","heartbeat_front_addr":"172.21.15.192:6805/2734551320","state":["exists","up"]},{"osd":4,"uuid":"80b72db8-682a-4a51-8d33-b0addda91fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":32,"last_clean_end":76,"up_from":80,"up_thru":92,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6809","nonce":2295139175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6811","nonce":2295139175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6815","nonce":2295139175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6813","nonce":2295139175}]},"public_addr":"172.21.15.192:6809/2295139175","cluster_addr":"172.21.15.192:6811/2295139175","heartbeat_back_addr":"172.21.15.192:6815/2295139175","heartbeat_front_addr":"172.21.15.192:6813/2295139175","state":["exists","up"]},{"osd":5,"uuid":"c8e1e98f-5edb-4fec-bdf6-bcda40562682","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":37,"last_clean_end":88,"up_from":92,"up_thru":92,"down_at":89,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6817","nonce":2093584908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6819","nonce":2093584908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6823","nonce":2093584908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6821","nonce":2093584908}]},"public_addr":"172.21.15.192:6817/2093584908","cluster_addr":"172.21.15.192:6819/2093584908","heartbeat_back_addr":"172.21.15.192:6823/2093584908","heartbeat_front_addr":"172.21.15.192:6821/2093584908","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:02:27.514514+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:01.358930+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:30.203057+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:54.928612+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:21.788000+0000","dead_epoch":76},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:48.830652+0000","dead_epoch":88}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.19","mappings":[{"from":0,"to":3}]},{"pgid":"2.1b","mappings":[{"from":0,"to":3}]},{"pgid":"3.4","mappings":[{"from":1,"to":5}]},{"pgid":"3.d","mappings":[{"from":4,"to":0}]},{"pgid":"3.10","mappings":[{"from":1,"to":3}]},{"pgid":"3.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.113:6827/1262609849":"2024-04-13T05:18:43.274072+0000","172.21.15.113:6801/3082109414":"2024-04-13T05:13:51.744570+0000","172.21.15.192:6824/616350866":"2024-04-13T05:18:43.274091+0000","172.21.15.113:6800/3082109414":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/321762285":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/2190156688":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/3618578044":"2024-04-13T05:13:51.744570+0000","172.21.15.192:6825/616350866":"2024-04-13T05:18:43.274091+0000","172.21.15.113:0/313346589":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/4036112881":"2024-04-13T05:08:26.214053+0000","172.21.15.113:6801/593576174":"2024-04-13T05:08:26.214053+0000","172.21.15.113:6800/593576174":"2024-04-13T05:08:26.214053+0000","172.21.15.113:0/1777682971":"2024-04-13T04:59:57.862720+0000","172.21.15.192:0/2210150409":"2024-04-13T05:10:55.934327+0000","172.21.15.113:6800/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.192:0/2589268524":"2024-04-13T05:10:55.934327+0000","172.21.15.192:6828/2257000201":"2024-04-13T05:10:55.934327+0000","172.21.15.113:0/516367458":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/781838919":"2024-04-13T05:08:26.214053+0000","172.21.15.113:6801/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.113:6801/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/3213888755":"2024-04-13T05:01:23.491618+0000","172.21.15.192:6829/2257000201":"2024-04-13T05:10:55.934327+0000","172.21.15.113:6826/1262609849":"2024-04-13T05:18:43.274072+0000","172.21.15.192:0/1673786821":"2024-04-13T05:10:55.934327+0000","172.21.15.113:0/2902584436":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/4242055626":"2024-04-13T05:00:29.828900+0000","172.21.15.113:6800/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.113:6801/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.192:0/2053192105":"2024-04-13T05:10:55.934327+0000","172.21.15.113:6800/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/3450096338":"2024-04-13T04:59:57.862720+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T05:51:20.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.702+0000 7ff4e3f16700 1 -- 172.21.15.113:0/935269263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff4c806caa0 msgr2=0x7ff4c806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.702+0000 7ff4e3f16700 1 --2- 172.21.15.113:0/935269263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff4c806caa0 0x7ff4c806ef60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7ff4d8004f10 tx=0x7ff4d800b040).stop 2024-04-12T05:51:20.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.702+0000 7ff4e3f16700 1 -- 172.21.15.113:0/935269263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4dc0fc730 msgr2=0x7ff4dc10ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.702+0000 7ff4e3f16700 1 --2- 172.21.15.113:0/935269263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4dc0fc730 0x7ff4dc10ce90 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7ff4d000e970 tx=0x7ff4d000ed30).stop 2024-04-12T05:51:20.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.702+0000 7ff4e3f16700 1 -- 172.21.15.113:0/935269263 shutdown_connections 2024-04-12T05:51:20.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.702+0000 7ff4e3f16700 1 --2- 172.21.15.113:0/935269263 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff4c806caa0 0x7ff4c806ef60 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:20.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.702+0000 7ff4e3f16700 1 --2- 172.21.15.113:0/935269263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4dc0fc730 0x7ff4dc10ce90 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:20.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.702+0000 7ff4e3f16700 1 --2- 172.21.15.113:0/935269263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4dc0fd090 0x7ff4dc10d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:20.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.702+0000 7ff4e3f16700 1 -- 172.21.15.113:0/935269263 >> 172.21.15.113:0/935269263 conn(0x7ff4dc06c760 msgr2=0x7ff4dc0faaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:20.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.702+0000 7ff4e3f16700 1 -- 172.21.15.113:0/935269263 shutdown_connections 2024-04-12T05:51:20.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.702+0000 7ff4e3f16700 1 -- 172.21.15.113:0/935269263 wait complete. 2024-04-12T05:51:20.710 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T05:03:31.839630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}, 'mgr_devicehealth': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-12T05:06:11.000491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-12T05:06:11.375495+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-12T05:51:20.711 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T05:51:20.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.835+0000 7f97caad3700 1 -- 172.21.15.113:0/1418852536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97c40fd370 msgr2=0x7f97c40f6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.835+0000 7f97caad3700 1 --2- 172.21.15.113:0/1418852536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97c40fd370 0x7f97c40f6b30 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7f97b00047f0 tx=0x7f97b0009cb0).stop 2024-04-12T05:51:20.832 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.835+0000 7f97caad3700 1 -- 172.21.15.113:0/1418852536 shutdown_connections 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.835+0000 7f97caad3700 1 --2- 172.21.15.113:0/1418852536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97c40fd370 0x7f97c40f6b30 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.835+0000 7f97caad3700 1 --2- 172.21.15.113:0/1418852536 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97c40fca10 0x7f97c40fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.835+0000 7f97caad3700 1 -- 172.21.15.113:0/1418852536 >> 172.21.15.113:0/1418852536 conn(0x7f97c40f2590 msgr2=0x7f97c40f49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.835+0000 7f97caad3700 1 -- 172.21.15.113:0/1418852536 shutdown_connections 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.835+0000 7f97caad3700 1 -- 172.21.15.113:0/1418852536 wait complete. 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.835+0000 7f97caad3700 1 Processor -- start 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.835+0000 7f97caad3700 1 -- start start 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97caad3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97c40fca10 0x7f97c40f8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97caad3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97c40fd370 0x7f97c40f8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97caad3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40f90e0 con 0x7f97c40fca10 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97caad3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40f9220 con 0x7f97c40fd370 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97c886f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97c40fca10 0x7f97c40f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97c886f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97c40fca10 0x7f97c40f8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33824/0 (socket says 172.21.15.113:33824) 2024-04-12T05:51:20.833 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97c886f700 1 -- 172.21.15.113:0/2218448269 learned_addr learned my addr 172.21.15.113:0/2218448269 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:20.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97c886f700 1 -- 172.21.15.113:0/2218448269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97c40fd370 msgr2=0x7f97c40f8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:20.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97c886f700 1 --2- 172.21.15.113:0/2218448269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97c40fd370 0x7f97c40f8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:20.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97c886f700 1 -- 172.21.15.113:0/2218448269 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b0005040 con 0x7f97c40fca10 2024-04-12T05:51:20.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97c886f700 1 --2- 172.21.15.113:0/2218448269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97c40fca10 0x7f97c40f8700 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7f97b80041d0 tx=0x7f97b8009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:20.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97c1f76700 1 -- 172.21.15.113:0/2218448269 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97b8002d90 con 0x7f97c40fca10 2024-04-12T05:51:20.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97c1f76700 1 -- 172.21.15.113:0/2218448269 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97b800f040 con 0x7f97c40fca10 2024-04-12T05:51:20.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97c1f76700 1 -- 172.21.15.113:0/2218448269 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97b800d800 con 0x7f97c40fca10 2024-04-12T05:51:20.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97caad3700 1 -- 172.21.15.113:0/2218448269 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c40f94d0 con 0x7f97c40fca10 2024-04-12T05:51:20.834 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.836+0000 7f97caad3700 1 -- 172.21.15.113:0/2218448269 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c4114870 con 0x7f97c40fca10 2024-04-12T05:51:20.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.838+0000 7f97caad3700 1 -- 172.21.15.113:0/2218448269 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c40603b0 con 0x7f97c40fca10 2024-04-12T05:51:20.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.838+0000 7f97c1f76700 1 -- 172.21.15.113:0/2218448269 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f97b800dac0 con 0x7f97c40fca10 2024-04-12T05:51:20.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.838+0000 7f97c1f76700 1 --2- 172.21.15.113:0/2218448269 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97b406caa0 0x7f97b406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:20.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.838+0000 7f97c1f76700 1 -- 172.21.15.113:0/2218448269 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f97b808bea0 con 0x7f97c40fca10 2024-04-12T05:51:20.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.838+0000 7f97c3fff700 1 --2- 172.21.15.113:0/2218448269 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97b406caa0 0x7f97b406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:20.835 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.839+0000 7f97c3fff700 1 --2- 172.21.15.113:0/2218448269 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97b406caa0 0x7f97b406ef60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f97c40f9ea0 tx=0x7f97b0009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:20.838 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:20.843+0000 7f97c1f76700 1 -- 172.21.15.113:0/2218448269 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f97b8055ec0 con 0x7f97c40fca10 2024-04-12T05:51:21.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.037+0000 7f97caad3700 1 -- 172.21.15.113:0/2218448269 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f97c40f9be0 con 0x7f97c40fca10 2024-04-12T05:51:21.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.038+0000 7f97c1f76700 1 -- 172.21.15.113:0/2218448269 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97b80594e0 con 0x7f97c40fca10 2024-04-12T05:51:21.033 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-04-12T05:51:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.040+0000 7f97caad3700 1 -- 172.21.15.113:0/2218448269 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97b406caa0 msgr2=0x7f97b406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.040+0000 7f97caad3700 1 --2- 172.21.15.113:0/2218448269 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97b406caa0 0x7f97b406ef60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f97c40f9ea0 tx=0x7f97b0009cb0).stop 2024-04-12T05:51:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.040+0000 7f97caad3700 1 -- 172.21.15.113:0/2218448269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97c40fca10 msgr2=0x7f97c40f8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.040+0000 7f97caad3700 1 --2- 172.21.15.113:0/2218448269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97c40fca10 0x7f97c40f8700 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7f97b80041d0 tx=0x7f97b8009f60).stop 2024-04-12T05:51:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.040+0000 7f97caad3700 1 -- 172.21.15.113:0/2218448269 shutdown_connections 2024-04-12T05:51:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.040+0000 7f97caad3700 1 --2- 172.21.15.113:0/2218448269 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f97b406caa0 0x7f97b406ef60 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.040+0000 7f97caad3700 1 --2- 172.21.15.113:0/2218448269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97c40fca10 0x7f97c40f8700 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.040+0000 7f97caad3700 1 --2- 172.21.15.113:0/2218448269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97c40fd370 0x7f97c40f8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.040+0000 7f97caad3700 1 -- 172.21.15.113:0/2218448269 >> 172.21.15.113:0/2218448269 conn(0x7f97c40f2590 msgr2=0x7f97c4060c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.040+0000 7f97caad3700 1 -- 172.21.15.113:0/2218448269 shutdown_connections 2024-04-12T05:51:21.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.040+0000 7f97caad3700 1 -- 172.21.15.113:0/2218448269 wait complete. 2024-04-12T05:51:21.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-12T05:51:21.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.170+0000 7f633d251700 1 -- 172.21.15.113:0/2540668511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63380fd100 msgr2=0x7f63380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.170+0000 7f633d251700 1 --2- 172.21.15.113:0/2540668511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63380fd100 0x7f63380f6930 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f63280047f0 tx=0x7f6328009cb0).stop 2024-04-12T05:51:21.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.170+0000 7f633d251700 1 -- 172.21.15.113:0/2540668511 shutdown_connections 2024-04-12T05:51:21.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.170+0000 7f633d251700 1 --2- 172.21.15.113:0/2540668511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63380fd100 0x7f63380f6930 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.170+0000 7f633d251700 1 --2- 172.21.15.113:0/2540668511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63380fc830 0x7f63380fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.170+0000 7f633d251700 1 -- 172.21.15.113:0/2540668511 >> 172.21.15.113:0/2540668511 conn(0x7f633806c760 msgr2=0x7f633806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:21.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.171+0000 7f633d251700 1 -- 172.21.15.113:0/2540668511 shutdown_connections 2024-04-12T05:51:21.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.171+0000 7f633d251700 1 -- 172.21.15.113:0/2540668511 wait complete. 2024-04-12T05:51:21.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.171+0000 7f633d251700 1 Processor -- start 2024-04-12T05:51:21.167 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.171+0000 7f633d251700 1 -- start start 2024-04-12T05:51:21.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f633d251700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63380fc830 0x7f63380f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:21.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f633d251700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63380fd100 0x7f63380f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:21.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f633d251700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63380f8f20 con 0x7f63380fc830 2024-04-12T05:51:21.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f633d251700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63380f9060 con 0x7f63380fd100 2024-04-12T05:51:21.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f6336d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63380fc830 0x7f63380f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f6336d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63380fc830 0x7f63380f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33852/0 (socket says 172.21.15.113:33852) 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f6336d9d700 1 -- 172.21.15.113:0/2815991855 learned_addr learned my addr 172.21.15.113:0/2815991855 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f633659c700 1 --2- 172.21.15.113:0/2815991855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63380fd100 0x7f63380f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f6336d9d700 1 -- 172.21.15.113:0/2815991855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63380fd100 msgr2=0x7f63380f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f6336d9d700 1 --2- 172.21.15.113:0/2815991855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63380fd100 0x7f63380f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f6336d9d700 1 -- 172.21.15.113:0/2815991855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6328005040 con 0x7f63380fc830 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f6336d9d700 1 --2- 172.21.15.113:0/2815991855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63380fc830 0x7f63380f8540 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f632c00c740 tx=0x7f632c007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f631ffff700 1 -- 172.21.15.113:0/2815991855 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f632c00e680 con 0x7f63380fc830 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f631ffff700 1 -- 172.21.15.113:0/2815991855 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f632c00ec90 con 0x7f63380fc830 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f631ffff700 1 -- 172.21.15.113:0/2815991855 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f632c013d30 con 0x7f63380fc830 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f633d251700 1 -- 172.21.15.113:0/2815991855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63380f9310 con 0x7f63380fc830 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.172+0000 7f633d251700 1 -- 172.21.15.113:0/2815991855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6338114650 con 0x7f63380fc830 2024-04-12T05:51:21.170 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.173+0000 7f633d251700 1 -- 172.21.15.113:0/2815991855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633805f380 con 0x7f63380fc830 2024-04-12T05:51:21.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.174+0000 7f631ffff700 1 -- 172.21.15.113:0/2815991855 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f632c013410 con 0x7f63380fc830 2024-04-12T05:51:21.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.175+0000 7f631ffff700 1 --2- 172.21.15.113:0/2815991855 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f632006caa0 0x7f632006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:21.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.175+0000 7f631ffff700 1 -- 172.21.15.113:0/2815991855 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f632c08c460 con 0x7f63380fc830 2024-04-12T05:51:21.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.175+0000 7f633659c700 1 --2- 172.21.15.113:0/2815991855 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f632006caa0 0x7f632006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:21.171 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.175+0000 7f633659c700 1 --2- 172.21.15.113:0/2815991855 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f632006caa0 0x7f632006ef60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f63380f9ce0 tx=0x7f632800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:21.174 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.179+0000 7f631ffff700 1 -- 172.21.15.113:0/2815991855 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f632c056480 con 0x7f63380fc830 2024-04-12T05:51:21.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.365+0000 7f633d251700 1 -- 172.21.15.113:0/2815991855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f633805eec0 con 0x7f63380fc830 2024-04-12T05:51:21.361 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.366+0000 7f631ffff700 1 -- 172.21.15.113:0/2815991855 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v98) v1 ==== 107+0+11 (secure 0 0 0) 0x7f632c059aa0 con 0x7f63380fc830 2024-04-12T05:51:21.361 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-04-12T05:51:21.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.368+0000 7f633d251700 1 -- 172.21.15.113:0/2815991855 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f632006caa0 msgr2=0x7f632006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.368+0000 7f633d251700 1 --2- 172.21.15.113:0/2815991855 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f632006caa0 0x7f632006ef60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f63380f9ce0 tx=0x7f632800b040).stop 2024-04-12T05:51:21.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.368+0000 7f633d251700 1 -- 172.21.15.113:0/2815991855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63380fc830 msgr2=0x7f63380f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.368+0000 7f633d251700 1 --2- 172.21.15.113:0/2815991855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63380fc830 0x7f63380f8540 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f632c00c740 tx=0x7f632c007ef0).stop 2024-04-12T05:51:21.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.368+0000 7f633d251700 1 -- 172.21.15.113:0/2815991855 shutdown_connections 2024-04-12T05:51:21.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.368+0000 7f633d251700 1 --2- 172.21.15.113:0/2815991855 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f632006caa0 0x7f632006ef60 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.368+0000 7f633d251700 1 --2- 172.21.15.113:0/2815991855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63380fc830 0x7f63380f8540 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.368+0000 7f633d251700 1 --2- 172.21.15.113:0/2815991855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63380fd100 0x7f63380f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.368+0000 7f633d251700 1 -- 172.21.15.113:0/2815991855 >> 172.21.15.113:0/2815991855 conn(0x7f633806c760 msgr2=0x7f6338060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:21.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.368+0000 7f633d251700 1 -- 172.21.15.113:0/2815991855 shutdown_connections 2024-04-12T05:51:21.364 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.368+0000 7f633d251700 1 -- 172.21.15.113:0/2815991855 wait complete. 2024-04-12T05:51:21.374 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:20 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/935269263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:51:21.375 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-12T05:51:21.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:20 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/935269263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:51:21.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.498+0000 7f67a5086700 1 -- 172.21.15.113:0/3111251876 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67a00fd150 msgr2=0x7f67a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.498+0000 7f67a5086700 1 --2- 172.21.15.113:0/3111251876 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67a00fd150 0x7f67a00f6930 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f67900047f0 tx=0x7f6790009cb0).stop 2024-04-12T05:51:21.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.498+0000 7f67a5086700 1 -- 172.21.15.113:0/3111251876 shutdown_connections 2024-04-12T05:51:21.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.498+0000 7f67a5086700 1 --2- 172.21.15.113:0/3111251876 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67a00fd150 0x7f67a00f6930 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.498+0000 7f67a5086700 1 --2- 172.21.15.113:0/3111251876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67a00fc7f0 0x7f67a00fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.498+0000 7f67a5086700 1 -- 172.21.15.113:0/3111251876 >> 172.21.15.113:0/3111251876 conn(0x7f67a00f23f0 msgr2=0x7f67a00f4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:21.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.498+0000 7f67a5086700 1 -- 172.21.15.113:0/3111251876 shutdown_connections 2024-04-12T05:51:21.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.498+0000 7f67a5086700 1 -- 172.21.15.113:0/3111251876 wait complete. 2024-04-12T05:51:21.494 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f67a5086700 1 Processor -- start 2024-04-12T05:51:21.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f67a5086700 1 -- start start 2024-04-12T05:51:21.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f67a5086700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67a00fc7f0 0x7f67a00f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:21.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f67a5086700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67a00fd150 0x7f67a00f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f67a5086700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00f8f20 con 0x7f67a00fc7f0 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f67a5086700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00f9060 con 0x7f67a00fd150 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f679ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67a00fc7f0 0x7f67a00f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f679ed9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67a00fc7f0 0x7f67a00f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33872/0 (socket says 172.21.15.113:33872) 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f679ed9d700 1 -- 172.21.15.113:0/4257544851 learned_addr learned my addr 172.21.15.113:0/4257544851 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f679e59c700 1 --2- 172.21.15.113:0/4257544851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67a00fd150 0x7f67a00f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f679ed9d700 1 -- 172.21.15.113:0/4257544851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67a00fd150 msgr2=0x7f67a00f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f679ed9d700 1 --2- 172.21.15.113:0/4257544851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67a00fd150 0x7f67a00f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.499+0000 7f679ed9d700 1 -- 172.21.15.113:0/4257544851 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6790005040 con 0x7f67a00fc7f0 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.500+0000 7f679ed9d700 1 --2- 172.21.15.113:0/4257544851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67a00fc7f0 0x7f67a00f8540 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7f67940041d0 tx=0x7f6794009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.500+0000 7f6787fff700 1 -- 172.21.15.113:0/4257544851 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6794007b40 con 0x7f67a00fc7f0 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.500+0000 7f67a5086700 1 -- 172.21.15.113:0/4257544851 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a00f9310 con 0x7f67a00fc7f0 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.500+0000 7f6787fff700 1 -- 172.21.15.113:0/4257544851 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f679400f040 con 0x7f67a00fc7f0 2024-04-12T05:51:21.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.500+0000 7f6787fff700 1 -- 172.21.15.113:0/4257544851 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67940137d0 con 0x7f67a00fc7f0 2024-04-12T05:51:21.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.500+0000 7f67a5086700 1 -- 172.21.15.113:0/4257544851 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a0114690 con 0x7f67a00fc7f0 2024-04-12T05:51:21.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.501+0000 7f67a5086700 1 -- 172.21.15.113:0/4257544851 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a005f2e0 con 0x7f67a00fc7f0 2024-04-12T05:51:21.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.502+0000 7f6787fff700 1 -- 172.21.15.113:0/4257544851 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6794007ca0 con 0x7f67a00fc7f0 2024-04-12T05:51:21.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.502+0000 7f6787fff700 1 --2- 172.21.15.113:0/4257544851 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f678806caa0 0x7f678806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:21.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.502+0000 7f6787fff700 1 -- 172.21.15.113:0/4257544851 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f679408b7a0 con 0x7f67a00fc7f0 2024-04-12T05:51:21.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.502+0000 7f679e59c700 1 --2- 172.21.15.113:0/4257544851 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f678806caa0 0x7f678806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:21.499 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.503+0000 7f679e59c700 1 --2- 172.21.15.113:0/4257544851 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f678806caa0 0x7f678806ef60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f67a00f9ce0 tx=0x7f679000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:21.501 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.506+0000 7f6787fff700 1 -- 172.21.15.113:0/4257544851 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f67940557c0 con 0x7f67a00fc7f0 2024-04-12T05:51:21.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.693+0000 7f67a5086700 1 -- 172.21.15.113:0/4257544851 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f67a005eec0 con 0x7f67a00fc7f0 2024-04-12T05:51:21.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.694+0000 7f6787fff700 1 -- 172.21.15.113:0/4257544851 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v98) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6794058de0 con 0x7f67a00fc7f0 2024-04-12T05:51:21.689 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-04-12T05:51:21.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.696+0000 7f67a5086700 1 -- 172.21.15.113:0/4257544851 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f678806caa0 msgr2=0x7f678806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.696+0000 7f67a5086700 1 --2- 172.21.15.113:0/4257544851 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f678806caa0 0x7f678806ef60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f67a00f9ce0 tx=0x7f679000b040).stop 2024-04-12T05:51:21.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.696+0000 7f67a5086700 1 -- 172.21.15.113:0/4257544851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67a00fc7f0 msgr2=0x7f67a00f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.696+0000 7f67a5086700 1 --2- 172.21.15.113:0/4257544851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67a00fc7f0 0x7f67a00f8540 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7f67940041d0 tx=0x7f6794009f60).stop 2024-04-12T05:51:21.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.696+0000 7f67a5086700 1 -- 172.21.15.113:0/4257544851 shutdown_connections 2024-04-12T05:51:21.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.696+0000 7f67a5086700 1 --2- 172.21.15.113:0/4257544851 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f678806caa0 0x7f678806ef60 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.696+0000 7f67a5086700 1 --2- 172.21.15.113:0/4257544851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67a00fc7f0 0x7f67a00f8540 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.696+0000 7f67a5086700 1 --2- 172.21.15.113:0/4257544851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67a00fd150 0x7f67a00f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.696+0000 7f67a5086700 1 -- 172.21.15.113:0/4257544851 >> 172.21.15.113:0/4257544851 conn(0x7f67a00f23f0 msgr2=0x7f67a0060e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:21.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.696+0000 7f67a5086700 1 -- 172.21.15.113:0/4257544851 shutdown_connections 2024-04-12T05:51:21.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.696+0000 7f67a5086700 1 -- 172.21.15.113:0/4257544851 wait complete. 2024-04-12T05:51:21.704 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T05:51:21.820 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.824+0000 7f67c7baf700 1 -- 172.21.15.113:0/3389353441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67c00fc730 msgr2=0x7f67c00fcb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.824+0000 7f67c7baf700 1 --2- 172.21.15.113:0/3389353441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67c00fc730 0x7f67c00fcb10 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7f67b00047f0 tx=0x7f67b0009cb0).stop 2024-04-12T05:51:21.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.825+0000 7f67c7baf700 1 -- 172.21.15.113:0/3389353441 shutdown_connections 2024-04-12T05:51:21.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.825+0000 7f67c7baf700 1 --2- 172.21.15.113:0/3389353441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c00fd090 0x7f67c00f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.825+0000 7f67c7baf700 1 --2- 172.21.15.113:0/3389353441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67c00fc730 0x7f67c00fcb10 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.825+0000 7f67c7baf700 1 -- 172.21.15.113:0/3389353441 >> 172.21.15.113:0/3389353441 conn(0x7f67c006c760 msgr2=0x7f67c006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:21.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.825+0000 7f67c7baf700 1 -- 172.21.15.113:0/3389353441 shutdown_connections 2024-04-12T05:51:21.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.825+0000 7f67c7baf700 1 -- 172.21.15.113:0/3389353441 wait complete. 2024-04-12T05:51:21.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.825+0000 7f67c7baf700 1 Processor -- start 2024-04-12T05:51:21.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.825+0000 7f67c7baf700 1 -- start start 2024-04-12T05:51:21.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c7baf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c00fc730 0x7f67c010cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c7baf700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67c00fd090 0x7f67c010d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c7baf700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c010d8e0 con 0x7f67c00fd090 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c7baf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c010da20 con 0x7f67c00fc730 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c514a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67c00fd090 0x7f67c010d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c514a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67c00fd090 0x7f67c010d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33894/0 (socket says 172.21.15.113:33894) 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c514a700 1 -- 172.21.15.113:0/2713552448 learned_addr learned my addr 172.21.15.113:0/2713552448 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c514a700 1 -- 172.21.15.113:0/2713552448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c00fc730 msgr2=0x7f67c010cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c594b700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c00fc730 0x7f67c010cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c514a700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c00fc730 0x7f67c010cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c514a700 1 -- 172.21.15.113:0/2713552448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b0005040 con 0x7f67c00fd090 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c594b700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c00fc730 0x7f67c010cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c514a700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67c00fd090 0x7f67c010d2e0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f67bc00ea20 tx=0x7f67bc007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67b6ffd700 1 -- 172.21.15.113:0/2713552448 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67bc00cb90 con 0x7f67c00fd090 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67b6ffd700 1 -- 172.21.15.113:0/2713552448 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67bc00ccf0 con 0x7f67c00fd090 2024-04-12T05:51:21.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c7baf700 1 -- 172.21.15.113:0/2713552448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c01070b0 con 0x7f67c00fd090 2024-04-12T05:51:21.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67b6ffd700 1 -- 172.21.15.113:0/2713552448 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67bc010630 con 0x7f67c00fd090 2024-04-12T05:51:21.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.826+0000 7f67c7baf700 1 -- 172.21.15.113:0/2713552448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c0107670 con 0x7f67c00fd090 2024-04-12T05:51:21.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.827+0000 7f67c7baf700 1 -- 172.21.15.113:0/2713552448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c005f380 con 0x7f67c00fd090 2024-04-12T05:51:21.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.828+0000 7f67b6ffd700 1 -- 172.21.15.113:0/2713552448 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f67bc011470 con 0x7f67c00fd090 2024-04-12T05:51:21.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.828+0000 7f67b6ffd700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f67ac06caa0 0x7f67ac06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:21.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.828+0000 7f67c594b700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f67ac06caa0 0x7f67ac06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:21.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.828+0000 7f67b6ffd700 1 -- 172.21.15.113:0/2713552448 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f67bc08e710 con 0x7f67c00fd090 2024-04-12T05:51:21.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.829+0000 7f67c594b700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f67ac06caa0 0x7f67ac06ef60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f67b00043d0 tx=0x7f67b000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:21.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:21.833+0000 7f67b6ffd700 1 -- 172.21.15.113:0/2713552448 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f67bc05c1c0 con 0x7f67c00fd090 2024-04-12T05:51:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.071+0000 7f67c7baf700 1 -- 172.21.15.113:0/2713552448 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67c005eec0 con 0x7f67c00fd090 2024-04-12T05:51:22.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.072+0000 7f67b6ffd700 1 -- 172.21.15.113:0/2713552448 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4848 (secure 0 0 0) 0x7f67bc05bd50 con 0x7f67c00fd090 2024-04-12T05:51:22.067 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:22.068 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44331,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/335558856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":335558856},{"type":"v1","addr":"172.21.15.192:6825","nonce":335558856}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44817,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/946017279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946017279},{"type":"v1","addr":"172.21.15.192:6827","nonce":946017279}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:49:50.608720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34412,"mds_1":34444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34412":{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":0,"incarnation":30,"state":"up:active","state_seq":468,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34444":{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":1,"incarnation":30,"state":"up:active","state_seq":463,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:22.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.074+0000 7f67c7baf700 1 -- 172.21.15.113:0/2713552448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f67ac06caa0 msgr2=0x7f67ac06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.074+0000 7f67c7baf700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f67ac06caa0 0x7f67ac06ef60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f67b00043d0 tx=0x7f67b000b040).stop 2024-04-12T05:51:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.074+0000 7f67c7baf700 1 -- 172.21.15.113:0/2713552448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67c00fd090 msgr2=0x7f67c010d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.074+0000 7f67c7baf700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67c00fd090 0x7f67c010d2e0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f67bc00ea20 tx=0x7f67bc007f50).stop 2024-04-12T05:51:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.074+0000 7f67c7baf700 1 -- 172.21.15.113:0/2713552448 shutdown_connections 2024-04-12T05:51:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.074+0000 7f67c7baf700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f67ac06caa0 0x7f67ac06ef60 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.074+0000 7f67c7baf700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c00fc730 0x7f67c010cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.074+0000 7f67c7baf700 1 --2- 172.21.15.113:0/2713552448 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67c00fd090 0x7f67c010d2e0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.074+0000 7f67c7baf700 1 -- 172.21.15.113:0/2713552448 >> 172.21.15.113:0/2713552448 conn(0x7f67c006c760 msgr2=0x7f67c00faaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.074+0000 7f67c7baf700 1 -- 172.21.15.113:0/2713552448 shutdown_connections 2024-04-12T05:51:22.070 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.074+0000 7f67c7baf700 1 -- 172.21.15.113:0/2713552448 wait complete. 2024-04-12T05:51:22.071 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 35 2024-04-12T05:51:22.081 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 14, 'max_mds': 2, 'flags': 18} 2024-04-12T05:51:22.081 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 15 2024-04-12T05:51:22.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.202+0000 7fd37b3da700 1 -- 172.21.15.113:0/3413677862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3740fc890 msgr2=0x7fd3740fcc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.202+0000 7fd37b3da700 1 --2- 172.21.15.113:0/3413677862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3740fc890 0x7fd3740fcc70 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0x7fd3640047f0 tx=0x7fd364009cb0).stop 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.202+0000 7fd37b3da700 1 -- 172.21.15.113:0/3413677862 shutdown_connections 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.202+0000 7fd37b3da700 1 --2- 172.21.15.113:0/3413677862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3740fd1f0 0x7fd3740f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.202+0000 7fd37b3da700 1 --2- 172.21.15.113:0/3413677862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3740fc890 0x7fd3740fcc70 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.202+0000 7fd37b3da700 1 -- 172.21.15.113:0/3413677862 >> 172.21.15.113:0/3413677862 conn(0x7fd3740f2490 msgr2=0x7fd3740f48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.202+0000 7fd37b3da700 1 -- 172.21.15.113:0/3413677862 shutdown_connections 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.202+0000 7fd37b3da700 1 -- 172.21.15.113:0/3413677862 wait complete. 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd37b3da700 1 Processor -- start 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd37b3da700 1 -- start start 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd37b3da700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3740fc890 0x7fd37410cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd37b3da700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3740fd1f0 0x7fd37410d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:22.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd37b3da700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37410da20 con 0x7fd3740fc890 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd37b3da700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37410db60 con 0x7fd3740fd1f0 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd379176700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3740fc890 0x7fd37410cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd379176700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3740fc890 0x7fd37410cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33904/0 (socket says 172.21.15.113:33904) 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd379176700 1 -- 172.21.15.113:0/971698443 learned_addr learned my addr 172.21.15.113:0/971698443 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd378975700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3740fd1f0 0x7fd37410d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd379176700 1 -- 172.21.15.113:0/971698443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3740fd1f0 msgr2=0x7fd37410d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd379176700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3740fd1f0 0x7fd37410d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd379176700 1 -- 172.21.15.113:0/971698443 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd364005040 con 0x7fd3740fc890 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd378975700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3740fd1f0 0x7fd37410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd379176700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3740fc890 0x7fd37410cf30 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7fd364005010 tx=0x7fd364009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd36a7fc700 1 -- 172.21.15.113:0/971698443 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd36400e030 con 0x7fd3740fc890 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd36a7fc700 1 -- 172.21.15.113:0/971698443 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd364007e90 con 0x7fd3740fc890 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.203+0000 7fd37b3da700 1 -- 172.21.15.113:0/971698443 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd374107190 con 0x7fd3740fc890 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.204+0000 7fd36a7fc700 1 -- 172.21.15.113:0/971698443 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd364018070 con 0x7fd3740fc890 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.204+0000 7fd37b3da700 1 -- 172.21.15.113:0/971698443 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3741076f0 con 0x7fd3740fc890 2024-04-12T05:51:22.201 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.204+0000 7fd37b3da700 1 -- 172.21.15.113:0/971698443 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37405f2e0 con 0x7fd3740fc890 2024-04-12T05:51:22.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.205+0000 7fd36a7fc700 1 -- 172.21.15.113:0/971698443 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd36401d6e0 con 0x7fd3740fc890 2024-04-12T05:51:22.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.205+0000 7fd36a7fc700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd36006ca30 0x7fd36006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:22.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.205+0000 7fd36a7fc700 1 -- 172.21.15.113:0/971698443 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd36408d100 con 0x7fd3740fc890 2024-04-12T05:51:22.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.205+0000 7fd378975700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd36006ca30 0x7fd36006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:22.203 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.206+0000 7fd378975700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd36006ca30 0x7fd36006eef0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fd374108870 tx=0x7fd370009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:22.205 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.210+0000 7fd36a7fc700 1 -- 172.21.15.113:0/971698443 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd36405a920 con 0x7fd3740fc890 2024-04-12T05:51:22.263 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:21 smithi113 ceph-mon[103371]: pgmap v1178: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:22.263 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:21 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2218448269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T05:51:22.263 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:21 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2815991855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T05:51:22.264 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:21 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/4257544851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T05:51:22.264 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:21 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:22.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.441+0000 7fd37b3da700 1 -- 172.21.15.113:0/971698443 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 15, "format": "json"} v 0) v1 -- 0x7fd37405eec0 con 0x7fd3740fc890 2024-04-12T05:51:22.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.442+0000 7fd36a7fc700 1 -- 172.21.15.113:0/971698443 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 15, "format": "json"}]=0 dumped fsmap epoch 15 v35) v1 ==== 107+0+4848 (secure 0 0 0) 0x7fd36405a740 con 0x7fd3740fc890 2024-04-12T05:51:22.438 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:22.438 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14438,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/2511637182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2511637182},{"type":"v1","addr":"172.21.15.113:6829","nonce":2511637182}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:07:56.641492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14426,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14426":{"gid":14426,"name":"cephfs.smithi113.ovmadh","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.113:6827/1262609849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1262609849},{"type":"v1","addr":"172.21.15.113:6827","nonce":1262609849}]},"join_fscid":1,"export_targets":[1],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi192.ammjap","rank":1,"incarnation":9,"state":"up:active","state_seq":2,"addr":"172.21.15.192:6825/616350866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":616350866},{"type":"v1","addr":"172.21.15.192:6825","nonce":616350866}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.444+0000 7fd37b3da700 1 -- 172.21.15.113:0/971698443 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd36006ca30 msgr2=0x7fd36006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.444+0000 7fd37b3da700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd36006ca30 0x7fd36006eef0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fd374108870 tx=0x7fd370009d60).stop 2024-04-12T05:51:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.444+0000 7fd37b3da700 1 -- 172.21.15.113:0/971698443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3740fc890 msgr2=0x7fd37410cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.444+0000 7fd37b3da700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3740fc890 0x7fd37410cf30 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7fd364005010 tx=0x7fd364009cb0).stop 2024-04-12T05:51:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.444+0000 7fd37b3da700 1 -- 172.21.15.113:0/971698443 shutdown_connections 2024-04-12T05:51:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.444+0000 7fd37b3da700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd36006ca30 0x7fd36006eef0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.444+0000 7fd37b3da700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3740fc890 0x7fd37410cf30 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.444+0000 7fd37b3da700 1 --2- 172.21.15.113:0/971698443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3740fd1f0 0x7fd37410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.444+0000 7fd37b3da700 1 -- 172.21.15.113:0/971698443 >> 172.21.15.113:0/971698443 conn(0x7fd3740f2490 msgr2=0x7fd3740fac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.444+0000 7fd37b3da700 1 -- 172.21.15.113:0/971698443 shutdown_connections 2024-04-12T05:51:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.444+0000 7fd37b3da700 1 -- 172.21.15.113:0/971698443 wait complete. 2024-04-12T05:51:22.441 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 15 2024-04-12T05:51:22.441 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:21 smithi192 ceph-mon[85304]: pgmap v1178: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:22.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:21 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2218448269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T05:51:22.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:21 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2815991855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T05:51:22.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:21 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/4257544851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T05:51:22.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:21 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:22.454 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 16 2024-04-12T05:51:22.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.576+0000 7f53fb440700 1 -- 172.21.15.113:0/859275821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f40fc830 msgr2=0x7f53f40fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.576+0000 7f53fb440700 1 --2- 172.21.15.113:0/859275821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f40fc830 0x7f53f40fcc10 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f53f00047f0 tx=0x7f53f0009cb0).stop 2024-04-12T05:51:22.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.576+0000 7f53fb440700 1 -- 172.21.15.113:0/859275821 shutdown_connections 2024-04-12T05:51:22.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.576+0000 7f53fb440700 1 --2- 172.21.15.113:0/859275821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53f40fd100 0x7f53f40f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.576+0000 7f53fb440700 1 --2- 172.21.15.113:0/859275821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f40fc830 0x7f53f40fcc10 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.576+0000 7f53fb440700 1 -- 172.21.15.113:0/859275821 >> 172.21.15.113:0/859275821 conn(0x7f53f406c760 msgr2=0x7f53f406cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:22.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.576+0000 7f53fb440700 1 -- 172.21.15.113:0/859275821 shutdown_connections 2024-04-12T05:51:22.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.576+0000 7f53fb440700 1 -- 172.21.15.113:0/859275821 wait complete. 2024-04-12T05:51:22.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53fb440700 1 Processor -- start 2024-04-12T05:51:22.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53fb440700 1 -- start start 2024-04-12T05:51:22.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53fb440700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f40fc830 0x7f53f4109090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:22.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53fb440700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53f40fd100 0x7f53f4109580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:22.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53fb440700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f4109a70 con 0x7f53f40fc830 2024-04-12T05:51:22.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53fb440700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f4109bb0 con 0x7f53f40fd100 2024-04-12T05:51:22.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53f91dc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f40fc830 0x7f53f4109090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:22.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53f91dc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f40fc830 0x7f53f4109090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33922/0 (socket says 172.21.15.113:33922) 2024-04-12T05:51:22.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53f91dc700 1 -- 172.21.15.113:0/1869389466 learned_addr learned my addr 172.21.15.113:0/1869389466 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:22.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53f89db700 1 --2- 172.21.15.113:0/1869389466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53f40fd100 0x7f53f4109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:22.575 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53f91dc700 1 -- 172.21.15.113:0/1869389466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53f40fd100 msgr2=0x7f53f4109580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53f91dc700 1 --2- 172.21.15.113:0/1869389466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53f40fd100 0x7f53f4109580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53f91dc700 1 -- 172.21.15.113:0/1869389466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f0005040 con 0x7f53f40fc830 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.577+0000 7f53f91dc700 1 --2- 172.21.15.113:0/1869389466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f40fc830 0x7f53f4109090 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f53f0005010 tx=0x7f53f0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.578+0000 7f53e67fc700 1 -- 172.21.15.113:0/1869389466 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53f000e030 con 0x7f53f40fc830 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.578+0000 7f53e67fc700 1 -- 172.21.15.113:0/1869389466 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53f0007e90 con 0x7f53f40fc830 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.578+0000 7f53e67fc700 1 -- 172.21.15.113:0/1869389466 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53f0018070 con 0x7f53f40fc830 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.578+0000 7f53fb440700 1 -- 172.21.15.113:0/1869389466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f4109e00 con 0x7f53f40fc830 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.578+0000 7f53fb440700 1 -- 172.21.15.113:0/1869389466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f41123f0 con 0x7f53f40fc830 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.579+0000 7f53fb440700 1 -- 172.21.15.113:0/1869389466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f405f380 con 0x7f53f40fc830 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.579+0000 7f53e67fc700 1 -- 172.21.15.113:0/1869389466 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f53f001d420 con 0x7f53f40fc830 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.579+0000 7f53e67fc700 1 --2- 172.21.15.113:0/1869389466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f53e006caa0 0x7f53e006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.580+0000 7f53e67fc700 1 -- 172.21.15.113:0/1869389466 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f53f008d180 con 0x7f53f40fc830 2024-04-12T05:51:22.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.580+0000 7f53f89db700 1 --2- 172.21.15.113:0/1869389466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f53e006caa0 0x7f53e006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:22.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.580+0000 7f53f89db700 1 --2- 172.21.15.113:0/1869389466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f53e006caa0 0x7f53e006ef60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f53f4060f00 tx=0x7f53e8009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:22.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.584+0000 7f53e67fc700 1 -- 172.21.15.113:0/1869389466 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f53f00571a0 con 0x7f53f40fc830 2024-04-12T05:51:22.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.817+0000 7f53fb440700 1 -- 172.21.15.113:0/1869389466 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 16, "format": "json"} v 0) v1 -- 0x7f53f40f9960 con 0x7f53f40fc830 2024-04-12T05:51:22.813 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.818+0000 7f53e67fc700 1 -- 172.21.15.113:0/1869389466 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 16, "format": "json"}]=0 dumped fsmap epoch 16 v35) v1 ==== 107+0+4849 (secure 0 0 0) 0x7f53f005a7c0 con 0x7f53f40fc830 2024-04-12T05:51:22.814 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:22.814 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14438,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/2511637182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2511637182},{"type":"v1","addr":"172.21.15.113:6829","nonce":2511637182}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:08:59.962849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14426,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14426":{"gid":14426,"name":"cephfs.smithi113.ovmadh","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.113:6827/1262609849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1262609849},{"type":"v1","addr":"172.21.15.113:6827","nonce":1262609849}]},"join_fscid":1,"export_targets":[1],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi192.ammjap","rank":1,"incarnation":9,"state":"up:active","state_seq":2,"addr":"172.21.15.192:6825/616350866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":616350866},{"type":"v1","addr":"172.21.15.192:6825","nonce":616350866}]},"join_fscid":1,"export_targets":[0],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:22.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.820+0000 7f53fb440700 1 -- 172.21.15.113:0/1869389466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f53e006caa0 msgr2=0x7f53e006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.820+0000 7f53fb440700 1 --2- 172.21.15.113:0/1869389466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f53e006caa0 0x7f53e006ef60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f53f4060f00 tx=0x7f53e8009da0).stop 2024-04-12T05:51:22.816 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.820+0000 7f53fb440700 1 -- 172.21.15.113:0/1869389466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f40fc830 msgr2=0x7f53f4109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.820+0000 7f53fb440700 1 --2- 172.21.15.113:0/1869389466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f40fc830 0x7f53f4109090 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f53f0005010 tx=0x7f53f0009cb0).stop 2024-04-12T05:51:22.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.821+0000 7f53fb440700 1 -- 172.21.15.113:0/1869389466 shutdown_connections 2024-04-12T05:51:22.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.821+0000 7f53fb440700 1 --2- 172.21.15.113:0/1869389466 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f53e006caa0 0x7f53e006ef60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.821+0000 7f53fb440700 1 --2- 172.21.15.113:0/1869389466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f40fc830 0x7f53f4109090 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.821+0000 7f53fb440700 1 --2- 172.21.15.113:0/1869389466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f53f40fd100 0x7f53f4109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.821+0000 7f53fb440700 1 -- 172.21.15.113:0/1869389466 >> 172.21.15.113:0/1869389466 conn(0x7f53f406c760 msgr2=0x7f53f40ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:22.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.821+0000 7f53fb440700 1 -- 172.21.15.113:0/1869389466 shutdown_connections 2024-04-12T05:51:22.817 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.821+0000 7f53fb440700 1 -- 172.21.15.113:0/1869389466 wait complete. 2024-04-12T05:51:22.817 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 16 2024-04-12T05:51:22.828 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 17 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.950+0000 7f13a8488700 1 -- 172.21.15.113:0/2717977120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13a00fc830 msgr2=0x7f13a00fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.950+0000 7f13a8488700 1 --2- 172.21.15.113:0/2717977120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13a00fc830 0x7f13a00fcc10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7f13900047f0 tx=0x7f1390009cb0).stop 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.950+0000 7f13a8488700 1 -- 172.21.15.113:0/2717977120 shutdown_connections 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.950+0000 7f13a8488700 1 --2- 172.21.15.113:0/2717977120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13a00fd100 0x7f13a00f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.950+0000 7f13a8488700 1 --2- 172.21.15.113:0/2717977120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13a00fc830 0x7f13a00fcc10 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.950+0000 7f13a8488700 1 -- 172.21.15.113:0/2717977120 >> 172.21.15.113:0/2717977120 conn(0x7f13a006c760 msgr2=0x7f13a006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.950+0000 7f13a8488700 1 -- 172.21.15.113:0/2717977120 shutdown_connections 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.950+0000 7f13a8488700 1 -- 172.21.15.113:0/2717977120 wait complete. 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.951+0000 7f13a8488700 1 Processor -- start 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.951+0000 7f13a8488700 1 -- start start 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.951+0000 7f13a8488700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13a00fc830 0x7f13a0109140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:22.948 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.951+0000 7f13a8488700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13a00fd100 0x7f13a0109630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:22.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.951+0000 7f13a8488700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0109bb0 con 0x7f13a00fc830 2024-04-12T05:51:22.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.951+0000 7f13a8488700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0109cf0 con 0x7f13a00fd100 2024-04-12T05:51:22.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.951+0000 7f13a6224700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13a00fc830 0x7f13a0109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:22.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.951+0000 7f13a6224700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13a00fc830 0x7f13a0109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33926/0 (socket says 172.21.15.113:33926) 2024-04-12T05:51:22.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.951+0000 7f13a6224700 1 -- 172.21.15.113:0/1976276101 learned_addr learned my addr 172.21.15.113:0/1976276101 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:22.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.951+0000 7f13a6224700 1 -- 172.21.15.113:0/1976276101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13a00fd100 msgr2=0x7f13a0109630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:22.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.951+0000 7f13a6224700 1 --2- 172.21.15.113:0/1976276101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13a00fd100 0x7f13a0109630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:22.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.952+0000 7f13a6224700 1 -- 172.21.15.113:0/1976276101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1390005040 con 0x7f13a00fc830 2024-04-12T05:51:22.949 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.952+0000 7f13a6224700 1 --2- 172.21.15.113:0/1976276101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13a00fc830 0x7f13a0109140 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7f139000b120 tx=0x7f1390009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:22.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.952+0000 7f13977fe700 1 -- 172.21.15.113:0/1976276101 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f139000e030 con 0x7f13a00fc830 2024-04-12T05:51:22.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.952+0000 7f13977fe700 1 -- 172.21.15.113:0/1976276101 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1390007e90 con 0x7f13a00fc830 2024-04-12T05:51:22.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.952+0000 7f13977fe700 1 -- 172.21.15.113:0/1976276101 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1390018070 con 0x7f13a00fc830 2024-04-12T05:51:22.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.952+0000 7f13a8488700 1 -- 172.21.15.113:0/1976276101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a0109f40 con 0x7f13a00fc830 2024-04-12T05:51:22.950 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.952+0000 7f13a8488700 1 -- 172.21.15.113:0/1976276101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a0112480 con 0x7f13a00fc830 2024-04-12T05:51:22.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.953+0000 7f13a8488700 1 -- 172.21.15.113:0/1976276101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a005f380 con 0x7f13a00fc830 2024-04-12T05:51:22.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.953+0000 7f13977fe700 1 -- 172.21.15.113:0/1976276101 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1390009710 con 0x7f13a00fc830 2024-04-12T05:51:22.951 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.954+0000 7f13977fe700 1 --2- 172.21.15.113:0/1976276101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f138c06caa0 0x7f138c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:22.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.954+0000 7f13977fe700 1 -- 172.21.15.113:0/1976276101 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f139008d2f0 con 0x7f13a00fc830 2024-04-12T05:51:22.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.954+0000 7f13a5a23700 1 --2- 172.21.15.113:0/1976276101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f138c06caa0 0x7f138c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:22.952 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.954+0000 7f13a5a23700 1 --2- 172.21.15.113:0/1976276101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f138c06caa0 0x7f138c06ef60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f13a0060f00 tx=0x7f139c006c30).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:22.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:22.958+0000 7f13977fe700 1 -- 172.21.15.113:0/1976276101 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f1390057310 con 0x7f13a00fc830 2024-04-12T05:51:23.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:22 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2713552448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T05:51:23.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:22 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/971698443' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 15, "format": "json"}]: dispatch 2024-04-12T05:51:23.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:22 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1869389466' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-04-12T05:51:23.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.195+0000 7f13a8488700 1 -- 172.21.15.113:0/1976276101 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 17, "format": "json"} v 0) v1 -- 0x7f13a00f9960 con 0x7f13a00fc830 2024-04-12T05:51:23.192 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.195+0000 7f13977fe700 1 -- 172.21.15.113:0/1976276101 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 17, "format": "json"}]=0 dumped fsmap epoch 17 v35) v1 ==== 107+0+4848 (secure 0 0 0) 0x7f139005a930 con 0x7f13a00fc830 2024-04-12T05:51:23.193 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:23.193 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14438,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/2511637182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2511637182},{"type":"v1","addr":"172.21.15.113:6829","nonce":2511637182}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:12:02.055397+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14426,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14426":{"gid":14426,"name":"cephfs.smithi113.ovmadh","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.113:6827/1262609849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1262609849},{"type":"v1","addr":"172.21.15.113:6827","nonce":1262609849}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi192.ammjap","rank":1,"incarnation":9,"state":"up:active","state_seq":2,"addr":"172.21.15.192:6825/616350866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":616350866},{"type":"v1","addr":"172.21.15.192:6825","nonce":616350866}]},"join_fscid":1,"export_targets":[0],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:23.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.198+0000 7f13a8488700 1 -- 172.21.15.113:0/1976276101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f138c06caa0 msgr2=0x7f138c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:23.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.198+0000 7f13a8488700 1 --2- 172.21.15.113:0/1976276101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f138c06caa0 0x7f138c06ef60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f13a0060f00 tx=0x7f139c006c30).stop 2024-04-12T05:51:23.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.198+0000 7f13a8488700 1 -- 172.21.15.113:0/1976276101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13a00fc830 msgr2=0x7f13a0109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:23.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.198+0000 7f13a8488700 1 --2- 172.21.15.113:0/1976276101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13a00fc830 0x7f13a0109140 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7f139000b120 tx=0x7f1390009cb0).stop 2024-04-12T05:51:23.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.198+0000 7f13a8488700 1 -- 172.21.15.113:0/1976276101 shutdown_connections 2024-04-12T05:51:23.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.198+0000 7f13a8488700 1 --2- 172.21.15.113:0/1976276101 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f138c06caa0 0x7f138c06ef60 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:23.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.198+0000 7f13a8488700 1 --2- 172.21.15.113:0/1976276101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13a00fc830 0x7f13a0109140 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:23.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.198+0000 7f13a8488700 1 --2- 172.21.15.113:0/1976276101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13a00fd100 0x7f13a0109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:23.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.198+0000 7f13a8488700 1 -- 172.21.15.113:0/1976276101 >> 172.21.15.113:0/1976276101 conn(0x7f13a006c760 msgr2=0x7f13a00ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:23.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.198+0000 7f13a8488700 1 -- 172.21.15.113:0/1976276101 shutdown_connections 2024-04-12T05:51:23.194 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.198+0000 7f13a8488700 1 -- 172.21.15.113:0/1976276101 wait complete. 2024-04-12T05:51:23.195 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 17 2024-04-12T05:51:23.205 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 18 2024-04-12T05:51:23.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.328+0000 7fec0cace700 1 -- 172.21.15.113:0/4086652480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec080fc830 msgr2=0x7fec080fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:23.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.328+0000 7fec0cace700 1 --2- 172.21.15.113:0/4086652480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec080fc830 0x7fec080fcc10 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0x7febf80047f0 tx=0x7febf8009cb0).stop 2024-04-12T05:51:23.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.328+0000 7fec0cace700 1 -- 172.21.15.113:0/4086652480 shutdown_connections 2024-04-12T05:51:23.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.328+0000 7fec0cace700 1 --2- 172.21.15.113:0/4086652480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec080fd100 0x7fec080f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:23.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.328+0000 7fec0cace700 1 --2- 172.21.15.113:0/4086652480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec080fc830 0x7fec080fcc10 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:23.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.328+0000 7fec0cace700 1 -- 172.21.15.113:0/4086652480 >> 172.21.15.113:0/4086652480 conn(0x7fec0806c760 msgr2=0x7fec0806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:23.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.328+0000 7fec0cace700 1 -- 172.21.15.113:0/4086652480 shutdown_connections 2024-04-12T05:51:23.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.329+0000 7fec0cace700 1 -- 172.21.15.113:0/4086652480 wait complete. 2024-04-12T05:51:23.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.329+0000 7fec0cace700 1 Processor -- start 2024-04-12T05:51:23.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.329+0000 7fec0cace700 1 -- start start 2024-04-12T05:51:23.326 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.329+0000 7fec0cace700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec080fc830 0x7fec0810cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.329+0000 7fec0cace700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec080fd100 0x7fec0810d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.329+0000 7fec0cace700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0810dad0 con 0x7fec080fd100 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.329+0000 7fec0cace700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec081070f0 con 0x7fec080fc830 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7fec05d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec080fd100 0x7fec0810d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7fec05d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec080fd100 0x7fec0810d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33940/0 (socket says 172.21.15.113:33940) 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7fec05d9b700 1 -- 172.21.15.113:0/3398855912 learned_addr learned my addr 172.21.15.113:0/3398855912 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7fec05d9b700 1 -- 172.21.15.113:0/3398855912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec080fc830 msgr2=0x7fec0810cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7fec05d9b700 1 --2- 172.21.15.113:0/3398855912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec080fc830 0x7fec0810cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7fec05d9b700 1 -- 172.21.15.113:0/3398855912 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf8005040 con 0x7fec080fd100 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7fec05d9b700 1 --2- 172.21.15.113:0/3398855912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec080fd100 0x7fec0810d440 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7febfc00b4b0 tx=0x7febfc0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7febf77fe700 1 -- 172.21.15.113:0/3398855912 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febfc002500 con 0x7fec080fd100 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7fec0cace700 1 -- 172.21.15.113:0/3398855912 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec081073a0 con 0x7fec080fd100 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7febf77fe700 1 -- 172.21.15.113:0/3398855912 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febfc002b10 con 0x7fec080fd100 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7febf77fe700 1 -- 172.21.15.113:0/3398855912 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febfc012cf0 con 0x7fec080fd100 2024-04-12T05:51:23.327 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.330+0000 7fec0cace700 1 -- 172.21.15.113:0/3398855912 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec081078e0 con 0x7fec080fd100 2024-04-12T05:51:23.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.331+0000 7fec0cace700 1 -- 172.21.15.113:0/3398855912 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec0805f380 con 0x7fec080fd100 2024-04-12T05:51:23.328 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.331+0000 7febf77fe700 1 -- 172.21.15.113:0/3398855912 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7febfc002660 con 0x7fec080fd100 2024-04-12T05:51:23.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.332+0000 7febf77fe700 1 --2- 172.21.15.113:0/3398855912 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7febf006cab0 0x7febf006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:23.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.332+0000 7febf77fe700 1 -- 172.21.15.113:0/3398855912 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7febfc08bf30 con 0x7fec080fd100 2024-04-12T05:51:23.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.332+0000 7fec0659c700 1 --2- 172.21.15.113:0/3398855912 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7febf006cab0 0x7febf006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:23.329 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.332+0000 7fec0659c700 1 --2- 172.21.15.113:0/3398855912 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7febf006cab0 0x7febf006ef70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7febf8004710 tx=0x7febf8009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:23.332 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.336+0000 7febf77fe700 1 -- 172.21.15.113:0/3398855912 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7febfc0bf2c0 con 0x7fec080fd100 2024-04-12T05:51:23.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:23 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2713552448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T05:51:23.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:23 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/971698443' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 15, "format": "json"}]: dispatch 2024-04-12T05:51:23.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:23 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1869389466' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-04-12T05:51:23.563 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.568+0000 7fec0cace700 1 -- 172.21.15.113:0/3398855912 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 18, "format": "json"} v 0) v1 -- 0x7fec0805eec0 con 0x7fec080fd100 2024-04-12T05:51:23.564 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.568+0000 7febf77fe700 1 -- 172.21.15.113:0/3398855912 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 18, "format": "json"}]=0 dumped fsmap epoch 18 v35) v1 ==== 107+0+4847 (secure 0 0 0) 0x7febfc0599e0 con 0x7fec080fd100 2024-04-12T05:51:23.564 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:23.564 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14438,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/2511637182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2511637182},{"type":"v1","addr":"172.21.15.113:6829","nonce":2511637182}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:12:34.568261+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14426,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14426":{"gid":14426,"name":"cephfs.smithi113.ovmadh","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.113:6827/1262609849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":1262609849},{"type":"v1","addr":"172.21.15.113:6827","nonce":1262609849}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi192.ammjap","rank":1,"incarnation":9,"state":"up:active","state_seq":2,"addr":"172.21.15.192:6825/616350866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":616350866},{"type":"v1","addr":"172.21.15.192:6825","nonce":616350866}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:23.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.571+0000 7fec0cace700 1 -- 172.21.15.113:0/3398855912 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7febf006cab0 msgr2=0x7febf006ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:23.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.571+0000 7fec0cace700 1 --2- 172.21.15.113:0/3398855912 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7febf006cab0 0x7febf006ef70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7febf8004710 tx=0x7febf8009cb0).stop 2024-04-12T05:51:23.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.571+0000 7fec0cace700 1 -- 172.21.15.113:0/3398855912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec080fd100 msgr2=0x7fec0810d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:23.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.571+0000 7fec0cace700 1 --2- 172.21.15.113:0/3398855912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec080fd100 0x7fec0810d440 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7febfc00b4b0 tx=0x7febfc0090f0).stop 2024-04-12T05:51:23.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.571+0000 7fec0cace700 1 -- 172.21.15.113:0/3398855912 shutdown_connections 2024-04-12T05:51:23.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.571+0000 7fec0cace700 1 --2- 172.21.15.113:0/3398855912 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7febf006cab0 0x7febf006ef70 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:23.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.571+0000 7fec0cace700 1 --2- 172.21.15.113:0/3398855912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec080fc830 0x7fec0810cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:23.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.571+0000 7fec0cace700 1 --2- 172.21.15.113:0/3398855912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec080fd100 0x7fec0810d440 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:23.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.571+0000 7fec0cace700 1 -- 172.21.15.113:0/3398855912 >> 172.21.15.113:0/3398855912 conn(0x7fec0806c760 msgr2=0x7fec080ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:23.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.571+0000 7fec0cace700 1 -- 172.21.15.113:0/3398855912 shutdown_connections 2024-04-12T05:51:23.567 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.571+0000 7fec0cace700 1 -- 172.21.15.113:0/3398855912 wait complete. 2024-04-12T05:51:23.567 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 18 2024-04-12T05:51:23.579 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-04-12T05:51:23.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.699+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1215761049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39c80fd100 msgr2=0x7f39c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.699+0000 7f39cfd7d700 1 --2- 172.21.15.113:0/1215761049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39c80fd100 0x7f39c80f6930 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f39c40047f0 tx=0x7f39c4009cb0).stop 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.699+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1215761049 shutdown_connections 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.699+0000 7f39cfd7d700 1 --2- 172.21.15.113:0/1215761049 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39c80fd100 0x7f39c80f6930 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.699+0000 7f39cfd7d700 1 --2- 172.21.15.113:0/1215761049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39c80fc830 0x7f39c80fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.699+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1215761049 >> 172.21.15.113:0/1215761049 conn(0x7f39c806c760 msgr2=0x7f39c806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.699+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1215761049 shutdown_connections 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.699+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1215761049 wait complete. 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cfd7d700 1 Processor -- start 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cfd7d700 1 -- start start 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cfd7d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39c80fc830 0x7f39c810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cfd7d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39c80fd100 0x7f39c810e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:23.697 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cfd7d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c810ead0 con 0x7f39c80fc830 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cfd7d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c810ec10 con 0x7f39c80fd100 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cdb19700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39c80fc830 0x7f39c810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cdb19700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39c80fc830 0x7f39c810dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33970/0 (socket says 172.21.15.113:33970) 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cdb19700 1 -- 172.21.15.113:0/1354215938 learned_addr learned my addr 172.21.15.113:0/1354215938 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cd318700 1 --2- 172.21.15.113:0/1354215938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39c80fd100 0x7f39c810e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cdb19700 1 -- 172.21.15.113:0/1354215938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39c80fd100 msgr2=0x7f39c810e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cdb19700 1 --2- 172.21.15.113:0/1354215938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39c80fd100 0x7f39c810e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cdb19700 1 -- 172.21.15.113:0/1354215938 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c4005040 con 0x7f39c80fc830 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.700+0000 7f39cdb19700 1 --2- 172.21.15.113:0/1354215938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39c80fc830 0x7f39c810dfe0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7f39b80041d0 tx=0x7f39b8009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.701+0000 7f39beffd700 1 -- 172.21.15.113:0/1354215938 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39b8007b40 con 0x7f39c80fc830 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.701+0000 7f39beffd700 1 -- 172.21.15.113:0/1354215938 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39b800f040 con 0x7f39c80fc830 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.701+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1354215938 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c81082a0 con 0x7f39c80fc830 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.701+0000 7f39beffd700 1 -- 172.21.15.113:0/1354215938 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39b80137d0 con 0x7f39c80fc830 2024-04-12T05:51:23.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.701+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1354215938 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c8108860 con 0x7f39c80fc830 2024-04-12T05:51:23.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.702+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1354215938 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c805f380 con 0x7f39c80fc830 2024-04-12T05:51:23.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.702+0000 7f39beffd700 1 -- 172.21.15.113:0/1354215938 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f39b8007ca0 con 0x7f39c80fc830 2024-04-12T05:51:23.699 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.702+0000 7f39beffd700 1 --2- 172.21.15.113:0/1354215938 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f39b406caa0 0x7f39b406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:23.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.702+0000 7f39beffd700 1 -- 172.21.15.113:0/1354215938 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f39b808b860 con 0x7f39c80fc830 2024-04-12T05:51:23.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.702+0000 7f39cd318700 1 --2- 172.21.15.113:0/1354215938 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f39b406caa0 0x7f39b406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:23.700 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.703+0000 7f39cd318700 1 --2- 172.21.15.113:0/1354215938 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f39b406caa0 0x7f39b406ef60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f39c8109920 tx=0x7f39c400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:23.702 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.707+0000 7f39beffd700 1 -- 172.21.15.113:0/1354215938 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f39b8059310 con 0x7f39c80fc830 2024-04-12T05:51:23.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.935+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1354215938 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7f39c805eec0 con 0x7f39c80fc830 2024-04-12T05:51:23.931 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.936+0000 7f39beffd700 1 -- 172.21.15.113:0/1354215938 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v35) v1 ==== 107+0+3267 (secure 0 0 0) 0x7f39b8058ea0 con 0x7f39c80fc830 2024-04-12T05:51:23.932 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:23.932 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14438,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/2511637182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2511637182},{"type":"v1","addr":"172.21.15.113:6829","nonce":2511637182}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:18:43.275880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:23.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.938+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1354215938 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f39b406caa0 msgr2=0x7f39b406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:23.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.938+0000 7f39cfd7d700 1 --2- 172.21.15.113:0/1354215938 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f39b406caa0 0x7f39b406ef60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f39c8109920 tx=0x7f39c400b040).stop 2024-04-12T05:51:23.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.938+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1354215938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39c80fc830 msgr2=0x7f39c810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:23.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.938+0000 7f39cfd7d700 1 --2- 172.21.15.113:0/1354215938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39c80fc830 0x7f39c810dfe0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7f39b80041d0 tx=0x7f39b8009f60).stop 2024-04-12T05:51:23.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.938+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1354215938 shutdown_connections 2024-04-12T05:51:23.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.938+0000 7f39cfd7d700 1 --2- 172.21.15.113:0/1354215938 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f39b406caa0 0x7f39b406ef60 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:23.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.938+0000 7f39cfd7d700 1 --2- 172.21.15.113:0/1354215938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39c80fc830 0x7f39c810dfe0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:23.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.938+0000 7f39cfd7d700 1 --2- 172.21.15.113:0/1354215938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39c80fd100 0x7f39c810e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:23.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.938+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1354215938 >> 172.21.15.113:0/1354215938 conn(0x7f39c806c760 msgr2=0x7f39c8060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:23.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.938+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1354215938 shutdown_connections 2024-04-12T05:51:23.934 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:23.938+0000 7f39cfd7d700 1 -- 172.21.15.113:0/1354215938 wait complete. 2024-04-12T05:51:23.935 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 19 2024-04-12T05:51:23.945 DEBUG:tasks.fs:max_mds not reduced in epoch 19 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T05:51:23.946 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-04-12T05:51:24.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.068+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2038731157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb840fd190 msgr2=0x7fcb840f7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:24.065 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.068+0000 7fcb8bbb4700 1 --2- 172.21.15.113:0/2038731157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb840fd190 0x7fcb840f7060 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7fcb800047f0 tx=0x7fcb80009cb0).stop 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.068+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2038731157 shutdown_connections 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.068+0000 7fcb8bbb4700 1 --2- 172.21.15.113:0/2038731157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb840fd190 0x7fcb840f7060 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.068+0000 7fcb8bbb4700 1 --2- 172.21.15.113:0/2038731157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb840fc830 0x7fcb840fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.068+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2038731157 >> 172.21.15.113:0/2038731157 conn(0x7fcb8406de70 msgr2=0x7fcb8406e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.068+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2038731157 shutdown_connections 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.068+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2038731157 wait complete. 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb8bbb4700 1 Processor -- start 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb8bbb4700 1 -- start start 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb8bbb4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb840fc830 0x7fcb841090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb8bbb4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb840fd190 0x7fcb841095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb8bbb4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84109aa0 con 0x7fcb840fc830 2024-04-12T05:51:24.066 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb8bbb4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84109be0 con 0x7fcb840fd190 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb89950700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb840fc830 0x7fcb841090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb89950700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb840fc830 0x7fcb841090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:33986/0 (socket says 172.21.15.113:33986) 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb89950700 1 -- 172.21.15.113:0/2524503107 learned_addr learned my addr 172.21.15.113:0/2524503107 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb8914f700 1 --2- 172.21.15.113:0/2524503107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb840fd190 0x7fcb841095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb89950700 1 -- 172.21.15.113:0/2524503107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb840fd190 msgr2=0x7fcb841095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb89950700 1 --2- 172.21.15.113:0/2524503107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb840fd190 0x7fcb841095b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb89950700 1 -- 172.21.15.113:0/2524503107 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb80005040 con 0x7fcb840fc830 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.069+0000 7fcb89950700 1 --2- 172.21.15.113:0/2524503107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb840fc830 0x7fcb841090c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7fcb74002770 tx=0x7fcb74002b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.070+0000 7fcb7affd700 1 -- 172.21.15.113:0/2524503107 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb74007b40 con 0x7fcb840fc830 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.070+0000 7fcb7affd700 1 -- 172.21.15.113:0/2524503107 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb7400f040 con 0x7fcb840fc830 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.070+0000 7fcb7affd700 1 -- 172.21.15.113:0/2524503107 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb7400d830 con 0x7fcb840fc830 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.070+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2524503107 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb84109e90 con 0x7fcb840fc830 2024-04-12T05:51:24.067 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.070+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2524503107 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb84112400 con 0x7fcb840fc830 2024-04-12T05:51:24.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.071+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2524503107 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8405fb30 con 0x7fcb840fc830 2024-04-12T05:51:24.068 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.071+0000 7fcb7affd700 1 -- 172.21.15.113:0/2524503107 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcb74007ca0 con 0x7fcb840fc830 2024-04-12T05:51:24.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.071+0000 7fcb7affd700 1 --2- 172.21.15.113:0/2524503107 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb7006caa0 0x7fcb7006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:24.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.072+0000 7fcb7affd700 1 -- 172.21.15.113:0/2524503107 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fcb7408b9e0 con 0x7fcb840fc830 2024-04-12T05:51:24.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.072+0000 7fcb8914f700 1 --2- 172.21.15.113:0/2524503107 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb7006caa0 0x7fcb7006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:24.069 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.072+0000 7fcb8914f700 1 --2- 172.21.15.113:0/2524503107 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb7006caa0 0x7fcb7006ef60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fcb80000c00 tx=0x7fcb8000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:24.071 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.076+0000 7fcb7affd700 1 -- 172.21.15.113:0/2524503107 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fcb74055a00 con 0x7fcb840fc830 2024-04-12T05:51:24.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.306+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2524503107 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7fcb841035c0 con 0x7fcb840fc830 2024-04-12T05:51:24.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.307+0000 7fcb7affd700 1 -- 172.21.15.113:0/2524503107 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v35) v1 ==== 107+0+4831 (secure 0 0 0) 0x7fcb74059020 con 0x7fcb840fc830 2024-04-12T05:51:24.302 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:24.303 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14438,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/2511637182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2511637182},{"type":"v1","addr":"172.21.15.113:6829","nonce":2511637182}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34408,"name":"cephfs.smithi113.ovmadh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6827/2944373762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2944373762},{"type":"v1","addr":"172.21.15.113:6827","nonce":2944373762}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":44303,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/2802712879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2802712879},{"type":"v1","addr":"172.21.15.192:6825","nonce":2802712879}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:18:43.275880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:24.304 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.309+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2524503107 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb7006caa0 msgr2=0x7fcb7006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:24.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.309+0000 7fcb8bbb4700 1 --2- 172.21.15.113:0/2524503107 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb7006caa0 0x7fcb7006ef60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fcb80000c00 tx=0x7fcb8000b040).stop 2024-04-12T05:51:24.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.309+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2524503107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb840fc830 msgr2=0x7fcb841090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:24.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.309+0000 7fcb8bbb4700 1 --2- 172.21.15.113:0/2524503107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb840fc830 0x7fcb841090c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7fcb74002770 tx=0x7fcb74002b30).stop 2024-04-12T05:51:24.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.309+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2524503107 shutdown_connections 2024-04-12T05:51:24.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.309+0000 7fcb8bbb4700 1 --2- 172.21.15.113:0/2524503107 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcb7006caa0 0x7fcb7006ef60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.309+0000 7fcb8bbb4700 1 --2- 172.21.15.113:0/2524503107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb840fc830 0x7fcb841090c0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.309+0000 7fcb8bbb4700 1 --2- 172.21.15.113:0/2524503107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb840fd190 0x7fcb841095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.309+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2524503107 >> 172.21.15.113:0/2524503107 conn(0x7fcb8406de70 msgr2=0x7fcb840ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:24.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.309+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2524503107 shutdown_connections 2024-04-12T05:51:24.306 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.309+0000 7fcb8bbb4700 1 -- 172.21.15.113:0/2524503107 wait complete. 2024-04-12T05:51:24.306 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 20 2024-04-12T05:51:24.315 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:24 smithi113 ceph-mon[103371]: pgmap v1179: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:24.315 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:24 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1976276101' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-12T05:51:24.315 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:24 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3398855912' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-12T05:51:24.315 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:24 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1354215938' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-12T05:51:24.316 DEBUG:tasks.fs:max_mds not reduced in epoch 20 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T05:51:24.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-12T05:51:24.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.439+0000 7fdd44e80700 1 -- 172.21.15.113:0/231547996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd400fd100 msgr2=0x7fdd400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:24.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.439+0000 7fdd44e80700 1 --2- 172.21.15.113:0/231547996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd400fd100 0x7fdd400f6930 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7fdd300047f0 tx=0x7fdd30009cb0).stop 2024-04-12T05:51:24.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.439+0000 7fdd44e80700 1 -- 172.21.15.113:0/231547996 shutdown_connections 2024-04-12T05:51:24.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.439+0000 7fdd44e80700 1 --2- 172.21.15.113:0/231547996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd400fd100 0x7fdd400f6930 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.439+0000 7fdd44e80700 1 --2- 172.21.15.113:0/231547996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400fc830 0x7fdd400fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.439+0000 7fdd44e80700 1 -- 172.21.15.113:0/231547996 >> 172.21.15.113:0/231547996 conn(0x7fdd4006c760 msgr2=0x7fdd4006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:24.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.439+0000 7fdd44e80700 1 -- 172.21.15.113:0/231547996 shutdown_connections 2024-04-12T05:51:24.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.439+0000 7fdd44e80700 1 -- 172.21.15.113:0/231547996 wait complete. 2024-04-12T05:51:24.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd44e80700 1 Processor -- start 2024-04-12T05:51:24.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd44e80700 1 -- start start 2024-04-12T05:51:24.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd44e80700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd400fc830 0x7fdd4010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:24.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd44e80700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400fd100 0x7fdd4010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:24.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd44e80700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4010ead0 con 0x7fdd400fc830 2024-04-12T05:51:24.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd44e80700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4010ec10 con 0x7fdd400fd100 2024-04-12T05:51:24.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd3e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd400fc830 0x7fdd4010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:24.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd3e59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd400fc830 0x7fdd4010dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34012/0 (socket says 172.21.15.113:34012) 2024-04-12T05:51:24.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd3e59c700 1 -- 172.21.15.113:0/785096278 learned_addr learned my addr 172.21.15.113:0/785096278 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:24.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd3dd9b700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400fd100 0x7fdd4010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:24.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd3e59c700 1 -- 172.21.15.113:0/785096278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400fd100 msgr2=0x7fdd4010e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:24.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd3e59c700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400fd100 0x7fdd4010e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd3e59c700 1 -- 172.21.15.113:0/785096278 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd30005040 con 0x7fdd400fc830 2024-04-12T05:51:24.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd3dd9b700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400fd100 0x7fdd4010e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:51:24.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.440+0000 7fdd3e59c700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd400fc830 0x7fdd4010dfe0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7fdd3400c790 tx=0x7fdd3400cb50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:24.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.441+0000 7fdd2f7fe700 1 -- 172.21.15.113:0/785096278 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd340106f0 con 0x7fdd400fc830 2024-04-12T05:51:24.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.441+0000 7fdd2f7fe700 1 -- 172.21.15.113:0/785096278 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd34010d00 con 0x7fdd400fc830 2024-04-12T05:51:24.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.441+0000 7fdd44e80700 1 -- 172.21.15.113:0/785096278 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd401082a0 con 0x7fdd400fc830 2024-04-12T05:51:24.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.441+0000 7fdd2f7fe700 1 -- 172.21.15.113:0/785096278 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd34012e50 con 0x7fdd400fc830 2024-04-12T05:51:24.439 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.441+0000 7fdd44e80700 1 -- 172.21.15.113:0/785096278 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd40108860 con 0x7fdd400fc830 2024-04-12T05:51:24.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.442+0000 7fdd44e80700 1 -- 172.21.15.113:0/785096278 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4005f380 con 0x7fdd400fc830 2024-04-12T05:51:24.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.442+0000 7fdd2f7fe700 1 -- 172.21.15.113:0/785096278 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdd34010850 con 0x7fdd400fc830 2024-04-12T05:51:24.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.443+0000 7fdd2f7fe700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd2806caa0 0x7fdd2806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:24.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.443+0000 7fdd2f7fe700 1 -- 172.21.15.113:0/785096278 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fdd3408b850 con 0x7fdd400fc830 2024-04-12T05:51:24.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.443+0000 7fdd3dd9b700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd2806caa0 0x7fdd2806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:24.440 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.443+0000 7fdd3dd9b700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd2806caa0 0x7fdd2806ef60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7fdd300043d0 tx=0x7fdd3000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:24.441 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:24 smithi192 ceph-mon[85304]: pgmap v1179: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:24.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:24 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1976276101' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-12T05:51:24.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:24 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3398855912' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-12T05:51:24.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:24 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1354215938' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-12T05:51:24.442 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.447+0000 7fdd2f7fe700 1 -- 172.21.15.113:0/785096278 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fdd34059240 con 0x7fdd400fc830 2024-04-12T05:51:24.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.676+0000 7fdd44e80700 1 -- 172.21.15.113:0/785096278 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7fdd40108f00 con 0x7fdd400fc830 2024-04-12T05:51:24.672 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.677+0000 7fdd2f7fe700 1 -- 172.21.15.113:0/785096278 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v35) v1 ==== 107+0+4049 (secure 0 0 0) 0x7fdd34058dd0 con 0x7fdd400fc830 2024-04-12T05:51:24.673 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:24.673 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14438,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/2511637182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2511637182},{"type":"v1","addr":"172.21.15.113:6829","nonce":2511637182}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":44303,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/2802712879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2802712879},{"type":"v1","addr":"172.21.15.192:6825","nonce":2802712879}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:18:43.275880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:24.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.679+0000 7fdd44e80700 1 -- 172.21.15.113:0/785096278 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd2806caa0 msgr2=0x7fdd2806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:24.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.679+0000 7fdd44e80700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd2806caa0 0x7fdd2806ef60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7fdd300043d0 tx=0x7fdd3000b040).stop 2024-04-12T05:51:24.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.679+0000 7fdd44e80700 1 -- 172.21.15.113:0/785096278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd400fc830 msgr2=0x7fdd4010dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:24.675 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.679+0000 7fdd44e80700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd400fc830 0x7fdd4010dfe0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7fdd3400c790 tx=0x7fdd3400cb50).stop 2024-04-12T05:51:24.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.679+0000 7fdd44e80700 1 -- 172.21.15.113:0/785096278 shutdown_connections 2024-04-12T05:51:24.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.679+0000 7fdd44e80700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdd2806caa0 0x7fdd2806ef60 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.679+0000 7fdd44e80700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd400fc830 0x7fdd4010dfe0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.679+0000 7fdd44e80700 1 --2- 172.21.15.113:0/785096278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400fd100 0x7fdd4010e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.679+0000 7fdd44e80700 1 -- 172.21.15.113:0/785096278 >> 172.21.15.113:0/785096278 conn(0x7fdd4006c760 msgr2=0x7fdd40060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:24.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.679+0000 7fdd44e80700 1 -- 172.21.15.113:0/785096278 shutdown_connections 2024-04-12T05:51:24.676 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.679+0000 7fdd44e80700 1 -- 172.21.15.113:0/785096278 wait complete. 2024-04-12T05:51:24.676 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 21 2024-04-12T05:51:24.687 DEBUG:tasks.fs:max_mds not reduced in epoch 21 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T05:51:24.687 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-12T05:51:24.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.809+0000 7eff8f205700 1 -- 172.21.15.113:0/3670061630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff880f9200 msgr2=0x7eff88101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:24.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.809+0000 7eff8f205700 1 --2- 172.21.15.113:0/3670061630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff880f9200 0x7eff88101910 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7eff740047f0 tx=0x7eff74009cb0).stop 2024-04-12T05:51:24.806 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.809+0000 7eff8f205700 1 -- 172.21.15.113:0/3670061630 shutdown_connections 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.809+0000 7eff8f205700 1 --2- 172.21.15.113:0/3670061630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff880f9200 0x7eff88101910 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.809+0000 7eff8f205700 1 --2- 172.21.15.113:0/3670061630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff880f88a0 0x7eff880f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.809+0000 7eff8f205700 1 -- 172.21.15.113:0/3670061630 >> 172.21.15.113:0/3670061630 conn(0x7eff880f23f0 msgr2=0x7eff880f4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.809+0000 7eff8f205700 1 -- 172.21.15.113:0/3670061630 shutdown_connections 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.809+0000 7eff8f205700 1 -- 172.21.15.113:0/3670061630 wait complete. 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.809+0000 7eff8f205700 1 Processor -- start 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8f205700 1 -- start start 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8f205700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff880f88a0 0x7eff881102b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8f205700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff880f9200 0x7eff881107a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8f205700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff88110da0 con 0x7eff880f88a0 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8f205700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff88110ee0 con 0x7eff880f9200 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8cfa1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff880f88a0 0x7eff881102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8cfa1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff880f88a0 0x7eff881102b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34028/0 (socket says 172.21.15.113:34028) 2024-04-12T05:51:24.807 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8cfa1700 1 -- 172.21.15.113:0/2314340967 learned_addr learned my addr 172.21.15.113:0/2314340967 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:24.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8cfa1700 1 -- 172.21.15.113:0/2314340967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff880f9200 msgr2=0x7eff881107a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:24.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8cfa1700 1 --2- 172.21.15.113:0/2314340967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff880f9200 0x7eff881107a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:24.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8cfa1700 1 -- 172.21.15.113:0/2314340967 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff74005040 con 0x7eff880f88a0 2024-04-12T05:51:24.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8cfa1700 1 --2- 172.21.15.113:0/2314340967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff880f88a0 0x7eff881102b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7eff7c00c790 tx=0x7eff7c00cb50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:24.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff85ffb700 1 -- 172.21.15.113:0/2314340967 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff7c0106f0 con 0x7eff880f88a0 2024-04-12T05:51:24.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff85ffb700 1 -- 172.21.15.113:0/2314340967 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff7c010d00 con 0x7eff880f88a0 2024-04-12T05:51:24.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff85ffb700 1 -- 172.21.15.113:0/2314340967 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff7c012e50 con 0x7eff880f88a0 2024-04-12T05:51:24.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.810+0000 7eff8f205700 1 -- 172.21.15.113:0/2314340967 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff8810a570 con 0x7eff880f88a0 2024-04-12T05:51:24.808 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.811+0000 7eff8f205700 1 -- 172.21.15.113:0/2314340967 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8810aab0 con 0x7eff880f88a0 2024-04-12T05:51:24.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.811+0000 7eff8f205700 1 -- 172.21.15.113:0/2314340967 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff8805f2e0 con 0x7eff880f88a0 2024-04-12T05:51:24.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.812+0000 7eff85ffb700 1 -- 172.21.15.113:0/2314340967 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7eff7c010850 con 0x7eff880f88a0 2024-04-12T05:51:24.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.812+0000 7eff85ffb700 1 --2- 172.21.15.113:0/2314340967 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff7806caa0 0x7eff7806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:24.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.812+0000 7eff85ffb700 1 -- 172.21.15.113:0/2314340967 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7eff7c08b790 con 0x7eff880f88a0 2024-04-12T05:51:24.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.812+0000 7eff87fff700 1 --2- 172.21.15.113:0/2314340967 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff7806caa0 0x7eff7806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:24.809 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.813+0000 7eff87fff700 1 --2- 172.21.15.113:0/2314340967 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff7806caa0 0x7eff7806ef60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7eff8810bbf0 tx=0x7eff74009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:24.812 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:24.817+0000 7eff85ffb700 1 -- 172.21.15.113:0/2314340967 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7eff7c059240 con 0x7eff880f88a0 2024-04-12T05:51:25.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.052+0000 7eff8f205700 1 -- 172.21.15.113:0/2314340967 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7eff8810b9d0 con 0x7eff880f88a0 2024-04-12T05:51:25.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.053+0000 7eff85ffb700 1 -- 172.21.15.113:0/2314340967 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v35) v1 ==== 107+0+4831 (secure 0 0 0) 0x7eff7c058dd0 con 0x7eff880f88a0 2024-04-12T05:51:25.049 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:25.049 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14438,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/2511637182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2511637182},{"type":"v1","addr":"172.21.15.113:6829","nonce":2511637182}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":44303,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/2802712879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2802712879},{"type":"v1","addr":"172.21.15.192:6825","nonce":2802712879}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:18:43.275880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:25.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.055+0000 7eff8f205700 1 -- 172.21.15.113:0/2314340967 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff7806caa0 msgr2=0x7eff7806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.055+0000 7eff8f205700 1 --2- 172.21.15.113:0/2314340967 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff7806caa0 0x7eff7806ef60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7eff8810bbf0 tx=0x7eff74009cb0).stop 2024-04-12T05:51:25.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.055+0000 7eff8f205700 1 -- 172.21.15.113:0/2314340967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff880f88a0 msgr2=0x7eff881102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.055+0000 7eff8f205700 1 --2- 172.21.15.113:0/2314340967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff880f88a0 0x7eff881102b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7eff7c00c790 tx=0x7eff7c00cb50).stop 2024-04-12T05:51:25.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.055+0000 7eff8f205700 1 -- 172.21.15.113:0/2314340967 shutdown_connections 2024-04-12T05:51:25.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.055+0000 7eff8f205700 1 --2- 172.21.15.113:0/2314340967 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7eff7806caa0 0x7eff7806ef60 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.055+0000 7eff8f205700 1 --2- 172.21.15.113:0/2314340967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff880f88a0 0x7eff881102b0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.051 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.055+0000 7eff8f205700 1 --2- 172.21.15.113:0/2314340967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff880f9200 0x7eff881107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:25.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.055+0000 7eff8f205700 1 -- 172.21.15.113:0/2314340967 >> 172.21.15.113:0/2314340967 conn(0x7eff880f23f0 msgr2=0x7eff880f6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:25.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.055+0000 7eff8f205700 1 -- 172.21.15.113:0/2314340967 shutdown_connections 2024-04-12T05:51:25.052 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.055+0000 7eff8f205700 1 -- 172.21.15.113:0/2314340967 wait complete. 2024-04-12T05:51:25.052 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 22 2024-04-12T05:51:25.064 DEBUG:tasks.fs:max_mds not reduced in epoch 22 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T05:51:25.064 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-12T05:51:25.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.185+0000 7f9e74441700 1 -- 172.21.15.113:0/1570130910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6c0fc830 msgr2=0x7f9e6c0fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.185+0000 7f9e74441700 1 --2- 172.21.15.113:0/1570130910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6c0fc830 0x7f9e6c0fcc10 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f9e680047f0 tx=0x7f9e68009cb0).stop 2024-04-12T05:51:25.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 -- 172.21.15.113:0/1570130910 shutdown_connections 2024-04-12T05:51:25.183 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 --2- 172.21.15.113:0/1570130910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e6c0fd100 0x7f9e6c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 --2- 172.21.15.113:0/1570130910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6c0fc830 0x7f9e6c0fcc10 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 -- 172.21.15.113:0/1570130910 >> 172.21.15.113:0/1570130910 conn(0x7f9e6c06c760 msgr2=0x7f9e6c06cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 -- 172.21.15.113:0/1570130910 shutdown_connections 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 -- 172.21.15.113:0/1570130910 wait complete. 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 Processor -- start 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 -- start start 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e6c0fc830 0x7f9e6c0fd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6c0fd100 0x7f9e6c0fdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c0fe160 con 0x7f9e6c0fd100 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e74441700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c0fe2a0 con 0x7f9e6c0fc830 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e719dc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6c0fd100 0x7f9e6c0fdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e719dc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6c0fd100 0x7f9e6c0fdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34044/0 (socket says 172.21.15.113:34044) 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.186+0000 7f9e719dc700 1 -- 172.21.15.113:0/935692336 learned_addr learned my addr 172.21.15.113:0/935692336 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.187+0000 7f9e721dd700 1 --2- 172.21.15.113:0/935692336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e6c0fc830 0x7f9e6c0fd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:25.184 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.187+0000 7f9e719dc700 1 -- 172.21.15.113:0/935692336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e6c0fc830 msgr2=0x7f9e6c0fd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.187+0000 7f9e719dc700 1 --2- 172.21.15.113:0/935692336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e6c0fc830 0x7f9e6c0fd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.187+0000 7f9e719dc700 1 -- 172.21.15.113:0/935692336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e68005040 con 0x7f9e6c0fd100 2024-04-12T05:51:25.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.187+0000 7f9e719dc700 1 --2- 172.21.15.113:0/935692336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6c0fd100 0x7f9e6c0fdc70 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f9e5c0041d0 tx=0x7f9e5c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:25.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.187+0000 7f9e637fe700 1 -- 172.21.15.113:0/935692336 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e5c007b40 con 0x7f9e6c0fd100 2024-04-12T05:51:25.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.187+0000 7f9e637fe700 1 -- 172.21.15.113:0/935692336 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e5c00f040 con 0x7f9e6c0fd100 2024-04-12T05:51:25.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.187+0000 7f9e74441700 1 -- 172.21.15.113:0/935692336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6c0fe550 con 0x7f9e6c0fd100 2024-04-12T05:51:25.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.187+0000 7f9e637fe700 1 -- 172.21.15.113:0/935692336 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e5c0137d0 con 0x7f9e6c0fd100 2024-04-12T05:51:25.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.187+0000 7f9e74441700 1 -- 172.21.15.113:0/935692336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c114670 con 0x7f9e6c0fd100 2024-04-12T05:51:25.185 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.188+0000 7f9e74441700 1 -- 172.21.15.113:0/935692336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6c05f380 con 0x7f9e6c0fd100 2024-04-12T05:51:25.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.188+0000 7f9e637fe700 1 -- 172.21.15.113:0/935692336 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9e5c007ca0 con 0x7f9e6c0fd100 2024-04-12T05:51:25.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.189+0000 7f9e637fe700 1 --2- 172.21.15.113:0/935692336 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e5806caa0 0x7f9e5806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:25.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.189+0000 7f9e637fe700 1 -- 172.21.15.113:0/935692336 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9e5c08b860 con 0x7f9e6c0fd100 2024-04-12T05:51:25.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.189+0000 7f9e721dd700 1 --2- 172.21.15.113:0/935692336 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e5806caa0 0x7f9e5806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:25.186 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.189+0000 7f9e721dd700 1 --2- 172.21.15.113:0/935692336 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e5806caa0 0x7f9e5806ef60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f9e68004f10 tx=0x7f9e6800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:25.189 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.193+0000 7f9e637fe700 1 -- 172.21.15.113:0/935692336 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9e5c055880 con 0x7f9e6c0fd100 2024-04-12T05:51:25.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:25 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2524503107' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-12T05:51:25.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:25 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/785096278' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-12T05:51:25.419 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.423+0000 7f9e74441700 1 -- 172.21.15.113:0/935692336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f9e6c0fed00 con 0x7f9e6c0fd100 2024-04-12T05:51:25.420 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.424+0000 7f9e637fe700 1 -- 172.21.15.113:0/935692336 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v35) v1 ==== 107+0+4049 (secure 0 0 0) 0x7f9e5c058ea0 con 0x7f9e6c0fd100 2024-04-12T05:51:25.420 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:25.420 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":44303,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/2802712879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2802712879},{"type":"v1","addr":"172.21.15.192:6825","nonce":2802712879}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:18:43.275880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:25.423 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.426+0000 7f9e74441700 1 -- 172.21.15.113:0/935692336 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e5806caa0 msgr2=0x7f9e5806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.426+0000 7f9e74441700 1 --2- 172.21.15.113:0/935692336 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e5806caa0 0x7f9e5806ef60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f9e68004f10 tx=0x7f9e6800b040).stop 2024-04-12T05:51:25.424 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.426+0000 7f9e74441700 1 -- 172.21.15.113:0/935692336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6c0fd100 msgr2=0x7f9e6c0fdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.426+0000 7f9e74441700 1 --2- 172.21.15.113:0/935692336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6c0fd100 0x7f9e6c0fdc70 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f9e5c0041d0 tx=0x7f9e5c009f60).stop 2024-04-12T05:51:25.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.426+0000 7f9e74441700 1 -- 172.21.15.113:0/935692336 shutdown_connections 2024-04-12T05:51:25.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.426+0000 7f9e74441700 1 --2- 172.21.15.113:0/935692336 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9e5806caa0 0x7f9e5806ef60 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.426+0000 7f9e74441700 1 --2- 172.21.15.113:0/935692336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e6c0fc830 0x7f9e6c0fd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.426+0000 7f9e74441700 1 --2- 172.21.15.113:0/935692336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e6c0fd100 0x7f9e6c0fdc70 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.425 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.426+0000 7f9e74441700 1 -- 172.21.15.113:0/935692336 >> 172.21.15.113:0/935692336 conn(0x7f9e6c06c760 msgr2=0x7f9e6c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:25.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.426+0000 7f9e74441700 1 -- 172.21.15.113:0/935692336 shutdown_connections 2024-04-12T05:51:25.426 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.426+0000 7f9e74441700 1 -- 172.21.15.113:0/935692336 wait complete. 2024-04-12T05:51:25.426 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 23 2024-04-12T05:51:25.433 DEBUG:tasks.fs:max_mds not reduced in epoch 23 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T05:51:25.434 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-12T05:51:25.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:25 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2524503107' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-12T05:51:25.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:25 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/785096278' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-12T05:51:25.552 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.554+0000 7fc4186d8700 1 -- 172.21.15.113:0/1215537138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4100f9220 msgr2=0x7fc410101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.554+0000 7fc4186d8700 1 --2- 172.21.15.113:0/1215537138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4100f9220 0x7fc410101930 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7fc4000047f0 tx=0x7fc400009cb0).stop 2024-04-12T05:51:25.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.554+0000 7fc4186d8700 1 -- 172.21.15.113:0/1215537138 shutdown_connections 2024-04-12T05:51:25.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.554+0000 7fc4186d8700 1 --2- 172.21.15.113:0/1215537138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4100f9220 0x7fc410101930 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.554+0000 7fc4186d8700 1 --2- 172.21.15.113:0/1215537138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4100f88c0 0x7fc4100f8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.554+0000 7fc4186d8700 1 -- 172.21.15.113:0/1215537138 >> 172.21.15.113:0/1215537138 conn(0x7fc4100f23f0 msgr2=0x7fc4100f4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:25.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.554+0000 7fc4186d8700 1 -- 172.21.15.113:0/1215537138 shutdown_connections 2024-04-12T05:51:25.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.554+0000 7fc4186d8700 1 -- 172.21.15.113:0/1215537138 wait complete. 2024-04-12T05:51:25.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.555+0000 7fc4186d8700 1 Processor -- start 2024-04-12T05:51:25.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.555+0000 7fc4186d8700 1 -- start start 2024-04-12T05:51:25.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.555+0000 7fc4186d8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4100f88c0 0x7fc41010e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:25.553 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.555+0000 7fc4186d8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4100f9220 0x7fc41010e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.555+0000 7fc4186d8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41010eb20 con 0x7fc4100f9220 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.555+0000 7fc4186d8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41010ec60 con 0x7fc4100f88c0 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.555+0000 7fc415c73700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4100f9220 0x7fc41010e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.555+0000 7fc415c73700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4100f9220 0x7fc41010e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34074/0 (socket says 172.21.15.113:34074) 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.555+0000 7fc415c73700 1 -- 172.21.15.113:0/3289306289 learned_addr learned my addr 172.21.15.113:0/3289306289 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.555+0000 7fc416474700 1 --2- 172.21.15.113:0/3289306289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4100f88c0 0x7fc41010e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.556+0000 7fc415c73700 1 -- 172.21.15.113:0/3289306289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4100f88c0 msgr2=0x7fc41010e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.556+0000 7fc415c73700 1 --2- 172.21.15.113:0/3289306289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4100f88c0 0x7fc41010e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.556+0000 7fc415c73700 1 -- 172.21.15.113:0/3289306289 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc400005040 con 0x7fc4100f9220 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.556+0000 7fc415c73700 1 --2- 172.21.15.113:0/3289306289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4100f9220 0x7fc41010e520 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7fc40000b120 tx=0x7fc400009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.556+0000 7fc4077fe700 1 -- 172.21.15.113:0/3289306289 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc40000e030 con 0x7fc4100f9220 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.556+0000 7fc4077fe700 1 -- 172.21.15.113:0/3289306289 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc400007e90 con 0x7fc4100f9220 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.556+0000 7fc4186d8700 1 -- 172.21.15.113:0/3289306289 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc410108290 con 0x7fc4100f9220 2024-04-12T05:51:25.554 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.556+0000 7fc4077fe700 1 -- 172.21.15.113:0/3289306289 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc400018070 con 0x7fc4100f9220 2024-04-12T05:51:25.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.556+0000 7fc4186d8700 1 -- 172.21.15.113:0/3289306289 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc410108770 con 0x7fc4100f9220 2024-04-12T05:51:25.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.557+0000 7fc4186d8700 1 -- 172.21.15.113:0/3289306289 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41005f2e0 con 0x7fc4100f9220 2024-04-12T05:51:25.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.557+0000 7fc4077fe700 1 -- 172.21.15.113:0/3289306289 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc40001d420 con 0x7fc4100f9220 2024-04-12T05:51:25.555 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.557+0000 7fc4077fe700 1 --2- 172.21.15.113:0/3289306289 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc3fc06caa0 0x7fc3fc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:25.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.558+0000 7fc4077fe700 1 -- 172.21.15.113:0/3289306289 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc40008d250 con 0x7fc4100f9220 2024-04-12T05:51:25.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.558+0000 7fc416474700 1 --2- 172.21.15.113:0/3289306289 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc3fc06caa0 0x7fc3fc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:25.556 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.558+0000 7fc416474700 1 --2- 172.21.15.113:0/3289306289 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc3fc06caa0 0x7fc3fc06ef60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7fc40c004ac0 tx=0x7fc40c009040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:25.557 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.562+0000 7fc4077fe700 1 -- 172.21.15.113:0/3289306289 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc40005ad00 con 0x7fc4100f9220 2024-04-12T05:51:25.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.793+0000 7fc4186d8700 1 -- 172.21.15.113:0/3289306289 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7fc4101096e0 con 0x7fc4100f9220 2024-04-12T05:51:25.789 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.794+0000 7fc4077fe700 1 -- 172.21.15.113:0/3289306289 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v35) v1 ==== 107+0+4828 (secure 0 0 0) 0x7fc40005a890 con 0x7fc4100f9220 2024-04-12T05:51:25.789 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:25.789 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44303,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/2802712879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2802712879},{"type":"v1","addr":"172.21.15.192:6825","nonce":2802712879}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:18:43.275880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:25.791 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.796+0000 7fc4186d8700 1 -- 172.21.15.113:0/3289306289 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc3fc06caa0 msgr2=0x7fc3fc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.796+0000 7fc4186d8700 1 --2- 172.21.15.113:0/3289306289 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc3fc06caa0 0x7fc3fc06ef60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7fc40c004ac0 tx=0x7fc40c009040).stop 2024-04-12T05:51:25.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.796+0000 7fc4186d8700 1 -- 172.21.15.113:0/3289306289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4100f9220 msgr2=0x7fc41010e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.796+0000 7fc4186d8700 1 --2- 172.21.15.113:0/3289306289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4100f9220 0x7fc41010e520 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7fc40000b120 tx=0x7fc400009cb0).stop 2024-04-12T05:51:25.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.796+0000 7fc4186d8700 1 -- 172.21.15.113:0/3289306289 shutdown_connections 2024-04-12T05:51:25.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.796+0000 7fc4186d8700 1 --2- 172.21.15.113:0/3289306289 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc3fc06caa0 0x7fc3fc06ef60 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.796+0000 7fc4186d8700 1 --2- 172.21.15.113:0/3289306289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4100f88c0 0x7fc41010e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.796+0000 7fc4186d8700 1 --2- 172.21.15.113:0/3289306289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4100f9220 0x7fc41010e520 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.796+0000 7fc4186d8700 1 -- 172.21.15.113:0/3289306289 >> 172.21.15.113:0/3289306289 conn(0x7fc4100f23f0 msgr2=0x7fc4100fc170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:25.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.796+0000 7fc4186d8700 1 -- 172.21.15.113:0/3289306289 shutdown_connections 2024-04-12T05:51:25.792 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.796+0000 7fc4186d8700 1 -- 172.21.15.113:0/3289306289 wait complete. 2024-04-12T05:51:25.792 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 24 2024-04-12T05:51:25.803 DEBUG:tasks.fs:max_mds not reduced in epoch 24 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T05:51:25.803 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-12T05:51:25.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.925+0000 7f5a20f81700 1 -- 172.21.15.113:0/2670337368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a1c0fca10 msgr2=0x7f5a1c0fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.925+0000 7f5a20f81700 1 --2- 172.21.15.113:0/2670337368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a1c0fca10 0x7f5a1c0fcdf0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f5a100047f0 tx=0x7f5a10009cb0).stop 2024-04-12T05:51:25.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.925+0000 7f5a20f81700 1 -- 172.21.15.113:0/2670337368 shutdown_connections 2024-04-12T05:51:25.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.925+0000 7f5a20f81700 1 --2- 172.21.15.113:0/2670337368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a1c0fd370 0x7f5a1c0f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.921 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.925+0000 7f5a20f81700 1 --2- 172.21.15.113:0/2670337368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a1c0fca10 0x7f5a1c0fcdf0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.925+0000 7f5a20f81700 1 -- 172.21.15.113:0/2670337368 >> 172.21.15.113:0/2670337368 conn(0x7f5a1c0f2590 msgr2=0x7f5a1c0f49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:25.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.925+0000 7f5a20f81700 1 -- 172.21.15.113:0/2670337368 shutdown_connections 2024-04-12T05:51:25.922 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.925+0000 7f5a20f81700 1 -- 172.21.15.113:0/2670337368 wait complete. 2024-04-12T05:51:25.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.926+0000 7f5a20f81700 1 Processor -- start 2024-04-12T05:51:25.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.926+0000 7f5a20f81700 1 -- start start 2024-04-12T05:51:25.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.926+0000 7f5a20f81700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a1c0fca10 0x7f5a1c10d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:25.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a20f81700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a1c0fd370 0x7f5a1c10d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:25.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a20f81700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c10dc40 con 0x7f5a1c0fd370 2024-04-12T05:51:25.924 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a20f81700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c10dd80 con 0x7f5a1c0fca10 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a19d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a1c0fd370 0x7f5a1c10d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a19d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a1c0fd370 0x7f5a1c10d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34084/0 (socket says 172.21.15.113:34084) 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a19d9b700 1 -- 172.21.15.113:0/1586136804 learned_addr learned my addr 172.21.15.113:0/1586136804 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a1a59c700 1 --2- 172.21.15.113:0/1586136804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a1c0fca10 0x7f5a1c10d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a19d9b700 1 -- 172.21.15.113:0/1586136804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a1c0fca10 msgr2=0x7f5a1c10d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a19d9b700 1 --2- 172.21.15.113:0/1586136804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a1c0fca10 0x7f5a1c10d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a19d9b700 1 -- 172.21.15.113:0/1586136804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a10005040 con 0x7f5a1c0fd370 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a19d9b700 1 --2- 172.21.15.113:0/1586136804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a1c0fd370 0x7f5a1c10d640 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f5a0c00c6e0 tx=0x7f5a0c007e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a0b7fe700 1 -- 172.21.15.113:0/1586136804 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a0c00e680 con 0x7f5a1c0fd370 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a0b7fe700 1 -- 172.21.15.113:0/1586136804 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a0c00ec90 con 0x7f5a1c0fd370 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a20f81700 1 -- 172.21.15.113:0/1586136804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1c107410 con 0x7f5a1c0fd370 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a0b7fe700 1 -- 172.21.15.113:0/1586136804 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a0c013d40 con 0x7f5a1c0fd370 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.927+0000 7f5a20f81700 1 -- 172.21.15.113:0/1586136804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1c1079d0 con 0x7f5a1c0fd370 2024-04-12T05:51:25.925 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.928+0000 7f5a20f81700 1 -- 172.21.15.113:0/1586136804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1c0603b0 con 0x7f5a1c0fd370 2024-04-12T05:51:25.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.929+0000 7f5a0b7fe700 1 -- 172.21.15.113:0/1586136804 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5a0c00e7e0 con 0x7f5a1c0fd370 2024-04-12T05:51:25.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.929+0000 7f5a0b7fe700 1 --2- 172.21.15.113:0/1586136804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a0406caa0 0x7f5a0406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:25.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.929+0000 7f5a0b7fe700 1 -- 172.21.15.113:0/1586136804 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f5a0c08c100 con 0x7f5a1c0fd370 2024-04-12T05:51:25.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.929+0000 7f5a1a59c700 1 --2- 172.21.15.113:0/1586136804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a0406caa0 0x7f5a0406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:25.926 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.929+0000 7f5a1a59c700 1 --2- 172.21.15.113:0/1586136804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a0406caa0 0x7f5a0406ef60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f5a10000c00 tx=0x7f5a1000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:25.929 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:25.933+0000 7f5a0b7fe700 1 -- 172.21.15.113:0/1586136804 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f5a0c059bb0 con 0x7f5a1c0fd370 2024-04-12T05:51:26.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.163+0000 7f5a20f81700 1 -- 172.21.15.113:0/1586136804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7f5a1c05fb30 con 0x7f5a1c0fd370 2024-04-12T05:51:26.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.164+0000 7f5a0b7fe700 1 -- 172.21.15.113:0/1586136804 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v35) v1 ==== 107+0+4046 (secure 0 0 0) 0x7f5a0c059740 con 0x7f5a1c0fd370 2024-04-12T05:51:26.160 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:26.160 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:18:43.275880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:26.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.166+0000 7f5a20f81700 1 -- 172.21.15.113:0/1586136804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a0406caa0 msgr2=0x7f5a0406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:26.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.166+0000 7f5a20f81700 1 --2- 172.21.15.113:0/1586136804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a0406caa0 0x7f5a0406ef60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f5a10000c00 tx=0x7f5a1000b040).stop 2024-04-12T05:51:26.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.166+0000 7f5a20f81700 1 -- 172.21.15.113:0/1586136804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a1c0fd370 msgr2=0x7f5a1c10d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:26.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.166+0000 7f5a20f81700 1 --2- 172.21.15.113:0/1586136804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a1c0fd370 0x7f5a1c10d640 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f5a0c00c6e0 tx=0x7f5a0c007e90).stop 2024-04-12T05:51:26.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.166+0000 7f5a20f81700 1 -- 172.21.15.113:0/1586136804 shutdown_connections 2024-04-12T05:51:26.162 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.166+0000 7f5a20f81700 1 --2- 172.21.15.113:0/1586136804 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f5a0406caa0 0x7f5a0406ef60 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.166+0000 7f5a20f81700 1 --2- 172.21.15.113:0/1586136804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a1c0fca10 0x7f5a1c10d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.166+0000 7f5a20f81700 1 --2- 172.21.15.113:0/1586136804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a1c0fd370 0x7f5a1c10d640 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.166+0000 7f5a20f81700 1 -- 172.21.15.113:0/1586136804 >> 172.21.15.113:0/1586136804 conn(0x7f5a1c0f2590 msgr2=0x7f5a1c0f3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:26.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.166+0000 7f5a20f81700 1 -- 172.21.15.113:0/1586136804 shutdown_connections 2024-04-12T05:51:26.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.166+0000 7f5a20f81700 1 -- 172.21.15.113:0/1586136804 wait complete. 2024-04-12T05:51:26.163 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 25 2024-04-12T05:51:26.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:26 smithi113 ceph-mon[103371]: pgmap v1180: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:26.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:26 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2314340967' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-12T05:51:26.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:26 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/935692336' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-12T05:51:26.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:26 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3289306289' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-12T05:51:26.174 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T05:51:26.174 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-12T05:51:26.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.296+0000 7f0529116700 1 -- 172.21.15.113:0/210028306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240fc890 msgr2=0x7f05240fcc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:26.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.296+0000 7f0529116700 1 --2- 172.21.15.113:0/210028306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240fc890 0x7f05240fcc70 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7f05140047f0 tx=0x7f0514009cb0).stop 2024-04-12T05:51:26.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.296+0000 7f0529116700 1 -- 172.21.15.113:0/210028306 shutdown_connections 2024-04-12T05:51:26.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.296+0000 7f0529116700 1 --2- 172.21.15.113:0/210028306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05240fd1f0 0x7f05240f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.296+0000 7f0529116700 1 --2- 172.21.15.113:0/210028306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240fc890 0x7f05240fcc70 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.296+0000 7f0529116700 1 -- 172.21.15.113:0/210028306 >> 172.21.15.113:0/210028306 conn(0x7f05240f2490 msgr2=0x7f05240f48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:26.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.296+0000 7f0529116700 1 -- 172.21.15.113:0/210028306 shutdown_connections 2024-04-12T05:51:26.292 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.296+0000 7f0529116700 1 -- 172.21.15.113:0/210028306 wait complete. 2024-04-12T05:51:26.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.297+0000 7f0529116700 1 Processor -- start 2024-04-12T05:51:26.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.297+0000 7f0529116700 1 -- start start 2024-04-12T05:51:26.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.297+0000 7f0529116700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240fc890 0x7f052410cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:26.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.297+0000 7f0529116700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05240fd1f0 0x7f052410d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:26.294 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.297+0000 7f0529116700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052410da60 con 0x7f05240fc890 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.297+0000 7f0529116700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052410dba0 con 0x7f05240fd1f0 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.297+0000 7f0522d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240fc890 0x7f052410cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.297+0000 7f0522d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240fc890 0x7f052410cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34114/0 (socket says 172.21.15.113:34114) 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.297+0000 7f0522d9d700 1 -- 172.21.15.113:0/2725487052 learned_addr learned my addr 172.21.15.113:0/2725487052 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.298+0000 7f052259c700 1 --2- 172.21.15.113:0/2725487052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05240fd1f0 0x7f052410d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.298+0000 7f0522d9d700 1 -- 172.21.15.113:0/2725487052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05240fd1f0 msgr2=0x7f052410d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.298+0000 7f0522d9d700 1 --2- 172.21.15.113:0/2725487052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05240fd1f0 0x7f052410d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.298+0000 7f0522d9d700 1 -- 172.21.15.113:0/2725487052 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514005040 con 0x7f05240fc890 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.298+0000 7f0522d9d700 1 --2- 172.21.15.113:0/2725487052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240fc890 0x7f052410cf70 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7f0514005010 tx=0x7f0514009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.298+0000 7f050bfff700 1 -- 172.21.15.113:0/2725487052 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f051400e030 con 0x7f05240fc890 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.298+0000 7f050bfff700 1 -- 172.21.15.113:0/2725487052 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0514007e90 con 0x7f05240fc890 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.298+0000 7f0529116700 1 -- 172.21.15.113:0/2725487052 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05241071d0 con 0x7f05240fc890 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.298+0000 7f050bfff700 1 -- 172.21.15.113:0/2725487052 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0514018070 con 0x7f05240fc890 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.298+0000 7f0529116700 1 -- 172.21.15.113:0/2725487052 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05241076b0 con 0x7f05240fc890 2024-04-12T05:51:26.295 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.299+0000 7f0529116700 1 -- 172.21.15.113:0/2725487052 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052405f2e0 con 0x7f05240fc890 2024-04-12T05:51:26.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.299+0000 7f050bfff700 1 -- 172.21.15.113:0/2725487052 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f051401d6e0 con 0x7f05240fc890 2024-04-12T05:51:26.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.300+0000 7f050bfff700 1 --2- 172.21.15.113:0/2725487052 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f050c06ca30 0x7f050c06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:26.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.300+0000 7f050bfff700 1 -- 172.21.15.113:0/2725487052 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f051408d100 con 0x7f05240fc890 2024-04-12T05:51:26.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.300+0000 7f052259c700 1 --2- 172.21.15.113:0/2725487052 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f050c06ca30 0x7f050c06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:26.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.300+0000 7f052259c700 1 --2- 172.21.15.113:0/2725487052 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f050c06ca30 0x7f050c06eef0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f05241088b0 tx=0x7f0518009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:26.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.304+0000 7f050bfff700 1 -- 172.21.15.113:0/2725487052 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f051405a920 con 0x7f05240fc890 2024-04-12T05:51:26.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:26 smithi192 ceph-mon[85304]: pgmap v1180: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:26.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:26 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2314340967' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-12T05:51:26.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:26 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/935692336' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-12T05:51:26.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:26 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3289306289' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-12T05:51:26.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.532+0000 7f0529116700 1 -- 172.21.15.113:0/2725487052 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f052405eec0 con 0x7f05240fc890 2024-04-12T05:51:26.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.533+0000 7f050bfff700 1 -- 172.21.15.113:0/2725487052 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v35) v1 ==== 107+0+4825 (secure 0 0 0) 0x7f051405a740 con 0x7f05240fc890 2024-04-12T05:51:26.529 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:26.529 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24271,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/3401306304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3401306304},{"type":"v1","addr":"172.21.15.192:6827","nonce":3401306304}]},"join_fscid":1,"export_targets":[],"features":4540138314316775423,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44331,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/335558856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":335558856},{"type":"v1","addr":"172.21.15.192:6825","nonce":335558856}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:18:43.275880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:26.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.535+0000 7f0529116700 1 -- 172.21.15.113:0/2725487052 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f050c06ca30 msgr2=0x7f050c06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:26.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.535+0000 7f0529116700 1 --2- 172.21.15.113:0/2725487052 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f050c06ca30 0x7f050c06eef0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f05241088b0 tx=0x7f0518009da0).stop 2024-04-12T05:51:26.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.535+0000 7f0529116700 1 -- 172.21.15.113:0/2725487052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240fc890 msgr2=0x7f052410cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:26.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.535+0000 7f0529116700 1 --2- 172.21.15.113:0/2725487052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240fc890 0x7f052410cf70 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7f0514005010 tx=0x7f0514009cb0).stop 2024-04-12T05:51:26.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.535+0000 7f0529116700 1 -- 172.21.15.113:0/2725487052 shutdown_connections 2024-04-12T05:51:26.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.535+0000 7f0529116700 1 --2- 172.21.15.113:0/2725487052 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f050c06ca30 0x7f050c06eef0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.535+0000 7f0529116700 1 --2- 172.21.15.113:0/2725487052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05240fc890 0x7f052410cf70 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.535+0000 7f0529116700 1 --2- 172.21.15.113:0/2725487052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05240fd1f0 0x7f052410d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.531 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.535+0000 7f0529116700 1 -- 172.21.15.113:0/2725487052 >> 172.21.15.113:0/2725487052 conn(0x7f05240f2490 msgr2=0x7f05240fac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:26.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.535+0000 7f0529116700 1 -- 172.21.15.113:0/2725487052 shutdown_connections 2024-04-12T05:51:26.532 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.535+0000 7f0529116700 1 -- 172.21.15.113:0/2725487052 wait complete. 2024-04-12T05:51:26.532 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 26 2024-04-12T05:51:26.543 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T05:51:26.543 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-12T05:51:26.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.664+0000 7fe937fab700 1 -- 172.21.15.113:0/988519932 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9300fd100 msgr2=0x7fe9300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:26.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.664+0000 7fe937fab700 1 --2- 172.21.15.113:0/988519932 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9300fd100 0x7fe9300f6930 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7fe92c0047f0 tx=0x7fe92c009cb0).stop 2024-04-12T05:51:26.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.664+0000 7fe937fab700 1 -- 172.21.15.113:0/988519932 shutdown_connections 2024-04-12T05:51:26.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.664+0000 7fe937fab700 1 --2- 172.21.15.113:0/988519932 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9300fd100 0x7fe9300f6930 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.664+0000 7fe937fab700 1 --2- 172.21.15.113:0/988519932 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9300fc830 0x7fe9300fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.664+0000 7fe937fab700 1 -- 172.21.15.113:0/988519932 >> 172.21.15.113:0/988519932 conn(0x7fe93006c760 msgr2=0x7fe93006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:26.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.664+0000 7fe937fab700 1 -- 172.21.15.113:0/988519932 shutdown_connections 2024-04-12T05:51:26.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.664+0000 7fe937fab700 1 -- 172.21.15.113:0/988519932 wait complete. 2024-04-12T05:51:26.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe937fab700 1 Processor -- start 2024-04-12T05:51:26.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe937fab700 1 -- start start 2024-04-12T05:51:26.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe937fab700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9300fc830 0x7fe93010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:26.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe937fab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9300fd100 0x7fe93010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:26.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe937fab700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93010ead0 con 0x7fe9300fd100 2024-04-12T05:51:26.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe937fab700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93010ec10 con 0x7fe9300fc830 2024-04-12T05:51:26.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe935546700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9300fd100 0x7fe93010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:26.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe935546700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9300fd100 0x7fe93010e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34130/0 (socket says 172.21.15.113:34130) 2024-04-12T05:51:26.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe935546700 1 -- 172.21.15.113:0/2366316432 learned_addr learned my addr 172.21.15.113:0/2366316432 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:26.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe935d47700 1 --2- 172.21.15.113:0/2366316432 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9300fc830 0x7fe93010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:26.663 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe935546700 1 -- 172.21.15.113:0/2366316432 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9300fc830 msgr2=0x7fe93010dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:26.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe935546700 1 --2- 172.21.15.113:0/2366316432 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9300fc830 0x7fe93010dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe935546700 1 -- 172.21.15.113:0/2366316432 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c005040 con 0x7fe9300fd100 2024-04-12T05:51:26.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe935546700 1 --2- 172.21.15.113:0/2366316432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9300fd100 0x7fe93010e4d0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7fe92c005010 tx=0x7fe92c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:26.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe922ffd700 1 -- 172.21.15.113:0/2366316432 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe92c00e030 con 0x7fe9300fd100 2024-04-12T05:51:26.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe922ffd700 1 -- 172.21.15.113:0/2366316432 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe92c007e90 con 0x7fe9300fd100 2024-04-12T05:51:26.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.665+0000 7fe937fab700 1 -- 172.21.15.113:0/2366316432 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe930108240 con 0x7fe9300fd100 2024-04-12T05:51:26.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.666+0000 7fe922ffd700 1 -- 172.21.15.113:0/2366316432 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe92c018070 con 0x7fe9300fd100 2024-04-12T05:51:26.664 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.666+0000 7fe937fab700 1 -- 172.21.15.113:0/2366316432 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9301087a0 con 0x7fe9300fd100 2024-04-12T05:51:26.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.666+0000 7fe937fab700 1 -- 172.21.15.113:0/2366316432 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93005f380 con 0x7fe9300fd100 2024-04-12T05:51:26.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.667+0000 7fe922ffd700 1 -- 172.21.15.113:0/2366316432 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe92c01d420 con 0x7fe9300fd100 2024-04-12T05:51:26.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.667+0000 7fe922ffd700 1 --2- 172.21.15.113:0/2366316432 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe91c06ca30 0x7fe91c06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:26.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.667+0000 7fe922ffd700 1 -- 172.21.15.113:0/2366316432 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fe92c08d270 con 0x7fe9300fd100 2024-04-12T05:51:26.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.667+0000 7fe935d47700 1 --2- 172.21.15.113:0/2366316432 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe91c06ca30 0x7fe91c06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:26.666 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.668+0000 7fe935d47700 1 --2- 172.21.15.113:0/2366316432 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe91c06ca30 0x7fe91c06eef0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7fe924009170 tx=0x7fe924009da0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:26.667 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.672+0000 7fe922ffd700 1 -- 172.21.15.113:0/2366316432 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fe92c05ad20 con 0x7fe9300fd100 2024-04-12T05:51:26.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.901+0000 7fe937fab700 1 -- 172.21.15.113:0/2366316432 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7fe93005eec0 con 0x7fe9300fd100 2024-04-12T05:51:26.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.902+0000 7fe922ffd700 1 -- 172.21.15.113:0/2366316432 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v35) v1 ==== 107+0+4043 (secure 0 0 0) 0x7fe92c05a8b0 con 0x7fe9300fd100 2024-04-12T05:51:26.898 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:26.898 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44331,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/335558856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":335558856},{"type":"v1","addr":"172.21.15.192:6825","nonce":335558856}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:18:43.275880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:26.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.904+0000 7fe937fab700 1 -- 172.21.15.113:0/2366316432 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe91c06ca30 msgr2=0x7fe91c06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:26.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.904+0000 7fe937fab700 1 --2- 172.21.15.113:0/2366316432 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe91c06ca30 0x7fe91c06eef0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7fe924009170 tx=0x7fe924009da0).stop 2024-04-12T05:51:26.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.904+0000 7fe937fab700 1 -- 172.21.15.113:0/2366316432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9300fd100 msgr2=0x7fe93010e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:26.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.904+0000 7fe937fab700 1 --2- 172.21.15.113:0/2366316432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9300fd100 0x7fe93010e4d0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7fe92c005010 tx=0x7fe92c009cb0).stop 2024-04-12T05:51:26.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.904+0000 7fe937fab700 1 -- 172.21.15.113:0/2366316432 shutdown_connections 2024-04-12T05:51:26.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.904+0000 7fe937fab700 1 --2- 172.21.15.113:0/2366316432 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fe91c06ca30 0x7fe91c06eef0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.904+0000 7fe937fab700 1 --2- 172.21.15.113:0/2366316432 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9300fc830 0x7fe93010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.904+0000 7fe937fab700 1 --2- 172.21.15.113:0/2366316432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9300fd100 0x7fe93010e4d0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:26.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.904+0000 7fe937fab700 1 -- 172.21.15.113:0/2366316432 >> 172.21.15.113:0/2366316432 conn(0x7fe93006c760 msgr2=0x7fe930060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:26.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.904+0000 7fe937fab700 1 -- 172.21.15.113:0/2366316432 shutdown_connections 2024-04-12T05:51:26.901 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:26.904+0000 7fe937fab700 1 -- 172.21.15.113:0/2366316432 wait complete. 2024-04-12T05:51:26.901 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 27 2024-04-12T05:51:26.912 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T05:51:26.912 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-12T05:51:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.035+0000 7f50bfbd7700 1 -- 172.21.15.113:0/3798580800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50b80fc830 msgr2=0x7f50b80fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.035+0000 7f50bfbd7700 1 --2- 172.21.15.113:0/3798580800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50b80fc830 0x7f50b80fcc10 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f50a80047f0 tx=0x7f50a8009cb0).stop 2024-04-12T05:51:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.035+0000 7f50bfbd7700 1 -- 172.21.15.113:0/3798580800 shutdown_connections 2024-04-12T05:51:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.035+0000 7f50bfbd7700 1 --2- 172.21.15.113:0/3798580800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b80fd100 0x7f50b80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.035+0000 7f50bfbd7700 1 --2- 172.21.15.113:0/3798580800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50b80fc830 0x7f50b80fcc10 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.035+0000 7f50bfbd7700 1 -- 172.21.15.113:0/3798580800 >> 172.21.15.113:0/3798580800 conn(0x7f50b806c760 msgr2=0x7f50b806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.035+0000 7f50bfbd7700 1 -- 172.21.15.113:0/3798580800 shutdown_connections 2024-04-12T05:51:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.035+0000 7f50bfbd7700 1 -- 172.21.15.113:0/3798580800 wait complete. 2024-04-12T05:51:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bfbd7700 1 Processor -- start 2024-04-12T05:51:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bfbd7700 1 -- start start 2024-04-12T05:51:27.033 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bfbd7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50b80fc830 0x7f50b8109050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bfbd7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b80fd100 0x7f50b8109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bfbd7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b8109a30 con 0x7f50b80fc830 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bfbd7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b8109b70 con 0x7f50b80fd100 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bd973700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50b80fc830 0x7f50b8109050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bd973700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50b80fc830 0x7f50b8109050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34146/0 (socket says 172.21.15.113:34146) 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bd973700 1 -- 172.21.15.113:0/2626425105 learned_addr learned my addr 172.21.15.113:0/2626425105 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bd172700 1 --2- 172.21.15.113:0/2626425105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b80fd100 0x7f50b8109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bd973700 1 -- 172.21.15.113:0/2626425105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b80fd100 msgr2=0x7f50b8109540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bd973700 1 --2- 172.21.15.113:0/2626425105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b80fd100 0x7f50b8109540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bd973700 1 -- 172.21.15.113:0/2626425105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a8005040 con 0x7f50b80fc830 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bd973700 1 --2- 172.21.15.113:0/2626425105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50b80fc830 0x7f50b8109050 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f50a8005010 tx=0x7f50a8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50aeffd700 1 -- 172.21.15.113:0/2626425105 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50a800e030 con 0x7f50b80fc830 2024-04-12T05:51:27.034 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50aeffd700 1 -- 172.21.15.113:0/2626425105 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50a8007e90 con 0x7f50b80fc830 2024-04-12T05:51:27.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50bfbd7700 1 -- 172.21.15.113:0/2626425105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b8109dc0 con 0x7f50b80fc830 2024-04-12T05:51:27.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.036+0000 7f50aeffd700 1 -- 172.21.15.113:0/2626425105 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50a8018070 con 0x7f50b80fc830 2024-04-12T05:51:27.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.037+0000 7f50bfbd7700 1 -- 172.21.15.113:0/2626425105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b81123b0 con 0x7f50b80fc830 2024-04-12T05:51:27.035 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.037+0000 7f50bfbd7700 1 -- 172.21.15.113:0/2626425105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b805f380 con 0x7f50b80fc830 2024-04-12T05:51:27.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.038+0000 7f50aeffd700 1 -- 172.21.15.113:0/2626425105 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f50a801d420 con 0x7f50b80fc830 2024-04-12T05:51:27.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.038+0000 7f50aeffd700 1 --2- 172.21.15.113:0/2626425105 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50a406ca30 0x7f50a406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:27.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.038+0000 7f50aeffd700 1 -- 172.21.15.113:0/2626425105 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f50a808d120 con 0x7f50b80fc830 2024-04-12T05:51:27.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.038+0000 7f50bd172700 1 --2- 172.21.15.113:0/2626425105 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50a406ca30 0x7f50a406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:27.036 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.039+0000 7f50bd172700 1 --2- 172.21.15.113:0/2626425105 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50a406ca30 0x7f50a406eef0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f50b8060f20 tx=0x7f50b400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:27.038 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.043+0000 7f50aeffd700 1 -- 172.21.15.113:0/2626425105 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f50a8057140 con 0x7f50b80fc830 2024-04-12T05:51:27.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:27 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1586136804' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-12T05:51:27.167 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:27 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2725487052' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-12T05:51:27.168 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:27 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2366316432' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-12T05:51:27.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.273+0000 7f50bfbd7700 1 -- 172.21.15.113:0/2626425105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7f50b80f9960 con 0x7f50b80fc830 2024-04-12T05:51:27.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.274+0000 7f50aeffd700 1 -- 172.21.15.113:0/2626425105 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v35) v1 ==== 107+0+4822 (secure 0 0 0) 0x7f50a801d6d0 con 0x7f50b80fc830 2024-04-12T05:51:27.269 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:27.269 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44331,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/335558856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":335558856},{"type":"v1","addr":"172.21.15.192:6825","nonce":335558856}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44817,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/946017279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946017279},{"type":"v1","addr":"172.21.15.192:6827","nonce":946017279}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:18:43.275880+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:27.271 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.276+0000 7f50bfbd7700 1 -- 172.21.15.113:0/2626425105 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50a406ca30 msgr2=0x7f50a406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:27.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.276+0000 7f50bfbd7700 1 --2- 172.21.15.113:0/2626425105 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50a406ca30 0x7f50a406eef0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f50b8060f20 tx=0x7f50b400b040).stop 2024-04-12T05:51:27.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.276+0000 7f50bfbd7700 1 -- 172.21.15.113:0/2626425105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50b80fc830 msgr2=0x7f50b8109050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:27.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.276+0000 7f50bfbd7700 1 --2- 172.21.15.113:0/2626425105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50b80fc830 0x7f50b8109050 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f50a8005010 tx=0x7f50a8009cb0).stop 2024-04-12T05:51:27.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.276+0000 7f50bfbd7700 1 -- 172.21.15.113:0/2626425105 shutdown_connections 2024-04-12T05:51:27.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.276+0000 7f50bfbd7700 1 --2- 172.21.15.113:0/2626425105 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f50a406ca30 0x7f50a406eef0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.276+0000 7f50bfbd7700 1 --2- 172.21.15.113:0/2626425105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f50b80fc830 0x7f50b8109050 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.276+0000 7f50bfbd7700 1 --2- 172.21.15.113:0/2626425105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b80fd100 0x7f50b8109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.276+0000 7f50bfbd7700 1 -- 172.21.15.113:0/2626425105 >> 172.21.15.113:0/2626425105 conn(0x7f50b806c760 msgr2=0x7f50b80ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:27.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.276+0000 7f50bfbd7700 1 -- 172.21.15.113:0/2626425105 shutdown_connections 2024-04-12T05:51:27.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.276+0000 7f50bfbd7700 1 -- 172.21.15.113:0/2626425105 wait complete. 2024-04-12T05:51:27.273 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 28 2024-04-12T05:51:27.283 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-12T05:51:27.283 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-12T05:51:27.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.405+0000 7fd029d9e700 1 -- 172.21.15.113:0/1831502738 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0240fd170 msgr2=0x7fd0240f6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:27.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.405+0000 7fd029d9e700 1 --2- 172.21.15.113:0/1831502738 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0240fd170 0x7fd0240f6950 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7fd0140047f0 tx=0x7fd014009cb0).stop 2024-04-12T05:51:27.401 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.405+0000 7fd029d9e700 1 -- 172.21.15.113:0/1831502738 shutdown_connections 2024-04-12T05:51:27.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.405+0000 7fd029d9e700 1 --2- 172.21.15.113:0/1831502738 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0240fd170 0x7fd0240f6950 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.405+0000 7fd029d9e700 1 --2- 172.21.15.113:0/1831502738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0240fc810 0x7fd0240fcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.405+0000 7fd029d9e700 1 -- 172.21.15.113:0/1831502738 >> 172.21.15.113:0/1831502738 conn(0x7fd0240f23f0 msgr2=0x7fd0240f4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:27.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.405+0000 7fd029d9e700 1 -- 172.21.15.113:0/1831502738 shutdown_connections 2024-04-12T05:51:27.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.405+0000 7fd029d9e700 1 -- 172.21.15.113:0/1831502738 wait complete. 2024-04-12T05:51:27.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.406+0000 7fd029d9e700 1 Processor -- start 2024-04-12T05:51:27.402 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.406+0000 7fd029d9e700 1 -- start start 2024-04-12T05:51:27.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.406+0000 7fd029d9e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0240fc810 0x7fd02410d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:27.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.406+0000 7fd029d9e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0240fd170 0x7fd02410d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:27.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.406+0000 7fd029d9e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02410daf0 con 0x7fd0240fd170 2024-04-12T05:51:27.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.406+0000 7fd029d9e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02410dc30 con 0x7fd0240fc810 2024-04-12T05:51:27.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.406+0000 7fd022ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0240fd170 0x7fd02410d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:27.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.406+0000 7fd022ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0240fd170 0x7fd02410d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34168/0 (socket says 172.21.15.113:34168) 2024-04-12T05:51:27.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.406+0000 7fd022ffd700 1 -- 172.21.15.113:0/2808292937 learned_addr learned my addr 172.21.15.113:0/2808292937 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:27.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.407+0000 7fd0237fe700 1 --2- 172.21.15.113:0/2808292937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0240fc810 0x7fd02410d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:27.404 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.407+0000 7fd022ffd700 1 -- 172.21.15.113:0/2808292937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0240fc810 msgr2=0x7fd02410d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:27.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.407+0000 7fd022ffd700 1 --2- 172.21.15.113:0/2808292937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0240fc810 0x7fd02410d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.407+0000 7fd022ffd700 1 -- 172.21.15.113:0/2808292937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd014005040 con 0x7fd0240fd170 2024-04-12T05:51:27.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.407+0000 7fd022ffd700 1 --2- 172.21.15.113:0/2808292937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0240fd170 0x7fd02410d4f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7fd014000c00 tx=0x7fd014009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:27.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.407+0000 7fd020ff9700 1 -- 172.21.15.113:0/2808292937 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd014018070 con 0x7fd0240fd170 2024-04-12T05:51:27.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.407+0000 7fd020ff9700 1 -- 172.21.15.113:0/2808292937 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd014007e90 con 0x7fd0240fd170 2024-04-12T05:51:27.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.407+0000 7fd029d9e700 1 -- 172.21.15.113:0/2808292937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd024107260 con 0x7fd0240fd170 2024-04-12T05:51:27.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.407+0000 7fd020ff9700 1 -- 172.21.15.113:0/2808292937 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd01401de20 con 0x7fd0240fd170 2024-04-12T05:51:27.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.407+0000 7fd029d9e700 1 -- 172.21.15.113:0/2808292937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0241077c0 con 0x7fd0240fd170 2024-04-12T05:51:27.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.408+0000 7fd029d9e700 1 -- 172.21.15.113:0/2808292937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02405f2e0 con 0x7fd0240fd170 2024-04-12T05:51:27.405 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.408+0000 7fd020ff9700 1 -- 172.21.15.113:0/2808292937 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd01401d420 con 0x7fd0240fd170 2024-04-12T05:51:27.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.409+0000 7fd020ff9700 1 --2- 172.21.15.113:0/2808292937 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd00c06caa0 0x7fd00c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:27.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.409+0000 7fd020ff9700 1 -- 172.21.15.113:0/2808292937 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fd01408c950 con 0x7fd0240fd170 2024-04-12T05:51:27.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.409+0000 7fd0237fe700 1 --2- 172.21.15.113:0/2808292937 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd00c06caa0 0x7fd00c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:27.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.409+0000 7fd0237fe700 1 --2- 172.21.15.113:0/2808292937 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd00c06caa0 0x7fd00c06ef60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fd018004ee0 tx=0x7fd01800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:27.409 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.413+0000 7fd020ff9700 1 -- 172.21.15.113:0/2808292937 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fd01405a400 con 0x7fd0240fd170 2024-04-12T05:51:27.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:27 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1586136804' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-12T05:51:27.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:27 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2725487052' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-12T05:51:27.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:27 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2366316432' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-12T05:51:27.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.644+0000 7fd029d9e700 1 -- 172.21.15.113:0/2808292937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7fd02405eec0 con 0x7fd0240fd170 2024-04-12T05:51:27.640 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.645+0000 7fd020ff9700 1 -- 172.21.15.113:0/2808292937 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v35) v1 ==== 107+0+4821 (secure 0 0 0) 0x7fd014059f90 con 0x7fd0240fd170 2024-04-12T05:51:27.641 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:27.641 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":22},{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44331,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/335558856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":335558856},{"type":"v1","addr":"172.21.15.192:6825","nonce":335558856}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44817,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/946017279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946017279},{"type":"v1","addr":"172.21.15.192:6827","nonce":946017279}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:49:40.025571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:27.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.647+0000 7fd029d9e700 1 -- 172.21.15.113:0/2808292937 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd00c06caa0 msgr2=0x7fd00c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:27.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.647+0000 7fd029d9e700 1 --2- 172.21.15.113:0/2808292937 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd00c06caa0 0x7fd00c06ef60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fd018004ee0 tx=0x7fd01800b040).stop 2024-04-12T05:51:27.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.647+0000 7fd029d9e700 1 -- 172.21.15.113:0/2808292937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0240fd170 msgr2=0x7fd02410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:27.643 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.647+0000 7fd029d9e700 1 --2- 172.21.15.113:0/2808292937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0240fd170 0x7fd02410d4f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7fd014000c00 tx=0x7fd014009cb0).stop 2024-04-12T05:51:27.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.647+0000 7fd029d9e700 1 -- 172.21.15.113:0/2808292937 shutdown_connections 2024-04-12T05:51:27.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.647+0000 7fd029d9e700 1 --2- 172.21.15.113:0/2808292937 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fd00c06caa0 0x7fd00c06ef60 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.647+0000 7fd029d9e700 1 --2- 172.21.15.113:0/2808292937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0240fc810 0x7fd02410d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.647+0000 7fd029d9e700 1 --2- 172.21.15.113:0/2808292937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0240fd170 0x7fd02410d4f0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.647+0000 7fd029d9e700 1 -- 172.21.15.113:0/2808292937 >> 172.21.15.113:0/2808292937 conn(0x7fd0240f23f0 msgr2=0x7fd0240fabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:27.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.648+0000 7fd029d9e700 1 -- 172.21.15.113:0/2808292937 shutdown_connections 2024-04-12T05:51:27.644 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.648+0000 7fd029d9e700 1 -- 172.21.15.113:0/2808292937 wait complete. 2024-04-12T05:51:27.644 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 29 2024-04-12T05:51:27.655 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-12T05:51:27.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.778+0000 7f406c8d3700 1 -- 172.21.15.113:0/1123684712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40680fd160 msgr2=0x7f40680f6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:27.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.778+0000 7f406c8d3700 1 --2- 172.21.15.113:0/1123684712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40680fd160 0x7f40680f6f60 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0x7f40580047f0 tx=0x7f4058009cb0).stop 2024-04-12T05:51:27.779 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.778+0000 7f406c8d3700 1 -- 172.21.15.113:0/1123684712 shutdown_connections 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.778+0000 7f406c8d3700 1 --2- 172.21.15.113:0/1123684712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40680fd160 0x7f40680f6f60 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.778+0000 7f406c8d3700 1 --2- 172.21.15.113:0/1123684712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40680fc800 0x7f40680fcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.778+0000 7f406c8d3700 1 -- 172.21.15.113:0/1123684712 >> 172.21.15.113:0/1123684712 conn(0x7f406806cb60 msgr2=0x7f40680f3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.778+0000 7f406c8d3700 1 -- 172.21.15.113:0/1123684712 shutdown_connections 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.778+0000 7f406c8d3700 1 -- 172.21.15.113:0/1123684712 wait complete. 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406c8d3700 1 Processor -- start 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406c8d3700 1 -- start start 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406c8d3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40680fc800 0x7f40681090b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406c8d3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40681095a0 0x7f4068109a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406c8d3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4068109f10 con 0x7f40680fc800 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406c8d3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4068112120 con 0x7f40681095a0 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40680fc800 0x7f40681090b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406659c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40680fc800 0x7f40681090b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34192/0 (socket says 172.21.15.113:34192) 2024-04-12T05:51:27.780 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406659c700 1 -- 172.21.15.113:0/741273735 learned_addr learned my addr 172.21.15.113:0/741273735 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f4065d9b700 1 --2- 172.21.15.113:0/741273735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40681095a0 0x7f4068109a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406659c700 1 -- 172.21.15.113:0/741273735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40681095a0 msgr2=0x7f4068109a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406659c700 1 --2- 172.21.15.113:0/741273735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40681095a0 0x7f4068109a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.779+0000 7f406659c700 1 -- 172.21.15.113:0/741273735 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4058005040 con 0x7f40680fc800 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.780+0000 7f406659c700 1 --2- 172.21.15.113:0/741273735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40680fc800 0x7f40681090b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f405c00c7c0 tx=0x7f405c004330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.780+0000 7f40577fe700 1 -- 172.21.15.113:0/741273735 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f405c00e680 con 0x7f40680fc800 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.780+0000 7f40577fe700 1 -- 172.21.15.113:0/741273735 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f405c00ec90 con 0x7f40680fc800 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.780+0000 7f406c8d3700 1 -- 172.21.15.113:0/741273735 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40681122c0 con 0x7f40680fc800 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.780+0000 7f40577fe700 1 -- 172.21.15.113:0/741273735 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f405c012d60 con 0x7f40680fc800 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.780+0000 7f406c8d3700 1 -- 172.21.15.113:0/741273735 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4068112880 con 0x7f40680fc800 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.781+0000 7f406c8d3700 1 -- 172.21.15.113:0/741273735 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40680606f0 con 0x7f40680fc800 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.781+0000 7f40577fe700 1 -- 172.21.15.113:0/741273735 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f405c012400 con 0x7f40680fc800 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.781+0000 7f40577fe700 1 --2- 172.21.15.113:0/741273735 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f405006caa0 0x7f405006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.782+0000 7f40577fe700 1 -- 172.21.15.113:0/741273735 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f405c08b720 con 0x7f40680fc800 2024-04-12T05:51:27.781 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.782+0000 7f4065d9b700 1 --2- 172.21.15.113:0/741273735 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f405006caa0 0x7f405006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:27.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.782+0000 7f4065d9b700 1 --2- 172.21.15.113:0/741273735 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f405006caa0 0x7f405006ef60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f4058007cf0 tx=0x7f405800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:27.782 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:27.786+0000 7f40577fe700 1 -- 172.21.15.113:0/741273735 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f405c055740 con 0x7f40680fc800 2024-04-12T05:51:28.014 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.018+0000 7f406c8d3700 1 -- 172.21.15.113:0/741273735 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7f40680f6510 con 0x7f40680fc800 2024-04-12T05:51:28.015 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.019+0000 7f40577fe700 1 -- 172.21.15.113:0/741273735 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v35) v1 ==== 107+0+4844 (secure 0 0 0) 0x7f405c058d60 con 0x7f40680fc800 2024-04-12T05:51:28.015 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:28.015 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44331,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/335558856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":335558856},{"type":"v1","addr":"172.21.15.192:6825","nonce":335558856}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44817,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/946017279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946017279},{"type":"v1","addr":"172.21.15.192:6827","nonce":946017279}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:49:40.030028+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34412,"mds_1":34444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34412":{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34444":{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":1,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:28.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.021+0000 7f406c8d3700 1 -- 172.21.15.113:0/741273735 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f405006caa0 msgr2=0x7f405006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.021+0000 7f406c8d3700 1 --2- 172.21.15.113:0/741273735 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f405006caa0 0x7f405006ef60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f4058007cf0 tx=0x7f405800b040).stop 2024-04-12T05:51:28.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.021+0000 7f406c8d3700 1 -- 172.21.15.113:0/741273735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40680fc800 msgr2=0x7f40681090b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.021+0000 7f406c8d3700 1 --2- 172.21.15.113:0/741273735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40680fc800 0x7f40681090b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f405c00c7c0 tx=0x7f405c004330).stop 2024-04-12T05:51:28.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.021+0000 7f406c8d3700 1 -- 172.21.15.113:0/741273735 shutdown_connections 2024-04-12T05:51:28.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.021+0000 7f406c8d3700 1 --2- 172.21.15.113:0/741273735 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f405006caa0 0x7f405006ef60 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.018 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.021+0000 7f406c8d3700 1 --2- 172.21.15.113:0/741273735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40680fc800 0x7f40681090b0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.021+0000 7f406c8d3700 1 --2- 172.21.15.113:0/741273735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40681095a0 0x7f4068109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.021+0000 7f406c8d3700 1 -- 172.21.15.113:0/741273735 >> 172.21.15.113:0/741273735 conn(0x7f406806cb60 msgr2=0x7f40680fabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:28.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.021+0000 7f406c8d3700 1 -- 172.21.15.113:0/741273735 shutdown_connections 2024-04-12T05:51:28.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.021+0000 7f406c8d3700 1 -- 172.21.15.113:0/741273735 wait complete. 2024-04-12T05:51:28.019 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 30 2024-04-12T05:51:28.029 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-12T05:51:28.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 -- 172.21.15.113:0/2477968940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90a80fc830 msgr2=0x7f90a80fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 --2- 172.21.15.113:0/2477968940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90a80fc830 0x7f90a80fcc10 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7f909c0047f0 tx=0x7f909c009cb0).stop 2024-04-12T05:51:28.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 -- 172.21.15.113:0/2477968940 shutdown_connections 2024-04-12T05:51:28.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 --2- 172.21.15.113:0/2477968940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90a80fd100 0x7f90a80f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.149 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 --2- 172.21.15.113:0/2477968940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90a80fc830 0x7f90a80fcc10 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 -- 172.21.15.113:0/2477968940 >> 172.21.15.113:0/2477968940 conn(0x7f90a806c760 msgr2=0x7f90a806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 -- 172.21.15.113:0/2477968940 shutdown_connections 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 -- 172.21.15.113:0/2477968940 wait complete. 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 Processor -- start 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 -- start start 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90a80fc830 0x7f90a810cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90a80fd100 0x7f90a810d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a810d990 con 0x7f90a80fd100 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.152+0000 7f90aed61700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a810dad0 con 0x7f90a80fc830 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90a80fd100 0x7f90a810d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90a80fd100 0x7f90a810d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34206/0 (socket says 172.21.15.113:34206) 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90a7fff700 1 -- 172.21.15.113:0/2166269201 learned_addr learned my addr 172.21.15.113:0/2166269201 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90acafd700 1 --2- 172.21.15.113:0/2166269201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90a80fc830 0x7f90a810cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:28.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90a7fff700 1 -- 172.21.15.113:0/2166269201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90a80fc830 msgr2=0x7f90a810cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90a7fff700 1 --2- 172.21.15.113:0/2166269201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90a80fc830 0x7f90a810cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90a7fff700 1 -- 172.21.15.113:0/2166269201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c005040 con 0x7f90a80fd100 2024-04-12T05:51:28.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90a7fff700 1 --2- 172.21.15.113:0/2166269201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90a80fd100 0x7f90a810d390 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f90940041d0 tx=0x7f9094009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:28.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90a5ffb700 1 -- 172.21.15.113:0/2166269201 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9094007b40 con 0x7f90a80fd100 2024-04-12T05:51:28.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90a5ffb700 1 -- 172.21.15.113:0/2166269201 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f909400f040 con 0x7f90a80fd100 2024-04-12T05:51:28.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90aed61700 1 -- 172.21.15.113:0/2166269201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a8107160 con 0x7f90a80fd100 2024-04-12T05:51:28.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90a5ffb700 1 -- 172.21.15.113:0/2166269201 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f909400d800 con 0x7f90a80fd100 2024-04-12T05:51:28.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.153+0000 7f90aed61700 1 -- 172.21.15.113:0/2166269201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a8107720 con 0x7f90a80fd100 2024-04-12T05:51:28.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.154+0000 7f90aed61700 1 -- 172.21.15.113:0/2166269201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a805f380 con 0x7f90a80fd100 2024-04-12T05:51:28.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.155+0000 7f90a5ffb700 1 -- 172.21.15.113:0/2166269201 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9094007ca0 con 0x7f90a80fd100 2024-04-12T05:51:28.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.155+0000 7f90a5ffb700 1 --2- 172.21.15.113:0/2166269201 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f909806caa0 0x7f909806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:28.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.155+0000 7f90a5ffb700 1 -- 172.21.15.113:0/2166269201 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f909408b950 con 0x7f90a80fd100 2024-04-12T05:51:28.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.155+0000 7f90acafd700 1 --2- 172.21.15.113:0/2166269201 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f909806caa0 0x7f909806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:28.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.156+0000 7f90acafd700 1 --2- 172.21.15.113:0/2166269201 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f909806caa0 0x7f909806ef60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f909c000c00 tx=0x7f909c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:28.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.160+0000 7f90a5ffb700 1 -- 172.21.15.113:0/2166269201 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9094059400 con 0x7f90a80fd100 2024-04-12T05:51:28.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.392+0000 7f90aed61700 1 -- 172.21.15.113:0/2166269201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7f90a805eec0 con 0x7f90a80fd100 2024-04-12T05:51:28.390 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.392+0000 7f90a5ffb700 1 -- 172.21.15.113:0/2166269201 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v35) v1 ==== 107+0+4847 (secure 0 0 0) 0x7f90a805eec0 con 0x7f90a80fd100 2024-04-12T05:51:28.391 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:28.391 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44331,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/335558856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":335558856},{"type":"v1","addr":"172.21.15.192:6825","nonce":335558856}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44817,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/946017279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946017279},{"type":"v1","addr":"172.21.15.192:6827","nonce":946017279}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:49:46.589995+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34412,"mds_1":34444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34412":{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34444":{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":1,"incarnation":30,"state":"up:resolve","state_seq":460,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:28.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.395+0000 7f90aed61700 1 -- 172.21.15.113:0/2166269201 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f909806caa0 msgr2=0x7f909806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.395+0000 7f90aed61700 1 --2- 172.21.15.113:0/2166269201 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f909806caa0 0x7f909806ef60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f909c000c00 tx=0x7f909c00b040).stop 2024-04-12T05:51:28.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.395+0000 7f90aed61700 1 -- 172.21.15.113:0/2166269201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90a80fd100 msgr2=0x7f90a810d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.396+0000 7f90aed61700 1 --2- 172.21.15.113:0/2166269201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90a80fd100 0x7f90a810d390 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f90940041d0 tx=0x7f9094009f60).stop 2024-04-12T05:51:28.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.396+0000 7f90aed61700 1 -- 172.21.15.113:0/2166269201 shutdown_connections 2024-04-12T05:51:28.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.396+0000 7f90aed61700 1 --2- 172.21.15.113:0/2166269201 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f909806caa0 0x7f909806ef60 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.392 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.396+0000 7f90aed61700 1 --2- 172.21.15.113:0/2166269201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90a80fc830 0x7f90a810cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.396+0000 7f90aed61700 1 --2- 172.21.15.113:0/2166269201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90a80fd100 0x7f90a810d390 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.396+0000 7f90aed61700 1 -- 172.21.15.113:0/2166269201 >> 172.21.15.113:0/2166269201 conn(0x7f90a806c760 msgr2=0x7f90a80ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:28.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.396+0000 7f90aed61700 1 -- 172.21.15.113:0/2166269201 shutdown_connections 2024-04-12T05:51:28.393 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.396+0000 7f90aed61700 1 -- 172.21.15.113:0/2166269201 wait complete. 2024-04-12T05:51:28.393 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 31 2024-04-12T05:51:28.402 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:28 smithi113 ceph-mon[103371]: pgmap v1181: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:28.402 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:28 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2626425105' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-12T05:51:28.402 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:28 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2808292937' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-12T05:51:28.404 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-12T05:51:28.441 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:28 smithi192 ceph-mon[85304]: pgmap v1181: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:28.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:28 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2626425105' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-12T05:51:28.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:28 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2808292937' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-12T05:51:28.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.525+0000 7fcac336b700 1 -- 172.21.15.113:0/2009109112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcabc0fd100 msgr2=0x7fcabc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.525+0000 7fcac336b700 1 --2- 172.21.15.113:0/2009109112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcabc0fd100 0x7fcabc0f6930 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7fcaac0047f0 tx=0x7fcaac009cb0).stop 2024-04-12T05:51:28.522 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.525+0000 7fcac336b700 1 -- 172.21.15.113:0/2009109112 shutdown_connections 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.525+0000 7fcac336b700 1 --2- 172.21.15.113:0/2009109112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcabc0fd100 0x7fcabc0f6930 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.525+0000 7fcac336b700 1 --2- 172.21.15.113:0/2009109112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcabc0fc830 0x7fcabc0fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.525+0000 7fcac336b700 1 -- 172.21.15.113:0/2009109112 >> 172.21.15.113:0/2009109112 conn(0x7fcabc06c760 msgr2=0x7fcabc06cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.525+0000 7fcac336b700 1 -- 172.21.15.113:0/2009109112 shutdown_connections 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.525+0000 7fcac336b700 1 -- 172.21.15.113:0/2009109112 wait complete. 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac336b700 1 Processor -- start 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac336b700 1 -- start start 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac336b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcabc0fc830 0x7fcabc10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac336b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcabc0fd100 0x7fcabc10d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac336b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcabc10d9b0 con 0x7fcabc0fd100 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac336b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcabc10daf0 con 0x7fcabc0fc830 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac0906700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcabc0fd100 0x7fcabc10d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac0906700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcabc0fd100 0x7fcabc10d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34228/0 (socket says 172.21.15.113:34228) 2024-04-12T05:51:28.523 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac0906700 1 -- 172.21.15.113:0/2977542896 learned_addr learned my addr 172.21.15.113:0/2977542896 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac1107700 1 --2- 172.21.15.113:0/2977542896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcabc0fc830 0x7fcabc10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac0906700 1 -- 172.21.15.113:0/2977542896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcabc0fc830 msgr2=0x7fcabc10cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac0906700 1 --2- 172.21.15.113:0/2977542896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcabc0fc830 0x7fcabc10cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac0906700 1 -- 172.21.15.113:0/2977542896 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaac005040 con 0x7fcabc0fd100 2024-04-12T05:51:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac0906700 1 --2- 172.21.15.113:0/2977542896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcabc0fd100 0x7fcabc10d3b0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7fcaac005010 tx=0x7fcaac009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcab27fc700 1 -- 172.21.15.113:0/2977542896 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaac00e030 con 0x7fcabc0fd100 2024-04-12T05:51:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcab27fc700 1 -- 172.21.15.113:0/2977542896 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcaac007e90 con 0x7fcabc0fd100 2024-04-12T05:51:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcab27fc700 1 -- 172.21.15.113:0/2977542896 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaac018070 con 0x7fcabc0fd100 2024-04-12T05:51:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.526+0000 7fcac336b700 1 -- 172.21.15.113:0/2977542896 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcabc107120 con 0x7fcabc0fd100 2024-04-12T05:51:28.524 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.527+0000 7fcac336b700 1 -- 172.21.15.113:0/2977542896 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcabc107680 con 0x7fcabc0fd100 2024-04-12T05:51:28.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.527+0000 7fcac336b700 1 -- 172.21.15.113:0/2977542896 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcabc05f380 con 0x7fcabc0fd100 2024-04-12T05:51:28.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.528+0000 7fcab27fc700 1 -- 172.21.15.113:0/2977542896 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcaac01d420 con 0x7fcabc0fd100 2024-04-12T05:51:28.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.529+0000 7fcab27fc700 1 --2- 172.21.15.113:0/2977542896 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcaa806ca30 0x7fcaa806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:28.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.529+0000 7fcab27fc700 1 -- 172.21.15.113:0/2977542896 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fcaac08d450 con 0x7fcabc0fd100 2024-04-12T05:51:28.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.529+0000 7fcac1107700 1 --2- 172.21.15.113:0/2977542896 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcaa806ca30 0x7fcaa806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:28.525 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.529+0000 7fcac1107700 1 --2- 172.21.15.113:0/2977542896 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcaa806ca30 0x7fcaa806eef0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7fcab8004dc0 tx=0x7fcab800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:28.528 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.533+0000 7fcab27fc700 1 -- 172.21.15.113:0/2977542896 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fcaac05af00 con 0x7fcabc0fd100 2024-04-12T05:51:28.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.765+0000 7fcac336b700 1 -- 172.21.15.113:0/2977542896 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7fcabc05eec0 con 0x7fcabc0fd100 2024-04-12T05:51:28.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.765+0000 7fcab27fc700 1 -- 172.21.15.113:0/2977542896 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v35) v1 ==== 107+0+4850 (secure 0 0 0) 0x7fcaac01d6d0 con 0x7fcabc0fd100 2024-04-12T05:51:28.761 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:28.761 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44331,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/335558856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":335558856},{"type":"v1","addr":"172.21.15.192:6825","nonce":335558856}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44817,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/946017279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946017279},{"type":"v1","addr":"172.21.15.192:6827","nonce":946017279}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:49:47.591454+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34412,"mds_1":34444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34412":{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":0,"incarnation":30,"state":"up:resolve","state_seq":465,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34444":{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":1,"incarnation":30,"state":"up:resolve","state_seq":460,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:28.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.767+0000 7fcac336b700 1 -- 172.21.15.113:0/2977542896 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcaa806ca30 msgr2=0x7fcaa806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.767+0000 7fcac336b700 1 --2- 172.21.15.113:0/2977542896 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcaa806ca30 0x7fcaa806eef0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7fcab8004dc0 tx=0x7fcab800b040).stop 2024-04-12T05:51:28.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.767+0000 7fcac336b700 1 -- 172.21.15.113:0/2977542896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcabc0fd100 msgr2=0x7fcabc10d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.767+0000 7fcac336b700 1 --2- 172.21.15.113:0/2977542896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcabc0fd100 0x7fcabc10d3b0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7fcaac005010 tx=0x7fcaac009cb0).stop 2024-04-12T05:51:28.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.767+0000 7fcac336b700 1 -- 172.21.15.113:0/2977542896 shutdown_connections 2024-04-12T05:51:28.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.767+0000 7fcac336b700 1 --2- 172.21.15.113:0/2977542896 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fcaa806ca30 0x7fcaa806eef0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.767+0000 7fcac336b700 1 --2- 172.21.15.113:0/2977542896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcabc0fc830 0x7fcabc10cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.767+0000 7fcac336b700 1 --2- 172.21.15.113:0/2977542896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcabc0fd100 0x7fcabc10d3b0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.768+0000 7fcac336b700 1 -- 172.21.15.113:0/2977542896 >> 172.21.15.113:0/2977542896 conn(0x7fcabc06c760 msgr2=0x7fcabc0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:28.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.768+0000 7fcac336b700 1 -- 172.21.15.113:0/2977542896 shutdown_connections 2024-04-12T05:51:28.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.768+0000 7fcac336b700 1 -- 172.21.15.113:0/2977542896 wait complete. 2024-04-12T05:51:28.764 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 32 2024-04-12T05:51:28.775 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-12T05:51:28.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.897+0000 7f2b06199700 1 -- 172.21.15.113:0/159531550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b000fd0f0 msgr2=0x7f2b000f6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.894 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.897+0000 7f2b06199700 1 --2- 172.21.15.113:0/159531550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b000fd0f0 0x7f2b000f6f80 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7f2ae80047f0 tx=0x7f2ae8009cb0).stop 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.897+0000 7f2b06199700 1 -- 172.21.15.113:0/159531550 shutdown_connections 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.897+0000 7f2b06199700 1 --2- 172.21.15.113:0/159531550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b000fd0f0 0x7f2b000f6f80 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.897+0000 7f2b06199700 1 --2- 172.21.15.113:0/159531550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b000fc790 0x7f2b000fcb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.897+0000 7f2b06199700 1 -- 172.21.15.113:0/159531550 >> 172.21.15.113:0/159531550 conn(0x7f2b0006cb60 msgr2=0x7f2b000f3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.897+0000 7f2b06199700 1 -- 172.21.15.113:0/159531550 shutdown_connections 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.897+0000 7f2b06199700 1 -- 172.21.15.113:0/159531550 wait complete. 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.897+0000 7f2b06199700 1 Processor -- start 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2b06199700 1 -- start start 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2b06199700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b000fc790 0x7f2b0010d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2b06199700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b000fd0f0 0x7f2b0010d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2b06199700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0010db00 con 0x7f2b000fc790 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2b06199700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0010dc40 con 0x7f2b000fd0f0 2024-04-12T05:51:28.895 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2aff7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b000fc790 0x7f2b0010d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2aff7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b000fc790 0x7f2b0010d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34248/0 (socket says 172.21.15.113:34248) 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2aff7fe700 1 -- 172.21.15.113:0/3660152062 learned_addr learned my addr 172.21.15.113:0/3660152062 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2afeffd700 1 --2- 172.21.15.113:0/3660152062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b000fd0f0 0x7f2b0010d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2aff7fe700 1 -- 172.21.15.113:0/3660152062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b000fd0f0 msgr2=0x7f2b0010d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2aff7fe700 1 --2- 172.21.15.113:0/3660152062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b000fd0f0 0x7f2b0010d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2aff7fe700 1 -- 172.21.15.113:0/3660152062 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae8005040 con 0x7f2b000fc790 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2aff7fe700 1 --2- 172.21.15.113:0/3660152062 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b000fc790 0x7f2b0010d010 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0x7f2af000e910 tx=0x7f2af000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.898+0000 7f2afcff9700 1 -- 172.21.15.113:0/3660152062 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2af000eeb0 con 0x7f2b000fc790 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.899+0000 7f2afcff9700 1 -- 172.21.15.113:0/3660152062 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2af000f040 con 0x7f2b000fc790 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.899+0000 7f2b06199700 1 -- 172.21.15.113:0/3660152062 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b00107270 con 0x7f2b000fc790 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.899+0000 7f2afcff9700 1 -- 172.21.15.113:0/3660152062 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2af000c8a0 con 0x7f2b000fc790 2024-04-12T05:51:28.896 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.899+0000 7f2b06199700 1 -- 172.21.15.113:0/3660152062 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b00107830 con 0x7f2b000fc790 2024-04-12T05:51:28.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.900+0000 7f2b06199700 1 -- 172.21.15.113:0/3660152062 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b000606f0 con 0x7f2b000fc790 2024-04-12T05:51:28.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.900+0000 7f2afcff9700 1 -- 172.21.15.113:0/3660152062 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2af00090d0 con 0x7f2b000fc790 2024-04-12T05:51:28.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.900+0000 7f2afcff9700 1 --2- 172.21.15.113:0/3660152062 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2aec06cab0 0x7f2aec06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:28.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.900+0000 7f2afcff9700 1 -- 172.21.15.113:0/3660152062 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2af008b920 con 0x7f2b000fc790 2024-04-12T05:51:28.897 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.900+0000 7f2afeffd700 1 --2- 172.21.15.113:0/3660152062 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2aec06cab0 0x7f2aec06ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:28.898 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.901+0000 7f2afeffd700 1 --2- 172.21.15.113:0/3660152062 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2aec06cab0 0x7f2aec06ef70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f2ae8005010 tx=0x7f2ae800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:28.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:28.905+0000 7f2afcff9700 1 -- 172.21.15.113:0/3660152062 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2af00593d0 con 0x7f2b000fc790 2024-04-12T05:51:29.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.136+0000 7f2b06199700 1 -- 172.21.15.113:0/3660152062 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f2b0005eec0 con 0x7f2b000fc790 2024-04-12T05:51:29.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.137+0000 7f2afcff9700 1 -- 172.21.15.113:0/3660152062 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v35) v1 ==== 107+0+4854 (secure 0 0 0) 0x7f2af0058f60 con 0x7f2b000fc790 2024-04-12T05:51:29.133 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:29.133 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44331,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/335558856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":335558856},{"type":"v1","addr":"172.21.15.192:6825","nonce":335558856}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44817,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/946017279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946017279},{"type":"v1","addr":"172.21.15.192:6827","nonce":946017279}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:49:48.596071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34412,"mds_1":34444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34412":{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":0,"incarnation":30,"state":"up:reconnect","state_seq":466,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34444":{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":1,"incarnation":30,"state":"up:reconnect","state_seq":461,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:29.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.139+0000 7f2b06199700 1 -- 172.21.15.113:0/3660152062 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2aec06cab0 msgr2=0x7f2aec06ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:29.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.139+0000 7f2b06199700 1 --2- 172.21.15.113:0/3660152062 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2aec06cab0 0x7f2aec06ef70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f2ae8005010 tx=0x7f2ae800b040).stop 2024-04-12T05:51:29.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.139+0000 7f2b06199700 1 -- 172.21.15.113:0/3660152062 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b000fc790 msgr2=0x7f2b0010d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:29.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.139+0000 7f2b06199700 1 --2- 172.21.15.113:0/3660152062 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b000fc790 0x7f2b0010d010 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0x7f2af000e910 tx=0x7f2af000b040).stop 2024-04-12T05:51:29.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.139+0000 7f2b06199700 1 -- 172.21.15.113:0/3660152062 shutdown_connections 2024-04-12T05:51:29.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.139+0000 7f2b06199700 1 --2- 172.21.15.113:0/3660152062 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2aec06cab0 0x7f2aec06ef70 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.139+0000 7f2b06199700 1 --2- 172.21.15.113:0/3660152062 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b000fc790 0x7f2b0010d010 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.135 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.139+0000 7f2b06199700 1 --2- 172.21.15.113:0/3660152062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b000fd0f0 0x7f2b0010d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.139+0000 7f2b06199700 1 -- 172.21.15.113:0/3660152062 >> 172.21.15.113:0/3660152062 conn(0x7f2b0006cb60 msgr2=0x7f2b000fabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:29.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.139+0000 7f2b06199700 1 -- 172.21.15.113:0/3660152062 shutdown_connections 2024-04-12T05:51:29.136 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.139+0000 7f2b06199700 1 -- 172.21.15.113:0/3660152062 wait complete. 2024-04-12T05:51:29.136 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 33 2024-04-12T05:51:29.146 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:29 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/741273735' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-12T05:51:29.146 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:29 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2166269201' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-12T05:51:29.146 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:29 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2977542896' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-12T05:51:29.147 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-12T05:51:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.268+0000 7fdaca0ae700 1 -- 172.21.15.113:0/217308972 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdac40fd100 msgr2=0x7fdac40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.268+0000 7fdaca0ae700 1 --2- 172.21.15.113:0/217308972 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdac40fd100 0x7fdac40f6930 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7fdab40047f0 tx=0x7fdab4009cb0).stop 2024-04-12T05:51:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.268+0000 7fdaca0ae700 1 -- 172.21.15.113:0/217308972 shutdown_connections 2024-04-12T05:51:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.268+0000 7fdaca0ae700 1 --2- 172.21.15.113:0/217308972 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdac40fd100 0x7fdac40f6930 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.268+0000 7fdaca0ae700 1 --2- 172.21.15.113:0/217308972 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdac40fc830 0x7fdac40fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.268+0000 7fdaca0ae700 1 -- 172.21.15.113:0/217308972 >> 172.21.15.113:0/217308972 conn(0x7fdac406c760 msgr2=0x7fdac406cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.268+0000 7fdaca0ae700 1 -- 172.21.15.113:0/217308972 shutdown_connections 2024-04-12T05:51:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.268+0000 7fdaca0ae700 1 -- 172.21.15.113:0/217308972 wait complete. 2024-04-12T05:51:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.269+0000 7fdaca0ae700 1 Processor -- start 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.269+0000 7fdaca0ae700 1 -- start start 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.269+0000 7fdaca0ae700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdac40fc830 0x7fdac410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.269+0000 7fdaca0ae700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdac40fd100 0x7fdac410d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.269+0000 7fdaca0ae700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac410d9b0 con 0x7fdac40fd100 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.269+0000 7fdaca0ae700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac410daf0 con 0x7fdac40fc830 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.269+0000 7fdac2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdac40fd100 0x7fdac410d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.269+0000 7fdac2ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdac40fd100 0x7fdac410d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34270/0 (socket says 172.21.15.113:34270) 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.269+0000 7fdac2ffd700 1 -- 172.21.15.113:0/2776096802 learned_addr learned my addr 172.21.15.113:0/2776096802 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.270+0000 7fdac2ffd700 1 -- 172.21.15.113:0/2776096802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdac40fc830 msgr2=0x7fdac410cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.270+0000 7fdac2ffd700 1 --2- 172.21.15.113:0/2776096802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdac40fc830 0x7fdac410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.270+0000 7fdac2ffd700 1 -- 172.21.15.113:0/2776096802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab4005040 con 0x7fdac40fd100 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.270+0000 7fdac2ffd700 1 --2- 172.21.15.113:0/2776096802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdac40fd100 0x7fdac410d3b0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7fdab4009920 tx=0x7fdab4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.270+0000 7fdac0ff9700 1 -- 172.21.15.113:0/2776096802 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdab400e030 con 0x7fdac40fd100 2024-04-12T05:51:29.267 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.270+0000 7fdaca0ae700 1 -- 172.21.15.113:0/2776096802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac4107120 con 0x7fdac40fd100 2024-04-12T05:51:29.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.270+0000 7fdac0ff9700 1 -- 172.21.15.113:0/2776096802 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdab4007e90 con 0x7fdac40fd100 2024-04-12T05:51:29.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.270+0000 7fdac0ff9700 1 -- 172.21.15.113:0/2776096802 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdab401a070 con 0x7fdac40fd100 2024-04-12T05:51:29.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.270+0000 7fdaca0ae700 1 -- 172.21.15.113:0/2776096802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac4107680 con 0x7fdac40fd100 2024-04-12T05:51:29.268 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.271+0000 7fdaca0ae700 1 -- 172.21.15.113:0/2776096802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac405f380 con 0x7fdac40fd100 2024-04-12T05:51:29.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.271+0000 7fdac0ff9700 1 -- 172.21.15.113:0/2776096802 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdab40127b0 con 0x7fdac40fd100 2024-04-12T05:51:29.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.272+0000 7fdac0ff9700 1 --2- 172.21.15.113:0/2776096802 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdaac06caa0 0x7fdaac06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:29.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.272+0000 7fdac0ff9700 1 -- 172.21.15.113:0/2776096802 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fdab408f940 con 0x7fdac40fd100 2024-04-12T05:51:29.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.272+0000 7fdac37fe700 1 --2- 172.21.15.113:0/2776096802 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdaac06caa0 0x7fdaac06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:29.269 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.272+0000 7fdac37fe700 1 --2- 172.21.15.113:0/2776096802 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdaac06caa0 0x7fdaac06ef60 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fdab8004950 tx=0x7fdab8007bc0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:29.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.276+0000 7fdac0ff9700 1 -- 172.21.15.113:0/2776096802 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fdab405d3f0 con 0x7fdac40fd100 2024-04-12T05:51:29.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:29 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/741273735' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-12T05:51:29.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:29 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2166269201' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-12T05:51:29.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:29 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2977542896' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-12T05:51:29.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.510+0000 7fdaca0ae700 1 -- 172.21.15.113:0/2776096802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7fdac40feb20 con 0x7fdac40fd100 2024-04-12T05:51:29.506 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.511+0000 7fdac0ff9700 1 -- 172.21.15.113:0/2776096802 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v35) v1 ==== 107+0+4848 (secure 0 0 0) 0x7fdab405cf80 con 0x7fdac40fd100 2024-04-12T05:51:29.506 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:29.507 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44331,"name":"cephfs.smithi192.ammjap","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6825/335558856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":335558856},{"type":"v1","addr":"172.21.15.192:6825","nonce":335558856}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44817,"name":"cephfs.smithi192.dtnaoi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6827/946017279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":946017279},{"type":"v1","addr":"172.21.15.192:6827","nonce":946017279}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T05:06:12.143448+0000","modified":"2024-04-12T05:49:49.603866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34412,"mds_1":34444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34412":{"gid":34412,"name":"cephfs.smithi113.ovmadh","rank":0,"incarnation":30,"state":"up:rejoin","state_seq":467,"addr":"172.21.15.113:6827/3491701432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":3491701432},{"type":"v1","addr":"172.21.15.113:6827","nonce":3491701432}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34444":{"gid":34444,"name":"cephfs.smithi113.ydsbei","rank":1,"incarnation":30,"state":"up:rejoin","state_seq":462,"addr":"172.21.15.113:6829/280747625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":280747625},{"type":"v1","addr":"172.21.15.113:6829","nonce":280747625}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-12T05:51:29.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.513+0000 7fdaca0ae700 1 -- 172.21.15.113:0/2776096802 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdaac06caa0 msgr2=0x7fdaac06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:29.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.513+0000 7fdaca0ae700 1 --2- 172.21.15.113:0/2776096802 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdaac06caa0 0x7fdaac06ef60 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fdab8004950 tx=0x7fdab8007bc0).stop 2024-04-12T05:51:29.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.513+0000 7fdaca0ae700 1 -- 172.21.15.113:0/2776096802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdac40fd100 msgr2=0x7fdac410d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:29.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.513+0000 7fdaca0ae700 1 --2- 172.21.15.113:0/2776096802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdac40fd100 0x7fdac410d3b0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7fdab4009920 tx=0x7fdab4009cb0).stop 2024-04-12T05:51:29.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.513+0000 7fdaca0ae700 1 -- 172.21.15.113:0/2776096802 shutdown_connections 2024-04-12T05:51:29.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.513+0000 7fdaca0ae700 1 --2- 172.21.15.113:0/2776096802 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdaac06caa0 0x7fdaac06ef60 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.513+0000 7fdaca0ae700 1 --2- 172.21.15.113:0/2776096802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdac40fc830 0x7fdac410cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:29.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.513+0000 7fdaca0ae700 1 --2- 172.21.15.113:0/2776096802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdac40fd100 0x7fdac410d3b0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.513+0000 7fdaca0ae700 1 -- 172.21.15.113:0/2776096802 >> 172.21.15.113:0/2776096802 conn(0x7fdac406c760 msgr2=0x7fdac40fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:29.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.513+0000 7fdaca0ae700 1 -- 172.21.15.113:0/2776096802 shutdown_connections 2024-04-12T05:51:29.509 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.513+0000 7fdaca0ae700 1 -- 172.21.15.113:0/2776096802 wait complete. 2024-04-12T05:51:29.510 INFO:teuthology.orchestra.run.smithi113.stderr:dumped fsmap epoch 34 2024-04-12T05:51:29.521 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-12T05:51:29.532 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-12T05:51:29.532 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:51:29.532 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-12T05:51:29.552 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:51:29.552 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-12T05:51:29.611 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T05:51:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.772+0000 7f88fdd5d700 1 -- 172.21.15.113:0/2866688744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f80f9220 msgr2=0x7f88f81019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.772+0000 7f88fdd5d700 1 --2- 172.21.15.113:0/2866688744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f80f9220 0x7f88f81019c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7f88ec0047f0 tx=0x7f88ec009cb0).stop 2024-04-12T05:51:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.772+0000 7f88fdd5d700 1 -- 172.21.15.113:0/2866688744 shutdown_connections 2024-04-12T05:51:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.772+0000 7f88fdd5d700 1 --2- 172.21.15.113:0/2866688744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f80f9220 0x7f88f81019c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.772+0000 7f88fdd5d700 1 --2- 172.21.15.113:0/2866688744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f80f8950 0x7f88f80f8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.772+0000 7f88fdd5d700 1 -- 172.21.15.113:0/2866688744 >> 172.21.15.113:0/2866688744 conn(0x7f88f806c760 msgr2=0x7f88f806cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.772+0000 7f88fdd5d700 1 -- 172.21.15.113:0/2866688744 shutdown_connections 2024-04-12T05:51:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.772+0000 7f88fdd5d700 1 -- 172.21.15.113:0/2866688744 wait complete. 2024-04-12T05:51:29.770 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.773+0000 7f88fdd5d700 1 Processor -- start 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.773+0000 7f88fdd5d700 1 -- start start 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.773+0000 7f88fdd5d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f80f8950 0x7f88f810df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.773+0000 7f88fdd5d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f80f9220 0x7f88f810e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.773+0000 7f88fdd5d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f810ea40 con 0x7f88f80f9220 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.773+0000 7f88fdd5d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f810eb80 con 0x7f88f80f8950 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.773+0000 7f88f6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f80f9220 0x7f88f810e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.773+0000 7f88f6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f80f9220 0x7f88f810e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34280/0 (socket says 172.21.15.113:34280) 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.773+0000 7f88f6ffd700 1 -- 172.21.15.113:0/3289482138 learned_addr learned my addr 172.21.15.113:0/3289482138 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.774+0000 7f88f77fe700 1 --2- 172.21.15.113:0/3289482138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f80f8950 0x7f88f810df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.774+0000 7f88f6ffd700 1 -- 172.21.15.113:0/3289482138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f80f8950 msgr2=0x7f88f810df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.774+0000 7f88f6ffd700 1 --2- 172.21.15.113:0/3289482138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f80f8950 0x7f88f810df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.774+0000 7f88f6ffd700 1 -- 172.21.15.113:0/3289482138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec005040 con 0x7f88f80f9220 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.774+0000 7f88f6ffd700 1 --2- 172.21.15.113:0/3289482138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f80f9220 0x7f88f810e440 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f88ec005010 tx=0x7f88ec009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:29.771 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.774+0000 7f88f4ff9700 1 -- 172.21.15.113:0/3289482138 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88ec00e030 con 0x7f88f80f9220 2024-04-12T05:51:29.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.774+0000 7f88fdd5d700 1 -- 172.21.15.113:0/3289482138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f81081b0 con 0x7f88f80f9220 2024-04-12T05:51:29.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.774+0000 7f88f4ff9700 1 -- 172.21.15.113:0/3289482138 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88ec007e90 con 0x7f88f80f9220 2024-04-12T05:51:29.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.774+0000 7f88f4ff9700 1 -- 172.21.15.113:0/3289482138 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88ec018070 con 0x7f88f80f9220 2024-04-12T05:51:29.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.774+0000 7f88fdd5d700 1 -- 172.21.15.113:0/3289482138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f8108690 con 0x7f88f80f9220 2024-04-12T05:51:29.772 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.775+0000 7f88fdd5d700 1 -- 172.21.15.113:0/3289482138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f805f380 con 0x7f88f80f9220 2024-04-12T05:51:29.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.775+0000 7f88f4ff9700 1 -- 172.21.15.113:0/3289482138 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f88ec01d6e0 con 0x7f88f80f9220 2024-04-12T05:51:29.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.776+0000 7f88f4ff9700 1 --2- 172.21.15.113:0/3289482138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88e006ca30 0x7f88e006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:29.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.776+0000 7f88f4ff9700 1 -- 172.21.15.113:0/3289482138 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f88ec08d100 con 0x7f88f80f9220 2024-04-12T05:51:29.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.776+0000 7f88f77fe700 1 --2- 172.21.15.113:0/3289482138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88e006ca30 0x7f88e006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:29.773 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.776+0000 7f88f77fe700 1 --2- 172.21.15.113:0/3289482138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88e006ca30 0x7f88e006eef0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f88e8009170 tx=0x7f88e8009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:29.776 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.780+0000 7f88f4ff9700 1 -- 172.21.15.113:0/3289482138 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f88ec05a920 con 0x7f88f80f9220 2024-04-12T05:51:29.967 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.971+0000 7f88fdd5d700 1 -- 172.21.15.113:0/3289482138 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88f805eec0 con 0x7f88f80f9220 2024-04-12T05:51:29.968 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.972+0000 7f88f4ff9700 1 -- 172.21.15.113:0/3289482138 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16556 (secure 0 0 0) 0x7f88ec05a740 con 0x7f88f80f9220 2024-04-12T05:51:29.968 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:29.968 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":98,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","created":"2024-04-12T04:59:30.005034+0000","modified":"2024-04-12T05:34:33.348722+0000","last_up_change":"2024-04-12T05:18:43.266143+0000","last_in_change":"2024-04-12T05:04:31.248597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T05:03:31.839630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{},"mgr_devicehealth":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-12T05:06:11.000491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-12T05:06:11.375495+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fc942a17-378a-48a6-b1d3-4f5730550810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":92,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6803","nonce":2361630947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6805","nonce":2361630947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6809","nonce":2361630947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6807","nonce":2361630947}]},"public_addr":"172.21.15.113:6803/2361630947","cluster_addr":"172.21.15.113:6805/2361630947","heartbeat_back_addr":"172.21.15.113:6809/2361630947","heartbeat_front_addr":"172.21.15.113:6807/2361630947","state":["exists","up"]},{"osd":1,"uuid":"39fec899-371b-4d48-99a8-52f90f52a71f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":92,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6811","nonce":1423470151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6813","nonce":1423470151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6817","nonce":1423470151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6815","nonce":1423470151}]},"public_addr":"172.21.15.113:6811/1423470151","cluster_addr":"172.21.15.113:6813/1423470151","heartbeat_back_addr":"172.21.15.113:6817/1423470151","heartbeat_front_addr":"172.21.15.113:6815/1423470151","state":["exists","up"]},{"osd":2,"uuid":"b135f3a9-d2f0-4737-8a30-9f6dc3f2571b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":63,"up_thru":92,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6819","nonce":2501657120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6821","nonce":2501657120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6825","nonce":2501657120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6823","nonce":2501657120}]},"public_addr":"172.21.15.113:6819/2501657120","cluster_addr":"172.21.15.113:6821/2501657120","heartbeat_back_addr":"172.21.15.113:6825/2501657120","heartbeat_front_addr":"172.21.15.113:6823/2501657120","state":["exists","up"]},{"osd":3,"uuid":"49fd856c-c351-4e90-9088-ece0024cd2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":72,"up_thru":92,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6801","nonce":2734551320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6803","nonce":2734551320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6807","nonce":2734551320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6805","nonce":2734551320}]},"public_addr":"172.21.15.192:6801/2734551320","cluster_addr":"172.21.15.192:6803/2734551320","heartbeat_back_addr":"172.21.15.192:6807/2734551320","heartbeat_front_addr":"172.21.15.192:6805/2734551320","state":["exists","up"]},{"osd":4,"uuid":"80b72db8-682a-4a51-8d33-b0addda91fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":32,"last_clean_end":76,"up_from":80,"up_thru":92,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6809","nonce":2295139175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6811","nonce":2295139175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6815","nonce":2295139175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6813","nonce":2295139175}]},"public_addr":"172.21.15.192:6809/2295139175","cluster_addr":"172.21.15.192:6811/2295139175","heartbeat_back_addr":"172.21.15.192:6815/2295139175","heartbeat_front_addr":"172.21.15.192:6813/2295139175","state":["exists","up"]},{"osd":5,"uuid":"c8e1e98f-5edb-4fec-bdf6-bcda40562682","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":37,"last_clean_end":88,"up_from":92,"up_thru":92,"down_at":89,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6817","nonce":2093584908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6819","nonce":2093584908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6823","nonce":2093584908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6821","nonce":2093584908}]},"public_addr":"172.21.15.192:6817/2093584908","cluster_addr":"172.21.15.192:6819/2093584908","heartbeat_back_addr":"172.21.15.192:6823/2093584908","heartbeat_front_addr":"172.21.15.192:6821/2093584908","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:02:27.514514+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:01.358930+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:30.203057+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:54.928612+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:21.788000+0000","dead_epoch":76},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:48.830652+0000","dead_epoch":88}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.19","mappings":[{"from":0,"to":3}]},{"pgid":"2.1b","mappings":[{"from":0,"to":3}]},{"pgid":"3.4","mappings":[{"from":1,"to":5}]},{"pgid":"3.d","mappings":[{"from":4,"to":0}]},{"pgid":"3.10","mappings":[{"from":1,"to":3}]},{"pgid":"3.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.113:6827/1262609849":"2024-04-13T05:18:43.274072+0000","172.21.15.113:6801/3082109414":"2024-04-13T05:13:51.744570+0000","172.21.15.192:6824/616350866":"2024-04-13T05:18:43.274091+0000","172.21.15.113:6800/3082109414":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/321762285":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/2190156688":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/3618578044":"2024-04-13T05:13:51.744570+0000","172.21.15.192:6825/616350866":"2024-04-13T05:18:43.274091+0000","172.21.15.113:0/313346589":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/4036112881":"2024-04-13T05:08:26.214053+0000","172.21.15.113:6801/593576174":"2024-04-13T05:08:26.214053+0000","172.21.15.113:6800/593576174":"2024-04-13T05:08:26.214053+0000","172.21.15.113:0/1777682971":"2024-04-13T04:59:57.862720+0000","172.21.15.192:0/2210150409":"2024-04-13T05:10:55.934327+0000","172.21.15.113:6800/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.192:0/2589268524":"2024-04-13T05:10:55.934327+0000","172.21.15.192:6828/2257000201":"2024-04-13T05:10:55.934327+0000","172.21.15.113:0/516367458":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/781838919":"2024-04-13T05:08:26.214053+0000","172.21.15.113:6801/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.113:6801/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/3213888755":"2024-04-13T05:01:23.491618+0000","172.21.15.192:6829/2257000201":"2024-04-13T05:10:55.934327+0000","172.21.15.113:6826/1262609849":"2024-04-13T05:18:43.274072+0000","172.21.15.192:0/1673786821":"2024-04-13T05:10:55.934327+0000","172.21.15.113:0/2902584436":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/4242055626":"2024-04-13T05:00:29.828900+0000","172.21.15.113:6800/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.113:6801/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.192:0/2053192105":"2024-04-13T05:10:55.934327+0000","172.21.15.113:6800/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/3450096338":"2024-04-13T04:59:57.862720+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T05:51:29.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.974+0000 7f88fdd5d700 1 -- 172.21.15.113:0/3289482138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88e006ca30 msgr2=0x7f88e006eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:29.970 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.974+0000 7f88fdd5d700 1 --2- 172.21.15.113:0/3289482138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88e006ca30 0x7f88e006eef0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f88e8009170 tx=0x7f88e8009d60).stop 2024-04-12T05:51:29.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.974+0000 7f88fdd5d700 1 -- 172.21.15.113:0/3289482138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f80f9220 msgr2=0x7f88f810e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:29.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.974+0000 7f88fdd5d700 1 --2- 172.21.15.113:0/3289482138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f80f9220 0x7f88f810e440 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f88ec005010 tx=0x7f88ec009cb0).stop 2024-04-12T05:51:29.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.975+0000 7f88fdd5d700 1 -- 172.21.15.113:0/3289482138 shutdown_connections 2024-04-12T05:51:29.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.975+0000 7f88fdd5d700 1 --2- 172.21.15.113:0/3289482138 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f88e006ca30 0x7f88e006eef0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.975+0000 7f88fdd5d700 1 --2- 172.21.15.113:0/3289482138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88f80f8950 0x7f88f810df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.975+0000 7f88fdd5d700 1 --2- 172.21.15.113:0/3289482138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88f80f9220 0x7f88f810e440 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:29.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.975+0000 7f88fdd5d700 1 -- 172.21.15.113:0/3289482138 >> 172.21.15.113:0/3289482138 conn(0x7f88f806c760 msgr2=0x7f88f80fc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:29.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.975+0000 7f88fdd5d700 1 -- 172.21.15.113:0/3289482138 shutdown_connections 2024-04-12T05:51:29.971 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:29.975+0000 7f88fdd5d700 1 -- 172.21.15.113:0/3289482138 wait complete. 2024-04-12T05:51:29.982 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T05:03:31.839630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}, 'mgr_devicehealth': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-12T05:06:11.000491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-12T05:06:11.375495+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-12T05:51:29.983 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T05:51:30.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.107+0000 7fdcd6586700 1 -- 172.21.15.113:0/2576854933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcd00f89f0 msgr2=0x7fdcd00f8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.107+0000 7fdcd6586700 1 --2- 172.21.15.113:0/2576854933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcd00f89f0 0x7fdcd00f8dd0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7fdcc40047f0 tx=0x7fdcc4009cb0).stop 2024-04-12T05:51:30.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.107+0000 7fdcd6586700 1 -- 172.21.15.113:0/2576854933 shutdown_connections 2024-04-12T05:51:30.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.107+0000 7fdcd6586700 1 --2- 172.21.15.113:0/2576854933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd00f92c0 0x7fdcd0101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.107+0000 7fdcd6586700 1 --2- 172.21.15.113:0/2576854933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcd00f89f0 0x7fdcd00f8dd0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.107+0000 7fdcd6586700 1 -- 172.21.15.113:0/2576854933 >> 172.21.15.113:0/2576854933 conn(0x7fdcd006c760 msgr2=0x7fdcd006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:30.104 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.107+0000 7fdcd6586700 1 -- 172.21.15.113:0/2576854933 shutdown_connections 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.107+0000 7fdcd6586700 1 -- 172.21.15.113:0/2576854933 wait complete. 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.107+0000 7fdcd6586700 1 Processor -- start 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.107+0000 7fdcd6586700 1 -- start start 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdcd6586700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd00f89f0 0x7fdcd00f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdcd6586700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcd00f92c0 0x7fdcd00f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdcd6586700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd00f7380 con 0x7fdcd00f92c0 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdcd6586700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd00f74c0 con 0x7fdcd00f89f0 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdccf7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcd00f92c0 0x7fdcd00f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdccf7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcd00f92c0 0x7fdcd00f6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34304/0 (socket says 172.21.15.113:34304) 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdccf7fe700 1 -- 172.21.15.113:0/3457638739 learned_addr learned my addr 172.21.15.113:0/3457638739 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdccffff700 1 --2- 172.21.15.113:0/3457638739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd00f89f0 0x7fdcd00f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdccf7fe700 1 -- 172.21.15.113:0/3457638739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd00f89f0 msgr2=0x7fdcd00f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdccf7fe700 1 --2- 172.21.15.113:0/3457638739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd00f89f0 0x7fdcd00f69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.105 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdccf7fe700 1 -- 172.21.15.113:0/3457638739 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc4005040 con 0x7fdcd00f92c0 2024-04-12T05:51:30.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdccf7fe700 1 --2- 172.21.15.113:0/3457638739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcd00f92c0 0x7fdcd00f6e90 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7fdcc00041d0 tx=0x7fdcc0009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:30.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdccd7fa700 1 -- 172.21.15.113:0/3457638739 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcc0007b40 con 0x7fdcd00f92c0 2024-04-12T05:51:30.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdccd7fa700 1 -- 172.21.15.113:0/3457638739 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcc000f040 con 0x7fdcd00f92c0 2024-04-12T05:51:30.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdcd6586700 1 -- 172.21.15.113:0/3457638739 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd00f7770 con 0x7fdcd00f92c0 2024-04-12T05:51:30.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdccd7fa700 1 -- 172.21.15.113:0/3457638739 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcc00147b0 con 0x7fdcd00f92c0 2024-04-12T05:51:30.106 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.108+0000 7fdcd6586700 1 -- 172.21.15.113:0/3457638739 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd0114630 con 0x7fdcd00f92c0 2024-04-12T05:51:30.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.109+0000 7fdcd6586700 1 -- 172.21.15.113:0/3457638739 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd005f380 con 0x7fdcd00f92c0 2024-04-12T05:51:30.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.110+0000 7fdccd7fa700 1 -- 172.21.15.113:0/3457638739 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdcc0009110 con 0x7fdcd00f92c0 2024-04-12T05:51:30.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.110+0000 7fdccd7fa700 1 --2- 172.21.15.113:0/3457638739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcb806caa0 0x7fdcb806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:30.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.110+0000 7fdccffff700 1 --2- 172.21.15.113:0/3457638739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcb806caa0 0x7fdcb806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:30.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.110+0000 7fdccd7fa700 1 -- 172.21.15.113:0/3457638739 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fdcc008be50 con 0x7fdcd00f92c0 2024-04-12T05:51:30.107 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.111+0000 7fdccffff700 1 --2- 172.21.15.113:0/3457638739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcb806caa0 0x7fdcb806ef60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7fdcc4009ee0 tx=0x7fdcc400b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:30.110 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.115+0000 7fdccd7fa700 1 -- 172.21.15.113:0/3457638739 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fdcc0055e70 con 0x7fdcd00f92c0 2024-04-12T05:51:30.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.300+0000 7fdcd6586700 1 -- 172.21.15.113:0/3457638739 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdcd00f7f20 con 0x7fdcd00f92c0 2024-04-12T05:51:30.296 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.301+0000 7fdccd7fa700 1 -- 172.21.15.113:0/3457638739 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdcc0059490 con 0x7fdcd00f92c0 2024-04-12T05:51:30.296 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-04-12T05:51:30.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.303+0000 7fdcd6586700 1 -- 172.21.15.113:0/3457638739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcb806caa0 msgr2=0x7fdcb806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.303+0000 7fdcd6586700 1 --2- 172.21.15.113:0/3457638739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcb806caa0 0x7fdcb806ef60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7fdcc4009ee0 tx=0x7fdcc400b040).stop 2024-04-12T05:51:30.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.303+0000 7fdcd6586700 1 -- 172.21.15.113:0/3457638739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcd00f92c0 msgr2=0x7fdcd00f6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.303+0000 7fdcd6586700 1 --2- 172.21.15.113:0/3457638739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcd00f92c0 0x7fdcd00f6e90 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7fdcc00041d0 tx=0x7fdcc0009f60).stop 2024-04-12T05:51:30.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.303+0000 7fdcd6586700 1 -- 172.21.15.113:0/3457638739 shutdown_connections 2024-04-12T05:51:30.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.303+0000 7fdcd6586700 1 --2- 172.21.15.113:0/3457638739 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fdcb806caa0 0x7fdcb806ef60 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.303+0000 7fdcd6586700 1 --2- 172.21.15.113:0/3457638739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd00f89f0 0x7fdcd00f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.303+0000 7fdcd6586700 1 --2- 172.21.15.113:0/3457638739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcd00f92c0 0x7fdcd00f6e90 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.303+0000 7fdcd6586700 1 -- 172.21.15.113:0/3457638739 >> 172.21.15.113:0/3457638739 conn(0x7fdcd006c760 msgr2=0x7fdcd00fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:30.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.303+0000 7fdcd6586700 1 -- 172.21.15.113:0/3457638739 shutdown_connections 2024-04-12T05:51:30.299 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.303+0000 7fdcd6586700 1 -- 172.21.15.113:0/3457638739 wait complete. 2024-04-12T05:51:30.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:30 smithi113 ceph-mon[103371]: pgmap v1182: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:30.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:30 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3660152062' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-12T05:51:30.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:30 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2776096802' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-12T05:51:30.309 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:30 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3289482138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:51:30.310 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-12T05:51:30.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.434+0000 7fc951c67700 1 -- 172.21.15.113:0/429047558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc94c0f70b0 msgr2=0x7fc94c103e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.434+0000 7fc951c67700 1 --2- 172.21.15.113:0/429047558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc94c0f70b0 0x7fc94c103e50 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7fc9400047f0 tx=0x7fc940009cb0).stop 2024-04-12T05:51:30.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.434+0000 7fc951c67700 1 -- 172.21.15.113:0/429047558 shutdown_connections 2024-04-12T05:51:30.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.434+0000 7fc951c67700 1 --2- 172.21.15.113:0/429047558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc94c0f70b0 0x7fc94c103e50 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.434+0000 7fc951c67700 1 --2- 172.21.15.113:0/429047558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc94c0f67e0 0x7fc94c0f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.434+0000 7fc951c67700 1 -- 172.21.15.113:0/429047558 >> 172.21.15.113:0/429047558 conn(0x7fc94c06c760 msgr2=0x7fc94c06cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:30.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.434+0000 7fc951c67700 1 -- 172.21.15.113:0/429047558 shutdown_connections 2024-04-12T05:51:30.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.434+0000 7fc951c67700 1 -- 172.21.15.113:0/429047558 wait complete. 2024-04-12T05:51:30.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.435+0000 7fc951c67700 1 Processor -- start 2024-04-12T05:51:30.432 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.435+0000 7fc951c67700 1 -- start start 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.435+0000 7fc951c67700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc94c0f67e0 0x7fc94c110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.435+0000 7fc951c67700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc94c0f70b0 0x7fc94c110670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.435+0000 7fc951c67700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c110c70 con 0x7fc94c0f67e0 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.435+0000 7fc951c67700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c110db0 con 0x7fc94c0f70b0 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.435+0000 7fc94b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc94c0f67e0 0x7fc94c110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.435+0000 7fc94b7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc94c0f67e0 0x7fc94c110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51270/0 (socket says 172.21.15.113:51270) 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.435+0000 7fc94b7fe700 1 -- 172.21.15.113:0/1120032118 learned_addr learned my addr 172.21.15.113:0/1120032118 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.435+0000 7fc94affd700 1 --2- 172.21.15.113:0/1120032118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc94c0f70b0 0x7fc94c110670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.436+0000 7fc94b7fe700 1 -- 172.21.15.113:0/1120032118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc94c0f70b0 msgr2=0x7fc94c110670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.436+0000 7fc94b7fe700 1 --2- 172.21.15.113:0/1120032118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc94c0f70b0 0x7fc94c110670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.436+0000 7fc94b7fe700 1 -- 172.21.15.113:0/1120032118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc940005040 con 0x7fc94c0f67e0 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.436+0000 7fc94b7fe700 1 --2- 172.21.15.113:0/1120032118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc94c0f67e0 0x7fc94c110180 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7fc93c0041d0 tx=0x7fc93c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.436+0000 7fc948ff9700 1 -- 172.21.15.113:0/1120032118 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc93c002d90 con 0x7fc94c0f67e0 2024-04-12T05:51:30.433 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.436+0000 7fc951c67700 1 -- 172.21.15.113:0/1120032118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94c10a440 con 0x7fc94c0f67e0 2024-04-12T05:51:30.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.436+0000 7fc948ff9700 1 -- 172.21.15.113:0/1120032118 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc93c00f040 con 0x7fc94c0f67e0 2024-04-12T05:51:30.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.436+0000 7fc948ff9700 1 -- 172.21.15.113:0/1120032118 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc93c00d800 con 0x7fc94c0f67e0 2024-04-12T05:51:30.434 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.436+0000 7fc951c67700 1 -- 172.21.15.113:0/1120032118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94c10aa00 con 0x7fc94c0f67e0 2024-04-12T05:51:30.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.437+0000 7fc951c67700 1 -- 172.21.15.113:0/1120032118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94c05f380 con 0x7fc94c0f67e0 2024-04-12T05:51:30.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.437+0000 7fc948ff9700 1 -- 172.21.15.113:0/1120032118 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc93c00dae0 con 0x7fc94c0f67e0 2024-04-12T05:51:30.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.438+0000 7fc948ff9700 1 --2- 172.21.15.113:0/1120032118 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc93406cab0 0x7fc93406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:30.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.438+0000 7fc948ff9700 1 -- 172.21.15.113:0/1120032118 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7fc93c08be80 con 0x7fc94c0f67e0 2024-04-12T05:51:30.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.438+0000 7fc94affd700 1 --2- 172.21.15.113:0/1120032118 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc93406cab0 0x7fc93406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:30.435 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.438+0000 7fc94affd700 1 --2- 172.21.15.113:0/1120032118 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc93406cab0 0x7fc93406ef70 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7fc94c10bac0 tx=0x7fc940009cb0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:30.437 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.442+0000 7fc948ff9700 1 -- 172.21.15.113:0/1120032118 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7fc93c059930 con 0x7fc94c0f67e0 2024-04-12T05:51:30.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:30 smithi192 ceph-mon[85304]: pgmap v1182: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:30.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:30 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3660152062' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-12T05:51:30.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:30 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2776096802' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-12T05:51:30.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:30 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3289482138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:51:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.628+0000 7fc951c67700 1 -- 172.21.15.113:0/1120032118 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fc94c05eec0 con 0x7fc94c0f67e0 2024-04-12T05:51:30.624 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.629+0000 7fc948ff9700 1 -- 172.21.15.113:0/1120032118 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v98) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc94c05eec0 con 0x7fc94c0f67e0 2024-04-12T05:51:30.624 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-04-12T05:51:30.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.631+0000 7fc951c67700 1 -- 172.21.15.113:0/1120032118 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc93406cab0 msgr2=0x7fc93406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.631+0000 7fc951c67700 1 --2- 172.21.15.113:0/1120032118 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc93406cab0 0x7fc93406ef70 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7fc94c10bac0 tx=0x7fc940009cb0).stop 2024-04-12T05:51:30.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.631+0000 7fc951c67700 1 -- 172.21.15.113:0/1120032118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc94c0f67e0 msgr2=0x7fc94c110180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.626 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.631+0000 7fc951c67700 1 --2- 172.21.15.113:0/1120032118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc94c0f67e0 0x7fc94c110180 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7fc93c0041d0 tx=0x7fc93c009f60).stop 2024-04-12T05:51:30.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.631+0000 7fc951c67700 1 -- 172.21.15.113:0/1120032118 shutdown_connections 2024-04-12T05:51:30.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.631+0000 7fc951c67700 1 --2- 172.21.15.113:0/1120032118 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7fc93406cab0 0x7fc93406ef70 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.631+0000 7fc951c67700 1 --2- 172.21.15.113:0/1120032118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc94c0f67e0 0x7fc94c110180 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.631+0000 7fc951c67700 1 --2- 172.21.15.113:0/1120032118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc94c0f70b0 0x7fc94c110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.631+0000 7fc951c67700 1 -- 172.21.15.113:0/1120032118 >> 172.21.15.113:0/1120032118 conn(0x7fc94c06c760 msgr2=0x7fc94c0f57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:30.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.631+0000 7fc951c67700 1 -- 172.21.15.113:0/1120032118 shutdown_connections 2024-04-12T05:51:30.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.631+0000 7fc951c67700 1 -- 172.21.15.113:0/1120032118 wait complete. 2024-04-12T05:51:30.638 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-12T05:51:30.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.761+0000 7f0b9d413700 1 -- 172.21.15.113:0/802469347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b980fd360 msgr2=0x7f0b980f6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.761+0000 7f0b9d413700 1 --2- 172.21.15.113:0/802469347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b980fd360 0x7f0b980f6b20 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0x7f0b880047f0 tx=0x7f0b88009cb0).stop 2024-04-12T05:51:30.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.761+0000 7f0b9d413700 1 -- 172.21.15.113:0/802469347 shutdown_connections 2024-04-12T05:51:30.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.761+0000 7f0b9d413700 1 --2- 172.21.15.113:0/802469347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b980fd360 0x7f0b980f6b20 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.761+0000 7f0b9d413700 1 --2- 172.21.15.113:0/802469347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b980fca00 0x7f0b980fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.761+0000 7f0b9d413700 1 -- 172.21.15.113:0/802469347 >> 172.21.15.113:0/802469347 conn(0x7f0b980f25a0 msgr2=0x7f0b980f49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:30.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.761+0000 7f0b9d413700 1 -- 172.21.15.113:0/802469347 shutdown_connections 2024-04-12T05:51:30.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.761+0000 7f0b9d413700 1 -- 172.21.15.113:0/802469347 wait complete. 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.762+0000 7f0b9d413700 1 Processor -- start 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.762+0000 7f0b9d413700 1 -- start start 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.762+0000 7f0b9d413700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b980fca00 0x7f0b9810e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.762+0000 7f0b9d413700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b980fd360 0x7f0b9810e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.762+0000 7f0b9d413700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9810ede0 con 0x7f0b980fd360 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.762+0000 7f0b9d413700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9810ef20 con 0x7f0b980fca00 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.762+0000 7f0b967fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b980fd360 0x7f0b9810e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b967fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b980fd360 0x7f0b9810e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51282/0 (socket says 172.21.15.113:51282) 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b967fc700 1 -- 172.21.15.113:0/2581996194 learned_addr learned my addr 172.21.15.113:0/2581996194 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b96ffd700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b980fca00 0x7f0b9810e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b967fc700 1 -- 172.21.15.113:0/2581996194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b980fca00 msgr2=0x7f0b9810e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b967fc700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b980fca00 0x7f0b9810e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b967fc700 1 -- 172.21.15.113:0/2581996194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b88005040 con 0x7f0b980fd360 2024-04-12T05:51:30.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b96ffd700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b980fca00 0x7f0b9810e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T05:51:30.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b967fc700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b980fd360 0x7f0b9810e7e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7f0b88000c00 tx=0x7f0b88009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:30.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b7ffff700 1 -- 172.21.15.113:0/2581996194 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b88018070 con 0x7f0b980fd360 2024-04-12T05:51:30.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b9d413700 1 -- 172.21.15.113:0/2581996194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b98108550 con 0x7f0b980fd360 2024-04-12T05:51:30.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b7ffff700 1 -- 172.21.15.113:0/2581996194 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b88007e90 con 0x7f0b980fd360 2024-04-12T05:51:30.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b7ffff700 1 -- 172.21.15.113:0/2581996194 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b880123c0 con 0x7f0b980fd360 2024-04-12T05:51:30.761 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.763+0000 7f0b9d413700 1 -- 172.21.15.113:0/2581996194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b98108ab0 con 0x7f0b980fd360 2024-04-12T05:51:30.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.764+0000 7f0b9d413700 1 -- 172.21.15.113:0/2581996194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b980603d0 con 0x7f0b980fd360 2024-04-12T05:51:30.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.764+0000 7f0b7ffff700 1 -- 172.21.15.113:0/2581996194 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0b8801d420 con 0x7f0b980fd360 2024-04-12T05:51:30.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.765+0000 7f0b7ffff700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8006caa0 0x7f0b8006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:30.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.765+0000 7f0b7ffff700 1 -- 172.21.15.113:0/2581996194 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f0b8808cbc0 con 0x7f0b980fd360 2024-04-12T05:51:30.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.765+0000 7f0b96ffd700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8006caa0 0x7f0b8006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:30.762 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.765+0000 7f0b96ffd700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8006caa0 0x7f0b8006ef60 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f0b8c007fd0 tx=0x7f0b8c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:30.765 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.769+0000 7f0b7ffff700 1 -- 172.21.15.113:0/2581996194 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f0b8805a670 con 0x7f0b980fd360 2024-04-12T05:51:30.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.957+0000 7f0b9d413700 1 -- 172.21.15.113:0/2581996194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f0b9805fb30 con 0x7f0b980fd360 2024-04-12T05:51:30.953 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.958+0000 7f0b7ffff700 1 -- 172.21.15.113:0/2581996194 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v98) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0b8805a200 con 0x7f0b980fd360 2024-04-12T05:51:30.953 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-04-12T05:51:30.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.960+0000 7f0b9d413700 1 -- 172.21.15.113:0/2581996194 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8006caa0 msgr2=0x7f0b8006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.960+0000 7f0b9d413700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8006caa0 0x7f0b8006ef60 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f0b8c007fd0 tx=0x7f0b8c00b040).stop 2024-04-12T05:51:30.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.960+0000 7f0b9d413700 1 -- 172.21.15.113:0/2581996194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b980fd360 msgr2=0x7f0b9810e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:30.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.960+0000 7f0b9d413700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b980fd360 0x7f0b9810e7e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7f0b88000c00 tx=0x7f0b88009cb0).stop 2024-04-12T05:51:30.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.960+0000 7f0b9d413700 1 -- 172.21.15.113:0/2581996194 shutdown_connections 2024-04-12T05:51:30.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.960+0000 7f0b9d413700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f0b8006caa0 0x7f0b8006ef60 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.960+0000 7f0b9d413700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b980fca00 0x7f0b9810e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.960+0000 7f0b9d413700 1 --2- 172.21.15.113:0/2581996194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b980fd360 0x7f0b9810e7e0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:30.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.960+0000 7f0b9d413700 1 -- 172.21.15.113:0/2581996194 >> 172.21.15.113:0/2581996194 conn(0x7f0b980f25a0 msgr2=0x7f0b980613a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:30.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.960+0000 7f0b9d413700 1 -- 172.21.15.113:0/2581996194 shutdown_connections 2024-04-12T05:51:30.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:30.960+0000 7f0b9d413700 1 -- 172.21.15.113:0/2581996194 wait complete. 2024-04-12T05:51:30.968 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-12T05:51:31.088 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.091+0000 7f9a9c420700 1 -- 172.21.15.113:0/1872424563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a940fe1f0 msgr2=0x7f9a940f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.091+0000 7f9a9c420700 1 --2- 172.21.15.113:0/1872424563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a940fe1f0 0x7f9a940f5d70 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7f9a900047f0 tx=0x7f9a90009cb0).stop 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.091+0000 7f9a9c420700 1 -- 172.21.15.113:0/1872424563 shutdown_connections 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.091+0000 7f9a9c420700 1 --2- 172.21.15.113:0/1872424563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a940fe1f0 0x7f9a940f5d70 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.091+0000 7f9a9c420700 1 --2- 172.21.15.113:0/1872424563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a940fd890 0x7f9a940fdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.091+0000 7f9a9c420700 1 -- 172.21.15.113:0/1872424563 >> 172.21.15.113:0/1872424563 conn(0x7f9a9406cb60 msgr2=0x7f9a940f3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.091+0000 7f9a9c420700 1 -- 172.21.15.113:0/1872424563 shutdown_connections 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.091+0000 7f9a9c420700 1 -- 172.21.15.113:0/1872424563 wait complete. 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9c420700 1 Processor -- start 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9c420700 1 -- start start 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9c420700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a940fd890 0x7f9a941101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9c420700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a940fe1f0 0x7f9a941106c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:31.089 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9c420700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a94110d50 con 0x7f9a940fd890 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9c420700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9410a370 con 0x7f9a940fe1f0 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9a1bc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a940fd890 0x7f9a941101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9a1bc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a940fd890 0x7f9a941101d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51304/0 (socket says 172.21.15.113:51304) 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9a1bc700 1 -- 172.21.15.113:0/4031749076 learned_addr learned my addr 172.21.15.113:0/4031749076 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a999bb700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a940fe1f0 0x7f9a941106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9a1bc700 1 -- 172.21.15.113:0/4031749076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a940fe1f0 msgr2=0x7f9a941106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9a1bc700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a940fe1f0 0x7f9a941106c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9a1bc700 1 -- 172.21.15.113:0/4031749076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a90005040 con 0x7f9a940fd890 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a999bb700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a940fe1f0 0x7f9a941106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.092+0000 7f9a9a1bc700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a940fd890 0x7f9a941101d0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7f9a8400b4b0 tx=0x7f9a840090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.093+0000 7f9a8b7fe700 1 -- 172.21.15.113:0/4031749076 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a84002500 con 0x7f9a940fd890 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.093+0000 7f9a9c420700 1 -- 172.21.15.113:0/4031749076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9410a620 con 0x7f9a940fd890 2024-04-12T05:51:31.090 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.093+0000 7f9a8b7fe700 1 -- 172.21.15.113:0/4031749076 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a84002b10 con 0x7f9a940fd890 2024-04-12T05:51:31.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.093+0000 7f9a8b7fe700 1 -- 172.21.15.113:0/4031749076 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a84012cf0 con 0x7f9a940fd890 2024-04-12T05:51:31.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.093+0000 7f9a9c420700 1 -- 172.21.15.113:0/4031749076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9410ab60 con 0x7f9a940fd890 2024-04-12T05:51:31.091 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.094+0000 7f9a9c420700 1 -- 172.21.15.113:0/4031749076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a940606f0 con 0x7f9a940fd890 2024-04-12T05:51:31.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.094+0000 7f9a8b7fe700 1 -- 172.21.15.113:0/4031749076 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9a84002660 con 0x7f9a940fd890 2024-04-12T05:51:31.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.095+0000 7f9a8b7fe700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a8006caa0 0x7f9a8006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:31.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.095+0000 7f9a8b7fe700 1 -- 172.21.15.113:0/4031749076 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9a8408bf30 con 0x7f9a940fd890 2024-04-12T05:51:31.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.095+0000 7f9a999bb700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a8006caa0 0x7f9a8006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:31.092 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.095+0000 7f9a999bb700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a8006caa0 0x7f9a8006ef60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f9a900043d0 tx=0x7f9a9000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:31.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.099+0000 7f9a8b7fe700 1 -- 172.21.15.113:0/4031749076 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9a840bf2c0 con 0x7f9a940fd890 2024-04-12T05:51:31.297 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.302+0000 7f9a9c420700 1 -- 172.21.15.113:0/4031749076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9a9410b850 con 0x7f9a940fd890 2024-04-12T05:51:31.298 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.302+0000 7f9a8b7fe700 1 -- 172.21.15.113:0/4031749076 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 32 entries v98) v1 ==== 81+0+1928 (secure 0 0 0) 0x7f9a840599e0 con 0x7f9a940fd890 2024-04-12T05:51:31.298 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6827/1262609849 2024-04-13T05:18:43.274072+0000 2024-04-12T05:51:31.298 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6801/3082109414 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:31.298 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:6824/616350866 2024-04-13T05:18:43.274091+0000 2024-04-12T05:51:31.298 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6800/3082109414 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/321762285 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/2190156688 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/3618578044 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:6825/616350866 2024-04-13T05:18:43.274091+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/313346589 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/4036112881 2024-04-13T05:08:26.214053+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6801/593576174 2024-04-13T05:08:26.214053+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6800/593576174 2024-04-13T05:08:26.214053+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/1777682971 2024-04-13T04:59:57.862720+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:0/2210150409 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6800/1349703403 2024-04-13T05:01:23.491618+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:0/2589268524 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:31.299 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:6828/2257000201 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/516367458 2024-04-13T05:01:23.491618+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/781838919 2024-04-13T05:08:26.214053+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6801/1349703403 2024-04-13T05:01:23.491618+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6801/2567243935 2024-04-13T05:00:29.828900+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/3213888755 2024-04-13T05:01:23.491618+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:6829/2257000201 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6826/1262609849 2024-04-13T05:18:43.274072+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:0/1673786821 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/2902584436 2024-04-13T05:00:29.828900+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/4242055626 2024-04-13T05:00:29.828900+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6800/3275382936 2024-04-13T04:59:57.862720+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6801/3275382936 2024-04-13T04:59:57.862720+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:0/2053192105 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:31.300 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6800/2567243935 2024-04-13T05:00:29.828900+0000 2024-04-12T05:51:31.301 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/3450096338 2024-04-13T04:59:57.862720+0000 2024-04-12T05:51:31.301 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.304+0000 7f9a9c420700 1 -- 172.21.15.113:0/4031749076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a8006caa0 msgr2=0x7f9a8006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:31.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.304+0000 7f9a9c420700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a8006caa0 0x7f9a8006ef60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f9a900043d0 tx=0x7f9a9000b040).stop 2024-04-12T05:51:31.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.304+0000 7f9a9c420700 1 -- 172.21.15.113:0/4031749076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a940fd890 msgr2=0x7f9a941101d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:31.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.304+0000 7f9a9c420700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a940fd890 0x7f9a941101d0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7f9a8400b4b0 tx=0x7f9a840090f0).stop 2024-04-12T05:51:31.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.305+0000 7f9a9c420700 1 -- 172.21.15.113:0/4031749076 shutdown_connections 2024-04-12T05:51:31.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.305+0000 7f9a9c420700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9a8006caa0 0x7f9a8006ef60 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.305+0000 7f9a9c420700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a940fd890 0x7f9a941101d0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.305+0000 7f9a9c420700 1 --2- 172.21.15.113:0/4031749076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a940fe1f0 0x7f9a941106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.305+0000 7f9a9c420700 1 -- 172.21.15.113:0/4031749076 >> 172.21.15.113:0/4031749076 conn(0x7f9a9406cb60 msgr2=0x7f9a940feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:31.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.305+0000 7f9a9c420700 1 -- 172.21.15.113:0/4031749076 shutdown_connections 2024-04-12T05:51:31.302 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.305+0000 7f9a9c420700 1 -- 172.21.15.113:0/4031749076 wait complete. 2024-04-12T05:51:31.302 INFO:teuthology.orchestra.run.smithi113.stderr:listed 32 entries 2024-04-12T05:51:31.311 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:31 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3457638739' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T05:51:31.311 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:31 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/1120032118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T05:51:31.311 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:31 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2581996194' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T05:51:31.312 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-04-12T05:51:31.312 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-12T05:51:31.332 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T05:51:31.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:31 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3457638739' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T05:51:31.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:31 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/1120032118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T05:51:31.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:31 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2581996194' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T05:51:31.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.492+0000 7f2b31416700 1 -- 172.21.15.113:0/4164195967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b2c0fd090 msgr2=0x7f2b2c0f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:31.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.492+0000 7f2b31416700 1 --2- 172.21.15.113:0/4164195967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b2c0fd090 0x7f2b2c0f6870 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7f2b200047f0 tx=0x7f2b20009cb0).stop 2024-04-12T05:51:31.488 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.492+0000 7f2b31416700 1 -- 172.21.15.113:0/4164195967 shutdown_connections 2024-04-12T05:51:31.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.492+0000 7f2b31416700 1 --2- 172.21.15.113:0/4164195967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b2c0fd090 0x7f2b2c0f6870 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.492+0000 7f2b31416700 1 --2- 172.21.15.113:0/4164195967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b2c0fc730 0x7f2b2c0fcb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.492+0000 7f2b31416700 1 -- 172.21.15.113:0/4164195967 >> 172.21.15.113:0/4164195967 conn(0x7f2b2c06c760 msgr2=0x7f2b2c06cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:31.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.492+0000 7f2b31416700 1 -- 172.21.15.113:0/4164195967 shutdown_connections 2024-04-12T05:51:31.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.492+0000 7f2b31416700 1 -- 172.21.15.113:0/4164195967 wait complete. 2024-04-12T05:51:31.489 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.493+0000 7f2b31416700 1 Processor -- start 2024-04-12T05:51:31.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.493+0000 7f2b31416700 1 -- start start 2024-04-12T05:51:31.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b31416700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b2c0fc730 0x7f2b2c10ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:31.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b31416700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b2c0fd090 0x7f2b2c10d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:31.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b31416700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c10d9a0 con 0x7f2b2c0fd090 2024-04-12T05:51:31.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b31416700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c10dae0 con 0x7f2b2c0fc730 2024-04-12T05:51:31.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b2a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b2c0fd090 0x7f2b2c10d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:31.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b2a7fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b2c0fd090 0x7f2b2c10d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51322/0 (socket says 172.21.15.113:51322) 2024-04-12T05:51:31.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b2a7fc700 1 -- 172.21.15.113:0/2135056429 learned_addr learned my addr 172.21.15.113:0/2135056429 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:31.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b2affd700 1 --2- 172.21.15.113:0/2135056429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b2c0fc730 0x7f2b2c10ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:31.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b2a7fc700 1 -- 172.21.15.113:0/2135056429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b2c0fc730 msgr2=0x7f2b2c10ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:31.491 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b2a7fc700 1 --2- 172.21.15.113:0/2135056429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b2c0fc730 0x7f2b2c10ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b2a7fc700 1 -- 172.21.15.113:0/2135056429 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b20005040 con 0x7f2b2c0fd090 2024-04-12T05:51:31.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b2a7fc700 1 --2- 172.21.15.113:0/2135056429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b2c0fd090 0x7f2b2c10d3a0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7f2b200099f0 tx=0x7f2b20009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:31.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b13fff700 1 -- 172.21.15.113:0/2135056429 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b2000e030 con 0x7f2b2c0fd090 2024-04-12T05:51:31.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b31416700 1 -- 172.21.15.113:0/2135056429 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2c107110 con 0x7f2b2c0fd090 2024-04-12T05:51:31.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b13fff700 1 -- 172.21.15.113:0/2135056429 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b20007e90 con 0x7f2b2c0fd090 2024-04-12T05:51:31.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b13fff700 1 -- 172.21.15.113:0/2135056429 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b20018070 con 0x7f2b2c0fd090 2024-04-12T05:51:31.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.494+0000 7f2b31416700 1 -- 172.21.15.113:0/2135056429 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2c1075f0 con 0x7f2b2c0fd090 2024-04-12T05:51:31.492 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.495+0000 7f2b31416700 1 -- 172.21.15.113:0/2135056429 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2c05f380 con 0x7f2b2c0fd090 2024-04-12T05:51:31.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.496+0000 7f2b13fff700 1 -- 172.21.15.113:0/2135056429 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2b200094c0 con 0x7f2b2c0fd090 2024-04-12T05:51:31.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.496+0000 7f2b13fff700 1 --2- 172.21.15.113:0/2135056429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b1406caa0 0x7f2b1406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:31.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.496+0000 7f2b13fff700 1 -- 172.21.15.113:0/2135056429 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f2b2008d0c0 con 0x7f2b2c0fd090 2024-04-12T05:51:31.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.496+0000 7f2b2affd700 1 --2- 172.21.15.113:0/2135056429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b1406caa0 0x7f2b1406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:31.493 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.497+0000 7f2b2affd700 1 --2- 172.21.15.113:0/2135056429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b1406caa0 0x7f2b1406ef60 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f2b1c004de0 tx=0x7f2b1c00b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:31.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.501+0000 7f2b13fff700 1 -- 172.21.15.113:0/2135056429 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2b2005ab70 con 0x7f2b2c0fd090 2024-04-12T05:51:31.686 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.690+0000 7f2b31416700 1 -- 172.21.15.113:0/2135056429 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b2c0fea80 con 0x7f2b2c0fd090 2024-04-12T05:51:31.687 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.692+0000 7f2b13fff700 1 -- 172.21.15.113:0/2135056429 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16556 (secure 0 0 0) 0x7f2b2005a700 con 0x7f2b2c0fd090 2024-04-12T05:51:31.687 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-04-12T05:51:31.687 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":98,"fsid":"52321e2c-f889-11ee-bc8c-c7b262605968","created":"2024-04-12T04:59:30.005034+0000","modified":"2024-04-12T05:34:33.348722+0000","last_up_change":"2024-04-12T05:18:43.266143+0000","last_in_change":"2024-04-12T05:04:31.248597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T05:03:31.839630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{},"mgr_devicehealth":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-12T05:06:11.000491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-12T05:06:11.375495+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fc942a17-378a-48a6-b1d3-4f5730550810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":92,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6803","nonce":2361630947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6805","nonce":2361630947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6809","nonce":2361630947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2361630947},{"type":"v1","addr":"172.21.15.113:6807","nonce":2361630947}]},"public_addr":"172.21.15.113:6803/2361630947","cluster_addr":"172.21.15.113:6805/2361630947","heartbeat_back_addr":"172.21.15.113:6809/2361630947","heartbeat_front_addr":"172.21.15.113:6807/2361630947","state":["exists","up"]},{"osd":1,"uuid":"39fec899-371b-4d48-99a8-52f90f52a71f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":92,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6811","nonce":1423470151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6813","nonce":1423470151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6817","nonce":1423470151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":1423470151},{"type":"v1","addr":"172.21.15.113:6815","nonce":1423470151}]},"public_addr":"172.21.15.113:6811/1423470151","cluster_addr":"172.21.15.113:6813/1423470151","heartbeat_back_addr":"172.21.15.113:6817/1423470151","heartbeat_front_addr":"172.21.15.113:6815/1423470151","state":["exists","up"]},{"osd":2,"uuid":"b135f3a9-d2f0-4737-8a30-9f6dc3f2571b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":63,"up_thru":92,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6819","nonce":2501657120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6821","nonce":2501657120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6825","nonce":2501657120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2501657120},{"type":"v1","addr":"172.21.15.113:6823","nonce":2501657120}]},"public_addr":"172.21.15.113:6819/2501657120","cluster_addr":"172.21.15.113:6821/2501657120","heartbeat_back_addr":"172.21.15.113:6825/2501657120","heartbeat_front_addr":"172.21.15.113:6823/2501657120","state":["exists","up"]},{"osd":3,"uuid":"49fd856c-c351-4e90-9088-ece0024cd2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":72,"up_thru":92,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6801","nonce":2734551320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6803","nonce":2734551320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6807","nonce":2734551320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2734551320},{"type":"v1","addr":"172.21.15.192:6805","nonce":2734551320}]},"public_addr":"172.21.15.192:6801/2734551320","cluster_addr":"172.21.15.192:6803/2734551320","heartbeat_back_addr":"172.21.15.192:6807/2734551320","heartbeat_front_addr":"172.21.15.192:6805/2734551320","state":["exists","up"]},{"osd":4,"uuid":"80b72db8-682a-4a51-8d33-b0addda91fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":32,"last_clean_end":76,"up_from":80,"up_thru":92,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6809","nonce":2295139175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6811","nonce":2295139175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6815","nonce":2295139175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2295139175},{"type":"v1","addr":"172.21.15.192:6813","nonce":2295139175}]},"public_addr":"172.21.15.192:6809/2295139175","cluster_addr":"172.21.15.192:6811/2295139175","heartbeat_back_addr":"172.21.15.192:6815/2295139175","heartbeat_front_addr":"172.21.15.192:6813/2295139175","state":["exists","up"]},{"osd":5,"uuid":"c8e1e98f-5edb-4fec-bdf6-bcda40562682","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":37,"last_clean_end":88,"up_from":92,"up_thru":92,"down_at":89,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6817","nonce":2093584908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6819","nonce":2093584908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6823","nonce":2093584908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":2093584908},{"type":"v1","addr":"172.21.15.192:6821","nonce":2093584908}]},"public_addr":"172.21.15.192:6817/2093584908","cluster_addr":"172.21.15.192:6819/2093584908","heartbeat_back_addr":"172.21.15.192:6823/2093584908","heartbeat_front_addr":"172.21.15.192:6821/2093584908","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:02:27.514514+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:01.358930+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:30.203057+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:03:54.928612+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:21.788000+0000","dead_epoch":76},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T05:04:48.830652+0000","dead_epoch":88}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.19","mappings":[{"from":0,"to":3}]},{"pgid":"2.1b","mappings":[{"from":0,"to":3}]},{"pgid":"3.4","mappings":[{"from":1,"to":5}]},{"pgid":"3.d","mappings":[{"from":4,"to":0}]},{"pgid":"3.10","mappings":[{"from":1,"to":3}]},{"pgid":"3.1c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.113:6827/1262609849":"2024-04-13T05:18:43.274072+0000","172.21.15.113:6801/3082109414":"2024-04-13T05:13:51.744570+0000","172.21.15.192:6824/616350866":"2024-04-13T05:18:43.274091+0000","172.21.15.113:6800/3082109414":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/321762285":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/2190156688":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/3618578044":"2024-04-13T05:13:51.744570+0000","172.21.15.192:6825/616350866":"2024-04-13T05:18:43.274091+0000","172.21.15.113:0/313346589":"2024-04-13T05:13:51.744570+0000","172.21.15.113:0/4036112881":"2024-04-13T05:08:26.214053+0000","172.21.15.113:6801/593576174":"2024-04-13T05:08:26.214053+0000","172.21.15.113:6800/593576174":"2024-04-13T05:08:26.214053+0000","172.21.15.113:0/1777682971":"2024-04-13T04:59:57.862720+0000","172.21.15.192:0/2210150409":"2024-04-13T05:10:55.934327+0000","172.21.15.113:6800/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.192:0/2589268524":"2024-04-13T05:10:55.934327+0000","172.21.15.192:6828/2257000201":"2024-04-13T05:10:55.934327+0000","172.21.15.113:0/516367458":"2024-04-13T05:01:23.491618+0000","172.21.15.113:0/781838919":"2024-04-13T05:08:26.214053+0000","172.21.15.113:6801/1349703403":"2024-04-13T05:01:23.491618+0000","172.21.15.113:6801/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/3213888755":"2024-04-13T05:01:23.491618+0000","172.21.15.192:6829/2257000201":"2024-04-13T05:10:55.934327+0000","172.21.15.113:6826/1262609849":"2024-04-13T05:18:43.274072+0000","172.21.15.192:0/1673786821":"2024-04-13T05:10:55.934327+0000","172.21.15.113:0/2902584436":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/4242055626":"2024-04-13T05:00:29.828900+0000","172.21.15.113:6800/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.113:6801/3275382936":"2024-04-13T04:59:57.862720+0000","172.21.15.192:0/2053192105":"2024-04-13T05:10:55.934327+0000","172.21.15.113:6800/2567243935":"2024-04-13T05:00:29.828900+0000","172.21.15.113:0/3450096338":"2024-04-13T04:59:57.862720+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T05:51:31.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.694+0000 7f2b31416700 1 -- 172.21.15.113:0/2135056429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b1406caa0 msgr2=0x7f2b1406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:31.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.694+0000 7f2b31416700 1 --2- 172.21.15.113:0/2135056429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b1406caa0 0x7f2b1406ef60 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f2b1c004de0 tx=0x7f2b1c00b040).stop 2024-04-12T05:51:31.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.694+0000 7f2b31416700 1 -- 172.21.15.113:0/2135056429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b2c0fd090 msgr2=0x7f2b2c10d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:31.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.694+0000 7f2b31416700 1 --2- 172.21.15.113:0/2135056429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b2c0fd090 0x7f2b2c10d3a0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7f2b200099f0 tx=0x7f2b20009cb0).stop 2024-04-12T05:51:31.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.694+0000 7f2b31416700 1 -- 172.21.15.113:0/2135056429 shutdown_connections 2024-04-12T05:51:31.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.694+0000 7f2b31416700 1 --2- 172.21.15.113:0/2135056429 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f2b1406caa0 0x7f2b1406ef60 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.694+0000 7f2b31416700 1 --2- 172.21.15.113:0/2135056429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b2c0fc730 0x7f2b2c10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.694+0000 7f2b31416700 1 --2- 172.21.15.113:0/2135056429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b2c0fd090 0x7f2b2c10d3a0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.694+0000 7f2b31416700 1 -- 172.21.15.113:0/2135056429 >> 172.21.15.113:0/2135056429 conn(0x7f2b2c06c760 msgr2=0x7f2b2c0faaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:31.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.694+0000 7f2b31416700 1 -- 172.21.15.113:0/2135056429 shutdown_connections 2024-04-12T05:51:31.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.694+0000 7f2b31416700 1 -- 172.21.15.113:0/2135056429 wait complete. 2024-04-12T05:51:31.702 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T05:03:31.839630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}, 'mgr_devicehealth': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-12T05:06:11.000491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-12T05:06:11.375495+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-12T05:51:31.702 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.826+0000 7f2215f12700 1 -- 172.21.15.113:0/2831910562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22100fd900 msgr2=0x7f22100fdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.826+0000 7f2215f12700 1 --2- 172.21.15.113:0/2831910562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22100fd900 0x7f22100fdce0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f21f80047f0 tx=0x7f21f8009cb0).stop 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.826+0000 7f2215f12700 1 -- 172.21.15.113:0/2831910562 shutdown_connections 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.826+0000 7f2215f12700 1 --2- 172.21.15.113:0/2831910562 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22100fe260 0x7f22100f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.826+0000 7f2215f12700 1 --2- 172.21.15.113:0/2831910562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22100fd900 0x7f22100fdce0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.826+0000 7f2215f12700 1 -- 172.21.15.113:0/2831910562 >> 172.21.15.113:0/2831910562 conn(0x7f22100f34a0 msgr2=0x7f22100f58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.826+0000 7f2215f12700 1 -- 172.21.15.113:0/2831910562 shutdown_connections 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.826+0000 7f2215f12700 1 -- 172.21.15.113:0/2831910562 wait complete. 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.826+0000 7f2215f12700 1 Processor -- start 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f2215f12700 1 -- start start 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f2215f12700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22100fd900 0x7f221010e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:31.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f2215f12700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22100fe260 0x7f221010e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f2215f12700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221010eaf0 con 0x7f22100fe260 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f2215f12700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221010ec30 con 0x7f22100fd900 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22100fe260 0x7f221010e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22100fe260 0x7f221010e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51340/0 (socket says 172.21.15.113:51340) 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220effd700 1 -- 172.21.15.113:0/2397272457 learned_addr learned my addr 172.21.15.113:0/2397272457 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220f7fe700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22100fd900 0x7f221010e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220effd700 1 -- 172.21.15.113:0/2397272457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22100fd900 msgr2=0x7f221010e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220effd700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22100fd900 0x7f221010e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220effd700 1 -- 172.21.15.113:0/2397272457 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f8005040 con 0x7f22100fe260 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220f7fe700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22100fd900 0x7f221010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220effd700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22100fe260 0x7f221010e4f0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f220000c710 tx=0x7f2200007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220cff9700 1 -- 172.21.15.113:0/2397272457 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220000e680 con 0x7f22100fe260 2024-04-12T05:51:31.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220cff9700 1 -- 172.21.15.113:0/2397272457 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f220000ec90 con 0x7f22100fe260 2024-04-12T05:51:31.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f2215f12700 1 -- 172.21.15.113:0/2397272457 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22101082c0 con 0x7f22100fe260 2024-04-12T05:51:31.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f220cff9700 1 -- 172.21.15.113:0/2397272457 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2200013cd0 con 0x7f22100fe260 2024-04-12T05:51:31.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.827+0000 7f2215f12700 1 -- 172.21.15.113:0/2397272457 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2210108800 con 0x7f22100fe260 2024-04-12T05:51:31.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.828+0000 7f2215f12700 1 -- 172.21.15.113:0/2397272457 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2210060370 con 0x7f22100fe260 2024-04-12T05:51:31.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.829+0000 7f220cff9700 1 -- 172.21.15.113:0/2397272457 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2200013410 con 0x7f22100fe260 2024-04-12T05:51:31.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.829+0000 7f220cff9700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21fc06caa0 0x7f21fc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:31.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.829+0000 7f220f7fe700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21fc06caa0 0x7f21fc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:31.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.829+0000 7f220cff9700 1 -- 172.21.15.113:0/2397272457 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f220008c130 con 0x7f22100fe260 2024-04-12T05:51:31.827 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.830+0000 7f220f7fe700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21fc06caa0 0x7f21fc06ef60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f21f80043d0 tx=0x7f21f800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:31.829 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:31.834+0000 7f220cff9700 1 -- 172.21.15.113:0/2397272457 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f2200059be0 con 0x7f22100fe260 2024-04-12T05:51:32.019 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.023+0000 7f2215f12700 1 -- 172.21.15.113:0/2397272457 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2210109720 con 0x7f22100fe260 2024-04-12T05:51:32.022 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.026+0000 7f220cff9700 1 -- 172.21.15.113:0/2397272457 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2200059770 con 0x7f22100fe260 2024-04-12T05:51:32.022 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 1 2024-04-12T05:51:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.029+0000 7f2215f12700 1 -- 172.21.15.113:0/2397272457 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21fc06caa0 msgr2=0x7f21fc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.029+0000 7f2215f12700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21fc06caa0 0x7f21fc06ef60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f21f80043d0 tx=0x7f21f800b040).stop 2024-04-12T05:51:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.029+0000 7f2215f12700 1 -- 172.21.15.113:0/2397272457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22100fe260 msgr2=0x7f221010e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.029+0000 7f2215f12700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22100fe260 0x7f221010e4f0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f220000c710 tx=0x7f2200007ec0).stop 2024-04-12T05:51:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.029+0000 7f2215f12700 1 -- 172.21.15.113:0/2397272457 shutdown_connections 2024-04-12T05:51:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.029+0000 7f2215f12700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f21fc06caa0 0x7f21fc06ef60 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.029+0000 7f2215f12700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22100fd900 0x7f221010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.029+0000 7f2215f12700 1 --2- 172.21.15.113:0/2397272457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22100fe260 0x7f221010e4f0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.029+0000 7f2215f12700 1 -- 172.21.15.113:0/2397272457 >> 172.21.15.113:0/2397272457 conn(0x7f22100f34a0 msgr2=0x7f22100fbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.029+0000 7f2215f12700 1 -- 172.21.15.113:0/2397272457 shutdown_connections 2024-04-12T05:51:32.025 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.029+0000 7f2215f12700 1 -- 172.21.15.113:0/2397272457 wait complete. 2024-04-12T05:51:32.037 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-12T05:51:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.160+0000 7f6f92e08700 1 -- 172.21.15.113:0/2974063800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f8c0fc830 msgr2=0x7f6f8c0fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.160+0000 7f6f92e08700 1 --2- 172.21.15.113:0/2974063800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f8c0fc830 0x7f6f8c0fcc10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7f6f800047f0 tx=0x7f6f80009cb0).stop 2024-04-12T05:51:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.160+0000 7f6f92e08700 1 -- 172.21.15.113:0/2974063800 shutdown_connections 2024-04-12T05:51:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.160+0000 7f6f92e08700 1 --2- 172.21.15.113:0/2974063800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f8c0fd100 0x7f6f8c0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.160+0000 7f6f92e08700 1 --2- 172.21.15.113:0/2974063800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f8c0fc830 0x7f6f8c0fcc10 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.160+0000 7f6f92e08700 1 -- 172.21.15.113:0/2974063800 >> 172.21.15.113:0/2974063800 conn(0x7f6f8c06c760 msgr2=0x7f6f8c06cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:32.157 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.160+0000 7f6f92e08700 1 -- 172.21.15.113:0/2974063800 shutdown_connections 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.160+0000 7f6f92e08700 1 -- 172.21.15.113:0/2974063800 wait complete. 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.160+0000 7f6f92e08700 1 Processor -- start 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.160+0000 7f6f92e08700 1 -- start start 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f92e08700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f8c0fc830 0x7f6f8c1091a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f92e08700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f8c0fd100 0x7f6f8c109690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f92e08700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c109b80 con 0x7f6f8c0fd100 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f92e08700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c109cc0 con 0x7f6f8c0fc830 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f8bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f8c0fd100 0x7f6f8c109690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f8bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f8c0fd100 0x7f6f8c109690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51364/0 (socket says 172.21.15.113:51364) 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f8bfff700 1 -- 172.21.15.113:0/3255426978 learned_addr learned my addr 172.21.15.113:0/3255426978 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f90ba4700 1 --2- 172.21.15.113:0/3255426978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f8c0fc830 0x7f6f8c1091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:32.158 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f8bfff700 1 -- 172.21.15.113:0/3255426978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f8c0fc830 msgr2=0x7f6f8c1091a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f8bfff700 1 --2- 172.21.15.113:0/3255426978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f8c0fc830 0x7f6f8c1091a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f8bfff700 1 -- 172.21.15.113:0/3255426978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f80005040 con 0x7f6f8c0fd100 2024-04-12T05:51:32.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f8bfff700 1 --2- 172.21.15.113:0/3255426978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f8c0fd100 0x7f6f8c109690 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7f6f7800e910 tx=0x7f6f7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:32.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f89ffb700 1 -- 172.21.15.113:0/3255426978 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f7800eeb0 con 0x7f6f8c0fd100 2024-04-12T05:51:32.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f89ffb700 1 -- 172.21.15.113:0/3255426978 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f7800f040 con 0x7f6f8c0fd100 2024-04-12T05:51:32.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f92e08700 1 -- 172.21.15.113:0/3255426978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8c109f10 con 0x7f6f8c0fd100 2024-04-12T05:51:32.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.161+0000 7f6f89ffb700 1 -- 172.21.15.113:0/3255426978 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f7800c8a0 con 0x7f6f8c0fd100 2024-04-12T05:51:32.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.162+0000 7f6f92e08700 1 -- 172.21.15.113:0/3255426978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8c112560 con 0x7f6f8c0fd100 2024-04-12T05:51:32.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.163+0000 7f6f92e08700 1 -- 172.21.15.113:0/3255426978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8c05f380 con 0x7f6f8c0fd100 2024-04-12T05:51:32.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.163+0000 7f6f89ffb700 1 -- 172.21.15.113:0/3255426978 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6f780090d0 con 0x7f6f8c0fd100 2024-04-12T05:51:32.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.163+0000 7f6f89ffb700 1 --2- 172.21.15.113:0/3255426978 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f7c06caa0 0x7f6f7c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:32.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.163+0000 7f6f89ffb700 1 -- 172.21.15.113:0/3255426978 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f6f7808b920 con 0x7f6f8c0fd100 2024-04-12T05:51:32.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.163+0000 7f6f90ba4700 1 --2- 172.21.15.113:0/3255426978 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f7c06caa0 0x7f6f7c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:32.160 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.164+0000 7f6f90ba4700 1 --2- 172.21.15.113:0/3255426978 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f7c06caa0 0x7f6f7c06ef60 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f6f80004e60 tx=0x7f6f8000b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:32.163 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.168+0000 7f6f89ffb700 1 -- 172.21.15.113:0/3255426978 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f6f78055940 con 0x7f6f8c0fd100 2024-04-12T05:51:32.359 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.364+0000 7f6f92e08700 1 -- 172.21.15.113:0/3255426978 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f6f8c05eec0 con 0x7f6f8c0fd100 2024-04-12T05:51:32.360 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.364+0000 7f6f89ffb700 1 -- 172.21.15.113:0/3255426978 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v98) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6f8c05eec0 con 0x7f6f8c0fd100 2024-04-12T05:51:32.360 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-04-12T05:51:32.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.366+0000 7f6f92e08700 1 -- 172.21.15.113:0/3255426978 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f7c06caa0 msgr2=0x7f6f7c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.366+0000 7f6f92e08700 1 --2- 172.21.15.113:0/3255426978 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f7c06caa0 0x7f6f7c06ef60 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f6f80004e60 tx=0x7f6f8000b040).stop 2024-04-12T05:51:32.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.366+0000 7f6f92e08700 1 -- 172.21.15.113:0/3255426978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f8c0fd100 msgr2=0x7f6f8c109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.366+0000 7f6f92e08700 1 --2- 172.21.15.113:0/3255426978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f8c0fd100 0x7f6f8c109690 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7f6f7800e910 tx=0x7f6f7800b040).stop 2024-04-12T05:51:32.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.366+0000 7f6f92e08700 1 -- 172.21.15.113:0/3255426978 shutdown_connections 2024-04-12T05:51:32.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.366+0000 7f6f92e08700 1 --2- 172.21.15.113:0/3255426978 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f6f7c06caa0 0x7f6f7c06ef60 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.366+0000 7f6f92e08700 1 --2- 172.21.15.113:0/3255426978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f8c0fc830 0x7f6f8c1091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.366+0000 7f6f92e08700 1 --2- 172.21.15.113:0/3255426978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f8c0fd100 0x7f6f8c109690 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.366+0000 7f6f92e08700 1 -- 172.21.15.113:0/3255426978 >> 172.21.15.113:0/3255426978 conn(0x7f6f8c06c760 msgr2=0x7f6f8c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:32.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.366+0000 7f6f92e08700 1 -- 172.21.15.113:0/3255426978 shutdown_connections 2024-04-12T05:51:32.363 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.366+0000 7f6f92e08700 1 -- 172.21.15.113:0/3255426978 wait complete. 2024-04-12T05:51:32.373 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:32 smithi113 ceph-mon[103371]: pgmap v1183: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:32.373 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:32 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/4031749076' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T05:51:32.373 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:32 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2135056429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:51:32.374 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-12T05:51:32.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:32 smithi192 ceph-mon[85304]: pgmap v1183: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:32.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:32 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/4031749076' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T05:51:32.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:32 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2135056429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T05:51:32.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.497+0000 7ff21fb84700 1 -- 172.21.15.113:0/764152050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2180fca30 msgr2=0x7ff2180fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.497+0000 7ff21fb84700 1 --2- 172.21.15.113:0/764152050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2180fca30 0x7ff2180fce10 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7ff2080047f0 tx=0x7ff208009cb0).stop 2024-04-12T05:51:32.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.497+0000 7ff21fb84700 1 -- 172.21.15.113:0/764152050 shutdown_connections 2024-04-12T05:51:32.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.497+0000 7ff21fb84700 1 --2- 172.21.15.113:0/764152050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2180fd390 0x7ff2180f6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.497+0000 7ff21fb84700 1 --2- 172.21.15.113:0/764152050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2180fca30 0x7ff2180fce10 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.497+0000 7ff21fb84700 1 -- 172.21.15.113:0/764152050 >> 172.21.15.113:0/764152050 conn(0x7ff2180f25d0 msgr2=0x7ff2180f49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:32.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.497+0000 7ff21fb84700 1 -- 172.21.15.113:0/764152050 shutdown_connections 2024-04-12T05:51:32.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.497+0000 7ff21fb84700 1 -- 172.21.15.113:0/764152050 wait complete. 2024-04-12T05:51:32.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21fb84700 1 Processor -- start 2024-04-12T05:51:32.495 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21fb84700 1 -- start start 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21fb84700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2180fca30 0x7ff218109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21fb84700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2180fd390 0x7ff218109820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21fb84700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff218109d10 con 0x7ff2180fd390 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21fb84700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff218109e50 con 0x7ff2180fca30 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21d11f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2180fd390 0x7ff218109820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21d11f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2180fd390 0x7ff218109820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51376/0 (socket says 172.21.15.113:51376) 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21d11f700 1 -- 172.21.15.113:0/3606546690 learned_addr learned my addr 172.21.15.113:0/3606546690 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21d920700 1 --2- 172.21.15.113:0/3606546690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2180fca30 0x7ff218109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21d11f700 1 -- 172.21.15.113:0/3606546690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2180fca30 msgr2=0x7ff218109330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21d11f700 1 --2- 172.21.15.113:0/3606546690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2180fca30 0x7ff218109330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21d11f700 1 -- 172.21.15.113:0/3606546690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208005040 con 0x7ff2180fd390 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.498+0000 7ff21d11f700 1 --2- 172.21.15.113:0/3606546690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2180fd390 0x7ff218109820 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7ff214002750 tx=0x7ff214009ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.499+0000 7ff20effd700 1 -- 172.21.15.113:0/3606546690 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff214007b40 con 0x7ff2180fd390 2024-04-12T05:51:32.496 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.499+0000 7ff21fb84700 1 -- 172.21.15.113:0/3606546690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21810a100 con 0x7ff2180fd390 2024-04-12T05:51:32.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.499+0000 7ff20effd700 1 -- 172.21.15.113:0/3606546690 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff21400f040 con 0x7ff2180fd390 2024-04-12T05:51:32.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.499+0000 7ff20effd700 1 -- 172.21.15.113:0/3606546690 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff214014840 con 0x7ff2180fd390 2024-04-12T05:51:32.497 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.499+0000 7ff21fb84700 1 -- 172.21.15.113:0/3606546690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2181126f0 con 0x7ff2180fd390 2024-04-12T05:51:32.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.500+0000 7ff21fb84700 1 -- 172.21.15.113:0/3606546690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2180603b0 con 0x7ff2180fd390 2024-04-12T05:51:32.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.500+0000 7ff20effd700 1 -- 172.21.15.113:0/3606546690 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff214007ca0 con 0x7ff2180fd390 2024-04-12T05:51:32.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.500+0000 7ff20effd700 1 --2- 172.21.15.113:0/3606546690 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff20406caa0 0x7ff20406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:32.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.501+0000 7ff20effd700 1 -- 172.21.15.113:0/3606546690 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7ff21408bcd0 con 0x7ff2180fd390 2024-04-12T05:51:32.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.501+0000 7ff21d920700 1 --2- 172.21.15.113:0/3606546690 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff20406caa0 0x7ff20406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:32.498 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.501+0000 7ff21d920700 1 --2- 172.21.15.113:0/3606546690 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff20406caa0 0x7ff20406ef60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7ff208004f10 tx=0x7ff20800b040).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:32.500 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.505+0000 7ff20effd700 1 -- 172.21.15.113:0/3606546690 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7ff214055cf0 con 0x7ff2180fd390 2024-04-12T05:51:32.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.693+0000 7ff21fb84700 1 -- 172.21.15.113:0/3606546690 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7ff21805fb30 con 0x7ff2180fd390 2024-04-12T05:51:32.689 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.693+0000 7ff20effd700 1 -- 172.21.15.113:0/3606546690 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v98) v1 ==== 107+0+11 (secure 0 0 0) 0x7ff214059310 con 0x7ff2180fd390 2024-04-12T05:51:32.689 INFO:teuthology.orchestra.run.smithi113.stdout:pg_num: 32 2024-04-12T05:51:32.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.695+0000 7ff21fb84700 1 -- 172.21.15.113:0/3606546690 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff20406caa0 msgr2=0x7ff20406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.695+0000 7ff21fb84700 1 --2- 172.21.15.113:0/3606546690 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff20406caa0 0x7ff20406ef60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7ff208004f10 tx=0x7ff20800b040).stop 2024-04-12T05:51:32.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.695+0000 7ff21fb84700 1 -- 172.21.15.113:0/3606546690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2180fd390 msgr2=0x7ff218109820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.695+0000 7ff21fb84700 1 --2- 172.21.15.113:0/3606546690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2180fd390 0x7ff218109820 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7ff214002750 tx=0x7ff214009ec0).stop 2024-04-12T05:51:32.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.695+0000 7ff21fb84700 1 -- 172.21.15.113:0/3606546690 shutdown_connections 2024-04-12T05:51:32.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.695+0000 7ff21fb84700 1 --2- 172.21.15.113:0/3606546690 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7ff20406caa0 0x7ff20406ef60 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.695+0000 7ff21fb84700 1 --2- 172.21.15.113:0/3606546690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2180fca30 0x7ff218109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.695+0000 7ff21fb84700 1 --2- 172.21.15.113:0/3606546690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2180fd390 0x7ff218109820 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.695+0000 7ff21fb84700 1 -- 172.21.15.113:0/3606546690 >> 172.21.15.113:0/3606546690 conn(0x7ff2180f25d0 msgr2=0x7ff2180f3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:32.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.695+0000 7ff21fb84700 1 -- 172.21.15.113:0/3606546690 shutdown_connections 2024-04-12T05:51:32.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.695+0000 7ff21fb84700 1 -- 172.21.15.113:0/3606546690 wait complete. 2024-04-12T05:51:32.703 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-12T05:51:32.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.825+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2541753765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0fc830 msgr2=0x7f9cdc0fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.825+0000 7f9ce0d69700 1 --2- 172.21.15.113:0/2541753765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0fc830 0x7f9cdc0fcc10 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7f9ccc0047f0 tx=0x7f9ccc009cb0).stop 2024-04-12T05:51:32.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.825+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2541753765 shutdown_connections 2024-04-12T05:51:32.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.825+0000 7f9ce0d69700 1 --2- 172.21.15.113:0/2541753765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cdc0fd100 0x7f9cdc0f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.825+0000 7f9ce0d69700 1 --2- 172.21.15.113:0/2541753765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0fc830 0x7f9cdc0fcc10 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.825+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2541753765 >> 172.21.15.113:0/2541753765 conn(0x7f9cdc06c760 msgr2=0x7f9cdc06cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:32.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.825+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2541753765 shutdown_connections 2024-04-12T05:51:32.821 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.825+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2541753765 wait complete. 2024-04-12T05:51:32.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9ce0d69700 1 Processor -- start 2024-04-12T05:51:32.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9ce0d69700 1 -- start start 2024-04-12T05:51:32.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9ce0d69700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0fc830 0x7f9cdc10ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:32.823 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9ce0d69700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cdc0fd100 0x7f9cdc10d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9ce0d69700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc10d970 con 0x7f9cdc0fc830 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9ce0d69700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc10dab0 con 0x7f9cdc0fd100 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9cda59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0fc830 0x7f9cdc10ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9cda59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0fc830 0x7f9cdc10ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51394/0 (socket says 172.21.15.113:51394) 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9cda59c700 1 -- 172.21.15.113:0/2644852092 learned_addr learned my addr 172.21.15.113:0/2644852092 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9cd9d9b700 1 --2- 172.21.15.113:0/2644852092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cdc0fd100 0x7f9cdc10d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9cda59c700 1 -- 172.21.15.113:0/2644852092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cdc0fd100 msgr2=0x7f9cdc10d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9cda59c700 1 --2- 172.21.15.113:0/2644852092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cdc0fd100 0x7f9cdc10d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.826+0000 7f9cda59c700 1 -- 172.21.15.113:0/2644852092 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc005040 con 0x7f9cdc0fc830 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.827+0000 7f9cda59c700 1 --2- 172.21.15.113:0/2644852092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0fc830 0x7f9cdc10ce80 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f9ccc005010 tx=0x7f9ccc009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.827+0000 7f9cd37fe700 1 -- 172.21.15.113:0/2644852092 <== mon.0 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ccc00e030 con 0x7f9cdc0fc830 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.827+0000 7f9cd37fe700 1 -- 172.21.15.113:0/2644852092 <== mon.0 v2:172.21.15.113:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ccc007e90 con 0x7f9cdc0fc830 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.827+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2644852092 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cdc1070e0 con 0x7f9cdc0fc830 2024-04-12T05:51:32.824 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.827+0000 7f9cd37fe700 1 -- 172.21.15.113:0/2644852092 <== mon.0 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ccc018070 con 0x7f9cdc0fc830 2024-04-12T05:51:32.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.827+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2644852092 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cdc107640 con 0x7f9cdc0fc830 2024-04-12T05:51:32.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.828+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2644852092 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cdc05f380 con 0x7f9cdc0fc830 2024-04-12T05:51:32.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.828+0000 7f9cd37fe700 1 -- 172.21.15.113:0/2644852092 <== mon.0 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 38) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9ccc01d420 con 0x7f9cdc0fc830 2024-04-12T05:51:32.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.829+0000 7f9cd37fe700 1 --2- 172.21.15.113:0/2644852092 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9cc806ca30 0x7f9cc806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-12T05:51:32.825 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.829+0000 7f9cd37fe700 1 -- 172.21.15.113:0/2644852092 <== mon.0 v2:172.21.15.113:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6397+0+0 (secure 0 0 0) 0x7f9ccc08d270 con 0x7f9cdc0fc830 2024-04-12T05:51:32.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.829+0000 7f9cd9d9b700 1 --2- 172.21.15.113:0/2644852092 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9cc806ca30 0x7f9cc806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T05:51:32.826 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.829+0000 7f9cd9d9b700 1 --2- 172.21.15.113:0/2644852092 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9cc806ca30 0x7f9cc806eef0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f9cdc1087c0 tx=0x7f9cc4009d60).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T05:51:32.828 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:32.833+0000 7f9cd37fe700 1 -- 172.21.15.113:0/2644852092 <== mon.0 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180903 (secure 0 0 0) 0x7f9ccc05ad20 con 0x7f9cdc0fc830 2024-04-12T05:51:33.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.044+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2644852092 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9cdc0f9920 con 0x7f9cdc0fc830 2024-04-12T05:51:33.039 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.044+0000 7f9cd37fe700 1 -- 172.21.15.113:0/2644852092 <== mon.0 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 32 entries v98) v1 ==== 81+0+1928 (secure 0 0 0) 0x7f9ccc05a8b0 con 0x7f9cdc0fc830 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6827/1262609849 2024-04-13T05:18:43.274072+0000 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6801/3082109414 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:6824/616350866 2024-04-13T05:18:43.274091+0000 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6800/3082109414 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/321762285 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/2190156688 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/3618578044 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:6825/616350866 2024-04-13T05:18:43.274091+0000 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/313346589 2024-04-13T05:13:51.744570+0000 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/4036112881 2024-04-13T05:08:26.214053+0000 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6801/593576174 2024-04-13T05:08:26.214053+0000 2024-04-12T05:51:33.040 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6800/593576174 2024-04-13T05:08:26.214053+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/1777682971 2024-04-13T04:59:57.862720+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:0/2210150409 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6800/1349703403 2024-04-13T05:01:23.491618+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:0/2589268524 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:6828/2257000201 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/516367458 2024-04-13T05:01:23.491618+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/781838919 2024-04-13T05:08:26.214053+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6801/1349703403 2024-04-13T05:01:23.491618+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6801/2567243935 2024-04-13T05:00:29.828900+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/3213888755 2024-04-13T05:01:23.491618+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:6829/2257000201 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6826/1262609849 2024-04-13T05:18:43.274072+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:0/1673786821 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/2902584436 2024-04-13T05:00:29.828900+0000 2024-04-12T05:51:33.041 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/4242055626 2024-04-13T05:00:29.828900+0000 2024-04-12T05:51:33.042 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6800/3275382936 2024-04-13T04:59:57.862720+0000 2024-04-12T05:51:33.042 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6801/3275382936 2024-04-13T04:59:57.862720+0000 2024-04-12T05:51:33.042 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.192:0/2053192105 2024-04-13T05:10:55.934327+0000 2024-04-12T05:51:33.042 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:6800/2567243935 2024-04-13T05:00:29.828900+0000 2024-04-12T05:51:33.042 INFO:teuthology.orchestra.run.smithi113.stdout:172.21.15.113:0/3450096338 2024-04-13T04:59:57.862720+0000 2024-04-12T05:51:33.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.047+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2644852092 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9cc806ca30 msgr2=0x7f9cc806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:33.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.047+0000 7f9ce0d69700 1 --2- 172.21.15.113:0/2644852092 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9cc806ca30 0x7f9cc806eef0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f9cdc1087c0 tx=0x7f9cc4009d60).stop 2024-04-12T05:51:33.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.047+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2644852092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0fc830 msgr2=0x7f9cdc10ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T05:51:33.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.047+0000 7f9ce0d69700 1 --2- 172.21.15.113:0/2644852092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0fc830 0x7f9cdc10ce80 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f9ccc005010 tx=0x7f9ccc009cb0).stop 2024-04-12T05:51:33.043 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.047+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2644852092 shutdown_connections 2024-04-12T05:51:33.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.047+0000 7f9ce0d69700 1 --2- 172.21.15.113:0/2644852092 >> [v2:172.21.15.113:6800/2746473633,v1:172.21.15.113:6801/2746473633] conn(0x7f9cc806ca30 0x7f9cc806eef0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:33.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.047+0000 7f9ce0d69700 1 --2- 172.21.15.113:0/2644852092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cdc0fc830 0x7f9cdc10ce80 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:33.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.047+0000 7f9ce0d69700 1 --2- 172.21.15.113:0/2644852092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9cdc0fd100 0x7f9cdc10d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-12T05:51:33.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.047+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2644852092 >> 172.21.15.113:0/2644852092 conn(0x7f9cdc06c760 msgr2=0x7f9cdc0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T05:51:33.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.047+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2644852092 shutdown_connections 2024-04-12T05:51:33.044 INFO:teuthology.orchestra.run.smithi113.stderr:2024-04-12T05:51:33.047+0000 7f9ce0d69700 1 -- 172.21.15.113:0/2644852092 wait complete. 2024-04-12T05:51:33.044 INFO:teuthology.orchestra.run.smithi113.stderr:listed 32 entries 2024-04-12T05:51:33.054 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi113.front.sepia.ceph.com... 2024-04-12T05:51:33.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T05:51:33.055 DEBUG:teuthology.orchestra.run.smithi113:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-12T05:51:33.092 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T05:51:33.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:33 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2397272457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T05:51:33.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:33 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3255426978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T05:51:33.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:33 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/3606546690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T05:51:33.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:33 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2397272457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T05:51:33.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:33 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3255426978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-12T05:51:33.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:33 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/3606546690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-12T05:51:34.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:34 smithi113 ceph-mon[103371]: pgmap v1184: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:34.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:34 smithi113 ceph-mon[103371]: from='client.? 172.21.15.113:0/2644852092' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T05:51:34.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:34 smithi192 ceph-mon[85304]: pgmap v1184: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:34.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:34 smithi192 ceph-mon[85304]: from='client.? 172.21.15.113:0/2644852092' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T05:51:36.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:36 smithi113 ceph-mon[103371]: pgmap v1185: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:36.441 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:36 smithi192 ceph-mon[85304]: pgmap v1185: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:37.418 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:37.442 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:38.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:38 smithi113 ceph-mon[103371]: pgmap v1186: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:38.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:38 smithi192 ceph-mon[85304]: pgmap v1186: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:40.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:40 smithi113 ceph-mon[103371]: pgmap v1187: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:40.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:40 smithi192 ceph-mon[85304]: pgmap v1187: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:42.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:42 smithi113 ceph-mon[103371]: pgmap v1188: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:42.693 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:42 smithi192 ceph-mon[85304]: pgmap v1188: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:44.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:44 smithi113 ceph-mon[103371]: pgmap v1189: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:44.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:44 smithi192 ceph-mon[85304]: pgmap v1189: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:46.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:46 smithi113 ceph-mon[103371]: pgmap v1190: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:46.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:46 smithi192 ceph-mon[85304]: pgmap v1190: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:48.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:48 smithi113 ceph-mon[103371]: pgmap v1191: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:48.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:48 smithi192 ceph-mon[85304]: pgmap v1191: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:50.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:50 smithi113 ceph-mon[103371]: pgmap v1192: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:50.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:50 smithi192 ceph-mon[85304]: pgmap v1192: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:52.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:52 smithi113 ceph-mon[103371]: pgmap v1193: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:52.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:52.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:52 smithi192 ceph-mon[85304]: pgmap v1193: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:52.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:51:53.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:53 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:51:53.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:53 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:51:54.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:54 smithi113 ceph-mon[103371]: pgmap v1194: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:54.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:54 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:51:54.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:54 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:51:54.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:54 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:51:54.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:54 smithi192 ceph-mon[85304]: pgmap v1194: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:54.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:54 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:51:54.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:54 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:51:54.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:54 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:51:56.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:56 smithi113 ceph-mon[103371]: pgmap v1195: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:56.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:56 smithi192 ceph-mon[85304]: pgmap v1195: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:58.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:51:58 smithi113 ceph-mon[103371]: pgmap v1196: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:51:58.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:51:58 smithi192 ceph-mon[85304]: pgmap v1196: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:00.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:00 smithi113 ceph-mon[103371]: pgmap v1197: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:00.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:00 smithi192 ceph-mon[85304]: pgmap v1197: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:02.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:02 smithi113 ceph-mon[103371]: pgmap v1198: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:02.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:02 smithi192 ceph-mon[85304]: pgmap v1198: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:04.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:04 smithi113 ceph-mon[103371]: pgmap v1199: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:04.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:04 smithi192 ceph-mon[85304]: pgmap v1199: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:06.668 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:06 smithi113 ceph-mon[103371]: pgmap v1200: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:06.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:06 smithi192 ceph-mon[85304]: pgmap v1200: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:07.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:07.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:08.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:08 smithi113 ceph-mon[103371]: pgmap v1201: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:08.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:08 smithi192 ceph-mon[85304]: pgmap v1201: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:10.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:10 smithi113 ceph-mon[103371]: pgmap v1202: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:10.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:10 smithi192 ceph-mon[85304]: pgmap v1202: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:12.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:12 smithi113 ceph-mon[103371]: pgmap v1203: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:12.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:12 smithi192 ceph-mon[85304]: pgmap v1203: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:14.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:14 smithi113 ceph-mon[103371]: pgmap v1204: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:14.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:14 smithi192 ceph-mon[85304]: pgmap v1204: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:16.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:16 smithi113 ceph-mon[103371]: pgmap v1205: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:16.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:16 smithi192 ceph-mon[85304]: pgmap v1205: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:18.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:18 smithi113 ceph-mon[103371]: pgmap v1206: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:18.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:18 smithi192 ceph-mon[85304]: pgmap v1206: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:20.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:20 smithi113 ceph-mon[103371]: pgmap v1207: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:20.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:20 smithi192 ceph-mon[85304]: pgmap v1207: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:22.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:22 smithi113 ceph-mon[103371]: pgmap v1208: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:22.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:22.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:22 smithi192 ceph-mon[85304]: pgmap v1208: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:22.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:23.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:23 smithi113 ceph-mon[103371]: pgmap v1209: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:23.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:23 smithi192 ceph-mon[85304]: pgmap v1209: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:25.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:25 smithi113 ceph-mon[103371]: pgmap v1210: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:25.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:25 smithi192 ceph-mon[85304]: pgmap v1210: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:27.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:27 smithi113 ceph-mon[103371]: pgmap v1211: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:27.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:27 smithi192 ceph-mon[85304]: pgmap v1211: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:29.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:29 smithi113 ceph-mon[103371]: pgmap v1212: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:29.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:29 smithi192 ceph-mon[85304]: pgmap v1212: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:31.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:31 smithi113 ceph-mon[103371]: pgmap v1213: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:31.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:31 smithi192 ceph-mon[85304]: pgmap v1213: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:33.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:33 smithi113 ceph-mon[103371]: pgmap v1214: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:33.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:33 smithi192 ceph-mon[85304]: pgmap v1214: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:35.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:35 smithi113 ceph-mon[103371]: pgmap v1215: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:35.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:35 smithi192 ceph-mon[85304]: pgmap v1215: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:37.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:37 smithi113 ceph-mon[103371]: pgmap v1216: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:37.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:37.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:37 smithi192 ceph-mon[85304]: pgmap v1216: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:37.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:39.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:39 smithi113 ceph-mon[103371]: pgmap v1217: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:39.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:39 smithi192 ceph-mon[85304]: pgmap v1217: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:41.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:41 smithi113 ceph-mon[103371]: pgmap v1218: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:41.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:41 smithi192 ceph-mon[85304]: pgmap v1218: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:43.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:43 smithi113 ceph-mon[103371]: pgmap v1219: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:43.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:43 smithi192 ceph-mon[85304]: pgmap v1219: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:45.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:45 smithi113 ceph-mon[103371]: pgmap v1220: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:45.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:45 smithi192 ceph-mon[85304]: pgmap v1220: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:47.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:47 smithi113 ceph-mon[103371]: pgmap v1221: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:47.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:47 smithi192 ceph-mon[85304]: pgmap v1221: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:49.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:49 smithi113 ceph-mon[103371]: pgmap v1222: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:49.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:49 smithi192 ceph-mon[85304]: pgmap v1222: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:51.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:51 smithi113 ceph-mon[103371]: pgmap v1223: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:51.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:51 smithi192 ceph-mon[85304]: pgmap v1223: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:52.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:52.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:52:53.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:53 smithi113 ceph-mon[103371]: pgmap v1224: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:53.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:53 smithi192 ceph-mon[85304]: pgmap v1224: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:54.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:54 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:52:54.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:54 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:52:54.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:54 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:52:54.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:54 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:52:54.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:54 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:52:54.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:54 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:52:54.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:54 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:52:54.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:54 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:52:56.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:56 smithi113 ceph-mon[103371]: pgmap v1225: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:56.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:56 smithi192 ceph-mon[85304]: pgmap v1225: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:58.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:52:58 smithi113 ceph-mon[103371]: pgmap v1226: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:52:58.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:52:58 smithi192 ceph-mon[85304]: pgmap v1226: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:00.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:00 smithi113 ceph-mon[103371]: pgmap v1227: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:00.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:00 smithi192 ceph-mon[85304]: pgmap v1227: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:02.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:02 smithi113 ceph-mon[103371]: pgmap v1228: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:02.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:02 smithi192 ceph-mon[85304]: pgmap v1228: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:04.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:04 smithi113 ceph-mon[103371]: pgmap v1229: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:04.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:04 smithi192 ceph-mon[85304]: pgmap v1229: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:06.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:06 smithi113 ceph-mon[103371]: pgmap v1230: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:06.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:06 smithi192 ceph-mon[85304]: pgmap v1230: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:07.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:07.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:08.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:08 smithi113 ceph-mon[103371]: pgmap v1231: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:08.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:08 smithi192 ceph-mon[85304]: pgmap v1231: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:10.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:10 smithi113 ceph-mon[103371]: pgmap v1232: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:10.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:10 smithi192 ceph-mon[85304]: pgmap v1232: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:12.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:12 smithi113 ceph-mon[103371]: pgmap v1233: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:12.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:12 smithi192 ceph-mon[85304]: pgmap v1233: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:14.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:14 smithi113 ceph-mon[103371]: pgmap v1234: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:14.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:14 smithi192 ceph-mon[85304]: pgmap v1234: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:16.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:16 smithi113 ceph-mon[103371]: pgmap v1235: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:16.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:16 smithi192 ceph-mon[85304]: pgmap v1235: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:18.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:18 smithi113 ceph-mon[103371]: pgmap v1236: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:18.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:18 smithi192 ceph-mon[85304]: pgmap v1236: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:20.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:20 smithi113 ceph-mon[103371]: pgmap v1237: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:20.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:20 smithi192 ceph-mon[85304]: pgmap v1237: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:22.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:22 smithi113 ceph-mon[103371]: pgmap v1238: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:22.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:22.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:22 smithi192 ceph-mon[85304]: pgmap v1238: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:22.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:24.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:24 smithi113 ceph-mon[103371]: pgmap v1239: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:24.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:24 smithi192 ceph-mon[85304]: pgmap v1239: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:26.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:26 smithi113 ceph-mon[103371]: pgmap v1240: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:26.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:26 smithi192 ceph-mon[85304]: pgmap v1240: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:28.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:28 smithi113 ceph-mon[103371]: pgmap v1241: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:28.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:28 smithi192 ceph-mon[85304]: pgmap v1241: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:30.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:30 smithi113 ceph-mon[103371]: pgmap v1242: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:30.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:30 smithi192 ceph-mon[85304]: pgmap v1242: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:32.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:32 smithi113 ceph-mon[103371]: pgmap v1243: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:32.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:32 smithi192 ceph-mon[85304]: pgmap v1243: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:34.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:34 smithi113 ceph-mon[103371]: pgmap v1244: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:34.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:34 smithi192 ceph-mon[85304]: pgmap v1244: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:36.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:36 smithi113 ceph-mon[103371]: pgmap v1245: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:36.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:36 smithi192 ceph-mon[85304]: pgmap v1245: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:37.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:37.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:38.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:38 smithi113 ceph-mon[103371]: pgmap v1246: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:38.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:38 smithi192 ceph-mon[85304]: pgmap v1246: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:39.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:39 smithi113 ceph-mon[103371]: pgmap v1247: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:39.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:39 smithi192 ceph-mon[85304]: pgmap v1247: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:41.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:41 smithi113 ceph-mon[103371]: pgmap v1248: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:41.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:41 smithi192 ceph-mon[85304]: pgmap v1248: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:43.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:43 smithi113 ceph-mon[103371]: pgmap v1249: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:43.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:43 smithi192 ceph-mon[85304]: pgmap v1249: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:45.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:45 smithi113 ceph-mon[103371]: pgmap v1250: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:45.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:45 smithi192 ceph-mon[85304]: pgmap v1250: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:47.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:47 smithi113 ceph-mon[103371]: pgmap v1251: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:47.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:47 smithi192 ceph-mon[85304]: pgmap v1251: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:49.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:49 smithi113 ceph-mon[103371]: pgmap v1252: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:49.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:49 smithi192 ceph-mon[85304]: pgmap v1252: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:51.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:51 smithi113 ceph-mon[103371]: pgmap v1253: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:51.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:51 smithi192 ceph-mon[85304]: pgmap v1253: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:52.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:52.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:53:53.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:53 smithi113 ceph-mon[103371]: pgmap v1254: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:53.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:53 smithi192 ceph-mon[85304]: pgmap v1254: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:54.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:54 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:53:54.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:54 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:53:55.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:55 smithi113 ceph-mon[103371]: pgmap v1255: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:55.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:53:55.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:53:55.918 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:53:55.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:55 smithi192 ceph-mon[85304]: pgmap v1255: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:55.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:53:55.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:53:55.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:53:57.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:57 smithi113 ceph-mon[103371]: pgmap v1256: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:57.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:57 smithi192 ceph-mon[85304]: pgmap v1256: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:59.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:53:59 smithi113 ceph-mon[103371]: pgmap v1257: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:53:59.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:53:59 smithi192 ceph-mon[85304]: pgmap v1257: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:01.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:01 smithi113 ceph-mon[103371]: pgmap v1258: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:01.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:01 smithi192 ceph-mon[85304]: pgmap v1258: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:03.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:03 smithi113 ceph-mon[103371]: pgmap v1259: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:03.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:03 smithi192 ceph-mon[85304]: pgmap v1259: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:05.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:05 smithi113 ceph-mon[103371]: pgmap v1260: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:05.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:05 smithi192 ceph-mon[85304]: pgmap v1260: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:07.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:07 smithi113 ceph-mon[103371]: pgmap v1261: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:07.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:07.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:07 smithi192 ceph-mon[85304]: pgmap v1261: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:07.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:09.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:09 smithi113 ceph-mon[103371]: pgmap v1262: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:09.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:09 smithi192 ceph-mon[85304]: pgmap v1262: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:11.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:11 smithi113 ceph-mon[103371]: pgmap v1263: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:11.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:11 smithi192 ceph-mon[85304]: pgmap v1263: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:14.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:14 smithi113 ceph-mon[103371]: pgmap v1264: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:14.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:14 smithi192 ceph-mon[85304]: pgmap v1264: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:16.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:16 smithi113 ceph-mon[103371]: pgmap v1265: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:16.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:16 smithi192 ceph-mon[85304]: pgmap v1265: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:18.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:18 smithi113 ceph-mon[103371]: pgmap v1266: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:18.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:18 smithi192 ceph-mon[85304]: pgmap v1266: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:20.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:20 smithi113 ceph-mon[103371]: pgmap v1267: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:20.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:20 smithi192 ceph-mon[85304]: pgmap v1267: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:22.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:22 smithi113 ceph-mon[103371]: pgmap v1268: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:22.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:22.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:22 smithi192 ceph-mon[85304]: pgmap v1268: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:22.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:24.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:24 smithi113 ceph-mon[103371]: pgmap v1269: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:24.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:24 smithi192 ceph-mon[85304]: pgmap v1269: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:26.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:26 smithi113 ceph-mon[103371]: pgmap v1270: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:26.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:26 smithi192 ceph-mon[85304]: pgmap v1270: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:28.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:28 smithi113 ceph-mon[103371]: pgmap v1271: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:28.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:28 smithi192 ceph-mon[85304]: pgmap v1271: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:30.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:30 smithi113 ceph-mon[103371]: pgmap v1272: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:30.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:30 smithi192 ceph-mon[85304]: pgmap v1272: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:32.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:32 smithi113 ceph-mon[103371]: pgmap v1273: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:32.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:32 smithi192 ceph-mon[85304]: pgmap v1273: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:34.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:34 smithi113 ceph-mon[103371]: pgmap v1274: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:34.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:34 smithi192 ceph-mon[85304]: pgmap v1274: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:36.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:36 smithi113 ceph-mon[103371]: pgmap v1275: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:36.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:36 smithi192 ceph-mon[85304]: pgmap v1275: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:37.670 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:37.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:38.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:38 smithi113 ceph-mon[103371]: pgmap v1276: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:38.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:38 smithi192 ceph-mon[85304]: pgmap v1276: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:40.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:40 smithi113 ceph-mon[103371]: pgmap v1277: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:40.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:40 smithi192 ceph-mon[85304]: pgmap v1277: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:42.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:42 smithi113 ceph-mon[103371]: pgmap v1278: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:42.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:42 smithi192 ceph-mon[85304]: pgmap v1278: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:44.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:44 smithi113 ceph-mon[103371]: pgmap v1279: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:44.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:44 smithi192 ceph-mon[85304]: pgmap v1279: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:46.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:46 smithi113 ceph-mon[103371]: pgmap v1280: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:46.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:46 smithi192 ceph-mon[85304]: pgmap v1280: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:48.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:48 smithi113 ceph-mon[103371]: pgmap v1281: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:48.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:48 smithi192 ceph-mon[85304]: pgmap v1281: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:50.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:50 smithi113 ceph-mon[103371]: pgmap v1282: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:50.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:50 smithi192 ceph-mon[85304]: pgmap v1282: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:52.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:52 smithi113 ceph-mon[103371]: pgmap v1283: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:52.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:52.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:52 smithi192 ceph-mon[85304]: pgmap v1283: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:52.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:54:54.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:54 smithi192 ceph-mon[85304]: pgmap v1284: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:54.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:54 smithi113 ceph-mon[103371]: pgmap v1284: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:55.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:55 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:54:55.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:55 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:54:56.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:56 smithi192 ceph-mon[85304]: pgmap v1285: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:56.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:54:56.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:54:56.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:54:56.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:56 smithi113 ceph-mon[103371]: pgmap v1285: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:56.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:54:56.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:54:56.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:54:57.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:57 smithi192 ceph-mon[85304]: pgmap v1286: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:57.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:57 smithi113 ceph-mon[103371]: pgmap v1286: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:59.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:54:59 smithi113 ceph-mon[103371]: pgmap v1287: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:54:59.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:54:59 smithi192 ceph-mon[85304]: pgmap v1287: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:01.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:01 smithi113 ceph-mon[103371]: pgmap v1288: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:01.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:01 smithi192 ceph-mon[85304]: pgmap v1288: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:03.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:03 smithi113 ceph-mon[103371]: pgmap v1289: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:03.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:03 smithi192 ceph-mon[85304]: pgmap v1289: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:05.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:05 smithi113 ceph-mon[103371]: pgmap v1290: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:05.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:05 smithi192 ceph-mon[85304]: pgmap v1290: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:07.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:07 smithi113 ceph-mon[103371]: pgmap v1291: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:07.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:07.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:07 smithi192 ceph-mon[85304]: pgmap v1291: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:07.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:09.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:09 smithi113 ceph-mon[103371]: pgmap v1292: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:09.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:09 smithi192 ceph-mon[85304]: pgmap v1292: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:11.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:11 smithi113 ceph-mon[103371]: pgmap v1293: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:11.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:11 smithi192 ceph-mon[85304]: pgmap v1293: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:13.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:13 smithi113 ceph-mon[103371]: pgmap v1294: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:13.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:13 smithi192 ceph-mon[85304]: pgmap v1294: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:15.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:15 smithi113 ceph-mon[103371]: pgmap v1295: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:15.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:15 smithi192 ceph-mon[85304]: pgmap v1295: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:17.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:17 smithi113 ceph-mon[103371]: pgmap v1296: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:17.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:17 smithi192 ceph-mon[85304]: pgmap v1296: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:19.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:19 smithi113 ceph-mon[103371]: pgmap v1297: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:19.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:19 smithi192 ceph-mon[85304]: pgmap v1297: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:21.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:21 smithi113 ceph-mon[103371]: pgmap v1298: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:21.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:21 smithi192 ceph-mon[85304]: pgmap v1298: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:22.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:22.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:23.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:23 smithi113 ceph-mon[103371]: pgmap v1299: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:23.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:23 smithi192 ceph-mon[85304]: pgmap v1299: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:25.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:25 smithi113 ceph-mon[103371]: pgmap v1300: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:25.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:25 smithi192 ceph-mon[85304]: pgmap v1300: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:27.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:27 smithi113 ceph-mon[103371]: pgmap v1301: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:27.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:27 smithi192 ceph-mon[85304]: pgmap v1301: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:29.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:29 smithi113 ceph-mon[103371]: pgmap v1302: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:29.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:29 smithi192 ceph-mon[85304]: pgmap v1302: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:31.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:31 smithi113 ceph-mon[103371]: pgmap v1303: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:31.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:31 smithi192 ceph-mon[85304]: pgmap v1303: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:34.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:34 smithi113 ceph-mon[103371]: pgmap v1304: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:34.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:34 smithi192 ceph-mon[85304]: pgmap v1304: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:36.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:36 smithi113 ceph-mon[103371]: pgmap v1305: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:36.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:36 smithi192 ceph-mon[85304]: pgmap v1305: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:37.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:37 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:37.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:37 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:38.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:38 smithi113 ceph-mon[103371]: pgmap v1306: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:38.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:38 smithi192 ceph-mon[85304]: pgmap v1306: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:40.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:40 smithi113 ceph-mon[103371]: pgmap v1307: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:40.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:40 smithi192 ceph-mon[85304]: pgmap v1307: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:42.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:42 smithi113 ceph-mon[103371]: pgmap v1308: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:42.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:42 smithi192 ceph-mon[85304]: pgmap v1308: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:44.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:44 smithi113 ceph-mon[103371]: pgmap v1309: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:44.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:44 smithi192 ceph-mon[85304]: pgmap v1309: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:46.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:46 smithi113 ceph-mon[103371]: pgmap v1310: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:46.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:46 smithi192 ceph-mon[85304]: pgmap v1310: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:48.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:48 smithi113 ceph-mon[103371]: pgmap v1311: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:48.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:48 smithi192 ceph-mon[85304]: pgmap v1311: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:50.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:50 smithi113 ceph-mon[103371]: pgmap v1312: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:50.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:50 smithi192 ceph-mon[85304]: pgmap v1312: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:52.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:52 smithi113 ceph-mon[103371]: pgmap v1313: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:52.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:52 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:52.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:52 smithi192 ceph-mon[85304]: pgmap v1313: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:52.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:52 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:55:54.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:54 smithi113 ceph-mon[103371]: pgmap v1314: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:54.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:54 smithi192 ceph-mon[85304]: pgmap v1314: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:56.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:56 smithi113 ceph-mon[103371]: pgmap v1315: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:56.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:55:56.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:55:56.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:55:56.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:56 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:55:56.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:56 smithi192 ceph-mon[85304]: pgmap v1315: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:56.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T05:55:56.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T05:55:56.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T05:55:56.695 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:56 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' 2024-04-12T05:55:58.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:55:58 smithi113 ceph-mon[103371]: pgmap v1316: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:55:58.692 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:55:58 smithi192 ceph-mon[85304]: pgmap v1316: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:00.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:00 smithi113 ceph-mon[103371]: pgmap v1317: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:00.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:00 smithi192 ceph-mon[85304]: pgmap v1317: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:02.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:02 smithi113 ceph-mon[103371]: pgmap v1318: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:02.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:02 smithi192 ceph-mon[85304]: pgmap v1318: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:04.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:04 smithi113 ceph-mon[103371]: pgmap v1319: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:04.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:04 smithi192 ceph-mon[85304]: pgmap v1319: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:06.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:06 smithi192 ceph-mon[85304]: pgmap v1320: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:06.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:06 smithi113 ceph-mon[103371]: pgmap v1320: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:07.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:07 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:07.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:07 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:08.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:08 smithi192 ceph-mon[85304]: pgmap v1321: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:08.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:08 smithi113 ceph-mon[103371]: pgmap v1321: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:10.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:10 smithi192 ceph-mon[85304]: pgmap v1322: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:10.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:10 smithi113 ceph-mon[103371]: pgmap v1322: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:12.667 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:12 smithi113 ceph-mon[103371]: pgmap v1323: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:12.691 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:12 smithi192 ceph-mon[85304]: pgmap v1323: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:13.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:13 smithi113 ceph-mon[103371]: pgmap v1324: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:13.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:13 smithi192 ceph-mon[85304]: pgmap v1324: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:15.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:15 smithi113 ceph-mon[103371]: pgmap v1325: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:15.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:15 smithi192 ceph-mon[85304]: pgmap v1325: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:17.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:17 smithi113 ceph-mon[103371]: pgmap v1326: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:17.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:17 smithi192 ceph-mon[85304]: pgmap v1326: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:19.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:19 smithi113 ceph-mon[103371]: pgmap v1327: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:19.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:19 smithi192 ceph-mon[85304]: pgmap v1327: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:21.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:21 smithi113 ceph-mon[103371]: pgmap v1328: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:21.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:21 smithi192 ceph-mon[85304]: pgmap v1328: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:22.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:22 smithi113 ceph-mon[103371]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:22.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:22 smithi192 ceph-mon[85304]: from='mgr.34104 172.21.15.113:0/77816307' entity='mgr.smithi113.ctsazg' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T05:56:23.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:23 smithi113 ceph-mon[103371]: pgmap v1329: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:23.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:23 smithi192 ceph-mon[85304]: pgmap v1329: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:25.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:25 smithi113 ceph-mon[103371]: pgmap v1330: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:25.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:25 smithi192 ceph-mon[85304]: pgmap v1330: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:27.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:27 smithi113 ceph-mon[103371]: pgmap v1331: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:27.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:27 smithi192 ceph-mon[85304]: pgmap v1331: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:29.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:29 smithi113 ceph-mon[103371]: pgmap v1332: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:29.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:29 smithi192 ceph-mon[85304]: pgmap v1332: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:31.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:31 smithi113 ceph-mon[103371]: pgmap v1333: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:31.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:31 smithi192 ceph-mon[85304]: pgmap v1333: 65 pgs: 65 active+clean; 259 MiB data, 1007 MiB used, 535 GiB / 536 GiB avail 2024-04-12T05:56:32.166 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-12T05:56:32.167 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-12T05:56:32.168 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-12T05:56:32.180 INFO:tasks.cephadm:Teardown begin 2024-04-12T05:56:32.180 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/cephadm.py", line 1750, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-12T05:56:32.181 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T05:56:32.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T05:56:32.255 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-12T05:56:32.256 DEBUG:teuthology.orchestra.run.smithi113:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-12T05:56:32.278 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-12T05:56:32.317 INFO:tasks.cephadm:Stopping all daemons... 2024-04-12T05:56:32.317 INFO:tasks.cephadm.mon.smithi113:Stopping mon.smithi113... 2024-04-12T05:56:32.318 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi113 2024-04-12T05:56:32.611 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:32 smithi113 systemd[1]: Stopping Ceph mon.smithi113 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:56:32.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:56:32.615+0000 7f142bc39700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi113 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:56:32.917 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:32 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113[103348]: 2024-04-12T05:56:32.615+0000 7f142bc39700 -1 mon.smithi113@0(leader) e3 *** Got Signal Terminated *** 2024-04-12T05:56:33.417 INFO:journalctl@ceph.mon.smithi113.smithi113.stdout:Apr 12 05:56:33 smithi113 bash[169449]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi113 2024-04-12T05:56:33.546 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi113.service' 2024-04-12T05:56:33.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T05:56:33.593 INFO:tasks.cephadm.mon.smithi113:Stopped mon.smithi113 2024-04-12T05:56:33.594 INFO:tasks.cephadm.mon.smithi192:Stopping mon.smithi192... 2024-04-12T05:56:33.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi192 2024-04-12T05:56:33.941 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:33 smithi192 systemd[1]: Stopping Ceph mon.smithi192 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:56:33.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[85281]: 2024-04-12T05:56:33.871+0000 7efd65334700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi192 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:56:33.942 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:33 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192[85281]: 2024-04-12T05:56:33.871+0000 7efd65334700 -1 mon.smithi192@1(peon) e3 *** Got Signal Terminated *** 2024-04-12T05:56:34.441 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Apr 12 05:56:34 smithi192 bash[133522]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-mon-smithi192 2024-04-12T05:56:34.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@mon.smithi192.service' 2024-04-12T05:56:34.782 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T05:56:34.783 INFO:tasks.cephadm.mon.smithi192:Stopped mon.smithi192 2024-04-12T05:56:34.783 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-12T05:56:34.783 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.0 2024-04-12T05:56:35.167 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:56:34 smithi113 systemd[1]: Stopping Ceph osd.0 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:56:35.667 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:56:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0[116980]: 2024-04-12T05:56:35.377+0000 7f66e25e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:56:35.667 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:56:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0[116980]: 2024-04-12T05:56:35.377+0000 7f66e25e0700 -1 osd.0 98 *** Got signal Terminated *** 2024-04-12T05:56:35.667 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:56:35 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0[116980]: 2024-04-12T05:56:35.377+0000 7f66e25e0700 -1 osd.0 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:56:41.417 INFO:journalctl@ceph.osd.0.smithi113.stdout:Apr 12 05:56:41 smithi113 bash[169611]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-0 2024-04-12T05:56:42.826 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.0.service' 2024-04-12T05:56:42.867 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T05:56:42.867 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-12T05:56:42.867 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-12T05:56:42.867 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.1 2024-04-12T05:56:43.164 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:56:42 smithi113 systemd[1]: Stopping Ceph osd.1 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:56:43.417 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:56:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[124118]: 2024-04-12T05:56:43.168+0000 7f9a9890e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:56:43.417 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:56:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[124118]: 2024-04-12T05:56:43.168+0000 7f9a9890e700 -1 osd.1 98 *** Got signal Terminated *** 2024-04-12T05:56:43.417 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:56:43 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1[124118]: 2024-04-12T05:56:43.168+0000 7f9a9890e700 -1 osd.1 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:56:49.167 INFO:journalctl@ceph.osd.1.smithi113.stdout:Apr 12 05:56:48 smithi113 bash[169878]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-1 2024-04-12T05:56:50.888 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.1.service' 2024-04-12T05:56:50.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T05:56:50.933 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-12T05:56:50.933 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-12T05:56:50.933 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.2 2024-04-12T05:56:51.167 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:56:51 smithi113 systemd[1]: Stopping Ceph osd.2 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:56:51.667 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:56:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[131495]: 2024-04-12T05:56:51.282+0000 7f5237dbe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:56:51.667 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:56:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[131495]: 2024-04-12T05:56:51.282+0000 7f5237dbe700 -1 osd.2 98 *** Got signal Terminated *** 2024-04-12T05:56:51.667 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:56:51 smithi113 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2[131495]: 2024-04-12T05:56:51.282+0000 7f5237dbe700 -1 osd.2 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:56:57.417 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:56:57 smithi113 bash[170149]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-2 2024-04-12T05:56:58.689 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.2.service' 2024-04-12T05:56:58.717 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:56:58 smithi113 systemd[1]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.2.service: Succeeded. 2024-04-12T05:56:58.717 INFO:journalctl@ceph.osd.2.smithi113.stdout:Apr 12 05:56:58 smithi113 systemd[1]: Stopped Ceph osd.2 for 52321e2c-f889-11ee-bc8c-c7b262605968. 2024-04-12T05:56:58.730 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T05:56:58.730 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-12T05:56:58.730 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-12T05:56:58.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.3 2024-04-12T05:56:59.028 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:56:58 smithi192 systemd[1]: Stopping Ceph osd.3 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:56:59.441 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:56:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[97959]: 2024-04-12T05:56:59.031+0000 7f6e16bd4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:56:59.441 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:56:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[97959]: 2024-04-12T05:56:59.031+0000 7f6e16bd4700 -1 osd.3 98 *** Got signal Terminated *** 2024-04-12T05:56:59.441 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:56:59 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[97959]: 2024-04-12T05:56:59.031+0000 7f6e16bd4700 -1 osd.3 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:57:02.691 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:57:02 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[97959]: 2024-04-12T05:57:02.345+0000 7f6e0f9c3700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:40.141461+0000 front 2024-04-12T05:56:40.141448+0000 (oldest deadline 2024-04-12T05:57:01.841456+0000) 2024-04-12T05:57:03.691 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:57:03 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3[97959]: 2024-04-12T05:57:03.319+0000 7f6e0f9c3700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:40.141461+0000 front 2024-04-12T05:56:40.141448+0000 (oldest deadline 2024-04-12T05:57:01.841456+0000) 2024-04-12T05:57:04.441 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:04 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:57:04.016+0000 7f8486b64700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:37.181169+0000 front 2024-04-12T05:56:37.181083+0000 (oldest deadline 2024-04-12T05:57:03.080968+0000) 2024-04-12T05:57:05.015 INFO:journalctl@ceph.osd.3.smithi192.stdout:Apr 12 05:57:04 smithi192 bash[133671]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-3 2024-04-12T05:57:05.441 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:57:05.019+0000 7f8486b64700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:37.181169+0000 front 2024-04-12T05:56:37.181083+0000 (oldest deadline 2024-04-12T05:57:03.080968+0000) 2024-04-12T05:57:05.442 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:05 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:05.112+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:06.320 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:57:06.025+0000 7f8486b64700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:37.181169+0000 front 2024-04-12T05:56:37.181083+0000 (oldest deadline 2024-04-12T05:57:03.080968+0000) 2024-04-12T05:57:06.320 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:06.133+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:06.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.3.service' 2024-04-12T05:57:06.667 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T05:57:06.667 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-12T05:57:06.667 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-12T05:57:06.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.4 2024-04-12T05:57:06.941 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:06 smithi192 systemd[1]: Stopping Ceph osd.4 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:57:07.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:07.087+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:07.442 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:06 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:57:06.997+0000 7f8486b64700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:37.181169+0000 front 2024-04-12T05:56:37.181083+0000 (oldest deadline 2024-04-12T05:57:03.080968+0000) 2024-04-12T05:57:07.442 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:57:07.001+0000 7f848dd75700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:57:07.442 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:57:07.001+0000 7f848dd75700 -1 osd.4 98 *** Got signal Terminated *** 2024-04-12T05:57:07.442 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:57:07.001+0000 7f848dd75700 -1 osd.4 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:57:08.441 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:07 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:57:07.993+0000 7f8486b64700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:37.181169+0000 front 2024-04-12T05:56:37.181083+0000 (oldest deadline 2024-04-12T05:57:03.080968+0000) 2024-04-12T05:57:08.442 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:08 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:08.126+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:09.441 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:08 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:57:08.983+0000 7f8486b64700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:37.181169+0000 front 2024-04-12T05:56:37.181083+0000 (oldest deadline 2024-04-12T05:57:03.080968+0000) 2024-04-12T05:57:09.442 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:09 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:09.134+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:10.441 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:57:10.033+0000 7f8486b64700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:37.181169+0000 front 2024-04-12T05:56:37.181083+0000 (oldest deadline 2024-04-12T05:57:03.080968+0000) 2024-04-12T05:57:10.442 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:10 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:10.134+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:11.441 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4[104848]: 2024-04-12T05:57:11.064+0000 7f8486b64700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:37.181169+0000 front 2024-04-12T05:56:37.181083+0000 (oldest deadline 2024-04-12T05:57:03.080968+0000) 2024-04-12T05:57:11.442 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:11 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:11.157+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:12.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:12.171+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:12.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:12 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:12.171+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6814 osd.1 since back 2024-04-12T05:56:45.396045+0000 front 2024-04-12T05:56:45.396082+0000 (oldest deadline 2024-04-12T05:57:11.295952+0000) 2024-04-12T05:57:13.164 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 12 05:57:12 smithi192 bash[133936]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-4 2024-04-12T05:57:13.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:13.168+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:13.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:13 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:13.168+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6814 osd.1 since back 2024-04-12T05:56:45.396045+0000 front 2024-04-12T05:56:45.396082+0000 (oldest deadline 2024-04-12T05:57:11.295952+0000) 2024-04-12T05:57:14.486 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:14.196+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:14.486 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:14.196+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6814 osd.1 since back 2024-04-12T05:56:45.396045+0000 front 2024-04-12T05:56:45.396082+0000 (oldest deadline 2024-04-12T05:57:11.295952+0000) 2024-04-12T05:57:14.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.4.service' 2024-04-12T05:57:14.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T05:57:14.534 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-12T05:57:14.534 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-12T05:57:14.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.5 2024-04-12T05:57:14.841 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:14 smithi192 systemd[1]: Stopping Ceph osd.5 for 52321e2c-f889-11ee-bc8c-c7b262605968... 2024-04-12T05:57:14.841 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:14.845+0000 7ff4080bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T05:57:15.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:14.845+0000 7ff4080bf700 -1 osd.5 98 *** Got signal Terminated *** 2024-04-12T05:57:15.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:14 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:14.845+0000 7ff4080bf700 -1 osd.5 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T05:57:15.158 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:15.162+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:15.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:15 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:15.162+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6814 osd.1 since back 2024-04-12T05:56:45.396045+0000 front 2024-04-12T05:56:45.396082+0000 (oldest deadline 2024-04-12T05:57:11.295952+0000) 2024-04-12T05:57:16.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:16.129+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:16.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:16 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:16.129+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6814 osd.1 since back 2024-04-12T05:56:45.396045+0000 front 2024-04-12T05:56:45.396082+0000 (oldest deadline 2024-04-12T05:57:11.295952+0000) 2024-04-12T05:57:17.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:17.128+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:17.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:17 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:17.128+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6814 osd.1 since back 2024-04-12T05:56:45.396045+0000 front 2024-04-12T05:56:45.396082+0000 (oldest deadline 2024-04-12T05:57:11.295952+0000) 2024-04-12T05:57:18.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:18.107+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:18.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:18 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:18.107+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6814 osd.1 since back 2024-04-12T05:56:45.396045+0000 front 2024-04-12T05:56:45.396082+0000 (oldest deadline 2024-04-12T05:57:11.295952+0000) 2024-04-12T05:57:19.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:19.153+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6806 osd.0 since back 2024-04-12T05:56:38.995088+0000 front 2024-04-12T05:56:38.995233+0000 (oldest deadline 2024-04-12T05:57:04.295132+0000) 2024-04-12T05:57:19.441 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:19 smithi192 ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5[111837]: 2024-04-12T05:57:19.153+0000 7ff400eae700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.113:6814 osd.1 since back 2024-04-12T05:56:45.396045+0000 front 2024-04-12T05:56:45.396082+0000 (oldest deadline 2024-04-12T05:57:11.295952+0000) 2024-04-12T05:57:20.941 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 12 05:57:20 smithi192 bash[134201]: ceph-52321e2c-f889-11ee-bc8c-c7b262605968-osd-5 2024-04-12T05:57:22.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52321e2c-f889-11ee-bc8c-c7b262605968@osd.5.service' 2024-04-12T05:57:22.270 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T05:57:22.270 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-12T05:57:22.270 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 --force --keep-logs 2024-04-12T05:57:27.444 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi113.stderr:ceph-fuse[77892]: fuse finished with error 0 and tester_r 0 2024-04-12T05:57:43.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 --force --keep-logs 2024-04-12T05:57:58.976 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T05:57:59.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T05:57:59.044 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-12T05:57:59.046 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627/remote/smithi113/crash 2024-04-12T05:57:59.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash -- . 2024-04-12T05:57:59.083 INFO:teuthology.orchestra.run.smithi113.stderr:tar: /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash: Cannot open: No such file or directory 2024-04-12T05:57:59.083 INFO:teuthology.orchestra.run.smithi113.stderr:tar: Error is not recoverable: exiting now 2024-04-12T05:57:59.085 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627/remote/smithi192/crash 2024-04-12T05:57:59.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash -- . 2024-04-12T05:57:59.112 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/crash: Cannot open: No such file or directory 2024-04-12T05:57:59.112 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2024-04-12T05:57:59.114 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-12T05:57:59.114 DEBUG:teuthology.orchestra.run.smithi113:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-04-12T05:57:59.157 INFO:teuthology.orchestra.run.smithi113.stderr:grep: /var/log/ceph/52321e2c-f889-11ee-bc8c-c7b262605968/ceph.log: No such file or directory 2024-04-12T05:57:59.159 INFO:tasks.cephadm:Compressing logs... 2024-04-12T05:57:59.159 DEBUG:teuthology.orchestra.run.smithi113:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T05:57:59.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T05:57:59.228 INFO:teuthology.orchestra.run.smithi113.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-12T05:57:59.229 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-12T05:59:21.138 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-04-12T05:59:40.551 INFO:tasks.cephadm:Archiving logs... 2024-04-12T05:59:40.552 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627/remote/smithi113/log 2024-04-12T05:59:40.553 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-12T05:59:52.882 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627/remote/smithi192/log 2024-04-12T05:59:52.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-12T06:00:03.273 INFO:tasks.cephadm:Removing cluster... 2024-04-12T06:00:03.274 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 --force 2024-04-12T06:00:03.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 52321e2c-f889-11ee-bc8c-c7b262605968 --force 2024-04-12T06:00:04.342 INFO:tasks.cephadm:Removing cephadm ... 2024-04-12T06:00:04.342 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-12T06:00:04.363 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-12T06:00:04.381 INFO:tasks.cephadm:Teardown complete 2024-04-12T06:00:04.381 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-12T06:00:04.395 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-12T06:00:04.396 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-12T06:00:04.396 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-12T06:00:04.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-12T06:00:04.534 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-12T06:00:04.545 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-12T06:00:04.546 DEBUG:teuthology.orchestra.run.smithi113:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T06:00:04.548 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T06:00:04.564 INFO:teuthology.orchestra.run.smithi113.stderr:bash: ntpq: command not found 2024-04-12T06:00:04.565 INFO:teuthology.orchestra.run.smithi192.stderr:bash: ntpq: command not found 2024-04-12T06:00:04.569 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T06:00:04.569 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-04-12T06:00:04.569 INFO:teuthology.orchestra.run.smithi113.stdout:^* hv01.front.sepia.ceph.com 3 7 377 122 +1009us[+1009us] +/- 63ms 2024-04-12T06:00:04.569 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv02.front.sepia.ceph.com 3 8 377 64 -595us[ -595us] +/- 69ms 2024-04-12T06:00:04.569 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 63 -651us[ -651us] +/- 82ms 2024-04-12T06:00:04.569 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T06:00:04.571 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T06:00:04.571 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-04-12T06:00:04.571 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 +1040us[+1038us] +/- 63ms 2024-04-12T06:00:04.571 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 62 -553us[ -553us] +/- 69ms 2024-04-12T06:00:04.571 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 126 -576us[ -578us] +/- 81ms 2024-04-12T06:00:04.571 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T06:00:04.571 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-12T06:00:04.583 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-12T06:00:04.584 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-12T06:00:04.596 DEBUG:teuthology.orchestra.run.smithi113:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-12T06:00:04.685 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-12T06:00:04.752 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T06:00:04.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T06:00:04.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T06:00:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T06:00:04.813 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-12T06:00:04.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-12T06:00:04.835 INFO:teuthology.task.internal:Duration was 4234.357634 seconds 2024-04-12T06:00:04.835 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-12T06:00:04.846 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-12T06:00:04.847 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-12T06:00:04.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-12T06:00:04.892 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T06:00:04.899 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T06:00:05.178 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-12T06:00:05.179 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi113.front.sepia.ceph.com 2024-04-12T06:00:05.179 DEBUG:teuthology.orchestra.run.smithi113:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-12T06:00:05.202 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-04-12T06:00:05.202 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-12T06:00:05.224 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-12T06:00:05.224 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T06:00:05.244 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T06:00:05.327 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-12T06:00:05.327 DEBUG:teuthology.orchestra.run.smithi113:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-12T06:00:05.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-12T06:00:05.469 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-12T06:00:05.481 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-12T06:00:05.481 DEBUG:teuthology.orchestra.run.smithi113:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-12T06:00:05.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-12T06:00:05.536 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-12T06:00:05.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-12T06:00:05.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-12T06:00:05.576 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = core 2024-04-12T06:00:05.602 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-04-12T06:00:05.626 DEBUG:teuthology.orchestra.run.smithi113:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-12T06:00:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T06:00:05.664 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-12T06:00:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T06:00:05.683 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-12T06:00:05.695 INFO:teuthology.task.internal:Transferring archived files... 2024-04-12T06:00:05.696 DEBUG:teuthology.misc:Transferring archived files from smithi113:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627/remote/smithi113 2024-04-12T06:00:05.697 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-12T06:00:05.750 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627/remote/smithi192 2024-04-12T06:00:05.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-12T06:00:05.793 INFO:teuthology.task.internal:Removing archive directory... 2024-04-12T06:00:05.793 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-12T06:00:05.795 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-12T06:00:05.843 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-12T06:00:05.856 INFO:teuthology.task.internal:Not uploading archives. 2024-04-12T06:00:05.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-12T06:00:05.873 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-12T06:00:05.874 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-12T06:00:05.876 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-12T06:00:05.891 INFO:teuthology.orchestra.run.smithi113.stdout: 525357 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 12 06:00 /home/ubuntu/cephtest 2024-04-12T06:00:05.891 INFO:teuthology.orchestra.run.smithi113.stdout: 527564 4 d--------- 2 ubuntu ubuntu 4096 Apr 12 05:06 /home/ubuntu/cephtest/mnt.0 2024-04-12T06:00:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T06:00:05.892 INFO:teuthology.orchestra.run.smithi113.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-12T06:00:05.892 INFO:teuthology.orchestra.run.smithi113.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-12T06:00:05.893 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi113 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-12T06:00:05.893 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-12T06:00:05.913 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-12T06:00:05.929 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-12T06:00:05.930 INFO:teuthology.nuke:Checking targets against current locks 2024-04-12T06:00:05.956 DEBUG:teuthology.nuke:shortname: smithi113 2024-04-12T06:00:05.956 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T06:00:05.981 DEBUG:teuthology.nuke:shortname: smithi192 2024-04-12T06:00:05.981 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T06:00:05.984 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 04:40:26.943143', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOuzn2QoBu1JU0pSxKraKaKiJGW6M8Q2m/AYPeBQyjrZZlCjDV3uF0Ib9qe3ZfRBGzHzH5X/+dMe/TdBp3nWoGE='} 2024-04-12T06:00:06.008 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:32:36-orch-reef-release-distro-default-smithi/7652627', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 04:40:26.944967', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJci6ze4WnN5nmpQkZPWaN69kYBi6vSuQ0l7tFn5JES2IS0STsffymi0mVGzvcR+4IwSf1EK4nvAMDIrlmGqDfs='} 2024-04-12T06:00:06.034 INFO:teuthology.orchestra.console.smithi113:Power off 2024-04-12T06:00:06.035 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-12T06:00:06.061 DEBUG:teuthology.orchestra.console.smithi113:power off output: Chassis Power Control: Down/Off 2024-04-12T06:00:06.062 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:00:06.078 INFO:teuthology.orchestra.console.smithi192:Power off 2024-04-12T06:00:06.079 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-12T06:00:06.088 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-04-12T06:00:06.103 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2024-04-12T06:00:06.103 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:00:06.126 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-04-12T06:00:10.089 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:00:10.127 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:00:10.199 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-04-12T06:00:10.235 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-04-12T06:00:14.203 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:00:14.236 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T06:00:14.313 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is off 2024-04-12T06:00:14.346 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2024-04-12T06:00:14.415 INFO:teuthology.orchestra.console.smithi113:Power off completed 2024-04-12T06:00:14.448 INFO:teuthology.orchestra.console.smithi192:Power off completed 2024-04-12T06:00:14.552 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 4234.357634305954 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-04-12T06:00:14.553 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T06:00:14.625 INFO:teuthology.run:FAIL